2024-03-11T18:47:46.141 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-11T18:47:46.141 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} --verbose --name teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591250 -- /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591250/orig.config.yaml 2024-03-11T18:47:46.187 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T18:47:46.271 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591250 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591250' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 94a539707733325322964c6a187c6a1b41be1daf subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported dry_run: true install: ceph: flavor: default sha1: 94a539707733325322964c6a187c6a1b41be1daf kclient: client.0: mount_subvol_num: 1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: b6a4997be2b8bb74411268cc42753f29ae9a78ad owner: scheduled_teuthology@teuthology priority: 250 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8179 sha1: 94a539707733325322964c6a187c6a1b41be1daf sleep_before_teardown: 0 subset: 297/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: b6a4997be2b8bb74411268cc42753f29ae9a78ad syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGYZilMnOnggr1Y/gS2ZXlbZcuw//vXJ/iU1NMGtqA6Wd8bQYlifRUH//C0wNj5O5nuN56O4JOfysQGUkthuPV8= smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBETp9xZZU8hpJWpm3QSjIWV/gEDDHQ3IfwEvgofjAyO0IozeUhdwLNWbcmTLw4cRsz0hmpFImxcBetisVFs1014= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMS0O7LVYEEcLwSKh+rOHokIgBjlbuuLyRFdpoBkfP/KLb72O3NsTW8FVvwqQN0Yky+HYVXQZ/8iaJEUUrEva28= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - workunit: clients: all: - suites/iogen.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_14:02:35 tube: smithi user: teuthology verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-11T18:47:46.271 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa; will attempt to use it 2024-03-11T18:47:46.271 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa/tasks 2024-03-11T18:47:46.272 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-11T18:47:46.273 INFO:teuthology.task.internal:Checking packages... 2024-03-11T18:47:46.295 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '94a539707733325322964c6a187c6a1b41be1daf' 2024-03-11T18:47:46.295 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T18:47:46.295 INFO:teuthology.packaging:ref: None 2024-03-11T18:47:46.295 INFO:teuthology.packaging:tag: None 2024-03-11T18:47:46.295 INFO:teuthology.packaging:branch: squid 2024-03-11T18:47:46.295 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:47:46.295 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-11T18:47:46.455 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1607.g94a53970 2024-03-11T18:47:46.456 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-11T18:47:46.463 INFO:teuthology.task.internal:no buildpackages task found 2024-03-11T18:47:46.463 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-11T18:47:46.483 INFO:teuthology.task.internal:Saving configuration 2024-03-11T18:47:46.500 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-11T18:47:46.505 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T18:47:46.531 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591250', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 18:41:46.997298', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGYZilMnOnggr1Y/gS2ZXlbZcuw//vXJ/iU1NMGtqA6Wd8bQYlifRUH//C0wNj5O5nuN56O4JOfysQGUkthuPV8='} 2024-03-11T18:47:46.551 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591250', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 18:41:46.995827', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBETp9xZZU8hpJWpm3QSjIWV/gEDDHQ3IfwEvgofjAyO0IozeUhdwLNWbcmTLw4cRsz0hmpFImxcBetisVFs1014='} 2024-03-11T18:47:46.574 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591250', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 18:41:46.998544', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMS0O7LVYEEcLwSKh+rOHokIgBjlbuuLyRFdpoBkfP/KLb72O3NsTW8FVvwqQN0Yky+HYVXQZ/8iaJEUUrEva28='} 2024-03-11T18:47:46.574 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-11T18:47:46.579 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-11T18:47:46.579 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-11T18:47:46.579 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-11T18:47:46.579 INFO:teuthology.run_tasks:Running task console_log... 2024-03-11T18:47:46.687 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f590b4a2160>, signals=[15]) 2024-03-11T18:47:46.687 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-11T18:47:46.693 INFO:teuthology.task.internal:Opening connections... 2024-03-11T18:47:46.693 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2024-03-11T18:47:46.695 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:47:46.799 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2024-03-11T18:47:46.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:47:46.887 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2024-03-11T18:47:46.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:47:46.977 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-11T18:47:46.983 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-03-11T18:47:47.002 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-03-11T18:47:47.002 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-03-11T18:47:47.059 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2024-03-11T18:47:47.059 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="9" 2024-03-11T18:47:47.059 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2024-03-11T18:47:47.059 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2024-03-11T18:47:47.059 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="9" 2024-03-11T18:47:47.059 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el9" 2024-03-11T18:47:47.060 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T18:47:47.060 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2024-03-11T18:47:47.060 INFO:teuthology.orchestra.run.smithi029.stdout:LOGO="fedora-logo-icon" 2024-03-11T18:47:47.060 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T18:47:47.060 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2024-03-11T18:47:47.060 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T18:47:47.060 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T18:47:47.060 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T18:47:47.060 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-03-11T18:47:47.085 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2024-03-11T18:47:47.103 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2024-03-11T18:47:47.103 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2024-03-11T18:47:47.159 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream" 2024-03-11T18:47:47.159 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="9" 2024-03-11T18:47:47.159 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2024-03-11T18:47:47.159 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2024-03-11T18:47:47.159 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="9" 2024-03-11T18:47:47.159 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el9" 2024-03-11T18:47:47.159 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T18:47:47.160 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2024-03-11T18:47:47.160 INFO:teuthology.orchestra.run.smithi170.stdout:LOGO="fedora-logo-icon" 2024-03-11T18:47:47.160 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T18:47:47.160 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2024-03-11T18:47:47.160 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T18:47:47.160 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T18:47:47.160 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T18:47:47.160 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2024-03-11T18:47:47.185 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2024-03-11T18:47:47.203 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2024-03-11T18:47:47.203 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2024-03-11T18:47:47.260 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="CentOS Stream" 2024-03-11T18:47:47.260 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="9" 2024-03-11T18:47:47.260 INFO:teuthology.orchestra.run.smithi190.stdout:ID="centos" 2024-03-11T18:47:47.260 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="rhel fedora" 2024-03-11T18:47:47.260 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="9" 2024-03-11T18:47:47.260 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el9" 2024-03-11T18:47:47.260 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T18:47:47.260 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31" 2024-03-11T18:47:47.260 INFO:teuthology.orchestra.run.smithi190.stdout:LOGO="fedora-logo-icon" 2024-03-11T18:47:47.260 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T18:47:47.260 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://centos.org/" 2024-03-11T18:47:47.261 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T18:47:47.261 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T18:47:47.261 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T18:47:47.261 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2024-03-11T18:47:47.285 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-11T18:47:47.295 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-11T18:47:47.321 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-11T18:47:47.321 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2024-03-11T18:47:47.324 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2024-03-11T18:47:47.326 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2024-03-11T18:47:47.342 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-11T18:47:47.348 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-11T18:47:47.348 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2024-03-11T18:47:47.381 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2024-03-11T18:47:47.384 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2024-03-11T18:47:47.496 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-11T18:47:47.573 INFO:teuthology.run_tasks:Running task kernel... 2024-03-11T18:47:47.588 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-11T18:47:47.588 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-11T18:47:47.588 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T18:47:47.588 DEBUG:teuthology.orchestra.run.smithi170:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T18:47:47.589 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T18:47:47.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T18:47:47.605 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2024-03-11T18:47:47.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T18:47:47.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T18:47:47.607 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-03-11T18:47:47.608 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-03-11T18:47:47.663 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-361.el9.x86_64 2024-03-11T18:47:47.663 INFO:teuthology.task.kernel:Running kernel on smithi170: 5.14.0-361.el9.x86_64 2024-03-11T18:47:47.663 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum install -y kernel 2024-03-11T18:47:47.664 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-361.el9.x86_64 2024-03-11T18:47:47.664 INFO:teuthology.task.kernel:Running kernel on smithi190: 5.14.0-361.el9.x86_64 2024-03-11T18:47:47.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum install -y kernel 2024-03-11T18:47:47.665 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-361.el9.x86_64 2024-03-11T18:47:47.665 INFO:teuthology.task.kernel:Running kernel on smithi029: 5.14.0-361.el9.x86_64 2024-03-11T18:47:47.666 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2024-03-11T18:47:49.908 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-11T18:47:49.960 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 9.9 MB/s | 6.7 MB 00:00 2024-03-11T18:47:50.904 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-11T18:47:51.500 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-11T18:47:51.576 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-11T18:47:52.355 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-11T18:47:52.726 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 9.0 MB/s | 8.0 MB 00:00 2024-03-11T18:47:53.390 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-11T18:47:54.450 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-03-11T18:47:55.409 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-11T18:47:56.408 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.0 MB 00:04 2024-03-11T18:47:59.059 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-11T18:47:59.475 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.1 MB 00:01 2024-03-11T18:48:00.736 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.1 MB 00:01 2024-03-11T18:48:01.101 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 15 kB 00:00 2024-03-11T18:48:02.480 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-11T18:48:03.264 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 15 kB 00:01 2024-03-11T18:48:03.393 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 6.8 MB/s | 6.1 MB 00:00 2024-03-11T18:48:04.844 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 15 kB 00:00 2024-03-11T18:48:06.673 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-11T18:48:07.444 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-11T18:48:09.601 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T18:48:09.601 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T18:48:09.682 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-11T18:48:09.689 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-11T18:48:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-03-11T18:48:09.690 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-11T18:48:09.690 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-11T18:48:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T18:48:09.690 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-11T18:48:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T18:48:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T18:48:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T18:48:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:48:09.690 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-11T18:48:09.690 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-11T18:48:09.690 INFO:teuthology.orchestra.run.smithi190.stdout:Install 4 Packages 2024-03-11T18:48:09.690 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:48:09.691 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 100 M 2024-03-11T18:48:09.691 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 123 M 2024-03-11T18:48:09.691 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-11T18:48:11.712 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-11T18:48:13.844 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T18:48:13.844 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T18:48:13.932 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-11T18:48:13.940 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-11T18:48:13.940 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2024-03-11T18:48:13.940 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-11T18:48:13.940 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-11T18:48:13.940 INFO:teuthology.orchestra.run.smithi170.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T18:48:13.940 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-11T18:48:13.941 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T18:48:13.941 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T18:48:13.941 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T18:48:13.941 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:48:13.941 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-11T18:48:13.941 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-11T18:48:13.941 INFO:teuthology.orchestra.run.smithi170.stdout:Install 4 Packages 2024-03-11T18:48:13.941 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:48:13.941 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 100 M 2024-03-11T18:48:13.941 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 123 M 2024-03-11T18:48:13.941 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-11T18:48:15.530 INFO:teuthology.orchestra.run.smithi170.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 4.6 MB/s | 6.0 MB 00:01 2024-03-11T18:48:20.028 INFO:teuthology.orchestra.run.smithi190.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 602 kB/s | 6.0 MB 00:10 2024-03-11T18:48:23.129 INFO:teuthology.orchestra.run.smithi170.stdout:(2/4): kernel-modules-core-5.14.0-427.el9.x86_6 4.4 MB/s | 33 MB 00:07 2024-03-11T18:48:23.629 INFO:teuthology.orchestra.run.smithi170.stdout:(3/4): kernel-core-5.14.0-427.el9.x86_64.rpm 2.2 MB/s | 21 MB 00:09 2024-03-11T18:48:26.435 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 914 kB/s | 21 MB 00:23 2024-03-11T18:48:31.541 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-11T18:48:33.981 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T18:48:33.981 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T18:48:34.087 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-11T18:48:34.101 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-11T18:48:34.101 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-03-11T18:48:34.102 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-11T18:48:34.102 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-11T18:48:34.102 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T18:48:34.102 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-11T18:48:34.102 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T18:48:34.102 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T18:48:34.102 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T18:48:34.102 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:48:34.102 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-11T18:48:34.102 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-11T18:48:34.102 INFO:teuthology.orchestra.run.smithi029.stdout:Install 4 Packages 2024-03-11T18:48:34.102 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:48:34.103 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 100 M 2024-03-11T18:48:34.103 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 123 M 2024-03-11T18:48:34.103 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-11T18:48:34.704 INFO:teuthology.orchestra.run.smithi029.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 11 MB/s | 6.0 MB 00:00 2024-03-11T18:48:35.455 INFO:teuthology.orchestra.run.smithi029.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 16 MB/s | 21 MB 00:01 2024-03-11T18:48:36.488 INFO:teuthology.orchestra.run.smithi029.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 17 MB/s | 39 MB 00:02 2024-03-11T18:48:37.039 INFO:teuthology.orchestra.run.smithi029.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 14 MB/s | 33 MB 00:02 2024-03-11T18:48:37.039 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-11T18:48:37.039 INFO:teuthology.orchestra.run.smithi029.stdout:Total 34 MB/s | 100 MB 00:02 2024-03-11T18:48:37.461 INFO:teuthology.orchestra.run.smithi170.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 1.7 MB/s | 39 MB 00:23 2024-03-11T18:48:37.462 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-11T18:48:37.462 INFO:teuthology.orchestra.run.smithi170.stdout:Total 4.2 MB/s | 100 MB 00:23 2024-03-11T18:48:37.963 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-11T18:48:38.347 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-11T18:48:38.428 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-11T18:48:38.429 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-11T18:48:38.696 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-11T18:48:38.696 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-11T18:48:39.821 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-11T18:48:39.821 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-11T18:48:40.054 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-11T18:48:40.054 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-11T18:48:42.706 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-11T18:48:42.770 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-11T18:48:43.429 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T18:48:43.485 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T18:48:43.604 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T18:48:43.670 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T18:48:45.792 INFO:teuthology.orchestra.run.smithi190.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 594 kB/s | 21 MB 00:35 2024-03-11T18:48:46.392 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T18:48:46.583 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T18:48:46.602 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T18:48:46.888 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T18:48:51.404 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T18:48:51.526 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T18:48:52.016 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T18:48:52.164 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T18:48:56.307 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T18:48:57.318 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T18:49:21.437 INFO:teuthology.orchestra.run.smithi190.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 557 kB/s | 33 MB 01:01 2024-03-11T18:49:25.525 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T18:49:25.525 INFO:teuthology.orchestra.run.smithi170.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T18:49:25.526 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:49:25.610 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T18:49:26.073 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T18:49:26.073 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T18:49:26.073 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T18:49:26.073 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T18:49:26.518 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T18:49:26.519 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:49:26.519 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-11T18:49:26.519 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T18:49:26.519 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T18:49:26.519 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T18:49:26.519 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T18:49:26.519 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:49:26.519 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-11T18:49:27.035 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T18:49:27.178 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T18:49:27.179 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T18:49:27.179 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T18:49:27.179 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T18:49:27.179 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T18:49:27.180 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-11T18:49:27.180 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi170.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T18:49:27.180 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T18:49:27.303 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T18:49:27.303 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T18:49:27.303 DEBUG:teuthology.orchestra.run.smithi170:> sudo rpm -qi grub2-tools 2024-03-11T18:49:27.344 INFO:teuthology.orchestra.run.smithi170.stdout:Name : grub2-tools 2024-03-11T18:49:27.344 INFO:teuthology.orchestra.run.smithi170.stdout:Epoch : 1 2024-03-11T18:49:27.344 INFO:teuthology.orchestra.run.smithi170.stdout:Version : 2.06 2024-03-11T18:49:27.344 INFO:teuthology.orchestra.run.smithi170.stdout:Release : 68.el9 2024-03-11T18:49:27.344 INFO:teuthology.orchestra.run.smithi170.stdout:Architecture: x86_64 2024-03-11T18:49:27.344 INFO:teuthology.orchestra.run.smithi170.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T18:49:27.344 INFO:teuthology.orchestra.run.smithi170.stdout:Group : Unspecified 2024-03-11T18:49:27.344 INFO:teuthology.orchestra.run.smithi170.stdout:Size : 8264254 2024-03-11T18:49:27.344 INFO:teuthology.orchestra.run.smithi170.stdout:License : GPLv3+ 2024-03-11T18:49:27.344 INFO:teuthology.orchestra.run.smithi170.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T18:49:27.345 INFO:teuthology.orchestra.run.smithi170.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T18:49:27.345 INFO:teuthology.orchestra.run.smithi170.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T18:49:27.345 INFO:teuthology.orchestra.run.smithi170.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T18:49:27.345 INFO:teuthology.orchestra.run.smithi170.stdout:Packager : builder@centos.org 2024-03-11T18:49:27.345 INFO:teuthology.orchestra.run.smithi170.stdout:Vendor : CentOS 2024-03-11T18:49:27.345 INFO:teuthology.orchestra.run.smithi170.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T18:49:27.345 INFO:teuthology.orchestra.run.smithi170.stdout:Summary : Support tools for GRUB. 2024-03-11T18:49:27.345 INFO:teuthology.orchestra.run.smithi170.stdout:Description : 2024-03-11T18:49:27.345 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:49:27.345 INFO:teuthology.orchestra.run.smithi170.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T18:49:27.345 INFO:teuthology.orchestra.run.smithi170.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T18:49:27.345 INFO:teuthology.orchestra.run.smithi170.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T18:49:27.345 INFO:teuthology.orchestra.run.smithi170.stdout:hardware devices. 2024-03-11T18:49:27.345 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:49:27.346 INFO:teuthology.orchestra.run.smithi170.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T18:49:27.346 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T18:49:27.346 INFO:teuthology.task.kernel:Updating grub on smithi170 to boot 5.14.0-427.el9.x86_64 2024-03-11T18:49:27.346 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T18:49:27.964 INFO:teuthology.orchestra.run.smithi170.stderr:Generating grub configuration file ... 2024-03-11T18:49:28.828 INFO:teuthology.orchestra.run.smithi190.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 510 kB/s | 39 MB 01:18 2024-03-11T18:49:28.828 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-11T18:49:28.828 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.3 MB/s | 100 MB 01:19 2024-03-11T18:49:29.372 INFO:teuthology.orchestra.run.smithi170.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T18:49:29.400 INFO:teuthology.orchestra.run.smithi170.stderr:done 2024-03-11T18:49:29.403 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2024-03-11T18:49:29.417 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.fSunXLFsKy 2024-03-11T18:49:29.418 DEBUG:teuthology.orchestra.run.smithi170:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fSunXLFsKy 2024-03-11T18:49:29.486 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.fSunXLFsKy 2024-03-11T18:49:29.622 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.fSunXLFsKy is 6KB 2024-03-11T18:49:29.640 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.fSunXLFsKy 2024-03-11T18:49:29.654 DEBUG:teuthology.orchestra.run.smithi170:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T18:49:29.715 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-11T18:49:29.718 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T18:49:29.718 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T18:49:29.718 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T18:49:29.719 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T18:49:29.720 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T18:49:29.881 DEBUG:teuthology.orchestra.run.smithi170:> sudo shutdown -r now 2024-03-11T18:49:30.093 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-11T18:49:30.093 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-11T18:49:30.332 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T18:49:30.332 INFO:teuthology.orchestra.run.smithi029.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T18:49:30.332 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:49:30.420 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T18:49:30.853 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T18:49:30.853 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T18:49:30.853 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T18:49:30.853 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T18:49:31.298 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T18:49:31.299 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:49:31.299 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-11T18:49:31.299 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T18:49:31.299 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T18:49:31.299 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T18:49:31.299 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T18:49:31.299 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:49:31.299 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-11T18:49:31.716 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-11T18:49:31.716 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-11T18:49:32.049 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T18:49:32.244 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T18:49:32.244 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T18:49:32.244 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T18:49:32.244 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T18:49:32.245 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T18:49:32.245 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-11T18:49:32.245 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T18:49:32.245 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T18:49:32.370 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T18:49:32.371 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T18:49:32.371 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2024-03-11T18:49:32.414 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2024-03-11T18:49:32.414 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2024-03-11T18:49:32.414 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.06 2024-03-11T18:49:32.414 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 68.el9 2024-03-11T18:49:32.414 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2024-03-11T18:49:32.414 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T18:49:32.414 INFO:teuthology.orchestra.run.smithi029.stdout:Group : Unspecified 2024-03-11T18:49:32.414 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 8264254 2024-03-11T18:49:32.415 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2024-03-11T18:49:32.415 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T18:49:32.415 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T18:49:32.415 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T18:49:32.415 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T18:49:32.415 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : builder@centos.org 2024-03-11T18:49:32.415 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2024-03-11T18:49:32.415 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T18:49:32.415 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2024-03-11T18:49:32.415 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2024-03-11T18:49:32.415 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:49:32.415 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T18:49:32.415 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T18:49:32.415 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T18:49:32.415 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2024-03-11T18:49:32.415 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:49:32.416 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T18:49:32.416 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T18:49:32.416 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 5.14.0-427.el9.x86_64 2024-03-11T18:49:32.416 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T18:49:33.242 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2024-03-11T18:49:34.432 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-11T18:49:34.861 INFO:teuthology.orchestra.run.smithi029.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T18:49:34.886 INFO:teuthology.orchestra.run.smithi029.stderr:done 2024-03-11T18:49:34.888 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-03-11T18:49:34.903 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.s2FLAVit7j 2024-03-11T18:49:34.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.s2FLAVit7j 2024-03-11T18:49:34.974 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.s2FLAVit7j 2024-03-11T18:49:35.087 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T18:49:35.116 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.s2FLAVit7j is 6KB 2024-03-11T18:49:35.166 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.s2FLAVit7j 2024-03-11T18:49:35.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T18:49:35.228 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T18:49:35.255 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T18:49:35.255 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T18:49:35.255 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T18:49:35.255 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T18:49:35.256 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T18:49:35.424 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2024-03-11T18:49:38.006 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T18:49:38.189 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T18:49:42.884 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T18:49:43.124 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T18:49:47.860 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T18:49:59.884 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-11T18:49:59.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-11T18:49:59.885 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:50:05.426 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-11T18:50:05.426 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-11T18:50:05.427 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:50:16.931 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T18:50:16.932 INFO:teuthology.orchestra.run.smithi190.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T18:50:16.932 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:50:17.016 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T18:50:17.480 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T18:50:17.481 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T18:50:17.481 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T18:50:17.481 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T18:50:17.914 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T18:50:17.914 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:50:17.915 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-11T18:50:17.915 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T18:50:17.915 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T18:50:17.915 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T18:50:17.915 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T18:50:17.915 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:50:17.915 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-11T18:50:18.463 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T18:50:18.604 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T18:50:18.605 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T18:50:18.605 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T18:50:18.605 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T18:50:18.605 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T18:50:18.605 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-11T18:50:18.605 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi190.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T18:50:18.605 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T18:50:18.726 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T18:50:18.727 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T18:50:18.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo rpm -qi grub2-tools 2024-03-11T18:50:18.771 INFO:teuthology.orchestra.run.smithi190.stdout:Name : grub2-tools 2024-03-11T18:50:18.771 INFO:teuthology.orchestra.run.smithi190.stdout:Epoch : 1 2024-03-11T18:50:18.771 INFO:teuthology.orchestra.run.smithi190.stdout:Version : 2.06 2024-03-11T18:50:18.771 INFO:teuthology.orchestra.run.smithi190.stdout:Release : 68.el9 2024-03-11T18:50:18.772 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: x86_64 2024-03-11T18:50:18.772 INFO:teuthology.orchestra.run.smithi190.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T18:50:18.772 INFO:teuthology.orchestra.run.smithi190.stdout:Group : Unspecified 2024-03-11T18:50:18.772 INFO:teuthology.orchestra.run.smithi190.stdout:Size : 8264254 2024-03-11T18:50:18.772 INFO:teuthology.orchestra.run.smithi190.stdout:License : GPLv3+ 2024-03-11T18:50:18.772 INFO:teuthology.orchestra.run.smithi190.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T18:50:18.772 INFO:teuthology.orchestra.run.smithi190.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T18:50:18.772 INFO:teuthology.orchestra.run.smithi190.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T18:50:18.772 INFO:teuthology.orchestra.run.smithi190.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T18:50:18.772 INFO:teuthology.orchestra.run.smithi190.stdout:Packager : builder@centos.org 2024-03-11T18:50:18.772 INFO:teuthology.orchestra.run.smithi190.stdout:Vendor : CentOS 2024-03-11T18:50:18.772 INFO:teuthology.orchestra.run.smithi190.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T18:50:18.772 INFO:teuthology.orchestra.run.smithi190.stdout:Summary : Support tools for GRUB. 2024-03-11T18:50:18.772 INFO:teuthology.orchestra.run.smithi190.stdout:Description : 2024-03-11T18:50:18.772 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:50:18.772 INFO:teuthology.orchestra.run.smithi190.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T18:50:18.773 INFO:teuthology.orchestra.run.smithi190.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T18:50:18.773 INFO:teuthology.orchestra.run.smithi190.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T18:50:18.773 INFO:teuthology.orchestra.run.smithi190.stdout:hardware devices. 2024-03-11T18:50:18.773 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:50:18.773 INFO:teuthology.orchestra.run.smithi190.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T18:50:18.773 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T18:50:18.774 INFO:teuthology.task.kernel:Updating grub on smithi190 to boot 5.14.0-427.el9.x86_64 2024-03-11T18:50:18.774 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T18:50:19.342 INFO:teuthology.orchestra.run.smithi190.stderr:Generating grub configuration file ... 2024-03-11T18:50:20.762 INFO:teuthology.orchestra.run.smithi190.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T18:50:20.797 INFO:teuthology.orchestra.run.smithi190.stderr:done 2024-03-11T18:50:20.799 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-03-11T18:50:20.814 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.ZYXISNBlhX 2024-03-11T18:50:20.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ZYXISNBlhX 2024-03-11T18:50:20.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.ZYXISNBlhX 2024-03-11T18:50:21.034 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.ZYXISNBlhX is 6KB 2024-03-11T18:50:21.082 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.ZYXISNBlhX 2024-03-11T18:50:21.096 DEBUG:teuthology.orchestra.run.smithi190:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T18:50:21.162 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T18:50:21.162 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T18:50:21.163 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T18:50:21.163 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T18:50:21.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T18:50:21.319 DEBUG:teuthology.orchestra.run.smithi190:> sudo shutdown -r now 2024-03-11T18:50:51.350 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-11T18:50:51.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-11T18:50:51.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:50:59.892 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T18:51:05.432 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T18:51:08.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-11T18:51:08.895 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:51:09.726 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-03-11T18:51:12.334 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-03-11T18:51:12.747 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-11T18:51:12.747 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-11T18:51:12.747 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2024-03-11T18:51:12.761 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-427.el9.x86_64 2024-03-11T18:51:12.761 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T18:51:12.761 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T18:51:12.761 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T18:51:13.762 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-11T18:51:13.762 DEBUG:teuthology.orchestra.run.smithi170:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T18:51:13.880 INFO:teuthology.orchestra.run.smithi170.stdout:ttyS1 2024-03-11T18:51:13.922 DEBUG:teuthology.parallel:result is None 2024-03-11T18:51:14.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-11T18:51:14.433 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:51:18.062 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-03-11T18:51:18.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-11T18:51:18.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:51:18.927 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-11T18:51:18.927 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-11T18:51:18.927 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-03-11T18:51:18.941 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-427.el9.x86_64 2024-03-11T18:51:18.941 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T18:51:18.941 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T18:51:18.942 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T18:51:19.942 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-11T18:51:19.943 DEBUG:teuthology.orchestra.run.smithi029:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T18:51:20.066 INFO:teuthology.orchestra.run.smithi029.stdout:ttyS1 2024-03-11T18:51:20.113 DEBUG:teuthology.parallel:result is None 2024-03-11T18:51:21.794 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-03-11T18:51:33.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-11T18:51:33.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:51:52.222 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-03-11T18:52:07.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-11T18:52:07.225 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:52:07.649 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-03-11T18:52:08.014 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-11T18:52:08.014 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-427.el9.x86_64"... 2024-03-11T18:52:08.015 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-03-11T18:52:08.028 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-427.el9.x86_64 2024-03-11T18:52:08.029 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T18:52:08.029 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T18:52:08.029 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T18:52:09.029 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-11T18:52:09.030 DEBUG:teuthology.orchestra.run.smithi190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T18:52:09.092 INFO:teuthology.orchestra.run.smithi190.stdout:ttyS1 2024-03-11T18:52:09.132 DEBUG:teuthology.parallel:result is None 2024-03-11T18:52:09.133 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-11T18:52:09.140 INFO:teuthology.task.internal:Creating test directory... 2024-03-11T18:52:09.140 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T18:52:09.143 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T18:52:09.146 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T18:52:09.162 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-11T18:52:09.170 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-11T18:52:09.179 INFO:teuthology.task.internal:Creating archive directory... 2024-03-11T18:52:09.179 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T18:52:09.202 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T18:52:09.205 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T18:52:09.280 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-11T18:52:09.288 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-11T18:52:09.288 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T18:52:09.291 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T18:52:09.293 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T18:52:09.329 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T18:52:09.331 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T18:52:09.342 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T18:52:09.345 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T18:52:09.353 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T18:52:09.367 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T18:52:09.369 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-11T18:52:09.377 INFO:teuthology.task.internal:Configuring sudo... 2024-03-11T18:52:09.377 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T18:52:09.389 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T18:52:09.392 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T18:52:09.419 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-11T18:52:09.428 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-11T18:52:09.428 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T18:52:09.459 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T18:52:09.462 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T18:52:09.476 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T18:52:09.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T18:52:09.610 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T18:52:09.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T18:52:09.736 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-11T18:52:09.736 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T18:52:09.803 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T18:52:09.847 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T18:52:09.902 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T18:52:09.989 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T18:52:10.032 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-11T18:52:10.032 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T18:52:10.101 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T18:52:10.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T18:52:10.187 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T18:52:10.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T18:52:10.304 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-11T18:52:10.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T18:52:10.367 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2024-03-11T18:52:10.370 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2024-03-11T18:52:10.372 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2024-03-11T18:52:10.421 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T18:52:10.428 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T18:52:10.459 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T18:52:10.885 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-11T18:52:10.895 INFO:teuthology.task.internal:Starting timer... 2024-03-11T18:52:10.895 INFO:teuthology.run_tasks:Running task pcp... 2024-03-11T18:52:10.912 INFO:teuthology.run_tasks:Running task selinux... 2024-03-11T18:52:10.926 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-11T18:52:10.927 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2024-03-11T18:52:11.022 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2024-03-11T18:52:11.025 DEBUG:teuthology.orchestra.run.smithi170:> sudo service auditd rotate 2024-03-11T18:52:11.108 INFO:teuthology.orchestra.run.smithi170.stdout:Rotating logs: 2024-03-11T18:52:11.110 DEBUG:teuthology.orchestra.run.smithi190:> sudo service auditd rotate 2024-03-11T18:52:11.151 INFO:teuthology.orchestra.run.smithi190.stdout:Rotating logs: 2024-03-11T18:52:11.153 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-11T18:52:11.153 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2024-03-11T18:52:11.175 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2024-03-11T18:52:11.176 DEBUG:teuthology.orchestra.run.smithi170:> /usr/sbin/getenforce 2024-03-11T18:52:11.201 INFO:teuthology.orchestra.run.smithi170.stdout:Permissive 2024-03-11T18:52:11.201 DEBUG:teuthology.orchestra.run.smithi190:> /usr/sbin/getenforce 2024-03-11T18:52:11.278 INFO:teuthology.orchestra.run.smithi190.stdout:Permissive 2024-03-11T18:52:11.279 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi170.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi190.front.sepia.ceph.com': 'permissive'} 2024-03-11T18:52:11.279 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T18:52:11.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T18:52:11.309 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T18:52:11.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T18:52:11.342 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T18:52:11.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T18:52:11.370 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-11T18:52:11.370 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2024-03-11T18:52:11.410 DEBUG:teuthology.orchestra.run.smithi170:> sudo /usr/sbin/setenforce permissive 2024-03-11T18:52:11.444 DEBUG:teuthology.orchestra.run.smithi190:> sudo /usr/sbin/setenforce permissive 2024-03-11T18:52:11.497 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-11T18:52:11.508 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-11T18:52:11.527 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-11T18:52:11.528 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi170.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-11T18:56:39.088 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi170.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2024-03-11T18:56:39.090 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-11T18:56:39.091 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:56:39.180 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-03-11T18:56:39.268 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-11T18:56:39.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-11T18:56:39.269 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:56:39.359 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-03-11T18:56:39.441 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-11T18:56:39.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-11T18:56:39.442 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:56:39.536 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-03-11T18:56:39.617 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-11T18:56:39.617 INFO:teuthology.run_tasks:Running task clock... 2024-03-11T18:56:39.628 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-11T18:56:39.629 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T18:56:39.629 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T18:56:39.631 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T18:56:39.631 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T18:56:39.634 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T18:56:39.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T18:56:39.661 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T18:56:39.665 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T18:56:39.678 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T18:56:39.682 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T18:56:39.689 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T18:56:39.706 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T18:56:39.707 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2024-03-11T18:56:39.712 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: ntpd: command not found 2024-03-11T18:56:39.721 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2024-03-11T18:56:39.726 INFO:teuthology.orchestra.run.smithi170.stdout:506 Cannot talk to daemon 2024-03-11T18:56:39.735 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: ntpd: command not found 2024-03-11T18:56:39.738 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T18:56:39.742 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T18:56:39.749 INFO:teuthology.orchestra.run.smithi190.stdout:506 Cannot talk to daemon 2024-03-11T18:56:39.754 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T18:56:39.758 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T18:56:39.765 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T18:56:39.781 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T18:56:39.803 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-03-11T18:56:39.806 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T18:56:39.806 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-03-11T18:56:39.806 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:56:39.806 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:56:39.806 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:56:39.807 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:56:39.809 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-03-11T18:56:39.813 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T18:56:39.813 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-03-11T18:56:39.813 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:56:39.813 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:56:39.813 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:56:39.813 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:56:39.842 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-03-11T18:56:39.845 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T18:56:39.845 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-03-11T18:56:39.845 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:56:39.846 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:56:39.846 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:56:39.846 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:56:39.846 INFO:teuthology.run_tasks:Running task install... 2024-03-11T18:56:39.857 DEBUG:teuthology.task.install:project ceph 2024-03-11T18:56:39.857 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'}} 2024-03-11T18:56:39.857 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'} 2024-03-11T18:56:39.857 INFO:teuthology.task.install:Using flavor: default 2024-03-11T18:56:39.864 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-11T18:56:39.864 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-11T18:56:39.865 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'tag': None, 'wait_for_package': False} 2024-03-11T18:56:39.865 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:56:39.867 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'tag': None, 'wait_for_package': False} 2024-03-11T18:56:39.867 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:56:39.868 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'tag': None, 'wait_for_package': False} 2024-03-11T18:56:39.868 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:56:40.015 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/ 2024-03-11T18:56:40.016 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1607.g94a53970 2024-03-11T18:56:40.136 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/ 2024-03-11T18:56:40.136 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1607.g94a53970 2024-03-11T18:56:40.140 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/ 2024-03-11T18:56:40.140 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1607.g94a53970 2024-03-11T18:56:40.202 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-11T18:56:40.202 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-11T18:56:40.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-11T18:56:40.230 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-11T18:56:40.231 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/94a539707733325322964c6a187c6a1b41be1daf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-11T18:56:40.299 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-11T18:56:40.338 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-11T18:56:40.338 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-11T18:56:40.338 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-11T18:56:40.346 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-11T18:56:40.347 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-11T18:56:40.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-11T18:56:40.368 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-11T18:56:40.368 DEBUG:teuthology.orchestra.run.smithi170:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/94a539707733325322964c6a187c6a1b41be1daf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-11T18:56:40.375 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-11T18:56:40.375 DEBUG:teuthology.orchestra.run.smithi190:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/94a539707733325322964c6a187c6a1b41be1daf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-11T18:56:40.379 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-11T18:56:40.437 DEBUG:teuthology.orchestra.run.smithi170:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-11T18:56:40.443 DEBUG:teuthology.orchestra.run.smithi190:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-11T18:56:40.445 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2024-03-11T18:56:40.446 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2024-03-11T18:56:40.517 DEBUG:teuthology.orchestra.run.smithi170:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-11T18:56:40.522 DEBUG:teuthology.orchestra.run.smithi190:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-11T18:56:40.584 INFO:teuthology.orchestra.run.smithi170.stdout:check_obsoletes = 1 2024-03-11T18:56:40.585 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum clean all 2024-03-11T18:56:40.588 INFO:teuthology.orchestra.run.smithi190.stdout:check_obsoletes = 1 2024-03-11T18:56:40.590 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum clean all 2024-03-11T18:56:40.697 INFO:teuthology.orchestra.run.smithi029.stdout:74 files removed 2024-03-11T18:56:40.724 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-11T18:56:40.820 INFO:teuthology.orchestra.run.smithi170.stdout:74 files removed 2024-03-11T18:56:40.822 INFO:teuthology.orchestra.run.smithi190.stdout:74 files removed 2024-03-11T18:56:40.847 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-11T18:56:40.849 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-11T18:56:41.484 INFO:teuthology.orchestra.run.smithi170.stdout:ceph packages for x86_64 258 kB/s | 82 kB 00:00 2024-03-11T18:56:41.769 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 113 kB/s | 82 kB 00:00 2024-03-11T18:56:41.793 INFO:teuthology.orchestra.run.smithi170.stdout:ceph noarch packages 39 kB/s | 10 kB 00:00 2024-03-11T18:56:41.798 INFO:teuthology.orchestra.run.smithi190.stdout:ceph packages for x86_64 130 kB/s | 82 kB 00:00 2024-03-11T18:56:42.136 INFO:teuthology.orchestra.run.smithi170.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-03-11T18:56:42.220 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-11T18:56:42.232 INFO:teuthology.orchestra.run.smithi190.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-11T18:56:42.579 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-03-11T18:56:42.699 INFO:teuthology.orchestra.run.smithi190.stdout:ceph source packages 4.4 kB/s | 1.9 kB 00:00 2024-03-11T18:56:42.890 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 9.3 MB/s | 6.7 MB 00:00 2024-03-11T18:56:43.367 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 8.9 MB/s | 6.7 MB 00:00 2024-03-11T18:56:43.401 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-11T18:56:44.340 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2024-03-11T18:56:44.902 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-11T18:56:44.917 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 5.0 MB/s | 2.0 MB 00:00 2024-03-11T18:56:45.827 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-11T18:56:46.668 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 8.0 MB 00:01 2024-03-11T18:56:48.739 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.0 MB 00:04 2024-03-11T18:56:51.369 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 5.2 MB/s | 19 MB 00:03 2024-03-11T18:56:52.648 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 6.6 MB/s | 19 MB 00:02 2024-03-11T18:56:56.438 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.1 MB 00:01 2024-03-11T18:56:56.590 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-11T18:56:56.821 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 1.9 MB/s | 19 MB 00:09 2024-03-11T18:56:57.975 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 15 kB 00:00 2024-03-11T18:56:58.004 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 15 kB 00:00 2024-03-11T18:56:59.710 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-11T18:57:00.598 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 8.3 MB/s | 21 MB 00:02 2024-03-11T18:57:01.273 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 7.5 MB/s | 6.1 MB 00:00 2024-03-11T18:57:02.752 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 15 kB 00:00 2024-03-11T18:57:04.618 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-11T18:57:04.649 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-11T18:57:05.487 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-11T18:57:07.154 INFO:teuthology.orchestra.run.smithi170.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-11T18:57:07.155 INFO:teuthology.orchestra.run.smithi170.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-11T18:57:07.167 INFO:teuthology.orchestra.run.smithi170.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-11T18:57:07.168 INFO:teuthology.orchestra.run.smithi170.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-11T18:57:07.169 INFO:teuthology.orchestra.run.smithi170.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-11T18:57:07.170 INFO:teuthology.orchestra.run.smithi170.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-11T18:57:07.171 INFO:teuthology.orchestra.run.smithi170.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-11T18:57:07.174 INFO:teuthology.orchestra.run.smithi170.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-11T18:57:07.175 INFO:teuthology.orchestra.run.smithi170.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-11T18:57:07.254 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-11T18:57:07.281 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-03-11T18:57:07.282 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-11T18:57:07.282 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-03-11T18:57:07.282 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-11T18:57:07.282 INFO:teuthology.orchestra.run.smithi170.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-11T18:57:07.282 INFO:teuthology.orchestra.run.smithi170.stdout: ceph x86_64 2:19.0.0-1607.g94a53970.el9 ceph 6.6 k 2024-03-11T18:57:07.282 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.3 M 2024-03-11T18:57:07.282 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 886 k 2024-03-11T18:57:07.285 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1607.g94a53970.el9 ceph 145 k 2024-03-11T18:57:07.285 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr x86_64 2:19.0.0-1607.g94a53970.el9 ceph 1.5 M 2024-03-11T18:57:07.285 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 138 k 2024-03-11T18:57:07.285 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 1.8 M 2024-03-11T18:57:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 7.4 M 2024-03-11T18:57:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 49 k 2024-03-11T18:57:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 9.8 M 2024-03-11T18:57:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test x86_64 2:19.0.0-1607.g94a53970.el9 ceph 47 M 2024-03-11T18:57:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 264 k 2024-03-11T18:57:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 748 k 2024-03-11T18:57:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 217 k 2024-03-11T18:57:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-shell x86_64 2:19.0.0-1607.g94a53970.el9 ceph 27 k 2024-03-11T18:57:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-top noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 18 k 2024-03-11T18:57:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-11T18:57:07.286 INFO:teuthology.orchestra.run.smithi170.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-11T18:57:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-11T18:57:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 31 k 2024-03-11T18:57:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 742 k 2024-03-11T18:57:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 127 k 2024-03-11T18:57:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-11T18:57:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-11T18:57:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-11T18:57:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-11T18:57:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-11T18:57:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-11T18:57:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs x86_64 2:19.0.0-1607.g94a53970.el9 ceph 161 k 2024-03-11T18:57:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados x86_64 2:19.0.0-1607.g94a53970.el9 ceph 320 k 2024-03-11T18:57:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 298 k 2024-03-11T18:57:07.287 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 99 k 2024-03-11T18:57:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 85 k 2024-03-11T18:57:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T18:57:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 171 k 2024-03-11T18:57:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-11T18:57:07.288 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrading: 2024-03-11T18:57:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-11T18:57:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-11T18:57:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-11T18:57:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-11T18:57:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-11T18:57:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-11T18:57:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-11T18:57:07.288 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-11T18:57:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-11T18:57:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-11T18:57:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-11T18:57:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-11T18:57:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-11T18:57:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: librados2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.4 M 2024-03-11T18:57:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T18:57:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-11T18:57:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-11T18:57:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-11T18:57:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-11T18:57:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-11T18:57:07.289 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-11T18:57:07.290 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-11T18:57:07.290 INFO:teuthology.orchestra.run.smithi170.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-11T18:57:07.290 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-11T18:57:07.290 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-11T18:57:07.290 INFO:teuthology.orchestra.run.smithi170.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-11T18:57:07.290 INFO:teuthology.orchestra.run.smithi170.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-11T18:57:07.290 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-11T18:57:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 20 M 2024-03-11T18:57:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 31 k 2024-03-11T18:57:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds x86_64 2:19.0.0-1607.g94a53970.el9 ceph 2.1 M 2024-03-11T18:57:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 247 k 2024-03-11T18:57:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon x86_64 2:19.0.0-1607.g94a53970.el9 ceph 4.7 M 2024-03-11T18:57:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 17 M 2024-03-11T18:57:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 15 k 2024-03-11T18:57:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux x86_64 2:19.0.0-1607.g94a53970.el9 ceph 25 k 2024-03-11T18:57:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-11T18:57:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-11T18:57:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-11T18:57:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-11T18:57:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-11T18:57:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-11T18:57:07.291 INFO:teuthology.orchestra.run.smithi170.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-11T18:57:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-11T18:57:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-11T18:57:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite x86_64 2:19.0.0-1607.g94a53970.el9 ceph 163 k 2024-03-11T18:57:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-11T18:57:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-11T18:57:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-11T18:57:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-11T18:57:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-11T18:57:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 498 k 2024-03-11T18:57:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-11T18:57:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.1 M 2024-03-11T18:57:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-11T18:57:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-11T18:57:07.292 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-11T18:57:07.293 INFO:teuthology.orchestra.run.smithi170.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-11T18:57:07.293 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-11T18:57:07.293 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-11T18:57:07.293 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-11T18:57:07.293 INFO:teuthology.orchestra.run.smithi170.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-11T18:57:07.293 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-11T18:57:07.293 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-11T18:57:07.293 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-11T18:57:07.293 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-11T18:57:07.293 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-11T18:57:07.293 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-11T18:57:07.293 INFO:teuthology.orchestra.run.smithi170.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-11T18:57:07.293 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-11T18:57:07.294 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-11T18:57:07.294 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-11T18:57:07.294 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 45 k 2024-03-11T18:57:07.294 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 131 k 2024-03-11T18:57:07.294 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-11T18:57:07.294 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-11T18:57:07.294 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-11T18:57:07.294 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-11T18:57:07.294 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-11T18:57:07.294 INFO:teuthology.orchestra.run.smithi170.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-11T18:57:07.294 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-11T18:57:07.294 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-11T18:57:07.294 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-11T18:57:07.294 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-11T18:57:07.295 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-11T18:57:07.295 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-11T18:57:07.295 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-11T18:57:07.295 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-11T18:57:07.295 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-11T18:57:07.295 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-11T18:57:07.295 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-11T18:57:07.295 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-11T18:57:07.295 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-11T18:57:07.295 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-11T18:57:07.295 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-11T18:57:07.295 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-11T18:57:07.295 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-11T18:57:07.296 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-11T18:57:07.296 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-11T18:57:07.296 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-11T18:57:07.296 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-11T18:57:07.296 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-11T18:57:07.296 INFO:teuthology.orchestra.run.smithi170.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-11T18:57:07.296 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-11T18:57:07.296 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-11T18:57:07.296 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-11T18:57:07.296 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-11T18:57:07.296 INFO:teuthology.orchestra.run.smithi170.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-11T18:57:07.296 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-11T18:57:07.296 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-11T18:57:07.296 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-11T18:57:07.297 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-11T18:57:07.297 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-11T18:57:07.297 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-11T18:57:07.297 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-11T18:57:07.297 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-11T18:57:07.297 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-11T18:57:07.297 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-11T18:57:07.297 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-11T18:57:07.297 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-11T18:57:07.297 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-11T18:57:07.297 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-11T18:57:07.297 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-11T18:57:07.297 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-11T18:57:07.297 INFO:teuthology.orchestra.run.smithi170.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-11T18:57:07.298 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-11T18:57:07.298 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-11T18:57:07.298 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-11T18:57:07.298 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-11T18:57:07.298 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-11T18:57:07.298 INFO:teuthology.orchestra.run.smithi170.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-11T18:57:07.298 INFO:teuthology.orchestra.run.smithi170.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-11T18:57:07.298 INFO:teuthology.orchestra.run.smithi170.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-11T18:57:07.298 INFO:teuthology.orchestra.run.smithi170.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-11T18:57:07.298 INFO:teuthology.orchestra.run.smithi170.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-11T18:57:07.298 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-11T18:57:07.298 INFO:teuthology.orchestra.run.smithi170.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-11T18:57:07.298 INFO:teuthology.orchestra.run.smithi170.stdout:Installing weak dependencies: 2024-03-11T18:57:07.299 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-11T18:57:07.299 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:57:07.299 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-11T18:57:07.299 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-03-11T18:57:07.299 INFO:teuthology.orchestra.run.smithi170.stdout:Install 150 Packages 2024-03-11T18:57:07.299 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrade 24 Packages 2024-03-11T18:57:07.299 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:57:07.300 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 223 M 2024-03-11T18:57:07.300 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-11T18:57:07.982 INFO:teuthology.orchestra.run.smithi190.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-11T18:57:07.982 INFO:teuthology.orchestra.run.smithi190.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-11T18:57:07.994 INFO:teuthology.orchestra.run.smithi190.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-11T18:57:07.995 INFO:teuthology.orchestra.run.smithi190.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-11T18:57:07.996 INFO:teuthology.orchestra.run.smithi190.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-11T18:57:07.997 INFO:teuthology.orchestra.run.smithi190.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-11T18:57:07.999 INFO:teuthology.orchestra.run.smithi190.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-11T18:57:08.001 INFO:teuthology.orchestra.run.smithi190.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-11T18:57:08.002 INFO:teuthology.orchestra.run.smithi190.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-11T18:57:08.081 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-11T18:57:08.101 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================ 2024-03-11T18:57:08.102 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-11T18:57:08.102 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================ 2024-03-11T18:57:08.102 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-11T18:57:08.102 INFO:teuthology.orchestra.run.smithi190.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-11T18:57:08.102 INFO:teuthology.orchestra.run.smithi190.stdout: ceph x86_64 2:19.0.0-1607.g94a53970.el9 ceph 6.6 k 2024-03-11T18:57:08.102 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.3 M 2024-03-11T18:57:08.102 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 886 k 2024-03-11T18:57:08.102 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1607.g94a53970.el9 ceph 145 k 2024-03-11T18:57:08.102 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr x86_64 2:19.0.0-1607.g94a53970.el9 ceph 1.5 M 2024-03-11T18:57:08.102 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 138 k 2024-03-11T18:57:08.102 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 1.8 M 2024-03-11T18:57:08.102 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 7.4 M 2024-03-11T18:57:08.102 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 49 k 2024-03-11T18:57:08.103 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 9.8 M 2024-03-11T18:57:08.103 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test x86_64 2:19.0.0-1607.g94a53970.el9 ceph 47 M 2024-03-11T18:57:08.103 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 264 k 2024-03-11T18:57:08.103 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 748 k 2024-03-11T18:57:08.103 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 217 k 2024-03-11T18:57:08.103 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-shell x86_64 2:19.0.0-1607.g94a53970.el9 ceph 27 k 2024-03-11T18:57:08.103 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-top noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 18 k 2024-03-11T18:57:08.103 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-11T18:57:08.103 INFO:teuthology.orchestra.run.smithi190.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-11T18:57:08.103 INFO:teuthology.orchestra.run.smithi190.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-11T18:57:08.103 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 31 k 2024-03-11T18:57:08.103 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 742 k 2024-03-11T18:57:08.103 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 127 k 2024-03-11T18:57:08.103 INFO:teuthology.orchestra.run.smithi190.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-11T18:57:08.104 INFO:teuthology.orchestra.run.smithi190.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-11T18:57:08.104 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-11T18:57:08.104 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-11T18:57:08.104 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-11T18:57:08.104 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-11T18:57:08.104 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs x86_64 2:19.0.0-1607.g94a53970.el9 ceph 161 k 2024-03-11T18:57:08.104 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados x86_64 2:19.0.0-1607.g94a53970.el9 ceph 320 k 2024-03-11T18:57:08.104 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 298 k 2024-03-11T18:57:08.104 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 99 k 2024-03-11T18:57:08.105 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 85 k 2024-03-11T18:57:08.105 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T18:57:08.105 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 171 k 2024-03-11T18:57:08.105 INFO:teuthology.orchestra.run.smithi190.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-11T18:57:08.105 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrading: 2024-03-11T18:57:08.105 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-11T18:57:08.105 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-11T18:57:08.105 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-11T18:57:08.105 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-11T18:57:08.105 INFO:teuthology.orchestra.run.smithi190.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-11T18:57:08.105 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-11T18:57:08.105 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-11T18:57:08.105 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-11T18:57:08.105 INFO:teuthology.orchestra.run.smithi190.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-11T18:57:08.106 INFO:teuthology.orchestra.run.smithi190.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-11T18:57:08.106 INFO:teuthology.orchestra.run.smithi190.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-11T18:57:08.106 INFO:teuthology.orchestra.run.smithi190.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-11T18:57:08.106 INFO:teuthology.orchestra.run.smithi190.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-11T18:57:08.106 INFO:teuthology.orchestra.run.smithi190.stdout: librados2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.4 M 2024-03-11T18:57:08.106 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T18:57:08.106 INFO:teuthology.orchestra.run.smithi190.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-11T18:57:08.106 INFO:teuthology.orchestra.run.smithi190.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-11T18:57:08.106 INFO:teuthology.orchestra.run.smithi190.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-11T18:57:08.106 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-11T18:57:08.106 INFO:teuthology.orchestra.run.smithi190.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-11T18:57:08.106 INFO:teuthology.orchestra.run.smithi190.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-11T18:57:08.106 INFO:teuthology.orchestra.run.smithi190.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-11T18:57:08.106 INFO:teuthology.orchestra.run.smithi190.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-11T18:57:08.107 INFO:teuthology.orchestra.run.smithi190.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-11T18:57:08.107 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-11T18:57:08.107 INFO:teuthology.orchestra.run.smithi190.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-11T18:57:08.107 INFO:teuthology.orchestra.run.smithi190.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-11T18:57:08.107 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-11T18:57:08.107 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 20 M 2024-03-11T18:57:08.107 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 31 k 2024-03-11T18:57:08.107 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds x86_64 2:19.0.0-1607.g94a53970.el9 ceph 2.1 M 2024-03-11T18:57:08.107 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 247 k 2024-03-11T18:57:08.107 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon x86_64 2:19.0.0-1607.g94a53970.el9 ceph 4.7 M 2024-03-11T18:57:08.107 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 17 M 2024-03-11T18:57:08.107 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 15 k 2024-03-11T18:57:08.107 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux x86_64 2:19.0.0-1607.g94a53970.el9 ceph 25 k 2024-03-11T18:57:08.107 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-11T18:57:08.107 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-11T18:57:08.108 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-11T18:57:08.108 INFO:teuthology.orchestra.run.smithi190.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-11T18:57:08.108 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-11T18:57:08.108 INFO:teuthology.orchestra.run.smithi190.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-11T18:57:08.108 INFO:teuthology.orchestra.run.smithi190.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-11T18:57:08.108 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-11T18:57:08.108 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-11T18:57:08.108 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite x86_64 2:19.0.0-1607.g94a53970.el9 ceph 163 k 2024-03-11T18:57:08.108 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-11T18:57:08.108 INFO:teuthology.orchestra.run.smithi190.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-11T18:57:08.108 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-11T18:57:08.108 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-11T18:57:08.108 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-11T18:57:08.108 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 498 k 2024-03-11T18:57:08.109 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-11T18:57:08.109 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.1 M 2024-03-11T18:57:08.109 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-11T18:57:08.109 INFO:teuthology.orchestra.run.smithi190.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-11T18:57:08.109 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-11T18:57:08.109 INFO:teuthology.orchestra.run.smithi190.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-11T18:57:08.109 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-11T18:57:08.109 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-11T18:57:08.109 INFO:teuthology.orchestra.run.smithi190.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-11T18:57:08.109 INFO:teuthology.orchestra.run.smithi190.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-11T18:57:08.109 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-11T18:57:08.109 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-11T18:57:08.109 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-11T18:57:08.109 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-11T18:57:08.110 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-11T18:57:08.110 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-11T18:57:08.110 INFO:teuthology.orchestra.run.smithi190.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-11T18:57:08.110 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-11T18:57:08.110 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-11T18:57:08.110 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-11T18:57:08.110 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 45 k 2024-03-11T18:57:08.110 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 131 k 2024-03-11T18:57:08.110 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-11T18:57:08.110 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-11T18:57:08.110 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-11T18:57:08.110 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-11T18:57:08.110 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-11T18:57:08.111 INFO:teuthology.orchestra.run.smithi190.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-11T18:57:08.111 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-11T18:57:08.111 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-11T18:57:08.111 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-11T18:57:08.111 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-11T18:57:08.111 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-11T18:57:08.111 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-11T18:57:08.111 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-11T18:57:08.111 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-11T18:57:08.111 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-11T18:57:08.111 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-11T18:57:08.111 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-11T18:57:08.111 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-11T18:57:08.112 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-11T18:57:08.112 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-11T18:57:08.112 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-11T18:57:08.112 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-11T18:57:08.112 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-11T18:57:08.112 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-11T18:57:08.112 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-11T18:57:08.112 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-11T18:57:08.112 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-11T18:57:08.112 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-11T18:57:08.112 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-11T18:57:08.112 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-11T18:57:08.112 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-11T18:57:08.113 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-11T18:57:08.113 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-11T18:57:08.113 INFO:teuthology.orchestra.run.smithi190.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-11T18:57:08.113 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-11T18:57:08.113 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-11T18:57:08.113 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-11T18:57:08.113 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-11T18:57:08.113 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-11T18:57:08.113 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-11T18:57:08.113 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-11T18:57:08.113 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-11T18:57:08.113 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-11T18:57:08.113 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-11T18:57:08.113 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-11T18:57:08.114 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-11T18:57:08.114 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-11T18:57:08.114 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-11T18:57:08.114 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-11T18:57:08.114 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-11T18:57:08.114 INFO:teuthology.orchestra.run.smithi190.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-11T18:57:08.114 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-11T18:57:08.114 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-11T18:57:08.114 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-11T18:57:08.114 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-11T18:57:08.114 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-11T18:57:08.114 INFO:teuthology.orchestra.run.smithi190.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-11T18:57:08.114 INFO:teuthology.orchestra.run.smithi190.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-11T18:57:08.115 INFO:teuthology.orchestra.run.smithi190.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-11T18:57:08.115 INFO:teuthology.orchestra.run.smithi190.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-11T18:57:08.115 INFO:teuthology.orchestra.run.smithi190.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-11T18:57:08.115 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-11T18:57:08.115 INFO:teuthology.orchestra.run.smithi190.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-11T18:57:08.115 INFO:teuthology.orchestra.run.smithi190.stdout:Installing weak dependencies: 2024-03-11T18:57:08.115 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-11T18:57:08.115 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:57:08.115 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-11T18:57:08.115 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================ 2024-03-11T18:57:08.115 INFO:teuthology.orchestra.run.smithi190.stdout:Install 150 Packages 2024-03-11T18:57:08.115 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrade 24 Packages 2024-03-11T18:57:08.115 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:57:08.116 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 223 M 2024-03-11T18:57:08.116 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-11T18:57:08.161 INFO:teuthology.orchestra.run.smithi170.stdout:(1/174): ceph-19.0.0-1607.g94a53970.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-03-11T18:57:08.721 INFO:teuthology.orchestra.run.smithi170.stdout:(2/174): ceph-fuse-19.0.0-1607.g94a53970.el9.x8 1.6 MB/s | 886 kB 00:00 2024-03-11T18:57:08.763 INFO:teuthology.orchestra.run.smithi190.stdout:(1/174): ceph-19.0.0-1607.g94a53970.el9.x86_64. 29 kB/s | 6.6 kB 00:00 2024-03-11T18:57:08.819 INFO:teuthology.orchestra.run.smithi170.stdout:(3/174): ceph-immutable-object-cache-19.0.0-160 1.4 MB/s | 145 kB 00:00 2024-03-11T18:57:09.464 INFO:teuthology.orchestra.run.smithi190.stdout:(2/174): ceph-fuse-19.0.0-1607.g94a53970.el9.x8 1.2 MB/s | 886 kB 00:00 2024-03-11T18:57:09.694 INFO:teuthology.orchestra.run.smithi190.stdout:(3/174): ceph-immutable-object-cache-19.0.0-160 633 kB/s | 145 kB 00:00 2024-03-11T18:57:09.754 INFO:teuthology.orchestra.run.smithi170.stdout:(4/174): ceph-mds-19.0.0-1607.g94a53970.el9.x86 2.3 MB/s | 2.1 MB 00:00 2024-03-11T18:57:09.772 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-11T18:57:10.370 INFO:teuthology.orchestra.run.smithi170.stdout:(5/174): ceph-mgr-19.0.0-1607.g94a53970.el9.x86 2.4 MB/s | 1.5 MB 00:00 2024-03-11T18:57:10.612 INFO:teuthology.orchestra.run.smithi170.stdout:(6/174): ceph-base-19.0.0-1607.g94a53970.el9.x8 2.0 MB/s | 5.3 MB 00:02 2024-03-11T18:57:10.656 INFO:teuthology.orchestra.run.smithi190.stdout:(4/174): ceph-common-19.0.0-1607.g94a53970.el9. 9.4 MB/s | 20 MB 00:02 2024-03-11T18:57:10.697 INFO:teuthology.orchestra.run.smithi190.stdout:(5/174): ceph-mds-19.0.0-1607.g94a53970.el9.x86 2.1 MB/s | 2.1 MB 00:01 2024-03-11T18:57:10.881 INFO:teuthology.orchestra.run.smithi190.stdout:(6/174): ceph-mgr-19.0.0-1607.g94a53970.el9.x86 6.6 MB/s | 1.5 MB 00:00 2024-03-11T18:57:11.673 INFO:teuthology.orchestra.run.smithi190.stdout:(7/174): ceph-base-19.0.0-1607.g94a53970.el9.x8 1.7 MB/s | 5.3 MB 00:03 2024-03-11T18:57:11.862 INFO:teuthology.orchestra.run.smithi170.stdout:(7/174): ceph-mon-19.0.0-1607.g94a53970.el9.x86 3.1 MB/s | 4.7 MB 00:01 2024-03-11T18:57:12.423 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-11T18:57:12.424 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-11T18:57:12.440 INFO:teuthology.orchestra.run.smithi029.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-11T18:57:12.440 INFO:teuthology.orchestra.run.smithi029.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-11T18:57:12.442 INFO:teuthology.orchestra.run.smithi029.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-11T18:57:12.443 INFO:teuthology.orchestra.run.smithi029.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-11T18:57:12.445 INFO:teuthology.orchestra.run.smithi029.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-11T18:57:12.448 INFO:teuthology.orchestra.run.smithi029.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-11T18:57:12.449 INFO:teuthology.orchestra.run.smithi029.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-11T18:57:12.535 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-11T18:57:12.560 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2024-03-11T18:57:12.560 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-11T18:57:12.561 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2024-03-11T18:57:12.561 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-11T18:57:12.561 INFO:teuthology.orchestra.run.smithi029.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-11T18:57:12.561 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:19.0.0-1607.g94a53970.el9 ceph 6.6 k 2024-03-11T18:57:12.561 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.3 M 2024-03-11T18:57:12.561 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 886 k 2024-03-11T18:57:12.561 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1607.g94a53970.el9 ceph 145 k 2024-03-11T18:57:12.561 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:19.0.0-1607.g94a53970.el9 ceph 1.5 M 2024-03-11T18:57:12.561 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 138 k 2024-03-11T18:57:12.561 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 1.8 M 2024-03-11T18:57:12.561 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 7.4 M 2024-03-11T18:57:12.561 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 49 k 2024-03-11T18:57:12.561 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 9.8 M 2024-03-11T18:57:12.562 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:19.0.0-1607.g94a53970.el9 ceph 47 M 2024-03-11T18:57:12.562 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 264 k 2024-03-11T18:57:12.562 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 748 k 2024-03-11T18:57:12.562 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 217 k 2024-03-11T18:57:12.562 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-shell x86_64 2:19.0.0-1607.g94a53970.el9 ceph 27 k 2024-03-11T18:57:12.562 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-top noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 18 k 2024-03-11T18:57:12.562 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-11T18:57:12.562 INFO:teuthology.orchestra.run.smithi029.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-11T18:57:12.562 INFO:teuthology.orchestra.run.smithi029.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-11T18:57:12.562 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 31 k 2024-03-11T18:57:12.562 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 742 k 2024-03-11T18:57:12.562 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 127 k 2024-03-11T18:57:12.562 INFO:teuthology.orchestra.run.smithi029.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-11T18:57:12.562 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-11T18:57:12.563 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-11T18:57:12.563 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-11T18:57:12.563 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-11T18:57:12.563 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-11T18:57:12.563 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:19.0.0-1607.g94a53970.el9 ceph 161 k 2024-03-11T18:57:12.563 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:19.0.0-1607.g94a53970.el9 ceph 320 k 2024-03-11T18:57:12.563 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 298 k 2024-03-11T18:57:12.563 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 99 k 2024-03-11T18:57:12.563 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 85 k 2024-03-11T18:57:12.563 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T18:57:12.563 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 171 k 2024-03-11T18:57:12.563 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-11T18:57:12.563 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2024-03-11T18:57:12.563 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-11T18:57:12.564 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-11T18:57:12.564 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-11T18:57:12.564 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-11T18:57:12.564 INFO:teuthology.orchestra.run.smithi029.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-11T18:57:12.564 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-11T18:57:12.564 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-11T18:57:12.564 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-11T18:57:12.564 INFO:teuthology.orchestra.run.smithi029.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-11T18:57:12.564 INFO:teuthology.orchestra.run.smithi029.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-11T18:57:12.564 INFO:teuthology.orchestra.run.smithi029.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-11T18:57:12.564 INFO:teuthology.orchestra.run.smithi029.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-11T18:57:12.564 INFO:teuthology.orchestra.run.smithi029.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-11T18:57:12.564 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.4 M 2024-03-11T18:57:12.564 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T18:57:12.565 INFO:teuthology.orchestra.run.smithi029.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-11T18:57:12.565 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-11T18:57:12.565 INFO:teuthology.orchestra.run.smithi029.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-11T18:57:12.565 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-11T18:57:12.565 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-11T18:57:12.565 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-11T18:57:12.565 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-11T18:57:12.565 INFO:teuthology.orchestra.run.smithi029.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-11T18:57:12.565 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-11T18:57:12.565 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-11T18:57:12.565 INFO:teuthology.orchestra.run.smithi029.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-11T18:57:12.565 INFO:teuthology.orchestra.run.smithi029.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-11T18:57:12.565 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-11T18:57:12.565 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 20 M 2024-03-11T18:57:12.565 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 31 k 2024-03-11T18:57:12.566 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:19.0.0-1607.g94a53970.el9 ceph 2.1 M 2024-03-11T18:57:12.566 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 247 k 2024-03-11T18:57:12.566 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:19.0.0-1607.g94a53970.el9 ceph 4.7 M 2024-03-11T18:57:12.566 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 17 M 2024-03-11T18:57:12.566 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 15 k 2024-03-11T18:57:12.566 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:19.0.0-1607.g94a53970.el9 ceph 25 k 2024-03-11T18:57:12.566 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-11T18:57:12.566 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-11T18:57:12.566 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-11T18:57:12.566 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-11T18:57:12.566 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-11T18:57:12.566 INFO:teuthology.orchestra.run.smithi029.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-11T18:57:12.566 INFO:teuthology.orchestra.run.smithi029.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-11T18:57:12.566 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-11T18:57:12.567 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-11T18:57:12.567 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:19.0.0-1607.g94a53970.el9 ceph 163 k 2024-03-11T18:57:12.567 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-11T18:57:12.567 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-11T18:57:12.567 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-11T18:57:12.567 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-11T18:57:12.567 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-11T18:57:12.567 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 498 k 2024-03-11T18:57:12.567 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-11T18:57:12.567 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.1 M 2024-03-11T18:57:12.567 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-11T18:57:12.567 INFO:teuthology.orchestra.run.smithi029.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-11T18:57:12.567 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-11T18:57:12.567 INFO:teuthology.orchestra.run.smithi029.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-11T18:57:12.567 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-11T18:57:12.568 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-11T18:57:12.568 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-11T18:57:12.568 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-11T18:57:12.568 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-11T18:57:12.568 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-11T18:57:12.568 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-11T18:57:12.568 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-11T18:57:12.568 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-11T18:57:12.568 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-11T18:57:12.568 INFO:teuthology.orchestra.run.smithi029.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-11T18:57:12.568 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-11T18:57:12.568 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-11T18:57:12.568 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-11T18:57:12.569 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 45 k 2024-03-11T18:57:12.569 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 131 k 2024-03-11T18:57:12.569 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-11T18:57:12.569 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-11T18:57:12.569 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-11T18:57:12.569 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-11T18:57:12.569 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-11T18:57:12.569 INFO:teuthology.orchestra.run.smithi029.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-11T18:57:12.569 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-11T18:57:12.569 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-11T18:57:12.569 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-11T18:57:12.569 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-11T18:57:12.569 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-11T18:57:12.569 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-11T18:57:12.569 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-11T18:57:12.570 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-11T18:57:12.570 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-11T18:57:12.570 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-11T18:57:12.570 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-11T18:57:12.570 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-11T18:57:12.570 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-11T18:57:12.570 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-11T18:57:12.570 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-11T18:57:12.570 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-11T18:57:12.570 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-11T18:57:12.570 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-11T18:57:12.570 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-11T18:57:12.570 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-11T18:57:12.570 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-11T18:57:12.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-11T18:57:12.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-11T18:57:12.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-11T18:57:12.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-11T18:57:12.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-11T18:57:12.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-11T18:57:12.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-11T18:57:12.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-11T18:57:12.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-11T18:57:12.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-11T18:57:12.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-11T18:57:12.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-11T18:57:12.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-11T18:57:12.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-11T18:57:12.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-11T18:57:12.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-11T18:57:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-11T18:57:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-11T18:57:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-11T18:57:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-11T18:57:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-11T18:57:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-11T18:57:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-11T18:57:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-11T18:57:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-11T18:57:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-11T18:57:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-11T18:57:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-11T18:57:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-11T18:57:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-11T18:57:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-11T18:57:12.572 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-11T18:57:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-11T18:57:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-11T18:57:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-11T18:57:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-11T18:57:12.573 INFO:teuthology.orchestra.run.smithi029.stdout:Installing weak dependencies: 2024-03-11T18:57:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-11T18:57:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:57:12.573 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-11T18:57:12.573 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2024-03-11T18:57:12.573 INFO:teuthology.orchestra.run.smithi029.stdout:Install 150 Packages 2024-03-11T18:57:12.573 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 24 Packages 2024-03-11T18:57:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:57:12.574 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 223 M 2024-03-11T18:57:12.574 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-11T18:57:12.748 INFO:teuthology.orchestra.run.smithi190.stdout:(8/174): ceph-mon-19.0.0-1607.g94a53970.el9.x86 2.3 MB/s | 4.7 MB 00:02 2024-03-11T18:57:12.857 INFO:teuthology.orchestra.run.smithi190.stdout:(9/174): ceph-selinux-19.0.0-1607.g94a53970.el9 233 kB/s | 25 kB 00:00 2024-03-11T18:57:13.368 INFO:teuthology.orchestra.run.smithi029.stdout:(1/174): ceph-19.0.0-1607.g94a53970.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-03-11T18:57:13.612 INFO:teuthology.orchestra.run.smithi170.stdout:(8/174): ceph-osd-19.0.0-1607.g94a53970.el9.x86 5.6 MB/s | 17 MB 00:02 2024-03-11T18:57:13.712 INFO:teuthology.orchestra.run.smithi170.stdout:(9/174): ceph-selinux-19.0.0-1607.g94a53970.el9 252 kB/s | 25 kB 00:00 2024-03-11T18:57:13.899 INFO:teuthology.orchestra.run.smithi190.stdout:(10/174): ceph-radosgw-19.0.0-1607.g94a53970.el 4.4 MB/s | 9.8 MB 00:02 2024-03-11T18:57:14.015 INFO:teuthology.orchestra.run.smithi190.stdout:(11/174): cephfs-mirror-19.0.0-1607.g94a53970.e 1.8 MB/s | 217 kB 00:00 2024-03-11T18:57:14.116 INFO:teuthology.orchestra.run.smithi190.stdout:(12/174): cephfs-shell-19.0.0-1607.g94a53970.el 270 kB/s | 27 kB 00:00 2024-03-11T18:57:14.219 INFO:teuthology.orchestra.run.smithi029.stdout:(2/174): ceph-fuse-19.0.0-1607.g94a53970.el9.x8 1.0 MB/s | 886 kB 00:00 2024-03-11T18:57:14.233 INFO:teuthology.orchestra.run.smithi190.stdout:(13/174): libcephfs-devel-19.0.0-1607.g94a53970 267 kB/s | 31 kB 00:00 2024-03-11T18:57:14.369 INFO:teuthology.orchestra.run.smithi029.stdout:(3/174): ceph-immutable-object-cache-19.0.0-160 970 kB/s | 145 kB 00:00 2024-03-11T18:57:14.466 INFO:teuthology.orchestra.run.smithi190.stdout:(14/174): libcephfs2-19.0.0-1607.g94a53970.el9. 3.1 MB/s | 742 kB 00:00 2024-03-11T18:57:14.558 INFO:teuthology.orchestra.run.smithi190.stdout:(15/174): libcephsqlite-19.0.0-1607.g94a53970.e 1.7 MB/s | 163 kB 00:00 2024-03-11T18:57:14.650 INFO:teuthology.orchestra.run.smithi190.stdout:(16/174): librados-devel-19.0.0-1607.g94a53970. 1.4 MB/s | 127 kB 00:00 2024-03-11T18:57:14.825 INFO:teuthology.orchestra.run.smithi190.stdout:(17/174): libradosstriper1-19.0.0-1607.g94a5397 2.8 MB/s | 498 kB 00:00 2024-03-11T18:57:14.871 INFO:teuthology.orchestra.run.smithi170.stdout:(10/174): ceph-radosgw-19.0.0-1607.g94a53970.el 3.2 MB/s | 9.8 MB 00:03 2024-03-11T18:57:14.979 INFO:teuthology.orchestra.run.smithi170.stdout:(11/174): cephfs-mirror-19.0.0-1607.g94a53970.e 2.0 MB/s | 217 kB 00:00 2024-03-11T18:57:15.079 INFO:teuthology.orchestra.run.smithi170.stdout:(12/174): cephfs-shell-19.0.0-1607.g94a53970.el 270 kB/s | 27 kB 00:00 2024-03-11T18:57:15.163 INFO:teuthology.orchestra.run.smithi170.stdout:(13/174): libcephfs-devel-19.0.0-1607.g94a53970 375 kB/s | 31 kB 00:00 2024-03-11T18:57:15.438 INFO:teuthology.orchestra.run.smithi170.stdout:(14/174): libcephfs2-19.0.0-1607.g94a53970.el9. 2.6 MB/s | 742 kB 00:00 2024-03-11T18:57:15.445 INFO:teuthology.orchestra.run.smithi029.stdout:(4/174): ceph-mds-19.0.0-1607.g94a53970.el9.x86 2.0 MB/s | 2.1 MB 00:01 2024-03-11T18:57:15.588 INFO:teuthology.orchestra.run.smithi170.stdout:(15/174): libcephsqlite-19.0.0-1607.g94a53970.e 1.1 MB/s | 163 kB 00:00 2024-03-11T18:57:15.695 INFO:teuthology.orchestra.run.smithi029.stdout:(5/174): ceph-base-19.0.0-1607.g94a53970.el9.x8 2.0 MB/s | 5.3 MB 00:02 2024-03-11T18:57:15.722 INFO:teuthology.orchestra.run.smithi170.stdout:(16/174): librados-devel-19.0.0-1607.g94a53970. 949 kB/s | 127 kB 00:00 2024-03-11T18:57:15.909 INFO:teuthology.orchestra.run.smithi190.stdout:(18/174): ceph-osd-19.0.0-1607.g94a53970.el9.x8 3.3 MB/s | 17 MB 00:05 2024-03-11T18:57:15.939 INFO:teuthology.orchestra.run.smithi170.stdout:(17/174): libradosstriper1-19.0.0-1607.g94a5397 2.2 MB/s | 498 kB 00:00 2024-03-11T18:57:15.984 INFO:teuthology.orchestra.run.smithi190.stdout:(19/174): librgw2-19.0.0-1607.g94a53970.el9.x86 4.4 MB/s | 5.1 MB 00:01 2024-03-11T18:57:15.996 INFO:teuthology.orchestra.run.smithi029.stdout:(6/174): ceph-mgr-19.0.0-1607.g94a53970.el9.x86 2.7 MB/s | 1.5 MB 00:00 2024-03-11T18:57:16.009 INFO:teuthology.orchestra.run.smithi190.stdout:(20/174): python3-ceph-argparse-19.0.0-1607.g94 451 kB/s | 45 kB 00:00 2024-03-11T18:57:16.084 INFO:teuthology.orchestra.run.smithi190.stdout:(21/174): python3-ceph-common-19.0.0-1607.g94a5 1.3 MB/s | 131 kB 00:00 2024-03-11T18:57:16.110 INFO:teuthology.orchestra.run.smithi190.stdout:(22/174): python3-cephfs-19.0.0-1607.g94a53970. 1.6 MB/s | 161 kB 00:00 2024-03-11T18:57:16.227 INFO:teuthology.orchestra.run.smithi190.stdout:(23/174): python3-rados-19.0.0-1607.g94a53970.e 2.2 MB/s | 320 kB 00:00 2024-03-11T18:57:16.252 INFO:teuthology.orchestra.run.smithi190.stdout:(24/174): python3-rbd-19.0.0-1607.g94a53970.el9 2.1 MB/s | 298 kB 00:00 2024-03-11T18:57:16.319 INFO:teuthology.orchestra.run.smithi190.stdout:(25/174): python3-rgw-19.0.0-1607.g94a53970.el9 1.1 MB/s | 99 kB 00:00 2024-03-11T18:57:16.344 INFO:teuthology.orchestra.run.smithi190.stdout:(26/174): rbd-fuse-19.0.0-1607.g94a53970.el9.x8 929 kB/s | 85 kB 00:00 2024-03-11T18:57:16.459 INFO:teuthology.orchestra.run.smithi190.stdout:(27/174): rbd-nbd-19.0.0-1607.g94a53970.el9.x86 1.5 MB/s | 171 kB 00:00 2024-03-11T18:57:16.528 INFO:teuthology.orchestra.run.smithi190.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1607.g 455 kB/s | 31 kB 00:00 2024-03-11T18:57:16.670 INFO:teuthology.orchestra.run.smithi190.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1607.g94a5397 973 kB/s | 138 kB 00:00 2024-03-11T18:57:16.895 INFO:teuthology.orchestra.run.smithi190.stdout:(30/174): rbd-mirror-19.0.0-1607.g94a53970.el9. 5.4 MB/s | 3.1 MB 00:00 2024-03-11T18:57:16.988 INFO:teuthology.orchestra.run.smithi029.stdout:(7/174): ceph-mon-19.0.0-1607.g94a53970.el9.x86 3.6 MB/s | 4.7 MB 00:01 2024-03-11T18:57:17.223 INFO:teuthology.orchestra.run.smithi170.stdout:(18/174): librgw2-19.0.0-1607.g94a53970.el9.x86 4.0 MB/s | 5.1 MB 00:01 2024-03-11T18:57:17.304 INFO:teuthology.orchestra.run.smithi190.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1607.g94a53 2.8 MB/s | 1.8 MB 00:00 2024-03-11T18:57:17.314 INFO:teuthology.orchestra.run.smithi170.stdout:(19/174): python3-ceph-argparse-19.0.0-1607.g94 493 kB/s | 45 kB 00:00 2024-03-11T18:57:17.423 INFO:teuthology.orchestra.run.smithi170.stdout:(20/174): python3-ceph-common-19.0.0-1607.g94a5 1.2 MB/s | 131 kB 00:00 2024-03-11T18:57:17.429 INFO:teuthology.orchestra.run.smithi190.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1607.g94 1.9 MB/s | 247 kB 00:00 2024-03-11T18:57:17.496 INFO:teuthology.orchestra.run.smithi190.stdout:(33/174): ceph-mgr-rook-19.0.0-1607.g94a53970.e 741 kB/s | 49 kB 00:00 2024-03-11T18:57:17.515 INFO:teuthology.orchestra.run.smithi170.stdout:(21/174): python3-cephfs-19.0.0-1607.g94a53970. 1.7 MB/s | 161 kB 00:00 2024-03-11T18:57:17.554 INFO:teuthology.orchestra.run.smithi190.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1607.g9 264 kB/s | 15 kB 00:00 2024-03-11T18:57:17.682 INFO:teuthology.orchestra.run.smithi170.stdout:(22/174): python3-rados-19.0.0-1607.g94a53970.e 1.9 MB/s | 320 kB 00:00 2024-03-11T18:57:17.696 INFO:teuthology.orchestra.run.smithi190.stdout:(35/174): ceph-volume-19.0.0-1607.g94a53970.el9 1.8 MB/s | 264 kB 00:00 2024-03-11T18:57:17.783 INFO:teuthology.orchestra.run.smithi170.stdout:(23/174): python3-rbd-19.0.0-1607.g94a53970.el9 2.9 MB/s | 298 kB 00:00 2024-03-11T18:57:17.883 INFO:teuthology.orchestra.run.smithi170.stdout:(24/174): python3-rgw-19.0.0-1607.g94a53970.el9 996 kB/s | 99 kB 00:00 2024-03-11T18:57:17.983 INFO:teuthology.orchestra.run.smithi170.stdout:(25/174): rbd-fuse-19.0.0-1607.g94a53970.el9.x8 853 kB/s | 85 kB 00:00 2024-03-11T18:57:18.005 INFO:teuthology.orchestra.run.smithi190.stdout:(36/174): cephadm-19.0.0-1607.g94a53970.el9.noa 2.4 MB/s | 748 kB 00:00 2024-03-11T18:57:18.089 INFO:teuthology.orchestra.run.smithi190.stdout:(37/174): ceph-mgr-diskprediction-local-19.0.0- 6.2 MB/s | 7.4 MB 00:01 2024-03-11T18:57:18.114 INFO:teuthology.orchestra.run.smithi190.stdout:(38/174): cephfs-top-19.0.0-1607.g94a53970.el9. 168 kB/s | 18 kB 00:00 2024-03-11T18:57:18.273 INFO:teuthology.orchestra.run.smithi190.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.5 MB/s | 662 kB 00:00 2024-03-11T18:57:18.315 INFO:teuthology.orchestra.run.smithi190.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.5 MB/s | 920 kB 00:00 2024-03-11T18:57:18.340 INFO:teuthology.orchestra.run.smithi190.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-11T18:57:18.365 INFO:teuthology.orchestra.run.smithi190.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-11T18:57:18.390 INFO:teuthology.orchestra.run.smithi190.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 636 kB/s | 32 kB 00:00 2024-03-11T18:57:18.474 INFO:teuthology.orchestra.run.smithi190.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 27 MB/s | 3.0 MB 00:00 2024-03-11T18:57:18.499 INFO:teuthology.orchestra.run.smithi190.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 161 kB/s | 17 kB 00:00 2024-03-11T18:57:18.525 INFO:teuthology.orchestra.run.smithi190.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-03-11T18:57:18.550 INFO:teuthology.orchestra.run.smithi190.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-11T18:57:18.575 INFO:teuthology.orchestra.run.smithi190.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2024-03-11T18:57:18.600 INFO:teuthology.orchestra.run.smithi190.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-11T18:57:18.626 INFO:teuthology.orchestra.run.smithi190.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-11T18:57:18.651 INFO:teuthology.orchestra.run.smithi190.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 940 kB/s | 46 kB 00:00 2024-03-11T18:57:18.676 INFO:teuthology.orchestra.run.smithi190.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-11T18:57:18.700 INFO:teuthology.orchestra.run.smithi170.stdout:(26/174): ceph-common-19.0.0-1607.g94a53970.el9 1.8 MB/s | 20 MB 00:10 2024-03-11T18:57:18.701 INFO:teuthology.orchestra.run.smithi190.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-11T18:57:18.727 INFO:teuthology.orchestra.run.smithi190.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-11T18:57:18.743 INFO:teuthology.orchestra.run.smithi190.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 825 kB/s | 34 kB 00:00 2024-03-11T18:57:18.758 INFO:teuthology.orchestra.run.smithi170.stdout:(27/174): rbd-mirror-19.0.0-1607.g94a53970.el9. 4.0 MB/s | 3.1 MB 00:00 2024-03-11T18:57:18.867 INFO:teuthology.orchestra.run.smithi170.stdout:(28/174): rbd-nbd-19.0.0-1607.g94a53970.el9.x86 1.0 MB/s | 171 kB 00:00 2024-03-11T18:57:18.892 INFO:teuthology.orchestra.run.smithi170.stdout:(29/174): ceph-grafana-dashboards-19.0.0-1607.g 235 kB/s | 31 kB 00:00 2024-03-11T18:57:18.944 INFO:teuthology.orchestra.run.smithi190.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 22 MB/s | 4.7 MB 00:00 2024-03-11T18:57:18.976 INFO:teuthology.orchestra.run.smithi170.stdout:(30/174): ceph-mgr-cephadm-19.0.0-1607.g94a5397 1.2 MB/s | 138 kB 00:00 2024-03-11T18:57:18.986 INFO:teuthology.orchestra.run.smithi190.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 6.3 MB/s | 1.5 MB 00:00 2024-03-11T18:57:19.011 INFO:teuthology.orchestra.run.smithi190.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 12 MB/s | 811 kB 00:00 2024-03-11T18:57:19.036 INFO:teuthology.orchestra.run.smithi190.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.7 MB/s | 134 kB 00:00 2024-03-11T18:57:19.188 INFO:teuthology.orchestra.run.smithi029.stdout:(8/174): ceph-radosgw-19.0.0-1607.g94a53970.el9 4.4 MB/s | 9.8 MB 00:02 2024-03-11T18:57:19.195 INFO:teuthology.orchestra.run.smithi190.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 31 MB/s | 5.7 MB 00:00 2024-03-11T18:57:19.229 INFO:teuthology.orchestra.run.smithi190.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.3 MB/s | 1.0 MB 00:00 2024-03-11T18:57:19.234 INFO:teuthology.orchestra.run.smithi170.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1607.g94a53 5.2 MB/s | 1.8 MB 00:00 2024-03-11T18:57:19.239 INFO:teuthology.orchestra.run.smithi029.stdout:(9/174): ceph-selinux-19.0.0-1607.g94a53970.el9 504 kB/s | 25 kB 00:00 2024-03-11T18:57:19.254 INFO:teuthology.orchestra.run.smithi190.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-11T18:57:19.335 INFO:teuthology.orchestra.run.smithi170.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1607.g94 2.4 MB/s | 247 kB 00:00 2024-03-11T18:57:19.402 INFO:teuthology.orchestra.run.smithi170.stdout:(33/174): ceph-mgr-rook-19.0.0-1607.g94a53970.e 740 kB/s | 49 kB 00:00 2024-03-11T18:57:19.413 INFO:teuthology.orchestra.run.smithi190.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 31 MB/s | 5.8 MB 00:00 2024-03-11T18:57:19.429 INFO:teuthology.orchestra.run.smithi190.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 140 kB/s | 24 kB 00:00 2024-03-11T18:57:19.454 INFO:teuthology.orchestra.run.smithi190.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 730 kB/s | 18 kB 00:00 2024-03-11T18:57:19.480 INFO:teuthology.orchestra.run.smithi190.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-11T18:57:19.497 INFO:teuthology.orchestra.run.smithi190.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 756 kB/s | 32 kB 00:00 2024-03-11T18:57:19.502 INFO:teuthology.orchestra.run.smithi170.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1607.g9 153 kB/s | 15 kB 00:00 2024-03-11T18:57:19.619 INFO:teuthology.orchestra.run.smithi170.stdout:(35/174): ceph-volume-19.0.0-1607.g94a53970.el9 2.2 MB/s | 264 kB 00:00 2024-03-11T18:57:19.639 INFO:teuthology.orchestra.run.smithi190.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-11T18:57:19.655 INFO:teuthology.orchestra.run.smithi190.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-11T18:57:19.689 INFO:teuthology.orchestra.run.smithi190.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-11T18:57:19.731 INFO:teuthology.orchestra.run.smithi190.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 912 kB/s | 69 kB 00:00 2024-03-11T18:57:19.757 INFO:teuthology.orchestra.run.smithi190.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 617 kB/s | 41 kB 00:00 2024-03-11T18:57:19.782 INFO:teuthology.orchestra.run.smithi190.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-11T18:57:19.807 INFO:teuthology.orchestra.run.smithi190.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-11T18:57:19.811 INFO:teuthology.orchestra.run.smithi170.stdout:(36/174): cephadm-19.0.0-1607.g94a53970.el9.noa 3.8 MB/s | 748 kB 00:00 2024-03-11T18:57:19.860 INFO:teuthology.orchestra.run.smithi190.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.6 MB/s | 211 kB 00:00 2024-03-11T18:57:19.886 INFO:teuthology.orchestra.run.smithi170.stdout:(37/174): cephfs-top-19.0.0-1607.g94a53970.el9. 243 kB/s | 18 kB 00:00 2024-03-11T18:57:19.891 INFO:teuthology.orchestra.run.smithi190.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 570 kB/s | 47 kB 00:00 2024-03-11T18:57:19.916 INFO:teuthology.orchestra.run.smithi190.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 766 kB/s | 43 kB 00:00 2024-03-11T18:57:20.225 INFO:teuthology.orchestra.run.smithi190.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 125 kB/s | 39 kB 00:00 2024-03-11T18:57:20.231 INFO:teuthology.orchestra.run.smithi029.stdout:(10/174): ceph-osd-19.0.0-1607.g94a53970.el9.x8 4.0 MB/s | 17 MB 00:04 2024-03-11T18:57:20.295 INFO:teuthology.orchestra.run.smithi170.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 1.6 MB/s | 662 kB 00:00 2024-03-11T18:57:20.315 INFO:teuthology.orchestra.run.smithi029.stdout:(11/174): cephfs-mirror-19.0.0-1607.g94a53970.e 2.6 MB/s | 217 kB 00:00 2024-03-11T18:57:20.357 INFO:teuthology.orchestra.run.smithi029.stdout:(12/174): cephfs-shell-19.0.0-1607.g94a53970.el 648 kB/s | 27 kB 00:00 2024-03-11T18:57:20.362 INFO:teuthology.orchestra.run.smithi170.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 920 kB 00:00 2024-03-11T18:57:20.395 INFO:teuthology.orchestra.run.smithi170.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-11T18:57:20.437 INFO:teuthology.orchestra.run.smithi170.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-11T18:57:20.459 INFO:teuthology.orchestra.run.smithi190.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-03-11T18:57:20.484 INFO:teuthology.orchestra.run.smithi190.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 160 kB/s | 41 kB 00:00 2024-03-11T18:57:20.487 INFO:teuthology.orchestra.run.smithi170.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 633 kB/s | 32 kB 00:00 2024-03-11T18:57:20.542 INFO:teuthology.orchestra.run.smithi190.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.5 MB/s | 300 kB 00:00 2024-03-11T18:57:20.626 INFO:teuthology.orchestra.run.smithi190.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 396 kB/s | 56 kB 00:00 2024-03-11T18:57:20.709 INFO:teuthology.orchestra.run.smithi029.stdout:(13/174): ceph-common-19.0.0-1607.g94a53970.el9 2.6 MB/s | 20 MB 00:07 2024-03-11T18:57:20.726 INFO:teuthology.orchestra.run.smithi190.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 341 kB/s | 63 kB 00:00 2024-03-11T18:57:20.734 INFO:teuthology.orchestra.run.smithi029.stdout:(14/174): libcephfs-devel-19.0.0-1607.g94a53970 83 kB/s | 31 kB 00:00 2024-03-11T18:57:20.751 INFO:teuthology.orchestra.run.smithi190.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 352 kB/s | 44 kB 00:00 2024-03-11T18:57:20.796 INFO:teuthology.orchestra.run.smithi170.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.7 MB/s | 3.0 MB 00:00 2024-03-11T18:57:20.810 INFO:teuthology.orchestra.run.smithi029.stdout:(15/174): libcephsqlite-19.0.0-1607.g94a53970.e 2.1 MB/s | 163 kB 00:00 2024-03-11T18:57:20.810 INFO:teuthology.orchestra.run.smithi190.stdout:(85/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-11T18:57:20.835 INFO:teuthology.orchestra.run.smithi190.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 941 kB/s | 79 kB 00:00 2024-03-11T18:57:20.846 INFO:teuthology.orchestra.run.smithi170.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 346 kB/s | 17 kB 00:00 2024-03-11T18:57:20.869 INFO:teuthology.orchestra.run.smithi029.stdout:(16/174): libcephfs2-19.0.0-1607.g94a53970.el9. 4.6 MB/s | 742 kB 00:00 2024-03-11T18:57:20.894 INFO:teuthology.orchestra.run.smithi029.stdout:(17/174): librados-devel-19.0.0-1607.g94a53970. 1.5 MB/s | 127 kB 00:00 2024-03-11T18:57:20.921 INFO:teuthology.orchestra.run.smithi170.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 134 kB/s | 10 kB 00:00 2024-03-11T18:57:20.970 INFO:teuthology.orchestra.run.smithi029.stdout:(18/174): libradosstriper1-19.0.0-1607.g94a5397 4.8 MB/s | 498 kB 00:00 2024-03-11T18:57:20.997 INFO:teuthology.orchestra.run.smithi170.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-03-11T18:57:21.010 INFO:teuthology.orchestra.run.smithi190.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 5.7 MB/s | 1.1 MB 00:00 2024-03-11T18:57:21.022 INFO:teuthology.orchestra.run.smithi029.stdout:(19/174): python3-ceph-argparse-19.0.0-1607.g94 901 kB/s | 45 kB 00:00 2024-03-11T18:57:21.036 INFO:teuthology.orchestra.run.smithi190.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 514 kB/s | 103 kB 00:00 2024-03-11T18:57:21.055 INFO:teuthology.orchestra.run.smithi170.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 760 kB/s | 44 kB 00:00 2024-03-11T18:57:21.079 INFO:teuthology.orchestra.run.smithi029.stdout:(20/174): python3-ceph-common-19.0.0-1607.g94a5 2.2 MB/s | 131 kB 00:00 2024-03-11T18:57:21.111 INFO:teuthology.orchestra.run.smithi190.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 124 kB 00:00 2024-03-11T18:57:21.136 INFO:teuthology.orchestra.run.smithi190.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 1.8 MB/s | 187 kB 00:00 2024-03-11T18:57:21.138 INFO:teuthology.orchestra.run.smithi029.stdout:(21/174): python3-cephfs-19.0.0-1607.g94a53970. 2.7 MB/s | 161 kB 00:00 2024-03-11T18:57:21.139 INFO:teuthology.orchestra.run.smithi170.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-03-11T18:57:21.206 INFO:teuthology.orchestra.run.smithi170.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-03-11T18:57:21.222 INFO:teuthology.orchestra.run.smithi029.stdout:(22/174): python3-rados-19.0.0-1607.g94a53970.e 3.7 MB/s | 320 kB 00:00 2024-03-11T18:57:21.240 INFO:teuthology.orchestra.run.smithi170.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-11T18:57:21.282 INFO:teuthology.orchestra.run.smithi170.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-11T18:57:21.306 INFO:teuthology.orchestra.run.smithi029.stdout:(23/174): python3-rbd-19.0.0-1607.g94a53970.el9 3.5 MB/s | 298 kB 00:00 2024-03-11T18:57:21.324 INFO:teuthology.orchestra.run.smithi170.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-11T18:57:21.366 INFO:teuthology.orchestra.run.smithi170.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-03-11T18:57:21.399 INFO:teuthology.orchestra.run.smithi170.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-11T18:57:21.432 INFO:teuthology.orchestra.run.smithi029.stdout:(24/174): python3-rgw-19.0.0-1607.g94a53970.el9 792 kB/s | 99 kB 00:00 2024-03-11T18:57:21.482 INFO:teuthology.orchestra.run.smithi029.stdout:(25/174): rbd-fuse-19.0.0-1607.g94a53970.el9.x8 1.7 MB/s | 85 kB 00:00 2024-03-11T18:57:21.708 INFO:teuthology.orchestra.run.smithi170.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-03-11T18:57:21.753 INFO:teuthology.orchestra.run.smithi190.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 292 kB/s | 187 kB 00:00 2024-03-11T18:57:21.808 INFO:teuthology.orchestra.run.smithi029.stdout:(26/174): librgw2-19.0.0-1607.g94a53970.el9.x86 5.6 MB/s | 5.1 MB 00:00 2024-03-11T18:57:21.808 INFO:teuthology.orchestra.run.smithi170.stdout:(56/174): postgresql-13.11-1.el9.x86_64.rpm 15 MB/s | 1.5 MB 00:00 2024-03-11T18:57:21.875 INFO:teuthology.orchestra.run.smithi029.stdout:(27/174): rbd-nbd-19.0.0-1607.g94a53970.el9.x86 2.5 MB/s | 171 kB 00:00 2024-03-11T18:57:21.876 INFO:teuthology.orchestra.run.smithi170.stdout:(57/174): postgresql-contrib-13.11-1.el9.x86_64 12 MB/s | 811 kB 00:00 2024-03-11T18:57:21.917 INFO:teuthology.orchestra.run.smithi170.stdout:(58/174): postgresql-private-libs-13.11-1.el9.x 3.2 MB/s | 134 kB 00:00 2024-03-11T18:57:21.927 INFO:teuthology.orchestra.run.smithi029.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1607.g 609 kB/s | 31 kB 00:00 2024-03-11T18:57:21.962 INFO:teuthology.orchestra.run.smithi190.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 938 kB/s | 195 kB 00:00 2024-03-11T18:57:22.036 INFO:teuthology.orchestra.run.smithi029.stdout:(29/174): rbd-mirror-19.0.0-1607.g94a53970.el9. 5.6 MB/s | 3.1 MB 00:00 2024-03-11T18:57:22.062 INFO:teuthology.orchestra.run.smithi029.stdout:(30/174): ceph-mgr-cephadm-19.0.0-1607.g94a5397 1.0 MB/s | 138 kB 00:00 2024-03-11T18:57:22.095 INFO:teuthology.orchestra.run.smithi190.stdout:(93/174): python3-requests-2.25.1-8.el9.noarch. 938 kB/s | 125 kB 00:00 2024-03-11T18:57:22.196 INFO:teuthology.orchestra.run.smithi190.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 215 kB/s | 21 kB 00:00 2024-03-11T18:57:22.218 INFO:teuthology.orchestra.run.smithi170.stdout:(59/174): postgresql-server-13.11-1.el9.x86_64. 19 MB/s | 5.7 MB 00:00 2024-03-11T18:57:22.296 INFO:teuthology.orchestra.run.smithi190.stdout:(95/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 254 kB/s | 25 kB 00:00 2024-03-11T18:57:22.301 INFO:teuthology.orchestra.run.smithi170.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-03-11T18:57:22.335 INFO:teuthology.orchestra.run.smithi170.stdout:(61/174): python3-attrs-20.3.0-7.el9.noarch.rpm 2.3 MB/s | 78 kB 00:00 2024-03-11T18:57:22.337 INFO:teuthology.orchestra.run.smithi029.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1607.g94a53 5.9 MB/s | 1.8 MB 00:00 2024-03-11T18:57:22.413 INFO:teuthology.orchestra.run.smithi029.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1607.g94 3.2 MB/s | 247 kB 00:00 2024-03-11T18:57:22.472 INFO:teuthology.orchestra.run.smithi029.stdout:(33/174): ceph-mgr-rook-19.0.0-1607.g94a53970.e 842 kB/s | 49 kB 00:00 2024-03-11T18:57:22.531 INFO:teuthology.orchestra.run.smithi029.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1607.g9 262 kB/s | 15 kB 00:00 2024-03-11T18:57:22.602 INFO:teuthology.orchestra.run.smithi170.stdout:(62/174): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-11T18:57:22.614 INFO:teuthology.orchestra.run.smithi029.stdout:(35/174): ceph-volume-19.0.0-1607.g94a53970.el9 3.1 MB/s | 264 kB 00:00 2024-03-11T18:57:22.644 INFO:teuthology.orchestra.run.smithi170.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 590 kB/s | 24 kB 00:00 2024-03-11T18:57:22.678 INFO:teuthology.orchestra.run.smithi170.stdout:(64/174): python3-jsonpointer-2.0-4.el9.noarch. 545 kB/s | 18 kB 00:00 2024-03-11T18:57:22.719 INFO:teuthology.orchestra.run.smithi170.stdout:(65/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-11T18:57:22.770 INFO:teuthology.orchestra.run.smithi170.stdout:(66/174): python3-markupsafe-1.1.1-12.el9.x86_6 635 kB/s | 32 kB 00:00 2024-03-11T18:57:22.773 INFO:teuthology.orchestra.run.smithi029.stdout:(36/174): cephadm-19.0.0-1607.g94a53970.el9.noa 4.6 MB/s | 748 kB 00:00 2024-03-11T18:57:22.832 INFO:teuthology.orchestra.run.smithi029.stdout:(37/174): cephfs-top-19.0.0-1607.g94a53970.el9. 312 kB/s | 18 kB 00:00 2024-03-11T18:57:22.936 INFO:teuthology.orchestra.run.smithi170.stdout:(67/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-11T18:57:22.979 INFO:teuthology.orchestra.run.smithi170.stdout:(68/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.1 MB/s | 217 kB 00:00 2024-03-11T18:57:22.983 INFO:teuthology.orchestra.run.smithi029.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.3 MB/s | 662 kB 00:00 2024-03-11T18:57:23.021 INFO:teuthology.orchestra.run.smithi170.stdout:(69/174): python3-oauthlib-3.1.1-5.el9.noarch.r 4.1 MB/s | 174 kB 00:00 2024-03-11T18:57:23.042 INFO:teuthology.orchestra.run.smithi029.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 15 MB/s | 920 kB 00:00 2024-03-11T18:57:23.054 INFO:teuthology.orchestra.run.smithi170.stdout:(70/174): python3-packaging-20.9-5.el9.noarch.r 2.0 MB/s | 69 kB 00:00 2024-03-11T18:57:23.063 INFO:teuthology.orchestra.run.smithi190.stdout:(96/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 5.4 MB/s | 4.1 MB 00:00 2024-03-11T18:57:23.075 INFO:teuthology.orchestra.run.smithi029.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-11T18:57:23.088 INFO:teuthology.orchestra.run.smithi170.stdout:(71/174): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-11T18:57:23.118 INFO:teuthology.orchestra.run.smithi029.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-11T18:57:23.130 INFO:teuthology.orchestra.run.smithi170.stdout:(72/174): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-11T18:57:23.152 INFO:teuthology.orchestra.run.smithi029.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 942 kB/s | 32 kB 00:00 2024-03-11T18:57:23.163 INFO:teuthology.orchestra.run.smithi170.stdout:(73/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.9 MB/s | 132 kB 00:00 2024-03-11T18:57:23.222 INFO:teuthology.orchestra.run.smithi170.stdout:(74/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-11T18:57:23.254 INFO:teuthology.orchestra.run.smithi190.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 379 kB/s | 803 kB 00:02 2024-03-11T18:57:23.255 INFO:teuthology.orchestra.run.smithi170.stdout:(75/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-11T18:57:23.297 INFO:teuthology.orchestra.run.smithi170.stdout:(76/174): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 43 kB 00:00 2024-03-11T18:57:23.329 INFO:teuthology.orchestra.run.smithi029.stdout:(43/174): ceph-mgr-diskprediction-local-19.0.0- 5.8 MB/s | 7.4 MB 00:01 2024-03-11T18:57:23.363 INFO:teuthology.orchestra.run.smithi190.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 818 kB/s | 245 kB 00:00 2024-03-11T18:57:23.413 INFO:teuthology.orchestra.run.smithi029.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-03-11T18:57:23.438 INFO:teuthology.orchestra.run.smithi029.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 162 kB/s | 17 kB 00:00 2024-03-11T18:57:23.464 INFO:teuthology.orchestra.run.smithi029.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 199 kB/s | 10 kB 00:00 2024-03-11T18:57:23.498 INFO:teuthology.orchestra.run.smithi029.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.6 MB/s | 159 kB 00:00 2024-03-11T18:57:23.531 INFO:teuthology.orchestra.run.smithi029.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 656 kB/s | 44 kB 00:00 2024-03-11T18:57:23.566 INFO:teuthology.orchestra.run.smithi029.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.6 MB/s | 661 kB 00:00 2024-03-11T18:57:23.608 INFO:teuthology.orchestra.run.smithi029.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 578 kB 00:00 2024-03-11T18:57:23.633 INFO:teuthology.orchestra.run.smithi029.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 701 kB/s | 46 kB 00:00 2024-03-11T18:57:23.640 INFO:teuthology.orchestra.run.smithi170.stdout:(77/174): ceph-mgr-diskprediction-local-19.0.0- 1.6 MB/s | 7.4 MB 00:04 2024-03-11T18:57:23.663 INFO:teuthology.orchestra.run.smithi190.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 722 kB/s | 216 kB 00:00 2024-03-11T18:57:23.667 INFO:teuthology.orchestra.run.smithi029.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-11T18:57:23.701 INFO:teuthology.orchestra.run.smithi029.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-11T18:57:23.705 INFO:teuthology.orchestra.run.smithi190.stdout:(100/174): python3-jinja2-2.11.3-5.el9.noarch.r 550 kB/s | 248 kB 00:00 2024-03-11T18:57:23.727 INFO:teuthology.orchestra.run.smithi029.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-03-11T18:57:23.760 INFO:teuthology.orchestra.run.smithi029.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 583 kB/s | 34 kB 00:00 2024-03-11T18:57:23.781 INFO:teuthology.orchestra.run.smithi190.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 298 kB/s | 35 kB 00:00 2024-03-11T18:57:23.790 INFO:teuthology.orchestra.run.smithi170.stdout:(78/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-03-11T18:57:23.815 INFO:teuthology.orchestra.run.smithi170.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 220 kB/s | 39 kB 00:00 2024-03-11T18:57:23.831 INFO:teuthology.orchestra.run.smithi190.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 355 kB/s | 44 kB 00:00 2024-03-11T18:57:23.840 INFO:teuthology.orchestra.run.smithi170.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 820 kB/s | 41 kB 00:00 2024-03-11T18:57:23.866 INFO:teuthology.orchestra.run.smithi170.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-03-11T18:57:23.890 INFO:teuthology.orchestra.run.smithi190.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 863 kB/s | 94 kB 00:00 2024-03-11T18:57:23.891 INFO:teuthology.orchestra.run.smithi170.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-11T18:57:23.916 INFO:teuthology.orchestra.run.smithi170.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-11T18:57:23.928 INFO:teuthology.orchestra.run.smithi029.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-03-11T18:57:23.942 INFO:teuthology.orchestra.run.smithi170.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-03-11T18:57:23.967 INFO:teuthology.orchestra.run.smithi170.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-11T18:57:23.978 INFO:teuthology.orchestra.run.smithi029.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 7.1 MB/s | 1.5 MB 00:00 2024-03-11T18:57:24.009 INFO:teuthology.orchestra.run.smithi170.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-11T18:57:24.022 INFO:teuthology.orchestra.run.smithi029.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 8.7 MB/s | 811 kB 00:00 2024-03-11T18:57:24.056 INFO:teuthology.orchestra.run.smithi029.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 1.7 MB/s | 134 kB 00:00 2024-03-11T18:57:24.059 INFO:teuthology.orchestra.run.smithi170.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-11T18:57:24.084 INFO:teuthology.orchestra.run.smithi170.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-11T18:57:24.127 INFO:teuthology.orchestra.run.smithi170.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 1.8 MB/s | 124 kB 00:00 2024-03-11T18:57:24.160 INFO:teuthology.orchestra.run.smithi170.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-11T18:57:24.298 INFO:teuthology.orchestra.run.smithi029.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 21 MB/s | 5.7 MB 00:00 2024-03-11T18:57:24.340 INFO:teuthology.orchestra.run.smithi029.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 3.5 MB/s | 1.0 MB 00:00 2024-03-11T18:57:24.366 INFO:teuthology.orchestra.run.smithi029.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.2 MB/s | 78 kB 00:00 2024-03-11T18:57:24.607 INFO:teuthology.orchestra.run.smithi190.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 29 kB/s | 22 kB 00:00 2024-03-11T18:57:24.633 INFO:teuthology.orchestra.run.smithi029.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-11T18:57:24.659 INFO:teuthology.orchestra.run.smithi029.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 83 kB/s | 24 kB 00:00 2024-03-11T18:57:24.684 INFO:teuthology.orchestra.run.smithi029.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 358 kB/s | 18 kB 00:00 2024-03-11T18:57:24.710 INFO:teuthology.orchestra.run.smithi029.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-11T18:57:24.710 INFO:teuthology.orchestra.run.smithi170.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 321 kB/s | 187 kB 00:00 2024-03-11T18:57:24.735 INFO:teuthology.orchestra.run.smithi029.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 625 kB/s | 32 kB 00:00 2024-03-11T18:57:24.902 INFO:teuthology.orchestra.run.smithi170.stdout:(92/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-03-11T18:57:24.903 INFO:teuthology.orchestra.run.smithi029.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2024-03-11T18:57:24.928 INFO:teuthology.orchestra.run.smithi170.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 901 kB/s | 195 kB 00:00 2024-03-11T18:57:24.936 INFO:teuthology.orchestra.run.smithi029.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.1 MB/s | 217 kB 00:00 2024-03-11T18:57:24.970 INFO:teuthology.orchestra.run.smithi029.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 174 kB 00:00 2024-03-11T18:57:24.996 INFO:teuthology.orchestra.run.smithi029.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 69 kB 00:00 2024-03-11T18:57:25.011 INFO:teuthology.orchestra.run.smithi170.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-03-11T18:57:25.015 INFO:teuthology.orchestra.run.smithi190.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-03-11T18:57:25.023 INFO:teuthology.orchestra.run.smithi029.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 790 kB/s | 41 kB 00:00 2024-03-11T18:57:25.028 INFO:teuthology.orchestra.run.smithi170.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 215 kB/s | 21 kB 00:00 2024-03-11T18:57:25.057 INFO:teuthology.orchestra.run.smithi029.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 3.8 MB/s | 238 kB 00:00 2024-03-11T18:57:25.082 INFO:teuthology.orchestra.run.smithi029.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-11T18:57:25.095 INFO:teuthology.orchestra.run.smithi170.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 305 kB/s | 25 kB 00:00 2024-03-11T18:57:25.108 INFO:teuthology.orchestra.run.smithi029.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-11T18:57:25.115 INFO:teuthology.orchestra.run.smithi190.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 3.0 MB/s | 309 kB 00:00 2024-03-11T18:57:25.133 INFO:teuthology.orchestra.run.smithi029.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 945 kB/s | 47 kB 00:00 2024-03-11T18:57:25.159 INFO:teuthology.orchestra.run.smithi029.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 846 kB/s | 43 kB 00:00 2024-03-11T18:57:25.182 INFO:teuthology.orchestra.run.smithi190.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 292 kB/s | 19 kB 00:00 2024-03-11T18:57:25.245 INFO:teuthology.orchestra.run.smithi170.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-11T18:57:25.329 INFO:teuthology.orchestra.run.smithi170.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-03-11T18:57:25.387 INFO:teuthology.orchestra.run.smithi170.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 3.6 MB/s | 216 kB 00:00 2024-03-11T18:57:25.466 INFO:teuthology.orchestra.run.smithi190.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-03-11T18:57:25.529 INFO:teuthology.orchestra.run.smithi170.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 246 kB/s | 35 kB 00:00 2024-03-11T18:57:25.532 INFO:teuthology.orchestra.run.smithi190.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 391 kB/s | 26 kB 00:00 2024-03-11T18:57:25.616 INFO:teuthology.orchestra.run.smithi190.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 582 kB/s | 49 kB 00:00 2024-03-11T18:57:25.621 INFO:teuthology.orchestra.run.smithi170.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 485 kB/s | 44 kB 00:00 2024-03-11T18:57:25.700 INFO:teuthology.orchestra.run.smithi190.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 809 kB/s | 67 kB 00:00 2024-03-11T18:57:25.738 INFO:teuthology.orchestra.run.smithi170.stdout:(102/174): openssl-devel-3.0.7-27.el9.x86_64.rp 5.8 MB/s | 4.1 MB 00:00 2024-03-11T18:57:25.764 INFO:teuthology.orchestra.run.smithi170.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 661 kB/s | 94 kB 00:00 2024-03-11T18:57:25.784 INFO:teuthology.orchestra.run.smithi190.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-03-11T18:57:25.875 INFO:teuthology.orchestra.run.smithi190.stdout:(113/174): protobuf-compiler-3.14.0-13.el9.x86_ 435 kB/s | 863 kB 00:01 2024-03-11T18:57:25.909 INFO:teuthology.orchestra.run.smithi190.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.5 MB/s | 838 kB 00:00 2024-03-11T18:57:25.910 INFO:teuthology.orchestra.run.smithi029.stdout:(78/174): ceph-test-19.0.0-1607.g94a53970.el9.x 7.1 MB/s | 47 MB 00:06 2024-03-11T18:57:25.930 INFO:teuthology.orchestra.run.smithi170.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 116 kB/s | 22 kB 00:00 2024-03-11T18:57:26.002 INFO:teuthology.orchestra.run.smithi029.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 46 kB/s | 39 kB 00:00 2024-03-11T18:57:26.068 INFO:teuthology.orchestra.run.smithi190.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 274 kB/s | 43 kB 00:00 2024-03-11T18:57:26.143 INFO:teuthology.orchestra.run.smithi190.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 430 kB/s | 32 kB 00:00 2024-03-11T18:57:26.218 INFO:teuthology.orchestra.run.smithi190.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 200 kB/s | 15 kB 00:00 2024-03-11T18:57:26.252 INFO:teuthology.orchestra.run.smithi190.stdout:(118/174): python3-asyncssh-2.13.2-4.el9.noarch 1.4 MB/s | 548 kB 00:00 2024-03-11T18:57:26.264 INFO:teuthology.orchestra.run.smithi170.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.7 MB/s | 863 kB 00:00 2024-03-11T18:57:26.311 INFO:teuthology.orchestra.run.smithi190.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-03-11T18:57:26.336 INFO:teuthology.orchestra.run.smithi029.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-03-11T18:57:26.370 INFO:teuthology.orchestra.run.smithi029.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 90 kB/s | 41 kB 00:00 2024-03-11T18:57:26.377 INFO:teuthology.orchestra.run.smithi190.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-03-11T18:57:26.381 INFO:teuthology.orchestra.run.smithi170.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.2 MB/s | 548 kB 00:00 2024-03-11T18:57:26.403 INFO:teuthology.orchestra.run.smithi029.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 754 kB/s | 300 kB 00:00 2024-03-11T18:57:26.406 INFO:teuthology.orchestra.run.smithi170.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 2.1 MB/s | 309 kB 00:00 2024-03-11T18:57:26.429 INFO:teuthology.orchestra.run.smithi029.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 615 kB/s | 56 kB 00:00 2024-03-11T18:57:26.444 INFO:teuthology.orchestra.run.smithi190.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-03-11T18:57:26.448 INFO:teuthology.orchestra.run.smithi170.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 294 kB/s | 19 kB 00:00 2024-03-11T18:57:26.454 INFO:teuthology.orchestra.run.smithi029.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 875 kB/s | 44 kB 00:00 2024-03-11T18:57:26.480 INFO:teuthology.orchestra.run.smithi029.stdout:(85/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 825 kB/s | 63 kB 00:00 2024-03-11T18:57:26.486 INFO:teuthology.orchestra.run.smithi190.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 325 kB/s | 35 kB 00:00 2024-03-11T18:57:26.505 INFO:teuthology.orchestra.run.smithi029.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-11T18:57:26.528 INFO:teuthology.orchestra.run.smithi190.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-03-11T18:57:26.539 INFO:teuthology.orchestra.run.smithi029.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.8 MB/s | 241 kB 00:00 2024-03-11T18:57:26.566 INFO:teuthology.orchestra.run.smithi029.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 103 kB 00:00 2024-03-11T18:57:26.616 INFO:teuthology.orchestra.run.smithi029.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 10 MB/s | 1.1 MB 00:00 2024-03-11T18:57:26.642 INFO:teuthology.orchestra.run.smithi029.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 124 kB 00:00 2024-03-11T18:57:26.653 INFO:teuthology.orchestra.run.smithi190.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-03-11T18:57:26.674 INFO:teuthology.orchestra.run.smithi170.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 115 kB/s | 26 kB 00:00 2024-03-11T18:57:26.675 INFO:teuthology.orchestra.run.smithi029.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 1.7 MB/s | 187 kB 00:00 2024-03-11T18:57:26.749 INFO:teuthology.orchestra.run.smithi170.stdout:(110/174): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-03-11T18:57:26.759 INFO:teuthology.orchestra.run.smithi029.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-03-11T18:57:26.762 INFO:teuthology.orchestra.run.smithi190.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 7.4 MB/s | 2.0 MB 00:00 2024-03-11T18:57:26.774 INFO:teuthology.orchestra.run.smithi170.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 488 kB/s | 49 kB 00:00 2024-03-11T18:57:26.787 INFO:teuthology.orchestra.run.smithi190.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 80 kB/s | 11 kB 00:00 2024-03-11T18:57:26.793 INFO:teuthology.orchestra.run.smithi029.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-11T18:57:26.816 INFO:teuthology.orchestra.run.smithi170.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-11T18:57:26.842 INFO:teuthology.orchestra.run.smithi170.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-03-11T18:57:26.844 INFO:teuthology.orchestra.run.smithi029.stdout:(94/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 803 kB 00:00 2024-03-11T18:57:26.900 INFO:teuthology.orchestra.run.smithi170.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-11T18:57:26.903 INFO:teuthology.orchestra.run.smithi029.stdout:(95/174): python3-requests-2.25.1-8.el9.noarch. 875 kB/s | 125 kB 00:00 2024-03-11T18:57:26.926 INFO:teuthology.orchestra.run.smithi170.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-03-11T18:57:26.963 INFO:teuthology.orchestra.run.smithi190.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 88 kB/s | 18 kB 00:00 2024-03-11T18:57:26.967 INFO:teuthology.orchestra.run.smithi170.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 660 kB/s | 43 kB 00:00 2024-03-11T18:57:26.988 INFO:teuthology.orchestra.run.smithi190.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 115 kB/s | 23 kB 00:00 2024-03-11T18:57:26.993 INFO:teuthology.orchestra.run.smithi170.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 482 kB/s | 32 kB 00:00 2024-03-11T18:57:27.035 INFO:teuthology.orchestra.run.smithi170.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 223 kB/s | 15 kB 00:00 2024-03-11T18:57:27.045 INFO:teuthology.orchestra.run.smithi029.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 85 kB/s | 21 kB 00:00 2024-03-11T18:57:27.055 INFO:teuthology.orchestra.run.smithi190.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 212 kB/s | 19 kB 00:00 2024-03-11T18:57:27.060 INFO:teuthology.orchestra.run.smithi170.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-03-11T18:57:27.080 INFO:teuthology.orchestra.run.smithi190.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-03-11T18:57:27.111 INFO:teuthology.orchestra.run.smithi170.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-03-11T18:57:27.112 INFO:teuthology.orchestra.run.smithi029.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 106 kB/s | 25 kB 00:00 2024-03-11T18:57:27.136 INFO:teuthology.orchestra.run.smithi170.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-03-11T18:57:27.156 INFO:teuthology.orchestra.run.smithi190.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 619 kB/s | 46 kB 00:00 2024-03-11T18:57:27.177 INFO:teuthology.orchestra.run.smithi170.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 534 kB/s | 35 kB 00:00 2024-03-11T18:57:27.203 INFO:teuthology.orchestra.run.smithi170.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-03-11T18:57:27.239 INFO:teuthology.orchestra.run.smithi190.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 945 kB/s | 79 kB 00:00 2024-03-11T18:57:27.290 INFO:teuthology.orchestra.run.smithi190.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 4.4 MB/s | 1.0 MB 00:00 2024-03-11T18:57:27.303 INFO:teuthology.orchestra.run.smithi170.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 16 MB/s | 2.0 MB 00:00 2024-03-11T18:57:27.315 INFO:teuthology.orchestra.run.smithi190.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 769 kB/s | 58 kB 00:00 2024-03-11T18:57:27.328 INFO:teuthology.orchestra.run.smithi170.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-03-11T18:57:27.370 INFO:teuthology.orchestra.run.smithi170.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-03-11T18:57:27.387 INFO:teuthology.orchestra.run.smithi170.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 301 kB/s | 18 kB 00:00 2024-03-11T18:57:27.399 INFO:teuthology.orchestra.run.smithi190.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-03-11T18:57:27.416 INFO:teuthology.orchestra.run.smithi190.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-03-11T18:57:27.429 INFO:teuthology.orchestra.run.smithi170.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 396 kB/s | 23 kB 00:00 2024-03-11T18:57:27.446 INFO:teuthology.orchestra.run.smithi170.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 331 kB/s | 19 kB 00:00 2024-03-11T18:57:27.475 INFO:teuthology.orchestra.run.smithi190.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-11T18:57:27.488 INFO:teuthology.orchestra.run.smithi170.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 336 kB/s | 20 kB 00:00 2024-03-11T18:57:27.491 INFO:teuthology.orchestra.run.smithi190.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 375 kB/s | 28 kB 00:00 2024-03-11T18:57:27.521 INFO:teuthology.orchestra.run.smithi029.stdout:(98/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 6.7 MB/s | 4.1 MB 00:00 2024-03-11T18:57:27.530 INFO:teuthology.orchestra.run.smithi170.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-03-11T18:57:27.547 INFO:teuthology.orchestra.run.smithi170.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 790 kB/s | 46 kB 00:00 2024-03-11T18:57:27.549 INFO:teuthology.orchestra.run.smithi190.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 409 kB/s | 31 kB 00:00 2024-03-11T18:57:27.556 INFO:teuthology.orchestra.run.smithi029.stdout:(99/174): python3-devel-3.9.17-2.el9.x86_64.rpm 482 kB/s | 245 kB 00:00 2024-03-11T18:57:27.583 INFO:teuthology.orchestra.run.smithi190.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-03-11T18:57:27.588 INFO:teuthology.orchestra.run.smithi029.stdout:(100/174): python3-jinja2-2.11.3-5.el9.noarch.r 551 kB/s | 248 kB 00:00 2024-03-11T18:57:27.597 INFO:teuthology.orchestra.run.smithi170.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-03-11T18:57:27.617 INFO:teuthology.orchestra.run.smithi190.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 887 kB/s | 59 kB 00:00 2024-03-11T18:57:27.622 INFO:teuthology.orchestra.run.smithi170.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 767 kB/s | 58 kB 00:00 2024-03-11T18:57:27.667 INFO:teuthology.orchestra.run.smithi190.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 427 kB/s | 36 kB 00:00 2024-03-11T18:57:27.681 INFO:teuthology.orchestra.run.smithi170.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-11T18:57:27.701 INFO:teuthology.orchestra.run.smithi190.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 610 kB/s | 51 kB 00:00 2024-03-11T18:57:27.706 INFO:teuthology.orchestra.run.smithi170.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-03-11T18:57:27.749 INFO:teuthology.orchestra.run.smithi170.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-03-11T18:57:27.759 INFO:teuthology.orchestra.run.smithi190.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-11T18:57:27.774 INFO:teuthology.orchestra.run.smithi170.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 421 kB/s | 28 kB 00:00 2024-03-11T18:57:27.785 INFO:teuthology.orchestra.run.smithi190.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-03-11T18:57:27.816 INFO:teuthology.orchestra.run.smithi170.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 460 kB/s | 31 kB 00:00 2024-03-11T18:57:27.831 INFO:teuthology.orchestra.run.smithi029.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 126 kB/s | 35 kB 00:00 2024-03-11T18:57:27.833 INFO:teuthology.orchestra.run.smithi170.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 3.2 MB/s | 188 kB 00:00 2024-03-11T18:57:27.860 INFO:teuthology.orchestra.run.smithi190.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 267 kB/s | 20 kB 00:00 2024-03-11T18:57:27.883 INFO:teuthology.orchestra.run.smithi170.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 884 kB/s | 59 kB 00:00 2024-03-11T18:57:27.893 INFO:teuthology.orchestra.run.smithi190.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-03-11T18:57:27.898 INFO:teuthology.orchestra.run.smithi029.stdout:(102/174): qatlib-23.11.0-1.el9.x86_64.rpm 575 kB/s | 216 kB 00:00 2024-03-11T18:57:27.908 INFO:teuthology.orchestra.run.smithi170.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 474 kB/s | 36 kB 00:00 2024-03-11T18:57:27.949 INFO:teuthology.orchestra.run.smithi029.stdout:(103/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 145 kB/s | 44 kB 00:00 2024-03-11T18:57:27.950 INFO:teuthology.orchestra.run.smithi170.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 765 kB/s | 51 kB 00:00 2024-03-11T18:57:27.952 INFO:teuthology.orchestra.run.smithi190.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-11T18:57:27.975 INFO:teuthology.orchestra.run.smithi170.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-11T18:57:28.018 INFO:teuthology.orchestra.run.smithi170.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-03-11T18:57:28.057 INFO:teuthology.orchestra.run.smithi029.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 168 kB/s | 22 kB 00:00 2024-03-11T18:57:28.068 INFO:teuthology.orchestra.run.smithi170.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-03-11T18:57:28.077 INFO:teuthology.orchestra.run.smithi190.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-03-11T18:57:28.094 INFO:teuthology.orchestra.run.smithi170.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-03-11T18:57:28.135 INFO:teuthology.orchestra.run.smithi170.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-11T18:57:28.184 INFO:teuthology.orchestra.run.smithi029.stdout:(105/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 265 kB/s | 94 kB 00:00 2024-03-11T18:57:28.202 INFO:teuthology.orchestra.run.smithi170.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-11T18:57:28.352 INFO:teuthology.orchestra.run.smithi029.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-03-11T18:57:28.710 INFO:teuthology.orchestra.run.smithi029.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 54 kB/s | 19 kB 00:00 2024-03-11T18:57:28.744 INFO:teuthology.orchestra.run.smithi029.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 798 kB/s | 548 kB 00:00 2024-03-11T18:57:28.777 INFO:teuthology.orchestra.run.smithi190.stdout:(150/174): librados2-19.0.0-1607.g94a53970.el9. 4.1 MB/s | 3.4 MB 00:00 2024-03-11T18:57:28.819 INFO:teuthology.orchestra.run.smithi190.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-11T18:57:28.820 INFO:teuthology.orchestra.run.smithi029.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 486 kB/s | 309 kB 00:00 2024-03-11T18:57:28.845 INFO:teuthology.orchestra.run.smithi029.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 257 kB/s | 26 kB 00:00 2024-03-11T18:57:28.929 INFO:teuthology.orchestra.run.smithi029.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 446 kB/s | 49 kB 00:00 2024-03-11T18:57:28.955 INFO:teuthology.orchestra.run.smithi029.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 618 kB/s | 67 kB 00:00 2024-03-11T18:57:29.047 INFO:teuthology.orchestra.run.smithi029.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-03-11T18:57:29.164 INFO:teuthology.orchestra.run.smithi029.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 838 kB 00:00 2024-03-11T18:57:29.186 INFO:teuthology.orchestra.run.smithi170.stdout:(150/174): ceph-test-19.0.0-1607.g94a53970.el9. 3.1 MB/s | 47 MB 00:15 2024-03-11T18:57:29.227 INFO:teuthology.orchestra.run.smithi170.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-11T18:57:29.256 INFO:teuthology.orchestra.run.smithi029.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 474 kB/s | 43 kB 00:00 2024-03-11T18:57:29.290 INFO:teuthology.orchestra.run.smithi029.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 2.2 MB/s | 548 kB 00:00 2024-03-11T18:57:29.349 INFO:teuthology.orchestra.run.smithi029.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 348 kB/s | 32 kB 00:00 2024-03-11T18:57:29.383 INFO:teuthology.orchestra.run.smithi029.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 162 kB/s | 15 kB 00:00 2024-03-11T18:57:29.386 INFO:teuthology.orchestra.run.smithi190.stdout:(152/174): librbd1-19.0.0-1607.g94a53970.el9.x8 2.4 MB/s | 3.1 MB 00:01 2024-03-11T18:57:29.394 INFO:teuthology.orchestra.run.smithi170.stdout:(152/174): librados2-19.0.0-1607.g94a53970.el9. 2.7 MB/s | 3.4 MB 00:01 2024-03-11T18:57:29.475 INFO:teuthology.orchestra.run.smithi029.stdout:(119/174): libarrow-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 4.4 MB 00:00 2024-03-11T18:57:29.511 INFO:teuthology.orchestra.run.smithi029.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.0 MB/s | 172 kB 00:00 2024-03-11T18:57:29.586 INFO:teuthology.orchestra.run.smithi029.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 1.7 MB/s | 358 kB 00:00 2024-03-11T18:57:29.612 INFO:teuthology.orchestra.run.smithi029.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 351 kB/s | 35 kB 00:00 2024-03-11T18:57:29.645 INFO:teuthology.orchestra.run.smithi029.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.7 MB/s | 303 kB 00:00 2024-03-11T18:57:29.696 INFO:teuthology.orchestra.run.smithi029.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 1.8 MB/s | 203 kB 00:00 2024-03-11T18:57:29.746 INFO:teuthology.orchestra.run.smithi029.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-11T18:57:29.770 INFO:teuthology.orchestra.run.smithi190.stdout:(153/174): NetworkManager-1.46.0-1.el9.x86_64.r 2.4 MB/s | 2.3 MB 00:00 2024-03-11T18:57:29.797 INFO:teuthology.orchestra.run.smithi029.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 106 kB/s | 11 kB 00:00 2024-03-11T18:57:29.864 INFO:teuthology.orchestra.run.smithi029.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 8.1 MB/s | 2.0 MB 00:00 2024-03-11T18:57:29.878 INFO:teuthology.orchestra.run.smithi190.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 392 kB/s | 42 kB 00:00 2024-03-11T18:57:29.882 INFO:teuthology.orchestra.run.smithi029.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 131 kB/s | 18 kB 00:00 2024-03-11T18:57:29.915 INFO:teuthology.orchestra.run.smithi029.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 196 kB/s | 23 kB 00:00 2024-03-11T18:57:29.966 INFO:teuthology.orchestra.run.smithi029.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 192 kB/s | 19 kB 00:00 2024-03-11T18:57:29.978 INFO:teuthology.orchestra.run.smithi190.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 2.4 MB/s | 248 kB 00:00 2024-03-11T18:57:29.991 INFO:teuthology.orchestra.run.smithi029.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 180 kB/s | 20 kB 00:00 2024-03-11T18:57:30.059 INFO:teuthology.orchestra.run.smithi029.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 502 kB/s | 46 kB 00:00 2024-03-11T18:57:30.087 INFO:teuthology.orchestra.run.smithi190.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 659 kB/s | 71 kB 00:00 2024-03-11T18:57:30.101 INFO:teuthology.orchestra.run.smithi029.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 5.5 MB/s | 1.0 MB 00:00 2024-03-11T18:57:30.103 INFO:teuthology.orchestra.run.smithi170.stdout:(153/174): NetworkManager-1.46.0-1.el9.x86_64.r 2.6 MB/s | 2.3 MB 00:00 2024-03-11T18:57:30.126 INFO:teuthology.orchestra.run.smithi029.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 586 kB/s | 79 kB 00:00 2024-03-11T18:57:30.145 INFO:teuthology.orchestra.run.smithi170.stdout:(154/174): NetworkManager-libnm-1.46.0-1.el9.x8 2.4 MB/s | 1.8 MB 00:00 2024-03-11T18:57:30.152 INFO:teuthology.orchestra.run.smithi029.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 622 kB/s | 58 kB 00:00 2024-03-11T18:57:30.196 INFO:teuthology.orchestra.run.smithi190.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 336 kB/s | 36 kB 00:00 2024-03-11T18:57:30.212 INFO:teuthology.orchestra.run.smithi170.stdout:(155/174): NetworkManager-team-1.46.0-1.el9.x86 391 kB/s | 42 kB 00:00 2024-03-11T18:57:30.212 INFO:teuthology.orchestra.run.smithi029.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-11T18:57:30.238 INFO:teuthology.orchestra.run.smithi190.stdout:(158/174): NetworkManager-libnm-1.46.0-1.el9.x8 2.1 MB/s | 1.8 MB 00:00 2024-03-11T18:57:30.238 INFO:teuthology.orchestra.run.smithi029.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 148 kB/s | 16 kB 00:00 2024-03-11T18:57:30.254 INFO:teuthology.orchestra.run.smithi170.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 2.2 MB/s | 248 kB 00:00 2024-03-11T18:57:30.263 INFO:teuthology.orchestra.run.smithi029.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 809 kB/s | 90 kB 00:00 2024-03-11T18:57:30.305 INFO:teuthology.orchestra.run.smithi190.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.7 MB/s | 192 kB 00:00 2024-03-11T18:57:30.314 INFO:teuthology.orchestra.run.smithi029.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 279 kB/s | 28 kB 00:00 2024-03-11T18:57:30.337 INFO:teuthology.orchestra.run.smithi170.stdout:(157/174): librbd1-19.0.0-1607.g94a53970.el9.x8 1.5 MB/s | 3.1 MB 00:02 2024-03-11T18:57:30.339 INFO:teuthology.orchestra.run.smithi029.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 303 kB/s | 31 kB 00:00 2024-03-11T18:57:30.347 INFO:teuthology.orchestra.run.smithi190.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-11T18:57:30.363 INFO:teuthology.orchestra.run.smithi170.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 335 kB/s | 36 kB 00:00 2024-03-11T18:57:30.365 INFO:teuthology.orchestra.run.smithi029.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-03-11T18:57:30.379 INFO:teuthology.orchestra.run.smithi170.stdout:(159/174): acl-2.3.1-4.el9.x86_64.rpm 426 kB/s | 71 kB 00:00 2024-03-11T18:57:30.405 INFO:teuthology.orchestra.run.smithi190.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 230 kB/s | 23 kB 00:00 2024-03-11T18:57:30.415 INFO:teuthology.orchestra.run.smithi029.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 584 kB/s | 59 kB 00:00 2024-03-11T18:57:30.441 INFO:teuthology.orchestra.run.smithi029.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 353 kB/s | 36 kB 00:00 2024-03-11T18:57:30.447 INFO:teuthology.orchestra.run.smithi190.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 108 kB 00:00 2024-03-11T18:57:30.466 INFO:teuthology.orchestra.run.smithi029.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 503 kB/s | 51 kB 00:00 2024-03-11T18:57:30.480 INFO:teuthology.orchestra.run.smithi170.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.1 MB/s | 256 kB 00:00 2024-03-11T18:57:30.505 INFO:teuthology.orchestra.run.smithi170.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 184 kB/s | 23 kB 00:00 2024-03-11T18:57:30.514 INFO:teuthology.orchestra.run.smithi190.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 155 kB 00:00 2024-03-11T18:57:30.525 INFO:teuthology.orchestra.run.smithi029.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.0 MB/s | 230 kB 00:00 2024-03-11T18:57:30.548 INFO:teuthology.orchestra.run.smithi190.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 136 kB 00:00 2024-03-11T18:57:30.551 INFO:teuthology.orchestra.run.smithi029.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 820 kB/s | 90 kB 00:00 2024-03-11T18:57:30.585 INFO:teuthology.orchestra.run.smithi029.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.5 MB/s | 427 kB 00:00 2024-03-11T18:57:30.588 INFO:teuthology.orchestra.run.smithi170.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.0 MB/s | 108 kB 00:00 2024-03-11T18:57:30.614 INFO:teuthology.orchestra.run.smithi170.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 155 kB 00:00 2024-03-11T18:57:30.615 INFO:teuthology.orchestra.run.smithi190.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 637 kB/s | 64 kB 00:00 2024-03-11T18:57:30.627 INFO:teuthology.orchestra.run.smithi029.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 197 kB/s | 20 kB 00:00 2024-03-11T18:57:30.648 INFO:teuthology.orchestra.run.smithi190.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 286 kB/s | 29 kB 00:00 2024-03-11T18:57:30.652 INFO:teuthology.orchestra.run.smithi029.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.8 MB/s | 192 kB 00:00 2024-03-11T18:57:30.698 INFO:teuthology.orchestra.run.smithi170.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 136 kB 00:00 2024-03-11T18:57:30.723 INFO:teuthology.orchestra.run.smithi170.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 587 kB/s | 64 kB 00:00 2024-03-11T18:57:30.728 INFO:teuthology.orchestra.run.smithi029.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-11T18:57:30.740 INFO:teuthology.orchestra.run.smithi190.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 9.7 MB/s | 1.2 MB 00:00 2024-03-11T18:57:30.778 INFO:teuthology.orchestra.run.smithi029.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-11T18:57:30.798 INFO:teuthology.orchestra.run.smithi170.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 286 kB/s | 29 kB 00:00 2024-03-11T18:57:30.849 INFO:teuthology.orchestra.run.smithi170.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 9.7 MB/s | 1.2 MB 00:00 2024-03-11T18:57:30.874 INFO:teuthology.orchestra.run.smithi170.stdout:(168/174): elfutils-libelf-0.190-2.el9.x86_64.r 359 kB/s | 192 kB 00:00 2024-03-11T18:57:30.981 INFO:teuthology.orchestra.run.smithi029.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 11 MB/s | 2.3 MB 00:00 2024-03-11T18:57:31.016 INFO:teuthology.orchestra.run.smithi170.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 9.9 MB/s | 2.2 MB 00:00 2024-03-11T18:57:31.073 INFO:teuthology.orchestra.run.smithi029.stdout:(153/174): librbd1-19.0.0-1607.g94a53970.el9.x8 7.5 MB/s | 3.1 MB 00:00 2024-03-11T18:57:31.116 INFO:teuthology.orchestra.run.smithi170.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 311 kB/s | 31 kB 00:00 2024-03-11T18:57:31.132 INFO:teuthology.orchestra.run.smithi029.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 727 kB/s | 42 kB 00:00 2024-03-11T18:57:31.166 INFO:teuthology.orchestra.run.smithi170.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 1.6 MB/s | 465 kB 00:00 2024-03-11T18:57:31.240 INFO:teuthology.orchestra.run.smithi029.stdout:(155/174): librados2-19.0.0-1607.g94a53970.el9. 5.5 MB/s | 3.4 MB 00:00 2024-03-11T18:57:31.250 INFO:teuthology.orchestra.run.smithi170.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 892 kB/s | 74 kB 00:00 2024-03-11T18:57:31.316 INFO:teuthology.orchestra.run.smithi029.stdout:(156/174): NetworkManager-libnm-1.46.0-1.el9.x8 5.4 MB/s | 1.8 MB 00:00 2024-03-11T18:57:31.350 INFO:teuthology.orchestra.run.smithi029.stdout:(157/174): NetworkManager-tui-1.46.0-1.el9.x86_ 1.1 MB/s | 248 kB 00:00 2024-03-11T18:57:31.375 INFO:teuthology.orchestra.run.smithi029.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 531 kB/s | 71 kB 00:00 2024-03-11T18:57:31.392 INFO:teuthology.orchestra.run.smithi029.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 478 kB/s | 36 kB 00:00 2024-03-11T18:57:31.408 INFO:teuthology.orchestra.run.smithi170.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.1 MB/s | 2.3 MB 00:00 2024-03-11T18:57:31.424 INFO:teuthology.orchestra.run.smithi190.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 2.8 MB/s | 2.2 MB 00:00 2024-03-11T18:57:31.451 INFO:teuthology.orchestra.run.smithi029.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.9 MB/s | 192 kB 00:00 2024-03-11T18:57:31.469 INFO:teuthology.orchestra.run.smithi029.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 304 kB/s | 23 kB 00:00 2024-03-11T18:57:31.474 INFO:teuthology.orchestra.run.smithi190.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 3.1 MB/s | 2.3 MB 00:00 2024-03-11T18:57:31.502 INFO:teuthology.orchestra.run.smithi029.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.0 MB/s | 256 kB 00:00 2024-03-11T18:57:31.545 INFO:teuthology.orchestra.run.smithi029.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 108 kB 00:00 2024-03-11T18:57:31.761 INFO:teuthology.orchestra.run.smithi029.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 530 kB/s | 155 kB 00:00 2024-03-11T18:57:31.891 INFO:teuthology.orchestra.run.smithi190.stdout:(170/174): ceph-test-19.0.0-1607.g94a53970.el9. 2.5 MB/s | 47 MB 00:19 2024-03-11T18:57:31.916 INFO:teuthology.orchestra.run.smithi190.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 70 kB/s | 31 kB 00:00 2024-03-11T18:57:31.929 INFO:teuthology.orchestra.run.smithi029.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 318 kB/s | 136 kB 00:00 2024-03-11T18:57:31.954 INFO:teuthology.orchestra.run.smithi029.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 156 kB/s | 64 kB 00:00 2024-03-11T18:57:31.973 INFO:teuthology.orchestra.run.smithi029.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 135 kB/s | 29 kB 00:00 2024-03-11T18:57:32.025 INFO:teuthology.orchestra.run.smithi190.stdout:(172/174): util-linux-core-2.37.4-18.el9.x86_64 775 kB/s | 465 kB 00:00 2024-03-11T18:57:32.050 INFO:teuthology.orchestra.run.smithi190.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 557 kB/s | 74 kB 00:00 2024-03-11T18:57:32.099 INFO:teuthology.orchestra.run.smithi029.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 7.2 MB/s | 1.2 MB 00:00 2024-03-11T18:57:32.191 INFO:teuthology.orchestra.run.smithi029.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 9.1 MB/s | 2.2 MB 00:00 2024-03-11T18:57:32.250 INFO:teuthology.orchestra.run.smithi029.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 8.3 MB/s | 2.3 MB 00:00 2024-03-11T18:57:32.284 INFO:teuthology.orchestra.run.smithi029.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 2.5 MB/s | 465 kB 00:00 2024-03-11T18:57:32.309 INFO:teuthology.orchestra.run.smithi029.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 264 kB/s | 31 kB 00:00 2024-03-11T18:57:32.418 INFO:teuthology.orchestra.run.smithi029.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 6.4 MB/s | 1.1 MB 00:00 2024-03-11T18:57:32.575 INFO:teuthology.orchestra.run.smithi170.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 753 kB/s | 1.1 MB 00:01 2024-03-11T18:57:32.581 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-11T18:57:32.581 INFO:teuthology.orchestra.run.smithi170.stdout:Total 8.8 MB/s | 223 MB 00:25 2024-03-11T18:57:32.610 INFO:teuthology.orchestra.run.smithi029.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 228 kB/s | 74 kB 00:00 2024-03-11T18:57:32.618 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-11T18:57:32.618 INFO:teuthology.orchestra.run.smithi029.stdout:Total 11 MB/s | 223 MB 00:20 2024-03-11T18:57:32.750 INFO:teuthology.orchestra.run.smithi190.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 1.3 MB/s | 1.1 MB 00:00 2024-03-11T18:57:32.755 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-11T18:57:32.755 INFO:teuthology.orchestra.run.smithi190.stdout:Total 9.0 MB/s | 223 MB 00:24 2024-03-11T18:57:33.061 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-11T18:57:33.103 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-11T18:57:33.217 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-11T18:57:33.369 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-11T18:57:33.369 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-11T18:57:33.434 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-11T18:57:33.434 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-11T18:57:33.525 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-11T18:57:33.526 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-11T18:57:35.372 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-11T18:57:35.373 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-11T18:57:35.518 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-11T18:57:35.518 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-11T18:57:35.577 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-11T18:57:35.577 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-11T18:57:42.566 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-11T18:57:42.663 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-11T18:57:42.667 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-11T18:57:42.689 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-11T18:57:42.735 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-11T18:57:42.746 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-11T18:57:42.779 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-11T18:57:42.790 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-11T18:57:42.805 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-11T18:57:42.843 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-11T18:57:42.847 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-11T18:57:42.853 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-11T18:57:42.885 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-11T18:57:42.901 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-11T18:57:42.911 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-11T18:57:42.955 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-11T18:57:42.956 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-11T18:57:42.979 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-11T18:57:42.997 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-11T18:57:43.022 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-11T18:57:43.068 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-11T18:57:43.102 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-11T18:57:43.154 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-11T18:57:43.184 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-11T18:57:43.194 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-11T18:57:43.243 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-11T18:57:43.279 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-11T18:57:43.298 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-11T18:57:43.337 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T18:57:43.339 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-11T18:57:43.373 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-11T18:57:43.391 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-11T18:57:43.449 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T18:57:43.458 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T18:57:43.488 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T18:57:43.570 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T18:57:43.603 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T18:57:43.616 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T18:57:43.619 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-11T18:57:43.638 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-11T18:57:43.685 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-11T18:57:43.715 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T18:57:43.749 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-11T18:57:43.794 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-11T18:57:43.862 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-11T18:57:43.915 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-11T18:57:43.951 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T18:57:43.969 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-11T18:57:43.970 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-11T18:57:44.018 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-11T18:57:44.046 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T18:57:44.053 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T18:57:44.077 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-11T18:57:44.118 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T18:57:44.134 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-11T18:57:44.149 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T18:57:44.172 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-11T18:57:44.173 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-11T18:57:44.195 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-11T18:57:44.211 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-11T18:57:44.233 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T18:57:44.254 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-11T18:57:44.261 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T18:57:44.292 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-11T18:57:44.315 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-11T18:57:44.331 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-11T18:57:44.342 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T18:57:44.390 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-11T18:57:44.395 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-11T18:57:44.428 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-11T18:57:44.447 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-11T18:57:44.478 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-11T18:57:44.500 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-11T18:57:44.531 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-11T18:57:44.570 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-11T18:57:44.637 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-11T18:57:44.657 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-11T18:57:44.695 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-11T18:57:44.731 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T18:57:44.753 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-11T18:57:44.787 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-11T18:57:44.836 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T18:57:44.855 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-11T18:57:44.866 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-11T18:57:44.887 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-11T18:57:44.934 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-11T18:57:44.948 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-11T18:57:44.961 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T18:57:44.973 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-11T18:57:44.998 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T18:57:45.004 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-11T18:57:45.020 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-11T18:57:45.054 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T18:57:45.099 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T18:57:45.231 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-11T18:57:45.238 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T18:57:45.263 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-11T18:57:45.311 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-11T18:57:45.312 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T18:57:45.343 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-11T18:57:45.420 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-11T18:57:45.441 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-11T18:57:45.487 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-11T18:57:45.521 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-11T18:57:45.523 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-11T18:57:45.554 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-11T18:57:45.557 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-11T18:57:45.590 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-11T18:57:45.604 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-11T18:57:45.632 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-11T18:57:45.633 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-11T18:57:45.686 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-11T18:57:45.705 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-11T18:57:45.723 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-11T18:57:45.740 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-11T18:57:45.760 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-11T18:57:45.789 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-11T18:57:45.807 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-11T18:57:45.807 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-11T18:57:45.823 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-11T18:57:45.866 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-11T18:57:45.870 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-11T18:57:45.900 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-11T18:57:45.914 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-11T18:57:45.928 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-11T18:57:45.957 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-11T18:57:45.974 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T18:57:45.990 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-11T18:57:45.995 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-11T18:57:46.018 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-11T18:57:46.031 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 39/198 2024-03-11T18:57:46.048 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T18:57:46.074 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 40/198 2024-03-11T18:57:46.103 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-11T18:57:46.104 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 41/198 2024-03-11T18:57:46.139 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 39/198 2024-03-11T18:57:46.174 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-11T18:57:46.182 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 40/198 2024-03-11T18:57:46.218 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-11T18:57:46.222 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 41/198 2024-03-11T18:57:46.280 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T18:57:46.288 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-11T18:57:46.293 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-11T18:57:46.334 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-11T18:57:46.390 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-11T18:57:46.406 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-11T18:57:46.451 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-11T18:57:46.504 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-11T18:57:46.536 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-11T18:57:46.540 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-11T18:57:46.571 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-11T18:57:46.595 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-11T18:57:46.599 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-11T18:57:46.645 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-11T18:57:46.730 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-11T18:57:46.793 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-11T18:57:46.830 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-11T18:57:46.905 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-11T18:57:46.947 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-11T18:57:46.991 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-11T18:57:47.029 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-11T18:57:47.034 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-11T18:57:47.070 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-11T18:57:47.108 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-11T18:57:47.108 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-11T18:57:47.151 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-11T18:57:47.189 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-11T18:57:47.193 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 39/198 2024-03-11T18:57:47.213 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-11T18:57:47.237 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 40/198 2024-03-11T18:57:47.266 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 41/198 2024-03-11T18:57:47.269 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-11T18:57:47.278 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-11T18:57:47.292 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-11T18:57:47.310 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-11T18:57:47.329 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-11T18:57:47.337 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-11T18:57:47.338 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-11T18:57:47.358 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-11T18:57:47.370 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-11T18:57:47.392 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-11T18:57:47.396 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 53/198 2024-03-11T18:57:47.409 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-11T18:57:47.426 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-11T18:57:47.447 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 53/198 2024-03-11T18:57:47.476 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-11T18:57:47.478 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-11T18:57:47.712 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-11T18:57:47.721 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-11T18:57:47.830 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-11T18:57:47.897 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-11T18:57:47.917 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-11T18:57:47.988 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-11T18:57:47.996 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-11T18:57:48.015 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-11T18:57:48.031 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-11T18:57:48.040 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-11T18:57:48.159 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-11T18:57:48.177 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-11T18:57:48.218 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-11T18:57:48.227 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-11T18:57:48.273 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-11T18:57:48.282 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-11T18:57:48.347 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-11T18:57:48.355 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-11T18:57:48.385 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-11T18:57:48.386 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-11T18:57:48.421 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-11T18:57:48.425 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-11T18:57:48.469 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-11T18:57:48.470 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-11T18:57:48.505 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-11T18:57:48.505 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-11T18:57:48.548 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-11T18:57:48.549 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-11T18:57:48.578 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-11T18:57:48.589 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-11T18:57:48.591 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-11T18:57:48.664 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-11T18:57:48.722 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-11T18:57:48.734 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-11T18:57:48.761 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-11T18:57:48.788 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-11T18:57:48.792 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-11T18:57:48.799 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-11T18:57:48.818 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-11T18:57:48.833 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-11T18:57:48.845 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-11T18:57:48.853 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-11T18:57:48.869 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-11T18:57:48.900 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-11T18:57:48.928 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-11T18:57:48.952 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-11T18:57:48.964 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-11T18:57:48.967 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-11T18:57:48.999 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 53/198 2024-03-11T18:57:49.002 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-11T18:57:49.007 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-11T18:57:49.033 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-11T18:57:49.039 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-11T18:57:49.042 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-11T18:57:49.058 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-11T18:57:49.067 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-11T18:57:49.092 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-11T18:57:49.135 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-11T18:57:49.168 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-11T18:57:49.258 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-11T18:57:49.302 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-11T18:57:49.322 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-11T18:57:49.367 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-11T18:57:49.416 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-11T18:57:49.417 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-11T18:57:49.459 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-11T18:57:49.464 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-11T18:57:49.508 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-11T18:57:49.520 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-11T18:57:49.540 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-11T18:57:49.564 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-11T18:57:49.626 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-11T18:57:49.688 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-11T18:57:49.703 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-11T18:57:49.752 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-11T18:57:49.753 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-11T18:57:49.797 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-11T18:57:49.831 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-11T18:57:49.882 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-11T18:57:49.886 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-11T18:57:49.918 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-11T18:57:49.937 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-11T18:57:49.943 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-11T18:57:49.950 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-11T18:57:49.981 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-11T18:57:49.988 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-11T18:57:50.017 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-11T18:57:50.018 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-11T18:57:50.025 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-11T18:57:50.049 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-11T18:57:50.052 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-11T18:57:50.064 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-11T18:57:50.086 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-11T18:57:50.091 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-11T18:57:50.102 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-11T18:57:50.120 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-11T18:57:50.122 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-11T18:57:50.150 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-11T18:57:50.152 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-11T18:57:50.159 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-11T18:57:50.191 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-11T18:57:50.196 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-11T18:57:50.221 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-11T18:57:50.228 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-11T18:57:50.247 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-11T18:57:50.263 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-11T18:57:50.296 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-11T18:57:50.297 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-11T18:57:50.298 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-11T18:57:50.331 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-11T18:57:50.337 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-11T18:57:50.364 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-11T18:57:50.397 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-11T18:57:50.406 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-11T18:57:50.438 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-11T18:57:50.440 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-11T18:57:50.486 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-11T18:57:50.491 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-11T18:57:50.529 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-11T18:57:50.555 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-11T18:57:50.567 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-11T18:57:50.597 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-11T18:57:50.612 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-11T18:57:50.619 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-11T18:57:50.652 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-11T18:57:50.655 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-11T18:57:50.686 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-11T18:57:50.695 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-11T18:57:50.699 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-11T18:57:50.731 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-11T18:57:50.735 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-11T18:57:50.774 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-11T18:57:50.795 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-11T18:57:50.808 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-11T18:57:50.834 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-11T18:57:50.841 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-11T18:57:50.844 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-11T18:57:50.888 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-11T18:57:50.924 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-11T18:57:51.060 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-11T18:57:51.168 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-11T18:57:51.216 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-11T18:57:51.275 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-11T18:57:51.341 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-11T18:57:51.543 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-11T18:57:51.567 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-11T18:57:51.600 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-11T18:57:51.625 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-11T18:57:51.638 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-11T18:57:51.658 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-11T18:57:51.685 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-11T18:57:51.706 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-11T18:57:51.713 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-11T18:57:51.743 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-11T18:57:51.762 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-11T18:57:51.785 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-11T18:57:51.785 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-11T18:57:51.817 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-11T18:57:51.818 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-11T18:57:51.849 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-11T18:57:51.857 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-11T18:57:51.894 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-11T18:57:51.897 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-11T18:57:51.929 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-11T18:57:51.969 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-11T18:57:51.989 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-11T18:57:52.001 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-11T18:57:52.027 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-11T18:57:52.041 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-11T18:57:52.072 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-11T18:57:52.103 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-11T18:57:52.105 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-11T18:57:52.162 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-11T18:57:52.203 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-11T18:57:52.232 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-11T18:57:52.237 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-11T18:57:52.279 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-11T18:57:52.322 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-11T18:57:52.415 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-11T18:57:52.507 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-11T18:57:52.565 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-11T18:57:52.605 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-11T18:57:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-11T18:57:52.733 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-11T18:57:52.762 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-11T18:57:52.819 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-11T18:57:52.848 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-11T18:57:53.089 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-11T18:57:53.193 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-11T18:57:53.586 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-11T18:57:53.646 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-11T18:57:53.681 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-11T18:57:53.730 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-11T18:57:53.831 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-11T18:57:53.868 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-11T18:57:54.052 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-11T18:57:54.053 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-11T18:57:54.091 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-11T18:57:54.128 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-11T18:57:54.159 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-11T18:57:54.172 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-11T18:57:54.190 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-11T18:57:54.217 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-11T18:57:54.428 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-11T18:57:54.522 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-11T18:57:54.525 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-11T18:57:54.553 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-11T18:57:54.565 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-11T18:57:54.583 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-11T18:57:54.611 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-11T18:57:54.655 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-11T18:57:54.686 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 125/198 2024-03-11T18:57:54.722 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-11T18:57:54.864 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-11T18:57:54.918 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-11T18:57:54.949 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-11T18:57:54.991 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-11T18:57:55.023 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-11T18:57:55.054 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 125/198 2024-03-11T18:57:55.100 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-11T18:57:55.175 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-11T18:57:55.201 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-11T18:57:55.272 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-11T18:57:55.294 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 129/198 2024-03-11T18:57:55.552 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-11T18:57:55.586 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-11T18:57:55.657 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-11T18:57:55.688 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 129/198 2024-03-11T18:57:56.294 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-11T18:57:56.391 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-11T18:57:56.423 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-11T18:57:56.455 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-11T18:57:56.483 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-11T18:57:56.818 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-11T18:57:56.848 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-11T18:57:56.940 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-11T18:57:56.971 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 125/198 2024-03-11T18:57:57.016 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-11T18:57:57.468 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T18:57:57.498 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-11T18:57:57.507 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T18:57:57.522 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-11T18:57:57.602 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-11T18:57:57.633 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 129/198 2024-03-11T18:57:57.853 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T18:57:57.901 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T18:57:58.069 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T18:57:58.111 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-11T18:57:58.204 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-11T18:57:58.204 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-11T18:57:58.204 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-11T18:57:58.204 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:57:58.237 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-11T18:57:58.448 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T18:57:58.489 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-11T18:57:58.582 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-11T18:57:58.582 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-11T18:57:58.583 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-11T18:57:58.583 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:57:58.622 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-11T18:57:59.948 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T18:57:59.994 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T18:58:00.564 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T18:58:00.610 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-11T18:58:00.695 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-11T18:58:00.695 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-11T18:58:00.695 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-11T18:58:00.695 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:58:00.738 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-11T18:58:09.661 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-11T18:58:09.661 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /sys 2024-03-11T18:58:09.661 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /proc 2024-03-11T18:58:09.661 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /mnt 2024-03-11T18:58:09.661 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /var/tmp 2024-03-11T18:58:09.661 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /home 2024-03-11T18:58:09.661 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /root 2024-03-11T18:58:09.661 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /tmp 2024-03-11T18:58:09.662 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:58:09.827 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-11T18:58:09.842 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-11T18:58:09.842 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:09.842 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-11T18:58:09.842 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:58:10.107 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-11T18:58:10.108 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /sys 2024-03-11T18:58:10.108 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /proc 2024-03-11T18:58:10.108 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /mnt 2024-03-11T18:58:10.108 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /var/tmp 2024-03-11T18:58:10.108 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /home 2024-03-11T18:58:10.108 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /root 2024-03-11T18:58:10.108 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /tmp 2024-03-11T18:58:10.108 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:58:10.270 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-11T18:58:10.285 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-11T18:58:10.285 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:10.285 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-11T18:58:10.286 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:58:11.288 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-11T18:58:11.313 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-11T18:58:11.313 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:11.314 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-11T18:58:11.314 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T18:58:11.314 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T18:58:11.314 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:58:11.522 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-11T18:58:11.544 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-11T18:58:11.544 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:11.544 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-11T18:58:11.544 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T18:58:11.544 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T18:58:11.544 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:58:11.765 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-11T18:58:11.791 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-11T18:58:11.791 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:11.791 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-11T18:58:11.791 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T18:58:11.791 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T18:58:11.791 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:58:11.981 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-11T18:58:11.999 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-11T18:58:12.003 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-11T18:58:12.003 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:12.003 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-11T18:58:12.003 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T18:58:12.003 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T18:58:12.003 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:58:12.022 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-11T18:58:12.022 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:12.022 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-11T18:58:12.022 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T18:58:12.022 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T18:58:12.022 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:58:12.091 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-11T18:58:12.458 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-11T18:58:12.481 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-11T18:58:12.481 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:12.481 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-11T18:58:12.481 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T18:58:12.481 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T18:58:12.481 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:58:12.484 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-11T18:58:12.514 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-11T18:58:12.545 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 139/198 2024-03-11T18:58:12.576 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-11T18:58:12.605 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 140/198 2024-03-11T18:58:12.629 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-11T18:58:12.862 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-11T18:58:12.889 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-11T18:58:12.977 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-11T18:58:13.008 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-11T18:58:13.039 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 139/198 2024-03-11T18:58:13.099 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 140/198 2024-03-11T18:58:13.123 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-11T18:58:13.354 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-11T18:58:13.382 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-11T18:58:13.847 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-11T18:58:13.881 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-11T18:58:13.979 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-11T18:58:14.134 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 144/198 2024-03-11T18:58:14.175 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-11T18:58:14.197 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-11T18:58:14.197 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:14.197 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-11T18:58:14.198 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T18:58:14.198 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T18:58:14.198 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:58:14.267 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-11T18:58:14.277 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-11T18:58:14.333 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 147/198 2024-03-11T18:58:14.338 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-11T18:58:14.366 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-11T18:58:14.466 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-11T18:58:14.619 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 144/198 2024-03-11T18:58:14.651 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-11T18:58:14.674 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-11T18:58:14.674 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:14.674 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-11T18:58:14.674 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T18:58:14.674 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T18:58:14.674 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:58:14.718 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-11T18:58:14.730 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-11T18:58:14.781 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 147/198 2024-03-11T18:58:15.185 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-11T18:58:15.211 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-11T18:58:15.233 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-11T18:58:15.233 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:15.234 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-11T18:58:15.234 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T18:58:15.234 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T18:58:15.234 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:58:15.278 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-11T18:58:15.300 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-11T18:58:15.300 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:15.300 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-11T18:58:15.300 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:58:15.345 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T18:58:15.367 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T18:58:15.368 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:15.368 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-11T18:58:15.368 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-11T18:58:15.368 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-11T18:58:15.368 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:58:15.639 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-11T18:58:15.670 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-11T18:58:15.680 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-11T18:58:15.693 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-11T18:58:15.694 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:15.694 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-11T18:58:15.694 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T18:58:15.694 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T18:58:15.694 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:58:15.702 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-11T18:58:15.702 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:15.702 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-11T18:58:15.702 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T18:58:15.702 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T18:58:15.702 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:58:15.738 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-11T18:58:15.759 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-11T18:58:15.760 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:15.760 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-11T18:58:15.760 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:58:15.805 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T18:58:15.828 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T18:58:15.828 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:15.828 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-11T18:58:15.829 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-11T18:58:15.829 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-11T18:58:15.829 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:58:16.148 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-11T18:58:16.170 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-11T18:58:16.171 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:16.171 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-11T18:58:16.171 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T18:58:16.171 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T18:58:16.171 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:58:17.019 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-11T18:58:17.019 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2024-03-11T18:58:17.019 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2024-03-11T18:58:17.020 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2024-03-11T18:58:17.020 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2024-03-11T18:58:17.020 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2024-03-11T18:58:17.020 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /root 2024-03-11T18:58:17.020 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2024-03-11T18:58:17.020 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:58:17.195 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-11T18:58:17.211 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-11T18:58:17.211 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:17.211 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-11T18:58:17.211 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:58:18.785 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-11T18:58:18.809 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-11T18:58:18.810 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:18.810 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-11T18:58:18.810 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T18:58:18.810 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T18:58:18.810 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:58:19.028 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-11T18:58:19.050 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-11T18:58:19.050 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:19.050 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-11T18:58:19.050 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T18:58:19.050 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T18:58:19.051 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:58:19.514 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-11T18:58:19.536 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-11T18:58:19.536 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:19.536 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-11T18:58:19.536 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T18:58:19.537 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T18:58:19.537 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:58:19.634 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-11T18:58:20.073 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-11T18:58:20.101 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-11T18:58:20.104 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-11T18:58:20.136 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 139/198 2024-03-11T18:58:20.142 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 154/198 2024-03-11T18:58:20.172 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 155/198 2024-03-11T18:58:20.198 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 140/198 2024-03-11T18:58:20.199 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-11T18:58:20.228 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-11T18:58:20.388 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 157/198 2024-03-11T18:58:20.468 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-11T18:58:20.499 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-11T18:58:20.505 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-11T18:58:20.539 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 159/198 2024-03-11T18:58:20.577 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-11T18:58:20.608 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-11T18:58:20.618 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 154/198 2024-03-11T18:58:20.648 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 155/198 2024-03-11T18:58:20.665 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-11T18:58:20.676 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-11T18:58:20.708 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-11T18:58:20.741 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 163/198 2024-03-11T18:58:20.781 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-11T18:58:20.811 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-11T18:58:20.863 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 157/198 2024-03-11T18:58:20.866 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-11T18:58:20.932 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-11T18:58:20.967 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-11T18:58:20.975 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-11T18:58:21.025 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 159/198 2024-03-11T18:58:21.073 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-11T18:58:21.132 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-11T18:58:21.175 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-11T18:58:21.209 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 163/198 2024-03-11T18:58:21.232 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-11T18:58:21.262 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-11T18:58:21.336 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-11T18:58:21.416 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-11T18:58:21.452 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-11T18:58:21.517 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-11T18:58:21.555 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-11T18:58:21.621 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T18:58:21.658 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T18:58:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-11T18:58:21.677 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T18:58:21.828 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 144/198 2024-03-11T18:58:21.861 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-11T18:58:21.884 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-11T18:58:21.884 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:21.885 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-11T18:58:21.885 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T18:58:21.885 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T18:58:21.885 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:58:21.946 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-11T18:58:21.957 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-11T18:58:22.019 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 147/198 2024-03-11T18:58:22.029 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T18:58:22.068 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T18:58:22.088 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T18:58:22.481 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 170/198 2024-03-11T18:58:22.561 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-11T18:58:22.611 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-11T18:58:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-11T18:58:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-11T18:58:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-11T18:58:22.784 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-11T18:58:22.810 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-11T18:58:22.811 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-11T18:58:22.890 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-11T18:58:22.910 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 170/198 2024-03-11T18:58:22.920 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-11T18:58:22.920 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-11T18:58:22.923 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-11T18:58:22.960 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-11T18:58:22.982 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-11T18:58:22.982 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:22.983 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-11T18:58:22.983 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T18:58:22.983 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T18:58:22.983 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:58:22.987 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-11T18:58:22.988 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-11T18:58:23.014 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-11T18:58:23.020 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-11T18:58:23.027 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-11T18:58:23.049 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-11T18:58:23.049 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:23.049 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-11T18:58:23.049 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:58:23.055 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-11T18:58:23.081 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-11T18:58:23.104 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T18:58:23.106 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-11T18:58:23.114 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-11T18:58:23.126 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T18:58:23.126 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:23.126 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-11T18:58:23.126 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-11T18:58:23.126 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-11T18:58:23.126 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:58:23.147 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-11T18:58:23.150 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-11T18:58:23.175 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-11T18:58:23.186 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-11T18:58:23.200 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-11T18:58:23.219 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-11T18:58:23.226 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-11T18:58:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-11T18:58:23.244 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-11T18:58:23.270 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-11T18:58:23.295 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-11T18:58:23.314 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-11T18:58:23.320 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-11T18:58:23.326 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T18:58:23.327 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T18:58:23.353 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-11T18:58:23.353 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-11T18:58:23.420 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-11T18:58:23.447 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-11T18:58:23.447 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-11T18:58:23.469 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-11T18:58:23.470 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:58:23.470 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-11T18:58:23.470 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T18:58:23.470 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T18:58:23.470 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:58:23.480 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-11T18:58:23.505 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-11T18:58:23.514 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T18:58:23.515 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-11T18:58:23.530 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-11T18:58:23.564 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-11T18:58:23.598 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-11T18:58:23.603 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-11T18:58:23.622 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-11T18:58:23.647 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-11T18:58:23.660 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-11T18:58:23.680 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-11T18:58:23.685 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-11T18:58:23.715 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-11T18:58:23.719 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-11T18:58:23.745 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-11T18:58:23.770 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-11T18:58:23.776 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T18:58:23.777 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T18:58:23.796 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-11T18:58:23.973 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T18:58:23.973 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-11T18:58:24.055 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-11T18:58:24.080 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-11T18:58:24.105 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-11T18:58:24.139 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-11T18:58:24.165 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-11T18:58:24.232 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-11T18:58:28.166 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-11T18:58:28.209 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 154/198 2024-03-11T18:58:28.247 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 155/198 2024-03-11T18:58:28.285 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-11T18:58:28.474 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 157/198 2024-03-11T18:58:28.602 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-11T18:58:28.617 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-11T18:58:28.618 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 1/198 2024-03-11T18:58:28.618 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2/198 2024-03-11T18:58:28.618 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 3/198 2024-03-11T18:58:28.618 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 4/198 2024-03-11T18:58:28.618 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 5/198 2024-03-11T18:58:28.618 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 6/198 2024-03-11T18:58:28.618 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 7/198 2024-03-11T18:58:28.618 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 8/198 2024-03-11T18:58:28.618 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 9/198 2024-03-11T18:58:28.618 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T18:58:28.618 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T18:58:28.618 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-11T18:58:28.619 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 13/198 2024-03-11T18:58:28.619 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 14/198 2024-03-11T18:58:28.619 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 15/198 2024-03-11T18:58:28.619 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T18:58:28.619 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 17/198 2024-03-11T18:58:28.619 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 18/198 2024-03-11T18:58:28.619 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 19/198 2024-03-11T18:58:28.619 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 20/198 2024-03-11T18:58:28.619 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 21/198 2024-03-11T18:58:28.619 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 22/198 2024-03-11T18:58:28.619 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 23/198 2024-03-11T18:58:28.619 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 24/198 2024-03-11T18:58:28.619 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 25/198 2024-03-11T18:58:28.620 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 26/198 2024-03-11T18:58:28.620 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 27/198 2024-03-11T18:58:28.637 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 28/198 2024-03-11T18:58:28.637 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 29/198 2024-03-11T18:58:28.637 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 30/198 2024-03-11T18:58:28.637 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 31/198 2024-03-11T18:58:28.637 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 32/198 2024-03-11T18:58:28.637 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 33/198 2024-03-11T18:58:28.637 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 34/198 2024-03-11T18:58:28.637 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 35/198 2024-03-11T18:58:28.637 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 36/198 2024-03-11T18:58:28.637 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 37/198 2024-03-11T18:58:28.637 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 38/198 2024-03-11T18:58:28.637 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 39/198 2024-03-11T18:58:28.638 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-11T18:58:28.638 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-11T18:58:28.638 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-11T18:58:28.638 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-11T18:58:28.638 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-11T18:58:28.638 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-11T18:58:28.638 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-11T18:58:28.638 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-11T18:58:28.638 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-11T18:58:28.638 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-11T18:58:28.638 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-11T18:58:28.638 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-11T18:58:28.638 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-11T18:58:28.638 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-11T18:58:28.639 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-11T18:58:28.639 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-11T18:58:28.639 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-11T18:58:28.639 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-11T18:58:28.639 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-11T18:58:28.639 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-11T18:58:28.639 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-11T18:58:28.639 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-11T18:58:28.639 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-11T18:58:28.639 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-11T18:58:28.639 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-11T18:58:28.639 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-11T18:58:28.639 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-11T18:58:28.639 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-11T18:58:28.639 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-11T18:58:28.640 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-11T18:58:28.640 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-11T18:58:28.640 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-11T18:58:28.640 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-11T18:58:28.640 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-11T18:58:28.640 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-11T18:58:28.640 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-11T18:58:28.640 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-11T18:58:28.640 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-11T18:58:28.640 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-11T18:58:28.640 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-11T18:58:28.640 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-11T18:58:28.640 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-11T18:58:28.640 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-11T18:58:28.641 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-11T18:58:28.641 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-11T18:58:28.641 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-11T18:58:28.641 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-11T18:58:28.641 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-11T18:58:28.641 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-11T18:58:28.641 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-11T18:58:28.641 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-11T18:58:28.641 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-11T18:58:28.641 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-11T18:58:28.641 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-11T18:58:28.641 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-11T18:58:28.641 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-11T18:58:28.641 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-11T18:58:28.642 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-11T18:58:28.642 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-11T18:58:28.642 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-11T18:58:28.642 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-11T18:58:28.642 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-11T18:58:28.642 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-11T18:58:28.642 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-11T18:58:28.642 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-11T18:58:28.642 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-11T18:58:28.642 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-11T18:58:28.642 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-11T18:58:28.642 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-11T18:58:28.642 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-11T18:58:28.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-11T18:58:28.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-11T18:58:28.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-11T18:58:28.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-11T18:58:28.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-11T18:58:28.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-11T18:58:28.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-11T18:58:28.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-11T18:58:28.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-11T18:58:28.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-11T18:58:28.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-11T18:58:28.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-11T18:58:28.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-11T18:58:28.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-11T18:58:28.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-11T18:58:28.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-11T18:58:28.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-11T18:58:28.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-11T18:58:28.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-11T18:58:28.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-11T18:58:28.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-11T18:58:28.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-11T18:58:28.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-11T18:58:28.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-11T18:58:28.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-11T18:58:28.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-11T18:58:28.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-11T18:58:28.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-11T18:58:28.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-11T18:58:28.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-11T18:58:28.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-11T18:58:28.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-11T18:58:28.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-11T18:58:28.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-11T18:58:28.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-11T18:58:28.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-11T18:58:28.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-11T18:58:28.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-11T18:58:28.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-11T18:58:28.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-11T18:58:28.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-11T18:58:28.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T18:58:28.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-11T18:58:28.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-11T18:58:28.646 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-11T18:58:28.646 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-11T18:58:28.646 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-11T18:58:28.646 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-11T18:58:28.646 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-11T18:58:28.646 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-11T18:58:28.646 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-11T18:58:28.646 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-11T18:58:28.646 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-11T18:58:28.646 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-11T18:58:28.646 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-11T18:58:28.646 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-11T18:58:28.646 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-11T18:58:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-11T18:58:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-11T18:58:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-11T18:58:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-11T18:58:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-11T18:58:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-11T18:58:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-11T18:58:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-11T18:58:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-11T18:58:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-11T18:58:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-11T18:58:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-11T18:58:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-11T18:58:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-11T18:58:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-11T18:58:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-11T18:58:28.648 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-11T18:58:28.648 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-11T18:58:28.648 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-11T18:58:28.648 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-11T18:58:28.648 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-11T18:58:28.648 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-11T18:58:28.648 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-11T18:58:28.648 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-11T18:58:28.648 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-11T18:58:28.648 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-11T18:58:28.648 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-11T18:58:28.648 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-11T18:58:28.648 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-11T18:58:28.648 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-11T18:58:28.649 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-11T18:58:28.657 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 159/198 2024-03-11T18:58:28.744 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-11T18:58:28.808 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-11T18:58:28.845 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-11T18:58:28.878 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 163/198 2024-03-11T18:58:28.911 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-11T18:58:28.950 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-11T18:58:29.032 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-11T18:58:29.107 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-11T18:58:29.152 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-11T18:58:29.197 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-11T18:58:29.197 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 1/198 2024-03-11T18:58:29.197 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2/198 2024-03-11T18:58:29.197 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 3/198 2024-03-11T18:58:29.197 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 4/198 2024-03-11T18:58:29.197 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 5/198 2024-03-11T18:58:29.197 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 6/198 2024-03-11T18:58:29.197 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 7/198 2024-03-11T18:58:29.197 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 8/198 2024-03-11T18:58:29.197 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 9/198 2024-03-11T18:58:29.198 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T18:58:29.198 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T18:58:29.198 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-11T18:58:29.198 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 13/198 2024-03-11T18:58:29.198 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 14/198 2024-03-11T18:58:29.198 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 15/198 2024-03-11T18:58:29.198 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T18:58:29.198 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 17/198 2024-03-11T18:58:29.198 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 18/198 2024-03-11T18:58:29.198 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 19/198 2024-03-11T18:58:29.198 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 20/198 2024-03-11T18:58:29.198 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 21/198 2024-03-11T18:58:29.198 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 22/198 2024-03-11T18:58:29.198 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 23/198 2024-03-11T18:58:29.199 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 24/198 2024-03-11T18:58:29.199 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 25/198 2024-03-11T18:58:29.199 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 26/198 2024-03-11T18:58:29.199 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 27/198 2024-03-11T18:58:29.199 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 28/198 2024-03-11T18:58:29.199 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 29/198 2024-03-11T18:58:29.199 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 30/198 2024-03-11T18:58:29.199 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 31/198 2024-03-11T18:58:29.199 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 32/198 2024-03-11T18:58:29.199 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 33/198 2024-03-11T18:58:29.199 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 34/198 2024-03-11T18:58:29.199 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 35/198 2024-03-11T18:58:29.199 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 36/198 2024-03-11T18:58:29.199 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 37/198 2024-03-11T18:58:29.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 38/198 2024-03-11T18:58:29.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 39/198 2024-03-11T18:58:29.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-11T18:58:29.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-11T18:58:29.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-11T18:58:29.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-11T18:58:29.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-11T18:58:29.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-11T18:58:29.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-11T18:58:29.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-11T18:58:29.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-11T18:58:29.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-11T18:58:29.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-11T18:58:29.200 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-11T18:58:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-11T18:58:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-11T18:58:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-11T18:58:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-11T18:58:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-11T18:58:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-11T18:58:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-11T18:58:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-11T18:58:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-11T18:58:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-11T18:58:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-11T18:58:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-11T18:58:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-11T18:58:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-11T18:58:29.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-11T18:58:29.202 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-11T18:58:29.202 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-11T18:58:29.202 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-11T18:58:29.202 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-11T18:58:29.202 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-11T18:58:29.202 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-11T18:58:29.202 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-11T18:58:29.202 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-11T18:58:29.202 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-11T18:58:29.202 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-11T18:58:29.202 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-11T18:58:29.202 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-11T18:58:29.202 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-11T18:58:29.202 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-11T18:58:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-11T18:58:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-11T18:58:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-11T18:58:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-11T18:58:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-11T18:58:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-11T18:58:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-11T18:58:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-11T18:58:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-11T18:58:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-11T18:58:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-11T18:58:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-11T18:58:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-11T18:58:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-11T18:58:29.203 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-11T18:58:29.204 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-11T18:58:29.204 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-11T18:58:29.204 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-11T18:58:29.204 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-11T18:58:29.204 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-11T18:58:29.204 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-11T18:58:29.204 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-11T18:58:29.204 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-11T18:58:29.204 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-11T18:58:29.204 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-11T18:58:29.204 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-11T18:58:29.204 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-11T18:58:29.204 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-11T18:58:29.204 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-11T18:58:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-11T18:58:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-11T18:58:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-11T18:58:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-11T18:58:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-11T18:58:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-11T18:58:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-11T18:58:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-11T18:58:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-11T18:58:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-11T18:58:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-11T18:58:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-11T18:58:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-11T18:58:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-11T18:58:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-11T18:58:29.205 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-11T18:58:29.206 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-11T18:58:29.206 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-11T18:58:29.206 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-11T18:58:29.206 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-11T18:58:29.206 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-11T18:58:29.206 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-11T18:58:29.206 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-11T18:58:29.206 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-11T18:58:29.206 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-11T18:58:29.207 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-11T18:58:29.207 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-11T18:58:29.207 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-11T18:58:29.207 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-11T18:58:29.207 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-11T18:58:29.207 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-11T18:58:29.207 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-11T18:58:29.207 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-11T18:58:29.207 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-11T18:58:29.207 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-11T18:58:29.207 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-11T18:58:29.207 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-11T18:58:29.207 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-11T18:58:29.207 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-11T18:58:29.207 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-11T18:58:29.208 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-11T18:58:29.208 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T18:58:29.208 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-11T18:58:29.208 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-11T18:58:29.208 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-11T18:58:29.208 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-11T18:58:29.208 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-11T18:58:29.208 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-11T18:58:29.208 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-11T18:58:29.208 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-11T18:58:29.208 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-11T18:58:29.208 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-11T18:58:29.208 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-11T18:58:29.208 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-11T18:58:29.208 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-11T18:58:29.209 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-11T18:58:29.209 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-11T18:58:29.209 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-11T18:58:29.209 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-11T18:58:29.209 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-11T18:58:29.209 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-11T18:58:29.209 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-11T18:58:29.209 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-11T18:58:29.209 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-11T18:58:29.209 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-11T18:58:29.209 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-11T18:58:29.209 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-11T18:58:29.209 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-11T18:58:29.209 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-11T18:58:29.209 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-11T18:58:29.210 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-11T18:58:29.210 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-11T18:58:29.210 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-11T18:58:29.210 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-11T18:58:29.210 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-11T18:58:29.210 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-11T18:58:29.210 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-11T18:58:29.210 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-11T18:58:29.210 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-11T18:58:29.210 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-11T18:58:29.210 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-11T18:58:29.210 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-11T18:58:29.210 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-11T18:58:29.210 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-11T18:58:29.210 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-11T18:58:29.211 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-11T18:58:29.211 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-11T18:58:29.211 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-11T18:58:29.768 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T18:58:29.817 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T18:58:29.836 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T18:58:30.729 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 170/198 2024-03-11T18:58:30.834 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-11T18:58:30.877 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-11T18:58:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-11T18:58:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-11T18:58:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-11T18:58:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-11T18:58:31.096 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-11T18:58:31.096 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-11T18:58:31.193 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-11T18:58:31.231 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-11T18:58:31.231 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-11T18:58:31.325 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-11T18:58:31.361 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-11T18:58:31.404 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-11T18:58:31.430 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-11T18:58:31.455 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-11T18:58:31.497 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-11T18:58:31.543 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-11T18:58:31.543 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:58:31.543 INFO:teuthology.orchestra.run.smithi190.stdout:Upgraded: 2024-03-11T18:58:31.543 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-11T18:58:31.543 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-11T18:58:31.543 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-11T18:58:31.543 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-11T18:58:31.543 INFO:teuthology.orchestra.run.smithi190.stdout: acl-2.3.1-4.el9.x86_64 2024-03-11T18:58:31.543 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-11T18:58:31.543 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-11T18:58:31.543 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-11T18:58:31.544 INFO:teuthology.orchestra.run.smithi190.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-11T18:58:31.544 INFO:teuthology.orchestra.run.smithi190.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-11T18:58:31.544 INFO:teuthology.orchestra.run.smithi190.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-11T18:58:31.544 INFO:teuthology.orchestra.run.smithi190.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-11T18:58:31.544 INFO:teuthology.orchestra.run.smithi190.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-11T18:58:31.544 INFO:teuthology.orchestra.run.smithi190.stdout: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.544 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.544 INFO:teuthology.orchestra.run.smithi190.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-11T18:58:31.544 INFO:teuthology.orchestra.run.smithi190.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-11T18:58:31.544 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-11T18:58:31.544 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-11T18:58:31.544 INFO:teuthology.orchestra.run.smithi190.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-11T18:58:31.544 INFO:teuthology.orchestra.run.smithi190.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-11T18:58:31.544 INFO:teuthology.orchestra.run.smithi190.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-11T18:58:31.545 INFO:teuthology.orchestra.run.smithi190.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-11T18:58:31.545 INFO:teuthology.orchestra.run.smithi190.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-11T18:58:31.545 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-11T18:58:31.545 INFO:teuthology.orchestra.run.smithi190.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-11T18:58:31.545 INFO:teuthology.orchestra.run.smithi190.stdout: automake-1.16.2-8.el9.noarch 2024-03-11T18:58:31.545 INFO:teuthology.orchestra.run.smithi190.stdout: bison-3.7.4-5.el9.x86_64 2024-03-11T18:58:31.545 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-11T18:58:31.545 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.545 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.545 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.545 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.545 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:31.545 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.545 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.546 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.546 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:31.546 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:31.546 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:31.546 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:31.546 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:31.546 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.546 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.546 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:31.546 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.546 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.546 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.546 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:31.546 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:31.547 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.547 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.547 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:31.547 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-11T18:58:31.547 INFO:teuthology.orchestra.run.smithi190.stdout: flex-2.6.4-9.el9.x86_64 2024-03-11T18:58:31.547 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-11T18:58:31.548 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-11T18:58:31.548 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-11T18:58:31.548 INFO:teuthology.orchestra.run.smithi190.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-11T18:58:31.548 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-11T18:58:31.548 INFO:teuthology.orchestra.run.smithi190.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-11T18:58:31.548 INFO:teuthology.orchestra.run.smithi190.stdout: jq-1.6-16.el9.x86_64 2024-03-11T18:58:31.549 INFO:teuthology.orchestra.run.smithi190.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-11T18:58:31.549 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-11T18:58:31.549 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-11T18:58:31.549 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.549 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.549 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.549 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-11T18:58:31.549 INFO:teuthology.orchestra.run.smithi190.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-11T18:58:31.550 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-11T18:58:31.550 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-11T18:58:31.550 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-11T18:58:31.550 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.550 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.550 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-11T18:58:31.550 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.550 INFO:teuthology.orchestra.run.smithi190.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-11T18:58:31.551 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-11T18:58:31.551 INFO:teuthology.orchestra.run.smithi190.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-11T18:58:31.551 INFO:teuthology.orchestra.run.smithi190.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-11T18:58:31.551 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-11T18:58:31.551 INFO:teuthology.orchestra.run.smithi190.stdout: lua-5.4.4-4.el9.x86_64 2024-03-11T18:58:31.551 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-11T18:58:31.551 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-11T18:58:31.552 INFO:teuthology.orchestra.run.smithi190.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-11T18:58:31.552 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-11T18:58:31.552 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-11T18:58:31.552 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-11T18:58:31.552 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-11T18:58:31.552 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-13.11-1.el9.x86_64 2024-03-11T18:58:31.552 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-11T18:58:31.553 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-11T18:58:31.553 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-11T18:58:31.553 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-11T18:58:31.553 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-11T18:58:31.553 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-11T18:58:31.553 INFO:teuthology.orchestra.run.smithi190.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-11T18:58:31.553 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-11T18:58:31.554 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-11T18:58:31.554 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-11T18:58:31.554 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.554 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.554 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.555 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-11T18:58:31.555 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-11T18:58:31.555 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-11T18:58:31.555 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-11T18:58:31.555 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-11T18:58:31.555 INFO:teuthology.orchestra.run.smithi190.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-11T18:58:31.555 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-11T18:58:31.556 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-11T18:58:31.556 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-11T18:58:31.556 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-11T18:58:31.556 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-11T18:58:31.556 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-11T18:58:31.556 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-11T18:58:31.556 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-11T18:58:31.557 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-11T18:58:31.557 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-11T18:58:31.557 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-11T18:58:31.557 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-11T18:58:31.557 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-11T18:58:31.557 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-11T18:58:31.558 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-11T18:58:31.558 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-11T18:58:31.558 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-11T18:58:31.558 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-11T18:58:31.558 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-11T18:58:31.558 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-11T18:58:31.559 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-11T18:58:31.559 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-11T18:58:31.559 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-11T18:58:31.559 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-11T18:58:31.559 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply-3.11-14.el9.noarch 2024-03-11T18:58:31.559 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-11T18:58:31.560 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-11T18:58:31.560 INFO:teuthology.orchestra.run.smithi190.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-11T18:58:31.560 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-11T18:58:31.560 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-11T18:58:31.560 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-11T18:58:31.560 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-11T18:58:31.561 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-11T18:58:31.561 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-11T18:58:31.561 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.561 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.561 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-11T18:58:31.561 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-11T18:58:31.561 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-11T18:58:31.562 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.562 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-11T18:58:31.562 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-11T18:58:31.562 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-11T18:58:31.562 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-11T18:58:31.562 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-11T18:58:31.562 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-11T18:58:31.562 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-11T18:58:31.562 INFO:teuthology.orchestra.run.smithi190.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-11T18:58:31.563 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-11T18:58:31.563 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-11T18:58:31.563 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-11T18:58:31.563 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-11T18:58:31.563 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-11T18:58:31.563 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-11T18:58:31.563 INFO:teuthology.orchestra.run.smithi190.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-11T18:58:31.563 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.563 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.563 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:31.563 INFO:teuthology.orchestra.run.smithi190.stdout: re2-1:20211101-3.el9.x86_64 2024-03-11T18:58:31.563 INFO:teuthology.orchestra.run.smithi190.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-11T18:58:31.563 INFO:teuthology.orchestra.run.smithi190.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-11T18:58:31.563 INFO:teuthology.orchestra.run.smithi190.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-11T18:58:31.564 INFO:teuthology.orchestra.run.smithi190.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-11T18:58:31.564 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-11T18:58:31.564 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-11T18:58:31.564 INFO:teuthology.orchestra.run.smithi190.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-11T18:58:31.564 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T18:58:31.565 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-11T18:58:31.582 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-11T18:58:31.608 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-11T18:58:31.643 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-11T18:58:31.675 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-11T18:58:31.710 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-11T18:58:31.717 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T18:58:31.717 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T18:58:31.915 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T18:58:31.915 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-11T18:58:32.046 DEBUG:teuthology.parallel:result is None 2024-03-11T18:58:32.048 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-11T18:58:32.073 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-11T18:58:32.099 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-11T18:58:32.132 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-11T18:58:32.168 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-11T18:58:32.193 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-11T18:58:32.194 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:58:32.194 INFO:teuthology.orchestra.run.smithi170.stdout:Upgraded: 2024-03-11T18:58:32.194 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-11T18:58:32.194 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-11T18:58:32.194 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-11T18:58:32.194 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-11T18:58:32.194 INFO:teuthology.orchestra.run.smithi170.stdout: acl-2.3.1-4.el9.x86_64 2024-03-11T18:58:32.195 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-11T18:58:32.195 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-11T18:58:32.195 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-11T18:58:32.195 INFO:teuthology.orchestra.run.smithi170.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-11T18:58:32.195 INFO:teuthology.orchestra.run.smithi170.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-11T18:58:32.195 INFO:teuthology.orchestra.run.smithi170.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-11T18:58:32.195 INFO:teuthology.orchestra.run.smithi170.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-11T18:58:32.196 INFO:teuthology.orchestra.run.smithi170.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-11T18:58:32.196 INFO:teuthology.orchestra.run.smithi170.stdout: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.196 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.196 INFO:teuthology.orchestra.run.smithi170.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-11T18:58:32.196 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-11T18:58:32.196 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-11T18:58:32.196 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-11T18:58:32.197 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-11T18:58:32.197 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-11T18:58:32.197 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-11T18:58:32.197 INFO:teuthology.orchestra.run.smithi170.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-11T18:58:32.197 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-11T18:58:32.197 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-11T18:58:32.198 INFO:teuthology.orchestra.run.smithi170.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-11T18:58:32.198 INFO:teuthology.orchestra.run.smithi170.stdout: automake-1.16.2-8.el9.noarch 2024-03-11T18:58:32.198 INFO:teuthology.orchestra.run.smithi170.stdout: bison-3.7.4-5.el9.x86_64 2024-03-11T18:58:32.198 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-11T18:58:32.198 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.199 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.199 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.199 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.199 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:32.199 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.199 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.200 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.200 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:32.200 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:32.200 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:32.200 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:32.200 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:32.200 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.201 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.201 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:32.201 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.201 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.201 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.201 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:32.201 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:32.201 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.202 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.202 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:32.202 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-11T18:58:32.202 INFO:teuthology.orchestra.run.smithi170.stdout: flex-2.6.4-9.el9.x86_64 2024-03-11T18:58:32.202 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-11T18:58:32.202 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-11T18:58:32.203 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-11T18:58:32.203 INFO:teuthology.orchestra.run.smithi170.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-11T18:58:32.203 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-11T18:58:32.203 INFO:teuthology.orchestra.run.smithi170.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-11T18:58:32.203 INFO:teuthology.orchestra.run.smithi170.stdout: jq-1.6-16.el9.x86_64 2024-03-11T18:58:32.203 INFO:teuthology.orchestra.run.smithi170.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-11T18:58:32.204 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-11T18:58:32.204 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-11T18:58:32.204 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.204 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.204 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.204 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-11T18:58:32.204 INFO:teuthology.orchestra.run.smithi170.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-11T18:58:32.205 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-11T18:58:32.205 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-11T18:58:32.205 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-11T18:58:32.205 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.205 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.205 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-11T18:58:32.205 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.205 INFO:teuthology.orchestra.run.smithi170.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-11T18:58:32.205 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-11T18:58:32.205 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-11T18:58:32.205 INFO:teuthology.orchestra.run.smithi170.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-11T18:58:32.205 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-11T18:58:32.205 INFO:teuthology.orchestra.run.smithi170.stdout: lua-5.4.4-4.el9.x86_64 2024-03-11T18:58:32.206 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-11T18:58:32.206 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-11T18:58:32.206 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-11T18:58:32.206 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-11T18:58:32.206 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-11T18:58:32.206 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-11T18:58:32.206 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-11T18:58:32.206 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-13.11-1.el9.x86_64 2024-03-11T18:58:32.206 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-11T18:58:32.206 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-11T18:58:32.206 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-11T18:58:32.206 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-11T18:58:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-11T18:58:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-11T18:58:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-11T18:58:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-11T18:58:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-11T18:58:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-11T18:58:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-11T18:58:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-11T18:58:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-11T18:58:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-11T18:58:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-11T18:58:32.207 INFO:teuthology.orchestra.run.smithi170.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-11T18:58:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-11T18:58:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-11T18:58:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-11T18:58:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-11T18:58:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-11T18:58:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-11T18:58:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-11T18:58:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-11T18:58:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-11T18:58:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-11T18:58:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-11T18:58:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-11T18:58:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-11T18:58:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-11T18:58:32.208 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-11T18:58:32.209 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-11T18:58:32.209 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-11T18:58:32.209 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-11T18:58:32.209 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-11T18:58:32.209 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-11T18:58:32.209 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-11T18:58:32.209 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-11T18:58:32.209 INFO:teuthology.orchestra.run.smithi170.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-11T18:58:32.209 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-11T18:58:32.209 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply-3.11-14.el9.noarch 2024-03-11T18:58:32.209 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-11T18:58:32.209 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-11T18:58:32.209 INFO:teuthology.orchestra.run.smithi170.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-11T18:58:32.210 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-11T18:58:32.210 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-11T18:58:32.210 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-11T18:58:32.210 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-11T18:58:32.210 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-11T18:58:32.210 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-11T18:58:32.210 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.210 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.210 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-11T18:58:32.210 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-11T18:58:32.210 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-11T18:58:32.210 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.210 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-11T18:58:32.211 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-11T18:58:32.211 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-11T18:58:32.211 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-11T18:58:32.211 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-11T18:58:32.211 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-11T18:58:32.211 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-11T18:58:32.211 INFO:teuthology.orchestra.run.smithi170.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-11T18:58:32.211 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-11T18:58:32.211 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-11T18:58:32.211 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-11T18:58:32.211 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-11T18:58:32.211 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-11T18:58:32.211 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-11T18:58:32.211 INFO:teuthology.orchestra.run.smithi170.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-11T18:58:32.212 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.212 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.212 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:32.212 INFO:teuthology.orchestra.run.smithi170.stdout: re2-1:20211101-3.el9.x86_64 2024-03-11T18:58:32.212 INFO:teuthology.orchestra.run.smithi170.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-11T18:58:32.212 INFO:teuthology.orchestra.run.smithi170.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-11T18:58:32.212 INFO:teuthology.orchestra.run.smithi170.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-11T18:58:32.212 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-11T18:58:32.212 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-11T18:58:32.212 INFO:teuthology.orchestra.run.smithi170.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-11T18:58:32.213 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-11T18:58:32.213 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-11T18:58:32.245 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-11T18:58:32.684 DEBUG:teuthology.parallel:result is None 2024-03-11T18:58:41.267 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-11T18:58:41.268 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 1/198 2024-03-11T18:58:41.268 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2/198 2024-03-11T18:58:41.268 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 3/198 2024-03-11T18:58:41.268 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 4/198 2024-03-11T18:58:41.268 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 5/198 2024-03-11T18:58:41.268 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 6/198 2024-03-11T18:58:41.268 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 7/198 2024-03-11T18:58:41.268 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 8/198 2024-03-11T18:58:41.268 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 9/198 2024-03-11T18:58:41.268 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T18:58:41.268 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T18:58:41.268 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-11T18:58:41.268 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 13/198 2024-03-11T18:58:41.269 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 14/198 2024-03-11T18:58:41.269 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 15/198 2024-03-11T18:58:41.269 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T18:58:41.269 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 17/198 2024-03-11T18:58:41.269 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 18/198 2024-03-11T18:58:41.269 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 19/198 2024-03-11T18:58:41.269 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 20/198 2024-03-11T18:58:41.269 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 21/198 2024-03-11T18:58:41.269 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 22/198 2024-03-11T18:58:41.269 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 23/198 2024-03-11T18:58:41.269 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 24/198 2024-03-11T18:58:41.269 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 25/198 2024-03-11T18:58:41.269 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 26/198 2024-03-11T18:58:41.269 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 27/198 2024-03-11T18:58:41.269 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 28/198 2024-03-11T18:58:41.269 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 29/198 2024-03-11T18:58:41.270 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 30/198 2024-03-11T18:58:41.270 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 31/198 2024-03-11T18:58:41.270 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 32/198 2024-03-11T18:58:41.270 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 33/198 2024-03-11T18:58:41.270 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 34/198 2024-03-11T18:58:41.270 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 35/198 2024-03-11T18:58:41.270 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 36/198 2024-03-11T18:58:41.270 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 37/198 2024-03-11T18:58:41.270 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 38/198 2024-03-11T18:58:41.270 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 39/198 2024-03-11T18:58:41.270 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-11T18:58:41.270 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-11T18:58:41.270 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-11T18:58:41.270 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-11T18:58:41.271 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-11T18:58:41.271 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-11T18:58:41.271 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-11T18:58:41.271 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-11T18:58:41.271 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-11T18:58:41.271 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-11T18:58:41.271 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-11T18:58:41.271 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-11T18:58:41.271 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-11T18:58:41.271 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-11T18:58:41.271 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-11T18:58:41.271 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-11T18:58:41.271 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-11T18:58:41.272 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-11T18:58:41.272 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-11T18:58:41.272 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-11T18:58:41.272 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-11T18:58:41.272 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-11T18:58:41.272 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-11T18:58:41.272 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-11T18:58:41.272 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-11T18:58:41.272 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-11T18:58:41.272 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-11T18:58:41.272 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-11T18:58:41.272 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-11T18:58:41.272 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-11T18:58:41.272 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-11T18:58:41.273 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-11T18:58:41.273 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-11T18:58:41.273 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-11T18:58:41.273 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-11T18:58:41.273 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-11T18:58:41.273 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-11T18:58:41.273 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-11T18:58:41.273 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-11T18:58:41.273 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-11T18:58:41.273 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-11T18:58:41.273 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-11T18:58:41.273 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-11T18:58:41.273 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-11T18:58:41.274 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-11T18:58:41.274 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-11T18:58:41.274 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-11T18:58:41.274 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-11T18:58:41.274 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-11T18:58:41.274 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-11T18:58:41.274 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-11T18:58:41.274 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-11T18:58:41.274 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-11T18:58:41.274 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-11T18:58:41.274 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-11T18:58:41.274 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-11T18:58:41.274 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-11T18:58:41.274 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-11T18:58:41.275 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-11T18:58:41.275 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-11T18:58:41.275 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-11T18:58:41.275 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-11T18:58:41.275 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-11T18:58:41.275 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-11T18:58:41.275 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-11T18:58:41.275 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-11T18:58:41.275 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-11T18:58:41.275 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-11T18:58:41.275 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-11T18:58:41.275 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-11T18:58:41.275 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-11T18:58:41.275 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-11T18:58:41.275 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-11T18:58:41.276 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-11T18:58:41.276 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-11T18:58:41.276 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-11T18:58:41.276 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-11T18:58:41.276 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-11T18:58:41.276 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-11T18:58:41.276 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-11T18:58:41.276 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-11T18:58:41.276 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-11T18:58:41.276 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-11T18:58:41.276 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-11T18:58:41.276 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-11T18:58:41.276 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-11T18:58:41.276 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-11T18:58:41.277 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-11T18:58:41.277 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-11T18:58:41.277 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-11T18:58:41.277 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-11T18:58:41.277 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-11T18:58:41.277 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-11T18:58:41.277 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-11T18:58:41.277 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-11T18:58:41.277 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-11T18:58:41.277 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-11T18:58:41.277 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-11T18:58:41.277 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-11T18:58:41.277 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-11T18:58:41.277 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-11T18:58:41.278 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-11T18:58:41.278 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-11T18:58:41.278 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-11T18:58:41.278 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-11T18:58:41.278 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-11T18:58:41.278 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-11T18:58:41.278 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-11T18:58:41.278 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-11T18:58:41.278 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-11T18:58:41.278 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-11T18:58:41.278 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T18:58:41.278 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-11T18:58:41.278 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-11T18:58:41.278 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-11T18:58:41.278 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-11T18:58:41.279 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-11T18:58:41.279 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-11T18:58:41.279 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-11T18:58:41.279 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-11T18:58:41.279 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-11T18:58:41.279 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-11T18:58:41.279 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-11T18:58:41.279 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-11T18:58:41.279 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-11T18:58:41.279 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-11T18:58:41.279 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-11T18:58:41.279 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-11T18:58:41.279 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-11T18:58:41.279 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-11T18:58:41.280 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-11T18:58:41.280 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-11T18:58:41.280 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-11T18:58:41.280 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-11T18:58:41.280 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-11T18:58:41.280 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-11T18:58:41.280 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-11T18:58:41.280 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-11T18:58:41.280 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-11T18:58:41.280 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-11T18:58:41.280 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-11T18:58:41.280 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-11T18:58:41.280 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-11T18:58:41.280 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-11T18:58:41.280 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-11T18:58:41.280 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-11T18:58:41.281 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-11T18:58:41.281 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-11T18:58:41.281 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-11T18:58:41.281 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-11T18:58:41.281 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-11T18:58:41.281 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-11T18:58:41.281 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-11T18:58:41.281 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-11T18:58:41.281 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-11T18:58:41.281 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-11T18:58:41.281 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-11T18:58:41.281 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-11T18:58:44.308 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-11T18:58:44.308 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:58:44.308 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2024-03-11T18:58:44.308 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-11T18:58:44.308 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-11T18:58:44.308 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-11T18:58:44.309 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-11T18:58:44.309 INFO:teuthology.orchestra.run.smithi029.stdout: acl-2.3.1-4.el9.x86_64 2024-03-11T18:58:44.309 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-11T18:58:44.309 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-11T18:58:44.309 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-11T18:58:44.309 INFO:teuthology.orchestra.run.smithi029.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-11T18:58:44.309 INFO:teuthology.orchestra.run.smithi029.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-11T18:58:44.309 INFO:teuthology.orchestra.run.smithi029.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-11T18:58:44.309 INFO:teuthology.orchestra.run.smithi029.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-11T18:58:44.309 INFO:teuthology.orchestra.run.smithi029.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-11T18:58:44.309 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.309 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.309 INFO:teuthology.orchestra.run.smithi029.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-11T18:58:44.309 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-11T18:58:44.310 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-11T18:58:44.310 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-11T18:58:44.310 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-11T18:58:44.310 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-11T18:58:44.310 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-11T18:58:44.310 INFO:teuthology.orchestra.run.smithi029.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-11T18:58:44.310 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-11T18:58:44.310 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-11T18:58:44.310 INFO:teuthology.orchestra.run.smithi029.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-11T18:58:44.310 INFO:teuthology.orchestra.run.smithi029.stdout: automake-1.16.2-8.el9.noarch 2024-03-11T18:58:44.310 INFO:teuthology.orchestra.run.smithi029.stdout: bison-3.7.4-5.el9.x86_64 2024-03-11T18:58:44.310 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-11T18:58:44.311 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.311 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.311 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.311 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.311 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:44.311 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.311 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.311 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.311 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:44.311 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:44.311 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:44.311 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:44.311 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:44.311 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.312 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.312 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:44.312 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.312 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.312 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.312 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:44.312 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:44.312 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.312 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.312 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:58:44.312 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-11T18:58:44.312 INFO:teuthology.orchestra.run.smithi029.stdout: flex-2.6.4-9.el9.x86_64 2024-03-11T18:58:44.312 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-11T18:58:44.312 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-11T18:58:44.312 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-11T18:58:44.312 INFO:teuthology.orchestra.run.smithi029.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-11T18:58:44.313 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-11T18:58:44.313 INFO:teuthology.orchestra.run.smithi029.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-11T18:58:44.313 INFO:teuthology.orchestra.run.smithi029.stdout: jq-1.6-16.el9.x86_64 2024-03-11T18:58:44.313 INFO:teuthology.orchestra.run.smithi029.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-11T18:58:44.313 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-11T18:58:44.313 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-11T18:58:44.313 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.313 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.313 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.313 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-11T18:58:44.313 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-11T18:58:44.313 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-11T18:58:44.313 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-11T18:58:44.313 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-11T18:58:44.313 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.314 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.314 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-11T18:58:44.314 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.314 INFO:teuthology.orchestra.run.smithi029.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-11T18:58:44.314 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-11T18:58:44.314 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-11T18:58:44.314 INFO:teuthology.orchestra.run.smithi029.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-11T18:58:44.314 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-11T18:58:44.314 INFO:teuthology.orchestra.run.smithi029.stdout: lua-5.4.4-4.el9.x86_64 2024-03-11T18:58:44.314 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-11T18:58:44.314 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-11T18:58:44.314 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-11T18:58:44.314 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-11T18:58:44.314 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-11T18:58:44.315 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-11T18:58:44.315 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-11T18:58:44.315 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-13.11-1.el9.x86_64 2024-03-11T18:58:44.315 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-11T18:58:44.315 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-11T18:58:44.315 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-11T18:58:44.315 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-11T18:58:44.315 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-11T18:58:44.315 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-11T18:58:44.315 INFO:teuthology.orchestra.run.smithi029.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-11T18:58:44.315 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-11T18:58:44.315 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-11T18:58:44.315 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-11T18:58:44.315 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.315 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.316 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.316 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-11T18:58:44.316 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-11T18:58:44.316 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-11T18:58:44.316 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-11T18:58:44.316 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-11T18:58:44.316 INFO:teuthology.orchestra.run.smithi029.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-11T18:58:44.316 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-11T18:58:44.316 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-11T18:58:44.316 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-11T18:58:44.316 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-11T18:58:44.316 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-11T18:58:44.316 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-11T18:58:44.316 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-11T18:58:44.317 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-11T18:58:44.317 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-11T18:58:44.317 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-11T18:58:44.317 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-11T18:58:44.317 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-11T18:58:44.317 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-11T18:58:44.317 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-11T18:58:44.317 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-11T18:58:44.317 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-11T18:58:44.317 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-11T18:58:44.317 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-11T18:58:44.317 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-11T18:58:44.317 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-11T18:58:44.317 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-11T18:58:44.318 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-11T18:58:44.318 INFO:teuthology.orchestra.run.smithi029.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-11T18:58:44.318 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-11T18:58:44.318 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply-3.11-14.el9.noarch 2024-03-11T18:58:44.318 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-11T18:58:44.318 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-11T18:58:44.318 INFO:teuthology.orchestra.run.smithi029.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-11T18:58:44.318 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-11T18:58:44.318 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-11T18:58:44.318 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-11T18:58:44.318 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-11T18:58:44.318 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-11T18:58:44.318 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-11T18:58:44.318 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.318 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.319 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-11T18:58:44.319 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-11T18:58:44.319 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-11T18:58:44.319 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.319 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-11T18:58:44.319 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-11T18:58:44.319 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-11T18:58:44.319 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-11T18:58:44.319 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-11T18:58:44.319 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-11T18:58:44.319 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-11T18:58:44.319 INFO:teuthology.orchestra.run.smithi029.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-11T18:58:44.319 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-11T18:58:44.319 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-11T18:58:44.319 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-11T18:58:44.320 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-11T18:58:44.320 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-11T18:58:44.320 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-11T18:58:44.320 INFO:teuthology.orchestra.run.smithi029.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-11T18:58:44.320 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.320 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.320 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:58:44.320 INFO:teuthology.orchestra.run.smithi029.stdout: re2-1:20211101-3.el9.x86_64 2024-03-11T18:58:44.320 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-11T18:58:44.320 INFO:teuthology.orchestra.run.smithi029.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-11T18:58:44.320 INFO:teuthology.orchestra.run.smithi029.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-11T18:58:44.320 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-11T18:58:44.320 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-11T18:58:44.321 INFO:teuthology.orchestra.run.smithi029.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-11T18:58:44.321 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:58:44.321 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-11T18:58:44.966 DEBUG:teuthology.parallel:result is None 2024-03-11T18:58:44.967 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:58:45.122 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-11T18:58:45.147 INFO:teuthology.orchestra.run.smithi029.stdout:19.0.0-1607.g94a53970.el9 2024-03-11T18:58:45.147 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1607.g94a53970.el9 2024-03-11T18:58:45.148 INFO:teuthology.task.install:The correct ceph version 19.0.0-1607.g94a53970 is installed. 2024-03-11T18:58:45.150 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:58:45.299 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-11T18:58:45.325 INFO:teuthology.orchestra.run.smithi170.stdout:19.0.0-1607.g94a53970.el9 2024-03-11T18:58:45.325 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1607.g94a53970.el9 2024-03-11T18:58:45.325 INFO:teuthology.task.install:The correct ceph version 19.0.0-1607.g94a53970 is installed. 2024-03-11T18:58:45.327 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:58:45.475 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-11T18:58:45.501 INFO:teuthology.orchestra.run.smithi190.stdout:19.0.0-1607.g94a53970.el9 2024-03-11T18:58:45.501 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1607.g94a53970.el9 2024-03-11T18:58:45.501 INFO:teuthology.task.install:The correct ceph version 19.0.0-1607.g94a53970 is installed. 2024-03-11T18:58:45.503 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-11T18:58:45.503 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-11T18:58:45.503 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T18:58:45.537 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-11T18:58:45.537 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T18:58:45.572 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-11T18:58:45.572 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T18:58:45.606 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-11T18:58:45.606 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-11T18:58:45.607 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T18:58:45.636 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T18:58:45.704 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-11T18:58:45.705 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T18:58:45.739 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T18:58:45.807 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-11T18:58:45.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T18:58:45.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T18:58:45.911 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-11T18:58:45.911 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-11T18:58:45.911 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T18:58:45.941 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T18:58:46.010 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-11T18:58:46.010 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T18:58:46.037 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T18:58:46.105 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-11T18:58:46.106 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T18:58:46.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T18:58:46.200 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-11T18:58:46.201 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-11T18:58:46.201 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T18:58:46.230 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T18:58:46.300 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-11T18:58:46.300 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T18:58:46.327 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T18:58:46.394 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-11T18:58:46.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T18:58:46.422 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T18:58:46.489 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-11T18:58:46.588 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-11T18:58:46.588 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:58:46.589 INFO:tasks.cephadm:Cluster fsid is 638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T18:58:46.589 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-11T18:58:46.589 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.29', 'mon.b': '172.21.15.170', 'mon.c': '172.21.15.190'} 2024-03-11T18:58:46.589 INFO:tasks.cephadm:First mon is mon.a on smithi029 2024-03-11T18:58:46.589 INFO:tasks.cephadm:First mgr is x 2024-03-11T18:58:46.589 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-11T18:58:46.589 DEBUG:teuthology.orchestra.run.smithi029:> sudo hostname $(hostname -s) 2024-03-11T18:58:46.620 DEBUG:teuthology.orchestra.run.smithi170:> sudo hostname $(hostname -s) 2024-03-11T18:58:46.646 DEBUG:teuthology.orchestra.run.smithi190:> sudo hostname $(hostname -s) 2024-03-11T18:58:46.673 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-11T18:58:46.674 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:58:47.044 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77431/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:08:19.449750', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77429/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:06:59.696700', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-11T18:58:47.194 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 94a539707733325322964c6a187c6a1b41be1daf from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:58:47.196 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm 2024-03-11T18:58:47.196 DEBUG:teuthology.orchestra.run.smithi029:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T18:58:48.191 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 18:58 /home/ubuntu/cephtest/cephadm 2024-03-11T18:58:48.191 DEBUG:teuthology.orchestra.run.smithi170:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T18:58:50.661 INFO:teuthology.orchestra.run.smithi170.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 18:58 /home/ubuntu/cephtest/cephadm 2024-03-11T18:58:50.662 DEBUG:teuthology.orchestra.run.smithi190:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T18:58:51.561 INFO:teuthology.orchestra.run.smithi190.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 18:58 /home/ubuntu/cephtest/cephadm 2024-03-11T18:58:51.561 DEBUG:teuthology.orchestra.run.smithi029:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T18:58:51.579 DEBUG:teuthology.orchestra.run.smithi170:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T18:58:51.598 DEBUG:teuthology.orchestra.run.smithi190:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T18:58:51.632 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf on all hosts... 2024-03-11T18:58:51.632 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T18:58:51.634 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T18:58:51.641 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T18:58:52.469 INFO:teuthology.orchestra.run.smithi170.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T18:58:52.534 INFO:teuthology.orchestra.run.smithi190.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T18:58:52.739 INFO:teuthology.orchestra.run.smithi029.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T18:59:21.534 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-03-11T18:59:21.534 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T18:59:21.534 INFO:teuthology.orchestra.run.smithi170.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T18:59:21.535 INFO:teuthology.orchestra.run.smithi170.stdout: "repo_digests": [ 2024-03-11T18:59:21.535 INFO:teuthology.orchestra.run.smithi170.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T18:59:21.535 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-03-11T18:59:21.535 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-03-11T18:59:24.562 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T18:59:24.562 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T18:59:24.562 INFO:teuthology.orchestra.run.smithi029.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T18:59:24.562 INFO:teuthology.orchestra.run.smithi029.stdout: "repo_digests": [ 2024-03-11T18:59:24.562 INFO:teuthology.orchestra.run.smithi029.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T18:59:24.562 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-11T18:59:24.562 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T18:59:27.928 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-11T18:59:27.928 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T18:59:27.928 INFO:teuthology.orchestra.run.smithi190.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T18:59:27.928 INFO:teuthology.orchestra.run.smithi190.stdout: "repo_digests": [ 2024-03-11T18:59:27.928 INFO:teuthology.orchestra.run.smithi190.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T18:59:27.928 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-11T18:59:27.928 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-11T18:59:27.952 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph 2024-03-11T18:59:27.987 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph 2024-03-11T18:59:28.022 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph 2024-03-11T18:59:28.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /etc/ceph 2024-03-11T18:59:28.083 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 777 /etc/ceph 2024-03-11T18:59:28.113 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /etc/ceph 2024-03-11T18:59:28.140 INFO:tasks.cephadm:Writing seed config... 2024-03-11T18:59:28.141 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-11T18:59:28.141 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-11T18:59:28.141 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-11T18:59:28.141 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-11T18:59:28.142 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-11T18:59:28.142 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-11T18:59:28.142 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-11T18:59:28.142 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-11T18:59:28.142 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-11T18:59:28.142 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-11T18:59:28.142 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-11T18:59:28.142 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-11T18:59:28.142 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-11T18:59:28.142 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-11T18:59:28.142 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-11T18:59:28.142 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-11T18:59:28.143 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-11T18:59:28.143 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-11T18:59:28.143 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-11T18:59:28.143 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-11T18:59:28.143 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-11T18:59:28.143 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-11T18:59:28.143 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-11T18:59:28.143 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-11T18:59:28.143 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-11T18:59:28.143 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-11T18:59:28.143 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-11T18:59:28.143 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-11T18:59:28.143 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-11T18:59:28.143 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-11T18:59:28.144 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-11T18:59:28.144 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-11T18:59:28.144 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-11T18:59:28.144 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-11T18:59:28.144 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-11T18:59:28.144 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-11T18:59:28.144 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-11T18:59:28.144 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-11T18:59:28.144 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-11T18:59:28.144 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-11T18:59:28.144 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-11T18:59:28.144 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-11T18:59:28.145 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-11T18:59:28.145 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-11T18:59:28.145 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-11T18:59:28.163 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 638915c2-dfd9-11ee-95c8-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-11T18:59:28.163 DEBUG:teuthology.orchestra.run.smithi029:mon.a> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mon.a.service 2024-03-11T18:59:28.206 DEBUG:teuthology.orchestra.run.smithi029:mgr.x> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mgr.x.service 2024-03-11T18:59:28.248 INFO:tasks.cephadm:Bootstrapping... 2024-03-11T18:59:28.248 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf -v bootstrap --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.29 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-11T18:59:28.449 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-11T18:59:28.449 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf', '-v', 'bootstrap', '--fsid', '638915c2-dfd9-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.29', '--skip-admin-label'] 2024-03-11T18:59:28.473 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T18:59:28.474 INFO:teuthology.orchestra.run.smithi029.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-11T18:59:28.474 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying podman|docker is present... 2024-03-11T18:59:28.498 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T18:59:28.498 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying lvm2 is present... 2024-03-11T18:59:28.498 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying time synchronization is in place... 2024-03-11T18:59:28.505 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T18:59:28.505 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T18:59:28.511 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T18:59:28.511 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-11T18:59:28.517 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout enabled 2024-03-11T18:59:28.523 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout active 2024-03-11T18:59:28.523 INFO:teuthology.orchestra.run.smithi029.stdout:Unit chronyd.service is enabled and running 2024-03-11T18:59:28.523 INFO:teuthology.orchestra.run.smithi029.stdout:Repeating the final host check... 2024-03-11T18:59:28.547 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T18:59:28.547 INFO:teuthology.orchestra.run.smithi029.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-11T18:59:28.547 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl is present 2024-03-11T18:59:28.547 INFO:teuthology.orchestra.run.smithi029.stdout:lvcreate is present 2024-03-11T18:59:28.553 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T18:59:28.553 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T18:59:28.559 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T18:59:28.559 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-11T18:59:28.565 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout enabled 2024-03-11T18:59:28.571 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout active 2024-03-11T18:59:28.571 INFO:teuthology.orchestra.run.smithi029.stdout:Unit chronyd.service is enabled and running 2024-03-11T18:59:28.571 INFO:teuthology.orchestra.run.smithi029.stdout:Host looks OK 2024-03-11T18:59:28.571 INFO:teuthology.orchestra.run.smithi029.stdout:Cluster fsid: 638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T18:59:28.571 INFO:teuthology.orchestra.run.smithi029.stdout:Acquiring lock 140514893766608 on /run/cephadm/638915c2-dfd9-11ee-95c8-87774f69a715.lock 2024-03-11T18:59:28.571 INFO:teuthology.orchestra.run.smithi029.stdout:Lock 140514893766608 acquired on /run/cephadm/638915c2-dfd9-11ee-95c8-87774f69a715.lock 2024-03-11T18:59:28.572 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying IP 172.21.15.29 port 3300 ... 2024-03-11T18:59:28.572 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying IP 172.21.15.29 port 6789 ... 2024-03-11T18:59:28.572 INFO:teuthology.orchestra.run.smithi029.stdout:Base mon IP(s) is [172.21.15.29:3300, 172.21.15.29:6789], mon addrv is [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-03-11T18:59:28.575 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.29 metric 100 2024-03-11T18:59:28.575 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.29 metric 100 2024-03-11T18:59:28.577 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-11T18:59:28.578 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-11T18:59:28.578 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-11T18:59:28.580 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-11T18:59:28.580 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-11T18:59:28.580 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T18:59:28.580 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-11T18:59:28.580 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2024-03-11T18:59:28.580 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T18:59:28.581 INFO:teuthology.orchestra.run.smithi029.stdout:Mon IP `172.21.15.29` is in CIDR network `172.21.0.0/20` 2024-03-11T18:59:28.581 INFO:teuthology.orchestra.run.smithi029.stdout:Mon IP `172.21.15.29` is in CIDR network `172.21.0.0/20` 2024-03-11T18:59:28.581 INFO:teuthology.orchestra.run.smithi029.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-11T18:59:28.581 INFO:teuthology.orchestra.run.smithi029.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-11T18:59:28.582 INFO:teuthology.orchestra.run.smithi029.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T18:59:29.060 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T18:59:29.060 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T18:59:29.060 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Getting image source signatures 2024-03-11T18:59:29.060 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying blob sha256:0b3c7bee4e66dab4b5376aaf0baf8ba5fc1c3e66c27dc601157782d617bd9dc6 2024-03-11T18:59:29.061 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-11T18:59:29.061 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying config sha256:a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T18:59:29.061 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-11T18:59:30.013 INFO:teuthology.orchestra.run.smithi029.stdout:ceph: stdout ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T18:59:30.013 INFO:teuthology.orchestra.run.smithi029.stdout:Ceph version: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T18:59:30.013 INFO:teuthology.orchestra.run.smithi029.stdout:Extracting ceph user uid/gid from container image... 2024-03-11T18:59:30.908 INFO:teuthology.orchestra.run.smithi029.stdout:stat: stdout 167 167 2024-03-11T18:59:30.908 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial keys... 2024-03-11T18:59:31.828 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQCTVO9lOOAVDxAAqAhL2qZk9w0nTBMX9/Kwuw== 2024-03-11T18:59:32.808 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQCUVO9ljZRJBxAAQowXXKaZNzI3SPR+SGauIg== 2024-03-11T18:59:33.687 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQCVVO9lXX2YBxAAhH8s4FfRc26bMZCkAe9vJg== 2024-03-11T18:59:33.688 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial monmap... 2024-03-11T18:59:34.616 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T18:59:34.616 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-11T18:59:34.616 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T18:59:34.616 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T18:59:34.616 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool for a [v2:172.21.15.29:3300,v1:172.21.15.29:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T18:59:34.617 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = quincy 2024-03-11T18:59:34.617 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: set fsid to 638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T18:59:34.617 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T18:59:34.617 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T18:59:34.617 INFO:teuthology.orchestra.run.smithi029.stdout:Creating mon... 2024-03-11T18:59:35.411 INFO:teuthology.orchestra.run.smithi029.stdout:create mon.a on 2024-03-11T18:59:35.614 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-11T18:59:35.750 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-11T18:59:35.940 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-638915c2-dfd9-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-638915c2-dfd9-11ee-95c8-87774f69a715.target. 2024-03-11T18:59:35.940 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-638915c2-dfd9-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-638915c2-dfd9-11ee-95c8-87774f69a715.target. 2024-03-11T18:59:36.170 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mon.a 2024-03-11T18:59:36.170 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to reset failed state of unit ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-11T18:59:36.335 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-638915c2-dfd9-11ee-95c8-87774f69a715.target.wants/ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-638915c2-dfd9-11ee-95c8-87774f69a715@.service. 2024-03-11T18:59:36.944 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T18:59:36.944 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-03-11T18:59:36.951 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T18:59:36.951 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-11T18:59:36.951 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-03-11T18:59:36.951 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T18:59:36.952 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mon to start... 2024-03-11T18:59:36.952 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mon... 2024-03-11T18:59:37.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout cluster: 2024-03-11T18:59:37.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout id: 638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T18:59:37.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-11T18:59:37.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-11T18:59:37.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout services: 2024-03-11T18:59:37.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.562734s) 2024-03-11T18:59:37.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-11T18:59:37.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-11T18:59:37.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-11T18:59:37.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout data: 2024-03-11T18:59:37.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-11T18:59:37.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-11T18:59:37.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-11T18:59:37.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout pgs: 2024-03-11T18:59:37.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-11T18:59:37.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.374+0000 7f1c2f968700 1 Processor -- start 2024-03-11T18:59:37.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.374+0000 7f1c2f968700 1 -- start start 2024-03-11T18:59:37.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.374+0000 7f1c2f968700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c28107ef0 0x7f1c2810a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:59:37.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.374+0000 7f1c2f968700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2807ae50 con 0x7f1c28107ef0 2024-03-11T18:59:37.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.375+0000 7f1c2d704700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c28107ef0 0x7f1c2810a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:59:37.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.375+0000 7f1c2d704700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c28107ef0 0x7f1c2810a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51386/0 (socket says 172.21.15.29:51386) 2024-03-11T18:59:37.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.375+0000 7f1c2d704700 1 -- 172.21.15.29:0/1808984028 learned_addr learned my addr 172.21.15.29:0/1808984028 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T18:59:37.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.375+0000 7f1c2d704700 1 -- 172.21.15.29:0/1808984028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2810a820 con 0x7f1c28107ef0 2024-03-11T18:59:37.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.375+0000 7f1c2d704700 1 --2- 172.21.15.29:0/1808984028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c28107ef0 0x7f1c2810a2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1c240098a0 tx=0x7f1c24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65728b657072ec25 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:59:37.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.375+0000 7f1c1ffff700 1 -- 172.21.15.29:0/1808984028 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c24004160 con 0x7f1c28107ef0 2024-03-11T18:59:37.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.375+0000 7f1c1ffff700 1 -- 172.21.15.29:0/1808984028 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c240042c0 con 0x7f1c28107ef0 2024-03-11T18:59:37.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.375+0000 7f1c1ffff700 1 -- 172.21.15.29:0/1808984028 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c240045a0 con 0x7f1c28107ef0 2024-03-11T18:59:37.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.376+0000 7f1c2f968700 1 -- 172.21.15.29:0/1808984028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c28107ef0 msgr2=0x7f1c2810a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:59:37.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.376+0000 7f1c2f968700 1 --2- 172.21.15.29:0/1808984028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c28107ef0 0x7f1c2810a2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1c240098a0 tx=0x7f1c24009bb0 comp rx=0 tx=0).stop 2024-03-11T18:59:37.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.376+0000 7f1c2f968700 1 -- 172.21.15.29:0/1808984028 shutdown_connections 2024-03-11T18:59:37.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.376+0000 7f1c2f968700 1 --2- 172.21.15.29:0/1808984028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c28107ef0 0x7f1c2810a2e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:59:37.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.376+0000 7f1c2f968700 1 -- 172.21.15.29:0/1808984028 >> 172.21.15.29:0/1808984028 conn(0x7f1c28101940 msgr2=0x7f1c28103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:59:37.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.376+0000 7f1c2f968700 1 -- 172.21.15.29:0/1808984028 shutdown_connections 2024-03-11T18:59:37.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.376+0000 7f1c2f968700 1 -- 172.21.15.29:0/1808984028 wait complete. 2024-03-11T18:59:37.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.377+0000 7f1c2f968700 1 Processor -- start 2024-03-11T18:59:37.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.377+0000 7f1c2f968700 1 -- start start 2024-03-11T18:59:37.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.377+0000 7f1c2f968700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c28107ef0 0x7f1c2819a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:59:37.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.377+0000 7f1c2f968700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2819ae00 con 0x7f1c28107ef0 2024-03-11T18:59:37.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.377+0000 7f1c2d704700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c28107ef0 0x7f1c2819a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:59:37.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.377+0000 7f1c2d704700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c28107ef0 0x7f1c2819a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51388/0 (socket says 172.21.15.29:51388) 2024-03-11T18:59:37.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.377+0000 7f1c2d704700 1 -- 172.21.15.29:0/4270333894 learned_addr learned my addr 172.21.15.29:0/4270333894 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T18:59:37.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.377+0000 7f1c2d704700 1 -- 172.21.15.29:0/4270333894 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c24009550 con 0x7f1c28107ef0 2024-03-11T18:59:37.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.377+0000 7f1c2d704700 1 --2- 172.21.15.29:0/4270333894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c28107ef0 0x7f1c2819a8c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1c2807a290 tx=0x7f1c24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:59:37.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.378+0000 7f1c1e7fc700 1 -- 172.21.15.29:0/4270333894 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c24011630 con 0x7f1c28107ef0 2024-03-11T18:59:37.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.378+0000 7f1c1e7fc700 1 -- 172.21.15.29:0/4270333894 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c24011790 con 0x7f1c28107ef0 2024-03-11T18:59:37.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.378+0000 7f1c2f968700 1 -- 172.21.15.29:0/4270333894 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2819b000 con 0x7f1c28107ef0 2024-03-11T18:59:37.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.378+0000 7f1c2f968700 1 -- 172.21.15.29:0/4270333894 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2819b4a0 con 0x7f1c28107ef0 2024-03-11T18:59:37.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.378+0000 7f1c1e7fc700 1 -- 172.21.15.29:0/4270333894 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c24011a30 con 0x7f1c28107ef0 2024-03-11T18:59:37.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.378+0000 7f1c1e7fc700 1 -- 172.21.15.29:0/4270333894 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f1c24022070 con 0x7f1c28107ef0 2024-03-11T18:59:37.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.378+0000 7f1c1e7fc700 1 -- 172.21.15.29:0/4270333894 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1c2401c450 con 0x7f1c28107ef0 2024-03-11T18:59:37.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.379+0000 7f1c2f968700 1 -- 172.21.15.29:0/4270333894 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c0c005360 con 0x7f1c28107ef0 2024-03-11T18:59:37.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.381+0000 7f1c1e7fc700 1 -- 172.21.15.29:0/4270333894 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f1c24011b90 con 0x7f1c28107ef0 2024-03-11T18:59:37.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.455+0000 7f1c2f968700 1 -- 172.21.15.29:0/4270333894 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f1c0c0058e0 con 0x7f1c28107ef0 2024-03-11T18:59:37.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.456+0000 7f1c1e7fc700 1 -- 172.21.15.29:0/4270333894 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f1c24011d30 con 0x7f1c28107ef0 2024-03-11T18:59:37.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.456+0000 7f1c2f968700 1 -- 172.21.15.29:0/4270333894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c28107ef0 msgr2=0x7f1c2819a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:59:37.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.456+0000 7f1c2f968700 1 --2- 172.21.15.29:0/4270333894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c28107ef0 0x7f1c2819a8c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1c2807a290 tx=0x7f1c24004750 comp rx=0 tx=0).stop 2024-03-11T18:59:37.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.457+0000 7f1c2f968700 1 -- 172.21.15.29:0/4270333894 shutdown_connections 2024-03-11T18:59:37.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.457+0000 7f1c2f968700 1 --2- 172.21.15.29:0/4270333894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c28107ef0 0x7f1c2819a8c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:59:37.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.457+0000 7f1c2f968700 1 -- 172.21.15.29:0/4270333894 >> 172.21.15.29:0/4270333894 conn(0x7f1c28101940 msgr2=0x7f1c28103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:59:37.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.457+0000 7f1c2f968700 1 -- 172.21.15.29:0/4270333894 shutdown_connections 2024-03-11T18:59:37.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:37.457+0000 7f1c2f968700 1 -- 172.21.15.29:0/4270333894 wait complete. 2024-03-11T18:59:37.998 INFO:teuthology.orchestra.run.smithi029.stdout:mon is available 2024-03-11T18:59:37.998 INFO:teuthology.orchestra.run.smithi029.stdout:Assimilating anything we can from ceph.conf... 2024-03-11T18:59:38.990 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-11T18:59:38.991 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T18:59:38.991 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout fsid = 638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T18:59:38.991 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-03-11T18:59:38.991 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T18:59:38.991 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T18:59:38.991 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T18:59:38.991 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T18:59:38.991 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-11T18:59:38.991 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T18:59:38.991 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T18:59:38.991 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-11T18:59:38.991 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T18:59:38.991 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-11T18:59:38.992 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-11T18:59:38.992 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-11T18:59:38.992 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T18:59:38.992 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T18:59:38.992 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.398+0000 7f1dd3da5700 1 Processor -- start 2024-03-11T18:59:38.992 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.398+0000 7f1dd3da5700 1 -- start start 2024-03-11T18:59:38.992 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.398+0000 7f1dd3da5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dcc105750 0x7f1dcc105b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:59:38.992 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.398+0000 7f1dd3da5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dcc106140 con 0x7f1dcc105750 2024-03-11T18:59:38.992 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.398+0000 7f1dd1b41700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dcc105750 0x7f1dcc105b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:59:38.992 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.399+0000 7f1dd1b41700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dcc105750 0x7f1dcc105b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51396/0 (socket says 172.21.15.29:51396) 2024-03-11T18:59:38.992 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.399+0000 7f1dd1b41700 1 -- 172.21.15.29:0/2987466049 learned_addr learned my addr 172.21.15.29:0/2987466049 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T18:59:38.992 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.399+0000 7f1dd1b41700 1 -- 172.21.15.29:0/2987466049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc1086e0 con 0x7f1dcc105750 2024-03-11T18:59:38.992 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.399+0000 7f1dd1b41700 1 --2- 172.21.15.29:0/2987466049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dcc105750 0x7f1dcc105b70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1dc80098a0 tx=0x7f1dc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=625b4fb62830d9be server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:59:38.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.399+0000 7f1dd0b3f700 1 -- 172.21.15.29:0/2987466049 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1dc8004160 con 0x7f1dcc105750 2024-03-11T18:59:38.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.399+0000 7f1dd0b3f700 1 -- 172.21.15.29:0/2987466049 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc80042c0 con 0x7f1dcc105750 2024-03-11T18:59:38.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.399+0000 7f1dd0b3f700 1 -- 172.21.15.29:0/2987466049 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1dc80045a0 con 0x7f1dcc105750 2024-03-11T18:59:38.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.400+0000 7f1dd3da5700 1 -- 172.21.15.29:0/2987466049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dcc105750 msgr2=0x7f1dcc105b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:59:38.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.400+0000 7f1dd3da5700 1 --2- 172.21.15.29:0/2987466049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dcc105750 0x7f1dcc105b70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1dc80098a0 tx=0x7f1dc8009bb0 comp rx=0 tx=0).stop 2024-03-11T18:59:38.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.400+0000 7f1dd3da5700 1 -- 172.21.15.29:0/2987466049 shutdown_connections 2024-03-11T18:59:38.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.400+0000 7f1dd3da5700 1 --2- 172.21.15.29:0/2987466049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dcc105750 0x7f1dcc105b70 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:59:38.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.400+0000 7f1dd3da5700 1 -- 172.21.15.29:0/2987466049 >> 172.21.15.29:0/2987466049 conn(0x7f1dcc1010a0 msgr2=0x7f1dcc1034c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:59:38.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.400+0000 7f1dd3da5700 1 -- 172.21.15.29:0/2987466049 shutdown_connections 2024-03-11T18:59:38.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.400+0000 7f1dd3da5700 1 -- 172.21.15.29:0/2987466049 wait complete. 2024-03-11T18:59:38.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.400+0000 7f1dd3da5700 1 Processor -- start 2024-03-11T18:59:38.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.400+0000 7f1dd3da5700 1 -- start start 2024-03-11T18:59:38.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.401+0000 7f1dd3da5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dcc105750 0x7f1dcc19cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:59:38.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.401+0000 7f1dd3da5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dcc19d060 con 0x7f1dcc105750 2024-03-11T18:59:38.993 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.401+0000 7f1dd1b41700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dcc105750 0x7f1dcc19cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:59:38.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.401+0000 7f1dd1b41700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dcc105750 0x7f1dcc19cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51408/0 (socket says 172.21.15.29:51408) 2024-03-11T18:59:38.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.401+0000 7f1dd1b41700 1 -- 172.21.15.29:0/3176013722 learned_addr learned my addr 172.21.15.29:0/3176013722 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T18:59:38.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.401+0000 7f1dd1b41700 1 -- 172.21.15.29:0/3176013722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc8009550 con 0x7f1dcc105750 2024-03-11T18:59:38.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.401+0000 7f1dd1b41700 1 --2- 172.21.15.29:0/3176013722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dcc105750 0x7f1dcc19cb20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8004000 tx=0x7f1dc8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:59:38.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.401+0000 7f1dc2ffd700 1 -- 172.21.15.29:0/3176013722 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1dc8011630 con 0x7f1dcc105750 2024-03-11T18:59:38.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.401+0000 7f1dc2ffd700 1 -- 172.21.15.29:0/3176013722 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc8011790 con 0x7f1dcc105750 2024-03-11T18:59:38.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.401+0000 7f1dd3da5700 1 -- 172.21.15.29:0/3176013722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dcc19d260 con 0x7f1dcc105750 2024-03-11T18:59:38.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.401+0000 7f1dc2ffd700 1 -- 172.21.15.29:0/3176013722 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1dc8011a30 con 0x7f1dcc105750 2024-03-11T18:59:38.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.401+0000 7f1dd3da5700 1 -- 172.21.15.29:0/3176013722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dcc19d700 con 0x7f1dcc105750 2024-03-11T18:59:38.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.402+0000 7f1dc2ffd700 1 -- 172.21.15.29:0/3176013722 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f1dc8022070 con 0x7f1dcc105750 2024-03-11T18:59:38.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.402+0000 7f1dc2ffd700 1 -- 172.21.15.29:0/3176013722 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1dc801c450 con 0x7f1dcc105750 2024-03-11T18:59:38.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.402+0000 7f1dd3da5700 1 -- 172.21.15.29:0/3176013722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dcc108a50 con 0x7f1dcc105750 2024-03-11T18:59:38.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.404+0000 7f1dc2ffd700 1 -- 172.21.15.29:0/3176013722 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f1dc8011b90 con 0x7f1dcc105750 2024-03-11T18:59:38.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.475+0000 7f1dd3da5700 1 -- 172.21.15.29:0/3176013722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f1dcc108c60 con 0x7f1dcc105750 2024-03-11T18:59:38.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.478+0000 7f1dc2ffd700 1 -- 172.21.15.29:0/3176013722 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f1dc8011de0 con 0x7f1dcc105750 2024-03-11T18:59:38.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.478+0000 7f1dc2ffd700 1 -- 172.21.15.29:0/3176013722 <== mon.0 v2:172.21.15.29:3300/0 8 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f1dc801c6b0 con 0x7f1dcc105750 2024-03-11T18:59:38.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.479+0000 7f1dd3da5700 1 -- 172.21.15.29:0/3176013722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dcc105750 msgr2=0x7f1dcc19cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:59:38.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.479+0000 7f1dd3da5700 1 --2- 172.21.15.29:0/3176013722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dcc105750 0x7f1dcc19cb20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8004000 tx=0x7f1dc8004750 comp rx=0 tx=0).stop 2024-03-11T18:59:38.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.479+0000 7f1dd3da5700 1 -- 172.21.15.29:0/3176013722 shutdown_connections 2024-03-11T18:59:38.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.479+0000 7f1dd3da5700 1 --2- 172.21.15.29:0/3176013722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1dcc105750 0x7f1dcc19cb20 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:59:38.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.479+0000 7f1dd3da5700 1 -- 172.21.15.29:0/3176013722 >> 172.21.15.29:0/3176013722 conn(0x7f1dcc1010a0 msgr2=0x7f1dcc106d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:59:38.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.479+0000 7f1dd3da5700 1 -- 172.21.15.29:0/3176013722 shutdown_connections 2024-03-11T18:59:38.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:38.479+0000 7f1dd3da5700 1 -- 172.21.15.29:0/3176013722 wait complete. 2024-03-11T18:59:38.995 INFO:teuthology.orchestra.run.smithi029.stdout:Generating new minimal ceph.conf... 2024-03-11T18:59:39.986 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.361+0000 7fa69daab700 1 Processor -- start 2024-03-11T18:59:39.986 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.361+0000 7fa69daab700 1 -- start start 2024-03-11T18:59:39.986 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.361+0000 7fa69daab700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698107480 0x7fa6981078a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:59:39.986 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.361+0000 7fa69daab700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa698107e70 con 0x7fa698107480 2024-03-11T18:59:39.986 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.361+0000 7fa6977fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698107480 0x7fa6981078a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:59:39.986 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.361+0000 7fa6977fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698107480 0x7fa6981078a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51412/0 (socket says 172.21.15.29:51412) 2024-03-11T18:59:39.986 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.361+0000 7fa6977fe700 1 -- 172.21.15.29:0/37582511 learned_addr learned my addr 172.21.15.29:0/37582511 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T18:59:39.986 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.362+0000 7fa6977fe700 1 -- 172.21.15.29:0/37582511 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa698108630 con 0x7fa698107480 2024-03-11T18:59:39.987 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.362+0000 7fa6977fe700 1 --2- 172.21.15.29:0/37582511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698107480 0x7fa6981078a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa6880098a0 tx=0x7fa688009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191f00772033a477 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:59:39.987 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.362+0000 7fa6967fc700 1 -- 172.21.15.29:0/37582511 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa688004160 con 0x7fa698107480 2024-03-11T18:59:39.987 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.362+0000 7fa6967fc700 1 -- 172.21.15.29:0/37582511 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fa6880042c0 con 0x7fa698107480 2024-03-11T18:59:39.987 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.362+0000 7fa6967fc700 1 -- 172.21.15.29:0/37582511 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa688011a50 con 0x7fa698107480 2024-03-11T18:59:39.987 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.363+0000 7fa69daab700 1 -- 172.21.15.29:0/37582511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698107480 msgr2=0x7fa6981078a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:59:39.987 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.363+0000 7fa69daab700 1 --2- 172.21.15.29:0/37582511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698107480 0x7fa6981078a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa6880098a0 tx=0x7fa688009bb0 comp rx=0 tx=0).stop 2024-03-11T18:59:39.987 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.363+0000 7fa69daab700 1 -- 172.21.15.29:0/37582511 shutdown_connections 2024-03-11T18:59:39.987 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.363+0000 7fa69daab700 1 --2- 172.21.15.29:0/37582511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698107480 0x7fa6981078a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:59:39.987 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.363+0000 7fa69daab700 1 -- 172.21.15.29:0/37582511 >> 172.21.15.29:0/37582511 conn(0x7fa698102a80 msgr2=0x7fa698104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:59:39.987 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.363+0000 7fa69daab700 1 -- 172.21.15.29:0/37582511 shutdown_connections 2024-03-11T18:59:39.987 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.363+0000 7fa69daab700 1 -- 172.21.15.29:0/37582511 wait complete. 2024-03-11T18:59:39.987 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.363+0000 7fa69daab700 1 Processor -- start 2024-03-11T18:59:39.987 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.364+0000 7fa69daab700 1 -- start start 2024-03-11T18:59:39.987 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.364+0000 7fa69daab700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698107480 0x7fa69819b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:59:39.988 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.364+0000 7fa69daab700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69819b660 con 0x7fa698107480 2024-03-11T18:59:39.988 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.364+0000 7fa6977fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698107480 0x7fa69819b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:59:39.988 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.364+0000 7fa6977fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698107480 0x7fa69819b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51422/0 (socket says 172.21.15.29:51422) 2024-03-11T18:59:39.988 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.364+0000 7fa6977fe700 1 -- 172.21.15.29:0/3387543920 learned_addr learned my addr 172.21.15.29:0/3387543920 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T18:59:39.988 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.364+0000 7fa6977fe700 1 -- 172.21.15.29:0/3387543920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa688009550 con 0x7fa698107480 2024-03-11T18:59:39.988 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.364+0000 7fa6977fe700 1 --2- 172.21.15.29:0/3387543920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698107480 0x7fa69819b120 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa688000c00 tx=0x7fa688011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:59:39.988 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.364+0000 7fa694ff9700 1 -- 172.21.15.29:0/3387543920 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa688003a40 con 0x7fa698107480 2024-03-11T18:59:39.988 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.364+0000 7fa694ff9700 1 -- 172.21.15.29:0/3387543920 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fa688003ba0 con 0x7fa698107480 2024-03-11T18:59:39.988 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.364+0000 7fa69daab700 1 -- 172.21.15.29:0/3387543920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa69819b860 con 0x7fa698107480 2024-03-11T18:59:39.988 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.364+0000 7fa694ff9700 1 -- 172.21.15.29:0/3387543920 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa68801a700 con 0x7fa698107480 2024-03-11T18:59:39.988 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.364+0000 7fa69daab700 1 -- 172.21.15.29:0/3387543920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa69819bd00 con 0x7fa698107480 2024-03-11T18:59:39.988 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.365+0000 7fa694ff9700 1 -- 172.21.15.29:0/3387543920 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fa68801acc0 con 0x7fa698107480 2024-03-11T18:59:39.988 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.365+0000 7fa694ff9700 1 -- 172.21.15.29:0/3387543920 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa6880196c0 con 0x7fa698107480 2024-03-11T18:59:39.988 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.366+0000 7fa69daab700 1 -- 172.21.15.29:0/3387543920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6980632a0 con 0x7fa698107480 2024-03-11T18:59:39.988 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.368+0000 7fa694ff9700 1 -- 172.21.15.29:0/3387543920 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa688024b80 con 0x7fa698107480 2024-03-11T18:59:39.988 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.438+0000 7fa69daab700 1 -- 172.21.15.29:0/3387543920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fa69819e870 con 0x7fa698107480 2024-03-11T18:59:39.989 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.438+0000 7fa694ff9700 1 -- 172.21.15.29:0/3387543920 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fa688019920 con 0x7fa698107480 2024-03-11T18:59:39.989 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.439+0000 7fa69daab700 1 -- 172.21.15.29:0/3387543920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698107480 msgr2=0x7fa69819b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:59:39.989 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.439+0000 7fa69daab700 1 --2- 172.21.15.29:0/3387543920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698107480 0x7fa69819b120 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa688000c00 tx=0x7fa688011ab0 comp rx=0 tx=0).stop 2024-03-11T18:59:39.989 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.439+0000 7fa69daab700 1 -- 172.21.15.29:0/3387543920 shutdown_connections 2024-03-11T18:59:39.989 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.439+0000 7fa69daab700 1 --2- 172.21.15.29:0/3387543920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa698107480 0x7fa69819b120 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:59:39.989 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.439+0000 7fa69daab700 1 -- 172.21.15.29:0/3387543920 >> 172.21.15.29:0/3387543920 conn(0x7fa698102a80 msgr2=0x7fa698103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:59:39.989 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.439+0000 7fa69daab700 1 -- 172.21.15.29:0/3387543920 shutdown_connections 2024-03-11T18:59:39.989 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:39.439+0000 7fa69daab700 1 -- 172.21.15.29:0/3387543920 wait complete. 2024-03-11T18:59:39.989 INFO:teuthology.orchestra.run.smithi029.stdout:Restarting the monitor... 2024-03-11T18:59:40.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:39 smithi029 systemd[1]: Stopping Ceph mon.a for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T18:59:40.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:40 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-a[32057]: 2024-03-11T18:59:40.229+0000 7f7008888700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:59:40.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:40 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-a[32057]: 2024-03-11T18:59:40.229+0000 7f7008888700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T18:59:40.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:40 smithi029 podman[32308]: 2024-03-11 18:59:40.394220377 +0000 UTC m=+0.251610314 container died 0dee521acb53f2d6f15def0db86b715a2d5d43ace0cbb6f5523ffeeb89971ef2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-03-11T18:59:40.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:40 smithi029 podman[32308]: 2024-03-11 18:59:40.723428207 +0000 UTC m=+0.580818193 container cleanup 0dee521acb53f2d6f15def0db86b715a2d5d43ace0cbb6f5523ffeeb89971ef2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T18:59:40.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:40 smithi029 bash[32308]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-a 2024-03-11T18:59:41.346 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:40 smithi029 podman[32323]: 2024-03-11 18:59:40.909896616 +0000 UTC m=+0.510157660 container remove 0dee521acb53f2d6f15def0db86b715a2d5d43ace0cbb6f5523ffeeb89971ef2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T18:59:41.347 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 systemd[1]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-11T18:59:41.347 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 systemd[1]: Stopped Ceph mon.a for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T18:59:41.347 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 systemd[1]: Starting Ceph mon.a for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T18:59:41.617 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 podman[32420]: 2024-03-11 18:59:41.344654915 +0000 UTC m=+0.096258735 container create 3d682d103af0cf3c34c3451d8fcc2bc22be7d675cede943758165c2d82c4d417 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-11T18:59:41.618 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 podman[32420]: 2024-03-11 18:59:41.266647506 +0000 UTC m=+0.018251301 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:59:41.618 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 podman[32420]: 2024-03-11 18:59:41.54001207 +0000 UTC m=+0.291615849 container init 3d682d103af0cf3c34c3451d8fcc2bc22be7d675cede943758165c2d82c4d417 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T18:59:41.618 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 podman[32420]: 2024-03-11 18:59:41.542842565 +0000 UTC m=+0.294446350 container start 3d682d103af0cf3c34c3451d8fcc2bc22be7d675cede943758165c2d82c4d417 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-11T18:59:41.618 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T18:59:41.618 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 2 2024-03-11T18:59:41.618 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: pidfile_write: ignore empty --pid-file 2024-03-11T18:59:41.618 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: load: jerasure load: lrc 2024-03-11T18:59:41.618 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: RocksDB version: 7.9.2 2024-03-11T18:59:41.618 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Git sha 0 2024-03-11T18:59:41.618 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T18:59:41.618 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: DB SUMMARY 2024-03-11T18:59:41.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: DB Session ID: E742LWBJAVN0V0F89XBZ 2024-03-11T18:59:41.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: CURRENT file: CURRENT 2024-03-11T18:59:41.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T18:59:41.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-11T18:59:41.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-11T18:59:41.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 99342 ; 2024-03-11T18:59:41.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.error_if_exists: 0 2024-03-11T18:59:41.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.create_if_missing: 0 2024-03-11T18:59:41.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.paranoid_checks: 1 2024-03-11T18:59:41.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T18:59:41.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T18:59:41.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T18:59:41.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.env: 0x5570fba9aae0 2024-03-11T18:59:41.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T18:59:41.620 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.info_log: 0x5570fce958c0 2024-03-11T18:59:41.620 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T18:59:41.620 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.statistics: (nil) 2024-03-11T18:59:41.620 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.use_fsync: 0 2024-03-11T18:59:41.620 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_log_file_size: 0 2024-03-11T18:59:41.620 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T18:59:41.620 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T18:59:41.620 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T18:59:41.620 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T18:59:41.620 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.allow_fallocate: 1 2024-03-11T18:59:41.620 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T18:59:41.620 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T18:59:41.620 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.use_direct_reads: 0 2024-03-11T18:59:41.620 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T18:59:41.620 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T18:59:41.621 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.db_log_dir: 2024-03-11T18:59:41.621 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.wal_dir: 2024-03-11T18:59:41.621 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T18:59:41.621 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T18:59:41.621 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T18:59:41.621 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T18:59:41.621 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T18:59:41.621 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T18:59:41.621 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T18:59:41.621 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T18:59:41.621 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.write_buffer_manager: 0x5570fce69400 2024-03-11T18:59:41.621 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T18:59:41.621 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T18:59:41.621 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T18:59:41.621 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T18:59:41.622 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T18:59:41.622 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T18:59:41.623 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T18:59:41.623 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T18:59:41.623 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.unordered_write: 0 2024-03-11T18:59:41.623 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T18:59:41.623 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T18:59:41.623 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T18:59:41.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T18:59:41.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.row_cache: None 2024-03-11T18:59:41.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.wal_filter: None 2024-03-11T18:59:41.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T18:59:41.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T18:59:41.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.two_write_queues: 0 2024-03-11T18:59:41.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T18:59:41.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.wal_compression: 0 2024-03-11T18:59:41.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.atomic_flush: 0 2024-03-11T18:59:41.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T18:59:41.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T18:59:41.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T18:59:41.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.log_readahead_size: 0 2024-03-11T18:59:41.625 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T18:59:41.625 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T18:59:41.625 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T18:59:41.625 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T18:59:41.625 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T18:59:41.625 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T18:59:41.625 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T18:59:41.625 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_background_jobs: 2 2024-03-11T18:59:41.625 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_background_compactions: -1 2024-03-11T18:59:41.625 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_subcompactions: 1 2024-03-11T18:59:41.626 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T18:59:41.626 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T18:59:41.626 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T18:59:41.626 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T18:59:41.626 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T18:59:41.626 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T18:59:41.626 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T18:59:41.626 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T18:59:41.626 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_open_files: -1 2024-03-11T18:59:41.626 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T18:59:41.626 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T18:59:41.626 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T18:59:41.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T18:59:41.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_background_flushes: -1 2024-03-11T18:59:41.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Compression algorithms supported: 2024-03-11T18:59:41.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T18:59:41.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: kZSTD supported: 0 2024-03-11T18:59:41.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: kXpressCompression supported: 0 2024-03-11T18:59:41.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T18:59:41.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: kZlibCompression supported: 1 2024-03-11T18:59:41.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: kSnappyCompression supported: 1 2024-03-11T18:59:41.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: kLZ4Compression supported: 1 2024-03-11T18:59:41.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: kBZip2Compression supported: 0 2024-03-11T18:59:41.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T18:59:41.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T18:59:41.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-11T18:59:41.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T18:59:41.628 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T18:59:41.628 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.merge_operator: 2024-03-11T18:59:41.628 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compaction_filter: None 2024-03-11T18:59:41.628 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T18:59:41.628 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T18:59:41.628 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T18:59:41.628 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T18:59:41.628 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5570fce959c0) 2024-03-11T18:59:41.628 INFO:journalctl@ceph.mon.a.smithi029.stdout: cache_index_and_filter_blocks: 1 2024-03-11T18:59:41.628 INFO:journalctl@ceph.mon.a.smithi029.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T18:59:41.628 INFO:journalctl@ceph.mon.a.smithi029.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T18:59:41.628 INFO:journalctl@ceph.mon.a.smithi029.stdout: pin_top_level_index_and_filter: 1 2024-03-11T18:59:41.628 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_type: 0 2024-03-11T18:59:41.628 INFO:journalctl@ceph.mon.a.smithi029.stdout: data_block_index_type: 0 2024-03-11T18:59:41.628 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_shortening: 1 2024-03-11T18:59:41.629 INFO:journalctl@ceph.mon.a.smithi029.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T18:59:41.629 INFO:journalctl@ceph.mon.a.smithi029.stdout: checksum: 4 2024-03-11T18:59:41.629 INFO:journalctl@ceph.mon.a.smithi029.stdout: no_block_cache: 0 2024-03-11T18:59:41.629 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache: 0x5570fce99090 2024-03-11T18:59:41.629 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_name: BinnedLRUCache 2024-03-11T18:59:41.629 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_options: 2024-03-11T18:59:41.629 INFO:journalctl@ceph.mon.a.smithi029.stdout: capacity : 536870912 2024-03-11T18:59:41.629 INFO:journalctl@ceph.mon.a.smithi029.stdout: num_shard_bits : 4 2024-03-11T18:59:41.629 INFO:journalctl@ceph.mon.a.smithi029.stdout: strict_capacity_limit : 0 2024-03-11T18:59:41.629 INFO:journalctl@ceph.mon.a.smithi029.stdout: high_pri_pool_ratio: 0.000 2024-03-11T18:59:41.629 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_compressed: (nil) 2024-03-11T18:59:41.630 INFO:journalctl@ceph.mon.a.smithi029.stdout: persistent_cache: (nil) 2024-03-11T18:59:41.630 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_size: 4096 2024-03-11T18:59:41.630 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_size_deviation: 10 2024-03-11T18:59:41.630 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_restart_interval: 16 2024-03-11T18:59:41.630 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_block_restart_interval: 1 2024-03-11T18:59:41.630 INFO:journalctl@ceph.mon.a.smithi029.stdout: metadata_block_size: 4096 2024-03-11T18:59:41.630 INFO:journalctl@ceph.mon.a.smithi029.stdout: partition_filters: 0 2024-03-11T18:59:41.630 INFO:journalctl@ceph.mon.a.smithi029.stdout: use_delta_encoding: 1 2024-03-11T18:59:41.630 INFO:journalctl@ceph.mon.a.smithi029.stdout: filter_policy: bloomfilter 2024-03-11T18:59:41.630 INFO:journalctl@ceph.mon.a.smithi029.stdout: whole_key_filtering: 1 2024-03-11T18:59:41.630 INFO:journalctl@ceph.mon.a.smithi029.stdout: verify_compression: 0 2024-03-11T18:59:41.630 INFO:journalctl@ceph.mon.a.smithi029.stdout: read_amp_bytes_per_bit: 0 2024-03-11T18:59:41.630 INFO:journalctl@ceph.mon.a.smithi029.stdout: format_version: 5 2024-03-11T18:59:41.631 INFO:journalctl@ceph.mon.a.smithi029.stdout: enable_index_compression: 1 2024-03-11T18:59:41.631 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_align: 0 2024-03-11T18:59:41.631 INFO:journalctl@ceph.mon.a.smithi029.stdout: max_auto_readahead_size: 262144 2024-03-11T18:59:41.631 INFO:journalctl@ceph.mon.a.smithi029.stdout: prepopulate_block_cache: 0 2024-03-11T18:59:41.631 INFO:journalctl@ceph.mon.a.smithi029.stdout: initial_auto_readahead_size: 8192 2024-03-11T18:59:41.631 INFO:journalctl@ceph.mon.a.smithi029.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T18:59:41.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T18:59:41.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T18:59:41.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compression: NoCompression 2024-03-11T18:59:41.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T18:59:41.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T18:59:41.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T18:59:41.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.num_levels: 7 2024-03-11T18:59:41.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T18:59:41.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T18:59:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T18:59:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T18:59:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T18:59:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T18:59:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T18:59:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T18:59:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T18:59:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T18:59:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T18:59:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T18:59:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T18:59:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T18:59:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T18:59:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T18:59:41.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T18:59:41.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T18:59:41.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T18:59:41.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T18:59:41.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T18:59:41.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T18:59:41.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T18:59:41.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T18:59:41.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T18:59:41.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T18:59:41.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T18:59:41.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T18:59:41.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T18:59:41.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T18:59:41.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T18:59:41.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T18:59:41.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T18:59:41.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T18:59:41.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T18:59:41.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T18:59:41.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T18:59:41.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T18:59:41.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T18:59:41.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T18:59:41.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T18:59:41.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T18:59:41.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T18:59:41.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T18:59:41.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T18:59:41.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T18:59:41.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T18:59:41.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T18:59:41.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T18:59:41.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T18:59:41.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T18:59:41.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T18:59:41.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T18:59:41.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.table_properties_collectors: 2024-03-11T18:59:41.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.inplace_update_support: 0 2024-03-11T18:59:41.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T18:59:41.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T18:59:41.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T18:59:41.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T18:59:41.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.bloom_locality: 0 2024-03-11T18:59:41.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.max_successive_merges: 0 2024-03-11T18:59:41.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T18:59:41.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T18:59:41.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T18:59:41.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T18:59:41.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.ttl: 2592000 2024-03-11T18:59:41.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T18:59:41.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T18:59:41.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T18:59:41.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.enable_blob_files: false 2024-03-11T18:59:41.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.min_blob_size: 0 2024-03-11T18:59:41.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T18:59:41.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T18:59:41.637 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T18:59:41.637 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T18:59:41.637 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T18:59:41.637 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T18:59:41.637 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T18:59:41.637 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T18:59:41.637 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-11T18:59:41.637 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-11T18:59:41.637 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: afdd6816-8617-461e-a52c-1156e5830f46 2024-03-11T18:59:41.637 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710183581571645, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-11T18:59:41.637 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-11T18:59:41.637 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710183581572636, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94534, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 92545, "index_size": 271, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13666, "raw_average_key_size": 50, "raw_value_size": 85127, "raw_average_value_size": 311, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710183581, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "afdd6816-8617-461e-a52c-1156e5830f46", "db_session_id": "E742LWBJAVN0V0F89XBZ", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-11T18:59:41.637 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710183581572719, "job": 1, "event": "recovery_finished"} 2024-03-11T18:59:41.638 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-11T18:59:41.638 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T18:59:41.638 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5570fcf8a000 2024-03-11T18:59:41.638 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: DB pointer 0x5570fcf74000 2024-03-11T18:59:41.638 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T18:59:41.638 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T18:59:41.638 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** DB Stats ** 2024-03-11T18:59:41.638 INFO:journalctl@ceph.mon.a.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T18:59:41.638 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T18:59:41.638 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T18:59:41.638 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T18:59:41.638 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T18:59:41.638 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T18:59:41.638 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T18:59:41.639 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-03-11T18:59:41.639 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** Compaction Stats [default] ** 2024-03-11T18:59:41.639 INFO:journalctl@ceph.mon.a.smithi029.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T18:59:41.639 INFO:journalctl@ceph.mon.a.smithi029.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T18:59:41.639 INFO:journalctl@ceph.mon.a.smithi029.stdout: L0 2/0 94.15 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T18:59:41.639 INFO:journalctl@ceph.mon.a.smithi029.stdout: Sum 2/0 94.15 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T18:59:41.639 INFO:journalctl@ceph.mon.a.smithi029.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T18:59:41.639 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-03-11T18:59:41.639 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** Compaction Stats [default] ** 2024-03-11T18:59:41.639 INFO:journalctl@ceph.mon.a.smithi029.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T18:59:41.639 INFO:journalctl@ceph.mon.a.smithi029.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T18:59:41.639 INFO:journalctl@ceph.mon.a.smithi029.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 141.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T18:59:41.639 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-03-11T18:59:41.640 INFO:journalctl@ceph.mon.a.smithi029.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T18:59:41.640 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-03-11T18:59:41.640 INFO:journalctl@ceph.mon.a.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T18:59:41.640 INFO:journalctl@ceph.mon.a.smithi029.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T18:59:41.640 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T18:59:41.640 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T18:59:41.640 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T18:59:41.640 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T18:59:41.640 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative compaction: 0.00 GB write, 30.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T18:59:41.640 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval compaction: 0.00 GB write, 30.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T18:59:41.640 INFO:journalctl@ceph.mon.a.smithi029.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T18:59:41.640 INFO:journalctl@ceph.mon.a.smithi029.stdout: Block cache BinnedLRUCache@0x5570fce99090#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-11T18:59:41.640 INFO:journalctl@ceph.mon.a.smithi029.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-11T18:59:41.641 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-03-11T18:59:41.641 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T18:59:41.641 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: starting mon.a rank 0 at public addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] at bind addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T18:59:41.641 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: mon.a@-1(???) e1 preinit fsid 638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T18:59:41.641 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: mon.a@-1(???).mds e1 new map 2024-03-11T18:59:41.641 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: mon.a@-1(???).mds e1 print_map 2024-03-11T18:59:41.641 INFO:journalctl@ceph.mon.a.smithi029.stdout: e1 2024-03-11T18:59:41.641 INFO:journalctl@ceph.mon.a.smithi029.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T18:59:41.641 INFO:journalctl@ceph.mon.a.smithi029.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T18:59:41.641 INFO:journalctl@ceph.mon.a.smithi029.stdout: legacy client fscid: -1 2024-03-11T18:59:41.641 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-03-11T18:59:41.641 INFO:journalctl@ceph.mon.a.smithi029.stdout: No filesystems configured 2024-03-11T18:59:41.641 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T18:59:41.641 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T18:59:41.641 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T18:59:41.642 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T18:59:41.642 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-11T18:59:41.642 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 bash[32420]: 3d682d103af0cf3c34c3451d8fcc2bc22be7d675cede943758165c2d82c4d417 2024-03-11T18:59:41.642 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 systemd[1]: Started Ceph mon.a for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T18:59:41.642 INFO:teuthology.orchestra.run.smithi029.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-11T18:59:41.890 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T18:59:41.891 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: monmap e1: 1 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T18:59:41.891 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: fsmap 2024-03-11T18:59:41.891 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T18:59:41.891 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:41 smithi029 ceph-mon[32439]: mgrmap e1: no daemons active 2024-03-11T18:59:42.601 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.991+0000 7f9f9c36d700 1 Processor -- start 2024-03-11T18:59:42.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.991+0000 7f9f9c36d700 1 -- start start 2024-03-11T18:59:42.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.992+0000 7f9f9c36d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f94105d30 0x7f9f94106150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:59:42.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.992+0000 7f9f9c36d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9407ae10 con 0x7f9f94105d30 2024-03-11T18:59:42.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.992+0000 7f9f9a109700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f94105d30 0x7f9f94106150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:59:42.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.992+0000 7f9f9a109700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f94105d30 0x7f9f94106150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51424/0 (socket says 172.21.15.29:51424) 2024-03-11T18:59:42.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.992+0000 7f9f9a109700 1 -- 172.21.15.29:0/3165435520 learned_addr learned my addr 172.21.15.29:0/3165435520 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T18:59:42.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.992+0000 7f9f9a109700 1 -- 172.21.15.29:0/3165435520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f94106690 con 0x7f9f94105d30 2024-03-11T18:59:42.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.993+0000 7f9f9a109700 1 --2- 172.21.15.29:0/3165435520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f94105d30 0x7f9f94106150 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9f900098a0 tx=0x7f9f90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d3f92fe3c9a81c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:59:42.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.993+0000 7f9f99107700 1 -- 172.21.15.29:0/3165435520 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f90004160 con 0x7f9f94105d30 2024-03-11T18:59:42.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.993+0000 7f9f99107700 1 -- 172.21.15.29:0/3165435520 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f9f900042c0 con 0x7f9f94105d30 2024-03-11T18:59:42.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.993+0000 7f9f99107700 1 -- 172.21.15.29:0/3165435520 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f90011a50 con 0x7f9f94105d30 2024-03-11T18:59:42.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.993+0000 7f9f9c36d700 1 -- 172.21.15.29:0/3165435520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f94105d30 msgr2=0x7f9f94106150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:59:42.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.993+0000 7f9f9c36d700 1 --2- 172.21.15.29:0/3165435520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f94105d30 0x7f9f94106150 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9f900098a0 tx=0x7f9f90009bb0 comp rx=0 tx=0).stop 2024-03-11T18:59:42.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.993+0000 7f9f9c36d700 1 -- 172.21.15.29:0/3165435520 shutdown_connections 2024-03-11T18:59:42.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.993+0000 7f9f9c36d700 1 --2- 172.21.15.29:0/3165435520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f94105d30 0x7f9f94106150 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:59:42.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.993+0000 7f9f9c36d700 1 -- 172.21.15.29:0/3165435520 >> 172.21.15.29:0/3165435520 conn(0x7f9f94101930 msgr2=0x7f9f94103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:59:42.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.993+0000 7f9f9c36d700 1 -- 172.21.15.29:0/3165435520 shutdown_connections 2024-03-11T18:59:42.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.993+0000 7f9f9c36d700 1 -- 172.21.15.29:0/3165435520 wait complete. 2024-03-11T18:59:42.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.994+0000 7f9f9c36d700 1 Processor -- start 2024-03-11T18:59:42.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.994+0000 7f9f9c36d700 1 -- start start 2024-03-11T18:59:42.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.994+0000 7f9f9c36d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f94105d30 0x7f9f9419d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:59:42.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.994+0000 7f9f9c36d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9419d880 con 0x7f9f94105d30 2024-03-11T18:59:42.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.994+0000 7f9f9a109700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f94105d30 0x7f9f9419d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:59:42.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.994+0000 7f9f9a109700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f94105d30 0x7f9f9419d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51436/0 (socket says 172.21.15.29:51436) 2024-03-11T18:59:42.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.994+0000 7f9f9a109700 1 -- 172.21.15.29:0/2007071363 learned_addr learned my addr 172.21.15.29:0/2007071363 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T18:59:42.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.995+0000 7f9f9a109700 1 -- 172.21.15.29:0/2007071363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f90009550 con 0x7f9f94105d30 2024-03-11T18:59:42.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.995+0000 7f9f9a109700 1 --2- 172.21.15.29:0/2007071363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f94105d30 0x7f9f9419d340 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9f90006af0 tx=0x7f9f90011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:59:42.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.995+0000 7f9f8b7fe700 1 -- 172.21.15.29:0/2007071363 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f90003930 con 0x7f9f94105d30 2024-03-11T18:59:42.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.995+0000 7f9f8b7fe700 1 -- 172.21.15.29:0/2007071363 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f9f90003a90 con 0x7f9f94105d30 2024-03-11T18:59:42.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.995+0000 7f9f9c36d700 1 -- 172.21.15.29:0/2007071363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9419da80 con 0x7f9f94105d30 2024-03-11T18:59:42.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.995+0000 7f9f8b7fe700 1 -- 172.21.15.29:0/2007071363 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f90019670 con 0x7f9f94105d30 2024-03-11T18:59:42.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.995+0000 7f9f9c36d700 1 -- 172.21.15.29:0/2007071363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f9419df20 con 0x7f9f94105d30 2024-03-11T18:59:42.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.995+0000 7f9f8b7fe700 1 -- 172.21.15.29:0/2007071363 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9f90019cc0 con 0x7f9f94105d30 2024-03-11T18:59:42.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.995+0000 7f9f8b7fe700 1 -- 172.21.15.29:0/2007071363 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9f90024a40 con 0x7f9f94105d30 2024-03-11T18:59:42.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.996+0000 7f9f9c36d700 1 -- 172.21.15.29:0/2007071363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f94197010 con 0x7f9f94105d30 2024-03-11T18:59:42.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:41.998+0000 7f9f8b7fe700 1 -- 172.21.15.29:0/2007071363 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9f9001aa30 con 0x7f9f94105d30 2024-03-11T18:59:42.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:42.068+0000 7f9f9c36d700 1 -- 172.21.15.29:0/2007071363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f9f94107490 con 0x7f9f94105d30 2024-03-11T18:59:42.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:42.070+0000 7f9f8b7fe700 1 -- 172.21.15.29:0/2007071363 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9f9001abd0 con 0x7f9f94105d30 2024-03-11T18:59:42.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:42.071+0000 7f9f9c36d700 1 -- 172.21.15.29:0/2007071363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f94105d30 msgr2=0x7f9f9419d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:59:42.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:42.071+0000 7f9f9c36d700 1 --2- 172.21.15.29:0/2007071363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f94105d30 0x7f9f9419d340 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9f90006af0 tx=0x7f9f90011ab0 comp rx=0 tx=0).stop 2024-03-11T18:59:42.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:42.071+0000 7f9f9c36d700 1 -- 172.21.15.29:0/2007071363 shutdown_connections 2024-03-11T18:59:42.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:42.071+0000 7f9f9c36d700 1 --2- 172.21.15.29:0/2007071363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f94105d30 0x7f9f9419d340 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:59:42.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:42.071+0000 7f9f9c36d700 1 -- 172.21.15.29:0/2007071363 >> 172.21.15.29:0/2007071363 conn(0x7f9f94101930 msgr2=0x7f9f94194150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:59:42.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:42.071+0000 7f9f9c36d700 1 -- 172.21.15.29:0/2007071363 shutdown_connections 2024-03-11T18:59:42.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:42.071+0000 7f9f9c36d700 1 -- 172.21.15.29:0/2007071363 wait complete. 2024-03-11T18:59:42.609 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-11T18:59:42.624 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-11T18:59:42.624 INFO:teuthology.orchestra.run.smithi029.stdout:Creating mgr... 2024-03-11T18:59:42.624 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-11T18:59:42.624 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-11T18:59:42.871 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mgr.x 2024-03-11T18:59:42.871 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to reset failed state of unit ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mgr.x.service: Unit ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mgr.x.service not loaded. 2024-03-11T18:59:43.013 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-638915c2-dfd9-11ee-95c8-87774f69a715.target.wants/ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-638915c2-dfd9-11ee-95c8-87774f69a715@.service. 2024-03-11T18:59:43.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:43 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2007071363' entity='client.admin' 2024-03-11T18:59:43.484 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:43 smithi029 podman[32652]: 2024-03-11 18:59:43.408760989 +0000 UTC m=+0.285147121 container start f43e9b0478f2758e59775146c378ee26e6d47d7fddc4fa997d9dad6119e10082 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, ceph=True) 2024-03-11T18:59:43.484 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:43 smithi029 bash[32652]: f43e9b0478f2758e59775146c378ee26e6d47d7fddc4fa997d9dad6119e10082 2024-03-11T18:59:43.496 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T18:59:43.496 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-03-11T18:59:43.503 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T18:59:43.503 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-11T18:59:43.503 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-03-11T18:59:43.503 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T18:59:43.511 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T18:59:43.511 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-03-11T18:59:43.519 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T18:59:43.519 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-11T18:59:43.519 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-03-11T18:59:43.519 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-11T18:59:43.519 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr to start... 2024-03-11T18:59:43.519 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr... 2024-03-11T18:59:43.742 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:43 smithi029 systemd[1]: Started Ceph mgr.x for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T18:59:44.004 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:43 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:43.973+0000 7f0edbd8f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T18:59:44.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:44 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/659282304' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T18:59:44.406 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:44 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:44.073+0000 7f0edbd8f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T18:59:44.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-11T18:59:44.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-11T18:59:44.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "638915c2-dfd9-11ee-95c8-87774f69a715", 2024-03-11T18:59:44.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T18:59:44.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T18:59:44.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T18:59:44.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T18:59:44.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:59:44.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T18:59:44.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T18:59:44.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-03-11T18:59:44.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-11T18:59:44.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T18:59:44.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T18:59:44.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-11T18:59:44.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-11T18:59:44.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T18:59:44.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:59:44.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T18:59:44.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T18:59:44.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:59:44.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T18:59:44.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:59:44.511 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T18:59:44.519 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T18:59:44.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T18:59:44.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T18:59:44.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T18:59:44.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T18:59:44.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:59:44.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T18:59:44.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T18:59:44.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T18:59:44.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T18:59:44.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T18:59:44.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T18:59:44.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T18:59:44.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T18:59:44.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T18:59:44.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:59:44.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T18:59:44.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:59:44.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T18:59:44.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T18:59:44.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:59:44.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T18:59:44.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T18:59:44.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T18:59:44.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T18:59:44.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T18:59:44.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T18:59:44.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T18:59:44.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-11T18:59:44.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T18:59:44.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:59:44.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T18:59:44.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:59:44.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T18:59:36.895333+0000", 2024-03-11T18:59:44.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T18:59:44.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:59:44.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T18:59:44.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-11T18:59:44.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.921+0000 7f3c516d4700 1 Processor -- start 2024-03-11T18:59:44.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.921+0000 7f3c516d4700 1 -- start start 2024-03-11T18:59:44.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.921+0000 7f3c516d4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c4c107fe0 0x7f3c4c10a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:59:44.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.921+0000 7f3c516d4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c07ae30 con 0x7f3c4c107fe0 2024-03-11T18:59:44.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.921+0000 7f3c4affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c4c107fe0 0x7f3c4c10a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:59:44.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.921+0000 7f3c4affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c4c107fe0 0x7f3c4c10a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51478/0 (socket says 172.21.15.29:51478) 2024-03-11T18:59:44.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.921+0000 7f3c4affd700 1 -- 172.21.15.29:0/226267972 learned_addr learned my addr 172.21.15.29:0/226267972 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T18:59:44.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.922+0000 7f3c4affd700 1 -- 172.21.15.29:0/226267972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4c10a910 con 0x7f3c4c107fe0 2024-03-11T18:59:44.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.922+0000 7f3c4affd700 1 --2- 172.21.15.29:0/226267972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c4c107fe0 0x7f3c4c10a3d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c0098a0 tx=0x7f3c3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f6ae095e5e8b0c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:59:44.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.922+0000 7f3c49ffb700 1 -- 172.21.15.29:0/226267972 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c3c004160 con 0x7f3c4c107fe0 2024-03-11T18:59:44.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.922+0000 7f3c49ffb700 1 -- 172.21.15.29:0/226267972 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f3c3c0042c0 con 0x7f3c4c107fe0 2024-03-11T18:59:44.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.922+0000 7f3c49ffb700 1 -- 172.21.15.29:0/226267972 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c3c011a50 con 0x7f3c4c107fe0 2024-03-11T18:59:44.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.923+0000 7f3c516d4700 1 -- 172.21.15.29:0/226267972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c4c107fe0 msgr2=0x7f3c4c10a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:59:44.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.923+0000 7f3c516d4700 1 --2- 172.21.15.29:0/226267972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c4c107fe0 0x7f3c4c10a3d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c0098a0 tx=0x7f3c3c009bb0 comp rx=0 tx=0).stop 2024-03-11T18:59:44.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.923+0000 7f3c516d4700 1 -- 172.21.15.29:0/226267972 shutdown_connections 2024-03-11T18:59:44.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.923+0000 7f3c516d4700 1 --2- 172.21.15.29:0/226267972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c4c107fe0 0x7f3c4c10a3d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:59:44.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.923+0000 7f3c516d4700 1 -- 172.21.15.29:0/226267972 >> 172.21.15.29:0/226267972 conn(0x7f3c4c101a30 msgr2=0x7f3c4c103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:59:44.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.923+0000 7f3c516d4700 1 -- 172.21.15.29:0/226267972 shutdown_connections 2024-03-11T18:59:44.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.923+0000 7f3c516d4700 1 -- 172.21.15.29:0/226267972 wait complete. 2024-03-11T18:59:44.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.924+0000 7f3c516d4700 1 Processor -- start 2024-03-11T18:59:44.524 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.924+0000 7f3c516d4700 1 -- start start 2024-03-11T18:59:44.524 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.924+0000 7f3c516d4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c4c107fe0 0x7f3c4c19d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:59:44.524 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.924+0000 7f3c516d4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c19d920 con 0x7f3c4c107fe0 2024-03-11T18:59:44.524 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.924+0000 7f3c4affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c4c107fe0 0x7f3c4c19d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:59:44.524 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.924+0000 7f3c4affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c4c107fe0 0x7f3c4c19d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51484/0 (socket says 172.21.15.29:51484) 2024-03-11T18:59:44.524 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.924+0000 7f3c4affd700 1 -- 172.21.15.29:0/659282304 learned_addr learned my addr 172.21.15.29:0/659282304 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T18:59:44.524 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.924+0000 7f3c4affd700 1 -- 172.21.15.29:0/659282304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3c009550 con 0x7f3c4c107fe0 2024-03-11T18:59:44.524 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.924+0000 7f3c4affd700 1 --2- 172.21.15.29:0/659282304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c4c107fe0 0x7f3c4c19d3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c011a00 tx=0x7f3c3c011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:59:44.524 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.925+0000 7f3c33fff700 1 -- 172.21.15.29:0/659282304 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c3c003a40 con 0x7f3c4c107fe0 2024-03-11T18:59:44.524 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.925+0000 7f3c516d4700 1 -- 172.21.15.29:0/659282304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c4c19db20 con 0x7f3c4c107fe0 2024-03-11T18:59:44.524 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.925+0000 7f3c33fff700 1 -- 172.21.15.29:0/659282304 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f3c3c003ba0 con 0x7f3c4c107fe0 2024-03-11T18:59:44.524 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.925+0000 7f3c33fff700 1 -- 172.21.15.29:0/659282304 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c3c01a6e0 con 0x7f3c4c107fe0 2024-03-11T18:59:44.524 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.925+0000 7f3c516d4700 1 -- 172.21.15.29:0/659282304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c4c19dfc0 con 0x7f3c4c107fe0 2024-03-11T18:59:44.524 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.925+0000 7f3c33fff700 1 -- 172.21.15.29:0/659282304 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f3c3c01ad30 con 0x7f3c4c107fe0 2024-03-11T18:59:44.524 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.925+0000 7f3c33fff700 1 -- 172.21.15.29:0/659282304 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3c3c019670 con 0x7f3c4c107fe0 2024-03-11T18:59:44.525 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.926+0000 7f3c516d4700 1 -- 172.21.15.29:0/659282304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c4c10b640 con 0x7f3c4c107fe0 2024-03-11T18:59:44.525 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:43.928+0000 7f3c33fff700 1 -- 172.21.15.29:0/659282304 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f3c3c024af0 con 0x7f3c4c107fe0 2024-03-11T18:59:44.525 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:44.001+0000 7f3c516d4700 1 -- 172.21.15.29:0/659282304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3c4c1a0ab0 con 0x7f3c4c107fe0 2024-03-11T18:59:44.525 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:44.002+0000 7f3c33fff700 1 -- 172.21.15.29:0/659282304 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f3c3c024440 con 0x7f3c4c107fe0 2024-03-11T18:59:44.525 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:44.003+0000 7f3c516d4700 1 -- 172.21.15.29:0/659282304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c4c107fe0 msgr2=0x7f3c4c19d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:59:44.525 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:44.003+0000 7f3c516d4700 1 --2- 172.21.15.29:0/659282304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c4c107fe0 0x7f3c4c19d3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c011a00 tx=0x7f3c3c011ae0 comp rx=0 tx=0).stop 2024-03-11T18:59:44.525 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:44.003+0000 7f3c516d4700 1 -- 172.21.15.29:0/659282304 shutdown_connections 2024-03-11T18:59:44.525 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:44.003+0000 7f3c516d4700 1 --2- 172.21.15.29:0/659282304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c4c107fe0 0x7f3c4c19d3e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:59:44.525 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:44.003+0000 7f3c516d4700 1 -- 172.21.15.29:0/659282304 >> 172.21.15.29:0/659282304 conn(0x7f3c4c101a30 msgr2=0x7f3c4c1024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:59:44.525 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:44.004+0000 7f3c516d4700 1 -- 172.21.15.29:0/659282304 shutdown_connections 2024-03-11T18:59:44.525 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:44.004+0000 7f3c516d4700 1 -- 172.21.15.29:0/659282304 wait complete. 2024-03-11T18:59:44.525 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (1/15)... 2024-03-11T18:59:44.834 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:44 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:44.738+0000 7f0edbd8f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T18:59:44.835 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:44 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:44.832+0000 7f0edbd8f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T18:59:45.155 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:44 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:44.974+0000 7f0edbd8f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T18:59:45.155 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:45 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:45.071+0000 7f0edbd8f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T18:59:45.435 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:45 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:45.156+0000 7f0edbd8f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T18:59:45.435 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:45 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:45.432+0000 7f0edbd8f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T18:59:45.730 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:45 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:45.544+0000 7f0edbd8f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T18:59:45.730 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:45 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:45.631+0000 7f0edbd8f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T18:59:45.731 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:45 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:45.727+0000 7f0edbd8f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T18:59:46.155 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:45 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:45.905+0000 7f0edbd8f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T18:59:47.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:47 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2213150157' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T18:59:47.156 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:46 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:46.915+0000 7f0edbd8f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T18:59:47.156 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:47 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:47.064+0000 7f0edbd8f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T18:59:47.603 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:47 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:47.409+0000 7f0edbd8f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T18:59:47.604 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:47 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:47.505+0000 7f0edbd8f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T18:59:47.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-11T18:59:47.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-11T18:59:47.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "638915c2-dfd9-11ee-95c8-87774f69a715", 2024-03-11T18:59:47.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T18:59:47.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T18:59:47.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T18:59:47.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T18:59:47.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:59:47.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T18:59:47.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T18:59:47.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-03-11T18:59:47.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-11T18:59:47.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T18:59:47.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T18:59:47.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-11T18:59:47.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-11T18:59:47.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T18:59:47.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:59:47.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T18:59:47.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T18:59:47.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:59:47.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T18:59:47.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:59:47.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T18:59:47.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T18:59:47.617 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T18:59:47.617 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T18:59:47.617 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T18:59:47.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T18:59:47.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:59:47.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T18:59:47.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T18:59:47.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T18:59:47.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T18:59:47.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T18:59:47.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T18:59:47.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T18:59:47.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T18:59:47.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T18:59:47.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:59:47.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T18:59:47.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:59:47.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T18:59:47.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T18:59:47.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:59:47.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T18:59:47.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T18:59:47.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T18:59:47.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T18:59:47.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T18:59:47.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T18:59:47.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T18:59:47.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-11T18:59:47.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T18:59:47.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:59:47.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T18:59:47.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:59:47.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T18:59:36.895333+0000", 2024-03-11T18:59:47.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T18:59:47.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:59:47.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T18:59:47.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-11T18:59:47.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.936+0000 7fe64cc13700 1 Processor -- start 2024-03-11T18:59:47.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.936+0000 7fe64cc13700 1 -- start start 2024-03-11T18:59:47.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.936+0000 7fe64cc13700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe648107480 0x7fe6481078a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:59:47.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.936+0000 7fe64cc13700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe648107e70 con 0x7fe648107480 2024-03-11T18:59:47.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.937+0000 7fe64659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe648107480 0x7fe6481078a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:59:47.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.937+0000 7fe64659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe648107480 0x7fe6481078a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52326/0 (socket says 172.21.15.29:52326) 2024-03-11T18:59:47.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.937+0000 7fe64659c700 1 -- 172.21.15.29:0/2253196766 learned_addr learned my addr 172.21.15.29:0/2253196766 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T18:59:47.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.937+0000 7fe64659c700 1 -- 172.21.15.29:0/2253196766 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe648108630 con 0x7fe648107480 2024-03-11T18:59:47.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.937+0000 7fe64659c700 1 --2- 172.21.15.29:0/2253196766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe648107480 0x7fe6481078a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe6380098a0 tx=0x7fe638009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6e97cc45a95aa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:59:47.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.937+0000 7fe64559a700 1 -- 172.21.15.29:0/2253196766 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe638004160 con 0x7fe648107480 2024-03-11T18:59:47.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.937+0000 7fe64559a700 1 -- 172.21.15.29:0/2253196766 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fe6380042c0 con 0x7fe648107480 2024-03-11T18:59:47.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.937+0000 7fe64559a700 1 -- 172.21.15.29:0/2253196766 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe638011a50 con 0x7fe648107480 2024-03-11T18:59:47.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.938+0000 7fe64cc13700 1 -- 172.21.15.29:0/2253196766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe648107480 msgr2=0x7fe6481078a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:59:47.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.938+0000 7fe64cc13700 1 --2- 172.21.15.29:0/2253196766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe648107480 0x7fe6481078a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe6380098a0 tx=0x7fe638009bb0 comp rx=0 tx=0).stop 2024-03-11T18:59:47.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.938+0000 7fe64cc13700 1 -- 172.21.15.29:0/2253196766 shutdown_connections 2024-03-11T18:59:47.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.938+0000 7fe64cc13700 1 --2- 172.21.15.29:0/2253196766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe648107480 0x7fe6481078a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:59:47.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.938+0000 7fe64cc13700 1 -- 172.21.15.29:0/2253196766 >> 172.21.15.29:0/2253196766 conn(0x7fe648102a80 msgr2=0x7fe648104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:59:47.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.938+0000 7fe64cc13700 1 -- 172.21.15.29:0/2253196766 shutdown_connections 2024-03-11T18:59:47.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.938+0000 7fe64cc13700 1 -- 172.21.15.29:0/2253196766 wait complete. 2024-03-11T18:59:47.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.939+0000 7fe64cc13700 1 Processor -- start 2024-03-11T18:59:47.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.939+0000 7fe64cc13700 1 -- start start 2024-03-11T18:59:47.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.939+0000 7fe64cc13700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe648107480 0x7fe64819b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:59:47.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.939+0000 7fe64cc13700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64819b680 con 0x7fe648107480 2024-03-11T18:59:47.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.940+0000 7fe64659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe648107480 0x7fe64819b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:59:47.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.940+0000 7fe64659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe648107480 0x7fe64819b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52334/0 (socket says 172.21.15.29:52334) 2024-03-11T18:59:47.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.940+0000 7fe64659c700 1 -- 172.21.15.29:0/2213150157 learned_addr learned my addr 172.21.15.29:0/2213150157 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T18:59:47.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.940+0000 7fe64659c700 1 -- 172.21.15.29:0/2213150157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe638009550 con 0x7fe648107480 2024-03-11T18:59:47.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.940+0000 7fe64659c700 1 --2- 172.21.15.29:0/2213150157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe648107480 0x7fe64819b140 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe638000c00 tx=0x7fe638011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:59:47.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.940+0000 7fe6377fe700 1 -- 172.21.15.29:0/2213150157 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe638003a40 con 0x7fe648107480 2024-03-11T18:59:47.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.940+0000 7fe6377fe700 1 -- 172.21.15.29:0/2213150157 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fe638003ba0 con 0x7fe648107480 2024-03-11T18:59:47.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.940+0000 7fe64cc13700 1 -- 172.21.15.29:0/2213150157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe64819b880 con 0x7fe648107480 2024-03-11T18:59:47.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.940+0000 7fe64cc13700 1 -- 172.21.15.29:0/2213150157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe64819bd20 con 0x7fe648107480 2024-03-11T18:59:47.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.940+0000 7fe6377fe700 1 -- 172.21.15.29:0/2213150157 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe63801a700 con 0x7fe648107480 2024-03-11T18:59:47.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.941+0000 7fe6377fe700 1 -- 172.21.15.29:0/2213150157 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe638024b80 con 0x7fe648107480 2024-03-11T18:59:47.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.941+0000 7fe6377fe700 1 -- 172.21.15.29:0/2213150157 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe63802d710 con 0x7fe648107480 2024-03-11T18:59:47.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.941+0000 7fe64cc13700 1 -- 172.21.15.29:0/2213150157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6480632a0 con 0x7fe648107480 2024-03-11T18:59:47.623 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:46.945+0000 7fe6377fe700 1 -- 172.21.15.29:0/2213150157 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe638042dd0 con 0x7fe648107480 2024-03-11T18:59:47.623 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:47.019+0000 7fe64cc13700 1 -- 172.21.15.29:0/2213150157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe64819e890 con 0x7fe648107480 2024-03-11T18:59:47.623 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:47.019+0000 7fe6377fe700 1 -- 172.21.15.29:0/2213150157 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fe638031030 con 0x7fe648107480 2024-03-11T18:59:47.623 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:47.021+0000 7fe64cc13700 1 -- 172.21.15.29:0/2213150157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe648107480 msgr2=0x7fe64819b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:59:47.623 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:47.021+0000 7fe64cc13700 1 --2- 172.21.15.29:0/2213150157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe648107480 0x7fe64819b140 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe638000c00 tx=0x7fe638011ab0 comp rx=0 tx=0).stop 2024-03-11T18:59:47.623 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:47.021+0000 7fe64cc13700 1 -- 172.21.15.29:0/2213150157 shutdown_connections 2024-03-11T18:59:47.623 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:47.021+0000 7fe64cc13700 1 --2- 172.21.15.29:0/2213150157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe648107480 0x7fe64819b140 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:59:47.623 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:47.021+0000 7fe64cc13700 1 -- 172.21.15.29:0/2213150157 >> 172.21.15.29:0/2213150157 conn(0x7fe648102a80 msgr2=0x7fe648103630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:59:47.623 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:47.021+0000 7fe64cc13700 1 -- 172.21.15.29:0/2213150157 shutdown_connections 2024-03-11T18:59:47.623 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:47.021+0000 7fe64cc13700 1 -- 172.21.15.29:0/2213150157 wait complete. 2024-03-11T18:59:47.623 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (2/15)... 2024-03-11T18:59:47.897 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:47 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:47.706+0000 7f0edbd8f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T18:59:47.898 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:47 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:47.895+0000 7f0edbd8f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T18:59:48.450 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:48 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:48.185+0000 7f0edbd8f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T18:59:48.450 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:48 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:48.447+0000 7f0edbd8f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T18:59:48.905 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:48 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:48.689+0000 7f0edbd8f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T18:59:49.905 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:49 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:49.513+0000 7f0edbd8f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T18:59:50.405 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:50 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:50.096+0000 7f0edbd8f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T18:59:50.405 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:50 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:50.196+0000 7f0edbd8f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T18:59:50.905 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 18:59:50 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T18:59:50.415+0000 7f0edbd8f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T18:59:50.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:50 smithi029 ceph-mon[32439]: Activating manager daemon x 2024-03-11T18:59:50.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:50 smithi029 ceph-mon[32439]: mgrmap e2: x(active, starting, since 0.00301899s) 2024-03-11T18:59:50.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:50 smithi029 ceph-mon[32439]: from='mgr.14100 172.21.15.29:0/3800011673' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T18:59:50.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:50 smithi029 ceph-mon[32439]: from='mgr.14100 172.21.15.29:0/3800011673' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T18:59:50.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:50 smithi029 ceph-mon[32439]: from='mgr.14100 172.21.15.29:0/3800011673' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T18:59:50.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:50 smithi029 ceph-mon[32439]: from='mgr.14100 172.21.15.29:0/3800011673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:59:50.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:50 smithi029 ceph-mon[32439]: from='mgr.14100 172.21.15.29:0/3800011673' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T18:59:50.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:50 smithi029 ceph-mon[32439]: Manager daemon x is now available 2024-03-11T18:59:50.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:50 smithi029 ceph-mon[32439]: from='mgr.14100 172.21.15.29:0/3800011673' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T18:59:50.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:50 smithi029 ceph-mon[32439]: from='mgr.14100 172.21.15.29:0/3800011673' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T18:59:50.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:50 smithi029 ceph-mon[32439]: from='mgr.14100 172.21.15.29:0/3800011673' entity='mgr.x' 2024-03-11T18:59:50.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:50 smithi029 ceph-mon[32439]: from='mgr.14100 172.21.15.29:0/3800011673' entity='mgr.x' 2024-03-11T18:59:50.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:50 smithi029 ceph-mon[32439]: from='mgr.14100 172.21.15.29:0/3800011673' entity='mgr.x' 2024-03-11T18:59:52.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:52 smithi029 ceph-mon[32439]: mgrmap e3: x(active, since 1.00718s) 2024-03-11T18:59:54.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:54 smithi029 ceph-mon[32439]: mgrmap e4: x(active, since 3s) 2024-03-11T18:59:55.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 18:59:55 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3046746509' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T19:00:05.204 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-11T19:00:05.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-11T19:00:05.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "638915c2-dfd9-11ee-95c8-87774f69a715", 2024-03-11T19:00:05.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T19:00:05.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T19:00:05.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T19:00:05.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T19:00:05.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:00:05.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T19:00:05.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T19:00:05.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-03-11T19:00:05.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-11T19:00:05.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T19:00:05.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T19:00:05.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-11T19:00:05.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-03-11T19:00:05.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T19:00:05.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:00:05.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T19:00:05.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T19:00:05.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:00:05.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T19:00:05.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:00:05.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T19:00:05.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T19:00:05.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T19:00:05.214 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T19:00:05.215 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T19:00:05.215 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T19:00:05.215 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:00:05.215 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T19:00:05.215 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T19:00:05.215 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T19:00:05.215 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T19:00:05.215 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T19:00:05.215 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T19:00:05.215 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T19:00:05.215 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T19:00:05.215 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T19:00:05.215 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:00:05.215 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T19:00:05.215 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:00:05.215 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T19:00:05.216 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T19:00:05.216 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:00:05.216 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T19:00:05.216 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T19:00:05.216 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T19:00:05.216 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T19:00:05.216 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T19:00:05.216 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T19:00:05.216 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T19:00:05.216 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-11T19:00:05.216 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T19:00:05.216 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:00:05.216 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T19:00:05.216 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:00:05.216 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T18:59:36.895333+0000", 2024-03-11T19:00:05.216 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T19:00:05.216 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:00:05.217 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T19:00:05.217 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-11T19:00:05.217 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.649+0000 7f6258a3e700 1 Processor -- start 2024-03-11T19:00:05.217 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.649+0000 7f6258a3e700 1 -- start start 2024-03-11T19:00:05.217 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.649+0000 7f6258a3e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6254107480 0x7f62541078a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:05.217 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.649+0000 7f6258a3e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6254107e70 con 0x7f6254107480 2024-03-11T19:00:05.217 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.649+0000 7f625259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6254107480 0x7f62541078a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:05.217 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.649+0000 7f625259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6254107480 0x7f62541078a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52414/0 (socket says 172.21.15.29:52414) 2024-03-11T19:00:05.217 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.649+0000 7f625259c700 1 -- 172.21.15.29:0/2129303659 learned_addr learned my addr 172.21.15.29:0/2129303659 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:05.217 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.649+0000 7f625259c700 1 -- 172.21.15.29:0/2129303659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6254108630 con 0x7f6254107480 2024-03-11T19:00:05.217 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.650+0000 7f625259c700 1 --2- 172.21.15.29:0/2129303659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6254107480 0x7f62541078a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f624c009e80 tx=0x7f624c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff824e3f89f2e9a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:05.217 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.650+0000 7f625159a700 1 -- 172.21.15.29:0/2129303659 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f624c00e070 con 0x7f6254107480 2024-03-11T19:00:05.217 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.650+0000 7f625159a700 1 -- 172.21.15.29:0/2129303659 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f624c00b8c0 con 0x7f6254107480 2024-03-11T19:00:05.217 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.650+0000 7f625159a700 1 -- 172.21.15.29:0/2129303659 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f624c0044b0 con 0x7f6254107480 2024-03-11T19:00:05.218 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.650+0000 7f6258a3e700 1 -- 172.21.15.29:0/2129303659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6254107480 msgr2=0x7f62541078a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:05.218 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.650+0000 7f6258a3e700 1 --2- 172.21.15.29:0/2129303659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6254107480 0x7f62541078a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f624c009e80 tx=0x7f624c00b330 comp rx=0 tx=0).stop 2024-03-11T19:00:05.218 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.650+0000 7f6258a3e700 1 -- 172.21.15.29:0/2129303659 shutdown_connections 2024-03-11T19:00:05.218 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.650+0000 7f6258a3e700 1 --2- 172.21.15.29:0/2129303659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6254107480 0x7f62541078a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:05.218 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.650+0000 7f6258a3e700 1 -- 172.21.15.29:0/2129303659 >> 172.21.15.29:0/2129303659 conn(0x7f6254102a80 msgr2=0x7f6254104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:05.218 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.650+0000 7f6258a3e700 1 -- 172.21.15.29:0/2129303659 shutdown_connections 2024-03-11T19:00:05.218 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.650+0000 7f6258a3e700 1 -- 172.21.15.29:0/2129303659 wait complete. 2024-03-11T19:00:05.218 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.651+0000 7f6258a3e700 1 Processor -- start 2024-03-11T19:00:05.218 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.651+0000 7f6258a3e700 1 -- start start 2024-03-11T19:00:05.218 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.651+0000 7f6258a3e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6254107480 0x7f625419b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:05.218 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.651+0000 7f6258a3e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625419b560 con 0x7f6254107480 2024-03-11T19:00:05.218 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.651+0000 7f625259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6254107480 0x7f625419b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:05.218 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.651+0000 7f625259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6254107480 0x7f625419b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52428/0 (socket says 172.21.15.29:52428) 2024-03-11T19:00:05.218 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.652+0000 7f625259c700 1 -- 172.21.15.29:0/3046746509 learned_addr learned my addr 172.21.15.29:0/3046746509 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:05.219 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.652+0000 7f625259c700 1 -- 172.21.15.29:0/3046746509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f624c009b30 con 0x7f6254107480 2024-03-11T19:00:05.219 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.652+0000 7f625259c700 1 --2- 172.21.15.29:0/3046746509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6254107480 0x7f625419b020 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f624c004070 tx=0x7f624c0093c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:05.219 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.652+0000 7f624b7fe700 1 -- 172.21.15.29:0/3046746509 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f624c00e040 con 0x7f6254107480 2024-03-11T19:00:05.219 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.652+0000 7f624b7fe700 1 -- 172.21.15.29:0/3046746509 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f624c00bd70 con 0x7f6254107480 2024-03-11T19:00:05.219 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.652+0000 7f6258a3e700 1 -- 172.21.15.29:0/3046746509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f625419b760 con 0x7f6254107480 2024-03-11T19:00:05.219 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.652+0000 7f624b7fe700 1 -- 172.21.15.29:0/3046746509 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f624c01c7a0 con 0x7f6254107480 2024-03-11T19:00:05.219 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.652+0000 7f6258a3e700 1 -- 172.21.15.29:0/3046746509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f625419bc60 con 0x7f6254107480 2024-03-11T19:00:05.219 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.653+0000 7f624b7fe700 1 -- 172.21.15.29:0/3046746509 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) v1 ==== 45758+0+0 (secure 0 0 0) 0x7f624c01c9e0 con 0x7f6254107480 2024-03-11T19:00:05.219 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.653+0000 7f624b7fe700 1 --2- 172.21.15.29:0/3046746509 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f6240038c10 0x7f624003b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:05.219 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.653+0000 7f624b7fe700 1 -- 172.21.15.29:0/3046746509 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f624c04cd40 con 0x7f6254107480 2024-03-11T19:00:05.219 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.653+0000 7f6251d9b700 1 --2- 172.21.15.29:0/3046746509 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f6240038c10 0x7f624003b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:05.219 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.653+0000 7f6258a3e700 1 -- 172.21.15.29:0/3046746509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f625410b860 con 0x7f6254107480 2024-03-11T19:00:05.219 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.654+0000 7f6251d9b700 1 --2- 172.21.15.29:0/3046746509 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f6240038c10 0x7f624003b0d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f623c006fd0 tx=0x7f623c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:05.219 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.658+0000 7f624b7fe700 1 -- 172.21.15.29:0/3046746509 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f624c017020 con 0x7f6254107480 2024-03-11T19:00:05.219 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.958+0000 7f6258a3e700 1 -- 172.21.15.29:0/3046746509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f625419e760 con 0x7f6254107480 2024-03-11T19:00:05.220 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.959+0000 7f624b7fe700 1 -- 172.21.15.29:0/3046746509 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f624c020030 con 0x7f6254107480 2024-03-11T19:00:05.220 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.961+0000 7f6258a3e700 1 -- 172.21.15.29:0/3046746509 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f6240038c10 msgr2=0x7f624003b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:05.220 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.961+0000 7f6258a3e700 1 --2- 172.21.15.29:0/3046746509 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f6240038c10 0x7f624003b0d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f623c006fd0 tx=0x7f623c006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:05.220 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.961+0000 7f6258a3e700 1 -- 172.21.15.29:0/3046746509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6254107480 msgr2=0x7f625419b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:05.220 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.961+0000 7f6258a3e700 1 --2- 172.21.15.29:0/3046746509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6254107480 0x7f625419b020 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f624c004070 tx=0x7f624c0093c0 comp rx=0 tx=0).stop 2024-03-11T19:00:05.220 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.961+0000 7f6258a3e700 1 -- 172.21.15.29:0/3046746509 shutdown_connections 2024-03-11T19:00:05.220 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.961+0000 7f6258a3e700 1 --2- 172.21.15.29:0/3046746509 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f6240038c10 0x7f624003b0d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:05.220 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.961+0000 7f6258a3e700 1 --2- 172.21.15.29:0/3046746509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6254107480 0x7f625419b020 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:05.220 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.961+0000 7f6258a3e700 1 -- 172.21.15.29:0/3046746509 >> 172.21.15.29:0/3046746509 conn(0x7f6254102a80 msgr2=0x7f6254103520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:05.220 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.961+0000 7f6258a3e700 1 -- 172.21.15.29:0/3046746509 shutdown_connections 2024-03-11T19:00:05.220 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T18:59:54.961+0000 7f6258a3e700 1 -- 172.21.15.29:0/3046746509 wait complete. 2024-03-11T19:00:05.220 INFO:teuthology.orchestra.run.smithi029.stdout:mgr is available 2024-03-11T19:00:07.602 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:07 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3408875217' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T19:00:07.720 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-11T19:00:07.720 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T19:00:07.720 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout fsid = 638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T19:00:07.720 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T19:00:07.720 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T19:00:07.720 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T19:00:07.720 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T19:00:07.720 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-11T19:00:07.720 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T19:00:07.720 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T19:00:07.720 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-11T19:00:07.721 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T19:00:07.721 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-11T19:00:07.721 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-11T19:00:07.721 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-11T19:00:07.721 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T19:00:07.721 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T19:00:07.721 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.902+0000 7f4600e87700 1 Processor -- start 2024-03-11T19:00:07.721 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.902+0000 7f4600e87700 1 -- start start 2024-03-11T19:00:07.721 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.903+0000 7f4600e87700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45fc105cd0 0x7f45fc1060f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:07.721 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.903+0000 7f4600e87700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc07ae90 con 0x7f45fc105cd0 2024-03-11T19:00:07.721 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.903+0000 7f45fa59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45fc105cd0 0x7f45fc1060f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:07.721 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.903+0000 7f45fa59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45fc105cd0 0x7f45fc1060f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43704/0 (socket says 172.21.15.29:43704) 2024-03-11T19:00:07.721 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.903+0000 7f45fa59c700 1 -- 172.21.15.29:0/3212804813 learned_addr learned my addr 172.21.15.29:0/3212804813 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:07.722 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.903+0000 7f45fa59c700 1 -- 172.21.15.29:0/3212804813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45fc106630 con 0x7f45fc105cd0 2024-03-11T19:00:07.722 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.903+0000 7f45fa59c700 1 --2- 172.21.15.29:0/3212804813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45fc105cd0 0x7f45fc1060f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f45e40098a0 tx=0x7f45e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0e714f06286e843 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:07.722 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.904+0000 7f45f959a700 1 -- 172.21.15.29:0/3212804813 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45e4004160 con 0x7f45fc105cd0 2024-03-11T19:00:07.722 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.904+0000 7f45f959a700 1 -- 172.21.15.29:0/3212804813 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f45e40042c0 con 0x7f45fc105cd0 2024-03-11T19:00:07.722 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.904+0000 7f45f959a700 1 -- 172.21.15.29:0/3212804813 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45e4011a50 con 0x7f45fc105cd0 2024-03-11T19:00:07.722 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.904+0000 7f4600e87700 1 -- 172.21.15.29:0/3212804813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45fc105cd0 msgr2=0x7f45fc1060f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:07.722 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.904+0000 7f4600e87700 1 --2- 172.21.15.29:0/3212804813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45fc105cd0 0x7f45fc1060f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f45e40098a0 tx=0x7f45e4009bb0 comp rx=0 tx=0).stop 2024-03-11T19:00:07.722 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.904+0000 7f4600e87700 1 -- 172.21.15.29:0/3212804813 shutdown_connections 2024-03-11T19:00:07.722 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.904+0000 7f4600e87700 1 --2- 172.21.15.29:0/3212804813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45fc105cd0 0x7f45fc1060f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:07.722 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.904+0000 7f4600e87700 1 -- 172.21.15.29:0/3212804813 >> 172.21.15.29:0/3212804813 conn(0x7f45fc101930 msgr2=0x7f45fc103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:07.722 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.904+0000 7f4600e87700 1 -- 172.21.15.29:0/3212804813 shutdown_connections 2024-03-11T19:00:07.722 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.904+0000 7f4600e87700 1 -- 172.21.15.29:0/3212804813 wait complete. 2024-03-11T19:00:07.722 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.905+0000 7f4600e87700 1 Processor -- start 2024-03-11T19:00:07.723 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.905+0000 7f4600e87700 1 -- start start 2024-03-11T19:00:07.723 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.905+0000 7f4600e87700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45fc105cd0 0x7f45fc19d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:07.723 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.905+0000 7f4600e87700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc19d870 con 0x7f45fc105cd0 2024-03-11T19:00:07.723 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.906+0000 7f45fa59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45fc105cd0 0x7f45fc19d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:07.723 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.906+0000 7f45fa59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45fc105cd0 0x7f45fc19d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43710/0 (socket says 172.21.15.29:43710) 2024-03-11T19:00:07.723 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.906+0000 7f45fa59c700 1 -- 172.21.15.29:0/3408875217 learned_addr learned my addr 172.21.15.29:0/3408875217 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:07.723 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.906+0000 7f45fa59c700 1 -- 172.21.15.29:0/3408875217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e4009550 con 0x7f45fc105cd0 2024-03-11T19:00:07.723 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.906+0000 7f45fa59c700 1 --2- 172.21.15.29:0/3408875217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45fc105cd0 0x7f45fc19d330 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f45e4011a00 tx=0x7f45e4011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:07.723 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.906+0000 7f45f37fe700 1 -- 172.21.15.29:0/3408875217 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45e4003a40 con 0x7f45fc105cd0 2024-03-11T19:00:07.723 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.906+0000 7f45f37fe700 1 -- 172.21.15.29:0/3408875217 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f45e4003ba0 con 0x7f45fc105cd0 2024-03-11T19:00:07.723 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.906+0000 7f45f37fe700 1 -- 172.21.15.29:0/3408875217 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45e401a6e0 con 0x7f45fc105cd0 2024-03-11T19:00:07.723 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.906+0000 7f4600e87700 1 -- 172.21.15.29:0/3408875217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45fc19da70 con 0x7f45fc105cd0 2024-03-11T19:00:07.723 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.906+0000 7f4600e87700 1 -- 172.21.15.29:0/3408875217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45fc19df10 con 0x7f45fc105cd0 2024-03-11T19:00:07.723 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.907+0000 7f45f37fe700 1 -- 172.21.15.29:0/3408875217 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) v1 ==== 45758+0+0 (secure 0 0 0) 0x7f45e401a9a0 con 0x7f45fc105cd0 2024-03-11T19:00:07.723 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.907+0000 7f45f37fe700 1 --2- 172.21.15.29:0/3408875217 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f45e8038cb0 0x7f45e803b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:07.723 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.907+0000 7f45f37fe700 1 -- 172.21.15.29:0/3408875217 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f45e404ca90 con 0x7f45fc105cd0 2024-03-11T19:00:07.724 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.907+0000 7f45f9d9b700 1 --2- 172.21.15.29:0/3408875217 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f45e8038cb0 0x7f45e803b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:07.724 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.907+0000 7f4600e87700 1 -- 172.21.15.29:0/3408875217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45fc197000 con 0x7f45fc105cd0 2024-03-11T19:00:07.724 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.908+0000 7f45f9d9b700 1 --2- 172.21.15.29:0/3408875217 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f45e8038cb0 0x7f45e803b170 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f45ec006fd0 tx=0x7f45ec006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:07.724 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:06.913+0000 7f45f37fe700 1 -- 172.21.15.29:0/3408875217 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f45e401ae30 con 0x7f45fc105cd0 2024-03-11T19:00:07.724 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:07.114+0000 7f4600e87700 1 -- 172.21.15.29:0/3408875217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f45fc107430 con 0x7f45fc105cd0 2024-03-11T19:00:07.724 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:07.115+0000 7f45f37fe700 1 -- 172.21.15.29:0/3408875217 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f45e4018350 con 0x7f45fc105cd0 2024-03-11T19:00:07.724 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:07.117+0000 7f4600e87700 1 -- 172.21.15.29:0/3408875217 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f45e8038cb0 msgr2=0x7f45e803b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:07.724 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:07.117+0000 7f4600e87700 1 --2- 172.21.15.29:0/3408875217 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f45e8038cb0 0x7f45e803b170 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f45ec006fd0 tx=0x7f45ec006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:07.724 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:07.117+0000 7f4600e87700 1 -- 172.21.15.29:0/3408875217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45fc105cd0 msgr2=0x7f45fc19d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:07.724 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:07.117+0000 7f4600e87700 1 --2- 172.21.15.29:0/3408875217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45fc105cd0 0x7f45fc19d330 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f45e4011a00 tx=0x7f45e4011ae0 comp rx=0 tx=0).stop 2024-03-11T19:00:07.724 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:07.117+0000 7f4600e87700 1 -- 172.21.15.29:0/3408875217 shutdown_connections 2024-03-11T19:00:07.724 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:07.117+0000 7f4600e87700 1 --2- 172.21.15.29:0/3408875217 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f45e8038cb0 0x7f45e803b170 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:07.724 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:07.117+0000 7f4600e87700 1 --2- 172.21.15.29:0/3408875217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45fc105cd0 0x7f45fc19d330 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:07.724 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:07.117+0000 7f4600e87700 1 -- 172.21.15.29:0/3408875217 >> 172.21.15.29:0/3408875217 conn(0x7f45fc101930 msgr2=0x7f45fc194140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:07.724 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:07.117+0000 7f4600e87700 1 -- 172.21.15.29:0/3408875217 shutdown_connections 2024-03-11T19:00:07.724 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:07.117+0000 7f4600e87700 1 -- 172.21.15.29:0/3408875217 wait complete. 2024-03-11T19:00:07.725 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling cephadm module... 2024-03-11T19:00:08.722 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:08 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3035156743' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-11T19:00:08.723 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:08 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: ignoring --setuser ceph since I am not root 2024-03-11T19:00:08.723 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:08 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: ignoring --setgroup ceph since I am not root 2024-03-11T19:00:08.982 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.137+0000 7f8f6dc59700 1 Processor -- start 2024-03-11T19:00:08.982 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.137+0000 7f8f6dc59700 1 -- start start 2024-03-11T19:00:08.982 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.137+0000 7f8f6dc59700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f68107f20 0x7f8f6810a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:08.982 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.137+0000 7f8f6dc59700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6807ae30 con 0x7f8f68107f20 2024-03-11T19:00:08.982 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.137+0000 7f8f677fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f68107f20 0x7f8f6810a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:08.982 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.137+0000 7f8f677fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f68107f20 0x7f8f6810a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43712/0 (socket says 172.21.15.29:43712) 2024-03-11T19:00:08.982 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.138+0000 7f8f677fe700 1 -- 172.21.15.29:0/2423238925 learned_addr learned my addr 172.21.15.29:0/2423238925 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:08.982 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.138+0000 7f8f677fe700 1 -- 172.21.15.29:0/2423238925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6810a850 con 0x7f8f68107f20 2024-03-11T19:00:08.983 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.138+0000 7f8f677fe700 1 --2- 172.21.15.29:0/2423238925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f68107f20 0x7f8f6810a310 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8f580098a0 tx=0x7f8f58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d50bab608e2e365 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:08.983 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.138+0000 7f8f667fc700 1 -- 172.21.15.29:0/2423238925 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f58004160 con 0x7f8f68107f20 2024-03-11T19:00:08.983 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.138+0000 7f8f667fc700 1 -- 172.21.15.29:0/2423238925 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f8f580042c0 con 0x7f8f68107f20 2024-03-11T19:00:08.983 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.138+0000 7f8f667fc700 1 -- 172.21.15.29:0/2423238925 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f58011a50 con 0x7f8f68107f20 2024-03-11T19:00:08.983 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.139+0000 7f8f6dc59700 1 -- 172.21.15.29:0/2423238925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f68107f20 msgr2=0x7f8f6810a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:08.983 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.139+0000 7f8f6dc59700 1 --2- 172.21.15.29:0/2423238925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f68107f20 0x7f8f6810a310 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8f580098a0 tx=0x7f8f58009bb0 comp rx=0 tx=0).stop 2024-03-11T19:00:08.983 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.139+0000 7f8f6dc59700 1 -- 172.21.15.29:0/2423238925 shutdown_connections 2024-03-11T19:00:08.983 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.139+0000 7f8f6dc59700 1 --2- 172.21.15.29:0/2423238925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f68107f20 0x7f8f6810a310 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:08.983 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.139+0000 7f8f6dc59700 1 -- 172.21.15.29:0/2423238925 >> 172.21.15.29:0/2423238925 conn(0x7f8f68101930 msgr2=0x7f8f68103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:08.983 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.139+0000 7f8f6dc59700 1 -- 172.21.15.29:0/2423238925 shutdown_connections 2024-03-11T19:00:08.983 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.139+0000 7f8f6dc59700 1 -- 172.21.15.29:0/2423238925 wait complete. 2024-03-11T19:00:08.983 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.139+0000 7f8f6dc59700 1 Processor -- start 2024-03-11T19:00:08.983 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.140+0000 7f8f6dc59700 1 -- start start 2024-03-11T19:00:08.983 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.140+0000 7f8f6dc59700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f68107f20 0x7f8f6819b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:08.984 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.140+0000 7f8f6dc59700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6819b670 con 0x7f8f68107f20 2024-03-11T19:00:08.984 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.140+0000 7f8f677fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f68107f20 0x7f8f6819b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:08.984 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.140+0000 7f8f677fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f68107f20 0x7f8f6819b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43718/0 (socket says 172.21.15.29:43718) 2024-03-11T19:00:08.984 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.140+0000 7f8f677fe700 1 -- 172.21.15.29:0/3035156743 learned_addr learned my addr 172.21.15.29:0/3035156743 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:08.984 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.140+0000 7f8f677fe700 1 -- 172.21.15.29:0/3035156743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f58009550 con 0x7f8f68107f20 2024-03-11T19:00:08.984 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.140+0000 7f8f677fe700 1 --2- 172.21.15.29:0/3035156743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f68107f20 0x7f8f6819b130 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8f58011a00 tx=0x7f8f58011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:08.984 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.141+0000 7f8f64ff9700 1 -- 172.21.15.29:0/3035156743 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f58003a40 con 0x7f8f68107f20 2024-03-11T19:00:08.984 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.141+0000 7f8f64ff9700 1 -- 172.21.15.29:0/3035156743 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f8f58003ba0 con 0x7f8f68107f20 2024-03-11T19:00:08.984 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.141+0000 7f8f64ff9700 1 -- 172.21.15.29:0/3035156743 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f5801a6e0 con 0x7f8f68107f20 2024-03-11T19:00:08.984 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.141+0000 7f8f6dc59700 1 -- 172.21.15.29:0/3035156743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f6819b870 con 0x7f8f68107f20 2024-03-11T19:00:08.984 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.141+0000 7f8f6dc59700 1 -- 172.21.15.29:0/3035156743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6819bd10 con 0x7f8f68107f20 2024-03-11T19:00:08.984 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.141+0000 7f8f64ff9700 1 -- 172.21.15.29:0/3035156743 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) v1 ==== 45758+0+0 (secure 0 0 0) 0x7f8f5801a950 con 0x7f8f68107f20 2024-03-11T19:00:08.984 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.142+0000 7f8f64ff9700 1 --2- 172.21.15.29:0/3035156743 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f8f50038cb0 0x7f8f5003b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:08.984 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.142+0000 7f8f64ff9700 1 -- 172.21.15.29:0/3035156743 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8f5804ca30 con 0x7f8f68107f20 2024-03-11T19:00:08.985 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.142+0000 7f8f6dc59700 1 -- 172.21.15.29:0/3035156743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f6810b580 con 0x7f8f68107f20 2024-03-11T19:00:08.985 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.142+0000 7f8f66ffd700 1 --2- 172.21.15.29:0/3035156743 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f8f50038cb0 0x7f8f5003b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:08.985 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.142+0000 7f8f66ffd700 1 --2- 172.21.15.29:0/3035156743 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f8f50038cb0 0x7f8f5003b170 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c006fd0 tx=0x7f8f5c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:08.985 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.147+0000 7f8f64ff9700 1 -- 172.21.15.29:0/3035156743 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8f5801ac00 con 0x7f8f68107f20 2024-03-11T19:00:08.985 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.409+0000 7f8f6dc59700 1 -- 172.21.15.29:0/3035156743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f8f6819e8f0 con 0x7f8f68107f20 2024-03-11T19:00:08.985 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.460+0000 7f8f64ff9700 1 -- 172.21.15.29:0/3035156743 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f8f58018350 con 0x7f8f68107f20 2024-03-11T19:00:08.985 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.462+0000 7f8f64ff9700 1 -- 172.21.15.29:0/3035156743 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 5) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f8f5802d910 con 0x7f8f68107f20 2024-03-11T19:00:08.985 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.463+0000 7f8f6dc59700 1 -- 172.21.15.29:0/3035156743 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f8f50038cb0 msgr2=0x7f8f5003b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:08.985 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.463+0000 7f8f6dc59700 1 --2- 172.21.15.29:0/3035156743 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f8f50038cb0 0x7f8f5003b170 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c006fd0 tx=0x7f8f5c006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:08.985 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.463+0000 7f8f6dc59700 1 -- 172.21.15.29:0/3035156743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f68107f20 msgr2=0x7f8f6819b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:08.985 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.463+0000 7f8f6dc59700 1 --2- 172.21.15.29:0/3035156743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f68107f20 0x7f8f6819b130 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8f58011a00 tx=0x7f8f58011ae0 comp rx=0 tx=0).stop 2024-03-11T19:00:08.985 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.463+0000 7f8f6dc59700 1 -- 172.21.15.29:0/3035156743 shutdown_connections 2024-03-11T19:00:08.985 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.463+0000 7f8f6dc59700 1 --2- 172.21.15.29:0/3035156743 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f8f50038cb0 0x7f8f5003b170 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:08.985 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.463+0000 7f8f6dc59700 1 --2- 172.21.15.29:0/3035156743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f68107f20 0x7f8f6819b130 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:08.986 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.463+0000 7f8f6dc59700 1 -- 172.21.15.29:0/3035156743 >> 172.21.15.29:0/3035156743 conn(0x7f8f68101930 msgr2=0x7f8f68102460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:08.986 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.463+0000 7f8f6dc59700 1 -- 172.21.15.29:0/3035156743 shutdown_connections 2024-03-11T19:00:08.986 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:08.463+0000 7f8f6dc59700 1 -- 172.21.15.29:0/3035156743 wait complete. 2024-03-11T19:00:09.278 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:09 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:09.048+0000 7fcd9e2a9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T19:00:09.278 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:09 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:09.145+0000 7fcd9e2a9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T19:00:09.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:09 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3035156743' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-11T19:00:09.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:09 smithi029 ceph-mon[32439]: mgrmap e5: x(active, since 18s) 2024-03-11T19:00:09.905 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:09 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:09.822+0000 7fcd9e2a9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T19:00:10.162 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-11T19:00:10.162 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-11T19:00:10.162 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T19:00:10.163 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-11T19:00:10.163 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T19:00:10.163 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-11T19:00:10.163 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.375+0000 7f972a4f8700 1 Processor -- start 2024-03-11T19:00:10.163 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.375+0000 7f972a4f8700 1 -- start start 2024-03-11T19:00:10.163 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.375+0000 7f972a4f8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f972407ce00 0x7f972407b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:10.163 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.375+0000 7f972a4f8700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972407b840 con 0x7f972407ce00 2024-03-11T19:00:10.163 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.376+0000 7f9723fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f972407ce00 0x7f972407b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:10.163 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.376+0000 7f9723fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f972407ce00 0x7f972407b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43754/0 (socket says 172.21.15.29:43754) 2024-03-11T19:00:10.163 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.376+0000 7f9723fff700 1 -- 172.21.15.29:0/221894176 learned_addr learned my addr 172.21.15.29:0/221894176 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:10.163 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.376+0000 7f9723fff700 1 -- 172.21.15.29:0/221894176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972407b980 con 0x7f972407ce00 2024-03-11T19:00:10.163 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.376+0000 7f9723fff700 1 --2- 172.21.15.29:0/221894176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f972407ce00 0x7f972407b300 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f970c0098a0 tx=0x7f970c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=572fcf4aaae6acb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:10.163 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.376+0000 7f9722ffd700 1 -- 172.21.15.29:0/221894176 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f970c004160 con 0x7f972407ce00 2024-03-11T19:00:10.163 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.376+0000 7f9722ffd700 1 -- 172.21.15.29:0/221894176 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f970c0042c0 con 0x7f972407ce00 2024-03-11T19:00:10.164 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.376+0000 7f9722ffd700 1 -- 172.21.15.29:0/221894176 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f970c011a50 con 0x7f972407ce00 2024-03-11T19:00:10.164 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.377+0000 7f972a4f8700 1 -- 172.21.15.29:0/221894176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f972407ce00 msgr2=0x7f972407b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:10.164 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.377+0000 7f972a4f8700 1 --2- 172.21.15.29:0/221894176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f972407ce00 0x7f972407b300 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f970c0098a0 tx=0x7f970c009bb0 comp rx=0 tx=0).stop 2024-03-11T19:00:10.164 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.377+0000 7f972a4f8700 1 -- 172.21.15.29:0/221894176 shutdown_connections 2024-03-11T19:00:10.164 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.377+0000 7f972a4f8700 1 --2- 172.21.15.29:0/221894176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f972407ce00 0x7f972407b300 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:10.164 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.377+0000 7f972a4f8700 1 -- 172.21.15.29:0/221894176 >> 172.21.15.29:0/221894176 conn(0x7f9724102a50 msgr2=0x7f9724104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:10.164 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.377+0000 7f972a4f8700 1 -- 172.21.15.29:0/221894176 shutdown_connections 2024-03-11T19:00:10.164 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.377+0000 7f972a4f8700 1 -- 172.21.15.29:0/221894176 wait complete. 2024-03-11T19:00:10.164 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.377+0000 7f972a4f8700 1 Processor -- start 2024-03-11T19:00:10.164 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.377+0000 7f972a4f8700 1 -- start start 2024-03-11T19:00:10.164 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.378+0000 7f972a4f8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f972407ce00 0x7f972419d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:10.164 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.378+0000 7f972a4f8700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972419d870 con 0x7f972407ce00 2024-03-11T19:00:10.164 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.378+0000 7f9723fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f972407ce00 0x7f972419d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:10.164 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.378+0000 7f9723fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f972407ce00 0x7f972419d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43764/0 (socket says 172.21.15.29:43764) 2024-03-11T19:00:10.165 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.378+0000 7f9723fff700 1 -- 172.21.15.29:0/1179469994 learned_addr learned my addr 172.21.15.29:0/1179469994 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:10.165 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.378+0000 7f9723fff700 1 -- 172.21.15.29:0/1179469994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970c009550 con 0x7f972407ce00 2024-03-11T19:00:10.165 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.378+0000 7f9723fff700 1 --2- 172.21.15.29:0/1179469994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f972407ce00 0x7f972419d330 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f970c011710 tx=0x7f970c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:10.165 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.378+0000 7f97217fa700 1 -- 172.21.15.29:0/1179469994 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f970c003af0 con 0x7f972407ce00 2024-03-11T19:00:10.165 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.379+0000 7f97217fa700 1 -- 172.21.15.29:0/1179469994 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f970c003c50 con 0x7f972407ce00 2024-03-11T19:00:10.165 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.379+0000 7f97217fa700 1 -- 172.21.15.29:0/1179469994 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f970c0197a0 con 0x7f972407ce00 2024-03-11T19:00:10.165 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.379+0000 7f972a4f8700 1 -- 172.21.15.29:0/1179469994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f972419da70 con 0x7f972407ce00 2024-03-11T19:00:10.165 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.379+0000 7f972a4f8700 1 -- 172.21.15.29:0/1179469994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f972419df10 con 0x7f972407ce00 2024-03-11T19:00:10.165 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.379+0000 7f97217fa700 1 -- 172.21.15.29:0/1179469994 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f970c01ac30 con 0x7f972407ce00 2024-03-11T19:00:10.165 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.380+0000 7f972a4f8700 1 -- 172.21.15.29:0/1179469994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f972410d740 con 0x7f972407ce00 2024-03-11T19:00:10.165 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.380+0000 7f97217fa700 1 --2- 172.21.15.29:0/1179469994 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f9710038cb0 0x7f971003b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:10.165 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.380+0000 7f97237fe700 1 -- 172.21.15.29:0/1179469994 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f9710038cb0 msgr2=0x7f971003b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3550575954 2024-03-11T19:00:10.165 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.380+0000 7f97217fa700 1 -- 172.21.15.29:0/1179469994 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f970c04cc50 con 0x7f972407ce00 2024-03-11T19:00:10.165 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.380+0000 7f97237fe700 1 --2- 172.21.15.29:0/1179469994 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f9710038cb0 0x7f971003b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T19:00:10.165 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.385+0000 7f97217fa700 1 -- 172.21.15.29:0/1179469994 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f970c01a430 con 0x7f972407ce00 2024-03-11T19:00:10.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.581+0000 7f97237fe700 1 -- 172.21.15.29:0/1179469994 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f9710038cb0 msgr2=0x7f971003b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3550575954 2024-03-11T19:00:10.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.581+0000 7f97237fe700 1 --2- 172.21.15.29:0/1179469994 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f9710038cb0 0x7f971003b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T19:00:10.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.643+0000 7f972a4f8700 1 -- 172.21.15.29:0/1179469994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f97241a0c20 con 0x7f972407ce00 2024-03-11T19:00:10.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.643+0000 7f97217fa700 1 -- 172.21.15.29:0/1179469994 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f970c021b70 con 0x7f972407ce00 2024-03-11T19:00:10.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.645+0000 7f972a4f8700 1 -- 172.21.15.29:0/1179469994 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f9710038cb0 msgr2=0x7f971003b170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:00:10.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.645+0000 7f972a4f8700 1 --2- 172.21.15.29:0/1179469994 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f9710038cb0 0x7f971003b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:10.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.645+0000 7f972a4f8700 1 -- 172.21.15.29:0/1179469994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f972407ce00 msgr2=0x7f972419d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:10.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.645+0000 7f972a4f8700 1 --2- 172.21.15.29:0/1179469994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f972407ce00 0x7f972419d330 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f970c011710 tx=0x7f970c011ab0 comp rx=0 tx=0).stop 2024-03-11T19:00:10.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.646+0000 7f972a4f8700 1 -- 172.21.15.29:0/1179469994 shutdown_connections 2024-03-11T19:00:10.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.646+0000 7f972a4f8700 1 --2- 172.21.15.29:0/1179469994 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f9710038cb0 0x7f971003b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:10.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.646+0000 7f972a4f8700 1 --2- 172.21.15.29:0/1179469994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f972407ce00 0x7f972419d330 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:10.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.646+0000 7f972a4f8700 1 -- 172.21.15.29:0/1179469994 >> 172.21.15.29:0/1179469994 conn(0x7f9724102a50 msgr2=0x7f97241035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:10.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.646+0000 7f972a4f8700 1 -- 172.21.15.29:0/1179469994 shutdown_connections 2024-03-11T19:00:10.166 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:09.646+0000 7f972a4f8700 1 -- 172.21.15.29:0/1179469994 wait complete. 2024-03-11T19:00:10.166 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for the mgr to restart... 2024-03-11T19:00:10.167 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr epoch 5... 2024-03-11T19:00:10.255 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:09 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:09.919+0000 7fcd9e2a9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T19:00:10.255 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:10 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:10.064+0000 7fcd9e2a9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T19:00:10.256 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:10 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:10.165+0000 7fcd9e2a9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T19:00:10.256 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:10 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:10.253+0000 7fcd9e2a9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T19:00:10.527 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:10 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1179469994' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T19:00:10.783 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:10 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:10.524+0000 7fcd9e2a9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T19:00:10.783 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:10 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:10.638+0000 7fcd9e2a9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T19:00:10.784 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:10 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:10.726+0000 7fcd9e2a9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T19:00:11.155 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:10 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:10.823+0000 7fcd9e2a9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T19:00:11.155 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:11 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:11.000+0000 7fcd9e2a9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T19:00:12.405 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:11 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:11.981+0000 7fcd9e2a9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T19:00:12.405 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:12 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:12.114+0000 7fcd9e2a9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T19:00:12.746 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:12 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:12.452+0000 7fcd9e2a9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T19:00:12.746 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:12 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:12.544+0000 7fcd9e2a9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T19:00:12.746 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:12 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:12.743+0000 7fcd9e2a9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:00:13.155 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:12 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:12.931+0000 7fcd9e2a9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T19:00:13.482 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:13 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:13.220+0000 7fcd9e2a9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:00:13.483 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:13 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:13.480+0000 7fcd9e2a9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T19:00:13.905 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:13 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:13.720+0000 7fcd9e2a9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T19:00:14.905 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:14 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:14.539+0000 7fcd9e2a9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T19:00:15.405 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:15 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:15.117+0000 7fcd9e2a9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T19:00:15.406 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:15 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:15.217+0000 7fcd9e2a9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T19:00:15.905 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:15 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:15.435+0000 7fcd9e2a9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T19:00:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:15 smithi029 ceph-mon[32439]: Active manager daemon x restarted 2024-03-11T19:00:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:15 smithi029 ceph-mon[32439]: Activating manager daemon x 2024-03-11T19:00:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:15 smithi029 ceph-mon[32439]: osdmap e2: 0 total, 0 up, 0 in 2024-03-11T19:00:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:15 smithi029 ceph-mon[32439]: mgrmap e6: x(active, starting, since 0.00343899s) 2024-03-11T19:00:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:15 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:00:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:15 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T19:00:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:15 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T19:00:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:15 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T19:00:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:15 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T19:00:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:15 smithi029 ceph-mon[32439]: Manager daemon x is now available 2024-03-11T19:00:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:15 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' 2024-03-11T19:00:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:15 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' 2024-03-11T19:00:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:15 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:00:15.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:15 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:00:16.776 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:16 smithi029 ceph-mon[32439]: Found migration_current of "None". Setting to last migration. 2024-03-11T19:00:16.777 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:16 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:00:16.777 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:16 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T19:00:16.777 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:16 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T19:00:16.777 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:16 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' 2024-03-11T19:00:16.777 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:16 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' 2024-03-11T19:00:16.777 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:16 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:00:16.777 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:16 smithi029 ceph-mon[32439]: mgrmap e7: x(active, since 1.00621s) 2024-03-11T19:00:17.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-11T19:00:17.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-11T19:00:17.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T19:00:17.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-11T19:00:17.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.575+0000 7f6839b3c700 1 Processor -- start 2024-03-11T19:00:17.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.575+0000 7f6839b3c700 1 -- start start 2024-03-11T19:00:17.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.575+0000 7f6839b3c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6834107eb0 0x7f683410a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:17.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.575+0000 7f6839b3c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683407ae90 con 0x7f6834107eb0 2024-03-11T19:00:17.056 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.575+0000 7f68337fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6834107eb0 0x7f683410a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:17.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.575+0000 7f68337fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6834107eb0 0x7f683410a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43768/0 (socket says 172.21.15.29:43768) 2024-03-11T19:00:17.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.575+0000 7f68337fe700 1 -- 172.21.15.29:0/3845856269 learned_addr learned my addr 172.21.15.29:0/3845856269 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:17.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.576+0000 7f68337fe700 1 -- 172.21.15.29:0/3845856269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f683410a7e0 con 0x7f6834107eb0 2024-03-11T19:00:17.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.576+0000 7f68337fe700 1 --2- 172.21.15.29:0/3845856269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6834107eb0 0x7f683410a2a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f68240098a0 tx=0x7f6824009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3def088e387a2785 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:17.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.576+0000 7f68327fc700 1 -- 172.21.15.29:0/3845856269 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6824004160 con 0x7f6834107eb0 2024-03-11T19:00:17.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.576+0000 7f68327fc700 1 -- 172.21.15.29:0/3845856269 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f68240042c0 con 0x7f6834107eb0 2024-03-11T19:00:17.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.576+0000 7f68327fc700 1 -- 172.21.15.29:0/3845856269 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6824011a50 con 0x7f6834107eb0 2024-03-11T19:00:17.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.577+0000 7f6839b3c700 1 -- 172.21.15.29:0/3845856269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6834107eb0 msgr2=0x7f683410a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:17.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.577+0000 7f6839b3c700 1 --2- 172.21.15.29:0/3845856269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6834107eb0 0x7f683410a2a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f68240098a0 tx=0x7f6824009bb0 comp rx=0 tx=0).stop 2024-03-11T19:00:17.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.577+0000 7f6839b3c700 1 -- 172.21.15.29:0/3845856269 shutdown_connections 2024-03-11T19:00:17.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.577+0000 7f6839b3c700 1 --2- 172.21.15.29:0/3845856269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6834107eb0 0x7f683410a2a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:17.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.577+0000 7f6839b3c700 1 -- 172.21.15.29:0/3845856269 >> 172.21.15.29:0/3845856269 conn(0x7f68341018a0 msgr2=0x7f6834103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:17.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.577+0000 7f6839b3c700 1 -- 172.21.15.29:0/3845856269 shutdown_connections 2024-03-11T19:00:17.057 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.577+0000 7f6839b3c700 1 -- 172.21.15.29:0/3845856269 wait complete. 2024-03-11T19:00:17.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.577+0000 7f6839b3c700 1 Processor -- start 2024-03-11T19:00:17.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.578+0000 7f6839b3c700 1 -- start start 2024-03-11T19:00:17.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.578+0000 7f6839b3c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6834107eb0 0x7f6834107730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:17.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.578+0000 7f6839b3c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6834107c70 con 0x7f6834107eb0 2024-03-11T19:00:17.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.578+0000 7f68337fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6834107eb0 0x7f6834107730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:17.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.578+0000 7f68337fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6834107eb0 0x7f6834107730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43778/0 (socket says 172.21.15.29:43778) 2024-03-11T19:00:17.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.578+0000 7f68337fe700 1 -- 172.21.15.29:0/2547494390 learned_addr learned my addr 172.21.15.29:0/2547494390 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:17.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.578+0000 7f68337fe700 1 -- 172.21.15.29:0/2547494390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6824009550 con 0x7f6834107eb0 2024-03-11T19:00:17.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.578+0000 7f68337fe700 1 --2- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6834107eb0 0x7f6834107730 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6824011710 tx=0x7f6824011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:17.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.578+0000 7f6830ff9700 1 -- 172.21.15.29:0/2547494390 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6824003af0 con 0x7f6834107eb0 2024-03-11T19:00:17.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.579+0000 7f6830ff9700 1 -- 172.21.15.29:0/2547494390 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f6824003c50 con 0x7f6834107eb0 2024-03-11T19:00:17.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.579+0000 7f6830ff9700 1 -- 172.21.15.29:0/2547494390 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f68240197a0 con 0x7f6834107eb0 2024-03-11T19:00:17.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.579+0000 7f6839b3c700 1 -- 172.21.15.29:0/2547494390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6834105e60 con 0x7f6834107eb0 2024-03-11T19:00:17.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.579+0000 7f6839b3c700 1 -- 172.21.15.29:0/2547494390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6834106300 con 0x7f6834107eb0 2024-03-11T19:00:17.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.580+0000 7f6830ff9700 1 -- 172.21.15.29:0/2547494390 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f682401ac30 con 0x7f6834107eb0 2024-03-11T19:00:17.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.580+0000 7f6830ff9700 1 --2- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f681c038d00 0x7f681c03b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:17.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.580+0000 7f6830ff9700 1 -- 172.21.15.29:0/2547494390 --> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f681c03b8d0 con 0x7f681c038d00 2024-03-11T19:00:17.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.580+0000 7f6830ff9700 1 -- 172.21.15.29:0/2547494390 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f682404cb70 con 0x7f6834107eb0 2024-03-11T19:00:17.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.580+0000 7f6832ffd700 1 -- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f681c038d00 msgr2=0x7f681c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3550575954 2024-03-11T19:00:17.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.580+0000 7f6832ffd700 1 --2- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f681c038d00 0x7f681c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T19:00:17.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.781+0000 7f6832ffd700 1 -- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f681c038d00 msgr2=0x7f681c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3550575954 2024-03-11T19:00:17.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:10.781+0000 7f6832ffd700 1 --2- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f681c038d00 0x7f681c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T19:00:17.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:11.182+0000 7f6832ffd700 1 -- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f681c038d00 msgr2=0x7f681c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3550575954 2024-03-11T19:00:17.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:11.182+0000 7f6832ffd700 1 --2- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f681c038d00 0x7f681c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-11T19:00:17.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:11.983+0000 7f6832ffd700 1 -- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f681c038d00 msgr2=0x7f681c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3550575954 2024-03-11T19:00:17.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:11.983+0000 7f6832ffd700 1 --2- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f681c038d00 0x7f681c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-11T19:00:17.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:13.584+0000 7f6832ffd700 1 -- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f681c038d00 msgr2=0x7f681c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3550575954 2024-03-11T19:00:17.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:13.584+0000 7f6832ffd700 1 --2- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f681c038d00 0x7f681c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-11T19:00:17.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:15.451+0000 7f6830ff9700 1 -- 172.21.15.29:0/2547494390 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mgrmap(e 6) v1 ==== 45536+0+0 (secure 0 0 0) 0x7f682402d350 con 0x7f6834107eb0 2024-03-11T19:00:17.060 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:15.451+0000 7f6830ff9700 1 -- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f681c038d00 msgr2=0x7f681c03b1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:00:17.060 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:15.451+0000 7f6830ff9700 1 --2- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f681c038d00 0x7f681c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:17.060 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:16.452+0000 7f6830ff9700 1 -- 172.21.15.29:0/2547494390 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 7) v1 ==== 45663+0+0 (secure 0 0 0) 0x7f6824019d90 con 0x7f6834107eb0 2024-03-11T19:00:17.060 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:16.453+0000 7f6830ff9700 1 --2- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f681c03c5f0 0x7f681c03e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:17.060 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:16.453+0000 7f6830ff9700 1 -- 172.21.15.29:0/2547494390 --> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f681c03b8d0 con 0x7f681c03c5f0 2024-03-11T19:00:17.060 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:16.453+0000 7f6832ffd700 1 --2- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f681c03c5f0 0x7f681c03e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:17.060 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:16.454+0000 7f6832ffd700 1 --2- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f681c03c5f0 0x7f681c03e9e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6828003a10 tx=0x7f68280092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:17.060 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:16.455+0000 7f6830ff9700 1 -- 172.21.15.29:0/2547494390 <== mgr.14118 v2:172.21.15.29:6800/3585665950 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f681c03b8d0 con 0x7f681c03c5f0 2024-03-11T19:00:17.060 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:16.461+0000 7f6839b3c700 1 -- 172.21.15.29:0/2547494390 --> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f68340632a0 con 0x7f681c03c5f0 2024-03-11T19:00:17.060 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:16.461+0000 7f6830ff9700 1 -- 172.21.15.29:0/2547494390 <== mgr.14118 v2:172.21.15.29:6800/3585665950 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f68340632a0 con 0x7f681c03c5f0 2024-03-11T19:00:17.060 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:16.462+0000 7f6839b3c700 1 -- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f681c03c5f0 msgr2=0x7f681c03e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:17.060 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:16.462+0000 7f6839b3c700 1 --2- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f681c03c5f0 0x7f681c03e9e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6828003a10 tx=0x7f68280092b0 comp rx=0 tx=0).stop 2024-03-11T19:00:17.060 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:16.462+0000 7f6839b3c700 1 -- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6834107eb0 msgr2=0x7f6834107730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:17.060 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:16.462+0000 7f6839b3c700 1 --2- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6834107eb0 0x7f6834107730 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6824011710 tx=0x7f6824011ab0 comp rx=0 tx=0).stop 2024-03-11T19:00:17.060 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:16.462+0000 7f6839b3c700 1 -- 172.21.15.29:0/2547494390 shutdown_connections 2024-03-11T19:00:17.061 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:16.462+0000 7f6839b3c700 1 --2- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f681c03c5f0 0x7f681c03e9e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:17.061 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:16.462+0000 7f6839b3c700 1 --2- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6834107eb0 0x7f6834107730 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:17.061 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:16.462+0000 7f6839b3c700 1 --2- 172.21.15.29:0/2547494390 >> [v2:172.21.15.29:6800/3550575954,v1:172.21.15.29:6801/3550575954] conn(0x7f681c038d00 0x7f681c03b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:17.061 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:16.462+0000 7f6839b3c700 1 -- 172.21.15.29:0/2547494390 >> 172.21.15.29:0/2547494390 conn(0x7f68341018a0 msgr2=0x7f68341024e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:17.061 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:16.462+0000 7f6839b3c700 1 -- 172.21.15.29:0/2547494390 shutdown_connections 2024-03-11T19:00:17.061 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:16.462+0000 7f6839b3c700 1 -- 172.21.15.29:0/2547494390 wait complete. 2024-03-11T19:00:17.061 INFO:teuthology.orchestra.run.smithi029.stdout:mgr epoch 5 is available 2024-03-11T19:00:17.061 INFO:teuthology.orchestra.run.smithi029.stdout:Setting orchestrator backend to cephadm... 2024-03-11T19:00:17.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:17 smithi029 ceph-mon[32439]: [11/Mar/2024:19:00:16] ENGINE Bus STARTING 2024-03-11T19:00:17.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:17 smithi029 ceph-mon[32439]: [11/Mar/2024:19:00:16] ENGINE Serving on https://172.21.15.29:7150 2024-03-11T19:00:17.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:17 smithi029 ceph-mon[32439]: [11/Mar/2024:19:00:16] ENGINE Serving on http://172.21.15.29:8765 2024-03-11T19:00:17.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:17 smithi029 ceph-mon[32439]: [11/Mar/2024:19:00:16] ENGINE Bus STARTED 2024-03-11T19:00:17.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:17 smithi029 ceph-mon[32439]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T19:00:17.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:17 smithi029 ceph-mon[32439]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T19:00:18.252 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.458+0000 7fb4ec528700 1 Processor -- start 2024-03-11T19:00:18.252 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.458+0000 7fb4ec528700 1 -- start start 2024-03-11T19:00:18.252 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.458+0000 7fb4ec528700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4e4107ec0 0x7fb4e410a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:18.252 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.458+0000 7fb4ec528700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e407ae90 con 0x7fb4e4107ec0 2024-03-11T19:00:18.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.458+0000 7fb4ea2c4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4e4107ec0 0x7fb4e410a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:18.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.458+0000 7fb4ea2c4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4e4107ec0 0x7fb4e410a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36922/0 (socket says 172.21.15.29:36922) 2024-03-11T19:00:18.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.458+0000 7fb4ea2c4700 1 -- 172.21.15.29:0/3978627623 learned_addr learned my addr 172.21.15.29:0/3978627623 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:18.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.459+0000 7fb4ea2c4700 1 -- 172.21.15.29:0/3978627623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e410a7f0 con 0x7fb4e4107ec0 2024-03-11T19:00:18.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.459+0000 7fb4ea2c4700 1 --2- 172.21.15.29:0/3978627623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4e4107ec0 0x7fb4e410a2b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00098a0 tx=0x7fb4e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b2fc61ace3078f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:18.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.459+0000 7fb4e92c2700 1 -- 172.21.15.29:0/3978627623 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4e0004160 con 0x7fb4e4107ec0 2024-03-11T19:00:18.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.459+0000 7fb4e92c2700 1 -- 172.21.15.29:0/3978627623 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fb4e00042c0 con 0x7fb4e4107ec0 2024-03-11T19:00:18.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.459+0000 7fb4e92c2700 1 -- 172.21.15.29:0/3978627623 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4e0011a50 con 0x7fb4e4107ec0 2024-03-11T19:00:18.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.460+0000 7fb4ec528700 1 -- 172.21.15.29:0/3978627623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4e4107ec0 msgr2=0x7fb4e410a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:18.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.460+0000 7fb4ec528700 1 --2- 172.21.15.29:0/3978627623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4e4107ec0 0x7fb4e410a2b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00098a0 tx=0x7fb4e0009bb0 comp rx=0 tx=0).stop 2024-03-11T19:00:18.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.460+0000 7fb4ec528700 1 -- 172.21.15.29:0/3978627623 shutdown_connections 2024-03-11T19:00:18.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.460+0000 7fb4ec528700 1 --2- 172.21.15.29:0/3978627623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4e4107ec0 0x7fb4e410a2b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:18.253 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.460+0000 7fb4ec528700 1 -- 172.21.15.29:0/3978627623 >> 172.21.15.29:0/3978627623 conn(0x7fb4e4101930 msgr2=0x7fb4e4103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:18.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.460+0000 7fb4ec528700 1 -- 172.21.15.29:0/3978627623 shutdown_connections 2024-03-11T19:00:18.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.460+0000 7fb4ec528700 1 -- 172.21.15.29:0/3978627623 wait complete. 2024-03-11T19:00:18.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.460+0000 7fb4ec528700 1 Processor -- start 2024-03-11T19:00:18.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.460+0000 7fb4ec528700 1 -- start start 2024-03-11T19:00:18.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.461+0000 7fb4ec528700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4e4107ec0 0x7fb4e4107740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:18.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.461+0000 7fb4ec528700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e4107c80 con 0x7fb4e4107ec0 2024-03-11T19:00:18.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.461+0000 7fb4ea2c4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4e4107ec0 0x7fb4e4107740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:18.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.461+0000 7fb4ea2c4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4e4107ec0 0x7fb4e4107740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36926/0 (socket says 172.21.15.29:36926) 2024-03-11T19:00:18.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.461+0000 7fb4ea2c4700 1 -- 172.21.15.29:0/4184485508 learned_addr learned my addr 172.21.15.29:0/4184485508 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:18.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.461+0000 7fb4ea2c4700 1 -- 172.21.15.29:0/4184485508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e0009550 con 0x7fb4e4107ec0 2024-03-11T19:00:18.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.461+0000 7fb4ea2c4700 1 --2- 172.21.15.29:0/4184485508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4e4107ec0 0x7fb4e4107740 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0006af0 tx=0x7fb4e0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:18.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.461+0000 7fb4db7fe700 1 -- 172.21.15.29:0/4184485508 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4e0003930 con 0x7fb4e4107ec0 2024-03-11T19:00:18.254 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.461+0000 7fb4db7fe700 1 -- 172.21.15.29:0/4184485508 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fb4e0003a90 con 0x7fb4e4107ec0 2024-03-11T19:00:18.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.461+0000 7fb4db7fe700 1 -- 172.21.15.29:0/4184485508 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4e0019670 con 0x7fb4e4107ec0 2024-03-11T19:00:18.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.461+0000 7fb4ec528700 1 -- 172.21.15.29:0/4184485508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4e4105e70 con 0x7fb4e4107ec0 2024-03-11T19:00:18.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.461+0000 7fb4ec528700 1 -- 172.21.15.29:0/4184485508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4e4106310 con 0x7fb4e4107ec0 2024-03-11T19:00:18.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.462+0000 7fb4db7fe700 1 -- 172.21.15.29:0/4184485508 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 7) v1 ==== 45663+0+0 (secure 0 0 0) 0x7fb4e001aa30 con 0x7fb4e4107ec0 2024-03-11T19:00:18.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.462+0000 7fb4ec528700 1 -- 172.21.15.29:0/4184485508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4e410b580 con 0x7fb4e4107ec0 2024-03-11T19:00:18.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.462+0000 7fb4db7fe700 1 --2- 172.21.15.29:0/4184485508 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fb4d0038ba0 0x7fb4d003b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:18.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.462+0000 7fb4db7fe700 1 -- 172.21.15.29:0/4184485508 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb4e004cd70 con 0x7fb4e4107ec0 2024-03-11T19:00:18.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.463+0000 7fb4e9ac3700 1 --2- 172.21.15.29:0/4184485508 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fb4d0038ba0 0x7fb4d003b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:18.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.463+0000 7fb4e9ac3700 1 --2- 172.21.15.29:0/4184485508 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fb4d0038ba0 0x7fb4d003b060 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4006fd0 tx=0x7fb4d4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:18.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.467+0000 7fb4db7fe700 1 -- 172.21.15.29:0/4184485508 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb4e0018730 con 0x7fb4e4107ec0 2024-03-11T19:00:18.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.503+0000 7fb4db7fe700 1 -- 172.21.15.29:0/4184485508 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fb4e000b3e0 con 0x7fb4e4107ec0 2024-03-11T19:00:18.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.689+0000 7fb4ec528700 1 -- 172.21.15.29:0/4184485508 --> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4e4106a00 con 0x7fb4d0038ba0 2024-03-11T19:00:18.255 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.694+0000 7fb4db7fe700 1 -- 172.21.15.29:0/4184485508 <== mgr.14118 v2:172.21.15.29:6800/3585665950 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb4e4106a00 con 0x7fb4d0038ba0 2024-03-11T19:00:18.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.696+0000 7fb4ec528700 1 -- 172.21.15.29:0/4184485508 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fb4d0038ba0 msgr2=0x7fb4d003b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:18.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.696+0000 7fb4ec528700 1 --2- 172.21.15.29:0/4184485508 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fb4d0038ba0 0x7fb4d003b060 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4006fd0 tx=0x7fb4d4006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:18.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.696+0000 7fb4ec528700 1 -- 172.21.15.29:0/4184485508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4e4107ec0 msgr2=0x7fb4e4107740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:18.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.696+0000 7fb4ec528700 1 --2- 172.21.15.29:0/4184485508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4e4107ec0 0x7fb4e4107740 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0006af0 tx=0x7fb4e0011ab0 comp rx=0 tx=0).stop 2024-03-11T19:00:18.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.696+0000 7fb4ec528700 1 -- 172.21.15.29:0/4184485508 shutdown_connections 2024-03-11T19:00:18.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.696+0000 7fb4ec528700 1 --2- 172.21.15.29:0/4184485508 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fb4d0038ba0 0x7fb4d003b060 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:18.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.696+0000 7fb4ec528700 1 --2- 172.21.15.29:0/4184485508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4e4107ec0 0x7fb4e4107740 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:18.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.696+0000 7fb4ec528700 1 -- 172.21.15.29:0/4184485508 >> 172.21.15.29:0/4184485508 conn(0x7fb4e4101930 msgr2=0x7fb4e4102400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:18.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.696+0000 7fb4ec528700 1 -- 172.21.15.29:0/4184485508 shutdown_connections 2024-03-11T19:00:18.256 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:17.696+0000 7fb4ec528700 1 -- 172.21.15.29:0/4184485508 wait complete. 2024-03-11T19:00:18.880 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:18 smithi029 ceph-mon[32439]: mgrmap e8: x(active, since 2s) 2024-03-11T19:00:18.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:18 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' 2024-03-11T19:00:18.881 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:18 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:00:19.432 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-11T19:00:19.433 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.663+0000 7ff3ed3d6700 1 Processor -- start 2024-03-11T19:00:19.433 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.663+0000 7ff3ed3d6700 1 -- start start 2024-03-11T19:00:19.433 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.663+0000 7ff3ed3d6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3e8105d70 0x7ff3e8106190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:19.433 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.663+0000 7ff3ed3d6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e807adf0 con 0x7ff3e8105d70 2024-03-11T19:00:19.433 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.663+0000 7ff3e6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3e8105d70 0x7ff3e8106190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:19.433 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.663+0000 7ff3e6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3e8105d70 0x7ff3e8106190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36928/0 (socket says 172.21.15.29:36928) 2024-03-11T19:00:19.433 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.663+0000 7ff3e6ffd700 1 -- 172.21.15.29:0/1357412450 learned_addr learned my addr 172.21.15.29:0/1357412450 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:19.433 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.664+0000 7ff3e6ffd700 1 -- 172.21.15.29:0/1357412450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e807af30 con 0x7ff3e8105d70 2024-03-11T19:00:19.433 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.664+0000 7ff3e6ffd700 1 --2- 172.21.15.29:0/1357412450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3e8105d70 0x7ff3e8106190 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0009e80 tx=0x7ff3e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1b425bead66bbbf server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:19.433 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.664+0000 7ff3e5ffb700 1 -- 172.21.15.29:0/1357412450 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff3e000e070 con 0x7ff3e8105d70 2024-03-11T19:00:19.433 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.664+0000 7ff3e5ffb700 1 -- 172.21.15.29:0/1357412450 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7ff3e000b8c0 con 0x7ff3e8105d70 2024-03-11T19:00:19.434 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.664+0000 7ff3e5ffb700 1 -- 172.21.15.29:0/1357412450 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff3e00044b0 con 0x7ff3e8105d70 2024-03-11T19:00:19.434 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.665+0000 7ff3ed3d6700 1 -- 172.21.15.29:0/1357412450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3e8105d70 msgr2=0x7ff3e8106190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:19.434 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.665+0000 7ff3ed3d6700 1 --2- 172.21.15.29:0/1357412450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3e8105d70 0x7ff3e8106190 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0009e80 tx=0x7ff3e000b330 comp rx=0 tx=0).stop 2024-03-11T19:00:19.434 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.665+0000 7ff3ed3d6700 1 -- 172.21.15.29:0/1357412450 shutdown_connections 2024-03-11T19:00:19.434 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.665+0000 7ff3ed3d6700 1 --2- 172.21.15.29:0/1357412450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3e8105d70 0x7ff3e8106190 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:19.434 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.665+0000 7ff3ed3d6700 1 -- 172.21.15.29:0/1357412450 >> 172.21.15.29:0/1357412450 conn(0x7ff3e8101930 msgr2=0x7ff3e8103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:19.434 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.665+0000 7ff3ed3d6700 1 -- 172.21.15.29:0/1357412450 shutdown_connections 2024-03-11T19:00:19.434 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.665+0000 7ff3ed3d6700 1 -- 172.21.15.29:0/1357412450 wait complete. 2024-03-11T19:00:19.434 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.665+0000 7ff3ed3d6700 1 Processor -- start 2024-03-11T19:00:19.434 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.666+0000 7ff3ed3d6700 1 -- start start 2024-03-11T19:00:19.434 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.666+0000 7ff3ed3d6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3e8105d70 0x7ff3e819d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:19.434 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.666+0000 7ff3ed3d6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e819d7f0 con 0x7ff3e8105d70 2024-03-11T19:00:19.434 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.666+0000 7ff3e6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3e8105d70 0x7ff3e819d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:19.434 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.666+0000 7ff3e6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3e8105d70 0x7ff3e819d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36934/0 (socket says 172.21.15.29:36934) 2024-03-11T19:00:19.434 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.666+0000 7ff3e6ffd700 1 -- 172.21.15.29:0/166958718 learned_addr learned my addr 172.21.15.29:0/166958718 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:19.435 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.666+0000 7ff3e6ffd700 1 -- 172.21.15.29:0/166958718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e0009b30 con 0x7ff3e8105d70 2024-03-11T19:00:19.435 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.666+0000 7ff3e6ffd700 1 --2- 172.21.15.29:0/166958718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3e8105d70 0x7ff3e819d2b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0004070 tx=0x7ff3e00093c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:19.435 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.666+0000 7ff3dffff700 1 -- 172.21.15.29:0/166958718 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff3e000e040 con 0x7ff3e8105d70 2024-03-11T19:00:19.435 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.666+0000 7ff3ed3d6700 1 -- 172.21.15.29:0/166958718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e819d9f0 con 0x7ff3e8105d70 2024-03-11T19:00:19.435 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.666+0000 7ff3dffff700 1 -- 172.21.15.29:0/166958718 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7ff3e000bd70 con 0x7ff3e8105d70 2024-03-11T19:00:19.435 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.666+0000 7ff3dffff700 1 -- 172.21.15.29:0/166958718 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff3e001c7a0 con 0x7ff3e8105d70 2024-03-11T19:00:19.435 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.666+0000 7ff3ed3d6700 1 -- 172.21.15.29:0/166958718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e819def0 con 0x7ff3e8105d70 2024-03-11T19:00:19.435 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.667+0000 7ff3dffff700 1 -- 172.21.15.29:0/166958718 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7ff3e001c9e0 con 0x7ff3e8105d70 2024-03-11T19:00:19.435 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.668+0000 7ff3ed3d6700 1 -- 172.21.15.29:0/166958718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e8196d80 con 0x7ff3e8105d70 2024-03-11T19:00:19.435 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.668+0000 7ff3dffff700 1 --2- 172.21.15.29:0/166958718 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7ff3d4038c60 0x7ff3d403b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:19.435 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.668+0000 7ff3dffff700 1 -- 172.21.15.29:0/166958718 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff3e004cea0 con 0x7ff3e8105d70 2024-03-11T19:00:19.435 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.668+0000 7ff3e67fc700 1 --2- 172.21.15.29:0/166958718 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7ff3d4038c60 0x7ff3d403b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:19.435 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.668+0000 7ff3e67fc700 1 --2- 172.21.15.29:0/166958718 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7ff3d4038c60 0x7ff3d403b120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0006fd0 tx=0x7ff3d0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:19.436 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.673+0000 7ff3dffff700 1 -- 172.21.15.29:0/166958718 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff3e0023a10 con 0x7ff3e8105d70 2024-03-11T19:00:19.436 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.877+0000 7ff3ed3d6700 1 -- 172.21.15.29:0/166958718 --> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3e802da40 con 0x7ff3d4038c60 2024-03-11T19:00:19.436 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.878+0000 7ff3dffff700 1 -- 172.21.15.29:0/166958718 <== mgr.14118 v2:172.21.15.29:6800/3585665950 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7ff3e802da40 con 0x7ff3d4038c60 2024-03-11T19:00:19.436 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.881+0000 7ff3ed3d6700 1 -- 172.21.15.29:0/166958718 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7ff3d4038c60 msgr2=0x7ff3d403b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:19.436 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.881+0000 7ff3ed3d6700 1 --2- 172.21.15.29:0/166958718 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7ff3d4038c60 0x7ff3d403b120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0006fd0 tx=0x7ff3d0006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:19.436 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.881+0000 7ff3ed3d6700 1 -- 172.21.15.29:0/166958718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3e8105d70 msgr2=0x7ff3e819d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:19.436 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.881+0000 7ff3ed3d6700 1 --2- 172.21.15.29:0/166958718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3e8105d70 0x7ff3e819d2b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0004070 tx=0x7ff3e00093c0 comp rx=0 tx=0).stop 2024-03-11T19:00:19.436 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.881+0000 7ff3ed3d6700 1 -- 172.21.15.29:0/166958718 shutdown_connections 2024-03-11T19:00:19.436 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.881+0000 7ff3ed3d6700 1 --2- 172.21.15.29:0/166958718 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7ff3d4038c60 0x7ff3d403b120 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:19.436 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.881+0000 7ff3ed3d6700 1 --2- 172.21.15.29:0/166958718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3e8105d70 0x7ff3e819d2b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:19.436 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.881+0000 7ff3ed3d6700 1 -- 172.21.15.29:0/166958718 >> 172.21.15.29:0/166958718 conn(0x7ff3e8101930 msgr2=0x7ff3e8191800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:19.436 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.881+0000 7ff3ed3d6700 1 -- 172.21.15.29:0/166958718 shutdown_connections 2024-03-11T19:00:19.436 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:18.881+0000 7ff3ed3d6700 1 -- 172.21.15.29:0/166958718 wait complete. 2024-03-11T19:00:19.436 INFO:teuthology.orchestra.run.smithi029.stdout:Generating ssh key... 2024-03-11T19:00:19.822 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:19 smithi029 ceph-mon[32439]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:00:20.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:20 smithi029 ceph-mon[32439]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:00:20.905 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: Generating public/private rsa key pair. 2024-03-11T19:00:20.906 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: Your identification has been saved in /tmp/tmpte5bdoxx/key. 2024-03-11T19:00:20.906 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: Your public key has been saved in /tmp/tmpte5bdoxx/key.pub. 2024-03-11T19:00:20.906 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: The key fingerprint is: 2024-03-11T19:00:20.906 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: SHA256:hY67LxC7oQt/dUK4urmGdjLzy7M+jUot7zBvBgRyhwY ceph-638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T19:00:20.906 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: The key's randomart image is: 2024-03-11T19:00:20.906 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: +---[RSA 3072]----+ 2024-03-11T19:00:20.906 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: |E. . | 2024-03-11T19:00:20.906 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: |o + . . | 2024-03-11T19:00:20.906 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: |oo . . . . | 2024-03-11T19:00:20.906 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: | . o .o . | 2024-03-11T19:00:20.906 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: |. =. S | 2024-03-11T19:00:20.906 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: | .. = o.. | 2024-03-11T19:00:20.906 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: |.*.+o=.o | 2024-03-11T19:00:20.906 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: |oB@Oo... | 2024-03-11T19:00:20.907 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: |.o^#= .o. | 2024-03-11T19:00:20.907 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: +----[SHA256]-----+ 2024-03-11T19:00:21.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.849+0000 7fae69052700 1 Processor -- start 2024-03-11T19:00:21.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.849+0000 7fae69052700 1 -- start start 2024-03-11T19:00:21.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.849+0000 7fae69052700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae64107eb0 0x7fae6410a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:21.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.849+0000 7fae69052700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6407ae90 con 0x7fae64107eb0 2024-03-11T19:00:21.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.849+0000 7fae62d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae64107eb0 0x7fae6410a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:21.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.849+0000 7fae62d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae64107eb0 0x7fae6410a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36946/0 (socket says 172.21.15.29:36946) 2024-03-11T19:00:21.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.849+0000 7fae62d9d700 1 -- 172.21.15.29:0/2370489498 learned_addr learned my addr 172.21.15.29:0/2370489498 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:21.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.850+0000 7fae62d9d700 1 -- 172.21.15.29:0/2370489498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae6410a7e0 con 0x7fae64107eb0 2024-03-11T19:00:21.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.850+0000 7fae62d9d700 1 --2- 172.21.15.29:0/2370489498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae64107eb0 0x7fae6410a2a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fae5c009e80 tx=0x7fae5c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb84ef3fdb11103 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:21.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.850+0000 7fae61d9b700 1 -- 172.21.15.29:0/2370489498 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae5c00e070 con 0x7fae64107eb0 2024-03-11T19:00:21.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.850+0000 7fae61d9b700 1 -- 172.21.15.29:0/2370489498 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fae5c00b8c0 con 0x7fae64107eb0 2024-03-11T19:00:21.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.850+0000 7fae61d9b700 1 -- 172.21.15.29:0/2370489498 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae5c0044b0 con 0x7fae64107eb0 2024-03-11T19:00:21.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.850+0000 7fae69052700 1 -- 172.21.15.29:0/2370489498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae64107eb0 msgr2=0x7fae6410a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:21.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.850+0000 7fae69052700 1 --2- 172.21.15.29:0/2370489498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae64107eb0 0x7fae6410a2a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fae5c009e80 tx=0x7fae5c00b330 comp rx=0 tx=0).stop 2024-03-11T19:00:21.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.851+0000 7fae69052700 1 -- 172.21.15.29:0/2370489498 shutdown_connections 2024-03-11T19:00:21.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.851+0000 7fae69052700 1 --2- 172.21.15.29:0/2370489498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae64107eb0 0x7fae6410a2a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:21.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.851+0000 7fae69052700 1 -- 172.21.15.29:0/2370489498 >> 172.21.15.29:0/2370489498 conn(0x7fae641018a0 msgr2=0x7fae64103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:21.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.851+0000 7fae69052700 1 -- 172.21.15.29:0/2370489498 shutdown_connections 2024-03-11T19:00:21.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.851+0000 7fae69052700 1 -- 172.21.15.29:0/2370489498 wait complete. 2024-03-11T19:00:21.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.851+0000 7fae69052700 1 Processor -- start 2024-03-11T19:00:21.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.852+0000 7fae69052700 1 -- start start 2024-03-11T19:00:21.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.852+0000 7fae69052700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae64107eb0 0x7fae6419b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:21.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.852+0000 7fae69052700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6419b600 con 0x7fae64107eb0 2024-03-11T19:00:21.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.852+0000 7fae62d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae64107eb0 0x7fae6419b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:21.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.852+0000 7fae62d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae64107eb0 0x7fae6419b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36956/0 (socket says 172.21.15.29:36956) 2024-03-11T19:00:21.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.852+0000 7fae62d9d700 1 -- 172.21.15.29:0/583838555 learned_addr learned my addr 172.21.15.29:0/583838555 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:21.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.852+0000 7fae62d9d700 1 -- 172.21.15.29:0/583838555 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5c009b30 con 0x7fae64107eb0 2024-03-11T19:00:21.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.852+0000 7fae62d9d700 1 --2- 172.21.15.29:0/583838555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae64107eb0 0x7fae6419b0c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fae5c004240 tx=0x7fae5c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:21.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.852+0000 7fae5bfff700 1 -- 172.21.15.29:0/583838555 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae5c00e040 con 0x7fae64107eb0 2024-03-11T19:00:21.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.852+0000 7fae5bfff700 1 -- 172.21.15.29:0/583838555 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fae5c00bd70 con 0x7fae64107eb0 2024-03-11T19:00:21.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.852+0000 7fae69052700 1 -- 172.21.15.29:0/583838555 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae6419b800 con 0x7fae64107eb0 2024-03-11T19:00:21.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.853+0000 7fae5bfff700 1 -- 172.21.15.29:0/583838555 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae5c01c6b0 con 0x7fae64107eb0 2024-03-11T19:00:21.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.853+0000 7fae69052700 1 -- 172.21.15.29:0/583838555 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae6419bd00 con 0x7fae64107eb0 2024-03-11T19:00:21.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.853+0000 7fae5bfff700 1 -- 172.21.15.29:0/583838555 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fae5c01c8f0 con 0x7fae64107eb0 2024-03-11T19:00:21.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.853+0000 7fae69052700 1 -- 172.21.15.29:0/583838555 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae6410b570 con 0x7fae64107eb0 2024-03-11T19:00:21.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.854+0000 7fae5bfff700 1 --2- 172.21.15.29:0/583838555 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fae50038cb0 0x7fae5003b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:21.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.854+0000 7fae5bfff700 1 -- 172.21.15.29:0/583838555 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fae5c04cce0 con 0x7fae64107eb0 2024-03-11T19:00:21.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.854+0000 7fae6259c700 1 --2- 172.21.15.29:0/583838555 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fae50038cb0 0x7fae5003b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:21.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.854+0000 7fae6259c700 1 --2- 172.21.15.29:0/583838555 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fae50038cb0 0x7fae5003b170 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fae4c006fd0 tx=0x7fae4c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:21.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:19.859+0000 7fae5bfff700 1 -- 172.21.15.29:0/583838555 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fae5c011570 con 0x7fae64107eb0 2024-03-11T19:00:21.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:20.060+0000 7fae69052700 1 -- 172.21.15.29:0/583838555 --> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fae64102ae0 con 0x7fae50038cb0 2024-03-11T19:00:21.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:20.681+0000 7fae5bfff700 1 -- 172.21.15.29:0/583838555 <== mgr.14118 v2:172.21.15.29:6800/3585665950 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fae64102ae0 con 0x7fae50038cb0 2024-03-11T19:00:21.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:20.683+0000 7fae69052700 1 -- 172.21.15.29:0/583838555 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fae50038cb0 msgr2=0x7fae5003b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:21.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:20.683+0000 7fae69052700 1 --2- 172.21.15.29:0/583838555 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fae50038cb0 0x7fae5003b170 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fae4c006fd0 tx=0x7fae4c006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:21.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:20.683+0000 7fae69052700 1 -- 172.21.15.29:0/583838555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae64107eb0 msgr2=0x7fae6419b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:21.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:20.683+0000 7fae69052700 1 --2- 172.21.15.29:0/583838555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae64107eb0 0x7fae6419b0c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fae5c004240 tx=0x7fae5c004600 comp rx=0 tx=0).stop 2024-03-11T19:00:21.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:20.683+0000 7fae69052700 1 -- 172.21.15.29:0/583838555 shutdown_connections 2024-03-11T19:00:21.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:20.683+0000 7fae69052700 1 --2- 172.21.15.29:0/583838555 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fae50038cb0 0x7fae5003b170 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:21.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:20.683+0000 7fae69052700 1 --2- 172.21.15.29:0/583838555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae64107eb0 0x7fae6419b0c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:21.300 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:20.683+0000 7fae69052700 1 -- 172.21.15.29:0/583838555 >> 172.21.15.29:0/583838555 conn(0x7fae641018a0 msgr2=0x7fae641023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:21.300 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:20.683+0000 7fae69052700 1 -- 172.21.15.29:0/583838555 shutdown_connections 2024-03-11T19:00:21.300 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:20.683+0000 7fae69052700 1 -- 172.21.15.29:0/583838555 wait complete. 2024-03-11T19:00:21.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:21 smithi029 ceph-mon[32439]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:00:21.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:21 smithi029 ceph-mon[32439]: Generating ssh key... 2024-03-11T19:00:21.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:21 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' 2024-03-11T19:00:21.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:21 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' 2024-03-11T19:00:22.494 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDP0ONkhl+F6vFrvdZYF63qAOjY6/P8wA7M06kw3BCc3LajkG/j5Ik5c2R20mSlZ7DXtfDVGxV77ORnttjoT2MCKr6qJAOOdoovc973s3Twe05VFlnpE5g6hQ3zhbD+zdZOiAydkhq42mFHLkgmCSl//K08QJLsakKvlKvAbEppkZA6aIXhoNtfXoGBT92B35Kgpw1PgusmnDs9v2WCtcQko5fFU2o1Nre+rBqsPlZR1/JJkR1B1qXRsjb0fQwMq1lCiPFyK6aVQMdsmBzOkDLh8YTqWipFQ+GHTnerwJd1IUEcFJyz0EOSJE0bhPVkANwlkDbZpb+I6xeEM4G3zjHABWJhPFRshwbX0tSFUeiIxP3TgI9EnqT1CU2IS+GvLuIRILAY8KeOoinek9xsYMwQ+TtHsyGONDMUe2uIHM9Uhdeu9ahqKodd3fHxtivMXj6k6EORB2xnPLvzYsCEv+Z+gll9rjVZRrzNloYsxe8yZgIIz1amlDOnlNLmaG77B1c= ceph-638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T19:00:22.494 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.714+0000 7fe3410ab700 1 Processor -- start 2024-03-11T19:00:22.494 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.714+0000 7fe3410ab700 1 -- start start 2024-03-11T19:00:22.494 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.714+0000 7fe3410ab700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe33c1074d0 0x7fe33c1078f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:22.495 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.714+0000 7fe3410ab700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33c107ec0 con 0x7fe33c1074d0 2024-03-11T19:00:22.495 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.714+0000 7fe33ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe33c1074d0 0x7fe33c1078f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:22.495 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.714+0000 7fe33ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe33c1074d0 0x7fe33c1078f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36970/0 (socket says 172.21.15.29:36970) 2024-03-11T19:00:22.495 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.714+0000 7fe33ad9d700 1 -- 172.21.15.29:0/1680937226 learned_addr learned my addr 172.21.15.29:0/1680937226 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:22.495 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.714+0000 7fe33ad9d700 1 -- 172.21.15.29:0/1680937226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33c108630 con 0x7fe33c1074d0 2024-03-11T19:00:22.495 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.714+0000 7fe33ad9d700 1 --2- 172.21.15.29:0/1680937226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe33c1074d0 0x7fe33c1078f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0098a0 tx=0x7fe32c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b796e7994a2bb1a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:22.495 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.715+0000 7fe339d9b700 1 -- 172.21.15.29:0/1680937226 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe32c004160 con 0x7fe33c1074d0 2024-03-11T19:00:22.495 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.715+0000 7fe339d9b700 1 -- 172.21.15.29:0/1680937226 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fe32c0042c0 con 0x7fe33c1074d0 2024-03-11T19:00:22.495 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.715+0000 7fe339d9b700 1 -- 172.21.15.29:0/1680937226 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe32c011a50 con 0x7fe33c1074d0 2024-03-11T19:00:22.495 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.715+0000 7fe3410ab700 1 -- 172.21.15.29:0/1680937226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe33c1074d0 msgr2=0x7fe33c1078f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:22.495 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.715+0000 7fe3410ab700 1 --2- 172.21.15.29:0/1680937226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe33c1074d0 0x7fe33c1078f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0098a0 tx=0x7fe32c009bb0 comp rx=0 tx=0).stop 2024-03-11T19:00:22.495 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.715+0000 7fe3410ab700 1 -- 172.21.15.29:0/1680937226 shutdown_connections 2024-03-11T19:00:22.495 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.715+0000 7fe3410ab700 1 --2- 172.21.15.29:0/1680937226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe33c1074d0 0x7fe33c1078f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:22.495 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.715+0000 7fe3410ab700 1 -- 172.21.15.29:0/1680937226 >> 172.21.15.29:0/1680937226 conn(0x7fe33c102ad0 msgr2=0x7fe33c104ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:22.496 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.715+0000 7fe3410ab700 1 -- 172.21.15.29:0/1680937226 shutdown_connections 2024-03-11T19:00:22.496 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.715+0000 7fe3410ab700 1 -- 172.21.15.29:0/1680937226 wait complete. 2024-03-11T19:00:22.496 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.716+0000 7fe3410ab700 1 Processor -- start 2024-03-11T19:00:22.496 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.716+0000 7fe3410ab700 1 -- start start 2024-03-11T19:00:22.496 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.716+0000 7fe3410ab700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe33c1074d0 0x7fe33c19b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:22.496 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.716+0000 7fe3410ab700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33c19b6e0 con 0x7fe33c1074d0 2024-03-11T19:00:22.496 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.717+0000 7fe33ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe33c1074d0 0x7fe33c19b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:22.496 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.717+0000 7fe33ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe33c1074d0 0x7fe33c19b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36986/0 (socket says 172.21.15.29:36986) 2024-03-11T19:00:22.496 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.717+0000 7fe33ad9d700 1 -- 172.21.15.29:0/1785896314 learned_addr learned my addr 172.21.15.29:0/1785896314 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:22.496 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.717+0000 7fe33ad9d700 1 -- 172.21.15.29:0/1785896314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32c009550 con 0x7fe33c1074d0 2024-03-11T19:00:22.496 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.717+0000 7fe33ad9d700 1 --2- 172.21.15.29:0/1785896314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe33c1074d0 0x7fe33c19b1a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe32c006af0 tx=0x7fe32c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:22.496 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.717+0000 7fe323fff700 1 -- 172.21.15.29:0/1785896314 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe32c003930 con 0x7fe33c1074d0 2024-03-11T19:00:22.496 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.717+0000 7fe323fff700 1 -- 172.21.15.29:0/1785896314 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fe32c003a90 con 0x7fe33c1074d0 2024-03-11T19:00:22.496 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.717+0000 7fe3410ab700 1 -- 172.21.15.29:0/1785896314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe33c19b8e0 con 0x7fe33c1074d0 2024-03-11T19:00:22.496 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.717+0000 7fe323fff700 1 -- 172.21.15.29:0/1785896314 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe32c019670 con 0x7fe33c1074d0 2024-03-11T19:00:22.497 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.717+0000 7fe3410ab700 1 -- 172.21.15.29:0/1785896314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe33c19bd80 con 0x7fe33c1074d0 2024-03-11T19:00:22.497 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.718+0000 7fe323fff700 1 -- 172.21.15.29:0/1785896314 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fe32c019930 con 0x7fe33c1074d0 2024-03-11T19:00:22.497 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.718+0000 7fe3410ab700 1 -- 172.21.15.29:0/1785896314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe33c10b860 con 0x7fe33c1074d0 2024-03-11T19:00:22.497 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.718+0000 7fe323fff700 1 --2- 172.21.15.29:0/1785896314 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fe324038c60 0x7fe32403b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:22.497 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.718+0000 7fe323fff700 1 -- 172.21.15.29:0/1785896314 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe32c04cf50 con 0x7fe33c1074d0 2024-03-11T19:00:22.497 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.719+0000 7fe33a59c700 1 --2- 172.21.15.29:0/1785896314 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fe324038c60 0x7fe32403b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:22.497 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.719+0000 7fe33a59c700 1 --2- 172.21.15.29:0/1785896314 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fe324038c60 0x7fe32403b120 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe330006fd0 tx=0x7fe330006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:22.497 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.723+0000 7fe323fff700 1 -- 172.21.15.29:0/1785896314 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe32c02b340 con 0x7fe33c1074d0 2024-03-11T19:00:22.497 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.927+0000 7fe3410ab700 1 -- 172.21.15.29:0/1785896314 --> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe33c103d60 con 0x7fe324038c60 2024-03-11T19:00:22.497 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.928+0000 7fe323fff700 1 -- 172.21.15.29:0/1785896314 <== mgr.14118 v2:172.21.15.29:6800/3585665950 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fe33c103d60 con 0x7fe324038c60 2024-03-11T19:00:22.497 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.930+0000 7fe3410ab700 1 -- 172.21.15.29:0/1785896314 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fe324038c60 msgr2=0x7fe32403b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:22.497 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.930+0000 7fe3410ab700 1 --2- 172.21.15.29:0/1785896314 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fe324038c60 0x7fe32403b120 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe330006fd0 tx=0x7fe330006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:22.497 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.930+0000 7fe3410ab700 1 -- 172.21.15.29:0/1785896314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe33c1074d0 msgr2=0x7fe33c19b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:22.497 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.930+0000 7fe3410ab700 1 --2- 172.21.15.29:0/1785896314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe33c1074d0 0x7fe33c19b1a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe32c006af0 tx=0x7fe32c011ab0 comp rx=0 tx=0).stop 2024-03-11T19:00:22.497 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.930+0000 7fe3410ab700 1 -- 172.21.15.29:0/1785896314 shutdown_connections 2024-03-11T19:00:22.498 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.930+0000 7fe3410ab700 1 --2- 172.21.15.29:0/1785896314 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fe324038c60 0x7fe32403b120 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:22.498 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.930+0000 7fe3410ab700 1 --2- 172.21.15.29:0/1785896314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe33c1074d0 0x7fe33c19b1a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:22.498 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.930+0000 7fe3410ab700 1 -- 172.21.15.29:0/1785896314 >> 172.21.15.29:0/1785896314 conn(0x7fe33c102ad0 msgr2=0x7fe33c103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:22.498 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.931+0000 7fe3410ab700 1 -- 172.21.15.29:0/1785896314 shutdown_connections 2024-03-11T19:00:22.498 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:21.931+0000 7fe3410ab700 1 -- 172.21.15.29:0/1785896314 wait complete. 2024-03-11T19:00:22.498 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-11T19:00:22.498 INFO:teuthology.orchestra.run.smithi029.stdout:Adding key to root@localhost authorized_keys... 2024-03-11T19:00:22.498 INFO:teuthology.orchestra.run.smithi029.stdout:Adding host smithi029... 2024-03-11T19:00:22.794 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:22 smithi029 ceph-mon[32439]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:00:23.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:23 smithi029 ceph-mon[32439]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi029", "addr": "172.21.15.29", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:00:24.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:24 smithi029 ceph-mon[32439]: Deploying cephadm binary to smithi029 2024-03-11T19:00:28.064 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Added host 'smithi029' with addr '172.21.15.29' 2024-03-11T19:00:28.064 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.900+0000 7fddc8307700 1 Processor -- start 2024-03-11T19:00:28.064 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.900+0000 7fddc8307700 1 -- start start 2024-03-11T19:00:28.065 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.900+0000 7fddc8307700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc0107460 0x7fddc0107880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:28.065 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.900+0000 7fddc8307700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc0107e50 con 0x7fddc0107460 2024-03-11T19:00:28.065 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.900+0000 7fddc60a3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc0107460 0x7fddc0107880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:28.065 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.900+0000 7fddc60a3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc0107460 0x7fddc0107880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36988/0 (socket says 172.21.15.29:36988) 2024-03-11T19:00:28.065 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.900+0000 7fddc60a3700 1 -- 172.21.15.29:0/2559155812 learned_addr learned my addr 172.21.15.29:0/2559155812 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:28.065 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.901+0000 7fddc60a3700 1 -- 172.21.15.29:0/2559155812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc0108610 con 0x7fddc0107460 2024-03-11T19:00:28.065 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.901+0000 7fddc60a3700 1 --2- 172.21.15.29:0/2559155812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc0107460 0x7fddc0107880 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fddbc0098a0 tx=0x7fddbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db71004a41d3a69e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:28.065 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.901+0000 7fddc50a1700 1 -- 172.21.15.29:0/2559155812 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fddbc004160 con 0x7fddc0107460 2024-03-11T19:00:28.065 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.901+0000 7fddc50a1700 1 -- 172.21.15.29:0/2559155812 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fddbc0042c0 con 0x7fddc0107460 2024-03-11T19:00:28.065 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.901+0000 7fddc50a1700 1 -- 172.21.15.29:0/2559155812 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fddbc011a50 con 0x7fddc0107460 2024-03-11T19:00:28.065 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.901+0000 7fddc8307700 1 -- 172.21.15.29:0/2559155812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc0107460 msgr2=0x7fddc0107880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:28.065 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.901+0000 7fddc8307700 1 --2- 172.21.15.29:0/2559155812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc0107460 0x7fddc0107880 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fddbc0098a0 tx=0x7fddbc009bb0 comp rx=0 tx=0).stop 2024-03-11T19:00:28.065 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.902+0000 7fddc8307700 1 -- 172.21.15.29:0/2559155812 shutdown_connections 2024-03-11T19:00:28.065 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.902+0000 7fddc8307700 1 --2- 172.21.15.29:0/2559155812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc0107460 0x7fddc0107880 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:28.066 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.902+0000 7fddc8307700 1 -- 172.21.15.29:0/2559155812 >> 172.21.15.29:0/2559155812 conn(0x7fddc0102a40 msgr2=0x7fddc0104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:28.066 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.902+0000 7fddc8307700 1 -- 172.21.15.29:0/2559155812 shutdown_connections 2024-03-11T19:00:28.066 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.902+0000 7fddc8307700 1 -- 172.21.15.29:0/2559155812 wait complete. 2024-03-11T19:00:28.066 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.902+0000 7fddc8307700 1 Processor -- start 2024-03-11T19:00:28.066 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.902+0000 7fddc8307700 1 -- start start 2024-03-11T19:00:28.066 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.902+0000 7fddc8307700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc0107460 0x7fddc019b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:28.066 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.902+0000 7fddc8307700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc019b690 con 0x7fddc0107460 2024-03-11T19:00:28.066 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.903+0000 7fddc60a3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc0107460 0x7fddc019b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:28.066 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.903+0000 7fddc60a3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc0107460 0x7fddc019b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36998/0 (socket says 172.21.15.29:36998) 2024-03-11T19:00:28.066 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.903+0000 7fddc60a3700 1 -- 172.21.15.29:0/633673834 learned_addr learned my addr 172.21.15.29:0/633673834 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:28.066 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.903+0000 7fddc60a3700 1 -- 172.21.15.29:0/633673834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddbc009550 con 0x7fddc0107460 2024-03-11T19:00:28.066 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.903+0000 7fddc60a3700 1 --2- 172.21.15.29:0/633673834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc0107460 0x7fddc019b150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fddbc006af0 tx=0x7fddbc011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:28.066 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.903+0000 7fddb77fe700 1 -- 172.21.15.29:0/633673834 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fddbc003930 con 0x7fddc0107460 2024-03-11T19:00:28.067 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.903+0000 7fddb77fe700 1 -- 172.21.15.29:0/633673834 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fddbc003a90 con 0x7fddc0107460 2024-03-11T19:00:28.067 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.903+0000 7fddc8307700 1 -- 172.21.15.29:0/633673834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc019b890 con 0x7fddc0107460 2024-03-11T19:00:28.067 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.903+0000 7fddb77fe700 1 -- 172.21.15.29:0/633673834 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fddbc019670 con 0x7fddc0107460 2024-03-11T19:00:28.067 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.903+0000 7fddc8307700 1 -- 172.21.15.29:0/633673834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc019bd30 con 0x7fddc0107460 2024-03-11T19:00:28.067 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.904+0000 7fddb77fe700 1 -- 172.21.15.29:0/633673834 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fddbc01aa30 con 0x7fddc0107460 2024-03-11T19:00:28.067 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.904+0000 7fddc8307700 1 -- 172.21.15.29:0/633673834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc010b840 con 0x7fddc0107460 2024-03-11T19:00:28.067 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.904+0000 7fddb77fe700 1 --2- 172.21.15.29:0/633673834 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fddac038cb0 0x7fddac03b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:28.067 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.904+0000 7fddb77fe700 1 -- 172.21.15.29:0/633673834 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fddbc04cf10 con 0x7fddc0107460 2024-03-11T19:00:28.067 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.904+0000 7fddc58a2700 1 --2- 172.21.15.29:0/633673834 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fddac038cb0 0x7fddac03b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:28.067 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.905+0000 7fddc58a2700 1 --2- 172.21.15.29:0/633673834 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fddac038cb0 0x7fddac03b170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fddb0006fd0 tx=0x7fddb0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:28.067 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:22.909+0000 7fddb77fe700 1 -- 172.21.15.29:0/633673834 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fddbc018760 con 0x7fddc0107460 2024-03-11T19:00:28.067 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:23.115+0000 7fddc8307700 1 -- 172.21.15.29:0/633673834 --> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi029", "addr": "172.21.15.29", "target": ["mon-mgr", ""]}) v1 -- 0x7fddc0103ce0 con 0x7fddac038cb0 2024-03-11T19:00:28.067 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:27.533+0000 7fddb77fe700 1 -- 172.21.15.29:0/633673834 <== mgr.14118 v2:172.21.15.29:6800/3585665950 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fddc0103ce0 con 0x7fddac038cb0 2024-03-11T19:00:28.067 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:27.536+0000 7fddc8307700 1 -- 172.21.15.29:0/633673834 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fddac038cb0 msgr2=0x7fddac03b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:28.067 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:27.536+0000 7fddc8307700 1 --2- 172.21.15.29:0/633673834 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fddac038cb0 0x7fddac03b170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fddb0006fd0 tx=0x7fddb0006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:28.068 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:27.536+0000 7fddc8307700 1 -- 172.21.15.29:0/633673834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc0107460 msgr2=0x7fddc019b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:28.068 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:27.536+0000 7fddc8307700 1 --2- 172.21.15.29:0/633673834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc0107460 0x7fddc019b150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fddbc006af0 tx=0x7fddbc011ab0 comp rx=0 tx=0).stop 2024-03-11T19:00:28.068 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:27.536+0000 7fddc8307700 1 -- 172.21.15.29:0/633673834 shutdown_connections 2024-03-11T19:00:28.068 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:27.536+0000 7fddc8307700 1 --2- 172.21.15.29:0/633673834 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fddac038cb0 0x7fddac03b170 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:28.068 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:27.536+0000 7fddc8307700 1 --2- 172.21.15.29:0/633673834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc0107460 0x7fddc019b150 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:28.068 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:27.536+0000 7fddc8307700 1 -- 172.21.15.29:0/633673834 >> 172.21.15.29:0/633673834 conn(0x7fddc0102a40 msgr2=0x7fddc01035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:28.068 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:27.536+0000 7fddc8307700 1 -- 172.21.15.29:0/633673834 shutdown_connections 2024-03-11T19:00:28.068 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:27.536+0000 7fddc8307700 1 -- 172.21.15.29:0/633673834 wait complete. 2024-03-11T19:00:28.068 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying unmanaged mon service... 2024-03-11T19:00:28.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:28 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' 2024-03-11T19:00:28.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:28 smithi029 ceph-mon[32439]: Added host smithi029 2024-03-11T19:00:28.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:28 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:00:30.002 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:29 smithi029 ceph-mon[32439]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:00:30.002 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:29 smithi029 ceph-mon[32439]: Saving service mon spec with placement count:5 2024-03-11T19:00:30.002 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:29 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' 2024-03-11T19:00:30.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-11T19:00:30.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.481+0000 7fa3725a2700 1 Processor -- start 2024-03-11T19:00:30.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.481+0000 7fa3725a2700 1 -- start start 2024-03-11T19:00:30.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.481+0000 7fa3725a2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa36c1072f0 0x7fa36c107710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:30.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.481+0000 7fa3725a2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c107ce0 con 0x7fa36c1072f0 2024-03-11T19:00:30.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.481+0000 7fa3715a0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa36c1072f0 0x7fa36c107710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:30.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.481+0000 7fa3715a0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa36c1072f0 0x7fa36c107710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56636/0 (socket says 172.21.15.29:56636) 2024-03-11T19:00:30.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.481+0000 7fa3715a0700 1 -- 172.21.15.29:0/3546359242 learned_addr learned my addr 172.21.15.29:0/3546359242 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:30.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.481+0000 7fa3715a0700 1 -- 172.21.15.29:0/3546359242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36c108450 con 0x7fa36c1072f0 2024-03-11T19:00:30.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.482+0000 7fa3715a0700 1 --2- 172.21.15.29:0/3546359242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa36c1072f0 0x7fa36c107710 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa3680098a0 tx=0x7fa368009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57857c2b9b218772 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:30.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.482+0000 7fa363fff700 1 -- 172.21.15.29:0/3546359242 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa368004160 con 0x7fa36c1072f0 2024-03-11T19:00:30.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.482+0000 7fa363fff700 1 -- 172.21.15.29:0/3546359242 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fa3680042c0 con 0x7fa36c1072f0 2024-03-11T19:00:30.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.482+0000 7fa363fff700 1 -- 172.21.15.29:0/3546359242 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa368011a50 con 0x7fa36c1072f0 2024-03-11T19:00:30.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.482+0000 7fa3725a2700 1 -- 172.21.15.29:0/3546359242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa36c1072f0 msgr2=0x7fa36c107710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:30.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.482+0000 7fa3725a2700 1 --2- 172.21.15.29:0/3546359242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa36c1072f0 0x7fa36c107710 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa3680098a0 tx=0x7fa368009bb0 comp rx=0 tx=0).stop 2024-03-11T19:00:30.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.482+0000 7fa3725a2700 1 -- 172.21.15.29:0/3546359242 shutdown_connections 2024-03-11T19:00:30.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.482+0000 7fa3725a2700 1 --2- 172.21.15.29:0/3546359242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa36c1072f0 0x7fa36c107710 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:30.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.482+0000 7fa3725a2700 1 -- 172.21.15.29:0/3546359242 >> 172.21.15.29:0/3546359242 conn(0x7fa36c1028b0 msgr2=0x7fa36c104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:30.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.483+0000 7fa3725a2700 1 -- 172.21.15.29:0/3546359242 shutdown_connections 2024-03-11T19:00:30.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.483+0000 7fa3725a2700 1 -- 172.21.15.29:0/3546359242 wait complete. 2024-03-11T19:00:30.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.483+0000 7fa3725a2700 1 Processor -- start 2024-03-11T19:00:30.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.483+0000 7fa3725a2700 1 -- start start 2024-03-11T19:00:30.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.483+0000 7fa3725a2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa36c1072f0 0x7fa36c19af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:30.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.483+0000 7fa3725a2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c19b4d0 con 0x7fa36c1072f0 2024-03-11T19:00:30.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.483+0000 7fa3715a0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa36c1072f0 0x7fa36c19af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:30.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.483+0000 7fa3715a0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa36c1072f0 0x7fa36c19af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56640/0 (socket says 172.21.15.29:56640) 2024-03-11T19:00:30.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.483+0000 7fa3715a0700 1 -- 172.21.15.29:0/771129742 learned_addr learned my addr 172.21.15.29:0/771129742 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:30.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.484+0000 7fa3715a0700 1 -- 172.21.15.29:0/771129742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa368009550 con 0x7fa36c1072f0 2024-03-11T19:00:30.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.484+0000 7fa3715a0700 1 --2- 172.21.15.29:0/771129742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa36c1072f0 0x7fa36c19af90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa368006af0 tx=0x7fa368011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:30.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.484+0000 7fa3627fc700 1 -- 172.21.15.29:0/771129742 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa368003930 con 0x7fa36c1072f0 2024-03-11T19:00:30.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.484+0000 7fa3627fc700 1 -- 172.21.15.29:0/771129742 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fa368003a90 con 0x7fa36c1072f0 2024-03-11T19:00:30.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.484+0000 7fa3725a2700 1 -- 172.21.15.29:0/771129742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa36c19b6d0 con 0x7fa36c1072f0 2024-03-11T19:00:30.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.484+0000 7fa3627fc700 1 -- 172.21.15.29:0/771129742 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa368019670 con 0x7fa36c1072f0 2024-03-11T19:00:30.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.484+0000 7fa3725a2700 1 -- 172.21.15.29:0/771129742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa36c19bb70 con 0x7fa36c1072f0 2024-03-11T19:00:30.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.485+0000 7fa3627fc700 1 -- 172.21.15.29:0/771129742 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fa3680198e0 con 0x7fa36c1072f0 2024-03-11T19:00:30.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.485+0000 7fa3627fc700 1 --2- 172.21.15.29:0/771129742 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fa358038cb0 0x7fa35803b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:30.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.485+0000 7fa3725a2700 1 -- 172.21.15.29:0/771129742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa36c10b680 con 0x7fa36c1072f0 2024-03-11T19:00:30.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.485+0000 7fa3627fc700 1 -- 172.21.15.29:0/771129742 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa36804d130 con 0x7fa36c1072f0 2024-03-11T19:00:30.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.485+0000 7fa370d9f700 1 --2- 172.21.15.29:0/771129742 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fa358038cb0 0x7fa35803b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:30.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.486+0000 7fa370d9f700 1 --2- 172.21.15.29:0/771129742 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fa358038cb0 0x7fa35803b170 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa35c006fd0 tx=0x7fa35c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:30.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.490+0000 7fa3627fc700 1 -- 172.21.15.29:0/771129742 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa368018c20 con 0x7fa36c1072f0 2024-03-11T19:00:30.470 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.706+0000 7fa3725a2700 1 -- 172.21.15.29:0/771129742 --> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fa36c103b70 con 0x7fa358038cb0 2024-03-11T19:00:30.470 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.711+0000 7fa3627fc700 1 -- 172.21.15.29:0/771129742 <== mgr.14118 v2:172.21.15.29:6800/3585665950 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa36c103b70 con 0x7fa358038cb0 2024-03-11T19:00:30.470 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.713+0000 7fa3725a2700 1 -- 172.21.15.29:0/771129742 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fa358038cb0 msgr2=0x7fa35803b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:30.470 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.713+0000 7fa3725a2700 1 --2- 172.21.15.29:0/771129742 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fa358038cb0 0x7fa35803b170 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa35c006fd0 tx=0x7fa35c006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:30.470 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.713+0000 7fa3725a2700 1 -- 172.21.15.29:0/771129742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa36c1072f0 msgr2=0x7fa36c19af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:30.470 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.713+0000 7fa3725a2700 1 --2- 172.21.15.29:0/771129742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa36c1072f0 0x7fa36c19af90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa368006af0 tx=0x7fa368011ab0 comp rx=0 tx=0).stop 2024-03-11T19:00:30.470 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.713+0000 7fa3725a2700 1 -- 172.21.15.29:0/771129742 shutdown_connections 2024-03-11T19:00:30.470 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.713+0000 7fa3725a2700 1 --2- 172.21.15.29:0/771129742 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fa358038cb0 0x7fa35803b170 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:30.470 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.713+0000 7fa3725a2700 1 --2- 172.21.15.29:0/771129742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa36c1072f0 0x7fa36c19af90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:30.470 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.713+0000 7fa3725a2700 1 -- 172.21.15.29:0/771129742 >> 172.21.15.29:0/771129742 conn(0x7fa36c1028b0 msgr2=0x7fa36c103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:30.470 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.713+0000 7fa3725a2700 1 -- 172.21.15.29:0/771129742 shutdown_connections 2024-03-11T19:00:30.470 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:28.713+0000 7fa3725a2700 1 -- 172.21.15.29:0/771129742 wait complete. 2024-03-11T19:00:30.470 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying unmanaged mgr service... 2024-03-11T19:00:31.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:31 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' 2024-03-11T19:00:31.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:31 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' 2024-03-11T19:00:32.304 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-11T19:00:32.304 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.904+0000 7f185e827700 1 Processor -- start 2024-03-11T19:00:32.304 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.904+0000 7f185e827700 1 -- start start 2024-03-11T19:00:32.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.904+0000 7f185e827700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185807ce50 0x7f185807b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:32.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.904+0000 7f185e827700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185807b8d0 con 0x7f185807ce50 2024-03-11T19:00:32.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.905+0000 7f1857fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185807ce50 0x7f185807b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:32.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.905+0000 7f1857fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185807ce50 0x7f185807b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56648/0 (socket says 172.21.15.29:56648) 2024-03-11T19:00:32.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.905+0000 7f1857fff700 1 -- 172.21.15.29:0/2553949898 learned_addr learned my addr 172.21.15.29:0/2553949898 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:32.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.905+0000 7f1857fff700 1 -- 172.21.15.29:0/2553949898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185807ba10 con 0x7f185807ce50 2024-03-11T19:00:32.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.905+0000 7f1857fff700 1 --2- 172.21.15.29:0/2553949898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185807ce50 0x7f185807b300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f18400098a0 tx=0x7f1840009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb12eb95c209af09 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:32.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.905+0000 7f1856ffd700 1 -- 172.21.15.29:0/2553949898 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1840004160 con 0x7f185807ce50 2024-03-11T19:00:32.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.905+0000 7f1856ffd700 1 -- 172.21.15.29:0/2553949898 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f18400042c0 con 0x7f185807ce50 2024-03-11T19:00:32.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.905+0000 7f1856ffd700 1 -- 172.21.15.29:0/2553949898 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18400119b0 con 0x7f185807ce50 2024-03-11T19:00:32.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.906+0000 7f185e827700 1 -- 172.21.15.29:0/2553949898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185807ce50 msgr2=0x7f185807b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:32.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.906+0000 7f185e827700 1 --2- 172.21.15.29:0/2553949898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185807ce50 0x7f185807b300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f18400098a0 tx=0x7f1840009bb0 comp rx=0 tx=0).stop 2024-03-11T19:00:32.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.906+0000 7f185e827700 1 -- 172.21.15.29:0/2553949898 shutdown_connections 2024-03-11T19:00:32.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.906+0000 7f185e827700 1 --2- 172.21.15.29:0/2553949898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185807ce50 0x7f185807b300 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:32.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.906+0000 7f185e827700 1 -- 172.21.15.29:0/2553949898 >> 172.21.15.29:0/2553949898 conn(0x7f1858102a40 msgr2=0x7f1858104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:32.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.907+0000 7f185e827700 1 -- 172.21.15.29:0/2553949898 shutdown_connections 2024-03-11T19:00:32.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.907+0000 7f185e827700 1 -- 172.21.15.29:0/2553949898 wait complete. 2024-03-11T19:00:32.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.907+0000 7f185e827700 1 Processor -- start 2024-03-11T19:00:32.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.907+0000 7f185e827700 1 -- start start 2024-03-11T19:00:32.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.907+0000 7f185e827700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185807ce50 0x7f185819d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:32.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.907+0000 7f185e827700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185819d850 con 0x7f185807ce50 2024-03-11T19:00:32.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.908+0000 7f1857fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185807ce50 0x7f185819d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:32.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.908+0000 7f1857fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185807ce50 0x7f185819d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56660/0 (socket says 172.21.15.29:56660) 2024-03-11T19:00:32.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.908+0000 7f1857fff700 1 -- 172.21.15.29:0/2178902012 learned_addr learned my addr 172.21.15.29:0/2178902012 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:32.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.908+0000 7f1857fff700 1 -- 172.21.15.29:0/2178902012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1840009550 con 0x7f185807ce50 2024-03-11T19:00:32.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.908+0000 7f1857fff700 1 --2- 172.21.15.29:0/2178902012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185807ce50 0x7f185819d310 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1840011980 tx=0x7f1840011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:32.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.908+0000 7f18557fa700 1 -- 172.21.15.29:0/2178902012 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18400038b0 con 0x7f185807ce50 2024-03-11T19:00:32.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.908+0000 7f185e827700 1 -- 172.21.15.29:0/2178902012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f185819da50 con 0x7f185807ce50 2024-03-11T19:00:32.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.908+0000 7f18557fa700 1 -- 172.21.15.29:0/2178902012 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1840003a10 con 0x7f185807ce50 2024-03-11T19:00:32.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.908+0000 7f18557fa700 1 -- 172.21.15.29:0/2178902012 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18400195e0 con 0x7f185807ce50 2024-03-11T19:00:32.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.908+0000 7f185e827700 1 -- 172.21.15.29:0/2178902012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185819def0 con 0x7f185807ce50 2024-03-11T19:00:32.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.909+0000 7f18557fa700 1 -- 172.21.15.29:0/2178902012 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f1840019840 con 0x7f185807ce50 2024-03-11T19:00:32.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.909+0000 7f18557fa700 1 --2- 172.21.15.29:0/2178902012 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f1844038c60 0x7f184403b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:32.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.909+0000 7f18557fa700 1 -- 172.21.15.29:0/2178902012 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f184004ce70 con 0x7f185807ce50 2024-03-11T19:00:32.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.909+0000 7f18577fe700 1 --2- 172.21.15.29:0/2178902012 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f1844038c60 0x7f184403b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:32.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.910+0000 7f185e827700 1 -- 172.21.15.29:0/2178902012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f185810d800 con 0x7f185807ce50 2024-03-11T19:00:32.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.910+0000 7f18577fe700 1 --2- 172.21.15.29:0/2178902012 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f1844038c60 0x7f184403b120 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1848006fd0 tx=0x7f1848006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:32.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:30.914+0000 7f18557fa700 1 -- 172.21.15.29:0/2178902012 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f184002b340 con 0x7f185807ce50 2024-03-11T19:00:32.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:31.181+0000 7f185e827700 1 -- 172.21.15.29:0/2178902012 --> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1858103cc0 con 0x7f1844038c60 2024-03-11T19:00:32.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:31.186+0000 7f18557fa700 1 -- 172.21.15.29:0/2178902012 <== mgr.14118 v2:172.21.15.29:6800/3585665950 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1858103cc0 con 0x7f1844038c60 2024-03-11T19:00:32.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:31.188+0000 7f185e827700 1 -- 172.21.15.29:0/2178902012 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f1844038c60 msgr2=0x7f184403b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:32.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:31.188+0000 7f185e827700 1 --2- 172.21.15.29:0/2178902012 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f1844038c60 0x7f184403b120 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1848006fd0 tx=0x7f1848006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:32.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:31.188+0000 7f185e827700 1 -- 172.21.15.29:0/2178902012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185807ce50 msgr2=0x7f185819d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:32.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:31.188+0000 7f185e827700 1 --2- 172.21.15.29:0/2178902012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185807ce50 0x7f185819d310 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1840011980 tx=0x7f1840011a60 comp rx=0 tx=0).stop 2024-03-11T19:00:32.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:31.188+0000 7f185e827700 1 -- 172.21.15.29:0/2178902012 shutdown_connections 2024-03-11T19:00:32.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:31.188+0000 7f185e827700 1 --2- 172.21.15.29:0/2178902012 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f1844038c60 0x7f184403b120 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:32.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:31.188+0000 7f185e827700 1 --2- 172.21.15.29:0/2178902012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185807ce50 0x7f185819d310 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:32.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:31.188+0000 7f185e827700 1 -- 172.21.15.29:0/2178902012 >> 172.21.15.29:0/2178902012 conn(0x7f1858102a40 msgr2=0x7f18581035b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:32.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:31.188+0000 7f185e827700 1 -- 172.21.15.29:0/2178902012 shutdown_connections 2024-03-11T19:00:32.309 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:31.189+0000 7f185e827700 1 -- 172.21.15.29:0/2178902012 wait complete. 2024-03-11T19:00:32.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:32 smithi029 ceph-mon[32439]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:00:32.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:32 smithi029 ceph-mon[32439]: Saving service mgr spec with placement count:2 2024-03-11T19:00:32.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:32 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' 2024-03-11T19:00:32.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:32 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' 2024-03-11T19:00:33.592 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.747+0000 7ffa68198700 1 Processor -- start 2024-03-11T19:00:33.593 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.747+0000 7ffa68198700 1 -- start start 2024-03-11T19:00:33.593 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.747+0000 7ffa68198700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa60107f10 0x7ffa6010a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:33.593 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.747+0000 7ffa68198700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6007ae30 con 0x7ffa60107f10 2024-03-11T19:00:33.593 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.747+0000 7ffa65f34700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa60107f10 0x7ffa6010a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:33.593 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.747+0000 7ffa65f34700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa60107f10 0x7ffa6010a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56666/0 (socket says 172.21.15.29:56666) 2024-03-11T19:00:33.593 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.747+0000 7ffa65f34700 1 -- 172.21.15.29:0/1949153995 learned_addr learned my addr 172.21.15.29:0/1949153995 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:33.593 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.748+0000 7ffa65f34700 1 -- 172.21.15.29:0/1949153995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6010a840 con 0x7ffa60107f10 2024-03-11T19:00:33.593 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.748+0000 7ffa65f34700 1 --2- 172.21.15.29:0/1949153995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa60107f10 0x7ffa6010a300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c0098a0 tx=0x7ffa5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5591a3ab42db7072 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:33.593 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.748+0000 7ffa64f32700 1 -- 172.21.15.29:0/1949153995 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa5c004160 con 0x7ffa60107f10 2024-03-11T19:00:33.593 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.748+0000 7ffa64f32700 1 -- 172.21.15.29:0/1949153995 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ffa5c0042c0 con 0x7ffa60107f10 2024-03-11T19:00:33.593 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.748+0000 7ffa64f32700 1 -- 172.21.15.29:0/1949153995 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa5c0119b0 con 0x7ffa60107f10 2024-03-11T19:00:33.593 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.749+0000 7ffa68198700 1 -- 172.21.15.29:0/1949153995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa60107f10 msgr2=0x7ffa6010a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:33.594 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.749+0000 7ffa68198700 1 --2- 172.21.15.29:0/1949153995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa60107f10 0x7ffa6010a300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c0098a0 tx=0x7ffa5c009bb0 comp rx=0 tx=0).stop 2024-03-11T19:00:33.594 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.749+0000 7ffa68198700 1 -- 172.21.15.29:0/1949153995 shutdown_connections 2024-03-11T19:00:33.594 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.749+0000 7ffa68198700 1 --2- 172.21.15.29:0/1949153995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa60107f10 0x7ffa6010a300 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:33.594 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.749+0000 7ffa68198700 1 -- 172.21.15.29:0/1949153995 >> 172.21.15.29:0/1949153995 conn(0x7ffa601018a0 msgr2=0x7ffa60103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:33.594 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.749+0000 7ffa68198700 1 -- 172.21.15.29:0/1949153995 shutdown_connections 2024-03-11T19:00:33.594 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.749+0000 7ffa68198700 1 -- 172.21.15.29:0/1949153995 wait complete. 2024-03-11T19:00:33.594 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.749+0000 7ffa68198700 1 Processor -- start 2024-03-11T19:00:33.594 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.750+0000 7ffa68198700 1 -- start start 2024-03-11T19:00:33.594 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.750+0000 7ffa68198700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa60107f10 0x7ffa60107790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:33.594 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.750+0000 7ffa68198700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa60107cd0 con 0x7ffa60107f10 2024-03-11T19:00:33.594 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.750+0000 7ffa65f34700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa60107f10 0x7ffa60107790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:33.594 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.750+0000 7ffa65f34700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa60107f10 0x7ffa60107790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56680/0 (socket says 172.21.15.29:56680) 2024-03-11T19:00:33.594 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.750+0000 7ffa65f34700 1 -- 172.21.15.29:0/285390311 learned_addr learned my addr 172.21.15.29:0/285390311 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:33.594 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.750+0000 7ffa65f34700 1 -- 172.21.15.29:0/285390311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5c009550 con 0x7ffa60107f10 2024-03-11T19:00:33.595 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.750+0000 7ffa65f34700 1 --2- 172.21.15.29:0/285390311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa60107f10 0x7ffa60107790 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c011980 tx=0x7ffa5c011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:33.595 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.750+0000 7ffa56ffd700 1 -- 172.21.15.29:0/285390311 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa5c003970 con 0x7ffa60107f10 2024-03-11T19:00:33.595 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.750+0000 7ffa68198700 1 -- 172.21.15.29:0/285390311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa60105ec0 con 0x7ffa60107f10 2024-03-11T19:00:33.595 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.750+0000 7ffa56ffd700 1 -- 172.21.15.29:0/285390311 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ffa5c003ad0 con 0x7ffa60107f10 2024-03-11T19:00:33.595 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.750+0000 7ffa56ffd700 1 -- 172.21.15.29:0/285390311 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa5c0195e0 con 0x7ffa60107f10 2024-03-11T19:00:33.595 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.750+0000 7ffa68198700 1 -- 172.21.15.29:0/285390311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa60106360 con 0x7ffa60107f10 2024-03-11T19:00:33.595 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.751+0000 7ffa56ffd700 1 -- 172.21.15.29:0/285390311 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7ffa5c01aae0 con 0x7ffa60107f10 2024-03-11T19:00:33.595 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.751+0000 7ffa68198700 1 -- 172.21.15.29:0/285390311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa6010b620 con 0x7ffa60107f10 2024-03-11T19:00:33.595 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.752+0000 7ffa56ffd700 1 --2- 172.21.15.29:0/285390311 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7ffa4c038c60 0x7ffa4c03b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:33.595 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.752+0000 7ffa56ffd700 1 -- 172.21.15.29:0/285390311 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ffa5c04cc50 con 0x7ffa60107f10 2024-03-11T19:00:33.595 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.752+0000 7ffa65733700 1 --2- 172.21.15.29:0/285390311 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7ffa4c038c60 0x7ffa4c03b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:33.595 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.752+0000 7ffa65733700 1 --2- 172.21.15.29:0/285390311 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7ffa4c038c60 0x7ffa4c03b120 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ffa50006fd0 tx=0x7ffa50006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:33.595 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.757+0000 7ffa56ffd700 1 -- 172.21.15.29:0/285390311 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ffa5c0237a0 con 0x7ffa60107f10 2024-03-11T19:00:33.596 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.958+0000 7ffa68198700 1 -- 172.21.15.29:0/285390311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7ffa6010b820 con 0x7ffa60107f10 2024-03-11T19:00:33.596 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.961+0000 7ffa56ffd700 1 -- 172.21.15.29:0/285390311 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7ffa5c02c3e0 con 0x7ffa60107f10 2024-03-11T19:00:33.596 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.963+0000 7ffa68198700 1 -- 172.21.15.29:0/285390311 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7ffa4c038c60 msgr2=0x7ffa4c03b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:33.596 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.963+0000 7ffa68198700 1 --2- 172.21.15.29:0/285390311 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7ffa4c038c60 0x7ffa4c03b120 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ffa50006fd0 tx=0x7ffa50006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:33.596 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.963+0000 7ffa68198700 1 -- 172.21.15.29:0/285390311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa60107f10 msgr2=0x7ffa60107790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:33.596 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.963+0000 7ffa68198700 1 --2- 172.21.15.29:0/285390311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa60107f10 0x7ffa60107790 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c011980 tx=0x7ffa5c011a60 comp rx=0 tx=0).stop 2024-03-11T19:00:33.596 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.963+0000 7ffa68198700 1 -- 172.21.15.29:0/285390311 shutdown_connections 2024-03-11T19:00:33.596 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.963+0000 7ffa68198700 1 --2- 172.21.15.29:0/285390311 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7ffa4c038c60 0x7ffa4c03b120 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:33.596 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.963+0000 7ffa68198700 1 --2- 172.21.15.29:0/285390311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa60107f10 0x7ffa60107790 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:33.596 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.963+0000 7ffa68198700 1 -- 172.21.15.29:0/285390311 >> 172.21.15.29:0/285390311 conn(0x7ffa601018a0 msgr2=0x7ffa60102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:33.596 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.964+0000 7ffa68198700 1 -- 172.21.15.29:0/285390311 shutdown_connections 2024-03-11T19:00:33.596 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:32.964+0000 7ffa68198700 1 -- 172.21.15.29:0/285390311 wait complete. 2024-03-11T19:00:34.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:33 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/285390311' entity='client.admin' 2024-03-11T19:00:34.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:33 smithi029 ceph-mon[32439]: from='mgr.14118 172.21.15.29:0/3656454746' entity='mgr.x' 2024-03-11T19:00:35.283 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.059+0000 7f49fe685700 1 Processor -- start 2024-03-11T19:00:35.283 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.059+0000 7f49fe685700 1 -- start start 2024-03-11T19:00:35.283 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.059+0000 7f49fe685700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f8107ec0 0x7f49f810a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:35.283 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.059+0000 7f49fe685700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f807ae90 con 0x7f49f8107ec0 2024-03-11T19:00:35.283 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.059+0000 7f49f7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f8107ec0 0x7f49f810a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:35.283 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.059+0000 7f49f7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f8107ec0 0x7f49f810a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56694/0 (socket says 172.21.15.29:56694) 2024-03-11T19:00:35.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.059+0000 7f49f7fff700 1 -- 172.21.15.29:0/1290692133 learned_addr learned my addr 172.21.15.29:0/1290692133 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:35.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.059+0000 7f49f7fff700 1 -- 172.21.15.29:0/1290692133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f810a7f0 con 0x7f49f8107ec0 2024-03-11T19:00:35.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.060+0000 7f49f7fff700 1 --2- 172.21.15.29:0/1290692133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f8107ec0 0x7f49f810a2b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f49e80098a0 tx=0x7f49e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eafd4ce18baf963 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:35.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.060+0000 7f49f6ffd700 1 -- 172.21.15.29:0/1290692133 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f49e8004160 con 0x7f49f8107ec0 2024-03-11T19:00:35.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.060+0000 7f49f6ffd700 1 -- 172.21.15.29:0/1290692133 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f49e80042c0 con 0x7f49f8107ec0 2024-03-11T19:00:35.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.060+0000 7f49f6ffd700 1 -- 172.21.15.29:0/1290692133 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f49e80119b0 con 0x7f49f8107ec0 2024-03-11T19:00:35.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.060+0000 7f49fe685700 1 -- 172.21.15.29:0/1290692133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f8107ec0 msgr2=0x7f49f810a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:35.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.060+0000 7f49fe685700 1 --2- 172.21.15.29:0/1290692133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f8107ec0 0x7f49f810a2b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f49e80098a0 tx=0x7f49e8009bb0 comp rx=0 tx=0).stop 2024-03-11T19:00:35.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.060+0000 7f49fe685700 1 -- 172.21.15.29:0/1290692133 shutdown_connections 2024-03-11T19:00:35.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.060+0000 7f49fe685700 1 --2- 172.21.15.29:0/1290692133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f8107ec0 0x7f49f810a2b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:35.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.060+0000 7f49fe685700 1 -- 172.21.15.29:0/1290692133 >> 172.21.15.29:0/1290692133 conn(0x7f49f8101930 msgr2=0x7f49f8103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:35.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.060+0000 7f49fe685700 1 -- 172.21.15.29:0/1290692133 shutdown_connections 2024-03-11T19:00:35.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.060+0000 7f49fe685700 1 -- 172.21.15.29:0/1290692133 wait complete. 2024-03-11T19:00:35.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.061+0000 7f49fe685700 1 Processor -- start 2024-03-11T19:00:35.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.061+0000 7f49fe685700 1 -- start start 2024-03-11T19:00:35.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.061+0000 7f49fe685700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f8107ec0 0x7f49f819b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:35.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.061+0000 7f49fe685700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f819b680 con 0x7f49f8107ec0 2024-03-11T19:00:35.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.061+0000 7f49f7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f8107ec0 0x7f49f819b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:35.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.061+0000 7f49f7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f8107ec0 0x7f49f819b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56696/0 (socket says 172.21.15.29:56696) 2024-03-11T19:00:35.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.062+0000 7f49f7fff700 1 -- 172.21.15.29:0/3647288671 learned_addr learned my addr 172.21.15.29:0/3647288671 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:35.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.062+0000 7f49f7fff700 1 -- 172.21.15.29:0/3647288671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e8009550 con 0x7f49f8107ec0 2024-03-11T19:00:35.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.062+0000 7f49f7fff700 1 --2- 172.21.15.29:0/3647288671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f8107ec0 0x7f49f819b140 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f49e8011980 tx=0x7f49e8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:35.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.062+0000 7f49f57fa700 1 -- 172.21.15.29:0/3647288671 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f49e80039c0 con 0x7f49f8107ec0 2024-03-11T19:00:35.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.062+0000 7f49f57fa700 1 -- 172.21.15.29:0/3647288671 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f49e8003b20 con 0x7f49f8107ec0 2024-03-11T19:00:35.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.062+0000 7f49fe685700 1 -- 172.21.15.29:0/3647288671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f819b880 con 0x7f49f8107ec0 2024-03-11T19:00:35.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.062+0000 7f49f57fa700 1 -- 172.21.15.29:0/3647288671 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f49e8019650 con 0x7f49f8107ec0 2024-03-11T19:00:35.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.062+0000 7f49fe685700 1 -- 172.21.15.29:0/3647288671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f819bd20 con 0x7f49f8107ec0 2024-03-11T19:00:35.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.063+0000 7f49f57fa700 1 -- 172.21.15.29:0/3647288671 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f49e8019910 con 0x7f49f8107ec0 2024-03-11T19:00:35.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.063+0000 7f49fe685700 1 -- 172.21.15.29:0/3647288671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f810b630 con 0x7f49f8107ec0 2024-03-11T19:00:35.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.063+0000 7f49f57fa700 1 --2- 172.21.15.29:0/3647288671 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f49e0038cb0 0x7f49e003b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:35.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.063+0000 7f49f57fa700 1 -- 172.21.15.29:0/3647288671 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f49e804d040 con 0x7f49f8107ec0 2024-03-11T19:00:35.286 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.063+0000 7f49f77fe700 1 --2- 172.21.15.29:0/3647288671 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f49e0038cb0 0x7f49e003b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:35.286 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.064+0000 7f49f77fe700 1 --2- 172.21.15.29:0/3647288671 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f49e0038cb0 0x7f49e003b170 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f49ec006fd0 tx=0x7f49ec006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:35.286 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.068+0000 7f49f57fa700 1 -- 172.21.15.29:0/3647288671 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f49e802b340 con 0x7f49f8107ec0 2024-03-11T19:00:35.286 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.281+0000 7f49fe685700 1 -- 172.21.15.29:0/3647288671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f49f819e660 con 0x7f49f8107ec0 2024-03-11T19:00:35.286 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.283+0000 7f49f57fa700 1 -- 172.21.15.29:0/3647288671 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f49e8018be0 con 0x7f49f8107ec0 2024-03-11T19:00:35.286 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.286+0000 7f49fe685700 1 -- 172.21.15.29:0/3647288671 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f49e0038cb0 msgr2=0x7f49e003b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:35.286 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.286+0000 7f49fe685700 1 --2- 172.21.15.29:0/3647288671 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f49e0038cb0 0x7f49e003b170 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f49ec006fd0 tx=0x7f49ec006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:35.286 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.286+0000 7f49fe685700 1 -- 172.21.15.29:0/3647288671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f8107ec0 msgr2=0x7f49f819b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:35.286 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.286+0000 7f49fe685700 1 --2- 172.21.15.29:0/3647288671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f8107ec0 0x7f49f819b140 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f49e8011980 tx=0x7f49e8011a60 comp rx=0 tx=0).stop 2024-03-11T19:00:35.286 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.286+0000 7f49fe685700 1 -- 172.21.15.29:0/3647288671 shutdown_connections 2024-03-11T19:00:35.286 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.286+0000 7f49fe685700 1 --2- 172.21.15.29:0/3647288671 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f49e0038cb0 0x7f49e003b170 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:35.286 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.286+0000 7f49fe685700 1 --2- 172.21.15.29:0/3647288671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f8107ec0 0x7f49f819b140 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:35.286 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.286+0000 7f49fe685700 1 -- 172.21.15.29:0/3647288671 >> 172.21.15.29:0/3647288671 conn(0x7f49f8101930 msgr2=0x7f49f8102440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:35.286 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.286+0000 7f49fe685700 1 -- 172.21.15.29:0/3647288671 shutdown_connections 2024-03-11T19:00:35.286 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:34.286+0000 7f49fe685700 1 -- 172.21.15.29:0/3647288671 wait complete. 2024-03-11T19:00:35.287 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling the dashboard module... 2024-03-11T19:00:35.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:35 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3647288671' entity='client.admin' 2024-03-11T19:00:36.775 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:36 smithi029 ceph-mon[32439]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:00:36.775 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:36 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/731993295' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-11T19:00:36.775 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:36 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: ignoring --setuser ceph since I am not root 2024-03-11T19:00:36.775 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:36 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: ignoring --setgroup ceph since I am not root 2024-03-11T19:00:37.078 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.738+0000 7f319c5c3700 1 Processor -- start 2024-03-11T19:00:37.079 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.738+0000 7f319c5c3700 1 -- start start 2024-03-11T19:00:37.079 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.738+0000 7f319c5c3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194107460 0x7f3194107880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:37.079 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.738+0000 7f319c5c3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3194107e50 con 0x7f3194107460 2024-03-11T19:00:37.079 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.738+0000 7f319a35f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194107460 0x7f3194107880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:37.079 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.738+0000 7f319a35f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194107460 0x7f3194107880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45586/0 (socket says 172.21.15.29:45586) 2024-03-11T19:00:37.079 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.738+0000 7f319a35f700 1 -- 172.21.15.29:0/3351162512 learned_addr learned my addr 172.21.15.29:0/3351162512 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:37.079 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.738+0000 7f319a35f700 1 -- 172.21.15.29:0/3351162512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3194108610 con 0x7f3194107460 2024-03-11T19:00:37.079 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.739+0000 7f319a35f700 1 --2- 172.21.15.29:0/3351162512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194107460 0x7f3194107880 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f31900098a0 tx=0x7f3190009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb98dca8731ccfd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:37.079 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.739+0000 7f319935d700 1 -- 172.21.15.29:0/3351162512 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3190004160 con 0x7f3194107460 2024-03-11T19:00:37.079 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.739+0000 7f319935d700 1 -- 172.21.15.29:0/3351162512 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f31900042c0 con 0x7f3194107460 2024-03-11T19:00:37.079 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.739+0000 7f319935d700 1 -- 172.21.15.29:0/3351162512 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f31900119b0 con 0x7f3194107460 2024-03-11T19:00:37.079 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.739+0000 7f319c5c3700 1 -- 172.21.15.29:0/3351162512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194107460 msgr2=0x7f3194107880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:37.079 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.739+0000 7f319c5c3700 1 --2- 172.21.15.29:0/3351162512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194107460 0x7f3194107880 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f31900098a0 tx=0x7f3190009bb0 comp rx=0 tx=0).stop 2024-03-11T19:00:37.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.739+0000 7f319c5c3700 1 -- 172.21.15.29:0/3351162512 shutdown_connections 2024-03-11T19:00:37.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.739+0000 7f319c5c3700 1 --2- 172.21.15.29:0/3351162512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194107460 0x7f3194107880 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:37.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.739+0000 7f319c5c3700 1 -- 172.21.15.29:0/3351162512 >> 172.21.15.29:0/3351162512 conn(0x7f3194102a40 msgr2=0x7f3194104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:37.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.740+0000 7f319c5c3700 1 -- 172.21.15.29:0/3351162512 shutdown_connections 2024-03-11T19:00:37.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.740+0000 7f319c5c3700 1 -- 172.21.15.29:0/3351162512 wait complete. 2024-03-11T19:00:37.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.740+0000 7f319c5c3700 1 Processor -- start 2024-03-11T19:00:37.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.740+0000 7f319c5c3700 1 -- start start 2024-03-11T19:00:37.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.740+0000 7f319c5c3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194107460 0x7f319419b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:37.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.740+0000 7f319c5c3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319419b6a0 con 0x7f3194107460 2024-03-11T19:00:37.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.741+0000 7f319a35f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194107460 0x7f319419b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:37.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.741+0000 7f319a35f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194107460 0x7f319419b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45600/0 (socket says 172.21.15.29:45600) 2024-03-11T19:00:37.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.741+0000 7f319a35f700 1 -- 172.21.15.29:0/731993295 learned_addr learned my addr 172.21.15.29:0/731993295 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:37.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.741+0000 7f319a35f700 1 -- 172.21.15.29:0/731993295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3190009550 con 0x7f3194107460 2024-03-11T19:00:37.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.741+0000 7f319a35f700 1 --2- 172.21.15.29:0/731993295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194107460 0x7f319419b160 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3190011670 tx=0x7f3190011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:37.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.741+0000 7f31877fe700 1 -- 172.21.15.29:0/731993295 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f31900038b0 con 0x7f3194107460 2024-03-11T19:00:37.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.741+0000 7f319c5c3700 1 -- 172.21.15.29:0/731993295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f319419b8a0 con 0x7f3194107460 2024-03-11T19:00:37.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.741+0000 7f31877fe700 1 -- 172.21.15.29:0/731993295 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3190003a10 con 0x7f3194107460 2024-03-11T19:00:37.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.741+0000 7f31877fe700 1 -- 172.21.15.29:0/731993295 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3190019530 con 0x7f3194107460 2024-03-11T19:00:37.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.741+0000 7f319c5c3700 1 -- 172.21.15.29:0/731993295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319419bd40 con 0x7f3194107460 2024-03-11T19:00:37.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.742+0000 7f31877fe700 1 -- 172.21.15.29:0/731993295 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f3190019830 con 0x7f3194107460 2024-03-11T19:00:37.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.742+0000 7f31877fe700 1 --2- 172.21.15.29:0/731993295 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f3180038c60 0x7f318003b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:37.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.742+0000 7f319c5c3700 1 -- 172.21.15.29:0/731993295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f319410b840 con 0x7f3194107460 2024-03-11T19:00:37.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.742+0000 7f31877fe700 1 -- 172.21.15.29:0/731993295 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f319004cea0 con 0x7f3194107460 2024-03-11T19:00:37.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.742+0000 7f3199b5e700 1 --2- 172.21.15.29:0/731993295 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f3180038c60 0x7f318003b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:37.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.743+0000 7f3199b5e700 1 --2- 172.21.15.29:0/731993295 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f3180038c60 0x7f318003b120 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3188006fd0 tx=0x7f3188006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:37.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:35.747+0000 7f31877fe700 1 -- 172.21.15.29:0/731993295 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3190018530 con 0x7f3194107460 2024-03-11T19:00:37.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:36.011+0000 7f319c5c3700 1 -- 172.21.15.29:0/731993295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f319419e620 con 0x7f3194107460 2024-03-11T19:00:37.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:36.457+0000 7f31877fe700 1 -- 172.21.15.29:0/731993295 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f3190029ae0 con 0x7f3194107460 2024-03-11T19:00:37.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:36.459+0000 7f31877fe700 1 -- 172.21.15.29:0/731993295 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 9) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f319001a430 con 0x7f3194107460 2024-03-11T19:00:37.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:36.461+0000 7f319c5c3700 1 -- 172.21.15.29:0/731993295 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f3180038c60 msgr2=0x7f318003b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:37.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:36.461+0000 7f319c5c3700 1 --2- 172.21.15.29:0/731993295 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f3180038c60 0x7f318003b120 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3188006fd0 tx=0x7f3188006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:37.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:36.461+0000 7f319c5c3700 1 -- 172.21.15.29:0/731993295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194107460 msgr2=0x7f319419b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:37.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:36.461+0000 7f319c5c3700 1 --2- 172.21.15.29:0/731993295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194107460 0x7f319419b160 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3190011670 tx=0x7f3190011a30 comp rx=0 tx=0).stop 2024-03-11T19:00:37.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:36.461+0000 7f319c5c3700 1 -- 172.21.15.29:0/731993295 shutdown_connections 2024-03-11T19:00:37.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:36.461+0000 7f319c5c3700 1 --2- 172.21.15.29:0/731993295 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f3180038c60 0x7f318003b120 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:37.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:36.461+0000 7f319c5c3700 1 --2- 172.21.15.29:0/731993295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194107460 0x7f319419b160 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:37.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:36.461+0000 7f319c5c3700 1 -- 172.21.15.29:0/731993295 >> 172.21.15.29:0/731993295 conn(0x7f3194102a40 msgr2=0x7f3194103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:37.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:36.461+0000 7f319c5c3700 1 -- 172.21.15.29:0/731993295 shutdown_connections 2024-03-11T19:00:37.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:36.461+0000 7f319c5c3700 1 -- 172.21.15.29:0/731993295 wait complete. 2024-03-11T19:00:37.304 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:37 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:37.048+0000 7f36c14a4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T19:00:37.304 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:37 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:37.156+0000 7f36c14a4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T19:00:37.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:37 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/731993295' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-11T19:00:37.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:37 smithi029 ceph-mon[32439]: mgrmap e9: x(active, since 21s) 2024-03-11T19:00:38.072 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:37 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:37.827+0000 7f36c14a4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T19:00:38.072 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:37 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:37.922+0000 7f36c14a4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T19:00:38.072 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:38 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:38.069+0000 7f36c14a4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T19:00:38.376 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:38 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:38.169+0000 7f36c14a4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T19:00:38.376 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:38 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:38.254+0000 7f36c14a4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T19:00:38.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-11T19:00:38.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-11T19:00:38.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T19:00:38.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-11T19:00:38.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T19:00:38.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-11T19:00:38.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.490+0000 7fba3319f700 1 Processor -- start 2024-03-11T19:00:38.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.490+0000 7fba3319f700 1 -- start start 2024-03-11T19:00:38.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.490+0000 7fba3319f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba2c107ef0 0x7fba2c10a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:38.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.490+0000 7fba3319f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2c07ae50 con 0x7fba2c107ef0 2024-03-11T19:00:38.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.490+0000 7fba30f3b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba2c107ef0 0x7fba2c10a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:38.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.490+0000 7fba30f3b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba2c107ef0 0x7fba2c10a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45622/0 (socket says 172.21.15.29:45622) 2024-03-11T19:00:38.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.490+0000 7fba30f3b700 1 -- 172.21.15.29:0/4228499302 learned_addr learned my addr 172.21.15.29:0/4228499302 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:38.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.491+0000 7fba30f3b700 1 -- 172.21.15.29:0/4228499302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2c10a820 con 0x7fba2c107ef0 2024-03-11T19:00:38.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.491+0000 7fba30f3b700 1 --2- 172.21.15.29:0/4228499302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba2c107ef0 0x7fba2c10a2e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fba200098a0 tx=0x7fba20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d7f2b99da14317d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:38.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.491+0000 7fba2b7fe700 1 -- 172.21.15.29:0/4228499302 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba20004160 con 0x7fba2c107ef0 2024-03-11T19:00:38.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.491+0000 7fba2b7fe700 1 -- 172.21.15.29:0/4228499302 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fba200042c0 con 0x7fba2c107ef0 2024-03-11T19:00:38.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.491+0000 7fba2b7fe700 1 -- 172.21.15.29:0/4228499302 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba200119b0 con 0x7fba2c107ef0 2024-03-11T19:00:38.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.491+0000 7fba3319f700 1 -- 172.21.15.29:0/4228499302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba2c107ef0 msgr2=0x7fba2c10a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:38.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.491+0000 7fba3319f700 1 --2- 172.21.15.29:0/4228499302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba2c107ef0 0x7fba2c10a2e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fba200098a0 tx=0x7fba20009bb0 comp rx=0 tx=0).stop 2024-03-11T19:00:38.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.492+0000 7fba3319f700 1 -- 172.21.15.29:0/4228499302 shutdown_connections 2024-03-11T19:00:38.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.492+0000 7fba3319f700 1 --2- 172.21.15.29:0/4228499302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba2c107ef0 0x7fba2c10a2e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:38.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.492+0000 7fba3319f700 1 -- 172.21.15.29:0/4228499302 >> 172.21.15.29:0/4228499302 conn(0x7fba2c1018a0 msgr2=0x7fba2c103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:38.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.492+0000 7fba3319f700 1 -- 172.21.15.29:0/4228499302 shutdown_connections 2024-03-11T19:00:38.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.492+0000 7fba3319f700 1 -- 172.21.15.29:0/4228499302 wait complete. 2024-03-11T19:00:38.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.492+0000 7fba3319f700 1 Processor -- start 2024-03-11T19:00:38.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.492+0000 7fba3319f700 1 -- start start 2024-03-11T19:00:38.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.492+0000 7fba3319f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba2c107ef0 0x7fba2c19b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:38.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.492+0000 7fba3319f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2c19b6f0 con 0x7fba2c107ef0 2024-03-11T19:00:38.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.493+0000 7fba30f3b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba2c107ef0 0x7fba2c19b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:38.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.493+0000 7fba30f3b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba2c107ef0 0x7fba2c19b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45634/0 (socket says 172.21.15.29:45634) 2024-03-11T19:00:38.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.493+0000 7fba30f3b700 1 -- 172.21.15.29:0/954214034 learned_addr learned my addr 172.21.15.29:0/954214034 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:38.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.493+0000 7fba30f3b700 1 -- 172.21.15.29:0/954214034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba20009550 con 0x7fba2c107ef0 2024-03-11T19:00:38.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.493+0000 7fba30f3b700 1 --2- 172.21.15.29:0/954214034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba2c107ef0 0x7fba2c19b1b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fba20011980 tx=0x7fba20011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:38.387 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.493+0000 7fba29ffb700 1 -- 172.21.15.29:0/954214034 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba20003970 con 0x7fba2c107ef0 2024-03-11T19:00:38.387 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.493+0000 7fba29ffb700 1 -- 172.21.15.29:0/954214034 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fba20003ad0 con 0x7fba2c107ef0 2024-03-11T19:00:38.387 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.493+0000 7fba3319f700 1 -- 172.21.15.29:0/954214034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba2c19b8f0 con 0x7fba2c107ef0 2024-03-11T19:00:38.387 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.493+0000 7fba29ffb700 1 -- 172.21.15.29:0/954214034 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba200195e0 con 0x7fba2c107ef0 2024-03-11T19:00:38.387 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.493+0000 7fba3319f700 1 -- 172.21.15.29:0/954214034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba2c19bd90 con 0x7fba2c107ef0 2024-03-11T19:00:38.387 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.494+0000 7fba29ffb700 1 -- 172.21.15.29:0/954214034 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 45782+0+0 (secure 0 0 0) 0x7fba200197d0 con 0x7fba2c107ef0 2024-03-11T19:00:38.387 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.494+0000 7fba29ffb700 1 --2- 172.21.15.29:0/954214034 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fba1c038d10 0x7fba1c03b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:38.387 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.494+0000 7fba29ffb700 1 -- 172.21.15.29:0/954214034 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fba2004cce0 con 0x7fba2c107ef0 2024-03-11T19:00:38.387 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.494+0000 7fba3319f700 1 -- 172.21.15.29:0/954214034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba2c10b600 con 0x7fba2c107ef0 2024-03-11T19:00:38.387 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.494+0000 7fba2bfff700 1 -- 172.21.15.29:0/954214034 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fba1c038d10 msgr2=0x7fba1c03b1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3585665950 2024-03-11T19:00:38.387 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.494+0000 7fba2bfff700 1 --2- 172.21.15.29:0/954214034 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fba1c038d10 0x7fba1c03b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T19:00:38.387 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.499+0000 7fba29ffb700 1 -- 172.21.15.29:0/954214034 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fba20018350 con 0x7fba2c107ef0 2024-03-11T19:00:38.387 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.695+0000 7fba2bfff700 1 -- 172.21.15.29:0/954214034 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fba1c038d10 msgr2=0x7fba1c03b1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3585665950 2024-03-11T19:00:38.387 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.695+0000 7fba2bfff700 1 --2- 172.21.15.29:0/954214034 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fba1c038d10 0x7fba1c03b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T19:00:38.387 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.760+0000 7fba3319f700 1 -- 172.21.15.29:0/954214034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fba2c19e9f0 con 0x7fba2c107ef0 2024-03-11T19:00:38.388 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.761+0000 7fba29ffb700 1 -- 172.21.15.29:0/954214034 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fba2004b070 con 0x7fba2c107ef0 2024-03-11T19:00:38.388 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.763+0000 7fba3319f700 1 -- 172.21.15.29:0/954214034 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fba1c038d10 msgr2=0x7fba1c03b1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:00:38.388 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.763+0000 7fba3319f700 1 --2- 172.21.15.29:0/954214034 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fba1c038d10 0x7fba1c03b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:38.388 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.763+0000 7fba3319f700 1 -- 172.21.15.29:0/954214034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba2c107ef0 msgr2=0x7fba2c19b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:38.388 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.763+0000 7fba3319f700 1 --2- 172.21.15.29:0/954214034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba2c107ef0 0x7fba2c19b1b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fba20011980 tx=0x7fba20011a60 comp rx=0 tx=0).stop 2024-03-11T19:00:38.388 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.763+0000 7fba3319f700 1 -- 172.21.15.29:0/954214034 shutdown_connections 2024-03-11T19:00:38.388 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.763+0000 7fba3319f700 1 --2- 172.21.15.29:0/954214034 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7fba1c038d10 0x7fba1c03b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:38.388 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.763+0000 7fba3319f700 1 --2- 172.21.15.29:0/954214034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba2c107ef0 0x7fba2c19b1b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:38.388 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.763+0000 7fba3319f700 1 -- 172.21.15.29:0/954214034 >> 172.21.15.29:0/954214034 conn(0x7fba2c1018a0 msgr2=0x7fba2c1024e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:38.388 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.763+0000 7fba3319f700 1 -- 172.21.15.29:0/954214034 shutdown_connections 2024-03-11T19:00:38.388 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:37.763+0000 7fba3319f700 1 -- 172.21.15.29:0/954214034 wait complete. 2024-03-11T19:00:38.388 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for the mgr to restart... 2024-03-11T19:00:38.388 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr epoch 9... 2024-03-11T19:00:38.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:38 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/954214034' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T19:00:38.628 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:38 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:38.529+0000 7f36c14a4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T19:00:38.905 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:38 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:38.644+0000 7f36c14a4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T19:00:38.905 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:38 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:38.732+0000 7f36c14a4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T19:00:38.906 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:38 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:38.832+0000 7f36c14a4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T19:00:39.405 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:39 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:39.013+0000 7f36c14a4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T19:00:40.259 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:40 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:40.006+0000 7f36c14a4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T19:00:40.259 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:40 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:40.140+0000 7f36c14a4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T19:00:40.573 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:40 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:40.477+0000 7f36c14a4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T19:00:40.574 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:40 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:40.571+0000 7f36c14a4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T19:00:40.905 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:40 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:40.771+0000 7f36c14a4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:00:41.255 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:40 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:40.961+0000 7f36c14a4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T19:00:41.517 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:41 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:41.252+0000 7f36c14a4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:00:41.517 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:41 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:41.514+0000 7f36c14a4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T19:00:41.860 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:41 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:41.757+0000 7f36c14a4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T19:00:42.905 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:42 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:42.589+0000 7f36c14a4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T19:00:43.514 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:43 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:43.183+0000 7f36c14a4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T19:00:43.514 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:43 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:43.286+0000 7f36c14a4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T19:00:43.514 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:00:43 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:00:43.511+0000 7f36c14a4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T19:00:43.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:43 smithi029 ceph-mon[32439]: Active manager daemon x restarted 2024-03-11T19:00:43.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:43 smithi029 ceph-mon[32439]: Activating manager daemon x 2024-03-11T19:00:43.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:43 smithi029 ceph-mon[32439]: osdmap e3: 0 total, 0 up, 0 in 2024-03-11T19:00:43.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:43 smithi029 ceph-mon[32439]: mgrmap e10: x(active, starting, since 0.00321284s) 2024-03-11T19:00:43.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:00:43.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T19:00:43.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T19:00:43.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T19:00:43.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T19:00:43.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:43 smithi029 ceph-mon[32439]: Manager daemon x is now available 2024-03-11T19:00:43.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:00:43.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:00:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T19:00:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T19:00:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:44 smithi029 ceph-mon[32439]: [11/Mar/2024:19:00:44] ENGINE Bus STARTING 2024-03-11T19:00:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:44 smithi029 ceph-mon[32439]: [11/Mar/2024:19:00:44] ENGINE Serving on http://172.21.15.29:8765 2024-03-11T19:00:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:44 smithi029 ceph-mon[32439]: [11/Mar/2024:19:00:44] ENGINE Serving on https://172.21.15.29:7150 2024-03-11T19:00:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:44 smithi029 ceph-mon[32439]: [11/Mar/2024:19:00:44] ENGINE Bus STARTED 2024-03-11T19:00:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:44 smithi029 ceph-mon[32439]: mgrmap e11: x(active, since 1.00558s) 2024-03-11T19:00:45.441 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-11T19:00:45.441 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-11T19:00:45.441 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T19:00:45.441 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-11T19:00:45.441 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.849+0000 7f046c764700 1 Processor -- start 2024-03-11T19:00:45.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.849+0000 7f046c764700 1 -- start start 2024-03-11T19:00:45.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.849+0000 7f046c764700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0464107f30 0x7f046410a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:45.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.849+0000 7f046c764700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046407ae30 con 0x7f0464107f30 2024-03-11T19:00:45.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.849+0000 7f046a500700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0464107f30 0x7f046410a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:45.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.849+0000 7f046a500700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0464107f30 0x7f046410a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45644/0 (socket says 172.21.15.29:45644) 2024-03-11T19:00:45.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.849+0000 7f046a500700 1 -- 172.21.15.29:0/3694150099 learned_addr learned my addr 172.21.15.29:0/3694150099 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:45.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.849+0000 7f046a500700 1 -- 172.21.15.29:0/3694150099 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046410a860 con 0x7f0464107f30 2024-03-11T19:00:45.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.850+0000 7f046a500700 1 --2- 172.21.15.29:0/3694150099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0464107f30 0x7f046410a320 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f04600098a0 tx=0x7f0460009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca8518eea67332a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:45.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.850+0000 7f04694fe700 1 -- 172.21.15.29:0/3694150099 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0460004160 con 0x7f0464107f30 2024-03-11T19:00:45.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.850+0000 7f04694fe700 1 -- 172.21.15.29:0/3694150099 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f04600042c0 con 0x7f0464107f30 2024-03-11T19:00:45.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.850+0000 7f04694fe700 1 -- 172.21.15.29:0/3694150099 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04600119b0 con 0x7f0464107f30 2024-03-11T19:00:45.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.850+0000 7f046c764700 1 -- 172.21.15.29:0/3694150099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0464107f30 msgr2=0x7f046410a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:45.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.850+0000 7f046c764700 1 --2- 172.21.15.29:0/3694150099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0464107f30 0x7f046410a320 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f04600098a0 tx=0x7f0460009bb0 comp rx=0 tx=0).stop 2024-03-11T19:00:45.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.851+0000 7f046c764700 1 -- 172.21.15.29:0/3694150099 shutdown_connections 2024-03-11T19:00:45.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.851+0000 7f046c764700 1 --2- 172.21.15.29:0/3694150099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0464107f30 0x7f046410a320 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:45.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.851+0000 7f046c764700 1 -- 172.21.15.29:0/3694150099 >> 172.21.15.29:0/3694150099 conn(0x7f0464101980 msgr2=0x7f0464103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:45.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.851+0000 7f046c764700 1 -- 172.21.15.29:0/3694150099 shutdown_connections 2024-03-11T19:00:45.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.851+0000 7f046c764700 1 -- 172.21.15.29:0/3694150099 wait complete. 2024-03-11T19:00:45.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.851+0000 7f046c764700 1 Processor -- start 2024-03-11T19:00:45.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.851+0000 7f046c764700 1 -- start start 2024-03-11T19:00:45.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.851+0000 7f046c764700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0464107f30 0x7f046419b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:45.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.851+0000 7f046c764700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046419b680 con 0x7f0464107f30 2024-03-11T19:00:45.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.852+0000 7f046a500700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0464107f30 0x7f046419b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:45.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.852+0000 7f046a500700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0464107f30 0x7f046419b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45646/0 (socket says 172.21.15.29:45646) 2024-03-11T19:00:45.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.852+0000 7f046a500700 1 -- 172.21.15.29:0/3130234581 learned_addr learned my addr 172.21.15.29:0/3130234581 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:45.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.852+0000 7f046a500700 1 -- 172.21.15.29:0/3130234581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0460009550 con 0x7f0464107f30 2024-03-11T19:00:45.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.852+0000 7f046a500700 1 --2- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0464107f30 0x7f046419b140 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0460011980 tx=0x7f0460011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:45.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.852+0000 7f045b7fe700 1 -- 172.21.15.29:0/3130234581 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04600039c0 con 0x7f0464107f30 2024-03-11T19:00:45.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.852+0000 7f045b7fe700 1 -- 172.21.15.29:0/3130234581 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0460003b20 con 0x7f0464107f30 2024-03-11T19:00:45.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.852+0000 7f046c764700 1 -- 172.21.15.29:0/3130234581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f046419b880 con 0x7f0464107f30 2024-03-11T19:00:45.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.852+0000 7f045b7fe700 1 -- 172.21.15.29:0/3130234581 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0460019650 con 0x7f0464107f30 2024-03-11T19:00:45.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.852+0000 7f046c764700 1 -- 172.21.15.29:0/3130234581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f046419bd20 con 0x7f0464107f30 2024-03-11T19:00:45.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.853+0000 7f045b7fe700 1 -- 172.21.15.29:0/3130234581 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f0460019910 con 0x7f0464107f30 2024-03-11T19:00:45.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.853+0000 7f045b7fe700 1 --2- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f0450038d00 0x7f045003b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:45.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.853+0000 7f045b7fe700 1 -- 172.21.15.29:0/3130234581 --> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f045003b8d0 con 0x7f0450038d00 2024-03-11T19:00:45.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.853+0000 7f045b7fe700 1 -- 172.21.15.29:0/3130234581 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f046004d050 con 0x7f0464107f30 2024-03-11T19:00:45.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.853+0000 7f0469cff700 1 -- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f0450038d00 msgr2=0x7f045003b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3585665950 2024-03-11T19:00:45.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:38.853+0000 7f0469cff700 1 --2- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f0450038d00 0x7f045003b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T19:00:45.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:39.054+0000 7f0469cff700 1 -- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f0450038d00 msgr2=0x7f045003b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3585665950 2024-03-11T19:00:45.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:39.054+0000 7f0469cff700 1 --2- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f0450038d00 0x7f045003b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T19:00:45.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:39.455+0000 7f0469cff700 1 -- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f0450038d00 msgr2=0x7f045003b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3585665950 2024-03-11T19:00:45.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:39.455+0000 7f0469cff700 1 --2- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f0450038d00 0x7f045003b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-11T19:00:45.445 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:40.256+0000 7f0469cff700 1 -- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f0450038d00 msgr2=0x7f045003b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3585665950 2024-03-11T19:00:45.445 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:40.256+0000 7f0469cff700 1 --2- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f0450038d00 0x7f045003b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-11T19:00:45.445 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:41.857+0000 7f0469cff700 1 -- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f0450038d00 msgr2=0x7f045003b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/3585665950 2024-03-11T19:00:45.445 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:41.857+0000 7f0469cff700 1 --2- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f0450038d00 0x7f045003b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-11T19:00:45.445 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:43.526+0000 7f045b7fe700 1 -- 172.21.15.29:0/3130234581 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mgrmap(e 10) v1 ==== 45549+0+0 (secure 0 0 0) 0x7f046002da00 con 0x7f0464107f30 2024-03-11T19:00:45.445 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:43.526+0000 7f045b7fe700 1 -- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f0450038d00 msgr2=0x7f045003b1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:00:45.445 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:43.526+0000 7f045b7fe700 1 --2- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f0450038d00 0x7f045003b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:45.445 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:44.527+0000 7f045b7fe700 1 -- 172.21.15.29:0/3130234581 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 11) v1 ==== 45676+0+0 (secure 0 0 0) 0x7f046004ead0 con 0x7f0464107f30 2024-03-11T19:00:45.445 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:44.527+0000 7f045b7fe700 1 --2- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f045003c780 0x7f045003eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:45.445 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:44.527+0000 7f045b7fe700 1 -- 172.21.15.29:0/3130234581 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f045003b8d0 con 0x7f045003c780 2024-03-11T19:00:45.445 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:44.528+0000 7f0469cff700 1 --2- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f045003c780 0x7f045003eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:45.445 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:44.528+0000 7f0469cff700 1 --2- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f045003c780 0x7f045003eb70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0454003a10 tx=0x7f04540092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:45.445 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:44.529+0000 7f045b7fe700 1 -- 172.21.15.29:0/3130234581 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f045003b8d0 con 0x7f045003c780 2024-03-11T19:00:45.445 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:44.536+0000 7f046c764700 1 -- 172.21.15.29:0/3130234581 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f04640632a0 con 0x7f045003c780 2024-03-11T19:00:45.445 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:44.536+0000 7f045b7fe700 1 -- 172.21.15.29:0/3130234581 <== mgr.14150 v2:172.21.15.29:6800/1587871898 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f04640632a0 con 0x7f045003c780 2024-03-11T19:00:45.446 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:44.536+0000 7f046c764700 1 -- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f045003c780 msgr2=0x7f045003eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:45.446 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:44.536+0000 7f046c764700 1 --2- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f045003c780 0x7f045003eb70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0454003a10 tx=0x7f04540092b0 comp rx=0 tx=0).stop 2024-03-11T19:00:45.446 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:44.536+0000 7f046c764700 1 -- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0464107f30 msgr2=0x7f046419b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:45.446 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:44.536+0000 7f046c764700 1 --2- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0464107f30 0x7f046419b140 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0460011980 tx=0x7f0460011a60 comp rx=0 tx=0).stop 2024-03-11T19:00:45.446 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:44.536+0000 7f046c764700 1 -- 172.21.15.29:0/3130234581 shutdown_connections 2024-03-11T19:00:45.446 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:44.536+0000 7f046c764700 1 --2- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f045003c780 0x7f045003eb70 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:45.446 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:44.536+0000 7f046c764700 1 --2- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0464107f30 0x7f046419b140 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:45.446 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:44.536+0000 7f046c764700 1 --2- 172.21.15.29:0/3130234581 >> [v2:172.21.15.29:6800/3585665950,v1:172.21.15.29:6801/3585665950] conn(0x7f0450038d00 0x7f045003b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:45.446 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:44.536+0000 7f046c764700 1 -- 172.21.15.29:0/3130234581 >> 172.21.15.29:0/3130234581 conn(0x7f0464101980 msgr2=0x7f04641024e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:45.447 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:44.536+0000 7f046c764700 1 -- 172.21.15.29:0/3130234581 shutdown_connections 2024-03-11T19:00:45.447 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:44.536+0000 7f046c764700 1 -- 172.21.15.29:0/3130234581 wait complete. 2024-03-11T19:00:45.447 INFO:teuthology.orchestra.run.smithi029.stdout:mgr epoch 9 is available 2024-03-11T19:00:45.447 INFO:teuthology.orchestra.run.smithi029.stdout:Generating a dashboard self-signed certificate... 2024-03-11T19:00:46.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:46 smithi029 ceph-mon[32439]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T19:00:46.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:46 smithi029 ceph-mon[32439]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T19:00:46.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:46 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:00:46.754 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-11T19:00:46.754 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.876+0000 7fddd1535700 1 Processor -- start 2024-03-11T19:00:46.754 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.876+0000 7fddd1535700 1 -- start start 2024-03-11T19:00:46.754 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.876+0000 7fddd1535700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc107520 0x7fddcc107940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:46.754 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.877+0000 7fddd1535700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc107f10 con 0x7fddcc107520 2024-03-11T19:00:46.754 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.877+0000 7fddcaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc107520 0x7fddcc107940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:46.754 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.877+0000 7fddcaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc107520 0x7fddcc107940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49860/0 (socket says 172.21.15.29:49860) 2024-03-11T19:00:46.754 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.877+0000 7fddcaffd700 1 -- 172.21.15.29:0/3994918019 learned_addr learned my addr 172.21.15.29:0/3994918019 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:46.754 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.877+0000 7fddcaffd700 1 -- 172.21.15.29:0/3994918019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddcc1086d0 con 0x7fddcc107520 2024-03-11T19:00:46.754 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.877+0000 7fddcaffd700 1 --2- 172.21.15.29:0/3994918019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc107520 0x7fddcc107940 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fddbc0098a0 tx=0x7fddbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56349a225615947f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:46.755 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.877+0000 7fddc9ffb700 1 -- 172.21.15.29:0/3994918019 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fddbc004160 con 0x7fddcc107520 2024-03-11T19:00:46.755 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.877+0000 7fddc9ffb700 1 -- 172.21.15.29:0/3994918019 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fddbc0042c0 con 0x7fddcc107520 2024-03-11T19:00:46.755 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.877+0000 7fddc9ffb700 1 -- 172.21.15.29:0/3994918019 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fddbc0119b0 con 0x7fddcc107520 2024-03-11T19:00:46.755 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.878+0000 7fddd1535700 1 -- 172.21.15.29:0/3994918019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc107520 msgr2=0x7fddcc107940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:46.755 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.878+0000 7fddd1535700 1 --2- 172.21.15.29:0/3994918019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc107520 0x7fddcc107940 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fddbc0098a0 tx=0x7fddbc009bb0 comp rx=0 tx=0).stop 2024-03-11T19:00:46.755 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.878+0000 7fddd1535700 1 -- 172.21.15.29:0/3994918019 shutdown_connections 2024-03-11T19:00:46.755 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.878+0000 7fddd1535700 1 --2- 172.21.15.29:0/3994918019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc107520 0x7fddcc107940 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:46.755 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.878+0000 7fddd1535700 1 -- 172.21.15.29:0/3994918019 >> 172.21.15.29:0/3994918019 conn(0x7fddcc102b40 msgr2=0x7fddcc104f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:46.755 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.878+0000 7fddd1535700 1 -- 172.21.15.29:0/3994918019 shutdown_connections 2024-03-11T19:00:46.755 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.878+0000 7fddd1535700 1 -- 172.21.15.29:0/3994918019 wait complete. 2024-03-11T19:00:46.755 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.878+0000 7fddd1535700 1 Processor -- start 2024-03-11T19:00:46.755 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.879+0000 7fddd1535700 1 -- start start 2024-03-11T19:00:46.755 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.879+0000 7fddd1535700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc107520 0x7fddcc19b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:46.755 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.879+0000 7fddd1535700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc19b710 con 0x7fddcc107520 2024-03-11T19:00:46.756 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.879+0000 7fddcaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc107520 0x7fddcc19b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:46.756 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.879+0000 7fddcaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc107520 0x7fddcc19b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49866/0 (socket says 172.21.15.29:49866) 2024-03-11T19:00:46.756 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.879+0000 7fddcaffd700 1 -- 172.21.15.29:0/3306387806 learned_addr learned my addr 172.21.15.29:0/3306387806 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:46.756 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.879+0000 7fddcaffd700 1 -- 172.21.15.29:0/3306387806 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddbc009550 con 0x7fddcc107520 2024-03-11T19:00:46.756 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.879+0000 7fddcaffd700 1 --2- 172.21.15.29:0/3306387806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc107520 0x7fddcc19b1d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fddbc011670 tx=0x7fddbc011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:46.756 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.879+0000 7fddb3fff700 1 -- 172.21.15.29:0/3306387806 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fddbc0038b0 con 0x7fddcc107520 2024-03-11T19:00:46.756 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.880+0000 7fddd1535700 1 -- 172.21.15.29:0/3306387806 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddcc19b910 con 0x7fddcc107520 2024-03-11T19:00:46.756 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.880+0000 7fddb3fff700 1 -- 172.21.15.29:0/3306387806 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fddbc003a10 con 0x7fddcc107520 2024-03-11T19:00:46.756 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.880+0000 7fddb3fff700 1 -- 172.21.15.29:0/3306387806 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fddbc019530 con 0x7fddcc107520 2024-03-11T19:00:46.756 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.880+0000 7fddd1535700 1 -- 172.21.15.29:0/3306387806 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddcc19bdb0 con 0x7fddcc107520 2024-03-11T19:00:46.756 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.880+0000 7fddb3fff700 1 -- 172.21.15.29:0/3306387806 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 45676+0+0 (secure 0 0 0) 0x7fddbc0197f0 con 0x7fddcc107520 2024-03-11T19:00:46.756 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.881+0000 7fddb3fff700 1 --2- 172.21.15.29:0/3306387806 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fddb4038ba0 0x7fddb403b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:46.756 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.881+0000 7fddb3fff700 1 -- 172.21.15.29:0/3306387806 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fddbc04ce50 con 0x7fddcc107520 2024-03-11T19:00:46.756 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.881+0000 7fddca7fc700 1 --2- 172.21.15.29:0/3306387806 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fddb4038ba0 0x7fddb403b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:46.756 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.881+0000 7fddd1535700 1 -- 172.21.15.29:0/3306387806 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddcc10b900 con 0x7fddcc107520 2024-03-11T19:00:46.757 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.881+0000 7fddca7fc700 1 --2- 172.21.15.29:0/3306387806 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fddb4038ba0 0x7fddb403b060 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fddc0006fd0 tx=0x7fddc0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:46.757 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:45.886+0000 7fddb3fff700 1 -- 172.21.15.29:0/3306387806 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fddbc07ad20 con 0x7fddcc107520 2024-03-11T19:00:46.757 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:46.045+0000 7fddb3fff700 1 -- 172.21.15.29:0/3306387806 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 12) v1 ==== 45782+0+0 (secure 0 0 0) 0x7fddbc019d20 con 0x7fddcc107520 2024-03-11T19:00:46.757 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:46.124+0000 7fddd1535700 1 -- 172.21.15.29:0/3306387806 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fddcc103d70 con 0x7fddb4038ba0 2024-03-11T19:00:46.757 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:46.221+0000 7fddb3fff700 1 -- 172.21.15.29:0/3306387806 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fddcc103d70 con 0x7fddb4038ba0 2024-03-11T19:00:46.757 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:46.223+0000 7fddd1535700 1 -- 172.21.15.29:0/3306387806 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fddb4038ba0 msgr2=0x7fddb403b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:46.757 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:46.223+0000 7fddd1535700 1 --2- 172.21.15.29:0/3306387806 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fddb4038ba0 0x7fddb403b060 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fddc0006fd0 tx=0x7fddc0006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:46.757 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:46.223+0000 7fddd1535700 1 -- 172.21.15.29:0/3306387806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc107520 msgr2=0x7fddcc19b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:46.757 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:46.223+0000 7fddd1535700 1 --2- 172.21.15.29:0/3306387806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc107520 0x7fddcc19b1d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fddbc011670 tx=0x7fddbc011a30 comp rx=0 tx=0).stop 2024-03-11T19:00:46.757 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:46.223+0000 7fddd1535700 1 -- 172.21.15.29:0/3306387806 shutdown_connections 2024-03-11T19:00:46.757 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:46.223+0000 7fddd1535700 1 --2- 172.21.15.29:0/3306387806 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fddb4038ba0 0x7fddb403b060 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:46.757 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:46.223+0000 7fddd1535700 1 --2- 172.21.15.29:0/3306387806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc107520 0x7fddcc19b1d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:46.757 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:46.223+0000 7fddd1535700 1 -- 172.21.15.29:0/3306387806 >> 172.21.15.29:0/3306387806 conn(0x7fddcc102b40 msgr2=0x7fddcc103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:46.757 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:46.224+0000 7fddd1535700 1 -- 172.21.15.29:0/3306387806 shutdown_connections 2024-03-11T19:00:46.758 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:46.224+0000 7fddd1535700 1 -- 172.21.15.29:0/3306387806 wait complete. 2024-03-11T19:00:46.758 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial admin user... 2024-03-11T19:00:47.153 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:47 smithi029 ceph-mon[32439]: mgrmap e12: x(active, since 2s) 2024-03-11T19:00:47.153 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:47 smithi029 ceph-mon[32439]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:00:47.153 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:00:47.153 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:00:47.153 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:00:48.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:48 smithi029 ceph-mon[32439]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:00:48.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:48 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:00:48.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$8O/16gwEIef4XskVNvh5tujoV/Bc2Y6D2e72bXDU7WoOJW4WRax.O", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710183647, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-11T19:00:48.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.256+0000 7fa9265a2700 1 Processor -- start 2024-03-11T19:00:48.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.256+0000 7fa9265a2700 1 -- start start 2024-03-11T19:00:48.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.257+0000 7fa9265a2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa920106370 0x7fa920106790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:48.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.257+0000 7fa9265a2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa920106d60 con 0x7fa920106370 2024-03-11T19:00:48.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.257+0000 7fa91ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa920106370 0x7fa920106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:48.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.257+0000 7fa91ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa920106370 0x7fa920106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49876/0 (socket says 172.21.15.29:49876) 2024-03-11T19:00:48.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.257+0000 7fa91ffff700 1 -- 172.21.15.29:0/930559978 learned_addr learned my addr 172.21.15.29:0/930559978 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:48.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.257+0000 7fa91ffff700 1 -- 172.21.15.29:0/930559978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa920107570 con 0x7fa920106370 2024-03-11T19:00:48.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.257+0000 7fa91ffff700 1 --2- 172.21.15.29:0/930559978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa920106370 0x7fa920106790 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa9140098a0 tx=0x7fa914009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ae3b8d98da6b7e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:48.417 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.257+0000 7fa91effd700 1 -- 172.21.15.29:0/930559978 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa914004160 con 0x7fa920106370 2024-03-11T19:00:48.417 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.257+0000 7fa91effd700 1 -- 172.21.15.29:0/930559978 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa9140042c0 con 0x7fa920106370 2024-03-11T19:00:48.417 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.257+0000 7fa91effd700 1 -- 172.21.15.29:0/930559978 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa9140119b0 con 0x7fa920106370 2024-03-11T19:00:48.417 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.258+0000 7fa9265a2700 1 -- 172.21.15.29:0/930559978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa920106370 msgr2=0x7fa920106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:48.417 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.258+0000 7fa9265a2700 1 --2- 172.21.15.29:0/930559978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa920106370 0x7fa920106790 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa9140098a0 tx=0x7fa914009bb0 comp rx=0 tx=0).stop 2024-03-11T19:00:48.417 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.258+0000 7fa9265a2700 1 -- 172.21.15.29:0/930559978 shutdown_connections 2024-03-11T19:00:48.417 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.258+0000 7fa9265a2700 1 --2- 172.21.15.29:0/930559978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa920106370 0x7fa920106790 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:48.417 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.258+0000 7fa9265a2700 1 -- 172.21.15.29:0/930559978 >> 172.21.15.29:0/930559978 conn(0x7fa920101930 msgr2=0x7fa920103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:48.417 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.258+0000 7fa9265a2700 1 -- 172.21.15.29:0/930559978 shutdown_connections 2024-03-11T19:00:48.417 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.258+0000 7fa9265a2700 1 -- 172.21.15.29:0/930559978 wait complete. 2024-03-11T19:00:48.417 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.259+0000 7fa9265a2700 1 Processor -- start 2024-03-11T19:00:48.417 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.259+0000 7fa9265a2700 1 -- start start 2024-03-11T19:00:48.417 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.259+0000 7fa9265a2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa920106370 0x7fa92019a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:48.417 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.259+0000 7fa9265a2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92019a600 con 0x7fa920106370 2024-03-11T19:00:48.417 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.259+0000 7fa91ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa920106370 0x7fa92019a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:48.418 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.259+0000 7fa91ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa920106370 0x7fa92019a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49886/0 (socket says 172.21.15.29:49886) 2024-03-11T19:00:48.418 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.259+0000 7fa91ffff700 1 -- 172.21.15.29:0/3348194350 learned_addr learned my addr 172.21.15.29:0/3348194350 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:48.418 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.260+0000 7fa91ffff700 1 -- 172.21.15.29:0/3348194350 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa914009550 con 0x7fa920106370 2024-03-11T19:00:48.418 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.260+0000 7fa91ffff700 1 --2- 172.21.15.29:0/3348194350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa920106370 0x7fa92019a0c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa914011670 tx=0x7fa914011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:48.418 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.260+0000 7fa91d7fa700 1 -- 172.21.15.29:0/3348194350 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa9140038b0 con 0x7fa920106370 2024-03-11T19:00:48.418 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.260+0000 7fa91d7fa700 1 -- 172.21.15.29:0/3348194350 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa914003a10 con 0x7fa920106370 2024-03-11T19:00:48.418 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.260+0000 7fa9265a2700 1 -- 172.21.15.29:0/3348194350 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa92019a800 con 0x7fa920106370 2024-03-11T19:00:48.418 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.260+0000 7fa91d7fa700 1 -- 172.21.15.29:0/3348194350 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa914019530 con 0x7fa920106370 2024-03-11T19:00:48.418 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.260+0000 7fa9265a2700 1 -- 172.21.15.29:0/3348194350 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa92019aca0 con 0x7fa920106370 2024-03-11T19:00:48.418 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.261+0000 7fa91d7fa700 1 -- 172.21.15.29:0/3348194350 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 45782+0+0 (secure 0 0 0) 0x7fa914019810 con 0x7fa920106370 2024-03-11T19:00:48.418 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.261+0000 7fa91d7fa700 1 --2- 172.21.15.29:0/3348194350 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa908038cb0 0x7fa90803b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:48.418 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.261+0000 7fa91d7fa700 1 -- 172.21.15.29:0/3348194350 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa91404df30 con 0x7fa920106370 2024-03-11T19:00:48.418 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.261+0000 7fa9265a2700 1 -- 172.21.15.29:0/3348194350 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa92010a7a0 con 0x7fa920106370 2024-03-11T19:00:48.418 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.261+0000 7fa91f7fe700 1 --2- 172.21.15.29:0/3348194350 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa908038cb0 0x7fa90803b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:48.419 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.261+0000 7fa91f7fe700 1 --2- 172.21.15.29:0/3348194350 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa908038cb0 0x7fa90803b170 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa910006fd0 tx=0x7fa910006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:48.419 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.266+0000 7fa91d7fa700 1 -- 172.21.15.29:0/3348194350 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa914019b60 con 0x7fa920106370 2024-03-11T19:00:48.419 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.489+0000 7fa9265a2700 1 -- 172.21.15.29:0/3348194350 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fa9200630c0 con 0x7fa908038cb0 2024-03-11T19:00:48.419 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.736+0000 7fa91d7fa700 1 -- 172.21.15.29:0/3348194350 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fa9200630c0 con 0x7fa908038cb0 2024-03-11T19:00:48.419 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.738+0000 7fa9265a2700 1 -- 172.21.15.29:0/3348194350 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa908038cb0 msgr2=0x7fa90803b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:48.419 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.738+0000 7fa9265a2700 1 --2- 172.21.15.29:0/3348194350 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa908038cb0 0x7fa90803b170 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa910006fd0 tx=0x7fa910006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:48.419 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.738+0000 7fa9265a2700 1 -- 172.21.15.29:0/3348194350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa920106370 msgr2=0x7fa92019a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:48.419 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.738+0000 7fa9265a2700 1 --2- 172.21.15.29:0/3348194350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa920106370 0x7fa92019a0c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa914011670 tx=0x7fa914011a30 comp rx=0 tx=0).stop 2024-03-11T19:00:48.419 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.739+0000 7fa9265a2700 1 -- 172.21.15.29:0/3348194350 shutdown_connections 2024-03-11T19:00:48.419 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.739+0000 7fa9265a2700 1 --2- 172.21.15.29:0/3348194350 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa908038cb0 0x7fa90803b170 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:48.419 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.739+0000 7fa9265a2700 1 --2- 172.21.15.29:0/3348194350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa920106370 0x7fa92019a0c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:48.419 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.739+0000 7fa9265a2700 1 -- 172.21.15.29:0/3348194350 >> 172.21.15.29:0/3348194350 conn(0x7fa920101930 msgr2=0x7fa920102440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:48.419 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.739+0000 7fa9265a2700 1 -- 172.21.15.29:0/3348194350 shutdown_connections 2024-03-11T19:00:48.419 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:47.739+0000 7fa9265a2700 1 -- 172.21.15.29:0/3348194350 wait complete. 2024-03-11T19:00:48.419 INFO:teuthology.orchestra.run.smithi029.stdout:Fetching dashboard port number... 2024-03-11T19:00:49.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:49 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2038902857' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-11T19:00:50.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 8443 2024-03-11T19:00:50.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.864+0000 7fb72e324700 1 Processor -- start 2024-03-11T19:00:50.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.864+0000 7fb72e324700 1 -- start start 2024-03-11T19:00:50.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.864+0000 7fb72e324700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb728107f60 0x7fb72810a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:50.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.864+0000 7fb72e324700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72807adf0 con 0x7fb728107f60 2024-03-11T19:00:50.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.864+0000 7fb727fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb728107f60 0x7fb72810a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:50.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.864+0000 7fb727fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb728107f60 0x7fb72810a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49902/0 (socket says 172.21.15.29:49902) 2024-03-11T19:00:50.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.864+0000 7fb727fff700 1 -- 172.21.15.29:0/2966636463 learned_addr learned my addr 172.21.15.29:0/2966636463 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:50.198 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.864+0000 7fb727fff700 1 -- 172.21.15.29:0/2966636463 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72807af30 con 0x7fb728107f60 2024-03-11T19:00:50.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.864+0000 7fb727fff700 1 --2- 172.21.15.29:0/2966636463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb728107f60 0x7fb72810a350 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb7180098a0 tx=0x7fb718009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b354fd0857c8f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:50.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.865+0000 7fb726ffd700 1 -- 172.21.15.29:0/2966636463 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb718004160 con 0x7fb728107f60 2024-03-11T19:00:50.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.865+0000 7fb726ffd700 1 -- 172.21.15.29:0/2966636463 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb7180042c0 con 0x7fb728107f60 2024-03-11T19:00:50.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.865+0000 7fb726ffd700 1 -- 172.21.15.29:0/2966636463 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7180119b0 con 0x7fb728107f60 2024-03-11T19:00:50.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.865+0000 7fb72e324700 1 -- 172.21.15.29:0/2966636463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb728107f60 msgr2=0x7fb72810a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:50.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.865+0000 7fb72e324700 1 --2- 172.21.15.29:0/2966636463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb728107f60 0x7fb72810a350 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb7180098a0 tx=0x7fb718009bb0 comp rx=0 tx=0).stop 2024-03-11T19:00:50.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.865+0000 7fb72e324700 1 -- 172.21.15.29:0/2966636463 shutdown_connections 2024-03-11T19:00:50.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.865+0000 7fb72e324700 1 --2- 172.21.15.29:0/2966636463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb728107f60 0x7fb72810a350 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:50.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.865+0000 7fb72e324700 1 -- 172.21.15.29:0/2966636463 >> 172.21.15.29:0/2966636463 conn(0x7fb728101930 msgr2=0x7fb728103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:50.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.865+0000 7fb72e324700 1 -- 172.21.15.29:0/2966636463 shutdown_connections 2024-03-11T19:00:50.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.865+0000 7fb72e324700 1 -- 172.21.15.29:0/2966636463 wait complete. 2024-03-11T19:00:50.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.866+0000 7fb72e324700 1 Processor -- start 2024-03-11T19:00:50.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.866+0000 7fb72e324700 1 -- start start 2024-03-11T19:00:50.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.866+0000 7fb72e324700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb728107f60 0x7fb72819b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:50.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.866+0000 7fb72e324700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72819b690 con 0x7fb728107f60 2024-03-11T19:00:50.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.866+0000 7fb727fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb728107f60 0x7fb72819b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:50.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.866+0000 7fb727fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb728107f60 0x7fb72819b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49904/0 (socket says 172.21.15.29:49904) 2024-03-11T19:00:50.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.866+0000 7fb727fff700 1 -- 172.21.15.29:0/2038902857 learned_addr learned my addr 172.21.15.29:0/2038902857 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:50.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.866+0000 7fb727fff700 1 -- 172.21.15.29:0/2038902857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb718009550 con 0x7fb728107f60 2024-03-11T19:00:50.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.867+0000 7fb727fff700 1 --2- 172.21.15.29:0/2038902857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb728107f60 0x7fb72819b150 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb718011980 tx=0x7fb718011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:50.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.867+0000 7fb7257fa700 1 -- 172.21.15.29:0/2038902857 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7180039c0 con 0x7fb728107f60 2024-03-11T19:00:50.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.867+0000 7fb7257fa700 1 -- 172.21.15.29:0/2038902857 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb718003b20 con 0x7fb728107f60 2024-03-11T19:00:50.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.867+0000 7fb7257fa700 1 -- 172.21.15.29:0/2038902857 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb718019650 con 0x7fb728107f60 2024-03-11T19:00:50.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.867+0000 7fb72e324700 1 -- 172.21.15.29:0/2038902857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72819b890 con 0x7fb728107f60 2024-03-11T19:00:50.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.867+0000 7fb72e324700 1 -- 172.21.15.29:0/2038902857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72819bd30 con 0x7fb728107f60 2024-03-11T19:00:50.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.868+0000 7fb7257fa700 1 -- 172.21.15.29:0/2038902857 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 45782+0+0 (secure 0 0 0) 0x7fb71801a430 con 0x7fb728107f60 2024-03-11T19:00:50.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.868+0000 7fb72e324700 1 -- 172.21.15.29:0/2038902857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb72810b600 con 0x7fb728107f60 2024-03-11T19:00:50.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.868+0000 7fb7257fa700 1 --2- 172.21.15.29:0/2038902857 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb710038d60 0x7fb71003b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:50.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.868+0000 7fb7257fa700 1 -- 172.21.15.29:0/2038902857 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb71804d0d0 con 0x7fb728107f60 2024-03-11T19:00:50.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.868+0000 7fb7277fe700 1 --2- 172.21.15.29:0/2038902857 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb710038d60 0x7fb71003b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:50.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.869+0000 7fb7277fe700 1 --2- 172.21.15.29:0/2038902857 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb710038d60 0x7fb71003b220 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb71c006fd0 tx=0x7fb71c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:50.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:48.873+0000 7fb7257fa700 1 -- 172.21.15.29:0/2038902857 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb718018350 con 0x7fb728107f60 2024-03-11T19:00:50.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:49.078+0000 7fb72e324700 1 -- 172.21.15.29:0/2038902857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fb72819e610 con 0x7fb728107f60 2024-03-11T19:00:50.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:49.079+0000 7fb7257fa700 1 -- 172.21.15.29:0/2038902857 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fb718018350 con 0x7fb728107f60 2024-03-11T19:00:50.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:49.081+0000 7fb72e324700 1 -- 172.21.15.29:0/2038902857 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb710038d60 msgr2=0x7fb71003b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:50.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:49.081+0000 7fb72e324700 1 --2- 172.21.15.29:0/2038902857 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb710038d60 0x7fb71003b220 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb71c006fd0 tx=0x7fb71c006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:50.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:49.081+0000 7fb72e324700 1 -- 172.21.15.29:0/2038902857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb728107f60 msgr2=0x7fb72819b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:50.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:49.081+0000 7fb72e324700 1 --2- 172.21.15.29:0/2038902857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb728107f60 0x7fb72819b150 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb718011980 tx=0x7fb718011a60 comp rx=0 tx=0).stop 2024-03-11T19:00:50.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:49.081+0000 7fb72e324700 1 -- 172.21.15.29:0/2038902857 shutdown_connections 2024-03-11T19:00:50.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:49.081+0000 7fb72e324700 1 --2- 172.21.15.29:0/2038902857 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb710038d60 0x7fb71003b220 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:50.201 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:49.081+0000 7fb72e324700 1 --2- 172.21.15.29:0/2038902857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb728107f60 0x7fb72819b150 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:50.202 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:49.081+0000 7fb72e324700 1 -- 172.21.15.29:0/2038902857 >> 172.21.15.29:0/2038902857 conn(0x7fb728101930 msgr2=0x7fb728102460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:50.202 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:49.081+0000 7fb72e324700 1 -- 172.21.15.29:0/2038902857 shutdown_connections 2024-03-11T19:00:50.202 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:49.081+0000 7fb72e324700 1 -- 172.21.15.29:0/2038902857 wait complete. 2024-03-11T19:00:50.207 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T19:00:50.207 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-03-11T19:00:50.214 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T19:00:50.214 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-11T19:00:50.214 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-03-11T19:00:50.215 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-11T19:00:50.216 INFO:teuthology.orchestra.run.smithi029.stdout:Ceph Dashboard is now available at: 2024-03-11T19:00:50.217 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:00:50.217 INFO:teuthology.orchestra.run.smithi029.stdout: URL: https://smithi029.front.sepia.ceph.com:8443/ 2024-03-11T19:00:50.217 INFO:teuthology.orchestra.run.smithi029.stdout: User: admin 2024-03-11T19:00:50.217 INFO:teuthology.orchestra.run.smithi029.stdout: Password: wabkjqjsp3 2024-03-11T19:00:50.217 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:00:50.217 INFO:teuthology.orchestra.run.smithi029.stdout:Saving cluster configuration to /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/config directory 2024-03-11T19:00:50.218 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling autotune for osd_memory_target 2024-03-11T19:00:51.388 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:51 smithi029 ceph-mon[32439]: mgrmap e13: x(active, since 6s) 2024-03-11T19:00:51.397 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.630+0000 7fa37e3f9700 1 Processor -- start 2024-03-11T19:00:51.397 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.630+0000 7fa37e3f9700 1 -- start start 2024-03-11T19:00:51.397 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.630+0000 7fa37e3f9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa378106340 0x7fa378106760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:51.397 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.630+0000 7fa37e3f9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa378106d30 con 0x7fa378106340 2024-03-11T19:00:51.397 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.631+0000 7fa377fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa378106340 0x7fa378106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:51.397 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.631+0000 7fa377fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa378106340 0x7fa378106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49906/0 (socket says 172.21.15.29:49906) 2024-03-11T19:00:51.397 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.631+0000 7fa377fff700 1 -- 172.21.15.29:0/462896066 learned_addr learned my addr 172.21.15.29:0/462896066 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:51.398 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.631+0000 7fa377fff700 1 -- 172.21.15.29:0/462896066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa378107540 con 0x7fa378106340 2024-03-11T19:00:51.398 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.631+0000 7fa377fff700 1 --2- 172.21.15.29:0/462896066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa378106340 0x7fa378106760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa3680098a0 tx=0x7fa368009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20c837956e51f4fe server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:51.398 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.631+0000 7fa376ffd700 1 -- 172.21.15.29:0/462896066 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa368004160 con 0x7fa378106340 2024-03-11T19:00:51.398 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.631+0000 7fa376ffd700 1 -- 172.21.15.29:0/462896066 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa3680042c0 con 0x7fa378106340 2024-03-11T19:00:51.398 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.631+0000 7fa376ffd700 1 -- 172.21.15.29:0/462896066 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa3680119b0 con 0x7fa378106340 2024-03-11T19:00:51.398 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.632+0000 7fa37e3f9700 1 -- 172.21.15.29:0/462896066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa378106340 msgr2=0x7fa378106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:51.398 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.632+0000 7fa37e3f9700 1 --2- 172.21.15.29:0/462896066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa378106340 0x7fa378106760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa3680098a0 tx=0x7fa368009bb0 comp rx=0 tx=0).stop 2024-03-11T19:00:51.398 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.632+0000 7fa37e3f9700 1 -- 172.21.15.29:0/462896066 shutdown_connections 2024-03-11T19:00:51.398 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.632+0000 7fa37e3f9700 1 --2- 172.21.15.29:0/462896066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa378106340 0x7fa378106760 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:51.398 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.632+0000 7fa37e3f9700 1 -- 172.21.15.29:0/462896066 >> 172.21.15.29:0/462896066 conn(0x7fa378101900 msgr2=0x7fa378103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:51.398 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.632+0000 7fa37e3f9700 1 -- 172.21.15.29:0/462896066 shutdown_connections 2024-03-11T19:00:51.398 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.632+0000 7fa37e3f9700 1 -- 172.21.15.29:0/462896066 wait complete. 2024-03-11T19:00:51.398 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.633+0000 7fa37e3f9700 1 Processor -- start 2024-03-11T19:00:51.399 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.633+0000 7fa37e3f9700 1 -- start start 2024-03-11T19:00:51.399 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.633+0000 7fa37e3f9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa378106340 0x7fa37819b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:51.399 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.633+0000 7fa37e3f9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37819b720 con 0x7fa378106340 2024-03-11T19:00:51.399 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.633+0000 7fa377fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa378106340 0x7fa37819b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:51.399 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.633+0000 7fa377fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa378106340 0x7fa37819b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49916/0 (socket says 172.21.15.29:49916) 2024-03-11T19:00:51.399 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.633+0000 7fa377fff700 1 -- 172.21.15.29:0/1200682671 learned_addr learned my addr 172.21.15.29:0/1200682671 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:51.399 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.633+0000 7fa377fff700 1 -- 172.21.15.29:0/1200682671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa368009550 con 0x7fa378106340 2024-03-11T19:00:51.399 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.634+0000 7fa377fff700 1 --2- 172.21.15.29:0/1200682671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa378106340 0x7fa37819b1e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa368011670 tx=0x7fa368011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:51.399 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.634+0000 7fa3757fa700 1 -- 172.21.15.29:0/1200682671 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa3680038b0 con 0x7fa378106340 2024-03-11T19:00:51.399 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.634+0000 7fa3757fa700 1 -- 172.21.15.29:0/1200682671 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa368003a10 con 0x7fa378106340 2024-03-11T19:00:51.399 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.634+0000 7fa37e3f9700 1 -- 172.21.15.29:0/1200682671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa37819b920 con 0x7fa378106340 2024-03-11T19:00:51.399 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.634+0000 7fa3757fa700 1 -- 172.21.15.29:0/1200682671 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa36801a630 con 0x7fa378106340 2024-03-11T19:00:51.399 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.634+0000 7fa37e3f9700 1 -- 172.21.15.29:0/1200682671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa37819bdc0 con 0x7fa378106340 2024-03-11T19:00:51.399 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.635+0000 7fa3757fa700 1 -- 172.21.15.29:0/1200682671 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7fa36801a8a0 con 0x7fa378106340 2024-03-11T19:00:51.399 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.635+0000 7fa3757fa700 1 --2- 172.21.15.29:0/1200682671 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa360038db0 0x7fa36003b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:51.400 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.635+0000 7fa3757fa700 1 -- 172.21.15.29:0/1200682671 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa36804d030 con 0x7fa378106340 2024-03-11T19:00:51.400 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.635+0000 7fa37e3f9700 1 -- 172.21.15.29:0/1200682671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa378195350 con 0x7fa378106340 2024-03-11T19:00:51.400 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.635+0000 7fa3777fe700 1 --2- 172.21.15.29:0/1200682671 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa360038db0 0x7fa36003b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:51.400 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.635+0000 7fa3777fe700 1 --2- 172.21.15.29:0/1200682671 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa360038db0 0x7fa36003b270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa36c006fd0 tx=0x7fa36c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:51.400 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.640+0000 7fa3757fa700 1 -- 172.21.15.29:0/1200682671 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa368018cb0 con 0x7fa378106340 2024-03-11T19:00:51.400 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.842+0000 7fa37e3f9700 1 -- 172.21.15.29:0/1200682671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fa37810a890 con 0x7fa378106340 2024-03-11T19:00:51.400 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.842+0000 7fa3757fa700 1 -- 172.21.15.29:0/1200682671 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fa3680188a0 con 0x7fa378106340 2024-03-11T19:00:51.400 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.844+0000 7fa37e3f9700 1 -- 172.21.15.29:0/1200682671 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa360038db0 msgr2=0x7fa36003b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:51.400 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.844+0000 7fa37e3f9700 1 --2- 172.21.15.29:0/1200682671 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa360038db0 0x7fa36003b270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa36c006fd0 tx=0x7fa36c006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:51.400 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.844+0000 7fa37e3f9700 1 -- 172.21.15.29:0/1200682671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa378106340 msgr2=0x7fa37819b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:51.400 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.844+0000 7fa37e3f9700 1 --2- 172.21.15.29:0/1200682671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa378106340 0x7fa37819b1e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa368011670 tx=0x7fa368011a30 comp rx=0 tx=0).stop 2024-03-11T19:00:51.400 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.845+0000 7fa37e3f9700 1 -- 172.21.15.29:0/1200682671 shutdown_connections 2024-03-11T19:00:51.400 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.845+0000 7fa37e3f9700 1 --2- 172.21.15.29:0/1200682671 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa360038db0 0x7fa36003b270 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:51.400 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.845+0000 7fa37e3f9700 1 --2- 172.21.15.29:0/1200682671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa378106340 0x7fa37819b1e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:51.400 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.845+0000 7fa37e3f9700 1 -- 172.21.15.29:0/1200682671 >> 172.21.15.29:0/1200682671 conn(0x7fa378101900 msgr2=0x7fa3781027e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:51.400 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.845+0000 7fa37e3f9700 1 -- 172.21.15.29:0/1200682671 shutdown_connections 2024-03-11T19:00:51.401 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:50.845+0000 7fa37e3f9700 1 -- 172.21.15.29:0/1200682671 wait complete. 2024-03-11T19:00:52.701 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.803+0000 7fe77a63d700 1 Processor -- start 2024-03-11T19:00:52.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.803+0000 7fe77a63d700 1 -- start start 2024-03-11T19:00:52.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.803+0000 7fe77a63d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe774107460 0x7fe774107880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:52.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.803+0000 7fe77a63d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe774107e50 con 0x7fe774107460 2024-03-11T19:00:52.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.804+0000 7fe773fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe774107460 0x7fe774107880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:52.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.804+0000 7fe773fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe774107460 0x7fe774107880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49922/0 (socket says 172.21.15.29:49922) 2024-03-11T19:00:52.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.804+0000 7fe773fff700 1 -- 172.21.15.29:0/1460117533 learned_addr learned my addr 172.21.15.29:0/1460117533 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:52.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.804+0000 7fe773fff700 1 -- 172.21.15.29:0/1460117533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe774108610 con 0x7fe774107460 2024-03-11T19:00:52.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.804+0000 7fe773fff700 1 --2- 172.21.15.29:0/1460117533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe774107460 0x7fe774107880 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe7640098a0 tx=0x7fe764009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3581b04ad4e8613 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:52.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.804+0000 7fe7737fe700 1 -- 172.21.15.29:0/1460117533 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe764004160 con 0x7fe774107460 2024-03-11T19:00:52.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.804+0000 7fe7737fe700 1 -- 172.21.15.29:0/1460117533 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe7640042c0 con 0x7fe774107460 2024-03-11T19:00:52.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.804+0000 7fe7737fe700 1 -- 172.21.15.29:0/1460117533 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe7640119b0 con 0x7fe774107460 2024-03-11T19:00:52.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.805+0000 7fe77a63d700 1 -- 172.21.15.29:0/1460117533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe774107460 msgr2=0x7fe774107880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:52.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.805+0000 7fe77a63d700 1 --2- 172.21.15.29:0/1460117533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe774107460 0x7fe774107880 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe7640098a0 tx=0x7fe764009bb0 comp rx=0 tx=0).stop 2024-03-11T19:00:52.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.805+0000 7fe77a63d700 1 -- 172.21.15.29:0/1460117533 shutdown_connections 2024-03-11T19:00:52.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.805+0000 7fe77a63d700 1 --2- 172.21.15.29:0/1460117533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe774107460 0x7fe774107880 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:52.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.805+0000 7fe77a63d700 1 -- 172.21.15.29:0/1460117533 >> 172.21.15.29:0/1460117533 conn(0x7fe774102a40 msgr2=0x7fe774104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:52.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.805+0000 7fe77a63d700 1 -- 172.21.15.29:0/1460117533 shutdown_connections 2024-03-11T19:00:52.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.805+0000 7fe77a63d700 1 -- 172.21.15.29:0/1460117533 wait complete. 2024-03-11T19:00:52.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.805+0000 7fe77a63d700 1 Processor -- start 2024-03-11T19:00:52.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.805+0000 7fe77a63d700 1 -- start start 2024-03-11T19:00:52.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.806+0000 7fe77a63d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe774107460 0x7fe77419b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:52.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.806+0000 7fe77a63d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77419b680 con 0x7fe774107460 2024-03-11T19:00:52.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.806+0000 7fe773fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe774107460 0x7fe77419b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:52.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.806+0000 7fe773fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe774107460 0x7fe77419b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49928/0 (socket says 172.21.15.29:49928) 2024-03-11T19:00:52.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.806+0000 7fe773fff700 1 -- 172.21.15.29:0/3025866400 learned_addr learned my addr 172.21.15.29:0/3025866400 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:52.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.806+0000 7fe773fff700 1 -- 172.21.15.29:0/3025866400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe764009550 con 0x7fe774107460 2024-03-11T19:00:52.704 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.806+0000 7fe773fff700 1 --2- 172.21.15.29:0/3025866400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe774107460 0x7fe77419b140 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe764011980 tx=0x7fe764011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:52.704 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.806+0000 7fe771ffb700 1 -- 172.21.15.29:0/3025866400 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe764003970 con 0x7fe774107460 2024-03-11T19:00:52.704 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.806+0000 7fe771ffb700 1 -- 172.21.15.29:0/3025866400 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe764003ad0 con 0x7fe774107460 2024-03-11T19:00:52.704 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.806+0000 7fe77a63d700 1 -- 172.21.15.29:0/3025866400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe77419b880 con 0x7fe774107460 2024-03-11T19:00:52.704 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.806+0000 7fe771ffb700 1 -- 172.21.15.29:0/3025866400 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe764019610 con 0x7fe774107460 2024-03-11T19:00:52.704 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.806+0000 7fe77a63d700 1 -- 172.21.15.29:0/3025866400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe77419bd20 con 0x7fe774107460 2024-03-11T19:00:52.704 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.807+0000 7fe771ffb700 1 -- 172.21.15.29:0/3025866400 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7fe764019880 con 0x7fe774107460 2024-03-11T19:00:52.704 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.807+0000 7fe771ffb700 1 --2- 172.21.15.29:0/3025866400 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe760038e00 0x7fe76003b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:52.704 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.807+0000 7fe77a63d700 1 -- 172.21.15.29:0/3025866400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe77410b840 con 0x7fe774107460 2024-03-11T19:00:52.704 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.807+0000 7fe771ffb700 1 -- 172.21.15.29:0/3025866400 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe76404d1b0 con 0x7fe774107460 2024-03-11T19:00:52.704 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.808+0000 7fe76bfff700 1 --2- 172.21.15.29:0/3025866400 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe760038e00 0x7fe76003b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:52.704 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.808+0000 7fe76bfff700 1 --2- 172.21.15.29:0/3025866400 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe760038e00 0x7fe76003b2c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe75c006fd0 tx=0x7fe75c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:52.704 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:51.812+0000 7fe771ffb700 1 -- 172.21.15.29:0/3025866400 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe764018c20 con 0x7fe774107460 2024-03-11T19:00:52.704 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:52.125+0000 7fe77a63d700 1 -- 172.21.15.29:0/3025866400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fe77410ba50 con 0x7fe774107460 2024-03-11T19:00:52.705 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:52.127+0000 7fe771ffb700 1 -- 172.21.15.29:0/3025866400 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fe764018350 con 0x7fe774107460 2024-03-11T19:00:52.705 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:52.129+0000 7fe77a63d700 1 -- 172.21.15.29:0/3025866400 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe760038e00 msgr2=0x7fe76003b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:52.705 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:52.129+0000 7fe77a63d700 1 --2- 172.21.15.29:0/3025866400 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe760038e00 0x7fe76003b2c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe75c006fd0 tx=0x7fe75c006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:52.705 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:52.129+0000 7fe77a63d700 1 -- 172.21.15.29:0/3025866400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe774107460 msgr2=0x7fe77419b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:52.705 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:52.129+0000 7fe77a63d700 1 --2- 172.21.15.29:0/3025866400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe774107460 0x7fe77419b140 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe764011980 tx=0x7fe764011a60 comp rx=0 tx=0).stop 2024-03-11T19:00:52.705 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:52.129+0000 7fe77a63d700 1 -- 172.21.15.29:0/3025866400 shutdown_connections 2024-03-11T19:00:52.705 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:52.129+0000 7fe77a63d700 1 --2- 172.21.15.29:0/3025866400 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe760038e00 0x7fe76003b2c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:52.705 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:52.129+0000 7fe77a63d700 1 --2- 172.21.15.29:0/3025866400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe774107460 0x7fe77419b140 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:52.705 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:52.129+0000 7fe77a63d700 1 -- 172.21.15.29:0/3025866400 >> 172.21.15.29:0/3025866400 conn(0x7fe774102a40 msgr2=0x7fe774103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:52.705 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:52.130+0000 7fe77a63d700 1 -- 172.21.15.29:0/3025866400 shutdown_connections 2024-03-11T19:00:52.705 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-11T19:00:52.130+0000 7fe77a63d700 1 -- 172.21.15.29:0/3025866400 wait complete. 2024-03-11T19:00:52.705 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-11T19:00:52.705 INFO:teuthology.orchestra.run.smithi029.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-11T19:00:52.705 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:00:52.705 INFO:teuthology.orchestra.run.smithi029.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-11T19:00:52.705 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:00:52.706 INFO:teuthology.orchestra.run.smithi029.stdout:Or, if you are only running a single cluster on this host: 2024-03-11T19:00:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:00:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-11T19:00:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:00:52.706 INFO:teuthology.orchestra.run.smithi029.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-11T19:00:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:00:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: ceph telemetry on 2024-03-11T19:00:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:00:52.706 INFO:teuthology.orchestra.run.smithi029.stdout:For more information see: 2024-03-11T19:00:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:00:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-11T19:00:52.706 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:00:52.706 INFO:teuthology.orchestra.run.smithi029.stdout:Bootstrap complete. 2024-03-11T19:00:52.740 INFO:tasks.cephadm:Fetching config... 2024-03-11T19:00:52.741 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-11T19:00:52.741 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-11T19:00:52.756 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-11T19:00:52.756 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-11T19:00:52.756 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-11T19:00:52.812 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-11T19:00:52.812 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-11T19:00:52.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-11T19:00:52.879 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-11T19:00:52.879 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-11T19:00:52.879 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-11T19:00:52.934 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-11T19:00:52.935 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDP0ONkhl+F6vFrvdZYF63qAOjY6/P8wA7M06kw3BCc3LajkG/j5Ik5c2R20mSlZ7DXtfDVGxV77ORnttjoT2MCKr6qJAOOdoovc973s3Twe05VFlnpE5g6hQ3zhbD+zdZOiAydkhq42mFHLkgmCSl//K08QJLsakKvlKvAbEppkZA6aIXhoNtfXoGBT92B35Kgpw1PgusmnDs9v2WCtcQko5fFU2o1Nre+rBqsPlZR1/JJkR1B1qXRsjb0fQwMq1lCiPFyK6aVQMdsmBzOkDLh8YTqWipFQ+GHTnerwJd1IUEcFJyz0EOSJE0bhPVkANwlkDbZpb+I6xeEM4G3zjHABWJhPFRshwbX0tSFUeiIxP3TgI9EnqT1CU2IS+GvLuIRILAY8KeOoinek9xsYMwQ+TtHsyGONDMUe2uIHM9Uhdeu9ahqKodd3fHxtivMXj6k6EORB2xnPLvzYsCEv+Z+gll9rjVZRrzNloYsxe8yZgIIz1amlDOnlNLmaG77B1c= ceph-638915c2-dfd9-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T19:00:53.012 INFO:teuthology.orchestra.run.smithi029.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDP0ONkhl+F6vFrvdZYF63qAOjY6/P8wA7M06kw3BCc3LajkG/j5Ik5c2R20mSlZ7DXtfDVGxV77ORnttjoT2MCKr6qJAOOdoovc973s3Twe05VFlnpE5g6hQ3zhbD+zdZOiAydkhq42mFHLkgmCSl//K08QJLsakKvlKvAbEppkZA6aIXhoNtfXoGBT92B35Kgpw1PgusmnDs9v2WCtcQko5fFU2o1Nre+rBqsPlZR1/JJkR1B1qXRsjb0fQwMq1lCiPFyK6aVQMdsmBzOkDLh8YTqWipFQ+GHTnerwJd1IUEcFJyz0EOSJE0bhPVkANwlkDbZpb+I6xeEM4G3zjHABWJhPFRshwbX0tSFUeiIxP3TgI9EnqT1CU2IS+GvLuIRILAY8KeOoinek9xsYMwQ+TtHsyGONDMUe2uIHM9Uhdeu9ahqKodd3fHxtivMXj6k6EORB2xnPLvzYsCEv+Z+gll9rjVZRrzNloYsxe8yZgIIz1amlDOnlNLmaG77B1c= ceph-638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T19:00:53.026 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDP0ONkhl+F6vFrvdZYF63qAOjY6/P8wA7M06kw3BCc3LajkG/j5Ik5c2R20mSlZ7DXtfDVGxV77ORnttjoT2MCKr6qJAOOdoovc973s3Twe05VFlnpE5g6hQ3zhbD+zdZOiAydkhq42mFHLkgmCSl//K08QJLsakKvlKvAbEppkZA6aIXhoNtfXoGBT92B35Kgpw1PgusmnDs9v2WCtcQko5fFU2o1Nre+rBqsPlZR1/JJkR1B1qXRsjb0fQwMq1lCiPFyK6aVQMdsmBzOkDLh8YTqWipFQ+GHTnerwJd1IUEcFJyz0EOSJE0bhPVkANwlkDbZpb+I6xeEM4G3zjHABWJhPFRshwbX0tSFUeiIxP3TgI9EnqT1CU2IS+GvLuIRILAY8KeOoinek9xsYMwQ+TtHsyGONDMUe2uIHM9Uhdeu9ahqKodd3fHxtivMXj6k6EORB2xnPLvzYsCEv+Z+gll9rjVZRrzNloYsxe8yZgIIz1amlDOnlNLmaG77B1c= ceph-638915c2-dfd9-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T19:00:53.076 INFO:teuthology.orchestra.run.smithi170.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDP0ONkhl+F6vFrvdZYF63qAOjY6/P8wA7M06kw3BCc3LajkG/j5Ik5c2R20mSlZ7DXtfDVGxV77ORnttjoT2MCKr6qJAOOdoovc973s3Twe05VFlnpE5g6hQ3zhbD+zdZOiAydkhq42mFHLkgmCSl//K08QJLsakKvlKvAbEppkZA6aIXhoNtfXoGBT92B35Kgpw1PgusmnDs9v2WCtcQko5fFU2o1Nre+rBqsPlZR1/JJkR1B1qXRsjb0fQwMq1lCiPFyK6aVQMdsmBzOkDLh8YTqWipFQ+GHTnerwJd1IUEcFJyz0EOSJE0bhPVkANwlkDbZpb+I6xeEM4G3zjHABWJhPFRshwbX0tSFUeiIxP3TgI9EnqT1CU2IS+GvLuIRILAY8KeOoinek9xsYMwQ+TtHsyGONDMUe2uIHM9Uhdeu9ahqKodd3fHxtivMXj6k6EORB2xnPLvzYsCEv+Z+gll9rjVZRrzNloYsxe8yZgIIz1amlDOnlNLmaG77B1c= ceph-638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T19:00:53.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDP0ONkhl+F6vFrvdZYF63qAOjY6/P8wA7M06kw3BCc3LajkG/j5Ik5c2R20mSlZ7DXtfDVGxV77ORnttjoT2MCKr6qJAOOdoovc973s3Twe05VFlnpE5g6hQ3zhbD+zdZOiAydkhq42mFHLkgmCSl//K08QJLsakKvlKvAbEppkZA6aIXhoNtfXoGBT92B35Kgpw1PgusmnDs9v2WCtcQko5fFU2o1Nre+rBqsPlZR1/JJkR1B1qXRsjb0fQwMq1lCiPFyK6aVQMdsmBzOkDLh8YTqWipFQ+GHTnerwJd1IUEcFJyz0EOSJE0bhPVkANwlkDbZpb+I6xeEM4G3zjHABWJhPFRshwbX0tSFUeiIxP3TgI9EnqT1CU2IS+GvLuIRILAY8KeOoinek9xsYMwQ+TtHsyGONDMUe2uIHM9Uhdeu9ahqKodd3fHxtivMXj6k6EORB2xnPLvzYsCEv+Z+gll9rjVZRrzNloYsxe8yZgIIz1amlDOnlNLmaG77B1c= ceph-638915c2-dfd9-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T19:00:53.139 INFO:teuthology.orchestra.run.smithi190.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDP0ONkhl+F6vFrvdZYF63qAOjY6/P8wA7M06kw3BCc3LajkG/j5Ik5c2R20mSlZ7DXtfDVGxV77ORnttjoT2MCKr6qJAOOdoovc973s3Twe05VFlnpE5g6hQ3zhbD+zdZOiAydkhq42mFHLkgmCSl//K08QJLsakKvlKvAbEppkZA6aIXhoNtfXoGBT92B35Kgpw1PgusmnDs9v2WCtcQko5fFU2o1Nre+rBqsPlZR1/JJkR1B1qXRsjb0fQwMq1lCiPFyK6aVQMdsmBzOkDLh8YTqWipFQ+GHTnerwJd1IUEcFJyz0EOSJE0bhPVkANwlkDbZpb+I6xeEM4G3zjHABWJhPFRshwbX0tSFUeiIxP3TgI9EnqT1CU2IS+GvLuIRILAY8KeOoinek9xsYMwQ+TtHsyGONDMUe2uIHM9Uhdeu9ahqKodd3fHxtivMXj6k6EORB2xnPLvzYsCEv+Z+gll9rjVZRrzNloYsxe8yZgIIz1amlDOnlNLmaG77B1c= ceph-638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T19:00:53.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-11T19:00:53.251 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:53 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3025866400' entity='client.admin' 2024-03-11T19:00:53.396 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:00:55.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.566+0000 7fcea907e700 1 -- 172.21.15.29:0/377077129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcea40fffc0 msgr2=0x7fcea41003e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:55.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.566+0000 7fcea907e700 1 --2- 172.21.15.29:0/377077129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcea40fffc0 0x7fcea41003e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fce98009910 tx=0x7fce98009c20 comp rx=0 tx=0).stop 2024-03-11T19:00:55.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.566+0000 7fcea907e700 1 -- 172.21.15.29:0/377077129 shutdown_connections 2024-03-11T19:00:55.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.566+0000 7fcea907e700 1 --2- 172.21.15.29:0/377077129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcea40fffc0 0x7fcea41003e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:55.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.566+0000 7fcea907e700 1 -- 172.21.15.29:0/377077129 >> 172.21.15.29:0/377077129 conn(0x7fcea40fbb80 msgr2=0x7fcea40fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:55.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.566+0000 7fcea907e700 1 -- 172.21.15.29:0/377077129 shutdown_connections 2024-03-11T19:00:55.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.566+0000 7fcea907e700 1 -- 172.21.15.29:0/377077129 wait complete. 2024-03-11T19:00:55.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.567+0000 7fcea907e700 1 Processor -- start 2024-03-11T19:00:55.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.567+0000 7fcea907e700 1 -- start start 2024-03-11T19:00:55.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.567+0000 7fcea907e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcea40fffc0 0x7fcea4198d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:55.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.567+0000 7fcea907e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea4199270 con 0x7fcea40fffc0 2024-03-11T19:00:55.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.567+0000 7fcea2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcea40fffc0 0x7fcea4198d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:55.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.567+0000 7fcea2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcea40fffc0 0x7fcea4198d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45142/0 (socket says 172.21.15.29:45142) 2024-03-11T19:00:55.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.567+0000 7fcea2d9d700 1 -- 172.21.15.29:0/2135393082 learned_addr learned my addr 172.21.15.29:0/2135393082 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:55.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.568+0000 7fcea2d9d700 1 -- 172.21.15.29:0/2135393082 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce980095f0 con 0x7fcea40fffc0 2024-03-11T19:00:55.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.568+0000 7fcea2d9d700 1 --2- 172.21.15.29:0/2135393082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcea40fffc0 0x7fcea4198d30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fce98005b40 tx=0x7fce98004f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:55.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.568+0000 7fce8bfff700 1 -- 172.21.15.29:0/2135393082 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce9801d070 con 0x7fcea40fffc0 2024-03-11T19:00:55.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.568+0000 7fce8bfff700 1 -- 172.21.15.29:0/2135393082 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fce9800be10 con 0x7fcea40fffc0 2024-03-11T19:00:55.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.568+0000 7fcea907e700 1 -- 172.21.15.29:0/2135393082 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea4199470 con 0x7fcea40fffc0 2024-03-11T19:00:55.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.568+0000 7fce8bfff700 1 -- 172.21.15.29:0/2135393082 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce98022cf0 con 0x7fcea40fffc0 2024-03-11T19:00:55.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.568+0000 7fcea907e700 1 -- 172.21.15.29:0/2135393082 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea4199910 con 0x7fcea40fffc0 2024-03-11T19:00:55.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.569+0000 7fce8bfff700 1 -- 172.21.15.29:0/2135393082 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7fce9800f460 con 0x7fcea40fffc0 2024-03-11T19:00:55.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.569+0000 7fcea907e700 1 -- 172.21.15.29:0/2135393082 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea41925c0 con 0x7fcea40fffc0 2024-03-11T19:00:55.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.569+0000 7fce8bfff700 1 --2- 172.21.15.29:0/2135393082 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fce8c038d50 0x7fce8c03b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:55.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.569+0000 7fce8bfff700 1 -- 172.21.15.29:0/2135393082 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fce9804ce30 con 0x7fcea40fffc0 2024-03-11T19:00:55.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.569+0000 7fcea259c700 1 --2- 172.21.15.29:0/2135393082 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fce8c038d50 0x7fce8c03b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:55.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.570+0000 7fcea259c700 1 --2- 172.21.15.29:0/2135393082 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fce8c038d50 0x7fce8c03b210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fce94006fd0 tx=0x7fce94006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:55.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.574+0000 7fce8bfff700 1 -- 172.21.15.29:0/2135393082 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fce98017440 con 0x7fcea40fffc0 2024-03-11T19:00:55.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.778+0000 7fcea907e700 1 -- 172.21.15.29:0/2135393082 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fcea40632a0 con 0x7fcea40fffc0 2024-03-11T19:00:55.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.780+0000 7fce8bfff700 1 -- 172.21.15.29:0/2135393082 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fce98027030 con 0x7fcea40fffc0 2024-03-11T19:00:55.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.783+0000 7fcea907e700 1 -- 172.21.15.29:0/2135393082 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fce8c038d50 msgr2=0x7fce8c03b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:55.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.783+0000 7fcea907e700 1 --2- 172.21.15.29:0/2135393082 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fce8c038d50 0x7fce8c03b210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fce94006fd0 tx=0x7fce94006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:55.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.783+0000 7fcea907e700 1 -- 172.21.15.29:0/2135393082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcea40fffc0 msgr2=0x7fcea4198d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:55.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.783+0000 7fcea907e700 1 --2- 172.21.15.29:0/2135393082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcea40fffc0 0x7fcea4198d30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fce98005b40 tx=0x7fce98004f40 comp rx=0 tx=0).stop 2024-03-11T19:00:55.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.783+0000 7fcea907e700 1 -- 172.21.15.29:0/2135393082 shutdown_connections 2024-03-11T19:00:55.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.783+0000 7fcea907e700 1 --2- 172.21.15.29:0/2135393082 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fce8c038d50 0x7fce8c03b210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:55.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.783+0000 7fcea907e700 1 --2- 172.21.15.29:0/2135393082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcea40fffc0 0x7fcea4198d30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:55.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.783+0000 7fcea907e700 1 -- 172.21.15.29:0/2135393082 >> 172.21.15.29:0/2135393082 conn(0x7fcea40fbb80 msgr2=0x7fcea40fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:55.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.783+0000 7fcea907e700 1 -- 172.21.15.29:0/2135393082 shutdown_connections 2024-03-11T19:00:55.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:55.783+0000 7fcea907e700 1 -- 172.21.15.29:0/2135393082 wait complete. 2024-03-11T19:00:55.819 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:55 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:00:55.819 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:55 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:00:55.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:55 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi029", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:00:55.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:55 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:00:55.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:55 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:00:55.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:55 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:00:56.651 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-11T19:00:56.651 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-11T19:00:56.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:56 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2135393082' entity='client.admin' 2024-03-11T19:00:56.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:56 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:00:56.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:00:56 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:00:57.540 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:00:59.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.076+0000 7f50b700c700 1 -- 172.21.15.29:0/3310703613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b01005a0 msgr2=0x7f50b01009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:59.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.076+0000 7f50b700c700 1 --2- 172.21.15.29:0/3310703613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b01005a0 0x7f50b01009c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f50a4009910 tx=0x7f50a4009c20 comp rx=0 tx=0).stop 2024-03-11T19:00:59.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.076+0000 7f50b700c700 1 -- 172.21.15.29:0/3310703613 shutdown_connections 2024-03-11T19:00:59.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.076+0000 7f50b700c700 1 --2- 172.21.15.29:0/3310703613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b01005a0 0x7f50b01009c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:59.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.076+0000 7f50b700c700 1 -- 172.21.15.29:0/3310703613 >> 172.21.15.29:0/3310703613 conn(0x7f50b00fbb40 msgr2=0x7f50b00fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:59.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.076+0000 7f50b700c700 1 -- 172.21.15.29:0/3310703613 shutdown_connections 2024-03-11T19:00:59.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.076+0000 7f50b700c700 1 -- 172.21.15.29:0/3310703613 wait complete. 2024-03-11T19:00:59.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.077+0000 7f50b700c700 1 Processor -- start 2024-03-11T19:00:59.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.077+0000 7f50b700c700 1 -- start start 2024-03-11T19:00:59.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.077+0000 7f50b700c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b01005a0 0x7f50b0196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:59.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.077+0000 7f50b700c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b0197030 con 0x7f50b01005a0 2024-03-11T19:00:59.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.077+0000 7f50b4da8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b01005a0 0x7f50b0196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:59.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.077+0000 7f50b4da8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b01005a0 0x7f50b0196af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45146/0 (socket says 172.21.15.29:45146) 2024-03-11T19:00:59.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.077+0000 7f50b4da8700 1 -- 172.21.15.29:0/1892936093 learned_addr learned my addr 172.21.15.29:0/1892936093 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:00:59.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.077+0000 7f50b4da8700 1 -- 172.21.15.29:0/1892936093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a40095f0 con 0x7f50b01005a0 2024-03-11T19:00:59.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.077+0000 7f50b4da8700 1 --2- 172.21.15.29:0/1892936093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b01005a0 0x7f50b0196af0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f50a4004f70 tx=0x7f50a4004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:59.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.078+0000 7f50adffb700 1 -- 172.21.15.29:0/1892936093 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50a401d070 con 0x7f50b01005a0 2024-03-11T19:00:59.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.078+0000 7f50adffb700 1 -- 172.21.15.29:0/1892936093 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f50a400be10 con 0x7f50b01005a0 2024-03-11T19:00:59.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.078+0000 7f50b700c700 1 -- 172.21.15.29:0/1892936093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b0197230 con 0x7f50b01005a0 2024-03-11T19:00:59.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.078+0000 7f50adffb700 1 -- 172.21.15.29:0/1892936093 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50a4022e90 con 0x7f50b01005a0 2024-03-11T19:00:59.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.078+0000 7f50b700c700 1 -- 172.21.15.29:0/1892936093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b01976d0 con 0x7f50b01005a0 2024-03-11T19:00:59.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.079+0000 7f50adffb700 1 -- 172.21.15.29:0/1892936093 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f50a400b960 con 0x7f50b01005a0 2024-03-11T19:00:59.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.079+0000 7f50b700c700 1 -- 172.21.15.29:0/1892936093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b0190560 con 0x7f50b01005a0 2024-03-11T19:00:59.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.079+0000 7f50adffb700 1 --2- 172.21.15.29:0/1892936093 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f50a0038d50 0x7f50a003b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:00:59.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.079+0000 7f50adffb700 1 -- 172.21.15.29:0/1892936093 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f50a404ce10 con 0x7f50b01005a0 2024-03-11T19:00:59.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.079+0000 7f50affff700 1 --2- 172.21.15.29:0/1892936093 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f50a0038d50 0x7f50a003b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:00:59.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.079+0000 7f50affff700 1 --2- 172.21.15.29:0/1892936093 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f50a0038d50 0x7f50a003b210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f509c006fd0 tx=0x7f509c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:00:59.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.084+0000 7f50adffb700 1 -- 172.21.15.29:0/1892936093 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f50a400f460 con 0x7f50b01005a0 2024-03-11T19:00:59.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.291+0000 7f50b700c700 1 -- 172.21.15.29:0/1892936093 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f50b002d7d0 con 0x7f50a0038d50 2024-03-11T19:00:59.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.295+0000 7f50adffb700 1 -- 172.21.15.29:0/1892936093 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f50b002d7d0 con 0x7f50a0038d50 2024-03-11T19:00:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.297+0000 7f50b700c700 1 -- 172.21.15.29:0/1892936093 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f50a0038d50 msgr2=0x7f50a003b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.297+0000 7f50b700c700 1 --2- 172.21.15.29:0/1892936093 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f50a0038d50 0x7f50a003b210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f509c006fd0 tx=0x7f509c006e40 comp rx=0 tx=0).stop 2024-03-11T19:00:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.297+0000 7f50b700c700 1 -- 172.21.15.29:0/1892936093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b01005a0 msgr2=0x7f50b0196af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:00:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.297+0000 7f50b700c700 1 --2- 172.21.15.29:0/1892936093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b01005a0 0x7f50b0196af0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f50a4004f70 tx=0x7f50a4004fa0 comp rx=0 tx=0).stop 2024-03-11T19:00:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.297+0000 7f50b700c700 1 -- 172.21.15.29:0/1892936093 shutdown_connections 2024-03-11T19:00:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.297+0000 7f50b700c700 1 --2- 172.21.15.29:0/1892936093 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f50a0038d50 0x7f50a003b210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.297+0000 7f50b700c700 1 --2- 172.21.15.29:0/1892936093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b01005a0 0x7f50b0196af0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:00:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.297+0000 7f50b700c700 1 -- 172.21.15.29:0/1892936093 >> 172.21.15.29:0/1892936093 conn(0x7f50b00fbb40 msgr2=0x7f50b0069dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:00:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.297+0000 7f50b700c700 1 -- 172.21.15.29:0/1892936093 shutdown_connections 2024-03-11T19:00:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:00:59.297+0000 7f50b700c700 1 -- 172.21.15.29:0/1892936093 wait complete. 2024-03-11T19:01:00.043 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi170 2024-03-11T19:01:00.043 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-11T19:01:00.043 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/etc/ceph/ceph.conf 2024-03-11T19:01:00.062 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-11T19:01:00.062 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:01:00.119 INFO:tasks.cephadm:Adding host smithi170 to orchestrator... 2024-03-11T19:01:00.119 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch host add smithi170 2024-03-11T19:01:00.357 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:01:00.565 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:00 smithi029 ceph-mon[32439]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:01:00.565 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:00.565 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:01:00.565 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:00.565 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:01:00.565 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:00 smithi029 ceph-mon[32439]: Updating smithi029:/etc/ceph/ceph.conf 2024-03-11T19:01:00.565 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:00 smithi029 ceph-mon[32439]: Updating smithi029:/var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:01:00.565 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:00.566 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:00.566 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:01.651 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:01 smithi029 ceph-mon[32439]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:01:01.651 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:01 smithi029 ceph-mon[32439]: Updating smithi029:/var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T19:01:01.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.887+0000 7fa79ed78700 1 -- 172.21.15.29:0/3228979656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa798106620 msgr2=0x7fa798106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:01.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.887+0000 7fa79ed78700 1 --2- 172.21.15.29:0/3228979656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa798106620 0x7fa798106a00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa78c009910 tx=0x7fa78c009c20 comp rx=0 tx=0).stop 2024-03-11T19:01:01.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.887+0000 7fa79ed78700 1 -- 172.21.15.29:0/3228979656 shutdown_connections 2024-03-11T19:01:01.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.887+0000 7fa79ed78700 1 --2- 172.21.15.29:0/3228979656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa798106620 0x7fa798106a00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:01.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.887+0000 7fa79ed78700 1 -- 172.21.15.29:0/3228979656 >> 172.21.15.29:0/3228979656 conn(0x7fa7980fc060 msgr2=0x7fa7980fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:01.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.887+0000 7fa79ed78700 1 -- 172.21.15.29:0/3228979656 shutdown_connections 2024-03-11T19:01:01.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.887+0000 7fa79ed78700 1 -- 172.21.15.29:0/3228979656 wait complete. 2024-03-11T19:01:01.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.888+0000 7fa79ed78700 1 Processor -- start 2024-03-11T19:01:01.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.888+0000 7fa79ed78700 1 -- start start 2024-03-11T19:01:01.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.888+0000 7fa79ed78700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa798106620 0x7fa798196de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:01.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.888+0000 7fa79ed78700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa798197320 con 0x7fa798106620 2024-03-11T19:01:01.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.888+0000 7fa79cb14700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa798106620 0x7fa798196de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:01.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.888+0000 7fa79cb14700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa798106620 0x7fa798196de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45178/0 (socket says 172.21.15.29:45178) 2024-03-11T19:01:01.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.888+0000 7fa79cb14700 1 -- 172.21.15.29:0/2193502100 learned_addr learned my addr 172.21.15.29:0/2193502100 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:01:01.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.888+0000 7fa79cb14700 1 -- 172.21.15.29:0/2193502100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78c0095f0 con 0x7fa798106620 2024-03-11T19:01:01.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.888+0000 7fa79cb14700 1 --2- 172.21.15.29:0/2193502100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa798106620 0x7fa798196de0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa78c004f40 tx=0x7fa78c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:01.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.888+0000 7fa795ffb700 1 -- 172.21.15.29:0/2193502100 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa78c01d070 con 0x7fa798106620 2024-03-11T19:01:01.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.888+0000 7fa795ffb700 1 -- 172.21.15.29:0/2193502100 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa78c00b750 con 0x7fa798106620 2024-03-11T19:01:01.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.889+0000 7fa79ed78700 1 -- 172.21.15.29:0/2193502100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa798197580 con 0x7fa798106620 2024-03-11T19:01:01.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.889+0000 7fa795ffb700 1 -- 172.21.15.29:0/2193502100 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa78c0037d0 con 0x7fa798106620 2024-03-11T19:01:01.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.889+0000 7fa79ed78700 1 -- 172.21.15.29:0/2193502100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa798197a80 con 0x7fa798106620 2024-03-11T19:01:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.890+0000 7fa795ffb700 1 -- 172.21.15.29:0/2193502100 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7fa78c003a40 con 0x7fa798106620 2024-03-11T19:01:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.890+0000 7fa79ed78700 1 -- 172.21.15.29:0/2193502100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7980632a0 con 0x7fa798106620 2024-03-11T19:01:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.890+0000 7fa795ffb700 1 --2- 172.21.15.29:0/2193502100 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa788038db0 0x7fa78803b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.890+0000 7fa795ffb700 1 -- 172.21.15.29:0/2193502100 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa78c04c8a0 con 0x7fa798106620 2024-03-11T19:01:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.890+0000 7fa797fff700 1 --2- 172.21.15.29:0/2193502100 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa788038db0 0x7fa78803b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.890+0000 7fa797fff700 1 --2- 172.21.15.29:0/2193502100 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa788038db0 0x7fa78803b270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa784006fd0 tx=0x7fa784006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:01.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:01.895+0000 7fa795ffb700 1 -- 172.21.15.29:0/2193502100 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa78c02ae20 con 0x7fa798106620 2024-03-11T19:01:02.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:02.103+0000 7fa79ed78700 1 -- 172.21.15.29:0/2193502100 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}) v1 -- 0x7fa798197a80 con 0x7fa788038db0 2024-03-11T19:01:02.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:02 smithi029 ceph-mon[32439]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:01:03.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:03 smithi029 ceph-mon[32439]: Deploying cephadm binary to smithi170 2024-03-11T19:01:04.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:04 smithi029 ceph-mon[32439]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:06.563 INFO:teuthology.orchestra.run.smithi029.stdout:Added host 'smithi170' with addr '172.21.15.170' 2024-03-11T19:01:06.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:06.560+0000 7fa795ffb700 1 -- 172.21.15.29:0/2193502100 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fa798197a80 con 0x7fa788038db0 2024-03-11T19:01:06.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:06.562+0000 7fa79ed78700 1 -- 172.21.15.29:0/2193502100 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa788038db0 msgr2=0x7fa78803b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:06.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:06.562+0000 7fa79ed78700 1 --2- 172.21.15.29:0/2193502100 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa788038db0 0x7fa78803b270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa784006fd0 tx=0x7fa784006e40 comp rx=0 tx=0).stop 2024-03-11T19:01:06.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:06.562+0000 7fa79ed78700 1 -- 172.21.15.29:0/2193502100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa798106620 msgr2=0x7fa798196de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:06.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:06.562+0000 7fa79ed78700 1 --2- 172.21.15.29:0/2193502100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa798106620 0x7fa798196de0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa78c004f40 tx=0x7fa78c005020 comp rx=0 tx=0).stop 2024-03-11T19:01:06.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:06.563+0000 7fa79ed78700 1 -- 172.21.15.29:0/2193502100 shutdown_connections 2024-03-11T19:01:06.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:06.563+0000 7fa79ed78700 1 --2- 172.21.15.29:0/2193502100 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa788038db0 0x7fa78803b270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:06.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:06.563+0000 7fa79ed78700 1 --2- 172.21.15.29:0/2193502100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa798106620 0x7fa798196de0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:06.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:06.563+0000 7fa79ed78700 1 -- 172.21.15.29:0/2193502100 >> 172.21.15.29:0/2193502100 conn(0x7fa7980fc060 msgr2=0x7fa798069ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:06.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:06.563+0000 7fa79ed78700 1 -- 172.21.15.29:0/2193502100 shutdown_connections 2024-03-11T19:01:06.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:06.563+0000 7fa79ed78700 1 -- 172.21.15.29:0/2193502100 wait complete. 2024-03-11T19:01:06.873 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:06 smithi029 ceph-mon[32439]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:06.873 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:06 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:07.184 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T19:01:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:01:07.833 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:07 smithi029 ceph-mon[32439]: Added host smithi170 2024-03-11T19:01:07.833 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:01:07.833 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:08.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:08 smithi029 ceph-mon[32439]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:08.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:08.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.937+0000 7f9bea55e700 1 -- 172.21.15.29:0/1649403096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9be4102590 msgr2=0x7f9be40ffaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:08.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.937+0000 7f9bea55e700 1 --2- 172.21.15.29:0/1649403096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9be4102590 0x7f9be40ffaa0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4009910 tx=0x7f9bd4009c20 comp rx=0 tx=0).stop 2024-03-11T19:01:08.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.937+0000 7f9bea55e700 1 -- 172.21.15.29:0/1649403096 shutdown_connections 2024-03-11T19:01:08.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.937+0000 7f9bea55e700 1 --2- 172.21.15.29:0/1649403096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9be4102590 0x7f9be40ffaa0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:08.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.937+0000 7f9bea55e700 1 -- 172.21.15.29:0/1649403096 >> 172.21.15.29:0/1649403096 conn(0x7f9be40fb7c0 msgr2=0x7f9be40fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:08.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.938+0000 7f9bea55e700 1 -- 172.21.15.29:0/1649403096 shutdown_connections 2024-03-11T19:01:08.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.938+0000 7f9bea55e700 1 -- 172.21.15.29:0/1649403096 wait complete. 2024-03-11T19:01:08.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.938+0000 7f9bea55e700 1 Processor -- start 2024-03-11T19:01:08.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.938+0000 7f9bea55e700 1 -- start start 2024-03-11T19:01:08.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.938+0000 7f9bea55e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9be4102590 0x7f9be41990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:08.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.938+0000 7f9bea55e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be4199630 con 0x7f9be4102590 2024-03-11T19:01:08.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.939+0000 7f9be3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9be4102590 0x7f9be41990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:08.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.939+0000 7f9be3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9be4102590 0x7f9be41990f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32824/0 (socket says 172.21.15.29:32824) 2024-03-11T19:01:08.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.939+0000 7f9be3fff700 1 -- 172.21.15.29:0/2060931765 learned_addr learned my addr 172.21.15.29:0/2060931765 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:01:08.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.939+0000 7f9be3fff700 1 -- 172.21.15.29:0/2060931765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd40095f0 con 0x7f9be4102590 2024-03-11T19:01:08.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.939+0000 7f9be3fff700 1 --2- 172.21.15.29:0/2060931765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9be4102590 0x7f9be41990f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4004f40 tx=0x7f9bd4005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:08.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.939+0000 7f9be17fa700 1 -- 172.21.15.29:0/2060931765 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bd401d070 con 0x7f9be4102590 2024-03-11T19:01:08.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.939+0000 7f9be17fa700 1 -- 172.21.15.29:0/2060931765 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9bd400bc00 con 0x7f9be4102590 2024-03-11T19:01:08.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.939+0000 7f9bea55e700 1 -- 172.21.15.29:0/2060931765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be4199830 con 0x7f9be4102590 2024-03-11T19:01:08.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.939+0000 7f9be17fa700 1 -- 172.21.15.29:0/2060931765 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bd4003920 con 0x7f9be4102590 2024-03-11T19:01:08.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.939+0000 7f9bea55e700 1 -- 172.21.15.29:0/2060931765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be4199cd0 con 0x7f9be4102590 2024-03-11T19:01:08.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.940+0000 7f9be17fa700 1 -- 172.21.15.29:0/2060931765 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f9bd4003be0 con 0x7f9be4102590 2024-03-11T19:01:08.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.940+0000 7f9bea55e700 1 -- 172.21.15.29:0/2060931765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be40632a0 con 0x7f9be4102590 2024-03-11T19:01:08.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.940+0000 7f9be17fa700 1 --2- 172.21.15.29:0/2060931765 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9bcc038d50 0x7f9bcc03b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:08.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.940+0000 7f9be17fa700 1 -- 172.21.15.29:0/2060931765 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9bd404cc70 con 0x7f9be4102590 2024-03-11T19:01:08.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.941+0000 7f9be37fe700 1 --2- 172.21.15.29:0/2060931765 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9bcc038d50 0x7f9bcc03b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:08.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.941+0000 7f9be37fe700 1 --2- 172.21.15.29:0/2060931765 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9bcc038d50 0x7f9bcc03b210 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8006fd0 tx=0x7f9bd8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:08.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:08.945+0000 7f9be17fa700 1 -- 172.21.15.29:0/2060931765 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9bd402ae20 con 0x7f9be4102590 2024-03-11T19:01:09.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:09.153+0000 7f9bea55e700 1 -- 172.21.15.29:0/2060931765 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9be410dd10 con 0x7f9bcc038d50 2024-03-11T19:01:09.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:09.154+0000 7f9be17fa700 1 -- 172.21.15.29:0/2060931765 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f9be410dd10 con 0x7f9bcc038d50 2024-03-11T19:01:09.157 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:01:09.157 INFO:teuthology.orchestra.run.smithi029.stdout:[{"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}, {"addr": "172.21.15.170", "hostname": "smithi170", "labels": [], "status": ""}] 2024-03-11T19:01:09.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:09.157+0000 7f9bea55e700 1 -- 172.21.15.29:0/2060931765 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9bcc038d50 msgr2=0x7f9bcc03b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:09.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:09.157+0000 7f9bea55e700 1 --2- 172.21.15.29:0/2060931765 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9bcc038d50 0x7f9bcc03b210 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8006fd0 tx=0x7f9bd8006e40 comp rx=0 tx=0).stop 2024-03-11T19:01:09.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:09.157+0000 7f9bea55e700 1 -- 172.21.15.29:0/2060931765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9be4102590 msgr2=0x7f9be41990f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:09.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:09.157+0000 7f9bea55e700 1 --2- 172.21.15.29:0/2060931765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9be4102590 0x7f9be41990f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4004f40 tx=0x7f9bd4005020 comp rx=0 tx=0).stop 2024-03-11T19:01:09.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:09.157+0000 7f9bea55e700 1 -- 172.21.15.29:0/2060931765 shutdown_connections 2024-03-11T19:01:09.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:09.157+0000 7f9bea55e700 1 --2- 172.21.15.29:0/2060931765 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9bcc038d50 0x7f9bcc03b210 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:09.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:09.157+0000 7f9bea55e700 1 --2- 172.21.15.29:0/2060931765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9be4102590 0x7f9be41990f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:09.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:09.157+0000 7f9bea55e700 1 -- 172.21.15.29:0/2060931765 >> 172.21.15.29:0/2060931765 conn(0x7f9be40fb7c0 msgr2=0x7f9be40fc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:09.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:09.157+0000 7f9bea55e700 1 -- 172.21.15.29:0/2060931765 shutdown_connections 2024-03-11T19:01:09.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:09.157+0000 7f9bea55e700 1 -- 172.21.15.29:0/2060931765 wait complete. 2024-03-11T19:01:09.782 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi190 2024-03-11T19:01:09.783 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-11T19:01:09.783 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.conf 2024-03-11T19:01:09.801 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-11T19:01:09.801 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:01:09.858 INFO:tasks.cephadm:Adding host smithi190 to orchestrator... 2024-03-11T19:01:09.859 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch host add smithi190 2024-03-11T19:01:10.100 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:01:10.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:10 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:10.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:10 smithi029 ceph-mon[32439]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:01:11.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:11 smithi029 ceph-mon[32439]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:11.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.545+0000 7f07d7a5f700 1 -- 172.21.15.29:0/2878641437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d0107690 msgr2=0x7f07d0107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:11.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.545+0000 7f07d7a5f700 1 --2- 172.21.15.29:0/2878641437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d0107690 0x7f07d0107a70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f07cc009910 tx=0x7f07cc009c20 comp rx=0 tx=0).stop 2024-03-11T19:01:11.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.545+0000 7f07d7a5f700 1 -- 172.21.15.29:0/2878641437 shutdown_connections 2024-03-11T19:01:11.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.545+0000 7f07d7a5f700 1 --2- 172.21.15.29:0/2878641437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d0107690 0x7f07d0107a70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:11.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.545+0000 7f07d7a5f700 1 -- 172.21.15.29:0/2878641437 >> 172.21.15.29:0/2878641437 conn(0x7f07d00fd1a0 msgr2=0x7f07d00ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:11.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.546+0000 7f07d7a5f700 1 -- 172.21.15.29:0/2878641437 shutdown_connections 2024-03-11T19:01:11.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.546+0000 7f07d7a5f700 1 -- 172.21.15.29:0/2878641437 wait complete. 2024-03-11T19:01:11.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.546+0000 7f07d7a5f700 1 Processor -- start 2024-03-11T19:01:11.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.546+0000 7f07d7a5f700 1 -- start start 2024-03-11T19:01:11.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.546+0000 7f07d7a5f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d0107690 0x7f07d0073f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:11.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.546+0000 7f07d7a5f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d0075e60 con 0x7f07d0107690 2024-03-11T19:01:11.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.547+0000 7f07d57fb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d0107690 0x7f07d0073f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:11.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.547+0000 7f07d57fb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d0107690 0x7f07d0073f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32852/0 (socket says 172.21.15.29:32852) 2024-03-11T19:01:11.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.547+0000 7f07d57fb700 1 -- 172.21.15.29:0/3051777379 learned_addr learned my addr 172.21.15.29:0/3051777379 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:01:11.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.547+0000 7f07d57fb700 1 -- 172.21.15.29:0/3051777379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07cc0095f0 con 0x7f07d0107690 2024-03-11T19:01:11.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.547+0000 7f07d57fb700 1 --2- 172.21.15.29:0/3051777379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d0107690 0x7f07d0073f70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f07cc004f40 tx=0x7f07cc005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:11.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.547+0000 7f07c67fc700 1 -- 172.21.15.29:0/3051777379 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07cc01d070 con 0x7f07d0107690 2024-03-11T19:01:11.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.547+0000 7f07c67fc700 1 -- 172.21.15.29:0/3051777379 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f07cc00b6f0 con 0x7f07d0107690 2024-03-11T19:01:11.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.547+0000 7f07d7a5f700 1 -- 172.21.15.29:0/3051777379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d00744b0 con 0x7f07d0107690 2024-03-11T19:01:11.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.547+0000 7f07c67fc700 1 -- 172.21.15.29:0/3051777379 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07cc0037d0 con 0x7f07d0107690 2024-03-11T19:01:11.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.547+0000 7f07d7a5f700 1 -- 172.21.15.29:0/3051777379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d0074930 con 0x7f07d0107690 2024-03-11T19:01:11.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.548+0000 7f07c67fc700 1 -- 172.21.15.29:0/3051777379 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f07cc003a90 con 0x7f07d0107690 2024-03-11T19:01:11.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.548+0000 7f07d7a5f700 1 -- 172.21.15.29:0/3051777379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d00632a0 con 0x7f07d0107690 2024-03-11T19:01:11.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.548+0000 7f07c67fc700 1 --2- 172.21.15.29:0/3051777379 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f07bc038d50 0x7f07bc03b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:11.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.548+0000 7f07c67fc700 1 -- 172.21.15.29:0/3051777379 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f07cc04ca20 con 0x7f07d0107690 2024-03-11T19:01:11.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.548+0000 7f07d4ffa700 1 --2- 172.21.15.29:0/3051777379 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f07bc038d50 0x7f07bc03b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:11.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.549+0000 7f07d4ffa700 1 --2- 172.21.15.29:0/3051777379 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f07bc038d50 0x7f07bc03b210 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f07c0006fd0 tx=0x7f07c0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:11.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.553+0000 7f07c67fc700 1 -- 172.21.15.29:0/3051777379 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f07cc029720 con 0x7f07d0107690 2024-03-11T19:01:11.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:11.760+0000 7f07d7a5f700 1 -- 172.21.15.29:0/3051777379 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}) v1 -- 0x7f07d0074c30 con 0x7f07bc038d50 2024-03-11T19:01:13.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:13 smithi029 ceph-mon[32439]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:13 smithi029 ceph-mon[32439]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:01:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:13 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:13 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:13 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:13 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:13 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:01:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:13 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:13 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:01:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:13 smithi029 ceph-mon[32439]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-11T19:01:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:13 smithi029 ceph-mon[32439]: Updating smithi170:/var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:01:14.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:14 smithi029 ceph-mon[32439]: Deploying cephadm binary to smithi190 2024-03-11T19:01:14.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:14 smithi029 ceph-mon[32439]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:01:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:14 smithi029 ceph-mon[32439]: Updating smithi170:/var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T19:01:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:15.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:15 smithi029 ceph-mon[32439]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:16.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:16.348+0000 7f07c67fc700 1 -- 172.21.15.29:0/3051777379 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f07d0074c30 con 0x7f07bc038d50 2024-03-11T19:01:16.352 INFO:teuthology.orchestra.run.smithi029.stdout:Added host 'smithi190' with addr '172.21.15.190' 2024-03-11T19:01:16.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:16.351+0000 7f07d7a5f700 1 -- 172.21.15.29:0/3051777379 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f07bc038d50 msgr2=0x7f07bc03b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:16.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:16.351+0000 7f07d7a5f700 1 --2- 172.21.15.29:0/3051777379 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f07bc038d50 0x7f07bc03b210 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f07c0006fd0 tx=0x7f07c0006e40 comp rx=0 tx=0).stop 2024-03-11T19:01:16.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:16.351+0000 7f07d7a5f700 1 -- 172.21.15.29:0/3051777379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d0107690 msgr2=0x7f07d0073f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:16.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:16.351+0000 7f07d7a5f700 1 --2- 172.21.15.29:0/3051777379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d0107690 0x7f07d0073f70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f07cc004f40 tx=0x7f07cc005020 comp rx=0 tx=0).stop 2024-03-11T19:01:16.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:16.351+0000 7f07d7a5f700 1 -- 172.21.15.29:0/3051777379 shutdown_connections 2024-03-11T19:01:16.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:16.351+0000 7f07d7a5f700 1 --2- 172.21.15.29:0/3051777379 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f07bc038d50 0x7f07bc03b210 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:16.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:16.351+0000 7f07d7a5f700 1 --2- 172.21.15.29:0/3051777379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d0107690 0x7f07d0073f70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:16.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:16.351+0000 7f07d7a5f700 1 -- 172.21.15.29:0/3051777379 >> 172.21.15.29:0/3051777379 conn(0x7f07d00fd1a0 msgr2=0x7f07d00fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:16.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:16.351+0000 7f07d7a5f700 1 -- 172.21.15.29:0/3051777379 shutdown_connections 2024-03-11T19:01:16.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:16.351+0000 7f07d7a5f700 1 -- 172.21.15.29:0/3051777379 wait complete. 2024-03-11T19:01:16.990 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T19:01:17.235 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:01:17.638 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:17 smithi029 ceph-mon[32439]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:17.638 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:17 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:17.639 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:17 smithi029 ceph-mon[32439]: Added host smithi190 2024-03-11T19:01:17.639 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:17 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:01:17.639 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:17 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:18.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.594+0000 7f4496e54700 1 -- 172.21.15.29:0/3946587364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4490106590 msgr2=0x7f4490106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:18.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.594+0000 7f4496e54700 1 --2- 172.21.15.29:0/3946587364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4490106590 0x7f4490106970 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4484009910 tx=0x7f4484009c20 comp rx=0 tx=0).stop 2024-03-11T19:01:18.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.594+0000 7f4496e54700 1 -- 172.21.15.29:0/3946587364 shutdown_connections 2024-03-11T19:01:18.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.594+0000 7f4496e54700 1 --2- 172.21.15.29:0/3946587364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4490106590 0x7f4490106970 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:18.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.594+0000 7f4496e54700 1 -- 172.21.15.29:0/3946587364 >> 172.21.15.29:0/3946587364 conn(0x7f44900fb7c0 msgr2=0x7f44900fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:18.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.594+0000 7f4496e54700 1 -- 172.21.15.29:0/3946587364 shutdown_connections 2024-03-11T19:01:18.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.594+0000 7f4496e54700 1 -- 172.21.15.29:0/3946587364 wait complete. 2024-03-11T19:01:18.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.595+0000 7f4496e54700 1 Processor -- start 2024-03-11T19:01:18.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.595+0000 7f4496e54700 1 -- start start 2024-03-11T19:01:18.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.595+0000 7f4496e54700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4490106590 0x7f44900fbc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:18.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.595+0000 7f4496e54700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44900fdb60 con 0x7f4490106590 2024-03-11T19:01:18.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.595+0000 7f4494bf0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4490106590 0x7f44900fbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:18.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.595+0000 7f4494bf0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4490106590 0x7f44900fbc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53308/0 (socket says 172.21.15.29:53308) 2024-03-11T19:01:18.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.595+0000 7f4494bf0700 1 -- 172.21.15.29:0/847578395 learned_addr learned my addr 172.21.15.29:0/847578395 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:01:18.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.595+0000 7f4494bf0700 1 -- 172.21.15.29:0/847578395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44840095f0 con 0x7f4490106590 2024-03-11T19:01:18.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.595+0000 7f4494bf0700 1 --2- 172.21.15.29:0/847578395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4490106590 0x7f44900fbc70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4484004f40 tx=0x7f4484005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:18.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.595+0000 7f448dffb700 1 -- 172.21.15.29:0/847578395 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f448401d070 con 0x7f4490106590 2024-03-11T19:01:18.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.595+0000 7f448dffb700 1 -- 172.21.15.29:0/847578395 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f448400bc00 con 0x7f4490106590 2024-03-11T19:01:18.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.596+0000 7f4496e54700 1 -- 172.21.15.29:0/847578395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44900fc1b0 con 0x7f4490106590 2024-03-11T19:01:18.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.596+0000 7f448dffb700 1 -- 172.21.15.29:0/847578395 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4484003920 con 0x7f4490106590 2024-03-11T19:01:18.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.596+0000 7f4496e54700 1 -- 172.21.15.29:0/847578395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44900fc5d0 con 0x7f4490106590 2024-03-11T19:01:18.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.596+0000 7f448dffb700 1 -- 172.21.15.29:0/847578395 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f4484003be0 con 0x7f4490106590 2024-03-11T19:01:18.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.597+0000 7f4496e54700 1 -- 172.21.15.29:0/847578395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44900632a0 con 0x7f4490106590 2024-03-11T19:01:18.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.597+0000 7f448dffb700 1 --2- 172.21.15.29:0/847578395 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4480038d50 0x7f448003b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:18.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.597+0000 7f448dffb700 1 -- 172.21.15.29:0/847578395 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f448404ce00 con 0x7f4490106590 2024-03-11T19:01:18.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.597+0000 7f448ffff700 1 --2- 172.21.15.29:0/847578395 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4480038d50 0x7f448003b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:18.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.597+0000 7f448ffff700 1 --2- 172.21.15.29:0/847578395 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4480038d50 0x7f448003b210 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f447c006fd0 tx=0x7f447c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:18.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.602+0000 7f448dffb700 1 -- 172.21.15.29:0/847578395 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4484029950 con 0x7f4490106590 2024-03-11T19:01:18.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:18 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:18.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.809+0000 7f4496e54700 1 -- 172.21.15.29:0/847578395 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f449010a930 con 0x7f4480038d50 2024-03-11T19:01:18.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.811+0000 7f448dffb700 1 -- 172.21.15.29:0/847578395 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f449010a930 con 0x7f4480038d50 2024-03-11T19:01:18.813 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:01:18.813 INFO:teuthology.orchestra.run.smithi029.stdout:[{"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}, {"addr": "172.21.15.170", "hostname": "smithi170", "labels": [], "status": ""}, {"addr": "172.21.15.190", "hostname": "smithi190", "labels": [], "status": ""}] 2024-03-11T19:01:18.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.813+0000 7f4496e54700 1 -- 172.21.15.29:0/847578395 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4480038d50 msgr2=0x7f448003b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:18.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.813+0000 7f4496e54700 1 --2- 172.21.15.29:0/847578395 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4480038d50 0x7f448003b210 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f447c006fd0 tx=0x7f447c006e40 comp rx=0 tx=0).stop 2024-03-11T19:01:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.813+0000 7f4496e54700 1 -- 172.21.15.29:0/847578395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4490106590 msgr2=0x7f44900fbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.813+0000 7f4496e54700 1 --2- 172.21.15.29:0/847578395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4490106590 0x7f44900fbc70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4484004f40 tx=0x7f4484005020 comp rx=0 tx=0).stop 2024-03-11T19:01:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.813+0000 7f4496e54700 1 -- 172.21.15.29:0/847578395 shutdown_connections 2024-03-11T19:01:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.813+0000 7f4496e54700 1 --2- 172.21.15.29:0/847578395 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4480038d50 0x7f448003b210 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.813+0000 7f4496e54700 1 --2- 172.21.15.29:0/847578395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4490106590 0x7f44900fbc70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.813+0000 7f4496e54700 1 -- 172.21.15.29:0/847578395 >> 172.21.15.29:0/847578395 conn(0x7f44900fb7c0 msgr2=0x7f449018fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.813+0000 7f4496e54700 1 -- 172.21.15.29:0/847578395 shutdown_connections 2024-03-11T19:01:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:18.813+0000 7f4496e54700 1 -- 172.21.15.29:0/847578395 wait complete. 2024-03-11T19:01:19.426 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-11T19:01:19.426 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-11T19:01:19.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:19 smithi029 ceph-mon[32439]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:19.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:19 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:19.664 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:01:20.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:20 smithi029 ceph-mon[32439]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:01:21.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.168+0000 7f8dfee29700 1 -- 172.21.15.29:0/1156013343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df8107650 msgr2=0x7f8df8107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:21.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.168+0000 7f8dfee29700 1 --2- 172.21.15.29:0/1156013343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df8107650 0x7f8df8107a30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009910 tx=0x7f8de8009c20 comp rx=0 tx=0).stop 2024-03-11T19:01:21.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.168+0000 7f8dfee29700 1 -- 172.21.15.29:0/1156013343 shutdown_connections 2024-03-11T19:01:21.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.168+0000 7f8dfee29700 1 --2- 172.21.15.29:0/1156013343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df8107650 0x7f8df8107a30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:21.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.168+0000 7f8dfee29700 1 -- 172.21.15.29:0/1156013343 >> 172.21.15.29:0/1156013343 conn(0x7f8df80fb7c0 msgr2=0x7f8df80fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.168+0000 7f8dfee29700 1 -- 172.21.15.29:0/1156013343 shutdown_connections 2024-03-11T19:01:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.168+0000 7f8dfee29700 1 -- 172.21.15.29:0/1156013343 wait complete. 2024-03-11T19:01:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.169+0000 7f8dfee29700 1 Processor -- start 2024-03-11T19:01:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.169+0000 7f8dfee29700 1 -- start start 2024-03-11T19:01:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.169+0000 7f8dfee29700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df8107650 0x7f8df8196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.169+0000 7f8dfee29700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df8197360 con 0x7f8df8107650 2024-03-11T19:01:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.169+0000 7f8dfcbc5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df8107650 0x7f8df8196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.169+0000 7f8dfcbc5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df8107650 0x7f8df8196e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53322/0 (socket says 172.21.15.29:53322) 2024-03-11T19:01:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.169+0000 7f8dfcbc5700 1 -- 172.21.15.29:0/1597013024 learned_addr learned my addr 172.21.15.29:0/1597013024 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:01:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.169+0000 7f8dfcbc5700 1 -- 172.21.15.29:0/1597013024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de80095f0 con 0x7f8df8107650 2024-03-11T19:01:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.169+0000 7f8dfcbc5700 1 --2- 172.21.15.29:0/1597013024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df8107650 0x7f8df8196e20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8de8004f40 tx=0x7f8de8005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.169+0000 7f8df5ffb700 1 -- 172.21.15.29:0/1597013024 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8de801d070 con 0x7f8df8107650 2024-03-11T19:01:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.169+0000 7f8df5ffb700 1 -- 172.21.15.29:0/1597013024 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8de800b6f0 con 0x7f8df8107650 2024-03-11T19:01:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.169+0000 7f8df5ffb700 1 -- 172.21.15.29:0/1597013024 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8de80037d0 con 0x7f8df8107650 2024-03-11T19:01:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.169+0000 7f8dfee29700 1 -- 172.21.15.29:0/1597013024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df81975c0 con 0x7f8df8107650 2024-03-11T19:01:21.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.170+0000 7f8dfee29700 1 -- 172.21.15.29:0/1597013024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df8197a40 con 0x7f8df8107650 2024-03-11T19:01:21.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.171+0000 7f8df5ffb700 1 -- 172.21.15.29:0/1597013024 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f8de800b860 con 0x7f8df8107650 2024-03-11T19:01:21.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.171+0000 7f8dfee29700 1 -- 172.21.15.29:0/1597013024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df8102ec0 con 0x7f8df8107650 2024-03-11T19:01:21.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.171+0000 7f8df5ffb700 1 --2- 172.21.15.29:0/1597013024 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8de0038d50 0x7f8de003b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:21.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.171+0000 7f8df5ffb700 1 -- 172.21.15.29:0/1597013024 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8de804ca50 con 0x7f8df8107650 2024-03-11T19:01:21.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.171+0000 7f8df7fff700 1 --2- 172.21.15.29:0/1597013024 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8de0038d50 0x7f8de003b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:21.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.171+0000 7f8df7fff700 1 --2- 172.21.15.29:0/1597013024 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8de0038d50 0x7f8de003b210 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8dec006fd0 tx=0x7f8dec006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:21.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.176+0000 7f8df5ffb700 1 -- 172.21.15.29:0/1597013024 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8de800edd0 con 0x7f8df8107650 2024-03-11T19:01:21.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:21.381+0000 7f8dfee29700 1 -- 172.21.15.29:0/1597013024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f8df80632a0 con 0x7f8df8107650 2024-03-11T19:01:21.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:21 smithi029 ceph-mon[32439]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:22.369+0000 7f8df5ffb700 1 -- 172.21.15.29:0/1597013024 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f8df80632a0 con 0x7f8df8107650 2024-03-11T19:01:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:22.372+0000 7f8dfee29700 1 -- 172.21.15.29:0/1597013024 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8de0038d50 msgr2=0x7f8de003b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:22.372+0000 7f8dfee29700 1 --2- 172.21.15.29:0/1597013024 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8de0038d50 0x7f8de003b210 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8dec006fd0 tx=0x7f8dec006e40 comp rx=0 tx=0).stop 2024-03-11T19:01:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:22.372+0000 7f8dfee29700 1 -- 172.21.15.29:0/1597013024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df8107650 msgr2=0x7f8df8196e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:22.372+0000 7f8dfee29700 1 --2- 172.21.15.29:0/1597013024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df8107650 0x7f8df8196e20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8de8004f40 tx=0x7f8de8005020 comp rx=0 tx=0).stop 2024-03-11T19:01:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:22.372+0000 7f8dfee29700 1 -- 172.21.15.29:0/1597013024 shutdown_connections 2024-03-11T19:01:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:22.372+0000 7f8dfee29700 1 --2- 172.21.15.29:0/1597013024 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8de0038d50 0x7f8de003b210 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:22.372+0000 7f8dfee29700 1 --2- 172.21.15.29:0/1597013024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df8107650 0x7f8df8196e20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:22.372+0000 7f8dfee29700 1 -- 172.21.15.29:0/1597013024 >> 172.21.15.29:0/1597013024 conn(0x7f8df80fb7c0 msgr2=0x7f8df80fbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:22.372+0000 7f8dfee29700 1 -- 172.21.15.29:0/1597013024 shutdown_connections 2024-03-11T19:01:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:22.372+0000 7f8dfee29700 1 -- 172.21.15.29:0/1597013024 wait complete. 2024-03-11T19:01:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2024-03-11T19:01:22.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:22 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1597013024' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T19:01:22.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:22 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:22.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:22 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:22.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:22 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:22.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:22 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:22.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:22 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi190", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:01:22.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:22 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:22.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:22 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:01:23.051 INFO:tasks.cephadm:Adding mon.a on smithi029 2024-03-11T19:01:23.051 INFO:tasks.cephadm:Adding mon.b on smithi170 2024-03-11T19:01:23.051 INFO:tasks.cephadm:Adding mon.c on smithi190 2024-03-11T19:01:23.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi029:172.21.15.29=a;smithi170:172.21.15.170=b;smithi190:172.21.15.190=c' 2024-03-11T19:01:23.285 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:01:23.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:23 smithi029 ceph-mon[32439]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:23.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:23 smithi029 ceph-mon[32439]: Updating smithi190:/etc/ceph/ceph.conf 2024-03-11T19:01:23.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:23 smithi029 ceph-mon[32439]: Updating smithi190:/var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:01:23.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:23 smithi029 ceph-mon[32439]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:01:23.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:23 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1597013024' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T19:01:23.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:23 smithi029 ceph-mon[32439]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T19:01:23.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:23 smithi029 ceph-mon[32439]: Updating smithi190:/var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T19:01:23.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:23.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:23.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:24.686 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.678+0000 7f5bac5af700 1 -- 172.21.15.190:0/2424114839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba4107650 msgr2=0x7f5ba4107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:24.686 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.678+0000 7f5bac5af700 1 --2- 172.21.15.190:0/2424114839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba4107650 0x7f5ba4107a30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0009910 tx=0x7f5ba0009c20 comp rx=0 tx=0).stop 2024-03-11T19:01:24.686 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.678+0000 7f5bac5af700 1 -- 172.21.15.190:0/2424114839 shutdown_connections 2024-03-11T19:01:24.686 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.678+0000 7f5bac5af700 1 --2- 172.21.15.190:0/2424114839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba4107650 0x7f5ba4107a30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:24.686 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.678+0000 7f5bac5af700 1 -- 172.21.15.190:0/2424114839 >> 172.21.15.190:0/2424114839 conn(0x7f5ba40fd180 msgr2=0x7f5ba40ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:24.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.678+0000 7f5bac5af700 1 -- 172.21.15.190:0/2424114839 shutdown_connections 2024-03-11T19:01:24.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.678+0000 7f5bac5af700 1 -- 172.21.15.190:0/2424114839 wait complete. 2024-03-11T19:01:24.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.679+0000 7f5bac5af700 1 Processor -- start 2024-03-11T19:01:24.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.679+0000 7f5bac5af700 1 -- start start 2024-03-11T19:01:24.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.679+0000 7f5bac5af700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba4107650 0x7f5ba4196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:24.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.679+0000 7f5bac5af700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba41973e0 con 0x7f5ba4107650 2024-03-11T19:01:24.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.679+0000 7f5baa34b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba4107650 0x7f5ba4196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:24.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.679+0000 7f5baa34b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba4107650 0x7f5ba4196ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.190:54742/0 (socket says 172.21.15.190:54742) 2024-03-11T19:01:24.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.679+0000 7f5baa34b700 1 -- 172.21.15.190:0/3588700394 learned_addr learned my addr 172.21.15.190:0/3588700394 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-11T19:01:24.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.680+0000 7f5baa34b700 1 -- 172.21.15.190:0/3588700394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba00095f0 con 0x7f5ba4107650 2024-03-11T19:01:24.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.680+0000 7f5baa34b700 1 --2- 172.21.15.190:0/3588700394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba4107650 0x7f5ba4196ea0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0004f40 tx=0x7f5ba0005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:24.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.680+0000 7f5b9b7fe700 1 -- 172.21.15.190:0/3588700394 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ba001d070 con 0x7f5ba4107650 2024-03-11T19:01:24.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.680+0000 7f5b9b7fe700 1 -- 172.21.15.190:0/3588700394 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5ba000b6f0 con 0x7f5ba4107650 2024-03-11T19:01:24.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.680+0000 7f5bac5af700 1 -- 172.21.15.190:0/3588700394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba4197640 con 0x7f5ba4107650 2024-03-11T19:01:24.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.680+0000 7f5b9b7fe700 1 -- 172.21.15.190:0/3588700394 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ba00037d0 con 0x7f5ba4107650 2024-03-11T19:01:24.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.680+0000 7f5bac5af700 1 -- 172.21.15.190:0/3588700394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba4197ac0 con 0x7f5ba4107650 2024-03-11T19:01:24.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.681+0000 7f5b9b7fe700 1 -- 172.21.15.190:0/3588700394 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f5ba0004240 con 0x7f5ba4107650 2024-03-11T19:01:24.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.681+0000 7f5bac5af700 1 -- 172.21.15.190:0/3588700394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba40632a0 con 0x7f5ba4107650 2024-03-11T19:01:24.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.681+0000 7f5b9b7fe700 1 --2- 172.21.15.190:0/3588700394 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5b90038db0 0x7f5b9003b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:24.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.681+0000 7f5b9b7fe700 1 -- 172.21.15.190:0/3588700394 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5ba004ca90 con 0x7f5ba4107650 2024-03-11T19:01:24.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.681+0000 7f5ba9b4a700 1 --2- 172.21.15.190:0/3588700394 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5b90038db0 0x7f5b9003b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:24.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.682+0000 7f5ba9b4a700 1 --2- 172.21.15.190:0/3588700394 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5b90038db0 0x7f5b9003b270 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5b94006fd0 tx=0x7f5b94006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:24.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.686+0000 7f5b9b7fe700 1 -- 172.21.15.190:0/3588700394 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5ba0029360 con 0x7f5ba4107650 2024-03-11T19:01:24.895 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.892+0000 7f5bac5af700 1 -- 172.21.15.190:0/3588700394 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi170:172.21.15.170=b;smithi190:172.21.15.190=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ba410baf0 con 0x7f5b90038db0 2024-03-11T19:01:24.900 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.898+0000 7f5b9b7fe700 1 -- 172.21.15.190:0/3588700394 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5ba410baf0 con 0x7f5b90038db0 2024-03-11T19:01:24.900 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled mon update... 2024-03-11T19:01:24.902 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.899+0000 7f5bac5af700 1 -- 172.21.15.190:0/3588700394 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5b90038db0 msgr2=0x7f5b9003b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:24.902 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.899+0000 7f5bac5af700 1 --2- 172.21.15.190:0/3588700394 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5b90038db0 0x7f5b9003b270 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5b94006fd0 tx=0x7f5b94006e40 comp rx=0 tx=0).stop 2024-03-11T19:01:24.902 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.899+0000 7f5bac5af700 1 -- 172.21.15.190:0/3588700394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba4107650 msgr2=0x7f5ba4196ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:24.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.899+0000 7f5bac5af700 1 --2- 172.21.15.190:0/3588700394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba4107650 0x7f5ba4196ea0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0004f40 tx=0x7f5ba0005020 comp rx=0 tx=0).stop 2024-03-11T19:01:24.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.900+0000 7f5bac5af700 1 -- 172.21.15.190:0/3588700394 shutdown_connections 2024-03-11T19:01:24.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.900+0000 7f5bac5af700 1 --2- 172.21.15.190:0/3588700394 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5b90038db0 0x7f5b9003b270 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:24.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.900+0000 7f5bac5af700 1 --2- 172.21.15.190:0/3588700394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ba4107650 0x7f5ba4196ea0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:24.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.900+0000 7f5bac5af700 1 -- 172.21.15.190:0/3588700394 >> 172.21.15.190:0/3588700394 conn(0x7f5ba40fd180 msgr2=0x7f5ba41091a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:24.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.900+0000 7f5bac5af700 1 -- 172.21.15.190:0/3588700394 shutdown_connections 2024-03-11T19:01:24.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:24.900+0000 7f5bac5af700 1 -- 172.21.15.190:0/3588700394 wait complete. 2024-03-11T19:01:25.467 DEBUG:teuthology.orchestra.run.smithi170:mon.b> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mon.b.service 2024-03-11T19:01:25.470 DEBUG:teuthology.orchestra.run.smithi190:mon.c> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mon.c.service 2024-03-11T19:01:25.472 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T19:01:25.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T19:01:25.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:25 smithi029 ceph-mon[32439]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:25.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:25 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:25.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:25 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:01:25.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:25 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:25.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:25 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:01:25.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:25 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:25.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:25 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:01:25.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:25 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:26.543 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:01:26.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:26 smithi029 ceph-mon[32439]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi170:172.21.15.170=b;smithi190:172.21.15.190=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:01:26.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:26 smithi029 ceph-mon[32439]: Saving service mon spec with placement smithi029:172.21.15.29=a;smithi170:172.21.15.170=b;smithi190:172.21.15.190=c;count:3 2024-03-11T19:01:26.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:26 smithi029 ceph-mon[32439]: Deploying daemon mon.c on smithi190 2024-03-11T19:01:27.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:27 smithi029 ceph-mon[32439]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:29.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.185+0000 7ff6f93be700 1 -- 172.21.15.190:0/1501111822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f4074010 msgr2=0x7ff6f40743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:29.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.185+0000 7ff6f93be700 1 --2- 172.21.15.190:0/1501111822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f4074010 0x7ff6f40743f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4009910 tx=0x7ff6e4009c20 comp rx=0 tx=0).stop 2024-03-11T19:01:29.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.186+0000 7ff6f93be700 1 -- 172.21.15.190:0/1501111822 shutdown_connections 2024-03-11T19:01:29.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.186+0000 7ff6f93be700 1 --2- 172.21.15.190:0/1501111822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f4074010 0x7ff6f40743f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:29.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.186+0000 7ff6f93be700 1 -- 172.21.15.190:0/1501111822 >> 172.21.15.190:0/1501111822 conn(0x7ff6f40fd180 msgr2=0x7ff6f40ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:29.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.186+0000 7ff6f93be700 1 -- 172.21.15.190:0/1501111822 shutdown_connections 2024-03-11T19:01:29.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.186+0000 7ff6f93be700 1 -- 172.21.15.190:0/1501111822 wait complete. 2024-03-11T19:01:29.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.186+0000 7ff6f93be700 1 Processor -- start 2024-03-11T19:01:29.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.186+0000 7ff6f93be700 1 -- start start 2024-03-11T19:01:29.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.186+0000 7ff6f93be700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f4074010 0x7ff6f419d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:29.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.186+0000 7ff6f93be700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f419da20 con 0x7ff6f4074010 2024-03-11T19:01:29.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.187+0000 7ff6f2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f4074010 0x7ff6f419d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:29.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.187+0000 7ff6f2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f4074010 0x7ff6f419d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.190:43816/0 (socket says 172.21.15.190:43816) 2024-03-11T19:01:29.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.187+0000 7ff6f2ffd700 1 -- 172.21.15.190:0/2866050704 learned_addr learned my addr 172.21.15.190:0/2866050704 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-11T19:01:29.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.187+0000 7ff6f2ffd700 1 -- 172.21.15.190:0/2866050704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e40095f0 con 0x7ff6f4074010 2024-03-11T19:01:29.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.187+0000 7ff6f2ffd700 1 --2- 172.21.15.190:0/2866050704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f4074010 0x7ff6f419d4e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4004f40 tx=0x7ff6e4005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:29.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.187+0000 7ff6dbfff700 1 -- 172.21.15.190:0/2866050704 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff6e401d070 con 0x7ff6f4074010 2024-03-11T19:01:29.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.187+0000 7ff6f93be700 1 -- 172.21.15.190:0/2866050704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f419dc80 con 0x7ff6f4074010 2024-03-11T19:01:29.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.187+0000 7ff6dbfff700 1 -- 172.21.15.190:0/2866050704 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff6e400bba0 con 0x7ff6f4074010 2024-03-11T19:01:29.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.187+0000 7ff6f93be700 1 -- 172.21.15.190:0/2866050704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f419e100 con 0x7ff6f4074010 2024-03-11T19:01:29.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.187+0000 7ff6dbfff700 1 -- 172.21.15.190:0/2866050704 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff6e40038c0 con 0x7ff6f4074010 2024-03-11T19:01:29.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.188+0000 7ff6f93be700 1 -- 172.21.15.190:0/2866050704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f40632a0 con 0x7ff6f4074010 2024-03-11T19:01:29.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.188+0000 7ff6dbfff700 1 -- 172.21.15.190:0/2866050704 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7ff6e400bd10 con 0x7ff6f4074010 2024-03-11T19:01:29.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.188+0000 7ff6dbfff700 1 --2- 172.21.15.190:0/2866050704 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff6dc038d50 0x7ff6dc03b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:29.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.188+0000 7ff6dbfff700 1 -- 172.21.15.190:0/2866050704 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff6e404ca40 con 0x7ff6f4074010 2024-03-11T19:01:29.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.189+0000 7ff6f27fc700 1 --2- 172.21.15.190:0/2866050704 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff6dc038d50 0x7ff6dc03b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:29.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.189+0000 7ff6f27fc700 1 --2- 172.21.15.190:0/2866050704 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff6dc038d50 0x7ff6dc03b210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8006fd0 tx=0x7ff6e8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:29.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.194+0000 7ff6dbfff700 1 -- 172.21.15.190:0/2866050704 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff6e404ee60 con 0x7ff6f4074010 2024-03-11T19:01:29.510 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.507+0000 7ff6f93be700 1 -- 172.21.15.190:0/2866050704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff6f4111fd0 con 0x7ff6f4074010 2024-03-11T19:01:29.510 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.507+0000 7ff6dbfff700 1 -- 172.21.15.190:0/2866050704 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+745 (secure 0 0 0) 0x7ff6e4026020 con 0x7ff6f4074010 2024-03-11T19:01:29.510 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T19:01:29.511 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","modified":"2024-03-11T18:59:34.068259Z","created":"2024-03-11T18:59:34.068259Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T19:01:29.512 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.509+0000 7ff6f93be700 1 -- 172.21.15.190:0/2866050704 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff6dc038d50 msgr2=0x7ff6dc03b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:29.512 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.509+0000 7ff6f93be700 1 --2- 172.21.15.190:0/2866050704 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff6dc038d50 0x7ff6dc03b210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8006fd0 tx=0x7ff6e8006e40 comp rx=0 tx=0).stop 2024-03-11T19:01:29.512 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.509+0000 7ff6f93be700 1 -- 172.21.15.190:0/2866050704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f4074010 msgr2=0x7ff6f419d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:29.513 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.509+0000 7ff6f93be700 1 --2- 172.21.15.190:0/2866050704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f4074010 0x7ff6f419d4e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4004f40 tx=0x7ff6e4005020 comp rx=0 tx=0).stop 2024-03-11T19:01:29.513 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.510+0000 7ff6f93be700 1 -- 172.21.15.190:0/2866050704 shutdown_connections 2024-03-11T19:01:29.513 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.510+0000 7ff6f93be700 1 --2- 172.21.15.190:0/2866050704 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff6dc038d50 0x7ff6dc03b210 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:29.513 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.510+0000 7ff6f93be700 1 --2- 172.21.15.190:0/2866050704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f4074010 0x7ff6f419d4e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:29.513 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.510+0000 7ff6f93be700 1 -- 172.21.15.190:0/2866050704 >> 172.21.15.190:0/2866050704 conn(0x7ff6f40fd180 msgr2=0x7ff6f41033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:29.513 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.510+0000 7ff6f93be700 1 -- 172.21.15.190:0/2866050704 shutdown_connections 2024-03-11T19:01:29.513 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:29.510+0000 7ff6f93be700 1 -- 172.21.15.190:0/2866050704 wait complete. 2024-03-11T19:01:29.513 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-03-11T19:01:29.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:29 smithi029 ceph-mon[32439]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:30.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:30 smithi190 systemd[1]: Started Ceph mon.c for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:01:30.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:30 smithi029 ceph-mon[32439]: from='client.? 172.21.15.190:0/2866050704' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T19:01:31.213 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T19:01:31.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T19:01:31.449 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:01:35.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:35.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:01:35.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: mon.a calling monitor election 2024-03-11T19:01:35.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:01:35.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:01:35.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:35.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:01:35.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: mon.c calling monitor election 2024-03-11T19:01:35.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:01:35.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:35.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:01:35.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:01:35.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T19:01:35.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: monmap e2: 2 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],c=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T19:01:35.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: fsmap 2024-03-11T19:01:35.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T19:01:35.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: mgrmap e13: x(active, since 51s) 2024-03-11T19:01:35.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: overall HEALTH_OK 2024-03-11T19:01:35.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:35.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:35.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:35.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:01:35.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:35.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:35.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:01:35.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: mon.a calling monitor election 2024-03-11T19:01:35.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:01:35.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:01:35.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:35.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:01:35.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: mon.c calling monitor election 2024-03-11T19:01:35.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:01:35.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:35.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:01:35.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:01:35.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T19:01:35.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: monmap e2: 2 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],c=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T19:01:35.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: fsmap 2024-03-11T19:01:35.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T19:01:35.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: mgrmap e13: x(active, since 51s) 2024-03-11T19:01:35.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: overall HEALTH_OK 2024-03-11T19:01:35.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:35.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:35.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:35.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:01:35.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:35.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.883+0000 7fcbd4718700 1 -- 172.21.15.190:0/2942083076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbcc107750 msgr2=0x7fcbb8005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:35.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.883+0000 7fcbd4718700 1 --2- 172.21.15.190:0/2942083076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbcc107750 0x7fcbb8005610 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8005fa0 tx=0x7fcbc800ffc0 comp rx=0 tx=0).stop 2024-03-11T19:01:35.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.883+0000 7fcbd4718700 1 -- 172.21.15.190:0/2942083076 shutdown_connections 2024-03-11T19:01:35.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.883+0000 7fcbd4718700 1 --2- 172.21.15.190:0/2942083076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbcc107750 0x7fcbb8005610 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:35.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.883+0000 7fcbd4718700 1 -- 172.21.15.190:0/2942083076 >> 172.21.15.190:0/2942083076 conn(0x7fcbcc0fd240 msgr2=0x7fcbcc0ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:35.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.883+0000 7fcbd4718700 1 -- 172.21.15.190:0/2942083076 shutdown_connections 2024-03-11T19:01:35.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.883+0000 7fcbd4718700 1 -- 172.21.15.190:0/2942083076 wait complete. 2024-03-11T19:01:35.889 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.884+0000 7fcbd4718700 1 Processor -- start 2024-03-11T19:01:35.889 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.884+0000 7fcbd4718700 1 -- start start 2024-03-11T19:01:35.889 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.884+0000 7fcbd4718700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbcc107750 0x7fcbcc19dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:35.889 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.884+0000 7fcbd4718700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbcc19e320 0x7fcbcc199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:35.889 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.884+0000 7fcbd4718700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc19e840 con 0x7fcbcc19e320 2024-03-11T19:01:35.889 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.884+0000 7fcbd4718700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc19e9b0 con 0x7fcbcc107750 2024-03-11T19:01:35.890 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.884+0000 7fcbd24b4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbcc107750 0x7fcbcc19dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:35.890 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.884+0000 7fcbd24b4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbcc107750 0x7fcbcc19dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:59932/0 (socket says 172.21.15.190:59932) 2024-03-11T19:01:35.890 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.884+0000 7fcbd24b4700 1 -- 172.21.15.190:0/2788976671 learned_addr learned my addr 172.21.15.190:0/2788976671 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-11T19:01:35.890 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.884+0000 7fcbd24b4700 1 -- 172.21.15.190:0/2788976671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbcc107750 msgr2=0x7fcbcc19dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-11T19:01:35.890 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.884+0000 7fcbd24b4700 1 -- 172.21.15.190:0/2788976671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbcc107750 msgr2=0x7fcbcc19dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-11T19:01:35.890 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.884+0000 7fcbd24b4700 1 --2- 172.21.15.190:0/2788976671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbcc107750 0x7fcbcc19dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-11T19:01:35.890 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.884+0000 7fcbd24b4700 1 --2- 172.21.15.190:0/2788976671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbcc107750 0x7fcbcc19dde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T19:01:35.890 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.884+0000 7fcbd1cb3700 1 --2- 172.21.15.190:0/2788976671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbcc19e320 0x7fcbcc199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:35.890 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.885+0000 7fcbd1cb3700 1 -- 172.21.15.190:0/2788976671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbcc107750 msgr2=0x7fcbcc19dde0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:01:35.890 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.885+0000 7fcbd1cb3700 1 --2- 172.21.15.190:0/2788976671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbcc107750 0x7fcbcc19dde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:35.890 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.885+0000 7fcbd1cb3700 1 -- 172.21.15.190:0/2788976671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc800f9c0 con 0x7fcbcc19e320 2024-03-11T19:01:35.890 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.885+0000 7fcbd1cb3700 1 --2- 172.21.15.190:0/2788976671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbcc19e320 0x7fcbcc199140 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc00b500 tx=0x7fcbbc00b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:35.890 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.885+0000 7fcbc37fe700 1 -- 172.21.15.190:0/2788976671 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fcbbc011630 con 0x7fcbcc19e320 2024-03-11T19:01:35.890 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.885+0000 7fcbc37fe700 1 -- 172.21.15.190:0/2788976671 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcbbc011c40 con 0x7fcbcc19e320 2024-03-11T19:01:35.891 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.885+0000 7fcbd4718700 1 -- 172.21.15.190:0/2788976671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbcc1996e0 con 0x7fcbcc19e320 2024-03-11T19:01:35.891 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.885+0000 7fcbc37fe700 1 -- 172.21.15.190:0/2788976671 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fcbbc010c40 con 0x7fcbcc19e320 2024-03-11T19:01:35.891 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.885+0000 7fcbd4718700 1 -- 172.21.15.190:0/2788976671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbcc199c30 con 0x7fcbcc19e320 2024-03-11T19:01:35.891 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.886+0000 7fcbc37fe700 1 -- 172.21.15.190:0/2788976671 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7fcbbc01a410 con 0x7fcbcc19e320 2024-03-11T19:01:35.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.886+0000 7fcbc37fe700 1 --2- 172.21.15.190:0/2788976671 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcbb0038af0 0x7fcbb003afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:35.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.887+0000 7fcbc37fe700 1 -- 172.21.15.190:0/2788976671 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcbbc04aef0 con 0x7fcbcc19e320 2024-03-11T19:01:35.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.887+0000 7fcbd4718700 1 -- 172.21.15.190:0/2788976671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbcc106a00 con 0x7fcbcc19e320 2024-03-11T19:01:35.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.887+0000 7fcbd24b4700 1 --2- 172.21.15.190:0/2788976671 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcbb0038af0 0x7fcbb003afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:35.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.887+0000 7fcbd24b4700 1 --2- 172.21.15.190:0/2788976671 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcbb0038af0 0x7fcbb003afb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8004e90 tx=0x7fcbc8004da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:35.896 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:35.892+0000 7fcbc37fe700 1 -- 172.21.15.190:0/2788976671 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcbbc0193b0 con 0x7fcbcc19e320 2024-03-11T19:01:36.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:36.162+0000 7fcbd4718700 1 -- 172.21.15.190:0/2788976671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcbcc19a560 con 0x7fcbcc19e320 2024-03-11T19:01:36.166 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:36.163+0000 7fcbc37fe700 1 -- 172.21.15.190:0/2788976671 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7fcbbc018310 con 0x7fcbcc19e320 2024-03-11T19:01:36.166 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T19:01:36.166 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":2,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","modified":"2024-03-11T19:01:30.390497Z","created":"2024-03-11T18:59:34.068259Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T19:01:36.169 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:36.165+0000 7fcbd4718700 1 -- 172.21.15.190:0/2788976671 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcbb0038af0 msgr2=0x7fcbb003afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:36.169 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:36.165+0000 7fcbd4718700 1 --2- 172.21.15.190:0/2788976671 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcbb0038af0 0x7fcbb003afb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8004e90 tx=0x7fcbc8004da0 comp rx=0 tx=0).stop 2024-03-11T19:01:36.169 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:36.165+0000 7fcbd4718700 1 -- 172.21.15.190:0/2788976671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbcc19e320 msgr2=0x7fcbcc199140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:36.169 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:36.165+0000 7fcbd4718700 1 --2- 172.21.15.190:0/2788976671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbcc19e320 0x7fcbcc199140 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc00b500 tx=0x7fcbbc00b8c0 comp rx=0 tx=0).stop 2024-03-11T19:01:36.169 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:36.165+0000 7fcbd4718700 1 -- 172.21.15.190:0/2788976671 shutdown_connections 2024-03-11T19:01:36.169 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:36.165+0000 7fcbd4718700 1 --2- 172.21.15.190:0/2788976671 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcbb0038af0 0x7fcbb003afb0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:36.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:36.165+0000 7fcbd4718700 1 --2- 172.21.15.190:0/2788976671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbcc107750 0x7fcbcc19dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:36.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:36.165+0000 7fcbd4718700 1 --2- 172.21.15.190:0/2788976671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbcc19e320 0x7fcbcc199140 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:36.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:36.165+0000 7fcbd4718700 1 -- 172.21.15.190:0/2788976671 >> 172.21.15.190:0/2788976671 conn(0x7fcbcc0fd240 msgr2=0x7fcbcc0749d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:36.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:36.165+0000 7fcbd4718700 1 -- 172.21.15.190:0/2788976671 shutdown_connections 2024-03-11T19:01:36.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:36.165+0000 7fcbd4718700 1 -- 172.21.15.190:0/2788976671 wait complete. 2024-03-11T19:01:36.170 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 2 2024-03-11T19:01:36.756 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:36 smithi190 ceph-mon[36292]: Deploying daemon mon.b on smithi170 2024-03-11T19:01:36.756 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:36 smithi190 ceph-mon[36292]: from='client.? 172.21.15.190:0/2788976671' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T19:01:36.756 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:36 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:01:36.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:36 smithi029 ceph-mon[32439]: Deploying daemon mon.b on smithi170 2024-03-11T19:01:36.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:36 smithi029 ceph-mon[32439]: from='client.? 172.21.15.190:0/2788976671' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T19:01:36.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:36 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:01:37.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:37 smithi190 ceph-mon[36292]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:37.783 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T19:01:37.783 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T19:01:37.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:37 smithi029 ceph-mon[32439]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:38.026 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:01:39.545 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.539+0000 7f98cfea6700 1 -- 172.21.15.190:0/156792569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98c81006f0 msgr2=0x7f98c8100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:39.545 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.539+0000 7f98cfea6700 1 --2- 172.21.15.190:0/156792569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98c81006f0 0x7f98c8100ad0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f98c4009910 tx=0x7f98c4009c20 comp rx=0 tx=0).stop 2024-03-11T19:01:39.545 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.539+0000 7f98cfea6700 1 -- 172.21.15.190:0/156792569 shutdown_connections 2024-03-11T19:01:39.545 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.539+0000 7f98cfea6700 1 --2- 172.21.15.190:0/156792569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98c81006f0 0x7f98c8100ad0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:39.545 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.539+0000 7f98cfea6700 1 -- 172.21.15.190:0/156792569 >> 172.21.15.190:0/156792569 conn(0x7f98c80fc110 msgr2=0x7f98c80fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:39.545 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.539+0000 7f98cfea6700 1 -- 172.21.15.190:0/156792569 shutdown_connections 2024-03-11T19:01:39.545 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.539+0000 7f98cfea6700 1 -- 172.21.15.190:0/156792569 wait complete. 2024-03-11T19:01:39.545 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.540+0000 7f98cfea6700 1 Processor -- start 2024-03-11T19:01:39.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.540+0000 7f98cfea6700 1 -- start start 2024-03-11T19:01:39.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.540+0000 7f98cfea6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98c81006f0 0x7f98c81a35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:39.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.540+0000 7f98cfea6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98c81a3ae0 0x7f98c819d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:39.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.540+0000 7f98cfea6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c81a4000 con 0x7f98c81a3ae0 2024-03-11T19:01:39.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.540+0000 7f98cfea6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c819dbb0 con 0x7f98c81006f0 2024-03-11T19:01:39.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.540+0000 7f98cdc42700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98c81006f0 0x7f98c81a35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:39.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.540+0000 7f98cdc42700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98c81006f0 0x7f98c81a35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:59936/0 (socket says 172.21.15.190:59936) 2024-03-11T19:01:39.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.540+0000 7f98cdc42700 1 -- 172.21.15.190:0/3901084607 learned_addr learned my addr 172.21.15.190:0/3901084607 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-11T19:01:39.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.540+0000 7f98cd441700 1 --2- 172.21.15.190:0/3901084607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98c81a3ae0 0x7f98c819d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:39.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.540+0000 7f98cdc42700 1 -- 172.21.15.190:0/3901084607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98c81006f0 msgr2=0x7f98c81a35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-11T19:01:39.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.540+0000 7f98cdc42700 1 -- 172.21.15.190:0/3901084607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98c81006f0 msgr2=0x7f98c81a35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-11T19:01:39.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.540+0000 7f98cdc42700 1 --2- 172.21.15.190:0/3901084607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98c81006f0 0x7f98c81a35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-11T19:01:39.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.540+0000 7f98cdc42700 1 --2- 172.21.15.190:0/3901084607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98c81006f0 0x7f98c81a35a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T19:01:39.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.541+0000 7f98cd441700 1 -- 172.21.15.190:0/3901084607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98c81006f0 msgr2=0x7f98c81a35a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:01:39.547 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.541+0000 7f98cd441700 1 --2- 172.21.15.190:0/3901084607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98c81006f0 0x7f98c81a35a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:39.547 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.541+0000 7f98cd441700 1 -- 172.21.15.190:0/3901084607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c40095f0 con 0x7f98c81a3ae0 2024-03-11T19:01:39.547 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.541+0000 7f98cd441700 1 --2- 172.21.15.190:0/3901084607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98c81a3ae0 0x7f98c819d670 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f98b800b4d0 tx=0x7f98b800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:39.547 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.541+0000 7f98beffd700 1 -- 172.21.15.190:0/3901084607 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f98b8011630 con 0x7f98c81a3ae0 2024-03-11T19:01:39.547 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.541+0000 7f98beffd700 1 -- 172.21.15.190:0/3901084607 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f98b8011c40 con 0x7f98c81a3ae0 2024-03-11T19:01:39.547 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.541+0000 7f98cfea6700 1 -- 172.21.15.190:0/3901084607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c819de40 con 0x7f98c81a3ae0 2024-03-11T19:01:39.547 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.541+0000 7f98beffd700 1 -- 172.21.15.190:0/3901084607 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f98b8010cc0 con 0x7f98c81a3ae0 2024-03-11T19:01:39.547 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.541+0000 7f98cfea6700 1 -- 172.21.15.190:0/3901084607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c819e390 con 0x7f98c81a3ae0 2024-03-11T19:01:39.548 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.542+0000 7f98beffd700 1 -- 172.21.15.190:0/3901084607 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f98b8011790 con 0x7f98c81a3ae0 2024-03-11T19:01:39.548 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.542+0000 7f98cfea6700 1 -- 172.21.15.190:0/3901084607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c806a730 con 0x7f98c81a3ae0 2024-03-11T19:01:39.548 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.542+0000 7f98beffd700 1 --2- 172.21.15.190:0/3901084607 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f98b4038af0 0x7f98b403afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:39.548 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.542+0000 7f98beffd700 1 -- 172.21.15.190:0/3901084607 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f98b804b460 con 0x7f98c81a3ae0 2024-03-11T19:01:39.548 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.542+0000 7f98cdc42700 1 --2- 172.21.15.190:0/3901084607 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f98b4038af0 0x7f98b403afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:39.548 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.543+0000 7f98cdc42700 1 --2- 172.21.15.190:0/3901084607 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f98b4038af0 0x7f98b403afb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f98c4000c00 tx=0x7f98c400b580 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:39.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.548+0000 7f98beffd700 1 -- 172.21.15.190:0/3901084607 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f98b8010410 con 0x7f98c81a3ae0 2024-03-11T19:01:39.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:39 smithi190 ceph-mon[36292]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:39.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.760+0000 7f98beffd700 1 -- 172.21.15.190:0/3901084607 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8010ef0 con 0x7f98c81a3ae0 2024-03-11T19:01:39.823 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:39.821+0000 7f98cfea6700 1 -- 172.21.15.190:0/3901084607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f98c819eba0 con 0x7f98c81a3ae0 2024-03-11T19:01:39.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:39 smithi170 systemd[1]: Started Ceph mon.b for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:01:39.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:39 smithi029 ceph-mon[32439]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:44.782 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:44.779+0000 7f98beffd700 1 -- 172.21.15.190:0/3901084607 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1283 (secure 0 0 0) 0x7f98b8017b50 con 0x7f98c81a3ae0 2024-03-11T19:01:44.783 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-11T19:01:44.783 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":3,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","modified":"2024-03-11T19:01:39.759477Z","created":"2024-03-11T18:59:34.068259Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T19:01:44.785 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:44.782+0000 7f98cfea6700 1 -- 172.21.15.190:0/3901084607 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f98b4038af0 msgr2=0x7f98b403afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:44.785 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:44.782+0000 7f98cfea6700 1 --2- 172.21.15.190:0/3901084607 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f98b4038af0 0x7f98b403afb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f98c4000c00 tx=0x7f98c400b580 comp rx=0 tx=0).stop 2024-03-11T19:01:44.785 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:44.782+0000 7f98cfea6700 1 -- 172.21.15.190:0/3901084607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98c81a3ae0 msgr2=0x7f98c819d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:44.785 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:44.782+0000 7f98cfea6700 1 --2- 172.21.15.190:0/3901084607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98c81a3ae0 0x7f98c819d670 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f98b800b4d0 tx=0x7f98b800b890 comp rx=0 tx=0).stop 2024-03-11T19:01:44.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:44.782+0000 7f98cfea6700 1 -- 172.21.15.190:0/3901084607 shutdown_connections 2024-03-11T19:01:44.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:44.782+0000 7f98cfea6700 1 --2- 172.21.15.190:0/3901084607 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f98b4038af0 0x7f98b403afb0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:44.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:44.782+0000 7f98cfea6700 1 --2- 172.21.15.190:0/3901084607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98c81006f0 0x7f98c81a35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:44.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:44.782+0000 7f98cfea6700 1 --2- 172.21.15.190:0/3901084607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98c81a3ae0 0x7f98c819d670 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:44.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:44.782+0000 7f98cfea6700 1 -- 172.21.15.190:0/3901084607 >> 172.21.15.190:0/3901084607 conn(0x7f98c80fc110 msgr2=0x7f98c80fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:44.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:44.782+0000 7f98cfea6700 1 -- 172.21.15.190:0/3901084607 shutdown_connections 2024-03-11T19:01:44.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:44.783+0000 7f98cfea6700 1 -- 172.21.15.190:0/3901084607 wait complete. 2024-03-11T19:01:44.786 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 3 2024-03-11T19:01:45.118 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:45.118 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:01:45.118 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: mon.c calling monitor election 2024-03-11T19:01:45.118 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: mon.a calling monitor election 2024-03-11T19:01:45.118 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:45.118 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:01:45.118 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: from='client.? 172.21.15.190:0/3901084607' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T19:01:45.118 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:45.118 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:45.118 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:45.118 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:45.118 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:45.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:45.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:45.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T19:01:45.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T19:01:45.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: fsmap 2024-03-11T19:01:45.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T19:01:45.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: mgrmap e13: x(active, since 61s) 2024-03-11T19:01:45.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T19:01:45.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T19:01:45.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T19:01:45.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: mon.b (rank 2) addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] is down (out of quorum) 2024-03-11T19:01:45.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:45.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:45.119 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:45.120 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:45.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:45.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:01:45.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: mon.c calling monitor election 2024-03-11T19:01:45.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: mon.a calling monitor election 2024-03-11T19:01:45.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:45.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:01:45.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: from='client.? 172.21.15.190:0/3901084607' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T19:01:45.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T19:01:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T19:01:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: fsmap 2024-03-11T19:01:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T19:01:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: mgrmap e13: x(active, since 61s) 2024-03-11T19:01:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T19:01:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T19:01:45.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T19:01:45.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: mon.b (rank 2) addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] is down (out of quorum) 2024-03-11T19:01:45.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:45.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:45.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:45.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:45.532 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-11T19:01:45.532 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-11T19:01:45.785 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:01:46.105 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:45 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:01:46.105 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:45 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:46.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:45 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:01:46.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:45 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:01:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: mon.c calling monitor election 2024-03-11T19:01:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: mon.a calling monitor election 2024-03-11T19:01:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:01:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: from='client.? 172.21.15.190:0/3901084607' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T19:01:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T19:01:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T19:01:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: fsmap 2024-03-11T19:01:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T19:01:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: mgrmap e13: x(active, since 61s) 2024-03-11T19:01:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T19:01:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T19:01:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T19:01:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: mon.b (rank 2) addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] is down (out of quorum) 2024-03-11T19:01:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:01:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:47.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.410+0000 7f4a94e5c700 1 -- 172.21.15.29:0/3298415644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a901066b0 msgr2=0x7f4a90106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:47.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.410+0000 7f4a94e5c700 1 --2- 172.21.15.29:0/3298415644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a901066b0 0x7f4a90106a90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4a78009910 tx=0x7f4a78009c20 comp rx=0 tx=0).stop 2024-03-11T19:01:47.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.410+0000 7f4a94e5c700 1 -- 172.21.15.29:0/3298415644 shutdown_connections 2024-03-11T19:01:47.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.410+0000 7f4a94e5c700 1 --2- 172.21.15.29:0/3298415644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a901066b0 0x7f4a90106a90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:47.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.410+0000 7f4a94e5c700 1 -- 172.21.15.29:0/3298415644 >> 172.21.15.29:0/3298415644 conn(0x7f4a900fb7c0 msgr2=0x7f4a900fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:47.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.410+0000 7f4a94e5c700 1 -- 172.21.15.29:0/3298415644 shutdown_connections 2024-03-11T19:01:47.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.410+0000 7f4a94e5c700 1 -- 172.21.15.29:0/3298415644 wait complete. 2024-03-11T19:01:47.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.411+0000 7f4a94e5c700 1 Processor -- start 2024-03-11T19:01:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.411+0000 7f4a94e5c700 1 -- start start 2024-03-11T19:01:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.411+0000 7f4a94e5c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a901066b0 0x7f4a901970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.411+0000 7f4a94e5c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a90197610 0x7f4a9019b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.411+0000 7f4a94e5c700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a9019bee0 0x7f4a9019c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.411+0000 7f4a94e5c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a90197d00 con 0x7f4a901066b0 2024-03-11T19:01:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.411+0000 7f4a94e5c700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a901a3c50 con 0x7f4a9019bee0 2024-03-11T19:01:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.411+0000 7f4a94e5c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a901a3d90 con 0x7f4a90197610 2024-03-11T19:01:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.411+0000 7f4a8e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a901066b0 0x7f4a901970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.411+0000 7f4a8e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a901066b0 0x7f4a901970d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59616/0 (socket says 172.21.15.29:59616) 2024-03-11T19:01:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.411+0000 7f4a8e59c700 1 -- 172.21.15.29:0/2948313459 learned_addr learned my addr 172.21.15.29:0/2948313459 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:01:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.412+0000 7f4a8dd9b700 1 --2- 172.21.15.29:0/2948313459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a90197610 0x7f4a9019b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.412+0000 7f4a8ed9d700 1 --2- 172.21.15.29:0/2948313459 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a9019bee0 0x7f4a9019c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.412+0000 7f4a8e59c700 1 -- 172.21.15.29:0/2948313459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a90197610 msgr2=0x7f4a9019b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.412+0000 7f4a8e59c700 1 --2- 172.21.15.29:0/2948313459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a90197610 0x7f4a9019b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:47.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.412+0000 7f4a8e59c700 1 -- 172.21.15.29:0/2948313459 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a9019bee0 msgr2=0x7f4a9019c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:47.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.412+0000 7f4a8e59c700 1 --2- 172.21.15.29:0/2948313459 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a9019bee0 0x7f4a9019c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:47.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.412+0000 7f4a8e59c700 1 -- 172.21.15.29:0/2948313459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a780095f0 con 0x7f4a901066b0 2024-03-11T19:01:47.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.412+0000 7f4a8ed9d700 1 --2- 172.21.15.29:0/2948313459 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a9019bee0 0x7f4a9019c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:01:47.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.412+0000 7f4a8e59c700 1 --2- 172.21.15.29:0/2948313459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a901066b0 0x7f4a901970d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4a78006010 tx=0x7f4a7800bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:47.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.412+0000 7f4a877fe700 1 -- 172.21.15.29:0/2948313459 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7801d070 con 0x7f4a901066b0 2024-03-11T19:01:47.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.412+0000 7f4a94e5c700 1 -- 172.21.15.29:0/2948313459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a901a4010 con 0x7f4a901066b0 2024-03-11T19:01:47.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.412+0000 7f4a877fe700 1 -- 172.21.15.29:0/2948313459 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4a78003d40 con 0x7f4a901066b0 2024-03-11T19:01:47.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.412+0000 7f4a877fe700 1 -- 172.21.15.29:0/2948313459 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7800fda0 con 0x7f4a901066b0 2024-03-11T19:01:47.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.412+0000 7f4a94e5c700 1 -- 172.21.15.29:0/2948313459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a901a4530 con 0x7f4a901066b0 2024-03-11T19:01:47.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.413+0000 7f4a877fe700 1 -- 172.21.15.29:0/2948313459 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f4a7802b440 con 0x7f4a901066b0 2024-03-11T19:01:47.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.413+0000 7f4a877fe700 1 --2- 172.21.15.29:0/2948313459 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4a7c0391e0 0x7f4a7c03b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:47.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.413+0000 7f4a877fe700 1 -- 172.21.15.29:0/2948313459 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4a7804dad0 con 0x7f4a901066b0 2024-03-11T19:01:47.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.413+0000 7f4a94e5c700 1 -- 172.21.15.29:0/2948313459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a900632a0 con 0x7f4a901066b0 2024-03-11T19:01:47.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.414+0000 7f4a8dd9b700 1 --2- 172.21.15.29:0/2948313459 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4a7c0391e0 0x7f4a7c03b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:47.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.414+0000 7f4a8dd9b700 1 --2- 172.21.15.29:0/2948313459 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4a7c0391e0 0x7f4a7c03b6a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4a80009850 tx=0x7f4a80008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:47.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.419+0000 7f4a877fe700 1 -- 172.21.15.29:0/2948313459 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4a7802b740 con 0x7f4a901066b0 2024-03-11T19:01:47.534 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: mon.b calling monitor election 2024-03-11T19:01:47.534 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: mon.b calling monitor election 2024-03-11T19:01:47.534 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: mon.a calling monitor election 2024-03-11T19:01:47.534 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: mon.c calling monitor election 2024-03-11T19:01:47.534 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T19:01:47.534 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T19:01:47.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: fsmap 2024-03-11T19:01:47.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T19:01:47.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: mgrmap e13: x(active, since 63s) 2024-03-11T19:01:47.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T19:01:47.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: Cluster is now healthy 2024-03-11T19:01:47.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: overall HEALTH_OK 2024-03-11T19:01:47.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.536 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:01:47.536 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T19:01:47.536 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:47.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.623+0000 7f4a94e5c700 1 -- 172.21.15.29:0/2948313459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f4a9010a8f0 con 0x7f4a901066b0 2024-03-11T19:01:47.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.623+0000 7f4a877fe700 1 -- 172.21.15.29:0/2948313459 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f4a7802b740 con 0x7f4a901066b0 2024-03-11T19:01:47.627 INFO:teuthology.orchestra.run.smithi029.stdout:# minimal ceph.conf for 638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T19:01:47.627 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2024-03-11T19:01:47.627 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = 638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T19:01:47.627 INFO:teuthology.orchestra.run.smithi029.stdout: mon_host = [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-03-11T19:01:47.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.625+0000 7f4a94e5c700 1 -- 172.21.15.29:0/2948313459 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4a7c0391e0 msgr2=0x7f4a7c03b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:47.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.625+0000 7f4a94e5c700 1 --2- 172.21.15.29:0/2948313459 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4a7c0391e0 0x7f4a7c03b6a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4a80009850 tx=0x7f4a80008040 comp rx=0 tx=0).stop 2024-03-11T19:01:47.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.625+0000 7f4a94e5c700 1 -- 172.21.15.29:0/2948313459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a901066b0 msgr2=0x7f4a901970d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:47.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.625+0000 7f4a94e5c700 1 --2- 172.21.15.29:0/2948313459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a901066b0 0x7f4a901970d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4a78006010 tx=0x7f4a7800bd80 comp rx=0 tx=0).stop 2024-03-11T19:01:47.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.625+0000 7f4a94e5c700 1 -- 172.21.15.29:0/2948313459 shutdown_connections 2024-03-11T19:01:47.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.625+0000 7f4a94e5c700 1 --2- 172.21.15.29:0/2948313459 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4a7c0391e0 0x7f4a7c03b6a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:47.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.625+0000 7f4a94e5c700 1 --2- 172.21.15.29:0/2948313459 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a9019bee0 0x7f4a9019c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:47.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.625+0000 7f4a94e5c700 1 --2- 172.21.15.29:0/2948313459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a901066b0 0x7f4a901970d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:47.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.625+0000 7f4a94e5c700 1 --2- 172.21.15.29:0/2948313459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a90197610 0x7f4a9019b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:47.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.626+0000 7f4a94e5c700 1 -- 172.21.15.29:0/2948313459 >> 172.21.15.29:0/2948313459 conn(0x7f4a900fb7c0 msgr2=0x7f4a900fbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:47.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.626+0000 7f4a94e5c700 1 -- 172.21.15.29:0/2948313459 shutdown_connections 2024-03-11T19:01:47.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:47.626+0000 7f4a94e5c700 1 -- 172.21.15.29:0/2948313459 wait complete. 2024-03-11T19:01:47.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: mon.b calling monitor election 2024-03-11T19:01:47.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: mon.b calling monitor election 2024-03-11T19:01:47.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: mon.a calling monitor election 2024-03-11T19:01:47.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: mon.c calling monitor election 2024-03-11T19:01:47.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T19:01:47.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T19:01:47.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: fsmap 2024-03-11T19:01:47.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T19:01:47.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: mgrmap e13: x(active, since 63s) 2024-03-11T19:01:47.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T19:01:47.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: Cluster is now healthy 2024-03-11T19:01:47.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: overall HEALTH_OK 2024-03-11T19:01:47.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.764 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.764 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:01:47.764 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T19:01:47.764 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:47.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: mon.b calling monitor election 2024-03-11T19:01:47.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: mon.b calling monitor election 2024-03-11T19:01:47.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: mon.a calling monitor election 2024-03-11T19:01:47.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: mon.c calling monitor election 2024-03-11T19:01:47.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T19:01:47.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T19:01:47.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: fsmap 2024-03-11T19:01:47.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T19:01:47.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: mgrmap e13: x(active, since 63s) 2024-03-11T19:01:47.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T19:01:47.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: Cluster is now healthy 2024-03-11T19:01:47.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: overall HEALTH_OK 2024-03-11T19:01:47.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:47.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:01:47.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T19:01:47.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:48.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:48 smithi029 ceph-mon[32439]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T19:01:48.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:48 smithi029 ceph-mon[32439]: Reconfiguring daemon mon.a on smithi029 2024-03-11T19:01:48.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:48 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2948313459' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:48.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:48 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:48.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:48 smithi190 ceph-mon[36292]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T19:01:48.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:48 smithi190 ceph-mon[36292]: Reconfiguring daemon mon.a on smithi029 2024-03-11T19:01:48.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:48 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2948313459' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:48.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:48 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:48.815 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-11T19:01:48.815 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-11T19:01:48.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T19:01:48.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:48 smithi170 ceph-mon[35690]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T19:01:48.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:48 smithi170 ceph-mon[35690]: Reconfiguring daemon mon.a on smithi029 2024-03-11T19:01:48.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:48 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2948313459' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:48.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:48 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:01:48.848 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-11T19:01:48.848 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:01:48.917 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-11T19:01:48.917 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T19:01:48.952 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-11T19:01:48.952 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:01:49.023 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-11T19:01:49.023 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T19:01:49.058 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-11T19:01:49.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:01:49.128 INFO:tasks.cephadm:Adding mgr.x on smithi029 2024-03-11T19:01:49.128 INFO:tasks.cephadm:Adding mgr.y on smithi170 2024-03-11T19:01:49.128 INFO:tasks.cephadm:Adding mgr.z on smithi190 2024-03-11T19:01:49.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch apply mgr '3;smithi029=x;smithi170=y;smithi190=z' 2024-03-11T19:01:49.402 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:01:49.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:49 smithi029 ceph-mon[32439]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:49.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:49 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:49.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:49 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:49.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:49 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:49.662 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:49 smithi190 ceph-mon[36292]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:49.662 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:49 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:49.662 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:49 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:49.663 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:49 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:49.739 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:49 smithi170 ceph-mon[35690]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:49.740 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:49 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:49.740 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:49 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:49.740 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:49 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:50.566 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:50 smithi190 ceph-mon[36292]: Reconfiguring mon.b (monmap changed)... 2024-03-11T19:01:50.566 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:01:50.566 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T19:01:50.566 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:50.566 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:50 smithi190 ceph-mon[36292]: Reconfiguring daemon mon.b on smithi170 2024-03-11T19:01:50.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:50 smithi029 ceph-mon[32439]: Reconfiguring mon.b (monmap changed)... 2024-03-11T19:01:50.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:01:50.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T19:01:50.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:50.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:50 smithi029 ceph-mon[32439]: Reconfiguring daemon mon.b on smithi170 2024-03-11T19:01:50.809 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.803+0000 7fa277b93700 1 -- 172.21.15.190:0/3140381164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa270100690 msgr2=0x7fa270100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:50.809 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.803+0000 7fa277b93700 1 --2- 172.21.15.190:0/3140381164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa270100690 0x7fa270100ab0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa260009910 tx=0x7fa260009c20 comp rx=0 tx=0).stop 2024-03-11T19:01:50.809 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.803+0000 7fa277b93700 1 -- 172.21.15.190:0/3140381164 shutdown_connections 2024-03-11T19:01:50.809 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.803+0000 7fa277b93700 1 --2- 172.21.15.190:0/3140381164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa270100690 0x7fa270100ab0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:50.809 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.803+0000 7fa277b93700 1 -- 172.21.15.190:0/3140381164 >> 172.21.15.190:0/3140381164 conn(0x7fa2700fbc50 msgr2=0x7fa2700fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:50.809 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.803+0000 7fa277b93700 1 -- 172.21.15.190:0/3140381164 shutdown_connections 2024-03-11T19:01:50.809 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.803+0000 7fa277b93700 1 -- 172.21.15.190:0/3140381164 wait complete. 2024-03-11T19:01:50.809 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.804+0000 7fa277b93700 1 Processor -- start 2024-03-11T19:01:50.809 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.804+0000 7fa277b93700 1 -- start start 2024-03-11T19:01:50.809 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.804+0000 7fa277b93700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa270100690 0x7fa270198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:50.809 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.804+0000 7fa277b93700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa270199430 0x7fa27019e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:50.809 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.804+0000 7fa277b93700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2701998b0 0x7fa2701a0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:50.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.804+0000 7fa277b93700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2701a11d0 con 0x7fa2701998b0 2024-03-11T19:01:50.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.804+0000 7fa277b93700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2701a1340 con 0x7fa270100690 2024-03-11T19:01:50.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.804+0000 7fa277b93700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2701a1500 con 0x7fa270199430 2024-03-11T19:01:50.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.804+0000 7fa27512e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa270199430 0x7fa27019e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:50.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.804+0000 7fa27512e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa270199430 0x7fa27019e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:35978/0 (socket says 172.21.15.190:35978) 2024-03-11T19:01:50.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.804+0000 7fa27512e700 1 -- 172.21.15.190:0/1992381511 learned_addr learned my addr 172.21.15.190:0/1992381511 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-11T19:01:50.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.804+0000 7fa276130700 1 --2- 172.21.15.190:0/1992381511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2701998b0 0x7fa2701a0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:50.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.804+0000 7fa27592f700 1 --2- 172.21.15.190:0/1992381511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa270100690 0x7fa270198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:50.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.805+0000 7fa27512e700 1 -- 172.21.15.190:0/1992381511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa270100690 msgr2=0x7fa270198ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:50.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.805+0000 7fa27512e700 1 --2- 172.21.15.190:0/1992381511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa270100690 0x7fa270198ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:50.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.805+0000 7fa27512e700 1 -- 172.21.15.190:0/1992381511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2701998b0 msgr2=0x7fa2701a0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:50.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.805+0000 7fa27512e700 1 --2- 172.21.15.190:0/1992381511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2701998b0 0x7fa2701a0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.805+0000 7fa27512e700 1 -- 172.21.15.190:0/1992381511 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2600095f0 con 0x7fa270199430 2024-03-11T19:01:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.805+0000 7fa27592f700 1 --2- 172.21.15.190:0/1992381511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa270100690 0x7fa270198ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:01:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.805+0000 7fa27512e700 1 --2- 172.21.15.190:0/1992381511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa270199430 0x7fa27019e4a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa26c00b780 tx=0x7fa26c00ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.805+0000 7fa266ffd700 1 -- 172.21.15.190:0/1992381511 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c004230 con 0x7fa270199430 2024-03-11T19:01:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.805+0000 7fa277b93700 1 -- 172.21.15.190:0/1992381511 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2701a17e0 con 0x7fa270199430 2024-03-11T19:01:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.805+0000 7fa266ffd700 1 -- 172.21.15.190:0/1992381511 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa26c004390 con 0x7fa270199430 2024-03-11T19:01:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.805+0000 7fa266ffd700 1 -- 172.21.15.190:0/1992381511 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c011af0 con 0x7fa270199430 2024-03-11T19:01:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.805+0000 7fa277b93700 1 -- 172.21.15.190:0/1992381511 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2701a1d60 con 0x7fa270199430 2024-03-11T19:01:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.806+0000 7fa266ffd700 1 -- 172.21.15.190:0/1992381511 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7fa26c011c50 con 0x7fa270199430 2024-03-11T19:01:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.806+0000 7fa277b93700 1 -- 172.21.15.190:0/1992381511 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa270192e70 con 0x7fa270199430 2024-03-11T19:01:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.806+0000 7fa266ffd700 1 --2- 172.21.15.190:0/1992381511 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa25c0391e0 0x7fa25c03b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.806+0000 7fa266ffd700 1 -- 172.21.15.190:0/1992381511 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa26c04b310 con 0x7fa270199430 2024-03-11T19:01:50.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.806+0000 7fa27592f700 1 --2- 172.21.15.190:0/1992381511 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa25c0391e0 0x7fa25c03b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:50.812 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.807+0000 7fa27592f700 1 --2- 172.21.15.190:0/1992381511 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa25c0391e0 0x7fa25c03b6a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa260005190 tx=0x7fa26000b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:50.814 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:50.811+0000 7fa266ffd700 1 -- 172.21.15.190:0/1992381511 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa26c0036a0 con 0x7fa270199430 2024-03-11T19:01:50.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:50 smithi170 ceph-mon[35690]: Reconfiguring mon.b (monmap changed)... 2024-03-11T19:01:50.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:01:50.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T19:01:50.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:50.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:50 smithi170 ceph-mon[35690]: Reconfiguring daemon mon.b on smithi170 2024-03-11T19:01:51.022 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:51.019+0000 7fa277b93700 1 -- 172.21.15.190:0/1992381511 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi029=x;smithi170=y;smithi190=z", "target": ["mon-mgr", ""]}) v1 -- 0x7fa27002d7d0 con 0x7fa25c0391e0 2024-03-11T19:01:51.028 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:51.025+0000 7fa266ffd700 1 -- 172.21.15.190:0/1992381511 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa27002d7d0 con 0x7fa25c0391e0 2024-03-11T19:01:51.028 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled mgr update... 2024-03-11T19:01:51.030 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:51.027+0000 7fa277b93700 1 -- 172.21.15.190:0/1992381511 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa25c0391e0 msgr2=0x7fa25c03b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:51.030 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:51.027+0000 7fa277b93700 1 --2- 172.21.15.190:0/1992381511 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa25c0391e0 0x7fa25c03b6a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa260005190 tx=0x7fa26000b540 comp rx=0 tx=0).stop 2024-03-11T19:01:51.030 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:51.027+0000 7fa277b93700 1 -- 172.21.15.190:0/1992381511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa270199430 msgr2=0x7fa27019e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:51.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:51.027+0000 7fa277b93700 1 --2- 172.21.15.190:0/1992381511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa270199430 0x7fa27019e4a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa26c00b780 tx=0x7fa26c00ba90 comp rx=0 tx=0).stop 2024-03-11T19:01:51.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:51.027+0000 7fa277b93700 1 -- 172.21.15.190:0/1992381511 shutdown_connections 2024-03-11T19:01:51.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:51.027+0000 7fa277b93700 1 --2- 172.21.15.190:0/1992381511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2701998b0 0x7fa2701a0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:51.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:51.027+0000 7fa277b93700 1 --2- 172.21.15.190:0/1992381511 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa25c0391e0 0x7fa25c03b6a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:51.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:51.027+0000 7fa277b93700 1 --2- 172.21.15.190:0/1992381511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa270100690 0x7fa270198ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:51.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:51.027+0000 7fa277b93700 1 --2- 172.21.15.190:0/1992381511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa270199430 0x7fa27019e4a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:51.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:51.027+0000 7fa277b93700 1 -- 172.21.15.190:0/1992381511 >> 172.21.15.190:0/1992381511 conn(0x7fa2700fbc50 msgr2=0x7fa27018ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:51.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:51.027+0000 7fa277b93700 1 -- 172.21.15.190:0/1992381511 shutdown_connections 2024-03-11T19:01:51.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:01:51.027+0000 7fa277b93700 1 -- 172.21.15.190:0/1992381511 wait complete. 2024-03-11T19:01:51.558 DEBUG:teuthology.orchestra.run.smithi170:mgr.y> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mgr.y.service 2024-03-11T19:01:51.561 DEBUG:teuthology.orchestra.run.smithi190:mgr.z> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mgr.z.service 2024-03-11T19:01:51.563 INFO:tasks.cephadm:Deploying OSDs... 2024-03-11T19:01:51.563 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-11T19:01:51.563 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T19:01:51.583 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T19:01:51.583 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2024-03-11T19:01:51.641 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T19:01:51.641 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:01:51.641 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T19:01:51.641 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:01:51.641 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:01:51.641 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-11 19:00:56.635934203 +0000 2024-03-11T19:01:51.641 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-11 18:56:16.048425972 +0000 2024-03-11T19:01:51.641 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-11 18:56:16.048425972 +0000 2024-03-11T19:01:51.642 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-11 18:56:16.048425972 +0000 2024-03-11T19:01:51.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T19:01:51.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:51 smithi029 ceph-mon[32439]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:51.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:51 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:51.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:51 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:51.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:51 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:01:51.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:51 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T19:01:51.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:51 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:51.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:51 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:51.684 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-11T19:01:51.684 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-11T19:01:51.685 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000120681 s, 4.2 MB/s 2024-03-11T19:01:51.686 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T19:01:51.692 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:51 smithi190 ceph-mon[36292]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:51.692 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:51 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:51.692 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:51 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:51.692 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:51 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:01:51.692 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:51 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T19:01:51.693 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:51 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:51.693 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:51 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:51.744 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2024-03-11T19:01:51.800 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T19:01:51.800 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:01:51.800 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-11T19:01:51.800 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:01:51.801 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:01:51.801 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-11 19:00:56.663933759 +0000 2024-03-11T19:01:51.801 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-11 18:56:16.373420719 +0000 2024-03-11T19:01:51.801 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-11 18:56:16.373420719 +0000 2024-03-11T19:01:51.801 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-11 18:56:16.373420719 +0000 2024-03-11T19:01:51.801 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T19:01:51.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:51 smithi170 ceph-mon[35690]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:51.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:51 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:51.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:51 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:51.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:51 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:01:51.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:51 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T19:01:51.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:51 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:51.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:51 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:51.868 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-11T19:01:51.868 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-11T19:01:51.868 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.00010966 s, 4.7 MB/s 2024-03-11T19:01:51.869 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T19:01:51.926 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2024-03-11T19:01:51.982 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T19:01:51.983 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:01:51.983 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-11T19:01:51.983 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:01:51.983 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:01:51.983 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-11 19:00:56.691933316 +0000 2024-03-11T19:01:51.983 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-11 18:56:16.709415288 +0000 2024-03-11T19:01:51.983 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-11 18:56:16.709415288 +0000 2024-03-11T19:01:51.983 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-11 18:56:16.709415288 +0000 2024-03-11T19:01:51.983 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T19:01:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-11T19:01:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-11T19:01:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000109572 s, 4.7 MB/s 2024-03-11T19:01:52.050 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T19:01:52.106 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2024-03-11T19:01:52.161 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T19:01:52.161 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:01:52.162 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-11T19:01:52.162 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:01:52.162 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:01:52.162 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-11 19:00:56.716932919 +0000 2024-03-11T19:01:52.162 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-11 18:56:17.048409809 +0000 2024-03-11T19:01:52.162 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-11 18:56:17.048409809 +0000 2024-03-11T19:01:52.162 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-11 18:56:17.048409809 +0000 2024-03-11T19:01:52.162 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T19:01:52.227 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-11T19:01:52.228 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-11T19:01:52.228 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000107787 s, 4.8 MB/s 2024-03-11T19:01:52.229 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T19:01:52.285 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-11T19:01:52.285 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T19:01:52.304 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T19:01:52.304 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_1 2024-03-11T19:01:52.363 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T19:01:52.363 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:01:52.363 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-11T19:01:52.363 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:01:52.363 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:01:52.363 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-11 19:01:39.157771146 +0000 2024-03-11T19:01:52.363 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-11 18:56:04.574431068 +0000 2024-03-11T19:01:52.363 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-11 18:56:04.574431068 +0000 2024-03-11T19:01:52.363 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-11 18:56:04.574431068 +0000 2024-03-11T19:01:52.364 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T19:01:52.432 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-11T19:01:52.433 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-11T19:01:52.433 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000107234 s, 4.8 MB/s 2024-03-11T19:01:52.434 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T19:01:52.491 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_2 2024-03-11T19:01:52.547 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T19:01:52.548 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:01:52.548 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-11T19:01:52.548 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:01:52.548 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:01:52.548 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-11 19:01:39.157771146 +0000 2024-03-11T19:01:52.548 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-11 18:56:04.917422401 +0000 2024-03-11T19:01:52.548 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-11 18:56:04.917422401 +0000 2024-03-11T19:01:52.548 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-11 18:56:04.917422401 +0000 2024-03-11T19:01:52.548 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T19:01:52.615 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-11T19:01:52.615 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-11T19:01:52.615 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000115688 s, 4.4 MB/s 2024-03-11T19:01:52.617 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T19:01:52.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:52 smithi029 ceph-mon[32439]: Reconfiguring mon.c (monmap changed)... 2024-03-11T19:01:52.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:52 smithi029 ceph-mon[32439]: Reconfiguring daemon mon.c on smithi190 2024-03-11T19:01:52.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:52 smithi029 ceph-mon[32439]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi029=x;smithi170=y;smithi190=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:01:52.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:52 smithi029 ceph-mon[32439]: Saving service mgr spec with placement smithi029=x;smithi170=y;smithi190=z;count:3 2024-03-11T19:01:52.673 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_3 2024-03-11T19:01:52.729 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T19:01:52.729 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:01:52.730 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-11T19:01:52.730 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:01:52.730 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:01:52.730 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-11 19:01:39.158771122 +0000 2024-03-11T19:01:52.730 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-11 18:56:05.241414215 +0000 2024-03-11T19:01:52.730 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-11 18:56:05.241414215 +0000 2024-03-11T19:01:52.730 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-11 18:56:05.241414215 +0000 2024-03-11T19:01:52.730 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T19:01:52.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:52 smithi190 ceph-mon[36292]: Reconfiguring mon.c (monmap changed)... 2024-03-11T19:01:52.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:52 smithi190 ceph-mon[36292]: Reconfiguring daemon mon.c on smithi190 2024-03-11T19:01:52.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:52 smithi190 ceph-mon[36292]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi029=x;smithi170=y;smithi190=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:01:52.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:52 smithi190 ceph-mon[36292]: Saving service mgr spec with placement smithi029=x;smithi170=y;smithi190=z;count:3 2024-03-11T19:01:52.793 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:52 smithi170 ceph-mon[35690]: Reconfiguring mon.c (monmap changed)... 2024-03-11T19:01:52.794 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:52 smithi170 ceph-mon[35690]: Reconfiguring daemon mon.c on smithi190 2024-03-11T19:01:52.794 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:52 smithi170 ceph-mon[35690]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi029=x;smithi170=y;smithi190=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:01:52.794 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:52 smithi170 ceph-mon[35690]: Saving service mgr spec with placement smithi029=x;smithi170=y;smithi190=z;count:3 2024-03-11T19:01:52.798 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-11T19:01:52.798 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-11T19:01:52.798 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000110401 s, 4.6 MB/s 2024-03-11T19:01:52.799 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T19:01:52.854 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_4 2024-03-11T19:01:52.909 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T19:01:52.910 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:01:52.910 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-11T19:01:52.910 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:01:52.910 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:01:52.910 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-11 19:01:39.158771122 +0000 2024-03-11T19:01:52.910 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-11 18:56:05.569405927 +0000 2024-03-11T19:01:52.910 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-11 18:56:05.569405927 +0000 2024-03-11T19:01:52.910 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-11 18:56:05.569405927 +0000 2024-03-11T19:01:52.910 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T19:01:52.975 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-11T19:01:52.975 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-11T19:01:52.976 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 8.9121e-05 s, 5.7 MB/s 2024-03-11T19:01:52.977 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T19:01:53.032 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-11T19:01:53.032 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T19:01:53.051 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T19:01:53.051 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2024-03-11T19:01:53.109 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T19:01:53.110 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:01:53.110 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T19:01:53.110 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:01:53.110 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:01:53.110 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-11 19:01:29.106180339 +0000 2024-03-11T19:01:53.110 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-11 18:56:04.583667440 +0000 2024-03-11T19:01:53.110 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-11 18:56:04.583667440 +0000 2024-03-11T19:01:53.110 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-11 18:56:04.583667440 +0000 2024-03-11T19:01:53.110 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T19:01:53.180 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-11T19:01:53.180 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-11T19:01:53.180 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000229298 s, 2.2 MB/s 2024-03-11T19:01:53.181 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T19:01:53.239 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2024-03-11T19:01:53.295 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T19:01:53.296 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:01:53.296 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-11T19:01:53.296 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:01:53.296 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:01:53.296 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-11 19:01:29.106180339 +0000 2024-03-11T19:01:53.296 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-11 18:56:04.930657416 +0000 2024-03-11T19:01:53.296 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-11 18:56:04.930657416 +0000 2024-03-11T19:01:53.296 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-11 18:56:04.930657416 +0000 2024-03-11T19:01:53.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T19:01:53.364 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-11T19:01:53.365 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-11T19:01:53.365 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000203423 s, 2.5 MB/s 2024-03-11T19:01:53.366 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T19:01:53.423 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2024-03-11T19:01:53.479 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T19:01:53.479 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:01:53.479 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-11T19:01:53.479 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:01:53.479 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:01:53.479 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-11 19:01:29.106180339 +0000 2024-03-11T19:01:53.479 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-11 18:56:05.251648142 +0000 2024-03-11T19:01:53.479 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-11 18:56:05.251648142 +0000 2024-03-11T19:01:53.479 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-11 18:56:05.251648142 +0000 2024-03-11T19:01:53.480 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T19:01:53.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:53 smithi190 ceph-mon[36292]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:53.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:53 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:53.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:53 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:53.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:53 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:01:53.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:53 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:53.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:53 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:01:53.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:53 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:53.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:53 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T19:01:53.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:53 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T19:01:53.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:53 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T19:01:53.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:53 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:53.538 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-11T19:01:53.538 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-11T19:01:53.538 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000185375 s, 2.8 MB/s 2024-03-11T19:01:53.539 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T19:01:53.596 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2024-03-11T19:01:53.651 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T19:01:53.651 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:01:53.651 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-11T19:01:53.651 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:01:53.651 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:01:53.651 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-11 19:01:29.107180310 +0000 2024-03-11T19:01:53.651 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-11 18:56:05.583638551 +0000 2024-03-11T19:01:53.651 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-11 18:56:05.583638551 +0000 2024-03-11T19:01:53.652 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-11 18:56:05.583638551 +0000 2024-03-11T19:01:53.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T19:01:53.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:53 smithi029 ceph-mon[32439]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:53.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:53 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:53.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:53 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:53.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:53 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:01:53.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:53 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:53.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:53 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:01:53.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:53 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:53.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:53 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T19:01:53.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:53 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T19:01:53.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:53 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T19:01:53.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:53 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:53.718 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-11T19:01:53.718 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-11T19:01:53.718 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000169739 s, 3.0 MB/s 2024-03-11T19:01:53.719 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T19:01:53.776 INFO:tasks.cephadm:Deploying osd.0 on smithi029 with /dev/vg_nvme/lv_4... 2024-03-11T19:01:53.776 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T19:01:53.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:53 smithi170 ceph-mon[35690]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:53.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:53 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:53.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:53 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:53.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:53 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:01:53.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:53 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:53.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:53 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:01:53.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:53 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:53.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:53 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T19:01:53.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:53 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T19:01:53.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:53 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T19:01:53.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:53 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:54.016 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:01:54.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:54 smithi029 ceph-mon[32439]: Deploying daemon mgr.z on smithi190 2024-03-11T19:01:54.757 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:54 smithi190 ceph-mon[36292]: Deploying daemon mgr.z on smithi190 2024-03-11T19:01:54.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:54 smithi170 ceph-mon[35690]: Deploying daemon mgr.z on smithi190 2024-03-11T19:01:55.400 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:55 smithi190 systemd[1]: Starting Ceph mgr.z for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:01:55.500 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:55 smithi029 ceph-mon[32439]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:55.746 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:55 smithi170 ceph-mon[35690]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:55.757 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:55 smithi190 ceph-mon[36292]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:55.757 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:55 smithi190 podman[38134]: 2024-03-11 19:01:55.419839297 +0000 UTC m=+0.099172604 container create 8b60ab5ea9125925525c2ed4248e8051f5014deef150dea51ad2f5f496b7ac82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, ceph=True) 2024-03-11T19:01:55.758 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:55 smithi190 podman[38134]: 2024-03-11 19:01:55.338443029 +0000 UTC m=+0.017776342 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:01:55.758 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:55 smithi190 podman[38134]: 2024-03-11 19:01:55.755932347 +0000 UTC m=+0.435265652 container init 8b60ab5ea9125925525c2ed4248e8051f5014deef150dea51ad2f5f496b7ac82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:01:55.862 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T19:01:55.862 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T19:01:55.862 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-03-11T19:01:55.862 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-03-11T19:01:55.862 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00873924 s, 1.2 GB/s 2024-03-11T19:01:55.862 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-03-11T19:01:56.011 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:55 smithi190 podman[38134]: 2024-03-11 19:01:55.758710686 +0000 UTC m=+0.438044002 container start 8b60ab5ea9125925525c2ed4248e8051f5014deef150dea51ad2f5f496b7ac82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:01:56.012 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:55 smithi190 ceph-mgr[38153]: -- 172.21.15.190:0/944376222 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cdea7461a0 con 0x55cdea71d000 2024-03-11T19:01:56.012 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:55 smithi190 bash[38134]: 8b60ab5ea9125925525c2ed4248e8051f5014deef150dea51ad2f5f496b7ac82 2024-03-11T19:01:56.012 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:55 smithi190 systemd[1]: Started Ceph mgr.z for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:01:56.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_4 2024-03-11T19:01:56.690 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:01:56.761 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:56 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:01:56.297+0000 7fb4feb0d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T19:01:56.761 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:56 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:01:56.388+0000 7fb4feb0d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T19:01:57.124 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:56 smithi029 ceph-mon[32439]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:57.124 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:56 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:57.124 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:56 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:57.124 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:56 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:57.124 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:56 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T19:01:57.124 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:56 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T19:01:57.125 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:56 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T19:01:57.125 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:56 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:57.125 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:56 smithi029 ceph-mon[32439]: Deploying daemon mgr.y on smithi170 2024-03-11T19:01:57.258 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:01:57.025+0000 7fb4feb0d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T19:01:57.259 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:01:57.115+0000 7fb4feb0d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T19:01:57.259 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:01:57.255+0000 7fb4feb0d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T19:01:57.259 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:56 smithi190 ceph-mon[36292]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:57.259 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:56 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:57.259 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:56 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:57.259 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:56 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:57.259 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:56 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T19:01:57.259 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:56 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T19:01:57.259 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:56 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T19:01:57.260 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:56 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:57.260 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:56 smithi190 ceph-mon[36292]: Deploying daemon mgr.y on smithi170 2024-03-11T19:01:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:56 smithi170 ceph-mon[35690]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:56 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:56 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:56 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:56 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T19:01:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:56 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T19:01:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:56 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T19:01:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:56 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:56 smithi170 ceph-mon[35690]: Deploying daemon mgr.y on smithi170 2024-03-11T19:01:57.511 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:01:57.349+0000 7fb4feb0d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T19:01:57.512 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:01:57.430+0000 7fb4feb0d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T19:01:57.948 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:01:57.691+0000 7fb4feb0d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T19:01:57.948 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:01:57.797+0000 7fb4feb0d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T19:01:57.948 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:01:57.881+0000 7fb4feb0d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T19:01:58.039 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:01:57 smithi170 systemd[1]: Starting Ceph mgr.y for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:01:58.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.230+0000 7fe1fa3c7700 1 -- 172.21.15.29:0/415588784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1f41050d0 msgr2=0x7fe1f41074c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.230+0000 7fe1fa3c7700 1 --2- 172.21.15.29:0/415588784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1f41050d0 0x7fe1f41074c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc009990 tx=0x7fe1dc009ca0 comp rx=0 tx=0).stop 2024-03-11T19:01:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.230+0000 7fe1fa3c7700 1 -- 172.21.15.29:0/415588784 shutdown_connections 2024-03-11T19:01:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.230+0000 7fe1fa3c7700 1 --2- 172.21.15.29:0/415588784 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1f4107a00 0x7fe1f4109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.230+0000 7fe1fa3c7700 1 --2- 172.21.15.29:0/415588784 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1f41027a0 0x7fe1f4104b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.230+0000 7fe1fa3c7700 1 --2- 172.21.15.29:0/415588784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1f41050d0 0x7fe1f41074c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.230+0000 7fe1fa3c7700 1 -- 172.21.15.29:0/415588784 >> 172.21.15.29:0/415588784 conn(0x7fe1f40fc130 msgr2=0x7fe1f40fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:01:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.230+0000 7fe1fa3c7700 1 -- 172.21.15.29:0/415588784 shutdown_connections 2024-03-11T19:01:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.230+0000 7fe1fa3c7700 1 -- 172.21.15.29:0/415588784 wait complete. 2024-03-11T19:01:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.231+0000 7fe1fa3c7700 1 Processor -- start 2024-03-11T19:01:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.231+0000 7fe1fa3c7700 1 -- start start 2024-03-11T19:01:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.231+0000 7fe1fa3c7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1f41027a0 0x7fe1f4197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.231+0000 7fe1fa3c7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1f41050d0 0x7fe1f4198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.231+0000 7fe1fa3c7700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1f4107a00 0x7fe1f419f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.231+0000 7fe1fa3c7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f419f6e0 con 0x7fe1f41050d0 2024-03-11T19:01:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.231+0000 7fe1fa3c7700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f419f820 con 0x7fe1f4107a00 2024-03-11T19:01:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.231+0000 7fe1fa3c7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f419f9b0 con 0x7fe1f41027a0 2024-03-11T19:01:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.231+0000 7fe1ebfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1f41050d0 0x7fe1f4198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.231+0000 7fe1ebfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1f41050d0 0x7fe1f4198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58368/0 (socket says 172.21.15.29:58368) 2024-03-11T19:01:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.231+0000 7fe1ebfff700 1 -- 172.21.15.29:0/1553282507 learned_addr learned my addr 172.21.15.29:0/1553282507 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:01:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.231+0000 7fe1f3fff700 1 --2- 172.21.15.29:0/1553282507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1f41027a0 0x7fe1f4197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.231+0000 7fe1f8964700 1 --2- 172.21.15.29:0/1553282507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1f4107a00 0x7fe1f419f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.231+0000 7fe1ebfff700 1 -- 172.21.15.29:0/1553282507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1f41027a0 msgr2=0x7fe1f4197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.231+0000 7fe1ebfff700 1 --2- 172.21.15.29:0/1553282507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1f41027a0 0x7fe1f4197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.231+0000 7fe1ebfff700 1 -- 172.21.15.29:0/1553282507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1f4107a00 msgr2=0x7fe1f419f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:01:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.231+0000 7fe1ebfff700 1 --2- 172.21.15.29:0/1553282507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1f4107a00 0x7fe1f419f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:01:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.231+0000 7fe1ebfff700 1 -- 172.21.15.29:0/1553282507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc0095f0 con 0x7fe1f41050d0 2024-03-11T19:01:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.231+0000 7fe1ebfff700 1 --2- 172.21.15.29:0/1553282507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1f41050d0 0x7fe1f4198090 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc005910 tx=0x7fe1dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.232+0000 7fe1f1ffb700 1 -- 172.21.15.29:0/1553282507 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc01d070 con 0x7fe1f41050d0 2024-03-11T19:01:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.232+0000 7fe1f1ffb700 1 -- 172.21.15.29:0/1553282507 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe1dc0054e0 con 0x7fe1f41050d0 2024-03-11T19:01:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.232+0000 7fe1fa3c7700 1 -- 172.21.15.29:0/1553282507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f419fc30 con 0x7fe1f41050d0 2024-03-11T19:01:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.232+0000 7fe1f1ffb700 1 -- 172.21.15.29:0/1553282507 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc004a00 con 0x7fe1f41050d0 2024-03-11T19:01:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.232+0000 7fe1fa3c7700 1 -- 172.21.15.29:0/1553282507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f41a00a0 con 0x7fe1f41050d0 2024-03-11T19:01:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.233+0000 7fe1f1ffb700 1 -- 172.21.15.29:0/1553282507 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7fe1dc003be0 con 0x7fe1f41050d0 2024-03-11T19:01:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.233+0000 7fe1fa3c7700 1 -- 172.21.15.29:0/1553282507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f40fdcb0 con 0x7fe1f41050d0 2024-03-11T19:01:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.233+0000 7fe1f1ffb700 1 --2- 172.21.15.29:0/1553282507 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe1e0038f90 0x7fe1e003b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:01:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.233+0000 7fe1f1ffb700 1 -- 172.21.15.29:0/1553282507 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe1dc04c900 con 0x7fe1f41050d0 2024-03-11T19:01:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.233+0000 7fe1f3fff700 1 --2- 172.21.15.29:0/1553282507 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe1e0038f90 0x7fe1e003b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:01:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.233+0000 7fe1f3fff700 1 --2- 172.21.15.29:0/1553282507 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe1e0038f90 0x7fe1e003b450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe1e4005950 tx=0x7fe1e400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:01:58.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.238+0000 7fe1f1ffb700 1 -- 172.21.15.29:0/1553282507 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe1dc00e3e0 con 0x7fe1f41050d0 2024-03-11T19:01:58.261 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:01:57.984+0000 7fb4feb0d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T19:01:58.261 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:58 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:01:58.160+0000 7fb4feb0d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T19:01:58.308 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:01:58 smithi170 podman[37070]: 2024-03-11 19:01:58.03724516 +0000 UTC m=+0.112942740 container create 1172ef466c1c3d0fc59e2df6a62e9d8b0fd1aa388ca242a7f50c2fd6be9a2c1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-11T19:01:58.308 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:01:58 smithi170 podman[37070]: 2024-03-11 19:01:57.942914363 +0000 UTC m=+0.018611905 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:01:58.308 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:01:58 smithi170 podman[37070]: 2024-03-11 19:01:58.231193967 +0000 UTC m=+0.306891503 container init 1172ef466c1c3d0fc59e2df6a62e9d8b0fd1aa388ca242a7f50c2fd6be9a2c1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-11T19:01:58.308 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:01:58 smithi170 podman[37070]: 2024-03-11 19:01:58.233639433 +0000 UTC m=+0.309336971 container start 1172ef466c1c3d0fc59e2df6a62e9d8b0fd1aa388ca242a7f50c2fd6be9a2c1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:01:58.309 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:01:58 smithi170 ceph-mgr[37089]: -- 172.21.15.170:0/3312101506 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559c789e61a0 con 0x559c789bc800 2024-03-11T19:01:58.309 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:01:58 smithi170 bash[37070]: 1172ef466c1c3d0fc59e2df6a62e9d8b0fd1aa388ca242a7f50c2fd6be9a2c1e 2024-03-11T19:01:58.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:01:58.454+0000 7fe1fa3c7700 1 -- 172.21.15.29:0/1553282507 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1f402d800 con 0x7fe1e0038f90 2024-03-11T19:01:58.595 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:01:58 smithi170 systemd[1]: Started Ceph mgr.y for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:01:59.095 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:01:58 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:01:58.806+0000 7f1e9e590200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T19:01:59.095 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:01:58 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:01:58.902+0000 7f1e9e590200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T19:01:59.511 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:59 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:01:59.173+0000 7fb4feb0d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T19:01:59.512 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:59 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:01:59.302+0000 7fb4feb0d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T19:01:59.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:59 smithi190 ceph-mon[36292]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:59.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:59 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:59.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:59 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:59.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:59 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:59.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:59 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:59.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:59 smithi190 ceph-mon[36292]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:01:59.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:59 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:01:59.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:59 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:01:59.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:59 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:59.513 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:59 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:01:59.513 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:01:59 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:59.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:59 smithi029 ceph-mon[32439]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:59.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:59 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:59.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:59 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:59.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:59 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:59.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:59 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:59.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:59 smithi029 ceph-mon[32439]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:01:59.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:59 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:01:59.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:59 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:01:59.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:59 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:59.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:59 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:01:59.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:01:59 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:59.681 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:01:59 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:01:59.584+0000 7f1e9e590200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T19:01:59.681 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:01:59 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:01:59.677+0000 7f1e9e590200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T19:01:59.681 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:59 smithi170 ceph-mon[35690]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:01:59.681 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:59 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:59.681 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:59 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:59.681 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:59 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:59.681 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:59 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:59.682 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:59 smithi170 ceph-mon[35690]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:01:59.682 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:59 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:01:59.682 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:59 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:01:59.682 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:59 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:01:59.682 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:59 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:01:59.682 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:01:59 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:01:59.946 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:59 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:01:59.646+0000 7fb4feb0d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T19:01:59.946 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:59 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:01:59.737+0000 7fb4feb0d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T19:01:59.946 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:01:59 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:01:59.943+0000 7fb4feb0d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:02:00.014 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:01:59 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:01:59.828+0000 7f1e9e590200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T19:02:00.014 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:01:59 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:01:59.925+0000 7f1e9e590200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T19:02:00.014 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:02:00 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:02:00.011+0000 7f1e9e590200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T19:02:00.261 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:02:00 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:02:00.139+0000 7fb4feb0d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T19:02:00.567 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:02:00 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:02:00.276+0000 7f1e9e590200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T19:02:00.567 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:02:00 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:02:00.386+0000 7f1e9e590200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T19:02:00.567 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:02:00 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:02:00.471+0000 7f1e9e590200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T19:02:00.568 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:02:00 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:02:00.564+0000 7f1e9e590200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T19:02:00.686 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:02:00 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:02:00.423+0000 7fb4feb0d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:02:00.686 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:02:00 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:02:00.683+0000 7fb4feb0d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T19:02:00.842 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:02:00 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:02:00.739+0000 7f1e9e590200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T19:02:01.011 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:02:00 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:02:00.917+0000 7fb4feb0d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T19:02:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:00 smithi190 ceph-mon[36292]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:02:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:00 smithi190 ceph-mon[36292]: Reconfiguring mgr.x (unknown last config time)... 2024-03-11T19:02:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T19:02:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T19:02:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:00 smithi190 ceph-mon[36292]: Reconfiguring daemon mgr.x on smithi029 2024-03-11T19:02:01.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:00 smithi170 ceph-mon[35690]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:01.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:01.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:01.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:01.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:01.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:01.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:02:01.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:01.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:00 smithi170 ceph-mon[35690]: Reconfiguring mgr.x (unknown last config time)... 2024-03-11T19:02:01.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T19:02:01.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T19:02:01.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:01.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:00 smithi170 ceph-mon[35690]: Reconfiguring daemon mgr.x on smithi029 2024-03-11T19:02:01.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:00 smithi029 ceph-mon[32439]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:01.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:01.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:01.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:01.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:01.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:01.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:02:01.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:01.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:00 smithi029 ceph-mon[32439]: Reconfiguring mgr.x (unknown last config time)... 2024-03-11T19:02:01.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T19:02:01.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T19:02:01.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:01.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:00 smithi029 ceph-mon[32439]: Reconfiguring daemon mgr.x on smithi029 2024-03-11T19:02:02.011 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:02:01 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:02:01.706+0000 7fb4feb0d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T19:02:02.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:01 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2702979515' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62b6ad23-8635-47d9-9485-c91c1d1f942d"}]: dispatch 2024-03-11T19:02:02.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:01 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2702979515' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62b6ad23-8635-47d9-9485-c91c1d1f942d"}]': finished 2024-03-11T19:02:02.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:01 smithi190 ceph-mon[36292]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T19:02:02.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:01 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:02:02.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:01 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1241710088' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:02:02.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:01 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2702979515' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62b6ad23-8635-47d9-9485-c91c1d1f942d"}]: dispatch 2024-03-11T19:02:02.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:01 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2702979515' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62b6ad23-8635-47d9-9485-c91c1d1f942d"}]': finished 2024-03-11T19:02:02.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:01 smithi170 ceph-mon[35690]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T19:02:02.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:01 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:02:02.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:01 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1241710088' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:02:02.096 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:02:01 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:02:01.704+0000 7f1e9e590200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T19:02:02.096 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:02:01 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:02:01.833+0000 7f1e9e590200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T19:02:02.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:01 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2702979515' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62b6ad23-8635-47d9-9485-c91c1d1f942d"}]: dispatch 2024-03-11T19:02:02.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:01 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2702979515' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62b6ad23-8635-47d9-9485-c91c1d1f942d"}]': finished 2024-03-11T19:02:02.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:01 smithi029 ceph-mon[32439]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T19:02:02.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:01 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:02:02.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:01 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1241710088' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:02:02.453 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:02:02 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:02:02.170+0000 7f1e9e590200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T19:02:02.453 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:02:02 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:02:02.259+0000 7f1e9e590200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T19:02:02.453 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:02:02 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:02:02.450+0000 7f1e9e590200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:02:02.586 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:02:02 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:02:02.279+0000 7fb4feb0d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T19:02:02.586 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:02:02 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:02:02.373+0000 7fb4feb0d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T19:02:02.586 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:02:02 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z[38149]: 2024-03-11T19:02:02.583+0000 7fb4feb0d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T19:02:02.845 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:02:02 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:02:02.633+0000 7f1e9e590200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T19:02:03.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:02:03.073+0000 7fe1f1ffb700 1 -- 172.21.15.29:0/1553282507 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 14) v1 ==== 91024+0+0 (secure 0 0 0) 0x7fe1dc004ee0 con 0x7fe1f41050d0 2024-03-11T19:02:03.190 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:03 smithi170 ceph-mon[35690]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:03.190 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:03 smithi170 ceph-mon[35690]: Standby manager daemon z started 2024-03-11T19:02:03.190 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:03 smithi170 ceph-mon[35690]: from='mgr.? 172.21.15.190:0/1545354178' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T19:02:03.190 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:03 smithi170 ceph-mon[35690]: from='mgr.? 172.21.15.190:0/1545354178' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T19:02:03.190 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:03 smithi170 ceph-mon[35690]: from='mgr.? 172.21.15.190:0/1545354178' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T19:02:03.190 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:03 smithi170 ceph-mon[35690]: from='mgr.? 172.21.15.190:0/1545354178' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T19:02:03.191 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:02:02 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:02:02.916+0000 7f1e9e590200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:02:03.191 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:02:03 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:02:03.187+0000 7f1e9e590200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T19:02:03.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:03 smithi029 ceph-mon[32439]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:03.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:03 smithi029 ceph-mon[32439]: Standby manager daemon z started 2024-03-11T19:02:03.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:03 smithi029 ceph-mon[32439]: from='mgr.? 172.21.15.190:0/1545354178' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T19:02:03.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:03 smithi029 ceph-mon[32439]: from='mgr.? 172.21.15.190:0/1545354178' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T19:02:03.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:03 smithi029 ceph-mon[32439]: from='mgr.? 172.21.15.190:0/1545354178' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T19:02:03.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:03 smithi029 ceph-mon[32439]: from='mgr.? 172.21.15.190:0/1545354178' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T19:02:03.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:03 smithi190 ceph-mon[36292]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:03 smithi190 ceph-mon[36292]: Standby manager daemon z started 2024-03-11T19:02:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:03 smithi190 ceph-mon[36292]: from='mgr.? 172.21.15.190:0/1545354178' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T19:02:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:03 smithi190 ceph-mon[36292]: from='mgr.? 172.21.15.190:0/1545354178' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T19:02:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:03 smithi190 ceph-mon[36292]: from='mgr.? 172.21.15.190:0/1545354178' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T19:02:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:03 smithi190 ceph-mon[36292]: from='mgr.? 172.21.15.190:0/1545354178' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T19:02:03.595 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:02:03 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:02:03.420+0000 7f1e9e590200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T19:02:04.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:04 smithi029 ceph-mon[32439]: mgrmap e14: x(active, since 79s), standbys: z 2024-03-11T19:02:04.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:04 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T19:02:04.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:04 smithi170 ceph-mon[35690]: mgrmap e14: x(active, since 79s), standbys: z 2024-03-11T19:02:04.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:04 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T19:02:04.346 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:02:04 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:02:04.210+0000 7f1e9e590200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T19:02:04.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:04 smithi190 ceph-mon[36292]: mgrmap e14: x(active, since 79s), standbys: z 2024-03-11T19:02:04.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:04 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T19:02:05.086 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:02:04 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:02:04.777+0000 7f1e9e590200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T19:02:05.087 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:02:04 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:02:04.873+0000 7f1e9e590200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T19:02:05.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:05 smithi029 ceph-mon[32439]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:05.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:05 smithi170 ceph-mon[35690]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:05.345 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:02:05 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y[37085]: 2024-03-11T19:02:05.088+0000 7f1e9e590200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T19:02:05.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:05 smithi190 ceph-mon[36292]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:06.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:02:06.093+0000 7fe1f1ffb700 1 -- 172.21.15.29:0/1553282507 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7fe1dc04c410 con 0x7fe1f41050d0 2024-03-11T19:02:06.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:06 smithi170 ceph-mon[35690]: from='mgr.? 172.21.15.170:0/632535989' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T19:02:06.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:06 smithi170 ceph-mon[35690]: from='mgr.? 172.21.15.170:0/632535989' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T19:02:06.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:06 smithi170 ceph-mon[35690]: Standby manager daemon y started 2024-03-11T19:02:06.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:06 smithi170 ceph-mon[35690]: from='mgr.? 172.21.15.170:0/632535989' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T19:02:06.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:06 smithi170 ceph-mon[35690]: from='mgr.? 172.21.15.170:0/632535989' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T19:02:06.400 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:06 smithi029 ceph-mon[32439]: from='mgr.? 172.21.15.170:0/632535989' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T19:02:06.400 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:06 smithi029 ceph-mon[32439]: from='mgr.? 172.21.15.170:0/632535989' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T19:02:06.400 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:06 smithi029 ceph-mon[32439]: Standby manager daemon y started 2024-03-11T19:02:06.400 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:06 smithi029 ceph-mon[32439]: from='mgr.? 172.21.15.170:0/632535989' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T19:02:06.401 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:06 smithi029 ceph-mon[32439]: from='mgr.? 172.21.15.170:0/632535989' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T19:02:06.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:06 smithi190 ceph-mon[36292]: from='mgr.? 172.21.15.170:0/632535989' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T19:02:06.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:06 smithi190 ceph-mon[36292]: from='mgr.? 172.21.15.170:0/632535989' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T19:02:06.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:06 smithi190 ceph-mon[36292]: Standby manager daemon y started 2024-03-11T19:02:06.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:06 smithi190 ceph-mon[36292]: from='mgr.? 172.21.15.170:0/632535989' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T19:02:06.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:06 smithi190 ceph-mon[36292]: from='mgr.? 172.21.15.170:0/632535989' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T19:02:07.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:07 smithi170 ceph-mon[35690]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:07.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:07 smithi170 ceph-mon[35690]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-03-11T19:02:07.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T19:02:07.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:07.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:07.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:02:07.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:07.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:02:07.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:07.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:07 smithi029 ceph-mon[32439]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:07.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:07 smithi029 ceph-mon[32439]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-03-11T19:02:07.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T19:02:07.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:07.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:07.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:02:07.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:07.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:02:07.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:07.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:07 smithi190 ceph-mon[36292]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:07.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:07 smithi190 ceph-mon[36292]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-03-11T19:02:07.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T19:02:07.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:07.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:07.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:02:07.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:07.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:02:07.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:09.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:09 smithi170 ceph-mon[35690]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:09.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:09 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T19:02:09.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:09 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:09.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:09 smithi029 ceph-mon[32439]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:09.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:09 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T19:02:09.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:09 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:09.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:09 smithi190 ceph-mon[36292]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:09.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:09 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T19:02:09.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:09 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:10.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:10 smithi029 ceph-mon[32439]: Deploying daemon osd.0 on smithi029 2024-03-11T19:02:10.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:10 smithi190 ceph-mon[36292]: Deploying daemon osd.0 on smithi029 2024-03-11T19:02:10.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:10 smithi170 ceph-mon[35690]: Deploying daemon osd.0 on smithi029 2024-03-11T19:02:11.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:11 smithi029 ceph-mon[32439]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:11.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:11 smithi190 ceph-mon[36292]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:11.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:11 smithi170 ceph-mon[35690]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:13.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:13 smithi029 ceph-mon[32439]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:13.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:13 smithi190 ceph-mon[36292]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:13.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:13 smithi170 ceph-mon[35690]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:15.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:15 smithi029 ceph-mon[32439]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:15.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:15 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:15.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:15 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:15.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:15 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:02:15.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:15 smithi170 ceph-mon[35690]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:15.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:15 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:15.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:15 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:15.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:15 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:02:15.514 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:15 smithi190 ceph-mon[36292]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:15.514 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:15 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:15.514 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:15 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:15.514 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:15 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:02:17.646 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:17 smithi029 ceph-mon[32439]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:17.647 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:17 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:17.647 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:17 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:17.647 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:17 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:17.647 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:17 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:02:17.647 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:17 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:17.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:02:17.715+0000 7fe1f1ffb700 1 -- 172.21.15.29:0/1553282507 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe1f402d800 con 0x7fe1e0038f90 2024-03-11T19:02:17.719 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 0 on host 'smithi029' 2024-03-11T19:02:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:02:17.718+0000 7fe1fa3c7700 1 -- 172.21.15.29:0/1553282507 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe1e0038f90 msgr2=0x7fe1e003b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:02:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:02:17.718+0000 7fe1fa3c7700 1 --2- 172.21.15.29:0/1553282507 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe1e0038f90 0x7fe1e003b450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe1e4005950 tx=0x7fe1e400b410 comp rx=0 tx=0).stop 2024-03-11T19:02:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:02:17.718+0000 7fe1fa3c7700 1 -- 172.21.15.29:0/1553282507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1f41050d0 msgr2=0x7fe1f4198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:02:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:02:17.718+0000 7fe1fa3c7700 1 --2- 172.21.15.29:0/1553282507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1f41050d0 0x7fe1f4198090 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc005910 tx=0x7fe1dc004730 comp rx=0 tx=0).stop 2024-03-11T19:02:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:02:17.718+0000 7fe1fa3c7700 1 -- 172.21.15.29:0/1553282507 shutdown_connections 2024-03-11T19:02:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:02:17.718+0000 7fe1fa3c7700 1 --2- 172.21.15.29:0/1553282507 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe1e0038f90 0x7fe1e003b450 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:02:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:02:17.718+0000 7fe1fa3c7700 1 --2- 172.21.15.29:0/1553282507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1f4107a00 0x7fe1f419f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:02:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:02:17.718+0000 7fe1fa3c7700 1 --2- 172.21.15.29:0/1553282507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1f41027a0 0x7fe1f4197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:02:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:02:17.718+0000 7fe1fa3c7700 1 --2- 172.21.15.29:0/1553282507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1f41050d0 0x7fe1f4198090 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:02:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:02:17.718+0000 7fe1fa3c7700 1 -- 172.21.15.29:0/1553282507 >> 172.21.15.29:0/1553282507 conn(0x7fe1f40fc130 msgr2=0x7fe1f4103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:02:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:02:17.718+0000 7fe1fa3c7700 1 -- 172.21.15.29:0/1553282507 shutdown_connections 2024-03-11T19:02:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:02:17.718+0000 7fe1fa3c7700 1 -- 172.21.15.29:0/1553282507 wait complete. 2024-03-11T19:02:17.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:17 smithi170 ceph-mon[35690]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:17.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:17 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:17.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:17 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:17.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:17 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:17.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:17 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:02:17.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:17 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:18.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:17 smithi190 ceph-mon[36292]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:18.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:17 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:18.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:17 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:18.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:17 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:18.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:17 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:02:18.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:17 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:18.350 DEBUG:teuthology.orchestra.run.smithi029:osd.0> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.0.service 2024-03-11T19:02:18.353 INFO:tasks.cephadm:Deploying osd.1 on smithi170 with /dev/vg_nvme/lv_4... 2024-03-11T19:02:18.353 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T19:02:18.596 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:02:18.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:18 smithi029 ceph-mon[32439]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:18.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:18 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:18.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:18 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:18.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:18 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:02:18.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:18 smithi029 ceph-mon[32439]: from='osd.0 [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T19:02:19.004 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:18 smithi170 ceph-mon[35690]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:19.005 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:18 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:19.005 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:18 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:19.005 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:18 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:02:19.005 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:18 smithi170 ceph-mon[35690]: from='osd.0 [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T19:02:19.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:18 smithi190 ceph-mon[36292]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:19.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:18 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:19.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:18 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:19.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:18 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:02:19.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:18 smithi190 ceph-mon[36292]: from='osd.0 [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T19:02:19.757 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:19 smithi029 ceph-mon[32439]: from='osd.0 [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T19:02:19.757 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:19 smithi029 ceph-mon[32439]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T19:02:19.757 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:19 smithi029 ceph-mon[32439]: from='osd.0 [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-11T19:02:19.757 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:19 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:02:19.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:19 smithi170 ceph-mon[35690]: from='osd.0 [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T19:02:19.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:19 smithi170 ceph-mon[35690]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T19:02:19.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:19 smithi170 ceph-mon[35690]: from='osd.0 [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-11T19:02:19.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:19 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:02:20.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:19 smithi190 ceph-mon[36292]: from='osd.0 [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T19:02:20.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:19 smithi190 ceph-mon[36292]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T19:02:20.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:19 smithi190 ceph-mon[36292]: from='osd.0 [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-11T19:02:20.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:19 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:02:20.411 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T19:02:20.411 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T19:02:20.411 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-03-11T19:02:20.411 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-03-11T19:02:20.411 INFO:teuthology.orchestra.run.smithi170.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00839022 s, 1.2 GB/s 2024-03-11T19:02:20.411 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-03-11T19:02:20.592 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:02:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-0[46761]: 2024-03-11T19:02:20.200+0000 7fe8f4b46700 -1 osd.0 0 waiting for initial osdmap 2024-03-11T19:02:20.593 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:02:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-0[46761]: 2024-03-11T19:02:20.204+0000 7fe8ec931700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:02:20.900 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:20 smithi029 ceph-mon[32439]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:20.900 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:20 smithi029 ceph-mon[32439]: from='osd.0 [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-11T19:02:20.900 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:20 smithi029 ceph-mon[32439]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T19:02:20.900 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:02:20.901 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:02:20.901 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:20 smithi029 ceph-mon[32439]: from='osd.0 [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152]' entity='osd.0' 2024-03-11T19:02:20.931 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:20 smithi170 ceph-mon[35690]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:20.931 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:20 smithi170 ceph-mon[35690]: from='osd.0 [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-11T19:02:20.931 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:20 smithi170 ceph-mon[35690]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T19:02:20.931 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:02:20.931 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:02:20.931 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:20 smithi170 ceph-mon[35690]: from='osd.0 [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152]' entity='osd.0' 2024-03-11T19:02:20.960 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi170:vg_nvme/lv_4 2024-03-11T19:02:21.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:20 smithi190 ceph-mon[36292]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:21.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:20 smithi190 ceph-mon[36292]: from='osd.0 [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-11T19:02:21.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:20 smithi190 ceph-mon[36292]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T19:02:21.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:02:21.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:02:21.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:20 smithi190 ceph-mon[36292]: from='osd.0 [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152]' entity='osd.0' 2024-03-11T19:02:21.204 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:02:21.886 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:21 smithi170 ceph-mon[35690]: purged_snaps scrub starts 2024-03-11T19:02:21.887 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:21 smithi170 ceph-mon[35690]: purged_snaps scrub ok 2024-03-11T19:02:21.887 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:21 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:02:21.887 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:21 smithi170 ceph-mon[35690]: osd.0 [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152] boot 2024-03-11T19:02:21.887 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:21 smithi170 ceph-mon[35690]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T19:02:21.887 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:21 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:02:21.887 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:21 smithi170 ceph-mon[35690]: Detected new or changed devices on smithi029 2024-03-11T19:02:21.887 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:21 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:21.887 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:21 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:21.887 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:21 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:02:21.887 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:21 smithi170 ceph-mon[35690]: Adjusting osd_memory_target on smithi029 to 17093M 2024-03-11T19:02:21.887 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:21 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:21.887 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:21 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:21.887 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:21 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:02:21.888 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:21 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:22.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:21 smithi190 ceph-mon[36292]: purged_snaps scrub starts 2024-03-11T19:02:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:21 smithi190 ceph-mon[36292]: purged_snaps scrub ok 2024-03-11T19:02:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:21 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:02:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:21 smithi190 ceph-mon[36292]: osd.0 [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152] boot 2024-03-11T19:02:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:21 smithi190 ceph-mon[36292]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T19:02:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:21 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:02:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:21 smithi190 ceph-mon[36292]: Detected new or changed devices on smithi029 2024-03-11T19:02:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:21 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:21 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:21 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:02:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:21 smithi190 ceph-mon[36292]: Adjusting osd_memory_target on smithi029 to 17093M 2024-03-11T19:02:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:21 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:21 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:21 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:02:22.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:21 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:22.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:21 smithi029 ceph-mon[32439]: purged_snaps scrub starts 2024-03-11T19:02:22.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:21 smithi029 ceph-mon[32439]: purged_snaps scrub ok 2024-03-11T19:02:22.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:21 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:02:22.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:21 smithi029 ceph-mon[32439]: osd.0 [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152] boot 2024-03-11T19:02:22.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:21 smithi029 ceph-mon[32439]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T19:02:22.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:21 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:02:22.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:21 smithi029 ceph-mon[32439]: Detected new or changed devices on smithi029 2024-03-11T19:02:22.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:21 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:22.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:21 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:22.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:21 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:02:22.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:21 smithi029 ceph-mon[32439]: Adjusting osd_memory_target on smithi029 to 17093M 2024-03-11T19:02:22.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:21 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:22.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:21 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:22.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:21 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:02:22.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:21 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:22.667 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.657+0000 7fd6a1fdf700 1 -- 172.21.15.170:0/3178407858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd69c100f70 msgr2=0x7fd69c1013f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:02:22.667 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.657+0000 7fd6a1fdf700 1 --2- 172.21.15.170:0/3178407858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd69c100f70 0x7fd69c1013f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd690009990 tx=0x7fd690009ca0 comp rx=0 tx=0).stop 2024-03-11T19:02:22.667 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.657+0000 7fd6a1fdf700 1 -- 172.21.15.170:0/3178407858 shutdown_connections 2024-03-11T19:02:22.667 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.657+0000 7fd6a1fdf700 1 --2- 172.21.15.170:0/3178407858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd69c101930 0x7fd69c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:02:22.667 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.657+0000 7fd6a1fdf700 1 --2- 172.21.15.170:0/3178407858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd69c100610 0x7fd69c100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:02:22.667 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.657+0000 7fd6a1fdf700 1 --2- 172.21.15.170:0/3178407858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd69c100f70 0x7fd69c1013f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:02:22.667 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.657+0000 7fd6a1fdf700 1 -- 172.21.15.170:0/3178407858 >> 172.21.15.170:0/3178407858 conn(0x7fd69c0fc1d0 msgr2=0x7fd69c0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:02:22.667 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.657+0000 7fd6a1fdf700 1 -- 172.21.15.170:0/3178407858 shutdown_connections 2024-03-11T19:02:22.667 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.657+0000 7fd6a1fdf700 1 -- 172.21.15.170:0/3178407858 wait complete. 2024-03-11T19:02:22.667 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.658+0000 7fd6a1fdf700 1 Processor -- start 2024-03-11T19:02:22.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.658+0000 7fd6a1fdf700 1 -- start start 2024-03-11T19:02:22.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.658+0000 7fd6a1fdf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd69c100610 0x7fd69c199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:02:22.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.658+0000 7fd6a1fdf700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd69c100f70 0x7fd69c19a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:02:22.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.658+0000 7fd6a1fdf700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd69c101930 0x7fd69c1a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:02:22.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.658+0000 7fd6a1fdf700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c1a1900 con 0x7fd69c100610 2024-03-11T19:02:22.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.658+0000 7fd6a1fdf700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c1a1a40 con 0x7fd69c101930 2024-03-11T19:02:22.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.658+0000 7fd6a1fdf700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c1a1c00 con 0x7fd69c100f70 2024-03-11T19:02:22.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.659+0000 7fd69bfff700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd69c101930 0x7fd69c1a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:02:22.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.659+0000 7fd69bfff700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd69c101930 0x7fd69c1a1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:49124/0 (socket says 172.21.15.170:49124) 2024-03-11T19:02:22.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.659+0000 7fd69bfff700 1 -- 172.21.15.170:0/595358887 learned_addr learned my addr 172.21.15.170:0/595358887 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-11T19:02:22.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.659+0000 7fd69b7fe700 1 --2- 172.21.15.170:0/595358887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd69c100610 0x7fd69c199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:02:22.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.659+0000 7fd69affd700 1 --2- 172.21.15.170:0/595358887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd69c100f70 0x7fd69c19a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:02:22.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.659+0000 7fd69bfff700 1 -- 172.21.15.170:0/595358887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd69c100f70 msgr2=0x7fd69c19a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:02:22.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.659+0000 7fd69bfff700 1 --2- 172.21.15.170:0/595358887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd69c100f70 0x7fd69c19a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:02:22.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.659+0000 7fd69bfff700 1 -- 172.21.15.170:0/595358887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd69c100610 msgr2=0x7fd69c199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:02:22.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.659+0000 7fd69bfff700 1 --2- 172.21.15.170:0/595358887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd69c100610 0x7fd69c199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:02:22.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.659+0000 7fd69bfff700 1 -- 172.21.15.170:0/595358887 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6900095f0 con 0x7fd69c101930 2024-03-11T19:02:22.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.659+0000 7fd69affd700 1 --2- 172.21.15.170:0/595358887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd69c100f70 0x7fd69c19a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:02:22.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.659+0000 7fd69bfff700 1 --2- 172.21.15.170:0/595358887 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd69c101930 0x7fd69c1a1330 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd69400b970 tx=0x7fd69400bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:02:22.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.659+0000 7fd698ff9700 1 -- 172.21.15.170:0/595358887 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69400c930 con 0x7fd69c101930 2024-03-11T19:02:22.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.659+0000 7fd698ff9700 1 -- 172.21.15.170:0/595358887 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd694014440 con 0x7fd69c101930 2024-03-11T19:02:22.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.659+0000 7fd6a1fdf700 1 -- 172.21.15.170:0/595358887 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd69c1a1ee0 con 0x7fd69c101930 2024-03-11T19:02:22.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.659+0000 7fd698ff9700 1 -- 172.21.15.170:0/595358887 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6940075d0 con 0x7fd69c101930 2024-03-11T19:02:22.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.659+0000 7fd6a1fdf700 1 -- 172.21.15.170:0/595358887 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69c1a2300 con 0x7fd69c101930 2024-03-11T19:02:22.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.660+0000 7fd6a1fdf700 1 -- 172.21.15.170:0/595358887 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd69c068930 con 0x7fd69c101930 2024-03-11T19:02:22.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.661+0000 7fd698ff9700 1 -- 172.21.15.170:0/595358887 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7fd6940145b0 con 0x7fd69c101930 2024-03-11T19:02:22.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.662+0000 7fd698ff9700 1 --2- 172.21.15.170:0/595358887 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd6840a1ab0 0x7fd6840a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:02:22.670 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.662+0000 7fd698ff9700 1 -- 172.21.15.170:0/595358887 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1767+0+0 (secure 0 0 0) 0x7fd694006e40 con 0x7fd69c101930 2024-03-11T19:02:22.670 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.662+0000 7fd69b7fe700 1 --2- 172.21.15.170:0/595358887 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd6840a1ab0 0x7fd6840a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:02:22.670 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.663+0000 7fd69b7fe700 1 --2- 172.21.15.170:0/595358887 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd6840a1ab0 0x7fd6840a3f70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd68c005950 tx=0x7fd68c00a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:02:22.670 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.666+0000 7fd698ff9700 1 -- 172.21.15.170:0/595358887 <== mon.2 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd694095e00 con 0x7fd69c101930 2024-03-11T19:02:22.870 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:22.867+0000 7fd6a1fdf700 1 -- 172.21.15.170:0/595358887 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd69c02d950 con 0x7fd6840a1ab0 2024-03-11T19:02:23.591 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:23 smithi170 ceph-mon[35690]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:23.591 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:23 smithi170 ceph-mon[35690]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T19:02:23.591 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:23 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:02:23.591 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:23 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:02:23.591 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:23 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:23.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:23 smithi029 ceph-mon[32439]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:23.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:23 smithi029 ceph-mon[32439]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T19:02:23.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:02:23.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:02:23.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:23.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:23 smithi190 ceph-mon[36292]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:02:23.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:23 smithi190 ceph-mon[36292]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T19:02:23.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:23 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:02:23.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:23 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:02:23.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:23 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:24.506 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:24 smithi170 ceph-mon[35690]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:02:24.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:24 smithi029 ceph-mon[32439]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:02:24.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:24 smithi190 ceph-mon[36292]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:02:25.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:25 smithi170 ceph-mon[35690]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:25.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:25 smithi029 ceph-mon[32439]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:25.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:25 smithi190 ceph-mon[36292]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:26.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:26 smithi170 ceph-mon[35690]: from='client.? 172.21.15.170:0/2103006311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ae8a614-7d37-4e39-b8cc-542895cc5a30"}]: dispatch 2024-03-11T19:02:26.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:26 smithi170 ceph-mon[35690]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ae8a614-7d37-4e39-b8cc-542895cc5a30"}]: dispatch 2024-03-11T19:02:26.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:26 smithi170 ceph-mon[35690]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4ae8a614-7d37-4e39-b8cc-542895cc5a30"}]': finished 2024-03-11T19:02:26.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:26 smithi170 ceph-mon[35690]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T19:02:26.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:26 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:02:26.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:26 smithi170 ceph-mon[35690]: from='client.? 172.21.15.170:0/337572145' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:02:26.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:26 smithi029 ceph-mon[32439]: from='client.? 172.21.15.170:0/2103006311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ae8a614-7d37-4e39-b8cc-542895cc5a30"}]: dispatch 2024-03-11T19:02:26.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:26 smithi029 ceph-mon[32439]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ae8a614-7d37-4e39-b8cc-542895cc5a30"}]: dispatch 2024-03-11T19:02:26.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:26 smithi029 ceph-mon[32439]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4ae8a614-7d37-4e39-b8cc-542895cc5a30"}]': finished 2024-03-11T19:02:26.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:26 smithi029 ceph-mon[32439]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T19:02:26.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:26 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:02:26.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:26 smithi029 ceph-mon[32439]: from='client.? 172.21.15.170:0/337572145' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:02:26.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:26 smithi190 ceph-mon[36292]: from='client.? 172.21.15.170:0/2103006311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ae8a614-7d37-4e39-b8cc-542895cc5a30"}]: dispatch 2024-03-11T19:02:26.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:26 smithi190 ceph-mon[36292]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ae8a614-7d37-4e39-b8cc-542895cc5a30"}]: dispatch 2024-03-11T19:02:26.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:26 smithi190 ceph-mon[36292]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4ae8a614-7d37-4e39-b8cc-542895cc5a30"}]': finished 2024-03-11T19:02:26.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:26 smithi190 ceph-mon[36292]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T19:02:26.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:26 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:02:26.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:26 smithi190 ceph-mon[36292]: from='client.? 172.21.15.170:0/337572145' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:02:27.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:27 smithi029 ceph-mon[32439]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:27.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:27 smithi190 ceph-mon[36292]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:27.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:27 smithi170 ceph-mon[35690]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:29.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:29 smithi170 ceph-mon[35690]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:29.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:29 smithi029 ceph-mon[32439]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:29.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:29 smithi190 ceph-mon[36292]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:31.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:31 smithi170 ceph-mon[35690]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:31.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:31 smithi029 ceph-mon[32439]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:31.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:31 smithi190 ceph-mon[36292]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:32.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:32 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T19:02:32.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:32 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:32.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:32 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T19:02:32.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:32 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:32.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:32 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T19:02:32.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:32 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:33.624 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:33 smithi170 ceph-mon[35690]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:33.624 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:33 smithi170 ceph-mon[35690]: Deploying daemon osd.1 on smithi170 2024-03-11T19:02:33.718 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:33 smithi029 ceph-mon[32439]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:33.719 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:33 smithi029 ceph-mon[32439]: Deploying daemon osd.1 on smithi170 2024-03-11T19:02:33.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:33 smithi190 ceph-mon[36292]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:33.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:33 smithi190 ceph-mon[36292]: Deploying daemon osd.1 on smithi170 2024-03-11T19:02:35.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:35 smithi170 ceph-mon[35690]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:35.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:35 smithi190 ceph-mon[36292]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:35.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:35 smithi029 ceph-mon[32439]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:37.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:37 smithi190 ceph-mon[36292]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:37.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:37 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:37.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:37 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:37.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:37 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:02:37.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:37 smithi170 ceph-mon[35690]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:37.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:37 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:37.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:37 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:37.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:37 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:02:37.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:37 smithi029 ceph-mon[32439]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:37.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:37 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:37.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:37 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:37.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:37 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:02:39.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:39 smithi190 ceph-mon[36292]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:39.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:39 smithi170 ceph-mon[35690]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:39.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:39 smithi029 ceph-mon[32439]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:40.486 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:40.483+0000 7fd698ff9700 1 -- 172.21.15.170:0/595358887 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd69c02d950 con 0x7fd6840a1ab0 2024-03-11T19:02:40.487 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 1 on host 'smithi170' 2024-03-11T19:02:40.488 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:40.485+0000 7fd6a1fdf700 1 -- 172.21.15.170:0/595358887 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd6840a1ab0 msgr2=0x7fd6840a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:02:40.489 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:40.485+0000 7fd6a1fdf700 1 --2- 172.21.15.170:0/595358887 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd6840a1ab0 0x7fd6840a3f70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd68c005950 tx=0x7fd68c00a300 comp rx=0 tx=0).stop 2024-03-11T19:02:40.489 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:40.485+0000 7fd6a1fdf700 1 -- 172.21.15.170:0/595358887 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd69c101930 msgr2=0x7fd69c1a1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:02:40.489 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:40.485+0000 7fd6a1fdf700 1 --2- 172.21.15.170:0/595358887 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd69c101930 0x7fd69c1a1330 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd69400b970 tx=0x7fd69400bc80 comp rx=0 tx=0).stop 2024-03-11T19:02:40.489 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:40.485+0000 7fd6a1fdf700 1 -- 172.21.15.170:0/595358887 shutdown_connections 2024-03-11T19:02:40.489 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:40.485+0000 7fd6a1fdf700 1 --2- 172.21.15.170:0/595358887 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd6840a1ab0 0x7fd6840a3f70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:02:40.489 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:40.485+0000 7fd6a1fdf700 1 --2- 172.21.15.170:0/595358887 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd69c101930 0x7fd69c1a1330 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:02:40.489 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:40.485+0000 7fd6a1fdf700 1 --2- 172.21.15.170:0/595358887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd69c100610 0x7fd69c199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:02:40.489 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:40.485+0000 7fd6a1fdf700 1 --2- 172.21.15.170:0/595358887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd69c100f70 0x7fd69c19a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:02:40.489 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:40.485+0000 7fd6a1fdf700 1 -- 172.21.15.170:0/595358887 >> 172.21.15.170:0/595358887 conn(0x7fd69c0fc1d0 msgr2=0x7fd69c0fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:02:40.489 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:40.485+0000 7fd6a1fdf700 1 -- 172.21.15.170:0/595358887 shutdown_connections 2024-03-11T19:02:40.489 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:02:40.485+0000 7fd6a1fdf700 1 -- 172.21.15.170:0/595358887 wait complete. 2024-03-11T19:02:41.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:40 smithi190 ceph-mon[36292]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:41.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:40 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:41.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:40 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:41.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:40 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:41.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:40 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:02:41.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:40 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:41.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:40 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:41.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:40 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:41.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:40 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:02:41.063 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:40 smithi170 ceph-mon[35690]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:41.063 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:40 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:41.063 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:40 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:41.063 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:40 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:41.063 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:40 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:02:41.063 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:40 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:41.063 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:40 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:41.063 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:40 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:41.063 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:40 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:02:41.095 DEBUG:teuthology.orchestra.run.smithi170:osd.1> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.1.service 2024-03-11T19:02:41.098 INFO:tasks.cephadm:Deploying osd.2 on smithi190 with /dev/vg_nvme/lv_4... 2024-03-11T19:02:41.098 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T19:02:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:40 smithi029 ceph-mon[32439]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:02:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:02:41.340 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:02:42.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:41 smithi190 ceph-mon[36292]: from='osd.1 [v2:172.21.15.170:6800/531888562,v1:172.21.15.170:6801/531888562]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T19:02:42.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:41 smithi190 ceph-mon[36292]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T19:02:42.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:41 smithi170 ceph-mon[35690]: from='osd.1 [v2:172.21.15.170:6800/531888562,v1:172.21.15.170:6801/531888562]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T19:02:42.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:41 smithi170 ceph-mon[35690]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T19:02:42.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:41 smithi029 ceph-mon[32439]: from='osd.1 [v2:172.21.15.170:6800/531888562,v1:172.21.15.170:6801/531888562]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T19:02:42.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:41 smithi029 ceph-mon[32439]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T19:02:42.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:42 smithi170 ceph-mon[35690]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:42.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:42 smithi170 ceph-mon[35690]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T19:02:42.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:42 smithi170 ceph-mon[35690]: from='osd.1 [v2:172.21.15.170:6800/531888562,v1:172.21.15.170:6801/531888562]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:02:42.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:42 smithi170 ceph-mon[35690]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T19:02:42.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:42 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:02:42.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:42 smithi170 ceph-mon[35690]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:02:42.846 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:02:42 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-1[40296]: 2024-03-11T19:02:42.751+0000 7f2b32afe700 -1 osd.1 0 waiting for initial osdmap 2024-03-11T19:02:42.846 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:02:42 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-1[40296]: 2024-03-11T19:02:42.756+0000 7f2b2e8f1700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:02:43.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:42 smithi190 ceph-mon[36292]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:43.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:42 smithi190 ceph-mon[36292]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T19:02:43.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:42 smithi190 ceph-mon[36292]: from='osd.1 [v2:172.21.15.170:6800/531888562,v1:172.21.15.170:6801/531888562]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:02:43.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:42 smithi190 ceph-mon[36292]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T19:02:43.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:42 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:02:43.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:42 smithi190 ceph-mon[36292]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:02:43.150 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T19:02:43.151 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T19:02:43.151 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-03-11T19:02:43.151 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-03-11T19:02:43.151 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112751 s, 930 MB/s 2024-03-11T19:02:43.151 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-03-11T19:02:43.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:42 smithi029 ceph-mon[32439]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:43.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:42 smithi029 ceph-mon[32439]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T19:02:43.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:42 smithi029 ceph-mon[32439]: from='osd.1 [v2:172.21.15.170:6800/531888562,v1:172.21.15.170:6801/531888562]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:02:43.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:42 smithi029 ceph-mon[32439]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T19:02:43.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:42 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:02:43.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:42 smithi029 ceph-mon[32439]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:02:43.730 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi190:vg_nvme/lv_4 2024-03-11T19:02:43.909 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:43 smithi170 ceph-mon[35690]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-11T19:02:43.910 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:43 smithi170 ceph-mon[35690]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T19:02:43.910 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:43 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:02:43.910 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:43 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:02:43.968 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:02:44.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:43 smithi190 ceph-mon[36292]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-11T19:02:44.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:43 smithi190 ceph-mon[36292]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T19:02:44.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:43 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:02:44.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:43 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:02:44.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:43 smithi029 ceph-mon[32439]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-11T19:02:44.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:43 smithi029 ceph-mon[32439]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T19:02:44.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:02:44.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:02:44.943 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:44 smithi190 ceph-mon[36292]: purged_snaps scrub starts 2024-03-11T19:02:44.943 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:44 smithi190 ceph-mon[36292]: purged_snaps scrub ok 2024-03-11T19:02:44.943 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:44 smithi190 ceph-mon[36292]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:44.943 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:44 smithi190 ceph-mon[36292]: osd.1 [v2:172.21.15.170:6800/531888562,v1:172.21.15.170:6801/531888562] boot 2024-03-11T19:02:44.943 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:44 smithi190 ceph-mon[36292]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T19:02:44.943 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:02:44.943 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:44 smithi190 ceph-mon[36292]: Detected new or changed devices on smithi170 2024-03-11T19:02:44.943 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:44.943 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:44.943 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:02:44.943 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:44 smithi190 ceph-mon[36292]: Adjusting osd_memory_target on smithi170 to 17088M 2024-03-11T19:02:44.943 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:44.943 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:44.944 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:02:44.944 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:45.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:44 smithi170 ceph-mon[35690]: purged_snaps scrub starts 2024-03-11T19:02:45.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:44 smithi170 ceph-mon[35690]: purged_snaps scrub ok 2024-03-11T19:02:45.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:44 smithi170 ceph-mon[35690]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:45.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:44 smithi170 ceph-mon[35690]: osd.1 [v2:172.21.15.170:6800/531888562,v1:172.21.15.170:6801/531888562] boot 2024-03-11T19:02:45.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:44 smithi170 ceph-mon[35690]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T19:02:45.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:44 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:02:45.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:44 smithi170 ceph-mon[35690]: Detected new or changed devices on smithi170 2024-03-11T19:02:45.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:44 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:45.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:44 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:45.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:44 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:02:45.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:44 smithi170 ceph-mon[35690]: Adjusting osd_memory_target on smithi170 to 17088M 2024-03-11T19:02:45.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:44 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:45.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:44 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:45.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:44 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:02:45.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:44 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:45.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:44 smithi029 ceph-mon[32439]: purged_snaps scrub starts 2024-03-11T19:02:45.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:44 smithi029 ceph-mon[32439]: purged_snaps scrub ok 2024-03-11T19:02:45.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:44 smithi029 ceph-mon[32439]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:02:45.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:44 smithi029 ceph-mon[32439]: osd.1 [v2:172.21.15.170:6800/531888562,v1:172.21.15.170:6801/531888562] boot 2024-03-11T19:02:45.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:44 smithi029 ceph-mon[32439]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T19:02:45.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:02:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:44 smithi029 ceph-mon[32439]: Detected new or changed devices on smithi170 2024-03-11T19:02:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:02:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:44 smithi029 ceph-mon[32439]: Adjusting osd_memory_target on smithi170 to 17088M 2024-03-11T19:02:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:02:45.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:02:45.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.437+0000 7f20aee99700 1 -- 172.21.15.190:0/2983629398 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20a8101da0 msgr2=0x7f20a8102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:02:45.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.437+0000 7f20aee99700 1 --2- 172.21.15.190:0/2983629398 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20a8101da0 0x7f20a8102200 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2094009990 tx=0x7f2094009ca0 comp rx=0 tx=0).stop 2024-03-11T19:02:45.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.437+0000 7f20aee99700 1 -- 172.21.15.190:0/2983629398 shutdown_connections 2024-03-11T19:02:45.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.437+0000 7f20aee99700 1 --2- 172.21.15.190:0/2983629398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20a8102740 0x7f20a8108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:02:45.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.437+0000 7f20aee99700 1 --2- 172.21.15.190:0/2983629398 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20a8100ba0 0x7f20a8100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:02:45.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.437+0000 7f20aee99700 1 --2- 172.21.15.190:0/2983629398 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20a8101da0 0x7f20a8102200 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:02:45.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.437+0000 7f20aee99700 1 -- 172.21.15.190:0/2983629398 >> 172.21.15.190:0/2983629398 conn(0x7f20a80fc160 msgr2=0x7f20a80fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:02:45.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.437+0000 7f20aee99700 1 -- 172.21.15.190:0/2983629398 shutdown_connections 2024-03-11T19:02:45.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.437+0000 7f20aee99700 1 -- 172.21.15.190:0/2983629398 wait complete. 2024-03-11T19:02:45.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.438+0000 7f20aee99700 1 Processor -- start 2024-03-11T19:02:45.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.438+0000 7f20aee99700 1 -- start start 2024-03-11T19:02:45.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.438+0000 7f20aee99700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20a8100ba0 0x7f20a8104ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:02:45.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.438+0000 7f20aee99700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20a8101da0 0x7f20a8103180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:02:45.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.438+0000 7f20aee99700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20a8102740 0x7f20a81036c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:02:45.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.438+0000 7f20aee99700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a8103e60 con 0x7f20a8100ba0 2024-03-11T19:02:45.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.438+0000 7f20aee99700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a8103fd0 con 0x7f20a8101da0 2024-03-11T19:02:45.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.438+0000 7f20aee99700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a81a1cc0 con 0x7f20a8102740 2024-03-11T19:02:45.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.438+0000 7f20ad436700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20a8102740 0x7f20a81036c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:02:45.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.438+0000 7f20ad436700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20a8102740 0x7f20a81036c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:34420/0 (socket says 172.21.15.190:34420) 2024-03-11T19:02:45.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.438+0000 7f20ad436700 1 -- 172.21.15.190:0/2689257686 learned_addr learned my addr 172.21.15.190:0/2689257686 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-11T19:02:45.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.438+0000 7f20acc35700 1 --2- 172.21.15.190:0/2689257686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20a8100ba0 0x7f20a8104ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:02:45.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.439+0000 7f20a7fff700 1 --2- 172.21.15.190:0/2689257686 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20a8101da0 0x7f20a8103180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:02:45.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.439+0000 7f20ad436700 1 -- 172.21.15.190:0/2689257686 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20a8101da0 msgr2=0x7f20a8103180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:02:45.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.439+0000 7f20ad436700 1 --2- 172.21.15.190:0/2689257686 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20a8101da0 0x7f20a8103180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:02:45.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.439+0000 7f20ad436700 1 -- 172.21.15.190:0/2689257686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20a8100ba0 msgr2=0x7f20a8104ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:02:45.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.439+0000 7f20ad436700 1 --2- 172.21.15.190:0/2689257686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20a8100ba0 0x7f20a8104ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:02:45.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.439+0000 7f20ad436700 1 -- 172.21.15.190:0/2689257686 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20940095f0 con 0x7f20a8102740 2024-03-11T19:02:45.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.439+0000 7f20ad436700 1 --2- 172.21.15.190:0/2689257686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20a8102740 0x7f20a81036c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f20a0007ee0 tx=0x7f20a000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:02:45.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.439+0000 7f20a5ffb700 1 -- 172.21.15.190:0/2689257686 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0017070 con 0x7f20a8102740 2024-03-11T19:02:45.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.439+0000 7f20a5ffb700 1 -- 172.21.15.190:0/2689257686 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f20a000f040 con 0x7f20a8102740 2024-03-11T19:02:45.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.439+0000 7f20aee99700 1 -- 172.21.15.190:0/2689257686 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a81a1f50 con 0x7f20a8102740 2024-03-11T19:02:45.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.439+0000 7f20a5ffb700 1 -- 172.21.15.190:0/2689257686 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0005730 con 0x7f20a8102740 2024-03-11T19:02:45.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.439+0000 7f20aee99700 1 -- 172.21.15.190:0/2689257686 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a81a2440 con 0x7f20a8102740 2024-03-11T19:02:45.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.440+0000 7f20aee99700 1 -- 172.21.15.190:0/2689257686 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a8068960 con 0x7f20a8102740 2024-03-11T19:02:45.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.441+0000 7f20a5ffb700 1 -- 172.21.15.190:0/2689257686 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7f20a0005890 con 0x7f20a8102740 2024-03-11T19:02:45.447 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.442+0000 7f20a5ffb700 1 --2- 172.21.15.190:0/2689257686 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f20980a1ab0 0x7f20980a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:02:45.447 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.442+0000 7f20a5ffb700 1 -- 172.21.15.190:0/2689257686 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f20a00cb320 con 0x7f20a8102740 2024-03-11T19:02:45.447 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.442+0000 7f20acc35700 1 --2- 172.21.15.190:0/2689257686 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f20980a1ab0 0x7f20980a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:02:45.447 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.443+0000 7f20acc35700 1 --2- 172.21.15.190:0/2689257686 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f20980a1ab0 0x7f20980a3f70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f20a8101c00 tx=0x7f209c009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:02:45.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.446+0000 7f20a5ffb700 1 -- 172.21.15.190:0/2689257686 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f20a0095d80 con 0x7f20a8102740 2024-03-11T19:02:45.651 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:02:45.648+0000 7f20aee99700 1 -- 172.21.15.190:0/2689257686 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f20a80fe360 con 0x7f20980a1ab0 2024-03-11T19:02:46.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:46 smithi170 ceph-mon[35690]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T19:02:46.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:02:46.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:02:46.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:46.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:46 smithi190 ceph-mon[36292]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T19:02:46.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:46 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:02:46.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:46 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:02:46.374 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:46 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:46.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:46 smithi029 ceph-mon[32439]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T19:02:46.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:46 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:02:46.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:46 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:02:46.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:46 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:47.118 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:47 smithi190 ceph-mon[36292]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:47.118 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:47 smithi190 ceph-mon[36292]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:02:47.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:47 smithi170 ceph-mon[35690]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:47.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:47 smithi170 ceph-mon[35690]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:02:47.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:47 smithi029 ceph-mon[32439]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:47.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:47 smithi029 ceph-mon[32439]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:02:49.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:49 smithi170 ceph-mon[35690]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:49.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:49 smithi170 ceph-mon[35690]: from='client.? 172.21.15.190:0/496599704' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "799a6a3b-63d6-47b8-a88f-c3a08ee03a53"}]: dispatch 2024-03-11T19:02:49.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:49 smithi170 ceph-mon[35690]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "799a6a3b-63d6-47b8-a88f-c3a08ee03a53"}]: dispatch 2024-03-11T19:02:49.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:49 smithi170 ceph-mon[35690]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "799a6a3b-63d6-47b8-a88f-c3a08ee03a53"}]': finished 2024-03-11T19:02:49.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:49 smithi170 ceph-mon[35690]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T19:02:49.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:49 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:02:49.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:49 smithi170 ceph-mon[35690]: from='client.? 172.21.15.190:0/2115795713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:02:49.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:49 smithi029 ceph-mon[32439]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:49.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:49 smithi029 ceph-mon[32439]: from='client.? 172.21.15.190:0/496599704' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "799a6a3b-63d6-47b8-a88f-c3a08ee03a53"}]: dispatch 2024-03-11T19:02:49.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:49 smithi029 ceph-mon[32439]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "799a6a3b-63d6-47b8-a88f-c3a08ee03a53"}]: dispatch 2024-03-11T19:02:49.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:49 smithi029 ceph-mon[32439]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "799a6a3b-63d6-47b8-a88f-c3a08ee03a53"}]': finished 2024-03-11T19:02:49.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:49 smithi029 ceph-mon[32439]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T19:02:49.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:49 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:02:49.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:49 smithi029 ceph-mon[32439]: from='client.? 172.21.15.190:0/2115795713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:02:49.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:49 smithi190 ceph-mon[36292]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:49.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:49 smithi190 ceph-mon[36292]: from='client.? 172.21.15.190:0/496599704' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "799a6a3b-63d6-47b8-a88f-c3a08ee03a53"}]: dispatch 2024-03-11T19:02:49.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:49 smithi190 ceph-mon[36292]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "799a6a3b-63d6-47b8-a88f-c3a08ee03a53"}]: dispatch 2024-03-11T19:02:49.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:49 smithi190 ceph-mon[36292]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "799a6a3b-63d6-47b8-a88f-c3a08ee03a53"}]': finished 2024-03-11T19:02:49.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:49 smithi190 ceph-mon[36292]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T19:02:49.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:49 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:02:49.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:49 smithi190 ceph-mon[36292]: from='client.? 172.21.15.190:0/2115795713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:02:51.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:51 smithi170 ceph-mon[35690]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:51.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:51 smithi029 ceph-mon[32439]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:51.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:51 smithi190 ceph-mon[36292]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:53.329 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:53 smithi190 ceph-mon[36292]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:53.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:53 smithi170 ceph-mon[35690]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:53.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:53 smithi029 ceph-mon[32439]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:55.331 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:55 smithi190 ceph-mon[36292]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:55.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:55 smithi170 ceph-mon[35690]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:55.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:55 smithi029 ceph-mon[32439]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:56.242 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:56 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T19:02:56.242 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:56 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:56.242 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:56 smithi190 ceph-mon[36292]: Deploying daemon osd.2 on smithi190 2024-03-11T19:02:56.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:56 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T19:02:56.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:56 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:56.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:56 smithi170 ceph-mon[35690]: Deploying daemon osd.2 on smithi190 2024-03-11T19:02:56.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:56 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T19:02:56.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:56 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:02:56.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:56 smithi029 ceph-mon[32439]: Deploying daemon osd.2 on smithi190 2024-03-11T19:02:57.182 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:57 smithi190 ceph-mon[36292]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:57.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:57 smithi170 ceph-mon[35690]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:57.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:57 smithi029 ceph-mon[32439]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:59.131 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:02:59 smithi190 ceph-mon[36292]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:59.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:02:59 smithi170 ceph-mon[35690]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:02:59.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:02:59 smithi029 ceph-mon[32439]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:03:01.306 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:01 smithi190 ceph-mon[36292]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:03:01.306 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:01 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:01.306 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:01 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:01.306 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:01 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:03:01.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:01 smithi170 ceph-mon[35690]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:03:01.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:01 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:01.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:01 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:01.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:01 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:03:01.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:01 smithi029 ceph-mon[32439]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:03:01.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:01 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:01.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:01 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:01.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:01 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:03:03.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:03 smithi190 ceph-mon[36292]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:03:03.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:03 smithi170 ceph-mon[35690]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:03:03.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:03 smithi029 ceph-mon[32439]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:03:04.021 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 2 on host 'smithi190' 2024-03-11T19:03:04.021 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:03:04.017+0000 7f20a5ffb700 1 -- 172.21.15.190:0/2689257686 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f20a80fe360 con 0x7f20980a1ab0 2024-03-11T19:03:04.023 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:03:04.020+0000 7f20aee99700 1 -- 172.21.15.190:0/2689257686 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f20980a1ab0 msgr2=0x7f20980a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:03:04.023 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:03:04.020+0000 7f20aee99700 1 --2- 172.21.15.190:0/2689257686 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f20980a1ab0 0x7f20980a3f70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f20a8101c00 tx=0x7f209c009190 comp rx=0 tx=0).stop 2024-03-11T19:03:04.023 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:03:04.020+0000 7f20aee99700 1 -- 172.21.15.190:0/2689257686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20a8102740 msgr2=0x7f20a81036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:03:04.023 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:03:04.020+0000 7f20aee99700 1 --2- 172.21.15.190:0/2689257686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20a8102740 0x7f20a81036c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f20a0007ee0 tx=0x7f20a000d3b0 comp rx=0 tx=0).stop 2024-03-11T19:03:04.023 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:03:04.020+0000 7f20aee99700 1 -- 172.21.15.190:0/2689257686 shutdown_connections 2024-03-11T19:03:04.023 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:03:04.020+0000 7f20aee99700 1 --2- 172.21.15.190:0/2689257686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20a8102740 0x7f20a81036c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:04.023 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:03:04.020+0000 7f20aee99700 1 --2- 172.21.15.190:0/2689257686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20a8100ba0 0x7f20a8104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:04.023 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:03:04.020+0000 7f20aee99700 1 --2- 172.21.15.190:0/2689257686 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f20980a1ab0 0x7f20980a3f70 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:04.023 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:03:04.020+0000 7f20aee99700 1 --2- 172.21.15.190:0/2689257686 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20a8101da0 0x7f20a8103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:04.023 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:03:04.020+0000 7f20aee99700 1 -- 172.21.15.190:0/2689257686 >> 172.21.15.190:0/2689257686 conn(0x7f20a80fc160 msgr2=0x7f20a80fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:03:04.023 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:03:04.020+0000 7f20aee99700 1 -- 172.21.15.190:0/2689257686 shutdown_connections 2024-03-11T19:03:04.024 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:03:04.020+0000 7f20aee99700 1 -- 172.21.15.190:0/2689257686 wait complete. 2024-03-11T19:03:04.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:04 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:04.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:04 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:04.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:04 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:04.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:04 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:03:04.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:04 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:04.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:04 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:04.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:04 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:04.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:04 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:03:04.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:04 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:04.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:04 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:04.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:04 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:04.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:04 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:03:04.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:04 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:04.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:04 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:04.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:04 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:04.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:04 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:03:04.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:04 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:04.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:04 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:04.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:04 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:04.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:04 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:03:04.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:04 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:04.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:04 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:04.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:04 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:04.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:04 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:03:04.627 DEBUG:teuthology.orchestra.run.smithi190:osd.2> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.2.service 2024-03-11T19:03:04.629 INFO:tasks.cephadm:Deploying osd.3 on smithi029 with /dev/vg_nvme/lv_3... 2024-03-11T19:03:04.630 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T19:03:04.880 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:03:05.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:05 smithi190 ceph-mon[36292]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:03:05.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:05 smithi190 ceph-mon[36292]: from='osd.2 [v2:172.21.15.190:6800/1010896180,v1:172.21.15.190:6801/1010896180]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T19:03:05.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:05 smithi190 ceph-mon[36292]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T19:03:05.303 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:05 smithi029 ceph-mon[32439]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:03:05.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:05 smithi029 ceph-mon[32439]: from='osd.2 [v2:172.21.15.190:6800/1010896180,v1:172.21.15.190:6801/1010896180]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T19:03:05.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:05 smithi029 ceph-mon[32439]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T19:03:05.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:05 smithi170 ceph-mon[35690]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:03:05.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:05 smithi170 ceph-mon[35690]: from='osd.2 [v2:172.21.15.190:6800/1010896180,v1:172.21.15.190:6801/1010896180]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T19:03:05.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:05 smithi170 ceph-mon[35690]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T19:03:06.265 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:06 smithi029 ceph-mon[32439]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T19:03:06.265 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:06 smithi029 ceph-mon[32439]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T19:03:06.265 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:06 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:03:06.265 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:06 smithi029 ceph-mon[32439]: from='osd.2 [v2:172.21.15.190:6800/1010896180,v1:172.21.15.190:6801/1010896180]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:03:06.265 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:06 smithi029 ceph-mon[32439]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:03:06.405 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:06 smithi170 ceph-mon[35690]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T19:03:06.406 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:06 smithi170 ceph-mon[35690]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T19:03:06.406 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:06 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:03:06.406 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:06 smithi170 ceph-mon[35690]: from='osd.2 [v2:172.21.15.190:6800/1010896180,v1:172.21.15.190:6801/1010896180]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:03:06.406 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:06 smithi170 ceph-mon[35690]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:03:06.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:06 smithi190 ceph-mon[36292]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T19:03:06.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:06 smithi190 ceph-mon[36292]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T19:03:06.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:06 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:03:06.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:06 smithi190 ceph-mon[36292]: from='osd.2 [v2:172.21.15.190:6800/1010896180,v1:172.21.15.190:6801/1010896180]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:03:06.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:06 smithi190 ceph-mon[36292]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:03:06.841 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T19:03:06.841 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T19:03:06.841 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-03-11T19:03:06.841 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-03-11T19:03:06.841 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00817705 s, 1.3 GB/s 2024-03-11T19:03:06.841 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-03-11T19:03:06.889 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:03:06 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-2[41403]: 2024-03-11T19:03:06.622+0000 7f7fa2214700 -1 osd.2 0 waiting for initial osdmap 2024-03-11T19:03:06.889 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:03:06 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-2[41403]: 2024-03-11T19:03:06.627+0000 7f7f997fe700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:03:07.219 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:07 smithi190 ceph-mon[36292]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:03:07.219 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:07 smithi190 ceph-mon[36292]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-11T19:03:07.219 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:07 smithi190 ceph-mon[36292]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T19:03:07.219 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:03:07.219 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:03:07.220 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:07 smithi190 ceph-mon[36292]: from='osd.2 ' entity='osd.2' 2024-03-11T19:03:07.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:07 smithi029 ceph-mon[32439]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:03:07.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:07 smithi029 ceph-mon[32439]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-11T19:03:07.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:07 smithi029 ceph-mon[32439]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T19:03:07.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:03:07.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:03:07.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:07 smithi029 ceph-mon[32439]: from='osd.2 ' entity='osd.2' 2024-03-11T19:03:07.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_3 2024-03-11T19:03:07.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:07 smithi170 ceph-mon[35690]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:03:07.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:07 smithi170 ceph-mon[35690]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-11T19:03:07.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:07 smithi170 ceph-mon[35690]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T19:03:07.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:03:07.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:03:07.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:07 smithi170 ceph-mon[35690]: from='osd.2 ' entity='osd.2' 2024-03-11T19:03:07.737 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:03:08.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:08 smithi029 ceph-mon[32439]: purged_snaps scrub starts 2024-03-11T19:03:08.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:08 smithi029 ceph-mon[32439]: purged_snaps scrub ok 2024-03-11T19:03:08.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:03:08.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:08 smithi029 ceph-mon[32439]: osd.2 [v2:172.21.15.190:6800/1010896180,v1:172.21.15.190:6801/1010896180] boot 2024-03-11T19:03:08.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:08 smithi029 ceph-mon[32439]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T19:03:08.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:03:08.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:08.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:08.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:03:08.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:08.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:08.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:03:08.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:08.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:08 smithi190 ceph-mon[36292]: purged_snaps scrub starts 2024-03-11T19:03:08.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:08 smithi190 ceph-mon[36292]: purged_snaps scrub ok 2024-03-11T19:03:08.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:03:08.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:08 smithi190 ceph-mon[36292]: osd.2 [v2:172.21.15.190:6800/1010896180,v1:172.21.15.190:6801/1010896180] boot 2024-03-11T19:03:08.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:08 smithi190 ceph-mon[36292]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T19:03:08.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:03:08.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:08.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:08.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:03:08.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:08.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:08.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:03:08.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:08.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:08 smithi170 ceph-mon[35690]: purged_snaps scrub starts 2024-03-11T19:03:08.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:08 smithi170 ceph-mon[35690]: purged_snaps scrub ok 2024-03-11T19:03:08.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:03:08.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:08 smithi170 ceph-mon[35690]: osd.2 [v2:172.21.15.190:6800/1010896180,v1:172.21.15.190:6801/1010896180] boot 2024-03-11T19:03:08.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:08 smithi170 ceph-mon[35690]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T19:03:08.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:03:08.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:08.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:08.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:03:08.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:08.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:08.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:03:08.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:09.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:09 smithi029 ceph-mon[32439]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:03:09.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:09 smithi029 ceph-mon[32439]: Detected new or changed devices on smithi190 2024-03-11T19:03:09.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:09 smithi029 ceph-mon[32439]: Adjusting osd_memory_target on smithi190 to 17088M 2024-03-11T19:03:09.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.443+0000 7fbaf1b78700 1 -- 172.21.15.29:0/4067483058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaec1017d0 msgr2=0x7fbaec109070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:03:09.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.443+0000 7fbaf1b78700 1 --2- 172.21.15.29:0/4067483058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaec1017d0 0x7fbaec109070 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbadc009990 tx=0x7fbadc009ca0 comp rx=0 tx=0).stop 2024-03-11T19:03:09.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.443+0000 7fbaf1b78700 1 -- 172.21.15.29:0/4067483058 shutdown_connections 2024-03-11T19:03:09.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.443+0000 7fbaf1b78700 1 --2- 172.21.15.29:0/4067483058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaec1017d0 0x7fbaec109070 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:09.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.443+0000 7fbaf1b78700 1 --2- 172.21.15.29:0/4067483058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaec1021b0 0x7fbaec1025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:09.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.443+0000 7fbaf1b78700 1 --2- 172.21.15.29:0/4067483058 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbaec100e10 0x7fbaec101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:09.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.443+0000 7fbaf1b78700 1 -- 172.21.15.29:0/4067483058 >> 172.21.15.29:0/4067483058 conn(0x7fbaec0fb8e0 msgr2=0x7fbaec0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:03:09.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.443+0000 7fbaf1b78700 1 -- 172.21.15.29:0/4067483058 shutdown_connections 2024-03-11T19:03:09.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.443+0000 7fbaf1b78700 1 -- 172.21.15.29:0/4067483058 wait complete. 2024-03-11T19:03:09.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.443+0000 7fbaf1b78700 1 Processor -- start 2024-03-11T19:03:09.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.444+0000 7fbaf1b78700 1 -- start start 2024-03-11T19:03:09.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.444+0000 7fbaf1b78700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaec100e10 0x7fbaec197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:03:09.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.444+0000 7fbaf1b78700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbaec1017d0 0x7fbaec198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:03:09.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.444+0000 7fbaf1b78700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaec1021b0 0x7fbaec19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:03:09.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.444+0000 7fbaf1b78700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec19f6a0 con 0x7fbaec100e10 2024-03-11T19:03:09.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.444+0000 7fbaf1b78700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec19f7e0 con 0x7fbaec1017d0 2024-03-11T19:03:09.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.444+0000 7fbaf1b78700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec19f970 con 0x7fbaec1021b0 2024-03-11T19:03:09.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.444+0000 7fbaeb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaec100e10 0x7fbaec197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:03:09.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.444+0000 7fbaeb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaec100e10 0x7fbaec197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54332/0 (socket says 172.21.15.29:54332) 2024-03-11T19:03:09.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.444+0000 7fbaeb7fe700 1 -- 172.21.15.29:0/1112582183 learned_addr learned my addr 172.21.15.29:0/1112582183 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:03:09.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.444+0000 7fbaebfff700 1 --2- 172.21.15.29:0/1112582183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaec1021b0 0x7fbaec19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:03:09.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.444+0000 7fbaeaffd700 1 --2- 172.21.15.29:0/1112582183 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbaec1017d0 0x7fbaec198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:03:09.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.444+0000 7fbaeb7fe700 1 -- 172.21.15.29:0/1112582183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaec1021b0 msgr2=0x7fbaec19f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:03:09.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.444+0000 7fbaeb7fe700 1 --2- 172.21.15.29:0/1112582183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaec1021b0 0x7fbaec19f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:09.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.444+0000 7fbaeb7fe700 1 -- 172.21.15.29:0/1112582183 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbaec1017d0 msgr2=0x7fbaec198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:03:09.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.444+0000 7fbaeb7fe700 1 --2- 172.21.15.29:0/1112582183 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbaec1017d0 0x7fbaec198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:09.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.444+0000 7fbaeb7fe700 1 -- 172.21.15.29:0/1112582183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbadc0095f0 con 0x7fbaec100e10 2024-03-11T19:03:09.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.444+0000 7fbaebfff700 1 --2- 172.21.15.29:0/1112582183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaec1021b0 0x7fbaec19f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:03:09.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.445+0000 7fbaeb7fe700 1 --2- 172.21.15.29:0/1112582183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaec100e10 0x7fbaec197b10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbae400b9f0 tx=0x7fbae400bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:03:09.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.445+0000 7fbae8ff9700 1 -- 172.21.15.29:0/1112582183 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae400ca30 con 0x7fbaec100e10 2024-03-11T19:03:09.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.445+0000 7fbae8ff9700 1 -- 172.21.15.29:0/1112582183 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbae400cb90 con 0x7fbaec100e10 2024-03-11T19:03:09.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.445+0000 7fbaf1b78700 1 -- 172.21.15.29:0/1112582183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaec19fc50 con 0x7fbaec100e10 2024-03-11T19:03:09.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.445+0000 7fbae8ff9700 1 -- 172.21.15.29:0/1112582183 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae40125a0 con 0x7fbaec100e10 2024-03-11T19:03:09.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.445+0000 7fbaf1b78700 1 -- 172.21.15.29:0/1112582183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaec1a01a0 con 0x7fbaec100e10 2024-03-11T19:03:09.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.446+0000 7fbaf1b78700 1 -- 172.21.15.29:0/1112582183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaec068930 con 0x7fbaec100e10 2024-03-11T19:03:09.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.447+0000 7fbae8ff9700 1 -- 172.21.15.29:0/1112582183 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7fbae400cd00 con 0x7fbaec100e10 2024-03-11T19:03:09.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.447+0000 7fbae8ff9700 1 --2- 172.21.15.29:0/1112582183 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbad80a1b30 0x7fbad80a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:03:09.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.448+0000 7fbaeaffd700 1 --2- 172.21.15.29:0/1112582183 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbad80a1b30 0x7fbad80a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:03:09.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.448+0000 7fbae8ff9700 1 -- 172.21.15.29:0/1112582183 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 2851+0+0 (secure 0 0 0) 0x7fbae4006e40 con 0x7fbaec100e10 2024-03-11T19:03:09.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.448+0000 7fbaeaffd700 1 --2- 172.21.15.29:0/1112582183 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbad80a1b30 0x7fbad80a3ff0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbad4005ea0 tx=0x7fbad4005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:03:09.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.451+0000 7fbae8ff9700 1 -- 172.21.15.29:0/1112582183 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbae4096610 con 0x7fbaec100e10 2024-03-11T19:03:09.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:09 smithi190 ceph-mon[36292]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:03:09.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:09 smithi190 ceph-mon[36292]: Detected new or changed devices on smithi190 2024-03-11T19:03:09.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:09 smithi190 ceph-mon[36292]: Adjusting osd_memory_target on smithi190 to 17088M 2024-03-11T19:03:09.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:09 smithi170 ceph-mon[35690]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:03:09.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:09 smithi170 ceph-mon[35690]: Detected new or changed devices on smithi190 2024-03-11T19:03:09.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:09 smithi170 ceph-mon[35690]: Adjusting osd_memory_target on smithi190 to 17088M 2024-03-11T19:03:09.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:09.665+0000 7fbaf1b78700 1 -- 172.21.15.29:0/1112582183 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fbaec0fd9e0 con 0x7fbad80a1b30 2024-03-11T19:03:10.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:10 smithi029 ceph-mon[32439]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T19:03:10.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:10 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T19:03:10.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:10 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:03:10.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:10 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:03:10.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:10 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:10.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:10 smithi190 ceph-mon[36292]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T19:03:10.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:10 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T19:03:10.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:10 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:03:10.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:10 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:03:10.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:10 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:10.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:10 smithi170 ceph-mon[35690]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T19:03:10.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:10 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T19:03:10.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:10 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:03:10.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:10 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:03:10.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:10 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:11.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:11 smithi029 ceph-mon[32439]: pgmap v82: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:11.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:11 smithi029 ceph-mon[32439]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:03:11.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:11 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T19:03:11.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:11 smithi029 ceph-mon[32439]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T19:03:11.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:11 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T19:03:11.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:11 smithi190 ceph-mon[36292]: pgmap v82: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:11.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:11 smithi190 ceph-mon[36292]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:03:11.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:11 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T19:03:11.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:11 smithi190 ceph-mon[36292]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T19:03:11.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:11 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T19:03:11.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:11 smithi170 ceph-mon[35690]: pgmap v82: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:11.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:11 smithi170 ceph-mon[35690]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:03:11.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:11 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T19:03:11.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:11 smithi170 ceph-mon[35690]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T19:03:11.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:11 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T19:03:11.769 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:03:11 smithi029 sudo[51624]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T19:03:11.769 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:03:11 smithi029 sudo[51624]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:03:11.769 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:03:11 smithi029 sudo[51624]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:03:11.769 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:03:11 smithi029 sudo[51624]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:03:12.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:03:11 smithi029 sudo[51645]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T19:03:12.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:03:11 smithi029 sudo[51645]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:03:12.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:03:11 smithi029 sudo[51645]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:03:12.156 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:03:11 smithi029 sudo[51645]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:03:12.318 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:03:12 smithi170 sudo[44420]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T19:03:12.318 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:03:12 smithi170 sudo[44420]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:03:12.318 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:03:12 smithi170 sudo[44420]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:03:12.318 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:03:12 smithi170 sudo[44420]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:03:12.319 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:12 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T19:03:12.319 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:12 smithi170 ceph-mon[35690]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T19:03:12.319 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:12 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:12.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:12 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T19:03:12.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:12 smithi190 ceph-mon[36292]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T19:03:12.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:12 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:12.595 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:03:12 smithi170 sudo[44423]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T19:03:12.595 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:03:12 smithi170 sudo[44423]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:03:12.595 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:03:12 smithi170 sudo[44423]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:03:12.595 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:03:12 smithi170 sudo[44423]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:03:12.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:12 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T19:03:12.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:12 smithi029 ceph-mon[32439]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T19:03:12.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:12 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:12.841 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:03:12 smithi190 sudo[45020]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T19:03:12.841 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:03:12 smithi190 sudo[45020]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:03:12.841 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:03:12 smithi190 sudo[45020]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:03:12.841 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:03:12 smithi190 sudo[45020]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:03:13.261 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:03:12 smithi190 sudo[45023]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T19:03:13.261 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:03:12 smithi190 sudo[45023]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:03:13.261 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:03:12 smithi190 sudo[45023]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:03:13.261 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:03:12 smithi190 sudo[45023]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:03:13.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:13 smithi029 sudo[51719]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T19:03:13.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:13 smithi029 sudo[51719]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:03:13.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:13 smithi029 sudo[51719]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:03:13.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:13 smithi029 sudo[51719]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:03:13.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:13.397+0000 7fbae8ff9700 1 -- 172.21.15.29:0/1112582183 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fbae40969e0 con 0x7fbaec100e10 2024-03-11T19:03:13.649 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:13 smithi190 ceph-mon[36292]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:13.649 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:13 smithi190 ceph-mon[36292]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T19:03:13.649 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:13 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:13.649 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:13 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3007855255' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e"}]: dispatch 2024-03-11T19:03:13.649 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:13 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3007855255' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e"}]': finished 2024-03-11T19:03:13.649 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:13 smithi190 ceph-mon[36292]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T19:03:13.649 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:13 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:03:13.649 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:13 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:13.649 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:13 smithi190 ceph-mon[36292]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T19:03:13.650 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:13 smithi190 sudo[45026]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T19:03:13.650 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:13 smithi190 sudo[45026]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:03:13.650 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:13 smithi190 sudo[45026]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:03:13.650 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:13 smithi190 sudo[45026]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:03:13.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:13 smithi029 sudo[51722]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T19:03:13.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:13 smithi029 sudo[51722]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:03:13.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:13 smithi029 sudo[51722]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:03:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:13 smithi029 sudo[51722]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:03:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:13 smithi029 ceph-mon[32439]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:13 smithi029 ceph-mon[32439]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T19:03:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:13 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:13 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3007855255' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e"}]: dispatch 2024-03-11T19:03:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:13 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3007855255' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e"}]': finished 2024-03-11T19:03:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:13 smithi029 ceph-mon[32439]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T19:03:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:13 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:03:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:13 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:13 smithi029 ceph-mon[32439]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T19:03:13.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:13 smithi170 ceph-mon[35690]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:13.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:13 smithi170 ceph-mon[35690]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T19:03:13.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:13 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:13.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:13 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3007855255' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e"}]: dispatch 2024-03-11T19:03:13.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:13 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3007855255' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e"}]': finished 2024-03-11T19:03:13.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:13 smithi170 ceph-mon[35690]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T19:03:13.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:13 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:03:13.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:13 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:13.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:13 smithi170 ceph-mon[35690]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T19:03:14.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:13 smithi190 sudo[45029]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T19:03:14.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:13 smithi190 sudo[45029]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:03:14.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:13 smithi190 sudo[45029]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:03:14.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:13 smithi190 sudo[45029]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:03:14.378 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 sudo[44426]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T19:03:14.379 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 sudo[44426]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:03:14.379 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 sudo[44426]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:03:14.379 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 sudo[44426]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:03:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:14 smithi029 ceph-mon[32439]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T19:03:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:03:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:03:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:03:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:14 smithi029 ceph-mon[32439]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T19:03:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:14 smithi029 ceph-mon[32439]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-11T19:03:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:03:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:03:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:03:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:14 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1430548844' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:03:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:14 smithi029 ceph-mon[32439]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T19:03:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:14 smithi029 ceph-mon[32439]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T19:03:14.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:03:14.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:03:14.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:03:14.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:14 smithi029 ceph-mon[32439]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T19:03:14.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:14 smithi190 ceph-mon[36292]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T19:03:14.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:14 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:03:14.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:14 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:03:14.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:14 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:03:14.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:14 smithi190 ceph-mon[36292]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T19:03:14.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:14 smithi190 ceph-mon[36292]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-11T19:03:14.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:14 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:03:14.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:14 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:03:14.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:14 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:03:14.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:14 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1430548844' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:03:14.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:14 smithi190 ceph-mon[36292]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T19:03:14.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:14 smithi190 ceph-mon[36292]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T19:03:14.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:14 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:03:14.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:14 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:03:14.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:14 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:03:14.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:14 smithi190 ceph-mon[36292]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T19:03:14.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 sudo[44429]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T19:03:14.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 sudo[44429]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:03:14.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 sudo[44429]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:03:14.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 sudo[44429]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:03:14.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 ceph-mon[35690]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T19:03:14.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:03:14.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:03:14.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:03:14.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 ceph-mon[35690]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T19:03:14.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 ceph-mon[35690]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-11T19:03:14.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:03:14.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:03:14.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:03:14.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1430548844' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:03:14.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 ceph-mon[35690]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T19:03:14.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 ceph-mon[35690]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T19:03:14.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:03:14.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:03:14.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:03:14.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:14 smithi170 ceph-mon[35690]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T19:03:15.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:15 smithi029 ceph-mon[32439]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:15.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:15 smithi190 ceph-mon[36292]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:15.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:15 smithi170 ceph-mon[35690]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:17.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:17 smithi029 ceph-mon[32439]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:17.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:17 smithi190 ceph-mon[36292]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:17.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:17 smithi170 ceph-mon[35690]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:19.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:19 smithi029 ceph-mon[32439]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:19.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:19 smithi190 ceph-mon[36292]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:19.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:19 smithi170 ceph-mon[35690]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:21.491 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:21 smithi029 ceph-mon[32439]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:21.492 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:21 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T19:03:21.492 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:21 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:21.492 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:21 smithi029 ceph-mon[32439]: Deploying daemon osd.3 on smithi029 2024-03-11T19:03:21.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:21 smithi190 ceph-mon[36292]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:21.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:21 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T19:03:21.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:21 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:21.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:21 smithi190 ceph-mon[36292]: Deploying daemon osd.3 on smithi029 2024-03-11T19:03:21.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:21 smithi170 ceph-mon[35690]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:21.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:21 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T19:03:21.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:21 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:21.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:21 smithi170 ceph-mon[35690]: Deploying daemon osd.3 on smithi029 2024-03-11T19:03:23.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:23 smithi190 ceph-mon[36292]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:23.818 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:23 smithi029 ceph-mon[32439]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:23.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:23 smithi170 ceph-mon[35690]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:25.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:25 smithi029 ceph-mon[32439]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:25.662 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:25 smithi170 ceph-mon[35690]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:25.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:25 smithi190 ceph-mon[36292]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:27.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:27 smithi170 ceph-mon[35690]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:27.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:27 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:27.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:27 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:27.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:27 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:03:27.622 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:27 smithi029 ceph-mon[32439]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:27.622 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:27 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:27.623 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:27 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:27.623 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:27 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:03:27.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:27 smithi190 ceph-mon[36292]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:27.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:27 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:27.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:27 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:27.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:27 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:03:29.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:29 smithi170 ceph-mon[35690]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:29.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:29.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:29.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:29.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:03:29.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:29.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:29 smithi029 ceph-mon[32439]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:29.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:29.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:29.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:29.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:03:29.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:29.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:29 smithi190 ceph-mon[36292]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:03:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:30.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:30.133+0000 7fbae8ff9700 1 -- 172.21.15.29:0/1112582183 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbaec0fd9e0 con 0x7fbad80a1b30 2024-03-11T19:03:30.137 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 3 on host 'smithi029' 2024-03-11T19:03:30.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:30.135+0000 7fbaf1b78700 1 -- 172.21.15.29:0/1112582183 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbad80a1b30 msgr2=0x7fbad80a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:03:30.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:30.135+0000 7fbaf1b78700 1 --2- 172.21.15.29:0/1112582183 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbad80a1b30 0x7fbad80a3ff0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbad4005ea0 tx=0x7fbad4005e30 comp rx=0 tx=0).stop 2024-03-11T19:03:30.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:30.135+0000 7fbaf1b78700 1 -- 172.21.15.29:0/1112582183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaec100e10 msgr2=0x7fbaec197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:03:30.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:30.135+0000 7fbaf1b78700 1 --2- 172.21.15.29:0/1112582183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaec100e10 0x7fbaec197b10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbae400b9f0 tx=0x7fbae400bdb0 comp rx=0 tx=0).stop 2024-03-11T19:03:30.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:30.136+0000 7fbaf1b78700 1 -- 172.21.15.29:0/1112582183 shutdown_connections 2024-03-11T19:03:30.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:30.136+0000 7fbaf1b78700 1 --2- 172.21.15.29:0/1112582183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaec1021b0 0x7fbaec19f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:30.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:30.136+0000 7fbaf1b78700 1 --2- 172.21.15.29:0/1112582183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaec100e10 0x7fbaec197b10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:30.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:30.136+0000 7fbaf1b78700 1 --2- 172.21.15.29:0/1112582183 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbad80a1b30 0x7fbad80a3ff0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:30.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:30.136+0000 7fbaf1b78700 1 --2- 172.21.15.29:0/1112582183 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbaec1017d0 0x7fbaec198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:30.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:30.136+0000 7fbaf1b78700 1 -- 172.21.15.29:0/1112582183 >> 172.21.15.29:0/1112582183 conn(0x7fbaec0fb8e0 msgr2=0x7fbaec0fd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:03:30.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:30.136+0000 7fbaf1b78700 1 -- 172.21.15.29:0/1112582183 shutdown_connections 2024-03-11T19:03:30.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:03:30.136+0000 7fbaf1b78700 1 -- 172.21.15.29:0/1112582183 wait complete. 2024-03-11T19:03:30.970 DEBUG:teuthology.orchestra.run.smithi029:osd.3> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.3.service 2024-03-11T19:03:30.972 INFO:tasks.cephadm:Deploying osd.4 on smithi170 with /dev/vg_nvme/lv_3... 2024-03-11T19:03:30.972 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T19:03:31.201 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:31 smithi029 ceph-mon[32439]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:31.202 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:31 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:31.202 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:31 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:31.202 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:31 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:03:31.202 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:31 smithi029 ceph-mon[32439]: from='osd.3 [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T19:03:31.217 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:03:31.339 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:31 smithi170 ceph-mon[35690]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:31.339 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:31 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:31.339 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:31 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:31.339 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:31 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:03:31.339 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:31 smithi170 ceph-mon[35690]: from='osd.3 [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T19:03:31.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:31 smithi190 ceph-mon[36292]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:31.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:31 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:31.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:31 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:31.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:31 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:03:31.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:31 smithi190 ceph-mon[36292]: from='osd.3 [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T19:03:32.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:32 smithi170 ceph-mon[35690]: from='osd.3 [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T19:03:32.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:32 smithi170 ceph-mon[35690]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T19:03:32.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:32 smithi170 ceph-mon[35690]: from='osd.3 [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-11T19:03:32.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:32 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:03:32.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:32 smithi029 ceph-mon[32439]: from='osd.3 [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T19:03:32.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:32 smithi029 ceph-mon[32439]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T19:03:32.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:32 smithi029 ceph-mon[32439]: from='osd.3 [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-11T19:03:32.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:32 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:03:32.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:32 smithi190 ceph-mon[36292]: from='osd.3 [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T19:03:32.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:32 smithi190 ceph-mon[36292]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T19:03:32.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:32 smithi190 ceph-mon[36292]: from='osd.3 [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-11T19:03:32.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:32 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:03:32.655 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:03:32 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-3[53739]: 2024-03-11T19:03:32.598+0000 7f94cb6a2700 -1 osd.3 0 waiting for initial osdmap 2024-03-11T19:03:32.656 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:03:32 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-3[53739]: 2024-03-11T19:03:32.603+0000 7f94c348d700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:03:33.042 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T19:03:33.042 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T19:03:33.042 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-03-11T19:03:33.042 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-03-11T19:03:33.042 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00840773 s, 1.2 GB/s 2024-03-11T19:03:33.043 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-03-11T19:03:33.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:33 smithi029 ceph-mon[32439]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:33.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:33 smithi029 ceph-mon[32439]: from='osd.3 [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-11T19:03:33.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:33 smithi029 ceph-mon[32439]: osdmap e25: 4 total, 3 up, 4 in 2024-03-11T19:03:33.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:33 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:03:33.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:33 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:03:33.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:33 smithi029 ceph-mon[32439]: from='osd.3 [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055]' entity='osd.3' 2024-03-11T19:03:33.416 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:33 smithi170 ceph-mon[35690]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:33.416 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:33 smithi170 ceph-mon[35690]: from='osd.3 [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-11T19:03:33.416 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:33 smithi170 ceph-mon[35690]: osdmap e25: 4 total, 3 up, 4 in 2024-03-11T19:03:33.416 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:33 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:03:33.416 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:33 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:03:33.416 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:33 smithi170 ceph-mon[35690]: from='osd.3 [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055]' entity='osd.3' 2024-03-11T19:03:33.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:33 smithi190 ceph-mon[36292]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:03:33.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:33 smithi190 ceph-mon[36292]: from='osd.3 [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-11T19:03:33.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:33 smithi190 ceph-mon[36292]: osdmap e25: 4 total, 3 up, 4 in 2024-03-11T19:03:33.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:33 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:03:33.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:33 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:03:33.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:33 smithi190 ceph-mon[36292]: from='osd.3 [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055]' entity='osd.3' 2024-03-11T19:03:33.671 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi170:vg_nvme/lv_3 2024-03-11T19:03:33.906 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:03:34.315 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:34 smithi170 ceph-mon[35690]: purged_snaps scrub starts 2024-03-11T19:03:34.315 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:34 smithi170 ceph-mon[35690]: purged_snaps scrub ok 2024-03-11T19:03:34.315 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:34 smithi170 ceph-mon[35690]: osd.3 [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055] boot 2024-03-11T19:03:34.315 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:34 smithi170 ceph-mon[35690]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T19:03:34.315 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:34 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:03:34.316 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:34 smithi029 ceph-mon[32439]: purged_snaps scrub starts 2024-03-11T19:03:34.316 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:34 smithi029 ceph-mon[32439]: purged_snaps scrub ok 2024-03-11T19:03:34.316 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:34 smithi029 ceph-mon[32439]: osd.3 [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055] boot 2024-03-11T19:03:34.316 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:34 smithi029 ceph-mon[32439]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T19:03:34.316 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:34 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:03:34.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:34 smithi190 ceph-mon[36292]: purged_snaps scrub starts 2024-03-11T19:03:34.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:34 smithi190 ceph-mon[36292]: purged_snaps scrub ok 2024-03-11T19:03:34.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:34 smithi190 ceph-mon[36292]: osd.3 [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055] boot 2024-03-11T19:03:34.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:34 smithi190 ceph-mon[36292]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T19:03:34.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:34 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:03:35.255 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:35 smithi170 ceph-mon[35690]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:35.255 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:35 smithi170 ceph-mon[35690]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T19:03:35.255 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:35 smithi170 ceph-mon[35690]: Detected new or changed devices on smithi029 2024-03-11T19:03:35.256 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:35 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:35.256 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:35 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:35.256 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:35 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:03:35.256 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:35 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:03:35.256 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:35 smithi170 ceph-mon[35690]: Adjusting osd_memory_target on smithi029 to 8546M 2024-03-11T19:03:35.256 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:35 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:35.256 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:35 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:35.256 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:35 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:03:35.256 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:35 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:35.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:35 smithi029 ceph-mon[32439]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:35.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:35 smithi029 ceph-mon[32439]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T19:03:35.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:35 smithi029 ceph-mon[32439]: Detected new or changed devices on smithi029 2024-03-11T19:03:35.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:35.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:35.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:03:35.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:03:35.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:35 smithi029 ceph-mon[32439]: Adjusting osd_memory_target on smithi029 to 8546M 2024-03-11T19:03:35.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:35.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:35.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:03:35.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:35.459 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.454+0000 7f1d15462700 1 -- 172.21.15.170:0/2687699436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d10101ca0 msgr2=0x7f1d101020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:03:35.459 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.454+0000 7f1d15462700 1 --2- 172.21.15.170:0/2687699436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d10101ca0 0x7f1d101020c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1d040099c0 tx=0x7f1d04009cd0 comp rx=0 tx=0).stop 2024-03-11T19:03:35.459 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.454+0000 7f1d15462700 1 -- 172.21.15.170:0/2687699436 shutdown_connections 2024-03-11T19:03:35.459 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.454+0000 7f1d15462700 1 --2- 172.21.15.170:0/2687699436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d10103840 0x7f1d1010a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:35.459 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.454+0000 7f1d15462700 1 --2- 172.21.15.170:0/2687699436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d10101ca0 0x7f1d101020c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:35.460 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.454+0000 7f1d15462700 1 --2- 172.21.15.170:0/2687699436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d10102ea0 0x7f1d10103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:35.460 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.454+0000 7f1d15462700 1 -- 172.21.15.170:0/2687699436 >> 172.21.15.170:0/2687699436 conn(0x7f1d100fd280 msgr2=0x7f1d100ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:03:35.460 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.454+0000 7f1d15462700 1 -- 172.21.15.170:0/2687699436 shutdown_connections 2024-03-11T19:03:35.460 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.454+0000 7f1d15462700 1 -- 172.21.15.170:0/2687699436 wait complete. 2024-03-11T19:03:35.460 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.455+0000 7f1d15462700 1 Processor -- start 2024-03-11T19:03:35.460 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.455+0000 7f1d15462700 1 -- start start 2024-03-11T19:03:35.460 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.455+0000 7f1d15462700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d10101ca0 0x7f1d10197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:03:35.460 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.455+0000 7f1d15462700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d10102ea0 0x7f1d101980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:03:35.460 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.455+0000 7f1d15462700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d10103840 0x7f1d1019f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:03:35.460 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.455+0000 7f1d15462700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1019f700 con 0x7f1d10103840 2024-03-11T19:03:35.460 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.455+0000 7f1d15462700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1019f840 con 0x7f1d10101ca0 2024-03-11T19:03:35.460 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.455+0000 7f1d15462700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1019f9d0 con 0x7f1d10102ea0 2024-03-11T19:03:35.460 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.455+0000 7f1d0effd700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d10101ca0 0x7f1d10197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:03:35.460 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.455+0000 7f1d0effd700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d10101ca0 0x7f1d10197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:57734/0 (socket says 172.21.15.170:57734) 2024-03-11T19:03:35.461 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.455+0000 7f1d0effd700 1 -- 172.21.15.170:0/75963207 learned_addr learned my addr 172.21.15.170:0/75963207 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-11T19:03:35.461 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.455+0000 7f1d0e7fc700 1 --2- 172.21.15.170:0/75963207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d10102ea0 0x7f1d101980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:03:35.461 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.455+0000 7f1d0f7fe700 1 --2- 172.21.15.170:0/75963207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d10103840 0x7f1d1019f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:03:35.461 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.455+0000 7f1d0effd700 1 -- 172.21.15.170:0/75963207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d10102ea0 msgr2=0x7f1d101980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:03:35.461 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.455+0000 7f1d0effd700 1 --2- 172.21.15.170:0/75963207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d10102ea0 0x7f1d101980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:35.461 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.455+0000 7f1d0effd700 1 -- 172.21.15.170:0/75963207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d10103840 msgr2=0x7f1d1019f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:03:35.461 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.455+0000 7f1d0effd700 1 --2- 172.21.15.170:0/75963207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d10103840 0x7f1d1019f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:35.461 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.455+0000 7f1d0effd700 1 -- 172.21.15.170:0/75963207 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d040095f0 con 0x7f1d10101ca0 2024-03-11T19:03:35.461 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.456+0000 7f1d0f7fe700 1 --2- 172.21.15.170:0/75963207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d10103840 0x7f1d1019f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:03:35.461 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.456+0000 7f1d0e7fc700 1 --2- 172.21.15.170:0/75963207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d10102ea0 0x7f1d101980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:03:35.461 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.456+0000 7f1d0effd700 1 --2- 172.21.15.170:0/75963207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d10101ca0 0x7f1d10197b70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1d04009f50 tx=0x7f1d040057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:03:35.461 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.456+0000 7f1cf7fff700 1 -- 172.21.15.170:0/75963207 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0401d070 con 0x7f1d10101ca0 2024-03-11T19:03:35.461 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.456+0000 7f1cf7fff700 1 -- 172.21.15.170:0/75963207 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1d040041c0 con 0x7f1d10101ca0 2024-03-11T19:03:35.461 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.456+0000 7f1d15462700 1 -- 172.21.15.170:0/75963207 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d1019fc50 con 0x7f1d10101ca0 2024-03-11T19:03:35.462 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.456+0000 7f1cf7fff700 1 -- 172.21.15.170:0/75963207 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0400f460 con 0x7f1d10101ca0 2024-03-11T19:03:35.462 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.456+0000 7f1d15462700 1 -- 172.21.15.170:0/75963207 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d101a0060 con 0x7f1d10101ca0 2024-03-11T19:03:35.463 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.457+0000 7f1d15462700 1 -- 172.21.15.170:0/75963207 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d101090b0 con 0x7f1d10101ca0 2024-03-11T19:03:35.463 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.458+0000 7f1cf7fff700 1 -- 172.21.15.170:0/75963207 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1d04022470 con 0x7f1d10101ca0 2024-03-11T19:03:35.463 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.458+0000 7f1cf7fff700 1 --2- 172.21.15.170:0/75963207 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1cf80a1ac0 0x7f1cf80a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:03:35.463 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.459+0000 7f1cf7fff700 1 -- 172.21.15.170:0/75963207 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 3698+0+0 (secure 0 0 0) 0x7f1d040cd730 con 0x7f1d10101ca0 2024-03-11T19:03:35.463 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.459+0000 7f1d0e7fc700 1 --2- 172.21.15.170:0/75963207 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1cf80a1ac0 0x7f1cf80a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:03:35.463 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.459+0000 7f1d0e7fc700 1 --2- 172.21.15.170:0/75963207 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1cf80a1ac0 0x7f1cf80a3f80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1d00005fd0 tx=0x7f1d00005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:03:35.466 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.463+0000 7f1cf7fff700 1 -- 172.21.15.170:0/75963207 <== mon.2 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1d04097bd0 con 0x7f1d10101ca0 2024-03-11T19:03:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:35 smithi190 ceph-mon[36292]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:35 smithi190 ceph-mon[36292]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T19:03:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:35 smithi190 ceph-mon[36292]: Detected new or changed devices on smithi029 2024-03-11T19:03:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:03:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:03:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:35 smithi190 ceph-mon[36292]: Adjusting osd_memory_target on smithi029 to 8546M 2024-03-11T19:03:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:03:35.513 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:35.670 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:35.667+0000 7f1d15462700 1 -- 172.21.15.170:0/75963207 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d1002d910 con 0x7f1cf80a1ac0 2024-03-11T19:03:36.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:36 smithi029 ceph-mon[32439]: osdmap e28: 4 total, 4 up, 4 in 2024-03-11T19:03:36.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:36 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:03:36.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:36 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:03:36.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:36 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:36.433 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:36 smithi170 ceph-mon[35690]: osdmap e28: 4 total, 4 up, 4 in 2024-03-11T19:03:36.433 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:36 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:03:36.433 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:36 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:03:36.433 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:36 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:36.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:36 smithi190 ceph-mon[36292]: osdmap e28: 4 total, 4 up, 4 in 2024-03-11T19:03:36.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:36 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:03:36.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:36 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:03:36.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:36 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:37.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:37 smithi170 ceph-mon[35690]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 134 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:37.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:37 smithi170 ceph-mon[35690]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:03:37.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:37 smithi170 ceph-mon[35690]: osdmap e29: 4 total, 4 up, 4 in 2024-03-11T19:03:37.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:37 smithi029 ceph-mon[32439]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 134 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:37.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:37 smithi029 ceph-mon[32439]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:03:37.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:37 smithi029 ceph-mon[32439]: osdmap e29: 4 total, 4 up, 4 in 2024-03-11T19:03:37.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:37 smithi190 ceph-mon[36292]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 134 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:37.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:37 smithi190 ceph-mon[36292]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:03:37.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:37 smithi190 ceph-mon[36292]: osdmap e29: 4 total, 4 up, 4 in 2024-03-11T19:03:38.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:38 smithi029 ceph-mon[32439]: osdmap e30: 4 total, 4 up, 4 in 2024-03-11T19:03:38.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:38 smithi190 ceph-mon[36292]: osdmap e30: 4 total, 4 up, 4 in 2024-03-11T19:03:38.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:38 smithi170 ceph-mon[35690]: osdmap e30: 4 total, 4 up, 4 in 2024-03-11T19:03:39.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:39 smithi190 ceph-mon[36292]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 134 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:39.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:39 smithi190 ceph-mon[36292]: from='client.? 172.21.15.170:0/2367931316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8719e9e6-6389-4554-931e-722f213781b2"}]: dispatch 2024-03-11T19:03:39.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:39 smithi190 ceph-mon[36292]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8719e9e6-6389-4554-931e-722f213781b2"}]: dispatch 2024-03-11T19:03:39.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:39 smithi190 ceph-mon[36292]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8719e9e6-6389-4554-931e-722f213781b2"}]': finished 2024-03-11T19:03:39.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:39 smithi190 ceph-mon[36292]: osdmap e31: 5 total, 4 up, 5 in 2024-03-11T19:03:39.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:39 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:03:39.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:39 smithi190 ceph-mon[36292]: from='client.? 172.21.15.170:0/1448717450' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:03:39.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:39 smithi170 ceph-mon[35690]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 134 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:39.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:39 smithi170 ceph-mon[35690]: from='client.? 172.21.15.170:0/2367931316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8719e9e6-6389-4554-931e-722f213781b2"}]: dispatch 2024-03-11T19:03:39.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:39 smithi170 ceph-mon[35690]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8719e9e6-6389-4554-931e-722f213781b2"}]: dispatch 2024-03-11T19:03:39.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:39 smithi170 ceph-mon[35690]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8719e9e6-6389-4554-931e-722f213781b2"}]': finished 2024-03-11T19:03:39.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:39 smithi170 ceph-mon[35690]: osdmap e31: 5 total, 4 up, 5 in 2024-03-11T19:03:39.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:39 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:03:39.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:39 smithi170 ceph-mon[35690]: from='client.? 172.21.15.170:0/1448717450' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:03:39.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:39 smithi029 ceph-mon[32439]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 134 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:39.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:39 smithi029 ceph-mon[32439]: from='client.? 172.21.15.170:0/2367931316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8719e9e6-6389-4554-931e-722f213781b2"}]: dispatch 2024-03-11T19:03:39.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:39 smithi029 ceph-mon[32439]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8719e9e6-6389-4554-931e-722f213781b2"}]: dispatch 2024-03-11T19:03:39.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:39 smithi029 ceph-mon[32439]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8719e9e6-6389-4554-931e-722f213781b2"}]': finished 2024-03-11T19:03:39.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:39 smithi029 ceph-mon[32439]: osdmap e31: 5 total, 4 up, 5 in 2024-03-11T19:03:39.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:39 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:03:39.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:39 smithi029 ceph-mon[32439]: from='client.? 172.21.15.170:0/1448717450' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:03:41.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:41 smithi190 ceph-mon[36292]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:41.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:41 smithi170 ceph-mon[35690]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:41.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:41 smithi029 ceph-mon[32439]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:43.424 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:43 smithi170 ceph-mon[35690]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:03:43.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:43 smithi190 ceph-mon[36292]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:03:43.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:43 smithi029 ceph-mon[32439]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:03:45.460 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:45 smithi170 ceph-mon[35690]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-11T19:03:45.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:45 smithi190 ceph-mon[36292]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-11T19:03:45.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:45 smithi029 ceph-mon[32439]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-11T19:03:46.421 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T19:03:46.421 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:46.421 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:46 smithi170 ceph-mon[35690]: Deploying daemon osd.4 on smithi170 2024-03-11T19:03:46.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:46 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T19:03:46.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:46 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:46.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:46 smithi190 ceph-mon[36292]: Deploying daemon osd.4 on smithi170 2024-03-11T19:03:46.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:46 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T19:03:46.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:46 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:46.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:46 smithi029 ceph-mon[32439]: Deploying daemon osd.4 on smithi170 2024-03-11T19:03:47.351 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:47 smithi170 ceph-mon[35690]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-11T19:03:47.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:47 smithi190 ceph-mon[36292]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-11T19:03:47.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:47 smithi029 ceph-mon[32439]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-11T19:03:49.437 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:49 smithi170 ceph-mon[35690]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T19:03:49.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:49 smithi190 ceph-mon[36292]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T19:03:49.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:49 smithi029 ceph-mon[32439]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T19:03:51.506 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:51 smithi170 ceph-mon[35690]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-11T19:03:51.506 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:51 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:51.506 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:51 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:51.506 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:51 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:03:51.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:51 smithi190 ceph-mon[36292]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-11T19:03:51.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:51 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:51.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:51 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:51.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:51 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:03:51.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:51 smithi029 ceph-mon[32439]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-11T19:03:51.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:51 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:51.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:51 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:51.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:51 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:03:53.458 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:53 smithi029 ceph-mon[32439]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-11T19:03:53.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:53 smithi190 ceph-mon[36292]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-11T19:03:53.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:53 smithi170 ceph-mon[35690]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-11T19:03:54.793 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:54.789+0000 7f1cf7fff700 1 -- 172.21.15.170:0/75963207 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1d1002d910 con 0x7f1cf80a1ac0 2024-03-11T19:03:54.793 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 4 on host 'smithi170' 2024-03-11T19:03:54.795 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:54.792+0000 7f1d15462700 1 -- 172.21.15.170:0/75963207 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1cf80a1ac0 msgr2=0x7f1cf80a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:03:54.795 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:54.792+0000 7f1d15462700 1 --2- 172.21.15.170:0/75963207 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1cf80a1ac0 0x7f1cf80a3f80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1d00005fd0 tx=0x7f1d00005e30 comp rx=0 tx=0).stop 2024-03-11T19:03:54.795 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:54.792+0000 7f1d15462700 1 -- 172.21.15.170:0/75963207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d10101ca0 msgr2=0x7f1d10197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:03:54.795 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:54.792+0000 7f1d15462700 1 --2- 172.21.15.170:0/75963207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d10101ca0 0x7f1d10197b70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1d04009f50 tx=0x7f1d040057f0 comp rx=0 tx=0).stop 2024-03-11T19:03:54.795 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:54.792+0000 7f1d15462700 1 -- 172.21.15.170:0/75963207 shutdown_connections 2024-03-11T19:03:54.795 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:54.792+0000 7f1d15462700 1 --2- 172.21.15.170:0/75963207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d10103840 0x7f1d1019f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:54.795 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:54.792+0000 7f1d15462700 1 --2- 172.21.15.170:0/75963207 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1cf80a1ac0 0x7f1cf80a3f80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:54.795 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:54.792+0000 7f1d15462700 1 --2- 172.21.15.170:0/75963207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d10101ca0 0x7f1d10197b70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:54.795 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:54.792+0000 7f1d15462700 1 --2- 172.21.15.170:0/75963207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d10102ea0 0x7f1d101980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:03:54.795 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:54.792+0000 7f1d15462700 1 -- 172.21.15.170:0/75963207 >> 172.21.15.170:0/75963207 conn(0x7f1d100fd280 msgr2=0x7f1d101061f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:03:54.795 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:54.792+0000 7f1d15462700 1 -- 172.21.15.170:0/75963207 shutdown_connections 2024-03-11T19:03:54.796 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:03:54.792+0000 7f1d15462700 1 -- 172.21.15.170:0/75963207 wait complete. 2024-03-11T19:03:55.006 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:54 smithi170 ceph-mon[35690]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:55.006 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:54 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:55.006 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:54 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:55.006 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:54 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:55.007 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:54 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:03:55.007 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:54 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:55.007 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:54 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:55.007 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:54 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:55.007 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:54 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:03:55.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:54 smithi029 ceph-mon[32439]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:55.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:54 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:55.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:54 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:55.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:54 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:55.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:54 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:03:55.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:54 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:55.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:54 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:55.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:54 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:55.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:54 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:03:55.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:54 smithi190 ceph-mon[36292]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:55.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:54 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:55.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:54 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:55.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:54 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:55.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:54 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:03:55.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:54 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:55.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:54 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:55.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:54 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:55.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:54 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:03:55.477 DEBUG:teuthology.orchestra.run.smithi170:osd.4> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.4.service 2024-03-11T19:03:55.479 INFO:tasks.cephadm:Deploying osd.5 on smithi190 with /dev/vg_nvme/lv_3... 2024-03-11T19:03:55.479 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T19:03:55.723 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:03:56.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:55 smithi170 ceph-mon[35690]: from='osd.4 [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T19:03:56.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:55 smithi029 ceph-mon[32439]: from='osd.4 [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T19:03:56.178 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:55 smithi190 ceph-mon[36292]: from='osd.4 [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T19:03:57.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:56 smithi190 ceph-mon[36292]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:57.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:56 smithi190 ceph-mon[36292]: from='osd.4 [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T19:03:57.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:56 smithi190 ceph-mon[36292]: osdmap e32: 5 total, 4 up, 5 in 2024-03-11T19:03:57.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:56 smithi190 ceph-mon[36292]: from='osd.4 [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:03:57.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:56 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:03:57.163 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:56 smithi170 ceph-mon[35690]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:57.164 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:56 smithi170 ceph-mon[35690]: from='osd.4 [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T19:03:57.164 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:56 smithi170 ceph-mon[35690]: osdmap e32: 5 total, 4 up, 5 in 2024-03-11T19:03:57.164 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:56 smithi170 ceph-mon[35690]: from='osd.4 [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:03:57.164 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:56 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:03:57.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:56 smithi029 ceph-mon[32439]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:57.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:56 smithi029 ceph-mon[32439]: from='osd.4 [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T19:03:57.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:56 smithi029 ceph-mon[32439]: osdmap e32: 5 total, 4 up, 5 in 2024-03-11T19:03:57.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:56 smithi029 ceph-mon[32439]: from='osd.4 [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:03:57.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:56 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:03:57.671 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T19:03:57.671 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T19:03:57.671 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-03-11T19:03:57.672 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-03-11T19:03:57.672 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012988 s, 807 MB/s 2024-03-11T19:03:57.672 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-03-11T19:03:57.829 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 11 19:03:57 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-4[47265]: 2024-03-11T19:03:57.426+0000 7f6595aa8700 -1 osd.4 0 waiting for initial osdmap 2024-03-11T19:03:57.829 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 11 19:03:57 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-4[47265]: 2024-03-11T19:03:57.434+0000 7f658d893700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:03:58.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:57 smithi170 ceph-mon[35690]: from='osd.4 [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226]' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-11T19:03:58.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:57 smithi170 ceph-mon[35690]: osdmap e33: 5 total, 4 up, 5 in 2024-03-11T19:03:58.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:57 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:03:58.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:57 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:03:58.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:57 smithi170 ceph-mon[35690]: from='osd.4 [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226]' entity='osd.4' 2024-03-11T19:03:58.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:57 smithi190 ceph-mon[36292]: from='osd.4 [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226]' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-11T19:03:58.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:57 smithi190 ceph-mon[36292]: osdmap e33: 5 total, 4 up, 5 in 2024-03-11T19:03:58.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:57 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:03:58.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:57 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:03:58.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:57 smithi190 ceph-mon[36292]: from='osd.4 [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226]' entity='osd.4' 2024-03-11T19:03:58.298 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi190:vg_nvme/lv_3 2024-03-11T19:03:58.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:57 smithi029 ceph-mon[32439]: from='osd.4 [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226]' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-11T19:03:58.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:57 smithi029 ceph-mon[32439]: osdmap e33: 5 total, 4 up, 5 in 2024-03-11T19:03:58.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:57 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:03:58.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:57 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:03:58.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:57 smithi029 ceph-mon[32439]: from='osd.4 [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226]' entity='osd.4' 2024-03-11T19:03:58.544 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:03:59.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:58 smithi190 ceph-mon[36292]: purged_snaps scrub starts 2024-03-11T19:03:59.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:58 smithi190 ceph-mon[36292]: purged_snaps scrub ok 2024-03-11T19:03:59.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:58 smithi190 ceph-mon[36292]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:59.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:58 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:03:59.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:58 smithi190 ceph-mon[36292]: osd.4 [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226] boot 2024-03-11T19:03:59.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:58 smithi190 ceph-mon[36292]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T19:03:59.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:58 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:03:59.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:58 smithi190 ceph-mon[36292]: Detected new or changed devices on smithi170 2024-03-11T19:03:59.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:58 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:59.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:58 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:59.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:58 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:03:59.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:58 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:03:59.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:58 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:59.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:58 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:59.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:58 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:03:59.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:58 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:59.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:58 smithi170 ceph-mon[35690]: purged_snaps scrub starts 2024-03-11T19:03:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:58 smithi170 ceph-mon[35690]: purged_snaps scrub ok 2024-03-11T19:03:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:58 smithi170 ceph-mon[35690]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:58 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:03:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:58 smithi170 ceph-mon[35690]: osd.4 [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226] boot 2024-03-11T19:03:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:58 smithi170 ceph-mon[35690]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T19:03:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:58 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:03:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:58 smithi170 ceph-mon[35690]: Detected new or changed devices on smithi170 2024-03-11T19:03:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:58 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:58 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:58 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:03:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:58 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:03:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:58 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:58 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:59.347 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:58 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:03:59.347 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:58 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:59.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:58 smithi029 ceph-mon[32439]: purged_snaps scrub starts 2024-03-11T19:03:59.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:58 smithi029 ceph-mon[32439]: purged_snaps scrub ok 2024-03-11T19:03:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:58 smithi029 ceph-mon[32439]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:03:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:58 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:03:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:58 smithi029 ceph-mon[32439]: osd.4 [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226] boot 2024-03-11T19:03:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:58 smithi029 ceph-mon[32439]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T19:03:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:58 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:03:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:58 smithi029 ceph-mon[32439]: Detected new or changed devices on smithi170 2024-03-11T19:03:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:58 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:58 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:58 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:03:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:58 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:03:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:58 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:03:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:58 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:03:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:58 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:03:59.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:58 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:00.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:03:59 smithi190 ceph-mon[36292]: Adjusting osd_memory_target on smithi170 to 8544M 2024-03-11T19:04:00.267 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.261+0000 7f9f63051700 1 -- 172.21.15.190:0/890247797 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f5c101da0 msgr2=0x7f9f5c102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:04:00.267 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.261+0000 7f9f63051700 1 --2- 172.21.15.190:0/890247797 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f5c101da0 0x7f9f5c102200 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c009990 tx=0x7f9f4c009ca0 comp rx=0 tx=0).stop 2024-03-11T19:04:00.267 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.261+0000 7f9f63051700 1 -- 172.21.15.190:0/890247797 shutdown_connections 2024-03-11T19:04:00.267 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.261+0000 7f9f63051700 1 --2- 172.21.15.190:0/890247797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5c102740 0x7f9f5c108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:00.268 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.261+0000 7f9f63051700 1 --2- 172.21.15.190:0/890247797 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f5c100ba0 0x7f9f5c100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:00.268 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.261+0000 7f9f63051700 1 --2- 172.21.15.190:0/890247797 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f5c101da0 0x7f9f5c102200 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:00.268 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.261+0000 7f9f63051700 1 -- 172.21.15.190:0/890247797 >> 172.21.15.190:0/890247797 conn(0x7f9f5c0fc140 msgr2=0x7f9f5c0fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:04:00.268 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.261+0000 7f9f63051700 1 -- 172.21.15.190:0/890247797 shutdown_connections 2024-03-11T19:04:00.268 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.261+0000 7f9f63051700 1 -- 172.21.15.190:0/890247797 wait complete. 2024-03-11T19:04:00.268 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.262+0000 7f9f63051700 1 Processor -- start 2024-03-11T19:04:00.268 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.262+0000 7f9f63051700 1 -- start start 2024-03-11T19:04:00.268 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.262+0000 7f9f63051700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5c100ba0 0x7f9f5c196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:04:00.268 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.262+0000 7f9f63051700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f5c101da0 0x7f9f5c197070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:04:00.268 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.262+0000 7f9f63051700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f5c102740 0x7f9f5c19e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:04:00.268 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.262+0000 7f9f63051700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c19e6c0 con 0x7f9f5c100ba0 2024-03-11T19:04:00.268 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.262+0000 7f9f63051700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c19e800 con 0x7f9f5c102740 2024-03-11T19:04:00.269 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.262+0000 7f9f63051700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c19e990 con 0x7f9f5c101da0 2024-03-11T19:04:00.269 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.262+0000 7f9f53fff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f5c101da0 0x7f9f5c197070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:04:00.269 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.262+0000 7f9f53fff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f5c101da0 0x7f9f5c197070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:57832/0 (socket says 172.21.15.190:57832) 2024-03-11T19:04:00.269 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.262+0000 7f9f53fff700 1 -- 172.21.15.190:0/1996194102 learned_addr learned my addr 172.21.15.190:0/1996194102 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-11T19:04:00.269 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.262+0000 7f9f615ee700 1 --2- 172.21.15.190:0/1996194102 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f5c102740 0x7f9f5c19e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:04:00.269 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.262+0000 7f9f53fff700 1 -- 172.21.15.190:0/1996194102 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f5c102740 msgr2=0x7f9f5c19e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:04:00.269 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.262+0000 7f9f53fff700 1 --2- 172.21.15.190:0/1996194102 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f5c102740 0x7f9f5c19e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:00.269 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.262+0000 7f9f53fff700 1 -- 172.21.15.190:0/1996194102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5c100ba0 msgr2=0x7f9f5c196b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:04:00.269 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.262+0000 7f9f60ded700 1 --2- 172.21.15.190:0/1996194102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5c100ba0 0x7f9f5c196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:04:00.269 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.262+0000 7f9f53fff700 1 --2- 172.21.15.190:0/1996194102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5c100ba0 0x7f9f5c196b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:00.269 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.262+0000 7f9f53fff700 1 -- 172.21.15.190:0/1996194102 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4c0095f0 con 0x7f9f5c101da0 2024-03-11T19:04:00.269 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.263+0000 7f9f53fff700 1 --2- 172.21.15.190:0/1996194102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f5c101da0 0x7f9f5c197070 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c0098e0 tx=0x7f9f4c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:04:00.269 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.263+0000 7f9f51ffb700 1 -- 172.21.15.190:0/1996194102 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c01d070 con 0x7f9f5c101da0 2024-03-11T19:04:00.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.263+0000 7f9f51ffb700 1 -- 172.21.15.190:0/1996194102 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9f4c004090 con 0x7f9f5c101da0 2024-03-11T19:04:00.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.263+0000 7f9f63051700 1 -- 172.21.15.190:0/1996194102 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f5c19ec10 con 0x7f9f5c101da0 2024-03-11T19:04:00.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.263+0000 7f9f51ffb700 1 -- 172.21.15.190:0/1996194102 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c022e60 con 0x7f9f5c101da0 2024-03-11T19:04:00.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.263+0000 7f9f63051700 1 -- 172.21.15.190:0/1996194102 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f5c19f080 con 0x7f9f5c101da0 2024-03-11T19:04:00.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.264+0000 7f9f63051700 1 -- 172.21.15.190:0/1996194102 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5c068930 con 0x7f9f5c101da0 2024-03-11T19:04:00.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.265+0000 7f9f51ffb700 1 -- 172.21.15.190:0/1996194102 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9f4c003be0 con 0x7f9f5c101da0 2024-03-11T19:04:00.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.266+0000 7f9f51ffb700 1 --2- 172.21.15.190:0/1996194102 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9f440a1b40 0x7f9f440a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:04:00.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.266+0000 7f9f51ffb700 1 -- 172.21.15.190:0/1996194102 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f9f4c0cd6b0 con 0x7f9f5c101da0 2024-03-11T19:04:00.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.266+0000 7f9f60ded700 1 --2- 172.21.15.190:0/1996194102 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9f440a1b40 0x7f9f440a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:04:00.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.267+0000 7f9f60ded700 1 --2- 172.21.15.190:0/1996194102 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9f440a1b40 0x7f9f440a4000 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c101c00 tx=0x7f9f58008640 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:04:00.273 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.271+0000 7f9f51ffb700 1 -- 172.21.15.190:0/1996194102 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9f4c05dd20 con 0x7f9f5c101da0 2024-03-11T19:04:00.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:03:59 smithi170 ceph-mon[35690]: Adjusting osd_memory_target on smithi170 to 8544M 2024-03-11T19:04:00.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:03:59 smithi029 ceph-mon[32439]: Adjusting osd_memory_target on smithi170 to 8544M 2024-03-11T19:04:00.476 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:00.473+0000 7f9f63051700 1 -- 172.21.15.190:0/1996194102 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f5c02d910 con 0x7f9f440a1b40 2024-03-11T19:04:01.187 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:00 smithi190 ceph-mon[36292]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:01.187 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:00 smithi190 ceph-mon[36292]: osdmap e35: 5 total, 5 up, 5 in 2024-03-11T19:04:01.187 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:00 smithi190 ceph-mon[36292]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:04:01.187 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:04:01.187 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:04:01.187 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:01.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:00 smithi170 ceph-mon[35690]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:01.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:00 smithi170 ceph-mon[35690]: osdmap e35: 5 total, 5 up, 5 in 2024-03-11T19:04:01.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:00 smithi170 ceph-mon[35690]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:04:01.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:04:01.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:04:01.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:01.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:00 smithi029 ceph-mon[32439]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:01.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:00 smithi029 ceph-mon[32439]: osdmap e35: 5 total, 5 up, 5 in 2024-03-11T19:04:01.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:00 smithi029 ceph-mon[32439]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:04:01.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:04:01.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:04:01.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:03.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:02 smithi190 ceph-mon[36292]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:03.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:02 smithi170 ceph-mon[35690]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:03.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:02 smithi029 ceph-mon[32439]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:04.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:03 smithi190 ceph-mon[36292]: from='client.? 172.21.15.190:0/4287598111' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8072585-c243-4a27-8a7a-b1af399aa467"}]: dispatch 2024-03-11T19:04:04.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:03 smithi190 ceph-mon[36292]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8072585-c243-4a27-8a7a-b1af399aa467"}]: dispatch 2024-03-11T19:04:04.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:03 smithi190 ceph-mon[36292]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8072585-c243-4a27-8a7a-b1af399aa467"}]': finished 2024-03-11T19:04:04.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:03 smithi190 ceph-mon[36292]: osdmap e36: 6 total, 5 up, 6 in 2024-03-11T19:04:04.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:03 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:04.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:03 smithi190 ceph-mon[36292]: from='client.? 172.21.15.190:0/3567276220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:04:04.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:03 smithi170 ceph-mon[35690]: from='client.? 172.21.15.190:0/4287598111' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8072585-c243-4a27-8a7a-b1af399aa467"}]: dispatch 2024-03-11T19:04:04.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:03 smithi170 ceph-mon[35690]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8072585-c243-4a27-8a7a-b1af399aa467"}]: dispatch 2024-03-11T19:04:04.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:03 smithi170 ceph-mon[35690]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8072585-c243-4a27-8a7a-b1af399aa467"}]': finished 2024-03-11T19:04:04.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:03 smithi170 ceph-mon[35690]: osdmap e36: 6 total, 5 up, 6 in 2024-03-11T19:04:04.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:03 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:04.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:03 smithi170 ceph-mon[35690]: from='client.? 172.21.15.190:0/3567276220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:04:04.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:03 smithi029 ceph-mon[32439]: from='client.? 172.21.15.190:0/4287598111' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8072585-c243-4a27-8a7a-b1af399aa467"}]: dispatch 2024-03-11T19:04:04.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:03 smithi029 ceph-mon[32439]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8072585-c243-4a27-8a7a-b1af399aa467"}]: dispatch 2024-03-11T19:04:04.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:03 smithi029 ceph-mon[32439]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8072585-c243-4a27-8a7a-b1af399aa467"}]': finished 2024-03-11T19:04:04.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:03 smithi029 ceph-mon[32439]: osdmap e36: 6 total, 5 up, 6 in 2024-03-11T19:04:04.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:03 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:04.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:03 smithi029 ceph-mon[32439]: from='client.? 172.21.15.190:0/3567276220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:04:05.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:05 smithi170 ceph-mon[35690]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:05.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:05 smithi029 ceph-mon[32439]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:05.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:05 smithi190 ceph-mon[36292]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:07.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:07 smithi170 ceph-mon[35690]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:07.348 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:07 smithi190 ceph-mon[36292]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:07.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:07 smithi029 ceph-mon[32439]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:09.260 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:09 smithi190 ceph-mon[36292]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:09.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:09 smithi170 ceph-mon[35690]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:09.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:09 smithi029 ceph-mon[32439]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:11.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:11 smithi190 ceph-mon[36292]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:11.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:11 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T19:04:11.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:11 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:11.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:11 smithi190 ceph-mon[36292]: Deploying daemon osd.5 on smithi190 2024-03-11T19:04:11.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:11 smithi170 ceph-mon[35690]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:11.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:11 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T19:04:11.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:11 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:11.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:11 smithi170 ceph-mon[35690]: Deploying daemon osd.5 on smithi190 2024-03-11T19:04:11.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:11 smithi029 ceph-mon[32439]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:11.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:11 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T19:04:11.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:11 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:11.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:11 smithi029 ceph-mon[32439]: Deploying daemon osd.5 on smithi190 2024-03-11T19:04:13.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:13 smithi170 ceph-mon[35690]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:13.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:13 smithi029 ceph-mon[32439]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:13.433 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:13 smithi190 ceph-mon[36292]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:15.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:15 smithi190 ceph-mon[36292]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:15.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:15 smithi170 ceph-mon[35690]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:15.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:15 smithi029 ceph-mon[32439]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:17.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:16 smithi190 ceph-mon[36292]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:17.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:17.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:17.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:04:17.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:16 smithi170 ceph-mon[35690]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:17.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:17.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:17.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:04:17.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:16 smithi029 ceph-mon[32439]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:17.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:17.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:17.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:04:19.546 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:19 smithi190 ceph-mon[36292]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:19.546 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:19 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:19.547 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:19 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:19.547 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:19 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:19.547 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:19 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:04:19.547 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:19 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:19.596 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:19.593+0000 7f9f51ffb700 1 -- 172.21.15.190:0/1996194102 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9f5c02d910 con 0x7f9f440a1b40 2024-03-11T19:04:19.596 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 5 on host 'smithi190' 2024-03-11T19:04:19.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:19.595+0000 7f9f63051700 1 -- 172.21.15.190:0/1996194102 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9f440a1b40 msgr2=0x7f9f440a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:04:19.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:19.595+0000 7f9f63051700 1 --2- 172.21.15.190:0/1996194102 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9f440a1b40 0x7f9f440a4000 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c101c00 tx=0x7f9f58008640 comp rx=0 tx=0).stop 2024-03-11T19:04:19.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:19.595+0000 7f9f63051700 1 -- 172.21.15.190:0/1996194102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f5c101da0 msgr2=0x7f9f5c197070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:04:19.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:19.595+0000 7f9f63051700 1 --2- 172.21.15.190:0/1996194102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f5c101da0 0x7f9f5c197070 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c0098e0 tx=0x7f9f4c004730 comp rx=0 tx=0).stop 2024-03-11T19:04:19.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:19.595+0000 7f9f63051700 1 -- 172.21.15.190:0/1996194102 shutdown_connections 2024-03-11T19:04:19.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:19.595+0000 7f9f63051700 1 --2- 172.21.15.190:0/1996194102 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9f440a1b40 0x7f9f440a4000 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:19.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:19.595+0000 7f9f63051700 1 --2- 172.21.15.190:0/1996194102 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f5c102740 0x7f9f5c19e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:19.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:19.595+0000 7f9f63051700 1 --2- 172.21.15.190:0/1996194102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5c100ba0 0x7f9f5c196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:19.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:19.595+0000 7f9f63051700 1 --2- 172.21.15.190:0/1996194102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f5c101da0 0x7f9f5c197070 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:19.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:19.595+0000 7f9f63051700 1 -- 172.21.15.190:0/1996194102 >> 172.21.15.190:0/1996194102 conn(0x7f9f5c0fc140 msgr2=0x7f9f5c1051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:04:19.599 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:19.595+0000 7f9f63051700 1 -- 172.21.15.190:0/1996194102 shutdown_connections 2024-03-11T19:04:19.599 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:04:19.595+0000 7f9f63051700 1 -- 172.21.15.190:0/1996194102 wait complete. 2024-03-11T19:04:19.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:19 smithi170 ceph-mon[35690]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:19.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:19 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:19.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:19 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:19.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:19 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:19.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:19 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:04:19.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:19 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:19.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:19 smithi029 ceph-mon[32439]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:19.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:19 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:19.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:19 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:19.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:19 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:19.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:19 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:04:19.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:19 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:20.234 DEBUG:teuthology.orchestra.run.smithi190:osd.5> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.5.service 2024-03-11T19:04:20.236 INFO:tasks.cephadm:Deploying osd.6 on smithi029 with /dev/vg_nvme/lv_2... 2024-03-11T19:04:20.236 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T19:04:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:04:20.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:20 smithi170 ceph-mon[35690]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:20.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:20.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:20.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:04:20.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:20 smithi170 ceph-mon[35690]: from='osd.5 [v2:172.21.15.190:6808/318560672,v1:172.21.15.190:6809/318560672]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T19:04:20.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:20 smithi170 ceph-mon[35690]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T19:04:20.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:20 smithi029 ceph-mon[32439]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:20.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:20.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:20.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:04:20.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:20 smithi029 ceph-mon[32439]: from='osd.5 [v2:172.21.15.190:6808/318560672,v1:172.21.15.190:6809/318560672]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T19:04:20.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:20 smithi029 ceph-mon[32439]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T19:04:20.981 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:20 smithi190 ceph-mon[36292]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:20.981 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:20.981 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:20.982 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:04:20.982 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:20 smithi190 ceph-mon[36292]: from='osd.5 [v2:172.21.15.190:6808/318560672,v1:172.21.15.190:6809/318560672]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T19:04:20.982 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:20 smithi190 ceph-mon[36292]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T19:04:21.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:21 smithi170 ceph-mon[35690]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T19:04:21.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:21 smithi170 ceph-mon[35690]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T19:04:21.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:21 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:21.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:21 smithi170 ceph-mon[35690]: from='osd.5 [v2:172.21.15.190:6808/318560672,v1:172.21.15.190:6809/318560672]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:04:21.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:21 smithi170 ceph-mon[35690]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:04:21.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:21 smithi029 ceph-mon[32439]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T19:04:21.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:21 smithi029 ceph-mon[32439]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T19:04:21.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:21 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:21.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:21 smithi029 ceph-mon[32439]: from='osd.5 [v2:172.21.15.190:6808/318560672,v1:172.21.15.190:6809/318560672]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:04:21.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:21 smithi029 ceph-mon[32439]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:04:22.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:21 smithi190 ceph-mon[36292]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T19:04:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:21 smithi190 ceph-mon[36292]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T19:04:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:21 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:21 smithi190 ceph-mon[36292]: from='osd.5 [v2:172.21.15.190:6808/318560672,v1:172.21.15.190:6809/318560672]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:04:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:21 smithi190 ceph-mon[36292]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:04:22.427 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:04:22 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:04:22.051+0000 7fd2e3b3f700 -1 osd.5 0 waiting for initial osdmap 2024-03-11T19:04:22.427 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:04:22 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:04:22.059+0000 7fd2db92a700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:04:22.523 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T19:04:22.523 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T19:04:22.523 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-03-11T19:04:22.523 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-03-11T19:04:22.523 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00864617 s, 1.2 GB/s 2024-03-11T19:04:22.523 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-03-11T19:04:22.687 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:22 smithi029 ceph-mon[32439]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:22.687 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:22 smithi029 ceph-mon[32439]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-11T19:04:22.687 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:22 smithi029 ceph-mon[32439]: osdmap e38: 6 total, 5 up, 6 in 2024-03-11T19:04:22.687 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:22 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:22.687 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:22 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:22.687 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:22 smithi029 ceph-mon[32439]: osdmap e39: 6 total, 5 up, 6 in 2024-03-11T19:04:22.688 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:22 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:22.688 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:22 smithi029 ceph-mon[32439]: from='osd.5 ' entity='osd.5' 2024-03-11T19:04:22.758 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:22 smithi190 ceph-mon[36292]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:22.758 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:22 smithi190 ceph-mon[36292]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-11T19:04:22.758 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:22 smithi190 ceph-mon[36292]: osdmap e38: 6 total, 5 up, 6 in 2024-03-11T19:04:22.758 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:22 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:22.758 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:22 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:22.758 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:22 smithi190 ceph-mon[36292]: osdmap e39: 6 total, 5 up, 6 in 2024-03-11T19:04:22.758 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:22 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:22.759 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:22 smithi190 ceph-mon[36292]: from='osd.5 ' entity='osd.5' 2024-03-11T19:04:22.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:22 smithi170 ceph-mon[35690]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:22.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:22 smithi170 ceph-mon[35690]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-11T19:04:22.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:22 smithi170 ceph-mon[35690]: osdmap e38: 6 total, 5 up, 6 in 2024-03-11T19:04:22.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:22 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:22.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:22 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:22.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:22 smithi170 ceph-mon[35690]: osdmap e39: 6 total, 5 up, 6 in 2024-03-11T19:04:22.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:22 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:22.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:22 smithi170 ceph-mon[35690]: from='osd.5 ' entity='osd.5' 2024-03-11T19:04:23.253 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_2 2024-03-11T19:04:23.498 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:04:23.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:23 smithi170 ceph-mon[35690]: purged_snaps scrub starts 2024-03-11T19:04:23.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:23 smithi170 ceph-mon[35690]: purged_snaps scrub ok 2024-03-11T19:04:23.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:23 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:23.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:23 smithi170 ceph-mon[35690]: osd.5 [v2:172.21.15.190:6808/318560672,v1:172.21.15.190:6809/318560672] boot 2024-03-11T19:04:23.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:23 smithi170 ceph-mon[35690]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T19:04:23.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:23 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:23.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:23 smithi170 ceph-mon[35690]: Detected new or changed devices on smithi190 2024-03-11T19:04:23.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:23 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:23.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:23 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:23.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:23 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:04:23.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:23 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:04:23.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:23 smithi170 ceph-mon[35690]: Adjusting osd_memory_target on smithi190 to 8544M 2024-03-11T19:04:23.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:23 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:23.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:23 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:23.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:23 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:04:23.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:23 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:23.866 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:23 smithi029 ceph-mon[32439]: purged_snaps scrub starts 2024-03-11T19:04:23.866 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:23 smithi029 ceph-mon[32439]: purged_snaps scrub ok 2024-03-11T19:04:23.866 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:23.866 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:23 smithi029 ceph-mon[32439]: osd.5 [v2:172.21.15.190:6808/318560672,v1:172.21.15.190:6809/318560672] boot 2024-03-11T19:04:23.867 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:23 smithi029 ceph-mon[32439]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T19:04:23.867 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:23.867 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:23 smithi029 ceph-mon[32439]: Detected new or changed devices on smithi190 2024-03-11T19:04:23.867 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:23.867 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:23.867 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:04:23.867 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:04:23.867 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:23 smithi029 ceph-mon[32439]: Adjusting osd_memory_target on smithi190 to 8544M 2024-03-11T19:04:23.867 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:23.867 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:23.867 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:04:23.867 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:24.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:23 smithi190 ceph-mon[36292]: purged_snaps scrub starts 2024-03-11T19:04:24.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:23 smithi190 ceph-mon[36292]: purged_snaps scrub ok 2024-03-11T19:04:24.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:23 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:24.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:23 smithi190 ceph-mon[36292]: osd.5 [v2:172.21.15.190:6808/318560672,v1:172.21.15.190:6809/318560672] boot 2024-03-11T19:04:24.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:23 smithi190 ceph-mon[36292]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T19:04:24.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:23 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:04:24.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:23 smithi190 ceph-mon[36292]: Detected new or changed devices on smithi190 2024-03-11T19:04:24.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:23 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:24.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:23 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:24.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:23 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:04:24.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:23 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:04:24.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:23 smithi190 ceph-mon[36292]: Adjusting osd_memory_target on smithi190 to 8544M 2024-03-11T19:04:24.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:23 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:24.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:23 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:24.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:23 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:04:24.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:23 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:24.795 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:24 smithi190 ceph-mon[36292]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:24.795 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:24 smithi190 ceph-mon[36292]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T19:04:24.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:24 smithi029 ceph-mon[32439]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:24.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:24 smithi029 ceph-mon[32439]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T19:04:25.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:24 smithi170 ceph-mon[35690]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:04:25.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:24 smithi170 ceph-mon[35690]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T19:04:25.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.306+0000 7f3366dea700 1 -- 172.21.15.29:0/2526802296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3360102ea0 msgr2=0x7f3360103300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:04:25.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.306+0000 7f3366dea700 1 --2- 172.21.15.29:0/2526802296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3360102ea0 0x7f3360103300 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f334c0099c0 tx=0x7f334c009cd0 comp rx=0 tx=0).stop 2024-03-11T19:04:25.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.306+0000 7f3366dea700 1 -- 172.21.15.29:0/2526802296 shutdown_connections 2024-03-11T19:04:25.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.306+0000 7f3366dea700 1 --2- 172.21.15.29:0/2526802296 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3360103840 0x7f336010a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:25.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.306+0000 7f3366dea700 1 --2- 172.21.15.29:0/2526802296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3360101ca0 0x7f33601020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:25.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.306+0000 7f3366dea700 1 --2- 172.21.15.29:0/2526802296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3360102ea0 0x7f3360103300 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:25.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.306+0000 7f3366dea700 1 -- 172.21.15.29:0/2526802296 >> 172.21.15.29:0/2526802296 conn(0x7f33600fd280 msgr2=0x7f33600ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:04:25.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.306+0000 7f3366dea700 1 -- 172.21.15.29:0/2526802296 shutdown_connections 2024-03-11T19:04:25.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.306+0000 7f3366dea700 1 -- 172.21.15.29:0/2526802296 wait complete. 2024-03-11T19:04:25.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.307+0000 7f3366dea700 1 Processor -- start 2024-03-11T19:04:25.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.307+0000 7f3366dea700 1 -- start start 2024-03-11T19:04:25.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.307+0000 7f3366dea700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3360101ca0 0x7f3360075a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:04:25.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.307+0000 7f3366dea700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3360102ea0 0x7f3360074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:04:25.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.307+0000 7f3366dea700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3360103840 0x7f33600745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:04:25.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.307+0000 7f3366dea700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3360074d20 con 0x7f3360101ca0 2024-03-11T19:04:25.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.307+0000 7f3366dea700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3360074e90 con 0x7f3360103840 2024-03-11T19:04:25.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.307+0000 7f3366dea700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33601a1b70 con 0x7f3360102ea0 2024-03-11T19:04:25.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.307+0000 7f3364b86700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3360101ca0 0x7f3360075a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:04:25.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.307+0000 7f3364b86700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3360101ca0 0x7f3360075a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38110/0 (socket says 172.21.15.29:38110) 2024-03-11T19:04:25.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.307+0000 7f3364b86700 1 -- 172.21.15.29:0/888532549 learned_addr learned my addr 172.21.15.29:0/888532549 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:04:25.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.307+0000 7f335ffff700 1 --2- 172.21.15.29:0/888532549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3360102ea0 0x7f3360074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:04:25.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.307+0000 7f3365387700 1 --2- 172.21.15.29:0/888532549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3360103840 0x7f33600745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:04:25.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.307+0000 7f3364b86700 1 -- 172.21.15.29:0/888532549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3360102ea0 msgr2=0x7f3360074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:04:25.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.307+0000 7f3364b86700 1 --2- 172.21.15.29:0/888532549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3360102ea0 0x7f3360074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:25.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.307+0000 7f3364b86700 1 -- 172.21.15.29:0/888532549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3360103840 msgr2=0x7f33600745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:04:25.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.307+0000 7f3364b86700 1 --2- 172.21.15.29:0/888532549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3360103840 0x7f33600745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:25.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.307+0000 7f3364b86700 1 -- 172.21.15.29:0/888532549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334c0095f0 con 0x7f3360101ca0 2024-03-11T19:04:25.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.307+0000 7f3364b86700 1 --2- 172.21.15.29:0/888532549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3360101ca0 0x7f3360075a10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f335400e970 tx=0x7f335400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:04:25.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.308+0000 7f335dffb700 1 -- 172.21.15.29:0/888532549 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3354009790 con 0x7f3360101ca0 2024-03-11T19:04:25.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.308+0000 7f335dffb700 1 -- 172.21.15.29:0/888532549 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f335400c7e0 con 0x7f3360101ca0 2024-03-11T19:04:25.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.308+0000 7f3366dea700 1 -- 172.21.15.29:0/888532549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33601a1e50 con 0x7f3360101ca0 2024-03-11T19:04:25.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.308+0000 7f335dffb700 1 -- 172.21.15.29:0/888532549 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3354003c40 con 0x7f3360101ca0 2024-03-11T19:04:25.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.308+0000 7f3366dea700 1 -- 172.21.15.29:0/888532549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33601a2270 con 0x7f3360101ca0 2024-03-11T19:04:25.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.309+0000 7f3366dea700 1 -- 172.21.15.29:0/888532549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3360068930 con 0x7f3360101ca0 2024-03-11T19:04:25.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.310+0000 7f335dffb700 1 -- 172.21.15.29:0/888532549 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f335400c950 con 0x7f3360101ca0 2024-03-11T19:04:25.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.310+0000 7f335dffb700 1 --2- 172.21.15.29:0/888532549 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f33500a1b40 0x7f33500a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:04:25.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.310+0000 7f335dffb700 1 -- 172.21.15.29:0/888532549 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4562+0+0 (secure 0 0 0) 0x7f3354014070 con 0x7f3360101ca0 2024-03-11T19:04:25.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.311+0000 7f335ffff700 1 --2- 172.21.15.29:0/888532549 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f33500a1b40 0x7f33500a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:04:25.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.311+0000 7f335ffff700 1 --2- 172.21.15.29:0/888532549 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f33500a1b40 0x7f33500a4000 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f334c00b580 tx=0x7f334c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:04:25.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.315+0000 7f335dffb700 1 -- 172.21.15.29:0/888532549 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3354096700 con 0x7f3360101ca0 2024-03-11T19:04:25.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:25.529+0000 7f3366dea700 1 -- 172.21.15.29:0/888532549 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f336002d950 con 0x7f33500a1b40 2024-03-11T19:04:25.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:25 smithi029 ceph-mon[32439]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T19:04:25.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:25 smithi029 ceph-mon[32439]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:04:25.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:25 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:04:25.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:25 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:04:25.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:25 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:25.997 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:25 smithi170 ceph-mon[35690]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T19:04:25.997 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:25 smithi170 ceph-mon[35690]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:04:25.997 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:25 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:04:25.997 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:25 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:04:25.997 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:25 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:26.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:25 smithi190 ceph-mon[36292]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T19:04:26.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:25 smithi190 ceph-mon[36292]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:04:26.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:25 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:04:26.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:25 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:04:26.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:25 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:26.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:26 smithi029 ceph-mon[32439]: pgmap v143: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-11T19:04:26.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:26 smithi029 ceph-mon[32439]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-11T19:04:26.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:26 smithi029 ceph-mon[32439]: osdmap e43: 6 total, 6 up, 6 in 2024-03-11T19:04:27.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:26 smithi190 ceph-mon[36292]: pgmap v143: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-11T19:04:27.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:26 smithi190 ceph-mon[36292]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-11T19:04:27.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:26 smithi190 ceph-mon[36292]: osdmap e43: 6 total, 6 up, 6 in 2024-03-11T19:04:27.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:26 smithi170 ceph-mon[35690]: pgmap v143: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-11T19:04:27.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:26 smithi170 ceph-mon[35690]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-11T19:04:27.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:26 smithi170 ceph-mon[35690]: osdmap e43: 6 total, 6 up, 6 in 2024-03-11T19:04:27.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:27 smithi029 ceph-mon[32439]: osdmap e44: 6 total, 6 up, 6 in 2024-03-11T19:04:28.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:27 smithi190 ceph-mon[36292]: osdmap e44: 6 total, 6 up, 6 in 2024-03-11T19:04:28.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:27 smithi170 ceph-mon[35690]: osdmap e44: 6 total, 6 up, 6 in 2024-03-11T19:04:28.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:28 smithi029 ceph-mon[32439]: pgmap v146: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-11T19:04:28.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:28 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/66518323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ab9b5a8-7c19-4e5f-b211-6201fee37f11"}]: dispatch 2024-03-11T19:04:28.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:28 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/66518323' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ab9b5a8-7c19-4e5f-b211-6201fee37f11"}]': finished 2024-03-11T19:04:28.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:28 smithi029 ceph-mon[32439]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T19:04:28.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:28 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:04:29.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:28 smithi190 ceph-mon[36292]: pgmap v146: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-11T19:04:29.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:28 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/66518323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ab9b5a8-7c19-4e5f-b211-6201fee37f11"}]: dispatch 2024-03-11T19:04:29.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:28 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/66518323' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ab9b5a8-7c19-4e5f-b211-6201fee37f11"}]': finished 2024-03-11T19:04:29.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:28 smithi190 ceph-mon[36292]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T19:04:29.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:28 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:04:29.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:28 smithi170 ceph-mon[35690]: pgmap v146: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-11T19:04:29.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:28 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/66518323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ab9b5a8-7c19-4e5f-b211-6201fee37f11"}]: dispatch 2024-03-11T19:04:29.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:28 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/66518323' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ab9b5a8-7c19-4e5f-b211-6201fee37f11"}]': finished 2024-03-11T19:04:29.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:28 smithi170 ceph-mon[35690]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T19:04:29.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:28 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:04:29.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:29 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1743487465' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:04:30.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:29 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1743487465' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:04:30.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:29 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1743487465' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:04:30.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:30 smithi029 ceph-mon[32439]: pgmap v148: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-11T19:04:31.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:30 smithi190 ceph-mon[36292]: pgmap v148: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-11T19:04:31.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:30 smithi170 ceph-mon[35690]: pgmap v148: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-11T19:04:32.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:31 smithi190 ceph-mon[36292]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-11T19:04:32.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:31 smithi190 ceph-mon[36292]: Cluster is now healthy 2024-03-11T19:04:32.043 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:31 smithi029 ceph-mon[32439]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-11T19:04:32.043 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:31 smithi029 ceph-mon[32439]: Cluster is now healthy 2024-03-11T19:04:32.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:31 smithi170 ceph-mon[35690]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-11T19:04:32.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:31 smithi170 ceph-mon[35690]: Cluster is now healthy 2024-03-11T19:04:33.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:32 smithi190 ceph-mon[36292]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:04:33.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:32 smithi170 ceph-mon[35690]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:04:33.122 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:32 smithi029 ceph-mon[32439]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:04:34.776 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:34 smithi029 ceph-mon[32439]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-11T19:04:35.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:34 smithi190 ceph-mon[36292]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-11T19:04:35.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:34 smithi170 ceph-mon[35690]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-11T19:04:36.915 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:36 smithi029 ceph-mon[32439]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T19:04:36.915 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:36 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T19:04:36.915 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:36 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:36.915 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:36 smithi029 ceph-mon[32439]: Deploying daemon osd.6 on smithi029 2024-03-11T19:04:37.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:36 smithi190 ceph-mon[36292]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T19:04:37.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:36 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T19:04:37.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:36 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:37.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:36 smithi190 ceph-mon[36292]: Deploying daemon osd.6 on smithi029 2024-03-11T19:04:37.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:36 smithi170 ceph-mon[35690]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T19:04:37.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:36 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T19:04:37.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:36 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:37.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:36 smithi170 ceph-mon[35690]: Deploying daemon osd.6 on smithi029 2024-03-11T19:04:38.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:38 smithi029 ceph-mon[32439]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T19:04:39.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:38 smithi190 ceph-mon[36292]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T19:04:39.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:38 smithi170 ceph-mon[35690]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T19:04:41.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:40 smithi190 ceph-mon[36292]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T19:04:41.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:40 smithi170 ceph-mon[35690]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T19:04:41.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:40 smithi029 ceph-mon[32439]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T19:04:43.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:43 smithi170 ceph-mon[35690]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-11T19:04:43.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:43 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:43.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:43 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:04:43.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:43 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:43.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:43 smithi029 ceph-mon[32439]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-11T19:04:43.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:43.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:04:43.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:43.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:43 smithi190 ceph-mon[36292]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-11T19:04:43.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:43 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:43.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:43 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:04:43.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:43 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:45.393 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:45 smithi029 ceph-mon[32439]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:04:45.393 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:45 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:45.394 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:45 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:45.394 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:45 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:45.394 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:45 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:04:45.394 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:45 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:45.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:45 smithi170 ceph-mon[35690]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:04:45.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:45 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:45.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:45 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:45.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:45 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:45.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:45 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:04:45.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:45 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:45.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:45 smithi190 ceph-mon[36292]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:04:45.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:45 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:45.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:45 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:45.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:45 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:45.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:45 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:04:45.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:45 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:46.397 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:46 smithi029 ceph-mon[32439]: from='osd.6 [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T19:04:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:46.438+0000 7f335dffb700 1 -- 172.21.15.29:0/888532549 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f336002d950 con 0x7f33500a1b40 2024-03-11T19:04:46.441 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 6 on host 'smithi029' 2024-03-11T19:04:46.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:46.440+0000 7f3366dea700 1 -- 172.21.15.29:0/888532549 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f33500a1b40 msgr2=0x7f33500a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:04:46.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:46.440+0000 7f3366dea700 1 --2- 172.21.15.29:0/888532549 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f33500a1b40 0x7f33500a4000 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f334c00b580 tx=0x7f334c005fb0 comp rx=0 tx=0).stop 2024-03-11T19:04:46.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:46.440+0000 7f3366dea700 1 -- 172.21.15.29:0/888532549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3360101ca0 msgr2=0x7f3360075a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:04:46.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:46.440+0000 7f3366dea700 1 --2- 172.21.15.29:0/888532549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3360101ca0 0x7f3360075a10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f335400e970 tx=0x7f335400ec80 comp rx=0 tx=0).stop 2024-03-11T19:04:46.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:46.440+0000 7f3366dea700 1 -- 172.21.15.29:0/888532549 shutdown_connections 2024-03-11T19:04:46.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:46.440+0000 7f3366dea700 1 --2- 172.21.15.29:0/888532549 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f33500a1b40 0x7f33500a4000 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:46.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:46.440+0000 7f3366dea700 1 --2- 172.21.15.29:0/888532549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3360103840 0x7f33600745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:46.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:46.440+0000 7f3366dea700 1 --2- 172.21.15.29:0/888532549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3360101ca0 0x7f3360075a10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:46.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:46.440+0000 7f3366dea700 1 --2- 172.21.15.29:0/888532549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3360102ea0 0x7f3360074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:46.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:46.440+0000 7f3366dea700 1 -- 172.21.15.29:0/888532549 >> 172.21.15.29:0/888532549 conn(0x7f33600fd280 msgr2=0x7f3360106220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:04:46.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:46.440+0000 7f3366dea700 1 -- 172.21.15.29:0/888532549 shutdown_connections 2024-03-11T19:04:46.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:04:46.440+0000 7f3366dea700 1 -- 172.21.15.29:0/888532549 wait complete. 2024-03-11T19:04:46.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:46 smithi170 ceph-mon[35690]: from='osd.6 [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T19:04:46.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:46 smithi190 ceph-mon[36292]: from='osd.6 [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T19:04:47.148 DEBUG:teuthology.orchestra.run.smithi029:osd.6> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.6.service 2024-03-11T19:04:47.151 INFO:tasks.cephadm:Deploying osd.7 on smithi170 with /dev/vg_nvme/lv_2... 2024-03-11T19:04:47.151 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T19:04:47.400 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:04:47.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:47 smithi029 ceph-mon[32439]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:04:47.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:47 smithi029 ceph-mon[32439]: from='osd.6 [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T19:04:47.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:47 smithi029 ceph-mon[32439]: osdmap e46: 7 total, 6 up, 7 in 2024-03-11T19:04:47.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:47 smithi029 ceph-mon[32439]: from='osd.6 [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-11T19:04:47.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:04:47.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:47.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:47.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:04:47.523 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:47 smithi170 ceph-mon[35690]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:04:47.523 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:47 smithi170 ceph-mon[35690]: from='osd.6 [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T19:04:47.523 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:47 smithi170 ceph-mon[35690]: osdmap e46: 7 total, 6 up, 7 in 2024-03-11T19:04:47.523 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:47 smithi170 ceph-mon[35690]: from='osd.6 [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-11T19:04:47.523 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:04:47.523 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:47.523 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:47.523 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:04:47.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:47 smithi190 ceph-mon[36292]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:04:47.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:47 smithi190 ceph-mon[36292]: from='osd.6 [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T19:04:47.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:47 smithi190 ceph-mon[36292]: osdmap e46: 7 total, 6 up, 7 in 2024-03-11T19:04:47.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:47 smithi190 ceph-mon[36292]: from='osd.6 [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-11T19:04:47.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:04:47.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:47.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:47.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:04:48.156 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:04:47 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:04:47.734+0000 7f30a459c700 -1 osd.6 0 waiting for initial osdmap 2024-03-11T19:04:48.156 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:04:47 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:04:47.743+0000 7f309c387700 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:04:48.569 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:48 smithi029 ceph-mon[32439]: from='osd.6 [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-11T19:04:48.570 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:48 smithi029 ceph-mon[32439]: osdmap e47: 7 total, 6 up, 7 in 2024-03-11T19:04:48.570 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:48 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:04:48.570 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:48 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:04:48.570 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:48 smithi029 ceph-mon[32439]: from='osd.6 [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214]' entity='osd.6' 2024-03-11T19:04:48.570 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:48 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:04:48.590 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:48 smithi170 ceph-mon[35690]: from='osd.6 [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-11T19:04:48.590 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:48 smithi170 ceph-mon[35690]: osdmap e47: 7 total, 6 up, 7 in 2024-03-11T19:04:48.590 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:48 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:04:48.590 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:48 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:04:48.590 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:48 smithi170 ceph-mon[35690]: from='osd.6 [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214]' entity='osd.6' 2024-03-11T19:04:48.590 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:48 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:04:48.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:48 smithi190 ceph-mon[36292]: from='osd.6 [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-11T19:04:48.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:48 smithi190 ceph-mon[36292]: osdmap e47: 7 total, 6 up, 7 in 2024-03-11T19:04:48.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:48 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:04:48.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:48 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:04:48.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:48 smithi190 ceph-mon[36292]: from='osd.6 [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214]' entity='osd.6' 2024-03-11T19:04:48.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:48 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:04:49.466 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T19:04:49.467 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T19:04:49.467 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-03-11T19:04:49.467 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-03-11T19:04:49.467 INFO:teuthology.orchestra.run.smithi170.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00866673 s, 1.2 GB/s 2024-03-11T19:04:49.467 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 2024-03-11T19:04:49.467 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-03-11T19:04:49.555 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:49 smithi029 ceph-mon[32439]: purged_snaps scrub starts 2024-03-11T19:04:49.556 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:49 smithi029 ceph-mon[32439]: purged_snaps scrub ok 2024-03-11T19:04:49.556 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:49 smithi029 ceph-mon[32439]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:04:49.556 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:49 smithi029 ceph-mon[32439]: osd.6 [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214] boot 2024-03-11T19:04:49.556 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:49 smithi029 ceph-mon[32439]: osdmap e48: 7 total, 7 up, 7 in 2024-03-11T19:04:49.556 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:49 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:04:49.581 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:49 smithi170 ceph-mon[35690]: purged_snaps scrub starts 2024-03-11T19:04:49.582 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:49 smithi170 ceph-mon[35690]: purged_snaps scrub ok 2024-03-11T19:04:49.582 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:49 smithi170 ceph-mon[35690]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:04:49.582 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:49 smithi170 ceph-mon[35690]: osd.6 [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214] boot 2024-03-11T19:04:49.582 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:49 smithi170 ceph-mon[35690]: osdmap e48: 7 total, 7 up, 7 in 2024-03-11T19:04:49.582 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:49 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:04:49.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:49 smithi190 ceph-mon[36292]: purged_snaps scrub starts 2024-03-11T19:04:49.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:49 smithi190 ceph-mon[36292]: purged_snaps scrub ok 2024-03-11T19:04:49.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:49 smithi190 ceph-mon[36292]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:04:49.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:49 smithi190 ceph-mon[36292]: osd.6 [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214] boot 2024-03-11T19:04:49.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:49 smithi190 ceph-mon[36292]: osdmap e48: 7 total, 7 up, 7 in 2024-03-11T19:04:49.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:49 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:04:50.134 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi170:vg_nvme/lv_2 2024-03-11T19:04:50.377 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:04:51.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:50 smithi190 ceph-mon[36292]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:04:51.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:50 smithi190 ceph-mon[36292]: osdmap e49: 7 total, 7 up, 7 in 2024-03-11T19:04:51.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:50 smithi190 ceph-mon[36292]: Detected new or changed devices on smithi029 2024-03-11T19:04:51.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:51.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:51.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:04:51.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:04:51.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:04:51.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:50 smithi190 ceph-mon[36292]: Adjusting osd_memory_target on smithi029 to 5697M 2024-03-11T19:04:51.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:51.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:51.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:04:51.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:51.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:50 smithi170 ceph-mon[35690]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:04:51.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:50 smithi170 ceph-mon[35690]: osdmap e49: 7 total, 7 up, 7 in 2024-03-11T19:04:51.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:50 smithi170 ceph-mon[35690]: Detected new or changed devices on smithi029 2024-03-11T19:04:51.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:51.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:51.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:04:51.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:04:51.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:04:51.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:50 smithi170 ceph-mon[35690]: Adjusting osd_memory_target on smithi029 to 5697M 2024-03-11T19:04:51.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:51.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:51.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:04:51.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:51.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:50 smithi029 ceph-mon[32439]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:04:51.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:50 smithi029 ceph-mon[32439]: osdmap e49: 7 total, 7 up, 7 in 2024-03-11T19:04:51.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:50 smithi029 ceph-mon[32439]: Detected new or changed devices on smithi029 2024-03-11T19:04:51.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:04:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:04:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:04:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:50 smithi029 ceph-mon[32439]: Adjusting osd_memory_target on smithi029 to 5697M 2024-03-11T19:04:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:04:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:04:52.080 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.077+0000 7f46b22d0700 1 -- 172.21.15.170:0/2684375437 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46ac105190 msgr2=0x7f46ac107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:04:52.080 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.077+0000 7f46b22d0700 1 --2- 172.21.15.170:0/2684375437 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46ac105190 0x7f46ac107580 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f46a0009990 tx=0x7f46a0009ca0 comp rx=0 tx=0).stop 2024-03-11T19:04:52.080 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.077+0000 7f46b22d0700 1 -- 172.21.15.170:0/2684375437 shutdown_connections 2024-03-11T19:04:52.080 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.077+0000 7f46b22d0700 1 --2- 172.21.15.170:0/2684375437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46ac107ac0 0x7f46ac109eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:52.080 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.077+0000 7f46b22d0700 1 --2- 172.21.15.170:0/2684375437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46ac102860 0x7f46ac104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:52.080 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.077+0000 7f46b22d0700 1 --2- 172.21.15.170:0/2684375437 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46ac105190 0x7f46ac107580 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:52.080 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.077+0000 7f46b22d0700 1 -- 172.21.15.170:0/2684375437 >> 172.21.15.170:0/2684375437 conn(0x7f46ac0fc1d0 msgr2=0x7f46ac0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:04:52.080 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.077+0000 7f46b22d0700 1 -- 172.21.15.170:0/2684375437 shutdown_connections 2024-03-11T19:04:52.080 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.077+0000 7f46b22d0700 1 -- 172.21.15.170:0/2684375437 wait complete. 2024-03-11T19:04:52.081 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.078+0000 7f46b22d0700 1 Processor -- start 2024-03-11T19:04:52.081 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.078+0000 7f46b22d0700 1 -- start start 2024-03-11T19:04:52.082 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.078+0000 7f46b22d0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46ac102860 0x7f46ac1021c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:04:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.078+0000 7f46b22d0700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46ac105190 0x7f46ac100810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:04:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.078+0000 7f46b22d0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46ac107ac0 0x7f46ac100d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:04:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.078+0000 7f46b22d0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ac1014d0 con 0x7f46ac102860 2024-03-11T19:04:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.078+0000 7f46b22d0700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ac101640 con 0x7f46ac105190 2024-03-11T19:04:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.078+0000 7f46b22d0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ac1a1bb0 con 0x7f46ac107ac0 2024-03-11T19:04:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.078+0000 7f46ab7fe700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46ac105190 0x7f46ac100810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:04:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.079+0000 7f46ab7fe700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46ac105190 0x7f46ac100810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:37834/0 (socket says 172.21.15.170:37834) 2024-03-11T19:04:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.079+0000 7f46ab7fe700 1 -- 172.21.15.170:0/1979927404 learned_addr learned my addr 172.21.15.170:0/1979927404 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-11T19:04:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.079+0000 7f46abfff700 1 --2- 172.21.15.170:0/1979927404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46ac102860 0x7f46ac1021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:04:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.079+0000 7f46b086d700 1 --2- 172.21.15.170:0/1979927404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46ac107ac0 0x7f46ac100d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:04:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.079+0000 7f46ab7fe700 1 -- 172.21.15.170:0/1979927404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46ac107ac0 msgr2=0x7f46ac100d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:04:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.079+0000 7f46ab7fe700 1 --2- 172.21.15.170:0/1979927404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46ac107ac0 0x7f46ac100d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:52.084 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.079+0000 7f46ab7fe700 1 -- 172.21.15.170:0/1979927404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46ac102860 msgr2=0x7f46ac1021c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:04:52.084 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.079+0000 7f46ab7fe700 1 --2- 172.21.15.170:0/1979927404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46ac102860 0x7f46ac1021c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:04:52.084 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.079+0000 7f46ab7fe700 1 -- 172.21.15.170:0/1979927404 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a00095f0 con 0x7f46ac105190 2024-03-11T19:04:52.084 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.079+0000 7f46b086d700 1 --2- 172.21.15.170:0/1979927404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46ac107ac0 0x7f46ac100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:04:52.084 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.079+0000 7f46ab7fe700 1 --2- 172.21.15.170:0/1979927404 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46ac105190 0x7f46ac100810 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f46a0000c00 tx=0x7f46a0005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:04:52.084 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.079+0000 7f46a97fa700 1 -- 172.21.15.170:0/1979927404 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a001d070 con 0x7f46ac105190 2024-03-11T19:04:52.084 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.079+0000 7f46b22d0700 1 -- 172.21.15.170:0/1979927404 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46ac1a1e30 con 0x7f46ac105190 2024-03-11T19:04:52.084 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.079+0000 7f46a97fa700 1 -- 172.21.15.170:0/1979927404 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f46a0004100 con 0x7f46ac105190 2024-03-11T19:04:52.084 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.079+0000 7f46a97fa700 1 -- 172.21.15.170:0/1979927404 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a0005290 con 0x7f46ac105190 2024-03-11T19:04:52.084 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.079+0000 7f46b22d0700 1 -- 172.21.15.170:0/1979927404 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46ac1a2240 con 0x7f46ac105190 2024-03-11T19:04:52.084 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.080+0000 7f46b22d0700 1 -- 172.21.15.170:0/1979927404 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46ac0fdd50 con 0x7f46ac105190 2024-03-11T19:04:52.085 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.081+0000 7f46a97fa700 1 -- 172.21.15.170:0/1979927404 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f46a0004950 con 0x7f46ac105190 2024-03-11T19:04:52.085 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.082+0000 7f46a97fa700 1 --2- 172.21.15.170:0/1979927404 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f46940a1ac0 0x7f46940a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:04:52.085 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.082+0000 7f46a97fa700 1 -- 172.21.15.170:0/1979927404 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f46a00cda60 con 0x7f46ac105190 2024-03-11T19:04:52.085 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.082+0000 7f46abfff700 1 --2- 172.21.15.170:0/1979927404 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f46940a1ac0 0x7f46940a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:04:52.085 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.083+0000 7f46abfff700 1 --2- 172.21.15.170:0/1979927404 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f46940a1ac0 0x7f46940a3f80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f469c00bbd0 tx=0x7f469c00b5b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:04:52.087 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.085+0000 7f46a97fa700 1 -- 172.21.15.170:0/1979927404 <== mon.2 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f46a0097b30 con 0x7f46ac105190 2024-03-11T19:04:52.289 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:04:52.287+0000 7f46b22d0700 1 -- 172.21.15.170:0/1979927404 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f46ac02d800 con 0x7f46940a1ac0 2024-03-11T19:04:53.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:52 smithi190 ceph-mon[36292]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:04:53.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:52 smithi190 ceph-mon[36292]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:04:53.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:52 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:04:53.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:52 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:04:53.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:52 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:53.015 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:52 smithi170 ceph-mon[35690]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:04:53.015 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:52 smithi170 ceph-mon[35690]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:04:53.015 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:52 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:04:53.015 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:52 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:04:53.015 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:52 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:53.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:52 smithi029 ceph-mon[32439]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:04:53.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:52 smithi029 ceph-mon[32439]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:04:53.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:52 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:04:53.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:52 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:04:53.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:52 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:04:55.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:54 smithi190 ceph-mon[36292]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:04:55.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:54 smithi170 ceph-mon[35690]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:04:55.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:54 smithi029 ceph-mon[32439]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:04:56.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:55 smithi190 ceph-mon[36292]: from='client.? 172.21.15.170:0/3483914322' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7d84364-5bf1-473c-9676-253fe01bff2b"}]: dispatch 2024-03-11T19:04:56.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:55 smithi190 ceph-mon[36292]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7d84364-5bf1-473c-9676-253fe01bff2b"}]: dispatch 2024-03-11T19:04:56.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:55 smithi190 ceph-mon[36292]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7d84364-5bf1-473c-9676-253fe01bff2b"}]': finished 2024-03-11T19:04:56.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:55 smithi190 ceph-mon[36292]: osdmap e50: 8 total, 7 up, 8 in 2024-03-11T19:04:56.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:55 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:04:56.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:55 smithi190 ceph-mon[36292]: from='client.? 172.21.15.170:0/3553842621' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:04:56.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:55 smithi170 ceph-mon[35690]: from='client.? 172.21.15.170:0/3483914322' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7d84364-5bf1-473c-9676-253fe01bff2b"}]: dispatch 2024-03-11T19:04:56.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:55 smithi170 ceph-mon[35690]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7d84364-5bf1-473c-9676-253fe01bff2b"}]: dispatch 2024-03-11T19:04:56.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:55 smithi170 ceph-mon[35690]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7d84364-5bf1-473c-9676-253fe01bff2b"}]': finished 2024-03-11T19:04:56.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:55 smithi170 ceph-mon[35690]: osdmap e50: 8 total, 7 up, 8 in 2024-03-11T19:04:56.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:55 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:04:56.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:55 smithi170 ceph-mon[35690]: from='client.? 172.21.15.170:0/3553842621' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:04:56.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:55 smithi029 ceph-mon[32439]: from='client.? 172.21.15.170:0/3483914322' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7d84364-5bf1-473c-9676-253fe01bff2b"}]: dispatch 2024-03-11T19:04:56.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:55 smithi029 ceph-mon[32439]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7d84364-5bf1-473c-9676-253fe01bff2b"}]: dispatch 2024-03-11T19:04:56.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:55 smithi029 ceph-mon[32439]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7d84364-5bf1-473c-9676-253fe01bff2b"}]': finished 2024-03-11T19:04:56.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:55 smithi029 ceph-mon[32439]: osdmap e50: 8 total, 7 up, 8 in 2024-03-11T19:04:56.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:55 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:04:56.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:55 smithi029 ceph-mon[32439]: from='client.? 172.21.15.170:0/3553842621' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:04:57.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:57 smithi170 ceph-mon[35690]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:04:57.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:57 smithi029 ceph-mon[32439]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:04:57.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:57 smithi190 ceph-mon[36292]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:04:59.285 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:04:59 smithi170 ceph-mon[35690]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:04:59.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:04:59 smithi029 ceph-mon[32439]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:04:59.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:04:59 smithi190 ceph-mon[36292]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:01.191 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:01 smithi170 ceph-mon[35690]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:01.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:01 smithi029 ceph-mon[32439]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:01.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:01 smithi190 ceph-mon[36292]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:03.146 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:03 smithi170 ceph-mon[35690]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:03.146 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:03 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T19:05:03.146 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:03 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:03.147 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:03 smithi170 ceph-mon[35690]: Deploying daemon osd.7 on smithi170 2024-03-11T19:05:03.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:03 smithi029 ceph-mon[32439]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:03.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:03 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T19:05:03.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:03 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:03.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:03 smithi029 ceph-mon[32439]: Deploying daemon osd.7 on smithi170 2024-03-11T19:05:03.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:03 smithi190 ceph-mon[36292]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:03 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T19:05:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:03 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:03 smithi190 ceph-mon[36292]: Deploying daemon osd.7 on smithi170 2024-03-11T19:05:05.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:05 smithi170 ceph-mon[35690]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:05.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:05 smithi029 ceph-mon[32439]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:05.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:05 smithi190 ceph-mon[36292]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:07.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:07 smithi170 ceph-mon[35690]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:07.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:07 smithi029 ceph-mon[32439]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:07.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:07 smithi190 ceph-mon[36292]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:09.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:08 smithi029 ceph-mon[32439]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:09.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:09.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:09.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:05:09.229 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:08 smithi170 ceph-mon[35690]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:09.229 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:09.229 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:09.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:05:09.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:08 smithi190 ceph-mon[36292]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:09.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:09.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:09.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:05:11.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:10 smithi190 ceph-mon[36292]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:11.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:10 smithi170 ceph-mon[35690]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:11.405 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 7 on host 'smithi170' 2024-03-11T19:05:11.406 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:05:11.402+0000 7f46a97fa700 1 -- 172.21.15.170:0/1979927404 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f46ac02d800 con 0x7f46940a1ac0 2024-03-11T19:05:11.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:10 smithi029 ceph-mon[32439]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:11.407 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:05:11.404+0000 7f46b22d0700 1 -- 172.21.15.170:0/1979927404 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f46940a1ac0 msgr2=0x7f46940a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:05:11.407 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:05:11.404+0000 7f46b22d0700 1 --2- 172.21.15.170:0/1979927404 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f46940a1ac0 0x7f46940a3f80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f469c00bbd0 tx=0x7f469c00b5b0 comp rx=0 tx=0).stop 2024-03-11T19:05:11.408 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:05:11.404+0000 7f46b22d0700 1 -- 172.21.15.170:0/1979927404 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46ac105190 msgr2=0x7f46ac100810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:05:11.408 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:05:11.404+0000 7f46b22d0700 1 --2- 172.21.15.170:0/1979927404 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46ac105190 0x7f46ac100810 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f46a0000c00 tx=0x7f46a0005ef0 comp rx=0 tx=0).stop 2024-03-11T19:05:11.408 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:05:11.404+0000 7f46b22d0700 1 -- 172.21.15.170:0/1979927404 shutdown_connections 2024-03-11T19:05:11.408 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:05:11.404+0000 7f46b22d0700 1 --2- 172.21.15.170:0/1979927404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46ac107ac0 0x7f46ac100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:05:11.408 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:05:11.404+0000 7f46b22d0700 1 --2- 172.21.15.170:0/1979927404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46ac102860 0x7f46ac1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:05:11.408 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:05:11.404+0000 7f46b22d0700 1 --2- 172.21.15.170:0/1979927404 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f46940a1ac0 0x7f46940a3f80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:05:11.408 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:05:11.404+0000 7f46b22d0700 1 --2- 172.21.15.170:0/1979927404 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46ac105190 0x7f46ac100810 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:05:11.408 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:05:11.404+0000 7f46b22d0700 1 -- 172.21.15.170:0/1979927404 >> 172.21.15.170:0/1979927404 conn(0x7f46ac0fc1d0 msgr2=0x7f46ac103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:05:11.408 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:05:11.405+0000 7f46b22d0700 1 -- 172.21.15.170:0/1979927404 shutdown_connections 2024-03-11T19:05:11.408 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:05:11.405+0000 7f46b22d0700 1 -- 172.21.15.170:0/1979927404 wait complete. 2024-03-11T19:05:12.074 DEBUG:teuthology.orchestra.run.smithi170:osd.7> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.7.service 2024-03-11T19:05:12.077 INFO:tasks.cephadm:Deploying osd.8 on smithi190 with /dev/vg_nvme/lv_2... 2024-03-11T19:05:12.077 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T19:05:12.327 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:05:12.448 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:12 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:12.449 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:12 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:12.449 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:12 smithi190 ceph-mon[36292]: from='osd.7 [v2:172.21.15.170:6816/3259207349,v1:172.21.15.170:6817/3259207349]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T19:05:12.449 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:12 smithi190 ceph-mon[36292]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T19:05:12.449 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:12 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:12.449 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:12 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:12.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:12 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:12.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:12 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:12.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:12 smithi029 ceph-mon[32439]: from='osd.7 [v2:172.21.15.170:6816/3259207349,v1:172.21.15.170:6817/3259207349]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T19:05:12.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:12 smithi029 ceph-mon[32439]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T19:05:12.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:12 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:12.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:12 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:12.782 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:12 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:12.782 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:12 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:12.782 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:12 smithi170 ceph-mon[35690]: from='osd.7 [v2:172.21.15.170:6816/3259207349,v1:172.21.15.170:6817/3259207349]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T19:05:12.783 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:12 smithi170 ceph-mon[35690]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T19:05:12.783 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:12 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:12.783 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:12 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:13.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:13 smithi029 ceph-mon[32439]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:13.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:13 smithi029 ceph-mon[32439]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T19:05:13.658 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:13 smithi029 ceph-mon[32439]: from='osd.7 [v2:172.21.15.170:6816/3259207349,v1:172.21.15.170:6817/3259207349]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:05:13.658 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:13 smithi029 ceph-mon[32439]: osdmap e51: 8 total, 7 up, 8 in 2024-03-11T19:05:13.658 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:13 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:05:13.658 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:13 smithi029 ceph-mon[32439]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:05:13.678 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:13 smithi190 ceph-mon[36292]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:13.679 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:13 smithi190 ceph-mon[36292]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T19:05:13.679 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:13 smithi190 ceph-mon[36292]: from='osd.7 [v2:172.21.15.170:6816/3259207349,v1:172.21.15.170:6817/3259207349]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:05:13.679 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:13 smithi190 ceph-mon[36292]: osdmap e51: 8 total, 7 up, 8 in 2024-03-11T19:05:13.679 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:13 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:05:13.679 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:13 smithi190 ceph-mon[36292]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:05:13.796 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:13 smithi170 ceph-mon[35690]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:13.796 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:13 smithi170 ceph-mon[35690]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T19:05:13.796 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:13 smithi170 ceph-mon[35690]: from='osd.7 [v2:172.21.15.170:6816/3259207349,v1:172.21.15.170:6817/3259207349]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:05:13.796 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:13 smithi170 ceph-mon[35690]: osdmap e51: 8 total, 7 up, 8 in 2024-03-11T19:05:13.796 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:13 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:05:13.796 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:13 smithi170 ceph-mon[35690]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:05:14.052 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:05:13 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:05:13.808+0000 7f4a77ed3700 -1 osd.7 0 waiting for initial osdmap 2024-03-11T19:05:14.053 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:05:13 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:05:13.816+0000 7f4a6fcbe700 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:05:14.409 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T19:05:14.410 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T19:05:14.410 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-03-11T19:05:14.410 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-03-11T19:05:14.410 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113681 s, 922 MB/s 2024-03-11T19:05:14.410 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-03-11T19:05:14.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:14 smithi170 ceph-mon[35690]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-11T19:05:14.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:14 smithi170 ceph-mon[35690]: osdmap e52: 8 total, 7 up, 8 in 2024-03-11T19:05:14.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:14 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:05:14.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:14 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:05:14.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:14 smithi170 ceph-mon[35690]: from='osd.7 ' entity='osd.7' 2024-03-11T19:05:14.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:14 smithi029 ceph-mon[32439]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-11T19:05:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:14 smithi029 ceph-mon[32439]: osdmap e52: 8 total, 7 up, 8 in 2024-03-11T19:05:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:05:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:05:14.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:14 smithi029 ceph-mon[32439]: from='osd.7 ' entity='osd.7' 2024-03-11T19:05:14.709 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:14 smithi190 ceph-mon[36292]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-11T19:05:14.709 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:14 smithi190 ceph-mon[36292]: osdmap e52: 8 total, 7 up, 8 in 2024-03-11T19:05:14.709 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:14 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:05:14.709 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:14 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:05:14.709 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:14 smithi190 ceph-mon[36292]: from='osd.7 ' entity='osd.7' 2024-03-11T19:05:15.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi190:vg_nvme/lv_2 2024-03-11T19:05:15.330 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:05:15.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:15 smithi029 ceph-mon[32439]: purged_snaps scrub starts 2024-03-11T19:05:15.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:15 smithi029 ceph-mon[32439]: purged_snaps scrub ok 2024-03-11T19:05:15.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:15 smithi029 ceph-mon[32439]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:15.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:15 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:05:15.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:15 smithi029 ceph-mon[32439]: osd.7 [v2:172.21.15.170:6816/3259207349,v1:172.21.15.170:6817/3259207349] boot 2024-03-11T19:05:15.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:15 smithi029 ceph-mon[32439]: osdmap e53: 8 total, 8 up, 8 in 2024-03-11T19:05:15.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:15 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:05:15.675 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:15 smithi190 ceph-mon[36292]: purged_snaps scrub starts 2024-03-11T19:05:15.675 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:15 smithi190 ceph-mon[36292]: purged_snaps scrub ok 2024-03-11T19:05:15.675 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:15 smithi190 ceph-mon[36292]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:15.675 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:15 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:05:15.676 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:15 smithi190 ceph-mon[36292]: osd.7 [v2:172.21.15.170:6816/3259207349,v1:172.21.15.170:6817/3259207349] boot 2024-03-11T19:05:15.676 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:15 smithi190 ceph-mon[36292]: osdmap e53: 8 total, 8 up, 8 in 2024-03-11T19:05:15.676 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:15 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:05:15.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:15 smithi170 ceph-mon[35690]: purged_snaps scrub starts 2024-03-11T19:05:15.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:15 smithi170 ceph-mon[35690]: purged_snaps scrub ok 2024-03-11T19:05:15.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:15 smithi170 ceph-mon[35690]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:05:15.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:15 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:05:15.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:15 smithi170 ceph-mon[35690]: osd.7 [v2:172.21.15.170:6816/3259207349,v1:172.21.15.170:6817/3259207349] boot 2024-03-11T19:05:15.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:15 smithi170 ceph-mon[35690]: osdmap e53: 8 total, 8 up, 8 in 2024-03-11T19:05:15.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:15 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:05:16.410 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:16 smithi190 ceph-mon[36292]: osdmap e54: 8 total, 8 up, 8 in 2024-03-11T19:05:16.410 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:16.410 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:16.410 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:05:16.410 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:05:16.410 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:05:16.410 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:16.410 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:16.410 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:05:16.411 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:16.411 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:05:16.411 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:16.411 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:05:16.411 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:16.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:16 smithi029 ceph-mon[32439]: osdmap e54: 8 total, 8 up, 8 in 2024-03-11T19:05:16.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:16.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:16.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:05:16.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:05:16.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:05:16.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:16.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:16.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:05:16.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:16.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:05:16.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:16.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:05:16.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:16 smithi170 ceph-mon[35690]: osdmap e54: 8 total, 8 up, 8 in 2024-03-11T19:05:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:05:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:05:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:05:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:16.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:16.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:05:16.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:16.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:05:16.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:16.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:05:16.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:16.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.989+0000 7f62891b3700 1 -- 172.21.15.190:0/300502262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6284103840 msgr2=0x7f628410a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:05:16.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.989+0000 7f62891b3700 1 --2- 172.21.15.190:0/300502262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6284103840 0x7f628410a0f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f627c00b0a0 tx=0x7f627c00b3b0 comp rx=0 tx=0).stop 2024-03-11T19:05:16.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.989+0000 7f62891b3700 1 -- 172.21.15.190:0/300502262 shutdown_connections 2024-03-11T19:05:16.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.989+0000 7f62891b3700 1 --2- 172.21.15.190:0/300502262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6284103840 0x7f628410a0f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:05:16.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.989+0000 7f62891b3700 1 --2- 172.21.15.190:0/300502262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6284101ca0 0x7f62841020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:05:16.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.989+0000 7f62891b3700 1 --2- 172.21.15.190:0/300502262 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6284102ea0 0x7f6284103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:05:16.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.989+0000 7f62891b3700 1 -- 172.21.15.190:0/300502262 >> 172.21.15.190:0/300502262 conn(0x7f62840fd280 msgr2=0x7f62840ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:05:16.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.989+0000 7f62891b3700 1 -- 172.21.15.190:0/300502262 shutdown_connections 2024-03-11T19:05:16.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.989+0000 7f62891b3700 1 -- 172.21.15.190:0/300502262 wait complete. 2024-03-11T19:05:16.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.990+0000 7f62891b3700 1 Processor -- start 2024-03-11T19:05:16.995 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.990+0000 7f62891b3700 1 -- start start 2024-03-11T19:05:16.995 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.990+0000 7f62891b3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6284101ca0 0x7f6284197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:05:16.995 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.990+0000 7f62891b3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6284102ea0 0x7f6284198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:05:16.995 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.990+0000 7f62891b3700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6284103840 0x7f628419f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:05:16.995 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.990+0000 7f62891b3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628419f870 con 0x7f6284102ea0 2024-03-11T19:05:16.995 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.990+0000 7f62891b3700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628419f9b0 con 0x7f6284103840 2024-03-11T19:05:16.995 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.990+0000 7f62891b3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628419fb70 con 0x7f6284101ca0 2024-03-11T19:05:16.995 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.990+0000 7f6282d9d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6284101ca0 0x7f6284197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:05:16.995 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.990+0000 7f6282d9d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6284101ca0 0x7f6284197d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:43216/0 (socket says 172.21.15.190:43216) 2024-03-11T19:05:16.995 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.990+0000 7f6282d9d700 1 -- 172.21.15.190:0/1451649773 learned_addr learned my addr 172.21.15.190:0/1451649773 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-11T19:05:16.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.991+0000 7f628359e700 1 --2- 172.21.15.190:0/1451649773 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6284103840 0x7f628419f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:05:16.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.991+0000 7f628259c700 1 --2- 172.21.15.190:0/1451649773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6284102ea0 0x7f6284198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:05:16.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.991+0000 7f6282d9d700 1 -- 172.21.15.190:0/1451649773 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6284103840 msgr2=0x7f628419f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:05:16.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.991+0000 7f6282d9d700 1 --2- 172.21.15.190:0/1451649773 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6284103840 0x7f628419f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:05:16.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.991+0000 7f6282d9d700 1 -- 172.21.15.190:0/1451649773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6284102ea0 msgr2=0x7f6284198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:05:16.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.991+0000 7f6282d9d700 1 --2- 172.21.15.190:0/1451649773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6284102ea0 0x7f6284198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:05:16.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.991+0000 7f6282d9d700 1 -- 172.21.15.190:0/1451649773 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627c009d00 con 0x7f6284101ca0 2024-03-11T19:05:16.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.991+0000 7f628259c700 1 --2- 172.21.15.190:0/1451649773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6284102ea0 0x7f6284198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:05:16.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.991+0000 7f6282d9d700 1 --2- 172.21.15.190:0/1451649773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6284101ca0 0x7f6284197d20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f627400c970 tx=0x7f627400cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:05:16.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.991+0000 7f626bfff700 1 -- 172.21.15.190:0/1451649773 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6274004220 con 0x7f6284101ca0 2024-03-11T19:05:16.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.991+0000 7f62891b3700 1 -- 172.21.15.190:0/1451649773 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628419fe50 con 0x7f6284101ca0 2024-03-11T19:05:16.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.991+0000 7f626bfff700 1 -- 172.21.15.190:0/1451649773 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6274007d50 con 0x7f6284101ca0 2024-03-11T19:05:16.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.991+0000 7f626bfff700 1 -- 172.21.15.190:0/1451649773 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6274003d00 con 0x7f6284101ca0 2024-03-11T19:05:16.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.991+0000 7f62891b3700 1 -- 172.21.15.190:0/1451649773 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62841a02f0 con 0x7f6284101ca0 2024-03-11T19:05:16.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.992+0000 7f62891b3700 1 -- 172.21.15.190:0/1451649773 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6284068930 con 0x7f6284101ca0 2024-03-11T19:05:16.997 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.994+0000 7f626bfff700 1 -- 172.21.15.190:0/1451649773 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f62740078a0 con 0x7f6284101ca0 2024-03-11T19:05:16.997 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.995+0000 7f626bfff700 1 --2- 172.21.15.190:0/1451649773 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f626c0a1ac0 0x7f626c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:05:16.997 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.995+0000 7f626bfff700 1 -- 172.21.15.190:0/1451649773 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5397+0+0 (secure 0 0 0) 0x7f62740cc960 con 0x7f6284101ca0 2024-03-11T19:05:16.997 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.995+0000 7f628259c700 1 --2- 172.21.15.190:0/1451649773 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f626c0a1ac0 0x7f626c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:05:16.998 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.995+0000 7f628259c700 1 --2- 172.21.15.190:0/1451649773 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f626c0a1ac0 0x7f626c0a3f80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6278007900 tx=0x7f6278008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:05:17.001 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:16.998+0000 7f626bfff700 1 -- 172.21.15.190:0/1451649773 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f62740966d0 con 0x7f6284101ca0 2024-03-11T19:05:17.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:17.195+0000 7f62891b3700 1 -- 172.21.15.190:0/1451649773 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f628402d990 con 0x7f626c0a1ac0 2024-03-11T19:05:17.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:17 smithi029 ceph-mon[32439]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:17.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:17 smithi029 ceph-mon[32439]: Detected new or changed devices on smithi170 2024-03-11T19:05:17.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:17 smithi029 ceph-mon[32439]: Adjusting osd_memory_target on smithi170 to 5696M 2024-03-11T19:05:17.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:17 smithi029 ceph-mon[32439]: osdmap e55: 8 total, 8 up, 8 in 2024-03-11T19:05:17.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:17 smithi029 ceph-mon[32439]: osdmap e56: 8 total, 8 up, 8 in 2024-03-11T19:05:17.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:17 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:05:17.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:17 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:05:17.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:17 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:17.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:17 smithi190 ceph-mon[36292]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:17.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:17 smithi190 ceph-mon[36292]: Detected new or changed devices on smithi170 2024-03-11T19:05:17.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:17 smithi190 ceph-mon[36292]: Adjusting osd_memory_target on smithi170 to 5696M 2024-03-11T19:05:17.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:17 smithi190 ceph-mon[36292]: osdmap e55: 8 total, 8 up, 8 in 2024-03-11T19:05:17.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:17 smithi190 ceph-mon[36292]: osdmap e56: 8 total, 8 up, 8 in 2024-03-11T19:05:17.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:17 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:05:17.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:17 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:05:17.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:17 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:17.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:17 smithi170 ceph-mon[35690]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:17.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:17 smithi170 ceph-mon[35690]: Detected new or changed devices on smithi170 2024-03-11T19:05:17.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:17 smithi170 ceph-mon[35690]: Adjusting osd_memory_target on smithi170 to 5696M 2024-03-11T19:05:17.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:17 smithi170 ceph-mon[35690]: osdmap e55: 8 total, 8 up, 8 in 2024-03-11T19:05:17.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:17 smithi170 ceph-mon[35690]: osdmap e56: 8 total, 8 up, 8 in 2024-03-11T19:05:17.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:17 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:05:17.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:17 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:05:17.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:17 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:18.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:18 smithi029 ceph-mon[32439]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:05:18.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:18 smithi029 ceph-mon[32439]: osdmap e57: 8 total, 8 up, 8 in 2024-03-11T19:05:18.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:18 smithi190 ceph-mon[36292]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:05:18.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:18 smithi190 ceph-mon[36292]: osdmap e57: 8 total, 8 up, 8 in 2024-03-11T19:05:18.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:18 smithi170 ceph-mon[35690]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:05:18.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:18 smithi170 ceph-mon[35690]: osdmap e57: 8 total, 8 up, 8 in 2024-03-11T19:05:19.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:19 smithi029 ceph-mon[32439]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:19.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:19 smithi190 ceph-mon[36292]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:19.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:19 smithi170 ceph-mon[35690]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:20.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:20 smithi029 ceph-mon[32439]: from='client.? 172.21.15.190:0/1503673387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d27edd6-a75d-4b20-ab1b-dfa99909419d"}]: dispatch 2024-03-11T19:05:20.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:20 smithi029 ceph-mon[32439]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d27edd6-a75d-4b20-ab1b-dfa99909419d"}]: dispatch 2024-03-11T19:05:20.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:20 smithi029 ceph-mon[32439]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d27edd6-a75d-4b20-ab1b-dfa99909419d"}]': finished 2024-03-11T19:05:20.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:20 smithi029 ceph-mon[32439]: osdmap e58: 9 total, 8 up, 9 in 2024-03-11T19:05:20.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:05:20.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:20 smithi190 ceph-mon[36292]: from='client.? 172.21.15.190:0/1503673387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d27edd6-a75d-4b20-ab1b-dfa99909419d"}]: dispatch 2024-03-11T19:05:20.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:20 smithi190 ceph-mon[36292]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d27edd6-a75d-4b20-ab1b-dfa99909419d"}]: dispatch 2024-03-11T19:05:20.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:20 smithi190 ceph-mon[36292]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d27edd6-a75d-4b20-ab1b-dfa99909419d"}]': finished 2024-03-11T19:05:20.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:20 smithi190 ceph-mon[36292]: osdmap e58: 9 total, 8 up, 9 in 2024-03-11T19:05:20.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:05:20.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:20 smithi170 ceph-mon[35690]: from='client.? 172.21.15.190:0/1503673387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d27edd6-a75d-4b20-ab1b-dfa99909419d"}]: dispatch 2024-03-11T19:05:20.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:20 smithi170 ceph-mon[35690]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d27edd6-a75d-4b20-ab1b-dfa99909419d"}]: dispatch 2024-03-11T19:05:20.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:20 smithi170 ceph-mon[35690]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d27edd6-a75d-4b20-ab1b-dfa99909419d"}]': finished 2024-03-11T19:05:20.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:20 smithi170 ceph-mon[35690]: osdmap e58: 9 total, 8 up, 9 in 2024-03-11T19:05:20.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:05:21.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:21 smithi029 ceph-mon[32439]: pgmap v185: 1 pgs: 1 unknown; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:21.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:21 smithi029 ceph-mon[32439]: from='client.? 172.21.15.190:0/857731617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:05:21.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:21 smithi190 ceph-mon[36292]: pgmap v185: 1 pgs: 1 unknown; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:21.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:21 smithi190 ceph-mon[36292]: from='client.? 172.21.15.190:0/857731617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:05:21.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:21 smithi170 ceph-mon[35690]: pgmap v185: 1 pgs: 1 unknown; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:21.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:21 smithi170 ceph-mon[35690]: from='client.? 172.21.15.190:0/857731617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:05:23.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:23 smithi029 ceph-mon[32439]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:23.663 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:23 smithi190 ceph-mon[36292]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:23.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:23 smithi170 ceph-mon[35690]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:25.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:25 smithi190 ceph-mon[36292]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:25.662 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:25 smithi170 ceph-mon[35690]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:25.741 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:25 smithi029 ceph-mon[32439]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:27.735 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:27 smithi190 ceph-mon[36292]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:27.735 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:27 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-11T19:05:27.735 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:27 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:27.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:27 smithi170 ceph-mon[35690]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:27.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:27 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-11T19:05:27.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:27 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:27.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:27 smithi029 ceph-mon[32439]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:27.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:27 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-11T19:05:27.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:27 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:28.709 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:28 smithi190 ceph-mon[36292]: Deploying daemon osd.8 on smithi190 2024-03-11T19:05:28.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:28 smithi170 ceph-mon[35690]: Deploying daemon osd.8 on smithi190 2024-03-11T19:05:28.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:28 smithi029 ceph-mon[32439]: Deploying daemon osd.8 on smithi190 2024-03-11T19:05:29.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:29 smithi190 ceph-mon[36292]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:29.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:29 smithi170 ceph-mon[35690]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:29.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:29 smithi029 ceph-mon[32439]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:31.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:31 smithi190 ceph-mon[36292]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:31.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:31 smithi170 ceph-mon[35690]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:31.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:31 smithi029 ceph-mon[32439]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:33.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:33 smithi190 ceph-mon[36292]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:33.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:33 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:33.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:33 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:33.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:33 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:05:33.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:33 smithi170 ceph-mon[35690]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:33.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:33 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:33.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:33 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:33.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:33 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:05:33.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:33 smithi029 ceph-mon[32439]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:33.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:33 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:33.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:33 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:33.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:33 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:05:35.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:35 smithi190 ceph-mon[36292]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:35.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:35 smithi170 ceph-mon[35690]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:35.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:35 smithi029 ceph-mon[32439]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:36.220 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:36.217+0000 7f626bfff700 1 -- 172.21.15.190:0/1451649773 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f628402d990 con 0x7f626c0a1ac0 2024-03-11T19:05:36.220 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 8 on host 'smithi190' 2024-03-11T19:05:36.222 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:36.219+0000 7f62891b3700 1 -- 172.21.15.190:0/1451649773 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f626c0a1ac0 msgr2=0x7f626c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:05:36.222 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:36.219+0000 7f62891b3700 1 --2- 172.21.15.190:0/1451649773 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f626c0a1ac0 0x7f626c0a3f80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6278007900 tx=0x7f6278008040 comp rx=0 tx=0).stop 2024-03-11T19:05:36.222 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:36.220+0000 7f62891b3700 1 -- 172.21.15.190:0/1451649773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6284101ca0 msgr2=0x7f6284197d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:05:36.223 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:36.220+0000 7f62891b3700 1 --2- 172.21.15.190:0/1451649773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6284101ca0 0x7f6284197d20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f627400c970 tx=0x7f627400cd30 comp rx=0 tx=0).stop 2024-03-11T19:05:36.223 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:36.220+0000 7f62891b3700 1 -- 172.21.15.190:0/1451649773 shutdown_connections 2024-03-11T19:05:36.223 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:36.220+0000 7f62891b3700 1 --2- 172.21.15.190:0/1451649773 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f626c0a1ac0 0x7f626c0a3f80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:05:36.223 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:36.220+0000 7f62891b3700 1 --2- 172.21.15.190:0/1451649773 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6284103840 0x7f628419f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:05:36.223 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:36.220+0000 7f62891b3700 1 --2- 172.21.15.190:0/1451649773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6284101ca0 0x7f6284197d20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:05:36.223 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:36.220+0000 7f62891b3700 1 --2- 172.21.15.190:0/1451649773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6284102ea0 0x7f6284198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:05:36.223 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:36.220+0000 7f62891b3700 1 -- 172.21.15.190:0/1451649773 >> 172.21.15.190:0/1451649773 conn(0x7f62840fd280 msgr2=0x7f6284106220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:05:36.223 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:36.220+0000 7f62891b3700 1 -- 172.21.15.190:0/1451649773 shutdown_connections 2024-03-11T19:05:36.223 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:05:36.220+0000 7f62891b3700 1 -- 172.21.15.190:0/1451649773 wait complete. 2024-03-11T19:05:36.878 DEBUG:teuthology.orchestra.run.smithi190:osd.8> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.8.service 2024-03-11T19:05:36.880 INFO:tasks.cephadm:Deploying osd.9 on smithi029 with /dev/vg_nvme/lv_1... 2024-03-11T19:05:36.880 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T19:05:37.132 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:05:37.442 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:37 smithi029 ceph-mon[32439]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:37.442 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:37 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:37.442 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:37 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:37.442 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:37 smithi029 ceph-mon[32439]: from='osd.8 [v2:172.21.15.190:6816/1474709011,v1:172.21.15.190:6817/1474709011]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-11T19:05:37.442 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:37 smithi029 ceph-mon[32439]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-11T19:05:37.442 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:37 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:37.443 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:37 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:37.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:37 smithi190 ceph-mon[36292]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:37.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:37 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:37.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:37 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:37.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:37 smithi190 ceph-mon[36292]: from='osd.8 [v2:172.21.15.190:6816/1474709011,v1:172.21.15.190:6817/1474709011]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-11T19:05:37.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:37 smithi190 ceph-mon[36292]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-11T19:05:37.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:37 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:37.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:37 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:37.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:37 smithi170 ceph-mon[35690]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:37.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:37 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:37.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:37 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:37.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:37 smithi170 ceph-mon[35690]: from='osd.8 [v2:172.21.15.190:6816/1474709011,v1:172.21.15.190:6817/1474709011]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-11T19:05:37.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:37 smithi170 ceph-mon[35690]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-11T19:05:37.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:37 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:37.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:37 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:38.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:38 smithi029 ceph-mon[32439]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-11T19:05:38.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:38 smithi029 ceph-mon[32439]: osdmap e59: 9 total, 8 up, 9 in 2024-03-11T19:05:38.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:38 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:05:38.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:38 smithi029 ceph-mon[32439]: from='osd.8 [v2:172.21.15.190:6816/1474709011,v1:172.21.15.190:6817/1474709011]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:05:38.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:38 smithi029 ceph-mon[32439]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:05:38.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:38 smithi190 ceph-mon[36292]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-11T19:05:38.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:38 smithi190 ceph-mon[36292]: osdmap e59: 9 total, 8 up, 9 in 2024-03-11T19:05:38.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:38 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:05:38.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:38 smithi190 ceph-mon[36292]: from='osd.8 [v2:172.21.15.190:6816/1474709011,v1:172.21.15.190:6817/1474709011]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:05:38.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:38 smithi190 ceph-mon[36292]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:05:38.763 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:05:38 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:05:38.433+0000 7f4c344a0700 -1 osd.8 0 waiting for initial osdmap 2024-03-11T19:05:38.763 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:05:38 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:05:38.443+0000 7f4c30293700 -1 osd.8 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:05:38.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:38 smithi170 ceph-mon[35690]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-11T19:05:38.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:38 smithi170 ceph-mon[35690]: osdmap e59: 9 total, 8 up, 9 in 2024-03-11T19:05:38.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:38 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:05:38.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:38 smithi170 ceph-mon[35690]: from='osd.8 [v2:172.21.15.190:6816/1474709011,v1:172.21.15.190:6817/1474709011]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:05:38.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:38 smithi170 ceph-mon[35690]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:05:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T19:05:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T19:05:39.455 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-03-11T19:05:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-03-11T19:05:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00857174 s, 1.2 GB/s 2024-03-11T19:05:39.455 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 2024-03-11T19:05:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-03-11T19:05:39.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:39 smithi190 ceph-mon[36292]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:39.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:39 smithi190 ceph-mon[36292]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-11T19:05:39.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:39 smithi190 ceph-mon[36292]: osdmap e60: 9 total, 8 up, 9 in 2024-03-11T19:05:39.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:39 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:05:39.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:39 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:05:39.767 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:39 smithi029 ceph-mon[32439]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:39.767 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:39 smithi029 ceph-mon[32439]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-11T19:05:39.767 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:39 smithi029 ceph-mon[32439]: osdmap e60: 9 total, 8 up, 9 in 2024-03-11T19:05:39.767 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:39 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:05:39.767 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:39 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:05:39.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:39 smithi170 ceph-mon[35690]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:39.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:39 smithi170 ceph-mon[35690]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-11T19:05:39.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:39 smithi170 ceph-mon[35690]: osdmap e60: 9 total, 8 up, 9 in 2024-03-11T19:05:39.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:39 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:05:39.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:39 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:05:40.248 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_1 2024-03-11T19:05:40.496 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:05:40.618 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:40 smithi029 ceph-mon[32439]: purged_snaps scrub starts 2024-03-11T19:05:40.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:40 smithi029 ceph-mon[32439]: purged_snaps scrub ok 2024-03-11T19:05:40.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:05:40.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:40 smithi029 ceph-mon[32439]: osd.8 [v2:172.21.15.190:6816/1474709011,v1:172.21.15.190:6817/1474709011] boot 2024-03-11T19:05:40.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:40 smithi029 ceph-mon[32439]: osdmap e61: 9 total, 9 up, 9 in 2024-03-11T19:05:40.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:05:40.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:40 smithi190 ceph-mon[36292]: purged_snaps scrub starts 2024-03-11T19:05:40.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:40 smithi190 ceph-mon[36292]: purged_snaps scrub ok 2024-03-11T19:05:40.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:40 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:05:40.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:40 smithi190 ceph-mon[36292]: osd.8 [v2:172.21.15.190:6816/1474709011,v1:172.21.15.190:6817/1474709011] boot 2024-03-11T19:05:40.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:40 smithi190 ceph-mon[36292]: osdmap e61: 9 total, 9 up, 9 in 2024-03-11T19:05:40.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:40 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:05:40.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:40 smithi170 ceph-mon[35690]: purged_snaps scrub starts 2024-03-11T19:05:40.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:40 smithi170 ceph-mon[35690]: purged_snaps scrub ok 2024-03-11T19:05:40.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:40 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:05:40.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:40 smithi170 ceph-mon[35690]: osd.8 [v2:172.21.15.190:6816/1474709011,v1:172.21.15.190:6817/1474709011] boot 2024-03-11T19:05:40.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:40 smithi170 ceph-mon[35690]: osdmap e61: 9 total, 9 up, 9 in 2024-03-11T19:05:40.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:40 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:05:41.652 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:41 smithi029 ceph-mon[32439]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:41.652 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:41 smithi029 ceph-mon[32439]: osdmap e62: 9 total, 9 up, 9 in 2024-03-11T19:05:41.652 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:41 smithi029 ceph-mon[32439]: Detected new or changed devices on smithi190 2024-03-11T19:05:41.652 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:41 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:41.652 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:41 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:41.652 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:41 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:05:41.653 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:41 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:05:41.653 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:41 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:05:41.653 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:41 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:41.653 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:41 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:41.653 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:41 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:05:41.653 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:41 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:41.653 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:41 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:05:41.653 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:41 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:41.653 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:41 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:05:41.653 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:41 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:41.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:41 smithi190 ceph-mon[36292]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:41.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:41 smithi190 ceph-mon[36292]: osdmap e62: 9 total, 9 up, 9 in 2024-03-11T19:05:41.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:41 smithi190 ceph-mon[36292]: Detected new or changed devices on smithi190 2024-03-11T19:05:41.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:41 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:41.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:41 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:41.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:41 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:05:41.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:41 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:05:41.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:41 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:05:41.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:41 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:41.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:41 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:41.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:41 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:05:41.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:41 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:41.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:41 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:05:41.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:41 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:41.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:41 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:05:41.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:41 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:41.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:41 smithi170 ceph-mon[35690]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:05:41.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:41 smithi170 ceph-mon[35690]: osdmap e62: 9 total, 9 up, 9 in 2024-03-11T19:05:41.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:41 smithi170 ceph-mon[35690]: Detected new or changed devices on smithi190 2024-03-11T19:05:41.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:41 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:41.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:41 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:41.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:41 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:05:41.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:41 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:05:41.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:41 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:05:41.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:41 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:41.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:41 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:41.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:41 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:05:41.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:41 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:41.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:41 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:05:41.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:41 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:41.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:41 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:05:41.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:41 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.172+0000 7f7ba40da700 1 -- 172.21.15.29:0/2847356121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b9c102ea0 msgr2=0x7f7b9c103300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:05:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.172+0000 7f7ba40da700 1 --2- 172.21.15.29:0/2847356121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b9c102ea0 0x7f7b9c103300 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c0099c0 tx=0x7f7b8c009cd0 comp rx=0 tx=0).stop 2024-03-11T19:05:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.172+0000 7f7ba40da700 1 -- 172.21.15.29:0/2847356121 shutdown_connections 2024-03-11T19:05:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.172+0000 7f7ba40da700 1 --2- 172.21.15.29:0/2847356121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b9c103840 0x7f7b9c10a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:05:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.172+0000 7f7ba40da700 1 --2- 172.21.15.29:0/2847356121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b9c101ca0 0x7f7b9c1020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:05:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.172+0000 7f7ba40da700 1 --2- 172.21.15.29:0/2847356121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b9c102ea0 0x7f7b9c103300 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:05:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.172+0000 7f7ba40da700 1 -- 172.21.15.29:0/2847356121 >> 172.21.15.29:0/2847356121 conn(0x7f7b9c0fd280 msgr2=0x7f7b9c0ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:05:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.172+0000 7f7ba40da700 1 -- 172.21.15.29:0/2847356121 shutdown_connections 2024-03-11T19:05:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.172+0000 7f7ba40da700 1 -- 172.21.15.29:0/2847356121 wait complete. 2024-03-11T19:05:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.173+0000 7f7ba40da700 1 Processor -- start 2024-03-11T19:05:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.173+0000 7f7ba40da700 1 -- start start 2024-03-11T19:05:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.173+0000 7f7ba40da700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b9c101ca0 0x7f7b9c075a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:05:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.173+0000 7f7ba40da700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b9c102ea0 0x7f7b9c074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:05:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.173+0000 7f7ba40da700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b9c103840 0x7f7b9c0745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:05:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.173+0000 7f7ba40da700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9c074d20 con 0x7f7b9c101ca0 2024-03-11T19:05:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.173+0000 7f7ba40da700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9c074e90 con 0x7f7b9c103840 2024-03-11T19:05:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.173+0000 7f7ba40da700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9c1a1b80 con 0x7f7b9c102ea0 2024-03-11T19:05:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.173+0000 7f7ba1e76700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b9c101ca0 0x7f7b9c075a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:05:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.173+0000 7f7ba1e76700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b9c101ca0 0x7f7b9c075a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33872/0 (socket says 172.21.15.29:33872) 2024-03-11T19:05:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.173+0000 7f7ba1e76700 1 -- 172.21.15.29:0/1933908821 learned_addr learned my addr 172.21.15.29:0/1933908821 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:05:42.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.174+0000 7f7ba1675700 1 --2- 172.21.15.29:0/1933908821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b9c102ea0 0x7f7b9c074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.174+0000 7f7ba2677700 1 --2- 172.21.15.29:0/1933908821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b9c103840 0x7f7b9c0745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.174+0000 7f7ba1e76700 1 -- 172.21.15.29:0/1933908821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b9c102ea0 msgr2=0x7f7b9c074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.174+0000 7f7ba1e76700 1 --2- 172.21.15.29:0/1933908821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b9c102ea0 0x7f7b9c074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.174+0000 7f7ba1e76700 1 -- 172.21.15.29:0/1933908821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b9c103840 msgr2=0x7f7b9c0745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.174+0000 7f7ba1e76700 1 --2- 172.21.15.29:0/1933908821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b9c103840 0x7f7b9c0745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.174+0000 7f7ba1e76700 1 -- 172.21.15.29:0/1933908821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8c0095f0 con 0x7f7b9c101ca0 2024-03-11T19:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.174+0000 7f7ba1e76700 1 --2- 172.21.15.29:0/1933908821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b9c101ca0 0x7f7b9c075a10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800d980 tx=0x7f7b9800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.174+0000 7f7b92ffd700 1 -- 172.21.15.29:0/1933908821 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b98004020 con 0x7f7b9c101ca0 2024-03-11T19:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.174+0000 7f7ba40da700 1 -- 172.21.15.29:0/1933908821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b9c1a1e60 con 0x7f7b9c101ca0 2024-03-11T19:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.174+0000 7f7b92ffd700 1 -- 172.21.15.29:0/1933908821 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7b9800f690 con 0x7f7b9c101ca0 2024-03-11T19:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.174+0000 7f7b92ffd700 1 -- 172.21.15.29:0/1933908821 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b980039e0 con 0x7f7b9c101ca0 2024-03-11T19:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.174+0000 7f7ba40da700 1 -- 172.21.15.29:0/1933908821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b9c1a2280 con 0x7f7b9c101ca0 2024-03-11T19:05:42.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.175+0000 7f7ba40da700 1 -- 172.21.15.29:0/1933908821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b9c109080 con 0x7f7b9c101ca0 2024-03-11T19:05:42.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.176+0000 7f7b92ffd700 1 -- 172.21.15.29:0/1933908821 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7b9800f800 con 0x7f7b9c101ca0 2024-03-11T19:05:42.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.177+0000 7f7b92ffd700 1 --2- 172.21.15.29:0/1933908821 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7b880a1ac0 0x7f7b880a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:05:42.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.177+0000 7f7ba1675700 1 --2- 172.21.15.29:0/1933908821 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7b880a1ac0 0x7f7b880a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:05:42.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.177+0000 7f7b92ffd700 1 -- 172.21.15.29:0/1933908821 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5829+0+0 (secure 0 0 0) 0x7f7b980cca30 con 0x7f7b9c101ca0 2024-03-11T19:05:42.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.177+0000 7f7ba1675700 1 --2- 172.21.15.29:0/1933908821 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7b880a1ac0 0x7f7b880a3f80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c009990 tx=0x7f7b8c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:05:42.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.181+0000 7f7b92ffd700 1 -- 172.21.15.29:0/1933908821 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7b98096550 con 0x7f7b9c101ca0 2024-03-11T19:05:42.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:05:42.391+0000 7f7ba40da700 1 -- 172.21.15.29:0/1933908821 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b9c02d8e0 con 0x7f7b880a1ac0 2024-03-11T19:05:42.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:42 smithi029 ceph-mon[32439]: Adjusting osd_memory_target on smithi190 to 5696M 2024-03-11T19:05:42.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:42 smithi029 ceph-mon[32439]: osdmap e63: 9 total, 9 up, 9 in 2024-03-11T19:05:42.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:42 smithi029 ceph-mon[32439]: osdmap e64: 9 total, 9 up, 9 in 2024-03-11T19:05:42.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:42 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:05:42.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:42 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:05:42.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:42 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:42.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:42 smithi190 ceph-mon[36292]: Adjusting osd_memory_target on smithi190 to 5696M 2024-03-11T19:05:42.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:42 smithi190 ceph-mon[36292]: osdmap e63: 9 total, 9 up, 9 in 2024-03-11T19:05:42.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:42 smithi190 ceph-mon[36292]: osdmap e64: 9 total, 9 up, 9 in 2024-03-11T19:05:42.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:42 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:05:42.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:42 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:05:42.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:42 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:42.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:42 smithi170 ceph-mon[35690]: Adjusting osd_memory_target on smithi190 to 5696M 2024-03-11T19:05:42.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:42 smithi170 ceph-mon[35690]: osdmap e63: 9 total, 9 up, 9 in 2024-03-11T19:05:42.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:42 smithi170 ceph-mon[35690]: osdmap e64: 9 total, 9 up, 9 in 2024-03-11T19:05:42.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:42 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:05:42.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:42 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:05:42.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:42 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:43.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:43 smithi029 ceph-mon[32439]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:05:43.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:43 smithi029 ceph-mon[32439]: from='client.14412 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:05:43.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:43 smithi029 ceph-mon[32439]: osdmap e65: 9 total, 9 up, 9 in 2024-03-11T19:05:43.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:43 smithi190 ceph-mon[36292]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:05:43.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:43 smithi190 ceph-mon[36292]: from='client.14412 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:05:43.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:43 smithi190 ceph-mon[36292]: osdmap e65: 9 total, 9 up, 9 in 2024-03-11T19:05:43.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:43 smithi170 ceph-mon[35690]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:05:43.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:43 smithi170 ceph-mon[35690]: from='client.14412 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:05:43.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:43 smithi170 ceph-mon[35690]: osdmap e65: 9 total, 9 up, 9 in 2024-03-11T19:05:45.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:45 smithi190 ceph-mon[36292]: pgmap v205: 1 pgs: 1 remapped+peering; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:05:45.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:45 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2403683785' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e593e338-5c8b-4d2e-a990-39fd8c3eff43"}]: dispatch 2024-03-11T19:05:45.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:45 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2403683785' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e593e338-5c8b-4d2e-a990-39fd8c3eff43"}]': finished 2024-03-11T19:05:45.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:45 smithi190 ceph-mon[36292]: osdmap e66: 10 total, 9 up, 10 in 2024-03-11T19:05:45.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:45 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:05:45.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:45 smithi170 ceph-mon[35690]: pgmap v205: 1 pgs: 1 remapped+peering; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:05:45.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:45 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2403683785' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e593e338-5c8b-4d2e-a990-39fd8c3eff43"}]: dispatch 2024-03-11T19:05:45.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:45 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2403683785' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e593e338-5c8b-4d2e-a990-39fd8c3eff43"}]': finished 2024-03-11T19:05:45.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:45 smithi170 ceph-mon[35690]: osdmap e66: 10 total, 9 up, 10 in 2024-03-11T19:05:45.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:45 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:05:45.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:45 smithi029 ceph-mon[32439]: pgmap v205: 1 pgs: 1 remapped+peering; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:05:45.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:45 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2403683785' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e593e338-5c8b-4d2e-a990-39fd8c3eff43"}]: dispatch 2024-03-11T19:05:45.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:45 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2403683785' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e593e338-5c8b-4d2e-a990-39fd8c3eff43"}]': finished 2024-03-11T19:05:45.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:45 smithi029 ceph-mon[32439]: osdmap e66: 10 total, 9 up, 10 in 2024-03-11T19:05:45.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:45 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:05:46.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:46 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1733626721' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:05:46.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:46 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1733626721' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:05:46.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:46 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1733626721' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:05:47.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:47 smithi190 ceph-mon[36292]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-03-11T19:05:47.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:47 smithi170 ceph-mon[35690]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-03-11T19:05:47.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:47 smithi029 ceph-mon[32439]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-03-11T19:05:49.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:49 smithi029 ceph-mon[32439]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:05:49.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:49 smithi190 ceph-mon[36292]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:05:49.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:49 smithi170 ceph-mon[35690]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:05:51.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:51 smithi190 ceph-mon[36292]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-11T19:05:51.767 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:51 smithi029 ceph-mon[32439]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-11T19:05:51.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:51 smithi170 ceph-mon[35690]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-11T19:05:53.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:53 smithi190 ceph-mon[36292]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-11T19:05:53.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:53 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-11T19:05:53.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:53 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:53.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:53 smithi029 ceph-mon[32439]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-11T19:05:53.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:53 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-11T19:05:53.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:53 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:53.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:53 smithi170 ceph-mon[35690]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-11T19:05:53.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:53 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-11T19:05:53.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:53 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:05:54.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:54 smithi029 ceph-mon[32439]: Deploying daemon osd.9 on smithi029 2024-03-11T19:05:54.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:54 smithi190 ceph-mon[36292]: Deploying daemon osd.9 on smithi029 2024-03-11T19:05:54.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:54 smithi170 ceph-mon[35690]: Deploying daemon osd.9 on smithi029 2024-03-11T19:05:55.747 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:55 smithi170 ceph-mon[35690]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T19:05:55.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:55 smithi190 ceph-mon[36292]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T19:05:55.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:55 smithi029 ceph-mon[32439]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T19:05:57.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:57 smithi029 ceph-mon[32439]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-11T19:05:57.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:57 smithi190 ceph-mon[36292]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-11T19:05:57.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:57 smithi170 ceph-mon[35690]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-11T19:05:59.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:59 smithi029 ceph-mon[32439]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:05:59.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:59 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:59.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:59 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:59.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:05:59 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:05:59.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:59 smithi190 ceph-mon[36292]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:05:59.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:59 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:59.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:59 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:59.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:05:59 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:05:59.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:59 smithi170 ceph-mon[35690]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:05:59.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:59 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:59.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:59 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:05:59.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:05:59 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:06:01.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:01 smithi029 ceph-mon[32439]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:06:01.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:01 smithi190 ceph-mon[36292]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:06:01.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:01 smithi170 ceph-mon[35690]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:06:03.058 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 9 on host 'smithi029' 2024-03-11T19:06:03.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:03.054+0000 7f7b92ffd700 1 -- 172.21.15.29:0/1933908821 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7b9c02d8e0 con 0x7f7b880a1ac0 2024-03-11T19:06:03.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:03.057+0000 7f7ba40da700 1 -- 172.21.15.29:0/1933908821 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7b880a1ac0 msgr2=0x7f7b880a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:06:03.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:03.057+0000 7f7ba40da700 1 --2- 172.21.15.29:0/1933908821 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7b880a1ac0 0x7f7b880a3f80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c009990 tx=0x7f7b8c000bc0 comp rx=0 tx=0).stop 2024-03-11T19:06:03.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:03.057+0000 7f7ba40da700 1 -- 172.21.15.29:0/1933908821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b9c101ca0 msgr2=0x7f7b9c075a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:06:03.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:03.057+0000 7f7ba40da700 1 --2- 172.21.15.29:0/1933908821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b9c101ca0 0x7f7b9c075a10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800d980 tx=0x7f7b9800dc90 comp rx=0 tx=0).stop 2024-03-11T19:06:03.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:03.057+0000 7f7ba40da700 1 -- 172.21.15.29:0/1933908821 shutdown_connections 2024-03-11T19:06:03.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:03.057+0000 7f7ba40da700 1 --2- 172.21.15.29:0/1933908821 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7b880a1ac0 0x7f7b880a3f80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:03.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:03.057+0000 7f7ba40da700 1 --2- 172.21.15.29:0/1933908821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b9c103840 0x7f7b9c0745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:03.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:03.057+0000 7f7ba40da700 1 --2- 172.21.15.29:0/1933908821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b9c101ca0 0x7f7b9c075a10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:03.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:03.057+0000 7f7ba40da700 1 --2- 172.21.15.29:0/1933908821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b9c102ea0 0x7f7b9c074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:03.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:03.057+0000 7f7ba40da700 1 -- 172.21.15.29:0/1933908821 >> 172.21.15.29:0/1933908821 conn(0x7f7b9c0fd280 msgr2=0x7f7b9c1061c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:06:03.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:03.057+0000 7f7ba40da700 1 -- 172.21.15.29:0/1933908821 shutdown_connections 2024-03-11T19:06:03.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:03.057+0000 7f7ba40da700 1 -- 172.21.15.29:0/1933908821 wait complete. 2024-03-11T19:06:03.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:03 smithi170 ceph-mon[35690]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:06:03.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:03 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:03.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:03 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:03.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:03 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:03.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:03 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:06:03.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:03 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:03.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:03 smithi170 ceph-mon[35690]: from='osd.9 [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-11T19:06:03.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:03 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:03.347 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:03 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:03.347 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:03 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:06:03.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:03 smithi029 ceph-mon[32439]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:06:03.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:03 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:03.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:03 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:03.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:03 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:03.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:03 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:06:03.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:03 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:03.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:03 smithi029 ceph-mon[32439]: from='osd.9 [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-11T19:06:03.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:03 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:03.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:03 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:03.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:03 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:06:03.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:03 smithi190 ceph-mon[36292]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:06:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:03 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:03 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:03 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:03 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:06:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:03 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:03 smithi190 ceph-mon[36292]: from='osd.9 [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-11T19:06:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:03 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:03 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:03 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:06:03.971 DEBUG:teuthology.orchestra.run.smithi029:osd.9> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.9.service 2024-03-11T19:06:03.973 INFO:tasks.cephadm:Deploying osd.10 on smithi170 with /dev/vg_nvme/lv_1... 2024-03-11T19:06:03.974 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T19:06:04.217 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:06:04.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:04 smithi170 ceph-mon[35690]: from='osd.9 [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-11T19:06:04.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:04 smithi170 ceph-mon[35690]: osdmap e67: 10 total, 9 up, 10 in 2024-03-11T19:06:04.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:04 smithi170 ceph-mon[35690]: from='osd.9 [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-11T19:06:04.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:04 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:06:04.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:04 smithi190 ceph-mon[36292]: from='osd.9 [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-11T19:06:04.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:04 smithi190 ceph-mon[36292]: osdmap e67: 10 total, 9 up, 10 in 2024-03-11T19:06:04.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:04 smithi190 ceph-mon[36292]: from='osd.9 [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-11T19:06:04.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:04 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:06:04.822 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:04 smithi029 ceph-mon[32439]: from='osd.9 [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-11T19:06:04.822 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:04 smithi029 ceph-mon[32439]: osdmap e67: 10 total, 9 up, 10 in 2024-03-11T19:06:04.822 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:04 smithi029 ceph-mon[32439]: from='osd.9 [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-11T19:06:04.822 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:04 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:06:04.823 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:06:04 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:06:04.493+0000 7f1503f51700 -1 osd.9 0 waiting for initial osdmap 2024-03-11T19:06:04.823 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:06:04 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:06:04.511+0000 7f14ffd44700 -1 osd.9 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:06:05.679 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:05 smithi170 ceph-mon[35690]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:06:05.679 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:05 smithi170 ceph-mon[35690]: from='osd.9 [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-11T19:06:05.679 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:05 smithi170 ceph-mon[35690]: osdmap e68: 10 total, 9 up, 10 in 2024-03-11T19:06:05.680 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:05 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:06:05.680 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:05 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:06:05.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:05 smithi190 ceph-mon[36292]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:06:05.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:05 smithi190 ceph-mon[36292]: from='osd.9 [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-11T19:06:05.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:05 smithi190 ceph-mon[36292]: osdmap e68: 10 total, 9 up, 10 in 2024-03-11T19:06:05.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:05 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:06:05.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:05 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:06:05.851 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:05 smithi029 ceph-mon[32439]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:06:05.852 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:05 smithi029 ceph-mon[32439]: from='osd.9 [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-11T19:06:05.852 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:05 smithi029 ceph-mon[32439]: osdmap e68: 10 total, 9 up, 10 in 2024-03-11T19:06:05.852 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:05 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:06:05.852 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:05 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:06:06.226 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T19:06:06.227 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T19:06:06.227 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-03-11T19:06:06.227 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-03-11T19:06:06.227 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00822838 s, 1.3 GB/s 2024-03-11T19:06:06.227 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-03-11T19:06:06.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:06 smithi029 ceph-mon[32439]: purged_snaps scrub starts 2024-03-11T19:06:06.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:06 smithi029 ceph-mon[32439]: purged_snaps scrub ok 2024-03-11T19:06:06.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:06 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:06:06.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:06 smithi029 ceph-mon[32439]: osd.9 [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080] boot 2024-03-11T19:06:06.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:06 smithi029 ceph-mon[32439]: osdmap e69: 10 total, 10 up, 10 in 2024-03-11T19:06:06.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:06 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:06:06.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:06 smithi190 ceph-mon[36292]: purged_snaps scrub starts 2024-03-11T19:06:06.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:06 smithi190 ceph-mon[36292]: purged_snaps scrub ok 2024-03-11T19:06:06.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:06 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:06:06.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:06 smithi190 ceph-mon[36292]: osd.9 [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080] boot 2024-03-11T19:06:06.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:06 smithi190 ceph-mon[36292]: osdmap e69: 10 total, 10 up, 10 in 2024-03-11T19:06:06.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:06 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:06:06.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:06 smithi170 ceph-mon[35690]: purged_snaps scrub starts 2024-03-11T19:06:06.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:06 smithi170 ceph-mon[35690]: purged_snaps scrub ok 2024-03-11T19:06:06.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:06 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:06:06.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:06 smithi170 ceph-mon[35690]: osd.9 [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080] boot 2024-03-11T19:06:06.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:06 smithi170 ceph-mon[35690]: osdmap e69: 10 total, 10 up, 10 in 2024-03-11T19:06:06.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:06 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:06:06.931 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi170:vg_nvme/lv_1 2024-03-11T19:06:07.170 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:06:07.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:07 smithi170 ceph-mon[35690]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:06:07.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:07 smithi170 ceph-mon[35690]: osdmap e70: 10 total, 10 up, 10 in 2024-03-11T19:06:07.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:07.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:07.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:07.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:07.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:07.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:07.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:07.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:07.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:06:07.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:07.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:07 smithi029 ceph-mon[32439]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:06:07.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:07 smithi029 ceph-mon[32439]: osdmap e70: 10 total, 10 up, 10 in 2024-03-11T19:06:07.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:07.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:07.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:07.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:07.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:07.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:07.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:07.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:07.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:06:07.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:08.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:07 smithi190 ceph-mon[36292]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:06:08.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:07 smithi190 ceph-mon[36292]: osdmap e70: 10 total, 10 up, 10 in 2024-03-11T19:06:08.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:08.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:08.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:08.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:08.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:08.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:08.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:08.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:08.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:06:08.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:08.582 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:08 smithi170 ceph-mon[35690]: Detected new or changed devices on smithi029 2024-03-11T19:06:08.582 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:08 smithi170 ceph-mon[35690]: Adjusting osd_memory_target on smithi029 to 4273M 2024-03-11T19:06:08.834 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.832+0000 7f372f13c700 1 -- 172.21.15.170:0/2743708678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3728100bd0 msgr2=0x7f3728100ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:06:08.835 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.832+0000 7f372f13c700 1 --2- 172.21.15.170:0/2743708678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3728100bd0 0x7f3728100ff0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f371c0099c0 tx=0x7f371c009cd0 comp rx=0 tx=0).stop 2024-03-11T19:06:08.835 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.832+0000 7f372f13c700 1 -- 172.21.15.170:0/2743708678 shutdown_connections 2024-03-11T19:06:08.835 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.832+0000 7f372f13c700 1 --2- 172.21.15.170:0/2743708678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3728102770 0x7f3728109020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:08.835 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.832+0000 7f372f13c700 1 --2- 172.21.15.170:0/2743708678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3728100bd0 0x7f3728100ff0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:08.835 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.832+0000 7f372f13c700 1 --2- 172.21.15.170:0/2743708678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3728101dd0 0x7f3728102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:08.835 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.832+0000 7f372f13c700 1 -- 172.21.15.170:0/2743708678 >> 172.21.15.170:0/2743708678 conn(0x7f37280fc190 msgr2=0x7f37280fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:06:08.835 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.832+0000 7f372f13c700 1 -- 172.21.15.170:0/2743708678 shutdown_connections 2024-03-11T19:06:08.835 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.832+0000 7f372f13c700 1 -- 172.21.15.170:0/2743708678 wait complete. 2024-03-11T19:06:08.842 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.833+0000 7f372f13c700 1 Processor -- start 2024-03-11T19:06:08.842 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.833+0000 7f372f13c700 1 -- start start 2024-03-11T19:06:08.843 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.833+0000 7f372f13c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3728100bd0 0x7f3728197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:06:08.843 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.833+0000 7f372f13c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3728101dd0 0x7f37281980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:06:08.843 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.833+0000 7f372f13c700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3728102770 0x7f372819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:06:08.843 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.833+0000 7f372f13c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372819f6a0 con 0x7f3728100bd0 2024-03-11T19:06:08.843 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.833+0000 7f372f13c700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372819f7e0 con 0x7f3728102770 2024-03-11T19:06:08.843 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.833+0000 7f372f13c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372819f970 con 0x7f3728101dd0 2024-03-11T19:06:08.843 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.833+0000 7f372d6d9700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3728102770 0x7f372819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:06:08.843 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.833+0000 7f372d6d9700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3728102770 0x7f372819f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:43906/0 (socket says 172.21.15.170:43906) 2024-03-11T19:06:08.844 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.833+0000 7f372d6d9700 1 -- 172.21.15.170:0/2921139163 learned_addr learned my addr 172.21.15.170:0/2921139163 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-11T19:06:08.844 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.833+0000 7f372ced8700 1 --2- 172.21.15.170:0/2921139163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3728100bd0 0x7f3728197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:06:08.844 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.833+0000 7f3727fff700 1 --2- 172.21.15.170:0/2921139163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3728101dd0 0x7f37281980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:06:08.844 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.834+0000 7f372d6d9700 1 -- 172.21.15.170:0/2921139163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3728101dd0 msgr2=0x7f37281980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:06:08.844 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.834+0000 7f372d6d9700 1 --2- 172.21.15.170:0/2921139163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3728101dd0 0x7f37281980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:08.844 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.834+0000 7f372d6d9700 1 -- 172.21.15.170:0/2921139163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3728100bd0 msgr2=0x7f3728197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:06:08.844 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.834+0000 7f372d6d9700 1 --2- 172.21.15.170:0/2921139163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3728100bd0 0x7f3728197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:08.844 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.834+0000 7f372d6d9700 1 -- 172.21.15.170:0/2921139163 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371c0095f0 con 0x7f3728102770 2024-03-11T19:06:08.844 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.834+0000 7f372d6d9700 1 --2- 172.21.15.170:0/2921139163 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3728102770 0x7f372819f0d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3720007d70 tx=0x7f372000a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:06:08.845 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.834+0000 7f3725ffb700 1 -- 172.21.15.170:0/2921139163 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372000ac90 con 0x7f3728102770 2024-03-11T19:06:08.845 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.834+0000 7f3725ffb700 1 -- 172.21.15.170:0/2921139163 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f372000adf0 con 0x7f3728102770 2024-03-11T19:06:08.845 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.834+0000 7f3725ffb700 1 -- 172.21.15.170:0/2921139163 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37200115e0 con 0x7f3728102770 2024-03-11T19:06:08.845 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.834+0000 7f372f13c700 1 -- 172.21.15.170:0/2921139163 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f372819fc50 con 0x7f3728102770 2024-03-11T19:06:08.845 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.834+0000 7f372f13c700 1 -- 172.21.15.170:0/2921139163 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37281a0100 con 0x7f3728102770 2024-03-11T19:06:08.845 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.835+0000 7f372f13c700 1 -- 172.21.15.170:0/2921139163 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3728068960 con 0x7f3728102770 2024-03-11T19:06:08.845 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.836+0000 7f3725ffb700 1 -- 172.21.15.170:0/2921139163 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3720012420 con 0x7f3728102770 2024-03-11T19:06:08.845 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.837+0000 7f3725ffb700 1 --2- 172.21.15.170:0/2921139163 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f37180a1ac0 0x7f37180a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:06:08.845 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.837+0000 7f3725ffb700 1 -- 172.21.15.170:0/2921139163 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f37200ccda0 con 0x7f3728102770 2024-03-11T19:06:08.846 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.837+0000 7f372ced8700 1 --2- 172.21.15.170:0/2921139163 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f37180a1ac0 0x7f37180a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:06:08.846 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.838+0000 7f372ced8700 1 --2- 172.21.15.170:0/2921139163 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f37180a1ac0 0x7f37180a3f80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f371c006010 tx=0x7f371c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:06:08.846 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:08.841+0000 7f3725ffb700 1 -- 172.21.15.170:0/2921139163 <== mon.2 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f37200967b0 con 0x7f3728102770 2024-03-11T19:06:08.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:08 smithi029 ceph-mon[32439]: Detected new or changed devices on smithi029 2024-03-11T19:06:08.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:08 smithi029 ceph-mon[32439]: Adjusting osd_memory_target on smithi029 to 4273M 2024-03-11T19:06:09.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:08 smithi190 ceph-mon[36292]: Detected new or changed devices on smithi029 2024-03-11T19:06:09.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:08 smithi190 ceph-mon[36292]: Adjusting osd_memory_target on smithi029 to 4273M 2024-03-11T19:06:09.050 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:09.045+0000 7f372f13c700 1 -- 172.21.15.170:0/2921139163 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f37280fe350 con 0x7f37180a1ac0 2024-03-11T19:06:09.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:09 smithi170 ceph-mon[35690]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:09.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:09 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:06:09.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:09 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:06:09.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:09 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:09.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:09 smithi029 ceph-mon[32439]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:09.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:09 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:06:09.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:09 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:06:09.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:09 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:10.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:09 smithi190 ceph-mon[36292]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:10.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:09 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:06:10.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:09 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:06:10.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:09 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:10.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:10 smithi170 ceph-mon[35690]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:06:10.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:10 smithi029 ceph-mon[32439]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:06:11.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:10 smithi190 ceph-mon[36292]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:06:11.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:11 smithi170 ceph-mon[35690]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:11.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:11 smithi029 ceph-mon[32439]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:12.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:11 smithi190 ceph-mon[36292]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:12.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:12 smithi170 ceph-mon[35690]: from='client.? 172.21.15.170:0/192941944' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06dde09c-a014-4cac-aa88-352b63ff644b"}]: dispatch 2024-03-11T19:06:12.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:12 smithi170 ceph-mon[35690]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06dde09c-a014-4cac-aa88-352b63ff644b"}]: dispatch 2024-03-11T19:06:12.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:12 smithi170 ceph-mon[35690]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06dde09c-a014-4cac-aa88-352b63ff644b"}]': finished 2024-03-11T19:06:12.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:12 smithi170 ceph-mon[35690]: osdmap e71: 11 total, 10 up, 11 in 2024-03-11T19:06:12.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:12 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:06:12.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:12 smithi170 ceph-mon[35690]: from='client.? 172.21.15.170:0/2715980328' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:06:12.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:12 smithi029 ceph-mon[32439]: from='client.? 172.21.15.170:0/192941944' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06dde09c-a014-4cac-aa88-352b63ff644b"}]: dispatch 2024-03-11T19:06:12.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:12 smithi029 ceph-mon[32439]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06dde09c-a014-4cac-aa88-352b63ff644b"}]: dispatch 2024-03-11T19:06:12.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:12 smithi029 ceph-mon[32439]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06dde09c-a014-4cac-aa88-352b63ff644b"}]': finished 2024-03-11T19:06:12.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:12 smithi029 ceph-mon[32439]: osdmap e71: 11 total, 10 up, 11 in 2024-03-11T19:06:12.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:12 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:06:12.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:12 smithi029 ceph-mon[32439]: from='client.? 172.21.15.170:0/2715980328' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:06:13.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:12 smithi190 ceph-mon[36292]: from='client.? 172.21.15.170:0/192941944' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06dde09c-a014-4cac-aa88-352b63ff644b"}]: dispatch 2024-03-11T19:06:13.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:12 smithi190 ceph-mon[36292]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06dde09c-a014-4cac-aa88-352b63ff644b"}]: dispatch 2024-03-11T19:06:13.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:12 smithi190 ceph-mon[36292]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06dde09c-a014-4cac-aa88-352b63ff644b"}]': finished 2024-03-11T19:06:13.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:12 smithi190 ceph-mon[36292]: osdmap e71: 11 total, 10 up, 11 in 2024-03-11T19:06:13.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:12 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:06:13.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:12 smithi190 ceph-mon[36292]: from='client.? 172.21.15.170:0/2715980328' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:06:13.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:13 smithi170 ceph-mon[35690]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:13.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:13 smithi029 ceph-mon[32439]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:14.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:13 smithi190 ceph-mon[36292]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:15.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:15 smithi170 ceph-mon[35690]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:15 smithi029 ceph-mon[32439]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:16.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:15 smithi190 ceph-mon[36292]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:17.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:17 smithi170 ceph-mon[35690]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:17.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:17 smithi029 ceph-mon[32439]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:18.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:17 smithi190 ceph-mon[36292]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:19.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:19 smithi170 ceph-mon[35690]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:19.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:19 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-11T19:06:19.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:19 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:19.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:19 smithi029 ceph-mon[32439]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:19.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:19 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-11T19:06:19.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:19 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:20.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:19 smithi190 ceph-mon[36292]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:20.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:19 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-11T19:06:20.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:19 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:20.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:20 smithi170 ceph-mon[35690]: Deploying daemon osd.10 on smithi170 2024-03-11T19:06:20.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:20 smithi029 ceph-mon[32439]: Deploying daemon osd.10 on smithi170 2024-03-11T19:06:21.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:20 smithi190 ceph-mon[36292]: Deploying daemon osd.10 on smithi170 2024-03-11T19:06:21.723 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:21 smithi170 ceph-mon[35690]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:21.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:21 smithi029 ceph-mon[32439]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:22.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:21 smithi190 ceph-mon[36292]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:23.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:23 smithi170 ceph-mon[35690]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:23.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:23 smithi029 ceph-mon[32439]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:24.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:23 smithi190 ceph-mon[36292]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:25.814 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:25 smithi190 ceph-mon[36292]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:25.814 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:25 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:25.814 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:25 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:25.814 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:25 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:06:25.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:25 smithi170 ceph-mon[35690]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:25.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:25 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:25.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:25 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:25.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:25 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:06:25.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:25 smithi029 ceph-mon[32439]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:25.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:25 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:25.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:25 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:25.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:25 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:06:27.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:27 smithi170 ceph-mon[35690]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:27.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:27 smithi029 ceph-mon[32439]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:28.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:27 smithi190 ceph-mon[36292]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:29.218 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:29.214+0000 7f3725ffb700 1 -- 172.21.15.170:0/2921139163 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f37280fe350 con 0x7f37180a1ac0 2024-03-11T19:06:29.218 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 10 on host 'smithi170' 2024-03-11T19:06:29.220 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:29.217+0000 7f372f13c700 1 -- 172.21.15.170:0/2921139163 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f37180a1ac0 msgr2=0x7f37180a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:06:29.220 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:29.217+0000 7f372f13c700 1 --2- 172.21.15.170:0/2921139163 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f37180a1ac0 0x7f37180a3f80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f371c006010 tx=0x7f371c000bc0 comp rx=0 tx=0).stop 2024-03-11T19:06:29.220 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:29.217+0000 7f372f13c700 1 -- 172.21.15.170:0/2921139163 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3728102770 msgr2=0x7f372819f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:06:29.221 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:29.217+0000 7f372f13c700 1 --2- 172.21.15.170:0/2921139163 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3728102770 0x7f372819f0d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3720007d70 tx=0x7f372000a340 comp rx=0 tx=0).stop 2024-03-11T19:06:29.221 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:29.217+0000 7f372f13c700 1 -- 172.21.15.170:0/2921139163 shutdown_connections 2024-03-11T19:06:29.221 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:29.217+0000 7f372f13c700 1 --2- 172.21.15.170:0/2921139163 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f37180a1ac0 0x7f37180a3f80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:29.221 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:29.217+0000 7f372f13c700 1 --2- 172.21.15.170:0/2921139163 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3728102770 0x7f372819f0d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:29.221 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:29.217+0000 7f372f13c700 1 --2- 172.21.15.170:0/2921139163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3728100bd0 0x7f3728197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:29.221 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:29.217+0000 7f372f13c700 1 --2- 172.21.15.170:0/2921139163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3728101dd0 0x7f37281980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:29.221 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:29.217+0000 7f372f13c700 1 -- 172.21.15.170:0/2921139163 >> 172.21.15.170:0/2921139163 conn(0x7f37280fc190 msgr2=0x7f37280fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:06:29.221 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:29.217+0000 7f372f13c700 1 -- 172.21.15.170:0/2921139163 shutdown_connections 2024-03-11T19:06:29.221 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-11T19:06:29.217+0000 7f372f13c700 1 -- 172.21.15.170:0/2921139163 wait complete. 2024-03-11T19:06:29.536 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:29 smithi170 ceph-mon[35690]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:29.536 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:29.537 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:29.537 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:29.537 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:06:29.537 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:29.537 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:29 smithi170 ceph-mon[35690]: from='osd.10 [v2:172.21.15.170:6824/3766541425,v1:172.21.15.170:6825/3766541425]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-11T19:06:29.537 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:29 smithi170 ceph-mon[35690]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-11T19:06:29.537 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:29.537 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:29.537 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:06:29.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:29 smithi029 ceph-mon[32439]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:29.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:29.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:29.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:29.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:06:29.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:29.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:29 smithi029 ceph-mon[32439]: from='osd.10 [v2:172.21.15.170:6824/3766541425,v1:172.21.15.170:6825/3766541425]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-11T19:06:29.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:29 smithi029 ceph-mon[32439]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-11T19:06:29.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:29.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:29.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:06:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:29 smithi190 ceph-mon[36292]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:06:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:29 smithi190 ceph-mon[36292]: from='osd.10 [v2:172.21.15.170:6824/3766541425,v1:172.21.15.170:6825/3766541425]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-11T19:06:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:29 smithi190 ceph-mon[36292]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-11T19:06:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:29.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:29.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:06:29.863 DEBUG:teuthology.orchestra.run.smithi170:osd.10> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.10.service 2024-03-11T19:06:29.865 INFO:tasks.cephadm:Deploying osd.11 on smithi190 with /dev/vg_nvme/lv_1... 2024-03-11T19:06:29.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T19:06:30.113 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:06:30.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:30 smithi170 ceph-mon[35690]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-11T19:06:30.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:30 smithi170 ceph-mon[35690]: from='osd.10 [v2:172.21.15.170:6824/3766541425,v1:172.21.15.170:6825/3766541425]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:06:30.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:30 smithi170 ceph-mon[35690]: osdmap e72: 11 total, 10 up, 11 in 2024-03-11T19:06:30.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:30 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:06:30.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:30 smithi170 ceph-mon[35690]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:06:30.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:06:30 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:06:30.564+0000 7f6585a63700 -1 osd.10 0 waiting for initial osdmap 2024-03-11T19:06:30.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:06:30 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:06:30.577+0000 7f6581856700 -1 osd.10 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:06:30.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:30 smithi029 ceph-mon[32439]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-11T19:06:30.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:30 smithi029 ceph-mon[32439]: from='osd.10 [v2:172.21.15.170:6824/3766541425,v1:172.21.15.170:6825/3766541425]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:06:30.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:30 smithi029 ceph-mon[32439]: osdmap e72: 11 total, 10 up, 11 in 2024-03-11T19:06:30.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:30 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:06:30.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:30 smithi029 ceph-mon[32439]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:06:31.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:30 smithi190 ceph-mon[36292]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-11T19:06:31.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:30 smithi190 ceph-mon[36292]: from='osd.10 [v2:172.21.15.170:6824/3766541425,v1:172.21.15.170:6825/3766541425]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:06:31.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:30 smithi190 ceph-mon[36292]: osdmap e72: 11 total, 10 up, 11 in 2024-03-11T19:06:31.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:30 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:06:31.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:30 smithi190 ceph-mon[36292]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-11T19:06:31.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:31 smithi170 ceph-mon[35690]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:31.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:31 smithi170 ceph-mon[35690]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-11T19:06:31.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:31 smithi170 ceph-mon[35690]: osdmap e73: 11 total, 10 up, 11 in 2024-03-11T19:06:31.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:31 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:06:31.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:31 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:06:31.860 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:31 smithi190 ceph-mon[36292]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:31.860 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:31 smithi190 ceph-mon[36292]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-11T19:06:31.860 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:31 smithi190 ceph-mon[36292]: osdmap e73: 11 total, 10 up, 11 in 2024-03-11T19:06:31.861 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:31 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:06:31.861 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:31 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:06:31.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:31 smithi029 ceph-mon[32439]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:31.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:31 smithi029 ceph-mon[32439]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-11T19:06:31.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:31 smithi029 ceph-mon[32439]: osdmap e73: 11 total, 10 up, 11 in 2024-03-11T19:06:31.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:31 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:06:31.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:31 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:06:32.313 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T19:06:32.313 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T19:06:32.313 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-03-11T19:06:32.313 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-03-11T19:06:32.313 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113123 s, 927 MB/s 2024-03-11T19:06:32.313 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-03-11T19:06:32.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:32 smithi170 ceph-mon[35690]: purged_snaps scrub starts 2024-03-11T19:06:32.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:32 smithi170 ceph-mon[35690]: purged_snaps scrub ok 2024-03-11T19:06:32.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:32 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:06:32.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:32 smithi170 ceph-mon[35690]: osd.10 [v2:172.21.15.170:6824/3766541425,v1:172.21.15.170:6825/3766541425] boot 2024-03-11T19:06:32.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:32 smithi170 ceph-mon[35690]: osdmap e74: 11 total, 11 up, 11 in 2024-03-11T19:06:32.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:32 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:06:32.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:32 smithi170 ceph-mon[35690]: osdmap e75: 11 total, 11 up, 11 in 2024-03-11T19:06:32.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:32 smithi029 ceph-mon[32439]: purged_snaps scrub starts 2024-03-11T19:06:32.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:32 smithi029 ceph-mon[32439]: purged_snaps scrub ok 2024-03-11T19:06:32.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:32 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:06:32.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:32 smithi029 ceph-mon[32439]: osd.10 [v2:172.21.15.170:6824/3766541425,v1:172.21.15.170:6825/3766541425] boot 2024-03-11T19:06:32.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:32 smithi029 ceph-mon[32439]: osdmap e74: 11 total, 11 up, 11 in 2024-03-11T19:06:32.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:32 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:06:32.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:32 smithi029 ceph-mon[32439]: osdmap e75: 11 total, 11 up, 11 in 2024-03-11T19:06:33.007 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:32 smithi190 ceph-mon[36292]: purged_snaps scrub starts 2024-03-11T19:06:33.007 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:32 smithi190 ceph-mon[36292]: purged_snaps scrub ok 2024-03-11T19:06:33.007 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:32 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:06:33.007 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:32 smithi190 ceph-mon[36292]: osd.10 [v2:172.21.15.170:6824/3766541425,v1:172.21.15.170:6825/3766541425] boot 2024-03-11T19:06:33.007 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:32 smithi190 ceph-mon[36292]: osdmap e74: 11 total, 11 up, 11 in 2024-03-11T19:06:33.007 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:32 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:06:33.007 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:32 smithi190 ceph-mon[36292]: osdmap e75: 11 total, 11 up, 11 in 2024-03-11T19:06:33.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi190:vg_nvme/lv_1 2024-03-11T19:06:33.285 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:06:33.661 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:33 smithi190 ceph-mon[36292]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:33.661 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:33 smithi190 ceph-mon[36292]: osdmap e76: 11 total, 11 up, 11 in 2024-03-11T19:06:33.661 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:33 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:33.661 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:33 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:33.661 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:33 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:33.661 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:33 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:33.661 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:33 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:33.661 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:33 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:33.661 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:33 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:33.661 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:33 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:33.661 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:33 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:06:33.661 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:33 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:33.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:33 smithi170 ceph-mon[35690]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:33.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:33 smithi170 ceph-mon[35690]: osdmap e76: 11 total, 11 up, 11 in 2024-03-11T19:06:33.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:33 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:33.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:33 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:33.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:33 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:33.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:33 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:33.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:33 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:33.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:33 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:33.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:33 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:33.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:33 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:33.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:33 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:06:33.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:33 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:33.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:33 smithi029 ceph-mon[32439]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:06:33.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:33 smithi029 ceph-mon[32439]: osdmap e76: 11 total, 11 up, 11 in 2024-03-11T19:06:33.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:33 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:33.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:33 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:33.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:33 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:33.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:33 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:33.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:33 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:33.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:33 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:06:33.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:33 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:33.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:33 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:33.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:33 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:06:33.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:33 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:34.741 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:34 smithi190 ceph-mon[36292]: Detected new or changed devices on smithi170 2024-03-11T19:06:34.741 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:34 smithi190 ceph-mon[36292]: Adjusting osd_memory_target on smithi170 to 4272M 2024-03-11T19:06:34.741 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:34 smithi190 ceph-mon[36292]: osdmap e77: 11 total, 11 up, 11 in 2024-03-11T19:06:34.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:34 smithi170 ceph-mon[35690]: Detected new or changed devices on smithi170 2024-03-11T19:06:34.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:34 smithi170 ceph-mon[35690]: Adjusting osd_memory_target on smithi170 to 4272M 2024-03-11T19:06:34.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:34 smithi170 ceph-mon[35690]: osdmap e77: 11 total, 11 up, 11 in 2024-03-11T19:06:34.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:34 smithi029 ceph-mon[32439]: Detected new or changed devices on smithi170 2024-03-11T19:06:34.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:34 smithi029 ceph-mon[32439]: Adjusting osd_memory_target on smithi170 to 4272M 2024-03-11T19:06:34.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:34 smithi029 ceph-mon[32439]: osdmap e77: 11 total, 11 up, 11 in 2024-03-11T19:06:34.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.967+0000 7f1e78da6700 1 -- 172.21.15.190:0/1866094430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e74102740 msgr2=0x7f1e74108ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:06:34.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.967+0000 7f1e78da6700 1 --2- 172.21.15.190:0/1866094430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e74102740 0x7f1e74108ff0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c00b0a0 tx=0x7f1e6c00b3b0 comp rx=0 tx=0).stop 2024-03-11T19:06:34.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.967+0000 7f1e78da6700 1 -- 172.21.15.190:0/1866094430 shutdown_connections 2024-03-11T19:06:34.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.967+0000 7f1e78da6700 1 --2- 172.21.15.190:0/1866094430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e74102740 0x7f1e74108ff0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:34.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.967+0000 7f1e78da6700 1 --2- 172.21.15.190:0/1866094430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e74100ba0 0x7f1e74100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:34.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.967+0000 7f1e78da6700 1 --2- 172.21.15.190:0/1866094430 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e74101da0 0x7f1e74102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:34.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.967+0000 7f1e78da6700 1 -- 172.21.15.190:0/1866094430 >> 172.21.15.190:0/1866094430 conn(0x7f1e740fc140 msgr2=0x7f1e740fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:06:34.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.967+0000 7f1e78da6700 1 -- 172.21.15.190:0/1866094430 shutdown_connections 2024-03-11T19:06:34.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.967+0000 7f1e78da6700 1 -- 172.21.15.190:0/1866094430 wait complete. 2024-03-11T19:06:34.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.968+0000 7f1e78da6700 1 Processor -- start 2024-03-11T19:06:34.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.968+0000 7f1e78da6700 1 -- start start 2024-03-11T19:06:34.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.968+0000 7f1e78da6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e74100ba0 0x7f1e7406a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:06:34.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.968+0000 7f1e78da6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e74101da0 0x7f1e7419c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:06:34.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.968+0000 7f1e78da6700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e74102740 0x7f1e7419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:06:34.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.968+0000 7f1e78da6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7419f7f0 con 0x7f1e74100ba0 2024-03-11T19:06:34.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.968+0000 7f1e78da6700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7419f960 con 0x7f1e74102740 2024-03-11T19:06:34.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.968+0000 7f1e78da6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7419fb20 con 0x7f1e74101da0 2024-03-11T19:06:34.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.968+0000 7f1e71d9b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e74101da0 0x7f1e7419c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:06:34.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.968+0000 7f1e71d9b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e74101da0 0x7f1e7419c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:52220/0 (socket says 172.21.15.190:52220) 2024-03-11T19:06:34.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.968+0000 7f1e71d9b700 1 -- 172.21.15.190:0/1106590247 learned_addr learned my addr 172.21.15.190:0/1106590247 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-11T19:06:34.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.969+0000 7f1e72d9d700 1 --2- 172.21.15.190:0/1106590247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e74102740 0x7f1e7419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:06:34.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.969+0000 7f1e7259c700 1 --2- 172.21.15.190:0/1106590247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e74100ba0 0x7f1e7406a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:06:34.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.969+0000 7f1e71d9b700 1 -- 172.21.15.190:0/1106590247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e74102740 msgr2=0x7f1e7419f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:06:34.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.969+0000 7f1e71d9b700 1 --2- 172.21.15.190:0/1106590247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e74102740 0x7f1e7419f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:34.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.969+0000 7f1e71d9b700 1 -- 172.21.15.190:0/1106590247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e74100ba0 msgr2=0x7f1e7406a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:06:34.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.969+0000 7f1e71d9b700 1 --2- 172.21.15.190:0/1106590247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e74100ba0 0x7f1e7406a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:34.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.969+0000 7f1e71d9b700 1 -- 172.21.15.190:0/1106590247 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c009d00 con 0x7f1e74101da0 2024-03-11T19:06:34.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.969+0000 7f1e7259c700 1 --2- 172.21.15.190:0/1106590247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e74100ba0 0x7f1e7406a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:06:34.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.969+0000 7f1e71d9b700 1 --2- 172.21.15.190:0/1106590247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e74101da0 0x7f1e7419c9c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800d950 tx=0x7f1e6800dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:06:34.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.969+0000 7f1e637fe700 1 -- 172.21.15.190:0/1106590247 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e68009b40 con 0x7f1e74101da0 2024-03-11T19:06:34.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.969+0000 7f1e78da6700 1 -- 172.21.15.190:0/1106590247 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e7419fe00 con 0x7f1e74101da0 2024-03-11T19:06:34.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.969+0000 7f1e637fe700 1 -- 172.21.15.190:0/1106590247 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1e68009ca0 con 0x7f1e74101da0 2024-03-11T19:06:34.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.969+0000 7f1e637fe700 1 -- 172.21.15.190:0/1106590247 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6800f620 con 0x7f1e74101da0 2024-03-11T19:06:34.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.969+0000 7f1e78da6700 1 -- 172.21.15.190:0/1106590247 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e741a02d0 con 0x7f1e74101da0 2024-03-11T19:06:34.975 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.970+0000 7f1e78da6700 1 -- 172.21.15.190:0/1106590247 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e74107fe0 con 0x7f1e74101da0 2024-03-11T19:06:34.975 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.971+0000 7f1e637fe700 1 -- 172.21.15.190:0/1106590247 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1e68010460 con 0x7f1e74101da0 2024-03-11T19:06:34.975 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.972+0000 7f1e637fe700 1 --2- 172.21.15.190:0/1106590247 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1e5c0a1ac0 0x7f1e5c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:06:34.975 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.972+0000 7f1e637fe700 1 -- 172.21.15.190:0/1106590247 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6694+0+0 (secure 0 0 0) 0x7f1e680cd120 con 0x7f1e74101da0 2024-03-11T19:06:34.975 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.972+0000 7f1e7259c700 1 --2- 172.21.15.190:0/1106590247 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1e5c0a1ac0 0x7f1e5c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:06:34.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.972+0000 7f1e7259c700 1 --2- 172.21.15.190:0/1106590247 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1e5c0a1ac0 0x7f1e5c0a3f80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1e64005fd0 tx=0x7f1e64005dc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:06:34.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:34.976+0000 7f1e637fe700 1 -- 172.21.15.190:0/1106590247 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1e68096980 con 0x7f1e74101da0 2024-03-11T19:06:35.182 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:35.179+0000 7f1e78da6700 1 -- 172.21.15.190:0/1106590247 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e7402d8e0 con 0x7f1e5c0a1ac0 2024-03-11T19:06:35.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:35 smithi170 ceph-mon[35690]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:35.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:35 smithi170 ceph-mon[35690]: osdmap e78: 11 total, 11 up, 11 in 2024-03-11T19:06:35.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:35 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:06:35.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:35 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:06:35.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:35 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:35.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:35 smithi029 ceph-mon[32439]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:35.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:35 smithi029 ceph-mon[32439]: osdmap e78: 11 total, 11 up, 11 in 2024-03-11T19:06:35.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:06:35.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:06:35.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:35.925 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:35 smithi190 ceph-mon[36292]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:35.925 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:35 smithi190 ceph-mon[36292]: osdmap e78: 11 total, 11 up, 11 in 2024-03-11T19:06:35.925 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:06:35.925 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:06:35.925 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:36.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:36 smithi170 ceph-mon[35690]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:06:36.895 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:36 smithi190 ceph-mon[36292]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:06:36.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:36 smithi029 ceph-mon[32439]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:06:37.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:37 smithi170 ceph-mon[35690]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T19:06:37.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:37 smithi029 ceph-mon[32439]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T19:06:38.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:37 smithi190 ceph-mon[36292]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T19:06:38.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:38 smithi190 ceph-mon[36292]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-11T19:06:38.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:38 smithi190 ceph-mon[36292]: from='client.? 172.21.15.190:0/952081148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0cf7ece9-e288-49e6-b833-e9935c3abfc7"}]: dispatch 2024-03-11T19:06:38.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:38 smithi190 ceph-mon[36292]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0cf7ece9-e288-49e6-b833-e9935c3abfc7"}]: dispatch 2024-03-11T19:06:38.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:38 smithi190 ceph-mon[36292]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0cf7ece9-e288-49e6-b833-e9935c3abfc7"}]': finished 2024-03-11T19:06:38.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:38 smithi190 ceph-mon[36292]: osdmap e79: 12 total, 11 up, 12 in 2024-03-11T19:06:38.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:38 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:06:38.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:38 smithi170 ceph-mon[35690]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-11T19:06:38.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:38 smithi170 ceph-mon[35690]: from='client.? 172.21.15.190:0/952081148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0cf7ece9-e288-49e6-b833-e9935c3abfc7"}]: dispatch 2024-03-11T19:06:38.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:38 smithi170 ceph-mon[35690]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0cf7ece9-e288-49e6-b833-e9935c3abfc7"}]: dispatch 2024-03-11T19:06:38.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:38 smithi170 ceph-mon[35690]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0cf7ece9-e288-49e6-b833-e9935c3abfc7"}]': finished 2024-03-11T19:06:38.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:38 smithi170 ceph-mon[35690]: osdmap e79: 12 total, 11 up, 12 in 2024-03-11T19:06:38.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:38 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:06:38.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:38 smithi029 ceph-mon[32439]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-11T19:06:38.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:38 smithi029 ceph-mon[32439]: from='client.? 172.21.15.190:0/952081148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0cf7ece9-e288-49e6-b833-e9935c3abfc7"}]: dispatch 2024-03-11T19:06:38.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:38 smithi029 ceph-mon[32439]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0cf7ece9-e288-49e6-b833-e9935c3abfc7"}]: dispatch 2024-03-11T19:06:38.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:38 smithi029 ceph-mon[32439]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0cf7ece9-e288-49e6-b833-e9935c3abfc7"}]': finished 2024-03-11T19:06:38.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:38 smithi029 ceph-mon[32439]: osdmap e79: 12 total, 11 up, 12 in 2024-03-11T19:06:38.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:38 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:06:39.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:39 smithi170 ceph-mon[35690]: from='client.? 172.21.15.190:0/2207832664' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:06:39.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:39 smithi029 ceph-mon[32439]: from='client.? 172.21.15.190:0/2207832664' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:06:40.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:39 smithi190 ceph-mon[36292]: from='client.? 172.21.15.190:0/2207832664' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:06:40.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:40 smithi170 ceph-mon[35690]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:06:40.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:40 smithi029 ceph-mon[32439]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:06:41.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:40 smithi190 ceph-mon[36292]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:06:42.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:42 smithi190 ceph-mon[36292]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-11T19:06:42.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:42 smithi029 ceph-mon[32439]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-11T19:06:43.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:42 smithi170 ceph-mon[35690]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-11T19:06:44.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:44 smithi190 ceph-mon[36292]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-11T19:06:44.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:44 smithi029 ceph-mon[32439]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-11T19:06:45.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:44 smithi170 ceph-mon[35690]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-11T19:06:46.617 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:46 smithi190 ceph-mon[36292]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:46.617 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:46 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-11T19:06:46.617 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:46 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:46.618 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:46 smithi190 ceph-mon[36292]: Deploying daemon osd.11 on smithi190 2024-03-11T19:06:46.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:46 smithi029 ceph-mon[32439]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:46.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:46 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-11T19:06:46.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:46 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:46.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:46 smithi029 ceph-mon[32439]: Deploying daemon osd.11 on smithi190 2024-03-11T19:06:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:46 smithi170 ceph-mon[35690]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-11T19:06:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:06:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:46 smithi170 ceph-mon[35690]: Deploying daemon osd.11 on smithi190 2024-03-11T19:06:48.654 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:48 smithi190 ceph-mon[36292]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:48.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:48 smithi029 ceph-mon[32439]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:49.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:48 smithi170 ceph-mon[35690]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:50.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:50 smithi190 ceph-mon[36292]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:50.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:50 smithi029 ceph-mon[32439]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:51.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:50 smithi170 ceph-mon[35690]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:52.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:52 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:52.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:52 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:52.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:52 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:06:52.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:52 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:52.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:52 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:52.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:52 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:06:52.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:52 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:52.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:52 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:52.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:52 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:06:53.727 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:53 smithi190 ceph-mon[36292]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:53.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:53 smithi170 ceph-mon[35690]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:53.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:53 smithi029 ceph-mon[32439]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:55.100 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:55.097+0000 7f1e637fe700 1 -- 172.21.15.190:0/1106590247 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f1e7402d8e0 con 0x7f1e5c0a1ac0 2024-03-11T19:06:55.100 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 11 on host 'smithi190' 2024-03-11T19:06:55.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:55.099+0000 7f1e78da6700 1 -- 172.21.15.190:0/1106590247 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1e5c0a1ac0 msgr2=0x7f1e5c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:06:55.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:55.099+0000 7f1e78da6700 1 --2- 172.21.15.190:0/1106590247 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1e5c0a1ac0 0x7f1e5c0a3f80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1e64005fd0 tx=0x7f1e64005dc0 comp rx=0 tx=0).stop 2024-03-11T19:06:55.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:55.099+0000 7f1e78da6700 1 -- 172.21.15.190:0/1106590247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e74101da0 msgr2=0x7f1e7419c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:06:55.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:55.099+0000 7f1e78da6700 1 --2- 172.21.15.190:0/1106590247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e74101da0 0x7f1e7419c9c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800d950 tx=0x7f1e6800dd10 comp rx=0 tx=0).stop 2024-03-11T19:06:55.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:55.099+0000 7f1e78da6700 1 -- 172.21.15.190:0/1106590247 shutdown_connections 2024-03-11T19:06:55.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:55.099+0000 7f1e78da6700 1 --2- 172.21.15.190:0/1106590247 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1e5c0a1ac0 0x7f1e5c0a3f80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:55.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:55.099+0000 7f1e78da6700 1 --2- 172.21.15.190:0/1106590247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e74102740 0x7f1e7419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:55.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:55.099+0000 7f1e78da6700 1 --2- 172.21.15.190:0/1106590247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e74100ba0 0x7f1e7406a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:55.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:55.099+0000 7f1e78da6700 1 --2- 172.21.15.190:0/1106590247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e74101da0 0x7f1e7419c9c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:55.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:55.099+0000 7f1e78da6700 1 -- 172.21.15.190:0/1106590247 >> 172.21.15.190:0/1106590247 conn(0x7f1e740fc140 msgr2=0x7f1e74105120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:06:55.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:55.099+0000 7f1e78da6700 1 -- 172.21.15.190:0/1106590247 shutdown_connections 2024-03-11T19:06:55.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:06:55.099+0000 7f1e78da6700 1 -- 172.21.15.190:0/1106590247 wait complete. 2024-03-11T19:06:55.708 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:55 smithi029 ceph-mon[32439]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:55.708 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:55 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:55.708 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:55 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:55.708 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:55 smithi029 ceph-mon[32439]: from='osd.11 [v2:172.21.15.190:6824/3926470159,v1:172.21.15.190:6825/3926470159]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-11T19:06:55.708 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:55 smithi029 ceph-mon[32439]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-11T19:06:55.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:55 smithi190 ceph-mon[36292]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:55.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:55 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:55.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:55 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:55.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:55 smithi190 ceph-mon[36292]: from='osd.11 [v2:172.21.15.190:6824/3926470159,v1:172.21.15.190:6825/3926470159]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-11T19:06:55.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:55 smithi190 ceph-mon[36292]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-11T19:06:55.823 DEBUG:teuthology.orchestra.run.smithi190:osd.11> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.11.service 2024-03-11T19:06:55.825 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-11T19:06:55.826 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T19:06:55.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:55 smithi170 ceph-mon[35690]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:55.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:55 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:55.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:55 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:55.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:55 smithi170 ceph-mon[35690]: from='osd.11 [v2:172.21.15.190:6824/3926470159,v1:172.21.15.190:6825/3926470159]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-11T19:06:55.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:55 smithi170 ceph-mon[35690]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-11T19:06:56.101 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:06:57.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:56 smithi029 ceph-mon[32439]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:57.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:56 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:57.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:56 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:57.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:56 smithi029 ceph-mon[32439]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-11T19:06:57.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:56 smithi029 ceph-mon[32439]: from='osd.11 [v2:172.21.15.190:6824/3926470159,v1:172.21.15.190:6825/3926470159]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:06:57.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:56 smithi029 ceph-mon[32439]: osdmap e80: 12 total, 11 up, 12 in 2024-03-11T19:06:57.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:56 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:06:57.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:56 smithi029 ceph-mon[32439]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:06:57.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:56 smithi190 ceph-mon[36292]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:57.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:56 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:57.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:56 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:57.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:56 smithi190 ceph-mon[36292]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-11T19:06:57.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:56 smithi190 ceph-mon[36292]: from='osd.11 [v2:172.21.15.190:6824/3926470159,v1:172.21.15.190:6825/3926470159]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:06:57.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:56 smithi190 ceph-mon[36292]: osdmap e80: 12 total, 11 up, 12 in 2024-03-11T19:06:57.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:56 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:06:57.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:56 smithi190 ceph-mon[36292]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:06:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:56 smithi170 ceph-mon[35690]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:56 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:56 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:06:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:56 smithi170 ceph-mon[35690]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-11T19:06:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:56 smithi170 ceph-mon[35690]: from='osd.11 [v2:172.21.15.190:6824/3926470159,v1:172.21.15.190:6825/3926470159]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:06:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:56 smithi170 ceph-mon[35690]: osdmap e80: 12 total, 11 up, 12 in 2024-03-11T19:06:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:56 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:06:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:56 smithi170 ceph-mon[35690]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-11T19:06:57.761 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:06:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:06:57.564+0000 7f05a440f700 -1 osd.11 0 waiting for initial osdmap 2024-03-11T19:06:57.761 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:06:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:06:57.583+0000 7f059c1fa700 -1 osd.11 81 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:06:57.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.880+0000 7fa106081700 1 -- 172.21.15.29:0/4010847709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa100101e40 msgr2=0x7fa1001022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:06:57.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.880+0000 7fa106081700 1 --2- 172.21.15.29:0/4010847709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa100101e40 0x7fa1001022a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa0f40099c0 tx=0x7fa0f4009cd0 comp rx=0 tx=0).stop 2024-03-11T19:06:57.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.880+0000 7fa106081700 1 -- 172.21.15.29:0/4010847709 shutdown_connections 2024-03-11T19:06:57.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.880+0000 7fa106081700 1 --2- 172.21.15.29:0/4010847709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1001027e0 0x7fa100109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.880+0000 7fa106081700 1 --2- 172.21.15.29:0/4010847709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa100100c40 0x7fa100101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.880+0000 7fa106081700 1 --2- 172.21.15.29:0/4010847709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa100101e40 0x7fa1001022a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.880+0000 7fa106081700 1 -- 172.21.15.29:0/4010847709 >> 172.21.15.29:0/4010847709 conn(0x7fa1000fc200 msgr2=0x7fa1000fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:06:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.880+0000 7fa106081700 1 -- 172.21.15.29:0/4010847709 shutdown_connections 2024-03-11T19:06:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.880+0000 7fa106081700 1 -- 172.21.15.29:0/4010847709 wait complete. 2024-03-11T19:06:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa106081700 1 Processor -- start 2024-03-11T19:06:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa106081700 1 -- start start 2024-03-11T19:06:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa106081700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa100100c40 0x7fa100197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:06:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa106081700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa100101e40 0x7fa100198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:06:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa106081700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1001027e0 0x7fa10019f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:06:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa106081700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10019f760 con 0x7fa100101e40 2024-03-11T19:06:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa106081700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10019f8a0 con 0x7fa1001027e0 2024-03-11T19:06:57.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa106081700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10019fa60 con 0x7fa100100c40 2024-03-11T19:06:57.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa0feffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa100101e40 0x7fa100198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:06:57.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa0ff7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa100100c40 0x7fa100197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:06:57.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa0feffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa100101e40 0x7fa100198150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36716/0 (socket says 172.21.15.29:36716) 2024-03-11T19:06:57.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa0feffd700 1 -- 172.21.15.29:0/2935378229 learned_addr learned my addr 172.21.15.29:0/2935378229 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:06:57.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa0fffff700 1 --2- 172.21.15.29:0/2935378229 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1001027e0 0x7fa10019f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:06:57.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa0ff7fe700 1 -- 172.21.15.29:0/2935378229 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1001027e0 msgr2=0x7fa10019f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:06:57.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa0ff7fe700 1 --2- 172.21.15.29:0/2935378229 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1001027e0 0x7fa10019f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:57.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa0ff7fe700 1 -- 172.21.15.29:0/2935378229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa100101e40 msgr2=0x7fa100198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:06:57.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa0ff7fe700 1 --2- 172.21.15.29:0/2935378229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa100101e40 0x7fa100198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:57.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa0ff7fe700 1 -- 172.21.15.29:0/2935378229 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f40095f0 con 0x7fa100100c40 2024-03-11T19:06:57.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa0feffd700 1 --2- 172.21.15.29:0/2935378229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa100101e40 0x7fa100198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:06:57.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.881+0000 7fa0fffff700 1 --2- 172.21.15.29:0/2935378229 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1001027e0 0x7fa10019f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:06:57.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.882+0000 7fa0ff7fe700 1 --2- 172.21.15.29:0/2935378229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa100100c40 0x7fa100197c10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa100101ca0 tx=0x7fa0f000b730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:06:57.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.882+0000 7fa0fcff9700 1 -- 172.21.15.29:0/2935378229 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0011830 con 0x7fa100100c40 2024-03-11T19:06:57.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.882+0000 7fa106081700 1 -- 172.21.15.29:0/2935378229 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa10019fd40 con 0x7fa100100c40 2024-03-11T19:06:57.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.882+0000 7fa0fcff9700 1 -- 172.21.15.29:0/2935378229 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa0f0011e40 con 0x7fa100100c40 2024-03-11T19:06:57.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.882+0000 7fa0fcff9700 1 -- 172.21.15.29:0/2935378229 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0010dd0 con 0x7fa100100c40 2024-03-11T19:06:57.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.882+0000 7fa106081700 1 -- 172.21.15.29:0/2935378229 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1001a0190 con 0x7fa100100c40 2024-03-11T19:06:57.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.883+0000 7fa106081700 1 -- 172.21.15.29:0/2935378229 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa10010c460 con 0x7fa100100c40 2024-03-11T19:06:57.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.884+0000 7fa0fcff9700 1 -- 172.21.15.29:0/2935378229 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fa0f0011990 con 0x7fa100100c40 2024-03-11T19:06:57.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.884+0000 7fa0fcff9700 1 --2- 172.21.15.29:0/2935378229 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa0e80a1ac0 0x7fa0e80a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:06:57.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.885+0000 7fa0feffd700 1 --2- 172.21.15.29:0/2935378229 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa0e80a1ac0 0x7fa0e80a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:06:57.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.885+0000 7fa0fcff9700 1 -- 172.21.15.29:0/2935378229 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6847+0+0 (secure 0 0 0) 0x7fa0f00cc890 con 0x7fa100100c40 2024-03-11T19:06:57.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.885+0000 7fa0feffd700 1 --2- 172.21.15.29:0/2935378229 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa0e80a1ac0 0x7fa0e80a3f80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4009fd0 tx=0x7fa0f40036f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:06:57.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:57.889+0000 7fa0fcff9700 1 -- 172.21.15.29:0/2935378229 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa0f0095fc0 con 0x7fa100100c40 2024-03-11T19:06:58.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:58.091+0000 7fa106081700 1 -- 172.21.15.29:0/2935378229 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fa10010c5f0 con 0x7fa100100c40 2024-03-11T19:06:58.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:58.092+0000 7fa0fcff9700 1 -- 172.21.15.29:0/2935378229 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) v1 ==== 74+0+133 (secure 0 0 0) 0x7fa0f0009280 con 0x7fa100100c40 2024-03-11T19:06:58.095 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:06:58.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:58.094+0000 7fa106081700 1 -- 172.21.15.29:0/2935378229 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa0e80a1ac0 msgr2=0x7fa0e80a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:06:58.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:58.094+0000 7fa106081700 1 --2- 172.21.15.29:0/2935378229 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa0e80a1ac0 0x7fa0e80a3f80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4009fd0 tx=0x7fa0f40036f0 comp rx=0 tx=0).stop 2024-03-11T19:06:58.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:58.094+0000 7fa106081700 1 -- 172.21.15.29:0/2935378229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa100100c40 msgr2=0x7fa100197c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:06:58.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:58.094+0000 7fa106081700 1 --2- 172.21.15.29:0/2935378229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa100100c40 0x7fa100197c10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa100101ca0 tx=0x7fa0f000b730 comp rx=0 tx=0).stop 2024-03-11T19:06:58.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:58.094+0000 7fa106081700 1 -- 172.21.15.29:0/2935378229 shutdown_connections 2024-03-11T19:06:58.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:58.094+0000 7fa106081700 1 --2- 172.21.15.29:0/2935378229 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa0e80a1ac0 0x7fa0e80a3f80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:58.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:58.094+0000 7fa106081700 1 --2- 172.21.15.29:0/2935378229 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1001027e0 0x7fa10019f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:58.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:58.094+0000 7fa106081700 1 --2- 172.21.15.29:0/2935378229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa100100c40 0x7fa100197c10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:58.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:58.094+0000 7fa106081700 1 --2- 172.21.15.29:0/2935378229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa100101e40 0x7fa100198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:06:58.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:58.094+0000 7fa106081700 1 -- 172.21.15.29:0/2935378229 >> 172.21.15.29:0/2935378229 conn(0x7fa1000fc200 msgr2=0x7fa1000fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:06:58.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:58.094+0000 7fa106081700 1 -- 172.21.15.29:0/2935378229 shutdown_connections 2024-03-11T19:06:58.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:06:58.094+0000 7fa106081700 1 -- 172.21.15.29:0/2935378229 wait complete. 2024-03-11T19:06:58.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:58 smithi170 ceph-mon[35690]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-11T19:06:58.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:58 smithi170 ceph-mon[35690]: osdmap e81: 12 total, 11 up, 12 in 2024-03-11T19:06:58.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:58 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:06:58.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:58 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:06:58.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:58 smithi170 ceph-mon[35690]: from='osd.11 ' entity='osd.11' 2024-03-11T19:06:58.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:58 smithi029 ceph-mon[32439]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-11T19:06:58.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:58 smithi029 ceph-mon[32439]: osdmap e81: 12 total, 11 up, 12 in 2024-03-11T19:06:58.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:58 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:06:58.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:58 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:06:58.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:58 smithi029 ceph-mon[32439]: from='osd.11 ' entity='osd.11' 2024-03-11T19:06:58.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:58 smithi190 ceph-mon[36292]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-11T19:06:58.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:58 smithi190 ceph-mon[36292]: osdmap e81: 12 total, 11 up, 12 in 2024-03-11T19:06:58.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:58 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:06:58.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:58 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:06:58.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:58 smithi190 ceph-mon[36292]: from='osd.11 ' entity='osd.11' 2024-03-11T19:06:58.811 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":81,"num_osds":12,"num_up_osds":11,"osd_up_since":1710183991,"num_in_osds":12,"osd_in_since":1710183998,"num_remapped_pgs":0} 2024-03-11T19:06:59.201 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:59 smithi190 ceph-mon[36292]: purged_snaps scrub starts 2024-03-11T19:06:59.201 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:59 smithi190 ceph-mon[36292]: purged_snaps scrub ok 2024-03-11T19:06:59.201 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:59 smithi190 ceph-mon[36292]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:59.201 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:59 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:06:59.202 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:59 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2935378229' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T19:06:59.202 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:59 smithi190 ceph-mon[36292]: osd.11 [v2:172.21.15.190:6824/3926470159,v1:172.21.15.190:6825/3926470159] boot 2024-03-11T19:06:59.202 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:59 smithi190 ceph-mon[36292]: osdmap e82: 12 total, 12 up, 12 in 2024-03-11T19:06:59.202 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:06:59 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:06:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:59 smithi170 ceph-mon[35690]: purged_snaps scrub starts 2024-03-11T19:06:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:59 smithi170 ceph-mon[35690]: purged_snaps scrub ok 2024-03-11T19:06:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:59 smithi170 ceph-mon[35690]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:59 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:06:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:59 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2935378229' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T19:06:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:59 smithi170 ceph-mon[35690]: osd.11 [v2:172.21.15.190:6824/3926470159,v1:172.21.15.190:6825/3926470159] boot 2024-03-11T19:06:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:59 smithi170 ceph-mon[35690]: osdmap e82: 12 total, 12 up, 12 in 2024-03-11T19:06:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:06:59 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:06:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:59 smithi029 ceph-mon[32439]: purged_snaps scrub starts 2024-03-11T19:06:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:59 smithi029 ceph-mon[32439]: purged_snaps scrub ok 2024-03-11T19:06:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:59 smithi029 ceph-mon[32439]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:06:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:59 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:06:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:59 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2935378229' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T19:06:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:59 smithi029 ceph-mon[32439]: osd.11 [v2:172.21.15.190:6824/3926470159,v1:172.21.15.190:6825/3926470159] boot 2024-03-11T19:06:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:59 smithi029 ceph-mon[32439]: osdmap e82: 12 total, 12 up, 12 in 2024-03-11T19:06:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:06:59 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:06:59.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T19:07:00.063 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:07:00.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:00 smithi170 ceph-mon[35690]: Detected new or changed devices on smithi190 2024-03-11T19:07:00.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:00.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:00.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:07:00.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:07:00.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:07:00.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:07:00.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:00.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:00.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:07:00.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:00.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:07:00.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:00 smithi170 ceph-mon[35690]: osdmap e83: 12 total, 12 up, 12 in 2024-03-11T19:07:00.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:00.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:07:00.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:00 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:00.903 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:00 smithi029 ceph-mon[32439]: Detected new or changed devices on smithi190 2024-03-11T19:07:00.903 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:00.903 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:00.903 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:07:00.903 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:07:00.903 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:07:00.904 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:07:00.904 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:00.904 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:00.904 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:07:00.904 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:00.904 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:07:00.904 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:00 smithi029 ceph-mon[32439]: osdmap e83: 12 total, 12 up, 12 in 2024-03-11T19:07:00.904 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:00.904 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:07:00.904 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:00 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:00 smithi190 ceph-mon[36292]: Detected new or changed devices on smithi190 2024-03-11T19:07:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:07:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:07:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:07:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:07:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:01.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:07:01.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:01.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:07:01.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:00 smithi190 ceph-mon[36292]: osdmap e83: 12 total, 12 up, 12 in 2024-03-11T19:07:01.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:01.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:07:01.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:00 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:01.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:01 smithi170 ceph-mon[35690]: Adjusting osd_memory_target on smithi190 to 4272M 2024-03-11T19:07:01.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:01 smithi170 ceph-mon[35690]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:01.856 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:01 smithi029 ceph-mon[32439]: Adjusting osd_memory_target on smithi190 to 4272M 2024-03-11T19:07:01.856 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:01 smithi029 ceph-mon[32439]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:01.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.858+0000 7f1b738c0700 1 -- 172.21.15.29:0/14929186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b6c102200 msgr2=0x7f1b6c102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:01.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.858+0000 7f1b738c0700 1 --2- 172.21.15.29:0/14929186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b6c102200 0x7f1b6c102620 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1b68009990 tx=0x7f1b68009ca0 comp rx=0 tx=0).stop 2024-03-11T19:07:01.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.858+0000 7f1b738c0700 1 -- 172.21.15.29:0/14929186 shutdown_connections 2024-03-11T19:07:01.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.858+0000 7f1b738c0700 1 --2- 172.21.15.29:0/14929186 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b6c101820 0x7f1b6c1090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:01.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.858+0000 7f1b738c0700 1 --2- 172.21.15.29:0/14929186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b6c102200 0x7f1b6c102620 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:01.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.858+0000 7f1b738c0700 1 --2- 172.21.15.29:0/14929186 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b6c100e60 0x7f1b6c1012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:01.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.858+0000 7f1b738c0700 1 -- 172.21.15.29:0/14929186 >> 172.21.15.29:0/14929186 conn(0x7f1b6c0fb910 msgr2=0x7f1b6c0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:01.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.859+0000 7f1b738c0700 1 -- 172.21.15.29:0/14929186 shutdown_connections 2024-03-11T19:07:01.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.859+0000 7f1b738c0700 1 -- 172.21.15.29:0/14929186 wait complete. 2024-03-11T19:07:01.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.859+0000 7f1b738c0700 1 Processor -- start 2024-03-11T19:07:01.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.859+0000 7f1b738c0700 1 -- start start 2024-03-11T19:07:01.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.859+0000 7f1b738c0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b6c100e60 0x7f1b6c10eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:01.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.859+0000 7f1b738c0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b6c101820 0x7f1b6c10d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:01.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.859+0000 7f1b738c0700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b6c102200 0x7f1b6c10d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:01.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.859+0000 7f1b738c0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c10de20 con 0x7f1b6c101820 2024-03-11T19:07:01.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.859+0000 7f1b738c0700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c10df90 con 0x7f1b6c102200 2024-03-11T19:07:01.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.859+0000 7f1b738c0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c1a1c50 con 0x7f1b6c100e60 2024-03-11T19:07:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.860+0000 7f1b70e5b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b6c101820 0x7f1b6c10d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.860+0000 7f1b70e5b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b6c101820 0x7f1b6c10d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36744/0 (socket says 172.21.15.29:36744) 2024-03-11T19:07:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.860+0000 7f1b70e5b700 1 -- 172.21.15.29:0/3478092368 learned_addr learned my addr 172.21.15.29:0/3478092368 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.860+0000 7f1b71e5d700 1 --2- 172.21.15.29:0/3478092368 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b6c102200 0x7f1b6c10d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.860+0000 7f1b7165c700 1 --2- 172.21.15.29:0/3478092368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b6c100e60 0x7f1b6c10eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.860+0000 7f1b70e5b700 1 -- 172.21.15.29:0/3478092368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b6c100e60 msgr2=0x7f1b6c10eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.860+0000 7f1b70e5b700 1 --2- 172.21.15.29:0/3478092368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b6c100e60 0x7f1b6c10eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.860+0000 7f1b70e5b700 1 -- 172.21.15.29:0/3478092368 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b6c102200 msgr2=0x7f1b6c10d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.860+0000 7f1b70e5b700 1 --2- 172.21.15.29:0/3478092368 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b6c102200 0x7f1b6c10d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.860+0000 7f1b70e5b700 1 -- 172.21.15.29:0/3478092368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b680095f0 con 0x7f1b6c101820 2024-03-11T19:07:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.860+0000 7f1b71e5d700 1 --2- 172.21.15.29:0/3478092368 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b6c102200 0x7f1b6c10d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:07:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.860+0000 7f1b70e5b700 1 --2- 172.21.15.29:0/3478092368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b6c101820 0x7f1b6c10d160 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c00d950 tx=0x7f1b5c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.860+0000 7f1b627fc700 1 -- 172.21.15.29:0/3478092368 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c009b40 con 0x7f1b6c101820 2024-03-11T19:07:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.860+0000 7f1b738c0700 1 -- 172.21.15.29:0/3478092368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b6c1a1f30 con 0x7f1b6c101820 2024-03-11T19:07:01.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.860+0000 7f1b627fc700 1 -- 172.21.15.29:0/3478092368 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1b5c009ca0 con 0x7f1b6c101820 2024-03-11T19:07:01.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.860+0000 7f1b627fc700 1 -- 172.21.15.29:0/3478092368 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c00f620 con 0x7f1b6c101820 2024-03-11T19:07:01.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.860+0000 7f1b738c0700 1 -- 172.21.15.29:0/3478092368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b6c1a23f0 con 0x7f1b6c101820 2024-03-11T19:07:01.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.861+0000 7f1b738c0700 1 -- 172.21.15.29:0/3478092368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b6c068930 con 0x7f1b6c101820 2024-03-11T19:07:01.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.862+0000 7f1b627fc700 1 -- 172.21.15.29:0/3478092368 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1b5c010460 con 0x7f1b6c101820 2024-03-11T19:07:01.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.863+0000 7f1b627fc700 1 --2- 172.21.15.29:0/3478092368 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1b580a1ac0 0x7f1b580a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:01.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.863+0000 7f1b7165c700 1 --2- 172.21.15.29:0/3478092368 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1b580a1ac0 0x7f1b580a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:01.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.863+0000 7f1b627fc700 1 -- 172.21.15.29:0/3478092368 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1b5c0cd340 con 0x7f1b6c101820 2024-03-11T19:07:01.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.863+0000 7f1b7165c700 1 --2- 172.21.15.29:0/3478092368 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1b580a1ac0 0x7f1b580a3f80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1b68003820 tx=0x7f1b68005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:01.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:01.867+0000 7f1b627fc700 1 -- 172.21.15.29:0/3478092368 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1b5c096950 con 0x7f1b6c101820 2024-03-11T19:07:02.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:01 smithi190 ceph-mon[36292]: Adjusting osd_memory_target on smithi190 to 4272M 2024-03-11T19:07:02.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:01 smithi190 ceph-mon[36292]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:02.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:02.070+0000 7f1b738c0700 1 -- 172.21.15.29:0/3478092368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f1b6c1a2850 con 0x7f1b6c101820 2024-03-11T19:07:02.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:02.070+0000 7f1b627fc700 1 -- 172.21.15.29:0/3478092368 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v83) v1 ==== 74+0+133 (secure 0 0 0) 0x7f1b5c09a010 con 0x7f1b6c101820 2024-03-11T19:07:02.073 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:02.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:02.072+0000 7f1b738c0700 1 -- 172.21.15.29:0/3478092368 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1b580a1ac0 msgr2=0x7f1b580a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:02.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:02.072+0000 7f1b738c0700 1 --2- 172.21.15.29:0/3478092368 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1b580a1ac0 0x7f1b580a3f80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1b68003820 tx=0x7f1b68005fb0 comp rx=0 tx=0).stop 2024-03-11T19:07:02.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:02.072+0000 7f1b738c0700 1 -- 172.21.15.29:0/3478092368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b6c101820 msgr2=0x7f1b6c10d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:02.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:02.072+0000 7f1b738c0700 1 --2- 172.21.15.29:0/3478092368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b6c101820 0x7f1b6c10d160 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c00d950 tx=0x7f1b5c00dd10 comp rx=0 tx=0).stop 2024-03-11T19:07:02.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:02.072+0000 7f1b738c0700 1 -- 172.21.15.29:0/3478092368 shutdown_connections 2024-03-11T19:07:02.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:02.072+0000 7f1b738c0700 1 --2- 172.21.15.29:0/3478092368 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1b580a1ac0 0x7f1b580a3f80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:02.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:02.072+0000 7f1b738c0700 1 --2- 172.21.15.29:0/3478092368 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b6c102200 0x7f1b6c10d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:02.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:02.072+0000 7f1b738c0700 1 --2- 172.21.15.29:0/3478092368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b6c100e60 0x7f1b6c10eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:02.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:02.072+0000 7f1b738c0700 1 --2- 172.21.15.29:0/3478092368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b6c101820 0x7f1b6c10d160 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:02.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:02.072+0000 7f1b738c0700 1 -- 172.21.15.29:0/3478092368 >> 172.21.15.29:0/3478092368 conn(0x7f1b6c0fb910 msgr2=0x7f1b6c0fbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:02.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:02.073+0000 7f1b738c0700 1 -- 172.21.15.29:0/3478092368 shutdown_connections 2024-03-11T19:07:02.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:02.073+0000 7f1b738c0700 1 -- 172.21.15.29:0/3478092368 wait complete. 2024-03-11T19:07:02.758 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:02 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3478092368' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T19:07:02.790 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":83,"num_osds":12,"num_up_osds":12,"osd_up_since":1710184018,"num_in_osds":12,"osd_in_since":1710183998,"num_remapped_pgs":0} 2024-03-11T19:07:02.790 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T19:07:02.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:02 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3478092368' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T19:07:03.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:02 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3478092368' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T19:07:03.036 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:07:03.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:03 smithi170 ceph-mon[35690]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:03.897 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:03 smithi029 ceph-mon[32439]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:04.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:03 smithi190 ceph-mon[36292]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.872+0000 7f4098f4e700 1 -- 172.21.15.29:0/2457809251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4094101510 msgr2=0x7f409410d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.872+0000 7f4098f4e700 1 --2- 172.21.15.29:0/2457809251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4094101510 0x7f409410d9b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f408c009990 tx=0x7f408c009ca0 comp rx=0 tx=0).stop 2024-03-11T19:07:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.872+0000 7f4098f4e700 1 -- 172.21.15.29:0/2457809251 shutdown_connections 2024-03-11T19:07:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.872+0000 7f4098f4e700 1 --2- 172.21.15.29:0/2457809251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4094101510 0x7f409410d9b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.872+0000 7f4098f4e700 1 --2- 172.21.15.29:0/2457809251 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4094105ab0 0x7f4094105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.872+0000 7f4098f4e700 1 --2- 172.21.15.29:0/2457809251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4094100b50 0x7f4094100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.872+0000 7f4098f4e700 1 -- 172.21.15.29:0/2457809251 >> 172.21.15.29:0/2457809251 conn(0x7f4094077b80 msgr2=0x7f4094077f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.872+0000 7f4098f4e700 1 -- 172.21.15.29:0/2457809251 shutdown_connections 2024-03-11T19:07:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.872+0000 7f4098f4e700 1 -- 172.21.15.29:0/2457809251 wait complete. 2024-03-11T19:07:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.873+0000 7f4098f4e700 1 Processor -- start 2024-03-11T19:07:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.873+0000 7f4098f4e700 1 -- start start 2024-03-11T19:07:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.873+0000 7f4098f4e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4094100b50 0x7f4094198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.873+0000 7f4098f4e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4094101510 0x7f40941985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.873+0000 7f4098f4e700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4094105ab0 0x7f409419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.873+0000 7f4098f4e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409419cf10 con 0x7f4094100b50 2024-03-11T19:07:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.873+0000 7f4098f4e700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409419d080 con 0x7f4094105ab0 2024-03-11T19:07:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.873+0000 7f4098f4e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409419d240 con 0x7f4094101510 2024-03-11T19:07:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.873+0000 7f409259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4094100b50 0x7f4094198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.873+0000 7f409259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4094100b50 0x7f4094198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36760/0 (socket says 172.21.15.29:36760) 2024-03-11T19:07:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.873+0000 7f409259c700 1 -- 172.21.15.29:0/992473234 learned_addr learned my addr 172.21.15.29:0/992473234 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.873+0000 7f4092d9d700 1 --2- 172.21.15.29:0/992473234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4094105ab0 0x7f409419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.873+0000 7f409259c700 1 -- 172.21.15.29:0/992473234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4094101510 msgr2=0x7f40941985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.873+0000 7f4091d9b700 1 --2- 172.21.15.29:0/992473234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4094101510 0x7f40941985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.873+0000 7f409259c700 1 --2- 172.21.15.29:0/992473234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4094101510 0x7f40941985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.873+0000 7f409259c700 1 -- 172.21.15.29:0/992473234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4094105ab0 msgr2=0x7f409419c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.873+0000 7f409259c700 1 --2- 172.21.15.29:0/992473234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4094105ab0 0x7f409419c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.873+0000 7f409259c700 1 -- 172.21.15.29:0/992473234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c0095f0 con 0x7f4094100b50 2024-03-11T19:07:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.873+0000 7f4092d9d700 1 --2- 172.21.15.29:0/992473234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4094105ab0 0x7f409419c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:07:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.874+0000 7f409259c700 1 --2- 172.21.15.29:0/992473234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4094100b50 0x7f4094198080 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4084007e20 tx=0x7f408400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.874+0000 7f408b7fe700 1 -- 172.21.15.29:0/992473234 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408400ae00 con 0x7f4094100b50 2024-03-11T19:07:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.874+0000 7f4098f4e700 1 -- 172.21.15.29:0/992473234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f409419d520 con 0x7f4094100b50 2024-03-11T19:07:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.874+0000 7f408b7fe700 1 -- 172.21.15.29:0/992473234 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f40840128d0 con 0x7f4094100b50 2024-03-11T19:07:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.874+0000 7f408b7fe700 1 -- 172.21.15.29:0/992473234 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4084011840 con 0x7f4094100b50 2024-03-11T19:07:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.874+0000 7f4098f4e700 1 -- 172.21.15.29:0/992473234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40941a4d80 con 0x7f4094100b50 2024-03-11T19:07:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.875+0000 7f4098f4e700 1 -- 172.21.15.29:0/992473234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f409410a0a0 con 0x7f4094100b50 2024-03-11T19:07:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.876+0000 7f408b7fe700 1 -- 172.21.15.29:0/992473234 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4084012a40 con 0x7f4094100b50 2024-03-11T19:07:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.876+0000 7f408b7fe700 1 --2- 172.21.15.29:0/992473234 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f40800a1ac0 0x7f40800a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.877+0000 7f4091d9b700 1 --2- 172.21.15.29:0/992473234 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f40800a1ac0 0x7f40800a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:04.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.877+0000 7f408b7fe700 1 -- 172.21.15.29:0/992473234 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f40840cd510 con 0x7f4094100b50 2024-03-11T19:07:04.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.877+0000 7f4091d9b700 1 --2- 172.21.15.29:0/992473234 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f40800a1ac0 0x7f40800a3f80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f407c005950 tx=0x7f407c0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:04.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:04.880+0000 7f408b7fe700 1 -- 172.21.15.29:0/992473234 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4084096b20 con 0x7f4094100b50 2024-03-11T19:07:05.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:05.086+0000 7f4098f4e700 1 -- 172.21.15.29:0/992473234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4094199390 con 0x7f4094100b50 2024-03-11T19:07:05.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:05.087+0000 7f408b7fe700 1 -- 172.21.15.29:0/992473234 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+18456 (secure 0 0 0) 0x7f408409a1e0 con 0x7f4094100b50 2024-03-11T19:07:05.090 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:05.090 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":83,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:07:00.078122+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:07:05.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:05.089+0000 7f4098f4e700 1 -- 172.21.15.29:0/992473234 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f40800a1ac0 msgr2=0x7f40800a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:05.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:05.089+0000 7f4098f4e700 1 --2- 172.21.15.29:0/992473234 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f40800a1ac0 0x7f40800a3f80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f407c005950 tx=0x7f407c0058e0 comp rx=0 tx=0).stop 2024-03-11T19:07:05.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:05.089+0000 7f4098f4e700 1 -- 172.21.15.29:0/992473234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4094100b50 msgr2=0x7f4094198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:05.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:05.089+0000 7f4098f4e700 1 --2- 172.21.15.29:0/992473234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4094100b50 0x7f4094198080 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4084007e20 tx=0x7f408400a340 comp rx=0 tx=0).stop 2024-03-11T19:07:05.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:05.089+0000 7f4098f4e700 1 -- 172.21.15.29:0/992473234 shutdown_connections 2024-03-11T19:07:05.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:05.089+0000 7f4098f4e700 1 --2- 172.21.15.29:0/992473234 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f40800a1ac0 0x7f40800a3f80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:05.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:05.089+0000 7f4098f4e700 1 --2- 172.21.15.29:0/992473234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4094105ab0 0x7f409419c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:05.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:05.089+0000 7f4098f4e700 1 --2- 172.21.15.29:0/992473234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4094100b50 0x7f4094198080 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:05.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:05.089+0000 7f4098f4e700 1 --2- 172.21.15.29:0/992473234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4094101510 0x7f40941985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:05.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:05.089+0000 7f4098f4e700 1 -- 172.21.15.29:0/992473234 >> 172.21.15.29:0/992473234 conn(0x7f4094077b80 msgr2=0x7f409406a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:05.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:05.089+0000 7f4098f4e700 1 -- 172.21.15.29:0/992473234 shutdown_connections 2024-03-11T19:07:05.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:05.089+0000 7f4098f4e700 1 -- 172.21.15.29:0/992473234 wait complete. 2024-03-11T19:07:05.809 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:05 smithi029 ceph-mon[32439]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:05.809 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:05 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/992473234' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:07:05.842 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:03:09.860019+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T19:07:05.843 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-11T19:07:05.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:05 smithi170 ceph-mon[35690]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:05 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/992473234' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:07:06.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:05 smithi190 ceph-mon[36292]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:06.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:05 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/992473234' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:07:06.090 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:07:06.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:06 smithi170 ceph-mon[35690]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:06.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:06 smithi029 ceph-mon[32439]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:07.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:06 smithi190 ceph-mon[36292]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.848+0000 7f96fffab700 1 -- 172.21.15.29:0/2722273186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96f8101cc0 msgr2=0x7f96f8102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.848+0000 7f96fffab700 1 --2- 172.21.15.29:0/2722273186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96f8101cc0 0x7f96f8102140 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f96e8009990 tx=0x7f96e8009ca0 comp rx=0 tx=0).stop 2024-03-11T19:07:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.848+0000 7f96fffab700 1 -- 172.21.15.29:0/2722273186 shutdown_connections 2024-03-11T19:07:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.848+0000 7f96fffab700 1 --2- 172.21.15.29:0/2722273186 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96f8102680 0x7f96f810eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.848+0000 7f96fffab700 1 --2- 172.21.15.29:0/2722273186 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96f8107d70 0x7f96f8108150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.848+0000 7f96fffab700 1 --2- 172.21.15.29:0/2722273186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96f8101cc0 0x7f96f8102140 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.848+0000 7f96fffab700 1 -- 172.21.15.29:0/2722273186 >> 172.21.15.29:0/2722273186 conn(0x7f96f8078da0 msgr2=0x7f96f80791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.848+0000 7f96fffab700 1 -- 172.21.15.29:0/2722273186 shutdown_connections 2024-03-11T19:07:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.848+0000 7f96fffab700 1 -- 172.21.15.29:0/2722273186 wait complete. 2024-03-11T19:07:07.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.848+0000 7f96fffab700 1 Processor -- start 2024-03-11T19:07:07.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.848+0000 7f96fffab700 1 -- start start 2024-03-11T19:07:07.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fffab700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96f8101cc0 0x7f96f8198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:07.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fffab700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96f8102680 0x7f96f81985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:07.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fffab700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96f8107d70 0x7f96f819c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:07.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fffab700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f819cf50 con 0x7f96f8101cc0 2024-03-11T19:07:07.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fffab700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f819d0c0 con 0x7f96f8102680 2024-03-11T19:07:07.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fffab700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f819d280 con 0x7f96f8107d70 2024-03-11T19:07:07.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fdd47700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96f8101cc0 0x7f96f8198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:07.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fdd47700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96f8101cc0 0x7f96f8198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57258/0 (socket says 172.21.15.29:57258) 2024-03-11T19:07:07.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fdd47700 1 -- 172.21.15.29:0/3170437029 learned_addr learned my addr 172.21.15.29:0/3170437029 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:07.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fe548700 1 --2- 172.21.15.29:0/3170437029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96f8107d70 0x7f96f819c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fd546700 1 --2- 172.21.15.29:0/3170437029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96f8102680 0x7f96f81985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fdd47700 1 -- 172.21.15.29:0/3170437029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96f8107d70 msgr2=0x7f96f819c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fdd47700 1 --2- 172.21.15.29:0/3170437029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96f8107d70 0x7f96f819c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fdd47700 1 -- 172.21.15.29:0/3170437029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96f8102680 msgr2=0x7f96f81985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fdd47700 1 --2- 172.21.15.29:0/3170437029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96f8102680 0x7f96f81985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fdd47700 1 -- 172.21.15.29:0/3170437029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e80095f0 con 0x7f96f8101cc0 2024-03-11T19:07:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fd546700 1 --2- 172.21.15.29:0/3170437029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96f8102680 0x7f96f81985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:07:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fdd47700 1 --2- 172.21.15.29:0/3170437029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96f8101cc0 0x7f96f8198070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f96f400d950 tx=0x7f96f400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96eeffd700 1 -- 172.21.15.29:0/3170437029 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f4009940 con 0x7f96f8101cc0 2024-03-11T19:07:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96eeffd700 1 -- 172.21.15.29:0/3170437029 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f96f4010460 con 0x7f96f8101cc0 2024-03-11T19:07:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fffab700 1 -- 172.21.15.29:0/3170437029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96f819d560 con 0x7f96f8101cc0 2024-03-11T19:07:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96eeffd700 1 -- 172.21.15.29:0/3170437029 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f400f620 con 0x7f96f8101cc0 2024-03-11T19:07:07.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.849+0000 7f96fffab700 1 -- 172.21.15.29:0/3170437029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96f81a4dc0 con 0x7f96f8101cc0 2024-03-11T19:07:07.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.850+0000 7f96fffab700 1 -- 172.21.15.29:0/3170437029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f810a0c0 con 0x7f96f8101cc0 2024-03-11T19:07:07.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.851+0000 7f96eeffd700 1 -- 172.21.15.29:0/3170437029 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f96f40105d0 con 0x7f96f8101cc0 2024-03-11T19:07:07.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.852+0000 7f96eeffd700 1 --2- 172.21.15.29:0/3170437029 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f96e40a1ac0 0x7f96e40a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:07.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.852+0000 7f96fd546700 1 --2- 172.21.15.29:0/3170437029 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f96e40a1ac0 0x7f96e40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:07.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.852+0000 7f96eeffd700 1 -- 172.21.15.29:0/3170437029 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f96f40ccdb0 con 0x7f96f8101cc0 2024-03-11T19:07:07.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.852+0000 7f96fd546700 1 --2- 172.21.15.29:0/3170437029 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f96e40a1ac0 0x7f96e40a3f80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f96e800b5c0 tx=0x7f96e8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:07.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:07.856+0000 7f96eeffd700 1 -- 172.21.15.29:0/3170437029 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f96f4096460 con 0x7f96f8101cc0 2024-03-11T19:07:08.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:08.057+0000 7f96fffab700 1 -- 172.21.15.29:0/3170437029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f96f81993d0 con 0x7f96f8101cc0 2024-03-11T19:07:08.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:08.058+0000 7f96eeffd700 1 -- 172.21.15.29:0/3170437029 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f96f40cc0a0 con 0x7f96f8101cc0 2024-03-11T19:07:08.061 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-11T19:07:08.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:08.060+0000 7f96fffab700 1 -- 172.21.15.29:0/3170437029 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f96e40a1ac0 msgr2=0x7f96e40a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:08.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:08.060+0000 7f96fffab700 1 --2- 172.21.15.29:0/3170437029 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f96e40a1ac0 0x7f96e40a3f80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f96e800b5c0 tx=0x7f96e8005fb0 comp rx=0 tx=0).stop 2024-03-11T19:07:08.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:08.060+0000 7f96fffab700 1 -- 172.21.15.29:0/3170437029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96f8101cc0 msgr2=0x7f96f8198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:08.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:08.060+0000 7f96fffab700 1 --2- 172.21.15.29:0/3170437029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96f8101cc0 0x7f96f8198070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f96f400d950 tx=0x7f96f400dc60 comp rx=0 tx=0).stop 2024-03-11T19:07:08.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:08.060+0000 7f96fffab700 1 -- 172.21.15.29:0/3170437029 shutdown_connections 2024-03-11T19:07:08.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:08.060+0000 7f96fffab700 1 --2- 172.21.15.29:0/3170437029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96f8107d70 0x7f96f819c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:08.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:08.060+0000 7f96fffab700 1 --2- 172.21.15.29:0/3170437029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96f8101cc0 0x7f96f8198070 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:08.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:08.060+0000 7f96fffab700 1 --2- 172.21.15.29:0/3170437029 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f96e40a1ac0 0x7f96e40a3f80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:08.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:08.060+0000 7f96fffab700 1 --2- 172.21.15.29:0/3170437029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96f8102680 0x7f96f81985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:08.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:08.060+0000 7f96fffab700 1 -- 172.21.15.29:0/3170437029 >> 172.21.15.29:0/3170437029 conn(0x7f96f8078da0 msgr2=0x7f96f8100c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:08.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:08.060+0000 7f96fffab700 1 -- 172.21.15.29:0/3170437029 shutdown_connections 2024-03-11T19:07:08.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:08.060+0000 7f96fffab700 1 -- 172.21.15.29:0/3170437029 wait complete. 2024-03-11T19:07:08.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:08 smithi029 ceph-mon[32439]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:08.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:08 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3170437029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:07:08.827 INFO:tasks.cephadm:Adding mds.a on smithi029 2024-03-11T19:07:08.827 INFO:tasks.cephadm:Adding mds.d on smithi029 2024-03-11T19:07:08.827 INFO:tasks.cephadm:Adding mds.g on smithi029 2024-03-11T19:07:08.827 INFO:tasks.cephadm:Adding mds.j on smithi029 2024-03-11T19:07:08.827 INFO:tasks.cephadm:Adding mds.b on smithi170 2024-03-11T19:07:08.827 INFO:tasks.cephadm:Adding mds.e on smithi170 2024-03-11T19:07:08.827 INFO:tasks.cephadm:Adding mds.h on smithi170 2024-03-11T19:07:08.827 INFO:tasks.cephadm:Adding mds.k on smithi170 2024-03-11T19:07:08.828 INFO:tasks.cephadm:Adding mds.c on smithi190 2024-03-11T19:07:08.828 INFO:tasks.cephadm:Adding mds.f on smithi190 2024-03-11T19:07:08.828 INFO:tasks.cephadm:Adding mds.i on smithi190 2024-03-11T19:07:08.828 INFO:tasks.cephadm:Adding mds.l on smithi190 2024-03-11T19:07:08.828 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph orch apply mds all '12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi170=b;smithi170=e;smithi170=h;smithi170=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l' 2024-03-11T19:07:09.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:08 smithi190 ceph-mon[36292]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:09.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:08 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3170437029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:07:09.080 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:07:09.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:08 smithi170 ceph-mon[35690]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:09.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:08 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3170437029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:07:10.680 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.675+0000 7f4e15664700 1 -- 172.21.15.190:0/1415155813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e10102ea0 msgr2=0x7f4e10103300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:10.680 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.675+0000 7f4e15664700 1 --2- 172.21.15.190:0/1415155813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e10102ea0 0x7f4e10103300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4e00009990 tx=0x7f4e00009ca0 comp rx=0 tx=0).stop 2024-03-11T19:07:10.680 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.675+0000 7f4e15664700 1 -- 172.21.15.190:0/1415155813 shutdown_connections 2024-03-11T19:07:10.680 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.675+0000 7f4e15664700 1 --2- 172.21.15.190:0/1415155813 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e10103840 0x7f4e1010a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:10.680 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.675+0000 7f4e15664700 1 --2- 172.21.15.190:0/1415155813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e10101ca0 0x7f4e101020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:10.680 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.675+0000 7f4e15664700 1 --2- 172.21.15.190:0/1415155813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e10102ea0 0x7f4e10103300 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:10.680 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.675+0000 7f4e15664700 1 -- 172.21.15.190:0/1415155813 >> 172.21.15.190:0/1415155813 conn(0x7f4e100fd280 msgr2=0x7f4e100ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:10.680 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.675+0000 7f4e15664700 1 -- 172.21.15.190:0/1415155813 shutdown_connections 2024-03-11T19:07:10.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.675+0000 7f4e15664700 1 -- 172.21.15.190:0/1415155813 wait complete. 2024-03-11T19:07:10.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.676+0000 7f4e15664700 1 Processor -- start 2024-03-11T19:07:10.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.676+0000 7f4e15664700 1 -- start start 2024-03-11T19:07:10.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.676+0000 7f4e15664700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e10101ca0 0x7f4e10075a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:10.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.676+0000 7f4e15664700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e10102ea0 0x7f4e10074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:10.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.676+0000 7f4e15664700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e10103840 0x7f4e100745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:10.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.676+0000 7f4e15664700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e10074d20 con 0x7f4e10102ea0 2024-03-11T19:07:10.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.676+0000 7f4e15664700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e10074e90 con 0x7f4e10101ca0 2024-03-11T19:07:10.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.676+0000 7f4e15664700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e101a1c10 con 0x7f4e10103840 2024-03-11T19:07:10.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.676+0000 7f4e0f7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e10103840 0x7f4e100745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:10.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.676+0000 7f4e0f7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e10103840 0x7f4e100745a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:48348/0 (socket says 172.21.15.190:48348) 2024-03-11T19:07:10.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.676+0000 7f4e0f7fe700 1 -- 172.21.15.190:0/1035057001 learned_addr learned my addr 172.21.15.190:0/1035057001 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-03-11T19:07:10.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.676+0000 7f4e0effd700 1 --2- 172.21.15.190:0/1035057001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e10101ca0 0x7f4e10075a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:10.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.676+0000 7f4e0e7fc700 1 --2- 172.21.15.190:0/1035057001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e10102ea0 0x7f4e10074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:10.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.676+0000 7f4e0f7fe700 1 -- 172.21.15.190:0/1035057001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e10101ca0 msgr2=0x7f4e10075a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:10.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.676+0000 7f4e0f7fe700 1 --2- 172.21.15.190:0/1035057001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e10101ca0 0x7f4e10075a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:10.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.676+0000 7f4e0f7fe700 1 -- 172.21.15.190:0/1035057001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e10102ea0 msgr2=0x7f4e10074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:10.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.676+0000 7f4e0f7fe700 1 --2- 172.21.15.190:0/1035057001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e10102ea0 0x7f4e10074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:10.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.676+0000 7f4e0f7fe700 1 -- 172.21.15.190:0/1035057001 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e000095f0 con 0x7f4e10103840 2024-03-11T19:07:10.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.677+0000 7f4e0e7fc700 1 --2- 172.21.15.190:0/1035057001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e10102ea0 0x7f4e10074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:07:10.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.677+0000 7f4e0f7fe700 1 --2- 172.21.15.190:0/1035057001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e10103840 0x7f4e100745a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4e0800b970 tx=0x7f4e0800bd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:10.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.677+0000 7f4df7fff700 1 -- 172.21.15.190:0/1035057001 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0800c9b0 con 0x7f4e10103840 2024-03-11T19:07:10.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.677+0000 7f4e15664700 1 -- 172.21.15.190:0/1035057001 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e101a1ef0 con 0x7f4e10103840 2024-03-11T19:07:10.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.677+0000 7f4df7fff700 1 -- 172.21.15.190:0/1035057001 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4e08014440 con 0x7f4e10103840 2024-03-11T19:07:10.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.677+0000 7f4e15664700 1 -- 172.21.15.190:0/1035057001 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e101a23e0 con 0x7f4e10103840 2024-03-11T19:07:10.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.677+0000 7f4df7fff700 1 -- 172.21.15.190:0/1035057001 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e080075d0 con 0x7f4e10103840 2024-03-11T19:07:10.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.678+0000 7f4e15664700 1 -- 172.21.15.190:0/1035057001 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e101090b0 con 0x7f4e10103840 2024-03-11T19:07:10.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.680+0000 7f4df7fff700 1 -- 172.21.15.190:0/1035057001 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4e080145b0 con 0x7f4e10103840 2024-03-11T19:07:10.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.680+0000 7f4df7fff700 1 --2- 172.21.15.190:0/1035057001 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4df80a1ac0 0x7f4df80a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:10.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.680+0000 7f4df7fff700 1 -- 172.21.15.190:0/1035057001 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4e08006e40 con 0x7f4e10103840 2024-03-11T19:07:10.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.680+0000 7f4e0effd700 1 --2- 172.21.15.190:0/1035057001 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4df80a1ac0 0x7f4df80a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:10.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.681+0000 7f4e0effd700 1 --2- 172.21.15.190:0/1035057001 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4df80a1ac0 0x7f4df80a3f80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4e04005950 tx=0x7f4e040058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:10.686 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.684+0000 7f4df7fff700 1 -- 172.21.15.190:0/1035057001 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4e080961c0 con 0x7f4e10103840 2024-03-11T19:07:10.761 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:10 smithi190 ceph-mon[36292]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:10.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.888+0000 7f4e15664700 1 -- 172.21.15.190:0/1035057001 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi170=b;smithi170=e;smithi170=h;smithi170=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e1002d970 con 0x7f4df80a1ac0 2024-03-11T19:07:10.899 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.897+0000 7f4df7fff700 1 -- 172.21.15.190:0/1035057001 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f4e1002d970 con 0x7f4df80a1ac0 2024-03-11T19:07:10.900 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled mds.all update... 2024-03-11T19:07:10.902 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.899+0000 7f4e15664700 1 -- 172.21.15.190:0/1035057001 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4df80a1ac0 msgr2=0x7f4df80a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:10.902 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.899+0000 7f4e15664700 1 --2- 172.21.15.190:0/1035057001 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4df80a1ac0 0x7f4df80a3f80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4e04005950 tx=0x7f4e040058e0 comp rx=0 tx=0).stop 2024-03-11T19:07:10.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.899+0000 7f4e15664700 1 -- 172.21.15.190:0/1035057001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e10103840 msgr2=0x7f4e100745a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:10.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.899+0000 7f4e15664700 1 --2- 172.21.15.190:0/1035057001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e10103840 0x7f4e100745a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4e0800b970 tx=0x7f4e0800bd30 comp rx=0 tx=0).stop 2024-03-11T19:07:10.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.899+0000 7f4e15664700 1 -- 172.21.15.190:0/1035057001 shutdown_connections 2024-03-11T19:07:10.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.899+0000 7f4e15664700 1 --2- 172.21.15.190:0/1035057001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e10103840 0x7f4e100745a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:10.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.899+0000 7f4e15664700 1 --2- 172.21.15.190:0/1035057001 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4df80a1ac0 0x7f4df80a3f80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:10.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.899+0000 7f4e15664700 1 --2- 172.21.15.190:0/1035057001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e10101ca0 0x7f4e10075a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:10.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.899+0000 7f4e15664700 1 --2- 172.21.15.190:0/1035057001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e10102ea0 0x7f4e10074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:10.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.899+0000 7f4e15664700 1 -- 172.21.15.190:0/1035057001 >> 172.21.15.190:0/1035057001 conn(0x7f4e100fd280 msgr2=0x7f4e101061f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:10.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.899+0000 7f4e15664700 1 -- 172.21.15.190:0/1035057001 shutdown_connections 2024-03-11T19:07:10.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-11T19:07:10.899+0000 7f4e15664700 1 -- 172.21.15.190:0/1035057001 wait complete. 2024-03-11T19:07:10.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:10 smithi029 ceph-mon[32439]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:11.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:10 smithi170 ceph-mon[35690]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:11.554 DEBUG:teuthology.orchestra.run.smithi029:mds.a> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.a.service 2024-03-11T19:07:11.557 DEBUG:teuthology.orchestra.run.smithi029:mds.d> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.d.service 2024-03-11T19:07:11.600 DEBUG:teuthology.orchestra.run.smithi029:mds.g> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.g.service 2024-03-11T19:07:11.643 DEBUG:teuthology.orchestra.run.smithi029:mds.j> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.j.service 2024-03-11T19:07:11.686 DEBUG:teuthology.orchestra.run.smithi170:mds.b> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.b.service 2024-03-11T19:07:11.689 DEBUG:teuthology.orchestra.run.smithi170:mds.e> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.e.service 2024-03-11T19:07:11.732 DEBUG:teuthology.orchestra.run.smithi170:mds.h> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.h.service 2024-03-11T19:07:11.775 DEBUG:teuthology.orchestra.run.smithi170:mds.k> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.k.service 2024-03-11T19:07:11.818 DEBUG:teuthology.orchestra.run.smithi190:mds.c> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.c.service 2024-03-11T19:07:11.820 DEBUG:teuthology.orchestra.run.smithi190:mds.f> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.f.service 2024-03-11T19:07:11.864 DEBUG:teuthology.orchestra.run.smithi190:mds.i> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.i.service 2024-03-11T19:07:11.906 DEBUG:teuthology.orchestra.run.smithi190:mds.l> sudo journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.l.service 2024-03-11T19:07:11.950 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-11T19:07:11.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:07:12.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:11 smithi190 ceph-mon[36292]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi170=b;smithi170=e;smithi170=h;smithi170=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:07:12.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:11 smithi190 ceph-mon[36292]: Saving service mds.all spec with placement smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi170=b;smithi170=e;smithi170=h;smithi170=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l;count:12 2024-03-11T19:07:12.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:11 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:12.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:11 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:07:12.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:11 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:12.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:11 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:07:12.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:11 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:12.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:11 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:12.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:11 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:12.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:11 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:12.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:11 smithi190 ceph-mon[36292]: Deploying daemon mds.l on smithi190 2024-03-11T19:07:12.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.069+0000 7f9a29e8f640 1 -- 172.21.15.29:0/1832189645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a241011b0 msgr2=0x7f9a241015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:12.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.069+0000 7f9a29e8f640 1 --2- 172.21.15.29:0/1832189645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a241011b0 0x7f9a241015d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9a14009900 tx=0x7f9a1402bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:12.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.069+0000 7f9a29e8f640 1 -- 172.21.15.29:0/1832189645 shutdown_connections 2024-03-11T19:07:12.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.069+0000 7f9a29e8f640 1 --2- 172.21.15.29:0/1832189645 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a24102d60 0x7f9a241095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.069+0000 7f9a29e8f640 1 --2- 172.21.15.29:0/1832189645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a241023a0 0x7f9a24102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.069+0000 7f9a29e8f640 1 --2- 172.21.15.29:0/1832189645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a241011b0 0x7f9a241015d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.069+0000 7f9a29e8f640 1 -- 172.21.15.29:0/1832189645 >> 172.21.15.29:0/1832189645 conn(0x7f9a240fc760 msgr2=0x7f9a240feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:12.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.070+0000 7f9a29e8f640 1 -- 172.21.15.29:0/1832189645 shutdown_connections 2024-03-11T19:07:12.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.070+0000 7f9a29e8f640 1 -- 172.21.15.29:0/1832189645 wait complete. 2024-03-11T19:07:12.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.070+0000 7f9a29e8f640 1 Processor -- start 2024-03-11T19:07:12.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.070+0000 7f9a29e8f640 1 -- start start 2024-03-11T19:07:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.070+0000 7f9a29e8f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a241011b0 0x7f9a24197210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.070+0000 7f9a29e8f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a241023a0 0x7f9a24197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.070+0000 7f9a29e8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a24102d60 0x7f9a2419e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a29e8f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2419ecb0 con 0x7f9a24102d60 2024-03-11T19:07:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a29e8f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2419ee20 con 0x7f9a241011b0 2024-03-11T19:07:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a29e8f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2419efe0 con 0x7f9a241023a0 2024-03-11T19:07:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a23fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a24102d60 0x7f9a2419e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a237fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a241011b0 0x7f9a24197210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a22ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a241023a0 0x7f9a24197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a237fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a241011b0 0x7f9a24197210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:36560/0 (socket says 172.21.15.29:36560) 2024-03-11T19:07:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a22ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a241023a0 0x7f9a24197750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:53534/0 (socket says 172.21.15.29:53534) 2024-03-11T19:07:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a237fe640 1 -- 172.21.15.29:0/1596420662 learned_addr learned my addr 172.21.15.29:0/1596420662 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a23fff640 1 -- 172.21.15.29:0/1596420662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a241023a0 msgr2=0x7f9a24197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a23fff640 1 --2- 172.21.15.29:0/1596420662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a241023a0 0x7f9a24197750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a23fff640 1 -- 172.21.15.29:0/1596420662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a241011b0 msgr2=0x7f9a24197210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a23fff640 1 --2- 172.21.15.29:0/1596420662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a241011b0 0x7f9a24197210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a23fff640 1 -- 172.21.15.29:0/1596420662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a14009560 con 0x7f9a24102d60 2024-03-11T19:07:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a23fff640 1 --2- 172.21.15.29:0/1596420662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a24102d60 0x7f9a2419e770 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00ca30 tx=0x7f9a1c00cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a20ff9640 1 -- 172.21.15.29:0/1596420662 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c007d90 con 0x7f9a24102d60 2024-03-11T19:07:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a20ff9640 1 -- 172.21.15.29:0/1596420662 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9a1c005020 con 0x7f9a24102d60 2024-03-11T19:07:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a29e8f640 1 -- 172.21.15.29:0/1596420662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a2419f2c0 con 0x7f9a24102d60 2024-03-11T19:07:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a20ff9640 1 -- 172.21.15.29:0/1596420662 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c00f650 con 0x7f9a24102d60 2024-03-11T19:07:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.071+0000 7f9a29e8f640 1 -- 172.21.15.29:0/1596420662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a2419f780 con 0x7f9a24102d60 2024-03-11T19:07:12.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.072+0000 7f9a29e8f640 1 -- 172.21.15.29:0/1596420662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a24108540 con 0x7f9a24102d60 2024-03-11T19:07:12.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.073+0000 7f9a20ff9640 1 -- 172.21.15.29:0/1596420662 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9a1c005190 con 0x7f9a24102d60 2024-03-11T19:07:12.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.074+0000 7f9a20ff9640 1 --2- 172.21.15.29:0/1596420662 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a100a1a30 0x7f9a100a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:12.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.074+0000 7f9a237fe640 1 --2- 172.21.15.29:0/1596420662 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a100a1a30 0x7f9a100a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:12.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.074+0000 7f9a20ff9640 1 -- 172.21.15.29:0/1596420662 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9a1c0cd610 con 0x7f9a24102d60 2024-03-11T19:07:12.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.074+0000 7f9a237fe640 1 --2- 172.21.15.29:0/1596420662 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a100a1a30 0x7f9a100a3ef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9a14006010 tx=0x7f9a14005d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:12.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.078+0000 7f9a20ff9640 1 -- 172.21.15.29:0/1596420662 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9a1c096cd0 con 0x7f9a24102d60 2024-03-11T19:07:12.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:11 smithi170 ceph-mon[35690]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi170=b;smithi170=e;smithi170=h;smithi170=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:07:12.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:11 smithi170 ceph-mon[35690]: Saving service mds.all spec with placement smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi170=b;smithi170=e;smithi170=h;smithi170=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l;count:12 2024-03-11T19:07:12.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:11 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:12.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:11 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:07:12.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:11 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:12.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:11 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:07:12.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:11 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:12.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:11 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:12.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:11 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:12.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:11 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:12.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:11 smithi170 ceph-mon[35690]: Deploying daemon mds.l on smithi190 2024-03-11T19:07:12.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:11 smithi029 ceph-mon[32439]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi170=b;smithi170=e;smithi170=h;smithi170=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:07:12.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:11 smithi029 ceph-mon[32439]: Saving service mds.all spec with placement smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi170=b;smithi170=e;smithi170=h;smithi170=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l;count:12 2024-03-11T19:07:12.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:11 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:12.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:11 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:07:12.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:11 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:12.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:11 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:07:12.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:11 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:12.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:11 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:12.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:11 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:12.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:11 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:12.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:11 smithi029 ceph-mon[32439]: Deploying daemon mds.l on smithi190 2024-03-11T19:07:12.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.261+0000 7f9a29e8f640 1 -- 172.21.15.29:0/1596420662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a2402ce40 con 0x7f9a24102d60 2024-03-11T19:07:12.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.262+0000 7f9a20ff9640 1 -- 172.21.15.29:0/1596420662 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+18456 (secure 0 0 0) 0x7f9a1c09a390 con 0x7f9a24102d60 2024-03-11T19:07:12.266 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:12.266 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":83,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:07:00.078122+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:07:12.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.264+0000 7f9a29e8f640 1 -- 172.21.15.29:0/1596420662 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a100a1a30 msgr2=0x7f9a100a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:12.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.264+0000 7f9a29e8f640 1 --2- 172.21.15.29:0/1596420662 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a100a1a30 0x7f9a100a3ef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9a14006010 tx=0x7f9a14005d50 comp rx=0 tx=0).stop 2024-03-11T19:07:12.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.264+0000 7f9a29e8f640 1 -- 172.21.15.29:0/1596420662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a24102d60 msgr2=0x7f9a2419e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:12.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.264+0000 7f9a29e8f640 1 --2- 172.21.15.29:0/1596420662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a24102d60 0x7f9a2419e770 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00ca30 tx=0x7f9a1c00cef0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.264+0000 7f9a29e8f640 1 -- 172.21.15.29:0/1596420662 shutdown_connections 2024-03-11T19:07:12.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.264+0000 7f9a29e8f640 1 --2- 172.21.15.29:0/1596420662 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a100a1a30 0x7f9a100a3ef0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.264+0000 7f9a29e8f640 1 --2- 172.21.15.29:0/1596420662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a24102d60 0x7f9a2419e770 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.264+0000 7f9a29e8f640 1 --2- 172.21.15.29:0/1596420662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a241023a0 0x7f9a24197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.264+0000 7f9a29e8f640 1 --2- 172.21.15.29:0/1596420662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a241011b0 0x7f9a24197210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.264+0000 7f9a29e8f640 1 -- 172.21.15.29:0/1596420662 >> 172.21.15.29:0/1596420662 conn(0x7f9a240fc760 msgr2=0x7f9a24105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:12.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.264+0000 7f9a29e8f640 1 -- 172.21.15.29:0/1596420662 shutdown_connections 2024-03-11T19:07:12.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.264+0000 7f9a29e8f640 1 -- 172.21.15.29:0/1596420662 wait complete. 2024-03-11T19:07:12.279 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:03:09.860019+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T19:07:12.280 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:07:12.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.382+0000 7fd7d572c640 1 -- 172.21.15.29:0/2385798327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d0100080 msgr2=0x7fd7d01004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:12.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.382+0000 7fd7d572c640 1 --2- 172.21.15.29:0/2385798327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d0100080 0x7fd7d01004a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8009900 tx=0x7fd7c802bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:12.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.382+0000 7fd7d572c640 1 -- 172.21.15.29:0/2385798327 shutdown_connections 2024-03-11T19:07:12.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.382+0000 7fd7d572c640 1 --2- 172.21.15.29:0/2385798327 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7d0101c30 0x7fd7d01084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.382+0000 7fd7d572c640 1 --2- 172.21.15.29:0/2385798327 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7d0101270 0x7fd7d01016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.382+0000 7fd7d572c640 1 --2- 172.21.15.29:0/2385798327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d0100080 0x7fd7d01004a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.382+0000 7fd7d572c640 1 -- 172.21.15.29:0/2385798327 >> 172.21.15.29:0/2385798327 conn(0x7fd7d00fb610 msgr2=0x7fd7d00fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:12.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.382+0000 7fd7d572c640 1 -- 172.21.15.29:0/2385798327 shutdown_connections 2024-03-11T19:07:12.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.382+0000 7fd7d572c640 1 -- 172.21.15.29:0/2385798327 wait complete. 2024-03-11T19:07:12.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.383+0000 7fd7d572c640 1 Processor -- start 2024-03-11T19:07:12.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.383+0000 7fd7d572c640 1 -- start start 2024-03-11T19:07:12.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.383+0000 7fd7d572c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d0100080 0x7fd7d0068e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:12.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.383+0000 7fd7d572c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7d0101270 0x7fd7d019bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:12.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.383+0000 7fd7d572c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7d0101c30 0x7fd7d019e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:12.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.383+0000 7fd7d572c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d019ecc0 con 0x7fd7d0100080 2024-03-11T19:07:12.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.383+0000 7fd7d572c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d019ee30 con 0x7fd7d0101c30 2024-03-11T19:07:12.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.383+0000 7fd7d572c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d019eff0 con 0x7fd7d0101270 2024-03-11T19:07:12.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.383+0000 7fd7ceffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d0100080 0x7fd7d0068e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:12.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.383+0000 7fd7ceffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d0100080 0x7fd7d0068e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57288/0 (socket says 172.21.15.29:57288) 2024-03-11T19:07:12.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.383+0000 7fd7ceffd640 1 -- 172.21.15.29:0/4266473914 learned_addr learned my addr 172.21.15.29:0/4266473914 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:12.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.383+0000 7fd7ce7fc640 1 --2- 172.21.15.29:0/4266473914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7d0101270 0x7fd7d019bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:12.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.383+0000 7fd7ceffd640 1 -- 172.21.15.29:0/4266473914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7d0101270 msgr2=0x7fd7d019bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:12.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.383+0000 7fd7ceffd640 1 --2- 172.21.15.29:0/4266473914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7d0101270 0x7fd7d019bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.383+0000 7fd7ceffd640 1 -- 172.21.15.29:0/4266473914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7d0101c30 msgr2=0x7fd7d019e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:12.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.384+0000 7fd7ceffd640 1 --2- 172.21.15.29:0/4266473914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7d0101c30 0x7fd7d019e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.384+0000 7fd7ceffd640 1 -- 172.21.15.29:0/4266473914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c8009560 con 0x7fd7d0100080 2024-03-11T19:07:12.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.384+0000 7fd7ceffd640 1 --2- 172.21.15.29:0/4266473914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d0100080 0x7fd7d0068e70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd7c802c7d0 tx=0x7fd7c80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:12.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.384+0000 7fd7c7fff640 1 -- 172.21.15.29:0/4266473914 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c8004aa0 con 0x7fd7d0100080 2024-03-11T19:07:12.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.384+0000 7fd7c7fff640 1 -- 172.21.15.29:0/4266473914 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd7c802de90 con 0x7fd7d0100080 2024-03-11T19:07:12.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.384+0000 7fd7d572c640 1 -- 172.21.15.29:0/4266473914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7d019f270 con 0x7fd7d0100080 2024-03-11T19:07:12.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.384+0000 7fd7c7fff640 1 -- 172.21.15.29:0/4266473914 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c8005440 con 0x7fd7d0100080 2024-03-11T19:07:12.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.384+0000 7fd7d572c640 1 -- 172.21.15.29:0/4266473914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7d019f780 con 0x7fd7d0100080 2024-03-11T19:07:12.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.385+0000 7fd7d572c640 1 -- 172.21.15.29:0/4266473914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7d0107510 con 0x7fd7d0100080 2024-03-11T19:07:12.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.386+0000 7fd7c7fff640 1 -- 172.21.15.29:0/4266473914 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd7c80055a0 con 0x7fd7d0100080 2024-03-11T19:07:12.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.387+0000 7fd7c7fff640 1 --2- 172.21.15.29:0/4266473914 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd7bc0a1a30 0x7fd7bc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:12.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.387+0000 7fd7ce7fc640 1 --2- 172.21.15.29:0/4266473914 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd7bc0a1a30 0x7fd7bc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:12.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.387+0000 7fd7c7fff640 1 -- 172.21.15.29:0/4266473914 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd7c80ef170 con 0x7fd7d0100080 2024-03-11T19:07:12.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.387+0000 7fd7ce7fc640 1 --2- 172.21.15.29:0/4266473914 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd7bc0a1a30 0x7fd7bc0a3ef0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0068be0 tx=0x7fd7b800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.391+0000 7fd7c7fff640 1 -- 172.21.15.29:0/4266473914 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd7c80b8820 con 0x7fd7d0100080 2024-03-11T19:07:12.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.573+0000 7fd7d572c640 1 -- 172.21.15.29:0/4266473914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd7d002cda0 con 0x7fd7d0100080 2024-03-11T19:07:12.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.573+0000 7fd7c7fff640 1 -- 172.21.15.29:0/4266473914 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd7c80bbee0 con 0x7fd7d0100080 2024-03-11T19:07:12.577 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-11T19:07:12.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.575+0000 7fd7d572c640 1 -- 172.21.15.29:0/4266473914 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd7bc0a1a30 msgr2=0x7fd7bc0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:12.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.575+0000 7fd7d572c640 1 --2- 172.21.15.29:0/4266473914 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd7bc0a1a30 0x7fd7bc0a3ef0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0068be0 tx=0x7fd7b800b410 comp rx=0 tx=0).stop 2024-03-11T19:07:12.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.575+0000 7fd7d572c640 1 -- 172.21.15.29:0/4266473914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d0100080 msgr2=0x7fd7d0068e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:12.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.575+0000 7fd7d572c640 1 --2- 172.21.15.29:0/4266473914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d0100080 0x7fd7d0068e70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd7c802c7d0 tx=0x7fd7c80042f0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.575+0000 7fd7d572c640 1 -- 172.21.15.29:0/4266473914 shutdown_connections 2024-03-11T19:07:12.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.575+0000 7fd7d572c640 1 --2- 172.21.15.29:0/4266473914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7d0101c30 0x7fd7d019e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.575+0000 7fd7d572c640 1 --2- 172.21.15.29:0/4266473914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7d0101270 0x7fd7d019bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.575+0000 7fd7d572c640 1 --2- 172.21.15.29:0/4266473914 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd7bc0a1a30 0x7fd7bc0a3ef0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.575+0000 7fd7d572c640 1 --2- 172.21.15.29:0/4266473914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d0100080 0x7fd7d0068e70 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.575+0000 7fd7d572c640 1 -- 172.21.15.29:0/4266473914 >> 172.21.15.29:0/4266473914 conn(0x7fd7d00fb610 msgr2=0x7fd7d01046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:12.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.575+0000 7fd7d572c640 1 -- 172.21.15.29:0/4266473914 shutdown_connections 2024-03-11T19:07:12.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.575+0000 7fd7d572c640 1 -- 172.21.15.29:0/4266473914 wait complete. 2024-03-11T19:07:12.590 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:07:12.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.689+0000 7f9d7bd4b640 1 -- 172.21.15.29:0/2701470489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d74101190 msgr2=0x7f9d741015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:12.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.689+0000 7f9d7bd4b640 1 --2- 172.21.15.29:0/2701470489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d74101190 0x7f9d741015b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9d70009900 tx=0x7f9d7002bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:12.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.689+0000 7f9d7bd4b640 1 -- 172.21.15.29:0/2701470489 shutdown_connections 2024-03-11T19:07:12.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.689+0000 7f9d7bd4b640 1 --2- 172.21.15.29:0/2701470489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d74102d40 0x7f9d741095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.689+0000 7f9d7bd4b640 1 --2- 172.21.15.29:0/2701470489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d74102380 0x7f9d74102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.689+0000 7f9d7bd4b640 1 --2- 172.21.15.29:0/2701470489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d74101190 0x7f9d741015b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.689+0000 7f9d7bd4b640 1 -- 172.21.15.29:0/2701470489 >> 172.21.15.29:0/2701470489 conn(0x7f9d740fc760 msgr2=0x7f9d740feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:12.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.689+0000 7f9d7bd4b640 1 -- 172.21.15.29:0/2701470489 shutdown_connections 2024-03-11T19:07:12.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.689+0000 7f9d7bd4b640 1 -- 172.21.15.29:0/2701470489 wait complete. 2024-03-11T19:07:12.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.689+0000 7f9d7bd4b640 1 Processor -- start 2024-03-11T19:07:12.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.689+0000 7f9d7bd4b640 1 -- start start 2024-03-11T19:07:12.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.689+0000 7f9d7bd4b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d74101190 0x7f9d74074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.689+0000 7f9d7bd4b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d74102380 0x7f9d74073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.689+0000 7f9d7bd4b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d74102d40 0x7f9d74073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.689+0000 7f9d7bd4b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d74074210 con 0x7f9d74102d40 2024-03-11T19:07:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.689+0000 7f9d7bd4b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d74074380 con 0x7f9d74102380 2024-03-11T19:07:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.689+0000 7f9d7bd4b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d741a11d0 con 0x7f9d74101190 2024-03-11T19:07:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.690+0000 7f9d7a2e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d74102d40 0x7f9d74073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.690+0000 7f9d7a2e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d74102d40 0x7f9d74073ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57298/0 (socket says 172.21.15.29:57298) 2024-03-11T19:07:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.690+0000 7f9d7a2e8640 1 -- 172.21.15.29:0/1423232888 learned_addr learned my addr 172.21.15.29:0/1423232888 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.690+0000 7f9d79ae7640 1 --2- 172.21.15.29:0/1423232888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d74101190 0x7f9d74074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.690+0000 7f9d792e6640 1 --2- 172.21.15.29:0/1423232888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d74102380 0x7f9d74073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.690+0000 7f9d7a2e8640 1 -- 172.21.15.29:0/1423232888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d74101190 msgr2=0x7f9d74074f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.690+0000 7f9d7a2e8640 1 --2- 172.21.15.29:0/1423232888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d74101190 0x7f9d74074f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.690+0000 7f9d7a2e8640 1 -- 172.21.15.29:0/1423232888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d74102380 msgr2=0x7f9d74073570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.690+0000 7f9d7a2e8640 1 --2- 172.21.15.29:0/1423232888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d74102380 0x7f9d74073570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.690+0000 7f9d7a2e8640 1 -- 172.21.15.29:0/1423232888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d70009560 con 0x7f9d74102d40 2024-03-11T19:07:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.690+0000 7f9d792e6640 1 --2- 172.21.15.29:0/1423232888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d74102380 0x7f9d74073570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:07:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.690+0000 7f9d7a2e8640 1 --2- 172.21.15.29:0/1423232888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d74102d40 0x7f9d74073ab0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00eb40 tx=0x7f9d6c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.690+0000 7f9d6affd640 1 -- 172.21.15.29:0/1423232888 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c004b20 con 0x7f9d74102d40 2024-03-11T19:07:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.690+0000 7f9d6affd640 1 -- 172.21.15.29:0/1423232888 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9d6c004c80 con 0x7f9d74102d40 2024-03-11T19:07:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.690+0000 7f9d7bd4b640 1 -- 172.21.15.29:0/1423232888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d741a1480 con 0x7f9d74102d40 2024-03-11T19:07:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.690+0000 7f9d6affd640 1 -- 172.21.15.29:0/1423232888 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c010690 con 0x7f9d74102d40 2024-03-11T19:07:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.690+0000 7f9d7bd4b640 1 -- 172.21.15.29:0/1423232888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d741a1990 con 0x7f9d74102d40 2024-03-11T19:07:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.691+0000 7f9d7bd4b640 1 -- 172.21.15.29:0/1423232888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d74108530 con 0x7f9d74102d40 2024-03-11T19:07:12.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.692+0000 7f9d6affd640 1 -- 172.21.15.29:0/1423232888 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9d6c005020 con 0x7f9d74102d40 2024-03-11T19:07:12.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.693+0000 7f9d6affd640 1 --2- 172.21.15.29:0/1423232888 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9d600a1ab0 0x7f9d600a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:12.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.693+0000 7f9d79ae7640 1 --2- 172.21.15.29:0/1423232888 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9d600a1ab0 0x7f9d600a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:12.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.693+0000 7f9d6affd640 1 -- 172.21.15.29:0/1423232888 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9d6c014070 con 0x7f9d74102d40 2024-03-11T19:07:12.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.693+0000 7f9d79ae7640 1 --2- 172.21.15.29:0/1423232888 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9d600a1ab0 0x7f9d600a3f70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9d7002c240 tx=0x7f9d7002c770 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:12.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.697+0000 7f9d6affd640 1 -- 172.21.15.29:0/1423232888 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9d6c096c30 con 0x7f9d74102d40 2024-03-11T19:07:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.916+0000 7f9d7bd4b640 1 -- 172.21.15.29:0/1423232888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d7402ce40 con 0x7f9d74102d40 2024-03-11T19:07:12.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.916+0000 7f9d6affd640 1 -- 172.21.15.29:0/1423232888 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7f9d6c09a2f0 con 0x7f9d74102d40 2024-03-11T19:07:12.920 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:12.920 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-11T19:07:12.920 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 1 2024-03-11T19:07:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.918+0000 7f9d7bd4b640 1 -- 172.21.15.29:0/1423232888 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9d600a1ab0 msgr2=0x7f9d600a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.918+0000 7f9d7bd4b640 1 --2- 172.21.15.29:0/1423232888 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9d600a1ab0 0x7f9d600a3f70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9d7002c240 tx=0x7f9d7002c770 comp rx=0 tx=0).stop 2024-03-11T19:07:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.918+0000 7f9d7bd4b640 1 -- 172.21.15.29:0/1423232888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d74102d40 msgr2=0x7f9d74073ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.918+0000 7f9d7bd4b640 1 --2- 172.21.15.29:0/1423232888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d74102d40 0x7f9d74073ab0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00eb40 tx=0x7f9d6c00c2d0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.918+0000 7f9d7bd4b640 1 -- 172.21.15.29:0/1423232888 shutdown_connections 2024-03-11T19:07:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.918+0000 7f9d7bd4b640 1 --2- 172.21.15.29:0/1423232888 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9d600a1ab0 0x7f9d600a3f70 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.918+0000 7f9d7bd4b640 1 --2- 172.21.15.29:0/1423232888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d74102d40 0x7f9d74073ab0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.918+0000 7f9d7bd4b640 1 --2- 172.21.15.29:0/1423232888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d74102380 0x7f9d74073570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.918+0000 7f9d7bd4b640 1 --2- 172.21.15.29:0/1423232888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d74101190 0x7f9d74074f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:12.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.918+0000 7f9d7bd4b640 1 -- 172.21.15.29:0/1423232888 >> 172.21.15.29:0/1423232888 conn(0x7f9d740fc760 msgr2=0x7f9d74105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:12.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.918+0000 7f9d7bd4b640 1 -- 172.21.15.29:0/1423232888 shutdown_connections 2024-03-11T19:07:12.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:12.918+0000 7f9d7bd4b640 1 -- 172.21.15.29:0/1423232888 wait complete. 2024-03-11T19:07:13.040 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:12 smithi190 ceph-mon[36292]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:13.040 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:12 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1596420662' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:07:13.041 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:12 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/4266473914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:07:13.041 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:12 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1423232888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:13.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:12 smithi170 ceph-mon[35690]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:13.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:12 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1596420662' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:07:13.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:12 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/4266473914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:07:13.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:12 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1423232888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:13.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:12 smithi029 ceph-mon[32439]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:13.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:12 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1596420662' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:07:13.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:12 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/4266473914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:07:13.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:12 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1423232888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:13.435 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:07:13 smithi190 systemd[1]: Starting Ceph mds.l for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:07:13.715 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:07:13 smithi190 podman[66059]: 2024-03-11 19:07:13.335456574 +0000 UTC m=+0.017946763 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:07:13.715 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:07:13 smithi190 podman[66059]: 2024-03-11 19:07:13.451764671 +0000 UTC m=+0.134254846 container create abe8b2b6897f4542c598c1fa04b6a91671964b15b1fd1446e292bdbb060f23a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-l, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T19:07:13.715 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:07:13 smithi190 podman[66059]: 2024-03-11 19:07:13.71276531 +0000 UTC m=+0.395255485 container init abe8b2b6897f4542c598c1fa04b6a91671964b15b1fd1446e292bdbb060f23a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-l, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-11T19:07:14.012 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:07:13 smithi190 podman[66059]: 2024-03-11 19:07:13.7155943 +0000 UTC m=+0.398084470 container start abe8b2b6897f4542c598c1fa04b6a91671964b15b1fd1446e292bdbb060f23a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-l, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:07:14.012 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:07:13 smithi190 ceph-mds[66077]: -- 172.21.15.190:0/2055261394 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562c250081a0 con 0x562c24fdf000 2024-03-11T19:07:14.012 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:07:13 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-l[66073]: starting mds.l at 2024-03-11T19:07:14.012 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:07:13 smithi190 bash[66059]: abe8b2b6897f4542c598c1fa04b6a91671964b15b1fd1446e292bdbb060f23a0 2024-03-11T19:07:14.012 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:07:13 smithi190 systemd[1]: Started Ceph mds.l for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:07:14.934 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:07:15.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:14 smithi190 ceph-mon[36292]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:15.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:14 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.190:6832/493725011,v1:172.21.15.190:6833/493725011] up:boot 2024-03-11T19:07:15.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:14 smithi190 ceph-mon[36292]: fsmap 1 up:standby 2024-03-11T19:07:15.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:14 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T19:07:15.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:14 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:15.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:14 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:15.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:14 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:15.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:14 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:15.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:14 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:15.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:14 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:15.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:14 smithi190 ceph-mon[36292]: Deploying daemon mds.k on smithi170 2024-03-11T19:07:15.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.044+0000 7f6c8ae64640 1 -- 172.21.15.29:0/2251830253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c84100120 msgr2=0x7f6c84100540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:15.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.044+0000 7f6c8ae64640 1 --2- 172.21.15.29:0/2251830253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c84100120 0x7f6c84100540 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009900 tx=0x7f6c7402bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:15.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.044+0000 7f6c8ae64640 1 -- 172.21.15.29:0/2251830253 shutdown_connections 2024-03-11T19:07:15.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.044+0000 7f6c8ae64640 1 --2- 172.21.15.29:0/2251830253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c84101cd0 0x7f6c84108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:15.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.044+0000 7f6c8ae64640 1 --2- 172.21.15.29:0/2251830253 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c84101310 0x7f6c84101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:15.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.044+0000 7f6c8ae64640 1 --2- 172.21.15.29:0/2251830253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c84100120 0x7f6c84100540 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:15.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.044+0000 7f6c8ae64640 1 -- 172.21.15.29:0/2251830253 >> 172.21.15.29:0/2251830253 conn(0x7f6c840fb6b0 msgr2=0x7f6c840fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:15.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.044+0000 7f6c8ae64640 1 -- 172.21.15.29:0/2251830253 shutdown_connections 2024-03-11T19:07:15.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.044+0000 7f6c8ae64640 1 -- 172.21.15.29:0/2251830253 wait complete. 2024-03-11T19:07:15.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c8ae64640 1 Processor -- start 2024-03-11T19:07:15.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c8ae64640 1 -- start start 2024-03-11T19:07:15.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c8ae64640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c84100120 0x7f6c841971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:15.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c8ae64640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c84101310 0x7f6c841976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:15.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c8ae64640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c84101cd0 0x7f6c8419e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:15.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c8ae64640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8419eca0 con 0x7f6c84101310 2024-03-11T19:07:15.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c8ae64640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8419ee10 con 0x7f6c84101cd0 2024-03-11T19:07:15.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c8ae64640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8419efd0 con 0x7f6c84100120 2024-03-11T19:07:15.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c83fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c84101310 0x7f6c841976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:15.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c83fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c84101310 0x7f6c841976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57328/0 (socket says 172.21.15.29:57328) 2024-03-11T19:07:15.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c83fff640 1 -- 172.21.15.29:0/3392535021 learned_addr learned my addr 172.21.15.29:0/3392535021 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:15.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c88c00640 1 --2- 172.21.15.29:0/3392535021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c84100120 0x7f6c841971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:15.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c89401640 1 --2- 172.21.15.29:0/3392535021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c84101cd0 0x7f6c8419e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:15.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c83fff640 1 -- 172.21.15.29:0/3392535021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c84100120 msgr2=0x7f6c841971b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:15.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c83fff640 1 --2- 172.21.15.29:0/3392535021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c84100120 0x7f6c841971b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:15.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c83fff640 1 -- 172.21.15.29:0/3392535021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c84101cd0 msgr2=0x7f6c8419e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:15.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c83fff640 1 --2- 172.21.15.29:0/3392535021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c84101cd0 0x7f6c8419e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:15.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c83fff640 1 -- 172.21.15.29:0/3392535021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c74009560 con 0x7f6c84101310 2024-03-11T19:07:15.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c89401640 1 --2- 172.21.15.29:0/3392535021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c84101cd0 0x7f6c8419e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:07:15.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.045+0000 7f6c88c00640 1 --2- 172.21.15.29:0/3392535021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c84100120 0x7f6c841971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:07:15.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.046+0000 7f6c83fff640 1 --2- 172.21.15.29:0/3392535021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c84101310 0x7f6c841976f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00c920 tx=0x7f6c6c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:15.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.046+0000 7f6c81ffb640 1 -- 172.21.15.29:0/3392535021 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c007c70 con 0x7f6c84101310 2024-03-11T19:07:15.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.046+0000 7f6c81ffb640 1 -- 172.21.15.29:0/3392535021 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6c6c007dd0 con 0x7f6c84101310 2024-03-11T19:07:15.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.046+0000 7f6c8ae64640 1 -- 172.21.15.29:0/3392535021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8419f2b0 con 0x7f6c84101310 2024-03-11T19:07:15.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.046+0000 7f6c81ffb640 1 -- 172.21.15.29:0/3392535021 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c0187c0 con 0x7f6c84101310 2024-03-11T19:07:15.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.046+0000 7f6c8ae64640 1 -- 172.21.15.29:0/3392535021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8419f7c0 con 0x7f6c84101310 2024-03-11T19:07:15.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.047+0000 7f6c8ae64640 1 -- 172.21.15.29:0/3392535021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c8410b970 con 0x7f6c84101310 2024-03-11T19:07:15.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.048+0000 7f6c81ffb640 1 -- 172.21.15.29:0/3392535021 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6c6c00f450 con 0x7f6c84101310 2024-03-11T19:07:15.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.048+0000 7f6c81ffb640 1 --2- 172.21.15.29:0/3392535021 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6c700a1a30 0x7f6c700a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:15.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.049+0000 7f6c88c00640 1 --2- 172.21.15.29:0/3392535021 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6c700a1a30 0x7f6c700a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:15.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.049+0000 7f6c81ffb640 1 -- 172.21.15.29:0/3392535021 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6c6c0cd150 con 0x7f6c84101310 2024-03-11T19:07:15.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.049+0000 7f6c88c00640 1 --2- 172.21.15.29:0/3392535021 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6c700a1a30 0x7f6c700a3ef0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6c74006010 tx=0x7f6c74000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:15.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.053+0000 7f6c81ffb640 1 -- 172.21.15.29:0/3392535021 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6c6c096800 con 0x7f6c84101310 2024-03-11T19:07:15.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:14 smithi170 ceph-mon[35690]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:15.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:14 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.190:6832/493725011,v1:172.21.15.190:6833/493725011] up:boot 2024-03-11T19:07:15.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:14 smithi170 ceph-mon[35690]: fsmap 1 up:standby 2024-03-11T19:07:15.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:14 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T19:07:15.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:14 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:15.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:14 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:15.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:14 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:15.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:14 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:15.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:14 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:15.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:14 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:15.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:14 smithi170 ceph-mon[35690]: Deploying daemon mds.k on smithi170 2024-03-11T19:07:15.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:14 smithi029 ceph-mon[32439]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:15.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:14 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.190:6832/493725011,v1:172.21.15.190:6833/493725011] up:boot 2024-03-11T19:07:15.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:14 smithi029 ceph-mon[32439]: fsmap 1 up:standby 2024-03-11T19:07:15.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T19:07:15.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:15.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:15.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:15.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:15.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:15.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:14 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:15.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:14 smithi029 ceph-mon[32439]: Deploying daemon mds.k on smithi170 2024-03-11T19:07:15.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.271+0000 7f6c8ae64640 1 -- 172.21.15.29:0/3392535021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c8419fa40 con 0x7f6c84101310 2024-03-11T19:07:15.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.271+0000 7f6c81ffb640 1 -- 172.21.15.29:0/3392535021 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1325 (secure 0 0 0) 0x7f6c6c099ec0 con 0x7f6c84101310 2024-03-11T19:07:15.275 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:15.275 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-11T19:07:15.275 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 2 2024-03-11T19:07:15.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.273+0000 7f6c8ae64640 1 -- 172.21.15.29:0/3392535021 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6c700a1a30 msgr2=0x7f6c700a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:15.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.273+0000 7f6c8ae64640 1 --2- 172.21.15.29:0/3392535021 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6c700a1a30 0x7f6c700a3ef0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6c74006010 tx=0x7f6c74000bc0 comp rx=0 tx=0).stop 2024-03-11T19:07:15.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.273+0000 7f6c8ae64640 1 -- 172.21.15.29:0/3392535021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c84101310 msgr2=0x7f6c841976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:15.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.273+0000 7f6c8ae64640 1 --2- 172.21.15.29:0/3392535021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c84101310 0x7f6c841976f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00c920 tx=0x7f6c6c00cde0 comp rx=0 tx=0).stop 2024-03-11T19:07:15.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.273+0000 7f6c8ae64640 1 -- 172.21.15.29:0/3392535021 shutdown_connections 2024-03-11T19:07:15.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.273+0000 7f6c8ae64640 1 --2- 172.21.15.29:0/3392535021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c84101cd0 0x7f6c8419e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:15.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.273+0000 7f6c8ae64640 1 --2- 172.21.15.29:0/3392535021 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6c700a1a30 0x7f6c700a3ef0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:15.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.273+0000 7f6c8ae64640 1 --2- 172.21.15.29:0/3392535021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c84101310 0x7f6c841976f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:15.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.273+0000 7f6c8ae64640 1 --2- 172.21.15.29:0/3392535021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c84100120 0x7f6c841971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:15.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.273+0000 7f6c8ae64640 1 -- 172.21.15.29:0/3392535021 >> 172.21.15.29:0/3392535021 conn(0x7f6c840fb6b0 msgr2=0x7f6c840fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:15.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.273+0000 7f6c8ae64640 1 -- 172.21.15.29:0/3392535021 shutdown_connections 2024-03-11T19:07:15.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:15.273+0000 7f6c8ae64640 1 -- 172.21.15.29:0/3392535021 wait complete. 2024-03-11T19:07:15.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:15 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3392535021' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:16.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:15 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3392535021' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:16.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:15 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3392535021' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:16.274 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:07:16 smithi170 systemd[1]: Starting Ceph mds.k for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:07:16.559 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:07:16 smithi170 podman[65429]: 2024-03-11 19:07:16.17491039 +0000 UTC m=+0.018689723 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:07:16.560 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:07:16 smithi170 podman[65429]: 2024-03-11 19:07:16.288157466 +0000 UTC m=+0.131936834 container create f35ef38595e55d38d2f499a2468eac9713d8d4aa79842c18933cce171b57add0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-k, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-03-11T19:07:16.560 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:07:16 smithi170 podman[65429]: 2024-03-11 19:07:16.557547996 +0000 UTC m=+0.401327322 container init f35ef38595e55d38d2f499a2468eac9713d8d4aa79842c18933cce171b57add0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-k, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:07:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:16 smithi170 ceph-mon[35690]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:16.847 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:07:16 smithi170 podman[65429]: 2024-03-11 19:07:16.560447379 +0000 UTC m=+0.404226719 container start f35ef38595e55d38d2f499a2468eac9713d8d4aa79842c18933cce171b57add0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-11T19:07:16.847 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:07:16 smithi170 ceph-mds[65448]: -- 172.21.15.170:0/1387656922 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5638320741a0 con 0x56383204a800 2024-03-11T19:07:16.847 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:07:16 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-k[65444]: starting mds.k at 2024-03-11T19:07:16.847 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:07:16 smithi170 bash[65429]: f35ef38595e55d38d2f499a2468eac9713d8d4aa79842c18933cce171b57add0 2024-03-11T19:07:16.847 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:07:16 smithi170 systemd[1]: Started Ceph mds.k for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:07:17.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:16 smithi029 ceph-mon[32439]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:17.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:17.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:17.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:17.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:17.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:17.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:17.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:16 smithi190 ceph-mon[36292]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:17.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:17.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:17.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:17.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:17.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:17.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:17.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:07:17.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.389+0000 7ff95b6c3640 1 -- 172.21.15.29:0/4176102667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954106e80 msgr2=0x7ff9541092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:17.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.389+0000 7ff95b6c3640 1 --2- 172.21.15.29:0/4176102667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954106e80 0x7ff9541092d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff944009ca0 tx=0x7ff94402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:07:17.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.389+0000 7ff95b6c3640 1 -- 172.21.15.29:0/4176102667 shutdown_connections 2024-03-11T19:07:17.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.389+0000 7ff95b6c3640 1 --2- 172.21.15.29:0/4176102667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954106e80 0x7ff9541092d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:17.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.389+0000 7ff95b6c3640 1 --2- 172.21.15.29:0/4176102667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff954104550 0x7ff954106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:17.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.389+0000 7ff95b6c3640 1 --2- 172.21.15.29:0/4176102667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff954101c20 0x7ff954104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:17.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.389+0000 7ff95b6c3640 1 -- 172.21.15.29:0/4176102667 >> 172.21.15.29:0/4176102667 conn(0x7ff9540fb610 msgr2=0x7ff9540fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:17.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.389+0000 7ff95b6c3640 1 -- 172.21.15.29:0/4176102667 shutdown_connections 2024-03-11T19:07:17.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.389+0000 7ff95b6c3640 1 -- 172.21.15.29:0/4176102667 wait complete. 2024-03-11T19:07:17.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff95b6c3640 1 Processor -- start 2024-03-11T19:07:17.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff95b6c3640 1 -- start start 2024-03-11T19:07:17.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff95b6c3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff954101c20 0x7ff954197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:17.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff95b6c3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954104550 0x7ff954197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:17.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff95b6c3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff954106e80 0x7ff95419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:17.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff95b6c3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95419ec20 con 0x7ff954104550 2024-03-11T19:07:17.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff95b6c3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95419ed90 con 0x7ff954101c20 2024-03-11T19:07:17.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff95b6c3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95419ef50 con 0x7ff954106e80 2024-03-11T19:07:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff958c5e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954104550 0x7ff954197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff958c5e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954104550 0x7ff954197670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42258/0 (socket says 172.21.15.29:42258) 2024-03-11T19:07:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff958c5e640 1 -- 172.21.15.29:0/323859205 learned_addr learned my addr 172.21.15.29:0/323859205 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff95945f640 1 --2- 172.21.15.29:0/323859205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff954101c20 0x7ff954197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff959c60640 1 --2- 172.21.15.29:0/323859205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff954106e80 0x7ff95419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff958c5e640 1 -- 172.21.15.29:0/323859205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff954106e80 msgr2=0x7ff95419e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff958c5e640 1 --2- 172.21.15.29:0/323859205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff954106e80 0x7ff95419e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff958c5e640 1 -- 172.21.15.29:0/323859205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff954101c20 msgr2=0x7ff954197130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff958c5e640 1 --2- 172.21.15.29:0/323859205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff954101c20 0x7ff954197130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff958c5e640 1 -- 172.21.15.29:0/323859205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff944009900 con 0x7ff954104550 2024-03-11T19:07:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff959c60640 1 --2- 172.21.15.29:0/323859205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff954106e80 0x7ff95419e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:07:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.390+0000 7ff958c5e640 1 --2- 172.21.15.29:0/323859205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954104550 0x7ff954197670 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff94800d770 tx=0x7ff94800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.391+0000 7ff9527fc640 1 -- 172.21.15.29:0/323859205 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff948009c30 con 0x7ff954104550 2024-03-11T19:07:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.391+0000 7ff9527fc640 1 -- 172.21.15.29:0/323859205 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff94800de80 con 0x7ff954104550 2024-03-11T19:07:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.391+0000 7ff95b6c3640 1 -- 172.21.15.29:0/323859205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff95419f230 con 0x7ff954104550 2024-03-11T19:07:17.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.391+0000 7ff9527fc640 1 -- 172.21.15.29:0/323859205 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94800f600 con 0x7ff954104550 2024-03-11T19:07:17.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.391+0000 7ff95b6c3640 1 -- 172.21.15.29:0/323859205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff95419f740 con 0x7ff954104550 2024-03-11T19:07:17.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.391+0000 7ff95b6c3640 1 -- 172.21.15.29:0/323859205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9540fd100 con 0x7ff954104550 2024-03-11T19:07:17.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.393+0000 7ff9527fc640 1 -- 172.21.15.29:0/323859205 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff948010460 con 0x7ff954104550 2024-03-11T19:07:17.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.394+0000 7ff9527fc640 1 --2- 172.21.15.29:0/323859205 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff93c0a1a30 0x7ff93c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:17.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.394+0000 7ff95945f640 1 --2- 172.21.15.29:0/323859205 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff93c0a1a30 0x7ff93c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:17.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.394+0000 7ff9527fc640 1 -- 172.21.15.29:0/323859205 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff9480ccce0 con 0x7ff954104550 2024-03-11T19:07:17.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.394+0000 7ff95945f640 1 --2- 172.21.15.29:0/323859205 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff93c0a1a30 0x7ff93c0a3ef0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff94c005fd0 tx=0x7ff94c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:17.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.397+0000 7ff9527fc640 1 -- 172.21.15.29:0/323859205 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff948096390 con 0x7ff954104550 2024-03-11T19:07:17.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.619+0000 7ff95b6c3640 1 -- 172.21.15.29:0/323859205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff95410cb80 con 0x7ff954104550 2024-03-11T19:07:17.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.619+0000 7ff9527fc640 1 -- 172.21.15.29:0/323859205 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2119 (secure 0 0 0) 0x7ff948021860 con 0x7ff954104550 2024-03-11T19:07:17.622 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:17.622 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-11T19:07:17.623 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 3 2024-03-11T19:07:17.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.621+0000 7ff95b6c3640 1 -- 172.21.15.29:0/323859205 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff93c0a1a30 msgr2=0x7ff93c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:17.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.621+0000 7ff95b6c3640 1 --2- 172.21.15.29:0/323859205 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff93c0a1a30 0x7ff93c0a3ef0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff94c005fd0 tx=0x7ff94c00b390 comp rx=0 tx=0).stop 2024-03-11T19:07:17.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.621+0000 7ff95b6c3640 1 -- 172.21.15.29:0/323859205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954104550 msgr2=0x7ff954197670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:17.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.621+0000 7ff95b6c3640 1 --2- 172.21.15.29:0/323859205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954104550 0x7ff954197670 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff94800d770 tx=0x7ff94800dc30 comp rx=0 tx=0).stop 2024-03-11T19:07:17.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.621+0000 7ff95b6c3640 1 -- 172.21.15.29:0/323859205 shutdown_connections 2024-03-11T19:07:17.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.621+0000 7ff95b6c3640 1 --2- 172.21.15.29:0/323859205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff954106e80 0x7ff95419e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:17.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.621+0000 7ff95b6c3640 1 --2- 172.21.15.29:0/323859205 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff93c0a1a30 0x7ff93c0a3ef0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:17.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.621+0000 7ff95b6c3640 1 --2- 172.21.15.29:0/323859205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954104550 0x7ff954197670 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:17.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.621+0000 7ff95b6c3640 1 --2- 172.21.15.29:0/323859205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff954101c20 0x7ff954197130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:17.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.621+0000 7ff95b6c3640 1 -- 172.21.15.29:0/323859205 >> 172.21.15.29:0/323859205 conn(0x7ff9540fb610 msgr2=0x7ff954102580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:17.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.621+0000 7ff95b6c3640 1 -- 172.21.15.29:0/323859205 shutdown_connections 2024-03-11T19:07:17.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:17.621+0000 7ff95b6c3640 1 -- 172.21.15.29:0/323859205 wait complete. 2024-03-11T19:07:17.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:17 smithi029 ceph-mon[32439]: Deploying daemon mds.j on smithi029 2024-03-11T19:07:17.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:17 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.170:6832/1035391240,v1:172.21.15.170:6833/1035391240] up:boot 2024-03-11T19:07:17.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:17 smithi029 ceph-mon[32439]: fsmap 2 up:standby 2024-03-11T19:07:17.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:17 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T19:07:17.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:17 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/323859205' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:18.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:17 smithi170 ceph-mon[35690]: Deploying daemon mds.j on smithi029 2024-03-11T19:07:18.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:17 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.170:6832/1035391240,v1:172.21.15.170:6833/1035391240] up:boot 2024-03-11T19:07:18.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:17 smithi170 ceph-mon[35690]: fsmap 2 up:standby 2024-03-11T19:07:18.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:17 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T19:07:18.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:17 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/323859205' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:18.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:17 smithi190 ceph-mon[36292]: Deploying daemon mds.j on smithi029 2024-03-11T19:07:18.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:17 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.170:6832/1035391240,v1:172.21.15.170:6833/1035391240] up:boot 2024-03-11T19:07:18.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:17 smithi190 ceph-mon[36292]: fsmap 2 up:standby 2024-03-11T19:07:18.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:17 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T19:07:18.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:17 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/323859205' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:18.971 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:18 smithi170 ceph-mon[35690]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:18.994 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:18 smithi029 ceph-mon[32439]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:19.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:18 smithi190 ceph-mon[36292]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:19.384 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 11 19:07:19 smithi029 systemd[1]: Starting Ceph mds.j for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:07:19.637 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:07:19.647 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 11 19:07:19 smithi029 podman[72667]: 2024-03-11 19:07:19.284228193 +0000 UTC m=+0.017911401 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:07:19.647 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 11 19:07:19 smithi029 podman[72667]: 2024-03-11 19:07:19.407188634 +0000 UTC m=+0.140871829 container create bbbf59d44f5797c2c253c2a3e709e3df3e722e40ca41fb437042c09ffb3c4147 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-j, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:07:19.648 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 11 19:07:19 smithi029 podman[72667]: 2024-03-11 19:07:19.645010619 +0000 UTC m=+0.378693832 container init bbbf59d44f5797c2c253c2a3e709e3df3e722e40ca41fb437042c09ffb3c4147 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-j, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-11T19:07:19.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.743+0000 7fc15809d640 1 -- 172.21.15.29:0/2846653919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc150101c70 msgr2=0x7fc150108500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:19.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.743+0000 7fc15809d640 1 --2- 172.21.15.29:0/2846653919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc150101c70 0x7fc150108500 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc148009ca0 tx=0x7fc14802c0f0 comp rx=0 tx=0).stop 2024-03-11T19:07:19.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.743+0000 7fc15809d640 1 -- 172.21.15.29:0/2846653919 shutdown_connections 2024-03-11T19:07:19.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.743+0000 7fc15809d640 1 --2- 172.21.15.29:0/2846653919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc150101c70 0x7fc150108500 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:19.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.743+0000 7fc15809d640 1 --2- 172.21.15.29:0/2846653919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1501012b0 0x7fc150101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:19.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.743+0000 7fc15809d640 1 --2- 172.21.15.29:0/2846653919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1501000c0 0x7fc1501004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:19.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.743+0000 7fc15809d640 1 -- 172.21.15.29:0/2846653919 >> 172.21.15.29:0/2846653919 conn(0x7fc1500fb630 msgr2=0x7fc1500fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:19.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.743+0000 7fc15809d640 1 -- 172.21.15.29:0/2846653919 shutdown_connections 2024-03-11T19:07:19.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.743+0000 7fc15809d640 1 -- 172.21.15.29:0/2846653919 wait complete. 2024-03-11T19:07:19.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc15809d640 1 Processor -- start 2024-03-11T19:07:19.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc15809d640 1 -- start start 2024-03-11T19:07:19.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc15809d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1501000c0 0x7fc150104050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:19.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc15809d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1501012b0 0x7fc150102690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:19.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc15809d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc150101c70 0x7fc150102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc15809d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc150103330 con 0x7fc1501000c0 2024-03-11T19:07:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc15809d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1501034a0 con 0x7fc150101c70 2024-03-11T19:07:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc15809d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1501a1180 con 0x7fc1501012b0 2024-03-11T19:07:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc155e39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1501000c0 0x7fc150104050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc155e39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1501000c0 0x7fc150104050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42294/0 (socket says 172.21.15.29:42294) 2024-03-11T19:07:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc155e39640 1 -- 172.21.15.29:0/2383958302 learned_addr learned my addr 172.21.15.29:0/2383958302 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc15663a640 1 --2- 172.21.15.29:0/2383958302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc150101c70 0x7fc150102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc155638640 1 --2- 172.21.15.29:0/2383958302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1501012b0 0x7fc150102690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc155e39640 1 -- 172.21.15.29:0/2383958302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1501012b0 msgr2=0x7fc150102690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc155e39640 1 --2- 172.21.15.29:0/2383958302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1501012b0 0x7fc150102690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc155e39640 1 -- 172.21.15.29:0/2383958302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc150101c70 msgr2=0x7fc150102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc155e39640 1 --2- 172.21.15.29:0/2383958302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc150101c70 0x7fc150102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc155e39640 1 -- 172.21.15.29:0/2383958302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc148009900 con 0x7fc1501000c0 2024-03-11T19:07:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc15663a640 1 --2- 172.21.15.29:0/2383958302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc150101c70 0x7fc150102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:07:19.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc155638640 1 --2- 172.21.15.29:0/2383958302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1501012b0 0x7fc150102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:07:19.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.744+0000 7fc155e39640 1 --2- 172.21.15.29:0/2383958302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1501000c0 0x7fc150104050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc14000c9a0 tx=0x7fc14000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:19.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.745+0000 7fc146ffd640 1 -- 172.21.15.29:0/2383958302 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc140004220 con 0x7fc1501000c0 2024-03-11T19:07:19.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.745+0000 7fc15809d640 1 -- 172.21.15.29:0/2383958302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1501a1430 con 0x7fc1501000c0 2024-03-11T19:07:19.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.745+0000 7fc146ffd640 1 -- 172.21.15.29:0/2383958302 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc140007e60 con 0x7fc1501000c0 2024-03-11T19:07:19.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.745+0000 7fc146ffd640 1 -- 172.21.15.29:0/2383958302 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc140004a00 con 0x7fc1501000c0 2024-03-11T19:07:19.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.745+0000 7fc15809d640 1 -- 172.21.15.29:0/2383958302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1501a1940 con 0x7fc1501000c0 2024-03-11T19:07:19.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.746+0000 7fc15809d640 1 -- 172.21.15.29:0/2383958302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc15010b980 con 0x7fc1501000c0 2024-03-11T19:07:19.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.747+0000 7fc146ffd640 1 -- 172.21.15.29:0/2383958302 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc1400079b0 con 0x7fc1501000c0 2024-03-11T19:07:19.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.748+0000 7fc146ffd640 1 --2- 172.21.15.29:0/2383958302 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc13c0a1a30 0x7fc13c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:19.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.748+0000 7fc155638640 1 --2- 172.21.15.29:0/2383958302 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc13c0a1a30 0x7fc13c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:19.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.748+0000 7fc146ffd640 1 -- 172.21.15.29:0/2383958302 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc1400cd020 con 0x7fc1501000c0 2024-03-11T19:07:19.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.749+0000 7fc155638640 1 --2- 172.21.15.29:0/2383958302 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc13c0a1a30 0x7fc13c0a3ef0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc150103dc0 tx=0x7fc14c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:19.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.752+0000 7fc146ffd640 1 -- 172.21.15.29:0/2383958302 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc1400966d0 con 0x7fc1501000c0 2024-03-11T19:07:19.906 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 11 19:07:19 smithi029 podman[72667]: 2024-03-11 19:07:19.648860294 +0000 UTC m=+0.382543499 container start bbbf59d44f5797c2c253c2a3e709e3df3e722e40ca41fb437042c09ffb3c4147 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-j, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:07:19.906 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 11 19:07:19 smithi029 ceph-mds[72701]: -- 172.21.15.29:0/3540728288 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555ccfcdc1a0 con 0x555ccfcb2800 2024-03-11T19:07:19.906 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 11 19:07:19 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-j[72682]: starting mds.j at 2024-03-11T19:07:19.906 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 11 19:07:19 smithi029 bash[72667]: bbbf59d44f5797c2c253c2a3e709e3df3e722e40ca41fb437042c09ffb3c4147 2024-03-11T19:07:19.906 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 11 19:07:19 smithi029 systemd[1]: Started Ceph mds.j for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:07:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.983+0000 7fc15809d640 1 -- 172.21.15.29:0/2383958302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc15010bb10 con 0x7fc1501000c0 2024-03-11T19:07:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.984+0000 7fc146ffd640 1 -- 172.21.15.29:0/2383958302 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2907 (secure 0 0 0) 0x7fc140099d90 con 0x7fc1501000c0 2024-03-11T19:07:19.987 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:19.987 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-11T19:07:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 4 2024-03-11T19:07:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.986+0000 7fc15809d640 1 -- 172.21.15.29:0/2383958302 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc13c0a1a30 msgr2=0x7fc13c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.986+0000 7fc15809d640 1 --2- 172.21.15.29:0/2383958302 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc13c0a1a30 0x7fc13c0a3ef0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc150103dc0 tx=0x7fc14c005f90 comp rx=0 tx=0).stop 2024-03-11T19:07:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.986+0000 7fc15809d640 1 -- 172.21.15.29:0/2383958302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1501000c0 msgr2=0x7fc150104050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.986+0000 7fc15809d640 1 --2- 172.21.15.29:0/2383958302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1501000c0 0x7fc150104050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc14000c9a0 tx=0x7fc14000ce60 comp rx=0 tx=0).stop 2024-03-11T19:07:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.986+0000 7fc15809d640 1 -- 172.21.15.29:0/2383958302 shutdown_connections 2024-03-11T19:07:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.986+0000 7fc15809d640 1 --2- 172.21.15.29:0/2383958302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc150101c70 0x7fc150102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.986+0000 7fc15809d640 1 --2- 172.21.15.29:0/2383958302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1501012b0 0x7fc150102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.986+0000 7fc15809d640 1 --2- 172.21.15.29:0/2383958302 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc13c0a1a30 0x7fc13c0a3ef0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.986+0000 7fc15809d640 1 --2- 172.21.15.29:0/2383958302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1501000c0 0x7fc150104050 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.986+0000 7fc15809d640 1 -- 172.21.15.29:0/2383958302 >> 172.21.15.29:0/2383958302 conn(0x7fc1500fb630 msgr2=0x7fc1500fd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.986+0000 7fc15809d640 1 -- 172.21.15.29:0/2383958302 shutdown_connections 2024-03-11T19:07:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:19.986+0000 7fc15809d640 1 -- 172.21.15.29:0/2383958302 wait complete. 2024-03-11T19:07:21.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:20 smithi170 ceph-mon[35690]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:21.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:20 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.29:6834/630953303,v1:172.21.15.29:6835/630953303] up:boot 2024-03-11T19:07:21.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:20 smithi170 ceph-mon[35690]: fsmap 3 up:standby 2024-03-11T19:07:21.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-11T19:07:21.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:21.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:21.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:21.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:21.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:21.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:21.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:20 smithi170 ceph-mon[35690]: Deploying daemon mds.d on smithi029 2024-03-11T19:07:21.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:20 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2383958302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:21.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:20 smithi029 ceph-mon[32439]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:21.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:20 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.29:6834/630953303,v1:172.21.15.29:6835/630953303] up:boot 2024-03-11T19:07:21.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:20 smithi029 ceph-mon[32439]: fsmap 3 up:standby 2024-03-11T19:07:21.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-11T19:07:21.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:21.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:21.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:21.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:21.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:21.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:21.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:20 smithi029 ceph-mon[32439]: Deploying daemon mds.d on smithi029 2024-03-11T19:07:21.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:20 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2383958302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:21.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:20 smithi190 ceph-mon[36292]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:21.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:20 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.29:6834/630953303,v1:172.21.15.29:6835/630953303] up:boot 2024-03-11T19:07:21.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:20 smithi190 ceph-mon[36292]: fsmap 3 up:standby 2024-03-11T19:07:21.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-11T19:07:21.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:21.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:21.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:21.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:21.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:21.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:21.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:20 smithi190 ceph-mon[36292]: Deploying daemon mds.d on smithi029 2024-03-11T19:07:21.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:20 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2383958302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:22.001 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:07:22.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.171+0000 7f6360e8f640 1 -- 172.21.15.29:0/2405363094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f635c1012a0 msgr2=0x7f635c101720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:22.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.171+0000 7f6360e8f640 1 --2- 172.21.15.29:0/2405363094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f635c1012a0 0x7f635c101720 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6344009900 tx=0x7f634402bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:22.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.171+0000 7f6360e8f640 1 -- 172.21.15.29:0/2405363094 shutdown_connections 2024-03-11T19:07:22.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.171+0000 7f6360e8f640 1 --2- 172.21.15.29:0/2405363094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f635c101c60 0x7f635c1084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:22.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.171+0000 7f6360e8f640 1 --2- 172.21.15.29:0/2405363094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f635c1012a0 0x7f635c101720 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:22.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.171+0000 7f6360e8f640 1 --2- 172.21.15.29:0/2405363094 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f635c1000b0 0x7f635c1004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:22.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.171+0000 7f6360e8f640 1 -- 172.21.15.29:0/2405363094 >> 172.21.15.29:0/2405363094 conn(0x7f635c0fb640 msgr2=0x7f635c0fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:22.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.171+0000 7f6360e8f640 1 -- 172.21.15.29:0/2405363094 shutdown_connections 2024-03-11T19:07:22.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.171+0000 7f6360e8f640 1 -- 172.21.15.29:0/2405363094 wait complete. 2024-03-11T19:07:22.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.172+0000 7f6360e8f640 1 Processor -- start 2024-03-11T19:07:22.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.172+0000 7f6360e8f640 1 -- start start 2024-03-11T19:07:22.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.172+0000 7f6360e8f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f635c1000b0 0x7f635c1971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:22.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.172+0000 7f6360e8f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f635c1012a0 0x7f635c1976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.172+0000 7f6360e8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f635c101c60 0x7f635c19e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.172+0000 7f6360e8f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c19ec90 con 0x7f635c101c60 2024-03-11T19:07:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.172+0000 7f6360e8f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c19ee00 con 0x7f635c1000b0 2024-03-11T19:07:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.172+0000 7f6360e8f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c19efc0 con 0x7f635c1012a0 2024-03-11T19:07:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.173+0000 7f635ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f635c101c60 0x7f635c19e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.173+0000 7f635ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f635c101c60 0x7f635c19e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42316/0 (socket says 172.21.15.29:42316) 2024-03-11T19:07:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.173+0000 7f635ad9d640 1 -- 172.21.15.29:0/223939733 learned_addr learned my addr 172.21.15.29:0/223939733 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.173+0000 7f6353fff640 1 --2- 172.21.15.29:0/223939733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f635c1012a0 0x7f635c1976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.173+0000 7f635a59c640 1 --2- 172.21.15.29:0/223939733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f635c1000b0 0x7f635c1971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.173+0000 7f635ad9d640 1 -- 172.21.15.29:0/223939733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f635c1012a0 msgr2=0x7f635c1976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.173+0000 7f635ad9d640 1 --2- 172.21.15.29:0/223939733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f635c1012a0 0x7f635c1976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.173+0000 7f635ad9d640 1 -- 172.21.15.29:0/223939733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f635c1000b0 msgr2=0x7f635c1971a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.173+0000 7f635ad9d640 1 --2- 172.21.15.29:0/223939733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f635c1000b0 0x7f635c1971a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.173+0000 7f635ad9d640 1 -- 172.21.15.29:0/223939733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6344009560 con 0x7f635c101c60 2024-03-11T19:07:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.173+0000 7f635ad9d640 1 --2- 172.21.15.29:0/223939733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f635c101c60 0x7f635c19e750 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f635400eb10 tx=0x7f635400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:22.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.173+0000 7f63537fe640 1 -- 172.21.15.29:0/223939733 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6354004b20 con 0x7f635c101c60 2024-03-11T19:07:22.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.173+0000 7f6360e8f640 1 -- 172.21.15.29:0/223939733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635c19f2a0 con 0x7f635c101c60 2024-03-11T19:07:22.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.173+0000 7f63537fe640 1 -- 172.21.15.29:0/223939733 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6354004c80 con 0x7f635c101c60 2024-03-11T19:07:22.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.173+0000 7f63537fe640 1 -- 172.21.15.29:0/223939733 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6354010690 con 0x7f635c101c60 2024-03-11T19:07:22.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.173+0000 7f6360e8f640 1 -- 172.21.15.29:0/223939733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635c19f730 con 0x7f635c101c60 2024-03-11T19:07:22.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.175+0000 7f6360e8f640 1 -- 172.21.15.29:0/223939733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f635c10b970 con 0x7f635c101c60 2024-03-11T19:07:22.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.176+0000 7f63537fe640 1 -- 172.21.15.29:0/223939733 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6354005020 con 0x7f635c101c60 2024-03-11T19:07:22.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.176+0000 7f63537fe640 1 --2- 172.21.15.29:0/223939733 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f63480a1a30 0x7f63480a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:22.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.176+0000 7f63537fe640 1 -- 172.21.15.29:0/223939733 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6354014070 con 0x7f635c101c60 2024-03-11T19:07:22.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.176+0000 7f635a59c640 1 --2- 172.21.15.29:0/223939733 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f63480a1a30 0x7f63480a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:22.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.177+0000 7f635a59c640 1 --2- 172.21.15.29:0/223939733 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f63480a1a30 0x7f63480a3ef0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f635c101100 tx=0x7f634c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:22.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.181+0000 7f63537fe640 1 -- 172.21.15.29:0/223939733 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6354096bc0 con 0x7f635c101c60 2024-03-11T19:07:22.332 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 11 19:07:22 smithi029 systemd[1]: Starting Ceph mds.d for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:07:22.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.405+0000 7f6360e8f640 1 -- 172.21.15.29:0/223939733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f635c10bb00 con 0x7f635c101c60 2024-03-11T19:07:22.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.405+0000 7f63537fe640 1 -- 172.21.15.29:0/223939733 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2907 (secure 0 0 0) 0x7f635c10bb00 con 0x7f635c101c60 2024-03-11T19:07:22.408 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:22.408 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-11T19:07:22.408 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 4 2024-03-11T19:07:22.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.407+0000 7f6360e8f640 1 -- 172.21.15.29:0/223939733 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f63480a1a30 msgr2=0x7f63480a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:22.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.407+0000 7f6360e8f640 1 --2- 172.21.15.29:0/223939733 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f63480a1a30 0x7f63480a3ef0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f635c101100 tx=0x7f634c005f90 comp rx=0 tx=0).stop 2024-03-11T19:07:22.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.407+0000 7f6360e8f640 1 -- 172.21.15.29:0/223939733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f635c101c60 msgr2=0x7f635c19e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:22.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.407+0000 7f6360e8f640 1 --2- 172.21.15.29:0/223939733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f635c101c60 0x7f635c19e750 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f635400eb10 tx=0x7f635400c2d0 comp rx=0 tx=0).stop 2024-03-11T19:07:22.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.407+0000 7f6360e8f640 1 -- 172.21.15.29:0/223939733 shutdown_connections 2024-03-11T19:07:22.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.407+0000 7f6360e8f640 1 --2- 172.21.15.29:0/223939733 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f63480a1a30 0x7f63480a3ef0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:22.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.407+0000 7f6360e8f640 1 --2- 172.21.15.29:0/223939733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f635c101c60 0x7f635c19e750 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:22.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.407+0000 7f6360e8f640 1 --2- 172.21.15.29:0/223939733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f635c1012a0 0x7f635c1976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:22.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.407+0000 7f6360e8f640 1 --2- 172.21.15.29:0/223939733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f635c1000b0 0x7f635c1971a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:22.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.407+0000 7f6360e8f640 1 -- 172.21.15.29:0/223939733 >> 172.21.15.29:0/223939733 conn(0x7f635c0fb640 msgr2=0x7f635c0fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:22.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.407+0000 7f6360e8f640 1 -- 172.21.15.29:0/223939733 shutdown_connections 2024-03-11T19:07:22.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:22.407+0000 7f6360e8f640 1 -- 172.21.15.29:0/223939733 wait complete. 2024-03-11T19:07:22.609 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 11 19:07:22 smithi029 podman[73056]: 2024-03-11 19:07:22.231945694 +0000 UTC m=+0.017851682 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:07:22.610 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 11 19:07:22 smithi029 podman[73056]: 2024-03-11 19:07:22.351152939 +0000 UTC m=+0.137058918 container create 3e10f89d8bfd0151fc03c71070937771920a284616fc6032bc09f14c421608d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-11T19:07:22.610 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 11 19:07:22 smithi029 podman[73056]: 2024-03-11 19:07:22.607608135 +0000 UTC m=+0.393514118 container init 3e10f89d8bfd0151fc03c71070937771920a284616fc6032bc09f14c421608d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-11T19:07:22.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:22 smithi029 ceph-mon[32439]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:22.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:22 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/223939733' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:22.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:22 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:22.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:22 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:22.906 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 11 19:07:22 smithi029 podman[73056]: 2024-03-11 19:07:22.610288676 +0000 UTC m=+0.396194657 container start 3e10f89d8bfd0151fc03c71070937771920a284616fc6032bc09f14c421608d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-11T19:07:22.906 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 11 19:07:22 smithi029 ceph-mds[73077]: -- 172.21.15.29:0/4197035535 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5595687f01a0 con 0x5595687c6c00 2024-03-11T19:07:22.906 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 11 19:07:22 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-d[73073]: starting mds.d at 2024-03-11T19:07:22.906 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 11 19:07:22 smithi029 bash[73056]: 3e10f89d8bfd0151fc03c71070937771920a284616fc6032bc09f14c421608d7 2024-03-11T19:07:22.907 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 11 19:07:22 smithi029 systemd[1]: Started Ceph mds.d for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:07:23.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:22 smithi170 ceph-mon[35690]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:23.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:22 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/223939733' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:23.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:22 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:23.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:22 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:23.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:22 smithi190 ceph-mon[36292]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:23.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:22 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/223939733' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:23.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:22 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:23.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:22 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:24.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:23 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.29:6836/368479661,v1:172.21.15.29:6837/368479661] up:boot 2024-03-11T19:07:24.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:23 smithi170 ceph-mon[35690]: fsmap 4 up:standby 2024-03-11T19:07:24.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:23 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-11T19:07:24.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:23 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:24.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:23 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:24.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:23 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:24.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:23 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:24.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:23 smithi170 ceph-mon[35690]: Deploying daemon mds.e on smithi170 2024-03-11T19:07:24.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:23 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.29:6836/368479661,v1:172.21.15.29:6837/368479661] up:boot 2024-03-11T19:07:24.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:23 smithi029 ceph-mon[32439]: fsmap 4 up:standby 2024-03-11T19:07:24.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-11T19:07:24.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:24.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:24.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:24.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:23 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:24.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:23 smithi029 ceph-mon[32439]: Deploying daemon mds.e on smithi170 2024-03-11T19:07:24.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:23 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.29:6836/368479661,v1:172.21.15.29:6837/368479661] up:boot 2024-03-11T19:07:24.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:23 smithi190 ceph-mon[36292]: fsmap 4 up:standby 2024-03-11T19:07:24.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:23 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-11T19:07:24.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:23 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:24.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:23 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:24.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:23 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:24.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:23 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:24.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:23 smithi190 ceph-mon[36292]: Deploying daemon mds.e on smithi170 2024-03-11T19:07:24.422 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:07:24.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.536+0000 7fd23a18c640 1 -- 172.21.15.29:0/632419761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd234073b60 msgr2=0x7fd234073fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:24.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.536+0000 7fd23a18c640 1 --2- 172.21.15.29:0/632419761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd234073b60 0x7fd234073fc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd228009900 tx=0x7fd22802bcd0 comp rx=0 tx=0).stop 2024-03-11T19:07:24.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.536+0000 7fd23a18c640 1 -- 172.21.15.29:0/632419761 shutdown_connections 2024-03-11T19:07:24.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.536+0000 7fd23a18c640 1 --2- 172.21.15.29:0/632419761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd234109060 0x7fd23410b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:24.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.536+0000 7fd23a18c640 1 --2- 172.21.15.29:0/632419761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd234073b60 0x7fd234073fc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:24.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.536+0000 7fd23a18c640 1 --2- 172.21.15.29:0/632419761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd234075140 0x7fd234073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:24.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.536+0000 7fd23a18c640 1 -- 172.21.15.29:0/632419761 >> 172.21.15.29:0/632419761 conn(0x7fd2340fc760 msgr2=0x7fd2340feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:24.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.536+0000 7fd23a18c640 1 -- 172.21.15.29:0/632419761 shutdown_connections 2024-03-11T19:07:24.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.536+0000 7fd23a18c640 1 -- 172.21.15.29:0/632419761 wait complete. 2024-03-11T19:07:24.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.536+0000 7fd23a18c640 1 Processor -- start 2024-03-11T19:07:24.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.537+0000 7fd23a18c640 1 -- start start 2024-03-11T19:07:24.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.537+0000 7fd23a18c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd234073b60 0x7fd2341993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:24.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.537+0000 7fd23a18c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd234075140 0x7fd2341998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:24.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.537+0000 7fd23a18c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd234109060 0x7fd2341a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:24.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.537+0000 7fd23a18c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2341a0ea0 con 0x7fd234073b60 2024-03-11T19:07:24.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.537+0000 7fd23a18c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2341a1010 con 0x7fd234075140 2024-03-11T19:07:24.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.537+0000 7fd23a18c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2341a11d0 con 0x7fd234109060 2024-03-11T19:07:24.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.537+0000 7fd2337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd234073b60 0x7fd2341993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:24.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.537+0000 7fd2337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd234073b60 0x7fd2341993b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42356/0 (socket says 172.21.15.29:42356) 2024-03-11T19:07:24.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.537+0000 7fd2337fe640 1 -- 172.21.15.29:0/3210239586 learned_addr learned my addr 172.21.15.29:0/3210239586 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:24.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.537+0000 7fd232ffd640 1 --2- 172.21.15.29:0/3210239586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd234075140 0x7fd2341998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:24.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.537+0000 7fd233fff640 1 --2- 172.21.15.29:0/3210239586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd234109060 0x7fd2341a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:24.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.537+0000 7fd2337fe640 1 -- 172.21.15.29:0/3210239586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd234109060 msgr2=0x7fd2341a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:24.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.537+0000 7fd2337fe640 1 --2- 172.21.15.29:0/3210239586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd234109060 0x7fd2341a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:24.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.537+0000 7fd2337fe640 1 -- 172.21.15.29:0/3210239586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd234075140 msgr2=0x7fd2341998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:24.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.537+0000 7fd2337fe640 1 --2- 172.21.15.29:0/3210239586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd234075140 0x7fd2341998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:24.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.537+0000 7fd2337fe640 1 -- 172.21.15.29:0/3210239586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd228009560 con 0x7fd234073b60 2024-03-11T19:07:24.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.537+0000 7fd2337fe640 1 --2- 172.21.15.29:0/3210239586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd234073b60 0x7fd2341993b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd22400eb90 tx=0x7fd22400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:24.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.538+0000 7fd230ff9640 1 -- 172.21.15.29:0/3210239586 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2240099b0 con 0x7fd234073b60 2024-03-11T19:07:24.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.538+0000 7fd230ff9640 1 -- 172.21.15.29:0/3210239586 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd22400ee90 con 0x7fd234073b60 2024-03-11T19:07:24.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.538+0000 7fd23a18c640 1 -- 172.21.15.29:0/3210239586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2341a14b0 con 0x7fd234073b60 2024-03-11T19:07:24.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.538+0000 7fd230ff9640 1 -- 172.21.15.29:0/3210239586 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd224010690 con 0x7fd234073b60 2024-03-11T19:07:24.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.538+0000 7fd23a18c640 1 -- 172.21.15.29:0/3210239586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2341a1940 con 0x7fd234073b60 2024-03-11T19:07:24.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.539+0000 7fd23a18c640 1 -- 172.21.15.29:0/3210239586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2340683c0 con 0x7fd234073b60 2024-03-11T19:07:24.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.540+0000 7fd230ff9640 1 -- 172.21.15.29:0/3210239586 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd224005000 con 0x7fd234073b60 2024-03-11T19:07:24.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.540+0000 7fd230ff9640 1 --2- 172.21.15.29:0/3210239586 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd21c0a1a30 0x7fd21c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:24.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.541+0000 7fd232ffd640 1 --2- 172.21.15.29:0/3210239586 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd21c0a1a30 0x7fd21c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:24.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.541+0000 7fd230ff9640 1 -- 172.21.15.29:0/3210239586 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd224014070 con 0x7fd234073b60 2024-03-11T19:07:24.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.541+0000 7fd232ffd640 1 --2- 172.21.15.29:0/3210239586 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd21c0a1a30 0x7fd21c0a3ef0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd228009230 tx=0x7fd228005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:24.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.544+0000 7fd230ff9640 1 -- 172.21.15.29:0/3210239586 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd224097110 con 0x7fd234073b60 2024-03-11T19:07:24.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.764+0000 7fd23a18c640 1 -- 172.21.15.29:0/3210239586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd234107e20 con 0x7fd234073b60 2024-03-11T19:07:24.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.764+0000 7fd230ff9640 1 -- 172.21.15.29:0/3210239586 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3695 (secure 0 0 0) 0x7fd22409a7d0 con 0x7fd234073b60 2024-03-11T19:07:24.767 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:24.767 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-11T19:07:24.767 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 5 2024-03-11T19:07:24.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.766+0000 7fd23a18c640 1 -- 172.21.15.29:0/3210239586 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd21c0a1a30 msgr2=0x7fd21c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:24.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.766+0000 7fd23a18c640 1 --2- 172.21.15.29:0/3210239586 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd21c0a1a30 0x7fd21c0a3ef0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd228009230 tx=0x7fd228005fd0 comp rx=0 tx=0).stop 2024-03-11T19:07:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.766+0000 7fd23a18c640 1 -- 172.21.15.29:0/3210239586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd234073b60 msgr2=0x7fd2341993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.766+0000 7fd23a18c640 1 --2- 172.21.15.29:0/3210239586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd234073b60 0x7fd2341993b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd22400eb90 tx=0x7fd22400c6a0 comp rx=0 tx=0).stop 2024-03-11T19:07:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.766+0000 7fd23a18c640 1 -- 172.21.15.29:0/3210239586 shutdown_connections 2024-03-11T19:07:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.766+0000 7fd23a18c640 1 --2- 172.21.15.29:0/3210239586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd234109060 0x7fd2341a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.766+0000 7fd23a18c640 1 --2- 172.21.15.29:0/3210239586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd234075140 0x7fd2341998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.766+0000 7fd23a18c640 1 --2- 172.21.15.29:0/3210239586 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd21c0a1a30 0x7fd21c0a3ef0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.766+0000 7fd23a18c640 1 --2- 172.21.15.29:0/3210239586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd234073b60 0x7fd2341993b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.766+0000 7fd23a18c640 1 -- 172.21.15.29:0/3210239586 >> 172.21.15.29:0/3210239586 conn(0x7fd2340fc760 msgr2=0x7fd234104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.766+0000 7fd23a18c640 1 -- 172.21.15.29:0/3210239586 shutdown_connections 2024-03-11T19:07:24.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:24.766+0000 7fd23a18c640 1 -- 172.21.15.29:0/3210239586 wait complete. 2024-03-11T19:07:25.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:24 smithi170 ceph-mon[35690]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:25.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:24 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3210239586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:25.096 INFO:journalctl@ceph.mds.e.smithi170.stdout:Mar 11 19:07:25 smithi170 systemd[1]: Starting Ceph mds.e for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:07:25.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:24 smithi029 ceph-mon[32439]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:25.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:24 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3210239586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:25.163 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:24 smithi190 ceph-mon[36292]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:25.163 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:24 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3210239586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:25.408 INFO:journalctl@ceph.mds.e.smithi170.stdout:Mar 11 19:07:25 smithi170 podman[65745]: 2024-03-11 19:07:25.165088464 +0000 UTC m=+0.018377225 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:07:25.408 INFO:journalctl@ceph.mds.e.smithi170.stdout:Mar 11 19:07:25 smithi170 podman[65745]: 2024-03-11 19:07:25.287047358 +0000 UTC m=+0.140336188 container create ce0f0e897c09006ae16ec94a50e179ee076aaa86c799d91a54ba86a9b5ff94fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-e, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-11T19:07:25.846 INFO:journalctl@ceph.mds.e.smithi170.stdout:Mar 11 19:07:25 smithi170 podman[65745]: 2024-03-11 19:07:25.548172248 +0000 UTC m=+0.401461025 container init ce0f0e897c09006ae16ec94a50e179ee076aaa86c799d91a54ba86a9b5ff94fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-e, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-11T19:07:25.846 INFO:journalctl@ceph.mds.e.smithi170.stdout:Mar 11 19:07:25 smithi170 podman[65745]: 2024-03-11 19:07:25.550911949 +0000 UTC m=+0.404200713 container start ce0f0e897c09006ae16ec94a50e179ee076aaa86c799d91a54ba86a9b5ff94fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-e, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-11T19:07:25.846 INFO:journalctl@ceph.mds.e.smithi170.stdout:Mar 11 19:07:25 smithi170 ceph-mds[65764]: -- 172.21.15.170:0/1938071862 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55afb32401a0 con 0x55afb3217000 2024-03-11T19:07:25.846 INFO:journalctl@ceph.mds.e.smithi170.stdout:Mar 11 19:07:25 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-e[65760]: starting mds.e at 2024-03-11T19:07:25.846 INFO:journalctl@ceph.mds.e.smithi170.stdout:Mar 11 19:07:25 smithi170 bash[65745]: ce0f0e897c09006ae16ec94a50e179ee076aaa86c799d91a54ba86a9b5ff94fd 2024-03-11T19:07:25.846 INFO:journalctl@ceph.mds.e.smithi170.stdout:Mar 11 19:07:25 smithi170 systemd[1]: Started Ceph mds.e for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:07:26.782 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:07:26.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.886+0000 7f72763a0640 1 -- 172.21.15.29:0/2529425789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72701043e0 msgr2=0x7f72701067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:26.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.886+0000 7f72763a0640 1 --2- 172.21.15.29:0/2529425789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72701043e0 0x7f72701067d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7260009900 tx=0x7f726002bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:26.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.886+0000 7f72763a0640 1 -- 172.21.15.29:0/2529425789 shutdown_connections 2024-03-11T19:07:26.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.886+0000 7f72763a0640 1 --2- 172.21.15.29:0/2529425789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7270106d10 0x7f7270109100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.886+0000 7f72763a0640 1 --2- 172.21.15.29:0/2529425789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72701043e0 0x7f72701067d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.886+0000 7f72763a0640 1 --2- 172.21.15.29:0/2529425789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7270101ab0 0x7f7270103ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.886+0000 7f72763a0640 1 -- 172.21.15.29:0/2529425789 >> 172.21.15.29:0/2529425789 conn(0x7f72700fb460 msgr2=0x7f72700fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.886+0000 7f72763a0640 1 -- 172.21.15.29:0/2529425789 shutdown_connections 2024-03-11T19:07:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.886+0000 7f72763a0640 1 -- 172.21.15.29:0/2529425789 wait complete. 2024-03-11T19:07:26.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.887+0000 7f72763a0640 1 Processor -- start 2024-03-11T19:07:26.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.887+0000 7f72763a0640 1 -- start start 2024-03-11T19:07:26.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.887+0000 7f72763a0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7270101ab0 0x7f7270196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:26.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.887+0000 7f72763a0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72701043e0 0x7f7270197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:26.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.887+0000 7f72763a0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7270106d10 0x7f727019e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:26.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.887+0000 7f72763a0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727019e9f0 con 0x7f72701043e0 2024-03-11T19:07:26.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.887+0000 7f72763a0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727019eb60 con 0x7f7270101ab0 2024-03-11T19:07:26.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.887+0000 7f72763a0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727019ed20 con 0x7f7270106d10 2024-03-11T19:07:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.887+0000 7f7274b9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72701043e0 0x7f7270197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.887+0000 7f7274b9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72701043e0 0x7f7270197440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50868/0 (socket says 172.21.15.29:50868) 2024-03-11T19:07:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.887+0000 7f7274b9d640 1 -- 172.21.15.29:0/3767889964 learned_addr learned my addr 172.21.15.29:0/3767889964 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.887+0000 7f7275b9f640 1 --2- 172.21.15.29:0/3767889964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7270106d10 0x7f727019e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.887+0000 7f727539e640 1 --2- 172.21.15.29:0/3767889964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7270101ab0 0x7f7270196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.888+0000 7f7274b9d640 1 -- 172.21.15.29:0/3767889964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7270106d10 msgr2=0x7f727019e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.888+0000 7f7274b9d640 1 --2- 172.21.15.29:0/3767889964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7270106d10 0x7f727019e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.888+0000 7f7274b9d640 1 -- 172.21.15.29:0/3767889964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7270101ab0 msgr2=0x7f7270196f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.888+0000 7f7274b9d640 1 --2- 172.21.15.29:0/3767889964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7270101ab0 0x7f7270196f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.888+0000 7f7274b9d640 1 -- 172.21.15.29:0/3767889964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7260009560 con 0x7f72701043e0 2024-03-11T19:07:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.888+0000 7f7274b9d640 1 --2- 172.21.15.29:0/3767889964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72701043e0 0x7f7270197440 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f726002c7d0 tx=0x7f72600042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.888+0000 7f72667fc640 1 -- 172.21.15.29:0/3767889964 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7260004aa0 con 0x7f72701043e0 2024-03-11T19:07:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.888+0000 7f72667fc640 1 -- 172.21.15.29:0/3767889964 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f726002de90 con 0x7f72701043e0 2024-03-11T19:07:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.888+0000 7f72763a0640 1 -- 172.21.15.29:0/3767889964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f727019efa0 con 0x7f72701043e0 2024-03-11T19:07:26.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.888+0000 7f72667fc640 1 -- 172.21.15.29:0/3767889964 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7260005440 con 0x7f72701043e0 2024-03-11T19:07:26.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.888+0000 7f72763a0640 1 -- 172.21.15.29:0/3767889964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f727019f420 con 0x7f72701043e0 2024-03-11T19:07:26.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.889+0000 7f72763a0640 1 -- 172.21.15.29:0/3767889964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72700fcf50 con 0x7f72701043e0 2024-03-11T19:07:26.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.890+0000 7f72667fc640 1 -- 172.21.15.29:0/3767889964 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f72600055a0 con 0x7f72701043e0 2024-03-11T19:07:26.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.890+0000 7f72667fc640 1 --2- 172.21.15.29:0/3767889964 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f725c0a1a30 0x7f725c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:26.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.891+0000 7f727539e640 1 --2- 172.21.15.29:0/3767889964 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f725c0a1a30 0x7f725c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:26.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.891+0000 7f72667fc640 1 -- 172.21.15.29:0/3767889964 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f72600ef640 con 0x7f72701043e0 2024-03-11T19:07:26.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.891+0000 7f727539e640 1 --2- 172.21.15.29:0/3767889964 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f725c0a1a30 0x7f725c0a3ef0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f726c005e10 tx=0x7f726c005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:26.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:26.894+0000 7f72667fc640 1 -- 172.21.15.29:0/3767889964 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f72600b8cf0 con 0x7f72701043e0 2024-03-11T19:07:26.983 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:26 smithi190 ceph-mon[36292]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:26.984 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:26 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:26.984 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:26 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:26.984 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:26 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:26.984 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:26 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:26.984 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:26 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:26.984 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:26 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:26.984 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:26 smithi190 ceph-mon[36292]: Deploying daemon mds.f on smithi190 2024-03-11T19:07:26.984 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:26 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.170:6834/2555518515,v1:172.21.15.170:6835/2555518515] up:boot 2024-03-11T19:07:26.984 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:26 smithi190 ceph-mon[36292]: fsmap 5 up:standby 2024-03-11T19:07:26.984 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:26 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T19:07:27.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:26 smithi170 ceph-mon[35690]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:27.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:26 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:27.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:26 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:27.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:26 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:27.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:26 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:27.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:26 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:27.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:26 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:27.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:26 smithi170 ceph-mon[35690]: Deploying daemon mds.f on smithi190 2024-03-11T19:07:27.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:26 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.170:6834/2555518515,v1:172.21.15.170:6835/2555518515] up:boot 2024-03-11T19:07:27.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:26 smithi170 ceph-mon[35690]: fsmap 5 up:standby 2024-03-11T19:07:27.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:26 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T19:07:27.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:27.112+0000 7f72763a0640 1 -- 172.21.15.29:0/3767889964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f727010c980 con 0x7f72701043e0 2024-03-11T19:07:27.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:27.113+0000 7f72667fc640 1 -- 172.21.15.29:0/3767889964 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4489 (secure 0 0 0) 0x7f72600bc3b0 con 0x7f72701043e0 2024-03-11T19:07:27.116 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:27.116 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-11T19:07:27.116 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 6 2024-03-11T19:07:27.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:27.115+0000 7f72763a0640 1 -- 172.21.15.29:0/3767889964 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f725c0a1a30 msgr2=0x7f725c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:27.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:27.115+0000 7f72763a0640 1 --2- 172.21.15.29:0/3767889964 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f725c0a1a30 0x7f725c0a3ef0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f726c005e10 tx=0x7f726c005da0 comp rx=0 tx=0).stop 2024-03-11T19:07:27.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:27.115+0000 7f72763a0640 1 -- 172.21.15.29:0/3767889964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72701043e0 msgr2=0x7f7270197440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:27.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:27.115+0000 7f72763a0640 1 --2- 172.21.15.29:0/3767889964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72701043e0 0x7f7270197440 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f726002c7d0 tx=0x7f72600042f0 comp rx=0 tx=0).stop 2024-03-11T19:07:27.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:27.115+0000 7f72763a0640 1 -- 172.21.15.29:0/3767889964 shutdown_connections 2024-03-11T19:07:27.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:27.115+0000 7f72763a0640 1 --2- 172.21.15.29:0/3767889964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7270106d10 0x7f727019e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:27.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:27.115+0000 7f72763a0640 1 --2- 172.21.15.29:0/3767889964 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f725c0a1a30 0x7f725c0a3ef0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:27.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:27.115+0000 7f72763a0640 1 --2- 172.21.15.29:0/3767889964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72701043e0 0x7f7270197440 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:27.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:27.115+0000 7f72763a0640 1 --2- 172.21.15.29:0/3767889964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7270101ab0 0x7f7270196f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:27.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:27.115+0000 7f72763a0640 1 -- 172.21.15.29:0/3767889964 >> 172.21.15.29:0/3767889964 conn(0x7f72700fb460 msgr2=0x7f7270102350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:27.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:27.115+0000 7f72763a0640 1 -- 172.21.15.29:0/3767889964 shutdown_connections 2024-03-11T19:07:27.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:27.115+0000 7f72763a0640 1 -- 172.21.15.29:0/3767889964 wait complete. 2024-03-11T19:07:27.127 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:26 smithi029 ceph-mon[32439]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:27.127 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:26 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:27.128 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:26 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:27.128 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:26 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:27.128 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:26 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:27.128 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:26 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:27.128 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:26 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:27.128 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:26 smithi029 ceph-mon[32439]: Deploying daemon mds.f on smithi190 2024-03-11T19:07:27.128 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:26 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.170:6834/2555518515,v1:172.21.15.170:6835/2555518515] up:boot 2024-03-11T19:07:27.128 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:26 smithi029 ceph-mon[32439]: fsmap 5 up:standby 2024-03-11T19:07:27.128 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:26 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T19:07:28.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:27 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3767889964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:28.012 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:07:27 smithi190 systemd[1]: Starting Ceph mds.f for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:07:28.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:27 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3767889964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:28.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:27 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3767889964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:28.369 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:07:28 smithi190 podman[66377]: 2024-03-11 19:07:28.006845815 +0000 UTC m=+0.017490798 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:07:28.370 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:07:28 smithi190 podman[66377]: 2024-03-11 19:07:28.123193951 +0000 UTC m=+0.133838923 container create 5cb8dff6283739da1dfc1043fe1fce1d0ec129a63ef9ba3f6c81e589e2933e57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-f, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-11T19:07:28.762 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:07:28 smithi190 podman[66377]: 2024-03-11 19:07:28.367700232 +0000 UTC m=+0.378345205 container init 5cb8dff6283739da1dfc1043fe1fce1d0ec129a63ef9ba3f6c81e589e2933e57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-f, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-11T19:07:28.762 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:07:28 smithi190 podman[66377]: 2024-03-11 19:07:28.37022096 +0000 UTC m=+0.380865946 container start 5cb8dff6283739da1dfc1043fe1fce1d0ec129a63ef9ba3f6c81e589e2933e57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-f, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, ceph=True) 2024-03-11T19:07:28.762 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:07:28 smithi190 ceph-mds[66396]: -- 172.21.15.190:0/1708804675 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d13e4961a0 con 0x55d13e46c800 2024-03-11T19:07:28.762 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:07:28 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-f[66392]: starting mds.f at 2024-03-11T19:07:28.762 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:07:28 smithi190 bash[66377]: 5cb8dff6283739da1dfc1043fe1fce1d0ec129a63ef9ba3f6c81e589e2933e57 2024-03-11T19:07:28.762 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:07:28 smithi190 systemd[1]: Started Ceph mds.f for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:07:29.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:07:29.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.242+0000 7f4f441e9640 1 -- 172.21.15.29:0/2361367939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f3c106e80 msgr2=0x7f4f3c1092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:29.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.242+0000 7f4f441e9640 1 --2- 172.21.15.29:0/2361367939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f3c106e80 0x7f4f3c1092d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4f38009ca0 tx=0x7f4f3802c0f0 comp rx=0 tx=0).stop 2024-03-11T19:07:29.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.242+0000 7f4f441e9640 1 -- 172.21.15.29:0/2361367939 shutdown_connections 2024-03-11T19:07:29.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.242+0000 7f4f441e9640 1 --2- 172.21.15.29:0/2361367939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f3c106e80 0x7f4f3c1092d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:29.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.242+0000 7f4f441e9640 1 --2- 172.21.15.29:0/2361367939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c104550 0x7f4f3c106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:29.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.242+0000 7f4f441e9640 1 --2- 172.21.15.29:0/2361367939 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f3c101c20 0x7f4f3c104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:29.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.242+0000 7f4f441e9640 1 -- 172.21.15.29:0/2361367939 >> 172.21.15.29:0/2361367939 conn(0x7f4f3c0fb610 msgr2=0x7f4f3c0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:29.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.243+0000 7f4f441e9640 1 -- 172.21.15.29:0/2361367939 shutdown_connections 2024-03-11T19:07:29.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.243+0000 7f4f441e9640 1 -- 172.21.15.29:0/2361367939 wait complete. 2024-03-11T19:07:29.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.243+0000 7f4f441e9640 1 Processor -- start 2024-03-11T19:07:29.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.243+0000 7f4f441e9640 1 -- start start 2024-03-11T19:07:29.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.243+0000 7f4f441e9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c101c20 0x7f4f3c101590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:29.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.243+0000 7f4f441e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f3c104550 0x7f4f3c0ffbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:29.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.243+0000 7f4f441e9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f3c106e80 0x7f4f3c100110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:29.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.243+0000 7f4f441e9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c100870 con 0x7f4f3c104550 2024-03-11T19:07:29.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.243+0000 7f4f441e9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c1009e0 con 0x7f4f3c106e80 2024-03-11T19:07:29.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.243+0000 7f4f441e9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c1a1190 con 0x7f4f3c101c20 2024-03-11T19:07:29.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.243+0000 7f4f41784640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f3c104550 0x7f4f3c0ffbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:29.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.244+0000 7f4f41784640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f3c104550 0x7f4f3c0ffbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50874/0 (socket says 172.21.15.29:50874) 2024-03-11T19:07:29.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.244+0000 7f4f41784640 1 -- 172.21.15.29:0/123390172 learned_addr learned my addr 172.21.15.29:0/123390172 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:29.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.244+0000 7f4f41f85640 1 --2- 172.21.15.29:0/123390172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c101c20 0x7f4f3c101590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:29.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.244+0000 7f4f42786640 1 --2- 172.21.15.29:0/123390172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f3c106e80 0x7f4f3c100110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:29.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.244+0000 7f4f41784640 1 -- 172.21.15.29:0/123390172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c101c20 msgr2=0x7f4f3c101590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:29.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.244+0000 7f4f41784640 1 --2- 172.21.15.29:0/123390172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c101c20 0x7f4f3c101590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:29.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.244+0000 7f4f41784640 1 -- 172.21.15.29:0/123390172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f3c106e80 msgr2=0x7f4f3c100110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:29.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.244+0000 7f4f41784640 1 --2- 172.21.15.29:0/123390172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f3c106e80 0x7f4f3c100110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:29.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.244+0000 7f4f41784640 1 -- 172.21.15.29:0/123390172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f38009900 con 0x7f4f3c104550 2024-03-11T19:07:29.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.244+0000 7f4f41784640 1 --2- 172.21.15.29:0/123390172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f3c104550 0x7f4f3c0ffbd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00d770 tx=0x7f4f2c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:29.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.244+0000 7f4f32ffd640 1 -- 172.21.15.29:0/123390172 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c009c30 con 0x7f4f3c104550 2024-03-11T19:07:29.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.244+0000 7f4f32ffd640 1 -- 172.21.15.29:0/123390172 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4f2c00de80 con 0x7f4f3c104550 2024-03-11T19:07:29.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.244+0000 7f4f441e9640 1 -- 172.21.15.29:0/123390172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f3c1a1440 con 0x7f4f3c104550 2024-03-11T19:07:29.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.244+0000 7f4f32ffd640 1 -- 172.21.15.29:0/123390172 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c004e40 con 0x7f4f3c104550 2024-03-11T19:07:29.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.244+0000 7f4f441e9640 1 -- 172.21.15.29:0/123390172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f3c1a1950 con 0x7f4f3c104550 2024-03-11T19:07:29.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.245+0000 7f4f441e9640 1 -- 172.21.15.29:0/123390172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f3c0fd100 con 0x7f4f3c104550 2024-03-11T19:07:29.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.246+0000 7f4f32ffd640 1 -- 172.21.15.29:0/123390172 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4f2c004fa0 con 0x7f4f3c104550 2024-03-11T19:07:29.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.247+0000 7f4f32ffd640 1 --2- 172.21.15.29:0/123390172 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4f280a1ab0 0x7f4f280a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:29.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.247+0000 7f4f41f85640 1 --2- 172.21.15.29:0/123390172 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4f280a1ab0 0x7f4f280a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:29.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.247+0000 7f4f32ffd640 1 -- 172.21.15.29:0/123390172 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4f2c0cccc0 con 0x7f4f3c104550 2024-03-11T19:07:29.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.247+0000 7f4f41f85640 1 --2- 172.21.15.29:0/123390172 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4f280a1ab0 0x7f4f280a3f70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4f340058c0 tx=0x7f4f3400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:29.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.250+0000 7f4f32ffd640 1 -- 172.21.15.29:0/123390172 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4f2c096400 con 0x7f4f3c104550 2024-03-11T19:07:29.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.469+0000 7f4f441e9640 1 -- 172.21.15.29:0/123390172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f3c10ca60 con 0x7f4f3c104550 2024-03-11T19:07:29.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.469+0000 7f4f32ffd640 1 -- 172.21.15.29:0/123390172 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5283 (secure 0 0 0) 0x7f4f2c020020 con 0x7f4f3c104550 2024-03-11T19:07:29.473 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:29.473 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-11T19:07:29.473 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 7 2024-03-11T19:07:29.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.471+0000 7f4f441e9640 1 -- 172.21.15.29:0/123390172 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4f280a1ab0 msgr2=0x7f4f280a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:29.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.471+0000 7f4f441e9640 1 --2- 172.21.15.29:0/123390172 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4f280a1ab0 0x7f4f280a3f70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4f340058c0 tx=0x7f4f3400a2d0 comp rx=0 tx=0).stop 2024-03-11T19:07:29.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.471+0000 7f4f441e9640 1 -- 172.21.15.29:0/123390172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f3c104550 msgr2=0x7f4f3c0ffbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:29.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.471+0000 7f4f441e9640 1 --2- 172.21.15.29:0/123390172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f3c104550 0x7f4f3c0ffbd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00d770 tx=0x7f4f2c00dc30 comp rx=0 tx=0).stop 2024-03-11T19:07:29.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.471+0000 7f4f441e9640 1 -- 172.21.15.29:0/123390172 shutdown_connections 2024-03-11T19:07:29.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.471+0000 7f4f441e9640 1 --2- 172.21.15.29:0/123390172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f3c106e80 0x7f4f3c100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:29.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.471+0000 7f4f441e9640 1 --2- 172.21.15.29:0/123390172 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4f280a1ab0 0x7f4f280a3f70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:29.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.471+0000 7f4f441e9640 1 --2- 172.21.15.29:0/123390172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f3c104550 0x7f4f3c0ffbd0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:29.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.471+0000 7f4f441e9640 1 --2- 172.21.15.29:0/123390172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c101c20 0x7f4f3c101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:29.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.471+0000 7f4f441e9640 1 -- 172.21.15.29:0/123390172 >> 172.21.15.29:0/123390172 conn(0x7f4f3c0fb610 msgr2=0x7f4f3c102580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:29.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.471+0000 7f4f441e9640 1 -- 172.21.15.29:0/123390172 shutdown_connections 2024-03-11T19:07:29.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:29.471+0000 7f4f441e9640 1 -- 172.21.15.29:0/123390172 wait complete. 2024-03-11T19:07:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:29 smithi190 ceph-mon[36292]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:29 smithi190 ceph-mon[36292]: Deploying daemon mds.c on smithi190 2024-03-11T19:07:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:29 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] up:boot 2024-03-11T19:07:29.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:29 smithi190 ceph-mon[36292]: fsmap 6 up:standby 2024-03-11T19:07:29.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:29 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T19:07:29.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:29 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/123390172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:29.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:29 smithi170 ceph-mon[35690]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:29.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:29.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:29.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:29.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:29.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:29.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:29.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:29 smithi170 ceph-mon[35690]: Deploying daemon mds.c on smithi190 2024-03-11T19:07:29.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:29 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] up:boot 2024-03-11T19:07:29.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:29 smithi170 ceph-mon[35690]: fsmap 6 up:standby 2024-03-11T19:07:29.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:29 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T19:07:29.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:29 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/123390172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:29.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:29 smithi029 ceph-mon[32439]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:29.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:29.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:29.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:29.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:29.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:29.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:29.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:29 smithi029 ceph-mon[32439]: Deploying daemon mds.c on smithi190 2024-03-11T19:07:29.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:29 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] up:boot 2024-03-11T19:07:29.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:29 smithi029 ceph-mon[32439]: fsmap 6 up:standby 2024-03-11T19:07:29.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:29 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T19:07:29.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:29 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/123390172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:30.762 INFO:journalctl@ceph.mds.c.smithi190.stdout:Mar 11 19:07:30 smithi190 systemd[1]: Starting Ceph mds.c for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:07:31.069 INFO:journalctl@ceph.mds.c.smithi190.stdout:Mar 11 19:07:30 smithi190 podman[66683]: 2024-03-11 19:07:30.717798775 +0000 UTC m=+0.018827652 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:07:31.069 INFO:journalctl@ceph.mds.c.smithi190.stdout:Mar 11 19:07:30 smithi190 podman[66683]: 2024-03-11 19:07:30.819962468 +0000 UTC m=+0.120991346 container create 656094af5d2608fc1c96ddd144fe5a6dd814113f1cdc80d15e8656c900f486c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-c, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-11T19:07:31.069 INFO:journalctl@ceph.mds.c.smithi190.stdout:Mar 11 19:07:31 smithi190 podman[66683]: 2024-03-11 19:07:31.06467949 +0000 UTC m=+0.365708367 container init 656094af5d2608fc1c96ddd144fe5a6dd814113f1cdc80d15e8656c900f486c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2) 2024-03-11T19:07:31.069 INFO:journalctl@ceph.mds.c.smithi190.stdout:Mar 11 19:07:31 smithi190 podman[66683]: 2024-03-11 19:07:31.067357809 +0000 UTC m=+0.368386696 container start 656094af5d2608fc1c96ddd144fe5a6dd814113f1cdc80d15e8656c900f486c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:07:31.485 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:07:31.511 INFO:journalctl@ceph.mds.c.smithi190.stdout:Mar 11 19:07:31 smithi190 ceph-mds[66701]: -- 172.21.15.190:0/2301024420 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fad089e1a0 con 0x55fad0874800 2024-03-11T19:07:31.512 INFO:journalctl@ceph.mds.c.smithi190.stdout:Mar 11 19:07:31 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-c[66697]: starting mds.c at 2024-03-11T19:07:31.512 INFO:journalctl@ceph.mds.c.smithi190.stdout:Mar 11 19:07:31 smithi190 bash[66683]: 656094af5d2608fc1c96ddd144fe5a6dd814113f1cdc80d15e8656c900f486c7 2024-03-11T19:07:31.512 INFO:journalctl@ceph.mds.c.smithi190.stdout:Mar 11 19:07:31 smithi190 systemd[1]: Started Ceph mds.c for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:07:31.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.598+0000 7f1557e9a640 1 -- 172.21.15.29:0/2477017582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1550100d10 msgr2=0x7f1550101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:31.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.598+0000 7f1557e9a640 1 --2- 172.21.15.29:0/2477017582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1550100d10 0x7f1550101190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1548009ca0 tx=0x7f154802c0f0 comp rx=0 tx=0).stop 2024-03-11T19:07:31.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.598+0000 7f1557e9a640 1 -- 172.21.15.29:0/2477017582 shutdown_connections 2024-03-11T19:07:31.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.598+0000 7f1557e9a640 1 --2- 172.21.15.29:0/2477017582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1550100d10 0x7f1550101190 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:31.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.598+0000 7f1557e9a640 1 --2- 172.21.15.29:0/2477017582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1550100370 0x7f15501007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:31.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.598+0000 7f1557e9a640 1 --2- 172.21.15.29:0/2477017582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15500ffa10 0x7f15500ffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:31.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.598+0000 7f1557e9a640 1 -- 172.21.15.29:0/2477017582 >> 172.21.15.29:0/2477017582 conn(0x7f15500fb610 msgr2=0x7f15500fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:31.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.598+0000 7f1557e9a640 1 -- 172.21.15.29:0/2477017582 shutdown_connections 2024-03-11T19:07:31.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.598+0000 7f1557e9a640 1 -- 172.21.15.29:0/2477017582 wait complete. 2024-03-11T19:07:31.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.598+0000 7f1557e9a640 1 Processor -- start 2024-03-11T19:07:31.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.599+0000 7f1557e9a640 1 -- start start 2024-03-11T19:07:31.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.599+0000 7f1557e9a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15500ffa10 0x7f1550199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:31.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.599+0000 7f1557e9a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1550100370 0x7f1550199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:31.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.599+0000 7f1557e9a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1550100d10 0x7f15501a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:31.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.599+0000 7f1557e9a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15501a0df0 con 0x7f15500ffa10 2024-03-11T19:07:31.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.599+0000 7f1557e9a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15501a0f60 con 0x7f1550100370 2024-03-11T19:07:31.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.599+0000 7f1557e9a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15501a1120 con 0x7f1550100d10 2024-03-11T19:07:31.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.599+0000 7f1555c36640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15500ffa10 0x7f1550199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:31.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.599+0000 7f1555c36640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15500ffa10 0x7f1550199300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50904/0 (socket says 172.21.15.29:50904) 2024-03-11T19:07:31.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.599+0000 7f1555c36640 1 -- 172.21.15.29:0/1232266751 learned_addr learned my addr 172.21.15.29:0/1232266751 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:31.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.599+0000 7f1555435640 1 --2- 172.21.15.29:0/1232266751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1550100370 0x7f1550199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:31.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.599+0000 7f1556437640 1 --2- 172.21.15.29:0/1232266751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1550100d10 0x7f15501a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:31.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.599+0000 7f1555c36640 1 -- 172.21.15.29:0/1232266751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1550100d10 msgr2=0x7f15501a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:31.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.599+0000 7f1555c36640 1 --2- 172.21.15.29:0/1232266751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1550100d10 0x7f15501a08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:31.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.599+0000 7f1555c36640 1 -- 172.21.15.29:0/1232266751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1550100370 msgr2=0x7f1550199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:31.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.599+0000 7f1555c36640 1 --2- 172.21.15.29:0/1232266751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1550100370 0x7f1550199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:31.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.599+0000 7f1555c36640 1 -- 172.21.15.29:0/1232266751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1548009900 con 0x7f15500ffa10 2024-03-11T19:07:31.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.599+0000 7f1555c36640 1 --2- 172.21.15.29:0/1232266751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15500ffa10 0x7f1550199300 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f154c00d770 tx=0x7f154c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:31.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.599+0000 7f1546ffd640 1 -- 172.21.15.29:0/1232266751 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c009c30 con 0x7f15500ffa10 2024-03-11T19:07:31.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.600+0000 7f1546ffd640 1 -- 172.21.15.29:0/1232266751 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f154c00de80 con 0x7f15500ffa10 2024-03-11T19:07:31.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.600+0000 7f1557e9a640 1 -- 172.21.15.29:0/1232266751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15501a1400 con 0x7f15500ffa10 2024-03-11T19:07:31.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.600+0000 7f1546ffd640 1 -- 172.21.15.29:0/1232266751 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c004e00 con 0x7f15500ffa10 2024-03-11T19:07:31.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.600+0000 7f1557e9a640 1 -- 172.21.15.29:0/1232266751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15501a1910 con 0x7f15500ffa10 2024-03-11T19:07:31.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.601+0000 7f1557e9a640 1 -- 172.21.15.29:0/1232266751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f155010a8e0 con 0x7f15500ffa10 2024-03-11T19:07:31.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.602+0000 7f1546ffd640 1 -- 172.21.15.29:0/1232266751 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f154c004f60 con 0x7f15500ffa10 2024-03-11T19:07:31.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.602+0000 7f1546ffd640 1 --2- 172.21.15.29:0/1232266751 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f153c0a1ab0 0x7f153c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:31.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.602+0000 7f1555435640 1 --2- 172.21.15.29:0/1232266751 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f153c0a1ab0 0x7f153c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:31.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.602+0000 7f1546ffd640 1 -- 172.21.15.29:0/1232266751 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f154c0cc890 con 0x7f15500ffa10 2024-03-11T19:07:31.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.603+0000 7f1555435640 1 --2- 172.21.15.29:0/1232266751 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f153c0a1ab0 0x7f153c0a3f70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f155019a810 tx=0x7f154000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:31.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.607+0000 7f1546ffd640 1 -- 172.21.15.29:0/1232266751 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f154c095ff0 con 0x7f15500ffa10 2024-03-11T19:07:31.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.828+0000 7f1557e9a640 1 -- 172.21.15.29:0/1232266751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f155010ec90 con 0x7f15500ffa10 2024-03-11T19:07:31.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.828+0000 7f1546ffd640 1 -- 172.21.15.29:0/1232266751 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6077 (secure 0 0 0) 0x7f155010ec90 con 0x7f15500ffa10 2024-03-11T19:07:31.832 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:31.832 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-11T19:07:31.832 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2024-03-11T19:07:31.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.830+0000 7f1557e9a640 1 -- 172.21.15.29:0/1232266751 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f153c0a1ab0 msgr2=0x7f153c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:31.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.830+0000 7f1557e9a640 1 --2- 172.21.15.29:0/1232266751 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f153c0a1ab0 0x7f153c0a3f70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f155019a810 tx=0x7f154000b410 comp rx=0 tx=0).stop 2024-03-11T19:07:31.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.830+0000 7f1557e9a640 1 -- 172.21.15.29:0/1232266751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15500ffa10 msgr2=0x7f1550199300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:31.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.830+0000 7f1557e9a640 1 --2- 172.21.15.29:0/1232266751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15500ffa10 0x7f1550199300 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f154c00d770 tx=0x7f154c00dc30 comp rx=0 tx=0).stop 2024-03-11T19:07:31.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.830+0000 7f1557e9a640 1 -- 172.21.15.29:0/1232266751 shutdown_connections 2024-03-11T19:07:31.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.830+0000 7f1557e9a640 1 --2- 172.21.15.29:0/1232266751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1550100d10 0x7f15501a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:31.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.830+0000 7f1557e9a640 1 --2- 172.21.15.29:0/1232266751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1550100370 0x7f1550199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:31.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.830+0000 7f1557e9a640 1 --2- 172.21.15.29:0/1232266751 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f153c0a1ab0 0x7f153c0a3f70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:31.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.830+0000 7f1557e9a640 1 --2- 172.21.15.29:0/1232266751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15500ffa10 0x7f1550199300 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:31.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.830+0000 7f1557e9a640 1 -- 172.21.15.29:0/1232266751 >> 172.21.15.29:0/1232266751 conn(0x7f15500fb610 msgr2=0x7f15500fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:31.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.830+0000 7f1557e9a640 1 -- 172.21.15.29:0/1232266751 shutdown_connections 2024-03-11T19:07:31.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:31.831+0000 7f1557e9a640 1 -- 172.21.15.29:0/1232266751 wait complete. 2024-03-11T19:07:31.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:31 smithi029 ceph-mon[32439]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:31.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:31 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:31.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:31 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:31.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:31 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:31.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:31 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:31.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:31 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:31.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:31 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:31.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:31 smithi170 ceph-mon[35690]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:31.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:31 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:31.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:31 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:31.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:31 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:31.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:31 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:31.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:31 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:31.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:31 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:32.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:31 smithi190 ceph-mon[36292]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:32.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:31 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:32.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:31 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:32.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:31 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:32.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:31 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:32.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:31 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:32.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:31 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:32.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:32 smithi170 ceph-mon[35690]: Deploying daemon mds.h on smithi170 2024-03-11T19:07:32.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:32 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] up:boot 2024-03-11T19:07:32.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:32 smithi170 ceph-mon[35690]: fsmap 7 up:standby 2024-03-11T19:07:32.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:32 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T19:07:32.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:32 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1232266751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:32.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:32 smithi029 ceph-mon[32439]: Deploying daemon mds.h on smithi170 2024-03-11T19:07:32.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:32 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] up:boot 2024-03-11T19:07:32.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:32 smithi029 ceph-mon[32439]: fsmap 7 up:standby 2024-03-11T19:07:32.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:32 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T19:07:32.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:32 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1232266751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:33.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:32 smithi190 ceph-mon[36292]: Deploying daemon mds.h on smithi170 2024-03-11T19:07:33.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:32 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] up:boot 2024-03-11T19:07:33.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:32 smithi190 ceph-mon[36292]: fsmap 7 up:standby 2024-03-11T19:07:33.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:32 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T19:07:33.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:32 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1232266751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:33.532 INFO:journalctl@ceph.mds.h.smithi170.stdout:Mar 11 19:07:33 smithi170 systemd[1]: Starting Ceph mds.h for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:07:33.843 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:33 smithi170 ceph-mon[35690]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:33.843 INFO:journalctl@ceph.mds.h.smithi170.stdout:Mar 11 19:07:33 smithi170 podman[66054]: 2024-03-11 19:07:33.576260336 +0000 UTC m=+0.017919415 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:07:33.843 INFO:journalctl@ceph.mds.h.smithi170.stdout:Mar 11 19:07:33 smithi170 podman[66054]: 2024-03-11 19:07:33.693717205 +0000 UTC m=+0.135376308 container create 9a47d71e1dc3db0db8cdbd2d4dd82c350188c83d4ed3a170d45b1a06f24e43d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-h, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-11T19:07:33.846 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:07:33.874 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:33 smithi029 ceph-mon[32439]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:33.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.957+0000 7f234ac15640 1 -- 172.21.15.29:0/1454148276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2344101c70 msgr2=0x7f2344108500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:33.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.957+0000 7f234ac15640 1 --2- 172.21.15.29:0/1454148276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2344101c70 0x7f2344108500 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f233c009ca0 tx=0x7f233c02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:07:33.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.957+0000 7f234ac15640 1 -- 172.21.15.29:0/1454148276 shutdown_connections 2024-03-11T19:07:33.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.957+0000 7f234ac15640 1 --2- 172.21.15.29:0/1454148276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2344101c70 0x7f2344108500 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:33.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.957+0000 7f234ac15640 1 --2- 172.21.15.29:0/1454148276 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23441012b0 0x7f2344101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:33.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.957+0000 7f234ac15640 1 --2- 172.21.15.29:0/1454148276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23441000c0 0x7f23441004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:33.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.957+0000 7f234ac15640 1 -- 172.21.15.29:0/1454148276 >> 172.21.15.29:0/1454148276 conn(0x7f23440fb630 msgr2=0x7f23440fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:33.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.957+0000 7f234ac15640 1 -- 172.21.15.29:0/1454148276 shutdown_connections 2024-03-11T19:07:33.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.957+0000 7f234ac15640 1 -- 172.21.15.29:0/1454148276 wait complete. 2024-03-11T19:07:33.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.958+0000 7f234ac15640 1 Processor -- start 2024-03-11T19:07:33.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.958+0000 7f234ac15640 1 -- start start 2024-03-11T19:07:33.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.958+0000 7f234ac15640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23441000c0 0x7f23441970a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:33.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.958+0000 7f234ac15640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23441012b0 0x7f23441975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:33.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.958+0000 7f234ac15640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2344101c70 0x7f234419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:33.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.958+0000 7f234ac15640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234419eb90 con 0x7f23441012b0 2024-03-11T19:07:33.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.958+0000 7f234ac15640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234419ed00 con 0x7f2344101c70 2024-03-11T19:07:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.958+0000 7f234ac15640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234419eec0 con 0x7f23441000c0 2024-03-11T19:07:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.958+0000 7f2343fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23441012b0 0x7f23441975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.958+0000 7f2343fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23441012b0 0x7f23441975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50922/0 (socket says 172.21.15.29:50922) 2024-03-11T19:07:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.958+0000 7f2343fff640 1 -- 172.21.15.29:0/2303606087 learned_addr learned my addr 172.21.15.29:0/2303606087 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.958+0000 7f23489b1640 1 --2- 172.21.15.29:0/2303606087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23441000c0 0x7f23441970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.958+0000 7f23491b2640 1 --2- 172.21.15.29:0/2303606087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2344101c70 0x7f234419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.958+0000 7f2343fff640 1 -- 172.21.15.29:0/2303606087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23441000c0 msgr2=0x7f23441970a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.958+0000 7f2343fff640 1 --2- 172.21.15.29:0/2303606087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23441000c0 0x7f23441970a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.958+0000 7f2343fff640 1 -- 172.21.15.29:0/2303606087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2344101c70 msgr2=0x7f234419e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.958+0000 7f2343fff640 1 --2- 172.21.15.29:0/2303606087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2344101c70 0x7f234419e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.958+0000 7f2343fff640 1 -- 172.21.15.29:0/2303606087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233c009900 con 0x7f23441012b0 2024-03-11T19:07:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.959+0000 7f2343fff640 1 --2- 172.21.15.29:0/2303606087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23441012b0 0x7f23441975e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f233000c920 tx=0x7f233000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.959+0000 7f2341ffb640 1 -- 172.21.15.29:0/2303606087 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2330007c70 con 0x7f23441012b0 2024-03-11T19:07:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.959+0000 7f2341ffb640 1 -- 172.21.15.29:0/2303606087 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2330007dd0 con 0x7f23441012b0 2024-03-11T19:07:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.959+0000 7f234ac15640 1 -- 172.21.15.29:0/2303606087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f234419f1a0 con 0x7f23441012b0 2024-03-11T19:07:33.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.959+0000 7f2341ffb640 1 -- 172.21.15.29:0/2303606087 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23300187c0 con 0x7f23441012b0 2024-03-11T19:07:33.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.959+0000 7f234ac15640 1 -- 172.21.15.29:0/2303606087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f234419f700 con 0x7f23441012b0 2024-03-11T19:07:33.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.960+0000 7f234ac15640 1 -- 172.21.15.29:0/2303606087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f234410b980 con 0x7f23441012b0 2024-03-11T19:07:33.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.961+0000 7f2341ffb640 1 -- 172.21.15.29:0/2303606087 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f233000f450 con 0x7f23441012b0 2024-03-11T19:07:33.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.962+0000 7f2341ffb640 1 --2- 172.21.15.29:0/2303606087 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23340a1ab0 0x7f23340a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:33.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.962+0000 7f23489b1640 1 --2- 172.21.15.29:0/2303606087 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23340a1ab0 0x7f23340a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:33.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.962+0000 7f2341ffb640 1 -- 172.21.15.29:0/2303606087 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f23300cd0f0 con 0x7f23441012b0 2024-03-11T19:07:33.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.962+0000 7f23489b1640 1 --2- 172.21.15.29:0/2303606087 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23340a1ab0 0x7f23340a3f70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2344101110 tx=0x7f2338005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:33.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:33.966+0000 7f2341ffb640 1 -- 172.21.15.29:0/2303606087 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f23300967a0 con 0x7f23441012b0 2024-03-11T19:07:34.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:33 smithi190 ceph-mon[36292]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:34.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:34.188+0000 7f234ac15640 1 -- 172.21.15.29:0/2303606087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f234419f980 con 0x7f23441012b0 2024-03-11T19:07:34.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:34.189+0000 7f2341ffb640 1 -- 172.21.15.29:0/2303606087 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6077 (secure 0 0 0) 0x7f2330099e60 con 0x7f23441012b0 2024-03-11T19:07:34.192 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:34.192 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-11T19:07:34.192 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2024-03-11T19:07:34.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:34.190+0000 7f234ac15640 1 -- 172.21.15.29:0/2303606087 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23340a1ab0 msgr2=0x7f23340a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:34.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:34.190+0000 7f234ac15640 1 --2- 172.21.15.29:0/2303606087 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23340a1ab0 0x7f23340a3f70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2344101110 tx=0x7f2338005f90 comp rx=0 tx=0).stop 2024-03-11T19:07:34.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:34.190+0000 7f234ac15640 1 -- 172.21.15.29:0/2303606087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23441012b0 msgr2=0x7f23441975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:34.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:34.191+0000 7f234ac15640 1 --2- 172.21.15.29:0/2303606087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23441012b0 0x7f23441975e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f233000c920 tx=0x7f233000cde0 comp rx=0 tx=0).stop 2024-03-11T19:07:34.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:34.191+0000 7f234ac15640 1 -- 172.21.15.29:0/2303606087 shutdown_connections 2024-03-11T19:07:34.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:34.191+0000 7f234ac15640 1 --2- 172.21.15.29:0/2303606087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2344101c70 0x7f234419e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:34.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:34.191+0000 7f234ac15640 1 --2- 172.21.15.29:0/2303606087 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23340a1ab0 0x7f23340a3f70 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:34.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:34.191+0000 7f234ac15640 1 --2- 172.21.15.29:0/2303606087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23441012b0 0x7f23441975e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:34.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:34.191+0000 7f234ac15640 1 --2- 172.21.15.29:0/2303606087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23441000c0 0x7f23441970a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:34.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:34.191+0000 7f234ac15640 1 -- 172.21.15.29:0/2303606087 >> 172.21.15.29:0/2303606087 conn(0x7f23440fb630 msgr2=0x7f23440fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:34.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:34.191+0000 7f234ac15640 1 -- 172.21.15.29:0/2303606087 shutdown_connections 2024-03-11T19:07:34.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:34.191+0000 7f234ac15640 1 -- 172.21.15.29:0/2303606087 wait complete. 2024-03-11T19:07:34.345 INFO:journalctl@ceph.mds.h.smithi170.stdout:Mar 11 19:07:33 smithi170 podman[66054]: 2024-03-11 19:07:33.971721423 +0000 UTC m=+0.413380501 container init 9a47d71e1dc3db0db8cdbd2d4dd82c350188c83d4ed3a170d45b1a06f24e43d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-h, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-94a5397) 2024-03-11T19:07:34.346 INFO:journalctl@ceph.mds.h.smithi170.stdout:Mar 11 19:07:33 smithi170 podman[66054]: 2024-03-11 19:07:33.974412367 +0000 UTC m=+0.416071435 container start 9a47d71e1dc3db0db8cdbd2d4dd82c350188c83d4ed3a170d45b1a06f24e43d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:07:34.346 INFO:journalctl@ceph.mds.h.smithi170.stdout:Mar 11 19:07:33 smithi170 ceph-mds[66073]: -- 172.21.15.170:0/1879484531 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560c49c421a0 con 0x560c49c19000 2024-03-11T19:07:34.346 INFO:journalctl@ceph.mds.h.smithi170.stdout:Mar 11 19:07:34 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-h[66069]: starting mds.h at 2024-03-11T19:07:34.346 INFO:journalctl@ceph.mds.h.smithi170.stdout:Mar 11 19:07:34 smithi170 bash[66054]: 9a47d71e1dc3db0db8cdbd2d4dd82c350188c83d4ed3a170d45b1a06f24e43d7 2024-03-11T19:07:34.346 INFO:journalctl@ceph.mds.h.smithi170.stdout:Mar 11 19:07:34 smithi170 systemd[1]: Started Ceph mds.h for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:07:35.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:35 smithi029 ceph-mon[32439]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:35.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:35.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:35.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:35.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:35.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:35.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:35.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:35 smithi029 ceph-mon[32439]: Deploying daemon mds.i on smithi190 2024-03-11T19:07:35.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:35 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2303606087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:35.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:35 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] up:boot 2024-03-11T19:07:35.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:35 smithi029 ceph-mon[32439]: fsmap 8 up:standby 2024-03-11T19:07:35.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:35 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T19:07:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:35 smithi190 ceph-mon[36292]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:35 smithi190 ceph-mon[36292]: Deploying daemon mds.i on smithi190 2024-03-11T19:07:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:35 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2303606087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:35 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] up:boot 2024-03-11T19:07:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:35 smithi190 ceph-mon[36292]: fsmap 8 up:standby 2024-03-11T19:07:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:35 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T19:07:35.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:35 smithi170 ceph-mon[35690]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:35.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:35 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:35.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:35 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:35.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:35 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:35.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:35 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:35.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:35 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:35.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:35 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:35.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:35 smithi170 ceph-mon[35690]: Deploying daemon mds.i on smithi190 2024-03-11T19:07:35.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:35 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2303606087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:35.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:35 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] up:boot 2024-03-11T19:07:35.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:35 smithi170 ceph-mon[35690]: fsmap 8 up:standby 2024-03-11T19:07:35.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:35 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T19:07:36.205 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:07:36.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.317+0000 7fc8d1365640 1 -- 172.21.15.29:0/2567803100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8cc101c50 msgr2=0x7fc8cc1084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.317+0000 7fc8d1365640 1 --2- 172.21.15.29:0/2567803100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8cc101c50 0x7fc8cc1084e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009ca0 tx=0x7fc8c402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:07:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.317+0000 7fc8d1365640 1 -- 172.21.15.29:0/2567803100 shutdown_connections 2024-03-11T19:07:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.317+0000 7fc8d1365640 1 --2- 172.21.15.29:0/2567803100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8cc101c50 0x7fc8cc1084e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.317+0000 7fc8d1365640 1 --2- 172.21.15.29:0/2567803100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8cc101290 0x7fc8cc101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.317+0000 7fc8d1365640 1 --2- 172.21.15.29:0/2567803100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8cc1000a0 0x7fc8cc1004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.317+0000 7fc8d1365640 1 -- 172.21.15.29:0/2567803100 >> 172.21.15.29:0/2567803100 conn(0x7fc8cc0fb610 msgr2=0x7fc8cc0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.317+0000 7fc8d1365640 1 -- 172.21.15.29:0/2567803100 shutdown_connections 2024-03-11T19:07:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.317+0000 7fc8d1365640 1 -- 172.21.15.29:0/2567803100 wait complete. 2024-03-11T19:07:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.317+0000 7fc8d1365640 1 Processor -- start 2024-03-11T19:07:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.318+0000 7fc8d1365640 1 -- start start 2024-03-11T19:07:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.318+0000 7fc8d1365640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8cc1000a0 0x7fc8cc068eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.318+0000 7fc8d1365640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8cc101290 0x7fc8cc19bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.318+0000 7fc8d1365640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8cc101c50 0x7fc8cc19e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.318+0000 7fc8d1365640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc19eba0 con 0x7fc8cc101290 2024-03-11T19:07:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.318+0000 7fc8d1365640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc19ed10 con 0x7fc8cc101c50 2024-03-11T19:07:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.318+0000 7fc8d1365640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc19eed0 con 0x7fc8cc1000a0 2024-03-11T19:07:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.318+0000 7fc8ca7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8cc101290 0x7fc8cc19bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.318+0000 7fc8ca7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8cc101290 0x7fc8cc19bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44056/0 (socket says 172.21.15.29:44056) 2024-03-11T19:07:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.318+0000 7fc8ca7fc640 1 -- 172.21.15.29:0/424545668 learned_addr learned my addr 172.21.15.29:0/424545668 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.318+0000 7fc8cb7fe640 1 --2- 172.21.15.29:0/424545668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8cc101c50 0x7fc8cc19e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.318+0000 7fc8caffd640 1 --2- 172.21.15.29:0/424545668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8cc1000a0 0x7fc8cc068eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.318+0000 7fc8ca7fc640 1 -- 172.21.15.29:0/424545668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8cc1000a0 msgr2=0x7fc8cc068eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.318+0000 7fc8ca7fc640 1 --2- 172.21.15.29:0/424545668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8cc1000a0 0x7fc8cc068eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.318+0000 7fc8ca7fc640 1 -- 172.21.15.29:0/424545668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8cc101c50 msgr2=0x7fc8cc19e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.318+0000 7fc8ca7fc640 1 --2- 172.21.15.29:0/424545668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8cc101c50 0x7fc8cc19e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.318+0000 7fc8ca7fc640 1 -- 172.21.15.29:0/424545668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c4009900 con 0x7fc8cc101290 2024-03-11T19:07:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.318+0000 7fc8caffd640 1 --2- 172.21.15.29:0/424545668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8cc1000a0 0x7fc8cc068eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:07:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.318+0000 7fc8cb7fe640 1 --2- 172.21.15.29:0/424545668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8cc101c50 0x7fc8cc19e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:07:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.319+0000 7fc8ca7fc640 1 --2- 172.21.15.29:0/424545668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8cc101290 0x7fc8cc19bda0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc00d770 tx=0x7fc8bc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.319+0000 7fc8b3fff640 1 -- 172.21.15.29:0/424545668 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc009c30 con 0x7fc8cc101290 2024-03-11T19:07:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.319+0000 7fc8d1365640 1 -- 172.21.15.29:0/424545668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8cc19f1b0 con 0x7fc8cc101290 2024-03-11T19:07:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.319+0000 7fc8b3fff640 1 -- 172.21.15.29:0/424545668 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc8bc00de80 con 0x7fc8cc101290 2024-03-11T19:07:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.319+0000 7fc8b3fff640 1 -- 172.21.15.29:0/424545668 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc004e00 con 0x7fc8cc101290 2024-03-11T19:07:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.319+0000 7fc8d1365640 1 -- 172.21.15.29:0/424545668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8cc19f670 con 0x7fc8cc101290 2024-03-11T19:07:36.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.320+0000 7fc8d1365640 1 -- 172.21.15.29:0/424545668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8cc1073c0 con 0x7fc8cc101290 2024-03-11T19:07:36.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.321+0000 7fc8b3fff640 1 -- 172.21.15.29:0/424545668 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc8bc004f60 con 0x7fc8cc101290 2024-03-11T19:07:36.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.321+0000 7fc8b3fff640 1 --2- 172.21.15.29:0/424545668 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc8b40a1a30 0x7fc8b40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:36.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.322+0000 7fc8caffd640 1 --2- 172.21.15.29:0/424545668 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc8b40a1a30 0x7fc8b40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:36.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.322+0000 7fc8b3fff640 1 -- 172.21.15.29:0/424545668 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc8bc0ccaa0 con 0x7fc8cc101290 2024-03-11T19:07:36.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.322+0000 7fc8caffd640 1 --2- 172.21.15.29:0/424545668 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc8b40a1a30 0x7fc8b40a3ef0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc1010f0 tx=0x7fc8c0005db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:36.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.326+0000 7fc8b3fff640 1 -- 172.21.15.29:0/424545668 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc8bc096150 con 0x7fc8cc101290 2024-03-11T19:07:36.471 INFO:journalctl@ceph.mds.i.smithi190.stdout:Mar 11 19:07:36 smithi190 systemd[1]: Starting Ceph mds.i for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:07:36.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.548+0000 7fc8d1365640 1 -- 172.21.15.29:0/424545668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8cc02ce40 con 0x7fc8cc101290 2024-03-11T19:07:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.548+0000 7fc8b3fff640 1 -- 172.21.15.29:0/424545668 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6868 (secure 0 0 0) 0x7fc8bc0cae60 con 0x7fc8cc101290 2024-03-11T19:07:36.551 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:36.551 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-11T19:07:36.552 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 9 2024-03-11T19:07:36.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.550+0000 7fc8d1365640 1 -- 172.21.15.29:0/424545668 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc8b40a1a30 msgr2=0x7fc8b40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:36.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.550+0000 7fc8d1365640 1 --2- 172.21.15.29:0/424545668 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc8b40a1a30 0x7fc8b40a3ef0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc1010f0 tx=0x7fc8c0005db0 comp rx=0 tx=0).stop 2024-03-11T19:07:36.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.550+0000 7fc8d1365640 1 -- 172.21.15.29:0/424545668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8cc101290 msgr2=0x7fc8cc19bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:36.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.550+0000 7fc8d1365640 1 --2- 172.21.15.29:0/424545668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8cc101290 0x7fc8cc19bda0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc00d770 tx=0x7fc8bc00dc30 comp rx=0 tx=0).stop 2024-03-11T19:07:36.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.550+0000 7fc8d1365640 1 -- 172.21.15.29:0/424545668 shutdown_connections 2024-03-11T19:07:36.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.550+0000 7fc8d1365640 1 --2- 172.21.15.29:0/424545668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8cc101c50 0x7fc8cc19e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:36.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.550+0000 7fc8d1365640 1 --2- 172.21.15.29:0/424545668 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc8b40a1a30 0x7fc8b40a3ef0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:36.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.550+0000 7fc8d1365640 1 --2- 172.21.15.29:0/424545668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8cc101290 0x7fc8cc19bda0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:36.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.550+0000 7fc8d1365640 1 --2- 172.21.15.29:0/424545668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8cc1000a0 0x7fc8cc068eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:36.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.550+0000 7fc8d1365640 1 -- 172.21.15.29:0/424545668 >> 172.21.15.29:0/424545668 conn(0x7fc8cc0fb610 msgr2=0x7fc8cc104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:36.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.550+0000 7fc8d1365640 1 -- 172.21.15.29:0/424545668 shutdown_connections 2024-03-11T19:07:36.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:36.550+0000 7fc8d1365640 1 -- 172.21.15.29:0/424545668 wait complete. 2024-03-11T19:07:36.762 INFO:journalctl@ceph.mds.i.smithi190.stdout:Mar 11 19:07:36 smithi190 podman[66990]: 2024-03-11 19:07:36.446361361 +0000 UTC m=+0.019341673 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:07:36.762 INFO:journalctl@ceph.mds.i.smithi190.stdout:Mar 11 19:07:36 smithi190 podman[66990]: 2024-03-11 19:07:36.558824532 +0000 UTC m=+0.131804834 container create b94348ec9f1e6eb0654d18f3864c99b7842b250c5af33978f3fb99f18ef4ba87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-i, ceph=True, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-11T19:07:37.128 INFO:journalctl@ceph.mds.i.smithi190.stdout:Mar 11 19:07:36 smithi190 podman[66990]: 2024-03-11 19:07:36.811535942 +0000 UTC m=+0.384516247 container init b94348ec9f1e6eb0654d18f3864c99b7842b250c5af33978f3fb99f18ef4ba87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-i, io.buildah.version=1.33.5, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:07:37.129 INFO:journalctl@ceph.mds.i.smithi190.stdout:Mar 11 19:07:36 smithi190 podman[66990]: 2024-03-11 19:07:36.814288931 +0000 UTC m=+0.387269232 container start b94348ec9f1e6eb0654d18f3864c99b7842b250c5af33978f3fb99f18ef4ba87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-i, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True) 2024-03-11T19:07:37.129 INFO:journalctl@ceph.mds.i.smithi190.stdout:Mar 11 19:07:36 smithi190 ceph-mds[67009]: -- 172.21.15.190:0/1797116168 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56440f0e61a0 con 0x56440f0bd000 2024-03-11T19:07:37.129 INFO:journalctl@ceph.mds.i.smithi190.stdout:Mar 11 19:07:36 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-i[67005]: starting mds.i at 2024-03-11T19:07:37.129 INFO:journalctl@ceph.mds.i.smithi190.stdout:Mar 11 19:07:36 smithi190 bash[66990]: b94348ec9f1e6eb0654d18f3864c99b7842b250c5af33978f3fb99f18ef4ba87 2024-03-11T19:07:37.129 INFO:journalctl@ceph.mds.i.smithi190.stdout:Mar 11 19:07:36 smithi190 systemd[1]: Started Ceph mds.i for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:07:37.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:37 smithi029 ceph-mon[32439]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:37.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:37 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/424545668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:37.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:37 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:37.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:37 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:37.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:37 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:37.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:37 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:37.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:37 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:37.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:37 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:37.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:37 smithi190 ceph-mon[36292]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:37.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:37 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/424545668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:37.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:37 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:37.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:37 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:37.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:37 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:37.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:37 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:37.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:37 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:37.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:37 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:37.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:37 smithi170 ceph-mon[35690]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:37.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:37 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/424545668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:37.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:37 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:37.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:37 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:37.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:37 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:37.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:37 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:37.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:37 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:37.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:37 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:38.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:38 smithi029 ceph-mon[32439]: Deploying daemon mds.g on smithi029 2024-03-11T19:07:38.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:38 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] up:boot 2024-03-11T19:07:38.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:38 smithi029 ceph-mon[32439]: fsmap 9 up:standby 2024-03-11T19:07:38.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:38 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T19:07:38.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:38 smithi190 ceph-mon[36292]: Deploying daemon mds.g on smithi029 2024-03-11T19:07:38.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:38 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] up:boot 2024-03-11T19:07:38.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:38 smithi190 ceph-mon[36292]: fsmap 9 up:standby 2024-03-11T19:07:38.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:38 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T19:07:38.564 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:07:38.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:38 smithi170 ceph-mon[35690]: Deploying daemon mds.g on smithi029 2024-03-11T19:07:38.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:38 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] up:boot 2024-03-11T19:07:38.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:38 smithi170 ceph-mon[35690]: fsmap 9 up:standby 2024-03-11T19:07:38.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:38 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T19:07:38.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.680+0000 7fdc73fc7640 1 -- 172.21.15.29:0/778233567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6c1045d0 msgr2=0x7fdc6c1069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:38.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.680+0000 7fdc73fc7640 1 --2- 172.21.15.29:0/778233567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6c1045d0 0x7fdc6c1069c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c009900 tx=0x7fdc5c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:38.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.680+0000 7fdc73fc7640 1 -- 172.21.15.29:0/778233567 shutdown_connections 2024-03-11T19:07:38.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.680+0000 7fdc73fc7640 1 --2- 172.21.15.29:0/778233567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc6c106f00 0x7fdc6c1092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:38.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.680+0000 7fdc73fc7640 1 --2- 172.21.15.29:0/778233567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6c1045d0 0x7fdc6c1069c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:38.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.680+0000 7fdc73fc7640 1 --2- 172.21.15.29:0/778233567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc6c101ca0 0x7fdc6c104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:38.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.680+0000 7fdc73fc7640 1 -- 172.21.15.29:0/778233567 >> 172.21.15.29:0/778233567 conn(0x7fdc6c0fb630 msgr2=0x7fdc6c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:38.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.680+0000 7fdc73fc7640 1 -- 172.21.15.29:0/778233567 shutdown_connections 2024-03-11T19:07:38.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.680+0000 7fdc73fc7640 1 -- 172.21.15.29:0/778233567 wait complete. 2024-03-11T19:07:38.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc73fc7640 1 Processor -- start 2024-03-11T19:07:38.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc73fc7640 1 -- start start 2024-03-11T19:07:38.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc73fc7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6c101ca0 0x7fdc6c1971d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:38.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc73fc7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc6c1045d0 0x7fdc6c197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:38.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc73fc7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc6c106f00 0x7fdc6c19e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:38.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc73fc7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c19ecc0 con 0x7fdc6c101ca0 2024-03-11T19:07:38.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc73fc7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c19ee30 con 0x7fdc6c1045d0 2024-03-11T19:07:38.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc73fc7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c19eff0 con 0x7fdc6c106f00 2024-03-11T19:07:38.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc71d63640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6c101ca0 0x7fdc6c1971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:38.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc71d63640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6c101ca0 0x7fdc6c1971d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44076/0 (socket says 172.21.15.29:44076) 2024-03-11T19:07:38.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc71d63640 1 -- 172.21.15.29:0/3192178238 learned_addr learned my addr 172.21.15.29:0/3192178238 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:38.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc71562640 1 --2- 172.21.15.29:0/3192178238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc6c1045d0 0x7fdc6c197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:38.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc72564640 1 --2- 172.21.15.29:0/3192178238 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc6c106f00 0x7fdc6c19e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:38.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc71d63640 1 -- 172.21.15.29:0/3192178238 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc6c106f00 msgr2=0x7fdc6c19e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:38.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc71d63640 1 --2- 172.21.15.29:0/3192178238 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc6c106f00 0x7fdc6c19e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:38.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc71d63640 1 -- 172.21.15.29:0/3192178238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc6c1045d0 msgr2=0x7fdc6c197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:38.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc71d63640 1 --2- 172.21.15.29:0/3192178238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc6c1045d0 0x7fdc6c197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:38.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc71d63640 1 -- 172.21.15.29:0/3192178238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5c009560 con 0x7fdc6c101ca0 2024-03-11T19:07:38.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc71562640 1 --2- 172.21.15.29:0/3192178238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc6c1045d0 0x7fdc6c197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:07:38.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc72564640 1 --2- 172.21.15.29:0/3192178238 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc6c106f00 0x7fdc6c19e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:07:38.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc71d63640 1 --2- 172.21.15.29:0/3192178238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6c101ca0 0x7fdc6c1971d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fdc6800d950 tx=0x7fdc6800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:38.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc62ffd640 1 -- 172.21.15.29:0/3192178238 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc68009a30 con 0x7fdc6c101ca0 2024-03-11T19:07:38.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc62ffd640 1 -- 172.21.15.29:0/3192178238 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdc680044d0 con 0x7fdc6c101ca0 2024-03-11T19:07:38.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc73fc7640 1 -- 172.21.15.29:0/3192178238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc6c19f2d0 con 0x7fdc6c101ca0 2024-03-11T19:07:38.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.681+0000 7fdc62ffd640 1 -- 172.21.15.29:0/3192178238 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc680047a0 con 0x7fdc6c101ca0 2024-03-11T19:07:38.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.682+0000 7fdc73fc7640 1 -- 172.21.15.29:0/3192178238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc6c19f830 con 0x7fdc6c101ca0 2024-03-11T19:07:38.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.682+0000 7fdc73fc7640 1 -- 172.21.15.29:0/3192178238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc6c0fd120 con 0x7fdc6c101ca0 2024-03-11T19:07:38.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.684+0000 7fdc62ffd640 1 -- 172.21.15.29:0/3192178238 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fdc68009b90 con 0x7fdc6c101ca0 2024-03-11T19:07:38.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.684+0000 7fdc62ffd640 1 --2- 172.21.15.29:0/3192178238 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdc580a1ab0 0x7fdc580a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:38.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.684+0000 7fdc71562640 1 --2- 172.21.15.29:0/3192178238 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdc580a1ab0 0x7fdc580a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:38.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.684+0000 7fdc62ffd640 1 -- 172.21.15.29:0/3192178238 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdc680cca20 con 0x7fdc6c101ca0 2024-03-11T19:07:38.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.685+0000 7fdc71562640 1 --2- 172.21.15.29:0/3192178238 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdc580a1ab0 0x7fdc580a3f70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c036040 tx=0x7fdc5c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:38.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.689+0000 7fdc62ffd640 1 -- 172.21.15.29:0/3192178238 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdc680960d0 con 0x7fdc6c101ca0 2024-03-11T19:07:38.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.916+0000 7fdc73fc7640 1 -- 172.21.15.29:0/3192178238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc6c10ca40 con 0x7fdc6c101ca0 2024-03-11T19:07:38.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.917+0000 7fdc62ffd640 1 -- 172.21.15.29:0/3192178238 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7664 (secure 0 0 0) 0x7fdc68099790 con 0x7fdc6c101ca0 2024-03-11T19:07:38.921 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:38.922 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-11T19:07:38.922 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 10 2024-03-11T19:07:38.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.920+0000 7fdc73fc7640 1 -- 172.21.15.29:0/3192178238 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdc580a1ab0 msgr2=0x7fdc580a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:38.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.920+0000 7fdc73fc7640 1 --2- 172.21.15.29:0/3192178238 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdc580a1ab0 0x7fdc580a3f70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c036040 tx=0x7fdc5c000bc0 comp rx=0 tx=0).stop 2024-03-11T19:07:38.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.920+0000 7fdc73fc7640 1 -- 172.21.15.29:0/3192178238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6c101ca0 msgr2=0x7fdc6c1971d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:38.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.920+0000 7fdc73fc7640 1 --2- 172.21.15.29:0/3192178238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6c101ca0 0x7fdc6c1971d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fdc6800d950 tx=0x7fdc6800de10 comp rx=0 tx=0).stop 2024-03-11T19:07:38.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.920+0000 7fdc73fc7640 1 -- 172.21.15.29:0/3192178238 shutdown_connections 2024-03-11T19:07:38.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.920+0000 7fdc73fc7640 1 --2- 172.21.15.29:0/3192178238 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc6c106f00 0x7fdc6c19e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:38.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.920+0000 7fdc73fc7640 1 --2- 172.21.15.29:0/3192178238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc6c1045d0 0x7fdc6c197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:38.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.920+0000 7fdc73fc7640 1 --2- 172.21.15.29:0/3192178238 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdc580a1ab0 0x7fdc580a3f70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:38.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.920+0000 7fdc73fc7640 1 --2- 172.21.15.29:0/3192178238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6c101ca0 0x7fdc6c1971d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:38.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.920+0000 7fdc73fc7640 1 -- 172.21.15.29:0/3192178238 >> 172.21.15.29:0/3192178238 conn(0x7fdc6c0fb630 msgr2=0x7fdc6c103b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:38.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.920+0000 7fdc73fc7640 1 -- 172.21.15.29:0/3192178238 shutdown_connections 2024-03-11T19:07:38.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:38.920+0000 7fdc73fc7640 1 -- 172.21.15.29:0/3192178238 wait complete. 2024-03-11T19:07:39.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:39 smithi029 ceph-mon[32439]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:39.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:39 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3192178238' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:39.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:39 smithi190 ceph-mon[36292]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:39.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:39 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3192178238' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:39.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:39 smithi170 ceph-mon[35690]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:39.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:39 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3192178238' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:39.905 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 11 19:07:39 smithi029 systemd[1]: Started Ceph mds.g for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:07:40.934 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:07:41.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.046+0000 7f28dd5d0640 1 -- 172.21.15.29:0/2545163064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28d8101cd0 msgr2=0x7f28d8108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:41.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.046+0000 7f28dd5d0640 1 --2- 172.21.15.29:0/2545163064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28d8101cd0 0x7f28d8108560 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f28c8009ca0 tx=0x7f28c802c0f0 comp rx=0 tx=0).stop 2024-03-11T19:07:41.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.046+0000 7f28dd5d0640 1 -- 172.21.15.29:0/2545163064 shutdown_connections 2024-03-11T19:07:41.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.046+0000 7f28dd5d0640 1 --2- 172.21.15.29:0/2545163064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28d8101cd0 0x7f28d8108560 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:41.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.046+0000 7f28dd5d0640 1 --2- 172.21.15.29:0/2545163064 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28d8101310 0x7f28d8101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:41.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.046+0000 7f28dd5d0640 1 --2- 172.21.15.29:0/2545163064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28d8100120 0x7f28d8100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:41.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.046+0000 7f28dd5d0640 1 -- 172.21.15.29:0/2545163064 >> 172.21.15.29:0/2545163064 conn(0x7f28d80fb6b0 msgr2=0x7f28d80fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:41.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.047+0000 7f28dd5d0640 1 -- 172.21.15.29:0/2545163064 shutdown_connections 2024-03-11T19:07:41.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.047+0000 7f28dd5d0640 1 -- 172.21.15.29:0/2545163064 wait complete. 2024-03-11T19:07:41.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.047+0000 7f28dd5d0640 1 Processor -- start 2024-03-11T19:07:41.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28dd5d0640 1 -- start start 2024-03-11T19:07:41.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28dd5d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28d8100120 0x7f28d806a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:41.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28dd5d0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28d8101310 0x7f28d819bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:41.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28dd5d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28d8101cd0 0x7f28d819e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:41.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28dd5d0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d819eb90 con 0x7f28d8101cd0 2024-03-11T19:07:41.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28dd5d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d819ed00 con 0x7f28d8100120 2024-03-11T19:07:41.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28dd5d0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d819eec0 con 0x7f28d8101310 2024-03-11T19:07:41.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28d77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28d8101cd0 0x7f28d819e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:41.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28d77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28d8101cd0 0x7f28d819e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44108/0 (socket says 172.21.15.29:44108) 2024-03-11T19:07:41.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28d77fe640 1 -- 172.21.15.29:0/3604633212 learned_addr learned my addr 172.21.15.29:0/3604633212 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:41.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28d6ffd640 1 --2- 172.21.15.29:0/3604633212 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28d8100120 0x7f28d806a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:41.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28d67fc640 1 --2- 172.21.15.29:0/3604633212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28d8101310 0x7f28d819bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:41.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28d77fe640 1 -- 172.21.15.29:0/3604633212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28d8101310 msgr2=0x7f28d819bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:41.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28d77fe640 1 --2- 172.21.15.29:0/3604633212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28d8101310 0x7f28d819bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:41.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28d77fe640 1 -- 172.21.15.29:0/3604633212 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28d8100120 msgr2=0x7f28d806a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:41.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28d77fe640 1 --2- 172.21.15.29:0/3604633212 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28d8100120 0x7f28d806a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:41.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28d77fe640 1 -- 172.21.15.29:0/3604633212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c8009900 con 0x7f28d8101cd0 2024-03-11T19:07:41.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28d67fc640 1 --2- 172.21.15.29:0/3604633212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28d8101310 0x7f28d819bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:07:41.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28d77fe640 1 --2- 172.21.15.29:0/3604633212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28d8101cd0 0x7f28d819e3b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f28c8037040 tx=0x7f28c8032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:41.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.048+0000 7f28cffff640 1 -- 172.21.15.29:0/3604633212 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c8034e70 con 0x7f28d8101cd0 2024-03-11T19:07:41.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.049+0000 7f28dd5d0640 1 -- 172.21.15.29:0/3604633212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d819f0e0 con 0x7f28d8101cd0 2024-03-11T19:07:41.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.049+0000 7f28cffff640 1 -- 172.21.15.29:0/3604633212 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f28c8003e20 con 0x7f28d8101cd0 2024-03-11T19:07:41.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.049+0000 7f28cffff640 1 -- 172.21.15.29:0/3604633212 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c8004800 con 0x7f28d8101cd0 2024-03-11T19:07:41.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.049+0000 7f28dd5d0640 1 -- 172.21.15.29:0/3604633212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d819f590 con 0x7f28d8101cd0 2024-03-11T19:07:41.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.050+0000 7f28dd5d0640 1 -- 172.21.15.29:0/3604633212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d8107370 con 0x7f28d8101cd0 2024-03-11T19:07:41.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.051+0000 7f28cffff640 1 -- 172.21.15.29:0/3604633212 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f28c803b040 con 0x7f28d8101cd0 2024-03-11T19:07:41.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.051+0000 7f28cffff640 1 --2- 172.21.15.29:0/3604633212 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f28c40a1a30 0x7f28c40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:41.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.051+0000 7f28d6ffd640 1 --2- 172.21.15.29:0/3604633212 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f28c40a1a30 0x7f28c40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:41.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.051+0000 7f28cffff640 1 -- 172.21.15.29:0/3604633212 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f28c80ef770 con 0x7f28d8101cd0 2024-03-11T19:07:41.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.052+0000 7f28d6ffd640 1 --2- 172.21.15.29:0/3604633212 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f28c40a1a30 0x7f28c40a3ef0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f28d00058c0 tx=0x7f28d0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:41.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.056+0000 7f28cffff640 1 -- 172.21.15.29:0/3604633212 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f28c80b8e20 con 0x7f28d8101cd0 2024-03-11T19:07:41.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:40 smithi170 ceph-mon[35690]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:41.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:40 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:41.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:40 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:41.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:40 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:41.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:40 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:41.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:40 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:41.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:40 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:41.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:40 smithi170 ceph-mon[35690]: Deploying daemon mds.a on smithi029 2024-03-11T19:07:41.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:40 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.29:6838/72765211,v1:172.21.15.29:6839/72765211] up:boot 2024-03-11T19:07:41.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:40 smithi170 ceph-mon[35690]: fsmap 10 up:standby 2024-03-11T19:07:41.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:40 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-11T19:07:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:40 smithi029 ceph-mon[32439]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:40 smithi029 ceph-mon[32439]: Deploying daemon mds.a on smithi029 2024-03-11T19:07:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:40 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.29:6838/72765211,v1:172.21.15.29:6839/72765211] up:boot 2024-03-11T19:07:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:40 smithi029 ceph-mon[32439]: fsmap 10 up:standby 2024-03-11T19:07:41.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-11T19:07:41.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:40 smithi190 ceph-mon[36292]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:41.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:40 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:41.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:40 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:41.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:40 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:41.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:40 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:41.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:40 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:41.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:40 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:41.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:40 smithi190 ceph-mon[36292]: Deploying daemon mds.a on smithi029 2024-03-11T19:07:41.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:40 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.29:6838/72765211,v1:172.21.15.29:6839/72765211] up:boot 2024-03-11T19:07:41.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:40 smithi190 ceph-mon[36292]: fsmap 10 up:standby 2024-03-11T19:07:41.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:40 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-11T19:07:41.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.276+0000 7f28dd5d0640 1 -- 172.21.15.29:0/3604633212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28d810bb70 con 0x7f28d8101cd0 2024-03-11T19:07:41.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.277+0000 7f28cffff640 1 -- 172.21.15.29:0/3604633212 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8450 (secure 0 0 0) 0x7f28c80bc4e0 con 0x7f28d8101cd0 2024-03-11T19:07:41.280 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:41.280 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-11T19:07:41.280 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 11 2024-03-11T19:07:41.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.278+0000 7f28dd5d0640 1 -- 172.21.15.29:0/3604633212 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f28c40a1a30 msgr2=0x7f28c40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:41.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.278+0000 7f28dd5d0640 1 --2- 172.21.15.29:0/3604633212 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f28c40a1a30 0x7f28c40a3ef0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f28d00058c0 tx=0x7f28d0005850 comp rx=0 tx=0).stop 2024-03-11T19:07:41.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.278+0000 7f28dd5d0640 1 -- 172.21.15.29:0/3604633212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28d8101cd0 msgr2=0x7f28d819e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:41.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.278+0000 7f28dd5d0640 1 --2- 172.21.15.29:0/3604633212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28d8101cd0 0x7f28d819e3b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f28c8037040 tx=0x7f28c8032980 comp rx=0 tx=0).stop 2024-03-11T19:07:41.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.278+0000 7f28dd5d0640 1 -- 172.21.15.29:0/3604633212 shutdown_connections 2024-03-11T19:07:41.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.278+0000 7f28dd5d0640 1 --2- 172.21.15.29:0/3604633212 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f28c40a1a30 0x7f28c40a3ef0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:41.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.278+0000 7f28dd5d0640 1 --2- 172.21.15.29:0/3604633212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28d8101cd0 0x7f28d819e3b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:41.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.278+0000 7f28dd5d0640 1 --2- 172.21.15.29:0/3604633212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28d8101310 0x7f28d819bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:41.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.278+0000 7f28dd5d0640 1 --2- 172.21.15.29:0/3604633212 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28d8100120 0x7f28d806a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:41.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.278+0000 7f28dd5d0640 1 -- 172.21.15.29:0/3604633212 >> 172.21.15.29:0/3604633212 conn(0x7f28d80fb6b0 msgr2=0x7f28d8104540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:41.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.279+0000 7f28dd5d0640 1 -- 172.21.15.29:0/3604633212 shutdown_connections 2024-03-11T19:07:41.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:41.279+0000 7f28dd5d0640 1 -- 172.21.15.29:0/3604633212 wait complete. 2024-03-11T19:07:42.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:41 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3604633212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:42.126 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:41 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3604633212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:42.126 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 11 19:07:41 smithi029 systemd[1]: Starting Ceph mds.a for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:07:42.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:41 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3604633212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:42.406 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 11 19:07:42 smithi029 podman[74026]: 2024-03-11 19:07:42.026198563 +0000 UTC m=+0.021926569 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:07:42.406 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 11 19:07:42 smithi029 podman[74026]: 2024-03-11 19:07:42.156286048 +0000 UTC m=+0.152014037 container create bc90ce3586d6099f1a60853a1c77d8b636da4539ed540fdff18d31e435c94e03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T19:07:42.726 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 11 19:07:42 smithi029 podman[74026]: 2024-03-11 19:07:42.471495373 +0000 UTC m=+0.467223385 container init bc90ce3586d6099f1a60853a1c77d8b636da4539ed540fdff18d31e435c94e03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-a, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-11T19:07:42.726 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 11 19:07:42 smithi029 podman[74026]: 2024-03-11 19:07:42.474287172 +0000 UTC m=+0.470015170 container start bc90ce3586d6099f1a60853a1c77d8b636da4539ed540fdff18d31e435c94e03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-a, GIT_CLEAN=True, ceph=True, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-11T19:07:42.726 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 11 19:07:42 smithi029 ceph-mds[74047]: -- 172.21.15.29:0/3288499282 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a69f4ee1a0 con 0x55a69f4c4800 2024-03-11T19:07:42.726 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 11 19:07:42 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-a[74043]: starting mds.a at 2024-03-11T19:07:42.726 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 11 19:07:42 smithi029 bash[74026]: bc90ce3586d6099f1a60853a1c77d8b636da4539ed540fdff18d31e435c94e03 2024-03-11T19:07:42.726 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 11 19:07:42 smithi029 systemd[1]: Started Ceph mds.a for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:07:43.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:42 smithi170 ceph-mon[35690]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:43.098 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:42 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:43.098 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:42 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:43.098 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:42 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:43.098 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:42 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:43.098 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:42 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:43.098 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:42 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:43.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:42 smithi029 ceph-mon[32439]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:43.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:42 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:43.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:42 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:43.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:42 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:43.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:42 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:43.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:42 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:43.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:42 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:43.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:42 smithi190 ceph-mon[36292]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:43.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:42 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:43.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:42 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:43.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:42 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:43.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:42 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:07:43.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:42 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:07:43.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:42 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:43.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:07:43.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.406+0000 7f1bf4793640 1 -- 172.21.15.29:0/4123558621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bec102d60 msgr2=0x7f1bec1095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:43.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.406+0000 7f1bf4793640 1 --2- 172.21.15.29:0/4123558621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bec102d60 0x7f1bec1095f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1be4009900 tx=0x7f1be402bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:43.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.406+0000 7f1bf4793640 1 -- 172.21.15.29:0/4123558621 shutdown_connections 2024-03-11T19:07:43.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.406+0000 7f1bf4793640 1 --2- 172.21.15.29:0/4123558621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bec102d60 0x7f1bec1095f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:43.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.406+0000 7f1bf4793640 1 --2- 172.21.15.29:0/4123558621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bec1023a0 0x7f1bec102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:43.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.406+0000 7f1bf4793640 1 --2- 172.21.15.29:0/4123558621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bec1011b0 0x7f1bec1015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:43.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.406+0000 7f1bf4793640 1 -- 172.21.15.29:0/4123558621 >> 172.21.15.29:0/4123558621 conn(0x7f1bec0fc760 msgr2=0x7f1bec0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:43.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.406+0000 7f1bf4793640 1 -- 172.21.15.29:0/4123558621 shutdown_connections 2024-03-11T19:07:43.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.406+0000 7f1bf4793640 1 -- 172.21.15.29:0/4123558621 wait complete. 2024-03-11T19:07:43.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.407+0000 7f1bf4793640 1 Processor -- start 2024-03-11T19:07:43.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.407+0000 7f1bf4793640 1 -- start start 2024-03-11T19:07:43.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.407+0000 7f1bf4793640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bec1011b0 0x7f1bec197070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:43.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.407+0000 7f1bf4793640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bec1023a0 0x7f1bec1975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:43.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.407+0000 7f1bf4793640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bec102d60 0x7f1bec19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:43.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.407+0000 7f1bf4793640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bec19eb60 con 0x7f1bec1011b0 2024-03-11T19:07:43.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.407+0000 7f1bf4793640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bec19ecd0 con 0x7f1bec102d60 2024-03-11T19:07:43.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.407+0000 7f1bf4793640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bec19ee90 con 0x7f1bec1023a0 2024-03-11T19:07:43.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.407+0000 7f1bf252f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bec1011b0 0x7f1bec197070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:43.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.407+0000 7f1bf252f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bec1011b0 0x7f1bec197070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44142/0 (socket says 172.21.15.29:44142) 2024-03-11T19:07:43.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.407+0000 7f1bf252f640 1 -- 172.21.15.29:0/2408563227 learned_addr learned my addr 172.21.15.29:0/2408563227 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:43.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.407+0000 7f1bf2d30640 1 --2- 172.21.15.29:0/2408563227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bec102d60 0x7f1bec19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:43.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.408+0000 7f1bf1d2e640 1 --2- 172.21.15.29:0/2408563227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bec1023a0 0x7f1bec1975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:43.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.408+0000 7f1bf252f640 1 -- 172.21.15.29:0/2408563227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bec1023a0 msgr2=0x7f1bec1975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:43.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.408+0000 7f1bf252f640 1 --2- 172.21.15.29:0/2408563227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bec1023a0 0x7f1bec1975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:43.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.408+0000 7f1bf252f640 1 -- 172.21.15.29:0/2408563227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bec102d60 msgr2=0x7f1bec19e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:43.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.408+0000 7f1bf252f640 1 --2- 172.21.15.29:0/2408563227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bec102d60 0x7f1bec19e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:43.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.408+0000 7f1bf252f640 1 -- 172.21.15.29:0/2408563227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be4009560 con 0x7f1bec1011b0 2024-03-11T19:07:43.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.408+0000 7f1bf1d2e640 1 --2- 172.21.15.29:0/2408563227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bec1023a0 0x7f1bec1975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:07:43.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.408+0000 7f1bf252f640 1 --2- 172.21.15.29:0/2408563227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bec1011b0 0x7f1bec197070 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1be800ea80 tx=0x7f1be800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:43.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.408+0000 7f1be37fe640 1 -- 172.21.15.29:0/2408563227 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be800ce40 con 0x7f1bec1011b0 2024-03-11T19:07:43.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.408+0000 7f1bf4793640 1 -- 172.21.15.29:0/2408563227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bec19f170 con 0x7f1bec1011b0 2024-03-11T19:07:43.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.408+0000 7f1be37fe640 1 -- 172.21.15.29:0/2408563227 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1be8011920 con 0x7f1bec1011b0 2024-03-11T19:07:43.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.408+0000 7f1be37fe640 1 -- 172.21.15.29:0/2408563227 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be801aa70 con 0x7f1bec1011b0 2024-03-11T19:07:43.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.408+0000 7f1bf4793640 1 -- 172.21.15.29:0/2408563227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bec19f680 con 0x7f1bec1011b0 2024-03-11T19:07:43.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.409+0000 7f1bf4793640 1 -- 172.21.15.29:0/2408563227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bec1084d0 con 0x7f1bec1011b0 2024-03-11T19:07:43.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.410+0000 7f1be37fe640 1 -- 172.21.15.29:0/2408563227 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1be8011a90 con 0x7f1bec1011b0 2024-03-11T19:07:43.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.411+0000 7f1be37fe640 1 --2- 172.21.15.29:0/2408563227 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1bd80a1a30 0x7f1bd80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:43.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.411+0000 7f1be37fe640 1 -- 172.21.15.29:0/2408563227 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1be80cd4d0 con 0x7f1bec1011b0 2024-03-11T19:07:43.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.411+0000 7f1bf1d2e640 1 --2- 172.21.15.29:0/2408563227 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1bd80a1a30 0x7f1bd80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:43.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.411+0000 7f1bf1d2e640 1 --2- 172.21.15.29:0/2408563227 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1bd80a1a30 0x7f1bd80a3ef0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1bec198580 tx=0x7f1bdc005d90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:43.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.415+0000 7f1be37fe640 1 -- 172.21.15.29:0/2408563227 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1be8096b80 con 0x7f1bec1011b0 2024-03-11T19:07:43.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.633+0000 7f1bf4793640 1 -- 172.21.15.29:0/2408563227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bec10ca70 con 0x7f1bec1011b0 2024-03-11T19:07:43.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.634+0000 7f1be37fe640 1 -- 172.21.15.29:0/2408563227 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9242 (secure 0 0 0) 0x7f1be809a240 con 0x7f1bec1011b0 2024-03-11T19:07:43.637 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:43.637 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-11T19:07:43.637 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 12 2024-03-11T19:07:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.636+0000 7f1bf4793640 1 -- 172.21.15.29:0/2408563227 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1bd80a1a30 msgr2=0x7f1bd80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.636+0000 7f1bf4793640 1 --2- 172.21.15.29:0/2408563227 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1bd80a1a30 0x7f1bd80a3ef0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1bec198580 tx=0x7f1bdc005d90 comp rx=0 tx=0).stop 2024-03-11T19:07:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.636+0000 7f1bf4793640 1 -- 172.21.15.29:0/2408563227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bec1011b0 msgr2=0x7f1bec197070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.636+0000 7f1bf4793640 1 --2- 172.21.15.29:0/2408563227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bec1011b0 0x7f1bec197070 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1be800ea80 tx=0x7f1be800ef40 comp rx=0 tx=0).stop 2024-03-11T19:07:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.636+0000 7f1bf4793640 1 -- 172.21.15.29:0/2408563227 shutdown_connections 2024-03-11T19:07:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.636+0000 7f1bf4793640 1 --2- 172.21.15.29:0/2408563227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bec102d60 0x7f1bec19e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.636+0000 7f1bf4793640 1 --2- 172.21.15.29:0/2408563227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bec1023a0 0x7f1bec1975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.636+0000 7f1bf4793640 1 --2- 172.21.15.29:0/2408563227 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1bd80a1a30 0x7f1bd80a3ef0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.636+0000 7f1bf4793640 1 --2- 172.21.15.29:0/2408563227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bec1011b0 0x7f1bec197070 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.636+0000 7f1bf4793640 1 -- 172.21.15.29:0/2408563227 >> 172.21.15.29:0/2408563227 conn(0x7f1bec0fc760 msgr2=0x7f1bec1056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:43.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.636+0000 7f1bf4793640 1 -- 172.21.15.29:0/2408563227 shutdown_connections 2024-03-11T19:07:43.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:43.636+0000 7f1bf4793640 1 -- 172.21.15.29:0/2408563227 wait complete. 2024-03-11T19:07:43.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:43 smithi029 ceph-mon[32439]: Deploying daemon mds.b on smithi170 2024-03-11T19:07:43.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:43 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.29:6840/3154444865,v1:172.21.15.29:6841/3154444865] up:boot 2024-03-11T19:07:43.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:43 smithi029 ceph-mon[32439]: fsmap 11 up:standby 2024-03-11T19:07:43.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T19:07:43.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:43 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2408563227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:44.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:43 smithi170 ceph-mon[35690]: Deploying daemon mds.b on smithi170 2024-03-11T19:07:44.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:43 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.29:6840/3154444865,v1:172.21.15.29:6841/3154444865] up:boot 2024-03-11T19:07:44.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:43 smithi170 ceph-mon[35690]: fsmap 11 up:standby 2024-03-11T19:07:44.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:43 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T19:07:44.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:43 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2408563227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:44.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:43 smithi190 ceph-mon[36292]: Deploying daemon mds.b on smithi170 2024-03-11T19:07:44.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:43 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.29:6840/3154444865,v1:172.21.15.29:6841/3154444865] up:boot 2024-03-11T19:07:44.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:43 smithi190 ceph-mon[36292]: fsmap 11 up:standby 2024-03-11T19:07:44.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:43 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T19:07:44.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:43 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2408563227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:45.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:44 smithi170 ceph-mon[35690]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:45.096 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:07:45 smithi170 systemd[1]: Starting Ceph mds.b for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:07:45.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:44 smithi029 ceph-mon[32439]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:45.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:44 smithi190 ceph-mon[36292]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:45.555 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:07:45 smithi170 podman[66364]: 2024-03-11 19:07:45.146040816 +0000 UTC m=+0.016889217 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:07:45.556 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:07:45 smithi170 podman[66364]: 2024-03-11 19:07:45.300395468 +0000 UTC m=+0.171243912 container create 32ee158abb0682e2a718c2a286f999af1e4cd98bb3b7e02dc349428721a37fdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-11T19:07:45.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:07:45.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.766+0000 7fb7e0c95640 1 -- 172.21.15.29:0/3088960909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc102d40 msgr2=0x7fb7dc1095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.766+0000 7fb7e0c95640 1 --2- 172.21.15.29:0/3088960909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc102d40 0x7fb7dc1095d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009ca0 tx=0x7fb7d402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:07:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.766+0000 7fb7e0c95640 1 -- 172.21.15.29:0/3088960909 shutdown_connections 2024-03-11T19:07:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.766+0000 7fb7e0c95640 1 --2- 172.21.15.29:0/3088960909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc102d40 0x7fb7dc1095d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.766+0000 7fb7e0c95640 1 --2- 172.21.15.29:0/3088960909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7dc102380 0x7fb7dc102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.766+0000 7fb7e0c95640 1 --2- 172.21.15.29:0/3088960909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7dc101190 0x7fb7dc1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.766+0000 7fb7e0c95640 1 -- 172.21.15.29:0/3088960909 >> 172.21.15.29:0/3088960909 conn(0x7fb7dc0fc760 msgr2=0x7fb7dc0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.766+0000 7fb7e0c95640 1 -- 172.21.15.29:0/3088960909 shutdown_connections 2024-03-11T19:07:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.766+0000 7fb7e0c95640 1 -- 172.21.15.29:0/3088960909 wait complete. 2024-03-11T19:07:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.767+0000 7fb7e0c95640 1 Processor -- start 2024-03-11T19:07:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.767+0000 7fb7e0c95640 1 -- start start 2024-03-11T19:07:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.767+0000 7fb7e0c95640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7dc101190 0x7fb7dc1970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.767+0000 7fb7e0c95640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc102380 0x7fb7dc197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.767+0000 7fb7e0c95640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7dc102d40 0x7fb7dc19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.767+0000 7fb7e0c95640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc19ebe0 con 0x7fb7dc102380 2024-03-11T19:07:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.767+0000 7fb7e0c95640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc19ed50 con 0x7fb7dc101190 2024-03-11T19:07:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.767+0000 7fb7e0c95640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc19ef10 con 0x7fb7dc102d40 2024-03-11T19:07:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.767+0000 7fb7d9d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc102380 0x7fb7dc197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.767+0000 7fb7d9d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc102380 0x7fb7dc197630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47982/0 (socket says 172.21.15.29:47982) 2024-03-11T19:07:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.767+0000 7fb7d9d9b640 1 -- 172.21.15.29:0/2870658419 learned_addr learned my addr 172.21.15.29:0/2870658419 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.768+0000 7fb7da59c640 1 --2- 172.21.15.29:0/2870658419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7dc101190 0x7fb7dc1970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.768+0000 7fb7dad9d640 1 --2- 172.21.15.29:0/2870658419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7dc102d40 0x7fb7dc19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.768+0000 7fb7da59c640 1 -- 172.21.15.29:0/2870658419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7dc102d40 msgr2=0x7fb7dc19e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.768+0000 7fb7da59c640 1 --2- 172.21.15.29:0/2870658419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7dc102d40 0x7fb7dc19e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.768+0000 7fb7da59c640 1 -- 172.21.15.29:0/2870658419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc102380 msgr2=0x7fb7dc197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.768+0000 7fb7da59c640 1 --2- 172.21.15.29:0/2870658419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc102380 0x7fb7dc197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.768+0000 7fb7da59c640 1 -- 172.21.15.29:0/2870658419 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d4009900 con 0x7fb7dc101190 2024-03-11T19:07:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.768+0000 7fb7d9d9b640 1 --2- 172.21.15.29:0/2870658419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc102380 0x7fb7dc197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:07:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.768+0000 7fb7da59c640 1 --2- 172.21.15.29:0/2870658419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7dc101190 0x7fb7dc1970f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00e9c0 tx=0x7fb7cc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.768+0000 7fb7cb7fe640 1 -- 172.21.15.29:0/2870658419 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc00ccc0 con 0x7fb7dc101190 2024-03-11T19:07:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.768+0000 7fb7e0c95640 1 -- 172.21.15.29:0/2870658419 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7dc19f1f0 con 0x7fb7dc101190 2024-03-11T19:07:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.768+0000 7fb7e0c95640 1 -- 172.21.15.29:0/2870658419 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7dc19f700 con 0x7fb7dc101190 2024-03-11T19:07:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.768+0000 7fb7cb7fe640 1 -- 172.21.15.29:0/2870658419 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb7cc0044e0 con 0x7fb7dc101190 2024-03-11T19:07:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.768+0000 7fb7cb7fe640 1 -- 172.21.15.29:0/2870658419 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc003ea0 con 0x7fb7dc101190 2024-03-11T19:07:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.769+0000 7fb7e0c95640 1 -- 172.21.15.29:0/2870658419 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7dc108500 con 0x7fb7dc101190 2024-03-11T19:07:45.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.770+0000 7fb7cb7fe640 1 -- 172.21.15.29:0/2870658419 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb7cc004030 con 0x7fb7dc101190 2024-03-11T19:07:45.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.771+0000 7fb7cb7fe640 1 --2- 172.21.15.29:0/2870658419 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7c40a1a30 0x7fb7c40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:45.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.771+0000 7fb7d9d9b640 1 --2- 172.21.15.29:0/2870658419 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7c40a1a30 0x7fb7c40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:45.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.771+0000 7fb7cb7fe640 1 -- 172.21.15.29:0/2870658419 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb7cc014070 con 0x7fb7dc101190 2024-03-11T19:07:45.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.771+0000 7fb7d9d9b640 1 --2- 172.21.15.29:0/2870658419 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7c40a1a30 0x7fb7c40a3ef0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc198600 tx=0x7fb7d0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:45.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:45.776+0000 7fb7cb7fe640 1 -- 172.21.15.29:0/2870658419 <== mon.2 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb7cc09a490 con 0x7fb7dc101190 2024-03-11T19:07:45.845 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:07:45 smithi170 podman[66364]: 2024-03-11 19:07:45.553055821 +0000 UTC m=+0.423904222 container init 32ee158abb0682e2a718c2a286f999af1e4cd98bb3b7e02dc349428721a37fdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-11T19:07:45.846 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:07:45 smithi170 podman[66364]: 2024-03-11 19:07:45.555762249 +0000 UTC m=+0.426610665 container start 32ee158abb0682e2a718c2a286f999af1e4cd98bb3b7e02dc349428721a37fdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-b, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-03-11T19:07:45.846 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:07:45 smithi170 ceph-mds[66384]: -- 172.21.15.170:0/676611727 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56473700c1a0 con 0x564736fe3000 2024-03-11T19:07:45.846 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:07:45 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-b[66380]: starting mds.b at 2024-03-11T19:07:45.846 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:07:45 smithi170 bash[66364]: 32ee158abb0682e2a718c2a286f999af1e4cd98bb3b7e02dc349428721a37fdb 2024-03-11T19:07:45.846 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:07:45 smithi170 systemd[1]: Started Ceph mds.b for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:07:46.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.004+0000 7fb7e0c95640 1 -- 172.21.15.29:0/2870658419 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7dc10ca50 con 0x7fb7dc101190 2024-03-11T19:07:46.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.005+0000 7fb7cb7fe640 1 -- 172.21.15.29:0/2870658419 <== mon.2 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10037 (secure 0 0 0) 0x7fb7cc09a020 con 0x7fb7dc101190 2024-03-11T19:07:46.008 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:46.008 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-11T19:07:46.008 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 13 2024-03-11T19:07:46.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.007+0000 7fb7e0c95640 1 -- 172.21.15.29:0/2870658419 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7c40a1a30 msgr2=0x7fb7c40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:46.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.007+0000 7fb7e0c95640 1 --2- 172.21.15.29:0/2870658419 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7c40a1a30 0x7fb7c40a3ef0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc198600 tx=0x7fb7d0005f90 comp rx=0 tx=0).stop 2024-03-11T19:07:46.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.007+0000 7fb7e0c95640 1 -- 172.21.15.29:0/2870658419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7dc101190 msgr2=0x7fb7dc1970f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:46.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.007+0000 7fb7e0c95640 1 --2- 172.21.15.29:0/2870658419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7dc101190 0x7fb7dc1970f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00e9c0 tx=0x7fb7cc00ee80 comp rx=0 tx=0).stop 2024-03-11T19:07:46.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.007+0000 7fb7e0c95640 1 -- 172.21.15.29:0/2870658419 shutdown_connections 2024-03-11T19:07:46.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.007+0000 7fb7e0c95640 1 --2- 172.21.15.29:0/2870658419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7dc102d40 0x7fb7dc19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.007+0000 7fb7e0c95640 1 --2- 172.21.15.29:0/2870658419 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7c40a1a30 0x7fb7c40a3ef0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.007+0000 7fb7e0c95640 1 --2- 172.21.15.29:0/2870658419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc102380 0x7fb7dc197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.007+0000 7fb7e0c95640 1 --2- 172.21.15.29:0/2870658419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7dc101190 0x7fb7dc1970f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.007+0000 7fb7e0c95640 1 -- 172.21.15.29:0/2870658419 >> 172.21.15.29:0/2870658419 conn(0x7fb7dc0fc760 msgr2=0x7fb7dc1056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:46.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.007+0000 7fb7e0c95640 1 -- 172.21.15.29:0/2870658419 shutdown_connections 2024-03-11T19:07:46.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.007+0000 7fb7e0c95640 1 -- 172.21.15.29:0/2870658419 wait complete. 2024-03-11T19:07:46.022 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:07:46.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.123+0000 7fb78e9d0640 1 -- 172.21.15.29:0/4211108695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb788101290 msgr2=0x7fb788101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:46.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.123+0000 7fb78e9d0640 1 --2- 172.21.15.29:0/4211108695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb788101290 0x7fb788101710 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb770009900 tx=0x7fb77002bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:46.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.123+0000 7fb78e9d0640 1 -- 172.21.15.29:0/4211108695 shutdown_connections 2024-03-11T19:07:46.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.123+0000 7fb78e9d0640 1 --2- 172.21.15.29:0/4211108695 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb788101c50 0x7fb7881084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.123+0000 7fb78e9d0640 1 --2- 172.21.15.29:0/4211108695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb788101290 0x7fb788101710 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.123+0000 7fb78e9d0640 1 --2- 172.21.15.29:0/4211108695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7881000a0 0x7fb7881004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.123+0000 7fb78e9d0640 1 -- 172.21.15.29:0/4211108695 >> 172.21.15.29:0/4211108695 conn(0x7fb7880fb610 msgr2=0x7fb7880fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:46.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.123+0000 7fb78e9d0640 1 -- 172.21.15.29:0/4211108695 shutdown_connections 2024-03-11T19:07:46.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.123+0000 7fb78e9d0640 1 -- 172.21.15.29:0/4211108695 wait complete. 2024-03-11T19:07:46.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.123+0000 7fb78e9d0640 1 Processor -- start 2024-03-11T19:07:46.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.123+0000 7fb78e9d0640 1 -- start start 2024-03-11T19:07:46.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb78e9d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7881000a0 0x7fb788104030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb78e9d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb788101290 0x7fb788102670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb78e9d0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb788101c50 0x7fb788102bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb78e9d0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb788103310 con 0x7fb7881000a0 2024-03-11T19:07:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb78e9d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb788103480 con 0x7fb788101290 2024-03-11T19:07:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb78e9d0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7881a1220 con 0x7fb788101c50 2024-03-11T19:07:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb787fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7881000a0 0x7fb788104030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb787fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7881000a0 0x7fb788104030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48006/0 (socket says 172.21.15.29:48006) 2024-03-11T19:07:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb787fff640 1 -- 172.21.15.29:0/2708749169 learned_addr learned my addr 172.21.15.29:0/2708749169 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb78cf6d640 1 --2- 172.21.15.29:0/2708749169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb788101c50 0x7fb788102bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb7877fe640 1 --2- 172.21.15.29:0/2708749169 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb788101290 0x7fb788102670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb787fff640 1 -- 172.21.15.29:0/2708749169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb788101c50 msgr2=0x7fb788102bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb787fff640 1 --2- 172.21.15.29:0/2708749169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb788101c50 0x7fb788102bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb787fff640 1 -- 172.21.15.29:0/2708749169 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb788101290 msgr2=0x7fb788102670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:46.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb787fff640 1 --2- 172.21.15.29:0/2708749169 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb788101290 0x7fb788102670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb787fff640 1 -- 172.21.15.29:0/2708749169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb770009560 con 0x7fb7881000a0 2024-03-11T19:07:46.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb787fff640 1 --2- 172.21.15.29:0/2708749169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7881000a0 0x7fb788104030 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb77800d980 tx=0x7fb77800de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:46.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb7857fa640 1 -- 172.21.15.29:0/2708749169 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778009c30 con 0x7fb7881000a0 2024-03-11T19:07:46.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb7857fa640 1 -- 172.21.15.29:0/2708749169 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb778009d90 con 0x7fb7881000a0 2024-03-11T19:07:46.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb78e9d0640 1 -- 172.21.15.29:0/2708749169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7881a14d0 con 0x7fb7881000a0 2024-03-11T19:07:46.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.124+0000 7fb7857fa640 1 -- 172.21.15.29:0/2708749169 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77800f620 con 0x7fb7881000a0 2024-03-11T19:07:46.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.125+0000 7fb78e9d0640 1 -- 172.21.15.29:0/2708749169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7881a1960 con 0x7fb7881000a0 2024-03-11T19:07:46.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.126+0000 7fb78e9d0640 1 -- 172.21.15.29:0/2708749169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78810b960 con 0x7fb7881000a0 2024-03-11T19:07:46.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.127+0000 7fb7857fa640 1 -- 172.21.15.29:0/2708749169 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb778010460 con 0x7fb7881000a0 2024-03-11T19:07:46.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.127+0000 7fb7857fa640 1 --2- 172.21.15.29:0/2708749169 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7740a1a30 0x7fb7740a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:46.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.127+0000 7fb7877fe640 1 --2- 172.21.15.29:0/2708749169 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7740a1a30 0x7fb7740a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:46.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.127+0000 7fb7857fa640 1 -- 172.21.15.29:0/2708749169 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb7780ccbd0 con 0x7fb7881000a0 2024-03-11T19:07:46.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.128+0000 7fb7877fe640 1 --2- 172.21.15.29:0/2708749169 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7740a1a30 0x7fb7740a3ef0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb770009230 tx=0x7fb770005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:46.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.131+0000 7fb7857fa640 1 -- 172.21.15.29:0/2708749169 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb778096330 con 0x7fb7881000a0 2024-03-11T19:07:46.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.315+0000 7fb78e9d0640 1 -- 172.21.15.29:0/2708749169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb78810baf0 con 0x7fb7881000a0 2024-03-11T19:07:46.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.317+0000 7fb7857fa640 1 -- 172.21.15.29:0/2708749169 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+18456 (secure 0 0 0) 0x7fb7780999f0 con 0x7fb7881000a0 2024-03-11T19:07:46.320 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:46.320 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":83,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:07:00.078122+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:07:46.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.318+0000 7fb78e9d0640 1 -- 172.21.15.29:0/2708749169 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7740a1a30 msgr2=0x7fb7740a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:46.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.318+0000 7fb78e9d0640 1 --2- 172.21.15.29:0/2708749169 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7740a1a30 0x7fb7740a3ef0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb770009230 tx=0x7fb770005fd0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.318+0000 7fb78e9d0640 1 -- 172.21.15.29:0/2708749169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7881000a0 msgr2=0x7fb788104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:46.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.318+0000 7fb78e9d0640 1 --2- 172.21.15.29:0/2708749169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7881000a0 0x7fb788104030 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb77800d980 tx=0x7fb77800de40 comp rx=0 tx=0).stop 2024-03-11T19:07:46.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.319+0000 7fb78e9d0640 1 -- 172.21.15.29:0/2708749169 shutdown_connections 2024-03-11T19:07:46.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.319+0000 7fb78e9d0640 1 --2- 172.21.15.29:0/2708749169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb788101c50 0x7fb788102bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.319+0000 7fb78e9d0640 1 --2- 172.21.15.29:0/2708749169 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb788101290 0x7fb788102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.319+0000 7fb78e9d0640 1 --2- 172.21.15.29:0/2708749169 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7740a1a30 0x7fb7740a3ef0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.319+0000 7fb78e9d0640 1 --2- 172.21.15.29:0/2708749169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7881000a0 0x7fb788104030 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.319+0000 7fb78e9d0640 1 -- 172.21.15.29:0/2708749169 >> 172.21.15.29:0/2708749169 conn(0x7fb7880fb610 msgr2=0x7fb788068350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:46.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.319+0000 7fb78e9d0640 1 -- 172.21.15.29:0/2708749169 shutdown_connections 2024-03-11T19:07:46.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.319+0000 7fb78e9d0640 1 -- 172.21.15.29:0/2708749169 wait complete. 2024-03-11T19:07:46.333 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:03:09.860019+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T19:07:46.334 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:07:46.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.451+0000 7f85cf2df640 1 -- 172.21.15.29:0/2805982657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85c8102d40 msgr2=0x7f85c81095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:46.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.451+0000 7f85cf2df640 1 --2- 172.21.15.29:0/2805982657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85c8102d40 0x7f85c81095d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f85c0009ca0 tx=0x7f85c002c0f0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.451+0000 7f85cf2df640 1 -- 172.21.15.29:0/2805982657 shutdown_connections 2024-03-11T19:07:46.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.451+0000 7f85cf2df640 1 --2- 172.21.15.29:0/2805982657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85c8102d40 0x7f85c81095d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.451+0000 7f85cf2df640 1 --2- 172.21.15.29:0/2805982657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85c8102380 0x7f85c8102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.451+0000 7f85cf2df640 1 --2- 172.21.15.29:0/2805982657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85c8101190 0x7f85c81015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.451+0000 7f85cf2df640 1 -- 172.21.15.29:0/2805982657 >> 172.21.15.29:0/2805982657 conn(0x7f85c80fc6e0 msgr2=0x7f85c80feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:46.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.451+0000 7f85cf2df640 1 -- 172.21.15.29:0/2805982657 shutdown_connections 2024-03-11T19:07:46.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.451+0000 7f85cf2df640 1 -- 172.21.15.29:0/2805982657 wait complete. 2024-03-11T19:07:46.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.452+0000 7f85cf2df640 1 Processor -- start 2024-03-11T19:07:46.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.452+0000 7f85cf2df640 1 -- start start 2024-03-11T19:07:46.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.452+0000 7f85cf2df640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85c8101190 0x7f85c8074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:46.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.452+0000 7f85cf2df640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85c8102380 0x7f85c8073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:46.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.452+0000 7f85cf2df640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85c8102d40 0x7f85c8073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:46.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.452+0000 7f85cf2df640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c8074210 con 0x7f85c8101190 2024-03-11T19:07:46.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.452+0000 7f85cf2df640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c8074380 con 0x7f85c8102380 2024-03-11T19:07:46.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.452+0000 7f85cf2df640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c81a1160 con 0x7f85c8102d40 2024-03-11T19:07:46.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.452+0000 7f85cd07b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85c8101190 0x7f85c8074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:46.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.452+0000 7f85cd07b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85c8101190 0x7f85c8074f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48028/0 (socket says 172.21.15.29:48028) 2024-03-11T19:07:46.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.452+0000 7f85cd07b640 1 -- 172.21.15.29:0/129079203 learned_addr learned my addr 172.21.15.29:0/129079203 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:46.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.452+0000 7f85cc87a640 1 --2- 172.21.15.29:0/129079203 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85c8102380 0x7f85c8073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:46.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.452+0000 7f85cd87c640 1 --2- 172.21.15.29:0/129079203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85c8102d40 0x7f85c8073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:46.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.452+0000 7f85cd07b640 1 -- 172.21.15.29:0/129079203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85c8102d40 msgr2=0x7f85c8073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:46.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.452+0000 7f85cd07b640 1 --2- 172.21.15.29:0/129079203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85c8102d40 0x7f85c8073ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.452+0000 7f85cd07b640 1 -- 172.21.15.29:0/129079203 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85c8102380 msgr2=0x7f85c8073570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:46.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.452+0000 7f85cd07b640 1 --2- 172.21.15.29:0/129079203 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85c8102380 0x7f85c8073570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.452+0000 7f85cd07b640 1 -- 172.21.15.29:0/129079203 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c0009900 con 0x7f85c8101190 2024-03-11T19:07:46.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.453+0000 7f85cd07b640 1 --2- 172.21.15.29:0/129079203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85c8101190 0x7f85c8074f30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f85b800e9f0 tx=0x7f85b800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:46.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.453+0000 7f85be7fc640 1 -- 172.21.15.29:0/129079203 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b80096e0 con 0x7f85c8101190 2024-03-11T19:07:46.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.453+0000 7f85be7fc640 1 -- 172.21.15.29:0/129079203 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f85b80044e0 con 0x7f85c8101190 2024-03-11T19:07:46.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.453+0000 7f85cf2df640 1 -- 172.21.15.29:0/129079203 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c81a1410 con 0x7f85c8101190 2024-03-11T19:07:46.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.453+0000 7f85be7fc640 1 -- 172.21.15.29:0/129079203 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b8010430 con 0x7f85c8101190 2024-03-11T19:07:46.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.453+0000 7f85cf2df640 1 -- 172.21.15.29:0/129079203 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c81a18d0 con 0x7f85c8101190 2024-03-11T19:07:46.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.454+0000 7f85cf2df640 1 -- 172.21.15.29:0/129079203 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c81085d0 con 0x7f85c8101190 2024-03-11T19:07:46.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.455+0000 7f85be7fc640 1 -- 172.21.15.29:0/129079203 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f85b8003f90 con 0x7f85c8101190 2024-03-11T19:07:46.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.455+0000 7f85be7fc640 1 --2- 172.21.15.29:0/129079203 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f85b40a1a30 0x7f85b40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:46.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.456+0000 7f85cc87a640 1 --2- 172.21.15.29:0/129079203 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f85b40a1a30 0x7f85b40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:46.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.456+0000 7f85be7fc640 1 -- 172.21.15.29:0/129079203 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f85b8014070 con 0x7f85c8101190 2024-03-11T19:07:46.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.456+0000 7f85cc87a640 1 --2- 172.21.15.29:0/129079203 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f85b40a1a30 0x7f85b40a3ef0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f85c8074ca0 tx=0x7f85c4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:46.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.460+0000 7f85be7fc640 1 -- 172.21.15.29:0/129079203 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f85b8096990 con 0x7f85c8101190 2024-03-11T19:07:46.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.661+0000 7f85cf2df640 1 -- 172.21.15.29:0/129079203 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f85c810ca50 con 0x7f85c8101190 2024-03-11T19:07:46.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.661+0000 7f85be7fc640 1 -- 172.21.15.29:0/129079203 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f85b80b8e60 con 0x7f85c8101190 2024-03-11T19:07:46.664 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-11T19:07:46.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.663+0000 7f85cf2df640 1 -- 172.21.15.29:0/129079203 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f85b40a1a30 msgr2=0x7f85b40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:46.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.663+0000 7f85cf2df640 1 --2- 172.21.15.29:0/129079203 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f85b40a1a30 0x7f85b40a3ef0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f85c8074ca0 tx=0x7f85c4005f90 comp rx=0 tx=0).stop 2024-03-11T19:07:46.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.663+0000 7f85cf2df640 1 -- 172.21.15.29:0/129079203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85c8101190 msgr2=0x7f85c8074f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:46.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.663+0000 7f85cf2df640 1 --2- 172.21.15.29:0/129079203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85c8101190 0x7f85c8074f30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f85b800e9f0 tx=0x7f85b800eeb0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.663+0000 7f85cf2df640 1 -- 172.21.15.29:0/129079203 shutdown_connections 2024-03-11T19:07:46.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.663+0000 7f85cf2df640 1 --2- 172.21.15.29:0/129079203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85c8102d40 0x7f85c8073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.663+0000 7f85cf2df640 1 --2- 172.21.15.29:0/129079203 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85c8102380 0x7f85c8073570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.663+0000 7f85cf2df640 1 --2- 172.21.15.29:0/129079203 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f85b40a1a30 0x7f85b40a3ef0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.663+0000 7f85cf2df640 1 --2- 172.21.15.29:0/129079203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85c8101190 0x7f85c8074f30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.663+0000 7f85cf2df640 1 -- 172.21.15.29:0/129079203 >> 172.21.15.29:0/129079203 conn(0x7f85c80fc6e0 msgr2=0x7f85c81057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:46.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.663+0000 7f85cf2df640 1 -- 172.21.15.29:0/129079203 shutdown_connections 2024-03-11T19:07:46.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.663+0000 7f85cf2df640 1 -- 172.21.15.29:0/129079203 wait complete. 2024-03-11T19:07:46.677 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-11T19:07:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.778+0000 7f64b12c0640 1 -- 172.21.15.29:0/2263184825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64ac106f20 msgr2=0x7f64ac109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.778+0000 7f64b12c0640 1 --2- 172.21.15.29:0/2263184825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64ac106f20 0x7f64ac109310 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f64a4009ca0 tx=0x7f64a402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.779+0000 7f64b12c0640 1 -- 172.21.15.29:0/2263184825 shutdown_connections 2024-03-11T19:07:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.779+0000 7f64b12c0640 1 --2- 172.21.15.29:0/2263184825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64ac106f20 0x7f64ac109310 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.779+0000 7f64b12c0640 1 --2- 172.21.15.29:0/2263184825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64ac1045f0 0x7f64ac1069e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.779+0000 7f64b12c0640 1 --2- 172.21.15.29:0/2263184825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64ac101cc0 0x7f64ac1040b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.779+0000 7f64b12c0640 1 -- 172.21.15.29:0/2263184825 >> 172.21.15.29:0/2263184825 conn(0x7f64ac0fb630 msgr2=0x7f64ac0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.779+0000 7f64b12c0640 1 -- 172.21.15.29:0/2263184825 shutdown_connections 2024-03-11T19:07:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.779+0000 7f64b12c0640 1 -- 172.21.15.29:0/2263184825 wait complete. 2024-03-11T19:07:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.779+0000 7f64b12c0640 1 Processor -- start 2024-03-11T19:07:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.779+0000 7f64b12c0640 1 -- start start 2024-03-11T19:07:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.779+0000 7f64b12c0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64ac101cc0 0x7f64ac197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.779+0000 7f64b12c0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64ac1045f0 0x7f64ac1976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.779+0000 7f64b12c0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64ac106f20 0x7f64ac19e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64b12c0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac19ec60 con 0x7f64ac106f20 2024-03-11T19:07:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64b12c0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac19edd0 con 0x7f64ac1045f0 2024-03-11T19:07:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64b12c0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac19ef90 con 0x7f64ac101cc0 2024-03-11T19:07:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64aaffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64ac101cc0 0x7f64ac197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64ac106f20 0x7f64ac19e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64aaffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64ac101cc0 0x7f64ac197170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:55518/0 (socket says 172.21.15.29:55518) 2024-03-11T19:07:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64aaffd640 1 -- 172.21.15.29:0/2941675171 learned_addr learned my addr 172.21.15.29:0/2941675171 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64ac106f20 0x7f64ac19e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48054/0 (socket says 172.21.15.29:48054) 2024-03-11T19:07:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64a27fc640 1 --2- 172.21.15.29:0/2941675171 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64ac1045f0 0x7f64ac1976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64aaffd640 1 -- 172.21.15.29:0/2941675171 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64ac1045f0 msgr2=0x7f64ac1976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64aaffd640 1 --2- 172.21.15.29:0/2941675171 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64ac1045f0 0x7f64ac1976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64aaffd640 1 -- 172.21.15.29:0/2941675171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64ac106f20 msgr2=0x7f64ac19e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64aaffd640 1 --2- 172.21.15.29:0/2941675171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64ac106f20 0x7f64ac19e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64aaffd640 1 -- 172.21.15.29:0/2941675171 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a4009900 con 0x7f64ac101cc0 2024-03-11T19:07:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64ab7fe640 1 --2- 172.21.15.29:0/2941675171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64ac106f20 0x7f64ac19e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:07:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64aaffd640 1 --2- 172.21.15.29:0/2941675171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64ac101cc0 0x7f64ac197170 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f649c00e9f0 tx=0x7f649c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64a8ff9640 1 -- 172.21.15.29:0/2941675171 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c009720 con 0x7f64ac101cc0 2024-03-11T19:07:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64b12c0640 1 -- 172.21.15.29:0/2941675171 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ac19f270 con 0x7f64ac101cc0 2024-03-11T19:07:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64a8ff9640 1 -- 172.21.15.29:0/2941675171 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f649c00cd80 con 0x7f64ac101cc0 2024-03-11T19:07:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64a8ff9640 1 -- 172.21.15.29:0/2941675171 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c010430 con 0x7f64ac101cc0 2024-03-11T19:07:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.780+0000 7f64b12c0640 1 -- 172.21.15.29:0/2941675171 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ac19f780 con 0x7f64ac101cc0 2024-03-11T19:07:46.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.781+0000 7f64b12c0640 1 -- 172.21.15.29:0/2941675171 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ac0fd120 con 0x7f64ac101cc0 2024-03-11T19:07:46.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.782+0000 7f64a8ff9640 1 -- 172.21.15.29:0/2941675171 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f649c005020 con 0x7f64ac101cc0 2024-03-11T19:07:46.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.783+0000 7f64a8ff9640 1 --2- 172.21.15.29:0/2941675171 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f64980a1a30 0x7f64980a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:46.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.783+0000 7f64a8ff9640 1 -- 172.21.15.29:0/2941675171 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f649c014070 con 0x7f64ac101cc0 2024-03-11T19:07:46.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.783+0000 7f64a27fc640 1 --2- 172.21.15.29:0/2941675171 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f64980a1a30 0x7f64980a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:46.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.783+0000 7f64a27fc640 1 --2- 172.21.15.29:0/2941675171 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f64980a1a30 0x7f64980a3ef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f64ac198680 tx=0x7f649400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:46.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.787+0000 7f64a8ff9640 1 -- 172.21.15.29:0/2941675171 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f649c0969c0 con 0x7f64ac101cc0 2024-03-11T19:07:46.973 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:46 smithi029 ceph-mon[32439]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:46.974 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:46 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:46.974 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:46 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:46.974 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:46 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:46.974 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:46 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:46.974 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:46 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:46.974 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:46 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] up:boot 2024-03-11T19:07:46.974 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:46 smithi029 ceph-mon[32439]: fsmap 12 up:standby 2024-03-11T19:07:46.974 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:46 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T19:07:46.974 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:46 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:07:46.974 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:46 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2870658419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:46.974 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:46 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2708749169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:07:46.974 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:46 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/129079203' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:07:46.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.972+0000 7f64b12c0640 1 -- 172.21.15.29:0/2941675171 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f64ac02cda0 con 0x7f64ac101cc0 2024-03-11T19:07:46.976 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:46 smithi190 ceph-mon[36292]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:46.976 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:46 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:46.976 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:46 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:46.976 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:46 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:46.976 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:46 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:46.976 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:46 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:46.976 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:46 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] up:boot 2024-03-11T19:07:46.976 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:46 smithi190 ceph-mon[36292]: fsmap 12 up:standby 2024-03-11T19:07:46.976 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:46 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T19:07:46.976 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:46 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:07:46.976 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:46 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2870658419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:46.976 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:46 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2708749169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:07:46.977 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:46 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/129079203' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:07:46.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.973+0000 7f64a8ff9640 1 -- 172.21.15.29:0/2941675171 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v83) v1 ==== 84+0+65 (secure 0 0 0) 0x7f649c09a080 con 0x7f64ac101cc0 2024-03-11T19:07:46.977 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:46.977 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2024-03-11T19:07:46.977 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:07:46.977 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2024-03-11T19:07:46.977 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2024-03-11T19:07:46.978 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:07:46.978 INFO:teuthology.orchestra.run.smithi029.stdout:] 2024-03-11T19:07:46.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.975+0000 7f64b12c0640 1 -- 172.21.15.29:0/2941675171 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f64980a1a30 msgr2=0x7f64980a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:46.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.975+0000 7f64b12c0640 1 --2- 172.21.15.29:0/2941675171 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f64980a1a30 0x7f64980a3ef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f64ac198680 tx=0x7f649400b410 comp rx=0 tx=0).stop 2024-03-11T19:07:46.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.975+0000 7f64b12c0640 1 -- 172.21.15.29:0/2941675171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64ac101cc0 msgr2=0x7f64ac197170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:46.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.975+0000 7f64b12c0640 1 --2- 172.21.15.29:0/2941675171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64ac101cc0 0x7f64ac197170 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f649c00e9f0 tx=0x7f649c00edf0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.975+0000 7f64b12c0640 1 -- 172.21.15.29:0/2941675171 shutdown_connections 2024-03-11T19:07:46.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.975+0000 7f64b12c0640 1 --2- 172.21.15.29:0/2941675171 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f64980a1a30 0x7f64980a3ef0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.975+0000 7f64b12c0640 1 --2- 172.21.15.29:0/2941675171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64ac106f20 0x7f64ac19e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.975+0000 7f64b12c0640 1 --2- 172.21.15.29:0/2941675171 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64ac1045f0 0x7f64ac1976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.975+0000 7f64b12c0640 1 --2- 172.21.15.29:0/2941675171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64ac101cc0 0x7f64ac197170 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:46.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.975+0000 7f64b12c0640 1 -- 172.21.15.29:0/2941675171 >> 172.21.15.29:0/2941675171 conn(0x7f64ac0fb630 msgr2=0x7f64ac102740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:46.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.975+0000 7f64b12c0640 1 -- 172.21.15.29:0/2941675171 shutdown_connections 2024-03-11T19:07:46.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:46.975+0000 7f64b12c0640 1 -- 172.21.15.29:0/2941675171 wait complete. 2024-03-11T19:07:46.990 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-11T19:07:46.990 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-11T19:07:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:46 smithi170 ceph-mon[35690]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:46 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] up:boot 2024-03-11T19:07:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:46 smithi170 ceph-mon[35690]: fsmap 12 up:standby 2024-03-11T19:07:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T19:07:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:46 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:07:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:46 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2870658419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:46 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2708749169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:07:47.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:46 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/129079203' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:07:47.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.095+0000 7f13b99e2640 1 -- 172.21.15.29:0/3829848472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b4102d40 msgr2=0x7f13b41095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:47.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.095+0000 7f13b99e2640 1 --2- 172.21.15.29:0/3829848472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b4102d40 0x7f13b41095d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f13ac009ca0 tx=0x7f13ac02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:07:47.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.095+0000 7f13b99e2640 1 -- 172.21.15.29:0/3829848472 shutdown_connections 2024-03-11T19:07:47.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.095+0000 7f13b99e2640 1 --2- 172.21.15.29:0/3829848472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b4102d40 0x7f13b41095d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:47.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.095+0000 7f13b99e2640 1 --2- 172.21.15.29:0/3829848472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13b4102380 0x7f13b4102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:47.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.095+0000 7f13b99e2640 1 --2- 172.21.15.29:0/3829848472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13b4101190 0x7f13b41015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:47.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.095+0000 7f13b99e2640 1 -- 172.21.15.29:0/3829848472 >> 172.21.15.29:0/3829848472 conn(0x7f13b40fc720 msgr2=0x7f13b40feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.095+0000 7f13b99e2640 1 -- 172.21.15.29:0/3829848472 shutdown_connections 2024-03-11T19:07:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.095+0000 7f13b99e2640 1 -- 172.21.15.29:0/3829848472 wait complete. 2024-03-11T19:07:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.095+0000 7f13b99e2640 1 Processor -- start 2024-03-11T19:07:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.096+0000 7f13b99e2640 1 -- start start 2024-03-11T19:07:47.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.096+0000 7f13b99e2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b4101190 0x7f13b4074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:47.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.096+0000 7f13b99e2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13b4102380 0x7f13b4073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:47.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.096+0000 7f13b99e2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13b4102d40 0x7f13b4073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:47.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.096+0000 7f13b99e2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b4074210 con 0x7f13b4101190 2024-03-11T19:07:47.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.096+0000 7f13b99e2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b4074380 con 0x7f13b4102d40 2024-03-11T19:07:47.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.096+0000 7f13b99e2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b41a1140 con 0x7f13b4102380 2024-03-11T19:07:47.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.096+0000 7f13b2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b4101190 0x7f13b4074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:47.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.096+0000 7f13b2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b4101190 0x7f13b4074f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48074/0 (socket says 172.21.15.29:48074) 2024-03-11T19:07:47.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.096+0000 7f13b2ffd640 1 -- 172.21.15.29:0/1237920874 learned_addr learned my addr 172.21.15.29:0/1237920874 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:47.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.096+0000 7f13b37fe640 1 --2- 172.21.15.29:0/1237920874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13b4102d40 0x7f13b4073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:47.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.096+0000 7f13b27fc640 1 --2- 172.21.15.29:0/1237920874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13b4102380 0x7f13b4073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:47.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.096+0000 7f13b2ffd640 1 -- 172.21.15.29:0/1237920874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13b4102380 msgr2=0x7f13b4073570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:47.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.096+0000 7f13b2ffd640 1 --2- 172.21.15.29:0/1237920874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13b4102380 0x7f13b4073570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:47.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.096+0000 7f13b2ffd640 1 -- 172.21.15.29:0/1237920874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13b4102d40 msgr2=0x7f13b4073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:47.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.096+0000 7f13b2ffd640 1 --2- 172.21.15.29:0/1237920874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13b4102d40 0x7f13b4073ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:47.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.096+0000 7f13b2ffd640 1 -- 172.21.15.29:0/1237920874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ac009900 con 0x7f13b4101190 2024-03-11T19:07:47.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.096+0000 7f13b2ffd640 1 --2- 172.21.15.29:0/1237920874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b4101190 0x7f13b4074f30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f13a400d980 tx=0x7f13a400de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:47.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.096+0000 7f13b89e0640 1 -- 172.21.15.29:0/1237920874 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a4009c30 con 0x7f13b4101190 2024-03-11T19:07:47.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.097+0000 7f13b99e2640 1 -- 172.21.15.29:0/1237920874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b41a13f0 con 0x7f13b4101190 2024-03-11T19:07:47.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.097+0000 7f13b89e0640 1 -- 172.21.15.29:0/1237920874 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f13a4009d90 con 0x7f13b4101190 2024-03-11T19:07:47.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.097+0000 7f13b89e0640 1 -- 172.21.15.29:0/1237920874 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a400f620 con 0x7f13b4101190 2024-03-11T19:07:47.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.097+0000 7f13b99e2640 1 -- 172.21.15.29:0/1237920874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b41a18b0 con 0x7f13b4101190 2024-03-11T19:07:47.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.098+0000 7f13b99e2640 1 -- 172.21.15.29:0/1237920874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b4108520 con 0x7f13b4101190 2024-03-11T19:07:47.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.099+0000 7f13b89e0640 1 -- 172.21.15.29:0/1237920874 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f13a4010460 con 0x7f13b4101190 2024-03-11T19:07:47.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.099+0000 7f13b89e0640 1 --2- 172.21.15.29:0/1237920874 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f139c0a1a30 0x7f139c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:47.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.099+0000 7f13b27fc640 1 --2- 172.21.15.29:0/1237920874 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f139c0a1a30 0x7f139c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:47.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.099+0000 7f13b89e0640 1 -- 172.21.15.29:0/1237920874 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f13a40ccc30 con 0x7f13b4101190 2024-03-11T19:07:47.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.100+0000 7f13b27fc640 1 --2- 172.21.15.29:0/1237920874 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f139c0a1a30 0x7f139c0a3ef0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f13b4074ca0 tx=0x7f13a8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:47.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.104+0000 7f13b89e0640 1 -- 172.21.15.29:0/1237920874 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f13a4096390 con 0x7f13b4101190 2024-03-11T19:07:47.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.306+0000 7f13b99e2640 1 -- 172.21.15.29:0/1237920874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f13b4107f90 con 0x7f13b4101190 2024-03-11T19:07:47.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.716+0000 7f13b89e0640 1 -- 172.21.15.29:0/1237920874 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v84) v1 ==== 138+0+0 (secure 0 0 0) 0x7f13a4099a50 con 0x7f13b4101190 2024-03-11T19:07:47.719 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2024-03-11T19:07:47.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.720+0000 7f13b99e2640 1 -- 172.21.15.29:0/1237920874 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f139c0a1a30 msgr2=0x7f139c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:47.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.720+0000 7f13b99e2640 1 --2- 172.21.15.29:0/1237920874 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f139c0a1a30 0x7f139c0a3ef0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f13b4074ca0 tx=0x7f13a8005f90 comp rx=0 tx=0).stop 2024-03-11T19:07:47.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.720+0000 7f13b99e2640 1 -- 172.21.15.29:0/1237920874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b4101190 msgr2=0x7f13b4074f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:47.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.720+0000 7f13b99e2640 1 --2- 172.21.15.29:0/1237920874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b4101190 0x7f13b4074f30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f13a400d980 tx=0x7f13a400de40 comp rx=0 tx=0).stop 2024-03-11T19:07:47.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.721+0000 7f13b99e2640 1 -- 172.21.15.29:0/1237920874 shutdown_connections 2024-03-11T19:07:47.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.721+0000 7f13b99e2640 1 --2- 172.21.15.29:0/1237920874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13b4102d40 0x7f13b4073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:47.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.721+0000 7f13b99e2640 1 --2- 172.21.15.29:0/1237920874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13b4102380 0x7f13b4073570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:47.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.721+0000 7f13b99e2640 1 --2- 172.21.15.29:0/1237920874 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f139c0a1a30 0x7f139c0a3ef0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:47.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.721+0000 7f13b99e2640 1 --2- 172.21.15.29:0/1237920874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b4101190 0x7f13b4074f30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:47.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.721+0000 7f13b99e2640 1 -- 172.21.15.29:0/1237920874 >> 172.21.15.29:0/1237920874 conn(0x7f13b40fc720 msgr2=0x7f13b41056f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:47.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.721+0000 7f13b99e2640 1 -- 172.21.15.29:0/1237920874 shutdown_connections 2024-03-11T19:07:47.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.721+0000 7f13b99e2640 1 -- 172.21.15.29:0/1237920874 wait complete. 2024-03-11T19:07:47.736 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-11T19:07:47.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.843+0000 7f4ccb7ac640 1 -- 172.21.15.29:0/2250187735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc4100080 msgr2=0x7f4cc41004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:47.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.843+0000 7f4ccb7ac640 1 --2- 172.21.15.29:0/2250187735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc4100080 0x7f4cc41004a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0009900 tx=0x7f4cc002bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.843+0000 7f4ccb7ac640 1 -- 172.21.15.29:0/2250187735 shutdown_connections 2024-03-11T19:07:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.843+0000 7f4ccb7ac640 1 --2- 172.21.15.29:0/2250187735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cc4101c30 0x7f4cc41084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.843+0000 7f4ccb7ac640 1 --2- 172.21.15.29:0/2250187735 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cc4101270 0x7f4cc41016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.843+0000 7f4ccb7ac640 1 --2- 172.21.15.29:0/2250187735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc4100080 0x7f4cc41004a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.843+0000 7f4ccb7ac640 1 -- 172.21.15.29:0/2250187735 >> 172.21.15.29:0/2250187735 conn(0x7f4cc40fb610 msgr2=0x7f4cc40fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.843+0000 7f4ccb7ac640 1 -- 172.21.15.29:0/2250187735 shutdown_connections 2024-03-11T19:07:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.843+0000 7f4ccb7ac640 1 -- 172.21.15.29:0/2250187735 wait complete. 2024-03-11T19:07:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.844+0000 7f4ccb7ac640 1 Processor -- start 2024-03-11T19:07:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.844+0000 7f4ccb7ac640 1 -- start start 2024-03-11T19:07:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.844+0000 7f4ccb7ac640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cc4100080 0x7f4cc4197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.844+0000 7f4ccb7ac640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cc4101270 0x7f4cc4197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.844+0000 7f4ccb7ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc4101c30 0x7f4cc419e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.844+0000 7f4ccb7ac640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc419ebf0 con 0x7f4cc4101c30 2024-03-11T19:07:47.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.844+0000 7f4ccb7ac640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc419ed60 con 0x7f4cc4100080 2024-03-11T19:07:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.844+0000 7f4ccb7ac640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc419ef20 con 0x7f4cc4101270 2024-03-11T19:07:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.844+0000 7f4cc9d49640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc4101c30 0x7f4cc419e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.844+0000 7f4cc9d49640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc4101c30 0x7f4cc419e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48090/0 (socket says 172.21.15.29:48090) 2024-03-11T19:07:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.844+0000 7f4cc9d49640 1 -- 172.21.15.29:0/4183156035 learned_addr learned my addr 172.21.15.29:0/4183156035 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.844+0000 7f4cc8d47640 1 --2- 172.21.15.29:0/4183156035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cc4101270 0x7f4cc4197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.844+0000 7f4cc9548640 1 --2- 172.21.15.29:0/4183156035 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cc4100080 0x7f4cc4197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.844+0000 7f4cc9d49640 1 -- 172.21.15.29:0/4183156035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cc4101270 msgr2=0x7f4cc4197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.844+0000 7f4cc9d49640 1 --2- 172.21.15.29:0/4183156035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cc4101270 0x7f4cc4197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.844+0000 7f4cc9d49640 1 -- 172.21.15.29:0/4183156035 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cc4100080 msgr2=0x7f4cc4197100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.844+0000 7f4cc9d49640 1 --2- 172.21.15.29:0/4183156035 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cc4100080 0x7f4cc4197100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.844+0000 7f4cc9d49640 1 -- 172.21.15.29:0/4183156035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc0009560 con 0x7f4cc4101c30 2024-03-11T19:07:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.844+0000 7f4cc8d47640 1 --2- 172.21.15.29:0/4183156035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cc4101270 0x7f4cc4197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:07:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.845+0000 7f4cc9d49640 1 --2- 172.21.15.29:0/4183156035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc4101c30 0x7f4cc419e6b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc00eb10 tx=0x7f4cbc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.845+0000 7f4cba7fc640 1 -- 172.21.15.29:0/4183156035 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cbc004b20 con 0x7f4cc4101c30 2024-03-11T19:07:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.845+0000 7f4ccb7ac640 1 -- 172.21.15.29:0/4183156035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc419f200 con 0x7f4cc4101c30 2024-03-11T19:07:47.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.845+0000 7f4cba7fc640 1 -- 172.21.15.29:0/4183156035 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4cbc004c80 con 0x7f4cc4101c30 2024-03-11T19:07:47.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.845+0000 7f4cba7fc640 1 -- 172.21.15.29:0/4183156035 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cbc010690 con 0x7f4cc4101c30 2024-03-11T19:07:47.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.845+0000 7f4ccb7ac640 1 -- 172.21.15.29:0/4183156035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc419f760 con 0x7f4cc4101c30 2024-03-11T19:07:47.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.846+0000 7f4ccb7ac640 1 -- 172.21.15.29:0/4183156035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc41073c0 con 0x7f4cc4101c30 2024-03-11T19:07:47.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.847+0000 7f4cba7fc640 1 -- 172.21.15.29:0/4183156035 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4cbc004df0 con 0x7f4cc4101c30 2024-03-11T19:07:47.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.847+0000 7f4cba7fc640 1 --2- 172.21.15.29:0/4183156035 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4cb00a1a30 0x7f4cb00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:47.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.848+0000 7f4cc9548640 1 --2- 172.21.15.29:0/4183156035 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4cb00a1a30 0x7f4cb00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:47.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.848+0000 7f4cba7fc640 1 -- 172.21.15.29:0/4183156035 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f4cbc014070 con 0x7f4cc4101c30 2024-03-11T19:07:47.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.848+0000 7f4cc9548640 1 --2- 172.21.15.29:0/4183156035 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4cb00a1a30 0x7f4cb00a3ef0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4cc002c7d0 tx=0x7f4cc0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:47.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:47.851+0000 7f4cba7fc640 1 -- 172.21.15.29:0/4183156035 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4cbc096d10 con 0x7f4cc4101c30 2024-03-11T19:07:48.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:47 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2941675171' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-11T19:07:48.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:47 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1237920874' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-11T19:07:48.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:48.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:48.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:48.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:48.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:48.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:48.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.035+0000 7f4ccb7ac640 1 -- 172.21.15.29:0/4183156035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4cc402cdd0 con 0x7f4cc4101c30 2024-03-11T19:07:48.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:47 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2941675171' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-11T19:07:48.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:47 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1237920874' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-11T19:07:48.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:48.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:48.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:48.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:48.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:48.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:48.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:47 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2941675171' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-11T19:07:48.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:47 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1237920874' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-11T19:07:48.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:48.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:48.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:48.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:48.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:48.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:48.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.722+0000 7f4cba7fc640 1 -- 172.21.15.29:0/4183156035 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v85) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4cbc09a3d0 con 0x7f4cc4101c30 2024-03-11T19:07:48.725 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2024-03-11T19:07:48.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.727+0000 7f4ccb7ac640 1 -- 172.21.15.29:0/4183156035 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4cb00a1a30 msgr2=0x7f4cb00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:48.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.727+0000 7f4ccb7ac640 1 --2- 172.21.15.29:0/4183156035 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4cb00a1a30 0x7f4cb00a3ef0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4cc002c7d0 tx=0x7f4cc0005fd0 comp rx=0 tx=0).stop 2024-03-11T19:07:48.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.727+0000 7f4ccb7ac640 1 -- 172.21.15.29:0/4183156035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc4101c30 msgr2=0x7f4cc419e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:48.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.727+0000 7f4ccb7ac640 1 --2- 172.21.15.29:0/4183156035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc4101c30 0x7f4cc419e6b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc00eb10 tx=0x7f4cbc00c2d0 comp rx=0 tx=0).stop 2024-03-11T19:07:48.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.731+0000 7f4ccb7ac640 1 -- 172.21.15.29:0/4183156035 shutdown_connections 2024-03-11T19:07:48.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.732+0000 7f4ccb7ac640 1 --2- 172.21.15.29:0/4183156035 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4cb00a1a30 0x7f4cb00a3ef0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:48.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.732+0000 7f4ccb7ac640 1 --2- 172.21.15.29:0/4183156035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc4101c30 0x7f4cc419e6b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:48.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.732+0000 7f4ccb7ac640 1 --2- 172.21.15.29:0/4183156035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cc4101270 0x7f4cc4197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:48.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.732+0000 7f4ccb7ac640 1 --2- 172.21.15.29:0/4183156035 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cc4100080 0x7f4cc4197100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:48.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.732+0000 7f4ccb7ac640 1 -- 172.21.15.29:0/4183156035 >> 172.21.15.29:0/4183156035 conn(0x7f4cc40fb610 msgr2=0x7f4cc4104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:48.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.732+0000 7f4ccb7ac640 1 -- 172.21.15.29:0/4183156035 shutdown_connections 2024-03-11T19:07:48.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.732+0000 7f4ccb7ac640 1 -- 172.21.15.29:0/4183156035 wait complete. 2024-03-11T19:07:48.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-11T19:07:48.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.858+0000 7f9740508640 1 -- 172.21.15.29:0/2678575382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9738102d60 msgr2=0x7f97381095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:48.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.858+0000 7f9740508640 1 --2- 172.21.15.29:0/2678575382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9738102d60 0x7f97381095f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9730009ca0 tx=0x7f973002c0f0 comp rx=0 tx=0).stop 2024-03-11T19:07:48.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.858+0000 7f9740508640 1 -- 172.21.15.29:0/2678575382 shutdown_connections 2024-03-11T19:07:48.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.858+0000 7f9740508640 1 --2- 172.21.15.29:0/2678575382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9738102d60 0x7f97381095f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:48.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.858+0000 7f9740508640 1 --2- 172.21.15.29:0/2678575382 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97381023a0 0x7f9738102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:48.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.858+0000 7f9740508640 1 --2- 172.21.15.29:0/2678575382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97381011b0 0x7f97381015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:48.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.858+0000 7f9740508640 1 -- 172.21.15.29:0/2678575382 >> 172.21.15.29:0/2678575382 conn(0x7f97380fc720 msgr2=0x7f97380feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:48.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.858+0000 7f9740508640 1 -- 172.21.15.29:0/2678575382 shutdown_connections 2024-03-11T19:07:48.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.858+0000 7f9740508640 1 -- 172.21.15.29:0/2678575382 wait complete. 2024-03-11T19:07:48.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.858+0000 7f9740508640 1 Processor -- start 2024-03-11T19:07:48.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f9740508640 1 -- start start 2024-03-11T19:07:48.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f9740508640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97381011b0 0x7f9738197110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:48.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f9740508640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97381023a0 0x7f9738197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:48.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f9740508640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9738102d60 0x7f973819e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:48.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f9740508640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973819ec00 con 0x7f9738102d60 2024-03-11T19:07:48.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f9740508640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973819ed70 con 0x7f97381011b0 2024-03-11T19:07:48.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f9740508640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973819ef30 con 0x7f97381023a0 2024-03-11T19:07:48.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f973eaa5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9738102d60 0x7f973819e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:48.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f973eaa5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9738102d60 0x7f973819e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48110/0 (socket says 172.21.15.29:48110) 2024-03-11T19:07:48.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f973eaa5640 1 -- 172.21.15.29:0/3210384188 learned_addr learned my addr 172.21.15.29:0/3210384188 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:48.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f973e2a4640 1 --2- 172.21.15.29:0/3210384188 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97381011b0 0x7f9738197110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:48.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f973daa3640 1 --2- 172.21.15.29:0/3210384188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97381023a0 0x7f9738197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:48.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f973eaa5640 1 -- 172.21.15.29:0/3210384188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97381023a0 msgr2=0x7f9738197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:48.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f973eaa5640 1 --2- 172.21.15.29:0/3210384188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97381023a0 0x7f9738197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:48.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f973eaa5640 1 -- 172.21.15.29:0/3210384188 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97381011b0 msgr2=0x7f9738197110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:48.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f973eaa5640 1 --2- 172.21.15.29:0/3210384188 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97381011b0 0x7f9738197110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:48.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f973eaa5640 1 -- 172.21.15.29:0/3210384188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9730009900 con 0x7f9738102d60 2024-03-11T19:07:48.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f973daa3640 1 --2- 172.21.15.29:0/3210384188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97381023a0 0x7f9738197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:07:48.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f973eaa5640 1 --2- 172.21.15.29:0/3210384188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9738102d60 0x7f973819e6c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9730037040 tx=0x7f9730032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:48.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.859+0000 7f972f7fe640 1 -- 172.21.15.29:0/3210384188 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9730034e70 con 0x7f9738102d60 2024-03-11T19:07:48.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.860+0000 7f972f7fe640 1 -- 172.21.15.29:0/3210384188 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9730003e20 con 0x7f9738102d60 2024-03-11T19:07:48.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.860+0000 7f9740508640 1 -- 172.21.15.29:0/3210384188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f973819f1b0 con 0x7f9738102d60 2024-03-11T19:07:48.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.860+0000 7f972f7fe640 1 -- 172.21.15.29:0/3210384188 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9730004800 con 0x7f9738102d60 2024-03-11T19:07:48.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.860+0000 7f9740508640 1 -- 172.21.15.29:0/3210384188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f973819f660 con 0x7f9738102d60 2024-03-11T19:07:48.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.860+0000 7f9740508640 1 -- 172.21.15.29:0/3210384188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9738108520 con 0x7f9738102d60 2024-03-11T19:07:48.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.862+0000 7f972f7fe640 1 -- 172.21.15.29:0/3210384188 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f973003b040 con 0x7f9738102d60 2024-03-11T19:07:48.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.862+0000 7f972f7fe640 1 --2- 172.21.15.29:0/3210384188 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97240a1a30 0x7f97240a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:48.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.862+0000 7f973e2a4640 1 --2- 172.21.15.29:0/3210384188 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97240a1a30 0x7f97240a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:48.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.862+0000 7f972f7fe640 1 -- 172.21.15.29:0/3210384188 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f97300efd80 con 0x7f9738102d60 2024-03-11T19:07:48.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.863+0000 7f973e2a4640 1 --2- 172.21.15.29:0/3210384188 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97240a1a30 0x7f97240a3ef0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9738102200 tx=0x7f9728005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:48.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:48.868+0000 7f972f7fe640 1 -- 172.21.15.29:0/3210384188 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f97300b9140 con 0x7f9738102d60 2024-03-11T19:07:49.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:48 smithi190 ceph-mon[36292]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:49.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:48 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1237920874' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-11T19:07:49.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:48 smithi190 ceph-mon[36292]: osdmap e84: 12 total, 12 up, 12 in 2024-03-11T19:07:49.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:48 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/4183156035' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-11T19:07:49.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:48 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:49.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:48 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:07:49.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:48 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:49.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:48 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:07:49.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:48 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:49.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:48 smithi170 ceph-mon[35690]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:49.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:48 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1237920874' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-11T19:07:49.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:48 smithi170 ceph-mon[35690]: osdmap e84: 12 total, 12 up, 12 in 2024-03-11T19:07:49.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:48 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/4183156035' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-11T19:07:49.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:48 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:49.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:48 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:07:49.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:48 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:49.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:48 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:07:49.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:48 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:49.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.121+0000 7f9740508640 1 -- 172.21.15.29:0/3210384188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f973802ceb0 con 0x7f9738102d60 2024-03-11T19:07:49.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:48 smithi029 ceph-mon[32439]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:49.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:48 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1237920874' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-11T19:07:49.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:48 smithi029 ceph-mon[32439]: osdmap e84: 12 total, 12 up, 12 in 2024-03-11T19:07:49.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:48 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/4183156035' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-11T19:07:49.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:48 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:49.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:48 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:07:49.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:48 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:49.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:48 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:07:49.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:48 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:49.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.732+0000 7f972f7fe640 1 -- 172.21.15.29:0/3210384188 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f97300bc800 con 0x7f9738102d60 2024-03-11T19:07:49.740 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 2 and data pool 3 2024-03-11T19:07:49.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.738+0000 7f9740508640 1 -- 172.21.15.29:0/3210384188 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97240a1a30 msgr2=0x7f97240a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:49.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.738+0000 7f9740508640 1 --2- 172.21.15.29:0/3210384188 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97240a1a30 0x7f97240a3ef0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9738102200 tx=0x7f9728005f90 comp rx=0 tx=0).stop 2024-03-11T19:07:49.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.738+0000 7f9740508640 1 -- 172.21.15.29:0/3210384188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9738102d60 msgr2=0x7f973819e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:49.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.738+0000 7f9740508640 1 --2- 172.21.15.29:0/3210384188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9738102d60 0x7f973819e6c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9730037040 tx=0x7f9730032980 comp rx=0 tx=0).stop 2024-03-11T19:07:49.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.741+0000 7f9740508640 1 -- 172.21.15.29:0/3210384188 shutdown_connections 2024-03-11T19:07:49.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.741+0000 7f9740508640 1 --2- 172.21.15.29:0/3210384188 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97240a1a30 0x7f97240a3ef0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:49.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.741+0000 7f9740508640 1 --2- 172.21.15.29:0/3210384188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9738102d60 0x7f973819e6c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:49.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.741+0000 7f9740508640 1 --2- 172.21.15.29:0/3210384188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97381023a0 0x7f9738197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.741+0000 7f9740508640 1 --2- 172.21.15.29:0/3210384188 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97381011b0 0x7f9738197110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.741+0000 7f9740508640 1 -- 172.21.15.29:0/3210384188 >> 172.21.15.29:0/3210384188 conn(0x7f97380fc720 msgr2=0x7f97381056f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:49.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.742+0000 7f9740508640 1 -- 172.21.15.29:0/3210384188 shutdown_connections 2024-03-11T19:07:49.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.744+0000 7f9740508640 1 -- 172.21.15.29:0/3210384188 wait complete. 2024-03-11T19:07:49.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:07:49.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.887+0000 7f2a7510c640 1 -- 172.21.15.29:0/2418077164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a70101c20 msgr2=0x7f2a70104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:49.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.887+0000 7f2a7510c640 1 --2- 172.21.15.29:0/2418077164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a70101c20 0x7f2a70104010 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2a68009900 tx=0x7f2a6802bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:49.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.887+0000 7f2a7510c640 1 -- 172.21.15.29:0/2418077164 shutdown_connections 2024-03-11T19:07:49.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.887+0000 7f2a7510c640 1 --2- 172.21.15.29:0/2418077164 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a70106e80 0x7f2a701092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:49.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.887+0000 7f2a7510c640 1 --2- 172.21.15.29:0/2418077164 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a70104550 0x7f2a70106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:49.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.887+0000 7f2a7510c640 1 --2- 172.21.15.29:0/2418077164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a70101c20 0x7f2a70104010 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:49.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.887+0000 7f2a7510c640 1 -- 172.21.15.29:0/2418077164 >> 172.21.15.29:0/2418077164 conn(0x7f2a700fb610 msgr2=0x7f2a700fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:49.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.887+0000 7f2a7510c640 1 -- 172.21.15.29:0/2418077164 shutdown_connections 2024-03-11T19:07:49.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.887+0000 7f2a7510c640 1 -- 172.21.15.29:0/2418077164 wait complete. 2024-03-11T19:07:49.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.888+0000 7f2a7510c640 1 Processor -- start 2024-03-11T19:07:49.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.888+0000 7f2a7510c640 1 -- start start 2024-03-11T19:07:49.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.888+0000 7f2a7510c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a70101c20 0x7f2a701993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:49.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.888+0000 7f2a7510c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a70104550 0x7f2a701998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:49.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.888+0000 7f2a7510c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a70106e80 0x7f2a701a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:49.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.888+0000 7f2a7510c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a701a0ea0 con 0x7f2a70104550 2024-03-11T19:07:49.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.888+0000 7f2a7510c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a701a1010 con 0x7f2a70101c20 2024-03-11T19:07:49.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.888+0000 7f2a7510c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a701a11d0 con 0x7f2a70106e80 2024-03-11T19:07:49.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.889+0000 7f2a6e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a70104550 0x7f2a701998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:49.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.889+0000 7f2a6e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a70104550 0x7f2a701998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48124/0 (socket says 172.21.15.29:48124) 2024-03-11T19:07:49.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.889+0000 7f2a6e59c640 1 -- 172.21.15.29:0/2855844531 learned_addr learned my addr 172.21.15.29:0/2855844531 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:49.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.889+0000 7f2a6ed9d640 1 --2- 172.21.15.29:0/2855844531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a70101c20 0x7f2a701993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:49.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.889+0000 7f2a6f59e640 1 --2- 172.21.15.29:0/2855844531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a70106e80 0x7f2a701a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:49.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.889+0000 7f2a6e59c640 1 -- 172.21.15.29:0/2855844531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a70106e80 msgr2=0x7f2a701a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:49.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.889+0000 7f2a6e59c640 1 --2- 172.21.15.29:0/2855844531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a70106e80 0x7f2a701a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:49.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.889+0000 7f2a6e59c640 1 -- 172.21.15.29:0/2855844531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a70101c20 msgr2=0x7f2a701993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:49.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.889+0000 7f2a6e59c640 1 --2- 172.21.15.29:0/2855844531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a70101c20 0x7f2a701993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:49.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.889+0000 7f2a6e59c640 1 -- 172.21.15.29:0/2855844531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a68009560 con 0x7f2a70104550 2024-03-11T19:07:49.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.889+0000 7f2a6f59e640 1 --2- 172.21.15.29:0/2855844531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a70106e80 0x7f2a701a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:07:49.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.889+0000 7f2a6e59c640 1 --2- 172.21.15.29:0/2855844531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a70104550 0x7f2a701998f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2a58005fd0 tx=0x7f2a5800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:49.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.889+0000 7f2a67fff640 1 -- 172.21.15.29:0/2855844531 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a58009c30 con 0x7f2a70104550 2024-03-11T19:07:49.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.889+0000 7f2a7510c640 1 -- 172.21.15.29:0/2855844531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a701a14b0 con 0x7f2a70104550 2024-03-11T19:07:49.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.889+0000 7f2a67fff640 1 -- 172.21.15.29:0/2855844531 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2a580044d0 con 0x7f2a70104550 2024-03-11T19:07:49.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.889+0000 7f2a67fff640 1 -- 172.21.15.29:0/2855844531 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a58003e50 con 0x7f2a70104550 2024-03-11T19:07:49.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.889+0000 7f2a7510c640 1 -- 172.21.15.29:0/2855844531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a701a19c0 con 0x7f2a70104550 2024-03-11T19:07:49.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.890+0000 7f2a7510c640 1 -- 172.21.15.29:0/2855844531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a701089d0 con 0x7f2a70104550 2024-03-11T19:07:49.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.891+0000 7f2a67fff640 1 -- 172.21.15.29:0/2855844531 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2a58004000 con 0x7f2a70104550 2024-03-11T19:07:49.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.892+0000 7f2a67fff640 1 --2- 172.21.15.29:0/2855844531 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2a5c0a1a30 0x7f2a5c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:49.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.892+0000 7f2a6ed9d640 1 --2- 172.21.15.29:0/2855844531 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2a5c0a1a30 0x7f2a5c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:49.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.892+0000 7f2a67fff640 1 -- 172.21.15.29:0/2855844531 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f2a580ccef0 con 0x7f2a70104550 2024-03-11T19:07:49.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.893+0000 7f2a6ed9d640 1 --2- 172.21.15.29:0/2855844531 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2a5c0a1a30 0x7f2a5c0a3ef0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2a68009230 tx=0x7f2a68005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:49.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:49.897+0000 7f2a67fff640 1 -- 172.21.15.29:0/2855844531 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2a58096250 con 0x7f2a70104550 2024-03-11T19:07:50.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:49 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/4183156035' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-11T19:07:50.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:49 smithi190 ceph-mon[36292]: osdmap e85: 12 total, 12 up, 12 in 2024-03-11T19:07:50.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:49 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3210384188' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-11T19:07:50.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:49 smithi190 ceph-mon[36292]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-11T19:07:50.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.087+0000 7f2a7510c640 1 -- 172.21.15.29:0/2855844531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a7010cc60 con 0x7f2a70104550 2024-03-11T19:07:50.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.088+0000 7f2a67fff640 1 -- 172.21.15.29:0/2855844531 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22232 (secure 0 0 0) 0x7f2a58099910 con 0x7f2a70104550 2024-03-11T19:07:50.091 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:50.092 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":86,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:07:49.123248+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:07:47.563184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:07:48.282461+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:07:50.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.091+0000 7f2a7510c640 1 -- 172.21.15.29:0/2855844531 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2a5c0a1a30 msgr2=0x7f2a5c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:50.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.091+0000 7f2a7510c640 1 --2- 172.21.15.29:0/2855844531 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2a5c0a1a30 0x7f2a5c0a3ef0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2a68009230 tx=0x7f2a68005fd0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.091+0000 7f2a7510c640 1 -- 172.21.15.29:0/2855844531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a70104550 msgr2=0x7f2a701998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:50.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.091+0000 7f2a7510c640 1 --2- 172.21.15.29:0/2855844531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a70104550 0x7f2a701998f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2a58005fd0 tx=0x7f2a5800dde0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.091+0000 7f2a7510c640 1 -- 172.21.15.29:0/2855844531 shutdown_connections 2024-03-11T19:07:50.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.091+0000 7f2a7510c640 1 --2- 172.21.15.29:0/2855844531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a70106e80 0x7f2a701a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.091+0000 7f2a7510c640 1 --2- 172.21.15.29:0/2855844531 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2a5c0a1a30 0x7f2a5c0a3ef0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.091+0000 7f2a7510c640 1 --2- 172.21.15.29:0/2855844531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a70104550 0x7f2a701998f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.091+0000 7f2a7510c640 1 --2- 172.21.15.29:0/2855844531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a70101c20 0x7f2a701993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.091+0000 7f2a7510c640 1 -- 172.21.15.29:0/2855844531 >> 172.21.15.29:0/2855844531 conn(0x7f2a700fb610 msgr2=0x7f2a701014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:50.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.092+0000 7f2a7510c640 1 -- 172.21.15.29:0/2855844531 shutdown_connections 2024-03-11T19:07:50.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.092+0000 7f2a7510c640 1 -- 172.21.15.29:0/2855844531 wait complete. 2024-03-11T19:07:50.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:49 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/4183156035' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-11T19:07:50.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:49 smithi170 ceph-mon[35690]: osdmap e85: 12 total, 12 up, 12 in 2024-03-11T19:07:50.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:49 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3210384188' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-11T19:07:50.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:49 smithi170 ceph-mon[35690]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-11T19:07:50.106 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:49 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/4183156035' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-11T19:07:50.106 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:49 smithi029 ceph-mon[32439]: osdmap e85: 12 total, 12 up, 12 in 2024-03-11T19:07:50.106 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:49 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3210384188' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-11T19:07:50.106 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:49 smithi029 ceph-mon[32439]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-11T19:07:50.108 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:07:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.211+0000 7fc9f3fe2640 1 -- 172.21.15.29:0/875485424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9ec1012b0 msgr2=0x7fc9ec101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.211+0000 7fc9f3fe2640 1 --2- 172.21.15.29:0/875485424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9ec1012b0 0x7fc9ec101730 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc009900 tx=0x7fc9dc02bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.212+0000 7fc9f3fe2640 1 -- 172.21.15.29:0/875485424 shutdown_connections 2024-03-11T19:07:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.212+0000 7fc9f3fe2640 1 --2- 172.21.15.29:0/875485424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9ec101c70 0x7fc9ec108500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.212+0000 7fc9f3fe2640 1 --2- 172.21.15.29:0/875485424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9ec1012b0 0x7fc9ec101730 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.212+0000 7fc9f3fe2640 1 --2- 172.21.15.29:0/875485424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9ec1000c0 0x7fc9ec1004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.212+0000 7fc9f3fe2640 1 -- 172.21.15.29:0/875485424 >> 172.21.15.29:0/875485424 conn(0x7fc9ec0fb630 msgr2=0x7fc9ec0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.212+0000 7fc9f3fe2640 1 -- 172.21.15.29:0/875485424 shutdown_connections 2024-03-11T19:07:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.212+0000 7fc9f3fe2640 1 -- 172.21.15.29:0/875485424 wait complete. 2024-03-11T19:07:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.212+0000 7fc9f3fe2640 1 Processor -- start 2024-03-11T19:07:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.212+0000 7fc9f3fe2640 1 -- start start 2024-03-11T19:07:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.212+0000 7fc9f3fe2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9ec1000c0 0x7fc9ec1960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.212+0000 7fc9f3fe2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9ec1012b0 0x7fc9ec1965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.212+0000 7fc9f3fe2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9ec101c70 0x7fc9ec19d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.212+0000 7fc9f3fe2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec19db90 con 0x7fc9ec1000c0 2024-03-11T19:07:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.212+0000 7fc9f3fe2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec19dd00 con 0x7fc9ec101c70 2024-03-11T19:07:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.212+0000 7fc9f3fe2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec19dec0 con 0x7fc9ec1012b0 2024-03-11T19:07:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.212+0000 7fc9f1d7e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9ec1000c0 0x7fc9ec1960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.212+0000 7fc9f1d7e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9ec1000c0 0x7fc9ec1960a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48138/0 (socket says 172.21.15.29:48138) 2024-03-11T19:07:50.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.212+0000 7fc9f1d7e640 1 -- 172.21.15.29:0/3316778716 learned_addr learned my addr 172.21.15.29:0/3316778716 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:50.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.212+0000 7fc9f157d640 1 --2- 172.21.15.29:0/3316778716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9ec1012b0 0x7fc9ec1965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:50.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.213+0000 7fc9f257f640 1 --2- 172.21.15.29:0/3316778716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9ec101c70 0x7fc9ec19d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:50.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.213+0000 7fc9f1d7e640 1 -- 172.21.15.29:0/3316778716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9ec1012b0 msgr2=0x7fc9ec1965e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:50.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.213+0000 7fc9f1d7e640 1 --2- 172.21.15.29:0/3316778716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9ec1012b0 0x7fc9ec1965e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.213+0000 7fc9f1d7e640 1 -- 172.21.15.29:0/3316778716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9ec101c70 msgr2=0x7fc9ec19d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:50.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.213+0000 7fc9f1d7e640 1 --2- 172.21.15.29:0/3316778716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9ec101c70 0x7fc9ec19d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.213+0000 7fc9f1d7e640 1 -- 172.21.15.29:0/3316778716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9dc009560 con 0x7fc9ec1000c0 2024-03-11T19:07:50.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.213+0000 7fc9f257f640 1 --2- 172.21.15.29:0/3316778716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9ec101c70 0x7fc9ec19d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:07:50.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.213+0000 7fc9f1d7e640 1 --2- 172.21.15.29:0/3316778716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9ec1000c0 0x7fc9ec1960a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc9e800e9c0 tx=0x7fc9e800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:50.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.213+0000 7fc9e2ffd640 1 -- 172.21.15.29:0/3316778716 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e80096e0 con 0x7fc9ec1000c0 2024-03-11T19:07:50.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.213+0000 7fc9f3fe2640 1 -- 172.21.15.29:0/3316778716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9ec19e1a0 con 0x7fc9ec1000c0 2024-03-11T19:07:50.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.213+0000 7fc9e2ffd640 1 -- 172.21.15.29:0/3316778716 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc9e80044e0 con 0x7fc9ec1000c0 2024-03-11T19:07:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.213+0000 7fc9e2ffd640 1 -- 172.21.15.29:0/3316778716 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8010430 con 0x7fc9ec1000c0 2024-03-11T19:07:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.213+0000 7fc9f3fe2640 1 -- 172.21.15.29:0/3316778716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9ec19e630 con 0x7fc9ec1000c0 2024-03-11T19:07:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.214+0000 7fc9f3fe2640 1 -- 172.21.15.29:0/3316778716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9ec107500 con 0x7fc9ec1000c0 2024-03-11T19:07:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.215+0000 7fc9e2ffd640 1 -- 172.21.15.29:0/3316778716 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc9e8003f80 con 0x7fc9ec1000c0 2024-03-11T19:07:50.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.215+0000 7fc9e2ffd640 1 --2- 172.21.15.29:0/3316778716 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc9d80a1a30 0x7fc9d80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:50.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.216+0000 7fc9f157d640 1 --2- 172.21.15.29:0/3316778716 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc9d80a1a30 0x7fc9d80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:50.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.216+0000 7fc9e2ffd640 1 -- 172.21.15.29:0/3316778716 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc9e8014070 con 0x7fc9ec1000c0 2024-03-11T19:07:50.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.216+0000 7fc9f157d640 1 --2- 172.21.15.29:0/3316778716 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc9d80a1a30 0x7fc9d80a3ef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc009230 tx=0x7fc9dc005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:50.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.220+0000 7fc9e2ffd640 1 -- 172.21.15.29:0/3316778716 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc9e8096a40 con 0x7fc9ec1000c0 2024-03-11T19:07:50.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.405+0000 7fc9f3fe2640 1 -- 172.21.15.29:0/3316778716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9ec10b980 con 0x7fc9ec1000c0 2024-03-11T19:07:50.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.407+0000 7fc9e2ffd640 1 -- 172.21.15.29:0/3316778716 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22232 (secure 0 0 0) 0x7fc9e809a100 con 0x7fc9ec1000c0 2024-03-11T19:07:50.410 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:50.410 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":86,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:07:49.123248+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:07:47.563184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:07:48.282461+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:07:50.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.409+0000 7fc9f3fe2640 1 -- 172.21.15.29:0/3316778716 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc9d80a1a30 msgr2=0x7fc9d80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:50.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.409+0000 7fc9f3fe2640 1 --2- 172.21.15.29:0/3316778716 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc9d80a1a30 0x7fc9d80a3ef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc009230 tx=0x7fc9dc005fd0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.409+0000 7fc9f3fe2640 1 -- 172.21.15.29:0/3316778716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9ec1000c0 msgr2=0x7fc9ec1960a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:50.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.409+0000 7fc9f3fe2640 1 --2- 172.21.15.29:0/3316778716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9ec1000c0 0x7fc9ec1960a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc9e800e9c0 tx=0x7fc9e800ee80 comp rx=0 tx=0).stop 2024-03-11T19:07:50.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.409+0000 7fc9f3fe2640 1 -- 172.21.15.29:0/3316778716 shutdown_connections 2024-03-11T19:07:50.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.409+0000 7fc9f3fe2640 1 --2- 172.21.15.29:0/3316778716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9ec101c70 0x7fc9ec19d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.409+0000 7fc9f3fe2640 1 --2- 172.21.15.29:0/3316778716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9ec1012b0 0x7fc9ec1965e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.409+0000 7fc9f3fe2640 1 --2- 172.21.15.29:0/3316778716 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc9d80a1a30 0x7fc9d80a3ef0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.409+0000 7fc9f3fe2640 1 --2- 172.21.15.29:0/3316778716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9ec1000c0 0x7fc9ec1960a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.409+0000 7fc9f3fe2640 1 -- 172.21.15.29:0/3316778716 >> 172.21.15.29:0/3316778716 conn(0x7fc9ec0fb630 msgr2=0x7fc9ec1046d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:50.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.409+0000 7fc9f3fe2640 1 -- 172.21.15.29:0/3316778716 shutdown_connections 2024-03-11T19:07:50.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.409+0000 7fc9f3fe2640 1 -- 172.21.15.29:0/3316778716 wait complete. 2024-03-11T19:07:50.423 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-11T19:07:50.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.525+0000 7efd1b240640 1 -- 172.21.15.29:0/854968688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd14101290 msgr2=0x7efd14101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:50.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.525+0000 7efd1b240640 1 --2- 172.21.15.29:0/854968688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd14101290 0x7efd14101710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7efd04009ca0 tx=0x7efd0402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.525+0000 7efd1b240640 1 -- 172.21.15.29:0/854968688 shutdown_connections 2024-03-11T19:07:50.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.525+0000 7efd1b240640 1 --2- 172.21.15.29:0/854968688 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd14101c50 0x7efd141084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.525+0000 7efd1b240640 1 --2- 172.21.15.29:0/854968688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd14101290 0x7efd14101710 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.525+0000 7efd1b240640 1 --2- 172.21.15.29:0/854968688 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd141000a0 0x7efd141004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.525+0000 7efd1b240640 1 -- 172.21.15.29:0/854968688 >> 172.21.15.29:0/854968688 conn(0x7efd140fb630 msgr2=0x7efd140fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:50.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.525+0000 7efd1b240640 1 -- 172.21.15.29:0/854968688 shutdown_connections 2024-03-11T19:07:50.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.525+0000 7efd1b240640 1 -- 172.21.15.29:0/854968688 wait complete. 2024-03-11T19:07:50.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.526+0000 7efd1b240640 1 Processor -- start 2024-03-11T19:07:50.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.526+0000 7efd1b240640 1 -- start start 2024-03-11T19:07:50.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.526+0000 7efd1b240640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd141000a0 0x7efd141993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:50.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.526+0000 7efd1b240640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd14101290 0x7efd141998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:50.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.526+0000 7efd1b240640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd14101c50 0x7efd141a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:50.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.526+0000 7efd1b240640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd141a0e50 con 0x7efd141000a0 2024-03-11T19:07:50.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.526+0000 7efd1b240640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd141a0fc0 con 0x7efd14101c50 2024-03-11T19:07:50.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.526+0000 7efd1b240640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd141a1180 con 0x7efd14101290 2024-03-11T19:07:50.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.526+0000 7efd18fdc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd141000a0 0x7efd141993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:50.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.526+0000 7efd18fdc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd141000a0 0x7efd141993b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48158/0 (socket says 172.21.15.29:48158) 2024-03-11T19:07:50.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.526+0000 7efd18fdc640 1 -- 172.21.15.29:0/397022972 learned_addr learned my addr 172.21.15.29:0/397022972 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:50.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.526+0000 7efd13fff640 1 --2- 172.21.15.29:0/397022972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd14101290 0x7efd141998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:50.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.526+0000 7efd197dd640 1 --2- 172.21.15.29:0/397022972 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd14101c50 0x7efd141a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:50.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.526+0000 7efd18fdc640 1 -- 172.21.15.29:0/397022972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd14101290 msgr2=0x7efd141998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:50.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.526+0000 7efd18fdc640 1 --2- 172.21.15.29:0/397022972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd14101290 0x7efd141998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.526+0000 7efd18fdc640 1 -- 172.21.15.29:0/397022972 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd14101c50 msgr2=0x7efd141a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.527+0000 7efd18fdc640 1 --2- 172.21.15.29:0/397022972 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd14101c50 0x7efd141a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.527+0000 7efd18fdc640 1 -- 172.21.15.29:0/397022972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd04009900 con 0x7efd141000a0 2024-03-11T19:07:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.527+0000 7efd18fdc640 1 --2- 172.21.15.29:0/397022972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd141000a0 0x7efd141993b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efd0c00d770 tx=0x7efd0c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.527+0000 7efd11ffb640 1 -- 172.21.15.29:0/397022972 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0c009c30 con 0x7efd141000a0 2024-03-11T19:07:50.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.527+0000 7efd11ffb640 1 -- 172.21.15.29:0/397022972 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efd0c0044d0 con 0x7efd141000a0 2024-03-11T19:07:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.527+0000 7efd1b240640 1 -- 172.21.15.29:0/397022972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd141a1460 con 0x7efd141000a0 2024-03-11T19:07:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.527+0000 7efd11ffb640 1 -- 172.21.15.29:0/397022972 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0c003de0 con 0x7efd141000a0 2024-03-11T19:07:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.527+0000 7efd1b240640 1 -- 172.21.15.29:0/397022972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd141a1940 con 0x7efd141000a0 2024-03-11T19:07:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.528+0000 7efd1b240640 1 -- 172.21.15.29:0/397022972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd1410b960 con 0x7efd141000a0 2024-03-11T19:07:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.529+0000 7efd11ffb640 1 -- 172.21.15.29:0/397022972 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7efd0c003f40 con 0x7efd141000a0 2024-03-11T19:07:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.529+0000 7efd11ffb640 1 --2- 172.21.15.29:0/397022972 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7efd000a1ab0 0x7efd000a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.530+0000 7efd13fff640 1 --2- 172.21.15.29:0/397022972 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7efd000a1ab0 0x7efd000a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:50.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.530+0000 7efd11ffb640 1 -- 172.21.15.29:0/397022972 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7efd0c0ccc10 con 0x7efd141000a0 2024-03-11T19:07:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.530+0000 7efd13fff640 1 --2- 172.21.15.29:0/397022972 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7efd000a1ab0 0x7efd000a3f70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efd04006010 tx=0x7efd040325d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.534+0000 7efd11ffb640 1 -- 172.21.15.29:0/397022972 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efd0c095f80 con 0x7efd141000a0 2024-03-11T19:07:50.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:50.758+0000 7efd1b240640 1 -- 172.21.15.29:0/397022972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7efd1410bb10 con 0x7efd141000a0 2024-03-11T19:07:51.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:50 smithi190 ceph-mon[36292]: pgmap v288: 97 pgs: 25 creating+peering, 7 active+clean, 65 unknown; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:51.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:50 smithi190 ceph-mon[36292]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T19:07:51.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:50 smithi190 ceph-mon[36292]: osdmap e86: 12 total, 12 up, 12 in 2024-03-11T19:07:51.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:50 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3210384188' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-11T19:07:51.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:50 smithi190 ceph-mon[36292]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-11T19:07:51.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:50 smithi190 ceph-mon[36292]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-11T19:07:51.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:50 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2855844531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:07:51.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:51.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:07:51.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:50 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:50 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3316778716' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:07:51.096 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:07:50 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-b[66380]: 2024-03-11T19:07:50.800+0000 7f358a6a7700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-11T19:07:51.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:50 smithi170 ceph-mon[35690]: pgmap v288: 97 pgs: 25 creating+peering, 7 active+clean, 65 unknown; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:51.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:50 smithi170 ceph-mon[35690]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T19:07:51.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:50 smithi170 ceph-mon[35690]: osdmap e86: 12 total, 12 up, 12 in 2024-03-11T19:07:51.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:50 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3210384188' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-11T19:07:51.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:50 smithi170 ceph-mon[35690]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-11T19:07:51.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:50 smithi170 ceph-mon[35690]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-11T19:07:51.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:50 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2855844531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:07:51.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:51.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:07:51.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:50 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:50 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3316778716' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:07:51.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:50 smithi029 ceph-mon[32439]: pgmap v288: 97 pgs: 25 creating+peering, 7 active+clean, 65 unknown; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:07:51.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:50 smithi029 ceph-mon[32439]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T19:07:51.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:50 smithi029 ceph-mon[32439]: osdmap e86: 12 total, 12 up, 12 in 2024-03-11T19:07:51.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:50 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3210384188' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-11T19:07:51.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:50 smithi029 ceph-mon[32439]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-11T19:07:51.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:50 smithi029 ceph-mon[32439]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-11T19:07:51.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:50 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2855844531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:07:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:07:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:07:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:50 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:07:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:50 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3316778716' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:07:51.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.745+0000 7efd11ffb640 1 -- 172.21.15.29:0/397022972 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7efd0c0200b0 con 0x7efd141000a0 2024-03-11T19:07:51.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.748+0000 7efd1b240640 1 -- 172.21.15.29:0/397022972 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7efd000a1ab0 msgr2=0x7efd000a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:51.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.748+0000 7efd1b240640 1 --2- 172.21.15.29:0/397022972 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7efd000a1ab0 0x7efd000a3f70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efd04006010 tx=0x7efd040325d0 comp rx=0 tx=0).stop 2024-03-11T19:07:51.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.748+0000 7efd1b240640 1 -- 172.21.15.29:0/397022972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd141000a0 msgr2=0x7efd141993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:51.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.748+0000 7efd1b240640 1 --2- 172.21.15.29:0/397022972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd141000a0 0x7efd141993b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efd0c00d770 tx=0x7efd0c00dc30 comp rx=0 tx=0).stop 2024-03-11T19:07:51.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.748+0000 7efd1b240640 1 -- 172.21.15.29:0/397022972 shutdown_connections 2024-03-11T19:07:51.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.748+0000 7efd1b240640 1 --2- 172.21.15.29:0/397022972 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd14101c50 0x7efd141a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:51.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.748+0000 7efd1b240640 1 --2- 172.21.15.29:0/397022972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd14101290 0x7efd141998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:51.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.748+0000 7efd1b240640 1 --2- 172.21.15.29:0/397022972 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7efd000a1ab0 0x7efd000a3f70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:51.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.748+0000 7efd1b240640 1 --2- 172.21.15.29:0/397022972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd141000a0 0x7efd141993b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:51.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.748+0000 7efd1b240640 1 -- 172.21.15.29:0/397022972 >> 172.21.15.29:0/397022972 conn(0x7efd140fb630 msgr2=0x7efd14069780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:51.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.748+0000 7efd1b240640 1 -- 172.21.15.29:0/397022972 shutdown_connections 2024-03-11T19:07:51.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.748+0000 7efd1b240640 1 -- 172.21.15.29:0/397022972 wait complete. 2024-03-11T19:07:51.765 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-11T19:07:51.765 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-11T19:07:51.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.874+0000 7f466589d640 1 -- 172.21.15.29:0/4077218261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4660101190 msgr2=0x7f46601015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:51.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.874+0000 7f466589d640 1 --2- 172.21.15.29:0/4077218261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4660101190 0x7f46601015b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4650009900 tx=0x7f465002bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:51.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.874+0000 7f466589d640 1 -- 172.21.15.29:0/4077218261 shutdown_connections 2024-03-11T19:07:51.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.874+0000 7f466589d640 1 --2- 172.21.15.29:0/4077218261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4660102d40 0x7f46601095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:51.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.874+0000 7f466589d640 1 --2- 172.21.15.29:0/4077218261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4660102380 0x7f4660102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:51.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.874+0000 7f466589d640 1 --2- 172.21.15.29:0/4077218261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4660101190 0x7f46601015b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:51.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.874+0000 7f466589d640 1 -- 172.21.15.29:0/4077218261 >> 172.21.15.29:0/4077218261 conn(0x7f46600fc720 msgr2=0x7f46600feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:51.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.874+0000 7f466589d640 1 -- 172.21.15.29:0/4077218261 shutdown_connections 2024-03-11T19:07:51.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.874+0000 7f466589d640 1 -- 172.21.15.29:0/4077218261 wait complete. 2024-03-11T19:07:51.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.875+0000 7f466589d640 1 Processor -- start 2024-03-11T19:07:51.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.875+0000 7f466589d640 1 -- start start 2024-03-11T19:07:51.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.875+0000 7f466589d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4660101190 0x7f4660197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:51.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.875+0000 7f466589d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4660102380 0x7f46601976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:51.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.875+0000 7f466589d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4660102d40 0x7f466019e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:51.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.875+0000 7f466589d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466019ec60 con 0x7f4660101190 2024-03-11T19:07:51.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.875+0000 7f466589d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466019edd0 con 0x7f4660102d40 2024-03-11T19:07:51.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.875+0000 7f466589d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466019ef90 con 0x7f4660102380 2024-03-11T19:07:51.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.875+0000 7f465effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4660101190 0x7f4660197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:51.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.875+0000 7f465effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4660101190 0x7f4660197170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48170/0 (socket says 172.21.15.29:48170) 2024-03-11T19:07:51.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.875+0000 7f465effd640 1 -- 172.21.15.29:0/1444597061 learned_addr learned my addr 172.21.15.29:0/1444597061 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:51.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.875+0000 7f465f7fe640 1 --2- 172.21.15.29:0/1444597061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4660102d40 0x7f466019e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:51.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.875+0000 7f465effd640 1 -- 172.21.15.29:0/1444597061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4660102380 msgr2=0x7f46601976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:51.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.875+0000 7f465effd640 1 --2- 172.21.15.29:0/1444597061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4660102380 0x7f46601976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:51.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.875+0000 7f465effd640 1 -- 172.21.15.29:0/1444597061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4660102d40 msgr2=0x7f466019e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:51.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.875+0000 7f465effd640 1 --2- 172.21.15.29:0/1444597061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4660102d40 0x7f466019e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:51.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.875+0000 7f465effd640 1 -- 172.21.15.29:0/1444597061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4650009560 con 0x7f4660101190 2024-03-11T19:07:51.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.875+0000 7f465effd640 1 --2- 172.21.15.29:0/1444597061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4660101190 0x7f4660197170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4650000c00 tx=0x7f46500042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:51.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.875+0000 7f466489b640 1 -- 172.21.15.29:0/1444597061 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46500044e0 con 0x7f4660101190 2024-03-11T19:07:51.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.876+0000 7f466489b640 1 -- 172.21.15.29:0/1444597061 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f465002de90 con 0x7f4660101190 2024-03-11T19:07:51.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.876+0000 7f466589d640 1 -- 172.21.15.29:0/1444597061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466019f210 con 0x7f4660101190 2024-03-11T19:07:51.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.876+0000 7f466489b640 1 -- 172.21.15.29:0/1444597061 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4650005410 con 0x7f4660101190 2024-03-11T19:07:51.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.876+0000 7f466589d640 1 -- 172.21.15.29:0/1444597061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466019f770 con 0x7f4660101190 2024-03-11T19:07:51.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.877+0000 7f466589d640 1 -- 172.21.15.29:0/1444597061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4660108620 con 0x7f4660101190 2024-03-11T19:07:51.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.878+0000 7f466489b640 1 -- 172.21.15.29:0/1444597061 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4650005570 con 0x7f4660101190 2024-03-11T19:07:51.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.878+0000 7f466489b640 1 --2- 172.21.15.29:0/1444597061 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f464c0a1ab0 0x7f464c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:51.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.878+0000 7f4657fff640 1 --2- 172.21.15.29:0/1444597061 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f464c0a1ab0 0x7f464c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:51.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.878+0000 7f466489b640 1 -- 172.21.15.29:0/1444597061 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f46500efc60 con 0x7f4660101190 2024-03-11T19:07:51.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.879+0000 7f4657fff640 1 --2- 172.21.15.29:0/1444597061 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f464c0a1ab0 0x7f464c0a3f70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4648005fd0 tx=0x7f4648005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:51.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:51.882+0000 7f466489b640 1 -- 172.21.15.29:0/1444597061 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f46500b8fc0 con 0x7f4660101190 2024-03-11T19:07:52.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:51 smithi190 ceph-mon[36292]: osdmap e87: 12 total, 12 up, 12 in 2024-03-11T19:07:52.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:51 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] up:active 2024-03-11T19:07:52.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:51 smithi190 ceph-mon[36292]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:07:52.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:51 smithi190 ceph-mon[36292]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:07:52.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:51 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/397022972' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-11T19:07:52.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:51 smithi170 ceph-mon[35690]: osdmap e87: 12 total, 12 up, 12 in 2024-03-11T19:07:52.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:51 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] up:active 2024-03-11T19:07:52.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:51 smithi170 ceph-mon[35690]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:07:52.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:51 smithi170 ceph-mon[35690]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:07:52.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:51 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/397022972' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-11T19:07:52.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.101+0000 7f466589d640 1 -- 172.21.15.29:0/1444597061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f466010cbe0 con 0x7f4660101190 2024-03-11T19:07:52.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:51 smithi029 ceph-mon[32439]: osdmap e87: 12 total, 12 up, 12 in 2024-03-11T19:07:52.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:51 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] up:active 2024-03-11T19:07:52.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:51 smithi029 ceph-mon[32439]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:07:52.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:51 smithi029 ceph-mon[32439]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:07:52.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:51 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/397022972' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-11T19:07:52.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.747+0000 7f466489b640 1 -- 172.21.15.29:0/1444597061 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7f46500430b0 con 0x7f4660101190 2024-03-11T19:07:52.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.751+0000 7f466589d640 1 -- 172.21.15.29:0/1444597061 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f464c0a1ab0 msgr2=0x7f464c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:52.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.751+0000 7f466589d640 1 --2- 172.21.15.29:0/1444597061 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f464c0a1ab0 0x7f464c0a3f70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4648005fd0 tx=0x7f4648005850 comp rx=0 tx=0).stop 2024-03-11T19:07:52.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.751+0000 7f466589d640 1 -- 172.21.15.29:0/1444597061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4660101190 msgr2=0x7f4660197170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:52.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.751+0000 7f466589d640 1 --2- 172.21.15.29:0/1444597061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4660101190 0x7f4660197170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4650000c00 tx=0x7f46500042f0 comp rx=0 tx=0).stop 2024-03-11T19:07:52.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.751+0000 7f466589d640 1 -- 172.21.15.29:0/1444597061 shutdown_connections 2024-03-11T19:07:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.751+0000 7f466589d640 1 --2- 172.21.15.29:0/1444597061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4660102d40 0x7f466019e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.751+0000 7f466589d640 1 --2- 172.21.15.29:0/1444597061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4660102380 0x7f46601976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.751+0000 7f466589d640 1 --2- 172.21.15.29:0/1444597061 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f464c0a1ab0 0x7f464c0a3f70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.751+0000 7f466589d640 1 --2- 172.21.15.29:0/1444597061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4660101190 0x7f4660197170 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.751+0000 7f466589d640 1 -- 172.21.15.29:0/1444597061 >> 172.21.15.29:0/1444597061 conn(0x7f46600fc720 msgr2=0x7f46601057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.751+0000 7f466589d640 1 -- 172.21.15.29:0/1444597061 shutdown_connections 2024-03-11T19:07:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.751+0000 7f466589d640 1 -- 172.21.15.29:0/1444597061 wait complete. 2024-03-11T19:07:52.776 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-11T19:07:52.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.885+0000 7fe156e62640 1 -- 172.21.15.29:0/848370443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15010bb90 msgr2=0x7fe15010bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:52.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.885+0000 7fe156e62640 1 --2- 172.21.15.29:0/848370443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15010bb90 0x7fe15010bfb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe140009900 tx=0x7fe14002bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:52.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.885+0000 7fe156e62640 1 -- 172.21.15.29:0/848370443 shutdown_connections 2024-03-11T19:07:52.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.885+0000 7fe156e62640 1 --2- 172.21.15.29:0/848370443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15010d740 0x7fe150113fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:52.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.885+0000 7fe156e62640 1 --2- 172.21.15.29:0/848370443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe15010cd80 0x7fe15010d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:52.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.885+0000 7fe156e62640 1 --2- 172.21.15.29:0/848370443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15010bb90 0x7fe15010bfb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:52.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.885+0000 7fe156e62640 1 -- 172.21.15.29:0/848370443 >> 172.21.15.29:0/848370443 conn(0x7fe15006bb40 msgr2=0x7fe15006bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.885+0000 7fe156e62640 1 -- 172.21.15.29:0/848370443 shutdown_connections 2024-03-11T19:07:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.885+0000 7fe156e62640 1 -- 172.21.15.29:0/848370443 wait complete. 2024-03-11T19:07:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.885+0000 7fe156e62640 1 Processor -- start 2024-03-11T19:07:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.885+0000 7fe156e62640 1 -- start start 2024-03-11T19:07:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.885+0000 7fe156e62640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15010bb90 0x7fe1501a1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.885+0000 7fe156e62640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe15010cd80 0x7fe1501a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.885+0000 7fe156e62640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15010d740 0x7fe1501a91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.885+0000 7fe156e62640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1501a96e0 con 0x7fe15010d740 2024-03-11T19:07:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.885+0000 7fe156e62640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1501a9850 con 0x7fe15010bb90 2024-03-11T19:07:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.885+0000 7fe156e62640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1501a9a10 con 0x7fe15010cd80 2024-03-11T19:07:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.886+0000 7fe1553ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15010d740 0x7fe1501a91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.886+0000 7fe1553ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15010d740 0x7fe1501a91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48188/0 (socket says 172.21.15.29:48188) 2024-03-11T19:07:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.886+0000 7fe1553ff640 1 -- 172.21.15.29:0/1020585242 learned_addr learned my addr 172.21.15.29:0/1020585242 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.886+0000 7fe154bfe640 1 --2- 172.21.15.29:0/1020585242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15010bb90 0x7fe1501a1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.886+0000 7fe14ffff640 1 --2- 172.21.15.29:0/1020585242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe15010cd80 0x7fe1501a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.886+0000 7fe1553ff640 1 -- 172.21.15.29:0/1020585242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe15010cd80 msgr2=0x7fe1501a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.886+0000 7fe1553ff640 1 --2- 172.21.15.29:0/1020585242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe15010cd80 0x7fe1501a2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.886+0000 7fe1553ff640 1 -- 172.21.15.29:0/1020585242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15010bb90 msgr2=0x7fe1501a1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.886+0000 7fe1553ff640 1 --2- 172.21.15.29:0/1020585242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15010bb90 0x7fe1501a1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.886+0000 7fe1553ff640 1 -- 172.21.15.29:0/1020585242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe140009560 con 0x7fe15010d740 2024-03-11T19:07:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.886+0000 7fe14ffff640 1 --2- 172.21.15.29:0/1020585242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe15010cd80 0x7fe1501a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:07:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.886+0000 7fe1553ff640 1 --2- 172.21.15.29:0/1020585242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15010d740 0x7fe1501a91a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe14800cad0 tx=0x7fe14800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.886+0000 7fe14dffb640 1 -- 172.21.15.29:0/1020585242 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14800ae20 con 0x7fe15010d740 2024-03-11T19:07:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.886+0000 7fe14dffb640 1 -- 172.21.15.29:0/1020585242 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe148005020 con 0x7fe15010d740 2024-03-11T19:07:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.886+0000 7fe156e62640 1 -- 172.21.15.29:0/1020585242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1501a9cf0 con 0x7fe15010d740 2024-03-11T19:07:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.886+0000 7fe14dffb640 1 -- 172.21.15.29:0/1020585242 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14800f690 con 0x7fe15010d740 2024-03-11T19:07:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.886+0000 7fe156e62640 1 -- 172.21.15.29:0/1020585242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1501aa1b0 con 0x7fe15010d740 2024-03-11T19:07:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.887+0000 7fe156e62640 1 -- 172.21.15.29:0/1020585242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe150113020 con 0x7fe15010d740 2024-03-11T19:07:52.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.888+0000 7fe14dffb640 1 -- 172.21.15.29:0/1020585242 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe148005190 con 0x7fe15010d740 2024-03-11T19:07:52.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.889+0000 7fe14dffb640 1 --2- 172.21.15.29:0/1020585242 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe1380a1ab0 0x7fe1380a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:52.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.889+0000 7fe154bfe640 1 --2- 172.21.15.29:0/1020585242 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe1380a1ab0 0x7fe1380a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:52.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.889+0000 7fe14dffb640 1 -- 172.21.15.29:0/1020585242 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe1480cd350 con 0x7fe15010d740 2024-03-11T19:07:52.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.889+0000 7fe154bfe640 1 --2- 172.21.15.29:0/1020585242 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe1380a1ab0 0x7fe1380a3f70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe14002c7d0 tx=0x7fe140005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:52.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:52.893+0000 7fe14dffb640 1 -- 172.21.15.29:0/1020585242 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe148096610 con 0x7fe15010d740 2024-03-11T19:07:53.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:52 smithi190 ceph-mon[36292]: pgmap v291: 97 pgs: 25 creating+peering, 40 active+clean, 32 unknown; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 6 op/s 2024-03-11T19:07:53.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:52 smithi190 ceph-mon[36292]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-11T19:07:53.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:52 smithi190 ceph-mon[36292]: Cluster is now healthy 2024-03-11T19:07:53.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:52 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/397022972' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-11T19:07:53.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:52 smithi190 ceph-mon[36292]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:07:53.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:52 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1444597061' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-11T19:07:53.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:52 smithi170 ceph-mon[35690]: pgmap v291: 97 pgs: 25 creating+peering, 40 active+clean, 32 unknown; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 6 op/s 2024-03-11T19:07:53.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:52 smithi170 ceph-mon[35690]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-11T19:07:53.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:52 smithi170 ceph-mon[35690]: Cluster is now healthy 2024-03-11T19:07:53.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:52 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/397022972' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-11T19:07:53.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:52 smithi170 ceph-mon[35690]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:07:53.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:52 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1444597061' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-11T19:07:53.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.113+0000 7fe156e62640 1 -- 172.21.15.29:0/1020585242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fe15002cea0 con 0x7fe15010d740 2024-03-11T19:07:53.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:52 smithi029 ceph-mon[32439]: pgmap v291: 97 pgs: 25 creating+peering, 40 active+clean, 32 unknown; 577 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 6 op/s 2024-03-11T19:07:53.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:52 smithi029 ceph-mon[32439]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-11T19:07:53.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:52 smithi029 ceph-mon[32439]: Cluster is now healthy 2024-03-11T19:07:53.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:52 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/397022972' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-11T19:07:53.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:52 smithi029 ceph-mon[32439]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:07:53.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:52 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1444597061' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-11T19:07:53.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.756+0000 7fe14dffb640 1 -- 172.21.15.29:0/1020585242 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7fe148099cd0 con 0x7fe15010d740 2024-03-11T19:07:53.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.759+0000 7fe156e62640 1 -- 172.21.15.29:0/1020585242 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe1380a1ab0 msgr2=0x7fe1380a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:53.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.759+0000 7fe156e62640 1 --2- 172.21.15.29:0/1020585242 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe1380a1ab0 0x7fe1380a3f70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe14002c7d0 tx=0x7fe140005fd0 comp rx=0 tx=0).stop 2024-03-11T19:07:53.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.759+0000 7fe156e62640 1 -- 172.21.15.29:0/1020585242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15010d740 msgr2=0x7fe1501a91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:53.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.759+0000 7fe156e62640 1 --2- 172.21.15.29:0/1020585242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15010d740 0x7fe1501a91a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe14800cad0 tx=0x7fe14800cf90 comp rx=0 tx=0).stop 2024-03-11T19:07:53.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.759+0000 7fe156e62640 1 -- 172.21.15.29:0/1020585242 shutdown_connections 2024-03-11T19:07:53.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.759+0000 7fe156e62640 1 --2- 172.21.15.29:0/1020585242 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe1380a1ab0 0x7fe1380a3f70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:53.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.759+0000 7fe156e62640 1 --2- 172.21.15.29:0/1020585242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15010d740 0x7fe1501a91a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:53.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.759+0000 7fe156e62640 1 --2- 172.21.15.29:0/1020585242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe15010cd80 0x7fe1501a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:53.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.759+0000 7fe156e62640 1 --2- 172.21.15.29:0/1020585242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15010bb90 0x7fe1501a1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:53.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.759+0000 7fe156e62640 1 -- 172.21.15.29:0/1020585242 >> 172.21.15.29:0/1020585242 conn(0x7fe15006bb40 msgr2=0x7fe1501101f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:53.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.760+0000 7fe156e62640 1 -- 172.21.15.29:0/1020585242 shutdown_connections 2024-03-11T19:07:53.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.760+0000 7fe156e62640 1 -- 172.21.15.29:0/1020585242 wait complete. 2024-03-11T19:07:53.779 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-11T19:07:53.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.887+0000 7fec4fc39640 1 -- 172.21.15.29:0/2699528770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec48100080 msgr2=0x7fec481004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:53.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.887+0000 7fec4fc39640 1 --2- 172.21.15.29:0/2699528770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec48100080 0x7fec481004a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fec44009900 tx=0x7fec4402bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:53.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.887+0000 7fec4fc39640 1 -- 172.21.15.29:0/2699528770 shutdown_connections 2024-03-11T19:07:53.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.887+0000 7fec4fc39640 1 --2- 172.21.15.29:0/2699528770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec48101c30 0x7fec481084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:53.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.887+0000 7fec4fc39640 1 --2- 172.21.15.29:0/2699528770 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec48101270 0x7fec481016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:53.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.887+0000 7fec4fc39640 1 --2- 172.21.15.29:0/2699528770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec48100080 0x7fec481004a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:53.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.887+0000 7fec4fc39640 1 -- 172.21.15.29:0/2699528770 >> 172.21.15.29:0/2699528770 conn(0x7fec480fb610 msgr2=0x7fec480fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:53.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.887+0000 7fec4fc39640 1 -- 172.21.15.29:0/2699528770 shutdown_connections 2024-03-11T19:07:53.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.887+0000 7fec4fc39640 1 -- 172.21.15.29:0/2699528770 wait complete. 2024-03-11T19:07:53.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4fc39640 1 Processor -- start 2024-03-11T19:07:53.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4fc39640 1 -- start start 2024-03-11T19:07:53.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4fc39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec48100080 0x7fec48103fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:53.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4fc39640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec48101270 0x7fec48102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:53.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4fc39640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec48101c30 0x7fec48102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:53.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4fc39640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec48103280 con 0x7fec48100080 2024-03-11T19:07:53.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4fc39640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec481033f0 con 0x7fec48101270 2024-03-11T19:07:53.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4fc39640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec481a11c0 con 0x7fec48101c30 2024-03-11T19:07:53.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4d9d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec48100080 0x7fec48103fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:53.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4d9d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec48100080 0x7fec48103fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48218/0 (socket says 172.21.15.29:48218) 2024-03-11T19:07:53.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4d9d5640 1 -- 172.21.15.29:0/3691423902 learned_addr learned my addr 172.21.15.29:0/3691423902 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:53.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4d1d4640 1 --2- 172.21.15.29:0/3691423902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec48101270 0x7fec48102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:53.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4e1d6640 1 --2- 172.21.15.29:0/3691423902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec48101c30 0x7fec48102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:53.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4d9d5640 1 -- 172.21.15.29:0/3691423902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec48101c30 msgr2=0x7fec48102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:53.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4d9d5640 1 --2- 172.21.15.29:0/3691423902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec48101c30 0x7fec48102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:53.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4d9d5640 1 -- 172.21.15.29:0/3691423902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec48101270 msgr2=0x7fec48102650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:53.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4d9d5640 1 --2- 172.21.15.29:0/3691423902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec48101270 0x7fec48102650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:53.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4d9d5640 1 -- 172.21.15.29:0/3691423902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec44009560 con 0x7fec48100080 2024-03-11T19:07:53.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4d1d4640 1 --2- 172.21.15.29:0/3691423902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec48101270 0x7fec48102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:07:53.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.888+0000 7fec4d9d5640 1 --2- 172.21.15.29:0/3691423902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec48100080 0x7fec48103fa0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fec4402c7d0 tx=0x7fec44004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:53.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.889+0000 7fec3effd640 1 -- 172.21.15.29:0/3691423902 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec44003800 con 0x7fec48100080 2024-03-11T19:07:53.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.889+0000 7fec3effd640 1 -- 172.21.15.29:0/3691423902 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fec4402de90 con 0x7fec48100080 2024-03-11T19:07:53.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.889+0000 7fec4fc39640 1 -- 172.21.15.29:0/3691423902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec481a13c0 con 0x7fec48100080 2024-03-11T19:07:53.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.889+0000 7fec3effd640 1 -- 172.21.15.29:0/3691423902 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec440055f0 con 0x7fec48100080 2024-03-11T19:07:53.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.889+0000 7fec4fc39640 1 -- 172.21.15.29:0/3691423902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec481a18f0 con 0x7fec48100080 2024-03-11T19:07:53.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.890+0000 7fec4fc39640 1 -- 172.21.15.29:0/3691423902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec4810b900 con 0x7fec48100080 2024-03-11T19:07:53.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.891+0000 7fec3effd640 1 -- 172.21.15.29:0/3691423902 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fec44004dc0 con 0x7fec48100080 2024-03-11T19:07:53.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.891+0000 7fec3effd640 1 --2- 172.21.15.29:0/3691423902 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fec340a1ab0 0x7fec340a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:53.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.891+0000 7fec4d1d4640 1 --2- 172.21.15.29:0/3691423902 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fec340a1ab0 0x7fec340a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:53.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.892+0000 7fec3effd640 1 -- 172.21.15.29:0/3691423902 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fec440ef620 con 0x7fec48100080 2024-03-11T19:07:53.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.892+0000 7fec4d1d4640 1 --2- 172.21.15.29:0/3691423902 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fec340a1ab0 0x7fec340a3f70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fec48103d10 tx=0x7fec3800a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:53.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:53.896+0000 7fec3effd640 1 -- 172.21.15.29:0/3691423902 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fec440b8980 con 0x7fec48100080 2024-03-11T19:07:54.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:53 smithi190 ceph-mon[36292]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-11T19:07:54.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:53 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1444597061' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-11T19:07:54.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:53 smithi190 ceph-mon[36292]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-11T19:07:54.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:53 smithi190 ceph-mon[36292]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:07:54.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:53 smithi190 ceph-mon[36292]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-11T19:07:54.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:53 smithi190 ceph-mon[36292]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-11T19:07:54.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:53 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1020585242' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-11T19:07:54.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:53 smithi170 ceph-mon[35690]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-11T19:07:54.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:53 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1444597061' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-11T19:07:54.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:53 smithi170 ceph-mon[35690]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-11T19:07:54.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:53 smithi170 ceph-mon[35690]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:07:54.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:53 smithi170 ceph-mon[35690]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-11T19:07:54.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:53 smithi170 ceph-mon[35690]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-11T19:07:54.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:53 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1020585242' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-11T19:07:54.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.117+0000 7fec4fc39640 1 -- 172.21.15.29:0/3691423902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fec481a1b10 con 0x7fec48100080 2024-03-11T19:07:54.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:53 smithi029 ceph-mon[32439]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-11T19:07:54.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:53 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1444597061' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-11T19:07:54.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:53 smithi029 ceph-mon[32439]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-11T19:07:54.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:53 smithi029 ceph-mon[32439]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:07:54.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:53 smithi029 ceph-mon[32439]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-11T19:07:54.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:53 smithi029 ceph-mon[32439]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-11T19:07:54.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:53 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1020585242' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-11T19:07:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.765+0000 7fec3effd640 1 -- 172.21.15.29:0/3691423902 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7fec481a1b10 con 0x7fec48100080 2024-03-11T19:07:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.768+0000 7fec4fc39640 1 -- 172.21.15.29:0/3691423902 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fec340a1ab0 msgr2=0x7fec340a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.768+0000 7fec4fc39640 1 --2- 172.21.15.29:0/3691423902 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fec340a1ab0 0x7fec340a3f70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fec48103d10 tx=0x7fec3800a5c0 comp rx=0 tx=0).stop 2024-03-11T19:07:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.768+0000 7fec4fc39640 1 -- 172.21.15.29:0/3691423902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec48100080 msgr2=0x7fec48103fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.768+0000 7fec4fc39640 1 --2- 172.21.15.29:0/3691423902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec48100080 0x7fec48103fa0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fec4402c7d0 tx=0x7fec44004360 comp rx=0 tx=0).stop 2024-03-11T19:07:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.769+0000 7fec4fc39640 1 -- 172.21.15.29:0/3691423902 shutdown_connections 2024-03-11T19:07:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.769+0000 7fec4fc39640 1 --2- 172.21.15.29:0/3691423902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec48101c30 0x7fec48102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.769+0000 7fec4fc39640 1 --2- 172.21.15.29:0/3691423902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec48101270 0x7fec48102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.769+0000 7fec4fc39640 1 --2- 172.21.15.29:0/3691423902 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fec340a1ab0 0x7fec340a3f70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.769+0000 7fec4fc39640 1 --2- 172.21.15.29:0/3691423902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec48100080 0x7fec48103fa0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.769+0000 7fec4fc39640 1 -- 172.21.15.29:0/3691423902 >> 172.21.15.29:0/3691423902 conn(0x7fec480fb610 msgr2=0x7fec480fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:54.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.769+0000 7fec4fc39640 1 -- 172.21.15.29:0/3691423902 shutdown_connections 2024-03-11T19:07:54.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.769+0000 7fec4fc39640 1 -- 172.21.15.29:0/3691423902 wait complete. 2024-03-11T19:07:54.788 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-03-11T19:07:54.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-11T19:07:54.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.897+0000 7f23f3ed0640 1 -- 172.21.15.29:0/3561760227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec106e80 msgr2=0x7f23ec1092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.897+0000 7f23f3ed0640 1 --2- 172.21.15.29:0/3561760227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec106e80 0x7f23ec1092d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f23e4009ca0 tx=0x7f23e402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:07:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.897+0000 7f23f3ed0640 1 -- 172.21.15.29:0/3561760227 shutdown_connections 2024-03-11T19:07:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.897+0000 7f23f3ed0640 1 --2- 172.21.15.29:0/3561760227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec106e80 0x7f23ec1092d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.897+0000 7f23f3ed0640 1 --2- 172.21.15.29:0/3561760227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23ec104550 0x7f23ec106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.897+0000 7f23f3ed0640 1 --2- 172.21.15.29:0/3561760227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23ec101c20 0x7f23ec104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.897+0000 7f23f3ed0640 1 -- 172.21.15.29:0/3561760227 >> 172.21.15.29:0/3561760227 conn(0x7f23ec0fb630 msgr2=0x7f23ec0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.897+0000 7f23f3ed0640 1 -- 172.21.15.29:0/3561760227 shutdown_connections 2024-03-11T19:07:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.897+0000 7f23f3ed0640 1 -- 172.21.15.29:0/3561760227 wait complete. 2024-03-11T19:07:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.898+0000 7f23f3ed0640 1 Processor -- start 2024-03-11T19:07:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.898+0000 7f23f3ed0640 1 -- start start 2024-03-11T19:07:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.898+0000 7f23f3ed0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec101c20 0x7f23ec101590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.898+0000 7f23f3ed0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23ec104550 0x7f23ec0ffbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.898+0000 7f23f3ed0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23ec106e80 0x7f23ec100110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.898+0000 7f23f3ed0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec100870 con 0x7f23ec101c20 2024-03-11T19:07:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.898+0000 7f23f3ed0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec1009e0 con 0x7f23ec104550 2024-03-11T19:07:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.898+0000 7f23f3ed0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec1a1160 con 0x7f23ec106e80 2024-03-11T19:07:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.898+0000 7f23f1c6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec101c20 0x7f23ec101590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.898+0000 7f23f1c6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec101c20 0x7f23ec101590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48230/0 (socket says 172.21.15.29:48230) 2024-03-11T19:07:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.898+0000 7f23f1c6c640 1 -- 172.21.15.29:0/3370926247 learned_addr learned my addr 172.21.15.29:0/3370926247 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.898+0000 7f23f146b640 1 --2- 172.21.15.29:0/3370926247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23ec104550 0x7f23ec0ffbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.898+0000 7f23f246d640 1 --2- 172.21.15.29:0/3370926247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23ec106e80 0x7f23ec100110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.898+0000 7f23f1c6c640 1 -- 172.21.15.29:0/3370926247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23ec106e80 msgr2=0x7f23ec100110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.898+0000 7f23f1c6c640 1 --2- 172.21.15.29:0/3370926247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23ec106e80 0x7f23ec100110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.898+0000 7f23f1c6c640 1 -- 172.21.15.29:0/3370926247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23ec104550 msgr2=0x7f23ec0ffbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.898+0000 7f23f1c6c640 1 --2- 172.21.15.29:0/3370926247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23ec104550 0x7f23ec0ffbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.898+0000 7f23f1c6c640 1 -- 172.21.15.29:0/3370926247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e4009900 con 0x7f23ec101c20 2024-03-11T19:07:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.898+0000 7f23f1c6c640 1 --2- 172.21.15.29:0/3370926247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec101c20 0x7f23ec101590 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f23e800d980 tx=0x7f23e800de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.899+0000 7f23e2ffd640 1 -- 172.21.15.29:0/3370926247 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e8009a30 con 0x7f23ec101c20 2024-03-11T19:07:54.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.899+0000 7f23e2ffd640 1 -- 172.21.15.29:0/3370926247 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f23e80044d0 con 0x7f23ec101c20 2024-03-11T19:07:54.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.899+0000 7f23f3ed0640 1 -- 172.21.15.29:0/3370926247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23ec1a1410 con 0x7f23ec101c20 2024-03-11T19:07:54.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.899+0000 7f23e2ffd640 1 -- 172.21.15.29:0/3370926247 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e80047a0 con 0x7f23ec101c20 2024-03-11T19:07:54.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.899+0000 7f23f3ed0640 1 -- 172.21.15.29:0/3370926247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23ec1a1970 con 0x7f23ec101c20 2024-03-11T19:07:54.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.900+0000 7f23f3ed0640 1 -- 172.21.15.29:0/3370926247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23ec0fd120 con 0x7f23ec101c20 2024-03-11T19:07:54.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.901+0000 7f23e2ffd640 1 -- 172.21.15.29:0/3370926247 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f23e8004020 con 0x7f23ec101c20 2024-03-11T19:07:54.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.901+0000 7f23e2ffd640 1 --2- 172.21.15.29:0/3370926247 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23d80a1ab0 0x7f23d80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:54.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.902+0000 7f23f146b640 1 --2- 172.21.15.29:0/3370926247 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23d80a1ab0 0x7f23d80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:54.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.902+0000 7f23e2ffd640 1 -- 172.21.15.29:0/3370926247 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f23e80ccc10 con 0x7f23ec101c20 2024-03-11T19:07:54.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.902+0000 7f23f146b640 1 --2- 172.21.15.29:0/3370926247 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23d80a1ab0 0x7f23d80a3f70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f23ec101300 tx=0x7f23dc00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:54.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:54.906+0000 7f23e2ffd640 1 -- 172.21.15.29:0/3370926247 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f23e8095f70 con 0x7f23ec101c20 2024-03-11T19:07:55.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:54 smithi029 ceph-mon[32439]: pgmap v292: 97 pgs: 20 creating+peering, 72 active+clean, 5 unknown; 579 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-03-11T19:07:55.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:54 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1020585242' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-11T19:07:55.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:54 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] up:active 2024-03-11T19:07:55.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:54 smithi029 ceph-mon[32439]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-11T19:07:55.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:54 smithi029 ceph-mon[32439]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-11T19:07:55.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:54 smithi029 ceph-mon[32439]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-11T19:07:55.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:54 smithi029 ceph-mon[32439]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-11T19:07:55.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:54 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3691423902' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-11T19:07:55.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:54 smithi170 ceph-mon[35690]: pgmap v292: 97 pgs: 20 creating+peering, 72 active+clean, 5 unknown; 579 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-03-11T19:07:55.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:54 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1020585242' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-11T19:07:55.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:54 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] up:active 2024-03-11T19:07:55.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:54 smithi170 ceph-mon[35690]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-11T19:07:55.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:54 smithi170 ceph-mon[35690]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-11T19:07:55.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:54 smithi170 ceph-mon[35690]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-11T19:07:55.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:54 smithi170 ceph-mon[35690]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-11T19:07:55.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:54 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3691423902' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-11T19:07:55.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.115+0000 7f23f3ed0640 1 -- 172.21.15.29:0/3370926247 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f23ec02ceb0 con 0x7f23d80a1ab0 2024-03-11T19:07:55.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:54 smithi190 ceph-mon[36292]: pgmap v292: 97 pgs: 20 creating+peering, 72 active+clean, 5 unknown; 579 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-03-11T19:07:55.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:54 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1020585242' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-11T19:07:55.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:54 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] up:active 2024-03-11T19:07:55.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:54 smithi190 ceph-mon[36292]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-11T19:07:55.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:54 smithi190 ceph-mon[36292]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-11T19:07:55.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:54 smithi190 ceph-mon[36292]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-11T19:07:55.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:54 smithi190 ceph-mon[36292]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-11T19:07:55.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:54 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3691423902' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-11T19:07:55.406 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:07:55 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:07:55.120+0000 7f36239d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:07:55.406 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:07:55 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:07:55.120+0000 7f36239d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:07:55.406 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:07:55 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:07:55.120+0000 7f36239d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:07:55.406 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:07:55 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:07:55.120+0000 7f36239d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:07:55.406 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:07:55 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:07:55.120+0000 7f36239d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:07:55.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.798+0000 7f23e2ffd640 1 -- 172.21.15.29:0/3370926247 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f23ec02ceb0 con 0x7f23d80a1ab0 2024-03-11T19:07:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.800+0000 7f23f3ed0640 1 -- 172.21.15.29:0/3370926247 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23d80a1ab0 msgr2=0x7f23d80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.800+0000 7f23f3ed0640 1 --2- 172.21.15.29:0/3370926247 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23d80a1ab0 0x7f23d80a3f70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f23ec101300 tx=0x7f23dc00b390 comp rx=0 tx=0).stop 2024-03-11T19:07:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.800+0000 7f23f3ed0640 1 -- 172.21.15.29:0/3370926247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec101c20 msgr2=0x7f23ec101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.800+0000 7f23f3ed0640 1 --2- 172.21.15.29:0/3370926247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec101c20 0x7f23ec101590 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f23e800d980 tx=0x7f23e800de40 comp rx=0 tx=0).stop 2024-03-11T19:07:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.801+0000 7f23f3ed0640 1 -- 172.21.15.29:0/3370926247 shutdown_connections 2024-03-11T19:07:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.801+0000 7f23f3ed0640 1 --2- 172.21.15.29:0/3370926247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23ec106e80 0x7f23ec100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.801+0000 7f23f3ed0640 1 --2- 172.21.15.29:0/3370926247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23ec104550 0x7f23ec0ffbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.801+0000 7f23f3ed0640 1 --2- 172.21.15.29:0/3370926247 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23d80a1ab0 0x7f23d80a3f70 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.801+0000 7f23f3ed0640 1 --2- 172.21.15.29:0/3370926247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23ec101c20 0x7f23ec101590 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.801+0000 7f23f3ed0640 1 -- 172.21.15.29:0/3370926247 >> 172.21.15.29:0/3370926247 conn(0x7f23ec0fb630 msgr2=0x7f23ec102580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.801+0000 7f23f3ed0640 1 -- 172.21.15.29:0/3370926247 shutdown_connections 2024-03-11T19:07:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.801+0000 7f23f3ed0640 1 -- 172.21.15.29:0/3370926247 wait complete. 2024-03-11T19:07:55.817 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-11T19:07:55.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.930+0000 7f1180e39640 1 -- 172.21.15.29:0/194542888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f117c102380 msgr2=0x7f117c102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:55.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.930+0000 7f1180e39640 1 --2- 172.21.15.29:0/194542888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f117c102380 0x7f117c102800 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f116c009900 tx=0x7f116c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:55.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.930+0000 7f1180e39640 1 -- 172.21.15.29:0/194542888 shutdown_connections 2024-03-11T19:07:55.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.930+0000 7f1180e39640 1 --2- 172.21.15.29:0/194542888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f117c102d40 0x7f117c1095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:55.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.930+0000 7f1180e39640 1 --2- 172.21.15.29:0/194542888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f117c102380 0x7f117c102800 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:55.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.930+0000 7f1180e39640 1 --2- 172.21.15.29:0/194542888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f117c101190 0x7f117c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:55.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.930+0000 7f1180e39640 1 -- 172.21.15.29:0/194542888 >> 172.21.15.29:0/194542888 conn(0x7f117c0fc720 msgr2=0x7f117c0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:55.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.930+0000 7f1180e39640 1 -- 172.21.15.29:0/194542888 shutdown_connections 2024-03-11T19:07:55.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.930+0000 7f1180e39640 1 -- 172.21.15.29:0/194542888 wait complete. 2024-03-11T19:07:55.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.930+0000 7f1180e39640 1 Processor -- start 2024-03-11T19:07:55.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.931+0000 7f1180e39640 1 -- start start 2024-03-11T19:07:55.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.931+0000 7f1180e39640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f117c101190 0x7f117c074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:55.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.931+0000 7f1180e39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f117c102380 0x7f117c073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:55.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.931+0000 7f1180e39640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f117c102d40 0x7f117c073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:55.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.931+0000 7f1180e39640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c074210 con 0x7f117c102380 2024-03-11T19:07:55.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.931+0000 7f1180e39640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c074380 con 0x7f117c102d40 2024-03-11T19:07:55.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.931+0000 7f1180e39640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c1a11a0 con 0x7f117c101190 2024-03-11T19:07:55.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.931+0000 7f1179d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f117c102380 0x7f117c073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:55.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.931+0000 7f1179d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f117c102380 0x7f117c073570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60730/0 (socket says 172.21.15.29:60730) 2024-03-11T19:07:55.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.931+0000 7f1179d9b640 1 -- 172.21.15.29:0/3747623269 learned_addr learned my addr 172.21.15.29:0/3747623269 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:55.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.931+0000 7f117ad9d640 1 --2- 172.21.15.29:0/3747623269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f117c102d40 0x7f117c073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:55.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.931+0000 7f117a59c640 1 --2- 172.21.15.29:0/3747623269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f117c101190 0x7f117c074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:55.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.931+0000 7f1179d9b640 1 -- 172.21.15.29:0/3747623269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f117c101190 msgr2=0x7f117c074f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:55.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.931+0000 7f1179d9b640 1 --2- 172.21.15.29:0/3747623269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f117c101190 0x7f117c074f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:55.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.931+0000 7f1179d9b640 1 -- 172.21.15.29:0/3747623269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f117c102d40 msgr2=0x7f117c073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:55.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.931+0000 7f1179d9b640 1 --2- 172.21.15.29:0/3747623269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f117c102d40 0x7f117c073ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:55.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.931+0000 7f1179d9b640 1 -- 172.21.15.29:0/3747623269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116c009560 con 0x7f117c102380 2024-03-11T19:07:55.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.931+0000 7f1179d9b640 1 --2- 172.21.15.29:0/3747623269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f117c102380 0x7f117c073570 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f116c036040 tx=0x7f116c004370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:55.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.932+0000 7f116b7fe640 1 -- 172.21.15.29:0/3747623269 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c004a50 con 0x7f117c102380 2024-03-11T19:07:55.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.932+0000 7f1180e39640 1 -- 172.21.15.29:0/3747623269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f117c1a13f0 con 0x7f117c102380 2024-03-11T19:07:55.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.932+0000 7f116b7fe640 1 -- 172.21.15.29:0/3747623269 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f116c02de90 con 0x7f117c102380 2024-03-11T19:07:55.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.932+0000 7f116b7fe640 1 -- 172.21.15.29:0/3747623269 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c003c20 con 0x7f117c102380 2024-03-11T19:07:55.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.932+0000 7f1180e39640 1 -- 172.21.15.29:0/3747623269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117c1a1950 con 0x7f117c102380 2024-03-11T19:07:55.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.933+0000 7f1180e39640 1 -- 172.21.15.29:0/3747623269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f117c1085d0 con 0x7f117c102380 2024-03-11T19:07:55.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.934+0000 7f116b7fe640 1 -- 172.21.15.29:0/3747623269 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f116c003d80 con 0x7f117c102380 2024-03-11T19:07:55.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.934+0000 7f116b7fe640 1 --2- 172.21.15.29:0/3747623269 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f11640a1a30 0x7f11640a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:55.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.935+0000 7f117a59c640 1 --2- 172.21.15.29:0/3747623269 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f11640a1a30 0x7f11640a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:55.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.935+0000 7f116b7fe640 1 -- 172.21.15.29:0/3747623269 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f116c0efd10 con 0x7f117c102380 2024-03-11T19:07:55.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.935+0000 7f117a59c640 1 --2- 172.21.15.29:0/3747623269 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f11640a1a30 0x7f11640a3ef0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f117c1021e0 tx=0x7f1170005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:55.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:55.938+0000 7f116b7fe640 1 -- 172.21.15.29:0/3747623269 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f116c0b90f0 con 0x7f117c102380 2024-03-11T19:07:56.095 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:07:55 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-b[66380]: 2024-03-11T19:07:55.800+0000 7f358a6a7700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-11T19:07:56.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:55 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3691423902' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-11T19:07:56.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:55 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] up:active 2024-03-11T19:07:56.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:55 smithi170 ceph-mon[35690]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-11T19:07:56.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:55 smithi170 ceph-mon[35690]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-11T19:07:56.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:55 smithi170 ceph-mon[35690]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-11T19:07:56.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:55 smithi170 ceph-mon[35690]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-11T19:07:56.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:55 smithi170 ceph-mon[35690]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:07:56.102 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:55 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3691423902' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-11T19:07:56.102 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:55 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] up:active 2024-03-11T19:07:56.102 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:55 smithi029 ceph-mon[32439]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-11T19:07:56.102 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:55 smithi029 ceph-mon[32439]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-11T19:07:56.102 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:55 smithi029 ceph-mon[32439]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-11T19:07:56.103 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:55 smithi029 ceph-mon[32439]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-11T19:07:56.103 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:55 smithi029 ceph-mon[32439]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:07:56.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.160+0000 7f1180e39640 1 -- 172.21.15.29:0/3747623269 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f117c0615c0 con 0x7f11640a1a30 2024-03-11T19:07:56.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:55 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3691423902' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-11T19:07:56.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:55 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] up:active 2024-03-11T19:07:56.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:55 smithi190 ceph-mon[36292]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-11T19:07:56.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:55 smithi190 ceph-mon[36292]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-11T19:07:56.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:55 smithi190 ceph-mon[36292]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-11T19:07:56.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:55 smithi190 ceph-mon[36292]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-11T19:07:56.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:55 smithi190 ceph-mon[36292]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:07:56.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.181+0000 7f116b7fe640 1 -- 172.21.15.29:0/3747623269 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f117c0615c0 con 0x7f11640a1a30 2024-03-11T19:07:56.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.183+0000 7f1180e39640 1 -- 172.21.15.29:0/3747623269 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f11640a1a30 msgr2=0x7f11640a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:56.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.183+0000 7f1180e39640 1 --2- 172.21.15.29:0/3747623269 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f11640a1a30 0x7f11640a3ef0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f117c1021e0 tx=0x7f1170005f90 comp rx=0 tx=0).stop 2024-03-11T19:07:56.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.183+0000 7f1180e39640 1 -- 172.21.15.29:0/3747623269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f117c102380 msgr2=0x7f117c073570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:56.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.183+0000 7f1180e39640 1 --2- 172.21.15.29:0/3747623269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f117c102380 0x7f117c073570 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f116c036040 tx=0x7f116c004370 comp rx=0 tx=0).stop 2024-03-11T19:07:56.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.183+0000 7f1180e39640 1 -- 172.21.15.29:0/3747623269 shutdown_connections 2024-03-11T19:07:56.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.183+0000 7f1180e39640 1 --2- 172.21.15.29:0/3747623269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f117c102d40 0x7f117c073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.183+0000 7f1180e39640 1 --2- 172.21.15.29:0/3747623269 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f11640a1a30 0x7f11640a3ef0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.183+0000 7f1180e39640 1 --2- 172.21.15.29:0/3747623269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f117c102380 0x7f117c073570 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.183+0000 7f1180e39640 1 --2- 172.21.15.29:0/3747623269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f117c101190 0x7f117c074f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.183+0000 7f1180e39640 1 -- 172.21.15.29:0/3747623269 >> 172.21.15.29:0/3747623269 conn(0x7f117c0fc720 msgr2=0x7f117c1057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:56.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.183+0000 7f1180e39640 1 -- 172.21.15.29:0/3747623269 shutdown_connections 2024-03-11T19:07:56.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.183+0000 7f1180e39640 1 -- 172.21.15.29:0/3747623269 wait complete. 2024-03-11T19:07:56.197 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-11T19:07:56.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.305+0000 7fa8a5cc3640 1 -- 172.21.15.29:0/2892540843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a0106ec0 msgr2=0x7fa8a01092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:56.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.305+0000 7fa8a5cc3640 1 --2- 172.21.15.29:0/2892540843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a0106ec0 0x7fa8a01092b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa898009ca0 tx=0x7fa89802c0f0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.305+0000 7fa8a5cc3640 1 -- 172.21.15.29:0/2892540843 shutdown_connections 2024-03-11T19:07:56.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.305+0000 7fa8a5cc3640 1 --2- 172.21.15.29:0/2892540843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a0106ec0 0x7fa8a01092b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.305+0000 7fa8a5cc3640 1 --2- 172.21.15.29:0/2892540843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8a0104590 0x7fa8a0106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.305+0000 7fa8a5cc3640 1 --2- 172.21.15.29:0/2892540843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8a0101c60 0x7fa8a0104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.305+0000 7fa8a5cc3640 1 -- 172.21.15.29:0/2892540843 >> 172.21.15.29:0/2892540843 conn(0x7fa8a00fb610 msgr2=0x7fa8a00fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:56.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.305+0000 7fa8a5cc3640 1 -- 172.21.15.29:0/2892540843 shutdown_connections 2024-03-11T19:07:56.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.305+0000 7fa8a5cc3640 1 -- 172.21.15.29:0/2892540843 wait complete. 2024-03-11T19:07:56.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.306+0000 7fa8a5cc3640 1 Processor -- start 2024-03-11T19:07:56.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.306+0000 7fa8a5cc3640 1 -- start start 2024-03-11T19:07:56.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.306+0000 7fa8a5cc3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8a0101c60 0x7fa8a0199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:56.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.306+0000 7fa8a5cc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a0104590 0x7fa8a01998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:56.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.306+0000 7fa8a5cc3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8a0106ec0 0x7fa8a01a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:56.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.306+0000 7fa8a5cc3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a01a0e80 con 0x7fa8a0104590 2024-03-11T19:07:56.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.306+0000 7fa8a5cc3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a01a0ff0 con 0x7fa8a0106ec0 2024-03-11T19:07:56.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.306+0000 7fa8a5cc3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a01a11b0 con 0x7fa8a0101c60 2024-03-11T19:07:56.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.306+0000 7fa89effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a0104590 0x7fa8a01998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:56.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.306+0000 7fa89effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a0104590 0x7fa8a01998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60744/0 (socket says 172.21.15.29:60744) 2024-03-11T19:07:56.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.306+0000 7fa89effd640 1 -- 172.21.15.29:0/126344012 learned_addr learned my addr 172.21.15.29:0/126344012 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:56.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.306+0000 7fa89ffff640 1 --2- 172.21.15.29:0/126344012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8a0106ec0 0x7fa8a01a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:56.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.307+0000 7fa89f7fe640 1 --2- 172.21.15.29:0/126344012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8a0101c60 0x7fa8a0199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:56.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.307+0000 7fa89effd640 1 -- 172.21.15.29:0/126344012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8a0101c60 msgr2=0x7fa8a0199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:56.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.307+0000 7fa89effd640 1 --2- 172.21.15.29:0/126344012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8a0101c60 0x7fa8a0199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.307+0000 7fa89effd640 1 -- 172.21.15.29:0/126344012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8a0106ec0 msgr2=0x7fa8a01a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:56.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.307+0000 7fa89effd640 1 --2- 172.21.15.29:0/126344012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8a0106ec0 0x7fa8a01a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.307+0000 7fa89effd640 1 -- 172.21.15.29:0/126344012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa898009900 con 0x7fa8a0104590 2024-03-11T19:07:56.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.307+0000 7fa89f7fe640 1 --2- 172.21.15.29:0/126344012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8a0101c60 0x7fa8a0199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:07:56.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.307+0000 7fa89effd640 1 --2- 172.21.15.29:0/126344012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a0104590 0x7fa8a01998d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa89000d740 tx=0x7fa89000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:56.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.307+0000 7fa89cff9640 1 -- 172.21.15.29:0/126344012 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa890009c30 con 0x7fa8a0104590 2024-03-11T19:07:56.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.307+0000 7fa89cff9640 1 -- 172.21.15.29:0/126344012 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa89000de50 con 0x7fa8a0104590 2024-03-11T19:07:56.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.307+0000 7fa8a5cc3640 1 -- 172.21.15.29:0/126344012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a01a1490 con 0x7fa8a0104590 2024-03-11T19:07:56.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.307+0000 7fa89cff9640 1 -- 172.21.15.29:0/126344012 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa890004e40 con 0x7fa8a0104590 2024-03-11T19:07:56.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.307+0000 7fa8a5cc3640 1 -- 172.21.15.29:0/126344012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a01a1950 con 0x7fa8a0104590 2024-03-11T19:07:56.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.308+0000 7fa8a5cc3640 1 -- 172.21.15.29:0/126344012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a01089b0 con 0x7fa8a0104590 2024-03-11T19:07:56.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.309+0000 7fa89cff9640 1 -- 172.21.15.29:0/126344012 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fa890004fa0 con 0x7fa8a0104590 2024-03-11T19:07:56.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.310+0000 7fa89cff9640 1 --2- 172.21.15.29:0/126344012 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa8880a1a30 0x7fa8880a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:56.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.310+0000 7fa89cff9640 1 -- 172.21.15.29:0/126344012 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa8900ccc70 con 0x7fa8a0104590 2024-03-11T19:07:56.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.310+0000 7fa89f7fe640 1 --2- 172.21.15.29:0/126344012 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa8880a1a30 0x7fa8880a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:56.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.310+0000 7fa89f7fe640 1 --2- 172.21.15.29:0/126344012 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa8880a1a30 0x7fa8880a3ef0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa894009b50 tx=0x7fa894005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:56.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.314+0000 7fa89cff9640 1 -- 172.21.15.29:0/126344012 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa890099b00 con 0x7fa8a0104590 2024-03-11T19:07:56.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.527+0000 7fa8a5cc3640 1 -- 172.21.15.29:0/126344012 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8a002ce40 con 0x7fa8880a1a30 2024-03-11T19:07:56.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.531+0000 7fa89cff9640 1 -- 172.21.15.29:0/126344012 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fa8a002ce40 con 0x7fa8880a1a30 2024-03-11T19:07:56.534 INFO:teuthology.orchestra.run.smithi029.stdout:/volumes/qa/sv_0/49ed91cb-cc36-4060-8530-37ec69b004c8 2024-03-11T19:07:56.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.533+0000 7fa8a5cc3640 1 -- 172.21.15.29:0/126344012 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa8880a1a30 msgr2=0x7fa8880a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:56.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.533+0000 7fa8a5cc3640 1 --2- 172.21.15.29:0/126344012 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa8880a1a30 0x7fa8880a3ef0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa894009b50 tx=0x7fa894005f90 comp rx=0 tx=0).stop 2024-03-11T19:07:56.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.533+0000 7fa8a5cc3640 1 -- 172.21.15.29:0/126344012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a0104590 msgr2=0x7fa8a01998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:56.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.533+0000 7fa8a5cc3640 1 --2- 172.21.15.29:0/126344012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a0104590 0x7fa8a01998d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa89000d740 tx=0x7fa89000dc00 comp rx=0 tx=0).stop 2024-03-11T19:07:56.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.533+0000 7fa8a5cc3640 1 -- 172.21.15.29:0/126344012 shutdown_connections 2024-03-11T19:07:56.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.533+0000 7fa8a5cc3640 1 --2- 172.21.15.29:0/126344012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8a0106ec0 0x7fa8a01a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.533+0000 7fa8a5cc3640 1 --2- 172.21.15.29:0/126344012 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa8880a1a30 0x7fa8880a3ef0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.533+0000 7fa8a5cc3640 1 --2- 172.21.15.29:0/126344012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8a0104590 0x7fa8a01998d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.533+0000 7fa8a5cc3640 1 --2- 172.21.15.29:0/126344012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8a0101c60 0x7fa8a0199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.533+0000 7fa8a5cc3640 1 -- 172.21.15.29:0/126344012 >> 172.21.15.29:0/126344012 conn(0x7fa8a00fb610 msgr2=0x7fa8a0101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:56.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.533+0000 7fa8a5cc3640 1 -- 172.21.15.29:0/126344012 shutdown_connections 2024-03-11T19:07:56.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.533+0000 7fa8a5cc3640 1 -- 172.21.15.29:0/126344012 wait complete. 2024-03-11T19:07:56.547 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-11T19:07:56.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.652+0000 7f6331c15640 1 -- 172.21.15.29:0/1044243928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f632c101c30 msgr2=0x7f632c1084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:56.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.652+0000 7f6331c15640 1 --2- 172.21.15.29:0/1044243928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f632c101c30 0x7f632c1084c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6324009ca0 tx=0x7f632402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.652+0000 7f6331c15640 1 -- 172.21.15.29:0/1044243928 shutdown_connections 2024-03-11T19:07:56.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.652+0000 7f6331c15640 1 --2- 172.21.15.29:0/1044243928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f632c101c30 0x7f632c1084c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.652+0000 7f6331c15640 1 --2- 172.21.15.29:0/1044243928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f632c101270 0x7f632c1016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.652+0000 7f6331c15640 1 --2- 172.21.15.29:0/1044243928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f632c100080 0x7f632c1004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.652+0000 7f6331c15640 1 -- 172.21.15.29:0/1044243928 >> 172.21.15.29:0/1044243928 conn(0x7f632c0fb610 msgr2=0x7f632c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:56.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.652+0000 7f6331c15640 1 -- 172.21.15.29:0/1044243928 shutdown_connections 2024-03-11T19:07:56.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.652+0000 7f6331c15640 1 -- 172.21.15.29:0/1044243928 wait complete. 2024-03-11T19:07:56.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.652+0000 7f6331c15640 1 Processor -- start 2024-03-11T19:07:56.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.652+0000 7f6331c15640 1 -- start start 2024-03-11T19:07:56.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.652+0000 7f6331c15640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f632c100080 0x7f632c068eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:56.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.652+0000 7f6331c15640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f632c101270 0x7f632c19be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:56.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.652+0000 7f6331c15640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f632c101c30 0x7f632c19e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:56.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.652+0000 7f6331c15640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c19ec10 con 0x7f632c101c30 2024-03-11T19:07:56.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.652+0000 7f6331c15640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c19ed80 con 0x7f632c100080 2024-03-11T19:07:56.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.652+0000 7f6331c15640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c19ef40 con 0x7f632c101270 2024-03-11T19:07:56.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.653+0000 7f632bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f632c101c30 0x7f632c19e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:56.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.653+0000 7f632bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f632c101c30 0x7f632c19e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60752/0 (socket says 172.21.15.29:60752) 2024-03-11T19:07:56.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.653+0000 7f632bfff640 1 -- 172.21.15.29:0/4103930938 learned_addr learned my addr 172.21.15.29:0/4103930938 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:56.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.653+0000 7f632b7fe640 1 --2- 172.21.15.29:0/4103930938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f632c100080 0x7f632c068eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:56.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.653+0000 7f6323fff640 1 --2- 172.21.15.29:0/4103930938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f632c101270 0x7f632c19be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:56.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.653+0000 7f632bfff640 1 -- 172.21.15.29:0/4103930938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f632c101270 msgr2=0x7f632c19be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:56.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.653+0000 7f632bfff640 1 --2- 172.21.15.29:0/4103930938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f632c101270 0x7f632c19be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.653+0000 7f632bfff640 1 -- 172.21.15.29:0/4103930938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f632c100080 msgr2=0x7f632c068eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:56.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.653+0000 7f632bfff640 1 --2- 172.21.15.29:0/4103930938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f632c100080 0x7f632c068eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.653+0000 7f632bfff640 1 -- 172.21.15.29:0/4103930938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6324009900 con 0x7f632c101c30 2024-03-11T19:07:56.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.653+0000 7f6323fff640 1 --2- 172.21.15.29:0/4103930938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f632c101270 0x7f632c19be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:07:56.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.653+0000 7f632bfff640 1 --2- 172.21.15.29:0/4103930938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f632c101c30 0x7f632c19e480 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6324037040 tx=0x7f6324032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:56.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.653+0000 7f63297fa640 1 -- 172.21.15.29:0/4103930938 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324034e70 con 0x7f632c101c30 2024-03-11T19:07:56.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.653+0000 7f63297fa640 1 -- 172.21.15.29:0/4103930938 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6324003e20 con 0x7f632c101c30 2024-03-11T19:07:56.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.653+0000 7f63297fa640 1 -- 172.21.15.29:0/4103930938 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324004800 con 0x7f632c101c30 2024-03-11T19:07:56.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.653+0000 7f6331c15640 1 -- 172.21.15.29:0/4103930938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632c19f1c0 con 0x7f632c101c30 2024-03-11T19:07:56.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.653+0000 7f6331c15640 1 -- 172.21.15.29:0/4103930938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632c19f670 con 0x7f632c101c30 2024-03-11T19:07:56.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.654+0000 7f6331c15640 1 -- 172.21.15.29:0/4103930938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632c107410 con 0x7f632c101c30 2024-03-11T19:07:56.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.655+0000 7f63297fa640 1 -- 172.21.15.29:0/4103930938 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f632403b040 con 0x7f632c101c30 2024-03-11T19:07:56.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.656+0000 7f63297fa640 1 --2- 172.21.15.29:0/4103930938 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f63180a1a30 0x7f63180a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:56.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.656+0000 7f632b7fe640 1 --2- 172.21.15.29:0/4103930938 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f63180a1a30 0x7f63180a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:56.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.656+0000 7f63297fa640 1 -- 172.21.15.29:0/4103930938 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f63240efab0 con 0x7f632c101c30 2024-03-11T19:07:56.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.657+0000 7f632b7fe640 1 --2- 172.21.15.29:0/4103930938 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f63180a1a30 0x7f63180a3ef0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f632c1010d0 tx=0x7f631c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:56.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.661+0000 7f63297fa640 1 -- 172.21.15.29:0/4103930938 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f63240b8d70 con 0x7f632c101c30 2024-03-11T19:07:56.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.794+0000 7f63297fa640 1 -- 172.21.15.29:0/4103930938 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f632400a040 con 0x7f632c101c30 2024-03-11T19:07:56.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.884+0000 7f6331c15640 1 -- 172.21.15.29:0/4103930938 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f632c02ceb0 con 0x7f63180a1a30 2024-03-11T19:07:56.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.902+0000 7f63297fa640 1 -- 172.21.15.29:0/4103930938 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f632c02ceb0 con 0x7f63180a1a30 2024-03-11T19:07:56.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:56 smithi029 ceph-mon[32439]: pgmap v293: 97 pgs: 97 active+clean; 581 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s wr, 15 op/s 2024-03-11T19:07:56.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:56 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] up:active 2024-03-11T19:07:56.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:56 smithi029 ceph-mon[32439]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-11T19:07:56.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:56 smithi029 ceph-mon[32439]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-11T19:07:56.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:56 smithi029 ceph-mon[32439]: Cluster is now healthy 2024-03-11T19:07:56.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:56 smithi029 ceph-mon[32439]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-03-11T19:07:56.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:56 smithi029 ceph-mon[32439]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-11T19:07:56.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:56 smithi029 ceph-mon[32439]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-11T19:07:56.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:56 smithi029 ceph-mon[32439]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:07:56.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:56 smithi029 ceph-mon[32439]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:07:56.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:56 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] up:active 2024-03-11T19:07:56.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:56 smithi029 ceph-mon[32439]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-11T19:07:56.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.904+0000 7f6331c15640 1 -- 172.21.15.29:0/4103930938 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f63180a1a30 msgr2=0x7f63180a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:56.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.905+0000 7f6331c15640 1 --2- 172.21.15.29:0/4103930938 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f63180a1a30 0x7f63180a3ef0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f632c1010d0 tx=0x7f631c005f90 comp rx=0 tx=0).stop 2024-03-11T19:07:56.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.905+0000 7f6331c15640 1 -- 172.21.15.29:0/4103930938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f632c101c30 msgr2=0x7f632c19e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:56.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.905+0000 7f6331c15640 1 --2- 172.21.15.29:0/4103930938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f632c101c30 0x7f632c19e480 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6324037040 tx=0x7f6324032980 comp rx=0 tx=0).stop 2024-03-11T19:07:56.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.905+0000 7f6331c15640 1 -- 172.21.15.29:0/4103930938 shutdown_connections 2024-03-11T19:07:56.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.905+0000 7f6331c15640 1 --2- 172.21.15.29:0/4103930938 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f63180a1a30 0x7f63180a3ef0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.905+0000 7f6331c15640 1 --2- 172.21.15.29:0/4103930938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f632c101c30 0x7f632c19e480 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.905+0000 7f6331c15640 1 --2- 172.21.15.29:0/4103930938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f632c101270 0x7f632c19be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.905+0000 7f6331c15640 1 --2- 172.21.15.29:0/4103930938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f632c100080 0x7f632c068eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:56.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.905+0000 7f6331c15640 1 -- 172.21.15.29:0/4103930938 >> 172.21.15.29:0/4103930938 conn(0x7f632c0fb610 msgr2=0x7f632c1045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:56.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.905+0000 7f6331c15640 1 -- 172.21.15.29:0/4103930938 shutdown_connections 2024-03-11T19:07:56.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:56.905+0000 7f6331c15640 1 -- 172.21.15.29:0/4103930938 wait complete. 2024-03-11T19:07:56.919 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-11T19:07:57.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.023+0000 7f7a32688640 1 -- 172.21.15.29:0/2515687289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a2c0ffa50 msgr2=0x7f7a2c0ffe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:57.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.023+0000 7f7a32688640 1 --2- 172.21.15.29:0/2515687289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a2c0ffa50 0x7f7a2c0ffe70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7a24009ca0 tx=0x7f7a2402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.023+0000 7f7a32688640 1 -- 172.21.15.29:0/2515687289 shutdown_connections 2024-03-11T19:07:57.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.023+0000 7f7a32688640 1 --2- 172.21.15.29:0/2515687289 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a2c100d50 0x7f7a2c1011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.023+0000 7f7a32688640 1 --2- 172.21.15.29:0/2515687289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a2c1003b0 0x7f7a2c100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.023+0000 7f7a32688640 1 --2- 172.21.15.29:0/2515687289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a2c0ffa50 0x7f7a2c0ffe70 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.023+0000 7f7a32688640 1 -- 172.21.15.29:0/2515687289 >> 172.21.15.29:0/2515687289 conn(0x7f7a2c0fb610 msgr2=0x7f7a2c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:57.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.023+0000 7f7a32688640 1 -- 172.21.15.29:0/2515687289 shutdown_connections 2024-03-11T19:07:57.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.023+0000 7f7a32688640 1 -- 172.21.15.29:0/2515687289 wait complete. 2024-03-11T19:07:57.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.024+0000 7f7a32688640 1 Processor -- start 2024-03-11T19:07:57.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.024+0000 7f7a32688640 1 -- start start 2024-03-11T19:07:57.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.024+0000 7f7a32688640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a2c0ffa50 0x7f7a2c1993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:57.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.024+0000 7f7a32688640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a2c1003b0 0x7f7a2c199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:57.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.024+0000 7f7a32688640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a2c100d50 0x7f7a2c1a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:57.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.024+0000 7f7a32688640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c1a0ec0 con 0x7f7a2c1003b0 2024-03-11T19:07:57.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.024+0000 7f7a32688640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c1a1030 con 0x7f7a2c0ffa50 2024-03-11T19:07:57.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.024+0000 7f7a32688640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c1a11f0 con 0x7f7a2c100d50 2024-03-11T19:07:57.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.024+0000 7f7a2b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a2c1003b0 0x7f7a2c199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:57.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.024+0000 7f7a2bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a2c0ffa50 0x7f7a2c1993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:57.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.024+0000 7f7a2b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a2c1003b0 0x7f7a2c199910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60766/0 (socket says 172.21.15.29:60766) 2024-03-11T19:07:57.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.024+0000 7f7a2b7fe640 1 -- 172.21.15.29:0/910680605 learned_addr learned my addr 172.21.15.29:0/910680605 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:57.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.024+0000 7f7a30c25640 1 --2- 172.21.15.29:0/910680605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a2c100d50 0x7f7a2c1a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:57.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.024+0000 7f7a2bfff640 1 -- 172.21.15.29:0/910680605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a2c100d50 msgr2=0x7f7a2c1a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:57.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.024+0000 7f7a2bfff640 1 --2- 172.21.15.29:0/910680605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a2c100d50 0x7f7a2c1a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.024+0000 7f7a2bfff640 1 -- 172.21.15.29:0/910680605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a2c1003b0 msgr2=0x7f7a2c199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:57.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.024+0000 7f7a2bfff640 1 --2- 172.21.15.29:0/910680605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a2c1003b0 0x7f7a2c199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.024+0000 7f7a2bfff640 1 -- 172.21.15.29:0/910680605 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a24009900 con 0x7f7a2c0ffa50 2024-03-11T19:07:57.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.025+0000 7f7a2bfff640 1 --2- 172.21.15.29:0/910680605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a2c0ffa50 0x7f7a2c1993d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7a2402c630 tx=0x7f7a24032980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:57.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.025+0000 7f7a297fa640 1 -- 172.21.15.29:0/910680605 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24034e70 con 0x7f7a2c0ffa50 2024-03-11T19:07:57.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.025+0000 7f7a32688640 1 -- 172.21.15.29:0/910680605 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2c1a1470 con 0x7f7a2c0ffa50 2024-03-11T19:07:57.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.025+0000 7f7a297fa640 1 -- 172.21.15.29:0/910680605 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7a24003e20 con 0x7f7a2c0ffa50 2024-03-11T19:07:57.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.025+0000 7f7a297fa640 1 -- 172.21.15.29:0/910680605 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24004840 con 0x7f7a2c0ffa50 2024-03-11T19:07:57.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.025+0000 7f7a32688640 1 -- 172.21.15.29:0/910680605 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2c1a1920 con 0x7f7a2c0ffa50 2024-03-11T19:07:57.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.026+0000 7f7a32688640 1 -- 172.21.15.29:0/910680605 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a2c10aa70 con 0x7f7a2c0ffa50 2024-03-11T19:07:57.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.027+0000 7f7a297fa640 1 -- 172.21.15.29:0/910680605 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f7a2403b040 con 0x7f7a2c0ffa50 2024-03-11T19:07:57.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.027+0000 7f7a297fa640 1 --2- 172.21.15.29:0/910680605 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7a140a1ac0 0x7f7a140a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:57.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.027+0000 7f7a2b7fe640 1 --2- 172.21.15.29:0/910680605 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7a140a1ac0 0x7f7a140a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:57.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.027+0000 7f7a297fa640 1 -- 172.21.15.29:0/910680605 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f7a240efb60 con 0x7f7a2c0ffa50 2024-03-11T19:07:57.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.028+0000 7f7a2b7fe640 1 --2- 172.21.15.29:0/910680605 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7a140a1ac0 0x7f7a140a3f80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c19a8e0 tx=0x7f7a2000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:57.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.032+0000 7f7a297fa640 1 -- 172.21.15.29:0/910680605 <== mon.2 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7a240b8f80 con 0x7f7a2c0ffa50 2024-03-11T19:07:57.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:56 smithi170 ceph-mon[35690]: pgmap v293: 97 pgs: 97 active+clean; 581 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s wr, 15 op/s 2024-03-11T19:07:57.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:56 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] up:active 2024-03-11T19:07:57.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:56 smithi170 ceph-mon[35690]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-11T19:07:57.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:56 smithi170 ceph-mon[35690]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-11T19:07:57.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:56 smithi170 ceph-mon[35690]: Cluster is now healthy 2024-03-11T19:07:57.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:56 smithi170 ceph-mon[35690]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-03-11T19:07:57.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:56 smithi170 ceph-mon[35690]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-11T19:07:57.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:56 smithi170 ceph-mon[35690]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-11T19:07:57.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:56 smithi170 ceph-mon[35690]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:07:57.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:56 smithi170 ceph-mon[35690]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:07:57.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:56 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] up:active 2024-03-11T19:07:57.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:56 smithi170 ceph-mon[35690]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-11T19:07:57.181 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:56 smithi190 ceph-mon[36292]: pgmap v293: 97 pgs: 97 active+clean; 581 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s wr, 15 op/s 2024-03-11T19:07:57.182 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:56 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] up:active 2024-03-11T19:07:57.182 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:56 smithi190 ceph-mon[36292]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-11T19:07:57.182 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:56 smithi190 ceph-mon[36292]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-11T19:07:57.182 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:56 smithi190 ceph-mon[36292]: Cluster is now healthy 2024-03-11T19:07:57.182 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:56 smithi190 ceph-mon[36292]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-03-11T19:07:57.182 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:56 smithi190 ceph-mon[36292]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-11T19:07:57.182 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:56 smithi190 ceph-mon[36292]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-11T19:07:57.182 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:56 smithi190 ceph-mon[36292]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:07:57.182 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:56 smithi190 ceph-mon[36292]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:07:57.182 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:56 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] up:active 2024-03-11T19:07:57.182 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:56 smithi190 ceph-mon[36292]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-11T19:07:57.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.266+0000 7f7a32688640 1 -- 172.21.15.29:0/910680605 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a2c02ce10 con 0x7f7a140a1ac0 2024-03-11T19:07:57.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.270+0000 7f7a297fa640 1 -- 172.21.15.29:0/910680605 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f7a2c02ce10 con 0x7f7a140a1ac0 2024-03-11T19:07:57.273 INFO:teuthology.orchestra.run.smithi029.stdout:/volumes/qa/sv_1/4183a391-7854-40e8-8885-8397644bcec5 2024-03-11T19:07:57.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.272+0000 7f7a32688640 1 -- 172.21.15.29:0/910680605 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7a140a1ac0 msgr2=0x7f7a140a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:57.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.272+0000 7f7a32688640 1 --2- 172.21.15.29:0/910680605 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7a140a1ac0 0x7f7a140a3f80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c19a8e0 tx=0x7f7a2000b410 comp rx=0 tx=0).stop 2024-03-11T19:07:57.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.272+0000 7f7a32688640 1 -- 172.21.15.29:0/910680605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a2c0ffa50 msgr2=0x7f7a2c1993d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:57.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.272+0000 7f7a32688640 1 --2- 172.21.15.29:0/910680605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a2c0ffa50 0x7f7a2c1993d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7a2402c630 tx=0x7f7a24032980 comp rx=0 tx=0).stop 2024-03-11T19:07:57.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.272+0000 7f7a32688640 1 -- 172.21.15.29:0/910680605 shutdown_connections 2024-03-11T19:07:57.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.272+0000 7f7a32688640 1 --2- 172.21.15.29:0/910680605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a2c100d50 0x7f7a2c1a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.272+0000 7f7a32688640 1 --2- 172.21.15.29:0/910680605 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7a140a1ac0 0x7f7a140a3f80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.272+0000 7f7a32688640 1 --2- 172.21.15.29:0/910680605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a2c1003b0 0x7f7a2c199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.272+0000 7f7a32688640 1 --2- 172.21.15.29:0/910680605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a2c0ffa50 0x7f7a2c1993d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.272+0000 7f7a32688640 1 -- 172.21.15.29:0/910680605 >> 172.21.15.29:0/910680605 conn(0x7f7a2c0fb610 msgr2=0x7f7a2c0fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:57.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.272+0000 7f7a32688640 1 -- 172.21.15.29:0/910680605 shutdown_connections 2024-03-11T19:07:57.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.272+0000 7f7a32688640 1 -- 172.21.15.29:0/910680605 wait complete. 2024-03-11T19:07:57.286 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:07:57.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.387+0000 7f12abd7a640 1 -- 172.21.15.29:0/2271338742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a4101310 msgr2=0x7f12a4101790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:57.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.387+0000 7f12abd7a640 1 --2- 172.21.15.29:0/2271338742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a4101310 0x7f12a4101790 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1298009900 tx=0x7f129802bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:57.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.387+0000 7f12abd7a640 1 -- 172.21.15.29:0/2271338742 shutdown_connections 2024-03-11T19:07:57.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.387+0000 7f12abd7a640 1 --2- 172.21.15.29:0/2271338742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12a4101cd0 0x7f12a4108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.387+0000 7f12abd7a640 1 --2- 172.21.15.29:0/2271338742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a4101310 0x7f12a4101790 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.387+0000 7f12abd7a640 1 --2- 172.21.15.29:0/2271338742 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12a4100120 0x7f12a4100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.387+0000 7f12abd7a640 1 -- 172.21.15.29:0/2271338742 >> 172.21.15.29:0/2271338742 conn(0x7f12a40fb6b0 msgr2=0x7f12a40fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:57.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.387+0000 7f12abd7a640 1 -- 172.21.15.29:0/2271338742 shutdown_connections 2024-03-11T19:07:57.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.387+0000 7f12abd7a640 1 -- 172.21.15.29:0/2271338742 wait complete. 2024-03-11T19:07:57.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.388+0000 7f12abd7a640 1 Processor -- start 2024-03-11T19:07:57.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.388+0000 7f12abd7a640 1 -- start start 2024-03-11T19:07:57.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.388+0000 7f12abd7a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12a4100120 0x7f12a4197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:57.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.388+0000 7f12abd7a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a4101310 0x7f12a41976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:57.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.388+0000 7f12abd7a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12a4101cd0 0x7f12a419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:57.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.388+0000 7f12abd7a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a419ec20 con 0x7f12a4101310 2024-03-11T19:07:57.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.388+0000 7f12abd7a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a419ed90 con 0x7f12a4100120 2024-03-11T19:07:57.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.388+0000 7f12abd7a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a419ef50 con 0x7f12a4101cd0 2024-03-11T19:07:57.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.388+0000 7f12a9315640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a4101310 0x7f12a41976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:57.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.388+0000 7f12a9315640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a4101310 0x7f12a41976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60782/0 (socket says 172.21.15.29:60782) 2024-03-11T19:07:57.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.388+0000 7f12a9315640 1 -- 172.21.15.29:0/3117632599 learned_addr learned my addr 172.21.15.29:0/3117632599 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:57.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.388+0000 7f12aa317640 1 --2- 172.21.15.29:0/3117632599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12a4101cd0 0x7f12a419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:57.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.388+0000 7f12a9b16640 1 --2- 172.21.15.29:0/3117632599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12a4100120 0x7f12a4197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:57.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.388+0000 7f12a9315640 1 -- 172.21.15.29:0/3117632599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12a4101cd0 msgr2=0x7f12a419e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:57.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.388+0000 7f12a9315640 1 --2- 172.21.15.29:0/3117632599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12a4101cd0 0x7f12a419e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.388+0000 7f12a9315640 1 -- 172.21.15.29:0/3117632599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12a4100120 msgr2=0x7f12a4197180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:57.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.388+0000 7f12a9315640 1 --2- 172.21.15.29:0/3117632599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12a4100120 0x7f12a4197180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.388+0000 7f12a9315640 1 -- 172.21.15.29:0/3117632599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1298009560 con 0x7f12a4101310 2024-03-11T19:07:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.388+0000 7f12a9b16640 1 --2- 172.21.15.29:0/3117632599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12a4100120 0x7f12a4197180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:07:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.389+0000 7f12a9315640 1 --2- 172.21.15.29:0/3117632599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a4101310 0x7f12a41976c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1298006010 tx=0x7f1298004370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.389+0000 7f12a2ffd640 1 -- 172.21.15.29:0/3117632599 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1298004a50 con 0x7f12a4101310 2024-03-11T19:07:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.389+0000 7f12a2ffd640 1 -- 172.21.15.29:0/3117632599 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f129802de90 con 0x7f12a4101310 2024-03-11T19:07:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.389+0000 7f12abd7a640 1 -- 172.21.15.29:0/3117632599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a419f1d0 con 0x7f12a4101310 2024-03-11T19:07:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.389+0000 7f12a2ffd640 1 -- 172.21.15.29:0/3117632599 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1298005410 con 0x7f12a4101310 2024-03-11T19:07:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.389+0000 7f12abd7a640 1 -- 172.21.15.29:0/3117632599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a419f6e0 con 0x7f12a4101310 2024-03-11T19:07:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.390+0000 7f12abd7a640 1 -- 172.21.15.29:0/3117632599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a410b970 con 0x7f12a4101310 2024-03-11T19:07:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.391+0000 7f12a2ffd640 1 -- 172.21.15.29:0/3117632599 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f1298005570 con 0x7f12a4101310 2024-03-11T19:07:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.391+0000 7f12a2ffd640 1 --2- 172.21.15.29:0/3117632599 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f128c0a1ac0 0x7f128c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.392+0000 7f12a9b16640 1 --2- 172.21.15.29:0/3117632599 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f128c0a1ac0 0x7f128c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.392+0000 7f12a2ffd640 1 -- 172.21.15.29:0/3117632599 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f12980ef3d0 con 0x7f12a4101310 2024-03-11T19:07:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.392+0000 7f12a9b16640 1 --2- 172.21.15.29:0/3117632599 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f128c0a1ac0 0x7f128c0a3f80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f12a4101170 tx=0x7f129c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:57.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.396+0000 7f12a2ffd640 1 -- 172.21.15.29:0/3117632599 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f12980b8740 con 0x7f12a4101310 2024-03-11T19:07:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.616+0000 7f12abd7a640 1 -- 172.21.15.29:0/3117632599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12a410bb00 con 0x7f12a4101310 2024-03-11T19:07:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.617+0000 7f12a2ffd640 1 -- 172.21.15.29:0/3117632599 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f12980bbe00 con 0x7f12a4101310 2024-03-11T19:07:57.619 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:57.620 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:07:56.662890+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:07:57.620 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 26 2024-03-11T19:07:57.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.619+0000 7f12abd7a640 1 -- 172.21.15.29:0/3117632599 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f128c0a1ac0 msgr2=0x7f128c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:57.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.619+0000 7f12abd7a640 1 --2- 172.21.15.29:0/3117632599 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f128c0a1ac0 0x7f128c0a3f80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f12a4101170 tx=0x7f129c005f90 comp rx=0 tx=0).stop 2024-03-11T19:07:57.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.619+0000 7f12abd7a640 1 -- 172.21.15.29:0/3117632599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a4101310 msgr2=0x7f12a41976c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:57.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.619+0000 7f12abd7a640 1 --2- 172.21.15.29:0/3117632599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a4101310 0x7f12a41976c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1298006010 tx=0x7f1298004370 comp rx=0 tx=0).stop 2024-03-11T19:07:57.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.619+0000 7f12abd7a640 1 -- 172.21.15.29:0/3117632599 shutdown_connections 2024-03-11T19:07:57.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.619+0000 7f12abd7a640 1 --2- 172.21.15.29:0/3117632599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12a4101cd0 0x7f12a419e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.619+0000 7f12abd7a640 1 --2- 172.21.15.29:0/3117632599 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f128c0a1ac0 0x7f128c0a3f80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.619+0000 7f12abd7a640 1 --2- 172.21.15.29:0/3117632599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a4101310 0x7f12a41976c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.619+0000 7f12abd7a640 1 --2- 172.21.15.29:0/3117632599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12a4100120 0x7f12a4197180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.619+0000 7f12abd7a640 1 -- 172.21.15.29:0/3117632599 >> 172.21.15.29:0/3117632599 conn(0x7f12a40fb6b0 msgr2=0x7f12a40697b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:57.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.619+0000 7f12abd7a640 1 -- 172.21.15.29:0/3117632599 shutdown_connections 2024-03-11T19:07:57.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.619+0000 7f12abd7a640 1 -- 172.21.15.29:0/3117632599 wait complete. 2024-03-11T19:07:57.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:07:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.733+0000 7f51a04d4640 1 -- 172.21.15.29:0/700357664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51981011b0 msgr2=0x7f51981015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.733+0000 7f51a04d4640 1 --2- 172.21.15.29:0/700357664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51981011b0 0x7f51981015d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5194009900 tx=0x7f519402bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.733+0000 7f51a04d4640 1 -- 172.21.15.29:0/700357664 shutdown_connections 2024-03-11T19:07:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.733+0000 7f51a04d4640 1 --2- 172.21.15.29:0/700357664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5198102d60 0x7f51981095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.733+0000 7f51a04d4640 1 --2- 172.21.15.29:0/700357664 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51981023a0 0x7f5198102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.733+0000 7f51a04d4640 1 --2- 172.21.15.29:0/700357664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51981011b0 0x7f51981015d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.733+0000 7f51a04d4640 1 -- 172.21.15.29:0/700357664 >> 172.21.15.29:0/700357664 conn(0x7f51980fc760 msgr2=0x7f51980feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.733+0000 7f51a04d4640 1 -- 172.21.15.29:0/700357664 shutdown_connections 2024-03-11T19:07:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.733+0000 7f51a04d4640 1 -- 172.21.15.29:0/700357664 wait complete. 2024-03-11T19:07:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.734+0000 7f51a04d4640 1 Processor -- start 2024-03-11T19:07:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.734+0000 7f51a04d4640 1 -- start start 2024-03-11T19:07:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.734+0000 7f51a04d4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51981011b0 0x7f5198197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.734+0000 7f51a04d4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51981023a0 0x7f5198197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.734+0000 7f51a04d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5198102d60 0x7f519819e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.734+0000 7f51a04d4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519819ec40 con 0x7f5198102d60 2024-03-11T19:07:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.734+0000 7f51a04d4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519819edb0 con 0x7f51981023a0 2024-03-11T19:07:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.734+0000 7f51a04d4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519819ef70 con 0x7f51981011b0 2024-03-11T19:07:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.734+0000 7f519ea71640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5198102d60 0x7f519819e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.734+0000 7f519ea71640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5198102d60 0x7f519819e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60808/0 (socket says 172.21.15.29:60808) 2024-03-11T19:07:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.734+0000 7f519ea71640 1 -- 172.21.15.29:0/3966982517 learned_addr learned my addr 172.21.15.29:0/3966982517 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.734+0000 7f519e270640 1 --2- 172.21.15.29:0/3966982517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51981011b0 0x7f5198197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.734+0000 7f519da6f640 1 --2- 172.21.15.29:0/3966982517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51981023a0 0x7f5198197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.734+0000 7f519ea71640 1 -- 172.21.15.29:0/3966982517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51981011b0 msgr2=0x7f5198197150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.734+0000 7f519ea71640 1 --2- 172.21.15.29:0/3966982517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51981011b0 0x7f5198197150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.734+0000 7f519ea71640 1 -- 172.21.15.29:0/3966982517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51981023a0 msgr2=0x7f5198197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.734+0000 7f519ea71640 1 --2- 172.21.15.29:0/3966982517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51981023a0 0x7f5198197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.734+0000 7f519ea71640 1 -- 172.21.15.29:0/3966982517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5194009560 con 0x7f5198102d60 2024-03-11T19:07:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.735+0000 7f519ea71640 1 --2- 172.21.15.29:0/3966982517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5198102d60 0x7f519819e700 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f519000eb10 tx=0x7f519000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.735+0000 7f518f7fe640 1 -- 172.21.15.29:0/3966982517 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5190004b20 con 0x7f5198102d60 2024-03-11T19:07:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.735+0000 7f518f7fe640 1 -- 172.21.15.29:0/3966982517 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5190004c80 con 0x7f5198102d60 2024-03-11T19:07:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.735+0000 7f51a04d4640 1 -- 172.21.15.29:0/3966982517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f519819f250 con 0x7f5198102d60 2024-03-11T19:07:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.735+0000 7f518f7fe640 1 -- 172.21.15.29:0/3966982517 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5190010690 con 0x7f5198102d60 2024-03-11T19:07:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.735+0000 7f51a04d4640 1 -- 172.21.15.29:0/3966982517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f519819f710 con 0x7f5198102d60 2024-03-11T19:07:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.736+0000 7f51a04d4640 1 -- 172.21.15.29:0/3966982517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51981085c0 con 0x7f5198102d60 2024-03-11T19:07:57.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.737+0000 7f518f7fe640 1 -- 172.21.15.29:0/3966982517 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f5190005020 con 0x7f5198102d60 2024-03-11T19:07:57.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.737+0000 7f518f7fe640 1 --2- 172.21.15.29:0/3966982517 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51840a1ac0 0x7f51840a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:57.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.737+0000 7f519e270640 1 --2- 172.21.15.29:0/3966982517 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51840a1ac0 0x7f51840a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:57.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.737+0000 7f518f7fe640 1 -- 172.21.15.29:0/3966982517 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f5190014070 con 0x7f5198102d60 2024-03-11T19:07:57.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.738+0000 7f519e270640 1 --2- 172.21.15.29:0/3966982517 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51840a1ac0 0x7f51840a3f80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f519402c7d0 tx=0x7f5194005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:57.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.743+0000 7f518f7fe640 1 -- 172.21.15.29:0/3966982517 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5190096c40 con 0x7f5198102d60 2024-03-11T19:07:57.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:57 smithi029 ceph-mon[32439]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-11T19:07:57.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:57 smithi029 ceph-mon[32439]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:07:57.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:57 smithi029 ceph-mon[32439]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:07:57.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:57 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3117632599' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:57.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.928+0000 7f51a04d4640 1 -- 172.21.15.29:0/3966982517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f519802ce40 con 0x7f5198102d60 2024-03-11T19:07:57.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.930+0000 7f518f7fe640 1 -- 172.21.15.29:0/3966982517 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22219 (secure 0 0 0) 0x7f5190085070 con 0x7f5198102d60 2024-03-11T19:07:57.933 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:57.934 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":87,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:07:50.732035+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:07:47.563184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:07:48.282461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:07:57.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.932+0000 7f51a04d4640 1 -- 172.21.15.29:0/3966982517 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51840a1ac0 msgr2=0x7f51840a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:57.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.932+0000 7f51a04d4640 1 --2- 172.21.15.29:0/3966982517 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51840a1ac0 0x7f51840a3f80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f519402c7d0 tx=0x7f5194005fd0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.932+0000 7f51a04d4640 1 -- 172.21.15.29:0/3966982517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5198102d60 msgr2=0x7f519819e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:57.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.932+0000 7f51a04d4640 1 --2- 172.21.15.29:0/3966982517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5198102d60 0x7f519819e700 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f519000eb10 tx=0x7f519000c2d0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.932+0000 7f51a04d4640 1 -- 172.21.15.29:0/3966982517 shutdown_connections 2024-03-11T19:07:57.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.932+0000 7f51a04d4640 1 --2- 172.21.15.29:0/3966982517 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51840a1ac0 0x7f51840a3f80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.932+0000 7f51a04d4640 1 --2- 172.21.15.29:0/3966982517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5198102d60 0x7f519819e700 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.932+0000 7f51a04d4640 1 --2- 172.21.15.29:0/3966982517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51981023a0 0x7f5198197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.932+0000 7f51a04d4640 1 --2- 172.21.15.29:0/3966982517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51981011b0 0x7f5198197150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:57.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.932+0000 7f51a04d4640 1 -- 172.21.15.29:0/3966982517 >> 172.21.15.29:0/3966982517 conn(0x7f51980fc760 msgr2=0x7f5198105790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:57.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.932+0000 7f51a04d4640 1 -- 172.21.15.29:0/3966982517 shutdown_connections 2024-03-11T19:07:57.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:57.932+0000 7f51a04d4640 1 -- 172.21.15.29:0/3966982517 wait complete. 2024-03-11T19:07:57.946 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-11T19:07:57.946 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-11T19:07:58.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.045+0000 7f9fc1b7a640 1 -- 172.21.15.29:0/1144593505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fbc1011a0 msgr2=0x7f9fbc1015c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:58.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.045+0000 7f9fc1b7a640 1 --2- 172.21.15.29:0/1144593505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fbc1011a0 0x7f9fbc1015c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9fac009900 tx=0x7f9fac02bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:58.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.045+0000 7f9fc1b7a640 1 -- 172.21.15.29:0/1144593505 shutdown_connections 2024-03-11T19:07:58.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.045+0000 7f9fc1b7a640 1 --2- 172.21.15.29:0/1144593505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9fbc102d50 0x7f9fbc1095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.045+0000 7f9fc1b7a640 1 --2- 172.21.15.29:0/1144593505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fbc102390 0x7f9fbc102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.045+0000 7f9fc1b7a640 1 --2- 172.21.15.29:0/1144593505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fbc1011a0 0x7f9fbc1015c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.045+0000 7f9fc1b7a640 1 -- 172.21.15.29:0/1144593505 >> 172.21.15.29:0/1144593505 conn(0x7f9fbc0fc730 msgr2=0x7f9fbc0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:58.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.045+0000 7f9fc1b7a640 1 -- 172.21.15.29:0/1144593505 shutdown_connections 2024-03-11T19:07:58.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.045+0000 7f9fc1b7a640 1 -- 172.21.15.29:0/1144593505 wait complete. 2024-03-11T19:07:58.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.046+0000 7f9fc1b7a640 1 Processor -- start 2024-03-11T19:07:58.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.046+0000 7f9fc1b7a640 1 -- start start 2024-03-11T19:07:58.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.046+0000 7f9fc1b7a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fbc1011a0 0x7f9fbc197110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:58.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.046+0000 7f9fc1b7a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fbc102390 0x7f9fbc197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:58.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.046+0000 7f9fc1b7a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9fbc102d50 0x7f9fbc19e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:58.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.046+0000 7f9fc1b7a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc19ec00 con 0x7f9fbc1011a0 2024-03-11T19:07:58.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.046+0000 7f9fc1b7a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc19ed70 con 0x7f9fbc102d50 2024-03-11T19:07:58.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.046+0000 7f9fc1b7a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc19ef30 con 0x7f9fbc102390 2024-03-11T19:07:58.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.046+0000 7f9fbb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fbc1011a0 0x7f9fbc197110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:58.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.046+0000 7f9fbb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fbc1011a0 0x7f9fbc197110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60836/0 (socket says 172.21.15.29:60836) 2024-03-11T19:07:58.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.046+0000 7f9fbb7fe640 1 -- 172.21.15.29:0/1188046858 learned_addr learned my addr 172.21.15.29:0/1188046858 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:58.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.046+0000 7f9fbaffd640 1 --2- 172.21.15.29:0/1188046858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fbc102390 0x7f9fbc197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:58.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.046+0000 7f9fbbfff640 1 --2- 172.21.15.29:0/1188046858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9fbc102d50 0x7f9fbc19e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:58.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.046+0000 7f9fbb7fe640 1 -- 172.21.15.29:0/1188046858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fbc102390 msgr2=0x7f9fbc197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:58.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.046+0000 7f9fbb7fe640 1 --2- 172.21.15.29:0/1188046858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fbc102390 0x7f9fbc197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.046+0000 7f9fbb7fe640 1 -- 172.21.15.29:0/1188046858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9fbc102d50 msgr2=0x7f9fbc19e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:58.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.046+0000 7f9fbb7fe640 1 --2- 172.21.15.29:0/1188046858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9fbc102d50 0x7f9fbc19e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.046+0000 7f9fbb7fe640 1 -- 172.21.15.29:0/1188046858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fac009560 con 0x7f9fbc1011a0 2024-03-11T19:07:58.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.047+0000 7f9fbb7fe640 1 --2- 172.21.15.29:0/1188046858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fbc1011a0 0x7f9fbc197110 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9fac02c7d0 tx=0x7f9fac0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:58.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.047+0000 7f9fb8ff9640 1 -- 172.21.15.29:0/1188046858 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fac004aa0 con 0x7f9fbc1011a0 2024-03-11T19:07:58.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.047+0000 7f9fb8ff9640 1 -- 172.21.15.29:0/1188046858 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9fac02de90 con 0x7f9fbc1011a0 2024-03-11T19:07:58.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.047+0000 7f9fc1b7a640 1 -- 172.21.15.29:0/1188046858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fbc19f1b0 con 0x7f9fbc1011a0 2024-03-11T19:07:58.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.047+0000 7f9fb8ff9640 1 -- 172.21.15.29:0/1188046858 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fac005440 con 0x7f9fbc1011a0 2024-03-11T19:07:58.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.047+0000 7f9fc1b7a640 1 -- 172.21.15.29:0/1188046858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fbc19f710 con 0x7f9fbc1011a0 2024-03-11T19:07:58.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.048+0000 7f9fc1b7a640 1 -- 172.21.15.29:0/1188046858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fbc108540 con 0x7f9fbc1011a0 2024-03-11T19:07:58.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.049+0000 7f9fb8ff9640 1 -- 172.21.15.29:0/1188046858 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f9fac0055a0 con 0x7f9fbc1011a0 2024-03-11T19:07:58.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.049+0000 7f9fb8ff9640 1 --2- 172.21.15.29:0/1188046858 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9fa40a1b40 0x7f9fa40a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:58.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.049+0000 7f9fbaffd640 1 --2- 172.21.15.29:0/1188046858 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9fa40a1b40 0x7f9fa40a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:58.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.049+0000 7f9fb8ff9640 1 -- 172.21.15.29:0/1188046858 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f9fac0ef3d0 con 0x7f9fbc1011a0 2024-03-11T19:07:58.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.050+0000 7f9fbaffd640 1 --2- 172.21.15.29:0/1188046858 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9fa40a1b40 0x7f9fa40a4000 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc198620 tx=0x7f9fb0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:58.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.054+0000 7f9fb8ff9640 1 -- 172.21.15.29:0/1188046858 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9fac0b8740 con 0x7f9fbc1011a0 2024-03-11T19:07:58.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:57 smithi170 ceph-mon[35690]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-11T19:07:58.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:57 smithi170 ceph-mon[35690]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:07:58.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:57 smithi170 ceph-mon[35690]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:07:58.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:57 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3117632599' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:58.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.231+0000 7f9fc1b7a640 1 -- 172.21.15.29:0/1188046858 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9fbc02ce40 con 0x7f9fa40a1b40 2024-03-11T19:07:58.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.239+0000 7f9fb8ff9640 1 -- 172.21.15.29:0/1188046858 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258913 (secure 0 0 0) 0x7f9fbc02ce40 con 0x7f9fa40a1b40 2024-03-11T19:07:58.245 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:58.247 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-11T19:07:58.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.241+0000 7f9fc1b7a640 1 -- 172.21.15.29:0/1188046858 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9fa40a1b40 msgr2=0x7f9fa40a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:58.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.241+0000 7f9fc1b7a640 1 --2- 172.21.15.29:0/1188046858 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9fa40a1b40 0x7f9fa40a4000 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc198620 tx=0x7f9fb0005f90 comp rx=0 tx=0).stop 2024-03-11T19:07:58.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.241+0000 7f9fc1b7a640 1 -- 172.21.15.29:0/1188046858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fbc1011a0 msgr2=0x7f9fbc197110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:58.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.241+0000 7f9fc1b7a640 1 --2- 172.21.15.29:0/1188046858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fbc1011a0 0x7f9fbc197110 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9fac02c7d0 tx=0x7f9fac0042f0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.241+0000 7f9fc1b7a640 1 -- 172.21.15.29:0/1188046858 shutdown_connections 2024-03-11T19:07:58.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.241+0000 7f9fc1b7a640 1 --2- 172.21.15.29:0/1188046858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9fbc102d50 0x7f9fbc19e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.241+0000 7f9fc1b7a640 1 --2- 172.21.15.29:0/1188046858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fbc102390 0x7f9fbc197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.241+0000 7f9fc1b7a640 1 --2- 172.21.15.29:0/1188046858 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9fa40a1b40 0x7f9fa40a4000 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.241+0000 7f9fc1b7a640 1 --2- 172.21.15.29:0/1188046858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fbc1011a0 0x7f9fbc197110 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.241+0000 7f9fc1b7a640 1 -- 172.21.15.29:0/1188046858 >> 172.21.15.29:0/1188046858 conn(0x7f9fbc0fc730 msgr2=0x7f9fbc105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:58.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.242+0000 7f9fc1b7a640 1 -- 172.21.15.29:0/1188046858 shutdown_connections 2024-03-11T19:07:58.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.242+0000 7f9fc1b7a640 1 -- 172.21.15.29:0/1188046858 wait complete. 2024-03-11T19:07:58.255 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-03-11T19:07:57.605536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631886,"num_objects":69,"num_object_clones":0,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":132,"num_read_kb":133,"num_write":336,"num_write_kb":3016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":272,"ondisk_log_size":272,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":389652,"kb_used_data":11356,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124552172,"statfs":{"total":1151940427776,"available":1151541424128,"internally_reserved":0,"allocated":11628544,"data_stored":7175090,"data_compressed":109264,"data_compressed_allocated":139264,"data_compressed_original":1777664,"omap_allocated":19142,"internal_metadata":387233082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41518,"num_objects":67,"num_object_clones":0,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":6,"num_read_kb":24,"num_write":155,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001832"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509020+0000","last_change":"2024-03-11T19:07:49.751738+0000","last_active":"2024-03-11T19:07:51.509020+0000","last_peered":"2024-03-11T19:07:51.509020+0000","last_clean":"2024-03-11T19:07:51.509020+0000","last_became_active":"2024-03-11T19:07:49.751173+0000","last_became_peered":"2024-03-11T19:07:49.751173+0000","last_unstale":"2024-03-11T19:07:51.509020+0000","last_undegraded":"2024-03-11T19:07:51.509020+0000","last_fullsized":"2024-03-11T19:07:51.509020+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:13:40.157763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739294+0000","last_change":"2024-03-11T19:07:49.749069+0000","last_active":"2024-03-11T19:07:50.739294+0000","last_peered":"2024-03-11T19:07:50.739294+0000","last_clean":"2024-03-11T19:07:50.739294+0000","last_became_active":"2024-03-11T19:07:49.747672+0000","last_became_peered":"2024-03-11T19:07:49.747672+0000","last_unstale":"2024-03-11T19:07:50.739294+0000","last_undegraded":"2024-03-11T19:07:50.739294+0000","last_fullsized":"2024-03-11T19:07:50.739294+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:57:26.379836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739214+0000","last_change":"2024-03-11T19:07:49.750298+0000","last_active":"2024-03-11T19:07:50.739214+0000","last_peered":"2024-03-11T19:07:50.739214+0000","last_clean":"2024-03-11T19:07:50.739214+0000","last_became_active":"2024-03-11T19:07:49.749985+0000","last_became_peered":"2024-03-11T19:07:49.749985+0000","last_unstale":"2024-03-11T19:07:50.739214+0000","last_undegraded":"2024-03-11T19:07:50.739214+0000","last_fullsized":"2024-03-11T19:07:50.739214+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:56.579872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.441461+0000","last_change":"2024-03-11T19:07:49.745246+0000","last_active":"2024-03-11T19:07:51.441461+0000","last_peered":"2024-03-11T19:07:51.441461+0000","last_clean":"2024-03-11T19:07:51.441461+0000","last_became_active":"2024-03-11T19:07:49.744907+0000","last_became_peered":"2024-03-11T19:07:49.744907+0000","last_unstale":"2024-03-11T19:07:51.441461+0000","last_undegraded":"2024-03-11T19:07:51.441461+0000","last_fullsized":"2024-03-11T19:07:51.441461+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:20:41.867658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739023+0000","last_change":"2024-03-11T19:07:49.747618+0000","last_active":"2024-03-11T19:07:50.739023+0000","last_peered":"2024-03-11T19:07:50.739023+0000","last_clean":"2024-03-11T19:07:50.739023+0000","last_became_active":"2024-03-11T19:07:49.747310+0000","last_became_peered":"2024-03-11T19:07:49.747310+0000","last_unstale":"2024-03-11T19:07:50.739023+0000","last_undegraded":"2024-03-11T19:07:50.739023+0000","last_fullsized":"2024-03-11T19:07:50.739023+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:15:42.260509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508731+0000","last_change":"2024-03-11T19:07:49.755564+0000","last_active":"2024-03-11T19:07:51.508731+0000","last_peered":"2024-03-11T19:07:51.508731+0000","last_clean":"2024-03-11T19:07:51.508731+0000","last_became_active":"2024-03-11T19:07:49.751322+0000","last_became_peered":"2024-03-11T19:07:49.751322+0000","last_unstale":"2024-03-11T19:07:51.508731+0000","last_undegraded":"2024-03-11T19:07:51.508731+0000","last_fullsized":"2024-03-11T19:07:51.508731+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:43:18.487887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739074+0000","last_change":"2024-03-11T19:07:49.731619+0000","last_active":"2024-03-11T19:07:50.739074+0000","last_peered":"2024-03-11T19:07:50.739074+0000","last_clean":"2024-03-11T19:07:50.739074+0000","last_became_active":"2024-03-11T19:07:49.731323+0000","last_became_peered":"2024-03-11T19:07:49.731323+0000","last_unstale":"2024-03-11T19:07:50.739074+0000","last_undegraded":"2024-03-11T19:07:50.739074+0000","last_fullsized":"2024-03-11T19:07:50.739074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:20:25.592709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.743380+0000","last_change":"2024-03-11T19:07:49.751851+0000","last_active":"2024-03-11T19:07:50.743380+0000","last_peered":"2024-03-11T19:07:50.743380+0000","last_clean":"2024-03-11T19:07:50.743380+0000","last_became_active":"2024-03-11T19:07:49.750068+0000","last_became_peered":"2024-03-11T19:07:49.750068+0000","last_unstale":"2024-03-11T19:07:50.743380+0000","last_undegraded":"2024-03-11T19:07:50.743380+0000","last_fullsized":"2024-03-11T19:07:50.743380+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:59:35.820107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508583+0000","last_change":"2024-03-11T19:07:49.752326+0000","last_active":"2024-03-11T19:07:51.508583+0000","last_peered":"2024-03-11T19:07:51.508583+0000","last_clean":"2024-03-11T19:07:51.508583+0000","last_became_active":"2024-03-11T19:07:49.748052+0000","last_became_peered":"2024-03-11T19:07:49.748052+0000","last_unstale":"2024-03-11T19:07:51.508583+0000","last_undegraded":"2024-03-11T19:07:51.508583+0000","last_fullsized":"2024-03-11T19:07:51.508583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:58:13.325964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508904+0000","last_change":"2024-03-11T19:07:49.748054+0000","last_active":"2024-03-11T19:07:51.508904+0000","last_peered":"2024-03-11T19:07:51.508904+0000","last_clean":"2024-03-11T19:07:51.508904+0000","last_became_active":"2024-03-11T19:07:49.747765+0000","last_became_peered":"2024-03-11T19:07:49.747765+0000","last_unstale":"2024-03-11T19:07:51.508904+0000","last_undegraded":"2024-03-11T19:07:51.508904+0000","last_fullsized":"2024-03-11T19:07:51.508904+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:18:05.599512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739677+0000","last_change":"2024-03-11T19:07:49.751622+0000","last_active":"2024-03-11T19:07:50.739677+0000","last_peered":"2024-03-11T19:07:50.739677+0000","last_clean":"2024-03-11T19:07:50.739677+0000","last_became_active":"2024-03-11T19:07:49.751317+0000","last_became_peered":"2024-03-11T19:07:49.751317+0000","last_unstale":"2024-03-11T19:07:50.739677+0000","last_undegraded":"2024-03-11T19:07:50.739677+0000","last_fullsized":"2024-03-11T19:07:50.739677+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:49.836096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.441285+0000","last_change":"2024-03-11T19:07:49.749160+0000","last_active":"2024-03-11T19:07:51.441285+0000","last_peered":"2024-03-11T19:07:51.441285+0000","last_clean":"2024-03-11T19:07:51.441285+0000","last_became_active":"2024-03-11T19:07:49.748838+0000","last_became_peered":"2024-03-11T19:07:49.748838+0000","last_unstale":"2024-03-11T19:07:51.441285+0000","last_undegraded":"2024-03-11T19:07:51.441285+0000","last_fullsized":"2024-03-11T19:07:51.441285+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:30:58.327196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508036+0000","last_change":"2024-03-11T19:07:49.749965+0000","last_active":"2024-03-11T19:07:51.508036+0000","last_peered":"2024-03-11T19:07:51.508036+0000","last_clean":"2024-03-11T19:07:51.508036+0000","last_became_active":"2024-03-11T19:07:49.749602+0000","last_became_peered":"2024-03-11T19:07:49.749602+0000","last_unstale":"2024-03-11T19:07:51.508036+0000","last_undegraded":"2024-03-11T19:07:51.508036+0000","last_fullsized":"2024-03-11T19:07:51.508036+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:36:08.807251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.738985+0000","last_change":"2024-03-11T19:07:49.747557+0000","last_active":"2024-03-11T19:07:50.738985+0000","last_peered":"2024-03-11T19:07:50.738985+0000","last_clean":"2024-03-11T19:07:50.738985+0000","last_became_active":"2024-03-11T19:07:49.745804+0000","last_became_peered":"2024-03-11T19:07:49.745804+0000","last_unstale":"2024-03-11T19:07:50.738985+0000","last_undegraded":"2024-03-11T19:07:50.738985+0000","last_fullsized":"2024-03-11T19:07:50.738985+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:07:26.939414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.738944+0000","last_change":"2024-03-11T19:07:49.744070+0000","last_active":"2024-03-11T19:07:50.738944+0000","last_peered":"2024-03-11T19:07:50.738944+0000","last_clean":"2024-03-11T19:07:50.738944+0000","last_became_active":"2024-03-11T19:07:49.743736+0000","last_became_peered":"2024-03-11T19:07:49.743736+0000","last_unstale":"2024-03-11T19:07:50.738944+0000","last_undegraded":"2024-03-11T19:07:50.738944+0000","last_fullsized":"2024-03-11T19:07:50.738944+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:50:33.862644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.507787+0000","last_change":"2024-03-11T19:07:49.748512+0000","last_active":"2024-03-11T19:07:51.507787+0000","last_peered":"2024-03-11T19:07:51.507787+0000","last_clean":"2024-03-11T19:07:51.507787+0000","last_became_active":"2024-03-11T19:07:49.748223+0000","last_became_peered":"2024-03-11T19:07:49.748223+0000","last_unstale":"2024-03-11T19:07:51.507787+0000","last_undegraded":"2024-03-11T19:07:51.507787+0000","last_fullsized":"2024-03-11T19:07:51.507787+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:49:43.305677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509376+0000","last_change":"2024-03-11T19:07:49.754322+0000","last_active":"2024-03-11T19:07:51.509376+0000","last_peered":"2024-03-11T19:07:51.509376+0000","last_clean":"2024-03-11T19:07:51.509376+0000","last_became_active":"2024-03-11T19:07:49.753757+0000","last_became_peered":"2024-03-11T19:07:49.753757+0000","last_unstale":"2024-03-11T19:07:51.509376+0000","last_undegraded":"2024-03-11T19:07:51.509376+0000","last_fullsized":"2024-03-11T19:07:51.509376+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:49.684978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.741917+0000","last_change":"2024-03-11T19:07:49.749546+0000","last_active":"2024-03-11T19:07:50.741917+0000","last_peered":"2024-03-11T19:07:50.741917+0000","last_clean":"2024-03-11T19:07:50.741917+0000","last_became_active":"2024-03-11T19:07:49.749200+0000","last_became_peered":"2024-03-11T19:07:49.749200+0000","last_unstale":"2024-03-11T19:07:50.741917+0000","last_undegraded":"2024-03-11T19:07:50.741917+0000","last_fullsized":"2024-03-11T19:07:50.741917+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:26:12.914015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509690+0000","last_change":"2024-03-11T19:07:49.750388+0000","last_active":"2024-03-11T19:07:51.509690+0000","last_peered":"2024-03-11T19:07:51.509690+0000","last_clean":"2024-03-11T19:07:51.509690+0000","last_became_active":"2024-03-11T19:07:49.750016+0000","last_became_peered":"2024-03-11T19:07:49.750016+0000","last_unstale":"2024-03-11T19:07:51.509690+0000","last_undegraded":"2024-03-11T19:07:51.509690+0000","last_fullsized":"2024-03-11T19:07:51.509690+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:32:23.125695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'13","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:55.790484+0000","last_change":"2024-03-11T19:07:48.738369+0000","last_active":"2024-03-11T19:07:55.790484+0000","last_peered":"2024-03-11T19:07:55.790484+0000","last_clean":"2024-03-11T19:07:55.790484+0000","last_became_active":"2024-03-11T19:07:48.738010+0000","last_became_peered":"2024-03-11T19:07:48.738010+0000","last_unstale":"2024-03-11T19:07:55.790484+0000","last_undegraded":"2024-03-11T19:07:55.790484+0000","last_fullsized":"2024-03-11T19:07:55.790484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:06:48.446454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.744152+0000","last_change":"2024-03-11T19:07:49.752622+0000","last_active":"2024-03-11T19:07:50.744152+0000","last_peered":"2024-03-11T19:07:50.744152+0000","last_clean":"2024-03-11T19:07:50.744152+0000","last_became_active":"2024-03-11T19:07:49.752242+0000","last_became_peered":"2024-03-11T19:07:49.752242+0000","last_unstale":"2024-03-11T19:07:50.744152+0000","last_undegraded":"2024-03-11T19:07:50.744152+0000","last_fullsized":"2024-03-11T19:07:50.744152+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:03:50.609641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.744038+0000","last_change":"2024-03-11T19:07:48.751657+0000","last_active":"2024-03-11T19:07:50.744038+0000","last_peered":"2024-03-11T19:07:50.744038+0000","last_clean":"2024-03-11T19:07:50.744038+0000","last_became_active":"2024-03-11T19:07:48.751294+0000","last_became_peered":"2024-03-11T19:07:48.751294+0000","last_unstale":"2024-03-11T19:07:50.744038+0000","last_undegraded":"2024-03-11T19:07:50.744038+0000","last_fullsized":"2024-03-11T19:07:50.744038+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:33:58.550707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739469+0000","last_change":"2024-03-11T19:07:49.759460+0000","last_active":"2024-03-11T19:07:50.739469+0000","last_peered":"2024-03-11T19:07:50.739469+0000","last_clean":"2024-03-11T19:07:50.739469+0000","last_became_active":"2024-03-11T19:07:49.759136+0000","last_became_peered":"2024-03-11T19:07:49.759136+0000","last_unstale":"2024-03-11T19:07:50.739469+0000","last_undegraded":"2024-03-11T19:07:50.739469+0000","last_fullsized":"2024-03-11T19:07:50.739469+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:10:37.273896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'7","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:54.782397+0000","last_change":"2024-03-11T19:07:48.744695+0000","last_active":"2024-03-11T19:07:54.782397+0000","last_peered":"2024-03-11T19:07:54.782397+0000","last_clean":"2024-03-11T19:07:54.782397+0000","last_became_active":"2024-03-11T19:07:48.744378+0000","last_became_peered":"2024-03-11T19:07:48.744378+0000","last_unstale":"2024-03-11T19:07:54.782397+0000","last_undegraded":"2024-03-11T19:07:54.782397+0000","last_fullsized":"2024-03-11T19:07:54.782397+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:55:46.601414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508089+0000","last_change":"2024-03-11T19:07:49.761429+0000","last_active":"2024-03-11T19:07:51.508089+0000","last_peered":"2024-03-11T19:07:51.508089+0000","last_clean":"2024-03-11T19:07:51.508089+0000","last_became_active":"2024-03-11T19:07:49.761120+0000","last_became_peered":"2024-03-11T19:07:49.761120+0000","last_unstale":"2024-03-11T19:07:51.508089+0000","last_undegraded":"2024-03-11T19:07:51.508089+0000","last_fullsized":"2024-03-11T19:07:51.508089+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:13:45.839528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739438+0000","last_change":"2024-03-11T19:07:48.740451+0000","last_active":"2024-03-11T19:07:50.739438+0000","last_peered":"2024-03-11T19:07:50.739438+0000","last_clean":"2024-03-11T19:07:50.739438+0000","last_became_active":"2024-03-11T19:07:48.740097+0000","last_became_peered":"2024-03-11T19:07:48.740097+0000","last_unstale":"2024-03-11T19:07:50.739438+0000","last_undegraded":"2024-03-11T19:07:50.739438+0000","last_fullsized":"2024-03-11T19:07:50.739438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:37.766790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509229+0000","last_change":"2024-03-11T19:07:49.750404+0000","last_active":"2024-03-11T19:07:51.509229+0000","last_peered":"2024-03-11T19:07:51.509229+0000","last_clean":"2024-03-11T19:07:51.509229+0000","last_became_active":"2024-03-11T19:07:49.749941+0000","last_became_peered":"2024-03-11T19:07:49.749941+0000","last_unstale":"2024-03-11T19:07:51.509229+0000","last_undegraded":"2024-03-11T19:07:51.509229+0000","last_fullsized":"2024-03-11T19:07:51.509229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:55.766729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'5","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.766952+0000","last_change":"2024-03-11T19:07:48.737100+0000","last_active":"2024-03-11T19:07:52.766952+0000","last_peered":"2024-03-11T19:07:52.766952+0000","last_clean":"2024-03-11T19:07:52.766952+0000","last_became_active":"2024-03-11T19:07:48.736768+0000","last_became_peered":"2024-03-11T19:07:48.736768+0000","last_unstale":"2024-03-11T19:07:52.766952+0000","last_undegraded":"2024-03-11T19:07:52.766952+0000","last_fullsized":"2024-03-11T19:07:52.766952+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:18:36.872151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739038+0000","last_change":"2024-03-11T19:07:49.739020+0000","last_active":"2024-03-11T19:07:50.739038+0000","last_peered":"2024-03-11T19:07:50.739038+0000","last_clean":"2024-03-11T19:07:50.739038+0000","last_became_active":"2024-03-11T19:07:49.738642+0000","last_became_peered":"2024-03-11T19:07:49.738642+0000","last_unstale":"2024-03-11T19:07:50.739038+0000","last_undegraded":"2024-03-11T19:07:50.739038+0000","last_fullsized":"2024-03-11T19:07:50.739038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:42:29.040242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'7","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:54.782428+0000","last_change":"2024-03-11T19:07:48.745192+0000","last_active":"2024-03-11T19:07:54.782428+0000","last_peered":"2024-03-11T19:07:54.782428+0000","last_clean":"2024-03-11T19:07:54.782428+0000","last_became_active":"2024-03-11T19:07:48.744868+0000","last_became_peered":"2024-03-11T19:07:48.744868+0000","last_unstale":"2024-03-11T19:07:54.782428+0000","last_undegraded":"2024-03-11T19:07:54.782428+0000","last_fullsized":"2024-03-11T19:07:54.782428+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:19:15.620186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508565+0000","last_change":"2024-03-11T19:07:49.737149+0000","last_active":"2024-03-11T19:07:51.508565+0000","last_peered":"2024-03-11T19:07:51.508565+0000","last_clean":"2024-03-11T19:07:51.508565+0000","last_became_active":"2024-03-11T19:07:49.736797+0000","last_became_peered":"2024-03-11T19:07:49.736797+0000","last_unstale":"2024-03-11T19:07:51.508565+0000","last_undegraded":"2024-03-11T19:07:51.508565+0000","last_fullsized":"2024-03-11T19:07:51.508565+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:12:10.282062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:53.773777+0000","last_change":"2024-03-11T19:07:48.743948+0000","last_active":"2024-03-11T19:07:53.773777+0000","last_peered":"2024-03-11T19:07:53.773777+0000","last_clean":"2024-03-11T19:07:53.773777+0000","last_became_active":"2024-03-11T19:07:48.743451+0000","last_became_peered":"2024-03-11T19:07:48.743451+0000","last_unstale":"2024-03-11T19:07:53.773777+0000","last_undegraded":"2024-03-11T19:07:53.773777+0000","last_fullsized":"2024-03-11T19:07:53.773777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:03:19.823287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.743297+0000","last_change":"2024-03-11T19:07:49.745518+0000","last_active":"2024-03-11T19:07:50.743297+0000","last_peered":"2024-03-11T19:07:50.743297+0000","last_clean":"2024-03-11T19:07:50.743297+0000","last_became_active":"2024-03-11T19:07:49.743719+0000","last_became_peered":"2024-03-11T19:07:49.743719+0000","last_unstale":"2024-03-11T19:07:50.743297+0000","last_undegraded":"2024-03-11T19:07:50.743297+0000","last_fullsized":"2024-03-11T19:07:50.743297+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:25:33.855679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509514+0000","last_change":"2024-03-11T19:07:48.734044+0000","last_active":"2024-03-11T19:07:51.509514+0000","last_peered":"2024-03-11T19:07:51.509514+0000","last_clean":"2024-03-11T19:07:51.509514+0000","last_became_active":"2024-03-11T19:07:48.733723+0000","last_became_peered":"2024-03-11T19:07:48.733723+0000","last_unstale":"2024-03-11T19:07:51.509514+0000","last_undegraded":"2024-03-11T19:07:51.509514+0000","last_fullsized":"2024-03-11T19:07:51.509514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:53:37.070296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.741698+0000","last_change":"2024-03-11T19:07:49.743729+0000","last_active":"2024-03-11T19:07:50.741698+0000","last_peered":"2024-03-11T19:07:50.741698+0000","last_clean":"2024-03-11T19:07:50.741698+0000","last_became_active":"2024-03-11T19:07:49.742992+0000","last_became_peered":"2024-03-11T19:07:49.742992+0000","last_unstale":"2024-03-11T19:07:50.741698+0000","last_undegraded":"2024-03-11T19:07:50.741698+0000","last_fullsized":"2024-03-11T19:07:50.741698+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:46.477177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:56.672097+0000","last_change":"2024-03-11T19:07:48.738775+0000","last_active":"2024-03-11T19:07:56.672097+0000","last_peered":"2024-03-11T19:07:56.672097+0000","last_clean":"2024-03-11T19:07:56.672097+0000","last_became_active":"2024-03-11T19:07:48.738437+0000","last_became_peered":"2024-03-11T19:07:48.738437+0000","last_unstale":"2024-03-11T19:07:56.672097+0000","last_undegraded":"2024-03-11T19:07:56.672097+0000","last_fullsized":"2024-03-11T19:07:56.672097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:17:20.020756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508324+0000","last_change":"2024-03-11T19:07:49.760293+0000","last_active":"2024-03-11T19:07:51.508324+0000","last_peered":"2024-03-11T19:07:51.508324+0000","last_clean":"2024-03-11T19:07:51.508324+0000","last_became_active":"2024-03-11T19:07:49.759977+0000","last_became_peered":"2024-03-11T19:07:49.759977+0000","last_unstale":"2024-03-11T19:07:51.508324+0000","last_undegraded":"2024-03-11T19:07:51.508324+0000","last_fullsized":"2024-03-11T19:07:51.508324+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:07:44.726756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.763455+0000","last_change":"2024-03-11T19:07:48.740672+0000","last_active":"2024-03-11T19:07:52.763455+0000","last_peered":"2024-03-11T19:07:52.763455+0000","last_clean":"2024-03-11T19:07:52.763455+0000","last_became_active":"2024-03-11T19:07:48.739331+0000","last_became_peered":"2024-03-11T19:07:48.739331+0000","last_unstale":"2024-03-11T19:07:52.763455+0000","last_undegraded":"2024-03-11T19:07:52.763455+0000","last_fullsized":"2024-03-11T19:07:52.763455+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:19:28.375515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739464+0000","last_change":"2024-03-11T19:07:49.742780+0000","last_active":"2024-03-11T19:07:50.739464+0000","last_peered":"2024-03-11T19:07:50.739464+0000","last_clean":"2024-03-11T19:07:50.739464+0000","last_became_active":"2024-03-11T19:07:49.742442+0000","last_became_peered":"2024-03-11T19:07:49.742442+0000","last_unstale":"2024-03-11T19:07:50.739464+0000","last_undegraded":"2024-03-11T19:07:50.739464+0000","last_fullsized":"2024-03-11T19:07:50.739464+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:11:13.838225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509039+0000","last_change":"2024-03-11T19:07:48.752617+0000","last_active":"2024-03-11T19:07:51.509039+0000","last_peered":"2024-03-11T19:07:51.509039+0000","last_clean":"2024-03-11T19:07:51.509039+0000","last_became_active":"2024-03-11T19:07:48.752181+0000","last_became_peered":"2024-03-11T19:07:48.752181+0000","last_unstale":"2024-03-11T19:07:51.509039+0000","last_undegraded":"2024-03-11T19:07:51.509039+0000","last_fullsized":"2024-03-11T19:07:51.509039+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:05:37.162866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508449+0000","last_change":"2024-03-11T19:07:49.753672+0000","last_active":"2024-03-11T19:07:51.508449+0000","last_peered":"2024-03-11T19:07:51.508449+0000","last_clean":"2024-03-11T19:07:51.508449+0000","last_became_active":"2024-03-11T19:07:49.753315+0000","last_became_peered":"2024-03-11T19:07:49.753315+0000","last_unstale":"2024-03-11T19:07:51.508449+0000","last_undegraded":"2024-03-11T19:07:51.508449+0000","last_fullsized":"2024-03-11T19:07:51.508449+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:30:58.503249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'21","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:56.897333+0000","last_change":"2024-03-11T19:07:48.740424+0000","last_active":"2024-03-11T19:07:56.897333+0000","last_peered":"2024-03-11T19:07:56.897333+0000","last_clean":"2024-03-11T19:07:56.897333+0000","last_became_active":"2024-03-11T19:07:48.740077+0000","last_became_peered":"2024-03-11T19:07:48.740077+0000","last_unstale":"2024-03-11T19:07:56.897333+0000","last_undegraded":"2024-03-11T19:07:56.897333+0000","last_fullsized":"2024-03-11T19:07:56.897333+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:30:29.193861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38193,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":22,"num_write":23,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739212+0000","last_change":"2024-03-11T19:07:49.746440+0000","last_active":"2024-03-11T19:07:50.739212+0000","last_peered":"2024-03-11T19:07:50.739212+0000","last_clean":"2024-03-11T19:07:50.739212+0000","last_became_active":"2024-03-11T19:07:49.746106+0000","last_became_peered":"2024-03-11T19:07:49.746106+0000","last_unstale":"2024-03-11T19:07:50.739212+0000","last_undegraded":"2024-03-11T19:07:50.739212+0000","last_fullsized":"2024-03-11T19:07:50.739212+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:44:58.039917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.765082+0000","last_change":"2024-03-11T19:07:48.744437+0000","last_active":"2024-03-11T19:07:52.765082+0000","last_peered":"2024-03-11T19:07:52.765082+0000","last_clean":"2024-03-11T19:07:52.765082+0000","last_became_active":"2024-03-11T19:07:48.744142+0000","last_became_peered":"2024-03-11T19:07:48.744142+0000","last_unstale":"2024-03-11T19:07:52.765082+0000","last_undegraded":"2024-03-11T19:07:52.765082+0000","last_fullsized":"2024-03-11T19:07:52.765082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:32:09.076975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.507894+0000","last_change":"2024-03-11T19:07:49.751191+0000","last_active":"2024-03-11T19:07:51.507894+0000","last_peered":"2024-03-11T19:07:51.507894+0000","last_clean":"2024-03-11T19:07:51.507894+0000","last_became_active":"2024-03-11T19:07:49.750849+0000","last_became_peered":"2024-03-11T19:07:49.750849+0000","last_unstale":"2024-03-11T19:07:51.507894+0000","last_undegraded":"2024-03-11T19:07:51.507894+0000","last_fullsized":"2024-03-11T19:07:51.507894+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:19:18.416905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.767963+0000","last_change":"2024-03-11T19:07:48.742808+0000","last_active":"2024-03-11T19:07:52.767963+0000","last_peered":"2024-03-11T19:07:52.767963+0000","last_clean":"2024-03-11T19:07:52.767963+0000","last_became_active":"2024-03-11T19:07:48.742462+0000","last_became_peered":"2024-03-11T19:07:48.742462+0000","last_unstale":"2024-03-11T19:07:52.767963+0000","last_undegraded":"2024-03-11T19:07:52.767963+0000","last_fullsized":"2024-03-11T19:07:52.767963+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:19.109299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739501+0000","last_change":"2024-03-11T19:07:49.744673+0000","last_active":"2024-03-11T19:07:50.739501+0000","last_peered":"2024-03-11T19:07:50.739501+0000","last_clean":"2024-03-11T19:07:50.739501+0000","last_became_active":"2024-03-11T19:07:49.744313+0000","last_became_peered":"2024-03-11T19:07:49.744313+0000","last_unstale":"2024-03-11T19:07:50.739501+0000","last_undegraded":"2024-03-11T19:07:50.739501+0000","last_fullsized":"2024-03-11T19:07:50.739501+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:21:04.759356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:53.774596+0000","last_change":"2024-03-11T19:07:48.736024+0000","last_active":"2024-03-11T19:07:53.774596+0000","last_peered":"2024-03-11T19:07:53.774596+0000","last_clean":"2024-03-11T19:07:53.774596+0000","last_became_active":"2024-03-11T19:07:48.735720+0000","last_became_peered":"2024-03-11T19:07:48.735720+0000","last_unstale":"2024-03-11T19:07:53.774596+0000","last_undegraded":"2024-03-11T19:07:53.774596+0000","last_fullsized":"2024-03-11T19:07:53.774596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:04:57.386267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739051+0000","last_change":"2024-03-11T19:07:49.742924+0000","last_active":"2024-03-11T19:07:50.739051+0000","last_peered":"2024-03-11T19:07:50.739051+0000","last_clean":"2024-03-11T19:07:50.739051+0000","last_became_active":"2024-03-11T19:07:49.742542+0000","last_became_peered":"2024-03-11T19:07:49.742542+0000","last_unstale":"2024-03-11T19:07:50.739051+0000","last_undegraded":"2024-03-11T19:07:50.739051+0000","last_fullsized":"2024-03-11T19:07:50.739051+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:08:11.944311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'16","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:53.774362+0000","last_change":"2024-03-11T19:07:48.753113+0000","last_active":"2024-03-11T19:07:53.774362+0000","last_peered":"2024-03-11T19:07:53.774362+0000","last_clean":"2024-03-11T19:07:53.774362+0000","last_became_active":"2024-03-11T19:07:48.752517+0000","last_became_peered":"2024-03-11T19:07:48.752517+0000","last_unstale":"2024-03-11T19:07:53.774362+0000","last_undegraded":"2024-03-11T19:07:53.774362+0000","last_fullsized":"2024-03-11T19:07:53.774362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:48:33.441125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509767+0000","last_change":"2024-03-11T19:07:49.748026+0000","last_active":"2024-03-11T19:07:51.509767+0000","last_peered":"2024-03-11T19:07:51.509767+0000","last_clean":"2024-03-11T19:07:51.509767+0000","last_became_active":"2024-03-11T19:07:49.747685+0000","last_became_peered":"2024-03-11T19:07:49.747685+0000","last_unstale":"2024-03-11T19:07:51.509767+0000","last_undegraded":"2024-03-11T19:07:51.509767+0000","last_fullsized":"2024-03-11T19:07:51.509767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:54:40.337723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739183+0000","last_change":"2024-03-11T19:07:48.739571+0000","last_active":"2024-03-11T19:07:50.739183+0000","last_peered":"2024-03-11T19:07:50.739183+0000","last_clean":"2024-03-11T19:07:50.739183+0000","last_became_active":"2024-03-11T19:07:48.739159+0000","last_became_peered":"2024-03-11T19:07:48.739159+0000","last_unstale":"2024-03-11T19:07:50.739183+0000","last_undegraded":"2024-03-11T19:07:50.739183+0000","last_fullsized":"2024-03-11T19:07:50.739183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:46:43.474591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508345+0000","last_change":"2024-03-11T19:07:49.744760+0000","last_active":"2024-03-11T19:07:51.508345+0000","last_peered":"2024-03-11T19:07:51.508345+0000","last_clean":"2024-03-11T19:07:51.508345+0000","last_became_active":"2024-03-11T19:07:49.744265+0000","last_became_peered":"2024-03-11T19:07:49.744265+0000","last_unstale":"2024-03-11T19:07:51.508345+0000","last_undegraded":"2024-03-11T19:07:51.508345+0000","last_fullsized":"2024-03-11T19:07:51.508345+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:09:49.823462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.502145+0000","last_change":"2024-03-11T19:07:48.740223+0000","last_active":"2024-03-11T19:07:51.502145+0000","last_peered":"2024-03-11T19:07:51.502145+0000","last_clean":"2024-03-11T19:07:51.502145+0000","last_became_active":"2024-03-11T19:07:48.739569+0000","last_became_peered":"2024-03-11T19:07:48.739569+0000","last_unstale":"2024-03-11T19:07:51.502145+0000","last_undegraded":"2024-03-11T19:07:51.502145+0000","last_fullsized":"2024-03-11T19:07:51.502145+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:04:19.384523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.573850+0000","last_change":"2024-03-11T19:07:49.755620+0000","last_active":"2024-03-11T19:07:52.573850+0000","last_peered":"2024-03-11T19:07:52.573850+0000","last_clean":"2024-03-11T19:07:52.573850+0000","last_became_active":"2024-03-11T19:07:49.750886+0000","last_became_peered":"2024-03-11T19:07:49.750886+0000","last_unstale":"2024-03-11T19:07:52.573850+0000","last_undegraded":"2024-03-11T19:07:52.573850+0000","last_fullsized":"2024-03-11T19:07:52.573850+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:35:54.191061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739273+0000","last_change":"2024-03-11T19:07:48.743340+0000","last_active":"2024-03-11T19:07:50.739273+0000","last_peered":"2024-03-11T19:07:50.739273+0000","last_clean":"2024-03-11T19:07:50.739273+0000","last_became_active":"2024-03-11T19:07:48.742947+0000","last_became_peered":"2024-03-11T19:07:48.742947+0000","last_unstale":"2024-03-11T19:07:50.739273+0000","last_undegraded":"2024-03-11T19:07:50.739273+0000","last_fullsized":"2024-03-11T19:07:50.739273+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:40:25.233150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508305+0000","last_change":"2024-03-11T19:07:49.753668+0000","last_active":"2024-03-11T19:07:51.508305+0000","last_peered":"2024-03-11T19:07:51.508305+0000","last_clean":"2024-03-11T19:07:51.508305+0000","last_became_active":"2024-03-11T19:07:49.750574+0000","last_became_peered":"2024-03-11T19:07:49.750574+0000","last_unstale":"2024-03-11T19:07:51.508305+0000","last_undegraded":"2024-03-11T19:07:51.508305+0000","last_fullsized":"2024-03-11T19:07:51.508305+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:12:32.711847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509695+0000","last_change":"2024-03-11T19:07:49.739538+0000","last_active":"2024-03-11T19:07:51.509695+0000","last_peered":"2024-03-11T19:07:51.509695+0000","last_clean":"2024-03-11T19:07:51.509695+0000","last_became_active":"2024-03-11T19:07:49.738617+0000","last_became_peered":"2024-03-11T19:07:49.738617+0000","last_unstale":"2024-03-11T19:07:51.509695+0000","last_undegraded":"2024-03-11T19:07:51.509695+0000","last_fullsized":"2024-03-11T19:07:51.509695+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:43:08.697125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:53.773088+0000","last_change":"2024-03-11T19:07:48.749527+0000","last_active":"2024-03-11T19:07:53.773088+0000","last_peered":"2024-03-11T19:07:53.773088+0000","last_clean":"2024-03-11T19:07:53.773088+0000","last_became_active":"2024-03-11T19:07:48.749026+0000","last_became_peered":"2024-03-11T19:07:48.749026+0000","last_unstale":"2024-03-11T19:07:53.773088+0000","last_undegraded":"2024-03-11T19:07:53.773088+0000","last_fullsized":"2024-03-11T19:07:53.773088+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:20:19.058429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.743699+0000","last_change":"2024-03-11T19:07:49.757696+0000","last_active":"2024-03-11T19:07:50.743699+0000","last_peered":"2024-03-11T19:07:50.743699+0000","last_clean":"2024-03-11T19:07:50.743699+0000","last_became_active":"2024-03-11T19:07:49.757380+0000","last_became_peered":"2024-03-11T19:07:49.757380+0000","last_unstale":"2024-03-11T19:07:50.743699+0000","last_undegraded":"2024-03-11T19:07:50.743699+0000","last_fullsized":"2024-03-11T19:07:50.743699+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:03:22.954619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.764299+0000","last_change":"2024-03-11T19:07:48.737504+0000","last_active":"2024-03-11T19:07:52.764299+0000","last_peered":"2024-03-11T19:07:52.764299+0000","last_clean":"2024-03-11T19:07:52.764299+0000","last_became_active":"2024-03-11T19:07:48.737147+0000","last_became_peered":"2024-03-11T19:07:48.737147+0000","last_unstale":"2024-03-11T19:07:52.764299+0000","last_undegraded":"2024-03-11T19:07:52.764299+0000","last_fullsized":"2024-03-11T19:07:52.764299+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:04.819617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":339,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739051+0000","last_change":"2024-03-11T19:06:35.035452+0000","last_active":"2024-03-11T19:07:50.739051+0000","last_peered":"2024-03-11T19:07:50.739051+0000","last_clean":"2024-03-11T19:07:50.739051+0000","last_became_active":"2024-03-11T19:06:35.035076+0000","last_became_peered":"2024-03-11T19:06:35.035076+0000","last_unstale":"2024-03-11T19:07:50.739051+0000","last_undegraded":"2024-03-11T19:07:50.739051+0000","last_fullsized":"2024-03-11T19:07:50.739051+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:03:10.143002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:03:10.143002+0000","last_clean_scrub_stamp":"2024-03-11T19:03:10.143002+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:39:37.348091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.507654+0000","last_change":"2024-03-11T19:07:49.743000+0000","last_active":"2024-03-11T19:07:51.507654+0000","last_peered":"2024-03-11T19:07:51.507654+0000","last_clean":"2024-03-11T19:07:51.507654+0000","last_became_active":"2024-03-11T19:07:49.742691+0000","last_became_peered":"2024-03-11T19:07:49.742691+0000","last_unstale":"2024-03-11T19:07:51.507654+0000","last_undegraded":"2024-03-11T19:07:51.507654+0000","last_fullsized":"2024-03-11T19:07:51.507654+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:40:21.519929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:53.772503+0000","last_change":"2024-03-11T19:07:48.739038+0000","last_active":"2024-03-11T19:07:53.772503+0000","last_peered":"2024-03-11T19:07:53.772503+0000","last_clean":"2024-03-11T19:07:53.772503+0000","last_became_active":"2024-03-11T19:07:48.738714+0000","last_became_peered":"2024-03-11T19:07:48.738714+0000","last_unstale":"2024-03-11T19:07:53.772503+0000","last_undegraded":"2024-03-11T19:07:53.772503+0000","last_fullsized":"2024-03-11T19:07:53.772503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:17:43.086679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:54.782084+0000","last_change":"2024-03-11T19:07:48.752699+0000","last_active":"2024-03-11T19:07:54.782084+0000","last_peered":"2024-03-11T19:07:54.782084+0000","last_clean":"2024-03-11T19:07:54.782084+0000","last_became_active":"2024-03-11T19:07:48.752331+0000","last_became_peered":"2024-03-11T19:07:48.752331+0000","last_unstale":"2024-03-11T19:07:54.782084+0000","last_undegraded":"2024-03-11T19:07:54.782084+0000","last_fullsized":"2024-03-11T19:07:54.782084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:03.912256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508129+0000","last_change":"2024-03-11T19:07:49.742434+0000","last_active":"2024-03-11T19:07:51.508129+0000","last_peered":"2024-03-11T19:07:51.508129+0000","last_clean":"2024-03-11T19:07:51.508129+0000","last_became_active":"2024-03-11T19:07:49.742048+0000","last_became_peered":"2024-03-11T19:07:49.742048+0000","last_unstale":"2024-03-11T19:07:51.508129+0000","last_undegraded":"2024-03-11T19:07:51.508129+0000","last_fullsized":"2024-03-11T19:07:51.508129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:39:22.685055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'7","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:54.784180+0000","last_change":"2024-03-11T19:07:48.744728+0000","last_active":"2024-03-11T19:07:54.784180+0000","last_peered":"2024-03-11T19:07:54.784180+0000","last_clean":"2024-03-11T19:07:54.784180+0000","last_became_active":"2024-03-11T19:07:48.744378+0000","last_became_peered":"2024-03-11T19:07:48.744378+0000","last_unstale":"2024-03-11T19:07:54.784180+0000","last_undegraded":"2024-03-11T19:07:54.784180+0000","last_fullsized":"2024-03-11T19:07:54.784180+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:42.823040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.743742+0000","last_change":"2024-03-11T19:07:49.744261+0000","last_active":"2024-03-11T19:07:50.743742+0000","last_peered":"2024-03-11T19:07:50.743742+0000","last_clean":"2024-03-11T19:07:50.743742+0000","last_became_active":"2024-03-11T19:07:49.743946+0000","last_became_peered":"2024-03-11T19:07:49.743946+0000","last_unstale":"2024-03-11T19:07:50.743742+0000","last_undegraded":"2024-03-11T19:07:50.743742+0000","last_fullsized":"2024-03-11T19:07:50.743742+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:54:07.928899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.763246+0000","last_change":"2024-03-11T19:07:48.740059+0000","last_active":"2024-03-11T19:07:52.763246+0000","last_peered":"2024-03-11T19:07:52.763246+0000","last_clean":"2024-03-11T19:07:52.763246+0000","last_became_active":"2024-03-11T19:07:48.739626+0000","last_became_peered":"2024-03-11T19:07:48.739626+0000","last_unstale":"2024-03-11T19:07:52.763246+0000","last_undegraded":"2024-03-11T19:07:52.763246+0000","last_fullsized":"2024-03-11T19:07:52.763246+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:45:12.219203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739482+0000","last_change":"2024-03-11T19:07:49.746041+0000","last_active":"2024-03-11T19:07:50.739482+0000","last_peered":"2024-03-11T19:07:50.739482+0000","last_clean":"2024-03-11T19:07:50.739482+0000","last_became_active":"2024-03-11T19:07:49.745602+0000","last_became_peered":"2024-03-11T19:07:49.745602+0000","last_unstale":"2024-03-11T19:07:50.739482+0000","last_undegraded":"2024-03-11T19:07:50.739482+0000","last_fullsized":"2024-03-11T19:07:50.739482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:23:44.596929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.763882+0000","last_change":"2024-03-11T19:07:48.748677+0000","last_active":"2024-03-11T19:07:52.763882+0000","last_peered":"2024-03-11T19:07:52.763882+0000","last_clean":"2024-03-11T19:07:52.763882+0000","last_became_active":"2024-03-11T19:07:48.748470+0000","last_became_peered":"2024-03-11T19:07:48.748470+0000","last_unstale":"2024-03-11T19:07:52.763882+0000","last_undegraded":"2024-03-11T19:07:52.763882+0000","last_fullsized":"2024-03-11T19:07:52.763882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:24:55.240722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.738941+0000","last_change":"2024-03-11T19:07:49.746522+0000","last_active":"2024-03-11T19:07:50.738941+0000","last_peered":"2024-03-11T19:07:50.738941+0000","last_clean":"2024-03-11T19:07:50.738941+0000","last_became_active":"2024-03-11T19:07:49.746179+0000","last_became_peered":"2024-03-11T19:07:49.746179+0000","last_unstale":"2024-03-11T19:07:50.738941+0000","last_undegraded":"2024-03-11T19:07:50.738941+0000","last_fullsized":"2024-03-11T19:07:50.738941+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:06:08.144850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.766543+0000","last_change":"2024-03-11T19:07:48.740926+0000","last_active":"2024-03-11T19:07:52.766543+0000","last_peered":"2024-03-11T19:07:52.766543+0000","last_clean":"2024-03-11T19:07:52.766543+0000","last_became_active":"2024-03-11T19:07:48.740625+0000","last_became_peered":"2024-03-11T19:07:48.740625+0000","last_unstale":"2024-03-11T19:07:52.766543+0000","last_undegraded":"2024-03-11T19:07:52.766543+0000","last_fullsized":"2024-03-11T19:07:52.766543+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:50:13.079608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508081+0000","last_change":"2024-03-11T19:07:49.742428+0000","last_active":"2024-03-11T19:07:51.508081+0000","last_peered":"2024-03-11T19:07:51.508081+0000","last_clean":"2024-03-11T19:07:51.508081+0000","last_became_active":"2024-03-11T19:07:49.742098+0000","last_became_peered":"2024-03-11T19:07:49.742098+0000","last_unstale":"2024-03-11T19:07:51.508081+0000","last_undegraded":"2024-03-11T19:07:51.508081+0000","last_fullsized":"2024-03-11T19:07:51.508081+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:19:24.615755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508717+0000","last_change":"2024-03-11T19:07:48.745092+0000","last_active":"2024-03-11T19:07:51.508717+0000","last_peered":"2024-03-11T19:07:51.508717+0000","last_clean":"2024-03-11T19:07:51.508717+0000","last_became_active":"2024-03-11T19:07:48.744714+0000","last_became_peered":"2024-03-11T19:07:48.744714+0000","last_unstale":"2024-03-11T19:07:51.508717+0000","last_undegraded":"2024-03-11T19:07:51.508717+0000","last_fullsized":"2024-03-11T19:07:51.508717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:10:48.274993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.743920+0000","last_change":"2024-03-11T19:07:49.754872+0000","last_active":"2024-03-11T19:07:50.743920+0000","last_peered":"2024-03-11T19:07:50.743920+0000","last_clean":"2024-03-11T19:07:50.743920+0000","last_became_active":"2024-03-11T19:07:49.749663+0000","last_became_peered":"2024-03-11T19:07:49.749663+0000","last_unstale":"2024-03-11T19:07:50.743920+0000","last_undegraded":"2024-03-11T19:07:50.743920+0000","last_fullsized":"2024-03-11T19:07:50.743920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:22:23.664814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.763388+0000","last_change":"2024-03-11T19:07:48.742227+0000","last_active":"2024-03-11T19:07:52.763388+0000","last_peered":"2024-03-11T19:07:52.763388+0000","last_clean":"2024-03-11T19:07:52.763388+0000","last_became_active":"2024-03-11T19:07:48.741912+0000","last_became_peered":"2024-03-11T19:07:48.741912+0000","last_unstale":"2024-03-11T19:07:52.763388+0000","last_undegraded":"2024-03-11T19:07:52.763388+0000","last_fullsized":"2024-03-11T19:07:52.763388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:10:06.120433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739388+0000","last_change":"2024-03-11T19:07:49.748947+0000","last_active":"2024-03-11T19:07:50.739388+0000","last_peered":"2024-03-11T19:07:50.739388+0000","last_clean":"2024-03-11T19:07:50.739388+0000","last_became_active":"2024-03-11T19:07:49.748621+0000","last_became_peered":"2024-03-11T19:07:49.748621+0000","last_unstale":"2024-03-11T19:07:50.739388+0000","last_undegraded":"2024-03-11T19:07:50.739388+0000","last_fullsized":"2024-03-11T19:07:50.739388+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:19:53.824590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509318+0000","last_change":"2024-03-11T19:07:48.748799+0000","last_active":"2024-03-11T19:07:51.509318+0000","last_peered":"2024-03-11T19:07:51.509318+0000","last_clean":"2024-03-11T19:07:51.509318+0000","last_became_active":"2024-03-11T19:07:48.748434+0000","last_became_peered":"2024-03-11T19:07:48.748434+0000","last_unstale":"2024-03-11T19:07:51.509318+0000","last_undegraded":"2024-03-11T19:07:51.509318+0000","last_fullsized":"2024-03-11T19:07:51.509318+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:10:09.836926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739740+0000","last_change":"2024-03-11T19:07:49.750122+0000","last_active":"2024-03-11T19:07:50.739740+0000","last_peered":"2024-03-11T19:07:50.739740+0000","last_clean":"2024-03-11T19:07:50.739740+0000","last_became_active":"2024-03-11T19:07:49.749801+0000","last_became_peered":"2024-03-11T19:07:49.749801+0000","last_unstale":"2024-03-11T19:07:50.739740+0000","last_undegraded":"2024-03-11T19:07:50.739740+0000","last_fullsized":"2024-03-11T19:07:50.739740+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:58:31.155275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508847+0000","last_change":"2024-03-11T19:07:48.741959+0000","last_active":"2024-03-11T19:07:51.508847+0000","last_peered":"2024-03-11T19:07:51.508847+0000","last_clean":"2024-03-11T19:07:51.508847+0000","last_became_active":"2024-03-11T19:07:48.741630+0000","last_became_peered":"2024-03-11T19:07:48.741630+0000","last_unstale":"2024-03-11T19:07:51.508847+0000","last_undegraded":"2024-03-11T19:07:51.508847+0000","last_fullsized":"2024-03-11T19:07:51.508847+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:27:08.606055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739222+0000","last_change":"2024-03-11T19:07:49.729838+0000","last_active":"2024-03-11T19:07:50.739222+0000","last_peered":"2024-03-11T19:07:50.739222+0000","last_clean":"2024-03-11T19:07:50.739222+0000","last_became_active":"2024-03-11T19:07:49.728912+0000","last_became_peered":"2024-03-11T19:07:49.728912+0000","last_unstale":"2024-03-11T19:07:50.739222+0000","last_undegraded":"2024-03-11T19:07:50.739222+0000","last_fullsized":"2024-03-11T19:07:50.739222+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:02:06.551734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508026+0000","last_change":"2024-03-11T19:07:48.744709+0000","last_active":"2024-03-11T19:07:51.508026+0000","last_peered":"2024-03-11T19:07:51.508026+0000","last_clean":"2024-03-11T19:07:51.508026+0000","last_became_active":"2024-03-11T19:07:48.744356+0000","last_became_peered":"2024-03-11T19:07:48.744356+0000","last_unstale":"2024-03-11T19:07:51.508026+0000","last_undegraded":"2024-03-11T19:07:51.508026+0000","last_fullsized":"2024-03-11T19:07:51.508026+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:48:47.098443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739422+0000","last_change":"2024-03-11T19:07:49.743288+0000","last_active":"2024-03-11T19:07:50.739422+0000","last_peered":"2024-03-11T19:07:50.739422+0000","last_clean":"2024-03-11T19:07:50.739422+0000","last_became_active":"2024-03-11T19:07:49.742942+0000","last_became_peered":"2024-03-11T19:07:49.742942+0000","last_unstale":"2024-03-11T19:07:50.739422+0000","last_undegraded":"2024-03-11T19:07:50.739422+0000","last_fullsized":"2024-03-11T19:07:50.739422+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:21:53.896192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508279+0000","last_change":"2024-03-11T19:07:49.741610+0000","last_active":"2024-03-11T19:07:51.508279+0000","last_peered":"2024-03-11T19:07:51.508279+0000","last_clean":"2024-03-11T19:07:51.508279+0000","last_became_active":"2024-03-11T19:07:49.741234+0000","last_became_peered":"2024-03-11T19:07:49.741234+0000","last_unstale":"2024-03-11T19:07:51.508279+0000","last_undegraded":"2024-03-11T19:07:51.508279+0000","last_fullsized":"2024-03-11T19:07:51.508279+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:53:03.320093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508376+0000","last_change":"2024-03-11T19:07:49.739625+0000","last_active":"2024-03-11T19:07:51.508376+0000","last_peered":"2024-03-11T19:07:51.508376+0000","last_clean":"2024-03-11T19:07:51.508376+0000","last_became_active":"2024-03-11T19:07:49.739283+0000","last_became_peered":"2024-03-11T19:07:49.739283+0000","last_unstale":"2024-03-11T19:07:51.508376+0000","last_undegraded":"2024-03-11T19:07:51.508376+0000","last_fullsized":"2024-03-11T19:07:51.508376+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:31:45.506952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508657+0000","last_change":"2024-03-11T19:07:49.745928+0000","last_active":"2024-03-11T19:07:51.508657+0000","last_peered":"2024-03-11T19:07:51.508657+0000","last_clean":"2024-03-11T19:07:51.508657+0000","last_became_active":"2024-03-11T19:07:49.745025+0000","last_became_peered":"2024-03-11T19:07:49.745025+0000","last_unstale":"2024-03-11T19:07:51.508657+0000","last_undegraded":"2024-03-11T19:07:51.508657+0000","last_fullsized":"2024-03-11T19:07:51.508657+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:40:30.141906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739082+0000","last_change":"2024-03-11T19:07:49.756274+0000","last_active":"2024-03-11T19:07:50.739082+0000","last_peered":"2024-03-11T19:07:50.739082+0000","last_clean":"2024-03-11T19:07:50.739082+0000","last_became_active":"2024-03-11T19:07:49.755954+0000","last_became_peered":"2024-03-11T19:07:49.755954+0000","last_unstale":"2024-03-11T19:07:50.739082+0000","last_undegraded":"2024-03-11T19:07:50.739082+0000","last_fullsized":"2024-03-11T19:07:50.739082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:15:36.825240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.507951+0000","last_change":"2024-03-11T19:07:49.752609+0000","last_active":"2024-03-11T19:07:51.507951+0000","last_peered":"2024-03-11T19:07:51.507951+0000","last_clean":"2024-03-11T19:07:51.507951+0000","last_became_active":"2024-03-11T19:07:49.752164+0000","last_became_peered":"2024-03-11T19:07:49.752164+0000","last_unstale":"2024-03-11T19:07:51.507951+0000","last_undegraded":"2024-03-11T19:07:51.507951+0000","last_fullsized":"2024-03-11T19:07:51.507951+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:56:49.705582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508382+0000","last_change":"2024-03-11T19:07:49.743963+0000","last_active":"2024-03-11T19:07:51.508382+0000","last_peered":"2024-03-11T19:07:51.508382+0000","last_clean":"2024-03-11T19:07:51.508382+0000","last_became_active":"2024-03-11T19:07:49.743568+0000","last_became_peered":"2024-03-11T19:07:49.743568+0000","last_unstale":"2024-03-11T19:07:51.508382+0000","last_undegraded":"2024-03-11T19:07:51.508382+0000","last_fullsized":"2024-03-11T19:07:51.508382+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:45:28.970286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509675+0000","last_change":"2024-03-11T19:07:49.750677+0000","last_active":"2024-03-11T19:07:51.509675+0000","last_peered":"2024-03-11T19:07:51.509675+0000","last_clean":"2024-03-11T19:07:51.509675+0000","last_became_active":"2024-03-11T19:07:49.750373+0000","last_became_peered":"2024-03-11T19:07:49.750373+0000","last_unstale":"2024-03-11T19:07:51.509675+0000","last_undegraded":"2024-03-11T19:07:51.509675+0000","last_fullsized":"2024-03-11T19:07:51.509675+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:07:28.767502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.743570+0000","last_change":"2024-03-11T19:07:49.745278+0000","last_active":"2024-03-11T19:07:50.743570+0000","last_peered":"2024-03-11T19:07:50.743570+0000","last_clean":"2024-03-11T19:07:50.743570+0000","last_became_active":"2024-03-11T19:07:49.744998+0000","last_became_peered":"2024-03-11T19:07:49.744998+0000","last_unstale":"2024-03-11T19:07:50.743570+0000","last_undegraded":"2024-03-11T19:07:50.743570+0000","last_fullsized":"2024-03-11T19:07:50.743570+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:23.778051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509608+0000","last_change":"2024-03-11T19:07:49.734318+0000","last_active":"2024-03-11T19:07:51.509608+0000","last_peered":"2024-03-11T19:07:51.509608+0000","last_clean":"2024-03-11T19:07:51.509608+0000","last_became_active":"2024-03-11T19:07:49.733998+0000","last_became_peered":"2024-03-11T19:07:49.733998+0000","last_unstale":"2024-03-11T19:07:51.509608+0000","last_undegraded":"2024-03-11T19:07:51.509608+0000","last_fullsized":"2024-03-11T19:07:51.509608+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:48:50.308499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.743448+0000","last_change":"2024-03-11T19:07:49.741009+0000","last_active":"2024-03-11T19:07:50.743448+0000","last_peered":"2024-03-11T19:07:50.743448+0000","last_clean":"2024-03-11T19:07:50.743448+0000","last_became_active":"2024-03-11T19:07:49.740094+0000","last_became_peered":"2024-03-11T19:07:49.740094+0000","last_unstale":"2024-03-11T19:07:50.743448+0000","last_undegraded":"2024-03-11T19:07:50.743448+0000","last_fullsized":"2024-03-11T19:07:50.743448+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:06:31.945387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508055+0000","last_change":"2024-03-11T19:07:49.759300+0000","last_active":"2024-03-11T19:07:51.508055+0000","last_peered":"2024-03-11T19:07:51.508055+0000","last_clean":"2024-03-11T19:07:51.508055+0000","last_became_active":"2024-03-11T19:07:49.758011+0000","last_became_peered":"2024-03-11T19:07:49.758011+0000","last_unstale":"2024-03-11T19:07:51.508055+0000","last_undegraded":"2024-03-11T19:07:51.508055+0000","last_fullsized":"2024-03-11T19:07:51.508055+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:03:40.651816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508121+0000","last_change":"2024-03-11T19:07:49.752905+0000","last_active":"2024-03-11T19:07:51.508121+0000","last_peered":"2024-03-11T19:07:51.508121+0000","last_clean":"2024-03-11T19:07:51.508121+0000","last_became_active":"2024-03-11T19:07:49.752383+0000","last_became_peered":"2024-03-11T19:07:49.752383+0000","last_unstale":"2024-03-11T19:07:51.508121+0000","last_undegraded":"2024-03-11T19:07:51.508121+0000","last_fullsized":"2024-03-11T19:07:51.508121+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:45:33.475526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508665+0000","last_change":"2024-03-11T19:07:49.744957+0000","last_active":"2024-03-11T19:07:51.508665+0000","last_peered":"2024-03-11T19:07:51.508665+0000","last_clean":"2024-03-11T19:07:51.508665+0000","last_became_active":"2024-03-11T19:07:49.744627+0000","last_became_peered":"2024-03-11T19:07:49.744627+0000","last_unstale":"2024-03-11T19:07:51.508665+0000","last_undegraded":"2024-03-11T19:07:51.508665+0000","last_fullsized":"2024-03-11T19:07:51.508665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:27:47.817004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245707,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":41401,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":6,"num_read_kb":24,"num_write":154,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":53297,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318286,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":933888,"data_stored":446066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579923,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1076480,"data_compressed":37658,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743449,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":446974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005086,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":446804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266723,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":1038898,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430248,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36248,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708904,"statfs":{"total":95995035648,"available":95957917696,"internally_reserved":0,"allocated":933888,"data_stored":446066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":946176,"data_stored":447678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":974848,"data_stored":448244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":950272,"data_stored":447712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36268,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708884,"statfs":{"total":95995035648,"available":95957897216,"internally_reserved":0,"allocated":942080,"data_stored":446170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888114,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36252,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708900,"statfs":{"total":95995035648,"available":95957913600,"internally_reserved":0,"allocated":937984,"data_stored":446088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691885,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":1037910,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":39929,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T19:07:58.262 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-11T19:07:58.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:57 smithi190 ceph-mon[36292]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-11T19:07:58.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:57 smithi190 ceph-mon[36292]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:07:58.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:57 smithi190 ceph-mon[36292]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:07:58.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:57 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3117632599' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:07:58.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.395+0000 7fa1a5b4f640 1 -- 172.21.15.29:0/3057132040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1a0102440 msgr2=0x7fa1a01028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:58.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.395+0000 7fa1a5b4f640 1 --2- 172.21.15.29:0/3057132040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1a0102440 0x7fa1a01028c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa194009900 tx=0x7fa19402bc30 comp rx=0 tx=0).stop 2024-03-11T19:07:58.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.395+0000 7fa1a5b4f640 1 -- 172.21.15.29:0/3057132040 shutdown_connections 2024-03-11T19:07:58.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.395+0000 7fa1a5b4f640 1 --2- 172.21.15.29:0/3057132040 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1a0102e00 0x7fa1a0109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.395+0000 7fa1a5b4f640 1 --2- 172.21.15.29:0/3057132040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1a0102440 0x7fa1a01028c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.395+0000 7fa1a5b4f640 1 --2- 172.21.15.29:0/3057132040 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1a00ff470 0x7fa1a00ff890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.395+0000 7fa1a5b4f640 1 -- 172.21.15.29:0/3057132040 >> 172.21.15.29:0/3057132040 conn(0x7fa1a00fad60 msgr2=0x7fa1a00fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:58.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.395+0000 7fa1a5b4f640 1 -- 172.21.15.29:0/3057132040 shutdown_connections 2024-03-11T19:07:58.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.395+0000 7fa1a5b4f640 1 -- 172.21.15.29:0/3057132040 wait complete. 2024-03-11T19:07:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.396+0000 7fa1a5b4f640 1 Processor -- start 2024-03-11T19:07:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.396+0000 7fa1a5b4f640 1 -- start start 2024-03-11T19:07:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.396+0000 7fa1a5b4f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1a00ff470 0x7fa1a0197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.396+0000 7fa1a5b4f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1a0102440 0x7fa1a01976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.396+0000 7fa1a5b4f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1a0102e00 0x7fa1a019e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.396+0000 7fa1a5b4f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a019ec50 con 0x7fa1a0102440 2024-03-11T19:07:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.396+0000 7fa1a5b4f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a019edc0 con 0x7fa1a0102e00 2024-03-11T19:07:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.396+0000 7fa1a5b4f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a019ef80 con 0x7fa1a00ff470 2024-03-11T19:07:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.396+0000 7fa19effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1a0102440 0x7fa1a01976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.396+0000 7fa19effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1a0102440 0x7fa1a01976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60854/0 (socket says 172.21.15.29:60854) 2024-03-11T19:07:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.396+0000 7fa19effd640 1 -- 172.21.15.29:0/4072247236 learned_addr learned my addr 172.21.15.29:0/4072247236 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:07:58.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.396+0000 7fa19f7fe640 1 --2- 172.21.15.29:0/4072247236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1a00ff470 0x7fa1a0197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:58.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.396+0000 7fa19effd640 1 -- 172.21.15.29:0/4072247236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1a00ff470 msgr2=0x7fa1a0197160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:58.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.396+0000 7fa19effd640 1 --2- 172.21.15.29:0/4072247236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1a00ff470 0x7fa1a0197160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.396+0000 7fa19effd640 1 -- 172.21.15.29:0/4072247236 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1a0102e00 msgr2=0x7fa1a019e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:58.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.396+0000 7fa19effd640 1 --2- 172.21.15.29:0/4072247236 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1a0102e00 0x7fa1a019e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.396+0000 7fa19effd640 1 -- 172.21.15.29:0/4072247236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa194009560 con 0x7fa1a0102440 2024-03-11T19:07:58.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.396+0000 7fa19f7fe640 1 --2- 172.21.15.29:0/4072247236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1a00ff470 0x7fa1a0197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:07:58.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.397+0000 7fa19effd640 1 --2- 172.21.15.29:0/4072247236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1a0102440 0x7fa1a01976a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa194036040 tx=0x7fa1940042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:58.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.397+0000 7fa19cff9640 1 -- 172.21.15.29:0/4072247236 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1940044e0 con 0x7fa1a0102440 2024-03-11T19:07:58.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.397+0000 7fa19cff9640 1 -- 172.21.15.29:0/4072247236 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa194003cb0 con 0x7fa1a0102440 2024-03-11T19:07:58.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.397+0000 7fa1a5b4f640 1 -- 172.21.15.29:0/4072247236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a019f200 con 0x7fa1a0102440 2024-03-11T19:07:58.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.397+0000 7fa19cff9640 1 -- 172.21.15.29:0/4072247236 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19403e450 con 0x7fa1a0102440 2024-03-11T19:07:58.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.397+0000 7fa1a5b4f640 1 -- 172.21.15.29:0/4072247236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a019f710 con 0x7fa1a0102440 2024-03-11T19:07:58.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.398+0000 7fa1a5b4f640 1 -- 172.21.15.29:0/4072247236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a0108660 con 0x7fa1a0102440 2024-03-11T19:07:58.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.399+0000 7fa19cff9640 1 -- 172.21.15.29:0/4072247236 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fa194004e50 con 0x7fa1a0102440 2024-03-11T19:07:58.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.399+0000 7fa19cff9640 1 --2- 172.21.15.29:0/4072247236 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa1880a1ac0 0x7fa1880a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:07:58.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.399+0000 7fa19f7fe640 1 --2- 172.21.15.29:0/4072247236 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa1880a1ac0 0x7fa1880a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:07:58.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.399+0000 7fa19cff9640 1 -- 172.21.15.29:0/4072247236 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa1940ef780 con 0x7fa1a0102440 2024-03-11T19:07:58.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.400+0000 7fa19f7fe640 1 --2- 172.21.15.29:0/4072247236 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa1880a1ac0 0x7fa1880a3f80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0069590 tx=0x7fa190009530 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:07:58.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.404+0000 7fa19cff9640 1 -- 172.21.15.29:0/4072247236 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa1940b8af0 con 0x7fa1a0102440 2024-03-11T19:07:58.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.581+0000 7fa1a5b4f640 1 -- 172.21.15.29:0/4072247236 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa1a002cf10 con 0x7fa1880a1ac0 2024-03-11T19:07:58.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.589+0000 7fa19cff9640 1 -- 172.21.15.29:0/4072247236 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258913 (secure 0 0 0) 0x7fa1a002cf10 con 0x7fa1880a1ac0 2024-03-11T19:07:58.594 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:07:58.595 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-11T19:07:58.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.591+0000 7fa1a5b4f640 1 -- 172.21.15.29:0/4072247236 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa1880a1ac0 msgr2=0x7fa1880a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:58.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.591+0000 7fa1a5b4f640 1 --2- 172.21.15.29:0/4072247236 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa1880a1ac0 0x7fa1880a3f80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0069590 tx=0x7fa190009530 comp rx=0 tx=0).stop 2024-03-11T19:07:58.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.591+0000 7fa1a5b4f640 1 -- 172.21.15.29:0/4072247236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1a0102440 msgr2=0x7fa1a01976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:07:58.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.591+0000 7fa1a5b4f640 1 --2- 172.21.15.29:0/4072247236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1a0102440 0x7fa1a01976a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa194036040 tx=0x7fa1940042f0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.591+0000 7fa1a5b4f640 1 -- 172.21.15.29:0/4072247236 shutdown_connections 2024-03-11T19:07:58.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.591+0000 7fa1a5b4f640 1 --2- 172.21.15.29:0/4072247236 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1a0102e00 0x7fa1a019e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.591+0000 7fa1a5b4f640 1 --2- 172.21.15.29:0/4072247236 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa1880a1ac0 0x7fa1880a3f80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.591+0000 7fa1a5b4f640 1 --2- 172.21.15.29:0/4072247236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1a0102440 0x7fa1a01976a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.591+0000 7fa1a5b4f640 1 --2- 172.21.15.29:0/4072247236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1a00ff470 0x7fa1a0197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:07:58.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.591+0000 7fa1a5b4f640 1 -- 172.21.15.29:0/4072247236 >> 172.21.15.29:0/4072247236 conn(0x7fa1a00fad60 msgr2=0x7fa1a0105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:07:58.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.592+0000 7fa1a5b4f640 1 -- 172.21.15.29:0/4072247236 shutdown_connections 2024-03-11T19:07:58.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:07:58.592+0000 7fa1a5b4f640 1 -- 172.21.15.29:0/4072247236 wait complete. 2024-03-11T19:07:58.605 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-03-11T19:07:57.605536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631886,"num_objects":69,"num_object_clones":0,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":132,"num_read_kb":133,"num_write":336,"num_write_kb":3016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":272,"ondisk_log_size":272,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":389652,"kb_used_data":11356,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124552172,"statfs":{"total":1151940427776,"available":1151541424128,"internally_reserved":0,"allocated":11628544,"data_stored":7175090,"data_compressed":109264,"data_compressed_allocated":139264,"data_compressed_original":1777664,"omap_allocated":19142,"internal_metadata":387233082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41518,"num_objects":67,"num_object_clones":0,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":6,"num_read_kb":24,"num_write":155,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001832"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509020+0000","last_change":"2024-03-11T19:07:49.751738+0000","last_active":"2024-03-11T19:07:51.509020+0000","last_peered":"2024-03-11T19:07:51.509020+0000","last_clean":"2024-03-11T19:07:51.509020+0000","last_became_active":"2024-03-11T19:07:49.751173+0000","last_became_peered":"2024-03-11T19:07:49.751173+0000","last_unstale":"2024-03-11T19:07:51.509020+0000","last_undegraded":"2024-03-11T19:07:51.509020+0000","last_fullsized":"2024-03-11T19:07:51.509020+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:13:40.157763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739294+0000","last_change":"2024-03-11T19:07:49.749069+0000","last_active":"2024-03-11T19:07:50.739294+0000","last_peered":"2024-03-11T19:07:50.739294+0000","last_clean":"2024-03-11T19:07:50.739294+0000","last_became_active":"2024-03-11T19:07:49.747672+0000","last_became_peered":"2024-03-11T19:07:49.747672+0000","last_unstale":"2024-03-11T19:07:50.739294+0000","last_undegraded":"2024-03-11T19:07:50.739294+0000","last_fullsized":"2024-03-11T19:07:50.739294+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:57:26.379836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739214+0000","last_change":"2024-03-11T19:07:49.750298+0000","last_active":"2024-03-11T19:07:50.739214+0000","last_peered":"2024-03-11T19:07:50.739214+0000","last_clean":"2024-03-11T19:07:50.739214+0000","last_became_active":"2024-03-11T19:07:49.749985+0000","last_became_peered":"2024-03-11T19:07:49.749985+0000","last_unstale":"2024-03-11T19:07:50.739214+0000","last_undegraded":"2024-03-11T19:07:50.739214+0000","last_fullsized":"2024-03-11T19:07:50.739214+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:56.579872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.441461+0000","last_change":"2024-03-11T19:07:49.745246+0000","last_active":"2024-03-11T19:07:51.441461+0000","last_peered":"2024-03-11T19:07:51.441461+0000","last_clean":"2024-03-11T19:07:51.441461+0000","last_became_active":"2024-03-11T19:07:49.744907+0000","last_became_peered":"2024-03-11T19:07:49.744907+0000","last_unstale":"2024-03-11T19:07:51.441461+0000","last_undegraded":"2024-03-11T19:07:51.441461+0000","last_fullsized":"2024-03-11T19:07:51.441461+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:20:41.867658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739023+0000","last_change":"2024-03-11T19:07:49.747618+0000","last_active":"2024-03-11T19:07:50.739023+0000","last_peered":"2024-03-11T19:07:50.739023+0000","last_clean":"2024-03-11T19:07:50.739023+0000","last_became_active":"2024-03-11T19:07:49.747310+0000","last_became_peered":"2024-03-11T19:07:49.747310+0000","last_unstale":"2024-03-11T19:07:50.739023+0000","last_undegraded":"2024-03-11T19:07:50.739023+0000","last_fullsized":"2024-03-11T19:07:50.739023+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:15:42.260509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508731+0000","last_change":"2024-03-11T19:07:49.755564+0000","last_active":"2024-03-11T19:07:51.508731+0000","last_peered":"2024-03-11T19:07:51.508731+0000","last_clean":"2024-03-11T19:07:51.508731+0000","last_became_active":"2024-03-11T19:07:49.751322+0000","last_became_peered":"2024-03-11T19:07:49.751322+0000","last_unstale":"2024-03-11T19:07:51.508731+0000","last_undegraded":"2024-03-11T19:07:51.508731+0000","last_fullsized":"2024-03-11T19:07:51.508731+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:43:18.487887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739074+0000","last_change":"2024-03-11T19:07:49.731619+0000","last_active":"2024-03-11T19:07:50.739074+0000","last_peered":"2024-03-11T19:07:50.739074+0000","last_clean":"2024-03-11T19:07:50.739074+0000","last_became_active":"2024-03-11T19:07:49.731323+0000","last_became_peered":"2024-03-11T19:07:49.731323+0000","last_unstale":"2024-03-11T19:07:50.739074+0000","last_undegraded":"2024-03-11T19:07:50.739074+0000","last_fullsized":"2024-03-11T19:07:50.739074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:20:25.592709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.743380+0000","last_change":"2024-03-11T19:07:49.751851+0000","last_active":"2024-03-11T19:07:50.743380+0000","last_peered":"2024-03-11T19:07:50.743380+0000","last_clean":"2024-03-11T19:07:50.743380+0000","last_became_active":"2024-03-11T19:07:49.750068+0000","last_became_peered":"2024-03-11T19:07:49.750068+0000","last_unstale":"2024-03-11T19:07:50.743380+0000","last_undegraded":"2024-03-11T19:07:50.743380+0000","last_fullsized":"2024-03-11T19:07:50.743380+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:59:35.820107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508583+0000","last_change":"2024-03-11T19:07:49.752326+0000","last_active":"2024-03-11T19:07:51.508583+0000","last_peered":"2024-03-11T19:07:51.508583+0000","last_clean":"2024-03-11T19:07:51.508583+0000","last_became_active":"2024-03-11T19:07:49.748052+0000","last_became_peered":"2024-03-11T19:07:49.748052+0000","last_unstale":"2024-03-11T19:07:51.508583+0000","last_undegraded":"2024-03-11T19:07:51.508583+0000","last_fullsized":"2024-03-11T19:07:51.508583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:58:13.325964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508904+0000","last_change":"2024-03-11T19:07:49.748054+0000","last_active":"2024-03-11T19:07:51.508904+0000","last_peered":"2024-03-11T19:07:51.508904+0000","last_clean":"2024-03-11T19:07:51.508904+0000","last_became_active":"2024-03-11T19:07:49.747765+0000","last_became_peered":"2024-03-11T19:07:49.747765+0000","last_unstale":"2024-03-11T19:07:51.508904+0000","last_undegraded":"2024-03-11T19:07:51.508904+0000","last_fullsized":"2024-03-11T19:07:51.508904+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:18:05.599512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739677+0000","last_change":"2024-03-11T19:07:49.751622+0000","last_active":"2024-03-11T19:07:50.739677+0000","last_peered":"2024-03-11T19:07:50.739677+0000","last_clean":"2024-03-11T19:07:50.739677+0000","last_became_active":"2024-03-11T19:07:49.751317+0000","last_became_peered":"2024-03-11T19:07:49.751317+0000","last_unstale":"2024-03-11T19:07:50.739677+0000","last_undegraded":"2024-03-11T19:07:50.739677+0000","last_fullsized":"2024-03-11T19:07:50.739677+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:49.836096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.441285+0000","last_change":"2024-03-11T19:07:49.749160+0000","last_active":"2024-03-11T19:07:51.441285+0000","last_peered":"2024-03-11T19:07:51.441285+0000","last_clean":"2024-03-11T19:07:51.441285+0000","last_became_active":"2024-03-11T19:07:49.748838+0000","last_became_peered":"2024-03-11T19:07:49.748838+0000","last_unstale":"2024-03-11T19:07:51.441285+0000","last_undegraded":"2024-03-11T19:07:51.441285+0000","last_fullsized":"2024-03-11T19:07:51.441285+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:30:58.327196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508036+0000","last_change":"2024-03-11T19:07:49.749965+0000","last_active":"2024-03-11T19:07:51.508036+0000","last_peered":"2024-03-11T19:07:51.508036+0000","last_clean":"2024-03-11T19:07:51.508036+0000","last_became_active":"2024-03-11T19:07:49.749602+0000","last_became_peered":"2024-03-11T19:07:49.749602+0000","last_unstale":"2024-03-11T19:07:51.508036+0000","last_undegraded":"2024-03-11T19:07:51.508036+0000","last_fullsized":"2024-03-11T19:07:51.508036+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:36:08.807251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.738985+0000","last_change":"2024-03-11T19:07:49.747557+0000","last_active":"2024-03-11T19:07:50.738985+0000","last_peered":"2024-03-11T19:07:50.738985+0000","last_clean":"2024-03-11T19:07:50.738985+0000","last_became_active":"2024-03-11T19:07:49.745804+0000","last_became_peered":"2024-03-11T19:07:49.745804+0000","last_unstale":"2024-03-11T19:07:50.738985+0000","last_undegraded":"2024-03-11T19:07:50.738985+0000","last_fullsized":"2024-03-11T19:07:50.738985+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:07:26.939414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.738944+0000","last_change":"2024-03-11T19:07:49.744070+0000","last_active":"2024-03-11T19:07:50.738944+0000","last_peered":"2024-03-11T19:07:50.738944+0000","last_clean":"2024-03-11T19:07:50.738944+0000","last_became_active":"2024-03-11T19:07:49.743736+0000","last_became_peered":"2024-03-11T19:07:49.743736+0000","last_unstale":"2024-03-11T19:07:50.738944+0000","last_undegraded":"2024-03-11T19:07:50.738944+0000","last_fullsized":"2024-03-11T19:07:50.738944+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:50:33.862644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.507787+0000","last_change":"2024-03-11T19:07:49.748512+0000","last_active":"2024-03-11T19:07:51.507787+0000","last_peered":"2024-03-11T19:07:51.507787+0000","last_clean":"2024-03-11T19:07:51.507787+0000","last_became_active":"2024-03-11T19:07:49.748223+0000","last_became_peered":"2024-03-11T19:07:49.748223+0000","last_unstale":"2024-03-11T19:07:51.507787+0000","last_undegraded":"2024-03-11T19:07:51.507787+0000","last_fullsized":"2024-03-11T19:07:51.507787+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:49:43.305677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509376+0000","last_change":"2024-03-11T19:07:49.754322+0000","last_active":"2024-03-11T19:07:51.509376+0000","last_peered":"2024-03-11T19:07:51.509376+0000","last_clean":"2024-03-11T19:07:51.509376+0000","last_became_active":"2024-03-11T19:07:49.753757+0000","last_became_peered":"2024-03-11T19:07:49.753757+0000","last_unstale":"2024-03-11T19:07:51.509376+0000","last_undegraded":"2024-03-11T19:07:51.509376+0000","last_fullsized":"2024-03-11T19:07:51.509376+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:49.684978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.741917+0000","last_change":"2024-03-11T19:07:49.749546+0000","last_active":"2024-03-11T19:07:50.741917+0000","last_peered":"2024-03-11T19:07:50.741917+0000","last_clean":"2024-03-11T19:07:50.741917+0000","last_became_active":"2024-03-11T19:07:49.749200+0000","last_became_peered":"2024-03-11T19:07:49.749200+0000","last_unstale":"2024-03-11T19:07:50.741917+0000","last_undegraded":"2024-03-11T19:07:50.741917+0000","last_fullsized":"2024-03-11T19:07:50.741917+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:26:12.914015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509690+0000","last_change":"2024-03-11T19:07:49.750388+0000","last_active":"2024-03-11T19:07:51.509690+0000","last_peered":"2024-03-11T19:07:51.509690+0000","last_clean":"2024-03-11T19:07:51.509690+0000","last_became_active":"2024-03-11T19:07:49.750016+0000","last_became_peered":"2024-03-11T19:07:49.750016+0000","last_unstale":"2024-03-11T19:07:51.509690+0000","last_undegraded":"2024-03-11T19:07:51.509690+0000","last_fullsized":"2024-03-11T19:07:51.509690+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:32:23.125695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'13","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:55.790484+0000","last_change":"2024-03-11T19:07:48.738369+0000","last_active":"2024-03-11T19:07:55.790484+0000","last_peered":"2024-03-11T19:07:55.790484+0000","last_clean":"2024-03-11T19:07:55.790484+0000","last_became_active":"2024-03-11T19:07:48.738010+0000","last_became_peered":"2024-03-11T19:07:48.738010+0000","last_unstale":"2024-03-11T19:07:55.790484+0000","last_undegraded":"2024-03-11T19:07:55.790484+0000","last_fullsized":"2024-03-11T19:07:55.790484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:06:48.446454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.744152+0000","last_change":"2024-03-11T19:07:49.752622+0000","last_active":"2024-03-11T19:07:50.744152+0000","last_peered":"2024-03-11T19:07:50.744152+0000","last_clean":"2024-03-11T19:07:50.744152+0000","last_became_active":"2024-03-11T19:07:49.752242+0000","last_became_peered":"2024-03-11T19:07:49.752242+0000","last_unstale":"2024-03-11T19:07:50.744152+0000","last_undegraded":"2024-03-11T19:07:50.744152+0000","last_fullsized":"2024-03-11T19:07:50.744152+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:03:50.609641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.744038+0000","last_change":"2024-03-11T19:07:48.751657+0000","last_active":"2024-03-11T19:07:50.744038+0000","last_peered":"2024-03-11T19:07:50.744038+0000","last_clean":"2024-03-11T19:07:50.744038+0000","last_became_active":"2024-03-11T19:07:48.751294+0000","last_became_peered":"2024-03-11T19:07:48.751294+0000","last_unstale":"2024-03-11T19:07:50.744038+0000","last_undegraded":"2024-03-11T19:07:50.744038+0000","last_fullsized":"2024-03-11T19:07:50.744038+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:33:58.550707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739469+0000","last_change":"2024-03-11T19:07:49.759460+0000","last_active":"2024-03-11T19:07:50.739469+0000","last_peered":"2024-03-11T19:07:50.739469+0000","last_clean":"2024-03-11T19:07:50.739469+0000","last_became_active":"2024-03-11T19:07:49.759136+0000","last_became_peered":"2024-03-11T19:07:49.759136+0000","last_unstale":"2024-03-11T19:07:50.739469+0000","last_undegraded":"2024-03-11T19:07:50.739469+0000","last_fullsized":"2024-03-11T19:07:50.739469+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:10:37.273896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'7","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:54.782397+0000","last_change":"2024-03-11T19:07:48.744695+0000","last_active":"2024-03-11T19:07:54.782397+0000","last_peered":"2024-03-11T19:07:54.782397+0000","last_clean":"2024-03-11T19:07:54.782397+0000","last_became_active":"2024-03-11T19:07:48.744378+0000","last_became_peered":"2024-03-11T19:07:48.744378+0000","last_unstale":"2024-03-11T19:07:54.782397+0000","last_undegraded":"2024-03-11T19:07:54.782397+0000","last_fullsized":"2024-03-11T19:07:54.782397+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:55:46.601414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508089+0000","last_change":"2024-03-11T19:07:49.761429+0000","last_active":"2024-03-11T19:07:51.508089+0000","last_peered":"2024-03-11T19:07:51.508089+0000","last_clean":"2024-03-11T19:07:51.508089+0000","last_became_active":"2024-03-11T19:07:49.761120+0000","last_became_peered":"2024-03-11T19:07:49.761120+0000","last_unstale":"2024-03-11T19:07:51.508089+0000","last_undegraded":"2024-03-11T19:07:51.508089+0000","last_fullsized":"2024-03-11T19:07:51.508089+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:13:45.839528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739438+0000","last_change":"2024-03-11T19:07:48.740451+0000","last_active":"2024-03-11T19:07:50.739438+0000","last_peered":"2024-03-11T19:07:50.739438+0000","last_clean":"2024-03-11T19:07:50.739438+0000","last_became_active":"2024-03-11T19:07:48.740097+0000","last_became_peered":"2024-03-11T19:07:48.740097+0000","last_unstale":"2024-03-11T19:07:50.739438+0000","last_undegraded":"2024-03-11T19:07:50.739438+0000","last_fullsized":"2024-03-11T19:07:50.739438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:37.766790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509229+0000","last_change":"2024-03-11T19:07:49.750404+0000","last_active":"2024-03-11T19:07:51.509229+0000","last_peered":"2024-03-11T19:07:51.509229+0000","last_clean":"2024-03-11T19:07:51.509229+0000","last_became_active":"2024-03-11T19:07:49.749941+0000","last_became_peered":"2024-03-11T19:07:49.749941+0000","last_unstale":"2024-03-11T19:07:51.509229+0000","last_undegraded":"2024-03-11T19:07:51.509229+0000","last_fullsized":"2024-03-11T19:07:51.509229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:55.766729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'5","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.766952+0000","last_change":"2024-03-11T19:07:48.737100+0000","last_active":"2024-03-11T19:07:52.766952+0000","last_peered":"2024-03-11T19:07:52.766952+0000","last_clean":"2024-03-11T19:07:52.766952+0000","last_became_active":"2024-03-11T19:07:48.736768+0000","last_became_peered":"2024-03-11T19:07:48.736768+0000","last_unstale":"2024-03-11T19:07:52.766952+0000","last_undegraded":"2024-03-11T19:07:52.766952+0000","last_fullsized":"2024-03-11T19:07:52.766952+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:18:36.872151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739038+0000","last_change":"2024-03-11T19:07:49.739020+0000","last_active":"2024-03-11T19:07:50.739038+0000","last_peered":"2024-03-11T19:07:50.739038+0000","last_clean":"2024-03-11T19:07:50.739038+0000","last_became_active":"2024-03-11T19:07:49.738642+0000","last_became_peered":"2024-03-11T19:07:49.738642+0000","last_unstale":"2024-03-11T19:07:50.739038+0000","last_undegraded":"2024-03-11T19:07:50.739038+0000","last_fullsized":"2024-03-11T19:07:50.739038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:42:29.040242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'7","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:54.782428+0000","last_change":"2024-03-11T19:07:48.745192+0000","last_active":"2024-03-11T19:07:54.782428+0000","last_peered":"2024-03-11T19:07:54.782428+0000","last_clean":"2024-03-11T19:07:54.782428+0000","last_became_active":"2024-03-11T19:07:48.744868+0000","last_became_peered":"2024-03-11T19:07:48.744868+0000","last_unstale":"2024-03-11T19:07:54.782428+0000","last_undegraded":"2024-03-11T19:07:54.782428+0000","last_fullsized":"2024-03-11T19:07:54.782428+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:19:15.620186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508565+0000","last_change":"2024-03-11T19:07:49.737149+0000","last_active":"2024-03-11T19:07:51.508565+0000","last_peered":"2024-03-11T19:07:51.508565+0000","last_clean":"2024-03-11T19:07:51.508565+0000","last_became_active":"2024-03-11T19:07:49.736797+0000","last_became_peered":"2024-03-11T19:07:49.736797+0000","last_unstale":"2024-03-11T19:07:51.508565+0000","last_undegraded":"2024-03-11T19:07:51.508565+0000","last_fullsized":"2024-03-11T19:07:51.508565+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:12:10.282062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:53.773777+0000","last_change":"2024-03-11T19:07:48.743948+0000","last_active":"2024-03-11T19:07:53.773777+0000","last_peered":"2024-03-11T19:07:53.773777+0000","last_clean":"2024-03-11T19:07:53.773777+0000","last_became_active":"2024-03-11T19:07:48.743451+0000","last_became_peered":"2024-03-11T19:07:48.743451+0000","last_unstale":"2024-03-11T19:07:53.773777+0000","last_undegraded":"2024-03-11T19:07:53.773777+0000","last_fullsized":"2024-03-11T19:07:53.773777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:03:19.823287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.743297+0000","last_change":"2024-03-11T19:07:49.745518+0000","last_active":"2024-03-11T19:07:50.743297+0000","last_peered":"2024-03-11T19:07:50.743297+0000","last_clean":"2024-03-11T19:07:50.743297+0000","last_became_active":"2024-03-11T19:07:49.743719+0000","last_became_peered":"2024-03-11T19:07:49.743719+0000","last_unstale":"2024-03-11T19:07:50.743297+0000","last_undegraded":"2024-03-11T19:07:50.743297+0000","last_fullsized":"2024-03-11T19:07:50.743297+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:25:33.855679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509514+0000","last_change":"2024-03-11T19:07:48.734044+0000","last_active":"2024-03-11T19:07:51.509514+0000","last_peered":"2024-03-11T19:07:51.509514+0000","last_clean":"2024-03-11T19:07:51.509514+0000","last_became_active":"2024-03-11T19:07:48.733723+0000","last_became_peered":"2024-03-11T19:07:48.733723+0000","last_unstale":"2024-03-11T19:07:51.509514+0000","last_undegraded":"2024-03-11T19:07:51.509514+0000","last_fullsized":"2024-03-11T19:07:51.509514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:53:37.070296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.741698+0000","last_change":"2024-03-11T19:07:49.743729+0000","last_active":"2024-03-11T19:07:50.741698+0000","last_peered":"2024-03-11T19:07:50.741698+0000","last_clean":"2024-03-11T19:07:50.741698+0000","last_became_active":"2024-03-11T19:07:49.742992+0000","last_became_peered":"2024-03-11T19:07:49.742992+0000","last_unstale":"2024-03-11T19:07:50.741698+0000","last_undegraded":"2024-03-11T19:07:50.741698+0000","last_fullsized":"2024-03-11T19:07:50.741698+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:46.477177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:56.672097+0000","last_change":"2024-03-11T19:07:48.738775+0000","last_active":"2024-03-11T19:07:56.672097+0000","last_peered":"2024-03-11T19:07:56.672097+0000","last_clean":"2024-03-11T19:07:56.672097+0000","last_became_active":"2024-03-11T19:07:48.738437+0000","last_became_peered":"2024-03-11T19:07:48.738437+0000","last_unstale":"2024-03-11T19:07:56.672097+0000","last_undegraded":"2024-03-11T19:07:56.672097+0000","last_fullsized":"2024-03-11T19:07:56.672097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:17:20.020756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508324+0000","last_change":"2024-03-11T19:07:49.760293+0000","last_active":"2024-03-11T19:07:51.508324+0000","last_peered":"2024-03-11T19:07:51.508324+0000","last_clean":"2024-03-11T19:07:51.508324+0000","last_became_active":"2024-03-11T19:07:49.759977+0000","last_became_peered":"2024-03-11T19:07:49.759977+0000","last_unstale":"2024-03-11T19:07:51.508324+0000","last_undegraded":"2024-03-11T19:07:51.508324+0000","last_fullsized":"2024-03-11T19:07:51.508324+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:07:44.726756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.763455+0000","last_change":"2024-03-11T19:07:48.740672+0000","last_active":"2024-03-11T19:07:52.763455+0000","last_peered":"2024-03-11T19:07:52.763455+0000","last_clean":"2024-03-11T19:07:52.763455+0000","last_became_active":"2024-03-11T19:07:48.739331+0000","last_became_peered":"2024-03-11T19:07:48.739331+0000","last_unstale":"2024-03-11T19:07:52.763455+0000","last_undegraded":"2024-03-11T19:07:52.763455+0000","last_fullsized":"2024-03-11T19:07:52.763455+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:19:28.375515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739464+0000","last_change":"2024-03-11T19:07:49.742780+0000","last_active":"2024-03-11T19:07:50.739464+0000","last_peered":"2024-03-11T19:07:50.739464+0000","last_clean":"2024-03-11T19:07:50.739464+0000","last_became_active":"2024-03-11T19:07:49.742442+0000","last_became_peered":"2024-03-11T19:07:49.742442+0000","last_unstale":"2024-03-11T19:07:50.739464+0000","last_undegraded":"2024-03-11T19:07:50.739464+0000","last_fullsized":"2024-03-11T19:07:50.739464+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:11:13.838225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509039+0000","last_change":"2024-03-11T19:07:48.752617+0000","last_active":"2024-03-11T19:07:51.509039+0000","last_peered":"2024-03-11T19:07:51.509039+0000","last_clean":"2024-03-11T19:07:51.509039+0000","last_became_active":"2024-03-11T19:07:48.752181+0000","last_became_peered":"2024-03-11T19:07:48.752181+0000","last_unstale":"2024-03-11T19:07:51.509039+0000","last_undegraded":"2024-03-11T19:07:51.509039+0000","last_fullsized":"2024-03-11T19:07:51.509039+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:05:37.162866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508449+0000","last_change":"2024-03-11T19:07:49.753672+0000","last_active":"2024-03-11T19:07:51.508449+0000","last_peered":"2024-03-11T19:07:51.508449+0000","last_clean":"2024-03-11T19:07:51.508449+0000","last_became_active":"2024-03-11T19:07:49.753315+0000","last_became_peered":"2024-03-11T19:07:49.753315+0000","last_unstale":"2024-03-11T19:07:51.508449+0000","last_undegraded":"2024-03-11T19:07:51.508449+0000","last_fullsized":"2024-03-11T19:07:51.508449+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:30:58.503249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'21","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:56.897333+0000","last_change":"2024-03-11T19:07:48.740424+0000","last_active":"2024-03-11T19:07:56.897333+0000","last_peered":"2024-03-11T19:07:56.897333+0000","last_clean":"2024-03-11T19:07:56.897333+0000","last_became_active":"2024-03-11T19:07:48.740077+0000","last_became_peered":"2024-03-11T19:07:48.740077+0000","last_unstale":"2024-03-11T19:07:56.897333+0000","last_undegraded":"2024-03-11T19:07:56.897333+0000","last_fullsized":"2024-03-11T19:07:56.897333+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:30:29.193861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38193,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":22,"num_write":23,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739212+0000","last_change":"2024-03-11T19:07:49.746440+0000","last_active":"2024-03-11T19:07:50.739212+0000","last_peered":"2024-03-11T19:07:50.739212+0000","last_clean":"2024-03-11T19:07:50.739212+0000","last_became_active":"2024-03-11T19:07:49.746106+0000","last_became_peered":"2024-03-11T19:07:49.746106+0000","last_unstale":"2024-03-11T19:07:50.739212+0000","last_undegraded":"2024-03-11T19:07:50.739212+0000","last_fullsized":"2024-03-11T19:07:50.739212+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:44:58.039917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.765082+0000","last_change":"2024-03-11T19:07:48.744437+0000","last_active":"2024-03-11T19:07:52.765082+0000","last_peered":"2024-03-11T19:07:52.765082+0000","last_clean":"2024-03-11T19:07:52.765082+0000","last_became_active":"2024-03-11T19:07:48.744142+0000","last_became_peered":"2024-03-11T19:07:48.744142+0000","last_unstale":"2024-03-11T19:07:52.765082+0000","last_undegraded":"2024-03-11T19:07:52.765082+0000","last_fullsized":"2024-03-11T19:07:52.765082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:32:09.076975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.507894+0000","last_change":"2024-03-11T19:07:49.751191+0000","last_active":"2024-03-11T19:07:51.507894+0000","last_peered":"2024-03-11T19:07:51.507894+0000","last_clean":"2024-03-11T19:07:51.507894+0000","last_became_active":"2024-03-11T19:07:49.750849+0000","last_became_peered":"2024-03-11T19:07:49.750849+0000","last_unstale":"2024-03-11T19:07:51.507894+0000","last_undegraded":"2024-03-11T19:07:51.507894+0000","last_fullsized":"2024-03-11T19:07:51.507894+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:19:18.416905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.767963+0000","last_change":"2024-03-11T19:07:48.742808+0000","last_active":"2024-03-11T19:07:52.767963+0000","last_peered":"2024-03-11T19:07:52.767963+0000","last_clean":"2024-03-11T19:07:52.767963+0000","last_became_active":"2024-03-11T19:07:48.742462+0000","last_became_peered":"2024-03-11T19:07:48.742462+0000","last_unstale":"2024-03-11T19:07:52.767963+0000","last_undegraded":"2024-03-11T19:07:52.767963+0000","last_fullsized":"2024-03-11T19:07:52.767963+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:19.109299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739501+0000","last_change":"2024-03-11T19:07:49.744673+0000","last_active":"2024-03-11T19:07:50.739501+0000","last_peered":"2024-03-11T19:07:50.739501+0000","last_clean":"2024-03-11T19:07:50.739501+0000","last_became_active":"2024-03-11T19:07:49.744313+0000","last_became_peered":"2024-03-11T19:07:49.744313+0000","last_unstale":"2024-03-11T19:07:50.739501+0000","last_undegraded":"2024-03-11T19:07:50.739501+0000","last_fullsized":"2024-03-11T19:07:50.739501+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:21:04.759356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:53.774596+0000","last_change":"2024-03-11T19:07:48.736024+0000","last_active":"2024-03-11T19:07:53.774596+0000","last_peered":"2024-03-11T19:07:53.774596+0000","last_clean":"2024-03-11T19:07:53.774596+0000","last_became_active":"2024-03-11T19:07:48.735720+0000","last_became_peered":"2024-03-11T19:07:48.735720+0000","last_unstale":"2024-03-11T19:07:53.774596+0000","last_undegraded":"2024-03-11T19:07:53.774596+0000","last_fullsized":"2024-03-11T19:07:53.774596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:04:57.386267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739051+0000","last_change":"2024-03-11T19:07:49.742924+0000","last_active":"2024-03-11T19:07:50.739051+0000","last_peered":"2024-03-11T19:07:50.739051+0000","last_clean":"2024-03-11T19:07:50.739051+0000","last_became_active":"2024-03-11T19:07:49.742542+0000","last_became_peered":"2024-03-11T19:07:49.742542+0000","last_unstale":"2024-03-11T19:07:50.739051+0000","last_undegraded":"2024-03-11T19:07:50.739051+0000","last_fullsized":"2024-03-11T19:07:50.739051+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:08:11.944311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'16","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:53.774362+0000","last_change":"2024-03-11T19:07:48.753113+0000","last_active":"2024-03-11T19:07:53.774362+0000","last_peered":"2024-03-11T19:07:53.774362+0000","last_clean":"2024-03-11T19:07:53.774362+0000","last_became_active":"2024-03-11T19:07:48.752517+0000","last_became_peered":"2024-03-11T19:07:48.752517+0000","last_unstale":"2024-03-11T19:07:53.774362+0000","last_undegraded":"2024-03-11T19:07:53.774362+0000","last_fullsized":"2024-03-11T19:07:53.774362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:48:33.441125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509767+0000","last_change":"2024-03-11T19:07:49.748026+0000","last_active":"2024-03-11T19:07:51.509767+0000","last_peered":"2024-03-11T19:07:51.509767+0000","last_clean":"2024-03-11T19:07:51.509767+0000","last_became_active":"2024-03-11T19:07:49.747685+0000","last_became_peered":"2024-03-11T19:07:49.747685+0000","last_unstale":"2024-03-11T19:07:51.509767+0000","last_undegraded":"2024-03-11T19:07:51.509767+0000","last_fullsized":"2024-03-11T19:07:51.509767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:54:40.337723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739183+0000","last_change":"2024-03-11T19:07:48.739571+0000","last_active":"2024-03-11T19:07:50.739183+0000","last_peered":"2024-03-11T19:07:50.739183+0000","last_clean":"2024-03-11T19:07:50.739183+0000","last_became_active":"2024-03-11T19:07:48.739159+0000","last_became_peered":"2024-03-11T19:07:48.739159+0000","last_unstale":"2024-03-11T19:07:50.739183+0000","last_undegraded":"2024-03-11T19:07:50.739183+0000","last_fullsized":"2024-03-11T19:07:50.739183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:46:43.474591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508345+0000","last_change":"2024-03-11T19:07:49.744760+0000","last_active":"2024-03-11T19:07:51.508345+0000","last_peered":"2024-03-11T19:07:51.508345+0000","last_clean":"2024-03-11T19:07:51.508345+0000","last_became_active":"2024-03-11T19:07:49.744265+0000","last_became_peered":"2024-03-11T19:07:49.744265+0000","last_unstale":"2024-03-11T19:07:51.508345+0000","last_undegraded":"2024-03-11T19:07:51.508345+0000","last_fullsized":"2024-03-11T19:07:51.508345+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:09:49.823462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.502145+0000","last_change":"2024-03-11T19:07:48.740223+0000","last_active":"2024-03-11T19:07:51.502145+0000","last_peered":"2024-03-11T19:07:51.502145+0000","last_clean":"2024-03-11T19:07:51.502145+0000","last_became_active":"2024-03-11T19:07:48.739569+0000","last_became_peered":"2024-03-11T19:07:48.739569+0000","last_unstale":"2024-03-11T19:07:51.502145+0000","last_undegraded":"2024-03-11T19:07:51.502145+0000","last_fullsized":"2024-03-11T19:07:51.502145+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:04:19.384523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.573850+0000","last_change":"2024-03-11T19:07:49.755620+0000","last_active":"2024-03-11T19:07:52.573850+0000","last_peered":"2024-03-11T19:07:52.573850+0000","last_clean":"2024-03-11T19:07:52.573850+0000","last_became_active":"2024-03-11T19:07:49.750886+0000","last_became_peered":"2024-03-11T19:07:49.750886+0000","last_unstale":"2024-03-11T19:07:52.573850+0000","last_undegraded":"2024-03-11T19:07:52.573850+0000","last_fullsized":"2024-03-11T19:07:52.573850+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:35:54.191061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739273+0000","last_change":"2024-03-11T19:07:48.743340+0000","last_active":"2024-03-11T19:07:50.739273+0000","last_peered":"2024-03-11T19:07:50.739273+0000","last_clean":"2024-03-11T19:07:50.739273+0000","last_became_active":"2024-03-11T19:07:48.742947+0000","last_became_peered":"2024-03-11T19:07:48.742947+0000","last_unstale":"2024-03-11T19:07:50.739273+0000","last_undegraded":"2024-03-11T19:07:50.739273+0000","last_fullsized":"2024-03-11T19:07:50.739273+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:40:25.233150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508305+0000","last_change":"2024-03-11T19:07:49.753668+0000","last_active":"2024-03-11T19:07:51.508305+0000","last_peered":"2024-03-11T19:07:51.508305+0000","last_clean":"2024-03-11T19:07:51.508305+0000","last_became_active":"2024-03-11T19:07:49.750574+0000","last_became_peered":"2024-03-11T19:07:49.750574+0000","last_unstale":"2024-03-11T19:07:51.508305+0000","last_undegraded":"2024-03-11T19:07:51.508305+0000","last_fullsized":"2024-03-11T19:07:51.508305+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:12:32.711847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509695+0000","last_change":"2024-03-11T19:07:49.739538+0000","last_active":"2024-03-11T19:07:51.509695+0000","last_peered":"2024-03-11T19:07:51.509695+0000","last_clean":"2024-03-11T19:07:51.509695+0000","last_became_active":"2024-03-11T19:07:49.738617+0000","last_became_peered":"2024-03-11T19:07:49.738617+0000","last_unstale":"2024-03-11T19:07:51.509695+0000","last_undegraded":"2024-03-11T19:07:51.509695+0000","last_fullsized":"2024-03-11T19:07:51.509695+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:43:08.697125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:53.773088+0000","last_change":"2024-03-11T19:07:48.749527+0000","last_active":"2024-03-11T19:07:53.773088+0000","last_peered":"2024-03-11T19:07:53.773088+0000","last_clean":"2024-03-11T19:07:53.773088+0000","last_became_active":"2024-03-11T19:07:48.749026+0000","last_became_peered":"2024-03-11T19:07:48.749026+0000","last_unstale":"2024-03-11T19:07:53.773088+0000","last_undegraded":"2024-03-11T19:07:53.773088+0000","last_fullsized":"2024-03-11T19:07:53.773088+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:20:19.058429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.743699+0000","last_change":"2024-03-11T19:07:49.757696+0000","last_active":"2024-03-11T19:07:50.743699+0000","last_peered":"2024-03-11T19:07:50.743699+0000","last_clean":"2024-03-11T19:07:50.743699+0000","last_became_active":"2024-03-11T19:07:49.757380+0000","last_became_peered":"2024-03-11T19:07:49.757380+0000","last_unstale":"2024-03-11T19:07:50.743699+0000","last_undegraded":"2024-03-11T19:07:50.743699+0000","last_fullsized":"2024-03-11T19:07:50.743699+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:03:22.954619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.764299+0000","last_change":"2024-03-11T19:07:48.737504+0000","last_active":"2024-03-11T19:07:52.764299+0000","last_peered":"2024-03-11T19:07:52.764299+0000","last_clean":"2024-03-11T19:07:52.764299+0000","last_became_active":"2024-03-11T19:07:48.737147+0000","last_became_peered":"2024-03-11T19:07:48.737147+0000","last_unstale":"2024-03-11T19:07:52.764299+0000","last_undegraded":"2024-03-11T19:07:52.764299+0000","last_fullsized":"2024-03-11T19:07:52.764299+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:04.819617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":339,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739051+0000","last_change":"2024-03-11T19:06:35.035452+0000","last_active":"2024-03-11T19:07:50.739051+0000","last_peered":"2024-03-11T19:07:50.739051+0000","last_clean":"2024-03-11T19:07:50.739051+0000","last_became_active":"2024-03-11T19:06:35.035076+0000","last_became_peered":"2024-03-11T19:06:35.035076+0000","last_unstale":"2024-03-11T19:07:50.739051+0000","last_undegraded":"2024-03-11T19:07:50.739051+0000","last_fullsized":"2024-03-11T19:07:50.739051+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:03:10.143002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:03:10.143002+0000","last_clean_scrub_stamp":"2024-03-11T19:03:10.143002+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:39:37.348091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.507654+0000","last_change":"2024-03-11T19:07:49.743000+0000","last_active":"2024-03-11T19:07:51.507654+0000","last_peered":"2024-03-11T19:07:51.507654+0000","last_clean":"2024-03-11T19:07:51.507654+0000","last_became_active":"2024-03-11T19:07:49.742691+0000","last_became_peered":"2024-03-11T19:07:49.742691+0000","last_unstale":"2024-03-11T19:07:51.507654+0000","last_undegraded":"2024-03-11T19:07:51.507654+0000","last_fullsized":"2024-03-11T19:07:51.507654+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:40:21.519929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:53.772503+0000","last_change":"2024-03-11T19:07:48.739038+0000","last_active":"2024-03-11T19:07:53.772503+0000","last_peered":"2024-03-11T19:07:53.772503+0000","last_clean":"2024-03-11T19:07:53.772503+0000","last_became_active":"2024-03-11T19:07:48.738714+0000","last_became_peered":"2024-03-11T19:07:48.738714+0000","last_unstale":"2024-03-11T19:07:53.772503+0000","last_undegraded":"2024-03-11T19:07:53.772503+0000","last_fullsized":"2024-03-11T19:07:53.772503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:17:43.086679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:54.782084+0000","last_change":"2024-03-11T19:07:48.752699+0000","last_active":"2024-03-11T19:07:54.782084+0000","last_peered":"2024-03-11T19:07:54.782084+0000","last_clean":"2024-03-11T19:07:54.782084+0000","last_became_active":"2024-03-11T19:07:48.752331+0000","last_became_peered":"2024-03-11T19:07:48.752331+0000","last_unstale":"2024-03-11T19:07:54.782084+0000","last_undegraded":"2024-03-11T19:07:54.782084+0000","last_fullsized":"2024-03-11T19:07:54.782084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:03.912256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508129+0000","last_change":"2024-03-11T19:07:49.742434+0000","last_active":"2024-03-11T19:07:51.508129+0000","last_peered":"2024-03-11T19:07:51.508129+0000","last_clean":"2024-03-11T19:07:51.508129+0000","last_became_active":"2024-03-11T19:07:49.742048+0000","last_became_peered":"2024-03-11T19:07:49.742048+0000","last_unstale":"2024-03-11T19:07:51.508129+0000","last_undegraded":"2024-03-11T19:07:51.508129+0000","last_fullsized":"2024-03-11T19:07:51.508129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:39:22.685055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'7","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:54.784180+0000","last_change":"2024-03-11T19:07:48.744728+0000","last_active":"2024-03-11T19:07:54.784180+0000","last_peered":"2024-03-11T19:07:54.784180+0000","last_clean":"2024-03-11T19:07:54.784180+0000","last_became_active":"2024-03-11T19:07:48.744378+0000","last_became_peered":"2024-03-11T19:07:48.744378+0000","last_unstale":"2024-03-11T19:07:54.784180+0000","last_undegraded":"2024-03-11T19:07:54.784180+0000","last_fullsized":"2024-03-11T19:07:54.784180+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:42.823040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.743742+0000","last_change":"2024-03-11T19:07:49.744261+0000","last_active":"2024-03-11T19:07:50.743742+0000","last_peered":"2024-03-11T19:07:50.743742+0000","last_clean":"2024-03-11T19:07:50.743742+0000","last_became_active":"2024-03-11T19:07:49.743946+0000","last_became_peered":"2024-03-11T19:07:49.743946+0000","last_unstale":"2024-03-11T19:07:50.743742+0000","last_undegraded":"2024-03-11T19:07:50.743742+0000","last_fullsized":"2024-03-11T19:07:50.743742+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:54:07.928899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.763246+0000","last_change":"2024-03-11T19:07:48.740059+0000","last_active":"2024-03-11T19:07:52.763246+0000","last_peered":"2024-03-11T19:07:52.763246+0000","last_clean":"2024-03-11T19:07:52.763246+0000","last_became_active":"2024-03-11T19:07:48.739626+0000","last_became_peered":"2024-03-11T19:07:48.739626+0000","last_unstale":"2024-03-11T19:07:52.763246+0000","last_undegraded":"2024-03-11T19:07:52.763246+0000","last_fullsized":"2024-03-11T19:07:52.763246+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:45:12.219203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739482+0000","last_change":"2024-03-11T19:07:49.746041+0000","last_active":"2024-03-11T19:07:50.739482+0000","last_peered":"2024-03-11T19:07:50.739482+0000","last_clean":"2024-03-11T19:07:50.739482+0000","last_became_active":"2024-03-11T19:07:49.745602+0000","last_became_peered":"2024-03-11T19:07:49.745602+0000","last_unstale":"2024-03-11T19:07:50.739482+0000","last_undegraded":"2024-03-11T19:07:50.739482+0000","last_fullsized":"2024-03-11T19:07:50.739482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:23:44.596929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.763882+0000","last_change":"2024-03-11T19:07:48.748677+0000","last_active":"2024-03-11T19:07:52.763882+0000","last_peered":"2024-03-11T19:07:52.763882+0000","last_clean":"2024-03-11T19:07:52.763882+0000","last_became_active":"2024-03-11T19:07:48.748470+0000","last_became_peered":"2024-03-11T19:07:48.748470+0000","last_unstale":"2024-03-11T19:07:52.763882+0000","last_undegraded":"2024-03-11T19:07:52.763882+0000","last_fullsized":"2024-03-11T19:07:52.763882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:24:55.240722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.738941+0000","last_change":"2024-03-11T19:07:49.746522+0000","last_active":"2024-03-11T19:07:50.738941+0000","last_peered":"2024-03-11T19:07:50.738941+0000","last_clean":"2024-03-11T19:07:50.738941+0000","last_became_active":"2024-03-11T19:07:49.746179+0000","last_became_peered":"2024-03-11T19:07:49.746179+0000","last_unstale":"2024-03-11T19:07:50.738941+0000","last_undegraded":"2024-03-11T19:07:50.738941+0000","last_fullsized":"2024-03-11T19:07:50.738941+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:06:08.144850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.766543+0000","last_change":"2024-03-11T19:07:48.740926+0000","last_active":"2024-03-11T19:07:52.766543+0000","last_peered":"2024-03-11T19:07:52.766543+0000","last_clean":"2024-03-11T19:07:52.766543+0000","last_became_active":"2024-03-11T19:07:48.740625+0000","last_became_peered":"2024-03-11T19:07:48.740625+0000","last_unstale":"2024-03-11T19:07:52.766543+0000","last_undegraded":"2024-03-11T19:07:52.766543+0000","last_fullsized":"2024-03-11T19:07:52.766543+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:50:13.079608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508081+0000","last_change":"2024-03-11T19:07:49.742428+0000","last_active":"2024-03-11T19:07:51.508081+0000","last_peered":"2024-03-11T19:07:51.508081+0000","last_clean":"2024-03-11T19:07:51.508081+0000","last_became_active":"2024-03-11T19:07:49.742098+0000","last_became_peered":"2024-03-11T19:07:49.742098+0000","last_unstale":"2024-03-11T19:07:51.508081+0000","last_undegraded":"2024-03-11T19:07:51.508081+0000","last_fullsized":"2024-03-11T19:07:51.508081+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:19:24.615755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508717+0000","last_change":"2024-03-11T19:07:48.745092+0000","last_active":"2024-03-11T19:07:51.508717+0000","last_peered":"2024-03-11T19:07:51.508717+0000","last_clean":"2024-03-11T19:07:51.508717+0000","last_became_active":"2024-03-11T19:07:48.744714+0000","last_became_peered":"2024-03-11T19:07:48.744714+0000","last_unstale":"2024-03-11T19:07:51.508717+0000","last_undegraded":"2024-03-11T19:07:51.508717+0000","last_fullsized":"2024-03-11T19:07:51.508717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:10:48.274993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.743920+0000","last_change":"2024-03-11T19:07:49.754872+0000","last_active":"2024-03-11T19:07:50.743920+0000","last_peered":"2024-03-11T19:07:50.743920+0000","last_clean":"2024-03-11T19:07:50.743920+0000","last_became_active":"2024-03-11T19:07:49.749663+0000","last_became_peered":"2024-03-11T19:07:49.749663+0000","last_unstale":"2024-03-11T19:07:50.743920+0000","last_undegraded":"2024-03-11T19:07:50.743920+0000","last_fullsized":"2024-03-11T19:07:50.743920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:22:23.664814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:52.763388+0000","last_change":"2024-03-11T19:07:48.742227+0000","last_active":"2024-03-11T19:07:52.763388+0000","last_peered":"2024-03-11T19:07:52.763388+0000","last_clean":"2024-03-11T19:07:52.763388+0000","last_became_active":"2024-03-11T19:07:48.741912+0000","last_became_peered":"2024-03-11T19:07:48.741912+0000","last_unstale":"2024-03-11T19:07:52.763388+0000","last_undegraded":"2024-03-11T19:07:52.763388+0000","last_fullsized":"2024-03-11T19:07:52.763388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:10:06.120433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739388+0000","last_change":"2024-03-11T19:07:49.748947+0000","last_active":"2024-03-11T19:07:50.739388+0000","last_peered":"2024-03-11T19:07:50.739388+0000","last_clean":"2024-03-11T19:07:50.739388+0000","last_became_active":"2024-03-11T19:07:49.748621+0000","last_became_peered":"2024-03-11T19:07:49.748621+0000","last_unstale":"2024-03-11T19:07:50.739388+0000","last_undegraded":"2024-03-11T19:07:50.739388+0000","last_fullsized":"2024-03-11T19:07:50.739388+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:19:53.824590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509318+0000","last_change":"2024-03-11T19:07:48.748799+0000","last_active":"2024-03-11T19:07:51.509318+0000","last_peered":"2024-03-11T19:07:51.509318+0000","last_clean":"2024-03-11T19:07:51.509318+0000","last_became_active":"2024-03-11T19:07:48.748434+0000","last_became_peered":"2024-03-11T19:07:48.748434+0000","last_unstale":"2024-03-11T19:07:51.509318+0000","last_undegraded":"2024-03-11T19:07:51.509318+0000","last_fullsized":"2024-03-11T19:07:51.509318+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:10:09.836926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739740+0000","last_change":"2024-03-11T19:07:49.750122+0000","last_active":"2024-03-11T19:07:50.739740+0000","last_peered":"2024-03-11T19:07:50.739740+0000","last_clean":"2024-03-11T19:07:50.739740+0000","last_became_active":"2024-03-11T19:07:49.749801+0000","last_became_peered":"2024-03-11T19:07:49.749801+0000","last_unstale":"2024-03-11T19:07:50.739740+0000","last_undegraded":"2024-03-11T19:07:50.739740+0000","last_fullsized":"2024-03-11T19:07:50.739740+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:58:31.155275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508847+0000","last_change":"2024-03-11T19:07:48.741959+0000","last_active":"2024-03-11T19:07:51.508847+0000","last_peered":"2024-03-11T19:07:51.508847+0000","last_clean":"2024-03-11T19:07:51.508847+0000","last_became_active":"2024-03-11T19:07:48.741630+0000","last_became_peered":"2024-03-11T19:07:48.741630+0000","last_unstale":"2024-03-11T19:07:51.508847+0000","last_undegraded":"2024-03-11T19:07:51.508847+0000","last_fullsized":"2024-03-11T19:07:51.508847+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:27:08.606055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739222+0000","last_change":"2024-03-11T19:07:49.729838+0000","last_active":"2024-03-11T19:07:50.739222+0000","last_peered":"2024-03-11T19:07:50.739222+0000","last_clean":"2024-03-11T19:07:50.739222+0000","last_became_active":"2024-03-11T19:07:49.728912+0000","last_became_peered":"2024-03-11T19:07:49.728912+0000","last_unstale":"2024-03-11T19:07:50.739222+0000","last_undegraded":"2024-03-11T19:07:50.739222+0000","last_fullsized":"2024-03-11T19:07:50.739222+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:02:06.551734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508026+0000","last_change":"2024-03-11T19:07:48.744709+0000","last_active":"2024-03-11T19:07:51.508026+0000","last_peered":"2024-03-11T19:07:51.508026+0000","last_clean":"2024-03-11T19:07:51.508026+0000","last_became_active":"2024-03-11T19:07:48.744356+0000","last_became_peered":"2024-03-11T19:07:48.744356+0000","last_unstale":"2024-03-11T19:07:51.508026+0000","last_undegraded":"2024-03-11T19:07:51.508026+0000","last_fullsized":"2024-03-11T19:07:51.508026+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:48:47.098443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739422+0000","last_change":"2024-03-11T19:07:49.743288+0000","last_active":"2024-03-11T19:07:50.739422+0000","last_peered":"2024-03-11T19:07:50.739422+0000","last_clean":"2024-03-11T19:07:50.739422+0000","last_became_active":"2024-03-11T19:07:49.742942+0000","last_became_peered":"2024-03-11T19:07:49.742942+0000","last_unstale":"2024-03-11T19:07:50.739422+0000","last_undegraded":"2024-03-11T19:07:50.739422+0000","last_fullsized":"2024-03-11T19:07:50.739422+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:21:53.896192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508279+0000","last_change":"2024-03-11T19:07:49.741610+0000","last_active":"2024-03-11T19:07:51.508279+0000","last_peered":"2024-03-11T19:07:51.508279+0000","last_clean":"2024-03-11T19:07:51.508279+0000","last_became_active":"2024-03-11T19:07:49.741234+0000","last_became_peered":"2024-03-11T19:07:49.741234+0000","last_unstale":"2024-03-11T19:07:51.508279+0000","last_undegraded":"2024-03-11T19:07:51.508279+0000","last_fullsized":"2024-03-11T19:07:51.508279+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:53:03.320093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508376+0000","last_change":"2024-03-11T19:07:49.739625+0000","last_active":"2024-03-11T19:07:51.508376+0000","last_peered":"2024-03-11T19:07:51.508376+0000","last_clean":"2024-03-11T19:07:51.508376+0000","last_became_active":"2024-03-11T19:07:49.739283+0000","last_became_peered":"2024-03-11T19:07:49.739283+0000","last_unstale":"2024-03-11T19:07:51.508376+0000","last_undegraded":"2024-03-11T19:07:51.508376+0000","last_fullsized":"2024-03-11T19:07:51.508376+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:31:45.506952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508657+0000","last_change":"2024-03-11T19:07:49.745928+0000","last_active":"2024-03-11T19:07:51.508657+0000","last_peered":"2024-03-11T19:07:51.508657+0000","last_clean":"2024-03-11T19:07:51.508657+0000","last_became_active":"2024-03-11T19:07:49.745025+0000","last_became_peered":"2024-03-11T19:07:49.745025+0000","last_unstale":"2024-03-11T19:07:51.508657+0000","last_undegraded":"2024-03-11T19:07:51.508657+0000","last_fullsized":"2024-03-11T19:07:51.508657+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:40:30.141906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.739082+0000","last_change":"2024-03-11T19:07:49.756274+0000","last_active":"2024-03-11T19:07:50.739082+0000","last_peered":"2024-03-11T19:07:50.739082+0000","last_clean":"2024-03-11T19:07:50.739082+0000","last_became_active":"2024-03-11T19:07:49.755954+0000","last_became_peered":"2024-03-11T19:07:49.755954+0000","last_unstale":"2024-03-11T19:07:50.739082+0000","last_undegraded":"2024-03-11T19:07:50.739082+0000","last_fullsized":"2024-03-11T19:07:50.739082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:15:36.825240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.507951+0000","last_change":"2024-03-11T19:07:49.752609+0000","last_active":"2024-03-11T19:07:51.507951+0000","last_peered":"2024-03-11T19:07:51.507951+0000","last_clean":"2024-03-11T19:07:51.507951+0000","last_became_active":"2024-03-11T19:07:49.752164+0000","last_became_peered":"2024-03-11T19:07:49.752164+0000","last_unstale":"2024-03-11T19:07:51.507951+0000","last_undegraded":"2024-03-11T19:07:51.507951+0000","last_fullsized":"2024-03-11T19:07:51.507951+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:56:49.705582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508382+0000","last_change":"2024-03-11T19:07:49.743963+0000","last_active":"2024-03-11T19:07:51.508382+0000","last_peered":"2024-03-11T19:07:51.508382+0000","last_clean":"2024-03-11T19:07:51.508382+0000","last_became_active":"2024-03-11T19:07:49.743568+0000","last_became_peered":"2024-03-11T19:07:49.743568+0000","last_unstale":"2024-03-11T19:07:51.508382+0000","last_undegraded":"2024-03-11T19:07:51.508382+0000","last_fullsized":"2024-03-11T19:07:51.508382+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:45:28.970286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509675+0000","last_change":"2024-03-11T19:07:49.750677+0000","last_active":"2024-03-11T19:07:51.509675+0000","last_peered":"2024-03-11T19:07:51.509675+0000","last_clean":"2024-03-11T19:07:51.509675+0000","last_became_active":"2024-03-11T19:07:49.750373+0000","last_became_peered":"2024-03-11T19:07:49.750373+0000","last_unstale":"2024-03-11T19:07:51.509675+0000","last_undegraded":"2024-03-11T19:07:51.509675+0000","last_fullsized":"2024-03-11T19:07:51.509675+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:07:28.767502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.743570+0000","last_change":"2024-03-11T19:07:49.745278+0000","last_active":"2024-03-11T19:07:50.743570+0000","last_peered":"2024-03-11T19:07:50.743570+0000","last_clean":"2024-03-11T19:07:50.743570+0000","last_became_active":"2024-03-11T19:07:49.744998+0000","last_became_peered":"2024-03-11T19:07:49.744998+0000","last_unstale":"2024-03-11T19:07:50.743570+0000","last_undegraded":"2024-03-11T19:07:50.743570+0000","last_fullsized":"2024-03-11T19:07:50.743570+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:23.778051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.509608+0000","last_change":"2024-03-11T19:07:49.734318+0000","last_active":"2024-03-11T19:07:51.509608+0000","last_peered":"2024-03-11T19:07:51.509608+0000","last_clean":"2024-03-11T19:07:51.509608+0000","last_became_active":"2024-03-11T19:07:49.733998+0000","last_became_peered":"2024-03-11T19:07:49.733998+0000","last_unstale":"2024-03-11T19:07:51.509608+0000","last_undegraded":"2024-03-11T19:07:51.509608+0000","last_fullsized":"2024-03-11T19:07:51.509608+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:48:50.308499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:50.743448+0000","last_change":"2024-03-11T19:07:49.741009+0000","last_active":"2024-03-11T19:07:50.743448+0000","last_peered":"2024-03-11T19:07:50.743448+0000","last_clean":"2024-03-11T19:07:50.743448+0000","last_became_active":"2024-03-11T19:07:49.740094+0000","last_became_peered":"2024-03-11T19:07:49.740094+0000","last_unstale":"2024-03-11T19:07:50.743448+0000","last_undegraded":"2024-03-11T19:07:50.743448+0000","last_fullsized":"2024-03-11T19:07:50.743448+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:06:31.945387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508055+0000","last_change":"2024-03-11T19:07:49.759300+0000","last_active":"2024-03-11T19:07:51.508055+0000","last_peered":"2024-03-11T19:07:51.508055+0000","last_clean":"2024-03-11T19:07:51.508055+0000","last_became_active":"2024-03-11T19:07:49.758011+0000","last_became_peered":"2024-03-11T19:07:49.758011+0000","last_unstale":"2024-03-11T19:07:51.508055+0000","last_undegraded":"2024-03-11T19:07:51.508055+0000","last_fullsized":"2024-03-11T19:07:51.508055+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:03:40.651816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508121+0000","last_change":"2024-03-11T19:07:49.752905+0000","last_active":"2024-03-11T19:07:51.508121+0000","last_peered":"2024-03-11T19:07:51.508121+0000","last_clean":"2024-03-11T19:07:51.508121+0000","last_became_active":"2024-03-11T19:07:49.752383+0000","last_became_peered":"2024-03-11T19:07:49.752383+0000","last_unstale":"2024-03-11T19:07:51.508121+0000","last_undegraded":"2024-03-11T19:07:51.508121+0000","last_fullsized":"2024-03-11T19:07:51.508121+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:45:33.475526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-11T19:07:51.508665+0000","last_change":"2024-03-11T19:07:49.744957+0000","last_active":"2024-03-11T19:07:51.508665+0000","last_peered":"2024-03-11T19:07:51.508665+0000","last_clean":"2024-03-11T19:07:51.508665+0000","last_became_active":"2024-03-11T19:07:49.744627+0000","last_became_peered":"2024-03-11T19:07:49.744627+0000","last_unstale":"2024-03-11T19:07:51.508665+0000","last_undegraded":"2024-03-11T19:07:51.508665+0000","last_fullsized":"2024-03-11T19:07:51.508665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:27:47.817004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245707,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":41401,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":6,"num_read_kb":24,"num_write":154,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":53297,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318286,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":933888,"data_stored":446066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579923,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1076480,"data_compressed":37658,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743449,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":446974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005086,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":446804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266723,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":1038898,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430248,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36248,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708904,"statfs":{"total":95995035648,"available":95957917696,"internally_reserved":0,"allocated":933888,"data_stored":446066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":946176,"data_stored":447678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":974848,"data_stored":448244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":950272,"data_stored":447712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36268,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708884,"statfs":{"total":95995035648,"available":95957897216,"internally_reserved":0,"allocated":942080,"data_stored":446170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888114,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36252,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708900,"statfs":{"total":95995035648,"available":95957913600,"internally_reserved":0,"allocated":937984,"data_stored":446088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691885,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":1037910,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":39929,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T19:07:58.609 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-11T19:07:58.610 INFO:tasks.cephadm:Setting up client nodes... 2024-03-11T19:07:58.610 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T19:07:58.890 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:07:58.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:58 smithi029 ceph-mon[32439]: pgmap v294: 97 pgs: 97 active+clean; 617 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 9.7 KiB/s wr, 20 op/s 2024-03-11T19:07:58.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:58 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3966982517' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:07:58.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:58 smithi029 ceph-mon[32439]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:07:58.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:07:58 smithi029 ceph-mon[32439]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:07:59.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:58 smithi170 ceph-mon[35690]: pgmap v294: 97 pgs: 97 active+clean; 617 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 9.7 KiB/s wr, 20 op/s 2024-03-11T19:07:59.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:58 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3966982517' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:07:59.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:58 smithi170 ceph-mon[35690]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:07:59.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:07:58 smithi170 ceph-mon[35690]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:07:59.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:58 smithi190 ceph-mon[36292]: pgmap v294: 97 pgs: 97 active+clean; 617 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 9.7 KiB/s wr, 20 op/s 2024-03-11T19:07:59.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:58 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3966982517' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:07:59.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:58 smithi190 ceph-mon[36292]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:07:59.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:07:58 smithi190 ceph-mon[36292]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:08:00.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.616+0000 7f6e08ff9700 1 -- 172.21.15.29:0/3076346717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e04101df0 msgr2=0x7f6e04102250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:00.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.616+0000 7f6e08ff9700 1 --2- 172.21.15.29:0/3076346717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e04101df0 0x7f6e04102250 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6df80099c0 tx=0x7f6df8009cd0 comp rx=0 tx=0).stop 2024-03-11T19:08:00.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.616+0000 7f6e08ff9700 1 -- 172.21.15.29:0/3076346717 shutdown_connections 2024-03-11T19:08:00.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.616+0000 7f6e08ff9700 1 --2- 172.21.15.29:0/3076346717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e04102790 0x7f6e04109040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:00.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.616+0000 7f6e08ff9700 1 --2- 172.21.15.29:0/3076346717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e04100bf0 0x7f6e04101010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:00.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.616+0000 7f6e08ff9700 1 --2- 172.21.15.29:0/3076346717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e04101df0 0x7f6e04102250 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:00.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.616+0000 7f6e08ff9700 1 -- 172.21.15.29:0/3076346717 >> 172.21.15.29:0/3076346717 conn(0x7f6e040fc190 msgr2=0x7f6e040fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:00.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.616+0000 7f6e08ff9700 1 -- 172.21.15.29:0/3076346717 shutdown_connections 2024-03-11T19:08:00.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.616+0000 7f6e08ff9700 1 -- 172.21.15.29:0/3076346717 wait complete. 2024-03-11T19:08:00.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.617+0000 7f6e08ff9700 1 Processor -- start 2024-03-11T19:08:00.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.617+0000 7f6e08ff9700 1 -- start start 2024-03-11T19:08:00.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.617+0000 7f6e08ff9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e04100bf0 0x7f6e0406af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:00.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.617+0000 7f6e08ff9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e04101df0 0x7f6e0419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:00.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.617+0000 7f6e08ff9700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e04102790 0x7f6e0419ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:00.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.617+0000 7f6e08ff9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0419f5b0 con 0x7f6e04100bf0 2024-03-11T19:08:00.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.617+0000 7f6e08ff9700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0419f720 con 0x7f6e04102790 2024-03-11T19:08:00.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.617+0000 7f6e08ff9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0419f8e0 con 0x7f6e04101df0 2024-03-11T19:08:00.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.617+0000 7f6e0259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e04100bf0 0x7f6e0406af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:00.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.617+0000 7f6e0259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e04100bf0 0x7f6e0406af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60870/0 (socket says 172.21.15.29:60870) 2024-03-11T19:08:00.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.617+0000 7f6e0259c700 1 -- 172.21.15.29:0/4184023720 learned_addr learned my addr 172.21.15.29:0/4184023720 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:00.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.617+0000 7f6e01d9b700 1 --2- 172.21.15.29:0/4184023720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e04101df0 0x7f6e0419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:00.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.618+0000 7f6e02d9d700 1 --2- 172.21.15.29:0/4184023720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e04102790 0x7f6e0419ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:00.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.618+0000 7f6e0259c700 1 -- 172.21.15.29:0/4184023720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e04101df0 msgr2=0x7f6e0419c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:00.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.618+0000 7f6e0259c700 1 --2- 172.21.15.29:0/4184023720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e04101df0 0x7f6e0419c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:00.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.618+0000 7f6e0259c700 1 -- 172.21.15.29:0/4184023720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e04102790 msgr2=0x7f6e0419ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:00.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.618+0000 7f6e0259c700 1 --2- 172.21.15.29:0/4184023720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e04102790 0x7f6e0419ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:00.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.618+0000 7f6e0259c700 1 -- 172.21.15.29:0/4184023720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df80095f0 con 0x7f6e04100bf0 2024-03-11T19:08:00.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.618+0000 7f6e02d9d700 1 --2- 172.21.15.29:0/4184023720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e04102790 0x7f6e0419ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:08:00.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.618+0000 7f6e0259c700 1 --2- 172.21.15.29:0/4184023720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e04100bf0 0x7f6e0406af10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6df400e970 tx=0x7f6df400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:00.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.618+0000 7f6df37fe700 1 -- 172.21.15.29:0/4184023720 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df4009790 con 0x7f6e04100bf0 2024-03-11T19:08:00.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.618+0000 7f6df37fe700 1 -- 172.21.15.29:0/4184023720 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6df400eec0 con 0x7f6e04100bf0 2024-03-11T19:08:00.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.618+0000 7f6df37fe700 1 -- 172.21.15.29:0/4184023720 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df4003d20 con 0x7f6e04100bf0 2024-03-11T19:08:00.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.618+0000 7f6e08ff9700 1 -- 172.21.15.29:0/4184023720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0419fbc0 con 0x7f6e04100bf0 2024-03-11T19:08:00.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.618+0000 7f6e08ff9700 1 -- 172.21.15.29:0/4184023720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e041a0110 con 0x7f6e04100bf0 2024-03-11T19:08:00.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.619+0000 7f6e08ff9700 1 -- 172.21.15.29:0/4184023720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e04107f50 con 0x7f6e04100bf0 2024-03-11T19:08:00.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.620+0000 7f6df37fe700 1 -- 172.21.15.29:0/4184023720 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f6df4003e80 con 0x7f6e04100bf0 2024-03-11T19:08:00.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.620+0000 7f6df37fe700 1 --2- 172.21.15.29:0/4184023720 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6dec0a1c80 0x7f6dec0a4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:00.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.621+0000 7f6e01d9b700 1 --2- 172.21.15.29:0/4184023720 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6dec0a1c80 0x7f6dec0a4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:00.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.621+0000 7f6df37fe700 1 -- 172.21.15.29:0/4184023720 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f6df4014070 con 0x7f6e04100bf0 2024-03-11T19:08:00.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.621+0000 7f6e01d9b700 1 --2- 172.21.15.29:0/4184023720 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6dec0a1c80 0x7f6dec0a4140 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6df800b580 tx=0x7f6df8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:00.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.625+0000 7f6df37fe700 1 -- 172.21.15.29:0/4184023720 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6df4096840 con 0x7f6e04100bf0 2024-03-11T19:08:00.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:00 smithi029 ceph-mon[32439]: pgmap v295: 97 pgs: 97 active+clean; 619 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 10 KiB/s wr, 23 op/s 2024-03-11T19:08:00.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.911+0000 7f6e08ff9700 1 -- 172.21.15.29:0/4184023720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f6e0410c4b0 con 0x7f6e04100bf0 2024-03-11T19:08:00.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.913+0000 7f6df37fe700 1 -- 172.21.15.29:0/4184023720 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f6df4099f00 con 0x7f6e04100bf0 2024-03-11T19:08:00.916 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2024-03-11T19:08:00.916 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCQVu9ljXJlNhAAVBEc6LheqWwJDOkoEo93NQ== 2024-03-11T19:08:00.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.915+0000 7f6e08ff9700 1 -- 172.21.15.29:0/4184023720 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6dec0a1c80 msgr2=0x7f6dec0a4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:00.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.915+0000 7f6e08ff9700 1 --2- 172.21.15.29:0/4184023720 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6dec0a1c80 0x7f6dec0a4140 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6df800b580 tx=0x7f6df8005fb0 comp rx=0 tx=0).stop 2024-03-11T19:08:00.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.915+0000 7f6e08ff9700 1 -- 172.21.15.29:0/4184023720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e04100bf0 msgr2=0x7f6e0406af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:00.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.915+0000 7f6e08ff9700 1 --2- 172.21.15.29:0/4184023720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e04100bf0 0x7f6e0406af10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6df400e970 tx=0x7f6df400ec80 comp rx=0 tx=0).stop 2024-03-11T19:08:00.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.916+0000 7f6e08ff9700 1 -- 172.21.15.29:0/4184023720 shutdown_connections 2024-03-11T19:08:00.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.916+0000 7f6e08ff9700 1 --2- 172.21.15.29:0/4184023720 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6dec0a1c80 0x7f6dec0a4140 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:00.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.916+0000 7f6e08ff9700 1 --2- 172.21.15.29:0/4184023720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e04102790 0x7f6e0419ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:00.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.916+0000 7f6e08ff9700 1 --2- 172.21.15.29:0/4184023720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e04100bf0 0x7f6e0406af10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:00.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.916+0000 7f6e08ff9700 1 --2- 172.21.15.29:0/4184023720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e04101df0 0x7f6e0419c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:00.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.916+0000 7f6e08ff9700 1 -- 172.21.15.29:0/4184023720 >> 172.21.15.29:0/4184023720 conn(0x7f6e040fc190 msgr2=0x7f6e04105090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:00.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.916+0000 7f6e08ff9700 1 -- 172.21.15.29:0/4184023720 shutdown_connections 2024-03-11T19:08:00.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:00.916+0000 7f6e08ff9700 1 -- 172.21.15.29:0/4184023720 wait complete. 2024-03-11T19:08:01.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:00 smithi170 ceph-mon[35690]: pgmap v295: 97 pgs: 97 active+clean; 619 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 10 KiB/s wr, 23 op/s 2024-03-11T19:08:01.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:00 smithi190 ceph-mon[36292]: pgmap v295: 97 pgs: 97 active+clean; 619 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 10 KiB/s wr, 23 op/s 2024-03-11T19:08:01.608 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-11T19:08:01.608 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-11T19:08:01.608 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-11T19:08:01.651 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-11T19:08:01.652 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-11T19:08:01.652 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-11T19:08:01.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:01 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/4184023720' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T19:08:01.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:01 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/4184023720' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T19:08:01.935 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:02.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:01 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/4184023720' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T19:08:02.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:01 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/4184023720' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T19:08:02.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:01 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/4184023720' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T19:08:02.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:01 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/4184023720' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T19:08:03.122 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:02 smithi029 ceph-mon[32439]: pgmap v296: 97 pgs: 97 active+clean; 621 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 9.7 KiB/s wr, 23 op/s 2024-03-11T19:08:03.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:02 smithi190 ceph-mon[36292]: pgmap v296: 97 pgs: 97 active+clean; 621 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 9.7 KiB/s wr, 23 op/s 2024-03-11T19:08:03.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:02 smithi170 ceph-mon[35690]: pgmap v296: 97 pgs: 97 active+clean; 621 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 9.7 KiB/s wr, 23 op/s 2024-03-11T19:08:03.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.688+0000 7fe93ea49700 1 -- 172.21.15.29:0/2249988595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9381056d0 msgr2=0x7fe93810d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:03.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.688+0000 7fe93ea49700 1 --2- 172.21.15.29:0/2249988595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9381056d0 0x7fe93810d210 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe93000b0d0 tx=0x7fe93000b3e0 comp rx=0 tx=0).stop 2024-03-11T19:08:03.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.688+0000 7fe93ea49700 1 -- 172.21.15.29:0/2249988595 shutdown_connections 2024-03-11T19:08:03.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.688+0000 7fe93ea49700 1 --2- 172.21.15.29:0/2249988595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9381056d0 0x7fe93810d210 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:03.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.688+0000 7fe93ea49700 1 --2- 172.21.15.29:0/2249988595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe938100800 0x7fe938100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:03.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.688+0000 7fe93ea49700 1 --2- 172.21.15.29:0/2249988595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9381011b0 0x7fe938104fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:03.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.688+0000 7fe93ea49700 1 -- 172.21.15.29:0/2249988595 >> 172.21.15.29:0/2249988595 conn(0x7fe938075ea0 msgr2=0x7fe9380762b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:03.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.688+0000 7fe93ea49700 1 -- 172.21.15.29:0/2249988595 shutdown_connections 2024-03-11T19:08:03.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.688+0000 7fe93ea49700 1 -- 172.21.15.29:0/2249988595 wait complete. 2024-03-11T19:08:03.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.689+0000 7fe93ea49700 1 Processor -- start 2024-03-11T19:08:03.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.689+0000 7fe93ea49700 1 -- start start 2024-03-11T19:08:03.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.689+0000 7fe93ea49700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe938100800 0x7fe938197060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:03.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.689+0000 7fe93ea49700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9381011b0 0x7fe9381975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:03.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.689+0000 7fe93ea49700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9381056d0 0x7fe93819b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:03.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.689+0000 7fe93ea49700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93819bf40 con 0x7fe938100800 2024-03-11T19:08:03.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.689+0000 7fe93ea49700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93819c0b0 con 0x7fe9381056d0 2024-03-11T19:08:03.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.689+0000 7fe93ea49700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93819c270 con 0x7fe9381011b0 2024-03-11T19:08:03.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.689+0000 7fe937fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe938100800 0x7fe938197060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:03.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.689+0000 7fe937fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe938100800 0x7fe938197060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60888/0 (socket says 172.21.15.29:60888) 2024-03-11T19:08:03.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.689+0000 7fe937fff700 1 -- 172.21.15.29:0/1654495789 learned_addr learned my addr 172.21.15.29:0/1654495789 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:03.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.689+0000 7fe9377fe700 1 --2- 172.21.15.29:0/1654495789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9381011b0 0x7fe9381975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:03.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.689+0000 7fe93cfe6700 1 --2- 172.21.15.29:0/1654495789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9381056d0 0x7fe93819b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:03.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.689+0000 7fe937fff700 1 -- 172.21.15.29:0/1654495789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9381011b0 msgr2=0x7fe9381975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:03.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.689+0000 7fe937fff700 1 --2- 172.21.15.29:0/1654495789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9381011b0 0x7fe9381975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:03.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.689+0000 7fe937fff700 1 -- 172.21.15.29:0/1654495789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9381056d0 msgr2=0x7fe93819b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:03.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.689+0000 7fe937fff700 1 --2- 172.21.15.29:0/1654495789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9381056d0 0x7fe93819b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:03.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.690+0000 7fe937fff700 1 -- 172.21.15.29:0/1654495789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe930009d00 con 0x7fe938100800 2024-03-11T19:08:03.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.690+0000 7fe937fff700 1 --2- 172.21.15.29:0/1654495789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe938100800 0x7fe938197060 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe92800d950 tx=0x7fe92800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:03.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.690+0000 7fe9357fa700 1 -- 172.21.15.29:0/1654495789 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92800f890 con 0x7fe938100800 2024-03-11T19:08:03.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.690+0000 7fe9357fa700 1 -- 172.21.15.29:0/1654495789 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe928010460 con 0x7fe938100800 2024-03-11T19:08:03.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.690+0000 7fe93ea49700 1 -- 172.21.15.29:0/1654495789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93819c550 con 0x7fe938100800 2024-03-11T19:08:03.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.690+0000 7fe9357fa700 1 -- 172.21.15.29:0/1654495789 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92800fea0 con 0x7fe938100800 2024-03-11T19:08:03.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.690+0000 7fe93ea49700 1 -- 172.21.15.29:0/1654495789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9381a3db0 con 0x7fe938100800 2024-03-11T19:08:03.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.691+0000 7fe93ea49700 1 -- 172.21.15.29:0/1654495789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe938108840 con 0x7fe938100800 2024-03-11T19:08:03.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.692+0000 7fe9357fa700 1 -- 172.21.15.29:0/1654495789 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fe9280105d0 con 0x7fe938100800 2024-03-11T19:08:03.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.693+0000 7fe9357fa700 1 --2- 172.21.15.29:0/1654495789 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe9240a1bd0 0x7fe9240a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:03.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.693+0000 7fe9377fe700 1 --2- 172.21.15.29:0/1654495789 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe9240a1bd0 0x7fe9240a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:03.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.693+0000 7fe9357fa700 1 -- 172.21.15.29:0/1654495789 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe9280cd0f0 con 0x7fe938100800 2024-03-11T19:08:03.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.693+0000 7fe9377fe700 1 --2- 172.21.15.29:0/1654495789 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe9240a1bd0 0x7fe9240a4090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe938100190 tx=0x7fe9200073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:03.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.697+0000 7fe9357fa700 1 -- 172.21.15.29:0/1654495789 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe928096350 con 0x7fe938100800 2024-03-11T19:08:03.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.961+0000 7fe93ea49700 1 -- 172.21.15.29:0/1654495789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fe9381983c0 con 0x7fe938100800 2024-03-11T19:08:03.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.965+0000 7fe9357fa700 1 -- 172.21.15.29:0/1654495789 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260662 (secure 0 0 0) 0x7fe928099a10 con 0x7fe938100800 2024-03-11T19:08:03.973 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:08:03.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.968+0000 7fe93ea49700 1 -- 172.21.15.29:0/1654495789 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe9240a1bd0 msgr2=0x7fe9240a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:03.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.968+0000 7fe93ea49700 1 --2- 172.21.15.29:0/1654495789 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe9240a1bd0 0x7fe9240a4090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe938100190 tx=0x7fe9200073d0 comp rx=0 tx=0).stop 2024-03-11T19:08:03.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.968+0000 7fe93ea49700 1 -- 172.21.15.29:0/1654495789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe938100800 msgr2=0x7fe938197060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:03.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.968+0000 7fe93ea49700 1 --2- 172.21.15.29:0/1654495789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe938100800 0x7fe938197060 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe92800d950 tx=0x7fe92800dc60 comp rx=0 tx=0).stop 2024-03-11T19:08:03.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.969+0000 7fe93ea49700 1 -- 172.21.15.29:0/1654495789 shutdown_connections 2024-03-11T19:08:03.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.969+0000 7fe93ea49700 1 --2- 172.21.15.29:0/1654495789 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe9240a1bd0 0x7fe9240a4090 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:03.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.969+0000 7fe93ea49700 1 --2- 172.21.15.29:0/1654495789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9381056d0 0x7fe93819b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:03.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.969+0000 7fe93ea49700 1 --2- 172.21.15.29:0/1654495789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe938100800 0x7fe938197060 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:03.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.969+0000 7fe93ea49700 1 --2- 172.21.15.29:0/1654495789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9381011b0 0x7fe9381975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:03.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.969+0000 7fe93ea49700 1 -- 172.21.15.29:0/1654495789 >> 172.21.15.29:0/1654495789 conn(0x7fe938075ea0 msgr2=0x7fe9381041f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:03.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.969+0000 7fe93ea49700 1 -- 172.21.15.29:0/1654495789 shutdown_connections 2024-03-11T19:08:03.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:03.969+0000 7fe93ea49700 1 -- 172.21.15.29:0/1654495789 wait complete. 2024-03-11T19:08:04.683 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":1587871898},{"type":"v1","addr":"172.21.15.29:6801","nonce":1587871898}]},"active_addr":"172.21.15.29:6801/1587871898","active_change":"2024-03-11T19:00:43.524986+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24121,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.29:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":4241238840}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":999793992}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":1189004412}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":2478434788}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":1735841543}]}]} 2024-03-11T19:08:04.689 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-11T19:08:04.689 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T19:08:04.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T19:08:04.934 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:05.187 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:04 smithi029 ceph-mon[32439]: pgmap v297: 97 pgs: 97 active+clean; 626 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 9.2 KiB/s wr, 23 op/s 2024-03-11T19:08:05.187 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:04 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1654495789' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T19:08:05.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:04 smithi190 ceph-mon[36292]: pgmap v297: 97 pgs: 97 active+clean; 626 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 9.2 KiB/s wr, 23 op/s 2024-03-11T19:08:05.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:04 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1654495789' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T19:08:05.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:04 smithi170 ceph-mon[35690]: pgmap v297: 97 pgs: 97 active+clean; 626 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 9.2 KiB/s wr, 23 op/s 2024-03-11T19:08:05.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:04 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1654495789' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T19:08:06.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.798+0000 7ff9b1d99700 1 -- 172.21.15.29:0/1917599134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac100b70 msgr2=0x7ff9ac100ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:06.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.798+0000 7ff9b1d99700 1 --2- 172.21.15.29:0/1917599134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac100b70 0x7ff9ac100ff0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0009990 tx=0x7ff9a0009ca0 comp rx=0 tx=0).stop 2024-03-11T19:08:06.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.798+0000 7ff9b1d99700 1 -- 172.21.15.29:0/1917599134 shutdown_connections 2024-03-11T19:08:06.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.798+0000 7ff9b1d99700 1 --2- 172.21.15.29:0/1917599134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9ac101530 0x7ff9ac10d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:06.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.798+0000 7ff9b1d99700 1 --2- 172.21.15.29:0/1917599134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9ac106b80 0x7ff9ac106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:06.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.798+0000 7ff9b1d99700 1 --2- 172.21.15.29:0/1917599134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac100b70 0x7ff9ac100ff0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:06.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.798+0000 7ff9b1d99700 1 -- 172.21.15.29:0/1917599134 >> 172.21.15.29:0/1917599134 conn(0x7ff9ac077aa0 msgr2=0x7ff9ac077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:06.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.798+0000 7ff9b1d99700 1 -- 172.21.15.29:0/1917599134 shutdown_connections 2024-03-11T19:08:06.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.798+0000 7ff9b1d99700 1 -- 172.21.15.29:0/1917599134 wait complete. 2024-03-11T19:08:06.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.799+0000 7ff9b1d99700 1 Processor -- start 2024-03-11T19:08:06.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.799+0000 7ff9b1d99700 1 -- start start 2024-03-11T19:08:06.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.799+0000 7ff9b1d99700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac100b70 0x7ff9ac1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:06.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.799+0000 7ff9b1d99700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9ac101530 0x7ff9ac1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:06.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.799+0000 7ff9b1d99700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9ac106b80 0x7ff9ac19c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:06.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.799+0000 7ff9b1d99700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac19cf80 con 0x7ff9ac100b70 2024-03-11T19:08:06.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.799+0000 7ff9b1d99700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac19d0f0 con 0x7ff9ac106b80 2024-03-11T19:08:06.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.799+0000 7ff9b1d99700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac19d2b0 con 0x7ff9ac101530 2024-03-11T19:08:06.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.799+0000 7ff9ab7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac100b70 0x7ff9ac1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:06.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.799+0000 7ff9ab7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac100b70 0x7ff9ac1980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56214/0 (socket says 172.21.15.29:56214) 2024-03-11T19:08:06.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.799+0000 7ff9ab7fe700 1 -- 172.21.15.29:0/2475848640 learned_addr learned my addr 172.21.15.29:0/2475848640 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:06.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.799+0000 7ff9abfff700 1 --2- 172.21.15.29:0/2475848640 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9ac106b80 0x7ff9ac19c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:06.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.799+0000 7ff9aaffd700 1 --2- 172.21.15.29:0/2475848640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9ac101530 0x7ff9ac1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:06.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.799+0000 7ff9ab7fe700 1 -- 172.21.15.29:0/2475848640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9ac101530 msgr2=0x7ff9ac1985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:06.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.799+0000 7ff9ab7fe700 1 --2- 172.21.15.29:0/2475848640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9ac101530 0x7ff9ac1985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:06.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.799+0000 7ff9ab7fe700 1 -- 172.21.15.29:0/2475848640 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9ac106b80 msgr2=0x7ff9ac19c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:06.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.799+0000 7ff9ab7fe700 1 --2- 172.21.15.29:0/2475848640 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9ac106b80 0x7ff9ac19c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:06.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.799+0000 7ff9ab7fe700 1 -- 172.21.15.29:0/2475848640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a00095f0 con 0x7ff9ac100b70 2024-03-11T19:08:06.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.800+0000 7ff9ab7fe700 1 --2- 172.21.15.29:0/2475848640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac100b70 0x7ff9ac1980a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff99c00b530 tx=0x7ff99c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:06.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.800+0000 7ff9a8ff9700 1 -- 172.21.15.29:0/2475848640 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c011830 con 0x7ff9ac100b70 2024-03-11T19:08:06.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.800+0000 7ff9b1d99700 1 -- 172.21.15.29:0/2475848640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9ac19d590 con 0x7ff9ac100b70 2024-03-11T19:08:06.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.800+0000 7ff9a8ff9700 1 -- 172.21.15.29:0/2475848640 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff99c011e40 con 0x7ff9ac100b70 2024-03-11T19:08:06.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.800+0000 7ff9a8ff9700 1 -- 172.21.15.29:0/2475848640 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c00f2e0 con 0x7ff9ac100b70 2024-03-11T19:08:06.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.800+0000 7ff9b1d99700 1 -- 172.21.15.29:0/2475848640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ac1a4df0 con 0x7ff9ac100b70 2024-03-11T19:08:06.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.801+0000 7ff9b1d99700 1 -- 172.21.15.29:0/2475848640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9ac109020 con 0x7ff9ac100b70 2024-03-11T19:08:06.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.802+0000 7ff9a8ff9700 1 -- 172.21.15.29:0/2475848640 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7ff99c011990 con 0x7ff9ac100b70 2024-03-11T19:08:06.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.802+0000 7ff9a8ff9700 1 --2- 172.21.15.29:0/2475848640 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff9940a1b50 0x7ff9940a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:06.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.803+0000 7ff9aaffd700 1 --2- 172.21.15.29:0/2475848640 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff9940a1b50 0x7ff9940a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:06.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.803+0000 7ff9a8ff9700 1 -- 172.21.15.29:0/2475848640 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff99c0cccc0 con 0x7ff9ac100b70 2024-03-11T19:08:06.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.803+0000 7ff9aaffd700 1 --2- 172.21.15.29:0/2475848640 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff9940a1b50 0x7ff9940a4010 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff9a000b5c0 tx=0x7ff9a0005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:06.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:06.807+0000 7ff9a8ff9700 1 -- 172.21.15.29:0/2475848640 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff99c096030 con 0x7ff9ac100b70 2024-03-11T19:08:07.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:07.015+0000 7ff9b1d99700 1 -- 172.21.15.29:0/2475848640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9ac068930 con 0x7ff9ac100b70 2024-03-11T19:08:07.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:07.016+0000 7ff9a8ff9700 1 -- 172.21.15.29:0/2475848640 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22219 (secure 0 0 0) 0x7ff99c0996f0 con 0x7ff9ac100b70 2024-03-11T19:08:07.019 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:08:07.020 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":87,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:07:50.732035+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:07:47.563184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:07:48.282461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:08:07.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:07.018+0000 7ff9b1d99700 1 -- 172.21.15.29:0/2475848640 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff9940a1b50 msgr2=0x7ff9940a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:07.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:07.018+0000 7ff9b1d99700 1 --2- 172.21.15.29:0/2475848640 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff9940a1b50 0x7ff9940a4010 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff9a000b5c0 tx=0x7ff9a0005960 comp rx=0 tx=0).stop 2024-03-11T19:08:07.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:07.018+0000 7ff9b1d99700 1 -- 172.21.15.29:0/2475848640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac100b70 msgr2=0x7ff9ac1980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:07.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:07.018+0000 7ff9b1d99700 1 --2- 172.21.15.29:0/2475848640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac100b70 0x7ff9ac1980a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff99c00b530 tx=0x7ff99c00b840 comp rx=0 tx=0).stop 2024-03-11T19:08:07.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:07.019+0000 7ff9b1d99700 1 -- 172.21.15.29:0/2475848640 shutdown_connections 2024-03-11T19:08:07.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:07.019+0000 7ff9b1d99700 1 --2- 172.21.15.29:0/2475848640 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff9940a1b50 0x7ff9940a4010 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:07.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:07.019+0000 7ff9b1d99700 1 --2- 172.21.15.29:0/2475848640 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9ac106b80 0x7ff9ac19c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:07.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:07.019+0000 7ff9b1d99700 1 --2- 172.21.15.29:0/2475848640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac100b70 0x7ff9ac1980a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:07.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:07.019+0000 7ff9b1d99700 1 --2- 172.21.15.29:0/2475848640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9ac101530 0x7ff9ac1985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:07.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:07.019+0000 7ff9b1d99700 1 -- 172.21.15.29:0/2475848640 >> 172.21.15.29:0/2475848640 conn(0x7ff9ac077aa0 msgr2=0x7ff9ac0ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:07.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:07.019+0000 7ff9b1d99700 1 -- 172.21.15.29:0/2475848640 shutdown_connections 2024-03-11T19:08:07.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:07.019+0000 7ff9b1d99700 1 -- 172.21.15.29:0/2475848640 wait complete. 2024-03-11T19:08:07.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:06 smithi170 ceph-mon[35690]: pgmap v298: 97 pgs: 97 active+clean; 626 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.2 KiB/s wr, 23 op/s 2024-03-11T19:08:07.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:06 smithi029 ceph-mon[32439]: pgmap v298: 97 pgs: 97 active+clean; 626 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.2 KiB/s wr, 23 op/s 2024-03-11T19:08:07.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:06 smithi190 ceph-mon[36292]: pgmap v298: 97 pgs: 97 active+clean; 626 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.2 KiB/s wr, 23 op/s 2024-03-11T19:08:07.846 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T19:08:07.846 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T19:08:08.089 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:08.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:07 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2475848640' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:08:08.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:07 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2475848640' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:08:08.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:07 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2475848640' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:08:09.248 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:08 smithi029 ceph-mon[32439]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 21 op/s 2024-03-11T19:08:09.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:08 smithi190 ceph-mon[36292]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 21 op/s 2024-03-11T19:08:09.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:08 smithi170 ceph-mon[35690]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 21 op/s 2024-03-11T19:08:09.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.940+0000 7fde9dd6d700 1 -- 172.21.15.29:0/86670058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde980747c0 msgr2=0x7fde9810e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:09.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.940+0000 7fde9dd6d700 1 --2- 172.21.15.29:0/86670058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde980747c0 0x7fde9810e130 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fde8c009990 tx=0x7fde8c009ca0 comp rx=0 tx=0).stop 2024-03-11T19:08:09.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.940+0000 7fde9dd6d700 1 -- 172.21.15.29:0/86670058 shutdown_connections 2024-03-11T19:08:09.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.940+0000 7fde9dd6d700 1 --2- 172.21.15.29:0/86670058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde9810e670 0x7fde98110a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:09.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.940+0000 7fde9dd6d700 1 --2- 172.21.15.29:0/86670058 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde98073ea0 0x7fde98074280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:09.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.940+0000 7fde9dd6d700 1 --2- 172.21.15.29:0/86670058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde980747c0 0x7fde9810e130 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:09.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.940+0000 7fde9dd6d700 1 -- 172.21.15.29:0/86670058 >> 172.21.15.29:0/86670058 conn(0x7fde98078da0 msgr2=0x7fde980791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:09.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.940+0000 7fde9dd6d700 1 -- 172.21.15.29:0/86670058 shutdown_connections 2024-03-11T19:08:09.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.940+0000 7fde9dd6d700 1 -- 172.21.15.29:0/86670058 wait complete. 2024-03-11T19:08:09.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.940+0000 7fde9dd6d700 1 Processor -- start 2024-03-11T19:08:09.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.940+0000 7fde9dd6d700 1 -- start start 2024-03-11T19:08:09.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde9dd6d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde98073ea0 0x7fde9819a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:09.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde9dd6d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde980747c0 0x7fde9819a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:09.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde9dd6d700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde9810e670 0x7fde9819ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:09.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde9dd6d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9819f220 con 0x7fde980747c0 2024-03-11T19:08:09.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde9dd6d700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9819f390 con 0x7fde9810e670 2024-03-11T19:08:09.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde9dd6d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9819f550 con 0x7fde98073ea0 2024-03-11T19:08:09.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde96ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde980747c0 0x7fde9819a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:09.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde96ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde980747c0 0x7fde9819a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56224/0 (socket says 172.21.15.29:56224) 2024-03-11T19:08:09.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde96ffd700 1 -- 172.21.15.29:0/642751137 learned_addr learned my addr 172.21.15.29:0/642751137 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:09.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde97fff700 1 --2- 172.21.15.29:0/642751137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde9810e670 0x7fde9819ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:09.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde977fe700 1 --2- 172.21.15.29:0/642751137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde98073ea0 0x7fde9819a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:09.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde96ffd700 1 -- 172.21.15.29:0/642751137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde98073ea0 msgr2=0x7fde9819a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:09.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde96ffd700 1 --2- 172.21.15.29:0/642751137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde98073ea0 0x7fde9819a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:09.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde96ffd700 1 -- 172.21.15.29:0/642751137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde9810e670 msgr2=0x7fde9819ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:09.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde96ffd700 1 --2- 172.21.15.29:0/642751137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde9810e670 0x7fde9819ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:09.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde96ffd700 1 -- 172.21.15.29:0/642751137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c0095f0 con 0x7fde980747c0 2024-03-11T19:08:09.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde96ffd700 1 --2- 172.21.15.29:0/642751137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde980747c0 0x7fde9819a8d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fde8c000c00 tx=0x7fde8c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:09.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde94ff9700 1 -- 172.21.15.29:0/642751137 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c01d070 con 0x7fde980747c0 2024-03-11T19:08:09.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde94ff9700 1 -- 172.21.15.29:0/642751137 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fde8c022470 con 0x7fde980747c0 2024-03-11T19:08:09.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde9dd6d700 1 -- 172.21.15.29:0/642751137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde9819f7d0 con 0x7fde980747c0 2024-03-11T19:08:09.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.941+0000 7fde94ff9700 1 -- 172.21.15.29:0/642751137 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c017650 con 0x7fde980747c0 2024-03-11T19:08:09.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.942+0000 7fde9dd6d700 1 -- 172.21.15.29:0/642751137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde981a7090 con 0x7fde980747c0 2024-03-11T19:08:09.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.943+0000 7fde9dd6d700 1 -- 172.21.15.29:0/642751137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde98109880 con 0x7fde980747c0 2024-03-11T19:08:09.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.943+0000 7fde94ff9700 1 -- 172.21.15.29:0/642751137 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fde8c0225e0 con 0x7fde980747c0 2024-03-11T19:08:09.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.944+0000 7fde94ff9700 1 --2- 172.21.15.29:0/642751137 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fde800a1b50 0x7fde800a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:09.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.944+0000 7fde977fe700 1 --2- 172.21.15.29:0/642751137 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fde800a1b50 0x7fde800a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:09.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.944+0000 7fde94ff9700 1 -- 172.21.15.29:0/642751137 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fde8c0ce670 con 0x7fde980747c0 2024-03-11T19:08:09.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.944+0000 7fde977fe700 1 --2- 172.21.15.29:0/642751137 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fde800a1b50 0x7fde800a4010 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fde88009bf0 tx=0x7fde88009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:09.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:09.948+0000 7fde94ff9700 1 -- 172.21.15.29:0/642751137 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fde8c0979f0 con 0x7fde980747c0 2024-03-11T19:08:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:10.150+0000 7fde9dd6d700 1 -- 172.21.15.29:0/642751137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde98068930 con 0x7fde980747c0 2024-03-11T19:08:10.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:10.152+0000 7fde94ff9700 1 -- 172.21.15.29:0/642751137 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22219 (secure 0 0 0) 0x7fde8c09b0b0 con 0x7fde980747c0 2024-03-11T19:08:10.155 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:08:10.156 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":87,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:07:50.732035+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:07:47.563184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:07:48.282461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:08:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:10.155+0000 7fde9dd6d700 1 -- 172.21.15.29:0/642751137 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fde800a1b50 msgr2=0x7fde800a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:10.155+0000 7fde9dd6d700 1 --2- 172.21.15.29:0/642751137 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fde800a1b50 0x7fde800a4010 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fde88009bf0 tx=0x7fde88009260 comp rx=0 tx=0).stop 2024-03-11T19:08:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:10.155+0000 7fde9dd6d700 1 -- 172.21.15.29:0/642751137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde980747c0 msgr2=0x7fde9819a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:10.155+0000 7fde9dd6d700 1 --2- 172.21.15.29:0/642751137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde980747c0 0x7fde9819a8d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fde8c000c00 tx=0x7fde8c004730 comp rx=0 tx=0).stop 2024-03-11T19:08:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:10.155+0000 7fde9dd6d700 1 -- 172.21.15.29:0/642751137 shutdown_connections 2024-03-11T19:08:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:10.155+0000 7fde9dd6d700 1 --2- 172.21.15.29:0/642751137 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fde800a1b50 0x7fde800a4010 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:10.155+0000 7fde9dd6d700 1 --2- 172.21.15.29:0/642751137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde9810e670 0x7fde9819ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:10.155+0000 7fde9dd6d700 1 --2- 172.21.15.29:0/642751137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde98073ea0 0x7fde9819a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:10.155+0000 7fde9dd6d700 1 --2- 172.21.15.29:0/642751137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde980747c0 0x7fde9819a8d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:10.155+0000 7fde9dd6d700 1 -- 172.21.15.29:0/642751137 >> 172.21.15.29:0/642751137 conn(0x7fde98078da0 msgr2=0x7fde980fef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:10.155+0000 7fde9dd6d700 1 -- 172.21.15.29:0/642751137 shutdown_connections 2024-03-11T19:08:10.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:10.155+0000 7fde9dd6d700 1 -- 172.21.15.29:0/642751137 wait complete. 2024-03-11T19:08:10.968 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T19:08:10.968 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T19:08:10.969 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T19:08:10.969 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-11T19:08:10.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-11T19:08:10.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-11T19:08:10.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-11T19:08:10.971 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-11T19:08:10.971 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-11T19:08:10.972 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-11T19:08:10.972 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-11T19:08:10.972 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-11T19:08:11.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:10 smithi190 ceph-mon[36292]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 KiB/s wr, 18 op/s 2024-03-11T19:08:11.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:10 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/642751137' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:08:11.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:10 smithi170 ceph-mon[35690]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 KiB/s wr, 18 op/s 2024-03-11T19:08:11.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:10 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/642751137' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:08:11.408 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:10 smithi029 ceph-mon[32439]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 KiB/s wr, 18 op/s 2024-03-11T19:08:11.408 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:10 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/642751137' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:08:11.676 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:11.681 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:11.681 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:11.769 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:12.933 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:13.121 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:12 smithi029 ceph-mon[32439]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-03-11T19:08:13.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:12 smithi190 ceph-mon[36292]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-03-11T19:08:13.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:12 smithi170 ceph-mon[35690]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-03-11T19:08:13.449 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:13.449 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:13.449 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:15.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:14 smithi029 ceph-mon[32439]: pgmap v302: 97 pgs: 97 active+clean; 638 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-03-11T19:08:15.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:14 smithi190 ceph-mon[36292]: pgmap v302: 97 pgs: 97 active+clean; 638 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-03-11T19:08:15.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:14 smithi170 ceph-mon[35690]: pgmap v302: 97 pgs: 97 active+clean; 638 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-03-11T19:08:16.670 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:16.684 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:16.684 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:17.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:16 smithi029 ceph-mon[32439]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-11T19:08:17.216 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:17.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:16 smithi190 ceph-mon[36292]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-11T19:08:17.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:16 smithi170 ceph-mon[35690]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-11T19:08:19.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:18 smithi029 ceph-mon[32439]: pgmap v304: 97 pgs: 97 active+clean; 641 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-11T19:08:19.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:18 smithi190 ceph-mon[36292]: pgmap v304: 97 pgs: 97 active+clean; 641 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-11T19:08:19.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:18 smithi170 ceph-mon[35690]: pgmap v304: 97 pgs: 97 active+clean; 641 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-11T19:08:21.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:20 smithi029 ceph-mon[32439]: pgmap v305: 97 pgs: 97 active+clean; 641 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-11T19:08:21.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:20 smithi190 ceph-mon[36292]: pgmap v305: 97 pgs: 97 active+clean; 641 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-11T19:08:21.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:20 smithi170 ceph-mon[35690]: pgmap v305: 97 pgs: 97 active+clean; 641 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-11T19:08:23.108 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:22 smithi029 ceph-mon[32439]: pgmap v306: 97 pgs: 97 active+clean; 641 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-11T19:08:23.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.105+0000 7fe48e467700 1 -- 172.21.15.29:0/2778927130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe488100590 msgr2=0x7fe488100970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:23.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.105+0000 7fe48e467700 1 --2- 172.21.15.29:0/2778927130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe488100590 0x7fe488100970 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009990 tx=0x7fe47c009ca0 comp rx=0 tx=0).stop 2024-03-11T19:08:23.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.105+0000 7fe48e467700 1 -- 172.21.15.29:0/2778927130 shutdown_connections 2024-03-11T19:08:23.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.105+0000 7fe48e467700 1 --2- 172.21.15.29:0/2778927130 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe488105670 0x7fe48810d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:23.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.105+0000 7fe48e467700 1 --2- 172.21.15.29:0/2778927130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe488100590 0x7fe488100970 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:23.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.105+0000 7fe48e467700 1 --2- 172.21.15.29:0/2778927130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe488100f40 0x7fe488104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:23.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.105+0000 7fe48e467700 1 -- 172.21.15.29:0/2778927130 >> 172.21.15.29:0/2778927130 conn(0x7fe488077170 msgr2=0x7fe4880fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:23.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.105+0000 7fe48e467700 1 -- 172.21.15.29:0/2778927130 shutdown_connections 2024-03-11T19:08:23.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.105+0000 7fe48e467700 1 -- 172.21.15.29:0/2778927130 wait complete. 2024-03-11T19:08:23.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.105+0000 7fe48e467700 1 Processor -- start 2024-03-11T19:08:23.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.105+0000 7fe48e467700 1 -- start start 2024-03-11T19:08:23.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.106+0000 7fe48e467700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe488100590 0x7fe4881981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:23.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.106+0000 7fe48e467700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe488100f40 0x7fe488198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:23.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.106+0000 7fe48e467700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe488105670 0x7fe48819caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:23.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.106+0000 7fe48e467700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48819d0b0 con 0x7fe488100f40 2024-03-11T19:08:23.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.106+0000 7fe48e467700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48819d220 con 0x7fe488100590 2024-03-11T19:08:23.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.106+0000 7fe48e467700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48819d3e0 con 0x7fe488105670 2024-03-11T19:08:23.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.106+0000 7fe4877fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe488100f40 0x7fe488198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:23.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.106+0000 7fe4877fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe488100f40 0x7fe488198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50778/0 (socket says 172.21.15.29:50778) 2024-03-11T19:08:23.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.106+0000 7fe487fff700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe488100590 0x7fe4881981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:23.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.106+0000 7fe4877fe700 1 -- 172.21.15.29:0/2476806763 learned_addr learned my addr 172.21.15.29:0/2476806763 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:23.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.106+0000 7fe48ca04700 1 --2- 172.21.15.29:0/2476806763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe488105670 0x7fe48819caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:23.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.106+0000 7fe4877fe700 1 -- 172.21.15.29:0/2476806763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe488105670 msgr2=0x7fe48819caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:23.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.106+0000 7fe4877fe700 1 --2- 172.21.15.29:0/2476806763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe488105670 0x7fe48819caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:23.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.106+0000 7fe4877fe700 1 -- 172.21.15.29:0/2476806763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe488100590 msgr2=0x7fe4881981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:23.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.106+0000 7fe4877fe700 1 --2- 172.21.15.29:0/2476806763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe488100590 0x7fe4881981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:23.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.106+0000 7fe4877fe700 1 -- 172.21.15.29:0/2476806763 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47c0095f0 con 0x7fe488100f40 2024-03-11T19:08:23.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.107+0000 7fe4877fe700 1 --2- 172.21.15.29:0/2476806763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe488100f40 0x7fe488198710 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe47800b500 tx=0x7fe47800b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:23.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.107+0000 7fe4857fa700 1 -- 172.21.15.29:0/2476806763 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe478011830 con 0x7fe488100f40 2024-03-11T19:08:23.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.107+0000 7fe4857fa700 1 -- 172.21.15.29:0/2476806763 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe478011e40 con 0x7fe488100f40 2024-03-11T19:08:23.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.107+0000 7fe48e467700 1 -- 172.21.15.29:0/2476806763 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48819d6c0 con 0x7fe488100f40 2024-03-11T19:08:23.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.107+0000 7fe4857fa700 1 -- 172.21.15.29:0/2476806763 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe478010dd0 con 0x7fe488100f40 2024-03-11T19:08:23.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.107+0000 7fe48e467700 1 -- 172.21.15.29:0/2476806763 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4881a4f20 con 0x7fe488100f40 2024-03-11T19:08:23.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.108+0000 7fe48e467700 1 -- 172.21.15.29:0/2476806763 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe488110e40 con 0x7fe488100f40 2024-03-11T19:08:23.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.109+0000 7fe4857fa700 1 -- 172.21.15.29:0/2476806763 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fe478024430 con 0x7fe488100f40 2024-03-11T19:08:23.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.109+0000 7fe4857fa700 1 --2- 172.21.15.29:0/2476806763 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe4700a1b50 0x7fe4700a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:23.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.110+0000 7fe487fff700 1 --2- 172.21.15.29:0/2476806763 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe4700a1b50 0x7fe4700a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:23.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.110+0000 7fe4857fa700 1 -- 172.21.15.29:0/2476806763 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe4780cd320 con 0x7fe488100f40 2024-03-11T19:08:23.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.110+0000 7fe4857fa700 1 --2- 172.21.15.29:0/2476806763 >> [v2:172.21.15.170:6824/3766541425,v1:172.21.15.170:6825/3766541425] conn(0x7fe4700ab330 0x7fe4700ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:23.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.110+0000 7fe4857fa700 1 -- 172.21.15.29:0/2476806763 --> [v2:172.21.15.170:6824/3766541425,v1:172.21.15.170:6825/3766541425] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe4700ade00 con 0x7fe4700ab330 2024-03-11T19:08:23.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.110+0000 7fe4857fa700 1 -- 172.21.15.29:0/2476806763 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4780cd7c0 con 0x7fe488100f40 2024-03-11T19:08:23.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.110+0000 7fe487fff700 1 --2- 172.21.15.29:0/2476806763 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe4700a1b50 0x7fe4700a4010 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe47c00b5c0 tx=0x7fe47c005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:23.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.110+0000 7fe48ca04700 1 --2- 172.21.15.29:0/2476806763 >> [v2:172.21.15.170:6824/3766541425,v1:172.21.15.170:6825/3766541425] conn(0x7fe4700ab330 0x7fe4700ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:23.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.111+0000 7fe48ca04700 1 --2- 172.21.15.29:0/2476806763 >> [v2:172.21.15.170:6824/3766541425,v1:172.21.15.170:6825/3766541425] conn(0x7fe4700ab330 0x7fe4700ad750 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:23.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.112+0000 7fe4857fa700 1 -- 172.21.15.29:0/2476806763 <== osd.10 v2:172.21.15.170:6824/3766541425 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe4700ade00 con 0x7fe4700ab330 2024-03-11T19:08:23.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.129+0000 7fe48e467700 1 -- 172.21.15.29:0/2476806763 --> [v2:172.21.15.170:6824/3766541425,v1:172.21.15.170:6825/3766541425] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe488062380 con 0x7fe4700ab330 2024-03-11T19:08:23.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.129+0000 7fe4857fa700 1 -- 172.21.15.29:0/2476806763 <== osd.10 v2:172.21.15.170:6824/3766541425 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe488062380 con 0x7fe4700ab330 2024-03-11T19:08:23.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.130+0000 7fe48e467700 1 -- 172.21.15.29:0/2476806763 >> [v2:172.21.15.170:6824/3766541425,v1:172.21.15.170:6825/3766541425] conn(0x7fe4700ab330 msgr2=0x7fe4700ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:23.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.130+0000 7fe48e467700 1 --2- 172.21.15.29:0/2476806763 >> [v2:172.21.15.170:6824/3766541425,v1:172.21.15.170:6825/3766541425] conn(0x7fe4700ab330 0x7fe4700ad750 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:23.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.130+0000 7fe48e467700 1 -- 172.21.15.29:0/2476806763 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe4700a1b50 msgr2=0x7fe4700a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:23.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.130+0000 7fe48e467700 1 --2- 172.21.15.29:0/2476806763 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe4700a1b50 0x7fe4700a4010 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe47c00b5c0 tx=0x7fe47c005960 comp rx=0 tx=0).stop 2024-03-11T19:08:23.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.130+0000 7fe48e467700 1 -- 172.21.15.29:0/2476806763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe488100f40 msgr2=0x7fe488198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:23.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.130+0000 7fe48e467700 1 --2- 172.21.15.29:0/2476806763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe488100f40 0x7fe488198710 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe47800b500 tx=0x7fe47800b810 comp rx=0 tx=0).stop 2024-03-11T19:08:23.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.130+0000 7fe48ca04700 1 -- 172.21.15.29:0/2476806763 reap_dead start 2024-03-11T19:08:23.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.130+0000 7fe48e467700 1 -- 172.21.15.29:0/2476806763 shutdown_connections 2024-03-11T19:08:23.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.130+0000 7fe48e467700 1 -- 172.21.15.29:0/2476806763 >> 172.21.15.29:0/2476806763 conn(0x7fe488077170 msgr2=0x7fe4880feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:23.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.130+0000 7fe48e467700 1 -- 172.21.15.29:0/2476806763 shutdown_connections 2024-03-11T19:08:23.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.130+0000 7fe48e467700 1 -- 172.21.15.29:0/2476806763 wait complete. 2024-03-11T19:08:23.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:22 smithi190 ceph-mon[36292]: pgmap v306: 97 pgs: 97 active+clean; 641 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-11T19:08:23.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:22 smithi170 ceph-mon[35690]: pgmap v306: 97 pgs: 97 active+clean; 641 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-11T19:08:24.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.121+0000 7fbfe6509700 1 -- 172.21.15.29:0/3002572710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfe0102680 msgr2=0x7fbfe010eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:24.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.121+0000 7fbfe6509700 1 --2- 172.21.15.29:0/3002572710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfe0102680 0x7fbfe010eac0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbfd800b0d0 tx=0x7fbfd800b3e0 comp rx=0 tx=0).stop 2024-03-11T19:08:24.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.122+0000 7fbfe6509700 1 -- 172.21.15.29:0/3002572710 shutdown_connections 2024-03-11T19:08:24.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.122+0000 7fbfe6509700 1 --2- 172.21.15.29:0/3002572710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfe0102680 0x7fbfe010eac0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:24.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.122+0000 7fbfe6509700 1 --2- 172.21.15.29:0/3002572710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfe0107ca0 0x7fbfe0108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:24.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.122+0000 7fbfe6509700 1 --2- 172.21.15.29:0/3002572710 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfe0101cc0 0x7fbfe0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:24.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.122+0000 7fbfe6509700 1 -- 172.21.15.29:0/3002572710 >> 172.21.15.29:0/3002572710 conn(0x7fbfe0078da0 msgr2=0x7fbfe00791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:24.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.122+0000 7fbfe6509700 1 -- 172.21.15.29:0/3002572710 shutdown_connections 2024-03-11T19:08:24.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.122+0000 7fbfe6509700 1 -- 172.21.15.29:0/3002572710 wait complete. 2024-03-11T19:08:24.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.122+0000 7fbfe6509700 1 Processor -- start 2024-03-11T19:08:24.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.122+0000 7fbfe6509700 1 -- start start 2024-03-11T19:08:24.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.123+0000 7fbfe6509700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfe0101cc0 0x7fbfe01981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:24.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.123+0000 7fbfe6509700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfe0102680 0x7fbfe01986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:24.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.123+0000 7fbfe6509700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfe0107ca0 0x7fbfe019ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:24.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.123+0000 7fbfe6509700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe019d090 con 0x7fbfe0107ca0 2024-03-11T19:08:24.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.123+0000 7fbfe6509700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe019d200 con 0x7fbfe0101cc0 2024-03-11T19:08:24.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.123+0000 7fbfe6509700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe019d3c0 con 0x7fbfe0102680 2024-03-11T19:08:24.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.123+0000 7fbfe4aa6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfe0107ca0 0x7fbfe019ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:24.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.123+0000 7fbfe4aa6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfe0107ca0 0x7fbfe019ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50788/0 (socket says 172.21.15.29:50788) 2024-03-11T19:08:24.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.123+0000 7fbfe4aa6700 1 -- 172.21.15.29:0/2952982335 learned_addr learned my addr 172.21.15.29:0/2952982335 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:24.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.123+0000 7fbfdf7fe700 1 --2- 172.21.15.29:0/2952982335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfe0102680 0x7fbfe01986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:24.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.123+0000 7fbfdffff700 1 --2- 172.21.15.29:0/2952982335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfe0101cc0 0x7fbfe01981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:24.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.123+0000 7fbfe4aa6700 1 -- 172.21.15.29:0/2952982335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfe0102680 msgr2=0x7fbfe01986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:24.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.123+0000 7fbfe4aa6700 1 --2- 172.21.15.29:0/2952982335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfe0102680 0x7fbfe01986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:24.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.123+0000 7fbfe4aa6700 1 -- 172.21.15.29:0/2952982335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfe0101cc0 msgr2=0x7fbfe01981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:24.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.123+0000 7fbfe4aa6700 1 --2- 172.21.15.29:0/2952982335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfe0101cc0 0x7fbfe01981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:24.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.123+0000 7fbfe4aa6700 1 -- 172.21.15.29:0/2952982335 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd8009d00 con 0x7fbfe0107ca0 2024-03-11T19:08:24.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.123+0000 7fbfdffff700 1 --2- 172.21.15.29:0/2952982335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfe0101cc0 0x7fbfe01981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:08:24.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.124+0000 7fbfe4aa6700 1 --2- 172.21.15.29:0/2952982335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfe0107ca0 0x7fbfe019ca80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbfd80079b0 tx=0x7fbfd80079e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:24.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.124+0000 7fbfdd7fa700 1 -- 172.21.15.29:0/2952982335 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd800e040 con 0x7fbfe0107ca0 2024-03-11T19:08:24.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.124+0000 7fbfdd7fa700 1 -- 172.21.15.29:0/2952982335 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbfd801cb70 con 0x7fbfe0107ca0 2024-03-11T19:08:24.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.124+0000 7fbfe6509700 1 -- 172.21.15.29:0/2952982335 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe019d640 con 0x7fbfe0107ca0 2024-03-11T19:08:24.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.124+0000 7fbfdd7fa700 1 -- 172.21.15.29:0/2952982335 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd8012c10 con 0x7fbfe0107ca0 2024-03-11T19:08:24.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.124+0000 7fbfe6509700 1 -- 172.21.15.29:0/2952982335 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe0074070 con 0x7fbfe0107ca0 2024-03-11T19:08:24.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.125+0000 7fbfe6509700 1 -- 172.21.15.29:0/2952982335 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbfe0103350 con 0x7fbfe0107ca0 2024-03-11T19:08:24.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.126+0000 7fbfdd7fa700 1 -- 172.21.15.29:0/2952982335 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fbfd801cce0 con 0x7fbfe0107ca0 2024-03-11T19:08:24.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.126+0000 7fbfdd7fa700 1 --2- 172.21.15.29:0/2952982335 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbfc80a1b50 0x7fbfc80a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:24.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.127+0000 7fbfdffff700 1 --2- 172.21.15.29:0/2952982335 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbfc80a1b50 0x7fbfc80a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:24.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.127+0000 7fbfdd7fa700 1 -- 172.21.15.29:0/2952982335 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fbfd80d2900 con 0x7fbfe0107ca0 2024-03-11T19:08:24.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.127+0000 7fbfdd7fa700 1 --2- 172.21.15.29:0/2952982335 >> [v2:172.21.15.190:6800/1010896180,v1:172.21.15.190:6801/1010896180] conn(0x7fbfc80ab330 0x7fbfc80ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:24.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.127+0000 7fbfdd7fa700 1 -- 172.21.15.29:0/2952982335 --> [v2:172.21.15.190:6800/1010896180,v1:172.21.15.190:6801/1010896180] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbfc80ade00 con 0x7fbfc80ab330 2024-03-11T19:08:24.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.127+0000 7fbfdd7fa700 1 -- 172.21.15.29:0/2952982335 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbfd80d2da0 con 0x7fbfe0107ca0 2024-03-11T19:08:24.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.127+0000 7fbfdffff700 1 --2- 172.21.15.29:0/2952982335 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbfc80a1b50 0x7fbfc80a4010 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000bb90 tx=0x7fbfd0005d80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:24.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.127+0000 7fbfdf7fe700 1 --2- 172.21.15.29:0/2952982335 >> [v2:172.21.15.190:6800/1010896180,v1:172.21.15.190:6801/1010896180] conn(0x7fbfc80ab330 0x7fbfc80ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:24.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.128+0000 7fbfdf7fe700 1 --2- 172.21.15.29:0/2952982335 >> [v2:172.21.15.190:6800/1010896180,v1:172.21.15.190:6801/1010896180] conn(0x7fbfc80ab330 0x7fbfc80ad750 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:24.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.129+0000 7fbfdd7fa700 1 -- 172.21.15.29:0/2952982335 <== osd.2 v2:172.21.15.190:6800/1010896180 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbfc80ade00 con 0x7fbfc80ab330 2024-03-11T19:08:24.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.145+0000 7fbfe6509700 1 -- 172.21.15.29:0/2952982335 --> [v2:172.21.15.190:6800/1010896180,v1:172.21.15.190:6801/1010896180] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbfe0111e70 con 0x7fbfc80ab330 2024-03-11T19:08:24.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.146+0000 7fbfdd7fa700 1 -- 172.21.15.29:0/2952982335 <== osd.2 v2:172.21.15.190:6800/1010896180 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbfe0111e70 con 0x7fbfc80ab330 2024-03-11T19:08:24.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.146+0000 7fbfe6509700 1 -- 172.21.15.29:0/2952982335 >> [v2:172.21.15.190:6800/1010896180,v1:172.21.15.190:6801/1010896180] conn(0x7fbfc80ab330 msgr2=0x7fbfc80ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:24.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.146+0000 7fbfe6509700 1 --2- 172.21.15.29:0/2952982335 >> [v2:172.21.15.190:6800/1010896180,v1:172.21.15.190:6801/1010896180] conn(0x7fbfc80ab330 0x7fbfc80ad750 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:24.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.146+0000 7fbfe6509700 1 -- 172.21.15.29:0/2952982335 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbfc80a1b50 msgr2=0x7fbfc80a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:24.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.146+0000 7fbfe6509700 1 --2- 172.21.15.29:0/2952982335 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbfc80a1b50 0x7fbfc80a4010 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000bb90 tx=0x7fbfd0005d80 comp rx=0 tx=0).stop 2024-03-11T19:08:24.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.146+0000 7fbfe6509700 1 -- 172.21.15.29:0/2952982335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfe0107ca0 msgr2=0x7fbfe019ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:24.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.146+0000 7fbfe6509700 1 --2- 172.21.15.29:0/2952982335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfe0107ca0 0x7fbfe019ca80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbfd80079b0 tx=0x7fbfd80079e0 comp rx=0 tx=0).stop 2024-03-11T19:08:24.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.146+0000 7fbfe4aa6700 1 -- 172.21.15.29:0/2952982335 reap_dead start 2024-03-11T19:08:24.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.146+0000 7fbfe6509700 1 -- 172.21.15.29:0/2952982335 shutdown_connections 2024-03-11T19:08:24.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.146+0000 7fbfe6509700 1 -- 172.21.15.29:0/2952982335 >> 172.21.15.29:0/2952982335 conn(0x7fbfe0078da0 msgr2=0x7fbfe0100a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:24.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.146+0000 7fbfe6509700 1 -- 172.21.15.29:0/2952982335 shutdown_connections 2024-03-11T19:08:24.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:24.146+0000 7fbfe6509700 1 -- 172.21.15.29:0/2952982335 wait complete. 2024-03-11T19:08:24.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.001+0000 7fa68053c700 1 -- 172.21.15.29:0/55120717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa678101530 msgr2=0x7fa67810d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:24.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.001+0000 7fa68053c700 1 --2- 172.21.15.29:0/55120717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa678101530 0x7fa67810d9c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa67000b0a0 tx=0x7fa67000b3b0 comp rx=0 tx=0).stop 2024-03-11T19:08:24.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.002+0000 7fa68053c700 1 -- 172.21.15.29:0/55120717 shutdown_connections 2024-03-11T19:08:24.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.002+0000 7fa68053c700 1 --2- 172.21.15.29:0/55120717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa678101530 0x7fa67810d9c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:24.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.002+0000 7fa68053c700 1 --2- 172.21.15.29:0/55120717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa678106b80 0x7fa678106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:24.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.002+0000 7fa68053c700 1 --2- 172.21.15.29:0/55120717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa678100b70 0x7fa678100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:24.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.002+0000 7fa68053c700 1 -- 172.21.15.29:0/55120717 >> 172.21.15.29:0/55120717 conn(0x7fa678077aa0 msgr2=0x7fa678077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:24.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.002+0000 7fa68053c700 1 -- 172.21.15.29:0/55120717 shutdown_connections 2024-03-11T19:08:24.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.002+0000 7fa68053c700 1 -- 172.21.15.29:0/55120717 wait complete. 2024-03-11T19:08:24.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.002+0000 7fa68053c700 1 Processor -- start 2024-03-11T19:08:24.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.002+0000 7fa68053c700 1 -- start start 2024-03-11T19:08:24.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.002+0000 7fa68053c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa678100b70 0x7fa678102ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:24.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa68053c700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa678101530 0x7fa678102ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:24.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa68053c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa678106b80 0x7fa678103530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:24.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa68053c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6781a6520 con 0x7fa678100b70 2024-03-11T19:08:24.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa68053c700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6781a6690 con 0x7fa678101530 2024-03-11T19:08:24.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa68053c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6781a6850 con 0x7fa678106b80 2024-03-11T19:08:24.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa67e2d8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa678100b70 0x7fa678102ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:24.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa67e2d8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa678100b70 0x7fa678102ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50758/0 (socket says 172.21.15.29:50758) 2024-03-11T19:08:24.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa67e2d8700 1 -- 172.21.15.29:0/2899918193 learned_addr learned my addr 172.21.15.29:0/2899918193 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:24.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa67ead9700 1 --2- 172.21.15.29:0/2899918193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa678106b80 0x7fa678103530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:24.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa67dad7700 1 --2- 172.21.15.29:0/2899918193 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa678101530 0x7fa678102ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:24.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa67e2d8700 1 -- 172.21.15.29:0/2899918193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa678106b80 msgr2=0x7fa678103530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:24.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa67e2d8700 1 --2- 172.21.15.29:0/2899918193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa678106b80 0x7fa678103530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:24.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa67e2d8700 1 -- 172.21.15.29:0/2899918193 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa678101530 msgr2=0x7fa678102ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:24.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa67e2d8700 1 --2- 172.21.15.29:0/2899918193 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa678101530 0x7fa678102ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:24.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa67e2d8700 1 -- 172.21.15.29:0/2899918193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa670009d00 con 0x7fa678100b70 2024-03-11T19:08:24.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa67dad7700 1 --2- 172.21.15.29:0/2899918193 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa678101530 0x7fa678102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:08:24.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa67e2d8700 1 --2- 172.21.15.29:0/2899918193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa678100b70 0x7fa678102ab0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa67400d950 tx=0x7fa67400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:24.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa66f7fe700 1 -- 172.21.15.29:0/2899918193 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa674009940 con 0x7fa678100b70 2024-03-11T19:08:24.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa68053c700 1 -- 172.21.15.29:0/2899918193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6781a6b30 con 0x7fa678100b70 2024-03-11T19:08:24.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa66f7fe700 1 -- 172.21.15.29:0/2899918193 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa674010460 con 0x7fa678100b70 2024-03-11T19:08:24.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa66f7fe700 1 -- 172.21.15.29:0/2899918193 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67400f620 con 0x7fa678100b70 2024-03-11T19:08:24.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.003+0000 7fa68053c700 1 -- 172.21.15.29:0/2899918193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6781a7030 con 0x7fa678100b70 2024-03-11T19:08:24.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.004+0000 7fa68053c700 1 -- 172.21.15.29:0/2899918193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa678110d40 con 0x7fa678100b70 2024-03-11T19:08:24.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.005+0000 7fa66f7fe700 1 -- 172.21.15.29:0/2899918193 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fa6740105d0 con 0x7fa678100b70 2024-03-11T19:08:24.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.006+0000 7fa66f7fe700 1 --2- 172.21.15.29:0/2899918193 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa6640a1b50 0x7fa6640a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:24.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.006+0000 7fa67dad7700 1 --2- 172.21.15.29:0/2899918193 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa6640a1b50 0x7fa6640a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:24.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.006+0000 7fa66f7fe700 1 -- 172.21.15.29:0/2899918193 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa6740cd810 con 0x7fa678100b70 2024-03-11T19:08:24.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.006+0000 7fa66f7fe700 1 --2- 172.21.15.29:0/2899918193 >> [v2:172.21.15.190:6808/318560672,v1:172.21.15.190:6809/318560672] conn(0x7fa6640ab330 0x7fa6640ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:24.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.006+0000 7fa66f7fe700 1 -- 172.21.15.29:0/2899918193 --> [v2:172.21.15.190:6808/318560672,v1:172.21.15.190:6809/318560672] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6640ade00 con 0x7fa6640ab330 2024-03-11T19:08:24.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.006+0000 7fa66f7fe700 1 -- 172.21.15.29:0/2899918193 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6740cdb80 con 0x7fa678100b70 2024-03-11T19:08:24.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.006+0000 7fa67dad7700 1 --2- 172.21.15.29:0/2899918193 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa6640a1b50 0x7fa6640a4010 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa668005fd0 tx=0x7fa668005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:24.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.007+0000 7fa67ead9700 1 --2- 172.21.15.29:0/2899918193 >> [v2:172.21.15.190:6808/318560672,v1:172.21.15.190:6809/318560672] conn(0x7fa6640ab330 0x7fa6640ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:24.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.007+0000 7fa67ead9700 1 --2- 172.21.15.29:0/2899918193 >> [v2:172.21.15.190:6808/318560672,v1:172.21.15.190:6809/318560672] conn(0x7fa6640ab330 0x7fa6640ad750 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:24.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.008+0000 7fa66f7fe700 1 -- 172.21.15.29:0/2899918193 <== osd.5 v2:172.21.15.190:6808/318560672 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa6640ade00 con 0x7fa6640ab330 2024-03-11T19:08:24.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.024+0000 7fa68053c700 1 -- 172.21.15.29:0/2899918193 --> [v2:172.21.15.190:6808/318560672,v1:172.21.15.190:6809/318560672] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa678069bf0 con 0x7fa6640ab330 2024-03-11T19:08:24.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.025+0000 7fa66f7fe700 1 -- 172.21.15.29:0/2899918193 <== osd.5 v2:172.21.15.190:6808/318560672 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa678069bf0 con 0x7fa6640ab330 2024-03-11T19:08:24.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.025+0000 7fa68053c700 1 -- 172.21.15.29:0/2899918193 >> [v2:172.21.15.190:6808/318560672,v1:172.21.15.190:6809/318560672] conn(0x7fa6640ab330 msgr2=0x7fa6640ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:24.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.025+0000 7fa68053c700 1 --2- 172.21.15.29:0/2899918193 >> [v2:172.21.15.190:6808/318560672,v1:172.21.15.190:6809/318560672] conn(0x7fa6640ab330 0x7fa6640ad750 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:24.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.025+0000 7fa68053c700 1 -- 172.21.15.29:0/2899918193 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa6640a1b50 msgr2=0x7fa6640a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:24.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.025+0000 7fa68053c700 1 --2- 172.21.15.29:0/2899918193 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa6640a1b50 0x7fa6640a4010 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa668005fd0 tx=0x7fa668005ee0 comp rx=0 tx=0).stop 2024-03-11T19:08:24.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.025+0000 7fa68053c700 1 -- 172.21.15.29:0/2899918193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa678100b70 msgr2=0x7fa678102ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:24.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.025+0000 7fa68053c700 1 --2- 172.21.15.29:0/2899918193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa678100b70 0x7fa678102ab0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa67400d950 tx=0x7fa67400dc60 comp rx=0 tx=0).stop 2024-03-11T19:08:24.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.025+0000 7fa67ead9700 1 -- 172.21.15.29:0/2899918193 reap_dead start 2024-03-11T19:08:24.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.025+0000 7fa68053c700 1 -- 172.21.15.29:0/2899918193 shutdown_connections 2024-03-11T19:08:24.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.025+0000 7fa68053c700 1 -- 172.21.15.29:0/2899918193 >> 172.21.15.29:0/2899918193 conn(0x7fa678077aa0 msgr2=0x7fa678104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:24.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.026+0000 7fa68053c700 1 -- 172.21.15.29:0/2899918193 shutdown_connections 2024-03-11T19:08:24.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:23.026+0000 7fa68053c700 1 -- 172.21.15.29:0/2899918193 wait complete. 2024-03-11T19:08:25.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:24 smithi029 ceph-mon[32439]: pgmap v307: 97 pgs: 97 active+clean; 641 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-11T19:08:25.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:24 smithi190 ceph-mon[36292]: pgmap v307: 97 pgs: 97 active+clean; 641 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-11T19:08:25.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:24 smithi170 ceph-mon[35690]: pgmap v307: 97 pgs: 97 active+clean; 641 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-11T19:08:26.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.182+0000 7f212a83e700 1 -- 172.21.15.29:0/2278586783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f212406aad0 msgr2=0x7f212410e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:26.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.182+0000 7f212a83e700 1 --2- 172.21.15.29:0/2278586783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f212406aad0 0x7f212410e210 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f21180098f0 tx=0x7f2118009c00 comp rx=0 tx=0).stop 2024-03-11T19:08:26.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.183+0000 7f212a83e700 1 -- 172.21.15.29:0/2278586783 shutdown_connections 2024-03-11T19:08:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.183+0000 7f212a83e700 1 --2- 172.21.15.29:0/2278586783 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f212410e750 0x7f2124110b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.183+0000 7f212a83e700 1 --2- 172.21.15.29:0/2278586783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f212406a1b0 0x7f212406a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.183+0000 7f212a83e700 1 --2- 172.21.15.29:0/2278586783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f212406aad0 0x7f212410e210 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.183+0000 7f212a83e700 1 -- 172.21.15.29:0/2278586783 >> 172.21.15.29:0/2278586783 conn(0x7f2124076bc0 msgr2=0x7f2124076fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.183+0000 7f212a83e700 1 -- 172.21.15.29:0/2278586783 shutdown_connections 2024-03-11T19:08:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.183+0000 7f212a83e700 1 -- 172.21.15.29:0/2278586783 wait complete. 2024-03-11T19:08:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.184+0000 7f212a83e700 1 Processor -- start 2024-03-11T19:08:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.184+0000 7f212a83e700 1 -- start start 2024-03-11T19:08:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.184+0000 7f212a83e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f212406a1b0 0x7f212419a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.184+0000 7f212a83e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f212406aad0 0x7f212419a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.184+0000 7f212a83e700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f212410e750 0x7f212419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.184+0000 7f212a83e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212419f1f0 con 0x7f212406aad0 2024-03-11T19:08:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.184+0000 7f212a83e700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212419f360 con 0x7f212410e750 2024-03-11T19:08:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.184+0000 7f212a83e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212419f520 con 0x7f212406a1b0 2024-03-11T19:08:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.184+0000 7f21237fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f212406aad0 0x7f212419a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.184+0000 7f21237fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f212406aad0 0x7f212419a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36738/0 (socket says 172.21.15.29:36738) 2024-03-11T19:08:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.184+0000 7f21237fe700 1 -- 172.21.15.29:0/1684544866 learned_addr learned my addr 172.21.15.29:0/1684544866 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.184+0000 7f2123fff700 1 --2- 172.21.15.29:0/1684544866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f212406a1b0 0x7f212419a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.184+0000 7f2128ddb700 1 --2- 172.21.15.29:0/1684544866 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f212410e750 0x7f212419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.185+0000 7f2123fff700 1 -- 172.21.15.29:0/1684544866 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f212410e750 msgr2=0x7f212419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.185+0000 7f2123fff700 1 --2- 172.21.15.29:0/1684544866 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f212410e750 0x7f212419ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.185+0000 7f2123fff700 1 -- 172.21.15.29:0/1684544866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f212406aad0 msgr2=0x7f212419a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.185+0000 7f2123fff700 1 --2- 172.21.15.29:0/1684544866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f212406aad0 0x7f212419a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.185+0000 7f2123fff700 1 -- 172.21.15.29:0/1684544866 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2118009520 con 0x7f212406a1b0 2024-03-11T19:08:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.185+0000 7f21237fe700 1 --2- 172.21.15.29:0/1684544866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f212406aad0 0x7f212419a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:08:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.185+0000 7f2128ddb700 1 --2- 172.21.15.29:0/1684544866 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f212410e750 0x7f212419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:08:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.185+0000 7f2123fff700 1 --2- 172.21.15.29:0/1684544866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f212406a1b0 0x7f212419a310 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f211400d950 tx=0x7f211400dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.185+0000 7f21217fa700 1 -- 172.21.15.29:0/1684544866 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2114009930 con 0x7f212406a1b0 2024-03-11T19:08:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.185+0000 7f21217fa700 1 -- 172.21.15.29:0/1684544866 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2114010460 con 0x7f212406a1b0 2024-03-11T19:08:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.185+0000 7f21217fa700 1 -- 172.21.15.29:0/1684544866 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211400b5d0 con 0x7f212406a1b0 2024-03-11T19:08:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.185+0000 7f212a83e700 1 -- 172.21.15.29:0/1684544866 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f212419f800 con 0x7f212406a1b0 2024-03-11T19:08:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.186+0000 7f212a83e700 1 -- 172.21.15.29:0/1684544866 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21241a7060 con 0x7f212406a1b0 2024-03-11T19:08:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.187+0000 7f212a83e700 1 -- 172.21.15.29:0/1684544866 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2124114090 con 0x7f212406a1b0 2024-03-11T19:08:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.187+0000 7f21217fa700 1 -- 172.21.15.29:0/1684544866 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f21140105d0 con 0x7f212406a1b0 2024-03-11T19:08:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.188+0000 7f21217fa700 1 --2- 172.21.15.29:0/1684544866 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f210c0a1bd0 0x7f210c0a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.188+0000 7f21237fe700 1 --2- 172.21.15.29:0/1684544866 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f210c0a1bd0 0x7f210c0a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.189+0000 7f21217fa700 1 -- 172.21.15.29:0/1684544866 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f21140ccfe0 con 0x7f212406a1b0 2024-03-11T19:08:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.189+0000 7f21217fa700 1 --2- 172.21.15.29:0/1684544866 >> [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152] conn(0x7f210c0ab3b0 0x7f210c0ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.189+0000 7f21217fa700 1 -- 172.21.15.29:0/1684544866 --> [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f210c0ade80 con 0x7f210c0ab3b0 2024-03-11T19:08:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.189+0000 7f21217fa700 1 -- 172.21.15.29:0/1684544866 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21140cf5a0 con 0x7f212406a1b0 2024-03-11T19:08:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.189+0000 7f2128ddb700 1 --2- 172.21.15.29:0/1684544866 >> [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152] conn(0x7f210c0ab3b0 0x7f210c0ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.189+0000 7f21237fe700 1 --2- 172.21.15.29:0/1684544866 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f210c0a1bd0 0x7f210c0a4090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2124102160 tx=0x7f2118005900 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.189+0000 7f2128ddb700 1 --2- 172.21.15.29:0/1684544866 >> [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152] conn(0x7f210c0ab3b0 0x7f210c0ad7d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.190+0000 7f21217fa700 1 -- 172.21.15.29:0/1684544866 <== osd.0 v2:172.21.15.29:6802/827892152 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f210c0ade80 con 0x7f210c0ab3b0 2024-03-11T19:08:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.209+0000 7f212a83e700 1 -- 172.21.15.29:0/1684544866 --> [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2124062380 con 0x7f210c0ab3b0 2024-03-11T19:08:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.209+0000 7f21217fa700 1 -- 172.21.15.29:0/1684544866 <== osd.0 v2:172.21.15.29:6802/827892152 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2124062380 con 0x7f210c0ab3b0 2024-03-11T19:08:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.209+0000 7f212a83e700 1 -- 172.21.15.29:0/1684544866 >> [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152] conn(0x7f210c0ab3b0 msgr2=0x7f210c0ad7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:26.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.209+0000 7f212a83e700 1 --2- 172.21.15.29:0/1684544866 >> [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152] conn(0x7f210c0ab3b0 0x7f210c0ad7d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:26.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.209+0000 7f212a83e700 1 -- 172.21.15.29:0/1684544866 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f210c0a1bd0 msgr2=0x7f210c0a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:26.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.209+0000 7f212a83e700 1 --2- 172.21.15.29:0/1684544866 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f210c0a1bd0 0x7f210c0a4090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2124102160 tx=0x7f2118005900 comp rx=0 tx=0).stop 2024-03-11T19:08:26.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.209+0000 7f212a83e700 1 -- 172.21.15.29:0/1684544866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f212406a1b0 msgr2=0x7f212419a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:26.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.209+0000 7f212a83e700 1 --2- 172.21.15.29:0/1684544866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f212406a1b0 0x7f212419a310 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f211400d950 tx=0x7f211400dc60 comp rx=0 tx=0).stop 2024-03-11T19:08:26.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.210+0000 7f2128ddb700 1 -- 172.21.15.29:0/1684544866 reap_dead start 2024-03-11T19:08:26.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.210+0000 7f212a83e700 1 -- 172.21.15.29:0/1684544866 shutdown_connections 2024-03-11T19:08:26.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.210+0000 7f212a83e700 1 -- 172.21.15.29:0/1684544866 >> 172.21.15.29:0/1684544866 conn(0x7f2124076bc0 msgr2=0x7f212410fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:26.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.210+0000 7f212a83e700 1 -- 172.21.15.29:0/1684544866 shutdown_connections 2024-03-11T19:08:26.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:26.210+0000 7f212a83e700 1 -- 172.21.15.29:0/1684544866 wait complete. 2024-03-11T19:08:26.655 INFO:teuthology.orchestra.run.smithi029.stdout:171798691891 2024-03-11T19:08:26.656 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-11T19:08:27.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:26 smithi190 ceph-mon[36292]: pgmap v308: 97 pgs: 97 active+clean; 641 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-11T19:08:27.292 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:26 smithi029 ceph-mon[32439]: pgmap v308: 97 pgs: 97 active+clean; 641 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-11T19:08:27.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:26 smithi170 ceph-mon[35690]: pgmap v308: 97 pgs: 97 active+clean; 641 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-11T19:08:28.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.674+0000 7f327df5a700 1 -- 172.21.15.29:0/3785628262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3278101530 msgr2=0x7f327810d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:28.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.674+0000 7f327df5a700 1 --2- 172.21.15.29:0/3785628262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3278101530 0x7f327810d9c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f327000b0a0 tx=0x7f327000b3b0 comp rx=0 tx=0).stop 2024-03-11T19:08:28.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.674+0000 7f327df5a700 1 -- 172.21.15.29:0/3785628262 shutdown_connections 2024-03-11T19:08:28.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.674+0000 7f327df5a700 1 --2- 172.21.15.29:0/3785628262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3278101530 0x7f327810d9c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:28.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.674+0000 7f327df5a700 1 --2- 172.21.15.29:0/3785628262 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3278106b80 0x7f3278106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:28.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.674+0000 7f327df5a700 1 --2- 172.21.15.29:0/3785628262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3278100b70 0x7f3278100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:28.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.674+0000 7f327df5a700 1 -- 172.21.15.29:0/3785628262 >> 172.21.15.29:0/3785628262 conn(0x7f3278077aa0 msgr2=0x7f3278077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:28.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.674+0000 7f327df5a700 1 -- 172.21.15.29:0/3785628262 shutdown_connections 2024-03-11T19:08:28.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.674+0000 7f327df5a700 1 -- 172.21.15.29:0/3785628262 wait complete. 2024-03-11T19:08:28.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.675+0000 7f327df5a700 1 Processor -- start 2024-03-11T19:08:28.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.675+0000 7f327df5a700 1 -- start start 2024-03-11T19:08:28.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.675+0000 7f327df5a700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3278100b70 0x7f327819a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:28.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.675+0000 7f327df5a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3278101530 0x7f327819a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:28.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.675+0000 7f327df5a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3278106b80 0x7f327819ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:28.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.675+0000 7f327df5a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327819f240 con 0x7f3278101530 2024-03-11T19:08:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.675+0000 7f327df5a700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327819f3b0 con 0x7f3278100b70 2024-03-11T19:08:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.675+0000 7f327df5a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327819f570 con 0x7f3278106b80 2024-03-11T19:08:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.675+0000 7f3276ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3278101530 0x7f327819a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.675+0000 7f3276ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3278101530 0x7f327819a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36758/0 (socket says 172.21.15.29:36758) 2024-03-11T19:08:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.675+0000 7f3276ffd700 1 -- 172.21.15.29:0/1136982023 learned_addr learned my addr 172.21.15.29:0/1136982023 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.675+0000 7f32777fe700 1 --2- 172.21.15.29:0/1136982023 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3278100b70 0x7f327819a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.675+0000 7f3277fff700 1 --2- 172.21.15.29:0/1136982023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3278106b80 0x7f327819ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.676+0000 7f3276ffd700 1 -- 172.21.15.29:0/1136982023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3278106b80 msgr2=0x7f327819ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:28.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.676+0000 7f3276ffd700 1 --2- 172.21.15.29:0/1136982023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3278106b80 0x7f327819ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.676+0000 7f3276ffd700 1 -- 172.21.15.29:0/1136982023 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3278100b70 msgr2=0x7f327819a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.676+0000 7f3276ffd700 1 --2- 172.21.15.29:0/1136982023 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3278100b70 0x7f327819a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.676+0000 7f3276ffd700 1 -- 172.21.15.29:0/1136982023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3270009d00 con 0x7f3278101530 2024-03-11T19:08:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.676+0000 7f32777fe700 1 --2- 172.21.15.29:0/1136982023 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3278100b70 0x7f327819a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:08:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.676+0000 7f3276ffd700 1 --2- 172.21.15.29:0/1136982023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3278101530 0x7f327819a8a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f326000d950 tx=0x7f326000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.676+0000 7f3274ff9700 1 -- 172.21.15.29:0/1136982023 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326000f890 con 0x7f3278101530 2024-03-11T19:08:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.676+0000 7f3274ff9700 1 -- 172.21.15.29:0/1136982023 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3260010460 con 0x7f3278101530 2024-03-11T19:08:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.676+0000 7f3274ff9700 1 -- 172.21.15.29:0/1136982023 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326000fea0 con 0x7f3278101530 2024-03-11T19:08:28.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.676+0000 7f327df5a700 1 -- 172.21.15.29:0/1136982023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f327819f850 con 0x7f3278101530 2024-03-11T19:08:28.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.677+0000 7f327df5a700 1 -- 172.21.15.29:0/1136982023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32781a70b0 con 0x7f3278101530 2024-03-11T19:08:28.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.678+0000 7f327df5a700 1 -- 172.21.15.29:0/1136982023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3278110d40 con 0x7f3278101530 2024-03-11T19:08:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.679+0000 7f3274ff9700 1 -- 172.21.15.29:0/1136982023 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f32600105d0 con 0x7f3278101530 2024-03-11T19:08:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.680+0000 7f3274ff9700 1 --2- 172.21.15.29:0/1136982023 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f32640a1b50 0x7f32640a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.680+0000 7f32777fe700 1 --2- 172.21.15.29:0/1136982023 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f32640a1b50 0x7f32640a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.680+0000 7f3274ff9700 1 -- 172.21.15.29:0/1136982023 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f32600cccc0 con 0x7f3278101530 2024-03-11T19:08:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.680+0000 7f3274ff9700 1 --2- 172.21.15.29:0/1136982023 >> [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055] conn(0x7f32640ab330 0x7f32640ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.680+0000 7f3274ff9700 1 -- 172.21.15.29:0/1136982023 --> [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32640ade00 con 0x7f32640ab330 2024-03-11T19:08:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.680+0000 7f3274ff9700 1 -- 172.21.15.29:0/1136982023 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32600cf2a0 con 0x7f3278101530 2024-03-11T19:08:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.680+0000 7f3277fff700 1 --2- 172.21.15.29:0/1136982023 >> [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055] conn(0x7f32640ab330 0x7f32640ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.680+0000 7f32777fe700 1 --2- 172.21.15.29:0/1136982023 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f32640a1b50 0x7f32640a4010 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3268005950 tx=0x7f326800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.681+0000 7f3277fff700 1 --2- 172.21.15.29:0/1136982023 >> [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055] conn(0x7f32640ab330 0x7f32640ad750 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.681+0000 7f3274ff9700 1 -- 172.21.15.29:0/1136982023 <== osd.3 v2:172.21.15.29:6810/2103856055 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f32640ade00 con 0x7f32640ab330 2024-03-11T19:08:28.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.699+0000 7f327df5a700 1 -- 172.21.15.29:0/1136982023 --> [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f327806a300 con 0x7f32640ab330 2024-03-11T19:08:28.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.699+0000 7f3274ff9700 1 -- 172.21.15.29:0/1136982023 <== osd.3 v2:172.21.15.29:6810/2103856055 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f327806a300 con 0x7f32640ab330 2024-03-11T19:08:28.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.700+0000 7f327df5a700 1 -- 172.21.15.29:0/1136982023 >> [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055] conn(0x7f32640ab330 msgr2=0x7f32640ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:28.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.700+0000 7f327df5a700 1 --2- 172.21.15.29:0/1136982023 >> [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055] conn(0x7f32640ab330 0x7f32640ad750 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:28.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.700+0000 7f327df5a700 1 -- 172.21.15.29:0/1136982023 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f32640a1b50 msgr2=0x7f32640a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:28.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.700+0000 7f327df5a700 1 --2- 172.21.15.29:0/1136982023 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f32640a1b50 0x7f32640a4010 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3268005950 tx=0x7f326800b410 comp rx=0 tx=0).stop 2024-03-11T19:08:28.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.700+0000 7f327df5a700 1 -- 172.21.15.29:0/1136982023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3278101530 msgr2=0x7f327819a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:28.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.700+0000 7f327df5a700 1 --2- 172.21.15.29:0/1136982023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3278101530 0x7f327819a8a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f326000d950 tx=0x7f326000dc60 comp rx=0 tx=0).stop 2024-03-11T19:08:28.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.700+0000 7f3277fff700 1 -- 172.21.15.29:0/1136982023 reap_dead start 2024-03-11T19:08:28.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.700+0000 7f327df5a700 1 -- 172.21.15.29:0/1136982023 shutdown_connections 2024-03-11T19:08:28.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.700+0000 7f327df5a700 1 -- 172.21.15.29:0/1136982023 >> 172.21.15.29:0/1136982023 conn(0x7f3278077aa0 msgr2=0x7f3278069920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:28.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.700+0000 7f327df5a700 1 -- 172.21.15.29:0/1136982023 shutdown_connections 2024-03-11T19:08:28.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:28.700+0000 7f327df5a700 1 -- 172.21.15.29:0/1136982023 wait complete. 2024-03-11T19:08:29.149 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:28 smithi029 ceph-mon[32439]: pgmap v309: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-11T19:08:29.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:28 smithi190 ceph-mon[36292]: pgmap v309: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-11T19:08:29.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:28 smithi170 ceph-mon[35690]: pgmap v309: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-11T19:08:29.644 INFO:teuthology.orchestra.run.smithi029.stdout:34359738444 2024-03-11T19:08:29.645 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T19:08:29.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.830+0000 7f8808931700 1 -- 172.21.15.29:0/3894876748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8804106ce0 msgr2=0x7f88041070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:29.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.830+0000 7f8808931700 1 --2- 172.21.15.29:0/3894876748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8804106ce0 0x7f88041070c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f87f40098f0 tx=0x7f87f4009c00 comp rx=0 tx=0).stop 2024-03-11T19:08:29.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.830+0000 7f8808931700 1 -- 172.21.15.29:0/3894876748 shutdown_connections 2024-03-11T19:08:29.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.830+0000 7f8808931700 1 --2- 172.21.15.29:0/3894876748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8804100c20 0x7f880410eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:29.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.830+0000 7f8808931700 1 --2- 172.21.15.29:0/3894876748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8804106ce0 0x7f88041070c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:29.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.830+0000 7f8808931700 1 --2- 172.21.15.29:0/3894876748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8804100260 0x7f88041006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:29.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.830+0000 7f8808931700 1 -- 172.21.15.29:0/3894876748 >> 172.21.15.29:0/3894876748 conn(0x7f8804077170 msgr2=0x7f88040fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:29.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.830+0000 7f8808931700 1 -- 172.21.15.29:0/3894876748 shutdown_connections 2024-03-11T19:08:29.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.830+0000 7f8808931700 1 -- 172.21.15.29:0/3894876748 wait complete. 2024-03-11T19:08:29.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.831+0000 7f8808931700 1 Processor -- start 2024-03-11T19:08:29.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.831+0000 7f8808931700 1 -- start start 2024-03-11T19:08:29.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f8808931700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8804100260 0x7f8804198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:29.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f8808931700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8804100c20 0x7f8804198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:29.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f8808931700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8804106ce0 0x7f880419cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:29.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f8808931700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880419d0e0 con 0x7f8804106ce0 2024-03-11T19:08:29.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f8808931700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880419d250 con 0x7f8804100260 2024-03-11T19:08:29.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f8808931700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880419d410 con 0x7f8804100c20 2024-03-11T19:08:29.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f8802d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8804106ce0 0x7f880419cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:29.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f8802d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8804106ce0 0x7f880419cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36784/0 (socket says 172.21.15.29:36784) 2024-03-11T19:08:29.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f8802d9d700 1 -- 172.21.15.29:0/3802817083 learned_addr learned my addr 172.21.15.29:0/3802817083 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:29.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f8801d9b700 1 --2- 172.21.15.29:0/3802817083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8804100c20 0x7f8804198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:29.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f880259c700 1 --2- 172.21.15.29:0/3802817083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8804100260 0x7f8804198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:29.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f8802d9d700 1 -- 172.21.15.29:0/3802817083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8804100c20 msgr2=0x7f8804198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:29.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f8802d9d700 1 --2- 172.21.15.29:0/3802817083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8804100c20 0x7f8804198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:29.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f8802d9d700 1 -- 172.21.15.29:0/3802817083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8804100260 msgr2=0x7f8804198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:29.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f8802d9d700 1 --2- 172.21.15.29:0/3802817083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8804100260 0x7f8804198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:29.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f8802d9d700 1 -- 172.21.15.29:0/3802817083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f8009520 con 0x7f8804106ce0 2024-03-11T19:08:29.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f8802d9d700 1 --2- 172.21.15.29:0/3802817083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8804106ce0 0x7f880419cad0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f87fc00e3f0 tx=0x7f87fc00e700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:29.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f87f37fe700 1 -- 172.21.15.29:0/3802817083 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87fc0090d0 con 0x7f8804106ce0 2024-03-11T19:08:29.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f8808931700 1 -- 172.21.15.29:0/3802817083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87fc009d00 con 0x7f8804106ce0 2024-03-11T19:08:29.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f87f37fe700 1 -- 172.21.15.29:0/3802817083 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f87fc00f040 con 0x7f8804106ce0 2024-03-11T19:08:29.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f87f37fe700 1 -- 172.21.15.29:0/3802817083 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87fc0146e0 con 0x7f8804106ce0 2024-03-11T19:08:29.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.832+0000 7f8808931700 1 -- 172.21.15.29:0/3802817083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f4009520 con 0x7f8804106ce0 2024-03-11T19:08:29.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.833+0000 7f8808931700 1 -- 172.21.15.29:0/3802817083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8804111fd0 con 0x7f8804106ce0 2024-03-11T19:08:29.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.834+0000 7f87f37fe700 1 -- 172.21.15.29:0/3802817083 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f87fc0075c0 con 0x7f8804106ce0 2024-03-11T19:08:29.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.835+0000 7f87f37fe700 1 --2- 172.21.15.29:0/3802817083 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f87ec0a1b50 0x7f87ec0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:29.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.835+0000 7f880259c700 1 --2- 172.21.15.29:0/3802817083 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f87ec0a1b50 0x7f87ec0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:29.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.835+0000 7f87f37fe700 1 -- 172.21.15.29:0/3802817083 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f87fc0cdf40 con 0x7f8804106ce0 2024-03-11T19:08:29.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.835+0000 7f87f37fe700 1 --2- 172.21.15.29:0/3802817083 >> [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214] conn(0x7f87ec0ab330 0x7f87ec0ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:29.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.835+0000 7f87f37fe700 1 -- 172.21.15.29:0/3802817083 --> [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87ec0ade00 con 0x7f87ec0ab330 2024-03-11T19:08:29.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.835+0000 7f87f37fe700 1 -- 172.21.15.29:0/3802817083 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87fc0ce3e0 con 0x7f8804106ce0 2024-03-11T19:08:29.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.835+0000 7f8801d9b700 1 --2- 172.21.15.29:0/3802817083 >> [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214] conn(0x7f87ec0ab330 0x7f87ec0ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:29.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.835+0000 7f880259c700 1 --2- 172.21.15.29:0/3802817083 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f87ec0a1b50 0x7f87ec0a4010 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f87f4009fd0 tx=0x7f87f4019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:29.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.836+0000 7f8801d9b700 1 --2- 172.21.15.29:0/3802817083 >> [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214] conn(0x7f87ec0ab330 0x7f87ec0ad750 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:29.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.836+0000 7f87f37fe700 1 -- 172.21.15.29:0/3802817083 <== osd.6 v2:172.21.15.29:6818/1792625214 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f87ec0ade00 con 0x7f87ec0ab330 2024-03-11T19:08:29.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.853+0000 7f8808931700 1 -- 172.21.15.29:0/3802817083 --> [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8804062300 con 0x7f87ec0ab330 2024-03-11T19:08:29.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.854+0000 7f87f37fe700 1 -- 172.21.15.29:0/3802817083 <== osd.6 v2:172.21.15.29:6818/1792625214 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8804062300 con 0x7f87ec0ab330 2024-03-11T19:08:29.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.854+0000 7f8808931700 1 -- 172.21.15.29:0/3802817083 >> [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214] conn(0x7f87ec0ab330 msgr2=0x7f87ec0ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:29.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.854+0000 7f8808931700 1 --2- 172.21.15.29:0/3802817083 >> [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214] conn(0x7f87ec0ab330 0x7f87ec0ad750 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:29.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.855+0000 7f8808931700 1 -- 172.21.15.29:0/3802817083 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f87ec0a1b50 msgr2=0x7f87ec0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:29.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.855+0000 7f8808931700 1 --2- 172.21.15.29:0/3802817083 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f87ec0a1b50 0x7f87ec0a4010 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f87f4009fd0 tx=0x7f87f4019040 comp rx=0 tx=0).stop 2024-03-11T19:08:29.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.855+0000 7f8808931700 1 -- 172.21.15.29:0/3802817083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8804106ce0 msgr2=0x7f880419cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:29.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.855+0000 7f8808931700 1 --2- 172.21.15.29:0/3802817083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8804106ce0 0x7f880419cad0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f87fc00e3f0 tx=0x7f87fc00e700 comp rx=0 tx=0).stop 2024-03-11T19:08:29.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.855+0000 7f8802d9d700 1 -- 172.21.15.29:0/3802817083 reap_dead start 2024-03-11T19:08:29.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.855+0000 7f8808931700 1 -- 172.21.15.29:0/3802817083 shutdown_connections 2024-03-11T19:08:29.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.855+0000 7f8808931700 1 -- 172.21.15.29:0/3802817083 >> 172.21.15.29:0/3802817083 conn(0x7f8804077170 msgr2=0x7f8804104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:29.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.855+0000 7f8808931700 1 -- 172.21.15.29:0/3802817083 shutdown_connections 2024-03-11T19:08:29.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:29.855+0000 7f8808931700 1 -- 172.21.15.29:0/3802817083 wait complete. 2024-03-11T19:08:30.680 INFO:teuthology.orchestra.run.smithi029.stdout:77309411395 2024-03-11T19:08:30.680 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T19:08:31.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:30 smithi190 ceph-mon[36292]: pgmap v310: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-11T19:08:31.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:30 smithi170 ceph-mon[35690]: pgmap v310: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-11T19:08:31.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:30 smithi029 ceph-mon[32439]: pgmap v310: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-11T19:08:31.492 INFO:teuthology.orchestra.run.smithi029.stdout:111669149758 2024-03-11T19:08:31.492 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-11T19:08:32.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.995+0000 7f8d1249b700 1 -- 172.21.15.29:0/1409057037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d0c101cc0 msgr2=0x7f8d0c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:32.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.995+0000 7f8d1249b700 1 --2- 172.21.15.29:0/1409057037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d0c101cc0 0x7f8d0c102140 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8d000099c0 tx=0x7f8d00009cd0 comp rx=0 tx=0).stop 2024-03-11T19:08:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.995+0000 7f8d1249b700 1 -- 172.21.15.29:0/1409057037 shutdown_connections 2024-03-11T19:08:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.995+0000 7f8d1249b700 1 --2- 172.21.15.29:0/1409057037 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d0c102680 0x7f8d0c10eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.995+0000 7f8d1249b700 1 --2- 172.21.15.29:0/1409057037 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d0c107ca0 0x7f8d0c108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.995+0000 7f8d1249b700 1 --2- 172.21.15.29:0/1409057037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d0c101cc0 0x7f8d0c102140 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.995+0000 7f8d1249b700 1 -- 172.21.15.29:0/1409057037 >> 172.21.15.29:0/1409057037 conn(0x7f8d0c078da0 msgr2=0x7f8d0c0791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.995+0000 7f8d1249b700 1 -- 172.21.15.29:0/1409057037 shutdown_connections 2024-03-11T19:08:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.995+0000 7f8d1249b700 1 -- 172.21.15.29:0/1409057037 wait complete. 2024-03-11T19:08:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.996+0000 7f8d1249b700 1 Processor -- start 2024-03-11T19:08:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.996+0000 7f8d1249b700 1 -- start start 2024-03-11T19:08:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.996+0000 7f8d1249b700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d0c101cc0 0x7f8d0c1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d1249b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d0c102680 0x7f8d0c198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d1249b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d0c107ca0 0x7f8d0c19c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d1249b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c19cf50 con 0x7f8d0c102680 2024-03-11T19:08:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d1249b700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c19d0c0 con 0x7f8d0c101cc0 2024-03-11T19:08:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d1249b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c19d280 con 0x7f8d0c107ca0 2024-03-11T19:08:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d0b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d0c102680 0x7f8d0c198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d0b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d0c102680 0x7f8d0c198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36810/0 (socket says 172.21.15.29:36810) 2024-03-11T19:08:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d0b7fe700 1 -- 172.21.15.29:0/2409273320 learned_addr learned my addr 172.21.15.29:0/2409273320 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d10a38700 1 --2- 172.21.15.29:0/2409273320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d0c107ca0 0x7f8d0c19c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d0b7fe700 1 -- 172.21.15.29:0/2409273320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d0c107ca0 msgr2=0x7f8d0c19c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d0b7fe700 1 --2- 172.21.15.29:0/2409273320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d0c107ca0 0x7f8d0c19c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d0b7fe700 1 -- 172.21.15.29:0/2409273320 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d0c101cc0 msgr2=0x7f8d0c1980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d0b7fe700 1 --2- 172.21.15.29:0/2409273320 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d0c101cc0 0x7f8d0c1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d0b7fe700 1 -- 172.21.15.29:0/2409273320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d000095f0 con 0x7f8d0c102680 2024-03-11T19:08:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d10a38700 1 --2- 172.21.15.29:0/2409273320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d0c107ca0 0x7f8d0c19c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:08:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d0b7fe700 1 --2- 172.21.15.29:0/2409273320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d0c102680 0x7f8d0c198600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000b580 tx=0x7f8d00009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d097fa700 1 -- 172.21.15.29:0/2409273320 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0001d070 con 0x7f8d0c102680 2024-03-11T19:08:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d1249b700 1 -- 172.21.15.29:0/2409273320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d0c19d500 con 0x7f8d0c102680 2024-03-11T19:08:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d097fa700 1 -- 172.21.15.29:0/2409273320 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8d0000fca0 con 0x7f8d0c102680 2024-03-11T19:08:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d097fa700 1 -- 172.21.15.29:0/2409273320 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d00022e00 con 0x7f8d0c102680 2024-03-11T19:08:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.997+0000 7f8d1249b700 1 -- 172.21.15.29:0/2409273320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d0c1a4dc0 con 0x7f8d0c102680 2024-03-11T19:08:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.998+0000 7f8d1249b700 1 -- 172.21.15.29:0/2409273320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8d0c103350 con 0x7f8d0c102680 2024-03-11T19:08:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:31.999+0000 7f8d097fa700 1 -- 172.21.15.29:0/2409273320 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f8d00022600 con 0x7f8d0c102680 2024-03-11T19:08:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.000+0000 7f8d097fa700 1 --2- 172.21.15.29:0/2409273320 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8cf40a1b50 0x7f8cf40a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.000+0000 7f8d0bfff700 1 --2- 172.21.15.29:0/2409273320 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8cf40a1b50 0x7f8cf40a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:32.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.000+0000 7f8d097fa700 1 -- 172.21.15.29:0/2409273320 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f8d000ce8d0 con 0x7f8d0c102680 2024-03-11T19:08:32.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.000+0000 7f8d097fa700 1 --2- 172.21.15.29:0/2409273320 >> [v2:172.21.15.190:6824/3926470159,v1:172.21.15.190:6825/3926470159] conn(0x7f8cf40ab330 0x7f8cf40ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:32.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.000+0000 7f8d097fa700 1 -- 172.21.15.29:0/2409273320 --> [v2:172.21.15.190:6824/3926470159,v1:172.21.15.190:6825/3926470159] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8cf40ade00 con 0x7f8cf40ab330 2024-03-11T19:08:32.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.000+0000 7f8d097fa700 1 -- 172.21.15.29:0/2409273320 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d000ced70 con 0x7f8d0c102680 2024-03-11T19:08:32.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.000+0000 7f8d0bfff700 1 --2- 172.21.15.29:0/2409273320 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8cf40a1b50 0x7f8cf40a4010 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc005950 tx=0x7f8cfc0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:32.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.000+0000 7f8d10a38700 1 --2- 172.21.15.29:0/2409273320 >> [v2:172.21.15.190:6824/3926470159,v1:172.21.15.190:6825/3926470159] conn(0x7f8cf40ab330 0x7f8cf40ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:32.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.001+0000 7f8d10a38700 1 --2- 172.21.15.29:0/2409273320 >> [v2:172.21.15.190:6824/3926470159,v1:172.21.15.190:6825/3926470159] conn(0x7f8cf40ab330 0x7f8cf40ad750 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:32.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.002+0000 7f8d097fa700 1 -- 172.21.15.29:0/2409273320 <== osd.11 v2:172.21.15.190:6824/3926470159 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8cf40ade00 con 0x7f8cf40ab330 2024-03-11T19:08:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.024+0000 7f8d1249b700 1 -- 172.21.15.29:0/2409273320 --> [v2:172.21.15.190:6824/3926470159,v1:172.21.15.190:6825/3926470159] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8d0c111ee0 con 0x7f8cf40ab330 2024-03-11T19:08:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.026+0000 7f8d097fa700 1 -- 172.21.15.29:0/2409273320 <== osd.11 v2:172.21.15.190:6824/3926470159 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8d0c111ee0 con 0x7f8cf40ab330 2024-03-11T19:08:32.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.026+0000 7f8d1249b700 1 -- 172.21.15.29:0/2409273320 >> [v2:172.21.15.190:6824/3926470159,v1:172.21.15.190:6825/3926470159] conn(0x7f8cf40ab330 msgr2=0x7f8cf40ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:32.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.026+0000 7f8d1249b700 1 --2- 172.21.15.29:0/2409273320 >> [v2:172.21.15.190:6824/3926470159,v1:172.21.15.190:6825/3926470159] conn(0x7f8cf40ab330 0x7f8cf40ad750 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:32.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.026+0000 7f8d1249b700 1 -- 172.21.15.29:0/2409273320 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8cf40a1b50 msgr2=0x7f8cf40a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:32.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.026+0000 7f8d1249b700 1 --2- 172.21.15.29:0/2409273320 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8cf40a1b50 0x7f8cf40a4010 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc005950 tx=0x7f8cfc0058e0 comp rx=0 tx=0).stop 2024-03-11T19:08:32.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.026+0000 7f8d1249b700 1 -- 172.21.15.29:0/2409273320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d0c102680 msgr2=0x7f8d0c198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:32.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.026+0000 7f8d1249b700 1 --2- 172.21.15.29:0/2409273320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d0c102680 0x7f8d0c198600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000b580 tx=0x7f8d00009510 comp rx=0 tx=0).stop 2024-03-11T19:08:32.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.026+0000 7f8d10a38700 1 -- 172.21.15.29:0/2409273320 reap_dead start 2024-03-11T19:08:32.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.026+0000 7f8d1249b700 1 -- 172.21.15.29:0/2409273320 shutdown_connections 2024-03-11T19:08:32.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.026+0000 7f8d1249b700 1 -- 172.21.15.29:0/2409273320 >> 172.21.15.29:0/2409273320 conn(0x7f8d0c078da0 msgr2=0x7f8d0c100a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:32.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.026+0000 7f8d1249b700 1 -- 172.21.15.29:0/2409273320 shutdown_connections 2024-03-11T19:08:32.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.026+0000 7f8d1249b700 1 -- 172.21.15.29:0/2409273320 wait complete. 2024-03-11T19:08:32.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.512+0000 7f3b8f133700 1 -- 172.21.15.29:0/4032765758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88105e90 msgr2=0x7f3b8810da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.512+0000 7f3b8f133700 1 --2- 172.21.15.29:0/4032765758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88105e90 0x7f3b8810da60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000b0a0 tx=0x7f3b8000b3b0 comp rx=0 tx=0).stop 2024-03-11T19:08:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.512+0000 7f3b8f133700 1 -- 172.21.15.29:0/4032765758 shutdown_connections 2024-03-11T19:08:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.512+0000 7f3b8f133700 1 --2- 172.21.15.29:0/4032765758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88105e90 0x7f3b8810da60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.512+0000 7f3b8f133700 1 --2- 172.21.15.29:0/4032765758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b88100e00 0x7f3b881011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.512+0000 7f3b8f133700 1 --2- 172.21.15.29:0/4032765758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b881017b0 0x7f3b881057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.512+0000 7f3b8f133700 1 -- 172.21.15.29:0/4032765758 >> 172.21.15.29:0/4032765758 conn(0x7f3b88076b90 msgr2=0x7f3b88076fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.513+0000 7f3b8f133700 1 -- 172.21.15.29:0/4032765758 shutdown_connections 2024-03-11T19:08:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.513+0000 7f3b8f133700 1 -- 172.21.15.29:0/4032765758 wait complete. 2024-03-11T19:08:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.513+0000 7f3b8f133700 1 Processor -- start 2024-03-11T19:08:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.513+0000 7f3b8f133700 1 -- start start 2024-03-11T19:08:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.513+0000 7f3b8f133700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88100e00 0x7f3b88102850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.513+0000 7f3b8f133700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b881017b0 0x7f3b88102d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.513+0000 7f3b8f133700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b88105e90 0x7f3b881032d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.513+0000 7f3b8f133700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b881a65d0 con 0x7f3b88100e00 2024-03-11T19:08:32.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.513+0000 7f3b8f133700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b881a6740 con 0x7f3b881017b0 2024-03-11T19:08:32.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.513+0000 7f3b8f133700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b881a6900 con 0x7f3b88105e90 2024-03-11T19:08:32.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.513+0000 7f3b8cecf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88100e00 0x7f3b88102850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:32.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.513+0000 7f3b8cecf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88100e00 0x7f3b88102850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36820/0 (socket says 172.21.15.29:36820) 2024-03-11T19:08:32.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.513+0000 7f3b8cecf700 1 -- 172.21.15.29:0/2860905401 learned_addr learned my addr 172.21.15.29:0/2860905401 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:32.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.513+0000 7f3b8d6d0700 1 --2- 172.21.15.29:0/2860905401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b88105e90 0x7f3b881032d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:32.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.514+0000 7f3b8cecf700 1 -- 172.21.15.29:0/2860905401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b88105e90 msgr2=0x7f3b881032d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:32.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.514+0000 7f3b8cecf700 1 --2- 172.21.15.29:0/2860905401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b88105e90 0x7f3b881032d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:32.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.514+0000 7f3b8cecf700 1 -- 172.21.15.29:0/2860905401 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b881017b0 msgr2=0x7f3b88102d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:32.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.514+0000 7f3b8cecf700 1 --2- 172.21.15.29:0/2860905401 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b881017b0 0x7f3b88102d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:32.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.514+0000 7f3b8cecf700 1 -- 172.21.15.29:0/2860905401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b80009d00 con 0x7f3b88100e00 2024-03-11T19:08:32.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.514+0000 7f3b8cecf700 1 --2- 172.21.15.29:0/2860905401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88100e00 0x7f3b88102850 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c00e9a0 tx=0x7f3b7c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:32.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.514+0000 7f3b85ffb700 1 -- 172.21.15.29:0/2860905401 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c009790 con 0x7f3b88100e00 2024-03-11T19:08:32.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.514+0000 7f3b85ffb700 1 -- 172.21.15.29:0/2860905401 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3b7c00cbd0 con 0x7f3b88100e00 2024-03-11T19:08:32.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.514+0000 7f3b85ffb700 1 -- 172.21.15.29:0/2860905401 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c018680 con 0x7f3b88100e00 2024-03-11T19:08:32.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.514+0000 7f3b8f133700 1 -- 172.21.15.29:0/2860905401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b881a6be0 con 0x7f3b88100e00 2024-03-11T19:08:32.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.514+0000 7f3b8f133700 1 -- 172.21.15.29:0/2860905401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b881a7130 con 0x7f3b88100e00 2024-03-11T19:08:32.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.515+0000 7f3b8f133700 1 -- 172.21.15.29:0/2860905401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3b88110d90 con 0x7f3b88100e00 2024-03-11T19:08:32.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.516+0000 7f3b85ffb700 1 -- 172.21.15.29:0/2860905401 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f3b7c00cd40 con 0x7f3b88100e00 2024-03-11T19:08:32.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.517+0000 7f3b85ffb700 1 --2- 172.21.15.29:0/2860905401 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3b780a1bd0 0x7f3b780a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:32.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.517+0000 7f3b87fff700 1 --2- 172.21.15.29:0/2860905401 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3b780a1bd0 0x7f3b780a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:32.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.517+0000 7f3b85ffb700 1 -- 172.21.15.29:0/2860905401 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3b7c014070 con 0x7f3b88100e00 2024-03-11T19:08:32.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.517+0000 7f3b85ffb700 1 --2- 172.21.15.29:0/2860905401 >> [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226] conn(0x7f3b780ab3b0 0x7f3b780ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:32.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.517+0000 7f3b85ffb700 1 -- 172.21.15.29:0/2860905401 --> [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b780ade80 con 0x7f3b780ab3b0 2024-03-11T19:08:32.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.517+0000 7f3b85ffb700 1 -- 172.21.15.29:0/2860905401 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b7c0cdd50 con 0x7f3b88100e00 2024-03-11T19:08:32.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.517+0000 7f3b87fff700 1 --2- 172.21.15.29:0/2860905401 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3b780a1bd0 0x7f3b780a4090 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3b88107440 tx=0x7f3b74005bf0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:32.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.517+0000 7f3b8d6d0700 1 --2- 172.21.15.29:0/2860905401 >> [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226] conn(0x7f3b780ab3b0 0x7f3b780ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:32.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.518+0000 7f3b8d6d0700 1 --2- 172.21.15.29:0/2860905401 >> [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226] conn(0x7f3b780ab3b0 0x7f3b780ad7d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:32.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.519+0000 7f3b85ffb700 1 -- 172.21.15.29:0/2860905401 <== osd.4 v2:172.21.15.170:6808/1979690226 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3b780ade80 con 0x7f3b780ab3b0 2024-03-11T19:08:32.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.538+0000 7f3b8f133700 1 -- 172.21.15.29:0/2860905401 --> [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3b88062300 con 0x7f3b780ab3b0 2024-03-11T19:08:32.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.538+0000 7f3b85ffb700 1 -- 172.21.15.29:0/2860905401 <== osd.4 v2:172.21.15.170:6808/1979690226 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f3b88062300 con 0x7f3b780ab3b0 2024-03-11T19:08:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.539+0000 7f3b8f133700 1 -- 172.21.15.29:0/2860905401 >> [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226] conn(0x7f3b780ab3b0 msgr2=0x7f3b780ad7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.539+0000 7f3b8f133700 1 --2- 172.21.15.29:0/2860905401 >> [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226] conn(0x7f3b780ab3b0 0x7f3b780ad7d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.539+0000 7f3b8f133700 1 -- 172.21.15.29:0/2860905401 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3b780a1bd0 msgr2=0x7f3b780a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.539+0000 7f3b8f133700 1 --2- 172.21.15.29:0/2860905401 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3b780a1bd0 0x7f3b780a4090 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3b88107440 tx=0x7f3b74005bf0 comp rx=0 tx=0).stop 2024-03-11T19:08:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.539+0000 7f3b8f133700 1 -- 172.21.15.29:0/2860905401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88100e00 msgr2=0x7f3b88102850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.539+0000 7f3b8f133700 1 --2- 172.21.15.29:0/2860905401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88100e00 0x7f3b88102850 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c00e9a0 tx=0x7f3b7c00ecb0 comp rx=0 tx=0).stop 2024-03-11T19:08:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.539+0000 7f3b8d6d0700 1 -- 172.21.15.29:0/2860905401 reap_dead start 2024-03-11T19:08:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.539+0000 7f3b8f133700 1 -- 172.21.15.29:0/2860905401 shutdown_connections 2024-03-11T19:08:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.539+0000 7f3b8f133700 1 -- 172.21.15.29:0/2860905401 >> 172.21.15.29:0/2860905401 conn(0x7f3b88076b90 msgr2=0x7f3b8806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.539+0000 7f3b8f133700 1 -- 172.21.15.29:0/2860905401 shutdown_connections 2024-03-11T19:08:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:32.539+0000 7f3b8f133700 1 -- 172.21.15.29:0/2860905401 wait complete. 2024-03-11T19:08:33.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:32 smithi190 ceph-mon[36292]: pgmap v311: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-11T19:08:33.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:32 smithi170 ceph-mon[35690]: pgmap v311: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-11T19:08:33.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:32 smithi029 ceph-mon[32439]: pgmap v311: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-11T19:08:33.533 INFO:teuthology.orchestra.run.smithi029.stdout:146028888122 2024-03-11T19:08:33.533 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-11T19:08:33.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.737+0000 7f2a08ee6700 1 -- 172.21.15.29:0/3082210286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a040743e0 msgr2=0x7f2a04074860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:33.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.737+0000 7f2a08ee6700 1 --2- 172.21.15.29:0/3082210286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a040743e0 0x7f2a04074860 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f29f8009990 tx=0x7f29f8009ca0 comp rx=0 tx=0).stop 2024-03-11T19:08:33.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.737+0000 7f2a08ee6700 1 -- 172.21.15.29:0/3082210286 shutdown_connections 2024-03-11T19:08:33.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.737+0000 7f2a08ee6700 1 --2- 172.21.15.29:0/3082210286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a04112930 0x7f2a04114d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:33.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.737+0000 7f2a08ee6700 1 --2- 172.21.15.29:0/3082210286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a04075c80 0x7f2a04073ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:33.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.737+0000 7f2a08ee6700 1 --2- 172.21.15.29:0/3082210286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a040743e0 0x7f2a04074860 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:33.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.737+0000 7f2a08ee6700 1 -- 172.21.15.29:0/3082210286 >> 172.21.15.29:0/3082210286 conn(0x7f2a04078d00 msgr2=0x7f2a04079110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:33.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.737+0000 7f2a08ee6700 1 -- 172.21.15.29:0/3082210286 shutdown_connections 2024-03-11T19:08:33.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.738+0000 7f2a08ee6700 1 -- 172.21.15.29:0/3082210286 wait complete. 2024-03-11T19:08:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.738+0000 7f2a08ee6700 1 Processor -- start 2024-03-11T19:08:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.738+0000 7f2a08ee6700 1 -- start start 2024-03-11T19:08:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.739+0000 7f2a08ee6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a040743e0 0x7f2a0419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.739+0000 7f2a08ee6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a04075c80 0x7f2a0419ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.739+0000 7f2a08ee6700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a04112930 0x7f2a041a2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.739+0000 7f2a08ee6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a041a3570 con 0x7f2a04075c80 2024-03-11T19:08:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.739+0000 7f2a08ee6700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a041a36e0 con 0x7f2a04112930 2024-03-11T19:08:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.739+0000 7f2a08ee6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a041a38a0 con 0x7f2a040743e0 2024-03-11T19:08:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.739+0000 7f2a01d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a04075c80 0x7f2a0419ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.739+0000 7f2a01d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a04075c80 0x7f2a0419ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36836/0 (socket says 172.21.15.29:36836) 2024-03-11T19:08:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.739+0000 7f2a01d9b700 1 -- 172.21.15.29:0/2584398269 learned_addr learned my addr 172.21.15.29:0/2584398269 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.739+0000 7f2a02d9d700 1 --2- 172.21.15.29:0/2584398269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a04112930 0x7f2a041a2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.739+0000 7f2a0259c700 1 --2- 172.21.15.29:0/2584398269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a040743e0 0x7f2a0419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.739+0000 7f2a01d9b700 1 -- 172.21.15.29:0/2584398269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a040743e0 msgr2=0x7f2a0419e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.739+0000 7f2a01d9b700 1 --2- 172.21.15.29:0/2584398269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a040743e0 0x7f2a0419e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.739+0000 7f2a01d9b700 1 -- 172.21.15.29:0/2584398269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a04112930 msgr2=0x7f2a041a2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.739+0000 7f2a01d9b700 1 --2- 172.21.15.29:0/2584398269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a04112930 0x7f2a041a2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.739+0000 7f2a01d9b700 1 -- 172.21.15.29:0/2584398269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f80095f0 con 0x7f2a04075c80 2024-03-11T19:08:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.739+0000 7f2a02d9d700 1 --2- 172.21.15.29:0/2584398269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a04112930 0x7f2a041a2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:08:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.739+0000 7f2a01d9b700 1 --2- 172.21.15.29:0/2584398269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a04075c80 0x7f2a0419ebd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f29f8000c00 tx=0x7f29f800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.740+0000 7f29f37fe700 1 -- 172.21.15.29:0/2584398269 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f801d070 con 0x7f2a04075c80 2024-03-11T19:08:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.740+0000 7f29f37fe700 1 -- 172.21.15.29:0/2584398269 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f29f800bdc0 con 0x7f2a04075c80 2024-03-11T19:08:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.740+0000 7f29f37fe700 1 -- 172.21.15.29:0/2584398269 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f8022ca0 con 0x7f2a04075c80 2024-03-11T19:08:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.740+0000 7f2a08ee6700 1 -- 172.21.15.29:0/2584398269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a041a3b20 con 0x7f2a04075c80 2024-03-11T19:08:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.740+0000 7f2a08ee6700 1 -- 172.21.15.29:0/2584398269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a041ab210 con 0x7f2a04075c80 2024-03-11T19:08:33.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.741+0000 7f2a08ee6700 1 -- 172.21.15.29:0/2584398269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2a04075530 con 0x7f2a04075c80 2024-03-11T19:08:33.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.742+0000 7f29f37fe700 1 -- 172.21.15.29:0/2584398269 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f29f8022470 con 0x7f2a04075c80 2024-03-11T19:08:33.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.742+0000 7f29f37fe700 1 --2- 172.21.15.29:0/2584398269 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f29ec0a1bd0 0x7f29ec0a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:33.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.742+0000 7f2a0259c700 1 --2- 172.21.15.29:0/2584398269 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f29ec0a1bd0 0x7f29ec0a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.743+0000 7f29f37fe700 1 -- 172.21.15.29:0/2584398269 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f29f80cee20 con 0x7f2a04075c80 2024-03-11T19:08:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.743+0000 7f29f37fe700 1 --2- 172.21.15.29:0/2584398269 >> [v2:172.21.15.170:6800/531888562,v1:172.21.15.170:6801/531888562] conn(0x7f29ec0ab3b0 0x7f29ec0ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.743+0000 7f29f37fe700 1 -- 172.21.15.29:0/2584398269 --> [v2:172.21.15.170:6800/531888562,v1:172.21.15.170:6801/531888562] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f29ec0ade80 con 0x7f29ec0ab3b0 2024-03-11T19:08:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.743+0000 7f29f37fe700 1 -- 172.21.15.29:0/2584398269 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29f80cf190 con 0x7f2a04075c80 2024-03-11T19:08:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.743+0000 7f2a0259c700 1 --2- 172.21.15.29:0/2584398269 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f29ec0a1bd0 0x7f29ec0a4090 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f29f4005950 tx=0x7f29f400a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.743+0000 7f2a02d9d700 1 --2- 172.21.15.29:0/2584398269 >> [v2:172.21.15.170:6800/531888562,v1:172.21.15.170:6801/531888562] conn(0x7f29ec0ab3b0 0x7f29ec0ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.743+0000 7f2a02d9d700 1 --2- 172.21.15.29:0/2584398269 >> [v2:172.21.15.170:6800/531888562,v1:172.21.15.170:6801/531888562] conn(0x7f29ec0ab3b0 0x7f29ec0ad7d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.744+0000 7f29f37fe700 1 -- 172.21.15.29:0/2584398269 <== osd.1 v2:172.21.15.170:6800/531888562 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f29ec0ade80 con 0x7f29ec0ab3b0 2024-03-11T19:08:33.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.767+0000 7f2a08ee6700 1 -- 172.21.15.29:0/2584398269 --> [v2:172.21.15.170:6800/531888562,v1:172.21.15.170:6801/531888562] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2a04118490 con 0x7f29ec0ab3b0 2024-03-11T19:08:33.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.767+0000 7f29f37fe700 1 -- 172.21.15.29:0/2584398269 <== osd.1 v2:172.21.15.170:6800/531888562 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2a04118490 con 0x7f29ec0ab3b0 2024-03-11T19:08:33.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.768+0000 7f2a08ee6700 1 -- 172.21.15.29:0/2584398269 >> [v2:172.21.15.170:6800/531888562,v1:172.21.15.170:6801/531888562] conn(0x7f29ec0ab3b0 msgr2=0x7f29ec0ad7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:33.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.768+0000 7f2a08ee6700 1 --2- 172.21.15.29:0/2584398269 >> [v2:172.21.15.170:6800/531888562,v1:172.21.15.170:6801/531888562] conn(0x7f29ec0ab3b0 0x7f29ec0ad7d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:33.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.768+0000 7f2a08ee6700 1 -- 172.21.15.29:0/2584398269 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f29ec0a1bd0 msgr2=0x7f29ec0a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:33.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.768+0000 7f2a08ee6700 1 --2- 172.21.15.29:0/2584398269 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f29ec0a1bd0 0x7f29ec0a4090 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f29f4005950 tx=0x7f29f400a300 comp rx=0 tx=0).stop 2024-03-11T19:08:33.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.768+0000 7f2a08ee6700 1 -- 172.21.15.29:0/2584398269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a04075c80 msgr2=0x7f2a0419ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:33.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.768+0000 7f2a08ee6700 1 --2- 172.21.15.29:0/2584398269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a04075c80 0x7f2a0419ebd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f29f8000c00 tx=0x7f29f800f690 comp rx=0 tx=0).stop 2024-03-11T19:08:33.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.768+0000 7f2a02d9d700 1 -- 172.21.15.29:0/2584398269 reap_dead start 2024-03-11T19:08:33.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.768+0000 7f2a08ee6700 1 -- 172.21.15.29:0/2584398269 shutdown_connections 2024-03-11T19:08:33.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.768+0000 7f2a08ee6700 1 -- 172.21.15.29:0/2584398269 >> 172.21.15.29:0/2584398269 conn(0x7f2a04078d00 msgr2=0x7f2a041116f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:33.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.768+0000 7f2a08ee6700 1 -- 172.21.15.29:0/2584398269 shutdown_connections 2024-03-11T19:08:33.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:33.768+0000 7f2a08ee6700 1 -- 172.21.15.29:0/2584398269 wait complete. 2024-03-11T19:08:34.377 INFO:teuthology.orchestra.run.smithi029.stdout:206158430256 2024-03-11T19:08:34.377 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-11T19:08:35.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:34 smithi029 ceph-mon[32439]: pgmap v312: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-11T19:08:35.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:34 smithi190 ceph-mon[36292]: pgmap v312: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-11T19:08:35.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:34 smithi170 ceph-mon[35690]: pgmap v312: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-11T19:08:35.780 INFO:teuthology.orchestra.run.smithi029.stdout:352187318294 2024-03-11T19:08:35.781 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-11T19:08:36.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.270+0000 7f562d66d700 1 -- 172.21.15.29:0/3698212334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5628101d20 msgr2=0x7f56281021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:36.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.270+0000 7f562d66d700 1 --2- 172.21.15.29:0/3698212334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5628101d20 0x7f56281021a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f561c009990 tx=0x7f561c009ca0 comp rx=0 tx=0).stop 2024-03-11T19:08:36.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.270+0000 7f562d66d700 1 -- 172.21.15.29:0/3698212334 shutdown_connections 2024-03-11T19:08:36.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.270+0000 7f562d66d700 1 --2- 172.21.15.29:0/3698212334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56281026e0 0x7f562810ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:36.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.270+0000 7f562d66d700 1 --2- 172.21.15.29:0/3698212334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5628107d90 0x7f5628108170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:36.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.270+0000 7f562d66d700 1 --2- 172.21.15.29:0/3698212334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5628101d20 0x7f56281021a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:36.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.270+0000 7f562d66d700 1 -- 172.21.15.29:0/3698212334 >> 172.21.15.29:0/3698212334 conn(0x7f5628078d40 msgr2=0x7f5628079150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:36.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.270+0000 7f562d66d700 1 -- 172.21.15.29:0/3698212334 shutdown_connections 2024-03-11T19:08:36.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.270+0000 7f562d66d700 1 -- 172.21.15.29:0/3698212334 wait complete. 2024-03-11T19:08:36.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.271+0000 7f562d66d700 1 Processor -- start 2024-03-11T19:08:36.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.271+0000 7f562d66d700 1 -- start start 2024-03-11T19:08:36.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.271+0000 7f562d66d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5628101d20 0x7f5628198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:36.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.271+0000 7f562d66d700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56281026e0 0x7f5628198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:36.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.271+0000 7f562d66d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5628107d90 0x7f562819c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:36.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.271+0000 7f562d66d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562819cfa0 con 0x7f5628101d20 2024-03-11T19:08:36.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.271+0000 7f562d66d700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562819d110 con 0x7f56281026e0 2024-03-11T19:08:36.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.271+0000 7f562d66d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562819d2d0 con 0x7f5628107d90 2024-03-11T19:08:36.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.271+0000 7f5626ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5628101d20 0x7f5628198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:36.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.271+0000 7f5626ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5628101d20 0x7f5628198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33520/0 (socket says 172.21.15.29:33520) 2024-03-11T19:08:36.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.271+0000 7f5626ffd700 1 -- 172.21.15.29:0/4007327530 learned_addr learned my addr 172.21.15.29:0/4007327530 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:36.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.271+0000 7f56267fc700 1 --2- 172.21.15.29:0/4007327530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56281026e0 0x7f5628198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:36.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.271+0000 7f56277fe700 1 --2- 172.21.15.29:0/4007327530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5628107d90 0x7f562819c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:36.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.272+0000 7f5626ffd700 1 -- 172.21.15.29:0/4007327530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5628107d90 msgr2=0x7f562819c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:36.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.272+0000 7f5626ffd700 1 --2- 172.21.15.29:0/4007327530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5628107d90 0x7f562819c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:36.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.272+0000 7f5626ffd700 1 -- 172.21.15.29:0/4007327530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56281026e0 msgr2=0x7f5628198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:36.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.272+0000 7f5626ffd700 1 --2- 172.21.15.29:0/4007327530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56281026e0 0x7f5628198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:36.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.272+0000 7f5626ffd700 1 -- 172.21.15.29:0/4007327530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561c0095f0 con 0x7f5628101d20 2024-03-11T19:08:36.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.272+0000 7f5626ffd700 1 --2- 172.21.15.29:0/4007327530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5628101d20 0x7f5628198110 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f561800c920 tx=0x7f561800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:36.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.272+0000 7f560ffff700 1 -- 172.21.15.29:0/4007327530 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5618007b00 con 0x7f5628101d20 2024-03-11T19:08:36.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.272+0000 7f560ffff700 1 -- 172.21.15.29:0/4007327530 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f561800ce70 con 0x7f5628101d20 2024-03-11T19:08:36.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.272+0000 7f562d66d700 1 -- 172.21.15.29:0/4007327530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562819d5b0 con 0x7f5628101d20 2024-03-11T19:08:36.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.272+0000 7f560ffff700 1 -- 172.21.15.29:0/4007327530 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56180054e0 con 0x7f5628101d20 2024-03-11T19:08:36.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.272+0000 7f562d66d700 1 -- 172.21.15.29:0/4007327530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56281a4e10 con 0x7f5628101d20 2024-03-11T19:08:36.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.273+0000 7f562d66d700 1 -- 172.21.15.29:0/4007327530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5628111eb0 con 0x7f5628101d20 2024-03-11T19:08:36.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.274+0000 7f560ffff700 1 -- 172.21.15.29:0/4007327530 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f5618004750 con 0x7f5628101d20 2024-03-11T19:08:36.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.275+0000 7f560ffff700 1 --2- 172.21.15.29:0/4007327530 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f56100a1bd0 0x7f56100a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:36.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.275+0000 7f56267fc700 1 --2- 172.21.15.29:0/4007327530 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f56100a1bd0 0x7f56100a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:36.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.275+0000 7f560ffff700 1 -- 172.21.15.29:0/4007327530 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f56180cd2a0 con 0x7f5628101d20 2024-03-11T19:08:36.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.275+0000 7f560ffff700 1 --2- 172.21.15.29:0/4007327530 >> [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080] conn(0x7f56100ab3b0 0x7f56100ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:36.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.275+0000 7f560ffff700 1 -- 172.21.15.29:0/4007327530 --> [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f56100ade80 con 0x7f56100ab3b0 2024-03-11T19:08:36.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.275+0000 7f560ffff700 1 -- 172.21.15.29:0/4007327530 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56180cd740 con 0x7f5628101d20 2024-03-11T19:08:36.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.275+0000 7f56277fe700 1 --2- 172.21.15.29:0/4007327530 >> [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080] conn(0x7f56100ab3b0 0x7f56100ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:36.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.275+0000 7f56267fc700 1 --2- 172.21.15.29:0/4007327530 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f56100a1bd0 0x7f56100a4090 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f561c00b5c0 tx=0x7f561c005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:36.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.275+0000 7f56277fe700 1 --2- 172.21.15.29:0/4007327530 >> [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080] conn(0x7f56100ab3b0 0x7f56100ad7d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:36.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.276+0000 7f560ffff700 1 -- 172.21.15.29:0/4007327530 <== osd.9 v2:172.21.15.29:6826/3553345080 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f56100ade80 con 0x7f56100ab3b0 2024-03-11T19:08:36.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.292+0000 7f562d66d700 1 -- 172.21.15.29:0/4007327530 --> [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5628062380 con 0x7f56100ab3b0 2024-03-11T19:08:36.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.293+0000 7f560ffff700 1 -- 172.21.15.29:0/4007327530 <== osd.9 v2:172.21.15.29:6826/3553345080 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5628062380 con 0x7f56100ab3b0 2024-03-11T19:08:36.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.293+0000 7f562d66d700 1 -- 172.21.15.29:0/4007327530 >> [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080] conn(0x7f56100ab3b0 msgr2=0x7f56100ad7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:36.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.293+0000 7f562d66d700 1 --2- 172.21.15.29:0/4007327530 >> [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080] conn(0x7f56100ab3b0 0x7f56100ad7d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:36.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.293+0000 7f562d66d700 1 -- 172.21.15.29:0/4007327530 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f56100a1bd0 msgr2=0x7f56100a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:36.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.293+0000 7f562d66d700 1 --2- 172.21.15.29:0/4007327530 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f56100a1bd0 0x7f56100a4090 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f561c00b5c0 tx=0x7f561c005960 comp rx=0 tx=0).stop 2024-03-11T19:08:36.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.293+0000 7f562d66d700 1 -- 172.21.15.29:0/4007327530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5628101d20 msgr2=0x7f5628198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:36.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.293+0000 7f562d66d700 1 --2- 172.21.15.29:0/4007327530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5628101d20 0x7f5628198110 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f561800c920 tx=0x7f561800cc30 comp rx=0 tx=0).stop 2024-03-11T19:08:36.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.293+0000 7f56277fe700 1 -- 172.21.15.29:0/4007327530 reap_dead start 2024-03-11T19:08:36.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.293+0000 7f562d66d700 1 -- 172.21.15.29:0/4007327530 shutdown_connections 2024-03-11T19:08:36.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.293+0000 7f562d66d700 1 -- 172.21.15.29:0/4007327530 >> 172.21.15.29:0/4007327530 conn(0x7f5628078d40 msgr2=0x7f5628100ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:36.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.293+0000 7f562d66d700 1 -- 172.21.15.29:0/4007327530 shutdown_connections 2024-03-11T19:08:36.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.293+0000 7f562d66d700 1 -- 172.21.15.29:0/4007327530 wait complete. 2024-03-11T19:08:36.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.524+0000 7f7776d29700 1 -- 172.21.15.29:0/946104187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7770069920 msgr2=0x7f7770069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:36.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.524+0000 7f7776d29700 1 --2- 172.21.15.29:0/946104187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7770069920 0x7f7770069d80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f77640099c0 tx=0x7f7764009cd0 comp rx=0 tx=0).stop 2024-03-11T19:08:36.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.524+0000 7f7776d29700 1 -- 172.21.15.29:0/946104187 shutdown_connections 2024-03-11T19:08:36.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.524+0000 7f7776d29700 1 --2- 172.21.15.29:0/946104187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f777006a2c0 0x7f777010eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:36.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.524+0000 7f7776d29700 1 --2- 172.21.15.29:0/946104187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7770107cc0 0x7f77701080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:36.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.524+0000 7f7776d29700 1 --2- 172.21.15.29:0/946104187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7770069920 0x7f7770069d80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:36.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.524+0000 7f7776d29700 1 -- 172.21.15.29:0/946104187 >> 172.21.15.29:0/946104187 conn(0x7f7770077aa0 msgr2=0x7f7770077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:36.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.524+0000 7f7776d29700 1 -- 172.21.15.29:0/946104187 shutdown_connections 2024-03-11T19:08:36.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.524+0000 7f7776d29700 1 -- 172.21.15.29:0/946104187 wait complete. 2024-03-11T19:08:36.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.524+0000 7f7776d29700 1 Processor -- start 2024-03-11T19:08:36.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.525+0000 7f7776d29700 1 -- start start 2024-03-11T19:08:36.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.525+0000 7f7776d29700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7770069920 0x7f7770100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:36.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.525+0000 7f7776d29700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f777006a2c0 0x7f7770101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:36.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.525+0000 7f7776d29700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7770107cc0 0x7f77701016c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:36.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.525+0000 7f7776d29700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77701a64c0 con 0x7f777006a2c0 2024-03-11T19:08:36.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.525+0000 7f7776d29700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77701a6630 con 0x7f7770107cc0 2024-03-11T19:08:36.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.525+0000 7f7776d29700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77701a67f0 con 0x7f7770069920 2024-03-11T19:08:36.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.525+0000 7f776ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f777006a2c0 0x7f7770101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:36.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.525+0000 7f776ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f777006a2c0 0x7f7770101180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33528/0 (socket says 172.21.15.29:33528) 2024-03-11T19:08:36.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.525+0000 7f776ffff700 1 -- 172.21.15.29:0/2473121061 learned_addr learned my addr 172.21.15.29:0/2473121061 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:36.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.525+0000 7f7774ac5700 1 --2- 172.21.15.29:0/2473121061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7770069920 0x7f7770100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:36.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.525+0000 7f77752c6700 1 --2- 172.21.15.29:0/2473121061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7770107cc0 0x7f77701016c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:36.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.525+0000 7f776ffff700 1 -- 172.21.15.29:0/2473121061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7770069920 msgr2=0x7f7770100c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:36.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.525+0000 7f776ffff700 1 --2- 172.21.15.29:0/2473121061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7770069920 0x7f7770100c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:36.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.525+0000 7f776ffff700 1 -- 172.21.15.29:0/2473121061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7770107cc0 msgr2=0x7f77701016c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:36.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.525+0000 7f776ffff700 1 --2- 172.21.15.29:0/2473121061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7770107cc0 0x7f77701016c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:36.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.525+0000 7f776ffff700 1 -- 172.21.15.29:0/2473121061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77640095f0 con 0x7f777006a2c0 2024-03-11T19:08:36.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.525+0000 7f776ffff700 1 --2- 172.21.15.29:0/2473121061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f777006a2c0 0x7f7770101180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7764005910 tx=0x7f77640039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:36.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.525+0000 7f776dffb700 1 -- 172.21.15.29:0/2473121061 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776401d070 con 0x7f777006a2c0 2024-03-11T19:08:36.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.526+0000 7f776dffb700 1 -- 172.21.15.29:0/2473121061 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7764003e30 con 0x7f777006a2c0 2024-03-11T19:08:36.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.526+0000 7f7776d29700 1 -- 172.21.15.29:0/2473121061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77701a6a70 con 0x7f777006a2c0 2024-03-11T19:08:36.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.526+0000 7f776dffb700 1 -- 172.21.15.29:0/2473121061 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776400faf0 con 0x7f777006a2c0 2024-03-11T19:08:36.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.526+0000 7f7776d29700 1 -- 172.21.15.29:0/2473121061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77701a6ee0 con 0x7f777006a2c0 2024-03-11T19:08:36.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.526+0000 7f7776d29700 1 -- 172.21.15.29:0/2473121061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7770111e20 con 0x7f777006a2c0 2024-03-11T19:08:36.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.528+0000 7f776dffb700 1 -- 172.21.15.29:0/2473121061 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f776400b730 con 0x7f777006a2c0 2024-03-11T19:08:36.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.528+0000 7f776dffb700 1 --2- 172.21.15.29:0/2473121061 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f77580a1bd0 0x7f77580a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:36.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.528+0000 7f7774ac5700 1 --2- 172.21.15.29:0/2473121061 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f77580a1bd0 0x7f77580a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:36.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.528+0000 7f776dffb700 1 -- 172.21.15.29:0/2473121061 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f77640ce870 con 0x7f777006a2c0 2024-03-11T19:08:36.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.529+0000 7f776dffb700 1 --2- 172.21.15.29:0/2473121061 >> [v2:172.21.15.190:6816/1474709011,v1:172.21.15.190:6817/1474709011] conn(0x7f77580ab3b0 0x7f77580ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:36.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.529+0000 7f776dffb700 1 -- 172.21.15.29:0/2473121061 --> [v2:172.21.15.190:6816/1474709011,v1:172.21.15.190:6817/1474709011] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77580ade80 con 0x7f77580ab3b0 2024-03-11T19:08:36.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.529+0000 7f776dffb700 1 -- 172.21.15.29:0/2473121061 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77640ced10 con 0x7f777006a2c0 2024-03-11T19:08:36.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.529+0000 7f7774ac5700 1 --2- 172.21.15.29:0/2473121061 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f77580a1bd0 0x7f77580a4090 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7760007b60 tx=0x7f77600073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:36.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.529+0000 7f77752c6700 1 --2- 172.21.15.29:0/2473121061 >> [v2:172.21.15.190:6816/1474709011,v1:172.21.15.190:6817/1474709011] conn(0x7f77580ab3b0 0x7f77580ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:36.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.529+0000 7f77752c6700 1 --2- 172.21.15.29:0/2473121061 >> [v2:172.21.15.190:6816/1474709011,v1:172.21.15.190:6817/1474709011] conn(0x7f77580ab3b0 0x7f77580ad7d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:36.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.530+0000 7f776dffb700 1 -- 172.21.15.29:0/2473121061 <== osd.8 v2:172.21.15.190:6816/1474709011 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f77580ade80 con 0x7f77580ab3b0 2024-03-11T19:08:36.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.547+0000 7f7776d29700 1 -- 172.21.15.29:0/2473121061 --> [v2:172.21.15.190:6816/1474709011,v1:172.21.15.190:6817/1474709011] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7770062300 con 0x7f77580ab3b0 2024-03-11T19:08:36.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.547+0000 7f776dffb700 1 -- 172.21.15.29:0/2473121061 <== osd.8 v2:172.21.15.190:6816/1474709011 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7770062300 con 0x7f77580ab3b0 2024-03-11T19:08:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.547+0000 7f7776d29700 1 -- 172.21.15.29:0/2473121061 >> [v2:172.21.15.190:6816/1474709011,v1:172.21.15.190:6817/1474709011] conn(0x7f77580ab3b0 msgr2=0x7f77580ad7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.547+0000 7f7776d29700 1 --2- 172.21.15.29:0/2473121061 >> [v2:172.21.15.190:6816/1474709011,v1:172.21.15.190:6817/1474709011] conn(0x7f77580ab3b0 0x7f77580ad7d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.547+0000 7f7776d29700 1 -- 172.21.15.29:0/2473121061 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f77580a1bd0 msgr2=0x7f77580a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.547+0000 7f7776d29700 1 --2- 172.21.15.29:0/2473121061 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f77580a1bd0 0x7f77580a4090 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7760007b60 tx=0x7f77600073d0 comp rx=0 tx=0).stop 2024-03-11T19:08:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.547+0000 7f7776d29700 1 -- 172.21.15.29:0/2473121061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f777006a2c0 msgr2=0x7f7770101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.547+0000 7f7776d29700 1 --2- 172.21.15.29:0/2473121061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f777006a2c0 0x7f7770101180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7764005910 tx=0x7f77640039f0 comp rx=0 tx=0).stop 2024-03-11T19:08:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.547+0000 7f77752c6700 1 -- 172.21.15.29:0/2473121061 reap_dead start 2024-03-11T19:08:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.547+0000 7f7776d29700 1 -- 172.21.15.29:0/2473121061 shutdown_connections 2024-03-11T19:08:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.548+0000 7f7776d29700 1 -- 172.21.15.29:0/2473121061 >> 172.21.15.29:0/2473121061 conn(0x7f7770077aa0 msgr2=0x7f77700ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.548+0000 7f7776d29700 1 -- 172.21.15.29:0/2473121061 shutdown_connections 2024-03-11T19:08:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:36.548+0000 7f7776d29700 1 -- 172.21.15.29:0/2473121061 wait complete. 2024-03-11T19:08:37.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:36 smithi029 ceph-mon[32439]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-11T19:08:37.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:36 smithi190 ceph-mon[36292]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-11T19:08:37.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:36 smithi170 ceph-mon[35690]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-11T19:08:37.511 INFO:teuthology.orchestra.run.smithi029.stdout:296352743457 2024-03-11T19:08:37.511 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-11T19:08:38.109 INFO:teuthology.orchestra.run.smithi029.stdout:261993005095 2024-03-11T19:08:38.110 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-11T19:08:38.599 INFO:teuthology.orchestra.run.smithi029.stdout:317827579929 2024-03-11T19:08:38.600 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-11T19:08:39.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:38 smithi029 ceph-mon[32439]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-11T19:08:39.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:38 smithi190 ceph-mon[36292]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-11T19:08:39.275 INFO:teuthology.orchestra.run.smithi029.stdout:55834574920 2024-03-11T19:08:39.275 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T19:08:39.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:38 smithi170 ceph-mon[35690]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-11T19:08:39.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.528+0000 7fac99506700 1 -- 172.21.15.29:0/2541869633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94100260 msgr2=0x7fac941006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:39.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.528+0000 7fac99506700 1 --2- 172.21.15.29:0/2541869633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94100260 0x7fac941006e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fac88009990 tx=0x7fac88009ca0 comp rx=0 tx=0).stop 2024-03-11T19:08:39.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.528+0000 7fac99506700 1 -- 172.21.15.29:0/2541869633 shutdown_connections 2024-03-11T19:08:39.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.528+0000 7fac99506700 1 --2- 172.21.15.29:0/2541869633 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac94100c20 0x7fac9410eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:39.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.528+0000 7fac99506700 1 --2- 172.21.15.29:0/2541869633 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac94106ce0 0x7fac941070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:39.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.528+0000 7fac99506700 1 --2- 172.21.15.29:0/2541869633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94100260 0x7fac941006e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:39.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.528+0000 7fac99506700 1 -- 172.21.15.29:0/2541869633 >> 172.21.15.29:0/2541869633 conn(0x7fac94077170 msgr2=0x7fac940fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:39.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.528+0000 7fac99506700 1 -- 172.21.15.29:0/2541869633 shutdown_connections 2024-03-11T19:08:39.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.528+0000 7fac99506700 1 -- 172.21.15.29:0/2541869633 wait complete. 2024-03-11T19:08:39.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.529+0000 7fac99506700 1 Processor -- start 2024-03-11T19:08:39.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.529+0000 7fac99506700 1 -- start start 2024-03-11T19:08:39.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.529+0000 7fac99506700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac94100260 0x7fac9419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:39.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.529+0000 7fac99506700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94100c20 0x7fac9419a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:39.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.529+0000 7fac99506700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac94106ce0 0x7fac9419ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:39.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.529+0000 7fac99506700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9419f300 con 0x7fac94100c20 2024-03-11T19:08:39.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.529+0000 7fac99506700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9419f470 con 0x7fac94106ce0 2024-03-11T19:08:39.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.529+0000 7fac99506700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9419f630 con 0x7fac94100260 2024-03-11T19:08:39.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.529+0000 7fac927fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94100c20 0x7fac9419a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:39.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.529+0000 7fac927fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94100c20 0x7fac9419a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33542/0 (socket says 172.21.15.29:33542) 2024-03-11T19:08:39.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.529+0000 7fac927fc700 1 -- 172.21.15.29:0/3670215607 learned_addr learned my addr 172.21.15.29:0/3670215607 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:39.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.529+0000 7fac92ffd700 1 --2- 172.21.15.29:0/3670215607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac94100260 0x7fac9419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:39.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.529+0000 7fac937fe700 1 --2- 172.21.15.29:0/3670215607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac94106ce0 0x7fac9419ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:39.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.529+0000 7fac927fc700 1 -- 172.21.15.29:0/3670215607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac94100260 msgr2=0x7fac9419a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:39.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.529+0000 7fac927fc700 1 --2- 172.21.15.29:0/3670215607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac94100260 0x7fac9419a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:39.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.529+0000 7fac927fc700 1 -- 172.21.15.29:0/3670215607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac94106ce0 msgr2=0x7fac9419ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:39.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.529+0000 7fac927fc700 1 --2- 172.21.15.29:0/3670215607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac94106ce0 0x7fac9419ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:39.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.529+0000 7fac927fc700 1 -- 172.21.15.29:0/3670215607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac880095f0 con 0x7fac94100c20 2024-03-11T19:08:39.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.530+0000 7fac937fe700 1 --2- 172.21.15.29:0/3670215607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac94106ce0 0x7fac9419ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:08:39.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.530+0000 7fac927fc700 1 --2- 172.21.15.29:0/3670215607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94100c20 0x7fac9419a960 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fac88000c00 tx=0x7fac880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:39.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.530+0000 7fac7bfff700 1 -- 172.21.15.29:0/3670215607 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8801d070 con 0x7fac94100c20 2024-03-11T19:08:39.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.530+0000 7fac7bfff700 1 -- 172.21.15.29:0/3670215607 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fac8800fc20 con 0x7fac94100c20 2024-03-11T19:08:39.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.530+0000 7fac99506700 1 -- 172.21.15.29:0/3670215607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9419f8b0 con 0x7fac94100c20 2024-03-11T19:08:39.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.530+0000 7fac7bfff700 1 -- 172.21.15.29:0/3670215607 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac88022d00 con 0x7fac94100c20 2024-03-11T19:08:39.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.530+0000 7fac99506700 1 -- 172.21.15.29:0/3670215607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac941a7170 con 0x7fac94100c20 2024-03-11T19:08:39.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.531+0000 7fac99506700 1 -- 172.21.15.29:0/3670215607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fac94111ff0 con 0x7fac94100c20 2024-03-11T19:08:39.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.532+0000 7fac7bfff700 1 -- 172.21.15.29:0/3670215607 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fac8800f770 con 0x7fac94100c20 2024-03-11T19:08:39.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.533+0000 7fac7bfff700 1 --2- 172.21.15.29:0/3670215607 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fac7c0a1b50 0x7fac7c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:39.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.533+0000 7fac92ffd700 1 --2- 172.21.15.29:0/3670215607 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fac7c0a1b50 0x7fac7c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:39.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.533+0000 7fac7bfff700 1 -- 172.21.15.29:0/3670215607 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fac880cefe0 con 0x7fac94100c20 2024-03-11T19:08:39.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.533+0000 7fac7bfff700 1 --2- 172.21.15.29:0/3670215607 >> [v2:172.21.15.170:6816/3259207349,v1:172.21.15.170:6817/3259207349] conn(0x7fac7c0ab330 0x7fac7c0ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:39.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.533+0000 7fac7bfff700 1 -- 172.21.15.29:0/3670215607 --> [v2:172.21.15.170:6816/3259207349,v1:172.21.15.170:6817/3259207349] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fac7c0ade00 con 0x7fac7c0ab330 2024-03-11T19:08:39.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.533+0000 7fac7bfff700 1 -- 172.21.15.29:0/3670215607 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac880cf480 con 0x7fac94100c20 2024-03-11T19:08:39.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.534+0000 7fac92ffd700 1 --2- 172.21.15.29:0/3670215607 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fac7c0a1b50 0x7fac7c0a4010 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fac84005fd0 tx=0x7fac8400af20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:39.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.534+0000 7fac937fe700 1 --2- 172.21.15.29:0/3670215607 >> [v2:172.21.15.170:6816/3259207349,v1:172.21.15.170:6817/3259207349] conn(0x7fac7c0ab330 0x7fac7c0ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:39.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.534+0000 7fac937fe700 1 --2- 172.21.15.29:0/3670215607 >> [v2:172.21.15.170:6816/3259207349,v1:172.21.15.170:6817/3259207349] conn(0x7fac7c0ab330 0x7fac7c0ad750 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:39.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.535+0000 7fac7bfff700 1 -- 172.21.15.29:0/3670215607 <== osd.7 v2:172.21.15.170:6816/3259207349 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fac7c0ade00 con 0x7fac7c0ab330 2024-03-11T19:08:39.546 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:39.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.568+0000 7fac99506700 1 -- 172.21.15.29:0/3670215607 --> [v2:172.21.15.170:6816/3259207349,v1:172.21.15.170:6817/3259207349] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fac9410e1a0 con 0x7fac7c0ab330 2024-03-11T19:08:39.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.569+0000 7fac7bfff700 1 -- 172.21.15.29:0/3670215607 <== osd.7 v2:172.21.15.170:6816/3259207349 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fac9410e1a0 con 0x7fac7c0ab330 2024-03-11T19:08:39.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.570+0000 7fac79ffb700 1 -- 172.21.15.29:0/3670215607 >> [v2:172.21.15.170:6816/3259207349,v1:172.21.15.170:6817/3259207349] conn(0x7fac7c0ab330 msgr2=0x7fac7c0ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:39.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.570+0000 7fac79ffb700 1 --2- 172.21.15.29:0/3670215607 >> [v2:172.21.15.170:6816/3259207349,v1:172.21.15.170:6817/3259207349] conn(0x7fac7c0ab330 0x7fac7c0ad750 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:39.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.571+0000 7fac79ffb700 1 -- 172.21.15.29:0/3670215607 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fac7c0a1b50 msgr2=0x7fac7c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:39.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.571+0000 7fac79ffb700 1 --2- 172.21.15.29:0/3670215607 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fac7c0a1b50 0x7fac7c0a4010 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fac84005fd0 tx=0x7fac8400af20 comp rx=0 tx=0).stop 2024-03-11T19:08:39.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.571+0000 7fac79ffb700 1 -- 172.21.15.29:0/3670215607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94100c20 msgr2=0x7fac9419a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:39.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.571+0000 7fac79ffb700 1 --2- 172.21.15.29:0/3670215607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94100c20 0x7fac9419a960 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fac88000c00 tx=0x7fac880057f0 comp rx=0 tx=0).stop 2024-03-11T19:08:39.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.571+0000 7fac937fe700 1 -- 172.21.15.29:0/3670215607 reap_dead start 2024-03-11T19:08:39.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.571+0000 7fac79ffb700 1 -- 172.21.15.29:0/3670215607 shutdown_connections 2024-03-11T19:08:39.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.571+0000 7fac79ffb700 1 -- 172.21.15.29:0/3670215607 >> 172.21.15.29:0/3670215607 conn(0x7fac94077170 msgr2=0x7fac94104f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:39.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.574+0000 7fac79ffb700 1 -- 172.21.15.29:0/3670215607 shutdown_connections 2024-03-11T19:08:39.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:39.574+0000 7fac79ffb700 1 -- 172.21.15.29:0/3670215607 wait complete. 2024-03-11T19:08:40.354 INFO:teuthology.orchestra.run.smithi029.stdout:227633266732 2024-03-11T19:08:40.355 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-11T19:08:40.463 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:40.464 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:40.464 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:40.466 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:40.467 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:40.469 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:40.481 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:40 smithi029 ceph-mon[32439]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-11T19:08:41.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:40 smithi190 ceph-mon[36292]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-11T19:08:41.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:40 smithi170 ceph-mon[35690]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-11T19:08:43.154 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:42 smithi029 ceph-mon[32439]: pgmap v316: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:08:43.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:42 smithi190 ceph-mon[36292]: pgmap v316: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:08:43.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:42 smithi170 ceph-mon[35690]: pgmap v316: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:08:44.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-11T19:08:44.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-11T19:08:44.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-03-11T19:08:44.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-03-11T19:08:44.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-11T19:08:44.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-03-11T19:08:44.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:43 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-11T19:08:44.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:43 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-11T19:08:44.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:43 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-03-11T19:08:44.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:43 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-03-11T19:08:44.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:43 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-11T19:08:44.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:43 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-03-11T19:08:44.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:43 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-11T19:08:44.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:43 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-11T19:08:44.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:43 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-03-11T19:08:44.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:43 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-03-11T19:08:44.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:43 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-11T19:08:44.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:43 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-03-11T19:08:45.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:44 smithi190 ceph-mon[36292]: pgmap v317: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-11T19:08:45.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-11T19:08:45.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-11T19:08:45.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-03-11T19:08:45.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-03-11T19:08:45.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-11T19:08:45.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-03-11T19:08:45.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:44 smithi190 ceph-mon[36292]: osdmap e88: 12 total, 12 up, 12 in 2024-03-11T19:08:45.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:44 smithi170 ceph-mon[35690]: pgmap v317: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-11T19:08:45.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:44 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-11T19:08:45.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:44 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-11T19:08:45.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:44 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-03-11T19:08:45.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:44 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-03-11T19:08:45.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:44 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-11T19:08:45.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:44 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-03-11T19:08:45.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:44 smithi170 ceph-mon[35690]: osdmap e88: 12 total, 12 up, 12 in 2024-03-11T19:08:45.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:44 smithi029 ceph-mon[32439]: pgmap v317: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-11T19:08:45.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-11T19:08:45.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-11T19:08:45.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-03-11T19:08:45.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-03-11T19:08:45.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-11T19:08:45.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-03-11T19:08:45.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:44 smithi029 ceph-mon[32439]: osdmap e88: 12 total, 12 up, 12 in 2024-03-11T19:08:46.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:45 smithi190 ceph-mon[36292]: osdmap e89: 12 total, 12 up, 12 in 2024-03-11T19:08:46.266 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:45 smithi029 ceph-mon[32439]: osdmap e89: 12 total, 12 up, 12 in 2024-03-11T19:08:46.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:45 smithi170 ceph-mon[35690]: osdmap e89: 12 total, 12 up, 12 in 2024-03-11T19:08:46.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.474+0000 7f31cf863700 1 -- 172.21.15.29:0/269378007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31c81015d0 msgr2=0x7f31c81019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:46.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.474+0000 7f31cf863700 1 --2- 172.21.15.29:0/269378007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31c81015d0 0x7f31c81019b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f31c4009990 tx=0x7f31c4009ca0 comp rx=0 tx=0).stop 2024-03-11T19:08:46.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.474+0000 7f31cf863700 1 -- 172.21.15.29:0/269378007 shutdown_connections 2024-03-11T19:08:46.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.474+0000 7f31cf863700 1 --2- 172.21.15.29:0/269378007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31c8105ef0 0x7f31c810dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:46.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.474+0000 7f31cf863700 1 --2- 172.21.15.29:0/269378007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31c81015d0 0x7f31c81019b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:46.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.474+0000 7f31cf863700 1 --2- 172.21.15.29:0/269378007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31c8101f80 0x7f31c8105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:46.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.474+0000 7f31cf863700 1 -- 172.21.15.29:0/269378007 >> 172.21.15.29:0/269378007 conn(0x7f31c8077170 msgr2=0x7f31c80fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:46.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.474+0000 7f31cf863700 1 -- 172.21.15.29:0/269378007 shutdown_connections 2024-03-11T19:08:46.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.474+0000 7f31cf863700 1 -- 172.21.15.29:0/269378007 wait complete. 2024-03-11T19:08:46.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.475+0000 7f31cf863700 1 Processor -- start 2024-03-11T19:08:46.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.475+0000 7f31cf863700 1 -- start start 2024-03-11T19:08:46.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.475+0000 7f31cf863700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31c81015d0 0x7f31c8198fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:46.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.475+0000 7f31cf863700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31c8101f80 0x7f31c8199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:46.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.475+0000 7f31cf863700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31c8105ef0 0x7f31c81a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:46.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.475+0000 7f31cf863700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c8199dd0 con 0x7f31c81015d0 2024-03-11T19:08:46.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.475+0000 7f31cf863700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c81a4590 con 0x7f31c8101f80 2024-03-11T19:08:46.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.475+0000 7f31cf863700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c81a46d0 con 0x7f31c8105ef0 2024-03-11T19:08:46.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.475+0000 7f31cd5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31c81015d0 0x7f31c8198fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:46.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.476+0000 7f31cd5ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31c81015d0 0x7f31c8198fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57620/0 (socket says 172.21.15.29:57620) 2024-03-11T19:08:46.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.476+0000 7f31cd5ff700 1 -- 172.21.15.29:0/558947630 learned_addr learned my addr 172.21.15.29:0/558947630 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:46.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.476+0000 7f31cde00700 1 --2- 172.21.15.29:0/558947630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31c8105ef0 0x7f31c81a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:46.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.476+0000 7f31ccdfe700 1 --2- 172.21.15.29:0/558947630 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31c8101f80 0x7f31c8199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:46.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.476+0000 7f31cd5ff700 1 -- 172.21.15.29:0/558947630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31c8105ef0 msgr2=0x7f31c81a4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:46.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.476+0000 7f31cd5ff700 1 --2- 172.21.15.29:0/558947630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31c8105ef0 0x7f31c81a4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:46.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.476+0000 7f31cd5ff700 1 -- 172.21.15.29:0/558947630 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31c8101f80 msgr2=0x7f31c8199510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:46.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.476+0000 7f31cd5ff700 1 --2- 172.21.15.29:0/558947630 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31c8101f80 0x7f31c8199510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:46.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.476+0000 7f31cd5ff700 1 -- 172.21.15.29:0/558947630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c40095f0 con 0x7f31c81015d0 2024-03-11T19:08:46.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.476+0000 7f31ccdfe700 1 --2- 172.21.15.29:0/558947630 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31c8101f80 0x7f31c8199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:08:46.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.476+0000 7f31cd5ff700 1 --2- 172.21.15.29:0/558947630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31c81015d0 0x7f31c8198fd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f31c400f6c0 tx=0x7f31c400f7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:46.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.476+0000 7f31be7fc700 1 -- 172.21.15.29:0/558947630 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c401d070 con 0x7f31c81015d0 2024-03-11T19:08:46.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.476+0000 7f31be7fc700 1 -- 172.21.15.29:0/558947630 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f31c4005a00 con 0x7f31c81015d0 2024-03-11T19:08:46.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.476+0000 7f31be7fc700 1 -- 172.21.15.29:0/558947630 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c4022b80 con 0x7f31c81015d0 2024-03-11T19:08:46.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.476+0000 7f31cf863700 1 -- 172.21.15.29:0/558947630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c81a4950 con 0x7f31c81015d0 2024-03-11T19:08:46.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.476+0000 7f31cf863700 1 -- 172.21.15.29:0/558947630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c81a4da0 con 0x7f31c81015d0 2024-03-11T19:08:46.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.478+0000 7f31cf863700 1 -- 172.21.15.29:0/558947630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c81091b0 con 0x7f31c81015d0 2024-03-11T19:08:46.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.480+0000 7f31be7fc700 1 -- 172.21.15.29:0/558947630 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f31c4022470 con 0x7f31c81015d0 2024-03-11T19:08:46.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.480+0000 7f31be7fc700 1 --2- 172.21.15.29:0/558947630 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f31b40a1b50 0x7f31b40a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:46.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.480+0000 7f31be7fc700 1 -- 172.21.15.29:0/558947630 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f31c40cf0f0 con 0x7f31c81015d0 2024-03-11T19:08:46.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.480+0000 7f31ccdfe700 1 --2- 172.21.15.29:0/558947630 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f31b40a1b50 0x7f31b40a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:46.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.481+0000 7f31ccdfe700 1 --2- 172.21.15.29:0/558947630 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f31b40a1b50 0x7f31b40a4010 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f31c81037b0 tx=0x7f31c0009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:46.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.484+0000 7f31be7fc700 1 -- 172.21.15.29:0/558947630 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f31c4098330 con 0x7f31c81015d0 2024-03-11T19:08:46.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.706+0000 7f31cf863700 1 -- 172.21.15.29:0/558947630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f31c8100870 con 0x7f31c81015d0 2024-03-11T19:08:46.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.707+0000 7f31be7fc700 1 -- 172.21.15.29:0/558947630 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f31c4027020 con 0x7f31c81015d0 2024-03-11T19:08:46.710 INFO:teuthology.orchestra.run.smithi029.stdout:146028888125 2024-03-11T19:08:46.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.709+0000 7f31cf863700 1 -- 172.21.15.29:0/558947630 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f31b40a1b50 msgr2=0x7f31b40a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:46.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.709+0000 7f31cf863700 1 --2- 172.21.15.29:0/558947630 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f31b40a1b50 0x7f31b40a4010 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f31c81037b0 tx=0x7f31c0009040 comp rx=0 tx=0).stop 2024-03-11T19:08:46.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.709+0000 7f31cf863700 1 -- 172.21.15.29:0/558947630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31c81015d0 msgr2=0x7f31c8198fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:46.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.709+0000 7f31cf863700 1 --2- 172.21.15.29:0/558947630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31c81015d0 0x7f31c8198fd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f31c400f6c0 tx=0x7f31c400f7a0 comp rx=0 tx=0).stop 2024-03-11T19:08:46.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.709+0000 7f31cf863700 1 -- 172.21.15.29:0/558947630 shutdown_connections 2024-03-11T19:08:46.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.709+0000 7f31cf863700 1 --2- 172.21.15.29:0/558947630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31c8105ef0 0x7f31c81a4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:46.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.709+0000 7f31cf863700 1 --2- 172.21.15.29:0/558947630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31c81015d0 0x7f31c8198fd0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:46.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.709+0000 7f31cf863700 1 --2- 172.21.15.29:0/558947630 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f31b40a1b50 0x7f31b40a4010 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:46.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.709+0000 7f31cf863700 1 --2- 172.21.15.29:0/558947630 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31c8101f80 0x7f31c8199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:46.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.709+0000 7f31cf863700 1 -- 172.21.15.29:0/558947630 >> 172.21.15.29:0/558947630 conn(0x7f31c8077170 msgr2=0x7f31c80fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:46.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.709+0000 7f31cf863700 1 -- 172.21.15.29:0/558947630 shutdown_connections 2024-03-11T19:08:46.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:46.709+0000 7f31cf863700 1 -- 172.21.15.29:0/558947630 wait complete. 2024-03-11T19:08:47.104 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:46 smithi029 ceph-mon[32439]: pgmap v320: 97 pgs: 3 peering, 94 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 9 op/s 2024-03-11T19:08:47.104 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:46 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/558947630' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T19:08:47.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:46 smithi190 ceph-mon[36292]: pgmap v320: 97 pgs: 3 peering, 94 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 9 op/s 2024-03-11T19:08:47.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:46 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/558947630' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T19:08:47.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:46 smithi170 ceph-mon[35690]: pgmap v320: 97 pgs: 3 peering, 94 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 9 op/s 2024-03-11T19:08:47.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:46 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/558947630' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T19:08:49.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:48 smithi029 ceph-mon[32439]: pgmap v321: 97 pgs: 3 peering, 94 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s 2024-03-11T19:08:49.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:48 smithi190 ceph-mon[36292]: pgmap v321: 97 pgs: 3 peering, 94 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s 2024-03-11T19:08:49.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:48 smithi170 ceph-mon[35690]: pgmap v321: 97 pgs: 3 peering, 94 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s 2024-03-11T19:08:49.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.966+0000 7f7d48c6b700 1 -- 172.21.15.29:0/2077354280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d441070b0 msgr2=0x7f7d4410ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:49.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.966+0000 7f7d48c6b700 1 --2- 172.21.15.29:0/2077354280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d441070b0 0x7f7d4410ec30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00b0d0 tx=0x7f7d3c00b3e0 comp rx=0 tx=0).stop 2024-03-11T19:08:49.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.966+0000 7f7d48c6b700 1 -- 172.21.15.29:0/2077354280 shutdown_connections 2024-03-11T19:08:49.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.966+0000 7f7d48c6b700 1 --2- 172.21.15.29:0/2077354280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d441070b0 0x7f7d4410ec30 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:49.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.966+0000 7f7d48c6b700 1 --2- 172.21.15.29:0/2077354280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d4406a610 0x7f7d4406a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:49.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.966+0000 7f7d48c6b700 1 --2- 172.21.15.29:0/2077354280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d4406afc0 0x7f7d44106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:49.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.966+0000 7f7d48c6b700 1 -- 172.21.15.29:0/2077354280 >> 172.21.15.29:0/2077354280 conn(0x7f7d440779d0 msgr2=0x7f7d44077de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:49.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.966+0000 7f7d48c6b700 1 -- 172.21.15.29:0/2077354280 shutdown_connections 2024-03-11T19:08:49.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.966+0000 7f7d48c6b700 1 -- 172.21.15.29:0/2077354280 wait complete. 2024-03-11T19:08:49.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.967+0000 7f7d48c6b700 1 Processor -- start 2024-03-11T19:08:49.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.967+0000 7f7d48c6b700 1 -- start start 2024-03-11T19:08:49.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.967+0000 7f7d48c6b700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d4406a610 0x7f7d441981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:49.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.967+0000 7f7d48c6b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d4406afc0 0x7f7d441986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:49.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.967+0000 7f7d48c6b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d441070b0 0x7f7d4419ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:49.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.967+0000 7f7d48c6b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4419d080 con 0x7f7d441070b0 2024-03-11T19:08:49.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.967+0000 7f7d48c6b700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4419d1f0 con 0x7f7d4406a610 2024-03-11T19:08:49.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.967+0000 7f7d48c6b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4419d3b0 con 0x7f7d4406afc0 2024-03-11T19:08:49.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.967+0000 7f7d42d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d441070b0 0x7f7d4419ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:49.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.967+0000 7f7d42d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d441070b0 0x7f7d4419ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57628/0 (socket says 172.21.15.29:57628) 2024-03-11T19:08:49.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.967+0000 7f7d42d9d700 1 -- 172.21.15.29:0/3786859824 learned_addr learned my addr 172.21.15.29:0/3786859824 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:49.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.967+0000 7f7d4259c700 1 --2- 172.21.15.29:0/3786859824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d4406a610 0x7f7d441981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:49.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.967+0000 7f7d41d9b700 1 --2- 172.21.15.29:0/3786859824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d4406afc0 0x7f7d441986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:49.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.968+0000 7f7d42d9d700 1 -- 172.21.15.29:0/3786859824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d4406afc0 msgr2=0x7f7d441986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:49.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.968+0000 7f7d42d9d700 1 --2- 172.21.15.29:0/3786859824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d4406afc0 0x7f7d441986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:49.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.968+0000 7f7d42d9d700 1 -- 172.21.15.29:0/3786859824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d4406a610 msgr2=0x7f7d441981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:49.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.968+0000 7f7d42d9d700 1 --2- 172.21.15.29:0/3786859824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d4406a610 0x7f7d441981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:49.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.968+0000 7f7d42d9d700 1 -- 172.21.15.29:0/3786859824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3c009d00 con 0x7f7d441070b0 2024-03-11T19:08:49.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.968+0000 7f7d41d9b700 1 --2- 172.21.15.29:0/3786859824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d4406afc0 0x7f7d441986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:08:49.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.968+0000 7f7d42d9d700 1 --2- 172.21.15.29:0/3786859824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d441070b0 0x7f7d4419ca70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c006070 tx=0x7f7d3c004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:49.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.968+0000 7f7d337fe700 1 -- 172.21.15.29:0/3786859824 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c00e040 con 0x7f7d441070b0 2024-03-11T19:08:49.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.968+0000 7f7d337fe700 1 -- 172.21.15.29:0/3786859824 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7d3c01b450 con 0x7f7d441070b0 2024-03-11T19:08:49.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.968+0000 7f7d337fe700 1 -- 172.21.15.29:0/3786859824 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c0052a0 con 0x7f7d441070b0 2024-03-11T19:08:49.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.968+0000 7f7d48c6b700 1 -- 172.21.15.29:0/3786859824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4419d630 con 0x7f7d441070b0 2024-03-11T19:08:49.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.968+0000 7f7d48c6b700 1 -- 172.21.15.29:0/3786859824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d441a4ef0 con 0x7f7d441070b0 2024-03-11T19:08:49.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.969+0000 7f7d48c6b700 1 -- 172.21.15.29:0/3786859824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4410a310 con 0x7f7d441070b0 2024-03-11T19:08:49.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.970+0000 7f7d337fe700 1 -- 172.21.15.29:0/3786859824 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f7d3c01b970 con 0x7f7d441070b0 2024-03-11T19:08:49.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.971+0000 7f7d337fe700 1 --2- 172.21.15.29:0/3786859824 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7d2c0a1b50 0x7f7d2c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:49.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.971+0000 7f7d4259c700 1 --2- 172.21.15.29:0/3786859824 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7d2c0a1b50 0x7f7d2c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:49.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.971+0000 7f7d337fe700 1 -- 172.21.15.29:0/3786859824 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7d3c0ce720 con 0x7f7d441070b0 2024-03-11T19:08:49.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.971+0000 7f7d4259c700 1 --2- 172.21.15.29:0/3786859824 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7d2c0a1b50 0x7f7d2c0a4010 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7d34009c60 tx=0x7f7d34009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:49.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:49.975+0000 7f7d337fe700 1 -- 172.21.15.29:0/3786859824 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7d3c097960 con 0x7f7d441070b0 2024-03-11T19:08:50.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:50.182+0000 7f7d48c6b700 1 -- 172.21.15.29:0/3786859824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7d44103430 con 0x7f7d441070b0 2024-03-11T19:08:50.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:50.182+0000 7f7d337fe700 1 -- 172.21.15.29:0/3786859824 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7d3c09b020 con 0x7f7d441070b0 2024-03-11T19:08:50.185 INFO:teuthology.orchestra.run.smithi029.stdout:34359738449 2024-03-11T19:08:50.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:50.184+0000 7f7d48c6b700 1 -- 172.21.15.29:0/3786859824 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7d2c0a1b50 msgr2=0x7f7d2c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:50.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:50.184+0000 7f7d48c6b700 1 --2- 172.21.15.29:0/3786859824 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7d2c0a1b50 0x7f7d2c0a4010 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7d34009c60 tx=0x7f7d34009260 comp rx=0 tx=0).stop 2024-03-11T19:08:50.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:50.184+0000 7f7d48c6b700 1 -- 172.21.15.29:0/3786859824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d441070b0 msgr2=0x7f7d4419ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:50.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:50.184+0000 7f7d48c6b700 1 --2- 172.21.15.29:0/3786859824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d441070b0 0x7f7d4419ca70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c006070 tx=0x7f7d3c004e40 comp rx=0 tx=0).stop 2024-03-11T19:08:50.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:50.184+0000 7f7d48c6b700 1 -- 172.21.15.29:0/3786859824 shutdown_connections 2024-03-11T19:08:50.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:50.184+0000 7f7d48c6b700 1 --2- 172.21.15.29:0/3786859824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d441070b0 0x7f7d4419ca70 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:50.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:50.184+0000 7f7d48c6b700 1 --2- 172.21.15.29:0/3786859824 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7d2c0a1b50 0x7f7d2c0a4010 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:50.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:50.184+0000 7f7d48c6b700 1 --2- 172.21.15.29:0/3786859824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d4406a610 0x7f7d441981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:50.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:50.184+0000 7f7d48c6b700 1 --2- 172.21.15.29:0/3786859824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d4406afc0 0x7f7d441986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:50.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:50.184+0000 7f7d48c6b700 1 -- 172.21.15.29:0/3786859824 >> 172.21.15.29:0/3786859824 conn(0x7f7d440779d0 msgr2=0x7f7d440fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:50.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:50.184+0000 7f7d48c6b700 1 -- 172.21.15.29:0/3786859824 shutdown_connections 2024-03-11T19:08:50.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:50.184+0000 7f7d48c6b700 1 -- 172.21.15.29:0/3786859824 wait complete. 2024-03-11T19:08:51.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.186+0000 7f97daeaa700 1 -- 172.21.15.29:0/2851673304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97d4100e00 msgr2=0x7f97d41011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:51.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.186+0000 7f97daeaa700 1 --2- 172.21.15.29:0/2851673304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97d4100e00 0x7f97d41011e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f97c8009990 tx=0x7f97c8009ca0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.186+0000 7f97daeaa700 1 -- 172.21.15.29:0/2851673304 shutdown_connections 2024-03-11T19:08:51.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.186+0000 7f97daeaa700 1 --2- 172.21.15.29:0/2851673304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97d4105e90 0x7f97d410da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.186+0000 7f97daeaa700 1 --2- 172.21.15.29:0/2851673304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97d4100e00 0x7f97d41011e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.186+0000 7f97daeaa700 1 --2- 172.21.15.29:0/2851673304 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97d41017b0 0x7f97d41057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.186+0000 7f97daeaa700 1 -- 172.21.15.29:0/2851673304 >> 172.21.15.29:0/2851673304 conn(0x7f97d4076b90 msgr2=0x7f97d4076fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:51.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.186+0000 7f97daeaa700 1 -- 172.21.15.29:0/2851673304 shutdown_connections 2024-03-11T19:08:51.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.186+0000 7f97daeaa700 1 -- 172.21.15.29:0/2851673304 wait complete. 2024-03-11T19:08:51.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.187+0000 7f97daeaa700 1 Processor -- start 2024-03-11T19:08:51.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.187+0000 7f97daeaa700 1 -- start start 2024-03-11T19:08:51.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.187+0000 7f97daeaa700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97d4100e00 0x7f97d41980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:51.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.187+0000 7f97daeaa700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97d41017b0 0x7f97d41985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:51.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.187+0000 7f97daeaa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97d4105e90 0x7f97d419c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:51.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.187+0000 7f97daeaa700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d419cf90 con 0x7f97d4105e90 2024-03-11T19:08:51.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.187+0000 7f97daeaa700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d419d100 con 0x7f97d4100e00 2024-03-11T19:08:51.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.187+0000 7f97daeaa700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d419d2c0 con 0x7f97d41017b0 2024-03-11T19:08:51.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.187+0000 7f97d9447700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97d4105e90 0x7f97d419c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:51.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.187+0000 7f97d9447700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97d4105e90 0x7f97d419c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57652/0 (socket says 172.21.15.29:57652) 2024-03-11T19:08:51.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.187+0000 7f97d9447700 1 -- 172.21.15.29:0/183438966 learned_addr learned my addr 172.21.15.29:0/183438966 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:51.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.187+0000 7f97d3fff700 1 --2- 172.21.15.29:0/183438966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97d41017b0 0x7f97d41985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:51.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.187+0000 7f97d8c46700 1 --2- 172.21.15.29:0/183438966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97d4100e00 0x7f97d41980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:51.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.188+0000 7f97d9447700 1 -- 172.21.15.29:0/183438966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97d41017b0 msgr2=0x7f97d41985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:51.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.188+0000 7f97d9447700 1 --2- 172.21.15.29:0/183438966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97d41017b0 0x7f97d41985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.188+0000 7f97d9447700 1 -- 172.21.15.29:0/183438966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97d4100e00 msgr2=0x7f97d41980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:51.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.188+0000 7f97d9447700 1 --2- 172.21.15.29:0/183438966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97d4100e00 0x7f97d41980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.188+0000 7f97d9447700 1 -- 172.21.15.29:0/183438966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c80095f0 con 0x7f97d4105e90 2024-03-11T19:08:51.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.188+0000 7f97d9447700 1 --2- 172.21.15.29:0/183438966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97d4105e90 0x7f97d419c980 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f97cc007d70 tx=0x7f97cc00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:51.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.188+0000 7f97d1ffb700 1 -- 172.21.15.29:0/183438966 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97cc00ac90 con 0x7f97d4105e90 2024-03-11T19:08:51.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.188+0000 7f97d1ffb700 1 -- 172.21.15.29:0/183438966 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f97cc00adf0 con 0x7f97d4105e90 2024-03-11T19:08:51.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.188+0000 7f97d1ffb700 1 -- 172.21.15.29:0/183438966 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97cc0115e0 con 0x7f97d4105e90 2024-03-11T19:08:51.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.188+0000 7f97daeaa700 1 -- 172.21.15.29:0/183438966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d419d5a0 con 0x7f97d4105e90 2024-03-11T19:08:51.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.188+0000 7f97daeaa700 1 -- 172.21.15.29:0/183438966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d41a4e00 con 0x7f97d4105e90 2024-03-11T19:08:51.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.189+0000 7f97daeaa700 1 -- 172.21.15.29:0/183438966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d410a130 con 0x7f97d4105e90 2024-03-11T19:08:51.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.190+0000 7f97d1ffb700 1 -- 172.21.15.29:0/183438966 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f97cc012420 con 0x7f97d4105e90 2024-03-11T19:08:51.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.190+0000 7f97d1ffb700 1 --2- 172.21.15.29:0/183438966 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97c40a1bd0 0x7f97c40a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:51.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.191+0000 7f97d8c46700 1 --2- 172.21.15.29:0/183438966 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97c40a1bd0 0x7f97c40a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:51.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.191+0000 7f97d1ffb700 1 -- 172.21.15.29:0/183438966 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f97cc0cd550 con 0x7f97d4105e90 2024-03-11T19:08:51.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.191+0000 7f97d8c46700 1 --2- 172.21.15.29:0/183438966 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97c40a1bd0 0x7f97c40a4090 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f97c8006010 tx=0x7f97c800b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:51.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.195+0000 7f97d1ffb700 1 -- 172.21.15.29:0/183438966 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f97cc096710 con 0x7f97d4105e90 2024-03-11T19:08:51.259 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:51 smithi170 ceph-mon[35690]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s; 13 B/s, 0 objects/s recovering 2024-03-11T19:08:51.259 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:51 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3786859824' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T19:08:51.259 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:51 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:08:51.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:51 smithi190 ceph-mon[36292]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s; 13 B/s, 0 objects/s recovering 2024-03-11T19:08:51.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:51 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3786859824' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T19:08:51.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:51 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:08:51.289 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:51 smithi029 ceph-mon[32439]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s; 13 B/s, 0 objects/s recovering 2024-03-11T19:08:51.289 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:51 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3786859824' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T19:08:51.289 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:51 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:08:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.388+0000 7f8038b79700 1 -- 172.21.15.29:0/3521551643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8034101530 msgr2=0x7f803410d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.388+0000 7f8038b79700 1 --2- 172.21.15.29:0/3521551643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8034101530 0x7f803410d9c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f802c00b0a0 tx=0x7f802c00b3b0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.388+0000 7f8038b79700 1 -- 172.21.15.29:0/3521551643 shutdown_connections 2024-03-11T19:08:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.388+0000 7f8038b79700 1 --2- 172.21.15.29:0/3521551643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8034101530 0x7f803410d9c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.388+0000 7f8038b79700 1 --2- 172.21.15.29:0/3521551643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8034106b80 0x7f8034106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.388+0000 7f8038b79700 1 --2- 172.21.15.29:0/3521551643 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8034100b70 0x7f8034100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.388+0000 7f8038b79700 1 -- 172.21.15.29:0/3521551643 >> 172.21.15.29:0/3521551643 conn(0x7f8034077aa0 msgr2=0x7f8034077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.388+0000 7f8038b79700 1 -- 172.21.15.29:0/3521551643 shutdown_connections 2024-03-11T19:08:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.388+0000 7f8038b79700 1 -- 172.21.15.29:0/3521551643 wait complete. 2024-03-11T19:08:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.389+0000 7f8038b79700 1 Processor -- start 2024-03-11T19:08:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.389+0000 7f8038b79700 1 -- start start 2024-03-11T19:08:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.389+0000 7f8038b79700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8034100b70 0x7f8034196f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.389+0000 7f8038b79700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8034101530 0x7f80341974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.389+0000 7f8038b79700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8034106b80 0x7f803419b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.389+0000 7f8038b79700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f803419be10 con 0x7f8034101530 2024-03-11T19:08:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.389+0000 7f8038b79700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f803419bf80 con 0x7f8034106b80 2024-03-11T19:08:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.389+0000 7f8038b79700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f803419c140 con 0x7f8034100b70 2024-03-11T19:08:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.389+0000 7f8031d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8034101530 0x7f80341974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.389+0000 7f8031d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8034101530 0x7f80341974c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57670/0 (socket says 172.21.15.29:57670) 2024-03-11T19:08:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.389+0000 7f8031d9b700 1 -- 172.21.15.29:0/1076713506 learned_addr learned my addr 172.21.15.29:0/1076713506 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.389+0000 7f803259c700 1 --2- 172.21.15.29:0/1076713506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8034100b70 0x7f8034196f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.389+0000 7f8032d9d700 1 --2- 172.21.15.29:0/1076713506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8034106b80 0x7f803419b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.390+0000 7f8031d9b700 1 -- 172.21.15.29:0/1076713506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8034100b70 msgr2=0x7f8034196f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.390+0000 7f8031d9b700 1 --2- 172.21.15.29:0/1076713506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8034100b70 0x7f8034196f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.390+0000 7f8031d9b700 1 -- 172.21.15.29:0/1076713506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8034106b80 msgr2=0x7f803419b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.390+0000 7f8031d9b700 1 --2- 172.21.15.29:0/1076713506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8034106b80 0x7f803419b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.390+0000 7f8031d9b700 1 -- 172.21.15.29:0/1076713506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802c009d00 con 0x7f8034101530 2024-03-11T19:08:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.390+0000 7f8031d9b700 1 --2- 172.21.15.29:0/1076713506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8034101530 0x7f80341974c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f802800e9a0 tx=0x7f802800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.390+0000 7f80237fe700 1 -- 172.21.15.29:0/1076713506 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8028009790 con 0x7f8034101530 2024-03-11T19:08:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.390+0000 7f80237fe700 1 -- 172.21.15.29:0/1076713506 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8028010430 con 0x7f8034101530 2024-03-11T19:08:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.390+0000 7f8038b79700 1 -- 172.21.15.29:0/1076713506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f803419c420 con 0x7f8034101530 2024-03-11T19:08:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.390+0000 7f80237fe700 1 -- 172.21.15.29:0/1076713506 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8028018680 con 0x7f8034101530 2024-03-11T19:08:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.390+0000 7f8038b79700 1 -- 172.21.15.29:0/1076713506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80341a3c80 con 0x7f8034101530 2024-03-11T19:08:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.391+0000 7f8038b79700 1 -- 172.21.15.29:0/1076713506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8034068930 con 0x7f8034101530 2024-03-11T19:08:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.393+0000 7f80237fe700 1 -- 172.21.15.29:0/1076713506 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f80280105a0 con 0x7f8034101530 2024-03-11T19:08:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.393+0000 7f80237fe700 1 --2- 172.21.15.29:0/1076713506 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f801c0a1b50 0x7f801c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.393+0000 7f803259c700 1 --2- 172.21.15.29:0/1076713506 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f801c0a1b50 0x7f801c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.394+0000 7f80237fe700 1 -- 172.21.15.29:0/1076713506 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f8028014070 con 0x7f8034101530 2024-03-11T19:08:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.394+0000 7f803259c700 1 --2- 172.21.15.29:0/1076713506 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f801c0a1b50 0x7f801c0a4010 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f802400ac50 tx=0x7f802400a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.398+0000 7f80237fe700 1 -- 172.21.15.29:0/1076713506 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8028052d20 con 0x7f8034101530 2024-03-11T19:08:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.401+0000 7f97daeaa700 1 -- 172.21.15.29:0/183438966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f97d4068930 con 0x7f97d4105e90 2024-03-11T19:08:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.401+0000 7f97d1ffb700 1 -- 172.21.15.29:0/183438966 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f97cc099dd0 con 0x7f97d4105e90 2024-03-11T19:08:51.404 INFO:teuthology.orchestra.run.smithi029.stdout:77309411400 2024-03-11T19:08:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.404+0000 7f97daeaa700 1 -- 172.21.15.29:0/183438966 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97c40a1bd0 msgr2=0x7f97c40a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.404+0000 7f97daeaa700 1 --2- 172.21.15.29:0/183438966 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97c40a1bd0 0x7f97c40a4090 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f97c8006010 tx=0x7f97c800b540 comp rx=0 tx=0).stop 2024-03-11T19:08:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.404+0000 7f97daeaa700 1 -- 172.21.15.29:0/183438966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97d4105e90 msgr2=0x7f97d419c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.404+0000 7f97daeaa700 1 --2- 172.21.15.29:0/183438966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97d4105e90 0x7f97d419c980 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f97cc007d70 tx=0x7f97cc00a340 comp rx=0 tx=0).stop 2024-03-11T19:08:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.404+0000 7f97daeaa700 1 -- 172.21.15.29:0/183438966 shutdown_connections 2024-03-11T19:08:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.404+0000 7f97daeaa700 1 --2- 172.21.15.29:0/183438966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97d4105e90 0x7f97d419c980 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.404+0000 7f97daeaa700 1 --2- 172.21.15.29:0/183438966 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97c40a1bd0 0x7f97c40a4090 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.404+0000 7f97daeaa700 1 --2- 172.21.15.29:0/183438966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97d4100e00 0x7f97d41980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.404+0000 7f97daeaa700 1 --2- 172.21.15.29:0/183438966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97d41017b0 0x7f97d41985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.404+0000 7f97daeaa700 1 -- 172.21.15.29:0/183438966 >> 172.21.15.29:0/183438966 conn(0x7f97d4076b90 msgr2=0x7f97d406a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.404+0000 7f97daeaa700 1 -- 172.21.15.29:0/183438966 shutdown_connections 2024-03-11T19:08:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.404+0000 7f97daeaa700 1 -- 172.21.15.29:0/183438966 wait complete. 2024-03-11T19:08:51.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.608+0000 7f8038b79700 1 -- 172.21.15.29:0/1076713506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f8034110a30 con 0x7f8034101530 2024-03-11T19:08:51.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.608+0000 7f80237fe700 1 -- 172.21.15.29:0/1076713506 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f80280967d0 con 0x7f8034101530 2024-03-11T19:08:51.612 INFO:teuthology.orchestra.run.smithi029.stdout:296352743460 2024-03-11T19:08:51.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.611+0000 7f8038b79700 1 -- 172.21.15.29:0/1076713506 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f801c0a1b50 msgr2=0x7f801c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.611+0000 7f8038b79700 1 --2- 172.21.15.29:0/1076713506 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f801c0a1b50 0x7f801c0a4010 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f802400ac50 tx=0x7f802400a380 comp rx=0 tx=0).stop 2024-03-11T19:08:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.611+0000 7f8038b79700 1 -- 172.21.15.29:0/1076713506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8034101530 msgr2=0x7f80341974c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.611+0000 7f8038b79700 1 --2- 172.21.15.29:0/1076713506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8034101530 0x7f80341974c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f802800e9a0 tx=0x7f802800ecb0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.611+0000 7f8038b79700 1 -- 172.21.15.29:0/1076713506 shutdown_connections 2024-03-11T19:08:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.611+0000 7f8038b79700 1 --2- 172.21.15.29:0/1076713506 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f801c0a1b50 0x7f801c0a4010 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.611+0000 7f8038b79700 1 --2- 172.21.15.29:0/1076713506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8034106b80 0x7f803419b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.611+0000 7f8038b79700 1 --2- 172.21.15.29:0/1076713506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8034100b70 0x7f8034196f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.611+0000 7f8038b79700 1 --2- 172.21.15.29:0/1076713506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8034101530 0x7f80341974c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.611+0000 7f8038b79700 1 -- 172.21.15.29:0/1076713506 >> 172.21.15.29:0/1076713506 conn(0x7f8034077aa0 msgr2=0x7f80340ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.611+0000 7f8038b79700 1 -- 172.21.15.29:0/1076713506 shutdown_connections 2024-03-11T19:08:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:51.611+0000 7f8038b79700 1 -- 172.21.15.29:0/1076713506 wait complete. 2024-03-11T19:08:52.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:52 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/183438966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T19:08:52.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:52 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1076713506' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-11T19:08:52.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:52 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:08:52.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:52 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:08:52.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:52 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/183438966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T19:08:52.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:52 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1076713506' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-11T19:08:52.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:52 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:08:52.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:52 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:08:52.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:52 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/183438966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T19:08:52.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:52 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1076713506' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-11T19:08:52.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:52 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:08:52.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:52 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:08:52.547 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888122 got 146028888125 for osd.4 2024-03-11T19:08:52.547 DEBUG:teuthology.parallel:result is None 2024-03-11T19:08:53.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.011+0000 7f187000d700 1 -- 172.21.15.29:0/75352038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1868102680 msgr2=0x7f186810eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:53.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.011+0000 7f187000d700 1 --2- 172.21.15.29:0/75352038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1868102680 0x7f186810eac0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f186000b0d0 tx=0x7f186000b3e0 comp rx=0 tx=0).stop 2024-03-11T19:08:53.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.012+0000 7f187000d700 1 -- 172.21.15.29:0/75352038 shutdown_connections 2024-03-11T19:08:53.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.012+0000 7f187000d700 1 --2- 172.21.15.29:0/75352038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1868102680 0x7f186810eac0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:53.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.012+0000 7f187000d700 1 --2- 172.21.15.29:0/75352038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1868107ca0 0x7f1868108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:53.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.012+0000 7f187000d700 1 --2- 172.21.15.29:0/75352038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1868101cc0 0x7f1868102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:53.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.012+0000 7f187000d700 1 -- 172.21.15.29:0/75352038 >> 172.21.15.29:0/75352038 conn(0x7f1868078da0 msgr2=0x7f18680791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:53.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.012+0000 7f187000d700 1 -- 172.21.15.29:0/75352038 shutdown_connections 2024-03-11T19:08:53.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.012+0000 7f187000d700 1 -- 172.21.15.29:0/75352038 wait complete. 2024-03-11T19:08:53.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.012+0000 7f187000d700 1 Processor -- start 2024-03-11T19:08:53.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.012+0000 7f187000d700 1 -- start start 2024-03-11T19:08:53.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.012+0000 7f187000d700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1868101cc0 0x7f1868073f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:53.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.012+0000 7f187000d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1868102680 0x7f18680744b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:53.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.012+0000 7f187000d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1868107ca0 0x7f18680749f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:53.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.012+0000 7f187000d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18681a6660 con 0x7f1868102680 2024-03-11T19:08:53.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.012+0000 7f187000d700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18681a67d0 con 0x7f1868101cc0 2024-03-11T19:08:53.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.012+0000 7f187000d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18681a6990 con 0x7f1868107ca0 2024-03-11T19:08:53.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.013+0000 7f186d5a8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1868102680 0x7f18680744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:53.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.013+0000 7f186d5a8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1868102680 0x7f18680744b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57690/0 (socket says 172.21.15.29:57690) 2024-03-11T19:08:53.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.013+0000 7f186d5a8700 1 -- 172.21.15.29:0/2995716430 learned_addr learned my addr 172.21.15.29:0/2995716430 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:53.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.013+0000 7f186dda9700 1 --2- 172.21.15.29:0/2995716430 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1868101cc0 0x7f1868073f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:53.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.013+0000 7f186d5a8700 1 -- 172.21.15.29:0/2995716430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1868107ca0 msgr2=0x7f18680749f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:53.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.013+0000 7f186d5a8700 1 --2- 172.21.15.29:0/2995716430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1868107ca0 0x7f18680749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:53.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.013+0000 7f186d5a8700 1 -- 172.21.15.29:0/2995716430 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1868101cc0 msgr2=0x7f1868073f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:53.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.013+0000 7f186d5a8700 1 --2- 172.21.15.29:0/2995716430 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1868101cc0 0x7f1868073f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:53.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.013+0000 7f186d5a8700 1 -- 172.21.15.29:0/2995716430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1860009d00 con 0x7f1868102680 2024-03-11T19:08:53.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.013+0000 7f186dda9700 1 --2- 172.21.15.29:0/2995716430 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1868101cc0 0x7f1868073f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:08:53.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.013+0000 7f186d5a8700 1 --2- 172.21.15.29:0/2995716430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1868102680 0x7f18680744b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f185800b530 tx=0x7f185800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:53.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.013+0000 7f185effd700 1 -- 172.21.15.29:0/2995716430 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1858011830 con 0x7f1868102680 2024-03-11T19:08:53.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.013+0000 7f185effd700 1 -- 172.21.15.29:0/2995716430 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1858011e40 con 0x7f1868102680 2024-03-11T19:08:53.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.013+0000 7f187000d700 1 -- 172.21.15.29:0/2995716430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18681a6c70 con 0x7f1868102680 2024-03-11T19:08:53.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.013+0000 7f185effd700 1 -- 172.21.15.29:0/2995716430 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185800f2e0 con 0x7f1868102680 2024-03-11T19:08:53.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.013+0000 7f187000d700 1 -- 172.21.15.29:0/2995716430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18681a7170 con 0x7f1868102680 2024-03-11T19:08:53.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.014+0000 7f187000d700 1 -- 172.21.15.29:0/2995716430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f186810a180 con 0x7f1868102680 2024-03-11T19:08:53.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.015+0000 7f185effd700 1 -- 172.21.15.29:0/2995716430 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f1858011990 con 0x7f1868102680 2024-03-11T19:08:53.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.016+0000 7f185effd700 1 --2- 172.21.15.29:0/2995716430 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f18540a1b50 0x7f18540a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:53.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.016+0000 7f186dda9700 1 --2- 172.21.15.29:0/2995716430 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f18540a1b50 0x7f18540a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:53.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.016+0000 7f185effd700 1 -- 172.21.15.29:0/2995716430 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f18580cd4a0 con 0x7f1868102680 2024-03-11T19:08:53.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.016+0000 7f186dda9700 1 --2- 172.21.15.29:0/2995716430 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f18540a1b50 0x7f18540a4010 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1864005fd0 tx=0x7f1864005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:53.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.020+0000 7f185effd700 1 -- 172.21.15.29:0/2995716430 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1858096760 con 0x7f1868102680 2024-03-11T19:08:53.175 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:53 smithi029 ceph-mon[32439]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 13 B/s, 0 objects/s recovering 2024-03-11T19:08:53.175 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:53 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:08:53.175 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:53 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:08:53.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.230+0000 7f187000d700 1 -- 172.21.15.29:0/2995716430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f1868068930 con 0x7f1868102680 2024-03-11T19:08:53.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.230+0000 7f185effd700 1 -- 172.21.15.29:0/2995716430 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1858010ed0 con 0x7f1868102680 2024-03-11T19:08:53.233 INFO:teuthology.orchestra.run.smithi029.stdout:111669149762 2024-03-11T19:08:53.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.232+0000 7f187000d700 1 -- 172.21.15.29:0/2995716430 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f18540a1b50 msgr2=0x7f18540a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:53.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.232+0000 7f187000d700 1 --2- 172.21.15.29:0/2995716430 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f18540a1b50 0x7f18540a4010 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1864005fd0 tx=0x7f1864005ee0 comp rx=0 tx=0).stop 2024-03-11T19:08:53.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.232+0000 7f187000d700 1 -- 172.21.15.29:0/2995716430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1868102680 msgr2=0x7f18680744b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:53.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.232+0000 7f187000d700 1 --2- 172.21.15.29:0/2995716430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1868102680 0x7f18680744b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f185800b530 tx=0x7f185800b8f0 comp rx=0 tx=0).stop 2024-03-11T19:08:53.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.232+0000 7f187000d700 1 -- 172.21.15.29:0/2995716430 shutdown_connections 2024-03-11T19:08:53.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.232+0000 7f187000d700 1 --2- 172.21.15.29:0/2995716430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1868107ca0 0x7f18680749f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:53.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.232+0000 7f187000d700 1 --2- 172.21.15.29:0/2995716430 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f18540a1b50 0x7f18540a4010 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:53.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.232+0000 7f187000d700 1 --2- 172.21.15.29:0/2995716430 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1868101cc0 0x7f1868073f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:53.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.232+0000 7f187000d700 1 --2- 172.21.15.29:0/2995716430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1868102680 0x7f18680744b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:53.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.232+0000 7f187000d700 1 -- 172.21.15.29:0/2995716430 >> 172.21.15.29:0/2995716430 conn(0x7f1868078da0 msgr2=0x7f18681009f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:53.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.232+0000 7f187000d700 1 -- 172.21.15.29:0/2995716430 shutdown_connections 2024-03-11T19:08:53.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:53.232+0000 7f187000d700 1 -- 172.21.15.29:0/2995716430 wait complete. 2024-03-11T19:08:53.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:53 smithi170 ceph-mon[35690]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 13 B/s, 0 objects/s recovering 2024-03-11T19:08:53.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:53 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:08:53.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:53 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:08:53.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:53 smithi190 ceph-mon[36292]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 13 B/s, 0 objects/s recovering 2024-03-11T19:08:53.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:53 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:08:53.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:53 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:08:54.294 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:54 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2995716430' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T19:08:54.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:54 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2995716430' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T19:08:54.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:54 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2995716430' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T19:08:54.678 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:55.090 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:55.104 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743457 got 296352743460 for osd.9 2024-03-11T19:08:55.104 DEBUG:teuthology.parallel:result is None 2024-03-11T19:08:55.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:55 smithi170 ceph-mon[35690]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s; 11 B/s, 0 objects/s recovering 2024-03-11T19:08:55.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:55 smithi029 ceph-mon[32439]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s; 11 B/s, 0 objects/s recovering 2024-03-11T19:08:55.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:55 smithi190 ceph-mon[36292]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s; 11 B/s, 0 objects/s recovering 2024-03-11T19:08:55.520 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738444 got 34359738449 for osd.0 2024-03-11T19:08:55.521 DEBUG:teuthology.parallel:result is None 2024-03-11T19:08:56.189 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:08:56.226 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149758 got 111669149762 for osd.3 2024-03-11T19:08:56.227 DEBUG:teuthology.parallel:result is None 2024-03-11T19:08:56.671 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411395 got 77309411400 for osd.2 2024-03-11T19:08:56.671 DEBUG:teuthology.parallel:result is None 2024-03-11T19:08:57.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:57 smithi029 ceph-mon[32439]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 10 B/s, 0 objects/s recovering 2024-03-11T19:08:57.314 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:57 smithi190 ceph-mon[36292]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 10 B/s, 0 objects/s recovering 2024-03-11T19:08:57.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:57 smithi170 ceph-mon[35690]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 10 B/s, 0 objects/s recovering 2024-03-11T19:08:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.553+0000 7f535a78c700 1 -- 172.21.15.29:0/2889418628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5354101530 msgr2=0x7f535410d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.553+0000 7f535a78c700 1 --2- 172.21.15.29:0/2889418628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5354101530 0x7f535410d9c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f534c00b0a0 tx=0x7f534c00b3b0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.553+0000 7f535a78c700 1 -- 172.21.15.29:0/2889418628 shutdown_connections 2024-03-11T19:08:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.553+0000 7f535a78c700 1 --2- 172.21.15.29:0/2889418628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5354101530 0x7f535410d9c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.553+0000 7f535a78c700 1 --2- 172.21.15.29:0/2889418628 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5354106b80 0x7f5354106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.553+0000 7f535a78c700 1 --2- 172.21.15.29:0/2889418628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5354100b70 0x7f5354100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.553+0000 7f535a78c700 1 -- 172.21.15.29:0/2889418628 >> 172.21.15.29:0/2889418628 conn(0x7f5354077aa0 msgr2=0x7f5354077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.553+0000 7f535a78c700 1 -- 172.21.15.29:0/2889418628 shutdown_connections 2024-03-11T19:08:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.553+0000 7f535a78c700 1 -- 172.21.15.29:0/2889418628 wait complete. 2024-03-11T19:08:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.554+0000 7f535a78c700 1 Processor -- start 2024-03-11T19:08:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.554+0000 7f535a78c700 1 -- start start 2024-03-11T19:08:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.554+0000 7f535a78c700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5354100b70 0x7f53541027b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.554+0000 7f535a78c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5354101530 0x7f5354102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.554+0000 7f535a78c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5354106b80 0x7f5354103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.554+0000 7f535a78c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53541a64d0 con 0x7f5354106b80 2024-03-11T19:08:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.554+0000 7f535a78c700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53541a6640 con 0x7f5354100b70 2024-03-11T19:08:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.554+0000 7f535a78c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53541a6800 con 0x7f5354101530 2024-03-11T19:08:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.554+0000 7f5358d29700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5354106b80 0x7f5354103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.554+0000 7f5358d29700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5354106b80 0x7f5354103230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51056/0 (socket says 172.21.15.29:51056) 2024-03-11T19:08:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.554+0000 7f5358d29700 1 -- 172.21.15.29:0/657947438 learned_addr learned my addr 172.21.15.29:0/657947438 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.555+0000 7f5353fff700 1 --2- 172.21.15.29:0/657947438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5354100b70 0x7f53541027b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.555+0000 7f5358d29700 1 -- 172.21.15.29:0/657947438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5354101530 msgr2=0x7f5354102cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.555+0000 7f5358d29700 1 --2- 172.21.15.29:0/657947438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5354101530 0x7f5354102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.555+0000 7f5358d29700 1 -- 172.21.15.29:0/657947438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5354100b70 msgr2=0x7f53541027b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.555+0000 7f5358d29700 1 --2- 172.21.15.29:0/657947438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5354100b70 0x7f53541027b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.555+0000 7f5358d29700 1 -- 172.21.15.29:0/657947438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534c009d00 con 0x7f5354106b80 2024-03-11T19:08:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.555+0000 7f5358d29700 1 --2- 172.21.15.29:0/657947438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5354106b80 0x7f5354103230 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f534c00b070 tx=0x7f534c009430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.555+0000 7f5353fff700 1 --2- 172.21.15.29:0/657947438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5354100b70 0x7f53541027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:08:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.555+0000 7f53517fa700 1 -- 172.21.15.29:0/657947438 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534c00e040 con 0x7f5354106b80 2024-03-11T19:08:58.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.555+0000 7f535a78c700 1 -- 172.21.15.29:0/657947438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53541a6a80 con 0x7f5354106b80 2024-03-11T19:08:58.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.555+0000 7f53517fa700 1 -- 172.21.15.29:0/657947438 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f534c007930 con 0x7f5354106b80 2024-03-11T19:08:58.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.555+0000 7f53517fa700 1 -- 172.21.15.29:0/657947438 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534c01b6b0 con 0x7f5354106b80 2024-03-11T19:08:58.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.555+0000 7f535a78c700 1 -- 172.21.15.29:0/657947438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53541a6ef0 con 0x7f5354106b80 2024-03-11T19:08:58.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.556+0000 7f535a78c700 1 -- 172.21.15.29:0/657947438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5354068930 con 0x7f5354106b80 2024-03-11T19:08:58.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.558+0000 7f53517fa700 1 -- 172.21.15.29:0/657947438 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f534c01d480 con 0x7f5354106b80 2024-03-11T19:08:58.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.559+0000 7f53517fa700 1 --2- 172.21.15.29:0/657947438 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f533c0a1bd0 0x7f533c0a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:58.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.559+0000 7f5353fff700 1 --2- 172.21.15.29:0/657947438 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f533c0a1bd0 0x7f533c0a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:58.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.559+0000 7f53517fa700 1 -- 172.21.15.29:0/657947438 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f534c0cf2e0 con 0x7f5354106b80 2024-03-11T19:08:58.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.560+0000 7f5353fff700 1 --2- 172.21.15.29:0/657947438 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f533c0a1bd0 0x7f533c0a4090 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5344009b50 tx=0x7f5344009550 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:58.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.565+0000 7f53517fa700 1 -- 172.21.15.29:0/657947438 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f534c098520 con 0x7f5354106b80 2024-03-11T19:08:58.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.769+0000 7f5e2a14a700 1 -- 172.21.15.29:0/1528698760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e24101530 msgr2=0x7f5e2410d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:58.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.769+0000 7f5e2a14a700 1 --2- 172.21.15.29:0/1528698760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e24101530 0x7f5e2410d9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c009990 tx=0x7f5e1c009ca0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.769+0000 7f5e2a14a700 1 -- 172.21.15.29:0/1528698760 shutdown_connections 2024-03-11T19:08:58.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.769+0000 7f5e2a14a700 1 --2- 172.21.15.29:0/1528698760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e24101530 0x7f5e2410d9c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.769+0000 7f5e2a14a700 1 --2- 172.21.15.29:0/1528698760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e24106b80 0x7f5e24106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.769+0000 7f5e2a14a700 1 --2- 172.21.15.29:0/1528698760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e24100b70 0x7f5e24100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.769+0000 7f5e2a14a700 1 -- 172.21.15.29:0/1528698760 >> 172.21.15.29:0/1528698760 conn(0x7f5e24077aa0 msgr2=0x7f5e24077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:58.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.769+0000 7f5e2a14a700 1 -- 172.21.15.29:0/1528698760 shutdown_connections 2024-03-11T19:08:58.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.769+0000 7f5e2a14a700 1 -- 172.21.15.29:0/1528698760 wait complete. 2024-03-11T19:08:58.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.770+0000 7f5e2a14a700 1 Processor -- start 2024-03-11T19:08:58.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.770+0000 7f5e2a14a700 1 -- start start 2024-03-11T19:08:58.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.770+0000 7f5e2a14a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e24100b70 0x7f5e24198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:58.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.770+0000 7f5e2a14a700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e24101530 0x7f5e241992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:58.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.770+0000 7f5e2a14a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e24106b80 0x7f5e241a3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:58.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.770+0000 7f5e2a14a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e24199b80 con 0x7f5e24100b70 2024-03-11T19:08:58.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.771+0000 7f5e237fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e24100b70 0x7f5e24198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:58.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.771+0000 7f5e237fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e24100b70 0x7f5e24198da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51082/0 (socket says 172.21.15.29:51082) 2024-03-11T19:08:58.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.771+0000 7f5e237fe700 1 -- 172.21.15.29:0/1240681049 learned_addr learned my addr 172.21.15.29:0/1240681049 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:08:58.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.771+0000 7f5e2a14a700 1 -- 172.21.15.29:0/1240681049 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e241a4360 con 0x7f5e24101530 2024-03-11T19:08:58.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.771+0000 7f5e2a14a700 1 -- 172.21.15.29:0/1240681049 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e241a44a0 con 0x7f5e24106b80 2024-03-11T19:08:58.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.771+0000 7f5e22ffd700 1 --2- 172.21.15.29:0/1240681049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e24101530 0x7f5e241992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:58.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.771+0000 7f5e23fff700 1 --2- 172.21.15.29:0/1240681049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e24106b80 0x7f5e241a3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:58.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.771+0000 7f5e237fe700 1 -- 172.21.15.29:0/1240681049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e24106b80 msgr2=0x7f5e241a3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:58.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.771+0000 7f5e237fe700 1 --2- 172.21.15.29:0/1240681049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e24106b80 0x7f5e241a3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.771+0000 7f5e237fe700 1 -- 172.21.15.29:0/1240681049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e24101530 msgr2=0x7f5e241992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:58.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.771+0000 7f5e237fe700 1 --2- 172.21.15.29:0/1240681049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e24101530 0x7f5e241992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.771+0000 7f5e237fe700 1 -- 172.21.15.29:0/1240681049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c0095f0 con 0x7f5e24100b70 2024-03-11T19:08:58.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.771+0000 7f5e237fe700 1 --2- 172.21.15.29:0/1240681049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e24100b70 0x7f5e24198da0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5e14007df0 tx=0x7f5e1400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:58.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.771+0000 7f5e20ff9700 1 -- 172.21.15.29:0/1240681049 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1400acb0 con 0x7f5e24100b70 2024-03-11T19:08:58.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.772+0000 7f5e2a14a700 1 -- 172.21.15.29:0/1240681049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e241a4750 con 0x7f5e24100b70 2024-03-11T19:08:58.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.772+0000 7f5e20ff9700 1 -- 172.21.15.29:0/1240681049 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5e1400ae10 con 0x7f5e24100b70 2024-03-11T19:08:58.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.772+0000 7f5e20ff9700 1 -- 172.21.15.29:0/1240681049 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e140116e0 con 0x7f5e24100b70 2024-03-11T19:08:58.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.772+0000 7f5e2a14a700 1 -- 172.21.15.29:0/1240681049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e241a4ca0 con 0x7f5e24100b70 2024-03-11T19:08:58.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.773+0000 7f5e2a14a700 1 -- 172.21.15.29:0/1240681049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e24109020 con 0x7f5e24100b70 2024-03-11T19:08:58.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.774+0000 7f535a78c700 1 -- 172.21.15.29:0/657947438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f5354110ad0 con 0x7f5354106b80 2024-03-11T19:08:58.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.774+0000 7f53517fa700 1 -- 172.21.15.29:0/657947438 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f534c021020 con 0x7f5354106b80 2024-03-11T19:08:58.777 INFO:teuthology.orchestra.run.smithi029.stdout:206158430261 2024-03-11T19:08:58.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.774+0000 7f5e20ff9700 1 -- 172.21.15.29:0/1240681049 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f5e14012420 con 0x7f5e24100b70 2024-03-11T19:08:58.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.775+0000 7f5e20ff9700 1 --2- 172.21.15.29:0/1240681049 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5e100a1bd0 0x7f5e100a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:08:58.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.775+0000 7f5e22ffd700 1 --2- 172.21.15.29:0/1240681049 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5e100a1bd0 0x7f5e100a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:08:58.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.775+0000 7f5e20ff9700 1 -- 172.21.15.29:0/1240681049 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5e140cd280 con 0x7f5e24100b70 2024-03-11T19:08:58.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.776+0000 7f5e22ffd700 1 --2- 172.21.15.29:0/1240681049 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5e100a1bd0 0x7f5e100a4090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5e24078030 tx=0x7f5e0c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:08:58.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.777+0000 7f535a78c700 1 -- 172.21.15.29:0/657947438 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f533c0a1bd0 msgr2=0x7f533c0a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:58.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.777+0000 7f535a78c700 1 --2- 172.21.15.29:0/657947438 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f533c0a1bd0 0x7f533c0a4090 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5344009b50 tx=0x7f5344009550 comp rx=0 tx=0).stop 2024-03-11T19:08:58.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.777+0000 7f535a78c700 1 -- 172.21.15.29:0/657947438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5354106b80 msgr2=0x7f5354103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:58.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.777+0000 7f535a78c700 1 --2- 172.21.15.29:0/657947438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5354106b80 0x7f5354103230 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f534c00b070 tx=0x7f534c009430 comp rx=0 tx=0).stop 2024-03-11T19:08:58.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.777+0000 7f535a78c700 1 -- 172.21.15.29:0/657947438 shutdown_connections 2024-03-11T19:08:58.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.777+0000 7f535a78c700 1 --2- 172.21.15.29:0/657947438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5354106b80 0x7f5354103230 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.777+0000 7f535a78c700 1 --2- 172.21.15.29:0/657947438 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f533c0a1bd0 0x7f533c0a4090 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.777+0000 7f535a78c700 1 --2- 172.21.15.29:0/657947438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5354100b70 0x7f53541027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.777+0000 7f535a78c700 1 --2- 172.21.15.29:0/657947438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5354101530 0x7f5354102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.777+0000 7f535a78c700 1 -- 172.21.15.29:0/657947438 >> 172.21.15.29:0/657947438 conn(0x7f5354077aa0 msgr2=0x7f53540ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:58.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.777+0000 7f535a78c700 1 -- 172.21.15.29:0/657947438 shutdown_connections 2024-03-11T19:08:58.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.777+0000 7f535a78c700 1 -- 172.21.15.29:0/657947438 wait complete. 2024-03-11T19:08:58.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.780+0000 7f5e20ff9700 1 -- 172.21.15.29:0/1240681049 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5e14096540 con 0x7f5e24100b70 2024-03-11T19:08:58.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.992+0000 7f5e2a14a700 1 -- 172.21.15.29:0/1240681049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f5e24068930 con 0x7f5e24100b70 2024-03-11T19:08:58.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.992+0000 7f5e20ff9700 1 -- 172.21.15.29:0/1240681049 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f5e14099c00 con 0x7f5e24100b70 2024-03-11T19:08:58.996 INFO:teuthology.orchestra.run.smithi029.stdout:352187318299 2024-03-11T19:08:58.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.994+0000 7f5e2a14a700 1 -- 172.21.15.29:0/1240681049 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5e100a1bd0 msgr2=0x7f5e100a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:58.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.995+0000 7f5e2a14a700 1 --2- 172.21.15.29:0/1240681049 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5e100a1bd0 0x7f5e100a4090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5e24078030 tx=0x7f5e0c009260 comp rx=0 tx=0).stop 2024-03-11T19:08:58.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.995+0000 7f5e2a14a700 1 -- 172.21.15.29:0/1240681049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e24100b70 msgr2=0x7f5e24198da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:08:58.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.995+0000 7f5e2a14a700 1 --2- 172.21.15.29:0/1240681049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e24100b70 0x7f5e24198da0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5e14007df0 tx=0x7f5e1400a340 comp rx=0 tx=0).stop 2024-03-11T19:08:58.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.995+0000 7f5e2a14a700 1 -- 172.21.15.29:0/1240681049 shutdown_connections 2024-03-11T19:08:58.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.995+0000 7f5e2a14a700 1 --2- 172.21.15.29:0/1240681049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e24106b80 0x7f5e241a3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.995+0000 7f5e2a14a700 1 --2- 172.21.15.29:0/1240681049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e24100b70 0x7f5e24198da0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.995+0000 7f5e2a14a700 1 --2- 172.21.15.29:0/1240681049 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5e100a1bd0 0x7f5e100a4090 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.995+0000 7f5e2a14a700 1 --2- 172.21.15.29:0/1240681049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e24101530 0x7f5e241992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:08:58.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.995+0000 7f5e2a14a700 1 -- 172.21.15.29:0/1240681049 >> 172.21.15.29:0/1240681049 conn(0x7f5e24077aa0 msgr2=0x7f5e240ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:08:58.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.995+0000 7f5e2a14a700 1 -- 172.21.15.29:0/1240681049 shutdown_connections 2024-03-11T19:08:58.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:08:58.995+0000 7f5e2a14a700 1 -- 172.21.15.29:0/1240681049 wait complete. 2024-03-11T19:08:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:59 smithi170 ceph-mon[35690]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s; 9 B/s, 0 objects/s recovering 2024-03-11T19:08:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:59 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/657947438' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T19:08:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:08:59 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1240681049' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-11T19:08:59.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:59 smithi029 ceph-mon[32439]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s; 9 B/s, 0 objects/s recovering 2024-03-11T19:08:59.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:59 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/657947438' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T19:08:59.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:08:59 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1240681049' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-11T19:08:59.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:59 smithi190 ceph-mon[36292]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s; 9 B/s, 0 objects/s recovering 2024-03-11T19:08:59.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:59 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/657947438' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T19:08:59.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:08:59 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1240681049' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-11T19:08:59.772 INFO:tasks.cephadm.ceph_manager.ceph:need seq 352187318294 got 352187318299 for osd.11 2024-03-11T19:08:59.772 DEBUG:teuthology.parallel:result is None 2024-03-11T19:09:00.435 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430256 got 206158430261 for osd.6 2024-03-11T19:09:00.435 DEBUG:teuthology.parallel:result is None 2024-03-11T19:09:01.220 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:01 smithi029 ceph-mon[32439]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-03-11T19:09:01.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:01 smithi170 ceph-mon[35690]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-03-11T19:09:01.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.428+0000 7fb049f19700 1 -- 172.21.15.29:0/4085396717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb044107cc0 msgr2=0x7fb0441080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:01.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.428+0000 7fb049f19700 1 --2- 172.21.15.29:0/4085396717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb044107cc0 0x7fb0441080a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb0340099c0 tx=0x7fb034009cd0 comp rx=0 tx=0).stop 2024-03-11T19:09:01.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.428+0000 7fb049f19700 1 -- 172.21.15.29:0/4085396717 shutdown_connections 2024-03-11T19:09:01.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.428+0000 7fb049f19700 1 --2- 172.21.15.29:0/4085396717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb04406a2c0 0x7fb04410eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:01.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.428+0000 7fb049f19700 1 --2- 172.21.15.29:0/4085396717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb044107cc0 0x7fb0441080a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:01.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.428+0000 7fb049f19700 1 --2- 172.21.15.29:0/4085396717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb044069920 0x7fb044069d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:01.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.428+0000 7fb049f19700 1 -- 172.21.15.29:0/4085396717 >> 172.21.15.29:0/4085396717 conn(0x7fb044077aa0 msgr2=0x7fb044077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:01.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.428+0000 7fb049f19700 1 -- 172.21.15.29:0/4085396717 shutdown_connections 2024-03-11T19:09:01.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.428+0000 7fb049f19700 1 -- 172.21.15.29:0/4085396717 wait complete. 2024-03-11T19:09:01.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.428+0000 7fb049f19700 1 Processor -- start 2024-03-11T19:09:01.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.428+0000 7fb049f19700 1 -- start start 2024-03-11T19:09:01.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb049f19700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb044069920 0x7fb044198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:01.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb049f19700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb04406a2c0 0x7fb044198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:01.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb049f19700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb044107cc0 0x7fb04419c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:01.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb049f19700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04419cf00 con 0x7fb044107cc0 2024-03-11T19:09:01.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb049f19700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04419d070 con 0x7fb044069920 2024-03-11T19:09:01.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb049f19700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04419d230 con 0x7fb04406a2c0 2024-03-11T19:09:01.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb043fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb044107cc0 0x7fb04419c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:01.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb043fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb044107cc0 0x7fb04419c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51112/0 (socket says 172.21.15.29:51112) 2024-03-11T19:09:01.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb043fff700 1 -- 172.21.15.29:0/1285211848 learned_addr learned my addr 172.21.15.29:0/1285211848 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:01.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb0437fe700 1 --2- 172.21.15.29:0/1285211848 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb044069920 0x7fb044198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:01.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb042ffd700 1 --2- 172.21.15.29:0/1285211848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb04406a2c0 0x7fb044198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:01.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb043fff700 1 -- 172.21.15.29:0/1285211848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb04406a2c0 msgr2=0x7fb044198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:01.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb043fff700 1 --2- 172.21.15.29:0/1285211848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb04406a2c0 0x7fb044198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:01.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb043fff700 1 -- 172.21.15.29:0/1285211848 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb044069920 msgr2=0x7fb044198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:01.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb043fff700 1 --2- 172.21.15.29:0/1285211848 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb044069920 0x7fb044198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:01.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb043fff700 1 -- 172.21.15.29:0/1285211848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0340095f0 con 0x7fb044107cc0 2024-03-11T19:09:01.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb043fff700 1 --2- 172.21.15.29:0/1285211848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb044107cc0 0x7fb04419c8f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00b970 tx=0x7fb03c00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:01.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb040ff9700 1 -- 172.21.15.29:0/1285211848 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c00c9b0 con 0x7fb044107cc0 2024-03-11T19:09:01.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb040ff9700 1 -- 172.21.15.29:0/1285211848 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb03c014440 con 0x7fb044107cc0 2024-03-11T19:09:01.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb049f19700 1 -- 172.21.15.29:0/1285211848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb04419d510 con 0x7fb044107cc0 2024-03-11T19:09:01.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.429+0000 7fb040ff9700 1 -- 172.21.15.29:0/1285211848 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c0075d0 con 0x7fb044107cc0 2024-03-11T19:09:01.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.430+0000 7fb049f19700 1 -- 172.21.15.29:0/1285211848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0441a4d70 con 0x7fb044107cc0 2024-03-11T19:09:01.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.430+0000 7fb049f19700 1 -- 172.21.15.29:0/1285211848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb04410a1e0 con 0x7fb044107cc0 2024-03-11T19:09:01.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.431+0000 7fb040ff9700 1 -- 172.21.15.29:0/1285211848 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fb03c0145b0 con 0x7fb044107cc0 2024-03-11T19:09:01.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.432+0000 7fb040ff9700 1 --2- 172.21.15.29:0/1285211848 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb02c0a1b50 0x7fb02c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:01.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.432+0000 7fb0437fe700 1 --2- 172.21.15.29:0/1285211848 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb02c0a1b50 0x7fb02c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:01.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.433+0000 7fb040ff9700 1 -- 172.21.15.29:0/1285211848 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb03c006e40 con 0x7fb044107cc0 2024-03-11T19:09:01.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.433+0000 7fb0437fe700 1 --2- 172.21.15.29:0/1285211848 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb02c0a1b50 0x7fb02c0a4010 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb034009fd0 tx=0x7fb034005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:01.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.436+0000 7fb040ff9700 1 -- 172.21.15.29:0/1285211848 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb03c096230 con 0x7fb044107cc0 2024-03-11T19:09:01.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:01 smithi190 ceph-mon[36292]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-03-11T19:09:01.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.643+0000 7fb049f19700 1 -- 172.21.15.29:0/1285211848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fb044111bb0 con 0x7fb044107cc0 2024-03-11T19:09:01.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.643+0000 7fb040ff9700 1 -- 172.21.15.29:0/1285211848 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb03c0998f0 con 0x7fb044107cc0 2024-03-11T19:09:01.646 INFO:teuthology.orchestra.run.smithi029.stdout:227633266736 2024-03-11T19:09:01.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.646+0000 7fb049f19700 1 -- 172.21.15.29:0/1285211848 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb02c0a1b50 msgr2=0x7fb02c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:01.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.646+0000 7fb049f19700 1 --2- 172.21.15.29:0/1285211848 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb02c0a1b50 0x7fb02c0a4010 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb034009fd0 tx=0x7fb034005fd0 comp rx=0 tx=0).stop 2024-03-11T19:09:01.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.646+0000 7fb049f19700 1 -- 172.21.15.29:0/1285211848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb044107cc0 msgr2=0x7fb04419c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:01.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.646+0000 7fb049f19700 1 --2- 172.21.15.29:0/1285211848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb044107cc0 0x7fb04419c8f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00b970 tx=0x7fb03c00bd30 comp rx=0 tx=0).stop 2024-03-11T19:09:01.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.647+0000 7fb049f19700 1 -- 172.21.15.29:0/1285211848 shutdown_connections 2024-03-11T19:09:01.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.647+0000 7fb049f19700 1 --2- 172.21.15.29:0/1285211848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb044107cc0 0x7fb04419c8f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:01.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.647+0000 7fb049f19700 1 --2- 172.21.15.29:0/1285211848 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb02c0a1b50 0x7fb02c0a4010 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:01.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.647+0000 7fb049f19700 1 --2- 172.21.15.29:0/1285211848 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb044069920 0x7fb044198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:01.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.647+0000 7fb049f19700 1 --2- 172.21.15.29:0/1285211848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb04406a2c0 0x7fb044198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:01.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.647+0000 7fb049f19700 1 -- 172.21.15.29:0/1285211848 >> 172.21.15.29:0/1285211848 conn(0x7fb044077aa0 msgr2=0x7fb0440ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:01.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.647+0000 7fb049f19700 1 -- 172.21.15.29:0/1285211848 shutdown_connections 2024-03-11T19:09:01.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:01.647+0000 7fb049f19700 1 -- 172.21.15.29:0/1285211848 wait complete. 2024-03-11T19:09:02.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.162+0000 7f3ba36a7700 1 -- 172.21.15.29:0/3712124753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c100e50 msgr2=0x7f3b9c1012d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:02.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.162+0000 7f3ba36a7700 1 --2- 172.21.15.29:0/3712124753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c100e50 0x7f3b9c1012d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c009990 tx=0x7f3b8c009ca0 comp rx=0 tx=0).stop 2024-03-11T19:09:02.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.162+0000 7f3ba36a7700 1 -- 172.21.15.29:0/3712124753 shutdown_connections 2024-03-11T19:09:02.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.162+0000 7f3ba36a7700 1 --2- 172.21.15.29:0/3712124753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b9c101810 0x7f3b9c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:02.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.162+0000 7f3ba36a7700 1 --2- 172.21.15.29:0/3712124753 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b9c106d60 0x7f3b9c107140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:02.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.162+0000 7f3ba36a7700 1 --2- 172.21.15.29:0/3712124753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c100e50 0x7f3b9c1012d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:02.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.162+0000 7f3ba36a7700 1 -- 172.21.15.29:0/3712124753 >> 172.21.15.29:0/3712124753 conn(0x7f3b9c0779d0 msgr2=0x7f3b9c077de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:02.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.162+0000 7f3ba36a7700 1 -- 172.21.15.29:0/3712124753 shutdown_connections 2024-03-11T19:09:02.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.162+0000 7f3ba36a7700 1 -- 172.21.15.29:0/3712124753 wait complete. 2024-03-11T19:09:02.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.163+0000 7f3ba36a7700 1 Processor -- start 2024-03-11T19:09:02.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.163+0000 7f3ba36a7700 1 -- start start 2024-03-11T19:09:02.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.163+0000 7f3ba36a7700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b9c100e50 0x7f3b9c1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:02.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.163+0000 7f3ba36a7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b9c101810 0x7f3b9c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:02.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.163+0000 7f3ba36a7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c106d60 0x7f3b9c19c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:02.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.163+0000 7f3ba36a7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c19cf80 con 0x7f3b9c106d60 2024-03-11T19:09:02.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.163+0000 7f3ba36a7700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c19d0f0 con 0x7f3b9c100e50 2024-03-11T19:09:02.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.163+0000 7f3ba36a7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c19d2b0 con 0x7f3b9c101810 2024-03-11T19:09:02.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.163+0000 7f3ba1c44700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c106d60 0x7f3b9c19c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:02.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.163+0000 7f3ba1c44700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c106d60 0x7f3b9c19c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51140/0 (socket says 172.21.15.29:51140) 2024-03-11T19:09:02.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.163+0000 7f3ba1c44700 1 -- 172.21.15.29:0/1518409655 learned_addr learned my addr 172.21.15.29:0/1518409655 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:02.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.163+0000 7f3ba0c42700 1 --2- 172.21.15.29:0/1518409655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b9c101810 0x7f3b9c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:02.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.163+0000 7f3ba1443700 1 --2- 172.21.15.29:0/1518409655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b9c100e50 0x7f3b9c1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:02.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.164+0000 7f3ba1c44700 1 -- 172.21.15.29:0/1518409655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b9c101810 msgr2=0x7f3b9c198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:02.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.164+0000 7f3ba1c44700 1 --2- 172.21.15.29:0/1518409655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b9c101810 0x7f3b9c198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:02.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.164+0000 7f3ba1c44700 1 -- 172.21.15.29:0/1518409655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b9c100e50 msgr2=0x7f3b9c1980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:02.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.164+0000 7f3ba1c44700 1 --2- 172.21.15.29:0/1518409655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b9c100e50 0x7f3b9c1980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:02.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.164+0000 7f3ba1c44700 1 -- 172.21.15.29:0/1518409655 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8c0095f0 con 0x7f3b9c106d60 2024-03-11T19:09:02.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.164+0000 7f3ba0c42700 1 --2- 172.21.15.29:0/1518409655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b9c101810 0x7f3b9c198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:02.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.164+0000 7f3ba1443700 1 --2- 172.21.15.29:0/1518409655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b9c100e50 0x7f3b9c1980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:02.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.164+0000 7f3ba1c44700 1 --2- 172.21.15.29:0/1518409655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c106d60 0x7f3b9c19c970 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3b94007f10 tx=0x7f3b9400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:02.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.164+0000 7f3b927fc700 1 -- 172.21.15.29:0/1518409655 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b94017070 con 0x7f3b9c106d60 2024-03-11T19:09:02.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.164+0000 7f3b927fc700 1 -- 172.21.15.29:0/1518409655 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3b9400f040 con 0x7f3b9c106d60 2024-03-11T19:09:02.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.164+0000 7f3ba36a7700 1 -- 172.21.15.29:0/1518409655 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b9c19d590 con 0x7f3b9c106d60 2024-03-11T19:09:02.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.164+0000 7f3b927fc700 1 -- 172.21.15.29:0/1518409655 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b94005730 con 0x7f3b9c106d60 2024-03-11T19:09:02.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.164+0000 7f3ba36a7700 1 -- 172.21.15.29:0/1518409655 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b9c1a4df0 con 0x7f3b9c106d60 2024-03-11T19:09:02.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.165+0000 7f3ba36a7700 1 -- 172.21.15.29:0/1518409655 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b9c1022f0 con 0x7f3b9c106d60 2024-03-11T19:09:02.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.166+0000 7f3b927fc700 1 -- 172.21.15.29:0/1518409655 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f3b94005890 con 0x7f3b9c106d60 2024-03-11T19:09:02.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.166+0000 7f3b927fc700 1 --2- 172.21.15.29:0/1518409655 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3b880a1b50 0x7f3b880a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:02.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.167+0000 7f3ba1443700 1 --2- 172.21.15.29:0/1518409655 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3b880a1b50 0x7f3b880a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:02.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.167+0000 7f3b927fc700 1 -- 172.21.15.29:0/1518409655 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f3b940ccd50 con 0x7f3b9c106d60 2024-03-11T19:09:02.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.167+0000 7f3ba1443700 1 --2- 172.21.15.29:0/1518409655 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3b880a1b50 0x7f3b880a4010 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3b98005e10 tx=0x7f3b98005d60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:02.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.171+0000 7f3b927fc700 1 -- 172.21.15.29:0/1518409655 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3b94095e00 con 0x7f3b9c106d60 2024-03-11T19:09:02.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:02 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1285211848' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T19:09:02.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.374+0000 7f3ba36a7700 1 -- 172.21.15.29:0/1518409655 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f3b9c068930 con 0x7f3b9c106d60 2024-03-11T19:09:02.378 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:02 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1285211848' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T19:09:02.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.375+0000 7f3b927fc700 1 -- 172.21.15.29:0/1518409655 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f3b940994c0 con 0x7f3b9c106d60 2024-03-11T19:09:02.378 INFO:teuthology.orchestra.run.smithi029.stdout:171798691899 2024-03-11T19:09:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.377+0000 7f3ba36a7700 1 -- 172.21.15.29:0/1518409655 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3b880a1b50 msgr2=0x7f3b880a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.377+0000 7f3ba36a7700 1 --2- 172.21.15.29:0/1518409655 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3b880a1b50 0x7f3b880a4010 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3b98005e10 tx=0x7f3b98005d60 comp rx=0 tx=0).stop 2024-03-11T19:09:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.377+0000 7f3ba36a7700 1 -- 172.21.15.29:0/1518409655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c106d60 msgr2=0x7f3b9c19c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.377+0000 7f3ba36a7700 1 --2- 172.21.15.29:0/1518409655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c106d60 0x7f3b9c19c970 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3b94007f10 tx=0x7f3b9400d3b0 comp rx=0 tx=0).stop 2024-03-11T19:09:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.377+0000 7f3ba36a7700 1 -- 172.21.15.29:0/1518409655 shutdown_connections 2024-03-11T19:09:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.377+0000 7f3ba36a7700 1 --2- 172.21.15.29:0/1518409655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c106d60 0x7f3b9c19c970 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.377+0000 7f3ba36a7700 1 --2- 172.21.15.29:0/1518409655 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3b880a1b50 0x7f3b880a4010 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.377+0000 7f3ba36a7700 1 --2- 172.21.15.29:0/1518409655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b9c100e50 0x7f3b9c1980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.377+0000 7f3ba36a7700 1 --2- 172.21.15.29:0/1518409655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b9c101810 0x7f3b9c198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.377+0000 7f3ba36a7700 1 -- 172.21.15.29:0/1518409655 >> 172.21.15.29:0/1518409655 conn(0x7f3b9c0779d0 msgr2=0x7f3b9c100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:02.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.377+0000 7f3ba36a7700 1 -- 172.21.15.29:0/1518409655 shutdown_connections 2024-03-11T19:09:02.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:02.377+0000 7f3ba36a7700 1 -- 172.21.15.29:0/1518409655 wait complete. 2024-03-11T19:09:02.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:02 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1285211848' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T19:09:03.153 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:03 smithi029 ceph-mon[32439]: pgmap v328: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:03.153 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:03 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1518409655' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T19:09:03.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:03 smithi170 ceph-mon[35690]: pgmap v328: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:03.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:03 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1518409655' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T19:09:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:03 smithi190 ceph-mon[36292]: pgmap v328: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:03 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1518409655' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T19:09:03.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.709+0000 7feff6f34700 1 -- 172.21.15.29:0/3563605033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feff0102680 msgr2=0x7feff010eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:03.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.709+0000 7feff6f34700 1 --2- 172.21.15.29:0/3563605033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feff0102680 0x7feff010eac0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fefe800b0d0 tx=0x7fefe800b3e0 comp rx=0 tx=0).stop 2024-03-11T19:09:03.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.709+0000 7feff6f34700 1 -- 172.21.15.29:0/3563605033 shutdown_connections 2024-03-11T19:09:03.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.709+0000 7feff6f34700 1 --2- 172.21.15.29:0/3563605033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feff0102680 0x7feff010eac0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:03.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.709+0000 7feff6f34700 1 --2- 172.21.15.29:0/3563605033 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feff0107ca0 0x7feff0108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:03.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.709+0000 7feff6f34700 1 --2- 172.21.15.29:0/3563605033 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feff0101cc0 0x7feff0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:03.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.709+0000 7feff6f34700 1 -- 172.21.15.29:0/3563605033 >> 172.21.15.29:0/3563605033 conn(0x7feff0078da0 msgr2=0x7feff00791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:03.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.709+0000 7feff6f34700 1 -- 172.21.15.29:0/3563605033 shutdown_connections 2024-03-11T19:09:03.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.709+0000 7feff6f34700 1 -- 172.21.15.29:0/3563605033 wait complete. 2024-03-11T19:09:03.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.709+0000 7feff6f34700 1 Processor -- start 2024-03-11T19:09:03.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.710+0000 7feff6f34700 1 -- start start 2024-03-11T19:09:03.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.710+0000 7feff6f34700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feff0101cc0 0x7feff01981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:03.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.710+0000 7feff6f34700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feff0102680 0x7feff0198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:03.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.710+0000 7feff6f34700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feff0107ca0 0x7feff019caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:03.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.710+0000 7feff6f34700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff019d0b0 con 0x7feff0102680 2024-03-11T19:09:03.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.710+0000 7feff6f34700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff019d220 con 0x7feff0107ca0 2024-03-11T19:09:03.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.710+0000 7feff6f34700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff019d3e0 con 0x7feff0101cc0 2024-03-11T19:09:03.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.710+0000 7fefeffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feff0102680 0x7feff0198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:03.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.710+0000 7fefeffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feff0102680 0x7feff0198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51170/0 (socket says 172.21.15.29:51170) 2024-03-11T19:09:03.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.710+0000 7fefeffff700 1 -- 172.21.15.29:0/994088881 learned_addr learned my addr 172.21.15.29:0/994088881 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:03.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.710+0000 7feff4cd0700 1 --2- 172.21.15.29:0/994088881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feff0101cc0 0x7feff01981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:03.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.710+0000 7feff54d1700 1 --2- 172.21.15.29:0/994088881 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feff0107ca0 0x7feff019caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:03.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.710+0000 7fefeffff700 1 -- 172.21.15.29:0/994088881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feff0101cc0 msgr2=0x7feff01981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:03.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.710+0000 7fefeffff700 1 --2- 172.21.15.29:0/994088881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feff0101cc0 0x7feff01981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:03.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.710+0000 7fefeffff700 1 -- 172.21.15.29:0/994088881 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feff0107ca0 msgr2=0x7feff019caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:03.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.710+0000 7fefeffff700 1 --2- 172.21.15.29:0/994088881 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feff0107ca0 0x7feff019caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:03.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.710+0000 7fefeffff700 1 -- 172.21.15.29:0/994088881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe8009d00 con 0x7feff0102680 2024-03-11T19:09:03.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.710+0000 7feff54d1700 1 --2- 172.21.15.29:0/994088881 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feff0107ca0 0x7feff019caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:03.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.711+0000 7fefeffff700 1 --2- 172.21.15.29:0/994088881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feff0102680 0x7feff0198710 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00e9a0 tx=0x7fefdc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:03.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.711+0000 7fefedffb700 1 -- 172.21.15.29:0/994088881 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefdc009790 con 0x7feff0102680 2024-03-11T19:09:03.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.711+0000 7feff6f34700 1 -- 172.21.15.29:0/994088881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff019d6c0 con 0x7feff0102680 2024-03-11T19:09:03.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.711+0000 7fefedffb700 1 -- 172.21.15.29:0/994088881 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fefdc00cbd0 con 0x7feff0102680 2024-03-11T19:09:03.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.711+0000 7fefedffb700 1 -- 172.21.15.29:0/994088881 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefdc018680 con 0x7feff0102680 2024-03-11T19:09:03.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.711+0000 7feff6f34700 1 -- 172.21.15.29:0/994088881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff01a4f20 con 0x7feff0102680 2024-03-11T19:09:03.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.712+0000 7feff6f34700 1 -- 172.21.15.29:0/994088881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff010a180 con 0x7feff0102680 2024-03-11T19:09:03.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.714+0000 7fefedffb700 1 -- 172.21.15.29:0/994088881 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fefdc00cd40 con 0x7feff0102680 2024-03-11T19:09:03.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.714+0000 7fefedffb700 1 --2- 172.21.15.29:0/994088881 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fefe00a1bd0 0x7fefe00a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:03.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.714+0000 7fefedffb700 1 -- 172.21.15.29:0/994088881 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fefdc014070 con 0x7feff0102680 2024-03-11T19:09:03.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.714+0000 7feff4cd0700 1 --2- 172.21.15.29:0/994088881 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fefe00a1bd0 0x7fefe00a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:03.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.715+0000 7feff4cd0700 1 --2- 172.21.15.29:0/994088881 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fefe00a1bd0 0x7fefe00a4090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fefe4005950 tx=0x7fefe400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:03.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.718+0000 7fefedffb700 1 -- 172.21.15.29:0/994088881 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fefdc096a30 con 0x7feff0102680 2024-03-11T19:09:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.926+0000 7feff6f34700 1 -- 172.21.15.29:0/994088881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7feff0111b60 con 0x7feff0102680 2024-03-11T19:09:03.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.927+0000 7fefedffb700 1 -- 172.21.15.29:0/994088881 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fefdc09a0f0 con 0x7feff0102680 2024-03-11T19:09:03.930 INFO:teuthology.orchestra.run.smithi029.stdout:317827579936 2024-03-11T19:09:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.929+0000 7feff6f34700 1 -- 172.21.15.29:0/994088881 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fefe00a1bd0 msgr2=0x7fefe00a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.929+0000 7feff6f34700 1 --2- 172.21.15.29:0/994088881 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fefe00a1bd0 0x7fefe00a4090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fefe4005950 tx=0x7fefe400b410 comp rx=0 tx=0).stop 2024-03-11T19:09:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.929+0000 7feff6f34700 1 -- 172.21.15.29:0/994088881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feff0102680 msgr2=0x7feff0198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.929+0000 7feff6f34700 1 --2- 172.21.15.29:0/994088881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feff0102680 0x7feff0198710 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00e9a0 tx=0x7fefdc00ecb0 comp rx=0 tx=0).stop 2024-03-11T19:09:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.929+0000 7feff6f34700 1 -- 172.21.15.29:0/994088881 shutdown_connections 2024-03-11T19:09:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.929+0000 7feff6f34700 1 --2- 172.21.15.29:0/994088881 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fefe00a1bd0 0x7fefe00a4090 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.929+0000 7feff6f34700 1 --2- 172.21.15.29:0/994088881 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feff0107ca0 0x7feff019caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.929+0000 7feff6f34700 1 --2- 172.21.15.29:0/994088881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feff0101cc0 0x7feff01981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.929+0000 7feff6f34700 1 --2- 172.21.15.29:0/994088881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feff0102680 0x7feff0198710 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.929+0000 7feff6f34700 1 -- 172.21.15.29:0/994088881 >> 172.21.15.29:0/994088881 conn(0x7feff0078da0 msgr2=0x7feff01009f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.929+0000 7feff6f34700 1 -- 172.21.15.29:0/994088881 shutdown_connections 2024-03-11T19:09:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:03.929+0000 7feff6f34700 1 -- 172.21.15.29:0/994088881 wait complete. 2024-03-11T19:09:04.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:04 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/994088881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-11T19:09:04.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:04 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/994088881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-11T19:09:04.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:04 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/994088881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-11T19:09:05.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:05 smithi170 ceph-mon[35690]: pgmap v329: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:09:05.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:05 smithi029 ceph-mon[32439]: pgmap v329: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:09:05.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:05 smithi190 ceph-mon[36292]: pgmap v329: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:09:05.908 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266732 got 227633266736 for osd.7 2024-03-11T19:09:05.908 DEBUG:teuthology.parallel:result is None 2024-03-11T19:09:05.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.485+0000 7fa38589c700 1 -- 172.21.15.29:0/2987259252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380101560 msgr2=0x7fa38010d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:05.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.485+0000 7fa38589c700 1 --2- 172.21.15.29:0/2987259252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380101560 0x7fa38010d9a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa37800b0a0 tx=0x7fa37800b3b0 comp rx=0 tx=0).stop 2024-03-11T19:09:05.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.485+0000 7fa38589c700 1 -- 172.21.15.29:0/2987259252 shutdown_connections 2024-03-11T19:09:05.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.485+0000 7fa38589c700 1 --2- 172.21.15.29:0/2987259252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380101560 0x7fa38010d9a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:05.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.485+0000 7fa38589c700 1 --2- 172.21.15.29:0/2987259252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa380106bd0 0x7fa380106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:05.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.485+0000 7fa38589c700 1 --2- 172.21.15.29:0/2987259252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa380100ba0 0x7fa380101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:05.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.485+0000 7fa38589c700 1 -- 172.21.15.29:0/2987259252 >> 172.21.15.29:0/2987259252 conn(0x7fa380076bc0 msgr2=0x7fa380076fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:05.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.485+0000 7fa38589c700 1 -- 172.21.15.29:0/2987259252 shutdown_connections 2024-03-11T19:09:05.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.485+0000 7fa38589c700 1 -- 172.21.15.29:0/2987259252 wait complete. 2024-03-11T19:09:05.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.486+0000 7fa38589c700 1 Processor -- start 2024-03-11T19:09:05.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.486+0000 7fa38589c700 1 -- start start 2024-03-11T19:09:05.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.486+0000 7fa38589c700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa380100ba0 0x7fa3801150f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:05.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.486+0000 7fa38589c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa380101560 0x7fa3801114c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:05.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.486+0000 7fa38589c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380106bd0 0x7fa380111a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:05.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.486+0000 7fa38589c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa380112060 con 0x7fa380106bd0 2024-03-11T19:09:05.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.486+0000 7fa38589c700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3801121d0 con 0x7fa380100ba0 2024-03-11T19:09:05.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.486+0000 7fa38589c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa380112390 con 0x7fa380101560 2024-03-11T19:09:05.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.486+0000 7fa37f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380106bd0 0x7fa380111a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:05.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.486+0000 7fa37f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380106bd0 0x7fa380111a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57574/0 (socket says 172.21.15.29:57574) 2024-03-11T19:09:05.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.486+0000 7fa37f7fe700 1 -- 172.21.15.29:0/2788113895 learned_addr learned my addr 172.21.15.29:0/2788113895 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:05.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.487+0000 7fa37f7fe700 1 -- 172.21.15.29:0/2788113895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa380101560 msgr2=0x7fa3801114c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:05.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.487+0000 7fa37f7fe700 1 --2- 172.21.15.29:0/2788113895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa380101560 0x7fa3801114c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:05.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.487+0000 7fa37effd700 1 --2- 172.21.15.29:0/2788113895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa380100ba0 0x7fa3801150f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:05.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.487+0000 7fa37f7fe700 1 -- 172.21.15.29:0/2788113895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa380100ba0 msgr2=0x7fa3801150f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:05.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.487+0000 7fa37f7fe700 1 --2- 172.21.15.29:0/2788113895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa380100ba0 0x7fa3801150f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:05.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.487+0000 7fa37f7fe700 1 -- 172.21.15.29:0/2788113895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa378009d00 con 0x7fa380106bd0 2024-03-11T19:09:05.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.487+0000 7fa37effd700 1 --2- 172.21.15.29:0/2788113895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa380100ba0 0x7fa3801150f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:05.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.487+0000 7fa37f7fe700 1 --2- 172.21.15.29:0/2788113895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380106bd0 0x7fa380111a00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa37800b830 tx=0x7fa378009430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:05.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.487+0000 7fa38489a700 1 -- 172.21.15.29:0/2788113895 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37800e040 con 0x7fa380106bd0 2024-03-11T19:09:05.995 INFO:teuthology.orchestra.run.smithi029.stdout:261993005100 2024-03-11T19:09:05.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.487+0000 7fa38489a700 1 -- 172.21.15.29:0/2788113895 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa378004e60 con 0x7fa380106bd0 2024-03-11T19:09:05.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.487+0000 7fa38589c700 1 -- 172.21.15.29:0/2788113895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3801aaf80 con 0x7fa380106bd0 2024-03-11T19:09:05.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.487+0000 7fa38489a700 1 -- 172.21.15.29:0/2788113895 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa378005460 con 0x7fa380106bd0 2024-03-11T19:09:05.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.487+0000 7fa38589c700 1 -- 172.21.15.29:0/2788113895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3801ab400 con 0x7fa380106bd0 2024-03-11T19:09:05.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.488+0000 7fa38589c700 1 -- 172.21.15.29:0/2788113895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa380068930 con 0x7fa380106bd0 2024-03-11T19:09:05.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.489+0000 7fa38489a700 1 -- 172.21.15.29:0/2788113895 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fa37801bbb0 con 0x7fa380106bd0 2024-03-11T19:09:05.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.490+0000 7fa38489a700 1 --2- 172.21.15.29:0/2788113895 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa3680a1b50 0x7fa3680a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:05.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.490+0000 7fa37effd700 1 --2- 172.21.15.29:0/2788113895 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa3680a1b50 0x7fa3680a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:05.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.490+0000 7fa38489a700 1 -- 172.21.15.29:0/2788113895 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fa3780cf3b0 con 0x7fa380106bd0 2024-03-11T19:09:05.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.490+0000 7fa37effd700 1 --2- 172.21.15.29:0/2788113895 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa3680a1b50 0x7fa3680a4010 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa370007900 tx=0x7fa370008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:05.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.494+0000 7fa38489a700 1 -- 172.21.15.29:0/2788113895 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa3780984f0 con 0x7fa380106bd0 2024-03-11T19:09:05.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.698+0000 7fa38589c700 1 -- 172.21.15.29:0/2788113895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7fa38010fa00 con 0x7fa380106bd0 2024-03-11T19:09:05.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.698+0000 7fa38489a700 1 -- 172.21.15.29:0/2788113895 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa37809bbb0 con 0x7fa380106bd0 2024-03-11T19:09:05.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.700+0000 7fa38589c700 1 -- 172.21.15.29:0/2788113895 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa3680a1b50 msgr2=0x7fa3680a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:05.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.700+0000 7fa38589c700 1 --2- 172.21.15.29:0/2788113895 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa3680a1b50 0x7fa3680a4010 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa370007900 tx=0x7fa370008040 comp rx=0 tx=0).stop 2024-03-11T19:09:05.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.700+0000 7fa38589c700 1 -- 172.21.15.29:0/2788113895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380106bd0 msgr2=0x7fa380111a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:05.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.700+0000 7fa38589c700 1 --2- 172.21.15.29:0/2788113895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380106bd0 0x7fa380111a00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa37800b830 tx=0x7fa378009430 comp rx=0 tx=0).stop 2024-03-11T19:09:05.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.700+0000 7fa38589c700 1 -- 172.21.15.29:0/2788113895 shutdown_connections 2024-03-11T19:09:05.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.700+0000 7fa38589c700 1 --2- 172.21.15.29:0/2788113895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa380106bd0 0x7fa380111a00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:05.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.700+0000 7fa38589c700 1 --2- 172.21.15.29:0/2788113895 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa3680a1b50 0x7fa3680a4010 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:05.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.700+0000 7fa38589c700 1 --2- 172.21.15.29:0/2788113895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa380100ba0 0x7fa3801150f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:05.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.700+0000 7fa38589c700 1 --2- 172.21.15.29:0/2788113895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa380101560 0x7fa3801114c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:05.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.700+0000 7fa38589c700 1 -- 172.21.15.29:0/2788113895 >> 172.21.15.29:0/2788113895 conn(0x7fa380076bc0 msgr2=0x7fa3800febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:05.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.700+0000 7fa38589c700 1 -- 172.21.15.29:0/2788113895 shutdown_connections 2024-03-11T19:09:05.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:05.700+0000 7fa38589c700 1 -- 172.21.15.29:0/2788113895 wait complete. 2024-03-11T19:09:06.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:06 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2788113895' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-11T19:09:06.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:06 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2788113895' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-11T19:09:06.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:06 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2788113895' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-11T19:09:06.798 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691891 got 171798691899 for osd.5 2024-03-11T19:09:06.799 DEBUG:teuthology.parallel:result is None 2024-03-11T19:09:07.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:07 smithi170 ceph-mon[35690]: pgmap v330: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:07.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:07 smithi029 ceph-mon[32439]: pgmap v330: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:07.443 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579929 got 317827579936 for osd.10 2024-03-11T19:09:07.443 DEBUG:teuthology.parallel:result is None 2024-03-11T19:09:07.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:07 smithi190 ceph-mon[36292]: pgmap v330: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:08.130 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005095 got 261993005100 for osd.8 2024-03-11T19:09:08.131 DEBUG:teuthology.parallel:result is None 2024-03-11T19:09:08.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.354+0000 7f0b63227700 1 -- 172.21.15.29:0/1946032687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5c06adc0 msgr2=0x7f0b5c1068a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:08.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.354+0000 7f0b63227700 1 --2- 172.21.15.29:0/1946032687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5c06adc0 0x7f0b5c1068a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009990 tx=0x7f0b50009ca0 comp rx=0 tx=0).stop 2024-03-11T19:09:08.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.354+0000 7f0b63227700 1 -- 172.21.15.29:0/1946032687 shutdown_connections 2024-03-11T19:09:08.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.354+0000 7f0b63227700 1 --2- 172.21.15.29:0/1946032687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b5c106f80 0x7f0b5c10eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:08.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.354+0000 7f0b63227700 1 --2- 172.21.15.29:0/1946032687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b5c06a410 0x7f0b5c06a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:08.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.354+0000 7f0b63227700 1 --2- 172.21.15.29:0/1946032687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5c06adc0 0x7f0b5c1068a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:08.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.354+0000 7f0b63227700 1 -- 172.21.15.29:0/1946032687 >> 172.21.15.29:0/1946032687 conn(0x7f0b5c076bc0 msgr2=0x7f0b5c076fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:08.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.354+0000 7f0b63227700 1 -- 172.21.15.29:0/1946032687 shutdown_connections 2024-03-11T19:09:08.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.354+0000 7f0b63227700 1 -- 172.21.15.29:0/1946032687 wait complete. 2024-03-11T19:09:08.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.354+0000 7f0b63227700 1 Processor -- start 2024-03-11T19:09:08.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.354+0000 7f0b63227700 1 -- start start 2024-03-11T19:09:08.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.355+0000 7f0b63227700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b5c06a410 0x7f0b5c100c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:08.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.355+0000 7f0b63227700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b5c06adc0 0x7f0b5c1011b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:08.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.355+0000 7f0b63227700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5c106f80 0x7f0b5c1016f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:08.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.355+0000 7f0b63227700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c1a64a0 con 0x7f0b5c106f80 2024-03-11T19:09:08.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.355+0000 7f0b63227700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c1a6610 con 0x7f0b5c06a410 2024-03-11T19:09:08.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.355+0000 7f0b63227700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c1a67d0 con 0x7f0b5c06adc0 2024-03-11T19:09:08.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.355+0000 7f0b617c4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5c106f80 0x7f0b5c1016f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:08.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.355+0000 7f0b617c4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5c106f80 0x7f0b5c1016f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57592/0 (socket says 172.21.15.29:57592) 2024-03-11T19:09:08.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.355+0000 7f0b617c4700 1 -- 172.21.15.29:0/2471845388 learned_addr learned my addr 172.21.15.29:0/2471845388 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:08.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.355+0000 7f0b60fc3700 1 --2- 172.21.15.29:0/2471845388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b5c06a410 0x7f0b5c100c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:08.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.355+0000 7f0b5bfff700 1 --2- 172.21.15.29:0/2471845388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b5c06adc0 0x7f0b5c1011b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:08.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.355+0000 7f0b617c4700 1 -- 172.21.15.29:0/2471845388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b5c06adc0 msgr2=0x7f0b5c1011b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:08.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.355+0000 7f0b617c4700 1 --2- 172.21.15.29:0/2471845388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b5c06adc0 0x7f0b5c1011b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:08.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.355+0000 7f0b617c4700 1 -- 172.21.15.29:0/2471845388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b5c06a410 msgr2=0x7f0b5c100c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:08.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.355+0000 7f0b617c4700 1 --2- 172.21.15.29:0/2471845388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b5c06a410 0x7f0b5c100c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:08.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.355+0000 7f0b617c4700 1 -- 172.21.15.29:0/2471845388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b500095f0 con 0x7f0b5c106f80 2024-03-11T19:09:08.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.355+0000 7f0b617c4700 1 --2- 172.21.15.29:0/2471845388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5c106f80 0x7f0b5c1016f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0b54007ee0 tx=0x7f0b5400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.355+0000 7f0b59ffb700 1 -- 172.21.15.29:0/2471845388 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5400dd20 con 0x7f0b5c106f80 2024-03-11T19:09:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.356+0000 7f0b63227700 1 -- 172.21.15.29:0/2471845388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b5c1a6ab0 con 0x7f0b5c106f80 2024-03-11T19:09:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.356+0000 7f0b59ffb700 1 -- 172.21.15.29:0/2471845388 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0b5400f040 con 0x7f0b5c106f80 2024-03-11T19:09:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.356+0000 7f0b59ffb700 1 -- 172.21.15.29:0/2471845388 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b54004840 con 0x7f0b5c106f80 2024-03-11T19:09:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.356+0000 7f0b63227700 1 -- 172.21.15.29:0/2471845388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b5c1a6f80 con 0x7f0b5c106f80 2024-03-11T19:09:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.357+0000 7f0b63227700 1 -- 172.21.15.29:0/2471845388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b5c10a1b0 con 0x7f0b5c106f80 2024-03-11T19:09:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.358+0000 7f0b59ffb700 1 -- 172.21.15.29:0/2471845388 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f0b54003a00 con 0x7f0b5c106f80 2024-03-11T19:09:08.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.359+0000 7f0b59ffb700 1 --2- 172.21.15.29:0/2471845388 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0b440a1bd0 0x7f0b440a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:08.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.359+0000 7f0b60fc3700 1 --2- 172.21.15.29:0/2471845388 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0b440a1bd0 0x7f0b440a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:08.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.359+0000 7f0b59ffb700 1 -- 172.21.15.29:0/2471845388 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0b540cccb0 con 0x7f0b5c106f80 2024-03-11T19:09:08.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.359+0000 7f0b60fc3700 1 --2- 172.21.15.29:0/2471845388 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0b440a1bd0 0x7f0b440a4090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c009c60 tx=0x7f0b4c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:08.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.363+0000 7f0b59ffb700 1 -- 172.21.15.29:0/2471845388 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0b54095d60 con 0x7f0b5c106f80 2024-03-11T19:09:08.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.585+0000 7f0b63227700 1 -- 172.21.15.29:0/2471845388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f0b5c111b90 con 0x7f0b5c106f80 2024-03-11T19:09:08.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.586+0000 7f0b59ffb700 1 -- 172.21.15.29:0/2471845388 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0b54099420 con 0x7f0b5c106f80 2024-03-11T19:09:08.589 INFO:teuthology.orchestra.run.smithi029.stdout:55834574927 2024-03-11T19:09:08.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.588+0000 7f0b63227700 1 -- 172.21.15.29:0/2471845388 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0b440a1bd0 msgr2=0x7f0b440a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:08.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.588+0000 7f0b63227700 1 --2- 172.21.15.29:0/2471845388 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0b440a1bd0 0x7f0b440a4090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c009c60 tx=0x7f0b4c009260 comp rx=0 tx=0).stop 2024-03-11T19:09:08.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.588+0000 7f0b63227700 1 -- 172.21.15.29:0/2471845388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5c106f80 msgr2=0x7f0b5c1016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:08.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.588+0000 7f0b63227700 1 --2- 172.21.15.29:0/2471845388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5c106f80 0x7f0b5c1016f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0b54007ee0 tx=0x7f0b5400d3b0 comp rx=0 tx=0).stop 2024-03-11T19:09:08.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.588+0000 7f0b63227700 1 -- 172.21.15.29:0/2471845388 shutdown_connections 2024-03-11T19:09:08.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.588+0000 7f0b63227700 1 --2- 172.21.15.29:0/2471845388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5c106f80 0x7f0b5c1016f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:08.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.588+0000 7f0b63227700 1 --2- 172.21.15.29:0/2471845388 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0b440a1bd0 0x7f0b440a4090 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:08.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.588+0000 7f0b63227700 1 --2- 172.21.15.29:0/2471845388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b5c06a410 0x7f0b5c100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:08.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.588+0000 7f0b63227700 1 --2- 172.21.15.29:0/2471845388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b5c06adc0 0x7f0b5c1011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:08.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.588+0000 7f0b63227700 1 -- 172.21.15.29:0/2471845388 >> 172.21.15.29:0/2471845388 conn(0x7f0b5c076bc0 msgr2=0x7f0b5c105b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:08.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.588+0000 7f0b63227700 1 -- 172.21.15.29:0/2471845388 shutdown_connections 2024-03-11T19:09:08.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:08.588+0000 7f0b63227700 1 -- 172.21.15.29:0/2471845388 wait complete. 2024-03-11T19:09:09.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:09 smithi170 ceph-mon[35690]: pgmap v331: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:09:09.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:09 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2471845388' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T19:09:09.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:09 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:09:09.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:09 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:09:09.361 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:09 smithi029 ceph-mon[32439]: pgmap v331: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:09:09.361 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:09 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2471845388' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T19:09:09.361 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:09 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:09:09.361 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:09 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:09:09.387 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574920 got 55834574927 for osd.1 2024-03-11T19:09:09.387 DEBUG:teuthology.parallel:result is None 2024-03-11T19:09:09.387 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T19:09:09.387 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T19:09:09.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:09 smithi190 ceph-mon[36292]: pgmap v331: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:09:09.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:09 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2471845388' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T19:09:09.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:09 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:09:09.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:09 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:09:09.643 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:09:10.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:10 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:09:10.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:10 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:09:10.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:10 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:09:10.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:10 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:09:10.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:10 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:09:10.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:10 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:09:10.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:10 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:09:10.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:10 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:09:10.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:10 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:09:11.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:11 smithi170 ceph-mon[35690]: pgmap v332: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:11.360 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:11 smithi029 ceph-mon[32439]: pgmap v332: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:11.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.359+0000 7f6349ca4700 1 -- 172.21.15.29:0/594698299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6344100e00 msgr2=0x7f63441011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:11.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.359+0000 7f6349ca4700 1 --2- 172.21.15.29:0/594698299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6344100e00 0x7f63441011e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6334009990 tx=0x7f6334009ca0 comp rx=0 tx=0).stop 2024-03-11T19:09:11.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.359+0000 7f6349ca4700 1 -- 172.21.15.29:0/594698299 shutdown_connections 2024-03-11T19:09:11.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.359+0000 7f6349ca4700 1 --2- 172.21.15.29:0/594698299 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6344105e90 0x7f634410da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:11.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.359+0000 7f6349ca4700 1 --2- 172.21.15.29:0/594698299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6344100e00 0x7f63441011e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:11.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.359+0000 7f6349ca4700 1 --2- 172.21.15.29:0/594698299 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63441017b0 0x7f63441057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:11.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.359+0000 7f6349ca4700 1 -- 172.21.15.29:0/594698299 >> 172.21.15.29:0/594698299 conn(0x7f6344076b90 msgr2=0x7f6344076fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:11.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.359+0000 7f6349ca4700 1 -- 172.21.15.29:0/594698299 shutdown_connections 2024-03-11T19:09:11.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.359+0000 7f6349ca4700 1 -- 172.21.15.29:0/594698299 wait complete. 2024-03-11T19:09:11.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.359+0000 7f6349ca4700 1 Processor -- start 2024-03-11T19:09:11.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f6349ca4700 1 -- start start 2024-03-11T19:09:11.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f6349ca4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6344100e00 0x7f6344112d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:11.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f6349ca4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63441017b0 0x7f63441113c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:11.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f6349ca4700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6344105e90 0x7f6344111900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:11.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f6349ca4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6344111fe0 con 0x7f6344100e00 2024-03-11T19:09:11.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f6349ca4700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6344112150 con 0x7f6344105e90 2024-03-11T19:09:11.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f6349ca4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63441a6760 con 0x7f63441017b0 2024-03-11T19:09:11.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f63437fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6344100e00 0x7f6344112d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:11.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f63437fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6344100e00 0x7f6344112d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57614/0 (socket says 172.21.15.29:57614) 2024-03-11T19:09:11.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f63437fe700 1 -- 172.21.15.29:0/3554821542 learned_addr learned my addr 172.21.15.29:0/3554821542 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:11.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f6343fff700 1 --2- 172.21.15.29:0/3554821542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6344105e90 0x7f6344111900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:11.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f6342ffd700 1 --2- 172.21.15.29:0/3554821542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63441017b0 0x7f63441113c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:11.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f63437fe700 1 -- 172.21.15.29:0/3554821542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63441017b0 msgr2=0x7f63441113c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:11.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f63437fe700 1 --2- 172.21.15.29:0/3554821542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63441017b0 0x7f63441113c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:11.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f63437fe700 1 -- 172.21.15.29:0/3554821542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6344105e90 msgr2=0x7f6344111900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:11.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f63437fe700 1 --2- 172.21.15.29:0/3554821542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6344105e90 0x7f6344111900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:11.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f63437fe700 1 -- 172.21.15.29:0/3554821542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63340095f0 con 0x7f6344100e00 2024-03-11T19:09:11.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f6342ffd700 1 --2- 172.21.15.29:0/3554821542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63441017b0 0x7f63441113c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:11.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f6343fff700 1 --2- 172.21.15.29:0/3554821542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6344105e90 0x7f6344111900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:11.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.360+0000 7f63437fe700 1 --2- 172.21.15.29:0/3554821542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6344100e00 0x7f6344112d90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6334000c00 tx=0x7f633400f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:11.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.361+0000 7f6340ff9700 1 -- 172.21.15.29:0/3554821542 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633401d070 con 0x7f6344100e00 2024-03-11T19:09:11.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.361+0000 7f6340ff9700 1 -- 172.21.15.29:0/3554821542 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6334022470 con 0x7f6344100e00 2024-03-11T19:09:11.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.361+0000 7f6349ca4700 1 -- 172.21.15.29:0/3554821542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63441a6960 con 0x7f6344100e00 2024-03-11T19:09:11.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.361+0000 7f6340ff9700 1 -- 172.21.15.29:0/3554821542 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6334003a20 con 0x7f6344100e00 2024-03-11T19:09:11.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.361+0000 7f6349ca4700 1 -- 172.21.15.29:0/3554821542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63441a6d70 con 0x7f6344100e00 2024-03-11T19:09:11.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.362+0000 7f6349ca4700 1 -- 172.21.15.29:0/3554821542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6344068930 con 0x7f6344100e00 2024-03-11T19:09:11.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.363+0000 7f6340ff9700 1 -- 172.21.15.29:0/3554821542 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f63340225e0 con 0x7f6344100e00 2024-03-11T19:09:11.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.363+0000 7f6340ff9700 1 --2- 172.21.15.29:0/3554821542 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f632c0a1b50 0x7f632c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:11.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.363+0000 7f6340ff9700 1 -- 172.21.15.29:0/3554821542 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f63340ce840 con 0x7f6344100e00 2024-03-11T19:09:11.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.363+0000 7f6342ffd700 1 --2- 172.21.15.29:0/3554821542 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f632c0a1b50 0x7f632c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:11.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.364+0000 7f6342ffd700 1 --2- 172.21.15.29:0/3554821542 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f632c0a1b50 0x7f632c0a4010 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6338005fd0 tx=0x7f6338005e20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:11.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.368+0000 7f6340ff9700 1 -- 172.21.15.29:0/3554821542 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6334097a80 con 0x7f6344100e00 2024-03-11T19:09:11.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:11 smithi190 ceph-mon[36292]: pgmap v332: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:11.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.568+0000 7f6349ca4700 1 -- 172.21.15.29:0/3554821542 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6344110b60 con 0x7f632c0a1b50 2024-03-11T19:09:11.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.575+0000 7f6340ff9700 1 -- 172.21.15.29:0/3554821542 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258964 (secure 0 0 0) 0x7f6344110b60 con 0x7f632c0a1b50 2024-03-11T19:09:11.585 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:11.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.578+0000 7f6349ca4700 1 -- 172.21.15.29:0/3554821542 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f632c0a1b50 msgr2=0x7f632c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:11.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.578+0000 7f6349ca4700 1 --2- 172.21.15.29:0/3554821542 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f632c0a1b50 0x7f632c0a4010 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6338005fd0 tx=0x7f6338005e20 comp rx=0 tx=0).stop 2024-03-11T19:09:11.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.578+0000 7f6349ca4700 1 -- 172.21.15.29:0/3554821542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6344100e00 msgr2=0x7f6344112d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:11.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.578+0000 7f6349ca4700 1 --2- 172.21.15.29:0/3554821542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6344100e00 0x7f6344112d90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6334000c00 tx=0x7f633400f510 comp rx=0 tx=0).stop 2024-03-11T19:09:11.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.578+0000 7f6349ca4700 1 -- 172.21.15.29:0/3554821542 shutdown_connections 2024-03-11T19:09:11.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.578+0000 7f6349ca4700 1 --2- 172.21.15.29:0/3554821542 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f632c0a1b50 0x7f632c0a4010 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:11.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.578+0000 7f6349ca4700 1 --2- 172.21.15.29:0/3554821542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6344105e90 0x7f6344111900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:11.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.578+0000 7f6349ca4700 1 --2- 172.21.15.29:0/3554821542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6344100e00 0x7f6344112d90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:11.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.578+0000 7f6349ca4700 1 --2- 172.21.15.29:0/3554821542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63441017b0 0x7f63441113c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:11.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.578+0000 7f6349ca4700 1 -- 172.21.15.29:0/3554821542 >> 172.21.15.29:0/3554821542 conn(0x7f6344076b90 msgr2=0x7f6344104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:11.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.579+0000 7f6349ca4700 1 -- 172.21.15.29:0/3554821542 shutdown_connections 2024-03-11T19:09:11.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:11.579+0000 7f6349ca4700 1 -- 172.21.15.29:0/3554821542 wait complete. 2024-03-11T19:09:11.588 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-11T19:09:12.279 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:12 smithi029 ceph-mon[32439]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:09:12.309 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-03-11T19:09:09.623807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658461,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":875,"num_read_kb":557,"num_write":420,"num_write_kb":3073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":345,"ondisk_log_size":345,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":394452,"kb_used_data":11940,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124547372,"statfs":{"total":1151940427776,"available":1151536508928,"internally_reserved":0,"allocated":12226560,"data_stored":7531119,"data_compressed":112974,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":391492922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":120,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003025"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.988138+0000","last_change":"2024-03-11T19:07:49.751738+0000","last_active":"2024-03-11T19:08:44.988138+0000","last_peered":"2024-03-11T19:08:44.988138+0000","last_clean":"2024-03-11T19:08:44.988138+0000","last_became_active":"2024-03-11T19:07:49.751173+0000","last_became_peered":"2024-03-11T19:07:49.751173+0000","last_unstale":"2024-03-11T19:08:44.988138+0000","last_undegraded":"2024-03-11T19:08:44.988138+0000","last_fullsized":"2024-03-11T19:08:44.988138+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:13:40.157763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991611+0000","last_change":"2024-03-11T19:07:49.749069+0000","last_active":"2024-03-11T19:08:44.991611+0000","last_peered":"2024-03-11T19:08:44.991611+0000","last_clean":"2024-03-11T19:08:44.991611+0000","last_became_active":"2024-03-11T19:07:49.747672+0000","last_became_peered":"2024-03-11T19:07:49.747672+0000","last_unstale":"2024-03-11T19:08:44.991611+0000","last_undegraded":"2024-03-11T19:08:44.991611+0000","last_fullsized":"2024-03-11T19:08:44.991611+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:57:26.379836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990986+0000","last_change":"2024-03-11T19:07:49.750298+0000","last_active":"2024-03-11T19:08:44.990986+0000","last_peered":"2024-03-11T19:08:44.990986+0000","last_clean":"2024-03-11T19:08:44.990986+0000","last_became_active":"2024-03-11T19:07:49.749985+0000","last_became_peered":"2024-03-11T19:07:49.749985+0000","last_unstale":"2024-03-11T19:08:44.990986+0000","last_undegraded":"2024-03-11T19:08:44.990986+0000","last_fullsized":"2024-03-11T19:08:44.990986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:56.579872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.993461+0000","last_change":"2024-03-11T19:08:44.993047+0000","last_active":"2024-03-11T19:08:44.993461+0000","last_peered":"2024-03-11T19:08:44.993461+0000","last_clean":"2024-03-11T19:08:44.993461+0000","last_became_active":"2024-03-11T19:08:44.992666+0000","last_became_peered":"2024-03-11T19:08:44.992666+0000","last_unstale":"2024-03-11T19:08:44.993461+0000","last_undegraded":"2024-03-11T19:08:44.993461+0000","last_fullsized":"2024-03-11T19:08:44.993461+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:32:18.748906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990386+0000","last_change":"2024-03-11T19:07:49.747618+0000","last_active":"2024-03-11T19:08:44.990386+0000","last_peered":"2024-03-11T19:08:44.990386+0000","last_clean":"2024-03-11T19:08:44.990386+0000","last_became_active":"2024-03-11T19:07:49.747310+0000","last_became_peered":"2024-03-11T19:07:49.747310+0000","last_unstale":"2024-03-11T19:08:44.990386+0000","last_undegraded":"2024-03-11T19:08:44.990386+0000","last_fullsized":"2024-03-11T19:08:44.990386+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:15:42.260509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037372+0000","last_change":"2024-03-11T19:07:49.755564+0000","last_active":"2024-03-11T19:08:45.037372+0000","last_peered":"2024-03-11T19:08:45.037372+0000","last_clean":"2024-03-11T19:08:45.037372+0000","last_became_active":"2024-03-11T19:07:49.751322+0000","last_became_peered":"2024-03-11T19:07:49.751322+0000","last_unstale":"2024-03-11T19:08:45.037372+0000","last_undegraded":"2024-03-11T19:08:45.037372+0000","last_fullsized":"2024-03-11T19:08:45.037372+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:43:18.487887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985513+0000","last_change":"2024-03-11T19:07:49.731619+0000","last_active":"2024-03-11T19:08:44.985513+0000","last_peered":"2024-03-11T19:08:44.985513+0000","last_clean":"2024-03-11T19:08:44.985513+0000","last_became_active":"2024-03-11T19:07:49.731323+0000","last_became_peered":"2024-03-11T19:07:49.731323+0000","last_unstale":"2024-03-11T19:08:44.985513+0000","last_undegraded":"2024-03-11T19:08:44.985513+0000","last_fullsized":"2024-03-11T19:08:44.985513+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:20:25.592709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.989401+0000","last_change":"2024-03-11T19:07:49.751851+0000","last_active":"2024-03-11T19:08:44.989401+0000","last_peered":"2024-03-11T19:08:44.989401+0000","last_clean":"2024-03-11T19:08:44.989401+0000","last_became_active":"2024-03-11T19:07:49.750068+0000","last_became_peered":"2024-03-11T19:07:49.750068+0000","last_unstale":"2024-03-11T19:08:44.989401+0000","last_undegraded":"2024-03-11T19:08:44.989401+0000","last_fullsized":"2024-03-11T19:08:44.989401+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:59:35.820107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037172+0000","last_change":"2024-03-11T19:07:49.752326+0000","last_active":"2024-03-11T19:08:45.037172+0000","last_peered":"2024-03-11T19:08:45.037172+0000","last_clean":"2024-03-11T19:08:45.037172+0000","last_became_active":"2024-03-11T19:07:49.748052+0000","last_became_peered":"2024-03-11T19:07:49.748052+0000","last_unstale":"2024-03-11T19:08:45.037172+0000","last_undegraded":"2024-03-11T19:08:45.037172+0000","last_fullsized":"2024-03-11T19:08:45.037172+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:58:13.325964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.038017+0000","last_change":"2024-03-11T19:07:49.748054+0000","last_active":"2024-03-11T19:08:45.038017+0000","last_peered":"2024-03-11T19:08:45.038017+0000","last_clean":"2024-03-11T19:08:45.038017+0000","last_became_active":"2024-03-11T19:07:49.747765+0000","last_became_peered":"2024-03-11T19:07:49.747765+0000","last_unstale":"2024-03-11T19:08:45.038017+0000","last_undegraded":"2024-03-11T19:08:45.038017+0000","last_fullsized":"2024-03-11T19:08:45.038017+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:18:05.599512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985983+0000","last_change":"2024-03-11T19:07:49.751622+0000","last_active":"2024-03-11T19:08:44.985983+0000","last_peered":"2024-03-11T19:08:44.985983+0000","last_clean":"2024-03-11T19:08:44.985983+0000","last_became_active":"2024-03-11T19:07:49.751317+0000","last_became_peered":"2024-03-11T19:07:49.751317+0000","last_unstale":"2024-03-11T19:08:44.985983+0000","last_undegraded":"2024-03-11T19:08:44.985983+0000","last_fullsized":"2024-03-11T19:08:44.985983+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:49.836096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037533+0000","last_change":"2024-03-11T19:07:49.749160+0000","last_active":"2024-03-11T19:08:45.037533+0000","last_peered":"2024-03-11T19:08:45.037533+0000","last_clean":"2024-03-11T19:08:45.037533+0000","last_became_active":"2024-03-11T19:07:49.748838+0000","last_became_peered":"2024-03-11T19:07:49.748838+0000","last_unstale":"2024-03-11T19:08:45.037533+0000","last_undegraded":"2024-03-11T19:08:45.037533+0000","last_fullsized":"2024-03-11T19:08:45.037533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:30:58.327196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.988839+0000","last_change":"2024-03-11T19:07:49.749965+0000","last_active":"2024-03-11T19:08:44.988839+0000","last_peered":"2024-03-11T19:08:44.988839+0000","last_clean":"2024-03-11T19:08:44.988839+0000","last_became_active":"2024-03-11T19:07:49.749602+0000","last_became_peered":"2024-03-11T19:07:49.749602+0000","last_unstale":"2024-03-11T19:08:44.988839+0000","last_undegraded":"2024-03-11T19:08:44.988839+0000","last_fullsized":"2024-03-11T19:08:44.988839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:36:08.807251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990351+0000","last_change":"2024-03-11T19:07:49.747557+0000","last_active":"2024-03-11T19:08:44.990351+0000","last_peered":"2024-03-11T19:08:44.990351+0000","last_clean":"2024-03-11T19:08:44.990351+0000","last_became_active":"2024-03-11T19:07:49.745804+0000","last_became_peered":"2024-03-11T19:07:49.745804+0000","last_unstale":"2024-03-11T19:08:44.990351+0000","last_undegraded":"2024-03-11T19:08:44.990351+0000","last_fullsized":"2024-03-11T19:08:44.990351+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:07:26.939414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990225+0000","last_change":"2024-03-11T19:07:49.744070+0000","last_active":"2024-03-11T19:08:44.990225+0000","last_peered":"2024-03-11T19:08:44.990225+0000","last_clean":"2024-03-11T19:08:44.990225+0000","last_became_active":"2024-03-11T19:07:49.743736+0000","last_became_peered":"2024-03-11T19:07:49.743736+0000","last_unstale":"2024-03-11T19:08:44.990225+0000","last_undegraded":"2024-03-11T19:08:44.990225+0000","last_fullsized":"2024-03-11T19:08:44.990225+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:50:33.862644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.035758+0000","last_change":"2024-03-11T19:07:49.748512+0000","last_active":"2024-03-11T19:08:45.035758+0000","last_peered":"2024-03-11T19:08:45.035758+0000","last_clean":"2024-03-11T19:08:45.035758+0000","last_became_active":"2024-03-11T19:07:49.748223+0000","last_became_peered":"2024-03-11T19:07:49.748223+0000","last_unstale":"2024-03-11T19:08:45.035758+0000","last_undegraded":"2024-03-11T19:08:45.035758+0000","last_fullsized":"2024-03-11T19:08:45.035758+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:49:43.305677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.988285+0000","last_change":"2024-03-11T19:07:49.754322+0000","last_active":"2024-03-11T19:08:44.988285+0000","last_peered":"2024-03-11T19:08:44.988285+0000","last_clean":"2024-03-11T19:08:44.988285+0000","last_became_active":"2024-03-11T19:07:49.753757+0000","last_became_peered":"2024-03-11T19:07:49.753757+0000","last_unstale":"2024-03-11T19:08:44.988285+0000","last_undegraded":"2024-03-11T19:08:44.988285+0000","last_fullsized":"2024-03-11T19:08:44.988285+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:49.684978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985342+0000","last_change":"2024-03-11T19:07:49.749546+0000","last_active":"2024-03-11T19:08:44.985342+0000","last_peered":"2024-03-11T19:08:44.985342+0000","last_clean":"2024-03-11T19:08:44.985342+0000","last_became_active":"2024-03-11T19:07:49.749200+0000","last_became_peered":"2024-03-11T19:07:49.749200+0000","last_unstale":"2024-03-11T19:08:44.985342+0000","last_undegraded":"2024-03-11T19:08:44.985342+0000","last_fullsized":"2024-03-11T19:08:44.985342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:26:12.914015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.986039+0000","last_change":"2024-03-11T19:07:49.750388+0000","last_active":"2024-03-11T19:08:44.986039+0000","last_peered":"2024-03-11T19:08:44.986039+0000","last_clean":"2024-03-11T19:08:44.986039+0000","last_became_active":"2024-03-11T19:07:49.750016+0000","last_became_peered":"2024-03-11T19:07:49.750016+0000","last_unstale":"2024-03-11T19:08:44.986039+0000","last_undegraded":"2024-03-11T19:08:44.986039+0000","last_fullsized":"2024-03-11T19:08:44.986039+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:32:23.125695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'13","reported_seq":103,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:04.764161+0000","last_change":"2024-03-11T19:07:48.738369+0000","last_active":"2024-03-11T19:09:04.764161+0000","last_peered":"2024-03-11T19:09:04.764161+0000","last_clean":"2024-03-11T19:09:04.764161+0000","last_became_active":"2024-03-11T19:07:48.738010+0000","last_became_peered":"2024-03-11T19:07:48.738010+0000","last_unstale":"2024-03-11T19:09:04.764161+0000","last_undegraded":"2024-03-11T19:09:04.764161+0000","last_fullsized":"2024-03-11T19:09:04.764161+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:06:48.446454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990356+0000","last_change":"2024-03-11T19:07:49.752622+0000","last_active":"2024-03-11T19:08:44.990356+0000","last_peered":"2024-03-11T19:08:44.990356+0000","last_clean":"2024-03-11T19:08:44.990356+0000","last_became_active":"2024-03-11T19:07:49.752242+0000","last_became_peered":"2024-03-11T19:07:49.752242+0000","last_unstale":"2024-03-11T19:08:44.990356+0000","last_undegraded":"2024-03-11T19:08:44.990356+0000","last_fullsized":"2024-03-11T19:08:44.990356+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:03:50.609641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990462+0000","last_change":"2024-03-11T19:07:48.751657+0000","last_active":"2024-03-11T19:08:44.990462+0000","last_peered":"2024-03-11T19:08:44.990462+0000","last_clean":"2024-03-11T19:08:44.990462+0000","last_became_active":"2024-03-11T19:07:48.751294+0000","last_became_peered":"2024-03-11T19:07:48.751294+0000","last_unstale":"2024-03-11T19:08:44.990462+0000","last_undegraded":"2024-03-11T19:08:44.990462+0000","last_fullsized":"2024-03-11T19:08:44.990462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:33:58.550707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991370+0000","last_change":"2024-03-11T19:07:49.759460+0000","last_active":"2024-03-11T19:08:44.991370+0000","last_peered":"2024-03-11T19:08:44.991370+0000","last_clean":"2024-03-11T19:08:44.991370+0000","last_became_active":"2024-03-11T19:07:49.759136+0000","last_became_peered":"2024-03-11T19:07:49.759136+0000","last_unstale":"2024-03-11T19:08:44.991370+0000","last_undegraded":"2024-03-11T19:08:44.991370+0000","last_fullsized":"2024-03-11T19:08:44.991370+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:10:37.273896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'12","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991106+0000","last_change":"2024-03-11T19:07:48.744695+0000","last_active":"2024-03-11T19:08:44.991106+0000","last_peered":"2024-03-11T19:08:44.991106+0000","last_clean":"2024-03-11T19:08:44.991106+0000","last_became_active":"2024-03-11T19:07:48.744378+0000","last_became_peered":"2024-03-11T19:07:48.744378+0000","last_unstale":"2024-03-11T19:08:44.991106+0000","last_undegraded":"2024-03-11T19:08:44.991106+0000","last_fullsized":"2024-03-11T19:08:44.991106+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:55:46.601414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.035982+0000","last_change":"2024-03-11T19:07:49.761429+0000","last_active":"2024-03-11T19:08:45.035982+0000","last_peered":"2024-03-11T19:08:45.035982+0000","last_clean":"2024-03-11T19:08:45.035982+0000","last_became_active":"2024-03-11T19:07:49.761120+0000","last_became_peered":"2024-03-11T19:07:49.761120+0000","last_unstale":"2024-03-11T19:08:45.035982+0000","last_undegraded":"2024-03-11T19:08:45.035982+0000","last_fullsized":"2024-03-11T19:08:45.035982+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:13:45.839528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990713+0000","last_change":"2024-03-11T19:07:48.740451+0000","last_active":"2024-03-11T19:08:44.990713+0000","last_peered":"2024-03-11T19:08:44.990713+0000","last_clean":"2024-03-11T19:08:44.990713+0000","last_became_active":"2024-03-11T19:07:48.740097+0000","last_became_peered":"2024-03-11T19:07:48.740097+0000","last_unstale":"2024-03-11T19:08:44.990713+0000","last_undegraded":"2024-03-11T19:08:44.990713+0000","last_fullsized":"2024-03-11T19:08:44.990713+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:37.766790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036530+0000","last_change":"2024-03-11T19:07:49.750404+0000","last_active":"2024-03-11T19:08:45.036530+0000","last_peered":"2024-03-11T19:08:45.036530+0000","last_clean":"2024-03-11T19:08:45.036530+0000","last_became_active":"2024-03-11T19:07:49.749941+0000","last_became_peered":"2024-03-11T19:07:49.749941+0000","last_unstale":"2024-03-11T19:08:45.036530+0000","last_undegraded":"2024-03-11T19:08:45.036530+0000","last_fullsized":"2024-03-11T19:08:45.036530+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:55.766729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'6","reported_seq":99,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:06.768459+0000","last_change":"2024-03-11T19:07:48.737100+0000","last_active":"2024-03-11T19:09:06.768459+0000","last_peered":"2024-03-11T19:09:06.768459+0000","last_clean":"2024-03-11T19:09:06.768459+0000","last_became_active":"2024-03-11T19:07:48.736768+0000","last_became_peered":"2024-03-11T19:07:48.736768+0000","last_unstale":"2024-03-11T19:09:06.768459+0000","last_undegraded":"2024-03-11T19:09:06.768459+0000","last_fullsized":"2024-03-11T19:09:06.768459+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:18:36.872151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990388+0000","last_change":"2024-03-11T19:07:49.739020+0000","last_active":"2024-03-11T19:08:44.990388+0000","last_peered":"2024-03-11T19:08:44.990388+0000","last_clean":"2024-03-11T19:08:44.990388+0000","last_became_active":"2024-03-11T19:07:49.738642+0000","last_became_peered":"2024-03-11T19:07:49.738642+0000","last_unstale":"2024-03-11T19:08:44.990388+0000","last_undegraded":"2024-03-11T19:08:44.990388+0000","last_fullsized":"2024-03-11T19:08:44.990388+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:42:29.040242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037233+0000","last_change":"2024-03-11T19:07:48.745192+0000","last_active":"2024-03-11T19:08:45.037233+0000","last_peered":"2024-03-11T19:08:45.037233+0000","last_clean":"2024-03-11T19:08:45.037233+0000","last_became_active":"2024-03-11T19:07:48.744868+0000","last_became_peered":"2024-03-11T19:07:48.744868+0000","last_unstale":"2024-03-11T19:08:45.037233+0000","last_undegraded":"2024-03-11T19:08:45.037233+0000","last_fullsized":"2024-03-11T19:08:45.037233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:19:15.620186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036228+0000","last_change":"2024-03-11T19:07:49.737149+0000","last_active":"2024-03-11T19:08:45.036228+0000","last_peered":"2024-03-11T19:08:45.036228+0000","last_clean":"2024-03-11T19:08:45.036228+0000","last_became_active":"2024-03-11T19:07:49.736797+0000","last_became_peered":"2024-03-11T19:07:49.736797+0000","last_unstale":"2024-03-11T19:08:45.036228+0000","last_undegraded":"2024-03-11T19:08:45.036228+0000","last_fullsized":"2024-03-11T19:08:45.036228+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:12:10.282062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.988596+0000","last_change":"2024-03-11T19:07:48.743948+0000","last_active":"2024-03-11T19:08:44.988596+0000","last_peered":"2024-03-11T19:08:44.988596+0000","last_clean":"2024-03-11T19:08:44.988596+0000","last_became_active":"2024-03-11T19:07:48.743451+0000","last_became_peered":"2024-03-11T19:07:48.743451+0000","last_unstale":"2024-03-11T19:08:44.988596+0000","last_undegraded":"2024-03-11T19:08:44.988596+0000","last_fullsized":"2024-03-11T19:08:44.988596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:03:19.823287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.989398+0000","last_change":"2024-03-11T19:07:49.745518+0000","last_active":"2024-03-11T19:08:44.989398+0000","last_peered":"2024-03-11T19:08:44.989398+0000","last_clean":"2024-03-11T19:08:44.989398+0000","last_became_active":"2024-03-11T19:07:49.743719+0000","last_became_peered":"2024-03-11T19:07:49.743719+0000","last_unstale":"2024-03-11T19:08:44.989398+0000","last_undegraded":"2024-03-11T19:08:44.989398+0000","last_fullsized":"2024-03-11T19:08:44.989398+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:25:33.855679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036218+0000","last_change":"2024-03-11T19:07:48.734044+0000","last_active":"2024-03-11T19:08:45.036218+0000","last_peered":"2024-03-11T19:08:45.036218+0000","last_clean":"2024-03-11T19:08:45.036218+0000","last_became_active":"2024-03-11T19:07:48.733723+0000","last_became_peered":"2024-03-11T19:07:48.733723+0000","last_unstale":"2024-03-11T19:08:45.036218+0000","last_undegraded":"2024-03-11T19:08:45.036218+0000","last_fullsized":"2024-03-11T19:08:45.036218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:53:37.070296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985073+0000","last_change":"2024-03-11T19:07:49.743729+0000","last_active":"2024-03-11T19:08:44.985073+0000","last_peered":"2024-03-11T19:08:44.985073+0000","last_clean":"2024-03-11T19:08:44.985073+0000","last_became_active":"2024-03-11T19:07:49.742992+0000","last_became_peered":"2024-03-11T19:07:49.742992+0000","last_unstale":"2024-03-11T19:08:44.985073+0000","last_undegraded":"2024-03-11T19:08:44.985073+0000","last_fullsized":"2024-03-11T19:08:44.985073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:46.477177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'4","reported_seq":165,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:06.767597+0000","last_change":"2024-03-11T19:07:48.738775+0000","last_active":"2024-03-11T19:09:06.767597+0000","last_peered":"2024-03-11T19:09:06.767597+0000","last_clean":"2024-03-11T19:09:06.767597+0000","last_became_active":"2024-03-11T19:07:48.738437+0000","last_became_peered":"2024-03-11T19:07:48.738437+0000","last_unstale":"2024-03-11T19:09:06.767597+0000","last_undegraded":"2024-03-11T19:09:06.767597+0000","last_fullsized":"2024-03-11T19:09:06.767597+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:17:20.020756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":142,"num_read_kb":142,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036862+0000","last_change":"2024-03-11T19:07:49.760293+0000","last_active":"2024-03-11T19:08:45.036862+0000","last_peered":"2024-03-11T19:08:45.036862+0000","last_clean":"2024-03-11T19:08:45.036862+0000","last_became_active":"2024-03-11T19:07:49.759977+0000","last_became_peered":"2024-03-11T19:07:49.759977+0000","last_unstale":"2024-03-11T19:08:45.036862+0000","last_undegraded":"2024-03-11T19:08:45.036862+0000","last_fullsized":"2024-03-11T19:08:45.036862+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:07:44.726756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985956+0000","last_change":"2024-03-11T19:07:48.740672+0000","last_active":"2024-03-11T19:08:44.985956+0000","last_peered":"2024-03-11T19:08:44.985956+0000","last_clean":"2024-03-11T19:08:44.985956+0000","last_became_active":"2024-03-11T19:07:48.739331+0000","last_became_peered":"2024-03-11T19:07:48.739331+0000","last_unstale":"2024-03-11T19:08:44.985956+0000","last_undegraded":"2024-03-11T19:08:44.985956+0000","last_fullsized":"2024-03-11T19:08:44.985956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:19:28.375515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991268+0000","last_change":"2024-03-11T19:07:49.742780+0000","last_active":"2024-03-11T19:08:44.991268+0000","last_peered":"2024-03-11T19:08:44.991268+0000","last_clean":"2024-03-11T19:08:44.991268+0000","last_became_active":"2024-03-11T19:07:49.742442+0000","last_became_peered":"2024-03-11T19:07:49.742442+0000","last_unstale":"2024-03-11T19:08:44.991268+0000","last_undegraded":"2024-03-11T19:08:44.991268+0000","last_fullsized":"2024-03-11T19:08:44.991268+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:11:13.838225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036582+0000","last_change":"2024-03-11T19:07:48.752617+0000","last_active":"2024-03-11T19:08:45.036582+0000","last_peered":"2024-03-11T19:08:45.036582+0000","last_clean":"2024-03-11T19:08:45.036582+0000","last_became_active":"2024-03-11T19:07:48.752181+0000","last_became_peered":"2024-03-11T19:07:48.752181+0000","last_unstale":"2024-03-11T19:08:45.036582+0000","last_undegraded":"2024-03-11T19:08:45.036582+0000","last_fullsized":"2024-03-11T19:08:45.036582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:05:37.162866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036332+0000","last_change":"2024-03-11T19:07:49.753672+0000","last_active":"2024-03-11T19:08:45.036332+0000","last_peered":"2024-03-11T19:08:45.036332+0000","last_clean":"2024-03-11T19:08:45.036332+0000","last_became_active":"2024-03-11T19:07:49.753315+0000","last_became_peered":"2024-03-11T19:07:49.753315+0000","last_unstale":"2024-03-11T19:08:45.036332+0000","last_undegraded":"2024-03-11T19:08:45.036332+0000","last_fullsized":"2024-03-11T19:08:45.036332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:30:58.503249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'31","reported_seq":199,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:06.767600+0000","last_change":"2024-03-11T19:07:48.740424+0000","last_active":"2024-03-11T19:09:06.767600+0000","last_peered":"2024-03-11T19:09:06.767600+0000","last_clean":"2024-03-11T19:09:06.767600+0000","last_became_active":"2024-03-11T19:07:48.740077+0000","last_became_peered":"2024-03-11T19:07:48.740077+0000","last_unstale":"2024-03-11T19:09:06.767600+0000","last_undegraded":"2024-03-11T19:09:06.767600+0000","last_fullsized":"2024-03-11T19:09:06.767600+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:30:29.193861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":149,"num_read_kb":134,"num_write":33,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990971+0000","last_change":"2024-03-11T19:07:49.746440+0000","last_active":"2024-03-11T19:08:44.990971+0000","last_peered":"2024-03-11T19:08:44.990971+0000","last_clean":"2024-03-11T19:08:44.990971+0000","last_became_active":"2024-03-11T19:07:49.746106+0000","last_became_peered":"2024-03-11T19:07:49.746106+0000","last_unstale":"2024-03-11T19:08:44.990971+0000","last_undegraded":"2024-03-11T19:08:44.990971+0000","last_fullsized":"2024-03-11T19:08:44.990971+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:44:58.039917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'9","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.986027+0000","last_change":"2024-03-11T19:07:48.744437+0000","last_active":"2024-03-11T19:08:44.986027+0000","last_peered":"2024-03-11T19:08:44.986027+0000","last_clean":"2024-03-11T19:08:44.986027+0000","last_became_active":"2024-03-11T19:07:48.744142+0000","last_became_peered":"2024-03-11T19:07:48.744142+0000","last_unstale":"2024-03-11T19:08:44.986027+0000","last_undegraded":"2024-03-11T19:08:44.986027+0000","last_fullsized":"2024-03-11T19:08:44.986027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:32:09.076975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.988450+0000","last_change":"2024-03-11T19:07:49.751191+0000","last_active":"2024-03-11T19:08:44.988450+0000","last_peered":"2024-03-11T19:08:44.988450+0000","last_clean":"2024-03-11T19:08:44.988450+0000","last_became_active":"2024-03-11T19:07:49.750849+0000","last_became_peered":"2024-03-11T19:07:49.750849+0000","last_unstale":"2024-03-11T19:08:44.988450+0000","last_undegraded":"2024-03-11T19:08:44.988450+0000","last_fullsized":"2024-03-11T19:08:44.988450+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:19:18.416905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036229+0000","last_change":"2024-03-11T19:07:48.742808+0000","last_active":"2024-03-11T19:08:45.036229+0000","last_peered":"2024-03-11T19:08:45.036229+0000","last_clean":"2024-03-11T19:08:45.036229+0000","last_became_active":"2024-03-11T19:07:48.742462+0000","last_became_peered":"2024-03-11T19:07:48.742462+0000","last_unstale":"2024-03-11T19:08:45.036229+0000","last_undegraded":"2024-03-11T19:08:45.036229+0000","last_fullsized":"2024-03-11T19:08:45.036229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:19.109299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985856+0000","last_change":"2024-03-11T19:07:49.744673+0000","last_active":"2024-03-11T19:08:44.985856+0000","last_peered":"2024-03-11T19:08:44.985856+0000","last_clean":"2024-03-11T19:08:44.985856+0000","last_became_active":"2024-03-11T19:07:49.744313+0000","last_became_peered":"2024-03-11T19:07:49.744313+0000","last_unstale":"2024-03-11T19:08:44.985856+0000","last_undegraded":"2024-03-11T19:08:44.985856+0000","last_fullsized":"2024-03-11T19:08:44.985856+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:21:04.759356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'5","reported_seq":99,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:07.772645+0000","last_change":"2024-03-11T19:07:48.736024+0000","last_active":"2024-03-11T19:09:07.772645+0000","last_peered":"2024-03-11T19:09:07.772645+0000","last_clean":"2024-03-11T19:09:07.772645+0000","last_became_active":"2024-03-11T19:07:48.735720+0000","last_became_peered":"2024-03-11T19:07:48.735720+0000","last_unstale":"2024-03-11T19:09:07.772645+0000","last_undegraded":"2024-03-11T19:09:07.772645+0000","last_fullsized":"2024-03-11T19:09:07.772645+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:04:57.386267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985463+0000","last_change":"2024-03-11T19:07:49.742924+0000","last_active":"2024-03-11T19:08:44.985463+0000","last_peered":"2024-03-11T19:08:44.985463+0000","last_clean":"2024-03-11T19:08:44.985463+0000","last_became_active":"2024-03-11T19:07:49.742542+0000","last_became_peered":"2024-03-11T19:07:49.742542+0000","last_unstale":"2024-03-11T19:08:44.985463+0000","last_undegraded":"2024-03-11T19:08:44.985463+0000","last_fullsized":"2024-03-11T19:08:44.985463+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:08:11.944311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'17","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990343+0000","last_change":"2024-03-11T19:07:48.753113+0000","last_active":"2024-03-11T19:08:44.990343+0000","last_peered":"2024-03-11T19:08:44.990343+0000","last_clean":"2024-03-11T19:08:44.990343+0000","last_became_active":"2024-03-11T19:07:48.752517+0000","last_became_peered":"2024-03-11T19:07:48.752517+0000","last_unstale":"2024-03-11T19:08:44.990343+0000","last_undegraded":"2024-03-11T19:08:44.990343+0000","last_fullsized":"2024-03-11T19:08:44.990343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:48:33.441125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.038561+0000","last_change":"2024-03-11T19:08:45.038243+0000","last_active":"2024-03-11T19:08:45.038561+0000","last_peered":"2024-03-11T19:08:45.038561+0000","last_clean":"2024-03-11T19:08:45.038561+0000","last_became_active":"2024-03-11T19:08:45.037969+0000","last_became_peered":"2024-03-11T19:08:45.037969+0000","last_unstale":"2024-03-11T19:08:45.038561+0000","last_undegraded":"2024-03-11T19:08:45.038561+0000","last_fullsized":"2024-03-11T19:08:45.038561+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:29:09.747958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991087+0000","last_change":"2024-03-11T19:07:48.739571+0000","last_active":"2024-03-11T19:08:44.991087+0000","last_peered":"2024-03-11T19:08:44.991087+0000","last_clean":"2024-03-11T19:08:44.991087+0000","last_became_active":"2024-03-11T19:07:48.739159+0000","last_became_peered":"2024-03-11T19:07:48.739159+0000","last_unstale":"2024-03-11T19:08:44.991087+0000","last_undegraded":"2024-03-11T19:08:44.991087+0000","last_fullsized":"2024-03-11T19:08:44.991087+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:46:43.474591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.987681+0000","last_change":"2024-03-11T19:07:49.744760+0000","last_active":"2024-03-11T19:08:44.987681+0000","last_peered":"2024-03-11T19:08:44.987681+0000","last_clean":"2024-03-11T19:08:44.987681+0000","last_became_active":"2024-03-11T19:07:49.744265+0000","last_became_peered":"2024-03-11T19:07:49.744265+0000","last_unstale":"2024-03-11T19:08:44.987681+0000","last_undegraded":"2024-03-11T19:08:44.987681+0000","last_fullsized":"2024-03-11T19:08:44.987681+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:09:49.823462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985159+0000","last_change":"2024-03-11T19:07:48.740223+0000","last_active":"2024-03-11T19:08:44.985159+0000","last_peered":"2024-03-11T19:08:44.985159+0000","last_clean":"2024-03-11T19:08:44.985159+0000","last_became_active":"2024-03-11T19:07:48.739569+0000","last_became_peered":"2024-03-11T19:07:48.739569+0000","last_unstale":"2024-03-11T19:08:44.985159+0000","last_undegraded":"2024-03-11T19:08:44.985159+0000","last_fullsized":"2024-03-11T19:08:44.985159+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:04:19.384523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.038056+0000","last_change":"2024-03-11T19:07:49.755620+0000","last_active":"2024-03-11T19:08:45.038056+0000","last_peered":"2024-03-11T19:08:45.038056+0000","last_clean":"2024-03-11T19:08:45.038056+0000","last_became_active":"2024-03-11T19:07:49.750886+0000","last_became_peered":"2024-03-11T19:07:49.750886+0000","last_unstale":"2024-03-11T19:08:45.038056+0000","last_undegraded":"2024-03-11T19:08:45.038056+0000","last_fullsized":"2024-03-11T19:08:45.038056+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:35:54.191061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'5","reported_seq":94,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:03.763279+0000","last_change":"2024-03-11T19:07:48.743340+0000","last_active":"2024-03-11T19:09:03.763279+0000","last_peered":"2024-03-11T19:09:03.763279+0000","last_clean":"2024-03-11T19:09:03.763279+0000","last_became_active":"2024-03-11T19:07:48.742947+0000","last_became_peered":"2024-03-11T19:07:48.742947+0000","last_unstale":"2024-03-11T19:09:03.763279+0000","last_undegraded":"2024-03-11T19:09:03.763279+0000","last_fullsized":"2024-03-11T19:09:03.763279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:40:25.233150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037599+0000","last_change":"2024-03-11T19:07:49.753668+0000","last_active":"2024-03-11T19:08:45.037599+0000","last_peered":"2024-03-11T19:08:45.037599+0000","last_clean":"2024-03-11T19:08:45.037599+0000","last_became_active":"2024-03-11T19:07:49.750574+0000","last_became_peered":"2024-03-11T19:07:49.750574+0000","last_unstale":"2024-03-11T19:08:45.037599+0000","last_undegraded":"2024-03-11T19:08:45.037599+0000","last_fullsized":"2024-03-11T19:08:45.037599+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:12:32.711847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036426+0000","last_change":"2024-03-11T19:07:49.739538+0000","last_active":"2024-03-11T19:08:45.036426+0000","last_peered":"2024-03-11T19:08:45.036426+0000","last_clean":"2024-03-11T19:08:45.036426+0000","last_became_active":"2024-03-11T19:07:49.738617+0000","last_became_peered":"2024-03-11T19:07:49.738617+0000","last_unstale":"2024-03-11T19:08:45.036426+0000","last_undegraded":"2024-03-11T19:08:45.036426+0000","last_fullsized":"2024-03-11T19:08:45.036426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:43:08.697125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":98,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:07.772140+0000","last_change":"2024-03-11T19:07:48.749527+0000","last_active":"2024-03-11T19:09:07.772140+0000","last_peered":"2024-03-11T19:09:07.772140+0000","last_clean":"2024-03-11T19:09:07.772140+0000","last_became_active":"2024-03-11T19:07:48.749026+0000","last_became_peered":"2024-03-11T19:07:48.749026+0000","last_unstale":"2024-03-11T19:09:07.772140+0000","last_undegraded":"2024-03-11T19:09:07.772140+0000","last_fullsized":"2024-03-11T19:09:07.772140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:20:19.058429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990018+0000","last_change":"2024-03-11T19:07:49.757696+0000","last_active":"2024-03-11T19:08:44.990018+0000","last_peered":"2024-03-11T19:08:44.990018+0000","last_clean":"2024-03-11T19:08:44.990018+0000","last_became_active":"2024-03-11T19:07:49.757380+0000","last_became_peered":"2024-03-11T19:07:49.757380+0000","last_unstale":"2024-03-11T19:08:44.990018+0000","last_undegraded":"2024-03-11T19:08:44.990018+0000","last_fullsized":"2024-03-11T19:08:44.990018+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:03:22.954619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'6","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036756+0000","last_change":"2024-03-11T19:07:48.737504+0000","last_active":"2024-03-11T19:08:45.036756+0000","last_peered":"2024-03-11T19:08:45.036756+0000","last_clean":"2024-03-11T19:08:45.036756+0000","last_became_active":"2024-03-11T19:07:48.737147+0000","last_became_peered":"2024-03-11T19:07:48.737147+0000","last_unstale":"2024-03-11T19:08:45.036756+0000","last_undegraded":"2024-03-11T19:08:45.036756+0000","last_fullsized":"2024-03-11T19:08:45.036756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:04.819617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":343,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985537+0000","last_change":"2024-03-11T19:06:35.035452+0000","last_active":"2024-03-11T19:08:44.985537+0000","last_peered":"2024-03-11T19:08:44.985537+0000","last_clean":"2024-03-11T19:08:44.985537+0000","last_became_active":"2024-03-11T19:06:35.035076+0000","last_became_peered":"2024-03-11T19:06:35.035076+0000","last_unstale":"2024-03-11T19:08:44.985537+0000","last_undegraded":"2024-03-11T19:08:44.985537+0000","last_fullsized":"2024-03-11T19:08:44.985537+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:03:10.143002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:03:10.143002+0000","last_clean_scrub_stamp":"2024-03-11T19:03:10.143002+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:39:37.348091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.988339+0000","last_change":"2024-03-11T19:07:49.743000+0000","last_active":"2024-03-11T19:08:44.988339+0000","last_peered":"2024-03-11T19:08:44.988339+0000","last_clean":"2024-03-11T19:08:44.988339+0000","last_became_active":"2024-03-11T19:07:49.742691+0000","last_became_peered":"2024-03-11T19:07:49.742691+0000","last_unstale":"2024-03-11T19:08:44.988339+0000","last_undegraded":"2024-03-11T19:08:44.988339+0000","last_fullsized":"2024-03-11T19:08:44.988339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:40:21.519929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985665+0000","last_change":"2024-03-11T19:07:48.739038+0000","last_active":"2024-03-11T19:08:44.985665+0000","last_peered":"2024-03-11T19:08:44.985665+0000","last_clean":"2024-03-11T19:08:44.985665+0000","last_became_active":"2024-03-11T19:07:48.738714+0000","last_became_peered":"2024-03-11T19:07:48.738714+0000","last_unstale":"2024-03-11T19:08:44.985665+0000","last_undegraded":"2024-03-11T19:08:44.985665+0000","last_fullsized":"2024-03-11T19:08:44.985665+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:17:43.086679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.998660+0000","last_change":"2024-03-11T19:08:44.998660+0000","last_active":"2024-03-11T19:08:44.998660+0000","last_peered":"2024-03-11T19:08:44.998660+0000","last_clean":"2024-03-11T19:08:44.998660+0000","last_became_active":"2024-03-11T19:08:44.991682+0000","last_became_peered":"2024-03-11T19:08:44.991682+0000","last_unstale":"2024-03-11T19:08:44.998660+0000","last_undegraded":"2024-03-11T19:08:44.998660+0000","last_fullsized":"2024-03-11T19:08:44.998660+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:19:44.266076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.988920+0000","last_change":"2024-03-11T19:07:49.742434+0000","last_active":"2024-03-11T19:08:44.988920+0000","last_peered":"2024-03-11T19:08:44.988920+0000","last_clean":"2024-03-11T19:08:44.988920+0000","last_became_active":"2024-03-11T19:07:49.742048+0000","last_became_peered":"2024-03-11T19:07:49.742048+0000","last_unstale":"2024-03-11T19:08:44.988920+0000","last_undegraded":"2024-03-11T19:08:44.988920+0000","last_fullsized":"2024-03-11T19:08:44.988920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:39:22.685055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'12","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985474+0000","last_change":"2024-03-11T19:07:48.744728+0000","last_active":"2024-03-11T19:08:44.985474+0000","last_peered":"2024-03-11T19:08:44.985474+0000","last_clean":"2024-03-11T19:08:44.985474+0000","last_became_active":"2024-03-11T19:07:48.744378+0000","last_became_peered":"2024-03-11T19:07:48.744378+0000","last_unstale":"2024-03-11T19:08:44.985474+0000","last_undegraded":"2024-03-11T19:08:44.985474+0000","last_fullsized":"2024-03-11T19:08:44.985474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:42.823040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990691+0000","last_change":"2024-03-11T19:07:49.744261+0000","last_active":"2024-03-11T19:08:44.990691+0000","last_peered":"2024-03-11T19:08:44.990691+0000","last_clean":"2024-03-11T19:08:44.990691+0000","last_became_active":"2024-03-11T19:07:49.743946+0000","last_became_peered":"2024-03-11T19:07:49.743946+0000","last_unstale":"2024-03-11T19:08:44.990691+0000","last_undegraded":"2024-03-11T19:08:44.990691+0000","last_fullsized":"2024-03-11T19:08:44.990691+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:54:07.928899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037193+0000","last_change":"2024-03-11T19:07:48.740059+0000","last_active":"2024-03-11T19:08:45.037193+0000","last_peered":"2024-03-11T19:08:45.037193+0000","last_clean":"2024-03-11T19:08:45.037193+0000","last_became_active":"2024-03-11T19:07:48.739626+0000","last_became_peered":"2024-03-11T19:07:48.739626+0000","last_unstale":"2024-03-11T19:08:45.037193+0000","last_undegraded":"2024-03-11T19:08:45.037193+0000","last_fullsized":"2024-03-11T19:08:45.037193+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:45:12.219203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991612+0000","last_change":"2024-03-11T19:07:49.746041+0000","last_active":"2024-03-11T19:08:44.991612+0000","last_peered":"2024-03-11T19:08:44.991612+0000","last_clean":"2024-03-11T19:08:44.991612+0000","last_became_active":"2024-03-11T19:07:49.745602+0000","last_became_peered":"2024-03-11T19:07:49.745602+0000","last_unstale":"2024-03-11T19:08:44.991612+0000","last_undegraded":"2024-03-11T19:08:44.991612+0000","last_fullsized":"2024-03-11T19:08:44.991612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:23:44.596929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.035863+0000","last_change":"2024-03-11T19:07:48.748677+0000","last_active":"2024-03-11T19:08:45.035863+0000","last_peered":"2024-03-11T19:08:45.035863+0000","last_clean":"2024-03-11T19:08:45.035863+0000","last_became_active":"2024-03-11T19:07:48.748470+0000","last_became_peered":"2024-03-11T19:07:48.748470+0000","last_unstale":"2024-03-11T19:08:45.035863+0000","last_undegraded":"2024-03-11T19:08:45.035863+0000","last_fullsized":"2024-03-11T19:08:45.035863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:24:55.240722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990208+0000","last_change":"2024-03-11T19:07:49.746522+0000","last_active":"2024-03-11T19:08:44.990208+0000","last_peered":"2024-03-11T19:08:44.990208+0000","last_clean":"2024-03-11T19:08:44.990208+0000","last_became_active":"2024-03-11T19:07:49.746179+0000","last_became_peered":"2024-03-11T19:07:49.746179+0000","last_unstale":"2024-03-11T19:08:44.990208+0000","last_undegraded":"2024-03-11T19:08:44.990208+0000","last_fullsized":"2024-03-11T19:08:44.990208+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:06:08.144850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.989685+0000","last_change":"2024-03-11T19:07:48.740926+0000","last_active":"2024-03-11T19:08:44.989685+0000","last_peered":"2024-03-11T19:08:44.989685+0000","last_clean":"2024-03-11T19:08:44.989685+0000","last_became_active":"2024-03-11T19:07:48.740625+0000","last_became_peered":"2024-03-11T19:07:48.740625+0000","last_unstale":"2024-03-11T19:08:44.989685+0000","last_undegraded":"2024-03-11T19:08:44.989685+0000","last_fullsized":"2024-03-11T19:08:44.989685+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:50:13.079608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.987248+0000","last_change":"2024-03-11T19:07:49.742428+0000","last_active":"2024-03-11T19:08:44.987248+0000","last_peered":"2024-03-11T19:08:44.987248+0000","last_clean":"2024-03-11T19:08:44.987248+0000","last_became_active":"2024-03-11T19:07:49.742098+0000","last_became_peered":"2024-03-11T19:07:49.742098+0000","last_unstale":"2024-03-11T19:08:44.987248+0000","last_undegraded":"2024-03-11T19:08:44.987248+0000","last_fullsized":"2024-03-11T19:08:44.987248+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:19:24.615755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'3","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.002189+0000","last_change":"2024-03-11T19:08:45.002189+0000","last_active":"2024-03-11T19:08:45.002189+0000","last_peered":"2024-03-11T19:08:45.002189+0000","last_clean":"2024-03-11T19:08:45.002189+0000","last_became_active":"2024-03-11T19:08:44.992818+0000","last_became_peered":"2024-03-11T19:08:44.992818+0000","last_unstale":"2024-03-11T19:08:45.002189+0000","last_undegraded":"2024-03-11T19:08:45.002189+0000","last_fullsized":"2024-03-11T19:08:45.002189+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:34:54.461734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.989833+0000","last_change":"2024-03-11T19:07:49.754872+0000","last_active":"2024-03-11T19:08:44.989833+0000","last_peered":"2024-03-11T19:08:44.989833+0000","last_clean":"2024-03-11T19:08:44.989833+0000","last_became_active":"2024-03-11T19:07:49.749663+0000","last_became_peered":"2024-03-11T19:07:49.749663+0000","last_unstale":"2024-03-11T19:08:44.989833+0000","last_undegraded":"2024-03-11T19:08:44.989833+0000","last_fullsized":"2024-03-11T19:08:44.989833+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:22:23.664814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990526+0000","last_change":"2024-03-11T19:07:48.742227+0000","last_active":"2024-03-11T19:08:44.990526+0000","last_peered":"2024-03-11T19:08:44.990526+0000","last_clean":"2024-03-11T19:08:44.990526+0000","last_became_active":"2024-03-11T19:07:48.741912+0000","last_became_peered":"2024-03-11T19:07:48.741912+0000","last_unstale":"2024-03-11T19:08:44.990526+0000","last_undegraded":"2024-03-11T19:08:44.990526+0000","last_fullsized":"2024-03-11T19:08:44.990526+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:10:06.120433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985806+0000","last_change":"2024-03-11T19:07:49.748947+0000","last_active":"2024-03-11T19:08:44.985806+0000","last_peered":"2024-03-11T19:08:44.985806+0000","last_clean":"2024-03-11T19:08:44.985806+0000","last_became_active":"2024-03-11T19:07:49.748621+0000","last_became_peered":"2024-03-11T19:07:49.748621+0000","last_unstale":"2024-03-11T19:08:44.985806+0000","last_undegraded":"2024-03-11T19:08:44.985806+0000","last_fullsized":"2024-03-11T19:08:44.985806+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:19:53.824590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'5","reported_seq":98,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:06.768362+0000","last_change":"2024-03-11T19:07:48.748799+0000","last_active":"2024-03-11T19:09:06.768362+0000","last_peered":"2024-03-11T19:09:06.768362+0000","last_clean":"2024-03-11T19:09:06.768362+0000","last_became_active":"2024-03-11T19:07:48.748434+0000","last_became_peered":"2024-03-11T19:07:48.748434+0000","last_unstale":"2024-03-11T19:09:06.768362+0000","last_undegraded":"2024-03-11T19:09:06.768362+0000","last_fullsized":"2024-03-11T19:09:06.768362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:10:09.836926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.986112+0000","last_change":"2024-03-11T19:07:49.750122+0000","last_active":"2024-03-11T19:08:44.986112+0000","last_peered":"2024-03-11T19:08:44.986112+0000","last_clean":"2024-03-11T19:08:44.986112+0000","last_became_active":"2024-03-11T19:07:49.749801+0000","last_became_peered":"2024-03-11T19:07:49.749801+0000","last_unstale":"2024-03-11T19:08:44.986112+0000","last_undegraded":"2024-03-11T19:08:44.986112+0000","last_fullsized":"2024-03-11T19:08:44.986112+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:58:31.155275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036439+0000","last_change":"2024-03-11T19:07:48.741959+0000","last_active":"2024-03-11T19:08:45.036439+0000","last_peered":"2024-03-11T19:08:45.036439+0000","last_clean":"2024-03-11T19:08:45.036439+0000","last_became_active":"2024-03-11T19:07:48.741630+0000","last_became_peered":"2024-03-11T19:07:48.741630+0000","last_unstale":"2024-03-11T19:08:45.036439+0000","last_undegraded":"2024-03-11T19:08:45.036439+0000","last_fullsized":"2024-03-11T19:08:45.036439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:27:08.606055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985962+0000","last_change":"2024-03-11T19:07:49.729838+0000","last_active":"2024-03-11T19:08:44.985962+0000","last_peered":"2024-03-11T19:08:44.985962+0000","last_clean":"2024-03-11T19:08:44.985962+0000","last_became_active":"2024-03-11T19:07:49.728912+0000","last_became_peered":"2024-03-11T19:07:49.728912+0000","last_unstale":"2024-03-11T19:08:44.985962+0000","last_undegraded":"2024-03-11T19:08:44.985962+0000","last_fullsized":"2024-03-11T19:08:44.985962+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:02:06.551734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.989032+0000","last_change":"2024-03-11T19:07:48.744709+0000","last_active":"2024-03-11T19:08:44.989032+0000","last_peered":"2024-03-11T19:08:44.989032+0000","last_clean":"2024-03-11T19:08:44.989032+0000","last_became_active":"2024-03-11T19:07:48.744356+0000","last_became_peered":"2024-03-11T19:07:48.744356+0000","last_unstale":"2024-03-11T19:08:44.989032+0000","last_undegraded":"2024-03-11T19:08:44.989032+0000","last_fullsized":"2024-03-11T19:08:44.989032+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:48:47.098443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.986084+0000","last_change":"2024-03-11T19:07:49.743288+0000","last_active":"2024-03-11T19:08:44.986084+0000","last_peered":"2024-03-11T19:08:44.986084+0000","last_clean":"2024-03-11T19:08:44.986084+0000","last_became_active":"2024-03-11T19:07:49.742942+0000","last_became_peered":"2024-03-11T19:07:49.742942+0000","last_unstale":"2024-03-11T19:08:44.986084+0000","last_undegraded":"2024-03-11T19:08:44.986084+0000","last_fullsized":"2024-03-11T19:08:44.986084+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:21:53.896192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.987288+0000","last_change":"2024-03-11T19:07:49.741610+0000","last_active":"2024-03-11T19:08:44.987288+0000","last_peered":"2024-03-11T19:08:44.987288+0000","last_clean":"2024-03-11T19:08:44.987288+0000","last_became_active":"2024-03-11T19:07:49.741234+0000","last_became_peered":"2024-03-11T19:07:49.741234+0000","last_unstale":"2024-03-11T19:08:44.987288+0000","last_undegraded":"2024-03-11T19:08:44.987288+0000","last_fullsized":"2024-03-11T19:08:44.987288+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:53:03.320093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037715+0000","last_change":"2024-03-11T19:08:45.037715+0000","last_active":"2024-03-11T19:08:45.037715+0000","last_peered":"2024-03-11T19:08:45.037715+0000","last_clean":"2024-03-11T19:08:45.037715+0000","last_became_active":"2024-03-11T19:08:45.037299+0000","last_became_peered":"2024-03-11T19:08:45.037299+0000","last_unstale":"2024-03-11T19:08:45.037715+0000","last_undegraded":"2024-03-11T19:08:45.037715+0000","last_fullsized":"2024-03-11T19:08:45.037715+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:01:49.025379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985697+0000","last_change":"2024-03-11T19:07:49.745928+0000","last_active":"2024-03-11T19:08:44.985697+0000","last_peered":"2024-03-11T19:08:44.985697+0000","last_clean":"2024-03-11T19:08:44.985697+0000","last_became_active":"2024-03-11T19:07:49.745025+0000","last_became_peered":"2024-03-11T19:07:49.745025+0000","last_unstale":"2024-03-11T19:08:44.985697+0000","last_undegraded":"2024-03-11T19:08:44.985697+0000","last_fullsized":"2024-03-11T19:08:44.985697+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:40:30.141906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991479+0000","last_change":"2024-03-11T19:07:49.756274+0000","last_active":"2024-03-11T19:08:44.991479+0000","last_peered":"2024-03-11T19:08:44.991479+0000","last_clean":"2024-03-11T19:08:44.991479+0000","last_became_active":"2024-03-11T19:07:49.755954+0000","last_became_peered":"2024-03-11T19:07:49.755954+0000","last_unstale":"2024-03-11T19:08:44.991479+0000","last_undegraded":"2024-03-11T19:08:44.991479+0000","last_fullsized":"2024-03-11T19:08:44.991479+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:15:36.825240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.992154+0000","last_change":"2024-03-11T19:08:44.992154+0000","last_active":"2024-03-11T19:08:44.992154+0000","last_peered":"2024-03-11T19:08:44.992154+0000","last_clean":"2024-03-11T19:08:44.992154+0000","last_became_active":"2024-03-11T19:08:44.991772+0000","last_became_peered":"2024-03-11T19:08:44.991772+0000","last_unstale":"2024-03-11T19:08:44.992154+0000","last_undegraded":"2024-03-11T19:08:44.992154+0000","last_fullsized":"2024-03-11T19:08:44.992154+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:10:33.882095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.987749+0000","last_change":"2024-03-11T19:07:49.743963+0000","last_active":"2024-03-11T19:08:44.987749+0000","last_peered":"2024-03-11T19:08:44.987749+0000","last_clean":"2024-03-11T19:08:44.987749+0000","last_became_active":"2024-03-11T19:07:49.743568+0000","last_became_peered":"2024-03-11T19:07:49.743568+0000","last_unstale":"2024-03-11T19:08:44.987749+0000","last_undegraded":"2024-03-11T19:08:44.987749+0000","last_fullsized":"2024-03-11T19:08:44.987749+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:45:28.970286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.986367+0000","last_change":"2024-03-11T19:07:49.750677+0000","last_active":"2024-03-11T19:08:44.986367+0000","last_peered":"2024-03-11T19:08:44.986367+0000","last_clean":"2024-03-11T19:08:44.986367+0000","last_became_active":"2024-03-11T19:07:49.750373+0000","last_became_peered":"2024-03-11T19:07:49.750373+0000","last_unstale":"2024-03-11T19:08:44.986367+0000","last_undegraded":"2024-03-11T19:08:44.986367+0000","last_fullsized":"2024-03-11T19:08:44.986367+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:07:28.767502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990146+0000","last_change":"2024-03-11T19:07:49.745278+0000","last_active":"2024-03-11T19:08:44.990146+0000","last_peered":"2024-03-11T19:08:44.990146+0000","last_clean":"2024-03-11T19:08:44.990146+0000","last_became_active":"2024-03-11T19:07:49.744998+0000","last_became_peered":"2024-03-11T19:07:49.744998+0000","last_unstale":"2024-03-11T19:08:44.990146+0000","last_undegraded":"2024-03-11T19:08:44.990146+0000","last_fullsized":"2024-03-11T19:08:44.990146+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:23.778051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036526+0000","last_change":"2024-03-11T19:07:49.734318+0000","last_active":"2024-03-11T19:08:45.036526+0000","last_peered":"2024-03-11T19:08:45.036526+0000","last_clean":"2024-03-11T19:08:45.036526+0000","last_became_active":"2024-03-11T19:07:49.733998+0000","last_became_peered":"2024-03-11T19:07:49.733998+0000","last_unstale":"2024-03-11T19:08:45.036526+0000","last_undegraded":"2024-03-11T19:08:45.036526+0000","last_fullsized":"2024-03-11T19:08:45.036526+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:48:50.308499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.989569+0000","last_change":"2024-03-11T19:07:49.741009+0000","last_active":"2024-03-11T19:08:44.989569+0000","last_peered":"2024-03-11T19:08:44.989569+0000","last_clean":"2024-03-11T19:08:44.989569+0000","last_became_active":"2024-03-11T19:07:49.740094+0000","last_became_peered":"2024-03-11T19:07:49.740094+0000","last_unstale":"2024-03-11T19:08:44.989569+0000","last_undegraded":"2024-03-11T19:08:44.989569+0000","last_fullsized":"2024-03-11T19:08:44.989569+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:06:31.945387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.035949+0000","last_change":"2024-03-11T19:07:49.759300+0000","last_active":"2024-03-11T19:08:45.035949+0000","last_peered":"2024-03-11T19:08:45.035949+0000","last_clean":"2024-03-11T19:08:45.035949+0000","last_became_active":"2024-03-11T19:07:49.758011+0000","last_became_peered":"2024-03-11T19:07:49.758011+0000","last_unstale":"2024-03-11T19:08:45.035949+0000","last_undegraded":"2024-03-11T19:08:45.035949+0000","last_fullsized":"2024-03-11T19:08:45.035949+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:03:40.651816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.988731+0000","last_change":"2024-03-11T19:07:49.752905+0000","last_active":"2024-03-11T19:08:44.988731+0000","last_peered":"2024-03-11T19:08:44.988731+0000","last_clean":"2024-03-11T19:08:44.988731+0000","last_became_active":"2024-03-11T19:07:49.752383+0000","last_became_peered":"2024-03-11T19:07:49.752383+0000","last_unstale":"2024-03-11T19:08:44.988731+0000","last_undegraded":"2024-03-11T19:08:44.988731+0000","last_fullsized":"2024-03-11T19:08:44.988731+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:45:33.475526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.987445+0000","last_change":"2024-03-11T19:07:49.744957+0000","last_active":"2024-03-11T19:08:44.987445+0000","last_peered":"2024-03-11T19:08:44.987445+0000","last_clean":"2024-03-11T19:08:44.987445+0000","last_became_active":"2024-03-11T19:07:49.744627+0000","last_became_peered":"2024-03-11T19:07:49.744627+0000","last_unstale":"2024-03-11T19:08:44.987445+0000","last_undegraded":"2024-03-11T19:08:44.987445+0000","last_fullsized":"2024-03-11T19:08:44.987445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:27:47.817004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245707,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67859,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":749,"num_read_kb":448,"num_write":237,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":203577,"data_compressed":5565,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":206,"ondisk_log_size":206,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318302,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":464282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579938,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":1116245,"data_compressed":37658,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743464,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":464100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005101,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":464366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266738,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1044480,"data_stored":1056996,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430264,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":978944,"data_stored":463817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":525776,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":999424,"data_stored":465156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1056768,"data_stored":526387,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149766,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":995328,"data_stored":463642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888130,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":978944,"data_stored":463678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691901,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1056768,"data_stored":1056674,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62798,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":63409,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":62782,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T19:09:12.313 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T19:09:12.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:12 smithi170 ceph-mon[35690]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:09:12.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:12 smithi190 ceph-mon[36292]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:09:12.560 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:09:13.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:13 smithi170 ceph-mon[35690]: pgmap v333: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:13.394 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:13 smithi029 ceph-mon[32439]: pgmap v333: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:13.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:13 smithi190 ceph-mon[36292]: pgmap v333: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:14.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.259+0000 7f9c4acf3700 1 -- 172.21.15.29:0/3365939019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c4406a370 msgr2=0x7f9c4406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:14.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.259+0000 7f9c4acf3700 1 --2- 172.21.15.29:0/3365939019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c4406a370 0x7f9c4406a7d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9c300099c0 tx=0x7f9c30009cd0 comp rx=0 tx=0).stop 2024-03-11T19:09:14.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.259+0000 7f9c4acf3700 1 -- 172.21.15.29:0/3365939019 shutdown_connections 2024-03-11T19:09:14.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.259+0000 7f9c4acf3700 1 --2- 172.21.15.29:0/3365939019 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c4406ad10 0x7f9c4410eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:14.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.259+0000 7f9c4acf3700 1 --2- 172.21.15.29:0/3365939019 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c44107cf0 0x7f9c441080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:14.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.259+0000 7f9c4acf3700 1 --2- 172.21.15.29:0/3365939019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c4406a370 0x7f9c4406a7d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:14.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.259+0000 7f9c4acf3700 1 -- 172.21.15.29:0/3365939019 >> 172.21.15.29:0/3365939019 conn(0x7f9c44076bc0 msgr2=0x7f9c44076fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:14.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.259+0000 7f9c4acf3700 1 -- 172.21.15.29:0/3365939019 shutdown_connections 2024-03-11T19:09:14.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.259+0000 7f9c4acf3700 1 -- 172.21.15.29:0/3365939019 wait complete. 2024-03-11T19:09:14.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.259+0000 7f9c4acf3700 1 Processor -- start 2024-03-11T19:09:14.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c4acf3700 1 -- start start 2024-03-11T19:09:14.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c4acf3700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c4406a370 0x7f9c441980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:14.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c4acf3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c4406ad10 0x7f9c44198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:14.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c4acf3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c44107cf0 0x7f9c4419c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:14.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c4acf3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4419cfa0 con 0x7f9c4406ad10 2024-03-11T19:09:14.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c4acf3700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4419d110 con 0x7f9c4406a370 2024-03-11T19:09:14.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c4acf3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4419d2d0 con 0x7f9c44107cf0 2024-03-11T19:09:14.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c43fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c4406ad10 0x7f9c44198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:14.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c43fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c4406ad10 0x7f9c44198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57624/0 (socket says 172.21.15.29:57624) 2024-03-11T19:09:14.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c43fff700 1 -- 172.21.15.29:0/3414536280 learned_addr learned my addr 172.21.15.29:0/3414536280 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:14.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c48a8f700 1 --2- 172.21.15.29:0/3414536280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c4406a370 0x7f9c441980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c49290700 1 --2- 172.21.15.29:0/3414536280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c44107cf0 0x7f9c4419c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c43fff700 1 -- 172.21.15.29:0/3414536280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c44107cf0 msgr2=0x7f9c4419c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c43fff700 1 --2- 172.21.15.29:0/3414536280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c44107cf0 0x7f9c4419c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c43fff700 1 -- 172.21.15.29:0/3414536280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c4406a370 msgr2=0x7f9c441980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c43fff700 1 --2- 172.21.15.29:0/3414536280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c4406a370 0x7f9c441980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c43fff700 1 -- 172.21.15.29:0/3414536280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c300095f0 con 0x7f9c4406ad10 2024-03-11T19:09:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c49290700 1 --2- 172.21.15.29:0/3414536280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c44107cf0 0x7f9c4419c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c43fff700 1 --2- 172.21.15.29:0/3414536280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c4406ad10 0x7f9c44198600 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9c300098e0 tx=0x7f9c30005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.260+0000 7f9c41ffb700 1 -- 172.21.15.29:0/3414536280 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3001d070 con 0x7f9c4406ad10 2024-03-11T19:09:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.261+0000 7f9c4acf3700 1 -- 172.21.15.29:0/3414536280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4419d550 con 0x7f9c4406ad10 2024-03-11T19:09:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.261+0000 7f9c41ffb700 1 -- 172.21.15.29:0/3414536280 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9c3000fb20 con 0x7f9c4406ad10 2024-03-11T19:09:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.261+0000 7f9c41ffb700 1 -- 172.21.15.29:0/3414536280 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c30022cd0 con 0x7f9c4406ad10 2024-03-11T19:09:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.261+0000 7f9c4acf3700 1 -- 172.21.15.29:0/3414536280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c441a4e10 con 0x7f9c4406ad10 2024-03-11T19:09:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.262+0000 7f9c4acf3700 1 -- 172.21.15.29:0/3414536280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c4410a210 con 0x7f9c4406ad10 2024-03-11T19:09:14.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.263+0000 7f9c41ffb700 1 -- 172.21.15.29:0/3414536280 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f9c3000fc90 con 0x7f9c4406ad10 2024-03-11T19:09:14.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.264+0000 7f9c41ffb700 1 --2- 172.21.15.29:0/3414536280 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9c340a1b50 0x7f9c340a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:14.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.264+0000 7f9c48a8f700 1 --2- 172.21.15.29:0/3414536280 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9c340a1b50 0x7f9c340a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:14.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.264+0000 7f9c41ffb700 1 -- 172.21.15.29:0/3414536280 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9c300ce7e0 con 0x7f9c4406ad10 2024-03-11T19:09:14.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.264+0000 7f9c48a8f700 1 --2- 172.21.15.29:0/3414536280 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9c340a1b50 0x7f9c340a4010 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800abd0 tx=0x7f9c3800a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:14.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.269+0000 7f9c41ffb700 1 -- 172.21.15.29:0/3414536280 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9c30097a20 con 0x7f9c4406ad10 2024-03-11T19:09:14.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.465+0000 7f9c4acf3700 1 -- 172.21.15.29:0/3414536280 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9c44199420 con 0x7f9c340a1b50 2024-03-11T19:09:14.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.472+0000 7f9c41ffb700 1 -- 172.21.15.29:0/3414536280 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258965 (secure 0 0 0) 0x7f9c44199420 con 0x7f9c340a1b50 2024-03-11T19:09:14.480 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:14.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.475+0000 7f9c4acf3700 1 -- 172.21.15.29:0/3414536280 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9c340a1b50 msgr2=0x7f9c340a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.475+0000 7f9c4acf3700 1 --2- 172.21.15.29:0/3414536280 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9c340a1b50 0x7f9c340a4010 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800abd0 tx=0x7f9c3800a380 comp rx=0 tx=0).stop 2024-03-11T19:09:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.475+0000 7f9c4acf3700 1 -- 172.21.15.29:0/3414536280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c4406ad10 msgr2=0x7f9c44198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.475+0000 7f9c4acf3700 1 --2- 172.21.15.29:0/3414536280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c4406ad10 0x7f9c44198600 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9c300098e0 tx=0x7f9c30005740 comp rx=0 tx=0).stop 2024-03-11T19:09:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.475+0000 7f9c4acf3700 1 -- 172.21.15.29:0/3414536280 shutdown_connections 2024-03-11T19:09:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.475+0000 7f9c4acf3700 1 --2- 172.21.15.29:0/3414536280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c44107cf0 0x7f9c4419c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.475+0000 7f9c4acf3700 1 --2- 172.21.15.29:0/3414536280 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9c340a1b50 0x7f9c340a4010 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.475+0000 7f9c4acf3700 1 --2- 172.21.15.29:0/3414536280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c4406a370 0x7f9c441980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.475+0000 7f9c4acf3700 1 --2- 172.21.15.29:0/3414536280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c4406ad10 0x7f9c44198600 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.475+0000 7f9c4acf3700 1 -- 172.21.15.29:0/3414536280 >> 172.21.15.29:0/3414536280 conn(0x7f9c44076bc0 msgr2=0x7f9c440ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.475+0000 7f9c4acf3700 1 -- 172.21.15.29:0/3414536280 shutdown_connections 2024-03-11T19:09:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:14.475+0000 7f9c4acf3700 1 -- 172.21.15.29:0/3414536280 wait complete. 2024-03-11T19:09:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-11T19:09:15.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:15 smithi170 ceph-mon[35690]: pgmap v334: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:09:15.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:15 smithi170 ceph-mon[35690]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:09:15.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:15 smithi029 ceph-mon[32439]: pgmap v334: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:09:15.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:15 smithi029 ceph-mon[32439]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:09:15.395 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-03-11T19:09:13.624828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658461,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":910,"num_read_kb":577,"num_write":420,"num_write_kb":3073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":345,"ondisk_log_size":345,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":394452,"kb_used_data":11940,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124547372,"statfs":{"total":1151940427776,"available":1151536508928,"internally_reserved":0,"allocated":12226560,"data_stored":7531119,"data_compressed":112974,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":391492922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":65,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002996"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.988138+0000","last_change":"2024-03-11T19:07:49.751738+0000","last_active":"2024-03-11T19:08:44.988138+0000","last_peered":"2024-03-11T19:08:44.988138+0000","last_clean":"2024-03-11T19:08:44.988138+0000","last_became_active":"2024-03-11T19:07:49.751173+0000","last_became_peered":"2024-03-11T19:07:49.751173+0000","last_unstale":"2024-03-11T19:08:44.988138+0000","last_undegraded":"2024-03-11T19:08:44.988138+0000","last_fullsized":"2024-03-11T19:08:44.988138+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:13:40.157763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991611+0000","last_change":"2024-03-11T19:07:49.749069+0000","last_active":"2024-03-11T19:08:44.991611+0000","last_peered":"2024-03-11T19:08:44.991611+0000","last_clean":"2024-03-11T19:08:44.991611+0000","last_became_active":"2024-03-11T19:07:49.747672+0000","last_became_peered":"2024-03-11T19:07:49.747672+0000","last_unstale":"2024-03-11T19:08:44.991611+0000","last_undegraded":"2024-03-11T19:08:44.991611+0000","last_fullsized":"2024-03-11T19:08:44.991611+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:57:26.379836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990986+0000","last_change":"2024-03-11T19:07:49.750298+0000","last_active":"2024-03-11T19:08:44.990986+0000","last_peered":"2024-03-11T19:08:44.990986+0000","last_clean":"2024-03-11T19:08:44.990986+0000","last_became_active":"2024-03-11T19:07:49.749985+0000","last_became_peered":"2024-03-11T19:07:49.749985+0000","last_unstale":"2024-03-11T19:08:44.990986+0000","last_undegraded":"2024-03-11T19:08:44.990986+0000","last_fullsized":"2024-03-11T19:08:44.990986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:56.579872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.993461+0000","last_change":"2024-03-11T19:08:44.993047+0000","last_active":"2024-03-11T19:08:44.993461+0000","last_peered":"2024-03-11T19:08:44.993461+0000","last_clean":"2024-03-11T19:08:44.993461+0000","last_became_active":"2024-03-11T19:08:44.992666+0000","last_became_peered":"2024-03-11T19:08:44.992666+0000","last_unstale":"2024-03-11T19:08:44.993461+0000","last_undegraded":"2024-03-11T19:08:44.993461+0000","last_fullsized":"2024-03-11T19:08:44.993461+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:32:18.748906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990386+0000","last_change":"2024-03-11T19:07:49.747618+0000","last_active":"2024-03-11T19:08:44.990386+0000","last_peered":"2024-03-11T19:08:44.990386+0000","last_clean":"2024-03-11T19:08:44.990386+0000","last_became_active":"2024-03-11T19:07:49.747310+0000","last_became_peered":"2024-03-11T19:07:49.747310+0000","last_unstale":"2024-03-11T19:08:44.990386+0000","last_undegraded":"2024-03-11T19:08:44.990386+0000","last_fullsized":"2024-03-11T19:08:44.990386+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:15:42.260509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037372+0000","last_change":"2024-03-11T19:07:49.755564+0000","last_active":"2024-03-11T19:08:45.037372+0000","last_peered":"2024-03-11T19:08:45.037372+0000","last_clean":"2024-03-11T19:08:45.037372+0000","last_became_active":"2024-03-11T19:07:49.751322+0000","last_became_peered":"2024-03-11T19:07:49.751322+0000","last_unstale":"2024-03-11T19:08:45.037372+0000","last_undegraded":"2024-03-11T19:08:45.037372+0000","last_fullsized":"2024-03-11T19:08:45.037372+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:43:18.487887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985513+0000","last_change":"2024-03-11T19:07:49.731619+0000","last_active":"2024-03-11T19:08:44.985513+0000","last_peered":"2024-03-11T19:08:44.985513+0000","last_clean":"2024-03-11T19:08:44.985513+0000","last_became_active":"2024-03-11T19:07:49.731323+0000","last_became_peered":"2024-03-11T19:07:49.731323+0000","last_unstale":"2024-03-11T19:08:44.985513+0000","last_undegraded":"2024-03-11T19:08:44.985513+0000","last_fullsized":"2024-03-11T19:08:44.985513+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:20:25.592709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.989401+0000","last_change":"2024-03-11T19:07:49.751851+0000","last_active":"2024-03-11T19:08:44.989401+0000","last_peered":"2024-03-11T19:08:44.989401+0000","last_clean":"2024-03-11T19:08:44.989401+0000","last_became_active":"2024-03-11T19:07:49.750068+0000","last_became_peered":"2024-03-11T19:07:49.750068+0000","last_unstale":"2024-03-11T19:08:44.989401+0000","last_undegraded":"2024-03-11T19:08:44.989401+0000","last_fullsized":"2024-03-11T19:08:44.989401+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:59:35.820107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037172+0000","last_change":"2024-03-11T19:07:49.752326+0000","last_active":"2024-03-11T19:08:45.037172+0000","last_peered":"2024-03-11T19:08:45.037172+0000","last_clean":"2024-03-11T19:08:45.037172+0000","last_became_active":"2024-03-11T19:07:49.748052+0000","last_became_peered":"2024-03-11T19:07:49.748052+0000","last_unstale":"2024-03-11T19:08:45.037172+0000","last_undegraded":"2024-03-11T19:08:45.037172+0000","last_fullsized":"2024-03-11T19:08:45.037172+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:58:13.325964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.038017+0000","last_change":"2024-03-11T19:07:49.748054+0000","last_active":"2024-03-11T19:08:45.038017+0000","last_peered":"2024-03-11T19:08:45.038017+0000","last_clean":"2024-03-11T19:08:45.038017+0000","last_became_active":"2024-03-11T19:07:49.747765+0000","last_became_peered":"2024-03-11T19:07:49.747765+0000","last_unstale":"2024-03-11T19:08:45.038017+0000","last_undegraded":"2024-03-11T19:08:45.038017+0000","last_fullsized":"2024-03-11T19:08:45.038017+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:18:05.599512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985983+0000","last_change":"2024-03-11T19:07:49.751622+0000","last_active":"2024-03-11T19:08:44.985983+0000","last_peered":"2024-03-11T19:08:44.985983+0000","last_clean":"2024-03-11T19:08:44.985983+0000","last_became_active":"2024-03-11T19:07:49.751317+0000","last_became_peered":"2024-03-11T19:07:49.751317+0000","last_unstale":"2024-03-11T19:08:44.985983+0000","last_undegraded":"2024-03-11T19:08:44.985983+0000","last_fullsized":"2024-03-11T19:08:44.985983+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:49.836096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037533+0000","last_change":"2024-03-11T19:07:49.749160+0000","last_active":"2024-03-11T19:08:45.037533+0000","last_peered":"2024-03-11T19:08:45.037533+0000","last_clean":"2024-03-11T19:08:45.037533+0000","last_became_active":"2024-03-11T19:07:49.748838+0000","last_became_peered":"2024-03-11T19:07:49.748838+0000","last_unstale":"2024-03-11T19:08:45.037533+0000","last_undegraded":"2024-03-11T19:08:45.037533+0000","last_fullsized":"2024-03-11T19:08:45.037533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:30:58.327196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.988839+0000","last_change":"2024-03-11T19:07:49.749965+0000","last_active":"2024-03-11T19:08:44.988839+0000","last_peered":"2024-03-11T19:08:44.988839+0000","last_clean":"2024-03-11T19:08:44.988839+0000","last_became_active":"2024-03-11T19:07:49.749602+0000","last_became_peered":"2024-03-11T19:07:49.749602+0000","last_unstale":"2024-03-11T19:08:44.988839+0000","last_undegraded":"2024-03-11T19:08:44.988839+0000","last_fullsized":"2024-03-11T19:08:44.988839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:36:08.807251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990351+0000","last_change":"2024-03-11T19:07:49.747557+0000","last_active":"2024-03-11T19:08:44.990351+0000","last_peered":"2024-03-11T19:08:44.990351+0000","last_clean":"2024-03-11T19:08:44.990351+0000","last_became_active":"2024-03-11T19:07:49.745804+0000","last_became_peered":"2024-03-11T19:07:49.745804+0000","last_unstale":"2024-03-11T19:08:44.990351+0000","last_undegraded":"2024-03-11T19:08:44.990351+0000","last_fullsized":"2024-03-11T19:08:44.990351+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:07:26.939414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990225+0000","last_change":"2024-03-11T19:07:49.744070+0000","last_active":"2024-03-11T19:08:44.990225+0000","last_peered":"2024-03-11T19:08:44.990225+0000","last_clean":"2024-03-11T19:08:44.990225+0000","last_became_active":"2024-03-11T19:07:49.743736+0000","last_became_peered":"2024-03-11T19:07:49.743736+0000","last_unstale":"2024-03-11T19:08:44.990225+0000","last_undegraded":"2024-03-11T19:08:44.990225+0000","last_fullsized":"2024-03-11T19:08:44.990225+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:50:33.862644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.035758+0000","last_change":"2024-03-11T19:07:49.748512+0000","last_active":"2024-03-11T19:08:45.035758+0000","last_peered":"2024-03-11T19:08:45.035758+0000","last_clean":"2024-03-11T19:08:45.035758+0000","last_became_active":"2024-03-11T19:07:49.748223+0000","last_became_peered":"2024-03-11T19:07:49.748223+0000","last_unstale":"2024-03-11T19:08:45.035758+0000","last_undegraded":"2024-03-11T19:08:45.035758+0000","last_fullsized":"2024-03-11T19:08:45.035758+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:49:43.305677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.988285+0000","last_change":"2024-03-11T19:07:49.754322+0000","last_active":"2024-03-11T19:08:44.988285+0000","last_peered":"2024-03-11T19:08:44.988285+0000","last_clean":"2024-03-11T19:08:44.988285+0000","last_became_active":"2024-03-11T19:07:49.753757+0000","last_became_peered":"2024-03-11T19:07:49.753757+0000","last_unstale":"2024-03-11T19:08:44.988285+0000","last_undegraded":"2024-03-11T19:08:44.988285+0000","last_fullsized":"2024-03-11T19:08:44.988285+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:49.684978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985342+0000","last_change":"2024-03-11T19:07:49.749546+0000","last_active":"2024-03-11T19:08:44.985342+0000","last_peered":"2024-03-11T19:08:44.985342+0000","last_clean":"2024-03-11T19:08:44.985342+0000","last_became_active":"2024-03-11T19:07:49.749200+0000","last_became_peered":"2024-03-11T19:07:49.749200+0000","last_unstale":"2024-03-11T19:08:44.985342+0000","last_undegraded":"2024-03-11T19:08:44.985342+0000","last_fullsized":"2024-03-11T19:08:44.985342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:26:12.914015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.986039+0000","last_change":"2024-03-11T19:07:49.750388+0000","last_active":"2024-03-11T19:08:44.986039+0000","last_peered":"2024-03-11T19:08:44.986039+0000","last_clean":"2024-03-11T19:08:44.986039+0000","last_became_active":"2024-03-11T19:07:49.750016+0000","last_became_peered":"2024-03-11T19:07:49.750016+0000","last_unstale":"2024-03-11T19:08:44.986039+0000","last_undegraded":"2024-03-11T19:08:44.986039+0000","last_fullsized":"2024-03-11T19:08:44.986039+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:32:23.125695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'13","reported_seq":108,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:09.771096+0000","last_change":"2024-03-11T19:07:48.738369+0000","last_active":"2024-03-11T19:09:09.771096+0000","last_peered":"2024-03-11T19:09:09.771096+0000","last_clean":"2024-03-11T19:09:09.771096+0000","last_became_active":"2024-03-11T19:07:48.738010+0000","last_became_peered":"2024-03-11T19:07:48.738010+0000","last_unstale":"2024-03-11T19:09:09.771096+0000","last_undegraded":"2024-03-11T19:09:09.771096+0000","last_fullsized":"2024-03-11T19:09:09.771096+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:06:48.446454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":76,"num_read_kb":76,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990356+0000","last_change":"2024-03-11T19:07:49.752622+0000","last_active":"2024-03-11T19:08:44.990356+0000","last_peered":"2024-03-11T19:08:44.990356+0000","last_clean":"2024-03-11T19:08:44.990356+0000","last_became_active":"2024-03-11T19:07:49.752242+0000","last_became_peered":"2024-03-11T19:07:49.752242+0000","last_unstale":"2024-03-11T19:08:44.990356+0000","last_undegraded":"2024-03-11T19:08:44.990356+0000","last_fullsized":"2024-03-11T19:08:44.990356+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:03:50.609641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990462+0000","last_change":"2024-03-11T19:07:48.751657+0000","last_active":"2024-03-11T19:08:44.990462+0000","last_peered":"2024-03-11T19:08:44.990462+0000","last_clean":"2024-03-11T19:08:44.990462+0000","last_became_active":"2024-03-11T19:07:48.751294+0000","last_became_peered":"2024-03-11T19:07:48.751294+0000","last_unstale":"2024-03-11T19:08:44.990462+0000","last_undegraded":"2024-03-11T19:08:44.990462+0000","last_fullsized":"2024-03-11T19:08:44.990462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:33:58.550707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991370+0000","last_change":"2024-03-11T19:07:49.759460+0000","last_active":"2024-03-11T19:08:44.991370+0000","last_peered":"2024-03-11T19:08:44.991370+0000","last_clean":"2024-03-11T19:08:44.991370+0000","last_became_active":"2024-03-11T19:07:49.759136+0000","last_became_peered":"2024-03-11T19:07:49.759136+0000","last_unstale":"2024-03-11T19:08:44.991370+0000","last_undegraded":"2024-03-11T19:08:44.991370+0000","last_fullsized":"2024-03-11T19:08:44.991370+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:10:37.273896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'12","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991106+0000","last_change":"2024-03-11T19:07:48.744695+0000","last_active":"2024-03-11T19:08:44.991106+0000","last_peered":"2024-03-11T19:08:44.991106+0000","last_clean":"2024-03-11T19:08:44.991106+0000","last_became_active":"2024-03-11T19:07:48.744378+0000","last_became_peered":"2024-03-11T19:07:48.744378+0000","last_unstale":"2024-03-11T19:08:44.991106+0000","last_undegraded":"2024-03-11T19:08:44.991106+0000","last_fullsized":"2024-03-11T19:08:44.991106+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:55:46.601414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.035982+0000","last_change":"2024-03-11T19:07:49.761429+0000","last_active":"2024-03-11T19:08:45.035982+0000","last_peered":"2024-03-11T19:08:45.035982+0000","last_clean":"2024-03-11T19:08:45.035982+0000","last_became_active":"2024-03-11T19:07:49.761120+0000","last_became_peered":"2024-03-11T19:07:49.761120+0000","last_unstale":"2024-03-11T19:08:45.035982+0000","last_undegraded":"2024-03-11T19:08:45.035982+0000","last_fullsized":"2024-03-11T19:08:45.035982+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:13:45.839528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990713+0000","last_change":"2024-03-11T19:07:48.740451+0000","last_active":"2024-03-11T19:08:44.990713+0000","last_peered":"2024-03-11T19:08:44.990713+0000","last_clean":"2024-03-11T19:08:44.990713+0000","last_became_active":"2024-03-11T19:07:48.740097+0000","last_became_peered":"2024-03-11T19:07:48.740097+0000","last_unstale":"2024-03-11T19:08:44.990713+0000","last_undegraded":"2024-03-11T19:08:44.990713+0000","last_fullsized":"2024-03-11T19:08:44.990713+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:37.766790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036530+0000","last_change":"2024-03-11T19:07:49.750404+0000","last_active":"2024-03-11T19:08:45.036530+0000","last_peered":"2024-03-11T19:08:45.036530+0000","last_clean":"2024-03-11T19:08:45.036530+0000","last_became_active":"2024-03-11T19:07:49.749941+0000","last_became_peered":"2024-03-11T19:07:49.749941+0000","last_unstale":"2024-03-11T19:08:45.036530+0000","last_undegraded":"2024-03-11T19:08:45.036530+0000","last_fullsized":"2024-03-11T19:08:45.036530+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:55.766729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'6","reported_seq":99,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:06.768459+0000","last_change":"2024-03-11T19:07:48.737100+0000","last_active":"2024-03-11T19:09:06.768459+0000","last_peered":"2024-03-11T19:09:06.768459+0000","last_clean":"2024-03-11T19:09:06.768459+0000","last_became_active":"2024-03-11T19:07:48.736768+0000","last_became_peered":"2024-03-11T19:07:48.736768+0000","last_unstale":"2024-03-11T19:09:06.768459+0000","last_undegraded":"2024-03-11T19:09:06.768459+0000","last_fullsized":"2024-03-11T19:09:06.768459+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:18:36.872151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990388+0000","last_change":"2024-03-11T19:07:49.739020+0000","last_active":"2024-03-11T19:08:44.990388+0000","last_peered":"2024-03-11T19:08:44.990388+0000","last_clean":"2024-03-11T19:08:44.990388+0000","last_became_active":"2024-03-11T19:07:49.738642+0000","last_became_peered":"2024-03-11T19:07:49.738642+0000","last_unstale":"2024-03-11T19:08:44.990388+0000","last_undegraded":"2024-03-11T19:08:44.990388+0000","last_fullsized":"2024-03-11T19:08:44.990388+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:42:29.040242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037233+0000","last_change":"2024-03-11T19:07:48.745192+0000","last_active":"2024-03-11T19:08:45.037233+0000","last_peered":"2024-03-11T19:08:45.037233+0000","last_clean":"2024-03-11T19:08:45.037233+0000","last_became_active":"2024-03-11T19:07:48.744868+0000","last_became_peered":"2024-03-11T19:07:48.744868+0000","last_unstale":"2024-03-11T19:08:45.037233+0000","last_undegraded":"2024-03-11T19:08:45.037233+0000","last_fullsized":"2024-03-11T19:08:45.037233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:19:15.620186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036228+0000","last_change":"2024-03-11T19:07:49.737149+0000","last_active":"2024-03-11T19:08:45.036228+0000","last_peered":"2024-03-11T19:08:45.036228+0000","last_clean":"2024-03-11T19:08:45.036228+0000","last_became_active":"2024-03-11T19:07:49.736797+0000","last_became_peered":"2024-03-11T19:07:49.736797+0000","last_unstale":"2024-03-11T19:08:45.036228+0000","last_undegraded":"2024-03-11T19:08:45.036228+0000","last_fullsized":"2024-03-11T19:08:45.036228+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:12:10.282062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.988596+0000","last_change":"2024-03-11T19:07:48.743948+0000","last_active":"2024-03-11T19:08:44.988596+0000","last_peered":"2024-03-11T19:08:44.988596+0000","last_clean":"2024-03-11T19:08:44.988596+0000","last_became_active":"2024-03-11T19:07:48.743451+0000","last_became_peered":"2024-03-11T19:07:48.743451+0000","last_unstale":"2024-03-11T19:08:44.988596+0000","last_undegraded":"2024-03-11T19:08:44.988596+0000","last_fullsized":"2024-03-11T19:08:44.988596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:03:19.823287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.989398+0000","last_change":"2024-03-11T19:07:49.745518+0000","last_active":"2024-03-11T19:08:44.989398+0000","last_peered":"2024-03-11T19:08:44.989398+0000","last_clean":"2024-03-11T19:08:44.989398+0000","last_became_active":"2024-03-11T19:07:49.743719+0000","last_became_peered":"2024-03-11T19:07:49.743719+0000","last_unstale":"2024-03-11T19:08:44.989398+0000","last_undegraded":"2024-03-11T19:08:44.989398+0000","last_fullsized":"2024-03-11T19:08:44.989398+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:25:33.855679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036218+0000","last_change":"2024-03-11T19:07:48.734044+0000","last_active":"2024-03-11T19:08:45.036218+0000","last_peered":"2024-03-11T19:08:45.036218+0000","last_clean":"2024-03-11T19:08:45.036218+0000","last_became_active":"2024-03-11T19:07:48.733723+0000","last_became_peered":"2024-03-11T19:07:48.733723+0000","last_unstale":"2024-03-11T19:08:45.036218+0000","last_undegraded":"2024-03-11T19:08:45.036218+0000","last_fullsized":"2024-03-11T19:08:45.036218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:53:37.070296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985073+0000","last_change":"2024-03-11T19:07:49.743729+0000","last_active":"2024-03-11T19:08:44.985073+0000","last_peered":"2024-03-11T19:08:44.985073+0000","last_clean":"2024-03-11T19:08:44.985073+0000","last_became_active":"2024-03-11T19:07:49.742992+0000","last_became_peered":"2024-03-11T19:07:49.742992+0000","last_unstale":"2024-03-11T19:08:44.985073+0000","last_undegraded":"2024-03-11T19:08:44.985073+0000","last_fullsized":"2024-03-11T19:08:44.985073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:46.477177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'4","reported_seq":175,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:11.774632+0000","last_change":"2024-03-11T19:07:48.738775+0000","last_active":"2024-03-11T19:09:11.774632+0000","last_peered":"2024-03-11T19:09:11.774632+0000","last_clean":"2024-03-11T19:09:11.774632+0000","last_became_active":"2024-03-11T19:07:48.738437+0000","last_became_peered":"2024-03-11T19:07:48.738437+0000","last_unstale":"2024-03-11T19:09:11.774632+0000","last_undegraded":"2024-03-11T19:09:11.774632+0000","last_fullsized":"2024-03-11T19:09:11.774632+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:17:20.020756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":152,"num_read_kb":152,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036862+0000","last_change":"2024-03-11T19:07:49.760293+0000","last_active":"2024-03-11T19:08:45.036862+0000","last_peered":"2024-03-11T19:08:45.036862+0000","last_clean":"2024-03-11T19:08:45.036862+0000","last_became_active":"2024-03-11T19:07:49.759977+0000","last_became_peered":"2024-03-11T19:07:49.759977+0000","last_unstale":"2024-03-11T19:08:45.036862+0000","last_undegraded":"2024-03-11T19:08:45.036862+0000","last_fullsized":"2024-03-11T19:08:45.036862+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:07:44.726756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985956+0000","last_change":"2024-03-11T19:07:48.740672+0000","last_active":"2024-03-11T19:08:44.985956+0000","last_peered":"2024-03-11T19:08:44.985956+0000","last_clean":"2024-03-11T19:08:44.985956+0000","last_became_active":"2024-03-11T19:07:48.739331+0000","last_became_peered":"2024-03-11T19:07:48.739331+0000","last_unstale":"2024-03-11T19:08:44.985956+0000","last_undegraded":"2024-03-11T19:08:44.985956+0000","last_fullsized":"2024-03-11T19:08:44.985956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:19:28.375515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991268+0000","last_change":"2024-03-11T19:07:49.742780+0000","last_active":"2024-03-11T19:08:44.991268+0000","last_peered":"2024-03-11T19:08:44.991268+0000","last_clean":"2024-03-11T19:08:44.991268+0000","last_became_active":"2024-03-11T19:07:49.742442+0000","last_became_peered":"2024-03-11T19:07:49.742442+0000","last_unstale":"2024-03-11T19:08:44.991268+0000","last_undegraded":"2024-03-11T19:08:44.991268+0000","last_fullsized":"2024-03-11T19:08:44.991268+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:11:13.838225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036582+0000","last_change":"2024-03-11T19:07:48.752617+0000","last_active":"2024-03-11T19:08:45.036582+0000","last_peered":"2024-03-11T19:08:45.036582+0000","last_clean":"2024-03-11T19:08:45.036582+0000","last_became_active":"2024-03-11T19:07:48.752181+0000","last_became_peered":"2024-03-11T19:07:48.752181+0000","last_unstale":"2024-03-11T19:08:45.036582+0000","last_undegraded":"2024-03-11T19:08:45.036582+0000","last_fullsized":"2024-03-11T19:08:45.036582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:05:37.162866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036332+0000","last_change":"2024-03-11T19:07:49.753672+0000","last_active":"2024-03-11T19:08:45.036332+0000","last_peered":"2024-03-11T19:08:45.036332+0000","last_clean":"2024-03-11T19:08:45.036332+0000","last_became_active":"2024-03-11T19:07:49.753315+0000","last_became_peered":"2024-03-11T19:07:49.753315+0000","last_unstale":"2024-03-11T19:08:45.036332+0000","last_undegraded":"2024-03-11T19:08:45.036332+0000","last_fullsized":"2024-03-11T19:08:45.036332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:30:58.503249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'31","reported_seq":209,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:11.774636+0000","last_change":"2024-03-11T19:07:48.740424+0000","last_active":"2024-03-11T19:09:11.774636+0000","last_peered":"2024-03-11T19:09:11.774636+0000","last_clean":"2024-03-11T19:09:11.774636+0000","last_became_active":"2024-03-11T19:07:48.740077+0000","last_became_peered":"2024-03-11T19:07:48.740077+0000","last_unstale":"2024-03-11T19:09:11.774636+0000","last_undegraded":"2024-03-11T19:09:11.774636+0000","last_fullsized":"2024-03-11T19:09:11.774636+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:30:29.193861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":159,"num_read_kb":139,"num_write":33,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990971+0000","last_change":"2024-03-11T19:07:49.746440+0000","last_active":"2024-03-11T19:08:44.990971+0000","last_peered":"2024-03-11T19:08:44.990971+0000","last_clean":"2024-03-11T19:08:44.990971+0000","last_became_active":"2024-03-11T19:07:49.746106+0000","last_became_peered":"2024-03-11T19:07:49.746106+0000","last_unstale":"2024-03-11T19:08:44.990971+0000","last_undegraded":"2024-03-11T19:08:44.990971+0000","last_fullsized":"2024-03-11T19:08:44.990971+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:44:58.039917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'9","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.986027+0000","last_change":"2024-03-11T19:07:48.744437+0000","last_active":"2024-03-11T19:08:44.986027+0000","last_peered":"2024-03-11T19:08:44.986027+0000","last_clean":"2024-03-11T19:08:44.986027+0000","last_became_active":"2024-03-11T19:07:48.744142+0000","last_became_peered":"2024-03-11T19:07:48.744142+0000","last_unstale":"2024-03-11T19:08:44.986027+0000","last_undegraded":"2024-03-11T19:08:44.986027+0000","last_fullsized":"2024-03-11T19:08:44.986027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:32:09.076975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.988450+0000","last_change":"2024-03-11T19:07:49.751191+0000","last_active":"2024-03-11T19:08:44.988450+0000","last_peered":"2024-03-11T19:08:44.988450+0000","last_clean":"2024-03-11T19:08:44.988450+0000","last_became_active":"2024-03-11T19:07:49.750849+0000","last_became_peered":"2024-03-11T19:07:49.750849+0000","last_unstale":"2024-03-11T19:08:44.988450+0000","last_undegraded":"2024-03-11T19:08:44.988450+0000","last_fullsized":"2024-03-11T19:08:44.988450+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:19:18.416905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036229+0000","last_change":"2024-03-11T19:07:48.742808+0000","last_active":"2024-03-11T19:08:45.036229+0000","last_peered":"2024-03-11T19:08:45.036229+0000","last_clean":"2024-03-11T19:08:45.036229+0000","last_became_active":"2024-03-11T19:07:48.742462+0000","last_became_peered":"2024-03-11T19:07:48.742462+0000","last_unstale":"2024-03-11T19:08:45.036229+0000","last_undegraded":"2024-03-11T19:08:45.036229+0000","last_fullsized":"2024-03-11T19:08:45.036229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:19.109299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985856+0000","last_change":"2024-03-11T19:07:49.744673+0000","last_active":"2024-03-11T19:08:44.985856+0000","last_peered":"2024-03-11T19:08:44.985856+0000","last_clean":"2024-03-11T19:08:44.985856+0000","last_became_active":"2024-03-11T19:07:49.744313+0000","last_became_peered":"2024-03-11T19:07:49.744313+0000","last_unstale":"2024-03-11T19:08:44.985856+0000","last_undegraded":"2024-03-11T19:08:44.985856+0000","last_fullsized":"2024-03-11T19:08:44.985856+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:21:04.759356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'5","reported_seq":99,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:07.772645+0000","last_change":"2024-03-11T19:07:48.736024+0000","last_active":"2024-03-11T19:09:07.772645+0000","last_peered":"2024-03-11T19:09:07.772645+0000","last_clean":"2024-03-11T19:09:07.772645+0000","last_became_active":"2024-03-11T19:07:48.735720+0000","last_became_peered":"2024-03-11T19:07:48.735720+0000","last_unstale":"2024-03-11T19:09:07.772645+0000","last_undegraded":"2024-03-11T19:09:07.772645+0000","last_fullsized":"2024-03-11T19:09:07.772645+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:04:57.386267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985463+0000","last_change":"2024-03-11T19:07:49.742924+0000","last_active":"2024-03-11T19:08:44.985463+0000","last_peered":"2024-03-11T19:08:44.985463+0000","last_clean":"2024-03-11T19:08:44.985463+0000","last_became_active":"2024-03-11T19:07:49.742542+0000","last_became_peered":"2024-03-11T19:07:49.742542+0000","last_unstale":"2024-03-11T19:08:44.985463+0000","last_undegraded":"2024-03-11T19:08:44.985463+0000","last_fullsized":"2024-03-11T19:08:44.985463+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:08:11.944311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'17","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990343+0000","last_change":"2024-03-11T19:07:48.753113+0000","last_active":"2024-03-11T19:08:44.990343+0000","last_peered":"2024-03-11T19:08:44.990343+0000","last_clean":"2024-03-11T19:08:44.990343+0000","last_became_active":"2024-03-11T19:07:48.752517+0000","last_became_peered":"2024-03-11T19:07:48.752517+0000","last_unstale":"2024-03-11T19:08:44.990343+0000","last_undegraded":"2024-03-11T19:08:44.990343+0000","last_fullsized":"2024-03-11T19:08:44.990343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:48:33.441125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.038561+0000","last_change":"2024-03-11T19:08:45.038243+0000","last_active":"2024-03-11T19:08:45.038561+0000","last_peered":"2024-03-11T19:08:45.038561+0000","last_clean":"2024-03-11T19:08:45.038561+0000","last_became_active":"2024-03-11T19:08:45.037969+0000","last_became_peered":"2024-03-11T19:08:45.037969+0000","last_unstale":"2024-03-11T19:08:45.038561+0000","last_undegraded":"2024-03-11T19:08:45.038561+0000","last_fullsized":"2024-03-11T19:08:45.038561+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:29:09.747958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991087+0000","last_change":"2024-03-11T19:07:48.739571+0000","last_active":"2024-03-11T19:08:44.991087+0000","last_peered":"2024-03-11T19:08:44.991087+0000","last_clean":"2024-03-11T19:08:44.991087+0000","last_became_active":"2024-03-11T19:07:48.739159+0000","last_became_peered":"2024-03-11T19:07:48.739159+0000","last_unstale":"2024-03-11T19:08:44.991087+0000","last_undegraded":"2024-03-11T19:08:44.991087+0000","last_fullsized":"2024-03-11T19:08:44.991087+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:46:43.474591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.987681+0000","last_change":"2024-03-11T19:07:49.744760+0000","last_active":"2024-03-11T19:08:44.987681+0000","last_peered":"2024-03-11T19:08:44.987681+0000","last_clean":"2024-03-11T19:08:44.987681+0000","last_became_active":"2024-03-11T19:07:49.744265+0000","last_became_peered":"2024-03-11T19:07:49.744265+0000","last_unstale":"2024-03-11T19:08:44.987681+0000","last_undegraded":"2024-03-11T19:08:44.987681+0000","last_fullsized":"2024-03-11T19:08:44.987681+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:09:49.823462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985159+0000","last_change":"2024-03-11T19:07:48.740223+0000","last_active":"2024-03-11T19:08:44.985159+0000","last_peered":"2024-03-11T19:08:44.985159+0000","last_clean":"2024-03-11T19:08:44.985159+0000","last_became_active":"2024-03-11T19:07:48.739569+0000","last_became_peered":"2024-03-11T19:07:48.739569+0000","last_unstale":"2024-03-11T19:08:44.985159+0000","last_undegraded":"2024-03-11T19:08:44.985159+0000","last_fullsized":"2024-03-11T19:08:44.985159+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:04:19.384523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.038056+0000","last_change":"2024-03-11T19:07:49.755620+0000","last_active":"2024-03-11T19:08:45.038056+0000","last_peered":"2024-03-11T19:08:45.038056+0000","last_clean":"2024-03-11T19:08:45.038056+0000","last_became_active":"2024-03-11T19:07:49.750886+0000","last_became_peered":"2024-03-11T19:07:49.750886+0000","last_unstale":"2024-03-11T19:08:45.038056+0000","last_undegraded":"2024-03-11T19:08:45.038056+0000","last_fullsized":"2024-03-11T19:08:45.038056+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:35:54.191061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'5","reported_seq":99,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:08.770279+0000","last_change":"2024-03-11T19:07:48.743340+0000","last_active":"2024-03-11T19:09:08.770279+0000","last_peered":"2024-03-11T19:09:08.770279+0000","last_clean":"2024-03-11T19:09:08.770279+0000","last_became_active":"2024-03-11T19:07:48.742947+0000","last_became_peered":"2024-03-11T19:07:48.742947+0000","last_unstale":"2024-03-11T19:09:08.770279+0000","last_undegraded":"2024-03-11T19:09:08.770279+0000","last_fullsized":"2024-03-11T19:09:08.770279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:40:25.233150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037599+0000","last_change":"2024-03-11T19:07:49.753668+0000","last_active":"2024-03-11T19:08:45.037599+0000","last_peered":"2024-03-11T19:08:45.037599+0000","last_clean":"2024-03-11T19:08:45.037599+0000","last_became_active":"2024-03-11T19:07:49.750574+0000","last_became_peered":"2024-03-11T19:07:49.750574+0000","last_unstale":"2024-03-11T19:08:45.037599+0000","last_undegraded":"2024-03-11T19:08:45.037599+0000","last_fullsized":"2024-03-11T19:08:45.037599+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:12:32.711847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036426+0000","last_change":"2024-03-11T19:07:49.739538+0000","last_active":"2024-03-11T19:08:45.036426+0000","last_peered":"2024-03-11T19:08:45.036426+0000","last_clean":"2024-03-11T19:08:45.036426+0000","last_became_active":"2024-03-11T19:07:49.738617+0000","last_became_peered":"2024-03-11T19:07:49.738617+0000","last_unstale":"2024-03-11T19:08:45.036426+0000","last_undegraded":"2024-03-11T19:08:45.036426+0000","last_fullsized":"2024-03-11T19:08:45.036426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:43:08.697125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":98,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:07.772140+0000","last_change":"2024-03-11T19:07:48.749527+0000","last_active":"2024-03-11T19:09:07.772140+0000","last_peered":"2024-03-11T19:09:07.772140+0000","last_clean":"2024-03-11T19:09:07.772140+0000","last_became_active":"2024-03-11T19:07:48.749026+0000","last_became_peered":"2024-03-11T19:07:48.749026+0000","last_unstale":"2024-03-11T19:09:07.772140+0000","last_undegraded":"2024-03-11T19:09:07.772140+0000","last_fullsized":"2024-03-11T19:09:07.772140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:20:19.058429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990018+0000","last_change":"2024-03-11T19:07:49.757696+0000","last_active":"2024-03-11T19:08:44.990018+0000","last_peered":"2024-03-11T19:08:44.990018+0000","last_clean":"2024-03-11T19:08:44.990018+0000","last_became_active":"2024-03-11T19:07:49.757380+0000","last_became_peered":"2024-03-11T19:07:49.757380+0000","last_unstale":"2024-03-11T19:08:44.990018+0000","last_undegraded":"2024-03-11T19:08:44.990018+0000","last_fullsized":"2024-03-11T19:08:44.990018+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:03:22.954619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'6","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036756+0000","last_change":"2024-03-11T19:07:48.737504+0000","last_active":"2024-03-11T19:08:45.036756+0000","last_peered":"2024-03-11T19:08:45.036756+0000","last_clean":"2024-03-11T19:08:45.036756+0000","last_became_active":"2024-03-11T19:07:48.737147+0000","last_became_peered":"2024-03-11T19:07:48.737147+0000","last_unstale":"2024-03-11T19:08:45.036756+0000","last_undegraded":"2024-03-11T19:08:45.036756+0000","last_fullsized":"2024-03-11T19:08:45.036756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:04.819617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":343,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985537+0000","last_change":"2024-03-11T19:06:35.035452+0000","last_active":"2024-03-11T19:08:44.985537+0000","last_peered":"2024-03-11T19:08:44.985537+0000","last_clean":"2024-03-11T19:08:44.985537+0000","last_became_active":"2024-03-11T19:06:35.035076+0000","last_became_peered":"2024-03-11T19:06:35.035076+0000","last_unstale":"2024-03-11T19:08:44.985537+0000","last_undegraded":"2024-03-11T19:08:44.985537+0000","last_fullsized":"2024-03-11T19:08:44.985537+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:03:10.143002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:03:10.143002+0000","last_clean_scrub_stamp":"2024-03-11T19:03:10.143002+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:39:37.348091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.988339+0000","last_change":"2024-03-11T19:07:49.743000+0000","last_active":"2024-03-11T19:08:44.988339+0000","last_peered":"2024-03-11T19:08:44.988339+0000","last_clean":"2024-03-11T19:08:44.988339+0000","last_became_active":"2024-03-11T19:07:49.742691+0000","last_became_peered":"2024-03-11T19:07:49.742691+0000","last_unstale":"2024-03-11T19:08:44.988339+0000","last_undegraded":"2024-03-11T19:08:44.988339+0000","last_fullsized":"2024-03-11T19:08:44.988339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:40:21.519929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985665+0000","last_change":"2024-03-11T19:07:48.739038+0000","last_active":"2024-03-11T19:08:44.985665+0000","last_peered":"2024-03-11T19:08:44.985665+0000","last_clean":"2024-03-11T19:08:44.985665+0000","last_became_active":"2024-03-11T19:07:48.738714+0000","last_became_peered":"2024-03-11T19:07:48.738714+0000","last_unstale":"2024-03-11T19:08:44.985665+0000","last_undegraded":"2024-03-11T19:08:44.985665+0000","last_fullsized":"2024-03-11T19:08:44.985665+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:17:43.086679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.998660+0000","last_change":"2024-03-11T19:08:44.998660+0000","last_active":"2024-03-11T19:08:44.998660+0000","last_peered":"2024-03-11T19:08:44.998660+0000","last_clean":"2024-03-11T19:08:44.998660+0000","last_became_active":"2024-03-11T19:08:44.991682+0000","last_became_peered":"2024-03-11T19:08:44.991682+0000","last_unstale":"2024-03-11T19:08:44.998660+0000","last_undegraded":"2024-03-11T19:08:44.998660+0000","last_fullsized":"2024-03-11T19:08:44.998660+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:19:44.266076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.988920+0000","last_change":"2024-03-11T19:07:49.742434+0000","last_active":"2024-03-11T19:08:44.988920+0000","last_peered":"2024-03-11T19:08:44.988920+0000","last_clean":"2024-03-11T19:08:44.988920+0000","last_became_active":"2024-03-11T19:07:49.742048+0000","last_became_peered":"2024-03-11T19:07:49.742048+0000","last_unstale":"2024-03-11T19:08:44.988920+0000","last_undegraded":"2024-03-11T19:08:44.988920+0000","last_fullsized":"2024-03-11T19:08:44.988920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:39:22.685055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'12","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985474+0000","last_change":"2024-03-11T19:07:48.744728+0000","last_active":"2024-03-11T19:08:44.985474+0000","last_peered":"2024-03-11T19:08:44.985474+0000","last_clean":"2024-03-11T19:08:44.985474+0000","last_became_active":"2024-03-11T19:07:48.744378+0000","last_became_peered":"2024-03-11T19:07:48.744378+0000","last_unstale":"2024-03-11T19:08:44.985474+0000","last_undegraded":"2024-03-11T19:08:44.985474+0000","last_fullsized":"2024-03-11T19:08:44.985474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:42.823040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990691+0000","last_change":"2024-03-11T19:07:49.744261+0000","last_active":"2024-03-11T19:08:44.990691+0000","last_peered":"2024-03-11T19:08:44.990691+0000","last_clean":"2024-03-11T19:08:44.990691+0000","last_became_active":"2024-03-11T19:07:49.743946+0000","last_became_peered":"2024-03-11T19:07:49.743946+0000","last_unstale":"2024-03-11T19:08:44.990691+0000","last_undegraded":"2024-03-11T19:08:44.990691+0000","last_fullsized":"2024-03-11T19:08:44.990691+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:54:07.928899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037193+0000","last_change":"2024-03-11T19:07:48.740059+0000","last_active":"2024-03-11T19:08:45.037193+0000","last_peered":"2024-03-11T19:08:45.037193+0000","last_clean":"2024-03-11T19:08:45.037193+0000","last_became_active":"2024-03-11T19:07:48.739626+0000","last_became_peered":"2024-03-11T19:07:48.739626+0000","last_unstale":"2024-03-11T19:08:45.037193+0000","last_undegraded":"2024-03-11T19:08:45.037193+0000","last_fullsized":"2024-03-11T19:08:45.037193+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:45:12.219203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991612+0000","last_change":"2024-03-11T19:07:49.746041+0000","last_active":"2024-03-11T19:08:44.991612+0000","last_peered":"2024-03-11T19:08:44.991612+0000","last_clean":"2024-03-11T19:08:44.991612+0000","last_became_active":"2024-03-11T19:07:49.745602+0000","last_became_peered":"2024-03-11T19:07:49.745602+0000","last_unstale":"2024-03-11T19:08:44.991612+0000","last_undegraded":"2024-03-11T19:08:44.991612+0000","last_fullsized":"2024-03-11T19:08:44.991612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:23:44.596929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.035863+0000","last_change":"2024-03-11T19:07:48.748677+0000","last_active":"2024-03-11T19:08:45.035863+0000","last_peered":"2024-03-11T19:08:45.035863+0000","last_clean":"2024-03-11T19:08:45.035863+0000","last_became_active":"2024-03-11T19:07:48.748470+0000","last_became_peered":"2024-03-11T19:07:48.748470+0000","last_unstale":"2024-03-11T19:08:45.035863+0000","last_undegraded":"2024-03-11T19:08:45.035863+0000","last_fullsized":"2024-03-11T19:08:45.035863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:24:55.240722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990208+0000","last_change":"2024-03-11T19:07:49.746522+0000","last_active":"2024-03-11T19:08:44.990208+0000","last_peered":"2024-03-11T19:08:44.990208+0000","last_clean":"2024-03-11T19:08:44.990208+0000","last_became_active":"2024-03-11T19:07:49.746179+0000","last_became_peered":"2024-03-11T19:07:49.746179+0000","last_unstale":"2024-03-11T19:08:44.990208+0000","last_undegraded":"2024-03-11T19:08:44.990208+0000","last_fullsized":"2024-03-11T19:08:44.990208+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:06:08.144850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.989685+0000","last_change":"2024-03-11T19:07:48.740926+0000","last_active":"2024-03-11T19:08:44.989685+0000","last_peered":"2024-03-11T19:08:44.989685+0000","last_clean":"2024-03-11T19:08:44.989685+0000","last_became_active":"2024-03-11T19:07:48.740625+0000","last_became_peered":"2024-03-11T19:07:48.740625+0000","last_unstale":"2024-03-11T19:08:44.989685+0000","last_undegraded":"2024-03-11T19:08:44.989685+0000","last_fullsized":"2024-03-11T19:08:44.989685+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:50:13.079608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.987248+0000","last_change":"2024-03-11T19:07:49.742428+0000","last_active":"2024-03-11T19:08:44.987248+0000","last_peered":"2024-03-11T19:08:44.987248+0000","last_clean":"2024-03-11T19:08:44.987248+0000","last_became_active":"2024-03-11T19:07:49.742098+0000","last_became_peered":"2024-03-11T19:07:49.742098+0000","last_unstale":"2024-03-11T19:08:44.987248+0000","last_undegraded":"2024-03-11T19:08:44.987248+0000","last_fullsized":"2024-03-11T19:08:44.987248+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:19:24.615755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'3","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.002189+0000","last_change":"2024-03-11T19:08:45.002189+0000","last_active":"2024-03-11T19:08:45.002189+0000","last_peered":"2024-03-11T19:08:45.002189+0000","last_clean":"2024-03-11T19:08:45.002189+0000","last_became_active":"2024-03-11T19:08:44.992818+0000","last_became_peered":"2024-03-11T19:08:44.992818+0000","last_unstale":"2024-03-11T19:08:45.002189+0000","last_undegraded":"2024-03-11T19:08:45.002189+0000","last_fullsized":"2024-03-11T19:08:45.002189+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:34:54.461734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.989833+0000","last_change":"2024-03-11T19:07:49.754872+0000","last_active":"2024-03-11T19:08:44.989833+0000","last_peered":"2024-03-11T19:08:44.989833+0000","last_clean":"2024-03-11T19:08:44.989833+0000","last_became_active":"2024-03-11T19:07:49.749663+0000","last_became_peered":"2024-03-11T19:07:49.749663+0000","last_unstale":"2024-03-11T19:08:44.989833+0000","last_undegraded":"2024-03-11T19:08:44.989833+0000","last_fullsized":"2024-03-11T19:08:44.989833+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:22:23.664814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990526+0000","last_change":"2024-03-11T19:07:48.742227+0000","last_active":"2024-03-11T19:08:44.990526+0000","last_peered":"2024-03-11T19:08:44.990526+0000","last_clean":"2024-03-11T19:08:44.990526+0000","last_became_active":"2024-03-11T19:07:48.741912+0000","last_became_peered":"2024-03-11T19:07:48.741912+0000","last_unstale":"2024-03-11T19:08:44.990526+0000","last_undegraded":"2024-03-11T19:08:44.990526+0000","last_fullsized":"2024-03-11T19:08:44.990526+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:10:06.120433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985806+0000","last_change":"2024-03-11T19:07:49.748947+0000","last_active":"2024-03-11T19:08:44.985806+0000","last_peered":"2024-03-11T19:08:44.985806+0000","last_clean":"2024-03-11T19:08:44.985806+0000","last_became_active":"2024-03-11T19:07:49.748621+0000","last_became_peered":"2024-03-11T19:07:49.748621+0000","last_unstale":"2024-03-11T19:08:44.985806+0000","last_undegraded":"2024-03-11T19:08:44.985806+0000","last_fullsized":"2024-03-11T19:08:44.985806+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:19:53.824590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'5","reported_seq":103,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:11.775352+0000","last_change":"2024-03-11T19:07:48.748799+0000","last_active":"2024-03-11T19:09:11.775352+0000","last_peered":"2024-03-11T19:09:11.775352+0000","last_clean":"2024-03-11T19:09:11.775352+0000","last_became_active":"2024-03-11T19:07:48.748434+0000","last_became_peered":"2024-03-11T19:07:48.748434+0000","last_unstale":"2024-03-11T19:09:11.775352+0000","last_undegraded":"2024-03-11T19:09:11.775352+0000","last_fullsized":"2024-03-11T19:09:11.775352+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:10:09.836926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":79,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.986112+0000","last_change":"2024-03-11T19:07:49.750122+0000","last_active":"2024-03-11T19:08:44.986112+0000","last_peered":"2024-03-11T19:08:44.986112+0000","last_clean":"2024-03-11T19:08:44.986112+0000","last_became_active":"2024-03-11T19:07:49.749801+0000","last_became_peered":"2024-03-11T19:07:49.749801+0000","last_unstale":"2024-03-11T19:08:44.986112+0000","last_undegraded":"2024-03-11T19:08:44.986112+0000","last_fullsized":"2024-03-11T19:08:44.986112+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:58:31.155275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036439+0000","last_change":"2024-03-11T19:07:48.741959+0000","last_active":"2024-03-11T19:08:45.036439+0000","last_peered":"2024-03-11T19:08:45.036439+0000","last_clean":"2024-03-11T19:08:45.036439+0000","last_became_active":"2024-03-11T19:07:48.741630+0000","last_became_peered":"2024-03-11T19:07:48.741630+0000","last_unstale":"2024-03-11T19:08:45.036439+0000","last_undegraded":"2024-03-11T19:08:45.036439+0000","last_fullsized":"2024-03-11T19:08:45.036439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:27:08.606055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985962+0000","last_change":"2024-03-11T19:07:49.729838+0000","last_active":"2024-03-11T19:08:44.985962+0000","last_peered":"2024-03-11T19:08:44.985962+0000","last_clean":"2024-03-11T19:08:44.985962+0000","last_became_active":"2024-03-11T19:07:49.728912+0000","last_became_peered":"2024-03-11T19:07:49.728912+0000","last_unstale":"2024-03-11T19:08:44.985962+0000","last_undegraded":"2024-03-11T19:08:44.985962+0000","last_fullsized":"2024-03-11T19:08:44.985962+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:02:06.551734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.989032+0000","last_change":"2024-03-11T19:07:48.744709+0000","last_active":"2024-03-11T19:08:44.989032+0000","last_peered":"2024-03-11T19:08:44.989032+0000","last_clean":"2024-03-11T19:08:44.989032+0000","last_became_active":"2024-03-11T19:07:48.744356+0000","last_became_peered":"2024-03-11T19:07:48.744356+0000","last_unstale":"2024-03-11T19:08:44.989032+0000","last_undegraded":"2024-03-11T19:08:44.989032+0000","last_fullsized":"2024-03-11T19:08:44.989032+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:48:47.098443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.986084+0000","last_change":"2024-03-11T19:07:49.743288+0000","last_active":"2024-03-11T19:08:44.986084+0000","last_peered":"2024-03-11T19:08:44.986084+0000","last_clean":"2024-03-11T19:08:44.986084+0000","last_became_active":"2024-03-11T19:07:49.742942+0000","last_became_peered":"2024-03-11T19:07:49.742942+0000","last_unstale":"2024-03-11T19:08:44.986084+0000","last_undegraded":"2024-03-11T19:08:44.986084+0000","last_fullsized":"2024-03-11T19:08:44.986084+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:21:53.896192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.987288+0000","last_change":"2024-03-11T19:07:49.741610+0000","last_active":"2024-03-11T19:08:44.987288+0000","last_peered":"2024-03-11T19:08:44.987288+0000","last_clean":"2024-03-11T19:08:44.987288+0000","last_became_active":"2024-03-11T19:07:49.741234+0000","last_became_peered":"2024-03-11T19:07:49.741234+0000","last_unstale":"2024-03-11T19:08:44.987288+0000","last_undegraded":"2024-03-11T19:08:44.987288+0000","last_fullsized":"2024-03-11T19:08:44.987288+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:53:03.320093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037715+0000","last_change":"2024-03-11T19:08:45.037715+0000","last_active":"2024-03-11T19:08:45.037715+0000","last_peered":"2024-03-11T19:08:45.037715+0000","last_clean":"2024-03-11T19:08:45.037715+0000","last_became_active":"2024-03-11T19:08:45.037299+0000","last_became_peered":"2024-03-11T19:08:45.037299+0000","last_unstale":"2024-03-11T19:08:45.037715+0000","last_undegraded":"2024-03-11T19:08:45.037715+0000","last_fullsized":"2024-03-11T19:08:45.037715+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:01:49.025379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985697+0000","last_change":"2024-03-11T19:07:49.745928+0000","last_active":"2024-03-11T19:08:44.985697+0000","last_peered":"2024-03-11T19:08:44.985697+0000","last_clean":"2024-03-11T19:08:44.985697+0000","last_became_active":"2024-03-11T19:07:49.745025+0000","last_became_peered":"2024-03-11T19:07:49.745025+0000","last_unstale":"2024-03-11T19:08:44.985697+0000","last_undegraded":"2024-03-11T19:08:44.985697+0000","last_fullsized":"2024-03-11T19:08:44.985697+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:40:30.141906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991479+0000","last_change":"2024-03-11T19:07:49.756274+0000","last_active":"2024-03-11T19:08:44.991479+0000","last_peered":"2024-03-11T19:08:44.991479+0000","last_clean":"2024-03-11T19:08:44.991479+0000","last_became_active":"2024-03-11T19:07:49.755954+0000","last_became_peered":"2024-03-11T19:07:49.755954+0000","last_unstale":"2024-03-11T19:08:44.991479+0000","last_undegraded":"2024-03-11T19:08:44.991479+0000","last_fullsized":"2024-03-11T19:08:44.991479+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:15:36.825240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.992154+0000","last_change":"2024-03-11T19:08:44.992154+0000","last_active":"2024-03-11T19:08:44.992154+0000","last_peered":"2024-03-11T19:08:44.992154+0000","last_clean":"2024-03-11T19:08:44.992154+0000","last_became_active":"2024-03-11T19:08:44.991772+0000","last_became_peered":"2024-03-11T19:08:44.991772+0000","last_unstale":"2024-03-11T19:08:44.992154+0000","last_undegraded":"2024-03-11T19:08:44.992154+0000","last_fullsized":"2024-03-11T19:08:44.992154+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:10:33.882095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.987749+0000","last_change":"2024-03-11T19:07:49.743963+0000","last_active":"2024-03-11T19:08:44.987749+0000","last_peered":"2024-03-11T19:08:44.987749+0000","last_clean":"2024-03-11T19:08:44.987749+0000","last_became_active":"2024-03-11T19:07:49.743568+0000","last_became_peered":"2024-03-11T19:07:49.743568+0000","last_unstale":"2024-03-11T19:08:44.987749+0000","last_undegraded":"2024-03-11T19:08:44.987749+0000","last_fullsized":"2024-03-11T19:08:44.987749+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:45:28.970286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.986367+0000","last_change":"2024-03-11T19:07:49.750677+0000","last_active":"2024-03-11T19:08:44.986367+0000","last_peered":"2024-03-11T19:08:44.986367+0000","last_clean":"2024-03-11T19:08:44.986367+0000","last_became_active":"2024-03-11T19:07:49.750373+0000","last_became_peered":"2024-03-11T19:07:49.750373+0000","last_unstale":"2024-03-11T19:08:44.986367+0000","last_undegraded":"2024-03-11T19:08:44.986367+0000","last_fullsized":"2024-03-11T19:08:44.986367+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:07:28.767502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990146+0000","last_change":"2024-03-11T19:07:49.745278+0000","last_active":"2024-03-11T19:08:44.990146+0000","last_peered":"2024-03-11T19:08:44.990146+0000","last_clean":"2024-03-11T19:08:44.990146+0000","last_became_active":"2024-03-11T19:07:49.744998+0000","last_became_peered":"2024-03-11T19:07:49.744998+0000","last_unstale":"2024-03-11T19:08:44.990146+0000","last_undegraded":"2024-03-11T19:08:44.990146+0000","last_fullsized":"2024-03-11T19:08:44.990146+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:23.778051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036526+0000","last_change":"2024-03-11T19:07:49.734318+0000","last_active":"2024-03-11T19:08:45.036526+0000","last_peered":"2024-03-11T19:08:45.036526+0000","last_clean":"2024-03-11T19:08:45.036526+0000","last_became_active":"2024-03-11T19:07:49.733998+0000","last_became_peered":"2024-03-11T19:07:49.733998+0000","last_unstale":"2024-03-11T19:08:45.036526+0000","last_undegraded":"2024-03-11T19:08:45.036526+0000","last_fullsized":"2024-03-11T19:08:45.036526+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:48:50.308499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.989569+0000","last_change":"2024-03-11T19:07:49.741009+0000","last_active":"2024-03-11T19:08:44.989569+0000","last_peered":"2024-03-11T19:08:44.989569+0000","last_clean":"2024-03-11T19:08:44.989569+0000","last_became_active":"2024-03-11T19:07:49.740094+0000","last_became_peered":"2024-03-11T19:07:49.740094+0000","last_unstale":"2024-03-11T19:08:44.989569+0000","last_undegraded":"2024-03-11T19:08:44.989569+0000","last_fullsized":"2024-03-11T19:08:44.989569+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:06:31.945387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.035949+0000","last_change":"2024-03-11T19:07:49.759300+0000","last_active":"2024-03-11T19:08:45.035949+0000","last_peered":"2024-03-11T19:08:45.035949+0000","last_clean":"2024-03-11T19:08:45.035949+0000","last_became_active":"2024-03-11T19:07:49.758011+0000","last_became_peered":"2024-03-11T19:07:49.758011+0000","last_unstale":"2024-03-11T19:08:45.035949+0000","last_undegraded":"2024-03-11T19:08:45.035949+0000","last_fullsized":"2024-03-11T19:08:45.035949+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:03:40.651816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.988731+0000","last_change":"2024-03-11T19:07:49.752905+0000","last_active":"2024-03-11T19:08:44.988731+0000","last_peered":"2024-03-11T19:08:44.988731+0000","last_clean":"2024-03-11T19:08:44.988731+0000","last_became_active":"2024-03-11T19:07:49.752383+0000","last_became_peered":"2024-03-11T19:07:49.752383+0000","last_unstale":"2024-03-11T19:08:44.988731+0000","last_undegraded":"2024-03-11T19:08:44.988731+0000","last_fullsized":"2024-03-11T19:08:44.988731+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:45:33.475526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.987445+0000","last_change":"2024-03-11T19:07:49.744957+0000","last_active":"2024-03-11T19:08:44.987445+0000","last_peered":"2024-03-11T19:08:44.987445+0000","last_clean":"2024-03-11T19:08:44.987445+0000","last_became_active":"2024-03-11T19:07:49.744627+0000","last_became_peered":"2024-03-11T19:07:49.744627+0000","last_unstale":"2024-03-11T19:08:44.987445+0000","last_undegraded":"2024-03-11T19:08:44.987445+0000","last_fullsized":"2024-03-11T19:08:44.987445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:27:47.817004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245707,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67859,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":784,"num_read_kb":468,"num_write":237,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":203577,"data_compressed":5565,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":206,"ondisk_log_size":206,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318302,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":464282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579939,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":1116245,"data_compressed":37658,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743465,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":464100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005102,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":464366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266739,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1044480,"data_stored":1056996,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430264,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":978944,"data_stored":463817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":525776,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738454,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":999424,"data_stored":465156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1056768,"data_stored":526387,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":995328,"data_stored":463642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888130,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":978944,"data_stored":463678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691901,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1056768,"data_stored":1056674,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62798,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":63409,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":62782,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T19:09:15.399 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T19:09:15.400 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-11T19:09:15.400 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-11T19:09:15.400 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-11T19:09:15.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:15 smithi190 ceph-mon[36292]: pgmap v334: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:09:15.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:15 smithi190 ceph-mon[36292]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:09:15.651 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:09:16.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:16 smithi029 ceph-mon[32439]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-11T19:09:16.514 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:16 smithi190 ceph-mon[36292]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-11T19:09:16.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:16 smithi170 ceph-mon[35690]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-11T19:09:17.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:17 smithi029 ceph-mon[32439]: pgmap v335: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-11T19:09:17.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.403+0000 7f049bcd9700 1 -- 172.21.15.29:0/3305283179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0494102680 msgr2=0x7f049410eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:17.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.403+0000 7f049bcd9700 1 --2- 172.21.15.29:0/3305283179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0494102680 0x7f049410eac0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f048c0099c0 tx=0x7f048c009cd0 comp rx=0 tx=0).stop 2024-03-11T19:09:17.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.403+0000 7f049bcd9700 1 -- 172.21.15.29:0/3305283179 shutdown_connections 2024-03-11T19:09:17.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.403+0000 7f049bcd9700 1 --2- 172.21.15.29:0/3305283179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0494102680 0x7f049410eac0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:17.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.403+0000 7f049bcd9700 1 --2- 172.21.15.29:0/3305283179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0494107ca0 0x7f0494108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:17.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.403+0000 7f049bcd9700 1 --2- 172.21.15.29:0/3305283179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0494101cc0 0x7f0494102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:17.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.403+0000 7f049bcd9700 1 -- 172.21.15.29:0/3305283179 >> 172.21.15.29:0/3305283179 conn(0x7f0494078da0 msgr2=0x7f04940791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:17.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.403+0000 7f049bcd9700 1 -- 172.21.15.29:0/3305283179 shutdown_connections 2024-03-11T19:09:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.403+0000 7f049bcd9700 1 -- 172.21.15.29:0/3305283179 wait complete. 2024-03-11T19:09:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.404+0000 7f049bcd9700 1 Processor -- start 2024-03-11T19:09:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.404+0000 7f049bcd9700 1 -- start start 2024-03-11T19:09:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.404+0000 7f049bcd9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0494101cc0 0x7f0494198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.404+0000 7f049bcd9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0494102680 0x7f0494198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.404+0000 7f049bcd9700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0494107ca0 0x7f049419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.404+0000 7f049bcd9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049419cf30 con 0x7f0494102680 2024-03-11T19:09:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.404+0000 7f049bcd9700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049419d0a0 con 0x7f0494107ca0 2024-03-11T19:09:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.404+0000 7f049bcd9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049419d260 con 0x7f0494101cc0 2024-03-11T19:09:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.404+0000 7f0499274700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0494102680 0x7f0494198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.404+0000 7f0499274700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0494102680 0x7f0494198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55320/0 (socket says 172.21.15.29:55320) 2024-03-11T19:09:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.404+0000 7f0499274700 1 -- 172.21.15.29:0/793014413 learned_addr learned my addr 172.21.15.29:0/793014413 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.404+0000 7f049a276700 1 --2- 172.21.15.29:0/793014413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0494107ca0 0x7f049419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:17.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.404+0000 7f0499a75700 1 --2- 172.21.15.29:0/793014413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0494101cc0 0x7f0494198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:17.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.404+0000 7f0499274700 1 -- 172.21.15.29:0/793014413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0494101cc0 msgr2=0x7f0494198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:17.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.404+0000 7f0499274700 1 --2- 172.21.15.29:0/793014413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0494101cc0 0x7f0494198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:17.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.404+0000 7f0499274700 1 -- 172.21.15.29:0/793014413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0494107ca0 msgr2=0x7f049419c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:17.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.404+0000 7f0499274700 1 --2- 172.21.15.29:0/793014413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0494107ca0 0x7f049419c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:17.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.404+0000 7f0499274700 1 -- 172.21.15.29:0/793014413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048c0095f0 con 0x7f0494102680 2024-03-11T19:09:17.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.404+0000 7f0499a75700 1 --2- 172.21.15.29:0/793014413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0494101cc0 0x7f0494198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:17.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.405+0000 7f0499274700 1 --2- 172.21.15.29:0/793014413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0494102680 0x7f0494198590 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f048800d950 tx=0x7f048800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:17.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.405+0000 7f0492ffd700 1 -- 172.21.15.29:0/793014413 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0488009940 con 0x7f0494102680 2024-03-11T19:09:17.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.405+0000 7f049bcd9700 1 -- 172.21.15.29:0/793014413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f049419d540 con 0x7f0494102680 2024-03-11T19:09:17.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.405+0000 7f0492ffd700 1 -- 172.21.15.29:0/793014413 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0488010460 con 0x7f0494102680 2024-03-11T19:09:17.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.405+0000 7f0492ffd700 1 -- 172.21.15.29:0/793014413 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048800f620 con 0x7f0494102680 2024-03-11T19:09:17.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.405+0000 7f049bcd9700 1 -- 172.21.15.29:0/793014413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04941a4da0 con 0x7f0494102680 2024-03-11T19:09:17.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.406+0000 7f049bcd9700 1 -- 172.21.15.29:0/793014413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f049410a180 con 0x7f0494102680 2024-03-11T19:09:17.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.407+0000 7f0492ffd700 1 -- 172.21.15.29:0/793014413 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f04880105d0 con 0x7f0494102680 2024-03-11T19:09:17.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.407+0000 7f0492ffd700 1 --2- 172.21.15.29:0/793014413 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f047c0a1ac0 0x7f047c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:17.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.407+0000 7f0499a75700 1 --2- 172.21.15.29:0/793014413 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f047c0a1ac0 0x7f047c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:17.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.407+0000 7f0492ffd700 1 -- 172.21.15.29:0/793014413 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f04880cd1f0 con 0x7f0494102680 2024-03-11T19:09:17.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.408+0000 7f0499a75700 1 --2- 172.21.15.29:0/793014413 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f047c0a1ac0 0x7f047c0a3f80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0484009d50 tx=0x7f0484009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:17.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.412+0000 7f0492ffd700 1 -- 172.21.15.29:0/793014413 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f04880964a0 con 0x7f0494102680 2024-03-11T19:09:17.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:17 smithi190 ceph-mon[36292]: pgmap v335: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-11T19:09:17.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:17 smithi170 ceph-mon[35690]: pgmap v335: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-11T19:09:17.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.660+0000 7f049bcd9700 1 -- 172.21.15.29:0/793014413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f04941993b0 con 0x7f0494102680 2024-03-11T19:09:17.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.660+0000 7f0492ffd700 1 -- 172.21.15.29:0/793014413 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f0488099b60 con 0x7f0494102680 2024-03-11T19:09:17.664 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:17.664 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T19:09:17.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.662+0000 7f049bcd9700 1 -- 172.21.15.29:0/793014413 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f047c0a1ac0 msgr2=0x7f047c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:17.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.662+0000 7f049bcd9700 1 --2- 172.21.15.29:0/793014413 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f047c0a1ac0 0x7f047c0a3f80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0484009d50 tx=0x7f0484009040 comp rx=0 tx=0).stop 2024-03-11T19:09:17.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.662+0000 7f049bcd9700 1 -- 172.21.15.29:0/793014413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0494102680 msgr2=0x7f0494198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:17.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.662+0000 7f049bcd9700 1 --2- 172.21.15.29:0/793014413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0494102680 0x7f0494198590 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f048800d950 tx=0x7f048800dc60 comp rx=0 tx=0).stop 2024-03-11T19:09:17.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.662+0000 7f049bcd9700 1 -- 172.21.15.29:0/793014413 shutdown_connections 2024-03-11T19:09:17.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.662+0000 7f049bcd9700 1 --2- 172.21.15.29:0/793014413 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f047c0a1ac0 0x7f047c0a3f80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:17.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.662+0000 7f049bcd9700 1 --2- 172.21.15.29:0/793014413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0494107ca0 0x7f049419c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:17.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.662+0000 7f049bcd9700 1 --2- 172.21.15.29:0/793014413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0494101cc0 0x7f0494198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:17.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.662+0000 7f049bcd9700 1 --2- 172.21.15.29:0/793014413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0494102680 0x7f0494198590 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:17.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.662+0000 7f049bcd9700 1 -- 172.21.15.29:0/793014413 >> 172.21.15.29:0/793014413 conn(0x7f0494078da0 msgr2=0x7f0494100a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:17.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.662+0000 7f049bcd9700 1 -- 172.21.15.29:0/793014413 shutdown_connections 2024-03-11T19:09:17.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:17.662+0000 7f049bcd9700 1 -- 172.21.15.29:0/793014413 wait complete. 2024-03-11T19:09:18.389 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:18 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/793014413' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T19:09:18.422 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-11T19:09:18.422 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:09:18.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:18 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/793014413' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T19:09:18.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.532+0000 7f8a61763640 1 -- 172.21.15.29:0/2938016462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a5c1045b0 msgr2=0x7f8a5c1069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:18.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.532+0000 7f8a61763640 1 --2- 172.21.15.29:0/2938016462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a5c1045b0 0x7f8a5c1069a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8a44009900 tx=0x7f8a4402bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:18.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.532+0000 7f8a61763640 1 -- 172.21.15.29:0/2938016462 shutdown_connections 2024-03-11T19:09:18.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.532+0000 7f8a61763640 1 --2- 172.21.15.29:0/2938016462 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a5c106ee0 0x7f8a5c1092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:18.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.532+0000 7f8a61763640 1 --2- 172.21.15.29:0/2938016462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a5c1045b0 0x7f8a5c1069a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:18.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.532+0000 7f8a61763640 1 --2- 172.21.15.29:0/2938016462 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a5c101c80 0x7f8a5c104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:18.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.532+0000 7f8a61763640 1 -- 172.21.15.29:0/2938016462 >> 172.21.15.29:0/2938016462 conn(0x7f8a5c0fb610 msgr2=0x7f8a5c0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:18.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.532+0000 7f8a61763640 1 -- 172.21.15.29:0/2938016462 shutdown_connections 2024-03-11T19:09:18.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.532+0000 7f8a61763640 1 -- 172.21.15.29:0/2938016462 wait complete. 2024-03-11T19:09:18.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.533+0000 7f8a61763640 1 Processor -- start 2024-03-11T19:09:18.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.533+0000 7f8a61763640 1 -- start start 2024-03-11T19:09:18.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.533+0000 7f8a61763640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a5c101c80 0x7f8a5c1971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:18.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.533+0000 7f8a61763640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a5c1045b0 0x7f8a5c1976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:18.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.533+0000 7f8a61763640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a5c106ee0 0x7f8a5c19e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:18.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.533+0000 7f8a61763640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c19eca0 con 0x7f8a5c1045b0 2024-03-11T19:09:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.533+0000 7f8a61763640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c19ee10 con 0x7f8a5c101c80 2024-03-11T19:09:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.533+0000 7f8a61763640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c19efd0 con 0x7f8a5c106ee0 2024-03-11T19:09:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.533+0000 7f8a5a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a5c1045b0 0x7f8a5c1976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.533+0000 7f8a5a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a5c1045b0 0x7f8a5c1976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55326/0 (socket says 172.21.15.29:55326) 2024-03-11T19:09:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.533+0000 7f8a5a7fc640 1 -- 172.21.15.29:0/776101252 learned_addr learned my addr 172.21.15.29:0/776101252 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.534+0000 7f8a5b7fe640 1 --2- 172.21.15.29:0/776101252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a5c106ee0 0x7f8a5c19e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.534+0000 7f8a5affd640 1 --2- 172.21.15.29:0/776101252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a5c101c80 0x7f8a5c1971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.534+0000 7f8a5a7fc640 1 -- 172.21.15.29:0/776101252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a5c106ee0 msgr2=0x7f8a5c19e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.534+0000 7f8a5a7fc640 1 --2- 172.21.15.29:0/776101252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a5c106ee0 0x7f8a5c19e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.534+0000 7f8a5a7fc640 1 -- 172.21.15.29:0/776101252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a5c101c80 msgr2=0x7f8a5c1971b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.534+0000 7f8a5a7fc640 1 --2- 172.21.15.29:0/776101252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a5c101c80 0x7f8a5c1971b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.534+0000 7f8a5a7fc640 1 -- 172.21.15.29:0/776101252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a44009560 con 0x7f8a5c1045b0 2024-03-11T19:09:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.534+0000 7f8a5b7fe640 1 --2- 172.21.15.29:0/776101252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a5c106ee0 0x7f8a5c19e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.534+0000 7f8a5a7fc640 1 --2- 172.21.15.29:0/776101252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a5c1045b0 0x7f8a5c1976f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8a44000c00 tx=0x7f8a440042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:18.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.534+0000 7f8a53fff640 1 -- 172.21.15.29:0/776101252 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a440044e0 con 0x7f8a5c1045b0 2024-03-11T19:09:18.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.534+0000 7f8a61763640 1 -- 172.21.15.29:0/776101252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a5c19f250 con 0x7f8a5c1045b0 2024-03-11T19:09:18.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.534+0000 7f8a53fff640 1 -- 172.21.15.29:0/776101252 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8a4402de90 con 0x7f8a5c1045b0 2024-03-11T19:09:18.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.534+0000 7f8a53fff640 1 -- 172.21.15.29:0/776101252 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a44005410 con 0x7f8a5c1045b0 2024-03-11T19:09:18.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.534+0000 7f8a61763640 1 -- 172.21.15.29:0/776101252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a5c19f7b0 con 0x7f8a5c1045b0 2024-03-11T19:09:18.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.535+0000 7f8a61763640 1 -- 172.21.15.29:0/776101252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a5c0fd100 con 0x7f8a5c1045b0 2024-03-11T19:09:18.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.536+0000 7f8a53fff640 1 -- 172.21.15.29:0/776101252 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8a44005570 con 0x7f8a5c1045b0 2024-03-11T19:09:18.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.537+0000 7f8a53fff640 1 --2- 172.21.15.29:0/776101252 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8a480a1a30 0x7f8a480a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:18.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.537+0000 7f8a5affd640 1 --2- 172.21.15.29:0/776101252 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8a480a1a30 0x7f8a480a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:18.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.537+0000 7f8a53fff640 1 -- 172.21.15.29:0/776101252 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f8a440ef660 con 0x7f8a5c1045b0 2024-03-11T19:09:18.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.537+0000 7f8a5affd640 1 --2- 172.21.15.29:0/776101252 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8a480a1a30 0x7f8a480a3ef0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8a54005fd0 tx=0x7f8a54005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:18.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.541+0000 7f8a53fff640 1 -- 172.21.15.29:0/776101252 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8a440b8910 con 0x7f8a5c1045b0 2024-03-11T19:09:18.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:18 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/793014413' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T19:09:18.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.726+0000 7f8a61763640 1 -- 172.21.15.29:0/776101252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a5c10cc60 con 0x7f8a5c1045b0 2024-03-11T19:09:18.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.728+0000 7f8a53fff640 1 -- 172.21.15.29:0/776101252 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22559 (secure 0 0 0) 0x7f8a440bbfd0 con 0x7f8a5c1045b0 2024-03-11T19:09:18.732 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:18.732 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":89,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:08:44.981806+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:07:47.563184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:07:48.282461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:09:18.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.730+0000 7f8a61763640 1 -- 172.21.15.29:0/776101252 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8a480a1a30 msgr2=0x7f8a480a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:18.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.730+0000 7f8a61763640 1 --2- 172.21.15.29:0/776101252 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8a480a1a30 0x7f8a480a3ef0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8a54005fd0 tx=0x7f8a54005850 comp rx=0 tx=0).stop 2024-03-11T19:09:18.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.730+0000 7f8a61763640 1 -- 172.21.15.29:0/776101252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a5c1045b0 msgr2=0x7f8a5c1976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:18.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.730+0000 7f8a61763640 1 --2- 172.21.15.29:0/776101252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a5c1045b0 0x7f8a5c1976f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8a44000c00 tx=0x7f8a440042f0 comp rx=0 tx=0).stop 2024-03-11T19:09:18.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.730+0000 7f8a61763640 1 -- 172.21.15.29:0/776101252 shutdown_connections 2024-03-11T19:09:18.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.730+0000 7f8a61763640 1 --2- 172.21.15.29:0/776101252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a5c106ee0 0x7f8a5c19e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:18.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.730+0000 7f8a61763640 1 --2- 172.21.15.29:0/776101252 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8a480a1a30 0x7f8a480a3ef0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:18.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.730+0000 7f8a61763640 1 --2- 172.21.15.29:0/776101252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a5c1045b0 0x7f8a5c1976f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:18.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.730+0000 7f8a61763640 1 --2- 172.21.15.29:0/776101252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a5c101c80 0x7f8a5c1971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:18.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.730+0000 7f8a61763640 1 -- 172.21.15.29:0/776101252 >> 172.21.15.29:0/776101252 conn(0x7f8a5c0fb610 msgr2=0x7f8a5c103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:18.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.730+0000 7f8a61763640 1 -- 172.21.15.29:0/776101252 shutdown_connections 2024-03-11T19:09:18.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.730+0000 7f8a61763640 1 -- 172.21.15.29:0/776101252 wait complete. 2024-03-11T19:09:18.746 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:03:09.860019+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T19:07:47.563184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T19:07:48.282461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T19:09:18.746 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:09:18.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.852+0000 7fcebd464640 1 -- 172.21.15.29:0/3722215415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceb81012e0 msgr2=0x7fceb8101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:18.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.852+0000 7fcebd464640 1 --2- 172.21.15.29:0/3722215415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceb81012e0 0x7fceb8101760 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fcea0009900 tx=0x7fcea002bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:18.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.852+0000 7fcebd464640 1 -- 172.21.15.29:0/3722215415 shutdown_connections 2024-03-11T19:09:18.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.852+0000 7fcebd464640 1 --2- 172.21.15.29:0/3722215415 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fceb8101ca0 0x7fceb8108530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:18.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.852+0000 7fcebd464640 1 --2- 172.21.15.29:0/3722215415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceb81012e0 0x7fceb8101760 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:18.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.852+0000 7fcebd464640 1 --2- 172.21.15.29:0/3722215415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fceb81005d0 0x7fceb81009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:18.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.852+0000 7fcebd464640 1 -- 172.21.15.29:0/3722215415 >> 172.21.15.29:0/3722215415 conn(0x7fceb80fad80 msgr2=0x7fceb80fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:18.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.852+0000 7fcebd464640 1 -- 172.21.15.29:0/3722215415 shutdown_connections 2024-03-11T19:09:18.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.852+0000 7fcebd464640 1 -- 172.21.15.29:0/3722215415 wait complete. 2024-03-11T19:09:18.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.853+0000 7fcebd464640 1 Processor -- start 2024-03-11T19:09:18.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.853+0000 7fcebd464640 1 -- start start 2024-03-11T19:09:18.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.853+0000 7fcebd464640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fceb81005d0 0x7fceb80ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:18.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.853+0000 7fcebd464640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceb81012e0 0x7fceb819bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:18.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.853+0000 7fcebd464640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fceb8101ca0 0x7fceb819e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:18.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.853+0000 7fcebd464640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb819ebe0 con 0x7fceb81012e0 2024-03-11T19:09:18.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.853+0000 7fcebd464640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb819ed50 con 0x7fceb8101ca0 2024-03-11T19:09:18.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.853+0000 7fcebd464640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb819ef10 con 0x7fceb81005d0 2024-03-11T19:09:18.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.853+0000 7fceb67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceb81012e0 0x7fceb819bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:18.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.853+0000 7fceb67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceb81012e0 0x7fceb819bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55350/0 (socket says 172.21.15.29:55350) 2024-03-11T19:09:18.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.853+0000 7fceb67fc640 1 -- 172.21.15.29:0/805024206 learned_addr learned my addr 172.21.15.29:0/805024206 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:18.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.853+0000 7fceb6ffd640 1 --2- 172.21.15.29:0/805024206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fceb81005d0 0x7fceb80ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:18.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.853+0000 7fceb77fe640 1 --2- 172.21.15.29:0/805024206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fceb8101ca0 0x7fceb819e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:18.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.853+0000 7fceb67fc640 1 -- 172.21.15.29:0/805024206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fceb81005d0 msgr2=0x7fceb80ffcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:18.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.853+0000 7fceb67fc640 1 --2- 172.21.15.29:0/805024206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fceb81005d0 0x7fceb80ffcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:18.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.853+0000 7fceb67fc640 1 -- 172.21.15.29:0/805024206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fceb8101ca0 msgr2=0x7fceb819e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:18.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.853+0000 7fceb67fc640 1 --2- 172.21.15.29:0/805024206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fceb8101ca0 0x7fceb819e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:18.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.853+0000 7fceb67fc640 1 -- 172.21.15.29:0/805024206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea0009560 con 0x7fceb81012e0 2024-03-11T19:09:18.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.854+0000 7fceb67fc640 1 --2- 172.21.15.29:0/805024206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceb81012e0 0x7fceb819bde0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fcea0036040 tx=0x7fcea0004140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:18.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.854+0000 7fceaffff640 1 -- 172.21.15.29:0/805024206 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea00035f0 con 0x7fceb81012e0 2024-03-11T19:09:18.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.854+0000 7fceaffff640 1 -- 172.21.15.29:0/805024206 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcea002de90 con 0x7fceb81012e0 2024-03-11T19:09:18.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.854+0000 7fcebd464640 1 -- 172.21.15.29:0/805024206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb819f190 con 0x7fceb81012e0 2024-03-11T19:09:18.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.854+0000 7fceaffff640 1 -- 172.21.15.29:0/805024206 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea0005490 con 0x7fceb81012e0 2024-03-11T19:09:18.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.854+0000 7fcebd464640 1 -- 172.21.15.29:0/805024206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb819f6f0 con 0x7fceb81012e0 2024-03-11T19:09:18.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.855+0000 7fcebd464640 1 -- 172.21.15.29:0/805024206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb8107410 con 0x7fceb81012e0 2024-03-11T19:09:18.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.856+0000 7fceaffff640 1 -- 172.21.15.29:0/805024206 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fcea0003750 con 0x7fceb81012e0 2024-03-11T19:09:18.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.857+0000 7fceaffff640 1 --2- 172.21.15.29:0/805024206 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcea40a1ab0 0x7fcea40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:18.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.857+0000 7fceb6ffd640 1 --2- 172.21.15.29:0/805024206 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcea40a1ab0 0x7fcea40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:18.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.857+0000 7fceaffff640 1 -- 172.21.15.29:0/805024206 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fcea00efe10 con 0x7fceb81012e0 2024-03-11T19:09:18.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.857+0000 7fceb6ffd640 1 --2- 172.21.15.29:0/805024206 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcea40a1ab0 0x7fcea40a3f70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fceb00058c0 tx=0x7fceb0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:18.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:18.861+0000 7fceaffff640 1 -- 172.21.15.29:0/805024206 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcea00b9020 con 0x7fceb81012e0 2024-03-11T19:09:19.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.046+0000 7fcebd464640 1 -- 172.21.15.29:0/805024206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fceb810bbb0 con 0x7fceb81012e0 2024-03-11T19:09:19.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.047+0000 7fceaffff640 1 -- 172.21.15.29:0/805024206 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcea00430a0 con 0x7fceb81012e0 2024-03-11T19:09:19.052 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-11T19:09:19.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.049+0000 7fcebd464640 1 -- 172.21.15.29:0/805024206 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcea40a1ab0 msgr2=0x7fcea40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:19.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.049+0000 7fcebd464640 1 --2- 172.21.15.29:0/805024206 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcea40a1ab0 0x7fcea40a3f70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fceb00058c0 tx=0x7fceb0005850 comp rx=0 tx=0).stop 2024-03-11T19:09:19.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.049+0000 7fcebd464640 1 -- 172.21.15.29:0/805024206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceb81012e0 msgr2=0x7fceb819bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:19.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.049+0000 7fcebd464640 1 --2- 172.21.15.29:0/805024206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceb81012e0 0x7fceb819bde0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fcea0036040 tx=0x7fcea0004140 comp rx=0 tx=0).stop 2024-03-11T19:09:19.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.049+0000 7fcebd464640 1 -- 172.21.15.29:0/805024206 shutdown_connections 2024-03-11T19:09:19.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.049+0000 7fcebd464640 1 --2- 172.21.15.29:0/805024206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fceb8101ca0 0x7fceb819e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.049+0000 7fcebd464640 1 --2- 172.21.15.29:0/805024206 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcea40a1ab0 0x7fcea40a3f70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.049+0000 7fcebd464640 1 --2- 172.21.15.29:0/805024206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceb81012e0 0x7fceb819bde0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.049+0000 7fcebd464640 1 --2- 172.21.15.29:0/805024206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fceb81005d0 0x7fceb80ffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.049+0000 7fcebd464640 1 -- 172.21.15.29:0/805024206 >> 172.21.15.29:0/805024206 conn(0x7fceb80fad80 msgr2=0x7fceb81045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:19.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.049+0000 7fcebd464640 1 -- 172.21.15.29:0/805024206 shutdown_connections 2024-03-11T19:09:19.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.049+0000 7fcebd464640 1 -- 172.21.15.29:0/805024206 wait complete. 2024-03-11T19:09:19.065 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T19:09:19.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.168+0000 7f1955823640 1 -- 172.21.15.29:0/2265153989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1950100080 msgr2=0x7f19501004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:19.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.168+0000 7f1955823640 1 --2- 172.21.15.29:0/2265153989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1950100080 0x7f19501004a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1940009900 tx=0x7f194002bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:19.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.168+0000 7f1955823640 1 -- 172.21.15.29:0/2265153989 shutdown_connections 2024-03-11T19:09:19.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.168+0000 7f1955823640 1 --2- 172.21.15.29:0/2265153989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1950101c30 0x7f19501084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.168+0000 7f1955823640 1 --2- 172.21.15.29:0/2265153989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1950101270 0x7f19501016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.168+0000 7f1955823640 1 --2- 172.21.15.29:0/2265153989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1950100080 0x7f19501004a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.168+0000 7f1955823640 1 -- 172.21.15.29:0/2265153989 >> 172.21.15.29:0/2265153989 conn(0x7f19500fb610 msgr2=0x7f19500fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:19.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.168+0000 7f1955823640 1 -- 172.21.15.29:0/2265153989 shutdown_connections 2024-03-11T19:09:19.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.168+0000 7f1955823640 1 -- 172.21.15.29:0/2265153989 wait complete. 2024-03-11T19:09:19.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f1955823640 1 Processor -- start 2024-03-11T19:09:19.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f1955823640 1 -- start start 2024-03-11T19:09:19.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f1955823640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1950100080 0x7f1950104010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f1955823640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1950101270 0x7f1950102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f1955823640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1950101c30 0x7f1950102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f1955823640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19501032f0 con 0x7f1950101c30 2024-03-11T19:09:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f1955823640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1950103460 con 0x7f1950101270 2024-03-11T19:09:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f1955823640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19501a11f0 con 0x7f1950100080 2024-03-11T19:09:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f194f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1950101c30 0x7f1950102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f194f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1950101c30 0x7f1950102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55362/0 (socket says 172.21.15.29:55362) 2024-03-11T19:09:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f194f7fe640 1 -- 172.21.15.29:0/886682663 learned_addr learned my addr 172.21.15.29:0/886682663 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f194effd640 1 --2- 172.21.15.29:0/886682663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1950100080 0x7f1950104010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f1947fff640 1 --2- 172.21.15.29:0/886682663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1950101270 0x7f1950102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f194f7fe640 1 -- 172.21.15.29:0/886682663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1950100080 msgr2=0x7f1950104010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f194f7fe640 1 --2- 172.21.15.29:0/886682663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1950100080 0x7f1950104010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f194f7fe640 1 -- 172.21.15.29:0/886682663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1950101270 msgr2=0x7f1950102650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f194f7fe640 1 --2- 172.21.15.29:0/886682663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1950101270 0x7f1950102650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f194f7fe640 1 -- 172.21.15.29:0/886682663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1940009560 con 0x7f1950101c30 2024-03-11T19:09:19.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f194effd640 1 --2- 172.21.15.29:0/886682663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1950100080 0x7f1950104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:19.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.169+0000 7f1947fff640 1 --2- 172.21.15.29:0/886682663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1950101270 0x7f1950102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:19.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.170+0000 7f194f7fe640 1 --2- 172.21.15.29:0/886682663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1950101c30 0x7f1950102b90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f194800cd10 tx=0x7f194800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:19.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.170+0000 7f1954821640 1 -- 172.21.15.29:0/886682663 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194800f450 con 0x7f1950101c30 2024-03-11T19:09:19.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.170+0000 7f1954821640 1 -- 172.21.15.29:0/886682663 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f194800fa60 con 0x7f1950101c30 2024-03-11T19:09:19.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.170+0000 7f1955823640 1 -- 172.21.15.29:0/886682663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19501a14a0 con 0x7f1950101c30 2024-03-11T19:09:19.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.170+0000 7f1954821640 1 -- 172.21.15.29:0/886682663 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1948018ad0 con 0x7f1950101c30 2024-03-11T19:09:19.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.170+0000 7f1955823640 1 -- 172.21.15.29:0/886682663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19501a1960 con 0x7f1950101c30 2024-03-11T19:09:19.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.171+0000 7f1955823640 1 -- 172.21.15.29:0/886682663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f195010b900 con 0x7f1950101c30 2024-03-11T19:09:19.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.172+0000 7f1954821640 1 -- 172.21.15.29:0/886682663 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f194800fbd0 con 0x7f1950101c30 2024-03-11T19:09:19.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.173+0000 7f1954821640 1 --2- 172.21.15.29:0/886682663 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f193c0a1a30 0x7f193c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:19.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.173+0000 7f194effd640 1 --2- 172.21.15.29:0/886682663 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f193c0a1a30 0x7f193c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:19.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.173+0000 7f1954821640 1 -- 172.21.15.29:0/886682663 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f19480cda90 con 0x7f1950101c30 2024-03-11T19:09:19.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.173+0000 7f194effd640 1 --2- 172.21.15.29:0/886682663 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f193c0a1a30 0x7f193c0a3ef0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1940006010 tx=0x7f1940005c00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:19.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.176+0000 7f1954821640 1 -- 172.21.15.29:0/886682663 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1948096ca0 con 0x7f1950101c30 2024-03-11T19:09:19.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.357+0000 7f1955823640 1 -- 172.21.15.29:0/886682663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f195010ba90 con 0x7f1950101c30 2024-03-11T19:09:19.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.358+0000 7f1954821640 1 -- 172.21.15.29:0/886682663 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f19480ba0a0 con 0x7f1950101c30 2024-03-11T19:09:19.361 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-03-11T19:09:19.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.359+0000 7f1955823640 1 -- 172.21.15.29:0/886682663 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f193c0a1a30 msgr2=0x7f193c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:19.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.359+0000 7f1955823640 1 --2- 172.21.15.29:0/886682663 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f193c0a1a30 0x7f193c0a3ef0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1940006010 tx=0x7f1940005c00 comp rx=0 tx=0).stop 2024-03-11T19:09:19.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.360+0000 7f1955823640 1 -- 172.21.15.29:0/886682663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1950101c30 msgr2=0x7f1950102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:19.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.360+0000 7f1955823640 1 --2- 172.21.15.29:0/886682663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1950101c30 0x7f1950102b90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f194800cd10 tx=0x7f194800a620 comp rx=0 tx=0).stop 2024-03-11T19:09:19.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.360+0000 7f1955823640 1 -- 172.21.15.29:0/886682663 shutdown_connections 2024-03-11T19:09:19.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.360+0000 7f1955823640 1 --2- 172.21.15.29:0/886682663 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f193c0a1a30 0x7f193c0a3ef0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.360+0000 7f1955823640 1 --2- 172.21.15.29:0/886682663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1950101c30 0x7f1950102b90 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.360+0000 7f1955823640 1 --2- 172.21.15.29:0/886682663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1950101270 0x7f1950102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.360+0000 7f1955823640 1 --2- 172.21.15.29:0/886682663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1950100080 0x7f1950104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.360+0000 7f1955823640 1 -- 172.21.15.29:0/886682663 >> 172.21.15.29:0/886682663 conn(0x7f19500fb610 msgr2=0x7f1950068350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:19.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.360+0000 7f1955823640 1 -- 172.21.15.29:0/886682663 shutdown_connections 2024-03-11T19:09:19.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.360+0000 7f1955823640 1 -- 172.21.15.29:0/886682663 wait complete. 2024-03-11T19:09:19.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:19 smithi029 ceph-mon[32439]: pgmap v336: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-11T19:09:19.373 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:19 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/776101252' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:19.373 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:19 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/805024206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:09:19.374 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T19:09:19.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.476+0000 7f1ac8f63640 1 -- 172.21.15.29:0/3173983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac4100080 msgr2=0x7f1ac41004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:19.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.476+0000 7f1ac8f63640 1 --2- 172.21.15.29:0/3173983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac4100080 0x7f1ac41004a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009900 tx=0x7f1ab402bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:19.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.476+0000 7f1ac8f63640 1 -- 172.21.15.29:0/3173983 shutdown_connections 2024-03-11T19:09:19.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.476+0000 7f1ac8f63640 1 --2- 172.21.15.29:0/3173983 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ac4101c30 0x7f1ac41084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.476+0000 7f1ac8f63640 1 --2- 172.21.15.29:0/3173983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ac4101270 0x7f1ac41016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.476+0000 7f1ac8f63640 1 --2- 172.21.15.29:0/3173983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac4100080 0x7f1ac41004a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.476+0000 7f1ac8f63640 1 -- 172.21.15.29:0/3173983 >> 172.21.15.29:0/3173983 conn(0x7f1ac40fb610 msgr2=0x7f1ac40fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:19.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.476+0000 7f1ac8f63640 1 -- 172.21.15.29:0/3173983 shutdown_connections 2024-03-11T19:09:19.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.476+0000 7f1ac8f63640 1 -- 172.21.15.29:0/3173983 wait complete. 2024-03-11T19:09:19.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.477+0000 7f1ac8f63640 1 Processor -- start 2024-03-11T19:09:19.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.477+0000 7f1ac8f63640 1 -- start start 2024-03-11T19:09:19.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.477+0000 7f1ac8f63640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac4100080 0x7f1ac4104010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:19.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.477+0000 7f1ac8f63640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ac4101270 0x7f1ac4102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.477+0000 7f1ac8f63640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ac4101c30 0x7f1ac4102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.477+0000 7f1ac8f63640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac41032f0 con 0x7f1ac4100080 2024-03-11T19:09:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.477+0000 7f1ac8f63640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac4103460 con 0x7f1ac4101270 2024-03-11T19:09:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.477+0000 7f1ac8f63640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac41a1260 con 0x7f1ac4101c30 2024-03-11T19:09:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.477+0000 7f1ac259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac4100080 0x7f1ac4104010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.477+0000 7f1ac259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac4100080 0x7f1ac4104010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55374/0 (socket says 172.21.15.29:55374) 2024-03-11T19:09:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.477+0000 7f1ac259c640 1 -- 172.21.15.29:0/4109692340 learned_addr learned my addr 172.21.15.29:0/4109692340 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.478+0000 7f1abbfff640 1 --2- 172.21.15.29:0/4109692340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ac4101270 0x7f1ac4102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.478+0000 7f1ac2d9d640 1 --2- 172.21.15.29:0/4109692340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ac4101c30 0x7f1ac4102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.478+0000 7f1ac259c640 1 -- 172.21.15.29:0/4109692340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ac4101c30 msgr2=0x7f1ac4102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.478+0000 7f1ac259c640 1 --2- 172.21.15.29:0/4109692340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ac4101c30 0x7f1ac4102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.478+0000 7f1ac259c640 1 -- 172.21.15.29:0/4109692340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ac4101270 msgr2=0x7f1ac4102650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.478+0000 7f1ac259c640 1 --2- 172.21.15.29:0/4109692340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ac4101270 0x7f1ac4102650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.478+0000 7f1ac259c640 1 -- 172.21.15.29:0/4109692340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab4009560 con 0x7f1ac4100080 2024-03-11T19:09:19.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.478+0000 7f1ac259c640 1 --2- 172.21.15.29:0/4109692340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac4100080 0x7f1ac4104010 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4000c00 tx=0x7f1ab40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:19.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.478+0000 7f1abb7fe640 1 -- 172.21.15.29:0/4109692340 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab4004aa0 con 0x7f1ac4100080 2024-03-11T19:09:19.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.478+0000 7f1abb7fe640 1 -- 172.21.15.29:0/4109692340 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1ab402de90 con 0x7f1ac4100080 2024-03-11T19:09:19.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.478+0000 7f1ac8f63640 1 -- 172.21.15.29:0/4109692340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac41a14b0 con 0x7f1ac4100080 2024-03-11T19:09:19.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.478+0000 7f1abb7fe640 1 -- 172.21.15.29:0/4109692340 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab4003c90 con 0x7f1ac4100080 2024-03-11T19:09:19.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.478+0000 7f1ac8f63640 1 -- 172.21.15.29:0/4109692340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac41a19c0 con 0x7f1ac4100080 2024-03-11T19:09:19.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.479+0000 7f1ac8f63640 1 -- 172.21.15.29:0/4109692340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac410b940 con 0x7f1ac4100080 2024-03-11T19:09:19.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.480+0000 7f1abb7fe640 1 -- 172.21.15.29:0/4109692340 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1ab4004fd0 con 0x7f1ac4100080 2024-03-11T19:09:19.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.481+0000 7f1abb7fe640 1 --2- 172.21.15.29:0/4109692340 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1ab00a1a30 0x7f1ab00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:19.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.481+0000 7f1abbfff640 1 --2- 172.21.15.29:0/4109692340 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1ab00a1a30 0x7f1ab00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:19.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.481+0000 7f1abb7fe640 1 -- 172.21.15.29:0/4109692340 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f1ab40ef5d0 con 0x7f1ac4100080 2024-03-11T19:09:19.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.481+0000 7f1abbfff640 1 --2- 172.21.15.29:0/4109692340 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1ab00a1a30 0x7f1ab00a3ef0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4103d80 tx=0x7f1aac00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:19.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.485+0000 7f1abb7fe640 1 -- 172.21.15.29:0/4109692340 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1ab40b87e0 con 0x7f1ac4100080 2024-03-11T19:09:19.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:19 smithi190 ceph-mon[36292]: pgmap v336: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-11T19:09:19.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:19 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/776101252' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:19.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:19 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/805024206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:09:19.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:19 smithi170 ceph-mon[35690]: pgmap v336: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-11T19:09:19.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:19 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/776101252' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:19.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:19 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/805024206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:09:19.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.671+0000 7f1ac8f63640 1 -- 172.21.15.29:0/4109692340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1ac41a1e30 con 0x7f1ac4100080 2024-03-11T19:09:19.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.671+0000 7f1abb7fe640 1 -- 172.21.15.29:0/4109692340 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1ab4005280 con 0x7f1ac4100080 2024-03-11T19:09:19.674 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-03-11T19:09:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.674+0000 7f1ac8f63640 1 -- 172.21.15.29:0/4109692340 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1ab00a1a30 msgr2=0x7f1ab00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.674+0000 7f1ac8f63640 1 --2- 172.21.15.29:0/4109692340 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1ab00a1a30 0x7f1ab00a3ef0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4103d80 tx=0x7f1aac00b390 comp rx=0 tx=0).stop 2024-03-11T19:09:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.674+0000 7f1ac8f63640 1 -- 172.21.15.29:0/4109692340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac4100080 msgr2=0x7f1ac4104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.674+0000 7f1ac8f63640 1 --2- 172.21.15.29:0/4109692340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac4100080 0x7f1ac4104010 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4000c00 tx=0x7f1ab40042f0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.674+0000 7f1ac8f63640 1 -- 172.21.15.29:0/4109692340 shutdown_connections 2024-03-11T19:09:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.674+0000 7f1ac8f63640 1 --2- 172.21.15.29:0/4109692340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ac4101c30 0x7f1ac4102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.674+0000 7f1ac8f63640 1 --2- 172.21.15.29:0/4109692340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ac4101270 0x7f1ac4102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.674+0000 7f1ac8f63640 1 --2- 172.21.15.29:0/4109692340 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1ab00a1a30 0x7f1ab00a3ef0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.674+0000 7f1ac8f63640 1 --2- 172.21.15.29:0/4109692340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac4100080 0x7f1ac4104010 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.674+0000 7f1ac8f63640 1 -- 172.21.15.29:0/4109692340 >> 172.21.15.29:0/4109692340 conn(0x7f1ac40fb610 msgr2=0x7f1ac4068350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:19.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.674+0000 7f1ac8f63640 1 -- 172.21.15.29:0/4109692340 shutdown_connections 2024-03-11T19:09:19.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.674+0000 7f1ac8f63640 1 -- 172.21.15.29:0/4109692340 wait complete. 2024-03-11T19:09:19.688 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:09:19.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.787+0000 7fa4b6239640 1 -- 172.21.15.29:0/440953031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4b0101290 msgr2=0x7fa4b0101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:19.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.787+0000 7fa4b6239640 1 --2- 172.21.15.29:0/440953031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4b0101290 0x7fa4b0101710 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4009900 tx=0x7fa4a402bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:19.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.788+0000 7fa4b6239640 1 -- 172.21.15.29:0/440953031 shutdown_connections 2024-03-11T19:09:19.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.788+0000 7fa4b6239640 1 --2- 172.21.15.29:0/440953031 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4b0101c50 0x7fa4b01084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.788+0000 7fa4b6239640 1 --2- 172.21.15.29:0/440953031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4b0101290 0x7fa4b0101710 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.788+0000 7fa4b6239640 1 --2- 172.21.15.29:0/440953031 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4b01000a0 0x7fa4b01004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.788+0000 7fa4b6239640 1 -- 172.21.15.29:0/440953031 >> 172.21.15.29:0/440953031 conn(0x7fa4b00fb610 msgr2=0x7fa4b00fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:19.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.788+0000 7fa4b6239640 1 -- 172.21.15.29:0/440953031 shutdown_connections 2024-03-11T19:09:19.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.788+0000 7fa4b6239640 1 -- 172.21.15.29:0/440953031 wait complete. 2024-03-11T19:09:19.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.788+0000 7fa4b6239640 1 Processor -- start 2024-03-11T19:09:19.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.788+0000 7fa4b6239640 1 -- start start 2024-03-11T19:09:19.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.788+0000 7fa4b6239640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4b01000a0 0x7fa4b0197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:19.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.788+0000 7fa4b6239640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4b0101290 0x7fa4b01976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:19.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.788+0000 7fa4b6239640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4b0101c50 0x7fa4b019e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.788+0000 7fa4b6239640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b019ec60 con 0x7fa4b0101c50 2024-03-11T19:09:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.788+0000 7fa4b6239640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b019edd0 con 0x7fa4b01000a0 2024-03-11T19:09:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.788+0000 7fa4b6239640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b019ef90 con 0x7fa4b0101290 2024-03-11T19:09:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.789+0000 7fa4affff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4b0101c50 0x7fa4b019e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.789+0000 7fa4affff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4b0101c50 0x7fa4b019e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55390/0 (socket says 172.21.15.29:55390) 2024-03-11T19:09:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.789+0000 7fa4affff640 1 -- 172.21.15.29:0/3329503386 learned_addr learned my addr 172.21.15.29:0/3329503386 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.789+0000 7fa4af7fe640 1 --2- 172.21.15.29:0/3329503386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4b01000a0 0x7fa4b0197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.789+0000 7fa4aeffd640 1 --2- 172.21.15.29:0/3329503386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4b0101290 0x7fa4b01976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.789+0000 7fa4affff640 1 -- 172.21.15.29:0/3329503386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4b0101290 msgr2=0x7fa4b01976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.789+0000 7fa4affff640 1 --2- 172.21.15.29:0/3329503386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4b0101290 0x7fa4b01976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.789+0000 7fa4affff640 1 -- 172.21.15.29:0/3329503386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4b01000a0 msgr2=0x7fa4b0197170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.789+0000 7fa4affff640 1 --2- 172.21.15.29:0/3329503386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4b01000a0 0x7fa4b0197170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:19.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.789+0000 7fa4affff640 1 -- 172.21.15.29:0/3329503386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a4009560 con 0x7fa4b0101c50 2024-03-11T19:09:19.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.789+0000 7fa4affff640 1 --2- 172.21.15.29:0/3329503386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4b0101c50 0x7fa4b019e720 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa4a800eb10 tx=0x7fa4a800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:19.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.789+0000 7fa4acff9640 1 -- 172.21.15.29:0/3329503386 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8004b20 con 0x7fa4b0101c50 2024-03-11T19:09:19.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.789+0000 7fa4acff9640 1 -- 172.21.15.29:0/3329503386 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa4a8004c80 con 0x7fa4b0101c50 2024-03-11T19:09:19.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.789+0000 7fa4b6239640 1 -- 172.21.15.29:0/3329503386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b019f270 con 0x7fa4b0101c50 2024-03-11T19:09:19.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.789+0000 7fa4acff9640 1 -- 172.21.15.29:0/3329503386 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8010690 con 0x7fa4b0101c50 2024-03-11T19:09:19.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.789+0000 7fa4b6239640 1 -- 172.21.15.29:0/3329503386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b019f750 con 0x7fa4b0101c50 2024-03-11T19:09:19.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.790+0000 7fa4b6239640 1 -- 172.21.15.29:0/3329503386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b010b960 con 0x7fa4b0101c50 2024-03-11T19:09:19.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.791+0000 7fa4acff9640 1 -- 172.21.15.29:0/3329503386 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fa4a8005020 con 0x7fa4b0101c50 2024-03-11T19:09:19.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.792+0000 7fa4acff9640 1 --2- 172.21.15.29:0/3329503386 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa4980a1a30 0x7fa4980a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:19.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.792+0000 7fa4af7fe640 1 --2- 172.21.15.29:0/3329503386 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa4980a1a30 0x7fa4980a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:19.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.792+0000 7fa4acff9640 1 -- 172.21.15.29:0/3329503386 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fa4a8014070 con 0x7fa4b0101c50 2024-03-11T19:09:19.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.792+0000 7fa4af7fe640 1 --2- 172.21.15.29:0/3329503386 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa4980a1a30 0x7fa4980a3ef0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa4b01010f0 tx=0x7fa4a0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:19.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:19.796+0000 7fa4acff9640 1 -- 172.21.15.29:0/3329503386 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa4a8096b70 con 0x7fa4b0101c50 2024-03-11T19:09:20.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.015+0000 7fa4b6239640 1 -- 172.21.15.29:0/3329503386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4b019f9d0 con 0x7fa4b0101c50 2024-03-11T19:09:20.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.015+0000 7fa4acff9640 1 -- 172.21.15.29:0/3329503386 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+11385 (secure 0 0 0) 0x7fa4b019f9d0 con 0x7fa4b0101c50 2024-03-11T19:09:20.018 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:20.019 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:07:56.662890+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:09:20.019 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 26 2024-03-11T19:09:20.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.017+0000 7fa4b6239640 1 -- 172.21.15.29:0/3329503386 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa4980a1a30 msgr2=0x7fa4980a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:20.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.017+0000 7fa4b6239640 1 --2- 172.21.15.29:0/3329503386 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa4980a1a30 0x7fa4980a3ef0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa4b01010f0 tx=0x7fa4a0005f90 comp rx=0 tx=0).stop 2024-03-11T19:09:20.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.017+0000 7fa4b6239640 1 -- 172.21.15.29:0/3329503386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4b0101c50 msgr2=0x7fa4b019e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:20.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.017+0000 7fa4b6239640 1 --2- 172.21.15.29:0/3329503386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4b0101c50 0x7fa4b019e720 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa4a800eb10 tx=0x7fa4a800c2d0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.017+0000 7fa4b6239640 1 -- 172.21.15.29:0/3329503386 shutdown_connections 2024-03-11T19:09:20.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.017+0000 7fa4b6239640 1 --2- 172.21.15.29:0/3329503386 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa4980a1a30 0x7fa4980a3ef0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.017+0000 7fa4b6239640 1 --2- 172.21.15.29:0/3329503386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4b0101c50 0x7fa4b019e720 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.017+0000 7fa4b6239640 1 --2- 172.21.15.29:0/3329503386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4b0101290 0x7fa4b01976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.017+0000 7fa4b6239640 1 --2- 172.21.15.29:0/3329503386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4b01000a0 0x7fa4b0197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.017+0000 7fa4b6239640 1 -- 172.21.15.29:0/3329503386 >> 172.21.15.29:0/3329503386 conn(0x7fa4b00fb610 msgr2=0x7fa4b00fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:20.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.017+0000 7fa4b6239640 1 -- 172.21.15.29:0/3329503386 shutdown_connections 2024-03-11T19:09:20.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.017+0000 7fa4b6239640 1 -- 172.21.15.29:0/3329503386 wait complete. 2024-03-11T19:09:20.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:09:20.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.131+0000 7f5430fa1640 1 -- 172.21.15.29:0/343638595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f542c101310 msgr2=0x7f542c101790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:20.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.131+0000 7f5430fa1640 1 --2- 172.21.15.29:0/343638595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f542c101310 0x7f542c101790 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5414009900 tx=0x7f541402bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:20.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.132+0000 7f5430fa1640 1 -- 172.21.15.29:0/343638595 shutdown_connections 2024-03-11T19:09:20.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.132+0000 7f5430fa1640 1 --2- 172.21.15.29:0/343638595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f542c101cd0 0x7f542c108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.132+0000 7f5430fa1640 1 --2- 172.21.15.29:0/343638595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f542c101310 0x7f542c101790 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.132+0000 7f5430fa1640 1 --2- 172.21.15.29:0/343638595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f542c100120 0x7f542c100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.132+0000 7f5430fa1640 1 -- 172.21.15.29:0/343638595 >> 172.21.15.29:0/343638595 conn(0x7f542c0fb6b0 msgr2=0x7f542c0fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:20.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.132+0000 7f5430fa1640 1 -- 172.21.15.29:0/343638595 shutdown_connections 2024-03-11T19:09:20.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.132+0000 7f5430fa1640 1 -- 172.21.15.29:0/343638595 wait complete. 2024-03-11T19:09:20.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.132+0000 7f5430fa1640 1 Processor -- start 2024-03-11T19:09:20.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.132+0000 7f5430fa1640 1 -- start start 2024-03-11T19:09:20.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f5430fa1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f542c100120 0x7f542c1971d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:20.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f5430fa1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f542c101310 0x7f542c197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:20.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f5430fa1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f542c101cd0 0x7f542c19e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:20.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f5430fa1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542c19ec70 con 0x7f542c100120 2024-03-11T19:09:20.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f5430fa1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542c19ede0 con 0x7f542c101cd0 2024-03-11T19:09:20.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f5430fa1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542c19efa0 con 0x7f542c101310 2024-03-11T19:09:20.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f542a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f542c100120 0x7f542c1971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:20.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f542a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f542c100120 0x7f542c1971d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55408/0 (socket says 172.21.15.29:55408) 2024-03-11T19:09:20.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f542a59c640 1 -- 172.21.15.29:0/1059086613 learned_addr learned my addr 172.21.15.29:0/1059086613 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:20.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f542ad9d640 1 --2- 172.21.15.29:0/1059086613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f542c101cd0 0x7f542c19e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:20.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f5429d9b640 1 --2- 172.21.15.29:0/1059086613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f542c101310 0x7f542c197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:20.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f542a59c640 1 -- 172.21.15.29:0/1059086613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f542c101310 msgr2=0x7f542c197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:20.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f542a59c640 1 --2- 172.21.15.29:0/1059086613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f542c101310 0x7f542c197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f542a59c640 1 -- 172.21.15.29:0/1059086613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f542c101cd0 msgr2=0x7f542c19e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:20.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f542a59c640 1 --2- 172.21.15.29:0/1059086613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f542c101cd0 0x7f542c19e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f542a59c640 1 -- 172.21.15.29:0/1059086613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5414009560 con 0x7f542c100120 2024-03-11T19:09:20.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f542ad9d640 1 --2- 172.21.15.29:0/1059086613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f542c101cd0 0x7f542c19e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:20.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f542a59c640 1 --2- 172.21.15.29:0/1059086613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f542c100120 0x7f542c1971d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f542400e9c0 tx=0x7f542400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:20.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f54237fe640 1 -- 172.21.15.29:0/1059086613 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54240096e0 con 0x7f542c100120 2024-03-11T19:09:20.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f54237fe640 1 -- 172.21.15.29:0/1059086613 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f54240044e0 con 0x7f542c100120 2024-03-11T19:09:20.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f5430fa1640 1 -- 172.21.15.29:0/1059086613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f542c19f280 con 0x7f542c100120 2024-03-11T19:09:20.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f54237fe640 1 -- 172.21.15.29:0/1059086613 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5424010430 con 0x7f542c100120 2024-03-11T19:09:20.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.133+0000 7f5430fa1640 1 -- 172.21.15.29:0/1059086613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f542c19f740 con 0x7f542c100120 2024-03-11T19:09:20.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.134+0000 7f5430fa1640 1 -- 172.21.15.29:0/1059086613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f542c10b970 con 0x7f542c100120 2024-03-11T19:09:20.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.135+0000 7f54237fe640 1 -- 172.21.15.29:0/1059086613 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5424003f60 con 0x7f542c100120 2024-03-11T19:09:20.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.136+0000 7f54237fe640 1 --2- 172.21.15.29:0/1059086613 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f54180a1a30 0x7f54180a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:20.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.136+0000 7f5429d9b640 1 --2- 172.21.15.29:0/1059086613 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f54180a1a30 0x7f54180a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:20.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.136+0000 7f54237fe640 1 -- 172.21.15.29:0/1059086613 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5424014070 con 0x7f542c100120 2024-03-11T19:09:20.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.136+0000 7f5429d9b640 1 --2- 172.21.15.29:0/1059086613 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f54180a1a30 0x7f54180a3ef0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f542c198690 tx=0x7f5414000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:20.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.139+0000 7f54237fe640 1 -- 172.21.15.29:0/1059086613 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5424096ba0 con 0x7f542c100120 2024-03-11T19:09:20.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.321+0000 7f5430fa1640 1 -- 172.21.15.29:0/1059086613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f542c10bb00 con 0x7f542c100120 2024-03-11T19:09:20.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.323+0000 7f54237fe640 1 -- 172.21.15.29:0/1059086613 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22559 (secure 0 0 0) 0x7f542409a260 con 0x7f542c100120 2024-03-11T19:09:20.326 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:20.327 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":89,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:08:44.981806+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:07:47.563184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:07:48.282461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:09:20.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.325+0000 7f5430fa1640 1 -- 172.21.15.29:0/1059086613 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f54180a1a30 msgr2=0x7f54180a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:20.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.325+0000 7f5430fa1640 1 --2- 172.21.15.29:0/1059086613 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f54180a1a30 0x7f54180a3ef0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f542c198690 tx=0x7f5414000bc0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.325+0000 7f5430fa1640 1 -- 172.21.15.29:0/1059086613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f542c100120 msgr2=0x7f542c1971d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.325+0000 7f5430fa1640 1 --2- 172.21.15.29:0/1059086613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f542c100120 0x7f542c1971d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f542400e9c0 tx=0x7f542400ee80 comp rx=0 tx=0).stop 2024-03-11T19:09:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.325+0000 7f5430fa1640 1 -- 172.21.15.29:0/1059086613 shutdown_connections 2024-03-11T19:09:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.325+0000 7f5430fa1640 1 --2- 172.21.15.29:0/1059086613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f542c101cd0 0x7f542c19e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.325+0000 7f5430fa1640 1 --2- 172.21.15.29:0/1059086613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f542c101310 0x7f542c197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.325+0000 7f5430fa1640 1 --2- 172.21.15.29:0/1059086613 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f54180a1a30 0x7f54180a3ef0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.325+0000 7f5430fa1640 1 --2- 172.21.15.29:0/1059086613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f542c100120 0x7f542c1971d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.325+0000 7f5430fa1640 1 -- 172.21.15.29:0/1059086613 >> 172.21.15.29:0/1059086613 conn(0x7f542c0fb6b0 msgr2=0x7f542c0fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:20.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.325+0000 7f5430fa1640 1 -- 172.21.15.29:0/1059086613 shutdown_connections 2024-03-11T19:09:20.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.325+0000 7f5430fa1640 1 -- 172.21.15.29:0/1059086613 wait complete. 2024-03-11T19:09:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:20 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/886682663' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:09:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:20 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/4109692340' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:09:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:20 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3329503386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:09:20.340 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:03:09.860019+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T19:07:47.563184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T19:07:48.282461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T19:09:20.340 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:09:20.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.442+0000 7f56a0440640 1 -- 172.21.15.29:0/2684033939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5698101290 msgr2=0x7f5698101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:20.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.442+0000 7f56a0440640 1 --2- 172.21.15.29:0/2684033939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5698101290 0x7f5698101710 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5688009900 tx=0x7f568802bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:20.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.442+0000 7f56a0440640 1 -- 172.21.15.29:0/2684033939 shutdown_connections 2024-03-11T19:09:20.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.442+0000 7f56a0440640 1 --2- 172.21.15.29:0/2684033939 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5698101c50 0x7f56981084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.442+0000 7f56a0440640 1 --2- 172.21.15.29:0/2684033939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5698101290 0x7f5698101710 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.442+0000 7f56a0440640 1 --2- 172.21.15.29:0/2684033939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56981000a0 0x7f56981004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.442+0000 7f56a0440640 1 -- 172.21.15.29:0/2684033939 >> 172.21.15.29:0/2684033939 conn(0x7f56980fb630 msgr2=0x7f56980fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:20.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.442+0000 7f56a0440640 1 -- 172.21.15.29:0/2684033939 shutdown_connections 2024-03-11T19:09:20.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.442+0000 7f56a0440640 1 -- 172.21.15.29:0/2684033939 wait complete. 2024-03-11T19:09:20.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.443+0000 7f56a0440640 1 Processor -- start 2024-03-11T19:09:20.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.443+0000 7f56a0440640 1 -- start start 2024-03-11T19:09:20.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.443+0000 7f56a0440640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56981000a0 0x7f56981970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:20.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.443+0000 7f56a0440640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5698101290 0x7f56981975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:20.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.443+0000 7f56a0440640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5698101c50 0x7f569819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:20.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.443+0000 7f56a0440640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569819eba0 con 0x7f56981000a0 2024-03-11T19:09:20.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.443+0000 7f56a0440640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569819ed10 con 0x7f5698101c50 2024-03-11T19:09:20.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.443+0000 7f56a0440640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569819eed0 con 0x7f5698101290 2024-03-11T19:09:20.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.443+0000 7f569e1dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56981000a0 0x7f56981970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:20.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.443+0000 7f569e1dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56981000a0 0x7f56981970b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55430/0 (socket says 172.21.15.29:55430) 2024-03-11T19:09:20.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.443+0000 7f569e1dc640 1 -- 172.21.15.29:0/2610602635 learned_addr learned my addr 172.21.15.29:0/2610602635 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:20.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.443+0000 7f569d9db640 1 --2- 172.21.15.29:0/2610602635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5698101290 0x7f56981975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.443+0000 7f569e9dd640 1 --2- 172.21.15.29:0/2610602635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5698101c50 0x7f569819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.443+0000 7f569e1dc640 1 -- 172.21.15.29:0/2610602635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5698101290 msgr2=0x7f56981975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.443+0000 7f569e1dc640 1 --2- 172.21.15.29:0/2610602635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5698101290 0x7f56981975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.443+0000 7f569e1dc640 1 -- 172.21.15.29:0/2610602635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5698101c50 msgr2=0x7f569819e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.443+0000 7f569e1dc640 1 --2- 172.21.15.29:0/2610602635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5698101c50 0x7f569819e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.444+0000 7f569e1dc640 1 -- 172.21.15.29:0/2610602635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5688009560 con 0x7f56981000a0 2024-03-11T19:09:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.444+0000 7f569e9dd640 1 --2- 172.21.15.29:0/2610602635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5698101c50 0x7f569819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.444+0000 7f569e1dc640 1 --2- 172.21.15.29:0/2610602635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56981000a0 0x7f56981970b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f569400c970 tx=0x7f569400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.444+0000 7f568f7fe640 1 -- 172.21.15.29:0/2610602635 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5694007c10 con 0x7f56981000a0 2024-03-11T19:09:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.444+0000 7f56a0440640 1 -- 172.21.15.29:0/2610602635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f569819f1b0 con 0x7f56981000a0 2024-03-11T19:09:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.444+0000 7f568f7fe640 1 -- 172.21.15.29:0/2610602635 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5694007d70 con 0x7f56981000a0 2024-03-11T19:09:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.444+0000 7f568f7fe640 1 -- 172.21.15.29:0/2610602635 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569400f740 con 0x7f56981000a0 2024-03-11T19:09:20.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.444+0000 7f56a0440640 1 -- 172.21.15.29:0/2610602635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f569819f710 con 0x7f56981000a0 2024-03-11T19:09:20.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.445+0000 7f56a0440640 1 -- 172.21.15.29:0/2610602635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f569810b960 con 0x7f56981000a0 2024-03-11T19:09:20.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.446+0000 7f568f7fe640 1 -- 172.21.15.29:0/2610602635 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5694018480 con 0x7f56981000a0 2024-03-11T19:09:20.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.446+0000 7f568f7fe640 1 --2- 172.21.15.29:0/2610602635 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f56840a1a30 0x7f56840a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:20.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.446+0000 7f569d9db640 1 --2- 172.21.15.29:0/2610602635 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f56840a1a30 0x7f56840a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:20.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.447+0000 7f568f7fe640 1 -- 172.21.15.29:0/2610602635 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5694013070 con 0x7f56981000a0 2024-03-11T19:09:20.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.447+0000 7f569d9db640 1 --2- 172.21.15.29:0/2610602635 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f56840a1a30 0x7f56840a3ef0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f56981985c0 tx=0x7f5688000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:20.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.451+0000 7f568f7fe640 1 -- 172.21.15.29:0/2610602635 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5694096a90 con 0x7f56981000a0 2024-03-11T19:09:20.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:20 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/886682663' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:09:20.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:20 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/4109692340' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:09:20.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:20 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3329503386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:09:20.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:20 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/886682663' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:09:20.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:20 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/4109692340' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:09:20.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:20 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3329503386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:09:20.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.631+0000 7f56a0440640 1 -- 172.21.15.29:0/2610602635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f569819f990 con 0x7f56981000a0 2024-03-11T19:09:20.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.631+0000 7f568f7fe640 1 -- 172.21.15.29:0/2610602635 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f569409a150 con 0x7f56981000a0 2024-03-11T19:09:20.635 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-11T19:09:20.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.633+0000 7f56a0440640 1 -- 172.21.15.29:0/2610602635 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f56840a1a30 msgr2=0x7f56840a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.633+0000 7f56a0440640 1 --2- 172.21.15.29:0/2610602635 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f56840a1a30 0x7f56840a3ef0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f56981985c0 tx=0x7f5688000bc0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.633+0000 7f56a0440640 1 -- 172.21.15.29:0/2610602635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56981000a0 msgr2=0x7f56981970b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.633+0000 7f56a0440640 1 --2- 172.21.15.29:0/2610602635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56981000a0 0x7f56981970b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f569400c970 tx=0x7f569400ce30 comp rx=0 tx=0).stop 2024-03-11T19:09:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.633+0000 7f56a0440640 1 -- 172.21.15.29:0/2610602635 shutdown_connections 2024-03-11T19:09:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.633+0000 7f56a0440640 1 --2- 172.21.15.29:0/2610602635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5698101c50 0x7f569819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.633+0000 7f56a0440640 1 --2- 172.21.15.29:0/2610602635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5698101290 0x7f56981975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.633+0000 7f56a0440640 1 --2- 172.21.15.29:0/2610602635 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f56840a1a30 0x7f56840a3ef0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.633+0000 7f56a0440640 1 --2- 172.21.15.29:0/2610602635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56981000a0 0x7f56981970b0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.633+0000 7f56a0440640 1 -- 172.21.15.29:0/2610602635 >> 172.21.15.29:0/2610602635 conn(0x7f56980fb630 msgr2=0x7f56980fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.633+0000 7f56a0440640 1 -- 172.21.15.29:0/2610602635 shutdown_connections 2024-03-11T19:09:20.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.633+0000 7f56a0440640 1 -- 172.21.15.29:0/2610602635 wait complete. 2024-03-11T19:09:20.647 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T19:09:20.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.749+0000 7f78772eb640 1 -- 172.21.15.29:0/3891107757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7870102790 msgr2=0x7f7870102bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:20.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.749+0000 7f78772eb640 1 --2- 172.21.15.29:0/3891107757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7870102790 0x7f7870102bb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f786c009900 tx=0x7f786c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:20.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.749+0000 7f78772eb640 1 -- 172.21.15.29:0/3891107757 shutdown_connections 2024-03-11T19:09:20.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.749+0000 7f78772eb640 1 --2- 172.21.15.29:0/3891107757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78700ffbc0 0x7f7870109770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.749+0000 7f78772eb640 1 --2- 172.21.15.29:0/3891107757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78700ff200 0x7f78700ff680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.749+0000 7f78772eb640 1 --2- 172.21.15.29:0/3891107757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7870102790 0x7f7870102bb0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.749+0000 7f78772eb640 1 -- 172.21.15.29:0/3891107757 >> 172.21.15.29:0/3891107757 conn(0x7f78700fad80 msgr2=0x7f78700fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:20.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.749+0000 7f78772eb640 1 -- 172.21.15.29:0/3891107757 shutdown_connections 2024-03-11T19:09:20.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.749+0000 7f78772eb640 1 -- 172.21.15.29:0/3891107757 wait complete. 2024-03-11T19:09:20.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.749+0000 7f78772eb640 1 Processor -- start 2024-03-11T19:09:20.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.749+0000 7f78772eb640 1 -- start start 2024-03-11T19:09:20.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f78772eb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78700ff200 0x7f7870197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:20.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f78772eb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78700ffbc0 0x7f78701976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:20.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f78772eb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7870102790 0x7f787019e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:20.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f78772eb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787019ec60 con 0x7f78700ff200 2024-03-11T19:09:20.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f78772eb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787019edd0 con 0x7f7870102790 2024-03-11T19:09:20.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f78772eb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787019ef90 con 0x7f78700ffbc0 2024-03-11T19:09:20.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f7875087640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78700ff200 0x7f7870197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:20.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f7875087640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78700ff200 0x7f7870197170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55438/0 (socket says 172.21.15.29:55438) 2024-03-11T19:09:20.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f7875087640 1 -- 172.21.15.29:0/2960811908 learned_addr learned my addr 172.21.15.29:0/2960811908 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:20.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f7874886640 1 --2- 172.21.15.29:0/2960811908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78700ffbc0 0x7f78701976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:20.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f7875888640 1 --2- 172.21.15.29:0/2960811908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7870102790 0x7f787019e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:20.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f7875087640 1 -- 172.21.15.29:0/2960811908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78700ffbc0 msgr2=0x7f78701976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:20.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f7875087640 1 --2- 172.21.15.29:0/2960811908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78700ffbc0 0x7f78701976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f7875087640 1 -- 172.21.15.29:0/2960811908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7870102790 msgr2=0x7f787019e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:20.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f7875087640 1 --2- 172.21.15.29:0/2960811908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7870102790 0x7f787019e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f7875087640 1 -- 172.21.15.29:0/2960811908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c009560 con 0x7f78700ff200 2024-03-11T19:09:20.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f7875888640 1 --2- 172.21.15.29:0/2960811908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7870102790 0x7f787019e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:20.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f7875087640 1 --2- 172.21.15.29:0/2960811908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78700ff200 0x7f7870197170 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f786c000c00 tx=0x7f786c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:20.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f78667fc640 1 -- 172.21.15.29:0/2960811908 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786c0044e0 con 0x7f78700ff200 2024-03-11T19:09:20.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f78667fc640 1 -- 172.21.15.29:0/2960811908 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f786c003cb0 con 0x7f78700ff200 2024-03-11T19:09:20.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f78772eb640 1 -- 172.21.15.29:0/2960811908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f787019f210 con 0x7f78700ff200 2024-03-11T19:09:20.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f78667fc640 1 -- 172.21.15.29:0/2960811908 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786c005690 con 0x7f78700ff200 2024-03-11T19:09:20.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.750+0000 7f78772eb640 1 -- 172.21.15.29:0/2960811908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f787019f720 con 0x7f78700ff200 2024-03-11T19:09:20.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.751+0000 7f78772eb640 1 -- 172.21.15.29:0/2960811908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7870108720 con 0x7f78700ff200 2024-03-11T19:09:20.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.752+0000 7f78667fc640 1 -- 172.21.15.29:0/2960811908 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f786c004dc0 con 0x7f78700ff200 2024-03-11T19:09:20.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.753+0000 7f78667fc640 1 --2- 172.21.15.29:0/2960811908 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f785c0a1a30 0x7f785c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:20.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.753+0000 7f78667fc640 1 -- 172.21.15.29:0/2960811908 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f786c0ef7f0 con 0x7f78700ff200 2024-03-11T19:09:20.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.753+0000 7f7874886640 1 --2- 172.21.15.29:0/2960811908 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f785c0a1a30 0x7f785c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:20.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.753+0000 7f7874886640 1 --2- 172.21.15.29:0/2960811908 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f785c0a1a30 0x7f785c0a3ef0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7870198680 tx=0x7f786000a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:20.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.757+0000 7f78667fc640 1 -- 172.21.15.29:0/2960811908 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f786c0b8aa0 con 0x7f78700ff200 2024-03-11T19:09:20.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.943+0000 7f78772eb640 1 -- 172.21.15.29:0/2960811908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f787010ca60 con 0x7f78700ff200 2024-03-11T19:09:20.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.944+0000 7f78667fc640 1 -- 172.21.15.29:0/2960811908 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f787010ca60 con 0x7f78700ff200 2024-03-11T19:09:20.947 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-03-11T19:09:20.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.946+0000 7f78772eb640 1 -- 172.21.15.29:0/2960811908 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f785c0a1a30 msgr2=0x7f785c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:20.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.946+0000 7f78772eb640 1 --2- 172.21.15.29:0/2960811908 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f785c0a1a30 0x7f785c0a3ef0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7870198680 tx=0x7f786000a4c0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.946+0000 7f78772eb640 1 -- 172.21.15.29:0/2960811908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78700ff200 msgr2=0x7f7870197170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:20.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.946+0000 7f78772eb640 1 --2- 172.21.15.29:0/2960811908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78700ff200 0x7f7870197170 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f786c000c00 tx=0x7f786c0042f0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.946+0000 7f78772eb640 1 -- 172.21.15.29:0/2960811908 shutdown_connections 2024-03-11T19:09:20.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.946+0000 7f78772eb640 1 --2- 172.21.15.29:0/2960811908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7870102790 0x7f787019e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.946+0000 7f78772eb640 1 --2- 172.21.15.29:0/2960811908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78700ffbc0 0x7f78701976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.946+0000 7f78772eb640 1 --2- 172.21.15.29:0/2960811908 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f785c0a1a30 0x7f785c0a3ef0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.946+0000 7f78772eb640 1 --2- 172.21.15.29:0/2960811908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78700ff200 0x7f7870197170 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:20.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.946+0000 7f78772eb640 1 -- 172.21.15.29:0/2960811908 >> 172.21.15.29:0/2960811908 conn(0x7f78700fad80 msgr2=0x7f78701058f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:20.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.946+0000 7f78772eb640 1 -- 172.21.15.29:0/2960811908 shutdown_connections 2024-03-11T19:09:20.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:20.946+0000 7f78772eb640 1 -- 172.21.15.29:0/2960811908 wait complete. 2024-03-11T19:09:20.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T19:09:21.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.063+0000 7fd68e88d640 1 -- 172.21.15.29:0/2789199545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd688102d40 msgr2=0x7fd6881095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:21.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.063+0000 7fd68e88d640 1 --2- 172.21.15.29:0/2789199545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd688102d40 0x7fd6881095d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd680009ca0 tx=0x7fd68002c0f0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.063+0000 7fd68e88d640 1 -- 172.21.15.29:0/2789199545 shutdown_connections 2024-03-11T19:09:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.063+0000 7fd68e88d640 1 --2- 172.21.15.29:0/2789199545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd688102d40 0x7fd6881095d0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.063+0000 7fd68e88d640 1 --2- 172.21.15.29:0/2789199545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd688102380 0x7fd688102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.063+0000 7fd68e88d640 1 --2- 172.21.15.29:0/2789199545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd688101190 0x7fd6881015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.063+0000 7fd68e88d640 1 -- 172.21.15.29:0/2789199545 >> 172.21.15.29:0/2789199545 conn(0x7fd6880fc720 msgr2=0x7fd6880feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.063+0000 7fd68e88d640 1 -- 172.21.15.29:0/2789199545 shutdown_connections 2024-03-11T19:09:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.063+0000 7fd68e88d640 1 -- 172.21.15.29:0/2789199545 wait complete. 2024-03-11T19:09:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.064+0000 7fd68e88d640 1 Processor -- start 2024-03-11T19:09:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.064+0000 7fd68e88d640 1 -- start start 2024-03-11T19:09:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.064+0000 7fd68e88d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd688101190 0x7fd6881970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.064+0000 7fd68e88d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd688102380 0x7fd688197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.064+0000 7fd68e88d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd688102d40 0x7fd68819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.064+0000 7fd68e88d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68819ebd0 con 0x7fd688102d40 2024-03-11T19:09:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.064+0000 7fd68e88d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68819ed40 con 0x7fd688101190 2024-03-11T19:09:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.064+0000 7fd68e88d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68819ef00 con 0x7fd688102380 2024-03-11T19:09:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.064+0000 7fd68ce2a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd688102d40 0x7fd68819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.064+0000 7fd68ce2a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd688102d40 0x7fd68819e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55460/0 (socket says 172.21.15.29:55460) 2024-03-11T19:09:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.064+0000 7fd68ce2a640 1 -- 172.21.15.29:0/3289698521 learned_addr learned my addr 172.21.15.29:0/3289698521 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.064+0000 7fd687fff640 1 --2- 172.21.15.29:0/3289698521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd688101190 0x7fd6881970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.064+0000 7fd6877fe640 1 --2- 172.21.15.29:0/3289698521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd688102380 0x7fd688197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.064+0000 7fd68ce2a640 1 -- 172.21.15.29:0/3289698521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd688102380 msgr2=0x7fd688197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.064+0000 7fd68ce2a640 1 --2- 172.21.15.29:0/3289698521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd688102380 0x7fd688197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.064+0000 7fd68ce2a640 1 -- 172.21.15.29:0/3289698521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd688101190 msgr2=0x7fd6881970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.064+0000 7fd68ce2a640 1 --2- 172.21.15.29:0/3289698521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd688101190 0x7fd6881970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.064+0000 7fd68ce2a640 1 -- 172.21.15.29:0/3289698521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd680009900 con 0x7fd688102d40 2024-03-11T19:09:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.065+0000 7fd68ce2a640 1 --2- 172.21.15.29:0/3289698521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd688102d40 0x7fd68819e690 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd680037040 tx=0x7fd680032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.065+0000 7fd6857fa640 1 -- 172.21.15.29:0/3289698521 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd680034e70 con 0x7fd688102d40 2024-03-11T19:09:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.065+0000 7fd6857fa640 1 -- 172.21.15.29:0/3289698521 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd680003e20 con 0x7fd688102d40 2024-03-11T19:09:21.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.065+0000 7fd68e88d640 1 -- 172.21.15.29:0/3289698521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68819f180 con 0x7fd688102d40 2024-03-11T19:09:21.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.065+0000 7fd6857fa640 1 -- 172.21.15.29:0/3289698521 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68003d3f0 con 0x7fd688102d40 2024-03-11T19:09:21.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.065+0000 7fd68e88d640 1 -- 172.21.15.29:0/3289698521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd68819f630 con 0x7fd688102d40 2024-03-11T19:09:21.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.066+0000 7fd68e88d640 1 -- 172.21.15.29:0/3289698521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6881084b0 con 0x7fd688102d40 2024-03-11T19:09:21.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.067+0000 7fd6857fa640 1 -- 172.21.15.29:0/3289698521 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd68003b040 con 0x7fd688102d40 2024-03-11T19:09:21.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.067+0000 7fd6857fa640 1 --2- 172.21.15.29:0/3289698521 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd6700a1a30 0x7fd6700a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:21.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.068+0000 7fd687fff640 1 --2- 172.21.15.29:0/3289698521 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd6700a1a30 0x7fd6700a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:21.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.068+0000 7fd6857fa640 1 -- 172.21.15.29:0/3289698521 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd6800efbc0 con 0x7fd688102d40 2024-03-11T19:09:21.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.068+0000 7fd687fff640 1 --2- 172.21.15.29:0/3289698521 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd6700a1a30 0x7fd6700a3ef0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd678005db0 tx=0x7fd678005d40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:21.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.072+0000 7fd6857fa640 1 -- 172.21.15.29:0/3289698521 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd6800b8e70 con 0x7fd688102d40 2024-03-11T19:09:21.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.255+0000 7fd68e88d640 1 -- 172.21.15.29:0/3289698521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd68802cda0 con 0x7fd688102d40 2024-03-11T19:09:21.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.255+0000 7fd6857fa640 1 -- 172.21.15.29:0/3289698521 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd6800bc530 con 0x7fd688102d40 2024-03-11T19:09:21.258 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-03-11T19:09:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.257+0000 7fd68e88d640 1 -- 172.21.15.29:0/3289698521 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd6700a1a30 msgr2=0x7fd6700a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.257+0000 7fd68e88d640 1 --2- 172.21.15.29:0/3289698521 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd6700a1a30 0x7fd6700a3ef0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd678005db0 tx=0x7fd678005d40 comp rx=0 tx=0).stop 2024-03-11T19:09:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.257+0000 7fd68e88d640 1 -- 172.21.15.29:0/3289698521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd688102d40 msgr2=0x7fd68819e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.257+0000 7fd68e88d640 1 --2- 172.21.15.29:0/3289698521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd688102d40 0x7fd68819e690 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd680037040 tx=0x7fd680032980 comp rx=0 tx=0).stop 2024-03-11T19:09:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.257+0000 7fd68e88d640 1 -- 172.21.15.29:0/3289698521 shutdown_connections 2024-03-11T19:09:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.257+0000 7fd68e88d640 1 --2- 172.21.15.29:0/3289698521 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd6700a1a30 0x7fd6700a3ef0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.257+0000 7fd68e88d640 1 --2- 172.21.15.29:0/3289698521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd688102d40 0x7fd68819e690 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.257+0000 7fd68e88d640 1 --2- 172.21.15.29:0/3289698521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd688102380 0x7fd688197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.257+0000 7fd68e88d640 1 --2- 172.21.15.29:0/3289698521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd688101190 0x7fd6881970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.257+0000 7fd68e88d640 1 -- 172.21.15.29:0/3289698521 >> 172.21.15.29:0/3289698521 conn(0x7fd6880fc720 msgr2=0x7fd688105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.257+0000 7fd68e88d640 1 -- 172.21.15.29:0/3289698521 shutdown_connections 2024-03-11T19:09:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.257+0000 7fd68e88d640 1 -- 172.21.15.29:0/3289698521 wait complete. 2024-03-11T19:09:21.270 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:21 smithi029 ceph-mon[32439]: pgmap v337: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-11T19:09:21.270 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:21 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1059086613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:21.270 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:21 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2610602635' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:09:21.270 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:21 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2960811908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:09:21.271 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:09:21.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.370+0000 7f9d79d02640 1 -- 172.21.15.29:0/3167163211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d741023a0 msgr2=0x7f9d74102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:21.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.370+0000 7f9d79d02640 1 --2- 172.21.15.29:0/3167163211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d741023a0 0x7f9d74102820 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9d68009900 tx=0x7f9d6802bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:21.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.370+0000 7f9d79d02640 1 -- 172.21.15.29:0/3167163211 shutdown_connections 2024-03-11T19:09:21.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.370+0000 7f9d79d02640 1 --2- 172.21.15.29:0/3167163211 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d74102d60 0x7f9d741095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.370+0000 7f9d79d02640 1 --2- 172.21.15.29:0/3167163211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d741023a0 0x7f9d74102820 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.370+0000 7f9d79d02640 1 --2- 172.21.15.29:0/3167163211 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d741011b0 0x7f9d741015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.370+0000 7f9d79d02640 1 -- 172.21.15.29:0/3167163211 >> 172.21.15.29:0/3167163211 conn(0x7f9d740fc760 msgr2=0x7f9d740feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:21.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.370+0000 7f9d79d02640 1 -- 172.21.15.29:0/3167163211 shutdown_connections 2024-03-11T19:09:21.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.370+0000 7f9d79d02640 1 -- 172.21.15.29:0/3167163211 wait complete. 2024-03-11T19:09:21.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.371+0000 7f9d79d02640 1 Processor -- start 2024-03-11T19:09:21.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.371+0000 7f9d79d02640 1 -- start start 2024-03-11T19:09:21.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.371+0000 7f9d79d02640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d741011b0 0x7f9d74074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:21.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.371+0000 7f9d79d02640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d741023a0 0x7f9d74073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:21.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.371+0000 7f9d79d02640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d74102d60 0x7f9d74073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:21.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.371+0000 7f9d79d02640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d74074210 con 0x7f9d741023a0 2024-03-11T19:09:21.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.371+0000 7f9d79d02640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d74074380 con 0x7f9d741011b0 2024-03-11T19:09:21.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.371+0000 7f9d79d02640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d741a10d0 con 0x7f9d74102d60 2024-03-11T19:09:21.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.371+0000 7f9d72ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d741023a0 0x7f9d74073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:21.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.371+0000 7f9d72ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d741023a0 0x7f9d74073570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55480/0 (socket says 172.21.15.29:55480) 2024-03-11T19:09:21.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.371+0000 7f9d72ffd640 1 -- 172.21.15.29:0/3059406438 learned_addr learned my addr 172.21.15.29:0/3059406438 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:21.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.372+0000 7f9d737fe640 1 --2- 172.21.15.29:0/3059406438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d741011b0 0x7f9d74074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:21.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.372+0000 7f9d72ffd640 1 -- 172.21.15.29:0/3059406438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d74102d60 msgr2=0x7f9d74073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:21.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.372+0000 7f9d72ffd640 1 --2- 172.21.15.29:0/3059406438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d74102d60 0x7f9d74073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.372+0000 7f9d72ffd640 1 -- 172.21.15.29:0/3059406438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d741011b0 msgr2=0x7f9d74074f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:21.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.372+0000 7f9d72ffd640 1 --2- 172.21.15.29:0/3059406438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d741011b0 0x7f9d74074f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.372+0000 7f9d72ffd640 1 -- 172.21.15.29:0/3059406438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d68009560 con 0x7f9d741023a0 2024-03-11T19:09:21.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.372+0000 7f9d72ffd640 1 --2- 172.21.15.29:0/3059406438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d741023a0 0x7f9d74073570 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9d68009430 tx=0x7f9d68004370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:21.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.372+0000 7f9d70ff9640 1 -- 172.21.15.29:0/3059406438 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d68004fd0 con 0x7f9d741023a0 2024-03-11T19:09:21.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.372+0000 7f9d70ff9640 1 -- 172.21.15.29:0/3059406438 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9d6802de90 con 0x7f9d741023a0 2024-03-11T19:09:21.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.372+0000 7f9d79d02640 1 -- 172.21.15.29:0/3059406438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d741a1320 con 0x7f9d741023a0 2024-03-11T19:09:21.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.372+0000 7f9d70ff9640 1 -- 172.21.15.29:0/3059406438 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d68003e20 con 0x7f9d741023a0 2024-03-11T19:09:21.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.372+0000 7f9d79d02640 1 -- 172.21.15.29:0/3059406438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d741a1830 con 0x7f9d741023a0 2024-03-11T19:09:21.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.373+0000 7f9d79d02640 1 -- 172.21.15.29:0/3059406438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d741084d0 con 0x7f9d741023a0 2024-03-11T19:09:21.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.374+0000 7f9d70ff9640 1 -- 172.21.15.29:0/3059406438 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9d680035f0 con 0x7f9d741023a0 2024-03-11T19:09:21.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.374+0000 7f9d70ff9640 1 --2- 172.21.15.29:0/3059406438 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9d5c0a1a30 0x7f9d5c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:21.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.375+0000 7f9d737fe640 1 --2- 172.21.15.29:0/3059406438 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9d5c0a1a30 0x7f9d5c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:21.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.375+0000 7f9d70ff9640 1 -- 172.21.15.29:0/3059406438 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9d680ef800 con 0x7f9d741023a0 2024-03-11T19:09:21.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.375+0000 7f9d737fe640 1 --2- 172.21.15.29:0/3059406438 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9d5c0a1a30 0x7f9d5c0a3ef0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9d64005fd0 tx=0x7f9d64005ce0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:21.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.379+0000 7f9d70ff9640 1 -- 172.21.15.29:0/3059406438 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9d680b8ab0 con 0x7f9d741023a0 2024-03-11T19:09:21.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:21 smithi190 ceph-mon[36292]: pgmap v337: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-11T19:09:21.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:21 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1059086613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:21.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:21 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2610602635' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:09:21.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:21 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2960811908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:09:21.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:21 smithi170 ceph-mon[35690]: pgmap v337: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-11T19:09:21.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:21 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1059086613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:21.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:21 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2610602635' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:09:21.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:21 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2960811908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:09:21.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.595+0000 7f9d79d02640 1 -- 172.21.15.29:0/3059406438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d7402ce00 con 0x7f9d741023a0 2024-03-11T19:09:21.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.596+0000 7f9d70ff9640 1 -- 172.21.15.29:0/3059406438 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f9d680bc170 con 0x7f9d741023a0 2024-03-11T19:09:21.599 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:21.599 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:07:56.662890+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:09:21.599 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 26 2024-03-11T19:09:21.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.598+0000 7f9d79d02640 1 -- 172.21.15.29:0/3059406438 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9d5c0a1a30 msgr2=0x7f9d5c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:21.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.598+0000 7f9d79d02640 1 --2- 172.21.15.29:0/3059406438 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9d5c0a1a30 0x7f9d5c0a3ef0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9d64005fd0 tx=0x7f9d64005ce0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.598+0000 7f9d79d02640 1 -- 172.21.15.29:0/3059406438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d741023a0 msgr2=0x7f9d74073570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:21.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.598+0000 7f9d79d02640 1 --2- 172.21.15.29:0/3059406438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d741023a0 0x7f9d74073570 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9d68009430 tx=0x7f9d68004370 comp rx=0 tx=0).stop 2024-03-11T19:09:21.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.598+0000 7f9d79d02640 1 -- 172.21.15.29:0/3059406438 shutdown_connections 2024-03-11T19:09:21.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.598+0000 7f9d79d02640 1 --2- 172.21.15.29:0/3059406438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d74102d60 0x7f9d74073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.598+0000 7f9d79d02640 1 --2- 172.21.15.29:0/3059406438 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9d5c0a1a30 0x7f9d5c0a3ef0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.598+0000 7f9d79d02640 1 --2- 172.21.15.29:0/3059406438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d741023a0 0x7f9d74073570 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.598+0000 7f9d79d02640 1 --2- 172.21.15.29:0/3059406438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d741011b0 0x7f9d74074f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.598+0000 7f9d79d02640 1 -- 172.21.15.29:0/3059406438 >> 172.21.15.29:0/3059406438 conn(0x7f9d740fc760 msgr2=0x7f9d741056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:21.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.598+0000 7f9d79d02640 1 -- 172.21.15.29:0/3059406438 shutdown_connections 2024-03-11T19:09:21.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.598+0000 7f9d79d02640 1 -- 172.21.15.29:0/3059406438 wait complete. 2024-03-11T19:09:21.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:09:21.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.711+0000 7f163aa98640 1 -- 172.21.15.29:0/3968263878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16340ff490 msgr2=0x7f16340ff8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:21.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.711+0000 7f163aa98640 1 --2- 172.21.15.29:0/3968263878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16340ff490 0x7f16340ff8b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f162c009ca0 tx=0x7f162c02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.711+0000 7f163aa98640 1 -- 172.21.15.29:0/3968263878 shutdown_connections 2024-03-11T19:09:21.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.711+0000 7f163aa98640 1 --2- 172.21.15.29:0/3968263878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1634102e20 0x7f16341096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.711+0000 7f163aa98640 1 --2- 172.21.15.29:0/3968263878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1634102460 0x7f16341028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.711+0000 7f163aa98640 1 --2- 172.21.15.29:0/3968263878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16340ff490 0x7f16340ff8b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.711+0000 7f163aa98640 1 -- 172.21.15.29:0/3968263878 >> 172.21.15.29:0/3968263878 conn(0x7f16340fadc0 msgr2=0x7f16340fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.711+0000 7f163aa98640 1 -- 172.21.15.29:0/3968263878 shutdown_connections 2024-03-11T19:09:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.711+0000 7f163aa98640 1 -- 172.21.15.29:0/3968263878 wait complete. 2024-03-11T19:09:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.712+0000 7f163aa98640 1 Processor -- start 2024-03-11T19:09:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.712+0000 7f163aa98640 1 -- start start 2024-03-11T19:09:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.712+0000 7f163aa98640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16340ff490 0x7f16341971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.712+0000 7f163aa98640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1634102460 0x7f16341976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.712+0000 7f163aa98640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1634102e20 0x7f163419e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.712+0000 7f163aa98640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163419eca0 con 0x7f1634102e20 2024-03-11T19:09:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.712+0000 7f163aa98640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163419ee10 con 0x7f16340ff490 2024-03-11T19:09:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.712+0000 7f163aa98640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163419efd0 con 0x7f1634102460 2024-03-11T19:09:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.712+0000 7f1639035640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1634102e20 0x7f163419e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.712+0000 7f1639035640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1634102e20 0x7f163419e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55492/0 (socket says 172.21.15.29:55492) 2024-03-11T19:09:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.712+0000 7f1639035640 1 -- 172.21.15.29:0/1673842143 learned_addr learned my addr 172.21.15.29:0/1673842143 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.712+0000 7f1638834640 1 --2- 172.21.15.29:0/1673842143 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16340ff490 0x7f16341971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.712+0000 7f1633fff640 1 --2- 172.21.15.29:0/1673842143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1634102460 0x7f16341976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.712+0000 7f1639035640 1 -- 172.21.15.29:0/1673842143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1634102460 msgr2=0x7f16341976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.712+0000 7f1639035640 1 --2- 172.21.15.29:0/1673842143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1634102460 0x7f16341976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.712+0000 7f1639035640 1 -- 172.21.15.29:0/1673842143 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16340ff490 msgr2=0x7f16341971b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.712+0000 7f1639035640 1 --2- 172.21.15.29:0/1673842143 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16340ff490 0x7f16341971b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.712+0000 7f1639035640 1 -- 172.21.15.29:0/1673842143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162c009900 con 0x7f1634102e20 2024-03-11T19:09:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.712+0000 7f1638834640 1 --2- 172.21.15.29:0/1673842143 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16340ff490 0x7f16341971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.713+0000 7f1639035640 1 --2- 172.21.15.29:0/1673842143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1634102e20 0x7f163419e760 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f162400d750 tx=0x7f162400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.713+0000 7f1631ffb640 1 -- 172.21.15.29:0/1673842143 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1624009bd0 con 0x7f1634102e20 2024-03-11T19:09:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.713+0000 7f1631ffb640 1 -- 172.21.15.29:0/1673842143 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f162400de60 con 0x7f1634102e20 2024-03-11T19:09:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.713+0000 7f163aa98640 1 -- 172.21.15.29:0/1673842143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f163419f2b0 con 0x7f1634102e20 2024-03-11T19:09:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.713+0000 7f1631ffb640 1 -- 172.21.15.29:0/1673842143 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1624004e00 con 0x7f1634102e20 2024-03-11T19:09:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.713+0000 7f163aa98640 1 -- 172.21.15.29:0/1673842143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f163419f7c0 con 0x7f1634102e20 2024-03-11T19:09:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.714+0000 7f163aa98640 1 -- 172.21.15.29:0/1673842143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1634108600 con 0x7f1634102e20 2024-03-11T19:09:21.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.715+0000 7f1631ffb640 1 -- 172.21.15.29:0/1673842143 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1624004f60 con 0x7f1634102e20 2024-03-11T19:09:21.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.716+0000 7f1631ffb640 1 --2- 172.21.15.29:0/1673842143 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f161c0a1ab0 0x7f161c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:21.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.716+0000 7f1638834640 1 --2- 172.21.15.29:0/1673842143 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f161c0a1ab0 0x7f161c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:21.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.716+0000 7f1631ffb640 1 -- 172.21.15.29:0/1673842143 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f16240cccb0 con 0x7f1634102e20 2024-03-11T19:09:21.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.716+0000 7f1638834640 1 --2- 172.21.15.29:0/1673842143 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f161c0a1ab0 0x7f161c0a3f70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f162c02c700 tx=0x7f162c0325f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:21.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.720+0000 7f1631ffb640 1 -- 172.21.15.29:0/1673842143 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1624095f60 con 0x7f1634102e20 2024-03-11T19:09:21.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.902+0000 7f163aa98640 1 -- 172.21.15.29:0/1673842143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1634108070 con 0x7f1634102e20 2024-03-11T19:09:21.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.904+0000 7f1631ffb640 1 -- 172.21.15.29:0/1673842143 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22559 (secure 0 0 0) 0x7f1624017090 con 0x7f1634102e20 2024-03-11T19:09:21.907 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:21.907 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":89,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:08:44.981806+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:07:47.563184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:07:48.282461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:09:21.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.906+0000 7f163aa98640 1 -- 172.21.15.29:0/1673842143 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f161c0a1ab0 msgr2=0x7f161c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:21.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.906+0000 7f163aa98640 1 --2- 172.21.15.29:0/1673842143 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f161c0a1ab0 0x7f161c0a3f70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f162c02c700 tx=0x7f162c0325f0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.906+0000 7f163aa98640 1 -- 172.21.15.29:0/1673842143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1634102e20 msgr2=0x7f163419e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:21.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.906+0000 7f163aa98640 1 --2- 172.21.15.29:0/1673842143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1634102e20 0x7f163419e760 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f162400d750 tx=0x7f162400dc10 comp rx=0 tx=0).stop 2024-03-11T19:09:21.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.906+0000 7f163aa98640 1 -- 172.21.15.29:0/1673842143 shutdown_connections 2024-03-11T19:09:21.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.906+0000 7f163aa98640 1 --2- 172.21.15.29:0/1673842143 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f161c0a1ab0 0x7f161c0a3f70 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.906+0000 7f163aa98640 1 --2- 172.21.15.29:0/1673842143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1634102e20 0x7f163419e760 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.906+0000 7f163aa98640 1 --2- 172.21.15.29:0/1673842143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1634102460 0x7f16341976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.906+0000 7f163aa98640 1 --2- 172.21.15.29:0/1673842143 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16340ff490 0x7f16341971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:21.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.906+0000 7f163aa98640 1 -- 172.21.15.29:0/1673842143 >> 172.21.15.29:0/1673842143 conn(0x7f16340fadc0 msgr2=0x7f16341057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:21.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.906+0000 7f163aa98640 1 -- 172.21.15.29:0/1673842143 shutdown_connections 2024-03-11T19:09:21.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:21.906+0000 7f163aa98640 1 -- 172.21.15.29:0/1673842143 wait complete. 2024-03-11T19:09:21.920 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:09:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.019+0000 7f24771ac640 1 -- 172.21.15.29:0/2318832368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2470101cd0 msgr2=0x7f2470108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.019+0000 7f24771ac640 1 --2- 172.21.15.29:0/2318832368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2470101cd0 0x7f2470108560 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2468009ca0 tx=0x7f246802c0f0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.019+0000 7f24771ac640 1 -- 172.21.15.29:0/2318832368 shutdown_connections 2024-03-11T19:09:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.019+0000 7f24771ac640 1 --2- 172.21.15.29:0/2318832368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2470101cd0 0x7f2470108560 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.019+0000 7f24771ac640 1 --2- 172.21.15.29:0/2318832368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2470101310 0x7f2470101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.019+0000 7f24771ac640 1 --2- 172.21.15.29:0/2318832368 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2470100120 0x7f2470100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.019+0000 7f24771ac640 1 -- 172.21.15.29:0/2318832368 >> 172.21.15.29:0/2318832368 conn(0x7f24700fb6b0 msgr2=0x7f24700fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.019+0000 7f24771ac640 1 -- 172.21.15.29:0/2318832368 shutdown_connections 2024-03-11T19:09:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.019+0000 7f24771ac640 1 -- 172.21.15.29:0/2318832368 wait complete. 2024-03-11T19:09:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.020+0000 7f24771ac640 1 Processor -- start 2024-03-11T19:09:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.020+0000 7f24771ac640 1 -- start start 2024-03-11T19:09:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.020+0000 7f24771ac640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2470100120 0x7f247006a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.020+0000 7f24771ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2470101310 0x7f247019bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.020+0000 7f24771ac640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2470101cd0 0x7f247019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.020+0000 7f24771ac640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247019ebd0 con 0x7f2470101310 2024-03-11T19:09:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.020+0000 7f24771ac640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247019ed40 con 0x7f2470101cd0 2024-03-11T19:09:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.020+0000 7f24771ac640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247019ef00 con 0x7f2470100120 2024-03-11T19:09:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.020+0000 7f246ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2470101310 0x7f247019bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.020+0000 7f246ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2470101310 0x7f247019bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55506/0 (socket says 172.21.15.29:55506) 2024-03-11T19:09:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.020+0000 7f246ffff640 1 -- 172.21.15.29:0/3160339995 learned_addr learned my addr 172.21.15.29:0/3160339995 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.020+0000 7f246ffff640 1 -- 172.21.15.29:0/3160339995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2470100120 msgr2=0x7f247006a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.020+0000 7f2474f48640 1 --2- 172.21.15.29:0/3160339995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2470100120 0x7f247006a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.020+0000 7f246ffff640 1 --2- 172.21.15.29:0/3160339995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2470100120 0x7f247006a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.020+0000 7f246ffff640 1 -- 172.21.15.29:0/3160339995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2470101cd0 msgr2=0x7f247019e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.020+0000 7f2475749640 1 --2- 172.21.15.29:0/3160339995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2470101cd0 0x7f247019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.020+0000 7f246ffff640 1 --2- 172.21.15.29:0/3160339995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2470101cd0 0x7f247019e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.020+0000 7f246ffff640 1 -- 172.21.15.29:0/3160339995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2468009900 con 0x7f2470101310 2024-03-11T19:09:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.020+0000 7f246ffff640 1 --2- 172.21.15.29:0/3160339995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2470101310 0x7f247019bdd0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f245c00c920 tx=0x7f245c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.021+0000 7f246dffb640 1 -- 172.21.15.29:0/3160339995 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c007c70 con 0x7f2470101310 2024-03-11T19:09:22.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.021+0000 7f24771ac640 1 -- 172.21.15.29:0/3160339995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f247019f180 con 0x7f2470101310 2024-03-11T19:09:22.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.021+0000 7f246dffb640 1 -- 172.21.15.29:0/3160339995 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f245c007dd0 con 0x7f2470101310 2024-03-11T19:09:22.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.021+0000 7f246dffb640 1 -- 172.21.15.29:0/3160339995 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c0187c0 con 0x7f2470101310 2024-03-11T19:09:22.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.021+0000 7f24771ac640 1 -- 172.21.15.29:0/3160339995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247019f690 con 0x7f2470101310 2024-03-11T19:09:22.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.022+0000 7f24771ac640 1 -- 172.21.15.29:0/3160339995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2470107440 con 0x7f2470101310 2024-03-11T19:09:22.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.023+0000 7f246dffb640 1 -- 172.21.15.29:0/3160339995 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f245c00f450 con 0x7f2470101310 2024-03-11T19:09:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.023+0000 7f246dffb640 1 --2- 172.21.15.29:0/3160339995 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f24600a1a30 0x7f24600a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.023+0000 7f2474f48640 1 --2- 172.21.15.29:0/3160339995 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f24600a1a30 0x7f24600a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.023+0000 7f246dffb640 1 -- 172.21.15.29:0/3160339995 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f245c0cd500 con 0x7f2470101310 2024-03-11T19:09:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.024+0000 7f2474f48640 1 --2- 172.21.15.29:0/3160339995 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f24600a1a30 0x7f24600a3ef0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2464005fd0 tx=0x7f2464005c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.028+0000 7f246dffb640 1 -- 172.21.15.29:0/3160339995 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f245c0967b0 con 0x7f2470101310 2024-03-11T19:09:22.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.246+0000 7f24771ac640 1 -- 172.21.15.29:0/3160339995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f247002ce40 con 0x7f2470101310 2024-03-11T19:09:22.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.246+0000 7f246dffb640 1 -- 172.21.15.29:0/3160339995 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f245c099e70 con 0x7f2470101310 2024-03-11T19:09:22.249 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:22.250 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:07:56.662890+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:09:22.250 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 26 2024-03-11T19:09:22.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.248+0000 7f24771ac640 1 -- 172.21.15.29:0/3160339995 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f24600a1a30 msgr2=0x7f24600a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.248+0000 7f24771ac640 1 --2- 172.21.15.29:0/3160339995 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f24600a1a30 0x7f24600a3ef0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2464005fd0 tx=0x7f2464005c70 comp rx=0 tx=0).stop 2024-03-11T19:09:22.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.248+0000 7f24771ac640 1 -- 172.21.15.29:0/3160339995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2470101310 msgr2=0x7f247019bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.248+0000 7f24771ac640 1 --2- 172.21.15.29:0/3160339995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2470101310 0x7f247019bdd0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f245c00c920 tx=0x7f245c00cde0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.248+0000 7f24771ac640 1 -- 172.21.15.29:0/3160339995 shutdown_connections 2024-03-11T19:09:22.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.248+0000 7f24771ac640 1 --2- 172.21.15.29:0/3160339995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2470101cd0 0x7f247019e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.248+0000 7f24771ac640 1 --2- 172.21.15.29:0/3160339995 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f24600a1a30 0x7f24600a3ef0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.248+0000 7f24771ac640 1 --2- 172.21.15.29:0/3160339995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2470101310 0x7f247019bdd0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.248+0000 7f24771ac640 1 --2- 172.21.15.29:0/3160339995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2470100120 0x7f247006a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.248+0000 7f24771ac640 1 -- 172.21.15.29:0/3160339995 >> 172.21.15.29:0/3160339995 conn(0x7f24700fb6b0 msgr2=0x7f2470104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.248+0000 7f24771ac640 1 -- 172.21.15.29:0/3160339995 shutdown_connections 2024-03-11T19:09:22.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.248+0000 7f24771ac640 1 -- 172.21.15.29:0/3160339995 wait complete. 2024-03-11T19:09:22.261 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:22 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3289698521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:09:22.261 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:22 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3059406438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:09:22.261 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:22 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1673842143' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:22.263 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-11T19:07:49.123624+0000', 'modified': '2024-03-11T19:07:56.662890+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24506, 'mds_1': 24496, 'mds_2': 24475, 'mds_3': 24473, 'mds_4': 24463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14616': {'gid': 14616, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.29:6839/72765211', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6838', 'nonce': 72765211}, {'type': 'v1', 'addr': '172.21.15.29:6839', 'nonce': 72765211}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_14628': {'gid': 14628, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.29:6841/3154444865', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6840', 'nonce': 3154444865}, {'type': 'v1', 'addr': '172.21.15.29:6841', 'nonce': 3154444865}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24407': {'gid': 24407, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.170:6833/1035391240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 1035391240}, {'type': 'v1', 'addr': '172.21.15.170:6833', 'nonce': 1035391240}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24418': {'gid': 24418, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.190:6833/493725011', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6832', 'nonce': 493725011}, {'type': 'v1', 'addr': '172.21.15.190:6833', 'nonce': 493725011}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24449': {'gid': 24449, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.170:6835/2555518515', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6834', 'nonce': 2555518515}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 2555518515}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24463': {'gid': 24463, 'name': 'f', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.190:6835/1048234253', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6834', 'nonce': 1048234253}, {'type': 'v1', 'addr': '172.21.15.190:6835', 'nonce': 1048234253}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24473': {'gid': 24473, 'name': 'h', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.170:6837/585739236', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6836', 'nonce': 585739236}, {'type': 'v1', 'addr': '172.21.15.170:6837', 'nonce': 585739236}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24475': {'gid': 24475, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.190:6837/3595456607', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6836', 'nonce': 3595456607}, {'type': 'v1', 'addr': '172.21.15.190:6837', 'nonce': 3595456607}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24496': {'gid': 24496, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.190:6839/2031437522', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6838', 'nonce': 2031437522}, {'type': 'v1', 'addr': '172.21.15.190:6839', 'nonce': 2031437522}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24506': {'gid': 24506, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.170:6839/4145997740', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6838', 'nonce': 4145997740}, {'type': 'v1', 'addr': '172.21.15.170:6839', 'nonce': 4145997740}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-11T19:09:22.263 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-11T19:09:22.263 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-11T19:09:22.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.364+0000 7fbe52cb5640 1 -- 172.21.15.29:0/524903065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe4c106e80 msgr2=0x7fbe4c1092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.364+0000 7fbe52cb5640 1 --2- 172.21.15.29:0/524903065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe4c106e80 0x7fbe4c1092d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009ca0 tx=0x7fbe4402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.364+0000 7fbe52cb5640 1 -- 172.21.15.29:0/524903065 shutdown_connections 2024-03-11T19:09:22.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.364+0000 7fbe52cb5640 1 --2- 172.21.15.29:0/524903065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe4c106e80 0x7fbe4c1092d0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.364+0000 7fbe52cb5640 1 --2- 172.21.15.29:0/524903065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe4c104550 0x7fbe4c106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.364+0000 7fbe52cb5640 1 --2- 172.21.15.29:0/524903065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe4c101c20 0x7fbe4c104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.364+0000 7fbe52cb5640 1 -- 172.21.15.29:0/524903065 >> 172.21.15.29:0/524903065 conn(0x7fbe4c0fb630 msgr2=0x7fbe4c0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.364+0000 7fbe52cb5640 1 -- 172.21.15.29:0/524903065 shutdown_connections 2024-03-11T19:09:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.364+0000 7fbe52cb5640 1 -- 172.21.15.29:0/524903065 wait complete. 2024-03-11T19:09:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.365+0000 7fbe52cb5640 1 Processor -- start 2024-03-11T19:09:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.365+0000 7fbe52cb5640 1 -- start start 2024-03-11T19:09:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.365+0000 7fbe52cb5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe4c101c20 0x7fbe4c197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.365+0000 7fbe52cb5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe4c104550 0x7fbe4c197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.365+0000 7fbe52cb5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe4c106e80 0x7fbe4c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.365+0000 7fbe52cb5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c19ec10 con 0x7fbe4c104550 2024-03-11T19:09:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.365+0000 7fbe52cb5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c19ed80 con 0x7fbe4c101c20 2024-03-11T19:09:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.365+0000 7fbe52cb5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c19ef40 con 0x7fbe4c106e80 2024-03-11T19:09:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.365+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe4c104550 0x7fbe4c197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.365+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe4c104550 0x7fbe4c197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55524/0 (socket says 172.21.15.29:55524) 2024-03-11T19:09:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.365+0000 7fbe4bfff640 1 -- 172.21.15.29:0/1562544078 learned_addr learned my addr 172.21.15.29:0/1562544078 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.365+0000 7fbe51252640 1 --2- 172.21.15.29:0/1562544078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe4c106e80 0x7fbe4c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.365+0000 7fbe50a51640 1 --2- 172.21.15.29:0/1562544078 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe4c101c20 0x7fbe4c197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.365+0000 7fbe4bfff640 1 -- 172.21.15.29:0/1562544078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe4c106e80 msgr2=0x7fbe4c19e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.365+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/1562544078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe4c106e80 0x7fbe4c19e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.365+0000 7fbe4bfff640 1 -- 172.21.15.29:0/1562544078 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe4c101c20 msgr2=0x7fbe4c197120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.365+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/1562544078 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe4c101c20 0x7fbe4c197120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.365+0000 7fbe4bfff640 1 -- 172.21.15.29:0/1562544078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe44009900 con 0x7fbe4c104550 2024-03-11T19:09:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.365+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/1562544078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe4c104550 0x7fbe4c197660 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbe3800c970 tx=0x7fbe3800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.366+0000 7fbe49ffb640 1 -- 172.21.15.29:0/1562544078 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe38007bb0 con 0x7fbe4c104550 2024-03-11T19:09:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.366+0000 7fbe49ffb640 1 -- 172.21.15.29:0/1562544078 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbe38007d10 con 0x7fbe4c104550 2024-03-11T19:09:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.366+0000 7fbe49ffb640 1 -- 172.21.15.29:0/1562544078 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3800f450 con 0x7fbe4c104550 2024-03-11T19:09:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.366+0000 7fbe52cb5640 1 -- 172.21.15.29:0/1562544078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe4c19f220 con 0x7fbe4c104550 2024-03-11T19:09:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.366+0000 7fbe52cb5640 1 -- 172.21.15.29:0/1562544078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe4c19f730 con 0x7fbe4c104550 2024-03-11T19:09:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.368+0000 7fbe49ffb640 1 -- 172.21.15.29:0/1562544078 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fbe38004eb0 con 0x7fbe4c104550 2024-03-11T19:09:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.368+0000 7fbe49ffb640 1 --2- 172.21.15.29:0/1562544078 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbe3c0a1a30 0x7fbe3c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.368+0000 7fbe55967740 1 Processor -- start 2024-03-11T19:09:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.368+0000 7fbe55967740 1 -- start start 2024-03-11T19:09:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.368+0000 7fbe50a51640 1 --2- 172.21.15.29:0/1562544078 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbe3c0a1a30 0x7fbe3c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.368+0000 7fbe49ffb640 1 -- 172.21.15.29:0/1562544078 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fbe380cd360 con 0x7fbe4c104550 2024-03-11T19:09:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.368+0000 7fbe55967740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28f6f80 0x5643e28f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.368+0000 7fbe55967740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 0x5643e28fc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.368+0000 7fbe55967740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643e28fcae0 0x5643e28fef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.368+0000 7fbe55967740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e28b2cd0 con 0x5643e28f7970 2024-03-11T19:09:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.368+0000 7fbe55967740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e2865f20 con 0x5643e28f6f80 2024-03-11T19:09:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.368+0000 7fbe55967740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e2863450 con 0x5643e28fcae0 2024-03-11T19:09:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.368+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 0x5643e28fc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.369+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 0x5643e28fc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55538/0 (socket says 172.21.15.29:55538) 2024-03-11T19:09:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.369+0000 7fbe4bfff640 1 -- 172.21.15.29:0/4185683751 learned_addr learned my addr 172.21.15.29:0/4185683751 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.369+0000 7fbe50a51640 1 --2- 172.21.15.29:0/1562544078 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbe3c0a1a30 0x7fbe3c0a3ef0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009bb0 tx=0x7fbe40005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.369+0000 7fbe51252640 1 --2- 172.21.15.29:0/4185683751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643e28fcae0 0x5643e28fef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.369+0000 7fbe50a51640 1 --2- 172.21.15.29:0/4185683751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28f6f80 0x5643e28f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.369+0000 7fbe4bfff640 1 -- 172.21.15.29:0/4185683751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643e28fcae0 msgr2=0x5643e28fef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.369+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/4185683751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643e28fcae0 0x5643e28fef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.369+0000 7fbe4bfff640 1 -- 172.21.15.29:0/4185683751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28f6f80 msgr2=0x5643e28f73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.369+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/4185683751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28f6f80 0x5643e28f73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.369+0000 7fbe4bfff640 1 -- 172.21.15.29:0/4185683751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5643e2813f20 con 0x5643e28f7970 2024-03-11T19:09:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.369+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/4185683751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 0x5643e28fc470 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbe380a25e0 tx=0x7fbe3809a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f7c4ad77445f5c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.369+0000 7fbe36ffd640 1 -- 172.21.15.29:0/4185683751 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3809a370 con 0x5643e28f7970 2024-03-11T19:09:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.369+0000 7fbe36ffd640 1 -- 172.21.15.29:0/4185683751 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbe3809e4c0 con 0x5643e28f7970 2024-03-11T19:09:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.369+0000 7fbe36ffd640 1 -- 172.21.15.29:0/4185683751 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe380969e0 con 0x5643e28f7970 2024-03-11T19:09:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.370+0000 7fbe55967740 1 -- 172.21.15.29:0/4185683751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 msgr2=0x5643e28fc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.370+0000 7fbe55967740 1 --2- 172.21.15.29:0/4185683751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 0x5643e28fc470 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbe380a25e0 tx=0x7fbe3809a270 comp rx=0 tx=0).stop 2024-03-11T19:09:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.370+0000 7fbe55967740 1 -- 172.21.15.29:0/4185683751 shutdown_connections 2024-03-11T19:09:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.370+0000 7fbe55967740 1 --2- 172.21.15.29:0/4185683751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643e28fcae0 0x5643e28fef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.370+0000 7fbe55967740 1 --2- 172.21.15.29:0/4185683751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 0x5643e28fc470 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.370+0000 7fbe55967740 1 --2- 172.21.15.29:0/4185683751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28f6f80 0x5643e28f73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.370+0000 7fbe55967740 1 -- 172.21.15.29:0/4185683751 >> 172.21.15.29:0/4185683751 conn(0x5643e28f24f0 msgr2=0x5643e28f28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.370+0000 7fbe55967740 1 -- 172.21.15.29:0/4185683751 shutdown_connections 2024-03-11T19:09:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.370+0000 7fbe55967740 1 -- 172.21.15.29:0/4185683751 wait complete. 2024-03-11T19:09:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.370+0000 7fbe55967740 1 -- start start 2024-03-11T19:09:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.370+0000 7fbe55967740 1 Processor -- start 2024-03-11T19:09:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.370+0000 7fbe55967740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643e28f6f80 0x5643e2901070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.370+0000 7fbe55967740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28f7970 0x5643e29015b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe55967740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28fcae0 0x5643e29085f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe55967740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e285ec30 con 0x5643e28fcae0 2024-03-11T19:09:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe55967740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e2803d30 con 0x5643e28f7970 2024-03-11T19:09:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe55967740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e2532310 con 0x5643e28f6f80 2024-03-11T19:09:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe55967740 10 client.0 _ll_register_callbacks cb 0x5643e280bd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe51252640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28fcae0 0x5643e29085f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe51252640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28fcae0 0x5643e29085f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55546/0 (socket says 172.21.15.29:55546) 2024-03-11T19:09:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe51252640 1 -- 172.21.15.29:0/2177664967 learned_addr learned my addr 172.21.15.29:0/2177664967 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe50a51640 1 --2- 172.21.15.29:0/2177664967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643e28f6f80 0x5643e2901070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/2177664967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28f7970 0x5643e29015b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe51252640 1 -- 172.21.15.29:0/2177664967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643e28f6f80 msgr2=0x5643e2901070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe51252640 1 --2- 172.21.15.29:0/2177664967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643e28f6f80 0x5643e2901070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe51252640 1 -- 172.21.15.29:0/2177664967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28f7970 msgr2=0x5643e29015b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe51252640 1 --2- 172.21.15.29:0/2177664967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28f7970 0x5643e29015b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe51252640 1 -- 172.21.15.29:0/2177664967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe4c19f730 con 0x5643e28fcae0 2024-03-11T19:09:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe51252640 1 --2- 172.21.15.29:0/2177664967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28fcae0 0x5643e29085f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009ca0 tx=0x7fbe44034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73716ebb6a75b35c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe2bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe2bfff640 1 -- 172.21.15.29:0/2177664967 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe440328d0 con 0x5643e28fcae0 2024-03-11T19:09:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe2bfff640 1 -- 172.21.15.29:0/2177664967 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fbe44032a30 con 0x5643e28fcae0 2024-03-11T19:09:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe55967740 1 -- 172.21.15.29:0/2177664967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5643e255e6c0 con 0x5643e28fcae0 2024-03-11T19:09:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe2bfff640 1 -- 172.21.15.29:0/2177664967 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe44034a60 con 0x5643e28fcae0 2024-03-11T19:09:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe55967740 10 client.15042 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.371+0000 7fbe55967740 1 -- 172.21.15.29:0/2177664967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5643e285d2b0 con 0x5643e28fcae0 2024-03-11T19:09:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.372+0000 7fbe2bfff640 1 -- 172.21.15.29:0/2177664967 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5643e255e6c0 con 0x5643e28fcae0 2024-03-11T19:09:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.372+0000 7fbe2bfff640 1 -- 172.21.15.29:0/2177664967 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbe440041e0 con 0x5643e28fcae0 2024-03-11T19:09:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.372+0000 7fbe55967740 10 client.15042 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.372+0000 7fbe55967740 10 client.15042 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-11T19:09:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.372+0000 7fbe55967740 20 client.15042 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.372+0000 7fbe55967740 1 --2- 172.21.15.29:0/2177664967 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5643e290dae0 0x5643e290cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.372+0000 7fbe55967740 4 client.15042 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:09:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.372+0000 7fbe55967740 1 -- 172.21.15.29:0/2177664967 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5643e25af270 con 0x5643e290dae0 2024-03-11T19:09:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.372+0000 7fbe50a51640 1 --2- 172.21.15.29:0/2177664967 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5643e290dae0 0x5643e290cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.372+0000 7fbe50a51640 1 --2- 172.21.15.29:0/2177664967 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5643e290dae0 0x5643e290cd10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=49697545af7fc87d server_cookie=1a064e36bfd91d3f in_seq=0 out_seq=0 2024-03-11T19:09:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.372+0000 7fbe2bfff640 10 client.15042 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:09:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.373+0000 7fbe2bfff640 1 -- 172.21.15.29:0/2177664967 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5643e25af270 con 0x5643e290dae0 2024-03-11T19:09:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.373+0000 7fbe2bfff640 10 client.15042 handle_command_reply: tid=0 2024-03-11T19:09:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.373+0000 7fbe55967740 1 client.15042 shutdown 2024-03-11T19:09:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.373+0000 7fbe55967740 1 -- 172.21.15.29:0/2177664967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28fcae0 msgr2=0x5643e29085f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.373+0000 7fbe55967740 1 --2- 172.21.15.29:0/2177664967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28fcae0 0x5643e29085f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009ca0 tx=0x7fbe44034670 comp rx=0 tx=0).stop 2024-03-11T19:09:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.373+0000 7fbe55967740 1 -- 172.21.15.29:0/2177664967 shutdown_connections 2024-03-11T19:09:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.373+0000 7fbe55967740 1 --2- 172.21.15.29:0/2177664967 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5643e290dae0 0x5643e290cd10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.373+0000 7fbe55967740 1 --2- 172.21.15.29:0/2177664967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28fcae0 0x5643e29085f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.373+0000 7fbe55967740 1 --2- 172.21.15.29:0/2177664967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28f7970 0x5643e29015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.373+0000 7fbe55967740 1 --2- 172.21.15.29:0/2177664967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643e28f6f80 0x5643e2901070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.373+0000 7fbe2bfff640 0 client.15042 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:09:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.373+0000 7fbe55967740 1 -- 172.21.15.29:0/2177664967 >> 172.21.15.29:0/2177664967 conn(0x5643e28f24f0 msgr2=0x5643e28f17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.373+0000 7fbe55967740 1 -- 172.21.15.29:0/2177664967 shutdown_connections 2024-03-11T19:09:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.373+0000 7fbe55967740 1 -- 172.21.15.29:0/2177664967 wait complete. 2024-03-11T19:09:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.374+0000 7fbe55967740 20 client.15042 trim_cache size 0 max 16384 2024-03-11T19:09:22.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.389+0000 7fbe55967740 1 Processor -- start 2024-03-11T19:09:22.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.389+0000 7fbe55967740 1 -- start start 2024-03-11T19:09:22.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.389+0000 7fbe55967740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643e28f6f80 0x5643e28ffce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.389+0000 7fbe55967740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 0x5643e29085b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.389+0000 7fbe55967740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28fcae0 0x5643e28f8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.389+0000 7fbe55967740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e27e4490 con 0x5643e28f7970 2024-03-11T19:09:22.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.389+0000 7fbe55967740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e29094a0 con 0x5643e28fcae0 2024-03-11T19:09:22.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.389+0000 7fbe55967740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e2901c80 con 0x5643e28f6f80 2024-03-11T19:09:22.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.389+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 0x5643e29085b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.389+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 0x5643e29085b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55556/0 (socket says 172.21.15.29:55556) 2024-03-11T19:09:22.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.389+0000 7fbe4bfff640 1 -- 172.21.15.29:0/1479168638 learned_addr learned my addr 172.21.15.29:0/1479168638 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.390+0000 7fbe4bfff640 1 -- 172.21.15.29:0/1479168638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643e28f6f80 msgr2=0x5643e28ffce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.390+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/1479168638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643e28f6f80 0x5643e28ffce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.390+0000 7fbe4bfff640 1 -- 172.21.15.29:0/1479168638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28fcae0 msgr2=0x5643e28f8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.390+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/1479168638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28fcae0 0x5643e28f8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.390+0000 7fbe4bfff640 1 -- 172.21.15.29:0/1479168638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5643e27f9a20 con 0x5643e28f7970 2024-03-11T19:09:22.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.390+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/1479168638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 0x5643e29085b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fbe38096050 tx=0x7fbe38096720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f56673995cc8c4ba server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.390+0000 7fbe367fc640 1 -- 172.21.15.29:0/1479168638 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3809a370 con 0x5643e28f7970 2024-03-11T19:09:22.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.390+0000 7fbe367fc640 1 -- 172.21.15.29:0/1479168638 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbe380b5070 con 0x5643e28f7970 2024-03-11T19:09:22.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.390+0000 7fbe367fc640 1 -- 172.21.15.29:0/1479168638 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe380969e0 con 0x5643e28f7970 2024-03-11T19:09:22.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.390+0000 7fbe55967740 1 -- 172.21.15.29:0/1479168638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 msgr2=0x5643e29085b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.390+0000 7fbe55967740 1 --2- 172.21.15.29:0/1479168638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 0x5643e29085b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fbe38096050 tx=0x7fbe38096720 comp rx=0 tx=0).stop 2024-03-11T19:09:22.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.390+0000 7fbe55967740 1 -- 172.21.15.29:0/1479168638 shutdown_connections 2024-03-11T19:09:22.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.390+0000 7fbe55967740 1 --2- 172.21.15.29:0/1479168638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28fcae0 0x5643e28f8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.390+0000 7fbe55967740 1 --2- 172.21.15.29:0/1479168638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 0x5643e29085b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.390+0000 7fbe55967740 1 --2- 172.21.15.29:0/1479168638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643e28f6f80 0x5643e28ffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.390+0000 7fbe55967740 1 -- 172.21.15.29:0/1479168638 >> 172.21.15.29:0/1479168638 conn(0x5643e28f24f0 msgr2=0x5643e28f28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.390+0000 7fbe55967740 1 -- 172.21.15.29:0/1479168638 shutdown_connections 2024-03-11T19:09:22.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.390+0000 7fbe55967740 1 -- 172.21.15.29:0/1479168638 wait complete. 2024-03-11T19:09:22.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.391+0000 7fbe55967740 1 -- start start 2024-03-11T19:09:22.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.391+0000 7fbe55967740 1 Processor -- start 2024-03-11T19:09:22.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.391+0000 7fbe55967740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28f6f80 0x5643e2905160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.391+0000 7fbe55967740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 0x5643e29056a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.391+0000 7fbe55967740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643e28fcae0 0x5643e29636e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.391+0000 7fbe55967740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e29086c0 con 0x5643e28f7970 2024-03-11T19:09:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.391+0000 7fbe55967740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e2908830 con 0x5643e28f6f80 2024-03-11T19:09:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.391+0000 7fbe55967740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643e2963c20 con 0x5643e28fcae0 2024-03-11T19:09:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.391+0000 7fbe55967740 10 client.0 _ll_register_callbacks cb 0x7fbe40005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.391+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 0x5643e29056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.391+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 0x5643e29056a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55566/0 (socket says 172.21.15.29:55566) 2024-03-11T19:09:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.391+0000 7fbe4bfff640 1 -- 172.21.15.29:0/2115528940 learned_addr learned my addr 172.21.15.29:0/2115528940 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.391+0000 7fbe50a51640 1 --2- 172.21.15.29:0/2115528940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28f6f80 0x5643e2905160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.391+0000 7fbe4bfff640 1 -- 172.21.15.29:0/2115528940 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643e28fcae0 msgr2=0x5643e29636e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.391+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/2115528940 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643e28fcae0 0x5643e29636e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.391+0000 7fbe4bfff640 1 -- 172.21.15.29:0/2115528940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28f6f80 msgr2=0x5643e2905160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.391+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/2115528940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28f6f80 0x5643e2905160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.391+0000 7fbe4bfff640 1 -- 172.21.15.29:0/2115528940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5643e2813f20 con 0x5643e28f7970 2024-03-11T19:09:22.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.391+0000 7fbe50a51640 1 --2- 172.21.15.29:0/2115528940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28f6f80 0x5643e2905160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:22.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.392+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/2115528940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 0x5643e29056a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fbe3809a070 tx=0x7fbe380047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38a40de652c054b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.392+0000 7fbe2bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:22.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.392+0000 7fbe2bfff640 1 -- 172.21.15.29:0/2115528940 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3809a370 con 0x5643e28f7970 2024-03-11T19:09:22.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.392+0000 7fbe2bfff640 1 -- 172.21.15.29:0/2115528940 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fbe38098670 con 0x5643e28f7970 2024-03-11T19:09:22.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.392+0000 7fbe55967740 1 -- 172.21.15.29:0/2115528940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5643e290e4b0 con 0x5643e28f7970 2024-03-11T19:09:22.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.392+0000 7fbe2bfff640 1 -- 172.21.15.29:0/2115528940 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe3809e170 con 0x5643e28f7970 2024-03-11T19:09:22.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.392+0000 7fbe55967740 10 client.15048 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:22.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.392+0000 7fbe55967740 1 -- 172.21.15.29:0/2115528940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5643e281f780 con 0x5643e28f7970 2024-03-11T19:09:22.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.392+0000 7fbe2bfff640 1 -- 172.21.15.29:0/2115528940 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5643e290e4b0 con 0x5643e28f7970 2024-03-11T19:09:22.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.392+0000 7fbe2bfff640 1 -- 172.21.15.29:0/2115528940 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbe38020070 con 0x5643e28f7970 2024-03-11T19:09:22.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.392+0000 7fbe55967740 10 client.15048 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:22.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.392+0000 7fbe55967740 10 client.15048 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-11T19:09:22.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.392+0000 7fbe55967740 20 client.15048 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:22.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.392+0000 7fbe55967740 1 --2- 172.21.15.29:0/2115528940 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5643e29689a0 0x5643e2964e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.392+0000 7fbe55967740 4 client.15048 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:09:22.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.392+0000 7fbe55967740 1 -- 172.21.15.29:0/2115528940 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fbe3809e4c0 con 0x5643e29689a0 2024-03-11T19:09:22.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.392+0000 7fbe50a51640 1 --2- 172.21.15.29:0/2115528940 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5643e29689a0 0x5643e2964e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.393+0000 7fbe50a51640 1 --2- 172.21.15.29:0/2115528940 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5643e29689a0 0x5643e2964e60 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=32c563782ef5e010 server_cookie=193c29dd14d08c6b in_seq=0 out_seq=0 2024-03-11T19:09:22.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.393+0000 7fbe2bfff640 10 client.15048 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:09:22.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.393+0000 7fbe2bfff640 1 -- 172.21.15.29:0/2115528940 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7fbe3809e4c0 con 0x5643e29689a0 2024-03-11T19:09:22.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.393+0000 7fbe2bfff640 10 client.15048 handle_command_reply: tid=0 2024-03-11T19:09:22.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.393+0000 7fbe55967740 1 client.15048 shutdown 2024-03-11T19:09:22.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.393+0000 7fbe55967740 1 -- 172.21.15.29:0/2115528940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 msgr2=0x5643e29056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.393+0000 7fbe55967740 1 --2- 172.21.15.29:0/2115528940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 0x5643e29056a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fbe3809a070 tx=0x7fbe380047c0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.393+0000 7fbe55967740 1 -- 172.21.15.29:0/2115528940 shutdown_connections 2024-03-11T19:09:22.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.393+0000 7fbe55967740 1 --2- 172.21.15.29:0/2115528940 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5643e29689a0 0x5643e2964e60 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.393+0000 7fbe55967740 1 --2- 172.21.15.29:0/2115528940 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643e28fcae0 0x5643e29636e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.393+0000 7fbe55967740 1 --2- 172.21.15.29:0/2115528940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643e28f7970 0x5643e29056a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.393+0000 7fbe55967740 1 --2- 172.21.15.29:0/2115528940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643e28f6f80 0x5643e2905160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.393+0000 7fbe2bfff640 0 client.15048 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:09:22.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.393+0000 7fbe55967740 1 -- 172.21.15.29:0/2115528940 >> 172.21.15.29:0/2115528940 conn(0x5643e28f24f0 msgr2=0x5643e28fbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.393+0000 7fbe55967740 1 -- 172.21.15.29:0/2115528940 shutdown_connections 2024-03-11T19:09:22.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.393+0000 7fbe55967740 1 -- 172.21.15.29:0/2115528940 wait complete. 2024-03-11T19:09:22.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.393+0000 7fbe55967740 20 client.15048 trim_cache size 0 max 16384 2024-03-11T19:09:22.405 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:09:22.405 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "638915c2-dfd9-11ee-95c8-87774f69a715", 2024-03-11T19:09:22.405 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 4, 2024-03-11T19:09:22.405 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24463, 2024-03-11T19:09:22.405 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-03-11T19:09:22.405 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-03-11T19:09:22.405 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-03-11T19:09:22.406 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 86.613927641999993, 2024-03-11T19:09:22.406 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 26, 2024-03-11T19:09:22.406 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 89, 2024-03-11T19:09:22.406 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 87, 2024-03-11T19:09:22.406 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 113.998111768 2024-03-11T19:09:22.406 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:09:22.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.394+0000 7fbe52cb5640 1 -- 172.21.15.29:0/1562544078 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbe3c0a1a30 msgr2=0x7fbe3c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.394+0000 7fbe52cb5640 1 --2- 172.21.15.29:0/1562544078 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbe3c0a1a30 0x7fbe3c0a3ef0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009bb0 tx=0x7fbe40005f90 comp rx=0 tx=0).stop 2024-03-11T19:09:22.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.394+0000 7fbe52cb5640 1 -- 172.21.15.29:0/1562544078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe4c104550 msgr2=0x7fbe4c197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.394+0000 7fbe52cb5640 1 --2- 172.21.15.29:0/1562544078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe4c104550 0x7fbe4c197660 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbe3800c970 tx=0x7fbe3800ce30 comp rx=0 tx=0).stop 2024-03-11T19:09:22.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.394+0000 7fbe52cb5640 1 -- 172.21.15.29:0/1562544078 shutdown_connections 2024-03-11T19:09:22.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.394+0000 7fbe52cb5640 1 --2- 172.21.15.29:0/1562544078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe4c106e80 0x7fbe4c19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.394+0000 7fbe52cb5640 1 --2- 172.21.15.29:0/1562544078 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbe3c0a1a30 0x7fbe3c0a3ef0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.394+0000 7fbe52cb5640 1 --2- 172.21.15.29:0/1562544078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe4c104550 0x7fbe4c197660 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.394+0000 7fbe52cb5640 1 --2- 172.21.15.29:0/1562544078 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe4c101c20 0x7fbe4c197120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.394+0000 7fbe52cb5640 1 -- 172.21.15.29:0/1562544078 >> 172.21.15.29:0/1562544078 conn(0x7fbe4c0fb630 msgr2=0x7fbe4c102580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.394+0000 7fbe52cb5640 1 -- 172.21.15.29:0/1562544078 shutdown_connections 2024-03-11T19:09:22.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.394+0000 7fbe52cb5640 1 -- 172.21.15.29:0/1562544078 wait complete. 2024-03-11T19:09:22.409 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-11T19:09:22.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:22 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3289698521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:09:22.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:22 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3059406438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:09:22.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:22 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1673842143' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:22.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.509+0000 7f8a3af8a640 1 -- 172.21.15.29:0/2377803148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a34102d60 msgr2=0x7f8a341095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.509+0000 7f8a3af8a640 1 --2- 172.21.15.29:0/2377803148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a34102d60 0x7f8a341095f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c009ca0 tx=0x7f8a2c02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.509+0000 7f8a3af8a640 1 -- 172.21.15.29:0/2377803148 shutdown_connections 2024-03-11T19:09:22.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.509+0000 7f8a3af8a640 1 --2- 172.21.15.29:0/2377803148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a34102d60 0x7f8a341095f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.509+0000 7f8a3af8a640 1 --2- 172.21.15.29:0/2377803148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a341023a0 0x7f8a34102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.509+0000 7f8a3af8a640 1 --2- 172.21.15.29:0/2377803148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a341011b0 0x7f8a341015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.509+0000 7f8a3af8a640 1 -- 172.21.15.29:0/2377803148 >> 172.21.15.29:0/2377803148 conn(0x7f8a340fc720 msgr2=0x7f8a340feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.509+0000 7f8a3af8a640 1 -- 172.21.15.29:0/2377803148 shutdown_connections 2024-03-11T19:09:22.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.509+0000 7f8a3af8a640 1 -- 172.21.15.29:0/2377803148 wait complete. 2024-03-11T19:09:22.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a3af8a640 1 Processor -- start 2024-03-11T19:09:22.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a3af8a640 1 -- start start 2024-03-11T19:09:22.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a3af8a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a341011b0 0x7f8a34197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a3af8a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a341023a0 0x7f8a341976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a3af8a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a34102d60 0x7f8a3419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a3af8a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3419ec20 con 0x7f8a341011b0 2024-03-11T19:09:22.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a3af8a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3419ed90 con 0x7f8a34102d60 2024-03-11T19:09:22.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a3af8a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3419ef50 con 0x7f8a341023a0 2024-03-11T19:09:22.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a38d26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a341011b0 0x7f8a34197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a38d26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a341011b0 0x7f8a34197180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55584/0 (socket says 172.21.15.29:55584) 2024-03-11T19:09:22.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a38d26640 1 -- 172.21.15.29:0/3064082168 learned_addr learned my addr 172.21.15.29:0/3064082168 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a33fff640 1 --2- 172.21.15.29:0/3064082168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a341023a0 0x7f8a341976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a39527640 1 --2- 172.21.15.29:0/3064082168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a34102d60 0x7f8a3419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a38d26640 1 -- 172.21.15.29:0/3064082168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a341023a0 msgr2=0x7f8a341976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a38d26640 1 --2- 172.21.15.29:0/3064082168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a341023a0 0x7f8a341976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a38d26640 1 -- 172.21.15.29:0/3064082168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a34102d60 msgr2=0x7f8a3419e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a38d26640 1 --2- 172.21.15.29:0/3064082168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a34102d60 0x7f8a3419e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a38d26640 1 -- 172.21.15.29:0/3064082168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a2c009900 con 0x7f8a341011b0 2024-03-11T19:09:22.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a39527640 1 --2- 172.21.15.29:0/3064082168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a34102d60 0x7f8a3419e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:22.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.510+0000 7f8a38d26640 1 --2- 172.21.15.29:0/3064082168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a341011b0 0x7f8a34197180 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800d950 tx=0x7f8a2800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.511+0000 7f8a31ffb640 1 -- 172.21.15.29:0/3064082168 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a28009c30 con 0x7f8a341011b0 2024-03-11T19:09:22.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.511+0000 7f8a3af8a640 1 -- 172.21.15.29:0/3064082168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a3419f230 con 0x7f8a341011b0 2024-03-11T19:09:22.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.511+0000 7f8a31ffb640 1 -- 172.21.15.29:0/3064082168 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8a28009d90 con 0x7f8a341011b0 2024-03-11T19:09:22.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.511+0000 7f8a31ffb640 1 -- 172.21.15.29:0/3064082168 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2800f720 con 0x7f8a341011b0 2024-03-11T19:09:22.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.511+0000 7f8a3af8a640 1 -- 172.21.15.29:0/3064082168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a3419f790 con 0x7f8a341011b0 2024-03-11T19:09:22.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.513+0000 7f8a31ffb640 1 -- 172.21.15.29:0/3064082168 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8a28010460 con 0x7f8a341011b0 2024-03-11T19:09:22.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.513+0000 7f8a3dc37740 1 Processor -- start 2024-03-11T19:09:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.513+0000 7f8a3dc37740 1 -- start start 2024-03-11T19:09:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a31ffb640 1 --2- 172.21.15.29:0/3064082168 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8a240a1a30 0x7f8a240a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a3dc37740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 0x56356a8c03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a3dc37740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c0970 0x56356a8c5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a3dc37740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c5ae0 0x56356a8c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a3dc37740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356a87bcd0 con 0x56356a8c0970 2024-03-11T19:09:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a3dc37740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356a82ef20 con 0x56356a8bff80 2024-03-11T19:09:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a3dc37740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356a82c450 con 0x56356a8c5ae0 2024-03-11T19:09:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a33fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c0970 0x56356a8c5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a39527640 1 --2- 172.21.15.29:0/3064082168 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8a240a1a30 0x7f8a240a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a33fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c0970 0x56356a8c5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55592/0 (socket says 172.21.15.29:55592) 2024-03-11T19:09:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a33fff640 1 -- 172.21.15.29:0/1313356475 learned_addr learned my addr 172.21.15.29:0/1313356475 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a31ffb640 1 -- 172.21.15.29:0/3064082168 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f8a280cd600 con 0x7f8a341011b0 2024-03-11T19:09:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a39527640 1 --2- 172.21.15.29:0/1313356475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c5ae0 0x56356a8c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a38d26640 1 --2- 172.21.15.29:0/1313356475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 0x56356a8c03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a33fff640 1 -- 172.21.15.29:0/1313356475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c5ae0 msgr2=0x56356a8c7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a33fff640 1 --2- 172.21.15.29:0/1313356475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c5ae0 0x56356a8c7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a33fff640 1 -- 172.21.15.29:0/1313356475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 msgr2=0x56356a8c03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a33fff640 1 --2- 172.21.15.29:0/1313356475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 0x56356a8c03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a33fff640 1 -- 172.21.15.29:0/1313356475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56356a7dcf20 con 0x56356a8c0970 2024-03-11T19:09:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a39527640 1 --2- 172.21.15.29:0/3064082168 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8a240a1a30 0x7f8a240a3ef0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c037040 tx=0x7f8a2c005d80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a33fff640 1 --2- 172.21.15.29:0/1313356475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c0970 0x56356a8c5470 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8a20009c20 tx=0x7f8a20011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59da1ed596d58cfa server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a1effd640 1 -- 172.21.15.29:0/1313356475 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a20004230 con 0x56356a8c0970 2024-03-11T19:09:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a1effd640 1 -- 172.21.15.29:0/1313356475 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8a20009e90 con 0x56356a8c0970 2024-03-11T19:09:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.514+0000 7f8a1effd640 1 -- 172.21.15.29:0/1313356475 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a20004ca0 con 0x56356a8c0970 2024-03-11T19:09:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.515+0000 7f8a3dc37740 1 -- 172.21.15.29:0/1313356475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c0970 msgr2=0x56356a8c5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.515+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/1313356475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c0970 0x56356a8c5470 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8a20009c20 tx=0x7f8a20011810 comp rx=0 tx=0).stop 2024-03-11T19:09:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.515+0000 7f8a3dc37740 1 -- 172.21.15.29:0/1313356475 shutdown_connections 2024-03-11T19:09:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.515+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/1313356475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c5ae0 0x56356a8c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.515+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/1313356475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c0970 0x56356a8c5470 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.515+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/1313356475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 0x56356a8c03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.515+0000 7f8a3dc37740 1 -- 172.21.15.29:0/1313356475 >> 172.21.15.29:0/1313356475 conn(0x56356a8bb4f0 msgr2=0x56356a8bb8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.515+0000 7f8a3dc37740 1 -- 172.21.15.29:0/1313356475 shutdown_connections 2024-03-11T19:09:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.515+0000 7f8a3dc37740 1 -- 172.21.15.29:0/1313356475 wait complete. 2024-03-11T19:09:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.515+0000 7f8a3dc37740 1 -- start start 2024-03-11T19:09:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.515+0000 7f8a3dc37740 1 Processor -- start 2024-03-11T19:09:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.516+0000 7f8a3dc37740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 0x56356a8ca190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.516+0000 7f8a3dc37740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c0970 0x56356a8ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.516+0000 7f8a3dc37740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 0x56356a8d1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.516+0000 7f8a3dc37740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356a827c30 con 0x56356a8c5ae0 2024-03-11T19:09:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.516+0000 7f8a3dc37740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356a7ccd30 con 0x56356a8bff80 2024-03-11T19:09:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.516+0000 7f8a3dc37740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356a4fb310 con 0x56356a8c0970 2024-03-11T19:09:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.516+0000 7f8a3dc37740 10 client.0 _ll_register_callbacks cb 0x56356a7d4d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.516+0000 7f8a39527640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 0x56356a8d1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.516+0000 7f8a39527640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 0x56356a8d1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55594/0 (socket says 172.21.15.29:55594) 2024-03-11T19:09:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.516+0000 7f8a39527640 1 -- 172.21.15.29:0/2443083836 learned_addr learned my addr 172.21.15.29:0/2443083836 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.516+0000 7f8a38d26640 1 --2- 172.21.15.29:0/2443083836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 0x56356a8ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.516+0000 7f8a33fff640 1 --2- 172.21.15.29:0/2443083836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c0970 0x56356a8ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.516+0000 7f8a39527640 1 -- 172.21.15.29:0/2443083836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c0970 msgr2=0x56356a8ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.516+0000 7f8a39527640 1 --2- 172.21.15.29:0/2443083836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c0970 0x56356a8ca6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.516+0000 7f8a39527640 1 -- 172.21.15.29:0/2443083836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 msgr2=0x56356a8ca190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.516+0000 7f8a39527640 1 --2- 172.21.15.29:0/2443083836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 0x56356a8ca190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.516+0000 7f8a39527640 1 -- 172.21.15.29:0/2443083836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a20009850 con 0x56356a8c5ae0 2024-03-11T19:09:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.516+0000 7f8a39527640 1 --2- 172.21.15.29:0/2443083836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 0x56356a8d1710 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c0346c0 tx=0x7f8a2c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68fa670b7b1e5fc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.516+0000 7f8a13fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.517+0000 7f8a13fff640 1 -- 172.21.15.29:0/2443083836 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c034e70 con 0x56356a8c5ae0 2024-03-11T19:09:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.517+0000 7f8a13fff640 1 -- 172.21.15.29:0/2443083836 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f8a2c003970 con 0x56356a8c5ae0 2024-03-11T19:09:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.517+0000 7f8a3dc37740 1 -- 172.21.15.29:0/2443083836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56356a5276c0 con 0x56356a8c5ae0 2024-03-11T19:09:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.517+0000 7f8a13fff640 1 -- 172.21.15.29:0/2443083836 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a2c003b30 con 0x56356a8c5ae0 2024-03-11T19:09:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.517+0000 7f8a3dc37740 10 client.15060 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.517+0000 7f8a3dc37740 1 -- 172.21.15.29:0/2443083836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56356a8262b0 con 0x56356a8c5ae0 2024-03-11T19:09:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.517+0000 7f8a13fff640 1 -- 172.21.15.29:0/2443083836 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x56356a5276c0 con 0x56356a8c5ae0 2024-03-11T19:09:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.517+0000 7f8a13fff640 1 -- 172.21.15.29:0/2443083836 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8a2c04c480 con 0x56356a8c5ae0 2024-03-11T19:09:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.517+0000 7f8a3dc37740 10 client.15060 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.517+0000 7f8a3dc37740 10 client.15060 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-11T19:09:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.517+0000 7f8a3dc37740 20 client.15060 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.517+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/2443083836 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x56356a8d6b40 0x56356a8d5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.517+0000 7f8a3dc37740 4 client.15060 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:09:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.517+0000 7f8a3dc37740 1 -- 172.21.15.29:0/2443083836 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56356a578270 con 0x56356a8d6b40 2024-03-11T19:09:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.517+0000 7f8a38d26640 1 --2- 172.21.15.29:0/2443083836 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x56356a8d6b40 0x56356a8d5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.518+0000 7f8a38d26640 1 --2- 172.21.15.29:0/2443083836 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x56356a8d6b40 0x56356a8d5d90 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=86eda46c1bd7212a server_cookie=82adea9145505280 in_seq=0 out_seq=0 2024-03-11T19:09:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.518+0000 7f8a13fff640 10 client.15060 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:09:22.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.519+0000 7f8a13fff640 1 -- 172.21.15.29:0/2443083836 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56356a578270 con 0x56356a8d6b40 2024-03-11T19:09:22.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.519+0000 7f8a13fff640 10 client.15060 handle_command_reply: tid=0 2024-03-11T19:09:22.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.519+0000 7f8a3dc37740 1 client.15060 shutdown 2024-03-11T19:09:22.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.519+0000 7f8a3dc37740 1 -- 172.21.15.29:0/2443083836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 msgr2=0x56356a8d1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.519+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/2443083836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 0x56356a8d1710 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c0346c0 tx=0x7f8a2c0095c0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.519+0000 7f8a3dc37740 1 -- 172.21.15.29:0/2443083836 shutdown_connections 2024-03-11T19:09:22.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.519+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/2443083836 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x56356a8d6b40 0x56356a8d5d90 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.519+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/2443083836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 0x56356a8d1710 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.519+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/2443083836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c0970 0x56356a8ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.519+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/2443083836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 0x56356a8ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.519+0000 7f8a13fff640 0 client.15060 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:09:22.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.519+0000 7f8a3dc37740 1 -- 172.21.15.29:0/2443083836 >> 172.21.15.29:0/2443083836 conn(0x56356a8bb4f0 msgr2=0x56356a8ba7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.519+0000 7f8a3dc37740 1 -- 172.21.15.29:0/2443083836 shutdown_connections 2024-03-11T19:09:22.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.519+0000 7f8a3dc37740 1 -- 172.21.15.29:0/2443083836 wait complete. 2024-03-11T19:09:22.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.519+0000 7f8a3dc37740 20 client.15060 trim_cache size 0 max 16384 2024-03-11T19:09:22.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.534+0000 7f8a3dc37740 1 Processor -- start 2024-03-11T19:09:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a3dc37740 1 -- start start 2024-03-11T19:09:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a3dc37740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 0x56356a8c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a3dc37740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c0970 0x56356a8d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a3dc37740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 0x56356a8bb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a3dc37740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356a7ad490 con 0x56356a8c5ae0 2024-03-11T19:09:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a3dc37740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356a5276c0 con 0x56356a8bff80 2024-03-11T19:09:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a3dc37740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356a8918c0 con 0x56356a8c0970 2024-03-11T19:09:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a39527640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 0x56356a8bb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a39527640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 0x56356a8bb8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55608/0 (socket says 172.21.15.29:55608) 2024-03-11T19:09:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a39527640 1 -- 172.21.15.29:0/1537980301 learned_addr learned my addr 172.21.15.29:0/1537980301 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a38d26640 1 --2- 172.21.15.29:0/1537980301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 0x56356a8c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a33fff640 1 --2- 172.21.15.29:0/1537980301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c0970 0x56356a8d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a39527640 1 -- 172.21.15.29:0/1537980301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c0970 msgr2=0x56356a8d16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a39527640 1 --2- 172.21.15.29:0/1537980301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c0970 0x56356a8d16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a39527640 1 -- 172.21.15.29:0/1537980301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 msgr2=0x56356a8c7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a39527640 1 --2- 172.21.15.29:0/1537980301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 0x56356a8c7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a39527640 1 -- 172.21.15.29:0/1537980301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56356a7c2a20 con 0x56356a8c5ae0 2024-03-11T19:09:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a38d26640 1 --2- 172.21.15.29:0/1537980301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 0x56356a8c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a33fff640 1 --2- 172.21.15.29:0/1537980301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c0970 0x56356a8d16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a39527640 1 --2- 172.21.15.29:0/1537980301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 0x56356a8bb8d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c02c630 tx=0x7f8a2c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60dbd34cfcad26db server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a1e7fc640 1 -- 172.21.15.29:0/1537980301 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c04ca70 con 0x56356a8c5ae0 2024-03-11T19:09:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a1e7fc640 1 -- 172.21.15.29:0/1537980301 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8a2c003e20 con 0x56356a8c5ae0 2024-03-11T19:09:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.535+0000 7f8a1e7fc640 1 -- 172.21.15.29:0/1537980301 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c0043a0 con 0x56356a8c5ae0 2024-03-11T19:09:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.536+0000 7f8a3dc37740 1 -- 172.21.15.29:0/1537980301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 msgr2=0x56356a8bb8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.536+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/1537980301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 0x56356a8bb8d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c02c630 tx=0x7f8a2c009b20 comp rx=0 tx=0).stop 2024-03-11T19:09:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.536+0000 7f8a3dc37740 1 -- 172.21.15.29:0/1537980301 shutdown_connections 2024-03-11T19:09:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.536+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/1537980301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 0x56356a8bb8d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.536+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/1537980301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c0970 0x56356a8d16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.536+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/1537980301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 0x56356a8c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.536+0000 7f8a3dc37740 1 -- 172.21.15.29:0/1537980301 >> 172.21.15.29:0/1537980301 conn(0x56356a8bb4f0 msgr2=0x56356a8c1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.536+0000 7f8a3dc37740 1 -- 172.21.15.29:0/1537980301 shutdown_connections 2024-03-11T19:09:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.536+0000 7f8a3dc37740 1 -- 172.21.15.29:0/1537980301 wait complete. 2024-03-11T19:09:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.536+0000 7f8a3dc37740 1 -- start start 2024-03-11T19:09:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.536+0000 7f8a3dc37740 1 Processor -- start 2024-03-11T19:09:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a3dc37740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 0x56356a8ce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a3dc37740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c0970 0x56356a8ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a3dc37740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 0x56356a92c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a3dc37740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356a8d1780 con 0x56356a8c5ae0 2024-03-11T19:09:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a3dc37740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356a8d18f0 con 0x56356a8bff80 2024-03-11T19:09:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a3dc37740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356a8d1ab0 con 0x56356a8c0970 2024-03-11T19:09:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a3dc37740 10 client.0 _ll_register_callbacks cb 0x7f8a20009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a39527640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 0x56356a92c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a39527640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 0x56356a92c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55614/0 (socket says 172.21.15.29:55614) 2024-03-11T19:09:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a39527640 1 -- 172.21.15.29:0/1240133006 learned_addr learned my addr 172.21.15.29:0/1240133006 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a33fff640 1 --2- 172.21.15.29:0/1240133006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c0970 0x56356a8ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a38d26640 1 --2- 172.21.15.29:0/1240133006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 0x56356a8ce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a39527640 1 -- 172.21.15.29:0/1240133006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c0970 msgr2=0x56356a8ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a39527640 1 --2- 172.21.15.29:0/1240133006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c0970 0x56356a8ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a39527640 1 -- 172.21.15.29:0/1240133006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 msgr2=0x56356a8ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a39527640 1 --2- 172.21.15.29:0/1240133006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 0x56356a8ce2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a39527640 1 -- 172.21.15.29:0/1240133006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56356a8262b0 con 0x56356a8c5ae0 2024-03-11T19:09:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a39527640 1 --2- 172.21.15.29:0/1240133006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 0x56356a92c740 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c032cb0 tx=0x7f8a2c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58c10f2e72949094 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a13fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a13fff640 1 -- 172.21.15.29:0/1240133006 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c0040a0 con 0x56356a8c5ae0 2024-03-11T19:09:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a13fff640 1 -- 172.21.15.29:0/1240133006 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f8a2c004200 con 0x56356a8c5ae0 2024-03-11T19:09:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a3dc37740 1 -- 172.21.15.29:0/1240133006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56356a8b6b10 con 0x56356a8c5ae0 2024-03-11T19:09:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.537+0000 7f8a13fff640 1 -- 172.21.15.29:0/1240133006 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a2c0043c0 con 0x56356a8c5ae0 2024-03-11T19:09:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.538+0000 7f8a3dc37740 10 client.15066 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.538+0000 7f8a3dc37740 1 -- 172.21.15.29:0/1240133006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56356a7e8780 con 0x56356a8c5ae0 2024-03-11T19:09:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.538+0000 7f8a13fff640 1 -- 172.21.15.29:0/1240133006 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x56356a8b6b10 con 0x56356a8c5ae0 2024-03-11T19:09:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.538+0000 7f8a13fff640 1 -- 172.21.15.29:0/1240133006 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8a2c068030 con 0x56356a8c5ae0 2024-03-11T19:09:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.538+0000 7f8a3dc37740 10 client.15066 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.538+0000 7f8a3dc37740 10 client.15066 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-11T19:09:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.538+0000 7f8a3dc37740 20 client.15066 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.538+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/1240133006 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x56356a931800 0x56356a92dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.538+0000 7f8a3dc37740 4 client.15066 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:09:22.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.538+0000 7f8a3dc37740 1 -- 172.21.15.29:0/1240133006 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56356a8d7b70 con 0x56356a931800 2024-03-11T19:09:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.538+0000 7f8a38d26640 1 --2- 172.21.15.29:0/1240133006 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x56356a931800 0x56356a92dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.538+0000 7f8a13fff640 10 client.15066 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:09:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.538+0000 7f8a38d26640 1 --2- 172.21.15.29:0/1240133006 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x56356a931800 0x56356a92dca0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8b93a5c86685e02c server_cookie=a1ed12e81f36e259 in_seq=0 out_seq=0 2024-03-11T19:09:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.539+0000 7f8a13fff640 1 -- 172.21.15.29:0/1240133006 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x56356a8d7b70 con 0x56356a931800 2024-03-11T19:09:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.539+0000 7f8a13fff640 10 client.15066 handle_command_reply: tid=0 2024-03-11T19:09:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.539+0000 7f8a3dc37740 1 client.15066 shutdown 2024-03-11T19:09:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.539+0000 7f8a3dc37740 1 -- 172.21.15.29:0/1240133006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 msgr2=0x56356a92c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.539+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/1240133006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 0x56356a92c740 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c032cb0 tx=0x7f8a2c003c20 comp rx=0 tx=0).stop 2024-03-11T19:09:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.539+0000 7f8a3dc37740 1 -- 172.21.15.29:0/1240133006 shutdown_connections 2024-03-11T19:09:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.539+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/1240133006 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x56356a931800 0x56356a92dca0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.539+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/1240133006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56356a8c5ae0 0x56356a92c740 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.539+0000 7f8a13fff640 0 client.15066 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:09:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.539+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/1240133006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56356a8c0970 0x56356a8ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.539+0000 7f8a3dc37740 1 --2- 172.21.15.29:0/1240133006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56356a8bff80 0x56356a8ce2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.539+0000 7f8a3dc37740 1 -- 172.21.15.29:0/1240133006 >> 172.21.15.29:0/1240133006 conn(0x56356a8bb4f0 msgr2=0x56356a8bb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.539+0000 7f8a3dc37740 1 -- 172.21.15.29:0/1240133006 shutdown_connections 2024-03-11T19:09:22.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.539+0000 7f8a3dc37740 1 -- 172.21.15.29:0/1240133006 wait complete. 2024-03-11T19:09:22.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.539+0000 7f8a3dc37740 20 client.15066 trim_cache size 0 max 16384 2024-03-11T19:09:22.549 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:09:22.549 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "638915c2-dfd9-11ee-95c8-87774f69a715", 2024-03-11T19:09:22.550 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 3, 2024-03-11T19:09:22.550 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24473, 2024-03-11T19:09:22.550 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-03-11T19:09:22.550 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-03-11T19:09:22.550 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-03-11T19:09:22.550 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 87.766810230000004, 2024-03-11T19:09:22.550 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 26, 2024-03-11T19:09:22.550 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 89, 2024-03-11T19:09:22.550 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 87, 2024-03-11T19:09:22.550 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 108.538493539 2024-03-11T19:09:22.550 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:09:22.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.539+0000 7f8a3af8a640 1 -- 172.21.15.29:0/3064082168 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8a240a1a30 msgr2=0x7f8a240a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.539+0000 7f8a3af8a640 1 --2- 172.21.15.29:0/3064082168 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8a240a1a30 0x7f8a240a3ef0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c037040 tx=0x7f8a2c005d80 comp rx=0 tx=0).stop 2024-03-11T19:09:22.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.539+0000 7f8a3af8a640 1 -- 172.21.15.29:0/3064082168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a341011b0 msgr2=0x7f8a34197180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.539+0000 7f8a3af8a640 1 --2- 172.21.15.29:0/3064082168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a341011b0 0x7f8a34197180 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800d950 tx=0x7f8a2800de10 comp rx=0 tx=0).stop 2024-03-11T19:09:22.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.540+0000 7f8a3af8a640 1 -- 172.21.15.29:0/3064082168 shutdown_connections 2024-03-11T19:09:22.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.540+0000 7f8a3af8a640 1 --2- 172.21.15.29:0/3064082168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a34102d60 0x7f8a3419e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.540+0000 7f8a3af8a640 1 --2- 172.21.15.29:0/3064082168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a341023a0 0x7f8a341976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.540+0000 7f8a3af8a640 1 --2- 172.21.15.29:0/3064082168 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8a240a1a30 0x7f8a240a3ef0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.540+0000 7f8a3af8a640 1 --2- 172.21.15.29:0/3064082168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a341011b0 0x7f8a34197180 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.540+0000 7f8a3af8a640 1 -- 172.21.15.29:0/3064082168 >> 172.21.15.29:0/3064082168 conn(0x7f8a340fc720 msgr2=0x7f8a341056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.540+0000 7f8a3af8a640 1 -- 172.21.15.29:0/3064082168 shutdown_connections 2024-03-11T19:09:22.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.540+0000 7f8a3af8a640 1 -- 172.21.15.29:0/3064082168 wait complete. 2024-03-11T19:09:22.553 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-11T19:09:22.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:22 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3289698521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:09:22.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:22 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3059406438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:09:22.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:22 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1673842143' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:22.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.654+0000 7f2cdae0f640 1 -- 172.21.15.29:0/3420560078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cd4101280 msgr2=0x7f2cd4101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.654+0000 7f2cdae0f640 1 --2- 172.21.15.29:0/3420560078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cd4101280 0x7f2cd4101700 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4009ca0 tx=0x7f2cc402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.654+0000 7f2cdae0f640 1 -- 172.21.15.29:0/3420560078 shutdown_connections 2024-03-11T19:09:22.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.654+0000 7f2cdae0f640 1 --2- 172.21.15.29:0/3420560078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2cd4101c40 0x7f2cd41084d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.654+0000 7f2cdae0f640 1 --2- 172.21.15.29:0/3420560078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cd4101280 0x7f2cd4101700 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.654+0000 7f2cdae0f640 1 --2- 172.21.15.29:0/3420560078 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cd4100090 0x7f2cd41004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.654+0000 7f2cdae0f640 1 -- 172.21.15.29:0/3420560078 >> 172.21.15.29:0/3420560078 conn(0x7f2cd40fb620 msgr2=0x7f2cd40fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.654+0000 7f2cdae0f640 1 -- 172.21.15.29:0/3420560078 shutdown_connections 2024-03-11T19:09:22.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.654+0000 7f2cdae0f640 1 -- 172.21.15.29:0/3420560078 wait complete. 2024-03-11T19:09:22.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.654+0000 7f2cdae0f640 1 Processor -- start 2024-03-11T19:09:22.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.654+0000 7f2cdae0f640 1 -- start start 2024-03-11T19:09:22.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.654+0000 7f2cdae0f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cd4100090 0x7f2cd4197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.654+0000 7f2cdae0f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cd4101280 0x7f2cd4197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.654+0000 7f2cdae0f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2cd4101c40 0x7f2cd419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.654+0000 7f2cdae0f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd419ec40 con 0x7f2cd4101280 2024-03-11T19:09:22.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.654+0000 7f2cdae0f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd419edb0 con 0x7f2cd4100090 2024-03-11T19:09:22.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.654+0000 7f2cdae0f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd419ef70 con 0x7f2cd4101c40 2024-03-11T19:09:22.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.654+0000 7f2cd3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cd4101280 0x7f2cd4197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.655+0000 7f2cd3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cd4101280 0x7f2cd4197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55630/0 (socket says 172.21.15.29:55630) 2024-03-11T19:09:22.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.655+0000 7f2cd3fff640 1 -- 172.21.15.29:0/3790753638 learned_addr learned my addr 172.21.15.29:0/3790753638 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.655+0000 7f2cd93ac640 1 --2- 172.21.15.29:0/3790753638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2cd4101c40 0x7f2cd419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.655+0000 7f2cd8bab640 1 --2- 172.21.15.29:0/3790753638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cd4100090 0x7f2cd4197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.655+0000 7f2cd3fff640 1 -- 172.21.15.29:0/3790753638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2cd4101c40 msgr2=0x7f2cd419e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.655+0000 7f2cd3fff640 1 --2- 172.21.15.29:0/3790753638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2cd4101c40 0x7f2cd419e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.655+0000 7f2cd3fff640 1 -- 172.21.15.29:0/3790753638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cd4100090 msgr2=0x7f2cd4197150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.655+0000 7f2cd3fff640 1 --2- 172.21.15.29:0/3790753638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cd4100090 0x7f2cd4197150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.655+0000 7f2cd3fff640 1 -- 172.21.15.29:0/3790753638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc4009900 con 0x7f2cd4101280 2024-03-11T19:09:22.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.655+0000 7f2cd8bab640 1 --2- 172.21.15.29:0/3790753638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cd4100090 0x7f2cd4197150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:22.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.655+0000 7f2cd3fff640 1 --2- 172.21.15.29:0/3790753638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cd4101280 0x7f2cd4197690 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4009dc0 tx=0x7f2cc4032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.655+0000 7f2cd1ffb640 1 -- 172.21.15.29:0/3790753638 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc4034e70 con 0x7f2cd4101280 2024-03-11T19:09:22.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.655+0000 7f2cd1ffb640 1 -- 172.21.15.29:0/3790753638 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2cc4003e20 con 0x7f2cd4101280 2024-03-11T19:09:22.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.655+0000 7f2cdae0f640 1 -- 172.21.15.29:0/3790753638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd419f1f0 con 0x7f2cd4101280 2024-03-11T19:09:22.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.655+0000 7f2cd1ffb640 1 -- 172.21.15.29:0/3790753638 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc403d3f0 con 0x7f2cd4101280 2024-03-11T19:09:22.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.655+0000 7f2cdae0f640 1 -- 172.21.15.29:0/3790753638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd419f6f0 con 0x7f2cd4101280 2024-03-11T19:09:22.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cd1ffb640 1 -- 172.21.15.29:0/3790753638 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2cc403b040 con 0x7f2cd4101280 2024-03-11T19:09:22.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cdd9d9740 1 Processor -- start 2024-03-11T19:09:22.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cdd9d9740 1 -- start start 2024-03-11T19:09:22.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cdd9d9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef809f80 0x55a0ef80a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cdd9d9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a0ef80a970 0x55a0ef80f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cdd9d9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80fae0 0x55a0ef811f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cdd9d9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0ef7c5cd0 con 0x55a0ef80fae0 2024-03-11T19:09:22.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cdd9d9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0ef778f20 con 0x55a0ef809f80 2024-03-11T19:09:22.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cdd9d9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0ef776450 con 0x55a0ef80a970 2024-03-11T19:09:22.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cd3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80fae0 0x55a0ef811f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cd3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80fae0 0x55a0ef811f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55642/0 (socket says 172.21.15.29:55642) 2024-03-11T19:09:22.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cd3fff640 1 -- 172.21.15.29:0/220914579 learned_addr learned my addr 172.21.15.29:0/220914579 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cd8bab640 1 --2- 172.21.15.29:0/220914579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a0ef80a970 0x55a0ef80f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cd1ffb640 1 --2- 172.21.15.29:0/3790753638 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2cc00a1ab0 0x7f2cc00a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cd93ac640 1 --2- 172.21.15.29:0/220914579 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef809f80 0x55a0ef80a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cd93ac640 1 --2- 172.21.15.29:0/3790753638 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2cc00a1ab0 0x7f2cc00a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cd3fff640 1 -- 172.21.15.29:0/220914579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a0ef80a970 msgr2=0x55a0ef80f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cd3fff640 1 --2- 172.21.15.29:0/220914579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a0ef80a970 0x55a0ef80f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cd3fff640 1 -- 172.21.15.29:0/220914579 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef809f80 msgr2=0x55a0ef80a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cd3fff640 1 --2- 172.21.15.29:0/220914579 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef809f80 0x55a0ef80a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.658+0000 7f2cd3fff640 1 -- 172.21.15.29:0/220914579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0ef726f20 con 0x55a0ef80fae0 2024-03-11T19:09:22.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.659+0000 7f2cd93ac640 1 --2- 172.21.15.29:0/220914579 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef809f80 0x55a0ef80a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:22.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.659+0000 7f2cd1ffb640 1 -- 172.21.15.29:0/3790753638 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f2cc40efa70 con 0x7f2cd4101280 2024-03-11T19:09:22.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.659+0000 7f2cd3fff640 1 --2- 172.21.15.29:0/220914579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80fae0 0x55a0ef811f70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2cc40bc780 tx=0x7f2cc402c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65162c75912aaa66 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.659+0000 7f2cd93ac640 1 --2- 172.21.15.29:0/3790753638 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2cc00a1ab0 0x7f2cc00a3f70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x55a0ef80b1d0 tx=0x7f2cbc004730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.659+0000 7f2ccaffd640 1 -- 172.21.15.29:0/220914579 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc4074ea0 con 0x55a0ef80fae0 2024-03-11T19:09:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.659+0000 7f2ccaffd640 1 -- 172.21.15.29:0/220914579 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2cc4004040 con 0x55a0ef80fae0 2024-03-11T19:09:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.659+0000 7f2ccaffd640 1 -- 172.21.15.29:0/220914579 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc40bca80 con 0x55a0ef80fae0 2024-03-11T19:09:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.659+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/220914579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80fae0 msgr2=0x55a0ef811f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.659+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/220914579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80fae0 0x55a0ef811f70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2cc40bc780 tx=0x7f2cc402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.660+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/220914579 shutdown_connections 2024-03-11T19:09:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.660+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/220914579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80fae0 0x55a0ef811f70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.660+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/220914579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a0ef80a970 0x55a0ef80f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.660+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/220914579 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef809f80 0x55a0ef80a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.660+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/220914579 >> 172.21.15.29:0/220914579 conn(0x55a0ef8054f0 msgr2=0x55a0ef8058f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.660+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/220914579 shutdown_connections 2024-03-11T19:09:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.660+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/220914579 wait complete. 2024-03-11T19:09:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.660+0000 7f2cdd9d9740 1 -- start start 2024-03-11T19:09:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.660+0000 7f2cdd9d9740 1 Processor -- start 2024-03-11T19:09:22.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.660+0000 7f2cdd9d9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef809f80 0x55a0ef813d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.660+0000 7f2cdd9d9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a0ef80a970 0x55a0ef814290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.660+0000 7f2cdd9d9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80fae0 0x55a0ef81b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.660+0000 7f2cdd9d9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0ef771c30 con 0x55a0ef80fae0 2024-03-11T19:09:22.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.660+0000 7f2cdd9d9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0ef716d30 con 0x55a0ef809f80 2024-03-11T19:09:22.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.660+0000 7f2cdd9d9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0ef445310 con 0x55a0ef80a970 2024-03-11T19:09:22.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.660+0000 7f2cdd9d9740 10 client.0 _ll_register_callbacks cb 0x55a0ef71ed40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:22.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.660+0000 7f2cd93ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80fae0 0x55a0ef81b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.660+0000 7f2cd93ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80fae0 0x55a0ef81b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55650/0 (socket says 172.21.15.29:55650) 2024-03-11T19:09:22.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.660+0000 7f2cd93ac640 1 -- 172.21.15.29:0/2430230964 learned_addr learned my addr 172.21.15.29:0/2430230964 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.661+0000 7f2cd3fff640 1 --2- 172.21.15.29:0/2430230964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a0ef80a970 0x55a0ef814290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.661+0000 7f2cd8bab640 1 --2- 172.21.15.29:0/2430230964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef809f80 0x55a0ef813d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.661+0000 7f2cd93ac640 1 -- 172.21.15.29:0/2430230964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a0ef80a970 msgr2=0x55a0ef814290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.661+0000 7f2cd93ac640 1 --2- 172.21.15.29:0/2430230964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a0ef80a970 0x55a0ef814290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.661+0000 7f2cd93ac640 1 -- 172.21.15.29:0/2430230964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef809f80 msgr2=0x55a0ef813d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.661+0000 7f2cd93ac640 1 --2- 172.21.15.29:0/2430230964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef809f80 0x55a0ef813d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.661+0000 7f2cd93ac640 1 -- 172.21.15.29:0/2430230964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd419f6f0 con 0x55a0ef80fae0 2024-03-11T19:09:22.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.661+0000 7f2cd93ac640 1 --2- 172.21.15.29:0/2430230964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80fae0 0x55a0ef81b2d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc003ee0 tx=0x7f2cbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f3dc2f8e2a191bd server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.661+0000 7f2cb3fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:22.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.661+0000 7f2cb3fff640 1 -- 172.21.15.29:0/2430230964 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc012870 con 0x55a0ef80fae0 2024-03-11T19:09:22.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.661+0000 7f2cb3fff640 1 -- 172.21.15.29:0/2430230964 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f2cbc0129d0 con 0x55a0ef80fae0 2024-03-11T19:09:22.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.661+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/2430230964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a0ef4716c0 con 0x55a0ef80fae0 2024-03-11T19:09:22.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.661+0000 7f2cb3fff640 1 -- 172.21.15.29:0/2430230964 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2cbc012b90 con 0x55a0ef80fae0 2024-03-11T19:09:22.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.661+0000 7f2cdd9d9740 10 client.15078 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:22.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.661+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/2430230964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55a0ef7702b0 con 0x55a0ef80fae0 2024-03-11T19:09:22.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.661+0000 7f2cb3fff640 1 -- 172.21.15.29:0/2430230964 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55a0ef4716c0 con 0x55a0ef80fae0 2024-03-11T19:09:22.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.662+0000 7f2cb3fff640 1 -- 172.21.15.29:0/2430230964 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2cbc01d040 con 0x55a0ef80fae0 2024-03-11T19:09:22.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.662+0000 7f2cdd9d9740 10 client.15078 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:22.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.662+0000 7f2cdd9d9740 10 client.15078 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-11T19:09:22.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.662+0000 7f2cdd9d9740 20 client.15078 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:22.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.662+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/2430230964 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55a0ef820770 0x55a0ef81f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.662+0000 7f2cdd9d9740 4 client.15078 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:09:22.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.662+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/2430230964 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a0ef4c2270 con 0x55a0ef820770 2024-03-11T19:09:22.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.662+0000 7f2cd8bab640 1 --2- 172.21.15.29:0/2430230964 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55a0ef820770 0x55a0ef81f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.662+0000 7f2cd8bab640 1 --2- 172.21.15.29:0/2430230964 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55a0ef820770 0x55a0ef81f9b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=effb4e62f7c519ea server_cookie=3553616d94f5ec26 in_seq=0 out_seq=0 2024-03-11T19:09:22.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.662+0000 7f2cb3fff640 10 client.15078 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:09:22.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.663+0000 7f2cb3fff640 1 -- 172.21.15.29:0/2430230964 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a0ef4c2270 con 0x55a0ef820770 2024-03-11T19:09:22.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.663+0000 7f2cb3fff640 10 client.15078 handle_command_reply: tid=0 2024-03-11T19:09:22.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.663+0000 7f2cdd9d9740 1 client.15078 shutdown 2024-03-11T19:09:22.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.663+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/2430230964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80fae0 msgr2=0x55a0ef81b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.663+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/2430230964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80fae0 0x55a0ef81b2d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc003ee0 tx=0x7f2cbc0046c0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.663+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/2430230964 shutdown_connections 2024-03-11T19:09:22.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.663+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/2430230964 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55a0ef820770 0x55a0ef81f9b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.663+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/2430230964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80fae0 0x55a0ef81b2d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.663+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/2430230964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a0ef80a970 0x55a0ef814290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.663+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/2430230964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef809f80 0x55a0ef813d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.663+0000 7f2cb3fff640 0 client.15078 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:09:22.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.663+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/2430230964 >> 172.21.15.29:0/2430230964 conn(0x55a0ef8054f0 msgr2=0x55a0ef8047d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.663+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/2430230964 shutdown_connections 2024-03-11T19:09:22.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.663+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/2430230964 wait complete. 2024-03-11T19:09:22.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.663+0000 7f2cdd9d9740 20 client.15078 trim_cache size 0 max 16384 2024-03-11T19:09:22.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.679+0000 7f2cdd9d9740 1 Processor -- start 2024-03-11T19:09:22.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.679+0000 7f2cdd9d9740 1 -- start start 2024-03-11T19:09:22.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.679+0000 7f2cdd9d9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef809f80 0x55a0ef81bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.679+0000 7f2cdd9d9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80a970 0x55a0ef81b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.679+0000 7f2cdd9d9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a0ef80fae0 0x55a0ef813d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.679+0000 7f2cdd9d9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0ef6f7490 con 0x55a0ef80a970 2024-03-11T19:09:22.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.679+0000 7f2cdd9d9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0ef812630 con 0x55a0ef809f80 2024-03-11T19:09:22.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.679+0000 7f2cdd9d9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0ef813b90 con 0x55a0ef80fae0 2024-03-11T19:09:22.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.679+0000 7f2cd3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80a970 0x55a0ef81b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.679+0000 7f2cd3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80a970 0x55a0ef81b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55666/0 (socket says 172.21.15.29:55666) 2024-03-11T19:09:22.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.679+0000 7f2cd3fff640 1 -- 172.21.15.29:0/59988373 learned_addr learned my addr 172.21.15.29:0/59988373 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.679+0000 7f2cd8bab640 1 --2- 172.21.15.29:0/59988373 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef809f80 0x55a0ef81bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.679+0000 7f2cd3fff640 1 -- 172.21.15.29:0/59988373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a0ef80fae0 msgr2=0x55a0ef813d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.679+0000 7f2cd3fff640 1 --2- 172.21.15.29:0/59988373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a0ef80fae0 0x55a0ef813d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.679+0000 7f2cd3fff640 1 -- 172.21.15.29:0/59988373 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef809f80 msgr2=0x55a0ef81bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.679+0000 7f2cd3fff640 1 --2- 172.21.15.29:0/59988373 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef809f80 0x55a0ef81bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.679+0000 7f2cd3fff640 1 -- 172.21.15.29:0/59988373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0ef70ca20 con 0x55a0ef80a970 2024-03-11T19:09:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.680+0000 7f2cd3fff640 1 --2- 172.21.15.29:0/59988373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80a970 0x55a0ef81b1f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2cc40b8ee0 tx=0x7f2cc40bc5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d18a9eae01484fb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.680+0000 7f2cca7fc640 1 -- 172.21.15.29:0/59988373 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc40bd200 con 0x55a0ef80a970 2024-03-11T19:09:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.680+0000 7f2cca7fc640 1 -- 172.21.15.29:0/59988373 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2cc40c0bd0 con 0x55a0ef80a970 2024-03-11T19:09:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.680+0000 7f2cca7fc640 1 -- 172.21.15.29:0/59988373 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc40041d0 con 0x55a0ef80a970 2024-03-11T19:09:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.680+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/59988373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80a970 msgr2=0x55a0ef81b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.680+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/59988373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80a970 0x55a0ef81b1f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2cc40b8ee0 tx=0x7f2cc40bc5c0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.680+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/59988373 shutdown_connections 2024-03-11T19:09:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.680+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/59988373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a0ef80fae0 0x55a0ef813d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.680+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/59988373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80a970 0x55a0ef81b1f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.680+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/59988373 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef809f80 0x55a0ef81bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.680+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/59988373 >> 172.21.15.29:0/59988373 conn(0x55a0ef8054f0 msgr2=0x55a0ef8058d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.680+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/59988373 shutdown_connections 2024-03-11T19:09:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.680+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/59988373 wait complete. 2024-03-11T19:09:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.681+0000 7f2cdd9d9740 1 -- start start 2024-03-11T19:09:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.681+0000 7f2cdd9d9740 1 Processor -- start 2024-03-11T19:09:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.681+0000 7f2cdd9d9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a0ef809f80 0x55a0ef818010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.681+0000 7f2cdd9d9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80a970 0x55a0ef818550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.681+0000 7f2cdd9d9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef80fae0 0x55a0ef876370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.681+0000 7f2cdd9d9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0ef81b500 con 0x55a0ef80a970 2024-03-11T19:09:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.681+0000 7f2cdd9d9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0ef81b670 con 0x55a0ef80fae0 2024-03-11T19:09:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.681+0000 7f2cdd9d9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0ef8768b0 con 0x55a0ef809f80 2024-03-11T19:09:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.681+0000 7f2cdd9d9740 10 client.0 _ll_register_callbacks cb 0x7f2cc40f4040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.681+0000 7f2cd3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80a970 0x55a0ef818550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.681+0000 7f2cd3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80a970 0x55a0ef818550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55678/0 (socket says 172.21.15.29:55678) 2024-03-11T19:09:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.681+0000 7f2cd3fff640 1 -- 172.21.15.29:0/328598318 learned_addr learned my addr 172.21.15.29:0/328598318 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.681+0000 7f2cd93ac640 1 --2- 172.21.15.29:0/328598318 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef80fae0 0x55a0ef876370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.681+0000 7f2cd3fff640 1 -- 172.21.15.29:0/328598318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a0ef809f80 msgr2=0x55a0ef818010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.681+0000 7f2cd3fff640 1 --2- 172.21.15.29:0/328598318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a0ef809f80 0x55a0ef818010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.681+0000 7f2cd3fff640 1 -- 172.21.15.29:0/328598318 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef80fae0 msgr2=0x55a0ef876370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.681+0000 7f2cd3fff640 1 --2- 172.21.15.29:0/328598318 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef80fae0 0x55a0ef876370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.681+0000 7f2cd3fff640 1 -- 172.21.15.29:0/328598318 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a0ef726f20 con 0x55a0ef80a970 2024-03-11T19:09:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.682+0000 7f2cd3fff640 1 --2- 172.21.15.29:0/328598318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80a970 0x55a0ef818550 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2cc40f1010 tx=0x7f2cc40b8d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b983f530a69fbea6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.682+0000 7f2cb3fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.682+0000 7f2cb3fff640 1 -- 172.21.15.29:0/328598318 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc40d9070 con 0x55a0ef80a970 2024-03-11T19:09:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.682+0000 7f2cb3fff640 1 -- 172.21.15.29:0/328598318 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f2cc40c5220 con 0x55a0ef80a970 2024-03-11T19:09:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.682+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/328598318 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2cc4004040 con 0x55a0ef80a970 2024-03-11T19:09:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.682+0000 7f2cb3fff640 1 -- 172.21.15.29:0/328598318 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2cc40bca80 con 0x55a0ef80a970 2024-03-11T19:09:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.682+0000 7f2cdd9d9740 10 client.15084 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.682+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/328598318 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55a0ef732780 con 0x55a0ef80a970 2024-03-11T19:09:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.682+0000 7f2cb3fff640 1 -- 172.21.15.29:0/328598318 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2cc4004040 con 0x55a0ef80a970 2024-03-11T19:09:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.682+0000 7f2cb3fff640 1 -- 172.21.15.29:0/328598318 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2cc409bb30 con 0x55a0ef80a970 2024-03-11T19:09:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.682+0000 7f2cdd9d9740 10 client.15084 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.682+0000 7f2cdd9d9740 10 client.15084 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-11T19:09:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.682+0000 7f2cdd9d9740 20 client.15084 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.682+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/328598318 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55a0ef87b730 0x55a0ef877bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.682+0000 7f2cdd9d9740 4 client.15084 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:09:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.682+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/328598318 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a0ef4ae610 con 0x55a0ef87b730 2024-03-11T19:09:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.682+0000 7f2cd8bab640 1 --2- 172.21.15.29:0/328598318 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55a0ef87b730 0x55a0ef877bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.683+0000 7f2cb3fff640 10 client.15084 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:09:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.683+0000 7f2cd8bab640 1 --2- 172.21.15.29:0/328598318 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55a0ef87b730 0x55a0ef877bb0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=76bc04bb88c712bc server_cookie=cb8cf9b7fa8cb878 in_seq=0 out_seq=0 2024-03-11T19:09:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.683+0000 7f2cb3fff640 1 -- 172.21.15.29:0/328598318 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55a0ef4ae610 con 0x55a0ef87b730 2024-03-11T19:09:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.683+0000 7f2cb3fff640 10 client.15084 handle_command_reply: tid=0 2024-03-11T19:09:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.683+0000 7f2cdd9d9740 1 client.15084 shutdown 2024-03-11T19:09:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.683+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/328598318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80a970 msgr2=0x55a0ef818550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.683+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/328598318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80a970 0x55a0ef818550 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2cc40f1010 tx=0x7f2cc40b8d20 comp rx=0 tx=0).stop 2024-03-11T19:09:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.683+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/328598318 shutdown_connections 2024-03-11T19:09:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.683+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/328598318 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55a0ef87b730 0x55a0ef877bb0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.683+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/328598318 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a0ef80fae0 0x55a0ef876370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.683+0000 7f2cb3fff640 0 client.15084 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:09:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.683+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/328598318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0ef80a970 0x55a0ef818550 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.683+0000 7f2cdd9d9740 1 --2- 172.21.15.29:0/328598318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a0ef809f80 0x55a0ef818010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.683+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/328598318 >> 172.21.15.29:0/328598318 conn(0x55a0ef8054f0 msgr2=0x55a0ef80bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.683+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/328598318 shutdown_connections 2024-03-11T19:09:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.683+0000 7f2cdd9d9740 1 -- 172.21.15.29:0/328598318 wait complete. 2024-03-11T19:09:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.683+0000 7f2cdd9d9740 20 client.15084 trim_cache size 0 max 16384 2024-03-11T19:09:22.693 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:09:22.693 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "638915c2-dfd9-11ee-95c8-87774f69a715", 2024-03-11T19:09:22.693 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 2, 2024-03-11T19:09:22.693 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24475, 2024-03-11T19:09:22.693 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-03-11T19:09:22.693 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-03-11T19:09:22.693 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-03-11T19:09:22.693 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 88.921158043000005, 2024-03-11T19:09:22.693 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 26, 2024-03-11T19:09:22.693 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 89, 2024-03-11T19:09:22.694 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 87, 2024-03-11T19:09:22.694 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 111.591274506 2024-03-11T19:09:22.694 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:09:22.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.684+0000 7f2cdae0f640 1 -- 172.21.15.29:0/3790753638 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2cc00a1ab0 msgr2=0x7f2cc00a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.684+0000 7f2cdae0f640 1 --2- 172.21.15.29:0/3790753638 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2cc00a1ab0 0x7f2cc00a3f70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x55a0ef80b1d0 tx=0x7f2cbc004730 comp rx=0 tx=0).stop 2024-03-11T19:09:22.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.684+0000 7f2cdae0f640 1 -- 172.21.15.29:0/3790753638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cd4101280 msgr2=0x7f2cd4197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.684+0000 7f2cdae0f640 1 --2- 172.21.15.29:0/3790753638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cd4101280 0x7f2cd4197690 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4009dc0 tx=0x7f2cc4032980 comp rx=0 tx=0).stop 2024-03-11T19:09:22.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.684+0000 7f2cdae0f640 1 -- 172.21.15.29:0/3790753638 shutdown_connections 2024-03-11T19:09:22.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.684+0000 7f2cdae0f640 1 --2- 172.21.15.29:0/3790753638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2cd4101c40 0x7f2cd419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.684+0000 7f2cdae0f640 1 --2- 172.21.15.29:0/3790753638 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2cc00a1ab0 0x7f2cc00a3f70 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.684+0000 7f2cdae0f640 1 --2- 172.21.15.29:0/3790753638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cd4101280 0x7f2cd4197690 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.684+0000 7f2cdae0f640 1 --2- 172.21.15.29:0/3790753638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cd4100090 0x7f2cd4197150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.684+0000 7f2cdae0f640 1 -- 172.21.15.29:0/3790753638 >> 172.21.15.29:0/3790753638 conn(0x7f2cd40fb620 msgr2=0x7f2cd4069770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.684+0000 7f2cdae0f640 1 -- 172.21.15.29:0/3790753638 shutdown_connections 2024-03-11T19:09:22.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.684+0000 7f2cdae0f640 1 -- 172.21.15.29:0/3790753638 wait complete. 2024-03-11T19:09:22.697 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-11T19:09:22.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.798+0000 7f2df92eb640 1 -- 172.21.15.29:0/3305263125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df4101280 msgr2=0x7f2df4101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.798+0000 7f2df92eb640 1 --2- 172.21.15.29:0/3305263125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df4101280 0x7f2df4101700 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc009900 tx=0x7f2ddc02bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:22.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.798+0000 7f2df92eb640 1 -- 172.21.15.29:0/3305263125 shutdown_connections 2024-03-11T19:09:22.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.798+0000 7f2df92eb640 1 --2- 172.21.15.29:0/3305263125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2df4101c40 0x7f2df41084d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.798+0000 7f2df92eb640 1 --2- 172.21.15.29:0/3305263125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df4101280 0x7f2df4101700 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.798+0000 7f2df92eb640 1 --2- 172.21.15.29:0/3305263125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2df4100090 0x7f2df41004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.798+0000 7f2df92eb640 1 -- 172.21.15.29:0/3305263125 >> 172.21.15.29:0/3305263125 conn(0x7f2df40fb620 msgr2=0x7f2df40fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.798+0000 7f2df92eb640 1 -- 172.21.15.29:0/3305263125 shutdown_connections 2024-03-11T19:09:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.798+0000 7f2df92eb640 1 -- 172.21.15.29:0/3305263125 wait complete. 2024-03-11T19:09:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.799+0000 7f2df92eb640 1 Processor -- start 2024-03-11T19:09:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.799+0000 7f2df92eb640 1 -- start start 2024-03-11T19:09:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.799+0000 7f2df92eb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2df4100090 0x7f2df41971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.799+0000 7f2df92eb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df4101280 0x7f2df4197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.799+0000 7f2df92eb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2df4101c40 0x7f2df419e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.799+0000 7f2df92eb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df419ecb0 con 0x7f2df4101280 2024-03-11T19:09:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.799+0000 7f2df92eb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df419ee20 con 0x7f2df4101c40 2024-03-11T19:09:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.799+0000 7f2df92eb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df419efe0 con 0x7f2df4100090 2024-03-11T19:09:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.799+0000 7f2debfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df4101280 0x7f2df4197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.799+0000 7f2debfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df4101280 0x7f2df4197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55696/0 (socket says 172.21.15.29:55696) 2024-03-11T19:09:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.799+0000 7f2debfff640 1 -- 172.21.15.29:0/940004102 learned_addr learned my addr 172.21.15.29:0/940004102 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.799+0000 7f2df2ffd640 1 --2- 172.21.15.29:0/940004102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2df4100090 0x7f2df41971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.799+0000 7f2df37fe640 1 --2- 172.21.15.29:0/940004102 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2df4101c40 0x7f2df419e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.799+0000 7f2debfff640 1 -- 172.21.15.29:0/940004102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2df4100090 msgr2=0x7f2df41971c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.799+0000 7f2debfff640 1 --2- 172.21.15.29:0/940004102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2df4100090 0x7f2df41971c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.799+0000 7f2debfff640 1 -- 172.21.15.29:0/940004102 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2df4101c40 msgr2=0x7f2df419e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.799+0000 7f2debfff640 1 --2- 172.21.15.29:0/940004102 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2df4101c40 0x7f2df419e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.799+0000 7f2debfff640 1 -- 172.21.15.29:0/940004102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc009560 con 0x7f2df4101280 2024-03-11T19:09:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.799+0000 7f2debfff640 1 --2- 172.21.15.29:0/940004102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df4101280 0x7f2df4197700 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc036040 tx=0x7f2ddc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.800+0000 7f2df0ff9640 1 -- 172.21.15.29:0/940004102 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc0044e0 con 0x7f2df4101280 2024-03-11T19:09:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.800+0000 7f2df92eb640 1 -- 172.21.15.29:0/940004102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df419f260 con 0x7f2df4101280 2024-03-11T19:09:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.800+0000 7f2df0ff9640 1 -- 172.21.15.29:0/940004102 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2ddc02de90 con 0x7f2df4101280 2024-03-11T19:09:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.800+0000 7f2df0ff9640 1 -- 172.21.15.29:0/940004102 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc005470 con 0x7f2df4101280 2024-03-11T19:09:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.800+0000 7f2df92eb640 1 -- 172.21.15.29:0/940004102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df419f7c0 con 0x7f2df4101280 2024-03-11T19:09:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.802+0000 7f2df0ff9640 1 -- 172.21.15.29:0/940004102 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2ddc003800 con 0x7f2df4101280 2024-03-11T19:09:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.802+0000 7f2dfbf98740 1 Processor -- start 2024-03-11T19:09:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.802+0000 7f2df0ff9640 1 --2- 172.21.15.29:0/940004102 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2de00a1a30 0x7f2de00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.802+0000 7f2dfbf98740 1 -- start start 2024-03-11T19:09:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.802+0000 7f2df37fe640 1 --2- 172.21.15.29:0/940004102 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2de00a1a30 0x7f2de00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.802+0000 7f2df0ff9640 1 -- 172.21.15.29:0/940004102 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f2ddc0ef690 con 0x7f2df4101280 2024-03-11T19:09:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.802+0000 7f2dfbf98740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 0x5571bf70b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.802+0000 7f2dfbf98740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70b970 0x5571bf710470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.802+0000 7f2dfbf98740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 0x5571bf712f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.802+0000 7f2dfbf98740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571bf6c6cd0 con 0x5571bf70af80 2024-03-11T19:09:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.802+0000 7f2dfbf98740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571bf679f20 con 0x5571bf70b970 2024-03-11T19:09:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.802+0000 7f2dfbf98740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571bf677450 con 0x5571bf710ae0 2024-03-11T19:09:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.803+0000 7f2df2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 0x5571bf70b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.803+0000 7f2df2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 0x5571bf70b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55710/0 (socket says 172.21.15.29:55710) 2024-03-11T19:09:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.803+0000 7f2df2ffd640 1 -- 172.21.15.29:0/2941016801 learned_addr learned my addr 172.21.15.29:0/2941016801 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.803+0000 7f2df37fe640 1 --2- 172.21.15.29:0/2941016801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 0x5571bf712f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.803+0000 7f2df37fe640 1 --2- 172.21.15.29:0/940004102 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2de00a1a30 0x7f2de00a3ef0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2dec005fd0 tx=0x7f2dec009520 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.803+0000 7f2debfff640 1 --2- 172.21.15.29:0/2941016801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70b970 0x5571bf710470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.803+0000 7f2df2ffd640 1 -- 172.21.15.29:0/2941016801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 msgr2=0x5571bf712f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.803+0000 7f2df2ffd640 1 --2- 172.21.15.29:0/2941016801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 0x5571bf712f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.803+0000 7f2df2ffd640 1 -- 172.21.15.29:0/2941016801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70b970 msgr2=0x5571bf710470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.803+0000 7f2df2ffd640 1 --2- 172.21.15.29:0/2941016801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70b970 0x5571bf710470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.803+0000 7f2df2ffd640 1 -- 172.21.15.29:0/2941016801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571bf627f20 con 0x5571bf70af80 2024-03-11T19:09:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.803+0000 7f2debfff640 1 --2- 172.21.15.29:0/2941016801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70b970 0x5571bf710470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.803+0000 7f2df2ffd640 1 --2- 172.21.15.29:0/2941016801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 0x5571bf70b3a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2de400cbc0 tx=0x7f2de400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d463e9fc864cd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.803+0000 7f2de97fa640 1 -- 172.21.15.29:0/2941016801 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de4010630 con 0x5571bf70af80 2024-03-11T19:09:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.803+0000 7f2de97fa640 1 -- 172.21.15.29:0/2941016801 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2de4010c40 con 0x5571bf70af80 2024-03-11T19:09:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.803+0000 7f2de97fa640 1 -- 172.21.15.29:0/2941016801 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de401aa40 con 0x5571bf70af80 2024-03-11T19:09:22.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.804+0000 7f2dfbf98740 1 -- 172.21.15.29:0/2941016801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 msgr2=0x5571bf70b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.804+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/2941016801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 0x5571bf70b3a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2de400cbc0 tx=0x7f2de400e720 comp rx=0 tx=0).stop 2024-03-11T19:09:22.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.804+0000 7f2dfbf98740 1 -- 172.21.15.29:0/2941016801 shutdown_connections 2024-03-11T19:09:22.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.804+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/2941016801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 0x5571bf712f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.804+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/2941016801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70b970 0x5571bf710470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.804+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/2941016801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 0x5571bf70b3a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.804+0000 7f2dfbf98740 1 -- 172.21.15.29:0/2941016801 >> 172.21.15.29:0/2941016801 conn(0x5571bf7064f0 msgr2=0x5571bf7068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.804+0000 7f2dfbf98740 1 -- 172.21.15.29:0/2941016801 shutdown_connections 2024-03-11T19:09:22.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.804+0000 7f2dfbf98740 1 -- 172.21.15.29:0/2941016801 wait complete. 2024-03-11T19:09:22.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.804+0000 7f2dfbf98740 1 -- start start 2024-03-11T19:09:22.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.804+0000 7f2dfbf98740 1 Processor -- start 2024-03-11T19:09:22.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2dfbf98740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 0x5571bf715130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2dfbf98740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70b970 0x5571bf715670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2dfbf98740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 0x5571bf71c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2dfbf98740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571bf672c30 con 0x5571bf70af80 2024-03-11T19:09:22.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2dfbf98740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571bf617d30 con 0x5571bf70b970 2024-03-11T19:09:22.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2dfbf98740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571bf346310 con 0x5571bf710ae0 2024-03-11T19:09:22.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2dfbf98740 10 client.0 _ll_register_callbacks cb 0x5571bf61fd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:22.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2df2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 0x5571bf715130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2df2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 0x5571bf715130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55714/0 (socket says 172.21.15.29:55714) 2024-03-11T19:09:22.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2df2ffd640 1 -- 172.21.15.29:0/2649554497 learned_addr learned my addr 172.21.15.29:0/2649554497 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2debfff640 1 --2- 172.21.15.29:0/2649554497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70b970 0x5571bf715670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2df37fe640 1 --2- 172.21.15.29:0/2649554497 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 0x5571bf71c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2df2ffd640 1 -- 172.21.15.29:0/2649554497 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 msgr2=0x5571bf71c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2df2ffd640 1 --2- 172.21.15.29:0/2649554497 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 0x5571bf71c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2df2ffd640 1 -- 172.21.15.29:0/2649554497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70b970 msgr2=0x5571bf715670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2df2ffd640 1 --2- 172.21.15.29:0/2649554497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70b970 0x5571bf715670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2df2ffd640 1 -- 172.21.15.29:0/2649554497 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de400c7f0 con 0x5571bf70af80 2024-03-11T19:09:22.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2debfff640 1 --2- 172.21.15.29:0/2649554497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70b970 0x5571bf715670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:22.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2df2ffd640 1 --2- 172.21.15.29:0/2649554497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 0x5571bf715130 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2de40058c0 tx=0x7f2de4010a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d776d32b166b78ee server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2dda7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:22.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2dda7fc640 1 -- 172.21.15.29:0/2649554497 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de401f070 con 0x5571bf70af80 2024-03-11T19:09:22.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2dda7fc640 1 -- 172.21.15.29:0/2649554497 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f2de4005260 con 0x5571bf70af80 2024-03-11T19:09:22.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.805+0000 7f2dfbf98740 1 -- 172.21.15.29:0/2649554497 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5571bf3726c0 con 0x5571bf70af80 2024-03-11T19:09:22.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.806+0000 7f2dda7fc640 1 -- 172.21.15.29:0/2649554497 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2de401a410 con 0x5571bf70af80 2024-03-11T19:09:22.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.806+0000 7f2dfbf98740 10 client.15096 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:22.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.806+0000 7f2dfbf98740 1 -- 172.21.15.29:0/2649554497 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5571bf6712b0 con 0x5571bf70af80 2024-03-11T19:09:22.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.806+0000 7f2dda7fc640 1 -- 172.21.15.29:0/2649554497 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5571bf3726c0 con 0x5571bf70af80 2024-03-11T19:09:22.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.806+0000 7f2dda7fc640 1 -- 172.21.15.29:0/2649554497 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2de4004830 con 0x5571bf70af80 2024-03-11T19:09:22.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.806+0000 7f2dfbf98740 10 client.15096 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:22.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.806+0000 7f2dfbf98740 10 client.15096 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-11T19:09:22.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.806+0000 7f2dfbf98740 20 client.15096 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:22.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.806+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/2649554497 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5571bf721260 0x5571bf725d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.806+0000 7f2dfbf98740 4 client.15096 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:09:22.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.806+0000 7f2dfbf98740 1 -- 172.21.15.29:0/2649554497 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5571bf3c3270 con 0x5571bf721260 2024-03-11T19:09:22.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.806+0000 7f2debfff640 1 --2- 172.21.15.29:0/2649554497 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5571bf721260 0x5571bf725d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.807+0000 7f2dda7fc640 10 client.15096 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:09:22.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.807+0000 7f2debfff640 1 --2- 172.21.15.29:0/2649554497 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5571bf721260 0x5571bf725d20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d3e2d96dd4d98da1 server_cookie=75d60eb0996651a2 in_seq=0 out_seq=0 2024-03-11T19:09:22.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.807+0000 7f2dda7fc640 1 -- 172.21.15.29:0/2649554497 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5571bf3c3270 con 0x5571bf721260 2024-03-11T19:09:22.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.808+0000 7f2dda7fc640 10 client.15096 handle_command_reply: tid=0 2024-03-11T19:09:22.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.808+0000 7f2dfbf98740 1 client.15096 shutdown 2024-03-11T19:09:22.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.808+0000 7f2dfbf98740 1 -- 172.21.15.29:0/2649554497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 msgr2=0x5571bf715130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.808+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/2649554497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 0x5571bf715130 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2de40058c0 tx=0x7f2de4010a90 comp rx=0 tx=0).stop 2024-03-11T19:09:22.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.808+0000 7f2dfbf98740 1 -- 172.21.15.29:0/2649554497 shutdown_connections 2024-03-11T19:09:22.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.808+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/2649554497 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 0x5571bf71c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.808+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/2649554497 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5571bf721260 0x5571bf725d20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.808+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/2649554497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70b970 0x5571bf715670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.808+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/2649554497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 0x5571bf715130 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.808+0000 7f2dda7fc640 0 client.15096 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:09:22.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.808+0000 7f2dfbf98740 1 -- 172.21.15.29:0/2649554497 >> 172.21.15.29:0/2649554497 conn(0x5571bf7064f0 msgr2=0x5571bf7057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.808+0000 7f2dfbf98740 1 -- 172.21.15.29:0/2649554497 shutdown_connections 2024-03-11T19:09:22.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.808+0000 7f2dfbf98740 1 -- 172.21.15.29:0/2649554497 wait complete. 2024-03-11T19:09:22.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.808+0000 7f2dfbf98740 20 client.15096 trim_cache size 0 max 16384 2024-03-11T19:09:22.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.823+0000 7f2dfbf98740 1 Processor -- start 2024-03-11T19:09:22.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.823+0000 7f2dfbf98740 1 -- start start 2024-03-11T19:09:22.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2dfbf98740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 0x5571bf712ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2dfbf98740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70b970 0x5571bf71c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2dfbf98740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 0x5571bf72a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2dfbf98740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571bf5f8490 con 0x5571bf70af80 2024-03-11T19:09:22.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2dfbf98740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571bf3726c0 con 0x5571bf70b970 2024-03-11T19:09:22.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2dfbf98740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571bf6dc8c0 con 0x5571bf710ae0 2024-03-11T19:09:22.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2df2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 0x5571bf712ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2df2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 0x5571bf712ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55724/0 (socket says 172.21.15.29:55724) 2024-03-11T19:09:22.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2df2ffd640 1 -- 172.21.15.29:0/4035637153 learned_addr learned my addr 172.21.15.29:0/4035637153 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2df37fe640 1 --2- 172.21.15.29:0/4035637153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 0x5571bf72a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2df2ffd640 1 -- 172.21.15.29:0/4035637153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 msgr2=0x5571bf72a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2df2ffd640 1 --2- 172.21.15.29:0/4035637153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 0x5571bf72a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2df2ffd640 1 -- 172.21.15.29:0/4035637153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70b970 msgr2=0x5571bf71c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2debfff640 1 --2- 172.21.15.29:0/4035637153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70b970 0x5571bf71c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2df2ffd640 1 --2- 172.21.15.29:0/4035637153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70b970 0x5571bf71c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2df2ffd640 1 -- 172.21.15.29:0/4035637153 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571bf60da20 con 0x5571bf70af80 2024-03-11T19:09:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2df2ffd640 1 --2- 172.21.15.29:0/4035637153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 0x5571bf712ed0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2de400d010 tx=0x7f2de400c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2ed53f90ab803e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2de8ff9640 1 -- 172.21.15.29:0/4035637153 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de401f070 con 0x5571bf70af80 2024-03-11T19:09:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2de8ff9640 1 -- 172.21.15.29:0/4035637153 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2de4020040 con 0x5571bf70af80 2024-03-11T19:09:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.824+0000 7f2de8ff9640 1 -- 172.21.15.29:0/4035637153 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de401a790 con 0x5571bf70af80 2024-03-11T19:09:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.825+0000 7f2dfbf98740 1 -- 172.21.15.29:0/4035637153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 msgr2=0x5571bf712ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.825+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/4035637153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 0x5571bf712ed0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2de400d010 tx=0x7f2de400c750 comp rx=0 tx=0).stop 2024-03-11T19:09:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.825+0000 7f2dfbf98740 1 -- 172.21.15.29:0/4035637153 shutdown_connections 2024-03-11T19:09:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.825+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/4035637153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 0x5571bf72a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.825+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/4035637153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70b970 0x5571bf71c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.825+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/4035637153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70af80 0x5571bf712ed0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.825+0000 7f2dfbf98740 1 -- 172.21.15.29:0/4035637153 >> 172.21.15.29:0/4035637153 conn(0x5571bf7064f0 msgr2=0x5571bf729d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.825+0000 7f2dfbf98740 1 -- 172.21.15.29:0/4035637153 shutdown_connections 2024-03-11T19:09:22.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.825+0000 7f2dfbf98740 1 -- 172.21.15.29:0/4035637153 wait complete. 2024-03-11T19:09:22.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.825+0000 7f2dfbf98740 1 -- start start 2024-03-11T19:09:22.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.825+0000 7f2dfbf98740 1 Processor -- start 2024-03-11T19:09:22.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2dfbf98740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70af80 0x5571bf7194d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2dfbf98740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70b970 0x5571bf719a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2dfbf98740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 0x5571bf7779e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2dfbf98740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571bf71a110 con 0x5571bf70b970 2024-03-11T19:09:22.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2dfbf98740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571bf777f20 con 0x5571bf70af80 2024-03-11T19:09:22.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2dfbf98740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571bf778060 con 0x5571bf710ae0 2024-03-11T19:09:22.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2dfbf98740 10 client.0 _ll_register_callbacks cb 0x7f2de400c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:22.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2debfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70b970 0x5571bf719a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2debfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70b970 0x5571bf719a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55732/0 (socket says 172.21.15.29:55732) 2024-03-11T19:09:22.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2debfff640 1 -- 172.21.15.29:0/3161849739 learned_addr learned my addr 172.21.15.29:0/3161849739 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2df2ffd640 1 --2- 172.21.15.29:0/3161849739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70af80 0x5571bf7194d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2df37fe640 1 --2- 172.21.15.29:0/3161849739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 0x5571bf7779e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2debfff640 1 -- 172.21.15.29:0/3161849739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 msgr2=0x5571bf7779e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2debfff640 1 --2- 172.21.15.29:0/3161849739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 0x5571bf7779e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2debfff640 1 -- 172.21.15.29:0/3161849739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70af80 msgr2=0x5571bf7194d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2debfff640 1 --2- 172.21.15.29:0/3161849739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70af80 0x5571bf7194d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2debfff640 1 -- 172.21.15.29:0/3161849739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5571bf6712b0 con 0x5571bf70b970 2024-03-11T19:09:22.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2df2ffd640 1 --2- 172.21.15.29:0/3161849739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70af80 0x5571bf7194d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:22.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2debfff640 1 --2- 172.21.15.29:0/3161849739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70b970 0x5571bf719a10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc0b9070 tx=0x7f2ddc0dc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=917cc8306885104c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2dda7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:22.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2dda7fc640 1 -- 172.21.15.29:0/3161849739 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc0bc6a0 con 0x5571bf70b970 2024-03-11T19:09:22.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2dda7fc640 1 -- 172.21.15.29:0/3161849739 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f2ddc0c07f0 con 0x5571bf70b970 2024-03-11T19:09:22.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2dfbf98740 1 -- 172.21.15.29:0/3161849739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5571bf7228d0 con 0x5571bf70b970 2024-03-11T19:09:22.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2dda7fc640 1 -- 172.21.15.29:0/3161849739 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ddc0c4e40 con 0x5571bf70b970 2024-03-11T19:09:22.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2dfbf98740 10 client.15102 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:22.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.826+0000 7f2dfbf98740 1 -- 172.21.15.29:0/3161849739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5571bf633780 con 0x5571bf70b970 2024-03-11T19:09:22.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.827+0000 7f2dda7fc640 1 -- 172.21.15.29:0/3161849739 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5571bf7228d0 con 0x5571bf70b970 2024-03-11T19:09:22.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.827+0000 7f2dda7fc640 1 -- 172.21.15.29:0/3161849739 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2ddc03c070 con 0x5571bf70b970 2024-03-11T19:09:22.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.827+0000 7f2dfbf98740 10 client.15102 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:22.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.827+0000 7f2dfbf98740 10 client.15102 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-11T19:09:22.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.827+0000 7f2dfbf98740 20 client.15102 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:22.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.827+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/3161849739 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5571bf77cff0 0x5571bf779470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.827+0000 7f2dfbf98740 4 client.15102 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:09:22.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.827+0000 7f2dfbf98740 1 -- 172.21.15.29:0/3161849739 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f2de4010c40 con 0x5571bf77cff0 2024-03-11T19:09:22.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.827+0000 7f2df2ffd640 1 --2- 172.21.15.29:0/3161849739 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5571bf77cff0 0x5571bf779470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.827+0000 7f2dda7fc640 10 client.15102 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:09:22.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.827+0000 7f2df2ffd640 1 --2- 172.21.15.29:0/3161849739 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5571bf77cff0 0x5571bf779470 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f4ce78ffeb9dc73f server_cookie=363fe895627e76a9 in_seq=0 out_seq=0 2024-03-11T19:09:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2dda7fc640 1 -- 172.21.15.29:0/3161849739 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x7f2de4010c40 con 0x5571bf77cff0 2024-03-11T19:09:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2dda7fc640 10 client.15102 handle_command_reply: tid=0 2024-03-11T19:09:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2dfbf98740 1 client.15102 shutdown 2024-03-11T19:09:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2dfbf98740 1 -- 172.21.15.29:0/3161849739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70b970 msgr2=0x5571bf719a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/3161849739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70b970 0x5571bf719a10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc0b9070 tx=0x7f2ddc0dc040 comp rx=0 tx=0).stop 2024-03-11T19:09:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2dfbf98740 1 -- 172.21.15.29:0/3161849739 shutdown_connections 2024-03-11T19:09:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/3161849739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571bf710ae0 0x5571bf7779e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/3161849739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5571bf70b970 0x5571bf719a10 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/3161849739 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5571bf77cff0 0x5571bf779470 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2dfbf98740 1 --2- 172.21.15.29:0/3161849739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5571bf70af80 0x5571bf7194d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2dda7fc640 0 client.15102 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:09:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2dfbf98740 1 -- 172.21.15.29:0/3161849739 >> 172.21.15.29:0/3161849739 conn(0x5571bf7064f0 msgr2=0x5571bf72a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2dfbf98740 1 -- 172.21.15.29:0/3161849739 shutdown_connections 2024-03-11T19:09:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2dfbf98740 1 -- 172.21.15.29:0/3161849739 wait complete. 2024-03-11T19:09:22.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2dfbf98740 20 client.15102 trim_cache size 0 max 16384 2024-03-11T19:09:22.838 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:09:22.838 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "638915c2-dfd9-11ee-95c8-87774f69a715", 2024-03-11T19:09:22.838 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 1, 2024-03-11T19:09:22.838 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24496, 2024-03-11T19:09:22.838 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-03-11T19:09:22.838 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-03-11T19:09:22.838 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-03-11T19:09:22.838 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 90.072615572999993, 2024-03-11T19:09:22.838 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 26, 2024-03-11T19:09:22.838 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 89, 2024-03-11T19:09:22.839 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 87, 2024-03-11T19:09:22.839 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 105.98946291599999 2024-03-11T19:09:22.839 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:09:22.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2df92eb640 1 -- 172.21.15.29:0/940004102 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2de00a1a30 msgr2=0x7f2de00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2df92eb640 1 --2- 172.21.15.29:0/940004102 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2de00a1a30 0x7f2de00a3ef0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2dec005fd0 tx=0x7f2dec009520 comp rx=0 tx=0).stop 2024-03-11T19:09:22.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2df92eb640 1 -- 172.21.15.29:0/940004102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df4101280 msgr2=0x7f2df4197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2df92eb640 1 --2- 172.21.15.29:0/940004102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df4101280 0x7f2df4197700 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc036040 tx=0x7f2ddc0042f0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2df92eb640 1 -- 172.21.15.29:0/940004102 shutdown_connections 2024-03-11T19:09:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2df92eb640 1 --2- 172.21.15.29:0/940004102 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2df4101c40 0x7f2df419e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2df92eb640 1 --2- 172.21.15.29:0/940004102 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2de00a1a30 0x7f2de00a3ef0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2df92eb640 1 --2- 172.21.15.29:0/940004102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2df4101280 0x7f2df4197700 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2df92eb640 1 --2- 172.21.15.29:0/940004102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2df4100090 0x7f2df41971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.828+0000 7f2df92eb640 1 -- 172.21.15.29:0/940004102 >> 172.21.15.29:0/940004102 conn(0x7f2df40fb620 msgr2=0x7f2df4069780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.829+0000 7f2df92eb640 1 -- 172.21.15.29:0/940004102 shutdown_connections 2024-03-11T19:09:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.829+0000 7f2df92eb640 1 -- 172.21.15.29:0/940004102 wait complete. 2024-03-11T19:09:22.842 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-11T19:09:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.942+0000 7fd890735640 1 -- 172.21.15.29:0/3711005443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8881000a0 msgr2=0x7fd8881004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.942+0000 7fd890735640 1 --2- 172.21.15.29:0/3711005443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8881000a0 0x7fd8881004c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd884009900 tx=0x7fd88402bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.942+0000 7fd890735640 1 -- 172.21.15.29:0/3711005443 shutdown_connections 2024-03-11T19:09:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.942+0000 7fd890735640 1 --2- 172.21.15.29:0/3711005443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd888101c50 0x7fd8881084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.943+0000 7fd890735640 1 --2- 172.21.15.29:0/3711005443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd888101290 0x7fd888101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.943+0000 7fd890735640 1 --2- 172.21.15.29:0/3711005443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8881000a0 0x7fd8881004c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.943+0000 7fd890735640 1 -- 172.21.15.29:0/3711005443 >> 172.21.15.29:0/3711005443 conn(0x7fd8880fb630 msgr2=0x7fd8880fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.943+0000 7fd890735640 1 -- 172.21.15.29:0/3711005443 shutdown_connections 2024-03-11T19:09:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.943+0000 7fd890735640 1 -- 172.21.15.29:0/3711005443 wait complete. 2024-03-11T19:09:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.943+0000 7fd890735640 1 Processor -- start 2024-03-11T19:09:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.943+0000 7fd890735640 1 -- start start 2024-03-11T19:09:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.943+0000 7fd890735640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8881000a0 0x7fd8881960e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.943+0000 7fd890735640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd888101290 0x7fd888196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.943+0000 7fd890735640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd888101c50 0x7fd88819d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.943+0000 7fd890735640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88819dbd0 con 0x7fd888101290 2024-03-11T19:09:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.943+0000 7fd890735640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88819dd40 con 0x7fd8881000a0 2024-03-11T19:09:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.943+0000 7fd890735640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88819df00 con 0x7fd888101c50 2024-03-11T19:09:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.943+0000 7fd88dcd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd888101290 0x7fd888196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.943+0000 7fd88dcd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd888101290 0x7fd888196620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55746/0 (socket says 172.21.15.29:55746) 2024-03-11T19:09:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.943+0000 7fd88dcd0640 1 -- 172.21.15.29:0/1626856867 learned_addr learned my addr 172.21.15.29:0/1626856867 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.944+0000 7fd88ecd2640 1 --2- 172.21.15.29:0/1626856867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd888101c50 0x7fd88819d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.944+0000 7fd88e4d1640 1 --2- 172.21.15.29:0/1626856867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8881000a0 0x7fd8881960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.944+0000 7fd88dcd0640 1 -- 172.21.15.29:0/1626856867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd888101c50 msgr2=0x7fd88819d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.944+0000 7fd88dcd0640 1 --2- 172.21.15.29:0/1626856867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd888101c50 0x7fd88819d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.944+0000 7fd88dcd0640 1 -- 172.21.15.29:0/1626856867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8881000a0 msgr2=0x7fd8881960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.944+0000 7fd88dcd0640 1 --2- 172.21.15.29:0/1626856867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8881000a0 0x7fd8881960e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.944+0000 7fd88dcd0640 1 -- 172.21.15.29:0/1626856867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd884009560 con 0x7fd888101290 2024-03-11T19:09:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.944+0000 7fd88dcd0640 1 --2- 172.21.15.29:0/1626856867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd888101290 0x7fd888196620 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd87800d770 tx=0x7fd87800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.944+0000 7fd87f7fe640 1 -- 172.21.15.29:0/1626856867 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd878009c30 con 0x7fd888101290 2024-03-11T19:09:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.944+0000 7fd87f7fe640 1 -- 172.21.15.29:0/1626856867 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd87800de80 con 0x7fd888101290 2024-03-11T19:09:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.944+0000 7fd890735640 1 -- 172.21.15.29:0/1626856867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd88819e1e0 con 0x7fd888101290 2024-03-11T19:09:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.944+0000 7fd87f7fe640 1 -- 172.21.15.29:0/1626856867 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd878004e00 con 0x7fd888101290 2024-03-11T19:09:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.944+0000 7fd890735640 1 -- 172.21.15.29:0/1626856867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd88819e6a0 con 0x7fd888101290 2024-03-11T19:09:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.946+0000 7fd87f7fe640 1 -- 172.21.15.29:0/1626856867 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd878004f60 con 0x7fd888101290 2024-03-11T19:09:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.946+0000 7fd8933df740 1 Processor -- start 2024-03-11T19:09:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd8933df740 1 -- start start 2024-03-11T19:09:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd8933df740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f0fd2c5190 0x55f0fd2c55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd87f7fe640 1 --2- 172.21.15.29:0/1626856867 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8740a1a30 0x7fd8740a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd8933df740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 0x55f0fd2ca680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd8933df740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2cacf0 0x55f0fd2cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd8933df740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0fd27ecd0 con 0x55f0fd2c5b80 2024-03-11T19:09:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd8933df740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0fd231f20 con 0x55f0fd2c5190 2024-03-11T19:09:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd8933df740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0fd22f450 con 0x55f0fd2cacf0 2024-03-11T19:09:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd88e4d1640 1 --2- 172.21.15.29:0/1626856867 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8740a1a30 0x7fd8740a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd88dcd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 0x55f0fd2ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd88dcd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 0x55f0fd2ca680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55760/0 (socket says 172.21.15.29:55760) 2024-03-11T19:09:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd88dcd0640 1 -- 172.21.15.29:0/4220497052 learned_addr learned my addr 172.21.15.29:0/4220497052 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd88ecd2640 1 --2- 172.21.15.29:0/4220497052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2cacf0 0x55f0fd2cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd87f7fe640 1 -- 172.21.15.29:0/1626856867 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd8780ccea0 con 0x7fd888101290 2024-03-11T19:09:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd88ecd2640 1 --2- 172.21.15.29:0/4220497052 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f0fd2c5190 0x55f0fd2c55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd88dcd0640 1 -- 172.21.15.29:0/4220497052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2cacf0 msgr2=0x55f0fd2cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd88dcd0640 1 --2- 172.21.15.29:0/4220497052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2cacf0 0x55f0fd2cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd88dcd0640 1 -- 172.21.15.29:0/4220497052 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f0fd2c5190 msgr2=0x55f0fd2c55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd88dcd0640 1 --2- 172.21.15.29:0/4220497052 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f0fd2c5190 0x55f0fd2c55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd88dcd0640 1 -- 172.21.15.29:0/4220497052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0fd1dff20 con 0x55f0fd2c5b80 2024-03-11T19:09:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd88e4d1640 1 --2- 172.21.15.29:0/1626856867 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8740a1a30 0x7fd8740a3ef0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd884036040 tx=0x7fd884005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.947+0000 7fd88dcd0640 1 --2- 172.21.15.29:0/4220497052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 0x55f0fd2ca680 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd8780a2120 tx=0x7fd87806c100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=822359b435215b10 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.948+0000 7fd87cff9640 1 -- 172.21.15.29:0/4220497052 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87809f520 con 0x55f0fd2c5b80 2024-03-11T19:09:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.948+0000 7fd87cff9640 1 -- 172.21.15.29:0/4220497052 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd87809ef20 con 0x55f0fd2c5b80 2024-03-11T19:09:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.948+0000 7fd87cff9640 1 -- 172.21.15.29:0/4220497052 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87809ecc0 con 0x55f0fd2c5b80 2024-03-11T19:09:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.948+0000 7fd8933df740 1 -- 172.21.15.29:0/4220497052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 msgr2=0x55f0fd2ca680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.948+0000 7fd8933df740 1 --2- 172.21.15.29:0/4220497052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 0x55f0fd2ca680 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd8780a2120 tx=0x7fd87806c100 comp rx=0 tx=0).stop 2024-03-11T19:09:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.948+0000 7fd8933df740 1 -- 172.21.15.29:0/4220497052 shutdown_connections 2024-03-11T19:09:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.948+0000 7fd8933df740 1 --2- 172.21.15.29:0/4220497052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2cacf0 0x55f0fd2cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.948+0000 7fd8933df740 1 --2- 172.21.15.29:0/4220497052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 0x55f0fd2ca680 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.948+0000 7fd8933df740 1 --2- 172.21.15.29:0/4220497052 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f0fd2c5190 0x55f0fd2c55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.948+0000 7fd8933df740 1 -- 172.21.15.29:0/4220497052 >> 172.21.15.29:0/4220497052 conn(0x55f0fd2c0700 msgr2=0x55f0fd2c0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd8933df740 1 -- 172.21.15.29:0/4220497052 shutdown_connections 2024-03-11T19:09:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd8933df740 1 -- 172.21.15.29:0/4220497052 wait complete. 2024-03-11T19:09:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd8933df740 1 -- start start 2024-03-11T19:09:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd8933df740 1 Processor -- start 2024-03-11T19:09:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd8933df740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2c5190 0x55f0fd2cf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd8933df740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 0x55f0fd2cf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd8933df740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f0fd2cacf0 0x55f0fd2d68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd8933df740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0fd22ac30 con 0x55f0fd2c5b80 2024-03-11T19:09:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd8933df740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0fd1cfd30 con 0x55f0fd2cacf0 2024-03-11T19:09:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd8933df740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0fcefe310 con 0x55f0fd2c5190 2024-03-11T19:09:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd8933df740 10 client.0 _ll_register_callbacks cb 0x55f0fd1d7d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd88dcd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 0x55f0fd2cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd88dcd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 0x55f0fd2cf880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55774/0 (socket says 172.21.15.29:55774) 2024-03-11T19:09:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd88dcd0640 1 -- 172.21.15.29:0/1572782748 learned_addr learned my addr 172.21.15.29:0/1572782748 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd88ecd2640 1 --2- 172.21.15.29:0/1572782748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f0fd2cacf0 0x55f0fd2d68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd88e4d1640 1 --2- 172.21.15.29:0/1572782748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2c5190 0x55f0fd2cf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd88dcd0640 1 -- 172.21.15.29:0/1572782748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2c5190 msgr2=0x55f0fd2cf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd88dcd0640 1 --2- 172.21.15.29:0/1572782748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2c5190 0x55f0fd2cf340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd88dcd0640 1 -- 172.21.15.29:0/1572782748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f0fd2cacf0 msgr2=0x55f0fd2d68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd88dcd0640 1 --2- 172.21.15.29:0/1572782748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f0fd2cacf0 0x55f0fd2d68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd88dcd0640 1 -- 172.21.15.29:0/1572782748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd88819e6a0 con 0x55f0fd2c5b80 2024-03-11T19:09:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.949+0000 7fd88e4d1640 1 --2- 172.21.15.29:0/1572782748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2c5190 0x55f0fd2cf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.950+0000 7fd88dcd0640 1 --2- 172.21.15.29:0/1572782748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 0x55f0fd2cf880 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd878099e40 tx=0x7fd878099eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed00e13bf585cfb server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.950+0000 7fd871ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.950+0000 7fd871ffb640 1 -- 172.21.15.29:0/1572782748 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8780a2650 con 0x55f0fd2c5b80 2024-03-11T19:09:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.950+0000 7fd871ffb640 1 -- 172.21.15.29:0/1572782748 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd87809ecc0 con 0x55f0fd2c5b80 2024-03-11T19:09:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.950+0000 7fd8933df740 1 -- 172.21.15.29:0/1572782748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f0fcf2a6c0 con 0x55f0fd2c5b80 2024-03-11T19:09:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.950+0000 7fd871ffb640 1 -- 172.21.15.29:0/1572782748 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd878009d90 con 0x55f0fd2c5b80 2024-03-11T19:09:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.950+0000 7fd8933df740 10 client.15114 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.950+0000 7fd8933df740 1 -- 172.21.15.29:0/1572782748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55f0fd2292b0 con 0x55f0fd2c5b80 2024-03-11T19:09:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.950+0000 7fd871ffb640 1 -- 172.21.15.29:0/1572782748 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd878065030 con 0x55f0fd2c5b80 2024-03-11T19:09:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.950+0000 7fd871ffb640 1 -- 172.21.15.29:0/1572782748 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd878096520 con 0x55f0fd2c5b80 2024-03-11T19:09:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.950+0000 7fd8933df740 10 client.15114 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.950+0000 7fd8933df740 10 client.15114 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-11T19:09:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.950+0000 7fd8933df740 20 client.15114 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.950+0000 7fd8933df740 1 --2- 172.21.15.29:0/1572782748 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f0fd2db1f0 0x55f0fd2dfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.950+0000 7fd8933df740 4 client.15114 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:09:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.950+0000 7fd8933df740 1 -- 172.21.15.29:0/1572782748 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f0fcf7b270 con 0x55f0fd2db1f0 2024-03-11T19:09:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.951+0000 7fd88e4d1640 1 --2- 172.21.15.29:0/1572782748 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f0fd2db1f0 0x55f0fd2dfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.951+0000 7fd88e4d1640 1 --2- 172.21.15.29:0/1572782748 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f0fd2db1f0 0x55f0fd2dfc70 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9a74b0e05c74bcb server_cookie=b98420443cb72cc9 in_seq=0 out_seq=0 2024-03-11T19:09:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.951+0000 7fd871ffb640 10 client.15114 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:09:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.952+0000 7fd871ffb640 1 -- 172.21.15.29:0/1572782748 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f0fcf7b270 con 0x55f0fd2db1f0 2024-03-11T19:09:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.952+0000 7fd871ffb640 10 client.15114 handle_command_reply: tid=0 2024-03-11T19:09:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.952+0000 7fd8933df740 1 client.15114 shutdown 2024-03-11T19:09:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.952+0000 7fd8933df740 1 -- 172.21.15.29:0/1572782748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 msgr2=0x55f0fd2cf880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.952+0000 7fd8933df740 1 --2- 172.21.15.29:0/1572782748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 0x55f0fd2cf880 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd878099e40 tx=0x7fd878099eb0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.952+0000 7fd8933df740 1 -- 172.21.15.29:0/1572782748 shutdown_connections 2024-03-11T19:09:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.952+0000 7fd8933df740 1 --2- 172.21.15.29:0/1572782748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f0fd2cacf0 0x55f0fd2d68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.952+0000 7fd8933df740 1 --2- 172.21.15.29:0/1572782748 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f0fd2db1f0 0x55f0fd2dfc70 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.952+0000 7fd8933df740 1 --2- 172.21.15.29:0/1572782748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 0x55f0fd2cf880 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.952+0000 7fd8933df740 1 --2- 172.21.15.29:0/1572782748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2c5190 0x55f0fd2cf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.952+0000 7fd871ffb640 0 client.15114 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:09:22.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.952+0000 7fd8933df740 1 -- 172.21.15.29:0/1572782748 >> 172.21.15.29:0/1572782748 conn(0x55f0fd2c0700 msgr2=0x55f0fd2bd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.952+0000 7fd8933df740 1 -- 172.21.15.29:0/1572782748 shutdown_connections 2024-03-11T19:09:22.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.952+0000 7fd8933df740 1 -- 172.21.15.29:0/1572782748 wait complete. 2024-03-11T19:09:22.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.952+0000 7fd8933df740 20 client.15114 trim_cache size 0 max 16384 2024-03-11T19:09:22.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd8933df740 1 Processor -- start 2024-03-11T19:09:22.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd8933df740 1 -- start start 2024-03-11T19:09:22.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd8933df740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2c0700 0x55f0fd2ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd8933df740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f0fd2c5190 0x55f0fd2d6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd8933df740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 0x55f0fd2ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd8933df740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0fd1b0490 con 0x55f0fd2c5b80 2024-03-11T19:09:22.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd8933df740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0fd2dba20 con 0x55f0fd2c5190 2024-03-11T19:09:22.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd8933df740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0fd2d7540 con 0x55f0fd2c0700 2024-03-11T19:09:22.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd88ecd2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 0x55f0fd2ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd88ecd2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 0x55f0fd2ceb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55780/0 (socket says 172.21.15.29:55780) 2024-03-11T19:09:22.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd88ecd2640 1 -- 172.21.15.29:0/3819329276 learned_addr learned my addr 172.21.15.29:0/3819329276 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd88ecd2640 1 -- 172.21.15.29:0/3819329276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2c0700 msgr2=0x55f0fd2ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd88e4d1640 1 --2- 172.21.15.29:0/3819329276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2c0700 0x55f0fd2ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd88ecd2640 1 --2- 172.21.15.29:0/3819329276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2c0700 0x55f0fd2ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd88ecd2640 1 -- 172.21.15.29:0/3819329276 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f0fd2c5190 msgr2=0x55f0fd2d6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd88ecd2640 1 --2- 172.21.15.29:0/3819329276 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f0fd2c5190 0x55f0fd2d6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd88ecd2640 1 -- 172.21.15.29:0/3819329276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0fd1c5a20 con 0x55f0fd2c5b80 2024-03-11T19:09:22.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd88e4d1640 1 --2- 172.21.15.29:0/3819329276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2c0700 0x55f0fd2ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:22.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd88ecd2640 1 --2- 172.21.15.29:0/3819329276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 0x55f0fd2ceb50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd8800142e0 tx=0x7fd880004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c884e0937557c34f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd873fff640 1 -- 172.21.15.29:0/3819329276 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd880004b20 con 0x55f0fd2c5b80 2024-03-11T19:09:22.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd873fff640 1 -- 172.21.15.29:0/3819329276 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd88001a070 con 0x55f0fd2c5b80 2024-03-11T19:09:22.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.968+0000 7fd873fff640 1 -- 172.21.15.29:0/3819329276 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd880005280 con 0x55f0fd2c5b80 2024-03-11T19:09:22.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.969+0000 7fd8933df740 1 -- 172.21.15.29:0/3819329276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 msgr2=0x55f0fd2ceb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.969+0000 7fd8933df740 1 --2- 172.21.15.29:0/3819329276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 0x55f0fd2ceb50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd8800142e0 tx=0x7fd880004ab0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.969+0000 7fd8933df740 1 -- 172.21.15.29:0/3819329276 shutdown_connections 2024-03-11T19:09:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.969+0000 7fd8933df740 1 --2- 172.21.15.29:0/3819329276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5b80 0x55f0fd2ceb50 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.969+0000 7fd8933df740 1 --2- 172.21.15.29:0/3819329276 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f0fd2c5190 0x55f0fd2d6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.969+0000 7fd8933df740 1 --2- 172.21.15.29:0/3819329276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2c0700 0x55f0fd2ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.969+0000 7fd8933df740 1 -- 172.21.15.29:0/3819329276 >> 172.21.15.29:0/3819329276 conn(0x55f0fd2c6960 msgr2=0x55f0fd2cf340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.969+0000 7fd8933df740 1 -- 172.21.15.29:0/3819329276 shutdown_connections 2024-03-11T19:09:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.969+0000 7fd8933df740 1 -- 172.21.15.29:0/3819329276 wait complete. 2024-03-11T19:09:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.969+0000 7fd8933df740 1 -- start start 2024-03-11T19:09:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.969+0000 7fd8933df740 1 Processor -- start 2024-03-11T19:09:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.969+0000 7fd8933df740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f0fd2c0700 0x55f0fd2d3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.969+0000 7fd8933df740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5190 0x55f0fd2d4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.969+0000 7fd8933df740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2c5b80 0x55f0fd331930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.969+0000 7fd8933df740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0fd2d4740 con 0x55f0fd2c5190 2024-03-11T19:09:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.969+0000 7fd8933df740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0fd331e70 con 0x55f0fd2c0700 2024-03-11T19:09:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.969+0000 7fd8933df740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0fd331fb0 con 0x55f0fd2c5b80 2024-03-11T19:09:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd8933df740 10 client.0 _ll_register_callbacks cb 0x7fd878004840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd88dcd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5190 0x55f0fd2d4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd88dcd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5190 0x55f0fd2d4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55786/0 (socket says 172.21.15.29:55786) 2024-03-11T19:09:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd88dcd0640 1 -- 172.21.15.29:0/1889361461 learned_addr learned my addr 172.21.15.29:0/1889361461 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd88e4d1640 1 --2- 172.21.15.29:0/1889361461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f0fd2c0700 0x55f0fd2d3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd88ecd2640 1 --2- 172.21.15.29:0/1889361461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2c5b80 0x55f0fd331930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd88dcd0640 1 -- 172.21.15.29:0/1889361461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2c5b80 msgr2=0x55f0fd331930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd88dcd0640 1 --2- 172.21.15.29:0/1889361461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2c5b80 0x55f0fd331930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd88dcd0640 1 -- 172.21.15.29:0/1889361461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f0fd2c0700 msgr2=0x55f0fd2d3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd88dcd0640 1 --2- 172.21.15.29:0/1889361461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f0fd2c0700 0x55f0fd2d3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd88dcd0640 1 -- 172.21.15.29:0/1889361461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd880014040 con 0x55f0fd2c5190 2024-03-11T19:09:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd88dcd0640 1 --2- 172.21.15.29:0/1889361461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5190 0x55f0fd2d4040 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd878098160 tx=0x7fd87809f780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e55c633e9654a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd871ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd871ffb640 1 -- 172.21.15.29:0/1889361461 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd878099810 con 0x55f0fd2c5190 2024-03-11T19:09:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd871ffb640 1 -- 172.21.15.29:0/1889361461 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd8780981b0 con 0x55f0fd2c5190 2024-03-11T19:09:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd8933df740 1 -- 172.21.15.29:0/1889361461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f0fd2dbb60 con 0x55f0fd2c5190 2024-03-11T19:09:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd871ffb640 1 -- 172.21.15.29:0/1889361461 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd87809e000 con 0x55f0fd2c5190 2024-03-11T19:09:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd8933df740 10 client.15120 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd8933df740 1 -- 172.21.15.29:0/1889361461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55f0fd1eb780 con 0x55f0fd2c5190 2024-03-11T19:09:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.970+0000 7fd871ffb640 1 -- 172.21.15.29:0/1889361461 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f0fd2dbb60 con 0x55f0fd2c5190 2024-03-11T19:09:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.971+0000 7fd871ffb640 1 -- 172.21.15.29:0/1889361461 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd8780b81a0 con 0x55f0fd2c5190 2024-03-11T19:09:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.971+0000 7fd8933df740 10 client.15120 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.971+0000 7fd8933df740 10 client.15120 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-11T19:09:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.971+0000 7fd8933df740 20 client.15120 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.971+0000 7fd8933df740 1 --2- 172.21.15.29:0/1889361461 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f0fd2cacf0 0x55f0fd333240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.971+0000 7fd8933df740 4 client.15120 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:09:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.971+0000 7fd8933df740 1 -- 172.21.15.29:0/1889361461 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fd87809ef20 con 0x55f0fd2cacf0 2024-03-11T19:09:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.971+0000 7fd88e4d1640 1 --2- 172.21.15.29:0/1889361461 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f0fd2cacf0 0x55f0fd333240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.971+0000 7fd88e4d1640 1 --2- 172.21.15.29:0/1889361461 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f0fd2cacf0 0x55f0fd333240 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=944a327cd54af2c9 server_cookie=d6b0e10f924f69b0 in_seq=0 out_seq=0 2024-03-11T19:09:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.971+0000 7fd871ffb640 10 client.15120 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:09:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.971+0000 7fd871ffb640 1 -- 172.21.15.29:0/1889361461 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x7fd87809ef20 con 0x55f0fd2cacf0 2024-03-11T19:09:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.971+0000 7fd871ffb640 10 client.15120 handle_command_reply: tid=0 2024-03-11T19:09:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.971+0000 7fd8933df740 1 client.15120 shutdown 2024-03-11T19:09:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd8933df740 1 -- 172.21.15.29:0/1889361461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5190 msgr2=0x55f0fd2d4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd8933df740 1 --2- 172.21.15.29:0/1889361461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5190 0x55f0fd2d4040 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd878098160 tx=0x7fd87809f780 comp rx=0 tx=0).stop 2024-03-11T19:09:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd8933df740 1 -- 172.21.15.29:0/1889361461 shutdown_connections 2024-03-11T19:09:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd8933df740 1 --2- 172.21.15.29:0/1889361461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0fd2c5b80 0x55f0fd331930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd8933df740 1 --2- 172.21.15.29:0/1889361461 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f0fd2cacf0 0x55f0fd333240 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd8933df740 1 --2- 172.21.15.29:0/1889361461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f0fd2c5190 0x55f0fd2d4040 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd871ffb640 0 client.15120 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:09:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd8933df740 1 --2- 172.21.15.29:0/1889361461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f0fd2c0700 0x55f0fd2d3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd8933df740 1 -- 172.21.15.29:0/1889361461 >> 172.21.15.29:0/1889361461 conn(0x55f0fd2c6960 msgr2=0x55f0fd2ce610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd8933df740 1 -- 172.21.15.29:0/1889361461 shutdown_connections 2024-03-11T19:09:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd8933df740 1 -- 172.21.15.29:0/1889361461 wait complete. 2024-03-11T19:09:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd8933df740 20 client.15120 trim_cache size 0 max 16384 2024-03-11T19:09:22.982 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:09:22.982 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "638915c2-dfd9-11ee-95c8-87774f69a715", 2024-03-11T19:09:22.982 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2024-03-11T19:09:22.982 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24506, 2024-03-11T19:09:22.982 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-03-11T19:09:22.983 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-03-11T19:09:22.983 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-03-11T19:09:22.983 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 93.237557100999993, 2024-03-11T19:09:22.983 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 26, 2024-03-11T19:09:22.983 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 89, 2024-03-11T19:09:22.983 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 87, 2024-03-11T19:09:22.983 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 97.391297182000002 2024-03-11T19:09:22.983 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:09:22.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd890735640 1 -- 172.21.15.29:0/1626856867 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8740a1a30 msgr2=0x7fd8740a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd890735640 1 --2- 172.21.15.29:0/1626856867 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8740a1a30 0x7fd8740a3ef0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd884036040 tx=0x7fd884005fd0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd890735640 1 -- 172.21.15.29:0/1626856867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd888101290 msgr2=0x7fd888196620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:22.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd890735640 1 --2- 172.21.15.29:0/1626856867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd888101290 0x7fd888196620 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd87800d770 tx=0x7fd87800dc30 comp rx=0 tx=0).stop 2024-03-11T19:09:22.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd890735640 1 -- 172.21.15.29:0/1626856867 shutdown_connections 2024-03-11T19:09:22.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd890735640 1 --2- 172.21.15.29:0/1626856867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd888101c50 0x7fd88819d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd890735640 1 --2- 172.21.15.29:0/1626856867 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8740a1a30 0x7fd8740a3ef0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd890735640 1 --2- 172.21.15.29:0/1626856867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd888101290 0x7fd888196620 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd890735640 1 --2- 172.21.15.29:0/1626856867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8881000a0 0x7fd8881960e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:22.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd890735640 1 -- 172.21.15.29:0/1626856867 >> 172.21.15.29:0/1626856867 conn(0x7fd8880fb630 msgr2=0x7fd8881046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd890735640 1 -- 172.21.15.29:0/1626856867 shutdown_connections 2024-03-11T19:09:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:22.972+0000 7fd890735640 1 -- 172.21.15.29:0/1626856867 wait complete. 2024-03-11T19:09:22.986 INFO:tasks.cephadm:Setup complete, yielding 2024-03-11T19:09:22.986 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T19:09:22.996 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-03-11T19:09:22.997 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch status' 2024-03-11T19:09:23.240 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:09:23.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:23 smithi029 ceph-mon[32439]: pgmap v338: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-11T19:09:23.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:23 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3160339995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:09:23.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:23 smithi190 ceph-mon[36292]: pgmap v338: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-11T19:09:23.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:23 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3160339995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:09:23.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:23 smithi170 ceph-mon[35690]: pgmap v338: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-11T19:09:23.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:23 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3160339995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:09:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.180+0000 7f7ff985f700 1 -- 172.21.15.29:0/910049695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ff4101ca0 msgr2=0x7f7ff41020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.180+0000 7f7ff985f700 1 --2- 172.21.15.29:0/910049695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ff4101ca0 0x7f7ff41020c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7fe40099c0 tx=0x7f7fe4009cd0 comp rx=0 tx=0).stop 2024-03-11T19:09:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.180+0000 7f7ff985f700 1 -- 172.21.15.29:0/910049695 shutdown_connections 2024-03-11T19:09:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.180+0000 7f7ff985f700 1 --2- 172.21.15.29:0/910049695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ff4103840 0x7f7ff410a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.180+0000 7f7ff985f700 1 --2- 172.21.15.29:0/910049695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ff4101ca0 0x7f7ff41020c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.180+0000 7f7ff985f700 1 --2- 172.21.15.29:0/910049695 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ff4102ea0 0x7f7ff4103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.180+0000 7f7ff985f700 1 -- 172.21.15.29:0/910049695 >> 172.21.15.29:0/910049695 conn(0x7f7ff40fd280 msgr2=0x7f7ff40ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.180+0000 7f7ff985f700 1 -- 172.21.15.29:0/910049695 shutdown_connections 2024-03-11T19:09:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.180+0000 7f7ff985f700 1 -- 172.21.15.29:0/910049695 wait complete. 2024-03-11T19:09:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.181+0000 7f7ff985f700 1 Processor -- start 2024-03-11T19:09:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.181+0000 7f7ff985f700 1 -- start start 2024-03-11T19:09:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.181+0000 7f7ff985f700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ff4101ca0 0x7f7ff4197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.181+0000 7f7ff985f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ff4102ea0 0x7f7ff41980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.181+0000 7f7ff985f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ff4103840 0x7f7ff419f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.181+0000 7f7ff985f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff419f740 con 0x7f7ff4103840 2024-03-11T19:09:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.181+0000 7f7ff985f700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff419f880 con 0x7f7ff4101ca0 2024-03-11T19:09:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.181+0000 7f7ff985f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff419fa10 con 0x7f7ff4102ea0 2024-03-11T19:09:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.181+0000 7f7ff37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ff4103840 0x7f7ff419f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.181+0000 7f7ff37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ff4103840 0x7f7ff419f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55796/0 (socket says 172.21.15.29:55796) 2024-03-11T19:09:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.182+0000 7f7ff37fe700 1 -- 172.21.15.29:0/2632686337 learned_addr learned my addr 172.21.15.29:0/2632686337 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.182+0000 7f7ff27fc700 1 --2- 172.21.15.29:0/2632686337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ff4102ea0 0x7f7ff41980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.182+0000 7f7ff2ffd700 1 --2- 172.21.15.29:0/2632686337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ff4101ca0 0x7f7ff4197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.182+0000 7f7ff37fe700 1 -- 172.21.15.29:0/2632686337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ff4102ea0 msgr2=0x7f7ff41980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.182+0000 7f7ff37fe700 1 --2- 172.21.15.29:0/2632686337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ff4102ea0 0x7f7ff41980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.182+0000 7f7ff37fe700 1 -- 172.21.15.29:0/2632686337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ff4101ca0 msgr2=0x7f7ff4197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.182+0000 7f7ff37fe700 1 --2- 172.21.15.29:0/2632686337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ff4101ca0 0x7f7ff4197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.182+0000 7f7ff37fe700 1 -- 172.21.15.29:0/2632686337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe40095f0 con 0x7f7ff4103840 2024-03-11T19:09:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.182+0000 7f7ff37fe700 1 --2- 172.21.15.29:0/2632686337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ff4103840 0x7f7ff419f170 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7fec007f10 tx=0x7f7fec00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.182+0000 7f7ff885d700 1 -- 172.21.15.29:0/2632686337 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec017070 con 0x7f7ff4103840 2024-03-11T19:09:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.182+0000 7f7ff885d700 1 -- 172.21.15.29:0/2632686337 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7fec00f040 con 0x7f7ff4103840 2024-03-11T19:09:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.182+0000 7f7ff985f700 1 -- 172.21.15.29:0/2632686337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff419fcf0 con 0x7f7ff4103840 2024-03-11T19:09:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.182+0000 7f7ff885d700 1 -- 172.21.15.29:0/2632686337 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec005730 con 0x7f7ff4103840 2024-03-11T19:09:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.182+0000 7f7ff985f700 1 -- 172.21.15.29:0/2632686337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff41a01a0 con 0x7f7ff4103840 2024-03-11T19:09:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.183+0000 7f7ff985f700 1 -- 172.21.15.29:0/2632686337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff4109180 con 0x7f7ff4103840 2024-03-11T19:09:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.185+0000 7f7ff885d700 1 -- 172.21.15.29:0/2632686337 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7fec005890 con 0x7f7ff4103840 2024-03-11T19:09:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.186+0000 7f7ff885d700 1 --2- 172.21.15.29:0/2632686337 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7fdc0a1ac0 0x7f7fdc0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.186+0000 7f7ff2ffd700 1 --2- 172.21.15.29:0/2632686337 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7fdc0a1ac0 0x7f7fdc0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.186+0000 7f7ff885d700 1 -- 172.21.15.29:0/2632686337 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7fec0cd280 con 0x7f7ff4103840 2024-03-11T19:09:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.186+0000 7f7ff2ffd700 1 --2- 172.21.15.29:0/2632686337 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7fdc0a1ac0 0x7f7fdc0a3f80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4000c00 tx=0x7f7fe4005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.189+0000 7f7ff885d700 1 -- 172.21.15.29:0/2632686337 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7fec096490 con 0x7f7ff4103840 2024-03-11T19:09:25.398 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:25 smithi170 ceph-mon[35690]: pgmap v339: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-11T19:09:25.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:25 smithi029 ceph-mon[32439]: pgmap v339: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-11T19:09:25.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.414+0000 7f7ff985f700 1 -- 172.21.15.29:0/2632686337 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ff402d8e0 con 0x7f7fdc0a1ac0 2024-03-11T19:09:25.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.415+0000 7f7ff885d700 1 -- 172.21.15.29:0/2632686337 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f7ff402d8e0 con 0x7f7fdc0a1ac0 2024-03-11T19:09:25.418 INFO:teuthology.orchestra.run.smithi029.stdout:Backend: cephadm 2024-03-11T19:09:25.418 INFO:teuthology.orchestra.run.smithi029.stdout:Available: Yes 2024-03-11T19:09:25.418 INFO:teuthology.orchestra.run.smithi029.stdout:Paused: No 2024-03-11T19:09:25.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.418+0000 7f7ff985f700 1 -- 172.21.15.29:0/2632686337 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7fdc0a1ac0 msgr2=0x7f7fdc0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:25.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.418+0000 7f7ff985f700 1 --2- 172.21.15.29:0/2632686337 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7fdc0a1ac0 0x7f7fdc0a3f80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4000c00 tx=0x7f7fe4005fd0 comp rx=0 tx=0).stop 2024-03-11T19:09:25.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.418+0000 7f7ff985f700 1 -- 172.21.15.29:0/2632686337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ff4103840 msgr2=0x7f7ff419f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:25.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.418+0000 7f7ff985f700 1 --2- 172.21.15.29:0/2632686337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ff4103840 0x7f7ff419f170 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7fec007f10 tx=0x7f7fec00d3b0 comp rx=0 tx=0).stop 2024-03-11T19:09:25.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.418+0000 7f7ff985f700 1 -- 172.21.15.29:0/2632686337 shutdown_connections 2024-03-11T19:09:25.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.418+0000 7f7ff985f700 1 --2- 172.21.15.29:0/2632686337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ff4103840 0x7f7ff419f170 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:25.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.418+0000 7f7ff985f700 1 --2- 172.21.15.29:0/2632686337 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7fdc0a1ac0 0x7f7fdc0a3f80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:25.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.418+0000 7f7ff985f700 1 --2- 172.21.15.29:0/2632686337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ff4101ca0 0x7f7ff4197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:25.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.418+0000 7f7ff985f700 1 --2- 172.21.15.29:0/2632686337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ff4102ea0 0x7f7ff41980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:25.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.418+0000 7f7ff985f700 1 -- 172.21.15.29:0/2632686337 >> 172.21.15.29:0/2632686337 conn(0x7f7ff40fd280 msgr2=0x7f7ff41062c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:25.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.418+0000 7f7ff985f700 1 -- 172.21.15.29:0/2632686337 shutdown_connections 2024-03-11T19:09:25.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:25.418+0000 7f7ff985f700 1 -- 172.21.15.29:0/2632686337 wait complete. 2024-03-11T19:09:25.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:25 smithi190 ceph-mon[36292]: pgmap v339: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-11T19:09:26.165 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-11T19:09:26.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:26 smithi029 ceph-mon[32439]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:09:26.412 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:09:26.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:26 smithi190 ceph-mon[36292]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:09:26.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:26 smithi170 ceph-mon[35690]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:09:27.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:27 smithi029 ceph-mon[32439]: pgmap v340: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-11T19:09:27.413 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:27 smithi190 ceph-mon[36292]: pgmap v340: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-11T19:09:27.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:27 smithi170 ceph-mon[35690]: pgmap v340: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-11T19:09:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.239+0000 7fa4dffdf700 1 -- 172.21.15.29:0/4270334846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4d8103840 msgr2=0x7fa4d810a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.239+0000 7fa4dffdf700 1 --2- 172.21.15.29:0/4270334846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4d8103840 0x7fa4d810a0f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa4d000b0d0 tx=0x7fa4d000b3e0 comp rx=0 tx=0).stop 2024-03-11T19:09:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.239+0000 7fa4dffdf700 1 -- 172.21.15.29:0/4270334846 shutdown_connections 2024-03-11T19:09:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.239+0000 7fa4dffdf700 1 --2- 172.21.15.29:0/4270334846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4d8103840 0x7fa4d810a0f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.239+0000 7fa4dffdf700 1 --2- 172.21.15.29:0/4270334846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4d8101ca0 0x7fa4d81020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.239+0000 7fa4dffdf700 1 --2- 172.21.15.29:0/4270334846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d8102ea0 0x7fa4d8103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.239+0000 7fa4dffdf700 1 -- 172.21.15.29:0/4270334846 >> 172.21.15.29:0/4270334846 conn(0x7fa4d80fd280 msgr2=0x7fa4d80ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.239+0000 7fa4dffdf700 1 -- 172.21.15.29:0/4270334846 shutdown_connections 2024-03-11T19:09:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.239+0000 7fa4dffdf700 1 -- 172.21.15.29:0/4270334846 wait complete. 2024-03-11T19:09:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4dffdf700 1 Processor -- start 2024-03-11T19:09:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4dffdf700 1 -- start start 2024-03-11T19:09:28.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4dffdf700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d8101ca0 0x7fa4d8197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:28.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4dffdf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4d8102ea0 0x7fa4d8198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:28.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4dffdf700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4d8103840 0x7fa4d819f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:28.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4dffdf700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d819f830 con 0x7fa4d8102ea0 2024-03-11T19:09:28.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4dffdf700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d819f970 con 0x7fa4d8103840 2024-03-11T19:09:28.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4dffdf700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d819fb30 con 0x7fa4d8101ca0 2024-03-11T19:09:28.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4dd57a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4d8102ea0 0x7fa4d8198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:28.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4dd57a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4d8102ea0 0x7fa4d8198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59668/0 (socket says 172.21.15.29:59668) 2024-03-11T19:09:28.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4dd57a700 1 -- 172.21.15.29:0/1843459598 learned_addr learned my addr 172.21.15.29:0/1843459598 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:28.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4dd57a700 1 -- 172.21.15.29:0/1843459598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d8101ca0 msgr2=0x7fa4d8197ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:28.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4ddd7b700 1 --2- 172.21.15.29:0/1843459598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d8101ca0 0x7fa4d8197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:28.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4de57c700 1 --2- 172.21.15.29:0/1843459598 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4d8103840 0x7fa4d819f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4dd57a700 1 --2- 172.21.15.29:0/1843459598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d8101ca0 0x7fa4d8197ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4dd57a700 1 -- 172.21.15.29:0/1843459598 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4d8103840 msgr2=0x7fa4d819f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4dd57a700 1 --2- 172.21.15.29:0/1843459598 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4d8103840 0x7fa4d819f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4dd57a700 1 -- 172.21.15.29:0/1843459598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d0009d00 con 0x7fa4d8102ea0 2024-03-11T19:09:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4ddd7b700 1 --2- 172.21.15.29:0/1843459598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d8101ca0 0x7fa4d8197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4de57c700 1 --2- 172.21.15.29:0/1843459598 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4d8103840 0x7fa4d819f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.240+0000 7fa4dd57a700 1 --2- 172.21.15.29:0/1843459598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4d8102ea0 0x7fa4d8198220 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa4c800d950 tx=0x7fa4c800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.241+0000 7fa4ceffd700 1 -- 172.21.15.29:0/1843459598 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c8009b40 con 0x7fa4d8102ea0 2024-03-11T19:09:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.241+0000 7fa4dffdf700 1 -- 172.21.15.29:0/1843459598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d819fe10 con 0x7fa4d8102ea0 2024-03-11T19:09:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.241+0000 7fa4ceffd700 1 -- 172.21.15.29:0/1843459598 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa4c8009ca0 con 0x7fa4d8102ea0 2024-03-11T19:09:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.241+0000 7fa4ceffd700 1 -- 172.21.15.29:0/1843459598 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c800f620 con 0x7fa4d8102ea0 2024-03-11T19:09:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.241+0000 7fa4dffdf700 1 -- 172.21.15.29:0/1843459598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d81a02b0 con 0x7fa4d8102ea0 2024-03-11T19:09:28.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.243+0000 7fa4dffdf700 1 -- 172.21.15.29:0/1843459598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d81090e0 con 0x7fa4d8102ea0 2024-03-11T19:09:28.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.243+0000 7fa4ceffd700 1 -- 172.21.15.29:0/1843459598 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fa4c8010460 con 0x7fa4d8102ea0 2024-03-11T19:09:28.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.244+0000 7fa4ceffd700 1 --2- 172.21.15.29:0/1843459598 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa4c40a1ac0 0x7fa4c40a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:28.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.244+0000 7fa4ddd7b700 1 --2- 172.21.15.29:0/1843459598 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa4c40a1ac0 0x7fa4c40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:28.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.244+0000 7fa4ceffd700 1 -- 172.21.15.29:0/1843459598 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fa4c80cd6c0 con 0x7fa4d8102ea0 2024-03-11T19:09:28.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.244+0000 7fa4ddd7b700 1 --2- 172.21.15.29:0/1843459598 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa4c40a1ac0 0x7fa4c40a3f80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa4d81991c0 tx=0x7fa4d4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:28.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.248+0000 7fa4ceffd700 1 -- 172.21.15.29:0/1843459598 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa4c8096970 con 0x7fa4d8102ea0 2024-03-11T19:09:28.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.460+0000 7fa4dffdf700 1 -- 172.21.15.29:0/1843459598 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4d802d870 con 0x7fa4c40a1ac0 2024-03-11T19:09:28.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.474+0000 7fa4ceffd700 1 -- 172.21.15.29:0/1843459598 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7fa4d802d870 con 0x7fa4c40a1ac0 2024-03-11T19:09:28.477 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T19:09:28.477 INFO:teuthology.orchestra.run.smithi029.stdout:mds.a smithi029 running (106s) 19s ago 105s 16.2M - 19.0.0-1607-g94a53970 a6477a474d71 bc90ce3586d6 2024-03-11T19:09:28.477 INFO:teuthology.orchestra.run.smithi029.stdout:mds.b smithi170 running (103s) 36s ago 102s 28.7M - 19.0.0-1607-g94a53970 a6477a474d71 32ee158abb06 2024-03-11T19:09:28.477 INFO:teuthology.orchestra.run.smithi029.stdout:mds.c smithi190 running (117s) 36s ago 117s 18.7M - 19.0.0-1607-g94a53970 a6477a474d71 656094af5d26 2024-03-11T19:09:28.477 INFO:teuthology.orchestra.run.smithi029.stdout:mds.d smithi029 running (2m) 19s ago 2m 16.2M - 19.0.0-1607-g94a53970 a6477a474d71 3e10f89d8bfd 2024-03-11T19:09:28.477 INFO:teuthology.orchestra.run.smithi029.stdout:mds.e smithi170 running (2m) 36s ago 2m 17.8M - 19.0.0-1607-g94a53970 a6477a474d71 ce0f0e897c09 2024-03-11T19:09:28.477 INFO:teuthology.orchestra.run.smithi029.stdout:mds.f smithi190 running (2m) 36s ago 119s 18.7M - 19.0.0-1607-g94a53970 a6477a474d71 5cb8dff62837 2024-03-11T19:09:28.477 INFO:teuthology.orchestra.run.smithi029.stdout:mds.g smithi029 running (109s) 19s ago 108s 17.0M - 19.0.0-1607-g94a53970 a6477a474d71 f31bf8334228 2024-03-11T19:09:28.477 INFO:teuthology.orchestra.run.smithi029.stdout:mds.h smithi170 running (114s) 36s ago 114s 17.2M - 19.0.0-1607-g94a53970 a6477a474d71 9a47d71e1dc3 2024-03-11T19:09:28.478 INFO:teuthology.orchestra.run.smithi029.stdout:mds.i smithi190 running (112s) 36s ago 111s 16.9M - 19.0.0-1607-g94a53970 a6477a474d71 b94348ec9f1e 2024-03-11T19:09:28.478 INFO:teuthology.orchestra.run.smithi029.stdout:mds.j smithi029 running (2m) 19s ago 2m 15.3M - 19.0.0-1607-g94a53970 a6477a474d71 bbbf59d44f57 2024-03-11T19:09:28.478 INFO:teuthology.orchestra.run.smithi029.stdout:mds.k smithi170 running (2m) 36s ago 2m 16.5M - 19.0.0-1607-g94a53970 a6477a474d71 f35ef38595e5 2024-03-11T19:09:28.478 INFO:teuthology.orchestra.run.smithi029.stdout:mds.l smithi190 running (2m) 36s ago 2m 19.4M - 19.0.0-1607-g94a53970 a6477a474d71 abe8b2b6897f 2024-03-11T19:09:28.478 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi029 *:9283,8765 running (9m) 19s ago 9m 512M - 19.0.0-1607-g94a53970 a6477a474d71 f43e9b0478f2 2024-03-11T19:09:28.478 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi170 *:8443,8765 running (7m) 36s ago 7m 454M - 19.0.0-1607-g94a53970 a6477a474d71 1172ef466c1c 2024-03-11T19:09:28.478 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.z smithi190 *:8443,8765 running (7m) 36s ago 7m 451M - 19.0.0-1607-g94a53970 a6477a474d71 8b60ab5ea912 2024-03-11T19:09:28.478 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (9m) 19s ago 9m 50.4M 2048M 19.0.0-1607-g94a53970 a6477a474d71 3d682d103af0 2024-03-11T19:09:28.478 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi170 running (7m) 36s ago 7m 37.0M 2048M 19.0.0-1607-g94a53970 a6477a474d71 c756922f8741 2024-03-11T19:09:28.478 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi190 running (7m) 36s ago 7m 39.8M 2048M 19.0.0-1607-g94a53970 a6477a474d71 632c5107f599 2024-03-11T19:09:28.478 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (7m) 19s ago 7m 56.6M 4273M 19.0.0-1607-g94a53970 a6477a474d71 123983e8ccce 2024-03-11T19:09:28.478 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi170 running (6m) 36s ago 6m 45.4M 4272M 19.0.0-1607-g94a53970 a6477a474d71 1154df2ddc29 2024-03-11T19:09:28.478 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi190 running (6m) 36s ago 6m 59.0M 4272M 19.0.0-1607-g94a53970 a6477a474d71 88797bf6b4f5 2024-03-11T19:09:28.479 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (6m) 19s ago 6m 55.6M 4273M 19.0.0-1607-g94a53970 a6477a474d71 f658c458ef86 2024-03-11T19:09:28.479 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi170 running (5m) 36s ago 5m 52.2M 4272M 19.0.0-1607-g94a53970 a6477a474d71 408bb11dcf7a 2024-03-11T19:09:28.479 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi190 running (5m) 36s ago 5m 53.7M 4272M 19.0.0-1607-g94a53970 a6477a474d71 95cb041a90b7 2024-03-11T19:09:28.479 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi029 running (4m) 19s ago 4m 56.7M 4273M 19.0.0-1607-g94a53970 a6477a474d71 4752209d7335 2024-03-11T19:09:28.479 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi170 running (4m) 36s ago 4m 54.5M 4272M 19.0.0-1607-g94a53970 a6477a474d71 328c46bb7033 2024-03-11T19:09:28.479 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8 smithi190 running (3m) 36s ago 3m 46.0M 4272M 19.0.0-1607-g94a53970 a6477a474d71 caeb4ff24047 2024-03-11T19:09:28.479 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9 smithi029 running (3m) 19s ago 3m 41.4M 4273M 19.0.0-1607-g94a53970 a6477a474d71 b37105f74307 2024-03-11T19:09:28.479 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10 smithi170 running (3m) 36s ago 3m 43.4M 4272M 19.0.0-1607-g94a53970 a6477a474d71 7f69282935bd 2024-03-11T19:09:28.479 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11 smithi190 running (2m) 36s ago 2m 51.7M 4272M 19.0.0-1607-g94a53970 a6477a474d71 609c688b2b95 2024-03-11T19:09:28.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.476+0000 7fa4dffdf700 1 -- 172.21.15.29:0/1843459598 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa4c40a1ac0 msgr2=0x7fa4c40a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:28.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.476+0000 7fa4dffdf700 1 --2- 172.21.15.29:0/1843459598 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa4c40a1ac0 0x7fa4c40a3f80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa4d81991c0 tx=0x7fa4d4008040 comp rx=0 tx=0).stop 2024-03-11T19:09:28.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.476+0000 7fa4dffdf700 1 -- 172.21.15.29:0/1843459598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4d8102ea0 msgr2=0x7fa4d8198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:28.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.476+0000 7fa4dffdf700 1 --2- 172.21.15.29:0/1843459598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4d8102ea0 0x7fa4d8198220 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa4c800d950 tx=0x7fa4c800dd10 comp rx=0 tx=0).stop 2024-03-11T19:09:28.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.476+0000 7fa4dffdf700 1 -- 172.21.15.29:0/1843459598 shutdown_connections 2024-03-11T19:09:28.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.476+0000 7fa4dffdf700 1 --2- 172.21.15.29:0/1843459598 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa4c40a1ac0 0x7fa4c40a3f80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:28.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.476+0000 7fa4dffdf700 1 --2- 172.21.15.29:0/1843459598 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4d8103840 0x7fa4d819f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:28.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.476+0000 7fa4dffdf700 1 --2- 172.21.15.29:0/1843459598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4d8101ca0 0x7fa4d8197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:28.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.476+0000 7fa4dffdf700 1 --2- 172.21.15.29:0/1843459598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4d8102ea0 0x7fa4d8198220 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:28.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.476+0000 7fa4dffdf700 1 -- 172.21.15.29:0/1843459598 >> 172.21.15.29:0/1843459598 conn(0x7fa4d80fd280 msgr2=0x7fa4d8106220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:28.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.476+0000 7fa4dffdf700 1 -- 172.21.15.29:0/1843459598 shutdown_connections 2024-03-11T19:09:28.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:28.476+0000 7fa4dffdf700 1 -- 172.21.15.29:0/1843459598 wait complete. 2024-03-11T19:09:29.197 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-11T19:09:29.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:29 smithi029 ceph-mon[32439]: pgmap v341: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-11T19:09:29.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:29 smithi029 ceph-mon[32439]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:09:29.439 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:09:29.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:29 smithi190 ceph-mon[36292]: pgmap v341: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-11T19:09:29.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:29 smithi190 ceph-mon[36292]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:09:29.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:29 smithi170 ceph-mon[35690]: pgmap v341: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-11T19:09:29.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:29 smithi170 ceph-mon[35690]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:09:31.256 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:31 smithi029 ceph-mon[32439]: pgmap v342: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-11T19:09:31.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.253+0000 7f518830e700 1 -- 172.21.15.29:0/2681978564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5180100ba0 msgr2=0x7f5180100fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:31.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.253+0000 7f518830e700 1 --2- 172.21.15.29:0/2681978564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5180100ba0 0x7f5180100fc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f517800b0d0 tx=0x7f517800b3e0 comp rx=0 tx=0).stop 2024-03-11T19:09:31.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.253+0000 7f518830e700 1 -- 172.21.15.29:0/2681978564 shutdown_connections 2024-03-11T19:09:31.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.253+0000 7f518830e700 1 --2- 172.21.15.29:0/2681978564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5180102740 0x7f5180108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:31.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.253+0000 7f518830e700 1 --2- 172.21.15.29:0/2681978564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5180100ba0 0x7f5180100fc0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:31.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.253+0000 7f518830e700 1 --2- 172.21.15.29:0/2681978564 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5180101da0 0x7f5180102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:31.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.253+0000 7f518830e700 1 -- 172.21.15.29:0/2681978564 >> 172.21.15.29:0/2681978564 conn(0x7f51800fc140 msgr2=0x7f51800fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:31.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.253+0000 7f518830e700 1 -- 172.21.15.29:0/2681978564 shutdown_connections 2024-03-11T19:09:31.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.253+0000 7f518830e700 1 -- 172.21.15.29:0/2681978564 wait complete. 2024-03-11T19:09:31.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.254+0000 7f518830e700 1 Processor -- start 2024-03-11T19:09:31.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.254+0000 7f518830e700 1 -- start start 2024-03-11T19:09:31.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.254+0000 7f518830e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5180100ba0 0x7f518006a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:31.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.254+0000 7f518830e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5180101da0 0x7f518019ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:31.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.254+0000 7f518830e700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5180102740 0x7f518019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:31.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.254+0000 7f518830e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518019f810 con 0x7f5180100ba0 2024-03-11T19:09:31.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.254+0000 7f518830e700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518019f980 con 0x7f5180102740 2024-03-11T19:09:31.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.254+0000 7f518830e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518019fb40 con 0x7f5180101da0 2024-03-11T19:09:31.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.254+0000 7f51860aa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5180100ba0 0x7f518006a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:31.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.254+0000 7f51860aa700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5180100ba0 0x7f518006a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59688/0 (socket says 172.21.15.29:59688) 2024-03-11T19:09:31.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.254+0000 7f51860aa700 1 -- 172.21.15.29:0/750391190 learned_addr learned my addr 172.21.15.29:0/750391190 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:31.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.254+0000 7f51858a9700 1 --2- 172.21.15.29:0/750391190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5180101da0 0x7f518019ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:31.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.254+0000 7f51868ab700 1 --2- 172.21.15.29:0/750391190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5180102740 0x7f518019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:31.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.254+0000 7f51860aa700 1 -- 172.21.15.29:0/750391190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5180101da0 msgr2=0x7f518019ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:31.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.255+0000 7f51860aa700 1 --2- 172.21.15.29:0/750391190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5180101da0 0x7f518019ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:31.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.255+0000 7f51860aa700 1 -- 172.21.15.29:0/750391190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5180102740 msgr2=0x7f518019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:31.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.255+0000 7f51860aa700 1 --2- 172.21.15.29:0/750391190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5180102740 0x7f518019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:31.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.255+0000 7f51860aa700 1 -- 172.21.15.29:0/750391190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5178009d00 con 0x7f5180100ba0 2024-03-11T19:09:31.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.255+0000 7f51860aa700 1 --2- 172.21.15.29:0/750391190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5180100ba0 0x7f518006a4a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5178000f80 tx=0x7f5178009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:31.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.255+0000 7f517f7fe700 1 -- 172.21.15.29:0/750391190 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517800e040 con 0x7f5180100ba0 2024-03-11T19:09:31.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.255+0000 7f517f7fe700 1 -- 172.21.15.29:0/750391190 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5178003e10 con 0x7f5180100ba0 2024-03-11T19:09:31.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.255+0000 7f518830e700 1 -- 172.21.15.29:0/750391190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f518019fdc0 con 0x7f5180100ba0 2024-03-11T19:09:31.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.255+0000 7f517f7fe700 1 -- 172.21.15.29:0/750391190 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517801d940 con 0x7f5180100ba0 2024-03-11T19:09:31.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.255+0000 7f518830e700 1 -- 172.21.15.29:0/750391190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51801a0180 con 0x7f5180100ba0 2024-03-11T19:09:31.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.256+0000 7f518830e700 1 -- 172.21.15.29:0/750391190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5180108020 con 0x7f5180100ba0 2024-03-11T19:09:31.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.257+0000 7f517f7fe700 1 -- 172.21.15.29:0/750391190 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5178026400 con 0x7f5180100ba0 2024-03-11T19:09:31.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.257+0000 7f517f7fe700 1 --2- 172.21.15.29:0/750391190 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51680a1ac0 0x7f51680a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:31.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.258+0000 7f51858a9700 1 --2- 172.21.15.29:0/750391190 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51680a1ac0 0x7f51680a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:31.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.258+0000 7f517f7fe700 1 -- 172.21.15.29:0/750391190 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f51780ce7e0 con 0x7f5180100ba0 2024-03-11T19:09:31.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.258+0000 7f51858a9700 1 --2- 172.21.15.29:0/750391190 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51680a1ac0 0x7f51680a3f80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f518006aa00 tx=0x7f517400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:31.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.262+0000 7f517f7fe700 1 -- 172.21.15.29:0/750391190 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5178097bb0 con 0x7f5180100ba0 2024-03-11T19:09:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.473+0000 7f518830e700 1 -- 172.21.15.29:0/750391190 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f518002d9f0 con 0x7f51680a1ac0 2024-03-11T19:09:31.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.479+0000 7f517f7fe700 1 -- 172.21.15.29:0/750391190 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f518002d9f0 con 0x7f51680a1ac0 2024-03-11T19:09:31.482 INFO:teuthology.orchestra.run.smithi029.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-11T19:09:31.482 INFO:teuthology.orchestra.run.smithi029.stdout:mds.all 12/12 39s ago 2m smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi170=b;smithi170=e;smithi170=h;smithi170=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l;count:12 2024-03-11T19:09:31.482 INFO:teuthology.orchestra.run.smithi029.stdout:mgr 3/3 39s ago 7m smithi029=x;smithi170=y;smithi190=z;count:3 2024-03-11T19:09:31.482 INFO:teuthology.orchestra.run.smithi029.stdout:mon 3/3 39s ago 8m smithi029:172.21.15.29=a;smithi170:172.21.15.170=b;smithi190:172.21.15.190=c;count:3 2024-03-11T19:09:31.482 INFO:teuthology.orchestra.run.smithi029.stdout:osd 12 39s ago - 2024-03-11T19:09:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.481+0000 7f518830e700 1 -- 172.21.15.29:0/750391190 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51680a1ac0 msgr2=0x7f51680a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.481+0000 7f518830e700 1 --2- 172.21.15.29:0/750391190 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51680a1ac0 0x7f51680a3f80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f518006aa00 tx=0x7f517400b410 comp rx=0 tx=0).stop 2024-03-11T19:09:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.481+0000 7f518830e700 1 -- 172.21.15.29:0/750391190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5180100ba0 msgr2=0x7f518006a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.481+0000 7f518830e700 1 --2- 172.21.15.29:0/750391190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5180100ba0 0x7f518006a4a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5178000f80 tx=0x7f5178009380 comp rx=0 tx=0).stop 2024-03-11T19:09:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.481+0000 7f518830e700 1 -- 172.21.15.29:0/750391190 shutdown_connections 2024-03-11T19:09:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.481+0000 7f518830e700 1 --2- 172.21.15.29:0/750391190 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51680a1ac0 0x7f51680a3f80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.481+0000 7f518830e700 1 --2- 172.21.15.29:0/750391190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5180102740 0x7f518019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:31.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.481+0000 7f518830e700 1 --2- 172.21.15.29:0/750391190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5180100ba0 0x7f518006a4a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:31.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.481+0000 7f518830e700 1 --2- 172.21.15.29:0/750391190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5180101da0 0x7f518019ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:31.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.481+0000 7f518830e700 1 -- 172.21.15.29:0/750391190 >> 172.21.15.29:0/750391190 conn(0x7f51800fc140 msgr2=0x7f5180105160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:31.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.481+0000 7f518830e700 1 -- 172.21.15.29:0/750391190 shutdown_connections 2024-03-11T19:09:31.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:31.481+0000 7f518830e700 1 -- 172.21.15.29:0/750391190 wait complete. 2024-03-11T19:09:31.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:31 smithi190 ceph-mon[36292]: pgmap v342: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-11T19:09:31.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:31 smithi170 ceph-mon[35690]: pgmap v342: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-11T19:09:32.256 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-11T19:09:32.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:32 smithi029 ceph-mon[32439]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:09:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:09:32.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:32 smithi190 ceph-mon[36292]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:09:32.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:32 smithi170 ceph-mon[35690]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:09:33.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:33 smithi029 ceph-mon[32439]: pgmap v343: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:33.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:33 smithi190 ceph-mon[36292]: pgmap v343: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:33.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:33 smithi170 ceph-mon[35690]: pgmap v343: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:34.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.312+0000 7fda185f1700 1 -- 172.21.15.29:0/1056701380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda10100ba0 msgr2=0x7fda10100fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:34.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.312+0000 7fda185f1700 1 --2- 172.21.15.29:0/1056701380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda10100ba0 0x7fda10100fc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fda0c0099c0 tx=0x7fda0c009cd0 comp rx=0 tx=0).stop 2024-03-11T19:09:34.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.312+0000 7fda185f1700 1 -- 172.21.15.29:0/1056701380 shutdown_connections 2024-03-11T19:09:34.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.312+0000 7fda185f1700 1 --2- 172.21.15.29:0/1056701380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda10102740 0x7fda10108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:34.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.312+0000 7fda185f1700 1 --2- 172.21.15.29:0/1056701380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda10100ba0 0x7fda10100fc0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:34.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.312+0000 7fda185f1700 1 --2- 172.21.15.29:0/1056701380 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda10101da0 0x7fda10102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:34.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.312+0000 7fda185f1700 1 -- 172.21.15.29:0/1056701380 >> 172.21.15.29:0/1056701380 conn(0x7fda100fc140 msgr2=0x7fda100fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:34.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.312+0000 7fda185f1700 1 -- 172.21.15.29:0/1056701380 shutdown_connections 2024-03-11T19:09:34.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.312+0000 7fda185f1700 1 -- 172.21.15.29:0/1056701380 wait complete. 2024-03-11T19:09:34.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.313+0000 7fda185f1700 1 Processor -- start 2024-03-11T19:09:34.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.313+0000 7fda185f1700 1 -- start start 2024-03-11T19:09:34.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.313+0000 7fda185f1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda10100ba0 0x7fda10197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:34.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.313+0000 7fda185f1700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda10101da0 0x7fda10198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:34.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.313+0000 7fda185f1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda10102740 0x7fda1019f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:34.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.313+0000 7fda185f1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1019f760 con 0x7fda10102740 2024-03-11T19:09:34.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.313+0000 7fda185f1700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1019f8a0 con 0x7fda10101da0 2024-03-11T19:09:34.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.313+0000 7fda185f1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1019fa30 con 0x7fda10100ba0 2024-03-11T19:09:34.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.313+0000 7fda16b8e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda10102740 0x7fda1019f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:34.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.313+0000 7fda16b8e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda10102740 0x7fda1019f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59702/0 (socket says 172.21.15.29:59702) 2024-03-11T19:09:34.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.313+0000 7fda16b8e700 1 -- 172.21.15.29:0/2502257201 learned_addr learned my addr 172.21.15.29:0/2502257201 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:34.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.313+0000 7fda15b8c700 1 --2- 172.21.15.29:0/2502257201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda10101da0 0x7fda10198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:34.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.314+0000 7fda1638d700 1 --2- 172.21.15.29:0/2502257201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda10100ba0 0x7fda10197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:34.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.314+0000 7fda16b8e700 1 -- 172.21.15.29:0/2502257201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda10100ba0 msgr2=0x7fda10197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:34.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.314+0000 7fda16b8e700 1 --2- 172.21.15.29:0/2502257201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda10100ba0 0x7fda10197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:34.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.314+0000 7fda16b8e700 1 -- 172.21.15.29:0/2502257201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda10101da0 msgr2=0x7fda10198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:34.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.314+0000 7fda16b8e700 1 --2- 172.21.15.29:0/2502257201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda10101da0 0x7fda10198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:34.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.314+0000 7fda16b8e700 1 -- 172.21.15.29:0/2502257201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0c0095f0 con 0x7fda10102740 2024-03-11T19:09:34.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.314+0000 7fda16b8e700 1 --2- 172.21.15.29:0/2502257201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda10102740 0x7fda1019f190 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fda08007f10 tx=0x7fda0800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:34.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.314+0000 7fda077fe700 1 -- 172.21.15.29:0/2502257201 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda08017070 con 0x7fda10102740 2024-03-11T19:09:34.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.314+0000 7fda185f1700 1 -- 172.21.15.29:0/2502257201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda1019fd10 con 0x7fda10102740 2024-03-11T19:09:34.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.314+0000 7fda077fe700 1 -- 172.21.15.29:0/2502257201 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fda0800f040 con 0x7fda10102740 2024-03-11T19:09:34.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.314+0000 7fda077fe700 1 -- 172.21.15.29:0/2502257201 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda08005730 con 0x7fda10102740 2024-03-11T19:09:34.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.314+0000 7fda185f1700 1 -- 172.21.15.29:0/2502257201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda101a01c0 con 0x7fda10102740 2024-03-11T19:09:34.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.315+0000 7fda185f1700 1 -- 172.21.15.29:0/2502257201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda10068930 con 0x7fda10102740 2024-03-11T19:09:34.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.316+0000 7fda077fe700 1 -- 172.21.15.29:0/2502257201 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fda08005890 con 0x7fda10102740 2024-03-11T19:09:34.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.316+0000 7fda077fe700 1 --2- 172.21.15.29:0/2502257201 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd9fc0a1ac0 0x7fd9fc0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:34.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.317+0000 7fda1638d700 1 --2- 172.21.15.29:0/2502257201 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd9fc0a1ac0 0x7fd9fc0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:34.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.317+0000 7fda077fe700 1 -- 172.21.15.29:0/2502257201 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fda080cd280 con 0x7fda10102740 2024-03-11T19:09:34.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.317+0000 7fda1638d700 1 --2- 172.21.15.29:0/2502257201 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd9fc0a1ac0 0x7fd9fc0a3f80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fda0c000c00 tx=0x7fda0c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:34.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.320+0000 7fda077fe700 1 -- 172.21.15.29:0/2502257201 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fda08096490 con 0x7fda10102740 2024-03-11T19:09:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.529+0000 7fda185f1700 1 -- 172.21.15.29:0/2502257201 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fda1006a030 con 0x7fd9fc0a1ac0 2024-03-11T19:09:34.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.531+0000 7fda077fe700 1 -- 172.21.15.29:0/2502257201 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fda1006a030 con 0x7fd9fc0a1ac0 2024-03-11T19:09:34.534 INFO:teuthology.orchestra.run.smithi029.stdout:HOST ADDR LABELS STATUS 2024-03-11T19:09:34.534 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 172.21.15.29 2024-03-11T19:09:34.534 INFO:teuthology.orchestra.run.smithi029.stdout:smithi170 172.21.15.170 2024-03-11T19:09:34.534 INFO:teuthology.orchestra.run.smithi029.stdout:smithi190 172.21.15.190 2024-03-11T19:09:34.534 INFO:teuthology.orchestra.run.smithi029.stdout:3 hosts in cluster 2024-03-11T19:09:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.533+0000 7fda185f1700 1 -- 172.21.15.29:0/2502257201 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd9fc0a1ac0 msgr2=0x7fd9fc0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.533+0000 7fda185f1700 1 --2- 172.21.15.29:0/2502257201 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd9fc0a1ac0 0x7fd9fc0a3f80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fda0c000c00 tx=0x7fda0c005fb0 comp rx=0 tx=0).stop 2024-03-11T19:09:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.533+0000 7fda185f1700 1 -- 172.21.15.29:0/2502257201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda10102740 msgr2=0x7fda1019f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.533+0000 7fda185f1700 1 --2- 172.21.15.29:0/2502257201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda10102740 0x7fda1019f190 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fda08007f10 tx=0x7fda0800d3b0 comp rx=0 tx=0).stop 2024-03-11T19:09:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.533+0000 7fda185f1700 1 -- 172.21.15.29:0/2502257201 shutdown_connections 2024-03-11T19:09:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.533+0000 7fda185f1700 1 --2- 172.21.15.29:0/2502257201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda10102740 0x7fda1019f190 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.533+0000 7fda185f1700 1 --2- 172.21.15.29:0/2502257201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda10100ba0 0x7fda10197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.533+0000 7fda185f1700 1 --2- 172.21.15.29:0/2502257201 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd9fc0a1ac0 0x7fd9fc0a3f80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.533+0000 7fda185f1700 1 --2- 172.21.15.29:0/2502257201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda10101da0 0x7fda10198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.533+0000 7fda185f1700 1 -- 172.21.15.29:0/2502257201 >> 172.21.15.29:0/2502257201 conn(0x7fda100fc140 msgr2=0x7fda10069920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.533+0000 7fda185f1700 1 -- 172.21.15.29:0/2502257201 shutdown_connections 2024-03-11T19:09:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:34.533+0000 7fda185f1700 1 -- 172.21.15.29:0/2502257201 wait complete. 2024-03-11T19:09:35.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:35 smithi029 ceph-mon[32439]: pgmap v344: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:09:35.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:35 smithi029 ceph-mon[32439]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:09:35.398 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-11T19:09:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:35 smithi190 ceph-mon[36292]: pgmap v344: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:09:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:35 smithi190 ceph-mon[36292]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:09:35.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:35 smithi170 ceph-mon[35690]: pgmap v344: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:09:35.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:35 smithi170 ceph-mon[35690]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:09:35.658 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:09:37.354 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:37 smithi029 ceph-mon[32439]: pgmap v345: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.460+0000 7f464a110700 1 -- 172.21.15.29:0/1399166677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4644103840 msgr2=0x7f464410a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.460+0000 7f464a110700 1 --2- 172.21.15.29:0/1399166677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4644103840 0x7f464410a0f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f463c00b0d0 tx=0x7f463c00b3e0 comp rx=0 tx=0).stop 2024-03-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.460+0000 7f464a110700 1 -- 172.21.15.29:0/1399166677 shutdown_connections 2024-03-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.460+0000 7f464a110700 1 --2- 172.21.15.29:0/1399166677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4644103840 0x7f464410a0f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.460+0000 7f464a110700 1 --2- 172.21.15.29:0/1399166677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4644101ca0 0x7f46441020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.460+0000 7f464a110700 1 --2- 172.21.15.29:0/1399166677 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4644102ea0 0x7f4644103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.460+0000 7f464a110700 1 -- 172.21.15.29:0/1399166677 >> 172.21.15.29:0/1399166677 conn(0x7f46440fd280 msgr2=0x7f46440ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.460+0000 7f464a110700 1 -- 172.21.15.29:0/1399166677 shutdown_connections 2024-03-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.460+0000 7f464a110700 1 -- 172.21.15.29:0/1399166677 wait complete. 2024-03-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.461+0000 7f464a110700 1 Processor -- start 2024-03-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.461+0000 7f464a110700 1 -- start start 2024-03-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.461+0000 7f464a110700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4644101ca0 0x7f4644197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.461+0000 7f464a110700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4644102ea0 0x7f4644198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.461+0000 7f464a110700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4644103840 0x7f464419f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.461+0000 7f464a110700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464419f780 con 0x7f4644103840 2024-03-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.461+0000 7f464a110700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464419f8c0 con 0x7f4644101ca0 2024-03-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.461+0000 7f464a110700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464419fa80 con 0x7f4644102ea0 2024-03-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.461+0000 7f4643fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4644103840 0x7f464419f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.461+0000 7f4643fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4644103840 0x7f464419f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58056/0 (socket says 172.21.15.29:58056) 2024-03-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.461+0000 7f4643fff700 1 -- 172.21.15.29:0/479073044 learned_addr learned my addr 172.21.15.29:0/479073044 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.462+0000 7f46437fe700 1 --2- 172.21.15.29:0/479073044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4644101ca0 0x7f4644197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.462+0000 7f4642ffd700 1 --2- 172.21.15.29:0/479073044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4644102ea0 0x7f4644198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.462+0000 7f4643fff700 1 -- 172.21.15.29:0/479073044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4644102ea0 msgr2=0x7f4644198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.462+0000 7f4643fff700 1 --2- 172.21.15.29:0/479073044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4644102ea0 0x7f4644198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.462+0000 7f4643fff700 1 -- 172.21.15.29:0/479073044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4644101ca0 msgr2=0x7f4644197c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.462+0000 7f4643fff700 1 --2- 172.21.15.29:0/479073044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4644101ca0 0x7f4644197c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.462+0000 7f4643fff700 1 -- 172.21.15.29:0/479073044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c009d00 con 0x7f4644103840 2024-03-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.462+0000 7f4642ffd700 1 --2- 172.21.15.29:0/479073044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4644102ea0 0x7f4644198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.462+0000 7f4643fff700 1 --2- 172.21.15.29:0/479073044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4644103840 0x7f464419f1b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f463c007bc0 tx=0x7f463c009410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.462+0000 7f4640ff9700 1 -- 172.21.15.29:0/479073044 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c00e040 con 0x7f4644103840 2024-03-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.462+0000 7f464a110700 1 -- 172.21.15.29:0/479073044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f464419fd00 con 0x7f4644103840 2024-03-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.462+0000 7f4640ff9700 1 -- 172.21.15.29:0/479073044 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f463c004e60 con 0x7f4644103840 2024-03-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.462+0000 7f4640ff9700 1 -- 172.21.15.29:0/479073044 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c012a60 con 0x7f4644103840 2024-03-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.462+0000 7f464a110700 1 -- 172.21.15.29:0/479073044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46441a0110 con 0x7f4644103840 2024-03-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.463+0000 7f464a110700 1 -- 172.21.15.29:0/479073044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4644109000 con 0x7f4644103840 2024-03-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.464+0000 7f4640ff9700 1 -- 172.21.15.29:0/479073044 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f463c005790 con 0x7f4644103840 2024-03-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.465+0000 7f4640ff9700 1 --2- 172.21.15.29:0/479073044 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f462c0a1ac0 0x7f462c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.465+0000 7f46437fe700 1 --2- 172.21.15.29:0/479073044 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f462c0a1ac0 0x7f462c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.465+0000 7f4640ff9700 1 -- 172.21.15.29:0/479073044 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f463c0cef20 con 0x7f4644103840 2024-03-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.466+0000 7f46437fe700 1 --2- 172.21.15.29:0/479073044 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f462c0a1ac0 0x7f462c0a3f80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4644199110 tx=0x7f463400a3b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.469+0000 7f4640ff9700 1 -- 172.21.15.29:0/479073044 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f463c0981c0 con 0x7f4644103840 2024-03-11T19:09:37.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:37 smithi190 ceph-mon[36292]: pgmap v345: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:37.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:37 smithi170 ceph-mon[35690]: pgmap v345: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:37.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.682+0000 7f464a110700 1 -- 172.21.15.29:0/479073044 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f464402da20 con 0x7f462c0a1ac0 2024-03-11T19:09:37.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.684+0000 7f4640ff9700 1 -- 172.21.15.29:0/479073044 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7f464402da20 con 0x7f462c0a1ac0 2024-03-11T19:09:37.687 INFO:teuthology.orchestra.run.smithi029.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-11T19:09:37.687 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7512009B480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-03-11T19:09:37.687 INFO:teuthology.orchestra.run.smithi029.stdout:smithi170 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476002R480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-03-11T19:09:37.687 INFO:teuthology.orchestra.run.smithi029.stdout:smithi190 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT63210012400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-11T19:09:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.686+0000 7f464a110700 1 -- 172.21.15.29:0/479073044 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f462c0a1ac0 msgr2=0x7f462c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.686+0000 7f464a110700 1 --2- 172.21.15.29:0/479073044 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f462c0a1ac0 0x7f462c0a3f80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4644199110 tx=0x7f463400a3b0 comp rx=0 tx=0).stop 2024-03-11T19:09:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.686+0000 7f464a110700 1 -- 172.21.15.29:0/479073044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4644103840 msgr2=0x7f464419f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.686+0000 7f464a110700 1 --2- 172.21.15.29:0/479073044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4644103840 0x7f464419f1b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f463c007bc0 tx=0x7f463c009410 comp rx=0 tx=0).stop 2024-03-11T19:09:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.686+0000 7f464a110700 1 -- 172.21.15.29:0/479073044 shutdown_connections 2024-03-11T19:09:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.686+0000 7f464a110700 1 --2- 172.21.15.29:0/479073044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4644103840 0x7f464419f1b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.686+0000 7f464a110700 1 --2- 172.21.15.29:0/479073044 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f462c0a1ac0 0x7f462c0a3f80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.686+0000 7f464a110700 1 --2- 172.21.15.29:0/479073044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4644101ca0 0x7f4644197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.686+0000 7f464a110700 1 --2- 172.21.15.29:0/479073044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4644102ea0 0x7f4644198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:37.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.686+0000 7f464a110700 1 -- 172.21.15.29:0/479073044 >> 172.21.15.29:0/479073044 conn(0x7f46440fd280 msgr2=0x7f4644106140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:37.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.686+0000 7f464a110700 1 -- 172.21.15.29:0/479073044 shutdown_connections 2024-03-11T19:09:37.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:37.686+0000 7f464a110700 1 -- 172.21.15.29:0/479073044 wait complete. 2024-03-11T19:09:38.455 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T19:09:38.464 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-03-11T19:09:38.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-11T19:09:38.715 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:09:39.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:39 smithi029 ceph-mon[32439]: pgmap v346: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:09:39.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:39 smithi029 ceph-mon[32439]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:09:39.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:39 smithi190 ceph-mon[36292]: pgmap v346: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:09:39.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:39 smithi190 ceph-mon[36292]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:09:39.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:39 smithi170 ceph-mon[35690]: pgmap v346: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:09:39.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:39 smithi170 ceph-mon[35690]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:09:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.382+0000 7f5f2d449700 1 -- 172.21.15.29:0/2489100361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28107990 msgr2=0x7f5f28109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.382+0000 7f5f2d449700 1 --2- 172.21.15.29:0/2489100361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28107990 0x7f5f28109e00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f5f2000b0a0 tx=0x7f5f2000b3b0 comp rx=0 tx=0).stop 2024-03-11T19:09:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.382+0000 7f5f2d449700 1 -- 172.21.15.29:0/2489100361 shutdown_connections 2024-03-11T19:09:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.382+0000 7f5f2d449700 1 --2- 172.21.15.29:0/2489100361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28107990 0x7f5f28109e00 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.382+0000 7f5f2d449700 1 --2- 172.21.15.29:0/2489100361 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f28102730 0x7f5f28104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.382+0000 7f5f2d449700 1 --2- 172.21.15.29:0/2489100361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f28105060 0x7f5f28107450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.382+0000 7f5f2d449700 1 -- 172.21.15.29:0/2489100361 >> 172.21.15.29:0/2489100361 conn(0x7f5f280fc140 msgr2=0x7f5f280fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.382+0000 7f5f2d449700 1 -- 172.21.15.29:0/2489100361 shutdown_connections 2024-03-11T19:09:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.383+0000 7f5f2d449700 1 -- 172.21.15.29:0/2489100361 wait complete. 2024-03-11T19:09:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.383+0000 7f5f2d449700 1 Processor -- start 2024-03-11T19:09:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.383+0000 7f5f2d449700 1 -- start start 2024-03-11T19:09:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.383+0000 7f5f2d449700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28102730 0x7f5f28197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.383+0000 7f5f2d449700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f28105060 0x7f5f281982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.383+0000 7f5f2d449700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f28107990 0x7f5f2819f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.383+0000 7f5f2d449700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2819f8f0 con 0x7f5f28102730 2024-03-11T19:09:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.383+0000 7f5f2d449700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2819fa30 con 0x7f5f28107990 2024-03-11T19:09:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.383+0000 7f5f2d449700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2819fbf0 con 0x7f5f28105060 2024-03-11T19:09:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.383+0000 7f5f26ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28102730 0x7f5f28197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.383+0000 7f5f26ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28102730 0x7f5f28197da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58082/0 (socket says 172.21.15.29:58082) 2024-03-11T19:09:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.384+0000 7f5f26ffd700 1 -- 172.21.15.29:0/1440325895 learned_addr learned my addr 172.21.15.29:0/1440325895 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.384+0000 7f5f267fc700 1 --2- 172.21.15.29:0/1440325895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f28105060 0x7f5f281982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.384+0000 7f5f277fe700 1 --2- 172.21.15.29:0/1440325895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f28107990 0x7f5f2819f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.384+0000 7f5f26ffd700 1 -- 172.21.15.29:0/1440325895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f28105060 msgr2=0x7f5f281982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.384+0000 7f5f26ffd700 1 --2- 172.21.15.29:0/1440325895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f28105060 0x7f5f281982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.384+0000 7f5f26ffd700 1 -- 172.21.15.29:0/1440325895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f28107990 msgr2=0x7f5f2819f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.384+0000 7f5f26ffd700 1 --2- 172.21.15.29:0/1440325895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f28107990 0x7f5f2819f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.384+0000 7f5f26ffd700 1 -- 172.21.15.29:0/1440325895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f20009d00 con 0x7f5f28102730 2024-03-11T19:09:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.384+0000 7f5f26ffd700 1 --2- 172.21.15.29:0/1440325895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28102730 0x7f5f28197da0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800d950 tx=0x7f5f1800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.384+0000 7f5f0ffff700 1 -- 172.21.15.29:0/1440325895 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f18009b40 con 0x7f5f28102730 2024-03-11T19:09:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.384+0000 7f5f0ffff700 1 -- 172.21.15.29:0/1440325895 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5f18009ca0 con 0x7f5f28102730 2024-03-11T19:09:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.384+0000 7f5f2d449700 1 -- 172.21.15.29:0/1440325895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f2819fed0 con 0x7f5f28102730 2024-03-11T19:09:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.384+0000 7f5f0ffff700 1 -- 172.21.15.29:0/1440325895 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1800f620 con 0x7f5f28102730 2024-03-11T19:09:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.384+0000 7f5f2d449700 1 -- 172.21.15.29:0/1440325895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f281a0360 con 0x7f5f28102730 2024-03-11T19:09:40.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.385+0000 7f5f2d449700 1 -- 172.21.15.29:0/1440325895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f280fdcc0 con 0x7f5f28102730 2024-03-11T19:09:40.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.386+0000 7f5f0ffff700 1 -- 172.21.15.29:0/1440325895 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5f18010460 con 0x7f5f28102730 2024-03-11T19:09:40.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.387+0000 7f5f0ffff700 1 --2- 172.21.15.29:0/1440325895 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5f100a1ac0 0x7f5f100a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:40.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.387+0000 7f5f267fc700 1 --2- 172.21.15.29:0/1440325895 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5f100a1ac0 0x7f5f100a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:40.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.387+0000 7f5f0ffff700 1 -- 172.21.15.29:0/1440325895 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5f180cd710 con 0x7f5f28102730 2024-03-11T19:09:40.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.387+0000 7f5f267fc700 1 --2- 172.21.15.29:0/1440325895 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5f100a1ac0 0x7f5f100a3f80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c006fd0 tx=0x7f5f1c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:40.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.390+0000 7f5f0ffff700 1 -- 172.21.15.29:0/1440325895 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5f18096920 con 0x7f5f28102730 2024-03-11T19:09:40.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.639+0000 7f5f2d449700 1 -- 172.21.15.29:0/1440325895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5f2810d750 con 0x7f5f28102730 2024-03-11T19:09:40.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.640+0000 7f5f0ffff700 1 -- 172.21.15.29:0/1440325895 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+2965 (secure 0 0 0) 0x7f5f18020bb0 con 0x7f5f28102730 2024-03-11T19:09:40.643 INFO:teuthology.orchestra.run.smithi029.stdout:e26 2024-03-11T19:09:40.643 INFO:teuthology.orchestra.run.smithi029.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T19:09:40.644 INFO:teuthology.orchestra.run.smithi029.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T19:09:40.644 INFO:teuthology.orchestra.run.smithi029.stdout:legacy client fscid: 1 2024-03-11T19:09:40.644 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:40.644 INFO:teuthology.orchestra.run.smithi029.stdout:Filesystem 'cephfs' (1) 2024-03-11T19:09:40.644 INFO:teuthology.orchestra.run.smithi029.stdout:fs_name cephfs 2024-03-11T19:09:40.644 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 26 2024-03-11T19:09:40.644 INFO:teuthology.orchestra.run.smithi029.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-11T19:09:40.644 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-03-11T19:07:49.123624+0000 2024-03-11T19:09:40.644 INFO:teuthology.orchestra.run.smithi029.stdout:modified 2024-03-11T19:07:56.662890+0000 2024-03-11T19:09:40.644 INFO:teuthology.orchestra.run.smithi029.stdout:tableserver 0 2024-03-11T19:09:40.644 INFO:teuthology.orchestra.run.smithi029.stdout:root 0 2024-03-11T19:09:40.644 INFO:teuthology.orchestra.run.smithi029.stdout:session_timeout 300 2024-03-11T19:09:40.644 INFO:teuthology.orchestra.run.smithi029.stdout:session_autoclose 300 2024-03-11T19:09:40.644 INFO:teuthology.orchestra.run.smithi029.stdout:max_file_size 1099511627776 2024-03-11T19:09:40.645 INFO:teuthology.orchestra.run.smithi029.stdout:max_xattr_size 65536 2024-03-11T19:09:40.645 INFO:teuthology.orchestra.run.smithi029.stdout:required_client_features {} 2024-03-11T19:09:40.645 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure 0 2024-03-11T19:09:40.645 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure_osd_epoch 0 2024-03-11T19:09:40.645 INFO:teuthology.orchestra.run.smithi029.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T19:09:40.645 INFO:teuthology.orchestra.run.smithi029.stdout:max_mds 5 2024-03-11T19:09:40.645 INFO:teuthology.orchestra.run.smithi029.stdout:in 0,1,2,3,4 2024-03-11T19:09:40.645 INFO:teuthology.orchestra.run.smithi029.stdout:up {0=24506,1=24496,2=24475,3=24473,4=24463} 2024-03-11T19:09:40.645 INFO:teuthology.orchestra.run.smithi029.stdout:failed 2024-03-11T19:09:40.645 INFO:teuthology.orchestra.run.smithi029.stdout:damaged 2024-03-11T19:09:40.645 INFO:teuthology.orchestra.run.smithi029.stdout:stopped 2024-03-11T19:09:40.645 INFO:teuthology.orchestra.run.smithi029.stdout:data_pools [3] 2024-03-11T19:09:40.645 INFO:teuthology.orchestra.run.smithi029.stdout:metadata_pool 2 2024-03-11T19:09:40.645 INFO:teuthology.orchestra.run.smithi029.stdout:inline_data disabled 2024-03-11T19:09:40.646 INFO:teuthology.orchestra.run.smithi029.stdout:balancer 2024-03-11T19:09:40.646 INFO:teuthology.orchestra.run.smithi029.stdout:bal_rank_mask -1 2024-03-11T19:09:40.646 INFO:teuthology.orchestra.run.smithi029.stdout:standby_count_wanted 0 2024-03-11T19:09:40.646 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.b{0:24506} state up:active seq 3 addr [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:09:40.646 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.e{0:24449} state up:standby-replay seq 1 addr [v2:172.21.15.170:6834/2555518515,v1:172.21.15.170:6835/2555518515] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:09:40.646 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.i{1:24496} state up:active seq 5 addr [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:09:40.646 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.l{1:24418} state up:standby-replay seq 1 addr [v2:172.21.15.190:6832/493725011,v1:172.21.15.190:6833/493725011] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:09:40.646 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.c{2:24475} state up:active seq 7 addr [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:09:40.646 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.k{2:24407} state up:standby-replay seq 1 addr [v2:172.21.15.170:6832/1035391240,v1:172.21.15.170:6833/1035391240] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:09:40.646 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.h{3:24473} state up:active seq 7 addr [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:09:40.646 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.a{3:14628} state up:standby-replay seq 1 addr [v2:172.21.15.29:6840/3154444865,v1:172.21.15.29:6841/3154444865] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:09:40.646 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.f{4:24463} state up:active seq 8 addr [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:09:40.646 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.g{4:14616} state up:standby-replay seq 1 addr [v2:172.21.15.29:6838/72765211,v1:172.21.15.29:6839/72765211] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:09:40.646 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:40.647 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:40.647 INFO:teuthology.orchestra.run.smithi029.stdout:Standby daemons: 2024-03-11T19:09:40.647 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:40.647 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.j{-1:14532} state up:standby seq 1 addr [v2:172.21.15.29:6834/630953303,v1:172.21.15.29:6835/630953303] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:09:40.647 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.d{-1:14550} state up:standby seq 1 addr [v2:172.21.15.29:6836/368479661,v1:172.21.15.29:6837/368479661] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:09:40.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.643+0000 7f5f2d449700 1 -- 172.21.15.29:0/1440325895 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5f100a1ac0 msgr2=0x7f5f100a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:40.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.643+0000 7f5f2d449700 1 --2- 172.21.15.29:0/1440325895 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5f100a1ac0 0x7f5f100a3f80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c006fd0 tx=0x7f5f1c008040 comp rx=0 tx=0).stop 2024-03-11T19:09:40.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.643+0000 7f5f2d449700 1 -- 172.21.15.29:0/1440325895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28102730 msgr2=0x7f5f28197da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:40.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.643+0000 7f5f2d449700 1 --2- 172.21.15.29:0/1440325895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28102730 0x7f5f28197da0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800d950 tx=0x7f5f1800dd10 comp rx=0 tx=0).stop 2024-03-11T19:09:40.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.643+0000 7f5f2d449700 1 -- 172.21.15.29:0/1440325895 shutdown_connections 2024-03-11T19:09:40.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.643+0000 7f5f2d449700 1 --2- 172.21.15.29:0/1440325895 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5f100a1ac0 0x7f5f100a3f80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:40.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.643+0000 7f5f2d449700 1 --2- 172.21.15.29:0/1440325895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f28107990 0x7f5f2819f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:40.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.643+0000 7f5f2d449700 1 --2- 172.21.15.29:0/1440325895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28102730 0x7f5f28197da0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:40.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.643+0000 7f5f2d449700 1 --2- 172.21.15.29:0/1440325895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f28105060 0x7f5f281982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:40.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.643+0000 7f5f2d449700 1 -- 172.21.15.29:0/1440325895 >> 172.21.15.29:0/1440325895 conn(0x7f5f280fc140 msgr2=0x7f5f28103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:40.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.643+0000 7f5f2d449700 1 -- 172.21.15.29:0/1440325895 shutdown_connections 2024-03-11T19:09:40.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:40.643+0000 7f5f2d449700 1 -- 172.21.15.29:0/1440325895 wait complete. 2024-03-11T19:09:40.649 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 26 2024-03-11T19:09:41.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:41 smithi029 ceph-mon[32439]: pgmap v347: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:41.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:41 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1440325895' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-11T19:09:41.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:41 smithi190 ceph-mon[36292]: pgmap v347: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:41.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:41 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1440325895' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-11T19:09:41.513 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-11T19:09:41.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:41 smithi170 ceph-mon[35690]: pgmap v347: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:41.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:41 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1440325895' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-11T19:09:41.760 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:09:43.326 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:43 smithi029 ceph-mon[32439]: pgmap v348: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:43.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:43 smithi190 ceph-mon[36292]: pgmap v348: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:43.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.561+0000 7f4a78642700 1 -- 172.21.15.29:0/1807867699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a70101cc0 msgr2=0x7f4a701020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:43.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.561+0000 7f4a78642700 1 --2- 172.21.15.29:0/1807867699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a70101cc0 0x7f4a701020e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c0099c0 tx=0x7f4a6c009cd0 comp rx=0 tx=0).stop 2024-03-11T19:09:43.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.561+0000 7f4a78642700 1 -- 172.21.15.29:0/1807867699 shutdown_connections 2024-03-11T19:09:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.561+0000 7f4a78642700 1 --2- 172.21.15.29:0/1807867699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a70103860 0x7f4a7010a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.561+0000 7f4a78642700 1 --2- 172.21.15.29:0/1807867699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a70101cc0 0x7f4a701020e0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.561+0000 7f4a78642700 1 --2- 172.21.15.29:0/1807867699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a70102ec0 0x7f4a70103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.561+0000 7f4a78642700 1 -- 172.21.15.29:0/1807867699 >> 172.21.15.29:0/1807867699 conn(0x7f4a700fd280 msgr2=0x7f4a700ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.561+0000 7f4a78642700 1 -- 172.21.15.29:0/1807867699 shutdown_connections 2024-03-11T19:09:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.561+0000 7f4a78642700 1 -- 172.21.15.29:0/1807867699 wait complete. 2024-03-11T19:09:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.562+0000 7f4a78642700 1 Processor -- start 2024-03-11T19:09:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.562+0000 7f4a78642700 1 -- start start 2024-03-11T19:09:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.562+0000 7f4a78642700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a70101cc0 0x7f4a70199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.562+0000 7f4a78642700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a70102ec0 0x7f4a7019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.562+0000 7f4a78642700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a70103860 0x7f4a701a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.562+0000 7f4a78642700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a701a18d0 con 0x7f4a70102ec0 2024-03-11T19:09:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.562+0000 7f4a78642700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a701a1a10 con 0x7f4a70101cc0 2024-03-11T19:09:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.562+0000 7f4a78642700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a701a1ba0 con 0x7f4a70103860 2024-03-11T19:09:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.562+0000 7f4a75bdd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a70102ec0 0x7f4a7019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.562+0000 7f4a75bdd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a70102ec0 0x7f4a7019a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58092/0 (socket says 172.21.15.29:58092) 2024-03-11T19:09:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.562+0000 7f4a75bdd700 1 -- 172.21.15.29:0/1407152444 learned_addr learned my addr 172.21.15.29:0/1407152444 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.562+0000 7f4a763de700 1 --2- 172.21.15.29:0/1407152444 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a70101cc0 0x7f4a70199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.563+0000 7f4a76bdf700 1 --2- 172.21.15.29:0/1407152444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a70103860 0x7f4a701a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.563+0000 7f4a75bdd700 1 -- 172.21.15.29:0/1407152444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a70103860 msgr2=0x7f4a701a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.563+0000 7f4a75bdd700 1 --2- 172.21.15.29:0/1407152444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a70103860 0x7f4a701a1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.563+0000 7f4a75bdd700 1 -- 172.21.15.29:0/1407152444 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a70101cc0 msgr2=0x7f4a70199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.563+0000 7f4a75bdd700 1 --2- 172.21.15.29:0/1407152444 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a70101cc0 0x7f4a70199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.563+0000 7f4a75bdd700 1 -- 172.21.15.29:0/1407152444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6c0095f0 con 0x7f4a70102ec0 2024-03-11T19:09:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.563+0000 7f4a76bdf700 1 --2- 172.21.15.29:0/1407152444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a70103860 0x7f4a701a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.563+0000 7f4a75bdd700 1 --2- 172.21.15.29:0/1407152444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a70102ec0 0x7f4a7019a280 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000d980 tx=0x7f4a6000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.563+0000 7f4a677fe700 1 -- 172.21.15.29:0/1407152444 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60009b40 con 0x7f4a70102ec0 2024-03-11T19:09:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.563+0000 7f4a677fe700 1 -- 172.21.15.29:0/1407152444 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4a6000b6e0 con 0x7f4a70102ec0 2024-03-11T19:09:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.563+0000 7f4a78642700 1 -- 172.21.15.29:0/1407152444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a701a1e80 con 0x7f4a70102ec0 2024-03-11T19:09:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.563+0000 7f4a677fe700 1 -- 172.21.15.29:0/1407152444 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6000f620 con 0x7f4a70102ec0 2024-03-11T19:09:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.563+0000 7f4a78642700 1 -- 172.21.15.29:0/1407152444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a701a22f0 con 0x7f4a70102ec0 2024-03-11T19:09:43.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.564+0000 7f4a78642700 1 -- 172.21.15.29:0/1407152444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a70068960 con 0x7f4a70102ec0 2024-03-11T19:09:43.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.565+0000 7f4a677fe700 1 -- 172.21.15.29:0/1407152444 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4a60010460 con 0x7f4a70102ec0 2024-03-11T19:09:43.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.566+0000 7f4a677fe700 1 --2- 172.21.15.29:0/1407152444 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4a5c0a1ac0 0x7f4a5c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:43.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.566+0000 7f4a763de700 1 --2- 172.21.15.29:0/1407152444 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4a5c0a1ac0 0x7f4a5c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:43.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.566+0000 7f4a677fe700 1 -- 172.21.15.29:0/1407152444 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4a600cd210 con 0x7f4a70102ec0 2024-03-11T19:09:43.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.566+0000 7f4a763de700 1 --2- 172.21.15.29:0/1407152444 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4a5c0a1ac0 0x7f4a5c0a3f80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c006010 tx=0x7f4a6c005ec0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:43.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.569+0000 7f4a677fe700 1 -- 172.21.15.29:0/1407152444 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4a60096420 con 0x7f4a70102ec0 2024-03-11T19:09:43.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:43 smithi170 ceph-mon[35690]: pgmap v348: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:43.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.774+0000 7f4a78642700 1 -- 172.21.15.29:0/1407152444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f4a70074480 con 0x7f4a70102ec0 2024-03-11T19:09:43.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.775+0000 7f4a677fe700 1 -- 172.21.15.29:0/1407152444 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v89) v1 ==== 56+0+5232 (secure 0 0 0) 0x7f4a60099ae0 con 0x7f4a70102ec0 2024-03-11T19:09:43.778 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 89 2024-03-11T19:09:43.778 INFO:teuthology.orchestra.run.smithi029.stdout:fsid 638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T19:09:43.778 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-03-11T18:59:36.894847+0000 2024-03-11T19:09:43.779 INFO:teuthology.orchestra.run.smithi029.stdout:modified 2024-03-11T19:08:44.981806+0000 2024-03-11T19:09:43.779 INFO:teuthology.orchestra.run.smithi029.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-11T19:09:43.779 INFO:teuthology.orchestra.run.smithi029.stdout:crush_version 26 2024-03-11T19:09:43.779 INFO:teuthology.orchestra.run.smithi029.stdout:full_ratio 0.95 2024-03-11T19:09:43.779 INFO:teuthology.orchestra.run.smithi029.stdout:backfillfull_ratio 0.9 2024-03-11T19:09:43.779 INFO:teuthology.orchestra.run.smithi029.stdout:nearfull_ratio 0.85 2024-03-11T19:09:43.779 INFO:teuthology.orchestra.run.smithi029.stdout:require_min_compat_client luminous 2024-03-11T19:09:43.779 INFO:teuthology.orchestra.run.smithi029.stdout:min_compat_client luminous 2024-03-11T19:09:43.779 INFO:teuthology.orchestra.run.smithi029.stdout:require_osd_release squid 2024-03-11T19:09:43.779 INFO:teuthology.orchestra.run.smithi029.stdout:stretch_mode_enabled false 2024-03-11T19:09:43.779 INFO:teuthology.orchestra.run.smithi029.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-11T19:09:43.779 INFO:teuthology.orchestra.run.smithi029.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-11T19:09:43.779 INFO:teuthology.orchestra.run.smithi029.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 87 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-11T19:09:43.779 INFO:teuthology.orchestra.run.smithi029.stdout:max_osd 12 2024-03-11T19:09:43.780 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 up in weight 1 up_from 8 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6802/827892152,v1:172.21.15.29:6803/827892152] [v2:172.21.15.29:6804/827892152,v1:172.21.15.29:6805/827892152] exists,up 62b6ad23-8635-47d9-9485-c91c1d1f942d 2024-03-11T19:09:43.780 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 up in weight 1 up_from 13 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6800/531888562,v1:172.21.15.170:6801/531888562] [v2:172.21.15.170:6802/531888562,v1:172.21.15.170:6803/531888562] exists,up 4ae8a614-7d37-4e39-b8cc-542895cc5a30 2024-03-11T19:09:43.780 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 up in weight 1 up_from 18 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6800/1010896180,v1:172.21.15.190:6801/1010896180] [v2:172.21.15.190:6802/1010896180,v1:172.21.15.190:6803/1010896180] exists,up 799a6a3b-63d6-47b8-a88f-c3a08ee03a53 2024-03-11T19:09:43.780 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 up in weight 1 up_from 26 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6810/2103856055,v1:172.21.15.29:6811/2103856055] [v2:172.21.15.29:6812/2103856055,v1:172.21.15.29:6813/2103856055] exists,up 2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e 2024-03-11T19:09:43.780 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 up in weight 1 up_from 34 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6808/1979690226,v1:172.21.15.170:6809/1979690226] [v2:172.21.15.170:6810/1979690226,v1:172.21.15.170:6811/1979690226] exists,up 8719e9e6-6389-4554-931e-722f213781b2 2024-03-11T19:09:43.780 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 up in weight 1 up_from 40 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6808/318560672,v1:172.21.15.190:6809/318560672] [v2:172.21.15.190:6810/318560672,v1:172.21.15.190:6811/318560672] exists,up e8072585-c243-4a27-8a7a-b1af399aa467 2024-03-11T19:09:43.780 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 up in weight 1 up_from 48 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6818/1792625214,v1:172.21.15.29:6819/1792625214] [v2:172.21.15.29:6820/1792625214,v1:172.21.15.29:6821/1792625214] exists,up 6ab9b5a8-7c19-4e5f-b211-6201fee37f11 2024-03-11T19:09:43.780 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 up in weight 1 up_from 53 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6816/3259207349,v1:172.21.15.170:6817/3259207349] [v2:172.21.15.170:6818/3259207349,v1:172.21.15.170:6819/3259207349] exists,up f7d84364-5bf1-473c-9676-253fe01bff2b 2024-03-11T19:09:43.780 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8 up in weight 1 up_from 61 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6816/1474709011,v1:172.21.15.190:6817/1474709011] [v2:172.21.15.190:6818/1474709011,v1:172.21.15.190:6819/1474709011] exists,up 7d27edd6-a75d-4b20-ab1b-dfa99909419d 2024-03-11T19:09:43.780 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9 up in weight 1 up_from 69 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6826/3553345080,v1:172.21.15.29:6827/3553345080] [v2:172.21.15.29:6828/3553345080,v1:172.21.15.29:6829/3553345080] exists,up e593e338-5c8b-4d2e-a990-39fd8c3eff43 2024-03-11T19:09:43.780 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10 up in weight 1 up_from 74 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6824/3766541425,v1:172.21.15.170:6825/3766541425] [v2:172.21.15.170:6826/3766541425,v1:172.21.15.170:6827/3766541425] exists,up 06dde09c-a014-4cac-aa88-352b63ff644b 2024-03-11T19:09:43.780 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11 up in weight 1 up_from 82 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6824/3926470159,v1:172.21.15.190:6825/3926470159] [v2:172.21.15.190:6826/3926470159,v1:172.21.15.190:6827/3926470159] exists,up 0cf7ece9-e288-49e6-b833-e9935c3abfc7 2024-03-11T19:09:43.780 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 2.5 [8,11] 2024-03-11T19:09:43.780 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 2.a [2,11] 2024-03-11T19:09:43.780 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 3.11 [5,9] 2024-03-11T19:09:43.781 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 3.21 [4,8] 2024-03-11T19:09:43.781 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 3.24 [4,9] 2024-03-11T19:09:43.781 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 3.3c [4,9] 2024-03-11T19:09:43.781 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/2913819020 expires 2024-03-12T19:00:43.524861+0000 2024-03-11T19:09:43.781 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/961617511 expires 2024-03-12T19:00:43.524861+0000 2024-03-11T19:09:43.781 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6800/3585665950 expires 2024-03-12T19:00:43.524861+0000 2024-03-11T19:09:43.781 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6801/3550575954 expires 2024-03-12T19:00:15.449726+0000 2024-03-11T19:09:43.781 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6800/3550575954 expires 2024-03-12T19:00:15.449726+0000 2024-03-11T19:09:43.781 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/385573026 expires 2024-03-12T19:00:15.449726+0000 2024-03-11T19:09:43.781 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6801/3585665950 expires 2024-03-12T19:00:43.524861+0000 2024-03-11T19:09:43.781 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/2164849075 expires 2024-03-12T19:00:15.449726+0000 2024-03-11T19:09:43.781 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/1037268269 expires 2024-03-12T19:00:43.524861+0000 2024-03-11T19:09:43.781 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/2099360722 expires 2024-03-12T19:00:15.449726+0000 2024-03-11T19:09:43.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.777+0000 7f4a78642700 1 -- 172.21.15.29:0/1407152444 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4a5c0a1ac0 msgr2=0x7f4a5c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:43.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.777+0000 7f4a78642700 1 --2- 172.21.15.29:0/1407152444 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4a5c0a1ac0 0x7f4a5c0a3f80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c006010 tx=0x7f4a6c005ec0 comp rx=0 tx=0).stop 2024-03-11T19:09:43.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.777+0000 7f4a78642700 1 -- 172.21.15.29:0/1407152444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a70102ec0 msgr2=0x7f4a7019a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:43.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.777+0000 7f4a78642700 1 --2- 172.21.15.29:0/1407152444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a70102ec0 0x7f4a7019a280 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000d980 tx=0x7f4a6000dc90 comp rx=0 tx=0).stop 2024-03-11T19:09:43.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.777+0000 7f4a78642700 1 -- 172.21.15.29:0/1407152444 shutdown_connections 2024-03-11T19:09:43.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.777+0000 7f4a78642700 1 --2- 172.21.15.29:0/1407152444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a70103860 0x7f4a701a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:43.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.777+0000 7f4a78642700 1 --2- 172.21.15.29:0/1407152444 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4a5c0a1ac0 0x7f4a5c0a3f80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:43.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.777+0000 7f4a78642700 1 --2- 172.21.15.29:0/1407152444 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a70101cc0 0x7f4a70199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:43.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.777+0000 7f4a78642700 1 --2- 172.21.15.29:0/1407152444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a70102ec0 0x7f4a7019a280 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:43.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.777+0000 7f4a78642700 1 -- 172.21.15.29:0/1407152444 >> 172.21.15.29:0/1407152444 conn(0x7f4a700fd280 msgr2=0x7f4a700fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:43.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.777+0000 7f4a78642700 1 -- 172.21.15.29:0/1407152444 shutdown_connections 2024-03-11T19:09:43.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:43.778+0000 7f4a78642700 1 -- 172.21.15.29:0/1407152444 wait complete. 2024-03-11T19:09:44.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:44 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1407152444' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T19:09:44.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:44 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1407152444' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T19:09:44.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:44 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1407152444' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T19:09:44.615 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-11T19:09:44.627 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:09:44.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.736+0000 7f74bdd77640 1 -- 172.21.15.29:0/3525520875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74b8101310 msgr2=0x7f74b8101790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:44.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.736+0000 7f74bdd77640 1 --2- 172.21.15.29:0/3525520875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74b8101310 0x7f74b8101790 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f74a8009900 tx=0x7f74a802bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:44.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.736+0000 7f74bdd77640 1 -- 172.21.15.29:0/3525520875 shutdown_connections 2024-03-11T19:09:44.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.736+0000 7f74bdd77640 1 --2- 172.21.15.29:0/3525520875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74b8101cd0 0x7f74b8108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:44.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.736+0000 7f74bdd77640 1 --2- 172.21.15.29:0/3525520875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74b8101310 0x7f74b8101790 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:44.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.736+0000 7f74bdd77640 1 --2- 172.21.15.29:0/3525520875 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74b8100120 0x7f74b8100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:44.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.736+0000 7f74bdd77640 1 -- 172.21.15.29:0/3525520875 >> 172.21.15.29:0/3525520875 conn(0x7f74b80fb6b0 msgr2=0x7f74b80fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:44.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.736+0000 7f74bdd77640 1 -- 172.21.15.29:0/3525520875 shutdown_connections 2024-03-11T19:09:44.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.736+0000 7f74bdd77640 1 -- 172.21.15.29:0/3525520875 wait complete. 2024-03-11T19:09:44.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.737+0000 7f74bdd77640 1 Processor -- start 2024-03-11T19:09:44.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.737+0000 7f74bdd77640 1 -- start start 2024-03-11T19:09:44.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.737+0000 7f74bdd77640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74b8100120 0x7f74b81970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:44.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.737+0000 7f74bdd77640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74b8101310 0x7f74b81975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:44.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.737+0000 7f74bdd77640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74b8101cd0 0x7f74b819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:44.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.737+0000 7f74bdd77640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b819eba0 con 0x7f74b8101310 2024-03-11T19:09:44.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.737+0000 7f74bdd77640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b819ed10 con 0x7f74b8101cd0 2024-03-11T19:09:44.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.737+0000 7f74bdd77640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b819eed0 con 0x7f74b8100120 2024-03-11T19:09:44.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.737+0000 7f74b6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74b8101310 0x7f74b81975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:44.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.737+0000 7f74b6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74b8101310 0x7f74b81975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58104/0 (socket says 172.21.15.29:58104) 2024-03-11T19:09:44.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.737+0000 7f74b6ffd640 1 -- 172.21.15.29:0/1224077080 learned_addr learned my addr 172.21.15.29:0/1224077080 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:44.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.738+0000 7f74b77fe640 1 --2- 172.21.15.29:0/1224077080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74b8100120 0x7f74b81970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:44.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.738+0000 7f74b7fff640 1 --2- 172.21.15.29:0/1224077080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74b8101cd0 0x7f74b819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:44.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.738+0000 7f74b6ffd640 1 -- 172.21.15.29:0/1224077080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74b8100120 msgr2=0x7f74b81970b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:44.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.738+0000 7f74b6ffd640 1 --2- 172.21.15.29:0/1224077080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74b8100120 0x7f74b81970b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:44.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.738+0000 7f74b6ffd640 1 -- 172.21.15.29:0/1224077080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74b8101cd0 msgr2=0x7f74b819e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:44.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.738+0000 7f74b6ffd640 1 --2- 172.21.15.29:0/1224077080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74b8101cd0 0x7f74b819e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:44.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.738+0000 7f74b6ffd640 1 -- 172.21.15.29:0/1224077080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a8009560 con 0x7f74b8101310 2024-03-11T19:09:44.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.738+0000 7f74b7fff640 1 --2- 172.21.15.29:0/1224077080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74b8101cd0 0x7f74b819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:44.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.738+0000 7f74b6ffd640 1 --2- 172.21.15.29:0/1224077080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74b8101310 0x7f74b81975f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f74a8036040 tx=0x7f74a80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:44.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.738+0000 7f74b4ff9640 1 -- 172.21.15.29:0/1224077080 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a80044e0 con 0x7f74b8101310 2024-03-11T19:09:44.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.738+0000 7f74bdd77640 1 -- 172.21.15.29:0/1224077080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b819f150 con 0x7f74b8101310 2024-03-11T19:09:44.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.738+0000 7f74b4ff9640 1 -- 172.21.15.29:0/1224077080 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f74a802de90 con 0x7f74b8101310 2024-03-11T19:09:44.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.738+0000 7f74b4ff9640 1 -- 172.21.15.29:0/1224077080 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a8005470 con 0x7f74b8101310 2024-03-11T19:09:44.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.738+0000 7f74bdd77640 1 -- 172.21.15.29:0/1224077080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b819f6b0 con 0x7f74b8101310 2024-03-11T19:09:44.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.739+0000 7f74bdd77640 1 -- 172.21.15.29:0/1224077080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b8107440 con 0x7f74b8101310 2024-03-11T19:09:44.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.740+0000 7f74b4ff9640 1 -- 172.21.15.29:0/1224077080 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f74a8003800 con 0x7f74b8101310 2024-03-11T19:09:44.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.741+0000 7f74b4ff9640 1 --2- 172.21.15.29:0/1224077080 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f74a00a1a30 0x7f74a00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:44.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.741+0000 7f74b77fe640 1 --2- 172.21.15.29:0/1224077080 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f74a00a1a30 0x7f74a00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:44.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.741+0000 7f74b4ff9640 1 -- 172.21.15.29:0/1224077080 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f74a80efe00 con 0x7f74b8101310 2024-03-11T19:09:44.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.741+0000 7f74b77fe640 1 --2- 172.21.15.29:0/1224077080 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f74a00a1a30 0x7f74a00a3ef0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f74ac0058c0 tx=0x7f74ac005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:44.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.745+0000 7f74b4ff9640 1 -- 172.21.15.29:0/1224077080 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f74a80b9090 con 0x7f74b8101310 2024-03-11T19:09:44.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.926+0000 7f74bdd77640 1 -- 172.21.15.29:0/1224077080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74b8106eb0 con 0x7f74b8101310 2024-03-11T19:09:44.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.928+0000 7f74b4ff9640 1 -- 172.21.15.29:0/1224077080 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22559 (secure 0 0 0) 0x7f74a80bc750 con 0x7f74b8101310 2024-03-11T19:09:44.931 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:44.931 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":89,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:08:44.981806+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:07:47.563184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:07:48.282461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:09:44.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.930+0000 7f74bdd77640 1 -- 172.21.15.29:0/1224077080 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f74a00a1a30 msgr2=0x7f74a00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:44.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.930+0000 7f74bdd77640 1 --2- 172.21.15.29:0/1224077080 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f74a00a1a30 0x7f74a00a3ef0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f74ac0058c0 tx=0x7f74ac005850 comp rx=0 tx=0).stop 2024-03-11T19:09:44.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.930+0000 7f74bdd77640 1 -- 172.21.15.29:0/1224077080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74b8101310 msgr2=0x7f74b81975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:44.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.930+0000 7f74bdd77640 1 --2- 172.21.15.29:0/1224077080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74b8101310 0x7f74b81975f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f74a8036040 tx=0x7f74a80042f0 comp rx=0 tx=0).stop 2024-03-11T19:09:44.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.930+0000 7f74bdd77640 1 -- 172.21.15.29:0/1224077080 shutdown_connections 2024-03-11T19:09:44.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.930+0000 7f74bdd77640 1 --2- 172.21.15.29:0/1224077080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74b8101cd0 0x7f74b819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:44.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.930+0000 7f74bdd77640 1 --2- 172.21.15.29:0/1224077080 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f74a00a1a30 0x7f74a00a3ef0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:44.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.930+0000 7f74bdd77640 1 --2- 172.21.15.29:0/1224077080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74b8101310 0x7f74b81975f0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:44.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.930+0000 7f74bdd77640 1 --2- 172.21.15.29:0/1224077080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74b8100120 0x7f74b81970b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:44.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.930+0000 7f74bdd77640 1 -- 172.21.15.29:0/1224077080 >> 172.21.15.29:0/1224077080 conn(0x7f74b80fb6b0 msgr2=0x7f74b8104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:44.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.930+0000 7f74bdd77640 1 -- 172.21.15.29:0/1224077080 shutdown_connections 2024-03-11T19:09:44.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:44.930+0000 7f74bdd77640 1 -- 172.21.15.29:0/1224077080 wait complete. 2024-03-11T19:09:44.945 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:03:09.860019+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T19:07:47.563184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T19:07:48.282461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T19:09:44.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:09:45.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.048+0000 7f31a2bac640 1 -- 172.21.15.29:0/2938858379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c101270 msgr2=0x7f319c1016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:45.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.048+0000 7f31a2bac640 1 --2- 172.21.15.29:0/2938858379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c101270 0x7f319c1016f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3188009900 tx=0x7f318802bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:45.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.048+0000 7f31a2bac640 1 -- 172.21.15.29:0/2938858379 shutdown_connections 2024-03-11T19:09:45.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.048+0000 7f31a2bac640 1 --2- 172.21.15.29:0/2938858379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f319c101c30 0x7f319c1084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.048+0000 7f31a2bac640 1 --2- 172.21.15.29:0/2938858379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c101270 0x7f319c1016f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.048+0000 7f31a2bac640 1 --2- 172.21.15.29:0/2938858379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f319c100080 0x7f319c1004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.048+0000 7f31a2bac640 1 -- 172.21.15.29:0/2938858379 >> 172.21.15.29:0/2938858379 conn(0x7f319c0fb610 msgr2=0x7f319c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:45.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.048+0000 7f31a2bac640 1 -- 172.21.15.29:0/2938858379 shutdown_connections 2024-03-11T19:09:45.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.048+0000 7f31a2bac640 1 -- 172.21.15.29:0/2938858379 wait complete. 2024-03-11T19:09:45.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.048+0000 7f31a2bac640 1 Processor -- start 2024-03-11T19:09:45.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.048+0000 7f31a2bac640 1 -- start start 2024-03-11T19:09:45.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.048+0000 7f31a2bac640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f319c100080 0x7f319c104010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:45.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.048+0000 7f31a2bac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c101270 0x7f319c102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:45.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.048+0000 7f31a2bac640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f319c101c30 0x7f319c102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:45.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.048+0000 7f31a2bac640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c1032f0 con 0x7f319c101270 2024-03-11T19:09:45.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.048+0000 7f31a2bac640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c103460 con 0x7f319c100080 2024-03-11T19:09:45.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.048+0000 7f31a2bac640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c1a1150 con 0x7f319c101c30 2024-03-11T19:09:45.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.048+0000 7f319bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c101270 0x7f319c102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:45.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.049+0000 7f319bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c101270 0x7f319c102650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58118/0 (socket says 172.21.15.29:58118) 2024-03-11T19:09:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.049+0000 7f319bfff640 1 -- 172.21.15.29:0/2181388643 learned_addr learned my addr 172.21.15.29:0/2181388643 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.049+0000 7f31a0948640 1 --2- 172.21.15.29:0/2181388643 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f319c100080 0x7f319c104010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.049+0000 7f31a1149640 1 --2- 172.21.15.29:0/2181388643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f319c101c30 0x7f319c102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.049+0000 7f319bfff640 1 -- 172.21.15.29:0/2181388643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f319c101c30 msgr2=0x7f319c102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.049+0000 7f319bfff640 1 --2- 172.21.15.29:0/2181388643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f319c101c30 0x7f319c102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.049+0000 7f319bfff640 1 -- 172.21.15.29:0/2181388643 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f319c100080 msgr2=0x7f319c104010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.049+0000 7f319bfff640 1 --2- 172.21.15.29:0/2181388643 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f319c100080 0x7f319c104010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.049+0000 7f319bfff640 1 -- 172.21.15.29:0/2181388643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3188009560 con 0x7f319c101270 2024-03-11T19:09:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.049+0000 7f31a1149640 1 --2- 172.21.15.29:0/2181388643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f319c101c30 0x7f319c102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.049+0000 7f319bfff640 1 --2- 172.21.15.29:0/2181388643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c101270 0x7f319c102650 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3188009430 tx=0x7f31880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.049+0000 7f3199ffb640 1 -- 172.21.15.29:0/2181388643 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3188004510 con 0x7f319c101270 2024-03-11T19:09:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.049+0000 7f3199ffb640 1 -- 172.21.15.29:0/2181388643 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f318802de90 con 0x7f319c101270 2024-03-11T19:09:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.049+0000 7f31a2bac640 1 -- 172.21.15.29:0/2181388643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f319c1a13a0 con 0x7f319c101270 2024-03-11T19:09:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.049+0000 7f3199ffb640 1 -- 172.21.15.29:0/2181388643 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3188003cc0 con 0x7f319c101270 2024-03-11T19:09:45.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.049+0000 7f31a2bac640 1 -- 172.21.15.29:0/2181388643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319c1a1900 con 0x7f319c101270 2024-03-11T19:09:45.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.050+0000 7f31a2bac640 1 -- 172.21.15.29:0/2181388643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f319c10b940 con 0x7f319c101270 2024-03-11T19:09:45.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.051+0000 7f3199ffb640 1 -- 172.21.15.29:0/2181388643 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3188004fd0 con 0x7f319c101270 2024-03-11T19:09:45.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.052+0000 7f3199ffb640 1 --2- 172.21.15.29:0/2181388643 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f318c0a1ab0 0x7f318c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:45.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.052+0000 7f31a0948640 1 --2- 172.21.15.29:0/2181388643 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f318c0a1ab0 0x7f318c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:45.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.052+0000 7f3199ffb640 1 -- 172.21.15.29:0/2181388643 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f31880ef560 con 0x7f319c101270 2024-03-11T19:09:45.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.052+0000 7f31a0948640 1 --2- 172.21.15.29:0/2181388643 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f318c0a1ab0 0x7f318c0a3f70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3190005fd0 tx=0x7f319000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:45.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.056+0000 7f3199ffb640 1 -- 172.21.15.29:0/2181388643 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f31880b8810 con 0x7f319c101270 2024-03-11T19:09:45.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.237+0000 7f31a2bac640 1 -- 172.21.15.29:0/2181388643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f319c1a1b20 con 0x7f319c101270 2024-03-11T19:09:45.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.238+0000 7f3199ffb640 1 -- 172.21.15.29:0/2181388643 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f31880bbed0 con 0x7f319c101270 2024-03-11T19:09:45.241 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-11T19:09:45.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.240+0000 7f31a2bac640 1 -- 172.21.15.29:0/2181388643 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f318c0a1ab0 msgr2=0x7f318c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:45.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.240+0000 7f31a2bac640 1 --2- 172.21.15.29:0/2181388643 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f318c0a1ab0 0x7f318c0a3f70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3190005fd0 tx=0x7f319000b410 comp rx=0 tx=0).stop 2024-03-11T19:09:45.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.240+0000 7f31a2bac640 1 -- 172.21.15.29:0/2181388643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c101270 msgr2=0x7f319c102650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:45.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.240+0000 7f31a2bac640 1 --2- 172.21.15.29:0/2181388643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c101270 0x7f319c102650 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3188009430 tx=0x7f31880042f0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.240+0000 7f31a2bac640 1 -- 172.21.15.29:0/2181388643 shutdown_connections 2024-03-11T19:09:45.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.240+0000 7f31a2bac640 1 --2- 172.21.15.29:0/2181388643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f319c101c30 0x7f319c102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.240+0000 7f31a2bac640 1 --2- 172.21.15.29:0/2181388643 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f318c0a1ab0 0x7f318c0a3f70 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.240+0000 7f31a2bac640 1 --2- 172.21.15.29:0/2181388643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f319c101270 0x7f319c102650 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.240+0000 7f31a2bac640 1 --2- 172.21.15.29:0/2181388643 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f319c100080 0x7f319c104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.240+0000 7f31a2bac640 1 -- 172.21.15.29:0/2181388643 >> 172.21.15.29:0/2181388643 conn(0x7f319c0fb610 msgr2=0x7f319c0682e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:45.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.240+0000 7f31a2bac640 1 -- 172.21.15.29:0/2181388643 shutdown_connections 2024-03-11T19:09:45.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.240+0000 7f31a2bac640 1 -- 172.21.15.29:0/2181388643 wait complete. 2024-03-11T19:09:45.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:45 smithi029 ceph-mon[32439]: pgmap v349: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:09:45.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:45 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1224077080' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:45.254 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T19:09:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.356+0000 7f76b3325640 1 -- 172.21.15.29:0/4033703914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac1000a0 msgr2=0x7f76ac1004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:45.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.356+0000 7f76b3325640 1 --2- 172.21.15.29:0/4033703914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac1000a0 0x7f76ac1004c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f76a8009900 tx=0x7f76a802bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:45.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.356+0000 7f76b3325640 1 -- 172.21.15.29:0/4033703914 shutdown_connections 2024-03-11T19:09:45.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.356+0000 7f76b3325640 1 --2- 172.21.15.29:0/4033703914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac101c50 0x7f76ac1084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.356+0000 7f76b3325640 1 --2- 172.21.15.29:0/4033703914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76ac101290 0x7f76ac101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.356+0000 7f76b3325640 1 --2- 172.21.15.29:0/4033703914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac1000a0 0x7f76ac1004c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.356+0000 7f76b3325640 1 -- 172.21.15.29:0/4033703914 >> 172.21.15.29:0/4033703914 conn(0x7f76ac0fb610 msgr2=0x7f76ac0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:45.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.356+0000 7f76b3325640 1 -- 172.21.15.29:0/4033703914 shutdown_connections 2024-03-11T19:09:45.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.356+0000 7f76b3325640 1 -- 172.21.15.29:0/4033703914 wait complete. 2024-03-11T19:09:45.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76b3325640 1 Processor -- start 2024-03-11T19:09:45.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76b3325640 1 -- start start 2024-03-11T19:09:45.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76b3325640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76ac1000a0 0x7f76ac197250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:45.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76b3325640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac101290 0x7f76ac197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76b3325640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac101c50 0x7f76ac19e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76b3325640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac19ed40 con 0x7f76ac101c50 2024-03-11T19:09:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76b3325640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac19eeb0 con 0x7f76ac101290 2024-03-11T19:09:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76b3325640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac19f070 con 0x7f76ac1000a0 2024-03-11T19:09:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76b18c2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac101c50 0x7f76ac19e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76b18c2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac101c50 0x7f76ac19e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58138/0 (socket says 172.21.15.29:58138) 2024-03-11T19:09:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76b18c2640 1 -- 172.21.15.29:0/1737154623 learned_addr learned my addr 172.21.15.29:0/1737154623 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76b10c1640 1 --2- 172.21.15.29:0/1737154623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76ac1000a0 0x7f76ac197250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76b08c0640 1 --2- 172.21.15.29:0/1737154623 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac101290 0x7f76ac197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76b18c2640 1 -- 172.21.15.29:0/1737154623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76ac1000a0 msgr2=0x7f76ac197250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76b18c2640 1 --2- 172.21.15.29:0/1737154623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76ac1000a0 0x7f76ac197250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76b18c2640 1 -- 172.21.15.29:0/1737154623 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac101290 msgr2=0x7f76ac197790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76b18c2640 1 --2- 172.21.15.29:0/1737154623 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac101290 0x7f76ac197790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76b18c2640 1 -- 172.21.15.29:0/1737154623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a8009560 con 0x7f76ac101c50 2024-03-11T19:09:45.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76b18c2640 1 --2- 172.21.15.29:0/1737154623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac101c50 0x7f76ac19e800 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f76a400eaa0 tx=0x7f76a400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:45.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.357+0000 7f76a27fc640 1 -- 172.21.15.29:0/1737154623 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a400cdb0 con 0x7f76ac101c50 2024-03-11T19:09:45.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.358+0000 7f76a27fc640 1 -- 172.21.15.29:0/1737154623 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f76a4005020 con 0x7f76ac101c50 2024-03-11T19:09:45.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.358+0000 7f76b3325640 1 -- 172.21.15.29:0/1737154623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76ac19f350 con 0x7f76ac101c50 2024-03-11T19:09:45.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.358+0000 7f76a27fc640 1 -- 172.21.15.29:0/1737154623 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a40116b0 con 0x7f76ac101c50 2024-03-11T19:09:45.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.358+0000 7f76b3325640 1 -- 172.21.15.29:0/1737154623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76ac19f8e0 con 0x7f76ac101c50 2024-03-11T19:09:45.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.358+0000 7f76b3325640 1 -- 172.21.15.29:0/1737154623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76ac10b9a0 con 0x7f76ac101c50 2024-03-11T19:09:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.360+0000 7f76a27fc640 1 -- 172.21.15.29:0/1737154623 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f76a4005190 con 0x7f76ac101c50 2024-03-11T19:09:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.360+0000 7f76a27fc640 1 --2- 172.21.15.29:0/1737154623 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f76980a1ab0 0x7f76980a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.360+0000 7f76b10c1640 1 --2- 172.21.15.29:0/1737154623 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f76980a1ab0 0x7f76980a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.360+0000 7f76a27fc640 1 -- 172.21.15.29:0/1737154623 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f76a40cd950 con 0x7f76ac101c50 2024-03-11T19:09:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.361+0000 7f76b10c1640 1 --2- 172.21.15.29:0/1737154623 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f76980a1ab0 0x7f76980a3f70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f76a8006010 tx=0x7f76a8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:45.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.364+0000 7f76a27fc640 1 -- 172.21.15.29:0/1737154623 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f76a4096c80 con 0x7f76ac101c50 2024-03-11T19:09:45.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:45 smithi190 ceph-mon[36292]: pgmap v349: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:09:45.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:45 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1224077080' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:45.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.549+0000 7f76b3325640 1 -- 172.21.15.29:0/1737154623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f76ac10bb30 con 0x7f76ac101c50 2024-03-11T19:09:45.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.549+0000 7f76a27fc640 1 -- 172.21.15.29:0/1737154623 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f76a401f020 con 0x7f76ac101c50 2024-03-11T19:09:45.552 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-03-11T19:09:45.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.551+0000 7f76b3325640 1 -- 172.21.15.29:0/1737154623 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f76980a1ab0 msgr2=0x7f76980a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:45.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.551+0000 7f76b3325640 1 --2- 172.21.15.29:0/1737154623 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f76980a1ab0 0x7f76980a3f70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f76a8006010 tx=0x7f76a8000bc0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.551+0000 7f76b3325640 1 -- 172.21.15.29:0/1737154623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac101c50 msgr2=0x7f76ac19e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:45.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.551+0000 7f76b3325640 1 --2- 172.21.15.29:0/1737154623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac101c50 0x7f76ac19e800 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f76a400eaa0 tx=0x7f76a400ef60 comp rx=0 tx=0).stop 2024-03-11T19:09:45.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.551+0000 7f76b3325640 1 -- 172.21.15.29:0/1737154623 shutdown_connections 2024-03-11T19:09:45.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.551+0000 7f76b3325640 1 --2- 172.21.15.29:0/1737154623 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f76980a1ab0 0x7f76980a3f70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.551+0000 7f76b3325640 1 --2- 172.21.15.29:0/1737154623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac101c50 0x7f76ac19e800 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.551+0000 7f76b3325640 1 --2- 172.21.15.29:0/1737154623 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac101290 0x7f76ac197790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.551+0000 7f76b3325640 1 --2- 172.21.15.29:0/1737154623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76ac1000a0 0x7f76ac197250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.551+0000 7f76b3325640 1 -- 172.21.15.29:0/1737154623 >> 172.21.15.29:0/1737154623 conn(0x7f76ac0fb610 msgr2=0x7f76ac069820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:45.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.551+0000 7f76b3325640 1 -- 172.21.15.29:0/1737154623 shutdown_connections 2024-03-11T19:09:45.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.551+0000 7f76b3325640 1 -- 172.21.15.29:0/1737154623 wait complete. 2024-03-11T19:09:45.565 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T19:09:45.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:45 smithi170 ceph-mon[35690]: pgmap v349: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:09:45.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:45 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1224077080' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:45.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.668+0000 7fafa356c640 1 -- 172.21.15.29:0/3756420972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf9c106e60 msgr2=0x7faf9c1092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:45.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.668+0000 7fafa356c640 1 --2- 172.21.15.29:0/3756420972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf9c106e60 0x7faf9c1092b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7faf94009ca0 tx=0x7faf9402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.668+0000 7fafa356c640 1 -- 172.21.15.29:0/3756420972 shutdown_connections 2024-03-11T19:09:45.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.668+0000 7fafa356c640 1 --2- 172.21.15.29:0/3756420972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf9c106e60 0x7faf9c1092b0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.668+0000 7fafa356c640 1 --2- 172.21.15.29:0/3756420972 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf9c104530 0x7faf9c106920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.668+0000 7fafa356c640 1 --2- 172.21.15.29:0/3756420972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf9c101c00 0x7faf9c103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.668+0000 7fafa356c640 1 -- 172.21.15.29:0/3756420972 >> 172.21.15.29:0/3756420972 conn(0x7faf9c0fb610 msgr2=0x7faf9c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:45.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.668+0000 7fafa356c640 1 -- 172.21.15.29:0/3756420972 shutdown_connections 2024-03-11T19:09:45.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.668+0000 7fafa356c640 1 -- 172.21.15.29:0/3756420972 wait complete. 2024-03-11T19:09:45.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.669+0000 7fafa356c640 1 Processor -- start 2024-03-11T19:09:45.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.669+0000 7fafa356c640 1 -- start start 2024-03-11T19:09:45.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.669+0000 7fafa356c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf9c101c00 0x7faf9c197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:45.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.669+0000 7fafa356c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf9c104530 0x7faf9c197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:45.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.669+0000 7fafa356c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf9c106e60 0x7faf9c19e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:45.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.669+0000 7fafa356c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9c19ec40 con 0x7faf9c104530 2024-03-11T19:09:45.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.669+0000 7fafa356c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9c19edb0 con 0x7faf9c106e60 2024-03-11T19:09:45.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.669+0000 7fafa356c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9c19ef70 con 0x7faf9c101c00 2024-03-11T19:09:45.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.669+0000 7fafa0b07640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf9c104530 0x7faf9c197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:45.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.669+0000 7fafa0b07640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf9c104530 0x7faf9c197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49320/0 (socket says 172.21.15.29:49320) 2024-03-11T19:09:45.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.669+0000 7fafa0b07640 1 -- 172.21.15.29:0/1034879017 learned_addr learned my addr 172.21.15.29:0/1034879017 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:45.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.669+0000 7fafa0b07640 1 -- 172.21.15.29:0/1034879017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf9c101c00 msgr2=0x7faf9c197150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:45.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.669+0000 7fafa0b07640 1 --2- 172.21.15.29:0/1034879017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf9c101c00 0x7faf9c197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.669+0000 7fafa0b07640 1 -- 172.21.15.29:0/1034879017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf9c106e60 msgr2=0x7faf9c19e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:45.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.669+0000 7fafa1b09640 1 --2- 172.21.15.29:0/1034879017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf9c106e60 0x7faf9c19e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:45.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.669+0000 7fafa0b07640 1 --2- 172.21.15.29:0/1034879017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf9c106e60 0x7faf9c19e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.669+0000 7fafa0b07640 1 -- 172.21.15.29:0/1034879017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf94009900 con 0x7faf9c104530 2024-03-11T19:09:45.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.670+0000 7fafa0b07640 1 --2- 172.21.15.29:0/1034879017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf9c104530 0x7faf9c197690 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7faf8c00c970 tx=0x7faf8c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:45.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.670+0000 7faf927fc640 1 -- 172.21.15.29:0/1034879017 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c007bb0 con 0x7faf9c104530 2024-03-11T19:09:45.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.670+0000 7faf927fc640 1 -- 172.21.15.29:0/1034879017 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faf8c007d10 con 0x7faf9c104530 2024-03-11T19:09:45.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.670+0000 7fafa356c640 1 -- 172.21.15.29:0/1034879017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf9c19f250 con 0x7faf9c104530 2024-03-11T19:09:45.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.670+0000 7faf927fc640 1 -- 172.21.15.29:0/1034879017 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c00f450 con 0x7faf9c104530 2024-03-11T19:09:45.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.670+0000 7fafa356c640 1 -- 172.21.15.29:0/1034879017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf9c19f760 con 0x7faf9c104530 2024-03-11T19:09:45.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.671+0000 7fafa356c640 1 -- 172.21.15.29:0/1034879017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf9c0fd100 con 0x7faf9c104530 2024-03-11T19:09:45.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.672+0000 7faf927fc640 1 -- 172.21.15.29:0/1034879017 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7faf8c004eb0 con 0x7faf9c104530 2024-03-11T19:09:45.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.672+0000 7faf927fc640 1 --2- 172.21.15.29:0/1034879017 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faf880a1a30 0x7faf880a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:45.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.672+0000 7fafa1308640 1 --2- 172.21.15.29:0/1034879017 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faf880a1a30 0x7faf880a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:45.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.672+0000 7faf927fc640 1 -- 172.21.15.29:0/1034879017 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7faf8c0cd410 con 0x7faf9c104530 2024-03-11T19:09:45.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.673+0000 7fafa1308640 1 --2- 172.21.15.29:0/1034879017 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faf880a1a30 0x7faf880a3ef0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7faf98005fd0 tx=0x7faf98005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:45.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.677+0000 7faf927fc640 1 -- 172.21.15.29:0/1034879017 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faf8c0966c0 con 0x7faf9c104530 2024-03-11T19:09:45.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.858+0000 7fafa356c640 1 -- 172.21.15.29:0/1034879017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7faf9c02ce40 con 0x7faf9c104530 2024-03-11T19:09:45.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.858+0000 7faf927fc640 1 -- 172.21.15.29:0/1034879017 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7faf8c099d80 con 0x7faf9c104530 2024-03-11T19:09:45.861 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-03-11T19:09:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.860+0000 7fafa356c640 1 -- 172.21.15.29:0/1034879017 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faf880a1a30 msgr2=0x7faf880a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.860+0000 7fafa356c640 1 --2- 172.21.15.29:0/1034879017 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faf880a1a30 0x7faf880a3ef0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7faf98005fd0 tx=0x7faf98005850 comp rx=0 tx=0).stop 2024-03-11T19:09:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.860+0000 7fafa356c640 1 -- 172.21.15.29:0/1034879017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf9c104530 msgr2=0x7faf9c197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.860+0000 7fafa356c640 1 --2- 172.21.15.29:0/1034879017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf9c104530 0x7faf9c197690 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7faf8c00c970 tx=0x7faf8c00ce30 comp rx=0 tx=0).stop 2024-03-11T19:09:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.860+0000 7fafa356c640 1 -- 172.21.15.29:0/1034879017 shutdown_connections 2024-03-11T19:09:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.860+0000 7fafa356c640 1 --2- 172.21.15.29:0/1034879017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf9c106e60 0x7faf9c19e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.860+0000 7fafa356c640 1 --2- 172.21.15.29:0/1034879017 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faf880a1a30 0x7faf880a3ef0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.860+0000 7fafa356c640 1 --2- 172.21.15.29:0/1034879017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf9c104530 0x7faf9c197690 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.860+0000 7fafa356c640 1 --2- 172.21.15.29:0/1034879017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf9c101c00 0x7faf9c197150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.860+0000 7fafa356c640 1 -- 172.21.15.29:0/1034879017 >> 172.21.15.29:0/1034879017 conn(0x7faf9c0fb610 msgr2=0x7faf9c1024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.860+0000 7fafa356c640 1 -- 172.21.15.29:0/1034879017 shutdown_connections 2024-03-11T19:09:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.860+0000 7fafa356c640 1 -- 172.21.15.29:0/1034879017 wait complete. 2024-03-11T19:09:45.874 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:09:45.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.973+0000 7fb5b031e640 1 -- 172.21.15.29:0/311897185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5a8101cd0 msgr2=0x7fb5a8108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:45.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.973+0000 7fb5b031e640 1 --2- 172.21.15.29:0/311897185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5a8101cd0 0x7fb5a8108560 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0009900 tx=0x7fb5a002bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:45.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.973+0000 7fb5b031e640 1 -- 172.21.15.29:0/311897185 shutdown_connections 2024-03-11T19:09:45.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.973+0000 7fb5b031e640 1 --2- 172.21.15.29:0/311897185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5a8101cd0 0x7fb5a8108560 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.973+0000 7fb5b031e640 1 --2- 172.21.15.29:0/311897185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5a8101310 0x7fb5a8101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.973+0000 7fb5b031e640 1 --2- 172.21.15.29:0/311897185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5a8100120 0x7fb5a8100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.973+0000 7fb5b031e640 1 -- 172.21.15.29:0/311897185 >> 172.21.15.29:0/311897185 conn(0x7fb5a80fb6b0 msgr2=0x7fb5a80fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.973+0000 7fb5b031e640 1 -- 172.21.15.29:0/311897185 shutdown_connections 2024-03-11T19:09:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.973+0000 7fb5b031e640 1 -- 172.21.15.29:0/311897185 wait complete. 2024-03-11T19:09:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.974+0000 7fb5b031e640 1 Processor -- start 2024-03-11T19:09:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.974+0000 7fb5b031e640 1 -- start start 2024-03-11T19:09:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.974+0000 7fb5b031e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5a8100120 0x7fb5a8197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.974+0000 7fb5b031e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5a8101310 0x7fb5a8197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.974+0000 7fb5b031e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5a8101cd0 0x7fb5a819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.974+0000 7fb5b031e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a819ebf0 con 0x7fb5a8100120 2024-03-11T19:09:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.974+0000 7fb5b031e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a819ed60 con 0x7fb5a8101cd0 2024-03-11T19:09:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.974+0000 7fb5b031e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a819ef20 con 0x7fb5a8101310 2024-03-11T19:09:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.974+0000 7fb5ae0ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5a8100120 0x7fb5a8197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.974+0000 7fb5ae0ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5a8100120 0x7fb5a8197100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49336/0 (socket says 172.21.15.29:49336) 2024-03-11T19:09:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.974+0000 7fb5ae0ba640 1 -- 172.21.15.29:0/2546104454 learned_addr learned my addr 172.21.15.29:0/2546104454 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.974+0000 7fb5ad8b9640 1 --2- 172.21.15.29:0/2546104454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5a8101310 0x7fb5a8197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.974+0000 7fb5ae8bb640 1 --2- 172.21.15.29:0/2546104454 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5a8101cd0 0x7fb5a819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.974+0000 7fb5ae0ba640 1 -- 172.21.15.29:0/2546104454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5a8101310 msgr2=0x7fb5a8197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.974+0000 7fb5ae0ba640 1 --2- 172.21.15.29:0/2546104454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5a8101310 0x7fb5a8197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.974+0000 7fb5ae0ba640 1 -- 172.21.15.29:0/2546104454 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5a8101cd0 msgr2=0x7fb5a819e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.974+0000 7fb5ae0ba640 1 --2- 172.21.15.29:0/2546104454 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5a8101cd0 0x7fb5a819e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.974+0000 7fb5ae0ba640 1 -- 172.21.15.29:0/2546104454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a0009560 con 0x7fb5a8100120 2024-03-11T19:09:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.974+0000 7fb5ae0ba640 1 --2- 172.21.15.29:0/2546104454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5a8100120 0x7fb5a8197100 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb59800ea80 tx=0x7fb59800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.975+0000 7fb5a77fe640 1 -- 172.21.15.29:0/2546104454 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59800ce40 con 0x7fb5a8100120 2024-03-11T19:09:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.975+0000 7fb5a77fe640 1 -- 172.21.15.29:0/2546104454 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb598011920 con 0x7fb5a8100120 2024-03-11T19:09:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.975+0000 7fb5b031e640 1 -- 172.21.15.29:0/2546104454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a819f200 con 0x7fb5a8100120 2024-03-11T19:09:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.975+0000 7fb5a77fe640 1 -- 172.21.15.29:0/2546104454 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59801aa70 con 0x7fb5a8100120 2024-03-11T19:09:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.975+0000 7fb5b031e640 1 -- 172.21.15.29:0/2546104454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a819f710 con 0x7fb5a8100120 2024-03-11T19:09:45.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.976+0000 7fb5b031e640 1 -- 172.21.15.29:0/2546104454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a810b970 con 0x7fb5a8100120 2024-03-11T19:09:45.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.976+0000 7fb5a77fe640 1 -- 172.21.15.29:0/2546104454 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb598011a90 con 0x7fb5a8100120 2024-03-11T19:09:45.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.977+0000 7fb5a77fe640 1 --2- 172.21.15.29:0/2546104454 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb5900a1a30 0x7fb5900a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.977+0000 7fb5ad8b9640 1 --2- 172.21.15.29:0/2546104454 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb5900a1a30 0x7fb5900a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.977+0000 7fb5a77fe640 1 -- 172.21.15.29:0/2546104454 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb5980cd970 con 0x7fb5a8100120 2024-03-11T19:09:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.978+0000 7fb5ad8b9640 1 --2- 172.21.15.29:0/2546104454 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb5900a1a30 0x7fb5900a3ef0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8198610 tx=0x7fb59c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:45.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:45.982+0000 7fb5a77fe640 1 -- 172.21.15.29:0/2546104454 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb598096b80 con 0x7fb5a8100120 2024-03-11T19:09:46.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.202+0000 7fb5b031e640 1 -- 172.21.15.29:0/2546104454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5a810bb00 con 0x7fb5a8100120 2024-03-11T19:09:46.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.203+0000 7fb5a77fe640 1 -- 172.21.15.29:0/2546104454 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+11385 (secure 0 0 0) 0x7fb59809a240 con 0x7fb5a8100120 2024-03-11T19:09:46.206 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:46.206 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:07:56.662890+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:09:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 26 2024-03-11T19:09:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.205+0000 7fb5b031e640 1 -- 172.21.15.29:0/2546104454 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb5900a1a30 msgr2=0x7fb5900a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.205+0000 7fb5b031e640 1 --2- 172.21.15.29:0/2546104454 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb5900a1a30 0x7fb5900a3ef0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8198610 tx=0x7fb59c00b410 comp rx=0 tx=0).stop 2024-03-11T19:09:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.205+0000 7fb5b031e640 1 -- 172.21.15.29:0/2546104454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5a8100120 msgr2=0x7fb5a8197100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.205+0000 7fb5b031e640 1 --2- 172.21.15.29:0/2546104454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5a8100120 0x7fb5a8197100 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb59800ea80 tx=0x7fb59800ef40 comp rx=0 tx=0).stop 2024-03-11T19:09:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.205+0000 7fb5b031e640 1 -- 172.21.15.29:0/2546104454 shutdown_connections 2024-03-11T19:09:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.205+0000 7fb5b031e640 1 --2- 172.21.15.29:0/2546104454 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5a8101cd0 0x7fb5a819e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.205+0000 7fb5b031e640 1 --2- 172.21.15.29:0/2546104454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5a8101310 0x7fb5a8197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.205+0000 7fb5b031e640 1 --2- 172.21.15.29:0/2546104454 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb5900a1a30 0x7fb5900a3ef0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.205+0000 7fb5b031e640 1 --2- 172.21.15.29:0/2546104454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5a8100120 0x7fb5a8197100 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.205+0000 7fb5b031e640 1 -- 172.21.15.29:0/2546104454 >> 172.21.15.29:0/2546104454 conn(0x7fb5a80fb6b0 msgr2=0x7fb5a80697b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.205+0000 7fb5b031e640 1 -- 172.21.15.29:0/2546104454 shutdown_connections 2024-03-11T19:09:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.205+0000 7fb5b031e640 1 -- 172.21.15.29:0/2546104454 wait complete. 2024-03-11T19:09:46.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:09:46.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.318+0000 7f18e1948640 1 -- 172.21.15.29:0/3286776098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18dc1023a0 msgr2=0x7f18dc102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.318+0000 7f18e1948640 1 --2- 172.21.15.29:0/3286776098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18dc1023a0 0x7f18dc102820 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f18d0009900 tx=0x7f18d002bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.318+0000 7f18e1948640 1 -- 172.21.15.29:0/3286776098 shutdown_connections 2024-03-11T19:09:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.318+0000 7f18e1948640 1 --2- 172.21.15.29:0/3286776098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18dc102d60 0x7f18dc1095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.318+0000 7f18e1948640 1 --2- 172.21.15.29:0/3286776098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18dc1023a0 0x7f18dc102820 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.318+0000 7f18e1948640 1 --2- 172.21.15.29:0/3286776098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18dc1011b0 0x7f18dc1015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.318+0000 7f18e1948640 1 -- 172.21.15.29:0/3286776098 >> 172.21.15.29:0/3286776098 conn(0x7f18dc0fc720 msgr2=0x7f18dc0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.318+0000 7f18e1948640 1 -- 172.21.15.29:0/3286776098 shutdown_connections 2024-03-11T19:09:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.318+0000 7f18e1948640 1 -- 172.21.15.29:0/3286776098 wait complete. 2024-03-11T19:09:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.319+0000 7f18e1948640 1 Processor -- start 2024-03-11T19:09:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.319+0000 7f18e1948640 1 -- start start 2024-03-11T19:09:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.319+0000 7f18e1948640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18dc1011b0 0x7f18dc197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.319+0000 7f18e1948640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18dc1023a0 0x7f18dc1976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.319+0000 7f18e1948640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18dc102d60 0x7f18dc19e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.319+0000 7f18e1948640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc19ec60 con 0x7f18dc1011b0 2024-03-11T19:09:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.319+0000 7f18e1948640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc19edd0 con 0x7f18dc102d60 2024-03-11T19:09:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.319+0000 7f18e1948640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc19ef90 con 0x7f18dc1023a0 2024-03-11T19:09:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.319+0000 7f18daffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18dc1011b0 0x7f18dc197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.319+0000 7f18daffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18dc1011b0 0x7f18dc197170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49344/0 (socket says 172.21.15.29:49344) 2024-03-11T19:09:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.319+0000 7f18daffd640 1 -- 172.21.15.29:0/2674493619 learned_addr learned my addr 172.21.15.29:0/2674493619 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.319+0000 7f18da7fc640 1 --2- 172.21.15.29:0/2674493619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18dc1023a0 0x7f18dc1976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.319+0000 7f18db7fe640 1 --2- 172.21.15.29:0/2674493619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18dc102d60 0x7f18dc19e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.319+0000 7f18daffd640 1 -- 172.21.15.29:0/2674493619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18dc1023a0 msgr2=0x7f18dc1976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.319+0000 7f18daffd640 1 --2- 172.21.15.29:0/2674493619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18dc1023a0 0x7f18dc1976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.319+0000 7f18daffd640 1 -- 172.21.15.29:0/2674493619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18dc102d60 msgr2=0x7f18dc19e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.319+0000 7f18daffd640 1 --2- 172.21.15.29:0/2674493619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18dc102d60 0x7f18dc19e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.319+0000 7f18daffd640 1 -- 172.21.15.29:0/2674493619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d0009560 con 0x7f18dc1011b0 2024-03-11T19:09:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.319+0000 7f18da7fc640 1 --2- 172.21.15.29:0/2674493619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18dc1023a0 0x7f18dc1976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.320+0000 7f18daffd640 1 --2- 172.21.15.29:0/2674493619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18dc1011b0 0x7f18dc197170 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00d770 tx=0x7f18cc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.320+0000 7f18e0946640 1 -- 172.21.15.29:0/2674493619 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc009c30 con 0x7f18dc1011b0 2024-03-11T19:09:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.320+0000 7f18e1948640 1 -- 172.21.15.29:0/2674493619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18dc19f270 con 0x7f18dc1011b0 2024-03-11T19:09:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.320+0000 7f18e0946640 1 -- 172.21.15.29:0/2674493619 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f18cc0044d0 con 0x7f18dc1011b0 2024-03-11T19:09:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.320+0000 7f18e0946640 1 -- 172.21.15.29:0/2674493619 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc003de0 con 0x7f18dc1011b0 2024-03-11T19:09:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.320+0000 7f18e1948640 1 -- 172.21.15.29:0/2674493619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18dc19f7d0 con 0x7f18dc1011b0 2024-03-11T19:09:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.321+0000 7f18e1948640 1 -- 172.21.15.29:0/2674493619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18dc1085a0 con 0x7f18dc1011b0 2024-03-11T19:09:46.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.322+0000 7f18e0946640 1 -- 172.21.15.29:0/2674493619 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f18cc003f40 con 0x7f18dc1011b0 2024-03-11T19:09:46.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.322+0000 7f18e0946640 1 --2- 172.21.15.29:0/2674493619 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f18c40a1a30 0x7f18c40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:46.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.323+0000 7f18da7fc640 1 --2- 172.21.15.29:0/2674493619 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f18c40a1a30 0x7f18c40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:46.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.323+0000 7f18e0946640 1 -- 172.21.15.29:0/2674493619 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f18cc0cd250 con 0x7f18dc1011b0 2024-03-11T19:09:46.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.323+0000 7f18da7fc640 1 --2- 172.21.15.29:0/2674493619 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f18c40a1a30 0x7f18c40a3ef0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f18d0036040 tx=0x7f18d0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.326+0000 7f18e0946640 1 -- 172.21.15.29:0/2674493619 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f18cc096460 con 0x7f18dc1011b0 2024-03-11T19:09:46.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:46 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2181388643' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:09:46.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:46 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1737154623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:09:46.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:46 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1034879017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:09:46.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.509+0000 7f18e1948640 1 -- 172.21.15.29:0/2674493619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18dc10ca70 con 0x7f18dc1011b0 2024-03-11T19:09:46.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.510+0000 7f18e0946640 1 -- 172.21.15.29:0/2674493619 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22559 (secure 0 0 0) 0x7f18cc017090 con 0x7f18dc1011b0 2024-03-11T19:09:46.513 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:46.514 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":89,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:08:44.981806+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:07:47.563184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:07:48.282461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:09:46.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.512+0000 7f18e1948640 1 -- 172.21.15.29:0/2674493619 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f18c40a1a30 msgr2=0x7f18c40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.512+0000 7f18e1948640 1 --2- 172.21.15.29:0/2674493619 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f18c40a1a30 0x7f18c40a3ef0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f18d0036040 tx=0x7f18d0000bc0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.512+0000 7f18e1948640 1 -- 172.21.15.29:0/2674493619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18dc1011b0 msgr2=0x7f18dc197170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.512+0000 7f18e1948640 1 --2- 172.21.15.29:0/2674493619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18dc1011b0 0x7f18dc197170 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00d770 tx=0x7f18cc00dc30 comp rx=0 tx=0).stop 2024-03-11T19:09:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.512+0000 7f18e1948640 1 -- 172.21.15.29:0/2674493619 shutdown_connections 2024-03-11T19:09:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.512+0000 7f18e1948640 1 --2- 172.21.15.29:0/2674493619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18dc102d60 0x7f18dc19e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.512+0000 7f18e1948640 1 --2- 172.21.15.29:0/2674493619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18dc1023a0 0x7f18dc1976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.512+0000 7f18e1948640 1 --2- 172.21.15.29:0/2674493619 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f18c40a1a30 0x7f18c40a3ef0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.512+0000 7f18e1948640 1 --2- 172.21.15.29:0/2674493619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f18dc1011b0 0x7f18dc197170 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.512+0000 7f18e1948640 1 -- 172.21.15.29:0/2674493619 >> 172.21.15.29:0/2674493619 conn(0x7f18dc0fc720 msgr2=0x7f18dc105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.513+0000 7f18e1948640 1 -- 172.21.15.29:0/2674493619 shutdown_connections 2024-03-11T19:09:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.513+0000 7f18e1948640 1 -- 172.21.15.29:0/2674493619 wait complete. 2024-03-11T19:09:46.525 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:46 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2181388643' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:09:46.526 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:46 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1737154623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:09:46.526 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:46 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1034879017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:09:46.527 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:03:09.860019+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T19:07:47.563184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T19:07:48.282461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T19:09:46.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:09:46.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:46 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2181388643' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:09:46.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:46 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1737154623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:09:46.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:46 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1034879017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:09:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.630+0000 7f3485dbc640 1 -- 172.21.15.29:0/2235383409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3480101c50 msgr2=0x7f34801084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.630+0000 7f3485dbc640 1 --2- 172.21.15.29:0/2235383409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3480101c50 0x7f34801084e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3478009ca0 tx=0x7f347802c0f0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.630+0000 7f3485dbc640 1 -- 172.21.15.29:0/2235383409 shutdown_connections 2024-03-11T19:09:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.630+0000 7f3485dbc640 1 --2- 172.21.15.29:0/2235383409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3480101c50 0x7f34801084e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.630+0000 7f3485dbc640 1 --2- 172.21.15.29:0/2235383409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3480101290 0x7f3480101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.630+0000 7f3485dbc640 1 --2- 172.21.15.29:0/2235383409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f34801000a0 0x7f34801004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.630+0000 7f3485dbc640 1 -- 172.21.15.29:0/2235383409 >> 172.21.15.29:0/2235383409 conn(0x7f34800fb630 msgr2=0x7f34800fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.630+0000 7f3485dbc640 1 -- 172.21.15.29:0/2235383409 shutdown_connections 2024-03-11T19:09:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.630+0000 7f3485dbc640 1 -- 172.21.15.29:0/2235383409 wait complete. 2024-03-11T19:09:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.630+0000 7f3485dbc640 1 Processor -- start 2024-03-11T19:09:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.630+0000 7f3485dbc640 1 -- start start 2024-03-11T19:09:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f3485dbc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34801000a0 0x7f3480197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f3485dbc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3480101290 0x7f3480197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f3485dbc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3480101c50 0x7f348019e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f3485dbc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348019ec10 con 0x7f34801000a0 2024-03-11T19:09:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f3485dbc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348019ed80 con 0x7f3480101c50 2024-03-11T19:09:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f3485dbc640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348019ef40 con 0x7f3480101290 2024-03-11T19:09:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f347f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34801000a0 0x7f3480197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f347f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34801000a0 0x7f3480197120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49364/0 (socket says 172.21.15.29:49364) 2024-03-11T19:09:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f347f7fe640 1 -- 172.21.15.29:0/1401870686 learned_addr learned my addr 172.21.15.29:0/1401870686 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f347effd640 1 --2- 172.21.15.29:0/1401870686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3480101290 0x7f3480197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f347ffff640 1 --2- 172.21.15.29:0/1401870686 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3480101c50 0x7f348019e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f347f7fe640 1 -- 172.21.15.29:0/1401870686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3480101290 msgr2=0x7f3480197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f347f7fe640 1 --2- 172.21.15.29:0/1401870686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3480101290 0x7f3480197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f347f7fe640 1 -- 172.21.15.29:0/1401870686 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3480101c50 msgr2=0x7f348019e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f347f7fe640 1 --2- 172.21.15.29:0/1401870686 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3480101c50 0x7f348019e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f347f7fe640 1 -- 172.21.15.29:0/1401870686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3478009900 con 0x7f34801000a0 2024-03-11T19:09:46.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f347ffff640 1 --2- 172.21.15.29:0/1401870686 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3480101c50 0x7f348019e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:46.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f347f7fe640 1 --2- 172.21.15.29:0/1401870686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34801000a0 0x7f3480197120 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f347000d770 tx=0x7f347000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:46.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f347cff9640 1 -- 172.21.15.29:0/1401870686 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3470009c30 con 0x7f34801000a0 2024-03-11T19:09:46.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f347cff9640 1 -- 172.21.15.29:0/1401870686 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f34700044d0 con 0x7f34801000a0 2024-03-11T19:09:46.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f3485dbc640 1 -- 172.21.15.29:0/1401870686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f348019f220 con 0x7f34801000a0 2024-03-11T19:09:46.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f347cff9640 1 -- 172.21.15.29:0/1401870686 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3470003de0 con 0x7f34801000a0 2024-03-11T19:09:46.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.631+0000 7f3485dbc640 1 -- 172.21.15.29:0/1401870686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f348019f730 con 0x7f34801000a0 2024-03-11T19:09:46.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.632+0000 7f3485dbc640 1 -- 172.21.15.29:0/1401870686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34801073c0 con 0x7f34801000a0 2024-03-11T19:09:46.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.633+0000 7f347cff9640 1 -- 172.21.15.29:0/1401870686 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3470003f40 con 0x7f34801000a0 2024-03-11T19:09:46.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.634+0000 7f347cff9640 1 --2- 172.21.15.29:0/1401870686 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f34680a1a30 0x7f34680a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:46.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.634+0000 7f347effd640 1 --2- 172.21.15.29:0/1401870686 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f34680a1a30 0x7f34680a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:46.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.634+0000 7f347cff9640 1 -- 172.21.15.29:0/1401870686 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f34700cce20 con 0x7f34801000a0 2024-03-11T19:09:46.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.634+0000 7f347effd640 1 --2- 172.21.15.29:0/1401870686 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f34680a1a30 0x7f34680a3ef0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3474005fd0 tx=0x7f3474005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:46.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.638+0000 7f347cff9640 1 -- 172.21.15.29:0/1401870686 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3470096030 con 0x7f34801000a0 2024-03-11T19:09:46.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.821+0000 7f3485dbc640 1 -- 172.21.15.29:0/1401870686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f348002ce40 con 0x7f34801000a0 2024-03-11T19:09:46.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.822+0000 7f347cff9640 1 -- 172.21.15.29:0/1401870686 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f34700200a0 con 0x7f34801000a0 2024-03-11T19:09:46.825 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-11T19:09:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.823+0000 7f3485dbc640 1 -- 172.21.15.29:0/1401870686 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f34680a1a30 msgr2=0x7f34680a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.823+0000 7f3485dbc640 1 --2- 172.21.15.29:0/1401870686 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f34680a1a30 0x7f34680a3ef0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3474005fd0 tx=0x7f3474005850 comp rx=0 tx=0).stop 2024-03-11T19:09:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.823+0000 7f3485dbc640 1 -- 172.21.15.29:0/1401870686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34801000a0 msgr2=0x7f3480197120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.823+0000 7f3485dbc640 1 --2- 172.21.15.29:0/1401870686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34801000a0 0x7f3480197120 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f347000d770 tx=0x7f347000dc30 comp rx=0 tx=0).stop 2024-03-11T19:09:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.823+0000 7f3485dbc640 1 -- 172.21.15.29:0/1401870686 shutdown_connections 2024-03-11T19:09:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.823+0000 7f3485dbc640 1 --2- 172.21.15.29:0/1401870686 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3480101c50 0x7f348019e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.823+0000 7f3485dbc640 1 --2- 172.21.15.29:0/1401870686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3480101290 0x7f3480197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.823+0000 7f3485dbc640 1 --2- 172.21.15.29:0/1401870686 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f34680a1a30 0x7f34680a3ef0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.823+0000 7f3485dbc640 1 --2- 172.21.15.29:0/1401870686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34801000a0 0x7f3480197120 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.824+0000 7f3485dbc640 1 -- 172.21.15.29:0/1401870686 >> 172.21.15.29:0/1401870686 conn(0x7f34800fb630 msgr2=0x7f3480104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.824+0000 7f3485dbc640 1 -- 172.21.15.29:0/1401870686 shutdown_connections 2024-03-11T19:09:46.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.824+0000 7f3485dbc640 1 -- 172.21.15.29:0/1401870686 wait complete. 2024-03-11T19:09:46.838 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T19:09:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.941+0000 7fbf902eb640 1 -- 172.21.15.29:0/2143278866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf88101c30 msgr2=0x7fbf881084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.941+0000 7fbf902eb640 1 --2- 172.21.15.29:0/2143278866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf88101c30 0x7fbf881084c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009ca0 tx=0x7fbf8002c0f0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.941+0000 7fbf902eb640 1 -- 172.21.15.29:0/2143278866 shutdown_connections 2024-03-11T19:09:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.941+0000 7fbf902eb640 1 --2- 172.21.15.29:0/2143278866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf88101c30 0x7fbf881084c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.941+0000 7fbf902eb640 1 --2- 172.21.15.29:0/2143278866 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf88101270 0x7fbf881016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.941+0000 7fbf902eb640 1 --2- 172.21.15.29:0/2143278866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf88100080 0x7fbf881004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.941+0000 7fbf902eb640 1 -- 172.21.15.29:0/2143278866 >> 172.21.15.29:0/2143278866 conn(0x7fbf880fb610 msgr2=0x7fbf880fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.941+0000 7fbf902eb640 1 -- 172.21.15.29:0/2143278866 shutdown_connections 2024-03-11T19:09:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.941+0000 7fbf902eb640 1 -- 172.21.15.29:0/2143278866 wait complete. 2024-03-11T19:09:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.941+0000 7fbf902eb640 1 Processor -- start 2024-03-11T19:09:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.941+0000 7fbf902eb640 1 -- start start 2024-03-11T19:09:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.941+0000 7fbf902eb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf88100080 0x7fbf88068e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.941+0000 7fbf902eb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf88101270 0x7fbf8819be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.941+0000 7fbf902eb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf88101c30 0x7fbf8819e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.941+0000 7fbf902eb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8819ec30 con 0x7fbf88101270 2024-03-11T19:09:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.941+0000 7fbf902eb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8819eda0 con 0x7fbf88101c30 2024-03-11T19:09:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.941+0000 7fbf902eb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8819ef60 con 0x7fbf88100080 2024-03-11T19:09:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.942+0000 7fbf8d886640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf88101270 0x7fbf8819be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.942+0000 7fbf8d886640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf88101270 0x7fbf8819be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49380/0 (socket says 172.21.15.29:49380) 2024-03-11T19:09:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.942+0000 7fbf8d886640 1 -- 172.21.15.29:0/1259552430 learned_addr learned my addr 172.21.15.29:0/1259552430 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.942+0000 7fbf8e087640 1 --2- 172.21.15.29:0/1259552430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf88100080 0x7fbf88068e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.942+0000 7fbf8e888640 1 --2- 172.21.15.29:0/1259552430 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf88101c30 0x7fbf8819e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.942+0000 7fbf8d886640 1 -- 172.21.15.29:0/1259552430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf88100080 msgr2=0x7fbf88068e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.942+0000 7fbf8d886640 1 --2- 172.21.15.29:0/1259552430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf88100080 0x7fbf88068e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.942+0000 7fbf8d886640 1 -- 172.21.15.29:0/1259552430 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf88101c30 msgr2=0x7fbf8819e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.942+0000 7fbf8d886640 1 --2- 172.21.15.29:0/1259552430 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf88101c30 0x7fbf8819e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:46.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.942+0000 7fbf8d886640 1 -- 172.21.15.29:0/1259552430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf80009900 con 0x7fbf88101270 2024-03-11T19:09:46.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.942+0000 7fbf8d886640 1 --2- 172.21.15.29:0/1259552430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf88101270 0x7fbf8819be60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fbf7800e9f0 tx=0x7fbf7800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:46.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.942+0000 7fbf7f7fe640 1 -- 172.21.15.29:0/1259552430 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf78009700 con 0x7fbf88101270 2024-03-11T19:09:46.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.942+0000 7fbf7f7fe640 1 -- 172.21.15.29:0/1259552430 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbf7800cd80 con 0x7fbf88101270 2024-03-11T19:09:46.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.942+0000 7fbf902eb640 1 -- 172.21.15.29:0/1259552430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf8819f240 con 0x7fbf88101270 2024-03-11T19:09:46.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.942+0000 7fbf7f7fe640 1 -- 172.21.15.29:0/1259552430 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf78018840 con 0x7fbf88101270 2024-03-11T19:09:46.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.942+0000 7fbf902eb640 1 -- 172.21.15.29:0/1259552430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf8819f7a0 con 0x7fbf88101270 2024-03-11T19:09:46.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.943+0000 7fbf902eb640 1 -- 172.21.15.29:0/1259552430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf881073c0 con 0x7fbf88101270 2024-03-11T19:09:46.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.944+0000 7fbf7f7fe640 1 -- 172.21.15.29:0/1259552430 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fbf780189a0 con 0x7fbf88101270 2024-03-11T19:09:46.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.945+0000 7fbf7f7fe640 1 --2- 172.21.15.29:0/1259552430 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbf740a1ab0 0x7fbf740a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:46.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.945+0000 7fbf8e087640 1 --2- 172.21.15.29:0/1259552430 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbf740a1ab0 0x7fbf740a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:46.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.945+0000 7fbf7f7fe640 1 -- 172.21.15.29:0/1259552430 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fbf78014070 con 0x7fbf88101270 2024-03-11T19:09:46.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.945+0000 7fbf8e087640 1 --2- 172.21.15.29:0/1259552430 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbf740a1ab0 0x7fbf740a3f70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbf881010d0 tx=0x7fbf84005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:46.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:46.950+0000 7fbf7f7fe640 1 -- 172.21.15.29:0/1259552430 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbf78052d20 con 0x7fbf88101270 2024-03-11T19:09:47.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.131+0000 7fbf902eb640 1 -- 172.21.15.29:0/1259552430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbf8802ce40 con 0x7fbf88101270 2024-03-11T19:09:47.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.131+0000 7fbf7f7fe640 1 -- 172.21.15.29:0/1259552430 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbf78096940 con 0x7fbf88101270 2024-03-11T19:09:47.134 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-03-11T19:09:47.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.133+0000 7fbf902eb640 1 -- 172.21.15.29:0/1259552430 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbf740a1ab0 msgr2=0x7fbf740a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:47.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.133+0000 7fbf902eb640 1 --2- 172.21.15.29:0/1259552430 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbf740a1ab0 0x7fbf740a3f70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbf881010d0 tx=0x7fbf84005f90 comp rx=0 tx=0).stop 2024-03-11T19:09:47.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.133+0000 7fbf902eb640 1 -- 172.21.15.29:0/1259552430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf88101270 msgr2=0x7fbf8819be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:47.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.133+0000 7fbf902eb640 1 --2- 172.21.15.29:0/1259552430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf88101270 0x7fbf8819be60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fbf7800e9f0 tx=0x7fbf7800eeb0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.133+0000 7fbf902eb640 1 -- 172.21.15.29:0/1259552430 shutdown_connections 2024-03-11T19:09:47.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.133+0000 7fbf902eb640 1 --2- 172.21.15.29:0/1259552430 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf88101c30 0x7fbf8819e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.133+0000 7fbf902eb640 1 --2- 172.21.15.29:0/1259552430 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbf740a1ab0 0x7fbf740a3f70 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.133+0000 7fbf902eb640 1 --2- 172.21.15.29:0/1259552430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf88101270 0x7fbf8819be60 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.133+0000 7fbf902eb640 1 --2- 172.21.15.29:0/1259552430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf88100080 0x7fbf88068e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.133+0000 7fbf902eb640 1 -- 172.21.15.29:0/1259552430 >> 172.21.15.29:0/1259552430 conn(0x7fbf880fb610 msgr2=0x7fbf88104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:47.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.133+0000 7fbf902eb640 1 -- 172.21.15.29:0/1259552430 shutdown_connections 2024-03-11T19:09:47.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.133+0000 7fbf902eb640 1 -- 172.21.15.29:0/1259552430 wait complete. 2024-03-11T19:09:47.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T19:09:47.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.251+0000 7fe3ac12e640 1 -- 172.21.15.29:0/3872013562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3a41000a0 msgr2=0x7fe3a41004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:47.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.251+0000 7fe3ac12e640 1 --2- 172.21.15.29:0/3872013562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3a41000a0 0x7fe3a41004c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0009900 tx=0x7fe3a002bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:47.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.251+0000 7fe3ac12e640 1 -- 172.21.15.29:0/3872013562 shutdown_connections 2024-03-11T19:09:47.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.251+0000 7fe3ac12e640 1 --2- 172.21.15.29:0/3872013562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3a4101c50 0x7fe3a41084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.251+0000 7fe3ac12e640 1 --2- 172.21.15.29:0/3872013562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3a4101290 0x7fe3a4101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.251+0000 7fe3ac12e640 1 --2- 172.21.15.29:0/3872013562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3a41000a0 0x7fe3a41004c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.251+0000 7fe3ac12e640 1 -- 172.21.15.29:0/3872013562 >> 172.21.15.29:0/3872013562 conn(0x7fe3a40fb630 msgr2=0x7fe3a40fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:47.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.251+0000 7fe3ac12e640 1 -- 172.21.15.29:0/3872013562 shutdown_connections 2024-03-11T19:09:47.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.251+0000 7fe3ac12e640 1 -- 172.21.15.29:0/3872013562 wait complete. 2024-03-11T19:09:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.251+0000 7fe3ac12e640 1 Processor -- start 2024-03-11T19:09:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.251+0000 7fe3ac12e640 1 -- start start 2024-03-11T19:09:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.251+0000 7fe3ac12e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3a41000a0 0x7fe3a4197190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.251+0000 7fe3ac12e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3a4101290 0x7fe3a41976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.251+0000 7fe3ac12e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3a4101c50 0x7fe3a419e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.251+0000 7fe3ac12e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a419ec80 con 0x7fe3a41000a0 2024-03-11T19:09:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.251+0000 7fe3ac12e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a419edf0 con 0x7fe3a4101c50 2024-03-11T19:09:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.251+0000 7fe3ac12e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a419efb0 con 0x7fe3a4101290 2024-03-11T19:09:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.252+0000 7fe3a9eca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3a41000a0 0x7fe3a4197190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.252+0000 7fe3a9eca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3a41000a0 0x7fe3a4197190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49394/0 (socket says 172.21.15.29:49394) 2024-03-11T19:09:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.252+0000 7fe3a9eca640 1 -- 172.21.15.29:0/1175398603 learned_addr learned my addr 172.21.15.29:0/1175398603 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.252+0000 7fe3a96c9640 1 --2- 172.21.15.29:0/1175398603 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3a4101290 0x7fe3a41976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.252+0000 7fe3a9eca640 1 -- 172.21.15.29:0/1175398603 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3a4101290 msgr2=0x7fe3a41976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.252+0000 7fe3a9eca640 1 --2- 172.21.15.29:0/1175398603 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3a4101290 0x7fe3a41976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.252+0000 7fe3a9eca640 1 -- 172.21.15.29:0/1175398603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3a4101c50 msgr2=0x7fe3a419e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:47.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.252+0000 7fe3a9eca640 1 --2- 172.21.15.29:0/1175398603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3a4101c50 0x7fe3a419e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.252+0000 7fe3a9eca640 1 -- 172.21.15.29:0/1175398603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a0009560 con 0x7fe3a41000a0 2024-03-11T19:09:47.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.252+0000 7fe3a9eca640 1 --2- 172.21.15.29:0/1175398603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3a41000a0 0x7fe3a4197190 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe3a002c7d0 tx=0x7fe3a0004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:47.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.252+0000 7fe39affd640 1 -- 172.21.15.29:0/1175398603 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0003800 con 0x7fe3a41000a0 2024-03-11T19:09:47.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.252+0000 7fe39affd640 1 -- 172.21.15.29:0/1175398603 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe3a002de90 con 0x7fe3a41000a0 2024-03-11T19:09:47.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.252+0000 7fe3ac12e640 1 -- 172.21.15.29:0/1175398603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a419f230 con 0x7fe3a41000a0 2024-03-11T19:09:47.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.252+0000 7fe39affd640 1 -- 172.21.15.29:0/1175398603 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a00055f0 con 0x7fe3a41000a0 2024-03-11T19:09:47.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.252+0000 7fe3ac12e640 1 -- 172.21.15.29:0/1175398603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a419f7c0 con 0x7fe3a41000a0 2024-03-11T19:09:47.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.253+0000 7fe3ac12e640 1 -- 172.21.15.29:0/1175398603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a410b9a0 con 0x7fe3a41000a0 2024-03-11T19:09:47.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.254+0000 7fe39affd640 1 -- 172.21.15.29:0/1175398603 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe3a0004dc0 con 0x7fe3a41000a0 2024-03-11T19:09:47.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.255+0000 7fe39affd640 1 --2- 172.21.15.29:0/1175398603 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3900a1a30 0x7fe3900a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:47.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.255+0000 7fe3a96c9640 1 --2- 172.21.15.29:0/1175398603 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3900a1a30 0x7fe3900a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:47.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.255+0000 7fe39affd640 1 -- 172.21.15.29:0/1175398603 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe3a00ef790 con 0x7fe3a41000a0 2024-03-11T19:09:47.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.255+0000 7fe3a96c9640 1 --2- 172.21.15.29:0/1175398603 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3900a1a30 0x7fe3900a3ef0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe394005e10 tx=0x7fe394005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:47.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.259+0000 7fe39affd640 1 -- 172.21.15.29:0/1175398603 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe3a00b8a40 con 0x7fe3a41000a0 2024-03-11T19:09:47.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:47 smithi029 ceph-mon[32439]: pgmap v350: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:47.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:47 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2546104454' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:09:47.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:47 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2674493619' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:47.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:47 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1401870686' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:09:47.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:47 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1259552430' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:09:47.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.440+0000 7fe3ac12e640 1 -- 172.21.15.29:0/1175398603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe3a419fd60 con 0x7fe3a41000a0 2024-03-11T19:09:47.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.440+0000 7fe39affd640 1 -- 172.21.15.29:0/1175398603 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe3a00bc100 con 0x7fe3a41000a0 2024-03-11T19:09:47.444 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-03-11T19:09:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.442+0000 7fe3ac12e640 1 -- 172.21.15.29:0/1175398603 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3900a1a30 msgr2=0x7fe3900a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.443+0000 7fe3ac12e640 1 --2- 172.21.15.29:0/1175398603 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3900a1a30 0x7fe3900a3ef0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe394005e10 tx=0x7fe394005da0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.443+0000 7fe3ac12e640 1 -- 172.21.15.29:0/1175398603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3a41000a0 msgr2=0x7fe3a4197190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.443+0000 7fe3ac12e640 1 --2- 172.21.15.29:0/1175398603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3a41000a0 0x7fe3a4197190 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe3a002c7d0 tx=0x7fe3a0004360 comp rx=0 tx=0).stop 2024-03-11T19:09:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.443+0000 7fe3ac12e640 1 -- 172.21.15.29:0/1175398603 shutdown_connections 2024-03-11T19:09:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.443+0000 7fe3ac12e640 1 --2- 172.21.15.29:0/1175398603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3a4101c50 0x7fe3a419e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.443+0000 7fe3ac12e640 1 --2- 172.21.15.29:0/1175398603 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3a4101290 0x7fe3a41976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.443+0000 7fe3ac12e640 1 --2- 172.21.15.29:0/1175398603 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3900a1a30 0x7fe3900a3ef0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.443+0000 7fe3ac12e640 1 --2- 172.21.15.29:0/1175398603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3a41000a0 0x7fe3a4197190 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.443+0000 7fe3ac12e640 1 -- 172.21.15.29:0/1175398603 >> 172.21.15.29:0/1175398603 conn(0x7fe3a40fb630 msgr2=0x7fe3a40fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:47.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.443+0000 7fe3ac12e640 1 -- 172.21.15.29:0/1175398603 shutdown_connections 2024-03-11T19:09:47.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.443+0000 7fe3ac12e640 1 -- 172.21.15.29:0/1175398603 wait complete. 2024-03-11T19:09:47.457 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:09:47.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:47 smithi190 ceph-mon[36292]: pgmap v350: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:47.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:47 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2546104454' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:09:47.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:47 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2674493619' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:47.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:47 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1401870686' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:09:47.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:47 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1259552430' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:09:47.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.556+0000 7f8376e64640 1 -- 172.21.15.29:0/3889921489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8370100a40 msgr2=0x7f8370100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.556+0000 7f8376e64640 1 --2- 172.21.15.29:0/3889921489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8370100a40 0x7f8370100ec0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f835c009900 tx=0x7f835c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.556+0000 7f8376e64640 1 -- 172.21.15.29:0/3889921489 shutdown_connections 2024-03-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.556+0000 7f8376e64640 1 --2- 172.21.15.29:0/3889921489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8370101400 0x7f8370107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.556+0000 7f8376e64640 1 --2- 172.21.15.29:0/3889921489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8370100a40 0x7f8370100ec0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.556+0000 7f8376e64640 1 --2- 172.21.15.29:0/3889921489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83700ff850 0x7f83700ffc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.556+0000 7f8376e64640 1 -- 172.21.15.29:0/3889921489 >> 172.21.15.29:0/3889921489 conn(0x7f83700fadc0 msgr2=0x7f83700fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.556+0000 7f8376e64640 1 -- 172.21.15.29:0/3889921489 shutdown_connections 2024-03-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.556+0000 7f8376e64640 1 -- 172.21.15.29:0/3889921489 wait complete. 2024-03-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.556+0000 7f8376e64640 1 Processor -- start 2024-03-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.556+0000 7f8376e64640 1 -- start start 2024-03-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f8376e64640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83700ff850 0x7f8370108e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f8376e64640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8370100a40 0x7f837019bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f8376e64640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8370101400 0x7f837019e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f8376e64640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837019ed90 con 0x7f83700ff850 2024-03-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f8376e64640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837019ef00 con 0x7f8370100a40 2024-03-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f8376e64640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837019f0c0 con 0x7f8370101400 2024-03-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f8374c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83700ff850 0x7f8370108e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f8374c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83700ff850 0x7f8370108e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49410/0 (socket says 172.21.15.29:49410) 2024-03-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f8374c00640 1 -- 172.21.15.29:0/2022853446 learned_addr learned my addr 172.21.15.29:0/2022853446 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f8375401640 1 --2- 172.21.15.29:0/2022853446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8370101400 0x7f837019e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f836ffff640 1 --2- 172.21.15.29:0/2022853446 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8370100a40 0x7f837019bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f8374c00640 1 -- 172.21.15.29:0/2022853446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8370101400 msgr2=0x7f837019e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f8374c00640 1 --2- 172.21.15.29:0/2022853446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8370101400 0x7f837019e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f8374c00640 1 -- 172.21.15.29:0/2022853446 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8370100a40 msgr2=0x7f837019bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f8374c00640 1 --2- 172.21.15.29:0/2022853446 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8370100a40 0x7f837019bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f8374c00640 1 -- 172.21.15.29:0/2022853446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835c009560 con 0x7f83700ff850 2024-03-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f836ffff640 1 --2- 172.21.15.29:0/2022853446 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8370100a40 0x7f837019bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f8374c00640 1 --2- 172.21.15.29:0/2022853446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83700ff850 0x7f8370108e80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f836400b450 tx=0x7f836400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:47.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f836dffb640 1 -- 172.21.15.29:0/2022853446 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8364011890 con 0x7f83700ff850 2024-03-11T19:09:47.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f8376e64640 1 -- 172.21.15.29:0/2022853446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f837019f340 con 0x7f83700ff850 2024-03-11T19:09:47.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f836dffb640 1 -- 172.21.15.29:0/2022853446 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f83640103e0 con 0x7f83700ff850 2024-03-11T19:09:47.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f836dffb640 1 -- 172.21.15.29:0/2022853446 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8364011ea0 con 0x7f83700ff850 2024-03-11T19:09:47.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.557+0000 7f8376e64640 1 -- 172.21.15.29:0/2022853446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f837019f920 con 0x7f83700ff850 2024-03-11T19:09:47.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.559+0000 7f8376e64640 1 -- 172.21.15.29:0/2022853446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8370106c90 con 0x7f83700ff850 2024-03-11T19:09:47.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.559+0000 7f836dffb640 1 -- 172.21.15.29:0/2022853446 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8364010550 con 0x7f83700ff850 2024-03-11T19:09:47.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.560+0000 7f836dffb640 1 --2- 172.21.15.29:0/2022853446 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f83600a1a30 0x7f83600a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:47.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.560+0000 7f836dffb640 1 -- 172.21.15.29:0/2022853446 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f83640cd1c0 con 0x7f83700ff850 2024-03-11T19:09:47.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.560+0000 7f836ffff640 1 --2- 172.21.15.29:0/2022853446 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f83600a1a30 0x7f83600a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:47.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.560+0000 7f836ffff640 1 --2- 172.21.15.29:0/2022853446 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f83600a1a30 0x7f83600a3ef0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f835c036040 tx=0x7f835c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:47.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.564+0000 7f836dffb640 1 -- 172.21.15.29:0/2022853446 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8364096470 con 0x7f83700ff850 2024-03-11T19:09:47.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:47 smithi170 ceph-mon[35690]: pgmap v350: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:47.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:47 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2546104454' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:09:47.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:47 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2674493619' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:47.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:47 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1401870686' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:09:47.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:47 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1259552430' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:09:47.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.785+0000 7f8376e64640 1 -- 172.21.15.29:0/2022853446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f837010bbb0 con 0x7f83700ff850 2024-03-11T19:09:47.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.785+0000 7f836dffb640 1 -- 172.21.15.29:0/2022853446 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f8364009280 con 0x7f83700ff850 2024-03-11T19:09:47.788 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:47.789 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:07:56.662890+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:09:47.789 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 26 2024-03-11T19:09:47.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.788+0000 7f8376e64640 1 -- 172.21.15.29:0/2022853446 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f83600a1a30 msgr2=0x7f83600a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:47.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.788+0000 7f8376e64640 1 --2- 172.21.15.29:0/2022853446 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f83600a1a30 0x7f83600a3ef0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f835c036040 tx=0x7f835c000bc0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.788+0000 7f8376e64640 1 -- 172.21.15.29:0/2022853446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83700ff850 msgr2=0x7f8370108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:47.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.788+0000 7f8376e64640 1 --2- 172.21.15.29:0/2022853446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83700ff850 0x7f8370108e80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f836400b450 tx=0x7f836400b910 comp rx=0 tx=0).stop 2024-03-11T19:09:47.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.788+0000 7f8376e64640 1 -- 172.21.15.29:0/2022853446 shutdown_connections 2024-03-11T19:09:47.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.788+0000 7f8376e64640 1 --2- 172.21.15.29:0/2022853446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8370101400 0x7f837019e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.788+0000 7f8376e64640 1 --2- 172.21.15.29:0/2022853446 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8370100a40 0x7f837019bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.788+0000 7f8376e64640 1 --2- 172.21.15.29:0/2022853446 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f83600a1a30 0x7f83600a3ef0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.788+0000 7f8376e64640 1 --2- 172.21.15.29:0/2022853446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83700ff850 0x7f8370108e80 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.788+0000 7f8376e64640 1 -- 172.21.15.29:0/2022853446 >> 172.21.15.29:0/2022853446 conn(0x7f83700fadc0 msgr2=0x7f8370103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:47.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.788+0000 7f8376e64640 1 -- 172.21.15.29:0/2022853446 shutdown_connections 2024-03-11T19:09:47.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.788+0000 7f8376e64640 1 -- 172.21.15.29:0/2022853446 wait complete. 2024-03-11T19:09:47.802 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:09:47.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.902+0000 7fb5f4d77640 1 -- 172.21.15.29:0/3576664980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f0109050 msgr2=0x7fb5f010b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:47.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.902+0000 7fb5f4d77640 1 --2- 172.21.15.29:0/3576664980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f0109050 0x7fb5f010b4e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8009ca0 tx=0x7fb5e802c180 comp rx=0 tx=0).stop 2024-03-11T19:09:47.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.902+0000 7fb5f4d77640 1 -- 172.21.15.29:0/3576664980 shutdown_connections 2024-03-11T19:09:47.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.902+0000 7fb5f4d77640 1 --2- 172.21.15.29:0/3576664980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f0109050 0x7fb5f010b4e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.902+0000 7fb5f4d77640 1 --2- 172.21.15.29:0/3576664980 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5f0073b60 0x7fb5f0073fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.902+0000 7fb5f4d77640 1 --2- 172.21.15.29:0/3576664980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5f0075140 0x7fb5f0073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.902+0000 7fb5f4d77640 1 -- 172.21.15.29:0/3576664980 >> 172.21.15.29:0/3576664980 conn(0x7fb5f00fc730 msgr2=0x7fb5f00feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:47.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.902+0000 7fb5f4d77640 1 -- 172.21.15.29:0/3576664980 shutdown_connections 2024-03-11T19:09:47.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.902+0000 7fb5f4d77640 1 -- 172.21.15.29:0/3576664980 wait complete. 2024-03-11T19:09:47.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.903+0000 7fb5f4d77640 1 Processor -- start 2024-03-11T19:09:47.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.903+0000 7fb5f4d77640 1 -- start start 2024-03-11T19:09:47.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.903+0000 7fb5f4d77640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5f0073b60 0x7fb5f0199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:47.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.903+0000 7fb5f4d77640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f0075140 0x7fb5f0199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:47.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.903+0000 7fb5f4d77640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5f0109050 0x7fb5f01a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:47.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.903+0000 7fb5f4d77640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f01a0d20 con 0x7fb5f0075140 2024-03-11T19:09:47.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.903+0000 7fb5f4d77640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f01a0e90 con 0x7fb5f0073b60 2024-03-11T19:09:47.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.903+0000 7fb5f4d77640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f01a1050 con 0x7fb5f0109050 2024-03-11T19:09:47.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.903+0000 7fb5e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f0075140 0x7fb5f0199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:47.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.903+0000 7fb5e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f0075140 0x7fb5f0199770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49420/0 (socket says 172.21.15.29:49420) 2024-03-11T19:09:47.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.903+0000 7fb5e7fff640 1 -- 172.21.15.29:0/611890101 learned_addr learned my addr 172.21.15.29:0/611890101 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:47.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.903+0000 7fb5eed9d640 1 --2- 172.21.15.29:0/611890101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5f0109050 0x7fb5f01a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:47.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.903+0000 7fb5ee59c640 1 --2- 172.21.15.29:0/611890101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5f0073b60 0x7fb5f0199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:47.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.903+0000 7fb5e7fff640 1 -- 172.21.15.29:0/611890101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5f0109050 msgr2=0x7fb5f01a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:47.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.903+0000 7fb5e7fff640 1 --2- 172.21.15.29:0/611890101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5f0109050 0x7fb5f01a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.903+0000 7fb5e7fff640 1 -- 172.21.15.29:0/611890101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5f0073b60 msgr2=0x7fb5f0199230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:47.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.903+0000 7fb5e7fff640 1 --2- 172.21.15.29:0/611890101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5f0073b60 0x7fb5f0199230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:47.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.903+0000 7fb5e7fff640 1 -- 172.21.15.29:0/611890101 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e8009900 con 0x7fb5f0075140 2024-03-11T19:09:47.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.903+0000 7fb5ee59c640 1 --2- 172.21.15.29:0/611890101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5f0073b60 0x7fb5f0199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:47.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.904+0000 7fb5e7fff640 1 --2- 172.21.15.29:0/611890101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f0075140 0x7fb5f0199770 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb5d800c920 tx=0x7fb5d800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:47.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.904+0000 7fb5e77fe640 1 -- 172.21.15.29:0/611890101 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8007c70 con 0x7fb5f0075140 2024-03-11T19:09:47.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.904+0000 7fb5e77fe640 1 -- 172.21.15.29:0/611890101 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb5d8007dd0 con 0x7fb5f0075140 2024-03-11T19:09:47.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.904+0000 7fb5f4d77640 1 -- 172.21.15.29:0/611890101 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f01a1330 con 0x7fb5f0075140 2024-03-11T19:09:47.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.904+0000 7fb5e77fe640 1 -- 172.21.15.29:0/611890101 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d80187c0 con 0x7fb5f0075140 2024-03-11T19:09:47.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.904+0000 7fb5f4d77640 1 -- 172.21.15.29:0/611890101 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f01a1890 con 0x7fb5f0075140 2024-03-11T19:09:47.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.905+0000 7fb5f4d77640 1 -- 172.21.15.29:0/611890101 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f0107e10 con 0x7fb5f0075140 2024-03-11T19:09:47.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.906+0000 7fb5e77fe640 1 -- 172.21.15.29:0/611890101 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb5d800f450 con 0x7fb5f0075140 2024-03-11T19:09:47.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.906+0000 7fb5e77fe640 1 --2- 172.21.15.29:0/611890101 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb5dc0a1ab0 0x7fb5dc0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:47.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.906+0000 7fb5ee59c640 1 --2- 172.21.15.29:0/611890101 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb5dc0a1ab0 0x7fb5dc0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:47.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.906+0000 7fb5e77fe640 1 -- 172.21.15.29:0/611890101 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb5d80cd440 con 0x7fb5f0075140 2024-03-11T19:09:47.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.907+0000 7fb5ee59c640 1 --2- 172.21.15.29:0/611890101 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb5dc0a1ab0 0x7fb5dc0a3f70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0005fd0 tx=0x7fb5e000b5f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:47.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:47.911+0000 7fb5e77fe640 1 -- 172.21.15.29:0/611890101 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb5d80966f0 con 0x7fb5f0075140 2024-03-11T19:09:48.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.092+0000 7fb5f4d77640 1 -- 172.21.15.29:0/611890101 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5f010ee40 con 0x7fb5f0075140 2024-03-11T19:09:48.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.093+0000 7fb5e77fe640 1 -- 172.21.15.29:0/611890101 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22559 (secure 0 0 0) 0x7fb5d8099db0 con 0x7fb5f0075140 2024-03-11T19:09:48.097 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:48.097 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":89,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:08:44.981806+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:07:47.563184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:07:48.282461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:09:48.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.095+0000 7fb5f4d77640 1 -- 172.21.15.29:0/611890101 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb5dc0a1ab0 msgr2=0x7fb5dc0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.095+0000 7fb5f4d77640 1 --2- 172.21.15.29:0/611890101 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb5dc0a1ab0 0x7fb5dc0a3f70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0005fd0 tx=0x7fb5e000b5f0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.095+0000 7fb5f4d77640 1 -- 172.21.15.29:0/611890101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f0075140 msgr2=0x7fb5f0199770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.095+0000 7fb5f4d77640 1 --2- 172.21.15.29:0/611890101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f0075140 0x7fb5f0199770 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb5d800c920 tx=0x7fb5d800cde0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.096+0000 7fb5f4d77640 1 -- 172.21.15.29:0/611890101 shutdown_connections 2024-03-11T19:09:48.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.096+0000 7fb5f4d77640 1 --2- 172.21.15.29:0/611890101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5f0109050 0x7fb5f01a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.096+0000 7fb5f4d77640 1 --2- 172.21.15.29:0/611890101 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb5dc0a1ab0 0x7fb5dc0a3f70 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.096+0000 7fb5f4d77640 1 --2- 172.21.15.29:0/611890101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5f0075140 0x7fb5f0199770 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.096+0000 7fb5f4d77640 1 --2- 172.21.15.29:0/611890101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5f0073b60 0x7fb5f0199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.096+0000 7fb5f4d77640 1 -- 172.21.15.29:0/611890101 >> 172.21.15.29:0/611890101 conn(0x7fb5f00fc730 msgr2=0x7fb5f0104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.096+0000 7fb5f4d77640 1 -- 172.21.15.29:0/611890101 shutdown_connections 2024-03-11T19:09:48.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.096+0000 7fb5f4d77640 1 -- 172.21.15.29:0/611890101 wait complete. 2024-03-11T19:09:48.110 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-11T19:07:49.123624+0000', 'modified': '2024-03-11T19:07:56.662890+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24506, 'mds_1': 24496, 'mds_2': 24475, 'mds_3': 24473, 'mds_4': 24463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14616': {'gid': 14616, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.29:6839/72765211', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6838', 'nonce': 72765211}, {'type': 'v1', 'addr': '172.21.15.29:6839', 'nonce': 72765211}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_14628': {'gid': 14628, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.29:6841/3154444865', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6840', 'nonce': 3154444865}, {'type': 'v1', 'addr': '172.21.15.29:6841', 'nonce': 3154444865}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24407': {'gid': 24407, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.170:6833/1035391240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 1035391240}, {'type': 'v1', 'addr': '172.21.15.170:6833', 'nonce': 1035391240}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24418': {'gid': 24418, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.190:6833/493725011', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6832', 'nonce': 493725011}, {'type': 'v1', 'addr': '172.21.15.190:6833', 'nonce': 493725011}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24449': {'gid': 24449, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.170:6835/2555518515', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6834', 'nonce': 2555518515}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 2555518515}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24463': {'gid': 24463, 'name': 'f', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.190:6835/1048234253', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6834', 'nonce': 1048234253}, {'type': 'v1', 'addr': '172.21.15.190:6835', 'nonce': 1048234253}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24473': {'gid': 24473, 'name': 'h', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.170:6837/585739236', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6836', 'nonce': 585739236}, {'type': 'v1', 'addr': '172.21.15.170:6837', 'nonce': 585739236}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24475': {'gid': 24475, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.190:6837/3595456607', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6836', 'nonce': 3595456607}, {'type': 'v1', 'addr': '172.21.15.190:6837', 'nonce': 3595456607}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24496': {'gid': 24496, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.190:6839/2031437522', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6838', 'nonce': 2031437522}, {'type': 'v1', 'addr': '172.21.15.190:6839', 'nonce': 2031437522}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24506': {'gid': 24506, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.170:6839/4145997740', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6838', 'nonce': 4145997740}, {'type': 'v1', 'addr': '172.21.15.170:6839', 'nonce': 4145997740}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-11T19:09:48.111 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-11T19:09:48.111 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-11T19:09:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.212+0000 7f3371925640 1 -- 172.21.15.29:0/1906117961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f336c101c30 msgr2=0x7f336c1084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.212+0000 7f3371925640 1 --2- 172.21.15.29:0/1906117961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f336c101c30 0x7f336c1084c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3364009ca0 tx=0x7f336402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.212+0000 7f3371925640 1 -- 172.21.15.29:0/1906117961 shutdown_connections 2024-03-11T19:09:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.212+0000 7f3371925640 1 --2- 172.21.15.29:0/1906117961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f336c101c30 0x7f336c1084c0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.212+0000 7f3371925640 1 --2- 172.21.15.29:0/1906117961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f336c101270 0x7f336c1016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.212+0000 7f3371925640 1 --2- 172.21.15.29:0/1906117961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f336c100080 0x7f336c1004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.212+0000 7f3371925640 1 -- 172.21.15.29:0/1906117961 >> 172.21.15.29:0/1906117961 conn(0x7f336c0fb610 msgr2=0x7f336c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.212+0000 7f3371925640 1 -- 172.21.15.29:0/1906117961 shutdown_connections 2024-03-11T19:09:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.212+0000 7f3371925640 1 -- 172.21.15.29:0/1906117961 wait complete. 2024-03-11T19:09:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.212+0000 7f3371925640 1 Processor -- start 2024-03-11T19:09:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.212+0000 7f3371925640 1 -- start start 2024-03-11T19:09:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f3371925640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f336c100080 0x7f336c195f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f3371925640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f336c101270 0x7f336c1964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f3371925640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f336c101c30 0x7f336c19d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f3371925640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c19da70 con 0x7f336c101270 2024-03-11T19:09:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f3371925640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c19dbe0 con 0x7f336c101c30 2024-03-11T19:09:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f3371925640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c19dda0 con 0x7f336c100080 2024-03-11T19:09:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f336a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f336c101270 0x7f336c1964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f336a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f336c101270 0x7f336c1964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49430/0 (socket says 172.21.15.29:49430) 2024-03-11T19:09:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f336a7fc640 1 -- 172.21.15.29:0/1185269543 learned_addr learned my addr 172.21.15.29:0/1185269543 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f336affd640 1 --2- 172.21.15.29:0/1185269543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f336c100080 0x7f336c195f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f336b7fe640 1 --2- 172.21.15.29:0/1185269543 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f336c101c30 0x7f336c19d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f336a7fc640 1 -- 172.21.15.29:0/1185269543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f336c100080 msgr2=0x7f336c195f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f336a7fc640 1 --2- 172.21.15.29:0/1185269543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f336c100080 0x7f336c195f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f336a7fc640 1 -- 172.21.15.29:0/1185269543 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f336c101c30 msgr2=0x7f336c19d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f336a7fc640 1 --2- 172.21.15.29:0/1185269543 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f336c101c30 0x7f336c19d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f336a7fc640 1 -- 172.21.15.29:0/1185269543 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3364009900 con 0x7f336c101270 2024-03-11T19:09:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f336affd640 1 --2- 172.21.15.29:0/1185269543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f336c100080 0x7f336c195f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f336a7fc640 1 --2- 172.21.15.29:0/1185269543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f336c101270 0x7f336c1964c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f335c00c920 tx=0x7f335c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f3370923640 1 -- 172.21.15.29:0/1185269543 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335c007c70 con 0x7f336c101270 2024-03-11T19:09:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f3370923640 1 -- 172.21.15.29:0/1185269543 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f335c007dd0 con 0x7f336c101270 2024-03-11T19:09:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f3371925640 1 -- 172.21.15.29:0/1185269543 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336c19e080 con 0x7f336c101270 2024-03-11T19:09:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f3370923640 1 -- 172.21.15.29:0/1185269543 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335c0187c0 con 0x7f336c101270 2024-03-11T19:09:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.213+0000 7f3371925640 1 -- 172.21.15.29:0/1185269543 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336c19e590 con 0x7f336c101270 2024-03-11T19:09:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.216+0000 7f3370923640 1 -- 172.21.15.29:0/1185269543 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f335c00f450 con 0x7f336c101270 2024-03-11T19:09:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.216+0000 7f33745d1740 1 Processor -- start 2024-03-11T19:09:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f33745d1740 1 -- start start 2024-03-11T19:09:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f33745d1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de98f600 0x5560de98dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f33745d1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de98e290 0x5560de98e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f33745d1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560de9982a0 0x5560de99a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f33745d1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560de949cd0 con 0x5560de98f600 2024-03-11T19:09:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f33745d1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560de8fcf20 con 0x5560de98e290 2024-03-11T19:09:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f33745d1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560de8fa450 con 0x5560de9982a0 2024-03-11T19:09:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f336b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de98f600 0x5560de98dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f336b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de98f600 0x5560de98dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49432/0 (socket says 172.21.15.29:49432) 2024-03-11T19:09:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f336b7fe640 1 -- 172.21.15.29:0/2783579418 learned_addr learned my addr 172.21.15.29:0/2783579418 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f3370923640 1 --2- 172.21.15.29:0/1185269543 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f33540a1a30 0x7f33540a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f336a7fc640 1 --2- 172.21.15.29:0/2783579418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560de9982a0 0x5560de99a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f336affd640 1 --2- 172.21.15.29:0/2783579418 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de98e290 0x5560de98e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f336b7fe640 1 --2- 172.21.15.29:0/1185269543 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f33540a1a30 0x7f33540a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f3370923640 1 -- 172.21.15.29:0/1185269543 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f335c0cd520 con 0x7f336c101270 2024-03-11T19:09:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f336b7fe640 1 -- 172.21.15.29:0/2783579418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560de9982a0 msgr2=0x5560de99a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f336b7fe640 1 --2- 172.21.15.29:0/2783579418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560de9982a0 0x5560de99a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f336b7fe640 1 -- 172.21.15.29:0/2783579418 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de98e290 msgr2=0x5560de98e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f336b7fe640 1 --2- 172.21.15.29:0/2783579418 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de98e290 0x5560de98e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.217+0000 7f336b7fe640 1 -- 172.21.15.29:0/2783579418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560de8aaf20 con 0x5560de98f600 2024-03-11T19:09:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.218+0000 7f336b7fe640 1 --2- 172.21.15.29:0/2783579418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de98f600 0x5560de98dd50 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f33640098d0 tx=0x7f3364032940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d340fac1d11d2b43 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.218+0000 7f336b7fe640 1 --2- 172.21.15.29:0/1185269543 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f33540a1a30 0x7f33540a3ef0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3364032630 tx=0x7f336402e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.218+0000 7f3351ffb640 1 -- 172.21.15.29:0/2783579418 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3364004d40 con 0x5560de98f600 2024-03-11T19:09:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.218+0000 7f3351ffb640 1 -- 172.21.15.29:0/2783579418 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3364034440 con 0x5560de98f600 2024-03-11T19:09:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.218+0000 7f3351ffb640 1 -- 172.21.15.29:0/2783579418 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3364004530 con 0x5560de98f600 2024-03-11T19:09:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.218+0000 7f33745d1740 1 -- 172.21.15.29:0/2783579418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de98f600 msgr2=0x5560de98dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.218+0000 7f33745d1740 1 --2- 172.21.15.29:0/2783579418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de98f600 0x5560de98dd50 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f33640098d0 tx=0x7f3364032940 comp rx=0 tx=0).stop 2024-03-11T19:09:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.218+0000 7f33745d1740 1 -- 172.21.15.29:0/2783579418 shutdown_connections 2024-03-11T19:09:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.218+0000 7f33745d1740 1 --2- 172.21.15.29:0/2783579418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560de9982a0 0x5560de99a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.218+0000 7f33745d1740 1 --2- 172.21.15.29:0/2783579418 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de98e290 0x5560de98e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.218+0000 7f33745d1740 1 --2- 172.21.15.29:0/2783579418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de98f600 0x5560de98dd50 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.218+0000 7f33745d1740 1 -- 172.21.15.29:0/2783579418 >> 172.21.15.29:0/2783579418 conn(0x5560de9894f0 msgr2=0x5560de9898f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.219+0000 7f33745d1740 1 -- 172.21.15.29:0/2783579418 shutdown_connections 2024-03-11T19:09:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.219+0000 7f33745d1740 1 -- 172.21.15.29:0/2783579418 wait complete. 2024-03-11T19:09:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.219+0000 7f33745d1740 1 -- start start 2024-03-11T19:09:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.219+0000 7f33745d1740 1 Processor -- start 2024-03-11T19:09:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.219+0000 7f33745d1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de98e290 0x5560de99c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.219+0000 7f33745d1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560de98f600 0x5560de99ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.219+0000 7f33745d1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de9982a0 0x5560de9a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.219+0000 7f33745d1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560de8f5c30 con 0x5560de9982a0 2024-03-11T19:09:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.219+0000 7f33745d1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560de89ad30 con 0x5560de98e290 2024-03-11T19:09:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.219+0000 7f33745d1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560de5c9310 con 0x5560de98f600 2024-03-11T19:09:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.219+0000 7f33745d1740 10 client.0 _ll_register_callbacks cb 0x5560de8a2d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.219+0000 7f336b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de9982a0 0x5560de9a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.219+0000 7f336b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de9982a0 0x5560de9a3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49434/0 (socket says 172.21.15.29:49434) 2024-03-11T19:09:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.219+0000 7f336b7fe640 1 -- 172.21.15.29:0/3182876644 learned_addr learned my addr 172.21.15.29:0/3182876644 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.220+0000 7f336affd640 1 --2- 172.21.15.29:0/3182876644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de98e290 0x5560de99c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.220+0000 7f336a7fc640 1 --2- 172.21.15.29:0/3182876644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560de98f600 0x5560de99ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.220+0000 7f336b7fe640 1 -- 172.21.15.29:0/3182876644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560de98f600 msgr2=0x5560de99ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.220+0000 7f336b7fe640 1 --2- 172.21.15.29:0/3182876644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560de98f600 0x5560de99ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.220+0000 7f336b7fe640 1 -- 172.21.15.29:0/3182876644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de98e290 msgr2=0x5560de99c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.220+0000 7f336b7fe640 1 --2- 172.21.15.29:0/3182876644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de98e290 0x5560de99c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.220+0000 7f336b7fe640 1 -- 172.21.15.29:0/3182876644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336c108dc0 con 0x5560de9982a0 2024-03-11T19:09:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.220+0000 7f336b7fe640 1 --2- 172.21.15.29:0/3182876644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de9982a0 0x5560de9a3ab0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3364009880 tx=0x7f3364003f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=333ce872fa8ec219 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.220+0000 7f334affd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.220+0000 7f334affd640 1 -- 172.21.15.29:0/3182876644 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33640042d0 con 0x5560de9982a0 2024-03-11T19:09:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.220+0000 7f334affd640 1 -- 172.21.15.29:0/3182876644 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f3364004430 con 0x5560de9982a0 2024-03-11T19:09:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.220+0000 7f33745d1740 1 -- 172.21.15.29:0/3182876644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5560de5f56c0 con 0x5560de9982a0 2024-03-11T19:09:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.220+0000 7f334affd640 1 -- 172.21.15.29:0/3182876644 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33640045f0 con 0x5560de9982a0 2024-03-11T19:09:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.220+0000 7f33745d1740 10 client.15240 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.220+0000 7f33745d1740 1 -- 172.21.15.29:0/3182876644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5560de8f42b0 con 0x5560de9982a0 2024-03-11T19:09:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.221+0000 7f334affd640 1 -- 172.21.15.29:0/3182876644 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5560de5f56c0 con 0x5560de9982a0 2024-03-11T19:09:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.221+0000 7f334affd640 1 -- 172.21.15.29:0/3182876644 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f336403e040 con 0x5560de9982a0 2024-03-11T19:09:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.221+0000 7f33745d1740 10 client.15240 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.221+0000 7f33745d1740 10 client.15240 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-11T19:09:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.221+0000 7f33745d1740 20 client.15240 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.221+0000 7f33745d1740 1 --2- 172.21.15.29:0/3182876644 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5560de9a7fa0 0x5560de9ad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.221+0000 7f33745d1740 4 client.15240 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:09:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.221+0000 7f33745d1740 1 -- 172.21.15.29:0/3182876644 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5560de646270 con 0x5560de9a7fa0 2024-03-11T19:09:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.221+0000 7f336affd640 1 --2- 172.21.15.29:0/3182876644 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5560de9a7fa0 0x5560de9ad0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.221+0000 7f334affd640 10 client.15240 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:09:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.221+0000 7f336affd640 1 --2- 172.21.15.29:0/3182876644 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5560de9a7fa0 0x5560de9ad0a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2a6531d609c198a5 server_cookie=3951dc6cf5763c9a in_seq=0 out_seq=0 2024-03-11T19:09:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.222+0000 7f334affd640 1 -- 172.21.15.29:0/3182876644 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5560de646270 con 0x5560de9a7fa0 2024-03-11T19:09:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.222+0000 7f334affd640 10 client.15240 handle_command_reply: tid=0 2024-03-11T19:09:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.222+0000 7f33745d1740 1 client.15240 shutdown 2024-03-11T19:09:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.222+0000 7f33745d1740 1 -- 172.21.15.29:0/3182876644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de9982a0 msgr2=0x5560de9a3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.222+0000 7f33745d1740 1 --2- 172.21.15.29:0/3182876644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de9982a0 0x5560de9a3ab0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3364009880 tx=0x7f3364003f30 comp rx=0 tx=0).stop 2024-03-11T19:09:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.222+0000 7f33745d1740 1 -- 172.21.15.29:0/3182876644 shutdown_connections 2024-03-11T19:09:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.222+0000 7f33745d1740 1 --2- 172.21.15.29:0/3182876644 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5560de9a7fa0 0x5560de9ad0a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.222+0000 7f33745d1740 1 --2- 172.21.15.29:0/3182876644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de9982a0 0x5560de9a3ab0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.222+0000 7f33745d1740 1 --2- 172.21.15.29:0/3182876644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560de98f600 0x5560de99ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.222+0000 7f33745d1740 1 --2- 172.21.15.29:0/3182876644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de98e290 0x5560de99c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.222+0000 7f334affd640 0 client.15240 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:09:48.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.222+0000 7f33745d1740 1 -- 172.21.15.29:0/3182876644 >> 172.21.15.29:0/3182876644 conn(0x5560de9894f0 msgr2=0x5560de9887d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.223+0000 7f33745d1740 1 -- 172.21.15.29:0/3182876644 shutdown_connections 2024-03-11T19:09:48.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.223+0000 7f33745d1740 1 -- 172.21.15.29:0/3182876644 wait complete. 2024-03-11T19:09:48.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.223+0000 7f33745d1740 20 client.15240 trim_cache size 0 max 16384 2024-03-11T19:09:48.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.239+0000 7f33745d1740 1 Processor -- start 2024-03-11T19:09:48.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.239+0000 7f33745d1740 1 -- start start 2024-03-11T19:09:48.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.239+0000 7f33745d1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de98e290 0x5560de997d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.239+0000 7f33745d1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560de98f600 0x5560de9a39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.239+0000 7f33745d1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de9982a0 0x5560de9b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.239+0000 7f33745d1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560de87b490 con 0x5560de98e290 2024-03-11T19:09:48.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.239+0000 7f33745d1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560de99d5c0 con 0x5560de9982a0 2024-03-11T19:09:48.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.239+0000 7f33745d1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560de9a87f0 con 0x5560de98f600 2024-03-11T19:09:48.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.239+0000 7f336affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de98e290 0x5560de997d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.239+0000 7f336affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de98e290 0x5560de997d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49440/0 (socket says 172.21.15.29:49440) 2024-03-11T19:09:48.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.239+0000 7f336affd640 1 -- 172.21.15.29:0/2162475054 learned_addr learned my addr 172.21.15.29:0/2162475054 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.239+0000 7f336affd640 1 -- 172.21.15.29:0/2162475054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560de98f600 msgr2=0x5560de9a39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.239+0000 7f336affd640 1 --2- 172.21.15.29:0/2162475054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560de98f600 0x5560de9a39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.239+0000 7f336affd640 1 -- 172.21.15.29:0/2162475054 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de9982a0 msgr2=0x5560de9b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.239+0000 7f336affd640 1 --2- 172.21.15.29:0/2162475054 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de9982a0 0x5560de9b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.239+0000 7f336affd640 1 -- 172.21.15.29:0/2162475054 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560de890a20 con 0x5560de98e290 2024-03-11T19:09:48.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.239+0000 7f336affd640 1 --2- 172.21.15.29:0/2162475054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de98e290 0x5560de997d60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3360005fd0 tx=0x7f3360011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b93c1129483ed07 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.240+0000 7f33517fa640 1 -- 172.21.15.29:0/2162475054 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360017040 con 0x5560de98e290 2024-03-11T19:09:48.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.240+0000 7f33517fa640 1 -- 172.21.15.29:0/2162475054 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3360020070 con 0x5560de98e290 2024-03-11T19:09:48.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.240+0000 7f33517fa640 1 -- 172.21.15.29:0/2162475054 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360010e90 con 0x5560de98e290 2024-03-11T19:09:48.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.240+0000 7f33745d1740 1 -- 172.21.15.29:0/2162475054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de98e290 msgr2=0x5560de997d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.240+0000 7f33745d1740 1 --2- 172.21.15.29:0/2162475054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de98e290 0x5560de997d60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3360005fd0 tx=0x7f3360011810 comp rx=0 tx=0).stop 2024-03-11T19:09:48.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.240+0000 7f33745d1740 1 -- 172.21.15.29:0/2162475054 shutdown_connections 2024-03-11T19:09:48.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.240+0000 7f33745d1740 1 --2- 172.21.15.29:0/2162475054 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de9982a0 0x5560de9b10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.240+0000 7f33745d1740 1 --2- 172.21.15.29:0/2162475054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560de98f600 0x5560de9a39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.240+0000 7f33745d1740 1 --2- 172.21.15.29:0/2162475054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de98e290 0x5560de997d60 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.240+0000 7f33745d1740 1 -- 172.21.15.29:0/2162475054 >> 172.21.15.29:0/2162475054 conn(0x5560de9894f0 msgr2=0x5560de9898d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.240+0000 7f33745d1740 1 -- 172.21.15.29:0/2162475054 shutdown_connections 2024-03-11T19:09:48.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.240+0000 7f33745d1740 1 -- 172.21.15.29:0/2162475054 wait complete. 2024-03-11T19:09:48.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.240+0000 7f33745d1740 1 -- start start 2024-03-11T19:09:48.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f33745d1740 1 Processor -- start 2024-03-11T19:09:48.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f33745d1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560de98e290 0x5560de9a3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f33745d1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de98f600 0x5560de9a3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f33745d1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de9982a0 0x5560de9a0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f33745d1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560de9aa170 con 0x5560de9982a0 2024-03-11T19:09:48.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f33745d1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560de9a0930 con 0x5560de98f600 2024-03-11T19:09:48.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f33745d1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560de9a0af0 con 0x5560de98e290 2024-03-11T19:09:48.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f33745d1740 10 client.0 _ll_register_callbacks cb 0x7f3364037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:48.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f336b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de9982a0 0x5560de9a0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f336b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de9982a0 0x5560de9a0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49446/0 (socket says 172.21.15.29:49446) 2024-03-11T19:09:48.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f336b7fe640 1 -- 172.21.15.29:0/1394953646 learned_addr learned my addr 172.21.15.29:0/1394953646 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f336affd640 1 --2- 172.21.15.29:0/1394953646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560de98e290 0x5560de9a3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f336a7fc640 1 --2- 172.21.15.29:0/1394953646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de98f600 0x5560de9a3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f336b7fe640 1 -- 172.21.15.29:0/1394953646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560de98e290 msgr2=0x5560de9a3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f336b7fe640 1 --2- 172.21.15.29:0/1394953646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560de98e290 0x5560de9a3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f336b7fe640 1 -- 172.21.15.29:0/1394953646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de98f600 msgr2=0x5560de9a3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f336b7fe640 1 --2- 172.21.15.29:0/1394953646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de98f600 0x5560de9a3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f336b7fe640 1 -- 172.21.15.29:0/1394953646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336000bb10 con 0x5560de9982a0 2024-03-11T19:09:48.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f336a7fc640 1 --2- 172.21.15.29:0/1394953646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de98f600 0x5560de9a3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:48.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f336b7fe640 1 --2- 172.21.15.29:0/1394953646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de9982a0 0x5560de9a0200 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3364009ca0 tx=0x7f3364009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=527523d9c738537 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f334affd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:48.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.241+0000 7f334affd640 1 -- 172.21.15.29:0/1394953646 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3364005bf0 con 0x5560de9982a0 2024-03-11T19:09:48.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.242+0000 7f334affd640 1 -- 172.21.15.29:0/1394953646 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f3364003b80 con 0x5560de9982a0 2024-03-11T19:09:48.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.242+0000 7f33745d1740 1 -- 172.21.15.29:0/1394953646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5560de9a8930 con 0x5560de9982a0 2024-03-11T19:09:48.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.242+0000 7f334affd640 1 -- 172.21.15.29:0/1394953646 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3364058dc0 con 0x5560de9982a0 2024-03-11T19:09:48.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.242+0000 7f33745d1740 10 client.15246 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:48.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.242+0000 7f33745d1740 1 -- 172.21.15.29:0/1394953646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5560de8b6780 con 0x5560de9982a0 2024-03-11T19:09:48.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.242+0000 7f334affd640 1 -- 172.21.15.29:0/1394953646 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3364075020 con 0x5560de9982a0 2024-03-11T19:09:48.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.242+0000 7f334affd640 1 -- 172.21.15.29:0/1394953646 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3364071480 con 0x5560de9982a0 2024-03-11T19:09:48.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.242+0000 7f33745d1740 10 client.15246 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:48.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.242+0000 7f33745d1740 10 client.15246 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-11T19:09:48.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.242+0000 7f33745d1740 20 client.15246 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:48.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.242+0000 7f33745d1740 1 --2- 172.21.15.29:0/1394953646 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5560dea03880 0x5560de9ffd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.242+0000 7f33745d1740 4 client.15246 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:09:48.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.242+0000 7f33745d1740 1 -- 172.21.15.29:0/1394953646 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5560de9aa2b0 con 0x5560dea03880 2024-03-11T19:09:48.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.242+0000 7f336affd640 1 --2- 172.21.15.29:0/1394953646 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5560dea03880 0x5560de9ffd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.243+0000 7f334affd640 10 client.15246 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:09:48.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.243+0000 7f336affd640 1 --2- 172.21.15.29:0/1394953646 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5560dea03880 0x5560de9ffd40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1257730d8d277ede server_cookie=d46f66fa69e95e4e in_seq=0 out_seq=0 2024-03-11T19:09:48.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.243+0000 7f334affd640 1 -- 172.21.15.29:0/1394953646 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5560de9aa2b0 con 0x5560dea03880 2024-03-11T19:09:48.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.243+0000 7f334affd640 10 client.15246 handle_command_reply: tid=0 2024-03-11T19:09:48.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.243+0000 7f33745d1740 1 client.15246 shutdown 2024-03-11T19:09:48.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.243+0000 7f33745d1740 1 -- 172.21.15.29:0/1394953646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de9982a0 msgr2=0x5560de9a0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.243+0000 7f33745d1740 1 --2- 172.21.15.29:0/1394953646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de9982a0 0x5560de9a0200 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3364009ca0 tx=0x7f3364009360 comp rx=0 tx=0).stop 2024-03-11T19:09:48.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.243+0000 7f33745d1740 1 -- 172.21.15.29:0/1394953646 shutdown_connections 2024-03-11T19:09:48.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.243+0000 7f33745d1740 1 --2- 172.21.15.29:0/1394953646 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5560dea03880 0x5560de9ffd40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.243+0000 7f33745d1740 1 --2- 172.21.15.29:0/1394953646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5560de9982a0 0x5560de9a0200 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.243+0000 7f33745d1740 1 --2- 172.21.15.29:0/1394953646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5560de98f600 0x5560de9a3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.243+0000 7f33745d1740 1 --2- 172.21.15.29:0/1394953646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560de98e290 0x5560de9a3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.243+0000 7f334affd640 0 client.15246 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:09:48.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.243+0000 7f33745d1740 1 -- 172.21.15.29:0/1394953646 >> 172.21.15.29:0/1394953646 conn(0x5560de9894f0 msgr2=0x5560de9b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.243+0000 7f33745d1740 1 -- 172.21.15.29:0/1394953646 shutdown_connections 2024-03-11T19:09:48.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.243+0000 7f33745d1740 1 -- 172.21.15.29:0/1394953646 wait complete. 2024-03-11T19:09:48.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.243+0000 7f33745d1740 20 client.15246 trim_cache size 0 max 16384 2024-03-11T19:09:48.253 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:09:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "638915c2-dfd9-11ee-95c8-87774f69a715", 2024-03-11T19:09:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 4, 2024-03-11T19:09:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24463, 2024-03-11T19:09:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-03-11T19:09:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-03-11T19:09:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-03-11T19:09:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 112.46378170600001, 2024-03-11T19:09:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 26, 2024-03-11T19:09:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 89, 2024-03-11T19:09:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 87, 2024-03-11T19:09:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 139.84796651799999 2024-03-11T19:09:48.254 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:09:48.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.244+0000 7f3371925640 1 -- 172.21.15.29:0/1185269543 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f33540a1a30 msgr2=0x7f33540a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.244+0000 7f3371925640 1 --2- 172.21.15.29:0/1185269543 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f33540a1a30 0x7f33540a3ef0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3364032630 tx=0x7f336402e000 comp rx=0 tx=0).stop 2024-03-11T19:09:48.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.244+0000 7f3371925640 1 -- 172.21.15.29:0/1185269543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f336c101270 msgr2=0x7f336c1964c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.244+0000 7f3371925640 1 --2- 172.21.15.29:0/1185269543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f336c101270 0x7f336c1964c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f335c00c920 tx=0x7f335c00cde0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.244+0000 7f3371925640 1 -- 172.21.15.29:0/1185269543 shutdown_connections 2024-03-11T19:09:48.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.244+0000 7f3371925640 1 --2- 172.21.15.29:0/1185269543 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f336c101c30 0x7f336c19d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.244+0000 7f3371925640 1 --2- 172.21.15.29:0/1185269543 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f33540a1a30 0x7f33540a3ef0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.244+0000 7f3371925640 1 --2- 172.21.15.29:0/1185269543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f336c101270 0x7f336c1964c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.244+0000 7f3371925640 1 --2- 172.21.15.29:0/1185269543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f336c100080 0x7f336c195f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.244+0000 7f3371925640 1 -- 172.21.15.29:0/1185269543 >> 172.21.15.29:0/1185269543 conn(0x7f336c0fb610 msgr2=0x7f336c104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.244+0000 7f3371925640 1 -- 172.21.15.29:0/1185269543 shutdown_connections 2024-03-11T19:09:48.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.244+0000 7f3371925640 1 -- 172.21.15.29:0/1185269543 wait complete. 2024-03-11T19:09:48.257 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-11T19:09:48.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.358+0000 7f162fc5b640 1 -- 172.21.15.29:0/3828358911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16281011b0 msgr2=0x7f16281015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.358+0000 7f162fc5b640 1 --2- 172.21.15.29:0/3828358911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16281011b0 0x7f16281015d0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1620009900 tx=0x7f162002bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:48.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.358+0000 7f162fc5b640 1 -- 172.21.15.29:0/3828358911 shutdown_connections 2024-03-11T19:09:48.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.358+0000 7f162fc5b640 1 --2- 172.21.15.29:0/3828358911 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1628102d60 0x7f16281095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.358+0000 7f162fc5b640 1 --2- 172.21.15.29:0/3828358911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16281023a0 0x7f1628102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.358+0000 7f162fc5b640 1 --2- 172.21.15.29:0/3828358911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16281011b0 0x7f16281015d0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.358+0000 7f162fc5b640 1 -- 172.21.15.29:0/3828358911 >> 172.21.15.29:0/3828358911 conn(0x7f16280fc720 msgr2=0x7f16280feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.358+0000 7f162fc5b640 1 -- 172.21.15.29:0/3828358911 shutdown_connections 2024-03-11T19:09:48.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.358+0000 7f162fc5b640 1 -- 172.21.15.29:0/3828358911 wait complete. 2024-03-11T19:09:48.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.358+0000 7f162fc5b640 1 Processor -- start 2024-03-11T19:09:48.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.358+0000 7f162fc5b640 1 -- start start 2024-03-11T19:09:48.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.358+0000 7f162fc5b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16281011b0 0x7f16281971f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.358+0000 7f162fc5b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16281023a0 0x7f1628197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.358+0000 7f162fc5b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1628102d60 0x7f162819e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.358+0000 7f162fc5b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162819ece0 con 0x7f16281023a0 2024-03-11T19:09:48.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.358+0000 7f162fc5b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162819ee50 con 0x7f16281011b0 2024-03-11T19:09:48.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.358+0000 7f162fc5b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162819f010 con 0x7f1628102d60 2024-03-11T19:09:48.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.359+0000 7f162d1f6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16281023a0 0x7f1628197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.359+0000 7f162d1f6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16281023a0 0x7f1628197730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49464/0 (socket says 172.21.15.29:49464) 2024-03-11T19:09:48.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.359+0000 7f162d1f6640 1 -- 172.21.15.29:0/242070790 learned_addr learned my addr 172.21.15.29:0/242070790 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.359+0000 7f162e1f8640 1 --2- 172.21.15.29:0/242070790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1628102d60 0x7f162819e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.359+0000 7f162d9f7640 1 --2- 172.21.15.29:0/242070790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16281011b0 0x7f16281971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.359+0000 7f162d1f6640 1 -- 172.21.15.29:0/242070790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1628102d60 msgr2=0x7f162819e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.359+0000 7f162d1f6640 1 --2- 172.21.15.29:0/242070790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1628102d60 0x7f162819e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.359+0000 7f162d1f6640 1 -- 172.21.15.29:0/242070790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16281011b0 msgr2=0x7f16281971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.359+0000 7f162d1f6640 1 --2- 172.21.15.29:0/242070790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16281011b0 0x7f16281971f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.359+0000 7f162d1f6640 1 -- 172.21.15.29:0/242070790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1620009560 con 0x7f16281023a0 2024-03-11T19:09:48.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.359+0000 7f162d1f6640 1 --2- 172.21.15.29:0/242070790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16281023a0 0x7f1628197730 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f161c00d770 tx=0x7f161c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.359+0000 7f1626ffd640 1 -- 172.21.15.29:0/242070790 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c009c30 con 0x7f16281023a0 2024-03-11T19:09:48.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.359+0000 7f1626ffd640 1 -- 172.21.15.29:0/242070790 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f161c00de80 con 0x7f16281023a0 2024-03-11T19:09:48.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.359+0000 7f162fc5b640 1 -- 172.21.15.29:0/242070790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f162819f2f0 con 0x7f16281023a0 2024-03-11T19:09:48.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.359+0000 7f1626ffd640 1 -- 172.21.15.29:0/242070790 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c004e00 con 0x7f16281023a0 2024-03-11T19:09:48.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.359+0000 7f162fc5b640 1 -- 172.21.15.29:0/242070790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f162819f800 con 0x7f16281023a0 2024-03-11T19:09:48.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.361+0000 7f1626ffd640 1 -- 172.21.15.29:0/242070790 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f161c004f60 con 0x7f16281023a0 2024-03-11T19:09:48.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.362+0000 7f163282a740 1 Processor -- start 2024-03-11T19:09:48.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.362+0000 7f163282a740 1 -- start start 2024-03-11T19:09:48.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.362+0000 7f163282a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6390 0x55f2e57d67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.362+0000 7f163282a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6d80 0x55f2e57db880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.362+0000 7f163282a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57de380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.362+0000 7f163282a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e578dcd0 con 0x55f2e57dbef0 2024-03-11T19:09:48.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.362+0000 7f163282a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e5740f20 con 0x55f2e57d6d80 2024-03-11T19:09:48.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.362+0000 7f163282a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e573e450 con 0x55f2e57d6390 2024-03-11T19:09:48.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.362+0000 7f1626ffd640 1 --2- 172.21.15.29:0/242070790 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f16100a1ab0 0x7f16100a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.362+0000 7f162d1f6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57de380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.362+0000 7f162d1f6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57de380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49466/0 (socket says 172.21.15.29:49466) 2024-03-11T19:09:48.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.362+0000 7f162d9f7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6d80 0x55f2e57db880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.362+0000 7f162d1f6640 1 -- 172.21.15.29:0/905567668 learned_addr learned my addr 172.21.15.29:0/905567668 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.362+0000 7f162e1f8640 1 --2- 172.21.15.29:0/242070790 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f16100a1ab0 0x7f16100a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.362+0000 7f162e1f8640 1 --2- 172.21.15.29:0/905567668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6390 0x55f2e57d67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.362+0000 7f1626ffd640 1 -- 172.21.15.29:0/242070790 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f161c0ccdf0 con 0x7f16281023a0 2024-03-11T19:09:48.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.363+0000 7f162d1f6640 1 -- 172.21.15.29:0/905567668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6390 msgr2=0x55f2e57d67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.363+0000 7f162d1f6640 1 --2- 172.21.15.29:0/905567668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6390 0x55f2e57d67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.363+0000 7f162d1f6640 1 -- 172.21.15.29:0/905567668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6d80 msgr2=0x55f2e57db880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.363+0000 7f162d1f6640 1 --2- 172.21.15.29:0/905567668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6d80 0x55f2e57db880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.363+0000 7f162d1f6640 1 -- 172.21.15.29:0/905567668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2e56eef20 con 0x55f2e57dbef0 2024-03-11T19:09:48.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.363+0000 7f162e1f8640 1 --2- 172.21.15.29:0/905567668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6390 0x55f2e57d67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:48.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.363+0000 7f162d9f7640 1 --2- 172.21.15.29:0/905567668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6d80 0x55f2e57db880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:09:48.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.363+0000 7f162d1f6640 1 --2- 172.21.15.29:0/905567668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57de380 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f161c0057c0 tx=0x7f161c01e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=974eacf97f44135e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.363+0000 7f162e1f8640 1 --2- 172.21.15.29:0/242070790 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f16100a1ab0 0x7f16100a3f70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55f2e57d75e0 tx=0x7f1618009b60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.363+0000 7f160ffff640 1 -- 172.21.15.29:0/905567668 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c014070 con 0x55f2e57dbef0 2024-03-11T19:09:48.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.363+0000 7f160ffff640 1 -- 172.21.15.29:0/905567668 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f161c09deb0 con 0x55f2e57dbef0 2024-03-11T19:09:48.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.363+0000 7f160ffff640 1 -- 172.21.15.29:0/905567668 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c098180 con 0x55f2e57dbef0 2024-03-11T19:09:48.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.364+0000 7f163282a740 1 -- 172.21.15.29:0/905567668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 msgr2=0x55f2e57de380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.364+0000 7f163282a740 1 --2- 172.21.15.29:0/905567668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57de380 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f161c0057c0 tx=0x7f161c01e000 comp rx=0 tx=0).stop 2024-03-11T19:09:48.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.364+0000 7f163282a740 1 -- 172.21.15.29:0/905567668 shutdown_connections 2024-03-11T19:09:48.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.364+0000 7f163282a740 1 --2- 172.21.15.29:0/905567668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57de380 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.364+0000 7f163282a740 1 --2- 172.21.15.29:0/905567668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6d80 0x55f2e57db880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.364+0000 7f163282a740 1 --2- 172.21.15.29:0/905567668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6390 0x55f2e57d67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.364+0000 7f163282a740 1 -- 172.21.15.29:0/905567668 >> 172.21.15.29:0/905567668 conn(0x55f2e57cd4f0 msgr2=0x55f2e57cd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.364+0000 7f163282a740 1 -- 172.21.15.29:0/905567668 shutdown_connections 2024-03-11T19:09:48.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.364+0000 7f163282a740 1 -- 172.21.15.29:0/905567668 wait complete. 2024-03-11T19:09:48.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.364+0000 7f163282a740 1 -- start start 2024-03-11T19:09:48.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.364+0000 7f163282a740 1 Processor -- start 2024-03-11T19:09:48.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f163282a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6390 0x55f2e57e0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f163282a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6d80 0x55f2e57e0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f163282a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57e7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f163282a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e5739c30 con 0x55f2e57dbef0 2024-03-11T19:09:48.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f163282a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e56ded30 con 0x55f2e57d6390 2024-03-11T19:09:48.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f163282a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e540d310 con 0x55f2e57d6d80 2024-03-11T19:09:48.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f163282a740 10 client.0 _ll_register_callbacks cb 0x55f2e56e6d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:48.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f162e1f8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57e7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f162e1f8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57e7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49482/0 (socket says 172.21.15.29:49482) 2024-03-11T19:09:48.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f162e1f8640 1 -- 172.21.15.29:0/2094301859 learned_addr learned my addr 172.21.15.29:0/2094301859 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f162d1f6640 1 --2- 172.21.15.29:0/2094301859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6d80 0x55f2e57e0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f162d9f7640 1 --2- 172.21.15.29:0/2094301859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6390 0x55f2e57e0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f162e1f8640 1 -- 172.21.15.29:0/2094301859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6d80 msgr2=0x55f2e57e0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f162e1f8640 1 --2- 172.21.15.29:0/2094301859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6d80 0x55f2e57e0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f162e1f8640 1 -- 172.21.15.29:0/2094301859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6390 msgr2=0x55f2e57e0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f162e1f8640 1 --2- 172.21.15.29:0/2094301859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6390 0x55f2e57e0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f162e1f8640 1 -- 172.21.15.29:0/2094301859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f162819f800 con 0x55f2e57dbef0 2024-03-11T19:09:48.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f162e1f8640 1 --2- 172.21.15.29:0/2094301859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57e7ac0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1618010b30 tx=0x7f1618004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5350d96e74ca7a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f160d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:48.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f160d7fa640 1 -- 172.21.15.29:0/2094301859 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161801edd0 con 0x55f2e57dbef0 2024-03-11T19:09:48.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f163282a740 1 -- 172.21.15.29:0/2094301859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f2e54396c0 con 0x55f2e57dbef0 2024-03-11T19:09:48.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f160d7fa640 1 -- 172.21.15.29:0/2094301859 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f161801f3f0 con 0x55f2e57dbef0 2024-03-11T19:09:48.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f160d7fa640 1 -- 172.21.15.29:0/2094301859 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f161801f5b0 con 0x55f2e57dbef0 2024-03-11T19:09:48.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f163282a740 10 client.15258 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:48.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.365+0000 7f163282a740 1 -- 172.21.15.29:0/2094301859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55f2e57382b0 con 0x55f2e57dbef0 2024-03-11T19:09:48.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.366+0000 7f160d7fa640 1 -- 172.21.15.29:0/2094301859 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f2e54396c0 con 0x55f2e57dbef0 2024-03-11T19:09:48.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.366+0000 7f160d7fa640 1 -- 172.21.15.29:0/2094301859 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1618027030 con 0x55f2e57dbef0 2024-03-11T19:09:48.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.366+0000 7f163282a740 10 client.15258 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:48.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.366+0000 7f163282a740 10 client.15258 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-11T19:09:48.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.366+0000 7f163282a740 20 client.15258 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.366+0000 7f163282a740 1 --2- 172.21.15.29:0/2094301859 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55f2e57ecf10 0x55f2e57ec1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.366+0000 7f163282a740 4 client.15258 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:09:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.366+0000 7f163282a740 1 -- 172.21.15.29:0/2094301859 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f2e548a270 con 0x55f2e57ecf10 2024-03-11T19:09:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.366+0000 7f162d9f7640 1 --2- 172.21.15.29:0/2094301859 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55f2e57ecf10 0x55f2e57ec1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.367+0000 7f162d9f7640 1 --2- 172.21.15.29:0/2094301859 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55f2e57ecf10 0x55f2e57ec1a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dc2d3d44cb717748 server_cookie=4bb9c3d572ea307d in_seq=0 out_seq=0 2024-03-11T19:09:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.367+0000 7f160d7fa640 10 client.15258 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:09:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.367+0000 7f160d7fa640 1 -- 172.21.15.29:0/2094301859 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f2e548a270 con 0x55f2e57ecf10 2024-03-11T19:09:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.367+0000 7f160d7fa640 10 client.15258 handle_command_reply: tid=0 2024-03-11T19:09:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.367+0000 7f163282a740 1 client.15258 shutdown 2024-03-11T19:09:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.368+0000 7f163282a740 1 -- 172.21.15.29:0/2094301859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 msgr2=0x55f2e57e7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.368+0000 7f163282a740 1 --2- 172.21.15.29:0/2094301859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57e7ac0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1618010b30 tx=0x7f1618004a40 comp rx=0 tx=0).stop 2024-03-11T19:09:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.368+0000 7f163282a740 1 -- 172.21.15.29:0/2094301859 shutdown_connections 2024-03-11T19:09:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.368+0000 7f163282a740 1 --2- 172.21.15.29:0/2094301859 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55f2e57ecf10 0x55f2e57ec1a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.368+0000 7f163282a740 1 --2- 172.21.15.29:0/2094301859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57e7ac0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.368+0000 7f163282a740 1 --2- 172.21.15.29:0/2094301859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6d80 0x55f2e57e0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.368+0000 7f160d7fa640 0 client.15258 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:09:48.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.368+0000 7f163282a740 1 --2- 172.21.15.29:0/2094301859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6390 0x55f2e57e0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.368+0000 7f163282a740 1 -- 172.21.15.29:0/2094301859 >> 172.21.15.29:0/2094301859 conn(0x55f2e57cd4f0 msgr2=0x55f2e57cc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.368+0000 7f163282a740 1 -- 172.21.15.29:0/2094301859 shutdown_connections 2024-03-11T19:09:48.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.368+0000 7f163282a740 1 -- 172.21.15.29:0/2094301859 wait complete. 2024-03-11T19:09:48.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.368+0000 7f163282a740 20 client.15258 trim_cache size 0 max 16384 2024-03-11T19:09:48.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f163282a740 1 Processor -- start 2024-03-11T19:09:48.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f163282a740 1 -- start start 2024-03-11T19:09:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f163282a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6390 0x55f2e57f53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f163282a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6d80 0x55f2e57e79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f163282a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57d8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f163282a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e56bf490 con 0x55f2e57dbef0 2024-03-11T19:09:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f163282a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e57eeef0 con 0x55f2e57d6d80 2024-03-11T19:09:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f163282a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e57cdfe0 con 0x55f2e57d6390 2024-03-11T19:09:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f162e1f8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57d8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f162e1f8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57d8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49492/0 (socket says 172.21.15.29:49492) 2024-03-11T19:09:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f162e1f8640 1 -- 172.21.15.29:0/3075500256 learned_addr learned my addr 172.21.15.29:0/3075500256 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f162d1f6640 1 --2- 172.21.15.29:0/3075500256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6d80 0x55f2e57e79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f162d9f7640 1 --2- 172.21.15.29:0/3075500256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6390 0x55f2e57f53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f162e1f8640 1 -- 172.21.15.29:0/3075500256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6390 msgr2=0x55f2e57f53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f162e1f8640 1 --2- 172.21.15.29:0/3075500256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6390 0x55f2e57f53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f162e1f8640 1 -- 172.21.15.29:0/3075500256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6d80 msgr2=0x55f2e57e79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f162e1f8640 1 --2- 172.21.15.29:0/3075500256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6d80 0x55f2e57e79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f162e1f8640 1 -- 172.21.15.29:0/3075500256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2e56d4a20 con 0x55f2e57dbef0 2024-03-11T19:09:48.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f162d9f7640 1 --2- 172.21.15.29:0/3075500256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6390 0x55f2e57f53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:48.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f162e1f8640 1 --2- 172.21.15.29:0/3075500256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57d8320 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1618012530 tx=0x7f1618012560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad3a5888ee416b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f160f7fe640 1 -- 172.21.15.29:0/3075500256 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1618029c40 con 0x55f2e57dbef0 2024-03-11T19:09:48.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f160f7fe640 1 -- 172.21.15.29:0/3075500256 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1618023070 con 0x55f2e57dbef0 2024-03-11T19:09:48.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.384+0000 7f160f7fe640 1 -- 172.21.15.29:0/3075500256 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1618033b10 con 0x55f2e57dbef0 2024-03-11T19:09:48.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.385+0000 7f163282a740 1 -- 172.21.15.29:0/3075500256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 msgr2=0x55f2e57d8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.385+0000 7f163282a740 1 --2- 172.21.15.29:0/3075500256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57d8320 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1618012530 tx=0x7f1618012560 comp rx=0 tx=0).stop 2024-03-11T19:09:48.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.385+0000 7f163282a740 1 -- 172.21.15.29:0/3075500256 shutdown_connections 2024-03-11T19:09:48.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.385+0000 7f163282a740 1 --2- 172.21.15.29:0/3075500256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57d8320 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.385+0000 7f163282a740 1 --2- 172.21.15.29:0/3075500256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6d80 0x55f2e57e79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.385+0000 7f163282a740 1 --2- 172.21.15.29:0/3075500256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6390 0x55f2e57f53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.385+0000 7f163282a740 1 -- 172.21.15.29:0/3075500256 >> 172.21.15.29:0/3075500256 conn(0x55f2e57cd4f0 msgr2=0x55f2e57cd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.385+0000 7f163282a740 1 -- 172.21.15.29:0/3075500256 shutdown_connections 2024-03-11T19:09:48.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.385+0000 7f163282a740 1 -- 172.21.15.29:0/3075500256 wait complete. 2024-03-11T19:09:48.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.385+0000 7f163282a740 1 -- start start 2024-03-11T19:09:48.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.385+0000 7f163282a740 1 Processor -- start 2024-03-11T19:09:48.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f163282a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6390 0x55f2e57e7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f163282a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6d80 0x55f2e57e4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f163282a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57e4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f163282a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e57e52d0 con 0x55f2e57dbef0 2024-03-11T19:09:48.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f163282a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e5842b40 con 0x55f2e57d6d80 2024-03-11T19:09:48.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f163282a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e5842d00 con 0x55f2e57d6390 2024-03-11T19:09:48.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f163282a740 10 client.0 _ll_register_callbacks cb 0x7f161c0958e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:48.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f162e1f8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57e4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f162e1f8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57e4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49506/0 (socket says 172.21.15.29:49506) 2024-03-11T19:09:48.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f162e1f8640 1 -- 172.21.15.29:0/406088321 learned_addr learned my addr 172.21.15.29:0/406088321 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f162d9f7640 1 --2- 172.21.15.29:0/406088321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6390 0x55f2e57e7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f162d1f6640 1 --2- 172.21.15.29:0/406088321 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6d80 0x55f2e57e4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f162e1f8640 1 -- 172.21.15.29:0/406088321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6390 msgr2=0x55f2e57e7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f162e1f8640 1 --2- 172.21.15.29:0/406088321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6390 0x55f2e57e7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f162e1f8640 1 -- 172.21.15.29:0/406088321 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6d80 msgr2=0x55f2e57e4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f162e1f8640 1 --2- 172.21.15.29:0/406088321 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6d80 0x55f2e57e4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f162e1f8640 1 -- 172.21.15.29:0/406088321 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f2e57382b0 con 0x55f2e57dbef0 2024-03-11T19:09:48.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f162e1f8640 1 --2- 172.21.15.29:0/406088321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57e4bd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1618029970 tx=0x7f16180296c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b736e91e0f40032 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f160d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:48.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f160d7fa640 1 -- 172.21.15.29:0/406088321 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16180049a0 con 0x55f2e57dbef0 2024-03-11T19:09:48.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f160d7fa640 1 -- 172.21.15.29:0/406088321 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f161802f030 con 0x55f2e57dbef0 2024-03-11T19:09:48.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.386+0000 7f163282a740 1 -- 172.21.15.29:0/406088321 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f161c09deb0 con 0x55f2e57dbef0 2024-03-11T19:09:48.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.387+0000 7f160d7fa640 1 -- 172.21.15.29:0/406088321 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1618004b00 con 0x55f2e57dbef0 2024-03-11T19:09:48.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.387+0000 7f163282a740 10 client.15264 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:48.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.387+0000 7f163282a740 1 -- 172.21.15.29:0/406088321 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55f2e56fa780 con 0x55f2e57dbef0 2024-03-11T19:09:48.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.387+0000 7f160d7fa640 1 -- 172.21.15.29:0/406088321 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f161c09deb0 con 0x55f2e57dbef0 2024-03-11T19:09:48.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.387+0000 7f160d7fa640 1 -- 172.21.15.29:0/406088321 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f161803a030 con 0x55f2e57dbef0 2024-03-11T19:09:48.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.387+0000 7f163282a740 10 client.15264 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:48.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.387+0000 7f163282a740 10 client.15264 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-11T19:09:48.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.387+0000 7f163282a740 20 client.15264 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:48.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.387+0000 7f163282a740 1 --2- 172.21.15.29:0/406088321 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55f2e5847ce0 0x55f2e58441a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.387+0000 7f163282a740 4 client.15264 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:09:48.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.387+0000 7f163282a740 1 -- 172.21.15.29:0/406088321 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f2e5476610 con 0x55f2e5847ce0 2024-03-11T19:09:48.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.387+0000 7f162d9f7640 1 --2- 172.21.15.29:0/406088321 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55f2e5847ce0 0x55f2e58441a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.388+0000 7f160d7fa640 10 client.15264 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:09:48.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.388+0000 7f162d9f7640 1 --2- 172.21.15.29:0/406088321 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55f2e5847ce0 0x55f2e58441a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=65c4c7979d05e8ed server_cookie=48504862fc40e664 in_seq=0 out_seq=0 2024-03-11T19:09:48.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.388+0000 7f160d7fa640 1 -- 172.21.15.29:0/406088321 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55f2e5476610 con 0x55f2e5847ce0 2024-03-11T19:09:48.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.388+0000 7f160d7fa640 10 client.15264 handle_command_reply: tid=0 2024-03-11T19:09:48.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.388+0000 7f163282a740 1 client.15264 shutdown 2024-03-11T19:09:48.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.388+0000 7f163282a740 1 -- 172.21.15.29:0/406088321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 msgr2=0x55f2e57e4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.388+0000 7f163282a740 1 --2- 172.21.15.29:0/406088321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57e4bd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1618029970 tx=0x7f16180296c0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.388+0000 7f163282a740 1 -- 172.21.15.29:0/406088321 shutdown_connections 2024-03-11T19:09:48.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.388+0000 7f163282a740 1 --2- 172.21.15.29:0/406088321 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55f2e5847ce0 0x55f2e58441a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.388+0000 7f163282a740 1 --2- 172.21.15.29:0/406088321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f2e57dbef0 0x55f2e57e4bd0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.388+0000 7f163282a740 1 --2- 172.21.15.29:0/406088321 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f2e57d6d80 0x55f2e57e4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.388+0000 7f160d7fa640 0 client.15264 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:09:48.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.388+0000 7f163282a740 1 --2- 172.21.15.29:0/406088321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2e57d6390 0x55f2e57e7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.388+0000 7f163282a740 1 -- 172.21.15.29:0/406088321 >> 172.21.15.29:0/406088321 conn(0x55f2e57cd4f0 msgr2=0x55f2e57f53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.388+0000 7f163282a740 1 -- 172.21.15.29:0/406088321 shutdown_connections 2024-03-11T19:09:48.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.388+0000 7f163282a740 1 -- 172.21.15.29:0/406088321 wait complete. 2024-03-11T19:09:48.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.389+0000 7f163282a740 20 client.15264 trim_cache size 0 max 16384 2024-03-11T19:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "638915c2-dfd9-11ee-95c8-87774f69a715", 2024-03-11T19:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 3, 2024-03-11T19:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24473, 2024-03-11T19:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-03-11T19:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-03-11T19:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-03-11T19:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 113.616357327, 2024-03-11T19:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 26, 2024-03-11T19:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 89, 2024-03-11T19:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 87, 2024-03-11T19:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 134.38804021600001 2024-03-11T19:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:09:48.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.389+0000 7f162fc5b640 1 -- 172.21.15.29:0/242070790 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f16100a1ab0 msgr2=0x7f16100a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.389+0000 7f162fc5b640 1 --2- 172.21.15.29:0/242070790 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f16100a1ab0 0x7f16100a3f70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55f2e57d75e0 tx=0x7f1618009b60 comp rx=0 tx=0).stop 2024-03-11T19:09:48.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.389+0000 7f162fc5b640 1 -- 172.21.15.29:0/242070790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16281023a0 msgr2=0x7f1628197730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.389+0000 7f162fc5b640 1 --2- 172.21.15.29:0/242070790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16281023a0 0x7f1628197730 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f161c00d770 tx=0x7f161c00dc30 comp rx=0 tx=0).stop 2024-03-11T19:09:48.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.389+0000 7f162fc5b640 1 -- 172.21.15.29:0/242070790 shutdown_connections 2024-03-11T19:09:48.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.389+0000 7f162fc5b640 1 --2- 172.21.15.29:0/242070790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1628102d60 0x7f162819e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.389+0000 7f162fc5b640 1 --2- 172.21.15.29:0/242070790 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f16100a1ab0 0x7f16100a3f70 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.389+0000 7f162fc5b640 1 --2- 172.21.15.29:0/242070790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16281023a0 0x7f1628197730 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.389+0000 7f162fc5b640 1 --2- 172.21.15.29:0/242070790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16281011b0 0x7f16281971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.389+0000 7f162fc5b640 1 -- 172.21.15.29:0/242070790 >> 172.21.15.29:0/242070790 conn(0x7f16280fc720 msgr2=0x7f16281057c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.389+0000 7f162fc5b640 1 -- 172.21.15.29:0/242070790 shutdown_connections 2024-03-11T19:09:48.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.389+0000 7f162fc5b640 1 -- 172.21.15.29:0/242070790 wait complete. 2024-03-11T19:09:48.402 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:48 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1175398603' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:09:48.402 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:48 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2022853446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:09:48.402 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:48 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/611890101' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:48.403 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-11T19:09:48.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.503+0000 7fa462c0e640 1 -- 172.21.15.29:0/2930913865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c101290 msgr2=0x7fa45c101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.503+0000 7fa462c0e640 1 --2- 172.21.15.29:0/2930913865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c101290 0x7fa45c101710 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa448009900 tx=0x7fa44802bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.503+0000 7fa462c0e640 1 -- 172.21.15.29:0/2930913865 shutdown_connections 2024-03-11T19:09:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.503+0000 7fa462c0e640 1 --2- 172.21.15.29:0/2930913865 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa45c101c50 0x7fa45c1084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.503+0000 7fa462c0e640 1 --2- 172.21.15.29:0/2930913865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c101290 0x7fa45c101710 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.503+0000 7fa462c0e640 1 --2- 172.21.15.29:0/2930913865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa45c1000a0 0x7fa45c1004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.503+0000 7fa462c0e640 1 -- 172.21.15.29:0/2930913865 >> 172.21.15.29:0/2930913865 conn(0x7fa45c0fb610 msgr2=0x7fa45c0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.503+0000 7fa462c0e640 1 -- 172.21.15.29:0/2930913865 shutdown_connections 2024-03-11T19:09:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.503+0000 7fa462c0e640 1 -- 172.21.15.29:0/2930913865 wait complete. 2024-03-11T19:09:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.504+0000 7fa462c0e640 1 Processor -- start 2024-03-11T19:09:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.504+0000 7fa462c0e640 1 -- start start 2024-03-11T19:09:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.504+0000 7fa462c0e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c1000a0 0x7fa45c1971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.504+0000 7fa462c0e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa45c101290 0x7fa45c197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.504+0000 7fa462c0e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa45c101c50 0x7fa45c19e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.504+0000 7fa462c0e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c19ecb0 con 0x7fa45c1000a0 2024-03-11T19:09:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.504+0000 7fa462c0e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c19ee20 con 0x7fa45c101c50 2024-03-11T19:09:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.504+0000 7fa462c0e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c19efe0 con 0x7fa45c101290 2024-03-11T19:09:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.504+0000 7fa4609aa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c1000a0 0x7fa45c1971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.504+0000 7fa4609aa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c1000a0 0x7fa45c1971c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49526/0 (socket says 172.21.15.29:49526) 2024-03-11T19:09:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.504+0000 7fa4609aa640 1 -- 172.21.15.29:0/3673787490 learned_addr learned my addr 172.21.15.29:0/3673787490 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.504+0000 7fa45bfff640 1 --2- 172.21.15.29:0/3673787490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa45c101290 0x7fa45c197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.505+0000 7fa4609aa640 1 -- 172.21.15.29:0/3673787490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa45c101290 msgr2=0x7fa45c197700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.505+0000 7fa4609aa640 1 --2- 172.21.15.29:0/3673787490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa45c101290 0x7fa45c197700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.505+0000 7fa4611ab640 1 --2- 172.21.15.29:0/3673787490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa45c101c50 0x7fa45c19e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.505+0000 7fa4609aa640 1 -- 172.21.15.29:0/3673787490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa45c101c50 msgr2=0x7fa45c19e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.505+0000 7fa4609aa640 1 --2- 172.21.15.29:0/3673787490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa45c101c50 0x7fa45c19e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.505+0000 7fa4609aa640 1 -- 172.21.15.29:0/3673787490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa448009560 con 0x7fa45c1000a0 2024-03-11T19:09:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.505+0000 7fa45bfff640 1 --2- 172.21.15.29:0/3673787490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa45c101290 0x7fa45c197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.505+0000 7fa4609aa640 1 --2- 172.21.15.29:0/3673787490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c1000a0 0x7fa45c1971c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa45000d770 tx=0x7fa45000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.505+0000 7fa459ffb640 1 -- 172.21.15.29:0/3673787490 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa450009c30 con 0x7fa45c1000a0 2024-03-11T19:09:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.505+0000 7fa459ffb640 1 -- 172.21.15.29:0/3673787490 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa4500044d0 con 0x7fa45c1000a0 2024-03-11T19:09:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.505+0000 7fa462c0e640 1 -- 172.21.15.29:0/3673787490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa45c19f2c0 con 0x7fa45c1000a0 2024-03-11T19:09:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.505+0000 7fa459ffb640 1 -- 172.21.15.29:0/3673787490 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa450003de0 con 0x7fa45c1000a0 2024-03-11T19:09:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.505+0000 7fa462c0e640 1 -- 172.21.15.29:0/3673787490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45c19f7d0 con 0x7fa45c1000a0 2024-03-11T19:09:48.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:48 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1175398603' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:09:48.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:48 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2022853446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:09:48.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:48 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/611890101' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.508+0000 7fa459ffb640 1 -- 172.21.15.29:0/3673787490 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fa450003f40 con 0x7fa45c1000a0 2024-03-11T19:09:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.508+0000 7fa4657e9740 1 Processor -- start 2024-03-11T19:09:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.508+0000 7fa4657e9740 1 -- start start 2024-03-11T19:09:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.508+0000 7fa4657e9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac2ba2f80 0x55aac2ba33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.508+0000 7fa4657e9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba3970 0x55aac2ba8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.508+0000 7fa4657e9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba8ae0 0x55aac2baaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.508+0000 7fa4657e9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac2b5ecd0 con 0x55aac2ba3970 2024-03-11T19:09:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.508+0000 7fa4657e9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac2b11f20 con 0x55aac2ba2f80 2024-03-11T19:09:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.508+0000 7fa4657e9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac2b0f450 con 0x55aac2ba8ae0 2024-03-11T19:09:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.508+0000 7fa459ffb640 1 --2- 172.21.15.29:0/3673787490 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa44c0a1a30 0x7fa44c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.508+0000 7fa4609aa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba3970 0x55aac2ba8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.508+0000 7fa4611ab640 1 --2- 172.21.15.29:0/3673787490 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa44c0a1a30 0x7fa44c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.508+0000 7fa4609aa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba3970 0x55aac2ba8470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49538/0 (socket says 172.21.15.29:49538) 2024-03-11T19:09:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.508+0000 7fa4609aa640 1 -- 172.21.15.29:0/1545474769 learned_addr learned my addr 172.21.15.29:0/1545474769 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.508+0000 7fa45bfff640 1 --2- 172.21.15.29:0/1545474769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba8ae0 0x55aac2baaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.509+0000 7fa459ffb640 1 -- 172.21.15.29:0/3673787490 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fa4500ccd80 con 0x7fa45c1000a0 2024-03-11T19:09:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.509+0000 7fa4611ab640 1 --2- 172.21.15.29:0/1545474769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac2ba2f80 0x55aac2ba33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.509+0000 7fa4609aa640 1 -- 172.21.15.29:0/1545474769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba8ae0 msgr2=0x55aac2baaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.509+0000 7fa4609aa640 1 --2- 172.21.15.29:0/1545474769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba8ae0 0x55aac2baaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.509+0000 7fa4609aa640 1 -- 172.21.15.29:0/1545474769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac2ba2f80 msgr2=0x55aac2ba33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.509+0000 7fa4609aa640 1 --2- 172.21.15.29:0/1545474769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac2ba2f80 0x55aac2ba33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.509+0000 7fa4609aa640 1 -- 172.21.15.29:0/1545474769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aac2abff20 con 0x55aac2ba3970 2024-03-11T19:09:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.509+0000 7fa45bfff640 1 --2- 172.21.15.29:0/1545474769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba8ae0 0x55aac2baaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.509+0000 7fa4611ab640 1 --2- 172.21.15.29:0/3673787490 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa44c0a1a30 0x7fa44c0a3ef0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa454005fd0 tx=0x7fa454013aa0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.509+0000 7fa4609aa640 1 --2- 172.21.15.29:0/1545474769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba3970 0x55aac2ba8470 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa45000d740 tx=0x7fa45006d090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d08b7ed9aa5bd4c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.509+0000 7fa446ffd640 1 -- 172.21.15.29:0/1545474769 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4500998d0 con 0x55aac2ba3970 2024-03-11T19:09:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.509+0000 7fa446ffd640 1 -- 172.21.15.29:0/1545474769 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa4500996f0 con 0x55aac2ba3970 2024-03-11T19:09:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.509+0000 7fa446ffd640 1 -- 172.21.15.29:0/1545474769 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4500041f0 con 0x55aac2ba3970 2024-03-11T19:09:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.510+0000 7fa4657e9740 1 -- 172.21.15.29:0/1545474769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba3970 msgr2=0x55aac2ba8470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.510+0000 7fa4657e9740 1 --2- 172.21.15.29:0/1545474769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba3970 0x55aac2ba8470 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa45000d740 tx=0x7fa45006d090 comp rx=0 tx=0).stop 2024-03-11T19:09:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.510+0000 7fa4657e9740 1 -- 172.21.15.29:0/1545474769 shutdown_connections 2024-03-11T19:09:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.510+0000 7fa4657e9740 1 --2- 172.21.15.29:0/1545474769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba8ae0 0x55aac2baaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.510+0000 7fa4657e9740 1 --2- 172.21.15.29:0/1545474769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba3970 0x55aac2ba8470 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.510+0000 7fa4657e9740 1 --2- 172.21.15.29:0/1545474769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac2ba2f80 0x55aac2ba33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.510+0000 7fa4657e9740 1 -- 172.21.15.29:0/1545474769 >> 172.21.15.29:0/1545474769 conn(0x55aac2b9e4f0 msgr2=0x55aac2b9e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.510+0000 7fa4657e9740 1 -- 172.21.15.29:0/1545474769 shutdown_connections 2024-03-11T19:09:48.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.510+0000 7fa4657e9740 1 -- 172.21.15.29:0/1545474769 wait complete. 2024-03-11T19:09:48.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.510+0000 7fa4657e9740 1 -- start start 2024-03-11T19:09:48.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.510+0000 7fa4657e9740 1 Processor -- start 2024-03-11T19:09:48.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.510+0000 7fa4657e9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba2f80 0x55aac2bad0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa4657e9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac2ba3970 0x55aac2bad620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa4657e9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 0x55aac2bb4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa4657e9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac2b0ac30 con 0x55aac2ba8ae0 2024-03-11T19:09:48.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa4657e9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac2aafd30 con 0x55aac2ba3970 2024-03-11T19:09:48.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa4657e9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac27de310 con 0x55aac2ba2f80 2024-03-11T19:09:48.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa4657e9740 10 client.0 _ll_register_callbacks cb 0x55aac2ab7d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:48.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa4611ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 0x55aac2bb4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa4609aa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba2f80 0x55aac2bad0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa4611ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 0x55aac2bb4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49542/0 (socket says 172.21.15.29:49542) 2024-03-11T19:09:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa4611ab640 1 -- 172.21.15.29:0/421046471 learned_addr learned my addr 172.21.15.29:0/421046471 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa45bfff640 1 --2- 172.21.15.29:0/421046471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac2ba3970 0x55aac2bad620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa4611ab640 1 -- 172.21.15.29:0/421046471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba2f80 msgr2=0x55aac2bad0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa4611ab640 1 --2- 172.21.15.29:0/421046471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba2f80 0x55aac2bad0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa4611ab640 1 -- 172.21.15.29:0/421046471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac2ba3970 msgr2=0x55aac2bad620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa4611ab640 1 --2- 172.21.15.29:0/421046471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac2ba3970 0x55aac2bad620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa4611ab640 1 -- 172.21.15.29:0/421046471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45c19f7d0 con 0x55aac2ba8ae0 2024-03-11T19:09:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa4611ab640 1 --2- 172.21.15.29:0/421046471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 0x55aac2bb4660 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa454011880 tx=0x7fa454009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b736a531ef9704 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa43bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa43bfff640 1 -- 172.21.15.29:0/421046471 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454011ec0 con 0x55aac2ba8ae0 2024-03-11T19:09:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa4657e9740 1 -- 172.21.15.29:0/421046471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aac280a6c0 con 0x55aac2ba8ae0 2024-03-11T19:09:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa43bfff640 1 -- 172.21.15.29:0/421046471 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fa454012ae0 con 0x55aac2ba8ae0 2024-03-11T19:09:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa43bfff640 1 -- 172.21.15.29:0/421046471 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa454012ca0 con 0x55aac2ba8ae0 2024-03-11T19:09:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa4657e9740 10 client.15276 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.511+0000 7fa4657e9740 1 -- 172.21.15.29:0/421046471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55aac2b092b0 con 0x55aac2ba8ae0 2024-03-11T19:09:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.512+0000 7fa43bfff640 1 -- 172.21.15.29:0/421046471 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa454030020 con 0x55aac2ba8ae0 2024-03-11T19:09:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.512+0000 7fa43bfff640 1 -- 172.21.15.29:0/421046471 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa45402a450 con 0x55aac2ba8ae0 2024-03-11T19:09:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.512+0000 7fa4657e9740 10 client.15276 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.512+0000 7fa4657e9740 10 client.15276 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-11T19:09:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.512+0000 7fa4657e9740 20 client.15276 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.512+0000 7fa4657e9740 1 --2- 172.21.15.29:0/421046471 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55aac2bb9ad0 0x55aac2bb8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.512+0000 7fa4657e9740 4 client.15276 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:09:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.512+0000 7fa4657e9740 1 -- 172.21.15.29:0/421046471 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aac285b270 con 0x55aac2bb9ad0 2024-03-11T19:09:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.512+0000 7fa4609aa640 1 --2- 172.21.15.29:0/421046471 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55aac2bb9ad0 0x55aac2bb8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.512+0000 7fa43bfff640 10 client.15276 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:09:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.512+0000 7fa4609aa640 1 --2- 172.21.15.29:0/421046471 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55aac2bb9ad0 0x55aac2bb8d80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=54761b3cda1573de server_cookie=11bbe17044ac0fdd in_seq=0 out_seq=0 2024-03-11T19:09:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.513+0000 7fa43bfff640 1 -- 172.21.15.29:0/421046471 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55aac285b270 con 0x55aac2bb9ad0 2024-03-11T19:09:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.513+0000 7fa43bfff640 10 client.15276 handle_command_reply: tid=0 2024-03-11T19:09:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.513+0000 7fa4657e9740 1 client.15276 shutdown 2024-03-11T19:09:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.513+0000 7fa4657e9740 1 -- 172.21.15.29:0/421046471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 msgr2=0x55aac2bb4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.513+0000 7fa4657e9740 1 --2- 172.21.15.29:0/421046471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 0x55aac2bb4660 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa454011880 tx=0x7fa454009500 comp rx=0 tx=0).stop 2024-03-11T19:09:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.513+0000 7fa4657e9740 1 -- 172.21.15.29:0/421046471 shutdown_connections 2024-03-11T19:09:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.513+0000 7fa4657e9740 1 --2- 172.21.15.29:0/421046471 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55aac2bb9ad0 0x55aac2bb8d80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.513+0000 7fa4657e9740 1 --2- 172.21.15.29:0/421046471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 0x55aac2bb4660 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.513+0000 7fa4657e9740 1 --2- 172.21.15.29:0/421046471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac2ba3970 0x55aac2bad620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.513+0000 7fa43bfff640 0 client.15276 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:09:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.513+0000 7fa4657e9740 1 --2- 172.21.15.29:0/421046471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba2f80 0x55aac2bad0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.513+0000 7fa4657e9740 1 -- 172.21.15.29:0/421046471 >> 172.21.15.29:0/421046471 conn(0x55aac2b9e4f0 msgr2=0x55aac2b9d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.513+0000 7fa4657e9740 1 -- 172.21.15.29:0/421046471 shutdown_connections 2024-03-11T19:09:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.513+0000 7fa4657e9740 1 -- 172.21.15.29:0/421046471 wait complete. 2024-03-11T19:09:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.513+0000 7fa4657e9740 20 client.15276 trim_cache size 0 max 16384 2024-03-11T19:09:48.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.529+0000 7fa4657e9740 1 Processor -- start 2024-03-11T19:09:48.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.529+0000 7fa4657e9740 1 -- start start 2024-03-11T19:09:48.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.529+0000 7fa4657e9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba2f80 0x55aac2ba4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.529+0000 7fa4657e9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac2ba3970 0x55aac2b9e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.529+0000 7fa4657e9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 0x55aac2bad0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.529+0000 7fa4657e9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac2a90490 con 0x55aac2ba8ae0 2024-03-11T19:09:48.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.529+0000 7fa4657e9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac2bbbab0 con 0x55aac2ba3970 2024-03-11T19:09:48.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.529+0000 7fa4657e9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac285b270 con 0x55aac2ba2f80 2024-03-11T19:09:48.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.529+0000 7fa4611ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 0x55aac2bad0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.529+0000 7fa4611ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 0x55aac2bad0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49550/0 (socket says 172.21.15.29:49550) 2024-03-11T19:09:48.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.529+0000 7fa4611ab640 1 -- 172.21.15.29:0/1815492952 learned_addr learned my addr 172.21.15.29:0/1815492952 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.530+0000 7fa4611ab640 1 -- 172.21.15.29:0/1815492952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba2f80 msgr2=0x55aac2ba4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.530+0000 7fa4611ab640 1 --2- 172.21.15.29:0/1815492952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba2f80 0x55aac2ba4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.530+0000 7fa4611ab640 1 -- 172.21.15.29:0/1815492952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac2ba3970 msgr2=0x55aac2b9e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.530+0000 7fa4611ab640 1 --2- 172.21.15.29:0/1815492952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac2ba3970 0x55aac2b9e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.530+0000 7fa4611ab640 1 -- 172.21.15.29:0/1815492952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aac2aa5a20 con 0x55aac2ba8ae0 2024-03-11T19:09:48.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.530+0000 7fa4611ab640 1 --2- 172.21.15.29:0/1815492952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 0x55aac2bad0e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa4540128a0 tx=0x7fa454011780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf2e30fc066e81d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.530+0000 7fa4467fc640 1 -- 172.21.15.29:0/1815492952 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454011ec0 con 0x55aac2ba8ae0 2024-03-11T19:09:48.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.530+0000 7fa4467fc640 1 -- 172.21.15.29:0/1815492952 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa454030040 con 0x55aac2ba8ae0 2024-03-11T19:09:48.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.530+0000 7fa4467fc640 1 -- 172.21.15.29:0/1815492952 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45402aea0 con 0x55aac2ba8ae0 2024-03-11T19:09:48.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.530+0000 7fa4657e9740 1 -- 172.21.15.29:0/1815492952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 msgr2=0x55aac2bad0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.530+0000 7fa4657e9740 1 --2- 172.21.15.29:0/1815492952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 0x55aac2bad0e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa4540128a0 tx=0x7fa454011780 comp rx=0 tx=0).stop 2024-03-11T19:09:48.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.530+0000 7fa4657e9740 1 -- 172.21.15.29:0/1815492952 shutdown_connections 2024-03-11T19:09:48.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.530+0000 7fa4657e9740 1 --2- 172.21.15.29:0/1815492952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 0x55aac2bad0e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.530+0000 7fa4657e9740 1 --2- 172.21.15.29:0/1815492952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac2ba3970 0x55aac2b9e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.530+0000 7fa4657e9740 1 --2- 172.21.15.29:0/1815492952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba2f80 0x55aac2ba4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.530+0000 7fa4657e9740 1 -- 172.21.15.29:0/1815492952 >> 172.21.15.29:0/1815492952 conn(0x55aac2b9e4f0 msgr2=0x55aac2babbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa4657e9740 1 -- 172.21.15.29:0/1815492952 shutdown_connections 2024-03-11T19:09:48.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa4657e9740 1 -- 172.21.15.29:0/1815492952 wait complete. 2024-03-11T19:09:48.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa4657e9740 1 -- start start 2024-03-11T19:09:48.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa4657e9740 1 Processor -- start 2024-03-11T19:09:48.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa4657e9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac2ba2f80 0x55aac2c0c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa4657e9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba3970 0x55aac2c0c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa4657e9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 0x55aac2c13800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa4657e9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac2c13d40 con 0x55aac2ba8ae0 2024-03-11T19:09:48.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa4657e9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac2c13e80 con 0x55aac2ba2f80 2024-03-11T19:09:48.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa4657e9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac2c13ff0 con 0x55aac2ba3970 2024-03-11T19:09:48.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa4657e9740 10 client.0 _ll_register_callbacks cb 0x7fa450009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:48.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa4611ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 0x55aac2c13800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa4611ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 0x55aac2c13800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49560/0 (socket says 172.21.15.29:49560) 2024-03-11T19:09:48.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa4611ab640 1 -- 172.21.15.29:0/821882316 learned_addr learned my addr 172.21.15.29:0/821882316 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa4609aa640 1 --2- 172.21.15.29:0/821882316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac2ba2f80 0x55aac2c0c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa45bfff640 1 --2- 172.21.15.29:0/821882316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba3970 0x55aac2c0c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa4611ab640 1 -- 172.21.15.29:0/821882316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba3970 msgr2=0x55aac2c0c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa4611ab640 1 --2- 172.21.15.29:0/821882316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba3970 0x55aac2c0c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa4611ab640 1 -- 172.21.15.29:0/821882316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac2ba2f80 msgr2=0x55aac2c0c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.531+0000 7fa4611ab640 1 --2- 172.21.15.29:0/821882316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac2ba2f80 0x55aac2c0c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.532+0000 7fa4611ab640 1 -- 172.21.15.29:0/821882316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aac2b092b0 con 0x55aac2ba8ae0 2024-03-11T19:09:48.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.532+0000 7fa45bfff640 1 --2- 172.21.15.29:0/821882316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba3970 0x55aac2c0c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:48.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.532+0000 7fa4611ab640 1 --2- 172.21.15.29:0/821882316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 0x55aac2c13800 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa45402aff0 tx=0x7fa45402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=613455c66106c92f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.532+0000 7fa43bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:48.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.532+0000 7fa43bfff640 1 -- 172.21.15.29:0/821882316 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454012cf0 con 0x55aac2ba8ae0 2024-03-11T19:09:48.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.532+0000 7fa43bfff640 1 -- 172.21.15.29:0/821882316 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fa454031080 con 0x55aac2ba8ae0 2024-03-11T19:09:48.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.532+0000 7fa4657e9740 1 -- 172.21.15.29:0/821882316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa4500996f0 con 0x55aac2ba8ae0 2024-03-11T19:09:48.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.532+0000 7fa43bfff640 1 -- 172.21.15.29:0/821882316 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa454012e50 con 0x55aac2ba8ae0 2024-03-11T19:09:48.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.532+0000 7fa4657e9740 10 client.15282 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:48.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.532+0000 7fa4657e9740 1 -- 172.21.15.29:0/821882316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55aac2acb780 con 0x55aac2ba8ae0 2024-03-11T19:09:48.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.532+0000 7fa43bfff640 1 -- 172.21.15.29:0/821882316 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa4500996f0 con 0x55aac2ba8ae0 2024-03-11T19:09:48.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.532+0000 7fa43bfff640 1 -- 172.21.15.29:0/821882316 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa454005730 con 0x55aac2ba8ae0 2024-03-11T19:09:48.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.532+0000 7fa4657e9740 10 client.15282 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.532+0000 7fa4657e9740 10 client.15282 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-11T19:09:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.532+0000 7fa4657e9740 20 client.15282 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.532+0000 7fa4657e9740 1 --2- 172.21.15.29:0/821882316 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55aac2c190f0 0x55aac2c15590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.532+0000 7fa4657e9740 4 client.15282 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:09:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.532+0000 7fa4657e9740 1 -- 172.21.15.29:0/821882316 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55aac2847610 con 0x55aac2c190f0 2024-03-11T19:09:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.533+0000 7fa4609aa640 1 --2- 172.21.15.29:0/821882316 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55aac2c190f0 0x55aac2c15590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.533+0000 7fa4609aa640 1 --2- 172.21.15.29:0/821882316 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55aac2c190f0 0x55aac2c15590 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c4633bef8c60074 server_cookie=18e7606c19397e6a in_seq=0 out_seq=0 2024-03-11T19:09:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.533+0000 7fa43bfff640 10 client.15282 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:09:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.533+0000 7fa43bfff640 1 -- 172.21.15.29:0/821882316 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55aac2847610 con 0x55aac2c190f0 2024-03-11T19:09:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.533+0000 7fa43bfff640 10 client.15282 handle_command_reply: tid=0 2024-03-11T19:09:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.533+0000 7fa4657e9740 1 client.15282 shutdown 2024-03-11T19:09:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.533+0000 7fa4657e9740 1 -- 172.21.15.29:0/821882316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 msgr2=0x55aac2c13800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.533+0000 7fa4657e9740 1 --2- 172.21.15.29:0/821882316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 0x55aac2c13800 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa45402aff0 tx=0x7fa45402a680 comp rx=0 tx=0).stop 2024-03-11T19:09:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa4657e9740 1 -- 172.21.15.29:0/821882316 shutdown_connections 2024-03-11T19:09:48.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa4657e9740 1 --2- 172.21.15.29:0/821882316 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55aac2c190f0 0x55aac2c15590 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa4657e9740 1 --2- 172.21.15.29:0/821882316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac2ba8ae0 0x55aac2c13800 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa4657e9740 1 --2- 172.21.15.29:0/821882316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55aac2ba3970 0x55aac2c0c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa43bfff640 0 client.15282 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:09:48.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa4657e9740 1 --2- 172.21.15.29:0/821882316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac2ba2f80 0x55aac2c0c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa4657e9740 1 -- 172.21.15.29:0/821882316 >> 172.21.15.29:0/821882316 conn(0x55aac2b9e4f0 msgr2=0x55aac2ba4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa4657e9740 1 -- 172.21.15.29:0/821882316 shutdown_connections 2024-03-11T19:09:48.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa4657e9740 1 -- 172.21.15.29:0/821882316 wait complete. 2024-03-11T19:09:48.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa4657e9740 20 client.15282 trim_cache size 0 max 16384 2024-03-11T19:09:48.543 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:09:48.543 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "638915c2-dfd9-11ee-95c8-87774f69a715", 2024-03-11T19:09:48.543 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 2, 2024-03-11T19:09:48.544 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24475, 2024-03-11T19:09:48.544 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-03-11T19:09:48.544 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-03-11T19:09:48.544 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-03-11T19:09:48.544 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 114.771481908, 2024-03-11T19:09:48.544 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 26, 2024-03-11T19:09:48.544 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 89, 2024-03-11T19:09:48.544 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 87, 2024-03-11T19:09:48.544 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 137.441598271 2024-03-11T19:09:48.544 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:09:48.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa462c0e640 1 -- 172.21.15.29:0/3673787490 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa44c0a1a30 msgr2=0x7fa44c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa462c0e640 1 --2- 172.21.15.29:0/3673787490 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa44c0a1a30 0x7fa44c0a3ef0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa454005fd0 tx=0x7fa454013aa0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa462c0e640 1 -- 172.21.15.29:0/3673787490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c1000a0 msgr2=0x7fa45c1971c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa462c0e640 1 --2- 172.21.15.29:0/3673787490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c1000a0 0x7fa45c1971c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa45000d770 tx=0x7fa45000dc30 comp rx=0 tx=0).stop 2024-03-11T19:09:48.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa462c0e640 1 -- 172.21.15.29:0/3673787490 shutdown_connections 2024-03-11T19:09:48.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa462c0e640 1 --2- 172.21.15.29:0/3673787490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa45c101c50 0x7fa45c19e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa462c0e640 1 --2- 172.21.15.29:0/3673787490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa45c101290 0x7fa45c197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa462c0e640 1 --2- 172.21.15.29:0/3673787490 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa44c0a1a30 0x7fa44c0a3ef0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa462c0e640 1 --2- 172.21.15.29:0/3673787490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c1000a0 0x7fa45c1971c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa462c0e640 1 -- 172.21.15.29:0/3673787490 >> 172.21.15.29:0/3673787490 conn(0x7fa45c0fb610 msgr2=0x7fa45c104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa462c0e640 1 -- 172.21.15.29:0/3673787490 shutdown_connections 2024-03-11T19:09:48.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.534+0000 7fa462c0e640 1 -- 172.21.15.29:0/3673787490 wait complete. 2024-03-11T19:09:48.548 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-11T19:09:48.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:48 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1175398603' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:09:48.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:48 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2022853446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:09:48.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:48 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/611890101' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:48.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.648+0000 7f28c9d25640 1 -- 172.21.15.29:0/477097138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c41016b0 msgr2=0x7f28c4101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.648+0000 7f28c9d25640 1 --2- 172.21.15.29:0/477097138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c41016b0 0x7f28c4101ad0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f28b4009900 tx=0x7f28b402bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:48.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.648+0000 7f28c9d25640 1 -- 172.21.15.29:0/477097138 shutdown_connections 2024-03-11T19:09:48.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.648+0000 7f28c9d25640 1 --2- 172.21.15.29:0/477097138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28c4100cc0 0x7f28c4108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.648+0000 7f28c9d25640 1 --2- 172.21.15.29:0/477097138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28c4100300 0x7f28c4100780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.648+0000 7f28c9d25640 1 --2- 172.21.15.29:0/477097138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c41016b0 0x7f28c4101ad0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.648+0000 7f28c9d25640 1 -- 172.21.15.29:0/477097138 >> 172.21.15.29:0/477097138 conn(0x7f28c40fad70 msgr2=0x7f28c40fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.648+0000 7f28c9d25640 1 -- 172.21.15.29:0/477097138 shutdown_connections 2024-03-11T19:09:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.648+0000 7f28c9d25640 1 -- 172.21.15.29:0/477097138 wait complete. 2024-03-11T19:09:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.649+0000 7f28c9d25640 1 Processor -- start 2024-03-11T19:09:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.649+0000 7f28c9d25640 1 -- start start 2024-03-11T19:09:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.649+0000 7f28c9d25640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28c4100300 0x7f28c40ffdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.649+0000 7f28c9d25640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c4100cc0 0x7f28c419be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.649+0000 7f28c9d25640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28c41016b0 0x7f28c419e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.649+0000 7f28c9d25640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c419ec30 con 0x7f28c4100cc0 2024-03-11T19:09:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.649+0000 7f28c9d25640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c419eda0 con 0x7f28c4100300 2024-03-11T19:09:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.649+0000 7f28c9d25640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c419ef60 con 0x7f28c41016b0 2024-03-11T19:09:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.649+0000 7f28c2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c4100cc0 0x7f28c419be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.649+0000 7f28c2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c4100cc0 0x7f28c419be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49572/0 (socket says 172.21.15.29:49572) 2024-03-11T19:09:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.649+0000 7f28c2ffd640 1 -- 172.21.15.29:0/211191535 learned_addr learned my addr 172.21.15.29:0/211191535 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.649+0000 7f28c37fe640 1 --2- 172.21.15.29:0/211191535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28c4100300 0x7f28c40ffdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.649+0000 7f28c3fff640 1 --2- 172.21.15.29:0/211191535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28c41016b0 0x7f28c419e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.649+0000 7f28c2ffd640 1 -- 172.21.15.29:0/211191535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28c41016b0 msgr2=0x7f28c419e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.649+0000 7f28c2ffd640 1 --2- 172.21.15.29:0/211191535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28c41016b0 0x7f28c419e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.650+0000 7f28c2ffd640 1 -- 172.21.15.29:0/211191535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28c4100300 msgr2=0x7f28c40ffdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.650+0000 7f28c2ffd640 1 --2- 172.21.15.29:0/211191535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28c4100300 0x7f28c40ffdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.650+0000 7f28c2ffd640 1 -- 172.21.15.29:0/211191535 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b4009560 con 0x7f28c4100cc0 2024-03-11T19:09:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.650+0000 7f28c2ffd640 1 --2- 172.21.15.29:0/211191535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c4100cc0 0x7f28c419be30 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f28b800d770 tx=0x7f28b800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.650+0000 7f28c0ff9640 1 -- 172.21.15.29:0/211191535 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b8009c30 con 0x7f28c4100cc0 2024-03-11T19:09:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.650+0000 7f28c0ff9640 1 -- 172.21.15.29:0/211191535 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f28b80044d0 con 0x7f28c4100cc0 2024-03-11T19:09:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.650+0000 7f28c9d25640 1 -- 172.21.15.29:0/211191535 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c419f240 con 0x7f28c4100cc0 2024-03-11T19:09:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.650+0000 7f28c0ff9640 1 -- 172.21.15.29:0/211191535 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b8003de0 con 0x7f28c4100cc0 2024-03-11T19:09:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.650+0000 7f28c9d25640 1 -- 172.21.15.29:0/211191535 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c419f750 con 0x7f28c4100cc0 2024-03-11T19:09:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.652+0000 7f28c0ff9640 1 -- 172.21.15.29:0/211191535 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f28b8003f40 con 0x7f28c4100cc0 2024-03-11T19:09:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.652+0000 7f28cc9d5740 1 Processor -- start 2024-03-11T19:09:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.652+0000 7f28cc9d5740 1 -- start start 2024-03-11T19:09:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28cc9d5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a1f80 0x5640df5a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28cc9d5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a2970 0x5640df5a7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28cc9d5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 0x5640df5a9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28cc9d5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640df55dcd0 con 0x5640df5a7ae0 2024-03-11T19:09:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28cc9d5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640df510f20 con 0x5640df5a2970 2024-03-11T19:09:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28cc9d5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640df50e450 con 0x5640df5a1f80 2024-03-11T19:09:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28c2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 0x5640df5a9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28c0ff9640 1 --2- 172.21.15.29:0/211191535 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f28ac0a1a30 0x7f28ac0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28c2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 0x5640df5a9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49582/0 (socket says 172.21.15.29:49582) 2024-03-11T19:09:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28c2ffd640 1 -- 172.21.15.29:0/2940548769 learned_addr learned my addr 172.21.15.29:0/2940548769 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28c3fff640 1 --2- 172.21.15.29:0/2940548769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a1f80 0x5640df5a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28c37fe640 1 --2- 172.21.15.29:0/2940548769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a2970 0x5640df5a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28c3fff640 1 --2- 172.21.15.29:0/211191535 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f28ac0a1a30 0x7f28ac0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28c0ff9640 1 -- 172.21.15.29:0/211191535 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f28b80cce20 con 0x7f28c4100cc0 2024-03-11T19:09:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28c2ffd640 1 -- 172.21.15.29:0/2940548769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a1f80 msgr2=0x5640df5a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28c2ffd640 1 --2- 172.21.15.29:0/2940548769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a1f80 0x5640df5a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28c2ffd640 1 -- 172.21.15.29:0/2940548769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a2970 msgr2=0x5640df5a7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28c2ffd640 1 --2- 172.21.15.29:0/2940548769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a2970 0x5640df5a7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28c2ffd640 1 -- 172.21.15.29:0/2940548769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5640df4bef20 con 0x5640df5a7ae0 2024-03-11T19:09:48.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28c3fff640 1 --2- 172.21.15.29:0/2940548769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a1f80 0x5640df5a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:09:48.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28c2ffd640 1 --2- 172.21.15.29:0/2940548769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 0x5640df5a9f70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f28b80ca920 tx=0x7f28b8084b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f57d1aa97ad23f71 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28c3fff640 1 --2- 172.21.15.29:0/211191535 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f28ac0a1a30 0x7f28ac0a3ef0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x5640df5a31d0 tx=0x7f28bc004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.653+0000 7f28a9ffb640 1 -- 172.21.15.29:0/2940548769 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b80998d0 con 0x5640df5a7ae0 2024-03-11T19:09:48.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.654+0000 7f28a9ffb640 1 -- 172.21.15.29:0/2940548769 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f28b80996f0 con 0x5640df5a7ae0 2024-03-11T19:09:48.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.654+0000 7f28a9ffb640 1 -- 172.21.15.29:0/2940548769 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b80a25c0 con 0x5640df5a7ae0 2024-03-11T19:09:48.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.654+0000 7f28cc9d5740 1 -- 172.21.15.29:0/2940548769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 msgr2=0x5640df5a9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.654+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/2940548769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 0x5640df5a9f70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f28b80ca920 tx=0x7f28b8084b50 comp rx=0 tx=0).stop 2024-03-11T19:09:48.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.654+0000 7f28cc9d5740 1 -- 172.21.15.29:0/2940548769 shutdown_connections 2024-03-11T19:09:48.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.654+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/2940548769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 0x5640df5a9f70 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.654+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/2940548769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a2970 0x5640df5a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.654+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/2940548769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a1f80 0x5640df5a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.654+0000 7f28cc9d5740 1 -- 172.21.15.29:0/2940548769 >> 172.21.15.29:0/2940548769 conn(0x5640df59d4f0 msgr2=0x5640df59d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.654+0000 7f28cc9d5740 1 -- 172.21.15.29:0/2940548769 shutdown_connections 2024-03-11T19:09:48.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.654+0000 7f28cc9d5740 1 -- 172.21.15.29:0/2940548769 wait complete. 2024-03-11T19:09:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.654+0000 7f28cc9d5740 1 -- start start 2024-03-11T19:09:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28cc9d5740 1 Processor -- start 2024-03-11T19:09:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28cc9d5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a1f80 0x5640df5ac110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28cc9d5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a2970 0x5640df5ac650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28cc9d5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 0x5640df5b3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28cc9d5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640df509c30 con 0x5640df5a7ae0 2024-03-11T19:09:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28cc9d5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640df4aed30 con 0x5640df5a1f80 2024-03-11T19:09:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28cc9d5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640df1dd310 con 0x5640df5a2970 2024-03-11T19:09:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28cc9d5740 10 client.0 _ll_register_callbacks cb 0x5640df4b6d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28c3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 0x5640df5b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28c3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 0x5640df5b3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49594/0 (socket says 172.21.15.29:49594) 2024-03-11T19:09:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28c3fff640 1 -- 172.21.15.29:0/3642934363 learned_addr learned my addr 172.21.15.29:0/3642934363 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28c2ffd640 1 --2- 172.21.15.29:0/3642934363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a2970 0x5640df5ac650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28c37fe640 1 --2- 172.21.15.29:0/3642934363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a1f80 0x5640df5ac110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28c3fff640 1 -- 172.21.15.29:0/3642934363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a2970 msgr2=0x5640df5ac650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28c3fff640 1 --2- 172.21.15.29:0/3642934363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a2970 0x5640df5ac650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28c3fff640 1 -- 172.21.15.29:0/3642934363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a1f80 msgr2=0x5640df5ac110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28c3fff640 1 --2- 172.21.15.29:0/3642934363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a1f80 0x5640df5ac110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28c3fff640 1 -- 172.21.15.29:0/3642934363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c419f750 con 0x5640df5a7ae0 2024-03-11T19:09:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28c37fe640 1 --2- 172.21.15.29:0/3642934363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a1f80 0x5640df5ac110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.655+0000 7f28c3fff640 1 --2- 172.21.15.29:0/3642934363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 0x5640df5b3620 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f28bc013390 tx=0x7f28bc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a89493ebc2a6feaf server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.656+0000 7f28a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.656+0000 7f28a2ffd640 1 -- 172.21.15.29:0/3642934363 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28bc012e60 con 0x5640df5a7ae0 2024-03-11T19:09:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.656+0000 7f28a2ffd640 1 -- 172.21.15.29:0/3642934363 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f28bc028410 con 0x5640df5a7ae0 2024-03-11T19:09:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.656+0000 7f28cc9d5740 1 -- 172.21.15.29:0/3642934363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5640df2096c0 con 0x5640df5a7ae0 2024-03-11T19:09:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.656+0000 7f28a2ffd640 1 -- 172.21.15.29:0/3642934363 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28bc0285d0 con 0x5640df5a7ae0 2024-03-11T19:09:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.656+0000 7f28cc9d5740 10 client.15294 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.656+0000 7f28cc9d5740 1 -- 172.21.15.29:0/3642934363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5640df5082b0 con 0x5640df5a7ae0 2024-03-11T19:09:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.656+0000 7f28a2ffd640 1 -- 172.21.15.29:0/3642934363 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5640df2096c0 con 0x5640df5a7ae0 2024-03-11T19:09:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.656+0000 7f28a2ffd640 1 -- 172.21.15.29:0/3642934363 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f28bc00e720 con 0x5640df5a7ae0 2024-03-11T19:09:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.656+0000 7f28cc9d5740 10 client.15294 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.656+0000 7f28cc9d5740 10 client.15294 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-11T19:09:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.656+0000 7f28cc9d5740 20 client.15294 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.656+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/3642934363 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5640df5b8ab0 0x5640df5b7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.656+0000 7f28cc9d5740 4 client.15294 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:09:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.656+0000 7f28cc9d5740 1 -- 172.21.15.29:0/3642934363 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5640df25a270 con 0x5640df5b8ab0 2024-03-11T19:09:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.657+0000 7f28c37fe640 1 --2- 172.21.15.29:0/3642934363 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5640df5b8ab0 0x5640df5b7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.657+0000 7f28c37fe640 1 --2- 172.21.15.29:0/3642934363 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5640df5b8ab0 0x5640df5b7cf0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4b86fa46e7180198 server_cookie=c9048c9d6f489cb6 in_seq=0 out_seq=0 2024-03-11T19:09:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.657+0000 7f28a2ffd640 10 client.15294 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:09:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.658+0000 7f28a2ffd640 1 -- 172.21.15.29:0/3642934363 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5640df25a270 con 0x5640df5b8ab0 2024-03-11T19:09:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.658+0000 7f28a2ffd640 10 client.15294 handle_command_reply: tid=0 2024-03-11T19:09:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.658+0000 7f28cc9d5740 1 client.15294 shutdown 2024-03-11T19:09:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.658+0000 7f28cc9d5740 1 -- 172.21.15.29:0/3642934363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 msgr2=0x5640df5b3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.658+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/3642934363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 0x5640df5b3620 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f28bc013390 tx=0x7f28bc004ab0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.658+0000 7f28cc9d5740 1 -- 172.21.15.29:0/3642934363 shutdown_connections 2024-03-11T19:09:48.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.658+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/3642934363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 0x5640df5b3620 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.658+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/3642934363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a2970 0x5640df5ac650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.658+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/3642934363 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5640df5b8ab0 0x5640df5b7cf0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.658+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/3642934363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a1f80 0x5640df5ac110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.658+0000 7f28a2ffd640 0 client.15294 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:09:48.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.658+0000 7f28cc9d5740 1 -- 172.21.15.29:0/3642934363 >> 172.21.15.29:0/3642934363 conn(0x5640df59d4f0 msgr2=0x5640df59c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.658+0000 7f28cc9d5740 1 -- 172.21.15.29:0/3642934363 shutdown_connections 2024-03-11T19:09:48.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.658+0000 7f28cc9d5740 1 -- 172.21.15.29:0/3642934363 wait complete. 2024-03-11T19:09:48.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.658+0000 7f28cc9d5740 20 client.15294 trim_cache size 0 max 16384 2024-03-11T19:09:48.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28cc9d5740 1 Processor -- start 2024-03-11T19:09:48.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28cc9d5740 1 -- start start 2024-03-11T19:09:48.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28cc9d5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a1f80 0x5640df5c0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28cc9d5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a2970 0x5640df5ac110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28cc9d5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a7ae0 0x5640df5ac650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28cc9d5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640df48f490 con 0x5640df5a1f80 2024-03-11T19:09:48.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28cc9d5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640df5baa90 con 0x5640df5a2970 2024-03-11T19:09:48.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28cc9d5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640df59d3a0 con 0x5640df5a7ae0 2024-03-11T19:09:48.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a1f80 0x5640df5c0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a1f80 0x5640df5c0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49596/0 (socket says 172.21.15.29:49596) 2024-03-11T19:09:48.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28c37fe640 1 -- 172.21.15.29:0/3294067932 learned_addr learned my addr 172.21.15.29:0/3294067932 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28c3fff640 1 --2- 172.21.15.29:0/3294067932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a7ae0 0x5640df5ac650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28c2ffd640 1 --2- 172.21.15.29:0/3294067932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a2970 0x5640df5ac110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28c37fe640 1 -- 172.21.15.29:0/3294067932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a7ae0 msgr2=0x5640df5ac650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28c37fe640 1 --2- 172.21.15.29:0/3294067932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a7ae0 0x5640df5ac650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28c37fe640 1 -- 172.21.15.29:0/3294067932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a2970 msgr2=0x5640df5ac110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28c37fe640 1 --2- 172.21.15.29:0/3294067932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a2970 0x5640df5ac110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28c37fe640 1 -- 172.21.15.29:0/3294067932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5640df4a4a20 con 0x5640df5a1f80 2024-03-11T19:09:48.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28c37fe640 1 --2- 172.21.15.29:0/3294067932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a1f80 0x5640df5c0f50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f28b4000c00 tx=0x7f28b402c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd54532f80f78f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28a97fa640 1 -- 172.21.15.29:0/3294067932 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b404c070 con 0x5640df5a1f80 2024-03-11T19:09:48.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28a97fa640 1 -- 172.21.15.29:0/3294067932 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f28b40044f0 con 0x5640df5a1f80 2024-03-11T19:09:48.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.674+0000 7f28a97fa640 1 -- 172.21.15.29:0/3294067932 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b40484d0 con 0x5640df5a1f80 2024-03-11T19:09:48.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.675+0000 7f28cc9d5740 1 -- 172.21.15.29:0/3294067932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a1f80 msgr2=0x5640df5c0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.675+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/3294067932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a1f80 0x5640df5c0f50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f28b4000c00 tx=0x7f28b402c770 comp rx=0 tx=0).stop 2024-03-11T19:09:48.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.675+0000 7f28cc9d5740 1 -- 172.21.15.29:0/3294067932 shutdown_connections 2024-03-11T19:09:48.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.675+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/3294067932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a7ae0 0x5640df5ac650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.675+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/3294067932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a2970 0x5640df5ac110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.675+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/3294067932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a1f80 0x5640df5c0f50 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.675+0000 7f28cc9d5740 1 -- 172.21.15.29:0/3294067932 >> 172.21.15.29:0/3294067932 conn(0x5640df59d4f0 msgr2=0x5640df5a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.675+0000 7f28cc9d5740 1 -- 172.21.15.29:0/3294067932 shutdown_connections 2024-03-11T19:09:48.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.675+0000 7f28cc9d5740 1 -- 172.21.15.29:0/3294067932 wait complete. 2024-03-11T19:09:48.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.675+0000 7f28cc9d5740 1 -- start start 2024-03-11T19:09:48.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28cc9d5740 1 Processor -- start 2024-03-11T19:09:48.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28cc9d5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a1f80 0x5640df60b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28cc9d5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a2970 0x5640df60bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28cc9d5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 0x5640df612e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28cc9d5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640df6133b0 con 0x5640df5a7ae0 2024-03-11T19:09:48.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28cc9d5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640df6134f0 con 0x5640df5a1f80 2024-03-11T19:09:48.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28cc9d5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640df613660 con 0x5640df5a2970 2024-03-11T19:09:48.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28cc9d5740 10 client.0 _ll_register_callbacks cb 0x7f28b8099b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:48.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28c3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 0x5640df612e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28c3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 0x5640df612e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49606/0 (socket says 172.21.15.29:49606) 2024-03-11T19:09:48.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28c3fff640 1 -- 172.21.15.29:0/256018030 learned_addr learned my addr 172.21.15.29:0/256018030 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28c37fe640 1 --2- 172.21.15.29:0/256018030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a1f80 0x5640df60b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28c2ffd640 1 --2- 172.21.15.29:0/256018030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a2970 0x5640df60bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28c3fff640 1 -- 172.21.15.29:0/256018030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a2970 msgr2=0x5640df60bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28c3fff640 1 --2- 172.21.15.29:0/256018030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a2970 0x5640df60bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28c3fff640 1 -- 172.21.15.29:0/256018030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a1f80 msgr2=0x5640df60b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28c3fff640 1 --2- 172.21.15.29:0/256018030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a1f80 0x5640df60b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28c3fff640 1 -- 172.21.15.29:0/256018030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c4108df0 con 0x5640df5a7ae0 2024-03-11T19:09:48.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28c37fe640 1 --2- 172.21.15.29:0/256018030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a1f80 0x5640df60b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:48.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28c3fff640 1 --2- 172.21.15.29:0/256018030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 0x5640df612e70 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f28bc007610 tx=0x7f28bc012d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88940d8fbdbeb19e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:48.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28a2ffd640 1 -- 172.21.15.29:0/256018030 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28bc00e430 con 0x5640df5a7ae0 2024-03-11T19:09:48.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28a2ffd640 1 -- 172.21.15.29:0/256018030 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f28bc00e590 con 0x5640df5a7ae0 2024-03-11T19:09:48.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.676+0000 7f28cc9d5740 1 -- 172.21.15.29:0/256018030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5640df5b87c0 con 0x5640df5a7ae0 2024-03-11T19:09:48.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.677+0000 7f28a2ffd640 1 -- 172.21.15.29:0/256018030 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28bc00e750 con 0x5640df5a7ae0 2024-03-11T19:09:48.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.677+0000 7f28cc9d5740 10 client.15300 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:48.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.677+0000 7f28cc9d5740 1 -- 172.21.15.29:0/256018030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5640df4ca780 con 0x5640df5a7ae0 2024-03-11T19:09:48.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.677+0000 7f28a2ffd640 1 -- 172.21.15.29:0/256018030 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5640df5b87c0 con 0x5640df5a7ae0 2024-03-11T19:09:48.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.677+0000 7f28a2ffd640 1 -- 172.21.15.29:0/256018030 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f28bc0043b0 con 0x5640df5a7ae0 2024-03-11T19:09:48.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.677+0000 7f28cc9d5740 10 client.15300 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:48.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.677+0000 7f28cc9d5740 10 client.15300 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-11T19:09:48.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.677+0000 7f28cc9d5740 20 client.15300 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:48.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.677+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/256018030 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5640df618960 0x5640df614d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.677+0000 7f28cc9d5740 4 client.15300 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:09:48.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.677+0000 7f28cc9d5740 1 -- 172.21.15.29:0/256018030 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5640df5b9ae0 con 0x5640df618960 2024-03-11T19:09:48.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.677+0000 7f28c37fe640 1 --2- 172.21.15.29:0/256018030 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5640df618960 0x5640df614d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.678+0000 7f28c37fe640 1 --2- 172.21.15.29:0/256018030 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5640df618960 0x5640df614d80 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=18e01665dc3d06a7 server_cookie=4365c12b0d442329 in_seq=0 out_seq=0 2024-03-11T19:09:48.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.678+0000 7f28a2ffd640 10 client.15300 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:09:48.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.678+0000 7f28a2ffd640 1 -- 172.21.15.29:0/256018030 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5640df5b9ae0 con 0x5640df618960 2024-03-11T19:09:48.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.678+0000 7f28a2ffd640 10 client.15300 handle_command_reply: tid=0 2024-03-11T19:09:48.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.678+0000 7f28cc9d5740 1 client.15300 shutdown 2024-03-11T19:09:48.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.678+0000 7f28cc9d5740 1 -- 172.21.15.29:0/256018030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 msgr2=0x5640df612e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.678+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/256018030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 0x5640df612e70 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f28bc007610 tx=0x7f28bc012d30 comp rx=0 tx=0).stop 2024-03-11T19:09:48.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.678+0000 7f28cc9d5740 1 -- 172.21.15.29:0/256018030 shutdown_connections 2024-03-11T19:09:48.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.678+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/256018030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5640df5a7ae0 0x5640df612e70 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.678+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/256018030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5640df5a2970 0x5640df60bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.678+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/256018030 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5640df618960 0x5640df614d80 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.678+0000 7f28cc9d5740 1 --2- 172.21.15.29:0/256018030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640df5a1f80 0x5640df60b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.678+0000 7f28a2ffd640 0 client.15300 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:09:48.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.678+0000 7f28cc9d5740 1 -- 172.21.15.29:0/256018030 >> 172.21.15.29:0/256018030 conn(0x5640df59d4f0 msgr2=0x5640df5c0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.678+0000 7f28cc9d5740 1 -- 172.21.15.29:0/256018030 shutdown_connections 2024-03-11T19:09:48.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.678+0000 7f28cc9d5740 1 -- 172.21.15.29:0/256018030 wait complete. 2024-03-11T19:09:48.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.678+0000 7f28cc9d5740 20 client.15300 trim_cache size 0 max 16384 2024-03-11T19:09:48.689 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:09:48.689 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "638915c2-dfd9-11ee-95c8-87774f69a715", 2024-03-11T19:09:48.690 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 1, 2024-03-11T19:09:48.690 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24496, 2024-03-11T19:09:48.690 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-03-11T19:09:48.690 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-03-11T19:09:48.690 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-03-11T19:09:48.690 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 115.922881115, 2024-03-11T19:09:48.690 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 26, 2024-03-11T19:09:48.690 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 89, 2024-03-11T19:09:48.690 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 87, 2024-03-11T19:09:48.690 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 131.83972849899999 2024-03-11T19:09:48.690 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:09:48.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.679+0000 7f28c9d25640 1 -- 172.21.15.29:0/211191535 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f28ac0a1a30 msgr2=0x7f28ac0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.679+0000 7f28c9d25640 1 --2- 172.21.15.29:0/211191535 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f28ac0a1a30 0x7f28ac0a3ef0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x5640df5a31d0 tx=0x7f28bc004a40 comp rx=0 tx=0).stop 2024-03-11T19:09:48.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.679+0000 7f28c9d25640 1 -- 172.21.15.29:0/211191535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c4100cc0 msgr2=0x7f28c419be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.679+0000 7f28c9d25640 1 --2- 172.21.15.29:0/211191535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c4100cc0 0x7f28c419be30 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f28b800d770 tx=0x7f28b800dc30 comp rx=0 tx=0).stop 2024-03-11T19:09:48.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.679+0000 7f28c9d25640 1 -- 172.21.15.29:0/211191535 shutdown_connections 2024-03-11T19:09:48.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.679+0000 7f28c9d25640 1 --2- 172.21.15.29:0/211191535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28c41016b0 0x7f28c419e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.679+0000 7f28c9d25640 1 --2- 172.21.15.29:0/211191535 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f28ac0a1a30 0x7f28ac0a3ef0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.679+0000 7f28c9d25640 1 --2- 172.21.15.29:0/211191535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c4100cc0 0x7f28c419be30 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.679+0000 7f28c9d25640 1 --2- 172.21.15.29:0/211191535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28c4100300 0x7f28c40ffdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.679+0000 7f28c9d25640 1 -- 172.21.15.29:0/211191535 >> 172.21.15.29:0/211191535 conn(0x7f28c40fad70 msgr2=0x7f28c41046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.679+0000 7f28c9d25640 1 -- 172.21.15.29:0/211191535 shutdown_connections 2024-03-11T19:09:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.679+0000 7f28c9d25640 1 -- 172.21.15.29:0/211191535 wait complete. 2024-03-11T19:09:48.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-11T19:09:48.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.794+0000 7f9f70281640 1 -- 172.21.15.29:0/664109848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f68101270 msgr2=0x7f9f681016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.794+0000 7f9f70281640 1 --2- 172.21.15.29:0/664109848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f68101270 0x7f9f681016f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9f58009900 tx=0x7f9f5802bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:48.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.794+0000 7f9f70281640 1 -- 172.21.15.29:0/664109848 shutdown_connections 2024-03-11T19:09:48.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.794+0000 7f9f70281640 1 --2- 172.21.15.29:0/664109848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f68101c30 0x7f9f681084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.794+0000 7f9f70281640 1 --2- 172.21.15.29:0/664109848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f68101270 0x7f9f681016f0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.794+0000 7f9f70281640 1 --2- 172.21.15.29:0/664109848 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f68100080 0x7f9f681004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.794+0000 7f9f70281640 1 -- 172.21.15.29:0/664109848 >> 172.21.15.29:0/664109848 conn(0x7f9f680fb610 msgr2=0x7f9f680fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.794+0000 7f9f70281640 1 -- 172.21.15.29:0/664109848 shutdown_connections 2024-03-11T19:09:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.794+0000 7f9f70281640 1 -- 172.21.15.29:0/664109848 wait complete. 2024-03-11T19:09:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.794+0000 7f9f70281640 1 Processor -- start 2024-03-11T19:09:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f70281640 1 -- start start 2024-03-11T19:09:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f70281640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f68100080 0x7f9f681971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f70281640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f68101270 0x7f9f681976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f70281640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f68101c30 0x7f9f6819e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f70281640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6819ec90 con 0x7f9f68101270 2024-03-11T19:09:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f70281640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6819ee00 con 0x7f9f68101c30 2024-03-11T19:09:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f70281640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6819efc0 con 0x7f9f68100080 2024-03-11T19:09:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f6d81c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f68101270 0x7f9f681976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f6d81c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f68101270 0x7f9f681976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49616/0 (socket says 172.21.15.29:49616) 2024-03-11T19:09:48.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f6d81c640 1 -- 172.21.15.29:0/2992173638 learned_addr learned my addr 172.21.15.29:0/2992173638 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f6e01d640 1 --2- 172.21.15.29:0/2992173638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f68100080 0x7f9f681971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f6e81e640 1 --2- 172.21.15.29:0/2992173638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f68101c30 0x7f9f6819e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f6d81c640 1 -- 172.21.15.29:0/2992173638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f68100080 msgr2=0x7f9f681971a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f6d81c640 1 --2- 172.21.15.29:0/2992173638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f68100080 0x7f9f681971a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f6d81c640 1 -- 172.21.15.29:0/2992173638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f68101c30 msgr2=0x7f9f6819e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f6d81c640 1 --2- 172.21.15.29:0/2992173638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f68101c30 0x7f9f6819e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f6d81c640 1 -- 172.21.15.29:0/2992173638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f58009560 con 0x7f9f68101270 2024-03-11T19:09:48.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f6e81e640 1 --2- 172.21.15.29:0/2992173638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f68101c30 0x7f9f6819e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:48.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f6e01d640 1 --2- 172.21.15.29:0/2992173638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f68100080 0x7f9f681971a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:48.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f6d81c640 1 --2- 172.21.15.29:0/2992173638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f68101270 0x7f9f681976e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9f58036040 tx=0x7f9f58004140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f5f7fe640 1 -- 172.21.15.29:0/2992173638 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f580035f0 con 0x7f9f68101270 2024-03-11T19:09:48.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f70281640 1 -- 172.21.15.29:0/2992173638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f6819f240 con 0x7f9f68101270 2024-03-11T19:09:48.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.795+0000 7f9f5f7fe640 1 -- 172.21.15.29:0/2992173638 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9f5802de90 con 0x7f9f68101270 2024-03-11T19:09:48.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.796+0000 7f9f5f7fe640 1 -- 172.21.15.29:0/2992173638 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f58005490 con 0x7f9f68101270 2024-03-11T19:09:48.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.796+0000 7f9f70281640 1 -- 172.21.15.29:0/2992173638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6819f750 con 0x7f9f68101270 2024-03-11T19:09:48.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.797+0000 7f9f5f7fe640 1 -- 172.21.15.29:0/2992173638 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9f58003750 con 0x7f9f68101270 2024-03-11T19:09:48.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.798+0000 7f9f72e51740 1 Processor -- start 2024-03-11T19:09:48.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.798+0000 7f9f72e51740 1 -- start start 2024-03-11T19:09:48.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.798+0000 7f9f5f7fe640 1 --2- 172.21.15.29:0/2992173638 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9f540a1a30 0x7f9f540a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.798+0000 7f9f72e51740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c26f80 0x55ba09c273a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.798+0000 7f9f72e51740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 0x55ba09c2c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.798+0000 7f9f72e51740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c2cae0 0x55ba09c2ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.798+0000 7f9f72e51740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba09be2cd0 con 0x55ba09c27970 2024-03-11T19:09:48.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.798+0000 7f9f5f7fe640 1 -- 172.21.15.29:0/2992173638 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9f580ef700 con 0x7f9f68101270 2024-03-11T19:09:48.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f6e81e640 1 --2- 172.21.15.29:0/2992173638 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9f540a1a30 0x7f9f540a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f72e51740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba09b95f20 con 0x55ba09c26f80 2024-03-11T19:09:48.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f72e51740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba09b93450 con 0x55ba09c2cae0 2024-03-11T19:09:48.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f6d81c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 0x55ba09c2c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f6d81c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 0x55ba09c2c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49628/0 (socket says 172.21.15.29:49628) 2024-03-11T19:09:48.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f6d81c640 1 -- 172.21.15.29:0/825477914 learned_addr learned my addr 172.21.15.29:0/825477914 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f6e01d640 1 --2- 172.21.15.29:0/825477914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c26f80 0x55ba09c273a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f6d81c640 1 -- 172.21.15.29:0/825477914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c2cae0 msgr2=0x55ba09c2ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f6e81e640 1 --2- 172.21.15.29:0/825477914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c2cae0 0x55ba09c2ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f6d81c640 1 --2- 172.21.15.29:0/825477914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c2cae0 0x55ba09c2ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f6d81c640 1 -- 172.21.15.29:0/825477914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c26f80 msgr2=0x55ba09c273a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f6d81c640 1 --2- 172.21.15.29:0/825477914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c26f80 0x55ba09c273a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f6d81c640 1 -- 172.21.15.29:0/825477914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba09b43f20 con 0x55ba09c27970 2024-03-11T19:09:48.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f6e01d640 1 --2- 172.21.15.29:0/825477914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c26f80 0x55ba09c273a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:48.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f6e81e640 1 --2- 172.21.15.29:0/825477914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c2cae0 0x55ba09c2ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:48.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f6e81e640 1 --2- 172.21.15.29:0/2992173638 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9f540a1a30 0x7f9f540a3ef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9f60005fd0 tx=0x7f9f600106e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f6d81c640 1 --2- 172.21.15.29:0/825477914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 0x55ba09c2c470 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f9f580b83f0 tx=0x7f9f58000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eb37ac2f7b408fb server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f5cbfe640 1 -- 172.21.15.29:0/825477914 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f580bc710 con 0x55ba09c27970 2024-03-11T19:09:48.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f5cbfe640 1 -- 172.21.15.29:0/825477914 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9f580c0860 con 0x55ba09c27970 2024-03-11T19:09:48.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.799+0000 7f9f5cbfe640 1 -- 172.21.15.29:0/825477914 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f580bc970 con 0x55ba09c27970 2024-03-11T19:09:48.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.800+0000 7f9f72e51740 1 -- 172.21.15.29:0/825477914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 msgr2=0x55ba09c2c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.800+0000 7f9f72e51740 1 --2- 172.21.15.29:0/825477914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 0x55ba09c2c470 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f9f580b83f0 tx=0x7f9f58000bc0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.800+0000 7f9f72e51740 1 -- 172.21.15.29:0/825477914 shutdown_connections 2024-03-11T19:09:48.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.800+0000 7f9f72e51740 1 --2- 172.21.15.29:0/825477914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c2cae0 0x55ba09c2ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.800+0000 7f9f72e51740 1 --2- 172.21.15.29:0/825477914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 0x55ba09c2c470 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.800+0000 7f9f72e51740 1 --2- 172.21.15.29:0/825477914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c26f80 0x55ba09c273a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.800+0000 7f9f72e51740 1 -- 172.21.15.29:0/825477914 >> 172.21.15.29:0/825477914 conn(0x55ba09c224f0 msgr2=0x55ba09c228f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.800+0000 7f9f72e51740 1 -- 172.21.15.29:0/825477914 shutdown_connections 2024-03-11T19:09:48.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.800+0000 7f9f72e51740 1 -- 172.21.15.29:0/825477914 wait complete. 2024-03-11T19:09:48.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.800+0000 7f9f72e51740 1 -- start start 2024-03-11T19:09:48.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f72e51740 1 Processor -- start 2024-03-11T19:09:48.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f72e51740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c26f80 0x55ba09c30e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f72e51740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c27970 0x55ba09c313b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f72e51740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c2cae0 0x55ba09c383f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f72e51740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba09b8ec30 con 0x55ba09c26f80 2024-03-11T19:09:48.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f72e51740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba09b33d30 con 0x55ba09c2cae0 2024-03-11T19:09:48.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f72e51740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba09862310 con 0x55ba09c27970 2024-03-11T19:09:48.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f72e51740 10 client.0 _ll_register_callbacks cb 0x55ba09b3bd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:48.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f6e01d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c26f80 0x55ba09c30e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f6e01d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c26f80 0x55ba09c30e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49642/0 (socket says 172.21.15.29:49642) 2024-03-11T19:09:48.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f6e01d640 1 -- 172.21.15.29:0/3568351786 learned_addr learned my addr 172.21.15.29:0/3568351786 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f6d81c640 1 --2- 172.21.15.29:0/3568351786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c27970 0x55ba09c313b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f6e81e640 1 --2- 172.21.15.29:0/3568351786 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c2cae0 0x55ba09c383f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f6e01d640 1 -- 172.21.15.29:0/3568351786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c27970 msgr2=0x55ba09c313b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f6e01d640 1 --2- 172.21.15.29:0/3568351786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c27970 0x55ba09c313b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f6e01d640 1 -- 172.21.15.29:0/3568351786 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c2cae0 msgr2=0x55ba09c383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f6e01d640 1 --2- 172.21.15.29:0/3568351786 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c2cae0 0x55ba09c383f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f6e01d640 1 -- 172.21.15.29:0/3568351786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6819f750 con 0x55ba09c26f80 2024-03-11T19:09:48.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f6d81c640 1 --2- 172.21.15.29:0/3568351786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c27970 0x55ba09c313b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:48.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f6e01d640 1 --2- 172.21.15.29:0/3568351786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c26f80 0x55ba09c30e70 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9f64012780 tx=0x7f9f64012c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2302887e6acb767 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f51ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:48.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f51ffb640 1 -- 172.21.15.29:0/3568351786 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f64010d60 con 0x55ba09c26f80 2024-03-11T19:09:48.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.801+0000 7f9f51ffb640 1 -- 172.21.15.29:0/3568351786 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9f64004020 con 0x55ba09c26f80 2024-03-11T19:09:48.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.802+0000 7f9f72e51740 1 -- 172.21.15.29:0/3568351786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ba0988e6c0 con 0x55ba09c26f80 2024-03-11T19:09:48.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.802+0000 7f9f51ffb640 1 -- 172.21.15.29:0/3568351786 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f64010ec0 con 0x55ba09c26f80 2024-03-11T19:09:48.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.802+0000 7f9f72e51740 10 client.15312 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:48.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.802+0000 7f9f72e51740 1 -- 172.21.15.29:0/3568351786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55ba09b8d2b0 con 0x55ba09c26f80 2024-03-11T19:09:48.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.802+0000 7f9f51ffb640 1 -- 172.21.15.29:0/3568351786 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55ba0988e6c0 con 0x55ba09c26f80 2024-03-11T19:09:48.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.802+0000 7f9f51ffb640 1 -- 172.21.15.29:0/3568351786 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9f64025460 con 0x55ba09c26f80 2024-03-11T19:09:48.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.802+0000 7f9f72e51740 10 client.15312 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:48.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.802+0000 7f9f72e51740 10 client.15312 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-11T19:09:48.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.802+0000 7f9f72e51740 20 client.15312 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:48.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.802+0000 7f9f72e51740 1 --2- 172.21.15.29:0/3568351786 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ba09c3d7e0 0x55ba09c3ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.802+0000 7f9f72e51740 4 client.15312 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:09:48.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.802+0000 7f9f72e51740 1 -- 172.21.15.29:0/3568351786 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ba098df270 con 0x55ba09c3d7e0 2024-03-11T19:09:48.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.803+0000 7f9f6d81c640 1 --2- 172.21.15.29:0/3568351786 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ba09c3d7e0 0x55ba09c3ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.803+0000 7f9f51ffb640 10 client.15312 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:09:48.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.803+0000 7f9f6d81c640 1 --2- 172.21.15.29:0/3568351786 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ba09c3d7e0 0x55ba09c3ca40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e1b5578d0a5592a server_cookie=f005cef65a99bf18 in_seq=0 out_seq=0 2024-03-11T19:09:48.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.804+0000 7f9f51ffb640 1 -- 172.21.15.29:0/3568351786 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ba098df270 con 0x55ba09c3d7e0 2024-03-11T19:09:48.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.804+0000 7f9f51ffb640 10 client.15312 handle_command_reply: tid=0 2024-03-11T19:09:48.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.804+0000 7f9f72e51740 1 client.15312 shutdown 2024-03-11T19:09:48.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.804+0000 7f9f72e51740 1 -- 172.21.15.29:0/3568351786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c26f80 msgr2=0x55ba09c30e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.804+0000 7f9f72e51740 1 --2- 172.21.15.29:0/3568351786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c26f80 0x55ba09c30e70 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9f64012780 tx=0x7f9f64012c40 comp rx=0 tx=0).stop 2024-03-11T19:09:48.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.804+0000 7f9f72e51740 1 -- 172.21.15.29:0/3568351786 shutdown_connections 2024-03-11T19:09:48.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.804+0000 7f9f72e51740 1 --2- 172.21.15.29:0/3568351786 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c2cae0 0x55ba09c383f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.804+0000 7f9f72e51740 1 --2- 172.21.15.29:0/3568351786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c27970 0x55ba09c313b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.804+0000 7f9f72e51740 1 --2- 172.21.15.29:0/3568351786 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ba09c3d7e0 0x55ba09c3ca40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.804+0000 7f9f72e51740 1 --2- 172.21.15.29:0/3568351786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c26f80 0x55ba09c30e70 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.804+0000 7f9f51ffb640 0 client.15312 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:09:48.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.804+0000 7f9f72e51740 1 -- 172.21.15.29:0/3568351786 >> 172.21.15.29:0/3568351786 conn(0x55ba09c224f0 msgr2=0x55ba09c217d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.804+0000 7f9f72e51740 1 -- 172.21.15.29:0/3568351786 shutdown_connections 2024-03-11T19:09:48.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.804+0000 7f9f72e51740 1 -- 172.21.15.29:0/3568351786 wait complete. 2024-03-11T19:09:48.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.804+0000 7f9f72e51740 20 client.15312 trim_cache size 0 max 16384 2024-03-11T19:09:48.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.819+0000 7f9f72e51740 1 Processor -- start 2024-03-11T19:09:48.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.819+0000 7f9f72e51740 1 -- start start 2024-03-11T19:09:48.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.820+0000 7f9f72e51740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c26f80 0x55ba09c28d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.820+0000 7f9f72e51740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 0x55ba09c292d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.820+0000 7f9f72e51740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c2cae0 0x55ba09c38310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.820+0000 7f9f72e51740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba09b14490 con 0x55ba09c27970 2024-03-11T19:09:48.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.820+0000 7f9f72e51740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba09c3f7c0 con 0x55ba09c26f80 2024-03-11T19:09:48.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.820+0000 7f9f72e51740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba0988e6c0 con 0x55ba09c2cae0 2024-03-11T19:09:48.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.820+0000 7f9f6d81c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 0x55ba09c292d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.820+0000 7f9f6d81c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 0x55ba09c292d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49644/0 (socket says 172.21.15.29:49644) 2024-03-11T19:09:48.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.820+0000 7f9f6d81c640 1 -- 172.21.15.29:0/2025787841 learned_addr learned my addr 172.21.15.29:0/2025787841 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.820+0000 7f9f6e81e640 1 --2- 172.21.15.29:0/2025787841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c2cae0 0x55ba09c38310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.820+0000 7f9f6d81c640 1 -- 172.21.15.29:0/2025787841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c2cae0 msgr2=0x55ba09c38310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.820+0000 7f9f6d81c640 1 --2- 172.21.15.29:0/2025787841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c2cae0 0x55ba09c38310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.820+0000 7f9f6d81c640 1 -- 172.21.15.29:0/2025787841 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c26f80 msgr2=0x55ba09c28d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.820+0000 7f9f6d81c640 1 --2- 172.21.15.29:0/2025787841 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c26f80 0x55ba09c28d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.820+0000 7f9f6d81c640 1 -- 172.21.15.29:0/2025787841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba09b29a20 con 0x55ba09c27970 2024-03-11T19:09:48.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.820+0000 7f9f6d81c640 1 --2- 172.21.15.29:0/2025787841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 0x55ba09c292d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9f58005940 tx=0x7f9f580a7040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6726d3a6af8913ae server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.820+0000 7f9f53fff640 1 -- 172.21.15.29:0/2025787841 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f580055f0 con 0x55ba09c27970 2024-03-11T19:09:48.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.820+0000 7f9f53fff640 1 -- 172.21.15.29:0/2025787841 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9f580d4070 con 0x55ba09c27970 2024-03-11T19:09:48.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.820+0000 7f9f53fff640 1 -- 172.21.15.29:0/2025787841 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f580b8d80 con 0x55ba09c27970 2024-03-11T19:09:48.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.821+0000 7f9f72e51740 1 -- 172.21.15.29:0/2025787841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 msgr2=0x55ba09c292d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.821+0000 7f9f72e51740 1 --2- 172.21.15.29:0/2025787841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 0x55ba09c292d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9f58005940 tx=0x7f9f580a7040 comp rx=0 tx=0).stop 2024-03-11T19:09:48.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.821+0000 7f9f72e51740 1 -- 172.21.15.29:0/2025787841 shutdown_connections 2024-03-11T19:09:48.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.821+0000 7f9f72e51740 1 --2- 172.21.15.29:0/2025787841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c2cae0 0x55ba09c38310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.821+0000 7f9f72e51740 1 --2- 172.21.15.29:0/2025787841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 0x55ba09c292d0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.821+0000 7f9f72e51740 1 --2- 172.21.15.29:0/2025787841 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c26f80 0x55ba09c28d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.821+0000 7f9f72e51740 1 -- 172.21.15.29:0/2025787841 >> 172.21.15.29:0/2025787841 conn(0x55ba09c224f0 msgr2=0x55ba09c228d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.821+0000 7f9f72e51740 1 -- 172.21.15.29:0/2025787841 shutdown_connections 2024-03-11T19:09:48.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.821+0000 7f9f72e51740 1 -- 172.21.15.29:0/2025787841 wait complete. 2024-03-11T19:09:48.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.821+0000 7f9f72e51740 1 -- start start 2024-03-11T19:09:48.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.821+0000 7f9f72e51740 1 Processor -- start 2024-03-11T19:09:48.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f72e51740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c26f80 0x55ba09c35100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f72e51740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 0x55ba09c35640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f72e51740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c2cae0 0x55ba09c933e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f72e51740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba09c38660 con 0x55ba09c27970 2024-03-11T19:09:48.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f72e51740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba09c387d0 con 0x55ba09c26f80 2024-03-11T19:09:48.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f72e51740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba09c93920 con 0x55ba09c2cae0 2024-03-11T19:09:48.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f72e51740 10 client.0 _ll_register_callbacks cb 0x7f9f580b89b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:09:48.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f6d81c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 0x55ba09c35640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f6d81c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 0x55ba09c35640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49648/0 (socket says 172.21.15.29:49648) 2024-03-11T19:09:48.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f6d81c640 1 -- 172.21.15.29:0/3691763260 learned_addr learned my addr 172.21.15.29:0/3691763260 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:48.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f6e01d640 1 --2- 172.21.15.29:0/3691763260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c26f80 0x55ba09c35100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f6e81e640 1 --2- 172.21.15.29:0/3691763260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c2cae0 0x55ba09c933e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f6d81c640 1 -- 172.21.15.29:0/3691763260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c2cae0 msgr2=0x55ba09c933e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f6d81c640 1 --2- 172.21.15.29:0/3691763260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c2cae0 0x55ba09c933e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f6d81c640 1 -- 172.21.15.29:0/3691763260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c26f80 msgr2=0x55ba09c35100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:09:48.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f6d81c640 1 --2- 172.21.15.29:0/3691763260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c26f80 0x55ba09c35100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f6d81c640 1 -- 172.21.15.29:0/3691763260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ba09b43f20 con 0x55ba09c27970 2024-03-11T19:09:48.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f6e01d640 1 --2- 172.21.15.29:0/3691763260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c26f80 0x55ba09c35100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:48.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f6d81c640 1 --2- 172.21.15.29:0/3691763260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 0x55ba09c35640 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9f580bc410 tx=0x7f9f580baa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d82a9c614a5a86e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:48.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f51ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:09:48.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f51ffb640 1 -- 172.21.15.29:0/3691763260 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f580d5170 con 0x55ba09c27970 2024-03-11T19:09:48.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f51ffb640 1 -- 172.21.15.29:0/3691763260 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9f58087e30 con 0x55ba09c27970 2024-03-11T19:09:48.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f72e51740 1 -- 172.21.15.29:0/3691763260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ba09c3e810 con 0x55ba09c27970 2024-03-11T19:09:48.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f51ffb640 1 -- 172.21.15.29:0/3691763260 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f580b8d80 con 0x55ba09c27970 2024-03-11T19:09:48.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f72e51740 10 client.15318 fetch_fsmap learned FSMap version 26 2024-03-11T19:09:48.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.822+0000 7f9f72e51740 1 -- 172.21.15.29:0/3691763260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55ba09b4f780 con 0x55ba09c27970 2024-03-11T19:09:48.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.823+0000 7f9f51ffb640 1 -- 172.21.15.29:0/3691763260 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55ba09c3e810 con 0x55ba09c27970 2024-03-11T19:09:48.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.823+0000 7f9f51ffb640 1 -- 172.21.15.29:0/3691763260 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9f580a4080 con 0x55ba09c27970 2024-03-11T19:09:48.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.823+0000 7f9f72e51740 10 client.15318 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T19:09:48.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.823+0000 7f9f72e51740 10 client.15318 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-11T19:09:48.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.823+0000 7f9f72e51740 20 client.15318 populate_metadata read hostname 'smithi029' 2024-03-11T19:09:48.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.823+0000 7f9f72e51740 1 --2- 172.21.15.29:0/3691763260 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ba09c986d0 0x55ba09c94bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:48.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.823+0000 7f9f72e51740 4 client.15318 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:09:48.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.823+0000 7f9f72e51740 1 -- 172.21.15.29:0/3691763260 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f9f580c0860 con 0x55ba09c986d0 2024-03-11T19:09:48.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.823+0000 7f9f6e01d640 1 --2- 172.21.15.29:0/3691763260 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ba09c986d0 0x55ba09c94bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:48.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.823+0000 7f9f51ffb640 10 client.15318 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:09:48.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.823+0000 7f9f6e01d640 1 --2- 172.21.15.29:0/3691763260 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ba09c986d0 0x55ba09c94bb0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81add69af11368b8 server_cookie=c77e8b406bd99f44 in_seq=0 out_seq=0 2024-03-11T19:09:48.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f51ffb640 1 -- 172.21.15.29:0/3691763260 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x7f9f580c0860 con 0x55ba09c986d0 2024-03-11T19:09:48.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f51ffb640 10 client.15318 handle_command_reply: tid=0 2024-03-11T19:09:48.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f72e51740 1 client.15318 shutdown 2024-03-11T19:09:48.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f72e51740 1 -- 172.21.15.29:0/3691763260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 msgr2=0x55ba09c35640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f72e51740 1 --2- 172.21.15.29:0/3691763260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 0x55ba09c35640 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9f580bc410 tx=0x7f9f580baa50 comp rx=0 tx=0).stop 2024-03-11T19:09:48.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f72e51740 1 -- 172.21.15.29:0/3691763260 shutdown_connections 2024-03-11T19:09:48.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f72e51740 1 --2- 172.21.15.29:0/3691763260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba09c2cae0 0x55ba09c933e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f72e51740 1 --2- 172.21.15.29:0/3691763260 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ba09c986d0 0x55ba09c94bb0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f72e51740 1 --2- 172.21.15.29:0/3691763260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba09c27970 0x55ba09c35640 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f72e51740 1 --2- 172.21.15.29:0/3691763260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba09c26f80 0x55ba09c35100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f51ffb640 0 client.15318 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:09:48.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f72e51740 1 -- 172.21.15.29:0/3691763260 >> 172.21.15.29:0/3691763260 conn(0x55ba09c224f0 msgr2=0x55ba09c28d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f72e51740 1 -- 172.21.15.29:0/3691763260 shutdown_connections 2024-03-11T19:09:48.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f72e51740 1 -- 172.21.15.29:0/3691763260 wait complete. 2024-03-11T19:09:48.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f72e51740 20 client.15318 trim_cache size 0 max 16384 2024-03-11T19:09:48.834 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:09:48.834 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "638915c2-dfd9-11ee-95c8-87774f69a715", 2024-03-11T19:09:48.834 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2024-03-11T19:09:48.834 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24506, 2024-03-11T19:09:48.834 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-03-11T19:09:48.834 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-03-11T19:09:48.834 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-03-11T19:09:48.834 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 119.089712353, 2024-03-11T19:09:48.834 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 26, 2024-03-11T19:09:48.835 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 89, 2024-03-11T19:09:48.835 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 87, 2024-03-11T19:09:48.835 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 123.24344413 2024-03-11T19:09:48.835 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:09:48.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f70281640 1 -- 172.21.15.29:0/2992173638 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9f540a1a30 msgr2=0x7f9f540a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f70281640 1 --2- 172.21.15.29:0/2992173638 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9f540a1a30 0x7f9f540a3ef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9f60005fd0 tx=0x7f9f600106e0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f70281640 1 -- 172.21.15.29:0/2992173638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f68101270 msgr2=0x7f9f681976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:48.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f70281640 1 --2- 172.21.15.29:0/2992173638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f68101270 0x7f9f681976e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9f58036040 tx=0x7f9f58004140 comp rx=0 tx=0).stop 2024-03-11T19:09:48.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f70281640 1 -- 172.21.15.29:0/2992173638 shutdown_connections 2024-03-11T19:09:48.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f70281640 1 --2- 172.21.15.29:0/2992173638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f68101c30 0x7f9f6819e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f70281640 1 --2- 172.21.15.29:0/2992173638 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9f540a1a30 0x7f9f540a3ef0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f70281640 1 --2- 172.21.15.29:0/2992173638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f68101270 0x7f9f681976e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f70281640 1 --2- 172.21.15.29:0/2992173638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f68100080 0x7f9f681971a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:48.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f70281640 1 -- 172.21.15.29:0/2992173638 >> 172.21.15.29:0/2992173638 conn(0x7f9f680fb610 msgr2=0x7f9f68104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:48.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f70281640 1 -- 172.21.15.29:0/2992173638 shutdown_connections 2024-03-11T19:09:48.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:48.824+0000 7f9f70281640 1 -- 172.21.15.29:0/2992173638 wait complete. 2024-03-11T19:09:48.838 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-11T19:09:48.862 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-11T19:09:48.862 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-03-11T19:09:48.862 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-03-11T19:09:48.863 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-11T19:09:48.863 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-11T19:09:48.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:09:48.863 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-03-11T19:09:48.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:09:48.878 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2024-03-11T19:09:48.947 INFO:teuthology.orchestra.run.smithi029.stderr:Cannot find device "ceph-brx" 2024-03-11T19:09:48.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T19:09:48.949 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-11T19:09:48.949 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph fs ls 2024-03-11T19:09:49.232 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:09:49.354 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:49 smithi029 ceph-mon[32439]: pgmap v351: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:09:49.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:49 smithi190 ceph-mon[36292]: pgmap v351: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:09:49.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:49 smithi170 ceph-mon[35690]: pgmap v351: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:09:51.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.238+0000 7f48f28b7700 1 -- 172.21.15.29:0/2394317836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48ec106ff0 msgr2=0x7f48ec10eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:51.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.238+0000 7f48f28b7700 1 --2- 172.21.15.29:0/2394317836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48ec106ff0 0x7f48ec10eb70 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f48e400b0a0 tx=0x7f48e400b3b0 comp rx=0 tx=0).stop 2024-03-11T19:09:51.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.238+0000 7f48f28b7700 1 -- 172.21.15.29:0/2394317836 shutdown_connections 2024-03-11T19:09:51.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.238+0000 7f48f28b7700 1 --2- 172.21.15.29:0/2394317836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48ec106ff0 0x7f48ec10eb70 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:51.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.238+0000 7f48f28b7700 1 --2- 172.21.15.29:0/2394317836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48ec06a410 0x7f48ec06a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:51.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.238+0000 7f48f28b7700 1 --2- 172.21.15.29:0/2394317836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48ec06adc0 0x7f48ec1068c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:51.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.238+0000 7f48f28b7700 1 -- 172.21.15.29:0/2394317836 >> 172.21.15.29:0/2394317836 conn(0x7f48ec076bc0 msgr2=0x7f48ec076fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:51.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.238+0000 7f48f28b7700 1 -- 172.21.15.29:0/2394317836 shutdown_connections 2024-03-11T19:09:51.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.238+0000 7f48f28b7700 1 -- 172.21.15.29:0/2394317836 wait complete. 2024-03-11T19:09:51.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.239+0000 7f48f28b7700 1 Processor -- start 2024-03-11T19:09:51.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.239+0000 7f48f28b7700 1 -- start start 2024-03-11T19:09:51.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.239+0000 7f48f28b7700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48ec06a410 0x7f48ec1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:51.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.239+0000 7f48f28b7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48ec06adc0 0x7f48ec1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:51.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.239+0000 7f48f28b7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48ec106ff0 0x7f48ec19ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:51.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.239+0000 7f48f28b7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ec19d040 con 0x7f48ec06adc0 2024-03-11T19:09:51.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.239+0000 7f48f28b7700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ec19d1b0 con 0x7f48ec06a410 2024-03-11T19:09:51.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.239+0000 7f48f28b7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ec19d370 con 0x7f48ec106ff0 2024-03-11T19:09:51.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.239+0000 7f48eb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48ec06adc0 0x7f48ec1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:51.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.239+0000 7f48eb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48ec06adc0 0x7f48ec1986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49654/0 (socket says 172.21.15.29:49654) 2024-03-11T19:09:51.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.239+0000 7f48eb7fe700 1 -- 172.21.15.29:0/2903487032 learned_addr learned my addr 172.21.15.29:0/2903487032 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:51.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.240+0000 7f48f0e54700 1 --2- 172.21.15.29:0/2903487032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48ec106ff0 0x7f48ec19ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:51.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.240+0000 7f48ebfff700 1 --2- 172.21.15.29:0/2903487032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48ec06a410 0x7f48ec1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:51.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.240+0000 7f48eb7fe700 1 -- 172.21.15.29:0/2903487032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48ec106ff0 msgr2=0x7f48ec19ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:51.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.240+0000 7f48eb7fe700 1 --2- 172.21.15.29:0/2903487032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48ec106ff0 0x7f48ec19ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:51.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.240+0000 7f48eb7fe700 1 -- 172.21.15.29:0/2903487032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48ec06a410 msgr2=0x7f48ec1981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:51.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.240+0000 7f48eb7fe700 1 --2- 172.21.15.29:0/2903487032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48ec06a410 0x7f48ec1981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:51.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.240+0000 7f48eb7fe700 1 -- 172.21.15.29:0/2903487032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e4009d00 con 0x7f48ec06adc0 2024-03-11T19:09:51.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.240+0000 7f48eb7fe700 1 --2- 172.21.15.29:0/2903487032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48ec06adc0 0x7f48ec1986f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f48e000d950 tx=0x7f48e000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:51.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.240+0000 7f48e97fa700 1 -- 172.21.15.29:0/2903487032 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e000f890 con 0x7f48ec06adc0 2024-03-11T19:09:51.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.240+0000 7f48f28b7700 1 -- 172.21.15.29:0/2903487032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48ec19d650 con 0x7f48ec06adc0 2024-03-11T19:09:51.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.240+0000 7f48e97fa700 1 -- 172.21.15.29:0/2903487032 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f48e0010460 con 0x7f48ec06adc0 2024-03-11T19:09:51.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.240+0000 7f48e97fa700 1 -- 172.21.15.29:0/2903487032 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e000fea0 con 0x7f48ec06adc0 2024-03-11T19:09:51.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.240+0000 7f48f28b7700 1 -- 172.21.15.29:0/2903487032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48ec1a4eb0 con 0x7f48ec06adc0 2024-03-11T19:09:51.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.241+0000 7f48f28b7700 1 -- 172.21.15.29:0/2903487032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48ec10a1e0 con 0x7f48ec06adc0 2024-03-11T19:09:51.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.242+0000 7f48e97fa700 1 -- 172.21.15.29:0/2903487032 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f48e00105d0 con 0x7f48ec06adc0 2024-03-11T19:09:51.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.243+0000 7f48e97fa700 1 --2- 172.21.15.29:0/2903487032 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f48d40a1ac0 0x7f48d40a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:51.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.243+0000 7f48ebfff700 1 --2- 172.21.15.29:0/2903487032 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f48d40a1ac0 0x7f48d40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:51.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.243+0000 7f48e97fa700 1 -- 172.21.15.29:0/2903487032 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f48e00cce50 con 0x7f48ec06adc0 2024-03-11T19:09:51.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.244+0000 7f48ebfff700 1 --2- 172.21.15.29:0/2903487032 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f48d40a1ac0 0x7f48d40a3f80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f48dc00ac50 tx=0x7f48dc00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:51.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.247+0000 7f48e97fa700 1 -- 172.21.15.29:0/2903487032 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f48e0096060 con 0x7f48ec06adc0 2024-03-11T19:09:51.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:51 smithi029 ceph-mon[32439]: pgmap v352: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.499+0000 7f48f28b7700 1 -- 172.21.15.29:0/2903487032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f48ec068930 con 0x7f48ec06adc0 2024-03-11T19:09:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.499+0000 7f48e97fa700 1 -- 172.21.15.29:0/2903487032 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) v1 ==== 53+0+73 (secure 0 0 0) 0x7f48e0099720 con 0x7f48ec06adc0 2024-03-11T19:09:51.503 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-11T19:09:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.502+0000 7f48f28b7700 1 -- 172.21.15.29:0/2903487032 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f48d40a1ac0 msgr2=0x7f48d40a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.502+0000 7f48f28b7700 1 --2- 172.21.15.29:0/2903487032 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f48d40a1ac0 0x7f48d40a3f80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f48dc00ac50 tx=0x7f48dc00a380 comp rx=0 tx=0).stop 2024-03-11T19:09:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.502+0000 7f48f28b7700 1 -- 172.21.15.29:0/2903487032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48ec06adc0 msgr2=0x7f48ec1986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.502+0000 7f48f28b7700 1 --2- 172.21.15.29:0/2903487032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48ec06adc0 0x7f48ec1986f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f48e000d950 tx=0x7f48e000dc60 comp rx=0 tx=0).stop 2024-03-11T19:09:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.502+0000 7f48f28b7700 1 -- 172.21.15.29:0/2903487032 shutdown_connections 2024-03-11T19:09:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.502+0000 7f48f28b7700 1 --2- 172.21.15.29:0/2903487032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48ec106ff0 0x7f48ec19ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.502+0000 7f48f28b7700 1 --2- 172.21.15.29:0/2903487032 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f48d40a1ac0 0x7f48d40a3f80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.502+0000 7f48f28b7700 1 --2- 172.21.15.29:0/2903487032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48ec06a410 0x7f48ec1981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.502+0000 7f48f28b7700 1 --2- 172.21.15.29:0/2903487032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48ec06adc0 0x7f48ec1986f0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.502+0000 7f48f28b7700 1 -- 172.21.15.29:0/2903487032 >> 172.21.15.29:0/2903487032 conn(0x7f48ec076bc0 msgr2=0x7f48ec0fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.502+0000 7f48f28b7700 1 -- 172.21.15.29:0/2903487032 shutdown_connections 2024-03-11T19:09:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:51.502+0000 7f48f28b7700 1 -- 172.21.15.29:0/2903487032 wait complete. 2024-03-11T19:09:51.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:51 smithi190 ceph-mon[36292]: pgmap v352: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:51.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:51 smithi170 ceph-mon[35690]: pgmap v352: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:52.223 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-11T19:09:52.223 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-11T19:09:52.223 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2024-03-11T19:09:52.223 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-11T19:09:52.223 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-11T19:09:52.223 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-11T19:09:52.223 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-11T19:09:52.223 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-11T19:09:52.223 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-11T19:09:52.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:09:52.224 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2024-03-11T19:09:52.239 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-11T19:09:52.239 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-11T19:09:52.239 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2024-03-11T19:09:52.239 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-03-11T19:09:52.239 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2024-03-11T19:09:52.239 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-03-11T19:09:52.240 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T19:09:52.240 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2024-03-11T19:09:52.240 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2024-03-11T19:09:52.240 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T19:09:52.240 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2024-03-11T19:09:52.240 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2024-03-11T19:09:52.240 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-11T19:09:52.240 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2024-03-11T19:09:52.240 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2024-03-11T19:09:52.240 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-11T19:09:52.240 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 42084sec preferred_lft 42084sec 2024-03-11T19:09:52.240 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2024-03-11T19:09:52.240 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-03-11T19:09:52.240 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T19:09:52.240 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2024-03-11T19:09:52.241 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2024-03-11T19:09:52.241 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-11T19:09:52.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:09:52.241 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2024-03-11T19:09:52.241 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-03-11T19:09:52.241 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add name ceph-brx type bridge 2024-03-11T19:09:52.241 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr flush dev ceph-brx 2024-03-11T19:09:52.242 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set ceph-brx up 2024-03-11T19:09:52.242 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-11T19:09:52.242 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-03-11T19:09:52.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:09:52.421 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-11T19:09:52.435 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:52 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2903487032' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-11T19:09:52.448 INFO:teuthology.orchestra.run.smithi029.stdout:1 2024-03-11T19:09:52.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:09:52.450 DEBUG:teuthology.orchestra.run.smithi029:> route 2024-03-11T19:09:52.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:52 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2903487032' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-11T19:09:52.527 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2024-03-11T19:09:52.527 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-11T19:09:52.527 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-11T19:09:52.527 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-11T19:09:52.527 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-11T19:09:52.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:09:52.528 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2024-03-11T19:09:52.528 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-03-11T19:09:52.528 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-11T19:09:52.528 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-11T19:09:52.528 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-11T19:09:52.528 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-03-11T19:09:52.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:52 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2903487032' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-11T19:09:52.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:09:52.795 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-03-11T19:09:52.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:09:52.810 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list-id 2024-03-11T19:09:52.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:09:52.865 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2024-03-11T19:09:52.865 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-03-11T19:09:52.865 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-11T19:09:52.865 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-11T19:09:52.865 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-03-11T19:09:52.948 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-11T19:09:52.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:09:52.948 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2024-03-11T19:09:52.948 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-03-11T19:09:52.948 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-11T19:09:52.948 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-11T19:09:52.948 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-11T19:09:52.948 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-11T19:09:52.948 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-11T19:09:52.948 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-03-11T19:09:53.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:09:53.087 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2024-03-11T19:09:53.087 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-03-11T19:09:53.087 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2024-03-11T19:09:53.087 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set dev brx.0 master ceph-brx 2024-03-11T19:09:53.087 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-03-11T19:09:53.128 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-03-11T19:09:53.128 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-11T19:09:53.128 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-11T19:09:53.183 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-11T19:09:53.184 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-11T19:09:53.184 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-11T19:09:53.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo modprobe fuse 2024-03-11T19:09:53.302 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:53 smithi029 ceph-mon[32439]: pgmap v353: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:53.306 DEBUG:teuthology.orchestra.run.smithi029:> cat /proc/self/mounts | awk '{print $2}' 2024-03-11T19:09:53.361 INFO:teuthology.orchestra.run.smithi029.stdout:/proc 2024-03-11T19:09:53.361 INFO:teuthology.orchestra.run.smithi029.stdout:/sys 2024-03-11T19:09:53.361 INFO:teuthology.orchestra.run.smithi029.stdout:/dev 2024-03-11T19:09:53.361 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/security 2024-03-11T19:09:53.361 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/shm 2024-03-11T19:09:53.361 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/pts 2024-03-11T19:09:53.361 INFO:teuthology.orchestra.run.smithi029.stdout:/run 2024-03-11T19:09:53.361 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/cgroup 2024-03-11T19:09:53.362 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/pstore 2024-03-11T19:09:53.362 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/bpf 2024-03-11T19:09:53.362 INFO:teuthology.orchestra.run.smithi029.stdout:/ 2024-03-11T19:09:53.362 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/selinux 2024-03-11T19:09:53.362 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-03-11T19:09:53.362 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/mqueue 2024-03-11T19:09:53.362 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/hugepages 2024-03-11T19:09:53.362 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/debug 2024-03-11T19:09:53.362 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/tracing 2024-03-11T19:09:53.362 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-11T19:09:53.362 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/fuse/connections 2024-03-11T19:09:53.362 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-sysctl.service 2024-03-11T19:09:53.362 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/config 2024-03-11T19:09:53.363 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-11T19:09:53.363 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/nfs/rpc_pipefs 2024-03-11T19:09:53.363 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/1000 2024-03-11T19:09:53.363 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph 2024-03-11T19:09:53.363 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay 2024-03-11T19:09:53.363 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/3d9bde034cb0627f34e8a089f1da44f2d86802e2c21695a7dd7a0399a061a122/merged 2024-03-11T19:09:53.363 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/dab85c6361a321dd8976b4b33d782331a247709bc318b26063e24a3bb154d720/merged 2024-03-11T19:09:53.363 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/0 2024-03-11T19:09:53.363 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-03-11T19:09:53.363 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/2adc7a11aa95e577c82636a9841da6c7d0c9a67d58ed448506009aca082aec8c/merged 2024-03-11T19:09:53.363 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/a2212dd798ea448b3dc872574654f93fcf2c565ec79a077317e41ac46c05881f/merged 2024-03-11T19:09:53.363 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/76477126e0e13b6937add7b589a7b7e11facdfd550d59a9826d11a6e57a8cb5e/merged 2024-03-11T19:09:53.363 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/9c9ad3511f0ac6d67e8198a5584968a8a3b33731195f8d6d67f20062e8c5c28f/merged 2024-03-11T19:09:53.363 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/425468d1abc18dda3f1dbb1033a9209aa51241c8e37eaea922f494bba3c1d60e/merged 2024-03-11T19:09:53.363 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/6a57644431584d613270fc0028dbb5abf460c6ca3d1866797e81b89ebdadef87/merged 2024-03-11T19:09:53.364 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/6e75a0280f7db62170730d17c20345775a62b26371191a1c5492faeea81bc56b/merged 2024-03-11T19:09:53.364 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/15b864bd9878bbf65c4f0f02b58abfadfd63cdf49b6710c733d3c9500c58a7ff/merged 2024-03-11T19:09:53.364 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns 2024-03-11T19:09:53.364 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-11T19:09:53.364 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-11T19:09:53.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:09:53.364 DEBUG:teuthology.orchestra.run.smithi029:> ls /sys/fs/fuse/connections 2024-03-11T19:09:53.416 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-11T19:09:53.416 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/49ed91cb-cc36-4060-8530-37ec69b004c8 --client_fs=cephfs) 2024-03-11T19:09:53.459 DEBUG:teuthology.orchestra.run.smithi029:> sudo modprobe fuse 2024-03-11T19:09:53.486 DEBUG:teuthology.orchestra.run.smithi029:> cat /proc/self/mounts | awk '{print $2}' 2024-03-11T19:09:53.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:53 smithi190 ceph-mon[36292]: pgmap v353: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:53.541 INFO:teuthology.orchestra.run.smithi029.stdout:/proc 2024-03-11T19:09:53.541 INFO:teuthology.orchestra.run.smithi029.stdout:/sys 2024-03-11T19:09:53.541 INFO:teuthology.orchestra.run.smithi029.stdout:/dev 2024-03-11T19:09:53.541 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/security 2024-03-11T19:09:53.541 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/shm 2024-03-11T19:09:53.541 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/pts 2024-03-11T19:09:53.541 INFO:teuthology.orchestra.run.smithi029.stdout:/run 2024-03-11T19:09:53.541 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/cgroup 2024-03-11T19:09:53.541 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/pstore 2024-03-11T19:09:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/bpf 2024-03-11T19:09:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:/ 2024-03-11T19:09:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/selinux 2024-03-11T19:09:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-03-11T19:09:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/mqueue 2024-03-11T19:09:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/hugepages 2024-03-11T19:09:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/debug 2024-03-11T19:09:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/tracing 2024-03-11T19:09:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-11T19:09:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/fuse/connections 2024-03-11T19:09:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-sysctl.service 2024-03-11T19:09:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/config 2024-03-11T19:09:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-11T19:09:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/nfs/rpc_pipefs 2024-03-11T19:09:53.542 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/1000 2024-03-11T19:09:53.543 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph 2024-03-11T19:09:53.543 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay 2024-03-11T19:09:53.543 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/3d9bde034cb0627f34e8a089f1da44f2d86802e2c21695a7dd7a0399a061a122/merged 2024-03-11T19:09:53.543 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/dab85c6361a321dd8976b4b33d782331a247709bc318b26063e24a3bb154d720/merged 2024-03-11T19:09:53.543 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/0 2024-03-11T19:09:53.543 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-03-11T19:09:53.543 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/2adc7a11aa95e577c82636a9841da6c7d0c9a67d58ed448506009aca082aec8c/merged 2024-03-11T19:09:53.543 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/a2212dd798ea448b3dc872574654f93fcf2c565ec79a077317e41ac46c05881f/merged 2024-03-11T19:09:53.543 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/76477126e0e13b6937add7b589a7b7e11facdfd550d59a9826d11a6e57a8cb5e/merged 2024-03-11T19:09:53.543 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/9c9ad3511f0ac6d67e8198a5584968a8a3b33731195f8d6d67f20062e8c5c28f/merged 2024-03-11T19:09:53.543 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/425468d1abc18dda3f1dbb1033a9209aa51241c8e37eaea922f494bba3c1d60e/merged 2024-03-11T19:09:53.543 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/6a57644431584d613270fc0028dbb5abf460c6ca3d1866797e81b89ebdadef87/merged 2024-03-11T19:09:53.543 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/6e75a0280f7db62170730d17c20345775a62b26371191a1c5492faeea81bc56b/merged 2024-03-11T19:09:53.543 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/15b864bd9878bbf65c4f0f02b58abfadfd63cdf49b6710c733d3c9500c58a7ff/merged 2024-03-11T19:09:53.543 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns 2024-03-11T19:09:53.544 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-11T19:09:53.544 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-11T19:09:53.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:09:53.544 DEBUG:teuthology.orchestra.run.smithi029:> ls /sys/fs/fuse/connections 2024-03-11T19:09:53.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi029.stderr:2024-03-11T19:09:53.553+0000 7fd7cf2da480 -1 init, newargv = 0x56108d446260 newargc=15 2024-03-11T19:09:53.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi029.stderr:ceph-fuse[84458]: starting ceph client 2024-03-11T19:09:53.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi029.stderr:ceph-fuse[84458]: starting fuse 2024-03-11T19:09:53.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:53 smithi170 ceph-mon[35690]: pgmap v353: 97 pgs: 97 active+clean; 643 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:09:54.671 DEBUG:teuthology.orchestra.run.smithi029:> sudo modprobe fuse 2024-03-11T19:09:54.705 DEBUG:teuthology.orchestra.run.smithi029:> cat /proc/self/mounts | awk '{print $2}' 2024-03-11T19:09:54.763 INFO:teuthology.orchestra.run.smithi029.stdout:/proc 2024-03-11T19:09:54.763 INFO:teuthology.orchestra.run.smithi029.stdout:/sys 2024-03-11T19:09:54.763 INFO:teuthology.orchestra.run.smithi029.stdout:/dev 2024-03-11T19:09:54.763 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/security 2024-03-11T19:09:54.763 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/shm 2024-03-11T19:09:54.763 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/pts 2024-03-11T19:09:54.763 INFO:teuthology.orchestra.run.smithi029.stdout:/run 2024-03-11T19:09:54.764 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/cgroup 2024-03-11T19:09:54.764 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/pstore 2024-03-11T19:09:54.764 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/bpf 2024-03-11T19:09:54.764 INFO:teuthology.orchestra.run.smithi029.stdout:/ 2024-03-11T19:09:54.764 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/selinux 2024-03-11T19:09:54.764 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-03-11T19:09:54.764 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/mqueue 2024-03-11T19:09:54.764 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/hugepages 2024-03-11T19:09:54.764 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/debug 2024-03-11T19:09:54.764 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/tracing 2024-03-11T19:09:54.764 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-11T19:09:54.764 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/fuse/connections 2024-03-11T19:09:54.764 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-sysctl.service 2024-03-11T19:09:54.765 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/config 2024-03-11T19:09:54.765 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-11T19:09:54.765 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/nfs/rpc_pipefs 2024-03-11T19:09:54.765 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/1000 2024-03-11T19:09:54.765 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph 2024-03-11T19:09:54.765 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay 2024-03-11T19:09:54.765 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/3d9bde034cb0627f34e8a089f1da44f2d86802e2c21695a7dd7a0399a061a122/merged 2024-03-11T19:09:54.765 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/dab85c6361a321dd8976b4b33d782331a247709bc318b26063e24a3bb154d720/merged 2024-03-11T19:09:54.765 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/0 2024-03-11T19:09:54.765 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-03-11T19:09:54.765 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/2adc7a11aa95e577c82636a9841da6c7d0c9a67d58ed448506009aca082aec8c/merged 2024-03-11T19:09:54.765 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/a2212dd798ea448b3dc872574654f93fcf2c565ec79a077317e41ac46c05881f/merged 2024-03-11T19:09:54.765 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/76477126e0e13b6937add7b589a7b7e11facdfd550d59a9826d11a6e57a8cb5e/merged 2024-03-11T19:09:54.766 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/9c9ad3511f0ac6d67e8198a5584968a8a3b33731195f8d6d67f20062e8c5c28f/merged 2024-03-11T19:09:54.766 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/425468d1abc18dda3f1dbb1033a9209aa51241c8e37eaea922f494bba3c1d60e/merged 2024-03-11T19:09:54.766 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/6a57644431584d613270fc0028dbb5abf460c6ca3d1866797e81b89ebdadef87/merged 2024-03-11T19:09:54.766 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/6e75a0280f7db62170730d17c20345775a62b26371191a1c5492faeea81bc56b/merged 2024-03-11T19:09:54.766 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/containers/storage/overlay/15b864bd9878bbf65c4f0f02b58abfadfd63cdf49b6710c733d3c9500c58a7ff/merged 2024-03-11T19:09:54.766 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns 2024-03-11T19:09:54.766 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-11T19:09:54.766 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-11T19:09:54.766 INFO:teuthology.orchestra.run.smithi029.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-11T19:09:54.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:09:54.767 DEBUG:teuthology.orchestra.run.smithi029:> ls /sys/fs/fuse/connections 2024-03-11T19:09:54.821 INFO:teuthology.orchestra.run.smithi029.stdout:119 2024-03-11T19:09:54.821 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-11T19:09:54.821 DEBUG:teuthology.orchestra.run.smithi029:> sudo stdin-killer -- python3 -c ' 2024-03-11T19:09:54.821 DEBUG:teuthology.orchestra.run.smithi029:> import glob 2024-03-11T19:09:54.821 DEBUG:teuthology.orchestra.run.smithi029:> import re 2024-03-11T19:09:54.822 DEBUG:teuthology.orchestra.run.smithi029:> import os 2024-03-11T19:09:54.822 DEBUG:teuthology.orchestra.run.smithi029:> import subprocess 2024-03-11T19:09:54.822 DEBUG:teuthology.orchestra.run.smithi029:> 2024-03-11T19:09:54.822 DEBUG:teuthology.orchestra.run.smithi029:> def _find_admin_socket(client_name): 2024-03-11T19:09:54.822 DEBUG:teuthology.orchestra.run.smithi029:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-11T19:09:54.822 DEBUG:teuthology.orchestra.run.smithi029:> files = glob.glob(asok_path) 2024-03-11T19:09:54.822 DEBUG:teuthology.orchestra.run.smithi029:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-11T19:09:54.822 DEBUG:teuthology.orchestra.run.smithi029:> 2024-03-11T19:09:54.822 DEBUG:teuthology.orchestra.run.smithi029:> # Given a non-glob path, it better be there 2024-03-11T19:09:54.822 DEBUG:teuthology.orchestra.run.smithi029:> if "*" not in asok_path: 2024-03-11T19:09:54.822 DEBUG:teuthology.orchestra.run.smithi029:> assert(len(files) == 1) 2024-03-11T19:09:54.822 DEBUG:teuthology.orchestra.run.smithi029:> return files[0] 2024-03-11T19:09:54.822 DEBUG:teuthology.orchestra.run.smithi029:> 2024-03-11T19:09:54.822 DEBUG:teuthology.orchestra.run.smithi029:> for f in files: 2024-03-11T19:09:54.822 DEBUG:teuthology.orchestra.run.smithi029:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-11T19:09:54.822 DEBUG:teuthology.orchestra.run.smithi029:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-11T19:09:54.823 DEBUG:teuthology.orchestra.run.smithi029:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-11T19:09:54.823 DEBUG:teuthology.orchestra.run.smithi029:> contents = proc_f.read() 2024-03-11T19:09:54.823 DEBUG:teuthology.orchestra.run.smithi029:> if mountpoint in contents: 2024-03-11T19:09:54.823 DEBUG:teuthology.orchestra.run.smithi029:> return f 2024-03-11T19:09:54.823 DEBUG:teuthology.orchestra.run.smithi029:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-11T19:09:54.823 DEBUG:teuthology.orchestra.run.smithi029:> 2024-03-11T19:09:54.823 DEBUG:teuthology.orchestra.run.smithi029:> print(_find_admin_socket("client.0")) 2024-03-11T19:09:54.823 DEBUG:teuthology.orchestra.run.smithi029:> ' 2024-03-11T19:09:54.950 INFO:teuthology.orchestra.run.smithi029.stdout:/var/run/ceph/ceph-client.0.84458.asok 2024-03-11T19:09:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-11T19:09:54.961 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.84458.asok 2024-03-11T19:09:54.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:09:54.961 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.84458.asok status 2024-03-11T19:09:55.101 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:09:55.101 INFO:teuthology.orchestra.run.smithi029.stdout: "metadata": { 2024-03-11T19:09:55.101 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_sha1": "94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T19:09:55.101 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T19:09:55.101 INFO:teuthology.orchestra.run.smithi029.stdout: "entity_id": "0", 2024-03-11T19:09:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: "hostname": "smithi029", 2024-03-11T19:09:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-11T19:09:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: "pid": "84458", 2024-03-11T19:09:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: "root": "/volumes/qa/sv_0/49ed91cb-cc36-4060-8530-37ec69b004c8" 2024-03-11T19:09:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:09:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: "dentry_count": 0, 2024-03-11T19:09:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: "dentry_pinned_count": 0, 2024-03-11T19:09:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 15330, 2024-03-11T19:09:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: "inst": { 2024-03-11T19:09:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: "name": { 2024-03-11T19:09:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "client", 2024-03-11T19:09:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: "num": 15330 2024-03-11T19:09:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:09:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": { 2024-03-11T19:09:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v1", 2024-03-11T19:09:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "192.168.0.1:0", 2024-03-11T19:09:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2594061520 2024-03-11T19:09:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:09:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:09:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": { 2024-03-11T19:09:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v1", 2024-03-11T19:09:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "192.168.0.1:0", 2024-03-11T19:09:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2594061520 2024-03-11T19:09:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:09:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: "inst_str": "client.15330 192.168.0.1:0/2594061520", 2024-03-11T19:09:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: "addr_str": "192.168.0.1:0/2594061520", 2024-03-11T19:09:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: "inode_count": 5, 2024-03-11T19:09:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_epoch": 26, 2024-03-11T19:09:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_epoch": 89, 2024-03-11T19:09:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_epoch_barrier": 0, 2024-03-11T19:09:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: "blocklisted": false, 2024-03-11T19:09:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs" 2024-03-11T19:09:55.104 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:09:55.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:09:55.109 DEBUG:teuthology.orchestra.run.smithi029:> stat --file-system '--printf=%T 2024-03-11T19:09:55.110 DEBUG:teuthology.orchestra.run.smithi029:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-11T19:09:55.125 INFO:teuthology.orchestra.run.smithi029.stdout:fuseblk 2024-03-11T19:09:55.125 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-11T19:09:55.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:09:55.125 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-11T19:09:55.195 INFO:teuthology.run_tasks:Running task exec... 2024-03-11T19:09:55.205 INFO:teuthology.task.exec:Executing custom commands... 2024-03-11T19:09:55.205 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-03-11T19:09:55.206 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-11T19:09:55.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:55 smithi029 ceph-mon[32439]: pgmap v354: 97 pgs: 97 active+clean; 644 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-11T19:09:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.336+0000 7ff64c642640 1 -- 172.21.15.29:0/3156046170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6441011b0 msgr2=0x7ff6441015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.336+0000 7ff64c642640 1 --2- 172.21.15.29:0/3156046170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6441011b0 0x7ff6441015d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff640009900 tx=0x7ff64002bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.336+0000 7ff64c642640 1 -- 172.21.15.29:0/3156046170 shutdown_connections 2024-03-11T19:09:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.336+0000 7ff64c642640 1 --2- 172.21.15.29:0/3156046170 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff644102d60 0x7ff6441095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.336+0000 7ff64c642640 1 --2- 172.21.15.29:0/3156046170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6441023a0 0x7ff644102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.336+0000 7ff64c642640 1 --2- 172.21.15.29:0/3156046170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6441011b0 0x7ff6441015d0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.337+0000 7ff64c642640 1 -- 172.21.15.29:0/3156046170 >> 172.21.15.29:0/3156046170 conn(0x7ff6440fc740 msgr2=0x7ff6440feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.337+0000 7ff64c642640 1 -- 172.21.15.29:0/3156046170 shutdown_connections 2024-03-11T19:09:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.337+0000 7ff64c642640 1 -- 172.21.15.29:0/3156046170 wait complete. 2024-03-11T19:09:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.337+0000 7ff64c642640 1 Processor -- start 2024-03-11T19:09:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.337+0000 7ff64c642640 1 -- start start 2024-03-11T19:09:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.337+0000 7ff64c642640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6441011b0 0x7ff6441970a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.337+0000 7ff64c642640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6441023a0 0x7ff6441975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.337+0000 7ff64c642640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff644102d60 0x7ff64419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.337+0000 7ff64c642640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64419eb90 con 0x7ff644102d60 2024-03-11T19:09:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.337+0000 7ff64c642640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64419ed00 con 0x7ff6441011b0 2024-03-11T19:09:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.337+0000 7ff64c642640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64419eec0 con 0x7ff6441023a0 2024-03-11T19:09:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.337+0000 7ff64abdf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff644102d60 0x7ff64419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.337+0000 7ff64abdf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff644102d60 0x7ff64419e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49666/0 (socket says 172.21.15.29:49666) 2024-03-11T19:09:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.337+0000 7ff64abdf640 1 -- 172.21.15.29:0/1741975988 learned_addr learned my addr 172.21.15.29:0/1741975988 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.338+0000 7ff64a3de640 1 --2- 172.21.15.29:0/1741975988 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6441011b0 0x7ff6441970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.338+0000 7ff64abdf640 1 -- 172.21.15.29:0/1741975988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6441023a0 msgr2=0x7ff6441975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.338+0000 7ff64abdf640 1 --2- 172.21.15.29:0/1741975988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6441023a0 0x7ff6441975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.338+0000 7ff64abdf640 1 -- 172.21.15.29:0/1741975988 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6441011b0 msgr2=0x7ff6441970a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.338+0000 7ff64abdf640 1 --2- 172.21.15.29:0/1741975988 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6441011b0 0x7ff6441970a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.338+0000 7ff64abdf640 1 -- 172.21.15.29:0/1741975988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff640009560 con 0x7ff644102d60 2024-03-11T19:09:55.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.338+0000 7ff64abdf640 1 --2- 172.21.15.29:0/1741975988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff644102d60 0x7ff64419e650 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00cce0 tx=0x7ff63c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:55.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.338+0000 7ff63b7fe640 1 -- 172.21.15.29:0/1741975988 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c00f450 con 0x7ff644102d60 2024-03-11T19:09:55.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.338+0000 7ff63b7fe640 1 -- 172.21.15.29:0/1741975988 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff63c00fa60 con 0x7ff644102d60 2024-03-11T19:09:55.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.338+0000 7ff64c642640 1 -- 172.21.15.29:0/1741975988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64419f1a0 con 0x7ff644102d60 2024-03-11T19:09:55.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.338+0000 7ff63b7fe640 1 -- 172.21.15.29:0/1741975988 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c018b10 con 0x7ff644102d60 2024-03-11T19:09:55.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.338+0000 7ff64c642640 1 -- 172.21.15.29:0/1741975988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64419f660 con 0x7ff644102d60 2024-03-11T19:09:55.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.339+0000 7ff64c642640 1 -- 172.21.15.29:0/1741975988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff644108470 con 0x7ff644102d60 2024-03-11T19:09:55.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.340+0000 7ff63b7fe640 1 -- 172.21.15.29:0/1741975988 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff63c00fbd0 con 0x7ff644102d60 2024-03-11T19:09:55.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.341+0000 7ff63b7fe640 1 --2- 172.21.15.29:0/1741975988 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff6300a1ad0 0x7ff6300a3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:55.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.341+0000 7ff63b7fe640 1 -- 172.21.15.29:0/1741975988 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff63c0cd7a0 con 0x7ff644102d60 2024-03-11T19:09:55.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.341+0000 7ff64a3de640 1 --2- 172.21.15.29:0/1741975988 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff6300a1ad0 0x7ff6300a3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:55.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.341+0000 7ff64a3de640 1 --2- 172.21.15.29:0/1741975988 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff6300a1ad0 0x7ff6300a3f90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff64002c7d0 tx=0x7ff640005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:55.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.345+0000 7ff63b7fe640 1 -- 172.21.15.29:0/1741975988 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff63c096a50 con 0x7ff644102d60 2024-03-11T19:09:55.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:55 smithi190 ceph-mon[36292]: pgmap v354: 97 pgs: 97 active+clean; 644 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-11T19:09:55.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:55.568+0000 7ff64c642640 1 -- 172.21.15.29:0/1741975988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7ff64410cc80 con 0x7ff644102d60 2024-03-11T19:09:55.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:55 smithi170 ceph-mon[35690]: pgmap v354: 97 pgs: 97 active+clean; 644 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-11T19:09:56.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.189+0000 7ff63b7fe640 1 -- 172.21.15.29:0/1741975988 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v27) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff63c09a110 con 0x7ff644102d60 2024-03-11T19:09:56.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.192+0000 7ff64c642640 1 -- 172.21.15.29:0/1741975988 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff6300a1ad0 msgr2=0x7ff6300a3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:56.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.192+0000 7ff64c642640 1 --2- 172.21.15.29:0/1741975988 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff6300a1ad0 0x7ff6300a3f90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff64002c7d0 tx=0x7ff640005fd0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.192+0000 7ff64c642640 1 -- 172.21.15.29:0/1741975988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff644102d60 msgr2=0x7ff64419e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:56.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.192+0000 7ff64c642640 1 --2- 172.21.15.29:0/1741975988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff644102d60 0x7ff64419e650 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00cce0 tx=0x7ff63c00a620 comp rx=0 tx=0).stop 2024-03-11T19:09:56.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.192+0000 7ff64c642640 1 -- 172.21.15.29:0/1741975988 shutdown_connections 2024-03-11T19:09:56.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.192+0000 7ff64c642640 1 --2- 172.21.15.29:0/1741975988 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff6300a1ad0 0x7ff6300a3f90 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.192+0000 7ff64c642640 1 --2- 172.21.15.29:0/1741975988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff644102d60 0x7ff64419e650 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.192+0000 7ff64c642640 1 --2- 172.21.15.29:0/1741975988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6441023a0 0x7ff6441975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.192+0000 7ff64c642640 1 --2- 172.21.15.29:0/1741975988 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6441011b0 0x7ff6441970a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.192+0000 7ff64c642640 1 -- 172.21.15.29:0/1741975988 >> 172.21.15.29:0/1741975988 conn(0x7ff6440fc740 msgr2=0x7ff644105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:56.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.192+0000 7ff64c642640 1 -- 172.21.15.29:0/1741975988 shutdown_connections 2024-03-11T19:09:56.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.192+0000 7ff64c642640 1 -- 172.21.15.29:0/1741975988 wait complete. 2024-03-11T19:09:56.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-11T19:09:56.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.360+0000 7f9df164e640 1 -- 172.21.15.29:0/4060477625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dec101c70 msgr2=0x7f9dec108500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:56.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.360+0000 7f9df164e640 1 --2- 172.21.15.29:0/4060477625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dec101c70 0x7f9dec108500 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc009ca0 tx=0x7f9ddc02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.361+0000 7f9df164e640 1 -- 172.21.15.29:0/4060477625 shutdown_connections 2024-03-11T19:09:56.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.361+0000 7f9df164e640 1 --2- 172.21.15.29:0/4060477625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dec101c70 0x7f9dec108500 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.361+0000 7f9df164e640 1 --2- 172.21.15.29:0/4060477625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dec1012b0 0x7f9dec101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.361+0000 7f9df164e640 1 --2- 172.21.15.29:0/4060477625 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dec1000c0 0x7f9dec1004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.361+0000 7f9df164e640 1 -- 172.21.15.29:0/4060477625 >> 172.21.15.29:0/4060477625 conn(0x7f9dec0fb630 msgr2=0x7f9dec0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:56.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.361+0000 7f9df164e640 1 -- 172.21.15.29:0/4060477625 shutdown_connections 2024-03-11T19:09:56.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.361+0000 7f9df164e640 1 -- 172.21.15.29:0/4060477625 wait complete. 2024-03-11T19:09:56.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.361+0000 7f9df164e640 1 Processor -- start 2024-03-11T19:09:56.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.361+0000 7f9df164e640 1 -- start start 2024-03-11T19:09:56.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.361+0000 7f9df164e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dec1000c0 0x7f9dec1970c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:56.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.361+0000 7f9df164e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dec1012b0 0x7f9dec197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:56.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.361+0000 7f9df164e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dec101c70 0x7f9dec19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:56.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.361+0000 7f9df164e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dec19ebb0 con 0x7f9dec1000c0 2024-03-11T19:09:56.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.361+0000 7f9df164e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dec19ed20 con 0x7f9dec101c70 2024-03-11T19:09:56.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.362+0000 7f9df164e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dec19eee0 con 0x7f9dec1012b0 2024-03-11T19:09:56.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.362+0000 7f9deaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dec1000c0 0x7f9dec1970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:56.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.362+0000 7f9deaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dec1000c0 0x7f9dec1970c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54254/0 (socket says 172.21.15.29:54254) 2024-03-11T19:09:56.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.362+0000 7f9deaffd640 1 -- 172.21.15.29:0/3848698212 learned_addr learned my addr 172.21.15.29:0/3848698212 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:56.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.362+0000 7f9deb7fe640 1 --2- 172.21.15.29:0/3848698212 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dec101c70 0x7f9dec19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:56.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.362+0000 7f9dea7fc640 1 --2- 172.21.15.29:0/3848698212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dec1012b0 0x7f9dec197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:56.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.362+0000 7f9deaffd640 1 -- 172.21.15.29:0/3848698212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dec1012b0 msgr2=0x7f9dec197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.362+0000 7f9deaffd640 1 --2- 172.21.15.29:0/3848698212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dec1012b0 0x7f9dec197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.362+0000 7f9deaffd640 1 -- 172.21.15.29:0/3848698212 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dec101c70 msgr2=0x7f9dec19e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.362+0000 7f9deaffd640 1 --2- 172.21.15.29:0/3848698212 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dec101c70 0x7f9dec19e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.362+0000 7f9deaffd640 1 -- 172.21.15.29:0/3848698212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ddc009900 con 0x7f9dec1000c0 2024-03-11T19:09:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.362+0000 7f9deaffd640 1 --2- 172.21.15.29:0/3848698212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dec1000c0 0x7f9dec1970c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9de400e9c0 tx=0x7f9de400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.362+0000 7f9dd3fff640 1 -- 172.21.15.29:0/3848698212 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de4009700 con 0x7f9dec1000c0 2024-03-11T19:09:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.362+0000 7f9dd3fff640 1 -- 172.21.15.29:0/3848698212 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9de40044d0 con 0x7f9dec1000c0 2024-03-11T19:09:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.362+0000 7f9dd3fff640 1 -- 172.21.15.29:0/3848698212 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de4010430 con 0x7f9dec1000c0 2024-03-11T19:09:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.362+0000 7f9df164e640 1 -- 172.21.15.29:0/3848698212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dec19f1c0 con 0x7f9dec1000c0 2024-03-11T19:09:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.362+0000 7f9df164e640 1 -- 172.21.15.29:0/3848698212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dec19f680 con 0x7f9dec1000c0 2024-03-11T19:09:56.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.363+0000 7f9df164e640 1 -- 172.21.15.29:0/3848698212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dec10b980 con 0x7f9dec1000c0 2024-03-11T19:09:56.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.365+0000 7f9dd3fff640 1 -- 172.21.15.29:0/3848698212 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9de4003f30 con 0x7f9dec1000c0 2024-03-11T19:09:56.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.365+0000 7f9dd3fff640 1 --2- 172.21.15.29:0/3848698212 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9dd40a1b50 0x7f9dd40a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:56.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.365+0000 7f9dea7fc640 1 --2- 172.21.15.29:0/3848698212 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9dd40a1b50 0x7f9dd40a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:56.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.366+0000 7f9dd3fff640 1 -- 172.21.15.29:0/3848698212 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9de4014070 con 0x7f9dec1000c0 2024-03-11T19:09:56.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.366+0000 7f9dea7fc640 1 --2- 172.21.15.29:0/3848698212 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9dd40a1b50 0x7f9dd40a4010 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9dec1985d0 tx=0x7f9de0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:56.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.370+0000 7f9dd3fff640 1 -- 172.21.15.29:0/3848698212 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9de4096a20 con 0x7f9dec1000c0 2024-03-11T19:09:56.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:56 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1741975988' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-11T19:09:56.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.578+0000 7f9df164e640 1 -- 172.21.15.29:0/3848698212 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) v1 -- 0x7f9dec0fd760 con 0x7f9dd40a1b50 2024-03-11T19:09:56.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:56 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1741975988' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-11T19:09:56.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:56 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1741975988' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-11T19:09:56.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.595+0000 7f9dd3fff640 1 -- 172.21.15.29:0/3848698212 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3 (secure 0 0 0) 0x7f9dec0fd760 con 0x7f9dd40a1b50 2024-03-11T19:09:56.598 INFO:teuthology.orchestra.run.smithi029.stdout:{} 2024-03-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.597+0000 7f9df164e640 1 -- 172.21.15.29:0/3848698212 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9dd40a1b50 msgr2=0x7f9dd40a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.597+0000 7f9df164e640 1 --2- 172.21.15.29:0/3848698212 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9dd40a1b50 0x7f9dd40a4010 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9dec1985d0 tx=0x7f9de0005f90 comp rx=0 tx=0).stop 2024-03-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.597+0000 7f9df164e640 1 -- 172.21.15.29:0/3848698212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dec1000c0 msgr2=0x7f9dec1970c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.597+0000 7f9df164e640 1 --2- 172.21.15.29:0/3848698212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dec1000c0 0x7f9dec1970c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9de400e9c0 tx=0x7f9de400ee80 comp rx=0 tx=0).stop 2024-03-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.597+0000 7f9df164e640 1 -- 172.21.15.29:0/3848698212 shutdown_connections 2024-03-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.597+0000 7f9df164e640 1 --2- 172.21.15.29:0/3848698212 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dec101c70 0x7f9dec19e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.597+0000 7f9df164e640 1 --2- 172.21.15.29:0/3848698212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dec1012b0 0x7f9dec197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.597+0000 7f9df164e640 1 --2- 172.21.15.29:0/3848698212 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9dd40a1b50 0x7f9dd40a4010 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.597+0000 7f9df164e640 1 --2- 172.21.15.29:0/3848698212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dec1000c0 0x7f9dec1970c0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.598+0000 7f9df164e640 1 -- 172.21.15.29:0/3848698212 >> 172.21.15.29:0/3848698212 conn(0x7f9dec0fb630 msgr2=0x7f9dec0fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.598+0000 7f9df164e640 1 -- 172.21.15.29:0/3848698212 shutdown_connections 2024-03-11T19:09:56.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.598+0000 7f9df164e640 1 -- 172.21.15.29:0/3848698212 wait complete. 2024-03-11T19:09:56.611 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-11T19:09:56.626 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-11T19:09:56.658 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:09:56.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.759+0000 7fecde9a7640 1 -- 172.21.15.29:0/3783116456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd81000a0 msgr2=0x7fecd81004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.759+0000 7fecde9a7640 1 --2- 172.21.15.29:0/3783116456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd81000a0 0x7fecd81004c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fecc0009900 tx=0x7fecc002bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.759+0000 7fecde9a7640 1 -- 172.21.15.29:0/3783116456 shutdown_connections 2024-03-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.759+0000 7fecde9a7640 1 --2- 172.21.15.29:0/3783116456 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecd8101c50 0x7fecd81084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.759+0000 7fecde9a7640 1 --2- 172.21.15.29:0/3783116456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecd8101290 0x7fecd8101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.759+0000 7fecde9a7640 1 --2- 172.21.15.29:0/3783116456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd81000a0 0x7fecd81004c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.759+0000 7fecde9a7640 1 -- 172.21.15.29:0/3783116456 >> 172.21.15.29:0/3783116456 conn(0x7fecd80fb610 msgr2=0x7fecd80fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.759+0000 7fecde9a7640 1 -- 172.21.15.29:0/3783116456 shutdown_connections 2024-03-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.759+0000 7fecde9a7640 1 -- 172.21.15.29:0/3783116456 wait complete. 2024-03-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.760+0000 7fecde9a7640 1 Processor -- start 2024-03-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.760+0000 7fecde9a7640 1 -- start start 2024-03-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.760+0000 7fecde9a7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecd81000a0 0x7fecd8196220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.760+0000 7fecde9a7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecd8101290 0x7fecd8196760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.760+0000 7fecde9a7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd8101c50 0x7fecd819d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.760+0000 7fecde9a7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd819dd10 con 0x7fecd8101c50 2024-03-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.760+0000 7fecde9a7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd819de80 con 0x7fecd8101290 2024-03-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.760+0000 7fecde9a7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd819e040 con 0x7fecd81000a0 2024-03-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.760+0000 7fecdcf44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd8101c50 0x7fecd819d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.760+0000 7fecdcf44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd8101c50 0x7fecd819d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54286/0 (socket says 172.21.15.29:54286) 2024-03-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.760+0000 7fecdcf44640 1 -- 172.21.15.29:0/3419587819 learned_addr learned my addr 172.21.15.29:0/3419587819 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.761+0000 7fecd77fe640 1 --2- 172.21.15.29:0/3419587819 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecd8101290 0x7fecd8196760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.761+0000 7fecdcf44640 1 -- 172.21.15.29:0/3419587819 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecd81000a0 msgr2=0x7fecd8196220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.761+0000 7fecd7fff640 1 --2- 172.21.15.29:0/3419587819 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecd81000a0 0x7fecd8196220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.761+0000 7fecdcf44640 1 --2- 172.21.15.29:0/3419587819 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecd81000a0 0x7fecd8196220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.761+0000 7fecdcf44640 1 -- 172.21.15.29:0/3419587819 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecd8101290 msgr2=0x7fecd8196760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.761+0000 7fecdcf44640 1 --2- 172.21.15.29:0/3419587819 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecd8101290 0x7fecd8196760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.761+0000 7fecdcf44640 1 -- 172.21.15.29:0/3419587819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc0009560 con 0x7fecd8101c50 2024-03-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.761+0000 7fecd77fe640 1 --2- 172.21.15.29:0/3419587819 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecd8101290 0x7fecd8196760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.761+0000 7fecdcf44640 1 --2- 172.21.15.29:0/3419587819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd8101c50 0x7fecd819d7d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fecd000b970 tx=0x7fecd000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.761+0000 7fecd57fa640 1 -- 172.21.15.29:0/3419587819 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd000cb40 con 0x7fecd8101c50 2024-03-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.761+0000 7fecd57fa640 1 -- 172.21.15.29:0/3419587819 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fecd000cca0 con 0x7fecd8101c50 2024-03-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.761+0000 7fecde9a7640 1 -- 172.21.15.29:0/3419587819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd819e320 con 0x7fecd8101c50 2024-03-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.761+0000 7fecd57fa640 1 -- 172.21.15.29:0/3419587819 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd00076e0 con 0x7fecd8101c50 2024-03-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.761+0000 7fecde9a7640 1 -- 172.21.15.29:0/3419587819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd819e8b0 con 0x7fecd8101c50 2024-03-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.762+0000 7fecde9a7640 1 -- 172.21.15.29:0/3419587819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd8107500 con 0x7fecd8101c50 2024-03-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.763+0000 7fecd57fa640 1 -- 172.21.15.29:0/3419587819 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fecd0014440 con 0x7fecd8101c50 2024-03-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.764+0000 7fecd57fa640 1 --2- 172.21.15.29:0/3419587819 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fecc40a1ab0 0x7fecc40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.764+0000 7fecd7fff640 1 --2- 172.21.15.29:0/3419587819 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fecc40a1ab0 0x7fecc40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.764+0000 7fecd57fa640 1 -- 172.21.15.29:0/3419587819 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fecd00ccd00 con 0x7fecd8101c50 2024-03-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.764+0000 7fecd7fff640 1 --2- 172.21.15.29:0/3419587819 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fecc40a1ab0 0x7fecc40a3f70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fecc0036040 tx=0x7fecc0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:56.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.768+0000 7fecd57fa640 1 -- 172.21.15.29:0/3419587819 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fecd0095f10 con 0x7fecd8101c50 2024-03-11T19:09:56.906 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:09:56 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:09:56.583+0000 7f36239d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:09:56.907 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:09:56 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:09:56.583+0000 7f36239d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:09:56.907 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:09:56 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:09:56.583+0000 7f36239d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:09:56.907 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:09:56 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:09:56.583+0000 7f36239d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:09:56.907 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:09:56 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:09:56.583+0000 7f36239d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:09:56.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.952+0000 7fecde9a7640 1 -- 172.21.15.29:0/3419587819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fecd802cdd0 con 0x7fecd8101c50 2024-03-11T19:09:56.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.954+0000 7fecd57fa640 1 -- 172.21.15.29:0/3419587819 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22559 (secure 0 0 0) 0x7fecd0019090 con 0x7fecd8101c50 2024-03-11T19:09:56.957 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:56.957 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":89,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:08:44.981806+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:07:47.563184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:07:48.282461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.956+0000 7fecde9a7640 1 -- 172.21.15.29:0/3419587819 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fecc40a1ab0 msgr2=0x7fecc40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.956+0000 7fecde9a7640 1 --2- 172.21.15.29:0/3419587819 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fecc40a1ab0 0x7fecc40a3f70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fecc0036040 tx=0x7fecc0005fd0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.956+0000 7fecde9a7640 1 -- 172.21.15.29:0/3419587819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd8101c50 msgr2=0x7fecd819d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.956+0000 7fecde9a7640 1 --2- 172.21.15.29:0/3419587819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd8101c50 0x7fecd819d7d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fecd000b970 tx=0x7fecd000be30 comp rx=0 tx=0).stop 2024-03-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.956+0000 7fecde9a7640 1 -- 172.21.15.29:0/3419587819 shutdown_connections 2024-03-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.956+0000 7fecde9a7640 1 --2- 172.21.15.29:0/3419587819 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fecc40a1ab0 0x7fecc40a3f70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.956+0000 7fecde9a7640 1 --2- 172.21.15.29:0/3419587819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecd8101c50 0x7fecd819d7d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.956+0000 7fecde9a7640 1 --2- 172.21.15.29:0/3419587819 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecd8101290 0x7fecd8196760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.956+0000 7fecde9a7640 1 --2- 172.21.15.29:0/3419587819 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecd81000a0 0x7fecd8196220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:56.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.956+0000 7fecde9a7640 1 -- 172.21.15.29:0/3419587819 >> 172.21.15.29:0/3419587819 conn(0x7fecd80fb610 msgr2=0x7fecd81046d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:56.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.956+0000 7fecde9a7640 1 -- 172.21.15.29:0/3419587819 shutdown_connections 2024-03-11T19:09:56.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:56.956+0000 7fecde9a7640 1 -- 172.21.15.29:0/3419587819 wait complete. 2024-03-11T19:09:56.970 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:03:09.860019+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T19:07:47.563184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T19:07:48.282461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T19:09:56.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:09:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.073+0000 7f5a45064640 1 -- 172.21.15.29:0/3919623820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a401005c0 msgr2=0x7f5a401009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.073+0000 7f5a45064640 1 --2- 172.21.15.29:0/3919623820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a401005c0 0x7f5a401009e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5a30009900 tx=0x7f5a3002bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.073+0000 7f5a45064640 1 -- 172.21.15.29:0/3919623820 shutdown_connections 2024-03-11T19:09:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.073+0000 7f5a45064640 1 --2- 172.21.15.29:0/3919623820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a40101c90 0x7f5a40108520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.073+0000 7f5a45064640 1 --2- 172.21.15.29:0/3919623820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a401012d0 0x7f5a40101750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.073+0000 7f5a45064640 1 --2- 172.21.15.29:0/3919623820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a401005c0 0x7f5a401009e0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.073+0000 7f5a45064640 1 -- 172.21.15.29:0/3919623820 >> 172.21.15.29:0/3919623820 conn(0x7f5a400fadb0 msgr2=0x7f5a400fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:57.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.073+0000 7f5a45064640 1 -- 172.21.15.29:0/3919623820 shutdown_connections 2024-03-11T19:09:57.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.073+0000 7f5a45064640 1 -- 172.21.15.29:0/3919623820 wait complete. 2024-03-11T19:09:57.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.074+0000 7f5a45064640 1 Processor -- start 2024-03-11T19:09:57.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.074+0000 7f5a45064640 1 -- start start 2024-03-11T19:09:57.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.074+0000 7f5a45064640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a401005c0 0x7f5a400ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:57.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.074+0000 7f5a45064640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a401012d0 0x7f5a4019bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:57.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.074+0000 7f5a45064640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a40101c90 0x7f5a4019e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:57.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.074+0000 7f5a45064640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4019ecb0 con 0x7f5a401012d0 2024-03-11T19:09:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.074+0000 7f5a45064640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4019ee20 con 0x7f5a40101c90 2024-03-11T19:09:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.074+0000 7f5a45064640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4019efe0 con 0x7f5a401005c0 2024-03-11T19:09:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.074+0000 7f5a3659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a401012d0 0x7f5a4019bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.074+0000 7f5a3659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a401012d0 0x7f5a4019bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54302/0 (socket says 172.21.15.29:54302) 2024-03-11T19:09:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.074+0000 7f5a3659c640 1 -- 172.21.15.29:0/1752244468 learned_addr learned my addr 172.21.15.29:0/1752244468 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.074+0000 7f5a3f59e640 1 --2- 172.21.15.29:0/1752244468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a40101c90 0x7f5a4019e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.074+0000 7f5a3659c640 1 -- 172.21.15.29:0/1752244468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a401005c0 msgr2=0x7f5a400ffca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.074+0000 7f5a3ed9d640 1 --2- 172.21.15.29:0/1752244468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a401005c0 0x7f5a400ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.075+0000 7f5a3659c640 1 --2- 172.21.15.29:0/1752244468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a401005c0 0x7f5a400ffca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.075+0000 7f5a3659c640 1 -- 172.21.15.29:0/1752244468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a40101c90 msgr2=0x7f5a4019e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.075+0000 7f5a3659c640 1 --2- 172.21.15.29:0/1752244468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a40101c90 0x7f5a4019e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.075+0000 7f5a3659c640 1 -- 172.21.15.29:0/1752244468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a30009560 con 0x7f5a401012d0 2024-03-11T19:09:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.075+0000 7f5a3ed9d640 1 --2- 172.21.15.29:0/1752244468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a401005c0 0x7f5a400ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.075+0000 7f5a3f59e640 1 --2- 172.21.15.29:0/1752244468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a40101c90 0x7f5a4019e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.075+0000 7f5a3659c640 1 --2- 172.21.15.29:0/1752244468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a401012d0 0x7f5a4019bee0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800d770 tx=0x7f5a2800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:57.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.075+0000 7f5a3cd99640 1 -- 172.21.15.29:0/1752244468 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28009c30 con 0x7f5a401012d0 2024-03-11T19:09:57.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.075+0000 7f5a3cd99640 1 -- 172.21.15.29:0/1752244468 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5a2800de80 con 0x7f5a401012d0 2024-03-11T19:09:57.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.075+0000 7f5a45064640 1 -- 172.21.15.29:0/1752244468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a4019f2c0 con 0x7f5a401012d0 2024-03-11T19:09:57.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.075+0000 7f5a3cd99640 1 -- 172.21.15.29:0/1752244468 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28004e00 con 0x7f5a401012d0 2024-03-11T19:09:57.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.075+0000 7f5a45064640 1 -- 172.21.15.29:0/1752244468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4019f780 con 0x7f5a401012d0 2024-03-11T19:09:57.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.076+0000 7f5a45064640 1 -- 172.21.15.29:0/1752244468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a401074a0 con 0x7f5a401012d0 2024-03-11T19:09:57.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.077+0000 7f5a3cd99640 1 -- 172.21.15.29:0/1752244468 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5a28004f60 con 0x7f5a401012d0 2024-03-11T19:09:57.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.078+0000 7f5a3cd99640 1 --2- 172.21.15.29:0/1752244468 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5a2c0a1a30 0x7f5a2c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:57.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.078+0000 7f5a3ed9d640 1 --2- 172.21.15.29:0/1752244468 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5a2c0a1a30 0x7f5a2c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:57.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.078+0000 7f5a3cd99640 1 -- 172.21.15.29:0/1752244468 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5a280ccea0 con 0x7f5a401012d0 2024-03-11T19:09:57.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.078+0000 7f5a3ed9d640 1 --2- 172.21.15.29:0/1752244468 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5a2c0a1a30 0x7f5a2c0a3ef0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5a30036040 tx=0x7f5a30000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:57.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.082+0000 7f5a3cd99640 1 -- 172.21.15.29:0/1752244468 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5a28096150 con 0x7f5a401012d0 2024-03-11T19:09:57.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.202+0000 7f5a3cd99640 1 -- 172.21.15.29:0/1752244468 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f5a28096520 con 0x7f5a401012d0 2024-03-11T19:09:57.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.268+0000 7f5a45064640 1 -- 172.21.15.29:0/1752244468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5a4010bb40 con 0x7f5a401012d0 2024-03-11T19:09:57.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.269+0000 7f5a3cd99640 1 -- 172.21.15.29:0/1752244468 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5a2805ec20 con 0x7f5a401012d0 2024-03-11T19:09:57.272 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-11T19:09:57.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.271+0000 7f5a45064640 1 -- 172.21.15.29:0/1752244468 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5a2c0a1a30 msgr2=0x7f5a2c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:57.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.271+0000 7f5a45064640 1 --2- 172.21.15.29:0/1752244468 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5a2c0a1a30 0x7f5a2c0a3ef0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5a30036040 tx=0x7f5a30000bc0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.271+0000 7f5a45064640 1 -- 172.21.15.29:0/1752244468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a401012d0 msgr2=0x7f5a4019bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:57.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.271+0000 7f5a45064640 1 --2- 172.21.15.29:0/1752244468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a401012d0 0x7f5a4019bee0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800d770 tx=0x7f5a2800dc30 comp rx=0 tx=0).stop 2024-03-11T19:09:57.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.271+0000 7f5a45064640 1 -- 172.21.15.29:0/1752244468 shutdown_connections 2024-03-11T19:09:57.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.271+0000 7f5a45064640 1 --2- 172.21.15.29:0/1752244468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a40101c90 0x7f5a4019e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.271+0000 7f5a45064640 1 --2- 172.21.15.29:0/1752244468 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5a2c0a1a30 0x7f5a2c0a3ef0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.271+0000 7f5a45064640 1 --2- 172.21.15.29:0/1752244468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a401012d0 0x7f5a4019bee0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.271+0000 7f5a45064640 1 --2- 172.21.15.29:0/1752244468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a401005c0 0x7f5a400ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.271+0000 7f5a45064640 1 -- 172.21.15.29:0/1752244468 >> 172.21.15.29:0/1752244468 conn(0x7f5a400fadb0 msgr2=0x7f5a40104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:57.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.271+0000 7f5a45064640 1 -- 172.21.15.29:0/1752244468 shutdown_connections 2024-03-11T19:09:57.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.271+0000 7f5a45064640 1 -- 172.21.15.29:0/1752244468 wait complete. 2024-03-11T19:09:57.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:57 smithi029 ceph-mon[32439]: pgmap v355: 97 pgs: 97 active+clean; 645 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-11T19:09:57.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:57 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1741975988' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-11T19:09:57.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:57 smithi029 ceph-mon[32439]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-11T19:09:57.286 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:57 smithi029 ceph-mon[32439]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:09:57.286 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:57 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3419587819' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:57.287 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T19:09:57.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.389+0000 7f9a653d0640 1 -- 172.21.15.29:0/194377131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a60101190 msgr2=0x7f9a601015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.389+0000 7f9a653d0640 1 --2- 172.21.15.29:0/194377131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a60101190 0x7f9a601015b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9a50009900 tx=0x7f9a5002bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.389+0000 7f9a653d0640 1 -- 172.21.15.29:0/194377131 shutdown_connections 2024-03-11T19:09:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.389+0000 7f9a653d0640 1 --2- 172.21.15.29:0/194377131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a60102d40 0x7f9a601095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.389+0000 7f9a653d0640 1 --2- 172.21.15.29:0/194377131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a60102380 0x7f9a60102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.389+0000 7f9a653d0640 1 --2- 172.21.15.29:0/194377131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a60101190 0x7f9a601015b0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.389+0000 7f9a653d0640 1 -- 172.21.15.29:0/194377131 >> 172.21.15.29:0/194377131 conn(0x7f9a600fc720 msgr2=0x7f9a600feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.389+0000 7f9a653d0640 1 -- 172.21.15.29:0/194377131 shutdown_connections 2024-03-11T19:09:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.389+0000 7f9a653d0640 1 -- 172.21.15.29:0/194377131 wait complete. 2024-03-11T19:09:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.390+0000 7f9a653d0640 1 Processor -- start 2024-03-11T19:09:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.390+0000 7f9a653d0640 1 -- start start 2024-03-11T19:09:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.390+0000 7f9a653d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a60101190 0x7f9a60074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.390+0000 7f9a653d0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a60102380 0x7f9a60073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.390+0000 7f9a653d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a60102d40 0x7f9a60073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.390+0000 7f9a653d0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a60074210 con 0x7f9a60102d40 2024-03-11T19:09:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.390+0000 7f9a653d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a60074380 con 0x7f9a60101190 2024-03-11T19:09:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.390+0000 7f9a653d0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a601a13a0 con 0x7f9a60102380 2024-03-11T19:09:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.390+0000 7f9a5f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a60102d40 0x7f9a60073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.391+0000 7f9a5f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a60102d40 0x7f9a60073ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54328/0 (socket says 172.21.15.29:54328) 2024-03-11T19:09:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.391+0000 7f9a5f7fe640 1 -- 172.21.15.29:0/1876325168 learned_addr learned my addr 172.21.15.29:0/1876325168 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.391+0000 7f9a5effd640 1 --2- 172.21.15.29:0/1876325168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a60101190 0x7f9a60074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.391+0000 7f9a5e7fc640 1 --2- 172.21.15.29:0/1876325168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a60102380 0x7f9a60073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.391+0000 7f9a5f7fe640 1 -- 172.21.15.29:0/1876325168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a60102380 msgr2=0x7f9a60073570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.391+0000 7f9a5f7fe640 1 --2- 172.21.15.29:0/1876325168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a60102380 0x7f9a60073570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.391+0000 7f9a5f7fe640 1 -- 172.21.15.29:0/1876325168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a60101190 msgr2=0x7f9a60074f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.391+0000 7f9a5f7fe640 1 --2- 172.21.15.29:0/1876325168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a60101190 0x7f9a60074f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.391+0000 7f9a5f7fe640 1 -- 172.21.15.29:0/1876325168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a50009560 con 0x7f9a60102d40 2024-03-11T19:09:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.391+0000 7f9a5f7fe640 1 --2- 172.21.15.29:0/1876325168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a60102d40 0x7f9a60073ab0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800b9a0 tx=0x7f9a5800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.391+0000 7f9a47fff640 1 -- 172.21.15.29:0/1876325168 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5800cba0 con 0x7f9a60102d40 2024-03-11T19:09:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.391+0000 7f9a653d0640 1 -- 172.21.15.29:0/1876325168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a601a1650 con 0x7f9a60102d40 2024-03-11T19:09:57.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.391+0000 7f9a47fff640 1 -- 172.21.15.29:0/1876325168 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9a5800cd00 con 0x7f9a60102d40 2024-03-11T19:09:57.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.391+0000 7f9a47fff640 1 -- 172.21.15.29:0/1876325168 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58007660 con 0x7f9a60102d40 2024-03-11T19:09:57.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.391+0000 7f9a653d0640 1 -- 172.21.15.29:0/1876325168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a601a1be0 con 0x7f9a60102d40 2024-03-11T19:09:57.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.392+0000 7f9a653d0640 1 -- 172.21.15.29:0/1876325168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a601085c0 con 0x7f9a60102d40 2024-03-11T19:09:57.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.393+0000 7f9a47fff640 1 -- 172.21.15.29:0/1876325168 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f9a58014440 con 0x7f9a60102d40 2024-03-11T19:09:57.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.394+0000 7f9a47fff640 1 --2- 172.21.15.29:0/1876325168 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a480a1ac0 0x7f9a480a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:57.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.394+0000 7f9a5effd640 1 --2- 172.21.15.29:0/1876325168 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a480a1ac0 0x7f9a480a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:57.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.394+0000 7f9a47fff640 1 -- 172.21.15.29:0/1876325168 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9a580ccd60 con 0x7f9a60102d40 2024-03-11T19:09:57.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.394+0000 7f9a5effd640 1 --2- 172.21.15.29:0/1876325168 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a480a1ac0 0x7f9a480a3f80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9a5002c7d0 tx=0x7f9a50005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:57.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.399+0000 7f9a47fff640 1 -- 172.21.15.29:0/1876325168 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9a58095fd0 con 0x7f9a60102d40 2024-03-11T19:09:57.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:57 smithi190 ceph-mon[36292]: pgmap v355: 97 pgs: 97 active+clean; 645 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-11T19:09:57.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:57 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1741975988' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-11T19:09:57.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:57 smithi190 ceph-mon[36292]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-11T19:09:57.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:57 smithi190 ceph-mon[36292]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:09:57.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:57 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3419587819' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:57.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.583+0000 7f9a653d0640 1 -- 172.21.15.29:0/1876325168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9a60108030 con 0x7f9a60102d40 2024-03-11T19:09:57.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.583+0000 7f9a47fff640 1 -- 172.21.15.29:0/1876325168 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9a58099690 con 0x7f9a60102d40 2024-03-11T19:09:57.586 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-03-11T19:09:57.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.585+0000 7f9a653d0640 1 -- 172.21.15.29:0/1876325168 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a480a1ac0 msgr2=0x7f9a480a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:57.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.585+0000 7f9a653d0640 1 --2- 172.21.15.29:0/1876325168 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a480a1ac0 0x7f9a480a3f80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9a5002c7d0 tx=0x7f9a50005fd0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.585+0000 7f9a653d0640 1 -- 172.21.15.29:0/1876325168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a60102d40 msgr2=0x7f9a60073ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:57.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.585+0000 7f9a653d0640 1 --2- 172.21.15.29:0/1876325168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a60102d40 0x7f9a60073ab0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800b9a0 tx=0x7f9a5800be60 comp rx=0 tx=0).stop 2024-03-11T19:09:57.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.585+0000 7f9a653d0640 1 -- 172.21.15.29:0/1876325168 shutdown_connections 2024-03-11T19:09:57.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.585+0000 7f9a653d0640 1 --2- 172.21.15.29:0/1876325168 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a480a1ac0 0x7f9a480a3f80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.585+0000 7f9a653d0640 1 --2- 172.21.15.29:0/1876325168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a60102d40 0x7f9a60073ab0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.585+0000 7f9a653d0640 1 --2- 172.21.15.29:0/1876325168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a60102380 0x7f9a60073570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.585+0000 7f9a653d0640 1 --2- 172.21.15.29:0/1876325168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a60101190 0x7f9a60074f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.585+0000 7f9a653d0640 1 -- 172.21.15.29:0/1876325168 >> 172.21.15.29:0/1876325168 conn(0x7f9a600fc720 msgr2=0x7f9a60105790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:57.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.585+0000 7f9a653d0640 1 -- 172.21.15.29:0/1876325168 shutdown_connections 2024-03-11T19:09:57.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.585+0000 7f9a653d0640 1 -- 172.21.15.29:0/1876325168 wait complete. 2024-03-11T19:09:57.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:57 smithi170 ceph-mon[35690]: pgmap v355: 97 pgs: 97 active+clean; 645 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-11T19:09:57.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:57 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1741975988' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-11T19:09:57.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:57 smithi170 ceph-mon[35690]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-11T19:09:57.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:57 smithi170 ceph-mon[35690]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:09:57.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:57 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3419587819' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:57.599 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T19:09:57.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.702+0000 7f0f5bbb5640 1 -- 172.21.15.29:0/29072839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f54101c30 msgr2=0x7f0f541084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:57.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.702+0000 7f0f5bbb5640 1 --2- 172.21.15.29:0/29072839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f54101c30 0x7f0f541084c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c009ca0 tx=0x7f0f4c02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.702+0000 7f0f5bbb5640 1 -- 172.21.15.29:0/29072839 shutdown_connections 2024-03-11T19:09:57.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.702+0000 7f0f5bbb5640 1 --2- 172.21.15.29:0/29072839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f54101c30 0x7f0f541084c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.702+0000 7f0f5bbb5640 1 --2- 172.21.15.29:0/29072839 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f54101270 0x7f0f541016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.702+0000 7f0f5bbb5640 1 --2- 172.21.15.29:0/29072839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f54100080 0x7f0f541004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.702+0000 7f0f5bbb5640 1 -- 172.21.15.29:0/29072839 >> 172.21.15.29:0/29072839 conn(0x7f0f540fb610 msgr2=0x7f0f540fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:57.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.702+0000 7f0f5bbb5640 1 -- 172.21.15.29:0/29072839 shutdown_connections 2024-03-11T19:09:57.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.702+0000 7f0f5bbb5640 1 -- 172.21.15.29:0/29072839 wait complete. 2024-03-11T19:09:57.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f5bbb5640 1 Processor -- start 2024-03-11T19:09:57.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f5bbb5640 1 -- start start 2024-03-11T19:09:57.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f5bbb5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f54100080 0x7f0f54196030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:57.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f5bbb5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f54101270 0x7f0f54196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:57.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f5bbb5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f54101c30 0x7f0f5419d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:57.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f5bbb5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5419db20 con 0x7f0f54101270 2024-03-11T19:09:57.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f5bbb5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5419dc90 con 0x7f0f54100080 2024-03-11T19:09:57.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f5bbb5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5419de50 con 0x7f0f54101c30 2024-03-11T19:09:57.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f59150640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f54101270 0x7f0f54196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:57.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f59150640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f54101270 0x7f0f54196570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54348/0 (socket says 172.21.15.29:54348) 2024-03-11T19:09:57.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f59150640 1 -- 172.21.15.29:0/218383455 learned_addr learned my addr 172.21.15.29:0/218383455 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:57.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f59951640 1 --2- 172.21.15.29:0/218383455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f54100080 0x7f0f54196030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:57.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f5a152640 1 --2- 172.21.15.29:0/218383455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f54101c30 0x7f0f5419d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:57.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f59150640 1 -- 172.21.15.29:0/218383455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f54101c30 msgr2=0x7f0f5419d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:57.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f59150640 1 --2- 172.21.15.29:0/218383455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f54101c30 0x7f0f5419d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f59150640 1 -- 172.21.15.29:0/218383455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f54100080 msgr2=0x7f0f54196030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:57.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f59150640 1 --2- 172.21.15.29:0/218383455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f54100080 0x7f0f54196030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f59150640 1 -- 172.21.15.29:0/218383455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4c009900 con 0x7f0f54101270 2024-03-11T19:09:57.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f5a152640 1 --2- 172.21.15.29:0/218383455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f54101c30 0x7f0f5419d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:57.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.703+0000 7f0f59150640 1 --2- 172.21.15.29:0/218383455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f54101270 0x7f0f54196570 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0f4400d950 tx=0x7f0f4400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:57.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.704+0000 7f0f4affd640 1 -- 172.21.15.29:0/218383455 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f44009c30 con 0x7f0f54101270 2024-03-11T19:09:57.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.704+0000 7f0f4affd640 1 -- 172.21.15.29:0/218383455 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0f44009d90 con 0x7f0f54101270 2024-03-11T19:09:57.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.704+0000 7f0f5bbb5640 1 -- 172.21.15.29:0/218383455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5419e130 con 0x7f0f54101270 2024-03-11T19:09:57.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.704+0000 7f0f4affd640 1 -- 172.21.15.29:0/218383455 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4400f6b0 con 0x7f0f54101270 2024-03-11T19:09:57.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.704+0000 7f0f5bbb5640 1 -- 172.21.15.29:0/218383455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5419e690 con 0x7f0f54101270 2024-03-11T19:09:57.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.705+0000 7f0f5bbb5640 1 -- 172.21.15.29:0/218383455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f541073c0 con 0x7f0f54101270 2024-03-11T19:09:57.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.706+0000 7f0f4affd640 1 -- 172.21.15.29:0/218383455 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f0f44010aa0 con 0x7f0f54101270 2024-03-11T19:09:57.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.706+0000 7f0f4affd640 1 --2- 172.21.15.29:0/218383455 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0f400a1b40 0x7f0f400a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:57.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.706+0000 7f0f59951640 1 --2- 172.21.15.29:0/218383455 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0f400a1b40 0x7f0f400a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:57.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.706+0000 7f0f4affd640 1 -- 172.21.15.29:0/218383455 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0f440cd0c0 con 0x7f0f54101270 2024-03-11T19:09:57.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.707+0000 7f0f59951640 1 --2- 172.21.15.29:0/218383455 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0f400a1b40 0x7f0f400a4000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0f541010d0 tx=0x7f0f5000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:57.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.711+0000 7f0f4affd640 1 -- 172.21.15.29:0/218383455 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0f44096270 con 0x7f0f54101270 2024-03-11T19:09:57.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.897+0000 7f0f5bbb5640 1 -- 172.21.15.29:0/218383455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0f5402ce40 con 0x7f0f54101270 2024-03-11T19:09:57.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.897+0000 7f0f4affd640 1 -- 172.21.15.29:0/218383455 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0f44099930 con 0x7f0f54101270 2024-03-11T19:09:57.900 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-03-11T19:09:57.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.899+0000 7f0f5bbb5640 1 -- 172.21.15.29:0/218383455 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0f400a1b40 msgr2=0x7f0f400a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:57.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.899+0000 7f0f5bbb5640 1 --2- 172.21.15.29:0/218383455 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0f400a1b40 0x7f0f400a4000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0f541010d0 tx=0x7f0f5000a2d0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.899+0000 7f0f5bbb5640 1 -- 172.21.15.29:0/218383455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f54101270 msgr2=0x7f0f54196570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:57.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.899+0000 7f0f5bbb5640 1 --2- 172.21.15.29:0/218383455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f54101270 0x7f0f54196570 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0f4400d950 tx=0x7f0f4400de10 comp rx=0 tx=0).stop 2024-03-11T19:09:57.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.899+0000 7f0f5bbb5640 1 -- 172.21.15.29:0/218383455 shutdown_connections 2024-03-11T19:09:57.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.899+0000 7f0f5bbb5640 1 --2- 172.21.15.29:0/218383455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f54101c30 0x7f0f5419d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.899+0000 7f0f5bbb5640 1 --2- 172.21.15.29:0/218383455 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0f400a1b40 0x7f0f400a4000 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.899+0000 7f0f5bbb5640 1 --2- 172.21.15.29:0/218383455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f54101270 0x7f0f54196570 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.899+0000 7f0f5bbb5640 1 --2- 172.21.15.29:0/218383455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f54100080 0x7f0f54196030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:57.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.899+0000 7f0f5bbb5640 1 -- 172.21.15.29:0/218383455 >> 172.21.15.29:0/218383455 conn(0x7f0f540fb610 msgr2=0x7f0f54104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:57.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.899+0000 7f0f5bbb5640 1 -- 172.21.15.29:0/218383455 shutdown_connections 2024-03-11T19:09:57.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:57.899+0000 7f0f5bbb5640 1 -- 172.21.15.29:0/218383455 wait complete. 2024-03-11T19:09:57.913 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:09:58.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.013+0000 7f2aecd77640 1 -- 172.21.15.29:0/2757124082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ae8100120 msgr2=0x7f2ae8100540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:58.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.013+0000 7f2aecd77640 1 --2- 172.21.15.29:0/2757124082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ae8100120 0x7f2ae8100540 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8009900 tx=0x7f2ad802bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:58.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.013+0000 7f2aecd77640 1 -- 172.21.15.29:0/2757124082 shutdown_connections 2024-03-11T19:09:58.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.013+0000 7f2aecd77640 1 --2- 172.21.15.29:0/2757124082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ae8101cd0 0x7f2ae8108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.013+0000 7f2aecd77640 1 --2- 172.21.15.29:0/2757124082 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ae8101310 0x7f2ae8101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.013+0000 7f2aecd77640 1 --2- 172.21.15.29:0/2757124082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ae8100120 0x7f2ae8100540 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.013+0000 7f2aecd77640 1 -- 172.21.15.29:0/2757124082 >> 172.21.15.29:0/2757124082 conn(0x7f2ae80fb6b0 msgr2=0x7f2ae80fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:58.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.013+0000 7f2aecd77640 1 -- 172.21.15.29:0/2757124082 shutdown_connections 2024-03-11T19:09:58.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.013+0000 7f2aecd77640 1 -- 172.21.15.29:0/2757124082 wait complete. 2024-03-11T19:09:58.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.014+0000 7f2aecd77640 1 Processor -- start 2024-03-11T19:09:58.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.014+0000 7f2aecd77640 1 -- start start 2024-03-11T19:09:58.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.014+0000 7f2aecd77640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ae8100120 0x7f2ae81971d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:58.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.014+0000 7f2aecd77640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ae8101310 0x7f2ae8197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:58.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.014+0000 7f2aecd77640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ae8101cd0 0x7f2ae819e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:58.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.014+0000 7f2aecd77640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae819ecc0 con 0x7f2ae8100120 2024-03-11T19:09:58.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.014+0000 7f2aecd77640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae819ee30 con 0x7f2ae8101cd0 2024-03-11T19:09:58.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.014+0000 7f2aecd77640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae819eff0 con 0x7f2ae8101310 2024-03-11T19:09:58.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.014+0000 7f2ae659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ae8100120 0x7f2ae81971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:58.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.014+0000 7f2ae659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ae8100120 0x7f2ae81971d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54360/0 (socket says 172.21.15.29:54360) 2024-03-11T19:09:58.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.014+0000 7f2ae659c640 1 -- 172.21.15.29:0/4246037696 learned_addr learned my addr 172.21.15.29:0/4246037696 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:58.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.014+0000 7f2addd9b640 1 --2- 172.21.15.29:0/4246037696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ae8101310 0x7f2ae8197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:58.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.014+0000 7f2ae6d9d640 1 --2- 172.21.15.29:0/4246037696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ae8101cd0 0x7f2ae819e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:58.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.014+0000 7f2ae659c640 1 -- 172.21.15.29:0/4246037696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ae8101310 msgr2=0x7f2ae8197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:58.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.015+0000 7f2ae659c640 1 --2- 172.21.15.29:0/4246037696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ae8101310 0x7f2ae8197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.015+0000 7f2ae659c640 1 -- 172.21.15.29:0/4246037696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ae8101cd0 msgr2=0x7f2ae819e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:58.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.015+0000 7f2ae659c640 1 --2- 172.21.15.29:0/4246037696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ae8101cd0 0x7f2ae819e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.015+0000 7f2ae659c640 1 -- 172.21.15.29:0/4246037696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad8009560 con 0x7f2ae8100120 2024-03-11T19:09:58.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.015+0000 7f2ae6d9d640 1 --2- 172.21.15.29:0/4246037696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ae8101cd0 0x7f2ae819e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:58.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.015+0000 7f2addd9b640 1 --2- 172.21.15.29:0/4246037696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ae8101310 0x7f2ae8197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:58.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.015+0000 7f2ae659c640 1 --2- 172.21.15.29:0/4246037696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ae8100120 0x7f2ae81971d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8000c00 tx=0x7f2ad80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:58.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.015+0000 7f2adffff640 1 -- 172.21.15.29:0/4246037696 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad80044e0 con 0x7f2ae8100120 2024-03-11T19:09:58.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.015+0000 7f2aecd77640 1 -- 172.21.15.29:0/4246037696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae819f270 con 0x7f2ae8100120 2024-03-11T19:09:58.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.015+0000 7f2adffff640 1 -- 172.21.15.29:0/4246037696 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2ad802de90 con 0x7f2ae8100120 2024-03-11T19:09:58.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.015+0000 7f2adffff640 1 -- 172.21.15.29:0/4246037696 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8005410 con 0x7f2ae8100120 2024-03-11T19:09:58.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.015+0000 7f2aecd77640 1 -- 172.21.15.29:0/4246037696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae819f780 con 0x7f2ae8100120 2024-03-11T19:09:58.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.016+0000 7f2aecd77640 1 -- 172.21.15.29:0/4246037696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae810b970 con 0x7f2ae8100120 2024-03-11T19:09:58.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.017+0000 7f2adffff640 1 -- 172.21.15.29:0/4246037696 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f2ad8005570 con 0x7f2ae8100120 2024-03-11T19:09:58.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.017+0000 7f2adffff640 1 --2- 172.21.15.29:0/4246037696 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2ad40a1ac0 0x7f2ad40a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:58.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.018+0000 7f2addd9b640 1 --2- 172.21.15.29:0/4246037696 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2ad40a1ac0 0x7f2ad40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:58.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.018+0000 7f2adffff640 1 -- 172.21.15.29:0/4246037696 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f2ad80ef680 con 0x7f2ae8100120 2024-03-11T19:09:58.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.018+0000 7f2addd9b640 1 --2- 172.21.15.29:0/4246037696 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2ad40a1ac0 0x7f2ad40a3f80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2ad00058c0 tx=0x7f2ad0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:58.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.022+0000 7f2adffff640 1 -- 172.21.15.29:0/4246037696 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2ad80b8940 con 0x7f2ae8100120 2024-03-11T19:09:58.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.203+0000 7f2aecd77640 1 -- 172.21.15.29:0/4246037696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ae810bb50 con 0x7f2ae8100120 2024-03-11T19:09:58.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.204+0000 7f2adffff640 1 -- 172.21.15.29:0/4246037696 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22559 (secure 0 0 0) 0x7f2ad80bc000 con 0x7f2ae8100120 2024-03-11T19:09:58.207 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:58.208 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":89,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:08:44.981806+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:07:47.563184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:07:48.282461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:09:58.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.206+0000 7f2aecd77640 1 -- 172.21.15.29:0/4246037696 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2ad40a1ac0 msgr2=0x7f2ad40a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:58.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.206+0000 7f2aecd77640 1 --2- 172.21.15.29:0/4246037696 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2ad40a1ac0 0x7f2ad40a3f80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2ad00058c0 tx=0x7f2ad0005850 comp rx=0 tx=0).stop 2024-03-11T19:09:58.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.206+0000 7f2aecd77640 1 -- 172.21.15.29:0/4246037696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ae8100120 msgr2=0x7f2ae81971d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:58.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.206+0000 7f2aecd77640 1 --2- 172.21.15.29:0/4246037696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ae8100120 0x7f2ae81971d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8000c00 tx=0x7f2ad80042f0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.206+0000 7f2aecd77640 1 -- 172.21.15.29:0/4246037696 shutdown_connections 2024-03-11T19:09:58.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.206+0000 7f2aecd77640 1 --2- 172.21.15.29:0/4246037696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ae8101cd0 0x7f2ae819e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.206+0000 7f2aecd77640 1 --2- 172.21.15.29:0/4246037696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ae8101310 0x7f2ae8197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.206+0000 7f2aecd77640 1 --2- 172.21.15.29:0/4246037696 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2ad40a1ac0 0x7f2ad40a3f80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.206+0000 7f2aecd77640 1 --2- 172.21.15.29:0/4246037696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ae8100120 0x7f2ae81971d0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.206+0000 7f2aecd77640 1 -- 172.21.15.29:0/4246037696 >> 172.21.15.29:0/4246037696 conn(0x7f2ae80fb6b0 msgr2=0x7f2ae80697b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:58.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.207+0000 7f2aecd77640 1 -- 172.21.15.29:0/4246037696 shutdown_connections 2024-03-11T19:09:58.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.207+0000 7f2aecd77640 1 -- 172.21.15.29:0/4246037696 wait complete. 2024-03-11T19:09:58.221 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:03:09.860019+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T19:07:47.563184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T19:07:48.282461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T19:09:58.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:09:58.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.325+0000 7f7898f02640 1 -- 172.21.15.29:0/665132712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78941000a0 msgr2=0x7f78941004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:58.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.325+0000 7f7898f02640 1 --2- 172.21.15.29:0/665132712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78941000a0 0x7f78941004c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7884009900 tx=0x7f788402bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:58.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.325+0000 7f7898f02640 1 -- 172.21.15.29:0/665132712 shutdown_connections 2024-03-11T19:09:58.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.325+0000 7f7898f02640 1 --2- 172.21.15.29:0/665132712 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7894101c50 0x7f78941084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.325+0000 7f7898f02640 1 --2- 172.21.15.29:0/665132712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7894101290 0x7f7894101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.325+0000 7f7898f02640 1 --2- 172.21.15.29:0/665132712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78941000a0 0x7f78941004c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.325+0000 7f7898f02640 1 -- 172.21.15.29:0/665132712 >> 172.21.15.29:0/665132712 conn(0x7f78940fb630 msgr2=0x7f78940fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:58.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.325+0000 7f7898f02640 1 -- 172.21.15.29:0/665132712 shutdown_connections 2024-03-11T19:09:58.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.325+0000 7f7898f02640 1 -- 172.21.15.29:0/665132712 wait complete. 2024-03-11T19:09:58.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.326+0000 7f7898f02640 1 Processor -- start 2024-03-11T19:09:58.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.326+0000 7f7898f02640 1 -- start start 2024-03-11T19:09:58.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.326+0000 7f7898f02640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78941000a0 0x7f78941971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:58.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.326+0000 7f7898f02640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7894101290 0x7f78941976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:58.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.326+0000 7f7898f02640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7894101c50 0x7f789419e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:58.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.326+0000 7f7898f02640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789419ec90 con 0x7f7894101c50 2024-03-11T19:09:58.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.326+0000 7f7898f02640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789419ee00 con 0x7f7894101290 2024-03-11T19:09:58.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.326+0000 7f7898f02640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789419efc0 con 0x7f78941000a0 2024-03-11T19:09:58.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.326+0000 7f7892d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7894101c50 0x7f789419e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:58.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.326+0000 7f7892d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7894101c50 0x7f789419e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54376/0 (socket says 172.21.15.29:54376) 2024-03-11T19:09:58.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.326+0000 7f7892d9d640 1 -- 172.21.15.29:0/3478030824 learned_addr learned my addr 172.21.15.29:0/3478030824 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:58.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.326+0000 7f788bfff640 1 --2- 172.21.15.29:0/3478030824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7894101290 0x7f78941976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:58.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.326+0000 7f789259c640 1 --2- 172.21.15.29:0/3478030824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78941000a0 0x7f78941971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:58.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.326+0000 7f7892d9d640 1 -- 172.21.15.29:0/3478030824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78941000a0 msgr2=0x7f78941971a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:58.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.326+0000 7f7892d9d640 1 --2- 172.21.15.29:0/3478030824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78941000a0 0x7f78941971a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.326+0000 7f7892d9d640 1 -- 172.21.15.29:0/3478030824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7894101290 msgr2=0x7f78941976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:58.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.326+0000 7f7892d9d640 1 --2- 172.21.15.29:0/3478030824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7894101290 0x7f78941976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.326+0000 7f7892d9d640 1 -- 172.21.15.29:0/3478030824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7884009560 con 0x7f7894101c50 2024-03-11T19:09:58.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.326+0000 7f7892d9d640 1 --2- 172.21.15.29:0/3478030824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7894101c50 0x7f789419e750 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f788c00eaa0 tx=0x7f788c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:58.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.327+0000 7f788b7fe640 1 -- 172.21.15.29:0/3478030824 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c00cdb0 con 0x7f7894101c50 2024-03-11T19:09:58.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.327+0000 7f7898f02640 1 -- 172.21.15.29:0/3478030824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f789419f2a0 con 0x7f7894101c50 2024-03-11T19:09:58.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.327+0000 7f788b7fe640 1 -- 172.21.15.29:0/3478030824 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f788c005020 con 0x7f7894101c50 2024-03-11T19:09:58.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.327+0000 7f788b7fe640 1 -- 172.21.15.29:0/3478030824 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c0116b0 con 0x7f7894101c50 2024-03-11T19:09:58.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.327+0000 7f7898f02640 1 -- 172.21.15.29:0/3478030824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f789419f760 con 0x7f7894101c50 2024-03-11T19:09:58.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.328+0000 7f7898f02640 1 -- 172.21.15.29:0/3478030824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f789410b960 con 0x7f7894101c50 2024-03-11T19:09:58.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.329+0000 7f788b7fe640 1 -- 172.21.15.29:0/3478030824 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f788c005190 con 0x7f7894101c50 2024-03-11T19:09:58.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.330+0000 7f788b7fe640 1 --2- 172.21.15.29:0/3478030824 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f78800a1ac0 0x7f78800a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:58.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.330+0000 7f789259c640 1 --2- 172.21.15.29:0/3478030824 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f78800a1ac0 0x7f78800a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:58.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.330+0000 7f788b7fe640 1 -- 172.21.15.29:0/3478030824 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f788c0cdac0 con 0x7f7894101c50 2024-03-11T19:09:58.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.330+0000 7f789259c640 1 --2- 172.21.15.29:0/3478030824 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f78800a1ac0 0x7f78800a3f80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f788402c7d0 tx=0x7f7884005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:58.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.334+0000 7f788b7fe640 1 -- 172.21.15.29:0/3478030824 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f788c096c20 con 0x7f7894101c50 2024-03-11T19:09:58.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:58 smithi190 ceph-mon[36292]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-11T19:09:58.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:58 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1752244468' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:09:58.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:58 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1876325168' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:09:58.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:58 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/218383455' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:09:58.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.514+0000 7f7898f02640 1 -- 172.21.15.29:0/3478030824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f789410baf0 con 0x7f7894101c50 2024-03-11T19:09:58.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.514+0000 7f788b7fe640 1 -- 172.21.15.29:0/3478030824 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f788c01f020 con 0x7f7894101c50 2024-03-11T19:09:58.517 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-11T19:09:58.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.516+0000 7f7898f02640 1 -- 172.21.15.29:0/3478030824 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f78800a1ac0 msgr2=0x7f78800a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:58.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.516+0000 7f7898f02640 1 --2- 172.21.15.29:0/3478030824 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f78800a1ac0 0x7f78800a3f80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f788402c7d0 tx=0x7f7884005fd0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.516+0000 7f7898f02640 1 -- 172.21.15.29:0/3478030824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7894101c50 msgr2=0x7f789419e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:58.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.516+0000 7f7898f02640 1 --2- 172.21.15.29:0/3478030824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7894101c50 0x7f789419e750 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f788c00eaa0 tx=0x7f788c00ef60 comp rx=0 tx=0).stop 2024-03-11T19:09:58.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.516+0000 7f7898f02640 1 -- 172.21.15.29:0/3478030824 shutdown_connections 2024-03-11T19:09:58.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.516+0000 7f7898f02640 1 --2- 172.21.15.29:0/3478030824 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f78800a1ac0 0x7f78800a3f80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.516+0000 7f7898f02640 1 --2- 172.21.15.29:0/3478030824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7894101c50 0x7f789419e750 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.516+0000 7f7898f02640 1 --2- 172.21.15.29:0/3478030824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7894101290 0x7f78941976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.516+0000 7f7898f02640 1 --2- 172.21.15.29:0/3478030824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78941000a0 0x7f78941971a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.516+0000 7f7898f02640 1 -- 172.21.15.29:0/3478030824 >> 172.21.15.29:0/3478030824 conn(0x7f78940fb630 msgr2=0x7f7894069780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:58.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.517+0000 7f7898f02640 1 -- 172.21.15.29:0/3478030824 shutdown_connections 2024-03-11T19:09:58.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.517+0000 7f7898f02640 1 -- 172.21.15.29:0/3478030824 wait complete. 2024-03-11T19:09:58.529 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:58 smithi029 ceph-mon[32439]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-11T19:09:58.530 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:58 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1752244468' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:09:58.530 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:58 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1876325168' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:09:58.530 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:58 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/218383455' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:09:58.531 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T19:09:58.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:58 smithi170 ceph-mon[35690]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-11T19:09:58.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:58 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1752244468' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:09:58.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:58 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1876325168' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:09:58.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:58 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/218383455' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:09:58.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.633+0000 7f4af34db640 1 -- 172.21.15.29:0/3821739724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4aec1045f0 msgr2=0x7f4aec1069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:58.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.633+0000 7f4af34db640 1 --2- 172.21.15.29:0/3821739724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4aec1045f0 0x7f4aec1069e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4adc009900 tx=0x7f4adc02bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:58.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.634+0000 7f4af34db640 1 -- 172.21.15.29:0/3821739724 shutdown_connections 2024-03-11T19:09:58.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.634+0000 7f4af34db640 1 --2- 172.21.15.29:0/3821739724 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4aec106f20 0x7f4aec109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.634+0000 7f4af34db640 1 --2- 172.21.15.29:0/3821739724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4aec1045f0 0x7f4aec1069e0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.634+0000 7f4af34db640 1 --2- 172.21.15.29:0/3821739724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4aec101cc0 0x7f4aec1040b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.634+0000 7f4af34db640 1 -- 172.21.15.29:0/3821739724 >> 172.21.15.29:0/3821739724 conn(0x7f4aec0fb630 msgr2=0x7f4aec0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:58.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.634+0000 7f4af34db640 1 -- 172.21.15.29:0/3821739724 shutdown_connections 2024-03-11T19:09:58.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.634+0000 7f4af34db640 1 -- 172.21.15.29:0/3821739724 wait complete. 2024-03-11T19:09:58.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.634+0000 7f4af34db640 1 Processor -- start 2024-03-11T19:09:58.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.634+0000 7f4af34db640 1 -- start start 2024-03-11T19:09:58.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.634+0000 7f4af34db640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4aec101cc0 0x7f4aec197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:58.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.634+0000 7f4af34db640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4aec1045f0 0x7f4aec197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:58.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.634+0000 7f4af34db640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4aec106f20 0x7f4aec19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:58.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.634+0000 7f4af34db640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec19ec10 con 0x7f4aec101cc0 2024-03-11T19:09:58.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.634+0000 7f4af34db640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec19ed80 con 0x7f4aec106f20 2024-03-11T19:09:58.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.634+0000 7f4af34db640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec19ef40 con 0x7f4aec1045f0 2024-03-11T19:09:58.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.634+0000 7f4af1277640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4aec101cc0 0x7f4aec197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:58.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.634+0000 7f4af1277640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4aec101cc0 0x7f4aec197120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54394/0 (socket says 172.21.15.29:54394) 2024-03-11T19:09:58.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.634+0000 7f4af1277640 1 -- 172.21.15.29:0/2500739399 learned_addr learned my addr 172.21.15.29:0/2500739399 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:58.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.635+0000 7f4af0a76640 1 --2- 172.21.15.29:0/2500739399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4aec1045f0 0x7f4aec197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:58.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.635+0000 7f4af1a78640 1 --2- 172.21.15.29:0/2500739399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4aec106f20 0x7f4aec19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:58.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.635+0000 7f4af1277640 1 -- 172.21.15.29:0/2500739399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4aec1045f0 msgr2=0x7f4aec197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:58.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.635+0000 7f4af1277640 1 --2- 172.21.15.29:0/2500739399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4aec1045f0 0x7f4aec197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.635+0000 7f4af1277640 1 -- 172.21.15.29:0/2500739399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4aec106f20 msgr2=0x7f4aec19e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:58.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.635+0000 7f4af1277640 1 --2- 172.21.15.29:0/2500739399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4aec106f20 0x7f4aec19e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.635+0000 7f4af1277640 1 -- 172.21.15.29:0/2500739399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4adc009560 con 0x7f4aec101cc0 2024-03-11T19:09:58.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.635+0000 7f4af1a78640 1 --2- 172.21.15.29:0/2500739399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4aec106f20 0x7f4aec19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:09:58.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.635+0000 7f4af1277640 1 --2- 172.21.15.29:0/2500739399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4aec101cc0 0x7f4aec197120 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4ae800c920 tx=0x7f4ae800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:58.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.635+0000 7f4ae27fc640 1 -- 172.21.15.29:0/2500739399 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8007c70 con 0x7f4aec101cc0 2024-03-11T19:09:58.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.635+0000 7f4ae27fc640 1 -- 172.21.15.29:0/2500739399 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4ae8007dd0 con 0x7f4aec101cc0 2024-03-11T19:09:58.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.635+0000 7f4af34db640 1 -- 172.21.15.29:0/2500739399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aec19f220 con 0x7f4aec101cc0 2024-03-11T19:09:58.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.635+0000 7f4ae27fc640 1 -- 172.21.15.29:0/2500739399 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae80187c0 con 0x7f4aec101cc0 2024-03-11T19:09:58.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.635+0000 7f4af34db640 1 -- 172.21.15.29:0/2500739399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aec19f750 con 0x7f4aec101cc0 2024-03-11T19:09:58.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.636+0000 7f4af34db640 1 -- 172.21.15.29:0/2500739399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aec0fd120 con 0x7f4aec101cc0 2024-03-11T19:09:58.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.637+0000 7f4ae27fc640 1 -- 172.21.15.29:0/2500739399 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f4ae800f450 con 0x7f4aec101cc0 2024-03-11T19:09:58.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.638+0000 7f4ae27fc640 1 --2- 172.21.15.29:0/2500739399 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4ad80a1b40 0x7f4ad80a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:58.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.638+0000 7f4af0a76640 1 --2- 172.21.15.29:0/2500739399 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4ad80a1b40 0x7f4ad80a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:58.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.638+0000 7f4ae27fc640 1 -- 172.21.15.29:0/2500739399 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4ae80cd570 con 0x7f4aec101cc0 2024-03-11T19:09:58.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.638+0000 7f4af0a76640 1 --2- 172.21.15.29:0/2500739399 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4ad80a1b40 0x7f4ad80a4000 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4adc02c7d0 tx=0x7f4adc005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:58.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.643+0000 7f4ae27fc640 1 -- 172.21.15.29:0/2500739399 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4ae8096720 con 0x7f4aec101cc0 2024-03-11T19:09:58.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.825+0000 7f4af34db640 1 -- 172.21.15.29:0/2500739399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4aec02ce40 con 0x7f4aec101cc0 2024-03-11T19:09:58.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.825+0000 7f4ae27fc640 1 -- 172.21.15.29:0/2500739399 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4ae8099de0 con 0x7f4aec101cc0 2024-03-11T19:09:58.829 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-03-11T19:09:58.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.828+0000 7f4af34db640 1 -- 172.21.15.29:0/2500739399 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4ad80a1b40 msgr2=0x7f4ad80a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:58.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.828+0000 7f4af34db640 1 --2- 172.21.15.29:0/2500739399 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4ad80a1b40 0x7f4ad80a4000 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4adc02c7d0 tx=0x7f4adc005fd0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.828+0000 7f4af34db640 1 -- 172.21.15.29:0/2500739399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4aec101cc0 msgr2=0x7f4aec197120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:58.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.828+0000 7f4af34db640 1 --2- 172.21.15.29:0/2500739399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4aec101cc0 0x7f4aec197120 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4ae800c920 tx=0x7f4ae800cde0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.828+0000 7f4af34db640 1 -- 172.21.15.29:0/2500739399 shutdown_connections 2024-03-11T19:09:58.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.828+0000 7f4af34db640 1 --2- 172.21.15.29:0/2500739399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4aec106f20 0x7f4aec19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.828+0000 7f4af34db640 1 --2- 172.21.15.29:0/2500739399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4aec1045f0 0x7f4aec197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.828+0000 7f4af34db640 1 --2- 172.21.15.29:0/2500739399 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4ad80a1b40 0x7f4ad80a4000 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.828+0000 7f4af34db640 1 --2- 172.21.15.29:0/2500739399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4aec101cc0 0x7f4aec197120 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.828+0000 7f4af34db640 1 -- 172.21.15.29:0/2500739399 >> 172.21.15.29:0/2500739399 conn(0x7f4aec0fb630 msgr2=0x7f4aec102780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:58.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.828+0000 7f4af34db640 1 -- 172.21.15.29:0/2500739399 shutdown_connections 2024-03-11T19:09:58.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.828+0000 7f4af34db640 1 -- 172.21.15.29:0/2500739399 wait complete. 2024-03-11T19:09:58.842 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T19:09:58.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.944+0000 7f4343172640 1 -- 172.21.15.29:0/2014162195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f433c101c10 msgr2=0x7f433c104000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:58.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.944+0000 7f4343172640 1 --2- 172.21.15.29:0/2014162195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f433c101c10 0x7f433c104000 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4330009900 tx=0x7f433002bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:58.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.944+0000 7f4343172640 1 -- 172.21.15.29:0/2014162195 shutdown_connections 2024-03-11T19:09:58.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.944+0000 7f4343172640 1 --2- 172.21.15.29:0/2014162195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f433c106e70 0x7f433c1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.944+0000 7f4343172640 1 --2- 172.21.15.29:0/2014162195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f433c104540 0x7f433c106930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.944+0000 7f4343172640 1 --2- 172.21.15.29:0/2014162195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f433c101c10 0x7f433c104000 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.944+0000 7f4343172640 1 -- 172.21.15.29:0/2014162195 >> 172.21.15.29:0/2014162195 conn(0x7f433c0fb620 msgr2=0x7f433c0fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:58.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.944+0000 7f4343172640 1 -- 172.21.15.29:0/2014162195 shutdown_connections 2024-03-11T19:09:58.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.944+0000 7f4343172640 1 -- 172.21.15.29:0/2014162195 wait complete. 2024-03-11T19:09:58.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.945+0000 7f4343172640 1 Processor -- start 2024-03-11T19:09:58.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.945+0000 7f4343172640 1 -- start start 2024-03-11T19:09:58.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.945+0000 7f4343172640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f433c101c10 0x7f433c101580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:58.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.945+0000 7f4343172640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f433c104540 0x7f433c0ffbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:58.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.945+0000 7f4343172640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f433c106e70 0x7f433c100100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:58.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.945+0000 7f4343172640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433c100860 con 0x7f433c106e70 2024-03-11T19:09:58.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.945+0000 7f4343172640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433c1009d0 con 0x7f433c104540 2024-03-11T19:09:58.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.945+0000 7f4343172640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433c1a11f0 con 0x7f433c101c10 2024-03-11T19:09:58.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.945+0000 7f434170f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f433c106e70 0x7f433c100100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:58.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.945+0000 7f434170f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f433c106e70 0x7f433c100100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54418/0 (socket says 172.21.15.29:54418) 2024-03-11T19:09:58.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.945+0000 7f434170f640 1 -- 172.21.15.29:0/3854345966 learned_addr learned my addr 172.21.15.29:0/3854345966 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:58.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.945+0000 7f433bfff640 1 --2- 172.21.15.29:0/3854345966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f433c104540 0x7f433c0ffbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:58.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.945+0000 7f4340f0e640 1 --2- 172.21.15.29:0/3854345966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f433c101c10 0x7f433c101580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:58.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.945+0000 7f434170f640 1 -- 172.21.15.29:0/3854345966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f433c101c10 msgr2=0x7f433c101580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:58.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.945+0000 7f434170f640 1 --2- 172.21.15.29:0/3854345966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f433c101c10 0x7f433c101580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.945+0000 7f434170f640 1 -- 172.21.15.29:0/3854345966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f433c104540 msgr2=0x7f433c0ffbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:58.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.945+0000 7f434170f640 1 --2- 172.21.15.29:0/3854345966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f433c104540 0x7f433c0ffbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:58.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.945+0000 7f434170f640 1 -- 172.21.15.29:0/3854345966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4330009560 con 0x7f433c106e70 2024-03-11T19:09:58.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.946+0000 7f434170f640 1 --2- 172.21.15.29:0/3854345966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f433c106e70 0x7f433c100100 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f433400ead0 tx=0x7f433400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:58.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.946+0000 7f4339ffb640 1 -- 172.21.15.29:0/3854345966 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433400cde0 con 0x7f433c106e70 2024-03-11T19:09:58.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.946+0000 7f4339ffb640 1 -- 172.21.15.29:0/3854345966 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4334005020 con 0x7f433c106e70 2024-03-11T19:09:58.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.946+0000 7f4343172640 1 -- 172.21.15.29:0/3854345966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f433c1a14a0 con 0x7f433c106e70 2024-03-11T19:09:58.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.946+0000 7f4339ffb640 1 -- 172.21.15.29:0/3854345966 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43340116b0 con 0x7f433c106e70 2024-03-11T19:09:58.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.946+0000 7f4343172640 1 -- 172.21.15.29:0/3854345966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f433c1a1960 con 0x7f433c106e70 2024-03-11T19:09:58.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.947+0000 7f4343172640 1 -- 172.21.15.29:0/3854345966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f433c0fd110 con 0x7f433c106e70 2024-03-11T19:09:58.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.948+0000 7f4339ffb640 1 -- 172.21.15.29:0/3854345966 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f4334005190 con 0x7f433c106e70 2024-03-11T19:09:58.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.948+0000 7f4339ffb640 1 --2- 172.21.15.29:0/3854345966 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f432c0a1ac0 0x7f432c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:58.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.948+0000 7f4339ffb640 1 -- 172.21.15.29:0/3854345966 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f43340cd5b0 con 0x7f433c106e70 2024-03-11T19:09:58.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.948+0000 7f4340f0e640 1 --2- 172.21.15.29:0/3854345966 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f432c0a1ac0 0x7f432c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:58.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.949+0000 7f4340f0e640 1 --2- 172.21.15.29:0/3854345966 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f432c0a1ac0 0x7f432c0a3f80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4330009230 tx=0x7f4330005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:58.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:58.953+0000 7f4339ffb640 1 -- 172.21.15.29:0/3854345966 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4334096810 con 0x7f433c106e70 2024-03-11T19:09:59.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.133+0000 7f4343172640 1 -- 172.21.15.29:0/3854345966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f433c02cda0 con 0x7f433c106e70 2024-03-11T19:09:59.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.133+0000 7f4339ffb640 1 -- 172.21.15.29:0/3854345966 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4334099ed0 con 0x7f433c106e70 2024-03-11T19:09:59.137 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-03-11T19:09:59.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.135+0000 7f4343172640 1 -- 172.21.15.29:0/3854345966 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f432c0a1ac0 msgr2=0x7f432c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:59.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.135+0000 7f4343172640 1 --2- 172.21.15.29:0/3854345966 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f432c0a1ac0 0x7f432c0a3f80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4330009230 tx=0x7f4330005fd0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.135+0000 7f4343172640 1 -- 172.21.15.29:0/3854345966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f433c106e70 msgr2=0x7f433c100100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:59.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.135+0000 7f4343172640 1 --2- 172.21.15.29:0/3854345966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f433c106e70 0x7f433c100100 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f433400ead0 tx=0x7f433400ef90 comp rx=0 tx=0).stop 2024-03-11T19:09:59.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.135+0000 7f4343172640 1 -- 172.21.15.29:0/3854345966 shutdown_connections 2024-03-11T19:09:59.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.135+0000 7f4343172640 1 --2- 172.21.15.29:0/3854345966 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f432c0a1ac0 0x7f432c0a3f80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.135+0000 7f4343172640 1 --2- 172.21.15.29:0/3854345966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f433c106e70 0x7f433c100100 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.135+0000 7f4343172640 1 --2- 172.21.15.29:0/3854345966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f433c104540 0x7f433c0ffbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.135+0000 7f4343172640 1 --2- 172.21.15.29:0/3854345966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f433c101c10 0x7f433c101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.135+0000 7f4343172640 1 -- 172.21.15.29:0/3854345966 >> 172.21.15.29:0/3854345966 conn(0x7f433c0fb620 msgr2=0x7f433c102710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:59.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.136+0000 7f4343172640 1 -- 172.21.15.29:0/3854345966 shutdown_connections 2024-03-11T19:09:59.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.136+0000 7f4343172640 1 -- 172.21.15.29:0/3854345966 wait complete. 2024-03-11T19:09:59.150 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-11T19:09:59.151 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:09:59.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.250+0000 7f277c7f4640 1 -- 172.21.15.29:0/2286918979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27741045b0 msgr2=0x7f27741069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:59.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.250+0000 7f277c7f4640 1 --2- 172.21.15.29:0/2286918979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27741045b0 0x7f27741069a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2764009900 tx=0x7f276402bc30 comp rx=0 tx=0).stop 2024-03-11T19:09:59.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.250+0000 7f277c7f4640 1 -- 172.21.15.29:0/2286918979 shutdown_connections 2024-03-11T19:09:59.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.250+0000 7f277c7f4640 1 --2- 172.21.15.29:0/2286918979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2774106ee0 0x7f27741092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.250+0000 7f277c7f4640 1 --2- 172.21.15.29:0/2286918979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27741045b0 0x7f27741069a0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.250+0000 7f277c7f4640 1 --2- 172.21.15.29:0/2286918979 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2774101c80 0x7f2774104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.250+0000 7f277c7f4640 1 -- 172.21.15.29:0/2286918979 >> 172.21.15.29:0/2286918979 conn(0x7f27740fb610 msgr2=0x7f27740fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:59.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.250+0000 7f277c7f4640 1 -- 172.21.15.29:0/2286918979 shutdown_connections 2024-03-11T19:09:59.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.250+0000 7f277c7f4640 1 -- 172.21.15.29:0/2286918979 wait complete. 2024-03-11T19:09:59.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.251+0000 7f277c7f4640 1 Processor -- start 2024-03-11T19:09:59.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.251+0000 7f277c7f4640 1 -- start start 2024-03-11T19:09:59.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.251+0000 7f277c7f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2774101c80 0x7f2774197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:59.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.251+0000 7f277c7f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f27741045b0 0x7f2774197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:59.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.251+0000 7f277c7f4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2774106ee0 0x7f277419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:59.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.251+0000 7f277c7f4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277419ec10 con 0x7f2774101c80 2024-03-11T19:09:59.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.251+0000 7f277c7f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277419ed80 con 0x7f27741045b0 2024-03-11T19:09:59.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.251+0000 7f277c7f4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277419ef40 con 0x7f2774106ee0 2024-03-11T19:09:59.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.251+0000 7f277a590640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2774101c80 0x7f2774197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:59.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.251+0000 7f277a590640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2774101c80 0x7f2774197120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54446/0 (socket says 172.21.15.29:54446) 2024-03-11T19:09:59.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.251+0000 7f277a590640 1 -- 172.21.15.29:0/1443096501 learned_addr learned my addr 172.21.15.29:0/1443096501 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.251+0000 7f277ad91640 1 --2- 172.21.15.29:0/1443096501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2774106ee0 0x7f277419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.251+0000 7f2779d8f640 1 --2- 172.21.15.29:0/1443096501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f27741045b0 0x7f2774197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.252+0000 7f277a590640 1 -- 172.21.15.29:0/1443096501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2774106ee0 msgr2=0x7f277419e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.252+0000 7f277a590640 1 --2- 172.21.15.29:0/1443096501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2774106ee0 0x7f277419e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.252+0000 7f277a590640 1 -- 172.21.15.29:0/1443096501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f27741045b0 msgr2=0x7f2774197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.252+0000 7f277a590640 1 --2- 172.21.15.29:0/1443096501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f27741045b0 0x7f2774197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.252+0000 7f277a590640 1 -- 172.21.15.29:0/1443096501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2764009560 con 0x7f2774101c80 2024-03-11T19:09:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.252+0000 7f277a590640 1 --2- 172.21.15.29:0/1443096501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2774101c80 0x7f2774197120 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f276c00c920 tx=0x7f276c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.252+0000 7f276b7fe640 1 -- 172.21.15.29:0/1443096501 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c007c70 con 0x7f2774101c80 2024-03-11T19:09:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.252+0000 7f276b7fe640 1 -- 172.21.15.29:0/1443096501 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f276c007dd0 con 0x7f2774101c80 2024-03-11T19:09:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.252+0000 7f277c7f4640 1 -- 172.21.15.29:0/1443096501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f277419f220 con 0x7f2774101c80 2024-03-11T19:09:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.252+0000 7f276b7fe640 1 -- 172.21.15.29:0/1443096501 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c0187c0 con 0x7f2774101c80 2024-03-11T19:09:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.252+0000 7f277c7f4640 1 -- 172.21.15.29:0/1443096501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f277419f750 con 0x7f2774101c80 2024-03-11T19:09:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.253+0000 7f277c7f4640 1 -- 172.21.15.29:0/1443096501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27740fd100 con 0x7f2774101c80 2024-03-11T19:09:59.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.254+0000 7f276b7fe640 1 -- 172.21.15.29:0/1443096501 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f276c00f450 con 0x7f2774101c80 2024-03-11T19:09:59.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.255+0000 7f276b7fe640 1 --2- 172.21.15.29:0/1443096501 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f27600a1b40 0x7f27600a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:59.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.255+0000 7f2779d8f640 1 --2- 172.21.15.29:0/1443096501 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f27600a1b40 0x7f27600a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:59.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.255+0000 7f276b7fe640 1 -- 172.21.15.29:0/1443096501 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f276c0cd670 con 0x7f2774101c80 2024-03-11T19:09:59.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.255+0000 7f2779d8f640 1 --2- 172.21.15.29:0/1443096501 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f27600a1b40 0x7f27600a4000 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2764006010 tx=0x7f2764000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:59.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.259+0000 7f276b7fe640 1 -- 172.21.15.29:0/1443096501 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f276c096720 con 0x7f2774101c80 2024-03-11T19:09:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:59 smithi029 ceph-mon[32439]: pgmap v356: 97 pgs: 97 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-11T19:09:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:59 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/4246037696' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:59 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3478030824' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:09:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:59 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2500739399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:09:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:09:59 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3854345966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:09:59.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.478+0000 7f277c7f4640 1 -- 172.21.15.29:0/1443096501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f277410ca60 con 0x7f2774101c80 2024-03-11T19:09:59.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.479+0000 7f276b7fe640 1 -- 172.21.15.29:0/1443096501 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f276c099de0 con 0x7f2774101c80 2024-03-11T19:09:59.482 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:59.482 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:09:59.482 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:09:59.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.480+0000 7f277c7f4640 1 -- 172.21.15.29:0/1443096501 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f27600a1b40 msgr2=0x7f27600a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:59.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.480+0000 7f277c7f4640 1 --2- 172.21.15.29:0/1443096501 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f27600a1b40 0x7f27600a4000 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2764006010 tx=0x7f2764000bc0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.480+0000 7f277c7f4640 1 -- 172.21.15.29:0/1443096501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2774101c80 msgr2=0x7f2774197120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:59.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.480+0000 7f277c7f4640 1 --2- 172.21.15.29:0/1443096501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2774101c80 0x7f2774197120 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f276c00c920 tx=0x7f276c00cde0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.480+0000 7f277c7f4640 1 -- 172.21.15.29:0/1443096501 shutdown_connections 2024-03-11T19:09:59.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.481+0000 7f277c7f4640 1 --2- 172.21.15.29:0/1443096501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2774106ee0 0x7f277419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.481+0000 7f277c7f4640 1 --2- 172.21.15.29:0/1443096501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f27741045b0 0x7f2774197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.481+0000 7f277c7f4640 1 --2- 172.21.15.29:0/1443096501 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f27600a1b40 0x7f27600a4000 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.481+0000 7f277c7f4640 1 --2- 172.21.15.29:0/1443096501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2774101c80 0x7f2774197120 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.481+0000 7f277c7f4640 1 -- 172.21.15.29:0/1443096501 >> 172.21.15.29:0/1443096501 conn(0x7f27740fb610 msgr2=0x7f27741026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:59.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.481+0000 7f277c7f4640 1 -- 172.21.15.29:0/1443096501 shutdown_connections 2024-03-11T19:09:59.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.481+0000 7f277c7f4640 1 -- 172.21.15.29:0/1443096501 wait complete. 2024-03-11T19:09:59.495 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-11T19:09:59.495 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-11T19:09:59.495 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-11T19:09:59.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:59 smithi190 ceph-mon[36292]: pgmap v356: 97 pgs: 97 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-11T19:09:59.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:59 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/4246037696' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:59.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:59 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3478030824' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:09:59.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:59 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2500739399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:09:59.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:09:59 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3854345966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:09:59.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:59 smithi170 ceph-mon[35690]: pgmap v356: 97 pgs: 97 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-11T19:09:59.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:59 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/4246037696' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:09:59.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:59 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3478030824' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:09:59.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:59 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2500739399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:09:59.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:09:59 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3854345966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:09:59.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.595+0000 7fc13c5d7640 1 -- 172.21.15.29:0/2311551445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1341027f0 msgr2=0x7fc134102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:59.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.595+0000 7fc13c5d7640 1 --2- 172.21.15.29:0/2311551445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1341027f0 0x7fc134102c10 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc124009ca0 tx=0x7fc12402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.595+0000 7fc13c5d7640 1 -- 172.21.15.29:0/2311551445 shutdown_connections 2024-03-11T19:09:59.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.595+0000 7fc13c5d7640 1 --2- 172.21.15.29:0/2311551445 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1340ffc20 0x7fc1341097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.595+0000 7fc13c5d7640 1 --2- 172.21.15.29:0/2311551445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1340ff260 0x7fc1340ff6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.595+0000 7fc13c5d7640 1 --2- 172.21.15.29:0/2311551445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1341027f0 0x7fc134102c10 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.595+0000 7fc13c5d7640 1 -- 172.21.15.29:0/2311551445 >> 172.21.15.29:0/2311551445 conn(0x7fc1340fadc0 msgr2=0x7fc1340fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:59.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.595+0000 7fc13c5d7640 1 -- 172.21.15.29:0/2311551445 shutdown_connections 2024-03-11T19:09:59.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.595+0000 7fc13c5d7640 1 -- 172.21.15.29:0/2311551445 wait complete. 2024-03-11T19:09:59.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc13c5d7640 1 Processor -- start 2024-03-11T19:09:59.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc13c5d7640 1 -- start start 2024-03-11T19:09:59.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc13c5d7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1340ff260 0x7fc1341971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:59.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc13c5d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1340ffc20 0x7fc134197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:59.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc13c5d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1341027f0 0x7fc13419e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:59.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc13c5d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13419ecd0 con 0x7fc1341027f0 2024-03-11T19:09:59.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc13c5d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13419ee40 con 0x7fc1340ffc20 2024-03-11T19:09:59.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc13c5d7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13419f000 con 0x7fc1340ff260 2024-03-11T19:09:59.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc13ab74640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1341027f0 0x7fc13419e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:59.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc13ab74640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1341027f0 0x7fc13419e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54474/0 (socket says 172.21.15.29:54474) 2024-03-11T19:09:59.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc13ab74640 1 -- 172.21.15.29:0/2493412024 learned_addr learned my addr 172.21.15.29:0/2493412024 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:59.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc13a373640 1 --2- 172.21.15.29:0/2493412024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1340ff260 0x7fc1341971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:59.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc139b72640 1 --2- 172.21.15.29:0/2493412024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1340ffc20 0x7fc134197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:59.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc13ab74640 1 -- 172.21.15.29:0/2493412024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1340ff260 msgr2=0x7fc1341971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:59.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc13ab74640 1 --2- 172.21.15.29:0/2493412024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1340ff260 0x7fc1341971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc13ab74640 1 -- 172.21.15.29:0/2493412024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1340ffc20 msgr2=0x7fc134197720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:59.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc13ab74640 1 --2- 172.21.15.29:0/2493412024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1340ffc20 0x7fc134197720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc13ab74640 1 -- 172.21.15.29:0/2493412024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc124009900 con 0x7fc1341027f0 2024-03-11T19:09:59.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc13a373640 1 --2- 172.21.15.29:0/2493412024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1340ff260 0x7fc1341971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:09:59.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.596+0000 7fc13ab74640 1 --2- 172.21.15.29:0/2493412024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1341027f0 0x7fc13419e790 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00e970 tx=0x7fc12c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:59.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.597+0000 7fc1337fe640 1 -- 172.21.15.29:0/2493412024 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c009720 con 0x7fc1341027f0 2024-03-11T19:09:59.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.597+0000 7fc1337fe640 1 -- 172.21.15.29:0/2493412024 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc12c0054d0 con 0x7fc1341027f0 2024-03-11T19:09:59.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.597+0000 7fc13c5d7640 1 -- 172.21.15.29:0/2493412024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13419f2e0 con 0x7fc1341027f0 2024-03-11T19:09:59.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.597+0000 7fc1337fe640 1 -- 172.21.15.29:0/2493412024 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c010430 con 0x7fc1341027f0 2024-03-11T19:09:59.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.597+0000 7fc13c5d7640 1 -- 172.21.15.29:0/2493412024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13419f7a0 con 0x7fc1341027f0 2024-03-11T19:09:59.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.598+0000 7fc13c5d7640 1 -- 172.21.15.29:0/2493412024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc134108870 con 0x7fc1341027f0 2024-03-11T19:09:59.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.599+0000 7fc1337fe640 1 -- 172.21.15.29:0/2493412024 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fc12c004f10 con 0x7fc1341027f0 2024-03-11T19:09:59.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.599+0000 7fc1337fe640 1 --2- 172.21.15.29:0/2493412024 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc11c0a1ac0 0x7fc11c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:59.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.599+0000 7fc13a373640 1 --2- 172.21.15.29:0/2493412024 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc11c0a1ac0 0x7fc11c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:59.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.599+0000 7fc1337fe640 1 -- 172.21.15.29:0/2493412024 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fc12c014070 con 0x7fc1341027f0 2024-03-11T19:09:59.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.600+0000 7fc13a373640 1 --2- 172.21.15.29:0/2493412024 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc11c0a1ac0 0x7fc11c0a3f80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc12402c700 tx=0x7fc1240325f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:59.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.603+0000 7fc1337fe640 1 -- 172.21.15.29:0/2493412024 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc12c096d50 con 0x7fc1341027f0 2024-03-11T19:09:59.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.780+0000 7fc13c5d7640 1 -- 172.21.15.29:0/2493412024 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc13402ceb0 con 0x7fc11c0a1ac0 2024-03-11T19:09:59.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.788+0000 7fc1337fe640 1 -- 172.21.15.29:0/2493412024 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259094 (secure 0 0 0) 0x7fc13402ceb0 con 0x7fc11c0a1ac0 2024-03-11T19:09:59.794 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:09:59.795 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-11T19:09:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.791+0000 7fc13c5d7640 1 -- 172.21.15.29:0/2493412024 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc11c0a1ac0 msgr2=0x7fc11c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.791+0000 7fc13c5d7640 1 --2- 172.21.15.29:0/2493412024 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc11c0a1ac0 0x7fc11c0a3f80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc12402c700 tx=0x7fc1240325f0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.791+0000 7fc13c5d7640 1 -- 172.21.15.29:0/2493412024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1341027f0 msgr2=0x7fc13419e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.791+0000 7fc13c5d7640 1 --2- 172.21.15.29:0/2493412024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1341027f0 0x7fc13419e790 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00e970 tx=0x7fc12c00ed70 comp rx=0 tx=0).stop 2024-03-11T19:09:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.791+0000 7fc13c5d7640 1 -- 172.21.15.29:0/2493412024 shutdown_connections 2024-03-11T19:09:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.791+0000 7fc13c5d7640 1 --2- 172.21.15.29:0/2493412024 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc11c0a1ac0 0x7fc11c0a3f80 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.791+0000 7fc13c5d7640 1 --2- 172.21.15.29:0/2493412024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1341027f0 0x7fc13419e790 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.791+0000 7fc13c5d7640 1 --2- 172.21.15.29:0/2493412024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1340ffc20 0x7fc134197720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.791+0000 7fc13c5d7640 1 --2- 172.21.15.29:0/2493412024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1340ff260 0x7fc1341971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.791+0000 7fc13c5d7640 1 -- 172.21.15.29:0/2493412024 >> 172.21.15.29:0/2493412024 conn(0x7fc1340fadc0 msgr2=0x7fc134105a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.791+0000 7fc13c5d7640 1 -- 172.21.15.29:0/2493412024 shutdown_connections 2024-03-11T19:09:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.791+0000 7fc13c5d7640 1 -- 172.21.15.29:0/2493412024 wait complete. 2024-03-11T19:09:59.805 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-03-11T19:09:59.636214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666021,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1384,"num_read_kb":820,"num_write":449,"num_write_kb":3099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":394388,"kb_used_data":11956,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124547436,"statfs":{"total":1151940427776,"available":1151536574464,"internally_reserved":0,"allocated":12242944,"data_stored":7549485,"data_compressed":112974,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":391492922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7170,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":127,"num_read_kb":66,"num_write":22,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002894"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.650725+0000","last_change":"2024-03-11T19:07:49.751738+0000","last_active":"2024-03-11T19:09:13.650725+0000","last_peered":"2024-03-11T19:09:13.650725+0000","last_clean":"2024-03-11T19:09:13.650725+0000","last_became_active":"2024-03-11T19:07:49.751173+0000","last_became_peered":"2024-03-11T19:07:49.751173+0000","last_unstale":"2024-03-11T19:09:13.650725+0000","last_undegraded":"2024-03-11T19:09:13.650725+0000","last_fullsized":"2024-03-11T19:09:13.650725+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:13:40.157763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.648231+0000","last_change":"2024-03-11T19:07:49.749069+0000","last_active":"2024-03-11T19:09:13.648231+0000","last_peered":"2024-03-11T19:09:13.648231+0000","last_clean":"2024-03-11T19:09:13.648231+0000","last_became_active":"2024-03-11T19:07:49.747672+0000","last_became_peered":"2024-03-11T19:07:49.747672+0000","last_unstale":"2024-03-11T19:09:13.648231+0000","last_undegraded":"2024-03-11T19:09:13.648231+0000","last_fullsized":"2024-03-11T19:09:13.648231+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:57:26.379836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.653423+0000","last_change":"2024-03-11T19:07:49.750298+0000","last_active":"2024-03-11T19:09:13.653423+0000","last_peered":"2024-03-11T19:09:13.653423+0000","last_clean":"2024-03-11T19:09:13.653423+0000","last_became_active":"2024-03-11T19:07:49.749985+0000","last_became_peered":"2024-03-11T19:07:49.749985+0000","last_unstale":"2024-03-11T19:09:13.653423+0000","last_undegraded":"2024-03-11T19:09:13.653423+0000","last_fullsized":"2024-03-11T19:09:13.653423+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:56.579872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.642014+0000","last_change":"2024-03-11T19:08:44.993047+0000","last_active":"2024-03-11T19:09:13.642014+0000","last_peered":"2024-03-11T19:09:13.642014+0000","last_clean":"2024-03-11T19:09:13.642014+0000","last_became_active":"2024-03-11T19:08:44.992666+0000","last_became_peered":"2024-03-11T19:08:44.992666+0000","last_unstale":"2024-03-11T19:09:13.642014+0000","last_undegraded":"2024-03-11T19:09:13.642014+0000","last_fullsized":"2024-03-11T19:09:13.642014+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:32:18.748906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.634551+0000","last_change":"2024-03-11T19:07:49.747618+0000","last_active":"2024-03-11T19:09:13.634551+0000","last_peered":"2024-03-11T19:09:13.634551+0000","last_clean":"2024-03-11T19:09:13.634551+0000","last_became_active":"2024-03-11T19:07:49.747310+0000","last_became_peered":"2024-03-11T19:07:49.747310+0000","last_unstale":"2024-03-11T19:09:13.634551+0000","last_undegraded":"2024-03-11T19:09:13.634551+0000","last_fullsized":"2024-03-11T19:09:13.634551+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:15:42.260509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.635452+0000","last_change":"2024-03-11T19:07:49.755564+0000","last_active":"2024-03-11T19:09:13.635452+0000","last_peered":"2024-03-11T19:09:13.635452+0000","last_clean":"2024-03-11T19:09:13.635452+0000","last_became_active":"2024-03-11T19:07:49.751322+0000","last_became_peered":"2024-03-11T19:07:49.751322+0000","last_unstale":"2024-03-11T19:09:13.635452+0000","last_undegraded":"2024-03-11T19:09:13.635452+0000","last_fullsized":"2024-03-11T19:09:13.635452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:43:18.487887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.637740+0000","last_change":"2024-03-11T19:07:49.731619+0000","last_active":"2024-03-11T19:09:13.637740+0000","last_peered":"2024-03-11T19:09:13.637740+0000","last_clean":"2024-03-11T19:09:13.637740+0000","last_became_active":"2024-03-11T19:07:49.731323+0000","last_became_peered":"2024-03-11T19:07:49.731323+0000","last_unstale":"2024-03-11T19:09:13.637740+0000","last_undegraded":"2024-03-11T19:09:13.637740+0000","last_fullsized":"2024-03-11T19:09:13.637740+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:20:25.592709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.646995+0000","last_change":"2024-03-11T19:07:49.751851+0000","last_active":"2024-03-11T19:09:13.646995+0000","last_peered":"2024-03-11T19:09:13.646995+0000","last_clean":"2024-03-11T19:09:13.646995+0000","last_became_active":"2024-03-11T19:07:49.750068+0000","last_became_peered":"2024-03-11T19:07:49.750068+0000","last_unstale":"2024-03-11T19:09:13.646995+0000","last_undegraded":"2024-03-11T19:09:13.646995+0000","last_fullsized":"2024-03-11T19:09:13.646995+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:59:35.820107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.652310+0000","last_change":"2024-03-11T19:07:49.752326+0000","last_active":"2024-03-11T19:09:13.652310+0000","last_peered":"2024-03-11T19:09:13.652310+0000","last_clean":"2024-03-11T19:09:13.652310+0000","last_became_active":"2024-03-11T19:07:49.748052+0000","last_became_peered":"2024-03-11T19:07:49.748052+0000","last_unstale":"2024-03-11T19:09:13.652310+0000","last_undegraded":"2024-03-11T19:09:13.652310+0000","last_fullsized":"2024-03-11T19:09:13.652310+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:58:13.325964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.639621+0000","last_change":"2024-03-11T19:07:49.748054+0000","last_active":"2024-03-11T19:09:13.639621+0000","last_peered":"2024-03-11T19:09:13.639621+0000","last_clean":"2024-03-11T19:09:13.639621+0000","last_became_active":"2024-03-11T19:07:49.747765+0000","last_became_peered":"2024-03-11T19:07:49.747765+0000","last_unstale":"2024-03-11T19:09:13.639621+0000","last_undegraded":"2024-03-11T19:09:13.639621+0000","last_fullsized":"2024-03-11T19:09:13.639621+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:18:05.599512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.651823+0000","last_change":"2024-03-11T19:07:49.751622+0000","last_active":"2024-03-11T19:09:13.651823+0000","last_peered":"2024-03-11T19:09:13.651823+0000","last_clean":"2024-03-11T19:09:13.651823+0000","last_became_active":"2024-03-11T19:07:49.751317+0000","last_became_peered":"2024-03-11T19:07:49.751317+0000","last_unstale":"2024-03-11T19:09:13.651823+0000","last_undegraded":"2024-03-11T19:09:13.651823+0000","last_fullsized":"2024-03-11T19:09:13.651823+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:49.836096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.630547+0000","last_change":"2024-03-11T19:07:49.749160+0000","last_active":"2024-03-11T19:09:13.630547+0000","last_peered":"2024-03-11T19:09:13.630547+0000","last_clean":"2024-03-11T19:09:13.630547+0000","last_became_active":"2024-03-11T19:07:49.748838+0000","last_became_peered":"2024-03-11T19:07:49.748838+0000","last_unstale":"2024-03-11T19:09:13.630547+0000","last_undegraded":"2024-03-11T19:09:13.630547+0000","last_fullsized":"2024-03-11T19:09:13.630547+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:30:58.327196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.647032+0000","last_change":"2024-03-11T19:07:49.749965+0000","last_active":"2024-03-11T19:09:13.647032+0000","last_peered":"2024-03-11T19:09:13.647032+0000","last_clean":"2024-03-11T19:09:13.647032+0000","last_became_active":"2024-03-11T19:07:49.749602+0000","last_became_peered":"2024-03-11T19:07:49.749602+0000","last_unstale":"2024-03-11T19:09:13.647032+0000","last_undegraded":"2024-03-11T19:09:13.647032+0000","last_fullsized":"2024-03-11T19:09:13.647032+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:36:08.807251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.640585+0000","last_change":"2024-03-11T19:07:49.747557+0000","last_active":"2024-03-11T19:09:13.640585+0000","last_peered":"2024-03-11T19:09:13.640585+0000","last_clean":"2024-03-11T19:09:13.640585+0000","last_became_active":"2024-03-11T19:07:49.745804+0000","last_became_peered":"2024-03-11T19:07:49.745804+0000","last_unstale":"2024-03-11T19:09:13.640585+0000","last_undegraded":"2024-03-11T19:09:13.640585+0000","last_fullsized":"2024-03-11T19:09:13.640585+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:07:26.939414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.649130+0000","last_change":"2024-03-11T19:07:49.744070+0000","last_active":"2024-03-11T19:09:13.649130+0000","last_peered":"2024-03-11T19:09:13.649130+0000","last_clean":"2024-03-11T19:09:13.649130+0000","last_became_active":"2024-03-11T19:07:49.743736+0000","last_became_peered":"2024-03-11T19:07:49.743736+0000","last_unstale":"2024-03-11T19:09:13.649130+0000","last_undegraded":"2024-03-11T19:09:13.649130+0000","last_fullsized":"2024-03-11T19:09:13.649130+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:50:33.862644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.657246+0000","last_change":"2024-03-11T19:07:49.748512+0000","last_active":"2024-03-11T19:09:13.657246+0000","last_peered":"2024-03-11T19:09:13.657246+0000","last_clean":"2024-03-11T19:09:13.657246+0000","last_became_active":"2024-03-11T19:07:49.748223+0000","last_became_peered":"2024-03-11T19:07:49.748223+0000","last_unstale":"2024-03-11T19:09:13.657246+0000","last_undegraded":"2024-03-11T19:09:13.657246+0000","last_fullsized":"2024-03-11T19:09:13.657246+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:49:43.305677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.655548+0000","last_change":"2024-03-11T19:07:49.754322+0000","last_active":"2024-03-11T19:09:13.655548+0000","last_peered":"2024-03-11T19:09:13.655548+0000","last_clean":"2024-03-11T19:09:13.655548+0000","last_became_active":"2024-03-11T19:07:49.753757+0000","last_became_peered":"2024-03-11T19:07:49.753757+0000","last_unstale":"2024-03-11T19:09:13.655548+0000","last_undegraded":"2024-03-11T19:09:13.655548+0000","last_fullsized":"2024-03-11T19:09:13.655548+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:49.684978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.657358+0000","last_change":"2024-03-11T19:07:49.749546+0000","last_active":"2024-03-11T19:09:13.657358+0000","last_peered":"2024-03-11T19:09:13.657358+0000","last_clean":"2024-03-11T19:09:13.657358+0000","last_became_active":"2024-03-11T19:07:49.749200+0000","last_became_peered":"2024-03-11T19:07:49.749200+0000","last_unstale":"2024-03-11T19:09:13.657358+0000","last_undegraded":"2024-03-11T19:09:13.657358+0000","last_fullsized":"2024-03-11T19:09:13.657358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:26:12.914015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.645227+0000","last_change":"2024-03-11T19:07:49.750388+0000","last_active":"2024-03-11T19:09:13.645227+0000","last_peered":"2024-03-11T19:09:13.645227+0000","last_clean":"2024-03-11T19:09:13.645227+0000","last_became_active":"2024-03-11T19:07:49.750016+0000","last_became_peered":"2024-03-11T19:07:49.750016+0000","last_unstale":"2024-03-11T19:09:13.645227+0000","last_undegraded":"2024-03-11T19:09:13.645227+0000","last_fullsized":"2024-03-11T19:09:13.645227+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:32:23.125695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'14","reported_seq":155,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:55.832399+0000","last_change":"2024-03-11T19:07:48.738369+0000","last_active":"2024-03-11T19:09:55.832399+0000","last_peered":"2024-03-11T19:09:55.832399+0000","last_clean":"2024-03-11T19:09:55.832399+0000","last_became_active":"2024-03-11T19:07:48.738010+0000","last_became_peered":"2024-03-11T19:07:48.738010+0000","last_unstale":"2024-03-11T19:09:55.832399+0000","last_undegraded":"2024-03-11T19:09:55.832399+0000","last_fullsized":"2024-03-11T19:09:55.832399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:06:48.446454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.646494+0000","last_change":"2024-03-11T19:07:49.752622+0000","last_active":"2024-03-11T19:09:13.646494+0000","last_peered":"2024-03-11T19:09:13.646494+0000","last_clean":"2024-03-11T19:09:13.646494+0000","last_became_active":"2024-03-11T19:07:49.752242+0000","last_became_peered":"2024-03-11T19:07:49.752242+0000","last_unstale":"2024-03-11T19:09:13.646494+0000","last_undegraded":"2024-03-11T19:09:13.646494+0000","last_fullsized":"2024-03-11T19:09:13.646494+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:03:50.609641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990514+0000","last_change":"2024-03-11T19:07:48.751657+0000","last_active":"2024-03-11T19:08:44.990514+0000","last_peered":"2024-03-11T19:08:44.990514+0000","last_clean":"2024-03-11T19:08:44.990514+0000","last_became_active":"2024-03-11T19:07:48.751294+0000","last_became_peered":"2024-03-11T19:07:48.751294+0000","last_unstale":"2024-03-11T19:08:44.990514+0000","last_undegraded":"2024-03-11T19:08:44.990514+0000","last_fullsized":"2024-03-11T19:08:44.990514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:33:58.550707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.655152+0000","last_change":"2024-03-11T19:07:49.759460+0000","last_active":"2024-03-11T19:09:13.655152+0000","last_peered":"2024-03-11T19:09:13.655152+0000","last_clean":"2024-03-11T19:09:13.655152+0000","last_became_active":"2024-03-11T19:07:49.759136+0000","last_became_peered":"2024-03-11T19:07:49.759136+0000","last_unstale":"2024-03-11T19:09:13.655152+0000","last_undegraded":"2024-03-11T19:09:13.655152+0000","last_fullsized":"2024-03-11T19:09:13.655152+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:10:37.273896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'12","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991106+0000","last_change":"2024-03-11T19:07:48.744695+0000","last_active":"2024-03-11T19:08:44.991106+0000","last_peered":"2024-03-11T19:08:44.991106+0000","last_clean":"2024-03-11T19:08:44.991106+0000","last_became_active":"2024-03-11T19:07:48.744378+0000","last_became_peered":"2024-03-11T19:07:48.744378+0000","last_unstale":"2024-03-11T19:08:44.991106+0000","last_undegraded":"2024-03-11T19:08:44.991106+0000","last_fullsized":"2024-03-11T19:08:44.991106+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:55:46.601414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.655586+0000","last_change":"2024-03-11T19:07:49.761429+0000","last_active":"2024-03-11T19:09:13.655586+0000","last_peered":"2024-03-11T19:09:13.655586+0000","last_clean":"2024-03-11T19:09:13.655586+0000","last_became_active":"2024-03-11T19:07:49.761120+0000","last_became_peered":"2024-03-11T19:07:49.761120+0000","last_unstale":"2024-03-11T19:09:13.655586+0000","last_undegraded":"2024-03-11T19:09:13.655586+0000","last_fullsized":"2024-03-11T19:09:13.655586+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:13:45.839528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990713+0000","last_change":"2024-03-11T19:07:48.740451+0000","last_active":"2024-03-11T19:08:44.990713+0000","last_peered":"2024-03-11T19:08:44.990713+0000","last_clean":"2024-03-11T19:08:44.990713+0000","last_became_active":"2024-03-11T19:07:48.740097+0000","last_became_peered":"2024-03-11T19:07:48.740097+0000","last_unstale":"2024-03-11T19:08:44.990713+0000","last_undegraded":"2024-03-11T19:08:44.990713+0000","last_fullsized":"2024-03-11T19:08:44.990713+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:37.766790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.657267+0000","last_change":"2024-03-11T19:07:49.750404+0000","last_active":"2024-03-11T19:09:13.657267+0000","last_peered":"2024-03-11T19:09:13.657267+0000","last_clean":"2024-03-11T19:09:13.657267+0000","last_became_active":"2024-03-11T19:07:49.749941+0000","last_became_peered":"2024-03-11T19:07:49.749941+0000","last_unstale":"2024-03-11T19:09:13.657267+0000","last_undegraded":"2024-03-11T19:09:13.657267+0000","last_fullsized":"2024-03-11T19:09:13.657267+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:55.766729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'8","reported_seq":153,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:56.835200+0000","last_change":"2024-03-11T19:07:48.737100+0000","last_active":"2024-03-11T19:09:56.835200+0000","last_peered":"2024-03-11T19:09:56.835200+0000","last_clean":"2024-03-11T19:09:56.835200+0000","last_became_active":"2024-03-11T19:07:48.736768+0000","last_became_peered":"2024-03-11T19:07:48.736768+0000","last_unstale":"2024-03-11T19:09:56.835200+0000","last_undegraded":"2024-03-11T19:09:56.835200+0000","last_fullsized":"2024-03-11T19:09:56.835200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:18:36.872151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1647,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.634648+0000","last_change":"2024-03-11T19:07:49.739020+0000","last_active":"2024-03-11T19:09:13.634648+0000","last_peered":"2024-03-11T19:09:13.634648+0000","last_clean":"2024-03-11T19:09:13.634648+0000","last_became_active":"2024-03-11T19:07:49.738642+0000","last_became_peered":"2024-03-11T19:07:49.738642+0000","last_unstale":"2024-03-11T19:09:13.634648+0000","last_undegraded":"2024-03-11T19:09:13.634648+0000","last_fullsized":"2024-03-11T19:09:13.634648+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:42:29.040242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037233+0000","last_change":"2024-03-11T19:07:48.745192+0000","last_active":"2024-03-11T19:08:45.037233+0000","last_peered":"2024-03-11T19:08:45.037233+0000","last_clean":"2024-03-11T19:08:45.037233+0000","last_became_active":"2024-03-11T19:07:48.744868+0000","last_became_peered":"2024-03-11T19:07:48.744868+0000","last_unstale":"2024-03-11T19:08:45.037233+0000","last_undegraded":"2024-03-11T19:08:45.037233+0000","last_fullsized":"2024-03-11T19:08:45.037233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:19:15.620186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.640619+0000","last_change":"2024-03-11T19:07:49.737149+0000","last_active":"2024-03-11T19:09:13.640619+0000","last_peered":"2024-03-11T19:09:13.640619+0000","last_clean":"2024-03-11T19:09:13.640619+0000","last_became_active":"2024-03-11T19:07:49.736797+0000","last_became_peered":"2024-03-11T19:07:49.736797+0000","last_unstale":"2024-03-11T19:09:13.640619+0000","last_undegraded":"2024-03-11T19:09:13.640619+0000","last_fullsized":"2024-03-11T19:09:13.640619+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:12:10.282062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.988596+0000","last_change":"2024-03-11T19:07:48.743948+0000","last_active":"2024-03-11T19:08:44.988596+0000","last_peered":"2024-03-11T19:08:44.988596+0000","last_clean":"2024-03-11T19:08:44.988596+0000","last_became_active":"2024-03-11T19:07:48.743451+0000","last_became_peered":"2024-03-11T19:07:48.743451+0000","last_unstale":"2024-03-11T19:08:44.988596+0000","last_undegraded":"2024-03-11T19:08:44.988596+0000","last_fullsized":"2024-03-11T19:08:44.988596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:03:19.823287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.636529+0000","last_change":"2024-03-11T19:07:49.745518+0000","last_active":"2024-03-11T19:09:13.636529+0000","last_peered":"2024-03-11T19:09:13.636529+0000","last_clean":"2024-03-11T19:09:13.636529+0000","last_became_active":"2024-03-11T19:07:49.743719+0000","last_became_peered":"2024-03-11T19:07:49.743719+0000","last_unstale":"2024-03-11T19:09:13.636529+0000","last_undegraded":"2024-03-11T19:09:13.636529+0000","last_fullsized":"2024-03-11T19:09:13.636529+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:25:33.855679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036218+0000","last_change":"2024-03-11T19:07:48.734044+0000","last_active":"2024-03-11T19:08:45.036218+0000","last_peered":"2024-03-11T19:08:45.036218+0000","last_clean":"2024-03-11T19:08:45.036218+0000","last_became_active":"2024-03-11T19:07:48.733723+0000","last_became_peered":"2024-03-11T19:07:48.733723+0000","last_unstale":"2024-03-11T19:08:45.036218+0000","last_undegraded":"2024-03-11T19:08:45.036218+0000","last_fullsized":"2024-03-11T19:08:45.036218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:53:37.070296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.635887+0000","last_change":"2024-03-11T19:07:49.743729+0000","last_active":"2024-03-11T19:09:13.635887+0000","last_peered":"2024-03-11T19:09:13.635887+0000","last_clean":"2024-03-11T19:09:13.635887+0000","last_became_active":"2024-03-11T19:07:49.742992+0000","last_became_peered":"2024-03-11T19:07:49.742992+0000","last_unstale":"2024-03-11T19:09:13.635887+0000","last_undegraded":"2024-03-11T19:09:13.635887+0000","last_fullsized":"2024-03-11T19:09:13.635887+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:46.477177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"89'7","reported_seq":268,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:56.834343+0000","last_change":"2024-03-11T19:07:48.738775+0000","last_active":"2024-03-11T19:09:56.834343+0000","last_peered":"2024-03-11T19:09:56.834343+0000","last_clean":"2024-03-11T19:09:56.834343+0000","last_became_active":"2024-03-11T19:07:48.738437+0000","last_became_peered":"2024-03-11T19:07:48.738437+0000","last_unstale":"2024-03-11T19:09:56.834343+0000","last_undegraded":"2024-03-11T19:09:56.834343+0000","last_fullsized":"2024-03-11T19:09:56.834343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:17:20.020756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":242,"num_read_kb":242,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.654932+0000","last_change":"2024-03-11T19:07:49.760293+0000","last_active":"2024-03-11T19:09:13.654932+0000","last_peered":"2024-03-11T19:09:13.654932+0000","last_clean":"2024-03-11T19:09:13.654932+0000","last_became_active":"2024-03-11T19:07:49.759977+0000","last_became_peered":"2024-03-11T19:07:49.759977+0000","last_unstale":"2024-03-11T19:09:13.654932+0000","last_undegraded":"2024-03-11T19:09:13.654932+0000","last_fullsized":"2024-03-11T19:09:13.654932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:07:44.726756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985956+0000","last_change":"2024-03-11T19:07:48.740672+0000","last_active":"2024-03-11T19:08:44.985956+0000","last_peered":"2024-03-11T19:08:44.985956+0000","last_clean":"2024-03-11T19:08:44.985956+0000","last_became_active":"2024-03-11T19:07:48.739331+0000","last_became_peered":"2024-03-11T19:07:48.739331+0000","last_unstale":"2024-03-11T19:08:44.985956+0000","last_undegraded":"2024-03-11T19:08:44.985956+0000","last_fullsized":"2024-03-11T19:08:44.985956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:19:28.375515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.650493+0000","last_change":"2024-03-11T19:07:49.742780+0000","last_active":"2024-03-11T19:09:13.650493+0000","last_peered":"2024-03-11T19:09:13.650493+0000","last_clean":"2024-03-11T19:09:13.650493+0000","last_became_active":"2024-03-11T19:07:49.742442+0000","last_became_peered":"2024-03-11T19:07:49.742442+0000","last_unstale":"2024-03-11T19:09:13.650493+0000","last_undegraded":"2024-03-11T19:09:13.650493+0000","last_fullsized":"2024-03-11T19:09:13.650493+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:11:13.838225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036582+0000","last_change":"2024-03-11T19:07:48.752617+0000","last_active":"2024-03-11T19:08:45.036582+0000","last_peered":"2024-03-11T19:08:45.036582+0000","last_clean":"2024-03-11T19:08:45.036582+0000","last_became_active":"2024-03-11T19:07:48.752181+0000","last_became_peered":"2024-03-11T19:07:48.752181+0000","last_unstale":"2024-03-11T19:08:45.036582+0000","last_undegraded":"2024-03-11T19:08:45.036582+0000","last_fullsized":"2024-03-11T19:08:45.036582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:05:37.162866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.654679+0000","last_change":"2024-03-11T19:07:49.753672+0000","last_active":"2024-03-11T19:09:13.654679+0000","last_peered":"2024-03-11T19:09:13.654679+0000","last_clean":"2024-03-11T19:09:13.654679+0000","last_became_active":"2024-03-11T19:07:49.753315+0000","last_became_peered":"2024-03-11T19:07:49.753315+0000","last_unstale":"2024-03-11T19:09:13.654679+0000","last_undegraded":"2024-03-11T19:09:13.654679+0000","last_fullsized":"2024-03-11T19:09:13.654679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:30:58.503249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'38","reported_seq":310,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:56.834983+0000","last_change":"2024-03-11T19:07:48.740424+0000","last_active":"2024-03-11T19:09:56.834983+0000","last_peered":"2024-03-11T19:09:56.834983+0000","last_clean":"2024-03-11T19:09:56.834983+0000","last_became_active":"2024-03-11T19:07:48.740077+0000","last_became_peered":"2024-03-11T19:07:48.740077+0000","last_unstale":"2024-03-11T19:09:56.834983+0000","last_undegraded":"2024-03-11T19:09:56.834983+0000","last_fullsized":"2024-03-11T19:09:56.834983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:30:29.193861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66493,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":253,"num_read_kb":192,"num_write":40,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.652816+0000","last_change":"2024-03-11T19:07:49.746440+0000","last_active":"2024-03-11T19:09:13.652816+0000","last_peered":"2024-03-11T19:09:13.652816+0000","last_clean":"2024-03-11T19:09:13.652816+0000","last_became_active":"2024-03-11T19:07:49.746106+0000","last_became_peered":"2024-03-11T19:07:49.746106+0000","last_unstale":"2024-03-11T19:09:13.652816+0000","last_undegraded":"2024-03-11T19:09:13.652816+0000","last_fullsized":"2024-03-11T19:09:13.652816+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:44:58.039917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'9","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.986027+0000","last_change":"2024-03-11T19:07:48.744437+0000","last_active":"2024-03-11T19:08:44.986027+0000","last_peered":"2024-03-11T19:08:44.986027+0000","last_clean":"2024-03-11T19:08:44.986027+0000","last_became_active":"2024-03-11T19:07:48.744142+0000","last_became_peered":"2024-03-11T19:07:48.744142+0000","last_unstale":"2024-03-11T19:08:44.986027+0000","last_undegraded":"2024-03-11T19:08:44.986027+0000","last_fullsized":"2024-03-11T19:08:44.986027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:32:09.076975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.653140+0000","last_change":"2024-03-11T19:07:49.751191+0000","last_active":"2024-03-11T19:09:13.653140+0000","last_peered":"2024-03-11T19:09:13.653140+0000","last_clean":"2024-03-11T19:09:13.653140+0000","last_became_active":"2024-03-11T19:07:49.750849+0000","last_became_peered":"2024-03-11T19:07:49.750849+0000","last_unstale":"2024-03-11T19:09:13.653140+0000","last_undegraded":"2024-03-11T19:09:13.653140+0000","last_fullsized":"2024-03-11T19:09:13.653140+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:19:18.416905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036229+0000","last_change":"2024-03-11T19:07:48.742808+0000","last_active":"2024-03-11T19:08:45.036229+0000","last_peered":"2024-03-11T19:08:45.036229+0000","last_clean":"2024-03-11T19:08:45.036229+0000","last_became_active":"2024-03-11T19:07:48.742462+0000","last_became_peered":"2024-03-11T19:07:48.742462+0000","last_unstale":"2024-03-11T19:08:45.036229+0000","last_undegraded":"2024-03-11T19:08:45.036229+0000","last_fullsized":"2024-03-11T19:08:45.036229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:19.109299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.651622+0000","last_change":"2024-03-11T19:07:49.744673+0000","last_active":"2024-03-11T19:09:13.651622+0000","last_peered":"2024-03-11T19:09:13.651622+0000","last_clean":"2024-03-11T19:09:13.651622+0000","last_became_active":"2024-03-11T19:07:49.744313+0000","last_became_peered":"2024-03-11T19:07:49.744313+0000","last_unstale":"2024-03-11T19:09:13.651622+0000","last_undegraded":"2024-03-11T19:09:13.651622+0000","last_fullsized":"2024-03-11T19:09:13.651622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:21:04.759356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":151,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:57.836159+0000","last_change":"2024-03-11T19:07:48.736024+0000","last_active":"2024-03-11T19:09:57.836159+0000","last_peered":"2024-03-11T19:09:57.836159+0000","last_clean":"2024-03-11T19:09:57.836159+0000","last_became_active":"2024-03-11T19:07:48.735720+0000","last_became_peered":"2024-03-11T19:07:48.735720+0000","last_unstale":"2024-03-11T19:09:57.836159+0000","last_undegraded":"2024-03-11T19:09:57.836159+0000","last_fullsized":"2024-03-11T19:09:57.836159+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:04:57.386267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":127,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.640229+0000","last_change":"2024-03-11T19:07:49.742924+0000","last_active":"2024-03-11T19:09:13.640229+0000","last_peered":"2024-03-11T19:09:13.640229+0000","last_clean":"2024-03-11T19:09:13.640229+0000","last_became_active":"2024-03-11T19:07:49.742542+0000","last_became_peered":"2024-03-11T19:07:49.742542+0000","last_unstale":"2024-03-11T19:09:13.640229+0000","last_undegraded":"2024-03-11T19:09:13.640229+0000","last_fullsized":"2024-03-11T19:09:13.640229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:08:11.944311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'17","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990343+0000","last_change":"2024-03-11T19:07:48.753113+0000","last_active":"2024-03-11T19:08:44.990343+0000","last_peered":"2024-03-11T19:08:44.990343+0000","last_clean":"2024-03-11T19:08:44.990343+0000","last_became_active":"2024-03-11T19:07:48.752517+0000","last_became_peered":"2024-03-11T19:07:48.752517+0000","last_unstale":"2024-03-11T19:08:44.990343+0000","last_undegraded":"2024-03-11T19:08:44.990343+0000","last_fullsized":"2024-03-11T19:08:44.990343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:48:33.441125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.653632+0000","last_change":"2024-03-11T19:08:45.038243+0000","last_active":"2024-03-11T19:09:13.653632+0000","last_peered":"2024-03-11T19:09:13.653632+0000","last_clean":"2024-03-11T19:09:13.653632+0000","last_became_active":"2024-03-11T19:08:45.037969+0000","last_became_peered":"2024-03-11T19:08:45.037969+0000","last_unstale":"2024-03-11T19:09:13.653632+0000","last_undegraded":"2024-03-11T19:09:13.653632+0000","last_fullsized":"2024-03-11T19:09:13.653632+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:29:09.747958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"89'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991151+0000","last_change":"2024-03-11T19:07:48.739571+0000","last_active":"2024-03-11T19:08:44.991151+0000","last_peered":"2024-03-11T19:08:44.991151+0000","last_clean":"2024-03-11T19:08:44.991151+0000","last_became_active":"2024-03-11T19:07:48.739159+0000","last_became_peered":"2024-03-11T19:07:48.739159+0000","last_unstale":"2024-03-11T19:08:44.991151+0000","last_undegraded":"2024-03-11T19:08:44.991151+0000","last_fullsized":"2024-03-11T19:08:44.991151+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:46:43.474591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.639217+0000","last_change":"2024-03-11T19:07:49.744760+0000","last_active":"2024-03-11T19:09:13.639217+0000","last_peered":"2024-03-11T19:09:13.639217+0000","last_clean":"2024-03-11T19:09:13.639217+0000","last_became_active":"2024-03-11T19:07:49.744265+0000","last_became_peered":"2024-03-11T19:07:49.744265+0000","last_unstale":"2024-03-11T19:09:13.639217+0000","last_undegraded":"2024-03-11T19:09:13.639217+0000","last_fullsized":"2024-03-11T19:09:13.639217+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:09:49.823462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"87'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985221+0000","last_change":"2024-03-11T19:07:48.740223+0000","last_active":"2024-03-11T19:08:44.985221+0000","last_peered":"2024-03-11T19:08:44.985221+0000","last_clean":"2024-03-11T19:08:44.985221+0000","last_became_active":"2024-03-11T19:07:48.739569+0000","last_became_peered":"2024-03-11T19:07:48.739569+0000","last_unstale":"2024-03-11T19:08:44.985221+0000","last_undegraded":"2024-03-11T19:08:44.985221+0000","last_fullsized":"2024-03-11T19:08:44.985221+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:04:19.384523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.641047+0000","last_change":"2024-03-11T19:07:49.755620+0000","last_active":"2024-03-11T19:09:13.641047+0000","last_peered":"2024-03-11T19:09:13.641047+0000","last_clean":"2024-03-11T19:09:13.641047+0000","last_became_active":"2024-03-11T19:07:49.750886+0000","last_became_peered":"2024-03-11T19:07:49.750886+0000","last_unstale":"2024-03-11T19:09:13.641047+0000","last_undegraded":"2024-03-11T19:09:13.641047+0000","last_fullsized":"2024-03-11T19:09:13.641047+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:35:54.191061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'6","reported_seq":147,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:54.831635+0000","last_change":"2024-03-11T19:07:48.743340+0000","last_active":"2024-03-11T19:09:54.831635+0000","last_peered":"2024-03-11T19:09:54.831635+0000","last_clean":"2024-03-11T19:09:54.831635+0000","last_became_active":"2024-03-11T19:07:48.742947+0000","last_became_peered":"2024-03-11T19:07:48.742947+0000","last_unstale":"2024-03-11T19:09:54.831635+0000","last_undegraded":"2024-03-11T19:09:54.831635+0000","last_fullsized":"2024-03-11T19:09:54.831635+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:40:25.233150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.643001+0000","last_change":"2024-03-11T19:07:49.753668+0000","last_active":"2024-03-11T19:09:13.643001+0000","last_peered":"2024-03-11T19:09:13.643001+0000","last_clean":"2024-03-11T19:09:13.643001+0000","last_became_active":"2024-03-11T19:07:49.750574+0000","last_became_peered":"2024-03-11T19:07:49.750574+0000","last_unstale":"2024-03-11T19:09:13.643001+0000","last_undegraded":"2024-03-11T19:09:13.643001+0000","last_fullsized":"2024-03-11T19:09:13.643001+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:12:32.711847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.640771+0000","last_change":"2024-03-11T19:07:49.739538+0000","last_active":"2024-03-11T19:09:13.640771+0000","last_peered":"2024-03-11T19:09:13.640771+0000","last_clean":"2024-03-11T19:09:13.640771+0000","last_became_active":"2024-03-11T19:07:49.738617+0000","last_became_peered":"2024-03-11T19:07:49.738617+0000","last_unstale":"2024-03-11T19:09:13.640771+0000","last_undegraded":"2024-03-11T19:09:13.640771+0000","last_fullsized":"2024-03-11T19:09:13.640771+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:43:08.697125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'6","reported_seq":150,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:57.835585+0000","last_change":"2024-03-11T19:07:48.749527+0000","last_active":"2024-03-11T19:09:57.835585+0000","last_peered":"2024-03-11T19:09:57.835585+0000","last_clean":"2024-03-11T19:09:57.835585+0000","last_became_active":"2024-03-11T19:07:48.749026+0000","last_became_peered":"2024-03-11T19:07:48.749026+0000","last_unstale":"2024-03-11T19:09:57.835585+0000","last_undegraded":"2024-03-11T19:09:57.835585+0000","last_fullsized":"2024-03-11T19:09:57.835585+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:20:19.058429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":124,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.646783+0000","last_change":"2024-03-11T19:07:49.757696+0000","last_active":"2024-03-11T19:09:13.646783+0000","last_peered":"2024-03-11T19:09:13.646783+0000","last_clean":"2024-03-11T19:09:13.646783+0000","last_became_active":"2024-03-11T19:07:49.757380+0000","last_became_peered":"2024-03-11T19:07:49.757380+0000","last_unstale":"2024-03-11T19:09:13.646783+0000","last_undegraded":"2024-03-11T19:09:13.646783+0000","last_fullsized":"2024-03-11T19:09:13.646783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:03:22.954619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036816+0000","last_change":"2024-03-11T19:07:48.737504+0000","last_active":"2024-03-11T19:08:45.036816+0000","last_peered":"2024-03-11T19:08:45.036816+0000","last_clean":"2024-03-11T19:08:45.036816+0000","last_became_active":"2024-03-11T19:07:48.737147+0000","last_became_peered":"2024-03-11T19:07:48.737147+0000","last_unstale":"2024-03-11T19:08:45.036816+0000","last_undegraded":"2024-03-11T19:08:45.036816+0000","last_fullsized":"2024-03-11T19:08:45.036816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:04.819617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":344,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985900+0000","last_change":"2024-03-11T19:06:35.035452+0000","last_active":"2024-03-11T19:08:44.985900+0000","last_peered":"2024-03-11T19:08:44.985900+0000","last_clean":"2024-03-11T19:08:44.985900+0000","last_became_active":"2024-03-11T19:06:35.035076+0000","last_became_peered":"2024-03-11T19:06:35.035076+0000","last_unstale":"2024-03-11T19:08:44.985900+0000","last_undegraded":"2024-03-11T19:08:44.985900+0000","last_fullsized":"2024-03-11T19:08:44.985900+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:03:10.143002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:03:10.143002+0000","last_clean_scrub_stamp":"2024-03-11T19:03:10.143002+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:39:37.348091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.642831+0000","last_change":"2024-03-11T19:07:49.743000+0000","last_active":"2024-03-11T19:09:13.642831+0000","last_peered":"2024-03-11T19:09:13.642831+0000","last_clean":"2024-03-11T19:09:13.642831+0000","last_became_active":"2024-03-11T19:07:49.742691+0000","last_became_peered":"2024-03-11T19:07:49.742691+0000","last_unstale":"2024-03-11T19:09:13.642831+0000","last_undegraded":"2024-03-11T19:09:13.642831+0000","last_fullsized":"2024-03-11T19:09:13.642831+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:40:21.519929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985719+0000","last_change":"2024-03-11T19:07:48.739038+0000","last_active":"2024-03-11T19:08:44.985719+0000","last_peered":"2024-03-11T19:08:44.985719+0000","last_clean":"2024-03-11T19:08:44.985719+0000","last_became_active":"2024-03-11T19:07:48.738714+0000","last_became_peered":"2024-03-11T19:07:48.738714+0000","last_unstale":"2024-03-11T19:08:44.985719+0000","last_undegraded":"2024-03-11T19:08:44.985719+0000","last_fullsized":"2024-03-11T19:08:44.985719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:17:43.086679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.998660+0000","last_change":"2024-03-11T19:08:44.998660+0000","last_active":"2024-03-11T19:08:44.998660+0000","last_peered":"2024-03-11T19:08:44.998660+0000","last_clean":"2024-03-11T19:08:44.998660+0000","last_became_active":"2024-03-11T19:08:44.991682+0000","last_became_peered":"2024-03-11T19:08:44.991682+0000","last_unstale":"2024-03-11T19:08:44.998660+0000","last_undegraded":"2024-03-11T19:08:44.998660+0000","last_fullsized":"2024-03-11T19:08:44.998660+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:19:44.266076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.645393+0000","last_change":"2024-03-11T19:07:49.742434+0000","last_active":"2024-03-11T19:09:13.645393+0000","last_peered":"2024-03-11T19:09:13.645393+0000","last_clean":"2024-03-11T19:09:13.645393+0000","last_became_active":"2024-03-11T19:07:49.742048+0000","last_became_peered":"2024-03-11T19:07:49.742048+0000","last_unstale":"2024-03-11T19:09:13.645393+0000","last_undegraded":"2024-03-11T19:09:13.645393+0000","last_fullsized":"2024-03-11T19:09:13.645393+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:39:22.685055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:53.757286+0000","last_change":"2024-03-11T19:07:48.744728+0000","last_active":"2024-03-11T19:09:53.757286+0000","last_peered":"2024-03-11T19:09:53.757286+0000","last_clean":"2024-03-11T19:09:53.757286+0000","last_became_active":"2024-03-11T19:07:48.744378+0000","last_became_peered":"2024-03-11T19:07:48.744378+0000","last_unstale":"2024-03-11T19:09:53.757286+0000","last_undegraded":"2024-03-11T19:09:53.757286+0000","last_fullsized":"2024-03-11T19:09:53.757286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:42.823040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.654164+0000","last_change":"2024-03-11T19:07:49.744261+0000","last_active":"2024-03-11T19:09:13.654164+0000","last_peered":"2024-03-11T19:09:13.654164+0000","last_clean":"2024-03-11T19:09:13.654164+0000","last_became_active":"2024-03-11T19:07:49.743946+0000","last_became_peered":"2024-03-11T19:07:49.743946+0000","last_unstale":"2024-03-11T19:09:13.654164+0000","last_undegraded":"2024-03-11T19:09:13.654164+0000","last_fullsized":"2024-03-11T19:09:13.654164+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:54:07.928899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'7","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037249+0000","last_change":"2024-03-11T19:07:48.740059+0000","last_active":"2024-03-11T19:08:45.037249+0000","last_peered":"2024-03-11T19:08:45.037249+0000","last_clean":"2024-03-11T19:08:45.037249+0000","last_became_active":"2024-03-11T19:07:48.739626+0000","last_became_peered":"2024-03-11T19:07:48.739626+0000","last_unstale":"2024-03-11T19:08:45.037249+0000","last_undegraded":"2024-03-11T19:08:45.037249+0000","last_fullsized":"2024-03-11T19:08:45.037249+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:45:12.219203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.647418+0000","last_change":"2024-03-11T19:07:49.746041+0000","last_active":"2024-03-11T19:09:13.647418+0000","last_peered":"2024-03-11T19:09:13.647418+0000","last_clean":"2024-03-11T19:09:13.647418+0000","last_became_active":"2024-03-11T19:07:49.745602+0000","last_became_peered":"2024-03-11T19:07:49.745602+0000","last_unstale":"2024-03-11T19:09:13.647418+0000","last_undegraded":"2024-03-11T19:09:13.647418+0000","last_fullsized":"2024-03-11T19:09:13.647418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:23:44.596929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.035863+0000","last_change":"2024-03-11T19:07:48.748677+0000","last_active":"2024-03-11T19:08:45.035863+0000","last_peered":"2024-03-11T19:08:45.035863+0000","last_clean":"2024-03-11T19:08:45.035863+0000","last_became_active":"2024-03-11T19:07:48.748470+0000","last_became_peered":"2024-03-11T19:07:48.748470+0000","last_unstale":"2024-03-11T19:08:45.035863+0000","last_undegraded":"2024-03-11T19:08:45.035863+0000","last_fullsized":"2024-03-11T19:08:45.035863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:24:55.240722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.620826+0000","last_change":"2024-03-11T19:07:49.746522+0000","last_active":"2024-03-11T19:09:13.620826+0000","last_peered":"2024-03-11T19:09:13.620826+0000","last_clean":"2024-03-11T19:09:13.620826+0000","last_became_active":"2024-03-11T19:07:49.746179+0000","last_became_peered":"2024-03-11T19:07:49.746179+0000","last_unstale":"2024-03-11T19:09:13.620826+0000","last_undegraded":"2024-03-11T19:09:13.620826+0000","last_fullsized":"2024-03-11T19:09:13.620826+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:06:08.144850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.989738+0000","last_change":"2024-03-11T19:07:48.740926+0000","last_active":"2024-03-11T19:08:44.989738+0000","last_peered":"2024-03-11T19:08:44.989738+0000","last_clean":"2024-03-11T19:08:44.989738+0000","last_became_active":"2024-03-11T19:07:48.740625+0000","last_became_peered":"2024-03-11T19:07:48.740625+0000","last_unstale":"2024-03-11T19:08:44.989738+0000","last_undegraded":"2024-03-11T19:08:44.989738+0000","last_fullsized":"2024-03-11T19:08:44.989738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:50:13.079608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.647609+0000","last_change":"2024-03-11T19:07:49.742428+0000","last_active":"2024-03-11T19:09:13.647609+0000","last_peered":"2024-03-11T19:09:13.647609+0000","last_clean":"2024-03-11T19:09:13.647609+0000","last_became_active":"2024-03-11T19:07:49.742098+0000","last_became_peered":"2024-03-11T19:07:49.742098+0000","last_unstale":"2024-03-11T19:09:13.647609+0000","last_undegraded":"2024-03-11T19:09:13.647609+0000","last_fullsized":"2024-03-11T19:09:13.647609+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:19:24.615755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'5","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:53.777258+0000","last_change":"2024-03-11T19:08:45.002189+0000","last_active":"2024-03-11T19:09:53.777258+0000","last_peered":"2024-03-11T19:09:53.777258+0000","last_clean":"2024-03-11T19:09:53.777258+0000","last_became_active":"2024-03-11T19:08:44.992818+0000","last_became_peered":"2024-03-11T19:08:44.992818+0000","last_unstale":"2024-03-11T19:09:53.777258+0000","last_undegraded":"2024-03-11T19:09:53.777258+0000","last_fullsized":"2024-03-11T19:09:53.777258+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:34:54.461734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.643596+0000","last_change":"2024-03-11T19:07:49.754872+0000","last_active":"2024-03-11T19:09:13.643596+0000","last_peered":"2024-03-11T19:09:13.643596+0000","last_clean":"2024-03-11T19:09:13.643596+0000","last_became_active":"2024-03-11T19:07:49.749663+0000","last_became_peered":"2024-03-11T19:07:49.749663+0000","last_unstale":"2024-03-11T19:09:13.643596+0000","last_undegraded":"2024-03-11T19:09:13.643596+0000","last_fullsized":"2024-03-11T19:09:13.643596+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:22:23.664814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990526+0000","last_change":"2024-03-11T19:07:48.742227+0000","last_active":"2024-03-11T19:08:44.990526+0000","last_peered":"2024-03-11T19:08:44.990526+0000","last_clean":"2024-03-11T19:08:44.990526+0000","last_became_active":"2024-03-11T19:07:48.741912+0000","last_became_peered":"2024-03-11T19:07:48.741912+0000","last_unstale":"2024-03-11T19:08:44.990526+0000","last_undegraded":"2024-03-11T19:08:44.990526+0000","last_fullsized":"2024-03-11T19:08:44.990526+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:10:06.120433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.657986+0000","last_change":"2024-03-11T19:07:49.748947+0000","last_active":"2024-03-11T19:09:13.657986+0000","last_peered":"2024-03-11T19:09:13.657986+0000","last_clean":"2024-03-11T19:09:13.657986+0000","last_became_active":"2024-03-11T19:07:49.748621+0000","last_became_peered":"2024-03-11T19:07:49.748621+0000","last_unstale":"2024-03-11T19:09:13.657986+0000","last_undegraded":"2024-03-11T19:09:13.657986+0000","last_fullsized":"2024-03-11T19:09:13.657986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:19:53.824590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'6","reported_seq":150,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:56.834864+0000","last_change":"2024-03-11T19:07:48.748799+0000","last_active":"2024-03-11T19:09:56.834864+0000","last_peered":"2024-03-11T19:09:56.834864+0000","last_clean":"2024-03-11T19:09:56.834864+0000","last_became_active":"2024-03-11T19:07:48.748434+0000","last_became_peered":"2024-03-11T19:07:48.748434+0000","last_unstale":"2024-03-11T19:09:56.834864+0000","last_undegraded":"2024-03-11T19:09:56.834864+0000","last_fullsized":"2024-03-11T19:09:56.834864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:10:09.836926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":125,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.657255+0000","last_change":"2024-03-11T19:07:49.750122+0000","last_active":"2024-03-11T19:09:13.657255+0000","last_peered":"2024-03-11T19:09:13.657255+0000","last_clean":"2024-03-11T19:09:13.657255+0000","last_became_active":"2024-03-11T19:07:49.749801+0000","last_became_peered":"2024-03-11T19:07:49.749801+0000","last_unstale":"2024-03-11T19:09:13.657255+0000","last_undegraded":"2024-03-11T19:09:13.657255+0000","last_fullsized":"2024-03-11T19:09:13.657255+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:58:31.155275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036439+0000","last_change":"2024-03-11T19:07:48.741959+0000","last_active":"2024-03-11T19:08:45.036439+0000","last_peered":"2024-03-11T19:08:45.036439+0000","last_clean":"2024-03-11T19:08:45.036439+0000","last_became_active":"2024-03-11T19:07:48.741630+0000","last_became_peered":"2024-03-11T19:07:48.741630+0000","last_unstale":"2024-03-11T19:08:45.036439+0000","last_undegraded":"2024-03-11T19:08:45.036439+0000","last_fullsized":"2024-03-11T19:08:45.036439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:27:08.606055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.652383+0000","last_change":"2024-03-11T19:07:49.729838+0000","last_active":"2024-03-11T19:09:13.652383+0000","last_peered":"2024-03-11T19:09:13.652383+0000","last_clean":"2024-03-11T19:09:13.652383+0000","last_became_active":"2024-03-11T19:07:49.728912+0000","last_became_peered":"2024-03-11T19:07:49.728912+0000","last_unstale":"2024-03-11T19:09:13.652383+0000","last_undegraded":"2024-03-11T19:09:13.652383+0000","last_fullsized":"2024-03-11T19:09:13.652383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:02:06.551734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.989032+0000","last_change":"2024-03-11T19:07:48.744709+0000","last_active":"2024-03-11T19:08:44.989032+0000","last_peered":"2024-03-11T19:08:44.989032+0000","last_clean":"2024-03-11T19:08:44.989032+0000","last_became_active":"2024-03-11T19:07:48.744356+0000","last_became_peered":"2024-03-11T19:07:48.744356+0000","last_unstale":"2024-03-11T19:08:44.989032+0000","last_undegraded":"2024-03-11T19:08:44.989032+0000","last_fullsized":"2024-03-11T19:08:44.989032+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:48:47.098443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.652239+0000","last_change":"2024-03-11T19:07:49.743288+0000","last_active":"2024-03-11T19:09:13.652239+0000","last_peered":"2024-03-11T19:09:13.652239+0000","last_clean":"2024-03-11T19:09:13.652239+0000","last_became_active":"2024-03-11T19:07:49.742942+0000","last_became_peered":"2024-03-11T19:07:49.742942+0000","last_unstale":"2024-03-11T19:09:13.652239+0000","last_undegraded":"2024-03-11T19:09:13.652239+0000","last_fullsized":"2024-03-11T19:09:13.652239+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:21:53.896192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.645073+0000","last_change":"2024-03-11T19:07:49.741610+0000","last_active":"2024-03-11T19:09:13.645073+0000","last_peered":"2024-03-11T19:09:13.645073+0000","last_clean":"2024-03-11T19:09:13.645073+0000","last_became_active":"2024-03-11T19:07:49.741234+0000","last_became_peered":"2024-03-11T19:07:49.741234+0000","last_unstale":"2024-03-11T19:09:13.645073+0000","last_undegraded":"2024-03-11T19:09:13.645073+0000","last_fullsized":"2024-03-11T19:09:13.645073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:53:03.320093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.638525+0000","last_change":"2024-03-11T19:08:45.037715+0000","last_active":"2024-03-11T19:09:13.638525+0000","last_peered":"2024-03-11T19:09:13.638525+0000","last_clean":"2024-03-11T19:09:13.638525+0000","last_became_active":"2024-03-11T19:08:45.037299+0000","last_became_peered":"2024-03-11T19:08:45.037299+0000","last_unstale":"2024-03-11T19:09:13.638525+0000","last_undegraded":"2024-03-11T19:09:13.638525+0000","last_fullsized":"2024-03-11T19:09:13.638525+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:01:49.025379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.645173+0000","last_change":"2024-03-11T19:07:49.745928+0000","last_active":"2024-03-11T19:09:13.645173+0000","last_peered":"2024-03-11T19:09:13.645173+0000","last_clean":"2024-03-11T19:09:13.645173+0000","last_became_active":"2024-03-11T19:07:49.745025+0000","last_became_peered":"2024-03-11T19:07:49.745025+0000","last_unstale":"2024-03-11T19:09:13.645173+0000","last_undegraded":"2024-03-11T19:09:13.645173+0000","last_fullsized":"2024-03-11T19:09:13.645173+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:40:30.141906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.648364+0000","last_change":"2024-03-11T19:07:49.756274+0000","last_active":"2024-03-11T19:09:13.648364+0000","last_peered":"2024-03-11T19:09:13.648364+0000","last_clean":"2024-03-11T19:09:13.648364+0000","last_became_active":"2024-03-11T19:07:49.755954+0000","last_became_peered":"2024-03-11T19:07:49.755954+0000","last_unstale":"2024-03-11T19:09:13.648364+0000","last_undegraded":"2024-03-11T19:09:13.648364+0000","last_fullsized":"2024-03-11T19:09:13.648364+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:15:36.825240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.639818+0000","last_change":"2024-03-11T19:08:44.992154+0000","last_active":"2024-03-11T19:09:13.639818+0000","last_peered":"2024-03-11T19:09:13.639818+0000","last_clean":"2024-03-11T19:09:13.639818+0000","last_became_active":"2024-03-11T19:08:44.991772+0000","last_became_peered":"2024-03-11T19:08:44.991772+0000","last_unstale":"2024-03-11T19:09:13.639818+0000","last_undegraded":"2024-03-11T19:09:13.639818+0000","last_fullsized":"2024-03-11T19:09:13.639818+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:10:33.882095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.641105+0000","last_change":"2024-03-11T19:07:49.743963+0000","last_active":"2024-03-11T19:09:13.641105+0000","last_peered":"2024-03-11T19:09:13.641105+0000","last_clean":"2024-03-11T19:09:13.641105+0000","last_became_active":"2024-03-11T19:07:49.743568+0000","last_became_peered":"2024-03-11T19:07:49.743568+0000","last_unstale":"2024-03-11T19:09:13.641105+0000","last_undegraded":"2024-03-11T19:09:13.641105+0000","last_fullsized":"2024-03-11T19:09:13.641105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:45:28.970286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.654180+0000","last_change":"2024-03-11T19:07:49.750677+0000","last_active":"2024-03-11T19:09:13.654180+0000","last_peered":"2024-03-11T19:09:13.654180+0000","last_clean":"2024-03-11T19:09:13.654180+0000","last_became_active":"2024-03-11T19:07:49.750373+0000","last_became_peered":"2024-03-11T19:07:49.750373+0000","last_unstale":"2024-03-11T19:09:13.654180+0000","last_undegraded":"2024-03-11T19:09:13.654180+0000","last_fullsized":"2024-03-11T19:09:13.654180+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:07:28.767502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.645768+0000","last_change":"2024-03-11T19:07:49.745278+0000","last_active":"2024-03-11T19:09:13.645768+0000","last_peered":"2024-03-11T19:09:13.645768+0000","last_clean":"2024-03-11T19:09:13.645768+0000","last_became_active":"2024-03-11T19:07:49.744998+0000","last_became_peered":"2024-03-11T19:07:49.744998+0000","last_unstale":"2024-03-11T19:09:13.645768+0000","last_undegraded":"2024-03-11T19:09:13.645768+0000","last_fullsized":"2024-03-11T19:09:13.645768+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:23.778051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.647869+0000","last_change":"2024-03-11T19:07:49.734318+0000","last_active":"2024-03-11T19:09:13.647869+0000","last_peered":"2024-03-11T19:09:13.647869+0000","last_clean":"2024-03-11T19:09:13.647869+0000","last_became_active":"2024-03-11T19:07:49.733998+0000","last_became_peered":"2024-03-11T19:07:49.733998+0000","last_unstale":"2024-03-11T19:09:13.647869+0000","last_undegraded":"2024-03-11T19:09:13.647869+0000","last_fullsized":"2024-03-11T19:09:13.647869+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:48:50.308499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.646683+0000","last_change":"2024-03-11T19:07:49.741009+0000","last_active":"2024-03-11T19:09:13.646683+0000","last_peered":"2024-03-11T19:09:13.646683+0000","last_clean":"2024-03-11T19:09:13.646683+0000","last_became_active":"2024-03-11T19:07:49.740094+0000","last_became_peered":"2024-03-11T19:07:49.740094+0000","last_unstale":"2024-03-11T19:09:13.646683+0000","last_undegraded":"2024-03-11T19:09:13.646683+0000","last_fullsized":"2024-03-11T19:09:13.646683+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:06:31.945387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.657556+0000","last_change":"2024-03-11T19:07:49.759300+0000","last_active":"2024-03-11T19:09:13.657556+0000","last_peered":"2024-03-11T19:09:13.657556+0000","last_clean":"2024-03-11T19:09:13.657556+0000","last_became_active":"2024-03-11T19:07:49.758011+0000","last_became_peered":"2024-03-11T19:07:49.758011+0000","last_unstale":"2024-03-11T19:09:13.657556+0000","last_undegraded":"2024-03-11T19:09:13.657556+0000","last_fullsized":"2024-03-11T19:09:13.657556+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:03:40.651816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.652991+0000","last_change":"2024-03-11T19:07:49.752905+0000","last_active":"2024-03-11T19:09:13.652991+0000","last_peered":"2024-03-11T19:09:13.652991+0000","last_clean":"2024-03-11T19:09:13.652991+0000","last_became_active":"2024-03-11T19:07:49.752383+0000","last_became_peered":"2024-03-11T19:07:49.752383+0000","last_unstale":"2024-03-11T19:09:13.652991+0000","last_undegraded":"2024-03-11T19:09:13.652991+0000","last_fullsized":"2024-03-11T19:09:13.652991+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:45:33.475526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.637754+0000","last_change":"2024-03-11T19:07:49.744957+0000","last_active":"2024-03-11T19:09:13.637754+0000","last_peered":"2024-03-11T19:09:13.637754+0000","last_clean":"2024-03-11T19:09:13.637754+0000","last_became_active":"2024-03-11T19:07:49.744627+0000","last_became_peered":"2024-03-11T19:07:49.744627+0000","last_unstale":"2024-03-11T19:09:13.637754+0000","last_undegraded":"2024-03-11T19:09:13.637754+0000","last_fullsized":"2024-03-11T19:09:13.637754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:27:47.817004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245707,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":75419,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1258,"num_read_kb":711,"num_write":266,"num_write_kb":159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":569344,"data_stored":221943,"data_compressed":5565,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318312,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":464775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579948,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":1122185,"data_compressed":37658,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743474,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":464100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005111,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":464366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266748,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":1058616,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430274,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":464701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":527011,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":465649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1064960,"data_stored":532961,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149776,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":463642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888140,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":464171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691911,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":1057308,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":64033,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":69983,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":68722,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T19:09:59.810 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-11T19:09:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.945+0000 7f7a70b16640 1 -- 172.21.15.29:0/2416744409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a6c106e80 msgr2=0x7f7a6c1092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.945+0000 7f7a70b16640 1 --2- 172.21.15.29:0/2416744409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a6c106e80 0x7f7a6c1092d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7a64009ca0 tx=0x7f7a6402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.945+0000 7f7a70b16640 1 -- 172.21.15.29:0/2416744409 shutdown_connections 2024-03-11T19:09:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.945+0000 7f7a70b16640 1 --2- 172.21.15.29:0/2416744409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a6c106e80 0x7f7a6c1092d0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.945+0000 7f7a70b16640 1 --2- 172.21.15.29:0/2416744409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a6c104550 0x7f7a6c106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.945+0000 7f7a70b16640 1 --2- 172.21.15.29:0/2416744409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a6c101c20 0x7f7a6c104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.945+0000 7f7a70b16640 1 -- 172.21.15.29:0/2416744409 >> 172.21.15.29:0/2416744409 conn(0x7f7a6c0fb610 msgr2=0x7f7a6c0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:09:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.945+0000 7f7a70b16640 1 -- 172.21.15.29:0/2416744409 shutdown_connections 2024-03-11T19:09:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.945+0000 7f7a70b16640 1 -- 172.21.15.29:0/2416744409 wait complete. 2024-03-11T19:09:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.946+0000 7f7a70b16640 1 Processor -- start 2024-03-11T19:09:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.946+0000 7f7a70b16640 1 -- start start 2024-03-11T19:09:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.946+0000 7f7a70b16640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a6c101c20 0x7f7a6c197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.946+0000 7f7a70b16640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a6c104550 0x7f7a6c197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.946+0000 7f7a70b16640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a6c106e80 0x7f7a6c19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.946+0000 7f7a70b16640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c19ec20 con 0x7f7a6c106e80 2024-03-11T19:09:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.946+0000 7f7a70b16640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c19ed90 con 0x7f7a6c101c20 2024-03-11T19:09:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.946+0000 7f7a70b16640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c19ef50 con 0x7f7a6c104550 2024-03-11T19:09:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.946+0000 7f7a6ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a6c106e80 0x7f7a6c19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.946+0000 7f7a6ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a6c106e80 0x7f7a6c19e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54492/0 (socket says 172.21.15.29:54492) 2024-03-11T19:09:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.946+0000 7f7a6ad9d640 1 -- 172.21.15.29:0/1255987888 learned_addr learned my addr 172.21.15.29:0/1255987888 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:09:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.947+0000 7f7a69d9b640 1 --2- 172.21.15.29:0/1255987888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a6c104550 0x7f7a6c197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.947+0000 7f7a6a59c640 1 --2- 172.21.15.29:0/1255987888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a6c101c20 0x7f7a6c197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.947+0000 7f7a6ad9d640 1 -- 172.21.15.29:0/1255987888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a6c104550 msgr2=0x7f7a6c197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.947+0000 7f7a6ad9d640 1 --2- 172.21.15.29:0/1255987888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a6c104550 0x7f7a6c197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.947+0000 7f7a6ad9d640 1 -- 172.21.15.29:0/1255987888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a6c101c20 msgr2=0x7f7a6c197130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:09:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.947+0000 7f7a6ad9d640 1 --2- 172.21.15.29:0/1255987888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a6c101c20 0x7f7a6c197130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:09:59.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.947+0000 7f7a6ad9d640 1 -- 172.21.15.29:0/1255987888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a64009900 con 0x7f7a6c106e80 2024-03-11T19:09:59.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.947+0000 7f7a6ad9d640 1 --2- 172.21.15.29:0/1255987888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a6c106e80 0x7f7a6c19e6e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7a64037040 tx=0x7f7a64032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:59.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.947+0000 7f7a637fe640 1 -- 172.21.15.29:0/1255987888 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a64034e70 con 0x7f7a6c106e80 2024-03-11T19:09:59.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.947+0000 7f7a637fe640 1 -- 172.21.15.29:0/1255987888 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7a64003e20 con 0x7f7a6c106e80 2024-03-11T19:09:59.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.947+0000 7f7a70b16640 1 -- 172.21.15.29:0/1255987888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6c19f1d0 con 0x7f7a6c106e80 2024-03-11T19:09:59.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.947+0000 7f7a637fe640 1 -- 172.21.15.29:0/1255987888 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a64004800 con 0x7f7a6c106e80 2024-03-11T19:09:59.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.947+0000 7f7a70b16640 1 -- 172.21.15.29:0/1255987888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6c19f680 con 0x7f7a6c106e80 2024-03-11T19:09:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.948+0000 7f7a70b16640 1 -- 172.21.15.29:0/1255987888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6c0fd100 con 0x7f7a6c106e80 2024-03-11T19:09:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.949+0000 7f7a637fe640 1 -- 172.21.15.29:0/1255987888 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f7a6403b040 con 0x7f7a6c106e80 2024-03-11T19:09:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.950+0000 7f7a637fe640 1 --2- 172.21.15.29:0/1255987888 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7a580a1ac0 0x7f7a580a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:09:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.950+0000 7f7a6a59c640 1 --2- 172.21.15.29:0/1255987888 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7a580a1ac0 0x7f7a580a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:09:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.950+0000 7f7a637fe640 1 -- 172.21.15.29:0/1255987888 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7a640efb90 con 0x7f7a6c106e80 2024-03-11T19:09:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.950+0000 7f7a6a59c640 1 --2- 172.21.15.29:0/1255987888 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7a580a1ac0 0x7f7a580a3f80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009b60 tx=0x7f7a5c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:09:59.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:09:59.954+0000 7f7a637fe640 1 -- 172.21.15.29:0/1255987888 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7a640b8e50 con 0x7f7a6c106e80 2024-03-11T19:10:00.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.134+0000 7f7a70b16640 1 -- 172.21.15.29:0/1255987888 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7a6c02ceb0 con 0x7f7a580a1ac0 2024-03-11T19:10:00.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.142+0000 7f7a637fe640 1 -- 172.21.15.29:0/1255987888 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259094 (secure 0 0 0) 0x7f7a6c02ceb0 con 0x7f7a580a1ac0 2024-03-11T19:10:00.147 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:10:00.148 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-11T19:10:00.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.144+0000 7f7a70b16640 1 -- 172.21.15.29:0/1255987888 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7a580a1ac0 msgr2=0x7f7a580a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:00.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.144+0000 7f7a70b16640 1 --2- 172.21.15.29:0/1255987888 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7a580a1ac0 0x7f7a580a3f80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009b60 tx=0x7f7a5c005f90 comp rx=0 tx=0).stop 2024-03-11T19:10:00.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.144+0000 7f7a70b16640 1 -- 172.21.15.29:0/1255987888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a6c106e80 msgr2=0x7f7a6c19e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:00.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.144+0000 7f7a70b16640 1 --2- 172.21.15.29:0/1255987888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a6c106e80 0x7f7a6c19e6e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7a64037040 tx=0x7f7a64032980 comp rx=0 tx=0).stop 2024-03-11T19:10:00.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.144+0000 7f7a70b16640 1 -- 172.21.15.29:0/1255987888 shutdown_connections 2024-03-11T19:10:00.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.144+0000 7f7a70b16640 1 --2- 172.21.15.29:0/1255987888 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7a580a1ac0 0x7f7a580a3f80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.144+0000 7f7a70b16640 1 --2- 172.21.15.29:0/1255987888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a6c106e80 0x7f7a6c19e6e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.144+0000 7f7a70b16640 1 --2- 172.21.15.29:0/1255987888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a6c104550 0x7f7a6c197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.144+0000 7f7a70b16640 1 --2- 172.21.15.29:0/1255987888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a6c101c20 0x7f7a6c197130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.144+0000 7f7a70b16640 1 -- 172.21.15.29:0/1255987888 >> 172.21.15.29:0/1255987888 conn(0x7f7a6c0fb610 msgr2=0x7f7a6c1024e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:00.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.145+0000 7f7a70b16640 1 -- 172.21.15.29:0/1255987888 shutdown_connections 2024-03-11T19:10:00.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.145+0000 7f7a70b16640 1 -- 172.21.15.29:0/1255987888 wait complete. 2024-03-11T19:10:00.162 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-03-11T19:09:59.636214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666021,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1384,"num_read_kb":820,"num_write":449,"num_write_kb":3099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":394388,"kb_used_data":11956,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124547436,"statfs":{"total":1151940427776,"available":1151536574464,"internally_reserved":0,"allocated":12242944,"data_stored":7549485,"data_compressed":112974,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":391492922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7170,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":127,"num_read_kb":66,"num_write":22,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002894"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.650725+0000","last_change":"2024-03-11T19:07:49.751738+0000","last_active":"2024-03-11T19:09:13.650725+0000","last_peered":"2024-03-11T19:09:13.650725+0000","last_clean":"2024-03-11T19:09:13.650725+0000","last_became_active":"2024-03-11T19:07:49.751173+0000","last_became_peered":"2024-03-11T19:07:49.751173+0000","last_unstale":"2024-03-11T19:09:13.650725+0000","last_undegraded":"2024-03-11T19:09:13.650725+0000","last_fullsized":"2024-03-11T19:09:13.650725+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:13:40.157763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.648231+0000","last_change":"2024-03-11T19:07:49.749069+0000","last_active":"2024-03-11T19:09:13.648231+0000","last_peered":"2024-03-11T19:09:13.648231+0000","last_clean":"2024-03-11T19:09:13.648231+0000","last_became_active":"2024-03-11T19:07:49.747672+0000","last_became_peered":"2024-03-11T19:07:49.747672+0000","last_unstale":"2024-03-11T19:09:13.648231+0000","last_undegraded":"2024-03-11T19:09:13.648231+0000","last_fullsized":"2024-03-11T19:09:13.648231+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:57:26.379836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.653423+0000","last_change":"2024-03-11T19:07:49.750298+0000","last_active":"2024-03-11T19:09:13.653423+0000","last_peered":"2024-03-11T19:09:13.653423+0000","last_clean":"2024-03-11T19:09:13.653423+0000","last_became_active":"2024-03-11T19:07:49.749985+0000","last_became_peered":"2024-03-11T19:07:49.749985+0000","last_unstale":"2024-03-11T19:09:13.653423+0000","last_undegraded":"2024-03-11T19:09:13.653423+0000","last_fullsized":"2024-03-11T19:09:13.653423+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:56.579872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.642014+0000","last_change":"2024-03-11T19:08:44.993047+0000","last_active":"2024-03-11T19:09:13.642014+0000","last_peered":"2024-03-11T19:09:13.642014+0000","last_clean":"2024-03-11T19:09:13.642014+0000","last_became_active":"2024-03-11T19:08:44.992666+0000","last_became_peered":"2024-03-11T19:08:44.992666+0000","last_unstale":"2024-03-11T19:09:13.642014+0000","last_undegraded":"2024-03-11T19:09:13.642014+0000","last_fullsized":"2024-03-11T19:09:13.642014+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:32:18.748906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.634551+0000","last_change":"2024-03-11T19:07:49.747618+0000","last_active":"2024-03-11T19:09:13.634551+0000","last_peered":"2024-03-11T19:09:13.634551+0000","last_clean":"2024-03-11T19:09:13.634551+0000","last_became_active":"2024-03-11T19:07:49.747310+0000","last_became_peered":"2024-03-11T19:07:49.747310+0000","last_unstale":"2024-03-11T19:09:13.634551+0000","last_undegraded":"2024-03-11T19:09:13.634551+0000","last_fullsized":"2024-03-11T19:09:13.634551+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:15:42.260509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.635452+0000","last_change":"2024-03-11T19:07:49.755564+0000","last_active":"2024-03-11T19:09:13.635452+0000","last_peered":"2024-03-11T19:09:13.635452+0000","last_clean":"2024-03-11T19:09:13.635452+0000","last_became_active":"2024-03-11T19:07:49.751322+0000","last_became_peered":"2024-03-11T19:07:49.751322+0000","last_unstale":"2024-03-11T19:09:13.635452+0000","last_undegraded":"2024-03-11T19:09:13.635452+0000","last_fullsized":"2024-03-11T19:09:13.635452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:43:18.487887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.637740+0000","last_change":"2024-03-11T19:07:49.731619+0000","last_active":"2024-03-11T19:09:13.637740+0000","last_peered":"2024-03-11T19:09:13.637740+0000","last_clean":"2024-03-11T19:09:13.637740+0000","last_became_active":"2024-03-11T19:07:49.731323+0000","last_became_peered":"2024-03-11T19:07:49.731323+0000","last_unstale":"2024-03-11T19:09:13.637740+0000","last_undegraded":"2024-03-11T19:09:13.637740+0000","last_fullsized":"2024-03-11T19:09:13.637740+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:20:25.592709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.646995+0000","last_change":"2024-03-11T19:07:49.751851+0000","last_active":"2024-03-11T19:09:13.646995+0000","last_peered":"2024-03-11T19:09:13.646995+0000","last_clean":"2024-03-11T19:09:13.646995+0000","last_became_active":"2024-03-11T19:07:49.750068+0000","last_became_peered":"2024-03-11T19:07:49.750068+0000","last_unstale":"2024-03-11T19:09:13.646995+0000","last_undegraded":"2024-03-11T19:09:13.646995+0000","last_fullsized":"2024-03-11T19:09:13.646995+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:59:35.820107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.652310+0000","last_change":"2024-03-11T19:07:49.752326+0000","last_active":"2024-03-11T19:09:13.652310+0000","last_peered":"2024-03-11T19:09:13.652310+0000","last_clean":"2024-03-11T19:09:13.652310+0000","last_became_active":"2024-03-11T19:07:49.748052+0000","last_became_peered":"2024-03-11T19:07:49.748052+0000","last_unstale":"2024-03-11T19:09:13.652310+0000","last_undegraded":"2024-03-11T19:09:13.652310+0000","last_fullsized":"2024-03-11T19:09:13.652310+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:58:13.325964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.639621+0000","last_change":"2024-03-11T19:07:49.748054+0000","last_active":"2024-03-11T19:09:13.639621+0000","last_peered":"2024-03-11T19:09:13.639621+0000","last_clean":"2024-03-11T19:09:13.639621+0000","last_became_active":"2024-03-11T19:07:49.747765+0000","last_became_peered":"2024-03-11T19:07:49.747765+0000","last_unstale":"2024-03-11T19:09:13.639621+0000","last_undegraded":"2024-03-11T19:09:13.639621+0000","last_fullsized":"2024-03-11T19:09:13.639621+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:18:05.599512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.651823+0000","last_change":"2024-03-11T19:07:49.751622+0000","last_active":"2024-03-11T19:09:13.651823+0000","last_peered":"2024-03-11T19:09:13.651823+0000","last_clean":"2024-03-11T19:09:13.651823+0000","last_became_active":"2024-03-11T19:07:49.751317+0000","last_became_peered":"2024-03-11T19:07:49.751317+0000","last_unstale":"2024-03-11T19:09:13.651823+0000","last_undegraded":"2024-03-11T19:09:13.651823+0000","last_fullsized":"2024-03-11T19:09:13.651823+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:49.836096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.630547+0000","last_change":"2024-03-11T19:07:49.749160+0000","last_active":"2024-03-11T19:09:13.630547+0000","last_peered":"2024-03-11T19:09:13.630547+0000","last_clean":"2024-03-11T19:09:13.630547+0000","last_became_active":"2024-03-11T19:07:49.748838+0000","last_became_peered":"2024-03-11T19:07:49.748838+0000","last_unstale":"2024-03-11T19:09:13.630547+0000","last_undegraded":"2024-03-11T19:09:13.630547+0000","last_fullsized":"2024-03-11T19:09:13.630547+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:30:58.327196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.647032+0000","last_change":"2024-03-11T19:07:49.749965+0000","last_active":"2024-03-11T19:09:13.647032+0000","last_peered":"2024-03-11T19:09:13.647032+0000","last_clean":"2024-03-11T19:09:13.647032+0000","last_became_active":"2024-03-11T19:07:49.749602+0000","last_became_peered":"2024-03-11T19:07:49.749602+0000","last_unstale":"2024-03-11T19:09:13.647032+0000","last_undegraded":"2024-03-11T19:09:13.647032+0000","last_fullsized":"2024-03-11T19:09:13.647032+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:36:08.807251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.640585+0000","last_change":"2024-03-11T19:07:49.747557+0000","last_active":"2024-03-11T19:09:13.640585+0000","last_peered":"2024-03-11T19:09:13.640585+0000","last_clean":"2024-03-11T19:09:13.640585+0000","last_became_active":"2024-03-11T19:07:49.745804+0000","last_became_peered":"2024-03-11T19:07:49.745804+0000","last_unstale":"2024-03-11T19:09:13.640585+0000","last_undegraded":"2024-03-11T19:09:13.640585+0000","last_fullsized":"2024-03-11T19:09:13.640585+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:07:26.939414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.649130+0000","last_change":"2024-03-11T19:07:49.744070+0000","last_active":"2024-03-11T19:09:13.649130+0000","last_peered":"2024-03-11T19:09:13.649130+0000","last_clean":"2024-03-11T19:09:13.649130+0000","last_became_active":"2024-03-11T19:07:49.743736+0000","last_became_peered":"2024-03-11T19:07:49.743736+0000","last_unstale":"2024-03-11T19:09:13.649130+0000","last_undegraded":"2024-03-11T19:09:13.649130+0000","last_fullsized":"2024-03-11T19:09:13.649130+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:50:33.862644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.657246+0000","last_change":"2024-03-11T19:07:49.748512+0000","last_active":"2024-03-11T19:09:13.657246+0000","last_peered":"2024-03-11T19:09:13.657246+0000","last_clean":"2024-03-11T19:09:13.657246+0000","last_became_active":"2024-03-11T19:07:49.748223+0000","last_became_peered":"2024-03-11T19:07:49.748223+0000","last_unstale":"2024-03-11T19:09:13.657246+0000","last_undegraded":"2024-03-11T19:09:13.657246+0000","last_fullsized":"2024-03-11T19:09:13.657246+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:49:43.305677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.655548+0000","last_change":"2024-03-11T19:07:49.754322+0000","last_active":"2024-03-11T19:09:13.655548+0000","last_peered":"2024-03-11T19:09:13.655548+0000","last_clean":"2024-03-11T19:09:13.655548+0000","last_became_active":"2024-03-11T19:07:49.753757+0000","last_became_peered":"2024-03-11T19:07:49.753757+0000","last_unstale":"2024-03-11T19:09:13.655548+0000","last_undegraded":"2024-03-11T19:09:13.655548+0000","last_fullsized":"2024-03-11T19:09:13.655548+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:49.684978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.657358+0000","last_change":"2024-03-11T19:07:49.749546+0000","last_active":"2024-03-11T19:09:13.657358+0000","last_peered":"2024-03-11T19:09:13.657358+0000","last_clean":"2024-03-11T19:09:13.657358+0000","last_became_active":"2024-03-11T19:07:49.749200+0000","last_became_peered":"2024-03-11T19:07:49.749200+0000","last_unstale":"2024-03-11T19:09:13.657358+0000","last_undegraded":"2024-03-11T19:09:13.657358+0000","last_fullsized":"2024-03-11T19:09:13.657358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:26:12.914015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.645227+0000","last_change":"2024-03-11T19:07:49.750388+0000","last_active":"2024-03-11T19:09:13.645227+0000","last_peered":"2024-03-11T19:09:13.645227+0000","last_clean":"2024-03-11T19:09:13.645227+0000","last_became_active":"2024-03-11T19:07:49.750016+0000","last_became_peered":"2024-03-11T19:07:49.750016+0000","last_unstale":"2024-03-11T19:09:13.645227+0000","last_undegraded":"2024-03-11T19:09:13.645227+0000","last_fullsized":"2024-03-11T19:09:13.645227+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:32:23.125695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'14","reported_seq":155,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:55.832399+0000","last_change":"2024-03-11T19:07:48.738369+0000","last_active":"2024-03-11T19:09:55.832399+0000","last_peered":"2024-03-11T19:09:55.832399+0000","last_clean":"2024-03-11T19:09:55.832399+0000","last_became_active":"2024-03-11T19:07:48.738010+0000","last_became_peered":"2024-03-11T19:07:48.738010+0000","last_unstale":"2024-03-11T19:09:55.832399+0000","last_undegraded":"2024-03-11T19:09:55.832399+0000","last_fullsized":"2024-03-11T19:09:55.832399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:06:48.446454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.646494+0000","last_change":"2024-03-11T19:07:49.752622+0000","last_active":"2024-03-11T19:09:13.646494+0000","last_peered":"2024-03-11T19:09:13.646494+0000","last_clean":"2024-03-11T19:09:13.646494+0000","last_became_active":"2024-03-11T19:07:49.752242+0000","last_became_peered":"2024-03-11T19:07:49.752242+0000","last_unstale":"2024-03-11T19:09:13.646494+0000","last_undegraded":"2024-03-11T19:09:13.646494+0000","last_fullsized":"2024-03-11T19:09:13.646494+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:03:50.609641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990514+0000","last_change":"2024-03-11T19:07:48.751657+0000","last_active":"2024-03-11T19:08:44.990514+0000","last_peered":"2024-03-11T19:08:44.990514+0000","last_clean":"2024-03-11T19:08:44.990514+0000","last_became_active":"2024-03-11T19:07:48.751294+0000","last_became_peered":"2024-03-11T19:07:48.751294+0000","last_unstale":"2024-03-11T19:08:44.990514+0000","last_undegraded":"2024-03-11T19:08:44.990514+0000","last_fullsized":"2024-03-11T19:08:44.990514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:33:58.550707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.655152+0000","last_change":"2024-03-11T19:07:49.759460+0000","last_active":"2024-03-11T19:09:13.655152+0000","last_peered":"2024-03-11T19:09:13.655152+0000","last_clean":"2024-03-11T19:09:13.655152+0000","last_became_active":"2024-03-11T19:07:49.759136+0000","last_became_peered":"2024-03-11T19:07:49.759136+0000","last_unstale":"2024-03-11T19:09:13.655152+0000","last_undegraded":"2024-03-11T19:09:13.655152+0000","last_fullsized":"2024-03-11T19:09:13.655152+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:10:37.273896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'12","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991106+0000","last_change":"2024-03-11T19:07:48.744695+0000","last_active":"2024-03-11T19:08:44.991106+0000","last_peered":"2024-03-11T19:08:44.991106+0000","last_clean":"2024-03-11T19:08:44.991106+0000","last_became_active":"2024-03-11T19:07:48.744378+0000","last_became_peered":"2024-03-11T19:07:48.744378+0000","last_unstale":"2024-03-11T19:08:44.991106+0000","last_undegraded":"2024-03-11T19:08:44.991106+0000","last_fullsized":"2024-03-11T19:08:44.991106+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:55:46.601414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.655586+0000","last_change":"2024-03-11T19:07:49.761429+0000","last_active":"2024-03-11T19:09:13.655586+0000","last_peered":"2024-03-11T19:09:13.655586+0000","last_clean":"2024-03-11T19:09:13.655586+0000","last_became_active":"2024-03-11T19:07:49.761120+0000","last_became_peered":"2024-03-11T19:07:49.761120+0000","last_unstale":"2024-03-11T19:09:13.655586+0000","last_undegraded":"2024-03-11T19:09:13.655586+0000","last_fullsized":"2024-03-11T19:09:13.655586+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:13:45.839528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990713+0000","last_change":"2024-03-11T19:07:48.740451+0000","last_active":"2024-03-11T19:08:44.990713+0000","last_peered":"2024-03-11T19:08:44.990713+0000","last_clean":"2024-03-11T19:08:44.990713+0000","last_became_active":"2024-03-11T19:07:48.740097+0000","last_became_peered":"2024-03-11T19:07:48.740097+0000","last_unstale":"2024-03-11T19:08:44.990713+0000","last_undegraded":"2024-03-11T19:08:44.990713+0000","last_fullsized":"2024-03-11T19:08:44.990713+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:37.766790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.657267+0000","last_change":"2024-03-11T19:07:49.750404+0000","last_active":"2024-03-11T19:09:13.657267+0000","last_peered":"2024-03-11T19:09:13.657267+0000","last_clean":"2024-03-11T19:09:13.657267+0000","last_became_active":"2024-03-11T19:07:49.749941+0000","last_became_peered":"2024-03-11T19:07:49.749941+0000","last_unstale":"2024-03-11T19:09:13.657267+0000","last_undegraded":"2024-03-11T19:09:13.657267+0000","last_fullsized":"2024-03-11T19:09:13.657267+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:55.766729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'8","reported_seq":153,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:56.835200+0000","last_change":"2024-03-11T19:07:48.737100+0000","last_active":"2024-03-11T19:09:56.835200+0000","last_peered":"2024-03-11T19:09:56.835200+0000","last_clean":"2024-03-11T19:09:56.835200+0000","last_became_active":"2024-03-11T19:07:48.736768+0000","last_became_peered":"2024-03-11T19:07:48.736768+0000","last_unstale":"2024-03-11T19:09:56.835200+0000","last_undegraded":"2024-03-11T19:09:56.835200+0000","last_fullsized":"2024-03-11T19:09:56.835200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:18:36.872151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1647,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.634648+0000","last_change":"2024-03-11T19:07:49.739020+0000","last_active":"2024-03-11T19:09:13.634648+0000","last_peered":"2024-03-11T19:09:13.634648+0000","last_clean":"2024-03-11T19:09:13.634648+0000","last_became_active":"2024-03-11T19:07:49.738642+0000","last_became_peered":"2024-03-11T19:07:49.738642+0000","last_unstale":"2024-03-11T19:09:13.634648+0000","last_undegraded":"2024-03-11T19:09:13.634648+0000","last_fullsized":"2024-03-11T19:09:13.634648+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:42:29.040242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037233+0000","last_change":"2024-03-11T19:07:48.745192+0000","last_active":"2024-03-11T19:08:45.037233+0000","last_peered":"2024-03-11T19:08:45.037233+0000","last_clean":"2024-03-11T19:08:45.037233+0000","last_became_active":"2024-03-11T19:07:48.744868+0000","last_became_peered":"2024-03-11T19:07:48.744868+0000","last_unstale":"2024-03-11T19:08:45.037233+0000","last_undegraded":"2024-03-11T19:08:45.037233+0000","last_fullsized":"2024-03-11T19:08:45.037233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:19:15.620186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.640619+0000","last_change":"2024-03-11T19:07:49.737149+0000","last_active":"2024-03-11T19:09:13.640619+0000","last_peered":"2024-03-11T19:09:13.640619+0000","last_clean":"2024-03-11T19:09:13.640619+0000","last_became_active":"2024-03-11T19:07:49.736797+0000","last_became_peered":"2024-03-11T19:07:49.736797+0000","last_unstale":"2024-03-11T19:09:13.640619+0000","last_undegraded":"2024-03-11T19:09:13.640619+0000","last_fullsized":"2024-03-11T19:09:13.640619+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:12:10.282062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.988596+0000","last_change":"2024-03-11T19:07:48.743948+0000","last_active":"2024-03-11T19:08:44.988596+0000","last_peered":"2024-03-11T19:08:44.988596+0000","last_clean":"2024-03-11T19:08:44.988596+0000","last_became_active":"2024-03-11T19:07:48.743451+0000","last_became_peered":"2024-03-11T19:07:48.743451+0000","last_unstale":"2024-03-11T19:08:44.988596+0000","last_undegraded":"2024-03-11T19:08:44.988596+0000","last_fullsized":"2024-03-11T19:08:44.988596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:03:19.823287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.636529+0000","last_change":"2024-03-11T19:07:49.745518+0000","last_active":"2024-03-11T19:09:13.636529+0000","last_peered":"2024-03-11T19:09:13.636529+0000","last_clean":"2024-03-11T19:09:13.636529+0000","last_became_active":"2024-03-11T19:07:49.743719+0000","last_became_peered":"2024-03-11T19:07:49.743719+0000","last_unstale":"2024-03-11T19:09:13.636529+0000","last_undegraded":"2024-03-11T19:09:13.636529+0000","last_fullsized":"2024-03-11T19:09:13.636529+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:25:33.855679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036218+0000","last_change":"2024-03-11T19:07:48.734044+0000","last_active":"2024-03-11T19:08:45.036218+0000","last_peered":"2024-03-11T19:08:45.036218+0000","last_clean":"2024-03-11T19:08:45.036218+0000","last_became_active":"2024-03-11T19:07:48.733723+0000","last_became_peered":"2024-03-11T19:07:48.733723+0000","last_unstale":"2024-03-11T19:08:45.036218+0000","last_undegraded":"2024-03-11T19:08:45.036218+0000","last_fullsized":"2024-03-11T19:08:45.036218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:53:37.070296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.635887+0000","last_change":"2024-03-11T19:07:49.743729+0000","last_active":"2024-03-11T19:09:13.635887+0000","last_peered":"2024-03-11T19:09:13.635887+0000","last_clean":"2024-03-11T19:09:13.635887+0000","last_became_active":"2024-03-11T19:07:49.742992+0000","last_became_peered":"2024-03-11T19:07:49.742992+0000","last_unstale":"2024-03-11T19:09:13.635887+0000","last_undegraded":"2024-03-11T19:09:13.635887+0000","last_fullsized":"2024-03-11T19:09:13.635887+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:46.477177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"89'7","reported_seq":268,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:56.834343+0000","last_change":"2024-03-11T19:07:48.738775+0000","last_active":"2024-03-11T19:09:56.834343+0000","last_peered":"2024-03-11T19:09:56.834343+0000","last_clean":"2024-03-11T19:09:56.834343+0000","last_became_active":"2024-03-11T19:07:48.738437+0000","last_became_peered":"2024-03-11T19:07:48.738437+0000","last_unstale":"2024-03-11T19:09:56.834343+0000","last_undegraded":"2024-03-11T19:09:56.834343+0000","last_fullsized":"2024-03-11T19:09:56.834343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:17:20.020756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":242,"num_read_kb":242,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.654932+0000","last_change":"2024-03-11T19:07:49.760293+0000","last_active":"2024-03-11T19:09:13.654932+0000","last_peered":"2024-03-11T19:09:13.654932+0000","last_clean":"2024-03-11T19:09:13.654932+0000","last_became_active":"2024-03-11T19:07:49.759977+0000","last_became_peered":"2024-03-11T19:07:49.759977+0000","last_unstale":"2024-03-11T19:09:13.654932+0000","last_undegraded":"2024-03-11T19:09:13.654932+0000","last_fullsized":"2024-03-11T19:09:13.654932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:07:44.726756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985956+0000","last_change":"2024-03-11T19:07:48.740672+0000","last_active":"2024-03-11T19:08:44.985956+0000","last_peered":"2024-03-11T19:08:44.985956+0000","last_clean":"2024-03-11T19:08:44.985956+0000","last_became_active":"2024-03-11T19:07:48.739331+0000","last_became_peered":"2024-03-11T19:07:48.739331+0000","last_unstale":"2024-03-11T19:08:44.985956+0000","last_undegraded":"2024-03-11T19:08:44.985956+0000","last_fullsized":"2024-03-11T19:08:44.985956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:19:28.375515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.650493+0000","last_change":"2024-03-11T19:07:49.742780+0000","last_active":"2024-03-11T19:09:13.650493+0000","last_peered":"2024-03-11T19:09:13.650493+0000","last_clean":"2024-03-11T19:09:13.650493+0000","last_became_active":"2024-03-11T19:07:49.742442+0000","last_became_peered":"2024-03-11T19:07:49.742442+0000","last_unstale":"2024-03-11T19:09:13.650493+0000","last_undegraded":"2024-03-11T19:09:13.650493+0000","last_fullsized":"2024-03-11T19:09:13.650493+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:11:13.838225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036582+0000","last_change":"2024-03-11T19:07:48.752617+0000","last_active":"2024-03-11T19:08:45.036582+0000","last_peered":"2024-03-11T19:08:45.036582+0000","last_clean":"2024-03-11T19:08:45.036582+0000","last_became_active":"2024-03-11T19:07:48.752181+0000","last_became_peered":"2024-03-11T19:07:48.752181+0000","last_unstale":"2024-03-11T19:08:45.036582+0000","last_undegraded":"2024-03-11T19:08:45.036582+0000","last_fullsized":"2024-03-11T19:08:45.036582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:05:37.162866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.654679+0000","last_change":"2024-03-11T19:07:49.753672+0000","last_active":"2024-03-11T19:09:13.654679+0000","last_peered":"2024-03-11T19:09:13.654679+0000","last_clean":"2024-03-11T19:09:13.654679+0000","last_became_active":"2024-03-11T19:07:49.753315+0000","last_became_peered":"2024-03-11T19:07:49.753315+0000","last_unstale":"2024-03-11T19:09:13.654679+0000","last_undegraded":"2024-03-11T19:09:13.654679+0000","last_fullsized":"2024-03-11T19:09:13.654679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:30:58.503249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'38","reported_seq":310,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:56.834983+0000","last_change":"2024-03-11T19:07:48.740424+0000","last_active":"2024-03-11T19:09:56.834983+0000","last_peered":"2024-03-11T19:09:56.834983+0000","last_clean":"2024-03-11T19:09:56.834983+0000","last_became_active":"2024-03-11T19:07:48.740077+0000","last_became_peered":"2024-03-11T19:07:48.740077+0000","last_unstale":"2024-03-11T19:09:56.834983+0000","last_undegraded":"2024-03-11T19:09:56.834983+0000","last_fullsized":"2024-03-11T19:09:56.834983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:30:29.193861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66493,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":253,"num_read_kb":192,"num_write":40,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.652816+0000","last_change":"2024-03-11T19:07:49.746440+0000","last_active":"2024-03-11T19:09:13.652816+0000","last_peered":"2024-03-11T19:09:13.652816+0000","last_clean":"2024-03-11T19:09:13.652816+0000","last_became_active":"2024-03-11T19:07:49.746106+0000","last_became_peered":"2024-03-11T19:07:49.746106+0000","last_unstale":"2024-03-11T19:09:13.652816+0000","last_undegraded":"2024-03-11T19:09:13.652816+0000","last_fullsized":"2024-03-11T19:09:13.652816+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:44:58.039917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'9","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.986027+0000","last_change":"2024-03-11T19:07:48.744437+0000","last_active":"2024-03-11T19:08:44.986027+0000","last_peered":"2024-03-11T19:08:44.986027+0000","last_clean":"2024-03-11T19:08:44.986027+0000","last_became_active":"2024-03-11T19:07:48.744142+0000","last_became_peered":"2024-03-11T19:07:48.744142+0000","last_unstale":"2024-03-11T19:08:44.986027+0000","last_undegraded":"2024-03-11T19:08:44.986027+0000","last_fullsized":"2024-03-11T19:08:44.986027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:32:09.076975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.653140+0000","last_change":"2024-03-11T19:07:49.751191+0000","last_active":"2024-03-11T19:09:13.653140+0000","last_peered":"2024-03-11T19:09:13.653140+0000","last_clean":"2024-03-11T19:09:13.653140+0000","last_became_active":"2024-03-11T19:07:49.750849+0000","last_became_peered":"2024-03-11T19:07:49.750849+0000","last_unstale":"2024-03-11T19:09:13.653140+0000","last_undegraded":"2024-03-11T19:09:13.653140+0000","last_fullsized":"2024-03-11T19:09:13.653140+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:19:18.416905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036229+0000","last_change":"2024-03-11T19:07:48.742808+0000","last_active":"2024-03-11T19:08:45.036229+0000","last_peered":"2024-03-11T19:08:45.036229+0000","last_clean":"2024-03-11T19:08:45.036229+0000","last_became_active":"2024-03-11T19:07:48.742462+0000","last_became_peered":"2024-03-11T19:07:48.742462+0000","last_unstale":"2024-03-11T19:08:45.036229+0000","last_undegraded":"2024-03-11T19:08:45.036229+0000","last_fullsized":"2024-03-11T19:08:45.036229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:19.109299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.651622+0000","last_change":"2024-03-11T19:07:49.744673+0000","last_active":"2024-03-11T19:09:13.651622+0000","last_peered":"2024-03-11T19:09:13.651622+0000","last_clean":"2024-03-11T19:09:13.651622+0000","last_became_active":"2024-03-11T19:07:49.744313+0000","last_became_peered":"2024-03-11T19:07:49.744313+0000","last_unstale":"2024-03-11T19:09:13.651622+0000","last_undegraded":"2024-03-11T19:09:13.651622+0000","last_fullsized":"2024-03-11T19:09:13.651622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:21:04.759356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":151,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:57.836159+0000","last_change":"2024-03-11T19:07:48.736024+0000","last_active":"2024-03-11T19:09:57.836159+0000","last_peered":"2024-03-11T19:09:57.836159+0000","last_clean":"2024-03-11T19:09:57.836159+0000","last_became_active":"2024-03-11T19:07:48.735720+0000","last_became_peered":"2024-03-11T19:07:48.735720+0000","last_unstale":"2024-03-11T19:09:57.836159+0000","last_undegraded":"2024-03-11T19:09:57.836159+0000","last_fullsized":"2024-03-11T19:09:57.836159+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:04:57.386267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":127,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.640229+0000","last_change":"2024-03-11T19:07:49.742924+0000","last_active":"2024-03-11T19:09:13.640229+0000","last_peered":"2024-03-11T19:09:13.640229+0000","last_clean":"2024-03-11T19:09:13.640229+0000","last_became_active":"2024-03-11T19:07:49.742542+0000","last_became_peered":"2024-03-11T19:07:49.742542+0000","last_unstale":"2024-03-11T19:09:13.640229+0000","last_undegraded":"2024-03-11T19:09:13.640229+0000","last_fullsized":"2024-03-11T19:09:13.640229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:08:11.944311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'17","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990343+0000","last_change":"2024-03-11T19:07:48.753113+0000","last_active":"2024-03-11T19:08:44.990343+0000","last_peered":"2024-03-11T19:08:44.990343+0000","last_clean":"2024-03-11T19:08:44.990343+0000","last_became_active":"2024-03-11T19:07:48.752517+0000","last_became_peered":"2024-03-11T19:07:48.752517+0000","last_unstale":"2024-03-11T19:08:44.990343+0000","last_undegraded":"2024-03-11T19:08:44.990343+0000","last_fullsized":"2024-03-11T19:08:44.990343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:48:33.441125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.653632+0000","last_change":"2024-03-11T19:08:45.038243+0000","last_active":"2024-03-11T19:09:13.653632+0000","last_peered":"2024-03-11T19:09:13.653632+0000","last_clean":"2024-03-11T19:09:13.653632+0000","last_became_active":"2024-03-11T19:08:45.037969+0000","last_became_peered":"2024-03-11T19:08:45.037969+0000","last_unstale":"2024-03-11T19:09:13.653632+0000","last_undegraded":"2024-03-11T19:09:13.653632+0000","last_fullsized":"2024-03-11T19:09:13.653632+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:29:09.747958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"89'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991151+0000","last_change":"2024-03-11T19:07:48.739571+0000","last_active":"2024-03-11T19:08:44.991151+0000","last_peered":"2024-03-11T19:08:44.991151+0000","last_clean":"2024-03-11T19:08:44.991151+0000","last_became_active":"2024-03-11T19:07:48.739159+0000","last_became_peered":"2024-03-11T19:07:48.739159+0000","last_unstale":"2024-03-11T19:08:44.991151+0000","last_undegraded":"2024-03-11T19:08:44.991151+0000","last_fullsized":"2024-03-11T19:08:44.991151+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:46:43.474591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.639217+0000","last_change":"2024-03-11T19:07:49.744760+0000","last_active":"2024-03-11T19:09:13.639217+0000","last_peered":"2024-03-11T19:09:13.639217+0000","last_clean":"2024-03-11T19:09:13.639217+0000","last_became_active":"2024-03-11T19:07:49.744265+0000","last_became_peered":"2024-03-11T19:07:49.744265+0000","last_unstale":"2024-03-11T19:09:13.639217+0000","last_undegraded":"2024-03-11T19:09:13.639217+0000","last_fullsized":"2024-03-11T19:09:13.639217+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:09:49.823462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"87'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985221+0000","last_change":"2024-03-11T19:07:48.740223+0000","last_active":"2024-03-11T19:08:44.985221+0000","last_peered":"2024-03-11T19:08:44.985221+0000","last_clean":"2024-03-11T19:08:44.985221+0000","last_became_active":"2024-03-11T19:07:48.739569+0000","last_became_peered":"2024-03-11T19:07:48.739569+0000","last_unstale":"2024-03-11T19:08:44.985221+0000","last_undegraded":"2024-03-11T19:08:44.985221+0000","last_fullsized":"2024-03-11T19:08:44.985221+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:04:19.384523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.641047+0000","last_change":"2024-03-11T19:07:49.755620+0000","last_active":"2024-03-11T19:09:13.641047+0000","last_peered":"2024-03-11T19:09:13.641047+0000","last_clean":"2024-03-11T19:09:13.641047+0000","last_became_active":"2024-03-11T19:07:49.750886+0000","last_became_peered":"2024-03-11T19:07:49.750886+0000","last_unstale":"2024-03-11T19:09:13.641047+0000","last_undegraded":"2024-03-11T19:09:13.641047+0000","last_fullsized":"2024-03-11T19:09:13.641047+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:35:54.191061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'6","reported_seq":147,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:54.831635+0000","last_change":"2024-03-11T19:07:48.743340+0000","last_active":"2024-03-11T19:09:54.831635+0000","last_peered":"2024-03-11T19:09:54.831635+0000","last_clean":"2024-03-11T19:09:54.831635+0000","last_became_active":"2024-03-11T19:07:48.742947+0000","last_became_peered":"2024-03-11T19:07:48.742947+0000","last_unstale":"2024-03-11T19:09:54.831635+0000","last_undegraded":"2024-03-11T19:09:54.831635+0000","last_fullsized":"2024-03-11T19:09:54.831635+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:40:25.233150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.643001+0000","last_change":"2024-03-11T19:07:49.753668+0000","last_active":"2024-03-11T19:09:13.643001+0000","last_peered":"2024-03-11T19:09:13.643001+0000","last_clean":"2024-03-11T19:09:13.643001+0000","last_became_active":"2024-03-11T19:07:49.750574+0000","last_became_peered":"2024-03-11T19:07:49.750574+0000","last_unstale":"2024-03-11T19:09:13.643001+0000","last_undegraded":"2024-03-11T19:09:13.643001+0000","last_fullsized":"2024-03-11T19:09:13.643001+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:12:32.711847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.640771+0000","last_change":"2024-03-11T19:07:49.739538+0000","last_active":"2024-03-11T19:09:13.640771+0000","last_peered":"2024-03-11T19:09:13.640771+0000","last_clean":"2024-03-11T19:09:13.640771+0000","last_became_active":"2024-03-11T19:07:49.738617+0000","last_became_peered":"2024-03-11T19:07:49.738617+0000","last_unstale":"2024-03-11T19:09:13.640771+0000","last_undegraded":"2024-03-11T19:09:13.640771+0000","last_fullsized":"2024-03-11T19:09:13.640771+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:43:08.697125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'6","reported_seq":150,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:57.835585+0000","last_change":"2024-03-11T19:07:48.749527+0000","last_active":"2024-03-11T19:09:57.835585+0000","last_peered":"2024-03-11T19:09:57.835585+0000","last_clean":"2024-03-11T19:09:57.835585+0000","last_became_active":"2024-03-11T19:07:48.749026+0000","last_became_peered":"2024-03-11T19:07:48.749026+0000","last_unstale":"2024-03-11T19:09:57.835585+0000","last_undegraded":"2024-03-11T19:09:57.835585+0000","last_fullsized":"2024-03-11T19:09:57.835585+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:20:19.058429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":124,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.646783+0000","last_change":"2024-03-11T19:07:49.757696+0000","last_active":"2024-03-11T19:09:13.646783+0000","last_peered":"2024-03-11T19:09:13.646783+0000","last_clean":"2024-03-11T19:09:13.646783+0000","last_became_active":"2024-03-11T19:07:49.757380+0000","last_became_peered":"2024-03-11T19:07:49.757380+0000","last_unstale":"2024-03-11T19:09:13.646783+0000","last_undegraded":"2024-03-11T19:09:13.646783+0000","last_fullsized":"2024-03-11T19:09:13.646783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:03:22.954619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036816+0000","last_change":"2024-03-11T19:07:48.737504+0000","last_active":"2024-03-11T19:08:45.036816+0000","last_peered":"2024-03-11T19:08:45.036816+0000","last_clean":"2024-03-11T19:08:45.036816+0000","last_became_active":"2024-03-11T19:07:48.737147+0000","last_became_peered":"2024-03-11T19:07:48.737147+0000","last_unstale":"2024-03-11T19:08:45.036816+0000","last_undegraded":"2024-03-11T19:08:45.036816+0000","last_fullsized":"2024-03-11T19:08:45.036816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:04.819617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":344,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985900+0000","last_change":"2024-03-11T19:06:35.035452+0000","last_active":"2024-03-11T19:08:44.985900+0000","last_peered":"2024-03-11T19:08:44.985900+0000","last_clean":"2024-03-11T19:08:44.985900+0000","last_became_active":"2024-03-11T19:06:35.035076+0000","last_became_peered":"2024-03-11T19:06:35.035076+0000","last_unstale":"2024-03-11T19:08:44.985900+0000","last_undegraded":"2024-03-11T19:08:44.985900+0000","last_fullsized":"2024-03-11T19:08:44.985900+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:03:10.143002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:03:10.143002+0000","last_clean_scrub_stamp":"2024-03-11T19:03:10.143002+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:39:37.348091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.642831+0000","last_change":"2024-03-11T19:07:49.743000+0000","last_active":"2024-03-11T19:09:13.642831+0000","last_peered":"2024-03-11T19:09:13.642831+0000","last_clean":"2024-03-11T19:09:13.642831+0000","last_became_active":"2024-03-11T19:07:49.742691+0000","last_became_peered":"2024-03-11T19:07:49.742691+0000","last_unstale":"2024-03-11T19:09:13.642831+0000","last_undegraded":"2024-03-11T19:09:13.642831+0000","last_fullsized":"2024-03-11T19:09:13.642831+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:40:21.519929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985719+0000","last_change":"2024-03-11T19:07:48.739038+0000","last_active":"2024-03-11T19:08:44.985719+0000","last_peered":"2024-03-11T19:08:44.985719+0000","last_clean":"2024-03-11T19:08:44.985719+0000","last_became_active":"2024-03-11T19:07:48.738714+0000","last_became_peered":"2024-03-11T19:07:48.738714+0000","last_unstale":"2024-03-11T19:08:44.985719+0000","last_undegraded":"2024-03-11T19:08:44.985719+0000","last_fullsized":"2024-03-11T19:08:44.985719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:17:43.086679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.998660+0000","last_change":"2024-03-11T19:08:44.998660+0000","last_active":"2024-03-11T19:08:44.998660+0000","last_peered":"2024-03-11T19:08:44.998660+0000","last_clean":"2024-03-11T19:08:44.998660+0000","last_became_active":"2024-03-11T19:08:44.991682+0000","last_became_peered":"2024-03-11T19:08:44.991682+0000","last_unstale":"2024-03-11T19:08:44.998660+0000","last_undegraded":"2024-03-11T19:08:44.998660+0000","last_fullsized":"2024-03-11T19:08:44.998660+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:19:44.266076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.645393+0000","last_change":"2024-03-11T19:07:49.742434+0000","last_active":"2024-03-11T19:09:13.645393+0000","last_peered":"2024-03-11T19:09:13.645393+0000","last_clean":"2024-03-11T19:09:13.645393+0000","last_became_active":"2024-03-11T19:07:49.742048+0000","last_became_peered":"2024-03-11T19:07:49.742048+0000","last_unstale":"2024-03-11T19:09:13.645393+0000","last_undegraded":"2024-03-11T19:09:13.645393+0000","last_fullsized":"2024-03-11T19:09:13.645393+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:39:22.685055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:53.757286+0000","last_change":"2024-03-11T19:07:48.744728+0000","last_active":"2024-03-11T19:09:53.757286+0000","last_peered":"2024-03-11T19:09:53.757286+0000","last_clean":"2024-03-11T19:09:53.757286+0000","last_became_active":"2024-03-11T19:07:48.744378+0000","last_became_peered":"2024-03-11T19:07:48.744378+0000","last_unstale":"2024-03-11T19:09:53.757286+0000","last_undegraded":"2024-03-11T19:09:53.757286+0000","last_fullsized":"2024-03-11T19:09:53.757286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:42.823040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.654164+0000","last_change":"2024-03-11T19:07:49.744261+0000","last_active":"2024-03-11T19:09:13.654164+0000","last_peered":"2024-03-11T19:09:13.654164+0000","last_clean":"2024-03-11T19:09:13.654164+0000","last_became_active":"2024-03-11T19:07:49.743946+0000","last_became_peered":"2024-03-11T19:07:49.743946+0000","last_unstale":"2024-03-11T19:09:13.654164+0000","last_undegraded":"2024-03-11T19:09:13.654164+0000","last_fullsized":"2024-03-11T19:09:13.654164+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:54:07.928899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'7","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037249+0000","last_change":"2024-03-11T19:07:48.740059+0000","last_active":"2024-03-11T19:08:45.037249+0000","last_peered":"2024-03-11T19:08:45.037249+0000","last_clean":"2024-03-11T19:08:45.037249+0000","last_became_active":"2024-03-11T19:07:48.739626+0000","last_became_peered":"2024-03-11T19:07:48.739626+0000","last_unstale":"2024-03-11T19:08:45.037249+0000","last_undegraded":"2024-03-11T19:08:45.037249+0000","last_fullsized":"2024-03-11T19:08:45.037249+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:45:12.219203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.647418+0000","last_change":"2024-03-11T19:07:49.746041+0000","last_active":"2024-03-11T19:09:13.647418+0000","last_peered":"2024-03-11T19:09:13.647418+0000","last_clean":"2024-03-11T19:09:13.647418+0000","last_became_active":"2024-03-11T19:07:49.745602+0000","last_became_peered":"2024-03-11T19:07:49.745602+0000","last_unstale":"2024-03-11T19:09:13.647418+0000","last_undegraded":"2024-03-11T19:09:13.647418+0000","last_fullsized":"2024-03-11T19:09:13.647418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:23:44.596929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.035863+0000","last_change":"2024-03-11T19:07:48.748677+0000","last_active":"2024-03-11T19:08:45.035863+0000","last_peered":"2024-03-11T19:08:45.035863+0000","last_clean":"2024-03-11T19:08:45.035863+0000","last_became_active":"2024-03-11T19:07:48.748470+0000","last_became_peered":"2024-03-11T19:07:48.748470+0000","last_unstale":"2024-03-11T19:08:45.035863+0000","last_undegraded":"2024-03-11T19:08:45.035863+0000","last_fullsized":"2024-03-11T19:08:45.035863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:24:55.240722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.620826+0000","last_change":"2024-03-11T19:07:49.746522+0000","last_active":"2024-03-11T19:09:13.620826+0000","last_peered":"2024-03-11T19:09:13.620826+0000","last_clean":"2024-03-11T19:09:13.620826+0000","last_became_active":"2024-03-11T19:07:49.746179+0000","last_became_peered":"2024-03-11T19:07:49.746179+0000","last_unstale":"2024-03-11T19:09:13.620826+0000","last_undegraded":"2024-03-11T19:09:13.620826+0000","last_fullsized":"2024-03-11T19:09:13.620826+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:06:08.144850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.989738+0000","last_change":"2024-03-11T19:07:48.740926+0000","last_active":"2024-03-11T19:08:44.989738+0000","last_peered":"2024-03-11T19:08:44.989738+0000","last_clean":"2024-03-11T19:08:44.989738+0000","last_became_active":"2024-03-11T19:07:48.740625+0000","last_became_peered":"2024-03-11T19:07:48.740625+0000","last_unstale":"2024-03-11T19:08:44.989738+0000","last_undegraded":"2024-03-11T19:08:44.989738+0000","last_fullsized":"2024-03-11T19:08:44.989738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:50:13.079608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.647609+0000","last_change":"2024-03-11T19:07:49.742428+0000","last_active":"2024-03-11T19:09:13.647609+0000","last_peered":"2024-03-11T19:09:13.647609+0000","last_clean":"2024-03-11T19:09:13.647609+0000","last_became_active":"2024-03-11T19:07:49.742098+0000","last_became_peered":"2024-03-11T19:07:49.742098+0000","last_unstale":"2024-03-11T19:09:13.647609+0000","last_undegraded":"2024-03-11T19:09:13.647609+0000","last_fullsized":"2024-03-11T19:09:13.647609+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:19:24.615755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'5","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:53.777258+0000","last_change":"2024-03-11T19:08:45.002189+0000","last_active":"2024-03-11T19:09:53.777258+0000","last_peered":"2024-03-11T19:09:53.777258+0000","last_clean":"2024-03-11T19:09:53.777258+0000","last_became_active":"2024-03-11T19:08:44.992818+0000","last_became_peered":"2024-03-11T19:08:44.992818+0000","last_unstale":"2024-03-11T19:09:53.777258+0000","last_undegraded":"2024-03-11T19:09:53.777258+0000","last_fullsized":"2024-03-11T19:09:53.777258+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:34:54.461734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.643596+0000","last_change":"2024-03-11T19:07:49.754872+0000","last_active":"2024-03-11T19:09:13.643596+0000","last_peered":"2024-03-11T19:09:13.643596+0000","last_clean":"2024-03-11T19:09:13.643596+0000","last_became_active":"2024-03-11T19:07:49.749663+0000","last_became_peered":"2024-03-11T19:07:49.749663+0000","last_unstale":"2024-03-11T19:09:13.643596+0000","last_undegraded":"2024-03-11T19:09:13.643596+0000","last_fullsized":"2024-03-11T19:09:13.643596+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:22:23.664814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990526+0000","last_change":"2024-03-11T19:07:48.742227+0000","last_active":"2024-03-11T19:08:44.990526+0000","last_peered":"2024-03-11T19:08:44.990526+0000","last_clean":"2024-03-11T19:08:44.990526+0000","last_became_active":"2024-03-11T19:07:48.741912+0000","last_became_peered":"2024-03-11T19:07:48.741912+0000","last_unstale":"2024-03-11T19:08:44.990526+0000","last_undegraded":"2024-03-11T19:08:44.990526+0000","last_fullsized":"2024-03-11T19:08:44.990526+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:10:06.120433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.657986+0000","last_change":"2024-03-11T19:07:49.748947+0000","last_active":"2024-03-11T19:09:13.657986+0000","last_peered":"2024-03-11T19:09:13.657986+0000","last_clean":"2024-03-11T19:09:13.657986+0000","last_became_active":"2024-03-11T19:07:49.748621+0000","last_became_peered":"2024-03-11T19:07:49.748621+0000","last_unstale":"2024-03-11T19:09:13.657986+0000","last_undegraded":"2024-03-11T19:09:13.657986+0000","last_fullsized":"2024-03-11T19:09:13.657986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:19:53.824590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'6","reported_seq":150,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:56.834864+0000","last_change":"2024-03-11T19:07:48.748799+0000","last_active":"2024-03-11T19:09:56.834864+0000","last_peered":"2024-03-11T19:09:56.834864+0000","last_clean":"2024-03-11T19:09:56.834864+0000","last_became_active":"2024-03-11T19:07:48.748434+0000","last_became_peered":"2024-03-11T19:07:48.748434+0000","last_unstale":"2024-03-11T19:09:56.834864+0000","last_undegraded":"2024-03-11T19:09:56.834864+0000","last_fullsized":"2024-03-11T19:09:56.834864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:10:09.836926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":125,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.657255+0000","last_change":"2024-03-11T19:07:49.750122+0000","last_active":"2024-03-11T19:09:13.657255+0000","last_peered":"2024-03-11T19:09:13.657255+0000","last_clean":"2024-03-11T19:09:13.657255+0000","last_became_active":"2024-03-11T19:07:49.749801+0000","last_became_peered":"2024-03-11T19:07:49.749801+0000","last_unstale":"2024-03-11T19:09:13.657255+0000","last_undegraded":"2024-03-11T19:09:13.657255+0000","last_fullsized":"2024-03-11T19:09:13.657255+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:58:31.155275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036439+0000","last_change":"2024-03-11T19:07:48.741959+0000","last_active":"2024-03-11T19:08:45.036439+0000","last_peered":"2024-03-11T19:08:45.036439+0000","last_clean":"2024-03-11T19:08:45.036439+0000","last_became_active":"2024-03-11T19:07:48.741630+0000","last_became_peered":"2024-03-11T19:07:48.741630+0000","last_unstale":"2024-03-11T19:08:45.036439+0000","last_undegraded":"2024-03-11T19:08:45.036439+0000","last_fullsized":"2024-03-11T19:08:45.036439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:27:08.606055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.652383+0000","last_change":"2024-03-11T19:07:49.729838+0000","last_active":"2024-03-11T19:09:13.652383+0000","last_peered":"2024-03-11T19:09:13.652383+0000","last_clean":"2024-03-11T19:09:13.652383+0000","last_became_active":"2024-03-11T19:07:49.728912+0000","last_became_peered":"2024-03-11T19:07:49.728912+0000","last_unstale":"2024-03-11T19:09:13.652383+0000","last_undegraded":"2024-03-11T19:09:13.652383+0000","last_fullsized":"2024-03-11T19:09:13.652383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:02:06.551734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.989032+0000","last_change":"2024-03-11T19:07:48.744709+0000","last_active":"2024-03-11T19:08:44.989032+0000","last_peered":"2024-03-11T19:08:44.989032+0000","last_clean":"2024-03-11T19:08:44.989032+0000","last_became_active":"2024-03-11T19:07:48.744356+0000","last_became_peered":"2024-03-11T19:07:48.744356+0000","last_unstale":"2024-03-11T19:08:44.989032+0000","last_undegraded":"2024-03-11T19:08:44.989032+0000","last_fullsized":"2024-03-11T19:08:44.989032+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:48:47.098443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.652239+0000","last_change":"2024-03-11T19:07:49.743288+0000","last_active":"2024-03-11T19:09:13.652239+0000","last_peered":"2024-03-11T19:09:13.652239+0000","last_clean":"2024-03-11T19:09:13.652239+0000","last_became_active":"2024-03-11T19:07:49.742942+0000","last_became_peered":"2024-03-11T19:07:49.742942+0000","last_unstale":"2024-03-11T19:09:13.652239+0000","last_undegraded":"2024-03-11T19:09:13.652239+0000","last_fullsized":"2024-03-11T19:09:13.652239+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:21:53.896192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.645073+0000","last_change":"2024-03-11T19:07:49.741610+0000","last_active":"2024-03-11T19:09:13.645073+0000","last_peered":"2024-03-11T19:09:13.645073+0000","last_clean":"2024-03-11T19:09:13.645073+0000","last_became_active":"2024-03-11T19:07:49.741234+0000","last_became_peered":"2024-03-11T19:07:49.741234+0000","last_unstale":"2024-03-11T19:09:13.645073+0000","last_undegraded":"2024-03-11T19:09:13.645073+0000","last_fullsized":"2024-03-11T19:09:13.645073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:53:03.320093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.638525+0000","last_change":"2024-03-11T19:08:45.037715+0000","last_active":"2024-03-11T19:09:13.638525+0000","last_peered":"2024-03-11T19:09:13.638525+0000","last_clean":"2024-03-11T19:09:13.638525+0000","last_became_active":"2024-03-11T19:08:45.037299+0000","last_became_peered":"2024-03-11T19:08:45.037299+0000","last_unstale":"2024-03-11T19:09:13.638525+0000","last_undegraded":"2024-03-11T19:09:13.638525+0000","last_fullsized":"2024-03-11T19:09:13.638525+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:01:49.025379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.645173+0000","last_change":"2024-03-11T19:07:49.745928+0000","last_active":"2024-03-11T19:09:13.645173+0000","last_peered":"2024-03-11T19:09:13.645173+0000","last_clean":"2024-03-11T19:09:13.645173+0000","last_became_active":"2024-03-11T19:07:49.745025+0000","last_became_peered":"2024-03-11T19:07:49.745025+0000","last_unstale":"2024-03-11T19:09:13.645173+0000","last_undegraded":"2024-03-11T19:09:13.645173+0000","last_fullsized":"2024-03-11T19:09:13.645173+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:40:30.141906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.648364+0000","last_change":"2024-03-11T19:07:49.756274+0000","last_active":"2024-03-11T19:09:13.648364+0000","last_peered":"2024-03-11T19:09:13.648364+0000","last_clean":"2024-03-11T19:09:13.648364+0000","last_became_active":"2024-03-11T19:07:49.755954+0000","last_became_peered":"2024-03-11T19:07:49.755954+0000","last_unstale":"2024-03-11T19:09:13.648364+0000","last_undegraded":"2024-03-11T19:09:13.648364+0000","last_fullsized":"2024-03-11T19:09:13.648364+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:15:36.825240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.639818+0000","last_change":"2024-03-11T19:08:44.992154+0000","last_active":"2024-03-11T19:09:13.639818+0000","last_peered":"2024-03-11T19:09:13.639818+0000","last_clean":"2024-03-11T19:09:13.639818+0000","last_became_active":"2024-03-11T19:08:44.991772+0000","last_became_peered":"2024-03-11T19:08:44.991772+0000","last_unstale":"2024-03-11T19:09:13.639818+0000","last_undegraded":"2024-03-11T19:09:13.639818+0000","last_fullsized":"2024-03-11T19:09:13.639818+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:10:33.882095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.641105+0000","last_change":"2024-03-11T19:07:49.743963+0000","last_active":"2024-03-11T19:09:13.641105+0000","last_peered":"2024-03-11T19:09:13.641105+0000","last_clean":"2024-03-11T19:09:13.641105+0000","last_became_active":"2024-03-11T19:07:49.743568+0000","last_became_peered":"2024-03-11T19:07:49.743568+0000","last_unstale":"2024-03-11T19:09:13.641105+0000","last_undegraded":"2024-03-11T19:09:13.641105+0000","last_fullsized":"2024-03-11T19:09:13.641105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:45:28.970286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.654180+0000","last_change":"2024-03-11T19:07:49.750677+0000","last_active":"2024-03-11T19:09:13.654180+0000","last_peered":"2024-03-11T19:09:13.654180+0000","last_clean":"2024-03-11T19:09:13.654180+0000","last_became_active":"2024-03-11T19:07:49.750373+0000","last_became_peered":"2024-03-11T19:07:49.750373+0000","last_unstale":"2024-03-11T19:09:13.654180+0000","last_undegraded":"2024-03-11T19:09:13.654180+0000","last_fullsized":"2024-03-11T19:09:13.654180+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:07:28.767502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.645768+0000","last_change":"2024-03-11T19:07:49.745278+0000","last_active":"2024-03-11T19:09:13.645768+0000","last_peered":"2024-03-11T19:09:13.645768+0000","last_clean":"2024-03-11T19:09:13.645768+0000","last_became_active":"2024-03-11T19:07:49.744998+0000","last_became_peered":"2024-03-11T19:07:49.744998+0000","last_unstale":"2024-03-11T19:09:13.645768+0000","last_undegraded":"2024-03-11T19:09:13.645768+0000","last_fullsized":"2024-03-11T19:09:13.645768+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:23.778051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.647869+0000","last_change":"2024-03-11T19:07:49.734318+0000","last_active":"2024-03-11T19:09:13.647869+0000","last_peered":"2024-03-11T19:09:13.647869+0000","last_clean":"2024-03-11T19:09:13.647869+0000","last_became_active":"2024-03-11T19:07:49.733998+0000","last_became_peered":"2024-03-11T19:07:49.733998+0000","last_unstale":"2024-03-11T19:09:13.647869+0000","last_undegraded":"2024-03-11T19:09:13.647869+0000","last_fullsized":"2024-03-11T19:09:13.647869+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:48:50.308499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.646683+0000","last_change":"2024-03-11T19:07:49.741009+0000","last_active":"2024-03-11T19:09:13.646683+0000","last_peered":"2024-03-11T19:09:13.646683+0000","last_clean":"2024-03-11T19:09:13.646683+0000","last_became_active":"2024-03-11T19:07:49.740094+0000","last_became_peered":"2024-03-11T19:07:49.740094+0000","last_unstale":"2024-03-11T19:09:13.646683+0000","last_undegraded":"2024-03-11T19:09:13.646683+0000","last_fullsized":"2024-03-11T19:09:13.646683+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:06:31.945387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.657556+0000","last_change":"2024-03-11T19:07:49.759300+0000","last_active":"2024-03-11T19:09:13.657556+0000","last_peered":"2024-03-11T19:09:13.657556+0000","last_clean":"2024-03-11T19:09:13.657556+0000","last_became_active":"2024-03-11T19:07:49.758011+0000","last_became_peered":"2024-03-11T19:07:49.758011+0000","last_unstale":"2024-03-11T19:09:13.657556+0000","last_undegraded":"2024-03-11T19:09:13.657556+0000","last_fullsized":"2024-03-11T19:09:13.657556+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:03:40.651816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.652991+0000","last_change":"2024-03-11T19:07:49.752905+0000","last_active":"2024-03-11T19:09:13.652991+0000","last_peered":"2024-03-11T19:09:13.652991+0000","last_clean":"2024-03-11T19:09:13.652991+0000","last_became_active":"2024-03-11T19:07:49.752383+0000","last_became_peered":"2024-03-11T19:07:49.752383+0000","last_unstale":"2024-03-11T19:09:13.652991+0000","last_undegraded":"2024-03-11T19:09:13.652991+0000","last_fullsized":"2024-03-11T19:09:13.652991+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:45:33.475526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.637754+0000","last_change":"2024-03-11T19:07:49.744957+0000","last_active":"2024-03-11T19:09:13.637754+0000","last_peered":"2024-03-11T19:09:13.637754+0000","last_clean":"2024-03-11T19:09:13.637754+0000","last_became_active":"2024-03-11T19:07:49.744627+0000","last_became_peered":"2024-03-11T19:07:49.744627+0000","last_unstale":"2024-03-11T19:09:13.637754+0000","last_undegraded":"2024-03-11T19:09:13.637754+0000","last_fullsized":"2024-03-11T19:09:13.637754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:27:47.817004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245707,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":75419,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1258,"num_read_kb":711,"num_write":266,"num_write_kb":159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":569344,"data_stored":221943,"data_compressed":5565,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318312,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":464775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579948,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":1122185,"data_compressed":37658,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743474,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":464100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005111,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":464366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266748,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":1058616,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430274,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":464701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":527011,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":465649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1064960,"data_stored":532961,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149776,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":463642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888140,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":464171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691911,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":1057308,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":64033,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":69983,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":68722,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T19:10:00.167 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-11T19:10:00.167 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-11T19:10:00.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.301+0000 7fcfd8d8c640 1 -- 172.21.15.29:0/654915725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfd4106fa0 msgr2=0x7fcfd4109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:00.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.301+0000 7fcfd8d8c640 1 --2- 172.21.15.29:0/654915725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfd4106fa0 0x7fcfd4109390 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc009ca0 tx=0x7fcfcc02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.301+0000 7fcfd8d8c640 1 -- 172.21.15.29:0/654915725 shutdown_connections 2024-03-11T19:10:00.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.301+0000 7fcfd8d8c640 1 --2- 172.21.15.29:0/654915725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfd4106fa0 0x7fcfd4109390 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.301+0000 7fcfd8d8c640 1 --2- 172.21.15.29:0/654915725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfd4104670 0x7fcfd4106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.301+0000 7fcfd8d8c640 1 --2- 172.21.15.29:0/654915725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfd4101d40 0x7fcfd4104130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.301+0000 7fcfd8d8c640 1 -- 172.21.15.29:0/654915725 >> 172.21.15.29:0/654915725 conn(0x7fcfd40fb6b0 msgr2=0x7fcfd40fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:00.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.301+0000 7fcfd8d8c640 1 -- 172.21.15.29:0/654915725 shutdown_connections 2024-03-11T19:10:00.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.301+0000 7fcfd8d8c640 1 -- 172.21.15.29:0/654915725 wait complete. 2024-03-11T19:10:00.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.302+0000 7fcfd8d8c640 1 Processor -- start 2024-03-11T19:10:00.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.302+0000 7fcfd8d8c640 1 -- start start 2024-03-11T19:10:00.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.302+0000 7fcfd8d8c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfd4101d40 0x7fcfd41970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:00.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.302+0000 7fcfd8d8c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfd4104670 0x7fcfd4197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:00.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.302+0000 7fcfd8d8c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfd4106fa0 0x7fcfd419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:00.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.302+0000 7fcfd8d8c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd419ebe0 con 0x7fcfd4106fa0 2024-03-11T19:10:00.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.302+0000 7fcfd8d8c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd419ed50 con 0x7fcfd4101d40 2024-03-11T19:10:00.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.302+0000 7fcfd8d8c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd419ef10 con 0x7fcfd4104670 2024-03-11T19:10:00.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.302+0000 7fcfd2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfd4106fa0 0x7fcfd419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:00.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.302+0000 7fcfd2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfd4106fa0 0x7fcfd419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54518/0 (socket says 172.21.15.29:54518) 2024-03-11T19:10:00.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.302+0000 7fcfd2d9d640 1 -- 172.21.15.29:0/3958167905 learned_addr learned my addr 172.21.15.29:0/3958167905 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:00.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.302+0000 7fcfd259c640 1 --2- 172.21.15.29:0/3958167905 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfd4101d40 0x7fcfd41970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:00.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.302+0000 7fcfc9d9b640 1 --2- 172.21.15.29:0/3958167905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfd4104670 0x7fcfd4197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:00.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.302+0000 7fcfd2d9d640 1 -- 172.21.15.29:0/3958167905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfd4104670 msgr2=0x7fcfd4197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:00.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.302+0000 7fcfd2d9d640 1 --2- 172.21.15.29:0/3958167905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfd4104670 0x7fcfd4197630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.302+0000 7fcfd2d9d640 1 -- 172.21.15.29:0/3958167905 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfd4101d40 msgr2=0x7fcfd41970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:00.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.302+0000 7fcfd2d9d640 1 --2- 172.21.15.29:0/3958167905 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfd4101d40 0x7fcfd41970f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.302+0000 7fcfd2d9d640 1 -- 172.21.15.29:0/3958167905 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfcc009900 con 0x7fcfd4106fa0 2024-03-11T19:10:00.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.303+0000 7fcfd2d9d640 1 --2- 172.21.15.29:0/3958167905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfd4106fa0 0x7fcfd419e6a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc037040 tx=0x7fcfcc032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:00.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.303+0000 7fcfcbfff640 1 -- 172.21.15.29:0/3958167905 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc034e70 con 0x7fcfd4106fa0 2024-03-11T19:10:00.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.303+0000 7fcfd8d8c640 1 -- 172.21.15.29:0/3958167905 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd419f190 con 0x7fcfd4106fa0 2024-03-11T19:10:00.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.303+0000 7fcfcbfff640 1 -- 172.21.15.29:0/3958167905 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcfcc003e20 con 0x7fcfd4106fa0 2024-03-11T19:10:00.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.303+0000 7fcfcbfff640 1 -- 172.21.15.29:0/3958167905 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc03d3f0 con 0x7fcfd4106fa0 2024-03-11T19:10:00.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.303+0000 7fcfd8d8c640 1 -- 172.21.15.29:0/3958167905 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd419f640 con 0x7fcfd4106fa0 2024-03-11T19:10:00.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.304+0000 7fcfd8d8c640 1 -- 172.21.15.29:0/3958167905 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd40fd1a0 con 0x7fcfd4106fa0 2024-03-11T19:10:00.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.305+0000 7fcfcbfff640 1 -- 172.21.15.29:0/3958167905 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fcfcc03b040 con 0x7fcfd4106fa0 2024-03-11T19:10:00.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.305+0000 7fcfcbfff640 1 --2- 172.21.15.29:0/3958167905 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcfc00a1ac0 0x7fcfc00a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:00.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.306+0000 7fcfd259c640 1 --2- 172.21.15.29:0/3958167905 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcfc00a1ac0 0x7fcfc00a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:00.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.306+0000 7fcfcbfff640 1 -- 172.21.15.29:0/3958167905 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fcfcc0efee0 con 0x7fcfd4106fa0 2024-03-11T19:10:00.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.306+0000 7fcfd259c640 1 --2- 172.21.15.29:0/3958167905 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcfc00a1ac0 0x7fcfc00a3f80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcfc40058c0 tx=0x7fcfc400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:00.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.309+0000 7fcfcbfff640 1 -- 172.21.15.29:0/3958167905 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcfcc0b91a0 con 0x7fcfd4106fa0 2024-03-11T19:10:00.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.486+0000 7fcfd8d8c640 1 -- 172.21.15.29:0/3958167905 --> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcfd402ceb0 con 0x7fcfc00a1ac0 2024-03-11T19:10:00.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.493+0000 7fcfcbfff640 1 -- 172.21.15.29:0/3958167905 <== mgr.14150 v2:172.21.15.29:6800/1587871898 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259094 (secure 0 0 0) 0x7fcfd402ceb0 con 0x7fcfc00a1ac0 2024-03-11T19:10:00.499 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:10:00.500 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-11T19:10:00.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.496+0000 7fcfd8d8c640 1 -- 172.21.15.29:0/3958167905 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcfc00a1ac0 msgr2=0x7fcfc00a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:00.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.496+0000 7fcfd8d8c640 1 --2- 172.21.15.29:0/3958167905 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcfc00a1ac0 0x7fcfc00a3f80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcfc40058c0 tx=0x7fcfc400a2d0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.496+0000 7fcfd8d8c640 1 -- 172.21.15.29:0/3958167905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfd4106fa0 msgr2=0x7fcfd419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:00.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.496+0000 7fcfd8d8c640 1 --2- 172.21.15.29:0/3958167905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfd4106fa0 0x7fcfd419e6a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc037040 tx=0x7fcfcc032980 comp rx=0 tx=0).stop 2024-03-11T19:10:00.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.496+0000 7fcfd8d8c640 1 -- 172.21.15.29:0/3958167905 shutdown_connections 2024-03-11T19:10:00.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.496+0000 7fcfd8d8c640 1 --2- 172.21.15.29:0/3958167905 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcfc00a1ac0 0x7fcfc00a3f80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.496+0000 7fcfd8d8c640 1 --2- 172.21.15.29:0/3958167905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfd4106fa0 0x7fcfd419e6a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.496+0000 7fcfd8d8c640 1 --2- 172.21.15.29:0/3958167905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfd4104670 0x7fcfd4197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.496+0000 7fcfd8d8c640 1 --2- 172.21.15.29:0/3958167905 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfd4101d40 0x7fcfd41970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.496+0000 7fcfd8d8c640 1 -- 172.21.15.29:0/3958167905 >> 172.21.15.29:0/3958167905 conn(0x7fcfd40fb6b0 msgr2=0x7fcfd41025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:00.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.496+0000 7fcfd8d8c640 1 -- 172.21.15.29:0/3958167905 shutdown_connections 2024-03-11T19:10:00.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.496+0000 7fcfd8d8c640 1 -- 172.21.15.29:0/3958167905 wait complete. 2024-03-11T19:10:00.509 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:00 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1443096501' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:10:00.509 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:00 smithi029 ceph-mon[32439]: overall HEALTH_OK 2024-03-11T19:10:00.510 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-03-11T19:09:59.636214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666021,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1384,"num_read_kb":820,"num_write":449,"num_write_kb":3099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":394388,"kb_used_data":11956,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124547436,"statfs":{"total":1151940427776,"available":1151536574464,"internally_reserved":0,"allocated":12242944,"data_stored":7549485,"data_compressed":112974,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":391492922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7170,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":127,"num_read_kb":66,"num_write":22,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002894"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.650725+0000","last_change":"2024-03-11T19:07:49.751738+0000","last_active":"2024-03-11T19:09:13.650725+0000","last_peered":"2024-03-11T19:09:13.650725+0000","last_clean":"2024-03-11T19:09:13.650725+0000","last_became_active":"2024-03-11T19:07:49.751173+0000","last_became_peered":"2024-03-11T19:07:49.751173+0000","last_unstale":"2024-03-11T19:09:13.650725+0000","last_undegraded":"2024-03-11T19:09:13.650725+0000","last_fullsized":"2024-03-11T19:09:13.650725+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:13:40.157763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.648231+0000","last_change":"2024-03-11T19:07:49.749069+0000","last_active":"2024-03-11T19:09:13.648231+0000","last_peered":"2024-03-11T19:09:13.648231+0000","last_clean":"2024-03-11T19:09:13.648231+0000","last_became_active":"2024-03-11T19:07:49.747672+0000","last_became_peered":"2024-03-11T19:07:49.747672+0000","last_unstale":"2024-03-11T19:09:13.648231+0000","last_undegraded":"2024-03-11T19:09:13.648231+0000","last_fullsized":"2024-03-11T19:09:13.648231+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:57:26.379836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.653423+0000","last_change":"2024-03-11T19:07:49.750298+0000","last_active":"2024-03-11T19:09:13.653423+0000","last_peered":"2024-03-11T19:09:13.653423+0000","last_clean":"2024-03-11T19:09:13.653423+0000","last_became_active":"2024-03-11T19:07:49.749985+0000","last_became_peered":"2024-03-11T19:07:49.749985+0000","last_unstale":"2024-03-11T19:09:13.653423+0000","last_undegraded":"2024-03-11T19:09:13.653423+0000","last_fullsized":"2024-03-11T19:09:13.653423+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:56.579872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.642014+0000","last_change":"2024-03-11T19:08:44.993047+0000","last_active":"2024-03-11T19:09:13.642014+0000","last_peered":"2024-03-11T19:09:13.642014+0000","last_clean":"2024-03-11T19:09:13.642014+0000","last_became_active":"2024-03-11T19:08:44.992666+0000","last_became_peered":"2024-03-11T19:08:44.992666+0000","last_unstale":"2024-03-11T19:09:13.642014+0000","last_undegraded":"2024-03-11T19:09:13.642014+0000","last_fullsized":"2024-03-11T19:09:13.642014+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:32:18.748906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.634551+0000","last_change":"2024-03-11T19:07:49.747618+0000","last_active":"2024-03-11T19:09:13.634551+0000","last_peered":"2024-03-11T19:09:13.634551+0000","last_clean":"2024-03-11T19:09:13.634551+0000","last_became_active":"2024-03-11T19:07:49.747310+0000","last_became_peered":"2024-03-11T19:07:49.747310+0000","last_unstale":"2024-03-11T19:09:13.634551+0000","last_undegraded":"2024-03-11T19:09:13.634551+0000","last_fullsized":"2024-03-11T19:09:13.634551+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:15:42.260509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.635452+0000","last_change":"2024-03-11T19:07:49.755564+0000","last_active":"2024-03-11T19:09:13.635452+0000","last_peered":"2024-03-11T19:09:13.635452+0000","last_clean":"2024-03-11T19:09:13.635452+0000","last_became_active":"2024-03-11T19:07:49.751322+0000","last_became_peered":"2024-03-11T19:07:49.751322+0000","last_unstale":"2024-03-11T19:09:13.635452+0000","last_undegraded":"2024-03-11T19:09:13.635452+0000","last_fullsized":"2024-03-11T19:09:13.635452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:43:18.487887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.637740+0000","last_change":"2024-03-11T19:07:49.731619+0000","last_active":"2024-03-11T19:09:13.637740+0000","last_peered":"2024-03-11T19:09:13.637740+0000","last_clean":"2024-03-11T19:09:13.637740+0000","last_became_active":"2024-03-11T19:07:49.731323+0000","last_became_peered":"2024-03-11T19:07:49.731323+0000","last_unstale":"2024-03-11T19:09:13.637740+0000","last_undegraded":"2024-03-11T19:09:13.637740+0000","last_fullsized":"2024-03-11T19:09:13.637740+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:20:25.592709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.646995+0000","last_change":"2024-03-11T19:07:49.751851+0000","last_active":"2024-03-11T19:09:13.646995+0000","last_peered":"2024-03-11T19:09:13.646995+0000","last_clean":"2024-03-11T19:09:13.646995+0000","last_became_active":"2024-03-11T19:07:49.750068+0000","last_became_peered":"2024-03-11T19:07:49.750068+0000","last_unstale":"2024-03-11T19:09:13.646995+0000","last_undegraded":"2024-03-11T19:09:13.646995+0000","last_fullsized":"2024-03-11T19:09:13.646995+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:59:35.820107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.652310+0000","last_change":"2024-03-11T19:07:49.752326+0000","last_active":"2024-03-11T19:09:13.652310+0000","last_peered":"2024-03-11T19:09:13.652310+0000","last_clean":"2024-03-11T19:09:13.652310+0000","last_became_active":"2024-03-11T19:07:49.748052+0000","last_became_peered":"2024-03-11T19:07:49.748052+0000","last_unstale":"2024-03-11T19:09:13.652310+0000","last_undegraded":"2024-03-11T19:09:13.652310+0000","last_fullsized":"2024-03-11T19:09:13.652310+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:58:13.325964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.639621+0000","last_change":"2024-03-11T19:07:49.748054+0000","last_active":"2024-03-11T19:09:13.639621+0000","last_peered":"2024-03-11T19:09:13.639621+0000","last_clean":"2024-03-11T19:09:13.639621+0000","last_became_active":"2024-03-11T19:07:49.747765+0000","last_became_peered":"2024-03-11T19:07:49.747765+0000","last_unstale":"2024-03-11T19:09:13.639621+0000","last_undegraded":"2024-03-11T19:09:13.639621+0000","last_fullsized":"2024-03-11T19:09:13.639621+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:18:05.599512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.651823+0000","last_change":"2024-03-11T19:07:49.751622+0000","last_active":"2024-03-11T19:09:13.651823+0000","last_peered":"2024-03-11T19:09:13.651823+0000","last_clean":"2024-03-11T19:09:13.651823+0000","last_became_active":"2024-03-11T19:07:49.751317+0000","last_became_peered":"2024-03-11T19:07:49.751317+0000","last_unstale":"2024-03-11T19:09:13.651823+0000","last_undegraded":"2024-03-11T19:09:13.651823+0000","last_fullsized":"2024-03-11T19:09:13.651823+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:49.836096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.630547+0000","last_change":"2024-03-11T19:07:49.749160+0000","last_active":"2024-03-11T19:09:13.630547+0000","last_peered":"2024-03-11T19:09:13.630547+0000","last_clean":"2024-03-11T19:09:13.630547+0000","last_became_active":"2024-03-11T19:07:49.748838+0000","last_became_peered":"2024-03-11T19:07:49.748838+0000","last_unstale":"2024-03-11T19:09:13.630547+0000","last_undegraded":"2024-03-11T19:09:13.630547+0000","last_fullsized":"2024-03-11T19:09:13.630547+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:30:58.327196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.647032+0000","last_change":"2024-03-11T19:07:49.749965+0000","last_active":"2024-03-11T19:09:13.647032+0000","last_peered":"2024-03-11T19:09:13.647032+0000","last_clean":"2024-03-11T19:09:13.647032+0000","last_became_active":"2024-03-11T19:07:49.749602+0000","last_became_peered":"2024-03-11T19:07:49.749602+0000","last_unstale":"2024-03-11T19:09:13.647032+0000","last_undegraded":"2024-03-11T19:09:13.647032+0000","last_fullsized":"2024-03-11T19:09:13.647032+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:36:08.807251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.640585+0000","last_change":"2024-03-11T19:07:49.747557+0000","last_active":"2024-03-11T19:09:13.640585+0000","last_peered":"2024-03-11T19:09:13.640585+0000","last_clean":"2024-03-11T19:09:13.640585+0000","last_became_active":"2024-03-11T19:07:49.745804+0000","last_became_peered":"2024-03-11T19:07:49.745804+0000","last_unstale":"2024-03-11T19:09:13.640585+0000","last_undegraded":"2024-03-11T19:09:13.640585+0000","last_fullsized":"2024-03-11T19:09:13.640585+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:07:26.939414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.649130+0000","last_change":"2024-03-11T19:07:49.744070+0000","last_active":"2024-03-11T19:09:13.649130+0000","last_peered":"2024-03-11T19:09:13.649130+0000","last_clean":"2024-03-11T19:09:13.649130+0000","last_became_active":"2024-03-11T19:07:49.743736+0000","last_became_peered":"2024-03-11T19:07:49.743736+0000","last_unstale":"2024-03-11T19:09:13.649130+0000","last_undegraded":"2024-03-11T19:09:13.649130+0000","last_fullsized":"2024-03-11T19:09:13.649130+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:50:33.862644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.657246+0000","last_change":"2024-03-11T19:07:49.748512+0000","last_active":"2024-03-11T19:09:13.657246+0000","last_peered":"2024-03-11T19:09:13.657246+0000","last_clean":"2024-03-11T19:09:13.657246+0000","last_became_active":"2024-03-11T19:07:49.748223+0000","last_became_peered":"2024-03-11T19:07:49.748223+0000","last_unstale":"2024-03-11T19:09:13.657246+0000","last_undegraded":"2024-03-11T19:09:13.657246+0000","last_fullsized":"2024-03-11T19:09:13.657246+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:49:43.305677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.655548+0000","last_change":"2024-03-11T19:07:49.754322+0000","last_active":"2024-03-11T19:09:13.655548+0000","last_peered":"2024-03-11T19:09:13.655548+0000","last_clean":"2024-03-11T19:09:13.655548+0000","last_became_active":"2024-03-11T19:07:49.753757+0000","last_became_peered":"2024-03-11T19:07:49.753757+0000","last_unstale":"2024-03-11T19:09:13.655548+0000","last_undegraded":"2024-03-11T19:09:13.655548+0000","last_fullsized":"2024-03-11T19:09:13.655548+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:49.684978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.657358+0000","last_change":"2024-03-11T19:07:49.749546+0000","last_active":"2024-03-11T19:09:13.657358+0000","last_peered":"2024-03-11T19:09:13.657358+0000","last_clean":"2024-03-11T19:09:13.657358+0000","last_became_active":"2024-03-11T19:07:49.749200+0000","last_became_peered":"2024-03-11T19:07:49.749200+0000","last_unstale":"2024-03-11T19:09:13.657358+0000","last_undegraded":"2024-03-11T19:09:13.657358+0000","last_fullsized":"2024-03-11T19:09:13.657358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:26:12.914015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.645227+0000","last_change":"2024-03-11T19:07:49.750388+0000","last_active":"2024-03-11T19:09:13.645227+0000","last_peered":"2024-03-11T19:09:13.645227+0000","last_clean":"2024-03-11T19:09:13.645227+0000","last_became_active":"2024-03-11T19:07:49.750016+0000","last_became_peered":"2024-03-11T19:07:49.750016+0000","last_unstale":"2024-03-11T19:09:13.645227+0000","last_undegraded":"2024-03-11T19:09:13.645227+0000","last_fullsized":"2024-03-11T19:09:13.645227+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:32:23.125695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'14","reported_seq":155,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:55.832399+0000","last_change":"2024-03-11T19:07:48.738369+0000","last_active":"2024-03-11T19:09:55.832399+0000","last_peered":"2024-03-11T19:09:55.832399+0000","last_clean":"2024-03-11T19:09:55.832399+0000","last_became_active":"2024-03-11T19:07:48.738010+0000","last_became_peered":"2024-03-11T19:07:48.738010+0000","last_unstale":"2024-03-11T19:09:55.832399+0000","last_undegraded":"2024-03-11T19:09:55.832399+0000","last_fullsized":"2024-03-11T19:09:55.832399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:06:48.446454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.646494+0000","last_change":"2024-03-11T19:07:49.752622+0000","last_active":"2024-03-11T19:09:13.646494+0000","last_peered":"2024-03-11T19:09:13.646494+0000","last_clean":"2024-03-11T19:09:13.646494+0000","last_became_active":"2024-03-11T19:07:49.752242+0000","last_became_peered":"2024-03-11T19:07:49.752242+0000","last_unstale":"2024-03-11T19:09:13.646494+0000","last_undegraded":"2024-03-11T19:09:13.646494+0000","last_fullsized":"2024-03-11T19:09:13.646494+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:03:50.609641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990514+0000","last_change":"2024-03-11T19:07:48.751657+0000","last_active":"2024-03-11T19:08:44.990514+0000","last_peered":"2024-03-11T19:08:44.990514+0000","last_clean":"2024-03-11T19:08:44.990514+0000","last_became_active":"2024-03-11T19:07:48.751294+0000","last_became_peered":"2024-03-11T19:07:48.751294+0000","last_unstale":"2024-03-11T19:08:44.990514+0000","last_undegraded":"2024-03-11T19:08:44.990514+0000","last_fullsized":"2024-03-11T19:08:44.990514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:33:58.550707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.655152+0000","last_change":"2024-03-11T19:07:49.759460+0000","last_active":"2024-03-11T19:09:13.655152+0000","last_peered":"2024-03-11T19:09:13.655152+0000","last_clean":"2024-03-11T19:09:13.655152+0000","last_became_active":"2024-03-11T19:07:49.759136+0000","last_became_peered":"2024-03-11T19:07:49.759136+0000","last_unstale":"2024-03-11T19:09:13.655152+0000","last_undegraded":"2024-03-11T19:09:13.655152+0000","last_fullsized":"2024-03-11T19:09:13.655152+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:10:37.273896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'12","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991106+0000","last_change":"2024-03-11T19:07:48.744695+0000","last_active":"2024-03-11T19:08:44.991106+0000","last_peered":"2024-03-11T19:08:44.991106+0000","last_clean":"2024-03-11T19:08:44.991106+0000","last_became_active":"2024-03-11T19:07:48.744378+0000","last_became_peered":"2024-03-11T19:07:48.744378+0000","last_unstale":"2024-03-11T19:08:44.991106+0000","last_undegraded":"2024-03-11T19:08:44.991106+0000","last_fullsized":"2024-03-11T19:08:44.991106+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:55:46.601414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.655586+0000","last_change":"2024-03-11T19:07:49.761429+0000","last_active":"2024-03-11T19:09:13.655586+0000","last_peered":"2024-03-11T19:09:13.655586+0000","last_clean":"2024-03-11T19:09:13.655586+0000","last_became_active":"2024-03-11T19:07:49.761120+0000","last_became_peered":"2024-03-11T19:07:49.761120+0000","last_unstale":"2024-03-11T19:09:13.655586+0000","last_undegraded":"2024-03-11T19:09:13.655586+0000","last_fullsized":"2024-03-11T19:09:13.655586+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:13:45.839528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990713+0000","last_change":"2024-03-11T19:07:48.740451+0000","last_active":"2024-03-11T19:08:44.990713+0000","last_peered":"2024-03-11T19:08:44.990713+0000","last_clean":"2024-03-11T19:08:44.990713+0000","last_became_active":"2024-03-11T19:07:48.740097+0000","last_became_peered":"2024-03-11T19:07:48.740097+0000","last_unstale":"2024-03-11T19:08:44.990713+0000","last_undegraded":"2024-03-11T19:08:44.990713+0000","last_fullsized":"2024-03-11T19:08:44.990713+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:37.766790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.657267+0000","last_change":"2024-03-11T19:07:49.750404+0000","last_active":"2024-03-11T19:09:13.657267+0000","last_peered":"2024-03-11T19:09:13.657267+0000","last_clean":"2024-03-11T19:09:13.657267+0000","last_became_active":"2024-03-11T19:07:49.749941+0000","last_became_peered":"2024-03-11T19:07:49.749941+0000","last_unstale":"2024-03-11T19:09:13.657267+0000","last_undegraded":"2024-03-11T19:09:13.657267+0000","last_fullsized":"2024-03-11T19:09:13.657267+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:55.766729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'8","reported_seq":153,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:56.835200+0000","last_change":"2024-03-11T19:07:48.737100+0000","last_active":"2024-03-11T19:09:56.835200+0000","last_peered":"2024-03-11T19:09:56.835200+0000","last_clean":"2024-03-11T19:09:56.835200+0000","last_became_active":"2024-03-11T19:07:48.736768+0000","last_became_peered":"2024-03-11T19:07:48.736768+0000","last_unstale":"2024-03-11T19:09:56.835200+0000","last_undegraded":"2024-03-11T19:09:56.835200+0000","last_fullsized":"2024-03-11T19:09:56.835200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:18:36.872151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1647,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.634648+0000","last_change":"2024-03-11T19:07:49.739020+0000","last_active":"2024-03-11T19:09:13.634648+0000","last_peered":"2024-03-11T19:09:13.634648+0000","last_clean":"2024-03-11T19:09:13.634648+0000","last_became_active":"2024-03-11T19:07:49.738642+0000","last_became_peered":"2024-03-11T19:07:49.738642+0000","last_unstale":"2024-03-11T19:09:13.634648+0000","last_undegraded":"2024-03-11T19:09:13.634648+0000","last_fullsized":"2024-03-11T19:09:13.634648+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:42:29.040242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037233+0000","last_change":"2024-03-11T19:07:48.745192+0000","last_active":"2024-03-11T19:08:45.037233+0000","last_peered":"2024-03-11T19:08:45.037233+0000","last_clean":"2024-03-11T19:08:45.037233+0000","last_became_active":"2024-03-11T19:07:48.744868+0000","last_became_peered":"2024-03-11T19:07:48.744868+0000","last_unstale":"2024-03-11T19:08:45.037233+0000","last_undegraded":"2024-03-11T19:08:45.037233+0000","last_fullsized":"2024-03-11T19:08:45.037233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:19:15.620186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.640619+0000","last_change":"2024-03-11T19:07:49.737149+0000","last_active":"2024-03-11T19:09:13.640619+0000","last_peered":"2024-03-11T19:09:13.640619+0000","last_clean":"2024-03-11T19:09:13.640619+0000","last_became_active":"2024-03-11T19:07:49.736797+0000","last_became_peered":"2024-03-11T19:07:49.736797+0000","last_unstale":"2024-03-11T19:09:13.640619+0000","last_undegraded":"2024-03-11T19:09:13.640619+0000","last_fullsized":"2024-03-11T19:09:13.640619+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:12:10.282062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.988596+0000","last_change":"2024-03-11T19:07:48.743948+0000","last_active":"2024-03-11T19:08:44.988596+0000","last_peered":"2024-03-11T19:08:44.988596+0000","last_clean":"2024-03-11T19:08:44.988596+0000","last_became_active":"2024-03-11T19:07:48.743451+0000","last_became_peered":"2024-03-11T19:07:48.743451+0000","last_unstale":"2024-03-11T19:08:44.988596+0000","last_undegraded":"2024-03-11T19:08:44.988596+0000","last_fullsized":"2024-03-11T19:08:44.988596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:03:19.823287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.636529+0000","last_change":"2024-03-11T19:07:49.745518+0000","last_active":"2024-03-11T19:09:13.636529+0000","last_peered":"2024-03-11T19:09:13.636529+0000","last_clean":"2024-03-11T19:09:13.636529+0000","last_became_active":"2024-03-11T19:07:49.743719+0000","last_became_peered":"2024-03-11T19:07:49.743719+0000","last_unstale":"2024-03-11T19:09:13.636529+0000","last_undegraded":"2024-03-11T19:09:13.636529+0000","last_fullsized":"2024-03-11T19:09:13.636529+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:25:33.855679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036218+0000","last_change":"2024-03-11T19:07:48.734044+0000","last_active":"2024-03-11T19:08:45.036218+0000","last_peered":"2024-03-11T19:08:45.036218+0000","last_clean":"2024-03-11T19:08:45.036218+0000","last_became_active":"2024-03-11T19:07:48.733723+0000","last_became_peered":"2024-03-11T19:07:48.733723+0000","last_unstale":"2024-03-11T19:08:45.036218+0000","last_undegraded":"2024-03-11T19:08:45.036218+0000","last_fullsized":"2024-03-11T19:08:45.036218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:53:37.070296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.635887+0000","last_change":"2024-03-11T19:07:49.743729+0000","last_active":"2024-03-11T19:09:13.635887+0000","last_peered":"2024-03-11T19:09:13.635887+0000","last_clean":"2024-03-11T19:09:13.635887+0000","last_became_active":"2024-03-11T19:07:49.742992+0000","last_became_peered":"2024-03-11T19:07:49.742992+0000","last_unstale":"2024-03-11T19:09:13.635887+0000","last_undegraded":"2024-03-11T19:09:13.635887+0000","last_fullsized":"2024-03-11T19:09:13.635887+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:46.477177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"89'7","reported_seq":268,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:56.834343+0000","last_change":"2024-03-11T19:07:48.738775+0000","last_active":"2024-03-11T19:09:56.834343+0000","last_peered":"2024-03-11T19:09:56.834343+0000","last_clean":"2024-03-11T19:09:56.834343+0000","last_became_active":"2024-03-11T19:07:48.738437+0000","last_became_peered":"2024-03-11T19:07:48.738437+0000","last_unstale":"2024-03-11T19:09:56.834343+0000","last_undegraded":"2024-03-11T19:09:56.834343+0000","last_fullsized":"2024-03-11T19:09:56.834343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:17:20.020756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":242,"num_read_kb":242,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.654932+0000","last_change":"2024-03-11T19:07:49.760293+0000","last_active":"2024-03-11T19:09:13.654932+0000","last_peered":"2024-03-11T19:09:13.654932+0000","last_clean":"2024-03-11T19:09:13.654932+0000","last_became_active":"2024-03-11T19:07:49.759977+0000","last_became_peered":"2024-03-11T19:07:49.759977+0000","last_unstale":"2024-03-11T19:09:13.654932+0000","last_undegraded":"2024-03-11T19:09:13.654932+0000","last_fullsized":"2024-03-11T19:09:13.654932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:07:44.726756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985956+0000","last_change":"2024-03-11T19:07:48.740672+0000","last_active":"2024-03-11T19:08:44.985956+0000","last_peered":"2024-03-11T19:08:44.985956+0000","last_clean":"2024-03-11T19:08:44.985956+0000","last_became_active":"2024-03-11T19:07:48.739331+0000","last_became_peered":"2024-03-11T19:07:48.739331+0000","last_unstale":"2024-03-11T19:08:44.985956+0000","last_undegraded":"2024-03-11T19:08:44.985956+0000","last_fullsized":"2024-03-11T19:08:44.985956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:19:28.375515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.650493+0000","last_change":"2024-03-11T19:07:49.742780+0000","last_active":"2024-03-11T19:09:13.650493+0000","last_peered":"2024-03-11T19:09:13.650493+0000","last_clean":"2024-03-11T19:09:13.650493+0000","last_became_active":"2024-03-11T19:07:49.742442+0000","last_became_peered":"2024-03-11T19:07:49.742442+0000","last_unstale":"2024-03-11T19:09:13.650493+0000","last_undegraded":"2024-03-11T19:09:13.650493+0000","last_fullsized":"2024-03-11T19:09:13.650493+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:11:13.838225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036582+0000","last_change":"2024-03-11T19:07:48.752617+0000","last_active":"2024-03-11T19:08:45.036582+0000","last_peered":"2024-03-11T19:08:45.036582+0000","last_clean":"2024-03-11T19:08:45.036582+0000","last_became_active":"2024-03-11T19:07:48.752181+0000","last_became_peered":"2024-03-11T19:07:48.752181+0000","last_unstale":"2024-03-11T19:08:45.036582+0000","last_undegraded":"2024-03-11T19:08:45.036582+0000","last_fullsized":"2024-03-11T19:08:45.036582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:05:37.162866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.654679+0000","last_change":"2024-03-11T19:07:49.753672+0000","last_active":"2024-03-11T19:09:13.654679+0000","last_peered":"2024-03-11T19:09:13.654679+0000","last_clean":"2024-03-11T19:09:13.654679+0000","last_became_active":"2024-03-11T19:07:49.753315+0000","last_became_peered":"2024-03-11T19:07:49.753315+0000","last_unstale":"2024-03-11T19:09:13.654679+0000","last_undegraded":"2024-03-11T19:09:13.654679+0000","last_fullsized":"2024-03-11T19:09:13.654679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:30:58.503249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'38","reported_seq":310,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:56.834983+0000","last_change":"2024-03-11T19:07:48.740424+0000","last_active":"2024-03-11T19:09:56.834983+0000","last_peered":"2024-03-11T19:09:56.834983+0000","last_clean":"2024-03-11T19:09:56.834983+0000","last_became_active":"2024-03-11T19:07:48.740077+0000","last_became_peered":"2024-03-11T19:07:48.740077+0000","last_unstale":"2024-03-11T19:09:56.834983+0000","last_undegraded":"2024-03-11T19:09:56.834983+0000","last_fullsized":"2024-03-11T19:09:56.834983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:30:29.193861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66493,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":253,"num_read_kb":192,"num_write":40,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.652816+0000","last_change":"2024-03-11T19:07:49.746440+0000","last_active":"2024-03-11T19:09:13.652816+0000","last_peered":"2024-03-11T19:09:13.652816+0000","last_clean":"2024-03-11T19:09:13.652816+0000","last_became_active":"2024-03-11T19:07:49.746106+0000","last_became_peered":"2024-03-11T19:07:49.746106+0000","last_unstale":"2024-03-11T19:09:13.652816+0000","last_undegraded":"2024-03-11T19:09:13.652816+0000","last_fullsized":"2024-03-11T19:09:13.652816+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:44:58.039917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'9","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.986027+0000","last_change":"2024-03-11T19:07:48.744437+0000","last_active":"2024-03-11T19:08:44.986027+0000","last_peered":"2024-03-11T19:08:44.986027+0000","last_clean":"2024-03-11T19:08:44.986027+0000","last_became_active":"2024-03-11T19:07:48.744142+0000","last_became_peered":"2024-03-11T19:07:48.744142+0000","last_unstale":"2024-03-11T19:08:44.986027+0000","last_undegraded":"2024-03-11T19:08:44.986027+0000","last_fullsized":"2024-03-11T19:08:44.986027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:32:09.076975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.653140+0000","last_change":"2024-03-11T19:07:49.751191+0000","last_active":"2024-03-11T19:09:13.653140+0000","last_peered":"2024-03-11T19:09:13.653140+0000","last_clean":"2024-03-11T19:09:13.653140+0000","last_became_active":"2024-03-11T19:07:49.750849+0000","last_became_peered":"2024-03-11T19:07:49.750849+0000","last_unstale":"2024-03-11T19:09:13.653140+0000","last_undegraded":"2024-03-11T19:09:13.653140+0000","last_fullsized":"2024-03-11T19:09:13.653140+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:19:18.416905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036229+0000","last_change":"2024-03-11T19:07:48.742808+0000","last_active":"2024-03-11T19:08:45.036229+0000","last_peered":"2024-03-11T19:08:45.036229+0000","last_clean":"2024-03-11T19:08:45.036229+0000","last_became_active":"2024-03-11T19:07:48.742462+0000","last_became_peered":"2024-03-11T19:07:48.742462+0000","last_unstale":"2024-03-11T19:08:45.036229+0000","last_undegraded":"2024-03-11T19:08:45.036229+0000","last_fullsized":"2024-03-11T19:08:45.036229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:19.109299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.651622+0000","last_change":"2024-03-11T19:07:49.744673+0000","last_active":"2024-03-11T19:09:13.651622+0000","last_peered":"2024-03-11T19:09:13.651622+0000","last_clean":"2024-03-11T19:09:13.651622+0000","last_became_active":"2024-03-11T19:07:49.744313+0000","last_became_peered":"2024-03-11T19:07:49.744313+0000","last_unstale":"2024-03-11T19:09:13.651622+0000","last_undegraded":"2024-03-11T19:09:13.651622+0000","last_fullsized":"2024-03-11T19:09:13.651622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:21:04.759356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":151,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:57.836159+0000","last_change":"2024-03-11T19:07:48.736024+0000","last_active":"2024-03-11T19:09:57.836159+0000","last_peered":"2024-03-11T19:09:57.836159+0000","last_clean":"2024-03-11T19:09:57.836159+0000","last_became_active":"2024-03-11T19:07:48.735720+0000","last_became_peered":"2024-03-11T19:07:48.735720+0000","last_unstale":"2024-03-11T19:09:57.836159+0000","last_undegraded":"2024-03-11T19:09:57.836159+0000","last_fullsized":"2024-03-11T19:09:57.836159+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:04:57.386267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":127,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.640229+0000","last_change":"2024-03-11T19:07:49.742924+0000","last_active":"2024-03-11T19:09:13.640229+0000","last_peered":"2024-03-11T19:09:13.640229+0000","last_clean":"2024-03-11T19:09:13.640229+0000","last_became_active":"2024-03-11T19:07:49.742542+0000","last_became_peered":"2024-03-11T19:07:49.742542+0000","last_unstale":"2024-03-11T19:09:13.640229+0000","last_undegraded":"2024-03-11T19:09:13.640229+0000","last_fullsized":"2024-03-11T19:09:13.640229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:08:11.944311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'17","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990343+0000","last_change":"2024-03-11T19:07:48.753113+0000","last_active":"2024-03-11T19:08:44.990343+0000","last_peered":"2024-03-11T19:08:44.990343+0000","last_clean":"2024-03-11T19:08:44.990343+0000","last_became_active":"2024-03-11T19:07:48.752517+0000","last_became_peered":"2024-03-11T19:07:48.752517+0000","last_unstale":"2024-03-11T19:08:44.990343+0000","last_undegraded":"2024-03-11T19:08:44.990343+0000","last_fullsized":"2024-03-11T19:08:44.990343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:48:33.441125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.653632+0000","last_change":"2024-03-11T19:08:45.038243+0000","last_active":"2024-03-11T19:09:13.653632+0000","last_peered":"2024-03-11T19:09:13.653632+0000","last_clean":"2024-03-11T19:09:13.653632+0000","last_became_active":"2024-03-11T19:08:45.037969+0000","last_became_peered":"2024-03-11T19:08:45.037969+0000","last_unstale":"2024-03-11T19:09:13.653632+0000","last_undegraded":"2024-03-11T19:09:13.653632+0000","last_fullsized":"2024-03-11T19:09:13.653632+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:29:09.747958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"89'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.991151+0000","last_change":"2024-03-11T19:07:48.739571+0000","last_active":"2024-03-11T19:08:44.991151+0000","last_peered":"2024-03-11T19:08:44.991151+0000","last_clean":"2024-03-11T19:08:44.991151+0000","last_became_active":"2024-03-11T19:07:48.739159+0000","last_became_peered":"2024-03-11T19:07:48.739159+0000","last_unstale":"2024-03-11T19:08:44.991151+0000","last_undegraded":"2024-03-11T19:08:44.991151+0000","last_fullsized":"2024-03-11T19:08:44.991151+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:46:43.474591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.639217+0000","last_change":"2024-03-11T19:07:49.744760+0000","last_active":"2024-03-11T19:09:13.639217+0000","last_peered":"2024-03-11T19:09:13.639217+0000","last_clean":"2024-03-11T19:09:13.639217+0000","last_became_active":"2024-03-11T19:07:49.744265+0000","last_became_peered":"2024-03-11T19:07:49.744265+0000","last_unstale":"2024-03-11T19:09:13.639217+0000","last_undegraded":"2024-03-11T19:09:13.639217+0000","last_fullsized":"2024-03-11T19:09:13.639217+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:09:49.823462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"87'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985221+0000","last_change":"2024-03-11T19:07:48.740223+0000","last_active":"2024-03-11T19:08:44.985221+0000","last_peered":"2024-03-11T19:08:44.985221+0000","last_clean":"2024-03-11T19:08:44.985221+0000","last_became_active":"2024-03-11T19:07:48.739569+0000","last_became_peered":"2024-03-11T19:07:48.739569+0000","last_unstale":"2024-03-11T19:08:44.985221+0000","last_undegraded":"2024-03-11T19:08:44.985221+0000","last_fullsized":"2024-03-11T19:08:44.985221+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:04:19.384523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.641047+0000","last_change":"2024-03-11T19:07:49.755620+0000","last_active":"2024-03-11T19:09:13.641047+0000","last_peered":"2024-03-11T19:09:13.641047+0000","last_clean":"2024-03-11T19:09:13.641047+0000","last_became_active":"2024-03-11T19:07:49.750886+0000","last_became_peered":"2024-03-11T19:07:49.750886+0000","last_unstale":"2024-03-11T19:09:13.641047+0000","last_undegraded":"2024-03-11T19:09:13.641047+0000","last_fullsized":"2024-03-11T19:09:13.641047+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:35:54.191061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'6","reported_seq":147,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:54.831635+0000","last_change":"2024-03-11T19:07:48.743340+0000","last_active":"2024-03-11T19:09:54.831635+0000","last_peered":"2024-03-11T19:09:54.831635+0000","last_clean":"2024-03-11T19:09:54.831635+0000","last_became_active":"2024-03-11T19:07:48.742947+0000","last_became_peered":"2024-03-11T19:07:48.742947+0000","last_unstale":"2024-03-11T19:09:54.831635+0000","last_undegraded":"2024-03-11T19:09:54.831635+0000","last_fullsized":"2024-03-11T19:09:54.831635+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:40:25.233150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.643001+0000","last_change":"2024-03-11T19:07:49.753668+0000","last_active":"2024-03-11T19:09:13.643001+0000","last_peered":"2024-03-11T19:09:13.643001+0000","last_clean":"2024-03-11T19:09:13.643001+0000","last_became_active":"2024-03-11T19:07:49.750574+0000","last_became_peered":"2024-03-11T19:07:49.750574+0000","last_unstale":"2024-03-11T19:09:13.643001+0000","last_undegraded":"2024-03-11T19:09:13.643001+0000","last_fullsized":"2024-03-11T19:09:13.643001+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:12:32.711847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.640771+0000","last_change":"2024-03-11T19:07:49.739538+0000","last_active":"2024-03-11T19:09:13.640771+0000","last_peered":"2024-03-11T19:09:13.640771+0000","last_clean":"2024-03-11T19:09:13.640771+0000","last_became_active":"2024-03-11T19:07:49.738617+0000","last_became_peered":"2024-03-11T19:07:49.738617+0000","last_unstale":"2024-03-11T19:09:13.640771+0000","last_undegraded":"2024-03-11T19:09:13.640771+0000","last_fullsized":"2024-03-11T19:09:13.640771+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:43:08.697125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'6","reported_seq":150,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:57.835585+0000","last_change":"2024-03-11T19:07:48.749527+0000","last_active":"2024-03-11T19:09:57.835585+0000","last_peered":"2024-03-11T19:09:57.835585+0000","last_clean":"2024-03-11T19:09:57.835585+0000","last_became_active":"2024-03-11T19:07:48.749026+0000","last_became_peered":"2024-03-11T19:07:48.749026+0000","last_unstale":"2024-03-11T19:09:57.835585+0000","last_undegraded":"2024-03-11T19:09:57.835585+0000","last_fullsized":"2024-03-11T19:09:57.835585+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:20:19.058429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":124,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.646783+0000","last_change":"2024-03-11T19:07:49.757696+0000","last_active":"2024-03-11T19:09:13.646783+0000","last_peered":"2024-03-11T19:09:13.646783+0000","last_clean":"2024-03-11T19:09:13.646783+0000","last_became_active":"2024-03-11T19:07:49.757380+0000","last_became_peered":"2024-03-11T19:07:49.757380+0000","last_unstale":"2024-03-11T19:09:13.646783+0000","last_undegraded":"2024-03-11T19:09:13.646783+0000","last_fullsized":"2024-03-11T19:09:13.646783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:03:22.954619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036816+0000","last_change":"2024-03-11T19:07:48.737504+0000","last_active":"2024-03-11T19:08:45.036816+0000","last_peered":"2024-03-11T19:08:45.036816+0000","last_clean":"2024-03-11T19:08:45.036816+0000","last_became_active":"2024-03-11T19:07:48.737147+0000","last_became_peered":"2024-03-11T19:07:48.737147+0000","last_unstale":"2024-03-11T19:08:45.036816+0000","last_undegraded":"2024-03-11T19:08:45.036816+0000","last_fullsized":"2024-03-11T19:08:45.036816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:04.819617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":344,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985900+0000","last_change":"2024-03-11T19:06:35.035452+0000","last_active":"2024-03-11T19:08:44.985900+0000","last_peered":"2024-03-11T19:08:44.985900+0000","last_clean":"2024-03-11T19:08:44.985900+0000","last_became_active":"2024-03-11T19:06:35.035076+0000","last_became_peered":"2024-03-11T19:06:35.035076+0000","last_unstale":"2024-03-11T19:08:44.985900+0000","last_undegraded":"2024-03-11T19:08:44.985900+0000","last_fullsized":"2024-03-11T19:08:44.985900+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:03:10.143002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:03:10.143002+0000","last_clean_scrub_stamp":"2024-03-11T19:03:10.143002+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:39:37.348091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.642831+0000","last_change":"2024-03-11T19:07:49.743000+0000","last_active":"2024-03-11T19:09:13.642831+0000","last_peered":"2024-03-11T19:09:13.642831+0000","last_clean":"2024-03-11T19:09:13.642831+0000","last_became_active":"2024-03-11T19:07:49.742691+0000","last_became_peered":"2024-03-11T19:07:49.742691+0000","last_unstale":"2024-03-11T19:09:13.642831+0000","last_undegraded":"2024-03-11T19:09:13.642831+0000","last_fullsized":"2024-03-11T19:09:13.642831+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:40:21.519929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.985719+0000","last_change":"2024-03-11T19:07:48.739038+0000","last_active":"2024-03-11T19:08:44.985719+0000","last_peered":"2024-03-11T19:08:44.985719+0000","last_clean":"2024-03-11T19:08:44.985719+0000","last_became_active":"2024-03-11T19:07:48.738714+0000","last_became_peered":"2024-03-11T19:07:48.738714+0000","last_unstale":"2024-03-11T19:08:44.985719+0000","last_undegraded":"2024-03-11T19:08:44.985719+0000","last_fullsized":"2024-03-11T19:08:44.985719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:17:43.086679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.998660+0000","last_change":"2024-03-11T19:08:44.998660+0000","last_active":"2024-03-11T19:08:44.998660+0000","last_peered":"2024-03-11T19:08:44.998660+0000","last_clean":"2024-03-11T19:08:44.998660+0000","last_became_active":"2024-03-11T19:08:44.991682+0000","last_became_peered":"2024-03-11T19:08:44.991682+0000","last_unstale":"2024-03-11T19:08:44.998660+0000","last_undegraded":"2024-03-11T19:08:44.998660+0000","last_fullsized":"2024-03-11T19:08:44.998660+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:19:44.266076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.645393+0000","last_change":"2024-03-11T19:07:49.742434+0000","last_active":"2024-03-11T19:09:13.645393+0000","last_peered":"2024-03-11T19:09:13.645393+0000","last_clean":"2024-03-11T19:09:13.645393+0000","last_became_active":"2024-03-11T19:07:49.742048+0000","last_became_peered":"2024-03-11T19:07:49.742048+0000","last_unstale":"2024-03-11T19:09:13.645393+0000","last_undegraded":"2024-03-11T19:09:13.645393+0000","last_fullsized":"2024-03-11T19:09:13.645393+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:39:22.685055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:53.757286+0000","last_change":"2024-03-11T19:07:48.744728+0000","last_active":"2024-03-11T19:09:53.757286+0000","last_peered":"2024-03-11T19:09:53.757286+0000","last_clean":"2024-03-11T19:09:53.757286+0000","last_became_active":"2024-03-11T19:07:48.744378+0000","last_became_peered":"2024-03-11T19:07:48.744378+0000","last_unstale":"2024-03-11T19:09:53.757286+0000","last_undegraded":"2024-03-11T19:09:53.757286+0000","last_fullsized":"2024-03-11T19:09:53.757286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:42.823040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.654164+0000","last_change":"2024-03-11T19:07:49.744261+0000","last_active":"2024-03-11T19:09:13.654164+0000","last_peered":"2024-03-11T19:09:13.654164+0000","last_clean":"2024-03-11T19:09:13.654164+0000","last_became_active":"2024-03-11T19:07:49.743946+0000","last_became_peered":"2024-03-11T19:07:49.743946+0000","last_unstale":"2024-03-11T19:09:13.654164+0000","last_undegraded":"2024-03-11T19:09:13.654164+0000","last_fullsized":"2024-03-11T19:09:13.654164+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:54:07.928899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'7","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.037249+0000","last_change":"2024-03-11T19:07:48.740059+0000","last_active":"2024-03-11T19:08:45.037249+0000","last_peered":"2024-03-11T19:08:45.037249+0000","last_clean":"2024-03-11T19:08:45.037249+0000","last_became_active":"2024-03-11T19:07:48.739626+0000","last_became_peered":"2024-03-11T19:07:48.739626+0000","last_unstale":"2024-03-11T19:08:45.037249+0000","last_undegraded":"2024-03-11T19:08:45.037249+0000","last_fullsized":"2024-03-11T19:08:45.037249+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:45:12.219203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.647418+0000","last_change":"2024-03-11T19:07:49.746041+0000","last_active":"2024-03-11T19:09:13.647418+0000","last_peered":"2024-03-11T19:09:13.647418+0000","last_clean":"2024-03-11T19:09:13.647418+0000","last_became_active":"2024-03-11T19:07:49.745602+0000","last_became_peered":"2024-03-11T19:07:49.745602+0000","last_unstale":"2024-03-11T19:09:13.647418+0000","last_undegraded":"2024-03-11T19:09:13.647418+0000","last_fullsized":"2024-03-11T19:09:13.647418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:23:44.596929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.035863+0000","last_change":"2024-03-11T19:07:48.748677+0000","last_active":"2024-03-11T19:08:45.035863+0000","last_peered":"2024-03-11T19:08:45.035863+0000","last_clean":"2024-03-11T19:08:45.035863+0000","last_became_active":"2024-03-11T19:07:48.748470+0000","last_became_peered":"2024-03-11T19:07:48.748470+0000","last_unstale":"2024-03-11T19:08:45.035863+0000","last_undegraded":"2024-03-11T19:08:45.035863+0000","last_fullsized":"2024-03-11T19:08:45.035863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:24:55.240722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.620826+0000","last_change":"2024-03-11T19:07:49.746522+0000","last_active":"2024-03-11T19:09:13.620826+0000","last_peered":"2024-03-11T19:09:13.620826+0000","last_clean":"2024-03-11T19:09:13.620826+0000","last_became_active":"2024-03-11T19:07:49.746179+0000","last_became_peered":"2024-03-11T19:07:49.746179+0000","last_unstale":"2024-03-11T19:09:13.620826+0000","last_undegraded":"2024-03-11T19:09:13.620826+0000","last_fullsized":"2024-03-11T19:09:13.620826+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:06:08.144850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.989738+0000","last_change":"2024-03-11T19:07:48.740926+0000","last_active":"2024-03-11T19:08:44.989738+0000","last_peered":"2024-03-11T19:08:44.989738+0000","last_clean":"2024-03-11T19:08:44.989738+0000","last_became_active":"2024-03-11T19:07:48.740625+0000","last_became_peered":"2024-03-11T19:07:48.740625+0000","last_unstale":"2024-03-11T19:08:44.989738+0000","last_undegraded":"2024-03-11T19:08:44.989738+0000","last_fullsized":"2024-03-11T19:08:44.989738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:50:13.079608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.647609+0000","last_change":"2024-03-11T19:07:49.742428+0000","last_active":"2024-03-11T19:09:13.647609+0000","last_peered":"2024-03-11T19:09:13.647609+0000","last_clean":"2024-03-11T19:09:13.647609+0000","last_became_active":"2024-03-11T19:07:49.742098+0000","last_became_peered":"2024-03-11T19:07:49.742098+0000","last_unstale":"2024-03-11T19:09:13.647609+0000","last_undegraded":"2024-03-11T19:09:13.647609+0000","last_fullsized":"2024-03-11T19:09:13.647609+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:19:24.615755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'5","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:53.777258+0000","last_change":"2024-03-11T19:08:45.002189+0000","last_active":"2024-03-11T19:09:53.777258+0000","last_peered":"2024-03-11T19:09:53.777258+0000","last_clean":"2024-03-11T19:09:53.777258+0000","last_became_active":"2024-03-11T19:08:44.992818+0000","last_became_peered":"2024-03-11T19:08:44.992818+0000","last_unstale":"2024-03-11T19:09:53.777258+0000","last_undegraded":"2024-03-11T19:09:53.777258+0000","last_fullsized":"2024-03-11T19:09:53.777258+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:34:54.461734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.643596+0000","last_change":"2024-03-11T19:07:49.754872+0000","last_active":"2024-03-11T19:09:13.643596+0000","last_peered":"2024-03-11T19:09:13.643596+0000","last_clean":"2024-03-11T19:09:13.643596+0000","last_became_active":"2024-03-11T19:07:49.749663+0000","last_became_peered":"2024-03-11T19:07:49.749663+0000","last_unstale":"2024-03-11T19:09:13.643596+0000","last_undegraded":"2024-03-11T19:09:13.643596+0000","last_fullsized":"2024-03-11T19:09:13.643596+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:22:23.664814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.990526+0000","last_change":"2024-03-11T19:07:48.742227+0000","last_active":"2024-03-11T19:08:44.990526+0000","last_peered":"2024-03-11T19:08:44.990526+0000","last_clean":"2024-03-11T19:08:44.990526+0000","last_became_active":"2024-03-11T19:07:48.741912+0000","last_became_peered":"2024-03-11T19:07:48.741912+0000","last_unstale":"2024-03-11T19:08:44.990526+0000","last_undegraded":"2024-03-11T19:08:44.990526+0000","last_fullsized":"2024-03-11T19:08:44.990526+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:10:06.120433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.657986+0000","last_change":"2024-03-11T19:07:49.748947+0000","last_active":"2024-03-11T19:09:13.657986+0000","last_peered":"2024-03-11T19:09:13.657986+0000","last_clean":"2024-03-11T19:09:13.657986+0000","last_became_active":"2024-03-11T19:07:49.748621+0000","last_became_peered":"2024-03-11T19:07:49.748621+0000","last_unstale":"2024-03-11T19:09:13.657986+0000","last_undegraded":"2024-03-11T19:09:13.657986+0000","last_fullsized":"2024-03-11T19:09:13.657986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:19:53.824590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'6","reported_seq":150,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:56.834864+0000","last_change":"2024-03-11T19:07:48.748799+0000","last_active":"2024-03-11T19:09:56.834864+0000","last_peered":"2024-03-11T19:09:56.834864+0000","last_clean":"2024-03-11T19:09:56.834864+0000","last_became_active":"2024-03-11T19:07:48.748434+0000","last_became_peered":"2024-03-11T19:07:48.748434+0000","last_unstale":"2024-03-11T19:09:56.834864+0000","last_undegraded":"2024-03-11T19:09:56.834864+0000","last_fullsized":"2024-03-11T19:09:56.834864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:10:09.836926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":125,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.657255+0000","last_change":"2024-03-11T19:07:49.750122+0000","last_active":"2024-03-11T19:09:13.657255+0000","last_peered":"2024-03-11T19:09:13.657255+0000","last_clean":"2024-03-11T19:09:13.657255+0000","last_became_active":"2024-03-11T19:07:49.749801+0000","last_became_peered":"2024-03-11T19:07:49.749801+0000","last_unstale":"2024-03-11T19:09:13.657255+0000","last_undegraded":"2024-03-11T19:09:13.657255+0000","last_fullsized":"2024-03-11T19:09:13.657255+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:58:31.155275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:45.036439+0000","last_change":"2024-03-11T19:07:48.741959+0000","last_active":"2024-03-11T19:08:45.036439+0000","last_peered":"2024-03-11T19:08:45.036439+0000","last_clean":"2024-03-11T19:08:45.036439+0000","last_became_active":"2024-03-11T19:07:48.741630+0000","last_became_peered":"2024-03-11T19:07:48.741630+0000","last_unstale":"2024-03-11T19:08:45.036439+0000","last_undegraded":"2024-03-11T19:08:45.036439+0000","last_fullsized":"2024-03-11T19:08:45.036439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:27:08.606055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.652383+0000","last_change":"2024-03-11T19:07:49.729838+0000","last_active":"2024-03-11T19:09:13.652383+0000","last_peered":"2024-03-11T19:09:13.652383+0000","last_clean":"2024-03-11T19:09:13.652383+0000","last_became_active":"2024-03-11T19:07:49.728912+0000","last_became_peered":"2024-03-11T19:07:49.728912+0000","last_unstale":"2024-03-11T19:09:13.652383+0000","last_undegraded":"2024-03-11T19:09:13.652383+0000","last_fullsized":"2024-03-11T19:09:13.652383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:02:06.551734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:08:44.989032+0000","last_change":"2024-03-11T19:07:48.744709+0000","last_active":"2024-03-11T19:08:44.989032+0000","last_peered":"2024-03-11T19:08:44.989032+0000","last_clean":"2024-03-11T19:08:44.989032+0000","last_became_active":"2024-03-11T19:07:48.744356+0000","last_became_peered":"2024-03-11T19:07:48.744356+0000","last_unstale":"2024-03-11T19:08:44.989032+0000","last_undegraded":"2024-03-11T19:08:44.989032+0000","last_fullsized":"2024-03-11T19:08:44.989032+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:47.711711+0000","last_clean_scrub_stamp":"2024-03-11T19:07:47.711711+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:48:47.098443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.652239+0000","last_change":"2024-03-11T19:07:49.743288+0000","last_active":"2024-03-11T19:09:13.652239+0000","last_peered":"2024-03-11T19:09:13.652239+0000","last_clean":"2024-03-11T19:09:13.652239+0000","last_became_active":"2024-03-11T19:07:49.742942+0000","last_became_peered":"2024-03-11T19:07:49.742942+0000","last_unstale":"2024-03-11T19:09:13.652239+0000","last_undegraded":"2024-03-11T19:09:13.652239+0000","last_fullsized":"2024-03-11T19:09:13.652239+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:21:53.896192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.645073+0000","last_change":"2024-03-11T19:07:49.741610+0000","last_active":"2024-03-11T19:09:13.645073+0000","last_peered":"2024-03-11T19:09:13.645073+0000","last_clean":"2024-03-11T19:09:13.645073+0000","last_became_active":"2024-03-11T19:07:49.741234+0000","last_became_peered":"2024-03-11T19:07:49.741234+0000","last_unstale":"2024-03-11T19:09:13.645073+0000","last_undegraded":"2024-03-11T19:09:13.645073+0000","last_fullsized":"2024-03-11T19:09:13.645073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:53:03.320093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.638525+0000","last_change":"2024-03-11T19:08:45.037715+0000","last_active":"2024-03-11T19:09:13.638525+0000","last_peered":"2024-03-11T19:09:13.638525+0000","last_clean":"2024-03-11T19:09:13.638525+0000","last_became_active":"2024-03-11T19:08:45.037299+0000","last_became_peered":"2024-03-11T19:08:45.037299+0000","last_unstale":"2024-03-11T19:09:13.638525+0000","last_undegraded":"2024-03-11T19:09:13.638525+0000","last_fullsized":"2024-03-11T19:09:13.638525+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:01:49.025379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.645173+0000","last_change":"2024-03-11T19:07:49.745928+0000","last_active":"2024-03-11T19:09:13.645173+0000","last_peered":"2024-03-11T19:09:13.645173+0000","last_clean":"2024-03-11T19:09:13.645173+0000","last_became_active":"2024-03-11T19:07:49.745025+0000","last_became_peered":"2024-03-11T19:07:49.745025+0000","last_unstale":"2024-03-11T19:09:13.645173+0000","last_undegraded":"2024-03-11T19:09:13.645173+0000","last_fullsized":"2024-03-11T19:09:13.645173+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:40:30.141906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.648364+0000","last_change":"2024-03-11T19:07:49.756274+0000","last_active":"2024-03-11T19:09:13.648364+0000","last_peered":"2024-03-11T19:09:13.648364+0000","last_clean":"2024-03-11T19:09:13.648364+0000","last_became_active":"2024-03-11T19:07:49.755954+0000","last_became_peered":"2024-03-11T19:07:49.755954+0000","last_unstale":"2024-03-11T19:09:13.648364+0000","last_undegraded":"2024-03-11T19:09:13.648364+0000","last_fullsized":"2024-03-11T19:09:13.648364+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:15:36.825240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.639818+0000","last_change":"2024-03-11T19:08:44.992154+0000","last_active":"2024-03-11T19:09:13.639818+0000","last_peered":"2024-03-11T19:09:13.639818+0000","last_clean":"2024-03-11T19:09:13.639818+0000","last_became_active":"2024-03-11T19:08:44.991772+0000","last_became_peered":"2024-03-11T19:08:44.991772+0000","last_unstale":"2024-03-11T19:09:13.639818+0000","last_undegraded":"2024-03-11T19:09:13.639818+0000","last_fullsized":"2024-03-11T19:09:13.639818+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:10:33.882095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.641105+0000","last_change":"2024-03-11T19:07:49.743963+0000","last_active":"2024-03-11T19:09:13.641105+0000","last_peered":"2024-03-11T19:09:13.641105+0000","last_clean":"2024-03-11T19:09:13.641105+0000","last_became_active":"2024-03-11T19:07:49.743568+0000","last_became_peered":"2024-03-11T19:07:49.743568+0000","last_unstale":"2024-03-11T19:09:13.641105+0000","last_undegraded":"2024-03-11T19:09:13.641105+0000","last_fullsized":"2024-03-11T19:09:13.641105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:45:28.970286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.654180+0000","last_change":"2024-03-11T19:07:49.750677+0000","last_active":"2024-03-11T19:09:13.654180+0000","last_peered":"2024-03-11T19:09:13.654180+0000","last_clean":"2024-03-11T19:09:13.654180+0000","last_became_active":"2024-03-11T19:07:49.750373+0000","last_became_peered":"2024-03-11T19:07:49.750373+0000","last_unstale":"2024-03-11T19:09:13.654180+0000","last_undegraded":"2024-03-11T19:09:13.654180+0000","last_fullsized":"2024-03-11T19:09:13.654180+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:07:28.767502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.645768+0000","last_change":"2024-03-11T19:07:49.745278+0000","last_active":"2024-03-11T19:09:13.645768+0000","last_peered":"2024-03-11T19:09:13.645768+0000","last_clean":"2024-03-11T19:09:13.645768+0000","last_became_active":"2024-03-11T19:07:49.744998+0000","last_became_peered":"2024-03-11T19:07:49.744998+0000","last_unstale":"2024-03-11T19:09:13.645768+0000","last_undegraded":"2024-03-11T19:09:13.645768+0000","last_fullsized":"2024-03-11T19:09:13.645768+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:23.778051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.647869+0000","last_change":"2024-03-11T19:07:49.734318+0000","last_active":"2024-03-11T19:09:13.647869+0000","last_peered":"2024-03-11T19:09:13.647869+0000","last_clean":"2024-03-11T19:09:13.647869+0000","last_became_active":"2024-03-11T19:07:49.733998+0000","last_became_peered":"2024-03-11T19:07:49.733998+0000","last_unstale":"2024-03-11T19:09:13.647869+0000","last_undegraded":"2024-03-11T19:09:13.647869+0000","last_fullsized":"2024-03-11T19:09:13.647869+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:48:50.308499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.646683+0000","last_change":"2024-03-11T19:07:49.741009+0000","last_active":"2024-03-11T19:09:13.646683+0000","last_peered":"2024-03-11T19:09:13.646683+0000","last_clean":"2024-03-11T19:09:13.646683+0000","last_became_active":"2024-03-11T19:07:49.740094+0000","last_became_peered":"2024-03-11T19:07:49.740094+0000","last_unstale":"2024-03-11T19:09:13.646683+0000","last_undegraded":"2024-03-11T19:09:13.646683+0000","last_fullsized":"2024-03-11T19:09:13.646683+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:06:31.945387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.657556+0000","last_change":"2024-03-11T19:07:49.759300+0000","last_active":"2024-03-11T19:09:13.657556+0000","last_peered":"2024-03-11T19:09:13.657556+0000","last_clean":"2024-03-11T19:09:13.657556+0000","last_became_active":"2024-03-11T19:07:49.758011+0000","last_became_peered":"2024-03-11T19:07:49.758011+0000","last_unstale":"2024-03-11T19:09:13.657556+0000","last_undegraded":"2024-03-11T19:09:13.657556+0000","last_fullsized":"2024-03-11T19:09:13.657556+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:03:40.651816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.652991+0000","last_change":"2024-03-11T19:07:49.752905+0000","last_active":"2024-03-11T19:09:13.652991+0000","last_peered":"2024-03-11T19:09:13.652991+0000","last_clean":"2024-03-11T19:09:13.652991+0000","last_became_active":"2024-03-11T19:07:49.752383+0000","last_became_peered":"2024-03-11T19:07:49.752383+0000","last_unstale":"2024-03-11T19:09:13.652991+0000","last_undegraded":"2024-03-11T19:09:13.652991+0000","last_fullsized":"2024-03-11T19:09:13.652991+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:45:33.475526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:09:13.637754+0000","last_change":"2024-03-11T19:07:49.744957+0000","last_active":"2024-03-11T19:09:13.637754+0000","last_peered":"2024-03-11T19:09:13.637754+0000","last_clean":"2024-03-11T19:09:13.637754+0000","last_became_active":"2024-03-11T19:07:49.744627+0000","last_became_peered":"2024-03-11T19:07:49.744627+0000","last_unstale":"2024-03-11T19:09:13.637754+0000","last_undegraded":"2024-03-11T19:09:13.637754+0000","last_fullsized":"2024-03-11T19:09:13.637754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:07:48.717860+0000","last_clean_scrub_stamp":"2024-03-11T19:07:48.717860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:27:47.817004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245707,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":75419,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1258,"num_read_kb":711,"num_write":266,"num_write_kb":159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":569344,"data_stored":221943,"data_compressed":5565,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318312,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":464775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579948,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":1122185,"data_compressed":37658,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743474,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":464100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005111,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":464366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266748,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":1058616,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430274,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":464701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":527011,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":465649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1064960,"data_stored":532961,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149776,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":463642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888140,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":464171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691911,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":1057308,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34165,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35803,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":64033,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":69983,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":68722,"data_compressed":1855,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T19:10:00.515 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:10:00.515 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:00 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1443096501' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:10:00.515 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:00 smithi190 ceph-mon[36292]: overall HEALTH_OK 2024-03-11T19:10:00.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:00 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1443096501' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:10:00.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:00 smithi170 ceph-mon[35690]: overall HEALTH_OK 2024-03-11T19:10:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.650+0000 7fd7ae223640 1 -- 172.21.15.29:0/3816266020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7a81003f0 msgr2=0x7fd7a8100850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.650+0000 7fd7ae223640 1 --2- 172.21.15.29:0/3816266020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7a81003f0 0x7fd7a8100850 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd79c009900 tx=0x7fd79c02bcd0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.651+0000 7fd7ae223640 1 -- 172.21.15.29:0/3816266020 shutdown_connections 2024-03-11T19:10:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.651+0000 7fd7ae223640 1 --2- 172.21.15.29:0/3816266020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7a8100d90 0x7fd7a8101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.651+0000 7fd7ae223640 1 --2- 172.21.15.29:0/3816266020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7a81003f0 0x7fd7a8100850 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.651+0000 7fd7ae223640 1 --2- 172.21.15.29:0/3816266020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a80ffa90 0x7fd7a80ffeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.651+0000 7fd7ae223640 1 -- 172.21.15.29:0/3816266020 >> 172.21.15.29:0/3816266020 conn(0x7fd7a80fb610 msgr2=0x7fd7a80fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.651+0000 7fd7ae223640 1 -- 172.21.15.29:0/3816266020 shutdown_connections 2024-03-11T19:10:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.651+0000 7fd7ae223640 1 -- 172.21.15.29:0/3816266020 wait complete. 2024-03-11T19:10:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.651+0000 7fd7ae223640 1 Processor -- start 2024-03-11T19:10:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.651+0000 7fd7ae223640 1 -- start start 2024-03-11T19:10:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.651+0000 7fd7ae223640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7a80ffa90 0x7fd7a8199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.651+0000 7fd7ae223640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a81003f0 0x7fd7a81998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.651+0000 7fd7ae223640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7a8100d90 0x7fd7a81a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.651+0000 7fd7ae223640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a81a0e60 con 0x7fd7a80ffa90 2024-03-11T19:10:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.651+0000 7fd7ae223640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a81a0fd0 con 0x7fd7a8100d90 2024-03-11T19:10:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.651+0000 7fd7ae223640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a81a1190 con 0x7fd7a81003f0 2024-03-11T19:10:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.652+0000 7fd7a77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7a80ffa90 0x7fd7a8199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.652+0000 7fd7a77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7a80ffa90 0x7fd7a8199370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54536/0 (socket says 172.21.15.29:54536) 2024-03-11T19:10:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.652+0000 7fd7a77fe640 1 -- 172.21.15.29:0/2866503842 learned_addr learned my addr 172.21.15.29:0/2866503842 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.652+0000 7fd7a6ffd640 1 --2- 172.21.15.29:0/2866503842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a81003f0 0x7fd7a81998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.652+0000 7fd7a7fff640 1 --2- 172.21.15.29:0/2866503842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7a8100d90 0x7fd7a81a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.652+0000 7fd7a77fe640 1 -- 172.21.15.29:0/2866503842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a81003f0 msgr2=0x7fd7a81998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.652+0000 7fd7a77fe640 1 --2- 172.21.15.29:0/2866503842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a81003f0 0x7fd7a81998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.652+0000 7fd7a77fe640 1 -- 172.21.15.29:0/2866503842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7a8100d90 msgr2=0x7fd7a81a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.652+0000 7fd7a77fe640 1 --2- 172.21.15.29:0/2866503842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7a8100d90 0x7fd7a81a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.652+0000 7fd7a77fe640 1 -- 172.21.15.29:0/2866503842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79c009560 con 0x7fd7a80ffa90 2024-03-11T19:10:00.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.652+0000 7fd7a77fe640 1 --2- 172.21.15.29:0/2866503842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7a80ffa90 0x7fd7a8199370 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd79800d950 tx=0x7fd79800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:00.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.652+0000 7fd7a4ff9640 1 -- 172.21.15.29:0/2866503842 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd798009a30 con 0x7fd7a80ffa90 2024-03-11T19:10:00.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.652+0000 7fd7a4ff9640 1 -- 172.21.15.29:0/2866503842 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd7980044d0 con 0x7fd7a80ffa90 2024-03-11T19:10:00.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.652+0000 7fd7ae223640 1 -- 172.21.15.29:0/2866503842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a81a1470 con 0x7fd7a80ffa90 2024-03-11T19:10:00.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.652+0000 7fd7a4ff9640 1 -- 172.21.15.29:0/2866503842 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7980047a0 con 0x7fd7a80ffa90 2024-03-11T19:10:00.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.652+0000 7fd7ae223640 1 -- 172.21.15.29:0/2866503842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a81a1950 con 0x7fd7a80ffa90 2024-03-11T19:10:00.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.653+0000 7fd7ae223640 1 -- 172.21.15.29:0/2866503842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a810aa60 con 0x7fd7a80ffa90 2024-03-11T19:10:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.654+0000 7fd7a4ff9640 1 -- 172.21.15.29:0/2866503842 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fd798009b90 con 0x7fd7a80ffa90 2024-03-11T19:10:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.655+0000 7fd7a4ff9640 1 --2- 172.21.15.29:0/2866503842 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd7900a1ac0 0x7fd7900a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.655+0000 7fd7a6ffd640 1 --2- 172.21.15.29:0/2866503842 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd7900a1ac0 0x7fd7900a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.655+0000 7fd7a4ff9640 1 -- 172.21.15.29:0/2866503842 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd7980cd020 con 0x7fd7a80ffa90 2024-03-11T19:10:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.655+0000 7fd7a6ffd640 1 --2- 172.21.15.29:0/2866503842 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd7900a1ac0 0x7fd7900a3f80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd79c009230 tx=0x7fd79c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:00.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.659+0000 7fd7a4ff9640 1 -- 172.21.15.29:0/2866503842 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd7980960d0 con 0x7fd7a80ffa90 2024-03-11T19:10:00.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.839+0000 7fd7ae223640 1 -- 172.21.15.29:0/2866503842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7a810ec30 con 0x7fd7a80ffa90 2024-03-11T19:10:00.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.841+0000 7fd7a4ff9640 1 -- 172.21.15.29:0/2866503842 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22559 (secure 0 0 0) 0x7fd798017090 con 0x7fd7a80ffa90 2024-03-11T19:10:00.844 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:10:00.844 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":89,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:08:44.981806+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:07:47.563184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:07:48.282461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:10:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.843+0000 7fd7ae223640 1 -- 172.21.15.29:0/2866503842 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd7900a1ac0 msgr2=0x7fd7900a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.843+0000 7fd7ae223640 1 --2- 172.21.15.29:0/2866503842 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd7900a1ac0 0x7fd7900a3f80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd79c009230 tx=0x7fd79c005fd0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.843+0000 7fd7ae223640 1 -- 172.21.15.29:0/2866503842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7a80ffa90 msgr2=0x7fd7a8199370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.843+0000 7fd7ae223640 1 --2- 172.21.15.29:0/2866503842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7a80ffa90 0x7fd7a8199370 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd79800d950 tx=0x7fd79800de10 comp rx=0 tx=0).stop 2024-03-11T19:10:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.843+0000 7fd7ae223640 1 -- 172.21.15.29:0/2866503842 shutdown_connections 2024-03-11T19:10:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.843+0000 7fd7ae223640 1 --2- 172.21.15.29:0/2866503842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7a8100d90 0x7fd7a81a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.843+0000 7fd7ae223640 1 --2- 172.21.15.29:0/2866503842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a81003f0 0x7fd7a81998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.843+0000 7fd7ae223640 1 --2- 172.21.15.29:0/2866503842 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd7900a1ac0 0x7fd7900a3f80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.843+0000 7fd7ae223640 1 --2- 172.21.15.29:0/2866503842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7a80ffa90 0x7fd7a8199370 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.843+0000 7fd7ae223640 1 -- 172.21.15.29:0/2866503842 >> 172.21.15.29:0/2866503842 conn(0x7fd7a80fb610 msgr2=0x7fd7a80fcdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:00.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.843+0000 7fd7ae223640 1 -- 172.21.15.29:0/2866503842 shutdown_connections 2024-03-11T19:10:00.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.843+0000 7fd7ae223640 1 -- 172.21.15.29:0/2866503842 wait complete. 2024-03-11T19:10:00.857 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:03:09.860019+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T19:07:47.563184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T19:07:48.282461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T19:10:00.858 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:10:00.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.960+0000 7fb7989c9640 1 -- 172.21.15.29:0/4155548381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb794101270 msgr2=0x7fb7941016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:00.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.960+0000 7fb7989c9640 1 --2- 172.21.15.29:0/4155548381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb794101270 0x7fb7941016f0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb77c009900 tx=0x7fb77c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:10:00.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.960+0000 7fb7989c9640 1 -- 172.21.15.29:0/4155548381 shutdown_connections 2024-03-11T19:10:00.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.960+0000 7fb7989c9640 1 --2- 172.21.15.29:0/4155548381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb794101c30 0x7fb7941084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.960+0000 7fb7989c9640 1 --2- 172.21.15.29:0/4155548381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb794101270 0x7fb7941016f0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.960+0000 7fb7989c9640 1 --2- 172.21.15.29:0/4155548381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb794100080 0x7fb7941004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.960+0000 7fb7989c9640 1 -- 172.21.15.29:0/4155548381 >> 172.21.15.29:0/4155548381 conn(0x7fb7940fb610 msgr2=0x7fb7940fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:00.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.960+0000 7fb7989c9640 1 -- 172.21.15.29:0/4155548381 shutdown_connections 2024-03-11T19:10:00.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.960+0000 7fb7989c9640 1 -- 172.21.15.29:0/4155548381 wait complete. 2024-03-11T19:10:00.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb7989c9640 1 Processor -- start 2024-03-11T19:10:00.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb7989c9640 1 -- start start 2024-03-11T19:10:00.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb7989c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb794100080 0x7fb794199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:00.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb7989c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb794101270 0x7fb794199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:00.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb7989c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb794101c30 0x7fb7941a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:00.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb7989c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7941a0e40 con 0x7fb794101270 2024-03-11T19:10:00.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb7989c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7941a0fb0 con 0x7fb794101c30 2024-03-11T19:10:00.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb7989c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7941a1170 con 0x7fb794100080 2024-03-11T19:10:00.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb78bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb794101270 0x7fb794199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:00.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb78bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb794101270 0x7fb794199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54554/0 (socket says 172.21.15.29:54554) 2024-03-11T19:10:00.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb78bfff640 1 -- 172.21.15.29:0/119695863 learned_addr learned my addr 172.21.15.29:0/119695863 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:00.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb79259c640 1 --2- 172.21.15.29:0/119695863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb794100080 0x7fb794199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:00.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb792d9d640 1 --2- 172.21.15.29:0/119695863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb794101c30 0x7fb7941a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:00.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb78bfff640 1 -- 172.21.15.29:0/119695863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb794100080 msgr2=0x7fb794199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:00.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb78bfff640 1 --2- 172.21.15.29:0/119695863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb794100080 0x7fb794199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb78bfff640 1 -- 172.21.15.29:0/119695863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb794101c30 msgr2=0x7fb7941a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:00.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb78bfff640 1 --2- 172.21.15.29:0/119695863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb794101c30 0x7fb7941a0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:00.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb78bfff640 1 -- 172.21.15.29:0/119695863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb77c009560 con 0x7fb794101270 2024-03-11T19:10:00.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb792d9d640 1 --2- 172.21.15.29:0/119695863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb794101c30 0x7fb7941a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:00.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb78bfff640 1 --2- 172.21.15.29:0/119695863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb794101270 0x7fb794199890 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb77c036040 tx=0x7fb77c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:00.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.961+0000 7fb78b7fe640 1 -- 172.21.15.29:0/119695863 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77c0044e0 con 0x7fb794101270 2024-03-11T19:10:00.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.962+0000 7fb7989c9640 1 -- 172.21.15.29:0/119695863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7941a13f0 con 0x7fb794101270 2024-03-11T19:10:00.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.962+0000 7fb78b7fe640 1 -- 172.21.15.29:0/119695863 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb77c02de90 con 0x7fb794101270 2024-03-11T19:10:00.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.962+0000 7fb78b7fe640 1 -- 172.21.15.29:0/119695863 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77c005470 con 0x7fb794101270 2024-03-11T19:10:00.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.962+0000 7fb7989c9640 1 -- 172.21.15.29:0/119695863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7941a1900 con 0x7fb794101270 2024-03-11T19:10:00.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.962+0000 7fb7989c9640 1 -- 172.21.15.29:0/119695863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb79410b940 con 0x7fb794101270 2024-03-11T19:10:00.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.963+0000 7fb78b7fe640 1 -- 172.21.15.29:0/119695863 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fb77c003800 con 0x7fb794101270 2024-03-11T19:10:00.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.964+0000 7fb78b7fe640 1 --2- 172.21.15.29:0/119695863 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7800a1ac0 0x7fb7800a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:00.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.964+0000 7fb79259c640 1 --2- 172.21.15.29:0/119695863 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7800a1ac0 0x7fb7800a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:00.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.964+0000 7fb78b7fe640 1 -- 172.21.15.29:0/119695863 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb77c0ef7b0 con 0x7fb794101270 2024-03-11T19:10:00.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.965+0000 7fb79259c640 1 --2- 172.21.15.29:0/119695863 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7800a1ac0 0x7fb7800a3f80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb784005fd0 tx=0x7fb784005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:00.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:00.969+0000 7fb78b7fe640 1 -- 172.21.15.29:0/119695863 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb77c0b8970 con 0x7fb794101270 2024-03-11T19:10:01.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.149+0000 7fb7989c9640 1 -- 172.21.15.29:0/119695863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb7941a1b20 con 0x7fb794101270 2024-03-11T19:10:01.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.150+0000 7fb78b7fe640 1 -- 172.21.15.29:0/119695863 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb77c0bc030 con 0x7fb794101270 2024-03-11T19:10:01.158 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-11T19:10:01.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.152+0000 7fb7989c9640 1 -- 172.21.15.29:0/119695863 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7800a1ac0 msgr2=0x7fb7800a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:01.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.152+0000 7fb7989c9640 1 --2- 172.21.15.29:0/119695863 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7800a1ac0 0x7fb7800a3f80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb784005fd0 tx=0x7fb784005850 comp rx=0 tx=0).stop 2024-03-11T19:10:01.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.152+0000 7fb7989c9640 1 -- 172.21.15.29:0/119695863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb794101270 msgr2=0x7fb794199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:01.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.152+0000 7fb7989c9640 1 --2- 172.21.15.29:0/119695863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb794101270 0x7fb794199890 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb77c036040 tx=0x7fb77c0042f0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.152+0000 7fb7989c9640 1 -- 172.21.15.29:0/119695863 shutdown_connections 2024-03-11T19:10:01.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.152+0000 7fb7989c9640 1 --2- 172.21.15.29:0/119695863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb794101c30 0x7fb7941a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.152+0000 7fb7989c9640 1 --2- 172.21.15.29:0/119695863 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7800a1ac0 0x7fb7800a3f80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.152+0000 7fb7989c9640 1 --2- 172.21.15.29:0/119695863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb794101270 0x7fb794199890 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.152+0000 7fb7989c9640 1 --2- 172.21.15.29:0/119695863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb794100080 0x7fb794199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.152+0000 7fb7989c9640 1 -- 172.21.15.29:0/119695863 >> 172.21.15.29:0/119695863 conn(0x7fb7940fb610 msgr2=0x7fb7940682e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:01.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.152+0000 7fb7989c9640 1 -- 172.21.15.29:0/119695863 shutdown_connections 2024-03-11T19:10:01.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.152+0000 7fb7989c9640 1 -- 172.21.15.29:0/119695863 wait complete. 2024-03-11T19:10:01.167 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T19:10:01.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.271+0000 7fe843f6c640 1 -- 172.21.15.29:0/3893584994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe83c104540 msgr2=0x7fe83c106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:01.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.271+0000 7fe843f6c640 1 --2- 172.21.15.29:0/3893584994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe83c104540 0x7fe83c106930 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe82c009900 tx=0x7fe82c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:10:01.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.271+0000 7fe843f6c640 1 -- 172.21.15.29:0/3893584994 shutdown_connections 2024-03-11T19:10:01.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.271+0000 7fe843f6c640 1 --2- 172.21.15.29:0/3893584994 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe83c106e70 0x7fe83c1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.271+0000 7fe843f6c640 1 --2- 172.21.15.29:0/3893584994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe83c104540 0x7fe83c106930 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.271+0000 7fe843f6c640 1 --2- 172.21.15.29:0/3893584994 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe83c101c10 0x7fe83c104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.271+0000 7fe843f6c640 1 -- 172.21.15.29:0/3893584994 >> 172.21.15.29:0/3893584994 conn(0x7fe83c0fb620 msgr2=0x7fe83c0fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:01.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.271+0000 7fe843f6c640 1 -- 172.21.15.29:0/3893584994 shutdown_connections 2024-03-11T19:10:01.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.271+0000 7fe843f6c640 1 -- 172.21.15.29:0/3893584994 wait complete. 2024-03-11T19:10:01.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.272+0000 7fe843f6c640 1 Processor -- start 2024-03-11T19:10:01.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.272+0000 7fe843f6c640 1 -- start start 2024-03-11T19:10:01.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.272+0000 7fe843f6c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe83c101c10 0x7fe83c197190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:01.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.272+0000 7fe843f6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe83c104540 0x7fe83c1976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:01.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.272+0000 7fe843f6c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe83c106e70 0x7fe83c19e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:01.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.272+0000 7fe843f6c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c19ec80 con 0x7fe83c104540 2024-03-11T19:10:01.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.272+0000 7fe843f6c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c19edf0 con 0x7fe83c106e70 2024-03-11T19:10:01.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.272+0000 7fe843f6c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c19efb0 con 0x7fe83c101c10 2024-03-11T19:10:01.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.272+0000 7fe841507640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe83c104540 0x7fe83c1976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.272+0000 7fe841507640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe83c104540 0x7fe83c1976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54572/0 (socket says 172.21.15.29:54572) 2024-03-11T19:10:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.272+0000 7fe841507640 1 -- 172.21.15.29:0/2805992351 learned_addr learned my addr 172.21.15.29:0/2805992351 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.272+0000 7fe841d08640 1 --2- 172.21.15.29:0/2805992351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe83c101c10 0x7fe83c197190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.272+0000 7fe842509640 1 --2- 172.21.15.29:0/2805992351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe83c106e70 0x7fe83c19e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.273+0000 7fe841507640 1 -- 172.21.15.29:0/2805992351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe83c101c10 msgr2=0x7fe83c197190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.273+0000 7fe841507640 1 --2- 172.21.15.29:0/2805992351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe83c101c10 0x7fe83c197190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.273+0000 7fe841507640 1 -- 172.21.15.29:0/2805992351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe83c106e70 msgr2=0x7fe83c19e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.273+0000 7fe841507640 1 --2- 172.21.15.29:0/2805992351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe83c106e70 0x7fe83c19e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.273+0000 7fe841507640 1 -- 172.21.15.29:0/2805992351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82c009560 con 0x7fe83c104540 2024-03-11T19:10:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.273+0000 7fe841507640 1 --2- 172.21.15.29:0/2805992351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe83c104540 0x7fe83c1976d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe82c000c00 tx=0x7fe82c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.273+0000 7fe832ffd640 1 -- 172.21.15.29:0/2805992351 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c0044e0 con 0x7fe83c104540 2024-03-11T19:10:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.273+0000 7fe832ffd640 1 -- 172.21.15.29:0/2805992351 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe82c02de90 con 0x7fe83c104540 2024-03-11T19:10:01.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.273+0000 7fe843f6c640 1 -- 172.21.15.29:0/2805992351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe83c19f230 con 0x7fe83c104540 2024-03-11T19:10:01.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.273+0000 7fe832ffd640 1 -- 172.21.15.29:0/2805992351 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c005410 con 0x7fe83c104540 2024-03-11T19:10:01.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.273+0000 7fe843f6c640 1 -- 172.21.15.29:0/2805992351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe83c19f740 con 0x7fe83c104540 2024-03-11T19:10:01.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.274+0000 7fe843f6c640 1 -- 172.21.15.29:0/2805992351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe83c0fd110 con 0x7fe83c104540 2024-03-11T19:10:01.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.275+0000 7fe832ffd640 1 -- 172.21.15.29:0/2805992351 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fe82c005570 con 0x7fe83c104540 2024-03-11T19:10:01.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.275+0000 7fe832ffd640 1 --2- 172.21.15.29:0/2805992351 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe8280a1ac0 0x7fe8280a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:01.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.276+0000 7fe841d08640 1 --2- 172.21.15.29:0/2805992351 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe8280a1ac0 0x7fe8280a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:01.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.276+0000 7fe832ffd640 1 -- 172.21.15.29:0/2805992351 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe82c0ef680 con 0x7fe83c104540 2024-03-11T19:10:01.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.276+0000 7fe841d08640 1 --2- 172.21.15.29:0/2805992351 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe8280a1ac0 0x7fe8280a3f80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe834005fd0 tx=0x7fe834005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:01.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.280+0000 7fe832ffd640 1 -- 172.21.15.29:0/2805992351 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe82c0b8940 con 0x7fe83c104540 2024-03-11T19:10:01.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.459+0000 7fe843f6c640 1 -- 172.21.15.29:0/2805992351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe83c10cca0 con 0x7fe83c104540 2024-03-11T19:10:01.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.460+0000 7fe832ffd640 1 -- 172.21.15.29:0/2805992351 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe82c0bc000 con 0x7fe83c104540 2024-03-11T19:10:01.463 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-03-11T19:10:01.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.462+0000 7fe843f6c640 1 -- 172.21.15.29:0/2805992351 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe8280a1ac0 msgr2=0x7fe8280a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:01.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.462+0000 7fe843f6c640 1 --2- 172.21.15.29:0/2805992351 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe8280a1ac0 0x7fe8280a3f80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe834005fd0 tx=0x7fe834005850 comp rx=0 tx=0).stop 2024-03-11T19:10:01.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.462+0000 7fe843f6c640 1 -- 172.21.15.29:0/2805992351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe83c104540 msgr2=0x7fe83c1976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:01.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.462+0000 7fe843f6c640 1 --2- 172.21.15.29:0/2805992351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe83c104540 0x7fe83c1976d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe82c000c00 tx=0x7fe82c0042f0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.462+0000 7fe843f6c640 1 -- 172.21.15.29:0/2805992351 shutdown_connections 2024-03-11T19:10:01.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.462+0000 7fe843f6c640 1 --2- 172.21.15.29:0/2805992351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe83c106e70 0x7fe83c19e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.462+0000 7fe843f6c640 1 --2- 172.21.15.29:0/2805992351 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe8280a1ac0 0x7fe8280a3f80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.462+0000 7fe843f6c640 1 --2- 172.21.15.29:0/2805992351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe83c104540 0x7fe83c1976d0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.462+0000 7fe843f6c640 1 --2- 172.21.15.29:0/2805992351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe83c101c10 0x7fe83c197190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.462+0000 7fe843f6c640 1 -- 172.21.15.29:0/2805992351 >> 172.21.15.29:0/2805992351 conn(0x7fe83c0fb620 msgr2=0x7fe83c103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:01.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.462+0000 7fe843f6c640 1 -- 172.21.15.29:0/2805992351 shutdown_connections 2024-03-11T19:10:01.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.462+0000 7fe843f6c640 1 -- 172.21.15.29:0/2805992351 wait complete. 2024-03-11T19:10:01.475 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:01 smithi029 ceph-mon[32439]: pgmap v357: 97 pgs: 97 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-11T19:10:01.475 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:01 smithi029 ceph-mon[32439]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:10:01.475 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:01 smithi029 ceph-mon[32439]: from='client.15414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:10:01.476 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:01 smithi029 ceph-mon[32439]: from='client.15420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:10:01.476 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:01 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2866503842' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:10:01.476 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:01 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/119695863' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:10:01.477 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T19:10:01.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:01 smithi190 ceph-mon[36292]: pgmap v357: 97 pgs: 97 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-11T19:10:01.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:01 smithi190 ceph-mon[36292]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:10:01.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:01 smithi190 ceph-mon[36292]: from='client.15414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:10:01.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:01 smithi190 ceph-mon[36292]: from='client.15420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:10:01.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:01 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2866503842' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:10:01.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:01 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/119695863' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:10:01.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.580+0000 7ff3c5e8a640 1 -- 172.21.15.29:0/1268280156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c0101c20 msgr2=0x7ff3c0104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:01.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.580+0000 7ff3c5e8a640 1 --2- 172.21.15.29:0/1268280156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c0101c20 0x7ff3c0104010 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0009900 tx=0x7ff3b002bc30 comp rx=0 tx=0).stop 2024-03-11T19:10:01.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.581+0000 7ff3c5e8a640 1 -- 172.21.15.29:0/1268280156 shutdown_connections 2024-03-11T19:10:01.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.581+0000 7ff3c5e8a640 1 --2- 172.21.15.29:0/1268280156 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3c0106e80 0x7ff3c01092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.581+0000 7ff3c5e8a640 1 --2- 172.21.15.29:0/1268280156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3c0104550 0x7ff3c0106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.581+0000 7ff3c5e8a640 1 --2- 172.21.15.29:0/1268280156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c0101c20 0x7ff3c0104010 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.581+0000 7ff3c5e8a640 1 -- 172.21.15.29:0/1268280156 >> 172.21.15.29:0/1268280156 conn(0x7ff3c00fb630 msgr2=0x7ff3c00fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:01.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.581+0000 7ff3c5e8a640 1 -- 172.21.15.29:0/1268280156 shutdown_connections 2024-03-11T19:10:01.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.581+0000 7ff3c5e8a640 1 -- 172.21.15.29:0/1268280156 wait complete. 2024-03-11T19:10:01.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.581+0000 7ff3c5e8a640 1 Processor -- start 2024-03-11T19:10:01.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.581+0000 7ff3c5e8a640 1 -- start start 2024-03-11T19:10:01.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.581+0000 7ff3c5e8a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c0101c20 0x7ff3c0197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:01.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.581+0000 7ff3c5e8a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3c0104550 0x7ff3c01976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:01.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.581+0000 7ff3c5e8a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3c0106e80 0x7ff3c019e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:01.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.582+0000 7ff3c5e8a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c019ec70 con 0x7ff3c0101c20 2024-03-11T19:10:01.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.582+0000 7ff3c5e8a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c019ede0 con 0x7ff3c0106e80 2024-03-11T19:10:01.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.582+0000 7ff3c5e8a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c019efa0 con 0x7ff3c0104550 2024-03-11T19:10:01.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.582+0000 7ff3bf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c0101c20 0x7ff3c0197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:01.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.582+0000 7ff3bf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c0101c20 0x7ff3c0197180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54596/0 (socket says 172.21.15.29:54596) 2024-03-11T19:10:01.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.582+0000 7ff3bf7fe640 1 -- 172.21.15.29:0/3476152718 learned_addr learned my addr 172.21.15.29:0/3476152718 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:01.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.582+0000 7ff3bffff640 1 --2- 172.21.15.29:0/3476152718 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3c0106e80 0x7ff3c019e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:01.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.582+0000 7ff3beffd640 1 --2- 172.21.15.29:0/3476152718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3c0104550 0x7ff3c01976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:01.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.582+0000 7ff3bf7fe640 1 -- 172.21.15.29:0/3476152718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3c0104550 msgr2=0x7ff3c01976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:01.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.582+0000 7ff3bf7fe640 1 --2- 172.21.15.29:0/3476152718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3c0104550 0x7ff3c01976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.582+0000 7ff3bf7fe640 1 -- 172.21.15.29:0/3476152718 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3c0106e80 msgr2=0x7ff3c019e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:01.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.582+0000 7ff3bf7fe640 1 --2- 172.21.15.29:0/3476152718 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3c0106e80 0x7ff3c019e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.582+0000 7ff3bf7fe640 1 -- 172.21.15.29:0/3476152718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b0009560 con 0x7ff3c0101c20 2024-03-11T19:10:01.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.582+0000 7ff3bf7fe640 1 --2- 172.21.15.29:0/3476152718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c0101c20 0x7ff3c0197180 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0036040 tx=0x7ff3b00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:01.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.582+0000 7ff3bcff9640 1 -- 172.21.15.29:0/3476152718 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b00044e0 con 0x7ff3c0101c20 2024-03-11T19:10:01.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.582+0000 7ff3bcff9640 1 -- 172.21.15.29:0/3476152718 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff3b002de90 con 0x7ff3c0101c20 2024-03-11T19:10:01.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.582+0000 7ff3c5e8a640 1 -- 172.21.15.29:0/3476152718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c019f220 con 0x7ff3c0101c20 2024-03-11T19:10:01.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.582+0000 7ff3bcff9640 1 -- 172.21.15.29:0/3476152718 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0005470 con 0x7ff3c0101c20 2024-03-11T19:10:01.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.582+0000 7ff3c5e8a640 1 -- 172.21.15.29:0/3476152718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c019f730 con 0x7ff3c0101c20 2024-03-11T19:10:01.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.583+0000 7ff3c5e8a640 1 -- 172.21.15.29:0/3476152718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c00fd120 con 0x7ff3c0101c20 2024-03-11T19:10:01.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.584+0000 7ff3bcff9640 1 -- 172.21.15.29:0/3476152718 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7ff3b0003800 con 0x7ff3c0101c20 2024-03-11T19:10:01.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.585+0000 7ff3bcff9640 1 --2- 172.21.15.29:0/3476152718 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff3a80a1ac0 0x7ff3a80a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:01.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.585+0000 7ff3beffd640 1 --2- 172.21.15.29:0/3476152718 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff3a80a1ac0 0x7ff3a80a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:01.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.585+0000 7ff3bcff9640 1 -- 172.21.15.29:0/3476152718 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff3b00ef710 con 0x7ff3c0101c20 2024-03-11T19:10:01.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.586+0000 7ff3beffd640 1 --2- 172.21.15.29:0/3476152718 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff3a80a1ac0 0x7ff3a80a3f80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4005fd0 tx=0x7ff3b4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:01.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.590+0000 7ff3bcff9640 1 -- 172.21.15.29:0/3476152718 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff3b00b88d0 con 0x7ff3c0101c20 2024-03-11T19:10:01.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:01 smithi170 ceph-mon[35690]: pgmap v357: 97 pgs: 97 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-11T19:10:01.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:01 smithi170 ceph-mon[35690]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:10:01.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:01 smithi170 ceph-mon[35690]: from='client.15414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:10:01.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:01 smithi170 ceph-mon[35690]: from='client.15420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:10:01.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:01 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2866503842' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:10:01.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:01 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/119695863' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:10:01.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.771+0000 7ff3c5e8a640 1 -- 172.21.15.29:0/3476152718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff3c002cda0 con 0x7ff3c0101c20 2024-03-11T19:10:01.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.772+0000 7ff3bcff9640 1 -- 172.21.15.29:0/3476152718 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff3b00bbf90 con 0x7ff3c0101c20 2024-03-11T19:10:01.775 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-03-11T19:10:01.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.774+0000 7ff3c5e8a640 1 -- 172.21.15.29:0/3476152718 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff3a80a1ac0 msgr2=0x7ff3a80a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:01.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.774+0000 7ff3c5e8a640 1 --2- 172.21.15.29:0/3476152718 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff3a80a1ac0 0x7ff3a80a3f80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4005fd0 tx=0x7ff3b4005850 comp rx=0 tx=0).stop 2024-03-11T19:10:01.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.774+0000 7ff3c5e8a640 1 -- 172.21.15.29:0/3476152718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c0101c20 msgr2=0x7ff3c0197180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:01.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.774+0000 7ff3c5e8a640 1 --2- 172.21.15.29:0/3476152718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c0101c20 0x7ff3c0197180 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0036040 tx=0x7ff3b00042f0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.774+0000 7ff3c5e8a640 1 -- 172.21.15.29:0/3476152718 shutdown_connections 2024-03-11T19:10:01.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.774+0000 7ff3c5e8a640 1 --2- 172.21.15.29:0/3476152718 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3c0106e80 0x7ff3c019e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.774+0000 7ff3c5e8a640 1 --2- 172.21.15.29:0/3476152718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3c0104550 0x7ff3c01976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.774+0000 7ff3c5e8a640 1 --2- 172.21.15.29:0/3476152718 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff3a80a1ac0 0x7ff3a80a3f80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.774+0000 7ff3c5e8a640 1 --2- 172.21.15.29:0/3476152718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c0101c20 0x7ff3c0197180 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.774+0000 7ff3c5e8a640 1 -- 172.21.15.29:0/3476152718 >> 172.21.15.29:0/3476152718 conn(0x7ff3c00fb630 msgr2=0x7ff3c01026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:01.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.774+0000 7ff3c5e8a640 1 -- 172.21.15.29:0/3476152718 shutdown_connections 2024-03-11T19:10:01.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.774+0000 7ff3c5e8a640 1 -- 172.21.15.29:0/3476152718 wait complete. 2024-03-11T19:10:01.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:10:01.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.888+0000 7f70f5bca640 1 -- 172.21.15.29:0/889358787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f0102390 msgr2=0x7f70f0102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:01.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.888+0000 7f70f5bca640 1 --2- 172.21.15.29:0/889358787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f0102390 0x7f70f0102810 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f70d8009900 tx=0x7f70d802bc30 comp rx=0 tx=0).stop 2024-03-11T19:10:01.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.888+0000 7f70f5bca640 1 -- 172.21.15.29:0/889358787 shutdown_connections 2024-03-11T19:10:01.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.888+0000 7f70f5bca640 1 --2- 172.21.15.29:0/889358787 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70f0102d50 0x7f70f01095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.888+0000 7f70f5bca640 1 --2- 172.21.15.29:0/889358787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f0102390 0x7f70f0102810 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.888+0000 7f70f5bca640 1 --2- 172.21.15.29:0/889358787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70f01011a0 0x7f70f01015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.888+0000 7f70f5bca640 1 -- 172.21.15.29:0/889358787 >> 172.21.15.29:0/889358787 conn(0x7f70f00fc730 msgr2=0x7f70f00feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.888+0000 7f70f5bca640 1 -- 172.21.15.29:0/889358787 shutdown_connections 2024-03-11T19:10:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.888+0000 7f70f5bca640 1 -- 172.21.15.29:0/889358787 wait complete. 2024-03-11T19:10:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.889+0000 7f70f5bca640 1 Processor -- start 2024-03-11T19:10:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.889+0000 7f70f5bca640 1 -- start start 2024-03-11T19:10:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.889+0000 7f70f5bca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f01011a0 0x7f70f0197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.889+0000 7f70f5bca640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70f0102390 0x7f70f0197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.889+0000 7f70f5bca640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70f0102d50 0x7f70f019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.889+0000 7f70f5bca640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f019ec20 con 0x7f70f01011a0 2024-03-11T19:10:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.889+0000 7f70f5bca640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f019ed90 con 0x7f70f0102390 2024-03-11T19:10:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.889+0000 7f70f5bca640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f019ef50 con 0x7f70f0102d50 2024-03-11T19:10:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.889+0000 7f70ef7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f01011a0 0x7f70f0197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:01.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.889+0000 7f70ef7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f01011a0 0x7f70f0197130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54612/0 (socket says 172.21.15.29:54612) 2024-03-11T19:10:01.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.889+0000 7f70ef7fe640 1 -- 172.21.15.29:0/3687888025 learned_addr learned my addr 172.21.15.29:0/3687888025 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:01.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.889+0000 7f70effff640 1 --2- 172.21.15.29:0/3687888025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70f0102d50 0x7f70f019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:01.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.889+0000 7f70eeffd640 1 --2- 172.21.15.29:0/3687888025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70f0102390 0x7f70f0197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:01.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.889+0000 7f70ef7fe640 1 -- 172.21.15.29:0/3687888025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70f0102d50 msgr2=0x7f70f019e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:01.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.889+0000 7f70ef7fe640 1 --2- 172.21.15.29:0/3687888025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70f0102d50 0x7f70f019e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.889+0000 7f70ef7fe640 1 -- 172.21.15.29:0/3687888025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70f0102390 msgr2=0x7f70f0197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:01.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.889+0000 7f70ef7fe640 1 --2- 172.21.15.29:0/3687888025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70f0102390 0x7f70f0197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:01.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.889+0000 7f70ef7fe640 1 -- 172.21.15.29:0/3687888025 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d8009560 con 0x7f70f01011a0 2024-03-11T19:10:01.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.889+0000 7f70eeffd640 1 --2- 172.21.15.29:0/3687888025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70f0102390 0x7f70f0197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:01.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.890+0000 7f70ef7fe640 1 --2- 172.21.15.29:0/3687888025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f01011a0 0x7f70f0197130 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f70e000d950 tx=0x7f70e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:01.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.890+0000 7f70ecff9640 1 -- 172.21.15.29:0/3687888025 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e0009c30 con 0x7f70f01011a0 2024-03-11T19:10:01.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.890+0000 7f70f5bca640 1 -- 172.21.15.29:0/3687888025 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f019f230 con 0x7f70f01011a0 2024-03-11T19:10:01.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.890+0000 7f70ecff9640 1 -- 172.21.15.29:0/3687888025 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f70e0009d90 con 0x7f70f01011a0 2024-03-11T19:10:01.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.890+0000 7f70ecff9640 1 -- 172.21.15.29:0/3687888025 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e000f7e0 con 0x7f70f01011a0 2024-03-11T19:10:01.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.890+0000 7f70f5bca640 1 -- 172.21.15.29:0/3687888025 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f019f790 con 0x7f70f01011a0 2024-03-11T19:10:01.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.891+0000 7f70f5bca640 1 -- 172.21.15.29:0/3687888025 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f0108510 con 0x7f70f01011a0 2024-03-11T19:10:01.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.892+0000 7f70ecff9640 1 -- 172.21.15.29:0/3687888025 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f70e0010460 con 0x7f70f01011a0 2024-03-11T19:10:01.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.892+0000 7f70ecff9640 1 --2- 172.21.15.29:0/3687888025 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f70dc0a1ac0 0x7f70dc0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:01.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.893+0000 7f70eeffd640 1 --2- 172.21.15.29:0/3687888025 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f70dc0a1ac0 0x7f70dc0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:01.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.893+0000 7f70ecff9640 1 -- 172.21.15.29:0/3687888025 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f70e00cd310 con 0x7f70f01011a0 2024-03-11T19:10:01.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.893+0000 7f70eeffd640 1 --2- 172.21.15.29:0/3687888025 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f70dc0a1ac0 0x7f70dc0a3f80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f70d8036040 tx=0x7f70d8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:01.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:01.897+0000 7f70ecff9640 1 -- 172.21.15.29:0/3687888025 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f70e00963c0 con 0x7f70f01011a0 2024-03-11T19:10:02.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.119+0000 7f70f5bca640 1 -- 172.21.15.29:0/3687888025 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70f010ca60 con 0x7f70f01011a0 2024-03-11T19:10:02.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.120+0000 7f70ecff9640 1 -- 172.21.15.29:0/3687888025 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f70e00b1c90 con 0x7f70f01011a0 2024-03-11T19:10:02.123 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:10:02.123 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:10:02.123 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:10:02.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.122+0000 7f70f5bca640 1 -- 172.21.15.29:0/3687888025 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f70dc0a1ac0 msgr2=0x7f70dc0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.122+0000 7f70f5bca640 1 --2- 172.21.15.29:0/3687888025 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f70dc0a1ac0 0x7f70dc0a3f80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f70d8036040 tx=0x7f70d8000bc0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.122+0000 7f70f5bca640 1 -- 172.21.15.29:0/3687888025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f01011a0 msgr2=0x7f70f0197130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.122+0000 7f70f5bca640 1 --2- 172.21.15.29:0/3687888025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f01011a0 0x7f70f0197130 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f70e000d950 tx=0x7f70e000de10 comp rx=0 tx=0).stop 2024-03-11T19:10:02.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.122+0000 7f70f5bca640 1 -- 172.21.15.29:0/3687888025 shutdown_connections 2024-03-11T19:10:02.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.122+0000 7f70f5bca640 1 --2- 172.21.15.29:0/3687888025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70f0102d50 0x7f70f019e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.122+0000 7f70f5bca640 1 --2- 172.21.15.29:0/3687888025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70f0102390 0x7f70f0197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.122+0000 7f70f5bca640 1 --2- 172.21.15.29:0/3687888025 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f70dc0a1ac0 0x7f70dc0a3f80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.122+0000 7f70f5bca640 1 --2- 172.21.15.29:0/3687888025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f01011a0 0x7f70f0197130 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.122+0000 7f70f5bca640 1 -- 172.21.15.29:0/3687888025 >> 172.21.15.29:0/3687888025 conn(0x7f70f00fc730 msgr2=0x7f70f01056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:02.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.122+0000 7f70f5bca640 1 -- 172.21.15.29:0/3687888025 shutdown_connections 2024-03-11T19:10:02.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.122+0000 7f70f5bca640 1 -- 172.21.15.29:0/3687888025 wait complete. 2024-03-11T19:10:02.136 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:10:02.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.237+0000 7f21af8df640 1 -- 172.21.15.29:0/4187578106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21a8102d80 msgr2=0x7f21a8109610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.237+0000 7f21af8df640 1 --2- 172.21.15.29:0/4187578106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21a8102d80 0x7f21a8109610 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f21a0009900 tx=0x7f21a002bc30 comp rx=0 tx=0).stop 2024-03-11T19:10:02.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.237+0000 7f21af8df640 1 -- 172.21.15.29:0/4187578106 shutdown_connections 2024-03-11T19:10:02.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.237+0000 7f21af8df640 1 --2- 172.21.15.29:0/4187578106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21a8102d80 0x7f21a8109610 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.237+0000 7f21af8df640 1 --2- 172.21.15.29:0/4187578106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21a81023c0 0x7f21a8102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.237+0000 7f21af8df640 1 --2- 172.21.15.29:0/4187578106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21a81011d0 0x7f21a81015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.237+0000 7f21af8df640 1 -- 172.21.15.29:0/4187578106 >> 172.21.15.29:0/4187578106 conn(0x7f21a80fc720 msgr2=0x7f21a80feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:02.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.237+0000 7f21af8df640 1 -- 172.21.15.29:0/4187578106 shutdown_connections 2024-03-11T19:10:02.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.238+0000 7f21af8df640 1 -- 172.21.15.29:0/4187578106 wait complete. 2024-03-11T19:10:02.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.238+0000 7f21af8df640 1 Processor -- start 2024-03-11T19:10:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.238+0000 7f21af8df640 1 -- start start 2024-03-11T19:10:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.238+0000 7f21af8df640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21a81011d0 0x7f21a8197110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.238+0000 7f21af8df640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21a81023c0 0x7f21a8197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.238+0000 7f21af8df640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21a8102d80 0x7f21a819e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.238+0000 7f21af8df640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a819ec00 con 0x7f21a81023c0 2024-03-11T19:10:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.238+0000 7f21af8df640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a819ed70 con 0x7f21a81011d0 2024-03-11T19:10:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.238+0000 7f21af8df640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a819ef30 con 0x7f21a8102d80 2024-03-11T19:10:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.238+0000 7f21ace7a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21a81023c0 0x7f21a8197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.238+0000 7f21ace7a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21a81023c0 0x7f21a8197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54628/0 (socket says 172.21.15.29:54628) 2024-03-11T19:10:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.238+0000 7f21ace7a640 1 -- 172.21.15.29:0/4152660295 learned_addr learned my addr 172.21.15.29:0/4152660295 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.238+0000 7f21ad67b640 1 --2- 172.21.15.29:0/4152660295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21a81011d0 0x7f21a8197110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.239+0000 7f21ace7a640 1 -- 172.21.15.29:0/4152660295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21a8102d80 msgr2=0x7f21a819e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.239+0000 7f21ace7a640 1 --2- 172.21.15.29:0/4152660295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21a8102d80 0x7f21a819e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.239+0000 7f21ace7a640 1 -- 172.21.15.29:0/4152660295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21a81011d0 msgr2=0x7f21a8197110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.239+0000 7f21ace7a640 1 --2- 172.21.15.29:0/4152660295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21a81011d0 0x7f21a8197110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.239+0000 7f21ace7a640 1 -- 172.21.15.29:0/4152660295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a0009560 con 0x7f21a81023c0 2024-03-11T19:10:02.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.239+0000 7f21ace7a640 1 --2- 172.21.15.29:0/4152660295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21a81023c0 0x7f21a8197650 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f219800c890 tx=0x7f219800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:02.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.239+0000 7f219e7fc640 1 -- 172.21.15.29:0/4152660295 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2198004220 con 0x7f21a81023c0 2024-03-11T19:10:02.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.239+0000 7f219e7fc640 1 -- 172.21.15.29:0/4152660295 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2198007e60 con 0x7f21a81023c0 2024-03-11T19:10:02.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.239+0000 7f21af8df640 1 -- 172.21.15.29:0/4152660295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a819f210 con 0x7f21a81023c0 2024-03-11T19:10:02.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.239+0000 7f219e7fc640 1 -- 172.21.15.29:0/4152660295 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2198004a00 con 0x7f21a81023c0 2024-03-11T19:10:02.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.239+0000 7f21af8df640 1 -- 172.21.15.29:0/4152660295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a819f770 con 0x7f21a81023c0 2024-03-11T19:10:02.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.240+0000 7f21af8df640 1 -- 172.21.15.29:0/4152660295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a8108570 con 0x7f21a81023c0 2024-03-11T19:10:02.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.242+0000 7f219e7fc640 1 -- 172.21.15.29:0/4152660295 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f21980079b0 con 0x7f21a81023c0 2024-03-11T19:10:02.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.242+0000 7f219e7fc640 1 --2- 172.21.15.29:0/4152660295 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f21940a1b40 0x7f21940a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.242+0000 7f21ad67b640 1 --2- 172.21.15.29:0/4152660295 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f21940a1b40 0x7f21940a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.242+0000 7f219e7fc640 1 -- 172.21.15.29:0/4152660295 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f21980cd4f0 con 0x7f21a81023c0 2024-03-11T19:10:02.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.243+0000 7f21ad67b640 1 --2- 172.21.15.29:0/4152660295 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f21940a1b40 0x7f21940a4000 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f21a8102220 tx=0x7f21a400a590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:02.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.246+0000 7f219e7fc640 1 -- 172.21.15.29:0/4152660295 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2198096720 con 0x7f21a81023c0 2024-03-11T19:10:02.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:02 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2805992351' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:10:02.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:02 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3476152718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:10:02.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:02 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3687888025' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:10:02.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.428+0000 7f21af8df640 1 -- 172.21.15.29:0/4152660295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21a810cbd0 con 0x7f21a81023c0 2024-03-11T19:10:02.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.429+0000 7f219e7fc640 1 -- 172.21.15.29:0/4152660295 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22559 (secure 0 0 0) 0x7f21a810cbd0 con 0x7f21a81023c0 2024-03-11T19:10:02.432 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:10:02.433 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":89,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:08:44.981806+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:07:47.563184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:07:48.282461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:10:02.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.431+0000 7f21af8df640 1 -- 172.21.15.29:0/4152660295 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f21940a1b40 msgr2=0x7f21940a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.431+0000 7f21af8df640 1 --2- 172.21.15.29:0/4152660295 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f21940a1b40 0x7f21940a4000 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f21a8102220 tx=0x7f21a400a590 comp rx=0 tx=0).stop 2024-03-11T19:10:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.431+0000 7f21af8df640 1 -- 172.21.15.29:0/4152660295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21a81023c0 msgr2=0x7f21a8197650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.431+0000 7f21af8df640 1 --2- 172.21.15.29:0/4152660295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21a81023c0 0x7f21a8197650 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f219800c890 tx=0x7f219800cd50 comp rx=0 tx=0).stop 2024-03-11T19:10:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.431+0000 7f21af8df640 1 -- 172.21.15.29:0/4152660295 shutdown_connections 2024-03-11T19:10:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.431+0000 7f21af8df640 1 --2- 172.21.15.29:0/4152660295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21a8102d80 0x7f21a819e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.431+0000 7f21af8df640 1 --2- 172.21.15.29:0/4152660295 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f21940a1b40 0x7f21940a4000 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.431+0000 7f21af8df640 1 --2- 172.21.15.29:0/4152660295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21a81023c0 0x7f21a8197650 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.431+0000 7f21af8df640 1 --2- 172.21.15.29:0/4152660295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21a81011d0 0x7f21a8197110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.431+0000 7f21af8df640 1 -- 172.21.15.29:0/4152660295 >> 172.21.15.29:0/4152660295 conn(0x7f21a80fc720 msgr2=0x7f21a8105740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.431+0000 7f21af8df640 1 -- 172.21.15.29:0/4152660295 shutdown_connections 2024-03-11T19:10:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.431+0000 7f21af8df640 1 -- 172.21.15.29:0/4152660295 wait complete. 2024-03-11T19:10:02.445 DEBUG:tasks.fwd_scrub:Yielding 2024-03-11T19:10:02.446 INFO:teuthology.run_tasks:Running task workunit... 2024-03-11T19:10:02.461 INFO:tasks.workunit:Pulling workunits from ref b6a4997be2b8bb74411268cc42753f29ae9a78ad 2024-03-11T19:10:02.462 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-11T19:10:02.462 INFO:tasks.workunit:timeout=3h 2024-03-11T19:10:02.462 INFO:tasks.workunit:cleanup=True 2024-03-11T19:10:02.463 DEBUG:teuthology.orchestra.run.smithi029:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-11T19:10:02.463 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-11T19:10:02.463 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:10:02.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:10:02.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:10:02.479 INFO:teuthology.orchestra.run.smithi029.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-11T19:10:02.479 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-11T19:10:02.480 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 77h/119d Inode: 1 Links: 2 2024-03-11T19:10:02.480 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:10:02.480 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-11T19:10:02.480 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-11 19:07:56.165765134 +0000 2024-03-11T19:10:02.480 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-11 19:07:56.165765134 +0000 2024-03-11T19:10:02.480 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-11 19:09:55.188654789 +0000 2024-03-11T19:10:02.480 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: - 2024-03-11T19:10:02.480 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-11T19:10:02.480 DEBUG:teuthology.orchestra.run.smithi029:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-11T19:10:02.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:02 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2805992351' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:10:02.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:02 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3476152718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:10:02.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:02 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3687888025' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:10:02.554 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout b6a4997be2b8bb74411268cc42753f29ae9a78ad 2024-03-11T19:10:02.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.570+0000 7f92a29a0640 1 -- 172.21.15.29:0/2387672690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f929c101aa0 msgr2=0x7f929c1083c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.570+0000 7f92a29a0640 1 --2- 172.21.15.29:0/2387672690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f929c101aa0 0x7f929c1083c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9294009ca0 tx=0x7f929402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.571+0000 7f92a29a0640 1 -- 172.21.15.29:0/2387672690 shutdown_connections 2024-03-11T19:10:02.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.571+0000 7f92a29a0640 1 --2- 172.21.15.29:0/2387672690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f929c101aa0 0x7f929c1083c0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.571+0000 7f92a29a0640 1 --2- 172.21.15.29:0/2387672690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f929c1010e0 0x7f929c101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.571+0000 7f92a29a0640 1 --2- 172.21.15.29:0/2387672690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f929c0fff10 0x7f929c100350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.571+0000 7f92a29a0640 1 -- 172.21.15.29:0/2387672690 >> 172.21.15.29:0/2387672690 conn(0x7f929c0fb4f0 msgr2=0x7f929c0fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:02.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.571+0000 7f92a29a0640 1 -- 172.21.15.29:0/2387672690 shutdown_connections 2024-03-11T19:10:02.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.571+0000 7f92a29a0640 1 -- 172.21.15.29:0/2387672690 wait complete. 2024-03-11T19:10:02.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.571+0000 7f92a29a0640 1 Processor -- start 2024-03-11T19:10:02.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.571+0000 7f92a29a0640 1 -- start start 2024-03-11T19:10:02.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.571+0000 7f92a29a0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f929c0fff10 0x7f929c195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.571+0000 7f92a29a0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f929c1010e0 0x7f929c1963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.571+0000 7f92a29a0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f929c101aa0 0x7f929c19d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.571+0000 7f92a29a0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c19d9a0 con 0x7f929c101aa0 2024-03-11T19:10:02.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.571+0000 7f92a29a0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c19db10 con 0x7f929c1010e0 2024-03-11T19:10:02.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.571+0000 7f92a29a0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c19dcd0 con 0x7f929c0fff10 2024-03-11T19:10:02.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.571+0000 7f92a219f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f929c101aa0 0x7f929c19d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.571+0000 7f92a219f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f929c101aa0 0x7f929c19d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54644/0 (socket says 172.21.15.29:54644) 2024-03-11T19:10:02.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.571+0000 7f92a219f640 1 -- 172.21.15.29:0/4106145825 learned_addr learned my addr 172.21.15.29:0/4106145825 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:02.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.572+0000 7f92a199e640 1 --2- 172.21.15.29:0/4106145825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f929c0fff10 0x7f929c195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.572+0000 7f92a219f640 1 -- 172.21.15.29:0/4106145825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f929c0fff10 msgr2=0x7f929c195eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.572+0000 7f92a219f640 1 --2- 172.21.15.29:0/4106145825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f929c0fff10 0x7f929c195eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.572+0000 7f92a219f640 1 -- 172.21.15.29:0/4106145825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f929c1010e0 msgr2=0x7f929c1963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.572+0000 7f92a119d640 1 --2- 172.21.15.29:0/4106145825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f929c1010e0 0x7f929c1963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.572+0000 7f92a219f640 1 --2- 172.21.15.29:0/4106145825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f929c1010e0 0x7f929c1963f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.572+0000 7f92a219f640 1 -- 172.21.15.29:0/4106145825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9294009900 con 0x7f929c101aa0 2024-03-11T19:10:02.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.572+0000 7f92a199e640 1 --2- 172.21.15.29:0/4106145825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f929c0fff10 0x7f929c195eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:02.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.572+0000 7f92a219f640 1 --2- 172.21.15.29:0/4106145825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f929c101aa0 0x7f929c19d460 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f929402c630 tx=0x7f9294034c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:02.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.572+0000 7f9292ffd640 1 -- 172.21.15.29:0/4106145825 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9294003970 con 0x7f929c101aa0 2024-03-11T19:10:02.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.572+0000 7f9292ffd640 1 -- 172.21.15.29:0/4106145825 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9294004040 con 0x7f929c101aa0 2024-03-11T19:10:02.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.572+0000 7f92a29a0640 1 -- 172.21.15.29:0/4106145825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f929c19df50 con 0x7f929c101aa0 2024-03-11T19:10:02.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.572+0000 7f9292ffd640 1 -- 172.21.15.29:0/4106145825 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92940406b0 con 0x7f929c101aa0 2024-03-11T19:10:02.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.572+0000 7f92a29a0640 1 -- 172.21.15.29:0/4106145825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929c19e370 con 0x7f929c101aa0 2024-03-11T19:10:02.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.574+0000 7f9292ffd640 1 -- 172.21.15.29:0/4106145825 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f929400a040 con 0x7f929c101aa0 2024-03-11T19:10:02.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.575+0000 7f92a67d5740 1 Processor -- start 2024-03-11T19:10:02.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.575+0000 7f92a67d5740 1 -- start start 2024-03-11T19:10:02.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.575+0000 7f92a67d5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d2690 0x55f81b2d2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.575+0000 7f92a67d5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d3860 0x55f81b2d3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.575+0000 7f92a67d5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d4220 0x55f81b2daab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.575+0000 7f92a67d5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f81b241f90 con 0x55f81b2d2690 2024-03-11T19:10:02.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.575+0000 7f92a67d5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f81b23f4c0 con 0x55f81b2d4220 2024-03-11T19:10:02.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.575+0000 7f92a67d5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f81b23aca0 con 0x55f81b2d3860 2024-03-11T19:10:02.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.575+0000 7f9292ffd640 1 --2- 172.21.15.29:0/4106145825 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f92880a1ac0 0x7f92880a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.575+0000 7f92a119d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d2690 0x55f81b2d2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.575+0000 7f92a119d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d2690 0x55f81b2d2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54646/0 (socket says 172.21.15.29:54646) 2024-03-11T19:10:02.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.575+0000 7f92a119d640 1 -- 172.21.15.29:0/158663694 learned_addr learned my addr 172.21.15.29:0/158663694 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:02.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.576+0000 7f92a119d640 1 --2- 172.21.15.29:0/158663694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d4220 0x55f81b2daab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.576+0000 7f92a219f640 1 --2- 172.21.15.29:0/4106145825 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f92880a1ac0 0x7f92880a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.576+0000 7f92a199e640 1 --2- 172.21.15.29:0/158663694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d3860 0x55f81b2d3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.576+0000 7f9292ffd640 1 -- 172.21.15.29:0/4106145825 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f92940f0330 con 0x7f929c101aa0 2024-03-11T19:10:02.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.576+0000 7f92a119d640 1 -- 172.21.15.29:0/158663694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d3860 msgr2=0x55f81b2d3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:02.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.576+0000 7f92a119d640 1 --2- 172.21.15.29:0/158663694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d3860 0x55f81b2d3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.576+0000 7f92a119d640 1 -- 172.21.15.29:0/158663694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d4220 msgr2=0x55f81b2daab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:02.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.576+0000 7f92a119d640 1 --2- 172.21.15.29:0/158663694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d4220 0x55f81b2daab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.576+0000 7f92a119d640 1 -- 172.21.15.29:0/158663694 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f81b1fd0b0 con 0x55f81b2d2690 2024-03-11T19:10:02.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.576+0000 7f92a119d640 1 --2- 172.21.15.29:0/158663694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d2690 0x55f81b2d2ab0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f928c0058c0 tx=0x7f928c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1424bda89fd6f43 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:02.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.576+0000 7f92a219f640 1 --2- 172.21.15.29:0/4106145825 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f92880a1ac0 0x7f92880a3f80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9294033000 tx=0x7f9294050000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:02.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.576+0000 7f9287fff640 1 -- 172.21.15.29:0/158663694 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c00e270 con 0x55f81b2d2690 2024-03-11T19:10:02.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.576+0000 7f9287fff640 1 -- 172.21.15.29:0/158663694 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f928c009e90 con 0x55f81b2d2690 2024-03-11T19:10:02.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.576+0000 7f9287fff640 1 -- 172.21.15.29:0/158663694 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c004dd0 con 0x55f81b2d2690 2024-03-11T19:10:02.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.577+0000 7f92a67d5740 1 -- 172.21.15.29:0/158663694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d2690 msgr2=0x55f81b2d2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.577+0000 7f92a67d5740 1 --2- 172.21.15.29:0/158663694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d2690 0x55f81b2d2ab0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f928c0058c0 tx=0x7f928c00e530 comp rx=0 tx=0).stop 2024-03-11T19:10:02.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.577+0000 7f92a67d5740 1 -- 172.21.15.29:0/158663694 shutdown_connections 2024-03-11T19:10:02.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.577+0000 7f92a67d5740 1 --2- 172.21.15.29:0/158663694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d4220 0x55f81b2daab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.577+0000 7f92a67d5740 1 --2- 172.21.15.29:0/158663694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d3860 0x55f81b2d3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.577+0000 7f92a67d5740 1 --2- 172.21.15.29:0/158663694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d2690 0x55f81b2d2ab0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.577+0000 7f92a67d5740 1 -- 172.21.15.29:0/158663694 >> 172.21.15.29:0/158663694 conn(0x55f81b2cd870 msgr2=0x55f81b2cdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:02.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.577+0000 7f92a67d5740 1 -- 172.21.15.29:0/158663694 shutdown_connections 2024-03-11T19:10:02.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.577+0000 7f92a67d5740 1 -- 172.21.15.29:0/158663694 wait complete. 2024-03-11T19:10:02.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.577+0000 7f92a67d5740 1 -- start start 2024-03-11T19:10:02.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.577+0000 7f92a67d5740 1 Processor -- start 2024-03-11T19:10:02.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.578+0000 7f92a67d5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d2690 0x55f81b2dc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.578+0000 7f92a67d5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d3860 0x55f81b2dcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.578+0000 7f92a67d5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 0x55f81b2e3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.578+0000 7f92a67d5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f81b1dff40 con 0x55f81b2d4220 2024-03-11T19:10:02.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.578+0000 7f92a67d5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f81af0e4b0 con 0x55f81b2d2690 2024-03-11T19:10:02.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.578+0000 7f92a67d5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f81b1c0670 con 0x55f81b2d3860 2024-03-11T19:10:02.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.578+0000 7f92a67d5740 10 client.0 _ll_register_callbacks cb 0x55f81b22df70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:02.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.578+0000 7f92a119d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 0x55f81b2e3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.578+0000 7f92a119d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 0x55f81b2e3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54650/0 (socket says 172.21.15.29:54650) 2024-03-11T19:10:02.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.578+0000 7f92a119d640 1 -- 172.21.15.29:0/3967470790 learned_addr learned my addr 172.21.15.29:0/3967470790 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:02.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.578+0000 7f92a119d640 1 --2- 172.21.15.29:0/3967470790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d2690 0x55f81b2dc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.578+0000 7f92a119d640 1 -- 172.21.15.29:0/3967470790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d3860 msgr2=0x55f81b2dcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:02.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.578+0000 7f92a119d640 1 --2- 172.21.15.29:0/3967470790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d3860 0x55f81b2dcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.578+0000 7f92a119d640 1 -- 172.21.15.29:0/3967470790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d2690 msgr2=0x55f81b2dc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:02.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.578+0000 7f92a119d640 1 --2- 172.21.15.29:0/3967470790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d2690 0x55f81b2dc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.578+0000 7f92a119d640 1 -- 172.21.15.29:0/3967470790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f928c009620 con 0x55f81b2d4220 2024-03-11T19:10:02.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.579+0000 7f92a119d640 1 --2- 172.21.15.29:0/3967470790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 0x55f81b2e3df0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f928c009fd0 tx=0x7f928c00ffc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5e000c33328115b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:02.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.579+0000 7f92857fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:02.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.579+0000 7f92857fa640 1 -- 172.21.15.29:0/3967470790 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c00e2d0 con 0x55f81b2d4220 2024-03-11T19:10:02.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.579+0000 7f92857fa640 1 -- 172.21.15.29:0/3967470790 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f928c005250 con 0x55f81b2d4220 2024-03-11T19:10:02.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.579+0000 7f92a67d5740 1 -- 172.21.15.29:0/3967470790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f81b214f40 con 0x55f81b2d4220 2024-03-11T19:10:02.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.579+0000 7f92857fa640 1 -- 172.21.15.29:0/3967470790 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f928c005410 con 0x55f81b2d4220 2024-03-11T19:10:02.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.579+0000 7f92a67d5740 10 client.15468 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:02.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.579+0000 7f92a67d5740 1 -- 172.21.15.29:0/3967470790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55f81b239320 con 0x55f81b2d4220 2024-03-11T19:10:02.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.579+0000 7f92857fa640 1 -- 172.21.15.29:0/3967470790 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f81b214f40 con 0x55f81b2d4220 2024-03-11T19:10:02.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.579+0000 7f92857fa640 1 -- 172.21.15.29:0/3967470790 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f928c00c580 con 0x55f81b2d4220 2024-03-11T19:10:02.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.579+0000 7f92a67d5740 10 client.15468 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:02.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.579+0000 7f92a67d5740 10 client.15468 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:10:02.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.579+0000 7f92a67d5740 20 client.15468 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:02.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.579+0000 7f92a67d5740 1 --2- 172.21.15.29:0/3967470790 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f81b2e8930 0x55f81b2ead50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.579+0000 7f92a67d5740 4 client.15468 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:10:02.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.579+0000 7f92a67d5740 1 -- 172.21.15.29:0/3967470790 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f81b231310 con 0x55f81b2e8930 2024-03-11T19:10:02.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.580+0000 7f92a199e640 1 --2- 172.21.15.29:0/3967470790 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f81b2e8930 0x55f81b2ead50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.581+0000 7f92857fa640 10 client.15468 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:02.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.580+0000 7f92a199e640 1 --2- 172.21.15.29:0/3967470790 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f81b2e8930 0x55f81b2ead50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ed8e6c148c90fe9 server_cookie=66be4921926ef9d1 in_seq=0 out_seq=0 2024-03-11T19:10:02.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.582+0000 7f92857fa640 1 -- 172.21.15.29:0/3967470790 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f81b231310 con 0x55f81b2e8930 2024-03-11T19:10:02.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.582+0000 7f92857fa640 10 client.15468 handle_command_reply: tid=0 2024-03-11T19:10:02.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.582+0000 7f92a67d5740 1 client.15468 shutdown 2024-03-11T19:10:02.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.582+0000 7f92a67d5740 1 -- 172.21.15.29:0/3967470790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 msgr2=0x55f81b2e3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.582+0000 7f92a67d5740 1 --2- 172.21.15.29:0/3967470790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 0x55f81b2e3df0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f928c009fd0 tx=0x7f928c00ffc0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.582+0000 7f92a67d5740 1 -- 172.21.15.29:0/3967470790 shutdown_connections 2024-03-11T19:10:02.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.582+0000 7f92a67d5740 1 --2- 172.21.15.29:0/3967470790 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f81b2e8930 0x55f81b2ead50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.582+0000 7f92a67d5740 1 --2- 172.21.15.29:0/3967470790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 0x55f81b2e3df0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.582+0000 7f92a67d5740 1 --2- 172.21.15.29:0/3967470790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d3860 0x55f81b2dcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.582+0000 7f92a67d5740 1 --2- 172.21.15.29:0/3967470790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d2690 0x55f81b2dc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.582+0000 7f92857fa640 0 client.15468 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:02.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.582+0000 7f92a67d5740 1 -- 172.21.15.29:0/3967470790 >> 172.21.15.29:0/3967470790 conn(0x55f81b2cd870 msgr2=0x55f81b2d0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:02.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.583+0000 7f92a67d5740 1 -- 172.21.15.29:0/3967470790 shutdown_connections 2024-03-11T19:10:02.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.583+0000 7f92a67d5740 1 -- 172.21.15.29:0/3967470790 wait complete. 2024-03-11T19:10:02.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.583+0000 7f92a67d5740 20 client.15468 trim_cache size 0 max 16384 2024-03-11T19:10:02.592 INFO:tasks.workunit.client.0.smithi029.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-11T19:10:02.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:02 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2805992351' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:10:02.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:02 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3476152718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:10:02.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:02 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3687888025' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:10:02.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.599+0000 7f92a67d5740 1 Processor -- start 2024-03-11T19:10:02.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.599+0000 7f92a67d5740 1 -- start start 2024-03-11T19:10:02.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.599+0000 7f92a67d5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d2690 0x55f81b2d7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.599+0000 7f92a67d5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d3860 0x55f81b2d7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.599+0000 7f92a67d5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 0x55f81b2e3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.599+0000 7f92a67d5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f81b2c2350 con 0x55f81b2d4220 2024-03-11T19:10:02.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.599+0000 7f92a67d5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f81b2d3360 con 0x55f81b2d2690 2024-03-11T19:10:02.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.599+0000 7f92a67d5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f81b231310 con 0x55f81b2d3860 2024-03-11T19:10:02.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.599+0000 7f92a119d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 0x55f81b2e3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.599+0000 7f92a119d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 0x55f81b2e3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54660/0 (socket says 172.21.15.29:54660) 2024-03-11T19:10:02.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.599+0000 7f92a119d640 1 -- 172.21.15.29:0/522382225 learned_addr learned my addr 172.21.15.29:0/522382225 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:02.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.599+0000 7f92a199e640 1 --2- 172.21.15.29:0/522382225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d3860 0x55f81b2d7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.599+0000 7f92a119d640 1 --2- 172.21.15.29:0/522382225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d2690 0x55f81b2d7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.599+0000 7f92a199e640 1 -- 172.21.15.29:0/522382225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d2690 msgr2=0x55f81b2d7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:02.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.599+0000 7f92a199e640 1 --2- 172.21.15.29:0/522382225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d2690 0x55f81b2d7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.599+0000 7f92a199e640 1 -- 172.21.15.29:0/522382225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 msgr2=0x55f81b2e3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:02.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.599+0000 7f92a199e640 1 --2- 172.21.15.29:0/522382225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 0x55f81b2e3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.599+0000 7f92a199e640 1 -- 172.21.15.29:0/522382225 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f81b1d5c00 con 0x55f81b2d3860 2024-03-11T19:10:02.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.599+0000 7f92a119d640 1 --2- 172.21.15.29:0/522382225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 0x55f81b2e3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:10:02.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.600+0000 7f92a199e640 1 --2- 172.21.15.29:0/522382225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d3860 0x55f81b2d7940 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x55f81b2eb620 tx=0x7f9298009890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9cf6f213bd6bde9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:02.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.600+0000 7f92877fe640 1 -- 172.21.15.29:0/522382225 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9298017040 con 0x55f81b2d3860 2024-03-11T19:10:02.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.600+0000 7f92877fe640 1 -- 172.21.15.29:0/522382225 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f929801e070 con 0x55f81b2d3860 2024-03-11T19:10:02.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.600+0000 7f92877fe640 1 -- 172.21.15.29:0/522382225 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9298011b30 con 0x55f81b2d3860 2024-03-11T19:10:02.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.601+0000 7f92a67d5740 1 -- 172.21.15.29:0/522382225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d3860 msgr2=0x55f81b2d7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.601+0000 7f92a67d5740 1 --2- 172.21.15.29:0/522382225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d3860 0x55f81b2d7940 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x55f81b2eb620 tx=0x7f9298009890 comp rx=0 tx=0).stop 2024-03-11T19:10:02.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.601+0000 7f92a67d5740 1 -- 172.21.15.29:0/522382225 shutdown_connections 2024-03-11T19:10:02.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.601+0000 7f92a67d5740 1 --2- 172.21.15.29:0/522382225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 0x55f81b2e3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.601+0000 7f92a67d5740 1 --2- 172.21.15.29:0/522382225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d3860 0x55f81b2d7940 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.601+0000 7f92a67d5740 1 --2- 172.21.15.29:0/522382225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d2690 0x55f81b2d7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.601+0000 7f92a67d5740 1 -- 172.21.15.29:0/522382225 >> 172.21.15.29:0/522382225 conn(0x55f81b2cd870 msgr2=0x55f81b2dc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:02.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.601+0000 7f92a67d5740 1 -- 172.21.15.29:0/522382225 shutdown_connections 2024-03-11T19:10:02.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.601+0000 7f92a67d5740 1 -- 172.21.15.29:0/522382225 wait complete. 2024-03-11T19:10:02.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.601+0000 7f92a67d5740 1 -- start start 2024-03-11T19:10:02.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.601+0000 7f92a67d5740 1 Processor -- start 2024-03-11T19:10:02.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.601+0000 7f92a67d5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d2690 0x55f81b2e3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.601+0000 7f92a67d5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d3860 0x55f81b2e3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.601+0000 7f92a67d5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 0x55f81b33ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.601+0000 7f92a67d5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f81b33f4c0 con 0x55f81b2d4220 2024-03-11T19:10:02.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.601+0000 7f92a67d5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f81b33f600 con 0x55f81b2d3860 2024-03-11T19:10:02.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.601+0000 7f92a67d5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f81b33f7c0 con 0x55f81b2d2690 2024-03-11T19:10:02.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.601+0000 7f92a67d5740 10 client.0 _ll_register_callbacks cb 0x7f928c00eb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:02.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.602+0000 7f92a119d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 0x55f81b33ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.602+0000 7f92a119d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 0x55f81b33ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54674/0 (socket says 172.21.15.29:54674) 2024-03-11T19:10:02.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.602+0000 7f92a119d640 1 -- 172.21.15.29:0/2907059271 learned_addr learned my addr 172.21.15.29:0/2907059271 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:02.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.602+0000 7f92a119d640 1 --2- 172.21.15.29:0/2907059271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d2690 0x55f81b2e3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.602+0000 7f92a199e640 1 --2- 172.21.15.29:0/2907059271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d3860 0x55f81b2e3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.602+0000 7f92a119d640 1 -- 172.21.15.29:0/2907059271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d2690 msgr2=0x55f81b2e3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:02.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.602+0000 7f92a119d640 1 --2- 172.21.15.29:0/2907059271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d2690 0x55f81b2e3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.602+0000 7f92a119d640 1 -- 172.21.15.29:0/2907059271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d3860 msgr2=0x55f81b2e3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:02.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.602+0000 7f92a119d640 1 --2- 172.21.15.29:0/2907059271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d3860 0x55f81b2e3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.602+0000 7f92a119d640 1 -- 172.21.15.29:0/2907059271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f81b239320 con 0x55f81b2d4220 2024-03-11T19:10:02.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.602+0000 7f92a199e640 1 --2- 172.21.15.29:0/2907059271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d3860 0x55f81b2e3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:10:02.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.602+0000 7f92a119d640 1 --2- 172.21.15.29:0/2907059271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 0x55f81b33ef80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f928c00ed70 tx=0x7f928c00eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f79a73eb9c0abaf server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:02.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.602+0000 7f92857fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:02.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.602+0000 7f92857fa640 1 -- 172.21.15.29:0/2907059271 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c00e2e0 con 0x55f81b2d4220 2024-03-11T19:10:02.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.602+0000 7f92857fa640 1 -- 172.21.15.29:0/2907059271 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f928c000e40 con 0x55f81b2d4220 2024-03-11T19:10:02.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.602+0000 7f92a67d5740 1 -- 172.21.15.29:0/2907059271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f81b260900 con 0x55f81b2d4220 2024-03-11T19:10:02.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.602+0000 7f92857fa640 1 -- 172.21.15.29:0/2907059271 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f928c00ce80 con 0x55f81b2d4220 2024-03-11T19:10:02.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.602+0000 7f92a67d5740 10 client.15474 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:02.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.602+0000 7f92a67d5740 1 -- 172.21.15.29:0/2907059271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7f9298009ce0 con 0x55f81b2d4220 2024-03-11T19:10:02.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.603+0000 7f92857fa640 1 -- 172.21.15.29:0/2907059271 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f928c04b080 con 0x55f81b2d4220 2024-03-11T19:10:02.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.603+0000 7f92857fa640 1 -- 172.21.15.29:0/2907059271 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f928c045030 con 0x55f81b2d4220 2024-03-11T19:10:02.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.603+0000 7f92a67d5740 10 client.15474 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:02.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.603+0000 7f92a67d5740 10 client.15474 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:10:02.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.603+0000 7f92a67d5740 20 client.15474 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:02.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.603+0000 7f92a67d5740 1 --2- 172.21.15.29:0/2907059271 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f81b340fc0 0x55f81b3433e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.603+0000 7f92a67d5740 4 client.15474 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:10:02.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.603+0000 7f92a67d5740 1 -- 172.21.15.29:0/2907059271 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f81af3a860 con 0x55f81b340fc0 2024-03-11T19:10:02.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.603+0000 7f92a199e640 1 --2- 172.21.15.29:0/2907059271 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f81b340fc0 0x55f81b3433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.604+0000 7f92a199e640 1 --2- 172.21.15.29:0/2907059271 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f81b340fc0 0x55f81b3433e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e344519227810575 server_cookie=6ce55298d392321 in_seq=0 out_seq=0 2024-03-11T19:10:02.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.604+0000 7f92857fa640 10 client.15474 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:02.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.605+0000 7f92857fa640 1 -- 172.21.15.29:0/2907059271 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f81af3a860 con 0x55f81b340fc0 2024-03-11T19:10:02.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.605+0000 7f92857fa640 10 client.15474 handle_command_reply: tid=0 2024-03-11T19:10:02.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.605+0000 7f92a67d5740 1 client.15474 shutdown 2024-03-11T19:10:02.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.605+0000 7f92a67d5740 1 -- 172.21.15.29:0/2907059271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 msgr2=0x55f81b33ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.605+0000 7f92a67d5740 1 --2- 172.21.15.29:0/2907059271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 0x55f81b33ef80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f928c00ed70 tx=0x7f928c00eda0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.605+0000 7f92a67d5740 1 -- 172.21.15.29:0/2907059271 shutdown_connections 2024-03-11T19:10:02.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.605+0000 7f92a67d5740 1 --2- 172.21.15.29:0/2907059271 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f81b340fc0 0x55f81b3433e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.605+0000 7f92a67d5740 1 --2- 172.21.15.29:0/2907059271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f81b2d4220 0x55f81b33ef80 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.605+0000 7f92a67d5740 1 --2- 172.21.15.29:0/2907059271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f81b2d3860 0x55f81b2e3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.605+0000 7f92a67d5740 1 --2- 172.21.15.29:0/2907059271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f81b2d2690 0x55f81b2e3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.605+0000 7f92857fa640 0 client.15474 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:02.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.605+0000 7f92a67d5740 1 -- 172.21.15.29:0/2907059271 >> 172.21.15.29:0/2907059271 conn(0x55f81b2cd870 msgr2=0x55f81b2dc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:02.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.605+0000 7f92a67d5740 1 -- 172.21.15.29:0/2907059271 shutdown_connections 2024-03-11T19:10:02.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.605+0000 7f92a67d5740 1 -- 172.21.15.29:0/2907059271 wait complete. 2024-03-11T19:10:02.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.605+0000 7f92a67d5740 20 client.15474 trim_cache size 0 max 16384 2024-03-11T19:10:02.616 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:10:02.616 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:10:02.616 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "d1feac02-b3b8-4507-9b63-2ecb0a75d9ad", 2024-03-11T19:10:02.617 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:10:02.617 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:10:02.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.606+0000 7f92a29a0640 1 -- 172.21.15.29:0/4106145825 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f92880a1ac0 msgr2=0x7f92880a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.606+0000 7f92a29a0640 1 --2- 172.21.15.29:0/4106145825 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f92880a1ac0 0x7f92880a3f80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9294033000 tx=0x7f9294050000 comp rx=0 tx=0).stop 2024-03-11T19:10:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.606+0000 7f92a29a0640 1 -- 172.21.15.29:0/4106145825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f929c101aa0 msgr2=0x7f929c19d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.606+0000 7f92a29a0640 1 --2- 172.21.15.29:0/4106145825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f929c101aa0 0x7f929c19d460 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f929402c630 tx=0x7f9294034c90 comp rx=0 tx=0).stop 2024-03-11T19:10:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.606+0000 7f92a29a0640 1 -- 172.21.15.29:0/4106145825 shutdown_connections 2024-03-11T19:10:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.606+0000 7f92a29a0640 1 --2- 172.21.15.29:0/4106145825 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f92880a1ac0 0x7f92880a3f80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.606+0000 7f92a29a0640 1 --2- 172.21.15.29:0/4106145825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f929c101aa0 0x7f929c19d460 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.606+0000 7f92a29a0640 1 --2- 172.21.15.29:0/4106145825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f929c1010e0 0x7f929c1963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.606+0000 7f92a29a0640 1 --2- 172.21.15.29:0/4106145825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f929c0fff10 0x7f929c195eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.606+0000 7f92a29a0640 1 -- 172.21.15.29:0/4106145825 >> 172.21.15.29:0/4106145825 conn(0x7f929c0fb4f0 msgr2=0x7f929c1045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.606+0000 7f92a29a0640 1 -- 172.21.15.29:0/4106145825 shutdown_connections 2024-03-11T19:10:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.606+0000 7f92a29a0640 1 -- 172.21.15.29:0/4106145825 wait complete. 2024-03-11T19:10:02.620 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:10:02.620 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:10:02.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.762+0000 7f7b351a0640 1 -- 172.21.15.29:0/3544107903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b301019b0 msgr2=0x7f7b301082d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.762+0000 7f7b351a0640 1 --2- 172.21.15.29:0/3544107903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b301019b0 0x7f7b301082d0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7b28009ca0 tx=0x7f7b2802c0f0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.762+0000 7f7b351a0640 1 -- 172.21.15.29:0/3544107903 shutdown_connections 2024-03-11T19:10:02.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.762+0000 7f7b351a0640 1 --2- 172.21.15.29:0/3544107903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b301019b0 0x7f7b301082d0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.762+0000 7f7b351a0640 1 --2- 172.21.15.29:0/3544107903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b30100ff0 0x7f7b30101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.762+0000 7f7b351a0640 1 --2- 172.21.15.29:0/3544107903 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b300ffea0 0x7f7b301002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.762+0000 7f7b351a0640 1 -- 172.21.15.29:0/3544107903 >> 172.21.15.29:0/3544107903 conn(0x7f7b300fb430 msgr2=0x7f7b300fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:02.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.762+0000 7f7b351a0640 1 -- 172.21.15.29:0/3544107903 shutdown_connections 2024-03-11T19:10:02.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.762+0000 7f7b351a0640 1 -- 172.21.15.29:0/3544107903 wait complete. 2024-03-11T19:10:02.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.762+0000 7f7b351a0640 1 Processor -- start 2024-03-11T19:10:02.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.762+0000 7f7b351a0640 1 -- start start 2024-03-11T19:10:02.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.763+0000 7f7b351a0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b300ffea0 0x7f7b30196fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.763+0000 7f7b351a0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b30100ff0 0x7f7b301974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.763+0000 7f7b351a0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b301019b0 0x7f7b3019e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.763+0000 7f7b351a0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3019eaa0 con 0x7f7b30100ff0 2024-03-11T19:10:02.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.763+0000 7f7b351a0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3019ec10 con 0x7f7b300ffea0 2024-03-11T19:10:02.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.763+0000 7f7b351a0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3019edd0 con 0x7f7b301019b0 2024-03-11T19:10:02.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.763+0000 7f7b2f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b30100ff0 0x7f7b301974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.763+0000 7f7b2f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b30100ff0 0x7f7b301974f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54694/0 (socket says 172.21.15.29:54694) 2024-03-11T19:10:02.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.763+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/3545584222 learned_addr learned my addr 172.21.15.29:0/3545584222 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:02.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.763+0000 7f7b2ffff640 1 --2- 172.21.15.29:0/3545584222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b300ffea0 0x7f7b30196fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.763+0000 7f7b3499f640 1 --2- 172.21.15.29:0/3545584222 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b301019b0 0x7f7b3019e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.763+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/3545584222 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b301019b0 msgr2=0x7f7b3019e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.763+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/3545584222 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b301019b0 0x7f7b3019e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.763+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/3545584222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b300ffea0 msgr2=0x7f7b30196fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.763+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/3545584222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b300ffea0 0x7f7b30196fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.763+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/3545584222 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b28009900 con 0x7f7b30100ff0 2024-03-11T19:10:02.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.763+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/3545584222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b30100ff0 0x7f7b301974f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7b2400d950 tx=0x7f7b2400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:02.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.764+0000 7f7b2d7fa640 1 -- 172.21.15.29:0/3545584222 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b24009c30 con 0x7f7b30100ff0 2024-03-11T19:10:02.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.764+0000 7f7b2d7fa640 1 -- 172.21.15.29:0/3545584222 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7b24009d90 con 0x7f7b30100ff0 2024-03-11T19:10:02.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.764+0000 7f7b351a0640 1 -- 172.21.15.29:0/3545584222 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b3019f0b0 con 0x7f7b30100ff0 2024-03-11T19:10:02.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.764+0000 7f7b2d7fa640 1 -- 172.21.15.29:0/3545584222 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2400f6b0 con 0x7f7b30100ff0 2024-03-11T19:10:02.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.764+0000 7f7b351a0640 1 -- 172.21.15.29:0/3545584222 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3019f570 con 0x7f7b30100ff0 2024-03-11T19:10:02.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.766+0000 7f7b2d7fa640 1 -- 172.21.15.29:0/3545584222 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f7b24010a30 con 0x7f7b30100ff0 2024-03-11T19:10:02.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.766+0000 7f7b38fcf740 1 Processor -- start 2024-03-11T19:10:02.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.766+0000 7f7b38fcf740 1 -- start start 2024-03-11T19:10:02.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.766+0000 7f7b2d7fa640 1 --2- 172.21.15.29:0/3545584222 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7b180a1ac0 0x7f7b180a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.766+0000 7f7b38fcf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6a8900 0x55fe2e6a8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.766+0000 7f7b38fcf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6addf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.766+0000 7f7b38fcf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6ae460 0x55fe2e6b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.766+0000 7f7b38fcf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe2e614280 con 0x55fe2e6a92f0 2024-03-11T19:10:02.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.766+0000 7f7b38fcf740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe2e6117b0 con 0x55fe2e6ae460 2024-03-11T19:10:02.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.766+0000 7f7b38fcf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe2e60cf90 con 0x55fe2e6a8900 2024-03-11T19:10:02.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.766+0000 7f7b3499f640 1 --2- 172.21.15.29:0/3545584222 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7b180a1ac0 0x7f7b180a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.766+0000 7f7b2d7fa640 1 -- 172.21.15.29:0/3545584222 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7b240cd170 con 0x7f7b30100ff0 2024-03-11T19:10:02.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.766+0000 7f7b2f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.766+0000 7f7b2f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6addf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54700/0 (socket says 172.21.15.29:54700) 2024-03-11T19:10:02.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.766+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/3624502619 learned_addr learned my addr 172.21.15.29:0/3624502619 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:02.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.767+0000 7f7b2ffff640 1 --2- 172.21.15.29:0/3624502619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6a8900 0x55fe2e6a8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.767+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/3624502619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6a8900 msgr2=0x55fe2e6a8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:02.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.767+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/3624502619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6a8900 0x55fe2e6a8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.767+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/3624502619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6ae460 msgr2=0x55fe2e6b08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:02.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.767+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/3624502619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6ae460 0x55fe2e6b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.767+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/3624502619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe2e5cb570 con 0x55fe2e6a92f0 2024-03-11T19:10:02.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.767+0000 7f7b3499f640 1 --2- 172.21.15.29:0/3545584222 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7b180a1ac0 0x7f7b180a3f80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7b28006010 tx=0x7f7b280323a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:02.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.767+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/3624502619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6addf0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009f20 tx=0x7f7b2409a030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76b0fa2bdf2e96b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:02.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.767+0000 7f7b167fc640 1 -- 172.21.15.29:0/3624502619 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2409a130 con 0x55fe2e6a92f0 2024-03-11T19:10:02.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.767+0000 7f7b167fc640 1 -- 172.21.15.29:0/3624502619 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7b2409e1d0 con 0x55fe2e6a92f0 2024-03-11T19:10:02.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.767+0000 7f7b167fc640 1 -- 172.21.15.29:0/3624502619 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b24096910 con 0x55fe2e6a92f0 2024-03-11T19:10:02.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.768+0000 7f7b38fcf740 1 -- 172.21.15.29:0/3624502619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 msgr2=0x55fe2e6addf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.768+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/3624502619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6addf0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009f20 tx=0x7f7b2409a030 comp rx=0 tx=0).stop 2024-03-11T19:10:02.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.768+0000 7f7b38fcf740 1 -- 172.21.15.29:0/3624502619 shutdown_connections 2024-03-11T19:10:02.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.768+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/3624502619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6ae460 0x55fe2e6b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.768+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/3624502619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6addf0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.768+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/3624502619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6a8900 0x55fe2e6a8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.768+0000 7f7b38fcf740 1 -- 172.21.15.29:0/3624502619 >> 172.21.15.29:0/3624502619 conn(0x55fe2e69fa60 msgr2=0x55fe2e69fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:02.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.768+0000 7f7b38fcf740 1 -- 172.21.15.29:0/3624502619 shutdown_connections 2024-03-11T19:10:02.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.768+0000 7f7b38fcf740 1 -- 172.21.15.29:0/3624502619 wait complete. 2024-03-11T19:10:02.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.768+0000 7f7b38fcf740 1 -- start start 2024-03-11T19:10:02.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.768+0000 7f7b38fcf740 1 Processor -- start 2024-03-11T19:10:02.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b38fcf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6a8900 0x55fe2e6b2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b38fcf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b38fcf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6ae460 0x55fe2e6b9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b38fcf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe2e2e0430 con 0x55fe2e6a92f0 2024-03-11T19:10:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b38fcf740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe2e29ce00 con 0x55fe2e6ae460 2024-03-11T19:10:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b38fcf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe2e591fd0 con 0x55fe2e6a8900 2024-03-11T19:10:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b38fcf740 10 client.0 _ll_register_callbacks cb 0x55fe2e5e7fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b2f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b2f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6b2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54706/0 (socket says 172.21.15.29:54706) 2024-03-11T19:10:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/4207824129 learned_addr learned my addr 172.21.15.29:0/4207824129 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b2ffff640 1 --2- 172.21.15.29:0/4207824129 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6a8900 0x55fe2e6b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b3499f640 1 --2- 172.21.15.29:0/4207824129 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6ae460 0x55fe2e6b9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/4207824129 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6a8900 msgr2=0x55fe2e6b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/4207824129 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6a8900 0x55fe2e6b2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/4207824129 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6ae460 msgr2=0x55fe2e6b9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:02.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/4207824129 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6ae460 0x55fe2e6b9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/4207824129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3019f570 con 0x55fe2e6a92f0 2024-03-11T19:10:02.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b3499f640 1 --2- 172.21.15.29:0/4207824129 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6ae460 0x55fe2e6b9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:10:02.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/4207824129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6b2d50 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7b240caca0 tx=0x7f7b24015340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c631f372da12dc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:02.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b0f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:02.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b0f7fe640 1 -- 172.21.15.29:0/4207824129 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b240010e0 con 0x55fe2e6a92f0 2024-03-11T19:10:02.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b0f7fe640 1 -- 172.21.15.29:0/4207824129 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7b240a2820 con 0x55fe2e6a92f0 2024-03-11T19:10:02.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b38fcf740 1 -- 172.21.15.29:0/4207824129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fe2e30c7e0 con 0x55fe2e6a92f0 2024-03-11T19:10:02.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b0f7fe640 1 -- 172.21.15.29:0/4207824129 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b240968f0 con 0x55fe2e6a92f0 2024-03-11T19:10:02.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.769+0000 7f7b38fcf740 10 client.15486 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:02.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.770+0000 7f7b38fcf740 1 -- 172.21.15.29:0/4207824129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55fe2e60b610 con 0x55fe2e6a92f0 2024-03-11T19:10:02.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.770+0000 7f7b0f7fe640 1 -- 172.21.15.29:0/4207824129 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55fe2e30c7e0 con 0x55fe2e6a92f0 2024-03-11T19:10:02.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.770+0000 7f7b0f7fe640 1 -- 172.21.15.29:0/4207824129 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7b240b2030 con 0x55fe2e6a92f0 2024-03-11T19:10:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.770+0000 7f7b38fcf740 10 client.15486 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.770+0000 7f7b38fcf740 10 client.15486 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:10:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.770+0000 7f7b38fcf740 20 client.15486 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.770+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/4207824129 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55fe2e6be750 0x55fe2e6c0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.770+0000 7f7b38fcf740 4 client.15486 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:10:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.770+0000 7f7b38fcf740 1 -- 172.21.15.29:0/4207824129 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe2e349720 con 0x55fe2e6be750 2024-03-11T19:10:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.770+0000 7f7b2ffff640 1 --2- 172.21.15.29:0/4207824129 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55fe2e6be750 0x55fe2e6c0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.771+0000 7f7b2ffff640 1 --2- 172.21.15.29:0/4207824129 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55fe2e6be750 0x55fe2e6c0b70 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7eafa40c7fb99b4e server_cookie=2c52874f8b197c3e in_seq=0 out_seq=0 2024-03-11T19:10:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.771+0000 7f7b0f7fe640 10 client.15486 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.771+0000 7f7b0f7fe640 1 -- 172.21.15.29:0/4207824129 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fe2e349720 con 0x55fe2e6be750 2024-03-11T19:10:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.771+0000 7f7b0f7fe640 10 client.15486 handle_command_reply: tid=0 2024-03-11T19:10:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.771+0000 7f7b38fcf740 1 client.15486 shutdown 2024-03-11T19:10:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.771+0000 7f7b38fcf740 1 -- 172.21.15.29:0/4207824129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 msgr2=0x55fe2e6b2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.771+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/4207824129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6b2d50 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7b240caca0 tx=0x7f7b24015340 comp rx=0 tx=0).stop 2024-03-11T19:10:02.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.771+0000 7f7b38fcf740 1 -- 172.21.15.29:0/4207824129 shutdown_connections 2024-03-11T19:10:02.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.771+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/4207824129 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6ae460 0x55fe2e6b9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.771+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/4207824129 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55fe2e6be750 0x55fe2e6c0b70 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.771+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/4207824129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6b2d50 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.771+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/4207824129 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6a8900 0x55fe2e6b2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.771+0000 7f7b0f7fe640 0 client.15486 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:02.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.771+0000 7f7b38fcf740 1 -- 172.21.15.29:0/4207824129 >> 172.21.15.29:0/4207824129 conn(0x55fe2e69fa60 msgr2=0x55fe2e69ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:02.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.772+0000 7f7b38fcf740 1 -- 172.21.15.29:0/4207824129 shutdown_connections 2024-03-11T19:10:02.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.772+0000 7f7b38fcf740 1 -- 172.21.15.29:0/4207824129 wait complete. 2024-03-11T19:10:02.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.772+0000 7f7b38fcf740 20 client.15486 trim_cache size 0 max 16384 2024-03-11T19:10:02.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.786+0000 7f7b38fcf740 1 Processor -- start 2024-03-11T19:10:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.786+0000 7f7b38fcf740 1 -- start start 2024-03-11T19:10:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b38fcf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a8900 0x55fe2e6a33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b38fcf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6b9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b38fcf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6ae460 0x55fe2e69e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b38fcf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe2e30c7e0 con 0x55fe2e6a8900 2024-03-11T19:10:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b38fcf740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe2e6a85a0 con 0x55fe2e6ae460 2024-03-11T19:10:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b38fcf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe2e6a99e0 con 0x55fe2e6a92f0 2024-03-11T19:10:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b2ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a8900 0x55fe2e6a33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b2ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a8900 0x55fe2e6a33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54718/0 (socket says 172.21.15.29:54718) 2024-03-11T19:10:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b2ffff640 1 -- 172.21.15.29:0/3529163693 learned_addr learned my addr 172.21.15.29:0/3529163693 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b2ffff640 1 -- 172.21.15.29:0/3529163693 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6a92f0 msgr2=0x55fe2e6b9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:02.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b2ffff640 1 --2- 172.21.15.29:0/3529163693 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6b9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b2ffff640 1 -- 172.21.15.29:0/3529163693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6ae460 msgr2=0x55fe2e69e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:02.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b3499f640 1 --2- 172.21.15.29:0/3529163693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6ae460 0x55fe2e69e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b2ffff640 1 --2- 172.21.15.29:0/3529163693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6ae460 0x55fe2e69e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b2ffff640 1 -- 172.21.15.29:0/3529163693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe2e5a7ac0 con 0x55fe2e6a8900 2024-03-11T19:10:02.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b3499f640 1 --2- 172.21.15.29:0/3529163693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6ae460 0x55fe2e69e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:02.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b2ffff640 1 --2- 172.21.15.29:0/3529163693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a8900 0x55fe2e6a33d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f7b20009e50 tx=0x7f7b20005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8075003780f0c5da server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:02.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b15ffb640 1 -- 172.21.15.29:0/3529163693 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20017070 con 0x55fe2e6a8900 2024-03-11T19:10:02.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b15ffb640 1 -- 172.21.15.29:0/3529163693 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7b20018040 con 0x55fe2e6a8900 2024-03-11T19:10:02.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.787+0000 7f7b15ffb640 1 -- 172.21.15.29:0/3529163693 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2001d380 con 0x55fe2e6a8900 2024-03-11T19:10:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.788+0000 7f7b38fcf740 1 -- 172.21.15.29:0/3529163693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a8900 msgr2=0x55fe2e6a33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.788+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/3529163693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a8900 0x55fe2e6a33d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f7b20009e50 tx=0x7f7b20005f90 comp rx=0 tx=0).stop 2024-03-11T19:10:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.788+0000 7f7b38fcf740 1 -- 172.21.15.29:0/3529163693 shutdown_connections 2024-03-11T19:10:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.788+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/3529163693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6ae460 0x55fe2e69e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.788+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/3529163693 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6b9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.788+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/3529163693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a8900 0x55fe2e6a33d0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.788+0000 7f7b38fcf740 1 -- 172.21.15.29:0/3529163693 >> 172.21.15.29:0/3529163693 conn(0x55fe2e69fa60 msgr2=0x55fe2e6a2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.788+0000 7f7b38fcf740 1 -- 172.21.15.29:0/3529163693 shutdown_connections 2024-03-11T19:10:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.788+0000 7f7b38fcf740 1 -- 172.21.15.29:0/3529163693 wait complete. 2024-03-11T19:10:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.788+0000 7f7b38fcf740 1 -- start start 2024-03-11T19:10:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b38fcf740 1 Processor -- start 2024-03-11T19:10:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b38fcf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6a8900 0x55fe2e6b8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b38fcf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b38fcf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6ae460 0x55fe2e714c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b38fcf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe2e6b94f0 con 0x55fe2e6a92f0 2024-03-11T19:10:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b38fcf740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe2e6b9660 con 0x55fe2e6a8900 2024-03-11T19:10:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b38fcf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe2e715180 con 0x55fe2e6ae460 2024-03-11T19:10:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b38fcf740 10 client.0 _ll_register_callbacks cb 0x7f7b2400bed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b2f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b2f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6b8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54726/0 (socket says 172.21.15.29:54726) 2024-03-11T19:10:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/378942690 learned_addr learned my addr 172.21.15.29:0/378942690 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b2ffff640 1 --2- 172.21.15.29:0/378942690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6a8900 0x55fe2e6b8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b3499f640 1 --2- 172.21.15.29:0/378942690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6ae460 0x55fe2e714c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/378942690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6ae460 msgr2=0x55fe2e714c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:02.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/378942690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6ae460 0x55fe2e714c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/378942690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6a8900 msgr2=0x55fe2e6b8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:02.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/378942690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6a8900 0x55fe2e6b8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/378942690 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b20010430 con 0x55fe2e6a92f0 2024-03-11T19:10:02.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b3499f640 1 --2- 172.21.15.29:0/378942690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6ae460 0x55fe2e714c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:10:02.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/378942690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6b8d90 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009610 tx=0x7f7b24009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7fb1d902eba472e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:02.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.789+0000 7f7b0f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:02.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.790+0000 7f7b0f7fe640 1 -- 172.21.15.29:0/378942690 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b240960d0 con 0x55fe2e6a92f0 2024-03-11T19:10:02.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.790+0000 7f7b0f7fe640 1 -- 172.21.15.29:0/378942690 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7b24005520 con 0x55fe2e6a92f0 2024-03-11T19:10:02.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.790+0000 7f7b38fcf740 1 -- 172.21.15.29:0/378942690 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b2409e1d0 con 0x55fe2e6a92f0 2024-03-11T19:10:02.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.790+0000 7f7b0f7fe640 1 -- 172.21.15.29:0/378942690 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b24096230 con 0x55fe2e6a92f0 2024-03-11T19:10:02.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.790+0000 7f7b38fcf740 10 client.15492 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:02.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.790+0000 7f7b38fcf740 1 -- 172.21.15.29:0/378942690 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55fe2e5cd6a0 con 0x55fe2e6a92f0 2024-03-11T19:10:02.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.790+0000 7f7b0f7fe640 1 -- 172.21.15.29:0/378942690 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7b2409e1d0 con 0x55fe2e6a92f0 2024-03-11T19:10:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.790+0000 7f7b0f7fe640 1 -- 172.21.15.29:0/378942690 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7b240b2040 con 0x55fe2e6a92f0 2024-03-11T19:10:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.790+0000 7f7b38fcf740 10 client.15492 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.790+0000 7f7b38fcf740 10 client.15492 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:10:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.790+0000 7f7b38fcf740 20 client.15492 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.790+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/378942690 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55fe2e7164a0 0x55fe2e718890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.790+0000 7f7b38fcf740 4 client.15492 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:10:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.790+0000 7f7b38fcf740 1 -- 172.21.15.29:0/378942690 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fe2e398890 con 0x55fe2e7164a0 2024-03-11T19:10:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.790+0000 7f7b2ffff640 1 --2- 172.21.15.29:0/378942690 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55fe2e7164a0 0x55fe2e718890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.791+0000 7f7b0f7fe640 10 client.15492 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.791+0000 7f7b2ffff640 1 --2- 172.21.15.29:0/378942690 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55fe2e7164a0 0x55fe2e718890 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f1ce52ca7a262b4 server_cookie=50e7c88dfea1fca2 in_seq=0 out_seq=0 2024-03-11T19:10:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.791+0000 7f7b0f7fe640 1 -- 172.21.15.29:0/378942690 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55fe2e398890 con 0x55fe2e7164a0 2024-03-11T19:10:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.791+0000 7f7b0f7fe640 10 client.15492 handle_command_reply: tid=0 2024-03-11T19:10:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.791+0000 7f7b38fcf740 1 client.15492 shutdown 2024-03-11T19:10:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.791+0000 7f7b38fcf740 1 -- 172.21.15.29:0/378942690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 msgr2=0x55fe2e6b8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.791+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/378942690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6b8d90 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009610 tx=0x7f7b24009f00 comp rx=0 tx=0).stop 2024-03-11T19:10:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.791+0000 7f7b38fcf740 1 -- 172.21.15.29:0/378942690 shutdown_connections 2024-03-11T19:10:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.791+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/378942690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2e6ae460 0x55fe2e714c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.791+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/378942690 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55fe2e7164a0 0x55fe2e718890 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.791+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/378942690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe2e6a92f0 0x55fe2e6b8d90 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.791+0000 7f7b0f7fe640 0 client.15492 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.791+0000 7f7b38fcf740 1 --2- 172.21.15.29:0/378942690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe2e6a8900 0x55fe2e6b8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.791+0000 7f7b38fcf740 1 -- 172.21.15.29:0/378942690 >> 172.21.15.29:0/378942690 conn(0x55fe2e69fa60 msgr2=0x55fe2e6a3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.791+0000 7f7b38fcf740 1 -- 172.21.15.29:0/378942690 shutdown_connections 2024-03-11T19:10:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.791+0000 7f7b38fcf740 1 -- 172.21.15.29:0/378942690 wait complete. 2024-03-11T19:10:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.791+0000 7f7b38fcf740 20 client.15492 trim_cache size 0 max 16384 2024-03-11T19:10:02.801 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:10:02.802 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:10:02.802 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:10:02.802 INFO:teuthology.orchestra.run.smithi029.stdout: "d1feac02-b3b8-4507-9b63-2ecb0a75d9ad": { 2024-03-11T19:10:02.802 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:10:02.802 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "d1feac02-b3b8-4507-9b63-2ecb0a75d9ad", 2024-03-11T19:10:02.802 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:10:02.802 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:10:02.802 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:10:02.802 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:10:02.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.792+0000 7f7b351a0640 1 -- 172.21.15.29:0/3545584222 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7b180a1ac0 msgr2=0x7f7b180a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.792+0000 7f7b351a0640 1 --2- 172.21.15.29:0/3545584222 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7b180a1ac0 0x7f7b180a3f80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7b28006010 tx=0x7f7b280323a0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.792+0000 7f7b351a0640 1 -- 172.21.15.29:0/3545584222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b30100ff0 msgr2=0x7f7b301974f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:02.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.792+0000 7f7b351a0640 1 --2- 172.21.15.29:0/3545584222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b30100ff0 0x7f7b301974f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7b2400d950 tx=0x7f7b2400de10 comp rx=0 tx=0).stop 2024-03-11T19:10:02.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.792+0000 7f7b351a0640 1 -- 172.21.15.29:0/3545584222 shutdown_connections 2024-03-11T19:10:02.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.792+0000 7f7b351a0640 1 --2- 172.21.15.29:0/3545584222 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b301019b0 0x7f7b3019e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.792+0000 7f7b351a0640 1 --2- 172.21.15.29:0/3545584222 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7b180a1ac0 0x7f7b180a3f80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.792+0000 7f7b351a0640 1 --2- 172.21.15.29:0/3545584222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b30100ff0 0x7f7b301974f0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.792+0000 7f7b351a0640 1 --2- 172.21.15.29:0/3545584222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b300ffea0 0x7f7b30196fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:02.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.792+0000 7f7b351a0640 1 -- 172.21.15.29:0/3545584222 >> 172.21.15.29:0/3545584222 conn(0x7f7b300fb430 msgr2=0x7f7b300fceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:02.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.792+0000 7f7b351a0640 1 -- 172.21.15.29:0/3545584222 shutdown_connections 2024-03-11T19:10:02.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:02.792+0000 7f7b351a0640 1 -- 172.21.15.29:0/3545584222 wait complete. 2024-03-11T19:10:02.806 INFO:tasks.cephfs.filesystem:scrub status for tag:d1feac02-b3b8-4507-9b63-2ecb0a75d9ad - {'path': '/', 'tag': 'd1feac02-b3b8-4507-9b63-2ecb0a75d9ad', 'options': 'recursive,force'} 2024-03-11T19:10:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:03 smithi190 ceph-mon[36292]: pgmap v358: 97 pgs: 97 active+clean; 651 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.0 KiB/s wr, 12 op/s 2024-03-11T19:10:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:03 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/4152660295' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:10:03.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:03 smithi170 ceph-mon[35690]: pgmap v358: 97 pgs: 97 active+clean; 651 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.0 KiB/s wr, 12 op/s 2024-03-11T19:10:03.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:03 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/4152660295' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:10:03.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:03 smithi029 ceph-mon[32439]: pgmap v358: 97 pgs: 97 active+clean; 651 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.0 KiB/s wr, 12 op/s 2024-03-11T19:10:03.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:03 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/4152660295' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:10:04.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:04 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:10:04.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:04 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:10:04.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:04 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:10:04.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:04 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:10:04.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:04 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:10:04.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:04 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:10:04.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:04 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:10:04.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:04 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:10:04.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:04 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:10:04.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:04 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:10:04.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:04 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:10:04.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:04 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:10:05.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:05 smithi190 ceph-mon[36292]: pgmap v359: 97 pgs: 97 active+clean; 659 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 13 op/s 2024-03-11T19:10:05.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:05 smithi170 ceph-mon[35690]: pgmap v359: 97 pgs: 97 active+clean; 659 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 13 op/s 2024-03-11T19:10:05.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:05 smithi029 ceph-mon[32439]: pgmap v359: 97 pgs: 97 active+clean; 659 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 13 op/s 2024-03-11T19:10:06.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:06 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:10:06.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:06 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:10:06.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:06 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:10:07.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:07 smithi190 ceph-mon[36292]: pgmap v360: 97 pgs: 97 active+clean; 659 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-03-11T19:10:07.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:07 smithi170 ceph-mon[35690]: pgmap v360: 97 pgs: 97 active+clean; 659 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-03-11T19:10:07.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:07 smithi029 ceph-mon[32439]: pgmap v360: 97 pgs: 97 active+clean; 659 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-03-11T19:10:09.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:09 smithi190 ceph-mon[36292]: pgmap v361: 97 pgs: 97 active+clean; 659 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-11T19:10:09.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:09 smithi170 ceph-mon[35690]: pgmap v361: 97 pgs: 97 active+clean; 659 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-11T19:10:09.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:09 smithi029 ceph-mon[32439]: pgmap v361: 97 pgs: 97 active+clean; 659 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-11T19:10:10.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:10 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:10:10.595 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:10 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:10:10.630 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:10 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:10:11.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:11 smithi190 ceph-mon[36292]: pgmap v362: 97 pgs: 97 active+clean; 659 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-11T19:10:11.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:11 smithi170 ceph-mon[35690]: pgmap v362: 97 pgs: 97 active+clean; 659 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-11T19:10:11.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:11 smithi029 ceph-mon[32439]: pgmap v362: 97 pgs: 97 active+clean; 659 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-11T19:10:12.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:12 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:12 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:12 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:12 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:12 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:12 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:12 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:10:12.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:12 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:10:12.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:12 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:12 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:12 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:12 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:12 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:12 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:12 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:12 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:10:12.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:12 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:10:12.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:12 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:12 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:12 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:12 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:12 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:12 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:12 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:12.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:12 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:10:12.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:12 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:10:12.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:12 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:10:13.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:13 smithi170 ceph-mon[35690]: pgmap v363: 97 pgs: 97 active+clean; 659 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-11T19:10:13.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:13 smithi029 ceph-mon[32439]: pgmap v363: 97 pgs: 97 active+clean; 659 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-11T19:10:14.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:13 smithi190 ceph-mon[36292]: pgmap v363: 97 pgs: 97 active+clean; 659 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-11T19:10:15.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:15 smithi170 ceph-mon[35690]: pgmap v364: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-03-11T19:10:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:15 smithi029 ceph-mon[32439]: pgmap v364: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-03-11T19:10:16.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:15 smithi190 ceph-mon[36292]: pgmap v364: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-03-11T19:10:17.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:17 smithi170 ceph-mon[35690]: pgmap v365: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-11T19:10:17.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:17 smithi029 ceph-mon[32439]: pgmap v365: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-11T19:10:18.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:17 smithi190 ceph-mon[36292]: pgmap v365: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-11T19:10:19.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:19 smithi170 ceph-mon[35690]: pgmap v366: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-11T19:10:19.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:19 smithi029 ceph-mon[32439]: pgmap v366: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-11T19:10:20.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:19 smithi190 ceph-mon[36292]: pgmap v366: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-11T19:10:21.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:21 smithi170 ceph-mon[35690]: pgmap v367: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-11T19:10:21.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:21 smithi029 ceph-mon[32439]: pgmap v367: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-11T19:10:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:21 smithi190 ceph-mon[36292]: pgmap v367: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-11T19:10:23.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:23 smithi170 ceph-mon[35690]: pgmap v368: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-11T19:10:23.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:23 smithi029 ceph-mon[32439]: pgmap v368: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-11T19:10:24.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:23 smithi190 ceph-mon[36292]: pgmap v368: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-11T19:10:25.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:25 smithi170 ceph-mon[35690]: pgmap v369: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-11T19:10:25.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:25 smithi029 ceph-mon[32439]: pgmap v369: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-11T19:10:26.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:25 smithi190 ceph-mon[36292]: pgmap v369: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-11T19:10:27.814 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:27 smithi190 ceph-mon[36292]: pgmap v370: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:27.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:27 smithi170 ceph-mon[35690]: pgmap v370: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:27.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:27 smithi029 ceph-mon[32439]: pgmap v370: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:29.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:29 smithi170 ceph-mon[35690]: pgmap v371: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:10:29.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:29 smithi029 ceph-mon[32439]: pgmap v371: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:10:30.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:29 smithi190 ceph-mon[36292]: pgmap v371: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:10:31.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:31 smithi170 ceph-mon[35690]: pgmap v372: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:31.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:31 smithi029 ceph-mon[32439]: pgmap v372: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:32.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:31 smithi190 ceph-mon[36292]: pgmap v372: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:32.807 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:10:32.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:10:32.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.918+0000 7f19376ae640 1 -- 172.21.15.29:0/4285440686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1930102d50 msgr2=0x7f19301095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:32.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.918+0000 7f19376ae640 1 --2- 172.21.15.29:0/4285440686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1930102d50 0x7f19301095e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f1928009ca0 tx=0x7f192802c0f0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.918+0000 7f19376ae640 1 -- 172.21.15.29:0/4285440686 shutdown_connections 2024-03-11T19:10:32.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.918+0000 7f19376ae640 1 --2- 172.21.15.29:0/4285440686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1930102d50 0x7f19301095e0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.918+0000 7f19376ae640 1 --2- 172.21.15.29:0/4285440686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1930102390 0x7f1930102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.918+0000 7f19376ae640 1 --2- 172.21.15.29:0/4285440686 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f19301011a0 0x7f19301015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.918+0000 7f19376ae640 1 -- 172.21.15.29:0/4285440686 >> 172.21.15.29:0/4285440686 conn(0x7f19300fc730 msgr2=0x7f19300feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:32.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.918+0000 7f19376ae640 1 -- 172.21.15.29:0/4285440686 shutdown_connections 2024-03-11T19:10:32.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.918+0000 7f19376ae640 1 -- 172.21.15.29:0/4285440686 wait complete. 2024-03-11T19:10:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.919+0000 7f19376ae640 1 Processor -- start 2024-03-11T19:10:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.919+0000 7f19376ae640 1 -- start start 2024-03-11T19:10:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.919+0000 7f19376ae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19301011a0 0x7f19301970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.919+0000 7f19376ae640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1930102390 0x7f19301975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.919+0000 7f19376ae640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1930102d50 0x7f193019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.919+0000 7f19376ae640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193019eba0 con 0x7f19301011a0 2024-03-11T19:10:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.919+0000 7f19376ae640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193019ed10 con 0x7f1930102d50 2024-03-11T19:10:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.919+0000 7f19376ae640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193019eed0 con 0x7f1930102390 2024-03-11T19:10:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.919+0000 7f193544a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19301011a0 0x7f19301970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.919+0000 7f193544a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19301011a0 0x7f19301970b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42360/0 (socket says 172.21.15.29:42360) 2024-03-11T19:10:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.919+0000 7f193544a640 1 -- 172.21.15.29:0/4206178272 learned_addr learned my addr 172.21.15.29:0/4206178272 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.920+0000 7f1934c49640 1 --2- 172.21.15.29:0/4206178272 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1930102390 0x7f19301975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.920+0000 7f1935c4b640 1 --2- 172.21.15.29:0/4206178272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1930102d50 0x7f193019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.920+0000 7f193544a640 1 -- 172.21.15.29:0/4206178272 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1930102390 msgr2=0x7f19301975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:32.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.920+0000 7f193544a640 1 --2- 172.21.15.29:0/4206178272 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1930102390 0x7f19301975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.920+0000 7f193544a640 1 -- 172.21.15.29:0/4206178272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1930102d50 msgr2=0x7f193019e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:32.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.920+0000 7f193544a640 1 --2- 172.21.15.29:0/4206178272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1930102d50 0x7f193019e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.920+0000 7f193544a640 1 -- 172.21.15.29:0/4206178272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1928009900 con 0x7f19301011a0 2024-03-11T19:10:32.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.920+0000 7f1935c4b640 1 --2- 172.21.15.29:0/4206178272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1930102d50 0x7f193019e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:32.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.920+0000 7f193544a640 1 --2- 172.21.15.29:0/4206178272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19301011a0 0x7f19301970b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f192c00d950 tx=0x7f192c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:32.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.920+0000 7f19267fc640 1 -- 172.21.15.29:0/4206178272 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c009c30 con 0x7f19301011a0 2024-03-11T19:10:32.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.920+0000 7f19267fc640 1 -- 172.21.15.29:0/4206178272 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f192c009d90 con 0x7f19301011a0 2024-03-11T19:10:32.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.920+0000 7f19376ae640 1 -- 172.21.15.29:0/4206178272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193019f1b0 con 0x7f19301011a0 2024-03-11T19:10:32.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.920+0000 7f19267fc640 1 -- 172.21.15.29:0/4206178272 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c00f7e0 con 0x7f19301011a0 2024-03-11T19:10:32.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.920+0000 7f19376ae640 1 -- 172.21.15.29:0/4206178272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193019f710 con 0x7f19301011a0 2024-03-11T19:10:32.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.922+0000 7f19267fc640 1 -- 172.21.15.29:0/4206178272 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f192c010460 con 0x7f19301011a0 2024-03-11T19:10:32.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.923+0000 7f193a280740 1 Processor -- start 2024-03-11T19:10:32.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.923+0000 7f19267fc640 1 --2- 172.21.15.29:0/4206178272 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f191c0a1b40 0x7f191c0a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:32.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.923+0000 7f193a280740 1 -- start start 2024-03-11T19:10:32.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.923+0000 7f193a280740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775eb2d0 0x558c775eb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:32.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.923+0000 7f19267fc640 1 -- 172.21.15.29:0/4206178272 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f192c0cd2b0 con 0x7f19301011a0 2024-03-11T19:10:32.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.923+0000 7f193a280740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775ebcc0 0x558c775f07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:32.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.923+0000 7f1935c4b640 1 --2- 172.21.15.29:0/4206178272 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f191c0a1b40 0x7f191c0a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:32.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.923+0000 7f193a280740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775f0e30 0x558c775f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:32.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.923+0000 7f193a280740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c7755b280 con 0x558c775f0e30 2024-03-11T19:10:32.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.923+0000 7f193a280740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c775587b0 con 0x558c775ebcc0 2024-03-11T19:10:32.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.923+0000 7f193a280740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c77553f90 con 0x558c775eb2d0 2024-03-11T19:10:32.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.923+0000 7f1935c4b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775f0e30 0x558c775f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:32.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.923+0000 7f1935c4b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775f0e30 0x558c775f32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42364/0 (socket says 172.21.15.29:42364) 2024-03-11T19:10:32.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.923+0000 7f1935c4b640 1 -- 172.21.15.29:0/2436845496 learned_addr learned my addr 172.21.15.29:0/2436845496 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:32.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.923+0000 7f193544a640 1 --2- 172.21.15.29:0/2436845496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775eb2d0 0x558c775eb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:32.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.923+0000 7f1934c49640 1 --2- 172.21.15.29:0/2436845496 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775ebcc0 0x558c775f07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:32.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.924+0000 7f1935c4b640 1 -- 172.21.15.29:0/2436845496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775eb2d0 msgr2=0x558c775eb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:32.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.924+0000 7f1935c4b640 1 --2- 172.21.15.29:0/2436845496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775eb2d0 0x558c775eb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.924+0000 7f1935c4b640 1 -- 172.21.15.29:0/2436845496 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775ebcc0 msgr2=0x558c775f07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:32.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.924+0000 7f1935c4b640 1 --2- 172.21.15.29:0/2436845496 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775ebcc0 0x558c775f07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.924+0000 7f1935c4b640 1 -- 172.21.15.29:0/2436845496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c77512570 con 0x558c775f0e30 2024-03-11T19:10:32.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.924+0000 7f1935c4b640 1 --2- 172.21.15.29:0/4206178272 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f191c0a1b40 0x7f191c0a4000 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1928032fc0 tx=0x7f192802ecc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:32.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.924+0000 7f1935c4b640 1 --2- 172.21.15.29:0/2436845496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775f0e30 0x558c775f32c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f192802c630 tx=0x7f1928046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8580f79295f9dc55 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:32.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.924+0000 7f191b7fe640 1 -- 172.21.15.29:0/2436845496 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1928004d40 con 0x558c775f0e30 2024-03-11T19:10:32.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.924+0000 7f191b7fe640 1 -- 172.21.15.29:0/2436845496 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1928034730 con 0x558c775f0e30 2024-03-11T19:10:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.924+0000 7f191b7fe640 1 -- 172.21.15.29:0/2436845496 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1928004040 con 0x558c775f0e30 2024-03-11T19:10:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.925+0000 7f193a280740 1 -- 172.21.15.29:0/2436845496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775f0e30 msgr2=0x558c775f32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.925+0000 7f193a280740 1 --2- 172.21.15.29:0/2436845496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775f0e30 0x558c775f32c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f192802c630 tx=0x7f1928046000 comp rx=0 tx=0).stop 2024-03-11T19:10:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.925+0000 7f193a280740 1 -- 172.21.15.29:0/2436845496 shutdown_connections 2024-03-11T19:10:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.925+0000 7f193a280740 1 --2- 172.21.15.29:0/2436845496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775f0e30 0x558c775f32c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.925+0000 7f193a280740 1 --2- 172.21.15.29:0/2436845496 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775ebcc0 0x558c775f07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.925+0000 7f193a280740 1 --2- 172.21.15.29:0/2436845496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775eb2d0 0x558c775eb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.925+0000 7f193a280740 1 -- 172.21.15.29:0/2436845496 >> 172.21.15.29:0/2436845496 conn(0x558c775e67f0 msgr2=0x558c775e6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.925+0000 7f193a280740 1 -- 172.21.15.29:0/2436845496 shutdown_connections 2024-03-11T19:10:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.925+0000 7f193a280740 1 -- 172.21.15.29:0/2436845496 wait complete. 2024-03-11T19:10:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.925+0000 7f193a280740 1 -- start start 2024-03-11T19:10:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.925+0000 7f193a280740 1 Processor -- start 2024-03-11T19:10:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f193a280740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775eb2d0 0x558c775f5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f193a280740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775ebcc0 0x558c775f5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f193a280740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775f0e30 0x558c775fc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:32.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f193a280740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c77227430 con 0x558c775ebcc0 2024-03-11T19:10:32.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f193a280740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c771e3e00 con 0x558c775f0e30 2024-03-11T19:10:32.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f193a280740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c774d8fd0 con 0x558c775eb2d0 2024-03-11T19:10:32.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f193a280740 10 client.0 _ll_register_callbacks cb 0x558c7752efd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:32.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f1934c49640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775ebcc0 0x558c775f5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:32.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f1934c49640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775ebcc0 0x558c775f5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42368/0 (socket says 172.21.15.29:42368) 2024-03-11T19:10:32.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f1934c49640 1 -- 172.21.15.29:0/2265402844 learned_addr learned my addr 172.21.15.29:0/2265402844 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:32.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f1935c4b640 1 --2- 172.21.15.29:0/2265402844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775f0e30 0x558c775fc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:32.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f193544a640 1 --2- 172.21.15.29:0/2265402844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775eb2d0 0x558c775f5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:32.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f1934c49640 1 -- 172.21.15.29:0/2265402844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775eb2d0 msgr2=0x558c775f5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:32.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f1934c49640 1 --2- 172.21.15.29:0/2265402844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775eb2d0 0x558c775f5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f1934c49640 1 -- 172.21.15.29:0/2265402844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775f0e30 msgr2=0x558c775fc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:32.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f1934c49640 1 --2- 172.21.15.29:0/2265402844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775f0e30 0x558c775fc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f1934c49640 1 -- 172.21.15.29:0/2265402844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1928049070 con 0x558c775ebcc0 2024-03-11T19:10:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f193544a640 1 --2- 172.21.15.29:0/2265402844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775eb2d0 0x558c775f5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:10:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f1934c49640 1 --2- 172.21.15.29:0/2265402844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775ebcc0 0x558c775f5760 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f192000bdc0 tx=0x7f1920012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4761f0dfc0934eec server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f1918ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.926+0000 7f1918ff9640 1 -- 172.21.15.29:0/2265402844 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19200108a0 con 0x558c775ebcc0 2024-03-11T19:10:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.927+0000 7f1918ff9640 1 -- 172.21.15.29:0/2265402844 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f1920010a00 con 0x558c775ebcc0 2024-03-11T19:10:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.927+0000 7f193a280740 1 -- 172.21.15.29:0/2265402844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558c772537e0 con 0x558c775ebcc0 2024-03-11T19:10:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.927+0000 7f1918ff9640 1 -- 172.21.15.29:0/2265402844 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1920010bc0 con 0x558c775ebcc0 2024-03-11T19:10:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.927+0000 7f193a280740 10 client.15504 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.927+0000 7f193a280740 1 -- 172.21.15.29:0/2265402844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x558c77552610 con 0x558c775ebcc0 2024-03-11T19:10:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.927+0000 7f1918ff9640 1 -- 172.21.15.29:0/2265402844 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x558c772537e0 con 0x558c775ebcc0 2024-03-11T19:10:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.927+0000 7f1918ff9640 1 -- 172.21.15.29:0/2265402844 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1920024390 con 0x558c775ebcc0 2024-03-11T19:10:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.927+0000 7f193a280740 10 client.15504 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.927+0000 7f193a280740 10 client.15504 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:10:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.927+0000 7f193a280740 20 client.15504 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.927+0000 7f193a280740 1 --2- 172.21.15.29:0/2265402844 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558c776014d0 0x558c776038f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.927+0000 7f193a280740 4 client.15504 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:10:32.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.927+0000 7f193a280740 1 -- 172.21.15.29:0/2265402844 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558c77290720 con 0x558c776014d0 2024-03-11T19:10:32.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.927+0000 7f193544a640 1 --2- 172.21.15.29:0/2265402844 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558c776014d0 0x558c776038f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:32.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.928+0000 7f1918ff9640 10 client.15504 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:32.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.928+0000 7f193544a640 1 --2- 172.21.15.29:0/2265402844 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558c776014d0 0x558c776038f0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9d306937360d10f server_cookie=ab4bb118bf83777a in_seq=0 out_seq=0 2024-03-11T19:10:32.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.928+0000 7f1918ff9640 1 -- 172.21.15.29:0/2265402844 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558c77290720 con 0x558c776014d0 2024-03-11T19:10:32.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.928+0000 7f1918ff9640 10 client.15504 handle_command_reply: tid=0 2024-03-11T19:10:32.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.928+0000 7f193a280740 1 client.15504 shutdown 2024-03-11T19:10:32.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.929+0000 7f193a280740 1 -- 172.21.15.29:0/2265402844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775ebcc0 msgr2=0x558c775f5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:32.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.929+0000 7f193a280740 1 --2- 172.21.15.29:0/2265402844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775ebcc0 0x558c775f5760 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f192000bdc0 tx=0x7f1920012820 comp rx=0 tx=0).stop 2024-03-11T19:10:32.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.929+0000 7f193a280740 1 -- 172.21.15.29:0/2265402844 shutdown_connections 2024-03-11T19:10:32.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.929+0000 7f193a280740 1 --2- 172.21.15.29:0/2265402844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775f0e30 0x558c775fc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.929+0000 7f193a280740 1 --2- 172.21.15.29:0/2265402844 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558c776014d0 0x558c776038f0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.929+0000 7f193a280740 1 --2- 172.21.15.29:0/2265402844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775ebcc0 0x558c775f5760 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.929+0000 7f193a280740 1 --2- 172.21.15.29:0/2265402844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775eb2d0 0x558c775f5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.929+0000 7f193a280740 1 -- 172.21.15.29:0/2265402844 >> 172.21.15.29:0/2265402844 conn(0x558c775e67f0 msgr2=0x558c775e5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:32.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.929+0000 7f1918ff9640 0 client.15504 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:32.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.929+0000 7f193a280740 1 -- 172.21.15.29:0/2265402844 shutdown_connections 2024-03-11T19:10:32.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.929+0000 7f193a280740 1 -- 172.21.15.29:0/2265402844 wait complete. 2024-03-11T19:10:32.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.929+0000 7f193a280740 20 client.15504 trim_cache size 0 max 16384 2024-03-11T19:10:32.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.946+0000 7f193a280740 1 Processor -- start 2024-03-11T19:10:32.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f193a280740 1 -- start start 2024-03-11T19:10:32.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f193a280740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775eb2d0 0x558c775e1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:32.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f193a280740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775ebcc0 0x558c775fc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:32.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f193a280740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775f0e30 0x558c775e6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:32.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f193a280740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c775ffa50 con 0x558c775eb2d0 2024-03-11T19:10:32.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f193a280740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c775e20c0 con 0x558c775f0e30 2024-03-11T19:10:32.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f193a280740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c775ec0a0 con 0x558c775ebcc0 2024-03-11T19:10:32.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f193544a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775eb2d0 0x558c775e1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:32.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f193544a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775eb2d0 0x558c775e1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42382/0 (socket says 172.21.15.29:42382) 2024-03-11T19:10:32.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f193544a640 1 -- 172.21.15.29:0/3238587377 learned_addr learned my addr 172.21.15.29:0/3238587377 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:32.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f1935c4b640 1 --2- 172.21.15.29:0/3238587377 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775f0e30 0x558c775e6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:32.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f193544a640 1 -- 172.21.15.29:0/3238587377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775ebcc0 msgr2=0x558c775fc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:32.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f1934c49640 1 --2- 172.21.15.29:0/3238587377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775ebcc0 0x558c775fc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:32.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f193544a640 1 --2- 172.21.15.29:0/3238587377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775ebcc0 0x558c775fc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f193544a640 1 -- 172.21.15.29:0/3238587377 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775f0e30 msgr2=0x558c775e6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:32.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f193544a640 1 --2- 172.21.15.29:0/3238587377 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775f0e30 0x558c775e6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f193544a640 1 -- 172.21.15.29:0/3238587377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c774eeac0 con 0x558c775eb2d0 2024-03-11T19:10:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f1935c4b640 1 --2- 172.21.15.29:0/3238587377 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775f0e30 0x558c775e6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f1934c49640 1 --2- 172.21.15.29:0/3238587377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775ebcc0 0x558c775fc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.947+0000 7f193544a640 1 --2- 172.21.15.29:0/3238587377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775eb2d0 0x558c775e1b80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f192c005fd0 tx=0x7f192c010e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5bfd7c294e0a386 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.948+0000 7f191affd640 1 -- 172.21.15.29:0/3238587377 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c09e210 con 0x558c775eb2d0 2024-03-11T19:10:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.948+0000 7f191affd640 1 -- 172.21.15.29:0/3238587377 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f192c0a2860 con 0x558c775eb2d0 2024-03-11T19:10:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.948+0000 7f191affd640 1 -- 172.21.15.29:0/3238587377 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c096730 con 0x558c775eb2d0 2024-03-11T19:10:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.948+0000 7f193a280740 1 -- 172.21.15.29:0/3238587377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775eb2d0 msgr2=0x558c775e1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.948+0000 7f193a280740 1 --2- 172.21.15.29:0/3238587377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775eb2d0 0x558c775e1b80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f192c005fd0 tx=0x7f192c010e20 comp rx=0 tx=0).stop 2024-03-11T19:10:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.948+0000 7f193a280740 1 -- 172.21.15.29:0/3238587377 shutdown_connections 2024-03-11T19:10:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.948+0000 7f193a280740 1 --2- 172.21.15.29:0/3238587377 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775f0e30 0x558c775e6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.948+0000 7f193a280740 1 --2- 172.21.15.29:0/3238587377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775ebcc0 0x558c775fc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.948+0000 7f193a280740 1 --2- 172.21.15.29:0/3238587377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775eb2d0 0x558c775e1b80 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.948+0000 7f193a280740 1 -- 172.21.15.29:0/3238587377 >> 172.21.15.29:0/3238587377 conn(0x558c775e67f0 msgr2=0x558c775f05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.948+0000 7f193a280740 1 -- 172.21.15.29:0/3238587377 shutdown_connections 2024-03-11T19:10:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.948+0000 7f193a280740 1 -- 172.21.15.29:0/3238587377 wait complete. 2024-03-11T19:10:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.949+0000 7f193a280740 1 -- start start 2024-03-11T19:10:32.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.949+0000 7f193a280740 1 Processor -- start 2024-03-11T19:10:32.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.949+0000 7f193a280740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775eb2d0 0x558c775fc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:32.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.949+0000 7f193a280740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775ebcc0 0x558c775fa940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:32.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.949+0000 7f193a280740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775f0e30 0x558c775fae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:32.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.949+0000 7f193a280740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c775fcb20 con 0x558c775ebcc0 2024-03-11T19:10:32.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.949+0000 7f193a280740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c775fb3c0 con 0x558c775eb2d0 2024-03-11T19:10:32.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.949+0000 7f193a280740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c775fb580 con 0x558c775f0e30 2024-03-11T19:10:32.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.949+0000 7f193a280740 10 client.0 _ll_register_callbacks cb 0x7f1928037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:32.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.949+0000 7f1934c49640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775ebcc0 0x558c775fa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:32.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.949+0000 7f1934c49640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775ebcc0 0x558c775fa940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42392/0 (socket says 172.21.15.29:42392) 2024-03-11T19:10:32.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.949+0000 7f1934c49640 1 -- 172.21.15.29:0/2454990429 learned_addr learned my addr 172.21.15.29:0/2454990429 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:32.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.949+0000 7f193544a640 1 --2- 172.21.15.29:0/2454990429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775eb2d0 0x558c775fc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:32.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.949+0000 7f1935c4b640 1 --2- 172.21.15.29:0/2454990429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775f0e30 0x558c775fae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:32.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f1934c49640 1 -- 172.21.15.29:0/2454990429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775f0e30 msgr2=0x558c775fae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f1934c49640 1 --2- 172.21.15.29:0/2454990429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775f0e30 0x558c775fae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f1934c49640 1 -- 172.21.15.29:0/2454990429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775eb2d0 msgr2=0x558c775fc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f1934c49640 1 --2- 172.21.15.29:0/2454990429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775eb2d0 0x558c775fc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f1934c49640 1 -- 172.21.15.29:0/2454990429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193019f710 con 0x558c775ebcc0 2024-03-11T19:10:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f193544a640 1 --2- 172.21.15.29:0/2454990429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775eb2d0 0x558c775fc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:10:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f1934c49640 1 --2- 172.21.15.29:0/2454990429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775ebcc0 0x558c775fa940 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f192000bd90 tx=0x7f1920024d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca8e767073cd2e48 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f1918ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f1918ff9640 1 -- 172.21.15.29:0/2454990429 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1920024590 con 0x558c775ebcc0 2024-03-11T19:10:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f1918ff9640 1 -- 172.21.15.29:0/2454990429 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f1920000e40 con 0x558c775ebcc0 2024-03-11T19:10:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f193a280740 1 -- 172.21.15.29:0/2454990429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1928034730 con 0x558c775ebcc0 2024-03-11T19:10:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f1918ff9640 1 -- 172.21.15.29:0/2454990429 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1920005040 con 0x558c775ebcc0 2024-03-11T19:10:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f193a280740 10 client.15510 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f193a280740 1 -- 172.21.15.29:0/2454990429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x558c775146a0 con 0x558c775ebcc0 2024-03-11T19:10:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f1918ff9640 1 -- 172.21.15.29:0/2454990429 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1928034730 con 0x558c775ebcc0 2024-03-11T19:10:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f1918ff9640 1 -- 172.21.15.29:0/2454990429 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1920010bd0 con 0x558c775ebcc0 2024-03-11T19:10:32.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f193a280740 10 client.15510 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:32.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f193a280740 10 client.15510 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:10:32.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f193a280740 20 client.15510 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:32.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.950+0000 7f193a280740 1 --2- 172.21.15.29:0/2454990429 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558c77658f30 0x558c7765b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.951+0000 7f193a280740 4 client.15510 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:10:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.951+0000 7f193a280740 1 -- 172.21.15.29:0/2454990429 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558c772df890 con 0x558c77658f30 2024-03-11T19:10:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.951+0000 7f193544a640 1 --2- 172.21.15.29:0/2454990429 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558c77658f30 0x558c7765b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.951+0000 7f1918ff9640 10 client.15510 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.951+0000 7f193544a640 1 --2- 172.21.15.29:0/2454990429 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558c77658f30 0x558c7765b350 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38bc955a66f13928 server_cookie=5af659835b058e6a in_seq=0 out_seq=0 2024-03-11T19:10:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.952+0000 7f1918ff9640 1 -- 172.21.15.29:0/2454990429 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x558c772df890 con 0x558c77658f30 2024-03-11T19:10:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.952+0000 7f1918ff9640 10 client.15510 handle_command_reply: tid=0 2024-03-11T19:10:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.952+0000 7f193a280740 1 client.15510 shutdown 2024-03-11T19:10:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.952+0000 7f193a280740 1 -- 172.21.15.29:0/2454990429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775ebcc0 msgr2=0x558c775fa940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.952+0000 7f193a280740 1 --2- 172.21.15.29:0/2454990429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775ebcc0 0x558c775fa940 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f192000bd90 tx=0x7f1920024d10 comp rx=0 tx=0).stop 2024-03-11T19:10:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.952+0000 7f193a280740 1 -- 172.21.15.29:0/2454990429 shutdown_connections 2024-03-11T19:10:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.952+0000 7f193a280740 1 --2- 172.21.15.29:0/2454990429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c775f0e30 0x558c775fae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.952+0000 7f193a280740 1 --2- 172.21.15.29:0/2454990429 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558c77658f30 0x558c7765b350 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.952+0000 7f193a280740 1 --2- 172.21.15.29:0/2454990429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c775ebcc0 0x558c775fa940 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.952+0000 7f193a280740 1 --2- 172.21.15.29:0/2454990429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c775eb2d0 0x558c775fc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.952+0000 7f193a280740 1 -- 172.21.15.29:0/2454990429 >> 172.21.15.29:0/2454990429 conn(0x558c775e67f0 msgr2=0x558c775e6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.952+0000 7f1918ff9640 0 client.15510 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.952+0000 7f193a280740 1 -- 172.21.15.29:0/2454990429 shutdown_connections 2024-03-11T19:10:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.952+0000 7f193a280740 1 -- 172.21.15.29:0/2454990429 wait complete. 2024-03-11T19:10:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.952+0000 7f193a280740 20 client.15510 trim_cache size 0 max 16384 2024-03-11T19:10:32.963 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:10:32.963 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:10:32.963 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:10:32.963 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:10:32.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.952+0000 7f19376ae640 1 -- 172.21.15.29:0/4206178272 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f191c0a1b40 msgr2=0x7f191c0a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:32.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.952+0000 7f19376ae640 1 --2- 172.21.15.29:0/4206178272 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f191c0a1b40 0x7f191c0a4000 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1928032fc0 tx=0x7f192802ecc0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.952+0000 7f19376ae640 1 -- 172.21.15.29:0/4206178272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19301011a0 msgr2=0x7f19301970b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:32.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.952+0000 7f19376ae640 1 --2- 172.21.15.29:0/4206178272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19301011a0 0x7f19301970b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f192c00d950 tx=0x7f192c00de10 comp rx=0 tx=0).stop 2024-03-11T19:10:32.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.953+0000 7f19376ae640 1 -- 172.21.15.29:0/4206178272 shutdown_connections 2024-03-11T19:10:32.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.953+0000 7f19376ae640 1 --2- 172.21.15.29:0/4206178272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1930102d50 0x7f193019e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.953+0000 7f19376ae640 1 --2- 172.21.15.29:0/4206178272 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1930102390 0x7f19301975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.953+0000 7f19376ae640 1 --2- 172.21.15.29:0/4206178272 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f191c0a1b40 0x7f191c0a4000 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.953+0000 7f19376ae640 1 --2- 172.21.15.29:0/4206178272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19301011a0 0x7f19301970b0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:32.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.953+0000 7f19376ae640 1 -- 172.21.15.29:0/4206178272 >> 172.21.15.29:0/4206178272 conn(0x7f19300fc730 msgr2=0x7f1930105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:32.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.953+0000 7f19376ae640 1 -- 172.21.15.29:0/4206178272 shutdown_connections 2024-03-11T19:10:32.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:32.953+0000 7f19376ae640 1 -- 172.21.15.29:0/4206178272 wait complete. 2024-03-11T19:10:32.966 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:10:32.967 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:10:33.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.069+0000 7f12b2adf640 1 -- 172.21.15.29:0/2476880999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12ac102d80 msgr2=0x7f12ac109610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.069+0000 7f12b2adf640 1 --2- 172.21.15.29:0/2476880999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12ac102d80 0x7f12ac109610 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f12a4009ca0 tx=0x7f12a402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.069+0000 7f12b2adf640 1 -- 172.21.15.29:0/2476880999 shutdown_connections 2024-03-11T19:10:33.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.069+0000 7f12b2adf640 1 --2- 172.21.15.29:0/2476880999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12ac102d80 0x7f12ac109610 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.069+0000 7f12b2adf640 1 --2- 172.21.15.29:0/2476880999 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12ac1023c0 0x7f12ac102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.069+0000 7f12b2adf640 1 --2- 172.21.15.29:0/2476880999 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12ac1011d0 0x7f12ac1015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.069+0000 7f12b2adf640 1 -- 172.21.15.29:0/2476880999 >> 172.21.15.29:0/2476880999 conn(0x7f12ac0fc720 msgr2=0x7f12ac0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.069+0000 7f12b2adf640 1 -- 172.21.15.29:0/2476880999 shutdown_connections 2024-03-11T19:10:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.069+0000 7f12b2adf640 1 -- 172.21.15.29:0/2476880999 wait complete. 2024-03-11T19:10:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12b2adf640 1 Processor -- start 2024-03-11T19:10:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12b2adf640 1 -- start start 2024-03-11T19:10:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12b2adf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12ac1011d0 0x7f12ac197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12b2adf640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12ac1023c0 0x7f12ac197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12b2adf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12ac102d80 0x7f12ac19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12b2adf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac19ec10 con 0x7f12ac1011d0 2024-03-11T19:10:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12b2adf640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac19ed80 con 0x7f12ac1023c0 2024-03-11T19:10:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12b2adf640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac19ef40 con 0x7f12ac102d80 2024-03-11T19:10:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12b087b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12ac1011d0 0x7f12ac197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12b087b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12ac1011d0 0x7f12ac197120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42406/0 (socket says 172.21.15.29:42406) 2024-03-11T19:10:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12b087b640 1 -- 172.21.15.29:0/4097938894 learned_addr learned my addr 172.21.15.29:0/4097938894 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12abfff640 1 --2- 172.21.15.29:0/4097938894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12ac1023c0 0x7f12ac197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12b107c640 1 --2- 172.21.15.29:0/4097938894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12ac102d80 0x7f12ac19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12b087b640 1 -- 172.21.15.29:0/4097938894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12ac102d80 msgr2=0x7f12ac19e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12b087b640 1 --2- 172.21.15.29:0/4097938894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12ac102d80 0x7f12ac19e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12b087b640 1 -- 172.21.15.29:0/4097938894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12ac1023c0 msgr2=0x7f12ac197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12b087b640 1 --2- 172.21.15.29:0/4097938894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12ac1023c0 0x7f12ac197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12b087b640 1 -- 172.21.15.29:0/4097938894 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a4009900 con 0x7f12ac1011d0 2024-03-11T19:10:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12b107c640 1 --2- 172.21.15.29:0/4097938894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12ac102d80 0x7f12ac19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:10:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.070+0000 7f12b087b640 1 --2- 172.21.15.29:0/4097938894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12ac1011d0 0x7f12ac197120 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f12a000d770 tx=0x7f12a000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.071+0000 7f12a9ffb640 1 -- 172.21.15.29:0/4097938894 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a0009c30 con 0x7f12ac1011d0 2024-03-11T19:10:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.071+0000 7f12b2adf640 1 -- 172.21.15.29:0/4097938894 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12ac19f220 con 0x7f12ac1011d0 2024-03-11T19:10:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.071+0000 7f12a9ffb640 1 -- 172.21.15.29:0/4097938894 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f12a00044d0 con 0x7f12ac1011d0 2024-03-11T19:10:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.071+0000 7f12a9ffb640 1 -- 172.21.15.29:0/4097938894 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a0003dd0 con 0x7f12ac1011d0 2024-03-11T19:10:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.071+0000 7f12b2adf640 1 -- 172.21.15.29:0/4097938894 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12ac19f730 con 0x7f12ac1011d0 2024-03-11T19:10:33.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.072+0000 7f12b2adf640 1 -- 172.21.15.29:0/4097938894 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12ac108560 con 0x7f12ac1011d0 2024-03-11T19:10:33.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.073+0000 7f12a9ffb640 1 -- 172.21.15.29:0/4097938894 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f12a0003f30 con 0x7f12ac1011d0 2024-03-11T19:10:33.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.074+0000 7f12a9ffb640 1 --2- 172.21.15.29:0/4097938894 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f129c0a1ac0 0x7f129c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.074+0000 7f12a9ffb640 1 -- 172.21.15.29:0/4097938894 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f12a00ccef0 con 0x7f12ac1011d0 2024-03-11T19:10:33.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.074+0000 7f12abfff640 1 --2- 172.21.15.29:0/4097938894 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f129c0a1ac0 0x7f129c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.074+0000 7f12abfff640 1 --2- 172.21.15.29:0/4097938894 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f129c0a1ac0 0x7f129c0a3f80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f12ac198630 tx=0x7f1298005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.082+0000 7f12a9ffb640 1 -- 172.21.15.29:0/4097938894 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f12a00960a0 con 0x7f12ac1011d0 2024-03-11T19:10:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.305+0000 7f12b2adf640 1 -- 172.21.15.29:0/4097938894 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12ac02ce40 con 0x7f12ac1011d0 2024-03-11T19:10:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.306+0000 7f12a9ffb640 1 -- 172.21.15.29:0/4097938894 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f12a0017090 con 0x7f12ac1011d0 2024-03-11T19:10:33.309 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:10:33.309 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:10:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:10:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.308+0000 7f12b2adf640 1 -- 172.21.15.29:0/4097938894 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f129c0a1ac0 msgr2=0x7f129c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.308+0000 7f12b2adf640 1 --2- 172.21.15.29:0/4097938894 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f129c0a1ac0 0x7f129c0a3f80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f12ac198630 tx=0x7f1298005f90 comp rx=0 tx=0).stop 2024-03-11T19:10:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.308+0000 7f12b2adf640 1 -- 172.21.15.29:0/4097938894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12ac1011d0 msgr2=0x7f12ac197120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.308+0000 7f12b2adf640 1 --2- 172.21.15.29:0/4097938894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12ac1011d0 0x7f12ac197120 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f12a000d770 tx=0x7f12a000dc30 comp rx=0 tx=0).stop 2024-03-11T19:10:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.308+0000 7f12b2adf640 1 -- 172.21.15.29:0/4097938894 shutdown_connections 2024-03-11T19:10:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.308+0000 7f12b2adf640 1 --2- 172.21.15.29:0/4097938894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12ac102d80 0x7f12ac19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.308+0000 7f12b2adf640 1 --2- 172.21.15.29:0/4097938894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12ac1023c0 0x7f12ac197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.308+0000 7f12b2adf640 1 --2- 172.21.15.29:0/4097938894 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f129c0a1ac0 0x7f129c0a3f80 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.308+0000 7f12b2adf640 1 --2- 172.21.15.29:0/4097938894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12ac1011d0 0x7f12ac197120 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.308+0000 7f12b2adf640 1 -- 172.21.15.29:0/4097938894 >> 172.21.15.29:0/4097938894 conn(0x7f12ac0fc720 msgr2=0x7f12ac105730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.308+0000 7f12b2adf640 1 -- 172.21.15.29:0/4097938894 shutdown_connections 2024-03-11T19:10:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.308+0000 7f12b2adf640 1 -- 172.21.15.29:0/4097938894 wait complete. 2024-03-11T19:10:33.322 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:10:33.322 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:10:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.425+0000 7f6645c55640 1 -- 172.21.15.29:0/3484463715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6640100080 msgr2=0x7f66401004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.425+0000 7f6645c55640 1 --2- 172.21.15.29:0/3484463715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6640100080 0x7f66401004a0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f6630009900 tx=0x7f663002bc30 comp rx=0 tx=0).stop 2024-03-11T19:10:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.425+0000 7f6645c55640 1 -- 172.21.15.29:0/3484463715 shutdown_connections 2024-03-11T19:10:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.425+0000 7f6645c55640 1 --2- 172.21.15.29:0/3484463715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6640101c30 0x7f66401084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.425+0000 7f6645c55640 1 --2- 172.21.15.29:0/3484463715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6640101270 0x7f66401016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.425+0000 7f6645c55640 1 --2- 172.21.15.29:0/3484463715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6640100080 0x7f66401004a0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.425+0000 7f6645c55640 1 -- 172.21.15.29:0/3484463715 >> 172.21.15.29:0/3484463715 conn(0x7f66400fb610 msgr2=0x7f66400fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.425+0000 7f6645c55640 1 -- 172.21.15.29:0/3484463715 shutdown_connections 2024-03-11T19:10:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.425+0000 7f6645c55640 1 -- 172.21.15.29:0/3484463715 wait complete. 2024-03-11T19:10:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.426+0000 7f6645c55640 1 Processor -- start 2024-03-11T19:10:33.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.426+0000 7f6645c55640 1 -- start start 2024-03-11T19:10:33.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.426+0000 7f6645c55640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6640100080 0x7f6640104010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.426+0000 7f6645c55640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6640101270 0x7f6640102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.426+0000 7f6645c55640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6640101c30 0x7f6640102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.426+0000 7f6645c55640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66401032f0 con 0x7f6640101c30 2024-03-11T19:10:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.426+0000 7f6645c55640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6640103460 con 0x7f6640100080 2024-03-11T19:10:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.426+0000 7f6645c55640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66401a1240 con 0x7f6640101270 2024-03-11T19:10:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.427+0000 7f663ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6640101c30 0x7f6640102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.427+0000 7f663ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6640101c30 0x7f6640102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42416/0 (socket says 172.21.15.29:42416) 2024-03-11T19:10:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.427+0000 7f663ffff640 1 -- 172.21.15.29:0/1155919138 learned_addr learned my addr 172.21.15.29:0/1155919138 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.427+0000 7f663f7fe640 1 --2- 172.21.15.29:0/1155919138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6640100080 0x7f6640104010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.427+0000 7f663effd640 1 --2- 172.21.15.29:0/1155919138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6640101270 0x7f6640102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.427+0000 7f663ffff640 1 -- 172.21.15.29:0/1155919138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6640101270 msgr2=0x7f6640102650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.427+0000 7f663ffff640 1 --2- 172.21.15.29:0/1155919138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6640101270 0x7f6640102650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.427+0000 7f663ffff640 1 -- 172.21.15.29:0/1155919138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6640100080 msgr2=0x7f6640104010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.427+0000 7f663ffff640 1 --2- 172.21.15.29:0/1155919138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6640100080 0x7f6640104010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.427+0000 7f663ffff640 1 -- 172.21.15.29:0/1155919138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6630009560 con 0x7f6640101c30 2024-03-11T19:10:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.427+0000 7f663effd640 1 --2- 172.21.15.29:0/1155919138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6640101270 0x7f6640102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:10:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.427+0000 7f663f7fe640 1 --2- 172.21.15.29:0/1155919138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6640100080 0x7f6640104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:10:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.427+0000 7f663ffff640 1 --2- 172.21.15.29:0/1155919138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6640101c30 0x7f6640102b90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f663800ead0 tx=0x7f663800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.427+0000 7f663cff9640 1 -- 172.21.15.29:0/1155919138 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663800cde0 con 0x7f6640101c30 2024-03-11T19:10:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.427+0000 7f663cff9640 1 -- 172.21.15.29:0/1155919138 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6638005020 con 0x7f6640101c30 2024-03-11T19:10:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.427+0000 7f6645c55640 1 -- 172.21.15.29:0/1155919138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66401a14f0 con 0x7f6640101c30 2024-03-11T19:10:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.427+0000 7f663cff9640 1 -- 172.21.15.29:0/1155919138 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66380116b0 con 0x7f6640101c30 2024-03-11T19:10:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.427+0000 7f6645c55640 1 -- 172.21.15.29:0/1155919138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66401a19b0 con 0x7f6640101c30 2024-03-11T19:10:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.430+0000 7f664882e740 1 Processor -- start 2024-03-11T19:10:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.430+0000 7f663cff9640 1 -- 172.21.15.29:0/1155919138 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f6638005190 con 0x7f6640101c30 2024-03-11T19:10:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.430+0000 7f664882e740 1 -- start start 2024-03-11T19:10:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f664882e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53270 0x55db3cd53690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f664882e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd53c60 0x55db3cd58760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f664882e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd58dd0 0x55db3cd5b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f664882e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3cd0f960 con 0x55db3cd53c60 2024-03-11T19:10:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f664882e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3ccc3310 con 0x55db3cd58dd0 2024-03-11T19:10:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f664882e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3ccc0840 con 0x55db3cd53270 2024-03-11T19:10:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f663f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd53c60 0x55db3cd58760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f663f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd53c60 0x55db3cd58760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42432/0 (socket says 172.21.15.29:42432) 2024-03-11T19:10:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f663f7fe640 1 -- 172.21.15.29:0/3883016150 learned_addr learned my addr 172.21.15.29:0/3883016150 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f663effd640 1 --2- 172.21.15.29:0/3883016150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd58dd0 0x55db3cd5b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f663cff9640 1 --2- 172.21.15.29:0/1155919138 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f66280a1ac0 0x7f66280a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f663effd640 1 --2- 172.21.15.29:0/3883016150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53270 0x55db3cd53690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f663f7fe640 1 -- 172.21.15.29:0/3883016150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53270 msgr2=0x55db3cd53690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f663f7fe640 1 --2- 172.21.15.29:0/3883016150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53270 0x55db3cd53690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f663f7fe640 1 -- 172.21.15.29:0/3883016150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd58dd0 msgr2=0x55db3cd5b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f663f7fe640 1 --2- 172.21.15.29:0/3883016150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd58dd0 0x55db3cd5b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f663f7fe640 1 -- 172.21.15.29:0/3883016150 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db3cc7e060 con 0x55db3cd53c60 2024-03-11T19:10:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f663ffff640 1 --2- 172.21.15.29:0/1155919138 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f66280a1ac0 0x7f66280a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f663cff9640 1 -- 172.21.15.29:0/1155919138 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f66380cd710 con 0x7f6640101c30 2024-03-11T19:10:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.431+0000 7f663f7fe640 1 --2- 172.21.15.29:0/3883016150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd53c60 0x55db3cd58760 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6630000c00 tx=0x7f663002c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e3067b29f42d0d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.432+0000 7f6625ffb640 1 -- 172.21.15.29:0/3883016150 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66300046c0 con 0x55db3cd53c60 2024-03-11T19:10:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.432+0000 7f6625ffb640 1 -- 172.21.15.29:0/3883016150 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f663002de90 con 0x55db3cd53c60 2024-03-11T19:10:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.432+0000 7f6625ffb640 1 -- 172.21.15.29:0/3883016150 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630003b30 con 0x55db3cd53c60 2024-03-11T19:10:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.432+0000 7f663ffff640 1 --2- 172.21.15.29:0/1155919138 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f66280a1ac0 0x7f66280a3f80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f663809a5b0 tx=0x7f66380cda20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.432+0000 7f664882e740 1 -- 172.21.15.29:0/3883016150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd53c60 msgr2=0x55db3cd58760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.432+0000 7f664882e740 1 --2- 172.21.15.29:0/3883016150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd53c60 0x55db3cd58760 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6630000c00 tx=0x7f663002c870 comp rx=0 tx=0).stop 2024-03-11T19:10:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.432+0000 7f664882e740 1 -- 172.21.15.29:0/3883016150 shutdown_connections 2024-03-11T19:10:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.432+0000 7f664882e740 1 --2- 172.21.15.29:0/3883016150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd58dd0 0x55db3cd5b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.432+0000 7f664882e740 1 --2- 172.21.15.29:0/3883016150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd53c60 0x55db3cd58760 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.432+0000 7f664882e740 1 --2- 172.21.15.29:0/3883016150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53270 0x55db3cd53690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.432+0000 7f664882e740 1 -- 172.21.15.29:0/3883016150 >> 172.21.15.29:0/3883016150 conn(0x55db3cd4e7e0 msgr2=0x55db3cd4ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.432+0000 7f664882e740 1 -- 172.21.15.29:0/3883016150 shutdown_connections 2024-03-11T19:10:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.432+0000 7f664882e740 1 -- 172.21.15.29:0/3883016150 wait complete. 2024-03-11T19:10:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.433+0000 7f664882e740 1 -- start start 2024-03-11T19:10:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.433+0000 7f664882e740 1 Processor -- start 2024-03-11T19:10:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.433+0000 7f664882e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53270 0x55db3cd5d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.433+0000 7f664882e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd53c60 0x55db3cd5d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.433+0000 7f664882e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd58dd0 0x55db3cd64970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.433+0000 7f664882e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3ccbc020 con 0x55db3cd58dd0 2024-03-11T19:10:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.433+0000 7f664882e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3c98f430 con 0x55db3cd53c60 2024-03-11T19:10:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.433+0000 7f664882e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3c94be00 con 0x55db3cd53270 2024-03-11T19:10:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.433+0000 7f664882e740 10 client.0 _ll_register_callbacks cb 0x55db3cc63cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.433+0000 7f663effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd58dd0 0x55db3cd64970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.433+0000 7f663effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd58dd0 0x55db3cd64970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42434/0 (socket says 172.21.15.29:42434) 2024-03-11T19:10:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.433+0000 7f663effd640 1 -- 172.21.15.29:0/2333984410 learned_addr learned my addr 172.21.15.29:0/2333984410 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.433+0000 7f663f7fe640 1 --2- 172.21.15.29:0/2333984410 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd53c60 0x55db3cd5d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.434+0000 7f663effd640 1 --2- 172.21.15.29:0/2333984410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53270 0x55db3cd5d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.434+0000 7f663effd640 1 -- 172.21.15.29:0/2333984410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53270 msgr2=0x55db3cd5d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.434+0000 7f663effd640 1 --2- 172.21.15.29:0/2333984410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53270 0x55db3cd5d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.434+0000 7f663effd640 1 -- 172.21.15.29:0/2333984410 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd53c60 msgr2=0x55db3cd5d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.434+0000 7f663effd640 1 --2- 172.21.15.29:0/2333984410 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd53c60 0x55db3cd5d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.434+0000 7f663effd640 1 -- 172.21.15.29:0/2333984410 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6640108dc0 con 0x55db3cd58dd0 2024-03-11T19:10:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.434+0000 7f663effd640 1 --2- 172.21.15.29:0/2333984410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd58dd0 0x55db3cd64970 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6634012de0 tx=0x7f66340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596b5e01ca2594e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.434+0000 7f661effd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.434+0000 7f661effd640 1 -- 172.21.15.29:0/2333984410 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6634012040 con 0x55db3cd58dd0 2024-03-11T19:10:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.434+0000 7f661effd640 1 -- 172.21.15.29:0/2333984410 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f66340121a0 con 0x55db3cd58dd0 2024-03-11T19:10:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.434+0000 7f664882e740 1 -- 172.21.15.29:0/2333984410 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55db3c9bb7e0 con 0x55db3cd58dd0 2024-03-11T19:10:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.434+0000 7f661effd640 1 -- 172.21.15.29:0/2333984410 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6634004020 con 0x55db3cd58dd0 2024-03-11T19:10:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.434+0000 7f664882e740 10 client.15528 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.434+0000 7f664882e740 1 -- 172.21.15.29:0/2333984410 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55db3ccba6a0 con 0x55db3cd58dd0 2024-03-11T19:10:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.434+0000 7f661effd640 1 -- 172.21.15.29:0/2333984410 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55db3c9bb7e0 con 0x55db3cd58dd0 2024-03-11T19:10:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.435+0000 7f661effd640 1 -- 172.21.15.29:0/2333984410 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f663401f030 con 0x55db3cd58dd0 2024-03-11T19:10:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.435+0000 7f664882e740 10 client.15528 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.435+0000 7f664882e740 10 client.15528 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:10:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.435+0000 7f664882e740 20 client.15528 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.435+0000 7f664882e740 1 --2- 172.21.15.29:0/2333984410 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55db3cd695f0 0x55db3cd6ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.435+0000 7f664882e740 4 client.15528 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:10:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.435+0000 7f664882e740 1 -- 172.21.15.29:0/2333984410 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55db3c9f8720 con 0x55db3cd695f0 2024-03-11T19:10:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.435+0000 7f663f7fe640 1 --2- 172.21.15.29:0/2333984410 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55db3cd695f0 0x55db3cd6ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.435+0000 7f661effd640 10 client.15528 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:10:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.435+0000 7f663f7fe640 1 --2- 172.21.15.29:0/2333984410 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55db3cd695f0 0x55db3cd6ba10 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c248de1572b848d8 server_cookie=1f3ec975a93dffd0 in_seq=0 out_seq=0 2024-03-11T19:10:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.436+0000 7f661effd640 1 -- 172.21.15.29:0/2333984410 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55db3c9f8720 con 0x55db3cd695f0 2024-03-11T19:10:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.436+0000 7f661effd640 10 client.15528 handle_command_reply: tid=0 2024-03-11T19:10:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.436+0000 7f664882e740 1 client.15528 shutdown 2024-03-11T19:10:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.436+0000 7f664882e740 1 -- 172.21.15.29:0/2333984410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd58dd0 msgr2=0x55db3cd64970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.436+0000 7f664882e740 1 --2- 172.21.15.29:0/2333984410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd58dd0 0x55db3cd64970 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6634012de0 tx=0x7f66340046c0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.436+0000 7f664882e740 1 -- 172.21.15.29:0/2333984410 shutdown_connections 2024-03-11T19:10:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.436+0000 7f664882e740 1 --2- 172.21.15.29:0/2333984410 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55db3cd695f0 0x55db3cd6ba10 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.436+0000 7f664882e740 1 --2- 172.21.15.29:0/2333984410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd58dd0 0x55db3cd64970 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.436+0000 7f664882e740 1 --2- 172.21.15.29:0/2333984410 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd53c60 0x55db3cd5d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.436+0000 7f661effd640 0 client.15528 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:10:33.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.436+0000 7f664882e740 1 --2- 172.21.15.29:0/2333984410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53270 0x55db3cd5d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.436+0000 7f664882e740 1 -- 172.21.15.29:0/2333984410 >> 172.21.15.29:0/2333984410 conn(0x55db3cd4e7e0 msgr2=0x55db3cd4db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.436+0000 7f664882e740 1 -- 172.21.15.29:0/2333984410 shutdown_connections 2024-03-11T19:10:33.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.436+0000 7f664882e740 1 -- 172.21.15.29:0/2333984410 wait complete. 2024-03-11T19:10:33.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.436+0000 7f664882e740 20 client.15528 trim_cache size 0 max 16384 2024-03-11T19:10:33.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.449+0000 7f664882e740 1 Processor -- start 2024-03-11T19:10:33.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.449+0000 7f664882e740 1 -- start start 2024-03-11T19:10:33.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.449+0000 7f664882e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53270 0x55db3cd4ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.449+0000 7f664882e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd53c60 0x55db3cd64930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.449+0000 7f664882e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd58dd0 0x55db3cd551b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.449+0000 7f664882e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3cc41130 con 0x55db3cd58dd0 2024-03-11T19:10:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.449+0000 7f664882e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3c9f8720 con 0x55db3cd53c60 2024-03-11T19:10:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.449+0000 7f664882e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3cd5b720 con 0x55db3cd53270 2024-03-11T19:10:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.449+0000 7f663effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd58dd0 0x55db3cd551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.449+0000 7f663f7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd53c60 0x55db3cd64930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.449+0000 7f663f7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd53c60 0x55db3cd64930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:36338/0 (socket says 172.21.15.29:36338) 2024-03-11T19:10:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.449+0000 7f663effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd58dd0 0x55db3cd551b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42448/0 (socket says 172.21.15.29:42448) 2024-03-11T19:10:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.449+0000 7f663f7fe640 1 -- 172.21.15.29:0/3923446781 learned_addr learned my addr 172.21.15.29:0/3923446781 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.449+0000 7f663effd640 1 --2- 172.21.15.29:0/3923446781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53270 0x55db3cd4ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.450+0000 7f663f7fe640 1 -- 172.21.15.29:0/3923446781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53270 msgr2=0x55db3cd4ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.450+0000 7f663f7fe640 1 --2- 172.21.15.29:0/3923446781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53270 0x55db3cd4ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.450+0000 7f663f7fe640 1 -- 172.21.15.29:0/3923446781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd58dd0 msgr2=0x55db3cd551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.450+0000 7f663f7fe640 1 --2- 172.21.15.29:0/3923446781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd58dd0 0x55db3cd551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.450+0000 7f663f7fe640 1 -- 172.21.15.29:0/3923446781 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db3cc56c20 con 0x55db3cd53c60 2024-03-11T19:10:33.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.450+0000 7f663effd640 1 --2- 172.21.15.29:0/3923446781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd58dd0 0x55db3cd551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:10:33.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.450+0000 7f663f7fe640 1 --2- 172.21.15.29:0/3923446781 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd53c60 0x55db3cd64930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f663002c170 tx=0x7f6630040040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d36006173cf567b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.450+0000 7f66257fa640 1 -- 172.21.15.29:0/3923446781 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630054070 con 0x55db3cd53c60 2024-03-11T19:10:33.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.450+0000 7f66257fa640 1 -- 172.21.15.29:0/3923446781 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f66300044e0 con 0x55db3cd53c60 2024-03-11T19:10:33.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.450+0000 7f66257fa640 1 -- 172.21.15.29:0/3923446781 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630050340 con 0x55db3cd53c60 2024-03-11T19:10:33.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.451+0000 7f664882e740 1 -- 172.21.15.29:0/3923446781 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd53c60 msgr2=0x55db3cd64930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.451+0000 7f664882e740 1 --2- 172.21.15.29:0/3923446781 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd53c60 0x55db3cd64930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f663002c170 tx=0x7f6630040040 comp rx=0 tx=0).stop 2024-03-11T19:10:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.451+0000 7f664882e740 1 -- 172.21.15.29:0/3923446781 shutdown_connections 2024-03-11T19:10:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.451+0000 7f664882e740 1 --2- 172.21.15.29:0/3923446781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd58dd0 0x55db3cd551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.451+0000 7f664882e740 1 --2- 172.21.15.29:0/3923446781 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd53c60 0x55db3cd64930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.451+0000 7f664882e740 1 --2- 172.21.15.29:0/3923446781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53270 0x55db3cd4ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.451+0000 7f664882e740 1 -- 172.21.15.29:0/3923446781 >> 172.21.15.29:0/3923446781 conn(0x55db3cd4e7e0 msgr2=0x55db3cd49a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.451+0000 7f664882e740 1 -- 172.21.15.29:0/3923446781 shutdown_connections 2024-03-11T19:10:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.451+0000 7f664882e740 1 -- 172.21.15.29:0/3923446781 wait complete. 2024-03-11T19:10:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.451+0000 7f664882e740 1 -- start start 2024-03-11T19:10:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.452+0000 7f664882e740 1 Processor -- start 2024-03-11T19:10:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.452+0000 7f664882e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd53270 0x55db3cd62680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.452+0000 7f664882e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53c60 0x55db3cd62bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.452+0000 7f664882e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd58dd0 0x55db3cd63bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.452+0000 7f664882e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3cd64110 con 0x55db3cd53270 2024-03-11T19:10:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.452+0000 7f664882e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3cd64280 con 0x55db3cd58dd0 2024-03-11T19:10:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.452+0000 7f664882e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3cd64440 con 0x55db3cd53c60 2024-03-11T19:10:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.452+0000 7f664882e740 10 client.0 _ll_register_callbacks cb 0x7f6634005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.452+0000 7f663effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd53270 0x55db3cd62680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.452+0000 7f663effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd53270 0x55db3cd62680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42464/0 (socket says 172.21.15.29:42464) 2024-03-11T19:10:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.452+0000 7f663effd640 1 -- 172.21.15.29:0/4118159469 learned_addr learned my addr 172.21.15.29:0/4118159469 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.452+0000 7f663effd640 1 --2- 172.21.15.29:0/4118159469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd58dd0 0x55db3cd63bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.452+0000 7f663f7fe640 1 --2- 172.21.15.29:0/4118159469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53c60 0x55db3cd62bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.453+0000 7f663effd640 1 -- 172.21.15.29:0/4118159469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53c60 msgr2=0x55db3cd62bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.453+0000 7f663effd640 1 --2- 172.21.15.29:0/4118159469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53c60 0x55db3cd62bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.453+0000 7f663effd640 1 -- 172.21.15.29:0/4118159469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd58dd0 msgr2=0x55db3cd63bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.453+0000 7f663effd640 1 --2- 172.21.15.29:0/4118159469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd58dd0 0x55db3cd63bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.453+0000 7f663effd640 1 -- 172.21.15.29:0/4118159469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6634034040 con 0x55db3cd53270 2024-03-11T19:10:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.453+0000 7f663f7fe640 1 --2- 172.21.15.29:0/4118159469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53c60 0x55db3cd62bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:10:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.453+0000 7f663effd640 1 --2- 172.21.15.29:0/4118159469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd53270 0x55db3cd62680 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6634011010 tx=0x7f663402e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b114df42de60a30 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.453+0000 7f661effd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.453+0000 7f661effd640 1 -- 172.21.15.29:0/4118159469 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6634031070 con 0x55db3cd53270 2024-03-11T19:10:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.453+0000 7f661effd640 1 -- 172.21.15.29:0/4118159469 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f6634000e40 con 0x55db3cd53270 2024-03-11T19:10:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.453+0000 7f664882e740 1 -- 172.21.15.29:0/4118159469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f663002de90 con 0x55db3cd53270 2024-03-11T19:10:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.453+0000 7f661effd640 1 -- 172.21.15.29:0/4118159469 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6634005040 con 0x55db3cd53270 2024-03-11T19:10:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.453+0000 7f664882e740 10 client.15534 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.453+0000 7f664882e740 1 -- 172.21.15.29:0/4118159469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55db3cc7e060 con 0x55db3cd53270 2024-03-11T19:10:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.453+0000 7f661effd640 1 -- 172.21.15.29:0/4118159469 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6634003e70 con 0x55db3cd53270 2024-03-11T19:10:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.454+0000 7f661effd640 1 -- 172.21.15.29:0/4118159469 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f663401f040 con 0x55db3cd53270 2024-03-11T19:10:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.454+0000 7f664882e740 10 client.15534 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.454+0000 7f664882e740 10 client.15534 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:10:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.454+0000 7f664882e740 20 client.15534 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.454+0000 7f664882e740 1 --2- 172.21.15.29:0/4118159469 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55db3cdc0a50 0x55db3cdc2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.454+0000 7f664882e740 4 client.15534 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:10:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.454+0000 7f664882e740 1 -- 172.21.15.29:0/4118159469 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f663002de90 con 0x55db3cdc0a50 2024-03-11T19:10:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.454+0000 7f663f7fe640 1 --2- 172.21.15.29:0/4118159469 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55db3cdc0a50 0x55db3cdc2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.454+0000 7f663f7fe640 1 --2- 172.21.15.29:0/4118159469 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55db3cdc0a50 0x55db3cdc2e70 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c80fa1a1d43e7fe server_cookie=da3a0181c38d65ea in_seq=0 out_seq=0 2024-03-11T19:10:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.454+0000 7f661effd640 10 client.15534 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:10:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.455+0000 7f661effd640 1 -- 172.21.15.29:0/4118159469 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f663002de90 con 0x55db3cdc0a50 2024-03-11T19:10:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.455+0000 7f661effd640 10 client.15534 handle_command_reply: tid=0 2024-03-11T19:10:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.455+0000 7f664882e740 1 client.15534 shutdown 2024-03-11T19:10:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.455+0000 7f664882e740 1 -- 172.21.15.29:0/4118159469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd53270 msgr2=0x55db3cd62680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.455+0000 7f664882e740 1 --2- 172.21.15.29:0/4118159469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd53270 0x55db3cd62680 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6634011010 tx=0x7f663402e040 comp rx=0 tx=0).stop 2024-03-11T19:10:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.455+0000 7f664882e740 1 -- 172.21.15.29:0/4118159469 shutdown_connections 2024-03-11T19:10:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.455+0000 7f664882e740 1 --2- 172.21.15.29:0/4118159469 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55db3cdc0a50 0x55db3cdc2e70 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.455+0000 7f664882e740 1 --2- 172.21.15.29:0/4118159469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55db3cd58dd0 0x55db3cd63bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.455+0000 7f664882e740 1 --2- 172.21.15.29:0/4118159469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55db3cd53c60 0x55db3cd62bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.455+0000 7f661effd640 0 client.15534 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:10:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.455+0000 7f664882e740 1 --2- 172.21.15.29:0/4118159469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55db3cd53270 0x55db3cd62680 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.455+0000 7f664882e740 1 -- 172.21.15.29:0/4118159469 >> 172.21.15.29:0/4118159469 conn(0x55db3cd4e7e0 msgr2=0x55db3cd4ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.455+0000 7f664882e740 1 -- 172.21.15.29:0/4118159469 shutdown_connections 2024-03-11T19:10:33.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.455+0000 7f664882e740 1 -- 172.21.15.29:0/4118159469 wait complete. 2024-03-11T19:10:33.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.455+0000 7f664882e740 20 client.15534 trim_cache size 0 max 16384 2024-03-11T19:10:33.464 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:10:33.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.455+0000 7f6645c55640 1 -- 172.21.15.29:0/1155919138 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f66280a1ac0 msgr2=0x7f66280a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.455+0000 7f6645c55640 1 --2- 172.21.15.29:0/1155919138 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f66280a1ac0 0x7f66280a3f80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f663809a5b0 tx=0x7f66380cda20 comp rx=0 tx=0).stop 2024-03-11T19:10:33.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.455+0000 7f6645c55640 1 -- 172.21.15.29:0/1155919138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6640101c30 msgr2=0x7f6640102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.455+0000 7f6645c55640 1 --2- 172.21.15.29:0/1155919138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6640101c30 0x7f6640102b90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f663800ead0 tx=0x7f663800ef90 comp rx=0 tx=0).stop 2024-03-11T19:10:33.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.456+0000 7f6645c55640 1 -- 172.21.15.29:0/1155919138 shutdown_connections 2024-03-11T19:10:33.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.456+0000 7f6645c55640 1 --2- 172.21.15.29:0/1155919138 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f66280a1ac0 0x7f66280a3f80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.456+0000 7f6645c55640 1 --2- 172.21.15.29:0/1155919138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6640101c30 0x7f6640102b90 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.456+0000 7f6645c55640 1 --2- 172.21.15.29:0/1155919138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6640101270 0x7f6640102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.456+0000 7f6645c55640 1 --2- 172.21.15.29:0/1155919138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6640100080 0x7f6640104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.456+0000 7f6645c55640 1 -- 172.21.15.29:0/1155919138 >> 172.21.15.29:0/1155919138 conn(0x7f66400fb610 msgr2=0x7f6640068390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.456+0000 7f6645c55640 1 -- 172.21.15.29:0/1155919138 shutdown_connections 2024-03-11T19:10:33.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.456+0000 7f6645c55640 1 -- 172.21.15.29:0/1155919138 wait complete. 2024-03-11T19:10:33.469 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:10:33.470 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:10:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.580+0000 7f543ee2e640 1 -- 172.21.15.29:0/2205054778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5438106f20 msgr2=0x7f5438109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.580+0000 7f543ee2e640 1 --2- 172.21.15.29:0/2205054778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5438106f20 0x7f5438109310 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f5430009ca0 tx=0x7f543002c0f0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.580+0000 7f543ee2e640 1 -- 172.21.15.29:0/2205054778 shutdown_connections 2024-03-11T19:10:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.580+0000 7f543ee2e640 1 --2- 172.21.15.29:0/2205054778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5438106f20 0x7f5438109310 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.580+0000 7f543ee2e640 1 --2- 172.21.15.29:0/2205054778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54381045f0 0x7f54381069e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.580+0000 7f543ee2e640 1 --2- 172.21.15.29:0/2205054778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5438101cc0 0x7f54381040b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.580+0000 7f543ee2e640 1 -- 172.21.15.29:0/2205054778 >> 172.21.15.29:0/2205054778 conn(0x7f54380fb610 msgr2=0x7f54380fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.580+0000 7f543ee2e640 1 -- 172.21.15.29:0/2205054778 shutdown_connections 2024-03-11T19:10:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.580+0000 7f543ee2e640 1 -- 172.21.15.29:0/2205054778 wait complete. 2024-03-11T19:10:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.581+0000 7f543ee2e640 1 Processor -- start 2024-03-11T19:10:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.581+0000 7f543ee2e640 1 -- start start 2024-03-11T19:10:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.581+0000 7f543ee2e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5438101cc0 0x7f5438101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.581+0000 7f543ee2e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54381045f0 0x7f54380ffc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.581+0000 7f543ee2e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5438106f20 0x7f54381001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.581+0000 7f543ee2e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5438100910 con 0x7f54381045f0 2024-03-11T19:10:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.581+0000 7f543ee2e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5438100a80 con 0x7f5438106f20 2024-03-11T19:10:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.581+0000 7f543ee2e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54381a11e0 con 0x7f5438101cc0 2024-03-11T19:10:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.581+0000 7f5437fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54381045f0 0x7f54380ffc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.581+0000 7f5437fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54381045f0 0x7f54380ffc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42482/0 (socket says 172.21.15.29:42482) 2024-03-11T19:10:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.581+0000 7f5437fff640 1 -- 172.21.15.29:0/3075914379 learned_addr learned my addr 172.21.15.29:0/3075914379 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.581+0000 7f543cbca640 1 --2- 172.21.15.29:0/3075914379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5438101cc0 0x7f5438101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.581+0000 7f543d3cb640 1 --2- 172.21.15.29:0/3075914379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5438106f20 0x7f54381001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.582+0000 7f5437fff640 1 -- 172.21.15.29:0/3075914379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5438101cc0 msgr2=0x7f5438101630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.582+0000 7f5437fff640 1 --2- 172.21.15.29:0/3075914379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5438101cc0 0x7f5438101630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.582+0000 7f5437fff640 1 -- 172.21.15.29:0/3075914379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5438106f20 msgr2=0x7f54381001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.582+0000 7f5437fff640 1 --2- 172.21.15.29:0/3075914379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5438106f20 0x7f54381001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.582+0000 7f5437fff640 1 -- 172.21.15.29:0/3075914379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5430009900 con 0x7f54381045f0 2024-03-11T19:10:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.582+0000 7f543cbca640 1 --2- 172.21.15.29:0/3075914379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5438101cc0 0x7f5438101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:10:33.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.582+0000 7f5437fff640 1 --2- 172.21.15.29:0/3075914379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54381045f0 0x7f54380ffc70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f542400ea20 tx=0x7f542400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.582+0000 7f5435ffb640 1 -- 172.21.15.29:0/3075914379 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54240096c0 con 0x7f54381045f0 2024-03-11T19:10:33.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.582+0000 7f543ee2e640 1 -- 172.21.15.29:0/3075914379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54381a1490 con 0x7f54381045f0 2024-03-11T19:10:33.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.582+0000 7f5435ffb640 1 -- 172.21.15.29:0/3075914379 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f542400cd20 con 0x7f54381045f0 2024-03-11T19:10:33.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.582+0000 7f5435ffb640 1 -- 172.21.15.29:0/3075914379 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5424010430 con 0x7f54381045f0 2024-03-11T19:10:33.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.582+0000 7f543ee2e640 1 -- 172.21.15.29:0/3075914379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54381a1950 con 0x7f54381045f0 2024-03-11T19:10:33.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.584+0000 7f5435ffb640 1 -- 172.21.15.29:0/3075914379 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f5424004f70 con 0x7f54381045f0 2024-03-11T19:10:33.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.585+0000 7f5435ffb640 1 --2- 172.21.15.29:0/3075914379 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f54280a1ac0 0x7f54280a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.585+0000 7f5441a01740 1 Processor -- start 2024-03-11T19:10:33.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.585+0000 7f543cbca640 1 --2- 172.21.15.29:0/3075914379 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f54280a1ac0 0x7f54280a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.585+0000 7f5441a01740 1 -- start start 2024-03-11T19:10:33.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.585+0000 7f5435ffb640 1 -- 172.21.15.29:0/3075914379 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5424014070 con 0x7f54381045f0 2024-03-11T19:10:33.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.585+0000 7f5441a01740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6a5270 0x562d2c6a5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.585+0000 7f5441a01740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 0x562d2c6aa760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.585+0000 7f5441a01740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6aadd0 0x562d2c6ad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.585+0000 7f5441a01740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2c661960 con 0x562d2c6a5c60 2024-03-11T19:10:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.585+0000 7f5441a01740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2c615310 con 0x562d2c6aadd0 2024-03-11T19:10:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.585+0000 7f5441a01740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2c612840 con 0x562d2c6a5270 2024-03-11T19:10:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.586+0000 7f5437fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 0x562d2c6aa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.586+0000 7f5437fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 0x562d2c6aa760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42498/0 (socket says 172.21.15.29:42498) 2024-03-11T19:10:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.586+0000 7f5437fff640 1 -- 172.21.15.29:0/3658911673 learned_addr learned my addr 172.21.15.29:0/3658911673 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.586+0000 7f543cbca640 1 --2- 172.21.15.29:0/3075914379 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f54280a1ac0 0x7f54280a3f80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f542c009c20 tx=0x7f542c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.586+0000 7f543cbca640 1 --2- 172.21.15.29:0/3658911673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6a5270 0x562d2c6a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.586+0000 7f543d3cb640 1 --2- 172.21.15.29:0/3658911673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6aadd0 0x562d2c6ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.586+0000 7f5437fff640 1 -- 172.21.15.29:0/3658911673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6a5270 msgr2=0x562d2c6a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.586+0000 7f5437fff640 1 --2- 172.21.15.29:0/3658911673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6a5270 0x562d2c6a5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.586+0000 7f5437fff640 1 -- 172.21.15.29:0/3658911673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6aadd0 msgr2=0x562d2c6ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.586+0000 7f5437fff640 1 --2- 172.21.15.29:0/3658911673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6aadd0 0x562d2c6ad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.586+0000 7f5437fff640 1 -- 172.21.15.29:0/3658911673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562d2c5d0060 con 0x562d2c6a5c60 2024-03-11T19:10:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.586+0000 7f543d3cb640 1 --2- 172.21.15.29:0/3658911673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6aadd0 0x562d2c6ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.586+0000 7f5437fff640 1 --2- 172.21.15.29:0/3658911673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 0x562d2c6aa760 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5424009380 tx=0x7f54240964d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d19ce6d8e17f2d3a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.586+0000 7f5422ffd640 1 -- 172.21.15.29:0/3658911673 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542409a830 con 0x562d2c6a5c60 2024-03-11T19:10:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.586+0000 7f5422ffd640 1 -- 172.21.15.29:0/3658911673 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f542400ce90 con 0x562d2c6a5c60 2024-03-11T19:10:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.586+0000 7f5422ffd640 1 -- 172.21.15.29:0/3658911673 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5424005220 con 0x562d2c6a5c60 2024-03-11T19:10:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.587+0000 7f5441a01740 1 -- 172.21.15.29:0/3658911673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 msgr2=0x562d2c6aa760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.587+0000 7f5441a01740 1 --2- 172.21.15.29:0/3658911673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 0x562d2c6aa760 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5424009380 tx=0x7f54240964d0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.587+0000 7f5441a01740 1 -- 172.21.15.29:0/3658911673 shutdown_connections 2024-03-11T19:10:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.587+0000 7f5441a01740 1 --2- 172.21.15.29:0/3658911673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6aadd0 0x562d2c6ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.587+0000 7f5441a01740 1 --2- 172.21.15.29:0/3658911673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 0x562d2c6aa760 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.587+0000 7f5441a01740 1 --2- 172.21.15.29:0/3658911673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6a5270 0x562d2c6a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.587+0000 7f5441a01740 1 -- 172.21.15.29:0/3658911673 >> 172.21.15.29:0/3658911673 conn(0x562d2c6a07e0 msgr2=0x562d2c6a0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.587+0000 7f5441a01740 1 -- 172.21.15.29:0/3658911673 shutdown_connections 2024-03-11T19:10:33.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.587+0000 7f5441a01740 1 -- 172.21.15.29:0/3658911673 wait complete. 2024-03-11T19:10:33.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.587+0000 7f5441a01740 1 -- start start 2024-03-11T19:10:33.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.588+0000 7f5441a01740 1 Processor -- start 2024-03-11T19:10:33.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.588+0000 7f5441a01740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6a5270 0x562d2c6af400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.588+0000 7f5441a01740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 0x562d2c6af940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.588+0000 7f5441a01740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6aadd0 0x562d2c6b6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.588+0000 7f5441a01740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2c60e020 con 0x562d2c6a5c60 2024-03-11T19:10:33.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.588+0000 7f5441a01740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2c2e1430 con 0x562d2c6aadd0 2024-03-11T19:10:33.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.588+0000 7f5441a01740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2c29de00 con 0x562d2c6a5270 2024-03-11T19:10:33.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.588+0000 7f5441a01740 10 client.0 _ll_register_callbacks cb 0x562d2c5b5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:33.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.588+0000 7f5437fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 0x562d2c6af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.588+0000 7f5437fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 0x562d2c6af940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42512/0 (socket says 172.21.15.29:42512) 2024-03-11T19:10:33.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.588+0000 7f5437fff640 1 -- 172.21.15.29:0/2204823886 learned_addr learned my addr 172.21.15.29:0/2204823886 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.588+0000 7f543cbca640 1 --2- 172.21.15.29:0/2204823886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6a5270 0x562d2c6af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.588+0000 7f543d3cb640 1 --2- 172.21.15.29:0/2204823886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6aadd0 0x562d2c6b6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.588+0000 7f5437fff640 1 -- 172.21.15.29:0/2204823886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6a5270 msgr2=0x562d2c6af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.588+0000 7f5437fff640 1 --2- 172.21.15.29:0/2204823886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6a5270 0x562d2c6af400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.588+0000 7f5437fff640 1 -- 172.21.15.29:0/2204823886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6aadd0 msgr2=0x562d2c6b6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.588+0000 7f5437fff640 1 --2- 172.21.15.29:0/2204823886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6aadd0 0x562d2c6b6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.588+0000 7f5437fff640 1 -- 172.21.15.29:0/2204823886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54381a1950 con 0x562d2c6a5c60 2024-03-11T19:10:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.589+0000 7f5437fff640 1 --2- 172.21.15.29:0/2204823886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 0x562d2c6af940 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f542409a7c0 tx=0x7f5424016df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe46f58ad02cf224 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.589+0000 7f5417fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.589+0000 7f5417fff640 1 -- 172.21.15.29:0/2204823886 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54240a2fd0 con 0x562d2c6a5c60 2024-03-11T19:10:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.589+0000 7f5417fff640 1 -- 172.21.15.29:0/2204823886 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f542409a830 con 0x562d2c6a5c60 2024-03-11T19:10:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.589+0000 7f5441a01740 1 -- 172.21.15.29:0/2204823886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562d2c30d7e0 con 0x562d2c6a5c60 2024-03-11T19:10:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.589+0000 7f5417fff640 1 -- 172.21.15.29:0/2204823886 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5424005430 con 0x562d2c6a5c60 2024-03-11T19:10:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.589+0000 7f5441a01740 10 client.15546 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.589+0000 7f5441a01740 1 -- 172.21.15.29:0/2204823886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x562d2c60c6a0 con 0x562d2c6a5c60 2024-03-11T19:10:33.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.589+0000 7f5417fff640 1 -- 172.21.15.29:0/2204823886 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x562d2c30d7e0 con 0x562d2c6a5c60 2024-03-11T19:10:33.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.589+0000 7f5417fff640 1 -- 172.21.15.29:0/2204823886 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f542401f060 con 0x562d2c6a5c60 2024-03-11T19:10:33.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.589+0000 7f5441a01740 10 client.15546 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:33.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.589+0000 7f5441a01740 10 client.15546 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:10:33.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.589+0000 7f5441a01740 20 client.15546 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:33.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.589+0000 7f5441a01740 1 --2- 172.21.15.29:0/2204823886 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x562d2c6bb260 0x562d2c6bd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.589+0000 7f5441a01740 4 client.15546 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:10:33.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.589+0000 7f5441a01740 1 -- 172.21.15.29:0/2204823886 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562d2c34a720 con 0x562d2c6bb260 2024-03-11T19:10:33.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.590+0000 7f543cbca640 1 --2- 172.21.15.29:0/2204823886 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x562d2c6bb260 0x562d2c6bd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.590+0000 7f543cbca640 1 --2- 172.21.15.29:0/2204823886 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x562d2c6bb260 0x562d2c6bd650 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=19f7bbdcc6bec12c server_cookie=ea92001404420487 in_seq=0 out_seq=0 2024-03-11T19:10:33.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.590+0000 7f5417fff640 10 client.15546 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:10:33.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.591+0000 7f5417fff640 1 -- 172.21.15.29:0/2204823886 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562d2c34a720 con 0x562d2c6bb260 2024-03-11T19:10:33.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.591+0000 7f5417fff640 10 client.15546 handle_command_reply: tid=0 2024-03-11T19:10:33.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.591+0000 7f5441a01740 1 client.15546 shutdown 2024-03-11T19:10:33.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.591+0000 7f5441a01740 1 -- 172.21.15.29:0/2204823886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 msgr2=0x562d2c6af940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.591+0000 7f5441a01740 1 --2- 172.21.15.29:0/2204823886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 0x562d2c6af940 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f542409a7c0 tx=0x7f5424016df0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.591+0000 7f5441a01740 1 -- 172.21.15.29:0/2204823886 shutdown_connections 2024-03-11T19:10:33.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.591+0000 7f5441a01740 1 --2- 172.21.15.29:0/2204823886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6aadd0 0x562d2c6b6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.591+0000 7f5441a01740 1 --2- 172.21.15.29:0/2204823886 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x562d2c6bb260 0x562d2c6bd650 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.591+0000 7f5441a01740 1 --2- 172.21.15.29:0/2204823886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 0x562d2c6af940 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.591+0000 7f5417fff640 0 client.15546 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:10:33.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.591+0000 7f5441a01740 1 --2- 172.21.15.29:0/2204823886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6a5270 0x562d2c6af400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.591+0000 7f5441a01740 1 -- 172.21.15.29:0/2204823886 >> 172.21.15.29:0/2204823886 conn(0x562d2c6a07e0 msgr2=0x562d2c69fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.591+0000 7f5441a01740 1 -- 172.21.15.29:0/2204823886 shutdown_connections 2024-03-11T19:10:33.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.591+0000 7f5441a01740 1 -- 172.21.15.29:0/2204823886 wait complete. 2024-03-11T19:10:33.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.591+0000 7f5441a01740 20 client.15546 trim_cache size 0 max 16384 2024-03-11T19:10:33.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.603+0000 7f5441a01740 1 Processor -- start 2024-03-11T19:10:33.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.603+0000 7f5441a01740 1 -- start start 2024-03-11T19:10:33.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.603+0000 7f5441a01740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5270 0x562d2c6af400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.603+0000 7f5441a01740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6a5c60 0x562d2c6af940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.603+0000 7f5441a01740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6aadd0 0x562d2c6b6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.603+0000 7f5441a01740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2c593130 con 0x562d2c6a5270 2024-03-11T19:10:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.603+0000 7f5441a01740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2c692a40 con 0x562d2c6a5c60 2024-03-11T19:10:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.603+0000 7f5441a01740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2c69c2b0 con 0x562d2c6aadd0 2024-03-11T19:10:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.603+0000 7f543cbca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5270 0x562d2c6af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.603+0000 7f543cbca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5270 0x562d2c6af400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42518/0 (socket says 172.21.15.29:42518) 2024-03-11T19:10:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.603+0000 7f543cbca640 1 -- 172.21.15.29:0/972115738 learned_addr learned my addr 172.21.15.29:0/972115738 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.604+0000 7f5437fff640 1 --2- 172.21.15.29:0/972115738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6a5c60 0x562d2c6af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.604+0000 7f543d3cb640 1 --2- 172.21.15.29:0/972115738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6aadd0 0x562d2c6b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.604+0000 7f543cbca640 1 -- 172.21.15.29:0/972115738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6aadd0 msgr2=0x562d2c6b6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.604+0000 7f543cbca640 1 --2- 172.21.15.29:0/972115738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6aadd0 0x562d2c6b6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.604+0000 7f543cbca640 1 -- 172.21.15.29:0/972115738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6a5c60 msgr2=0x562d2c6af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.604+0000 7f543cbca640 1 --2- 172.21.15.29:0/972115738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6a5c60 0x562d2c6af940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.604+0000 7f543cbca640 1 -- 172.21.15.29:0/972115738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562d2c5a8c20 con 0x562d2c6a5270 2024-03-11T19:10:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.604+0000 7f543cbca640 1 --2- 172.21.15.29:0/972115738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5270 0x562d2c6af400 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x562d2c6a64c0 tx=0x7f542c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c12b98b4a5c1d84f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.604+0000 7f54227fc640 1 -- 172.21.15.29:0/972115738 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542c026040 con 0x562d2c6a5270 2024-03-11T19:10:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.604+0000 7f54227fc640 1 -- 172.21.15.29:0/972115738 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f542c028d00 con 0x562d2c6a5270 2024-03-11T19:10:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.604+0000 7f54227fc640 1 -- 172.21.15.29:0/972115738 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542c012910 con 0x562d2c6a5270 2024-03-11T19:10:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.605+0000 7f5441a01740 1 -- 172.21.15.29:0/972115738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5270 msgr2=0x562d2c6af400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.605+0000 7f5441a01740 1 --2- 172.21.15.29:0/972115738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5270 0x562d2c6af400 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x562d2c6a64c0 tx=0x7f542c017000 comp rx=0 tx=0).stop 2024-03-11T19:10:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.605+0000 7f5441a01740 1 -- 172.21.15.29:0/972115738 shutdown_connections 2024-03-11T19:10:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.605+0000 7f5441a01740 1 --2- 172.21.15.29:0/972115738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6aadd0 0x562d2c6b6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.605+0000 7f5441a01740 1 --2- 172.21.15.29:0/972115738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6a5c60 0x562d2c6af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.605+0000 7f5441a01740 1 --2- 172.21.15.29:0/972115738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5270 0x562d2c6af400 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.605+0000 7f5441a01740 1 -- 172.21.15.29:0/972115738 >> 172.21.15.29:0/972115738 conn(0x562d2c6a07e0 msgr2=0x562d2c6a0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.605+0000 7f5441a01740 1 -- 172.21.15.29:0/972115738 shutdown_connections 2024-03-11T19:10:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.605+0000 7f5441a01740 1 -- 172.21.15.29:0/972115738 wait complete. 2024-03-11T19:10:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.605+0000 7f5441a01740 1 -- start start 2024-03-11T19:10:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.605+0000 7f5441a01740 1 Processor -- start 2024-03-11T19:10:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5441a01740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6a5270 0x562d2c6b5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5441a01740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 0x562d2c6b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5441a01740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6aadd0 0x562d2c6b5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5441a01740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2c711720 con 0x562d2c6a5c60 2024-03-11T19:10:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5441a01740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2c711860 con 0x562d2c6a5270 2024-03-11T19:10:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5441a01740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2c7119a0 con 0x562d2c6aadd0 2024-03-11T19:10:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5441a01740 10 client.0 _ll_register_callbacks cb 0x7f5424005660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5437fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 0x562d2c6b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5437fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 0x562d2c6b57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42534/0 (socket says 172.21.15.29:42534) 2024-03-11T19:10:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5437fff640 1 -- 172.21.15.29:0/3579312330 learned_addr learned my addr 172.21.15.29:0/3579312330 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f543cbca640 1 --2- 172.21.15.29:0/3579312330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6a5270 0x562d2c6b5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f543d3cb640 1 --2- 172.21.15.29:0/3579312330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6aadd0 0x562d2c6b5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5437fff640 1 -- 172.21.15.29:0/3579312330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6aadd0 msgr2=0x562d2c6b5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5437fff640 1 --2- 172.21.15.29:0/3579312330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6aadd0 0x562d2c6b5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5437fff640 1 -- 172.21.15.29:0/3579312330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6a5270 msgr2=0x562d2c6b5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5437fff640 1 --2- 172.21.15.29:0/3579312330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6a5270 0x562d2c6b5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5437fff640 1 -- 172.21.15.29:0/3579312330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f542c004310 con 0x562d2c6a5c60 2024-03-11T19:10:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f543d3cb640 1 --2- 172.21.15.29:0/3579312330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6aadd0 0x562d2c6b5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5437fff640 1 --2- 172.21.15.29:0/3579312330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 0x562d2c6b57a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f54240971b0 tx=0x7f5424004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9d098cd74847bc server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5417fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5417fff640 1 -- 172.21.15.29:0/3579312330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542409a830 con 0x562d2c6a5c60 2024-03-11T19:10:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5417fff640 1 -- 172.21.15.29:0/3579312330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f54240a2fd0 con 0x562d2c6a5c60 2024-03-11T19:10:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.606+0000 7f5441a01740 1 -- 172.21.15.29:0/3579312330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f542400ce90 con 0x562d2c6a5c60 2024-03-11T19:10:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.607+0000 7f5417fff640 1 -- 172.21.15.29:0/3579312330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f542409bc80 con 0x562d2c6a5c60 2024-03-11T19:10:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.607+0000 7f5441a01740 10 client.15552 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.607+0000 7f5441a01740 1 -- 172.21.15.29:0/3579312330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x562d2c5ce8c0 con 0x562d2c6a5c60 2024-03-11T19:10:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.607+0000 7f5417fff640 1 -- 172.21.15.29:0/3579312330 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f542400ce90 con 0x562d2c6a5c60 2024-03-11T19:10:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.607+0000 7f5417fff640 1 -- 172.21.15.29:0/3579312330 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f542401f070 con 0x562d2c6a5c60 2024-03-11T19:10:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.607+0000 7f5441a01740 10 client.15552 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.607+0000 7f5441a01740 10 client.15552 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:10:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.607+0000 7f5441a01740 20 client.15552 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.607+0000 7f5441a01740 1 --2- 172.21.15.29:0/3579312330 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x562d2c712ec0 0x562d2c7152e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.607+0000 7f5441a01740 4 client.15552 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:10:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.607+0000 7f5441a01740 1 -- 172.21.15.29:0/3579312330 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562d2c399890 con 0x562d2c712ec0 2024-03-11T19:10:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.607+0000 7f543cbca640 1 --2- 172.21.15.29:0/3579312330 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x562d2c712ec0 0x562d2c7152e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.608+0000 7f5417fff640 10 client.15552 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:10:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.608+0000 7f543cbca640 1 --2- 172.21.15.29:0/3579312330 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x562d2c712ec0 0x562d2c7152e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d2c229854762b9c4 server_cookie=441073ae17f4c6b6 in_seq=0 out_seq=0 2024-03-11T19:10:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.609+0000 7f5417fff640 1 -- 172.21.15.29:0/3579312330 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562d2c399890 con 0x562d2c712ec0 2024-03-11T19:10:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.609+0000 7f5417fff640 10 client.15552 handle_command_reply: tid=0 2024-03-11T19:10:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.609+0000 7f5441a01740 1 client.15552 shutdown 2024-03-11T19:10:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.609+0000 7f5441a01740 1 -- 172.21.15.29:0/3579312330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 msgr2=0x562d2c6b57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.609+0000 7f5441a01740 1 --2- 172.21.15.29:0/3579312330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 0x562d2c6b57a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f54240971b0 tx=0x7f5424004840 comp rx=0 tx=0).stop 2024-03-11T19:10:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.609+0000 7f5441a01740 1 -- 172.21.15.29:0/3579312330 shutdown_connections 2024-03-11T19:10:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.609+0000 7f5441a01740 1 --2- 172.21.15.29:0/3579312330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562d2c6aadd0 0x562d2c6b5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.609+0000 7f5441a01740 1 --2- 172.21.15.29:0/3579312330 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x562d2c712ec0 0x562d2c7152e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.609+0000 7f5441a01740 1 --2- 172.21.15.29:0/3579312330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d2c6a5c60 0x562d2c6b57a0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.609+0000 7f5417fff640 0 client.15552 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:10:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.609+0000 7f5441a01740 1 --2- 172.21.15.29:0/3579312330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d2c6a5270 0x562d2c6b5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.609+0000 7f5441a01740 1 -- 172.21.15.29:0/3579312330 >> 172.21.15.29:0/3579312330 conn(0x562d2c6a07e0 msgr2=0x562d2c6a71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.609+0000 7f5441a01740 1 -- 172.21.15.29:0/3579312330 shutdown_connections 2024-03-11T19:10:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.609+0000 7f5441a01740 1 -- 172.21.15.29:0/3579312330 wait complete. 2024-03-11T19:10:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.609+0000 7f5441a01740 20 client.15552 trim_cache size 0 max 16384 2024-03-11T19:10:33.618 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:10:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.610+0000 7f543ee2e640 1 -- 172.21.15.29:0/3075914379 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f54280a1ac0 msgr2=0x7f54280a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.610+0000 7f543ee2e640 1 --2- 172.21.15.29:0/3075914379 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f54280a1ac0 0x7f54280a3f80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f542c009c20 tx=0x7f542c005f90 comp rx=0 tx=0).stop 2024-03-11T19:10:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.610+0000 7f543ee2e640 1 -- 172.21.15.29:0/3075914379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54381045f0 msgr2=0x7f54380ffc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.610+0000 7f543ee2e640 1 --2- 172.21.15.29:0/3075914379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54381045f0 0x7f54380ffc70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f542400ea20 tx=0x7f542400eee0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.610+0000 7f543ee2e640 1 -- 172.21.15.29:0/3075914379 shutdown_connections 2024-03-11T19:10:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.610+0000 7f543ee2e640 1 --2- 172.21.15.29:0/3075914379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5438106f20 0x7f54381001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.610+0000 7f543ee2e640 1 --2- 172.21.15.29:0/3075914379 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f54280a1ac0 0x7f54280a3f80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.610+0000 7f543ee2e640 1 --2- 172.21.15.29:0/3075914379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54381045f0 0x7f54380ffc70 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.610+0000 7f543ee2e640 1 --2- 172.21.15.29:0/3075914379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5438101cc0 0x7f5438101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.610+0000 7f543ee2e640 1 -- 172.21.15.29:0/3075914379 >> 172.21.15.29:0/3075914379 conn(0x7f54380fb610 msgr2=0x7f5438102720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.610+0000 7f543ee2e640 1 -- 172.21.15.29:0/3075914379 shutdown_connections 2024-03-11T19:10:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.610+0000 7f543ee2e640 1 -- 172.21.15.29:0/3075914379 wait complete. 2024-03-11T19:10:33.623 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:33 smithi029 ceph-mon[32439]: pgmap v373: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:33.623 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:33 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/4097938894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:10:33.624 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:10:33.624 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:10:33.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.728+0000 7f6ed5977640 1 -- 172.21.15.29:0/3420210180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ed0101280 msgr2=0x7f6ed0101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.728+0000 7f6ed5977640 1 --2- 172.21.15.29:0/3420210180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ed0101280 0x7f6ed0101700 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4009900 tx=0x7f6ec402bc30 comp rx=0 tx=0).stop 2024-03-11T19:10:33.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.728+0000 7f6ed5977640 1 -- 172.21.15.29:0/3420210180 shutdown_connections 2024-03-11T19:10:33.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.728+0000 7f6ed5977640 1 --2- 172.21.15.29:0/3420210180 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ed0101c40 0x7f6ed01084d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.728+0000 7f6ed5977640 1 --2- 172.21.15.29:0/3420210180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ed0101280 0x7f6ed0101700 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.728+0000 7f6ed5977640 1 --2- 172.21.15.29:0/3420210180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ed0100090 0x7f6ed01004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.728+0000 7f6ed5977640 1 -- 172.21.15.29:0/3420210180 >> 172.21.15.29:0/3420210180 conn(0x7f6ed00fb620 msgr2=0x7f6ed00fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.728+0000 7f6ed5977640 1 -- 172.21.15.29:0/3420210180 shutdown_connections 2024-03-11T19:10:33.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.728+0000 7f6ed5977640 1 -- 172.21.15.29:0/3420210180 wait complete. 2024-03-11T19:10:33.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.729+0000 7f6ed5977640 1 Processor -- start 2024-03-11T19:10:33.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.729+0000 7f6ed5977640 1 -- start start 2024-03-11T19:10:33.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.729+0000 7f6ed5977640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ed0100090 0x7f6ed0195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.729+0000 7f6ed5977640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ed0101280 0x7f6ed0196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.729+0000 7f6ed5977640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ed0101c40 0x7f6ed019d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.729+0000 7f6ed5977640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed019dad0 con 0x7f6ed0100090 2024-03-11T19:10:33.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.729+0000 7f6ed5977640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed019dc40 con 0x7f6ed0101280 2024-03-11T19:10:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.729+0000 7f6ed5977640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed019de00 con 0x7f6ed0101c40 2024-03-11T19:10:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.729+0000 7f6eceffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ed0100090 0x7f6ed0195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.729+0000 7f6eceffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ed0100090 0x7f6ed0195fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42548/0 (socket says 172.21.15.29:42548) 2024-03-11T19:10:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.729+0000 7f6eceffd640 1 -- 172.21.15.29:0/3746060468 learned_addr learned my addr 172.21.15.29:0/3746060468 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.729+0000 7f6ece7fc640 1 --2- 172.21.15.29:0/3746060468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ed0101280 0x7f6ed0196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.730+0000 7f6ecf7fe640 1 --2- 172.21.15.29:0/3746060468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ed0101c40 0x7f6ed019d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.730+0000 7f6eceffd640 1 -- 172.21.15.29:0/3746060468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ed0101c40 msgr2=0x7f6ed019d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.730+0000 7f6eceffd640 1 --2- 172.21.15.29:0/3746060468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ed0101c40 0x7f6ed019d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.730+0000 7f6eceffd640 1 -- 172.21.15.29:0/3746060468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ed0101280 msgr2=0x7f6ed0196520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.730+0000 7f6eceffd640 1 --2- 172.21.15.29:0/3746060468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ed0101280 0x7f6ed0196520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.730+0000 7f6eceffd640 1 -- 172.21.15.29:0/3746060468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec4009560 con 0x7f6ed0100090 2024-03-11T19:10:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.730+0000 7f6ecf7fe640 1 --2- 172.21.15.29:0/3746060468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ed0101c40 0x7f6ed019d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.730+0000 7f6eceffd640 1 --2- 172.21.15.29:0/3746060468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ed0100090 0x7f6ed0195fe0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0007b50 tx=0x7f6ec000e420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.730+0000 7f6ed4975640 1 -- 172.21.15.29:0/3746060468 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0011070 con 0x7f6ed0100090 2024-03-11T19:10:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.730+0000 7f6ed4975640 1 -- 172.21.15.29:0/3746060468 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6ec000c2e0 con 0x7f6ed0100090 2024-03-11T19:10:33.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.730+0000 7f6ed5977640 1 -- 172.21.15.29:0/3746060468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed0069750 con 0x7f6ed0100090 2024-03-11T19:10:33.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.730+0000 7f6ed4975640 1 -- 172.21.15.29:0/3746060468 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec000aaa0 con 0x7f6ed0100090 2024-03-11T19:10:33.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.730+0000 7f6ed5977640 1 -- 172.21.15.29:0/3746060468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed0069c30 con 0x7f6ed0100090 2024-03-11T19:10:33.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.732+0000 7f6ed4975640 1 -- 172.21.15.29:0/3746060468 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f6ec000ac60 con 0x7f6ed0100090 2024-03-11T19:10:33.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6ed8621740 1 Processor -- start 2024-03-11T19:10:33.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6ed8621740 1 -- start start 2024-03-11T19:10:33.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6ed8621740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7270 0x563cfedc7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6ed8621740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedc7c60 0x563cfedcc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6ed8621740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedccdd0 0x563cfedcf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6ed8621740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cfed83960 con 0x563cfedccdd0 2024-03-11T19:10:33.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6ed8621740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cfed37310 con 0x563cfedc7c60 2024-03-11T19:10:33.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6ed8621740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cfed34840 con 0x563cfedc7270 2024-03-11T19:10:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6ed4975640 1 --2- 172.21.15.29:0/3746060468 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6eb80a1ac0 0x7f6eb80a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6ece7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedccdd0 0x563cfedcf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6ece7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedccdd0 0x563cfedcf260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42556/0 (socket says 172.21.15.29:42556) 2024-03-11T19:10:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6ece7fc640 1 -- 172.21.15.29:0/1563979053 learned_addr learned my addr 172.21.15.29:0/1563979053 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6eceffd640 1 --2- 172.21.15.29:0/1563979053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedc7c60 0x563cfedcc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6ecf7fe640 1 --2- 172.21.15.29:0/3746060468 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6eb80a1ac0 0x7f6eb80a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6ed4975640 1 -- 172.21.15.29:0/3746060468 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f6ec00ce9f0 con 0x7f6ed0100090 2024-03-11T19:10:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6ecf7fe640 1 --2- 172.21.15.29:0/1563979053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7270 0x563cfedc7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6ece7fc640 1 -- 172.21.15.29:0/1563979053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7270 msgr2=0x563cfedc7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6ece7fc640 1 --2- 172.21.15.29:0/1563979053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7270 0x563cfedc7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6ece7fc640 1 -- 172.21.15.29:0/1563979053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedc7c60 msgr2=0x563cfedcc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.733+0000 7f6ece7fc640 1 --2- 172.21.15.29:0/1563979053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedc7c60 0x563cfedcc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.734+0000 7f6ece7fc640 1 -- 172.21.15.29:0/1563979053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cfecf2060 con 0x563cfedccdd0 2024-03-11T19:10:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.734+0000 7f6ece7fc640 1 --2- 172.21.15.29:0/1563979053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedccdd0 0x563cfedcf260 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4009900 tx=0x7f6ec402c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8ac1dc575007ccd server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.734+0000 7f6ecf7fe640 1 --2- 172.21.15.29:0/3746060468 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6eb80a1ac0 0x7f6eb80a3f80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x563cfedc84c0 tx=0x7f6ec8009530 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.734+0000 7f6eb5ffb640 1 -- 172.21.15.29:0/1563979053 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec40046c0 con 0x563cfedccdd0 2024-03-11T19:10:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.734+0000 7f6eb5ffb640 1 -- 172.21.15.29:0/1563979053 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6ec402de90 con 0x563cfedccdd0 2024-03-11T19:10:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.734+0000 7f6eb5ffb640 1 -- 172.21.15.29:0/1563979053 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4003a80 con 0x563cfedccdd0 2024-03-11T19:10:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.735+0000 7f6ed8621740 1 -- 172.21.15.29:0/1563979053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedccdd0 msgr2=0x563cfedcf260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.735+0000 7f6ed8621740 1 --2- 172.21.15.29:0/1563979053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedccdd0 0x563cfedcf260 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4009900 tx=0x7f6ec402c9a0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.735+0000 7f6ed8621740 1 -- 172.21.15.29:0/1563979053 shutdown_connections 2024-03-11T19:10:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.735+0000 7f6ed8621740 1 --2- 172.21.15.29:0/1563979053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedccdd0 0x563cfedcf260 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.735+0000 7f6ed8621740 1 --2- 172.21.15.29:0/1563979053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedc7c60 0x563cfedcc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.735+0000 7f6ed8621740 1 --2- 172.21.15.29:0/1563979053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7270 0x563cfedc7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.735+0000 7f6ed8621740 1 -- 172.21.15.29:0/1563979053 >> 172.21.15.29:0/1563979053 conn(0x563cfedc27e0 msgr2=0x563cfedc2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.735+0000 7f6ed8621740 1 -- 172.21.15.29:0/1563979053 shutdown_connections 2024-03-11T19:10:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.735+0000 7f6ed8621740 1 -- 172.21.15.29:0/1563979053 wait complete. 2024-03-11T19:10:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.735+0000 7f6ed8621740 1 -- start start 2024-03-11T19:10:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.735+0000 7f6ed8621740 1 Processor -- start 2024-03-11T19:10:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.735+0000 7f6ed8621740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedc7270 0x563cfedd13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.735+0000 7f6ed8621740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7c60 0x563cfedd1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.735+0000 7f6ed8621740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedccdd0 0x563cfedd8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.735+0000 7f6ed8621740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cfed30020 con 0x563cfedc7270 2024-03-11T19:10:33.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6ed8621740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cfea03430 con 0x563cfedccdd0 2024-03-11T19:10:33.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6ed8621740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cfe9bfe00 con 0x563cfedc7c60 2024-03-11T19:10:33.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6ed8621740 10 client.0 _ll_register_callbacks cb 0x563cfecd7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:33.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6eceffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedc7270 0x563cfedd13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6eceffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedc7270 0x563cfedd13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42560/0 (socket says 172.21.15.29:42560) 2024-03-11T19:10:33.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6eceffd640 1 -- 172.21.15.29:0/2724804228 learned_addr learned my addr 172.21.15.29:0/2724804228 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6ece7fc640 1 --2- 172.21.15.29:0/2724804228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7c60 0x563cfedd1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6ecf7fe640 1 --2- 172.21.15.29:0/2724804228 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedccdd0 0x563cfedd8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6eceffd640 1 -- 172.21.15.29:0/2724804228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7c60 msgr2=0x563cfedd1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6eceffd640 1 --2- 172.21.15.29:0/2724804228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7c60 0x563cfedd1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6eceffd640 1 -- 172.21.15.29:0/2724804228 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedccdd0 msgr2=0x563cfedd8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6eceffd640 1 --2- 172.21.15.29:0/2724804228 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedccdd0 0x563cfedd8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6eceffd640 1 -- 172.21.15.29:0/2724804228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed0108dd0 con 0x563cfedc7270 2024-03-11T19:10:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6ecf7fe640 1 --2- 172.21.15.29:0/2724804228 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedccdd0 0x563cfedd8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6eceffd640 1 --2- 172.21.15.29:0/2724804228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedc7270 0x563cfedd13f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00cc4d0 tx=0x7f6ec000a440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=815544c004d66fb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6eaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6eaeffd640 1 -- 172.21.15.29:0/2724804228 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec009c090 con 0x563cfedc7270 2024-03-11T19:10:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6eaeffd640 1 -- 172.21.15.29:0/2724804228 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f6ec009b990 con 0x563cfedc7270 2024-03-11T19:10:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6ed8621740 1 -- 172.21.15.29:0/2724804228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563cfea2f7e0 con 0x563cfedc7270 2024-03-11T19:10:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6eaeffd640 1 -- 172.21.15.29:0/2724804228 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ec009fae0 con 0x563cfedc7270 2024-03-11T19:10:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6ed8621740 10 client.15564 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.736+0000 7f6ed8621740 1 -- 172.21.15.29:0/2724804228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563cfed2e6a0 con 0x563cfedc7270 2024-03-11T19:10:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.737+0000 7f6eaeffd640 1 -- 172.21.15.29:0/2724804228 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x563cfea2f7e0 con 0x563cfedc7270 2024-03-11T19:10:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.737+0000 7f6eaeffd640 1 -- 172.21.15.29:0/2724804228 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6ec001c020 con 0x563cfedc7270 2024-03-11T19:10:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.737+0000 7f6ed8621740 10 client.15564 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.737+0000 7f6ed8621740 10 client.15564 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:10:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.737+0000 7f6ed8621740 20 client.15564 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.737+0000 7f6ed8621740 1 --2- 172.21.15.29:0/2724804228 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x563cfeddd730 0x563cfeddfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.737+0000 7f6ed8621740 4 client.15564 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:10:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.737+0000 7f6ed8621740 1 -- 172.21.15.29:0/2724804228 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563cfea6c720 con 0x563cfeddd730 2024-03-11T19:10:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.737+0000 7f6ece7fc640 1 --2- 172.21.15.29:0/2724804228 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x563cfeddd730 0x563cfeddfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.738+0000 7f6ece7fc640 1 --2- 172.21.15.29:0/2724804228 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x563cfeddd730 0x563cfeddfb40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c9394dbffd5d1865 server_cookie=b9f2ab38edc67567 in_seq=0 out_seq=0 2024-03-11T19:10:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.738+0000 7f6eaeffd640 10 client.15564 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:10:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.738+0000 7f6eaeffd640 1 -- 172.21.15.29:0/2724804228 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563cfea6c720 con 0x563cfeddd730 2024-03-11T19:10:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.738+0000 7f6eaeffd640 10 client.15564 handle_command_reply: tid=0 2024-03-11T19:10:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.738+0000 7f6ed8621740 1 client.15564 shutdown 2024-03-11T19:10:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.738+0000 7f6ed8621740 1 -- 172.21.15.29:0/2724804228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedc7270 msgr2=0x563cfedd13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.738+0000 7f6ed8621740 1 --2- 172.21.15.29:0/2724804228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedc7270 0x563cfedd13f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00cc4d0 tx=0x7f6ec000a440 comp rx=0 tx=0).stop 2024-03-11T19:10:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.738+0000 7f6ed8621740 1 -- 172.21.15.29:0/2724804228 shutdown_connections 2024-03-11T19:10:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.738+0000 7f6ed8621740 1 --2- 172.21.15.29:0/2724804228 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x563cfeddd730 0x563cfeddfb40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.738+0000 7f6ed8621740 1 --2- 172.21.15.29:0/2724804228 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedccdd0 0x563cfedd8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.738+0000 7f6ed8621740 1 --2- 172.21.15.29:0/2724804228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7c60 0x563cfedd1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.738+0000 7f6ed8621740 1 --2- 172.21.15.29:0/2724804228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedc7270 0x563cfedd13f0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.738+0000 7f6ed8621740 1 -- 172.21.15.29:0/2724804228 >> 172.21.15.29:0/2724804228 conn(0x563cfedc27e0 msgr2=0x563cfedc1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.738+0000 7f6eaeffd640 0 client.15564 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:10:33.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.738+0000 7f6ed8621740 1 -- 172.21.15.29:0/2724804228 shutdown_connections 2024-03-11T19:10:33.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.739+0000 7f6ed8621740 1 -- 172.21.15.29:0/2724804228 wait complete. 2024-03-11T19:10:33.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.739+0000 7f6ed8621740 20 client.15564 trim_cache size 0 max 16384 2024-03-11T19:10:33.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.751+0000 7f6ed8621740 1 Processor -- start 2024-03-11T19:10:33.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.751+0000 7f6ed8621740 1 -- start start 2024-03-11T19:10:33.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.751+0000 7f6ed8621740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7270 0x563cfedc2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.751+0000 7f6ed8621740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedc7c60 0x563cfedd8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.751+0000 7f6ed8621740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedccdd0 0x563cfedc9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.751+0000 7f6ed8621740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cfecb5130 con 0x563cfedc7c60 2024-03-11T19:10:33.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.751+0000 7f6ed8621740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cfea2f7e0 con 0x563cfedccdd0 2024-03-11T19:10:33.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.751+0000 7f6ed8621740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cfedc8350 con 0x563cfedc7270 2024-03-11T19:10:33.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.751+0000 7f6ece7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedc7c60 0x563cfedd8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.751+0000 7f6ece7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedc7c60 0x563cfedd8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42568/0 (socket says 172.21.15.29:42568) 2024-03-11T19:10:33.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.751+0000 7f6ece7fc640 1 -- 172.21.15.29:0/2801757996 learned_addr learned my addr 172.21.15.29:0/2801757996 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.751+0000 7f6eceffd640 1 --2- 172.21.15.29:0/2801757996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7270 0x563cfedc2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.751+0000 7f6ecf7fe640 1 --2- 172.21.15.29:0/2801757996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedccdd0 0x563cfedc9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.751+0000 7f6ece7fc640 1 -- 172.21.15.29:0/2801757996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7270 msgr2=0x563cfedc2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.751+0000 7f6ece7fc640 1 --2- 172.21.15.29:0/2801757996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7270 0x563cfedc2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.751+0000 7f6ece7fc640 1 -- 172.21.15.29:0/2801757996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedccdd0 msgr2=0x563cfedc9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.751+0000 7f6ece7fc640 1 --2- 172.21.15.29:0/2801757996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedccdd0 0x563cfedc9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.751+0000 7f6ece7fc640 1 -- 172.21.15.29:0/2801757996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cfeccac20 con 0x563cfedc7c60 2024-03-11T19:10:33.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.751+0000 7f6ecf7fe640 1 --2- 172.21.15.29:0/2801757996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedccdd0 0x563cfedc9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:10:33.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.752+0000 7f6ece7fc640 1 --2- 172.21.15.29:0/2801757996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedc7c60 0x563cfedd8900 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x563cfedc3750 tx=0x7f6ec402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=babd38acceb5328 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.752+0000 7f6eb57fa640 1 -- 172.21.15.29:0/2801757996 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4004aa0 con 0x563cfedc7c60 2024-03-11T19:10:33.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.752+0000 7f6eb57fa640 1 -- 172.21.15.29:0/2801757996 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6ec40044f0 con 0x563cfedc7c60 2024-03-11T19:10:33.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.752+0000 7f6eb57fa640 1 -- 172.21.15.29:0/2801757996 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4050850 con 0x563cfedc7c60 2024-03-11T19:10:33.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.753+0000 7f6ed8621740 1 -- 172.21.15.29:0/2801757996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedc7c60 msgr2=0x563cfedd8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.753+0000 7f6ed8621740 1 --2- 172.21.15.29:0/2801757996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedc7c60 0x563cfedd8900 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x563cfedc3750 tx=0x7f6ec402bc30 comp rx=0 tx=0).stop 2024-03-11T19:10:33.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.753+0000 7f6ed8621740 1 -- 172.21.15.29:0/2801757996 shutdown_connections 2024-03-11T19:10:33.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.753+0000 7f6ed8621740 1 --2- 172.21.15.29:0/2801757996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedccdd0 0x563cfedc9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.753+0000 7f6ed8621740 1 --2- 172.21.15.29:0/2801757996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedc7c60 0x563cfedd8900 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.753+0000 7f6ed8621740 1 --2- 172.21.15.29:0/2801757996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7270 0x563cfedc2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.753+0000 7f6ed8621740 1 -- 172.21.15.29:0/2801757996 >> 172.21.15.29:0/2801757996 conn(0x563cfedc27e0 msgr2=0x563cfedbda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.753+0000 7f6ed8621740 1 -- 172.21.15.29:0/2801757996 shutdown_connections 2024-03-11T19:10:33.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.753+0000 7f6ed8621740 1 -- 172.21.15.29:0/2801757996 wait complete. 2024-03-11T19:10:33.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.753+0000 7f6ed8621740 1 -- start start 2024-03-11T19:10:33.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.753+0000 7f6ed8621740 1 Processor -- start 2024-03-11T19:10:33.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.753+0000 7f6ed8621740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7270 0x563cfedd68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.753+0000 7f6ed8621740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedc7c60 0x563cfedd6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.753+0000 7f6ed8621740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedccdd0 0x563cfedd7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.753+0000 7f6ed8621740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cfedd82d0 con 0x563cfedccdd0 2024-03-11T19:10:33.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.753+0000 7f6ed8621740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cfedd8440 con 0x563cfedc7c60 2024-03-11T19:10:33.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.753+0000 7f6ed8621740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cfedd8600 con 0x563cfedc7270 2024-03-11T19:10:33.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.753+0000 7f6ed8621740 10 client.0 _ll_register_callbacks cb 0x7f6ec4005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:33.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.754+0000 7f6ecf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedccdd0 0x563cfedd7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.754+0000 7f6ecf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedccdd0 0x563cfedd7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42572/0 (socket says 172.21.15.29:42572) 2024-03-11T19:10:33.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.754+0000 7f6ecf7fe640 1 -- 172.21.15.29:0/553340680 learned_addr learned my addr 172.21.15.29:0/553340680 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.754+0000 7f6eceffd640 1 --2- 172.21.15.29:0/553340680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7270 0x563cfedd68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.754+0000 7f6ece7fc640 1 --2- 172.21.15.29:0/553340680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedc7c60 0x563cfedd6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.754+0000 7f6ecf7fe640 1 -- 172.21.15.29:0/553340680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7270 msgr2=0x563cfedd68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.754+0000 7f6ecf7fe640 1 --2- 172.21.15.29:0/553340680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7270 0x563cfedd68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.754+0000 7f6ecf7fe640 1 -- 172.21.15.29:0/553340680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedc7c60 msgr2=0x563cfedd6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.754+0000 7f6ecf7fe640 1 --2- 172.21.15.29:0/553340680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedc7c60 0x563cfedd6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.754+0000 7f6ecf7fe640 1 -- 172.21.15.29:0/553340680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563cfecf2060 con 0x563cfedccdd0 2024-03-11T19:10:33.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.754+0000 7f6ece7fc640 1 --2- 172.21.15.29:0/553340680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedc7c60 0x563cfedd6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:33.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.754+0000 7f6ecf7fe640 1 --2- 172.21.15.29:0/553340680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedccdd0 0x563cfedd7d90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8005fd0 tx=0x7f6ec8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a17f1d9cbd278a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.754+0000 7f6eaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:33.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.754+0000 7f6eaeffd640 1 -- 172.21.15.29:0/553340680 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8020710 con 0x563cfedccdd0 2024-03-11T19:10:33.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.754+0000 7f6eaeffd640 1 -- 172.21.15.29:0/553340680 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f6ec8020870 con 0x563cfedccdd0 2024-03-11T19:10:33.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.754+0000 7f6ed8621740 1 -- 172.21.15.29:0/553340680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6ec402de90 con 0x563cfedccdd0 2024-03-11T19:10:33.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.754+0000 7f6eaeffd640 1 -- 172.21.15.29:0/553340680 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ec8004b20 con 0x563cfedccdd0 2024-03-11T19:10:33.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.754+0000 7f6ed8621740 10 client.15570 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:33.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.754+0000 7f6ed8621740 1 -- 172.21.15.29:0/553340680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563cfecf08c0 con 0x563cfedccdd0 2024-03-11T19:10:33.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.755+0000 7f6eaeffd640 1 -- 172.21.15.29:0/553340680 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6ec402de90 con 0x563cfedccdd0 2024-03-11T19:10:33.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.755+0000 7f6eaeffd640 1 -- 172.21.15.29:0/553340680 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6ec800c070 con 0x563cfedccdd0 2024-03-11T19:10:33.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.755+0000 7f6ed8621740 10 client.15570 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:33.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.755+0000 7f6ed8621740 10 client.15570 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:10:33.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.755+0000 7f6ed8621740 20 client.15570 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:33.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.755+0000 7f6ed8621740 1 --2- 172.21.15.29:0/553340680 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x563cfee34f70 0x563cfee37390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.755+0000 7f6ed8621740 4 client.15570 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:10:33.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.755+0000 7f6ed8621740 1 -- 172.21.15.29:0/553340680 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563cfeabb890 con 0x563cfee34f70 2024-03-11T19:10:33.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.755+0000 7f6eceffd640 1 --2- 172.21.15.29:0/553340680 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x563cfee34f70 0x563cfee37390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.755+0000 7f6eceffd640 1 --2- 172.21.15.29:0/553340680 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x563cfee34f70 0x563cfee37390 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b15b6138872cc6f2 server_cookie=7edbf72a3b11fc28 in_seq=0 out_seq=0 2024-03-11T19:10:33.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.755+0000 7f6eaeffd640 10 client.15570 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:10:33.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.756+0000 7f6eaeffd640 1 -- 172.21.15.29:0/553340680 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563cfeabb890 con 0x563cfee34f70 2024-03-11T19:10:33.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.756+0000 7f6eaeffd640 10 client.15570 handle_command_reply: tid=0 2024-03-11T19:10:33.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.756+0000 7f6ed8621740 1 client.15570 shutdown 2024-03-11T19:10:33.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.756+0000 7f6ed8621740 1 -- 172.21.15.29:0/553340680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedccdd0 msgr2=0x563cfedd7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.756+0000 7f6ed8621740 1 --2- 172.21.15.29:0/553340680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedccdd0 0x563cfedd7d90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8005fd0 tx=0x7f6ec8004ab0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.756+0000 7f6ed8621740 1 -- 172.21.15.29:0/553340680 shutdown_connections 2024-03-11T19:10:33.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.756+0000 7f6ed8621740 1 --2- 172.21.15.29:0/553340680 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x563cfee34f70 0x563cfee37390 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.756+0000 7f6ed8621740 1 --2- 172.21.15.29:0/553340680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cfedccdd0 0x563cfedd7d90 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.756+0000 7f6eaeffd640 0 client.15570 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:10:33.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.756+0000 7f6ed8621740 1 --2- 172.21.15.29:0/553340680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cfedc7c60 0x563cfedd6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.756+0000 7f6ed8621740 1 --2- 172.21.15.29:0/553340680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cfedc7270 0x563cfedd68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.756+0000 7f6ed8621740 1 -- 172.21.15.29:0/553340680 >> 172.21.15.29:0/553340680 conn(0x563cfedc27e0 msgr2=0x563cfedc2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.756+0000 7f6ed8621740 1 -- 172.21.15.29:0/553340680 shutdown_connections 2024-03-11T19:10:33.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.756+0000 7f6ed8621740 1 -- 172.21.15.29:0/553340680 wait complete. 2024-03-11T19:10:33.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.756+0000 7f6ed8621740 20 client.15570 trim_cache size 0 max 16384 2024-03-11T19:10:33.765 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:10:33.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.757+0000 7f6ed5977640 1 -- 172.21.15.29:0/3746060468 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6eb80a1ac0 msgr2=0x7f6eb80a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.757+0000 7f6ed5977640 1 --2- 172.21.15.29:0/3746060468 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6eb80a1ac0 0x7f6eb80a3f80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x563cfedc84c0 tx=0x7f6ec8009530 comp rx=0 tx=0).stop 2024-03-11T19:10:33.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.757+0000 7f6ed5977640 1 -- 172.21.15.29:0/3746060468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ed0100090 msgr2=0x7f6ed0195fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.757+0000 7f6ed5977640 1 --2- 172.21.15.29:0/3746060468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ed0100090 0x7f6ed0195fe0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0007b50 tx=0x7f6ec000e420 comp rx=0 tx=0).stop 2024-03-11T19:10:33.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.757+0000 7f6ed5977640 1 -- 172.21.15.29:0/3746060468 shutdown_connections 2024-03-11T19:10:33.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.757+0000 7f6ed5977640 1 --2- 172.21.15.29:0/3746060468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ed0101c40 0x7f6ed019d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.757+0000 7f6ed5977640 1 --2- 172.21.15.29:0/3746060468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ed0101280 0x7f6ed0196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.757+0000 7f6ed5977640 1 --2- 172.21.15.29:0/3746060468 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6eb80a1ac0 0x7f6eb80a3f80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.757+0000 7f6ed5977640 1 --2- 172.21.15.29:0/3746060468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ed0100090 0x7f6ed0195fe0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.757+0000 7f6ed5977640 1 -- 172.21.15.29:0/3746060468 >> 172.21.15.29:0/3746060468 conn(0x7f6ed00fb620 msgr2=0x7f6ed0104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.757+0000 7f6ed5977640 1 -- 172.21.15.29:0/3746060468 shutdown_connections 2024-03-11T19:10:33.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.757+0000 7f6ed5977640 1 -- 172.21.15.29:0/3746060468 wait complete. 2024-03-11T19:10:33.771 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:10:33.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:10:33.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:33 smithi170 ceph-mon[35690]: pgmap v373: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:33.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:33 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/4097938894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:10:33.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.883+0000 7f680dcaa640 1 -- 172.21.15.29:0/3415177425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6808104550 msgr2=0x7f6808106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.883+0000 7f680dcaa640 1 --2- 172.21.15.29:0/3415177425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6808104550 0x7f6808106940 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f67fc009900 tx=0x7f67fc02bc30 comp rx=0 tx=0).stop 2024-03-11T19:10:33.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.883+0000 7f680dcaa640 1 -- 172.21.15.29:0/3415177425 shutdown_connections 2024-03-11T19:10:33.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.883+0000 7f680dcaa640 1 --2- 172.21.15.29:0/3415177425 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6808106e80 0x7f68081092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.883+0000 7f680dcaa640 1 --2- 172.21.15.29:0/3415177425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6808104550 0x7f6808106940 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.883+0000 7f680dcaa640 1 --2- 172.21.15.29:0/3415177425 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6808101c20 0x7f6808104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.883+0000 7f680dcaa640 1 -- 172.21.15.29:0/3415177425 >> 172.21.15.29:0/3415177425 conn(0x7f68080fb630 msgr2=0x7f68080fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.883+0000 7f680dcaa640 1 -- 172.21.15.29:0/3415177425 shutdown_connections 2024-03-11T19:10:33.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.883+0000 7f680dcaa640 1 -- 172.21.15.29:0/3415177425 wait complete. 2024-03-11T19:10:33.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.884+0000 7f680dcaa640 1 Processor -- start 2024-03-11T19:10:33.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.884+0000 7f680dcaa640 1 -- start start 2024-03-11T19:10:33.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.884+0000 7f680dcaa640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6808101c20 0x7f68081971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.884+0000 7f680dcaa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6808104550 0x7f68081976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.884+0000 7f680dcaa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6808106e80 0x7f680819e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.884+0000 7f680dcaa640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680819eca0 con 0x7f6808104550 2024-03-11T19:10:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.884+0000 7f680dcaa640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680819ee10 con 0x7f6808101c20 2024-03-11T19:10:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.884+0000 7f680dcaa640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680819efd0 con 0x7f6808106e80 2024-03-11T19:10:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.885+0000 7f6806ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6808104550 0x7f68081976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.885+0000 7f6806ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6808104550 0x7f68081976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42586/0 (socket says 172.21.15.29:42586) 2024-03-11T19:10:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.885+0000 7f6806ffd640 1 -- 172.21.15.29:0/2799272776 learned_addr learned my addr 172.21.15.29:0/2799272776 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.885+0000 7f68077fe640 1 --2- 172.21.15.29:0/2799272776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6808101c20 0x7f68081971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.885+0000 7f6807fff640 1 --2- 172.21.15.29:0/2799272776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6808106e80 0x7f680819e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.885+0000 7f6806ffd640 1 -- 172.21.15.29:0/2799272776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6808106e80 msgr2=0x7f680819e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.885+0000 7f6806ffd640 1 --2- 172.21.15.29:0/2799272776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6808106e80 0x7f680819e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.885+0000 7f6806ffd640 1 -- 172.21.15.29:0/2799272776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6808101c20 msgr2=0x7f68081971b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.885+0000 7f6806ffd640 1 --2- 172.21.15.29:0/2799272776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6808101c20 0x7f68081971b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.885+0000 7f6806ffd640 1 -- 172.21.15.29:0/2799272776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67fc009560 con 0x7f6808104550 2024-03-11T19:10:33.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.885+0000 7f68077fe640 1 --2- 172.21.15.29:0/2799272776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6808101c20 0x7f68081971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:10:33.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.885+0000 7f6806ffd640 1 --2- 172.21.15.29:0/2799272776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6808104550 0x7f68081976f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f67fc000c00 tx=0x7f67fc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.885+0000 7f6804ff9640 1 -- 172.21.15.29:0/2799272776 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc0044e0 con 0x7f6808104550 2024-03-11T19:10:33.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.885+0000 7f680dcaa640 1 -- 172.21.15.29:0/2799272776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680819f250 con 0x7f6808104550 2024-03-11T19:10:33.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.885+0000 7f6804ff9640 1 -- 172.21.15.29:0/2799272776 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f67fc02de90 con 0x7f6808104550 2024-03-11T19:10:33.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.885+0000 7f6804ff9640 1 -- 172.21.15.29:0/2799272776 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc005410 con 0x7f6808104550 2024-03-11T19:10:33.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.885+0000 7f680dcaa640 1 -- 172.21.15.29:0/2799272776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680819f7b0 con 0x7f6808104550 2024-03-11T19:10:33.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.888+0000 7f6804ff9640 1 -- 172.21.15.29:0/2799272776 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f67fc005570 con 0x7f6808104550 2024-03-11T19:10:33.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.888+0000 7f681087d740 1 Processor -- start 2024-03-11T19:10:33.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.888+0000 7f681087d740 1 -- start start 2024-03-11T19:10:33.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.888+0000 7f681087d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84afb270 0x55da84afb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.888+0000 7f681087d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afbc60 0x55da84b00760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.888+0000 7f681087d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84b00dd0 0x55da84b03260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.888+0000 7f681087d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da84ab7960 con 0x55da84b00dd0 2024-03-11T19:10:33.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.888+0000 7f681087d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da84a6b310 con 0x55da84afb270 2024-03-11T19:10:33.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.888+0000 7f681087d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da84a68840 con 0x55da84afbc60 2024-03-11T19:10:33.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.888+0000 7f6804ff9640 1 --2- 172.21.15.29:0/2799272776 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f67f00a1ac0 0x7f67f00a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.888+0000 7f6806ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84b00dd0 0x55da84b03260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.888+0000 7f6806ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84b00dd0 0x55da84b03260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42596/0 (socket says 172.21.15.29:42596) 2024-03-11T19:10:33.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.888+0000 7f6806ffd640 1 -- 172.21.15.29:0/702768132 learned_addr learned my addr 172.21.15.29:0/702768132 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.888+0000 7f6807fff640 1 --2- 172.21.15.29:0/2799272776 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f67f00a1ac0 0x7f67f00a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.889+0000 7f68077fe640 1 --2- 172.21.15.29:0/702768132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afbc60 0x55da84b00760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.889+0000 7f6807fff640 1 --2- 172.21.15.29:0/702768132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84afb270 0x55da84afb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.889+0000 7f6804ff9640 1 -- 172.21.15.29:0/2799272776 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f67fc0ef4e0 con 0x7f6808104550 2024-03-11T19:10:33.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.889+0000 7f6806ffd640 1 -- 172.21.15.29:0/702768132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afbc60 msgr2=0x55da84b00760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.889+0000 7f6806ffd640 1 --2- 172.21.15.29:0/702768132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afbc60 0x55da84b00760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.889+0000 7f6806ffd640 1 -- 172.21.15.29:0/702768132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84afb270 msgr2=0x55da84afb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.889+0000 7f6806ffd640 1 --2- 172.21.15.29:0/702768132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84afb270 0x55da84afb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.889+0000 7f6806ffd640 1 -- 172.21.15.29:0/702768132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da84a26060 con 0x55da84b00dd0 2024-03-11T19:10:33.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.889+0000 7f6807fff640 1 --2- 172.21.15.29:0/2799272776 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f67f00a1ac0 0x7f67f00a3f80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f68000098b0 tx=0x7f6800009e00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.889+0000 7f6806ffd640 1 --2- 172.21.15.29:0/702768132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84b00dd0 0x55da84b03260 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f67fc0c4770 tx=0x7f67fc02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf0b343514f79b2f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.889+0000 7f67edffb640 1 -- 172.21.15.29:0/702768132 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc003e20 con 0x55da84b00dd0 2024-03-11T19:10:33.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.889+0000 7f67edffb640 1 -- 172.21.15.29:0/702768132 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f67fc0bcc00 con 0x55da84b00dd0 2024-03-11T19:10:33.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.889+0000 7f67edffb640 1 -- 172.21.15.29:0/702768132 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc0ba800 con 0x55da84b00dd0 2024-03-11T19:10:33.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.890+0000 7f681087d740 1 -- 172.21.15.29:0/702768132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84b00dd0 msgr2=0x55da84b03260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.890+0000 7f681087d740 1 --2- 172.21.15.29:0/702768132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84b00dd0 0x55da84b03260 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f67fc0c4770 tx=0x7f67fc02bc30 comp rx=0 tx=0).stop 2024-03-11T19:10:33.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.890+0000 7f681087d740 1 -- 172.21.15.29:0/702768132 shutdown_connections 2024-03-11T19:10:33.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.890+0000 7f681087d740 1 --2- 172.21.15.29:0/702768132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84b00dd0 0x55da84b03260 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.890+0000 7f681087d740 1 --2- 172.21.15.29:0/702768132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afbc60 0x55da84b00760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.890+0000 7f681087d740 1 --2- 172.21.15.29:0/702768132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84afb270 0x55da84afb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.890+0000 7f681087d740 1 -- 172.21.15.29:0/702768132 >> 172.21.15.29:0/702768132 conn(0x55da84af67e0 msgr2=0x55da84af6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.890+0000 7f681087d740 1 -- 172.21.15.29:0/702768132 shutdown_connections 2024-03-11T19:10:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.890+0000 7f681087d740 1 -- 172.21.15.29:0/702768132 wait complete. 2024-03-11T19:10:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.890+0000 7f681087d740 1 -- start start 2024-03-11T19:10:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f681087d740 1 Processor -- start 2024-03-11T19:10:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f681087d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afb270 0x55da84b05240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f681087d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84afbc60 0x55da84b05780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f681087d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84b00dd0 0x55da84b0c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f681087d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da84a64020 con 0x55da84b00dd0 2024-03-11T19:10:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f681087d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da84737430 con 0x55da84afbc60 2024-03-11T19:10:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f681087d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da846f3e00 con 0x55da84afb270 2024-03-11T19:10:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f681087d740 10 client.0 _ll_register_callbacks cb 0x55da84a0bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f6807fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84b00dd0 0x55da84b0c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f6807fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84b00dd0 0x55da84b0c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42602/0 (socket says 172.21.15.29:42602) 2024-03-11T19:10:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f6807fff640 1 -- 172.21.15.29:0/637912167 learned_addr learned my addr 172.21.15.29:0/637912167 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f68077fe640 1 --2- 172.21.15.29:0/637912167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afb270 0x55da84b05240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f6806ffd640 1 --2- 172.21.15.29:0/637912167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84afbc60 0x55da84b05780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f6807fff640 1 -- 172.21.15.29:0/637912167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afb270 msgr2=0x55da84b05240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f6807fff640 1 --2- 172.21.15.29:0/637912167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afb270 0x55da84b05240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f6807fff640 1 -- 172.21.15.29:0/637912167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84afbc60 msgr2=0x55da84b05780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f6807fff640 1 --2- 172.21.15.29:0/637912167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84afbc60 0x55da84b05780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f6807fff640 1 -- 172.21.15.29:0/637912167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680819f7b0 con 0x55da84b00dd0 2024-03-11T19:10:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f6806ffd640 1 --2- 172.21.15.29:0/637912167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84afbc60 0x55da84b05780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f6807fff640 1 --2- 172.21.15.29:0/637912167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84b00dd0 0x55da84b0c7c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6800010d80 tx=0x7f680001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ecf3b159324ce9f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f67e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f67e6ffd640 1 -- 172.21.15.29:0/637912167 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6800020400 con 0x55da84b00dd0 2024-03-11T19:10:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.891+0000 7f67e6ffd640 1 -- 172.21.15.29:0/637912167 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f6800020560 con 0x55da84b00dd0 2024-03-11T19:10:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.892+0000 7f681087d740 1 -- 172.21.15.29:0/637912167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55da847637e0 con 0x55da84b00dd0 2024-03-11T19:10:33.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.892+0000 7f67e6ffd640 1 -- 172.21.15.29:0/637912167 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6800020720 con 0x55da84b00dd0 2024-03-11T19:10:33.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.892+0000 7f681087d740 10 client.15582 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:33.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.892+0000 7f681087d740 1 -- 172.21.15.29:0/637912167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55da84a626a0 con 0x55da84b00dd0 2024-03-11T19:10:33.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.892+0000 7f67e6ffd640 1 -- 172.21.15.29:0/637912167 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55da847637e0 con 0x55da84b00dd0 2024-03-11T19:10:33.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.892+0000 7f67e6ffd640 1 -- 172.21.15.29:0/637912167 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6800034850 con 0x55da84b00dd0 2024-03-11T19:10:33.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.892+0000 7f681087d740 10 client.15582 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:33.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.892+0000 7f681087d740 10 client.15582 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:10:33.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.892+0000 7f681087d740 20 client.15582 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:33.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.892+0000 7f681087d740 1 --2- 172.21.15.29:0/637912167 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55da84b11440 0x55da84b13860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.892+0000 7f681087d740 4 client.15582 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:10:33.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.892+0000 7f681087d740 1 -- 172.21.15.29:0/637912167 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55da847a0720 con 0x55da84b11440 2024-03-11T19:10:33.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.892+0000 7f68077fe640 1 --2- 172.21.15.29:0/637912167 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55da84b11440 0x55da84b13860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.893+0000 7f67e6ffd640 10 client.15582 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:10:33.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.893+0000 7f68077fe640 1 --2- 172.21.15.29:0/637912167 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55da84b11440 0x55da84b13860 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cfe745f971ac4c5e server_cookie=fcf1b610ce0dee8 in_seq=0 out_seq=0 2024-03-11T19:10:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.893+0000 7f67e6ffd640 1 -- 172.21.15.29:0/637912167 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55da847a0720 con 0x55da84b11440 2024-03-11T19:10:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.893+0000 7f67e6ffd640 10 client.15582 handle_command_reply: tid=0 2024-03-11T19:10:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.894+0000 7f681087d740 1 client.15582 shutdown 2024-03-11T19:10:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.894+0000 7f681087d740 1 -- 172.21.15.29:0/637912167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84b00dd0 msgr2=0x55da84b0c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.894+0000 7f681087d740 1 --2- 172.21.15.29:0/637912167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84b00dd0 0x55da84b0c7c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6800010d80 tx=0x7f680001c000 comp rx=0 tx=0).stop 2024-03-11T19:10:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.894+0000 7f681087d740 1 -- 172.21.15.29:0/637912167 shutdown_connections 2024-03-11T19:10:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.894+0000 7f681087d740 1 --2- 172.21.15.29:0/637912167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84b00dd0 0x55da84b0c7c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.894+0000 7f681087d740 1 --2- 172.21.15.29:0/637912167 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55da84b11440 0x55da84b13860 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.894+0000 7f681087d740 1 --2- 172.21.15.29:0/637912167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84afbc60 0x55da84b05780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.894+0000 7f681087d740 1 --2- 172.21.15.29:0/637912167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afb270 0x55da84b05240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.894+0000 7f67e6ffd640 0 client.15582 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:10:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.894+0000 7f681087d740 1 -- 172.21.15.29:0/637912167 >> 172.21.15.29:0/637912167 conn(0x55da84af67e0 msgr2=0x55da84af5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.894+0000 7f681087d740 1 -- 172.21.15.29:0/637912167 shutdown_connections 2024-03-11T19:10:33.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.894+0000 7f681087d740 1 -- 172.21.15.29:0/637912167 wait complete. 2024-03-11T19:10:33.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.894+0000 7f681087d740 20 client.15582 trim_cache size 0 max 16384 2024-03-11T19:10:33.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f681087d740 1 Processor -- start 2024-03-11T19:10:33.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f681087d740 1 -- start start 2024-03-11T19:10:33.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f681087d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afb270 0x55da84b05240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f681087d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84afbc60 0x55da84b05780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f681087d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84b00dd0 0x55da84b0c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f681087d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da849e9130 con 0x55da84afbc60 2024-03-11T19:10:33.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f681087d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da84af72d0 con 0x55da84b00dd0 2024-03-11T19:10:33.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f681087d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da847a0720 con 0x55da84afb270 2024-03-11T19:10:33.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f6806ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84afbc60 0x55da84b05780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f6806ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84afbc60 0x55da84b05780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42618/0 (socket says 172.21.15.29:42618) 2024-03-11T19:10:33.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f6806ffd640 1 -- 172.21.15.29:0/4127707571 learned_addr learned my addr 172.21.15.29:0/4127707571 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f6807fff640 1 --2- 172.21.15.29:0/4127707571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84b00dd0 0x55da84b0c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f68077fe640 1 --2- 172.21.15.29:0/4127707571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afb270 0x55da84b05240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f6806ffd640 1 -- 172.21.15.29:0/4127707571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afb270 msgr2=0x55da84b05240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f6806ffd640 1 --2- 172.21.15.29:0/4127707571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afb270 0x55da84b05240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f6806ffd640 1 -- 172.21.15.29:0/4127707571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84b00dd0 msgr2=0x55da84b0c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f6806ffd640 1 --2- 172.21.15.29:0/4127707571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84b00dd0 0x55da84b0c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f6806ffd640 1 -- 172.21.15.29:0/4127707571 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da849fec20 con 0x55da84afbc60 2024-03-11T19:10:33.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f68077fe640 1 --2- 172.21.15.29:0/4127707571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afb270 0x55da84b05240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:33.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f6807fff640 1 --2- 172.21.15.29:0/4127707571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84b00dd0 0x55da84b0c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:10:33.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.908+0000 7f6806ffd640 1 --2- 172.21.15.29:0/4127707571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84afbc60 0x55da84b05780 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f67fc0b8960 tx=0x7f67fc0bbf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5263dcd66ae0f49d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.909+0000 7f67ed7fa640 1 -- 172.21.15.29:0/4127707571 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc003e20 con 0x55da84afbc60 2024-03-11T19:10:33.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.909+0000 7f67ed7fa640 1 -- 172.21.15.29:0/4127707571 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f67fc0c4ca0 con 0x55da84afbc60 2024-03-11T19:10:33.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.909+0000 7f67ed7fa640 1 -- 172.21.15.29:0/4127707571 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc0ba800 con 0x55da84afbc60 2024-03-11T19:10:33.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.909+0000 7f681087d740 1 -- 172.21.15.29:0/4127707571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84afbc60 msgr2=0x55da84b05780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.909+0000 7f681087d740 1 --2- 172.21.15.29:0/4127707571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84afbc60 0x55da84b05780 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f67fc0b8960 tx=0x7f67fc0bbf60 comp rx=0 tx=0).stop 2024-03-11T19:10:33.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.909+0000 7f681087d740 1 -- 172.21.15.29:0/4127707571 shutdown_connections 2024-03-11T19:10:33.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.909+0000 7f681087d740 1 --2- 172.21.15.29:0/4127707571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84b00dd0 0x55da84b0c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.909+0000 7f681087d740 1 --2- 172.21.15.29:0/4127707571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84afbc60 0x55da84b05780 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.909+0000 7f681087d740 1 --2- 172.21.15.29:0/4127707571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afb270 0x55da84b05240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.909+0000 7f681087d740 1 -- 172.21.15.29:0/4127707571 >> 172.21.15.29:0/4127707571 conn(0x55da84af67e0 msgr2=0x55da84af6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.909+0000 7f681087d740 1 -- 172.21.15.29:0/4127707571 shutdown_connections 2024-03-11T19:10:33.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.909+0000 7f681087d740 1 -- 172.21.15.29:0/4127707571 wait complete. 2024-03-11T19:10:33.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.910+0000 7f681087d740 1 -- start start 2024-03-11T19:10:33.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.910+0000 7f681087d740 1 Processor -- start 2024-03-11T19:10:33.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.910+0000 7f681087d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84afb270 0x55da84b0c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.910+0000 7f681087d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afbc60 0x55da84b0abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.910+0000 7f681087d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84b00dd0 0x55da84b0b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.910+0000 7f681087d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da84b0b640 con 0x55da84afb270 2024-03-11T19:10:33.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.910+0000 7f681087d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da84b0b7b0 con 0x55da84b00dd0 2024-03-11T19:10:33.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.910+0000 7f681087d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da84b0b970 con 0x55da84afbc60 2024-03-11T19:10:33.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.910+0000 7f681087d740 10 client.0 _ll_register_callbacks cb 0x7f67fc0bc2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:33.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.910+0000 7f68077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84afb270 0x55da84b0c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.910+0000 7f68077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84afb270 0x55da84b0c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42626/0 (socket says 172.21.15.29:42626) 2024-03-11T19:10:33.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.910+0000 7f68077fe640 1 -- 172.21.15.29:0/3969974919 learned_addr learned my addr 172.21.15.29:0/3969974919 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:33.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.910+0000 7f6807fff640 1 --2- 172.21.15.29:0/3969974919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84b00dd0 0x55da84b0b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.910+0000 7f6806ffd640 1 --2- 172.21.15.29:0/3969974919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afbc60 0x55da84b0abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.910+0000 7f68077fe640 1 -- 172.21.15.29:0/3969974919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afbc60 msgr2=0x55da84b0abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.911+0000 7f68077fe640 1 --2- 172.21.15.29:0/3969974919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afbc60 0x55da84b0abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.911+0000 7f68077fe640 1 -- 172.21.15.29:0/3969974919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84b00dd0 msgr2=0x55da84b0b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:33.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.911+0000 7f68077fe640 1 --2- 172.21.15.29:0/3969974919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84b00dd0 0x55da84b0b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.911+0000 7f68077fe640 1 -- 172.21.15.29:0/3969974919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55da84a26060 con 0x55da84afb270 2024-03-11T19:10:33.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.911+0000 7f6807fff640 1 --2- 172.21.15.29:0/3969974919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84b00dd0 0x55da84b0b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:33.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.911+0000 7f6806ffd640 1 --2- 172.21.15.29:0/3969974919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afbc60 0x55da84b0abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:33.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.911+0000 7f68077fe640 1 --2- 172.21.15.29:0/3969974919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84afb270 0x55da84b0c5b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x55da84afc060 tx=0x7f67f8011be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45c621538e54aace server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:33.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.911+0000 7f67e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:33.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.911+0000 7f67e6ffd640 1 -- 172.21.15.29:0/3969974919 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f801e600 con 0x55da84afb270 2024-03-11T19:10:33.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.911+0000 7f67e6ffd640 1 -- 172.21.15.29:0/3969974919 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f67f801e760 con 0x55da84afb270 2024-03-11T19:10:33.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.911+0000 7f681087d740 1 -- 172.21.15.29:0/3969974919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f67fc0bcc00 con 0x55da84afb270 2024-03-11T19:10:33.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.911+0000 7f67e6ffd640 1 -- 172.21.15.29:0/3969974919 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67f801e920 con 0x55da84afb270 2024-03-11T19:10:33.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.911+0000 7f681087d740 10 client.15588 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:33.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.911+0000 7f681087d740 1 -- 172.21.15.29:0/3969974919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55da84a248c0 con 0x55da84afb270 2024-03-11T19:10:33.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.911+0000 7f67e6ffd640 1 -- 172.21.15.29:0/3969974919 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f67fc0bcc00 con 0x55da84afb270 2024-03-11T19:10:33.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.912+0000 7f67e6ffd640 1 -- 172.21.15.29:0/3969974919 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f67f8027990 con 0x55da84afb270 2024-03-11T19:10:33.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.912+0000 7f681087d740 10 client.15588 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:33.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.912+0000 7f681087d740 10 client.15588 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:10:33.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.912+0000 7f681087d740 20 client.15588 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:33.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.912+0000 7f681087d740 1 --2- 172.21.15.29:0/3969974919 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55da84b691c0 0x55da84b6b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:33.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.912+0000 7f681087d740 4 client.15588 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:10:33.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.912+0000 7f681087d740 1 -- 172.21.15.29:0/3969974919 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55da847ef890 con 0x55da84b691c0 2024-03-11T19:10:33.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.912+0000 7f6806ffd640 1 --2- 172.21.15.29:0/3969974919 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55da84b691c0 0x55da84b6b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:33.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.912+0000 7f6806ffd640 1 --2- 172.21.15.29:0/3969974919 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55da84b691c0 0x55da84b6b5e0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bc0c492925003094 server_cookie=3b983b9f20849b09 in_seq=0 out_seq=0 2024-03-11T19:10:33.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.912+0000 7f67e6ffd640 10 client.15588 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:10:33.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.913+0000 7f67e6ffd640 1 -- 172.21.15.29:0/3969974919 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55da847ef890 con 0x55da84b691c0 2024-03-11T19:10:33.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.913+0000 7f67e6ffd640 10 client.15588 handle_command_reply: tid=0 2024-03-11T19:10:33.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.913+0000 7f681087d740 1 client.15588 shutdown 2024-03-11T19:10:33.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.913+0000 7f681087d740 1 -- 172.21.15.29:0/3969974919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84afb270 msgr2=0x55da84b0c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.913+0000 7f681087d740 1 --2- 172.21.15.29:0/3969974919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84afb270 0x55da84b0c5b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x55da84afc060 tx=0x7f67f8011be0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.913+0000 7f681087d740 1 -- 172.21.15.29:0/3969974919 shutdown_connections 2024-03-11T19:10:33.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.913+0000 7f681087d740 1 --2- 172.21.15.29:0/3969974919 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55da84b691c0 0x55da84b6b5e0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.913+0000 7f681087d740 1 --2- 172.21.15.29:0/3969974919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da84b00dd0 0x55da84b0b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.913+0000 7f681087d740 1 --2- 172.21.15.29:0/3969974919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da84afbc60 0x55da84b0abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.913+0000 7f67e6ffd640 0 client.15588 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:10:33.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.913+0000 7f681087d740 1 --2- 172.21.15.29:0/3969974919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da84afb270 0x55da84b0c5b0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.913+0000 7f681087d740 1 -- 172.21.15.29:0/3969974919 >> 172.21.15.29:0/3969974919 conn(0x55da84af67e0 msgr2=0x55da84b05220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.913+0000 7f681087d740 1 -- 172.21.15.29:0/3969974919 shutdown_connections 2024-03-11T19:10:33.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.913+0000 7f681087d740 1 -- 172.21.15.29:0/3969974919 wait complete. 2024-03-11T19:10:33.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.913+0000 7f681087d740 20 client.15588 trim_cache size 0 max 16384 2024-03-11T19:10:33.926 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:10:33.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.913+0000 7f680dcaa640 1 -- 172.21.15.29:0/2799272776 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f67f00a1ac0 msgr2=0x7f67f00a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.913+0000 7f680dcaa640 1 --2- 172.21.15.29:0/2799272776 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f67f00a1ac0 0x7f67f00a3f80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f68000098b0 tx=0x7f6800009e00 comp rx=0 tx=0).stop 2024-03-11T19:10:33.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.913+0000 7f680dcaa640 1 -- 172.21.15.29:0/2799272776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6808104550 msgr2=0x7f68081976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:33.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.913+0000 7f680dcaa640 1 --2- 172.21.15.29:0/2799272776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6808104550 0x7f68081976f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f67fc000c00 tx=0x7f67fc0042f0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.914+0000 7f680dcaa640 1 -- 172.21.15.29:0/2799272776 shutdown_connections 2024-03-11T19:10:33.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.914+0000 7f680dcaa640 1 --2- 172.21.15.29:0/2799272776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6808106e80 0x7f680819e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.914+0000 7f680dcaa640 1 --2- 172.21.15.29:0/2799272776 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f67f00a1ac0 0x7f67f00a3f80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.914+0000 7f680dcaa640 1 --2- 172.21.15.29:0/2799272776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6808104550 0x7f68081976f0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.914+0000 7f680dcaa640 1 --2- 172.21.15.29:0/2799272776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6808101c20 0x7f68081971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:33.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.914+0000 7f680dcaa640 1 -- 172.21.15.29:0/2799272776 >> 172.21.15.29:0/2799272776 conn(0x7f68080fb630 msgr2=0x7f6808102660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:33.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.914+0000 7f680dcaa640 1 -- 172.21.15.29:0/2799272776 shutdown_connections 2024-03-11T19:10:33.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:33.914+0000 7f680dcaa640 1 -- 172.21.15.29:0/2799272776 wait complete. 2024-03-11T19:10:33.930 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:10:33.930 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:10:34.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:33 smithi190 ceph-mon[36292]: pgmap v373: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:34.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:33 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/4097938894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:10:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.035+0000 7f98c333a640 1 -- 172.21.15.29:0/2228307700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc106ec0 msgr2=0x7f98bc1092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.035+0000 7f98c333a640 1 --2- 172.21.15.29:0/2228307700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc106ec0 0x7f98bc1092b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f98b4009ca0 tx=0x7f98b402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.035+0000 7f98c333a640 1 -- 172.21.15.29:0/2228307700 shutdown_connections 2024-03-11T19:10:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.035+0000 7f98c333a640 1 --2- 172.21.15.29:0/2228307700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc106ec0 0x7f98bc1092b0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.035+0000 7f98c333a640 1 --2- 172.21.15.29:0/2228307700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98bc104590 0x7f98bc106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.035+0000 7f98c333a640 1 --2- 172.21.15.29:0/2228307700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98bc101c60 0x7f98bc104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.035+0000 7f98c333a640 1 -- 172.21.15.29:0/2228307700 >> 172.21.15.29:0/2228307700 conn(0x7f98bc0fb610 msgr2=0x7f98bc0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.035+0000 7f98c333a640 1 -- 172.21.15.29:0/2228307700 shutdown_connections 2024-03-11T19:10:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.035+0000 7f98c333a640 1 -- 172.21.15.29:0/2228307700 wait complete. 2024-03-11T19:10:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.036+0000 7f98c333a640 1 Processor -- start 2024-03-11T19:10:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.036+0000 7f98c333a640 1 -- start start 2024-03-11T19:10:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.036+0000 7f98c333a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98bc101c60 0x7f98bc1015d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.036+0000 7f98c333a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98bc104590 0x7f98bc0ffc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.036+0000 7f98c333a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc106ec0 0x7f98bc100150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.036+0000 7f98c333a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc1008b0 con 0x7f98bc106ec0 2024-03-11T19:10:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.036+0000 7f98c333a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc100a20 con 0x7f98bc104590 2024-03-11T19:10:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.036+0000 7f98c333a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc1a1190 con 0x7f98bc101c60 2024-03-11T19:10:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.036+0000 7f98c18d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc106ec0 0x7f98bc100150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.036+0000 7f98c18d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc106ec0 0x7f98bc100150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42638/0 (socket says 172.21.15.29:42638) 2024-03-11T19:10:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.036+0000 7f98c18d7640 1 -- 172.21.15.29:0/3377616263 learned_addr learned my addr 172.21.15.29:0/3377616263 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.036+0000 7f98c08d5640 1 --2- 172.21.15.29:0/3377616263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98bc104590 0x7f98bc0ffc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.036+0000 7f98c10d6640 1 --2- 172.21.15.29:0/3377616263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98bc101c60 0x7f98bc1015d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.036+0000 7f98c18d7640 1 -- 172.21.15.29:0/3377616263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98bc101c60 msgr2=0x7f98bc1015d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.036+0000 7f98c18d7640 1 --2- 172.21.15.29:0/3377616263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98bc101c60 0x7f98bc1015d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.036+0000 7f98c18d7640 1 -- 172.21.15.29:0/3377616263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98bc104590 msgr2=0x7f98bc0ffc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.036+0000 7f98c18d7640 1 --2- 172.21.15.29:0/3377616263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98bc104590 0x7f98bc0ffc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.036+0000 7f98c18d7640 1 -- 172.21.15.29:0/3377616263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac009490 con 0x7f98bc106ec0 2024-03-11T19:10:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.037+0000 7f98c18d7640 1 --2- 172.21.15.29:0/3377616263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc106ec0 0x7f98bc100150 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f98b4033000 tx=0x7f98b4032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.037+0000 7f98b27fc640 1 -- 172.21.15.29:0/3377616263 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b4034e70 con 0x7f98bc106ec0 2024-03-11T19:10:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.037+0000 7f98c333a640 1 -- 172.21.15.29:0/3377616263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b4009900 con 0x7f98bc106ec0 2024-03-11T19:10:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.037+0000 7f98b27fc640 1 -- 172.21.15.29:0/3377616263 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f98b4003e20 con 0x7f98bc106ec0 2024-03-11T19:10:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.037+0000 7f98b27fc640 1 -- 172.21.15.29:0/3377616263 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b4004840 con 0x7f98bc106ec0 2024-03-11T19:10:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.037+0000 7f98c333a640 1 -- 172.21.15.29:0/3377616263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98bc1a1700 con 0x7f98bc106ec0 2024-03-11T19:10:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.039+0000 7f98b27fc640 1 -- 172.21.15.29:0/3377616263 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f98b403b040 con 0x7f98bc106ec0 2024-03-11T19:10:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98b27fc640 1 --2- 172.21.15.29:0/3377616263 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f98a80a1ac0 0x7f98a80a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98c5fe1740 1 Processor -- start 2024-03-11T19:10:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98c5fe1740 1 -- start start 2024-03-11T19:10:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98c10d6640 1 --2- 172.21.15.29:0/3377616263 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f98a80a1ac0 0x7f98a80a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98b27fc640 1 -- 172.21.15.29:0/3377616263 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f98b40ef980 con 0x7f98bc106ec0 2024-03-11T19:10:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98c5fe1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 0x55e800599690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98c5fe1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e800599c60 0x55e80059e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98c5fe1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e80059edd0 0x55e8005a1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98c5fe1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e800555960 con 0x55e800599270 2024-03-11T19:10:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98c5fe1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e800509310 con 0x55e800599c60 2024-03-11T19:10:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98c5fe1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e800506840 con 0x55e80059edd0 2024-03-11T19:10:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98c10d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 0x55e800599690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98c10d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 0x55e800599690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42654/0 (socket says 172.21.15.29:42654) 2024-03-11T19:10:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98c10d6640 1 -- 172.21.15.29:0/3233715322 learned_addr learned my addr 172.21.15.29:0/3233715322 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98c08d5640 1 --2- 172.21.15.29:0/3233715322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e800599c60 0x55e80059e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98c18d7640 1 --2- 172.21.15.29:0/3233715322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e80059edd0 0x55e8005a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98c10d6640 1 -- 172.21.15.29:0/3233715322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e80059edd0 msgr2=0x55e8005a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98c10d6640 1 --2- 172.21.15.29:0/3233715322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e80059edd0 0x55e8005a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98c10d6640 1 -- 172.21.15.29:0/3233715322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e800599c60 msgr2=0x55e80059e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98c10d6640 1 --2- 172.21.15.29:0/3233715322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e800599c60 0x55e80059e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.040+0000 7f98c10d6640 1 -- 172.21.15.29:0/3233715322 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e8004c4060 con 0x55e800599270 2024-03-11T19:10:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.041+0000 7f98c10d6640 1 --2- 172.21.15.29:0/3377616263 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f98a80a1ac0 0x7f98a80a3f80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f98b8012a20 tx=0x7f98b800bc20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.041+0000 7f98c10d6640 1 --2- 172.21.15.29:0/3233715322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 0x55e800599690 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f98b800e720 tx=0x7f98b801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f38c94ca56ca6dfe server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.041+0000 7f98a77fe640 1 -- 172.21.15.29:0/3233715322 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b800ed20 con 0x55e800599270 2024-03-11T19:10:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.041+0000 7f98a77fe640 1 -- 172.21.15.29:0/3233715322 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f98b800ee80 con 0x55e800599270 2024-03-11T19:10:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.041+0000 7f98a77fe640 1 -- 172.21.15.29:0/3233715322 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8010660 con 0x55e800599270 2024-03-11T19:10:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.042+0000 7f98c5fe1740 1 -- 172.21.15.29:0/3233715322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 msgr2=0x55e800599690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.042+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/3233715322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 0x55e800599690 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f98b800e720 tx=0x7f98b801c000 comp rx=0 tx=0).stop 2024-03-11T19:10:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.042+0000 7f98c5fe1740 1 -- 172.21.15.29:0/3233715322 shutdown_connections 2024-03-11T19:10:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.042+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/3233715322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e80059edd0 0x55e8005a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.042+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/3233715322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e800599c60 0x55e80059e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.042+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/3233715322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 0x55e800599690 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.042+0000 7f98c5fe1740 1 -- 172.21.15.29:0/3233715322 >> 172.21.15.29:0/3233715322 conn(0x55e8005947e0 msgr2=0x55e800594be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.042+0000 7f98c5fe1740 1 -- 172.21.15.29:0/3233715322 shutdown_connections 2024-03-11T19:10:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.042+0000 7f98c5fe1740 1 -- 172.21.15.29:0/3233715322 wait complete. 2024-03-11T19:10:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.042+0000 7f98c5fe1740 1 -- start start 2024-03-11T19:10:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.042+0000 7f98c5fe1740 1 Processor -- start 2024-03-11T19:10:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.042+0000 7f98c5fe1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 0x55e8005a3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.042+0000 7f98c5fe1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e800599c60 0x55e8005a3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.042+0000 7f98c5fe1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e80059edd0 0x55e8005aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.042+0000 7f98c5fe1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e800502020 con 0x55e800599270 2024-03-11T19:10:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.042+0000 7f98c5fe1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8001d5430 con 0x55e80059edd0 2024-03-11T19:10:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.042+0000 7f98c5fe1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e800191e00 con 0x55e800599c60 2024-03-11T19:10:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.042+0000 7f98c5fe1740 10 client.0 _ll_register_callbacks cb 0x55e8004a9cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.043+0000 7f98c10d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 0x55e8005a3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.043+0000 7f98c10d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 0x55e8005a3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42668/0 (socket says 172.21.15.29:42668) 2024-03-11T19:10:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.043+0000 7f98c10d6640 1 -- 172.21.15.29:0/3076798161 learned_addr learned my addr 172.21.15.29:0/3076798161 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.043+0000 7f98c18d7640 1 --2- 172.21.15.29:0/3076798161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e80059edd0 0x55e8005aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.043+0000 7f98c08d5640 1 --2- 172.21.15.29:0/3076798161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e800599c60 0x55e8005a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.043+0000 7f98c10d6640 1 -- 172.21.15.29:0/3076798161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e800599c60 msgr2=0x55e8005a3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.043+0000 7f98c10d6640 1 --2- 172.21.15.29:0/3076798161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e800599c60 0x55e8005a3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.043+0000 7f98c10d6640 1 -- 172.21.15.29:0/3076798161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e80059edd0 msgr2=0x55e8005aa9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.043+0000 7f98c10d6640 1 --2- 172.21.15.29:0/3076798161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e80059edd0 0x55e8005aa9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.043+0000 7f98c10d6640 1 -- 172.21.15.29:0/3076798161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b801a040 con 0x55e800599270 2024-03-11T19:10:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.043+0000 7f98c10d6640 1 --2- 172.21.15.29:0/3076798161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 0x55e8005a3450 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f98b8006010 tx=0x7f98b800d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec12de55e577be1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.043+0000 7f98a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.043+0000 7f98a4ff9640 1 -- 172.21.15.29:0/3076798161 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b800ed20 con 0x55e800599270 2024-03-11T19:10:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.043+0000 7f98a4ff9640 1 -- 172.21.15.29:0/3076798161 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f98b800de80 con 0x55e800599270 2024-03-11T19:10:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.043+0000 7f98c5fe1740 1 -- 172.21.15.29:0/3076798161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e80023e720 con 0x55e800599270 2024-03-11T19:10:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.043+0000 7f98a4ff9640 1 -- 172.21.15.29:0/3076798161 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98b800ba00 con 0x55e800599270 2024-03-11T19:10:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.043+0000 7f98c5fe1740 10 client.15600 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.043+0000 7f98c5fe1740 1 -- 172.21.15.29:0/3076798161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e8005006a0 con 0x55e800599270 2024-03-11T19:10:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.044+0000 7f98a4ff9640 1 -- 172.21.15.29:0/3076798161 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55e80023e720 con 0x55e800599270 2024-03-11T19:10:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.044+0000 7f98a4ff9640 1 -- 172.21.15.29:0/3076798161 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f98b802b5e0 con 0x55e800599270 2024-03-11T19:10:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.044+0000 7f98c5fe1740 10 client.15600 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.044+0000 7f98c5fe1740 10 client.15600 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:10:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.044+0000 7f98c5fe1740 20 client.15600 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.044+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/3076798161 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e8005af590 0x55e8005b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.044+0000 7f98c5fe1740 4 client.15600 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:10:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.044+0000 7f98c5fe1740 1 -- 172.21.15.29:0/3076798161 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e80028d890 con 0x55e8005af590 2024-03-11T19:10:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.044+0000 7f98c08d5640 1 --2- 172.21.15.29:0/3076798161 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e8005af590 0x55e8005b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.045+0000 7f98c08d5640 1 --2- 172.21.15.29:0/3076798161 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e8005af590 0x55e8005b19b0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=834173046ec89d55 server_cookie=f2215e5010ebfda0 in_seq=0 out_seq=0 2024-03-11T19:10:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.045+0000 7f98a4ff9640 10 client.15600 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.045+0000 7f98a4ff9640 1 -- 172.21.15.29:0/3076798161 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e80028d890 con 0x55e8005af590 2024-03-11T19:10:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.045+0000 7f98a4ff9640 10 client.15600 handle_command_reply: tid=0 2024-03-11T19:10:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.045+0000 7f98c5fe1740 1 client.15600 shutdown 2024-03-11T19:10:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.045+0000 7f98c5fe1740 1 -- 172.21.15.29:0/3076798161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 msgr2=0x55e8005a3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.045+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/3076798161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 0x55e8005a3450 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f98b8006010 tx=0x7f98b800d9f0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.046+0000 7f98c5fe1740 1 -- 172.21.15.29:0/3076798161 shutdown_connections 2024-03-11T19:10:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.046+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/3076798161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e80059edd0 0x55e8005aa9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.046+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/3076798161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e800599c60 0x55e8005a3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.046+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/3076798161 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e8005af590 0x55e8005b19b0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.046+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/3076798161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 0x55e8005a3450 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.046+0000 7f98a4ff9640 0 client.15600 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:34.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.046+0000 7f98c5fe1740 1 -- 172.21.15.29:0/3076798161 >> 172.21.15.29:0/3076798161 conn(0x55e8005947e0 msgr2=0x55e800593b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:34.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.046+0000 7f98c5fe1740 1 -- 172.21.15.29:0/3076798161 shutdown_connections 2024-03-11T19:10:34.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.046+0000 7f98c5fe1740 1 -- 172.21.15.29:0/3076798161 wait complete. 2024-03-11T19:10:34.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.046+0000 7f98c5fe1740 20 client.15600 trim_cache size 0 max 16384 2024-03-11T19:10:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.057+0000 7f98c5fe1740 1 Processor -- start 2024-03-11T19:10:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.057+0000 7f98c5fe1740 1 -- start start 2024-03-11T19:10:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98c5fe1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e800599270 0x55e8005a3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98c5fe1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e800599c60 0x55e8005aa960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98c5fe1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80059edd0 0x55e8005932f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98c5fe1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e800487130 con 0x55e80059edd0 2024-03-11T19:10:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98c5fe1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e800599ad0 con 0x55e800599c60 2024-03-11T19:10:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98c5fe1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e800598e30 con 0x55e800599270 2024-03-11T19:10:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98c18d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80059edd0 0x55e8005932f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98c18d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80059edd0 0x55e8005932f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42682/0 (socket says 172.21.15.29:42682) 2024-03-11T19:10:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98c18d7640 1 -- 172.21.15.29:0/3086816371 learned_addr learned my addr 172.21.15.29:0/3086816371 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98c08d5640 1 --2- 172.21.15.29:0/3086816371 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e800599c60 0x55e8005aa960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98c10d6640 1 --2- 172.21.15.29:0/3086816371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e800599270 0x55e8005a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98c18d7640 1 -- 172.21.15.29:0/3086816371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e800599270 msgr2=0x55e8005a3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98c18d7640 1 --2- 172.21.15.29:0/3086816371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e800599270 0x55e8005a3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98c18d7640 1 -- 172.21.15.29:0/3086816371 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e800599c60 msgr2=0x55e8005aa960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98c18d7640 1 --2- 172.21.15.29:0/3086816371 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e800599c60 0x55e8005aa960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98c18d7640 1 -- 172.21.15.29:0/3086816371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e80049cc20 con 0x55e80059edd0 2024-03-11T19:10:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98c08d5640 1 --2- 172.21.15.29:0/3086816371 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e800599c60 0x55e8005aa960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98c10d6640 1 --2- 172.21.15.29:0/3086816371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e800599270 0x55e8005a3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98c18d7640 1 --2- 172.21.15.29:0/3086816371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80059edd0 0x55e8005932f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f98b4005e30 tx=0x7f98b402c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a19ce09b7e4b326a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98a6ffd640 1 -- 172.21.15.29:0/3086816371 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b40bca50 con 0x55e80059edd0 2024-03-11T19:10:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98a6ffd640 1 -- 172.21.15.29:0/3086816371 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f98b40c0af0 con 0x55e80059edd0 2024-03-11T19:10:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.058+0000 7f98a6ffd640 1 -- 172.21.15.29:0/3086816371 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b40da640 con 0x55e80059edd0 2024-03-11T19:10:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.059+0000 7f98c5fe1740 1 -- 172.21.15.29:0/3086816371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80059edd0 msgr2=0x55e8005932f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.059+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/3086816371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80059edd0 0x55e8005932f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f98b4005e30 tx=0x7f98b402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.059+0000 7f98c5fe1740 1 -- 172.21.15.29:0/3086816371 shutdown_connections 2024-03-11T19:10:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.059+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/3086816371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e80059edd0 0x55e8005932f0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.059+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/3086816371 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e800599c60 0x55e8005aa960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.059+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/3086816371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e800599270 0x55e8005a3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.059+0000 7f98c5fe1740 1 -- 172.21.15.29:0/3086816371 >> 172.21.15.29:0/3086816371 conn(0x55e8005947e0 msgr2=0x55e8005a3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.059+0000 7f98c5fe1740 1 -- 172.21.15.29:0/3086816371 shutdown_connections 2024-03-11T19:10:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.059+0000 7f98c5fe1740 1 -- 172.21.15.29:0/3086816371 wait complete. 2024-03-11T19:10:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.059+0000 7f98c5fe1740 1 -- start start 2024-03-11T19:10:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.059+0000 7f98c5fe1740 1 Processor -- start 2024-03-11T19:10:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.060+0000 7f98c5fe1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 0x55e8005aa230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.060+0000 7f98c5fe1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e800599c60 0x55e8005aa770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.060+0000 7f98c5fe1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e80059edd0 0x55e8005a8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.060+0000 7f98c5fe1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8005a8e50 con 0x55e800599270 2024-03-11T19:10:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.060+0000 7f98c5fe1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8005a8fc0 con 0x55e800599c60 2024-03-11T19:10:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.060+0000 7f98c5fe1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8005a9180 con 0x55e80059edd0 2024-03-11T19:10:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.060+0000 7f98c5fe1740 10 client.0 _ll_register_callbacks cb 0x7f98b8012570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.060+0000 7f98c10d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 0x55e8005aa230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.060+0000 7f98c10d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 0x55e8005aa230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42696/0 (socket says 172.21.15.29:42696) 2024-03-11T19:10:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.060+0000 7f98c10d6640 1 -- 172.21.15.29:0/1168095681 learned_addr learned my addr 172.21.15.29:0/1168095681 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.060+0000 7f98c18d7640 1 --2- 172.21.15.29:0/1168095681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e80059edd0 0x55e8005a8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.060+0000 7f98c08d5640 1 --2- 172.21.15.29:0/1168095681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e800599c60 0x55e8005aa770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.060+0000 7f98c10d6640 1 -- 172.21.15.29:0/1168095681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e80059edd0 msgr2=0x55e8005a8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.060+0000 7f98c10d6640 1 --2- 172.21.15.29:0/1168095681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e80059edd0 0x55e8005a8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.060+0000 7f98c10d6640 1 -- 172.21.15.29:0/1168095681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e800599c60 msgr2=0x55e8005aa770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.060+0000 7f98c10d6640 1 --2- 172.21.15.29:0/1168095681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e800599c60 0x55e8005aa770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.060+0000 7f98c10d6640 1 -- 172.21.15.29:0/1168095681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98bc1a1700 con 0x55e800599270 2024-03-11T19:10:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.060+0000 7f98c08d5640 1 --2- 172.21.15.29:0/1168095681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e800599c60 0x55e8005aa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:10:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.061+0000 7f98c10d6640 1 --2- 172.21.15.29:0/1168095681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 0x55e8005aa230 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f98b8006010 tx=0x7f98b8012270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5594d541e627c976 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.061+0000 7f98a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.061+0000 7f98a4ff9640 1 -- 172.21.15.29:0/1168095681 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b800ba00 con 0x55e800599270 2024-03-11T19:10:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.061+0000 7f98a4ff9640 1 -- 172.21.15.29:0/1168095681 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f98b8005850 con 0x55e800599270 2024-03-11T19:10:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.061+0000 7f98c5fe1740 1 -- 172.21.15.29:0/1168095681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e8002017e0 con 0x55e800599270 2024-03-11T19:10:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.061+0000 7f98a4ff9640 1 -- 172.21.15.29:0/1168095681 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98b802b440 con 0x55e800599270 2024-03-11T19:10:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.061+0000 7f98c5fe1740 10 client.15606 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.061+0000 7f98c5fe1740 1 -- 172.21.15.29:0/1168095681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e8004c28c0 con 0x55e800599270 2024-03-11T19:10:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.061+0000 7f98a4ff9640 1 -- 172.21.15.29:0/1168095681 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55e8002017e0 con 0x55e800599270 2024-03-11T19:10:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.061+0000 7f98a4ff9640 1 -- 172.21.15.29:0/1168095681 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f98b8010970 con 0x55e800599270 2024-03-11T19:10:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.061+0000 7f98c5fe1740 10 client.15606 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.061+0000 7f98c5fe1740 10 client.15606 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:10:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.061+0000 7f98c5fe1740 20 client.15606 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.061+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/1168095681 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e800606aa0 0x55e800608ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.061+0000 7f98c5fe1740 4 client.15606 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:10:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.061+0000 7f98c5fe1740 1 -- 172.21.15.29:0/1168095681 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e8001ad700 con 0x55e800606aa0 2024-03-11T19:10:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.062+0000 7f98c08d5640 1 --2- 172.21.15.29:0/1168095681 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e800606aa0 0x55e800608ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.062+0000 7f98c08d5640 1 --2- 172.21.15.29:0/1168095681 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e800606aa0 0x55e800608ec0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=694d7a6f960e26fd server_cookie=5fbd61e415d7a25b in_seq=0 out_seq=0 2024-03-11T19:10:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.062+0000 7f98a4ff9640 10 client.15606 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.062+0000 7f98a4ff9640 1 -- 172.21.15.29:0/1168095681 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e8001ad700 con 0x55e800606aa0 2024-03-11T19:10:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.062+0000 7f98a4ff9640 10 client.15606 handle_command_reply: tid=0 2024-03-11T19:10:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.062+0000 7f98c5fe1740 1 client.15606 shutdown 2024-03-11T19:10:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.062+0000 7f98c5fe1740 1 -- 172.21.15.29:0/1168095681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 msgr2=0x55e8005aa230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.062+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/1168095681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 0x55e8005aa230 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f98b8006010 tx=0x7f98b8012270 comp rx=0 tx=0).stop 2024-03-11T19:10:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98c5fe1740 1 -- 172.21.15.29:0/1168095681 shutdown_connections 2024-03-11T19:10:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/1168095681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e80059edd0 0x55e8005a8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/1168095681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e800599c60 0x55e8005aa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/1168095681 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e800606aa0 0x55e800608ec0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98c5fe1740 1 --2- 172.21.15.29:0/1168095681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e800599270 0x55e8005aa230 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98a4ff9640 0 client.15606 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98c5fe1740 1 -- 172.21.15.29:0/1168095681 >> 172.21.15.29:0/1168095681 conn(0x55e8005947e0 msgr2=0x55e800597a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98c5fe1740 1 -- 172.21.15.29:0/1168095681 shutdown_connections 2024-03-11T19:10:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98c5fe1740 1 -- 172.21.15.29:0/1168095681 wait complete. 2024-03-11T19:10:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98c5fe1740 20 client.15606 trim_cache size 0 max 16384 2024-03-11T19:10:34.072 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:10:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98c333a640 1 -- 172.21.15.29:0/3377616263 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f98a80a1ac0 msgr2=0x7f98a80a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98c333a640 1 --2- 172.21.15.29:0/3377616263 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f98a80a1ac0 0x7f98a80a3f80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f98b8012a20 tx=0x7f98b800bc20 comp rx=0 tx=0).stop 2024-03-11T19:10:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98c333a640 1 -- 172.21.15.29:0/3377616263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc106ec0 msgr2=0x7f98bc100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98c333a640 1 --2- 172.21.15.29:0/3377616263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc106ec0 0x7f98bc100150 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f98b4033000 tx=0x7f98b4032980 comp rx=0 tx=0).stop 2024-03-11T19:10:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98c333a640 1 -- 172.21.15.29:0/3377616263 shutdown_connections 2024-03-11T19:10:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98c333a640 1 --2- 172.21.15.29:0/3377616263 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f98a80a1ac0 0x7f98a80a3f80 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98c333a640 1 --2- 172.21.15.29:0/3377616263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98bc106ec0 0x7f98bc100150 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98c333a640 1 --2- 172.21.15.29:0/3377616263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98bc104590 0x7f98bc0ffc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98c333a640 1 --2- 172.21.15.29:0/3377616263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98bc101c60 0x7f98bc1015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.063+0000 7f98c333a640 1 -- 172.21.15.29:0/3377616263 >> 172.21.15.29:0/3377616263 conn(0x7f98bc0fb610 msgr2=0x7f98bc1025a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.064+0000 7f98c333a640 1 -- 172.21.15.29:0/3377616263 shutdown_connections 2024-03-11T19:10:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:34.064+0000 7f98c333a640 1 -- 172.21.15.29:0/3377616263 wait complete. 2024-03-11T19:10:35.081 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:10:35.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:10:35.082 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:10:35.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.188+0000 7feec5be2640 1 -- 172.21.15.29:0/652929344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec0101270 msgr2=0x7feec01016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:35.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.188+0000 7feec5be2640 1 --2- 172.21.15.29:0/652929344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec0101270 0x7feec01016f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7feeb4009900 tx=0x7feeb402bc30 comp rx=0 tx=0).stop 2024-03-11T19:10:35.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.188+0000 7feec5be2640 1 -- 172.21.15.29:0/652929344 shutdown_connections 2024-03-11T19:10:35.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.188+0000 7feec5be2640 1 --2- 172.21.15.29:0/652929344 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feec0101c30 0x7feec01084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.188+0000 7feec5be2640 1 --2- 172.21.15.29:0/652929344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec0101270 0x7feec01016f0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.188+0000 7feec5be2640 1 --2- 172.21.15.29:0/652929344 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feec0100080 0x7feec01004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.188+0000 7feec5be2640 1 -- 172.21.15.29:0/652929344 >> 172.21.15.29:0/652929344 conn(0x7feec00fb610 msgr2=0x7feec00fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:35.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.188+0000 7feec5be2640 1 -- 172.21.15.29:0/652929344 shutdown_connections 2024-03-11T19:10:35.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.188+0000 7feec5be2640 1 -- 172.21.15.29:0/652929344 wait complete. 2024-03-11T19:10:35.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.189+0000 7feec5be2640 1 Processor -- start 2024-03-11T19:10:35.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.189+0000 7feec5be2640 1 -- start start 2024-03-11T19:10:35.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.189+0000 7feec5be2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feec0100080 0x7feec0197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.189+0000 7feec5be2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feec0101270 0x7feec01976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.189+0000 7feec5be2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec0101c30 0x7feec019e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.189+0000 7feec5be2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec019ec70 con 0x7feec0101c30 2024-03-11T19:10:35.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.189+0000 7feec5be2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec019ede0 con 0x7feec0100080 2024-03-11T19:10:35.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.189+0000 7feec5be2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec019efa0 con 0x7feec0101270 2024-03-11T19:10:35.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.189+0000 7feebffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec0101c30 0x7feec019e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.189+0000 7feebffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec0101c30 0x7feec019e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42718/0 (socket says 172.21.15.29:42718) 2024-03-11T19:10:35.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.189+0000 7feebffff640 1 -- 172.21.15.29:0/3887116674 learned_addr learned my addr 172.21.15.29:0/3887116674 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:35.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.190+0000 7feebf7fe640 1 --2- 172.21.15.29:0/3887116674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feec0100080 0x7feec0197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.190+0000 7feebeffd640 1 --2- 172.21.15.29:0/3887116674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feec0101270 0x7feec01976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.190+0000 7feebffff640 1 -- 172.21.15.29:0/3887116674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feec0101270 msgr2=0x7feec01976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:35.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.190+0000 7feebffff640 1 --2- 172.21.15.29:0/3887116674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feec0101270 0x7feec01976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.190+0000 7feebffff640 1 -- 172.21.15.29:0/3887116674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feec0100080 msgr2=0x7feec0197180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:35.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.190+0000 7feebffff640 1 --2- 172.21.15.29:0/3887116674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feec0100080 0x7feec0197180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.190+0000 7feebffff640 1 -- 172.21.15.29:0/3887116674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb4009560 con 0x7feec0101c30 2024-03-11T19:10:35.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.190+0000 7feebeffd640 1 --2- 172.21.15.29:0/3887116674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feec0101270 0x7feec01976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:10:35.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.190+0000 7feebffff640 1 --2- 172.21.15.29:0/3887116674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec0101c30 0x7feec019e730 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7feeb800cce0 tx=0x7feeb800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:35.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.190+0000 7feebcff9640 1 -- 172.21.15.29:0/3887116674 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb800f450 con 0x7feec0101c30 2024-03-11T19:10:35.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.190+0000 7feebcff9640 1 -- 172.21.15.29:0/3887116674 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feeb800fa60 con 0x7feec0101c30 2024-03-11T19:10:35.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.190+0000 7feec5be2640 1 -- 172.21.15.29:0/3887116674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec019f280 con 0x7feec0101c30 2024-03-11T19:10:35.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.190+0000 7feebcff9640 1 -- 172.21.15.29:0/3887116674 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb8018b10 con 0x7feec0101c30 2024-03-11T19:10:35.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.190+0000 7feec5be2640 1 -- 172.21.15.29:0/3887116674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec019f790 con 0x7feec0101c30 2024-03-11T19:10:35.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.192+0000 7feebcff9640 1 -- 172.21.15.29:0/3887116674 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7feeb800fbd0 con 0x7feec0101c30 2024-03-11T19:10:35.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.193+0000 7feec87bb740 1 Processor -- start 2024-03-11T19:10:35.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.193+0000 7feec87bb740 1 -- start start 2024-03-11T19:10:35.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.193+0000 7feec87bb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d824984c0 0x556d824988e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.193+0000 7feec87bb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d82499690 0x556d82499b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.193+0000 7feec87bb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d8249a050 0x556d824a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.193+0000 7feebcff9640 1 --2- 172.21.15.29:0/3887116674 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7feea80a1b40 0x7feea80a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.193+0000 7feec87bb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d82407f90 con 0x556d8249a050 2024-03-11T19:10:35.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.193+0000 7feec87bb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d824054c0 con 0x556d82499690 2024-03-11T19:10:35.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.193+0000 7feec87bb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d82400ca0 con 0x556d824984c0 2024-03-11T19:10:35.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.193+0000 7feebffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d8249a050 0x556d824a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.193+0000 7feebeffd640 1 --2- 172.21.15.29:0/3887116674 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7feea80a1b40 0x7feea80a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.193+0000 7feebffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d8249a050 0x556d824a08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42730/0 (socket says 172.21.15.29:42730) 2024-03-11T19:10:35.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.193+0000 7feebffff640 1 -- 172.21.15.29:0/2937790309 learned_addr learned my addr 172.21.15.29:0/2937790309 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:35.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.194+0000 7feebf7fe640 1 --2- 172.21.15.29:0/2937790309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d82499690 0x556d82499b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.194+0000 7feebcff9640 1 -- 172.21.15.29:0/3887116674 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7feeb80cd920 con 0x7feec0101c30 2024-03-11T19:10:35.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.194+0000 7feebeffd640 1 --2- 172.21.15.29:0/2937790309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d824984c0 0x556d824988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.194+0000 7feebffff640 1 -- 172.21.15.29:0/2937790309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d824984c0 msgr2=0x556d824988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:35.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.194+0000 7feebffff640 1 --2- 172.21.15.29:0/2937790309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d824984c0 0x556d824988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.194+0000 7feebffff640 1 -- 172.21.15.29:0/2937790309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d82499690 msgr2=0x556d82499b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:35.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.194+0000 7feebffff640 1 --2- 172.21.15.29:0/2937790309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d82499690 0x556d82499b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.194+0000 7feebffff640 1 -- 172.21.15.29:0/2937790309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d823c30b0 con 0x556d8249a050 2024-03-11T19:10:35.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.194+0000 7feebeffd640 1 --2- 172.21.15.29:0/3887116674 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7feea80a1b40 0x7feea80a4000 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7feec0198690 tx=0x7feeb402ca40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:35.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.194+0000 7feebffff640 1 --2- 172.21.15.29:0/2937790309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d8249a050 0x556d824a08e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7feeb8096b90 tx=0x7feeb809a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f2fec31a9f2c66 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:35.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.194+0000 7feea5ffb640 1 -- 172.21.15.29:0/2937790309 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb80b8ea0 con 0x556d8249a050 2024-03-11T19:10:35.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.194+0000 7feea5ffb640 1 -- 172.21.15.29:0/2937790309 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feeb809ae30 con 0x556d8249a050 2024-03-11T19:10:35.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.194+0000 7feea5ffb640 1 -- 172.21.15.29:0/2937790309 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb8018c70 con 0x556d8249a050 2024-03-11T19:10:35.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.195+0000 7feec87bb740 1 -- 172.21.15.29:0/2937790309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d8249a050 msgr2=0x556d824a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:35.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.195+0000 7feec87bb740 1 --2- 172.21.15.29:0/2937790309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d8249a050 0x556d824a08e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7feeb8096b90 tx=0x7feeb809a270 comp rx=0 tx=0).stop 2024-03-11T19:10:35.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.195+0000 7feec87bb740 1 -- 172.21.15.29:0/2937790309 shutdown_connections 2024-03-11T19:10:35.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.195+0000 7feec87bb740 1 --2- 172.21.15.29:0/2937790309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d8249a050 0x556d824a08e0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.195+0000 7feec87bb740 1 --2- 172.21.15.29:0/2937790309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d82499690 0x556d82499b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.195+0000 7feec87bb740 1 --2- 172.21.15.29:0/2937790309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d824984c0 0x556d824988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.195+0000 7feec87bb740 1 -- 172.21.15.29:0/2937790309 >> 172.21.15.29:0/2937790309 conn(0x556d824936a0 msgr2=0x556d82493aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:35.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.195+0000 7feec87bb740 1 -- 172.21.15.29:0/2937790309 shutdown_connections 2024-03-11T19:10:35.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.195+0000 7feec87bb740 1 -- 172.21.15.29:0/2937790309 wait complete. 2024-03-11T19:10:35.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.195+0000 7feec87bb740 1 -- start start 2024-03-11T19:10:35.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feec87bb740 1 Processor -- start 2024-03-11T19:10:35.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feec87bb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d824984c0 0x556d824a2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feec87bb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d82499690 0x556d824a2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feec87bb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d8249a050 0x556d824a9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feec87bb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d823a5f40 con 0x556d824984c0 2024-03-11T19:10:35.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feec87bb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d820d44b0 con 0x556d8249a050 2024-03-11T19:10:35.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feec87bb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d82386670 con 0x556d82499690 2024-03-11T19:10:35.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feec87bb740 10 client.0 _ll_register_callbacks cb 0x556d823f3f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:35.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feebf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d824984c0 0x556d824a2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feebf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d824984c0 0x556d824a2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42746/0 (socket says 172.21.15.29:42746) 2024-03-11T19:10:35.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feebf7fe640 1 -- 172.21.15.29:0/1994043045 learned_addr learned my addr 172.21.15.29:0/1994043045 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:35.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feebffff640 1 --2- 172.21.15.29:0/1994043045 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d8249a050 0x556d824a9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feebeffd640 1 --2- 172.21.15.29:0/1994043045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d82499690 0x556d824a2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feebf7fe640 1 -- 172.21.15.29:0/1994043045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d82499690 msgr2=0x556d824a2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feebf7fe640 1 --2- 172.21.15.29:0/1994043045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d82499690 0x556d824a2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feebf7fe640 1 -- 172.21.15.29:0/1994043045 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d8249a050 msgr2=0x556d824a9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feebf7fe640 1 --2- 172.21.15.29:0/1994043045 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d8249a050 0x556d824a9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feebf7fe640 1 -- 172.21.15.29:0/1994043045 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec019f790 con 0x556d824984c0 2024-03-11T19:10:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feebeffd640 1 --2- 172.21.15.29:0/1994043045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d82499690 0x556d824a2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7feebf7fe640 1 --2- 172.21.15.29:0/1994043045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d824984c0 0x556d824a2740 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7feeb000bad0 tx=0x7feeb000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f399681c6aa279bf server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.196+0000 7fee9effd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.197+0000 7fee9effd640 1 -- 172.21.15.29:0/1994043045 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb0016070 con 0x556d824984c0 2024-03-11T19:10:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.197+0000 7fee9effd640 1 -- 172.21.15.29:0/1994043045 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7feeb0003e20 con 0x556d824984c0 2024-03-11T19:10:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.197+0000 7feec87bb740 1 -- 172.21.15.29:0/1994043045 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556d823daf40 con 0x556d824984c0 2024-03-11T19:10:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.197+0000 7fee9effd640 1 -- 172.21.15.29:0/1994043045 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7feeb0011440 con 0x556d824984c0 2024-03-11T19:10:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.197+0000 7feec87bb740 10 client.15618 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.197+0000 7feec87bb740 1 -- 172.21.15.29:0/1994043045 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x556d823ff320 con 0x556d824984c0 2024-03-11T19:10:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.197+0000 7fee9effd640 1 -- 172.21.15.29:0/1994043045 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x556d823daf40 con 0x556d824984c0 2024-03-11T19:10:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.197+0000 7fee9effd640 1 -- 172.21.15.29:0/1994043045 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7feeb001b750 con 0x556d824984c0 2024-03-11T19:10:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.197+0000 7feec87bb740 10 client.15618 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.197+0000 7feec87bb740 10 client.15618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:10:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.197+0000 7feec87bb740 20 client.15618 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.197+0000 7feec87bb740 1 --2- 172.21.15.29:0/1994043045 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x556d824ae990 0x556d824b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.197+0000 7feec87bb740 4 client.15618 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:10:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.197+0000 7feec87bb740 1 -- 172.21.15.29:0/1994043045 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556d823f7310 con 0x556d824ae990 2024-03-11T19:10:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.197+0000 7feebeffd640 1 --2- 172.21.15.29:0/1994043045 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x556d824ae990 0x556d824b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.198+0000 7feebeffd640 1 --2- 172.21.15.29:0/1994043045 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x556d824ae990 0x556d824b0db0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90a9481df0d8929a server_cookie=596bc5d7ff661ac2 in_seq=0 out_seq=0 2024-03-11T19:10:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.198+0000 7fee9effd640 10 client.15618 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.199+0000 7fee9effd640 1 -- 172.21.15.29:0/1994043045 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556d823f7310 con 0x556d824ae990 2024-03-11T19:10:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.199+0000 7fee9effd640 10 client.15618 handle_command_reply: tid=0 2024-03-11T19:10:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.199+0000 7feec87bb740 1 client.15618 shutdown 2024-03-11T19:10:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.199+0000 7feec87bb740 1 -- 172.21.15.29:0/1994043045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d824984c0 msgr2=0x556d824a2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.199+0000 7feec87bb740 1 --2- 172.21.15.29:0/1994043045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d824984c0 0x556d824a2740 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7feeb000bad0 tx=0x7feeb000bf90 comp rx=0 tx=0).stop 2024-03-11T19:10:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.199+0000 7feec87bb740 1 -- 172.21.15.29:0/1994043045 shutdown_connections 2024-03-11T19:10:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.199+0000 7feec87bb740 1 --2- 172.21.15.29:0/1994043045 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d8249a050 0x556d824a9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.199+0000 7feec87bb740 1 --2- 172.21.15.29:0/1994043045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d82499690 0x556d824a2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.199+0000 7feec87bb740 1 --2- 172.21.15.29:0/1994043045 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x556d824ae990 0x556d824b0db0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.199+0000 7feec87bb740 1 --2- 172.21.15.29:0/1994043045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d824984c0 0x556d824a2740 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.199+0000 7fee9effd640 0 client.15618 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:35.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.199+0000 7feec87bb740 1 -- 172.21.15.29:0/1994043045 >> 172.21.15.29:0/1994043045 conn(0x556d824936a0 msgr2=0x556d824928e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:35.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.199+0000 7feec87bb740 1 -- 172.21.15.29:0/1994043045 shutdown_connections 2024-03-11T19:10:35.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.199+0000 7feec87bb740 1 -- 172.21.15.29:0/1994043045 wait complete. 2024-03-11T19:10:35.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.199+0000 7feec87bb740 20 client.15618 trim_cache size 0 max 16384 2024-03-11T19:10:35.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.228+0000 7feec87bb740 1 Processor -- start 2024-03-11T19:10:35.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.228+0000 7feec87bb740 1 -- start start 2024-03-11T19:10:35.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.228+0000 7feec87bb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d824984c0 0x556d82492050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.228+0000 7feec87bb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d82499690 0x556d824a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.228+0000 7feec87bb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d8249a050 0x556d82492590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.228+0000 7feec87bb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d824980c0 con 0x556d82499690 2024-03-11T19:10:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.228+0000 7feec87bb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d824acf70 con 0x556d8249a050 2024-03-11T19:10:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.228+0000 7feec87bb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d82499130 con 0x556d824984c0 2024-03-11T19:10:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.228+0000 7feebeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d82499690 0x556d824a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.228+0000 7feebeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d82499690 0x556d824a9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42748/0 (socket says 172.21.15.29:42748) 2024-03-11T19:10:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.228+0000 7feebeffd640 1 -- 172.21.15.29:0/233142995 learned_addr learned my addr 172.21.15.29:0/233142995 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.228+0000 7feebf7fe640 1 --2- 172.21.15.29:0/233142995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d824984c0 0x556d82492050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.228+0000 7feebffff640 1 --2- 172.21.15.29:0/233142995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d8249a050 0x556d82492590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.228+0000 7feebeffd640 1 -- 172.21.15.29:0/233142995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d824984c0 msgr2=0x556d82492050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.228+0000 7feebeffd640 1 --2- 172.21.15.29:0/233142995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d824984c0 0x556d82492050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.228+0000 7feebeffd640 1 -- 172.21.15.29:0/233142995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d8249a050 msgr2=0x556d82492590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.228+0000 7feebeffd640 1 --2- 172.21.15.29:0/233142995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d8249a050 0x556d82492590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.228+0000 7feebeffd640 1 -- 172.21.15.29:0/233142995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d8239bc00 con 0x556d82499690 2024-03-11T19:10:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.229+0000 7feebeffd640 1 --2- 172.21.15.29:0/233142995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d82499690 0x556d824a9c80 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7feeb402c170 tx=0x7feeb404e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4aad553048edfaf server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.229+0000 7feea57fa640 1 -- 172.21.15.29:0/233142995 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb4044040 con 0x556d82499690 2024-03-11T19:10:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.229+0000 7feea57fa640 1 -- 172.21.15.29:0/233142995 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feeb4055040 con 0x556d82499690 2024-03-11T19:10:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.229+0000 7feea57fa640 1 -- 172.21.15.29:0/233142995 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb403e740 con 0x556d82499690 2024-03-11T19:10:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.229+0000 7feec87bb740 1 -- 172.21.15.29:0/233142995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d82499690 msgr2=0x556d824a9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.229+0000 7feec87bb740 1 --2- 172.21.15.29:0/233142995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d82499690 0x556d824a9c80 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7feeb402c170 tx=0x7feeb404e000 comp rx=0 tx=0).stop 2024-03-11T19:10:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.229+0000 7feec87bb740 1 -- 172.21.15.29:0/233142995 shutdown_connections 2024-03-11T19:10:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.229+0000 7feec87bb740 1 --2- 172.21.15.29:0/233142995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d8249a050 0x556d82492590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.229+0000 7feec87bb740 1 --2- 172.21.15.29:0/233142995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d82499690 0x556d824a9c80 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.229+0000 7feec87bb740 1 --2- 172.21.15.29:0/233142995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d824984c0 0x556d82492050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.229+0000 7feec87bb740 1 -- 172.21.15.29:0/233142995 >> 172.21.15.29:0/233142995 conn(0x556d824936a0 msgr2=0x556d8249d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.230+0000 7feec87bb740 1 -- 172.21.15.29:0/233142995 shutdown_connections 2024-03-11T19:10:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.230+0000 7feec87bb740 1 -- 172.21.15.29:0/233142995 wait complete. 2024-03-11T19:10:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.230+0000 7feec87bb740 1 -- start start 2024-03-11T19:10:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.230+0000 7feec87bb740 1 Processor -- start 2024-03-11T19:10:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.230+0000 7feec87bb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d824984c0 0x556d824a9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.230+0000 7feec87bb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d82499690 0x556d824a6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.230+0000 7feec87bb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d8249a050 0x556d824a69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.230+0000 7feec87bb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d824a6f10 con 0x556d8249a050 2024-03-11T19:10:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.230+0000 7feec87bb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d824a7080 con 0x556d824984c0 2024-03-11T19:10:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.230+0000 7feec87bb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d82504ec0 con 0x556d82499690 2024-03-11T19:10:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.230+0000 7feec87bb740 10 client.0 _ll_register_callbacks cb 0x7feeb8018510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.230+0000 7feebffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d8249a050 0x556d824a69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.230+0000 7feebffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d8249a050 0x556d824a69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42756/0 (socket says 172.21.15.29:42756) 2024-03-11T19:10:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.230+0000 7feebffff640 1 -- 172.21.15.29:0/2740222451 learned_addr learned my addr 172.21.15.29:0/2740222451 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.231+0000 7feebeffd640 1 --2- 172.21.15.29:0/2740222451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d82499690 0x556d824a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.231+0000 7feebf7fe640 1 --2- 172.21.15.29:0/2740222451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d824984c0 0x556d824a9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.231+0000 7feebffff640 1 -- 172.21.15.29:0/2740222451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d82499690 msgr2=0x556d824a6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.231+0000 7feebffff640 1 --2- 172.21.15.29:0/2740222451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d82499690 0x556d824a6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.231+0000 7feebffff640 1 -- 172.21.15.29:0/2740222451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d824984c0 msgr2=0x556d824a9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.231+0000 7feebffff640 1 --2- 172.21.15.29:0/2740222451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d824984c0 0x556d824a9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.231+0000 7feebffff640 1 -- 172.21.15.29:0/2740222451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb4047040 con 0x556d8249a050 2024-03-11T19:10:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.231+0000 7feebffff640 1 --2- 172.21.15.29:0/2740222451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d8249a050 0x556d824a69d0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7feeb809a430 tx=0x7feeb80cb210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db1dabd1859dbb server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.231+0000 7fee9effd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.231+0000 7fee9effd640 1 -- 172.21.15.29:0/2740222451 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb80b8ea0 con 0x556d8249a050 2024-03-11T19:10:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.231+0000 7fee9effd640 1 -- 172.21.15.29:0/2740222451 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7feeb809e880 con 0x556d8249a050 2024-03-11T19:10:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.231+0000 7feec87bb740 1 -- 172.21.15.29:0/2740222451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556d82426900 con 0x556d8249a050 2024-03-11T19:10:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.231+0000 7fee9effd640 1 -- 172.21.15.29:0/2740222451 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7feeb809ae30 con 0x556d8249a050 2024-03-11T19:10:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.231+0000 7feec87bb740 10 client.15624 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.231+0000 7feec87bb740 1 -- 172.21.15.29:0/2740222451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x556d823c1910 con 0x556d8249a050 2024-03-11T19:10:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.231+0000 7fee9effd640 1 -- 172.21.15.29:0/2740222451 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x556d82426900 con 0x556d8249a050 2024-03-11T19:10:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.232+0000 7fee9effd640 1 -- 172.21.15.29:0/2740222451 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7feeb8016020 con 0x556d8249a050 2024-03-11T19:10:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.232+0000 7feec87bb740 10 client.15624 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.232+0000 7feec87bb740 10 client.15624 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:10:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.232+0000 7feec87bb740 20 client.15624 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:35.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.232+0000 7feec87bb740 1 --2- 172.21.15.29:0/2740222451 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x556d825065b0 0x556d825089d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.232+0000 7feec87bb740 4 client.15624 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:10:35.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.232+0000 7feec87bb740 1 -- 172.21.15.29:0/2740222451 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x556d82100860 con 0x556d825065b0 2024-03-11T19:10:35.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.232+0000 7feebf7fe640 1 --2- 172.21.15.29:0/2740222451 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x556d825065b0 0x556d825089d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.232+0000 7fee9effd640 10 client.15624 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:35.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.232+0000 7feebf7fe640 1 --2- 172.21.15.29:0/2740222451 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x556d825065b0 0x556d825089d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50a8e8e3c001573e server_cookie=e7b052fc3372b0a2 in_seq=0 out_seq=0 2024-03-11T19:10:35.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.233+0000 7fee9effd640 1 -- 172.21.15.29:0/2740222451 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x556d82100860 con 0x556d825065b0 2024-03-11T19:10:35.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.233+0000 7fee9effd640 10 client.15624 handle_command_reply: tid=0 2024-03-11T19:10:35.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.233+0000 7feec87bb740 1 client.15624 shutdown 2024-03-11T19:10:35.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec87bb740 1 -- 172.21.15.29:0/2740222451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d8249a050 msgr2=0x556d824a69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:35.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec87bb740 1 --2- 172.21.15.29:0/2740222451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d8249a050 0x556d824a69d0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7feeb809a430 tx=0x7feeb80cb210 comp rx=0 tx=0).stop 2024-03-11T19:10:35.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec87bb740 1 -- 172.21.15.29:0/2740222451 shutdown_connections 2024-03-11T19:10:35.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec87bb740 1 --2- 172.21.15.29:0/2740222451 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x556d825065b0 0x556d825089d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec87bb740 1 --2- 172.21.15.29:0/2740222451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556d8249a050 0x556d824a69d0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7fee9effd640 0 client.15624 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:35.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec87bb740 1 --2- 172.21.15.29:0/2740222451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556d82499690 0x556d824a6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec87bb740 1 --2- 172.21.15.29:0/2740222451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556d824984c0 0x556d824a9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec87bb740 1 -- 172.21.15.29:0/2740222451 >> 172.21.15.29:0/2740222451 conn(0x556d824936a0 msgr2=0x556d82496900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:35.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec87bb740 1 -- 172.21.15.29:0/2740222451 shutdown_connections 2024-03-11T19:10:35.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec87bb740 1 -- 172.21.15.29:0/2740222451 wait complete. 2024-03-11T19:10:35.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec87bb740 20 client.15624 trim_cache size 0 max 16384 2024-03-11T19:10:35.243 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:10:35.243 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:10:35.243 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "8e89ca3b-fe57-418a-9b51-c8d3d7f188c9", 2024-03-11T19:10:35.243 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:10:35.243 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:10:35.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec5be2640 1 -- 172.21.15.29:0/3887116674 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7feea80a1b40 msgr2=0x7feea80a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:35.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec5be2640 1 --2- 172.21.15.29:0/3887116674 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7feea80a1b40 0x7feea80a4000 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7feec0198690 tx=0x7feeb402ca40 comp rx=0 tx=0).stop 2024-03-11T19:10:35.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec5be2640 1 -- 172.21.15.29:0/3887116674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec0101c30 msgr2=0x7feec019e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:35.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec5be2640 1 --2- 172.21.15.29:0/3887116674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec0101c30 0x7feec019e730 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7feeb800cce0 tx=0x7feeb800a620 comp rx=0 tx=0).stop 2024-03-11T19:10:35.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec5be2640 1 -- 172.21.15.29:0/3887116674 shutdown_connections 2024-03-11T19:10:35.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec5be2640 1 --2- 172.21.15.29:0/3887116674 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7feea80a1b40 0x7feea80a4000 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec5be2640 1 --2- 172.21.15.29:0/3887116674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec0101c30 0x7feec019e730 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec5be2640 1 --2- 172.21.15.29:0/3887116674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feec0101270 0x7feec01976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec5be2640 1 --2- 172.21.15.29:0/3887116674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feec0100080 0x7feec0197180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.234+0000 7feec5be2640 1 -- 172.21.15.29:0/3887116674 >> 172.21.15.29:0/3887116674 conn(0x7feec00fb610 msgr2=0x7feec00fd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:35.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.235+0000 7feec5be2640 1 -- 172.21.15.29:0/3887116674 shutdown_connections 2024-03-11T19:10:35.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.235+0000 7feec5be2640 1 -- 172.21.15.29:0/3887116674 wait complete. 2024-03-11T19:10:35.248 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:10:35.249 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:10:35.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.353+0000 7fa0a849d640 1 -- 172.21.15.29:0/3985209154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0a0101c30 msgr2=0x7fa0a01084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:35.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.353+0000 7fa0a849d640 1 --2- 172.21.15.29:0/3985209154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0a0101c30 0x7fa0a01084c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fa098009ca0 tx=0x7fa09802c0f0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.353+0000 7fa0a849d640 1 -- 172.21.15.29:0/3985209154 shutdown_connections 2024-03-11T19:10:35.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.353+0000 7fa0a849d640 1 --2- 172.21.15.29:0/3985209154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0a0101c30 0x7fa0a01084c0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.353+0000 7fa0a849d640 1 --2- 172.21.15.29:0/3985209154 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0a0101270 0x7fa0a01016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.353+0000 7fa0a849d640 1 --2- 172.21.15.29:0/3985209154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0a0100080 0x7fa0a01004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.353+0000 7fa0a849d640 1 -- 172.21.15.29:0/3985209154 >> 172.21.15.29:0/3985209154 conn(0x7fa0a00fb5f0 msgr2=0x7fa0a00fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.354+0000 7fa0a849d640 1 -- 172.21.15.29:0/3985209154 shutdown_connections 2024-03-11T19:10:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.354+0000 7fa0a849d640 1 -- 172.21.15.29:0/3985209154 wait complete. 2024-03-11T19:10:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.354+0000 7fa0a849d640 1 Processor -- start 2024-03-11T19:10:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.354+0000 7fa0a849d640 1 -- start start 2024-03-11T19:10:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.354+0000 7fa0a849d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0a0100080 0x7fa0a0195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.354+0000 7fa0a849d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0a0101270 0x7fa0a0196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.354+0000 7fa0a849d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0a0101c30 0x7fa0a019d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.354+0000 7fa0a849d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a019dac0 con 0x7fa0a0101270 2024-03-11T19:10:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.354+0000 7fa0a849d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a019dc30 con 0x7fa0a0101c30 2024-03-11T19:10:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.354+0000 7fa0a849d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a019ddf0 con 0x7fa0a0100080 2024-03-11T19:10:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.354+0000 7fa0a5a38640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0a0101270 0x7fa0a0196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.354+0000 7fa0a5a38640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0a0101270 0x7fa0a0196510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42786/0 (socket says 172.21.15.29:42786) 2024-03-11T19:10:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.354+0000 7fa0a5a38640 1 -- 172.21.15.29:0/1193893157 learned_addr learned my addr 172.21.15.29:0/1193893157 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.354+0000 7fa0a6239640 1 --2- 172.21.15.29:0/1193893157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0a0100080 0x7fa0a0195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.355+0000 7fa0a6a3a640 1 --2- 172.21.15.29:0/1193893157 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0a0101c30 0x7fa0a019d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.355+0000 7fa0a5a38640 1 -- 172.21.15.29:0/1193893157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0a0100080 msgr2=0x7fa0a0195fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.355+0000 7fa0a5a38640 1 --2- 172.21.15.29:0/1193893157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0a0100080 0x7fa0a0195fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.355+0000 7fa0a5a38640 1 -- 172.21.15.29:0/1193893157 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0a0101c30 msgr2=0x7fa0a019d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.355+0000 7fa0a5a38640 1 --2- 172.21.15.29:0/1193893157 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0a0101c30 0x7fa0a019d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.355+0000 7fa0a5a38640 1 -- 172.21.15.29:0/1193893157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa098009900 con 0x7fa0a0101270 2024-03-11T19:10:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.355+0000 7fa0a6a3a640 1 --2- 172.21.15.29:0/1193893157 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0a0101c30 0x7fa0a019d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:10:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.355+0000 7fa0a5a38640 1 --2- 172.21.15.29:0/1193893157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0a0101270 0x7fa0a0196510 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fa09000d950 tx=0x7fa09000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.355+0000 7fa0977fe640 1 -- 172.21.15.29:0/1193893157 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa090009c30 con 0x7fa0a0101270 2024-03-11T19:10:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.355+0000 7fa0977fe640 1 -- 172.21.15.29:0/1193893157 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa090009d90 con 0x7fa0a0101270 2024-03-11T19:10:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.355+0000 7fa0a849d640 1 -- 172.21.15.29:0/1193893157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a019e0d0 con 0x7fa0a0101270 2024-03-11T19:10:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.355+0000 7fa0977fe640 1 -- 172.21.15.29:0/1193893157 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09000f6b0 con 0x7fa0a0101270 2024-03-11T19:10:35.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.355+0000 7fa0a849d640 1 -- 172.21.15.29:0/1193893157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a019e5e0 con 0x7fa0a0101270 2024-03-11T19:10:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.357+0000 7fa0977fe640 1 -- 172.21.15.29:0/1193893157 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fa090010aa0 con 0x7fa0a0101270 2024-03-11T19:10:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.358+0000 7fa0977fe640 1 --2- 172.21.15.29:0/1193893157 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa08c0a1ac0 0x7fa08c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.358+0000 7fa0ab076740 1 Processor -- start 2024-03-11T19:10:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.358+0000 7fa0a6239640 1 --2- 172.21.15.29:0/1193893157 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa08c0a1ac0 0x7fa08c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.358+0000 7fa0977fe640 1 -- 172.21.15.29:0/1193893157 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fa0900cd1b0 con 0x7fa0a0101270 2024-03-11T19:10:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.358+0000 7fa0ab076740 1 -- start start 2024-03-11T19:10:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.358+0000 7fa0ab076740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2eba4e0 0x5559c2eba900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.358+0000 7fa0ab076740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 0x5559c2ebf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.358+0000 7fa0ab076740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2ec0040 0x5559c2ec24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.358+0000 7fa0ab076740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559c2e28280 con 0x5559c2ebaed0 2024-03-11T19:10:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.358+0000 7fa0ab076740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559c2e257b0 con 0x5559c2eba4e0 2024-03-11T19:10:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.358+0000 7fa0ab076740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559c2e20f90 con 0x5559c2ec0040 2024-03-11T19:10:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.359+0000 7fa0a5a38640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 0x5559c2ebf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.359+0000 7fa0a6239640 1 --2- 172.21.15.29:0/1193893157 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa08c0a1ac0 0x7fa08c0a3f80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa0a01010d0 tx=0x7fa09c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.359+0000 7fa0a5a38640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 0x5559c2ebf9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42794/0 (socket says 172.21.15.29:42794) 2024-03-11T19:10:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.359+0000 7fa0a5a38640 1 -- 172.21.15.29:0/1529433695 learned_addr learned my addr 172.21.15.29:0/1529433695 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.359+0000 7fa0a6a3a640 1 --2- 172.21.15.29:0/1529433695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2ec0040 0x5559c2ec24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.359+0000 7fa0a5a38640 1 -- 172.21.15.29:0/1529433695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2ec0040 msgr2=0x5559c2ec24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.359+0000 7fa0a5a38640 1 --2- 172.21.15.29:0/1529433695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2ec0040 0x5559c2ec24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.359+0000 7fa0a5a38640 1 -- 172.21.15.29:0/1529433695 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2eba4e0 msgr2=0x5559c2eba900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.359+0000 7fa0a5a38640 1 --2- 172.21.15.29:0/1529433695 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2eba4e0 0x5559c2eba900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.359+0000 7fa0a5a38640 1 -- 172.21.15.29:0/1529433695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559c2ddf570 con 0x5559c2ebaed0 2024-03-11T19:10:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.359+0000 7fa0a5a38640 1 --2- 172.21.15.29:0/1529433695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 0x5559c2ebf9d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa09000bfd0 tx=0x7fa09009a900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e9be25d9de5ea39 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.359+0000 7fa094bfe640 1 -- 172.21.15.29:0/1529433695 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0900a2860 con 0x5559c2ebaed0 2024-03-11T19:10:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.359+0000 7fa094bfe640 1 -- 172.21.15.29:0/1529433695 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa090096730 con 0x5559c2ebaed0 2024-03-11T19:10:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.359+0000 7fa094bfe640 1 -- 172.21.15.29:0/1529433695 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa090096030 con 0x5559c2ebaed0 2024-03-11T19:10:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.360+0000 7fa0ab076740 1 -- 172.21.15.29:0/1529433695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 msgr2=0x5559c2ebf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.360+0000 7fa0ab076740 1 --2- 172.21.15.29:0/1529433695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 0x5559c2ebf9d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa09000bfd0 tx=0x7fa09009a900 comp rx=0 tx=0).stop 2024-03-11T19:10:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.360+0000 7fa0ab076740 1 -- 172.21.15.29:0/1529433695 shutdown_connections 2024-03-11T19:10:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.360+0000 7fa0ab076740 1 --2- 172.21.15.29:0/1529433695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2ec0040 0x5559c2ec24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.360+0000 7fa0ab076740 1 --2- 172.21.15.29:0/1529433695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 0x5559c2ebf9d0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.360+0000 7fa0ab076740 1 --2- 172.21.15.29:0/1529433695 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2eba4e0 0x5559c2eba900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.360+0000 7fa0ab076740 1 -- 172.21.15.29:0/1529433695 >> 172.21.15.29:0/1529433695 conn(0x5559c2eb37f0 msgr2=0x5559c2eb3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.360+0000 7fa0ab076740 1 -- 172.21.15.29:0/1529433695 shutdown_connections 2024-03-11T19:10:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.360+0000 7fa0ab076740 1 -- 172.21.15.29:0/1529433695 wait complete. 2024-03-11T19:10:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.360+0000 7fa0ab076740 1 -- start start 2024-03-11T19:10:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa0ab076740 1 Processor -- start 2024-03-11T19:10:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa0ab076740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2eba4e0 0x5559c2ec43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa0ab076740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 0x5559c2ec4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa0ab076740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2ec0040 0x5559c2ecb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa0ab076740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559c2af4430 con 0x5559c2ebaed0 2024-03-11T19:10:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa0ab076740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559c2ab0e00 con 0x5559c2ec0040 2024-03-11T19:10:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa0ab076740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559c2da5fd0 con 0x5559c2eba4e0 2024-03-11T19:10:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa0ab076740 10 client.0 _ll_register_callbacks cb 0x5559c2dfbfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa0a5a38640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 0x5559c2ec4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa0a5a38640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 0x5559c2ec4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42798/0 (socket says 172.21.15.29:42798) 2024-03-11T19:10:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa0a5a38640 1 -- 172.21.15.29:0/2953229265 learned_addr learned my addr 172.21.15.29:0/2953229265 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa0a6239640 1 --2- 172.21.15.29:0/2953229265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2eba4e0 0x5559c2ec43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa0a6a3a640 1 --2- 172.21.15.29:0/2953229265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2ec0040 0x5559c2ecb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa0a5a38640 1 -- 172.21.15.29:0/2953229265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2eba4e0 msgr2=0x5559c2ec43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa0a5a38640 1 --2- 172.21.15.29:0/2953229265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2eba4e0 0x5559c2ec43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa0a5a38640 1 -- 172.21.15.29:0/2953229265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2ec0040 msgr2=0x5559c2ecb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa0a5a38640 1 --2- 172.21.15.29:0/2953229265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2ec0040 0x5559c2ecb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa0a5a38640 1 -- 172.21.15.29:0/2953229265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a019e5e0 con 0x5559c2ebaed0 2024-03-11T19:10:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa0a5a38640 1 --2- 172.21.15.29:0/2953229265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 0x5559c2ec4930 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa090096a90 tx=0x7fa090000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebefed99f9f440aa server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.361+0000 7fa089ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.362+0000 7fa089ffb640 1 -- 172.21.15.29:0/2953229265 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0900a2860 con 0x5559c2ebaed0 2024-03-11T19:10:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.362+0000 7fa0ab076740 1 -- 172.21.15.29:0/2953229265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5559c2b207e0 con 0x5559c2ebaed0 2024-03-11T19:10:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.362+0000 7fa089ffb640 1 -- 172.21.15.29:0/2953229265 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fa090099720 con 0x5559c2ebaed0 2024-03-11T19:10:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.362+0000 7fa089ffb640 1 -- 172.21.15.29:0/2953229265 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa090095e10 con 0x5559c2ebaed0 2024-03-11T19:10:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.362+0000 7fa0ab076740 10 client.15636 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.362+0000 7fa0ab076740 1 -- 172.21.15.29:0/2953229265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5559c2e1f610 con 0x5559c2ebaed0 2024-03-11T19:10:35.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.362+0000 7fa089ffb640 1 -- 172.21.15.29:0/2953229265 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5559c2b207e0 con 0x5559c2ebaed0 2024-03-11T19:10:35.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.362+0000 7fa089ffb640 1 -- 172.21.15.29:0/2953229265 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa0900b84c0 con 0x5559c2ebaed0 2024-03-11T19:10:35.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.362+0000 7fa0ab076740 10 client.15636 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:35.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.362+0000 7fa0ab076740 10 client.15636 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:10:35.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.362+0000 7fa0ab076740 20 client.15636 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:35.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.362+0000 7fa0ab076740 1 --2- 172.21.15.29:0/2953229265 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5559c2ed02e0 0x5559c2ed26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.362+0000 7fa0ab076740 4 client.15636 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:10:35.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.362+0000 7fa0ab076740 1 -- 172.21.15.29:0/2953229265 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5559c2b5d720 con 0x5559c2ed02e0 2024-03-11T19:10:35.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.362+0000 7fa0a6239640 1 --2- 172.21.15.29:0/2953229265 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5559c2ed02e0 0x5559c2ed26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.363+0000 7fa089ffb640 10 client.15636 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:35.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.363+0000 7fa0a6239640 1 --2- 172.21.15.29:0/2953229265 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5559c2ed02e0 0x5559c2ed26d0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c836efdd7ca30c63 server_cookie=f15e93752acf488a in_seq=0 out_seq=0 2024-03-11T19:10:35.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.363+0000 7fa089ffb640 1 -- 172.21.15.29:0/2953229265 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5559c2b5d720 con 0x5559c2ed02e0 2024-03-11T19:10:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.363+0000 7fa089ffb640 10 client.15636 handle_command_reply: tid=0 2024-03-11T19:10:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.363+0000 7fa0ab076740 1 client.15636 shutdown 2024-03-11T19:10:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.364+0000 7fa0ab076740 1 -- 172.21.15.29:0/2953229265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 msgr2=0x5559c2ec4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.364+0000 7fa0ab076740 1 --2- 172.21.15.29:0/2953229265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 0x5559c2ec4930 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa090096a90 tx=0x7fa090000bc0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.364+0000 7fa0ab076740 1 -- 172.21.15.29:0/2953229265 shutdown_connections 2024-03-11T19:10:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.364+0000 7fa0ab076740 1 --2- 172.21.15.29:0/2953229265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2ec0040 0x5559c2ecb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.364+0000 7fa0ab076740 1 --2- 172.21.15.29:0/2953229265 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5559c2ed02e0 0x5559c2ed26d0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.364+0000 7fa0ab076740 1 --2- 172.21.15.29:0/2953229265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 0x5559c2ec4930 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.364+0000 7fa089ffb640 0 client.15636 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.364+0000 7fa0ab076740 1 --2- 172.21.15.29:0/2953229265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2eba4e0 0x5559c2ec43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.364+0000 7fa0ab076740 1 -- 172.21.15.29:0/2953229265 >> 172.21.15.29:0/2953229265 conn(0x5559c2eb37f0 msgr2=0x5559c2eb2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.364+0000 7fa0ab076740 1 -- 172.21.15.29:0/2953229265 shutdown_connections 2024-03-11T19:10:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.364+0000 7fa0ab076740 1 -- 172.21.15.29:0/2953229265 wait complete. 2024-03-11T19:10:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.364+0000 7fa0ab076740 20 client.15636 trim_cache size 0 max 16384 2024-03-11T19:10:35.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.379+0000 7fa0ab076740 1 Processor -- start 2024-03-11T19:10:35.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.379+0000 7fa0ab076740 1 -- start start 2024-03-11T19:10:35.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.379+0000 7fa0ab076740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2eba4e0 0x5559c2ebc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.379+0000 7fa0ab076740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 0x5559c2ecb930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.379+0000 7fa0ab076740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2ec0040 0x5559c2ebc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.379+0000 7fa0ab076740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559c2ec2660 con 0x5559c2ebaed0 2024-03-11T19:10:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.379+0000 7fa0ab076740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559c2ebfa10 con 0x5559c2ec0040 2024-03-11T19:10:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.379+0000 7fa0ab076740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559c2eaeeb0 con 0x5559c2eba4e0 2024-03-11T19:10:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.379+0000 7fa0a5a38640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 0x5559c2ecb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.379+0000 7fa0a5a38640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 0x5559c2ecb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42802/0 (socket says 172.21.15.29:42802) 2024-03-11T19:10:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.379+0000 7fa0a5a38640 1 -- 172.21.15.29:0/4213462427 learned_addr learned my addr 172.21.15.29:0/4213462427 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.379+0000 7fa0a6a3a640 1 --2- 172.21.15.29:0/4213462427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2ec0040 0x5559c2ebc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.379+0000 7fa0a6239640 1 --2- 172.21.15.29:0/4213462427 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2eba4e0 0x5559c2ebc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.379+0000 7fa0a5a38640 1 -- 172.21.15.29:0/4213462427 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2eba4e0 msgr2=0x5559c2ebc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.379+0000 7fa0a5a38640 1 --2- 172.21.15.29:0/4213462427 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2eba4e0 0x5559c2ebc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.379+0000 7fa0a5a38640 1 -- 172.21.15.29:0/4213462427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2ec0040 msgr2=0x5559c2ebc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.380+0000 7fa0a5a38640 1 --2- 172.21.15.29:0/4213462427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2ec0040 0x5559c2ebc6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.380+0000 7fa0a5a38640 1 -- 172.21.15.29:0/4213462427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559c2dbbac0 con 0x5559c2ebaed0 2024-03-11T19:10:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.380+0000 7fa0a5a38640 1 --2- 172.21.15.29:0/4213462427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 0x5559c2ecb930 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa090098370 tx=0x7fa090096490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db319614bc259266 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.380+0000 7fa08bfff640 1 -- 172.21.15.29:0/4213462427 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0900a2860 con 0x5559c2ebaed0 2024-03-11T19:10:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.380+0000 7fa08bfff640 1 -- 172.21.15.29:0/4213462427 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa0900a25e0 con 0x5559c2ebaed0 2024-03-11T19:10:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.380+0000 7fa08bfff640 1 -- 172.21.15.29:0/4213462427 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09009f4f0 con 0x5559c2ebaed0 2024-03-11T19:10:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.380+0000 7fa0ab076740 1 -- 172.21.15.29:0/4213462427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 msgr2=0x5559c2ecb930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.380+0000 7fa0ab076740 1 --2- 172.21.15.29:0/4213462427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 0x5559c2ecb930 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa090098370 tx=0x7fa090096490 comp rx=0 tx=0).stop 2024-03-11T19:10:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.381+0000 7fa0ab076740 1 -- 172.21.15.29:0/4213462427 shutdown_connections 2024-03-11T19:10:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.381+0000 7fa0ab076740 1 --2- 172.21.15.29:0/4213462427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2ec0040 0x5559c2ebc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.381+0000 7fa0ab076740 1 --2- 172.21.15.29:0/4213462427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ebaed0 0x5559c2ecb930 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.381+0000 7fa0ab076740 1 --2- 172.21.15.29:0/4213462427 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2eba4e0 0x5559c2ebc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.381+0000 7fa0ab076740 1 -- 172.21.15.29:0/4213462427 >> 172.21.15.29:0/4213462427 conn(0x5559c2eb37f0 msgr2=0x5559c2ec43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.381+0000 7fa0ab076740 1 -- 172.21.15.29:0/4213462427 shutdown_connections 2024-03-11T19:10:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.381+0000 7fa0ab076740 1 -- 172.21.15.29:0/4213462427 wait complete. 2024-03-11T19:10:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.381+0000 7fa0ab076740 1 -- start start 2024-03-11T19:10:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.381+0000 7fa0ab076740 1 Processor -- start 2024-03-11T19:10:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.381+0000 7fa0ab076740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2eba4e0 0x5559c2ec9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.381+0000 7fa0ab076740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2ebaed0 0x5559c2eca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.381+0000 7fa0ab076740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ec0040 0x5559c2ecaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.381+0000 7fa0ab076740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559c2ecbc00 con 0x5559c2ec0040 2024-03-11T19:10:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.381+0000 7fa0ab076740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559c2f26730 con 0x5559c2eba4e0 2024-03-11T19:10:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.381+0000 7fa0ab076740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559c2f26870 con 0x5559c2ebaed0 2024-03-11T19:10:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.381+0000 7fa0ab076740 10 client.0 _ll_register_callbacks cb 0x7fa0900963e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:10:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.381+0000 7fa0a6a3a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ec0040 0x5559c2ecaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.381+0000 7fa0a6a3a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ec0040 0x5559c2ecaa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42810/0 (socket says 172.21.15.29:42810) 2024-03-11T19:10:35.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.381+0000 7fa0a6a3a640 1 -- 172.21.15.29:0/1332119583 learned_addr learned my addr 172.21.15.29:0/1332119583 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:10:35.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.382+0000 7fa0a5a38640 1 --2- 172.21.15.29:0/1332119583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2ebaed0 0x5559c2eca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.382+0000 7fa0a6239640 1 --2- 172.21.15.29:0/1332119583 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2eba4e0 0x5559c2ec9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.382+0000 7fa0a6a3a640 1 -- 172.21.15.29:0/1332119583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2ebaed0 msgr2=0x5559c2eca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:35.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.382+0000 7fa0a6a3a640 1 --2- 172.21.15.29:0/1332119583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2ebaed0 0x5559c2eca4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.382+0000 7fa0a6a3a640 1 -- 172.21.15.29:0/1332119583 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2eba4e0 msgr2=0x5559c2ec9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:10:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.382+0000 7fa0a6a3a640 1 --2- 172.21.15.29:0/1332119583 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2eba4e0 0x5559c2ec9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.382+0000 7fa0a6a3a640 1 -- 172.21.15.29:0/1332119583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5559c2e1f610 con 0x5559c2ec0040 2024-03-11T19:10:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.382+0000 7fa0a5a38640 1 --2- 172.21.15.29:0/1332119583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2ebaed0 0x5559c2eca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.382+0000 7fa0a6239640 1 --2- 172.21.15.29:0/1332119583 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2eba4e0 0x5559c2ec9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:10:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.382+0000 7fa0a6a3a640 1 --2- 172.21.15.29:0/1332119583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ec0040 0x5559c2ecaa30 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa098032500 tx=0x7fa0980347b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd7667ea5b69d8d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:10:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.382+0000 7fa089ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:10:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.382+0000 7fa089ffb640 1 -- 172.21.15.29:0/1332119583 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098045070 con 0x5559c2ec0040 2024-03-11T19:10:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.382+0000 7fa089ffb640 1 -- 172.21.15.29:0/1332119583 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fa0980328e0 con 0x5559c2ec0040 2024-03-11T19:10:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.382+0000 7fa0ab076740 1 -- 172.21.15.29:0/1332119583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa090096730 con 0x5559c2ec0040 2024-03-11T19:10:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.382+0000 7fa089ffb640 1 -- 172.21.15.29:0/1332119583 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa098034a60 con 0x5559c2ec0040 2024-03-11T19:10:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.382+0000 7fa0ab076740 10 client.15642 fetch_fsmap learned FSMap version 27 2024-03-11T19:10:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.382+0000 7fa0ab076740 1 -- 172.21.15.29:0/1332119583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5559c2de16a0 con 0x5559c2ec0040 2024-03-11T19:10:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.382+0000 7fa089ffb640 1 -- 172.21.15.29:0/1332119583 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa090096730 con 0x5559c2ec0040 2024-03-11T19:10:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.382+0000 7fa089ffb640 1 -- 172.21.15.29:0/1332119583 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa0980040f0 con 0x5559c2ec0040 2024-03-11T19:10:35.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.383+0000 7fa0ab076740 10 client.15642 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:10:35.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.383+0000 7fa0ab076740 10 client.15642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:10:35.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.383+0000 7fa0ab076740 20 client.15642 populate_metadata read hostname 'smithi029' 2024-03-11T19:10:35.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.383+0000 7fa0ab076740 1 --2- 172.21.15.29:0/1332119583 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5559c2f28350 0x5559c2f2a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:10:35.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.383+0000 7fa0ab076740 4 client.15642 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:10:35.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.383+0000 7fa0ab076740 1 -- 172.21.15.29:0/1332119583 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5559c2bac890 con 0x5559c2f28350 2024-03-11T19:10:35.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.383+0000 7fa0a6239640 1 --2- 172.21.15.29:0/1332119583 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5559c2f28350 0x5559c2f2a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:10:35.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.383+0000 7fa089ffb640 10 client.15642 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:35.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.383+0000 7fa0a6239640 1 --2- 172.21.15.29:0/1332119583 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5559c2f28350 0x5559c2f2a770 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f41ceed55ea0ced server_cookie=e959cc4c6783f51c in_seq=0 out_seq=0 2024-03-11T19:10:35.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.383+0000 7fa089ffb640 1 -- 172.21.15.29:0/1332119583 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5559c2bac890 con 0x5559c2f28350 2024-03-11T19:10:35.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.383+0000 7fa089ffb640 10 client.15642 handle_command_reply: tid=0 2024-03-11T19:10:35.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.383+0000 7fa0ab076740 1 client.15642 shutdown 2024-03-11T19:10:35.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0ab076740 1 -- 172.21.15.29:0/1332119583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ec0040 msgr2=0x5559c2ecaa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:35.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0ab076740 1 --2- 172.21.15.29:0/1332119583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ec0040 0x5559c2ecaa30 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa098032500 tx=0x7fa0980347b0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0ab076740 1 -- 172.21.15.29:0/1332119583 shutdown_connections 2024-03-11T19:10:35.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0ab076740 1 --2- 172.21.15.29:0/1332119583 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5559c2f28350 0x5559c2f2a770 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0ab076740 1 --2- 172.21.15.29:0/1332119583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559c2ec0040 0x5559c2ecaa30 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0ab076740 1 --2- 172.21.15.29:0/1332119583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559c2ebaed0 0x5559c2eca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0ab076740 1 --2- 172.21.15.29:0/1332119583 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559c2eba4e0 0x5559c2ec9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0ab076740 1 -- 172.21.15.29:0/1332119583 >> 172.21.15.29:0/1332119583 conn(0x5559c2eb37f0 msgr2=0x5559c2ec42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:35.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa089ffb640 0 client.15642 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:10:35.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0ab076740 1 -- 172.21.15.29:0/1332119583 shutdown_connections 2024-03-11T19:10:35.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0ab076740 1 -- 172.21.15.29:0/1332119583 wait complete. 2024-03-11T19:10:35.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0ab076740 20 client.15642 trim_cache size 0 max 16384 2024-03-11T19:10:35.401 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:10:35.401 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:10:35.401 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:10:35.401 INFO:teuthology.orchestra.run.smithi029.stdout: "8e89ca3b-fe57-418a-9b51-c8d3d7f188c9": { 2024-03-11T19:10:35.401 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:10:35.401 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "8e89ca3b-fe57-418a-9b51-c8d3d7f188c9", 2024-03-11T19:10:35.401 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:10:35.401 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:10:35.401 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:10:35.401 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:10:35.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0a849d640 1 -- 172.21.15.29:0/1193893157 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa08c0a1ac0 msgr2=0x7fa08c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:35.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0a849d640 1 --2- 172.21.15.29:0/1193893157 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa08c0a1ac0 0x7fa08c0a3f80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa0a01010d0 tx=0x7fa09c00b390 comp rx=0 tx=0).stop 2024-03-11T19:10:35.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0a849d640 1 -- 172.21.15.29:0/1193893157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0a0101270 msgr2=0x7fa0a0196510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:10:35.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0a849d640 1 --2- 172.21.15.29:0/1193893157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0a0101270 0x7fa0a0196510 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fa09000d950 tx=0x7fa09000de10 comp rx=0 tx=0).stop 2024-03-11T19:10:35.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0a849d640 1 -- 172.21.15.29:0/1193893157 shutdown_connections 2024-03-11T19:10:35.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0a849d640 1 --2- 172.21.15.29:0/1193893157 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0a0101c30 0x7fa0a019d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0a849d640 1 --2- 172.21.15.29:0/1193893157 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa08c0a1ac0 0x7fa08c0a3f80 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0a849d640 1 --2- 172.21.15.29:0/1193893157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0a0101270 0x7fa0a0196510 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0a849d640 1 --2- 172.21.15.29:0/1193893157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0a0100080 0x7fa0a0195fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:10:35.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0a849d640 1 -- 172.21.15.29:0/1193893157 >> 172.21.15.29:0/1193893157 conn(0x7fa0a00fb5f0 msgr2=0x7fa0a0104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:10:35.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0a849d640 1 -- 172.21.15.29:0/1193893157 shutdown_connections 2024-03-11T19:10:35.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:10:35.384+0000 7fa0a849d640 1 -- 172.21.15.29:0/1193893157 wait complete. 2024-03-11T19:10:35.403 INFO:tasks.cephfs.filesystem:scrub status for tag:8e89ca3b-fe57-418a-9b51-c8d3d7f188c9 - {'path': '/', 'tag': '8e89ca3b-fe57-418a-9b51-c8d3d7f188c9', 'options': 'recursive,force'} 2024-03-11T19:10:35.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:35 smithi029 ceph-mon[32439]: pgmap v374: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:10:35.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:35 smithi170 ceph-mon[35690]: pgmap v374: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:10:36.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:35 smithi190 ceph-mon[36292]: pgmap v374: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:10:36.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:36 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:10:36.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:36 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:10:36.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:36 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:10:36.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:36 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:10:36.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:36 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:10:36.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:36 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:10:36.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:36 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:10:36.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:36 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:10:36.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:36 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:10:36.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:36 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:10:37.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:36 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:10:37.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:36 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:10:37.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:36 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:10:37.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:36 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:10:37.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:36 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:10:37.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:37 smithi170 ceph-mon[35690]: pgmap v375: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:37.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:37 smithi029 ceph-mon[32439]: pgmap v375: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:38.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:37 smithi190 ceph-mon[36292]: pgmap v375: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:39.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:39 smithi170 ceph-mon[35690]: pgmap v376: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-03-11T19:10:39.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:39 smithi029 ceph-mon[32439]: pgmap v376: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-03-11T19:10:40.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:39 smithi190 ceph-mon[36292]: pgmap v376: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-03-11T19:10:41.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:41 smithi170 ceph-mon[35690]: pgmap v377: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-11T19:10:41.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:41 smithi029 ceph-mon[32439]: pgmap v377: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-11T19:10:42.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:41 smithi190 ceph-mon[36292]: pgmap v377: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-11T19:10:43.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:43 smithi170 ceph-mon[35690]: pgmap v378: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-11T19:10:43.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:43 smithi029 ceph-mon[32439]: pgmap v378: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-11T19:10:44.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:43 smithi190 ceph-mon[36292]: pgmap v378: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-11T19:10:45.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:45 smithi170 ceph-mon[35690]: pgmap v379: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-11T19:10:45.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:45 smithi029 ceph-mon[32439]: pgmap v379: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-11T19:10:46.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:45 smithi190 ceph-mon[36292]: pgmap v379: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-11T19:10:47.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:47 smithi170 ceph-mon[35690]: pgmap v380: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-11T19:10:47.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:47 smithi029 ceph-mon[32439]: pgmap v380: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-11T19:10:48.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:47 smithi190 ceph-mon[36292]: pgmap v380: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-11T19:10:49.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:49 smithi170 ceph-mon[35690]: pgmap v381: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-03-11T19:10:49.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:49 smithi029 ceph-mon[32439]: pgmap v381: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-03-11T19:10:50.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:49 smithi190 ceph-mon[36292]: pgmap v381: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-03-11T19:10:51.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:51 smithi170 ceph-mon[35690]: pgmap v382: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:51.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:51 smithi029 ceph-mon[32439]: pgmap v382: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:52.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:51 smithi190 ceph-mon[36292]: pgmap v382: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:53.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:53 smithi170 ceph-mon[35690]: pgmap v383: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:53.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:53 smithi029 ceph-mon[32439]: pgmap v383: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:54.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:53 smithi190 ceph-mon[36292]: pgmap v383: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:55.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:55 smithi029 ceph-mon[32439]: pgmap v384: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:10:56.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:55 smithi190 ceph-mon[36292]: pgmap v384: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:10:56.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:55 smithi170 ceph-mon[35690]: pgmap v384: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:10:57.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:57 smithi029 ceph-mon[32439]: pgmap v385: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:58.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:57 smithi190 ceph-mon[36292]: pgmap v385: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:58.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:57 smithi170 ceph-mon[35690]: pgmap v385: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:10:59.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:10:59 smithi029 ceph-mon[32439]: pgmap v386: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:11:00.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:10:59 smithi190 ceph-mon[36292]: pgmap v386: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:11:00.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:10:59 smithi170 ceph-mon[35690]: pgmap v386: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:11:01.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:01 smithi029 ceph-mon[32439]: pgmap v387: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:11:02.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:01 smithi190 ceph-mon[36292]: pgmap v387: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:11:02.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:01 smithi170 ceph-mon[35690]: pgmap v387: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:11:03.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:03 smithi029 ceph-mon[32439]: pgmap v388: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:11:04.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:03 smithi190 ceph-mon[36292]: pgmap v388: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:11:04.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:03 smithi170 ceph-mon[35690]: pgmap v388: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:11:05.405 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:11:05.405 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:11:05.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.519+0000 7f0139202640 1 -- 172.21.15.29:0/3106261880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0134106ce0 msgr2=0x7f01341090d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:05.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.519+0000 7f0139202640 1 --2- 172.21.15.29:0/3106261880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0134106ce0 0x7f01341090d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f012c009ca0 tx=0x7f012c02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.519+0000 7f0139202640 1 -- 172.21.15.29:0/3106261880 shutdown_connections 2024-03-11T19:11:05.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.519+0000 7f0139202640 1 --2- 172.21.15.29:0/3106261880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0134106ce0 0x7f01341090d0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.519+0000 7f0139202640 1 --2- 172.21.15.29:0/3106261880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01341043b0 0x7f01341067a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.519+0000 7f0139202640 1 --2- 172.21.15.29:0/3106261880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0134101a80 0x7f0134103e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.519+0000 7f0139202640 1 -- 172.21.15.29:0/3106261880 >> 172.21.15.29:0/3106261880 conn(0x7f01340fb450 msgr2=0x7f01340fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:05.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.519+0000 7f0139202640 1 -- 172.21.15.29:0/3106261880 shutdown_connections 2024-03-11T19:11:05.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.519+0000 7f0139202640 1 -- 172.21.15.29:0/3106261880 wait complete. 2024-03-11T19:11:05.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.520+0000 7f0139202640 1 Processor -- start 2024-03-11T19:11:05.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.520+0000 7f0139202640 1 -- start start 2024-03-11T19:11:05.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.521+0000 7f0139202640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0134101a80 0x7f0134196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.521+0000 7f0139202640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01341043b0 0x7f0134197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.521+0000 7f0139202640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0134106ce0 0x7f013419e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.521+0000 7f0139202640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013419e9e0 con 0x7f0134101a80 2024-03-11T19:11:05.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.521+0000 7f0139202640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013419eb50 con 0x7f0134106ce0 2024-03-11T19:11:05.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.521+0000 7f0139202640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013419ed10 con 0x7f01341043b0 2024-03-11T19:11:05.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.521+0000 7f0132d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0134101a80 0x7f0134196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.521+0000 7f013259c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01341043b0 0x7f0134197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.521+0000 7f013259c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01341043b0 0x7f0134197430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:47900/0 (socket says 172.21.15.29:47900) 2024-03-11T19:11:05.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.521+0000 7f013259c640 1 -- 172.21.15.29:0/2347466455 learned_addr learned my addr 172.21.15.29:0/2347466455 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:05.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.521+0000 7f013359e640 1 --2- 172.21.15.29:0/2347466455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0134106ce0 0x7f013419e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.521+0000 7f013259c640 1 -- 172.21.15.29:0/2347466455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0134106ce0 msgr2=0x7f013419e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:05.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.521+0000 7f013259c640 1 --2- 172.21.15.29:0/2347466455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0134106ce0 0x7f013419e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.521+0000 7f013259c640 1 -- 172.21.15.29:0/2347466455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0134101a80 msgr2=0x7f0134196ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:05.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.521+0000 7f013259c640 1 --2- 172.21.15.29:0/2347466455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0134101a80 0x7f0134196ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.521+0000 7f013259c640 1 -- 172.21.15.29:0/2347466455 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012c009900 con 0x7f01341043b0 2024-03-11T19:11:05.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.521+0000 7f0132d9d640 1 --2- 172.21.15.29:0/2347466455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0134101a80 0x7f0134196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:05.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.521+0000 7f013259c640 1 --2- 172.21.15.29:0/2347466455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01341043b0 0x7f0134197430 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f012800e9c0 tx=0x7f012800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:05.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.522+0000 7f011bfff640 1 -- 172.21.15.29:0/2347466455 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0128009700 con 0x7f01341043b0 2024-03-11T19:11:05.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.522+0000 7f0139202640 1 -- 172.21.15.29:0/2347466455 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f013419eff0 con 0x7f01341043b0 2024-03-11T19:11:05.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.522+0000 7f011bfff640 1 -- 172.21.15.29:0/2347466455 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f012800cd80 con 0x7f01341043b0 2024-03-11T19:11:05.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.522+0000 7f0139202640 1 -- 172.21.15.29:0/2347466455 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f013419f4c0 con 0x7f01341043b0 2024-03-11T19:11:05.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.522+0000 7f011bfff640 1 -- 172.21.15.29:0/2347466455 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0128010430 con 0x7f01341043b0 2024-03-11T19:11:05.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.525+0000 7f011bfff640 1 -- 172.21.15.29:0/2347466455 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f0128004f50 con 0x7f01341043b0 2024-03-11T19:11:05.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.525+0000 7f013bdd0740 1 Processor -- start 2024-03-11T19:11:05.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.525+0000 7f013bdd0740 1 -- start start 2024-03-11T19:11:05.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.525+0000 7f013bdd0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a73a90 0x5595b4a73eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.525+0000 7f013bdd0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a74480 0x5595b4a789b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.525+0000 7f013bdd0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a79020 0x5595b4a7b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.525+0000 7f013bdd0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595b49e4280 con 0x5595b4a73a90 2024-03-11T19:11:05.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.525+0000 7f013bdd0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595b49e17b0 con 0x5595b4a79020 2024-03-11T19:11:05.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.525+0000 7f013bdd0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595b49dcf90 con 0x5595b4a74480 2024-03-11T19:11:05.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.525+0000 7f0132d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a74480 0x5595b4a789b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.525+0000 7f0132d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a74480 0x5595b4a789b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:47902/0 (socket says 172.21.15.29:47902) 2024-03-11T19:11:05.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.525+0000 7f0132d9d640 1 -- 172.21.15.29:0/1740297231 learned_addr learned my addr 172.21.15.29:0/1740297231 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:05.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.526+0000 7f013259c640 1 --2- 172.21.15.29:0/1740297231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a79020 0x5595b4a7b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.526+0000 7f013359e640 1 --2- 172.21.15.29:0/1740297231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a73a90 0x5595b4a73eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.526+0000 7f0132d9d640 1 -- 172.21.15.29:0/1740297231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a79020 msgr2=0x5595b4a7b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:05.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.526+0000 7f0132d9d640 1 --2- 172.21.15.29:0/1740297231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a79020 0x5595b4a7b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.526+0000 7f0132d9d640 1 -- 172.21.15.29:0/1740297231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a73a90 msgr2=0x5595b4a73eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:05.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.526+0000 7f0132d9d640 1 --2- 172.21.15.29:0/1740297231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a73a90 0x5595b4a73eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.526+0000 7f0132d9d640 1 -- 172.21.15.29:0/1740297231 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595b499b570 con 0x5595b4a74480 2024-03-11T19:11:05.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.526+0000 7f013359e640 1 --2- 172.21.15.29:0/1740297231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a73a90 0x5595b4a73eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:05.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.526+0000 7f011bfff640 1 --2- 172.21.15.29:0/2347466455 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f011c0a1b40 0x7f011c0a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.526+0000 7f0132d9d640 1 --2- 172.21.15.29:0/1740297231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a74480 0x5595b4a789b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f01240115c0 tx=0x7f0124011a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17d9fbe4f0e95950 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:05.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.526+0000 7f011bfff640 1 -- 172.21.15.29:0/2347466455 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0128014070 con 0x7f01341043b0 2024-03-11T19:11:05.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.526+0000 7f013359e640 1 --2- 172.21.15.29:0/2347466455 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f011c0a1b40 0x7f011c0a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.526+0000 7f01197fa640 1 -- 172.21.15.29:0/1740297231 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012400f640 con 0x5595b4a74480 2024-03-11T19:11:05.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.526+0000 7f01197fa640 1 -- 172.21.15.29:0/1740297231 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f012400fc50 con 0x5595b4a74480 2024-03-11T19:11:05.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.526+0000 7f01197fa640 1 -- 172.21.15.29:0/1740297231 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0124019cb0 con 0x5595b4a74480 2024-03-11T19:11:05.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.527+0000 7f013bdd0740 1 -- 172.21.15.29:0/1740297231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a74480 msgr2=0x5595b4a789b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:05.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.527+0000 7f013bdd0740 1 --2- 172.21.15.29:0/1740297231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a74480 0x5595b4a789b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f01240115c0 tx=0x7f0124011a80 comp rx=0 tx=0).stop 2024-03-11T19:11:05.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.527+0000 7f013359e640 1 --2- 172.21.15.29:0/2347466455 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f011c0a1b40 0x7f011c0a4000 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f012c005e30 tx=0x7f012c02e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:05.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.527+0000 7f013bdd0740 1 -- 172.21.15.29:0/1740297231 shutdown_connections 2024-03-11T19:11:05.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.527+0000 7f013bdd0740 1 --2- 172.21.15.29:0/1740297231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a79020 0x5595b4a7b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.527+0000 7f013bdd0740 1 --2- 172.21.15.29:0/1740297231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a74480 0x5595b4a789b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.527+0000 7f013bdd0740 1 --2- 172.21.15.29:0/1740297231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a73a90 0x5595b4a73eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.527+0000 7f013bdd0740 1 -- 172.21.15.29:0/1740297231 >> 172.21.15.29:0/1740297231 conn(0x5595b4a6f9a0 msgr2=0x5595b4a6fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:05.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.527+0000 7f013bdd0740 1 -- 172.21.15.29:0/1740297231 shutdown_connections 2024-03-11T19:11:05.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.527+0000 7f013bdd0740 1 -- 172.21.15.29:0/1740297231 wait complete. 2024-03-11T19:11:05.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.527+0000 7f013bdd0740 1 -- start start 2024-03-11T19:11:05.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.527+0000 7f013bdd0740 1 Processor -- start 2024-03-11T19:11:05.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f013bdd0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a73a90 0x5595b4a8f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f013bdd0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a74480 0x5595b4a8f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f013bdd0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a79020 0x5595b4a969e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f013bdd0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595b46b0430 con 0x5595b4a79020 2024-03-11T19:11:05.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f013bdd0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595b466ce00 con 0x5595b4a74480 2024-03-11T19:11:05.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f013bdd0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595b4961fd0 con 0x5595b4a73a90 2024-03-11T19:11:05.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f013bdd0740 10 client.0 _ll_register_callbacks cb 0x5595b49b7fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:05.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f0132d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a73a90 0x5595b4a8f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f0132d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a73a90 0x5595b4a8f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:47908/0 (socket says 172.21.15.29:47908) 2024-03-11T19:11:05.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f0132d9d640 1 -- 172.21.15.29:0/1407980294 learned_addr learned my addr 172.21.15.29:0/1407980294 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:05.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f013359e640 1 --2- 172.21.15.29:0/1407980294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a79020 0x5595b4a969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f013259c640 1 --2- 172.21.15.29:0/1407980294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a74480 0x5595b4a8f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f0132d9d640 1 -- 172.21.15.29:0/1407980294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a74480 msgr2=0x5595b4a8f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:05.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f0132d9d640 1 --2- 172.21.15.29:0/1407980294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a74480 0x5595b4a8f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f0132d9d640 1 -- 172.21.15.29:0/1407980294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a79020 msgr2=0x5595b4a969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:05.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f0132d9d640 1 --2- 172.21.15.29:0/1407980294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a79020 0x5595b4a969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f0132d9d640 1 -- 172.21.15.29:0/1407980294 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0124009d40 con 0x5595b4a73a90 2024-03-11T19:11:05.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f013259c640 1 --2- 172.21.15.29:0/1407980294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a74480 0x5595b4a8f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:05.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f0132d9d640 1 --2- 172.21.15.29:0/1407980294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a73a90 0x5595b4a8f460 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0124004650 tx=0x7f012400fb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c968305c08e06dad server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:05.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.528+0000 7f0116ff5640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:11:05.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.529+0000 7f0116ff5640 1 -- 172.21.15.29:0/1407980294 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012400fe40 con 0x5595b4a73a90 2024-03-11T19:11:05.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.529+0000 7f0116ff5640 1 -- 172.21.15.29:0/1407980294 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0124019bc0 con 0x5595b4a73a90 2024-03-11T19:11:05.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.529+0000 7f013bdd0740 1 -- 172.21.15.29:0/1407980294 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5595b46dc7e0 con 0x5595b4a73a90 2024-03-11T19:11:05.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.529+0000 7f0116ff5640 1 -- 172.21.15.29:0/1407980294 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0124019d80 con 0x5595b4a73a90 2024-03-11T19:11:05.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.529+0000 7f013bdd0740 10 client.25267 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:05.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.529+0000 7f013bdd0740 1 -- 172.21.15.29:0/1407980294 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5595b49db610 con 0x5595b4a73a90 2024-03-11T19:11:05.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.529+0000 7f0116ff5640 1 -- 172.21.15.29:0/1407980294 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5595b46dc7e0 con 0x5595b4a73a90 2024-03-11T19:11:05.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.530+0000 7f0116ff5640 1 -- 172.21.15.29:0/1407980294 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0124004b40 con 0x5595b4a73a90 2024-03-11T19:11:05.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.530+0000 7f013bdd0740 10 client.25267 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:05.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.530+0000 7f013bdd0740 10 client.25267 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:11:05.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.530+0000 7f013bdd0740 20 client.25267 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.530+0000 7f013bdd0740 1 --2- 172.21.15.29:0/1407980294 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5595b4a9b580 0x5595b4a9d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.530+0000 7f013bdd0740 4 client.25267 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:11:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.530+0000 7f013bdd0740 1 -- 172.21.15.29:0/1407980294 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5595b4719720 con 0x5595b4a9b580 2024-03-11T19:11:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.530+0000 7f013259c640 1 --2- 172.21.15.29:0/1407980294 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5595b4a9b580 0x5595b4a9d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.530+0000 7f0116ff5640 10 client.25267 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.530+0000 7f013259c640 1 --2- 172.21.15.29:0/1407980294 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5595b4a9b580 0x5595b4a9d9a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac0ebb54c621219a server_cookie=61557c3c614a496d in_seq=0 out_seq=0 2024-03-11T19:11:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.531+0000 7f0116ff5640 1 -- 172.21.15.29:0/1407980294 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5595b4719720 con 0x5595b4a9b580 2024-03-11T19:11:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.531+0000 7f0116ff5640 10 client.25267 handle_command_reply: tid=0 2024-03-11T19:11:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.531+0000 7f013bdd0740 1 client.25267 shutdown 2024-03-11T19:11:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.531+0000 7f013bdd0740 1 -- 172.21.15.29:0/1407980294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a73a90 msgr2=0x5595b4a8f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.531+0000 7f013bdd0740 1 --2- 172.21.15.29:0/1407980294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a73a90 0x5595b4a8f460 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0124004650 tx=0x7f012400fb90 comp rx=0 tx=0).stop 2024-03-11T19:11:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.531+0000 7f013bdd0740 1 -- 172.21.15.29:0/1407980294 shutdown_connections 2024-03-11T19:11:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.531+0000 7f013bdd0740 1 --2- 172.21.15.29:0/1407980294 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5595b4a9b580 0x5595b4a9d9a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.531+0000 7f013bdd0740 1 --2- 172.21.15.29:0/1407980294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a79020 0x5595b4a969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.531+0000 7f013bdd0740 1 --2- 172.21.15.29:0/1407980294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a74480 0x5595b4a8f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.531+0000 7f0116ff5640 0 client.25267 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:05.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.531+0000 7f013bdd0740 1 --2- 172.21.15.29:0/1407980294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a73a90 0x5595b4a8f460 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.531+0000 7f013bdd0740 1 -- 172.21.15.29:0/1407980294 >> 172.21.15.29:0/1407980294 conn(0x5595b4a6f9a0 msgr2=0x5595b4a7c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:05.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.531+0000 7f013bdd0740 1 -- 172.21.15.29:0/1407980294 shutdown_connections 2024-03-11T19:11:05.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.531+0000 7f013bdd0740 1 -- 172.21.15.29:0/1407980294 wait complete. 2024-03-11T19:11:05.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.531+0000 7f013bdd0740 20 client.25267 trim_cache size 0 max 16384 2024-03-11T19:11:05.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.561+0000 7f013bdd0740 1 Processor -- start 2024-03-11T19:11:05.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.561+0000 7f013bdd0740 1 -- start start 2024-03-11T19:11:05.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.561+0000 7f013bdd0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a73a90 0x5595b4a6f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.561+0000 7f013bdd0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a74480 0x5595b4a969a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.561+0000 7f013bdd0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a79020 0x5595b4a750b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.561+0000 7f013bdd0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595b4a904b0 con 0x5595b4a79020 2024-03-11T19:11:05.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.561+0000 7f013bdd0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595b4a6a010 con 0x5595b4a73a90 2024-03-11T19:11:05.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.561+0000 7f013bdd0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595b4a6b500 con 0x5595b4a74480 2024-03-11T19:11:05.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.561+0000 7f013359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a79020 0x5595b4a750b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.561+0000 7f013359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a79020 0x5595b4a750b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41378/0 (socket says 172.21.15.29:41378) 2024-03-11T19:11:05.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.561+0000 7f013359e640 1 -- 172.21.15.29:0/292217522 learned_addr learned my addr 172.21.15.29:0/292217522 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:05.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.562+0000 7f0132d9d640 1 --2- 172.21.15.29:0/292217522 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a73a90 0x5595b4a6f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.562+0000 7f013259c640 1 --2- 172.21.15.29:0/292217522 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a74480 0x5595b4a969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.562+0000 7f013359e640 1 -- 172.21.15.29:0/292217522 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a74480 msgr2=0x5595b4a969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:05.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.562+0000 7f013359e640 1 --2- 172.21.15.29:0/292217522 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a74480 0x5595b4a969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.562+0000 7f013359e640 1 -- 172.21.15.29:0/292217522 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a73a90 msgr2=0x5595b4a6f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:05.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.562+0000 7f013359e640 1 --2- 172.21.15.29:0/292217522 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a73a90 0x5595b4a6f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.562+0000 7f013359e640 1 -- 172.21.15.29:0/292217522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595b4977ac0 con 0x5595b4a79020 2024-03-11T19:11:05.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.562+0000 7f013359e640 1 --2- 172.21.15.29:0/292217522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a79020 0x5595b4a750b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f012c005cc0 tx=0x7f012c041070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53277c37bdfbec15 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:05.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.562+0000 7f0118ff9640 1 -- 172.21.15.29:0/292217522 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c034a60 con 0x5595b4a79020 2024-03-11T19:11:05.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.562+0000 7f0118ff9640 1 -- 172.21.15.29:0/292217522 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f012c043070 con 0x5595b4a79020 2024-03-11T19:11:05.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.562+0000 7f0118ff9640 1 -- 172.21.15.29:0/292217522 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c004540 con 0x5595b4a79020 2024-03-11T19:11:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.563+0000 7f013bdd0740 1 -- 172.21.15.29:0/292217522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a79020 msgr2=0x5595b4a750b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.563+0000 7f013bdd0740 1 --2- 172.21.15.29:0/292217522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a79020 0x5595b4a750b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f012c005cc0 tx=0x7f012c041070 comp rx=0 tx=0).stop 2024-03-11T19:11:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.563+0000 7f013bdd0740 1 -- 172.21.15.29:0/292217522 shutdown_connections 2024-03-11T19:11:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.563+0000 7f013bdd0740 1 --2- 172.21.15.29:0/292217522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a79020 0x5595b4a750b0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.563+0000 7f013bdd0740 1 --2- 172.21.15.29:0/292217522 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a74480 0x5595b4a969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.563+0000 7f013bdd0740 1 --2- 172.21.15.29:0/292217522 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a73a90 0x5595b4a6f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.563+0000 7f013bdd0740 1 -- 172.21.15.29:0/292217522 >> 172.21.15.29:0/292217522 conn(0x5595b4a6f9a0 msgr2=0x5595b4aa2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.563+0000 7f013bdd0740 1 -- 172.21.15.29:0/292217522 shutdown_connections 2024-03-11T19:11:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.563+0000 7f013bdd0740 1 -- 172.21.15.29:0/292217522 wait complete. 2024-03-11T19:11:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.563+0000 7f013bdd0740 1 -- start start 2024-03-11T19:11:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.564+0000 7f013bdd0740 1 Processor -- start 2024-03-11T19:11:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.564+0000 7f013bdd0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a73a90 0x5595b4a952b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.564+0000 7f013bdd0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a74480 0x5595b4a957f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.564+0000 7f013bdd0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a79020 0x5595b4a95d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.564+0000 7f013bdd0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595b4af1a80 con 0x5595b4a73a90 2024-03-11T19:11:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.564+0000 7f013bdd0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595b4af1bc0 con 0x5595b4a74480 2024-03-11T19:11:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.564+0000 7f013bdd0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595b4af1d80 con 0x5595b4a79020 2024-03-11T19:11:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.564+0000 7f013bdd0740 10 client.0 _ll_register_callbacks cb 0x7f0124009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.564+0000 7f0132d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a73a90 0x5595b4a952b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.564+0000 7f0132d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a73a90 0x5595b4a952b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41382/0 (socket says 172.21.15.29:41382) 2024-03-11T19:11:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.564+0000 7f0132d9d640 1 -- 172.21.15.29:0/752979858 learned_addr learned my addr 172.21.15.29:0/752979858 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:05.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.564+0000 7f013259c640 1 --2- 172.21.15.29:0/752979858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a74480 0x5595b4a957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.564+0000 7f0132d9d640 1 -- 172.21.15.29:0/752979858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a79020 msgr2=0x5595b4a95d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.564+0000 7f0132d9d640 1 --2- 172.21.15.29:0/752979858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a79020 0x5595b4a95d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.564+0000 7f0132d9d640 1 -- 172.21.15.29:0/752979858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a74480 msgr2=0x5595b4a957f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.564+0000 7f0132d9d640 1 --2- 172.21.15.29:0/752979858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a74480 0x5595b4a957f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.564+0000 7f0132d9d640 1 -- 172.21.15.29:0/752979858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0134109cb0 con 0x5595b4a73a90 2024-03-11T19:11:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.565+0000 7f0132d9d640 1 --2- 172.21.15.29:0/752979858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a73a90 0x5595b4a952b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f0124001eb0 tx=0x7f01240226f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cfb01fad294701f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.565+0000 7f0116ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.565+0000 7f0116ff5640 1 -- 172.21.15.29:0/752979858 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0124016070 con 0x5595b4a73a90 2024-03-11T19:11:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.565+0000 7f0116ff5640 1 -- 172.21.15.29:0/752979858 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0124000e40 con 0x5595b4a73a90 2024-03-11T19:11:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.565+0000 7f013bdd0740 1 -- 172.21.15.29:0/752979858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f012400fc50 con 0x5595b4a73a90 2024-03-11T19:11:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.565+0000 7f0116ff5640 1 -- 172.21.15.29:0/752979858 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0124009be0 con 0x5595b4a73a90 2024-03-11T19:11:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.565+0000 7f013bdd0740 10 client.15660 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.565+0000 7f013bdd0740 1 -- 172.21.15.29:0/752979858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5595b499d6a0 con 0x5595b4a73a90 2024-03-11T19:11:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.565+0000 7f0116ff5640 1 -- 172.21.15.29:0/752979858 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f012400fc50 con 0x5595b4a73a90 2024-03-11T19:11:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.565+0000 7f0116ff5640 1 -- 172.21.15.29:0/752979858 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0124022940 con 0x5595b4a73a90 2024-03-11T19:11:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.565+0000 7f013bdd0740 10 client.15660 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.565+0000 7f013bdd0740 10 client.15660 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:11:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.565+0000 7f013bdd0740 20 client.15660 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.565+0000 7f013bdd0740 1 --2- 172.21.15.29:0/752979858 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5595b4af3510 0x5595b4af5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.565+0000 7f013bdd0740 4 client.15660 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:11:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.565+0000 7f013bdd0740 1 -- 172.21.15.29:0/752979858 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5595b4768890 con 0x5595b4af3510 2024-03-11T19:11:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.566+0000 7f013259c640 1 --2- 172.21.15.29:0/752979858 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5595b4af3510 0x5595b4af5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.566+0000 7f0116ff5640 10 client.15660 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.566+0000 7f013259c640 1 --2- 172.21.15.29:0/752979858 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5595b4af3510 0x5595b4af5930 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1229108843588c6 server_cookie=7b2997cde2b61220 in_seq=0 out_seq=0 2024-03-11T19:11:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.566+0000 7f0116ff5640 1 -- 172.21.15.29:0/752979858 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5595b4768890 con 0x5595b4af3510 2024-03-11T19:11:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.566+0000 7f0116ff5640 10 client.15660 handle_command_reply: tid=0 2024-03-11T19:11:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.566+0000 7f013bdd0740 1 client.15660 shutdown 2024-03-11T19:11:05.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.567+0000 7f013bdd0740 1 -- 172.21.15.29:0/752979858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a73a90 msgr2=0x5595b4a952b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:05.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.567+0000 7f013bdd0740 1 --2- 172.21.15.29:0/752979858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a73a90 0x5595b4a952b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f0124001eb0 tx=0x7f01240226f0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.567+0000 7f013bdd0740 1 -- 172.21.15.29:0/752979858 shutdown_connections 2024-03-11T19:11:05.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.567+0000 7f013bdd0740 1 --2- 172.21.15.29:0/752979858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5595b4a79020 0x5595b4a95d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.567+0000 7f013bdd0740 1 --2- 172.21.15.29:0/752979858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5595b4a74480 0x5595b4a957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.567+0000 7f013bdd0740 1 --2- 172.21.15.29:0/752979858 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5595b4af3510 0x5595b4af5930 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.567+0000 7f0116ff5640 0 client.15660 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:05.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.567+0000 7f013bdd0740 1 --2- 172.21.15.29:0/752979858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5595b4a73a90 0x5595b4a952b0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.567+0000 7f013bdd0740 1 -- 172.21.15.29:0/752979858 >> 172.21.15.29:0/752979858 conn(0x5595b4a6f9a0 msgr2=0x5595b4a750b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:05.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.567+0000 7f013bdd0740 1 -- 172.21.15.29:0/752979858 shutdown_connections 2024-03-11T19:11:05.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.567+0000 7f013bdd0740 1 -- 172.21.15.29:0/752979858 wait complete. 2024-03-11T19:11:05.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.567+0000 7f013bdd0740 20 client.15660 trim_cache size 0 max 16384 2024-03-11T19:11:05.575 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:11:05.575 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:11:05.575 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:11:05.575 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:11:05.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.568+0000 7f0139202640 1 -- 172.21.15.29:0/2347466455 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f011c0a1b40 msgr2=0x7f011c0a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:05.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.568+0000 7f0139202640 1 --2- 172.21.15.29:0/2347466455 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f011c0a1b40 0x7f011c0a4000 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f012c005e30 tx=0x7f012c02e040 comp rx=0 tx=0).stop 2024-03-11T19:11:05.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.568+0000 7f0139202640 1 -- 172.21.15.29:0/2347466455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01341043b0 msgr2=0x7f0134197430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:05.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.568+0000 7f0139202640 1 --2- 172.21.15.29:0/2347466455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01341043b0 0x7f0134197430 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f012800e9c0 tx=0x7f012800ee80 comp rx=0 tx=0).stop 2024-03-11T19:11:05.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.568+0000 7f0139202640 1 -- 172.21.15.29:0/2347466455 shutdown_connections 2024-03-11T19:11:05.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.568+0000 7f0139202640 1 --2- 172.21.15.29:0/2347466455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0134106ce0 0x7f013419e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.568+0000 7f0139202640 1 --2- 172.21.15.29:0/2347466455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01341043b0 0x7f0134197430 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.568+0000 7f0139202640 1 --2- 172.21.15.29:0/2347466455 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f011c0a1b40 0x7f011c0a4000 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.568+0000 7f0139202640 1 --2- 172.21.15.29:0/2347466455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0134101a80 0x7f0134196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.568+0000 7f0139202640 1 -- 172.21.15.29:0/2347466455 >> 172.21.15.29:0/2347466455 conn(0x7f01340fb450 msgr2=0x7f0134102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:05.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.568+0000 7f0139202640 1 -- 172.21.15.29:0/2347466455 shutdown_connections 2024-03-11T19:11:05.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.568+0000 7f0139202640 1 -- 172.21.15.29:0/2347466455 wait complete. 2024-03-11T19:11:05.582 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:11:05.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:11:05.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.727+0000 7f15ce0c9640 1 -- 172.21.15.29:0/2359741649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15c81011c0 msgr2=0x7f15c81015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:05.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.727+0000 7f15ce0c9640 1 --2- 172.21.15.29:0/2359741649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15c81011c0 0x7f15c81015e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f15b8009900 tx=0x7f15b802bc30 comp rx=0 tx=0).stop 2024-03-11T19:11:05.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.727+0000 7f15ce0c9640 1 -- 172.21.15.29:0/2359741649 shutdown_connections 2024-03-11T19:11:05.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.727+0000 7f15ce0c9640 1 --2- 172.21.15.29:0/2359741649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15c8102d70 0x7f15c8109600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.727+0000 7f15ce0c9640 1 --2- 172.21.15.29:0/2359741649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c81023b0 0x7f15c8102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.727+0000 7f15ce0c9640 1 --2- 172.21.15.29:0/2359741649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15c81011c0 0x7f15c81015e0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.727+0000 7f15ce0c9640 1 -- 172.21.15.29:0/2359741649 >> 172.21.15.29:0/2359741649 conn(0x7f15c80fc730 msgr2=0x7f15c80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:05.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.727+0000 7f15ce0c9640 1 -- 172.21.15.29:0/2359741649 shutdown_connections 2024-03-11T19:11:05.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.727+0000 7f15ce0c9640 1 -- 172.21.15.29:0/2359741649 wait complete. 2024-03-11T19:11:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.728+0000 7f15ce0c9640 1 Processor -- start 2024-03-11T19:11:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.728+0000 7f15ce0c9640 1 -- start start 2024-03-11T19:11:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.728+0000 7f15ce0c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c81011c0 0x7f15c8074ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.728+0000 7f15ce0c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15c81023b0 0x7f15c8073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.728+0000 7f15ce0c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15c8102d70 0x7f15c8073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.728+0000 7f15ce0c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c8074240 con 0x7f15c81023b0 2024-03-11T19:11:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.728+0000 7f15ce0c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c80743b0 con 0x7f15c8102d70 2024-03-11T19:11:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.728+0000 7f15ce0c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c81a1230 con 0x7f15c81011c0 2024-03-11T19:11:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.728+0000 7f15c6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15c81023b0 0x7f15c8073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.728+0000 7f15c6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15c81023b0 0x7f15c8073570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41394/0 (socket says 172.21.15.29:41394) 2024-03-11T19:11:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.728+0000 7f15c6ffd640 1 -- 172.21.15.29:0/1504197947 learned_addr learned my addr 172.21.15.29:0/1504197947 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.728+0000 7f15c77fe640 1 --2- 172.21.15.29:0/1504197947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c81011c0 0x7f15c8074ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.728+0000 7f15c7fff640 1 --2- 172.21.15.29:0/1504197947 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15c8102d70 0x7f15c8073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.729+0000 7f15c6ffd640 1 -- 172.21.15.29:0/1504197947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c81011c0 msgr2=0x7f15c8074ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:05.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.729+0000 7f15c6ffd640 1 --2- 172.21.15.29:0/1504197947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c81011c0 0x7f15c8074ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.729+0000 7f15c6ffd640 1 -- 172.21.15.29:0/1504197947 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15c8102d70 msgr2=0x7f15c8073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:05.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.729+0000 7f15c6ffd640 1 --2- 172.21.15.29:0/1504197947 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15c8102d70 0x7f15c8073ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:05.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.729+0000 7f15c6ffd640 1 -- 172.21.15.29:0/1504197947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b8009560 con 0x7f15c81023b0 2024-03-11T19:11:05.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.729+0000 7f15c7fff640 1 --2- 172.21.15.29:0/1504197947 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15c8102d70 0x7f15c8073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:05.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.729+0000 7f15c6ffd640 1 --2- 172.21.15.29:0/1504197947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15c81023b0 0x7f15c8073570 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f15bc00ea20 tx=0x7f15bc00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:05.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.729+0000 7f15c4ff9640 1 -- 172.21.15.29:0/1504197947 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc009700 con 0x7f15c81023b0 2024-03-11T19:11:05.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.729+0000 7f15c4ff9640 1 -- 172.21.15.29:0/1504197947 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f15bc00cd80 con 0x7f15c81023b0 2024-03-11T19:11:05.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.730+0000 7f15c4ff9640 1 -- 172.21.15.29:0/1504197947 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc018710 con 0x7f15c81023b0 2024-03-11T19:11:05.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.730+0000 7f15ce0c9640 1 -- 172.21.15.29:0/1504197947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c81a1490 con 0x7f15c81023b0 2024-03-11T19:11:05.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.730+0000 7f15ce0c9640 1 -- 172.21.15.29:0/1504197947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c81a18b0 con 0x7f15c81023b0 2024-03-11T19:11:05.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.732+0000 7f15ce0c9640 1 -- 172.21.15.29:0/1504197947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c80676e0 con 0x7f15c81023b0 2024-03-11T19:11:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.733+0000 7f15c4ff9640 1 -- 172.21.15.29:0/1504197947 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f15bc018870 con 0x7f15c81023b0 2024-03-11T19:11:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.734+0000 7f15c4ff9640 1 --2- 172.21.15.29:0/1504197947 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f15b00a1a20 0x7f15b00a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.734+0000 7f15c4ff9640 1 -- 172.21.15.29:0/1504197947 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f15bc014070 con 0x7f15c81023b0 2024-03-11T19:11:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.734+0000 7f15c77fe640 1 --2- 172.21.15.29:0/1504197947 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f15b00a1a20 0x7f15b00a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.735+0000 7f15c77fe640 1 --2- 172.21.15.29:0/1504197947 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f15b00a1a20 0x7f15b00a3ee0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f15b802c7d0 tx=0x7f15b8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:05.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:05.743+0000 7f15c4ff9640 1 -- 172.21.15.29:0/1504197947 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f15bc096900 con 0x7f15c81023b0 2024-03-11T19:11:05.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:05 smithi029 ceph-mon[32439]: pgmap v389: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:11:06.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:05 smithi190 ceph-mon[36292]: pgmap v389: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:11:06.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:05 smithi170 ceph-mon[35690]: pgmap v389: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:11:06.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.191+0000 7f15ce0c9640 1 -- 172.21.15.29:0/1504197947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15c810ca40 con 0x7f15c81023b0 2024-03-11T19:11:06.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.192+0000 7f15c4ff9640 1 -- 172.21.15.29:0/1504197947 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f15bc099fc0 con 0x7f15c81023b0 2024-03-11T19:11:06.195 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:11:06.196 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:11:06.196 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:11:06.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.195+0000 7f15ce0c9640 1 -- 172.21.15.29:0/1504197947 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f15b00a1a20 msgr2=0x7f15b00a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.195+0000 7f15ce0c9640 1 --2- 172.21.15.29:0/1504197947 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f15b00a1a20 0x7f15b00a3ee0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f15b802c7d0 tx=0x7f15b8005fd0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.195+0000 7f15ce0c9640 1 -- 172.21.15.29:0/1504197947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15c81023b0 msgr2=0x7f15c8073570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.195+0000 7f15ce0c9640 1 --2- 172.21.15.29:0/1504197947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15c81023b0 0x7f15c8073570 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f15bc00ea20 tx=0x7f15bc00eee0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.195+0000 7f15ce0c9640 1 -- 172.21.15.29:0/1504197947 shutdown_connections 2024-03-11T19:11:06.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.195+0000 7f15ce0c9640 1 --2- 172.21.15.29:0/1504197947 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15c8102d70 0x7f15c8073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.195+0000 7f15ce0c9640 1 --2- 172.21.15.29:0/1504197947 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f15b00a1a20 0x7f15b00a3ee0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.195+0000 7f15ce0c9640 1 --2- 172.21.15.29:0/1504197947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15c81023b0 0x7f15c8073570 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.195+0000 7f15ce0c9640 1 --2- 172.21.15.29:0/1504197947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15c81011c0 0x7f15c8074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.195+0000 7f15ce0c9640 1 -- 172.21.15.29:0/1504197947 >> 172.21.15.29:0/1504197947 conn(0x7f15c80fc730 msgr2=0x7f15c8105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.195+0000 7f15ce0c9640 1 -- 172.21.15.29:0/1504197947 shutdown_connections 2024-03-11T19:11:06.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.195+0000 7f15ce0c9640 1 -- 172.21.15.29:0/1504197947 wait complete. 2024-03-11T19:11:06.213 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:11:06.213 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:11:06.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.321+0000 7f1eaf96c640 1 -- 172.21.15.29:0/1024415205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ea8101c30 msgr2=0x7f1ea81084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.321+0000 7f1eaf96c640 1 --2- 172.21.15.29:0/1024415205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ea8101c30 0x7f1ea81084c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0009ca0 tx=0x7f1ea002c0f0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.322+0000 7f1eaf96c640 1 -- 172.21.15.29:0/1024415205 shutdown_connections 2024-03-11T19:11:06.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.322+0000 7f1eaf96c640 1 --2- 172.21.15.29:0/1024415205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ea8101c30 0x7f1ea81084c0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.322+0000 7f1eaf96c640 1 --2- 172.21.15.29:0/1024415205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ea8101270 0x7f1ea81016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.322+0000 7f1eaf96c640 1 --2- 172.21.15.29:0/1024415205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ea8100080 0x7f1ea81004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.322+0000 7f1eaf96c640 1 -- 172.21.15.29:0/1024415205 >> 172.21.15.29:0/1024415205 conn(0x7f1ea80fb610 msgr2=0x7f1ea80fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.322+0000 7f1eaf96c640 1 -- 172.21.15.29:0/1024415205 shutdown_connections 2024-03-11T19:11:06.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.322+0000 7f1eaf96c640 1 -- 172.21.15.29:0/1024415205 wait complete. 2024-03-11T19:11:06.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.322+0000 7f1eaf96c640 1 Processor -- start 2024-03-11T19:11:06.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.322+0000 7f1eaf96c640 1 -- start start 2024-03-11T19:11:06.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.323+0000 7f1eaf96c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ea8100080 0x7f1ea8068e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.323+0000 7f1eaf96c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ea8101270 0x7f1ea819e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.323+0000 7f1eaf96c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ea8101c30 0x7f1ea81a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.323+0000 7f1eaf96c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea81a0e90 con 0x7f1ea8101c30 2024-03-11T19:11:06.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.323+0000 7f1eaf96c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea81a1000 con 0x7f1ea8101270 2024-03-11T19:11:06.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.323+0000 7f1eaf96c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea81a11c0 con 0x7f1ea8100080 2024-03-11T19:11:06.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.323+0000 7f1eadf09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ea8101c30 0x7f1ea81a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.323+0000 7f1eadf09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ea8101c30 0x7f1ea81a07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41408/0 (socket says 172.21.15.29:41408) 2024-03-11T19:11:06.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.323+0000 7f1eadf09640 1 -- 172.21.15.29:0/552954291 learned_addr learned my addr 172.21.15.29:0/552954291 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.323+0000 7f1ead708640 1 --2- 172.21.15.29:0/552954291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ea8100080 0x7f1ea8068e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.323+0000 7f1eacf07640 1 --2- 172.21.15.29:0/552954291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ea8101270 0x7f1ea819e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.323+0000 7f1eadf09640 1 -- 172.21.15.29:0/552954291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ea8100080 msgr2=0x7f1ea8068e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.323+0000 7f1eadf09640 1 --2- 172.21.15.29:0/552954291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ea8100080 0x7f1ea8068e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.323+0000 7f1eadf09640 1 -- 172.21.15.29:0/552954291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ea8101270 msgr2=0x7f1ea819e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.323+0000 7f1eadf09640 1 --2- 172.21.15.29:0/552954291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ea8101270 0x7f1ea819e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.323+0000 7f1eadf09640 1 -- 172.21.15.29:0/552954291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea0009900 con 0x7f1ea8101c30 2024-03-11T19:11:06.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.323+0000 7f1ead708640 1 --2- 172.21.15.29:0/552954291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ea8100080 0x7f1ea8068e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:06.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.323+0000 7f1eadf09640 1 --2- 172.21.15.29:0/552954291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ea8101c30 0x7f1ea81a07d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0037040 tx=0x7f1ea0032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.324+0000 7f1e9e7fc640 1 -- 172.21.15.29:0/552954291 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0034e70 con 0x7f1ea8101c30 2024-03-11T19:11:06.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.324+0000 7f1e9e7fc640 1 -- 172.21.15.29:0/552954291 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1ea0003e20 con 0x7f1ea8101c30 2024-03-11T19:11:06.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.324+0000 7f1eaf96c640 1 -- 172.21.15.29:0/552954291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea81a13e0 con 0x7f1ea8101c30 2024-03-11T19:11:06.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.324+0000 7f1e9e7fc640 1 -- 172.21.15.29:0/552954291 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea003d3f0 con 0x7f1ea8101c30 2024-03-11T19:11:06.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.324+0000 7f1eaf96c640 1 -- 172.21.15.29:0/552954291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea81a17d0 con 0x7f1ea8101c30 2024-03-11T19:11:06.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.327+0000 7f1e9e7fc640 1 -- 172.21.15.29:0/552954291 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f1ea003b040 con 0x7f1ea8101c30 2024-03-11T19:11:06.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.328+0000 7f1e9e7fc640 1 --2- 172.21.15.29:0/552954291 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1e940a1af0 0x7f1e940a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.328+0000 7f1e9e7fc640 1 -- 172.21.15.29:0/552954291 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f1ea00efae0 con 0x7f1ea8101c30 2024-03-11T19:11:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.328+0000 7f1ead708640 1 --2- 172.21.15.29:0/552954291 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1e940a1af0 0x7f1e940a3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.328+0000 7f1ead708640 1 --2- 172.21.15.29:0/552954291 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1e940a1af0 0x7f1e940a3fb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1ea40058c0 tx=0x7f1ea4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.329+0000 7f1eb261b740 1 Processor -- start 2024-03-11T19:11:06.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.329+0000 7f1eb261b740 1 -- start start 2024-03-11T19:11:06.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.329+0000 7f1eb261b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e41767270 0x563e41767690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.329+0000 7f1eb261b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e41767c60 0x563e4176c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.329+0000 7f1eb261b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e4176cdd0 0x563e4176f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.329+0000 7f1eb261b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e41723960 con 0x563e4176cdd0 2024-03-11T19:11:06.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.329+0000 7f1eb261b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e416d7310 con 0x563e41767270 2024-03-11T19:11:06.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.329+0000 7f1eb261b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e416d4840 con 0x563e41767c60 2024-03-11T19:11:06.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.329+0000 7f1ead708640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e41767270 0x563e41767690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.329+0000 7f1ead708640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e41767270 0x563e41767690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:52426/0 (socket says 172.21.15.29:52426) 2024-03-11T19:11:06.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.329+0000 7f1ead708640 1 -- 172.21.15.29:0/3686265100 learned_addr learned my addr 172.21.15.29:0/3686265100 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.330+0000 7f1eacf07640 1 --2- 172.21.15.29:0/3686265100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e41767c60 0x563e4176c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.330+0000 7f1eadf09640 1 --2- 172.21.15.29:0/3686265100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e4176cdd0 0x563e4176f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.330+0000 7f1ead708640 1 -- 172.21.15.29:0/3686265100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e41767c60 msgr2=0x563e4176c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.330+0000 7f1ead708640 1 --2- 172.21.15.29:0/3686265100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e41767c60 0x563e4176c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.330+0000 7f1ead708640 1 -- 172.21.15.29:0/3686265100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e4176cdd0 msgr2=0x563e4176f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.330+0000 7f1ead708640 1 --2- 172.21.15.29:0/3686265100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e4176cdd0 0x563e4176f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.330+0000 7f1ead708640 1 -- 172.21.15.29:0/3686265100 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e41692060 con 0x563e41767270 2024-03-11T19:11:06.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.330+0000 7f1eadf09640 1 --2- 172.21.15.29:0/3686265100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e4176cdd0 0x563e4176f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:06.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.330+0000 7f1ead708640 1 --2- 172.21.15.29:0/3686265100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e41767270 0x563e41767690 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ea400b450 tx=0x7f1ea4012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dc14734014e9134 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.330+0000 7f1e937fe640 1 -- 172.21.15.29:0/3686265100 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea401e650 con 0x563e41767270 2024-03-11T19:11:06.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.331+0000 7f1e937fe640 1 -- 172.21.15.29:0/3686265100 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1ea400b930 con 0x563e41767270 2024-03-11T19:11:06.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.331+0000 7f1e937fe640 1 -- 172.21.15.29:0/3686265100 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4010bf0 con 0x563e41767270 2024-03-11T19:11:06.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.332+0000 7f1eb261b740 1 -- 172.21.15.29:0/3686265100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e41767270 msgr2=0x563e41767690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.332+0000 7f1eb261b740 1 --2- 172.21.15.29:0/3686265100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e41767270 0x563e41767690 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ea400b450 tx=0x7f1ea4012000 comp rx=0 tx=0).stop 2024-03-11T19:11:06.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.332+0000 7f1eb261b740 1 -- 172.21.15.29:0/3686265100 shutdown_connections 2024-03-11T19:11:06.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.332+0000 7f1eb261b740 1 --2- 172.21.15.29:0/3686265100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e4176cdd0 0x563e4176f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.332+0000 7f1eb261b740 1 --2- 172.21.15.29:0/3686265100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e41767c60 0x563e4176c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.332+0000 7f1eb261b740 1 --2- 172.21.15.29:0/3686265100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e41767270 0x563e41767690 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.332+0000 7f1eb261b740 1 -- 172.21.15.29:0/3686265100 >> 172.21.15.29:0/3686265100 conn(0x563e417627e0 msgr2=0x563e41762be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.332+0000 7f1eb261b740 1 -- 172.21.15.29:0/3686265100 shutdown_connections 2024-03-11T19:11:06.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.332+0000 7f1eb261b740 1 -- 172.21.15.29:0/3686265100 wait complete. 2024-03-11T19:11:06.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.332+0000 7f1eb261b740 1 -- start start 2024-03-11T19:11:06.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.332+0000 7f1eb261b740 1 Processor -- start 2024-03-11T19:11:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.333+0000 7f1eb261b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e41767270 0x563e417714b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.333+0000 7f1eb261b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e41767c60 0x563e417719f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.333+0000 7f1eb261b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e4176cdd0 0x563e41778a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.333+0000 7f1ead708640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e41767270 0x563e417714b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.333+0000 7f1eb261b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e416d0020 con 0x563e41767270 2024-03-11T19:11:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.333+0000 7f1ead708640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e41767270 0x563e417714b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41428/0 (socket says 172.21.15.29:41428) 2024-03-11T19:11:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.333+0000 7f1eb261b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e413a3430 con 0x563e41767c60 2024-03-11T19:11:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.333+0000 7f1ead708640 1 -- 172.21.15.29:0/151603199 learned_addr learned my addr 172.21.15.29:0/151603199 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.333+0000 7f1eb261b740 1 -- 172.21.15.29:0/151603199 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e4135fe00 con 0x563e4176cdd0 2024-03-11T19:11:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.333+0000 7f1eacf07640 1 --2- 172.21.15.29:0/151603199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e41767c60 0x563e417719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.334+0000 7f1eadf09640 1 --2- 172.21.15.29:0/151603199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e4176cdd0 0x563e41778a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.334+0000 7f1eb261b740 10 client.0 _ll_register_callbacks cb 0x563e41677cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.334+0000 7f1ead708640 1 -- 172.21.15.29:0/151603199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e4176cdd0 msgr2=0x563e41778a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.334+0000 7f1ead708640 1 --2- 172.21.15.29:0/151603199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e4176cdd0 0x563e41778a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.334+0000 7f1ead708640 1 -- 172.21.15.29:0/151603199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e41767c60 msgr2=0x563e417719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.334+0000 7f1ead708640 1 --2- 172.21.15.29:0/151603199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e41767c60 0x563e417719f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.334+0000 7f1ead708640 1 -- 172.21.15.29:0/151603199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea400bca0 con 0x563e41767270 2024-03-11T19:11:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.334+0000 7f1eadf09640 1 --2- 172.21.15.29:0/151603199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e4176cdd0 0x563e41778a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.334+0000 7f1eacf07640 1 --2- 172.21.15.29:0/151603199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e41767c60 0x563e417719f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.334+0000 7f1ead708640 1 --2- 172.21.15.29:0/151603199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e41767270 0x563e417714b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f1ea400b5b0 tx=0x7f1ea4010970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c561357c4c7b44cb server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.334+0000 7f1e90ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.334+0000 7f1e90ff9640 1 -- 172.21.15.29:0/151603199 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea402d070 con 0x563e41767270 2024-03-11T19:11:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.334+0000 7f1e90ff9640 1 -- 172.21.15.29:0/151603199 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f1ea4004020 con 0x563e41767270 2024-03-11T19:11:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.334+0000 7f1eb261b740 1 -- 172.21.15.29:0/151603199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563e4140c720 con 0x563e41767270 2024-03-11T19:11:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.335+0000 7f1e90ff9640 1 -- 172.21.15.29:0/151603199 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ea4010e60 con 0x563e41767270 2024-03-11T19:11:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.335+0000 7f1eb261b740 10 client.15678 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.335+0000 7f1eb261b740 1 -- 172.21.15.29:0/151603199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563e416ce6a0 con 0x563e41767270 2024-03-11T19:11:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.335+0000 7f1e90ff9640 1 -- 172.21.15.29:0/151603199 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x563e4140c720 con 0x563e41767270 2024-03-11T19:11:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.335+0000 7f1e90ff9640 1 -- 172.21.15.29:0/151603199 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1ea4032470 con 0x563e41767270 2024-03-11T19:11:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.335+0000 7f1eb261b740 10 client.15678 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.335+0000 7f1eb261b740 10 client.15678 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:11:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.335+0000 7f1eb261b740 20 client.15678 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.335+0000 7f1eb261b740 1 --2- 172.21.15.29:0/151603199 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x563e4177d580 0x563e4177f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.335+0000 7f1eb261b740 4 client.15678 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:11:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.335+0000 7f1eb261b740 1 -- 172.21.15.29:0/151603199 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563e4145b890 con 0x563e4177d580 2024-03-11T19:11:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.335+0000 7f1eacf07640 1 --2- 172.21.15.29:0/151603199 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x563e4177d580 0x563e4177f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.336+0000 7f1eacf07640 1 --2- 172.21.15.29:0/151603199 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x563e4177d580 0x563e4177f9c0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ac277162f26f2d65 server_cookie=d097a62ce5e27676 in_seq=0 out_seq=0 2024-03-11T19:11:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.336+0000 7f1e90ff9640 10 client.15678 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:11:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.337+0000 7f1e90ff9640 1 -- 172.21.15.29:0/151603199 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563e4145b890 con 0x563e4177d580 2024-03-11T19:11:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.337+0000 7f1e90ff9640 10 client.15678 handle_command_reply: tid=0 2024-03-11T19:11:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.337+0000 7f1eb261b740 1 client.15678 shutdown 2024-03-11T19:11:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.337+0000 7f1eb261b740 1 -- 172.21.15.29:0/151603199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e41767270 msgr2=0x563e417714b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.337+0000 7f1eb261b740 1 --2- 172.21.15.29:0/151603199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e41767270 0x563e417714b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f1ea400b5b0 tx=0x7f1ea4010970 comp rx=0 tx=0).stop 2024-03-11T19:11:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.337+0000 7f1eb261b740 1 -- 172.21.15.29:0/151603199 shutdown_connections 2024-03-11T19:11:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.337+0000 7f1eb261b740 1 --2- 172.21.15.29:0/151603199 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x563e4177d580 0x563e4177f9c0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.337+0000 7f1eb261b740 1 --2- 172.21.15.29:0/151603199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e4176cdd0 0x563e41778a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.337+0000 7f1eb261b740 1 --2- 172.21.15.29:0/151603199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e41767c60 0x563e417719f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.337+0000 7f1e90ff9640 0 client.15678 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:11:06.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.337+0000 7f1eb261b740 1 --2- 172.21.15.29:0/151603199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e41767270 0x563e417714b0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.337+0000 7f1eb261b740 1 -- 172.21.15.29:0/151603199 >> 172.21.15.29:0/151603199 conn(0x563e417627e0 msgr2=0x563e41761b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.337+0000 7f1eb261b740 1 -- 172.21.15.29:0/151603199 shutdown_connections 2024-03-11T19:11:06.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.337+0000 7f1eb261b740 1 -- 172.21.15.29:0/151603199 wait complete. 2024-03-11T19:11:06.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.337+0000 7f1eb261b740 20 client.15678 trim_cache size 0 max 16384 2024-03-11T19:11:06.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.361+0000 7f1eb261b740 1 Processor -- start 2024-03-11T19:11:06.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.361+0000 7f1eb261b740 1 -- start start 2024-03-11T19:11:06.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.361+0000 7f1eb261b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e41767c60 0x563e417691b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.361+0000 7f1eb261b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e4176cdd0 0x563e417696f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.361+0000 7f1eb261b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e4177d580 0x563e417789c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.361+0000 7f1eb261b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e41655130 con 0x563e41767c60 2024-03-11T19:11:06.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.361+0000 7f1eb261b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e4140c720 con 0x563e4176cdd0 2024-03-11T19:11:06.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.361+0000 7f1eb261b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e41754a40 con 0x563e4177d580 2024-03-11T19:11:06.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.361+0000 7f1ead708640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e41767c60 0x563e417691b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.361+0000 7f1ead708640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e41767c60 0x563e417691b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41432/0 (socket says 172.21.15.29:41432) 2024-03-11T19:11:06.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.361+0000 7f1ead708640 1 -- 172.21.15.29:0/961467579 learned_addr learned my addr 172.21.15.29:0/961467579 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.361+0000 7f1eadf09640 1 --2- 172.21.15.29:0/961467579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e4177d580 0x563e417789c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.361+0000 7f1eacf07640 1 --2- 172.21.15.29:0/961467579 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e4176cdd0 0x563e417696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.362+0000 7f1ead708640 1 -- 172.21.15.29:0/961467579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e4177d580 msgr2=0x563e417789c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.362+0000 7f1ead708640 1 --2- 172.21.15.29:0/961467579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e4177d580 0x563e417789c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.362+0000 7f1ead708640 1 -- 172.21.15.29:0/961467579 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e4176cdd0 msgr2=0x563e417696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.362+0000 7f1ead708640 1 --2- 172.21.15.29:0/961467579 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e4176cdd0 0x563e417696f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.362+0000 7f1ead708640 1 -- 172.21.15.29:0/961467579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e4166ac20 con 0x563e41767c60 2024-03-11T19:11:06.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.362+0000 7f1ead708640 1 --2- 172.21.15.29:0/961467579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e41767c60 0x563e417691b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f1ea40328a0 tx=0x7f1ea400b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b81e4066b5853e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.362+0000 7f1e92ffd640 1 -- 172.21.15.29:0/961467579 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea402d040 con 0x563e41767c60 2024-03-11T19:11:06.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.362+0000 7f1e92ffd640 1 -- 172.21.15.29:0/961467579 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1ea40044d0 con 0x563e41767c60 2024-03-11T19:11:06.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.362+0000 7f1e92ffd640 1 -- 172.21.15.29:0/961467579 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea40052c0 con 0x563e41767c60 2024-03-11T19:11:06.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.363+0000 7f1eb261b740 1 -- 172.21.15.29:0/961467579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e41767c60 msgr2=0x563e417691b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.363+0000 7f1eb261b740 1 --2- 172.21.15.29:0/961467579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e41767c60 0x563e417691b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f1ea40328a0 tx=0x7f1ea400b4d0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.363+0000 7f1eb261b740 1 -- 172.21.15.29:0/961467579 shutdown_connections 2024-03-11T19:11:06.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.363+0000 7f1eb261b740 1 --2- 172.21.15.29:0/961467579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e4177d580 0x563e417789c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.363+0000 7f1eb261b740 1 --2- 172.21.15.29:0/961467579 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e4176cdd0 0x563e417696f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.363+0000 7f1eb261b740 1 --2- 172.21.15.29:0/961467579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e41767c60 0x563e417691b0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.363+0000 7f1eb261b740 1 -- 172.21.15.29:0/961467579 >> 172.21.15.29:0/961467579 conn(0x563e417627e0 msgr2=0x563e4175da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.363+0000 7f1eb261b740 1 -- 172.21.15.29:0/961467579 shutdown_connections 2024-03-11T19:11:06.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.363+0000 7f1eb261b740 1 -- 172.21.15.29:0/961467579 wait complete. 2024-03-11T19:11:06.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.363+0000 7f1eb261b740 1 -- start start 2024-03-11T19:11:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.363+0000 7f1eb261b740 1 Processor -- start 2024-03-11T19:11:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.364+0000 7f1eb261b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e41767c60 0x563e41776bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.364+0000 7f1eb261b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e4176cdd0 0x563e417770f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.364+0000 7f1eb261b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e4177d580 0x563e41777f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.364+0000 7f1eacf07640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e4176cdd0 0x563e417770f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.364+0000 7f1eadf09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e4177d580 0x563e41777f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.364+0000 7f1ead708640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e41767c60 0x563e41776bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.364+0000 7f1eadf09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e4177d580 0x563e41777f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41436/0 (socket says 172.21.15.29:41436) 2024-03-11T19:11:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.364+0000 7f1eacf07640 1 --2- 172.21.15.29:0/561267149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e4176cdd0 0x563e417770f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:47978/0 (socket says 172.21.15.29:47978) 2024-03-11T19:11:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.364+0000 7f1eadf09640 1 -- 172.21.15.29:0/561267149 learned_addr learned my addr 172.21.15.29:0/561267149 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.364+0000 7f1eb261b740 1 -- 172.21.15.29:0/561267149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e417784d0 con 0x563e4177d580 2024-03-11T19:11:06.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.364+0000 7f1eb261b740 1 -- 172.21.15.29:0/561267149 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e41778640 con 0x563e41767c60 2024-03-11T19:11:06.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.364+0000 7f1eb261b740 1 -- 172.21.15.29:0/561267149 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e41778800 con 0x563e4176cdd0 2024-03-11T19:11:06.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.364+0000 7f1eb261b740 10 client.0 _ll_register_callbacks cb 0x7f1ea4015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:06.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.364+0000 7f1eadf09640 1 -- 172.21.15.29:0/561267149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e4176cdd0 msgr2=0x563e417770f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.364+0000 7f1eadf09640 1 --2- 172.21.15.29:0/561267149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e4176cdd0 0x563e417770f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.364+0000 7f1eadf09640 1 -- 172.21.15.29:0/561267149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e41767c60 msgr2=0x563e41776bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.364+0000 7f1eadf09640 1 --2- 172.21.15.29:0/561267149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e41767c60 0x563e41776bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.364+0000 7f1eadf09640 1 -- 172.21.15.29:0/561267149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563e416ce6a0 con 0x563e4177d580 2024-03-11T19:11:06.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.364+0000 7f1ead708640 1 --2- 172.21.15.29:0/561267149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e41767c60 0x563e41776bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:06.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.365+0000 7f1eadf09640 1 --2- 172.21.15.29:0/561267149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e4177d580 0x563e41777f40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1ea00bca90 tx=0x7f1ea002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e4c422867e68bff server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.365+0000 7f1e90ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:06.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.365+0000 7f1e90ff9640 1 -- 172.21.15.29:0/561267149 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea00c5010 con 0x563e4177d580 2024-03-11T19:11:06.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.365+0000 7f1e90ff9640 1 -- 172.21.15.29:0/561267149 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f1ea00c0c50 con 0x563e4177d580 2024-03-11T19:11:06.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.365+0000 7f1eb261b740 1 -- 172.21.15.29:0/561267149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563e413cf7e0 con 0x563e4177d580 2024-03-11T19:11:06.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.365+0000 7f1e90ff9640 1 -- 172.21.15.29:0/561267149 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ea0004250 con 0x563e4177d580 2024-03-11T19:11:06.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.365+0000 7f1eb261b740 10 client.15684 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:06.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.365+0000 7f1eb261b740 1 -- 172.21.15.29:0/561267149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563e416908c0 con 0x563e4177d580 2024-03-11T19:11:06.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.365+0000 7f1e90ff9640 1 -- 172.21.15.29:0/561267149 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1ea00db030 con 0x563e4177d580 2024-03-11T19:11:06.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.365+0000 7f1e90ff9640 1 -- 172.21.15.29:0/561267149 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1ea0004b40 con 0x563e4177d580 2024-03-11T19:11:06.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.366+0000 7f1eb261b740 10 client.15684 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:06.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.366+0000 7f1eb261b740 10 client.15684 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:11:06.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.366+0000 7f1eb261b740 20 client.15684 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:06.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.366+0000 7f1eb261b740 1 --2- 172.21.15.29:0/561267149 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x563e417d4a50 0x563e417d6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.366+0000 7f1eb261b740 4 client.15684 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:11:06.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.366+0000 7f1eb261b740 1 -- 172.21.15.29:0/561267149 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563e413cf7e0 con 0x563e417d4a50 2024-03-11T19:11:06.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.366+0000 7f1ead708640 1 --2- 172.21.15.29:0/561267149 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x563e417d4a50 0x563e417d6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.366+0000 7f1e90ff9640 10 client.15684 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:11:06.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.366+0000 7f1ead708640 1 --2- 172.21.15.29:0/561267149 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x563e417d4a50 0x563e417d6e70 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5d57da36f014366 server_cookie=9f90878095209d82 in_seq=0 out_seq=0 2024-03-11T19:11:06.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.366+0000 7f1e90ff9640 1 -- 172.21.15.29:0/561267149 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563e413cf7e0 con 0x563e417d4a50 2024-03-11T19:11:06.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.366+0000 7f1e90ff9640 10 client.15684 handle_command_reply: tid=0 2024-03-11T19:11:06.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.366+0000 7f1eb261b740 1 client.15684 shutdown 2024-03-11T19:11:06.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.367+0000 7f1eb261b740 1 -- 172.21.15.29:0/561267149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e4177d580 msgr2=0x563e41777f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.367+0000 7f1eb261b740 1 --2- 172.21.15.29:0/561267149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e4177d580 0x563e41777f40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1ea00bca90 tx=0x7f1ea002c0f0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.367+0000 7f1eb261b740 1 -- 172.21.15.29:0/561267149 shutdown_connections 2024-03-11T19:11:06.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.367+0000 7f1eb261b740 1 --2- 172.21.15.29:0/561267149 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x563e417d4a50 0x563e417d6e70 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.367+0000 7f1eb261b740 1 --2- 172.21.15.29:0/561267149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e4177d580 0x563e41777f40 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.367+0000 7f1eb261b740 1 --2- 172.21.15.29:0/561267149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563e4176cdd0 0x563e417770f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.367+0000 7f1eb261b740 1 --2- 172.21.15.29:0/561267149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563e41767c60 0x563e41776bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.367+0000 7f1eb261b740 1 -- 172.21.15.29:0/561267149 >> 172.21.15.29:0/561267149 conn(0x563e417627e0 msgr2=0x563e417691b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.367+0000 7f1e90ff9640 0 client.15684 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:11:06.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.367+0000 7f1eb261b740 1 -- 172.21.15.29:0/561267149 shutdown_connections 2024-03-11T19:11:06.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.367+0000 7f1eb261b740 1 -- 172.21.15.29:0/561267149 wait complete. 2024-03-11T19:11:06.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.367+0000 7f1eb261b740 20 client.15684 trim_cache size 0 max 16384 2024-03-11T19:11:06.377 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:11:06.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.367+0000 7f1eaf96c640 1 -- 172.21.15.29:0/552954291 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1e940a1af0 msgr2=0x7f1e940a3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.367+0000 7f1eaf96c640 1 --2- 172.21.15.29:0/552954291 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1e940a1af0 0x7f1e940a3fb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1ea40058c0 tx=0x7f1ea4005850 comp rx=0 tx=0).stop 2024-03-11T19:11:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.367+0000 7f1eaf96c640 1 -- 172.21.15.29:0/552954291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ea8101c30 msgr2=0x7f1ea81a07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.367+0000 7f1eaf96c640 1 --2- 172.21.15.29:0/552954291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ea8101c30 0x7f1ea81a07d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0037040 tx=0x7f1ea0032980 comp rx=0 tx=0).stop 2024-03-11T19:11:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.368+0000 7f1eaf96c640 1 -- 172.21.15.29:0/552954291 shutdown_connections 2024-03-11T19:11:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.368+0000 7f1eaf96c640 1 --2- 172.21.15.29:0/552954291 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1e940a1af0 0x7f1e940a3fb0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.368+0000 7f1eaf96c640 1 --2- 172.21.15.29:0/552954291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ea8101c30 0x7f1ea81a07d0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.368+0000 7f1eaf96c640 1 --2- 172.21.15.29:0/552954291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ea8101270 0x7f1ea819e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.368+0000 7f1eaf96c640 1 --2- 172.21.15.29:0/552954291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ea8100080 0x7f1ea8068e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.368+0000 7f1eaf96c640 1 -- 172.21.15.29:0/552954291 >> 172.21.15.29:0/552954291 conn(0x7f1ea80fb610 msgr2=0x7f1ea81045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.368+0000 7f1eaf96c640 1 -- 172.21.15.29:0/552954291 shutdown_connections 2024-03-11T19:11:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.368+0000 7f1eaf96c640 1 -- 172.21.15.29:0/552954291 wait complete. 2024-03-11T19:11:06.382 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:11:06.382 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:11:06.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.492+0000 7ff2cf15c640 1 -- 172.21.15.29:0/1026783069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c8101c80 msgr2=0x7ff2c8104070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.492+0000 7ff2cf15c640 1 --2- 172.21.15.29:0/1026783069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c8101c80 0x7ff2c8104070 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc009900 tx=0x7ff2bc030040 comp rx=0 tx=0).stop 2024-03-11T19:11:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.493+0000 7ff2cf15c640 1 -- 172.21.15.29:0/1026783069 shutdown_connections 2024-03-11T19:11:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.493+0000 7ff2cf15c640 1 --2- 172.21.15.29:0/1026783069 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2c8106ee0 0x7ff2c81092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.493+0000 7ff2cf15c640 1 --2- 172.21.15.29:0/1026783069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c81045b0 0x7ff2c81069a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.493+0000 7ff2cf15c640 1 --2- 172.21.15.29:0/1026783069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c8101c80 0x7ff2c8104070 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.493+0000 7ff2cf15c640 1 -- 172.21.15.29:0/1026783069 >> 172.21.15.29:0/1026783069 conn(0x7ff2c80fb610 msgr2=0x7ff2c80fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.493+0000 7ff2cf15c640 1 -- 172.21.15.29:0/1026783069 shutdown_connections 2024-03-11T19:11:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.493+0000 7ff2cf15c640 1 -- 172.21.15.29:0/1026783069 wait complete. 2024-03-11T19:11:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.493+0000 7ff2cf15c640 1 Processor -- start 2024-03-11T19:11:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.493+0000 7ff2cf15c640 1 -- start start 2024-03-11T19:11:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.493+0000 7ff2cf15c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c8101c80 0x7ff2c8195020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.493+0000 7ff2cf15c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c81045b0 0x7ff2c8195560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.494+0000 7ff2cf15c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2c8106ee0 0x7ff2c819c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.494+0000 7ff2cf15c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c819cac0 con 0x7ff2c81045b0 2024-03-11T19:11:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.494+0000 7ff2cf15c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c819cc30 con 0x7ff2c8106ee0 2024-03-11T19:11:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.494+0000 7ff2cf15c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c819cdf0 con 0x7ff2c8101c80 2024-03-11T19:11:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.494+0000 7ff2c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c81045b0 0x7ff2c8195560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.494+0000 7ff2c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c81045b0 0x7ff2c8195560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41450/0 (socket says 172.21.15.29:41450) 2024-03-11T19:11:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.494+0000 7ff2c7fff640 1 -- 172.21.15.29:0/257905953 learned_addr learned my addr 172.21.15.29:0/257905953 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.494+0000 7ff2ccef8640 1 --2- 172.21.15.29:0/257905953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c8101c80 0x7ff2c8195020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.494+0000 7ff2cd6f9640 1 --2- 172.21.15.29:0/257905953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2c8106ee0 0x7ff2c819c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.494+0000 7ff2c7fff640 1 -- 172.21.15.29:0/257905953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c8101c80 msgr2=0x7ff2c8195020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.494+0000 7ff2c7fff640 1 --2- 172.21.15.29:0/257905953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c8101c80 0x7ff2c8195020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.494+0000 7ff2c7fff640 1 -- 172.21.15.29:0/257905953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2c8106ee0 msgr2=0x7ff2c819c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.494+0000 7ff2c7fff640 1 --2- 172.21.15.29:0/257905953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2c8106ee0 0x7ff2c819c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.494+0000 7ff2c7fff640 1 -- 172.21.15.29:0/257905953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2bc009560 con 0x7ff2c81045b0 2024-03-11T19:11:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.494+0000 7ff2cd6f9640 1 --2- 172.21.15.29:0/257905953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2c8106ee0 0x7ff2c819c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:11:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.494+0000 7ff2ccef8640 1 --2- 172.21.15.29:0/257905953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c8101c80 0x7ff2c8195020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.494+0000 7ff2c7fff640 1 --2- 172.21.15.29:0/257905953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c81045b0 0x7ff2c8195560 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff2b400e9f0 tx=0x7ff2b400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.494+0000 7ff2c5ffb640 1 -- 172.21.15.29:0/257905953 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b40096c0 con 0x7ff2c81045b0 2024-03-11T19:11:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.494+0000 7ff2cf15c640 1 -- 172.21.15.29:0/257905953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c819d0d0 con 0x7ff2c81045b0 2024-03-11T19:11:06.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.495+0000 7ff2c5ffb640 1 -- 172.21.15.29:0/257905953 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff2b40054d0 con 0x7ff2c81045b0 2024-03-11T19:11:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.495+0000 7ff2c5ffb640 1 -- 172.21.15.29:0/257905953 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4010430 con 0x7ff2c81045b0 2024-03-11T19:11:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.495+0000 7ff2cf15c640 1 -- 172.21.15.29:0/257905953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c819d5e0 con 0x7ff2c81045b0 2024-03-11T19:11:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.498+0000 7ff2c5ffb640 1 -- 172.21.15.29:0/257905953 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7ff2b4004fe0 con 0x7ff2c81045b0 2024-03-11T19:11:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.498+0000 7ff2d1e05740 1 Processor -- start 2024-03-11T19:11:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.498+0000 7ff2d1e05740 1 -- start start 2024-03-11T19:11:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.498+0000 7ff2d1e05740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85613270 0x560f85613690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.498+0000 7ff2d1e05740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613c60 0x560f85618760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.498+0000 7ff2d1e05740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85618dd0 0x560f8561b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.498+0000 7ff2d1e05740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f855cf960 con 0x560f85613270 2024-03-11T19:11:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.498+0000 7ff2d1e05740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f85583310 con 0x560f85613c60 2024-03-11T19:11:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.498+0000 7ff2d1e05740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f85580840 con 0x560f85618dd0 2024-03-11T19:11:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.498+0000 7ff2cd6f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85613270 0x560f85613690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.498+0000 7ff2cd6f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85613270 0x560f85613690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41456/0 (socket says 172.21.15.29:41456) 2024-03-11T19:11:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.498+0000 7ff2cd6f9640 1 -- 172.21.15.29:0/3031680100 learned_addr learned my addr 172.21.15.29:0/3031680100 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.498+0000 7ff2c7fff640 1 --2- 172.21.15.29:0/3031680100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85618dd0 0x560f8561b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.499+0000 7ff2cd6f9640 1 -- 172.21.15.29:0/3031680100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85618dd0 msgr2=0x560f8561b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.499+0000 7ff2cd6f9640 1 --2- 172.21.15.29:0/3031680100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85618dd0 0x560f8561b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.499+0000 7ff2cd6f9640 1 -- 172.21.15.29:0/3031680100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613c60 msgr2=0x560f85618760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:11:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.499+0000 7ff2c5ffb640 1 --2- 172.21.15.29:0/257905953 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff2b80a1b40 0x7ff2b80a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.499+0000 7ff2cd6f9640 1 --2- 172.21.15.29:0/3031680100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613c60 0x560f85618760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.499+0000 7ff2cd6f9640 1 -- 172.21.15.29:0/3031680100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f8553e060 con 0x560f85613270 2024-03-11T19:11:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.499+0000 7ff2c5ffb640 1 -- 172.21.15.29:0/257905953 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff2b4014070 con 0x7ff2c81045b0 2024-03-11T19:11:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.499+0000 7ff2cd6f9640 1 --2- 172.21.15.29:0/3031680100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85613270 0x560f85613690 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0009ea0 tx=0x7ff2c0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b62298a25a23f47 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.499+0000 7ff2cd6f9640 1 --2- 172.21.15.29:0/257905953 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff2b80a1b40 0x7ff2b80a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.499+0000 7ff2b2ffd640 1 -- 172.21.15.29:0/3031680100 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c001e040 con 0x560f85613270 2024-03-11T19:11:06.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.499+0000 7ff2b2ffd640 1 -- 172.21.15.29:0/3031680100 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff2c001ab70 con 0x560f85613270 2024-03-11T19:11:06.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.499+0000 7ff2b2ffd640 1 -- 172.21.15.29:0/3031680100 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0011ea0 con 0x560f85613270 2024-03-11T19:11:06.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.500+0000 7ff2d1e05740 1 -- 172.21.15.29:0/3031680100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85613270 msgr2=0x560f85613690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.500+0000 7ff2cd6f9640 1 --2- 172.21.15.29:0/257905953 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff2b80a1b40 0x7ff2b80a4000 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0019000 tx=0x7ff2c0019460 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.500+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/3031680100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85613270 0x560f85613690 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0009ea0 tx=0x7ff2c0009f90 comp rx=0 tx=0).stop 2024-03-11T19:11:06.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.500+0000 7ff2d1e05740 1 -- 172.21.15.29:0/3031680100 shutdown_connections 2024-03-11T19:11:06.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.500+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/3031680100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85618dd0 0x560f8561b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.500+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/3031680100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613c60 0x560f85618760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.500+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/3031680100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85613270 0x560f85613690 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.500+0000 7ff2d1e05740 1 -- 172.21.15.29:0/3031680100 >> 172.21.15.29:0/3031680100 conn(0x560f8560e7e0 msgr2=0x560f8560ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.500+0000 7ff2d1e05740 1 -- 172.21.15.29:0/3031680100 shutdown_connections 2024-03-11T19:11:06.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.500+0000 7ff2d1e05740 1 -- 172.21.15.29:0/3031680100 wait complete. 2024-03-11T19:11:06.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.501+0000 7ff2d1e05740 1 -- start start 2024-03-11T19:11:06.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.501+0000 7ff2d1e05740 1 Processor -- start 2024-03-11T19:11:06.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.501+0000 7ff2d1e05740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613270 0x560f8561d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.501+0000 7ff2d1e05740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85613c60 0x560f8561d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.501+0000 7ff2d1e05740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 0x560f856249a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.501+0000 7ff2d1e05740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f8557c020 con 0x560f85618dd0 2024-03-11T19:11:06.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.501+0000 7ff2d1e05740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f8524f430 con 0x560f85613270 2024-03-11T19:11:06.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.501+0000 7ff2d1e05740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f8520be00 con 0x560f85613c60 2024-03-11T19:11:06.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.501+0000 7ff2d1e05740 10 client.0 _ll_register_callbacks cb 0x560f85523cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:06.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.502+0000 7ff2c7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85613c60 0x560f8561d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.502+0000 7ff2c7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85613c60 0x560f8561d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48012/0 (socket says 172.21.15.29:48012) 2024-03-11T19:11:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.502+0000 7ff2cd6f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 0x560f856249a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.502+0000 7ff2c7fff640 1 -- 172.21.15.29:0/2521781560 learned_addr learned my addr 172.21.15.29:0/2521781560 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.502+0000 7ff2ccef8640 1 --2- 172.21.15.29:0/2521781560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613270 0x560f8561d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.502+0000 7ff2cd6f9640 1 -- 172.21.15.29:0/2521781560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85613c60 msgr2=0x560f8561d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.502+0000 7ff2cd6f9640 1 --2- 172.21.15.29:0/2521781560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85613c60 0x560f8561d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.502+0000 7ff2cd6f9640 1 -- 172.21.15.29:0/2521781560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613270 msgr2=0x560f8561d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.502+0000 7ff2cd6f9640 1 --2- 172.21.15.29:0/2521781560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613270 0x560f8561d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.502+0000 7ff2cd6f9640 1 -- 172.21.15.29:0/2521781560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c0009ad0 con 0x560f85618dd0 2024-03-11T19:11:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.502+0000 7ff2cd6f9640 1 --2- 172.21.15.29:0/2521781560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 0x560f856249a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ff2c00093c0 tx=0x7ff2c001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f51c3150476d6e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.502+0000 7ff2a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.502+0000 7ff2a7fff640 1 -- 172.21.15.29:0/2521781560 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c001e040 con 0x560f85618dd0 2024-03-11T19:11:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.502+0000 7ff2a7fff640 1 -- 172.21.15.29:0/2521781560 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff2c0015070 con 0x560f85618dd0 2024-03-11T19:11:06.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.502+0000 7ff2d1e05740 1 -- 172.21.15.29:0/2521781560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560f8527b7e0 con 0x560f85618dd0 2024-03-11T19:11:06.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.503+0000 7ff2a7fff640 1 -- 172.21.15.29:0/2521781560 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2c0019e80 con 0x560f85618dd0 2024-03-11T19:11:06.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.503+0000 7ff2d1e05740 10 client.15696 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:06.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.503+0000 7ff2d1e05740 1 -- 172.21.15.29:0/2521781560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x560f8557a6a0 con 0x560f85618dd0 2024-03-11T19:11:06.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.503+0000 7ff2a7fff640 1 -- 172.21.15.29:0/2521781560 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff2c0004e70 con 0x560f85618dd0 2024-03-11T19:11:06.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.503+0000 7ff2a7fff640 1 -- 172.21.15.29:0/2521781560 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff2c00109a0 con 0x560f85618dd0 2024-03-11T19:11:06.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.503+0000 7ff2d1e05740 10 client.15696 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:06.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.503+0000 7ff2d1e05740 10 client.15696 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:11:06.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.503+0000 7ff2d1e05740 20 client.15696 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:06.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.503+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/2521781560 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x560f856295b0 0x560f8562ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.503+0000 7ff2d1e05740 4 client.15696 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:11:06.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.503+0000 7ff2d1e05740 1 -- 172.21.15.29:0/2521781560 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560f8527b7e0 con 0x560f856295b0 2024-03-11T19:11:06.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.503+0000 7ff2ccef8640 1 --2- 172.21.15.29:0/2521781560 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x560f856295b0 0x560f8562ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.504+0000 7ff2a7fff640 10 client.15696 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:11:06.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.504+0000 7ff2ccef8640 1 --2- 172.21.15.29:0/2521781560 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x560f856295b0 0x560f8562ba00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e493f02fc6af78d4 server_cookie=25ea52625de4a084 in_seq=0 out_seq=0 2024-03-11T19:11:06.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.504+0000 7ff2a7fff640 1 -- 172.21.15.29:0/2521781560 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560f8527b7e0 con 0x560f856295b0 2024-03-11T19:11:06.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.504+0000 7ff2a7fff640 10 client.15696 handle_command_reply: tid=0 2024-03-11T19:11:06.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.504+0000 7ff2d1e05740 1 client.15696 shutdown 2024-03-11T19:11:06.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.505+0000 7ff2d1e05740 1 -- 172.21.15.29:0/2521781560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 msgr2=0x560f856249a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.505+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/2521781560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 0x560f856249a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ff2c00093c0 tx=0x7ff2c001a000 comp rx=0 tx=0).stop 2024-03-11T19:11:06.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.505+0000 7ff2d1e05740 1 -- 172.21.15.29:0/2521781560 shutdown_connections 2024-03-11T19:11:06.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.505+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/2521781560 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x560f856295b0 0x560f8562ba00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.505+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/2521781560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 0x560f856249a0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.505+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/2521781560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85613c60 0x560f8561d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.505+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/2521781560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613270 0x560f8561d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.505+0000 7ff2d1e05740 1 -- 172.21.15.29:0/2521781560 >> 172.21.15.29:0/2521781560 conn(0x560f8560e7e0 msgr2=0x560f8560db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.505+0000 7ff2a7fff640 0 client.15696 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:11:06.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.505+0000 7ff2d1e05740 1 -- 172.21.15.29:0/2521781560 shutdown_connections 2024-03-11T19:11:06.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.505+0000 7ff2d1e05740 1 -- 172.21.15.29:0/2521781560 wait complete. 2024-03-11T19:11:06.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.505+0000 7ff2d1e05740 20 client.15696 trim_cache size 0 max 16384 2024-03-11T19:11:06.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.517+0000 7ff2d1e05740 1 Processor -- start 2024-03-11T19:11:06.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.517+0000 7ff2d1e05740 1 -- start start 2024-03-11T19:11:06.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.517+0000 7ff2d1e05740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613270 0x560f8561ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.517+0000 7ff2d1e05740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85613c60 0x560f85624960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.517+0000 7ff2d1e05740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 0x560f8561d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.517+0000 7ff2d1e05740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f85501130 con 0x560f85618dd0 2024-03-11T19:11:06.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.517+0000 7ff2d1e05740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f85600a40 con 0x560f85613270 2024-03-11T19:11:06.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.517+0000 7ff2d1e05740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f8560e6a0 con 0x560f85613c60 2024-03-11T19:11:06.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.517+0000 7ff2cd6f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 0x560f8561d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.518+0000 7ff2cd6f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 0x560f8561d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41474/0 (socket says 172.21.15.29:41474) 2024-03-11T19:11:06.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.518+0000 7ff2cd6f9640 1 -- 172.21.15.29:0/1798807406 learned_addr learned my addr 172.21.15.29:0/1798807406 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.518+0000 7ff2ccef8640 1 --2- 172.21.15.29:0/1798807406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613270 0x560f8561ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.518+0000 7ff2c7fff640 1 --2- 172.21.15.29:0/1798807406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85613c60 0x560f85624960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.518+0000 7ff2cd6f9640 1 -- 172.21.15.29:0/1798807406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85613c60 msgr2=0x560f85624960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.518+0000 7ff2cd6f9640 1 --2- 172.21.15.29:0/1798807406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85613c60 0x560f85624960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.518+0000 7ff2cd6f9640 1 -- 172.21.15.29:0/1798807406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613270 msgr2=0x560f8561ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.518+0000 7ff2cd6f9640 1 --2- 172.21.15.29:0/1798807406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613270 0x560f8561ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.518+0000 7ff2cd6f9640 1 -- 172.21.15.29:0/1798807406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f85516c20 con 0x560f85618dd0 2024-03-11T19:11:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.518+0000 7ff2c7fff640 1 --2- 172.21.15.29:0/1798807406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85613c60 0x560f85624960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.518+0000 7ff2ccef8640 1 --2- 172.21.15.29:0/1798807406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613270 0x560f8561ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.518+0000 7ff2cd6f9640 1 --2- 172.21.15.29:0/1798807406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 0x560f8561d420 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0002230 tx=0x7ff2c000fc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ce19092462f2120 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.518+0000 7ff2b27fc640 1 -- 172.21.15.29:0/1798807406 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c001a480 con 0x560f85618dd0 2024-03-11T19:11:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.518+0000 7ff2b27fc640 1 -- 172.21.15.29:0/1798807406 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff2c0019e80 con 0x560f85618dd0 2024-03-11T19:11:06.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.518+0000 7ff2b27fc640 1 -- 172.21.15.29:0/1798807406 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0004c00 con 0x560f85618dd0 2024-03-11T19:11:06.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.519+0000 7ff2d1e05740 1 -- 172.21.15.29:0/1798807406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 msgr2=0x560f8561d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.519+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/1798807406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 0x560f8561d420 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0002230 tx=0x7ff2c000fc80 comp rx=0 tx=0).stop 2024-03-11T19:11:06.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.519+0000 7ff2d1e05740 1 -- 172.21.15.29:0/1798807406 shutdown_connections 2024-03-11T19:11:06.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.519+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/1798807406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 0x560f8561d420 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.519+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/1798807406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85613c60 0x560f85624960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.519+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/1798807406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613270 0x560f8561ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.519+0000 7ff2d1e05740 1 -- 172.21.15.29:0/1798807406 >> 172.21.15.29:0/1798807406 conn(0x560f8560e7e0 msgr2=0x560f8560ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.519+0000 7ff2d1e05740 1 -- 172.21.15.29:0/1798807406 shutdown_connections 2024-03-11T19:11:06.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.519+0000 7ff2d1e05740 1 -- 172.21.15.29:0/1798807406 wait complete. 2024-03-11T19:11:06.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.519+0000 7ff2d1e05740 1 -- start start 2024-03-11T19:11:06.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.519+0000 7ff2d1e05740 1 Processor -- start 2024-03-11T19:11:06.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.520+0000 7ff2d1e05740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85613270 0x560f856245a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.520+0000 7ff2d1e05740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613c60 0x560f85622bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.520+0000 7ff2d1e05740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 0x560f856230f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.520+0000 7ff2d1e05740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f85624d90 con 0x560f85618dd0 2024-03-11T19:11:06.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.520+0000 7ff2d1e05740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f85623660 con 0x560f85613c60 2024-03-11T19:11:06.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.520+0000 7ff2d1e05740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f85623820 con 0x560f85613270 2024-03-11T19:11:06.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.520+0000 7ff2d1e05740 10 client.0 _ll_register_callbacks cb 0x7ff2c0009630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:06.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.520+0000 7ff2cd6f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 0x560f856230f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.520+0000 7ff2cd6f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 0x560f856230f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41476/0 (socket says 172.21.15.29:41476) 2024-03-11T19:11:06.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.520+0000 7ff2cd6f9640 1 -- 172.21.15.29:0/2289714293 learned_addr learned my addr 172.21.15.29:0/2289714293 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.520+0000 7ff2ccef8640 1 --2- 172.21.15.29:0/2289714293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85613270 0x560f856245a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.520+0000 7ff2c7fff640 1 --2- 172.21.15.29:0/2289714293 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613c60 0x560f85622bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.520+0000 7ff2cd6f9640 1 -- 172.21.15.29:0/2289714293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85613270 msgr2=0x560f856245a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.520+0000 7ff2cd6f9640 1 --2- 172.21.15.29:0/2289714293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85613270 0x560f856245a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.520+0000 7ff2cd6f9640 1 -- 172.21.15.29:0/2289714293 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613c60 msgr2=0x560f85622bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.520+0000 7ff2cd6f9640 1 --2- 172.21.15.29:0/2289714293 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613c60 0x560f85622bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.520+0000 7ff2cd6f9640 1 -- 172.21.15.29:0/2289714293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f8557a6a0 con 0x560f85618dd0 2024-03-11T19:11:06.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.520+0000 7ff2ccef8640 1 --2- 172.21.15.29:0/2289714293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85613270 0x560f856245a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:06.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.521+0000 7ff2cd6f9640 1 --2- 172.21.15.29:0/2289714293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 0x560f856230f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0024040 tx=0x7ff2c0010f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31b0460f910a221 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.521+0000 7ff2a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:06.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.521+0000 7ff2a7fff640 1 -- 172.21.15.29:0/2289714293 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c001e020 con 0x560f85618dd0 2024-03-11T19:11:06.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.521+0000 7ff2d1e05740 1 -- 172.21.15.29:0/2289714293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560f852b8720 con 0x560f85618dd0 2024-03-11T19:11:06.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.521+0000 7ff2a7fff640 1 -- 172.21.15.29:0/2289714293 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff2c0015070 con 0x560f85618dd0 2024-03-11T19:11:06.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.521+0000 7ff2a7fff640 1 -- 172.21.15.29:0/2289714293 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2c0019e80 con 0x560f85618dd0 2024-03-11T19:11:06.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.521+0000 7ff2d1e05740 10 client.15702 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:06.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.521+0000 7ff2d1e05740 1 -- 172.21.15.29:0/2289714293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x560f8553c8c0 con 0x560f85618dd0 2024-03-11T19:11:06.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.521+0000 7ff2a7fff640 1 -- 172.21.15.29:0/2289714293 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x560f852b8720 con 0x560f85618dd0 2024-03-11T19:11:06.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.521+0000 7ff2a7fff640 1 -- 172.21.15.29:0/2289714293 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff2c0041070 con 0x560f85618dd0 2024-03-11T19:11:06.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.521+0000 7ff2d1e05740 10 client.15702 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:06.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.521+0000 7ff2d1e05740 10 client.15702 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:11:06.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.522+0000 7ff2d1e05740 20 client.15702 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:06.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.522+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/2289714293 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x560f85680ea0 0x560f856832c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.522+0000 7ff2d1e05740 4 client.15702 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:11:06.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.522+0000 7ff2d1e05740 1 -- 172.21.15.29:0/2289714293 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560f85307890 con 0x560f85680ea0 2024-03-11T19:11:06.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.522+0000 7ff2ccef8640 1 --2- 172.21.15.29:0/2289714293 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x560f85680ea0 0x560f856832c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.522+0000 7ff2a7fff640 10 client.15702 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:11:06.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.522+0000 7ff2ccef8640 1 --2- 172.21.15.29:0/2289714293 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x560f85680ea0 0x560f856832c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e9642d3aba1dc3f2 server_cookie=b4657fe515cbed6d in_seq=0 out_seq=0 2024-03-11T19:11:06.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.523+0000 7ff2a7fff640 1 -- 172.21.15.29:0/2289714293 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560f85307890 con 0x560f85680ea0 2024-03-11T19:11:06.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.523+0000 7ff2a7fff640 10 client.15702 handle_command_reply: tid=0 2024-03-11T19:11:06.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.523+0000 7ff2d1e05740 1 client.15702 shutdown 2024-03-11T19:11:06.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.523+0000 7ff2d1e05740 1 -- 172.21.15.29:0/2289714293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 msgr2=0x560f856230f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.523+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/2289714293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 0x560f856230f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0024040 tx=0x7ff2c0010f70 comp rx=0 tx=0).stop 2024-03-11T19:11:06.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.523+0000 7ff2d1e05740 1 -- 172.21.15.29:0/2289714293 shutdown_connections 2024-03-11T19:11:06.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.523+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/2289714293 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x560f85680ea0 0x560f856832c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.523+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/2289714293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f85618dd0 0x560f856230f0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.523+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/2289714293 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560f85613c60 0x560f85622bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.523+0000 7ff2d1e05740 1 --2- 172.21.15.29:0/2289714293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f85613270 0x560f856245a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.523+0000 7ff2a7fff640 0 client.15702 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:11:06.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.523+0000 7ff2d1e05740 1 -- 172.21.15.29:0/2289714293 >> 172.21.15.29:0/2289714293 conn(0x560f8560e7e0 msgr2=0x560f856150e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.523+0000 7ff2d1e05740 1 -- 172.21.15.29:0/2289714293 shutdown_connections 2024-03-11T19:11:06.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.523+0000 7ff2d1e05740 1 -- 172.21.15.29:0/2289714293 wait complete. 2024-03-11T19:11:06.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.523+0000 7ff2d1e05740 20 client.15702 trim_cache size 0 max 16384 2024-03-11T19:11:06.532 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:11:06.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.524+0000 7ff2cf15c640 1 -- 172.21.15.29:0/257905953 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff2b80a1b40 msgr2=0x7ff2b80a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.524+0000 7ff2cf15c640 1 --2- 172.21.15.29:0/257905953 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff2b80a1b40 0x7ff2b80a4000 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0019000 tx=0x7ff2c0019460 comp rx=0 tx=0).stop 2024-03-11T19:11:06.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.524+0000 7ff2cf15c640 1 -- 172.21.15.29:0/257905953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c81045b0 msgr2=0x7ff2c8195560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.524+0000 7ff2cf15c640 1 --2- 172.21.15.29:0/257905953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c81045b0 0x7ff2c8195560 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff2b400e9f0 tx=0x7ff2b400eeb0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.524+0000 7ff2cf15c640 1 -- 172.21.15.29:0/257905953 shutdown_connections 2024-03-11T19:11:06.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.524+0000 7ff2cf15c640 1 --2- 172.21.15.29:0/257905953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2c8106ee0 0x7ff2c819c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.524+0000 7ff2cf15c640 1 --2- 172.21.15.29:0/257905953 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff2b80a1b40 0x7ff2b80a4000 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.524+0000 7ff2cf15c640 1 --2- 172.21.15.29:0/257905953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c81045b0 0x7ff2c8195560 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.524+0000 7ff2cf15c640 1 --2- 172.21.15.29:0/257905953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2c8101c80 0x7ff2c8195020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.524+0000 7ff2cf15c640 1 -- 172.21.15.29:0/257905953 >> 172.21.15.29:0/257905953 conn(0x7ff2c80fb610 msgr2=0x7ff2c8102720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.524+0000 7ff2cf15c640 1 -- 172.21.15.29:0/257905953 shutdown_connections 2024-03-11T19:11:06.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.524+0000 7ff2cf15c640 1 -- 172.21.15.29:0/257905953 wait complete. 2024-03-11T19:11:06.538 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:11:06.538 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:11:06.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.645+0000 7fdbc5b8c640 1 -- 172.21.15.29:0/3896982872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc0101270 msgr2=0x7fdbc01016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.645+0000 7fdbc5b8c640 1 --2- 172.21.15.29:0/3896982872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc0101270 0x7fdbc01016f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4009900 tx=0x7fdbb402bc30 comp rx=0 tx=0).stop 2024-03-11T19:11:06.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.645+0000 7fdbc5b8c640 1 -- 172.21.15.29:0/3896982872 shutdown_connections 2024-03-11T19:11:06.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.645+0000 7fdbc5b8c640 1 --2- 172.21.15.29:0/3896982872 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdbc0101c30 0x7fdbc01084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.645+0000 7fdbc5b8c640 1 --2- 172.21.15.29:0/3896982872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc0101270 0x7fdbc01016f0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.645+0000 7fdbc5b8c640 1 --2- 172.21.15.29:0/3896982872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdbc0100080 0x7fdbc01004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.645+0000 7fdbc5b8c640 1 -- 172.21.15.29:0/3896982872 >> 172.21.15.29:0/3896982872 conn(0x7fdbc00fb610 msgr2=0x7fdbc00fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.645+0000 7fdbc5b8c640 1 -- 172.21.15.29:0/3896982872 shutdown_connections 2024-03-11T19:11:06.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.645+0000 7fdbc5b8c640 1 -- 172.21.15.29:0/3896982872 wait complete. 2024-03-11T19:11:06.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.645+0000 7fdbc5b8c640 1 Processor -- start 2024-03-11T19:11:06.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.646+0000 7fdbc5b8c640 1 -- start start 2024-03-11T19:11:06.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.646+0000 7fdbc5b8c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdbc0100080 0x7fdbc0197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.646+0000 7fdbc5b8c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdbc0101270 0x7fdbc0197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.646+0000 7fdbc5b8c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc0101c30 0x7fdbc019e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.646+0000 7fdbc5b8c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc019ec30 con 0x7fdbc0101c30 2024-03-11T19:11:06.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.646+0000 7fdbc5b8c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc019eda0 con 0x7fdbc0100080 2024-03-11T19:11:06.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.646+0000 7fdbc5b8c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc019ef60 con 0x7fdbc0101270 2024-03-11T19:11:06.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.646+0000 7fdbbffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc0101c30 0x7fdbc019e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.646+0000 7fdbbffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc0101c30 0x7fdbc019e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41502/0 (socket says 172.21.15.29:41502) 2024-03-11T19:11:06.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.646+0000 7fdbbffff640 1 -- 172.21.15.29:0/2638420746 learned_addr learned my addr 172.21.15.29:0/2638420746 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.646+0000 7fdbbf7fe640 1 --2- 172.21.15.29:0/2638420746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdbc0100080 0x7fdbc0197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.646+0000 7fdbbeffd640 1 --2- 172.21.15.29:0/2638420746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdbc0101270 0x7fdbc0197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.647+0000 7fdbbffff640 1 -- 172.21.15.29:0/2638420746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdbc0101270 msgr2=0x7fdbc0197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.647+0000 7fdbbffff640 1 --2- 172.21.15.29:0/2638420746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdbc0101270 0x7fdbc0197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.647+0000 7fdbbffff640 1 -- 172.21.15.29:0/2638420746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdbc0100080 msgr2=0x7fdbc0197140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.647+0000 7fdbbffff640 1 --2- 172.21.15.29:0/2638420746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdbc0100080 0x7fdbc0197140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.647+0000 7fdbbffff640 1 -- 172.21.15.29:0/2638420746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb4009560 con 0x7fdbc0101c30 2024-03-11T19:11:06.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.647+0000 7fdbbf7fe640 1 --2- 172.21.15.29:0/2638420746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdbc0100080 0x7fdbc0197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:06.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.647+0000 7fdbbffff640 1 --2- 172.21.15.29:0/2638420746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc0101c30 0x7fdbc019e6f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800eaa0 tx=0x7fdbb800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.647+0000 7fdbbcff9640 1 -- 172.21.15.29:0/2638420746 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb800cdb0 con 0x7fdbc0101c30 2024-03-11T19:11:06.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.647+0000 7fdbc5b8c640 1 -- 172.21.15.29:0/2638420746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc019f240 con 0x7fdbc0101c30 2024-03-11T19:11:06.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.647+0000 7fdbbcff9640 1 -- 172.21.15.29:0/2638420746 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdbb8005020 con 0x7fdbc0101c30 2024-03-11T19:11:06.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.647+0000 7fdbbcff9640 1 -- 172.21.15.29:0/2638420746 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb80116b0 con 0x7fdbc0101c30 2024-03-11T19:11:06.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.648+0000 7fdbc5b8c640 1 -- 172.21.15.29:0/2638420746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc019f720 con 0x7fdbc0101c30 2024-03-11T19:11:06.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.650+0000 7fdbc8766740 1 Processor -- start 2024-03-11T19:11:06.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.650+0000 7fdbbcff9640 1 -- 172.21.15.29:0/2638420746 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fdbb8005190 con 0x7fdbc0101c30 2024-03-11T19:11:06.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.651+0000 7fdbc8766740 1 -- start start 2024-03-11T19:11:06.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.651+0000 7fdbc8766740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559012919270 0x559012919690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.651+0000 7fdbc8766740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 0x55901291e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.651+0000 7fdbc8766740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55901291edd0 0x559012921260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.651+0000 7fdbc8766740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590128d5960 con 0x559012919c60 2024-03-11T19:11:06.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.651+0000 7fdbc8766740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559012889310 con 0x55901291edd0 2024-03-11T19:11:06.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.651+0000 7fdbc8766740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559012886840 con 0x559012919270 2024-03-11T19:11:06.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.651+0000 7fdbbf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 0x55901291e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.651+0000 7fdbbf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 0x55901291e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41516/0 (socket says 172.21.15.29:41516) 2024-03-11T19:11:06.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.651+0000 7fdbbf7fe640 1 -- 172.21.15.29:0/3467315007 learned_addr learned my addr 172.21.15.29:0/3467315007 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.651+0000 7fdbbeffd640 1 --2- 172.21.15.29:0/3467315007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55901291edd0 0x559012921260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.651+0000 7fdbbeffd640 1 --2- 172.21.15.29:0/3467315007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559012919270 0x559012919690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.651+0000 7fdbbcff9640 1 --2- 172.21.15.29:0/2638420746 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdba80a1ac0 0x7fdba80a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.651+0000 7fdbbf7fe640 1 -- 172.21.15.29:0/3467315007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559012919270 msgr2=0x559012919690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.651+0000 7fdbbf7fe640 1 --2- 172.21.15.29:0/3467315007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559012919270 0x559012919690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.651+0000 7fdbbf7fe640 1 -- 172.21.15.29:0/3467315007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55901291edd0 msgr2=0x559012921260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.651+0000 7fdbbf7fe640 1 --2- 172.21.15.29:0/3467315007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55901291edd0 0x559012921260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.652+0000 7fdbbf7fe640 1 -- 172.21.15.29:0/3467315007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559012844060 con 0x559012919c60 2024-03-11T19:11:06.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.652+0000 7fdbbffff640 1 --2- 172.21.15.29:0/2638420746 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdba80a1ac0 0x7fdba80a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.652+0000 7fdbbeffd640 1 --2- 172.21.15.29:0/3467315007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55901291edd0 0x559012921260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:06.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.652+0000 7fdbbcff9640 1 -- 172.21.15.29:0/2638420746 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fdbb80cd260 con 0x7fdbc0101c30 2024-03-11T19:11:06.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.652+0000 7fdbbf7fe640 1 --2- 172.21.15.29:0/3467315007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 0x55901291e760 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fdbb00114c0 tx=0x7fdbb0011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=620775912752dbb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.652+0000 7fdbbffff640 1 --2- 172.21.15.29:0/2638420746 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdba80a1ac0 0x7fdba80a3f80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8096bf0 tx=0x7fdbb8094c40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.652+0000 7fdba5ffb640 1 -- 172.21.15.29:0/3467315007 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0011250 con 0x559012919c60 2024-03-11T19:11:06.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.652+0000 7fdba5ffb640 1 -- 172.21.15.29:0/3467315007 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdbb00054d0 con 0x559012919c60 2024-03-11T19:11:06.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.652+0000 7fdba5ffb640 1 -- 172.21.15.29:0/3467315007 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb000f5c0 con 0x559012919c60 2024-03-11T19:11:06.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.653+0000 7fdbc8766740 1 -- 172.21.15.29:0/3467315007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 msgr2=0x55901291e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.653+0000 7fdbc8766740 1 --2- 172.21.15.29:0/3467315007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 0x55901291e760 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fdbb00114c0 tx=0x7fdbb0011980 comp rx=0 tx=0).stop 2024-03-11T19:11:06.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.653+0000 7fdbc8766740 1 -- 172.21.15.29:0/3467315007 shutdown_connections 2024-03-11T19:11:06.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.653+0000 7fdbc8766740 1 --2- 172.21.15.29:0/3467315007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55901291edd0 0x559012921260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.653+0000 7fdbc8766740 1 --2- 172.21.15.29:0/3467315007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 0x55901291e760 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.653+0000 7fdbc8766740 1 --2- 172.21.15.29:0/3467315007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559012919270 0x559012919690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.653+0000 7fdbc8766740 1 -- 172.21.15.29:0/3467315007 >> 172.21.15.29:0/3467315007 conn(0x5590129147e0 msgr2=0x559012914be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.653+0000 7fdbc8766740 1 -- 172.21.15.29:0/3467315007 shutdown_connections 2024-03-11T19:11:06.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.653+0000 7fdbc8766740 1 -- 172.21.15.29:0/3467315007 wait complete. 2024-03-11T19:11:06.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.653+0000 7fdbc8766740 1 -- start start 2024-03-11T19:11:06.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.654+0000 7fdbc8766740 1 Processor -- start 2024-03-11T19:11:06.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.654+0000 7fdbc8766740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559012919270 0x5590129344f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.654+0000 7fdbc8766740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559012919c60 0x559012934a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.654+0000 7fdbc8766740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55901291edd0 0x55901293ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.654+0000 7fdbc8766740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559012882020 con 0x55901291edd0 2024-03-11T19:11:06.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.654+0000 7fdbc8766740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559012555430 con 0x559012919c60 2024-03-11T19:11:06.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.654+0000 7fdbc8766740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559012511e00 con 0x559012919270 2024-03-11T19:11:06.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.654+0000 7fdbc8766740 10 client.0 _ll_register_callbacks cb 0x559012829cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:06.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.654+0000 7fdbbeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55901291edd0 0x55901293ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.654+0000 7fdbbeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55901291edd0 0x55901293ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41532/0 (socket says 172.21.15.29:41532) 2024-03-11T19:11:06.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.654+0000 7fdbbeffd640 1 -- 172.21.15.29:0/866236804 learned_addr learned my addr 172.21.15.29:0/866236804 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.654+0000 7fdbbeffd640 1 --2- 172.21.15.29:0/866236804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559012919270 0x5590129344f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.654+0000 7fdbbf7fe640 1 --2- 172.21.15.29:0/866236804 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559012919c60 0x559012934a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.654+0000 7fdbbeffd640 1 -- 172.21.15.29:0/866236804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559012919270 msgr2=0x5590129344f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.654+0000 7fdbbeffd640 1 --2- 172.21.15.29:0/866236804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559012919270 0x5590129344f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.654+0000 7fdbbeffd640 1 -- 172.21.15.29:0/866236804 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559012919c60 msgr2=0x559012934a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.654+0000 7fdbbeffd640 1 --2- 172.21.15.29:0/866236804 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559012919c60 0x559012934a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.654+0000 7fdbbeffd640 1 -- 172.21.15.29:0/866236804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb0009cc0 con 0x55901291edd0 2024-03-11T19:11:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.655+0000 7fdbbeffd640 1 --2- 172.21.15.29:0/866236804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55901291edd0 0x55901293ba70 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4009900 tx=0x7fdbb402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=831130072e8dc902 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.655+0000 7fdb9effd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.655+0000 7fdb9effd640 1 -- 172.21.15.29:0/866236804 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb403e650 con 0x55901291edd0 2024-03-11T19:11:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.655+0000 7fdb9effd640 1 -- 172.21.15.29:0/866236804 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fdbb402de40 con 0x55901291edd0 2024-03-11T19:11:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.655+0000 7fdbc8766740 1 -- 172.21.15.29:0/866236804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5590125817e0 con 0x55901291edd0 2024-03-11T19:11:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.655+0000 7fdb9effd640 1 -- 172.21.15.29:0/866236804 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdbb403e7b0 con 0x55901291edd0 2024-03-11T19:11:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.655+0000 7fdbc8766740 10 client.15714 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.655+0000 7fdbc8766740 1 -- 172.21.15.29:0/866236804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5590128806a0 con 0x55901291edd0 2024-03-11T19:11:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.655+0000 7fdb9effd640 1 -- 172.21.15.29:0/866236804 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5590125817e0 con 0x55901291edd0 2024-03-11T19:11:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.656+0000 7fdb9effd640 1 -- 172.21.15.29:0/866236804 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fdbb4003b20 con 0x55901291edd0 2024-03-11T19:11:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.656+0000 7fdbc8766740 10 client.15714 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.656+0000 7fdbc8766740 10 client.15714 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:11:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.656+0000 7fdbc8766740 20 client.15714 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.656+0000 7fdbc8766740 1 --2- 172.21.15.29:0/866236804 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5590129406f0 0x559012942ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.656+0000 7fdbc8766740 4 client.15714 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:11:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.656+0000 7fdbc8766740 1 -- 172.21.15.29:0/866236804 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5590125be720 con 0x5590129406f0 2024-03-11T19:11:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.656+0000 7fdbbf7fe640 1 --2- 172.21.15.29:0/866236804 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5590129406f0 0x559012942ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.656+0000 7fdb9effd640 10 client.15714 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:11:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.656+0000 7fdbbf7fe640 1 --2- 172.21.15.29:0/866236804 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5590129406f0 0x559012942ae0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=28df57da9a84e1e2 server_cookie=38cc4510f2677b5 in_seq=0 out_seq=0 2024-03-11T19:11:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.657+0000 7fdb9effd640 1 -- 172.21.15.29:0/866236804 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5590125be720 con 0x5590129406f0 2024-03-11T19:11:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.657+0000 7fdb9effd640 10 client.15714 handle_command_reply: tid=0 2024-03-11T19:11:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.657+0000 7fdbc8766740 1 client.15714 shutdown 2024-03-11T19:11:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.657+0000 7fdbc8766740 1 -- 172.21.15.29:0/866236804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55901291edd0 msgr2=0x55901293ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.657+0000 7fdbc8766740 1 --2- 172.21.15.29:0/866236804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55901291edd0 0x55901293ba70 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4009900 tx=0x7fdbb402bc30 comp rx=0 tx=0).stop 2024-03-11T19:11:06.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.657+0000 7fdbc8766740 1 -- 172.21.15.29:0/866236804 shutdown_connections 2024-03-11T19:11:06.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.657+0000 7fdbc8766740 1 --2- 172.21.15.29:0/866236804 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5590129406f0 0x559012942ae0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.657+0000 7fdbc8766740 1 --2- 172.21.15.29:0/866236804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55901291edd0 0x55901293ba70 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.657+0000 7fdb9effd640 0 client.15714 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:11:06.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.657+0000 7fdbc8766740 1 --2- 172.21.15.29:0/866236804 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559012919c60 0x559012934a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.657+0000 7fdbc8766740 1 --2- 172.21.15.29:0/866236804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559012919270 0x5590129344f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.657+0000 7fdbc8766740 1 -- 172.21.15.29:0/866236804 >> 172.21.15.29:0/866236804 conn(0x5590129147e0 msgr2=0x559012913b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.657+0000 7fdbc8766740 1 -- 172.21.15.29:0/866236804 shutdown_connections 2024-03-11T19:11:06.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.658+0000 7fdbc8766740 1 -- 172.21.15.29:0/866236804 wait complete. 2024-03-11T19:11:06.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.658+0000 7fdbc8766740 20 client.15714 trim_cache size 0 max 16384 2024-03-11T19:11:06.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.669+0000 7fdbc8766740 1 Processor -- start 2024-03-11T19:11:06.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.669+0000 7fdbc8766740 1 -- start start 2024-03-11T19:11:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.669+0000 7fdbc8766740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559012919270 0x55901290fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.669+0000 7fdbc8766740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 0x55901293b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.669+0000 7fdbc8766740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5590129344f0 0x559012934970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.670+0000 7fdbc8766740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559012807130 con 0x559012919c60 2024-03-11T19:11:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.670+0000 7fdbc8766740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590125817e0 con 0x559012919270 2024-03-11T19:11:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.670+0000 7fdbc8766740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559012906a40 con 0x5590129344f0 2024-03-11T19:11:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.670+0000 7fdbbf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 0x55901293b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.670+0000 7fdbbeffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559012919270 0x55901290fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.670+0000 7fdbbf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 0x55901293b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41536/0 (socket says 172.21.15.29:41536) 2024-03-11T19:11:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.670+0000 7fdbbf7fe640 1 -- 172.21.15.29:0/250825844 learned_addr learned my addr 172.21.15.29:0/250825844 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.670+0000 7fdbbeffd640 1 --2- 172.21.15.29:0/250825844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5590129344f0 0x559012934970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.670+0000 7fdbbeffd640 1 -- 172.21.15.29:0/250825844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5590129344f0 msgr2=0x559012934970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.670+0000 7fdbbeffd640 1 --2- 172.21.15.29:0/250825844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5590129344f0 0x559012934970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.670+0000 7fdbbeffd640 1 -- 172.21.15.29:0/250825844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 msgr2=0x55901293b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.670+0000 7fdbbeffd640 1 --2- 172.21.15.29:0/250825844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 0x55901293b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.670+0000 7fdbbeffd640 1 -- 172.21.15.29:0/250825844 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55901281cc20 con 0x559012919270 2024-03-11T19:11:06.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.670+0000 7fdbbf7fe640 1 --2- 172.21.15.29:0/250825844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 0x55901293b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:06.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.670+0000 7fdbbeffd640 1 --2- 172.21.15.29:0/250825844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559012919270 0x55901290fa10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4003ed0 tx=0x7fdbb402e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab4840857be9b3f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.670+0000 7fdba57fa640 1 -- 172.21.15.29:0/250825844 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4004ee0 con 0x559012919270 2024-03-11T19:11:06.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.671+0000 7fdba57fa640 1 -- 172.21.15.29:0/250825844 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdbb4003aa0 con 0x559012919270 2024-03-11T19:11:06.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.671+0000 7fdba57fa640 1 -- 172.21.15.29:0/250825844 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb40054c0 con 0x559012919270 2024-03-11T19:11:06.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.671+0000 7fdbc8766740 1 -- 172.21.15.29:0/250825844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559012919270 msgr2=0x55901290fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.671+0000 7fdbc8766740 1 --2- 172.21.15.29:0/250825844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559012919270 0x55901290fa10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4003ed0 tx=0x7fdbb402e040 comp rx=0 tx=0).stop 2024-03-11T19:11:06.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.671+0000 7fdbc8766740 1 -- 172.21.15.29:0/250825844 shutdown_connections 2024-03-11T19:11:06.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.671+0000 7fdbc8766740 1 --2- 172.21.15.29:0/250825844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5590129344f0 0x559012934970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.672+0000 7fdbc8766740 1 --2- 172.21.15.29:0/250825844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 0x55901293b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.672+0000 7fdbc8766740 1 --2- 172.21.15.29:0/250825844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559012919270 0x55901290fa10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.672+0000 7fdbc8766740 1 -- 172.21.15.29:0/250825844 >> 172.21.15.29:0/250825844 conn(0x5590129147e0 msgr2=0x559012932fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.672+0000 7fdbc8766740 1 -- 172.21.15.29:0/250825844 shutdown_connections 2024-03-11T19:11:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.672+0000 7fdbc8766740 1 -- 172.21.15.29:0/250825844 wait complete. 2024-03-11T19:11:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.672+0000 7fdbc8766740 1 -- start start 2024-03-11T19:11:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.672+0000 7fdbc8766740 1 Processor -- start 2024-03-11T19:11:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.672+0000 7fdbc8766740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559012919270 0x55901293b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.672+0000 7fdbc8766740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 0x559012939e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.673+0000 7fdbc8766740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5590129344f0 0x55901293a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.673+0000 7fdbbf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 0x559012939e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.673+0000 7fdbbf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 0x559012939e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41548/0 (socket says 172.21.15.29:41548) 2024-03-11T19:11:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.673+0000 7fdbbf7fe640 1 -- 172.21.15.29:0/245689151 learned_addr learned my addr 172.21.15.29:0/245689151 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.673+0000 7fdbbeffd640 1 --2- 172.21.15.29:0/245689151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559012919270 0x55901293b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.673+0000 7fdbc8766740 1 -- 172.21.15.29:0/245689151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55901293a8b0 con 0x559012919c60 2024-03-11T19:11:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.673+0000 7fdbc8766740 1 -- 172.21.15.29:0/245689151 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55901293aa20 con 0x559012919270 2024-03-11T19:11:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.673+0000 7fdbc8766740 1 -- 172.21.15.29:0/245689151 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55901293ab90 con 0x5590129344f0 2024-03-11T19:11:06.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.673+0000 7fdbbeffd640 1 --2- 172.21.15.29:0/245689151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5590129344f0 0x55901293a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.673+0000 7fdbbf7fe640 1 -- 172.21.15.29:0/245689151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5590129344f0 msgr2=0x55901293a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.673+0000 7fdbbf7fe640 1 --2- 172.21.15.29:0/245689151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5590129344f0 0x55901293a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.673+0000 7fdbbf7fe640 1 -- 172.21.15.29:0/245689151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559012919270 msgr2=0x55901293b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.673+0000 7fdbbf7fe640 1 --2- 172.21.15.29:0/245689151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559012919270 0x55901293b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.673+0000 7fdbbf7fe640 1 -- 172.21.15.29:0/245689151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559012844060 con 0x559012919c60 2024-03-11T19:11:06.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.673+0000 7fdbbf7fe640 1 --2- 172.21.15.29:0/245689151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 0x559012939e30 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0000c00 tx=0x7fdbb0011060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71b3810f38efb835 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.673+0000 7fdb9effd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:06.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.673+0000 7fdbc8766740 10 client.0 _ll_register_callbacks cb 0x7fdbb0009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:06.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.674+0000 7fdbc8766740 1 -- 172.21.15.29:0/245689151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdbb00054d0 con 0x559012919c60 2024-03-11T19:11:06.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.674+0000 7fdb9effd640 1 -- 172.21.15.29:0/245689151 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0004040 con 0x559012919c60 2024-03-11T19:11:06.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.674+0000 7fdb9effd640 1 -- 172.21.15.29:0/245689151 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fdbb00041a0 con 0x559012919c60 2024-03-11T19:11:06.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.674+0000 7fdb9effd640 1 -- 172.21.15.29:0/245689151 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdbb0004360 con 0x559012919c60 2024-03-11T19:11:06.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.674+0000 7fdbc8766740 10 client.15720 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:06.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.674+0000 7fdbc8766740 1 -- 172.21.15.29:0/245689151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x5590128806a0 con 0x559012919c60 2024-03-11T19:11:06.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.674+0000 7fdb9effd640 1 -- 172.21.15.29:0/245689151 <== mon.0 v2:172.21.15.29:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fdbb00044c0 con 0x559012919c60 2024-03-11T19:11:06.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.674+0000 7fdbc8766740 10 client.15720 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:06.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.674+0000 7fdbc8766740 10 client.15720 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:11:06.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.674+0000 7fdbc8766740 20 client.15720 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:06.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.674+0000 7fdbc8766740 1 --2- 172.21.15.29:0/245689151 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x559012998040 0x55901299a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.674+0000 7fdbc8766740 4 client.15720 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:11:06.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.674+0000 7fdbc8766740 1 -- 172.21.15.29:0/245689151 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55901260d890 con 0x559012998040 2024-03-11T19:11:06.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.675+0000 7fdbbeffd640 1 --2- 172.21.15.29:0/245689151 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x559012998040 0x55901299a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.675+0000 7fdb9effd640 10 client.15720 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:11:06.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.675+0000 7fdbbeffd640 1 --2- 172.21.15.29:0/245689151 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x559012998040 0x55901299a500 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3838daf3d87ae380 server_cookie=ff2a5656ec6f5205 in_seq=0 out_seq=0 2024-03-11T19:11:06.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.675+0000 7fdb9effd640 1 -- 172.21.15.29:0/245689151 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55901260d890 con 0x559012998040 2024-03-11T19:11:06.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.675+0000 7fdb9effd640 10 client.15720 handle_command_reply: tid=0 2024-03-11T19:11:06.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.675+0000 7fdbc8766740 1 client.15720 shutdown 2024-03-11T19:11:06.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.675+0000 7fdbc8766740 1 -- 172.21.15.29:0/245689151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 msgr2=0x559012939e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.675+0000 7fdbc8766740 1 --2- 172.21.15.29:0/245689151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 0x559012939e30 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0000c00 tx=0x7fdbb0011060 comp rx=0 tx=0).stop 2024-03-11T19:11:06.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.676+0000 7fdbc8766740 1 -- 172.21.15.29:0/245689151 shutdown_connections 2024-03-11T19:11:06.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.676+0000 7fdbc8766740 1 --2- 172.21.15.29:0/245689151 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x559012998040 0x55901299a500 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.676+0000 7fdbc8766740 1 --2- 172.21.15.29:0/245689151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5590129344f0 0x55901293a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.676+0000 7fdb9effd640 0 client.15720 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:11:06.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.676+0000 7fdbc8766740 1 --2- 172.21.15.29:0/245689151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559012919c60 0x559012939e30 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.676+0000 7fdbc8766740 1 --2- 172.21.15.29:0/245689151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559012919270 0x55901293b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.676+0000 7fdbc8766740 1 -- 172.21.15.29:0/245689151 >> 172.21.15.29:0/245689151 conn(0x5590129147e0 msgr2=0x55901290fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.676+0000 7fdbc8766740 1 -- 172.21.15.29:0/245689151 shutdown_connections 2024-03-11T19:11:06.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.676+0000 7fdbc8766740 1 -- 172.21.15.29:0/245689151 wait complete. 2024-03-11T19:11:06.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.676+0000 7fdbc8766740 20 client.15720 trim_cache size 0 max 16384 2024-03-11T19:11:06.685 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:11:06.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.677+0000 7fdbc5b8c640 1 -- 172.21.15.29:0/2638420746 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdba80a1ac0 msgr2=0x7fdba80a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.677+0000 7fdbc5b8c640 1 --2- 172.21.15.29:0/2638420746 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdba80a1ac0 0x7fdba80a3f80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8096bf0 tx=0x7fdbb8094c40 comp rx=0 tx=0).stop 2024-03-11T19:11:06.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.677+0000 7fdbc5b8c640 1 -- 172.21.15.29:0/2638420746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc0101c30 msgr2=0x7fdbc019e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.677+0000 7fdbc5b8c640 1 --2- 172.21.15.29:0/2638420746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc0101c30 0x7fdbc019e6f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800eaa0 tx=0x7fdbb800ef60 comp rx=0 tx=0).stop 2024-03-11T19:11:06.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.677+0000 7fdbc5b8c640 1 -- 172.21.15.29:0/2638420746 shutdown_connections 2024-03-11T19:11:06.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.677+0000 7fdbc5b8c640 1 --2- 172.21.15.29:0/2638420746 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdba80a1ac0 0x7fdba80a3f80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.677+0000 7fdbc5b8c640 1 --2- 172.21.15.29:0/2638420746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc0101c30 0x7fdbc019e6f0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.677+0000 7fdbc5b8c640 1 --2- 172.21.15.29:0/2638420746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdbc0101270 0x7fdbc0197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.677+0000 7fdbc5b8c640 1 --2- 172.21.15.29:0/2638420746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdbc0100080 0x7fdbc0197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.677+0000 7fdbc5b8c640 1 -- 172.21.15.29:0/2638420746 >> 172.21.15.29:0/2638420746 conn(0x7fdbc00fb610 msgr2=0x7fdbc00fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.677+0000 7fdbc5b8c640 1 -- 172.21.15.29:0/2638420746 shutdown_connections 2024-03-11T19:11:06.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.677+0000 7fdbc5b8c640 1 -- 172.21.15.29:0/2638420746 wait complete. 2024-03-11T19:11:06.694 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:11:06.695 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:11:06.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.852+0000 7fe3be29d640 1 -- 172.21.15.29:0/4235177976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3b81000a0 msgr2=0x7fe3b81004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.852+0000 7fe3be29d640 1 --2- 172.21.15.29:0/4235177976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3b81000a0 0x7fe3b81004c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8009900 tx=0x7fe3a802de40 comp rx=0 tx=0).stop 2024-03-11T19:11:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.852+0000 7fe3be29d640 1 -- 172.21.15.29:0/4235177976 shutdown_connections 2024-03-11T19:11:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.852+0000 7fe3be29d640 1 --2- 172.21.15.29:0/4235177976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3b8101c50 0x7fe3b81084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.852+0000 7fe3be29d640 1 --2- 172.21.15.29:0/4235177976 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3b8101290 0x7fe3b8101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.852+0000 7fe3be29d640 1 --2- 172.21.15.29:0/4235177976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3b81000a0 0x7fe3b81004c0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.852+0000 7fe3be29d640 1 -- 172.21.15.29:0/4235177976 >> 172.21.15.29:0/4235177976 conn(0x7fe3b80fb610 msgr2=0x7fe3b80fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.852+0000 7fe3be29d640 1 -- 172.21.15.29:0/4235177976 shutdown_connections 2024-03-11T19:11:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.852+0000 7fe3be29d640 1 -- 172.21.15.29:0/4235177976 wait complete. 2024-03-11T19:11:06.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.853+0000 7fe3be29d640 1 Processor -- start 2024-03-11T19:11:06.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.853+0000 7fe3be29d640 1 -- start start 2024-03-11T19:11:06.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.853+0000 7fe3be29d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3b81000a0 0x7fe3b8195020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.853+0000 7fe3be29d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3b8101290 0x7fe3b8195560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.853+0000 7fe3be29d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3b8101c50 0x7fe3b819c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.853+0000 7fe3be29d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b819cac0 con 0x7fe3b8101c50 2024-03-11T19:11:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.853+0000 7fe3be29d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b819cc30 con 0x7fe3b8101290 2024-03-11T19:11:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.853+0000 7fe3be29d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b819cdf0 con 0x7fe3b81000a0 2024-03-11T19:11:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.854+0000 7fe3bc83a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3b8101c50 0x7fe3b819c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.854+0000 7fe3bc83a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3b8101c50 0x7fe3b819c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41564/0 (socket says 172.21.15.29:41564) 2024-03-11T19:11:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.854+0000 7fe3bc83a640 1 -- 172.21.15.29:0/3907456989 learned_addr learned my addr 172.21.15.29:0/3907456989 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.854+0000 7fe3b7fff640 1 --2- 172.21.15.29:0/3907456989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3b81000a0 0x7fe3b8195020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.854+0000 7fe3b77fe640 1 --2- 172.21.15.29:0/3907456989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3b8101290 0x7fe3b8195560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.854+0000 7fe3bc83a640 1 -- 172.21.15.29:0/3907456989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3b81000a0 msgr2=0x7fe3b8195020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.854+0000 7fe3bc83a640 1 --2- 172.21.15.29:0/3907456989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3b81000a0 0x7fe3b8195020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.854+0000 7fe3bc83a640 1 -- 172.21.15.29:0/3907456989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3b8101290 msgr2=0x7fe3b8195560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.854+0000 7fe3bc83a640 1 --2- 172.21.15.29:0/3907456989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3b8101290 0x7fe3b8195560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.854+0000 7fe3bc83a640 1 -- 172.21.15.29:0/3907456989 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a8009560 con 0x7fe3b8101c50 2024-03-11T19:11:06.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.854+0000 7fe3b77fe640 1 --2- 172.21.15.29:0/3907456989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3b8101290 0x7fe3b8195560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:06.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.854+0000 7fe3b7fff640 1 --2- 172.21.15.29:0/3907456989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3b81000a0 0x7fe3b8195020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:06.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.854+0000 7fe3bc83a640 1 --2- 172.21.15.29:0/3907456989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3b8101c50 0x7fe3b819c580 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000eb10 tx=0x7fe3b000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.854+0000 7fe3b57fa640 1 -- 172.21.15.29:0/3907456989 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0004b20 con 0x7fe3b8101c50 2024-03-11T19:11:06.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.854+0000 7fe3be29d640 1 -- 172.21.15.29:0/3907456989 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b81a14e0 con 0x7fe3b8101c50 2024-03-11T19:11:06.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.855+0000 7fe3b57fa640 1 -- 172.21.15.29:0/3907456989 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe3b0004c80 con 0x7fe3b8101c50 2024-03-11T19:11:06.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.855+0000 7fe3b57fa640 1 -- 172.21.15.29:0/3907456989 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0010690 con 0x7fe3b8101c50 2024-03-11T19:11:06.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.855+0000 7fe3be29d640 1 -- 172.21.15.29:0/3907456989 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b81a19f0 con 0x7fe3b8101c50 2024-03-11T19:11:06.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.858+0000 7fe3b57fa640 1 -- 172.21.15.29:0/3907456989 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fe3b0005020 con 0x7fe3b8101c50 2024-03-11T19:11:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.858+0000 7fe3c0e79740 1 Processor -- start 2024-03-11T19:11:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.858+0000 7fe3c0e79740 1 -- start start 2024-03-11T19:11:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3c0e79740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fba7270 0x55618fba7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3b57fa640 1 --2- 172.21.15.29:0/3907456989 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3a00a1ac0 0x7fe3a00a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3c0e79740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 0x55618fbac760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3c0e79740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fbacdd0 0x55618fbaf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3c0e79740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55618fb63960 con 0x55618fba7c60 2024-03-11T19:11:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3c0e79740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55618fb17310 con 0x55618fba7270 2024-03-11T19:11:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3c0e79740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55618fb14840 con 0x55618fbacdd0 2024-03-11T19:11:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3b57fa640 1 -- 172.21.15.29:0/3907456989 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe3b0014070 con 0x7fe3b8101c50 2024-03-11T19:11:06.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3b7fff640 1 --2- 172.21.15.29:0/3907456989 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3a00a1ac0 0x7fe3a00a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3b77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 0x55618fbac760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3b77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 0x55618fbac760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41568/0 (socket says 172.21.15.29:41568) 2024-03-11T19:11:06.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3b77fe640 1 -- 172.21.15.29:0/3968206292 learned_addr learned my addr 172.21.15.29:0/3968206292 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3bc83a640 1 --2- 172.21.15.29:0/3968206292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fbacdd0 0x55618fbaf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3bc83a640 1 -- 172.21.15.29:0/3968206292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fba7270 msgr2=0x55618fba7690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:11:06.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3bc83a640 1 --2- 172.21.15.29:0/3968206292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fba7270 0x55618fba7690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3bc83a640 1 -- 172.21.15.29:0/3968206292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 msgr2=0x55618fbac760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3bc83a640 1 --2- 172.21.15.29:0/3968206292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 0x55618fbac760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3bc83a640 1 -- 172.21.15.29:0/3968206292 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55618fad2060 con 0x55618fbacdd0 2024-03-11T19:11:06.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3b77fe640 1 --2- 172.21.15.29:0/3968206292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 0x55618fbac760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:06.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.859+0000 7fe3b7fff640 1 --2- 172.21.15.29:0/3907456989 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3a00a1ac0 0x7fe3a00a3f80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe3a802e450 tx=0x7fe3a8004920 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.860+0000 7fe3bc83a640 1 --2- 172.21.15.29:0/3968206292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fbacdd0 0x55618fbaf260 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe3b00cb5a0 tx=0x7fe3b000cee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=750538547f574634 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.860+0000 7fe39e7fc640 1 -- 172.21.15.29:0/3968206292 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0065630 con 0x55618fbacdd0 2024-03-11T19:11:06.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.860+0000 7fe39e7fc640 1 -- 172.21.15.29:0/3968206292 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe3b009aa00 con 0x55618fbacdd0 2024-03-11T19:11:06.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.860+0000 7fe39e7fc640 1 -- 172.21.15.29:0/3968206292 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0097070 con 0x55618fbacdd0 2024-03-11T19:11:06.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.860+0000 7fe3c0e79740 1 -- 172.21.15.29:0/3968206292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fbacdd0 msgr2=0x55618fbaf260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.860+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/3968206292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fbacdd0 0x55618fbaf260 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe3b00cb5a0 tx=0x7fe3b000cee0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.860+0000 7fe3c0e79740 1 -- 172.21.15.29:0/3968206292 shutdown_connections 2024-03-11T19:11:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.860+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/3968206292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fbacdd0 0x55618fbaf260 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.861+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/3968206292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 0x55618fbac760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.861+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/3968206292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fba7270 0x55618fba7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.861+0000 7fe3c0e79740 1 -- 172.21.15.29:0/3968206292 >> 172.21.15.29:0/3968206292 conn(0x55618fba27e0 msgr2=0x55618fba2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.861+0000 7fe3c0e79740 1 -- 172.21.15.29:0/3968206292 shutdown_connections 2024-03-11T19:11:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.861+0000 7fe3c0e79740 1 -- 172.21.15.29:0/3968206292 wait complete. 2024-03-11T19:11:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.861+0000 7fe3c0e79740 1 -- start start 2024-03-11T19:11:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.861+0000 7fe3c0e79740 1 Processor -- start 2024-03-11T19:11:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.861+0000 7fe3c0e79740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fba7270 0x55618fbb1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3c0e79740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fba7c60 0x55618fbb1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3c0e79740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fbacdd0 0x55618fbb8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3c0e79740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55618fb10020 con 0x55618fbacdd0 2024-03-11T19:11:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3c0e79740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55618f7e3430 con 0x55618fba7c60 2024-03-11T19:11:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3c0e79740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55618f79fe00 con 0x55618fba7270 2024-03-11T19:11:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3c0e79740 10 client.0 _ll_register_callbacks cb 0x55618fab7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3b7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fba7270 0x55618fbb1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3b7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fba7270 0x55618fbb1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48132/0 (socket says 172.21.15.29:48132) 2024-03-11T19:11:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3b7fff640 1 -- 172.21.15.29:0/3154520920 learned_addr learned my addr 172.21.15.29:0/3154520920 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3bc83a640 1 --2- 172.21.15.29:0/3154520920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fbacdd0 0x55618fbb8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3b77fe640 1 --2- 172.21.15.29:0/3154520920 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fba7c60 0x55618fbb1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3b7fff640 1 -- 172.21.15.29:0/3154520920 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fba7c60 msgr2=0x55618fbb1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3b7fff640 1 --2- 172.21.15.29:0/3154520920 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fba7c60 0x55618fbb1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3b7fff640 1 -- 172.21.15.29:0/3154520920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fbacdd0 msgr2=0x55618fbb8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3b7fff640 1 --2- 172.21.15.29:0/3154520920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fbacdd0 0x55618fbb8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3b7fff640 1 -- 172.21.15.29:0/3154520920 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b81a19f0 con 0x55618fba7270 2024-03-11T19:11:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3b77fe640 1 --2- 172.21.15.29:0/3154520920 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fba7c60 0x55618fbb1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3b7fff640 1 --2- 172.21.15.29:0/3154520920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fba7270 0x55618fbb1550 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8009900 tx=0x7fe3a8009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb41bae9c3a99981 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3977fe640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:11:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.862+0000 7fe3977fe640 1 -- 172.21.15.29:0/3154520920 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a80053f0 con 0x55618fba7270 2024-03-11T19:11:06.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.863+0000 7fe3c0e79740 1 -- 172.21.15.29:0/3154520920 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55618f80f7e0 con 0x55618fba7270 2024-03-11T19:11:06.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.863+0000 7fe3977fe640 1 -- 172.21.15.29:0/3154520920 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe3a8047040 con 0x55618fba7270 2024-03-11T19:11:06.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.863+0000 7fe3977fe640 1 -- 172.21.15.29:0/3154520920 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3a8005550 con 0x55618fba7270 2024-03-11T19:11:06.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.863+0000 7fe3c0e79740 10 client.25342 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:06.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.863+0000 7fe3c0e79740 1 -- 172.21.15.29:0/3154520920 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55618fb0e6a0 con 0x55618fba7270 2024-03-11T19:11:06.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.863+0000 7fe3977fe640 1 -- 172.21.15.29:0/3154520920 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55618f80f7e0 con 0x55618fba7270 2024-03-11T19:11:06.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.863+0000 7fe3977fe640 1 -- 172.21.15.29:0/3154520920 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe3a803e040 con 0x55618fba7270 2024-03-11T19:11:06.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.863+0000 7fe3c0e79740 10 client.25342 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:06.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.863+0000 7fe3c0e79740 10 client.25342 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:11:06.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.863+0000 7fe3c0e79740 20 client.25342 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:06.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.863+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/3154520920 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55618fbbd5c0 0x55618fbbf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.863+0000 7fe3c0e79740 4 client.25342 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:11:06.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.863+0000 7fe3c0e79740 1 -- 172.21.15.29:0/3154520920 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55618f84c720 con 0x55618fbbd5c0 2024-03-11T19:11:06.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.864+0000 7fe3b77fe640 1 --2- 172.21.15.29:0/3154520920 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55618fbbd5c0 0x55618fbbf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.864+0000 7fe3977fe640 10 client.25342 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:11:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.864+0000 7fe3b77fe640 1 --2- 172.21.15.29:0/3154520920 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55618fbbd5c0 0x55618fbbf9e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=40e65975225e03fd server_cookie=16688655f6f5f48 in_seq=0 out_seq=0 2024-03-11T19:11:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.865+0000 7fe3977fe640 1 -- 172.21.15.29:0/3154520920 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55618f84c720 con 0x55618fbbd5c0 2024-03-11T19:11:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.865+0000 7fe3977fe640 10 client.25342 handle_command_reply: tid=0 2024-03-11T19:11:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.865+0000 7fe3c0e79740 1 client.25342 shutdown 2024-03-11T19:11:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.865+0000 7fe3c0e79740 1 -- 172.21.15.29:0/3154520920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fba7270 msgr2=0x55618fbb1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.865+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/3154520920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fba7270 0x55618fbb1550 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8009900 tx=0x7fe3a8009260 comp rx=0 tx=0).stop 2024-03-11T19:11:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.865+0000 7fe3c0e79740 1 -- 172.21.15.29:0/3154520920 shutdown_connections 2024-03-11T19:11:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.865+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/3154520920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fbacdd0 0x55618fbb8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.865+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/3154520920 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55618fbbd5c0 0x55618fbbf9e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.865+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/3154520920 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fba7c60 0x55618fbb1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.865+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/3154520920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fba7270 0x55618fbb1550 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.865+0000 7fe3c0e79740 1 -- 172.21.15.29:0/3154520920 >> 172.21.15.29:0/3154520920 conn(0x55618fba27e0 msgr2=0x55618fba1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.865+0000 7fe3977fe640 0 client.25342 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:11:06.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.865+0000 7fe3c0e79740 1 -- 172.21.15.29:0/3154520920 shutdown_connections 2024-03-11T19:11:06.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.865+0000 7fe3c0e79740 1 -- 172.21.15.29:0/3154520920 wait complete. 2024-03-11T19:11:06.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.866+0000 7fe3c0e79740 20 client.25342 trim_cache size 0 max 16384 2024-03-11T19:11:06.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.878+0000 7fe3c0e79740 1 Processor -- start 2024-03-11T19:11:06.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.878+0000 7fe3c0e79740 1 -- start start 2024-03-11T19:11:06.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.878+0000 7fe3c0e79740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fba7270 0x55618fbb1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.878+0000 7fe3c0e79740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 0x55618fbb8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.878+0000 7fe3c0e79740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fbacdd0 0x55618fbb16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.878+0000 7fe3c0e79740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55618fa95130 con 0x55618fba7c60 2024-03-11T19:11:06.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.878+0000 7fe3c0e79740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55618fba84f0 con 0x55618fbacdd0 2024-03-11T19:11:06.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.878+0000 7fe3c0e79740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55618fb94a40 con 0x55618fba7270 2024-03-11T19:11:06.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.878+0000 7fe3b77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 0x55618fbb8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.878+0000 7fe3b77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 0x55618fbb8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41588/0 (socket says 172.21.15.29:41588) 2024-03-11T19:11:06.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.878+0000 7fe3b77fe640 1 -- 172.21.15.29:0/1275477951 learned_addr learned my addr 172.21.15.29:0/1275477951 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.878+0000 7fe3bc83a640 1 --2- 172.21.15.29:0/1275477951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fbacdd0 0x55618fbb16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.878+0000 7fe3b7fff640 1 --2- 172.21.15.29:0/1275477951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fba7270 0x55618fbb1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.878+0000 7fe3b77fe640 1 -- 172.21.15.29:0/1275477951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fba7270 msgr2=0x55618fbb1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.878+0000 7fe3b77fe640 1 --2- 172.21.15.29:0/1275477951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fba7270 0x55618fbb1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.878+0000 7fe3b77fe640 1 -- 172.21.15.29:0/1275477951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fbacdd0 msgr2=0x55618fbb16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.878+0000 7fe3b77fe640 1 --2- 172.21.15.29:0/1275477951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fbacdd0 0x55618fbb16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.878+0000 7fe3b77fe640 1 -- 172.21.15.29:0/1275477951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55618faaac20 con 0x55618fba7c60 2024-03-11T19:11:06.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.878+0000 7fe3b77fe640 1 --2- 172.21.15.29:0/1275477951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 0x55618fbb8a60 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x55618fba86d0 tx=0x7fe3ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b33d85804b347a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.879+0000 7fe39dffb640 1 -- 172.21.15.29:0/1275477951 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac024070 con 0x55618fba7c60 2024-03-11T19:11:06.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.879+0000 7fe39dffb640 1 -- 172.21.15.29:0/1275477951 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe3ac018040 con 0x55618fba7c60 2024-03-11T19:11:06.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.879+0000 7fe39dffb640 1 -- 172.21.15.29:0/1275477951 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac01f2d0 con 0x55618fba7c60 2024-03-11T19:11:06.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.879+0000 7fe3c0e79740 1 -- 172.21.15.29:0/1275477951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 msgr2=0x55618fbb8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.879+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/1275477951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 0x55618fbb8a60 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x55618fba86d0 tx=0x7fe3ac0046c0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.879+0000 7fe3c0e79740 1 -- 172.21.15.29:0/1275477951 shutdown_connections 2024-03-11T19:11:06.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.879+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/1275477951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fbacdd0 0x55618fbb16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.879+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/1275477951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 0x55618fbb8a60 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.879+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/1275477951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fba7270 0x55618fbb1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.879+0000 7fe3c0e79740 1 -- 172.21.15.29:0/1275477951 >> 172.21.15.29:0/1275477951 conn(0x55618fba27e0 msgr2=0x55618fba9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.880+0000 7fe3c0e79740 1 -- 172.21.15.29:0/1275477951 shutdown_connections 2024-03-11T19:11:06.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.880+0000 7fe3c0e79740 1 -- 172.21.15.29:0/1275477951 wait complete. 2024-03-11T19:11:06.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.880+0000 7fe3c0e79740 1 -- start start 2024-03-11T19:11:06.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.880+0000 7fe3c0e79740 1 Processor -- start 2024-03-11T19:11:06.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.880+0000 7fe3c0e79740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fba7270 0x55618fbb8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.880+0000 7fe3c0e79740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 0x55618fbb87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.880+0000 7fe3c0e79740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fbacdd0 0x55618fbb68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.880+0000 7fe3c0e79740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55618fbb6e30 con 0x55618fba7c60 2024-03-11T19:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.880+0000 7fe3c0e79740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55618fbb6f70 con 0x55618fbacdd0 2024-03-11T19:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.880+0000 7fe3c0e79740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55618fbb7130 con 0x55618fba7270 2024-03-11T19:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.880+0000 7fe3b77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 0x55618fbb87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.880+0000 7fe3b77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 0x55618fbb87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41604/0 (socket says 172.21.15.29:41604) 2024-03-11T19:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.880+0000 7fe3b77fe640 1 -- 172.21.15.29:0/1254954809 learned_addr learned my addr 172.21.15.29:0/1254954809 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.881+0000 7fe3c0e79740 10 client.0 _ll_register_callbacks cb 0x7fe3b0096ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.881+0000 7fe3b7fff640 1 --2- 172.21.15.29:0/1254954809 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fba7270 0x55618fbb8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.881+0000 7fe3bc83a640 1 --2- 172.21.15.29:0/1254954809 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fbacdd0 0x55618fbb68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.881+0000 7fe3b77fe640 1 -- 172.21.15.29:0/1254954809 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fba7270 msgr2=0x55618fbb8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.881+0000 7fe3b77fe640 1 --2- 172.21.15.29:0/1254954809 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fba7270 0x55618fbb8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.881+0000 7fe3b77fe640 1 -- 172.21.15.29:0/1254954809 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fbacdd0 msgr2=0x55618fbb68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.881+0000 7fe3b77fe640 1 --2- 172.21.15.29:0/1254954809 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fbacdd0 0x55618fbb68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.881+0000 7fe3b77fe640 1 -- 172.21.15.29:0/1254954809 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ac009b90 con 0x55618fba7c60 2024-03-11T19:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.881+0000 7fe3b77fe640 1 --2- 172.21.15.29:0/1254954809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 0x55618fbb87d0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac022040 tx=0x7fe3ac0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68c700c49eec7a05 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.881+0000 7fe3977fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.881+0000 7fe3977fe640 1 -- 172.21.15.29:0/1254954809 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac02e030 con 0x55618fba7c60 2024-03-11T19:11:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.881+0000 7fe3977fe640 1 -- 172.21.15.29:0/1254954809 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe3ac02f070 con 0x55618fba7c60 2024-03-11T19:11:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.881+0000 7fe3c0e79740 1 -- 172.21.15.29:0/1254954809 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe3b009aa00 con 0x55618fba7c60 2024-03-11T19:11:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.882+0000 7fe3977fe640 1 -- 172.21.15.29:0/1254954809 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3ac024040 con 0x55618fba7c60 2024-03-11T19:11:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.882+0000 7fe3c0e79740 10 client.15738 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.882+0000 7fe3c0e79740 1 -- 172.21.15.29:0/1254954809 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55618fad08c0 con 0x55618fba7c60 2024-03-11T19:11:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.882+0000 7fe3977fe640 1 -- 172.21.15.29:0/1254954809 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe3ac039020 con 0x55618fba7c60 2024-03-11T19:11:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.882+0000 7fe3977fe640 1 -- 172.21.15.29:0/1254954809 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe3ac004a80 con 0x55618fba7c60 2024-03-11T19:11:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.882+0000 7fe3c0e79740 10 client.15738 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.882+0000 7fe3c0e79740 10 client.15738 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:11:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.882+0000 7fe3c0e79740 20 client.15738 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.882+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/1254954809 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55618fc14d40 0x55618fc17160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.882+0000 7fe3c0e79740 4 client.15738 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:11:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.882+0000 7fe3c0e79740 1 -- 172.21.15.29:0/1254954809 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55618f89b890 con 0x55618fc14d40 2024-03-11T19:11:06.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.882+0000 7fe3b7fff640 1 --2- 172.21.15.29:0/1254954809 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55618fc14d40 0x55618fc17160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:06.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.883+0000 7fe3b7fff640 1 --2- 172.21.15.29:0/1254954809 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55618fc14d40 0x55618fc17160 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=329925ef7eaf3cee server_cookie=e52b38c9dc9143f3 in_seq=0 out_seq=0 2024-03-11T19:11:06.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.883+0000 7fe3977fe640 10 client.15738 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:11:06.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.883+0000 7fe3977fe640 1 -- 172.21.15.29:0/1254954809 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55618f89b890 con 0x55618fc14d40 2024-03-11T19:11:06.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.883+0000 7fe3977fe640 10 client.15738 handle_command_reply: tid=0 2024-03-11T19:11:06.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.883+0000 7fe3c0e79740 1 client.15738 shutdown 2024-03-11T19:11:06.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.883+0000 7fe3c0e79740 1 -- 172.21.15.29:0/1254954809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 msgr2=0x55618fbb87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.883+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/1254954809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 0x55618fbb87d0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac022040 tx=0x7fe3ac0038a0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.883+0000 7fe3c0e79740 1 -- 172.21.15.29:0/1254954809 shutdown_connections 2024-03-11T19:11:06.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.883+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/1254954809 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55618fc14d40 0x55618fc17160 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.883+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/1254954809 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55618fbacdd0 0x55618fbb68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.883+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/1254954809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55618fba7c60 0x55618fbb87d0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.883+0000 7fe3c0e79740 1 --2- 172.21.15.29:0/1254954809 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55618fba7270 0x55618fbb8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.883+0000 7fe3c0e79740 1 -- 172.21.15.29:0/1254954809 >> 172.21.15.29:0/1254954809 conn(0x55618fba27e0 msgr2=0x55618fbb10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.883+0000 7fe3977fe640 0 client.15738 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:11:06.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.883+0000 7fe3c0e79740 1 -- 172.21.15.29:0/1254954809 shutdown_connections 2024-03-11T19:11:06.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.884+0000 7fe3c0e79740 1 -- 172.21.15.29:0/1254954809 wait complete. 2024-03-11T19:11:06.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.884+0000 7fe3c0e79740 20 client.15738 trim_cache size 0 max 16384 2024-03-11T19:11:06.891 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:11:06.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.884+0000 7fe3be29d640 1 -- 172.21.15.29:0/3907456989 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3a00a1ac0 msgr2=0x7fe3a00a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.884+0000 7fe3be29d640 1 --2- 172.21.15.29:0/3907456989 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3a00a1ac0 0x7fe3a00a3f80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe3a802e450 tx=0x7fe3a8004920 comp rx=0 tx=0).stop 2024-03-11T19:11:06.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.884+0000 7fe3be29d640 1 -- 172.21.15.29:0/3907456989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3b8101c50 msgr2=0x7fe3b819c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:06.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.884+0000 7fe3be29d640 1 --2- 172.21.15.29:0/3907456989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3b8101c50 0x7fe3b819c580 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000eb10 tx=0x7fe3b000c2d0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.884+0000 7fe3be29d640 1 -- 172.21.15.29:0/3907456989 shutdown_connections 2024-03-11T19:11:06.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.885+0000 7fe3be29d640 1 --2- 172.21.15.29:0/3907456989 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3a00a1ac0 0x7fe3a00a3f80 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.885+0000 7fe3be29d640 1 --2- 172.21.15.29:0/3907456989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3b8101c50 0x7fe3b819c580 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.885+0000 7fe3be29d640 1 --2- 172.21.15.29:0/3907456989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3b8101290 0x7fe3b8195560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.885+0000 7fe3be29d640 1 --2- 172.21.15.29:0/3907456989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3b81000a0 0x7fe3b8195020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:06.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.885+0000 7fe3be29d640 1 -- 172.21.15.29:0/3907456989 >> 172.21.15.29:0/3907456989 conn(0x7fe3b80fb610 msgr2=0x7fe3b8104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:06.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.885+0000 7fe3be29d640 1 -- 172.21.15.29:0/3907456989 shutdown_connections 2024-03-11T19:11:06.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:06.885+0000 7fe3be29d640 1 -- 172.21.15.29:0/3907456989 wait complete. 2024-03-11T19:11:06.901 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:06 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1504197947' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:11:06.902 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:11:06.902 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:11:07.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:06 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1504197947' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:11:07.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.041+0000 7efdd7c9d640 1 -- 172.21.15.29:0/3467982472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdd0102d60 msgr2=0x7efdd01095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:07.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.041+0000 7efdd7c9d640 1 --2- 172.21.15.29:0/3467982472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdd0102d60 0x7efdd01095f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7efdc8009ca0 tx=0x7efdc802c0f0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.041+0000 7efdd7c9d640 1 -- 172.21.15.29:0/3467982472 shutdown_connections 2024-03-11T19:11:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.041+0000 7efdd7c9d640 1 --2- 172.21.15.29:0/3467982472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdd0102d60 0x7efdd01095f0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.041+0000 7efdd7c9d640 1 --2- 172.21.15.29:0/3467982472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdd01023a0 0x7efdd0102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.041+0000 7efdd7c9d640 1 --2- 172.21.15.29:0/3467982472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdd01011b0 0x7efdd01015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.041+0000 7efdd7c9d640 1 -- 172.21.15.29:0/3467982472 >> 172.21.15.29:0/3467982472 conn(0x7efdd00fc720 msgr2=0x7efdd00feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.042+0000 7efdd7c9d640 1 -- 172.21.15.29:0/3467982472 shutdown_connections 2024-03-11T19:11:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.042+0000 7efdd7c9d640 1 -- 172.21.15.29:0/3467982472 wait complete. 2024-03-11T19:11:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.042+0000 7efdd7c9d640 1 Processor -- start 2024-03-11T19:11:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.042+0000 7efdd7c9d640 1 -- start start 2024-03-11T19:11:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.042+0000 7efdd7c9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdd01011b0 0x7efdd0197110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.042+0000 7efdd7c9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdd01023a0 0x7efdd0197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.042+0000 7efdd7c9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdd0102d60 0x7efdd019e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.042+0000 7efdd7c9d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd019ec00 con 0x7efdd01011b0 2024-03-11T19:11:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.042+0000 7efdd7c9d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd019ed70 con 0x7efdd01023a0 2024-03-11T19:11:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.042+0000 7efdd7c9d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd019ef30 con 0x7efdd0102d60 2024-03-11T19:11:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.042+0000 7efdd5a39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdd01011b0 0x7efdd0197110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.042+0000 7efdd5a39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdd01011b0 0x7efdd0197110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41618/0 (socket says 172.21.15.29:41618) 2024-03-11T19:11:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.043+0000 7efdd5a39640 1 -- 172.21.15.29:0/3936446075 learned_addr learned my addr 172.21.15.29:0/3936446075 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.043+0000 7efdd623a640 1 --2- 172.21.15.29:0/3936446075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdd0102d60 0x7efdd019e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.043+0000 7efdd5238640 1 --2- 172.21.15.29:0/3936446075 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdd01023a0 0x7efdd0197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.043+0000 7efdd5a39640 1 -- 172.21.15.29:0/3936446075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdd0102d60 msgr2=0x7efdd019e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.043+0000 7efdd5a39640 1 --2- 172.21.15.29:0/3936446075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdd0102d60 0x7efdd019e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.043+0000 7efdd5a39640 1 -- 172.21.15.29:0/3936446075 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdd01023a0 msgr2=0x7efdd0197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.043+0000 7efdd5a39640 1 --2- 172.21.15.29:0/3936446075 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdd01023a0 0x7efdd0197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.043+0000 7efdd5a39640 1 -- 172.21.15.29:0/3936446075 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc8009900 con 0x7efdd01011b0 2024-03-11T19:11:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.043+0000 7efdd5238640 1 --2- 172.21.15.29:0/3936446075 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdd01023a0 0x7efdd0197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.043+0000 7efdd5a39640 1 --2- 172.21.15.29:0/3936446075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdd01011b0 0x7efdd0197110 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7efdcc00e9c0 tx=0x7efdcc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.043+0000 7efdc6ffd640 1 -- 172.21.15.29:0/3936446075 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdcc0096e0 con 0x7efdd01011b0 2024-03-11T19:11:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.043+0000 7efdc6ffd640 1 -- 172.21.15.29:0/3936446075 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efdcc0044e0 con 0x7efdd01011b0 2024-03-11T19:11:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.043+0000 7efdd7c9d640 1 -- 172.21.15.29:0/3936446075 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd0073410 con 0x7efdd01011b0 2024-03-11T19:11:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.043+0000 7efdc6ffd640 1 -- 172.21.15.29:0/3936446075 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdcc010430 con 0x7efdd01011b0 2024-03-11T19:11:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.043+0000 7efdd7c9d640 1 -- 172.21.15.29:0/3936446075 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd00738d0 con 0x7efdd01011b0 2024-03-11T19:11:07.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.047+0000 7efdc6ffd640 1 -- 172.21.15.29:0/3936446075 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7efdcc003f80 con 0x7efdd01011b0 2024-03-11T19:11:07.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.047+0000 7efdda875740 1 Processor -- start 2024-03-11T19:11:07.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.047+0000 7efdda875740 1 -- start start 2024-03-11T19:11:07.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.047+0000 7efdda875740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328e270 0x56174328e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:07.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.047+0000 7efdda875740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56174328ec60 0x561743293760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:07.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.047+0000 7efdda875740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561743293dd0 0x561743296260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:07.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.048+0000 7efdda875740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56174324a960 con 0x56174328e270 2024-03-11T19:11:07.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.048+0000 7efdda875740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617431fe310 con 0x561743293dd0 2024-03-11T19:11:07.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.048+0000 7efdda875740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617431fb840 con 0x56174328ec60 2024-03-11T19:11:07.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.048+0000 7efdd623a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328e270 0x56174328e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:07.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.048+0000 7efdd5238640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561743293dd0 0x561743296260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:07.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.048+0000 7efdd623a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328e270 0x56174328e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41634/0 (socket says 172.21.15.29:41634) 2024-03-11T19:11:07.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.048+0000 7efdc6ffd640 1 --2- 172.21.15.29:0/3936446075 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7efdbc0a1ac0 0x7efdbc0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:07.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.048+0000 7efdd623a640 1 -- 172.21.15.29:0/3906481471 learned_addr learned my addr 172.21.15.29:0/3906481471 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.048+0000 7efdd623a640 1 --2- 172.21.15.29:0/3936446075 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7efdbc0a1ac0 0x7efdbc0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.048+0000 7efdc6ffd640 1 -- 172.21.15.29:0/3936446075 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7efdcc014070 con 0x7efdd01011b0 2024-03-11T19:11:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.048+0000 7efdd623a640 1 -- 172.21.15.29:0/3906481471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56174328ec60 msgr2=0x561743293760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.048+0000 7efdd5a39640 1 --2- 172.21.15.29:0/3906481471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56174328ec60 0x561743293760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.048+0000 7efdd623a640 1 --2- 172.21.15.29:0/3906481471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56174328ec60 0x561743293760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.048+0000 7efdd623a640 1 -- 172.21.15.29:0/3906481471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561743293dd0 msgr2=0x561743296260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.048+0000 7efdd623a640 1 --2- 172.21.15.29:0/3906481471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561743293dd0 0x561743296260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.048+0000 7efdd623a640 1 -- 172.21.15.29:0/3906481471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617431b9060 con 0x56174328e270 2024-03-11T19:11:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.048+0000 7efdd5a39640 1 --2- 172.21.15.29:0/3906481471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56174328ec60 0x561743293760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:11:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.049+0000 7efdd623a640 1 --2- 172.21.15.29:0/3906481471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328e270 0x56174328e690 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7efdc80098d0 tx=0x7efdc80344c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d34e4dfa284ead server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.049+0000 7efdd623a640 1 --2- 172.21.15.29:0/3936446075 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7efdbc0a1ac0 0x7efdbc0a3f80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7efdc8005e30 tx=0x7efdc8049000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.049+0000 7efdbbfff640 1 -- 172.21.15.29:0/3906481471 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc8032990 con 0x56174328e270 2024-03-11T19:11:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.049+0000 7efdbbfff640 1 -- 172.21.15.29:0/3906481471 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efdc803d3f0 con 0x56174328e270 2024-03-11T19:11:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.049+0000 7efdbbfff640 1 -- 172.21.15.29:0/3906481471 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc8040630 con 0x56174328e270 2024-03-11T19:11:07.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.050+0000 7efdda875740 1 -- 172.21.15.29:0/3906481471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328e270 msgr2=0x56174328e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:07.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.050+0000 7efdda875740 1 --2- 172.21.15.29:0/3906481471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328e270 0x56174328e690 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7efdc80098d0 tx=0x7efdc80344c0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.050+0000 7efdda875740 1 -- 172.21.15.29:0/3906481471 shutdown_connections 2024-03-11T19:11:07.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.050+0000 7efdda875740 1 --2- 172.21.15.29:0/3906481471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561743293dd0 0x561743296260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.050+0000 7efdda875740 1 --2- 172.21.15.29:0/3906481471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56174328ec60 0x561743293760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.050+0000 7efdda875740 1 --2- 172.21.15.29:0/3906481471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328e270 0x56174328e690 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.050+0000 7efdda875740 1 -- 172.21.15.29:0/3906481471 >> 172.21.15.29:0/3906481471 conn(0x5617432897e0 msgr2=0x561743289be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:07.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.050+0000 7efdda875740 1 -- 172.21.15.29:0/3906481471 shutdown_connections 2024-03-11T19:11:07.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.050+0000 7efdda875740 1 -- 172.21.15.29:0/3906481471 wait complete. 2024-03-11T19:11:07.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.050+0000 7efdda875740 1 -- start start 2024-03-11T19:11:07.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.050+0000 7efdda875740 1 Processor -- start 2024-03-11T19:11:07.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdda875740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56174328e270 0x561743298480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:07.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdda875740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328ec60 0x5617432989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:07.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdda875740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561743293dd0 0x56174329fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:07.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdda875740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617431f7020 con 0x56174328ec60 2024-03-11T19:11:07.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdda875740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x561742eca430 con 0x56174328e270 2024-03-11T19:11:07.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdda875740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561742e86e00 con 0x561743293dd0 2024-03-11T19:11:07.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdd5238640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328ec60 0x5617432989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:07.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdda875740 10 client.0 _ll_register_callbacks cb 0x56174319ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:07.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdd5238640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328ec60 0x5617432989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41644/0 (socket says 172.21.15.29:41644) 2024-03-11T19:11:07.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdd5238640 1 -- 172.21.15.29:0/594995538 learned_addr learned my addr 172.21.15.29:0/594995538 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:07.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdd5a39640 1 --2- 172.21.15.29:0/594995538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56174328e270 0x561743298480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:07.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdd623a640 1 --2- 172.21.15.29:0/594995538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561743293dd0 0x56174329fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:07.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdd5238640 1 -- 172.21.15.29:0/594995538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561743293dd0 msgr2=0x56174329fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:07.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdd5238640 1 --2- 172.21.15.29:0/594995538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561743293dd0 0x56174329fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdd5238640 1 -- 172.21.15.29:0/594995538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56174328e270 msgr2=0x561743298480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:07.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdd5238640 1 --2- 172.21.15.29:0/594995538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56174328e270 0x561743298480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdd5238640 1 -- 172.21.15.29:0/594995538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd0109ef0 con 0x56174328ec60 2024-03-11T19:11:07.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdd5a39640 1 --2- 172.21.15.29:0/594995538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56174328e270 0x561743298480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:07.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdd5238640 1 --2- 172.21.15.29:0/594995538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328ec60 0x5617432989c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7efdc0012770 tx=0x7efdc0012c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d51315ac5bb9e18 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:07.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.051+0000 7efdb97fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:07.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.052+0000 7efdb97fa640 1 -- 172.21.15.29:0/594995538 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0017070 con 0x56174328ec60 2024-03-11T19:11:07.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.052+0000 7efdda875740 1 -- 172.21.15.29:0/594995538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561742ef67e0 con 0x56174328ec60 2024-03-11T19:11:07.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.052+0000 7efdb97fa640 1 -- 172.21.15.29:0/594995538 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7efdc0004240 con 0x56174328ec60 2024-03-11T19:11:07.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.052+0000 7efdb97fa640 1 -- 172.21.15.29:0/594995538 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdc0004400 con 0x56174328ec60 2024-03-11T19:11:07.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.052+0000 7efdda875740 10 client.15750 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:07.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.052+0000 7efdda875740 1 -- 172.21.15.29:0/594995538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5617431f56a0 con 0x56174328ec60 2024-03-11T19:11:07.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.052+0000 7efdb97fa640 1 -- 172.21.15.29:0/594995538 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x561742ef67e0 con 0x56174328ec60 2024-03-11T19:11:07.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.052+0000 7efdb97fa640 1 -- 172.21.15.29:0/594995538 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efdc0003c80 con 0x56174328ec60 2024-03-11T19:11:07.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.052+0000 7efdda875740 10 client.15750 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:07.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.052+0000 7efdda875740 10 client.15750 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:11:07.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.052+0000 7efdda875740 20 client.15750 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:07.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.052+0000 7efdda875740 1 --2- 172.21.15.29:0/594995538 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5617432a46f0 0x5617432a6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:07.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.052+0000 7efdda875740 4 client.15750 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:11:07.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.052+0000 7efdda875740 1 -- 172.21.15.29:0/594995538 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561742f33720 con 0x5617432a46f0 2024-03-11T19:11:07.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.053+0000 7efdd5a39640 1 --2- 172.21.15.29:0/594995538 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5617432a46f0 0x5617432a6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:07.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.053+0000 7efdb97fa640 10 client.15750 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:07.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.053+0000 7efdd5a39640 1 --2- 172.21.15.29:0/594995538 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5617432a46f0 0x5617432a6b30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19ea55bce79cdbde server_cookie=ab5d335c3a3ba98f in_seq=0 out_seq=0 2024-03-11T19:11:07.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.054+0000 7efdb97fa640 1 -- 172.21.15.29:0/594995538 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561742f33720 con 0x5617432a46f0 2024-03-11T19:11:07.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.054+0000 7efdb97fa640 10 client.15750 handle_command_reply: tid=0 2024-03-11T19:11:07.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.054+0000 7efdda875740 1 client.15750 shutdown 2024-03-11T19:11:07.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.054+0000 7efdda875740 1 -- 172.21.15.29:0/594995538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328ec60 msgr2=0x5617432989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:07.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.054+0000 7efdda875740 1 --2- 172.21.15.29:0/594995538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328ec60 0x5617432989c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7efdc0012770 tx=0x7efdc0012c30 comp rx=0 tx=0).stop 2024-03-11T19:11:07.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.054+0000 7efdda875740 1 -- 172.21.15.29:0/594995538 shutdown_connections 2024-03-11T19:11:07.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.054+0000 7efdda875740 1 --2- 172.21.15.29:0/594995538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561743293dd0 0x56174329fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.054+0000 7efdda875740 1 --2- 172.21.15.29:0/594995538 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5617432a46f0 0x5617432a6b30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.054+0000 7efdda875740 1 --2- 172.21.15.29:0/594995538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328ec60 0x5617432989c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.054+0000 7efdda875740 1 --2- 172.21.15.29:0/594995538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56174328e270 0x561743298480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.054+0000 7efdb97fa640 0 client.15750 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:07.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.054+0000 7efdda875740 1 -- 172.21.15.29:0/594995538 >> 172.21.15.29:0/594995538 conn(0x5617432897e0 msgr2=0x561743288b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:07.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.054+0000 7efdda875740 1 -- 172.21.15.29:0/594995538 shutdown_connections 2024-03-11T19:11:07.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.054+0000 7efdda875740 1 -- 172.21.15.29:0/594995538 wait complete. 2024-03-11T19:11:07.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.054+0000 7efdda875740 20 client.15750 trim_cache size 0 max 16384 2024-03-11T19:11:07.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.066+0000 7efdda875740 1 Processor -- start 2024-03-11T19:11:07.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.066+0000 7efdda875740 1 -- start start 2024-03-11T19:11:07.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.066+0000 7efdda875740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56174328e270 0x561743289bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:07.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.066+0000 7efdda875740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56174328ec60 0x56174329f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:07.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.067+0000 7efdda875740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561743293dd0 0x561743298460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:07.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.067+0000 7efdda875740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56174317c130 con 0x561743293dd0 2024-03-11T19:11:07.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.067+0000 7efdda875740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x561742f33720 con 0x56174328e270 2024-03-11T19:11:07.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.067+0000 7efdda875740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561742ef67e0 con 0x56174328ec60 2024-03-11T19:11:07.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.067+0000 7efdd5a39640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56174328e270 0x561743289bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:07.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.067+0000 7efdd623a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561743293dd0 0x561743298460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:07.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.067+0000 7efdd623a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561743293dd0 0x561743298460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41660/0 (socket says 172.21.15.29:41660) 2024-03-11T19:11:07.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.067+0000 7efdd623a640 1 -- 172.21.15.29:0/1557258469 learned_addr learned my addr 172.21.15.29:0/1557258469 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:07.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.067+0000 7efdd5a39640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56174328e270 0x561743289bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:52682/0 (socket says 172.21.15.29:52682) 2024-03-11T19:11:07.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.067+0000 7efdd5238640 1 --2- 172.21.15.29:0/1557258469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56174328ec60 0x56174329f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:07.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.067+0000 7efdd5a39640 1 -- 172.21.15.29:0/1557258469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56174328ec60 msgr2=0x56174329f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:07.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.067+0000 7efdd5a39640 1 --2- 172.21.15.29:0/1557258469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56174328ec60 0x56174329f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.067+0000 7efdd5a39640 1 -- 172.21.15.29:0/1557258469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561743293dd0 msgr2=0x561743298460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:07.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.067+0000 7efdd5a39640 1 --2- 172.21.15.29:0/1557258469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561743293dd0 0x561743298460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.067+0000 7efdd5a39640 1 -- 172.21.15.29:0/1557258469 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561743191c20 con 0x56174328e270 2024-03-11T19:11:07.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.067+0000 7efdd623a640 1 --2- 172.21.15.29:0/1557258469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561743293dd0 0x561743298460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:07.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.067+0000 7efdd5a39640 1 --2- 172.21.15.29:0/1557258469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56174328e270 0x561743289bc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efdcc0cb370 tx=0x7efdcc085040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0f1d0cc23c225c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:07.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.068+0000 7efdbb7fe640 1 -- 172.21.15.29:0/1557258469 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdcc09a7d0 con 0x56174328e270 2024-03-11T19:11:07.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.068+0000 7efdbb7fe640 1 -- 172.21.15.29:0/1557258469 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efdcc09e920 con 0x56174328e270 2024-03-11T19:11:07.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.068+0000 7efdbb7fe640 1 -- 172.21.15.29:0/1557258469 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdcc096e40 con 0x56174328e270 2024-03-11T19:11:07.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.069+0000 7efdda875740 1 -- 172.21.15.29:0/1557258469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56174328e270 msgr2=0x561743289bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:07.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.069+0000 7efdda875740 1 --2- 172.21.15.29:0/1557258469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56174328e270 0x561743289bc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efdcc0cb370 tx=0x7efdcc085040 comp rx=0 tx=0).stop 2024-03-11T19:11:07.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.069+0000 7efdda875740 1 -- 172.21.15.29:0/1557258469 shutdown_connections 2024-03-11T19:11:07.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.069+0000 7efdda875740 1 --2- 172.21.15.29:0/1557258469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561743293dd0 0x561743298460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.069+0000 7efdda875740 1 --2- 172.21.15.29:0/1557258469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56174328ec60 0x56174329f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.069+0000 7efdda875740 1 --2- 172.21.15.29:0/1557258469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56174328e270 0x561743289bc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.069+0000 7efdda875740 1 -- 172.21.15.29:0/1557258469 >> 172.21.15.29:0/1557258469 conn(0x5617432897e0 msgr2=0x561743296f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:07.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.069+0000 7efdda875740 1 -- 172.21.15.29:0/1557258469 shutdown_connections 2024-03-11T19:11:07.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.069+0000 7efdda875740 1 -- 172.21.15.29:0/1557258469 wait complete. 2024-03-11T19:11:07.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.069+0000 7efdda875740 1 -- start start 2024-03-11T19:11:07.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.069+0000 7efdda875740 1 Processor -- start 2024-03-11T19:11:07.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdda875740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328e270 0x56174329f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:07.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdda875740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56174328ec60 0x56174329dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:07.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdda875740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561743293dd0 0x56174329e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:07.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdda875740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56174329ec80 con 0x56174328e270 2024-03-11T19:11:07.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdda875740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56174329edf0 con 0x561743293dd0 2024-03-11T19:11:07.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdda875740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617432fac40 con 0x56174328ec60 2024-03-11T19:11:07.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdd5a39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328e270 0x56174329f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:07.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdd5a39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328e270 0x56174329f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41666/0 (socket says 172.21.15.29:41666) 2024-03-11T19:11:07.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdd5a39640 1 -- 172.21.15.29:0/3551156152 learned_addr learned my addr 172.21.15.29:0/3551156152 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:07.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdd5238640 1 --2- 172.21.15.29:0/3551156152 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56174328ec60 0x56174329dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdd623a640 1 --2- 172.21.15.29:0/3551156152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561743293dd0 0x56174329e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdda875740 10 client.0 _ll_register_callbacks cb 0x7efdc0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdd5a39640 1 -- 172.21.15.29:0/3551156152 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56174328ec60 msgr2=0x56174329dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdd5a39640 1 --2- 172.21.15.29:0/3551156152 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56174328ec60 0x56174329dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdd5a39640 1 -- 172.21.15.29:0/3551156152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561743293dd0 msgr2=0x56174329e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdd5a39640 1 --2- 172.21.15.29:0/3551156152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561743293dd0 0x56174329e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdd5a39640 1 -- 172.21.15.29:0/3551156152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc8050070 con 0x56174328e270 2024-03-11T19:11:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdd623a640 1 --2- 172.21.15.29:0/3551156152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561743293dd0 0x56174329e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdda875740 1 -- 172.21.15.29:0/3551156152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efdc803d3f0 con 0x56174328e270 2024-03-11T19:11:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdd5a39640 1 --2- 172.21.15.29:0/3551156152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328e270 0x56174329f940 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7efdcc09a4d0 tx=0x7efdcc0038c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=502f9fd70c9091ff server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.070+0000 7efdb97fa640 10 client.15756 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.071+0000 7efdb97fa640 1 -- 172.21.15.29:0/3551156152 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdcc09a7d0 con 0x56174328e270 2024-03-11T19:11:07.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.071+0000 7efdb97fa640 1 -- 172.21.15.29:0/3551156152 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7efdcc0a2f70 con 0x56174328e270 2024-03-11T19:11:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.071+0000 7efdb97fa640 1 -- 172.21.15.29:0/3551156152 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdcc004230 con 0x56174328e270 2024-03-11T19:11:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.071+0000 7efdda875740 10 client.15756 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.071+0000 7efdda875740 1 -- 172.21.15.29:0/3551156152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x7efdd00738d0 con 0x56174328e270 2024-03-11T19:11:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.071+0000 7efdb97fa640 1 -- 172.21.15.29:0/3551156152 <== mon.0 v2:172.21.15.29:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efdcc01f060 con 0x56174328e270 2024-03-11T19:11:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.071+0000 7efdda875740 10 client.15756 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.071+0000 7efdda875740 10 client.15756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:11:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.071+0000 7efdda875740 20 client.15756 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.071+0000 7efdda875740 1 --2- 172.21.15.29:0/3551156152 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5617432fc100 0x5617432fe4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.071+0000 7efdda875740 4 client.15756 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:11:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.071+0000 7efdda875740 1 -- 172.21.15.29:0/3551156152 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561742f82890 con 0x5617432fc100 2024-03-11T19:11:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.071+0000 7efdd5238640 1 --2- 172.21.15.29:0/3551156152 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5617432fc100 0x5617432fe4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.072+0000 7efdb97fa640 10 client.15756 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.072+0000 7efdd5238640 1 --2- 172.21.15.29:0/3551156152 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5617432fc100 0x5617432fe4f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbf96e241979a41f server_cookie=95b9c961d9c3c174 in_seq=0 out_seq=0 2024-03-11T19:11:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.072+0000 7efdb97fa640 1 -- 172.21.15.29:0/3551156152 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561742f82890 con 0x5617432fc100 2024-03-11T19:11:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.072+0000 7efdb97fa640 10 client.15756 handle_command_reply: tid=0 2024-03-11T19:11:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.072+0000 7efdda875740 1 client.15756 shutdown 2024-03-11T19:11:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.072+0000 7efdda875740 1 -- 172.21.15.29:0/3551156152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328e270 msgr2=0x56174329f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.072+0000 7efdda875740 1 --2- 172.21.15.29:0/3551156152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328e270 0x56174329f940 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7efdcc09a4d0 tx=0x7efdcc0038c0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.072+0000 7efdda875740 1 -- 172.21.15.29:0/3551156152 shutdown_connections 2024-03-11T19:11:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.072+0000 7efdda875740 1 --2- 172.21.15.29:0/3551156152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561743293dd0 0x56174329e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.072+0000 7efdda875740 1 --2- 172.21.15.29:0/3551156152 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56174328ec60 0x56174329dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.072+0000 7efdda875740 1 --2- 172.21.15.29:0/3551156152 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5617432fc100 0x5617432fe4f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.072+0000 7efdda875740 1 --2- 172.21.15.29:0/3551156152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56174328e270 0x56174329f940 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.072+0000 7efdda875740 1 -- 172.21.15.29:0/3551156152 >> 172.21.15.29:0/3551156152 conn(0x5617432897e0 msgr2=0x561743289bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.072+0000 7efdb97fa640 0 client.15756 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.072+0000 7efdda875740 1 -- 172.21.15.29:0/3551156152 shutdown_connections 2024-03-11T19:11:07.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.073+0000 7efdda875740 1 -- 172.21.15.29:0/3551156152 wait complete. 2024-03-11T19:11:07.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.073+0000 7efdda875740 20 client.15756 trim_cache size 0 max 16384 2024-03-11T19:11:07.081 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:11:07.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.073+0000 7efdd7c9d640 1 -- 172.21.15.29:0/3936446075 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7efdbc0a1ac0 msgr2=0x7efdbc0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.073+0000 7efdd7c9d640 1 --2- 172.21.15.29:0/3936446075 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7efdbc0a1ac0 0x7efdbc0a3f80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7efdc8005e30 tx=0x7efdc8049000 comp rx=0 tx=0).stop 2024-03-11T19:11:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.073+0000 7efdd7c9d640 1 -- 172.21.15.29:0/3936446075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdd01011b0 msgr2=0x7efdd0197110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.073+0000 7efdd7c9d640 1 --2- 172.21.15.29:0/3936446075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdd01011b0 0x7efdd0197110 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7efdcc00e9c0 tx=0x7efdcc00ee80 comp rx=0 tx=0).stop 2024-03-11T19:11:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.073+0000 7efdd7c9d640 1 -- 172.21.15.29:0/3936446075 shutdown_connections 2024-03-11T19:11:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.073+0000 7efdd7c9d640 1 --2- 172.21.15.29:0/3936446075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdd0102d60 0x7efdd019e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.073+0000 7efdd7c9d640 1 --2- 172.21.15.29:0/3936446075 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdd01023a0 0x7efdd0197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.073+0000 7efdd7c9d640 1 --2- 172.21.15.29:0/3936446075 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7efdbc0a1ac0 0x7efdbc0a3f80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.073+0000 7efdd7c9d640 1 --2- 172.21.15.29:0/3936446075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdd01011b0 0x7efdd0197110 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.073+0000 7efdd7c9d640 1 -- 172.21.15.29:0/3936446075 >> 172.21.15.29:0/3936446075 conn(0x7efdd00fc720 msgr2=0x7efdd01056f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.073+0000 7efdd7c9d640 1 -- 172.21.15.29:0/3936446075 shutdown_connections 2024-03-11T19:11:07.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:07.074+0000 7efdd7c9d640 1 -- 172.21.15.29:0/3936446075 wait complete. 2024-03-11T19:11:07.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:06 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1504197947' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:11:07.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:07 smithi029 ceph-mon[32439]: pgmap v390: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:11:08.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:07 smithi190 ceph-mon[36292]: pgmap v390: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:11:08.092 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:11:08.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:11:08.092 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:11:08.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:07 smithi170 ceph-mon[35690]: pgmap v390: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:11:08.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.200+0000 7fdb19977640 1 -- 172.21.15.29:0/3104296820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb14101270 msgr2=0x7fdb141016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:08.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.200+0000 7fdb19977640 1 --2- 172.21.15.29:0/3104296820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb14101270 0x7fdb141016f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fdb08009900 tx=0x7fdb0802bc30 comp rx=0 tx=0).stop 2024-03-11T19:11:08.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.200+0000 7fdb19977640 1 -- 172.21.15.29:0/3104296820 shutdown_connections 2024-03-11T19:11:08.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.200+0000 7fdb19977640 1 --2- 172.21.15.29:0/3104296820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb14101c30 0x7fdb141084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.200+0000 7fdb19977640 1 --2- 172.21.15.29:0/3104296820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb14101270 0x7fdb141016f0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.200+0000 7fdb19977640 1 --2- 172.21.15.29:0/3104296820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb14100080 0x7fdb141004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.200+0000 7fdb19977640 1 -- 172.21.15.29:0/3104296820 >> 172.21.15.29:0/3104296820 conn(0x7fdb140fb610 msgr2=0x7fdb140fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:08.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.200+0000 7fdb19977640 1 -- 172.21.15.29:0/3104296820 shutdown_connections 2024-03-11T19:11:08.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.200+0000 7fdb19977640 1 -- 172.21.15.29:0/3104296820 wait complete. 2024-03-11T19:11:08.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.201+0000 7fdb19977640 1 Processor -- start 2024-03-11T19:11:08.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.201+0000 7fdb19977640 1 -- start start 2024-03-11T19:11:08.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.201+0000 7fdb19977640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb14100080 0x7fdb14197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.201+0000 7fdb19977640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb14101270 0x7fdb141976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.201+0000 7fdb19977640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb14101c30 0x7fdb1419e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.201+0000 7fdb19977640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1419ec60 con 0x7fdb14100080 2024-03-11T19:11:08.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.201+0000 7fdb19977640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1419edd0 con 0x7fdb14101c30 2024-03-11T19:11:08.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.201+0000 7fdb19977640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1419ef90 con 0x7fdb14101270 2024-03-11T19:11:08.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.201+0000 7fdb12ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb14100080 0x7fdb14197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.201+0000 7fdb12ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb14100080 0x7fdb14197170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41672/0 (socket says 172.21.15.29:41672) 2024-03-11T19:11:08.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.201+0000 7fdb12ffd640 1 -- 172.21.15.29:0/1236692481 learned_addr learned my addr 172.21.15.29:0/1236692481 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:08.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.201+0000 7fdb137fe640 1 --2- 172.21.15.29:0/1236692481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb14101c30 0x7fdb1419e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.201+0000 7fdb127fc640 1 --2- 172.21.15.29:0/1236692481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb14101270 0x7fdb141976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.202+0000 7fdb12ffd640 1 -- 172.21.15.29:0/1236692481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb14101270 msgr2=0x7fdb141976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:08.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.202+0000 7fdb12ffd640 1 --2- 172.21.15.29:0/1236692481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb14101270 0x7fdb141976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.202+0000 7fdb12ffd640 1 -- 172.21.15.29:0/1236692481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb14101c30 msgr2=0x7fdb1419e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:08.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.202+0000 7fdb12ffd640 1 --2- 172.21.15.29:0/1236692481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb14101c30 0x7fdb1419e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.202+0000 7fdb12ffd640 1 -- 172.21.15.29:0/1236692481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb08009560 con 0x7fdb14100080 2024-03-11T19:11:08.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.202+0000 7fdb12ffd640 1 --2- 172.21.15.29:0/1236692481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb14100080 0x7fdb14197170 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdb0400c970 tx=0x7fdb0400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:08.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.202+0000 7fdb18975640 1 -- 172.21.15.29:0/1236692481 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb04004220 con 0x7fdb14100080 2024-03-11T19:11:08.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.202+0000 7fdb19977640 1 -- 172.21.15.29:0/1236692481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb1419f270 con 0x7fdb14100080 2024-03-11T19:11:08.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.202+0000 7fdb18975640 1 -- 172.21.15.29:0/1236692481 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdb04007e60 con 0x7fdb14100080 2024-03-11T19:11:08.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.202+0000 7fdb18975640 1 -- 172.21.15.29:0/1236692481 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb04004a00 con 0x7fdb14100080 2024-03-11T19:11:08.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.202+0000 7fdb19977640 1 -- 172.21.15.29:0/1236692481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb1419f780 con 0x7fdb14100080 2024-03-11T19:11:08.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.205+0000 7fdb18975640 1 -- 172.21.15.29:0/1236692481 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fdb040079b0 con 0x7fdb14100080 2024-03-11T19:11:08.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.205+0000 7fdb1c623740 1 Processor -- start 2024-03-11T19:11:08.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.205+0000 7fdb1c623740 1 -- start start 2024-03-11T19:11:08.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb1c623740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761b4c0 0x561b9761b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb1c623740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761c690 0x561b9761cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb1c623740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761d050 0x561b976238e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb1c623740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b9758af90 con 0x561b9761d050 2024-03-11T19:11:08.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb1c623740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b975884c0 con 0x561b9761b4c0 2024-03-11T19:11:08.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb1c623740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b97583ca0 con 0x561b9761c690 2024-03-11T19:11:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb127fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761d050 0x561b976238e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb127fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761d050 0x561b976238e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41688/0 (socket says 172.21.15.29:41688) 2024-03-11T19:11:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb137fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761b4c0 0x561b9761b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb127fc640 1 -- 172.21.15.29:0/1389317125 learned_addr learned my addr 172.21.15.29:0/1389317125 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb12ffd640 1 --2- 172.21.15.29:0/1389317125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761c690 0x561b9761cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb18975640 1 --2- 172.21.15.29:0/1236692481 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdafc0a1ac0 0x7fdafc0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb127fc640 1 -- 172.21.15.29:0/1389317125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761c690 msgr2=0x561b9761cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb127fc640 1 --2- 172.21.15.29:0/1389317125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761c690 0x561b9761cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb127fc640 1 -- 172.21.15.29:0/1389317125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761b4c0 msgr2=0x561b9761b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb127fc640 1 --2- 172.21.15.29:0/1389317125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761b4c0 0x561b9761b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb127fc640 1 -- 172.21.15.29:0/1389317125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b975460b0 con 0x561b9761d050 2024-03-11T19:11:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb137fe640 1 --2- 172.21.15.29:0/1389317125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761b4c0 0x561b9761b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb137fe640 1 --2- 172.21.15.29:0/1236692481 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdafc0a1ac0 0x7fdafc0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb18975640 1 -- 172.21.15.29:0/1236692481 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fdb040cd500 con 0x7fdb14100080 2024-03-11T19:11:08.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.206+0000 7fdb127fc640 1 --2- 172.21.15.29:0/1389317125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761d050 0x561b976238e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fdb08009900 tx=0x7fdb0802c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f0e0c35ca56881 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:08.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.207+0000 7fdaf9ffb640 1 -- 172.21.15.29:0/1389317125 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb080046c0 con 0x561b9761d050 2024-03-11T19:11:08.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.207+0000 7fdaf9ffb640 1 -- 172.21.15.29:0/1389317125 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdb0802de90 con 0x561b9761d050 2024-03-11T19:11:08.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.207+0000 7fdaf9ffb640 1 -- 172.21.15.29:0/1389317125 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb08003b20 con 0x561b9761d050 2024-03-11T19:11:08.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.207+0000 7fdb137fe640 1 --2- 172.21.15.29:0/1236692481 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdafc0a1ac0 0x7fdafc0a3f80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c009fd0 tx=0x7fdb0c004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:08.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.207+0000 7fdb1c623740 1 -- 172.21.15.29:0/1389317125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761d050 msgr2=0x561b976238e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:08.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.207+0000 7fdb1c623740 1 --2- 172.21.15.29:0/1389317125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761d050 0x561b976238e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fdb08009900 tx=0x7fdb0802c8f0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.207+0000 7fdb1c623740 1 -- 172.21.15.29:0/1389317125 shutdown_connections 2024-03-11T19:11:08.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.207+0000 7fdb1c623740 1 --2- 172.21.15.29:0/1389317125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761d050 0x561b976238e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.207+0000 7fdb1c623740 1 --2- 172.21.15.29:0/1389317125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761c690 0x561b9761cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.207+0000 7fdb1c623740 1 --2- 172.21.15.29:0/1389317125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761b4c0 0x561b9761b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.207+0000 7fdb1c623740 1 -- 172.21.15.29:0/1389317125 >> 172.21.15.29:0/1389317125 conn(0x561b976166a0 msgr2=0x561b97616aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:08.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.208+0000 7fdb1c623740 1 -- 172.21.15.29:0/1389317125 shutdown_connections 2024-03-11T19:11:08.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.208+0000 7fdb1c623740 1 -- 172.21.15.29:0/1389317125 wait complete. 2024-03-11T19:11:08.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.208+0000 7fdb1c623740 1 -- start start 2024-03-11T19:11:08.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.208+0000 7fdb1c623740 1 Processor -- start 2024-03-11T19:11:08.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.208+0000 7fdb1c623740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761b4c0 0x561b97625650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.208+0000 7fdb1c623740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761c690 0x561b97625b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.208+0000 7fdb1c623740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761d050 0x561b9762cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.208+0000 7fdb1c623740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b97528f40 con 0x561b9761c690 2024-03-11T19:11:08.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.208+0000 7fdb1c623740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b972574b0 con 0x561b9761b4c0 2024-03-11T19:11:08.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.208+0000 7fdb1c623740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b97509670 con 0x561b9761d050 2024-03-11T19:11:08.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.208+0000 7fdb1c623740 10 client.0 _ll_register_callbacks cb 0x561b97576f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:08.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.208+0000 7fdb127fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761c690 0x561b97625b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.208+0000 7fdb127fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761c690 0x561b97625b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41696/0 (socket says 172.21.15.29:41696) 2024-03-11T19:11:08.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.208+0000 7fdb127fc640 1 -- 172.21.15.29:0/1191947289 learned_addr learned my addr 172.21.15.29:0/1191947289 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:08.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.209+0000 7fdb12ffd640 1 --2- 172.21.15.29:0/1191947289 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761b4c0 0x561b97625650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.209+0000 7fdb137fe640 1 --2- 172.21.15.29:0/1191947289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761d050 0x561b9762cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.209+0000 7fdb127fc640 1 -- 172.21.15.29:0/1191947289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761d050 msgr2=0x561b9762cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:08.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.209+0000 7fdb127fc640 1 --2- 172.21.15.29:0/1191947289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761d050 0x561b9762cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.209+0000 7fdb127fc640 1 -- 172.21.15.29:0/1191947289 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761b4c0 msgr2=0x561b97625650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:08.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.209+0000 7fdb127fc640 1 --2- 172.21.15.29:0/1191947289 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761b4c0 0x561b97625650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.209+0000 7fdb127fc640 1 -- 172.21.15.29:0/1191947289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb14108dc0 con 0x561b9761c690 2024-03-11T19:11:08.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.209+0000 7fdb127fc640 1 --2- 172.21.15.29:0/1191947289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761c690 0x561b97625b90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fdb08036040 tx=0x7fdb0802df30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67d5114b0b72136e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:08.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.209+0000 7fdaf2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:08.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.209+0000 7fdaf2ffd640 1 -- 172.21.15.29:0/1191947289 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb08005310 con 0x561b9761c690 2024-03-11T19:11:08.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.209+0000 7fdaf2ffd640 1 -- 172.21.15.29:0/1191947289 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fdb08005470 con 0x561b9761c690 2024-03-11T19:11:08.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.209+0000 7fdb1c623740 1 -- 172.21.15.29:0/1191947289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561b9755df40 con 0x561b9761c690 2024-03-11T19:11:08.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.209+0000 7fdaf2ffd640 1 -- 172.21.15.29:0/1191947289 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb08005630 con 0x561b9761c690 2024-03-11T19:11:08.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.209+0000 7fdb1c623740 10 client.15768 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:08.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.209+0000 7fdb1c623740 1 -- 172.21.15.29:0/1191947289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x561b97582320 con 0x561b9761c690 2024-03-11T19:11:08.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.209+0000 7fdaf2ffd640 1 -- 172.21.15.29:0/1191947289 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x561b9755df40 con 0x561b9761c690 2024-03-11T19:11:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.210+0000 7fdaf2ffd640 1 -- 172.21.15.29:0/1191947289 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fdb0803c040 con 0x561b9761c690 2024-03-11T19:11:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.210+0000 7fdb1c623740 10 client.15768 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.210+0000 7fdb1c623740 10 client.15768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:11:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.210+0000 7fdb1c623740 20 client.15768 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.210+0000 7fdb1c623740 1 --2- 172.21.15.29:0/1191947289 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x561b97631440 0x561b97633860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.210+0000 7fdb1c623740 4 client.15768 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:11:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.210+0000 7fdb1c623740 1 -- 172.21.15.29:0/1191947289 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561b9757a310 con 0x561b97631440 2024-03-11T19:11:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.210+0000 7fdb12ffd640 1 --2- 172.21.15.29:0/1191947289 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x561b97631440 0x561b97633860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.210+0000 7fdaf2ffd640 10 client.15768 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.210+0000 7fdb12ffd640 1 --2- 172.21.15.29:0/1191947289 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x561b97631440 0x561b97633860 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d561bfe03374c3c3 server_cookie=767c99e082c41e2d in_seq=0 out_seq=0 2024-03-11T19:11:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.211+0000 7fdaf2ffd640 1 -- 172.21.15.29:0/1191947289 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561b9757a310 con 0x561b97631440 2024-03-11T19:11:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.211+0000 7fdaf2ffd640 10 client.15768 handle_command_reply: tid=0 2024-03-11T19:11:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.211+0000 7fdb1c623740 1 client.15768 shutdown 2024-03-11T19:11:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.211+0000 7fdb1c623740 1 -- 172.21.15.29:0/1191947289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761c690 msgr2=0x561b97625b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:08.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.211+0000 7fdb1c623740 1 --2- 172.21.15.29:0/1191947289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761c690 0x561b97625b90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fdb08036040 tx=0x7fdb0802df30 comp rx=0 tx=0).stop 2024-03-11T19:11:08.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.211+0000 7fdb1c623740 1 -- 172.21.15.29:0/1191947289 shutdown_connections 2024-03-11T19:11:08.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.211+0000 7fdb1c623740 1 --2- 172.21.15.29:0/1191947289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761d050 0x561b9762cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.211+0000 7fdb1c623740 1 --2- 172.21.15.29:0/1191947289 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x561b97631440 0x561b97633860 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.211+0000 7fdb1c623740 1 --2- 172.21.15.29:0/1191947289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761c690 0x561b97625b90 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.211+0000 7fdb1c623740 1 --2- 172.21.15.29:0/1191947289 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761b4c0 0x561b97625650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.211+0000 7fdb1c623740 1 -- 172.21.15.29:0/1191947289 >> 172.21.15.29:0/1191947289 conn(0x561b976166a0 msgr2=0x561b976158e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:08.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.211+0000 7fdaf2ffd640 0 client.15768 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:08.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.211+0000 7fdb1c623740 1 -- 172.21.15.29:0/1191947289 shutdown_connections 2024-03-11T19:11:08.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.211+0000 7fdb1c623740 1 -- 172.21.15.29:0/1191947289 wait complete. 2024-03-11T19:11:08.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.212+0000 7fdb1c623740 20 client.15768 trim_cache size 0 max 16384 2024-03-11T19:11:08.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.227+0000 7fdb1c623740 1 Processor -- start 2024-03-11T19:11:08.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.227+0000 7fdb1c623740 1 -- start start 2024-03-11T19:11:08.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.227+0000 7fdb1c623740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761b4c0 0x561b976201c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.227+0000 7fdb1c623740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761c690 0x561b97620700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.227+0000 7fdb1c623740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761d050 0x561b9762cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.227+0000 7fdb1c623740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b976247c0 con 0x561b9761d050 2024-03-11T19:11:08.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.227+0000 7fdb1c623740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b97625490 con 0x561b9761b4c0 2024-03-11T19:11:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.227+0000 7fdb1c623740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b9755df40 con 0x561b9761c690 2024-03-11T19:11:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.227+0000 7fdb137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761d050 0x561b9762cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.227+0000 7fdb137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761d050 0x561b9762cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41712/0 (socket says 172.21.15.29:41712) 2024-03-11T19:11:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.227+0000 7fdb137fe640 1 -- 172.21.15.29:0/3936618301 learned_addr learned my addr 172.21.15.29:0/3936618301 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.228+0000 7fdb12ffd640 1 --2- 172.21.15.29:0/3936618301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761b4c0 0x561b976201c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.228+0000 7fdb127fc640 1 --2- 172.21.15.29:0/3936618301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761c690 0x561b97620700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.228+0000 7fdb137fe640 1 -- 172.21.15.29:0/3936618301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761c690 msgr2=0x561b97620700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.228+0000 7fdb137fe640 1 --2- 172.21.15.29:0/3936618301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761c690 0x561b97620700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.228+0000 7fdb137fe640 1 -- 172.21.15.29:0/3936618301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761b4c0 msgr2=0x561b976201c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.228+0000 7fdb137fe640 1 --2- 172.21.15.29:0/3936618301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761b4c0 0x561b976201c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.228+0000 7fdb137fe640 1 -- 172.21.15.29:0/3936618301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b9751ec00 con 0x561b9761d050 2024-03-11T19:11:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.228+0000 7fdb12ffd640 1 --2- 172.21.15.29:0/3936618301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761b4c0 0x561b976201c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.228+0000 7fdb137fe640 1 --2- 172.21.15.29:0/3936618301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761d050 0x561b9762cb90 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c0133c0 tx=0x7fdb0c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c616a48458be75d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.228+0000 7fdaf97fa640 1 -- 172.21.15.29:0/3936618301 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c023070 con 0x561b9761d050 2024-03-11T19:11:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.228+0000 7fdaf97fa640 1 -- 172.21.15.29:0/3936618301 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdb0c003e20 con 0x561b9761d050 2024-03-11T19:11:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.228+0000 7fdaf97fa640 1 -- 172.21.15.29:0/3936618301 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c004490 con 0x561b9761d050 2024-03-11T19:11:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.229+0000 7fdb1c623740 1 -- 172.21.15.29:0/3936618301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761d050 msgr2=0x561b9762cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.229+0000 7fdb1c623740 1 --2- 172.21.15.29:0/3936618301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761d050 0x561b9762cb90 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c0133c0 tx=0x7fdb0c016000 comp rx=0 tx=0).stop 2024-03-11T19:11:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.229+0000 7fdb1c623740 1 -- 172.21.15.29:0/3936618301 shutdown_connections 2024-03-11T19:11:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.229+0000 7fdb1c623740 1 --2- 172.21.15.29:0/3936618301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761d050 0x561b9762cb90 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.229+0000 7fdb1c623740 1 --2- 172.21.15.29:0/3936618301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761c690 0x561b97620700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.229+0000 7fdb1c623740 1 --2- 172.21.15.29:0/3936618301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761b4c0 0x561b976201c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.229+0000 7fdb1c623740 1 -- 172.21.15.29:0/3936618301 >> 172.21.15.29:0/3936618301 conn(0x561b976166a0 msgr2=0x561b97625650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.229+0000 7fdb1c623740 1 -- 172.21.15.29:0/3936618301 shutdown_connections 2024-03-11T19:11:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.229+0000 7fdb1c623740 1 -- 172.21.15.29:0/3936618301 wait complete. 2024-03-11T19:11:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.229+0000 7fdb1c623740 1 -- start start 2024-03-11T19:11:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.229+0000 7fdb1c623740 1 Processor -- start 2024-03-11T19:11:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb1c623740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761b4c0 0x561b9762b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb1c623740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761c690 0x561b9762ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb1c623740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761d050 0x561b9762bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb1c623740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b97687950 con 0x561b9761b4c0 2024-03-11T19:11:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb1c623740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b97687a90 con 0x561b9761d050 2024-03-11T19:11:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb1c623740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b97687c00 con 0x561b9761c690 2024-03-11T19:11:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb1c623740 10 client.0 _ll_register_callbacks cb 0x7fdb08009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb12ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761b4c0 0x561b9762b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb12ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761b4c0 0x561b9762b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41716/0 (socket says 172.21.15.29:41716) 2024-03-11T19:11:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb12ffd640 1 -- 172.21.15.29:0/2378326942 learned_addr learned my addr 172.21.15.29:0/2378326942 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb127fc640 1 --2- 172.21.15.29:0/2378326942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761c690 0x561b9762ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb137fe640 1 --2- 172.21.15.29:0/2378326942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761d050 0x561b9762bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb12ffd640 1 -- 172.21.15.29:0/2378326942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761c690 msgr2=0x561b9762ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb12ffd640 1 --2- 172.21.15.29:0/2378326942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761c690 0x561b9762ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb12ffd640 1 -- 172.21.15.29:0/2378326942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761d050 msgr2=0x561b9762bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb12ffd640 1 --2- 172.21.15.29:0/2378326942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761d050 0x561b9762bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb12ffd640 1 -- 172.21.15.29:0/2378326942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0c013070 con 0x561b9761b4c0 2024-03-11T19:11:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb137fe640 1 --2- 172.21.15.29:0/2378326942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761d050 0x561b9762bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb12ffd640 1 --2- 172.21.15.29:0/2378326942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761b4c0 0x561b9762b550 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fdb04004630 tx=0x7fdb040cfab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e1298df35c78e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdaf2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdaf2ffd640 1 -- 172.21.15.29:0/2378326942 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb04099d70 con 0x561b9761b4c0 2024-03-11T19:11:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdaf2ffd640 1 -- 172.21.15.29:0/2378326942 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fdb040cd310 con 0x561b9761b4c0 2024-03-11T19:11:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.230+0000 7fdb1c623740 1 -- 172.21.15.29:0/2378326942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561b975a9900 con 0x561b9761b4c0 2024-03-11T19:11:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.231+0000 7fdaf2ffd640 1 -- 172.21.15.29:0/2378326942 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb0409ab10 con 0x561b9761b4c0 2024-03-11T19:11:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.231+0000 7fdb1c623740 10 client.15774 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.231+0000 7fdb1c623740 1 -- 172.21.15.29:0/2378326942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x561b97544910 con 0x561b9761b4c0 2024-03-11T19:11:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.231+0000 7fdaf2ffd640 1 -- 172.21.15.29:0/2378326942 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x561b975a9900 con 0x561b9761b4c0 2024-03-11T19:11:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.231+0000 7fdaf2ffd640 1 -- 172.21.15.29:0/2378326942 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fdb04020070 con 0x561b9761b4c0 2024-03-11T19:11:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.231+0000 7fdb1c623740 10 client.15774 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.231+0000 7fdb1c623740 10 client.15774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:11:08.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.231+0000 7fdb1c623740 20 client.15774 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:08.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.231+0000 7fdb1c623740 1 --2- 172.21.15.29:0/2378326942 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x561b97689600 0x561b9768ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.231+0000 7fdb1c623740 4 client.15774 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:11:08.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.231+0000 7fdb1c623740 1 -- 172.21.15.29:0/2378326942 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x561b97283860 con 0x561b97689600 2024-03-11T19:11:08.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.231+0000 7fdb127fc640 1 --2- 172.21.15.29:0/2378326942 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x561b97689600 0x561b9768ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.232+0000 7fdaf2ffd640 10 client.15774 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:08.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.232+0000 7fdb127fc640 1 --2- 172.21.15.29:0/2378326942 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x561b97689600 0x561b9768ba20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7beaec4d0906fad server_cookie=e1cf6ef36bc02500 in_seq=0 out_seq=0 2024-03-11T19:11:08.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.233+0000 7fdaf2ffd640 1 -- 172.21.15.29:0/2378326942 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x561b97283860 con 0x561b97689600 2024-03-11T19:11:08.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.233+0000 7fdaf2ffd640 10 client.15774 handle_command_reply: tid=0 2024-03-11T19:11:08.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.233+0000 7fdb1c623740 1 client.15774 shutdown 2024-03-11T19:11:08.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.233+0000 7fdb1c623740 1 -- 172.21.15.29:0/2378326942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761b4c0 msgr2=0x561b9762b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:08.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.233+0000 7fdb1c623740 1 --2- 172.21.15.29:0/2378326942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761b4c0 0x561b9762b550 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fdb04004630 tx=0x7fdb040cfab0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.233+0000 7fdb1c623740 1 -- 172.21.15.29:0/2378326942 shutdown_connections 2024-03-11T19:11:08.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.233+0000 7fdb1c623740 1 --2- 172.21.15.29:0/2378326942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561b9761d050 0x561b9762bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.233+0000 7fdb1c623740 1 --2- 172.21.15.29:0/2378326942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561b9761c690 0x561b9762ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.233+0000 7fdb1c623740 1 --2- 172.21.15.29:0/2378326942 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x561b97689600 0x561b9768ba20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.233+0000 7fdb1c623740 1 --2- 172.21.15.29:0/2378326942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561b9761b4c0 0x561b9762b550 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.233+0000 7fdb1c623740 1 -- 172.21.15.29:0/2378326942 >> 172.21.15.29:0/2378326942 conn(0x561b976166a0 msgr2=0x561b976255d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:08.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.233+0000 7fdaf2ffd640 0 client.15774 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:08.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.233+0000 7fdb1c623740 1 -- 172.21.15.29:0/2378326942 shutdown_connections 2024-03-11T19:11:08.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.233+0000 7fdb1c623740 1 -- 172.21.15.29:0/2378326942 wait complete. 2024-03-11T19:11:08.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.233+0000 7fdb1c623740 20 client.15774 trim_cache size 0 max 16384 2024-03-11T19:11:08.243 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:11:08.243 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:11:08.243 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "44050744-3e3a-4a73-b30a-968aa0f5115b", 2024-03-11T19:11:08.243 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:11:08.243 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:11:08.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.234+0000 7fdb19977640 1 -- 172.21.15.29:0/1236692481 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdafc0a1ac0 msgr2=0x7fdafc0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:08.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.234+0000 7fdb19977640 1 --2- 172.21.15.29:0/1236692481 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdafc0a1ac0 0x7fdafc0a3f80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c009fd0 tx=0x7fdb0c004a40 comp rx=0 tx=0).stop 2024-03-11T19:11:08.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.234+0000 7fdb19977640 1 -- 172.21.15.29:0/1236692481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb14100080 msgr2=0x7fdb14197170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:08.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.234+0000 7fdb19977640 1 --2- 172.21.15.29:0/1236692481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb14100080 0x7fdb14197170 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdb0400c970 tx=0x7fdb0400ce30 comp rx=0 tx=0).stop 2024-03-11T19:11:08.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.234+0000 7fdb19977640 1 -- 172.21.15.29:0/1236692481 shutdown_connections 2024-03-11T19:11:08.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.234+0000 7fdb19977640 1 --2- 172.21.15.29:0/1236692481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb14101c30 0x7fdb1419e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.234+0000 7fdb19977640 1 --2- 172.21.15.29:0/1236692481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb14101270 0x7fdb141976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.234+0000 7fdb19977640 1 --2- 172.21.15.29:0/1236692481 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdafc0a1ac0 0x7fdafc0a3f80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.234+0000 7fdb19977640 1 --2- 172.21.15.29:0/1236692481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb14100080 0x7fdb14197170 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.234+0000 7fdb19977640 1 -- 172.21.15.29:0/1236692481 >> 172.21.15.29:0/1236692481 conn(0x7fdb140fb610 msgr2=0x7fdb14068310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:08.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.234+0000 7fdb19977640 1 -- 172.21.15.29:0/1236692481 shutdown_connections 2024-03-11T19:11:08.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.234+0000 7fdb19977640 1 -- 172.21.15.29:0/1236692481 wait complete. 2024-03-11T19:11:08.249 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:11:08.249 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:11:08.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.354+0000 7fc81a320640 1 -- 172.21.15.29:0/935160394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc814100080 msgr2=0x7fc8141004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.354+0000 7fc81a320640 1 --2- 172.21.15.29:0/935160394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc814100080 0x7fc8141004a0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc804009900 tx=0x7fc80402bc30 comp rx=0 tx=0).stop 2024-03-11T19:11:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.354+0000 7fc81a320640 1 -- 172.21.15.29:0/935160394 shutdown_connections 2024-03-11T19:11:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.354+0000 7fc81a320640 1 --2- 172.21.15.29:0/935160394 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc814101c30 0x7fc8141084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.354+0000 7fc81a320640 1 --2- 172.21.15.29:0/935160394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc814101270 0x7fc8141016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.354+0000 7fc81a320640 1 --2- 172.21.15.29:0/935160394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc814100080 0x7fc8141004a0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.354+0000 7fc81a320640 1 -- 172.21.15.29:0/935160394 >> 172.21.15.29:0/935160394 conn(0x7fc8140fb5f0 msgr2=0x7fc8140fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.354+0000 7fc81a320640 1 -- 172.21.15.29:0/935160394 shutdown_connections 2024-03-11T19:11:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.354+0000 7fc81a320640 1 -- 172.21.15.29:0/935160394 wait complete. 2024-03-11T19:11:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.354+0000 7fc81a320640 1 Processor -- start 2024-03-11T19:11:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.354+0000 7fc81a320640 1 -- start start 2024-03-11T19:11:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc81a320640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc814100080 0x7fc814068e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc81a320640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc814101270 0x7fc81419bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc81a320640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc814101c30 0x7fc81419e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc81a320640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81419ecd0 con 0x7fc814101270 2024-03-11T19:11:08.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc81a320640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81419ee40 con 0x7fc814101c30 2024-03-11T19:11:08.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc81a320640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81419f000 con 0x7fc814100080 2024-03-11T19:11:08.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc8137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc814101270 0x7fc81419bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc8137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc814101270 0x7fc81419bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41722/0 (socket says 172.21.15.29:41722) 2024-03-11T19:11:08.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc8137fe640 1 -- 172.21.15.29:0/1075343159 learned_addr learned my addr 172.21.15.29:0/1075343159 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:08.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc813fff640 1 --2- 172.21.15.29:0/1075343159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc814100080 0x7fc814068e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc8188bd640 1 --2- 172.21.15.29:0/1075343159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc814101c30 0x7fc81419e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc8137fe640 1 -- 172.21.15.29:0/1075343159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc814100080 msgr2=0x7fc814068e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:08.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc8137fe640 1 --2- 172.21.15.29:0/1075343159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc814100080 0x7fc814068e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc8137fe640 1 -- 172.21.15.29:0/1075343159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc814101c30 msgr2=0x7fc81419e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:08.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc8137fe640 1 --2- 172.21.15.29:0/1075343159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc814101c30 0x7fc81419e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc8137fe640 1 -- 172.21.15.29:0/1075343159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc804009560 con 0x7fc814101270 2024-03-11T19:11:08.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc8137fe640 1 --2- 172.21.15.29:0/1075343159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc814101270 0x7fc81419bed0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fc80800d770 tx=0x7fc80800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:08.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc8117fa640 1 -- 172.21.15.29:0/1075343159 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc808009c30 con 0x7fc814101270 2024-03-11T19:11:08.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc8117fa640 1 -- 172.21.15.29:0/1075343159 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc80800de80 con 0x7fc814101270 2024-03-11T19:11:08.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.355+0000 7fc81a320640 1 -- 172.21.15.29:0/1075343159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc81419f280 con 0x7fc814101270 2024-03-11T19:11:08.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.356+0000 7fc8117fa640 1 -- 172.21.15.29:0/1075343159 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc808004e00 con 0x7fc814101270 2024-03-11T19:11:08.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.356+0000 7fc81a320640 1 -- 172.21.15.29:0/1075343159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc81419f740 con 0x7fc814101270 2024-03-11T19:11:08.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.358+0000 7fc8117fa640 1 -- 172.21.15.29:0/1075343159 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fc808004f60 con 0x7fc814101270 2024-03-11T19:11:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.358+0000 7fc81cfcc740 1 Processor -- start 2024-03-11T19:11:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.358+0000 7fc81cfcc740 1 -- start start 2024-03-11T19:11:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.358+0000 7fc81cfcc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933daa2d0 0x555933daa6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.358+0000 7fc81cfcc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daacc0 0x555933daf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.358+0000 7fc81cfcc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933dafe30 0x555933db22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.358+0000 7fc81cfcc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555933d1a280 con 0x555933daacc0 2024-03-11T19:11:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.358+0000 7fc81cfcc740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555933d177b0 con 0x555933dafe30 2024-03-11T19:11:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.358+0000 7fc81cfcc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555933d12f90 con 0x555933daa2d0 2024-03-11T19:11:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.358+0000 7fc813fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daacc0 0x555933daf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.358+0000 7fc813fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daacc0 0x555933daf7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41736/0 (socket says 172.21.15.29:41736) 2024-03-11T19:11:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.358+0000 7fc813fff640 1 -- 172.21.15.29:0/2967568716 learned_addr learned my addr 172.21.15.29:0/2967568716 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.358+0000 7fc8188bd640 1 --2- 172.21.15.29:0/2967568716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933daa2d0 0x555933daa6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.358+0000 7fc8117fa640 1 --2- 172.21.15.29:0/1075343159 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc7fc0a1ac0 0x7fc7fc0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.358+0000 7fc8137fe640 1 --2- 172.21.15.29:0/2967568716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933dafe30 0x555933db22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.359+0000 7fc813fff640 1 -- 172.21.15.29:0/2967568716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933daa2d0 msgr2=0x555933daa6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.359+0000 7fc813fff640 1 --2- 172.21.15.29:0/2967568716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933daa2d0 0x555933daa6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.359+0000 7fc813fff640 1 -- 172.21.15.29:0/2967568716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933dafe30 msgr2=0x555933db22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.359+0000 7fc813fff640 1 --2- 172.21.15.29:0/2967568716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933dafe30 0x555933db22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.359+0000 7fc813fff640 1 -- 172.21.15.29:0/2967568716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555933cd1570 con 0x555933daacc0 2024-03-11T19:11:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.359+0000 7fc8137fe640 1 --2- 172.21.15.29:0/2967568716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933dafe30 0x555933db22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:11:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.359+0000 7fc8188bd640 1 --2- 172.21.15.29:0/2967568716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933daa2d0 0x555933daa6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.359+0000 7fc8188bd640 1 --2- 172.21.15.29:0/1075343159 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc7fc0a1ac0 0x7fc7fc0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.359+0000 7fc813fff640 1 --2- 172.21.15.29:0/2967568716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daacc0 0x555933daf7c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc8141010d0 tx=0x7fc804005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265a0f4aabdc35ac server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.359+0000 7fc8117fa640 1 -- 172.21.15.29:0/1075343159 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fc8080ccfd0 con 0x7fc814101270 2024-03-11T19:11:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.359+0000 7fc7fa7fc640 1 -- 172.21.15.29:0/2967568716 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80402cb70 con 0x555933daacc0 2024-03-11T19:11:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.359+0000 7fc8188bd640 1 --2- 172.21.15.29:0/1075343159 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc7fc0a1ac0 0x7fc7fc0a3f80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x555933dab520 tx=0x7fc80c004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.359+0000 7fc7fa7fc640 1 -- 172.21.15.29:0/2967568716 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc80402de90 con 0x555933daacc0 2024-03-11T19:11:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.359+0000 7fc7fa7fc640 1 -- 172.21.15.29:0/2967568716 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8040039b0 con 0x555933daacc0 2024-03-11T19:11:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.360+0000 7fc81cfcc740 1 -- 172.21.15.29:0/2967568716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daacc0 msgr2=0x555933daf7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.360+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/2967568716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daacc0 0x555933daf7c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc8141010d0 tx=0x7fc804005dd0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.360+0000 7fc81cfcc740 1 -- 172.21.15.29:0/2967568716 shutdown_connections 2024-03-11T19:11:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.360+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/2967568716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933dafe30 0x555933db22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.360+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/2967568716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daacc0 0x555933daf7c0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.360+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/2967568716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933daa2d0 0x555933daa6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.360+0000 7fc81cfcc740 1 -- 172.21.15.29:0/2967568716 >> 172.21.15.29:0/2967568716 conn(0x555933da57f0 msgr2=0x555933da5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.360+0000 7fc81cfcc740 1 -- 172.21.15.29:0/2967568716 shutdown_connections 2024-03-11T19:11:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.360+0000 7fc81cfcc740 1 -- 172.21.15.29:0/2967568716 wait complete. 2024-03-11T19:11:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.360+0000 7fc81cfcc740 1 -- start start 2024-03-11T19:11:08.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.360+0000 7fc81cfcc740 1 Processor -- start 2024-03-11T19:11:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc81cfcc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933daa2d0 0x555933db4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc81cfcc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933daacc0 0x555933db4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc81cfcc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933dafe30 0x555933dbb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc81cfcc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559339e6430 con 0x555933dafe30 2024-03-11T19:11:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc81cfcc740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559339a2e00 con 0x555933daa2d0 2024-03-11T19:11:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc81cfcc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555933c97fd0 con 0x555933daacc0 2024-03-11T19:11:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc81cfcc740 10 client.0 _ll_register_callbacks cb 0x555933cedfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc8188bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933dafe30 0x555933dbb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc8188bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933dafe30 0x555933dbb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41742/0 (socket says 172.21.15.29:41742) 2024-03-11T19:11:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc8188bd640 1 -- 172.21.15.29:0/1653132867 learned_addr learned my addr 172.21.15.29:0/1653132867 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc813fff640 1 --2- 172.21.15.29:0/1653132867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933daa2d0 0x555933db4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc8137fe640 1 --2- 172.21.15.29:0/1653132867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933daacc0 0x555933db4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc8188bd640 1 -- 172.21.15.29:0/1653132867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933daacc0 msgr2=0x555933db4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc8188bd640 1 --2- 172.21.15.29:0/1653132867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933daacc0 0x555933db4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc8188bd640 1 -- 172.21.15.29:0/1653132867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933daa2d0 msgr2=0x555933db4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc8188bd640 1 --2- 172.21.15.29:0/1653132867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933daa2d0 0x555933db4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc8188bd640 1 -- 172.21.15.29:0/1653132867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc814108dc0 con 0x555933dafe30 2024-03-11T19:11:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc8188bd640 1 --2- 172.21.15.29:0/1653132867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933dafe30 0x555933dbb7c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc80c009c40 tx=0x7fc80c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22e0c5ab9a208d26 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc7f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.361+0000 7fc7f37fe640 1 -- 172.21.15.29:0/1653132867 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80c021070 con 0x555933dafe30 2024-03-11T19:11:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.362+0000 7fc7f37fe640 1 -- 172.21.15.29:0/1653132867 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fc80c01ce10 con 0x555933dafe30 2024-03-11T19:11:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.362+0000 7fc81cfcc740 1 -- 172.21.15.29:0/1653132867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555933a127e0 con 0x555933dafe30 2024-03-11T19:11:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.362+0000 7fc7f37fe640 1 -- 172.21.15.29:0/1653132867 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc80c01c2e0 con 0x555933dafe30 2024-03-11T19:11:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.362+0000 7fc81cfcc740 10 client.15786 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.362+0000 7fc81cfcc740 1 -- 172.21.15.29:0/1653132867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555933d11610 con 0x555933dafe30 2024-03-11T19:11:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.362+0000 7fc7f37fe640 1 -- 172.21.15.29:0/1653132867 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x555933a127e0 con 0x555933dafe30 2024-03-11T19:11:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.362+0000 7fc7f37fe640 1 -- 172.21.15.29:0/1653132867 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc80c025410 con 0x555933dafe30 2024-03-11T19:11:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.362+0000 7fc81cfcc740 10 client.15786 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.362+0000 7fc81cfcc740 10 client.15786 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:11:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.362+0000 7fc81cfcc740 20 client.15786 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.362+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/1653132867 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555933dc04a0 0x555933dc28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.362+0000 7fc81cfcc740 4 client.15786 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:11:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.362+0000 7fc81cfcc740 1 -- 172.21.15.29:0/1653132867 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555933a4f720 con 0x555933dc04a0 2024-03-11T19:11:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.362+0000 7fc813fff640 1 --2- 172.21.15.29:0/1653132867 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555933dc04a0 0x555933dc28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.363+0000 7fc813fff640 1 --2- 172.21.15.29:0/1653132867 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555933dc04a0 0x555933dc28e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd8cff14114c5b1d server_cookie=eb8865c2e183f71a in_seq=0 out_seq=0 2024-03-11T19:11:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.363+0000 7fc7f37fe640 10 client.15786 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.363+0000 7fc7f37fe640 1 -- 172.21.15.29:0/1653132867 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555933a4f720 con 0x555933dc04a0 2024-03-11T19:11:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.363+0000 7fc7f37fe640 10 client.15786 handle_command_reply: tid=0 2024-03-11T19:11:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.363+0000 7fc81cfcc740 1 client.15786 shutdown 2024-03-11T19:11:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.364+0000 7fc81cfcc740 1 -- 172.21.15.29:0/1653132867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933dafe30 msgr2=0x555933dbb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.364+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/1653132867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933dafe30 0x555933dbb7c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc80c009c40 tx=0x7fc80c004ab0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.364+0000 7fc81cfcc740 1 -- 172.21.15.29:0/1653132867 shutdown_connections 2024-03-11T19:11:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.364+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/1653132867 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555933dc04a0 0x555933dc28e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.364+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/1653132867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933dafe30 0x555933dbb7c0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.364+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/1653132867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933daacc0 0x555933db4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.364+0000 7fc7f37fe640 0 client.15786 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.364+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/1653132867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933daa2d0 0x555933db4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.364+0000 7fc81cfcc740 1 -- 172.21.15.29:0/1653132867 >> 172.21.15.29:0/1653132867 conn(0x555933da57f0 msgr2=0x555933da4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.364+0000 7fc81cfcc740 1 -- 172.21.15.29:0/1653132867 shutdown_connections 2024-03-11T19:11:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.364+0000 7fc81cfcc740 1 -- 172.21.15.29:0/1653132867 wait complete. 2024-03-11T19:11:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.364+0000 7fc81cfcc740 20 client.15786 trim_cache size 0 max 16384 2024-03-11T19:11:08.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.379+0000 7fc81cfcc740 1 Processor -- start 2024-03-11T19:11:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.379+0000 7fc81cfcc740 1 -- start start 2024-03-11T19:11:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc81cfcc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daa2d0 0x555933dbc210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc81cfcc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933daacc0 0x555933dabf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc81cfcc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933dafe30 0x555933dac4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc81cfcc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555933a127e0 con 0x555933daa2d0 2024-03-11T19:11:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc81cfcc740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555933db2680 con 0x555933daacc0 2024-03-11T19:11:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc81cfcc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555933da1320 con 0x555933dafe30 2024-03-11T19:11:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc813fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daa2d0 0x555933dbc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc813fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daa2d0 0x555933dbc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41752/0 (socket says 172.21.15.29:41752) 2024-03-11T19:11:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc813fff640 1 -- 172.21.15.29:0/202038984 learned_addr learned my addr 172.21.15.29:0/202038984 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc8188bd640 1 --2- 172.21.15.29:0/202038984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933dafe30 0x555933dac4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc8137fe640 1 --2- 172.21.15.29:0/202038984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933daacc0 0x555933dabf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc813fff640 1 -- 172.21.15.29:0/202038984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933dafe30 msgr2=0x555933dac4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc813fff640 1 --2- 172.21.15.29:0/202038984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933dafe30 0x555933dac4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc813fff640 1 -- 172.21.15.29:0/202038984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933daacc0 msgr2=0x555933dabf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:08.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc813fff640 1 --2- 172.21.15.29:0/202038984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933daacc0 0x555933dabf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc813fff640 1 -- 172.21.15.29:0/202038984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555933cadac0 con 0x555933daa2d0 2024-03-11T19:11:08.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc813fff640 1 --2- 172.21.15.29:0/202038984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daa2d0 0x555933dbc210 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x555933dab730 tx=0x7fc804040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4afa6f44d3558ace server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:08.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc7f9ffb640 1 -- 172.21.15.29:0/202038984 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8040048c0 con 0x555933daa2d0 2024-03-11T19:11:08.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc7f9ffb640 1 -- 172.21.15.29:0/202038984 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc8040044e0 con 0x555933daa2d0 2024-03-11T19:11:08.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.380+0000 7fc7f9ffb640 1 -- 172.21.15.29:0/202038984 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8040505d0 con 0x555933daa2d0 2024-03-11T19:11:08.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.381+0000 7fc81cfcc740 1 -- 172.21.15.29:0/202038984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daa2d0 msgr2=0x555933dbc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:08.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.381+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/202038984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daa2d0 0x555933dbc210 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x555933dab730 tx=0x7fc804040040 comp rx=0 tx=0).stop 2024-03-11T19:11:08.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.381+0000 7fc81cfcc740 1 -- 172.21.15.29:0/202038984 shutdown_connections 2024-03-11T19:11:08.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.381+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/202038984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933dafe30 0x555933dac4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.381+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/202038984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933daacc0 0x555933dabf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.381+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/202038984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daa2d0 0x555933dbc210 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.381+0000 7fc81cfcc740 1 -- 172.21.15.29:0/202038984 >> 172.21.15.29:0/202038984 conn(0x555933da57f0 msgr2=0x555933db4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.381+0000 7fc81cfcc740 1 -- 172.21.15.29:0/202038984 shutdown_connections 2024-03-11T19:11:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.381+0000 7fc81cfcc740 1 -- 172.21.15.29:0/202038984 wait complete. 2024-03-11T19:11:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.382+0000 7fc81cfcc740 1 -- start start 2024-03-11T19:11:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.382+0000 7fc81cfcc740 1 Processor -- start 2024-03-11T19:11:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.382+0000 7fc81cfcc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933daa2d0 0x555933e12b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.382+0000 7fc81cfcc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daacc0 0x555933e130b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.382+0000 7fc81cfcc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933dafe30 0x555933e1a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.382+0000 7fc81cfcc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555933e1a630 con 0x555933daacc0 2024-03-11T19:11:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.382+0000 7fc81cfcc740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555933e1a770 con 0x555933dafe30 2024-03-11T19:11:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.382+0000 7fc81cfcc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555933e1a8e0 con 0x555933daa2d0 2024-03-11T19:11:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.382+0000 7fc81cfcc740 10 client.0 _ll_register_callbacks cb 0x7fc80c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.382+0000 7fc8137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daacc0 0x555933e130b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.382+0000 7fc8137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daacc0 0x555933e130b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41758/0 (socket says 172.21.15.29:41758) 2024-03-11T19:11:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.382+0000 7fc8137fe640 1 -- 172.21.15.29:0/2196451980 learned_addr learned my addr 172.21.15.29:0/2196451980 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.383+0000 7fc813fff640 1 --2- 172.21.15.29:0/2196451980 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933daa2d0 0x555933e12b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.383+0000 7fc8188bd640 1 --2- 172.21.15.29:0/2196451980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933dafe30 0x555933e1a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.383+0000 7fc8137fe640 1 -- 172.21.15.29:0/2196451980 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933daa2d0 msgr2=0x555933e12b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.383+0000 7fc8137fe640 1 --2- 172.21.15.29:0/2196451980 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933daa2d0 0x555933e12b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.383+0000 7fc8137fe640 1 -- 172.21.15.29:0/2196451980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933dafe30 msgr2=0x555933e1a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.383+0000 7fc8137fe640 1 --2- 172.21.15.29:0/2196451980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933dafe30 0x555933e1a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.383+0000 7fc8137fe640 1 -- 172.21.15.29:0/2196451980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555933cd1570 con 0x555933daacc0 2024-03-11T19:11:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.383+0000 7fc8188bd640 1 --2- 172.21.15.29:0/2196451980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933dafe30 0x555933e1a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:11:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.383+0000 7fc8137fe640 1 --2- 172.21.15.29:0/2196451980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daacc0 0x555933e130b0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc808009610 tx=0x7fc808099ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1aaa8dfafd5b18 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.383+0000 7fc7f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.383+0000 7fc7f37fe640 1 -- 172.21.15.29:0/2196451980 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8080a2680 con 0x555933daacc0 2024-03-11T19:11:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.383+0000 7fc7f37fe640 1 -- 172.21.15.29:0/2196451980 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fc8080cf0e0 con 0x555933daacc0 2024-03-11T19:11:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.383+0000 7fc81cfcc740 1 -- 172.21.15.29:0/2196451980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc80402de90 con 0x555933daacc0 2024-03-11T19:11:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.383+0000 7fc7f37fe640 1 -- 172.21.15.29:0/2196451980 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8080cf2a0 con 0x555933daacc0 2024-03-11T19:11:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.383+0000 7fc81cfcc740 10 client.15792 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.383+0000 7fc81cfcc740 1 -- 172.21.15.29:0/2196451980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555933cd36a0 con 0x555933daacc0 2024-03-11T19:11:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.383+0000 7fc7f37fe640 1 -- 172.21.15.29:0/2196451980 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc80402de90 con 0x555933daacc0 2024-03-11T19:11:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.384+0000 7fc7f37fe640 1 -- 172.21.15.29:0/2196451980 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc80809d030 con 0x555933daacc0 2024-03-11T19:11:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.384+0000 7fc81cfcc740 10 client.15792 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.384+0000 7fc81cfcc740 10 client.15792 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:11:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.384+0000 7fc81cfcc740 20 client.15792 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.384+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/2196451980 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555933e1c4b0 0x555933e1e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.384+0000 7fc81cfcc740 4 client.15792 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:11:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.384+0000 7fc81cfcc740 1 -- 172.21.15.29:0/2196451980 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555933a9e890 con 0x555933e1c4b0 2024-03-11T19:11:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.384+0000 7fc813fff640 1 --2- 172.21.15.29:0/2196451980 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555933e1c4b0 0x555933e1e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.384+0000 7fc7f37fe640 10 client.15792 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.384+0000 7fc813fff640 1 --2- 172.21.15.29:0/2196451980 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555933e1c4b0 0x555933e1e8d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=387ac44ea5f1e912 server_cookie=af4cc456f4ac0727 in_seq=0 out_seq=0 2024-03-11T19:11:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.385+0000 7fc7f37fe640 1 -- 172.21.15.29:0/2196451980 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x555933a9e890 con 0x555933e1c4b0 2024-03-11T19:11:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.385+0000 7fc7f37fe640 10 client.15792 handle_command_reply: tid=0 2024-03-11T19:11:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.385+0000 7fc81cfcc740 1 client.15792 shutdown 2024-03-11T19:11:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.385+0000 7fc81cfcc740 1 -- 172.21.15.29:0/2196451980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daacc0 msgr2=0x555933e130b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.385+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/2196451980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daacc0 0x555933e130b0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc808009610 tx=0x7fc808099ee0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.385+0000 7fc81cfcc740 1 -- 172.21.15.29:0/2196451980 shutdown_connections 2024-03-11T19:11:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.385+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/2196451980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555933dafe30 0x555933e1a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.385+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/2196451980 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555933e1c4b0 0x555933e1e8d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.385+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/2196451980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555933daacc0 0x555933e130b0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.385+0000 7fc81cfcc740 1 --2- 172.21.15.29:0/2196451980 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555933daa2d0 0x555933e12b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.385+0000 7fc7f37fe640 0 client.15792 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.385+0000 7fc81cfcc740 1 -- 172.21.15.29:0/2196451980 >> 172.21.15.29:0/2196451980 conn(0x555933da57f0 msgr2=0x555933db41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.385+0000 7fc81cfcc740 1 -- 172.21.15.29:0/2196451980 shutdown_connections 2024-03-11T19:11:08.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.385+0000 7fc81cfcc740 1 -- 172.21.15.29:0/2196451980 wait complete. 2024-03-11T19:11:08.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.385+0000 7fc81cfcc740 20 client.15792 trim_cache size 0 max 16384 2024-03-11T19:11:08.395 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:11:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:11:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:11:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "44050744-3e3a-4a73-b30a-968aa0f5115b": { 2024-03-11T19:11:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:11:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "44050744-3e3a-4a73-b30a-968aa0f5115b", 2024-03-11T19:11:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:11:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:11:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:11:08.396 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:11:08.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.385+0000 7fc81a320640 1 -- 172.21.15.29:0/1075343159 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc7fc0a1ac0 msgr2=0x7fc7fc0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:08.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.385+0000 7fc81a320640 1 --2- 172.21.15.29:0/1075343159 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc7fc0a1ac0 0x7fc7fc0a3f80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x555933dab520 tx=0x7fc80c004a40 comp rx=0 tx=0).stop 2024-03-11T19:11:08.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.385+0000 7fc81a320640 1 -- 172.21.15.29:0/1075343159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc814101270 msgr2=0x7fc81419bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:08.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.385+0000 7fc81a320640 1 --2- 172.21.15.29:0/1075343159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc814101270 0x7fc81419bed0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fc80800d770 tx=0x7fc80800dc30 comp rx=0 tx=0).stop 2024-03-11T19:11:08.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.386+0000 7fc81a320640 1 -- 172.21.15.29:0/1075343159 shutdown_connections 2024-03-11T19:11:08.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.386+0000 7fc81a320640 1 --2- 172.21.15.29:0/1075343159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc814101c30 0x7fc81419e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.386+0000 7fc81a320640 1 --2- 172.21.15.29:0/1075343159 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc7fc0a1ac0 0x7fc7fc0a3f80 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.386+0000 7fc81a320640 1 --2- 172.21.15.29:0/1075343159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc814101270 0x7fc81419bed0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.386+0000 7fc81a320640 1 --2- 172.21.15.29:0/1075343159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc814100080 0x7fc814068e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:08.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.386+0000 7fc81a320640 1 -- 172.21.15.29:0/1075343159 >> 172.21.15.29:0/1075343159 conn(0x7fc8140fb5f0 msgr2=0x7fc814104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:08.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.386+0000 7fc81a320640 1 -- 172.21.15.29:0/1075343159 shutdown_connections 2024-03-11T19:11:08.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:08.386+0000 7fc81a320640 1 -- 172.21.15.29:0/1075343159 wait complete. 2024-03-11T19:11:08.400 INFO:tasks.cephfs.filesystem:scrub status for tag:44050744-3e3a-4a73-b30a-968aa0f5115b - {'path': '/', 'tag': '44050744-3e3a-4a73-b30a-968aa0f5115b', 'options': 'recursive,force'} 2024-03-11T19:11:08.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:08 smithi029 ceph-mon[32439]: pgmap v391: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:11:08.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:08 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:11:08.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:08 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:11:08.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:08 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:11:08.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:08 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:11:09.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:08 smithi190 ceph-mon[36292]: pgmap v391: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:11:09.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:08 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:11:09.014 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:08 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:11:09.014 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:08 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:11:09.015 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:08 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:11:09.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:08 smithi170 ceph-mon[35690]: pgmap v391: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:11:09.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:08 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:11:09.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:08 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:11:09.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:08 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:11:09.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:08 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:11:10.318 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 97% (12026/12344) Updating files: 98% (12098/12344) Updating files: 99% (12221/12344) Updating files: 100% (12344/12344) Updating files: 100% (12344/12344), done. 2024-03-11T19:11:10.717 INFO:tasks.workunit.client.0.smithi029.stderr:Note: switching to 'b6a4997be2b8bb74411268cc42753f29ae9a78ad'. 2024-03-11T19:11:10.717 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-11T19:11:10.717 INFO:tasks.workunit.client.0.smithi029.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-11T19:11:10.718 INFO:tasks.workunit.client.0.smithi029.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-11T19:11:10.718 INFO:tasks.workunit.client.0.smithi029.stderr:state without impacting any branches by switching back to a branch. 2024-03-11T19:11:10.718 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-11T19:11:10.718 INFO:tasks.workunit.client.0.smithi029.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-11T19:11:10.718 INFO:tasks.workunit.client.0.smithi029.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-11T19:11:10.718 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-11T19:11:10.718 INFO:tasks.workunit.client.0.smithi029.stderr: git switch -c 2024-03-11T19:11:10.718 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-11T19:11:10.718 INFO:tasks.workunit.client.0.smithi029.stderr:Or undo this operation with: 2024-03-11T19:11:10.718 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-11T19:11:10.718 INFO:tasks.workunit.client.0.smithi029.stderr: git switch - 2024-03-11T19:11:10.718 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-11T19:11:10.718 INFO:tasks.workunit.client.0.smithi029.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-11T19:11:10.718 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-11T19:11:10.719 INFO:tasks.workunit.client.0.smithi029.stderr:HEAD is now at b6a4997be2b Merge pull request #56060 from rhcs-dashboard/wip-64808-squid 2024-03-11T19:11:10.721 DEBUG:teuthology.orchestra.run.smithi029:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-11T19:11:10.754 INFO:tasks.workunit.client.0.smithi029.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-11T19:11:10.755 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-11T19:11:10.756 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-11T19:11:11.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:10 smithi190 ceph-mon[36292]: pgmap v392: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:11:11.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:10 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:11:11.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:10 smithi170 ceph-mon[35690]: pgmap v392: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:11:11.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:10 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:11:11.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:10 smithi029 ceph-mon[32439]: pgmap v392: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:11:11.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:10 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:11:11.842 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-11T19:11:11.943 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-11T19:11:11.984 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-11T19:11:11.985 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-11T19:11:11.985 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-11T19:11:12.024 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-11T19:11:12.027 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-11T19:11:12.028 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-11T19:11:12.083 INFO:tasks.workunit:Running workunits matching suites/iogen.sh on client.0... 2024-03-11T19:11:12.084 INFO:tasks.workunit:Running workunit suites/iogen.sh... 2024-03-11T19:11:12.085 DEBUG:teuthology.orchestra.run.smithi029:workunit test suites/iogen.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b6a4997be2b8bb74411268cc42753f29ae9a78ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iogen.sh 2024-03-11T19:11:12.147 INFO:tasks.workunit.client.0.smithi029.stdout:getting iogen 2024-03-11T19:11:12.147 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'getting iogen' 2024-03-11T19:11:12.147 INFO:tasks.workunit.client.0.smithi029.stderr:+ wget http://download.ceph.com/qa/iogen_3.1p0.tar 2024-03-11T19:11:12.174 INFO:tasks.workunit.client.0.smithi029.stderr:--2024-03-11 19:11:12-- http://download.ceph.com/qa/iogen_3.1p0.tar 2024-03-11T19:11:12.174 INFO:tasks.workunit.client.0.smithi029.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-03-11T19:11:12.267 INFO:tasks.workunit.client.0.smithi029.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-03-11T19:11:12.363 INFO:tasks.workunit.client.0.smithi029.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-11T19:11:12.363 INFO:tasks.workunit.client.0.smithi029.stderr:Length: 8071 (7.9K) [application/octet-stream] 2024-03-11T19:11:12.366 INFO:tasks.workunit.client.0.smithi029.stderr:Saving to: ‘iogen_3.1p0.tar’ 2024-03-11T19:11:12.369 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-11T19:11:12.369 INFO:tasks.workunit.client.0.smithi029.stderr: 0K ....... 100% 1.22G=0s 2024-03-11T19:11:12.370 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-11T19:11:12.370 INFO:tasks.workunit.client.0.smithi029.stderr:2024-03-11 19:11:12 (1.22 GB/s) - ‘iogen_3.1p0.tar’ saved [8071/8071] 2024-03-11T19:11:12.370 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-11T19:11:12.370 INFO:tasks.workunit.client.0.smithi029.stderr:+ tar -xvzf iogen_3.1p0.tar 2024-03-11T19:11:12.390 INFO:tasks.workunit.client.0.smithi029.stdout:iogen_3.1p0 2024-03-11T19:11:12.392 INFO:tasks.workunit.client.0.smithi029.stdout:iogen_3.1p0/src 2024-03-11T19:11:12.394 INFO:tasks.workunit.client.0.smithi029.stdout:iogen_3.1p0/src/iogen.8 2024-03-11T19:11:12.397 INFO:tasks.workunit.client.0.smithi029.stdout:iogen_3.1p0/src/iogen.c 2024-03-11T19:11:12.399 INFO:tasks.workunit.client.0.smithi029.stdout:iogen_3.1p0/README 2024-03-11T19:11:12.402 INFO:tasks.workunit.client.0.smithi029.stdout:iogen_3.1p0/Makefile.bsd 2024-03-11T19:11:12.404 INFO:tasks.workunit.client.0.smithi029.stdout:iogen_3.1p0/Makefile 2024-03-11T19:11:12.407 INFO:tasks.workunit.client.0.smithi029.stderr:+ cd iogen_3.1p0 2024-03-11T19:11:12.408 INFO:tasks.workunit.client.0.smithi029.stdout:making iogen 2024-03-11T19:11:12.408 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'making iogen' 2024-03-11T19:11:12.409 INFO:tasks.workunit.client.0.smithi029.stderr:+ make 2024-03-11T19:11:12.420 INFO:tasks.workunit.client.0.smithi029.stdout:gcc -D_FILE_OFFSET_BITS=64 -DVERSION=\"3.1p0\" -Dstrlcpy=strncpy -Darc4random=rand -O2 src/iogen.c -o iogen 2024-03-11T19:11:12.782 INFO:tasks.workunit.client.0.smithi029.stdout:nroff -mandoc -Tascii src/iogen.8 > iogen.cat8 2024-03-11T19:11:13.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:12 smithi190 ceph-mon[36292]: pgmap v393: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-11T19:11:13.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:12 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:11:13.083 INFO:tasks.workunit.client.0.smithi029.stdout:running iogen 2024-03-11T19:11:13.083 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'running iogen' 2024-03-11T19:11:13.083 INFO:tasks.workunit.client.0.smithi029.stderr:+ ./iogen -n 5 -s 2g 2024-03-11T19:11:13.084 INFO:tasks.workunit.client.0.smithi029.stderr:child spawned 87567 2024-03-11T19:11:13.084 INFO:tasks.workunit.client.0.smithi029.stderr:child spawned 87568 2024-03-11T19:11:13.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:12 smithi029 ceph-mon[32439]: pgmap v393: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-11T19:11:13.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:12 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:11:13.085 INFO:tasks.workunit.client.0.smithi029.stderr:child spawned 87569 2024-03-11T19:11:13.085 INFO:tasks.workunit.client.0.smithi029.stderr:child spawned 87570 2024-03-11T19:11:13.085 INFO:tasks.workunit.client.0.smithi029.stderr:child spawned 87571 2024-03-11T19:11:13.086 INFO:tasks.workunit.client.0.smithi029.stdout:sleep for 10 min 2024-03-11T19:11:13.086 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'sleep for 10 min' 2024-03-11T19:11:13.086 INFO:tasks.workunit.client.0.smithi029.stderr:+ sleep 600 2024-03-11T19:11:13.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:12 smithi170 ceph-mon[35690]: pgmap v393: 97 pgs: 97 active+clean; 664 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-11T19:11:13.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:12 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:11:14.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:13 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:11:14.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:13 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:11:14.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:13 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:11:14.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:13 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:11:14.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:13 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:11:14.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:13 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:11:14.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:13 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:11:14.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:13 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:11:14.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:13 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:11:14.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:13 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:11:14.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:13 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:11:14.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:13 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:11:14.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:13 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:11:14.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:13 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:11:14.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:13 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:11:14.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:13 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:11:14.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:13 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:11:14.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:13 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:11:14.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:13 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:11:14.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:13 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:11:14.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:13 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:11:15.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:14 smithi170 ceph-mon[35690]: pgmap v394: 97 pgs: 97 active+clean; 9.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 792 KiB/s wr, 13 op/s 2024-03-11T19:11:15.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:14 smithi029 ceph-mon[32439]: pgmap v394: 97 pgs: 97 active+clean; 9.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 792 KiB/s wr, 13 op/s 2024-03-11T19:11:15.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:14 smithi190 ceph-mon[36292]: pgmap v394: 97 pgs: 97 active+clean; 9.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 792 KiB/s wr, 13 op/s 2024-03-11T19:11:17.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:16 smithi170 ceph-mon[35690]: pgmap v395: 97 pgs: 97 active+clean; 168 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 14 MiB/s wr, 18 op/s 2024-03-11T19:11:17.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:16 smithi170 ceph-mon[35690]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-11T19:11:17.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:16 smithi029 ceph-mon[32439]: pgmap v395: 97 pgs: 97 active+clean; 168 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 14 MiB/s wr, 18 op/s 2024-03-11T19:11:17.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:16 smithi029 ceph-mon[32439]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-11T19:11:17.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:16 smithi190 ceph-mon[36292]: pgmap v395: 97 pgs: 97 active+clean; 168 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 14 MiB/s wr, 18 op/s 2024-03-11T19:11:17.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:16 smithi190 ceph-mon[36292]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-11T19:11:19.098 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:18 smithi170 ceph-mon[35690]: pgmap v396: 97 pgs: 97 active+clean; 316 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 26 MiB/s wr, 30 op/s 2024-03-11T19:11:19.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:18 smithi029 ceph-mon[32439]: pgmap v396: 97 pgs: 97 active+clean; 316 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 26 MiB/s wr, 30 op/s 2024-03-11T19:11:19.264 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:18 smithi190 ceph-mon[36292]: pgmap v396: 97 pgs: 97 active+clean; 316 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 26 MiB/s wr, 30 op/s 2024-03-11T19:11:21.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:20 smithi170 ceph-mon[35690]: pgmap v397: 97 pgs: 97 active+clean; 674 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 56 MiB/s wr, 45 op/s 2024-03-11T19:11:21.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:20 smithi029 ceph-mon[32439]: pgmap v397: 97 pgs: 97 active+clean; 674 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 56 MiB/s wr, 45 op/s 2024-03-11T19:11:21.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:20 smithi190 ceph-mon[36292]: pgmap v397: 97 pgs: 97 active+clean; 674 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 56 MiB/s wr, 45 op/s 2024-03-11T19:11:23.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:22 smithi170 ceph-mon[35690]: pgmap v398: 97 pgs: 97 active+clean; 1021 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 85 MiB/s wr, 59 op/s 2024-03-11T19:11:23.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:22 smithi029 ceph-mon[32439]: pgmap v398: 97 pgs: 97 active+clean; 1021 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 85 MiB/s wr, 59 op/s 2024-03-11T19:11:23.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:22 smithi190 ceph-mon[36292]: pgmap v398: 97 pgs: 97 active+clean; 1021 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 85 MiB/s wr, 59 op/s 2024-03-11T19:11:25.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:24 smithi170 ceph-mon[35690]: pgmap v399: 97 pgs: 97 active+clean; 1.3 GiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 113 MiB/s wr, 76 op/s 2024-03-11T19:11:25.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:24 smithi029 ceph-mon[32439]: pgmap v399: 97 pgs: 97 active+clean; 1.3 GiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 113 MiB/s wr, 76 op/s 2024-03-11T19:11:25.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:24 smithi190 ceph-mon[36292]: pgmap v399: 97 pgs: 97 active+clean; 1.3 GiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 113 MiB/s wr, 76 op/s 2024-03-11T19:11:27.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:26 smithi170 ceph-mon[35690]: pgmap v400: 97 pgs: 97 active+clean; 1.8 GiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 153 MiB/s wr, 93 op/s 2024-03-11T19:11:27.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:26 smithi029 ceph-mon[32439]: pgmap v400: 97 pgs: 97 active+clean; 1.8 GiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 153 MiB/s wr, 93 op/s 2024-03-11T19:11:27.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:26 smithi190 ceph-mon[36292]: pgmap v400: 97 pgs: 97 active+clean; 1.8 GiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 153 MiB/s wr, 93 op/s 2024-03-11T19:11:29.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:28 smithi170 ceph-mon[35690]: pgmap v401: 97 pgs: 97 active+clean; 2.1 GiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 162 MiB/s wr, 98 op/s 2024-03-11T19:11:29.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:28 smithi029 ceph-mon[32439]: pgmap v401: 97 pgs: 97 active+clean; 2.1 GiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 162 MiB/s wr, 98 op/s 2024-03-11T19:11:29.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:28 smithi190 ceph-mon[36292]: pgmap v401: 97 pgs: 97 active+clean; 2.1 GiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 162 MiB/s wr, 98 op/s 2024-03-11T19:11:31.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:30 smithi170 ceph-mon[35690]: pgmap v402: 97 pgs: 97 active+clean; 2.4 GiB data, 846 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 182 MiB/s wr, 103 op/s 2024-03-11T19:11:31.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:30 smithi029 ceph-mon[32439]: pgmap v402: 97 pgs: 97 active+clean; 2.4 GiB data, 846 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 182 MiB/s wr, 103 op/s 2024-03-11T19:11:31.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:30 smithi190 ceph-mon[36292]: pgmap v402: 97 pgs: 97 active+clean; 2.4 GiB data, 846 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 182 MiB/s wr, 103 op/s 2024-03-11T19:11:33.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:32 smithi170 ceph-mon[35690]: pgmap v403: 97 pgs: 97 active+clean; 2.7 GiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 177 MiB/s wr, 100 op/s 2024-03-11T19:11:33.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:32 smithi029 ceph-mon[32439]: pgmap v403: 97 pgs: 97 active+clean; 2.7 GiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 177 MiB/s wr, 100 op/s 2024-03-11T19:11:33.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:32 smithi190 ceph-mon[36292]: pgmap v403: 97 pgs: 97 active+clean; 2.7 GiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 177 MiB/s wr, 100 op/s 2024-03-11T19:11:35.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:34 smithi170 ceph-mon[35690]: pgmap v404: 97 pgs: 97 active+clean; 3.1 GiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 181 MiB/s wr, 104 op/s 2024-03-11T19:11:35.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:34 smithi029 ceph-mon[32439]: pgmap v404: 97 pgs: 97 active+clean; 3.1 GiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 181 MiB/s wr, 104 op/s 2024-03-11T19:11:35.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:34 smithi190 ceph-mon[36292]: pgmap v404: 97 pgs: 97 active+clean; 3.1 GiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 181 MiB/s wr, 104 op/s 2024-03-11T19:11:37.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:36 smithi170 ceph-mon[35690]: pgmap v405: 97 pgs: 97 active+clean; 3.5 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 188 MiB/s wr, 105 op/s 2024-03-11T19:11:37.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:36 smithi029 ceph-mon[32439]: pgmap v405: 97 pgs: 97 active+clean; 3.5 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 188 MiB/s wr, 105 op/s 2024-03-11T19:11:37.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:36 smithi190 ceph-mon[36292]: pgmap v405: 97 pgs: 97 active+clean; 3.5 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 188 MiB/s wr, 105 op/s 2024-03-11T19:11:38.401 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:11:38.401 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:11:38.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.552+0000 7f9a99325640 1 -- 172.21.15.29:0/2782925241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a94101c30 msgr2=0x7f9a941084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:38.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.552+0000 7f9a99325640 1 --2- 172.21.15.29:0/2782925241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a94101c30 0x7f9a941084c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c009ca0 tx=0x7f9a8c02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.552+0000 7f9a99325640 1 -- 172.21.15.29:0/2782925241 shutdown_connections 2024-03-11T19:11:38.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.552+0000 7f9a99325640 1 --2- 172.21.15.29:0/2782925241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a94101c30 0x7f9a941084c0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.552+0000 7f9a99325640 1 --2- 172.21.15.29:0/2782925241 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a94101270 0x7f9a941016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.552+0000 7f9a99325640 1 --2- 172.21.15.29:0/2782925241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a94100080 0x7f9a941004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.553+0000 7f9a99325640 1 -- 172.21.15.29:0/2782925241 >> 172.21.15.29:0/2782925241 conn(0x7f9a940fb5f0 msgr2=0x7f9a940fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:38.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.553+0000 7f9a99325640 1 -- 172.21.15.29:0/2782925241 shutdown_connections 2024-03-11T19:11:38.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.553+0000 7f9a99325640 1 -- 172.21.15.29:0/2782925241 wait complete. 2024-03-11T19:11:38.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.553+0000 7f9a99325640 1 Processor -- start 2024-03-11T19:11:38.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.554+0000 7f9a99325640 1 -- start start 2024-03-11T19:11:38.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.554+0000 7f9a99325640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a94100080 0x7f9a94197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.554+0000 7f9a99325640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a94101270 0x7f9a941976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.554+0000 7f9a99325640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a94101c30 0x7f9a9419e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.554+0000 7f9a92ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a94100080 0x7f9a94197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.554+0000 7f9a99325640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9419ec60 con 0x7f9a94100080 2024-03-11T19:11:38.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.554+0000 7f9a92ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a94100080 0x7f9a94197170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45918/0 (socket says 172.21.15.29:45918) 2024-03-11T19:11:38.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.554+0000 7f9a92ffd640 1 -- 172.21.15.29:0/174500746 learned_addr learned my addr 172.21.15.29:0/174500746 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:38.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.554+0000 7f9a99325640 1 -- 172.21.15.29:0/174500746 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9419edd0 con 0x7f9a94101c30 2024-03-11T19:11:38.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.554+0000 7f9a99325640 1 -- 172.21.15.29:0/174500746 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9419ef90 con 0x7f9a94101270 2024-03-11T19:11:38.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.554+0000 7f9a937fe640 1 --2- 172.21.15.29:0/174500746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a94101c30 0x7f9a9419e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.554+0000 7f9a927fc640 1 --2- 172.21.15.29:0/174500746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a94101270 0x7f9a941976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.554+0000 7f9a92ffd640 1 -- 172.21.15.29:0/174500746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a94101270 msgr2=0x7f9a941976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:38.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.554+0000 7f9a92ffd640 1 --2- 172.21.15.29:0/174500746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a94101270 0x7f9a941976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.554+0000 7f9a92ffd640 1 -- 172.21.15.29:0/174500746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a94101c30 msgr2=0x7f9a9419e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:38.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.554+0000 7f9a92ffd640 1 --2- 172.21.15.29:0/174500746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a94101c30 0x7f9a9419e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.554+0000 7f9a92ffd640 1 -- 172.21.15.29:0/174500746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a84009490 con 0x7f9a94100080 2024-03-11T19:11:38.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.555+0000 7f9a92ffd640 1 --2- 172.21.15.29:0/174500746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a94100080 0x7f9a94197170 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f9a8400e9c0 tx=0x7f9a8400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:38.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.555+0000 7f9a7bfff640 1 -- 172.21.15.29:0/174500746 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8400cd80 con 0x7f9a94100080 2024-03-11T19:11:38.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.555+0000 7f9a7bfff640 1 -- 172.21.15.29:0/174500746 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9a840108e0 con 0x7f9a94100080 2024-03-11T19:11:38.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.555+0000 7f9a99325640 1 -- 172.21.15.29:0/174500746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a8c009900 con 0x7f9a94100080 2024-03-11T19:11:38.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.555+0000 7f9a7bfff640 1 -- 172.21.15.29:0/174500746 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a84018840 con 0x7f9a94100080 2024-03-11T19:11:38.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.555+0000 7f9a99325640 1 -- 172.21.15.29:0/174500746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a9419f600 con 0x7f9a94100080 2024-03-11T19:11:38.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.558+0000 7f9a7bfff640 1 -- 172.21.15.29:0/174500746 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9a84010a50 con 0x7f9a94100080 2024-03-11T19:11:38.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.558+0000 7f9a9bfd4740 1 Processor -- start 2024-03-11T19:11:38.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.558+0000 7f9a9bfd4740 1 -- start start 2024-03-11T19:11:38.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.558+0000 7f9a7bfff640 1 --2- 172.21.15.29:0/174500746 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a7c0a1ab0 0x7f9a7c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.559+0000 7f9a9bfd4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 0x556b8d11db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.559+0000 7f9a9bfd4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11e0d0 0x556b8d122bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.559+0000 7f9a9bfd4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d123240 0x556b8d1256d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.559+0000 7f9a9bfd4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b8d089280 con 0x556b8d11d6e0 2024-03-11T19:11:38.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.559+0000 7f9a9bfd4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b8d0867b0 con 0x556b8d123240 2024-03-11T19:11:38.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.559+0000 7f9a9bfd4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b8d081f90 con 0x556b8d11e0d0 2024-03-11T19:11:38.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.559+0000 7f9a92ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 0x556b8d11db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.559+0000 7f9a92ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 0x556b8d11db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45932/0 (socket says 172.21.15.29:45932) 2024-03-11T19:11:38.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.559+0000 7f9a92ffd640 1 -- 172.21.15.29:0/1839141717 learned_addr learned my addr 172.21.15.29:0/1839141717 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:38.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.559+0000 7f9a937fe640 1 --2- 172.21.15.29:0/174500746 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a7c0a1ab0 0x7f9a7c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.559+0000 7f9a7bfff640 1 -- 172.21.15.29:0/174500746 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9a84014070 con 0x7f9a94100080 2024-03-11T19:11:38.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.559+0000 7f9a927fc640 1 --2- 172.21.15.29:0/1839141717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11e0d0 0x556b8d122bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.559+0000 7f9a927fc640 1 -- 172.21.15.29:0/1839141717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d123240 msgr2=0x556b8d1256d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:11:38.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.559+0000 7f9a927fc640 1 --2- 172.21.15.29:0/1839141717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d123240 0x556b8d1256d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.559+0000 7f9a927fc640 1 -- 172.21.15.29:0/1839141717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 msgr2=0x556b8d11db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:38.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.559+0000 7f9a927fc640 1 --2- 172.21.15.29:0/1839141717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 0x556b8d11db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.559+0000 7f9a927fc640 1 -- 172.21.15.29:0/1839141717 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b8d040570 con 0x556b8d11e0d0 2024-03-11T19:11:38.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.559+0000 7f9a927fc640 1 --2- 172.21.15.29:0/1839141717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11e0d0 0x556b8d122bd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9a88009970 tx=0x7f9a88009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fd8d30348a87de4 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:38.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.560+0000 7f9a937fe640 1 --2- 172.21.15.29:0/174500746 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a7c0a1ab0 0x7f9a7c0a3f70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c02c630 tx=0x7f9a8c0325d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:38.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.560+0000 7f9a797fa640 1 -- 172.21.15.29:0/1839141717 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8800ea80 con 0x556b8d11e0d0 2024-03-11T19:11:38.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.560+0000 7f9a797fa640 1 -- 172.21.15.29:0/1839141717 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9a8800ebe0 con 0x556b8d11e0d0 2024-03-11T19:11:38.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.560+0000 7f9a797fa640 1 -- 172.21.15.29:0/1839141717 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a88012630 con 0x556b8d11e0d0 2024-03-11T19:11:38.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.561+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/1839141717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11e0d0 msgr2=0x556b8d122bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:38.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.561+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/1839141717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11e0d0 0x556b8d122bd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9a88009970 tx=0x7f9a88009e30 comp rx=0 tx=0).stop 2024-03-11T19:11:38.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.561+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/1839141717 shutdown_connections 2024-03-11T19:11:38.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.561+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/1839141717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d123240 0x556b8d1256d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.561+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/1839141717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11e0d0 0x556b8d122bd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.561+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/1839141717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 0x556b8d11db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.561+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/1839141717 >> 172.21.15.29:0/1839141717 conn(0x556b8d118c00 msgr2=0x556b8d119000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:38.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.561+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/1839141717 shutdown_connections 2024-03-11T19:11:38.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.561+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/1839141717 wait complete. 2024-03-11T19:11:38.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.561+0000 7f9a9bfd4740 1 -- start start 2024-03-11T19:11:38.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.561+0000 7f9a9bfd4740 1 Processor -- start 2024-03-11T19:11:38.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.562+0000 7f9a9bfd4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11d6e0 0x556b8d1276a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.562+0000 7f9a9bfd4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d11e0d0 0x556b8d127be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.562+0000 7f9a9bfd4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d123240 0x556b8d12ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.562+0000 7f9a9bfd4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b8cd55430 con 0x556b8d123240 2024-03-11T19:11:38.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.562+0000 7f9a9bfd4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b8cd11e00 con 0x556b8d11e0d0 2024-03-11T19:11:38.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.562+0000 7f9a9bfd4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b8d006fd0 con 0x556b8d11d6e0 2024-03-11T19:11:38.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.562+0000 7f9a9bfd4740 10 client.0 _ll_register_callbacks cb 0x556b8d05cfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:38.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.562+0000 7f9a937fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d123240 0x556b8d12ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.562+0000 7f9a937fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d123240 0x556b8d12ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45936/0 (socket says 172.21.15.29:45936) 2024-03-11T19:11:38.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.562+0000 7f9a937fe640 1 -- 172.21.15.29:0/696757684 learned_addr learned my addr 172.21.15.29:0/696757684 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:38.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.562+0000 7f9a92ffd640 1 --2- 172.21.15.29:0/696757684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11d6e0 0x556b8d1276a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.562+0000 7f9a927fc640 1 --2- 172.21.15.29:0/696757684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d11e0d0 0x556b8d127be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.562+0000 7f9a92ffd640 1 -- 172.21.15.29:0/696757684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d11e0d0 msgr2=0x556b8d127be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:38.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.563+0000 7f9a92ffd640 1 --2- 172.21.15.29:0/696757684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d11e0d0 0x556b8d127be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.563+0000 7f9a92ffd640 1 -- 172.21.15.29:0/696757684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d123240 msgr2=0x556b8d12ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:38.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.563+0000 7f9a92ffd640 1 --2- 172.21.15.29:0/696757684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d123240 0x556b8d12ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.563+0000 7f9a92ffd640 1 -- 172.21.15.29:0/696757684 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a880095d0 con 0x556b8d11d6e0 2024-03-11T19:11:38.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.563+0000 7f9a937fe640 1 --2- 172.21.15.29:0/696757684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d123240 0x556b8d12ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:38.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.563+0000 7f9a927fc640 1 --2- 172.21.15.29:0/696757684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d11e0d0 0x556b8d127be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:38.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.563+0000 7f9a92ffd640 1 --2- 172.21.15.29:0/696757684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11d6e0 0x556b8d1276a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9a840cb430 tx=0x7f9a8409b060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cce20b796539e045 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:38.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.563+0000 7f9a76ff5640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:11:38.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.563+0000 7f9a76ff5640 1 -- 172.21.15.29:0/696757684 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8406d1d0 con 0x556b8d11d6e0 2024-03-11T19:11:38.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.563+0000 7f9a76ff5640 1 -- 172.21.15.29:0/696757684 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9a8409a8a0 con 0x556b8d11d6e0 2024-03-11T19:11:38.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.563+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/696757684 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556b8cd817e0 con 0x556b8d11d6e0 2024-03-11T19:11:38.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.564+0000 7f9a76ff5640 1 -- 172.21.15.29:0/696757684 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a8409e9f0 con 0x556b8d11d6e0 2024-03-11T19:11:38.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.564+0000 7f9a9bfd4740 10 client.25408 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:38.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.564+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/696757684 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x556b8d080610 con 0x556b8d11d6e0 2024-03-11T19:11:38.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.564+0000 7f9a76ff5640 1 -- 172.21.15.29:0/696757684 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9a840a3040 con 0x556b8d11d6e0 2024-03-11T19:11:38.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.564+0000 7f9a76ff5640 1 -- 172.21.15.29:0/696757684 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9a8409ab00 con 0x556b8d11d6e0 2024-03-11T19:11:38.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.564+0000 7f9a9bfd4740 10 client.25408 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:38.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.564+0000 7f9a9bfd4740 10 client.25408 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:11:38.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.564+0000 7f9a9bfd4740 20 client.25408 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:38.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.564+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/696757684 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x556b8d137bf0 0x556b8d13a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.564+0000 7f9a9bfd4740 4 client.25408 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:11:38.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.564+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/696757684 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556b8cd817e0 con 0x556b8d137bf0 2024-03-11T19:11:38.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.565+0000 7f9a927fc640 1 --2- 172.21.15.29:0/696757684 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x556b8d137bf0 0x556b8d13a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.566+0000 7f9a927fc640 1 --2- 172.21.15.29:0/696757684 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x556b8d137bf0 0x556b8d13a080 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf0e525e0e4cfbfc server_cookie=452cbf2764c183d3 in_seq=0 out_seq=0 2024-03-11T19:11:38.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.566+0000 7f9a76ff5640 10 client.25408 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:38.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.567+0000 7f9a76ff5640 1 -- 172.21.15.29:0/696757684 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556b8cd817e0 con 0x556b8d137bf0 2024-03-11T19:11:38.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.567+0000 7f9a76ff5640 10 client.25408 handle_command_reply: tid=0 2024-03-11T19:11:38.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.567+0000 7f9a9bfd4740 1 client.25408 shutdown 2024-03-11T19:11:38.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.567+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/696757684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11d6e0 msgr2=0x556b8d1276a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:38.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.567+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/696757684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11d6e0 0x556b8d1276a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9a840cb430 tx=0x7f9a8409b060 comp rx=0 tx=0).stop 2024-03-11T19:11:38.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.567+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/696757684 shutdown_connections 2024-03-11T19:11:38.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.567+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/696757684 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x556b8d137bf0 0x556b8d13a080 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.567+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/696757684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d123240 0x556b8d12ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.567+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/696757684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d11e0d0 0x556b8d127be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.567+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/696757684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11d6e0 0x556b8d1276a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.567+0000 7f9a76ff5640 0 client.25408 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:38.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.567+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/696757684 >> 172.21.15.29:0/696757684 conn(0x556b8d118c00 msgr2=0x556b8d113a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:38.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.567+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/696757684 shutdown_connections 2024-03-11T19:11:38.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.567+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/696757684 wait complete. 2024-03-11T19:11:38.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.567+0000 7f9a9bfd4740 20 client.25408 trim_cache size 0 max 16384 2024-03-11T19:11:38.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.596+0000 7f9a9bfd4740 1 Processor -- start 2024-03-11T19:11:38.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.596+0000 7f9a9bfd4740 1 -- start start 2024-03-11T19:11:38.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.596+0000 7f9a9bfd4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 0x556b8d11f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.596+0000 7f9a9bfd4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11e0d0 0x556b8d12ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.596+0000 7f9a9bfd4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d123240 0x556b8d11f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.596+0000 7f9a9bfd4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b8d125ab0 con 0x556b8d11d6e0 2024-03-11T19:11:38.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.596+0000 7f9a9bfd4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b8d11e500 con 0x556b8d123240 2024-03-11T19:11:38.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.597+0000 7f9a9bfd4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b8d110320 con 0x556b8d11e0d0 2024-03-11T19:11:38.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.597+0000 7f9a92ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 0x556b8d11f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.597+0000 7f9a92ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 0x556b8d11f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45952/0 (socket says 172.21.15.29:45952) 2024-03-11T19:11:38.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.597+0000 7f9a92ffd640 1 -- 172.21.15.29:0/1047576798 learned_addr learned my addr 172.21.15.29:0/1047576798 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:38.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.597+0000 7f9a927fc640 1 --2- 172.21.15.29:0/1047576798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11e0d0 0x556b8d12ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.597+0000 7f9a937fe640 1 --2- 172.21.15.29:0/1047576798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d123240 0x556b8d11f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.597+0000 7f9a92ffd640 1 -- 172.21.15.29:0/1047576798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11e0d0 msgr2=0x556b8d12ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:38.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.597+0000 7f9a92ffd640 1 --2- 172.21.15.29:0/1047576798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11e0d0 0x556b8d12ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.597+0000 7f9a92ffd640 1 -- 172.21.15.29:0/1047576798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d123240 msgr2=0x556b8d11f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:38.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.597+0000 7f9a92ffd640 1 --2- 172.21.15.29:0/1047576798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d123240 0x556b8d11f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.597+0000 7f9a92ffd640 1 -- 172.21.15.29:0/1047576798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b8d01cac0 con 0x556b8d11d6e0 2024-03-11T19:11:38.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.597+0000 7f9a937fe640 1 --2- 172.21.15.29:0/1047576798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d123240 0x556b8d11f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:38.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.597+0000 7f9a92ffd640 1 --2- 172.21.15.29:0/1047576798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 0x556b8d11f410 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9a84000c00 tx=0x7f9a84096c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52361e3bf51ecc45 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:38.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.598+0000 7f9a78ff9640 1 -- 172.21.15.29:0/1047576798 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8409a8a0 con 0x556b8d11d6e0 2024-03-11T19:11:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.598+0000 7f9a78ff9640 1 -- 172.21.15.29:0/1047576798 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9a8409e9f0 con 0x556b8d11d6e0 2024-03-11T19:11:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.598+0000 7f9a78ff9640 1 -- 172.21.15.29:0/1047576798 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a840a2bb0 con 0x556b8d11d6e0 2024-03-11T19:11:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.599+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/1047576798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 msgr2=0x556b8d11f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.599+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/1047576798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 0x556b8d11f410 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9a84000c00 tx=0x7f9a84096c70 comp rx=0 tx=0).stop 2024-03-11T19:11:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.599+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/1047576798 shutdown_connections 2024-03-11T19:11:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.599+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/1047576798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d123240 0x556b8d11f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.599+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/1047576798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11e0d0 0x556b8d12ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.599+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/1047576798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 0x556b8d11f410 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.599+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/1047576798 >> 172.21.15.29:0/1047576798 conn(0x556b8d118c00 msgr2=0x556b8d13e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.599+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/1047576798 shutdown_connections 2024-03-11T19:11:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.599+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/1047576798 wait complete. 2024-03-11T19:11:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.599+0000 7f9a9bfd4740 1 -- start start 2024-03-11T19:11:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.599+0000 7f9a9bfd4740 1 Processor -- start 2024-03-11T19:11:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.600+0000 7f9a9bfd4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 0x556b8d12e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.600+0000 7f9a9bfd4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11e0d0 0x556b8d12eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.600+0000 7f9a9bfd4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d123240 0x556b8d12cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.600+0000 7f9a9bfd4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b8d12d2c0 con 0x556b8d11d6e0 2024-03-11T19:11:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.600+0000 7f9a9bfd4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b8d12d430 con 0x556b8d123240 2024-03-11T19:11:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.600+0000 7f9a9bfd4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b8d12d5f0 con 0x556b8d11e0d0 2024-03-11T19:11:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.600+0000 7f9a92ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 0x556b8d12e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.600+0000 7f9a9bfd4740 10 client.0 _ll_register_callbacks cb 0x7f9a880093c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.600+0000 7f9a92ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 0x556b8d12e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45962/0 (socket says 172.21.15.29:45962) 2024-03-11T19:11:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.600+0000 7f9a92ffd640 1 -- 172.21.15.29:0/3672229851 learned_addr learned my addr 172.21.15.29:0/3672229851 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.600+0000 7f9a927fc640 1 --2- 172.21.15.29:0/3672229851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11e0d0 0x556b8d12eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.600+0000 7f9a937fe640 1 --2- 172.21.15.29:0/3672229851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d123240 0x556b8d12cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.600+0000 7f9a92ffd640 1 -- 172.21.15.29:0/3672229851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11e0d0 msgr2=0x556b8d12eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.600+0000 7f9a92ffd640 1 --2- 172.21.15.29:0/3672229851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11e0d0 0x556b8d12eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.600+0000 7f9a92ffd640 1 -- 172.21.15.29:0/3672229851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d123240 msgr2=0x556b8d12cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.600+0000 7f9a92ffd640 1 --2- 172.21.15.29:0/3672229851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d123240 0x556b8d12cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.600+0000 7f9a92ffd640 1 -- 172.21.15.29:0/3672229851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b8d080610 con 0x556b8d11d6e0 2024-03-11T19:11:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.600+0000 7f9a927fc640 1 --2- 172.21.15.29:0/3672229851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11e0d0 0x556b8d12eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.601+0000 7f9a92ffd640 1 --2- 172.21.15.29:0/3672229851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 0x556b8d12e5b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9a840a2b10 tx=0x7f9a8400cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f7caa16e5155ec9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.601+0000 7f9a76ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.601+0000 7f9a76ff5640 1 -- 172.21.15.29:0/3672229851 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8409a8a0 con 0x556b8d11d6e0 2024-03-11T19:11:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.601+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/3672229851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9a8800ebe0 con 0x556b8d11d6e0 2024-03-11T19:11:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.601+0000 7f9a76ff5640 1 -- 172.21.15.29:0/3672229851 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9a840a3040 con 0x556b8d11d6e0 2024-03-11T19:11:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.601+0000 7f9a76ff5640 1 -- 172.21.15.29:0/3672229851 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a8406c100 con 0x556b8d11d6e0 2024-03-11T19:11:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.601+0000 7f9a9bfd4740 10 client.15810 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.601+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/3672229851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x556b8d0426a0 con 0x556b8d11d6e0 2024-03-11T19:11:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.601+0000 7f9a76ff5640 1 -- 172.21.15.29:0/3672229851 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9a8800ebe0 con 0x556b8d11d6e0 2024-03-11T19:11:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.602+0000 7f9a76ff5640 1 -- 172.21.15.29:0/3672229851 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9a8406d410 con 0x556b8d11d6e0 2024-03-11T19:11:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.602+0000 7f9a9bfd4740 10 client.15810 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.602+0000 7f9a9bfd4740 10 client.15810 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:11:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.602+0000 7f9a9bfd4740 20 client.15810 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.602+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/3672229851 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x556b8d18f1c0 0x556b8d1915e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.602+0000 7f9a9bfd4740 4 client.15810 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:11:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.602+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/3672229851 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556b8cdbe720 con 0x556b8d18f1c0 2024-03-11T19:11:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.602+0000 7f9a927fc640 1 --2- 172.21.15.29:0/3672229851 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x556b8d18f1c0 0x556b8d1915e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.603+0000 7f9a927fc640 1 --2- 172.21.15.29:0/3672229851 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x556b8d18f1c0 0x556b8d1915e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da526d9249e91448 server_cookie=edf1687b50432c68 in_seq=0 out_seq=0 2024-03-11T19:11:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.603+0000 7f9a76ff5640 10 client.15810 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.603+0000 7f9a76ff5640 1 -- 172.21.15.29:0/3672229851 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x556b8cdbe720 con 0x556b8d18f1c0 2024-03-11T19:11:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.603+0000 7f9a76ff5640 10 client.15810 handle_command_reply: tid=0 2024-03-11T19:11:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.603+0000 7f9a9bfd4740 1 client.15810 shutdown 2024-03-11T19:11:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.603+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/3672229851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 msgr2=0x556b8d12e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.603+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/3672229851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 0x556b8d12e5b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9a840a2b10 tx=0x7f9a8400cee0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.603+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/3672229851 shutdown_connections 2024-03-11T19:11:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.603+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/3672229851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b8d123240 0x556b8d12cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.603+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/3672229851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b8d11e0d0 0x556b8d12eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.603+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/3672229851 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x556b8d18f1c0 0x556b8d1915e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.603+0000 7f9a9bfd4740 1 --2- 172.21.15.29:0/3672229851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b8d11d6e0 0x556b8d12e5b0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.604+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/3672229851 >> 172.21.15.29:0/3672229851 conn(0x556b8d118c00 msgr2=0x556b8d11f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:38.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.604+0000 7f9a76ff5640 0 client.15810 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:38.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.604+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/3672229851 shutdown_connections 2024-03-11T19:11:38.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.604+0000 7f9a9bfd4740 1 -- 172.21.15.29:0/3672229851 wait complete. 2024-03-11T19:11:38.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.604+0000 7f9a9bfd4740 20 client.15810 trim_cache size 0 max 16384 2024-03-11T19:11:38.610 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:11:38.610 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:11:38.611 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:11:38.611 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:11:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.605+0000 7f9a99325640 1 -- 172.21.15.29:0/174500746 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a7c0a1ab0 msgr2=0x7f9a7c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.605+0000 7f9a99325640 1 --2- 172.21.15.29:0/174500746 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a7c0a1ab0 0x7f9a7c0a3f70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c02c630 tx=0x7f9a8c0325d0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.605+0000 7f9a99325640 1 -- 172.21.15.29:0/174500746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a94100080 msgr2=0x7f9a94197170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.605+0000 7f9a99325640 1 --2- 172.21.15.29:0/174500746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a94100080 0x7f9a94197170 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f9a8400e9c0 tx=0x7f9a8400edc0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.605+0000 7f9a99325640 1 -- 172.21.15.29:0/174500746 shutdown_connections 2024-03-11T19:11:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.605+0000 7f9a99325640 1 --2- 172.21.15.29:0/174500746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a94101c30 0x7f9a9419e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.605+0000 7f9a99325640 1 --2- 172.21.15.29:0/174500746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a94101270 0x7f9a941976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.605+0000 7f9a99325640 1 --2- 172.21.15.29:0/174500746 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a7c0a1ab0 0x7f9a7c0a3f70 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.605+0000 7f9a99325640 1 --2- 172.21.15.29:0/174500746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a94100080 0x7f9a94197170 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.605+0000 7f9a99325640 1 -- 172.21.15.29:0/174500746 >> 172.21.15.29:0/174500746 conn(0x7f9a940fb5f0 msgr2=0x7f9a940fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.605+0000 7f9a99325640 1 -- 172.21.15.29:0/174500746 shutdown_connections 2024-03-11T19:11:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.605+0000 7f9a99325640 1 -- 172.21.15.29:0/174500746 wait complete. 2024-03-11T19:11:38.623 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:11:38.623 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:11:38.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.771+0000 7ff42baae640 1 -- 172.21.15.29:0/1751908639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff424104550 msgr2=0x7ff424106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:38.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.771+0000 7ff42baae640 1 --2- 172.21.15.29:0/1751908639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff424104550 0x7ff424106940 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff414009900 tx=0x7ff41402bc30 comp rx=0 tx=0).stop 2024-03-11T19:11:38.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.771+0000 7ff42baae640 1 -- 172.21.15.29:0/1751908639 shutdown_connections 2024-03-11T19:11:38.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.771+0000 7ff42baae640 1 --2- 172.21.15.29:0/1751908639 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff424106e80 0x7ff4241092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.771+0000 7ff42baae640 1 --2- 172.21.15.29:0/1751908639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff424104550 0x7ff424106940 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.771+0000 7ff42baae640 1 --2- 172.21.15.29:0/1751908639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff424101c20 0x7ff424104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.771+0000 7ff42baae640 1 -- 172.21.15.29:0/1751908639 >> 172.21.15.29:0/1751908639 conn(0x7ff4240fb630 msgr2=0x7ff4240fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:38.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.771+0000 7ff42baae640 1 -- 172.21.15.29:0/1751908639 shutdown_connections 2024-03-11T19:11:38.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.771+0000 7ff42baae640 1 -- 172.21.15.29:0/1751908639 wait complete. 2024-03-11T19:11:38.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.772+0000 7ff42baae640 1 Processor -- start 2024-03-11T19:11:38.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.772+0000 7ff42baae640 1 -- start start 2024-03-11T19:11:38.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.772+0000 7ff42baae640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff424101c20 0x7ff424194fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.772+0000 7ff42baae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff424104550 0x7ff424195510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.772+0000 7ff42baae640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff424106e80 0x7ff42419c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.772+0000 7ff42baae640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42419cac0 con 0x7ff424104550 2024-03-11T19:11:38.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.772+0000 7ff42baae640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42419cc30 con 0x7ff424106e80 2024-03-11T19:11:38.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.772+0000 7ff42baae640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42419cdf0 con 0x7ff424101c20 2024-03-11T19:11:38.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.772+0000 7ff429049640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff424104550 0x7ff424195510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.773+0000 7ff429049640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff424104550 0x7ff424195510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45986/0 (socket says 172.21.15.29:45986) 2024-03-11T19:11:38.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.773+0000 7ff429049640 1 -- 172.21.15.29:0/3440139609 learned_addr learned my addr 172.21.15.29:0/3440139609 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:38.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.773+0000 7ff429049640 1 -- 172.21.15.29:0/3440139609 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff424101c20 msgr2=0x7ff424194fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-11T19:11:38.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.773+0000 7ff429049640 1 --2- 172.21.15.29:0/3440139609 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff424101c20 0x7ff424194fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.773+0000 7ff429049640 1 -- 172.21.15.29:0/3440139609 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff424106e80 msgr2=0x7ff42419c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:38.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.773+0000 7ff42a04b640 1 --2- 172.21.15.29:0/3440139609 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff424106e80 0x7ff42419c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.773+0000 7ff429049640 1 --2- 172.21.15.29:0/3440139609 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff424106e80 0x7ff42419c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:38.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.773+0000 7ff429049640 1 -- 172.21.15.29:0/3440139609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff414009560 con 0x7ff424104550 2024-03-11T19:11:38.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.773+0000 7ff42a04b640 1 --2- 172.21.15.29:0/3440139609 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff424106e80 0x7ff42419c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:11:38.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.773+0000 7ff429049640 1 --2- 172.21.15.29:0/3440139609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff424104550 0x7ff424195510 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff414000c00 tx=0x7ff414004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:38.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.774+0000 7ff41affd640 1 -- 172.21.15.29:0/3440139609 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414004500 con 0x7ff424104550 2024-03-11T19:11:38.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.774+0000 7ff42baae640 1 -- 172.21.15.29:0/3440139609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42419d070 con 0x7ff424104550 2024-03-11T19:11:38.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.774+0000 7ff42baae640 1 -- 172.21.15.29:0/3440139609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42419d5d0 con 0x7ff424104550 2024-03-11T19:11:38.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.774+0000 7ff41affd640 1 -- 172.21.15.29:0/3440139609 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff414003aa0 con 0x7ff424104550 2024-03-11T19:11:38.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.774+0000 7ff41affd640 1 -- 172.21.15.29:0/3440139609 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414005430 con 0x7ff424104550 2024-03-11T19:11:38.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.775+0000 7ff42baae640 1 -- 172.21.15.29:0/3440139609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff408005290 con 0x7ff424104550 2024-03-11T19:11:38.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.776+0000 7ff41affd640 1 -- 172.21.15.29:0/3440139609 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff414003c10 con 0x7ff424104550 2024-03-11T19:11:38.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.777+0000 7ff41affd640 1 --2- 172.21.15.29:0/3440139609 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff4100a1ab0 0x7ff4100a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:38.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.777+0000 7ff42984a640 1 --2- 172.21.15.29:0/3440139609 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff4100a1ab0 0x7ff4100a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:38.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.778+0000 7ff41affd640 1 -- 172.21.15.29:0/3440139609 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff41402de40 con 0x7ff424104550 2024-03-11T19:11:38.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.778+0000 7ff42984a640 1 --2- 172.21.15.29:0/3440139609 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff4100a1ab0 0x7ff4100a3f70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff420009a60 tx=0x7ff420005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:38.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:38.786+0000 7ff41affd640 1 -- 172.21.15.29:0/3440139609 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff4140b8dc0 con 0x7ff424104550 2024-03-11T19:11:39.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:38 smithi170 ceph-mon[35690]: pgmap v406: 97 pgs: 97 active+clean; 3.8 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 166 MiB/s wr, 95 op/s 2024-03-11T19:11:39.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:38 smithi029 ceph-mon[32439]: pgmap v406: 97 pgs: 97 active+clean; 3.8 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 166 MiB/s wr, 95 op/s 2024-03-11T19:11:39.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.176+0000 7ff42baae640 1 -- 172.21.15.29:0/3440139609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff408005810 con 0x7ff424104550 2024-03-11T19:11:39.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.177+0000 7ff41affd640 1 -- 172.21.15.29:0/3440139609 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7ff414043090 con 0x7ff424104550 2024-03-11T19:11:39.180 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:11:39.180 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:11:39.180 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:11:39.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.179+0000 7ff42baae640 1 -- 172.21.15.29:0/3440139609 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff4100a1ab0 msgr2=0x7ff4100a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.179+0000 7ff42baae640 1 --2- 172.21.15.29:0/3440139609 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff4100a1ab0 0x7ff4100a3f70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff420009a60 tx=0x7ff420005f90 comp rx=0 tx=0).stop 2024-03-11T19:11:39.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.179+0000 7ff42baae640 1 -- 172.21.15.29:0/3440139609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff424104550 msgr2=0x7ff424195510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.179+0000 7ff42baae640 1 --2- 172.21.15.29:0/3440139609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff424104550 0x7ff424195510 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff414000c00 tx=0x7ff414004040 comp rx=0 tx=0).stop 2024-03-11T19:11:39.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.179+0000 7ff42baae640 1 -- 172.21.15.29:0/3440139609 shutdown_connections 2024-03-11T19:11:39.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.179+0000 7ff42baae640 1 --2- 172.21.15.29:0/3440139609 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff424106e80 0x7ff42419c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.179+0000 7ff42baae640 1 --2- 172.21.15.29:0/3440139609 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff4100a1ab0 0x7ff4100a3f70 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.179+0000 7ff42baae640 1 --2- 172.21.15.29:0/3440139609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff424104550 0x7ff424195510 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.179+0000 7ff42baae640 1 --2- 172.21.15.29:0/3440139609 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff424101c20 0x7ff424194fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.179+0000 7ff42baae640 1 -- 172.21.15.29:0/3440139609 >> 172.21.15.29:0/3440139609 conn(0x7ff4240fb630 msgr2=0x7ff4241014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.179+0000 7ff42baae640 1 -- 172.21.15.29:0/3440139609 shutdown_connections 2024-03-11T19:11:39.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.179+0000 7ff42baae640 1 -- 172.21.15.29:0/3440139609 wait complete. 2024-03-11T19:11:39.197 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:11:39.197 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:11:39.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:38 smithi190 ceph-mon[36292]: pgmap v406: 97 pgs: 97 active+clean; 3.8 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 166 MiB/s wr, 95 op/s 2024-03-11T19:11:39.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.347+0000 7f5e6452c640 1 -- 172.21.15.29:0/1866520316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e5c1023a0 msgr2=0x7f5e5c102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.347+0000 7f5e6452c640 1 --2- 172.21.15.29:0/1866520316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e5c1023a0 0x7f5e5c102820 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009900 tx=0x7f5e4c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:11:39.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.347+0000 7f5e6452c640 1 -- 172.21.15.29:0/1866520316 shutdown_connections 2024-03-11T19:11:39.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.347+0000 7f5e6452c640 1 --2- 172.21.15.29:0/1866520316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e5c102d60 0x7f5e5c1095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.347+0000 7f5e6452c640 1 --2- 172.21.15.29:0/1866520316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e5c1023a0 0x7f5e5c102820 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.347+0000 7f5e6452c640 1 --2- 172.21.15.29:0/1866520316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e5c1011b0 0x7f5e5c1015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.347+0000 7f5e6452c640 1 -- 172.21.15.29:0/1866520316 >> 172.21.15.29:0/1866520316 conn(0x7f5e5c0fc720 msgr2=0x7f5e5c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.347+0000 7f5e6452c640 1 -- 172.21.15.29:0/1866520316 shutdown_connections 2024-03-11T19:11:39.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.347+0000 7f5e6452c640 1 -- 172.21.15.29:0/1866520316 wait complete. 2024-03-11T19:11:39.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.348+0000 7f5e6452c640 1 Processor -- start 2024-03-11T19:11:39.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.348+0000 7f5e6452c640 1 -- start start 2024-03-11T19:11:39.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.348+0000 7f5e6452c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e5c1011b0 0x7f5e5c197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.348+0000 7f5e6452c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e5c102d60 0x7f5e5c197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.348+0000 7f5e6452c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e5c19c8e0 0x7f5e5c19ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.348+0000 7f5e6452c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c19f210 con 0x7f5e5c1011b0 2024-03-11T19:11:39.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.348+0000 7f5e6452c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c19f380 con 0x7f5e5c19c8e0 2024-03-11T19:11:39.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.348+0000 7f5e6452c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c19f540 con 0x7f5e5c102d60 2024-03-11T19:11:39.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.348+0000 7f5e622c8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e5c1011b0 0x7f5e5c197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.348+0000 7f5e622c8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e5c1011b0 0x7f5e5c197340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46000/0 (socket says 172.21.15.29:46000) 2024-03-11T19:11:39.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.349+0000 7f5e622c8640 1 -- 172.21.15.29:0/2526043630 learned_addr learned my addr 172.21.15.29:0/2526043630 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.349+0000 7f5e61ac7640 1 --2- 172.21.15.29:0/2526043630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e5c102d60 0x7f5e5c197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.349+0000 7f5e622c8640 1 -- 172.21.15.29:0/2526043630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e5c102d60 msgr2=0x7f5e5c197880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.349+0000 7f5e622c8640 1 --2- 172.21.15.29:0/2526043630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e5c102d60 0x7f5e5c197880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.349+0000 7f5e622c8640 1 -- 172.21.15.29:0/2526043630 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e5c19c8e0 msgr2=0x7f5e5c19ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.349+0000 7f5e622c8640 1 --2- 172.21.15.29:0/2526043630 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e5c19c8e0 0x7f5e5c19ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.349+0000 7f5e622c8640 1 -- 172.21.15.29:0/2526043630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4c009560 con 0x7f5e5c1011b0 2024-03-11T19:11:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.349+0000 7f5e622c8640 1 --2- 172.21.15.29:0/2526043630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e5c1011b0 0x7f5e5c197340 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f5e5800d770 tx=0x7f5e5800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.349+0000 7f5e537fe640 1 -- 172.21.15.29:0/2526043630 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e58009c30 con 0x7f5e5c1011b0 2024-03-11T19:11:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.350+0000 7f5e537fe640 1 -- 172.21.15.29:0/2526043630 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5e580044d0 con 0x7f5e5c1011b0 2024-03-11T19:11:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.350+0000 7f5e6452c640 1 -- 172.21.15.29:0/2526043630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c19f820 con 0x7f5e5c1011b0 2024-03-11T19:11:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.350+0000 7f5e537fe640 1 -- 172.21.15.29:0/2526043630 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e58003de0 con 0x7f5e5c1011b0 2024-03-11T19:11:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.350+0000 7f5e6452c640 1 -- 172.21.15.29:0/2526043630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c19fd50 con 0x7f5e5c1011b0 2024-03-11T19:11:39.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.352+0000 7f5e537fe640 1 -- 172.21.15.29:0/2526043630 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5e58003f40 con 0x7f5e5c1011b0 2024-03-11T19:11:39.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.353+0000 7f5e537fe640 1 --2- 172.21.15.29:0/2526043630 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5e480a1ab0 0x7f5e480a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.353+0000 7f5e671d8740 1 Processor -- start 2024-03-11T19:11:39.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.353+0000 7f5e537fe640 1 -- 172.21.15.29:0/2526043630 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5e580ccd10 con 0x7f5e5c1011b0 2024-03-11T19:11:39.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.353+0000 7f5e61ac7640 1 --2- 172.21.15.29:0/2526043630 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5e480a1ab0 0x7f5e480a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.354+0000 7f5e671d8740 1 -- start start 2024-03-11T19:11:39.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.354+0000 7f5e671d8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902aff270 0x557902aff690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.354+0000 7f5e671d8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902affc60 0x557902b04760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.354+0000 7f5e671d8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902b04dd0 0x557902b07260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.354+0000 7f5e671d8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557902abb960 con 0x557902aff270 2024-03-11T19:11:39.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.354+0000 7f5e671d8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557902a6f310 con 0x557902affc60 2024-03-11T19:11:39.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.354+0000 7f5e671d8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557902a6c840 con 0x557902b04dd0 2024-03-11T19:11:39.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.354+0000 7f5e622c8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902aff270 0x557902aff690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.354+0000 7f5e622c8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902aff270 0x557902aff690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46012/0 (socket says 172.21.15.29:46012) 2024-03-11T19:11:39.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.354+0000 7f5e622c8640 1 -- 172.21.15.29:0/420548614 learned_addr learned my addr 172.21.15.29:0/420548614 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:39.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.354+0000 7f5e61ac7640 1 --2- 172.21.15.29:0/2526043630 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5e480a1ab0 0x7f5e480a3f70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c036040 tx=0x7f5e4c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.354+0000 7f5e61ac7640 1 --2- 172.21.15.29:0/420548614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902affc60 0x557902b04760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.354+0000 7f5e62ac9640 1 --2- 172.21.15.29:0/420548614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902b04dd0 0x557902b07260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.354+0000 7f5e622c8640 1 -- 172.21.15.29:0/420548614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902b04dd0 msgr2=0x557902b07260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.354+0000 7f5e622c8640 1 --2- 172.21.15.29:0/420548614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902b04dd0 0x557902b07260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.354+0000 7f5e622c8640 1 -- 172.21.15.29:0/420548614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902affc60 msgr2=0x557902b04760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.354+0000 7f5e622c8640 1 --2- 172.21.15.29:0/420548614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902affc60 0x557902b04760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.354+0000 7f5e622c8640 1 -- 172.21.15.29:0/420548614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557902a2a060 con 0x557902aff270 2024-03-11T19:11:39.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.355+0000 7f5e622c8640 1 --2- 172.21.15.29:0/420548614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902aff270 0x557902aff690 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f5e58096180 tx=0x7f5e58099bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a454bcb55d11e112 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.355+0000 7f5e50bfe640 1 -- 172.21.15.29:0/420548614 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5809a420 con 0x557902aff270 2024-03-11T19:11:39.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.355+0000 7f5e50bfe640 1 -- 172.21.15.29:0/420548614 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5e58099d20 con 0x557902aff270 2024-03-11T19:11:39.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.355+0000 7f5e50bfe640 1 -- 172.21.15.29:0/420548614 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e58096390 con 0x557902aff270 2024-03-11T19:11:39.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.356+0000 7f5e671d8740 1 -- 172.21.15.29:0/420548614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902aff270 msgr2=0x557902aff690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.356+0000 7f5e671d8740 1 --2- 172.21.15.29:0/420548614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902aff270 0x557902aff690 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f5e58096180 tx=0x7f5e58099bf0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.356+0000 7f5e671d8740 1 -- 172.21.15.29:0/420548614 shutdown_connections 2024-03-11T19:11:39.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.356+0000 7f5e671d8740 1 --2- 172.21.15.29:0/420548614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902b04dd0 0x557902b07260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.356+0000 7f5e671d8740 1 --2- 172.21.15.29:0/420548614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902affc60 0x557902b04760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.356+0000 7f5e671d8740 1 --2- 172.21.15.29:0/420548614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902aff270 0x557902aff690 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.356+0000 7f5e671d8740 1 -- 172.21.15.29:0/420548614 >> 172.21.15.29:0/420548614 conn(0x557902afa7e0 msgr2=0x557902afabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.356+0000 7f5e671d8740 1 -- 172.21.15.29:0/420548614 shutdown_connections 2024-03-11T19:11:39.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.356+0000 7f5e671d8740 1 -- 172.21.15.29:0/420548614 wait complete. 2024-03-11T19:11:39.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.357+0000 7f5e671d8740 1 -- start start 2024-03-11T19:11:39.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.357+0000 7f5e671d8740 1 Processor -- start 2024-03-11T19:11:39.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.357+0000 7f5e671d8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902aff270 0x557902b0d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.357+0000 7f5e671d8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902affc60 0x557902b0dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.357+0000 7f5e671d8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902b04dd0 0x557902b14d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.357+0000 7f5e671d8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557902a68020 con 0x557902affc60 2024-03-11T19:11:39.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.357+0000 7f5e671d8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55790273b430 con 0x557902aff270 2024-03-11T19:11:39.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.357+0000 7f5e671d8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579026f7e00 con 0x557902b04dd0 2024-03-11T19:11:39.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.357+0000 7f5e671d8740 10 client.0 _ll_register_callbacks cb 0x557902a0fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:39.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.357+0000 7f5e61ac7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902affc60 0x557902b0dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.357+0000 7f5e61ac7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902affc60 0x557902b0dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46028/0 (socket says 172.21.15.29:46028) 2024-03-11T19:11:39.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.357+0000 7f5e61ac7640 1 -- 172.21.15.29:0/3058213081 learned_addr learned my addr 172.21.15.29:0/3058213081 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:39.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.358+0000 7f5e62ac9640 1 --2- 172.21.15.29:0/3058213081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902b04dd0 0x557902b14d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.358+0000 7f5e622c8640 1 --2- 172.21.15.29:0/3058213081 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902aff270 0x557902b0d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.358+0000 7f5e61ac7640 1 -- 172.21.15.29:0/3058213081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902b04dd0 msgr2=0x557902b14d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.358+0000 7f5e61ac7640 1 --2- 172.21.15.29:0/3058213081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902b04dd0 0x557902b14d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.358+0000 7f5e61ac7640 1 -- 172.21.15.29:0/3058213081 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902aff270 msgr2=0x557902b0d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.358+0000 7f5e61ac7640 1 --2- 172.21.15.29:0/3058213081 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902aff270 0x557902b0d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.358+0000 7f5e61ac7640 1 -- 172.21.15.29:0/3058213081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c19fd50 con 0x557902affc60 2024-03-11T19:11:39.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.358+0000 7f5e61ac7640 1 --2- 172.21.15.29:0/3058213081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902affc60 0x557902b0dd30 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c005ec0 tx=0x7f5e4c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dd1cee49b856dd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.358+0000 7f5e45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:39.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.358+0000 7f5e45ffb640 1 -- 172.21.15.29:0/3058213081 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c02cbf0 con 0x557902affc60 2024-03-11T19:11:39.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.358+0000 7f5e45ffb640 1 -- 172.21.15.29:0/3058213081 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5e4c046040 con 0x557902affc60 2024-03-11T19:11:39.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.358+0000 7f5e671d8740 1 -- 172.21.15.29:0/3058213081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5579027677e0 con 0x557902affc60 2024-03-11T19:11:39.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.359+0000 7f5e45ffb640 1 -- 172.21.15.29:0/3058213081 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e4c003b40 con 0x557902affc60 2024-03-11T19:11:39.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.359+0000 7f5e671d8740 10 client.15828 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:39.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.359+0000 7f5e671d8740 1 -- 172.21.15.29:0/3058213081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x557902a666a0 con 0x557902affc60 2024-03-11T19:11:39.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.359+0000 7f5e45ffb640 1 -- 172.21.15.29:0/3058213081 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5579027677e0 con 0x557902affc60 2024-03-11T19:11:39.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.359+0000 7f5e45ffb640 1 -- 172.21.15.29:0/3058213081 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5e4c03c040 con 0x557902affc60 2024-03-11T19:11:39.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.359+0000 7f5e671d8740 10 client.15828 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:39.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.359+0000 7f5e671d8740 10 client.15828 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:11:39.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.359+0000 7f5e671d8740 20 client.15828 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:39.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.359+0000 7f5e671d8740 1 --2- 172.21.15.29:0/3058213081 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x557902b19a60 0x557902b1be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.359+0000 7f5e671d8740 4 client.15828 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:11:39.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.359+0000 7f5e671d8740 1 -- 172.21.15.29:0/3058213081 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5579027a4720 con 0x557902b19a60 2024-03-11T19:11:39.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.360+0000 7f5e622c8640 1 --2- 172.21.15.29:0/3058213081 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x557902b19a60 0x557902b1be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.360+0000 7f5e622c8640 1 --2- 172.21.15.29:0/3058213081 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x557902b19a60 0x557902b1be80 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=973a50a60e22a8a6 server_cookie=4c205daa114a65d3 in_seq=0 out_seq=0 2024-03-11T19:11:39.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.360+0000 7f5e45ffb640 10 client.15828 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:11:39.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.361+0000 7f5e45ffb640 1 -- 172.21.15.29:0/3058213081 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5579027a4720 con 0x557902b19a60 2024-03-11T19:11:39.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.361+0000 7f5e45ffb640 10 client.15828 handle_command_reply: tid=0 2024-03-11T19:11:39.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.361+0000 7f5e671d8740 1 client.15828 shutdown 2024-03-11T19:11:39.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.361+0000 7f5e671d8740 1 -- 172.21.15.29:0/3058213081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902affc60 msgr2=0x557902b0dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.361+0000 7f5e671d8740 1 --2- 172.21.15.29:0/3058213081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902affc60 0x557902b0dd30 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c005ec0 tx=0x7f5e4c009260 comp rx=0 tx=0).stop 2024-03-11T19:11:39.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.361+0000 7f5e671d8740 1 -- 172.21.15.29:0/3058213081 shutdown_connections 2024-03-11T19:11:39.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.361+0000 7f5e671d8740 1 --2- 172.21.15.29:0/3058213081 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x557902b19a60 0x557902b1be80 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.361+0000 7f5e671d8740 1 --2- 172.21.15.29:0/3058213081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902b04dd0 0x557902b14d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.361+0000 7f5e671d8740 1 --2- 172.21.15.29:0/3058213081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902affc60 0x557902b0dd30 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.361+0000 7f5e671d8740 1 --2- 172.21.15.29:0/3058213081 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902aff270 0x557902b0d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.361+0000 7f5e671d8740 1 -- 172.21.15.29:0/3058213081 >> 172.21.15.29:0/3058213081 conn(0x557902afa7e0 msgr2=0x557902af9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.361+0000 7f5e45ffb640 0 client.15828 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:11:39.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.361+0000 7f5e671d8740 1 -- 172.21.15.29:0/3058213081 shutdown_connections 2024-03-11T19:11:39.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.361+0000 7f5e671d8740 1 -- 172.21.15.29:0/3058213081 wait complete. 2024-03-11T19:11:39.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.361+0000 7f5e671d8740 20 client.15828 trim_cache size 0 max 16384 2024-03-11T19:11:39.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.380+0000 7f5e671d8740 1 Processor -- start 2024-03-11T19:11:39.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.380+0000 7f5e671d8740 1 -- start start 2024-03-11T19:11:39.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.380+0000 7f5e671d8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902aff270 0x557902b0c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.380+0000 7f5e671d8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902affc60 0x557902b14d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.380+0000 7f5e671d8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902b04dd0 0x557902b0d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.380+0000 7f5e671d8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579029ed130 con 0x557902affc60 2024-03-11T19:11:39.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.380+0000 7f5e671d8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557902767810 con 0x557902b04dd0 2024-03-11T19:11:39.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.380+0000 7f5e671d8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557902aeca40 con 0x557902aff270 2024-03-11T19:11:39.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.380+0000 7f5e61ac7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902affc60 0x557902b14d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.380+0000 7f5e62ac9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902b04dd0 0x557902b0d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.380+0000 7f5e61ac7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902affc60 0x557902b14d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46030/0 (socket says 172.21.15.29:46030) 2024-03-11T19:11:39.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.380+0000 7f5e61ac7640 1 -- 172.21.15.29:0/307636234 learned_addr learned my addr 172.21.15.29:0/307636234 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:39.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.380+0000 7f5e622c8640 1 --2- 172.21.15.29:0/307636234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902aff270 0x557902b0c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.381+0000 7f5e61ac7640 1 -- 172.21.15.29:0/307636234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902aff270 msgr2=0x557902b0c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.381+0000 7f5e61ac7640 1 --2- 172.21.15.29:0/307636234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902aff270 0x557902b0c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.381+0000 7f5e61ac7640 1 -- 172.21.15.29:0/307636234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902b04dd0 msgr2=0x557902b0d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.381+0000 7f5e61ac7640 1 --2- 172.21.15.29:0/307636234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902b04dd0 0x557902b0d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.381+0000 7f5e61ac7640 1 -- 172.21.15.29:0/307636234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557902a02c20 con 0x557902affc60 2024-03-11T19:11:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.381+0000 7f5e62ac9640 1 --2- 172.21.15.29:0/307636234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902b04dd0 0x557902b0d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.381+0000 7f5e622c8640 1 --2- 172.21.15.29:0/307636234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902aff270 0x557902b0c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.381+0000 7f5e61ac7640 1 --2- 172.21.15.29:0/307636234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902affc60 0x557902b14d30 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c02c7d0 tx=0x7f5e4c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1e23b856ff22f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.381+0000 7f5e47fff640 1 -- 172.21.15.29:0/307636234 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c02cb70 con 0x557902affc60 2024-03-11T19:11:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.381+0000 7f5e47fff640 1 -- 172.21.15.29:0/307636234 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5e4c003e70 con 0x557902affc60 2024-03-11T19:11:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.381+0000 7f5e47fff640 1 -- 172.21.15.29:0/307636234 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c034650 con 0x557902affc60 2024-03-11T19:11:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.382+0000 7f5e671d8740 1 -- 172.21.15.29:0/307636234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902affc60 msgr2=0x557902b14d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.382+0000 7f5e671d8740 1 --2- 172.21.15.29:0/307636234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902affc60 0x557902b14d30 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c02c7d0 tx=0x7f5e4c009780 comp rx=0 tx=0).stop 2024-03-11T19:11:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.382+0000 7f5e671d8740 1 -- 172.21.15.29:0/307636234 shutdown_connections 2024-03-11T19:11:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.382+0000 7f5e671d8740 1 --2- 172.21.15.29:0/307636234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902b04dd0 0x557902b0d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.382+0000 7f5e671d8740 1 --2- 172.21.15.29:0/307636234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902affc60 0x557902b14d30 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.382+0000 7f5e671d8740 1 --2- 172.21.15.29:0/307636234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902aff270 0x557902b0c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.382+0000 7f5e671d8740 1 -- 172.21.15.29:0/307636234 >> 172.21.15.29:0/307636234 conn(0x557902afa7e0 msgr2=0x557902af5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.382+0000 7f5e671d8740 1 -- 172.21.15.29:0/307636234 shutdown_connections 2024-03-11T19:11:39.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.382+0000 7f5e671d8740 1 -- 172.21.15.29:0/307636234 wait complete. 2024-03-11T19:11:39.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.382+0000 7f5e671d8740 1 -- start start 2024-03-11T19:11:39.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.383+0000 7f5e671d8740 1 Processor -- start 2024-03-11T19:11:39.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.383+0000 7f5e671d8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902aff270 0x557902b14c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.383+0000 7f5e671d8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902affc60 0x557902b132c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.383+0000 7f5e671d8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902b04dd0 0x557902b13800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.383+0000 7f5e671d8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557902b13f00 con 0x557902b04dd0 2024-03-11T19:11:39.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.383+0000 7f5e671d8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557902b14070 con 0x557902aff270 2024-03-11T19:11:39.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.383+0000 7f5e671d8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557902b6ff70 con 0x557902affc60 2024-03-11T19:11:39.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.383+0000 7f5e671d8740 10 client.0 _ll_register_callbacks cb 0x7f5e5800de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:39.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.383+0000 7f5e622c8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902aff270 0x557902b14c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.383+0000 7f5e62ac9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902b04dd0 0x557902b13800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.383+0000 7f5e622c8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902aff270 0x557902b14c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:48264/0 (socket says 172.21.15.29:48264) 2024-03-11T19:11:39.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.383+0000 7f5e622c8640 1 -- 172.21.15.29:0/2527168864 learned_addr learned my addr 172.21.15.29:0/2527168864 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:39.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.383+0000 7f5e61ac7640 1 --2- 172.21.15.29:0/2527168864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902affc60 0x557902b132c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.383+0000 7f5e622c8640 1 -- 172.21.15.29:0/2527168864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902affc60 msgr2=0x557902b132c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.383+0000 7f5e622c8640 1 --2- 172.21.15.29:0/2527168864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902affc60 0x557902b132c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.383+0000 7f5e622c8640 1 -- 172.21.15.29:0/2527168864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902b04dd0 msgr2=0x557902b13800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.383+0000 7f5e622c8640 1 --2- 172.21.15.29:0/2527168864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902b04dd0 0x557902b13800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.383+0000 7f5e622c8640 1 -- 172.21.15.29:0/2527168864 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557902a2a060 con 0x557902aff270 2024-03-11T19:11:39.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.384+0000 7f5e622c8640 1 --2- 172.21.15.29:0/2527168864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902aff270 0x557902b14c60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5e58099bc0 tx=0x7f5e5800b4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71b139978db4e928 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.384+0000 7f5e45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:11:39.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.384+0000 7f5e45ffb640 1 -- 172.21.15.29:0/2527168864 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5809de70 con 0x557902aff270 2024-03-11T19:11:39.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.384+0000 7f5e45ffb640 1 -- 172.21.15.29:0/2527168864 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5e580041f0 con 0x557902aff270 2024-03-11T19:11:39.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.384+0000 7f5e671d8740 1 -- 172.21.15.29:0/2527168864 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e58099d20 con 0x557902aff270 2024-03-11T19:11:39.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.384+0000 7f5e45ffb640 1 -- 172.21.15.29:0/2527168864 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e580b8120 con 0x557902aff270 2024-03-11T19:11:39.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.384+0000 7f5e671d8740 10 client.25385 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:39.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.384+0000 7f5e671d8740 1 -- 172.21.15.29:0/2527168864 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7f5e54009cd0 con 0x557902aff270 2024-03-11T19:11:39.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.384+0000 7f5e45ffb640 1 -- 172.21.15.29:0/2527168864 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5e58099d20 con 0x557902aff270 2024-03-11T19:11:39.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.384+0000 7f5e45ffb640 1 -- 172.21.15.29:0/2527168864 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5e58095a70 con 0x557902aff270 2024-03-11T19:11:39.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.384+0000 7f5e671d8740 10 client.25385 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:39.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.384+0000 7f5e671d8740 10 client.25385 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:11:39.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.384+0000 7f5e671d8740 20 client.25385 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:39.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.384+0000 7f5e671d8740 1 --2- 172.21.15.29:0/2527168864 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x557902b71500 0x557902b738f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.384+0000 7f5e671d8740 4 client.25385 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:11:39.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.385+0000 7f5e671d8740 1 -- 172.21.15.29:0/2527168864 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5579027f3890 con 0x557902b71500 2024-03-11T19:11:39.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.385+0000 7f5e61ac7640 1 --2- 172.21.15.29:0/2527168864 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x557902b71500 0x557902b738f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.385+0000 7f5e61ac7640 1 --2- 172.21.15.29:0/2527168864 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x557902b71500 0x557902b738f0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fabb4f0fd39dddf3 server_cookie=75d87a1c5711a2dd in_seq=0 out_seq=0 2024-03-11T19:11:39.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.385+0000 7f5e45ffb640 10 client.25385 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:11:39.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.385+0000 7f5e45ffb640 1 -- 172.21.15.29:0/2527168864 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5579027f3890 con 0x557902b71500 2024-03-11T19:11:39.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.385+0000 7f5e45ffb640 10 client.25385 handle_command_reply: tid=0 2024-03-11T19:11:39.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.385+0000 7f5e671d8740 1 client.25385 shutdown 2024-03-11T19:11:39.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.386+0000 7f5e671d8740 1 -- 172.21.15.29:0/2527168864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902aff270 msgr2=0x557902b14c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.386+0000 7f5e671d8740 1 --2- 172.21.15.29:0/2527168864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902aff270 0x557902b14c60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5e58099bc0 tx=0x7f5e5800b4a0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.386+0000 7f5e671d8740 1 -- 172.21.15.29:0/2527168864 shutdown_connections 2024-03-11T19:11:39.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.386+0000 7f5e671d8740 1 --2- 172.21.15.29:0/2527168864 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x557902b71500 0x557902b738f0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.386+0000 7f5e671d8740 1 --2- 172.21.15.29:0/2527168864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557902b04dd0 0x557902b13800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.386+0000 7f5e671d8740 1 --2- 172.21.15.29:0/2527168864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557902affc60 0x557902b132c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.386+0000 7f5e671d8740 1 --2- 172.21.15.29:0/2527168864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557902aff270 0x557902b14c60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.386+0000 7f5e45ffb640 0 client.25385 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:11:39.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.386+0000 7f5e671d8740 1 -- 172.21.15.29:0/2527168864 >> 172.21.15.29:0/2527168864 conn(0x557902afa7e0 msgr2=0x557902b011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.386+0000 7f5e671d8740 1 -- 172.21.15.29:0/2527168864 shutdown_connections 2024-03-11T19:11:39.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.386+0000 7f5e671d8740 1 -- 172.21.15.29:0/2527168864 wait complete. 2024-03-11T19:11:39.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.386+0000 7f5e671d8740 20 client.25385 trim_cache size 0 max 16384 2024-03-11T19:11:39.394 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:11:39.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.386+0000 7f5e6452c640 1 -- 172.21.15.29:0/2526043630 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5e480a1ab0 msgr2=0x7f5e480a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.386+0000 7f5e6452c640 1 --2- 172.21.15.29:0/2526043630 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5e480a1ab0 0x7f5e480a3f70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c036040 tx=0x7f5e4c005fd0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.387+0000 7f5e6452c640 1 -- 172.21.15.29:0/2526043630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e5c1011b0 msgr2=0x7f5e5c197340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.387+0000 7f5e6452c640 1 --2- 172.21.15.29:0/2526043630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e5c1011b0 0x7f5e5c197340 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f5e5800d770 tx=0x7f5e5800dc30 comp rx=0 tx=0).stop 2024-03-11T19:11:39.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.387+0000 7f5e6452c640 1 -- 172.21.15.29:0/2526043630 shutdown_connections 2024-03-11T19:11:39.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.387+0000 7f5e6452c640 1 --2- 172.21.15.29:0/2526043630 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e5c19c8e0 0x7f5e5c19ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.387+0000 7f5e6452c640 1 --2- 172.21.15.29:0/2526043630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e5c102d60 0x7f5e5c197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.387+0000 7f5e6452c640 1 --2- 172.21.15.29:0/2526043630 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5e480a1ab0 0x7f5e480a3f70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.387+0000 7f5e6452c640 1 --2- 172.21.15.29:0/2526043630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e5c1011b0 0x7f5e5c197340 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.387+0000 7f5e6452c640 1 -- 172.21.15.29:0/2526043630 >> 172.21.15.29:0/2526043630 conn(0x7f5e5c0fc720 msgr2=0x7f5e5c105790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.387+0000 7f5e6452c640 1 -- 172.21.15.29:0/2526043630 shutdown_connections 2024-03-11T19:11:39.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.387+0000 7f5e6452c640 1 -- 172.21.15.29:0/2526043630 wait complete. 2024-03-11T19:11:39.405 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:11:39.405 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:11:39.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.560+0000 7fa11d3c7640 1 -- 172.21.15.29:0/132464893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa118101c40 msgr2=0x7fa118104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.560+0000 7fa11d3c7640 1 --2- 172.21.15.29:0/132464893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa118101c40 0x7fa118104030 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa108009900 tx=0x7fa10802bc30 comp rx=0 tx=0).stop 2024-03-11T19:11:39.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.560+0000 7fa11d3c7640 1 -- 172.21.15.29:0/132464893 shutdown_connections 2024-03-11T19:11:39.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.560+0000 7fa11d3c7640 1 --2- 172.21.15.29:0/132464893 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa118106ea0 0x7fa1181092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.560+0000 7fa11d3c7640 1 --2- 172.21.15.29:0/132464893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa118104570 0x7fa118106960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.560+0000 7fa11d3c7640 1 --2- 172.21.15.29:0/132464893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa118101c40 0x7fa118104030 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.560+0000 7fa11d3c7640 1 -- 172.21.15.29:0/132464893 >> 172.21.15.29:0/132464893 conn(0x7fa1180fb630 msgr2=0x7fa1180fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.561+0000 7fa11d3c7640 1 -- 172.21.15.29:0/132464893 shutdown_connections 2024-03-11T19:11:39.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.561+0000 7fa11d3c7640 1 -- 172.21.15.29:0/132464893 wait complete. 2024-03-11T19:11:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.561+0000 7fa11d3c7640 1 Processor -- start 2024-03-11T19:11:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.561+0000 7fa11d3c7640 1 -- start start 2024-03-11T19:11:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.561+0000 7fa11d3c7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa118101c40 0x7fa118195010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.561+0000 7fa11d3c7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa118104570 0x7fa118195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.561+0000 7fa11d3c7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa118106ea0 0x7fa11819c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.561+0000 7fa11d3c7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11819cb00 con 0x7fa118104570 2024-03-11T19:11:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.561+0000 7fa11d3c7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11819cc70 con 0x7fa118101c40 2024-03-11T19:11:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.561+0000 7fa11d3c7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11819ce30 con 0x7fa118106ea0 2024-03-11T19:11:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.562+0000 7fa10ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa118104570 0x7fa118195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.562+0000 7fa116ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa118101c40 0x7fa118195010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.562+0000 7fa10ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa118104570 0x7fa118195550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46060/0 (socket says 172.21.15.29:46060) 2024-03-11T19:11:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.562+0000 7fa10ffff640 1 -- 172.21.15.29:0/447649943 learned_addr learned my addr 172.21.15.29:0/447649943 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.562+0000 7fa1177fe640 1 --2- 172.21.15.29:0/447649943 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa118106ea0 0x7fa11819c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.562+0000 7fa116ffd640 1 -- 172.21.15.29:0/447649943 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa118106ea0 msgr2=0x7fa11819c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.562+0000 7fa116ffd640 1 --2- 172.21.15.29:0/447649943 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa118106ea0 0x7fa11819c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.562+0000 7fa116ffd640 1 -- 172.21.15.29:0/447649943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa118104570 msgr2=0x7fa118195550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.562+0000 7fa116ffd640 1 --2- 172.21.15.29:0/447649943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa118104570 0x7fa118195550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.562+0000 7fa116ffd640 1 -- 172.21.15.29:0/447649943 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa108009560 con 0x7fa118101c40 2024-03-11T19:11:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.562+0000 7fa116ffd640 1 --2- 172.21.15.29:0/447649943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa118101c40 0x7fa118195010 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa108000c00 tx=0x7fa108005e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.562+0000 7fa114ff9640 1 -- 172.21.15.29:0/447649943 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1080043d0 con 0x7fa118101c40 2024-03-11T19:11:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.563+0000 7fa114ff9640 1 -- 172.21.15.29:0/447649943 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa10802de90 con 0x7fa118101c40 2024-03-11T19:11:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.563+0000 7fa114ff9640 1 -- 172.21.15.29:0/447649943 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1080052e0 con 0x7fa118101c40 2024-03-11T19:11:39.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.563+0000 7fa11d3c7640 1 -- 172.21.15.29:0/447649943 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa11819d0b0 con 0x7fa118101c40 2024-03-11T19:11:39.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.563+0000 7fa11d3c7640 1 -- 172.21.15.29:0/447649943 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa11819d540 con 0x7fa118101c40 2024-03-11T19:11:39.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.566+0000 7fa114ff9640 1 -- 172.21.15.29:0/447649943 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fa108003cc0 con 0x7fa118101c40 2024-03-11T19:11:39.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.566+0000 7fa11ff98740 1 Processor -- start 2024-03-11T19:11:39.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.566+0000 7fa11ff98740 1 -- start start 2024-03-11T19:11:39.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.567+0000 7fa114ff9640 1 --2- 172.21.15.29:0/447649943 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa1040a1ab0 0x7fa1040a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.567+0000 7fa11ff98740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555de40c9270 0x555de40c9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.567+0000 7fa11ff98740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9c60 0x555de40ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.567+0000 7fa11ff98740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40cedd0 0x555de40d1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.567+0000 7fa11ff98740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de4085960 con 0x555de40cedd0 2024-03-11T19:11:39.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.567+0000 7fa11ff98740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de4039310 con 0x555de40c9c60 2024-03-11T19:11:39.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.567+0000 7fa11ff98740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de4036840 con 0x555de40c9270 2024-03-11T19:11:39.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.567+0000 7fa116ffd640 1 --2- 172.21.15.29:0/447649943 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa1040a1ab0 0x7fa1040a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.567+0000 7fa1177fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40cedd0 0x555de40d1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.567+0000 7fa114ff9640 1 -- 172.21.15.29:0/447649943 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fa1080ef300 con 0x7fa118101c40 2024-03-11T19:11:39.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.567+0000 7fa1177fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40cedd0 0x555de40d1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46072/0 (socket says 172.21.15.29:46072) 2024-03-11T19:11:39.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.567+0000 7fa1177fe640 1 -- 172.21.15.29:0/68256781 learned_addr learned my addr 172.21.15.29:0/68256781 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:39.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.568+0000 7fa10ffff640 1 --2- 172.21.15.29:0/68256781 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9c60 0x555de40ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.568+0000 7fa1177fe640 1 --2- 172.21.15.29:0/68256781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555de40c9270 0x555de40c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.568+0000 7fa1177fe640 1 -- 172.21.15.29:0/68256781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555de40c9270 msgr2=0x555de40c9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.568+0000 7fa1177fe640 1 --2- 172.21.15.29:0/68256781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555de40c9270 0x555de40c9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.568+0000 7fa1177fe640 1 -- 172.21.15.29:0/68256781 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9c60 msgr2=0x555de40ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.568+0000 7fa1177fe640 1 --2- 172.21.15.29:0/68256781 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9c60 0x555de40ce760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.568+0000 7fa1177fe640 1 -- 172.21.15.29:0/68256781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555de3ff4060 con 0x555de40cedd0 2024-03-11T19:11:39.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.568+0000 7fa116ffd640 1 --2- 172.21.15.29:0/447649943 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa1040a1ab0 0x7fa1040a3f70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa10804c000 tx=0x7fa1080a7030 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.568+0000 7fa1177fe640 1 --2- 172.21.15.29:0/68256781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40cedd0 0x555de40d1260 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fa110013770 tx=0x7fa11000d910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41cbb519428f6e50 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.569+0000 7fa10d7fa640 1 -- 172.21.15.29:0/68256781 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110020070 con 0x555de40cedd0 2024-03-11T19:11:39.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.569+0000 7fa10d7fa640 1 -- 172.21.15.29:0/68256781 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa110013040 con 0x555de40cedd0 2024-03-11T19:11:39.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.569+0000 7fa10d7fa640 1 -- 172.21.15.29:0/68256781 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110012040 con 0x555de40cedd0 2024-03-11T19:11:39.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.570+0000 7fa11ff98740 1 -- 172.21.15.29:0/68256781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40cedd0 msgr2=0x555de40d1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.570+0000 7fa11ff98740 1 --2- 172.21.15.29:0/68256781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40cedd0 0x555de40d1260 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fa110013770 tx=0x7fa11000d910 comp rx=0 tx=0).stop 2024-03-11T19:11:39.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.570+0000 7fa11ff98740 1 -- 172.21.15.29:0/68256781 shutdown_connections 2024-03-11T19:11:39.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.570+0000 7fa11ff98740 1 --2- 172.21.15.29:0/68256781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40cedd0 0x555de40d1260 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.570+0000 7fa11ff98740 1 --2- 172.21.15.29:0/68256781 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9c60 0x555de40ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.570+0000 7fa11ff98740 1 --2- 172.21.15.29:0/68256781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555de40c9270 0x555de40c9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.570+0000 7fa11ff98740 1 -- 172.21.15.29:0/68256781 >> 172.21.15.29:0/68256781 conn(0x555de40c47e0 msgr2=0x555de40c4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.570+0000 7fa11ff98740 1 -- 172.21.15.29:0/68256781 shutdown_connections 2024-03-11T19:11:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.570+0000 7fa11ff98740 1 -- 172.21.15.29:0/68256781 wait complete. 2024-03-11T19:11:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.571+0000 7fa11ff98740 1 -- start start 2024-03-11T19:11:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.571+0000 7fa11ff98740 1 Processor -- start 2024-03-11T19:11:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.571+0000 7fa11ff98740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9270 0x555de40d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.571+0000 7fa11ff98740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9c60 0x555de40d39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.571+0000 7fa11ff98740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555de40cedd0 0x555de40daa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.571+0000 7fa11ff98740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de4032020 con 0x555de40c9c60 2024-03-11T19:11:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.571+0000 7fa11ff98740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de3d05430 con 0x555de40c9270 2024-03-11T19:11:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.571+0000 7fa11ff98740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de3cc1e00 con 0x555de40cedd0 2024-03-11T19:11:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.571+0000 7fa10ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9c60 0x555de40d39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.571+0000 7fa11ff98740 10 client.0 _ll_register_callbacks cb 0x555de3fd9cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.571+0000 7fa10ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9c60 0x555de40d39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46078/0 (socket says 172.21.15.29:46078) 2024-03-11T19:11:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.571+0000 7fa10ffff640 1 -- 172.21.15.29:0/3051630769 learned_addr learned my addr 172.21.15.29:0/3051630769 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.572+0000 7fa10ffff640 1 -- 172.21.15.29:0/3051630769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555de40cedd0 msgr2=0x555de40daa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.572+0000 7fa10ffff640 1 --2- 172.21.15.29:0/3051630769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555de40cedd0 0x555de40daa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.572+0000 7fa10ffff640 1 -- 172.21.15.29:0/3051630769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9270 msgr2=0x555de40d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.572+0000 7fa10ffff640 1 --2- 172.21.15.29:0/3051630769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9270 0x555de40d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.572+0000 7fa10ffff640 1 -- 172.21.15.29:0/3051630769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa110013300 con 0x555de40c9c60 2024-03-11T19:11:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.572+0000 7fa10ffff640 1 --2- 172.21.15.29:0/3051630769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9c60 0x555de40d39d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa1000126d0 tx=0x7fa100012ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6df43d85c416d21e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.572+0000 7fa0fe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.572+0000 7fa0fe7fc640 1 -- 172.21.15.29:0/3051630769 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa100017070 con 0x555de40c9c60 2024-03-11T19:11:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.572+0000 7fa0fe7fc640 1 -- 172.21.15.29:0/3051630769 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fa10000f5e0 con 0x555de40c9c60 2024-03-11T19:11:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.572+0000 7fa11ff98740 1 -- 172.21.15.29:0/3051630769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555de3d317e0 con 0x555de40c9c60 2024-03-11T19:11:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.572+0000 7fa0fe7fc640 1 -- 172.21.15.29:0/3051630769 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1000122e0 con 0x555de40c9c60 2024-03-11T19:11:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.572+0000 7fa11ff98740 10 client.15846 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.572+0000 7fa11ff98740 1 -- 172.21.15.29:0/3051630769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555de40306a0 con 0x555de40c9c60 2024-03-11T19:11:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.573+0000 7fa0fe7fc640 1 -- 172.21.15.29:0/3051630769 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x555de3d317e0 con 0x555de40c9c60 2024-03-11T19:11:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.573+0000 7fa0fe7fc640 1 -- 172.21.15.29:0/3051630769 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa100010960 con 0x555de40c9c60 2024-03-11T19:11:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.573+0000 7fa11ff98740 10 client.15846 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.573+0000 7fa11ff98740 10 client.15846 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:11:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.573+0000 7fa11ff98740 20 client.15846 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.573+0000 7fa11ff98740 1 --2- 172.21.15.29:0/3051630769 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x555de40df780 0x555de40e1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.573+0000 7fa11ff98740 4 client.15846 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:11:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.573+0000 7fa11ff98740 1 -- 172.21.15.29:0/3051630769 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555de3d6e720 con 0x555de40df780 2024-03-11T19:11:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.573+0000 7fa1177fe640 1 --2- 172.21.15.29:0/3051630769 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x555de40df780 0x555de40e1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.574+0000 7fa0fe7fc640 10 client.15846 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:11:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.574+0000 7fa1177fe640 1 --2- 172.21.15.29:0/3051630769 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x555de40df780 0x555de40e1ba0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=81210b37bc4cf48a server_cookie=f3d7f5207ee04927 in_seq=0 out_seq=0 2024-03-11T19:11:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.575+0000 7fa0fe7fc640 1 -- 172.21.15.29:0/3051630769 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555de3d6e720 con 0x555de40df780 2024-03-11T19:11:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.575+0000 7fa0fe7fc640 10 client.15846 handle_command_reply: tid=0 2024-03-11T19:11:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.575+0000 7fa11ff98740 1 client.15846 shutdown 2024-03-11T19:11:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.575+0000 7fa11ff98740 1 -- 172.21.15.29:0/3051630769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9c60 msgr2=0x555de40d39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.575+0000 7fa11ff98740 1 --2- 172.21.15.29:0/3051630769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9c60 0x555de40d39d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa1000126d0 tx=0x7fa100012ad0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.575+0000 7fa11ff98740 1 -- 172.21.15.29:0/3051630769 shutdown_connections 2024-03-11T19:11:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.575+0000 7fa11ff98740 1 --2- 172.21.15.29:0/3051630769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555de40cedd0 0x555de40daa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.575+0000 7fa11ff98740 1 --2- 172.21.15.29:0/3051630769 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x555de40df780 0x555de40e1ba0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.575+0000 7fa11ff98740 1 --2- 172.21.15.29:0/3051630769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9c60 0x555de40d39d0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.575+0000 7fa11ff98740 1 --2- 172.21.15.29:0/3051630769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9270 0x555de40d3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.575+0000 7fa0fe7fc640 0 client.15846 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:11:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.575+0000 7fa11ff98740 1 -- 172.21.15.29:0/3051630769 >> 172.21.15.29:0/3051630769 conn(0x555de40c47e0 msgr2=0x555de40c3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.575+0000 7fa11ff98740 1 -- 172.21.15.29:0/3051630769 shutdown_connections 2024-03-11T19:11:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.575+0000 7fa11ff98740 1 -- 172.21.15.29:0/3051630769 wait complete. 2024-03-11T19:11:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.575+0000 7fa11ff98740 20 client.15846 trim_cache size 0 max 16384 2024-03-11T19:11:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.593+0000 7fa11ff98740 1 Processor -- start 2024-03-11T19:11:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.593+0000 7fa11ff98740 1 -- start start 2024-03-11T19:11:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.594+0000 7fa11ff98740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9270 0x555de40c4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.594+0000 7fa11ff98740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9c60 0x555de40da9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.594+0000 7fa11ff98740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555de40cedd0 0x555de40cb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.594+0000 7fa11ff98740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de3fb7130 con 0x555de40c9270 2024-03-11T19:11:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.594+0000 7fa11ff98740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de40c4690 con 0x555de40c9c60 2024-03-11T19:11:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.594+0000 7fa11ff98740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de3d6e720 con 0x555de40cedd0 2024-03-11T19:11:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.594+0000 7fa1177fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9270 0x555de40c4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.594+0000 7fa1177fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9270 0x555de40c4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46090/0 (socket says 172.21.15.29:46090) 2024-03-11T19:11:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.594+0000 7fa1177fe640 1 -- 172.21.15.29:0/2499039719 learned_addr learned my addr 172.21.15.29:0/2499039719 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.594+0000 7fa10ffff640 1 --2- 172.21.15.29:0/2499039719 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9c60 0x555de40da9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.594+0000 7fa1177fe640 1 --2- 172.21.15.29:0/2499039719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555de40cedd0 0x555de40cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.594+0000 7fa10ffff640 1 -- 172.21.15.29:0/2499039719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555de40cedd0 msgr2=0x555de40cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.594+0000 7fa10ffff640 1 --2- 172.21.15.29:0/2499039719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555de40cedd0 0x555de40cb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.594+0000 7fa10ffff640 1 -- 172.21.15.29:0/2499039719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9270 msgr2=0x555de40c4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.594+0000 7fa10ffff640 1 --2- 172.21.15.29:0/2499039719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9270 0x555de40c4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.594+0000 7fa10ffff640 1 -- 172.21.15.29:0/2499039719 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555de3fccc20 con 0x555de40c9c60 2024-03-11T19:11:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.594+0000 7fa1177fe640 1 --2- 172.21.15.29:0/2499039719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9270 0x555de40c4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.595+0000 7fa10ffff640 1 --2- 172.21.15.29:0/2499039719 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9c60 0x555de40da9d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa100010d10 tx=0x7fa100005db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c793316202e5b66 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.595+0000 7fa10cff9640 1 -- 172.21.15.29:0/2499039719 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa100017070 con 0x555de40c9c60 2024-03-11T19:11:39.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.595+0000 7fa10cff9640 1 -- 172.21.15.29:0/2499039719 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa1000122e0 con 0x555de40c9c60 2024-03-11T19:11:39.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.595+0000 7fa10cff9640 1 -- 172.21.15.29:0/2499039719 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa100010b10 con 0x555de40c9c60 2024-03-11T19:11:39.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.596+0000 7fa11ff98740 1 -- 172.21.15.29:0/2499039719 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9c60 msgr2=0x555de40da9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.596+0000 7fa11ff98740 1 --2- 172.21.15.29:0/2499039719 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9c60 0x555de40da9d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa100010d10 tx=0x7fa100005db0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.596+0000 7fa11ff98740 1 -- 172.21.15.29:0/2499039719 shutdown_connections 2024-03-11T19:11:39.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.596+0000 7fa11ff98740 1 --2- 172.21.15.29:0/2499039719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555de40cedd0 0x555de40cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.596+0000 7fa11ff98740 1 --2- 172.21.15.29:0/2499039719 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9c60 0x555de40da9d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.596+0000 7fa11ff98740 1 --2- 172.21.15.29:0/2499039719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9270 0x555de40c4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.596+0000 7fa11ff98740 1 -- 172.21.15.29:0/2499039719 >> 172.21.15.29:0/2499039719 conn(0x555de40c47e0 msgr2=0x555de40bfa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.596+0000 7fa11ff98740 1 -- 172.21.15.29:0/2499039719 shutdown_connections 2024-03-11T19:11:39.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.596+0000 7fa11ff98740 1 -- 172.21.15.29:0/2499039719 wait complete. 2024-03-11T19:11:39.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.596+0000 7fa11ff98740 1 -- start start 2024-03-11T19:11:39.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.596+0000 7fa11ff98740 1 Processor -- start 2024-03-11T19:11:39.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.597+0000 7fa11ff98740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9270 0x555de40d9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.597+0000 7fa11ff98740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9c60 0x555de40d9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.597+0000 7fa11ff98740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555de40cedd0 0x555de40d9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.597+0000 7fa11ff98740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de4135c90 con 0x555de40c9270 2024-03-11T19:11:39.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.597+0000 7fa11ff98740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de4135dd0 con 0x555de40c9c60 2024-03-11T19:11:39.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.597+0000 7fa11ff98740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de4135f90 con 0x555de40cedd0 2024-03-11T19:11:39.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.597+0000 7fa11ff98740 10 client.0 _ll_register_callbacks cb 0x7fa1100131b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:39.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.597+0000 7fa1177fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9270 0x555de40d9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.597+0000 7fa1177fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9270 0x555de40d9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46094/0 (socket says 172.21.15.29:46094) 2024-03-11T19:11:39.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.597+0000 7fa1177fe640 1 -- 172.21.15.29:0/1067572270 learned_addr learned my addr 172.21.15.29:0/1067572270 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:39.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.597+0000 7fa10ffff640 1 --2- 172.21.15.29:0/1067572270 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9c60 0x555de40d9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.597+0000 7fa1177fe640 1 --2- 172.21.15.29:0/1067572270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555de40cedd0 0x555de40d9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.597+0000 7fa1177fe640 1 -- 172.21.15.29:0/1067572270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555de40cedd0 msgr2=0x555de40d9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.597+0000 7fa1177fe640 1 --2- 172.21.15.29:0/1067572270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555de40cedd0 0x555de40d9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.597+0000 7fa1177fe640 1 -- 172.21.15.29:0/1067572270 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9c60 msgr2=0x555de40d9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.597+0000 7fa1177fe640 1 --2- 172.21.15.29:0/1067572270 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9c60 0x555de40d9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.597+0000 7fa1177fe640 1 -- 172.21.15.29:0/1067572270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555de3ff4060 con 0x555de40c9270 2024-03-11T19:11:39.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.597+0000 7fa1177fe640 1 --2- 172.21.15.29:0/1067572270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9270 0x555de40d9340 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fa110002230 tx=0x7fa11000dad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1223d1eb92f0933 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.597+0000 7fa0fe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:39.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.598+0000 7fa0fe7fc640 1 -- 172.21.15.29:0/1067572270 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110012070 con 0x555de40c9270 2024-03-11T19:11:39.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.598+0000 7fa0fe7fc640 1 -- 172.21.15.29:0/1067572270 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fa110003a30 con 0x555de40c9270 2024-03-11T19:11:39.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.598+0000 7fa11ff98740 1 -- 172.21.15.29:0/1067572270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa110013040 con 0x555de40c9270 2024-03-11T19:11:39.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.598+0000 7fa0fe7fc640 1 -- 172.21.15.29:0/1067572270 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa110020040 con 0x555de40c9270 2024-03-11T19:11:39.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.598+0000 7fa11ff98740 10 client.15852 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:39.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.598+0000 7fa11ff98740 1 -- 172.21.15.29:0/1067572270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555de40306a0 con 0x555de40c9270 2024-03-11T19:11:39.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.598+0000 7fa0fe7fc640 1 -- 172.21.15.29:0/1067572270 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa110013040 con 0x555de40c9270 2024-03-11T19:11:39.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.599+0000 7fa0fe7fc640 1 -- 172.21.15.29:0/1067572270 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa11000b7f0 con 0x555de40c9270 2024-03-11T19:11:39.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.599+0000 7fa11ff98740 10 client.15852 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:39.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.599+0000 7fa11ff98740 10 client.15852 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:11:39.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.599+0000 7fa11ff98740 20 client.15852 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:39.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.599+0000 7fa11ff98740 1 --2- 172.21.15.29:0/1067572270 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x555de40df780 0x555de4139ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.599+0000 7fa11ff98740 4 client.15852 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:11:39.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.599+0000 7fa11ff98740 1 -- 172.21.15.29:0/1067572270 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555de3dbd890 con 0x555de40df780 2024-03-11T19:11:39.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.599+0000 7fa10ffff640 1 --2- 172.21.15.29:0/1067572270 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x555de40df780 0x555de4139ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.599+0000 7fa0fe7fc640 10 client.15852 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:11:39.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.599+0000 7fa10ffff640 1 --2- 172.21.15.29:0/1067572270 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x555de40df780 0x555de4139ac0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5b6eceef21edc41e server_cookie=c12e35157c88af5c in_seq=0 out_seq=0 2024-03-11T19:11:39.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.600+0000 7fa0fe7fc640 1 -- 172.21.15.29:0/1067572270 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555de3dbd890 con 0x555de40df780 2024-03-11T19:11:39.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.600+0000 7fa0fe7fc640 10 client.15852 handle_command_reply: tid=0 2024-03-11T19:11:39.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.600+0000 7fa11ff98740 1 client.15852 shutdown 2024-03-11T19:11:39.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.600+0000 7fa11ff98740 1 -- 172.21.15.29:0/1067572270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9270 msgr2=0x555de40d9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.600+0000 7fa11ff98740 1 --2- 172.21.15.29:0/1067572270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9270 0x555de40d9340 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fa110002230 tx=0x7fa11000dad0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.600+0000 7fa11ff98740 1 -- 172.21.15.29:0/1067572270 shutdown_connections 2024-03-11T19:11:39.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.600+0000 7fa11ff98740 1 --2- 172.21.15.29:0/1067572270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555de40cedd0 0x555de40d9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.600+0000 7fa11ff98740 1 --2- 172.21.15.29:0/1067572270 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555de40c9c60 0x555de40d9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.600+0000 7fa11ff98740 1 --2- 172.21.15.29:0/1067572270 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x555de40df780 0x555de4139ac0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.600+0000 7fa11ff98740 1 --2- 172.21.15.29:0/1067572270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555de40c9270 0x555de40d9340 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.600+0000 7fa11ff98740 1 -- 172.21.15.29:0/1067572270 >> 172.21.15.29:0/1067572270 conn(0x555de40c47e0 msgr2=0x555de40c4bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.600+0000 7fa0fe7fc640 0 client.15852 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:11:39.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.600+0000 7fa11ff98740 1 -- 172.21.15.29:0/1067572270 shutdown_connections 2024-03-11T19:11:39.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.600+0000 7fa11ff98740 1 -- 172.21.15.29:0/1067572270 wait complete. 2024-03-11T19:11:39.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.600+0000 7fa11ff98740 20 client.15852 trim_cache size 0 max 16384 2024-03-11T19:11:39.610 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:11:39.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.601+0000 7fa11d3c7640 1 -- 172.21.15.29:0/447649943 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa1040a1ab0 msgr2=0x7fa1040a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.601+0000 7fa11d3c7640 1 --2- 172.21.15.29:0/447649943 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa1040a1ab0 0x7fa1040a3f70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa10804c000 tx=0x7fa1080a7030 comp rx=0 tx=0).stop 2024-03-11T19:11:39.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.601+0000 7fa11d3c7640 1 -- 172.21.15.29:0/447649943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa118101c40 msgr2=0x7fa118195010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.601+0000 7fa11d3c7640 1 --2- 172.21.15.29:0/447649943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa118101c40 0x7fa118195010 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa108000c00 tx=0x7fa108005e70 comp rx=0 tx=0).stop 2024-03-11T19:11:39.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.601+0000 7fa11d3c7640 1 -- 172.21.15.29:0/447649943 shutdown_connections 2024-03-11T19:11:39.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.601+0000 7fa11d3c7640 1 --2- 172.21.15.29:0/447649943 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa118106ea0 0x7fa11819c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.601+0000 7fa11d3c7640 1 --2- 172.21.15.29:0/447649943 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa1040a1ab0 0x7fa1040a3f70 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.601+0000 7fa11d3c7640 1 --2- 172.21.15.29:0/447649943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa118104570 0x7fa118195550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.601+0000 7fa11d3c7640 1 --2- 172.21.15.29:0/447649943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa118101c40 0x7fa118195010 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.601+0000 7fa11d3c7640 1 -- 172.21.15.29:0/447649943 >> 172.21.15.29:0/447649943 conn(0x7fa1180fb630 msgr2=0x7fa118103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.601+0000 7fa11d3c7640 1 -- 172.21.15.29:0/447649943 shutdown_connections 2024-03-11T19:11:39.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.601+0000 7fa11d3c7640 1 -- 172.21.15.29:0/447649943 wait complete. 2024-03-11T19:11:39.618 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:11:39.619 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:11:39.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.777+0000 7f84169a1640 1 -- 172.21.15.29:0/2679239332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8410102d60 msgr2=0x7f84101095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.777+0000 7f84169a1640 1 --2- 172.21.15.29:0/2679239332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8410102d60 0x7f84101095f0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8408009ca0 tx=0x7f840802c0f0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.777+0000 7f84169a1640 1 -- 172.21.15.29:0/2679239332 shutdown_connections 2024-03-11T19:11:39.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.777+0000 7f84169a1640 1 --2- 172.21.15.29:0/2679239332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8410102d60 0x7f84101095f0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.777+0000 7f84169a1640 1 --2- 172.21.15.29:0/2679239332 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f84101023a0 0x7f8410102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.777+0000 7f84169a1640 1 --2- 172.21.15.29:0/2679239332 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84101011b0 0x7f84101015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.777+0000 7f84169a1640 1 -- 172.21.15.29:0/2679239332 >> 172.21.15.29:0/2679239332 conn(0x7f84100fc720 msgr2=0x7f84100feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.777+0000 7f84169a1640 1 -- 172.21.15.29:0/2679239332 shutdown_connections 2024-03-11T19:11:39.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.777+0000 7f84169a1640 1 -- 172.21.15.29:0/2679239332 wait complete. 2024-03-11T19:11:39.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.778+0000 7f84169a1640 1 Processor -- start 2024-03-11T19:11:39.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.778+0000 7f84169a1640 1 -- start start 2024-03-11T19:11:39.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.778+0000 7f84169a1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84101011b0 0x7f8410197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.778+0000 7f84169a1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84101023a0 0x7f8410197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.778+0000 7f84169a1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8410102d60 0x7f841019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.778+0000 7f84169a1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841019ec20 con 0x7f84101011b0 2024-03-11T19:11:39.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.778+0000 7f84169a1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841019ed90 con 0x7f8410102d60 2024-03-11T19:11:39.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.778+0000 7f84169a1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841019ef50 con 0x7f84101023a0 2024-03-11T19:11:39.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.778+0000 7f840ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84101011b0 0x7f8410197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.778+0000 7f840ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84101011b0 0x7f8410197130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46104/0 (socket says 172.21.15.29:46104) 2024-03-11T19:11:39.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.778+0000 7f840ffff640 1 -- 172.21.15.29:0/916223111 learned_addr learned my addr 172.21.15.29:0/916223111 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:39.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.778+0000 7f840f7fe640 1 --2- 172.21.15.29:0/916223111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84101023a0 0x7f8410197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.779+0000 7f8414f3e640 1 --2- 172.21.15.29:0/916223111 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8410102d60 0x7f841019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.779+0000 7f840f7fe640 1 -- 172.21.15.29:0/916223111 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8410102d60 msgr2=0x7f841019e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.779+0000 7f840f7fe640 1 --2- 172.21.15.29:0/916223111 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8410102d60 0x7f841019e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.779+0000 7f840f7fe640 1 -- 172.21.15.29:0/916223111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84101011b0 msgr2=0x7f8410197130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.779+0000 7f840f7fe640 1 --2- 172.21.15.29:0/916223111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84101011b0 0x7f8410197130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.779+0000 7f840f7fe640 1 -- 172.21.15.29:0/916223111 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8408009900 con 0x7f84101023a0 2024-03-11T19:11:39.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.779+0000 7f8414f3e640 1 --2- 172.21.15.29:0/916223111 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8410102d60 0x7f841019e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:11:39.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.779+0000 7f840f7fe640 1 --2- 172.21.15.29:0/916223111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84101023a0 0x7f8410197670 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f840400d6d0 tx=0x7f840400db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.779+0000 7f840d7fa640 1 -- 172.21.15.29:0/916223111 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8404005220 con 0x7f84101023a0 2024-03-11T19:11:39.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.779+0000 7f84169a1640 1 -- 172.21.15.29:0/916223111 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8410073410 con 0x7f84101023a0 2024-03-11T19:11:39.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.779+0000 7f840d7fa640 1 -- 172.21.15.29:0/916223111 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8404009cd0 con 0x7f84101023a0 2024-03-11T19:11:39.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.779+0000 7f840d7fa640 1 -- 172.21.15.29:0/916223111 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8404004d00 con 0x7f84101023a0 2024-03-11T19:11:39.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.779+0000 7f84169a1640 1 -- 172.21.15.29:0/916223111 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8410073970 con 0x7f84101023a0 2024-03-11T19:11:39.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.782+0000 7f840d7fa640 1 -- 172.21.15.29:0/916223111 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8404009820 con 0x7f84101023a0 2024-03-11T19:11:39.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.782+0000 7f8419579740 1 Processor -- start 2024-03-11T19:11:39.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.782+0000 7f8419579740 1 -- start start 2024-03-11T19:11:39.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f840d7fa640 1 --2- 172.21.15.29:0/916223111 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f83f80a1a30 0x7f83f80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f8419579740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 0x559e61690690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f8419579740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690c60 0x559e61695760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f8419579740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 0x559e61698260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f8419579740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e6164c960 con 0x559e61690270 2024-03-11T19:11:39.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f8419579740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e61600310 con 0x559e61695dd0 2024-03-11T19:11:39.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f8419579740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e615fd840 con 0x559e61690c60 2024-03-11T19:11:39.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f840d7fa640 1 -- 172.21.15.29:0/916223111 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f84040cce40 con 0x7f84101023a0 2024-03-11T19:11:39.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f8414f3e640 1 --2- 172.21.15.29:0/916223111 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f83f80a1a30 0x7f83f80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f840ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 0x559e61690690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f840f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690c60 0x559e61695760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f840f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690c60 0x559e61695760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:51578/0 (socket says 172.21.15.29:51578) 2024-03-11T19:11:39.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f840f7fe640 1 -- 172.21.15.29:0/3938481029 learned_addr learned my addr 172.21.15.29:0/3938481029 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:39.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f8414f3e640 1 --2- 172.21.15.29:0/3938481029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 0x559e61698260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f840f7fe640 1 -- 172.21.15.29:0/3938481029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 msgr2=0x559e61698260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f840f7fe640 1 --2- 172.21.15.29:0/3938481029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 0x559e61698260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f840f7fe640 1 -- 172.21.15.29:0/3938481029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 msgr2=0x559e61690690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f840f7fe640 1 --2- 172.21.15.29:0/3938481029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 0x559e61690690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f840f7fe640 1 -- 172.21.15.29:0/3938481029 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e615bb060 con 0x559e61690c60 2024-03-11T19:11:39.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.783+0000 7f840ffff640 1 --2- 172.21.15.29:0/3938481029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 0x559e61690690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:39.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.784+0000 7f8414f3e640 1 --2- 172.21.15.29:0/916223111 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f83f80a1a30 0x7f83f80a3ef0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8408032460 tx=0x7f8408032ec0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.784+0000 7f840f7fe640 1 --2- 172.21.15.29:0/3938481029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690c60 0x559e61695760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8404096820 tx=0x7f84040997b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeee39ba7869e009 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.784+0000 7f83f67fc640 1 -- 172.21.15.29:0/3938481029 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8404062a40 con 0x559e61690c60 2024-03-11T19:11:39.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.784+0000 7f83f67fc640 1 -- 172.21.15.29:0/3938481029 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8404062820 con 0x559e61690c60 2024-03-11T19:11:39.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.784+0000 7f83f67fc640 1 -- 172.21.15.29:0/3938481029 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840409dfa0 con 0x559e61690c60 2024-03-11T19:11:39.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.785+0000 7f8419579740 1 -- 172.21.15.29:0/3938481029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690c60 msgr2=0x559e61695760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.785+0000 7f8419579740 1 --2- 172.21.15.29:0/3938481029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690c60 0x559e61695760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8404096820 tx=0x7f84040997b0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.785+0000 7f8419579740 1 -- 172.21.15.29:0/3938481029 shutdown_connections 2024-03-11T19:11:39.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.785+0000 7f8419579740 1 --2- 172.21.15.29:0/3938481029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 0x559e61698260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.785+0000 7f8419579740 1 --2- 172.21.15.29:0/3938481029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690c60 0x559e61695760 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.785+0000 7f8419579740 1 --2- 172.21.15.29:0/3938481029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 0x559e61690690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.785+0000 7f8419579740 1 -- 172.21.15.29:0/3938481029 >> 172.21.15.29:0/3938481029 conn(0x559e6168b7e0 msgr2=0x559e6168bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.785+0000 7f8419579740 1 -- 172.21.15.29:0/3938481029 shutdown_connections 2024-03-11T19:11:39.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.785+0000 7f8419579740 1 -- 172.21.15.29:0/3938481029 wait complete. 2024-03-11T19:11:39.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.785+0000 7f8419579740 1 -- start start 2024-03-11T19:11:39.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.786+0000 7f8419579740 1 Processor -- start 2024-03-11T19:11:39.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.786+0000 7f8419579740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690270 0x559e6169a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.786+0000 7f8419579740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690c60 0x559e6169a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.786+0000 7f8419579740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 0x559e616a1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.786+0000 7f8419579740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e615f9020 con 0x559e61690c60 2024-03-11T19:11:39.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.786+0000 7f8419579740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e612cc430 con 0x559e61695dd0 2024-03-11T19:11:39.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.786+0000 7f8419579740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e61288e00 con 0x559e61690270 2024-03-11T19:11:39.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.786+0000 7f8419579740 10 client.0 _ll_register_callbacks cb 0x559e615a0cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:39.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.786+0000 7f840ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690270 0x559e6169a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.786+0000 7f840f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690c60 0x559e6169a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.786+0000 7f840ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690270 0x559e6169a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:51580/0 (socket says 172.21.15.29:51580) 2024-03-11T19:11:39.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.786+0000 7f840ffff640 1 -- 172.21.15.29:0/1213140271 learned_addr learned my addr 172.21.15.29:0/1213140271 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:39.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.787+0000 7f8414f3e640 1 --2- 172.21.15.29:0/1213140271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 0x559e616a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.787+0000 7f840ffff640 1 -- 172.21.15.29:0/1213140271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 msgr2=0x559e616a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.787+0000 7f840ffff640 1 --2- 172.21.15.29:0/1213140271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 0x559e616a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.787+0000 7f840ffff640 1 -- 172.21.15.29:0/1213140271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690c60 msgr2=0x559e6169a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.787+0000 7f840ffff640 1 --2- 172.21.15.29:0/1213140271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690c60 0x559e6169a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.787+0000 7f840ffff640 1 -- 172.21.15.29:0/1213140271 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8410073970 con 0x559e61690270 2024-03-11T19:11:39.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.787+0000 7f8414f3e640 1 --2- 172.21.15.29:0/1213140271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 0x559e616a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:39.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.787+0000 7f840ffff640 1 --2- 172.21.15.29:0/1213140271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690270 0x559e6169a400 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f840000bf60 tx=0x7f840000bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1829574aa39ef71e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.787+0000 7f83ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:11:39.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.787+0000 7f83ef7fe640 1 -- 172.21.15.29:0/1213140271 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8400010960 con 0x559e61690270 2024-03-11T19:11:39.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.787+0000 7f8419579740 1 -- 172.21.15.29:0/1213140271 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559e612f87e0 con 0x559e61690270 2024-03-11T19:11:39.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.787+0000 7f83ef7fe640 1 -- 172.21.15.29:0/1213140271 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f8400010ac0 con 0x559e61690270 2024-03-11T19:11:39.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.788+0000 7f83ef7fe640 1 -- 172.21.15.29:0/1213140271 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8400010c80 con 0x559e61690270 2024-03-11T19:11:39.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.788+0000 7f8419579740 10 client.25456 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:39.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.788+0000 7f8419579740 1 -- 172.21.15.29:0/1213140271 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x559e615f76a0 con 0x559e61690270 2024-03-11T19:11:39.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.788+0000 7f83ef7fe640 1 -- 172.21.15.29:0/1213140271 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x559e612f87e0 con 0x559e61690270 2024-03-11T19:11:39.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.788+0000 7f83ef7fe640 1 -- 172.21.15.29:0/1213140271 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8400024390 con 0x559e61690270 2024-03-11T19:11:39.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.788+0000 7f8419579740 10 client.25456 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:39.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.788+0000 7f8419579740 10 client.25456 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:11:39.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.788+0000 7f8419579740 20 client.25456 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:39.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.788+0000 7f8419579740 1 --2- 172.21.15.29:0/1213140271 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x559e616a6650 0x559e616a8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.789+0000 7f8419579740 4 client.25456 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:11:39.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.789+0000 7f8419579740 1 -- 172.21.15.29:0/1213140271 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559e61335720 con 0x559e616a6650 2024-03-11T19:11:39.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.790+0000 7f840f7fe640 1 --2- 172.21.15.29:0/1213140271 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x559e616a6650 0x559e616a8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.791+0000 7f840f7fe640 1 --2- 172.21.15.29:0/1213140271 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x559e616a6650 0x559e616a8a70 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9e0dd09ac1c00b79 server_cookie=e9d6c240fd7f2bc9 in_seq=0 out_seq=0 2024-03-11T19:11:39.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.791+0000 7f83ef7fe640 10 client.25456 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:11:39.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.792+0000 7f83ef7fe640 1 -- 172.21.15.29:0/1213140271 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559e61335720 con 0x559e616a6650 2024-03-11T19:11:39.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.792+0000 7f83ef7fe640 10 client.25456 handle_command_reply: tid=0 2024-03-11T19:11:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.792+0000 7f8419579740 1 client.25456 shutdown 2024-03-11T19:11:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.792+0000 7f8419579740 1 -- 172.21.15.29:0/1213140271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690270 msgr2=0x559e6169a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.792+0000 7f8419579740 1 --2- 172.21.15.29:0/1213140271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690270 0x559e6169a400 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f840000bf60 tx=0x7f840000bf90 comp rx=0 tx=0).stop 2024-03-11T19:11:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.792+0000 7f8419579740 1 -- 172.21.15.29:0/1213140271 shutdown_connections 2024-03-11T19:11:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.792+0000 7f8419579740 1 --2- 172.21.15.29:0/1213140271 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x559e616a6650 0x559e616a8a70 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.792+0000 7f8419579740 1 --2- 172.21.15.29:0/1213140271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 0x559e616a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.792+0000 7f83ef7fe640 0 client.25456 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:11:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.792+0000 7f8419579740 1 --2- 172.21.15.29:0/1213140271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690c60 0x559e6169a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.792+0000 7f8419579740 1 --2- 172.21.15.29:0/1213140271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690270 0x559e6169a400 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.792+0000 7f8419579740 1 -- 172.21.15.29:0/1213140271 >> 172.21.15.29:0/1213140271 conn(0x559e6168b7e0 msgr2=0x559e6168ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.792+0000 7f8419579740 1 -- 172.21.15.29:0/1213140271 shutdown_connections 2024-03-11T19:11:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.792+0000 7f8419579740 1 -- 172.21.15.29:0/1213140271 wait complete. 2024-03-11T19:11:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.792+0000 7f8419579740 20 client.25456 trim_cache size 0 max 16384 2024-03-11T19:11:39.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.804+0000 7f8419579740 1 Processor -- start 2024-03-11T19:11:39.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.804+0000 7f8419579740 1 -- start start 2024-03-11T19:11:39.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f8419579740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 0x559e6168a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f8419579740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690c60 0x559e616a1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f8419579740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 0x559e6168a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f8419579740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e6157e130 con 0x559e61690270 2024-03-11T19:11:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f8419579740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e6167da40 con 0x559e61695dd0 2024-03-11T19:11:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f8419579740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e61691090 con 0x559e61690c60 2024-03-11T19:11:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f840ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 0x559e6168a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f840ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 0x559e6168a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46134/0 (socket says 172.21.15.29:46134) 2024-03-11T19:11:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f840ffff640 1 -- 172.21.15.29:0/2297611993 learned_addr learned my addr 172.21.15.29:0/2297611993 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f840f7fe640 1 --2- 172.21.15.29:0/2297611993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690c60 0x559e616a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f8414f3e640 1 --2- 172.21.15.29:0/2297611993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 0x559e6168a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f840ffff640 1 -- 172.21.15.29:0/2297611993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690c60 msgr2=0x559e616a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f840ffff640 1 --2- 172.21.15.29:0/2297611993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690c60 0x559e616a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f840ffff640 1 -- 172.21.15.29:0/2297611993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 msgr2=0x559e6168a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f840ffff640 1 --2- 172.21.15.29:0/2297611993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 0x559e6168a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f840ffff640 1 -- 172.21.15.29:0/2297611993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e61593c20 con 0x559e61690270 2024-03-11T19:11:39.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f8414f3e640 1 --2- 172.21.15.29:0/2297611993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 0x559e6168a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:11:39.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f840f7fe640 1 --2- 172.21.15.29:0/2297611993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690c60 0x559e616a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:39.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f840ffff640 1 --2- 172.21.15.29:0/2297611993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 0x559e6168a2f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f840000ba30 tx=0x7f840000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b1d44b790cb987 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f83f5ffb640 1 -- 172.21.15.29:0/2297611993 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8400005850 con 0x559e61690270 2024-03-11T19:11:39.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f83f5ffb640 1 -- 172.21.15.29:0/2297611993 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f84000054f0 con 0x559e61690270 2024-03-11T19:11:39.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.805+0000 7f83f5ffb640 1 -- 172.21.15.29:0/2297611993 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84000109c0 con 0x559e61690270 2024-03-11T19:11:39.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.806+0000 7f8419579740 1 -- 172.21.15.29:0/2297611993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 msgr2=0x559e6168a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.806+0000 7f8419579740 1 --2- 172.21.15.29:0/2297611993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 0x559e6168a2f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f840000ba30 tx=0x7f840000b8a0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.806+0000 7f8419579740 1 -- 172.21.15.29:0/2297611993 shutdown_connections 2024-03-11T19:11:39.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.806+0000 7f8419579740 1 --2- 172.21.15.29:0/2297611993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 0x559e6168a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.806+0000 7f8419579740 1 --2- 172.21.15.29:0/2297611993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690c60 0x559e616a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.806+0000 7f8419579740 1 --2- 172.21.15.29:0/2297611993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 0x559e6168a2f0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.806+0000 7f8419579740 1 -- 172.21.15.29:0/2297611993 >> 172.21.15.29:0/2297611993 conn(0x559e6168b7e0 msgr2=0x559e6168bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.806+0000 7f8419579740 1 -- 172.21.15.29:0/2297611993 shutdown_connections 2024-03-11T19:11:39.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.806+0000 7f8419579740 1 -- 172.21.15.29:0/2297611993 wait complete. 2024-03-11T19:11:39.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.806+0000 7f8419579740 1 -- start start 2024-03-11T19:11:39.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f8419579740 1 Processor -- start 2024-03-11T19:11:39.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f8419579740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 0x559e616a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f8419579740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690c60 0x559e616a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f8419579740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 0x559e616fcb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f8419579740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e616a1220 con 0x559e61690270 2024-03-11T19:11:39.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f8419579740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e616a1390 con 0x559e61695dd0 2024-03-11T19:11:39.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f8419579740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e616fd140 con 0x559e61690c60 2024-03-11T19:11:39.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f8419579740 10 client.0 _ll_register_callbacks cb 0x7f8404004740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:39.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f840ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 0x559e616a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f840ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 0x559e616a04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46150/0 (socket says 172.21.15.29:46150) 2024-03-11T19:11:39.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f840ffff640 1 -- 172.21.15.29:0/2111237271 learned_addr learned my addr 172.21.15.29:0/2111237271 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:39.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f8414f3e640 1 --2- 172.21.15.29:0/2111237271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 0x559e616fcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f840f7fe640 1 --2- 172.21.15.29:0/2111237271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690c60 0x559e616a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f840ffff640 1 -- 172.21.15.29:0/2111237271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690c60 msgr2=0x559e616a0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f840ffff640 1 --2- 172.21.15.29:0/2111237271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690c60 0x559e616a0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f840ffff640 1 -- 172.21.15.29:0/2111237271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 msgr2=0x559e616fcb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f840ffff640 1 --2- 172.21.15.29:0/2111237271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 0x559e616fcb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f840ffff640 1 -- 172.21.15.29:0/2111237271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559e615f76a0 con 0x559e61690270 2024-03-11T19:11:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f840ffff640 1 --2- 172.21.15.29:0/2111237271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 0x559e616a04d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f84000247c0 tx=0x7f8400012e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=824e72804f46ea34 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.807+0000 7f83ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.808+0000 7f83ef7fe640 1 -- 172.21.15.29:0/2111237271 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8400010960 con 0x559e61690270 2024-03-11T19:11:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.808+0000 7f83ef7fe640 1 -- 172.21.15.29:0/2111237271 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f8400010ac0 con 0x559e61690270 2024-03-11T19:11:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.808+0000 7f8419579740 1 -- 172.21.15.29:0/2111237271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8404062820 con 0x559e61690270 2024-03-11T19:11:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.808+0000 7f83ef7fe640 1 -- 172.21.15.29:0/2111237271 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8400010c80 con 0x559e61690270 2024-03-11T19:11:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.808+0000 7f8419579740 10 client.15870 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.808+0000 7f8419579740 1 -- 172.21.15.29:0/2111237271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x559e615b98c0 con 0x559e61690270 2024-03-11T19:11:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.808+0000 7f83ef7fe640 1 -- 172.21.15.29:0/2111237271 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8404062820 con 0x559e61690270 2024-03-11T19:11:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.808+0000 7f83ef7fe640 1 -- 172.21.15.29:0/2111237271 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8400039430 con 0x559e61690270 2024-03-11T19:11:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.808+0000 7f8419579740 10 client.15870 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.808+0000 7f8419579740 10 client.15870 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:11:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.808+0000 7f8419579740 20 client.15870 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.808+0000 7f8419579740 1 --2- 172.21.15.29:0/2111237271 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x559e616fe9f0 0x559e61700e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.808+0000 7f8419579740 4 client.15870 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:11:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.808+0000 7f8419579740 1 -- 172.21.15.29:0/2111237271 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559e61384890 con 0x559e616fe9f0 2024-03-11T19:11:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.808+0000 7f840f7fe640 1 --2- 172.21.15.29:0/2111237271 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x559e616fe9f0 0x559e61700e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.809+0000 7f840f7fe640 1 --2- 172.21.15.29:0/2111237271 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x559e616fe9f0 0x559e61700e10 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=561f28f956fae38f server_cookie=ea0277f0e81ac0d6 in_seq=0 out_seq=0 2024-03-11T19:11:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.809+0000 7f83ef7fe640 10 client.15870 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:11:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.809+0000 7f83ef7fe640 1 -- 172.21.15.29:0/2111237271 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559e61384890 con 0x559e616fe9f0 2024-03-11T19:11:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.809+0000 7f83ef7fe640 10 client.15870 handle_command_reply: tid=0 2024-03-11T19:11:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.809+0000 7f8419579740 1 client.15870 shutdown 2024-03-11T19:11:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.809+0000 7f8419579740 1 -- 172.21.15.29:0/2111237271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 msgr2=0x559e616a04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.809+0000 7f8419579740 1 --2- 172.21.15.29:0/2111237271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 0x559e616a04d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f84000247c0 tx=0x7f8400012e70 comp rx=0 tx=0).stop 2024-03-11T19:11:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.809+0000 7f8419579740 1 -- 172.21.15.29:0/2111237271 shutdown_connections 2024-03-11T19:11:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.809+0000 7f8419579740 1 --2- 172.21.15.29:0/2111237271 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x559e616fe9f0 0x559e61700e10 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.809+0000 7f8419579740 1 --2- 172.21.15.29:0/2111237271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559e61695dd0 0x559e616fcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.809+0000 7f8419579740 1 --2- 172.21.15.29:0/2111237271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e61690c60 0x559e616a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.809+0000 7f83ef7fe640 0 client.15870 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:11:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.809+0000 7f8419579740 1 --2- 172.21.15.29:0/2111237271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e61690270 0x559e616a04d0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.809+0000 7f8419579740 1 -- 172.21.15.29:0/2111237271 >> 172.21.15.29:0/2111237271 conn(0x559e6168b7e0 msgr2=0x559e6168ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.809+0000 7f8419579740 1 -- 172.21.15.29:0/2111237271 shutdown_connections 2024-03-11T19:11:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.809+0000 7f8419579740 1 -- 172.21.15.29:0/2111237271 wait complete. 2024-03-11T19:11:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.809+0000 7f8419579740 20 client.15870 trim_cache size 0 max 16384 2024-03-11T19:11:39.821 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:11:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.810+0000 7f84169a1640 1 -- 172.21.15.29:0/916223111 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f83f80a1a30 msgr2=0x7f83f80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.810+0000 7f84169a1640 1 --2- 172.21.15.29:0/916223111 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f83f80a1a30 0x7f83f80a3ef0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8408032460 tx=0x7f8408032ec0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.810+0000 7f84169a1640 1 -- 172.21.15.29:0/916223111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84101023a0 msgr2=0x7f8410197670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.810+0000 7f84169a1640 1 --2- 172.21.15.29:0/916223111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84101023a0 0x7f8410197670 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f840400d6d0 tx=0x7f840400db90 comp rx=0 tx=0).stop 2024-03-11T19:11:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.810+0000 7f84169a1640 1 -- 172.21.15.29:0/916223111 shutdown_connections 2024-03-11T19:11:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.810+0000 7f84169a1640 1 --2- 172.21.15.29:0/916223111 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8410102d60 0x7f841019e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.810+0000 7f84169a1640 1 --2- 172.21.15.29:0/916223111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84101023a0 0x7f8410197670 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.810+0000 7f84169a1640 1 --2- 172.21.15.29:0/916223111 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f83f80a1a30 0x7f83f80a3ef0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.810+0000 7f84169a1640 1 --2- 172.21.15.29:0/916223111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84101011b0 0x7f8410197130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.810+0000 7f84169a1640 1 -- 172.21.15.29:0/916223111 >> 172.21.15.29:0/916223111 conn(0x7f84100fc720 msgr2=0x7f8410105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.810+0000 7f84169a1640 1 -- 172.21.15.29:0/916223111 shutdown_connections 2024-03-11T19:11:39.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.810+0000 7f84169a1640 1 -- 172.21.15.29:0/916223111 wait complete. 2024-03-11T19:11:39.824 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:11:39.824 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:11:39.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:39 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3440139609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:11:39.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.981+0000 7fb39cc4e640 1 -- 172.21.15.29:0/1685415257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3981005c0 msgr2=0x7fb3981009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.981+0000 7fb39cc4e640 1 --2- 172.21.15.29:0/1685415257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3981005c0 0x7fb3981009e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb388009900 tx=0x7fb38802bc30 comp rx=0 tx=0).stop 2024-03-11T19:11:39.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.981+0000 7fb39cc4e640 1 -- 172.21.15.29:0/1685415257 shutdown_connections 2024-03-11T19:11:39.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.981+0000 7fb39cc4e640 1 --2- 172.21.15.29:0/1685415257 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb398101c90 0x7fb398108520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.981+0000 7fb39cc4e640 1 --2- 172.21.15.29:0/1685415257 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3981012d0 0x7fb398101750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.981+0000 7fb39cc4e640 1 --2- 172.21.15.29:0/1685415257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3981005c0 0x7fb3981009e0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.981+0000 7fb39cc4e640 1 -- 172.21.15.29:0/1685415257 >> 172.21.15.29:0/1685415257 conn(0x7fb3980fad70 msgr2=0x7fb3980fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.981+0000 7fb39cc4e640 1 -- 172.21.15.29:0/1685415257 shutdown_connections 2024-03-11T19:11:39.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.981+0000 7fb39cc4e640 1 -- 172.21.15.29:0/1685415257 wait complete. 2024-03-11T19:11:39.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.982+0000 7fb39cc4e640 1 Processor -- start 2024-03-11T19:11:39.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.982+0000 7fb39cc4e640 1 -- start start 2024-03-11T19:11:39.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.982+0000 7fb39cc4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3981005c0 0x7fb39810a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.982+0000 7fb39cc4e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3981012d0 0x7fb398108fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.982+0000 7fb39cc4e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb398101c90 0x7fb3981094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.983+0000 7fb39cc4e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb398109c40 con 0x7fb3981005c0 2024-03-11T19:11:39.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.983+0000 7fb39cc4e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb398109db0 con 0x7fb398101c90 2024-03-11T19:11:39.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.983+0000 7fb39cc4e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3981a11e0 con 0x7fb3981012d0 2024-03-11T19:11:39.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.983+0000 7fb39659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3981005c0 0x7fb39810a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.983+0000 7fb39659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3981005c0 0x7fb39810a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46166/0 (socket says 172.21.15.29:46166) 2024-03-11T19:11:39.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.983+0000 7fb39659c640 1 -- 172.21.15.29:0/2069808955 learned_addr learned my addr 172.21.15.29:0/2069808955 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:39.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.983+0000 7fb395d9b640 1 --2- 172.21.15.29:0/2069808955 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3981012d0 0x7fb398108fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.983+0000 7fb39659c640 1 -- 172.21.15.29:0/2069808955 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3981012d0 msgr2=0x7fb398108fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.983+0000 7fb39659c640 1 --2- 172.21.15.29:0/2069808955 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3981012d0 0x7fb398108fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.983+0000 7fb39659c640 1 -- 172.21.15.29:0/2069808955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb398101c90 msgr2=0x7fb3981094e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:11:39.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.983+0000 7fb39659c640 1 --2- 172.21.15.29:0/2069808955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb398101c90 0x7fb3981094e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.983+0000 7fb39659c640 1 -- 172.21.15.29:0/2069808955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb388009560 con 0x7fb3981005c0 2024-03-11T19:11:39.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.984+0000 7fb39659c640 1 --2- 172.21.15.29:0/2069808955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3981005c0 0x7fb39810a960 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fb38802c240 tx=0x7fb3880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.984+0000 7fb3877fe640 1 -- 172.21.15.29:0/2069808955 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3880044e0 con 0x7fb3981005c0 2024-03-11T19:11:39.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.984+0000 7fb39cc4e640 1 -- 172.21.15.29:0/2069808955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3981a1430 con 0x7fb3981005c0 2024-03-11T19:11:39.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.984+0000 7fb39cc4e640 1 -- 172.21.15.29:0/2069808955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3981a1990 con 0x7fb3981005c0 2024-03-11T19:11:39.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.984+0000 7fb3877fe640 1 -- 172.21.15.29:0/2069808955 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb388003cb0 con 0x7fb3981005c0 2024-03-11T19:11:39.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.985+0000 7fb3877fe640 1 -- 172.21.15.29:0/2069808955 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3880055d0 con 0x7fb3981005c0 2024-03-11T19:11:39.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.987+0000 7fb39f829740 1 Processor -- start 2024-03-11T19:11:39.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.987+0000 7fb3877fe640 1 -- 172.21.15.29:0/2069808955 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb388004dc0 con 0x7fb3981005c0 2024-03-11T19:11:39.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.988+0000 7fb39f829740 1 -- start start 2024-03-11T19:11:39.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.988+0000 7fb39f829740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83feda80 0x55da83fedea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.988+0000 7fb39f829740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83fee470 0x55da83ff2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.988+0000 7fb39f829740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83ff35e0 0x55da83ff5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.988+0000 7fb39f829740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da83fa1960 con 0x55da83fee470 2024-03-11T19:11:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.988+0000 7fb39f829740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da83f55310 con 0x55da83ff35e0 2024-03-11T19:11:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.988+0000 7fb39f829740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da83f52840 con 0x55da83feda80 2024-03-11T19:11:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.988+0000 7fb39659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83fee470 0x55da83ff2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.988+0000 7fb395d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83ff35e0 0x55da83ff5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.988+0000 7fb396d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83feda80 0x55da83fedea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.988+0000 7fb396d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83feda80 0x55da83fedea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:51620/0 (socket says 172.21.15.29:51620) 2024-03-11T19:11:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.988+0000 7fb396d9d640 1 -- 172.21.15.29:0/3374524428 learned_addr learned my addr 172.21.15.29:0/3374524428 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:39.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.988+0000 7fb3877fe640 1 --2- 172.21.15.29:0/2069808955 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb3800a1ab0 0x7fb3800a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.988+0000 7fb39659c640 1 -- 172.21.15.29:0/3374524428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83feda80 msgr2=0x55da83fedea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.988+0000 7fb39659c640 1 --2- 172.21.15.29:0/3374524428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83feda80 0x55da83fedea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.989+0000 7fb39659c640 1 -- 172.21.15.29:0/3374524428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83ff35e0 msgr2=0x55da83ff5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.989+0000 7fb39659c640 1 --2- 172.21.15.29:0/3374524428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83ff35e0 0x55da83ff5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.989+0000 7fb39659c640 1 -- 172.21.15.29:0/3374524428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da83f10060 con 0x55da83fee470 2024-03-11T19:11:39.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.989+0000 7fb396d9d640 1 --2- 172.21.15.29:0/2069808955 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb3800a1ab0 0x7fb3800a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.989+0000 7fb3877fe640 1 -- 172.21.15.29:0/2069808955 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb38802de40 con 0x7fb3981005c0 2024-03-11T19:11:39.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.989+0000 7fb39659c640 1 --2- 172.21.15.29:0/3374524428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83fee470 0x55da83ff2f70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fb3880ed2f0 tx=0x7fb38804d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ead8c37bceb4773a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.989+0000 7fb396d9d640 1 --2- 172.21.15.29:0/2069808955 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb3800a1ab0 0x7fb3800a3f70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55da83feecd0 tx=0x7fb390004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.989+0000 7fb384ff9640 1 -- 172.21.15.29:0/3374524428 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3880bc300 con 0x55da83fee470 2024-03-11T19:11:39.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.989+0000 7fb384ff9640 1 -- 172.21.15.29:0/3374524428 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb3880bc120 con 0x55da83fee470 2024-03-11T19:11:39.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.989+0000 7fb384ff9640 1 -- 172.21.15.29:0/3374524428 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3880bc7c0 con 0x55da83fee470 2024-03-11T19:11:39.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.990+0000 7fb39f829740 1 -- 172.21.15.29:0/3374524428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83fee470 msgr2=0x55da83ff2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:39.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.990+0000 7fb39f829740 1 --2- 172.21.15.29:0/3374524428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83fee470 0x55da83ff2f70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fb3880ed2f0 tx=0x7fb38804d000 comp rx=0 tx=0).stop 2024-03-11T19:11:39.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.990+0000 7fb39f829740 1 -- 172.21.15.29:0/3374524428 shutdown_connections 2024-03-11T19:11:39.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.990+0000 7fb39f829740 1 --2- 172.21.15.29:0/3374524428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83ff35e0 0x55da83ff5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.990+0000 7fb39f829740 1 --2- 172.21.15.29:0/3374524428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83fee470 0x55da83ff2f70 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.990+0000 7fb39f829740 1 --2- 172.21.15.29:0/3374524428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83feda80 0x55da83fedea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.990+0000 7fb39f829740 1 -- 172.21.15.29:0/3374524428 >> 172.21.15.29:0/3374524428 conn(0x55da83fe07e0 msgr2=0x55da83fe0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:39.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.991+0000 7fb39f829740 1 -- 172.21.15.29:0/3374524428 shutdown_connections 2024-03-11T19:11:39.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.991+0000 7fb39f829740 1 -- 172.21.15.29:0/3374524428 wait complete. 2024-03-11T19:11:39.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.991+0000 7fb39f829740 1 -- start start 2024-03-11T19:11:39.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.991+0000 7fb39f829740 1 Processor -- start 2024-03-11T19:11:39.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.991+0000 7fb39f829740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83feda80 0x55da83ff78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.991+0000 7fb39f829740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83fee470 0x55da83ff7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.991+0000 7fb39f829740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83ff35e0 0x55da83ffee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:39.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.992+0000 7fb39f829740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da83f4e020 con 0x55da83ff35e0 2024-03-11T19:11:39.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.992+0000 7fb39f829740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da83c21430 con 0x55da83fee470 2024-03-11T19:11:39.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.992+0000 7fb39f829740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da83bdde00 con 0x55da83feda80 2024-03-11T19:11:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.992+0000 7fb39f829740 10 client.0 _ll_register_callbacks cb 0x55da83ef5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.992+0000 7fb39659c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83feda80 0x55da83ff78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.992+0000 7fb39659c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83feda80 0x55da83ff78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:51622/0 (socket says 172.21.15.29:51622) 2024-03-11T19:11:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.992+0000 7fb39659c640 1 -- 172.21.15.29:0/3546631348 learned_addr learned my addr 172.21.15.29:0/3546631348 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.992+0000 7fb395d9b640 1 --2- 172.21.15.29:0/3546631348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83fee470 0x55da83ff7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.992+0000 7fb396d9d640 1 --2- 172.21.15.29:0/3546631348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83ff35e0 0x55da83ffee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.993+0000 7fb39659c640 1 -- 172.21.15.29:0/3546631348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83fee470 msgr2=0x55da83ff7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.993+0000 7fb39659c640 1 --2- 172.21.15.29:0/3546631348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83fee470 0x55da83ff7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.993+0000 7fb39659c640 1 -- 172.21.15.29:0/3546631348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83ff35e0 msgr2=0x55da83ffee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.993+0000 7fb39659c640 1 --2- 172.21.15.29:0/3546631348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83ff35e0 0x55da83ffee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.993+0000 7fb39659c640 1 -- 172.21.15.29:0/3546631348 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3981a1990 con 0x55da83feda80 2024-03-11T19:11:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.993+0000 7fb39659c640 1 --2- 172.21.15.29:0/3546631348 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83feda80 0x55da83ff78f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb3880bc290 tx=0x7fb388009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b44695183ab568b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:39.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.993+0000 7fb37dffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:11:39.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.994+0000 7fb37dffb640 1 -- 172.21.15.29:0/3546631348 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3880bc300 con 0x55da83feda80 2024-03-11T19:11:40.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.994+0000 7fb37dffb640 1 -- 172.21.15.29:0/3546631348 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb388003e20 con 0x55da83feda80 2024-03-11T19:11:40.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.994+0000 7fb39f829740 1 -- 172.21.15.29:0/3546631348 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55da83c4d7e0 con 0x55da83feda80 2024-03-11T19:11:40.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.994+0000 7fb37dffb640 1 -- 172.21.15.29:0/3546631348 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3880c0910 con 0x55da83feda80 2024-03-11T19:11:40.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.994+0000 7fb39f829740 10 client.25474 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:40.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.994+0000 7fb39f829740 1 -- 172.21.15.29:0/3546631348 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55da83f4c6a0 con 0x55da83feda80 2024-03-11T19:11:40.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.995+0000 7fb37dffb640 1 -- 172.21.15.29:0/3546631348 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55da83c4d7e0 con 0x55da83feda80 2024-03-11T19:11:40.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.995+0000 7fb37dffb640 1 -- 172.21.15.29:0/3546631348 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb38803c070 con 0x55da83feda80 2024-03-11T19:11:40.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.995+0000 7fb39f829740 10 client.25474 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:40.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.995+0000 7fb39f829740 10 client.25474 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:11:40.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.995+0000 7fb39f829740 20 client.25474 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:40.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.995+0000 7fb39f829740 1 --2- 172.21.15.29:0/3546631348 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55da840037e0 0x55da84005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.995+0000 7fb39f829740 4 client.25474 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:11:40.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.995+0000 7fb39f829740 1 -- 172.21.15.29:0/3546631348 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55da83c8a720 con 0x55da840037e0 2024-03-11T19:11:40.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.996+0000 7fb395d9b640 1 --2- 172.21.15.29:0/3546631348 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55da840037e0 0x55da84005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.996+0000 7fb37dffb640 10 client.25474 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:11:40.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.996+0000 7fb395d9b640 1 --2- 172.21.15.29:0/3546631348 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55da840037e0 0x55da84005c00 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=20c2bd4d9f2a05ce server_cookie=74f07f51dead83e6 in_seq=0 out_seq=0 2024-03-11T19:11:40.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.997+0000 7fb37dffb640 1 -- 172.21.15.29:0/3546631348 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55da83c8a720 con 0x55da840037e0 2024-03-11T19:11:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.997+0000 7fb37dffb640 10 client.25474 handle_command_reply: tid=0 2024-03-11T19:11:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.997+0000 7fb39f829740 1 client.25474 shutdown 2024-03-11T19:11:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.998+0000 7fb39f829740 1 -- 172.21.15.29:0/3546631348 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83feda80 msgr2=0x55da83ff78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.998+0000 7fb39f829740 1 --2- 172.21.15.29:0/3546631348 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83feda80 0x55da83ff78f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb3880bc290 tx=0x7fb388009260 comp rx=0 tx=0).stop 2024-03-11T19:11:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.998+0000 7fb39f829740 1 -- 172.21.15.29:0/3546631348 shutdown_connections 2024-03-11T19:11:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.998+0000 7fb39f829740 1 --2- 172.21.15.29:0/3546631348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83ff35e0 0x55da83ffee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.998+0000 7fb39f829740 1 --2- 172.21.15.29:0/3546631348 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55da840037e0 0x55da84005c00 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.998+0000 7fb39f829740 1 --2- 172.21.15.29:0/3546631348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83fee470 0x55da83ff7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.998+0000 7fb39f829740 1 --2- 172.21.15.29:0/3546631348 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83feda80 0x55da83ff78f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.998+0000 7fb39f829740 1 -- 172.21.15.29:0/3546631348 >> 172.21.15.29:0/3546631348 conn(0x55da83fe07e0 msgr2=0x55da83fdfb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.998+0000 7fb37dffb640 0 client.25474 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:11:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.998+0000 7fb39f829740 1 -- 172.21.15.29:0/3546631348 shutdown_connections 2024-03-11T19:11:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.998+0000 7fb39f829740 1 -- 172.21.15.29:0/3546631348 wait complete. 2024-03-11T19:11:40.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:39.998+0000 7fb39f829740 20 client.25474 trim_cache size 0 max 16384 2024-03-11T19:11:40.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.019+0000 7fb39f829740 1 Processor -- start 2024-03-11T19:11:40.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.019+0000 7fb39f829740 1 -- start start 2024-03-11T19:11:40.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.019+0000 7fb39f829740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83feda80 0x55da83fef8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.019+0000 7fb39f829740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83fee470 0x55da83ffee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.019+0000 7fb39f829740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83ff35e0 0x55da83fefe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.019+0000 7fb39f829740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da83ed3130 con 0x55da83fee470 2024-03-11T19:11:40.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.019+0000 7fb39f829740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da83c4d810 con 0x55da83feda80 2024-03-11T19:11:40.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.019+0000 7fb39f829740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da83ff5f30 con 0x55da83ff35e0 2024-03-11T19:11:40.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.019+0000 7fb395d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83fee470 0x55da83ffee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.019+0000 7fb395d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83fee470 0x55da83ffee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46196/0 (socket says 172.21.15.29:46196) 2024-03-11T19:11:40.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.019+0000 7fb395d9b640 1 -- 172.21.15.29:0/3798434866 learned_addr learned my addr 172.21.15.29:0/3798434866 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:40.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.020+0000 7fb39659c640 1 --2- 172.21.15.29:0/3798434866 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83feda80 0x55da83fef8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.020+0000 7fb396d9d640 1 --2- 172.21.15.29:0/3798434866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83ff35e0 0x55da83fefe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.020+0000 7fb395d9b640 1 -- 172.21.15.29:0/3798434866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83ff35e0 msgr2=0x55da83fefe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:40.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.020+0000 7fb395d9b640 1 --2- 172.21.15.29:0/3798434866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83ff35e0 0x55da83fefe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.020+0000 7fb395d9b640 1 -- 172.21.15.29:0/3798434866 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83feda80 msgr2=0x55da83fef8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:40.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.020+0000 7fb395d9b640 1 --2- 172.21.15.29:0/3798434866 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83feda80 0x55da83fef8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.020+0000 7fb395d9b640 1 -- 172.21.15.29:0/3798434866 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da83ee8c20 con 0x55da83fee470 2024-03-11T19:11:40.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.020+0000 7fb39659c640 1 --2- 172.21.15.29:0/3798434866 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83feda80 0x55da83fef8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:40.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.020+0000 7fb395d9b640 1 --2- 172.21.15.29:0/3798434866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83fee470 0x55da83ffee00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x55da83fe1750 tx=0x7fb38c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c51176bf66d36ad7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:40.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.020+0000 7fb37ffff640 1 -- 172.21.15.29:0/3798434866 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c017040 con 0x55da83fee470 2024-03-11T19:11:40.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.020+0000 7fb37ffff640 1 -- 172.21.15.29:0/3798434866 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb38c01c860 con 0x55da83fee470 2024-03-11T19:11:40.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.021+0000 7fb37ffff640 1 -- 172.21.15.29:0/3798434866 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c004a00 con 0x55da83fee470 2024-03-11T19:11:40.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.021+0000 7fb39f829740 1 -- 172.21.15.29:0/3798434866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83fee470 msgr2=0x55da83ffee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:40.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.021+0000 7fb39f829740 1 --2- 172.21.15.29:0/3798434866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83fee470 0x55da83ffee00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x55da83fe1750 tx=0x7fb38c011810 comp rx=0 tx=0).stop 2024-03-11T19:11:40.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.021+0000 7fb39f829740 1 -- 172.21.15.29:0/3798434866 shutdown_connections 2024-03-11T19:11:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.021+0000 7fb39f829740 1 --2- 172.21.15.29:0/3798434866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83ff35e0 0x55da83fefe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.021+0000 7fb39f829740 1 --2- 172.21.15.29:0/3798434866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83fee470 0x55da83ffee00 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.021+0000 7fb39f829740 1 --2- 172.21.15.29:0/3798434866 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83feda80 0x55da83fef8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.021+0000 7fb39f829740 1 -- 172.21.15.29:0/3798434866 >> 172.21.15.29:0/3798434866 conn(0x55da83fe07e0 msgr2=0x55da8400a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.021+0000 7fb39f829740 1 -- 172.21.15.29:0/3798434866 shutdown_connections 2024-03-11T19:11:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.021+0000 7fb39f829740 1 -- 172.21.15.29:0/3798434866 wait complete. 2024-03-11T19:11:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.022+0000 7fb39f829740 1 -- start start 2024-03-11T19:11:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.022+0000 7fb39f829740 1 Processor -- start 2024-03-11T19:11:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.022+0000 7fb39f829740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83feda80 0x55da83ffebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.022+0000 7fb39f829740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83fee470 0x55da83ffd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.022+0000 7fb39f829740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83ff35e0 0x55da83ffd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.022+0000 7fb39f829740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da83ffdd20 con 0x55da83ff35e0 2024-03-11T19:11:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.022+0000 7fb39f829740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da83ffde90 con 0x55da83feda80 2024-03-11T19:11:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.022+0000 7fb39f829740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da83ffe050 con 0x55da83fee470 2024-03-11T19:11:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.022+0000 7fb39f829740 10 client.0 _ll_register_callbacks cb 0x7fb3880f5060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.022+0000 7fb396d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83ff35e0 0x55da83ffd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.022+0000 7fb395d9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83fee470 0x55da83ffd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.022+0000 7fb395d9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83fee470 0x55da83ffd200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:51632/0 (socket says 172.21.15.29:51632) 2024-03-11T19:11:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.022+0000 7fb396d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83ff35e0 0x55da83ffd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46202/0 (socket says 172.21.15.29:46202) 2024-03-11T19:11:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.022+0000 7fb395d9b640 1 -- 172.21.15.29:0/439642831 learned_addr learned my addr 172.21.15.29:0/439642831 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.022+0000 7fb39659c640 1 --2- 172.21.15.29:0/439642831 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83feda80 0x55da83ffebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.023+0000 7fb39659c640 1 -- 172.21.15.29:0/439642831 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83fee470 msgr2=0x55da83ffd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.023+0000 7fb39659c640 1 --2- 172.21.15.29:0/439642831 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83fee470 0x55da83ffd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.023+0000 7fb39659c640 1 -- 172.21.15.29:0/439642831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83ff35e0 msgr2=0x55da83ffd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.023+0000 7fb39659c640 1 --2- 172.21.15.29:0/439642831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83ff35e0 0x55da83ffd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.023+0000 7fb39659c640 1 -- 172.21.15.29:0/439642831 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55da83f4c6a0 con 0x55da83feda80 2024-03-11T19:11:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.023+0000 7fb395d9b640 1 --2- 172.21.15.29:0/439642831 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83fee470 0x55da83ffd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.023+0000 7fb39659c640 1 --2- 172.21.15.29:0/439642831 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83feda80 0x55da83ffebf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb3880bc4c0 tx=0x7fb388004bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c874d0b3b89e936 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:40.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.023+0000 7fb37dffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:11:40.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.024+0000 7fb37dffb640 1 -- 172.21.15.29:0/439642831 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3880bc300 con 0x55da83feda80 2024-03-11T19:11:40.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.024+0000 7fb37dffb640 1 -- 172.21.15.29:0/439642831 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb388003e20 con 0x55da83feda80 2024-03-11T19:11:40.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.024+0000 7fb39f829740 1 -- 172.21.15.29:0/439642831 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb3880bc120 con 0x55da83feda80 2024-03-11T19:11:40.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.024+0000 7fb37dffb640 1 -- 172.21.15.29:0/439642831 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3880bc590 con 0x55da83feda80 2024-03-11T19:11:40.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.024+0000 7fb39f829740 10 client.25427 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:40.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.025+0000 7fb39f829740 1 -- 172.21.15.29:0/439642831 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7fb38c00b2c0 con 0x55da83feda80 2024-03-11T19:11:40.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.025+0000 7fb37dffb640 1 -- 172.21.15.29:0/439642831 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb3880bc120 con 0x55da83feda80 2024-03-11T19:11:40.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.025+0000 7fb37dffb640 1 -- 172.21.15.29:0/439642831 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb38803c070 con 0x55da83feda80 2024-03-11T19:11:40.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.025+0000 7fb39f829740 10 client.25427 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:40.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.025+0000 7fb39f829740 10 client.25427 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:11:40.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.025+0000 7fb39f829740 20 client.25427 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:40.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.025+0000 7fb39f829740 1 --2- 172.21.15.29:0/439642831 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55da8405b170 0x55da8405d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.025+0000 7fb39f829740 4 client.25427 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:11:40.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.025+0000 7fb39f829740 1 -- 172.21.15.29:0/439642831 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55da83cd9890 con 0x55da8405b170 2024-03-11T19:11:40.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.026+0000 7fb395d9b640 1 --2- 172.21.15.29:0/439642831 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55da8405b170 0x55da8405d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.026+0000 7fb37dffb640 10 client.25427 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:11:40.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.026+0000 7fb395d9b640 1 --2- 172.21.15.29:0/439642831 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55da8405b170 0x55da8405d590 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=388610c0f1545811 server_cookie=832c61dba0e9ee0b in_seq=0 out_seq=0 2024-03-11T19:11:40.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.026+0000 7fb37dffb640 1 -- 172.21.15.29:0/439642831 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55da83cd9890 con 0x55da8405b170 2024-03-11T19:11:40.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.026+0000 7fb37dffb640 10 client.25427 handle_command_reply: tid=0 2024-03-11T19:11:40.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.026+0000 7fb39f829740 1 client.25427 shutdown 2024-03-11T19:11:40.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.026+0000 7fb39f829740 1 -- 172.21.15.29:0/439642831 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83feda80 msgr2=0x55da83ffebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:40.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.026+0000 7fb39f829740 1 --2- 172.21.15.29:0/439642831 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83feda80 0x55da83ffebf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb3880bc4c0 tx=0x7fb388004bc0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.027+0000 7fb39f829740 1 -- 172.21.15.29:0/439642831 shutdown_connections 2024-03-11T19:11:40.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.027+0000 7fb39f829740 1 --2- 172.21.15.29:0/439642831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55da83ff35e0 0x55da83ffd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.027+0000 7fb39f829740 1 --2- 172.21.15.29:0/439642831 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da83fee470 0x55da83ffd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.027+0000 7fb39f829740 1 --2- 172.21.15.29:0/439642831 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55da8405b170 0x55da8405d590 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.027+0000 7fb39f829740 1 --2- 172.21.15.29:0/439642831 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da83feda80 0x55da83ffebf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.027+0000 7fb37dffb640 0 client.25427 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:11:40.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.027+0000 7fb39f829740 1 -- 172.21.15.29:0/439642831 >> 172.21.15.29:0/439642831 conn(0x55da83fe07e0 msgr2=0x55da83fef8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:40.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.027+0000 7fb39f829740 1 -- 172.21.15.29:0/439642831 shutdown_connections 2024-03-11T19:11:40.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.027+0000 7fb39f829740 1 -- 172.21.15.29:0/439642831 wait complete. 2024-03-11T19:11:40.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.027+0000 7fb39f829740 20 client.25427 trim_cache size 0 max 16384 2024-03-11T19:11:40.034 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:11:40.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.027+0000 7fb39cc4e640 1 -- 172.21.15.29:0/2069808955 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb3800a1ab0 msgr2=0x7fb3800a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:40.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.027+0000 7fb39cc4e640 1 --2- 172.21.15.29:0/2069808955 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb3800a1ab0 0x7fb3800a3f70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55da83feecd0 tx=0x7fb390004a40 comp rx=0 tx=0).stop 2024-03-11T19:11:40.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.027+0000 7fb39cc4e640 1 -- 172.21.15.29:0/2069808955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3981005c0 msgr2=0x7fb39810a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:40.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.027+0000 7fb39cc4e640 1 --2- 172.21.15.29:0/2069808955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3981005c0 0x7fb39810a960 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fb38802c240 tx=0x7fb3880042f0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.028+0000 7fb39cc4e640 1 -- 172.21.15.29:0/2069808955 shutdown_connections 2024-03-11T19:11:40.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.028+0000 7fb39cc4e640 1 --2- 172.21.15.29:0/2069808955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb398101c90 0x7fb3981094e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.028+0000 7fb39cc4e640 1 --2- 172.21.15.29:0/2069808955 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3981012d0 0x7fb398108fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.028+0000 7fb39cc4e640 1 --2- 172.21.15.29:0/2069808955 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb3800a1ab0 0x7fb3800a3f70 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.028+0000 7fb39cc4e640 1 --2- 172.21.15.29:0/2069808955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3981005c0 0x7fb39810a960 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.028+0000 7fb39cc4e640 1 -- 172.21.15.29:0/2069808955 >> 172.21.15.29:0/2069808955 conn(0x7fb3980fad70 msgr2=0x7fb3981046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:40.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.028+0000 7fb39cc4e640 1 -- 172.21.15.29:0/2069808955 shutdown_connections 2024-03-11T19:11:40.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.028+0000 7fb39cc4e640 1 -- 172.21.15.29:0/2069808955 wait complete. 2024-03-11T19:11:40.046 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:11:40.046 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:11:40.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:39 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3440139609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:11:40.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.196+0000 7f9da0a9d640 1 -- 172.21.15.29:0/1546876233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d9c104f50 msgr2=0x7f9d9c10b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:40.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.196+0000 7f9da0a9d640 1 --2- 172.21.15.29:0/1546876233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d9c104f50 0x7f9d9c10b7e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f9d94009ca0 tx=0x7f9d9402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.197+0000 7f9da0a9d640 1 -- 172.21.15.29:0/1546876233 shutdown_connections 2024-03-11T19:11:40.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.197+0000 7f9da0a9d640 1 --2- 172.21.15.29:0/1546876233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d9c104f50 0x7f9d9c10b7e0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.197+0000 7f9da0a9d640 1 --2- 172.21.15.29:0/1546876233 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d9c104590 0x7f9d9c104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.197+0000 7f9da0a9d640 1 --2- 172.21.15.29:0/1546876233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d9c1033a0 0x7f9d9c1037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.197+0000 7f9da0a9d640 1 -- 172.21.15.29:0/1546876233 >> 172.21.15.29:0/1546876233 conn(0x7f9d9c0fe930 msgr2=0x7f9d9c100d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:40.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.197+0000 7f9da0a9d640 1 -- 172.21.15.29:0/1546876233 shutdown_connections 2024-03-11T19:11:40.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.197+0000 7f9da0a9d640 1 -- 172.21.15.29:0/1546876233 wait complete. 2024-03-11T19:11:40.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.198+0000 7f9da0a9d640 1 Processor -- start 2024-03-11T19:11:40.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.198+0000 7f9da0a9d640 1 -- start start 2024-03-11T19:11:40.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.198+0000 7f9da0a9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d9c1033a0 0x7f9d9c1992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.198+0000 7f9da0a9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d9c104590 0x7f9d9c199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.198+0000 7f9da0a9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d9c104f50 0x7f9d9c1a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.198+0000 7f9da0a9d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c1a0dd0 con 0x7f9d9c1033a0 2024-03-11T19:11:40.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.198+0000 7f9da0a9d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c1a0f40 con 0x7f9d9c104f50 2024-03-11T19:11:40.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.198+0000 7f9da0a9d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c1a1100 con 0x7f9d9c104590 2024-03-11T19:11:40.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.198+0000 7f9d9a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d9c1033a0 0x7f9d9c1992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.198+0000 7f9d9a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d9c1033a0 0x7f9d9c1992e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46220/0 (socket says 172.21.15.29:46220) 2024-03-11T19:11:40.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.198+0000 7f9d9a59c640 1 -- 172.21.15.29:0/1976959813 learned_addr learned my addr 172.21.15.29:0/1976959813 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:40.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.198+0000 7f9d9ad9d640 1 --2- 172.21.15.29:0/1976959813 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d9c104f50 0x7f9d9c1a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.198+0000 7f9d9a59c640 1 -- 172.21.15.29:0/1976959813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d9c104590 msgr2=0x7f9d9c199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:40.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.198+0000 7f9d9a59c640 1 --2- 172.21.15.29:0/1976959813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d9c104590 0x7f9d9c199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.198+0000 7f9d9a59c640 1 -- 172.21.15.29:0/1976959813 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d9c104f50 msgr2=0x7f9d9c1a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:40.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.198+0000 7f9d9a59c640 1 --2- 172.21.15.29:0/1976959813 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d9c104f50 0x7f9d9c1a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.198+0000 7f9d9a59c640 1 -- 172.21.15.29:0/1976959813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d94009900 con 0x7f9d9c1033a0 2024-03-11T19:11:40.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.199+0000 7f9d9ad9d640 1 --2- 172.21.15.29:0/1976959813 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d9c104f50 0x7f9d9c1a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:40.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.199+0000 7f9d9a59c640 1 --2- 172.21.15.29:0/1976959813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d9c1033a0 0x7f9d9c1992e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f9d8400e9c0 tx=0x7f9d8400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:40.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.199+0000 7f9d937fe640 1 -- 172.21.15.29:0/1976959813 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d840096e0 con 0x7f9d9c1033a0 2024-03-11T19:11:40.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.199+0000 7f9da0a9d640 1 -- 172.21.15.29:0/1976959813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d9c1a13e0 con 0x7f9d9c1033a0 2024-03-11T19:11:40.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.199+0000 7f9d937fe640 1 -- 172.21.15.29:0/1976959813 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9d840044e0 con 0x7f9d9c1033a0 2024-03-11T19:11:40.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.199+0000 7f9d937fe640 1 -- 172.21.15.29:0/1976959813 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d84010430 con 0x7f9d9c1033a0 2024-03-11T19:11:40.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.199+0000 7f9da0a9d640 1 -- 172.21.15.29:0/1976959813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9c1a18a0 con 0x7f9d9c1033a0 2024-03-11T19:11:40.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.202+0000 7f9d937fe640 1 -- 172.21.15.29:0/1976959813 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9d84003f80 con 0x7f9d9c1033a0 2024-03-11T19:11:40.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.203+0000 7f9da3677740 1 Processor -- start 2024-03-11T19:11:40.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.203+0000 7f9da3677740 1 -- start start 2024-03-11T19:11:40.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.204+0000 7f9d937fe640 1 --2- 172.21.15.29:0/1976959813 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9d880a1a30 0x7f9d880a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.204+0000 7f9da3677740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b69a80 0x5599d0b69ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.204+0000 7f9da3677740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b6a470 0x5599d0b6ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.204+0000 7f9da3677740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5599d0b6f5e0 0x5599d0b71a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.204+0000 7f9da3677740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d0b1d960 con 0x5599d0b69a80 2024-03-11T19:11:40.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.204+0000 7f9da3677740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d0ad1310 con 0x5599d0b6f5e0 2024-03-11T19:11:40.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.204+0000 7f9da3677740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d0ace840 con 0x5599d0b6a470 2024-03-11T19:11:40.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.204+0000 7f9d937fe640 1 -- 172.21.15.29:0/1976959813 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9d84014070 con 0x7f9d9c1033a0 2024-03-11T19:11:40.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.204+0000 7f9d9a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b69a80 0x5599d0b69ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.204+0000 7f9d9a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b69a80 0x5599d0b69ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46222/0 (socket says 172.21.15.29:46222) 2024-03-11T19:11:40.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.204+0000 7f9d9a59c640 1 -- 172.21.15.29:0/1196919834 learned_addr learned my addr 172.21.15.29:0/1196919834 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:40.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.204+0000 7f9d9ad9d640 1 --2- 172.21.15.29:0/1976959813 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9d880a1a30 0x7f9d880a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.204+0000 7f9d9a59c640 1 -- 172.21.15.29:0/1196919834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b6a470 msgr2=0x5599d0b6ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:40.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.204+0000 7f9d9a59c640 1 --2- 172.21.15.29:0/1196919834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b6a470 0x5599d0b6ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.204+0000 7f9d9a59c640 1 -- 172.21.15.29:0/1196919834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5599d0b6f5e0 msgr2=0x5599d0b71a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:40.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.204+0000 7f9d9a59c640 1 --2- 172.21.15.29:0/1196919834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5599d0b6f5e0 0x5599d0b71a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.204+0000 7f9d9a59c640 1 -- 172.21.15.29:0/1196919834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599d0a8c060 con 0x5599d0b69a80 2024-03-11T19:11:40.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.205+0000 7f9d9a59c640 1 --2- 172.21.15.29:0/1196919834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b69a80 0x5599d0b69ea0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009fd0 tx=0x7f9d8409a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74cf65da651400b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:40.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.205+0000 7f9d90ff9640 1 -- 172.21.15.29:0/1196919834 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8409a7a0 con 0x5599d0b69a80 2024-03-11T19:11:40.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.205+0000 7f9d90ff9640 1 -- 172.21.15.29:0/1196919834 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9d8409e8f0 con 0x5599d0b69a80 2024-03-11T19:11:40.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.205+0000 7f9d90ff9640 1 -- 172.21.15.29:0/1196919834 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d84004230 con 0x5599d0b69a80 2024-03-11T19:11:40.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.206+0000 7f9da3677740 1 -- 172.21.15.29:0/1196919834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b69a80 msgr2=0x5599d0b69ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:40.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.206+0000 7f9d9ad9d640 1 --2- 172.21.15.29:0/1976959813 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9d880a1a30 0x7f9d880a3ef0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9d94005e30 tx=0x7f9d94009380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:40.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.206+0000 7f9da3677740 1 --2- 172.21.15.29:0/1196919834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b69a80 0x5599d0b69ea0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009fd0 tx=0x7f9d8409a100 comp rx=0 tx=0).stop 2024-03-11T19:11:40.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.206+0000 7f9da3677740 1 -- 172.21.15.29:0/1196919834 shutdown_connections 2024-03-11T19:11:40.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.206+0000 7f9da3677740 1 --2- 172.21.15.29:0/1196919834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5599d0b6f5e0 0x5599d0b71a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.206+0000 7f9da3677740 1 --2- 172.21.15.29:0/1196919834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b6a470 0x5599d0b6ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.206+0000 7f9da3677740 1 --2- 172.21.15.29:0/1196919834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b69a80 0x5599d0b69ea0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.206+0000 7f9da3677740 1 -- 172.21.15.29:0/1196919834 >> 172.21.15.29:0/1196919834 conn(0x5599d0b5c7e0 msgr2=0x5599d0b5cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:40.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.206+0000 7f9da3677740 1 -- 172.21.15.29:0/1196919834 shutdown_connections 2024-03-11T19:11:40.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.206+0000 7f9da3677740 1 -- 172.21.15.29:0/1196919834 wait complete. 2024-03-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.206+0000 7f9da3677740 1 -- start start 2024-03-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.206+0000 7f9da3677740 1 Processor -- start 2024-03-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.207+0000 7f9da3677740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b69a80 0x5599d0b73d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.207+0000 7f9da3677740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5599d0b6a470 0x5599d0b74260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.207+0000 7f9d9a59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b69a80 0x5599d0b73d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.207+0000 7f9d9a59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b69a80 0x5599d0b73d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:51666/0 (socket says 172.21.15.29:51666) 2024-03-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.207+0000 7f9d9a59c640 1 -- 172.21.15.29:0/1024598497 learned_addr learned my addr 172.21.15.29:0/1024598497 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.207+0000 7f9da3677740 1 --2- 172.21.15.29:0/1024598497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b6f5e0 0x5599d0b7b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.207+0000 7f9da3677740 1 -- 172.21.15.29:0/1024598497 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d0aca020 con 0x5599d0b6f5e0 2024-03-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.207+0000 7f9da3677740 1 -- 172.21.15.29:0/1024598497 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d079d430 con 0x5599d0b6a470 2024-03-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.207+0000 7f9da3677740 1 -- 172.21.15.29:0/1024598497 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d0759e00 con 0x5599d0b69a80 2024-03-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.207+0000 7f9da3677740 10 client.0 _ll_register_callbacks cb 0x5599d0a71cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.207+0000 7f9d9ad9d640 1 --2- 172.21.15.29:0/1024598497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b6f5e0 0x5599d0b7b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.208+0000 7f9d9a59c640 1 -- 172.21.15.29:0/1024598497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5599d0b6a470 msgr2=0x5599d0b74260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.208+0000 7f9d9a59c640 1 --2- 172.21.15.29:0/1024598497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5599d0b6a470 0x5599d0b74260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.208+0000 7f9d9a59c640 1 -- 172.21.15.29:0/1024598497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b6f5e0 msgr2=0x5599d0b7b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.208+0000 7f9d9a59c640 1 --2- 172.21.15.29:0/1024598497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b6f5e0 0x5599d0b7b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.208+0000 7f9d9a59c640 1 -- 172.21.15.29:0/1024598497 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9c1a18a0 con 0x5599d0b69a80 2024-03-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.208+0000 7f9d9a59c640 1 --2- 172.21.15.29:0/1024598497 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b69a80 0x5599d0b73d20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9d840058c0 tx=0x7f9d84003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c5932e0d6d236c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.208+0000 7f9d81ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.208+0000 7f9d81ffb640 1 -- 172.21.15.29:0/1024598497 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d84085400 con 0x5599d0b69a80 2024-03-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.208+0000 7f9d81ffb640 1 -- 172.21.15.29:0/1024598497 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9d84085040 con 0x5599d0b69a80 2024-03-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.208+0000 7f9da3677740 1 -- 172.21.15.29:0/1024598497 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5599d07c97e0 con 0x5599d0b69a80 2024-03-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.208+0000 7f9d81ffb640 1 -- 172.21.15.29:0/1024598497 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d8409a7a0 con 0x5599d0b69a80 2024-03-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.208+0000 7f9da3677740 10 client.25486 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.208+0000 7f9da3677740 1 -- 172.21.15.29:0/1024598497 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5599d0ac86a0 con 0x5599d0b69a80 2024-03-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.209+0000 7f9d81ffb640 1 -- 172.21.15.29:0/1024598497 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5599d07c97e0 con 0x5599d0b69a80 2024-03-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.209+0000 7f9d81ffb640 1 -- 172.21.15.29:0/1024598497 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9d8401f060 con 0x5599d0b69a80 2024-03-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.209+0000 7f9da3677740 10 client.25486 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.209+0000 7f9da3677740 10 client.25486 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:11:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.209+0000 7f9da3677740 20 client.25486 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.209+0000 7f9da3677740 1 --2- 172.21.15.29:0/1024598497 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5599d0b7fc40 0x5599d0b82080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.209+0000 7f9da3677740 4 client.25486 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:11:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.209+0000 7f9da3677740 1 -- 172.21.15.29:0/1024598497 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5599d0806720 con 0x5599d0b7fc40 2024-03-11T19:11:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.209+0000 7f9d99d9b640 1 --2- 172.21.15.29:0/1024598497 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5599d0b7fc40 0x5599d0b82080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.209+0000 7f9d81ffb640 10 client.25486 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.209+0000 7f9d99d9b640 1 --2- 172.21.15.29:0/1024598497 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5599d0b7fc40 0x5599d0b82080 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=475ebe521cca56c server_cookie=a9c0f38d4e6a4912 in_seq=0 out_seq=0 2024-03-11T19:11:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.210+0000 7f9d81ffb640 1 -- 172.21.15.29:0/1024598497 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5599d0806720 con 0x5599d0b7fc40 2024-03-11T19:11:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.210+0000 7f9d81ffb640 10 client.25486 handle_command_reply: tid=0 2024-03-11T19:11:40.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.210+0000 7f9da3677740 1 client.25486 shutdown 2024-03-11T19:11:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.210+0000 7f9da3677740 1 -- 172.21.15.29:0/1024598497 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b69a80 msgr2=0x5599d0b73d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.210+0000 7f9da3677740 1 --2- 172.21.15.29:0/1024598497 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b69a80 0x5599d0b73d20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9d840058c0 tx=0x7f9d84003820 comp rx=0 tx=0).stop 2024-03-11T19:11:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.210+0000 7f9da3677740 1 -- 172.21.15.29:0/1024598497 shutdown_connections 2024-03-11T19:11:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.210+0000 7f9da3677740 1 --2- 172.21.15.29:0/1024598497 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5599d0b7fc40 0x5599d0b82080 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.210+0000 7f9da3677740 1 --2- 172.21.15.29:0/1024598497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b6f5e0 0x5599d0b7b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.210+0000 7f9da3677740 1 --2- 172.21.15.29:0/1024598497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5599d0b6a470 0x5599d0b74260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.210+0000 7f9d81ffb640 0 client.25486 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:40.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.210+0000 7f9da3677740 1 --2- 172.21.15.29:0/1024598497 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b69a80 0x5599d0b73d20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.210+0000 7f9da3677740 1 -- 172.21.15.29:0/1024598497 >> 172.21.15.29:0/1024598497 conn(0x5599d0b5c7e0 msgr2=0x5599d0b5bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:40.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.211+0000 7f9da3677740 1 -- 172.21.15.29:0/1024598497 shutdown_connections 2024-03-11T19:11:40.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.211+0000 7f9da3677740 1 -- 172.21.15.29:0/1024598497 wait complete. 2024-03-11T19:11:40.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.211+0000 7f9da3677740 20 client.25486 trim_cache size 0 max 16384 2024-03-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.230+0000 7f9da3677740 1 Processor -- start 2024-03-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.230+0000 7f9da3677740 1 -- start start 2024-03-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.230+0000 7f9da3677740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5599d0b6a470 0x5599d0b5cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.230+0000 7f9da3677740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b6f5e0 0x5599d0b7b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.231+0000 7f9da3677740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b5fa40 0x5599d0b5fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.231+0000 7f9d99d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b6f5e0 0x5599d0b7b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.231+0000 7f9d99d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b6f5e0 0x5599d0b7b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46236/0 (socket says 172.21.15.29:46236) 2024-03-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.231+0000 7f9d99d9b640 1 -- 172.21.15.29:0/84856898 learned_addr learned my addr 172.21.15.29:0/84856898 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.231+0000 7f9da3677740 1 -- 172.21.15.29:0/84856898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d0a4f130 con 0x5599d0b6f5e0 2024-03-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.231+0000 7f9da3677740 1 -- 172.21.15.29:0/84856898 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d0b72130 con 0x5599d0b6a470 2024-03-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.231+0000 7f9da3677740 1 -- 172.21.15.29:0/84856898 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d07c97e0 con 0x5599d0b5fa40 2024-03-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.231+0000 7f9d9a59c640 1 --2- 172.21.15.29:0/84856898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5599d0b6a470 0x5599d0b5cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.231+0000 7f9d9ad9d640 1 --2- 172.21.15.29:0/84856898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b5fa40 0x5599d0b5fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.231+0000 7f9d99d9b640 1 -- 172.21.15.29:0/84856898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b5fa40 msgr2=0x5599d0b5fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.231+0000 7f9d99d9b640 1 --2- 172.21.15.29:0/84856898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b5fa40 0x5599d0b5fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.231+0000 7f9d99d9b640 1 -- 172.21.15.29:0/84856898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5599d0b6a470 msgr2=0x5599d0b5cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.231+0000 7f9d99d9b640 1 --2- 172.21.15.29:0/84856898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5599d0b6a470 0x5599d0b5cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.231+0000 7f9d99d9b640 1 -- 172.21.15.29:0/84856898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599d0a64c20 con 0x5599d0b6f5e0 2024-03-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.231+0000 7f9d9a59c640 1 --2- 172.21.15.29:0/84856898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5599d0b6a470 0x5599d0b5cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.231+0000 7f9d99d9b640 1 --2- 172.21.15.29:0/84856898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b6f5e0 0x5599d0b7b260 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00b880 tx=0x7f9d8c013820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f29dd4fce5a0d322 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.232+0000 7f9d83fff640 1 -- 172.21.15.29:0/84856898 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c018070 con 0x5599d0b6f5e0 2024-03-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.232+0000 7f9d83fff640 1 -- 172.21.15.29:0/84856898 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9d8c0044e0 con 0x5599d0b6f5e0 2024-03-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.232+0000 7f9d83fff640 1 -- 172.21.15.29:0/84856898 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c0047a0 con 0x5599d0b6f5e0 2024-03-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.233+0000 7f9da3677740 1 -- 172.21.15.29:0/84856898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b6f5e0 msgr2=0x5599d0b7b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.233+0000 7f9da3677740 1 --2- 172.21.15.29:0/84856898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b6f5e0 0x5599d0b7b260 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00b880 tx=0x7f9d8c013820 comp rx=0 tx=0).stop 2024-03-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.233+0000 7f9da3677740 1 -- 172.21.15.29:0/84856898 shutdown_connections 2024-03-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.233+0000 7f9da3677740 1 --2- 172.21.15.29:0/84856898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b5fa40 0x5599d0b5fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.233+0000 7f9da3677740 1 --2- 172.21.15.29:0/84856898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b6f5e0 0x5599d0b7b260 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.233+0000 7f9da3677740 1 --2- 172.21.15.29:0/84856898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5599d0b6a470 0x5599d0b5cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.233+0000 7f9da3677740 1 -- 172.21.15.29:0/84856898 >> 172.21.15.29:0/84856898 conn(0x5599d0b5c7e0 msgr2=0x5599d0b57a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.233+0000 7f9da3677740 1 -- 172.21.15.29:0/84856898 shutdown_connections 2024-03-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.233+0000 7f9da3677740 1 -- 172.21.15.29:0/84856898 wait complete. 2024-03-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.233+0000 7f9da3677740 1 -- start start 2024-03-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.233+0000 7f9da3677740 1 Processor -- start 2024-03-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.234+0000 7f9da3677740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b5fa40 0x5599d0b7b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.234+0000 7f9da3677740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5599d0b6a470 0x5599d0b79720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.234+0000 7f9da3677740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b6f5e0 0x5599d0b79c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.234+0000 7f9da3677740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d0b7a270 con 0x5599d0b6f5e0 2024-03-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.234+0000 7f9da3677740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d0b7a3e0 con 0x5599d0b6a470 2024-03-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.234+0000 7f9da3677740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d0b7a5a0 con 0x5599d0b5fa40 2024-03-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.234+0000 7f9da3677740 10 client.0 _ll_register_callbacks cb 0x7f9d84009f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.234+0000 7f9d9a59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b5fa40 0x5599d0b7b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.234+0000 7f9d9ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b6f5e0 0x5599d0b79c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.234+0000 7f9d9a59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b5fa40 0x5599d0b7b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:51682/0 (socket says 172.21.15.29:51682) 2024-03-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.234+0000 7f9d9a59c640 1 -- 172.21.15.29:0/376425022 learned_addr learned my addr 172.21.15.29:0/376425022 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.234+0000 7f9d99d9b640 1 --2- 172.21.15.29:0/376425022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5599d0b6a470 0x5599d0b79720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.234+0000 7f9d9a59c640 1 -- 172.21.15.29:0/376425022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5599d0b6a470 msgr2=0x5599d0b79720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.234+0000 7f9d9a59c640 1 --2- 172.21.15.29:0/376425022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5599d0b6a470 0x5599d0b79720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.234+0000 7f9d9a59c640 1 -- 172.21.15.29:0/376425022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b6f5e0 msgr2=0x5599d0b79c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:40.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.234+0000 7f9d9a59c640 1 --2- 172.21.15.29:0/376425022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b6f5e0 0x5599d0b79c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.234+0000 7f9d9a59c640 1 -- 172.21.15.29:0/376425022 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5599d0ac86a0 con 0x5599d0b5fa40 2024-03-11T19:11:40.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.235+0000 7f9d9a59c640 1 --2- 172.21.15.29:0/376425022 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b5fa40 0x5599d0b7b0c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9d84098a50 tx=0x7f9d8409a8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abfe017d323222be server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:40.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.235+0000 7f9d81ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:11:40.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.235+0000 7f9d81ffb640 1 -- 172.21.15.29:0/376425022 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d84085200 con 0x5599d0b5fa40 2024-03-11T19:11:40.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.235+0000 7f9d81ffb640 1 -- 172.21.15.29:0/376425022 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9d84085040 con 0x5599d0b5fa40 2024-03-11T19:11:40.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.235+0000 7f9da3677740 1 -- 172.21.15.29:0/376425022 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d8409e8f0 con 0x5599d0b5fa40 2024-03-11T19:11:40.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.235+0000 7f9d81ffb640 1 -- 172.21.15.29:0/376425022 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d84085360 con 0x5599d0b5fa40 2024-03-11T19:11:40.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.235+0000 7f9da3677740 10 client.25492 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:40.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.235+0000 7f9da3677740 1 -- 172.21.15.29:0/376425022 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7f9d8c00ba10 con 0x5599d0b5fa40 2024-03-11T19:11:40.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.236+0000 7f9d81ffb640 1 -- 172.21.15.29:0/376425022 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9d840ba110 con 0x5599d0b5fa40 2024-03-11T19:11:40.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.236+0000 7f9d81ffb640 1 -- 172.21.15.29:0/376425022 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9d8401f070 con 0x5599d0b5fa40 2024-03-11T19:11:40.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.236+0000 7f9da3677740 10 client.25492 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:40.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.236+0000 7f9da3677740 10 client.25492 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:11:40.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.236+0000 7f9da3677740 20 client.25492 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:40.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.236+0000 7f9da3677740 1 --2- 172.21.15.29:0/376425022 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5599d0bd7630 0x5599d0bd9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:40.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.236+0000 7f9da3677740 4 client.25492 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:11:40.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.236+0000 7f9da3677740 1 -- 172.21.15.29:0/376425022 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f9d8409e8f0 con 0x5599d0bd7630 2024-03-11T19:11:40.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.236+0000 7f9d99d9b640 1 --2- 172.21.15.29:0/376425022 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5599d0bd7630 0x5599d0bd9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:40.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.237+0000 7f9d99d9b640 1 --2- 172.21.15.29:0/376425022 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5599d0bd7630 0x5599d0bd9a50 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f034226a1cb75bd server_cookie=51f2fad7069d78f in_seq=0 out_seq=0 2024-03-11T19:11:40.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.237+0000 7f9d81ffb640 10 client.25492 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:40.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.237+0000 7f9d81ffb640 1 -- 172.21.15.29:0/376425022 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f9d8409e8f0 con 0x5599d0bd7630 2024-03-11T19:11:40.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.237+0000 7f9d81ffb640 10 client.25492 handle_command_reply: tid=0 2024-03-11T19:11:40.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.237+0000 7f9da3677740 1 client.25492 shutdown 2024-03-11T19:11:40.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.237+0000 7f9da3677740 1 -- 172.21.15.29:0/376425022 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b5fa40 msgr2=0x5599d0b7b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:40.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.237+0000 7f9da3677740 1 --2- 172.21.15.29:0/376425022 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b5fa40 0x5599d0b7b0c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9d84098a50 tx=0x7f9d8409a8c0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.237+0000 7f9da3677740 1 -- 172.21.15.29:0/376425022 shutdown_connections 2024-03-11T19:11:40.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.237+0000 7f9da3677740 1 --2- 172.21.15.29:0/376425022 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5599d0bd7630 0x5599d0bd9a50 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.237+0000 7f9da3677740 1 --2- 172.21.15.29:0/376425022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5599d0b6f5e0 0x5599d0b79c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.237+0000 7f9da3677740 1 --2- 172.21.15.29:0/376425022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5599d0b6a470 0x5599d0b79720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.237+0000 7f9da3677740 1 --2- 172.21.15.29:0/376425022 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599d0b5fa40 0x5599d0b7b0c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.237+0000 7f9d81ffb640 0 client.25492 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:40.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.237+0000 7f9da3677740 1 -- 172.21.15.29:0/376425022 >> 172.21.15.29:0/376425022 conn(0x5599d0b5c7e0 msgr2=0x5599d0b5cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:40.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.237+0000 7f9da3677740 1 -- 172.21.15.29:0/376425022 shutdown_connections 2024-03-11T19:11:40.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.238+0000 7f9da3677740 1 -- 172.21.15.29:0/376425022 wait complete. 2024-03-11T19:11:40.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.238+0000 7f9da3677740 20 client.25492 trim_cache size 0 max 16384 2024-03-11T19:11:40.245 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:11:40.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.238+0000 7f9da0a9d640 1 -- 172.21.15.29:0/1976959813 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9d880a1a30 msgr2=0x7f9d880a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:40.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.238+0000 7f9da0a9d640 1 --2- 172.21.15.29:0/1976959813 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9d880a1a30 0x7f9d880a3ef0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9d94005e30 tx=0x7f9d94009380 comp rx=0 tx=0).stop 2024-03-11T19:11:40.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.238+0000 7f9da0a9d640 1 -- 172.21.15.29:0/1976959813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d9c1033a0 msgr2=0x7f9d9c1992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:40.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.238+0000 7f9da0a9d640 1 --2- 172.21.15.29:0/1976959813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d9c1033a0 0x7f9d9c1992e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f9d8400e9c0 tx=0x7f9d8400ee80 comp rx=0 tx=0).stop 2024-03-11T19:11:40.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.239+0000 7f9da0a9d640 1 -- 172.21.15.29:0/1976959813 shutdown_connections 2024-03-11T19:11:40.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.239+0000 7f9da0a9d640 1 --2- 172.21.15.29:0/1976959813 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d9c104f50 0x7f9d9c1a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.239+0000 7f9da0a9d640 1 --2- 172.21.15.29:0/1976959813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d9c104590 0x7f9d9c199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.239+0000 7f9da0a9d640 1 --2- 172.21.15.29:0/1976959813 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9d880a1a30 0x7f9d880a3ef0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.239+0000 7f9da0a9d640 1 --2- 172.21.15.29:0/1976959813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d9c1033a0 0x7f9d9c1992e0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:40.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.239+0000 7f9da0a9d640 1 -- 172.21.15.29:0/1976959813 >> 172.21.15.29:0/1976959813 conn(0x7f9d9c0fe930 msgr2=0x7f9d9c107930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:40.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.239+0000 7f9da0a9d640 1 -- 172.21.15.29:0/1976959813 shutdown_connections 2024-03-11T19:11:40.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:40.239+0000 7f9da0a9d640 1 -- 172.21.15.29:0/1976959813 wait complete. 2024-03-11T19:11:40.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:39 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3440139609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:11:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:40 smithi029 ceph-mon[32439]: pgmap v407: 97 pgs: 97 active+clean; 4.2 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 184 MiB/s wr, 103 op/s 2024-03-11T19:11:41.260 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:11:41.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:11:41.260 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:11:41.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:40 smithi190 ceph-mon[36292]: pgmap v407: 97 pgs: 97 active+clean; 4.2 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 184 MiB/s wr, 103 op/s 2024-03-11T19:11:41.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:40 smithi170 ceph-mon[35690]: pgmap v407: 97 pgs: 97 active+clean; 4.2 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 184 MiB/s wr, 103 op/s 2024-03-11T19:11:41.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.423+0000 7fb5c18df640 1 -- 172.21.15.29:0/2337343382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5bc1009f0 msgr2=0x7fb5bc100e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:41.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.423+0000 7fb5c18df640 1 --2- 172.21.15.29:0/2337343382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5bc1009f0 0x7fb5bc100e70 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009900 tx=0x7fb5a402bc30 comp rx=0 tx=0).stop 2024-03-11T19:11:41.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.423+0000 7fb5c18df640 1 -- 172.21.15.29:0/2337343382 shutdown_connections 2024-03-11T19:11:41.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.423+0000 7fb5c18df640 1 --2- 172.21.15.29:0/2337343382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5bc1013b0 0x7fb5bc107c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.423+0000 7fb5c18df640 1 --2- 172.21.15.29:0/2337343382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5bc1009f0 0x7fb5bc100e70 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.423+0000 7fb5c18df640 1 --2- 172.21.15.29:0/2337343382 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5bc0ff800 0x7fb5bc0ffc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.423+0000 7fb5c18df640 1 -- 172.21.15.29:0/2337343382 >> 172.21.15.29:0/2337343382 conn(0x7fb5bc0fad70 msgr2=0x7fb5bc0fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:41.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.423+0000 7fb5c18df640 1 -- 172.21.15.29:0/2337343382 shutdown_connections 2024-03-11T19:11:41.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.423+0000 7fb5c18df640 1 -- 172.21.15.29:0/2337343382 wait complete. 2024-03-11T19:11:41.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.424+0000 7fb5c18df640 1 Processor -- start 2024-03-11T19:11:41.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.424+0000 7fb5c18df640 1 -- start start 2024-03-11T19:11:41.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.424+0000 7fb5c18df640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5bc0ff800 0x7fb5bc199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.424+0000 7fb5c18df640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5bc1009f0 0x7fb5bc1998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.424+0000 7fb5c18df640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5bc1013b0 0x7fb5bc1a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.424+0000 7fb5c18df640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc1a0e70 con 0x7fb5bc1013b0 2024-03-11T19:11:41.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.424+0000 7fb5c18df640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc1a0fe0 con 0x7fb5bc1009f0 2024-03-11T19:11:41.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.424+0000 7fb5c18df640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc1a11a0 con 0x7fb5bc0ff800 2024-03-11T19:11:41.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.424+0000 7fb5bb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5bc1013b0 0x7fb5bc1a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.424+0000 7fb5bb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5bc1013b0 0x7fb5bc1a0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46262/0 (socket says 172.21.15.29:46262) 2024-03-11T19:11:41.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.424+0000 7fb5bb7fe640 1 -- 172.21.15.29:0/3560665582 learned_addr learned my addr 172.21.15.29:0/3560665582 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:41.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.424+0000 7fb5b3fff640 1 --2- 172.21.15.29:0/3560665582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5bc1009f0 0x7fb5bc1998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.424+0000 7fb5baffd640 1 --2- 172.21.15.29:0/3560665582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5bc0ff800 0x7fb5bc199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.424+0000 7fb5bb7fe640 1 -- 172.21.15.29:0/3560665582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5bc0ff800 msgr2=0x7fb5bc199380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:41.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.424+0000 7fb5bb7fe640 1 --2- 172.21.15.29:0/3560665582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5bc0ff800 0x7fb5bc199380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.424+0000 7fb5bb7fe640 1 -- 172.21.15.29:0/3560665582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5bc1009f0 msgr2=0x7fb5bc1998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:41.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.424+0000 7fb5bb7fe640 1 --2- 172.21.15.29:0/3560665582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5bc1009f0 0x7fb5bc1998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.424+0000 7fb5bb7fe640 1 -- 172.21.15.29:0/3560665582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a4009560 con 0x7fb5bc1013b0 2024-03-11T19:11:41.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.424+0000 7fb5baffd640 1 --2- 172.21.15.29:0/3560665582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5bc0ff800 0x7fb5bc199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:41.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.425+0000 7fb5bb7fe640 1 --2- 172.21.15.29:0/3560665582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5bc1013b0 0x7fb5bc1a0930 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400eb10 tx=0x7fb5b400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:41.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.425+0000 7fb5c08dd640 1 -- 172.21.15.29:0/3560665582 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b4004b20 con 0x7fb5bc1013b0 2024-03-11T19:11:41.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.425+0000 7fb5c18df640 1 -- 172.21.15.29:0/3560665582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5bc1a1480 con 0x7fb5bc1013b0 2024-03-11T19:11:41.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.425+0000 7fb5c18df640 1 -- 172.21.15.29:0/3560665582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5bc1a19b0 con 0x7fb5bc1013b0 2024-03-11T19:11:41.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.425+0000 7fb5c08dd640 1 -- 172.21.15.29:0/3560665582 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb5b4004c80 con 0x7fb5bc1013b0 2024-03-11T19:11:41.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.425+0000 7fb5c08dd640 1 -- 172.21.15.29:0/3560665582 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b4010690 con 0x7fb5bc1013b0 2024-03-11T19:11:41.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.428+0000 7fb5c458e740 1 Processor -- start 2024-03-11T19:11:41.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.428+0000 7fb5c08dd640 1 -- 172.21.15.29:0/3560665582 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb5b4005020 con 0x7fb5bc1013b0 2024-03-11T19:11:41.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.428+0000 7fb5c458e740 1 -- start start 2024-03-11T19:11:41.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.428+0000 7fb5c458e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2b4e0 0x561488c2b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.428+0000 7fb5c458e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c2c6b0 0x561488c2cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.428+0000 7fb5c458e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2d070 0x561488c33900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.428+0000 7fb5c458e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x561488b9af90 con 0x561488c2b4e0 2024-03-11T19:11:41.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.428+0000 7fb5c458e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x561488b984c0 con 0x561488c2c6b0 2024-03-11T19:11:41.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.428+0000 7fb5c458e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561488b93ca0 con 0x561488c2d070 2024-03-11T19:11:41.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.428+0000 7fb5b3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2b4e0 0x561488c2b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.428+0000 7fb5b3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2b4e0 0x561488c2b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46278/0 (socket says 172.21.15.29:46278) 2024-03-11T19:11:41.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.428+0000 7fb5b3fff640 1 -- 172.21.15.29:0/4226847835 learned_addr learned my addr 172.21.15.29:0/4226847835 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:41.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.428+0000 7fb5baffd640 1 --2- 172.21.15.29:0/4226847835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c2c6b0 0x561488c2cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.429+0000 7fb5b3fff640 1 --2- 172.21.15.29:0/4226847835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2d070 0x561488c33900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.429+0000 7fb5c08dd640 1 --2- 172.21.15.29:0/3560665582 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb5a80a1ab0 0x7fb5a80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.429+0000 7fb5baffd640 1 -- 172.21.15.29:0/4226847835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2d070 msgr2=0x561488c33900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:41.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.429+0000 7fb5baffd640 1 --2- 172.21.15.29:0/4226847835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2d070 0x561488c33900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.429+0000 7fb5baffd640 1 -- 172.21.15.29:0/4226847835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2b4e0 msgr2=0x561488c2b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:41.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.429+0000 7fb5baffd640 1 --2- 172.21.15.29:0/4226847835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2b4e0 0x561488c2b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.429+0000 7fb5baffd640 1 -- 172.21.15.29:0/4226847835 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561488b560b0 con 0x561488c2c6b0 2024-03-11T19:11:41.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.429+0000 7fb5b3fff640 1 --2- 172.21.15.29:0/4226847835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2b4e0 0x561488c2b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:41.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.429+0000 7fb5bb7fe640 1 --2- 172.21.15.29:0/3560665582 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb5a80a1ab0 0x7fb5a80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.429+0000 7fb5c08dd640 1 -- 172.21.15.29:0/3560665582 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb5b4014070 con 0x7fb5bc1013b0 2024-03-11T19:11:41.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.429+0000 7fb5baffd640 1 --2- 172.21.15.29:0/4226847835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c2c6b0 0x561488c2cb30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac009c00 tx=0x7fb5ac0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97dbba5155ddf75f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:41.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.429+0000 7fb5bb7fe640 1 --2- 172.21.15.29:0/3560665582 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb5a80a1ab0 0x7fb5a80a3f70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb5b40a2b90 tx=0x7fb5b406aa10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:41.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.430+0000 7fb5b1ffb640 1 -- 172.21.15.29:0/4226847835 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac005020 con 0x561488c2c6b0 2024-03-11T19:11:41.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.430+0000 7fb5b1ffb640 1 -- 172.21.15.29:0/4226847835 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb5ac005630 con 0x561488c2c6b0 2024-03-11T19:11:41.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.430+0000 7fb5b1ffb640 1 -- 172.21.15.29:0/4226847835 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac0126b0 con 0x561488c2c6b0 2024-03-11T19:11:41.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.430+0000 7fb5c458e740 1 -- 172.21.15.29:0/4226847835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c2c6b0 msgr2=0x561488c2cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:41.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.430+0000 7fb5c458e740 1 --2- 172.21.15.29:0/4226847835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c2c6b0 0x561488c2cb30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac009c00 tx=0x7fb5ac0117b0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.431+0000 7fb5c458e740 1 -- 172.21.15.29:0/4226847835 shutdown_connections 2024-03-11T19:11:41.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.431+0000 7fb5c458e740 1 --2- 172.21.15.29:0/4226847835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2d070 0x561488c33900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.431+0000 7fb5c458e740 1 --2- 172.21.15.29:0/4226847835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c2c6b0 0x561488c2cb30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.431+0000 7fb5c458e740 1 --2- 172.21.15.29:0/4226847835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2b4e0 0x561488c2b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.431+0000 7fb5c458e740 1 -- 172.21.15.29:0/4226847835 >> 172.21.15.29:0/4226847835 conn(0x561488c266c0 msgr2=0x561488c26ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.431+0000 7fb5c458e740 1 -- 172.21.15.29:0/4226847835 shutdown_connections 2024-03-11T19:11:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.431+0000 7fb5c458e740 1 -- 172.21.15.29:0/4226847835 wait complete. 2024-03-11T19:11:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.431+0000 7fb5c458e740 1 -- start start 2024-03-11T19:11:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.431+0000 7fb5c458e740 1 Processor -- start 2024-03-11T19:11:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.432+0000 7fb5c458e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2b4e0 0x561488c356f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.432+0000 7fb5c458e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2c6b0 0x561488c35c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.432+0000 7fb5c458e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c2d070 0x561488c3cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.432+0000 7fb5c458e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x561488b38f40 con 0x561488c2c6b0 2024-03-11T19:11:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.432+0000 7fb5c458e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614888674b0 con 0x561488c2d070 2024-03-11T19:11:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.432+0000 7fb5c458e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561488b19670 con 0x561488c2b4e0 2024-03-11T19:11:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.432+0000 7fb5c458e740 10 client.0 _ll_register_callbacks cb 0x561488b86f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:41.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.432+0000 7fb5baffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2c6b0 0x561488c35c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.432+0000 7fb5b3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2b4e0 0x561488c356f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.432+0000 7fb5baffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2c6b0 0x561488c35c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46292/0 (socket says 172.21.15.29:46292) 2024-03-11T19:11:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.432+0000 7fb5baffd640 1 -- 172.21.15.29:0/311102721 learned_addr learned my addr 172.21.15.29:0/311102721 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.432+0000 7fb5b3fff640 1 --2- 172.21.15.29:0/311102721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c2d070 0x561488c3cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.432+0000 7fb5baffd640 1 -- 172.21.15.29:0/311102721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2b4e0 msgr2=0x561488c356f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.432+0000 7fb5baffd640 1 --2- 172.21.15.29:0/311102721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2b4e0 0x561488c356f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.432+0000 7fb5baffd640 1 -- 172.21.15.29:0/311102721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c2d070 msgr2=0x561488c3cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.432+0000 7fb5baffd640 1 --2- 172.21.15.29:0/311102721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c2d070 0x561488c3cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.432+0000 7fb5baffd640 1 -- 172.21.15.29:0/311102721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5ac009860 con 0x561488c2c6b0 2024-03-11T19:11:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.433+0000 7fb5baffd640 1 --2- 172.21.15.29:0/311102721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2c6b0 0x561488c35c30 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac000c00 tx=0x7fb5ac009db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b03977b05c2163 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.433+0000 7fb59affd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.433+0000 7fb59affd640 1 -- 172.21.15.29:0/311102721 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac005210 con 0x561488c2c6b0 2024-03-11T19:11:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.433+0000 7fb59affd640 1 -- 172.21.15.29:0/311102721 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb5ac005370 con 0x561488c2c6b0 2024-03-11T19:11:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.433+0000 7fb5c458e740 1 -- 172.21.15.29:0/311102721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561488b6df40 con 0x561488c2c6b0 2024-03-11T19:11:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.433+0000 7fb59affd640 1 -- 172.21.15.29:0/311102721 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5ac005530 con 0x561488c2c6b0 2024-03-11T19:11:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.433+0000 7fb5c458e740 10 client.15918 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.433+0000 7fb5c458e740 1 -- 172.21.15.29:0/311102721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x561488b92320 con 0x561488c2c6b0 2024-03-11T19:11:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.434+0000 7fb59affd640 1 -- 172.21.15.29:0/311102721 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x561488b6df40 con 0x561488c2c6b0 2024-03-11T19:11:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.434+0000 7fb59affd640 1 -- 172.21.15.29:0/311102721 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb5ac0124e0 con 0x561488c2c6b0 2024-03-11T19:11:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.434+0000 7fb5c458e740 10 client.15918 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.434+0000 7fb5c458e740 10 client.15918 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:11:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.434+0000 7fb5c458e740 20 client.15918 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.434+0000 7fb5c458e740 1 --2- 172.21.15.29:0/311102721 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x561488c418b0 0x561488c43cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.434+0000 7fb5c458e740 4 client.15918 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:11:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.434+0000 7fb5c458e740 1 -- 172.21.15.29:0/311102721 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561488b8a310 con 0x561488c418b0 2024-03-11T19:11:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.434+0000 7fb5b3fff640 1 --2- 172.21.15.29:0/311102721 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x561488c418b0 0x561488c43cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.435+0000 7fb59affd640 10 client.15918 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:41.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.435+0000 7fb5b3fff640 1 --2- 172.21.15.29:0/311102721 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x561488c418b0 0x561488c43cd0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4522fe62e4750a3 server_cookie=a4b90fbc508e767f in_seq=0 out_seq=0 2024-03-11T19:11:41.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.435+0000 7fb59affd640 1 -- 172.21.15.29:0/311102721 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561488b8a310 con 0x561488c418b0 2024-03-11T19:11:41.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.435+0000 7fb59affd640 10 client.15918 handle_command_reply: tid=0 2024-03-11T19:11:41.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.436+0000 7fb5c458e740 1 client.15918 shutdown 2024-03-11T19:11:41.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.436+0000 7fb5c458e740 1 -- 172.21.15.29:0/311102721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2c6b0 msgr2=0x561488c35c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:41.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.436+0000 7fb5c458e740 1 --2- 172.21.15.29:0/311102721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2c6b0 0x561488c35c30 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac000c00 tx=0x7fb5ac009db0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.436+0000 7fb5c458e740 1 -- 172.21.15.29:0/311102721 shutdown_connections 2024-03-11T19:11:41.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.436+0000 7fb5c458e740 1 --2- 172.21.15.29:0/311102721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c2d070 0x561488c3cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.436+0000 7fb5c458e740 1 --2- 172.21.15.29:0/311102721 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x561488c418b0 0x561488c43cd0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.436+0000 7fb5c458e740 1 --2- 172.21.15.29:0/311102721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2c6b0 0x561488c35c30 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.436+0000 7fb5c458e740 1 --2- 172.21.15.29:0/311102721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2b4e0 0x561488c356f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.436+0000 7fb5c458e740 1 -- 172.21.15.29:0/311102721 >> 172.21.15.29:0/311102721 conn(0x561488c266c0 msgr2=0x561488c25a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:41.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.436+0000 7fb59affd640 0 client.15918 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:41.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.436+0000 7fb5c458e740 1 -- 172.21.15.29:0/311102721 shutdown_connections 2024-03-11T19:11:41.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.436+0000 7fb5c458e740 1 -- 172.21.15.29:0/311102721 wait complete. 2024-03-11T19:11:41.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.436+0000 7fb5c458e740 20 client.15918 trim_cache size 0 max 16384 2024-03-11T19:11:41.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.459+0000 7fb5c458e740 1 Processor -- start 2024-03-11T19:11:41.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.459+0000 7fb5c458e740 1 -- start start 2024-03-11T19:11:41.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.459+0000 7fb5c458e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c2b4e0 0x561488c48630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.459+0000 7fb5c458e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2c6b0 0x561488c3cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.459+0000 7fb5c458e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2d070 0x561488c356f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.459+0000 7fb5c458e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x561488c34840 con 0x561488c2d070 2024-03-11T19:11:41.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.459+0000 7fb5c458e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x561488c2b380 con 0x561488c2b4e0 2024-03-11T19:11:41.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.459+0000 7fb5c458e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561488b8a310 con 0x561488c2c6b0 2024-03-11T19:11:41.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.459+0000 7fb5b3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2d070 0x561488c356f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.459+0000 7fb5b3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2d070 0x561488c356f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46298/0 (socket says 172.21.15.29:46298) 2024-03-11T19:11:41.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.459+0000 7fb5b3fff640 1 -- 172.21.15.29:0/2535970548 learned_addr learned my addr 172.21.15.29:0/2535970548 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:41.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.459+0000 7fb5b3fff640 1 --2- 172.21.15.29:0/2535970548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c2b4e0 0x561488c48630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.459+0000 7fb5baffd640 1 --2- 172.21.15.29:0/2535970548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2c6b0 0x561488c3cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.459+0000 7fb5b3fff640 1 -- 172.21.15.29:0/2535970548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2c6b0 msgr2=0x561488c3cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:41.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.459+0000 7fb5b3fff640 1 --2- 172.21.15.29:0/2535970548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2c6b0 0x561488c3cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.459+0000 7fb5b3fff640 1 -- 172.21.15.29:0/2535970548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c2b4e0 msgr2=0x561488c48630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:41.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.459+0000 7fb5b3fff640 1 --2- 172.21.15.29:0/2535970548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c2b4e0 0x561488c48630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.459+0000 7fb5b3fff640 1 -- 172.21.15.29:0/2535970548 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561488b2ec00 con 0x561488c2d070 2024-03-11T19:11:41.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.460+0000 7fb5b3fff640 1 --2- 172.21.15.29:0/2535970548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2d070 0x561488c356f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009230 tx=0x7fb5a402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a82ae5b69019292 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:41.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.460+0000 7fb5b17fa640 1 -- 172.21.15.29:0/2535970548 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4048070 con 0x561488c2d070 2024-03-11T19:11:41.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.460+0000 7fb5b17fa640 1 -- 172.21.15.29:0/2535970548 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb5a402cd40 con 0x561488c2d070 2024-03-11T19:11:41.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.460+0000 7fb5b17fa640 1 -- 172.21.15.29:0/2535970548 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a404ca90 con 0x561488c2d070 2024-03-11T19:11:41.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.461+0000 7fb5c458e740 1 -- 172.21.15.29:0/2535970548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2d070 msgr2=0x561488c356f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:41.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.461+0000 7fb5c458e740 1 --2- 172.21.15.29:0/2535970548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2d070 0x561488c356f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009230 tx=0x7fb5a402bc30 comp rx=0 tx=0).stop 2024-03-11T19:11:41.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.461+0000 7fb5c458e740 1 -- 172.21.15.29:0/2535970548 shutdown_connections 2024-03-11T19:11:41.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.461+0000 7fb5c458e740 1 --2- 172.21.15.29:0/2535970548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2d070 0x561488c356f0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.461+0000 7fb5c458e740 1 --2- 172.21.15.29:0/2535970548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2c6b0 0x561488c3cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.461+0000 7fb5c458e740 1 --2- 172.21.15.29:0/2535970548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c2b4e0 0x561488c48630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.461+0000 7fb5c458e740 1 -- 172.21.15.29:0/2535970548 >> 172.21.15.29:0/2535970548 conn(0x561488c266c0 msgr2=0x561488c34d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:41.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.461+0000 7fb5c458e740 1 -- 172.21.15.29:0/2535970548 shutdown_connections 2024-03-11T19:11:41.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.461+0000 7fb5c458e740 1 -- 172.21.15.29:0/2535970548 wait complete. 2024-03-11T19:11:41.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.462+0000 7fb5c458e740 1 -- start start 2024-03-11T19:11:41.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.462+0000 7fb5c458e740 1 Processor -- start 2024-03-11T19:11:41.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.462+0000 7fb5c458e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2b4e0 0x561488c3c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.462+0000 7fb5c458e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2c6b0 0x561488c39370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.462+0000 7fb5c458e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c398b0 0x561488c97da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.462+0000 7fb5c458e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x561488c3d020 con 0x561488c2c6b0 2024-03-11T19:11:41.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.462+0000 7fb5c458e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x561488c39d30 con 0x561488c398b0 2024-03-11T19:11:41.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.462+0000 7fb5c458e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561488c39ec0 con 0x561488c2b4e0 2024-03-11T19:11:41.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.462+0000 7fb5c458e740 10 client.0 _ll_register_callbacks cb 0x7fb5ac0095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:41.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.462+0000 7fb5baffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2c6b0 0x561488c39370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.462+0000 7fb5baffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2c6b0 0x561488c39370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46300/0 (socket says 172.21.15.29:46300) 2024-03-11T19:11:41.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.463+0000 7fb5baffd640 1 -- 172.21.15.29:0/998629386 learned_addr learned my addr 172.21.15.29:0/998629386 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:41.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.463+0000 7fb5b3fff640 1 --2- 172.21.15.29:0/998629386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c398b0 0x561488c97da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.463+0000 7fb5b3fff640 1 --2- 172.21.15.29:0/998629386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2b4e0 0x561488c3c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.463+0000 7fb5baffd640 1 -- 172.21.15.29:0/998629386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2b4e0 msgr2=0x561488c3c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:41.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.463+0000 7fb5baffd640 1 --2- 172.21.15.29:0/998629386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2b4e0 0x561488c3c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.463+0000 7fb5baffd640 1 -- 172.21.15.29:0/998629386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c398b0 msgr2=0x561488c97da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:41.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.463+0000 7fb5baffd640 1 --2- 172.21.15.29:0/998629386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c398b0 0x561488c97da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.463+0000 7fb5baffd640 1 -- 172.21.15.29:0/998629386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a4030070 con 0x561488c2c6b0 2024-03-11T19:11:41.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.463+0000 7fb5baffd640 1 --2- 172.21.15.29:0/998629386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2c6b0 0x561488c39370 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac011d30 tx=0x7fb5ac012e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d2e165a64c41b6d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:41.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.463+0000 7fb59affd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:41.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.463+0000 7fb59affd640 1 -- 172.21.15.29:0/998629386 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac012450 con 0x561488c2c6b0 2024-03-11T19:11:41.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.463+0000 7fb59affd640 1 -- 172.21.15.29:0/998629386 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb5ac000e40 con 0x561488c2c6b0 2024-03-11T19:11:41.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.463+0000 7fb5c458e740 1 -- 172.21.15.29:0/998629386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561488bb9900 con 0x561488c2c6b0 2024-03-11T19:11:41.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.464+0000 7fb59affd640 1 -- 172.21.15.29:0/998629386 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5ac0125b0 con 0x561488c2c6b0 2024-03-11T19:11:41.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.464+0000 7fb5c458e740 10 client.15924 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:41.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.464+0000 7fb5c458e740 1 -- 172.21.15.29:0/998629386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x561488b54910 con 0x561488c2c6b0 2024-03-11T19:11:41.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.464+0000 7fb59affd640 1 -- 172.21.15.29:0/998629386 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x561488bb9900 con 0x561488c2c6b0 2024-03-11T19:11:41.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.464+0000 7fb59affd640 1 -- 172.21.15.29:0/998629386 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb5ac024990 con 0x561488c2c6b0 2024-03-11T19:11:41.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.464+0000 7fb5c458e740 10 client.15924 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:41.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.464+0000 7fb5c458e740 10 client.15924 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:11:41.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.464+0000 7fb5c458e740 20 client.15924 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:41.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.464+0000 7fb5c458e740 1 --2- 172.21.15.29:0/998629386 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x561488c99840 0x561488c9bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.464+0000 7fb5c458e740 4 client.15924 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:11:41.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.464+0000 7fb5c458e740 1 -- 172.21.15.29:0/998629386 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x561488893860 con 0x561488c99840 2024-03-11T19:11:41.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.465+0000 7fb5b3fff640 1 --2- 172.21.15.29:0/998629386 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x561488c99840 0x561488c9bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.466+0000 7fb5b3fff640 1 --2- 172.21.15.29:0/998629386 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x561488c99840 0x561488c9bc60 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d72321352a296722 server_cookie=8bbe8ad9e6240be7 in_seq=0 out_seq=0 2024-03-11T19:11:41.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.466+0000 7fb59affd640 10 client.15924 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:41.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.467+0000 7fb59affd640 1 -- 172.21.15.29:0/998629386 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x561488893860 con 0x561488c99840 2024-03-11T19:11:41.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.467+0000 7fb59affd640 10 client.15924 handle_command_reply: tid=0 2024-03-11T19:11:41.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.467+0000 7fb5c458e740 1 client.15924 shutdown 2024-03-11T19:11:41.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.467+0000 7fb5c458e740 1 -- 172.21.15.29:0/998629386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2c6b0 msgr2=0x561488c39370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.467+0000 7fb5c458e740 1 --2- 172.21.15.29:0/998629386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2c6b0 0x561488c39370 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac011d30 tx=0x7fb5ac012e20 comp rx=0 tx=0).stop 2024-03-11T19:11:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.467+0000 7fb5c458e740 1 -- 172.21.15.29:0/998629386 shutdown_connections 2024-03-11T19:11:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.467+0000 7fb5c458e740 1 --2- 172.21.15.29:0/998629386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561488c398b0 0x561488c97da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.467+0000 7fb5c458e740 1 --2- 172.21.15.29:0/998629386 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x561488c99840 0x561488c9bc60 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.467+0000 7fb5c458e740 1 --2- 172.21.15.29:0/998629386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561488c2c6b0 0x561488c39370 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.467+0000 7fb5c458e740 1 --2- 172.21.15.29:0/998629386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561488c2b4e0 0x561488c3c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.467+0000 7fb5c458e740 1 -- 172.21.15.29:0/998629386 >> 172.21.15.29:0/998629386 conn(0x561488c266c0 msgr2=0x561488c29920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.467+0000 7fb59affd640 0 client.15924 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.467+0000 7fb5c458e740 1 -- 172.21.15.29:0/998629386 shutdown_connections 2024-03-11T19:11:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.467+0000 7fb5c458e740 1 -- 172.21.15.29:0/998629386 wait complete. 2024-03-11T19:11:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.467+0000 7fb5c458e740 20 client.15924 trim_cache size 0 max 16384 2024-03-11T19:11:41.474 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:11:41.474 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:11:41.474 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "eaa91b78-4526-42dd-99ae-e3ceae511852", 2024-03-11T19:11:41.474 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:11:41.474 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:11:41.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.468+0000 7fb5c18df640 1 -- 172.21.15.29:0/3560665582 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb5a80a1ab0 msgr2=0x7fb5a80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:41.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.468+0000 7fb5c18df640 1 --2- 172.21.15.29:0/3560665582 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb5a80a1ab0 0x7fb5a80a3f70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb5b40a2b90 tx=0x7fb5b406aa10 comp rx=0 tx=0).stop 2024-03-11T19:11:41.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.468+0000 7fb5c18df640 1 -- 172.21.15.29:0/3560665582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5bc1013b0 msgr2=0x7fb5bc1a0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:41.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.468+0000 7fb5c18df640 1 --2- 172.21.15.29:0/3560665582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5bc1013b0 0x7fb5bc1a0930 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400eb10 tx=0x7fb5b400c2d0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.468+0000 7fb5c18df640 1 -- 172.21.15.29:0/3560665582 shutdown_connections 2024-03-11T19:11:41.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.468+0000 7fb5c18df640 1 --2- 172.21.15.29:0/3560665582 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb5a80a1ab0 0x7fb5a80a3f70 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.468+0000 7fb5c18df640 1 --2- 172.21.15.29:0/3560665582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5bc1013b0 0x7fb5bc1a0930 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.468+0000 7fb5c18df640 1 --2- 172.21.15.29:0/3560665582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5bc1009f0 0x7fb5bc1998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.468+0000 7fb5c18df640 1 --2- 172.21.15.29:0/3560665582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5bc0ff800 0x7fb5bc199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.468+0000 7fb5c18df640 1 -- 172.21.15.29:0/3560665582 >> 172.21.15.29:0/3560665582 conn(0x7fb5bc0fad70 msgr2=0x7fb5bc1093d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:41.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.468+0000 7fb5c18df640 1 -- 172.21.15.29:0/3560665582 shutdown_connections 2024-03-11T19:11:41.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.468+0000 7fb5c18df640 1 -- 172.21.15.29:0/3560665582 wait complete. 2024-03-11T19:11:41.488 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:11:41.488 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:11:41.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.640+0000 7f4a3817a640 1 -- 172.21.15.29:0/927835429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a30102450 msgr2=0x7f4a301028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:41.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.640+0000 7f4a3817a640 1 --2- 172.21.15.29:0/927835429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a30102450 0x7f4a301028d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4a24009900 tx=0x7f4a2402bc30 comp rx=0 tx=0).stop 2024-03-11T19:11:41.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.640+0000 7f4a3817a640 1 -- 172.21.15.29:0/927835429 shutdown_connections 2024-03-11T19:11:41.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.640+0000 7f4a3817a640 1 --2- 172.21.15.29:0/927835429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a30102e10 0x7f4a301096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.640+0000 7f4a3817a640 1 --2- 172.21.15.29:0/927835429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a30102450 0x7f4a301028d0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.640+0000 7f4a3817a640 1 --2- 172.21.15.29:0/927835429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a300ff480 0x7f4a300ff8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.640+0000 7f4a3817a640 1 -- 172.21.15.29:0/927835429 >> 172.21.15.29:0/927835429 conn(0x7f4a300fadb0 msgr2=0x7f4a300fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:41.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.640+0000 7f4a3817a640 1 -- 172.21.15.29:0/927835429 shutdown_connections 2024-03-11T19:11:41.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.640+0000 7f4a3817a640 1 -- 172.21.15.29:0/927835429 wait complete. 2024-03-11T19:11:41.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.641+0000 7f4a3817a640 1 Processor -- start 2024-03-11T19:11:41.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.641+0000 7f4a3817a640 1 -- start start 2024-03-11T19:11:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.641+0000 7f4a3817a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a300ff480 0x7f4a301971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.641+0000 7f4a3817a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a30102450 0x7f4a30197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.641+0000 7f4a3817a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a30102e10 0x7f4a3019e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.641+0000 7f4a3817a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3019ecb0 con 0x7f4a30102e10 2024-03-11T19:11:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.641+0000 7f4a3817a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3019ee20 con 0x7f4a300ff480 2024-03-11T19:11:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.641+0000 7f4a3817a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3019efe0 con 0x7f4a30102450 2024-03-11T19:11:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.641+0000 7f4a36717640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a30102e10 0x7f4a3019e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.641+0000 7f4a35f16640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a300ff480 0x7f4a301971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.641+0000 7f4a35715640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a30102450 0x7f4a30197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.641+0000 7f4a35715640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a30102450 0x7f4a30197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:51758/0 (socket says 172.21.15.29:51758) 2024-03-11T19:11:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.641+0000 7f4a35715640 1 -- 172.21.15.29:0/1766785383 learned_addr learned my addr 172.21.15.29:0/1766785383 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.642+0000 7f4a35715640 1 -- 172.21.15.29:0/1766785383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a300ff480 msgr2=0x7f4a301971c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.642+0000 7f4a35715640 1 --2- 172.21.15.29:0/1766785383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a300ff480 0x7f4a301971c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.642+0000 7f4a35715640 1 -- 172.21.15.29:0/1766785383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a30102e10 msgr2=0x7f4a3019e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:41.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.642+0000 7f4a35715640 1 --2- 172.21.15.29:0/1766785383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a30102e10 0x7f4a3019e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.642+0000 7f4a35715640 1 -- 172.21.15.29:0/1766785383 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a24009560 con 0x7f4a30102450 2024-03-11T19:11:41.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.642+0000 7f4a35f16640 1 --2- 172.21.15.29:0/1766785383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a300ff480 0x7f4a301971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:41.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.642+0000 7f4a35715640 1 --2- 172.21.15.29:0/1766785383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a30102450 0x7f4a30197700 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4a24036040 tx=0x7f4a24004370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:41.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.642+0000 7f4a2effd640 1 -- 172.21.15.29:0/1766785383 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a24004aa0 con 0x7f4a30102450 2024-03-11T19:11:41.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.642+0000 7f4a3817a640 1 -- 172.21.15.29:0/1766785383 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a3019f260 con 0x7f4a30102450 2024-03-11T19:11:41.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.642+0000 7f4a2effd640 1 -- 172.21.15.29:0/1766785383 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4a24003cb0 con 0x7f4a30102450 2024-03-11T19:11:41.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.642+0000 7f4a2effd640 1 -- 172.21.15.29:0/1766785383 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2403e450 con 0x7f4a30102450 2024-03-11T19:11:41.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.642+0000 7f4a3817a640 1 -- 172.21.15.29:0/1766785383 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3019f840 con 0x7f4a30102450 2024-03-11T19:11:41.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.645+0000 7f4a2effd640 1 -- 172.21.15.29:0/1766785383 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4a24004e50 con 0x7f4a30102450 2024-03-11T19:11:41.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.645+0000 7f4a2effd640 1 --2- 172.21.15.29:0/1766785383 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4a180a1a30 0x7f4a180a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.646+0000 7f4a2effd640 1 -- 172.21.15.29:0/1766785383 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4a240ef6a0 con 0x7f4a30102450 2024-03-11T19:11:41.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.646+0000 7f4a35f16640 1 --2- 172.21.15.29:0/1766785383 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4a180a1a30 0x7f4a180a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.646+0000 7f4a35f16640 1 --2- 172.21.15.29:0/1766785383 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4a180a1a30 0x7f4a180a3ef0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4a30069590 tx=0x7f4a2000d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:41.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.646+0000 7f4a3ae2e740 1 Processor -- start 2024-03-11T19:11:41.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.647+0000 7f4a3ae2e740 1 -- start start 2024-03-11T19:11:41.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.647+0000 7f4a3ae2e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f8021f2d0 0x558f8021f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.647+0000 7f4a3ae2e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 0x558f802247c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.647+0000 7f4a3ae2e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f80224e30 0x558f802272c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.647+0000 7f4a35f16640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f8021f2d0 0x558f8021f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.647+0000 7f4a35f16640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f8021f2d0 0x558f8021f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:48622/0 (socket says 172.21.15.29:48622) 2024-03-11T19:11:41.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.647+0000 7f4a35715640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 0x558f802247c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.647+0000 7f4a35f16640 1 -- 172.21.15.29:0/1879290511 learned_addr learned my addr 172.21.15.29:0/1879290511 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:41.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.647+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/1879290511 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f8018f280 con 0x558f8021fcc0 2024-03-11T19:11:41.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.647+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/1879290511 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f8018c7b0 con 0x558f8021f2d0 2024-03-11T19:11:41.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.647+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/1879290511 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f80187f90 con 0x558f80224e30 2024-03-11T19:11:41.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.648+0000 7f4a36717640 1 --2- 172.21.15.29:0/1879290511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f80224e30 0x558f802272c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.648+0000 7f4a35715640 1 -- 172.21.15.29:0/1879290511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f80224e30 msgr2=0x558f802272c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:41.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.648+0000 7f4a35715640 1 --2- 172.21.15.29:0/1879290511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f80224e30 0x558f802272c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.648+0000 7f4a35715640 1 -- 172.21.15.29:0/1879290511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f8021f2d0 msgr2=0x558f8021f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:41.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.648+0000 7f4a35715640 1 --2- 172.21.15.29:0/1879290511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f8021f2d0 0x558f8021f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.648+0000 7f4a35715640 1 -- 172.21.15.29:0/1879290511 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f80146570 con 0x558f8021fcc0 2024-03-11T19:11:41.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.648+0000 7f4a35f16640 1 --2- 172.21.15.29:0/1879290511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f8021f2d0 0x558f8021f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:11:41.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.648+0000 7f4a35715640 1 --2- 172.21.15.29:0/1879290511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 0x558f802247c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4a24004c20 tx=0x7f4a2402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7364f65c07ad57 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.648+0000 7f4a17fff640 1 -- 172.21.15.29:0/1879290511 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a24003e20 con 0x558f8021fcc0 2024-03-11T19:11:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.648+0000 7f4a17fff640 1 -- 172.21.15.29:0/1879290511 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4a240bc610 con 0x558f8021fcc0 2024-03-11T19:11:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.648+0000 7f4a17fff640 1 -- 172.21.15.29:0/1879290511 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a240dc210 con 0x558f8021fcc0 2024-03-11T19:11:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.649+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/1879290511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 msgr2=0x558f802247c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.649+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/1879290511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 0x558f802247c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4a24004c20 tx=0x7f4a2402bc30 comp rx=0 tx=0).stop 2024-03-11T19:11:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.649+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/1879290511 shutdown_connections 2024-03-11T19:11:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.649+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/1879290511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f80224e30 0x558f802272c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.649+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/1879290511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 0x558f802247c0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.649+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/1879290511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f8021f2d0 0x558f8021f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.649+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/1879290511 >> 172.21.15.29:0/1879290511 conn(0x558f8021a7f0 msgr2=0x558f8021abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.650+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/1879290511 shutdown_connections 2024-03-11T19:11:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.650+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/1879290511 wait complete. 2024-03-11T19:11:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.650+0000 7f4a3ae2e740 1 -- start start 2024-03-11T19:11:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.650+0000 7f4a3ae2e740 1 Processor -- start 2024-03-11T19:11:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.650+0000 7f4a3ae2e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f8021f2d0 0x558f802290a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.650+0000 7f4a3ae2e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 0x558f802295e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.650+0000 7f4a3ae2e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f80224e30 0x558f80230620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.650+0000 7f4a3ae2e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f7fe5b430 con 0x558f8021fcc0 2024-03-11T19:11:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.650+0000 7f4a3ae2e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f7fe17e00 con 0x558f80224e30 2024-03-11T19:11:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.650+0000 7f4a3ae2e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f8010cfd0 con 0x558f8021f2d0 2024-03-11T19:11:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.651+0000 7f4a3ae2e740 10 client.0 _ll_register_callbacks cb 0x558f80162fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.651+0000 7f4a35715640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 0x558f802295e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.651+0000 7f4a35715640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 0x558f802295e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46328/0 (socket says 172.21.15.29:46328) 2024-03-11T19:11:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.651+0000 7f4a35715640 1 -- 172.21.15.29:0/2104392674 learned_addr learned my addr 172.21.15.29:0/2104392674 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.651+0000 7f4a35f16640 1 --2- 172.21.15.29:0/2104392674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f8021f2d0 0x558f802290a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.651+0000 7f4a36717640 1 --2- 172.21.15.29:0/2104392674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f80224e30 0x558f80230620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.651+0000 7f4a35715640 1 -- 172.21.15.29:0/2104392674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f8021f2d0 msgr2=0x558f802290a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.651+0000 7f4a35715640 1 --2- 172.21.15.29:0/2104392674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f8021f2d0 0x558f802290a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.651+0000 7f4a35715640 1 -- 172.21.15.29:0/2104392674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f80224e30 msgr2=0x558f80230620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.651+0000 7f4a35715640 1 --2- 172.21.15.29:0/2104392674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f80224e30 0x558f80230620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.651+0000 7f4a35715640 1 -- 172.21.15.29:0/2104392674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3019f840 con 0x558f8021fcc0 2024-03-11T19:11:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.651+0000 7f4a36717640 1 --2- 172.21.15.29:0/2104392674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f80224e30 0x558f80230620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:11:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.651+0000 7f4a35715640 1 --2- 172.21.15.29:0/2104392674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 0x558f802295e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f4a240bab60 tx=0x7f4a240056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a887d276380b6f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.651+0000 7f4a157fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:11:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.651+0000 7f4a157fa640 1 -- 172.21.15.29:0/2104392674 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a24003e20 con 0x558f8021fcc0 2024-03-11T19:11:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.651+0000 7f4a157fa640 1 -- 172.21.15.29:0/2104392674 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4a240c0760 con 0x558f8021fcc0 2024-03-11T19:11:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.652+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/2104392674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558f7fe877e0 con 0x558f8021fcc0 2024-03-11T19:11:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.652+0000 7f4a157fa640 1 -- 172.21.15.29:0/2104392674 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a240a70b0 con 0x558f8021fcc0 2024-03-11T19:11:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.652+0000 7f4a3ae2e740 10 client.15936 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.652+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/2104392674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x558f80186610 con 0x558f8021fcc0 2024-03-11T19:11:41.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.652+0000 7f4a157fa640 1 -- 172.21.15.29:0/2104392674 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4a2409f080 con 0x558f8021fcc0 2024-03-11T19:11:41.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.652+0000 7f4a157fa640 1 -- 172.21.15.29:0/2104392674 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4a2403c070 con 0x558f8021fcc0 2024-03-11T19:11:41.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.652+0000 7f4a3ae2e740 10 client.15936 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:41.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.652+0000 7f4a3ae2e740 10 client.15936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:11:41.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.652+0000 7f4a3ae2e740 20 client.15936 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:41.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.652+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/2104392674 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558f80234f40 0x558f80237330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.652+0000 7f4a3ae2e740 4 client.15936 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:11:41.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.652+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/2104392674 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558f7fec4720 con 0x558f80234f40 2024-03-11T19:11:41.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.653+0000 7f4a35f16640 1 --2- 172.21.15.29:0/2104392674 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558f80234f40 0x558f80237330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.653+0000 7f4a35f16640 1 --2- 172.21.15.29:0/2104392674 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558f80234f40 0x558f80237330 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8de3569259337d1 server_cookie=7b287ca95182995 in_seq=0 out_seq=0 2024-03-11T19:11:41.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.653+0000 7f4a157fa640 10 client.15936 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:41.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.654+0000 7f4a157fa640 1 -- 172.21.15.29:0/2104392674 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558f7fec4720 con 0x558f80234f40 2024-03-11T19:11:41.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.654+0000 7f4a157fa640 10 client.15936 handle_command_reply: tid=0 2024-03-11T19:11:41.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.654+0000 7f4a3ae2e740 1 client.15936 shutdown 2024-03-11T19:11:41.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.654+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/2104392674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 msgr2=0x558f802295e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:41.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.654+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/2104392674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 0x558f802295e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f4a240bab60 tx=0x7f4a240056d0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.654+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/2104392674 shutdown_connections 2024-03-11T19:11:41.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.654+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/2104392674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f80224e30 0x558f80230620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.654+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/2104392674 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558f80234f40 0x558f80237330 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.654+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/2104392674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 0x558f802295e0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.654+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/2104392674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f8021f2d0 0x558f802290a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.654+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/2104392674 >> 172.21.15.29:0/2104392674 conn(0x558f8021a7f0 msgr2=0x558f80219a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:41.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.654+0000 7f4a157fa640 0 client.15936 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:41.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.654+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/2104392674 shutdown_connections 2024-03-11T19:11:41.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.654+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/2104392674 wait complete. 2024-03-11T19:11:41.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.654+0000 7f4a3ae2e740 20 client.15936 trim_cache size 0 max 16384 2024-03-11T19:11:41.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.679+0000 7f4a3ae2e740 1 Processor -- start 2024-03-11T19:11:41.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.679+0000 7f4a3ae2e740 1 -- start start 2024-03-11T19:11:41.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.679+0000 7f4a3ae2e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f8021f2d0 0x558f802290a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.679+0000 7f4a3ae2e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 0x558f802295e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.679+0000 7f4a3ae2e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f80224e30 0x558f802305e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.679+0000 7f4a3ae2e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f7fec4750 con 0x558f8021fcc0 2024-03-11T19:11:41.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.679+0000 7f4a3ae2e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f802277b0 con 0x558f8021f2d0 2024-03-11T19:11:41.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.679+0000 7f4a3ae2e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f802160d0 con 0x558f80224e30 2024-03-11T19:11:41.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.679+0000 7f4a35715640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 0x558f802295e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.679+0000 7f4a35715640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 0x558f802295e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46332/0 (socket says 172.21.15.29:46332) 2024-03-11T19:11:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.679+0000 7f4a35715640 1 -- 172.21.15.29:0/3777608242 learned_addr learned my addr 172.21.15.29:0/3777608242 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.679+0000 7f4a35f16640 1 --2- 172.21.15.29:0/3777608242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f8021f2d0 0x558f802290a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.679+0000 7f4a36717640 1 --2- 172.21.15.29:0/3777608242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f80224e30 0x558f802305e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.679+0000 7f4a35715640 1 -- 172.21.15.29:0/3777608242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f80224e30 msgr2=0x558f802305e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.679+0000 7f4a35715640 1 --2- 172.21.15.29:0/3777608242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f80224e30 0x558f802305e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.679+0000 7f4a35715640 1 -- 172.21.15.29:0/3777608242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f8021f2d0 msgr2=0x558f802290a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.679+0000 7f4a35715640 1 --2- 172.21.15.29:0/3777608242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f8021f2d0 0x558f802290a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.679+0000 7f4a35715640 1 -- 172.21.15.29:0/3777608242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f80122ac0 con 0x558f8021fcc0 2024-03-11T19:11:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.680+0000 7f4a36717640 1 --2- 172.21.15.29:0/3777608242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f80224e30 0x558f802305e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:11:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.680+0000 7f4a35715640 1 --2- 172.21.15.29:0/3777608242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 0x558f802295e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4a240b8fe0 tx=0x7f4a24000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e81f89413c7a270 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.680+0000 7f4a177fe640 1 -- 172.21.15.29:0/3777608242 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a240bbdc0 con 0x558f8021fcc0 2024-03-11T19:11:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.680+0000 7f4a177fe640 1 -- 172.21.15.29:0/3777608242 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x558f7fe877e0 con 0x558f8021fcc0 2024-03-11T19:11:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.680+0000 7f4a177fe640 1 -- 172.21.15.29:0/3777608242 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a240c4920 con 0x558f8021fcc0 2024-03-11T19:11:41.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.681+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/3777608242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 msgr2=0x558f802295e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:41.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.681+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/3777608242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 0x558f802295e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4a240b8fe0 tx=0x7f4a24000bc0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.681+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/3777608242 shutdown_connections 2024-03-11T19:11:41.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.681+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/3777608242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f80224e30 0x558f802305e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.681+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/3777608242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021fcc0 0x558f802295e0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.681+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/3777608242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f8021f2d0 0x558f802290a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.681+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/3777608242 >> 172.21.15.29:0/3777608242 conn(0x558f8021a7f0 msgr2=0x558f8023bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:41.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.681+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/3777608242 shutdown_connections 2024-03-11T19:11:41.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.681+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/3777608242 wait complete. 2024-03-11T19:11:41.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.682+0000 7f4a3ae2e740 1 -- start start 2024-03-11T19:11:41.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.682+0000 7f4a3ae2e740 1 Processor -- start 2024-03-11T19:11:41.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.682+0000 7f4a3ae2e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021f2d0 0x558f80230050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.682+0000 7f4a3ae2e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f8021fcc0 0x558f80230590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.682+0000 7f4a3ae2e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f80224e30 0x558f8022e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.682+0000 7f4a3ae2e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f8022ed60 con 0x558f8021f2d0 2024-03-11T19:11:41.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.682+0000 7f4a3ae2e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f8022eed0 con 0x558f80224e30 2024-03-11T19:11:41.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.682+0000 7f4a3ae2e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f8022f090 con 0x558f8021fcc0 2024-03-11T19:11:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.682+0000 7f4a3ae2e740 10 client.0 _ll_register_callbacks cb 0x7f4a24005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:11:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.683+0000 7f4a35f16640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021f2d0 0x558f80230050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.683+0000 7f4a36717640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f80224e30 0x558f8022e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.683+0000 7f4a35f16640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021f2d0 0x558f80230050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46346/0 (socket says 172.21.15.29:46346) 2024-03-11T19:11:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.683+0000 7f4a35f16640 1 -- 172.21.15.29:0/3818613490 learned_addr learned my addr 172.21.15.29:0/3818613490 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:11:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.683+0000 7f4a35715640 1 --2- 172.21.15.29:0/3818613490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f8021fcc0 0x558f80230590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.683+0000 7f4a36717640 1 -- 172.21.15.29:0/3818613490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f8021fcc0 msgr2=0x558f80230590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.683+0000 7f4a36717640 1 --2- 172.21.15.29:0/3818613490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f8021fcc0 0x558f80230590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.683+0000 7f4a36717640 1 -- 172.21.15.29:0/3818613490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021f2d0 msgr2=0x558f80230050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:11:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.683+0000 7f4a36717640 1 --2- 172.21.15.29:0/3818613490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021f2d0 0x558f80230050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.683+0000 7f4a36717640 1 -- 172.21.15.29:0/3818613490 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f80186610 con 0x558f80224e30 2024-03-11T19:11:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.683+0000 7f4a35715640 1 --2- 172.21.15.29:0/3818613490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f8021fcc0 0x558f80230590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:11:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.684+0000 7f4a36717640 1 --2- 172.21.15.29:0/3818613490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f80224e30 0x558f8022e6b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x558f80220a10 tx=0x7f4a28009c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d51c95013aa2b90 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:11:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.684+0000 7f4a157fa640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:11:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.684+0000 7f4a157fa640 1 -- 172.21.15.29:0/3818613490 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a28004260 con 0x558f80224e30 2024-03-11T19:11:41.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.684+0000 7f4a157fa640 1 -- 172.21.15.29:0/3818613490 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4a280043c0 con 0x558f80224e30 2024-03-11T19:11:41.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.684+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/3818613490 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4a240bc610 con 0x558f80224e30 2024-03-11T19:11:41.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.685+0000 7f4a157fa640 1 -- 172.21.15.29:0/3818613490 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a2800e8e0 con 0x558f80224e30 2024-03-11T19:11:41.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.685+0000 7f4a3ae2e740 10 client.25475 fetch_fsmap learned FSMap version 27 2024-03-11T19:11:41.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.685+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/3818613490 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x558f801486a0 con 0x558f80224e30 2024-03-11T19:11:41.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.685+0000 7f4a157fa640 1 -- 172.21.15.29:0/3818613490 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4a280035f0 con 0x558f80224e30 2024-03-11T19:11:41.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.685+0000 7f4a157fa640 1 -- 172.21.15.29:0/3818613490 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4a28020480 con 0x558f80224e30 2024-03-11T19:11:41.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.685+0000 7f4a3ae2e740 10 client.25475 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:11:41.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.685+0000 7f4a3ae2e740 10 client.25475 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:11:41.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.685+0000 7f4a3ae2e740 20 client.25475 populate_metadata read hostname 'smithi029' 2024-03-11T19:11:41.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.685+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/3818613490 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558f8028c940 0x558f8028ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:11:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.685+0000 7f4a3ae2e740 4 client.25475 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:11:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.685+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/3818613490 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f4a240bc610 con 0x558f8028c940 2024-03-11T19:11:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.686+0000 7f4a35f16640 1 --2- 172.21.15.29:0/3818613490 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558f8028c940 0x558f8028ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:11:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.686+0000 7f4a157fa640 10 client.25475 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.686+0000 7f4a35f16640 1 --2- 172.21.15.29:0/3818613490 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558f8028c940 0x558f8028ed60 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87777008c6dd1274 server_cookie=f17f7087056f65e0 in_seq=0 out_seq=0 2024-03-11T19:11:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.687+0000 7f4a157fa640 1 -- 172.21.15.29:0/3818613490 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7f4a240bc610 con 0x558f8028c940 2024-03-11T19:11:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.687+0000 7f4a157fa640 10 client.25475 handle_command_reply: tid=0 2024-03-11T19:11:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.687+0000 7f4a3ae2e740 1 client.25475 shutdown 2024-03-11T19:11:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.687+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/3818613490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f80224e30 msgr2=0x558f8022e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.687+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/3818613490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f80224e30 0x558f8022e6b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x558f80220a10 tx=0x7f4a28009c60 comp rx=0 tx=0).stop 2024-03-11T19:11:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.687+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/3818613490 shutdown_connections 2024-03-11T19:11:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.687+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/3818613490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f80224e30 0x558f8022e6b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.687+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/3818613490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f8021fcc0 0x558f80230590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.687+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/3818613490 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558f8028c940 0x558f8028ed60 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.687+0000 7f4a157fa640 0 client.25475 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:11:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.687+0000 7f4a3ae2e740 1 --2- 172.21.15.29:0/3818613490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f8021f2d0 0x558f80230050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.687+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/3818613490 >> 172.21.15.29:0/3818613490 conn(0x558f8021a7f0 msgr2=0x558f80220fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.687+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/3818613490 shutdown_connections 2024-03-11T19:11:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.687+0000 7f4a3ae2e740 1 -- 172.21.15.29:0/3818613490 wait complete. 2024-03-11T19:11:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.688+0000 7f4a3ae2e740 20 client.25475 trim_cache size 0 max 16384 2024-03-11T19:11:41.694 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:11:41.694 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:11:41.695 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:11:41.695 INFO:teuthology.orchestra.run.smithi029.stdout: "eaa91b78-4526-42dd-99ae-e3ceae511852": { 2024-03-11T19:11:41.695 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:11:41.695 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "eaa91b78-4526-42dd-99ae-e3ceae511852", 2024-03-11T19:11:41.695 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:11:41.695 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:11:41.695 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:11:41.695 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:11:41.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.689+0000 7f4a3817a640 1 -- 172.21.15.29:0/1766785383 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4a180a1a30 msgr2=0x7f4a180a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:41.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.689+0000 7f4a3817a640 1 --2- 172.21.15.29:0/1766785383 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4a180a1a30 0x7f4a180a3ef0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4a30069590 tx=0x7f4a2000d040 comp rx=0 tx=0).stop 2024-03-11T19:11:41.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.689+0000 7f4a3817a640 1 -- 172.21.15.29:0/1766785383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a30102450 msgr2=0x7f4a30197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:11:41.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.689+0000 7f4a3817a640 1 --2- 172.21.15.29:0/1766785383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a30102450 0x7f4a30197700 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4a24036040 tx=0x7f4a24004370 comp rx=0 tx=0).stop 2024-03-11T19:11:41.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.689+0000 7f4a3817a640 1 -- 172.21.15.29:0/1766785383 shutdown_connections 2024-03-11T19:11:41.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.689+0000 7f4a3817a640 1 --2- 172.21.15.29:0/1766785383 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4a180a1a30 0x7f4a180a3ef0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.689+0000 7f4a3817a640 1 --2- 172.21.15.29:0/1766785383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a30102e10 0x7f4a3019e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.689+0000 7f4a3817a640 1 --2- 172.21.15.29:0/1766785383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a30102450 0x7f4a30197700 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.689+0000 7f4a3817a640 1 --2- 172.21.15.29:0/1766785383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a300ff480 0x7f4a301971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:11:41.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.689+0000 7f4a3817a640 1 -- 172.21.15.29:0/1766785383 >> 172.21.15.29:0/1766785383 conn(0x7f4a300fadb0 msgr2=0x7f4a30105750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:11:41.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.689+0000 7f4a3817a640 1 -- 172.21.15.29:0/1766785383 shutdown_connections 2024-03-11T19:11:41.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:11:41.689+0000 7f4a3817a640 1 -- 172.21.15.29:0/1766785383 wait complete. 2024-03-11T19:11:41.707 INFO:tasks.cephfs.filesystem:scrub status for tag:eaa91b78-4526-42dd-99ae-e3ceae511852 - {'path': '/', 'tag': 'eaa91b78-4526-42dd-99ae-e3ceae511852', 'options': 'recursive,force'} 2024-03-11T19:11:42.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:41 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:11:42.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:41 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:11:42.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:41 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:11:42.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:41 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:11:42.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:41 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:11:42.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:41 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:11:42.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:41 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:11:42.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:41 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:11:42.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:41 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:11:42.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:41 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:11:42.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:41 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:11:42.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:41 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:11:43.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:42 smithi029 ceph-mon[32439]: pgmap v408: 97 pgs: 97 active+clean; 4.5 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 176 MiB/s wr, 98 op/s 2024-03-11T19:11:43.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:42 smithi190 ceph-mon[36292]: pgmap v408: 97 pgs: 97 active+clean; 4.5 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 176 MiB/s wr, 98 op/s 2024-03-11T19:11:43.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:42 smithi170 ceph-mon[35690]: pgmap v408: 97 pgs: 97 active+clean; 4.5 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 176 MiB/s wr, 98 op/s 2024-03-11T19:11:45.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:44 smithi029 ceph-mon[32439]: pgmap v409: 97 pgs: 97 active+clean; 4.7 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 172 MiB/s wr, 98 op/s 2024-03-11T19:11:45.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:44 smithi190 ceph-mon[36292]: pgmap v409: 97 pgs: 97 active+clean; 4.7 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 172 MiB/s wr, 98 op/s 2024-03-11T19:11:45.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:44 smithi170 ceph-mon[35690]: pgmap v409: 97 pgs: 97 active+clean; 4.7 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 172 MiB/s wr, 98 op/s 2024-03-11T19:11:46.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:45 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:11:46.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:45 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:11:46.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:45 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:11:47.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:46 smithi029 ceph-mon[32439]: pgmap v410: 97 pgs: 97 active+clean; 5.2 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 180 MiB/s wr, 100 op/s 2024-03-11T19:11:47.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:46 smithi190 ceph-mon[36292]: pgmap v410: 97 pgs: 97 active+clean; 5.2 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 180 MiB/s wr, 100 op/s 2024-03-11T19:11:47.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:46 smithi170 ceph-mon[35690]: pgmap v410: 97 pgs: 97 active+clean; 5.2 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 180 MiB/s wr, 100 op/s 2024-03-11T19:11:49.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:48 smithi029 ceph-mon[32439]: pgmap v411: 97 pgs: 97 active+clean; 5.4 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 162 MiB/s wr, 92 op/s 2024-03-11T19:11:49.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:48 smithi190 ceph-mon[36292]: pgmap v411: 97 pgs: 97 active+clean; 5.4 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 162 MiB/s wr, 92 op/s 2024-03-11T19:11:49.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:48 smithi170 ceph-mon[35690]: pgmap v411: 97 pgs: 97 active+clean; 5.4 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 162 MiB/s wr, 92 op/s 2024-03-11T19:11:51.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:50 smithi029 ceph-mon[32439]: pgmap v412: 97 pgs: 97 active+clean; 5.8 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 173 MiB/s wr, 97 op/s 2024-03-11T19:11:51.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:50 smithi190 ceph-mon[36292]: pgmap v412: 97 pgs: 97 active+clean; 5.8 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 173 MiB/s wr, 97 op/s 2024-03-11T19:11:51.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:50 smithi170 ceph-mon[35690]: pgmap v412: 97 pgs: 97 active+clean; 5.8 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 173 MiB/s wr, 97 op/s 2024-03-11T19:11:53.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:52 smithi029 ceph-mon[32439]: pgmap v413: 97 pgs: 97 active+clean; 6.1 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 160 MiB/s wr, 90 op/s 2024-03-11T19:11:53.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:52 smithi190 ceph-mon[36292]: pgmap v413: 97 pgs: 97 active+clean; 6.1 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 160 MiB/s wr, 90 op/s 2024-03-11T19:11:53.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:52 smithi170 ceph-mon[35690]: pgmap v413: 97 pgs: 97 active+clean; 6.1 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 160 MiB/s wr, 90 op/s 2024-03-11T19:11:55.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:54 smithi029 ceph-mon[32439]: pgmap v414: 97 pgs: 97 active+clean; 6.5 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 167 MiB/s wr, 95 op/s 2024-03-11T19:11:55.261 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:54 smithi190 ceph-mon[36292]: pgmap v414: 97 pgs: 97 active+clean; 6.5 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 167 MiB/s wr, 95 op/s 2024-03-11T19:11:55.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:54 smithi170 ceph-mon[35690]: pgmap v414: 97 pgs: 97 active+clean; 6.5 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 167 MiB/s wr, 95 op/s 2024-03-11T19:11:57.148 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:56 smithi170 ceph-mon[35690]: pgmap v415: 97 pgs: 97 active+clean; 6.9 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 184 MiB/s wr, 101 op/s 2024-03-11T19:11:57.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:56 smithi029 ceph-mon[32439]: pgmap v415: 97 pgs: 97 active+clean; 6.9 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 184 MiB/s wr, 101 op/s 2024-03-11T19:11:57.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:56 smithi190 ceph-mon[36292]: pgmap v415: 97 pgs: 97 active+clean; 6.9 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 184 MiB/s wr, 101 op/s 2024-03-11T19:11:59.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:11:58 smithi029 ceph-mon[32439]: pgmap v416: 97 pgs: 97 active+clean; 7.1 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 160 MiB/s wr, 91 op/s 2024-03-11T19:11:59.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:11:58 smithi190 ceph-mon[36292]: pgmap v416: 97 pgs: 97 active+clean; 7.1 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 160 MiB/s wr, 91 op/s 2024-03-11T19:11:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:11:58 smithi170 ceph-mon[35690]: pgmap v416: 97 pgs: 97 active+clean; 7.1 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 160 MiB/s wr, 91 op/s 2024-03-11T19:12:01.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:00 smithi029 ceph-mon[32439]: pgmap v417: 97 pgs: 97 active+clean; 7.5 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 179 MiB/s wr, 100 op/s 2024-03-11T19:12:01.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:00 smithi190 ceph-mon[36292]: pgmap v417: 97 pgs: 97 active+clean; 7.5 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 179 MiB/s wr, 100 op/s 2024-03-11T19:12:01.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:00 smithi170 ceph-mon[35690]: pgmap v417: 97 pgs: 97 active+clean; 7.5 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 179 MiB/s wr, 100 op/s 2024-03-11T19:12:03.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:02 smithi029 ceph-mon[32439]: pgmap v418: 97 pgs: 97 active+clean; 7.8 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 172 MiB/s wr, 95 op/s 2024-03-11T19:12:03.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:02 smithi190 ceph-mon[36292]: pgmap v418: 97 pgs: 97 active+clean; 7.8 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 172 MiB/s wr, 95 op/s 2024-03-11T19:12:03.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:02 smithi170 ceph-mon[35690]: pgmap v418: 97 pgs: 97 active+clean; 7.8 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 172 MiB/s wr, 95 op/s 2024-03-11T19:12:05.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:04 smithi029 ceph-mon[32439]: pgmap v419: 97 pgs: 97 active+clean; 8.2 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 180 MiB/s wr, 101 op/s 2024-03-11T19:12:05.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:04 smithi190 ceph-mon[36292]: pgmap v419: 97 pgs: 97 active+clean; 8.2 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 180 MiB/s wr, 101 op/s 2024-03-11T19:12:05.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:04 smithi170 ceph-mon[35690]: pgmap v419: 97 pgs: 97 active+clean; 8.2 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 180 MiB/s wr, 101 op/s 2024-03-11T19:12:07.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:06 smithi029 ceph-mon[32439]: pgmap v420: 97 pgs: 97 active+clean; 8.6 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 183 MiB/s wr, 102 op/s 2024-03-11T19:12:07.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:06 smithi190 ceph-mon[36292]: pgmap v420: 97 pgs: 97 active+clean; 8.6 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 183 MiB/s wr, 102 op/s 2024-03-11T19:12:07.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:06 smithi170 ceph-mon[35690]: pgmap v420: 97 pgs: 97 active+clean; 8.6 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 183 MiB/s wr, 102 op/s 2024-03-11T19:12:09.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:08 smithi029 ceph-mon[32439]: pgmap v421: 97 pgs: 97 active+clean; 8.8 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 163 MiB/s wr, 92 op/s 2024-03-11T19:12:09.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:08 smithi190 ceph-mon[36292]: pgmap v421: 97 pgs: 97 active+clean; 8.8 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 163 MiB/s wr, 92 op/s 2024-03-11T19:12:09.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:08 smithi170 ceph-mon[35690]: pgmap v421: 97 pgs: 97 active+clean; 8.8 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 163 MiB/s wr, 92 op/s 2024-03-11T19:12:11.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:10 smithi029 ceph-mon[32439]: pgmap v422: 97 pgs: 97 active+clean; 9.3 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 185 MiB/s wr, 102 op/s 2024-03-11T19:12:11.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:10 smithi190 ceph-mon[36292]: pgmap v422: 97 pgs: 97 active+clean; 9.3 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 185 MiB/s wr, 102 op/s 2024-03-11T19:12:11.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:10 smithi170 ceph-mon[35690]: pgmap v422: 97 pgs: 97 active+clean; 9.3 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 185 MiB/s wr, 102 op/s 2024-03-11T19:12:11.708 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:12:11.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:12:11.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.855+0000 7f66e58c9640 1 -- 172.21.15.29:0/1838942388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e0101270 msgr2=0x7f66e01016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:11.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.855+0000 7f66e58c9640 1 --2- 172.21.15.29:0/1838942388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e0101270 0x7f66e01016f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f66d4009900 tx=0x7f66d402bc30 comp rx=0 tx=0).stop 2024-03-11T19:12:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.855+0000 7f66e58c9640 1 -- 172.21.15.29:0/1838942388 shutdown_connections 2024-03-11T19:12:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.855+0000 7f66e58c9640 1 --2- 172.21.15.29:0/1838942388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66e0101c30 0x7f66e01084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.855+0000 7f66e58c9640 1 --2- 172.21.15.29:0/1838942388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e0101270 0x7f66e01016f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.855+0000 7f66e58c9640 1 --2- 172.21.15.29:0/1838942388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66e0100080 0x7f66e01004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.855+0000 7f66e58c9640 1 -- 172.21.15.29:0/1838942388 >> 172.21.15.29:0/1838942388 conn(0x7f66e00fb5f0 msgr2=0x7f66e00fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.855+0000 7f66e58c9640 1 -- 172.21.15.29:0/1838942388 shutdown_connections 2024-03-11T19:12:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.855+0000 7f66e58c9640 1 -- 172.21.15.29:0/1838942388 wait complete. 2024-03-11T19:12:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.856+0000 7f66e58c9640 1 Processor -- start 2024-03-11T19:12:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.856+0000 7f66e58c9640 1 -- start start 2024-03-11T19:12:11.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.856+0000 7f66e58c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66e0100080 0x7f66e019ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:11.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.856+0000 7f66e58c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e0101270 0x7f66e01a2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:11.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.856+0000 7f66e58c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66e0101c30 0x7f66e01a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:11.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.856+0000 7f66e58c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e01a5250 con 0x7f66e0101c30 2024-03-11T19:12:11.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.856+0000 7f66e58c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e01a53c0 con 0x7f66e0101270 2024-03-11T19:12:11.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.856+0000 7f66e58c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e01a5580 con 0x7f66e0100080 2024-03-11T19:12:11.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.856+0000 7f66df7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66e0101c30 0x7f66e01a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:11.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.857+0000 7f66df7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66e0101c30 0x7f66e01a4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43670/0 (socket says 172.21.15.29:43670) 2024-03-11T19:12:11.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.857+0000 7f66df7fe640 1 -- 172.21.15.29:0/2826148272 learned_addr learned my addr 172.21.15.29:0/2826148272 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:11.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.857+0000 7f66de7fc640 1 --2- 172.21.15.29:0/2826148272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e0101270 0x7f66e01a2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:11.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.857+0000 7f66deffd640 1 --2- 172.21.15.29:0/2826148272 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66e0100080 0x7f66e019ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:11.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.857+0000 7f66df7fe640 1 -- 172.21.15.29:0/2826148272 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66e0100080 msgr2=0x7f66e019ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:11.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.857+0000 7f66df7fe640 1 --2- 172.21.15.29:0/2826148272 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66e0100080 0x7f66e019ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.857+0000 7f66df7fe640 1 -- 172.21.15.29:0/2826148272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e0101270 msgr2=0x7f66e01a2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:11.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.857+0000 7f66df7fe640 1 --2- 172.21.15.29:0/2826148272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e0101270 0x7f66e01a2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.857+0000 7f66df7fe640 1 -- 172.21.15.29:0/2826148272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d8009840 con 0x7f66e0101c30 2024-03-11T19:12:11.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.857+0000 7f66df7fe640 1 --2- 172.21.15.29:0/2826148272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66e0101c30 0x7f66e01a4ac0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f66d800efd0 tx=0x7f66d800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:11.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.858+0000 7f66e48c7640 1 -- 172.21.15.29:0/2826148272 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8009b40 con 0x7f66e0101c30 2024-03-11T19:12:11.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.858+0000 7f66e58c9640 1 -- 172.21.15.29:0/2826148272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d4009560 con 0x7f66e0101c30 2024-03-11T19:12:11.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.858+0000 7f66e58c9640 1 -- 172.21.15.29:0/2826148272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e01a5bb0 con 0x7f66e0101c30 2024-03-11T19:12:11.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.858+0000 7f66e48c7640 1 -- 172.21.15.29:0/2826148272 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f66d8003b70 con 0x7f66e0101c30 2024-03-11T19:12:11.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.858+0000 7f66e48c7640 1 -- 172.21.15.29:0/2826148272 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d80107b0 con 0x7f66e0101c30 2024-03-11T19:12:11.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.861+0000 7f66e48c7640 1 -- 172.21.15.29:0/2826148272 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f66d8010910 con 0x7f66e0101c30 2024-03-11T19:12:11.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.861+0000 7f66e8570740 1 Processor -- start 2024-03-11T19:12:11.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.861+0000 7f66e8570740 1 -- start start 2024-03-11T19:12:11.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.861+0000 7f66e8570740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada62d0 0x55908ada66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:11.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.861+0000 7f66e8570740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908ada6cc0 0x55908adab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:11.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.861+0000 7f66e8570740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908adabe30 0x55908adae2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:11.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.861+0000 7f66e8570740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55908ad16280 con 0x55908ada62d0 2024-03-11T19:12:11.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.861+0000 7f66e8570740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55908ad137b0 con 0x55908ada6cc0 2024-03-11T19:12:11.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.861+0000 7f66e8570740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55908ad0ef90 con 0x55908adabe30 2024-03-11T19:12:11.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.861+0000 7f66de7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada62d0 0x55908ada66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:11.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.861+0000 7f66de7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada62d0 0x55908ada66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43682/0 (socket says 172.21.15.29:43682) 2024-03-11T19:12:11.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.861+0000 7f66de7fc640 1 -- 172.21.15.29:0/3420312195 learned_addr learned my addr 172.21.15.29:0/3420312195 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:11.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.861+0000 7f66deffd640 1 --2- 172.21.15.29:0/3420312195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908ada6cc0 0x55908adab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:11.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.861+0000 7f66de7fc640 1 --2- 172.21.15.29:0/3420312195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908adabe30 0x55908adae2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:11.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.862+0000 7f66e48c7640 1 --2- 172.21.15.29:0/2826148272 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f66c80a1ab0 0x7f66c80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:11.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.862+0000 7f66df7fe640 1 --2- 172.21.15.29:0/2826148272 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f66c80a1ab0 0x7f66c80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:11.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.862+0000 7f66e48c7640 1 -- 172.21.15.29:0/2826148272 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f66d8014070 con 0x7f66e0101c30 2024-03-11T19:12:11.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.862+0000 7f66deffd640 1 -- 172.21.15.29:0/3420312195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908adabe30 msgr2=0x55908adae2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:11.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.862+0000 7f66deffd640 1 --2- 172.21.15.29:0/3420312195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908adabe30 0x55908adae2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.862+0000 7f66deffd640 1 -- 172.21.15.29:0/3420312195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada62d0 msgr2=0x55908ada66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:11.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.862+0000 7f66deffd640 1 --2- 172.21.15.29:0/3420312195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada62d0 0x55908ada66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.862+0000 7f66deffd640 1 -- 172.21.15.29:0/3420312195 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55908accd570 con 0x55908ada6cc0 2024-03-11T19:12:11.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.862+0000 7f66de7fc640 1 --2- 172.21.15.29:0/3420312195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada62d0 0x55908ada66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:11.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.862+0000 7f66df7fe640 1 --2- 172.21.15.29:0/2826148272 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f66c80a1ab0 0x7f66c80a3f70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f66d8097260 tx=0x7f66d80ba040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:11.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.863+0000 7f66deffd640 1 --2- 172.21.15.29:0/3420312195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908ada6cc0 0x55908adab7c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f66e01010d0 tx=0x7f66d00113e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aeeec9464207c0dc server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:11.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.863+0000 7f66c5ffb640 1 -- 172.21.15.29:0/3420312195 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d0011de0 con 0x55908ada6cc0 2024-03-11T19:12:11.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.863+0000 7f66c5ffb640 1 -- 172.21.15.29:0/3420312195 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f66d0012900 con 0x55908ada6cc0 2024-03-11T19:12:11.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.863+0000 7f66c5ffb640 1 -- 172.21.15.29:0/3420312195 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d000f8e0 con 0x55908ada6cc0 2024-03-11T19:12:11.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.864+0000 7f66e8570740 1 -- 172.21.15.29:0/3420312195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908ada6cc0 msgr2=0x55908adab7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:11.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.864+0000 7f66e8570740 1 --2- 172.21.15.29:0/3420312195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908ada6cc0 0x55908adab7c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f66e01010d0 tx=0x7f66d00113e0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.864+0000 7f66e8570740 1 -- 172.21.15.29:0/3420312195 shutdown_connections 2024-03-11T19:12:11.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.864+0000 7f66e8570740 1 --2- 172.21.15.29:0/3420312195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908adabe30 0x55908adae2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.864+0000 7f66e8570740 1 --2- 172.21.15.29:0/3420312195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908ada6cc0 0x55908adab7c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.864+0000 7f66e8570740 1 --2- 172.21.15.29:0/3420312195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada62d0 0x55908ada66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.864+0000 7f66e8570740 1 -- 172.21.15.29:0/3420312195 >> 172.21.15.29:0/3420312195 conn(0x55908ada17f0 msgr2=0x55908ada1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:11.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.864+0000 7f66e8570740 1 -- 172.21.15.29:0/3420312195 shutdown_connections 2024-03-11T19:12:11.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.864+0000 7f66e8570740 1 -- 172.21.15.29:0/3420312195 wait complete. 2024-03-11T19:12:11.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.864+0000 7f66e8570740 1 -- start start 2024-03-11T19:12:11.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.865+0000 7f66e8570740 1 Processor -- start 2024-03-11T19:12:11.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.865+0000 7f66e8570740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada62d0 0x55908adb0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:11.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.865+0000 7f66e8570740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908ada6cc0 0x55908adb0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:11.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.865+0000 7f66e8570740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908adabe30 0x55908adb77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:11.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.865+0000 7f66e8570740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55908a9e2430 con 0x55908ada62d0 2024-03-11T19:12:11.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.865+0000 7f66e8570740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55908a99ee00 con 0x55908ada6cc0 2024-03-11T19:12:11.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.865+0000 7f66e8570740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55908ac93fd0 con 0x55908adabe30 2024-03-11T19:12:11.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.865+0000 7f66e8570740 10 client.0 _ll_register_callbacks cb 0x55908ace9fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:11.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.865+0000 7f66deffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908ada6cc0 0x55908adb0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:11.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.865+0000 7f66deffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908ada6cc0 0x55908adb0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:48304/0 (socket says 172.21.15.29:48304) 2024-03-11T19:12:11.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.865+0000 7f66deffd640 1 -- 172.21.15.29:0/1667513509 learned_addr learned my addr 172.21.15.29:0/1667513509 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:11.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.865+0000 7f66de7fc640 1 --2- 172.21.15.29:0/1667513509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada62d0 0x55908adb0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:11.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.865+0000 7f66deffd640 1 -- 172.21.15.29:0/1667513509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908adabe30 msgr2=0x55908adb77d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:12:11.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.865+0000 7f66deffd640 1 --2- 172.21.15.29:0/1667513509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908adabe30 0x55908adb77d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.865+0000 7f66deffd640 1 -- 172.21.15.29:0/1667513509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada62d0 msgr2=0x55908adb0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:11.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.865+0000 7f66deffd640 1 --2- 172.21.15.29:0/1667513509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada62d0 0x55908adb0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.865+0000 7f66deffd640 1 -- 172.21.15.29:0/1667513509 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d00097a0 con 0x55908ada6cc0 2024-03-11T19:12:11.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.866+0000 7f66de7fc640 1 --2- 172.21.15.29:0/1667513509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada62d0 0x55908adb0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:11.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.866+0000 7f66deffd640 1 --2- 172.21.15.29:0/1667513509 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908ada6cc0 0x55908adb0790 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f66d001b000 tx=0x7f66d0012ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77ffe0069f9bd8b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:11.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.866+0000 7f66beffd640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:12:11.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.866+0000 7f66beffd640 1 -- 172.21.15.29:0/1667513509 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d0005450 con 0x55908ada6cc0 2024-03-11T19:12:11.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.866+0000 7f66beffd640 1 -- 172.21.15.29:0/1667513509 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f66d00055b0 con 0x55908ada6cc0 2024-03-11T19:12:11.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.866+0000 7f66e8570740 1 -- 172.21.15.29:0/1667513509 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55908aa0e7e0 con 0x55908ada6cc0 2024-03-11T19:12:11.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.866+0000 7f66beffd640 1 -- 172.21.15.29:0/1667513509 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66d0011e40 con 0x55908ada6cc0 2024-03-11T19:12:11.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.866+0000 7f66e8570740 10 client.25487 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:11.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.866+0000 7f66e8570740 1 -- 172.21.15.29:0/1667513509 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55908ad0d610 con 0x55908ada6cc0 2024-03-11T19:12:11.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.866+0000 7f66beffd640 1 -- 172.21.15.29:0/1667513509 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55908aa0e7e0 con 0x55908ada6cc0 2024-03-11T19:12:11.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.867+0000 7f66beffd640 1 -- 172.21.15.29:0/1667513509 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f66d0022040 con 0x55908ada6cc0 2024-03-11T19:12:11.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.867+0000 7f66e8570740 10 client.25487 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:11.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.867+0000 7f66e8570740 10 client.25487 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:12:11.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.867+0000 7f66e8570740 20 client.25487 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:11.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.867+0000 7f66e8570740 1 --2- 172.21.15.29:0/1667513509 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55908adbc4c0 0x55908adbe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:11.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.867+0000 7f66e8570740 4 client.25487 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:12:11.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.867+0000 7f66e8570740 1 -- 172.21.15.29:0/1667513509 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55908aa4b720 con 0x55908adbc4c0 2024-03-11T19:12:11.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.867+0000 7f66de7fc640 1 --2- 172.21.15.29:0/1667513509 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55908adbc4c0 0x55908adbe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:11.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.867+0000 7f66beffd640 10 client.25487 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:11.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.867+0000 7f66de7fc640 1 --2- 172.21.15.29:0/1667513509 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55908adbc4c0 0x55908adbe8e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53c8c75d760fd611 server_cookie=d9b997a0684065b7 in_seq=0 out_seq=0 2024-03-11T19:12:11.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.868+0000 7f66beffd640 1 -- 172.21.15.29:0/1667513509 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55908aa4b720 con 0x55908adbc4c0 2024-03-11T19:12:11.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.868+0000 7f66beffd640 10 client.25487 handle_command_reply: tid=0 2024-03-11T19:12:11.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.868+0000 7f66e8570740 1 client.25487 shutdown 2024-03-11T19:12:11.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.868+0000 7f66e8570740 1 -- 172.21.15.29:0/1667513509 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908ada6cc0 msgr2=0x55908adb0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:11.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.868+0000 7f66e8570740 1 --2- 172.21.15.29:0/1667513509 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908ada6cc0 0x55908adb0790 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f66d001b000 tx=0x7f66d0012ca0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.868+0000 7f66e8570740 1 -- 172.21.15.29:0/1667513509 shutdown_connections 2024-03-11T19:12:11.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.868+0000 7f66e8570740 1 --2- 172.21.15.29:0/1667513509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908adabe30 0x55908adb77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.868+0000 7f66e8570740 1 --2- 172.21.15.29:0/1667513509 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908ada6cc0 0x55908adb0790 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.868+0000 7f66e8570740 1 --2- 172.21.15.29:0/1667513509 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55908adbc4c0 0x55908adbe8e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.868+0000 7f66e8570740 1 --2- 172.21.15.29:0/1667513509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada62d0 0x55908adb0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.868+0000 7f66e8570740 1 -- 172.21.15.29:0/1667513509 >> 172.21.15.29:0/1667513509 conn(0x55908ada17f0 msgr2=0x55908ada0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:11.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.868+0000 7f66beffd640 0 client.25487 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:11.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.868+0000 7f66e8570740 1 -- 172.21.15.29:0/1667513509 shutdown_connections 2024-03-11T19:12:11.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.869+0000 7f66e8570740 1 -- 172.21.15.29:0/1667513509 wait complete. 2024-03-11T19:12:11.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.869+0000 7f66e8570740 20 client.25487 trim_cache size 0 max 16384 2024-03-11T19:12:11.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.898+0000 7f66e8570740 1 Processor -- start 2024-03-11T19:12:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.898+0000 7f66e8570740 1 -- start start 2024-03-11T19:12:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.898+0000 7f66e8570740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908ada62d0 0x55908adb0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.898+0000 7f66e8570740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada6cc0 0x55908adb7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.898+0000 7f66e8570740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908adabe30 0x55908adb0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.898+0000 7f66e8570740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55908aa4b750 con 0x55908ada6cc0 2024-03-11T19:12:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.898+0000 7f66e8570740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55908aa0e810 con 0x55908adabe30 2024-03-11T19:12:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.898+0000 7f66e8570740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55908ada73b0 con 0x55908ada62d0 2024-03-11T19:12:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.899+0000 7f66deffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada6cc0 0x55908adb7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.899+0000 7f66de7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908ada62d0 0x55908adb0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.899+0000 7f66de7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908ada62d0 0x55908adb0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41820/0 (socket says 172.21.15.29:41820) 2024-03-11T19:12:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.899+0000 7f66deffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada6cc0 0x55908adb7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43702/0 (socket says 172.21.15.29:43702) 2024-03-11T19:12:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.899+0000 7f66de7fc640 1 -- 172.21.15.29:0/4125008786 learned_addr learned my addr 172.21.15.29:0/4125008786 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.899+0000 7f66de7fc640 1 --2- 172.21.15.29:0/4125008786 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908adabe30 0x55908adb0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.899+0000 7f66de7fc640 1 -- 172.21.15.29:0/4125008786 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908adabe30 msgr2=0x55908adb0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:11.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.899+0000 7f66de7fc640 1 --2- 172.21.15.29:0/4125008786 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908adabe30 0x55908adb0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.899+0000 7f66de7fc640 1 -- 172.21.15.29:0/4125008786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada6cc0 msgr2=0x55908adb7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:11.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.899+0000 7f66de7fc640 1 --2- 172.21.15.29:0/4125008786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada6cc0 0x55908adb7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.899+0000 7f66de7fc640 1 -- 172.21.15.29:0/4125008786 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55908aca9ac0 con 0x55908ada62d0 2024-03-11T19:12:11.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.899+0000 7f66de7fc640 1 --2- 172.21.15.29:0/4125008786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908ada62d0 0x55908adb0250 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f66d4009340 tx=0x7f66d402c770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57b74a8a2621cc1c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:11.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.899+0000 7f66c57fa640 1 -- 172.21.15.29:0/4125008786 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d4004030 con 0x55908ada62d0 2024-03-11T19:12:11.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.900+0000 7f66c57fa640 1 -- 172.21.15.29:0/4125008786 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f66d404a070 con 0x55908ada62d0 2024-03-11T19:12:11.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.900+0000 7f66c57fa640 1 -- 172.21.15.29:0/4125008786 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d4003850 con 0x55908ada62d0 2024-03-11T19:12:11.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.900+0000 7f66e8570740 1 -- 172.21.15.29:0/4125008786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908ada62d0 msgr2=0x55908adb0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:11.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.900+0000 7f66e8570740 1 --2- 172.21.15.29:0/4125008786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908ada62d0 0x55908adb0250 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f66d4009340 tx=0x7f66d402c770 comp rx=0 tx=0).stop 2024-03-11T19:12:11.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.901+0000 7f66e8570740 1 -- 172.21.15.29:0/4125008786 shutdown_connections 2024-03-11T19:12:11.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.901+0000 7f66e8570740 1 --2- 172.21.15.29:0/4125008786 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908adabe30 0x55908adb0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.901+0000 7f66e8570740 1 --2- 172.21.15.29:0/4125008786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada6cc0 0x55908adb7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.901+0000 7f66e8570740 1 --2- 172.21.15.29:0/4125008786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908ada62d0 0x55908adb0250 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.901+0000 7f66e8570740 1 -- 172.21.15.29:0/4125008786 >> 172.21.15.29:0/4125008786 conn(0x55908ada17f0 msgr2=0x55908ada1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.901+0000 7f66e8570740 1 -- 172.21.15.29:0/4125008786 shutdown_connections 2024-03-11T19:12:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.901+0000 7f66e8570740 1 -- 172.21.15.29:0/4125008786 wait complete. 2024-03-11T19:12:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.901+0000 7f66e8570740 1 -- start start 2024-03-11T19:12:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.902+0000 7f66e8570740 1 Processor -- start 2024-03-11T19:12:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.902+0000 7f66e8570740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908ada62d0 0x55908adb5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.902+0000 7f66e8570740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada6cc0 0x55908adb6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.902+0000 7f66e8570740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908adabe30 0x55908adb6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.902+0000 7f66e8570740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55908ae129d0 con 0x55908ada6cc0 2024-03-11T19:12:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.902+0000 7f66e8570740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55908ae12b10 con 0x55908adabe30 2024-03-11T19:12:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.902+0000 7f66e8570740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55908ae12c50 con 0x55908ada62d0 2024-03-11T19:12:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.902+0000 7f66e8570740 10 client.0 _ll_register_callbacks cb 0x7f66d0009590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.902+0000 7f66de7fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908adabe30 0x55908adb6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.902+0000 7f66deffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada6cc0 0x55908adb6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.902+0000 7f66de7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908ada62d0 0x55908adb5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:11.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.902+0000 7f66de7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908ada62d0 0x55908adb5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41834/0 (socket says 172.21.15.29:41834) 2024-03-11T19:12:11.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.902+0000 7f66de7fc640 1 -- 172.21.15.29:0/1531558963 learned_addr learned my addr 172.21.15.29:0/1531558963 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:11.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.903+0000 7f66de7fc640 1 -- 172.21.15.29:0/1531558963 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908adabe30 msgr2=0x55908adb6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:11.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.903+0000 7f66de7fc640 1 --2- 172.21.15.29:0/1531558963 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908adabe30 0x55908adb6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.903+0000 7f66de7fc640 1 -- 172.21.15.29:0/1531558963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada6cc0 msgr2=0x55908adb6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:11.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.903+0000 7f66de7fc640 1 --2- 172.21.15.29:0/1531558963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada6cc0 0x55908adb6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.903+0000 7f66de7fc640 1 -- 172.21.15.29:0/1531558963 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d404c040 con 0x55908ada62d0 2024-03-11T19:12:11.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.904+0000 7f66de7fc640 1 --2- 172.21.15.29:0/1531558963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908ada62d0 0x55908adb5fc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f66d4009850 tx=0x7f66d4005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=974e5603d2ab4fe7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:11.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.904+0000 7f66beffd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:12:11.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.904+0000 7f66beffd640 1 -- 172.21.15.29:0/1531558963 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d4004050 con 0x55908ada62d0 2024-03-11T19:12:11.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.904+0000 7f66beffd640 1 -- 172.21.15.29:0/1531558963 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f66d402de40 con 0x55908ada62d0 2024-03-11T19:12:11.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.904+0000 7f66e8570740 1 -- 172.21.15.29:0/1531558963 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f66d0012900 con 0x55908ada62d0 2024-03-11T19:12:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.904+0000 7f66beffd640 1 -- 172.21.15.29:0/1531558963 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66d40041b0 con 0x55908ada62d0 2024-03-11T19:12:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.904+0000 7f66e8570740 10 client.25534 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.904+0000 7f66e8570740 1 -- 172.21.15.29:0/1531558963 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55908accf6a0 con 0x55908ada62d0 2024-03-11T19:12:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.904+0000 7f66beffd640 1 -- 172.21.15.29:0/1531558963 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f66d0012900 con 0x55908ada62d0 2024-03-11T19:12:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.904+0000 7f66beffd640 1 -- 172.21.15.29:0/1531558963 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f66d403c040 con 0x55908ada62d0 2024-03-11T19:12:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.905+0000 7f66e8570740 10 client.25534 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.905+0000 7f66e8570740 10 client.25534 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:12:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.905+0000 7f66e8570740 20 client.25534 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.905+0000 7f66e8570740 1 --2- 172.21.15.29:0/1531558963 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55908ae142e0 0x55908ae16700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.905+0000 7f66deffd640 1 --2- 172.21.15.29:0/1531558963 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55908ae142e0 0x55908ae16700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.905+0000 7f66e8570740 4 client.25534 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:12:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.906+0000 7f66e8570740 1 -- 172.21.15.29:0/1531558963 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55908aa9a890 con 0x55908ae142e0 2024-03-11T19:12:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.906+0000 7f66deffd640 1 --2- 172.21.15.29:0/1531558963 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55908ae142e0 0x55908ae16700 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62282ff0f02799f2 server_cookie=2c52fa4b46116b71 in_seq=0 out_seq=0 2024-03-11T19:12:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.906+0000 7f66beffd640 10 client.25534 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.906+0000 7f66beffd640 1 -- 172.21.15.29:0/1531558963 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55908aa9a890 con 0x55908ae142e0 2024-03-11T19:12:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.906+0000 7f66beffd640 10 client.25534 handle_command_reply: tid=0 2024-03-11T19:12:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.906+0000 7f66e8570740 1 client.25534 shutdown 2024-03-11T19:12:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.906+0000 7f66e8570740 1 -- 172.21.15.29:0/1531558963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908ada62d0 msgr2=0x55908adb5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.906+0000 7f66e8570740 1 --2- 172.21.15.29:0/1531558963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908ada62d0 0x55908adb5fc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f66d4009850 tx=0x7f66d4005fd0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.907+0000 7f66e8570740 1 -- 172.21.15.29:0/1531558963 shutdown_connections 2024-03-11T19:12:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.907+0000 7f66e8570740 1 --2- 172.21.15.29:0/1531558963 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55908adabe30 0x55908adb6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.907+0000 7f66e8570740 1 --2- 172.21.15.29:0/1531558963 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55908ae142e0 0x55908ae16700 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.907+0000 7f66e8570740 1 --2- 172.21.15.29:0/1531558963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55908ada6cc0 0x55908adb6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.907+0000 7f66e8570740 1 --2- 172.21.15.29:0/1531558963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55908ada62d0 0x55908adb5fc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.907+0000 7f66beffd640 0 client.25534 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:11.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.907+0000 7f66e8570740 1 -- 172.21.15.29:0/1531558963 >> 172.21.15.29:0/1531558963 conn(0x55908ada17f0 msgr2=0x55908ada7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:11.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.907+0000 7f66e8570740 1 -- 172.21.15.29:0/1531558963 shutdown_connections 2024-03-11T19:12:11.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.907+0000 7f66e8570740 1 -- 172.21.15.29:0/1531558963 wait complete. 2024-03-11T19:12:11.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.907+0000 7f66e8570740 20 client.25534 trim_cache size 0 max 16384 2024-03-11T19:12:11.912 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:12:11.912 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:12:11.912 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:12:11.912 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:12:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.908+0000 7f66e58c9640 1 -- 172.21.15.29:0/2826148272 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f66c80a1ab0 msgr2=0x7f66c80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.908+0000 7f66e58c9640 1 --2- 172.21.15.29:0/2826148272 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f66c80a1ab0 0x7f66c80a3f70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f66d8097260 tx=0x7f66d80ba040 comp rx=0 tx=0).stop 2024-03-11T19:12:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.908+0000 7f66e58c9640 1 -- 172.21.15.29:0/2826148272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66e0101c30 msgr2=0x7f66e01a4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:11.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.908+0000 7f66e58c9640 1 --2- 172.21.15.29:0/2826148272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66e0101c30 0x7f66e01a4ac0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f66d800efd0 tx=0x7f66d800c6a0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.908+0000 7f66e58c9640 1 -- 172.21.15.29:0/2826148272 shutdown_connections 2024-03-11T19:12:11.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.908+0000 7f66e58c9640 1 --2- 172.21.15.29:0/2826148272 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f66c80a1ab0 0x7f66c80a3f70 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.908+0000 7f66e58c9640 1 --2- 172.21.15.29:0/2826148272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66e0101c30 0x7f66e01a4ac0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.908+0000 7f66e58c9640 1 --2- 172.21.15.29:0/2826148272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e0101270 0x7f66e01a2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.908+0000 7f66e58c9640 1 --2- 172.21.15.29:0/2826148272 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66e0100080 0x7f66e019ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:11.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.908+0000 7f66e58c9640 1 -- 172.21.15.29:0/2826148272 >> 172.21.15.29:0/2826148272 conn(0x7f66e00fb5f0 msgr2=0x7f66e00fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:11.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.909+0000 7f66e58c9640 1 -- 172.21.15.29:0/2826148272 shutdown_connections 2024-03-11T19:12:11.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:11.909+0000 7f66e58c9640 1 -- 172.21.15.29:0/2826148272 wait complete. 2024-03-11T19:12:11.929 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:12:11.930 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:12:12.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.091+0000 7f9965ab5640 1 -- 172.21.15.29:0/3131785621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996010d780 msgr2=0x7f996010fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.091+0000 7f9965ab5640 1 --2- 172.21.15.29:0/3131785621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996010d780 0x7f996010fb70 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f9958009cd0 tx=0x7f995802c0f0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.091+0000 7f9965ab5640 1 -- 172.21.15.29:0/3131785621 shutdown_connections 2024-03-11T19:12:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.091+0000 7f9965ab5640 1 --2- 172.21.15.29:0/3131785621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996010d780 0x7f996010fb70 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.091+0000 7f9965ab5640 1 --2- 172.21.15.29:0/3131785621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9960073be0 0x7f9960074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.091+0000 7f9965ab5640 1 --2- 172.21.15.29:0/3131785621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99600751c0 0x7f9960073610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.091+0000 7f9965ab5640 1 -- 172.21.15.29:0/3131785621 >> 172.21.15.29:0/3131785621 conn(0x7f99600fc720 msgr2=0x7f99600feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.091+0000 7f9965ab5640 1 -- 172.21.15.29:0/3131785621 shutdown_connections 2024-03-11T19:12:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.091+0000 7f9965ab5640 1 -- 172.21.15.29:0/3131785621 wait complete. 2024-03-11T19:12:12.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.092+0000 7f9965ab5640 1 Processor -- start 2024-03-11T19:12:12.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.092+0000 7f9965ab5640 1 -- start start 2024-03-11T19:12:12.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.092+0000 7f9965ab5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9960073be0 0x7f996019d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.092+0000 7f9965ab5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99600751c0 0x7f996019dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.092+0000 7f9965ab5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f996010d780 0x7f99601a4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.092+0000 7f9965ab5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99601a51d0 con 0x7f99600751c0 2024-03-11T19:12:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.092+0000 7f9965ab5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99601a5340 con 0x7f996010d780 2024-03-11T19:12:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.092+0000 7f9965ab5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99601a5500 con 0x7f9960073be0 2024-03-11T19:12:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.093+0000 7f9957fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99600751c0 0x7f996019dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.093+0000 7f995ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f996010d780 0x7f99601a4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.093+0000 7f995ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f996010d780 0x7f99601a4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:48326/0 (socket says 172.21.15.29:48326) 2024-03-11T19:12:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.093+0000 7f995ffff640 1 -- 172.21.15.29:0/1434186839 learned_addr learned my addr 172.21.15.29:0/1434186839 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.093+0000 7f995f7fe640 1 --2- 172.21.15.29:0/1434186839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9960073be0 0x7f996019d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.093+0000 7f995ffff640 1 -- 172.21.15.29:0/1434186839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9960073be0 msgr2=0x7f996019d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.093+0000 7f995ffff640 1 --2- 172.21.15.29:0/1434186839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9960073be0 0x7f996019d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.093+0000 7f995ffff640 1 -- 172.21.15.29:0/1434186839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99600751c0 msgr2=0x7f996019dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.093+0000 7f995ffff640 1 --2- 172.21.15.29:0/1434186839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99600751c0 0x7f996019dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.093+0000 7f995ffff640 1 -- 172.21.15.29:0/1434186839 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9948009490 con 0x7f996010d780 2024-03-11T19:12:12.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.093+0000 7f995ffff640 1 --2- 172.21.15.29:0/1434186839 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f996010d780 0x7f99601a4c90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f995802c630 tx=0x7f9958032900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:12.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.094+0000 7f995d7fa640 1 -- 172.21.15.29:0/1434186839 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9958034e70 con 0x7f996010d780 2024-03-11T19:12:12.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.094+0000 7f9965ab5640 1 -- 172.21.15.29:0/1434186839 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9958009900 con 0x7f996010d780 2024-03-11T19:12:12.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.094+0000 7f995d7fa640 1 -- 172.21.15.29:0/1434186839 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9958003e20 con 0x7f996010d780 2024-03-11T19:12:12.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.094+0000 7f995d7fa640 1 -- 172.21.15.29:0/1434186839 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99580046f0 con 0x7f996010d780 2024-03-11T19:12:12.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.094+0000 7f9965ab5640 1 -- 172.21.15.29:0/1434186839 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99601a5aa0 con 0x7f996010d780 2024-03-11T19:12:12.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.096+0000 7f9965ab5640 1 -- 172.21.15.29:0/1434186839 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9960103950 con 0x7f996010d780 2024-03-11T19:12:12.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.097+0000 7f995d7fa640 1 -- 172.21.15.29:0/1434186839 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f995803b040 con 0x7f996010d780 2024-03-11T19:12:12.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.097+0000 7f995d7fa640 1 --2- 172.21.15.29:0/1434186839 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f99400a1a30 0x7f99400a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.097+0000 7f995d7fa640 1 -- 172.21.15.29:0/1434186839 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f99580efc30 con 0x7f996010d780 2024-03-11T19:12:12.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.098+0000 7f995f7fe640 1 --2- 172.21.15.29:0/1434186839 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f99400a1a30 0x7f99400a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.098+0000 7f995f7fe640 1 --2- 172.21.15.29:0/1434186839 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f99400a1a30 0x7f99400a3ef0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f995000abe0 tx=0x7f995000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:12.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.103+0000 7f995d7fa640 1 -- 172.21.15.29:0/1434186839 <== mon.2 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f99580b8ee0 con 0x7f996010d780 2024-03-11T19:12:12.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.504+0000 7f9965ab5640 1 -- 172.21.15.29:0/1434186839 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f996002cdd0 con 0x7f996010d780 2024-03-11T19:12:12.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.506+0000 7f995d7fa640 1 -- 172.21.15.29:0/1434186839 <== mon.2 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f99580bc5a0 con 0x7f996010d780 2024-03-11T19:12:12.509 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:12:12.509 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:12:12.509 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:12:12.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.508+0000 7f9965ab5640 1 -- 172.21.15.29:0/1434186839 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f99400a1a30 msgr2=0x7f99400a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.508+0000 7f9965ab5640 1 --2- 172.21.15.29:0/1434186839 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f99400a1a30 0x7f99400a3ef0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f995000abe0 tx=0x7f995000a2d0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.508+0000 7f9965ab5640 1 -- 172.21.15.29:0/1434186839 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f996010d780 msgr2=0x7f99601a4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.508+0000 7f9965ab5640 1 --2- 172.21.15.29:0/1434186839 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f996010d780 0x7f99601a4c90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f995802c630 tx=0x7f9958032900 comp rx=0 tx=0).stop 2024-03-11T19:12:12.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.509+0000 7f9965ab5640 1 -- 172.21.15.29:0/1434186839 shutdown_connections 2024-03-11T19:12:12.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.509+0000 7f9965ab5640 1 --2- 172.21.15.29:0/1434186839 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f996010d780 0x7f99601a4c90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.509+0000 7f9965ab5640 1 --2- 172.21.15.29:0/1434186839 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f99400a1a30 0x7f99400a3ef0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.509+0000 7f9965ab5640 1 --2- 172.21.15.29:0/1434186839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99600751c0 0x7f996019dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.509+0000 7f9965ab5640 1 --2- 172.21.15.29:0/1434186839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9960073be0 0x7f996019d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.510+0000 7f9965ab5640 1 -- 172.21.15.29:0/1434186839 >> 172.21.15.29:0/1434186839 conn(0x7f99600fc720 msgr2=0x7f9960100b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:12.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.510+0000 7f9965ab5640 1 -- 172.21.15.29:0/1434186839 shutdown_connections 2024-03-11T19:12:12.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.510+0000 7f9965ab5640 1 -- 172.21.15.29:0/1434186839 wait complete. 2024-03-11T19:12:12.529 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:12:12.529 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:12:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.685+0000 7f4e3a5e4640 1 -- 172.21.15.29:0/1154508766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34101c30 msgr2=0x7f4e34069710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.685+0000 7f4e3a5e4640 1 --2- 172.21.15.29:0/1154508766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34101c30 0x7f4e34069710 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009ca0 tx=0x7f4e2c02c100 comp rx=0 tx=0).stop 2024-03-11T19:12:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.685+0000 7f4e3a5e4640 1 -- 172.21.15.29:0/1154508766 shutdown_connections 2024-03-11T19:12:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.685+0000 7f4e3a5e4640 1 --2- 172.21.15.29:0/1154508766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34101c30 0x7f4e34069710 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.685+0000 7f4e3a5e4640 1 --2- 172.21.15.29:0/1154508766 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e34101270 0x7f4e341016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.685+0000 7f4e3a5e4640 1 --2- 172.21.15.29:0/1154508766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e34100080 0x7f4e341004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.685+0000 7f4e3a5e4640 1 -- 172.21.15.29:0/1154508766 >> 172.21.15.29:0/1154508766 conn(0x7f4e340fb610 msgr2=0x7f4e340fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.685+0000 7f4e3a5e4640 1 -- 172.21.15.29:0/1154508766 shutdown_connections 2024-03-11T19:12:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.685+0000 7f4e3a5e4640 1 -- 172.21.15.29:0/1154508766 wait complete. 2024-03-11T19:12:12.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.686+0000 7f4e3a5e4640 1 Processor -- start 2024-03-11T19:12:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.686+0000 7f4e3a5e4640 1 -- start start 2024-03-11T19:12:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.686+0000 7f4e3a5e4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e34100080 0x7f4e341992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.686+0000 7f4e3a5e4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34101270 0x7f4e34199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.686+0000 7f4e3a5e4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e34101c30 0x7f4e341a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.686+0000 7f4e3a5e4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e341a0db0 con 0x7f4e34101270 2024-03-11T19:12:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.686+0000 7f4e3a5e4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e341a0f20 con 0x7f4e34100080 2024-03-11T19:12:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.686+0000 7f4e3a5e4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e341a10e0 con 0x7f4e34101c30 2024-03-11T19:12:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.686+0000 7f4e33fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e34100080 0x7f4e341992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.686+0000 7f4e337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34101270 0x7f4e34199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.686+0000 7f4e33fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e34100080 0x7f4e341992c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:48346/0 (socket says 172.21.15.29:48346) 2024-03-11T19:12:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.686+0000 7f4e33fff640 1 -- 172.21.15.29:0/3386127073 learned_addr learned my addr 172.21.15.29:0/3386127073 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.686+0000 7f4e337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34101270 0x7f4e34199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43740/0 (socket says 172.21.15.29:43740) 2024-03-11T19:12:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.686+0000 7f4e38b81640 1 --2- 172.21.15.29:0/3386127073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e34101c30 0x7f4e341a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.687+0000 7f4e38b81640 1 -- 172.21.15.29:0/3386127073 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e34100080 msgr2=0x7f4e341992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.687+0000 7f4e38b81640 1 --2- 172.21.15.29:0/3386127073 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e34100080 0x7f4e341992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.687+0000 7f4e38b81640 1 -- 172.21.15.29:0/3386127073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34101270 msgr2=0x7f4e34199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.687+0000 7f4e38b81640 1 --2- 172.21.15.29:0/3386127073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34101270 0x7f4e34199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.687+0000 7f4e38b81640 1 -- 172.21.15.29:0/3386127073 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c009900 con 0x7f4e34101c30 2024-03-11T19:12:12.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.687+0000 7f4e337fe640 1 --2- 172.21.15.29:0/3386127073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34101270 0x7f4e34199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:12.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.687+0000 7f4e38b81640 1 --2- 172.21.15.29:0/3386127073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e34101c30 0x7f4e341a0870 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c02c640 tx=0x7f4e2c0329a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:12.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.687+0000 7f4e317fa640 1 -- 172.21.15.29:0/3386127073 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c034850 con 0x7f4e34101c30 2024-03-11T19:12:12.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.687+0000 7f4e3a5e4640 1 -- 172.21.15.29:0/3386127073 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e341a1360 con 0x7f4e34101c30 2024-03-11T19:12:12.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.687+0000 7f4e317fa640 1 -- 172.21.15.29:0/3386127073 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4e2c003e20 con 0x7f4e34101c30 2024-03-11T19:12:12.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.687+0000 7f4e317fa640 1 -- 172.21.15.29:0/3386127073 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c004830 con 0x7f4e34101c30 2024-03-11T19:12:12.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.687+0000 7f4e3a5e4640 1 -- 172.21.15.29:0/3386127073 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e341a1830 con 0x7f4e34101c30 2024-03-11T19:12:12.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.690+0000 7f4e317fa640 1 -- 172.21.15.29:0/3386127073 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4e2c03b040 con 0x7f4e34101c30 2024-03-11T19:12:12.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.691+0000 7f4e317fa640 1 --2- 172.21.15.29:0/3386127073 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4e1c0a1a30 0x7f4e1c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.692+0000 7f4e317fa640 1 -- 172.21.15.29:0/3386127073 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4e2c0f0e40 con 0x7f4e34101c30 2024-03-11T19:12:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.692+0000 7f4e3d1b0740 1 Processor -- start 2024-03-11T19:12:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.692+0000 7f4e33fff640 1 --2- 172.21.15.29:0/3386127073 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4e1c0a1a30 0x7f4e1c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.692+0000 7f4e3d1b0740 1 -- start start 2024-03-11T19:12:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.692+0000 7f4e3d1b0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229382270 0x55d229382690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.692+0000 7f4e3d1b0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382c60 0x55d229387760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.692+0000 7f4e3d1b0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229387dd0 0x55d22938a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.692+0000 7f4e3d1b0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d22933e960 con 0x55d229387dd0 2024-03-11T19:12:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.692+0000 7f4e3d1b0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2292f2310 con 0x55d229382c60 2024-03-11T19:12:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.692+0000 7f4e3d1b0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2292ef840 con 0x55d229382270 2024-03-11T19:12:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.693+0000 7f4e38b81640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229387dd0 0x55d22938a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.693+0000 7f4e337fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382c60 0x55d229387760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.693+0000 7f4e337fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382c60 0x55d229387760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:48352/0 (socket says 172.21.15.29:48352) 2024-03-11T19:12:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.693+0000 7f4e337fe640 1 -- 172.21.15.29:0/180468415 learned_addr learned my addr 172.21.15.29:0/180468415 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.693+0000 7f4e337fe640 1 -- 172.21.15.29:0/180468415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229382270 msgr2=0x55d229382690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:12:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.693+0000 7f4e337fe640 1 --2- 172.21.15.29:0/180468415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229382270 0x55d229382690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.693+0000 7f4e337fe640 1 -- 172.21.15.29:0/180468415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229387dd0 msgr2=0x55d22938a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.693+0000 7f4e337fe640 1 --2- 172.21.15.29:0/180468415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229387dd0 0x55d22938a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.693+0000 7f4e337fe640 1 -- 172.21.15.29:0/180468415 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d2292ad060 con 0x55d229382c60 2024-03-11T19:12:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.693+0000 7f4e337fe640 1 --2- 172.21.15.29:0/180468415 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382c60 0x55d229387760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4e280114f0 tx=0x7f4e280119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19f9c49751ccb697 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:12.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.693+0000 7f4e1a7fc640 1 -- 172.21.15.29:0/180468415 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e28004030 con 0x55d229382c60 2024-03-11T19:12:12.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.693+0000 7f4e33fff640 1 --2- 172.21.15.29:0/3386127073 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4e1c0a1a30 0x7f4e1c0a3ef0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400bc80 tx=0x7f4e2400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:12.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.694+0000 7f4e1a7fc640 1 -- 172.21.15.29:0/180468415 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4e280035f0 con 0x55d229382c60 2024-03-11T19:12:12.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.694+0000 7f4e1a7fc640 1 -- 172.21.15.29:0/180468415 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2800f890 con 0x55d229382c60 2024-03-11T19:12:12.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.694+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/180468415 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382c60 msgr2=0x55d229387760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.694+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/180468415 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382c60 0x55d229387760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4e280114f0 tx=0x7f4e280119b0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.695+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/180468415 shutdown_connections 2024-03-11T19:12:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.695+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/180468415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229387dd0 0x55d22938a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.695+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/180468415 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382c60 0x55d229387760 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.695+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/180468415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229382270 0x55d229382690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.695+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/180468415 >> 172.21.15.29:0/180468415 conn(0x55d22937d7e0 msgr2=0x55d22937dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.695+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/180468415 shutdown_connections 2024-03-11T19:12:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.695+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/180468415 wait complete. 2024-03-11T19:12:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.695+0000 7f4e3d1b0740 1 -- start start 2024-03-11T19:12:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.696+0000 7f4e3d1b0740 1 Processor -- start 2024-03-11T19:12:12.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.696+0000 7f4e3d1b0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229382270 0x55d22938c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.696+0000 7f4e33fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229382270 0x55d22938c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.696+0000 7f4e33fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229382270 0x55d22938c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43772/0 (socket says 172.21.15.29:43772) 2024-03-11T19:12:12.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.696+0000 7f4e33fff640 1 -- 172.21.15.29:0/228699810 learned_addr learned my addr 172.21.15.29:0/228699810 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.696+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/228699810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382c60 0x55d22938ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.696+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/228699810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229387dd0 0x55d229393a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.696+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/228699810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2292eb020 con 0x55d229382270 2024-03-11T19:12:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.696+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/228699810 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d228fbe430 con 0x55d229382c60 2024-03-11T19:12:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.696+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/228699810 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d228f7ae00 con 0x55d229387dd0 2024-03-11T19:12:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.696+0000 7f4e337fe640 1 --2- 172.21.15.29:0/228699810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382c60 0x55d22938ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.696+0000 7f4e3d1b0740 10 client.0 _ll_register_callbacks cb 0x55d229292cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.696+0000 7f4e38b81640 1 --2- 172.21.15.29:0/228699810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229387dd0 0x55d229393a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.697+0000 7f4e33fff640 1 -- 172.21.15.29:0/228699810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229387dd0 msgr2=0x55d229393a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.697+0000 7f4e33fff640 1 --2- 172.21.15.29:0/228699810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229387dd0 0x55d229393a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.697+0000 7f4e33fff640 1 -- 172.21.15.29:0/228699810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382c60 msgr2=0x55d22938ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.697+0000 7f4e33fff640 1 --2- 172.21.15.29:0/228699810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382c60 0x55d22938ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.697+0000 7f4e33fff640 1 -- 172.21.15.29:0/228699810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e28009cc0 con 0x55d229382270 2024-03-11T19:12:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.697+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/228699810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d228fea7e0 con 0x55d229382270 2024-03-11T19:12:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.697+0000 7f4e33fff640 1 --2- 172.21.15.29:0/228699810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229382270 0x55d22938c510 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400b780 tx=0x7f4e24004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d706a86493a067 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.697+0000 7f4e137fe640 10 client.15978 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:12:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.697+0000 7f4e137fe640 1 -- 172.21.15.29:0/228699810 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2401cac0 con 0x55d229382270 2024-03-11T19:12:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.697+0000 7f4e137fe640 1 -- 172.21.15.29:0/228699810 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4e24022070 con 0x55d229382270 2024-03-11T19:12:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.697+0000 7f4e137fe640 1 -- 172.21.15.29:0/228699810 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e2401cc20 con 0x55d229382270 2024-03-11T19:12:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.697+0000 7f4e3d1b0740 10 client.15978 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.697+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/228699810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x55d2292e96a0 con 0x55d229382270 2024-03-11T19:12:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.698+0000 7f4e137fe640 1 -- 172.21.15.29:0/228699810 <== mon.0 v2:172.21.15.29:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4e24027480 con 0x55d229382270 2024-03-11T19:12:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.698+0000 7f4e3d1b0740 10 client.15978 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.698+0000 7f4e3d1b0740 10 client.15978 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:12:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.698+0000 7f4e3d1b0740 20 client.15978 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.698+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/228699810 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55d2293985a0 0x55d22939a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.698+0000 7f4e3d1b0740 4 client.15978 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:12:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.698+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/228699810 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d229027720 con 0x55d2293985a0 2024-03-11T19:12:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.698+0000 7f4e337fe640 1 --2- 172.21.15.29:0/228699810 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55d2293985a0 0x55d22939a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.698+0000 7f4e337fe640 1 --2- 172.21.15.29:0/228699810 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55d2293985a0 0x55d22939a9d0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b1165e6fa2cd1d91 server_cookie=5bc0ce7d2f674d7e in_seq=0 out_seq=0 2024-03-11T19:12:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.698+0000 7f4e137fe640 10 client.15978 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:12:12.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.699+0000 7f4e137fe640 1 -- 172.21.15.29:0/228699810 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d229027720 con 0x55d2293985a0 2024-03-11T19:12:12.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.699+0000 7f4e137fe640 10 client.15978 handle_command_reply: tid=0 2024-03-11T19:12:12.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.699+0000 7f4e3d1b0740 1 client.15978 shutdown 2024-03-11T19:12:12.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.699+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/228699810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229382270 msgr2=0x55d22938c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.699+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/228699810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229382270 0x55d22938c510 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400b780 tx=0x7f4e24004730 comp rx=0 tx=0).stop 2024-03-11T19:12:12.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.699+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/228699810 shutdown_connections 2024-03-11T19:12:12.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.699+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/228699810 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55d2293985a0 0x55d22939a9d0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.699+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/228699810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229387dd0 0x55d229393a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.699+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/228699810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382c60 0x55d22938ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.699+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/228699810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229382270 0x55d22938c510 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.699+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/228699810 >> 172.21.15.29:0/228699810 conn(0x55d22937d7e0 msgr2=0x55d22937cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:12.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.699+0000 7f4e137fe640 0 client.15978 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:12:12.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.699+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/228699810 shutdown_connections 2024-03-11T19:12:12.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.699+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/228699810 wait complete. 2024-03-11T19:12:12.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.700+0000 7f4e3d1b0740 20 client.15978 trim_cache size 0 max 16384 2024-03-11T19:12:12.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.717+0000 7f4e3d1b0740 1 Processor -- start 2024-03-11T19:12:12.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.717+0000 7f4e3d1b0740 1 -- start start 2024-03-11T19:12:12.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.717+0000 7f4e3d1b0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382270 0x55d22939f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.717+0000 7f4e3d1b0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229382c60 0x55d229393a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.717+0000 7f4e3d1b0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229387dd0 0x55d2293841f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.717+0000 7f4e3d1b0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d229270130 con 0x55d229387dd0 2024-03-11T19:12:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.717+0000 7f4e3d1b0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d229027720 con 0x55d229382270 2024-03-11T19:12:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.717+0000 7f4e3d1b0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d229379370 con 0x55d229382c60 2024-03-11T19:12:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.717+0000 7f4e33fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382270 0x55d22939f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.717+0000 7f4e33fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382270 0x55d22939f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:48370/0 (socket says 172.21.15.29:48370) 2024-03-11T19:12:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.718+0000 7f4e33fff640 1 -- 172.21.15.29:0/187606218 learned_addr learned my addr 172.21.15.29:0/187606218 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.718+0000 7f4e38b81640 1 --2- 172.21.15.29:0/187606218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229387dd0 0x55d2293841f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.718+0000 7f4e337fe640 1 --2- 172.21.15.29:0/187606218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229382c60 0x55d229393a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.718+0000 7f4e33fff640 1 -- 172.21.15.29:0/187606218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229382c60 msgr2=0x55d229393a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.718+0000 7f4e33fff640 1 --2- 172.21.15.29:0/187606218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229382c60 0x55d229393a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.718+0000 7f4e33fff640 1 -- 172.21.15.29:0/187606218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229387dd0 msgr2=0x55d2293841f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.718+0000 7f4e33fff640 1 --2- 172.21.15.29:0/187606218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229387dd0 0x55d2293841f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.718+0000 7f4e33fff640 1 -- 172.21.15.29:0/187606218 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d229285c20 con 0x55d229382270 2024-03-11T19:12:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.718+0000 7f4e337fe640 1 --2- 172.21.15.29:0/187606218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229382c60 0x55d229393a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:12:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.718+0000 7f4e33fff640 1 --2- 172.21.15.29:0/187606218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382270 0x55d22939f340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400b730 tx=0x7f4e2401f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b74629c3e1b5706 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:12.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.718+0000 7f4e19ffb640 1 -- 172.21.15.29:0/187606218 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2403f070 con 0x55d229382270 2024-03-11T19:12:12.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.719+0000 7f4e19ffb640 1 -- 172.21.15.29:0/187606218 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55d228fea7e0 con 0x55d229382270 2024-03-11T19:12:12.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.719+0000 7f4e19ffb640 1 -- 172.21.15.29:0/187606218 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e24027ca0 con 0x55d229382270 2024-03-11T19:12:12.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.720+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/187606218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382270 msgr2=0x55d22939f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.720+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/187606218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382270 0x55d22939f340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400b730 tx=0x7f4e2401f000 comp rx=0 tx=0).stop 2024-03-11T19:12:12.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.720+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/187606218 shutdown_connections 2024-03-11T19:12:12.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.720+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/187606218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229387dd0 0x55d2293841f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.720+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/187606218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229382c60 0x55d229393a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.720+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/187606218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382270 0x55d22939f340 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.720+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/187606218 >> 172.21.15.29:0/187606218 conn(0x55d22937d7e0 msgr2=0x55d22937dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:12.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.720+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/187606218 shutdown_connections 2024-03-11T19:12:12.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.720+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/187606218 wait complete. 2024-03-11T19:12:12.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.720+0000 7f4e3d1b0740 1 -- start start 2024-03-11T19:12:12.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.720+0000 7f4e3d1b0740 1 Processor -- start 2024-03-11T19:12:12.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.721+0000 7f4e3d1b0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382270 0x55d229393540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.721+0000 7f4e3d1b0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229382c60 0x55d229391b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.721+0000 7f4e3d1b0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229387dd0 0x55d229392090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.721+0000 7f4e3d1b0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d229393d30 con 0x55d229382c60 2024-03-11T19:12:12.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.721+0000 7f4e3d1b0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2293925d0 con 0x55d229382270 2024-03-11T19:12:12.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.721+0000 7f4e3d1b0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d229392790 con 0x55d229387dd0 2024-03-11T19:12:12.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.721+0000 7f4e3d1b0740 10 client.0 _ll_register_callbacks cb 0x7f4e28009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:12.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.721+0000 7f4e337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229382c60 0x55d229391b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.721+0000 7f4e38b81640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229387dd0 0x55d229392090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.721+0000 7f4e337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229382c60 0x55d229391b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43786/0 (socket says 172.21.15.29:43786) 2024-03-11T19:12:12.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.721+0000 7f4e337fe640 1 -- 172.21.15.29:0/4289026808 learned_addr learned my addr 172.21.15.29:0/4289026808 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:12.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.721+0000 7f4e33fff640 1 --2- 172.21.15.29:0/4289026808 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382270 0x55d229393540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.721+0000 7f4e38b81640 1 -- 172.21.15.29:0/4289026808 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382270 msgr2=0x55d229393540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:12.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.721+0000 7f4e38b81640 1 --2- 172.21.15.29:0/4289026808 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382270 0x55d229393540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.721+0000 7f4e38b81640 1 -- 172.21.15.29:0/4289026808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229382c60 msgr2=0x55d229391b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:12.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.722+0000 7f4e38b81640 1 --2- 172.21.15.29:0/4289026808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229382c60 0x55d229391b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.722+0000 7f4e38b81640 1 -- 172.21.15.29:0/4289026808 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d2292e96a0 con 0x55d229387dd0 2024-03-11T19:12:12.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.722+0000 7f4e337fe640 1 --2- 172.21.15.29:0/4289026808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229382c60 0x55d229391b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:12.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.722+0000 7f4e38b81640 1 --2- 172.21.15.29:0/4289026808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229387dd0 0x55d229392090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c0bdab0 tx=0x7f4e2c003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba7874c68a962157 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:12.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.722+0000 7f4e137fe640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:12:12.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.722+0000 7f4e137fe640 1 -- 172.21.15.29:0/4289026808 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c0c6550 con 0x55d229387dd0 2024-03-11T19:12:12.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.722+0000 7f4e137fe640 1 -- 172.21.15.29:0/4289026808 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4e2c0ba420 con 0x55d229387dd0 2024-03-11T19:12:12.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.722+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/4289026808 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4e280035f0 con 0x55d229387dd0 2024-03-11T19:12:12.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.723+0000 7f4e137fe640 1 -- 172.21.15.29:0/4289026808 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e2c0ba5e0 con 0x55d229387dd0 2024-03-11T19:12:12.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.723+0000 7f4e3d1b0740 10 client.25549 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:12.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.723+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/4289026808 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d2292ab8c0 con 0x55d229387dd0 2024-03-11T19:12:12.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.723+0000 7f4e137fe640 1 -- 172.21.15.29:0/4289026808 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4e2c0db050 con 0x55d229387dd0 2024-03-11T19:12:12.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.723+0000 7f4e137fe640 1 -- 172.21.15.29:0/4289026808 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4e2c0c6130 con 0x55d229387dd0 2024-03-11T19:12:12.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.723+0000 7f4e3d1b0740 10 client.25549 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:12.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.724+0000 7f4e3d1b0740 10 client.25549 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:12:12.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.724+0000 7f4e3d1b0740 20 client.25549 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:12.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.724+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/4289026808 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55d2293efe20 0x55d2293f2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.724+0000 7f4e3d1b0740 4 client.25549 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:12:12.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.724+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/4289026808 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f4e280035f0 con 0x55d2293efe20 2024-03-11T19:12:12.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.724+0000 7f4e33fff640 1 --2- 172.21.15.29:0/4289026808 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55d2293efe20 0x55d2293f2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.725+0000 7f4e33fff640 1 --2- 172.21.15.29:0/4289026808 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55d2293efe20 0x55d2293f2210 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=936c8421a4fdb8f2 server_cookie=c3a5787b3a7f1dba in_seq=0 out_seq=0 2024-03-11T19:12:12.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.725+0000 7f4e137fe640 10 client.25549 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:12:12.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.725+0000 7f4e137fe640 1 -- 172.21.15.29:0/4289026808 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f4e280035f0 con 0x55d2293efe20 2024-03-11T19:12:12.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.725+0000 7f4e137fe640 10 client.25549 handle_command_reply: tid=0 2024-03-11T19:12:12.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.725+0000 7f4e3d1b0740 1 client.25549 shutdown 2024-03-11T19:12:12.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.726+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/4289026808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229387dd0 msgr2=0x55d229392090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.726+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/4289026808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229387dd0 0x55d229392090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c0bdab0 tx=0x7f4e2c003f90 comp rx=0 tx=0).stop 2024-03-11T19:12:12.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.726+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/4289026808 shutdown_connections 2024-03-11T19:12:12.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.726+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/4289026808 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55d2293efe20 0x55d2293f2210 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.726+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/4289026808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d229387dd0 0x55d229392090 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.726+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/4289026808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d229382c60 0x55d229391b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.726+0000 7f4e3d1b0740 1 --2- 172.21.15.29:0/4289026808 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d229382270 0x55d229393540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.726+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/4289026808 >> 172.21.15.29:0/4289026808 conn(0x55d22937d7e0 msgr2=0x55d22939f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:12.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.726+0000 7f4e137fe640 0 client.25549 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:12:12.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.727+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/4289026808 shutdown_connections 2024-03-11T19:12:12.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.727+0000 7f4e3d1b0740 1 -- 172.21.15.29:0/4289026808 wait complete. 2024-03-11T19:12:12.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.727+0000 7f4e3d1b0740 20 client.25549 trim_cache size 0 max 16384 2024-03-11T19:12:12.732 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:12:12.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.728+0000 7f4e3a5e4640 1 -- 172.21.15.29:0/3386127073 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4e1c0a1a30 msgr2=0x7f4e1c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.728+0000 7f4e3a5e4640 1 --2- 172.21.15.29:0/3386127073 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4e1c0a1a30 0x7f4e1c0a3ef0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400bc80 tx=0x7f4e2400b410 comp rx=0 tx=0).stop 2024-03-11T19:12:12.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.728+0000 7f4e3a5e4640 1 -- 172.21.15.29:0/3386127073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e34101c30 msgr2=0x7f4e341a0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.728+0000 7f4e3a5e4640 1 --2- 172.21.15.29:0/3386127073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e34101c30 0x7f4e341a0870 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c02c640 tx=0x7f4e2c0329a0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.728+0000 7f4e3a5e4640 1 -- 172.21.15.29:0/3386127073 shutdown_connections 2024-03-11T19:12:12.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.728+0000 7f4e3a5e4640 1 --2- 172.21.15.29:0/3386127073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e34101c30 0x7f4e341a0870 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.728+0000 7f4e3a5e4640 1 --2- 172.21.15.29:0/3386127073 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4e1c0a1a30 0x7f4e1c0a3ef0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.728+0000 7f4e3a5e4640 1 --2- 172.21.15.29:0/3386127073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34101270 0x7f4e34199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.728+0000 7f4e3a5e4640 1 --2- 172.21.15.29:0/3386127073 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e34100080 0x7f4e341992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.728+0000 7f4e3a5e4640 1 -- 172.21.15.29:0/3386127073 >> 172.21.15.29:0/3386127073 conn(0x7f4e340fb610 msgr2=0x7f4e34102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:12.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.728+0000 7f4e3a5e4640 1 -- 172.21.15.29:0/3386127073 shutdown_connections 2024-03-11T19:12:12.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.729+0000 7f4e3a5e4640 1 -- 172.21.15.29:0/3386127073 wait complete. 2024-03-11T19:12:12.747 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:12:12.747 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:12:12.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.909+0000 7fa92e0a7640 1 -- 172.21.15.29:0/328118718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa928108990 msgr2=0x7fa928108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.909+0000 7fa92e0a7640 1 --2- 172.21.15.29:0/328118718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa928108990 0x7fa928108e10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa91c009900 tx=0x7fa91c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:12:12.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.909+0000 7fa92e0a7640 1 -- 172.21.15.29:0/328118718 shutdown_connections 2024-03-11T19:12:12.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.909+0000 7fa92e0a7640 1 --2- 172.21.15.29:0/328118718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa928109350 0x7fa92810fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.909+0000 7fa92e0a7640 1 --2- 172.21.15.29:0/328118718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa928108990 0x7fa928108e10 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.909+0000 7fa92e0a7640 1 --2- 172.21.15.29:0/328118718 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9281077a0 0x7fa928107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.909+0000 7fa92e0a7640 1 -- 172.21.15.29:0/328118718 >> 172.21.15.29:0/328118718 conn(0x7fa928076730 msgr2=0x7fa928078b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:12.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.909+0000 7fa92e0a7640 1 -- 172.21.15.29:0/328118718 shutdown_connections 2024-03-11T19:12:12.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.909+0000 7fa92e0a7640 1 -- 172.21.15.29:0/328118718 wait complete. 2024-03-11T19:12:12.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.910+0000 7fa92e0a7640 1 Processor -- start 2024-03-11T19:12:12.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.911+0000 7fa92e0a7640 1 -- start start 2024-03-11T19:12:12.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.911+0000 7fa92e0a7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9281077a0 0x7fa92819d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.911+0000 7fa92e0a7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa928108990 0x7fa92819dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.911+0000 7fa92e0a7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa928109350 0x7fa9281a4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.911+0000 7fa92e0a7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9281a52c0 con 0x7fa928109350 2024-03-11T19:12:12.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.911+0000 7fa92e0a7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9281a5430 con 0x7fa9281077a0 2024-03-11T19:12:12.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.911+0000 7fa92e0a7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9281a55f0 con 0x7fa928108990 2024-03-11T19:12:12.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.911+0000 7fa9277fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9281077a0 0x7fa92819d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.911+0000 7fa926ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa928108990 0x7fa92819dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.911+0000 7fa927fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa928109350 0x7fa9281a4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.911+0000 7fa9277fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9281077a0 0x7fa92819d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:48402/0 (socket says 172.21.15.29:48402) 2024-03-11T19:12:12.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.911+0000 7fa926ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa928108990 0x7fa92819dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41910/0 (socket says 172.21.15.29:41910) 2024-03-11T19:12:12.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.911+0000 7fa9277fe640 1 -- 172.21.15.29:0/2836724333 learned_addr learned my addr 172.21.15.29:0/2836724333 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.912+0000 7fa926ffd640 1 -- 172.21.15.29:0/2836724333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9281077a0 msgr2=0x7fa92819d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.912+0000 7fa926ffd640 1 --2- 172.21.15.29:0/2836724333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9281077a0 0x7fa92819d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.912+0000 7fa926ffd640 1 -- 172.21.15.29:0/2836724333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa928109350 msgr2=0x7fa9281a4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.912+0000 7fa926ffd640 1 --2- 172.21.15.29:0/2836724333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa928109350 0x7fa9281a4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.912+0000 7fa926ffd640 1 -- 172.21.15.29:0/2836724333 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa920009840 con 0x7fa928108990 2024-03-11T19:12:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.912+0000 7fa9277fe640 1 --2- 172.21.15.29:0/2836724333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9281077a0 0x7fa92819d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.912+0000 7fa926ffd640 1 --2- 172.21.15.29:0/2836724333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa928108990 0x7fa92819dd10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa91c02c170 tx=0x7fa91c004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.912+0000 7fa924ff9640 1 -- 172.21.15.29:0/2836724333 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c004440 con 0x7fa928108990 2024-03-11T19:12:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.912+0000 7fa924ff9640 1 -- 172.21.15.29:0/2836724333 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa91c003aa0 con 0x7fa928108990 2024-03-11T19:12:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.912+0000 7fa92e0a7640 1 -- 172.21.15.29:0/2836724333 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa91c009560 con 0x7fa928108990 2024-03-11T19:12:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.912+0000 7fa924ff9640 1 -- 172.21.15.29:0/2836724333 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c005390 con 0x7fa928108990 2024-03-11T19:12:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.912+0000 7fa92e0a7640 1 -- 172.21.15.29:0/2836724333 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9281a5bc0 con 0x7fa928108990 2024-03-11T19:12:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.915+0000 7fa924ff9640 1 -- 172.21.15.29:0/2836724333 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fa91c0054f0 con 0x7fa928108990 2024-03-11T19:12:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.916+0000 7fa930c7e740 1 Processor -- start 2024-03-11T19:12:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.916+0000 7fa930c7e740 1 -- start start 2024-03-11T19:12:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.916+0000 7fa924ff9640 1 --2- 172.21.15.29:0/2836724333 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa9100a1a30 0x7fa9100a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.916+0000 7fa924ff9640 1 -- 172.21.15.29:0/2836724333 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fa91c02de40 con 0x7fa928108990 2024-03-11T19:12:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.916+0000 7fa930c7e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b695270 0x55840b695690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.916+0000 7fa930c7e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b695c60 0x55840b69a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.916+0000 7fa927fff640 1 --2- 172.21.15.29:0/2836724333 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa9100a1a30 0x7fa9100a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.916+0000 7fa930c7e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b69add0 0x55840b69d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.916+0000 7fa930c7e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55840b651960 con 0x55840b69add0 2024-03-11T19:12:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.916+0000 7fa930c7e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55840b605310 con 0x55840b695270 2024-03-11T19:12:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.916+0000 7fa930c7e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55840b602840 con 0x55840b695c60 2024-03-11T19:12:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.916+0000 7fa9277fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b695270 0x55840b695690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.917+0000 7fa9277fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b695270 0x55840b695690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:48404/0 (socket says 172.21.15.29:48404) 2024-03-11T19:12:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.917+0000 7fa9277fe640 1 -- 172.21.15.29:0/3402314294 learned_addr learned my addr 172.21.15.29:0/3402314294 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.917+0000 7fa926ffd640 1 --2- 172.21.15.29:0/3402314294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b695c60 0x55840b69a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.917+0000 7fa927fff640 1 --2- 172.21.15.29:0/3402314294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b69add0 0x55840b69d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.917+0000 7fa9277fe640 1 -- 172.21.15.29:0/3402314294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b695c60 msgr2=0x55840b69a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.917+0000 7fa9277fe640 1 --2- 172.21.15.29:0/3402314294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b695c60 0x55840b69a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.917+0000 7fa9277fe640 1 -- 172.21.15.29:0/3402314294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b69add0 msgr2=0x55840b69d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.917+0000 7fa9277fe640 1 --2- 172.21.15.29:0/3402314294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b69add0 0x55840b69d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.917+0000 7fa9277fe640 1 -- 172.21.15.29:0/3402314294 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55840b5c0060 con 0x55840b695270 2024-03-11T19:12:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.917+0000 7fa927fff640 1 --2- 172.21.15.29:0/3402314294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b69add0 0x55840b69d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:12:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.917+0000 7fa927fff640 1 --2- 172.21.15.29:0/2836724333 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa9100a1a30 0x7fa9100a3ef0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa920005fd0 tx=0x7fa920011a00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.917+0000 7fa9277fe640 1 --2- 172.21.15.29:0/3402314294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b695270 0x55840b695690 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa9180115c0 tx=0x7fa918011a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16d00f5e900225de server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.918+0000 7fa90dffb640 1 -- 172.21.15.29:0/3402314294 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa918005020 con 0x55840b695270 2024-03-11T19:12:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.918+0000 7fa90dffb640 1 -- 172.21.15.29:0/3402314294 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa918005630 con 0x55840b695270 2024-03-11T19:12:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.918+0000 7fa90dffb640 1 -- 172.21.15.29:0/3402314294 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91800f8c0 con 0x55840b695270 2024-03-11T19:12:12.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.919+0000 7fa930c7e740 1 -- 172.21.15.29:0/3402314294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b695270 msgr2=0x55840b695690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.919+0000 7fa930c7e740 1 --2- 172.21.15.29:0/3402314294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b695270 0x55840b695690 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa9180115c0 tx=0x7fa918011a80 comp rx=0 tx=0).stop 2024-03-11T19:12:12.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.919+0000 7fa930c7e740 1 -- 172.21.15.29:0/3402314294 shutdown_connections 2024-03-11T19:12:12.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.919+0000 7fa930c7e740 1 --2- 172.21.15.29:0/3402314294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b69add0 0x55840b69d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.919+0000 7fa930c7e740 1 --2- 172.21.15.29:0/3402314294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b695c60 0x55840b69a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.919+0000 7fa930c7e740 1 --2- 172.21.15.29:0/3402314294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b695270 0x55840b695690 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.919+0000 7fa930c7e740 1 -- 172.21.15.29:0/3402314294 >> 172.21.15.29:0/3402314294 conn(0x55840b6907e0 msgr2=0x55840b690be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:12.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.919+0000 7fa930c7e740 1 -- 172.21.15.29:0/3402314294 shutdown_connections 2024-03-11T19:12:12.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.919+0000 7fa930c7e740 1 -- 172.21.15.29:0/3402314294 wait complete. 2024-03-11T19:12:12.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.919+0000 7fa930c7e740 1 -- start start 2024-03-11T19:12:12.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.920+0000 7fa930c7e740 1 Processor -- start 2024-03-11T19:12:12.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.920+0000 7fa930c7e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b695270 0x55840b69f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.920+0000 7fa930c7e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b695c60 0x55840b69f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.920+0000 7fa930c7e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b69add0 0x55840b6a69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.920+0000 7fa930c7e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55840b5fe020 con 0x55840b695270 2024-03-11T19:12:12.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.920+0000 7fa930c7e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55840b2d1430 con 0x55840b69add0 2024-03-11T19:12:12.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.920+0000 7fa930c7e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55840b28de00 con 0x55840b695c60 2024-03-11T19:12:12.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.920+0000 7fa9277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b695270 0x55840b69f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.920+0000 7fa927fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b69add0 0x55840b6a69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.920+0000 7fa9277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b695270 0x55840b69f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43824/0 (socket says 172.21.15.29:43824) 2024-03-11T19:12:12.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.920+0000 7fa9277fe640 1 -- 172.21.15.29:0/3850449085 learned_addr learned my addr 172.21.15.29:0/3850449085 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:12.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.920+0000 7fa930c7e740 10 client.0 _ll_register_callbacks cb 0x55840b5a5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:12.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.920+0000 7fa926ffd640 1 --2- 172.21.15.29:0/3850449085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b695c60 0x55840b69f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.921+0000 7fa9277fe640 1 -- 172.21.15.29:0/3850449085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b695c60 msgr2=0x55840b69f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:12.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.921+0000 7fa9277fe640 1 --2- 172.21.15.29:0/3850449085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b695c60 0x55840b69f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.921+0000 7fa9277fe640 1 -- 172.21.15.29:0/3850449085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b69add0 msgr2=0x55840b6a69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.921+0000 7fa9277fe640 1 --2- 172.21.15.29:0/3850449085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b69add0 0x55840b6a69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.921+0000 7fa9277fe640 1 -- 172.21.15.29:0/3850449085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa918009d80 con 0x55840b695270 2024-03-11T19:12:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.921+0000 7fa927fff640 1 --2- 172.21.15.29:0/3850449085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b69add0 0x55840b6a69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.921+0000 7fa9277fe640 1 --2- 172.21.15.29:0/3850449085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b695270 0x55840b69f4b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa918009380 tx=0x7fa918005230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dba4534c943fa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.921+0000 7fa906ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:12:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.921+0000 7fa906ffd640 1 -- 172.21.15.29:0/3850449085 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa918005420 con 0x55840b695270 2024-03-11T19:12:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.921+0000 7fa906ffd640 1 -- 172.21.15.29:0/3850449085 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fa91800f660 con 0x55840b695270 2024-03-11T19:12:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.921+0000 7fa930c7e740 1 -- 172.21.15.29:0/3850449085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55840b2fd7e0 con 0x55840b695270 2024-03-11T19:12:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.922+0000 7fa906ffd640 1 -- 172.21.15.29:0/3850449085 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa91800f820 con 0x55840b695270 2024-03-11T19:12:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.922+0000 7fa930c7e740 10 client.15996 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.922+0000 7fa930c7e740 1 -- 172.21.15.29:0/3850449085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55840b5fc6a0 con 0x55840b695270 2024-03-11T19:12:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.922+0000 7fa906ffd640 1 -- 172.21.15.29:0/3850449085 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55840b2fd7e0 con 0x55840b695270 2024-03-11T19:12:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.922+0000 7fa906ffd640 1 -- 172.21.15.29:0/3850449085 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa918023930 con 0x55840b695270 2024-03-11T19:12:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.922+0000 7fa930c7e740 10 client.15996 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.922+0000 7fa930c7e740 10 client.15996 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:12:12.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.922+0000 7fa930c7e740 20 client.15996 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:12.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.922+0000 7fa930c7e740 1 --2- 172.21.15.29:0/3850449085 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55840b6bc4b0 0x55840b6be8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.922+0000 7fa930c7e740 4 client.15996 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:12:12.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.922+0000 7fa930c7e740 1 -- 172.21.15.29:0/3850449085 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55840b33a720 con 0x55840b6bc4b0 2024-03-11T19:12:12.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.922+0000 7fa926ffd640 1 --2- 172.21.15.29:0/3850449085 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55840b6bc4b0 0x55840b6be8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.923+0000 7fa906ffd640 10 client.15996 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:12:12.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.923+0000 7fa926ffd640 1 --2- 172.21.15.29:0/3850449085 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55840b6bc4b0 0x55840b6be8c0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=81c8195d68d43c8c server_cookie=3c6afdceb6cf0c1c in_seq=0 out_seq=0 2024-03-11T19:12:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.923+0000 7fa906ffd640 1 -- 172.21.15.29:0/3850449085 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55840b33a720 con 0x55840b6bc4b0 2024-03-11T19:12:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.923+0000 7fa906ffd640 10 client.15996 handle_command_reply: tid=0 2024-03-11T19:12:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.924+0000 7fa930c7e740 1 client.15996 shutdown 2024-03-11T19:12:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.924+0000 7fa930c7e740 1 -- 172.21.15.29:0/3850449085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b695270 msgr2=0x55840b69f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.924+0000 7fa930c7e740 1 --2- 172.21.15.29:0/3850449085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b695270 0x55840b69f4b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa918009380 tx=0x7fa918005230 comp rx=0 tx=0).stop 2024-03-11T19:12:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.924+0000 7fa930c7e740 1 -- 172.21.15.29:0/3850449085 shutdown_connections 2024-03-11T19:12:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.924+0000 7fa930c7e740 1 --2- 172.21.15.29:0/3850449085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b69add0 0x55840b6a69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.924+0000 7fa930c7e740 1 --2- 172.21.15.29:0/3850449085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b695c60 0x55840b69f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.924+0000 7fa930c7e740 1 --2- 172.21.15.29:0/3850449085 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55840b6bc4b0 0x55840b6be8c0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.924+0000 7fa930c7e740 1 --2- 172.21.15.29:0/3850449085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b695270 0x55840b69f4b0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.924+0000 7fa930c7e740 1 -- 172.21.15.29:0/3850449085 >> 172.21.15.29:0/3850449085 conn(0x55840b6907e0 msgr2=0x55840b68fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.924+0000 7fa906ffd640 0 client.15996 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:12:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.924+0000 7fa930c7e740 1 -- 172.21.15.29:0/3850449085 shutdown_connections 2024-03-11T19:12:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.924+0000 7fa930c7e740 1 -- 172.21.15.29:0/3850449085 wait complete. 2024-03-11T19:12:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.924+0000 7fa930c7e740 20 client.15996 trim_cache size 0 max 16384 2024-03-11T19:12:12.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.946+0000 7fa930c7e740 1 Processor -- start 2024-03-11T19:12:12.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.946+0000 7fa930c7e740 1 -- start start 2024-03-11T19:12:12.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.946+0000 7fa930c7e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b695c60 0x55840b6971b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.946+0000 7fa930c7e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b69add0 0x55840b6976f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.946+0000 7fa930c7e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b6a6980 0x55840b68f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.946+0000 7fa930c7e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55840b583130 con 0x55840b69add0 2024-03-11T19:12:12.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.946+0000 7fa930c7e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55840b69d890 con 0x55840b6a6980 2024-03-11T19:12:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.946+0000 7fa930c7e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55840b696d20 con 0x55840b695c60 2024-03-11T19:12:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.946+0000 7fa926ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b69add0 0x55840b6976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.946+0000 7fa9277fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b695c60 0x55840b6971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.946+0000 7fa926ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b69add0 0x55840b6976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43828/0 (socket says 172.21.15.29:43828) 2024-03-11T19:12:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.947+0000 7fa927fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b6a6980 0x55840b68f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.947+0000 7fa926ffd640 1 -- 172.21.15.29:0/2432918702 learned_addr learned my addr 172.21.15.29:0/2432918702 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.947+0000 7fa926ffd640 1 -- 172.21.15.29:0/2432918702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b695c60 msgr2=0x55840b6971b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.947+0000 7fa926ffd640 1 --2- 172.21.15.29:0/2432918702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b695c60 0x55840b6971b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.947+0000 7fa926ffd640 1 -- 172.21.15.29:0/2432918702 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b6a6980 msgr2=0x55840b68f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.947+0000 7fa926ffd640 1 --2- 172.21.15.29:0/2432918702 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b6a6980 0x55840b68f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.947+0000 7fa926ffd640 1 -- 172.21.15.29:0/2432918702 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55840b598c20 con 0x55840b69add0 2024-03-11T19:12:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.947+0000 7fa9277fe640 1 --2- 172.21.15.29:0/2432918702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b695c60 0x55840b6971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.947+0000 7fa926ffd640 1 --2- 172.21.15.29:0/2432918702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b69add0 0x55840b6976f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa91c006010 tx=0x7fa91c0ed560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dabd6b0c28c789 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.948+0000 7fa90d7fa640 1 -- 172.21.15.29:0/2432918702 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c0c0c50 con 0x55840b69add0 2024-03-11T19:12:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.948+0000 7fa90d7fa640 1 -- 172.21.15.29:0/2432918702 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa91c0c52a0 con 0x55840b69add0 2024-03-11T19:12:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.948+0000 7fa90d7fa640 1 -- 172.21.15.29:0/2432918702 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c0b9170 con 0x55840b69add0 2024-03-11T19:12:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.949+0000 7fa930c7e740 1 -- 172.21.15.29:0/2432918702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b69add0 msgr2=0x55840b6976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.949+0000 7fa930c7e740 1 --2- 172.21.15.29:0/2432918702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b69add0 0x55840b6976f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa91c006010 tx=0x7fa91c0ed560 comp rx=0 tx=0).stop 2024-03-11T19:12:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.949+0000 7fa930c7e740 1 -- 172.21.15.29:0/2432918702 shutdown_connections 2024-03-11T19:12:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.949+0000 7fa930c7e740 1 --2- 172.21.15.29:0/2432918702 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b6a6980 0x55840b68f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.949+0000 7fa930c7e740 1 --2- 172.21.15.29:0/2432918702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b69add0 0x55840b6976f0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.949+0000 7fa930c7e740 1 --2- 172.21.15.29:0/2432918702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b695c60 0x55840b6971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.949+0000 7fa930c7e740 1 -- 172.21.15.29:0/2432918702 >> 172.21.15.29:0/2432918702 conn(0x55840b6907e0 msgr2=0x55840b68ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.949+0000 7fa930c7e740 1 -- 172.21.15.29:0/2432918702 shutdown_connections 2024-03-11T19:12:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.949+0000 7fa930c7e740 1 -- 172.21.15.29:0/2432918702 wait complete. 2024-03-11T19:12:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.949+0000 7fa930c7e740 1 -- start start 2024-03-11T19:12:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.950+0000 7fa930c7e740 1 Processor -- start 2024-03-11T19:12:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.950+0000 7fa930c7e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b695c60 0x55840b6a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.950+0000 7fa930c7e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b69add0 0x55840b6a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.950+0000 7fa930c7e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b6a6980 0x55840b6a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.950+0000 7fa930c7e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55840b6a5510 con 0x55840b695c60 2024-03-11T19:12:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.950+0000 7fa930c7e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55840b6a5680 con 0x55840b69add0 2024-03-11T19:12:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.950+0000 7fa930c7e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55840b7129a0 con 0x55840b6a6980 2024-03-11T19:12:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.950+0000 7fa930c7e740 10 client.0 _ll_register_callbacks cb 0x7fa918009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.950+0000 7fa9277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b695c60 0x55840b6a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.950+0000 7fa926ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b69add0 0x55840b6a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.950+0000 7fa9277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b695c60 0x55840b6a62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43834/0 (socket says 172.21.15.29:43834) 2024-03-11T19:12:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.950+0000 7fa9277fe640 1 -- 172.21.15.29:0/1549263475 learned_addr learned my addr 172.21.15.29:0/1549263475 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.950+0000 7fa927fff640 1 --2- 172.21.15.29:0/1549263475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b6a6980 0x55840b6a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.951+0000 7fa9277fe640 1 -- 172.21.15.29:0/1549263475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b6a6980 msgr2=0x55840b6a4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.951+0000 7fa9277fe640 1 --2- 172.21.15.29:0/1549263475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b6a6980 0x55840b6a4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.951+0000 7fa9277fe640 1 -- 172.21.15.29:0/1549263475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b69add0 msgr2=0x55840b6a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.951+0000 7fa9277fe640 1 --2- 172.21.15.29:0/1549263475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b69add0 0x55840b6a48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.951+0000 7fa9277fe640 1 -- 172.21.15.29:0/1549263475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55840b5fc6a0 con 0x55840b695c60 2024-03-11T19:12:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.951+0000 7fa9277fe640 1 --2- 172.21.15.29:0/1549263475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b695c60 0x55840b6a62c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa918009250 tx=0x7fa918011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a436a382ed79c0da server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.951+0000 7fa906ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:12:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.951+0000 7fa906ffd640 1 -- 172.21.15.29:0/1549263475 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa918004de0 con 0x55840b695c60 2024-03-11T19:12:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.951+0000 7fa906ffd640 1 -- 172.21.15.29:0/1549263475 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fa918000e40 con 0x55840b695c60 2024-03-11T19:12:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.951+0000 7fa930c7e740 1 -- 172.21.15.29:0/1549263475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa918005630 con 0x55840b695c60 2024-03-11T19:12:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.951+0000 7fa906ffd640 1 -- 172.21.15.29:0/1549263475 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa918004040 con 0x55840b695c60 2024-03-11T19:12:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.951+0000 7fa930c7e740 10 client.16002 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.951+0000 7fa930c7e740 1 -- 172.21.15.29:0/1549263475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7fa9281a5bc0 con 0x55840b695c60 2024-03-11T19:12:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.952+0000 7fa906ffd640 1 -- 172.21.15.29:0/1549263475 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa918023470 con 0x55840b695c60 2024-03-11T19:12:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.952+0000 7fa906ffd640 1 -- 172.21.15.29:0/1549263475 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa91800f660 con 0x55840b695c60 2024-03-11T19:12:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.952+0000 7fa930c7e740 10 client.16002 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.952+0000 7fa930c7e740 10 client.16002 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:12:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.952+0000 7fa930c7e740 20 client.16002 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.952+0000 7fa930c7e740 1 --2- 172.21.15.29:0/1549263475 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55840b714020 0x55840b716440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.952+0000 7fa930c7e740 4 client.16002 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:12:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.952+0000 7fa930c7e740 1 -- 172.21.15.29:0/1549263475 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7fa918005630 con 0x55840b714020 2024-03-11T19:12:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.952+0000 7fa926ffd640 1 --2- 172.21.15.29:0/1549263475 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55840b714020 0x55840b716440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.953+0000 7fa906ffd640 10 client.16002 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:12:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.953+0000 7fa926ffd640 1 --2- 172.21.15.29:0/1549263475 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55840b714020 0x55840b716440 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bb8e953d2d9869db server_cookie=768fd635e0334f78 in_seq=0 out_seq=0 2024-03-11T19:12:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.953+0000 7fa906ffd640 1 -- 172.21.15.29:0/1549263475 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fa918005630 con 0x55840b714020 2024-03-11T19:12:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.953+0000 7fa906ffd640 10 client.16002 handle_command_reply: tid=0 2024-03-11T19:12:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.953+0000 7fa930c7e740 1 client.16002 shutdown 2024-03-11T19:12:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.953+0000 7fa930c7e740 1 -- 172.21.15.29:0/1549263475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b695c60 msgr2=0x55840b6a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.953+0000 7fa930c7e740 1 --2- 172.21.15.29:0/1549263475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b695c60 0x55840b6a62c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa918009250 tx=0x7fa918011a80 comp rx=0 tx=0).stop 2024-03-11T19:12:12.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.953+0000 7fa930c7e740 1 -- 172.21.15.29:0/1549263475 shutdown_connections 2024-03-11T19:12:12.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.953+0000 7fa930c7e740 1 --2- 172.21.15.29:0/1549263475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55840b6a6980 0x55840b6a4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.953+0000 7fa930c7e740 1 --2- 172.21.15.29:0/1549263475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55840b69add0 0x55840b6a48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.953+0000 7fa930c7e740 1 --2- 172.21.15.29:0/1549263475 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55840b714020 0x55840b716440 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.953+0000 7fa906ffd640 0 client.16002 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:12:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.953+0000 7fa930c7e740 1 --2- 172.21.15.29:0/1549263475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55840b695c60 0x55840b6a62c0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.953+0000 7fa930c7e740 1 -- 172.21.15.29:0/1549263475 >> 172.21.15.29:0/1549263475 conn(0x55840b6907e0 msgr2=0x55840b6971b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.953+0000 7fa930c7e740 1 -- 172.21.15.29:0/1549263475 shutdown_connections 2024-03-11T19:12:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.953+0000 7fa930c7e740 1 -- 172.21.15.29:0/1549263475 wait complete. 2024-03-11T19:12:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.954+0000 7fa930c7e740 20 client.16002 trim_cache size 0 max 16384 2024-03-11T19:12:12.960 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:12:12.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.954+0000 7fa92e0a7640 1 -- 172.21.15.29:0/2836724333 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa9100a1a30 msgr2=0x7fa9100a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.954+0000 7fa92e0a7640 1 --2- 172.21.15.29:0/2836724333 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa9100a1a30 0x7fa9100a3ef0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa920005fd0 tx=0x7fa920011a00 comp rx=0 tx=0).stop 2024-03-11T19:12:12.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.954+0000 7fa92e0a7640 1 -- 172.21.15.29:0/2836724333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa928108990 msgr2=0x7fa92819dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:12.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.954+0000 7fa92e0a7640 1 --2- 172.21.15.29:0/2836724333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa928108990 0x7fa92819dd10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa91c02c170 tx=0x7fa91c004040 comp rx=0 tx=0).stop 2024-03-11T19:12:12.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.954+0000 7fa92e0a7640 1 -- 172.21.15.29:0/2836724333 shutdown_connections 2024-03-11T19:12:12.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.954+0000 7fa92e0a7640 1 --2- 172.21.15.29:0/2836724333 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa9100a1a30 0x7fa9100a3ef0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.954+0000 7fa92e0a7640 1 --2- 172.21.15.29:0/2836724333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa928109350 0x7fa9281a4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.954+0000 7fa92e0a7640 1 --2- 172.21.15.29:0/2836724333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa928108990 0x7fa92819dd10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.954+0000 7fa92e0a7640 1 --2- 172.21.15.29:0/2836724333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9281077a0 0x7fa92819d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:12.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.954+0000 7fa92e0a7640 1 -- 172.21.15.29:0/2836724333 >> 172.21.15.29:0/2836724333 conn(0x7fa928076730 msgr2=0x7fa92810bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:12.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.954+0000 7fa92e0a7640 1 -- 172.21.15.29:0/2836724333 shutdown_connections 2024-03-11T19:12:12.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:12.955+0000 7fa92e0a7640 1 -- 172.21.15.29:0/2836724333 wait complete. 2024-03-11T19:12:12.975 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:12:12.975 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:12:13.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.140+0000 7ff61cc1c640 1 -- 172.21.15.29:0/676127493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff618102360 msgr2=0x7ff6181047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.140+0000 7ff61cc1c640 1 --2- 172.21.15.29:0/676127493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff618102360 0x7ff6181047d0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ff600009900 tx=0x7ff60002bc30 comp rx=0 tx=0).stop 2024-03-11T19:12:13.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.140+0000 7ff61cc1c640 1 -- 172.21.15.29:0/676127493 shutdown_connections 2024-03-11T19:12:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.140+0000 7ff61cc1c640 1 --2- 172.21.15.29:0/676127493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff618104d10 0x7ff618105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.140+0000 7ff61cc1c640 1 --2- 172.21.15.29:0/676127493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff618102360 0x7ff6181047d0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.140+0000 7ff61cc1c640 1 --2- 172.21.15.29:0/676127493 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6180ff9f0 0x7ff618101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.140+0000 7ff61cc1c640 1 -- 172.21.15.29:0/676127493 >> 172.21.15.29:0/676127493 conn(0x7ff6180fb610 msgr2=0x7ff6180fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.140+0000 7ff61cc1c640 1 -- 172.21.15.29:0/676127493 shutdown_connections 2024-03-11T19:12:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.140+0000 7ff61cc1c640 1 -- 172.21.15.29:0/676127493 wait complete. 2024-03-11T19:12:13.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.141+0000 7ff61cc1c640 1 Processor -- start 2024-03-11T19:12:13.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.141+0000 7ff61cc1c640 1 -- start start 2024-03-11T19:12:13.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.141+0000 7ff61cc1c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6180ff9f0 0x7ff61819d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.141+0000 7ff61cc1c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff618102360 0x7ff61819dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.141+0000 7ff61cc1c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff618104d10 0x7ff6181a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.142+0000 7ff61cc1c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6181a52f0 con 0x7ff6180ff9f0 2024-03-11T19:12:13.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.142+0000 7ff61659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6180ff9f0 0x7ff61819d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.142+0000 7ff61659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6180ff9f0 0x7ff61819d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43854/0 (socket says 172.21.15.29:43854) 2024-03-11T19:12:13.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.142+0000 7ff616d9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff618104d10 0x7ff6181a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.142+0000 7ff60ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff618102360 0x7ff61819dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.142+0000 7ff616d9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff618104d10 0x7ff6181a4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:48454/0 (socket says 172.21.15.29:48454) 2024-03-11T19:12:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.142+0000 7ff60ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff618102360 0x7ff61819dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41960/0 (socket says 172.21.15.29:41960) 2024-03-11T19:12:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.142+0000 7ff616d9d640 1 -- 172.21.15.29:0/3563863239 learned_addr learned my addr 172.21.15.29:0/3563863239 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.142+0000 7ff61cc1c640 1 -- 172.21.15.29:0/3563863239 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6181a5460 con 0x7ff618104d10 2024-03-11T19:12:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.142+0000 7ff61cc1c640 1 -- 172.21.15.29:0/3563863239 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6181a5620 con 0x7ff618102360 2024-03-11T19:12:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.142+0000 7ff616d9d640 1 -- 172.21.15.29:0/3563863239 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff618102360 msgr2=0x7ff61819dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.142+0000 7ff616d9d640 1 --2- 172.21.15.29:0/3563863239 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff618102360 0x7ff61819dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.142+0000 7ff616d9d640 1 -- 172.21.15.29:0/3563863239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6180ff9f0 msgr2=0x7ff61819d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.142+0000 7ff616d9d640 1 --2- 172.21.15.29:0/3563863239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6180ff9f0 0x7ff61819d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.142+0000 7ff616d9d640 1 -- 172.21.15.29:0/3563863239 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff600009560 con 0x7ff618104d10 2024-03-11T19:12:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.142+0000 7ff616d9d640 1 --2- 172.21.15.29:0/3563863239 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff618104d10 0x7ff6181a4db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff61000eaa0 tx=0x7ff61000eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.143+0000 7ff60f7fe640 1 -- 172.21.15.29:0/3563863239 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610004d20 con 0x7ff618104d10 2024-03-11T19:12:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.143+0000 7ff60f7fe640 1 -- 172.21.15.29:0/3563863239 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff610004e80 con 0x7ff618104d10 2024-03-11T19:12:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.143+0000 7ff61cc1c640 1 -- 172.21.15.29:0/3563863239 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6181a5900 con 0x7ff618104d10 2024-03-11T19:12:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.143+0000 7ff60f7fe640 1 -- 172.21.15.29:0/3563863239 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6100116d0 con 0x7ff618104d10 2024-03-11T19:12:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.143+0000 7ff61cc1c640 1 -- 172.21.15.29:0/3563863239 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6181a5e00 con 0x7ff618104d10 2024-03-11T19:12:13.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.146+0000 7ff60f7fe640 1 -- 172.21.15.29:0/3563863239 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff610005020 con 0x7ff618104d10 2024-03-11T19:12:13.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.146+0000 7ff61f7f3740 1 Processor -- start 2024-03-11T19:12:13.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.146+0000 7ff61f7f3740 1 -- start start 2024-03-11T19:12:13.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.147+0000 7ff61f7f3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d0155270 0x5557d0155690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.147+0000 7ff61f7f3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155c60 0x5557d015a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.147+0000 7ff61f7f3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d015add0 0x5557d015d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.147+0000 7ff61f7f3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557d0111960 con 0x5557d0155c60 2024-03-11T19:12:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.147+0000 7ff61f7f3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557d00c5310 con 0x5557d015add0 2024-03-11T19:12:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.147+0000 7ff61f7f3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557d00c2840 con 0x5557d0155270 2024-03-11T19:12:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.147+0000 7ff60ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d0155270 0x5557d0155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.147+0000 7ff61659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155c60 0x5557d015a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.147+0000 7ff60ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d0155270 0x5557d0155690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41976/0 (socket says 172.21.15.29:41976) 2024-03-11T19:12:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.147+0000 7ff60ffff640 1 -- 172.21.15.29:0/1860622628 learned_addr learned my addr 172.21.15.29:0/1860622628 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.147+0000 7ff60ffff640 1 --2- 172.21.15.29:0/1860622628 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d015add0 0x5557d015d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.147+0000 7ff60ffff640 1 -- 172.21.15.29:0/1860622628 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d015add0 msgr2=0x5557d015d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.147+0000 7ff60ffff640 1 --2- 172.21.15.29:0/1860622628 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d015add0 0x5557d015d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.147+0000 7ff60ffff640 1 -- 172.21.15.29:0/1860622628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155c60 msgr2=0x5557d015a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.147+0000 7ff60ffff640 1 --2- 172.21.15.29:0/1860622628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155c60 0x5557d015a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.147+0000 7ff60ffff640 1 -- 172.21.15.29:0/1860622628 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557d0080060 con 0x5557d0155270 2024-03-11T19:12:13.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.147+0000 7ff60f7fe640 1 --2- 172.21.15.29:0/3563863239 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff5f80a1b00 0x7ff5f80a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.148+0000 7ff60ffff640 1 --2- 172.21.15.29:0/1860622628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d0155270 0x5557d0155690 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff600009230 tx=0x7ff60002c150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dee056bab93e3829 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:13.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.148+0000 7ff61659c640 1 --2- 172.21.15.29:0/3563863239 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff5f80a1b00 0x7ff5f80a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.148+0000 7ff60f7fe640 1 -- 172.21.15.29:0/3563863239 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff6100cd440 con 0x7ff618104d10 2024-03-11T19:12:13.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.148+0000 7ff60cff9640 1 -- 172.21.15.29:0/1860622628 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600004aa0 con 0x5557d0155270 2024-03-11T19:12:13.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.148+0000 7ff60cff9640 1 -- 172.21.15.29:0/1860622628 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff6000044e0 con 0x5557d0155270 2024-03-11T19:12:13.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.148+0000 7ff60cff9640 1 -- 172.21.15.29:0/1860622628 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600004dc0 con 0x5557d0155270 2024-03-11T19:12:13.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.149+0000 7ff61659c640 1 --2- 172.21.15.29:0/3563863239 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff5f80a1b00 0x7ff5f80a3fc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x5557d0156060 tx=0x7ff608009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:13.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.149+0000 7ff61f7f3740 1 -- 172.21.15.29:0/1860622628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d0155270 msgr2=0x5557d0155690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.149+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/1860622628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d0155270 0x5557d0155690 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff600009230 tx=0x7ff60002c150 comp rx=0 tx=0).stop 2024-03-11T19:12:13.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.149+0000 7ff61f7f3740 1 -- 172.21.15.29:0/1860622628 shutdown_connections 2024-03-11T19:12:13.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.149+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/1860622628 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d015add0 0x5557d015d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.149+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/1860622628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155c60 0x5557d015a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.149+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/1860622628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d0155270 0x5557d0155690 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.149+0000 7ff61f7f3740 1 -- 172.21.15.29:0/1860622628 >> 172.21.15.29:0/1860622628 conn(0x5557d01507e0 msgr2=0x5557d0150be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:13.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.149+0000 7ff61f7f3740 1 -- 172.21.15.29:0/1860622628 shutdown_connections 2024-03-11T19:12:13.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.149+0000 7ff61f7f3740 1 -- 172.21.15.29:0/1860622628 wait complete. 2024-03-11T19:12:13.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.149+0000 7ff61f7f3740 1 -- start start 2024-03-11T19:12:13.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.149+0000 7ff61f7f3740 1 Processor -- start 2024-03-11T19:12:13.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.150+0000 7ff61f7f3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155270 0x5557d015f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.150+0000 7ff61f7f3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d0155c60 0x5557d015f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.150+0000 7ff61f7f3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d015add0 0x5557d01669a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.150+0000 7ff61659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155270 0x5557d015f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.150+0000 7ff61659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155270 0x5557d015f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43864/0 (socket says 172.21.15.29:43864) 2024-03-11T19:12:13.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.150+0000 7ff61659c640 1 -- 172.21.15.29:0/2199178309 learned_addr learned my addr 172.21.15.29:0/2199178309 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:13.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.150+0000 7ff61f7f3740 1 -- 172.21.15.29:0/2199178309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557d00be020 con 0x5557d0155270 2024-03-11T19:12:13.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.150+0000 7ff61f7f3740 1 -- 172.21.15.29:0/2199178309 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557cfd91430 con 0x5557d015add0 2024-03-11T19:12:13.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.150+0000 7ff61f7f3740 1 -- 172.21.15.29:0/2199178309 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557cfd4de00 con 0x5557d0155c60 2024-03-11T19:12:13.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.150+0000 7ff60ffff640 1 --2- 172.21.15.29:0/2199178309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d0155c60 0x5557d015f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.150+0000 7ff616d9d640 1 --2- 172.21.15.29:0/2199178309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d015add0 0x5557d01669a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.151+0000 7ff61f7f3740 10 client.0 _ll_register_callbacks cb 0x5557d0065cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:13.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.151+0000 7ff61659c640 1 -- 172.21.15.29:0/2199178309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d0155c60 msgr2=0x5557d015f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.151+0000 7ff61659c640 1 --2- 172.21.15.29:0/2199178309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d0155c60 0x5557d015f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.151+0000 7ff61659c640 1 -- 172.21.15.29:0/2199178309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d015add0 msgr2=0x5557d01669a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.151+0000 7ff61659c640 1 --2- 172.21.15.29:0/2199178309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d015add0 0x5557d01669a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.151+0000 7ff61659c640 1 -- 172.21.15.29:0/2199178309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff618105c80 con 0x5557d0155270 2024-03-11T19:12:13.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.151+0000 7ff616d9d640 1 --2- 172.21.15.29:0/2199178309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d015add0 0x5557d01669a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:13.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.151+0000 7ff61f7f3740 1 -- 172.21.15.29:0/2199178309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5557cfdbd7e0 con 0x5557d0155270 2024-03-11T19:12:13.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.151+0000 7ff61659c640 1 --2- 172.21.15.29:0/2199178309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155270 0x5557d015f420 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ff6080149c0 tx=0x7ff6080047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e4103da43c4afd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:13.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.151+0000 7ff605ffb640 10 client.16014 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:12:13.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.151+0000 7ff605ffb640 1 -- 172.21.15.29:0/2199178309 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608021600 con 0x5557d0155270 2024-03-11T19:12:13.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.152+0000 7ff605ffb640 1 -- 172.21.15.29:0/2199178309 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff60801c040 con 0x5557d0155270 2024-03-11T19:12:13.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.152+0000 7ff605ffb640 1 -- 172.21.15.29:0/2199178309 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff608021760 con 0x5557d0155270 2024-03-11T19:12:13.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.152+0000 7ff61f7f3740 10 client.16014 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:13.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.152+0000 7ff61f7f3740 1 -- 172.21.15.29:0/2199178309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x5557d00bc6a0 con 0x5557d0155270 2024-03-11T19:12:13.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.152+0000 7ff605ffb640 1 -- 172.21.15.29:0/2199178309 <== mon.0 v2:172.21.15.29:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff6080218c0 con 0x5557d0155270 2024-03-11T19:12:13.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.152+0000 7ff61f7f3740 10 client.16014 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:13.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.152+0000 7ff61f7f3740 10 client.16014 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:12:13.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.152+0000 7ff61f7f3740 20 client.16014 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:13.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.152+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/2199178309 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5557d016b3c0 0x5557d016d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.153+0000 7ff61f7f3740 4 client.16014 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:12:13.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.153+0000 7ff61f7f3740 1 -- 172.21.15.29:0/2199178309 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5557cfdfa720 con 0x5557d016b3c0 2024-03-11T19:12:13.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:12 smithi029 ceph-mon[32439]: pgmap v423: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 168 MiB/s wr, 93 op/s 2024-03-11T19:12:13.159 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:12 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1434186839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:12:13.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.153+0000 7ff60ffff640 1 --2- 172.21.15.29:0/2199178309 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5557d016b3c0 0x5557d016d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.153+0000 7ff60ffff640 1 --2- 172.21.15.29:0/2199178309 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5557d016b3c0 0x5557d016d7f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a270486be986a562 server_cookie=e6b8736d1465e34e in_seq=0 out_seq=0 2024-03-11T19:12:13.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.153+0000 7ff605ffb640 10 client.16014 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:12:13.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.154+0000 7ff605ffb640 1 -- 172.21.15.29:0/2199178309 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5557cfdfa720 con 0x5557d016b3c0 2024-03-11T19:12:13.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.154+0000 7ff605ffb640 10 client.16014 handle_command_reply: tid=0 2024-03-11T19:12:13.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.154+0000 7ff61f7f3740 1 client.16014 shutdown 2024-03-11T19:12:13.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.154+0000 7ff61f7f3740 1 -- 172.21.15.29:0/2199178309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155270 msgr2=0x5557d015f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.154+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/2199178309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155270 0x5557d015f420 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ff6080149c0 tx=0x7ff6080047d0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.154+0000 7ff61f7f3740 1 -- 172.21.15.29:0/2199178309 shutdown_connections 2024-03-11T19:12:13.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.154+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/2199178309 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5557d016b3c0 0x5557d016d7f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.154+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/2199178309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d015add0 0x5557d01669a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.154+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/2199178309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d0155c60 0x5557d015f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.154+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/2199178309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155270 0x5557d015f420 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.154+0000 7ff605ffb640 0 client.16014 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:12:13.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.154+0000 7ff61f7f3740 1 -- 172.21.15.29:0/2199178309 >> 172.21.15.29:0/2199178309 conn(0x5557d01507e0 msgr2=0x5557d014fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:13.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.154+0000 7ff61f7f3740 1 -- 172.21.15.29:0/2199178309 shutdown_connections 2024-03-11T19:12:13.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.154+0000 7ff61f7f3740 1 -- 172.21.15.29:0/2199178309 wait complete. 2024-03-11T19:12:13.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.154+0000 7ff61f7f3740 20 client.16014 trim_cache size 0 max 16384 2024-03-11T19:12:13.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.174+0000 7ff61f7f3740 1 Processor -- start 2024-03-11T19:12:13.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.174+0000 7ff61f7f3740 1 -- start start 2024-03-11T19:12:13.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.174+0000 7ff61f7f3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d0155270 0x5557d0157160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.174+0000 7ff61f7f3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155c60 0x5557d01576a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.175+0000 7ff61f7f3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d015add0 0x5557d0166960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.175+0000 7ff61f7f3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557d0043130 con 0x5557d0155c60 2024-03-11T19:12:13.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.175+0000 7ff61f7f3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557d0151780 con 0x5557d0155270 2024-03-11T19:12:13.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.175+0000 7ff61f7f3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557d014c0b0 con 0x5557d015add0 2024-03-11T19:12:13.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.175+0000 7ff60ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155c60 0x5557d01576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.175+0000 7ff61659c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d0155270 0x5557d0157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.175+0000 7ff60ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155c60 0x5557d01576a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43872/0 (socket says 172.21.15.29:43872) 2024-03-11T19:12:13.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.175+0000 7ff60ffff640 1 -- 172.21.15.29:0/338870417 learned_addr learned my addr 172.21.15.29:0/338870417 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:13.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.175+0000 7ff616d9d640 1 --2- 172.21.15.29:0/338870417 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d015add0 0x5557d0166960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.175+0000 7ff60ffff640 1 -- 172.21.15.29:0/338870417 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d015add0 msgr2=0x5557d0166960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.175+0000 7ff60ffff640 1 --2- 172.21.15.29:0/338870417 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d015add0 0x5557d0166960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.175+0000 7ff60ffff640 1 -- 172.21.15.29:0/338870417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d0155270 msgr2=0x5557d0157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.175+0000 7ff60ffff640 1 --2- 172.21.15.29:0/338870417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d0155270 0x5557d0157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.175+0000 7ff60ffff640 1 -- 172.21.15.29:0/338870417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557d0058c20 con 0x5557d0155c60 2024-03-11T19:12:13.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.175+0000 7ff60ffff640 1 --2- 172.21.15.29:0/338870417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155c60 0x5557d01576a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff600009230 tx=0x7ff60002ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf76457d7de6594 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:13.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.176+0000 7ff607fff640 1 -- 172.21.15.29:0/338870417 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60005b070 con 0x5557d0155c60 2024-03-11T19:12:13.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.176+0000 7ff607fff640 1 -- 172.21.15.29:0/338870417 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff60002cad0 con 0x5557d0155c60 2024-03-11T19:12:13.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.176+0000 7ff607fff640 1 -- 172.21.15.29:0/338870417 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600053040 con 0x5557d0155c60 2024-03-11T19:12:13.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.177+0000 7ff61f7f3740 1 -- 172.21.15.29:0/338870417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155c60 msgr2=0x5557d01576a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.177+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/338870417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155c60 0x5557d01576a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff600009230 tx=0x7ff60002ca60 comp rx=0 tx=0).stop 2024-03-11T19:12:13.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.177+0000 7ff61f7f3740 1 -- 172.21.15.29:0/338870417 shutdown_connections 2024-03-11T19:12:13.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.177+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/338870417 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d015add0 0x5557d0166960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.177+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/338870417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155c60 0x5557d01576a0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.177+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/338870417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d0155270 0x5557d0157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.177+0000 7ff61f7f3740 1 -- 172.21.15.29:0/338870417 >> 172.21.15.29:0/338870417 conn(0x5557d01507e0 msgr2=0x5557d0172160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:13.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.177+0000 7ff61f7f3740 1 -- 172.21.15.29:0/338870417 shutdown_connections 2024-03-11T19:12:13.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.177+0000 7ff61f7f3740 1 -- 172.21.15.29:0/338870417 wait complete. 2024-03-11T19:12:13.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.178+0000 7ff61f7f3740 1 -- start start 2024-03-11T19:12:13.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.178+0000 7ff61f7f3740 1 Processor -- start 2024-03-11T19:12:13.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.178+0000 7ff61f7f3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155270 0x5557d0166990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.178+0000 7ff61f7f3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d0155c60 0x5557d0164fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.178+0000 7ff61f7f3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d015add0 0x5557d01655b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.178+0000 7ff61f7f3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557d0165cb0 con 0x5557d0155270 2024-03-11T19:12:13.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.178+0000 7ff61f7f3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557d0165e20 con 0x5557d015add0 2024-03-11T19:12:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.178+0000 7ff61f7f3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557d01d2910 con 0x5557d0155c60 2024-03-11T19:12:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.178+0000 7ff61f7f3740 10 client.0 _ll_register_callbacks cb 0x7ff60002c9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.178+0000 7ff61659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155270 0x5557d0166990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.178+0000 7ff61659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155270 0x5557d0166990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43878/0 (socket says 172.21.15.29:43878) 2024-03-11T19:12:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.178+0000 7ff61659c640 1 -- 172.21.15.29:0/2284057271 learned_addr learned my addr 172.21.15.29:0/2284057271 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.178+0000 7ff616d9d640 1 --2- 172.21.15.29:0/2284057271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d015add0 0x5557d01655b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.179+0000 7ff60ffff640 1 --2- 172.21.15.29:0/2284057271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d0155c60 0x5557d0164fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.179+0000 7ff61659c640 1 -- 172.21.15.29:0/2284057271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d0155c60 msgr2=0x5557d0164fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.179+0000 7ff61659c640 1 --2- 172.21.15.29:0/2284057271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d0155c60 0x5557d0164fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.179+0000 7ff61659c640 1 -- 172.21.15.29:0/2284057271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d015add0 msgr2=0x5557d01655b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.179+0000 7ff61659c640 1 --2- 172.21.15.29:0/2284057271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d015add0 0x5557d01655b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.179+0000 7ff61659c640 1 -- 172.21.15.29:0/2284057271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5557d0080060 con 0x5557d0155270 2024-03-11T19:12:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.179+0000 7ff61659c640 1 --2- 172.21.15.29:0/2284057271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155270 0x5557d0166990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff6080149c0 tx=0x7ff608019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb5dc3edc4608c3e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.179+0000 7ff605ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:12:13.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.180+0000 7ff605ffb640 1 -- 172.21.15.29:0/2284057271 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60802a850 con 0x5557d0155270 2024-03-11T19:12:13.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.180+0000 7ff605ffb640 1 -- 172.21.15.29:0/2284057271 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff60802a9b0 con 0x5557d0155270 2024-03-11T19:12:13.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.180+0000 7ff61f7f3740 1 -- 172.21.15.29:0/2284057271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff6000044e0 con 0x5557d0155270 2024-03-11T19:12:13.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.180+0000 7ff605ffb640 1 -- 172.21.15.29:0/2284057271 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff60802ab70 con 0x5557d0155270 2024-03-11T19:12:13.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.180+0000 7ff61f7f3740 10 client.16020 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.180+0000 7ff61f7f3740 1 -- 172.21.15.29:0/2284057271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5557d007e8c0 con 0x5557d0155270 2024-03-11T19:12:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.181+0000 7ff605ffb640 1 -- 172.21.15.29:0/2284057271 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff6000044e0 con 0x5557d0155270 2024-03-11T19:12:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.181+0000 7ff605ffb640 1 -- 172.21.15.29:0/2284057271 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff608047920 con 0x5557d0155270 2024-03-11T19:12:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.181+0000 7ff61f7f3740 10 client.16020 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.181+0000 7ff61f7f3740 10 client.16020 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:12:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.181+0000 7ff61f7f3740 20 client.16020 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.181+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/2284057271 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5557d01d41b0 0x5557d01d65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.181+0000 7ff61f7f3740 4 client.16020 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:12:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.181+0000 7ff61f7f3740 1 -- 172.21.15.29:0/2284057271 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5557cfe49890 con 0x5557d01d41b0 2024-03-11T19:12:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.181+0000 7ff60ffff640 1 --2- 172.21.15.29:0/2284057271 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5557d01d41b0 0x5557d01d65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.182+0000 7ff605ffb640 10 client.16020 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:12:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.182+0000 7ff60ffff640 1 --2- 172.21.15.29:0/2284057271 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5557d01d41b0 0x5557d01d65d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3288bf50956e35e7 server_cookie=2519b3e8e07ea40f in_seq=0 out_seq=0 2024-03-11T19:12:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.182+0000 7ff605ffb640 1 -- 172.21.15.29:0/2284057271 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5557cfe49890 con 0x5557d01d41b0 2024-03-11T19:12:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.182+0000 7ff605ffb640 10 client.16020 handle_command_reply: tid=0 2024-03-11T19:12:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.182+0000 7ff61f7f3740 1 client.16020 shutdown 2024-03-11T19:12:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.182+0000 7ff61f7f3740 1 -- 172.21.15.29:0/2284057271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155270 msgr2=0x5557d0166990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.182+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/2284057271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155270 0x5557d0166990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff6080149c0 tx=0x7ff608019000 comp rx=0 tx=0).stop 2024-03-11T19:12:13.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.182+0000 7ff61f7f3740 1 -- 172.21.15.29:0/2284057271 shutdown_connections 2024-03-11T19:12:13.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.182+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/2284057271 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5557d01d41b0 0x5557d01d65d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.182+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/2284057271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557d015add0 0x5557d01655b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.182+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/2284057271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557d0155c60 0x5557d0164fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.182+0000 7ff61f7f3740 1 --2- 172.21.15.29:0/2284057271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557d0155270 0x5557d0166990 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.182+0000 7ff61f7f3740 1 -- 172.21.15.29:0/2284057271 >> 172.21.15.29:0/2284057271 conn(0x5557d01507e0 msgr2=0x5557d0157160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:13.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.183+0000 7ff605ffb640 0 client.16020 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:12:13.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.183+0000 7ff61f7f3740 1 -- 172.21.15.29:0/2284057271 shutdown_connections 2024-03-11T19:12:13.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.183+0000 7ff61f7f3740 1 -- 172.21.15.29:0/2284057271 wait complete. 2024-03-11T19:12:13.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.183+0000 7ff61f7f3740 20 client.16020 trim_cache size 0 max 16384 2024-03-11T19:12:13.188 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:12:13.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.184+0000 7ff61cc1c640 1 -- 172.21.15.29:0/3563863239 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff5f80a1b00 msgr2=0x7ff5f80a3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.184+0000 7ff61cc1c640 1 --2- 172.21.15.29:0/3563863239 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff5f80a1b00 0x7ff5f80a3fc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x5557d0156060 tx=0x7ff608009110 comp rx=0 tx=0).stop 2024-03-11T19:12:13.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.184+0000 7ff61cc1c640 1 -- 172.21.15.29:0/3563863239 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff618104d10 msgr2=0x7ff6181a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.184+0000 7ff61cc1c640 1 --2- 172.21.15.29:0/3563863239 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff618104d10 0x7ff6181a4db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff61000eaa0 tx=0x7ff61000eea0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.184+0000 7ff61cc1c640 1 -- 172.21.15.29:0/3563863239 shutdown_connections 2024-03-11T19:12:13.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.184+0000 7ff61cc1c640 1 --2- 172.21.15.29:0/3563863239 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff618104d10 0x7ff6181a4db0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.184+0000 7ff61cc1c640 1 --2- 172.21.15.29:0/3563863239 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff618102360 0x7ff61819dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.184+0000 7ff61cc1c640 1 --2- 172.21.15.29:0/3563863239 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff5f80a1b00 0x7ff5f80a3fc0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.184+0000 7ff61cc1c640 1 --2- 172.21.15.29:0/3563863239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6180ff9f0 0x7ff61819d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.184+0000 7ff61cc1c640 1 -- 172.21.15.29:0/3563863239 >> 172.21.15.29:0/3563863239 conn(0x7ff6180fb610 msgr2=0x7ff6181004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:13.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.184+0000 7ff61cc1c640 1 -- 172.21.15.29:0/3563863239 shutdown_connections 2024-03-11T19:12:13.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.184+0000 7ff61cc1c640 1 -- 172.21.15.29:0/3563863239 wait complete. 2024-03-11T19:12:13.204 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:12:13.204 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:12:13.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:12 smithi190 ceph-mon[36292]: pgmap v423: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 168 MiB/s wr, 93 op/s 2024-03-11T19:12:13.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:12 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1434186839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:12:13.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:12 smithi170 ceph-mon[35690]: pgmap v423: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 168 MiB/s wr, 93 op/s 2024-03-11T19:12:13.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:12 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1434186839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:12:13.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.369+0000 7f2d795d0640 1 -- 172.21.15.29:0/2042239042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d74102d60 msgr2=0x7f2d741095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.369+0000 7f2d795d0640 1 --2- 172.21.15.29:0/2042239042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d74102d60 0x7f2d741095f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c009ca0 tx=0x7f2d6c02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.369+0000 7f2d795d0640 1 -- 172.21.15.29:0/2042239042 shutdown_connections 2024-03-11T19:12:13.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.370+0000 7f2d795d0640 1 --2- 172.21.15.29:0/2042239042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d74102d60 0x7f2d741095f0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.370+0000 7f2d795d0640 1 --2- 172.21.15.29:0/2042239042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d741023a0 0x7f2d74102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.370+0000 7f2d795d0640 1 --2- 172.21.15.29:0/2042239042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d741011b0 0x7f2d741015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.370+0000 7f2d795d0640 1 -- 172.21.15.29:0/2042239042 >> 172.21.15.29:0/2042239042 conn(0x7f2d740fc720 msgr2=0x7f2d740feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:13.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.370+0000 7f2d795d0640 1 -- 172.21.15.29:0/2042239042 shutdown_connections 2024-03-11T19:12:13.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.370+0000 7f2d795d0640 1 -- 172.21.15.29:0/2042239042 wait complete. 2024-03-11T19:12:13.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.371+0000 7f2d795d0640 1 Processor -- start 2024-03-11T19:12:13.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.371+0000 7f2d795d0640 1 -- start start 2024-03-11T19:12:13.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.371+0000 7f2d795d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d741011b0 0x7f2d74199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.371+0000 7f2d795d0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d741023a0 0x7f2d74199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.371+0000 7f2d795d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d74102d60 0x7f2d741a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.372+0000 7f2d795d0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d741a0e30 con 0x7f2d741011b0 2024-03-11T19:12:13.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.372+0000 7f2d72ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d741011b0 0x7f2d74199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.372+0000 7f2d727fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d741023a0 0x7f2d74199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.372+0000 7f2d727fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d741023a0 0x7f2d74199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:42024/0 (socket says 172.21.15.29:42024) 2024-03-11T19:12:13.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.372+0000 7f2d727fc640 1 -- 172.21.15.29:0/2234346669 learned_addr learned my addr 172.21.15.29:0/2234346669 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:13.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.372+0000 7f2d795d0640 1 -- 172.21.15.29:0/2234346669 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d741a0fa0 con 0x7f2d74102d60 2024-03-11T19:12:13.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.372+0000 7f2d795d0640 1 -- 172.21.15.29:0/2234346669 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d741a1160 con 0x7f2d741023a0 2024-03-11T19:12:13.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.372+0000 7f2d737fe640 1 --2- 172.21.15.29:0/2234346669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d74102d60 0x7f2d741a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.372+0000 7f2d727fc640 1 -- 172.21.15.29:0/2234346669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d74102d60 msgr2=0x7f2d741a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.372+0000 7f2d727fc640 1 --2- 172.21.15.29:0/2234346669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d74102d60 0x7f2d741a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.372+0000 7f2d727fc640 1 -- 172.21.15.29:0/2234346669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d741011b0 msgr2=0x7f2d74199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.372+0000 7f2d727fc640 1 --2- 172.21.15.29:0/2234346669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d741011b0 0x7f2d74199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.372+0000 7f2d727fc640 1 -- 172.21.15.29:0/2234346669 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d68009490 con 0x7f2d741023a0 2024-03-11T19:12:13.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.372+0000 7f2d737fe640 1 --2- 172.21.15.29:0/2234346669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d74102d60 0x7f2d741a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:13.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.372+0000 7f2d72ffd640 1 --2- 172.21.15.29:0/2234346669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d741011b0 0x7f2d74199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:13.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.373+0000 7f2d727fc640 1 --2- 172.21.15.29:0/2234346669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d741023a0 0x7f2d74199880 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800ea60 tx=0x7f2d6800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:13.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.373+0000 7f2d5bfff640 1 -- 172.21.15.29:0/2234346669 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6800cde0 con 0x7f2d741023a0 2024-03-11T19:12:13.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.373+0000 7f2d5bfff640 1 -- 172.21.15.29:0/2234346669 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2d680044d0 con 0x7f2d741023a0 2024-03-11T19:12:13.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.373+0000 7f2d795d0640 1 -- 172.21.15.29:0/2234346669 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d6c009900 con 0x7f2d741023a0 2024-03-11T19:12:13.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.373+0000 7f2d5bfff640 1 -- 172.21.15.29:0/2234346669 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d68010470 con 0x7f2d741023a0 2024-03-11T19:12:13.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.373+0000 7f2d795d0640 1 -- 172.21.15.29:0/2234346669 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d741a17c0 con 0x7f2d741023a0 2024-03-11T19:12:13.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.376+0000 7f2d5bfff640 1 -- 172.21.15.29:0/2234346669 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2d680035f0 con 0x7f2d741023a0 2024-03-11T19:12:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.376+0000 7f2d7c19b740 1 Processor -- start 2024-03-11T19:12:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.376+0000 7f2d7c19b740 1 -- start start 2024-03-11T19:12:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.377+0000 7f2d5bfff640 1 --2- 172.21.15.29:0/2234346669 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2d5c0a1ab0 0x7f2d5c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.377+0000 7f2d7c19b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b6c270 0x557cf5b6c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.377+0000 7f2d7c19b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b6cc60 0x557cf5b71760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.377+0000 7f2d7c19b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b71dd0 0x557cf5b74260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.377+0000 7f2d7c19b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf5b28960 con 0x557cf5b71dd0 2024-03-11T19:12:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.377+0000 7f2d7c19b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf5adc310 con 0x557cf5b6cc60 2024-03-11T19:12:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.377+0000 7f2d7c19b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf5ad9840 con 0x557cf5b6c270 2024-03-11T19:12:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.377+0000 7f2d737fe640 1 --2- 172.21.15.29:0/2234346669 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2d5c0a1ab0 0x7f2d5c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.377+0000 7f2d727fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b6cc60 0x557cf5b71760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.377+0000 7f2d737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b71dd0 0x557cf5b74260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.377+0000 7f2d72ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b6c270 0x557cf5b6c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.377+0000 7f2d72ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b6c270 0x557cf5b6c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:42040/0 (socket says 172.21.15.29:42040) 2024-03-11T19:12:13.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.377+0000 7f2d72ffd640 1 -- 172.21.15.29:0/3576867884 learned_addr learned my addr 172.21.15.29:0/3576867884 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:13.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.377+0000 7f2d737fe640 1 -- 172.21.15.29:0/3576867884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b6c270 msgr2=0x557cf5b6c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.377+0000 7f2d737fe640 1 --2- 172.21.15.29:0/3576867884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b6c270 0x557cf5b6c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.377+0000 7f2d737fe640 1 -- 172.21.15.29:0/3576867884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b6cc60 msgr2=0x557cf5b71760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.377+0000 7f2d737fe640 1 --2- 172.21.15.29:0/3576867884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b6cc60 0x557cf5b71760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.378+0000 7f2d737fe640 1 -- 172.21.15.29:0/3576867884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557cf5a97060 con 0x557cf5b71dd0 2024-03-11T19:12:13.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.378+0000 7f2d727fc640 1 --2- 172.21.15.29:0/3576867884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b6cc60 0x557cf5b71760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:13.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.378+0000 7f2d5bfff640 1 -- 172.21.15.29:0/2234346669 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f2d68014070 con 0x7f2d741023a0 2024-03-11T19:12:13.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.378+0000 7f2d737fe640 1 --2- 172.21.15.29:0/2234346669 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2d5c0a1ab0 0x7f2d5c0a3f70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c005e30 tx=0x7f2d6c0323c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:13.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.378+0000 7f2d737fe640 1 --2- 172.21.15.29:0/3576867884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b71dd0 0x557cf5b74260 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c02c630 tx=0x7f2d6c048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4376299af7ab8b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:13.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.378+0000 7f2d597fa640 1 -- 172.21.15.29:0/3576867884 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c050070 con 0x557cf5b71dd0 2024-03-11T19:12:13.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.378+0000 7f2d597fa640 1 -- 172.21.15.29:0/3576867884 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2d6c003e20 con 0x557cf5b71dd0 2024-03-11T19:12:13.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.378+0000 7f2d597fa640 1 -- 172.21.15.29:0/3576867884 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c048070 con 0x557cf5b71dd0 2024-03-11T19:12:13.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.379+0000 7f2d7c19b740 1 -- 172.21.15.29:0/3576867884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b71dd0 msgr2=0x557cf5b74260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.379+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/3576867884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b71dd0 0x557cf5b74260 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c02c630 tx=0x7f2d6c048000 comp rx=0 tx=0).stop 2024-03-11T19:12:13.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.380+0000 7f2d7c19b740 1 -- 172.21.15.29:0/3576867884 shutdown_connections 2024-03-11T19:12:13.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.380+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/3576867884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b71dd0 0x557cf5b74260 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.380+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/3576867884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b6cc60 0x557cf5b71760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.380+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/3576867884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b6c270 0x557cf5b6c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.380+0000 7f2d7c19b740 1 -- 172.21.15.29:0/3576867884 >> 172.21.15.29:0/3576867884 conn(0x557cf5b677e0 msgr2=0x557cf5b67be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:13.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.380+0000 7f2d7c19b740 1 -- 172.21.15.29:0/3576867884 shutdown_connections 2024-03-11T19:12:13.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.380+0000 7f2d7c19b740 1 -- 172.21.15.29:0/3576867884 wait complete. 2024-03-11T19:12:13.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.380+0000 7f2d7c19b740 1 -- start start 2024-03-11T19:12:13.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.380+0000 7f2d7c19b740 1 Processor -- start 2024-03-11T19:12:13.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.381+0000 7f2d7c19b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b6c270 0x557cf5b76450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.381+0000 7f2d7c19b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b6cc60 0x557cf5b76990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.381+0000 7f2d7c19b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b71dd0 0x557cf5b7d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.381+0000 7f2d7c19b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf5ad5020 con 0x557cf5b71dd0 2024-03-11T19:12:13.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.381+0000 7f2d7c19b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf57a8430 con 0x557cf5b6c270 2024-03-11T19:12:13.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.381+0000 7f2d7c19b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf5764e00 con 0x557cf5b6cc60 2024-03-11T19:12:13.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.381+0000 7f2d7c19b740 10 client.0 _ll_register_callbacks cb 0x557cf5a7ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:13.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.381+0000 7f2d72ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b6c270 0x557cf5b76450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.381+0000 7f2d737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b71dd0 0x557cf5b7d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.381+0000 7f2d727fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b6cc60 0x557cf5b76990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.381+0000 7f2d737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b71dd0 0x557cf5b7d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43910/0 (socket says 172.21.15.29:43910) 2024-03-11T19:12:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.381+0000 7f2d737fe640 1 -- 172.21.15.29:0/1618739912 learned_addr learned my addr 172.21.15.29:0/1618739912 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.381+0000 7f2d727fc640 1 -- 172.21.15.29:0/1618739912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b6c270 msgr2=0x557cf5b76450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.381+0000 7f2d727fc640 1 --2- 172.21.15.29:0/1618739912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b6c270 0x557cf5b76450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.381+0000 7f2d727fc640 1 -- 172.21.15.29:0/1618739912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b71dd0 msgr2=0x557cf5b7d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.381+0000 7f2d727fc640 1 --2- 172.21.15.29:0/1618739912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b71dd0 0x557cf5b7d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.381+0000 7f2d727fc640 1 -- 172.21.15.29:0/1618739912 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d74109ef0 con 0x557cf5b6cc60 2024-03-11T19:12:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.381+0000 7f2d737fe640 1 --2- 172.21.15.29:0/1618739912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b71dd0 0x557cf5b7d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.382+0000 7f2d727fc640 1 --2- 172.21.15.29:0/1618739912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b6cc60 0x557cf5b76990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x557cf5b6d6d0 tx=0x7f2d6809a370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7bc9fbac245f8c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.382+0000 7f2d56ff5640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:12:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.382+0000 7f2d56ff5640 1 -- 172.21.15.29:0/1618739912 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6809a770 con 0x557cf5b6cc60 2024-03-11T19:12:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.382+0000 7f2d56ff5640 1 -- 172.21.15.29:0/1618739912 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f2d6809e8c0 con 0x557cf5b6cc60 2024-03-11T19:12:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.382+0000 7f2d7c19b740 1 -- 172.21.15.29:0/1618739912 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557cf57d47e0 con 0x557cf5b6cc60 2024-03-11T19:12:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.382+0000 7f2d56ff5640 1 -- 172.21.15.29:0/1618739912 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d680a2f10 con 0x557cf5b6cc60 2024-03-11T19:12:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.382+0000 7f2d7c19b740 10 client.25597 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.382+0000 7f2d7c19b740 1 -- 172.21.15.29:0/1618739912 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x557cf5ad36a0 con 0x557cf5b6cc60 2024-03-11T19:12:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.383+0000 7f2d56ff5640 1 -- 172.21.15.29:0/1618739912 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x557cf57d47e0 con 0x557cf5b6cc60 2024-03-11T19:12:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.383+0000 7f2d56ff5640 1 -- 172.21.15.29:0/1618739912 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2d6801f060 con 0x557cf5b6cc60 2024-03-11T19:12:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.383+0000 7f2d7c19b740 10 client.25597 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.383+0000 7f2d7c19b740 10 client.25597 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:12:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.383+0000 7f2d7c19b740 20 client.25597 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.383+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/1618739912 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557cf5b827e0 0x557cf5b84bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.383+0000 7f2d7c19b740 4 client.25597 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:12:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.383+0000 7f2d7c19b740 1 -- 172.21.15.29:0/1618739912 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557cf5811720 con 0x557cf5b827e0 2024-03-11T19:12:13.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.383+0000 7f2d72ffd640 1 --2- 172.21.15.29:0/1618739912 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557cf5b827e0 0x557cf5b84bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.384+0000 7f2d72ffd640 1 --2- 172.21.15.29:0/1618739912 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557cf5b827e0 0x557cf5b84bd0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5733443cf0be50ad server_cookie=bbe74dff2dc9184a in_seq=0 out_seq=0 2024-03-11T19:12:13.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.384+0000 7f2d56ff5640 10 client.25597 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:12:13.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.385+0000 7f2d56ff5640 1 -- 172.21.15.29:0/1618739912 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557cf5811720 con 0x557cf5b827e0 2024-03-11T19:12:13.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.385+0000 7f2d56ff5640 10 client.25597 handle_command_reply: tid=0 2024-03-11T19:12:13.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.385+0000 7f2d7c19b740 1 client.25597 shutdown 2024-03-11T19:12:13.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.385+0000 7f2d7c19b740 1 -- 172.21.15.29:0/1618739912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b6cc60 msgr2=0x557cf5b76990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.385+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/1618739912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b6cc60 0x557cf5b76990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x557cf5b6d6d0 tx=0x7f2d6809a370 comp rx=0 tx=0).stop 2024-03-11T19:12:13.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.385+0000 7f2d7c19b740 1 -- 172.21.15.29:0/1618739912 shutdown_connections 2024-03-11T19:12:13.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.385+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/1618739912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b71dd0 0x557cf5b7d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.385+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/1618739912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b6cc60 0x557cf5b76990 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.385+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/1618739912 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557cf5b827e0 0x557cf5b84bd0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.385+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/1618739912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b6c270 0x557cf5b76450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.385+0000 7f2d7c19b740 1 -- 172.21.15.29:0/1618739912 >> 172.21.15.29:0/1618739912 conn(0x557cf5b677e0 msgr2=0x557cf5b66b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:13.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.385+0000 7f2d56ff5640 0 client.25597 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:12:13.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.386+0000 7f2d7c19b740 1 -- 172.21.15.29:0/1618739912 shutdown_connections 2024-03-11T19:12:13.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.386+0000 7f2d7c19b740 1 -- 172.21.15.29:0/1618739912 wait complete. 2024-03-11T19:12:13.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.386+0000 7f2d7c19b740 20 client.25597 trim_cache size 0 max 16384 2024-03-11T19:12:13.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.407+0000 7f2d7c19b740 1 Processor -- start 2024-03-11T19:12:13.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.408+0000 7f2d7c19b740 1 -- start start 2024-03-11T19:12:13.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.408+0000 7f2d7c19b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b6c270 0x557cf5b76450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.408+0000 7f2d7c19b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b6cc60 0x557cf5b76990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.408+0000 7f2d7c19b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b71dd0 0x557cf5b7d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.408+0000 7f2d7c19b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf5a5a130 con 0x557cf5b6cc60 2024-03-11T19:12:13.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.408+0000 7f2d7c19b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf5811720 con 0x557cf5b71dd0 2024-03-11T19:12:13.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.408+0000 7f2d7c19b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf57d47e0 con 0x557cf5b6c270 2024-03-11T19:12:13.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.408+0000 7f2d727fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b6cc60 0x557cf5b76990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.408+0000 7f2d72ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b6c270 0x557cf5b76450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.408+0000 7f2d727fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b6cc60 0x557cf5b76990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43912/0 (socket says 172.21.15.29:43912) 2024-03-11T19:12:13.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.408+0000 7f2d727fc640 1 -- 172.21.15.29:0/430928057 learned_addr learned my addr 172.21.15.29:0/430928057 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:13.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.408+0000 7f2d737fe640 1 --2- 172.21.15.29:0/430928057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b71dd0 0x557cf5b7d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.409+0000 7f2d72ffd640 1 -- 172.21.15.29:0/430928057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b71dd0 msgr2=0x557cf5b7d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.409+0000 7f2d72ffd640 1 --2- 172.21.15.29:0/430928057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b71dd0 0x557cf5b7d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.409+0000 7f2d72ffd640 1 -- 172.21.15.29:0/430928057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b6cc60 msgr2=0x557cf5b76990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.409+0000 7f2d72ffd640 1 --2- 172.21.15.29:0/430928057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b6cc60 0x557cf5b76990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.409+0000 7f2d72ffd640 1 -- 172.21.15.29:0/430928057 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557cf5a6fc20 con 0x557cf5b6c270 2024-03-11T19:12:13.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.409+0000 7f2d737fe640 1 --2- 172.21.15.29:0/430928057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b71dd0 0x557cf5b7d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:13.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.409+0000 7f2d727fc640 1 --2- 172.21.15.29:0/430928057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b6cc60 0x557cf5b76990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:13.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.409+0000 7f2d72ffd640 1 --2- 172.21.15.29:0/430928057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b6c270 0x557cf5b76450 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x557cf5b854e0 tx=0x7f2d64005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43d167e142b17633 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:13.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.409+0000 7f2d58ff9640 1 -- 172.21.15.29:0/430928057 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d64017030 con 0x557cf5b6c270 2024-03-11T19:12:13.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.409+0000 7f2d58ff9640 1 -- 172.21.15.29:0/430928057 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2d64018040 con 0x557cf5b6c270 2024-03-11T19:12:13.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.409+0000 7f2d58ff9640 1 -- 172.21.15.29:0/430928057 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d64004a00 con 0x557cf5b6c270 2024-03-11T19:12:13.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.410+0000 7f2d7c19b740 1 -- 172.21.15.29:0/430928057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b6c270 msgr2=0x557cf5b76450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.410+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/430928057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b6c270 0x557cf5b76450 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x557cf5b854e0 tx=0x7f2d64005f90 comp rx=0 tx=0).stop 2024-03-11T19:12:13.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.410+0000 7f2d7c19b740 1 -- 172.21.15.29:0/430928057 shutdown_connections 2024-03-11T19:12:13.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.410+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/430928057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b71dd0 0x557cf5b7d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.410+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/430928057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b6cc60 0x557cf5b76990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.410+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/430928057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b6c270 0x557cf5b76450 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.410+0000 7f2d7c19b740 1 -- 172.21.15.29:0/430928057 >> 172.21.15.29:0/430928057 conn(0x557cf5b677e0 msgr2=0x557cf5b89530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:13.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.411+0000 7f2d7c19b740 1 -- 172.21.15.29:0/430928057 shutdown_connections 2024-03-11T19:12:13.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.411+0000 7f2d7c19b740 1 -- 172.21.15.29:0/430928057 wait complete. 2024-03-11T19:12:13.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.411+0000 7f2d7c19b740 1 -- start start 2024-03-11T19:12:13.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.411+0000 7f2d7c19b740 1 Processor -- start 2024-03-11T19:12:13.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.411+0000 7f2d7c19b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b6c270 0x557cf5b7c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.411+0000 7f2d7c19b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b6cc60 0x557cf5b7c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.412+0000 7f2d7c19b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b71dd0 0x557cf5b7cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.412+0000 7f2d7c19b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf5bd8ca0 con 0x557cf5b6cc60 2024-03-11T19:12:13.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.412+0000 7f2d7c19b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf5bd8de0 con 0x557cf5b6c270 2024-03-11T19:12:13.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.412+0000 7f2d72ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b6c270 0x557cf5b7c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.412+0000 7f2d72ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b6c270 0x557cf5b7c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:48524/0 (socket says 172.21.15.29:48524) 2024-03-11T19:12:13.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.412+0000 7f2d72ffd640 1 -- 172.21.15.29:0/3905698769 learned_addr learned my addr 172.21.15.29:0/3905698769 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:13.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.412+0000 7f2d7c19b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf5bd8f20 con 0x557cf5b71dd0 2024-03-11T19:12:13.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.412+0000 7f2d7c19b740 10 client.0 _ll_register_callbacks cb 0x7f2d68005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:13.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.412+0000 7f2d727fc640 1 --2- 172.21.15.29:0/3905698769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b6cc60 0x557cf5b7c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.412+0000 7f2d72ffd640 1 -- 172.21.15.29:0/3905698769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b71dd0 msgr2=0x557cf5b7cc40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:12:13.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.412+0000 7f2d72ffd640 1 --2- 172.21.15.29:0/3905698769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b71dd0 0x557cf5b7cc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.412+0000 7f2d72ffd640 1 -- 172.21.15.29:0/3905698769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b6cc60 msgr2=0x557cf5b7c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.412+0000 7f2d72ffd640 1 --2- 172.21.15.29:0/3905698769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b6cc60 0x557cf5b7c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.412+0000 7f2d72ffd640 1 -- 172.21.15.29:0/3905698769 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557cf5ad36a0 con 0x557cf5b6c270 2024-03-11T19:12:13.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.412+0000 7f2d727fc640 1 --2- 172.21.15.29:0/3905698769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b6cc60 0x557cf5b7c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:12:13.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.412+0000 7f2d72ffd640 1 --2- 172.21.15.29:0/3905698769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b6c270 0x557cf5b7c1c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2d64014010 tx=0x7f2d640056f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=540e7ec1370f0541 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:13.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.413+0000 7f2d56ff5640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:12:13.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.413+0000 7f2d56ff5640 1 -- 172.21.15.29:0/3905698769 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d64017050 con 0x557cf5b6c270 2024-03-11T19:12:13.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.413+0000 7f2d56ff5640 1 -- 172.21.15.29:0/3905698769 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f2d640052b0 con 0x557cf5b6c270 2024-03-11T19:12:13.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.413+0000 7f2d7c19b740 1 -- 172.21.15.29:0/3905698769 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2d6c003e20 con 0x557cf5b6c270 2024-03-11T19:12:13.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.413+0000 7f2d56ff5640 1 -- 172.21.15.29:0/3905698769 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d6402b070 con 0x557cf5b6c270 2024-03-11T19:12:13.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.413+0000 7f2d7c19b740 10 client.25565 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:13.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.413+0000 7f2d7c19b740 1 -- 172.21.15.29:0/3905698769 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x557cf5a97060 con 0x557cf5b6c270 2024-03-11T19:12:13.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.413+0000 7f2d56ff5640 1 -- 172.21.15.29:0/3905698769 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2d6c003e20 con 0x557cf5b6c270 2024-03-11T19:12:13.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.413+0000 7f2d56ff5640 1 -- 172.21.15.29:0/3905698769 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2d64025070 con 0x557cf5b6c270 2024-03-11T19:12:13.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.413+0000 7f2d7c19b740 10 client.25565 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:13.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.413+0000 7f2d7c19b740 10 client.25565 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:12:13.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.413+0000 7f2d7c19b740 20 client.25565 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:13.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.413+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/3905698769 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557cf5bda890 0x557cf5bdccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.413+0000 7f2d7c19b740 4 client.25565 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:12:13.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.413+0000 7f2d7c19b740 1 -- 172.21.15.29:0/3905698769 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557cf5860890 con 0x557cf5bda890 2024-03-11T19:12:13.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.414+0000 7f2d727fc640 1 --2- 172.21.15.29:0/3905698769 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557cf5bda890 0x557cf5bdccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.415+0000 7f2d56ff5640 10 client.25565 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:12:13.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.415+0000 7f2d727fc640 1 --2- 172.21.15.29:0/3905698769 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557cf5bda890 0x557cf5bdccb0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=879ed5c67b50b589 server_cookie=51f717e7a2a9ba17 in_seq=0 out_seq=0 2024-03-11T19:12:13.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.415+0000 7f2d56ff5640 1 -- 172.21.15.29:0/3905698769 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557cf5860890 con 0x557cf5bda890 2024-03-11T19:12:13.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.415+0000 7f2d56ff5640 10 client.25565 handle_command_reply: tid=0 2024-03-11T19:12:13.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.415+0000 7f2d7c19b740 1 client.25565 shutdown 2024-03-11T19:12:13.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.415+0000 7f2d7c19b740 1 -- 172.21.15.29:0/3905698769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b6c270 msgr2=0x557cf5b7c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.415+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/3905698769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b6c270 0x557cf5b7c1c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2d64014010 tx=0x7f2d640056f0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.415+0000 7f2d7c19b740 1 -- 172.21.15.29:0/3905698769 shutdown_connections 2024-03-11T19:12:13.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.415+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/3905698769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557cf5b71dd0 0x557cf5b7cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.415+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/3905698769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557cf5b6cc60 0x557cf5b7c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.415+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/3905698769 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557cf5bda890 0x557cf5bdccb0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.415+0000 7f2d7c19b740 1 --2- 172.21.15.29:0/3905698769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cf5b6c270 0x557cf5b7c1c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.415+0000 7f2d56ff5640 0 client.25565 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:12:13.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.415+0000 7f2d7c19b740 1 -- 172.21.15.29:0/3905698769 >> 172.21.15.29:0/3905698769 conn(0x557cf5b677e0 msgr2=0x557cf5b6aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:13.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.416+0000 7f2d7c19b740 1 -- 172.21.15.29:0/3905698769 shutdown_connections 2024-03-11T19:12:13.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.416+0000 7f2d7c19b740 1 -- 172.21.15.29:0/3905698769 wait complete. 2024-03-11T19:12:13.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.416+0000 7f2d7c19b740 20 client.25565 trim_cache size 0 max 16384 2024-03-11T19:12:13.421 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:12:13.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.416+0000 7f2d795d0640 1 -- 172.21.15.29:0/2234346669 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2d5c0a1ab0 msgr2=0x7f2d5c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.416+0000 7f2d795d0640 1 --2- 172.21.15.29:0/2234346669 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2d5c0a1ab0 0x7f2d5c0a3f70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c005e30 tx=0x7f2d6c0323c0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.417+0000 7f2d795d0640 1 -- 172.21.15.29:0/2234346669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d741023a0 msgr2=0x7f2d74199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.417+0000 7f2d795d0640 1 --2- 172.21.15.29:0/2234346669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d741023a0 0x7f2d74199880 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800ea60 tx=0x7f2d6800ef20 comp rx=0 tx=0).stop 2024-03-11T19:12:13.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.417+0000 7f2d795d0640 1 -- 172.21.15.29:0/2234346669 shutdown_connections 2024-03-11T19:12:13.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.417+0000 7f2d795d0640 1 --2- 172.21.15.29:0/2234346669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d74102d60 0x7f2d741a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.417+0000 7f2d795d0640 1 --2- 172.21.15.29:0/2234346669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d741023a0 0x7f2d74199880 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.417+0000 7f2d795d0640 1 --2- 172.21.15.29:0/2234346669 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2d5c0a1ab0 0x7f2d5c0a3f70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.417+0000 7f2d795d0640 1 --2- 172.21.15.29:0/2234346669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d741011b0 0x7f2d74199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.417+0000 7f2d795d0640 1 -- 172.21.15.29:0/2234346669 >> 172.21.15.29:0/2234346669 conn(0x7f2d740fc720 msgr2=0x7f2d740733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:13.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.417+0000 7f2d795d0640 1 -- 172.21.15.29:0/2234346669 shutdown_connections 2024-03-11T19:12:13.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.417+0000 7f2d795d0640 1 -- 172.21.15.29:0/2234346669 wait complete. 2024-03-11T19:12:13.439 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:12:13.439 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:12:13.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.602+0000 7fafe0617640 1 -- 172.21.15.29:0/1932745908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd8100090 msgr2=0x7fafd81004b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.602+0000 7fafe0617640 1 --2- 172.21.15.29:0/1932745908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd8100090 0x7fafd81004b0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009900 tx=0x7fafc802bc50 comp rx=0 tx=0).stop 2024-03-11T19:12:13.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.602+0000 7fafe0617640 1 -- 172.21.15.29:0/1932745908 shutdown_connections 2024-03-11T19:12:13.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.602+0000 7fafe0617640 1 --2- 172.21.15.29:0/1932745908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafd8101c40 0x7fafd81084d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.602+0000 7fafe0617640 1 --2- 172.21.15.29:0/1932745908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafd8101280 0x7fafd8101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.602+0000 7fafe0617640 1 --2- 172.21.15.29:0/1932745908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd8100090 0x7fafd81004b0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.602+0000 7fafe0617640 1 -- 172.21.15.29:0/1932745908 >> 172.21.15.29:0/1932745908 conn(0x7fafd80fb600 msgr2=0x7fafd80fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:13.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.603+0000 7fafe0617640 1 -- 172.21.15.29:0/1932745908 shutdown_connections 2024-03-11T19:12:13.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.603+0000 7fafe0617640 1 -- 172.21.15.29:0/1932745908 wait complete. 2024-03-11T19:12:13.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.603+0000 7fafe0617640 1 Processor -- start 2024-03-11T19:12:13.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.603+0000 7fafe0617640 1 -- start start 2024-03-11T19:12:13.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.604+0000 7fafe0617640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafd8100090 0x7fafd810d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.604+0000 7fafe0617640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafd8101280 0x7fafd810d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.604+0000 7fafe0617640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd8101c40 0x7fafd8109ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.604+0000 7fafe0617640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd810a780 con 0x7fafd8101c40 2024-03-11T19:12:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.604+0000 7fafe0617640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd810a8f0 con 0x7fafd8100090 2024-03-11T19:12:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.604+0000 7fafe0617640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd810aab0 con 0x7fafd8101280 2024-03-11T19:12:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.604+0000 7fafdebb4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd8101c40 0x7fafd8109ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.604+0000 7fafddbb2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafd8101280 0x7fafd810d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.604+0000 7fafdebb4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd8101c40 0x7fafd8109ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43950/0 (socket says 172.21.15.29:43950) 2024-03-11T19:12:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.604+0000 7fafdebb4640 1 -- 172.21.15.29:0/52100981 learned_addr learned my addr 172.21.15.29:0/52100981 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.604+0000 7fafde3b3640 1 --2- 172.21.15.29:0/52100981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafd8100090 0x7fafd810d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.604+0000 7fafdebb4640 1 -- 172.21.15.29:0/52100981 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafd8101280 msgr2=0x7fafd810d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.604+0000 7fafdebb4640 1 --2- 172.21.15.29:0/52100981 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafd8101280 0x7fafd810d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.604+0000 7fafdebb4640 1 -- 172.21.15.29:0/52100981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafd8100090 msgr2=0x7fafd810d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.604+0000 7fafdebb4640 1 --2- 172.21.15.29:0/52100981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafd8100090 0x7fafd810d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.604+0000 7fafdebb4640 1 -- 172.21.15.29:0/52100981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc8009560 con 0x7fafd8101c40 2024-03-11T19:12:13.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.604+0000 7fafde3b3640 1 --2- 172.21.15.29:0/52100981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafd8100090 0x7fafd810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:12:13.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.604+0000 7fafdebb4640 1 --2- 172.21.15.29:0/52100981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd8101c40 0x7fafd8109ff0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fafd000eb10 tx=0x7fafd000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:13.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.605+0000 7fafcf7fe640 1 -- 172.21.15.29:0/52100981 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd0004b20 con 0x7fafd8101c40 2024-03-11T19:12:13.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.605+0000 7fafcf7fe640 1 -- 172.21.15.29:0/52100981 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fafd0004c80 con 0x7fafd8101c40 2024-03-11T19:12:13.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.605+0000 7fafcf7fe640 1 -- 172.21.15.29:0/52100981 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd0010690 con 0x7fafd8101c40 2024-03-11T19:12:13.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.605+0000 7fafe0617640 1 -- 172.21.15.29:0/52100981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd81a5910 con 0x7fafd8101c40 2024-03-11T19:12:13.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.605+0000 7fafe0617640 1 -- 172.21.15.29:0/52100981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd81a5e50 con 0x7fafd8101c40 2024-03-11T19:12:13.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.608+0000 7fafcf7fe640 1 -- 172.21.15.29:0/52100981 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fafd0005020 con 0x7fafd8101c40 2024-03-11T19:12:13.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.610+0000 7fafcf7fe640 1 --2- 172.21.15.29:0/52100981 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fafc40a1bb0 0x7fafc40a4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.611+0000 7fafcf7fe640 1 -- 172.21.15.29:0/52100981 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fafd0014070 con 0x7fafd8101c40 2024-03-11T19:12:13.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.611+0000 7fafe31df740 1 Processor -- start 2024-03-11T19:12:13.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.611+0000 7fafde3b3640 1 --2- 172.21.15.29:0/52100981 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fafc40a1bb0 0x7fafc40a4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.612+0000 7fafe31df740 1 -- start start 2024-03-11T19:12:13.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.612+0000 7fafe31df740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cb270 0x55d5c86cb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.612+0000 7fafe31df740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86cbc60 0x55d5c86d0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.612+0000 7fafe31df740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86d0dd0 0x55d5c86d3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.612+0000 7fafe31df740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5c8687960 con 0x55d5c86cbc60 2024-03-11T19:12:13.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.612+0000 7fafe31df740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5c863b310 con 0x55d5c86d0dd0 2024-03-11T19:12:13.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.612+0000 7fafe31df740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5c8638840 con 0x55d5c86cb270 2024-03-11T19:12:13.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.612+0000 7fafddbb2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86cbc60 0x55d5c86d0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.612+0000 7fafddbb2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86cbc60 0x55d5c86d0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43960/0 (socket says 172.21.15.29:43960) 2024-03-11T19:12:13.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.612+0000 7fafddbb2640 1 -- 172.21.15.29:0/2270779402 learned_addr learned my addr 172.21.15.29:0/2270779402 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:13.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.612+0000 7fafdebb4640 1 --2- 172.21.15.29:0/2270779402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86d0dd0 0x55d5c86d3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.612+0000 7fafde3b3640 1 --2- 172.21.15.29:0/52100981 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fafc40a1bb0 0x7fafc40a4070 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fafc802c260 tx=0x7fafc8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.612+0000 7fafde3b3640 1 --2- 172.21.15.29:0/2270779402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cb270 0x55d5c86cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.612+0000 7fafddbb2640 1 -- 172.21.15.29:0/2270779402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cb270 msgr2=0x55d5c86cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.612+0000 7fafddbb2640 1 --2- 172.21.15.29:0/2270779402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cb270 0x55d5c86cb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.612+0000 7fafddbb2640 1 -- 172.21.15.29:0/2270779402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86d0dd0 msgr2=0x55d5c86d3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.612+0000 7fafddbb2640 1 --2- 172.21.15.29:0/2270779402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86d0dd0 0x55d5c86d3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.612+0000 7fafddbb2640 1 -- 172.21.15.29:0/2270779402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5c85f6060 con 0x55d5c86cbc60 2024-03-11T19:12:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.613+0000 7fafddbb2640 1 --2- 172.21.15.29:0/2270779402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86cbc60 0x55d5c86d0760 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009a10 tx=0x7fafd4009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f4263fc825c84cc server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.613+0000 7fafccbfe640 1 -- 172.21.15.29:0/2270779402 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd40094e0 con 0x55d5c86cbc60 2024-03-11T19:12:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.613+0000 7fafccbfe640 1 -- 172.21.15.29:0/2270779402 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fafd4013470 con 0x55d5c86cbc60 2024-03-11T19:12:13.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.613+0000 7fafccbfe640 1 -- 172.21.15.29:0/2270779402 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd4012630 con 0x55d5c86cbc60 2024-03-11T19:12:13.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.614+0000 7fafe31df740 1 -- 172.21.15.29:0/2270779402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86cbc60 msgr2=0x55d5c86d0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.614+0000 7fafe31df740 1 --2- 172.21.15.29:0/2270779402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86cbc60 0x55d5c86d0760 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009a10 tx=0x7fafd4009ed0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.614+0000 7fafe31df740 1 -- 172.21.15.29:0/2270779402 shutdown_connections 2024-03-11T19:12:13.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.614+0000 7fafe31df740 1 --2- 172.21.15.29:0/2270779402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86d0dd0 0x55d5c86d3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.614+0000 7fafe31df740 1 --2- 172.21.15.29:0/2270779402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86cbc60 0x55d5c86d0760 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.614+0000 7fafe31df740 1 --2- 172.21.15.29:0/2270779402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cb270 0x55d5c86cb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.614+0000 7fafe31df740 1 -- 172.21.15.29:0/2270779402 >> 172.21.15.29:0/2270779402 conn(0x55d5c86c67e0 msgr2=0x55d5c86c6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:13.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.614+0000 7fafe31df740 1 -- 172.21.15.29:0/2270779402 shutdown_connections 2024-03-11T19:12:13.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.614+0000 7fafe31df740 1 -- 172.21.15.29:0/2270779402 wait complete. 2024-03-11T19:12:13.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.615+0000 7fafe31df740 1 -- start start 2024-03-11T19:12:13.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.615+0000 7fafe31df740 1 Processor -- start 2024-03-11T19:12:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.615+0000 7fafe31df740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cb270 0x55d5c86d5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.615+0000 7fafe31df740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86cbc60 0x55d5c86d5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.615+0000 7fafe31df740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86d0dd0 0x55d5c86dc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.615+0000 7fafe31df740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5c8634020 con 0x55d5c86d0dd0 2024-03-11T19:12:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.615+0000 7fafe31df740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5c8307430 con 0x55d5c86cbc60 2024-03-11T19:12:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.615+0000 7fafe31df740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5c82c3e00 con 0x55d5c86cb270 2024-03-11T19:12:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.615+0000 7fafe31df740 10 client.0 _ll_register_callbacks cb 0x55d5c85dbcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.615+0000 7fafde3b3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cb270 0x55d5c86d5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.615+0000 7fafddbb2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86cbc60 0x55d5c86d5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.615+0000 7fafde3b3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cb270 0x55d5c86d5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:42090/0 (socket says 172.21.15.29:42090) 2024-03-11T19:12:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.615+0000 7fafde3b3640 1 -- 172.21.15.29:0/843569433 learned_addr learned my addr 172.21.15.29:0/843569433 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.615+0000 7fafdebb4640 1 --2- 172.21.15.29:0/843569433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86d0dd0 0x55d5c86dc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.616+0000 7fafddbb2640 1 -- 172.21.15.29:0/843569433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cb270 msgr2=0x55d5c86d5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.616+0000 7fafddbb2640 1 --2- 172.21.15.29:0/843569433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cb270 0x55d5c86d5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.616+0000 7fafddbb2640 1 -- 172.21.15.29:0/843569433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86d0dd0 msgr2=0x55d5c86dc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.616+0000 7fafddbb2640 1 --2- 172.21.15.29:0/843569433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86d0dd0 0x55d5c86dc9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.616+0000 7fafddbb2640 1 -- 172.21.15.29:0/843569433 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd4009640 con 0x55d5c86cbc60 2024-03-11T19:12:13.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.616+0000 7fafc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:12:13.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.616+0000 7fafddbb2640 1 --2- 172.21.15.29:0/843569433 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86cbc60 0x55d5c86d5970 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x55d5c86c7750 tx=0x7fafd400ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37d777a0e680b41e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:13.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.616+0000 7fafc1ffb640 1 -- 172.21.15.29:0/843569433 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd40094e0 con 0x55d5c86cbc60 2024-03-11T19:12:13.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.616+0000 7fafc1ffb640 1 -- 172.21.15.29:0/843569433 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fafd401c660 con 0x55d5c86cbc60 2024-03-11T19:12:13.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.616+0000 7fafe31df740 1 -- 172.21.15.29:0/843569433 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d5c83337e0 con 0x55d5c86cbc60 2024-03-11T19:12:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.617+0000 7fafc1ffb640 1 -- 172.21.15.29:0/843569433 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafd401c840 con 0x55d5c86cbc60 2024-03-11T19:12:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.617+0000 7fafe31df740 10 client.25577 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.617+0000 7fafe31df740 1 -- 172.21.15.29:0/843569433 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d5c86326a0 con 0x55d5c86cbc60 2024-03-11T19:12:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.617+0000 7fafc1ffb640 1 -- 172.21.15.29:0/843569433 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fafd401ce50 con 0x55d5c86cbc60 2024-03-11T19:12:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.617+0000 7fafc1ffb640 1 -- 172.21.15.29:0/843569433 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fafd402c070 con 0x55d5c86cbc60 2024-03-11T19:12:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.617+0000 7fafe31df740 10 client.25577 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.618+0000 7fafe31df740 10 client.25577 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:12:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.618+0000 7fafe31df740 20 client.25577 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.618+0000 7fafe31df740 1 --2- 172.21.15.29:0/843569433 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d5c86e1400 0x55d5c86e3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.618+0000 7fafe31df740 4 client.25577 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:12:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.618+0000 7fafe31df740 1 -- 172.21.15.29:0/843569433 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d5c83337e0 con 0x55d5c86e1400 2024-03-11T19:12:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.618+0000 7fafde3b3640 1 --2- 172.21.15.29:0/843569433 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d5c86e1400 0x55d5c86e3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.618+0000 7fafc1ffb640 10 client.25577 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.618+0000 7fafde3b3640 1 --2- 172.21.15.29:0/843569433 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d5c86e1400 0x55d5c86e3860 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8e3516a4462d9b8 server_cookie=205310a0d9859459 in_seq=0 out_seq=0 2024-03-11T19:12:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.619+0000 7fafc1ffb640 1 -- 172.21.15.29:0/843569433 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d5c83337e0 con 0x55d5c86e1400 2024-03-11T19:12:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.619+0000 7fafc1ffb640 10 client.25577 handle_command_reply: tid=0 2024-03-11T19:12:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.619+0000 7fafe31df740 1 client.25577 shutdown 2024-03-11T19:12:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.619+0000 7fafe31df740 1 -- 172.21.15.29:0/843569433 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86cbc60 msgr2=0x55d5c86d5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.619+0000 7fafe31df740 1 --2- 172.21.15.29:0/843569433 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86cbc60 0x55d5c86d5970 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x55d5c86c7750 tx=0x7fafd400ed90 comp rx=0 tx=0).stop 2024-03-11T19:12:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.619+0000 7fafe31df740 1 -- 172.21.15.29:0/843569433 shutdown_connections 2024-03-11T19:12:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.619+0000 7fafe31df740 1 --2- 172.21.15.29:0/843569433 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d5c86e1400 0x55d5c86e3860 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.619+0000 7fafc1ffb640 0 client.25577 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.619+0000 7fafe31df740 1 --2- 172.21.15.29:0/843569433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86d0dd0 0x55d5c86dc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.619+0000 7fafe31df740 1 --2- 172.21.15.29:0/843569433 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86cbc60 0x55d5c86d5970 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.619+0000 7fafe31df740 1 --2- 172.21.15.29:0/843569433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cb270 0x55d5c86d5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.619+0000 7fafe31df740 1 -- 172.21.15.29:0/843569433 >> 172.21.15.29:0/843569433 conn(0x55d5c86c67e0 msgr2=0x55d5c86c5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.620+0000 7fafe31df740 1 -- 172.21.15.29:0/843569433 shutdown_connections 2024-03-11T19:12:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.620+0000 7fafe31df740 1 -- 172.21.15.29:0/843569433 wait complete. 2024-03-11T19:12:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.620+0000 7fafe31df740 20 client.25577 trim_cache size 0 max 16384 2024-03-11T19:12:13.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.637+0000 7fafe31df740 1 Processor -- start 2024-03-11T19:12:13.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.638+0000 7fafe31df740 1 -- start start 2024-03-11T19:12:13.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.638+0000 7fafe31df740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cb270 0x55d5c86cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.638+0000 7fafe31df740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86cbc60 0x55d5c86cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.638+0000 7fafe31df740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86d0dd0 0x55d5c86dc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.638+0000 7fafe31df740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5c85b9130 con 0x55d5c86cbc60 2024-03-11T19:12:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.638+0000 7fafe31df740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5c86d5270 con 0x55d5c86d0dd0 2024-03-11T19:12:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.638+0000 7fafe31df740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5c86d65b0 con 0x55d5c86cb270 2024-03-11T19:12:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.638+0000 7fafddbb2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86cbc60 0x55d5c86cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.638+0000 7fafddbb2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86cbc60 0x55d5c86cd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43990/0 (socket says 172.21.15.29:43990) 2024-03-11T19:12:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.638+0000 7fafddbb2640 1 -- 172.21.15.29:0/3564378914 learned_addr learned my addr 172.21.15.29:0/3564378914 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.638+0000 7fafdebb4640 1 --2- 172.21.15.29:0/3564378914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86d0dd0 0x55d5c86dc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.639+0000 7fafde3b3640 1 --2- 172.21.15.29:0/3564378914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cb270 0x55d5c86cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.639+0000 7fafddbb2640 1 -- 172.21.15.29:0/3564378914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cb270 msgr2=0x55d5c86cd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.639+0000 7fafddbb2640 1 --2- 172.21.15.29:0/3564378914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cb270 0x55d5c86cd1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.639+0000 7fafddbb2640 1 -- 172.21.15.29:0/3564378914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86d0dd0 msgr2=0x55d5c86dc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.639+0000 7fafddbb2640 1 --2- 172.21.15.29:0/3564378914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86d0dd0 0x55d5c86dc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.639+0000 7fafddbb2640 1 -- 172.21.15.29:0/3564378914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5c85cec20 con 0x55d5c86cbc60 2024-03-11T19:12:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.639+0000 7fafddbb2640 1 --2- 172.21.15.29:0/3564378914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86cbc60 0x55d5c86cd6f0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fafd4000c00 tx=0x7fafd4006e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee3b8a3fd4f87d01 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.639+0000 7fafc3fff640 1 -- 172.21.15.29:0/3564378914 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd40094e0 con 0x55d5c86cbc60 2024-03-11T19:12:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.639+0000 7fafc3fff640 1 -- 172.21.15.29:0/3564378914 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fafd400ee80 con 0x55d5c86cbc60 2024-03-11T19:12:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.639+0000 7fafc3fff640 1 -- 172.21.15.29:0/3564378914 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd401ca90 con 0x55d5c86cbc60 2024-03-11T19:12:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.641+0000 7fafe31df740 1 -- 172.21.15.29:0/3564378914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86cbc60 msgr2=0x55d5c86cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.641+0000 7fafe31df740 1 --2- 172.21.15.29:0/3564378914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86cbc60 0x55d5c86cd6f0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fafd4000c00 tx=0x7fafd4006e70 comp rx=0 tx=0).stop 2024-03-11T19:12:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.641+0000 7fafe31df740 1 -- 172.21.15.29:0/3564378914 shutdown_connections 2024-03-11T19:12:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.641+0000 7fafe31df740 1 --2- 172.21.15.29:0/3564378914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86d0dd0 0x55d5c86dc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.641+0000 7fafe31df740 1 --2- 172.21.15.29:0/3564378914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86cbc60 0x55d5c86cd6f0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.641+0000 7fafe31df740 1 --2- 172.21.15.29:0/3564378914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cb270 0x55d5c86cd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.641+0000 7fafe31df740 1 -- 172.21.15.29:0/3564378914 >> 172.21.15.29:0/3564378914 conn(0x55d5c86c67e0 msgr2=0x55d5c86c6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.641+0000 7fafe31df740 1 -- 172.21.15.29:0/3564378914 shutdown_connections 2024-03-11T19:12:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.641+0000 7fafe31df740 1 -- 172.21.15.29:0/3564378914 wait complete. 2024-03-11T19:12:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.642+0000 7fafe31df740 1 -- start start 2024-03-11T19:12:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.642+0000 7fafe31df740 1 Processor -- start 2024-03-11T19:12:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.642+0000 7fafe31df740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86cb270 0x55d5c86db500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.642+0000 7fafe31df740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cbc60 0x55d5c86dba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.642+0000 7fafe31df740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86d0dd0 0x55d5c8737970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.642+0000 7fafdebb4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86d0dd0 0x55d5c8737970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.642+0000 7fafddbb2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cbc60 0x55d5c86dba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.642+0000 7fafdebb4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86d0dd0 0x55d5c8737970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44000/0 (socket says 172.21.15.29:44000) 2024-03-11T19:12:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.642+0000 7fafdebb4640 1 -- 172.21.15.29:0/3159671432 learned_addr learned my addr 172.21.15.29:0/3159671432 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.642+0000 7fafde3b3640 1 --2- 172.21.15.29:0/3159671432 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86cb270 0x55d5c86db500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.642+0000 7fafe31df740 1 -- 172.21.15.29:0/3159671432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5c86dc280 con 0x55d5c86d0dd0 2024-03-11T19:12:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.642+0000 7fafe31df740 1 -- 172.21.15.29:0/3159671432 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5c8737eb0 con 0x55d5c86cb270 2024-03-11T19:12:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.642+0000 7fafe31df740 1 -- 172.21.15.29:0/3159671432 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5c8737ff0 con 0x55d5c86cbc60 2024-03-11T19:12:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.643+0000 7fafe31df740 10 client.0 _ll_register_callbacks cb 0x7fafd4009430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.643+0000 7fafddbb2640 1 -- 172.21.15.29:0/3159671432 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86cb270 msgr2=0x55d5c86db500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.643+0000 7fafddbb2640 1 --2- 172.21.15.29:0/3159671432 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86cb270 0x55d5c86db500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.643+0000 7fafddbb2640 1 -- 172.21.15.29:0/3159671432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86d0dd0 msgr2=0x55d5c8737970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:13.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.643+0000 7fafddbb2640 1 --2- 172.21.15.29:0/3159671432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86d0dd0 0x55d5c8737970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.643+0000 7fafddbb2640 1 -- 172.21.15.29:0/3159671432 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d5c86326a0 con 0x55d5c86cbc60 2024-03-11T19:12:13.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.643+0000 7fafdebb4640 1 --2- 172.21.15.29:0/3159671432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86d0dd0 0x55d5c8737970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:13.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.643+0000 7fafddbb2640 1 --2- 172.21.15.29:0/3159671432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cbc60 0x55d5c86dba40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fafd400b010 tx=0x7fafd4013e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1d56c73fe346f51 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:13.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.643+0000 7fafc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:12:13.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.644+0000 7fafc1ffb640 1 -- 172.21.15.29:0/3159671432 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd4005b70 con 0x55d5c86cbc60 2024-03-11T19:12:13.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.644+0000 7fafc1ffb640 1 -- 172.21.15.29:0/3159671432 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fafd4005cd0 con 0x55d5c86cbc60 2024-03-11T19:12:13.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.644+0000 7fafe31df740 1 -- 172.21.15.29:0/3159671432 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d5c8370720 con 0x55d5c86cbc60 2024-03-11T19:12:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.644+0000 7fafc1ffb640 1 -- 172.21.15.29:0/3159671432 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafd402e420 con 0x55d5c86cbc60 2024-03-11T19:12:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.644+0000 7fafe31df740 10 client.25615 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.644+0000 7fafe31df740 1 -- 172.21.15.29:0/3159671432 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d5c85f48c0 con 0x55d5c86cbc60 2024-03-11T19:12:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.644+0000 7fafc1ffb640 1 -- 172.21.15.29:0/3159671432 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55d5c8370720 con 0x55d5c86cbc60 2024-03-11T19:12:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.645+0000 7fafc1ffb640 1 -- 172.21.15.29:0/3159671432 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fafd403f070 con 0x55d5c86cbc60 2024-03-11T19:12:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.645+0000 7fafe31df740 10 client.25615 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.645+0000 7fafe31df740 10 client.25615 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:12:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.645+0000 7fafe31df740 20 client.25615 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.645+0000 7fafe31df740 1 --2- 172.21.15.29:0/3159671432 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d5c86e1400 0x55d5c873b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.645+0000 7fafe31df740 4 client.25615 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:12:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.645+0000 7fafe31df740 1 -- 172.21.15.29:0/3159671432 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d5c83bf890 con 0x55d5c86e1400 2024-03-11T19:12:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.645+0000 7fafde3b3640 1 --2- 172.21.15.29:0/3159671432 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d5c86e1400 0x55d5c873b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.646+0000 7fafc1ffb640 10 client.25615 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.646+0000 7fafde3b3640 1 --2- 172.21.15.29:0/3159671432 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d5c86e1400 0x55d5c873b500 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=918dd1f1a56fa527 server_cookie=174c9c6af032a06 in_seq=0 out_seq=0 2024-03-11T19:12:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.646+0000 7fafc1ffb640 1 -- 172.21.15.29:0/3159671432 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d5c83bf890 con 0x55d5c86e1400 2024-03-11T19:12:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.646+0000 7fafc1ffb640 10 client.25615 handle_command_reply: tid=0 2024-03-11T19:12:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.646+0000 7fafe31df740 1 client.25615 shutdown 2024-03-11T19:12:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.647+0000 7fafe31df740 1 -- 172.21.15.29:0/3159671432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cbc60 msgr2=0x55d5c86dba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.647+0000 7fafe31df740 1 --2- 172.21.15.29:0/3159671432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cbc60 0x55d5c86dba40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fafd400b010 tx=0x7fafd4013e10 comp rx=0 tx=0).stop 2024-03-11T19:12:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.647+0000 7fafe31df740 1 -- 172.21.15.29:0/3159671432 shutdown_connections 2024-03-11T19:12:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.647+0000 7fafe31df740 1 --2- 172.21.15.29:0/3159671432 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d5c86e1400 0x55d5c873b500 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.647+0000 7fafc1ffb640 0 client.25615 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.647+0000 7fafe31df740 1 --2- 172.21.15.29:0/3159671432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d5c86d0dd0 0x55d5c8737970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.647+0000 7fafe31df740 1 --2- 172.21.15.29:0/3159671432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5c86cbc60 0x55d5c86dba40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.647+0000 7fafe31df740 1 --2- 172.21.15.29:0/3159671432 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d5c86cb270 0x55d5c86db500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.647+0000 7fafe31df740 1 -- 172.21.15.29:0/3159671432 >> 172.21.15.29:0/3159671432 conn(0x55d5c86c67e0 msgr2=0x55d5c86d52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.647+0000 7fafe31df740 1 -- 172.21.15.29:0/3159671432 shutdown_connections 2024-03-11T19:12:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.647+0000 7fafe31df740 1 -- 172.21.15.29:0/3159671432 wait complete. 2024-03-11T19:12:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.647+0000 7fafe31df740 20 client.25615 trim_cache size 0 max 16384 2024-03-11T19:12:13.653 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:12:13.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.648+0000 7fafe0617640 1 -- 172.21.15.29:0/52100981 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fafc40a1bb0 msgr2=0x7fafc40a4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.648+0000 7fafe0617640 1 --2- 172.21.15.29:0/52100981 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fafc40a1bb0 0x7fafc40a4070 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fafc802c260 tx=0x7fafc8005fd0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.648+0000 7fafe0617640 1 -- 172.21.15.29:0/52100981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd8101c40 msgr2=0x7fafd8109ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.648+0000 7fafe0617640 1 --2- 172.21.15.29:0/52100981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd8101c40 0x7fafd8109ff0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fafd000eb10 tx=0x7fafd000c2d0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.649+0000 7fafe0617640 1 -- 172.21.15.29:0/52100981 shutdown_connections 2024-03-11T19:12:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.649+0000 7fafe0617640 1 --2- 172.21.15.29:0/52100981 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fafc40a1bb0 0x7fafc40a4070 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.649+0000 7fafe0617640 1 --2- 172.21.15.29:0/52100981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd8101c40 0x7fafd8109ff0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.649+0000 7fafe0617640 1 --2- 172.21.15.29:0/52100981 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fafd8101280 0x7fafd810d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.649+0000 7fafe0617640 1 --2- 172.21.15.29:0/52100981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafd8100090 0x7fafd810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.649+0000 7fafe0617640 1 -- 172.21.15.29:0/52100981 >> 172.21.15.29:0/52100981 conn(0x7fafd80fb600 msgr2=0x7fafd81045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.649+0000 7fafe0617640 1 -- 172.21.15.29:0/52100981 shutdown_connections 2024-03-11T19:12:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:13.649+0000 7fafe0617640 1 -- 172.21.15.29:0/52100981 wait complete. 2024-03-11T19:12:14.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:13 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:12:14.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:13 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:12:14.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:13 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:12:14.669 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:12:14.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:12:14.670 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:12:14.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.834+0000 7f73b85d0640 1 -- 172.21.15.29:0/1402479445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73b01033a0 msgr2=0x7f73b01037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:14.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.835+0000 7f73b85d0640 1 --2- 172.21.15.29:0/1402479445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73b01033a0 0x7f73b01037c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009900 tx=0x7f73ac02bc50 comp rx=0 tx=0).stop 2024-03-11T19:12:14.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.835+0000 7f73b85d0640 1 -- 172.21.15.29:0/1402479445 shutdown_connections 2024-03-11T19:12:14.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.835+0000 7f73b85d0640 1 --2- 172.21.15.29:0/1402479445 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73b0104f50 0x7f73b010b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.835+0000 7f73b85d0640 1 --2- 172.21.15.29:0/1402479445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73b0104590 0x7f73b0104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.835+0000 7f73b85d0640 1 --2- 172.21.15.29:0/1402479445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73b01033a0 0x7f73b01037c0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.835+0000 7f73b85d0640 1 -- 172.21.15.29:0/1402479445 >> 172.21.15.29:0/1402479445 conn(0x7f73b00fe930 msgr2=0x7f73b0100d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:14.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.835+0000 7f73b85d0640 1 -- 172.21.15.29:0/1402479445 shutdown_connections 2024-03-11T19:12:14.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.835+0000 7f73b85d0640 1 -- 172.21.15.29:0/1402479445 wait complete. 2024-03-11T19:12:14.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.836+0000 7f73b85d0640 1 Processor -- start 2024-03-11T19:12:14.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.836+0000 7f73b85d0640 1 -- start start 2024-03-11T19:12:14.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.836+0000 7f73b85d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73b01033a0 0x7f73b0199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:14.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.836+0000 7f73b85d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73b0104590 0x7f73b01998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:14.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.836+0000 7f73b636c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73b01033a0 0x7f73b0199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:14.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.836+0000 7f73b636c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73b01033a0 0x7f73b0199370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44014/0 (socket says 172.21.15.29:44014) 2024-03-11T19:12:14.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.836+0000 7f73b636c640 1 -- 172.21.15.29:0/810684633 learned_addr learned my addr 172.21.15.29:0/810684633 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:14.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.836+0000 7f73b85d0640 1 --2- 172.21.15.29:0/810684633 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73b0104f50 0x7f73b01a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:14.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.836+0000 7f73b85d0640 1 -- 172.21.15.29:0/810684633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b01a0e60 con 0x7f73b01033a0 2024-03-11T19:12:14.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.836+0000 7f73b85d0640 1 -- 172.21.15.29:0/810684633 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b01a0fd0 con 0x7f73b0104590 2024-03-11T19:12:14.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.836+0000 7f73b85d0640 1 -- 172.21.15.29:0/810684633 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b01a1190 con 0x7f73b0104f50 2024-03-11T19:12:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.837+0000 7f73b5b6b640 1 --2- 172.21.15.29:0/810684633 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73b0104590 0x7f73b01998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.837+0000 7f73b6b6d640 1 --2- 172.21.15.29:0/810684633 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73b0104f50 0x7f73b01a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.837+0000 7f73b636c640 1 -- 172.21.15.29:0/810684633 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73b0104f50 msgr2=0x7f73b01a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.837+0000 7f73b636c640 1 --2- 172.21.15.29:0/810684633 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73b0104f50 0x7f73b01a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.837+0000 7f73b636c640 1 -- 172.21.15.29:0/810684633 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73b0104590 msgr2=0x7f73b01998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.837+0000 7f73b636c640 1 --2- 172.21.15.29:0/810684633 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73b0104590 0x7f73b01998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.837+0000 7f73b636c640 1 -- 172.21.15.29:0/810684633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac009560 con 0x7f73b01033a0 2024-03-11T19:12:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.837+0000 7f73b6b6d640 1 --2- 172.21.15.29:0/810684633 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73b0104f50 0x7f73b01a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:12:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.837+0000 7f73b636c640 1 --2- 172.21.15.29:0/810684633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73b01033a0 0x7f73b0199370 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f73ac02c7f0 tx=0x7f73ac004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.837+0000 7f73a77fe640 1 -- 172.21.15.29:0/810684633 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac0035f0 con 0x7f73b01033a0 2024-03-11T19:12:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.837+0000 7f73a77fe640 1 -- 172.21.15.29:0/810684633 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f73ac003c00 con 0x7f73b01033a0 2024-03-11T19:12:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.838+0000 7f73a77fe640 1 -- 172.21.15.29:0/810684633 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac0054e0 con 0x7f73b01033a0 2024-03-11T19:12:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.838+0000 7f73b85d0640 1 -- 172.21.15.29:0/810684633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b01a1410 con 0x7f73b01033a0 2024-03-11T19:12:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.838+0000 7f73b85d0640 1 -- 172.21.15.29:0/810684633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b01a1970 con 0x7f73b01033a0 2024-03-11T19:12:14.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.841+0000 7f73a77fe640 1 -- 172.21.15.29:0/810684633 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f73ac003750 con 0x7f73b01033a0 2024-03-11T19:12:14.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.842+0000 7f73bb19c740 1 Processor -- start 2024-03-11T19:12:14.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.842+0000 7f73a77fe640 1 --2- 172.21.15.29:0/810684633 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f739c0a1b00 0x7f739c0a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:14.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.842+0000 7f73b6b6d640 1 --2- 172.21.15.29:0/810684633 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f739c0a1b00 0x7f739c0a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:14.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.842+0000 7f73a77fe640 1 -- 172.21.15.29:0/810684633 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f73ac0ef980 con 0x7f73b01033a0 2024-03-11T19:12:14.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.842+0000 7f73bb19c740 1 -- start start 2024-03-11T19:12:14.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.843+0000 7f73bb19c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c154520 0x55e94c154940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:14.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.843+0000 7f73b6b6d640 1 --2- 172.21.15.29:0/810684633 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f739c0a1b00 0x7f739c0a3fc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f73a8005fd0 tx=0x7f73a8009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:14.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.843+0000 7f73bb19c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c1556f0 0x55e94c155b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:14.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.843+0000 7f73bb19c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c1560b0 0x55e94c15c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:14.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.843+0000 7f73bb19c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94c0c3f90 con 0x55e94c154520 2024-03-11T19:12:14.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.843+0000 7f73b636c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c154520 0x55e94c154940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:14.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.843+0000 7f73b636c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c154520 0x55e94c154940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44030/0 (socket says 172.21.15.29:44030) 2024-03-11T19:12:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.843+0000 7f73b636c640 1 -- 172.21.15.29:0/533043185 learned_addr learned my addr 172.21.15.29:0/533043185 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.843+0000 7f73bb19c740 1 -- 172.21.15.29:0/533043185 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94c0c14c0 con 0x55e94c1560b0 2024-03-11T19:12:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.843+0000 7f73bb19c740 1 -- 172.21.15.29:0/533043185 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94c0bcca0 con 0x55e94c1556f0 2024-03-11T19:12:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.843+0000 7f73b5b6b640 1 --2- 172.21.15.29:0/533043185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c1556f0 0x55e94c155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.843+0000 7f73b6b6d640 1 --2- 172.21.15.29:0/533043185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c1560b0 0x55e94c15c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.844+0000 7f73b5b6b640 1 -- 172.21.15.29:0/533043185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c1560b0 msgr2=0x55e94c15c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.844+0000 7f73b5b6b640 1 --2- 172.21.15.29:0/533043185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c1560b0 0x55e94c15c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.844+0000 7f73b5b6b640 1 -- 172.21.15.29:0/533043185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c154520 msgr2=0x55e94c154940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.844+0000 7f73b5b6b640 1 --2- 172.21.15.29:0/533043185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c154520 0x55e94c154940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.844+0000 7f73b5b6b640 1 -- 172.21.15.29:0/533043185 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e94c07f0b0 con 0x55e94c1556f0 2024-03-11T19:12:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.844+0000 7f73b6b6d640 1 --2- 172.21.15.29:0/533043185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c1560b0 0x55e94c15c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.844+0000 7f73b636c640 1 --2- 172.21.15.29:0/533043185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c154520 0x55e94c154940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.844+0000 7f73b5b6b640 1 --2- 172.21.15.29:0/533043185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c1556f0 0x55e94c155b70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f73a0009bf0 tx=0x7f73a0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2eb09e245ffb173e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.844+0000 7f73a4bfe640 1 -- 172.21.15.29:0/533043185 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0004050 con 0x55e94c1556f0 2024-03-11T19:12:14.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.844+0000 7f73a4bfe640 1 -- 172.21.15.29:0/533043185 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f73a0005020 con 0x55e94c1556f0 2024-03-11T19:12:14.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.844+0000 7f73a4bfe640 1 -- 172.21.15.29:0/533043185 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0004c30 con 0x55e94c1556f0 2024-03-11T19:12:14.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.845+0000 7f73bb19c740 1 -- 172.21.15.29:0/533043185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c1556f0 msgr2=0x55e94c155b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:14.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.845+0000 7f73bb19c740 1 --2- 172.21.15.29:0/533043185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c1556f0 0x55e94c155b70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f73a0009bf0 tx=0x7f73a0011810 comp rx=0 tx=0).stop 2024-03-11T19:12:14.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.845+0000 7f73bb19c740 1 -- 172.21.15.29:0/533043185 shutdown_connections 2024-03-11T19:12:14.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.845+0000 7f73bb19c740 1 --2- 172.21.15.29:0/533043185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c1560b0 0x55e94c15c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.845+0000 7f73bb19c740 1 --2- 172.21.15.29:0/533043185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c1556f0 0x55e94c155b70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.845+0000 7f73bb19c740 1 --2- 172.21.15.29:0/533043185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c154520 0x55e94c154940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.845+0000 7f73bb19c740 1 -- 172.21.15.29:0/533043185 >> 172.21.15.29:0/533043185 conn(0x55e94c14f700 msgr2=0x55e94c14fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:14.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.846+0000 7f73bb19c740 1 -- 172.21.15.29:0/533043185 shutdown_connections 2024-03-11T19:12:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.846+0000 7f73bb19c740 1 -- 172.21.15.29:0/533043185 wait complete. 2024-03-11T19:12:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.846+0000 7f73bb19c740 1 -- start start 2024-03-11T19:12:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.846+0000 7f73bb19c740 1 Processor -- start 2024-03-11T19:12:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.846+0000 7f73bb19c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c154520 0x55e94c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.846+0000 7f73bb19c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c1556f0 0x55e94c15ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.847+0000 7f73bb19c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c1560b0 0x55e94c165cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.847+0000 7f73bb19c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94c061f40 con 0x55e94c1560b0 2024-03-11T19:12:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.847+0000 7f73bb19c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94bd904b0 con 0x55e94c1556f0 2024-03-11T19:12:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.847+0000 7f73bb19c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94c042670 con 0x55e94c154520 2024-03-11T19:12:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.847+0000 7f73bb19c740 10 client.0 _ll_register_callbacks cb 0x55e94c0aff70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.847+0000 7f73b6b6d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c1560b0 0x55e94c165cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.847+0000 7f73b6b6d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c1560b0 0x55e94c165cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44042/0 (socket says 172.21.15.29:44042) 2024-03-11T19:12:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.847+0000 7f73b6b6d640 1 -- 172.21.15.29:0/4224781505 learned_addr learned my addr 172.21.15.29:0/4224781505 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.847+0000 7f73b5b6b640 1 --2- 172.21.15.29:0/4224781505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c1556f0 0x55e94c15ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.847+0000 7f73b636c640 1 --2- 172.21.15.29:0/4224781505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c154520 0x55e94c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.847+0000 7f73b6b6d640 1 -- 172.21.15.29:0/4224781505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c154520 msgr2=0x55e94c15e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.847+0000 7f73b6b6d640 1 --2- 172.21.15.29:0/4224781505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c154520 0x55e94c15e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.847+0000 7f73b6b6d640 1 -- 172.21.15.29:0/4224781505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c1556f0 msgr2=0x55e94c15ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.847+0000 7f73b6b6d640 1 --2- 172.21.15.29:0/4224781505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c1556f0 0x55e94c15ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.847+0000 7f73b6b6d640 1 -- 172.21.15.29:0/4224781505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a0009850 con 0x55e94c1560b0 2024-03-11T19:12:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.847+0000 7f73b636c640 1 --2- 172.21.15.29:0/4224781505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c154520 0x55e94c15e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.847+0000 7f73b6b6d640 1 --2- 172.21.15.29:0/4224781505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c1560b0 0x55e94c165cd0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f73a8011e90 tx=0x7f73a8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7bd2683c7662c5a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.848+0000 7f7399ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:12:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.848+0000 7f7399ffb640 1 -- 172.21.15.29:0/4224781505 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a801a7b0 con 0x55e94c1560b0 2024-03-11T19:12:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.848+0000 7f7399ffb640 1 -- 172.21.15.29:0/4224781505 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f73a801fe40 con 0x55e94c1560b0 2024-03-11T19:12:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.848+0000 7f73bb19c740 1 -- 172.21.15.29:0/4224781505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e94c096f40 con 0x55e94c1560b0 2024-03-11T19:12:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.848+0000 7f7399ffb640 1 -- 172.21.15.29:0/4224781505 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73a801a910 con 0x55e94c1560b0 2024-03-11T19:12:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.848+0000 7f73bb19c740 10 client.16065 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.848+0000 7f73bb19c740 1 -- 172.21.15.29:0/4224781505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e94c0bb320 con 0x55e94c1560b0 2024-03-11T19:12:14.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.848+0000 7f7399ffb640 1 -- 172.21.15.29:0/4224781505 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55e94c096f40 con 0x55e94c1560b0 2024-03-11T19:12:14.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.849+0000 7f7399ffb640 1 -- 172.21.15.29:0/4224781505 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f73a8028be0 con 0x55e94c1560b0 2024-03-11T19:12:14.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.849+0000 7f73bb19c740 10 client.16065 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:14.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.849+0000 7f73bb19c740 10 client.16065 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:12:14.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.849+0000 7f73bb19c740 20 client.16065 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:14.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.849+0000 7f73bb19c740 1 --2- 172.21.15.29:0/4224781505 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e94c16aa40 0x55e94c16ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:14.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.849+0000 7f73bb19c740 4 client.16065 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:12:14.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.849+0000 7f73bb19c740 1 -- 172.21.15.29:0/4224781505 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e94c0b3310 con 0x55e94c16aa40 2024-03-11T19:12:14.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.849+0000 7f73b636c640 1 --2- 172.21.15.29:0/4224781505 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e94c16aa40 0x55e94c16ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:14.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.850+0000 7f7399ffb640 10 client.16065 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:14.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.850+0000 7f73b636c640 1 --2- 172.21.15.29:0/4224781505 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e94c16aa40 0x55e94c16ce60 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a03cd08b706a501 server_cookie=d1eb6beda64b22f7 in_seq=0 out_seq=0 2024-03-11T19:12:14.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.850+0000 7f7399ffb640 1 -- 172.21.15.29:0/4224781505 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e94c0b3310 con 0x55e94c16aa40 2024-03-11T19:12:14.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.850+0000 7f7399ffb640 10 client.16065 handle_command_reply: tid=0 2024-03-11T19:12:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.850+0000 7f73bb19c740 1 client.16065 shutdown 2024-03-11T19:12:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.851+0000 7f73bb19c740 1 -- 172.21.15.29:0/4224781505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c1560b0 msgr2=0x55e94c165cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.851+0000 7f73bb19c740 1 --2- 172.21.15.29:0/4224781505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c1560b0 0x55e94c165cd0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f73a8011e90 tx=0x7f73a8004ab0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.851+0000 7f73bb19c740 1 -- 172.21.15.29:0/4224781505 shutdown_connections 2024-03-11T19:12:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.851+0000 7f73bb19c740 1 --2- 172.21.15.29:0/4224781505 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e94c16aa40 0x55e94c16ce60 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.851+0000 7f73bb19c740 1 --2- 172.21.15.29:0/4224781505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c1560b0 0x55e94c165cd0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.851+0000 7f73bb19c740 1 --2- 172.21.15.29:0/4224781505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c1556f0 0x55e94c15ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.851+0000 7f73bb19c740 1 --2- 172.21.15.29:0/4224781505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c154520 0x55e94c15e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.851+0000 7f73bb19c740 1 -- 172.21.15.29:0/4224781505 >> 172.21.15.29:0/4224781505 conn(0x55e94c14f700 msgr2=0x55e94c14ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.851+0000 7f7399ffb640 0 client.16065 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.851+0000 7f73bb19c740 1 -- 172.21.15.29:0/4224781505 shutdown_connections 2024-03-11T19:12:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.851+0000 7f73bb19c740 1 -- 172.21.15.29:0/4224781505 wait complete. 2024-03-11T19:12:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.851+0000 7f73bb19c740 20 client.16065 trim_cache size 0 max 16384 2024-03-11T19:12:14.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.877+0000 7f73bb19c740 1 Processor -- start 2024-03-11T19:12:14.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.877+0000 7f73bb19c740 1 -- start start 2024-03-11T19:12:14.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.878+0000 7f73bb19c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c154520 0x55e94c159260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:14.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.878+0000 7f73bb19c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c1556f0 0x55e94c1597a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:14.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.878+0000 7f73bb19c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c1560b0 0x55e94c165c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:14.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.878+0000 7f73b636c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c154520 0x55e94c159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:14.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.878+0000 7f73b636c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c154520 0x55e94c159260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:42142/0 (socket says 172.21.15.29:42142) 2024-03-11T19:12:14.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.878+0000 7f73bb19c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94c14ff40 con 0x55e94c1560b0 2024-03-11T19:12:14.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.878+0000 7f73bb19c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94c14b4e0 con 0x55e94c1556f0 2024-03-11T19:12:14.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.878+0000 7f73bb19c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94c153f60 con 0x55e94c154520 2024-03-11T19:12:14.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.878+0000 7f73b5b6b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c1556f0 0x55e94c1597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:14.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.878+0000 7f73b5b6b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c1556f0 0x55e94c1597a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:48632/0 (socket says 172.21.15.29:48632) 2024-03-11T19:12:14.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.878+0000 7f73b5b6b640 1 -- 172.21.15.29:0/4294763565 learned_addr learned my addr 172.21.15.29:0/4294763565 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:14.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.878+0000 7f73b6b6d640 1 --2- 172.21.15.29:0/4294763565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c1560b0 0x55e94c165c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:14.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.878+0000 7f73b5b6b640 1 -- 172.21.15.29:0/4294763565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c154520 msgr2=0x55e94c159260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:14.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.879+0000 7f73b5b6b640 1 --2- 172.21.15.29:0/4294763565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c154520 0x55e94c159260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.879+0000 7f73b5b6b640 1 -- 172.21.15.29:0/4294763565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c1560b0 msgr2=0x55e94c165c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:14.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.879+0000 7f73b5b6b640 1 --2- 172.21.15.29:0/4294763565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c1560b0 0x55e94c165c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.879+0000 7f73b5b6b640 1 -- 172.21.15.29:0/4294763565 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e94c057c00 con 0x55e94c1556f0 2024-03-11T19:12:14.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.879+0000 7f73b6b6d640 1 --2- 172.21.15.29:0/4294763565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c1560b0 0x55e94c165c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:12:14.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.879+0000 7f73b5b6b640 1 --2- 172.21.15.29:0/4294763565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c1556f0 0x55e94c1597a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55e94c150670 tx=0x7f73a0009dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1520d58f68c6814e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:14.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.879+0000 7f739bfff640 1 -- 172.21.15.29:0/4294763565 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0005240 con 0x55e94c1556f0 2024-03-11T19:12:14.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.879+0000 7f739bfff640 1 -- 172.21.15.29:0/4294763565 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f73a0009ec0 con 0x55e94c1556f0 2024-03-11T19:12:14.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.879+0000 7f739bfff640 1 -- 172.21.15.29:0/4294763565 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0019690 con 0x55e94c1556f0 2024-03-11T19:12:14.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.880+0000 7f73bb19c740 1 -- 172.21.15.29:0/4294763565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c1556f0 msgr2=0x55e94c1597a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:14.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.880+0000 7f73bb19c740 1 --2- 172.21.15.29:0/4294763565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c1556f0 0x55e94c1597a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55e94c150670 tx=0x7f73a0009dd0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.880+0000 7f73bb19c740 1 -- 172.21.15.29:0/4294763565 shutdown_connections 2024-03-11T19:12:14.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.880+0000 7f73bb19c740 1 --2- 172.21.15.29:0/4294763565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c1560b0 0x55e94c165c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.880+0000 7f73bb19c740 1 --2- 172.21.15.29:0/4294763565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c1556f0 0x55e94c1597a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.880+0000 7f73bb19c740 1 --2- 172.21.15.29:0/4294763565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c154520 0x55e94c159260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.880+0000 7f73bb19c740 1 -- 172.21.15.29:0/4294763565 >> 172.21.15.29:0/4294763565 conn(0x55e94c14f700 msgr2=0x55e94c15e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:14.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.880+0000 7f73bb19c740 1 -- 172.21.15.29:0/4294763565 shutdown_connections 2024-03-11T19:12:14.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.880+0000 7f73bb19c740 1 -- 172.21.15.29:0/4294763565 wait complete. 2024-03-11T19:12:14.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.881+0000 7f73bb19c740 1 -- start start 2024-03-11T19:12:14.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.881+0000 7f73bb19c740 1 Processor -- start 2024-03-11T19:12:14.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.881+0000 7f73bb19c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c154520 0x55e94c162bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:14.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.881+0000 7f73bb19c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c1556f0 0x55e94c1630f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:14.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.882+0000 7f73bb19c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c1560b0 0x55e94c1c0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:14.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.882+0000 7f73bb19c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94c163700 con 0x55e94c1556f0 2024-03-11T19:12:14.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.882+0000 7f73bb19c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94c1c1470 con 0x55e94c154520 2024-03-11T19:12:14.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.882+0000 7f73bb19c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94c1c15b0 con 0x55e94c1560b0 2024-03-11T19:12:14.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.882+0000 7f73b636c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c154520 0x55e94c162bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:14.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.882+0000 7f73b636c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c154520 0x55e94c162bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:48644/0 (socket says 172.21.15.29:48644) 2024-03-11T19:12:14.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.882+0000 7f73b636c640 1 -- 172.21.15.29:0/3157407441 learned_addr learned my addr 172.21.15.29:0/3157407441 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:14.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.882+0000 7f73b5b6b640 1 --2- 172.21.15.29:0/3157407441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c1556f0 0x55e94c1630f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:14.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.882+0000 7f73b6b6d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c1560b0 0x55e94c1c0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:14.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.882+0000 7f73bb19c740 10 client.0 _ll_register_callbacks cb 0x7f73a00097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:14.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.882+0000 7f73b636c640 1 -- 172.21.15.29:0/3157407441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c1560b0 msgr2=0x55e94c1c0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:14.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.882+0000 7f73b636c640 1 --2- 172.21.15.29:0/3157407441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c1560b0 0x55e94c1c0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.882+0000 7f73b636c640 1 -- 172.21.15.29:0/3157407441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c1556f0 msgr2=0x55e94c1630f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:14.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.882+0000 7f73b636c640 1 --2- 172.21.15.29:0/3157407441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c1556f0 0x55e94c1630f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.882+0000 7f73b636c640 1 -- 172.21.15.29:0/3157407441 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e94c07f0b0 con 0x55e94c154520 2024-03-11T19:12:14.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.882+0000 7f73b6b6d640 1 --2- 172.21.15.29:0/3157407441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c1560b0 0x55e94c1c0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:12:14.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.882+0000 7f73b636c640 1 --2- 172.21.15.29:0/3157407441 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c154520 0x55e94c162bb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f73ac005c20 tx=0x7f73ac02c170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f548916809660dcc server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:14.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.882+0000 7f7399ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:12:14.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.883+0000 7f7399ffb640 1 -- 172.21.15.29:0/3157407441 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac003d70 con 0x55e94c154520 2024-03-11T19:12:14.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.883+0000 7f7399ffb640 1 -- 172.21.15.29:0/3157407441 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f73ac004dc0 con 0x55e94c154520 2024-03-11T19:12:14.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.883+0000 7f73bb19c740 1 -- 172.21.15.29:0/3157407441 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e94bdbc860 con 0x55e94c154520 2024-03-11T19:12:14.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.884+0000 7f7399ffb640 1 -- 172.21.15.29:0/3157407441 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73ac0d5240 con 0x55e94c154520 2024-03-11T19:12:14.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.884+0000 7f73bb19c740 10 client.25601 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:14.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.884+0000 7f73bb19c740 1 -- 172.21.15.29:0/3157407441 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e94c07d910 con 0x55e94c154520 2024-03-11T19:12:14.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.884+0000 7f7399ffb640 1 -- 172.21.15.29:0/3157407441 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55e94bdbc860 con 0x55e94c154520 2024-03-11T19:12:14.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.884+0000 7f7399ffb640 1 -- 172.21.15.29:0/3157407441 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f73ac0a2050 con 0x55e94c154520 2024-03-11T19:12:14.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.884+0000 7f73bb19c740 10 client.25601 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:14.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.884+0000 7f73bb19c740 10 client.25601 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:12:14.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.884+0000 7f73bb19c740 20 client.25601 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:14.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.884+0000 7f73bb19c740 1 --2- 172.21.15.29:0/3157407441 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e94c1c2d80 0x55e94c1c51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:14.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.884+0000 7f73bb19c740 4 client.25601 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:12:14.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.884+0000 7f73bb19c740 1 -- 172.21.15.29:0/3157407441 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55e94be0d410 con 0x55e94c1c2d80 2024-03-11T19:12:14.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.885+0000 7f73b5b6b640 1 --2- 172.21.15.29:0/3157407441 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e94c1c2d80 0x55e94c1c51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:14.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.885+0000 7f7399ffb640 10 client.25601 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:14.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.885+0000 7f73b5b6b640 1 --2- 172.21.15.29:0/3157407441 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e94c1c2d80 0x55e94c1c51a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd83c8d8b7e1e950 server_cookie=a6f8450ccf1c1392 in_seq=0 out_seq=0 2024-03-11T19:12:14.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.886+0000 7f7399ffb640 1 -- 172.21.15.29:0/3157407441 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e94be0d410 con 0x55e94c1c2d80 2024-03-11T19:12:14.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.887+0000 7f7399ffb640 10 client.25601 handle_command_reply: tid=0 2024-03-11T19:12:14.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.887+0000 7f73bb19c740 1 client.25601 shutdown 2024-03-11T19:12:14.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.887+0000 7f73bb19c740 1 -- 172.21.15.29:0/3157407441 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c154520 msgr2=0x55e94c162bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:14.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.887+0000 7f73bb19c740 1 --2- 172.21.15.29:0/3157407441 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c154520 0x55e94c162bb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f73ac005c20 tx=0x7f73ac02c170 comp rx=0 tx=0).stop 2024-03-11T19:12:14.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.887+0000 7f73bb19c740 1 -- 172.21.15.29:0/3157407441 shutdown_connections 2024-03-11T19:12:14.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.887+0000 7f73bb19c740 1 --2- 172.21.15.29:0/3157407441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e94c1560b0 0x55e94c1c0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.887+0000 7f73bb19c740 1 --2- 172.21.15.29:0/3157407441 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e94c1c2d80 0x55e94c1c51a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.887+0000 7f73bb19c740 1 --2- 172.21.15.29:0/3157407441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e94c1556f0 0x55e94c1630f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.887+0000 7f7399ffb640 0 client.25601 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:14.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.887+0000 7f73bb19c740 1 --2- 172.21.15.29:0/3157407441 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e94c154520 0x55e94c162bb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.887+0000 7f73bb19c740 1 -- 172.21.15.29:0/3157407441 >> 172.21.15.29:0/3157407441 conn(0x55e94c14f700 msgr2=0x55e94c159260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:14.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.887+0000 7f73bb19c740 1 -- 172.21.15.29:0/3157407441 shutdown_connections 2024-03-11T19:12:14.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.887+0000 7f73bb19c740 1 -- 172.21.15.29:0/3157407441 wait complete. 2024-03-11T19:12:14.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.887+0000 7f73bb19c740 20 client.25601 trim_cache size 0 max 16384 2024-03-11T19:12:14.894 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:12:14.894 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:12:14.894 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "c1ce4178-4642-4b30-883e-7f812db850fa", 2024-03-11T19:12:14.894 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:12:14.894 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:12:14.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.888+0000 7f73b85d0640 1 -- 172.21.15.29:0/810684633 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f739c0a1b00 msgr2=0x7f739c0a3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:14.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.888+0000 7f73b85d0640 1 --2- 172.21.15.29:0/810684633 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f739c0a1b00 0x7f739c0a3fc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f73a8005fd0 tx=0x7f73a8009500 comp rx=0 tx=0).stop 2024-03-11T19:12:14.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.888+0000 7f73b85d0640 1 -- 172.21.15.29:0/810684633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73b01033a0 msgr2=0x7f73b0199370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:14.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.888+0000 7f73b85d0640 1 --2- 172.21.15.29:0/810684633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73b01033a0 0x7f73b0199370 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f73ac02c7f0 tx=0x7f73ac004270 comp rx=0 tx=0).stop 2024-03-11T19:12:14.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.888+0000 7f73b85d0640 1 -- 172.21.15.29:0/810684633 shutdown_connections 2024-03-11T19:12:14.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.888+0000 7f73b85d0640 1 --2- 172.21.15.29:0/810684633 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73b0104f50 0x7f73b01a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.888+0000 7f73b85d0640 1 --2- 172.21.15.29:0/810684633 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73b0104590 0x7f73b01998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.888+0000 7f73b85d0640 1 --2- 172.21.15.29:0/810684633 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f739c0a1b00 0x7f739c0a3fc0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.888+0000 7f73b85d0640 1 --2- 172.21.15.29:0/810684633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73b01033a0 0x7f73b0199370 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:14.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.888+0000 7f73b85d0640 1 -- 172.21.15.29:0/810684633 >> 172.21.15.29:0/810684633 conn(0x7f73b00fe930 msgr2=0x7f73b0107960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:14.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.888+0000 7f73b85d0640 1 -- 172.21.15.29:0/810684633 shutdown_connections 2024-03-11T19:12:14.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:14.889+0000 7f73b85d0640 1 -- 172.21.15.29:0/810684633 wait complete. 2024-03-11T19:12:14.908 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:12:14.908 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:12:15.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.073+0000 7f329661c640 1 -- 172.21.15.29:0/1436118892 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3290104590 msgr2=0x7f3290104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:15.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.073+0000 7f329661c640 1 --2- 172.21.15.29:0/1436118892 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3290104590 0x7f3290104a10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3284009900 tx=0x7f328402bc40 comp rx=0 tx=0).stop 2024-03-11T19:12:15.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.073+0000 7f329661c640 1 -- 172.21.15.29:0/1436118892 shutdown_connections 2024-03-11T19:12:15.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.073+0000 7f329661c640 1 --2- 172.21.15.29:0/1436118892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3290104f50 0x7f329010b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.073+0000 7f329661c640 1 --2- 172.21.15.29:0/1436118892 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3290104590 0x7f3290104a10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.073+0000 7f329661c640 1 --2- 172.21.15.29:0/1436118892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32901033a0 0x7f32901037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.073+0000 7f329661c640 1 -- 172.21.15.29:0/1436118892 >> 172.21.15.29:0/1436118892 conn(0x7f32900fe930 msgr2=0x7f3290100d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:15.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.073+0000 7f329661c640 1 -- 172.21.15.29:0/1436118892 shutdown_connections 2024-03-11T19:12:15.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.073+0000 7f329661c640 1 -- 172.21.15.29:0/1436118892 wait complete. 2024-03-11T19:12:15.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.073+0000 7f329661c640 1 Processor -- start 2024-03-11T19:12:15.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.074+0000 7f329661c640 1 -- start start 2024-03-11T19:12:15.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.074+0000 7f329661c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32901033a0 0x7f329019d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:15.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.074+0000 7f329661c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3290104590 0x7f329019dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:15.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.074+0000 7f329661c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3290104f50 0x7f32901a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:15.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.074+0000 7f329661c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32901a5150 con 0x7f32901033a0 2024-03-11T19:12:15.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.074+0000 7f329661c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32901a52c0 con 0x7f3290104f50 2024-03-11T19:12:15.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.074+0000 7f329661c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32901a5480 con 0x7f3290104590 2024-03-11T19:12:15.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.074+0000 7f328ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32901033a0 0x7f329019d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:15.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.074+0000 7f328f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3290104590 0x7f329019dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:15.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.074+0000 7f3294bb9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3290104f50 0x7f32901a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:15.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.074+0000 7f328f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3290104590 0x7f329019dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:42176/0 (socket says 172.21.15.29:42176) 2024-03-11T19:12:15.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.074+0000 7f328ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32901033a0 0x7f329019d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44076/0 (socket says 172.21.15.29:44076) 2024-03-11T19:12:15.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.074+0000 7f328f7fe640 1 -- 172.21.15.29:0/595207181 learned_addr learned my addr 172.21.15.29:0/595207181 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:15.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.075+0000 7f3294bb9640 1 -- 172.21.15.29:0/595207181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3290104590 msgr2=0x7f329019dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:15.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.075+0000 7f3294bb9640 1 --2- 172.21.15.29:0/595207181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3290104590 0x7f329019dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.075+0000 7f3294bb9640 1 -- 172.21.15.29:0/595207181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32901033a0 msgr2=0x7f329019d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:15.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.075+0000 7f3294bb9640 1 --2- 172.21.15.29:0/595207181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32901033a0 0x7f329019d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.075+0000 7f3294bb9640 1 -- 172.21.15.29:0/595207181 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3288009840 con 0x7f3290104f50 2024-03-11T19:12:15.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.075+0000 7f3294bb9640 1 --2- 172.21.15.29:0/595207181 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3290104f50 0x7f32901a4c10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f328800ec50 tx=0x7f328800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:15.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.075+0000 7f328d7fa640 1 -- 172.21.15.29:0/595207181 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3288003b70 con 0x7f3290104f50 2024-03-11T19:12:15.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.076+0000 7f329661c640 1 -- 172.21.15.29:0/595207181 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3284009560 con 0x7f3290104f50 2024-03-11T19:12:15.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.076+0000 7f329661c640 1 -- 172.21.15.29:0/595207181 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32901a5ae0 con 0x7f3290104f50 2024-03-11T19:12:15.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.077+0000 7f328d7fa640 1 -- 172.21.15.29:0/595207181 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f328800ce70 con 0x7f3290104f50 2024-03-11T19:12:15.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.078+0000 7f328d7fa640 1 -- 172.21.15.29:0/595207181 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3288010670 con 0x7f3290104f50 2024-03-11T19:12:15.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.080+0000 7f32991f4740 1 Processor -- start 2024-03-11T19:12:15.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.080+0000 7f328d7fa640 1 -- 172.21.15.29:0/595207181 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3288004020 con 0x7f3290104f50 2024-03-11T19:12:15.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.080+0000 7f32991f4740 1 -- start start 2024-03-11T19:12:15.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.081+0000 7f32991f4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4c92e0 0x5565eb4c9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:15.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.081+0000 7f32991f4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c9cd0 0x5565eb4ce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:15.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.081+0000 7f32991f4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4cee40 0x5565eb4d12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:15.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.081+0000 7f32991f4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565eb439280 con 0x5565eb4c9cd0 2024-03-11T19:12:15.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.081+0000 7f32991f4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565eb4367b0 con 0x5565eb4cee40 2024-03-11T19:12:15.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.081+0000 7f32991f4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565eb431f90 con 0x5565eb4c92e0 2024-03-11T19:12:15.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.081+0000 7f328ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c9cd0 0x5565eb4ce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:15.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.081+0000 7f328ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c9cd0 0x5565eb4ce7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44086/0 (socket says 172.21.15.29:44086) 2024-03-11T19:12:15.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.081+0000 7f328ffff640 1 -- 172.21.15.29:0/1941841341 learned_addr learned my addr 172.21.15.29:0/1941841341 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:15.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.081+0000 7f328f7fe640 1 --2- 172.21.15.29:0/1941841341 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4cee40 0x5565eb4d12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:15.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.081+0000 7f328f7fe640 1 --2- 172.21.15.29:0/1941841341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4c92e0 0x5565eb4c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:15.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.082+0000 7f328d7fa640 1 --2- 172.21.15.29:0/595207181 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f32780a1990 0x7f32780a3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:15.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.082+0000 7f328ffff640 1 -- 172.21.15.29:0/1941841341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4c92e0 msgr2=0x5565eb4c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:15.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.082+0000 7f328ffff640 1 --2- 172.21.15.29:0/1941841341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4c92e0 0x5565eb4c9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.082+0000 7f328ffff640 1 -- 172.21.15.29:0/1941841341 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4cee40 msgr2=0x5565eb4d12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:15.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.082+0000 7f328ffff640 1 --2- 172.21.15.29:0/1941841341 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4cee40 0x5565eb4d12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.082+0000 7f328ffff640 1 -- 172.21.15.29:0/1941841341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565eb3f0570 con 0x5565eb4c9cd0 2024-03-11T19:12:15.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.082+0000 7f328f7fe640 1 --2- 172.21.15.29:0/1941841341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4c92e0 0x5565eb4c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:12:15.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.082+0000 7f328d7fa640 1 -- 172.21.15.29:0/595207181 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f3288014070 con 0x7f3290104f50 2024-03-11T19:12:15.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.082+0000 7f3294bb9640 1 --2- 172.21.15.29:0/595207181 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f32780a1990 0x7f32780a3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:15.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.082+0000 7f328ffff640 1 --2- 172.21.15.29:0/1941841341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c9cd0 0x5565eb4ce7d0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f328000ac70 tx=0x7f328000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=208986074756f7b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:15.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.082+0000 7f32767fc640 1 -- 172.21.15.29:0/1941841341 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3280005220 con 0x5565eb4c9cd0 2024-03-11T19:12:15.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.082+0000 7f32767fc640 1 -- 172.21.15.29:0/1941841341 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f32800047a0 con 0x5565eb4c9cd0 2024-03-11T19:12:15.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.082+0000 7f32767fc640 1 -- 172.21.15.29:0/1941841341 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32800187a0 con 0x5565eb4c9cd0 2024-03-11T19:12:15.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.083+0000 7f3294bb9640 1 --2- 172.21.15.29:0/595207181 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f32780a1990 0x7f32780a3e50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f328800cba0 tx=0x7f328800ef50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:15.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.083+0000 7f32991f4740 1 -- 172.21.15.29:0/1941841341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c9cd0 msgr2=0x5565eb4ce7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:15.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.083+0000 7f32991f4740 1 --2- 172.21.15.29:0/1941841341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c9cd0 0x5565eb4ce7d0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f328000ac70 tx=0x7f328000f790 comp rx=0 tx=0).stop 2024-03-11T19:12:15.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.084+0000 7f32991f4740 1 -- 172.21.15.29:0/1941841341 shutdown_connections 2024-03-11T19:12:15.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.084+0000 7f32991f4740 1 --2- 172.21.15.29:0/1941841341 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4cee40 0x5565eb4d12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.084+0000 7f32991f4740 1 --2- 172.21.15.29:0/1941841341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c9cd0 0x5565eb4ce7d0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.084+0000 7f32991f4740 1 --2- 172.21.15.29:0/1941841341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4c92e0 0x5565eb4c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.084+0000 7f32991f4740 1 -- 172.21.15.29:0/1941841341 >> 172.21.15.29:0/1941841341 conn(0x5565eb4c4850 msgr2=0x5565eb4c4c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:15.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.084+0000 7f32991f4740 1 -- 172.21.15.29:0/1941841341 shutdown_connections 2024-03-11T19:12:15.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.084+0000 7f32991f4740 1 -- 172.21.15.29:0/1941841341 wait complete. 2024-03-11T19:12:15.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.084+0000 7f32991f4740 1 -- start start 2024-03-11T19:12:15.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.085+0000 7f32991f4740 1 Processor -- start 2024-03-11T19:12:15.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.085+0000 7f32991f4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4c92e0 0x5565eb4d3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:15.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.085+0000 7f32991f4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c9cd0 0x5565eb4d37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:15.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.085+0000 7f32991f4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4cee40 0x5565eb4da800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:15.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.085+0000 7f32991f4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565eb105430 con 0x5565eb4c9cd0 2024-03-11T19:12:15.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.085+0000 7f32991f4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565eb0c1e00 con 0x5565eb4c92e0 2024-03-11T19:12:15.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.086+0000 7f328ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c9cd0 0x5565eb4d37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:15.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.086+0000 7f328ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c9cd0 0x5565eb4d37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44088/0 (socket says 172.21.15.29:44088) 2024-03-11T19:12:15.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.086+0000 7f328ffff640 1 -- 172.21.15.29:0/3927110079 learned_addr learned my addr 172.21.15.29:0/3927110079 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:15.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.086+0000 7f32991f4740 1 -- 172.21.15.29:0/3927110079 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565eb3b6fd0 con 0x5565eb4cee40 2024-03-11T19:12:15.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.086+0000 7f328f7fe640 1 --2- 172.21.15.29:0/3927110079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4c92e0 0x5565eb4d3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:15.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.086+0000 7f32991f4740 10 client.0 _ll_register_callbacks cb 0x5565eb40cfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:15.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.086+0000 7f328f7fe640 1 --2- 172.21.15.29:0/3927110079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4cee40 0x5565eb4da800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:15.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.086+0000 7f328f7fe640 1 -- 172.21.15.29:0/3927110079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4cee40 msgr2=0x5565eb4da800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:15.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.086+0000 7f328f7fe640 1 --2- 172.21.15.29:0/3927110079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4cee40 0x5565eb4da800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.086+0000 7f328f7fe640 1 -- 172.21.15.29:0/3927110079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c9cd0 msgr2=0x5565eb4d37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:15.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.086+0000 7f328f7fe640 1 --2- 172.21.15.29:0/3927110079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c9cd0 0x5565eb4d37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.086+0000 7f328f7fe640 1 -- 172.21.15.29:0/3927110079 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328000a8a0 con 0x5565eb4c92e0 2024-03-11T19:12:15.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.086+0000 7f328ffff640 1 --2- 172.21.15.29:0/3927110079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c9cd0 0x5565eb4d37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:15.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.086+0000 7f328f7fe640 1 --2- 172.21.15.29:0/3927110079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4c92e0 0x5565eb4d3280 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f328402c750 tx=0x7f3284004e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=831fdc6f3507a95 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:15.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.086+0000 7f326f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:12:15.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.087+0000 7f326f7fe640 1 -- 172.21.15.29:0/3927110079 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3284005220 con 0x5565eb4c92e0 2024-03-11T19:12:15.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.087+0000 7f326f7fe640 1 -- 172.21.15.29:0/3927110079 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f32840048d0 con 0x5565eb4c92e0 2024-03-11T19:12:15.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.087+0000 7f32991f4740 1 -- 172.21.15.29:0/3927110079 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5565eb1317e0 con 0x5565eb4c92e0 2024-03-11T19:12:15.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.087+0000 7f326f7fe640 1 -- 172.21.15.29:0/3927110079 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3284004a90 con 0x5565eb4c92e0 2024-03-11T19:12:15.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.087+0000 7f32991f4740 10 client.25613 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:15.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.087+0000 7f32991f4740 1 -- 172.21.15.29:0/3927110079 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5565eb430610 con 0x5565eb4c92e0 2024-03-11T19:12:15.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.088+0000 7f326f7fe640 1 -- 172.21.15.29:0/3927110079 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5565eb1317e0 con 0x5565eb4c92e0 2024-03-11T19:12:15.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.088+0000 7f326f7fe640 1 -- 172.21.15.29:0/3927110079 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3284003b10 con 0x5565eb4c92e0 2024-03-11T19:12:15.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.088+0000 7f32991f4740 10 client.25613 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:15.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.088+0000 7f32991f4740 10 client.25613 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:12:15.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.088+0000 7f32991f4740 20 client.25613 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:15.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.088+0000 7f32991f4740 1 --2- 172.21.15.29:0/3927110079 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5565eb4df3a0 0x5565eb4e17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:15.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.088+0000 7f32991f4740 4 client.25613 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:12:15.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.088+0000 7f32991f4740 1 -- 172.21.15.29:0/3927110079 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5565eb16e720 con 0x5565eb4df3a0 2024-03-11T19:12:15.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.088+0000 7f328ffff640 1 --2- 172.21.15.29:0/3927110079 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5565eb4df3a0 0x5565eb4e17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:15.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.089+0000 7f326f7fe640 10 client.25613 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:15.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.089+0000 7f328ffff640 1 --2- 172.21.15.29:0/3927110079 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5565eb4df3a0 0x5565eb4e17b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2e3599a4c955ef8 server_cookie=23ceed343d32d50f in_seq=0 out_seq=0 2024-03-11T19:12:15.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.089+0000 7f326f7fe640 1 -- 172.21.15.29:0/3927110079 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5565eb16e720 con 0x5565eb4df3a0 2024-03-11T19:12:15.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.089+0000 7f326f7fe640 10 client.25613 handle_command_reply: tid=0 2024-03-11T19:12:15.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.089+0000 7f32991f4740 1 client.25613 shutdown 2024-03-11T19:12:15.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.089+0000 7f32991f4740 1 -- 172.21.15.29:0/3927110079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4c92e0 msgr2=0x5565eb4d3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:15.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.089+0000 7f32991f4740 1 --2- 172.21.15.29:0/3927110079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4c92e0 0x5565eb4d3280 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f328402c750 tx=0x7f3284004e20 comp rx=0 tx=0).stop 2024-03-11T19:12:15.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.090+0000 7f32991f4740 1 -- 172.21.15.29:0/3927110079 shutdown_connections 2024-03-11T19:12:15.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.090+0000 7f32991f4740 1 --2- 172.21.15.29:0/3927110079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4cee40 0x5565eb4da800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.090+0000 7f32991f4740 1 --2- 172.21.15.29:0/3927110079 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5565eb4df3a0 0x5565eb4e17b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.090+0000 7f32991f4740 1 --2- 172.21.15.29:0/3927110079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c9cd0 0x5565eb4d37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.090+0000 7f326f7fe640 0 client.25613 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:15.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.090+0000 7f32991f4740 1 --2- 172.21.15.29:0/3927110079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4c92e0 0x5565eb4d3280 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.090+0000 7f32991f4740 1 -- 172.21.15.29:0/3927110079 >> 172.21.15.29:0/3927110079 conn(0x5565eb4c4850 msgr2=0x5565eb4c3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:15.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.090+0000 7f32991f4740 1 -- 172.21.15.29:0/3927110079 shutdown_connections 2024-03-11T19:12:15.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.090+0000 7f32991f4740 1 -- 172.21.15.29:0/3927110079 wait complete. 2024-03-11T19:12:15.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.090+0000 7f32991f4740 20 client.25613 trim_cache size 0 max 16384 2024-03-11T19:12:15.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.116+0000 7f32991f4740 1 Processor -- start 2024-03-11T19:12:15.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.116+0000 7f32991f4740 1 -- start start 2024-03-11T19:12:15.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.116+0000 7f32991f4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4c92e0 0x5565eb4cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:15.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.116+0000 7f32991f4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4c9cd0 0x5565eb4da790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:15.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.116+0000 7f32991f4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4cee40 0x5565eb4cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:15.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.116+0000 7f32991f4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565eb4dda60 con 0x5565eb4cee40 2024-03-11T19:12:15.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.116+0000 7f32991f4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565eb16e750 con 0x5565eb4c9cd0 2024-03-11T19:12:15.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.116+0000 7f32991f4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565eb4c02c0 con 0x5565eb4c92e0 2024-03-11T19:12:15.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.117+0000 7f328f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4c92e0 0x5565eb4cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:15.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.117+0000 7f328f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4c92e0 0x5565eb4cafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:42188/0 (socket says 172.21.15.29:42188) 2024-03-11T19:12:15.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.117+0000 7f328f7fe640 1 -- 172.21.15.29:0/1011236153 learned_addr learned my addr 172.21.15.29:0/1011236153 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:15.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.117+0000 7f328ffff640 1 --2- 172.21.15.29:0/1011236153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4c9cd0 0x5565eb4da790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:15.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.117+0000 7f328f7fe640 1 --2- 172.21.15.29:0/1011236153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4cee40 0x5565eb4cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:15.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.117+0000 7f328ffff640 1 -- 172.21.15.29:0/1011236153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4c92e0 msgr2=0x5565eb4cafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:15.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.117+0000 7f328ffff640 1 --2- 172.21.15.29:0/1011236153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4c92e0 0x5565eb4cafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.117+0000 7f328ffff640 1 -- 172.21.15.29:0/1011236153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4cee40 msgr2=0x5565eb4cb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:15.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.117+0000 7f328ffff640 1 --2- 172.21.15.29:0/1011236153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4cee40 0x5565eb4cb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.117+0000 7f328ffff640 1 -- 172.21.15.29:0/1011236153 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565eb3ccac0 con 0x5565eb4c9cd0 2024-03-11T19:12:15.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.117+0000 7f328ffff640 1 --2- 172.21.15.29:0/1011236153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4c9cd0 0x5565eb4da790 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3280000c00 tx=0x7f3280004e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55d9de514dbf56b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:15.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.118+0000 7f3275ffb640 1 -- 172.21.15.29:0/1011236153 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3280029070 con 0x5565eb4c9cd0 2024-03-11T19:12:15.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.118+0000 7f3275ffb640 1 -- 172.21.15.29:0/1011236153 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3280004c40 con 0x5565eb4c9cd0 2024-03-11T19:12:15.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.118+0000 7f3275ffb640 1 -- 172.21.15.29:0/1011236153 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3280005770 con 0x5565eb4c9cd0 2024-03-11T19:12:15.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.119+0000 7f32991f4740 1 -- 172.21.15.29:0/1011236153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4c9cd0 msgr2=0x5565eb4da790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:15.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.119+0000 7f32991f4740 1 --2- 172.21.15.29:0/1011236153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4c9cd0 0x5565eb4da790 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3280000c00 tx=0x7f3280004e70 comp rx=0 tx=0).stop 2024-03-11T19:12:15.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.119+0000 7f32991f4740 1 -- 172.21.15.29:0/1011236153 shutdown_connections 2024-03-11T19:12:15.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.119+0000 7f32991f4740 1 --2- 172.21.15.29:0/1011236153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4cee40 0x5565eb4cb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.119+0000 7f32991f4740 1 --2- 172.21.15.29:0/1011236153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4c9cd0 0x5565eb4da790 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.120+0000 7f32991f4740 1 --2- 172.21.15.29:0/1011236153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4c92e0 0x5565eb4cafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.120+0000 7f32991f4740 1 -- 172.21.15.29:0/1011236153 >> 172.21.15.29:0/1011236153 conn(0x5565eb4c4850 msgr2=0x5565eb4c4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:15.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.120+0000 7f32991f4740 1 -- 172.21.15.29:0/1011236153 shutdown_connections 2024-03-11T19:12:15.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.120+0000 7f32991f4740 1 -- 172.21.15.29:0/1011236153 wait complete. 2024-03-11T19:12:15.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.121+0000 7f32991f4740 1 -- start start 2024-03-11T19:12:15.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.121+0000 7f32991f4740 1 Processor -- start 2024-03-11T19:12:15.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.122+0000 7f32991f4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c92e0 0x5565eb4da090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:15.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.122+0000 7f32991f4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4c9cd0 0x5565eb4da5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:15.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.122+0000 7f32991f4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4cee40 0x5565eb4d86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:15.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.122+0000 7f32991f4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565eb4d8d30 con 0x5565eb4c92e0 2024-03-11T19:12:15.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.122+0000 7f328f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c92e0 0x5565eb4da090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:15.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.122+0000 7f32991f4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565eb4d8ea0 con 0x5565eb4c9cd0 2024-03-11T19:12:15.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.122+0000 7f32991f4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565eb4d9060 con 0x5565eb4cee40 2024-03-11T19:12:15.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.122+0000 7f328f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c92e0 0x5565eb4da090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44106/0 (socket says 172.21.15.29:44106) 2024-03-11T19:12:15.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.122+0000 7f328f7fe640 1 -- 172.21.15.29:0/1913778099 learned_addr learned my addr 172.21.15.29:0/1913778099 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:15.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.122+0000 7f32991f4740 10 client.0 _ll_register_callbacks cb 0x7f328000a690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:15.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.122+0000 7f328ffff640 1 --2- 172.21.15.29:0/1913778099 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4c9cd0 0x5565eb4da5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.122+0000 7f328f7fe640 1 --2- 172.21.15.29:0/1913778099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4cee40 0x5565eb4d86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.123+0000 7f328ffff640 1 -- 172.21.15.29:0/1913778099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4cee40 msgr2=0x5565eb4d86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.123+0000 7f328ffff640 1 --2- 172.21.15.29:0/1913778099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4cee40 0x5565eb4d86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.123+0000 7f328ffff640 1 -- 172.21.15.29:0/1913778099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c92e0 msgr2=0x5565eb4da090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.123+0000 7f328ffff640 1 --2- 172.21.15.29:0/1913778099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c92e0 0x5565eb4da090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.123+0000 7f328ffff640 1 -- 172.21.15.29:0/1913778099 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5565eb3f0570 con 0x5565eb4c9cd0 2024-03-11T19:12:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.123+0000 7f328ffff640 1 --2- 172.21.15.29:0/1913778099 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4c9cd0 0x5565eb4da5d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x5565eb4c9c10 tx=0x7f3280005ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44a0dd1a27a791c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.123+0000 7f326f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:12:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.123+0000 7f326f7fe640 1 -- 172.21.15.29:0/1913778099 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3280029040 con 0x5565eb4c9cd0 2024-03-11T19:12:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.124+0000 7f326f7fe640 1 -- 172.21.15.29:0/1913778099 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f3280005200 con 0x5565eb4c9cd0 2024-03-11T19:12:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.124+0000 7f32991f4740 1 -- 172.21.15.29:0/1913778099 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f32800047a0 con 0x5565eb4c9cd0 2024-03-11T19:12:15.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.124+0000 7f326f7fe640 1 -- 172.21.15.29:0/1913778099 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32800035f0 con 0x5565eb4c9cd0 2024-03-11T19:12:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.124+0000 7f32991f4740 10 client.25619 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.124+0000 7f32991f4740 1 -- 172.21.15.29:0/1913778099 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5565eb3f26a0 con 0x5565eb4c9cd0 2024-03-11T19:12:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.125+0000 7f326f7fe640 1 -- 172.21.15.29:0/1913778099 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f328003d030 con 0x5565eb4c9cd0 2024-03-11T19:12:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.125+0000 7f326f7fe640 1 -- 172.21.15.29:0/1913778099 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f32800376e0 con 0x5565eb4c9cd0 2024-03-11T19:12:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.125+0000 7f32991f4740 10 client.25619 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.125+0000 7f32991f4740 10 client.25619 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:12:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.125+0000 7f32991f4740 20 client.25619 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.125+0000 7f32991f4740 1 --2- 172.21.15.29:0/1913778099 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5565eb4df3a0 0x5565eb538950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.125+0000 7f32991f4740 4 client.25619 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:12:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.125+0000 7f32991f4740 1 -- 172.21.15.29:0/1913778099 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f32800047a0 con 0x5565eb4df3a0 2024-03-11T19:12:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.125+0000 7f328f7fe640 1 --2- 172.21.15.29:0/1913778099 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5565eb4df3a0 0x5565eb538950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.126+0000 7f326f7fe640 10 client.25619 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.126+0000 7f328f7fe640 1 --2- 172.21.15.29:0/1913778099 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5565eb4df3a0 0x5565eb538950 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e26b7f46754a0f0 server_cookie=a8de2e7602028a84 in_seq=0 out_seq=0 2024-03-11T19:12:15.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.126+0000 7f326f7fe640 1 -- 172.21.15.29:0/1913778099 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7f32800047a0 con 0x5565eb4df3a0 2024-03-11T19:12:15.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.126+0000 7f326f7fe640 10 client.25619 handle_command_reply: tid=0 2024-03-11T19:12:15.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.126+0000 7f32991f4740 1 client.25619 shutdown 2024-03-11T19:12:15.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.126+0000 7f32991f4740 1 -- 172.21.15.29:0/1913778099 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4c9cd0 msgr2=0x5565eb4da5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:15.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.126+0000 7f32991f4740 1 --2- 172.21.15.29:0/1913778099 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4c9cd0 0x5565eb4da5d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x5565eb4c9c10 tx=0x7f3280005ce0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.126+0000 7f32991f4740 1 -- 172.21.15.29:0/1913778099 shutdown_connections 2024-03-11T19:12:15.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.126+0000 7f32991f4740 1 --2- 172.21.15.29:0/1913778099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5565eb4cee40 0x5565eb4d86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.126+0000 7f32991f4740 1 --2- 172.21.15.29:0/1913778099 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565eb4c9cd0 0x5565eb4da5d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.126+0000 7f32991f4740 1 --2- 172.21.15.29:0/1913778099 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5565eb4df3a0 0x5565eb538950 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.126+0000 7f326f7fe640 0 client.25619 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:15.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.126+0000 7f32991f4740 1 --2- 172.21.15.29:0/1913778099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5565eb4c92e0 0x5565eb4da090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.126+0000 7f32991f4740 1 -- 172.21.15.29:0/1913778099 >> 172.21.15.29:0/1913778099 conn(0x5565eb4c4850 msgr2=0x5565eb4cafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:15.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.127+0000 7f32991f4740 1 -- 172.21.15.29:0/1913778099 shutdown_connections 2024-03-11T19:12:15.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.127+0000 7f32991f4740 1 -- 172.21.15.29:0/1913778099 wait complete. 2024-03-11T19:12:15.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.127+0000 7f32991f4740 20 client.25619 trim_cache size 0 max 16384 2024-03-11T19:12:15.133 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:12:15.133 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:12:15.133 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:12:15.133 INFO:teuthology.orchestra.run.smithi029.stdout: "c1ce4178-4642-4b30-883e-7f812db850fa": { 2024-03-11T19:12:15.133 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:12:15.133 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "c1ce4178-4642-4b30-883e-7f812db850fa", 2024-03-11T19:12:15.133 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:12:15.133 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:12:15.133 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:12:15.133 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:12:15.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.128+0000 7f329661c640 1 -- 172.21.15.29:0/595207181 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f32780a1990 msgr2=0x7f32780a3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:15.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.128+0000 7f329661c640 1 --2- 172.21.15.29:0/595207181 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f32780a1990 0x7f32780a3e50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f328800cba0 tx=0x7f328800ef50 comp rx=0 tx=0).stop 2024-03-11T19:12:15.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.128+0000 7f329661c640 1 -- 172.21.15.29:0/595207181 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3290104f50 msgr2=0x7f32901a4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:15.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.128+0000 7f329661c640 1 --2- 172.21.15.29:0/595207181 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3290104f50 0x7f32901a4c10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f328800ec50 tx=0x7f328800c6a0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.128+0000 7f329661c640 1 -- 172.21.15.29:0/595207181 shutdown_connections 2024-03-11T19:12:15.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.128+0000 7f329661c640 1 --2- 172.21.15.29:0/595207181 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3290104f50 0x7f32901a4c10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.128+0000 7f329661c640 1 --2- 172.21.15.29:0/595207181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3290104590 0x7f329019dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.128+0000 7f329661c640 1 --2- 172.21.15.29:0/595207181 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f32780a1990 0x7f32780a3e50 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.128+0000 7f329661c640 1 --2- 172.21.15.29:0/595207181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32901033a0 0x7f329019d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:15.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.128+0000 7f329661c640 1 -- 172.21.15.29:0/595207181 >> 172.21.15.29:0/595207181 conn(0x7f32900fe930 msgr2=0x7f32900775d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:15.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.128+0000 7f329661c640 1 -- 172.21.15.29:0/595207181 shutdown_connections 2024-03-11T19:12:15.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:15.128+0000 7f329661c640 1 -- 172.21.15.29:0/595207181 wait complete. 2024-03-11T19:12:15.148 INFO:tasks.cephfs.filesystem:scrub status for tag:c1ce4178-4642-4b30-883e-7f812db850fa - {'path': '/', 'tag': 'c1ce4178-4642-4b30-883e-7f812db850fa', 'options': 'recursive,force'} 2024-03-11T19:12:15.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:15 smithi190 ceph-mon[36292]: pgmap v424: 97 pgs: 97 active+clean; 9.8 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 173 MiB/s wr, 134 op/s 2024-03-11T19:12:15.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:15 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:12:15.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:15 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:12:15.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:15 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:12:15.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:15 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:12:15.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:15 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:12:15.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:15 smithi170 ceph-mon[35690]: pgmap v424: 97 pgs: 97 active+clean; 9.8 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 173 MiB/s wr, 134 op/s 2024-03-11T19:12:15.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:15 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:12:15.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:15 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:12:15.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:15 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:12:15.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:15 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:12:15.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:15 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:12:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:15 smithi029 ceph-mon[32439]: pgmap v424: 97 pgs: 97 active+clean; 9.8 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 173 MiB/s wr, 134 op/s 2024-03-11T19:12:15.937 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:15 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:12:15.937 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:15 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:12:15.937 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:15 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:12:15.937 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:15 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:12:15.937 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:15 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:12:16.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:16 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:12:16.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:16 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:12:16.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:16 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:12:16.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:16 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:12:16.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:16 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:12:16.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:16 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:12:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:16 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:12:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:16 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:12:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:16 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:12:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:16 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:12:16.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:16 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:12:16.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:16 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:12:16.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:16 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:12:16.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:16 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:12:16.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:16 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:12:17.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:17 smithi190 ceph-mon[36292]: pgmap v425: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 165 MiB/s wr, 265 op/s 2024-03-11T19:12:17.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:17 smithi170 ceph-mon[35690]: pgmap v425: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 165 MiB/s wr, 265 op/s 2024-03-11T19:12:17.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:17 smithi029 ceph-mon[32439]: pgmap v425: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 165 MiB/s wr, 265 op/s 2024-03-11T19:12:19.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:19 smithi190 ceph-mon[36292]: pgmap v426: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 137 MiB/s wr, 362 op/s 2024-03-11T19:12:19.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:19 smithi170 ceph-mon[35690]: pgmap v426: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 137 MiB/s wr, 362 op/s 2024-03-11T19:12:19.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:19 smithi029 ceph-mon[32439]: pgmap v426: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 137 MiB/s wr, 362 op/s 2024-03-11T19:12:21.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:21 smithi190 ceph-mon[36292]: pgmap v427: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 134 MiB/s wr, 589 op/s 2024-03-11T19:12:21.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:21 smithi170 ceph-mon[35690]: pgmap v427: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 134 MiB/s wr, 589 op/s 2024-03-11T19:12:21.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:21 smithi029 ceph-mon[32439]: pgmap v427: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 134 MiB/s wr, 589 op/s 2024-03-11T19:12:23.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:23 smithi190 ceph-mon[36292]: pgmap v428: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 108 MiB/s wr, 757 op/s 2024-03-11T19:12:23.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:23 smithi170 ceph-mon[35690]: pgmap v428: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 108 MiB/s wr, 757 op/s 2024-03-11T19:12:23.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:23 smithi029 ceph-mon[32439]: pgmap v428: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 108 MiB/s wr, 757 op/s 2024-03-11T19:12:25.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:25 smithi190 ceph-mon[36292]: pgmap v429: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 101 MiB/s wr, 974 op/s 2024-03-11T19:12:25.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:25 smithi170 ceph-mon[35690]: pgmap v429: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 101 MiB/s wr, 974 op/s 2024-03-11T19:12:25.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:25 smithi029 ceph-mon[32439]: pgmap v429: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 101 MiB/s wr, 974 op/s 2024-03-11T19:12:27.708 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:27 smithi029 ceph-mon[32439]: pgmap v430: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 90 MiB/s wr, 1.19k op/s 2024-03-11T19:12:27.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:27 smithi190 ceph-mon[36292]: pgmap v430: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 90 MiB/s wr, 1.19k op/s 2024-03-11T19:12:27.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:27 smithi170 ceph-mon[35690]: pgmap v430: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 90 MiB/s wr, 1.19k op/s 2024-03-11T19:12:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:29 smithi190 ceph-mon[36292]: pgmap v431: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-03-11T19:12:29.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:29 smithi170 ceph-mon[35690]: pgmap v431: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-03-11T19:12:29.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:29 smithi029 ceph-mon[32439]: pgmap v431: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-03-11T19:12:31.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:31 smithi190 ceph-mon[36292]: pgmap v432: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-11T19:12:31.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:31 smithi170 ceph-mon[35690]: pgmap v432: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-11T19:12:31.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:31 smithi029 ceph-mon[32439]: pgmap v432: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-11T19:12:33.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:33 smithi190 ceph-mon[36292]: pgmap v433: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-11T19:12:33.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:33 smithi170 ceph-mon[35690]: pgmap v433: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-11T19:12:33.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:33 smithi029 ceph-mon[32439]: pgmap v433: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-11T19:12:35.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:35 smithi190 ceph-mon[36292]: pgmap v434: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T19:12:35.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:35 smithi170 ceph-mon[35690]: pgmap v434: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T19:12:35.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:35 smithi029 ceph-mon[32439]: pgmap v434: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T19:12:37.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:37 smithi190 ceph-mon[36292]: pgmap v435: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T19:12:37.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:37 smithi170 ceph-mon[35690]: pgmap v435: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T19:12:37.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:37 smithi029 ceph-mon[32439]: pgmap v435: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T19:12:39.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:39 smithi190 ceph-mon[36292]: pgmap v436: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:12:39.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:39 smithi170 ceph-mon[35690]: pgmap v436: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:12:39.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:39 smithi029 ceph-mon[32439]: pgmap v436: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:12:41.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:41 smithi190 ceph-mon[36292]: pgmap v437: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-11T19:12:41.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:41 smithi170 ceph-mon[35690]: pgmap v437: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-11T19:12:41.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:41 smithi029 ceph-mon[32439]: pgmap v437: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-11T19:12:43.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:43 smithi190 ceph-mon[36292]: pgmap v438: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T19:12:43.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:43 smithi170 ceph-mon[35690]: pgmap v438: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T19:12:43.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:43 smithi029 ceph-mon[32439]: pgmap v438: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T19:12:45.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:12:45.149 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:12:45.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.312+0000 7f94b3a15640 1 -- 172.21.15.29:0/1823039757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94ac1077c0 msgr2=0x7f94ac107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:45.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.312+0000 7f94b3a15640 1 --2- 172.21.15.29:0/1823039757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94ac1077c0 0x7f94ac107be0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f94a8009860 tx=0x7f94a802bb70 comp rx=0 tx=0).stop 2024-03-11T19:12:45.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.312+0000 7f94b3a15640 1 -- 172.21.15.29:0/1823039757 shutdown_connections 2024-03-11T19:12:45.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.312+0000 7f94b3a15640 1 --2- 172.21.15.29:0/1823039757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94ac109370 0x7f94ac10fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.312+0000 7f94b3a15640 1 --2- 172.21.15.29:0/1823039757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94ac1089b0 0x7f94ac108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.312+0000 7f94b3a15640 1 --2- 172.21.15.29:0/1823039757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94ac1077c0 0x7f94ac107be0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.312+0000 7f94b3a15640 1 -- 172.21.15.29:0/1823039757 >> 172.21.15.29:0/1823039757 conn(0x7f94ac076770 msgr2=0x7f94ac078b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:45.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.313+0000 7f94b3a15640 1 -- 172.21.15.29:0/1823039757 shutdown_connections 2024-03-11T19:12:45.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.313+0000 7f94b3a15640 1 -- 172.21.15.29:0/1823039757 wait complete. 2024-03-11T19:12:45.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.313+0000 7f94b3a15640 1 Processor -- start 2024-03-11T19:12:45.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.314+0000 7f94b3a15640 1 -- start start 2024-03-11T19:12:45.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.314+0000 7f94b3a15640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94ac1077c0 0x7f94ac19d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.314+0000 7f94b3a15640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94ac1089b0 0x7f94ac19dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.314+0000 7f94b3a15640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94ac109370 0x7f94ac1a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.314+0000 7f94b3a15640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac1a5240 con 0x7f94ac1089b0 2024-03-11T19:12:45.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.314+0000 7f94b3a15640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac1a53b0 con 0x7f94ac109370 2024-03-11T19:12:45.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.314+0000 7f94b3a15640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac1a5570 con 0x7f94ac1077c0 2024-03-11T19:12:45.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.314+0000 7f94b1fb2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94ac109370 0x7f94ac1a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.314+0000 7f94b1fb2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94ac109370 0x7f94ac1a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49166/0 (socket says 172.21.15.29:49166) 2024-03-11T19:12:45.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.314+0000 7f94b1fb2640 1 -- 172.21.15.29:0/347660821 learned_addr learned my addr 172.21.15.29:0/347660821 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:45.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.314+0000 7f94b0fb0640 1 --2- 172.21.15.29:0/347660821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94ac1089b0 0x7f94ac19dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.314+0000 7f94b17b1640 1 --2- 172.21.15.29:0/347660821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94ac1077c0 0x7f94ac19d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.314+0000 7f94b1fb2640 1 -- 172.21.15.29:0/347660821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94ac1077c0 msgr2=0x7f94ac19d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:45.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.314+0000 7f94b1fb2640 1 --2- 172.21.15.29:0/347660821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94ac1077c0 0x7f94ac19d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.314+0000 7f94b1fb2640 1 -- 172.21.15.29:0/347660821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94ac1089b0 msgr2=0x7f94ac19dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:45.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.315+0000 7f94b1fb2640 1 --2- 172.21.15.29:0/347660821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94ac1089b0 0x7f94ac19dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.315+0000 7f94b1fb2640 1 -- 172.21.15.29:0/347660821 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f949c009560 con 0x7f94ac109370 2024-03-11T19:12:45.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.315+0000 7f94b0fb0640 1 --2- 172.21.15.29:0/347660821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94ac1089b0 0x7f94ac19dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:12:45.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.315+0000 7f94b1fb2640 1 --2- 172.21.15.29:0/347660821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94ac109370 0x7f94ac1a4d00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f94a400eaa0 tx=0x7f94a400ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:45.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.315+0000 7f94a27fc640 1 -- 172.21.15.29:0/347660821 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a400cdb0 con 0x7f94ac109370 2024-03-11T19:12:45.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.315+0000 7f94b3a15640 1 -- 172.21.15.29:0/347660821 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a8009490 con 0x7f94ac109370 2024-03-11T19:12:45.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.315+0000 7f94b3a15640 1 -- 172.21.15.29:0/347660821 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ac1a5bd0 con 0x7f94ac109370 2024-03-11T19:12:45.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.315+0000 7f94a27fc640 1 -- 172.21.15.29:0/347660821 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f94a4005020 con 0x7f94ac109370 2024-03-11T19:12:45.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.315+0000 7f94a27fc640 1 -- 172.21.15.29:0/347660821 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a40116d0 con 0x7f94ac109370 2024-03-11T19:12:45.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.318+0000 7f94a27fc640 1 -- 172.21.15.29:0/347660821 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f94a4011830 con 0x7f94ac109370 2024-03-11T19:12:45.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.319+0000 7f94b65ef740 1 Processor -- start 2024-03-11T19:12:45.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.319+0000 7f94b65ef740 1 -- start start 2024-03-11T19:12:45.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.319+0000 7f94b65ef740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc452d0 0x5593ebc456f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.319+0000 7f94b65ef740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc45cc0 0x5593ebc4a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.319+0000 7f94b65ef740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc4ae30 0x5593ebc4d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.319+0000 7f94b65ef740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593ebbb5280 con 0x5593ebc452d0 2024-03-11T19:12:45.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.319+0000 7f94b65ef740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593ebbb27b0 con 0x5593ebc4ae30 2024-03-11T19:12:45.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.319+0000 7f94b65ef740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593ebbadf90 con 0x5593ebc45cc0 2024-03-11T19:12:45.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.319+0000 7f94b0fb0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc452d0 0x5593ebc456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.319+0000 7f94b17b1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc45cc0 0x5593ebc4a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.319+0000 7f94b17b1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc45cc0 0x5593ebc4a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48446/0 (socket says 172.21.15.29:48446) 2024-03-11T19:12:45.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.319+0000 7f94b0fb0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc452d0 0x5593ebc456f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50850/0 (socket says 172.21.15.29:50850) 2024-03-11T19:12:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.319+0000 7f94b17b1640 1 -- 172.21.15.29:0/866141070 learned_addr learned my addr 172.21.15.29:0/866141070 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.320+0000 7f94a27fc640 1 --2- 172.21.15.29:0/347660821 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f94980a1b30 0x7f94980a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.320+0000 7f94b0fb0640 1 --2- 172.21.15.29:0/866141070 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc4ae30 0x5593ebc4d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.320+0000 7f94b0fb0640 1 -- 172.21.15.29:0/866141070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc45cc0 msgr2=0x5593ebc4a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.320+0000 7f94b0fb0640 1 --2- 172.21.15.29:0/866141070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc45cc0 0x5593ebc4a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.320+0000 7f94b0fb0640 1 -- 172.21.15.29:0/866141070 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc4ae30 msgr2=0x5593ebc4d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.320+0000 7f94b0fb0640 1 --2- 172.21.15.29:0/866141070 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc4ae30 0x5593ebc4d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.320+0000 7f94b0fb0640 1 -- 172.21.15.29:0/866141070 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593ebb6c570 con 0x5593ebc452d0 2024-03-11T19:12:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.320+0000 7f94b1fb2640 1 --2- 172.21.15.29:0/347660821 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f94980a1b30 0x7f94980a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.320+0000 7f94a27fc640 1 -- 172.21.15.29:0/347660821 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f94a40cd5b0 con 0x7f94ac109370 2024-03-11T19:12:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.320+0000 7f94b0fb0640 1 --2- 172.21.15.29:0/866141070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc452d0 0x5593ebc456f0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f94ac19ec60 tx=0x7f949c009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a3ee3d204c30fbf server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:45.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.321+0000 7f94977fe640 1 -- 172.21.15.29:0/866141070 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c010aa0 con 0x5593ebc452d0 2024-03-11T19:12:45.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.321+0000 7f94b1fb2640 1 --2- 172.21.15.29:0/347660821 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f94980a1b30 0x7f94980a3ff0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f94a409a4b0 tx=0x7f94a408fc40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:45.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.321+0000 7f94977fe640 1 -- 172.21.15.29:0/866141070 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f949c0054d0 con 0x5593ebc452d0 2024-03-11T19:12:45.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.321+0000 7f94977fe640 1 -- 172.21.15.29:0/866141070 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c004cf0 con 0x5593ebc452d0 2024-03-11T19:12:45.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.322+0000 7f94b65ef740 1 -- 172.21.15.29:0/866141070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc452d0 msgr2=0x5593ebc456f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:45.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.322+0000 7f94b65ef740 1 --2- 172.21.15.29:0/866141070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc452d0 0x5593ebc456f0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f94ac19ec60 tx=0x7f949c009900 comp rx=0 tx=0).stop 2024-03-11T19:12:45.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.323+0000 7f94b65ef740 1 -- 172.21.15.29:0/866141070 shutdown_connections 2024-03-11T19:12:45.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.323+0000 7f94b65ef740 1 --2- 172.21.15.29:0/866141070 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc4ae30 0x5593ebc4d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.323+0000 7f94b65ef740 1 --2- 172.21.15.29:0/866141070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc45cc0 0x5593ebc4a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.323+0000 7f94b65ef740 1 --2- 172.21.15.29:0/866141070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc452d0 0x5593ebc456f0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.323+0000 7f94b65ef740 1 -- 172.21.15.29:0/866141070 >> 172.21.15.29:0/866141070 conn(0x5593ebc407f0 msgr2=0x5593ebc40bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:45.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.323+0000 7f94b65ef740 1 -- 172.21.15.29:0/866141070 shutdown_connections 2024-03-11T19:12:45.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.324+0000 7f94b65ef740 1 -- 172.21.15.29:0/866141070 wait complete. 2024-03-11T19:12:45.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.324+0000 7f94b65ef740 1 -- start start 2024-03-11T19:12:45.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.324+0000 7f94b65ef740 1 Processor -- start 2024-03-11T19:12:45.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.325+0000 7f94b65ef740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc452d0 0x5593ebc4f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.325+0000 7f94b65ef740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc45cc0 0x5593ebc4f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.325+0000 7f94b65ef740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc4ae30 0x5593ebc56860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.325+0000 7f94b65ef740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593eb881430 con 0x5593ebc45cc0 2024-03-11T19:12:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.325+0000 7f94b65ef740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593eb83de00 con 0x5593ebc452d0 2024-03-11T19:12:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.325+0000 7f94b65ef740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593ebb32fd0 con 0x5593ebc4ae30 2024-03-11T19:12:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.325+0000 7f94b17b1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc45cc0 0x5593ebc4f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.325+0000 7f94b17b1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc45cc0 0x5593ebc4f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50860/0 (socket says 172.21.15.29:50860) 2024-03-11T19:12:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.325+0000 7f94b17b1640 1 -- 172.21.15.29:0/2744946123 learned_addr learned my addr 172.21.15.29:0/2744946123 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.325+0000 7f94b65ef740 10 client.0 _ll_register_callbacks cb 0x5593ebb88fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.325+0000 7f94b0fb0640 1 --2- 172.21.15.29:0/2744946123 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc452d0 0x5593ebc4f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.325+0000 7f94b17b1640 1 -- 172.21.15.29:0/2744946123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc4ae30 msgr2=0x5593ebc56860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.325+0000 7f94b0fb0640 1 --2- 172.21.15.29:0/2744946123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc4ae30 0x5593ebc56860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.325+0000 7f94b17b1640 1 --2- 172.21.15.29:0/2744946123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc4ae30 0x5593ebc56860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.325+0000 7f94b17b1640 1 -- 172.21.15.29:0/2744946123 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc452d0 msgr2=0x5593ebc4f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.325+0000 7f94b17b1640 1 --2- 172.21.15.29:0/2744946123 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc452d0 0x5593ebc4f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.325+0000 7f94b17b1640 1 -- 172.21.15.29:0/2744946123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f949c012450 con 0x5593ebc45cc0 2024-03-11T19:12:45.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.325+0000 7f94b0fb0640 1 --2- 172.21.15.29:0/2744946123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc4ae30 0x5593ebc56860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:12:45.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.326+0000 7f94b17b1640 1 --2- 172.21.15.29:0/2744946123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc45cc0 0x5593ebc4f820 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f94a80349e0 tx=0x7f94a8034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=536dcb274cb34c18 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:45.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.326+0000 7f9494ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:12:45.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.326+0000 7f9494ff9640 1 -- 172.21.15.29:0/2744946123 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a802dea0 con 0x5593ebc45cc0 2024-03-11T19:12:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.326+0000 7f9494ff9640 1 -- 172.21.15.29:0/2744946123 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f94a8004450 con 0x5593ebc45cc0 2024-03-11T19:12:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.326+0000 7f94b65ef740 1 -- 172.21.15.29:0/2744946123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5593eb8ad7e0 con 0x5593ebc45cc0 2024-03-11T19:12:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.326+0000 7f9494ff9640 1 -- 172.21.15.29:0/2744946123 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94a8005030 con 0x5593ebc45cc0 2024-03-11T19:12:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.326+0000 7f94b65ef740 10 client.16095 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.326+0000 7f94b65ef740 1 -- 172.21.15.29:0/2744946123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5593ebbac610 con 0x5593ebc45cc0 2024-03-11T19:12:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.326+0000 7f9494ff9640 1 -- 172.21.15.29:0/2744946123 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f94a8005640 con 0x5593ebc45cc0 2024-03-11T19:12:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.327+0000 7f9494ff9640 1 -- 172.21.15.29:0/2744946123 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f94a80046c0 con 0x5593ebc45cc0 2024-03-11T19:12:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.327+0000 7f94b65ef740 10 client.16095 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.327+0000 7f94b65ef740 10 client.16095 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:12:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.327+0000 7f94b65ef740 20 client.16095 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.327+0000 7f94b65ef740 1 --2- 172.21.15.29:0/2744946123 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5593ebc5b550 0x5593ebc5d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.327+0000 7f94b65ef740 4 client.16095 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:12:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.327+0000 7f94b65ef740 1 -- 172.21.15.29:0/2744946123 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5593eb8ad7e0 con 0x5593ebc5b550 2024-03-11T19:12:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.327+0000 7f94b0fb0640 1 --2- 172.21.15.29:0/2744946123 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5593ebc5b550 0x5593ebc5d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.327+0000 7f9494ff9640 10 client.16095 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.327+0000 7f94b0fb0640 1 --2- 172.21.15.29:0/2744946123 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5593ebc5b550 0x5593ebc5d990 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f4bbe296e34e84f server_cookie=a45e51f0c73476cf in_seq=0 out_seq=0 2024-03-11T19:12:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.328+0000 7f9494ff9640 1 -- 172.21.15.29:0/2744946123 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5593eb8ad7e0 con 0x5593ebc5b550 2024-03-11T19:12:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.328+0000 7f9494ff9640 10 client.16095 handle_command_reply: tid=0 2024-03-11T19:12:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.328+0000 7f94b65ef740 1 client.16095 shutdown 2024-03-11T19:12:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.328+0000 7f94b65ef740 1 -- 172.21.15.29:0/2744946123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc45cc0 msgr2=0x5593ebc4f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.328+0000 7f94b65ef740 1 --2- 172.21.15.29:0/2744946123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc45cc0 0x5593ebc4f820 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f94a80349e0 tx=0x7f94a8034670 comp rx=0 tx=0).stop 2024-03-11T19:12:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.329+0000 7f94b65ef740 1 -- 172.21.15.29:0/2744946123 shutdown_connections 2024-03-11T19:12:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.329+0000 7f94b65ef740 1 --2- 172.21.15.29:0/2744946123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc4ae30 0x5593ebc56860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.329+0000 7f94b65ef740 1 --2- 172.21.15.29:0/2744946123 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5593ebc5b550 0x5593ebc5d990 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.329+0000 7f94b65ef740 1 --2- 172.21.15.29:0/2744946123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc45cc0 0x5593ebc4f820 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.329+0000 7f94b65ef740 1 --2- 172.21.15.29:0/2744946123 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc452d0 0x5593ebc4f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.329+0000 7f9494ff9640 0 client.16095 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:45.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.329+0000 7f94b65ef740 1 -- 172.21.15.29:0/2744946123 >> 172.21.15.29:0/2744946123 conn(0x5593ebc407f0 msgr2=0x5593ebc3fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:45.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.329+0000 7f94b65ef740 1 -- 172.21.15.29:0/2744946123 shutdown_connections 2024-03-11T19:12:45.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.329+0000 7f94b65ef740 1 -- 172.21.15.29:0/2744946123 wait complete. 2024-03-11T19:12:45.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.329+0000 7f94b65ef740 20 client.16095 trim_cache size 0 max 16384 2024-03-11T19:12:45.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.351+0000 7f94b65ef740 1 Processor -- start 2024-03-11T19:12:45.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.351+0000 7f94b65ef740 1 -- start start 2024-03-11T19:12:45.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.351+0000 7f94b65ef740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc452d0 0x5593ebc4f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.351+0000 7f94b65ef740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc45cc0 0x5593ebc4f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.351+0000 7f94b65ef740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc4ae30 0x5593ebc56820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.351+0000 7f94b65ef740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593eb8ad7e0 con 0x5593ebc45cc0 2024-03-11T19:12:45.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.351+0000 7f94b65ef740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593ebc501e0 con 0x5593ebc452d0 2024-03-11T19:12:45.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.351+0000 7f94b65ef740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593ebc3c410 con 0x5593ebc4ae30 2024-03-11T19:12:45.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.352+0000 7f94b0fb0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc452d0 0x5593ebc4f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.352+0000 7f94b17b1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc45cc0 0x5593ebc4f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.352+0000 7f94b0fb0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc452d0 0x5593ebc4f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49192/0 (socket says 172.21.15.29:49192) 2024-03-11T19:12:45.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.352+0000 7f94b0fb0640 1 -- 172.21.15.29:0/308787489 learned_addr learned my addr 172.21.15.29:0/308787489 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:45.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.352+0000 7f94b0fb0640 1 --2- 172.21.15.29:0/308787489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc4ae30 0x5593ebc56820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.352+0000 7f94b0fb0640 1 -- 172.21.15.29:0/308787489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc4ae30 msgr2=0x5593ebc56820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:45.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.352+0000 7f94b0fb0640 1 --2- 172.21.15.29:0/308787489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc4ae30 0x5593ebc56820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.352+0000 7f94b0fb0640 1 -- 172.21.15.29:0/308787489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc45cc0 msgr2=0x5593ebc4f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:45.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.352+0000 7f94b0fb0640 1 --2- 172.21.15.29:0/308787489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc45cc0 0x5593ebc4f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.352+0000 7f94b0fb0640 1 -- 172.21.15.29:0/308787489 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593ebb48ac0 con 0x5593ebc452d0 2024-03-11T19:12:45.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.352+0000 7f94b17b1640 1 --2- 172.21.15.29:0/308787489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc45cc0 0x5593ebc4f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:45.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.353+0000 7f94b0fb0640 1 --2- 172.21.15.29:0/308787489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc452d0 0x5593ebc4f2e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f949c0056b0 tx=0x7f949c010f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b234d1a08a8c7de1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:45.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.354+0000 7f9496ffd640 1 -- 172.21.15.29:0/308787489 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c010920 con 0x5593ebc452d0 2024-03-11T19:12:45.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.354+0000 7f9496ffd640 1 -- 172.21.15.29:0/308787489 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f949c022040 con 0x5593ebc452d0 2024-03-11T19:12:45.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.354+0000 7f9496ffd640 1 -- 172.21.15.29:0/308787489 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c01d650 con 0x5593ebc452d0 2024-03-11T19:12:45.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.355+0000 7f94b65ef740 1 -- 172.21.15.29:0/308787489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc452d0 msgr2=0x5593ebc4f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:45.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.355+0000 7f94b65ef740 1 --2- 172.21.15.29:0/308787489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc452d0 0x5593ebc4f2e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f949c0056b0 tx=0x7f949c010f70 comp rx=0 tx=0).stop 2024-03-11T19:12:45.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.355+0000 7f94b65ef740 1 -- 172.21.15.29:0/308787489 shutdown_connections 2024-03-11T19:12:45.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.355+0000 7f94b65ef740 1 --2- 172.21.15.29:0/308787489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc4ae30 0x5593ebc56820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.355+0000 7f94b65ef740 1 --2- 172.21.15.29:0/308787489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc45cc0 0x5593ebc4f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.355+0000 7f94b65ef740 1 --2- 172.21.15.29:0/308787489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc452d0 0x5593ebc4f2e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.355+0000 7f94b65ef740 1 -- 172.21.15.29:0/308787489 >> 172.21.15.29:0/308787489 conn(0x5593ebc407f0 msgr2=0x5593ebc62330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:45.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.355+0000 7f94b65ef740 1 -- 172.21.15.29:0/308787489 shutdown_connections 2024-03-11T19:12:45.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.355+0000 7f94b65ef740 1 -- 172.21.15.29:0/308787489 wait complete. 2024-03-11T19:12:45.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.355+0000 7f94b65ef740 1 -- start start 2024-03-11T19:12:45.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.355+0000 7f94b65ef740 1 Processor -- start 2024-03-11T19:12:45.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.356+0000 7f94b65ef740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc452d0 0x5593ebc55120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.356+0000 7f94b65ef740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc45cc0 0x5593ebc55660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.356+0000 7f94b0fb0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc452d0 0x5593ebc55120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.356+0000 7f94b65ef740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc4ae30 0x5593ebc55ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.356+0000 7f94b65ef740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593ebc56d00 con 0x5593ebc4ae30 2024-03-11T19:12:45.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.356+0000 7f94b65ef740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593ebcb1ad0 con 0x5593ebc452d0 2024-03-11T19:12:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.356+0000 7f94b65ef740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593ebcb1c90 con 0x5593ebc45cc0 2024-03-11T19:12:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.356+0000 7f94b0fb0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc452d0 0x5593ebc55120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49198/0 (socket says 172.21.15.29:49198) 2024-03-11T19:12:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.356+0000 7f94b0fb0640 1 -- 172.21.15.29:0/2776204781 learned_addr learned my addr 172.21.15.29:0/2776204781 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.356+0000 7f94b17b1640 1 --2- 172.21.15.29:0/2776204781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc45cc0 0x5593ebc55660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.356+0000 7f94b65ef740 10 client.0 _ll_register_callbacks cb 0x7f949c009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.356+0000 7f94b0fb0640 1 --2- 172.21.15.29:0/2776204781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc4ae30 0x5593ebc55ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.357+0000 7f94b17b1640 1 -- 172.21.15.29:0/2776204781 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc452d0 msgr2=0x5593ebc55120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.357+0000 7f94b17b1640 1 --2- 172.21.15.29:0/2776204781 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc452d0 0x5593ebc55120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.357+0000 7f94b17b1640 1 -- 172.21.15.29:0/2776204781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc4ae30 msgr2=0x5593ebc55ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.357+0000 7f94b17b1640 1 --2- 172.21.15.29:0/2776204781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc4ae30 0x5593ebc55ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.357+0000 7f94b17b1640 1 -- 172.21.15.29:0/2776204781 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5593ebbac610 con 0x5593ebc45cc0 2024-03-11T19:12:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.357+0000 7f94b0fb0640 1 --2- 172.21.15.29:0/2776204781 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc452d0 0x5593ebc55120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.357+0000 7f94b17b1640 1 --2- 172.21.15.29:0/2776204781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc45cc0 0x5593ebc55660 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f94a8005b60 tx=0x7f94a8004ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19901a1c213745 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.357+0000 7f9494ff9640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:12:45.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.357+0000 7f9494ff9640 1 -- 172.21.15.29:0/2776204781 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a8004450 con 0x5593ebc45cc0 2024-03-11T19:12:45.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.357+0000 7f9494ff9640 1 -- 172.21.15.29:0/2776204781 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f94a802c510 con 0x5593ebc45cc0 2024-03-11T19:12:45.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.357+0000 7f94b65ef740 1 -- 172.21.15.29:0/2776204781 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f949c0054d0 con 0x5593ebc45cc0 2024-03-11T19:12:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.358+0000 7f9494ff9640 1 -- 172.21.15.29:0/2776204781 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94a80048d0 con 0x5593ebc45cc0 2024-03-11T19:12:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.358+0000 7f94b65ef740 10 client.25663 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.358+0000 7f94b65ef740 1 -- 172.21.15.29:0/2776204781 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7f949c034040 con 0x5593ebc45cc0 2024-03-11T19:12:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.358+0000 7f9494ff9640 1 -- 172.21.15.29:0/2776204781 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f949c0054d0 con 0x5593ebc45cc0 2024-03-11T19:12:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.358+0000 7f9494ff9640 1 -- 172.21.15.29:0/2776204781 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f94a8032600 con 0x5593ebc45cc0 2024-03-11T19:12:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.358+0000 7f94b65ef740 10 client.25663 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.358+0000 7f94b65ef740 10 client.25663 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:12:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.358+0000 7f94b65ef740 20 client.25663 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.358+0000 7f94b65ef740 1 --2- 172.21.15.29:0/2776204781 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5593ebcb3440 0x5593ebcb5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.358+0000 7f94b65ef740 4 client.25663 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:12:45.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.358+0000 7f94b65ef740 1 -- 172.21.15.29:0/2776204781 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5593eb8ea720 con 0x5593ebcb3440 2024-03-11T19:12:45.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.358+0000 7f94b0fb0640 1 --2- 172.21.15.29:0/2776204781 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5593ebcb3440 0x5593ebcb5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.359+0000 7f9494ff9640 10 client.25663 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.360+0000 7f94b0fb0640 1 --2- 172.21.15.29:0/2776204781 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5593ebcb3440 0x5593ebcb5860 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2a9f66c3e3dd762 server_cookie=1dd9d8f39afdf2ce in_seq=0 out_seq=0 2024-03-11T19:12:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.361+0000 7f9494ff9640 1 -- 172.21.15.29:0/2776204781 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5593eb8ea720 con 0x5593ebcb3440 2024-03-11T19:12:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.361+0000 7f9494ff9640 10 client.25663 handle_command_reply: tid=0 2024-03-11T19:12:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.361+0000 7f94b65ef740 1 client.25663 shutdown 2024-03-11T19:12:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.361+0000 7f94b65ef740 1 -- 172.21.15.29:0/2776204781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc45cc0 msgr2=0x5593ebc55660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.361+0000 7f94b65ef740 1 --2- 172.21.15.29:0/2776204781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc45cc0 0x5593ebc55660 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f94a8005b60 tx=0x7f94a8004ee0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.361+0000 7f94b65ef740 1 -- 172.21.15.29:0/2776204781 shutdown_connections 2024-03-11T19:12:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.361+0000 7f94b65ef740 1 --2- 172.21.15.29:0/2776204781 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5593ebcb3440 0x5593ebcb5860 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.361+0000 7f94b65ef740 1 --2- 172.21.15.29:0/2776204781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5593ebc4ae30 0x5593ebc55ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.361+0000 7f94b65ef740 1 --2- 172.21.15.29:0/2776204781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593ebc45cc0 0x5593ebc55660 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.361+0000 7f94b65ef740 1 --2- 172.21.15.29:0/2776204781 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593ebc452d0 0x5593ebc55120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.361+0000 7f94b65ef740 1 -- 172.21.15.29:0/2776204781 >> 172.21.15.29:0/2776204781 conn(0x5593ebc407f0 msgr2=0x5593ebc4f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:45.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.361+0000 7f9494ff9640 0 client.25663 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:45.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.361+0000 7f94b65ef740 1 -- 172.21.15.29:0/2776204781 shutdown_connections 2024-03-11T19:12:45.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.362+0000 7f94b65ef740 1 -- 172.21.15.29:0/2776204781 wait complete. 2024-03-11T19:12:45.366 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:12:45.366 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:12:45.366 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:12:45.366 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:12:45.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.362+0000 7f94b65ef740 20 client.25663 trim_cache size 0 max 16384 2024-03-11T19:12:45.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.362+0000 7f94b3a15640 1 -- 172.21.15.29:0/347660821 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f94980a1b30 msgr2=0x7f94980a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:45.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.362+0000 7f94b3a15640 1 --2- 172.21.15.29:0/347660821 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f94980a1b30 0x7f94980a3ff0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f94a409a4b0 tx=0x7f94a408fc40 comp rx=0 tx=0).stop 2024-03-11T19:12:45.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.362+0000 7f94b3a15640 1 -- 172.21.15.29:0/347660821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94ac109370 msgr2=0x7f94ac1a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:45.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.362+0000 7f94b3a15640 1 --2- 172.21.15.29:0/347660821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94ac109370 0x7f94ac1a4d00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f94a400eaa0 tx=0x7f94a400ef60 comp rx=0 tx=0).stop 2024-03-11T19:12:45.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.363+0000 7f94b3a15640 1 -- 172.21.15.29:0/347660821 shutdown_connections 2024-03-11T19:12:45.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.363+0000 7f94b3a15640 1 --2- 172.21.15.29:0/347660821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94ac109370 0x7f94ac1a4d00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.363+0000 7f94b3a15640 1 --2- 172.21.15.29:0/347660821 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f94980a1b30 0x7f94980a3ff0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.363+0000 7f94b3a15640 1 --2- 172.21.15.29:0/347660821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94ac1089b0 0x7f94ac19dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.363+0000 7f94b3a15640 1 --2- 172.21.15.29:0/347660821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94ac1077c0 0x7f94ac19d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.363+0000 7f94b3a15640 1 -- 172.21.15.29:0/347660821 >> 172.21.15.29:0/347660821 conn(0x7f94ac076770 msgr2=0x7f94ac10bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:45.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.363+0000 7f94b3a15640 1 -- 172.21.15.29:0/347660821 shutdown_connections 2024-03-11T19:12:45.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.363+0000 7f94b3a15640 1 -- 172.21.15.29:0/347660821 wait complete. 2024-03-11T19:12:45.383 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:12:45.384 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:12:45.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.546+0000 7f3ff9e23640 1 -- 172.21.15.29:0/671858188 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ff41000a0 msgr2=0x7f3ff41004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:45.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.546+0000 7f3ff9e23640 1 --2- 172.21.15.29:0/671858188 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ff41000a0 0x7f3ff41004c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4009980 tx=0x7f3fe402de60 comp rx=0 tx=0).stop 2024-03-11T19:12:45.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.546+0000 7f3ff9e23640 1 -- 172.21.15.29:0/671858188 shutdown_connections 2024-03-11T19:12:45.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.546+0000 7f3ff9e23640 1 --2- 172.21.15.29:0/671858188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ff4101c50 0x7f3ff41084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.546+0000 7f3ff9e23640 1 --2- 172.21.15.29:0/671858188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff4101290 0x7f3ff4101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.546+0000 7f3ff9e23640 1 --2- 172.21.15.29:0/671858188 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ff41000a0 0x7f3ff41004c0 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4009980 tx=0x7f3fe402de60 comp rx=0 tx=0).stop 2024-03-11T19:12:45.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.546+0000 7f3ff9e23640 1 -- 172.21.15.29:0/671858188 >> 172.21.15.29:0/671858188 conn(0x7f3ff40fb610 msgr2=0x7f3ff40fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:45.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.547+0000 7f3ff9e23640 1 -- 172.21.15.29:0/671858188 shutdown_connections 2024-03-11T19:12:45.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.547+0000 7f3ff9e23640 1 -- 172.21.15.29:0/671858188 wait complete. 2024-03-11T19:12:45.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.548+0000 7f3ff9e23640 1 Processor -- start 2024-03-11T19:12:45.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.548+0000 7f3ff9e23640 1 -- start start 2024-03-11T19:12:45.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.548+0000 7f3ff9e23640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ff4101290 0x7f3ff41951b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.548+0000 7f3ff9e23640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ff4101c50 0x7f3ff41956f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.548+0000 7f3ff9e23640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff419a750 0x7f3ff419cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.548+0000 7f3ff9e23640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff419d080 con 0x7f3ff419a750 2024-03-11T19:12:45.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.548+0000 7f3ff9e23640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff419d1f0 con 0x7f3ff4101290 2024-03-11T19:12:45.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.548+0000 7f3ff9e23640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff419d3b0 con 0x7f3ff4101c50 2024-03-11T19:12:45.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.548+0000 7f3ff3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff419a750 0x7f3ff419cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.548+0000 7f3ff37fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ff4101290 0x7f3ff41951b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.548+0000 7f3ff2ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ff4101c50 0x7f3ff41956f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.548+0000 7f3ff2ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ff4101c50 0x7f3ff41956f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:36866/0 (socket says 172.21.15.29:36866) 2024-03-11T19:12:45.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.548+0000 7f3ff37fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ff4101290 0x7f3ff41951b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:42736/0 (socket says 172.21.15.29:42736) 2024-03-11T19:12:45.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.549+0000 7f3ff2ffd640 1 -- 172.21.15.29:0/218829801 learned_addr learned my addr 172.21.15.29:0/218829801 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:45.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.548+0000 7f3ff3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff419a750 0x7f3ff419cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52278/0 (socket says 172.21.15.29:52278) 2024-03-11T19:12:45.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.549+0000 7f3ff3fff640 1 -- 172.21.15.29:0/218829801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ff4101c50 msgr2=0x7f3ff41956f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:45.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.549+0000 7f3ff3fff640 1 --2- 172.21.15.29:0/218829801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ff4101c50 0x7f3ff41956f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.549+0000 7f3ff3fff640 1 -- 172.21.15.29:0/218829801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ff4101290 msgr2=0x7f3ff41951b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:45.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.549+0000 7f3ff3fff640 1 --2- 172.21.15.29:0/218829801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ff4101290 0x7f3ff41951b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.549+0000 7f3ff3fff640 1 -- 172.21.15.29:0/218829801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe8009490 con 0x7f3ff419a750 2024-03-11T19:12:45.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.549+0000 7f3ff2ffd640 1 --2- 172.21.15.29:0/218829801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ff4101c50 0x7f3ff41956f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:45.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.549+0000 7f3ff3fff640 1 --2- 172.21.15.29:0/218829801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff419a750 0x7f3ff419cb40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00e940 tx=0x7f3fec00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:45.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.549+0000 7f3ff0ff9640 1 -- 172.21.15.29:0/218829801 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fec004d20 con 0x7f3ff419a750 2024-03-11T19:12:45.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.549+0000 7f3ff9e23640 1 -- 172.21.15.29:0/218829801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe4009560 con 0x7f3ff419a750 2024-03-11T19:12:45.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.549+0000 7f3ff0ff9640 1 -- 172.21.15.29:0/218829801 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3fec004e80 con 0x7f3ff419a750 2024-03-11T19:12:45.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.549+0000 7f3ff0ff9640 1 -- 172.21.15.29:0/218829801 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fec011470 con 0x7f3ff419a750 2024-03-11T19:12:45.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.550+0000 7f3ff9e23640 1 -- 172.21.15.29:0/218829801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff419d9c0 con 0x7f3ff419a750 2024-03-11T19:12:45.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.551+0000 7f3ff9e23640 1 -- 172.21.15.29:0/218829801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ff41074e0 con 0x7f3ff419a750 2024-03-11T19:12:45.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.552+0000 7f3ff0ff9640 1 -- 172.21.15.29:0/218829801 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3fec0052b0 con 0x7f3ff419a750 2024-03-11T19:12:45.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.553+0000 7f3ff0ff9640 1 --2- 172.21.15.29:0/218829801 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3fdc0a1a30 0x7f3fdc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:45.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.553+0000 7f3ff0ff9640 1 -- 172.21.15.29:0/218829801 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f3fec0cd3c0 con 0x7f3ff419a750 2024-03-11T19:12:45.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.553+0000 7f3ff37fe640 1 --2- 172.21.15.29:0/218829801 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3fdc0a1a30 0x7f3fdc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:45.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.554+0000 7f3ff37fe640 1 --2- 172.21.15.29:0/218829801 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3fdc0a1a30 0x7f3fdc0a3ef0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4009230 tx=0x7f3fe4004940 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:45.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.559+0000 7f3ff0ff9640 1 -- 172.21.15.29:0/218829801 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3fec096720 con 0x7f3ff419a750 2024-03-11T19:12:45.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:45 smithi190 ceph-mon[36292]: pgmap v439: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-11T19:12:45.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:45 smithi170 ceph-mon[35690]: pgmap v439: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-11T19:12:45.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:45 smithi029 ceph-mon[32439]: pgmap v439: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-11T19:12:45.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.973+0000 7f3ff9e23640 1 -- 172.21.15.29:0/218829801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ff410aa50 con 0x7f3ff419a750 2024-03-11T19:12:45.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.974+0000 7f3ff0ff9640 1 -- 172.21.15.29:0/218829801 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f3fec099de0 con 0x7f3ff419a750 2024-03-11T19:12:45.977 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:12:45.977 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:12:45.977 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:12:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.977+0000 7f3ff9e23640 1 -- 172.21.15.29:0/218829801 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3fdc0a1a30 msgr2=0x7f3fdc0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.977+0000 7f3ff9e23640 1 --2- 172.21.15.29:0/218829801 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3fdc0a1a30 0x7f3fdc0a3ef0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4009230 tx=0x7f3fe4004940 comp rx=0 tx=0).stop 2024-03-11T19:12:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.977+0000 7f3ff9e23640 1 -- 172.21.15.29:0/218829801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff419a750 msgr2=0x7f3ff419cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.977+0000 7f3ff9e23640 1 --2- 172.21.15.29:0/218829801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff419a750 0x7f3ff419cb40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00e940 tx=0x7f3fec00ee00 comp rx=0 tx=0).stop 2024-03-11T19:12:45.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.977+0000 7f3ff9e23640 1 -- 172.21.15.29:0/218829801 shutdown_connections 2024-03-11T19:12:45.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.977+0000 7f3ff9e23640 1 --2- 172.21.15.29:0/218829801 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3fdc0a1a30 0x7f3fdc0a3ef0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.977+0000 7f3ff9e23640 1 --2- 172.21.15.29:0/218829801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ff419a750 0x7f3ff419cb40 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.977+0000 7f3ff9e23640 1 --2- 172.21.15.29:0/218829801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ff4101c50 0x7f3ff41956f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.977+0000 7f3ff9e23640 1 --2- 172.21.15.29:0/218829801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ff4101290 0x7f3ff41951b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:45.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.977+0000 7f3ff9e23640 1 -- 172.21.15.29:0/218829801 >> 172.21.15.29:0/218829801 conn(0x7f3ff40fb610 msgr2=0x7f3ff41046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:45.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.977+0000 7f3ff9e23640 1 -- 172.21.15.29:0/218829801 shutdown_connections 2024-03-11T19:12:45.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:45.977+0000 7f3ff9e23640 1 -- 172.21.15.29:0/218829801 wait complete. 2024-03-11T19:12:45.997 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:12:45.997 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:12:46.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.160+0000 7f7963fff640 1 -- 172.21.15.29:0/3655639188 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c032340 con 0x7f796c1045f0 2024-03-11T19:12:46.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.160+0000 7f797390d640 1 -- 172.21.15.29:0/3655639188 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f796c1045f0 msgr2=0x7f796c1069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.160+0000 7f797390d640 1 --2- 172.21.15.29:0/3655639188 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f796c1045f0 0x7f796c1069e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f795c009860 tx=0x7f795c02bb70 comp rx=0 tx=0).stop 2024-03-11T19:12:46.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.160+0000 7f797390d640 1 -- 172.21.15.29:0/3655639188 shutdown_connections 2024-03-11T19:12:46.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.160+0000 7f797390d640 1 --2- 172.21.15.29:0/3655639188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796c106f20 0x7f796c109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.160+0000 7f797390d640 1 --2- 172.21.15.29:0/3655639188 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f796c1045f0 0x7f796c1069e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.160+0000 7f797390d640 1 --2- 172.21.15.29:0/3655639188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f796c101cc0 0x7f796c1040b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.160+0000 7f797390d640 1 -- 172.21.15.29:0/3655639188 >> 172.21.15.29:0/3655639188 conn(0x7f796c0fb610 msgr2=0x7f796c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.161+0000 7f797390d640 1 -- 172.21.15.29:0/3655639188 shutdown_connections 2024-03-11T19:12:46.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.161+0000 7f797390d640 1 -- 172.21.15.29:0/3655639188 wait complete. 2024-03-11T19:12:46.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.161+0000 7f797390d640 1 Processor -- start 2024-03-11T19:12:46.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.161+0000 7f797390d640 1 -- start start 2024-03-11T19:12:46.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.161+0000 7f797390d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f796c101cc0 0x7f796c1971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.162+0000 7f797390d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796c1045f0 0x7f796c1976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.162+0000 7f797390d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f796c106f20 0x7f796c1a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.162+0000 7f797390d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c1a0fb0 con 0x7f796c1045f0 2024-03-11T19:12:46.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.162+0000 7f797390d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c1a10f0 con 0x7f796c106f20 2024-03-11T19:12:46.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.162+0000 7f797390d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c1a1230 con 0x7f796c101cc0 2024-03-11T19:12:46.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.162+0000 7f79716a9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f796c101cc0 0x7f796c1971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.162+0000 7f7970ea8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796c1045f0 0x7f796c1976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.162+0000 7f7971eaa640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f796c106f20 0x7f796c1a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.162+0000 7f7970ea8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796c1045f0 0x7f796c1976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52300/0 (socket says 172.21.15.29:52300) 2024-03-11T19:12:46.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.162+0000 7f7970ea8640 1 -- 172.21.15.29:0/2551838483 learned_addr learned my addr 172.21.15.29:0/2551838483 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.162+0000 7f7971eaa640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f796c106f20 0x7f796c1a0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:42760/0 (socket says 172.21.15.29:42760) 2024-03-11T19:12:46.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.162+0000 7f79716a9640 1 -- 172.21.15.29:0/2551838483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f796c106f20 msgr2=0x7f796c1a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.162+0000 7f79716a9640 1 --2- 172.21.15.29:0/2551838483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f796c106f20 0x7f796c1a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.162+0000 7f79716a9640 1 -- 172.21.15.29:0/2551838483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796c1045f0 msgr2=0x7f796c1976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.162+0000 7f79716a9640 1 --2- 172.21.15.29:0/2551838483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796c1045f0 0x7f796c1976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.162+0000 7f79716a9640 1 -- 172.21.15.29:0/2551838483 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795c009490 con 0x7f796c101cc0 2024-03-11T19:12:46.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.162+0000 7f7970ea8640 1 --2- 172.21.15.29:0/2551838483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796c1045f0 0x7f796c1976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:46.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.162+0000 7f79716a9640 1 --2- 172.21.15.29:0/2551838483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f796c101cc0 0x7f796c1971a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f796800e9f0 tx=0x7f796800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.163+0000 7f79627fc640 1 -- 172.21.15.29:0/2551838483 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796800ccc0 con 0x7f796c101cc0 2024-03-11T19:12:46.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.163+0000 7f797390d640 1 -- 172.21.15.29:0/2551838483 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f796c1a14c0 con 0x7f796c101cc0 2024-03-11T19:12:46.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.163+0000 7f797390d640 1 -- 172.21.15.29:0/2551838483 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796c1a19f0 con 0x7f796c101cc0 2024-03-11T19:12:46.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.163+0000 7f79627fc640 1 -- 172.21.15.29:0/2551838483 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f796800ce20 con 0x7f796c101cc0 2024-03-11T19:12:46.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.163+0000 7f79627fc640 1 -- 172.21.15.29:0/2551838483 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79680187a0 con 0x7f796c101cc0 2024-03-11T19:12:46.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.165+0000 7f79627fc640 1 -- 172.21.15.29:0/2551838483 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7968018900 con 0x7f796c101cc0 2024-03-11T19:12:46.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.166+0000 7f79627fc640 1 --2- 172.21.15.29:0/2551838483 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f79580a1a30 0x7f79580a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.166+0000 7f79627fc640 1 -- 172.21.15.29:0/2551838483 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7968014070 con 0x7f796c101cc0 2024-03-11T19:12:46.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.166+0000 7f7970ea8640 1 --2- 172.21.15.29:0/2551838483 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f79580a1a30 0x7f79580a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.166+0000 7f79765b9740 1 Processor -- start 2024-03-11T19:12:46.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.167+0000 7f79765b9740 1 -- start start 2024-03-11T19:12:46.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.167+0000 7f79765b9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f97a80 0x5602e8f97ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.167+0000 7f79765b9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f98470 0x5602e8f9cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.167+0000 7f79765b9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f9d5e0 0x5602e8f9fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.167+0000 7f79765b9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e8f4b960 con 0x5602e8f98470 2024-03-11T19:12:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.167+0000 7f79765b9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e8eff310 con 0x5602e8f97a80 2024-03-11T19:12:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.167+0000 7f79765b9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e8efc840 con 0x5602e8f9d5e0 2024-03-11T19:12:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.167+0000 7f79716a9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f97a80 0x5602e8f97ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.167+0000 7f7971eaa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f9d5e0 0x5602e8f9fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.167+0000 7f7970ea8640 1 --2- 172.21.15.29:0/2551838483 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f79580a1a30 0x7f79580a3ef0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f795c0057c0 tx=0x7f795c02e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.167+0000 7f7970ea8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f98470 0x5602e8f9cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.167+0000 7f7971eaa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f9d5e0 0x5602e8f9fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:36888/0 (socket says 172.21.15.29:36888) 2024-03-11T19:12:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.167+0000 7f7971eaa640 1 -- 172.21.15.29:0/3177114733 learned_addr learned my addr 172.21.15.29:0/3177114733 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.168+0000 7f79716a9640 1 -- 172.21.15.29:0/3177114733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f9d5e0 msgr2=0x5602e8f9fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.168+0000 7f79716a9640 1 --2- 172.21.15.29:0/3177114733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f9d5e0 0x5602e8f9fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.168+0000 7f79716a9640 1 -- 172.21.15.29:0/3177114733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f98470 msgr2=0x5602e8f9cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.168+0000 7f79716a9640 1 --2- 172.21.15.29:0/3177114733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f98470 0x5602e8f9cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.168+0000 7f79716a9640 1 -- 172.21.15.29:0/3177114733 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602e8eba060 con 0x5602e8f97a80 2024-03-11T19:12:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.168+0000 7f7970ea8640 1 --2- 172.21.15.29:0/3177114733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f98470 0x5602e8f9cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.168+0000 7f79716a9640 1 --2- 172.21.15.29:0/3177114733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f97a80 0x5602e8f97ea0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f79680cb160 tx=0x7f796809a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b6775acbd21a3e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.168+0000 7f79577fe640 1 -- 172.21.15.29:0/3177114733 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79680986d0 con 0x5602e8f97a80 2024-03-11T19:12:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.168+0000 7f79577fe640 1 -- 172.21.15.29:0/3177114733 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7968098930 con 0x5602e8f97a80 2024-03-11T19:12:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.168+0000 7f79577fe640 1 -- 172.21.15.29:0/3177114733 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7968018bb0 con 0x5602e8f97a80 2024-03-11T19:12:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.169+0000 7f79765b9740 1 -- 172.21.15.29:0/3177114733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f97a80 msgr2=0x5602e8f97ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.169+0000 7f79765b9740 1 --2- 172.21.15.29:0/3177114733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f97a80 0x5602e8f97ea0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f79680cb160 tx=0x7f796809a080 comp rx=0 tx=0).stop 2024-03-11T19:12:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.169+0000 7f79765b9740 1 -- 172.21.15.29:0/3177114733 shutdown_connections 2024-03-11T19:12:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.169+0000 7f79765b9740 1 --2- 172.21.15.29:0/3177114733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f9d5e0 0x5602e8f9fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.169+0000 7f79765b9740 1 --2- 172.21.15.29:0/3177114733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f98470 0x5602e8f9cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.169+0000 7f79765b9740 1 --2- 172.21.15.29:0/3177114733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f97a80 0x5602e8f97ea0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.169+0000 7f79765b9740 1 -- 172.21.15.29:0/3177114733 >> 172.21.15.29:0/3177114733 conn(0x5602e8f92ff0 msgr2=0x5602e8f933f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.169+0000 7f79765b9740 1 -- 172.21.15.29:0/3177114733 shutdown_connections 2024-03-11T19:12:46.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.169+0000 7f79765b9740 1 -- 172.21.15.29:0/3177114733 wait complete. 2024-03-11T19:12:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.170+0000 7f79765b9740 1 -- start start 2024-03-11T19:12:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.170+0000 7f79765b9740 1 Processor -- start 2024-03-11T19:12:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.170+0000 7f79765b9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f97a80 0x5602e8fa1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.170+0000 7f79765b9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f98470 0x5602e8fa2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.170+0000 7f79765b9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f9d5e0 0x5602e8fa9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.170+0000 7f7970ea8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f98470 0x5602e8fa2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.170+0000 7f7970ea8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f98470 0x5602e8fa2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52312/0 (socket says 172.21.15.29:52312) 2024-03-11T19:12:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.170+0000 7f7970ea8640 1 -- 172.21.15.29:0/407623999 learned_addr learned my addr 172.21.15.29:0/407623999 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.170+0000 7f79765b9740 1 -- 172.21.15.29:0/407623999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e8ef8020 con 0x5602e8f98470 2024-03-11T19:12:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.170+0000 7f79765b9740 1 -- 172.21.15.29:0/407623999 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e8bcb430 con 0x5602e8f97a80 2024-03-11T19:12:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.170+0000 7f79765b9740 1 -- 172.21.15.29:0/407623999 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e8b87e00 con 0x5602e8f9d5e0 2024-03-11T19:12:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.171+0000 7f79716a9640 1 --2- 172.21.15.29:0/407623999 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f97a80 0x5602e8fa1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.171+0000 7f79765b9740 10 client.0 _ll_register_callbacks cb 0x5602e8e9fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:46.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.171+0000 7f7971eaa640 1 --2- 172.21.15.29:0/407623999 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f9d5e0 0x5602e8fa9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.171+0000 7f7970ea8640 1 -- 172.21.15.29:0/407623999 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f9d5e0 msgr2=0x5602e8fa9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.171+0000 7f7970ea8640 1 --2- 172.21.15.29:0/407623999 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f9d5e0 0x5602e8fa9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.171+0000 7f7970ea8640 1 -- 172.21.15.29:0/407623999 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f97a80 msgr2=0x5602e8fa1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.171+0000 7f7970ea8640 1 --2- 172.21.15.29:0/407623999 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f97a80 0x5602e8fa1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.171+0000 7f7970ea8640 1 -- 172.21.15.29:0/407623999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796c1a19f0 con 0x5602e8f98470 2024-03-11T19:12:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.171+0000 7f7970ea8640 1 --2- 172.21.15.29:0/407623999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f98470 0x5602e8fa2140 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f795c0349e0 tx=0x7f795c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f65e608d82cd41d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.171+0000 7f7954ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:12:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.171+0000 7f7954ff9640 1 -- 172.21.15.29:0/407623999 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c0037f0 con 0x5602e8f98470 2024-03-11T19:12:46.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.171+0000 7f7954ff9640 1 -- 172.21.15.29:0/407623999 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f795c046040 con 0x5602e8f98470 2024-03-11T19:12:46.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.171+0000 7f79765b9740 1 -- 172.21.15.29:0/407623999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5602e8bf77e0 con 0x5602e8f98470 2024-03-11T19:12:46.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.172+0000 7f7954ff9640 1 -- 172.21.15.29:0/407623999 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f795c003950 con 0x5602e8f98470 2024-03-11T19:12:46.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.172+0000 7f79765b9740 10 client.16119 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:46.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.172+0000 7f79765b9740 1 -- 172.21.15.29:0/407623999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5602e8ef66a0 con 0x5602e8f98470 2024-03-11T19:12:46.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.172+0000 7f7954ff9640 1 -- 172.21.15.29:0/407623999 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5602e8bf77e0 con 0x5602e8f98470 2024-03-11T19:12:46.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.172+0000 7f7954ff9640 1 -- 172.21.15.29:0/407623999 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f795c032cb0 con 0x5602e8f98470 2024-03-11T19:12:46.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.172+0000 7f79765b9740 10 client.16119 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:46.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.172+0000 7f79765b9740 10 client.16119 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:12:46.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.173+0000 7f79765b9740 20 client.16119 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:46.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.173+0000 7f79765b9740 1 --2- 172.21.15.29:0/407623999 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5602e8fadd50 0x5602e8fb0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.173+0000 7f79765b9740 4 client.16119 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:12:46.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.173+0000 7f79765b9740 1 -- 172.21.15.29:0/407623999 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5602e8c34720 con 0x5602e8fadd50 2024-03-11T19:12:46.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.173+0000 7f79716a9640 1 --2- 172.21.15.29:0/407623999 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5602e8fadd50 0x5602e8fb0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.174+0000 7f79716a9640 1 --2- 172.21.15.29:0/407623999 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5602e8fadd50 0x5602e8fb0170 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=63b1ea5b5c5bfdcf server_cookie=3b8dfda3dfd6fdcb in_seq=0 out_seq=0 2024-03-11T19:12:46.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.174+0000 7f7954ff9640 10 client.16119 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:12:46.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.175+0000 7f7954ff9640 1 -- 172.21.15.29:0/407623999 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5602e8c34720 con 0x5602e8fadd50 2024-03-11T19:12:46.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.175+0000 7f7954ff9640 10 client.16119 handle_command_reply: tid=0 2024-03-11T19:12:46.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.175+0000 7f79765b9740 1 client.16119 shutdown 2024-03-11T19:12:46.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.175+0000 7f79765b9740 1 -- 172.21.15.29:0/407623999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f98470 msgr2=0x5602e8fa2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.175+0000 7f79765b9740 1 --2- 172.21.15.29:0/407623999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f98470 0x5602e8fa2140 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f795c0349e0 tx=0x7f795c005f90 comp rx=0 tx=0).stop 2024-03-11T19:12:46.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.176+0000 7f79765b9740 1 -- 172.21.15.29:0/407623999 shutdown_connections 2024-03-11T19:12:46.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.176+0000 7f79765b9740 1 --2- 172.21.15.29:0/407623999 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5602e8fadd50 0x5602e8fb0170 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.176+0000 7f79765b9740 1 --2- 172.21.15.29:0/407623999 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f9d5e0 0x5602e8fa9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.176+0000 7f7954ff9640 0 client.16119 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:12:46.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.176+0000 7f79765b9740 1 --2- 172.21.15.29:0/407623999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f98470 0x5602e8fa2140 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.176+0000 7f79765b9740 1 --2- 172.21.15.29:0/407623999 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f97a80 0x5602e8fa1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.176+0000 7f79765b9740 1 -- 172.21.15.29:0/407623999 >> 172.21.15.29:0/407623999 conn(0x5602e8f92ff0 msgr2=0x5602e8f94240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.176+0000 7f79765b9740 1 -- 172.21.15.29:0/407623999 shutdown_connections 2024-03-11T19:12:46.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.176+0000 7f79765b9740 1 -- 172.21.15.29:0/407623999 wait complete. 2024-03-11T19:12:46.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.176+0000 7f79765b9740 20 client.16119 trim_cache size 0 max 16384 2024-03-11T19:12:46.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.198+0000 7f79765b9740 1 Processor -- start 2024-03-11T19:12:46.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.199+0000 7f79765b9740 1 -- start start 2024-03-11T19:12:46.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.199+0000 7f79765b9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f97a80 0x5602e8f999c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.199+0000 7f79765b9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f98470 0x5602e8f99f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.199+0000 7f79765b9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f9d5e0 0x5602e8fa9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.199+0000 7f79765b9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e8e7d130 con 0x5602e8f9d5e0 2024-03-11T19:12:46.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.199+0000 7f79765b9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e8fa2810 con 0x5602e8f97a80 2024-03-11T19:12:46.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.199+0000 7f79765b9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e8f98980 con 0x5602e8f98470 2024-03-11T19:12:46.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.199+0000 7f79716a9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f97a80 0x5602e8f999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.199+0000 7f7970ea8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f98470 0x5602e8f99f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.199+0000 7f79716a9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f97a80 0x5602e8f999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:42778/0 (socket says 172.21.15.29:42778) 2024-03-11T19:12:46.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.199+0000 7f7971eaa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f9d5e0 0x5602e8fa9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.199+0000 7f79716a9640 1 -- 172.21.15.29:0/2740089740 learned_addr learned my addr 172.21.15.29:0/2740089740 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.200+0000 7f7971eaa640 1 -- 172.21.15.29:0/2740089740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f98470 msgr2=0x5602e8f99f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.200+0000 7f7971eaa640 1 --2- 172.21.15.29:0/2740089740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f98470 0x5602e8f99f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.200+0000 7f7971eaa640 1 -- 172.21.15.29:0/2740089740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f97a80 msgr2=0x5602e8f999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.200+0000 7f7971eaa640 1 --2- 172.21.15.29:0/2740089740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f97a80 0x5602e8f999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.200+0000 7f7971eaa640 1 -- 172.21.15.29:0/2740089740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602e8e92c20 con 0x5602e8f9d5e0 2024-03-11T19:12:46.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.200+0000 7f7971eaa640 1 --2- 172.21.15.29:0/2740089740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f9d5e0 0x5602e8fa9140 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x5602e8f93f60 tx=0x7f79640095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a53d19e207f2063 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.200+0000 7f7956ffd640 1 -- 172.21.15.29:0/2740089740 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7964013040 con 0x5602e8f9d5e0 2024-03-11T19:12:46.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.200+0000 7f7956ffd640 1 -- 172.21.15.29:0/2740089740 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7964003e40 con 0x5602e8f9d5e0 2024-03-11T19:12:46.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.200+0000 7f7956ffd640 1 -- 172.21.15.29:0/2740089740 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796400d7e0 con 0x5602e8f9d5e0 2024-03-11T19:12:46.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.202+0000 7f79765b9740 1 -- 172.21.15.29:0/2740089740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f9d5e0 msgr2=0x5602e8fa9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.202+0000 7f79765b9740 1 --2- 172.21.15.29:0/2740089740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f9d5e0 0x5602e8fa9140 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x5602e8f93f60 tx=0x7f79640095c0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.202+0000 7f79765b9740 1 -- 172.21.15.29:0/2740089740 shutdown_connections 2024-03-11T19:12:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.202+0000 7f79765b9740 1 --2- 172.21.15.29:0/2740089740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f9d5e0 0x5602e8fa9140 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.202+0000 7f79765b9740 1 --2- 172.21.15.29:0/2740089740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f98470 0x5602e8f99f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.202+0000 7f79765b9740 1 --2- 172.21.15.29:0/2740089740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f97a80 0x5602e8f999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.202+0000 7f79765b9740 1 -- 172.21.15.29:0/2740089740 >> 172.21.15.29:0/2740089740 conn(0x5602e8f92ff0 msgr2=0x5602e8fa1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.202+0000 7f79765b9740 1 -- 172.21.15.29:0/2740089740 shutdown_connections 2024-03-11T19:12:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.202+0000 7f79765b9740 1 -- 172.21.15.29:0/2740089740 wait complete. 2024-03-11T19:12:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.202+0000 7f79765b9740 1 -- start start 2024-03-11T19:12:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.203+0000 7f79765b9740 1 Processor -- start 2024-03-11T19:12:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.203+0000 7f79765b9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f97a80 0x5602e8fa7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.203+0000 7f79765b9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f98470 0x5602e8fa7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.204+0000 7f79716a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f97a80 0x5602e8fa7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.204+0000 7f79716a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f97a80 0x5602e8fa7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52332/0 (socket says 172.21.15.29:52332) 2024-03-11T19:12:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.204+0000 7f79716a9640 1 -- 172.21.15.29:0/2796824501 learned_addr learned my addr 172.21.15.29:0/2796824501 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.204+0000 7f79765b9740 1 --2- 172.21.15.29:0/2796824501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f9d5e0 0x5602e8fa83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.204+0000 7f79765b9740 1 -- 172.21.15.29:0/2796824501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e8fa9590 con 0x5602e8f97a80 2024-03-11T19:12:46.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.204+0000 7f79765b9740 1 -- 172.21.15.29:0/2796824501 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e9004240 con 0x5602e8f9d5e0 2024-03-11T19:12:46.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.204+0000 7f79765b9740 1 -- 172.21.15.29:0/2796824501 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e90043b0 con 0x5602e8f98470 2024-03-11T19:12:46.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.204+0000 7f7971eaa640 1 --2- 172.21.15.29:0/2796824501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f9d5e0 0x5602e8fa83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.204+0000 7f79716a9640 1 -- 172.21.15.29:0/2796824501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f98470 msgr2=0x5602e8fa7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.204+0000 7f79765b9740 10 client.0 _ll_register_callbacks cb 0x7f7968096290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:46.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.204+0000 7f7970ea8640 1 --2- 172.21.15.29:0/2796824501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f98470 0x5602e8fa7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.204+0000 7f79716a9640 1 --2- 172.21.15.29:0/2796824501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f98470 0x5602e8fa7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.204+0000 7f79716a9640 1 -- 172.21.15.29:0/2796824501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f9d5e0 msgr2=0x5602e8fa83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.204+0000 7f79716a9640 1 --2- 172.21.15.29:0/2796824501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f9d5e0 0x5602e8fa83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.204+0000 7f79716a9640 1 -- 172.21.15.29:0/2796824501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7964009e10 con 0x5602e8f97a80 2024-03-11T19:12:46.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.204+0000 7f7971eaa640 1 --2- 172.21.15.29:0/2796824501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f9d5e0 0x5602e8fa83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:12:46.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.204+0000 7f79765b9740 1 -- 172.21.15.29:0/2796824501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7968098930 con 0x5602e8f97a80 2024-03-11T19:12:46.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.204+0000 7f7970ea8640 1 --2- 172.21.15.29:0/2796824501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f98470 0x5602e8fa7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:12:46.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.204+0000 7f79716a9640 1 --2- 172.21.15.29:0/2796824501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f97a80 0x5602e8fa7940 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f796800c6a0 tx=0x7f796809e850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b59611e8c88e2fce server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.205+0000 7f7954ff9640 10 client.16125 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:12:46.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.205+0000 7f7954ff9640 1 -- 172.21.15.29:0/2796824501 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7968096ca0 con 0x5602e8f97a80 2024-03-11T19:12:46.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.205+0000 7f7954ff9640 1 -- 172.21.15.29:0/2796824501 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f79680ba0d0 con 0x5602e8f97a80 2024-03-11T19:12:46.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.205+0000 7f7954ff9640 1 -- 172.21.15.29:0/2796824501 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79680850b0 con 0x5602e8f97a80 2024-03-11T19:12:46.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.205+0000 7f79765b9740 10 client.16125 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:46.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.205+0000 7f79765b9740 1 -- 172.21.15.29:0/2796824501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x5602e8eb88c0 con 0x5602e8f97a80 2024-03-11T19:12:46.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.205+0000 7f7954ff9640 1 -- 172.21.15.29:0/2796824501 <== mon.0 v2:172.21.15.29:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f79680b4070 con 0x5602e8f97a80 2024-03-11T19:12:46.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.205+0000 7f79765b9740 10 client.16125 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:46.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.205+0000 7f79765b9740 10 client.16125 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:12:46.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.205+0000 7f79765b9740 20 client.16125 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:46.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.205+0000 7f79765b9740 1 --2- 172.21.15.29:0/2796824501 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5602e8fadd50 0x5602e9007a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.205+0000 7f79765b9740 4 client.16125 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:12:46.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.205+0000 7f79765b9740 1 -- 172.21.15.29:0/2796824501 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5602e8c83890 con 0x5602e8fadd50 2024-03-11T19:12:46.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.206+0000 7f7970ea8640 1 --2- 172.21.15.29:0/2796824501 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5602e8fadd50 0x5602e9007a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.206+0000 7f7970ea8640 1 --2- 172.21.15.29:0/2796824501 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5602e8fadd50 0x5602e9007a60 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d19f27890940f6c server_cookie=bfb9d07f816a96db in_seq=0 out_seq=0 2024-03-11T19:12:46.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.206+0000 7f7954ff9640 10 client.16125 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:12:46.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.206+0000 7f7954ff9640 1 -- 172.21.15.29:0/2796824501 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5602e8c83890 con 0x5602e8fadd50 2024-03-11T19:12:46.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.206+0000 7f7954ff9640 10 client.16125 handle_command_reply: tid=0 2024-03-11T19:12:46.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.206+0000 7f79765b9740 1 client.16125 shutdown 2024-03-11T19:12:46.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.207+0000 7f79765b9740 1 -- 172.21.15.29:0/2796824501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f97a80 msgr2=0x5602e8fa7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.207+0000 7f79765b9740 1 --2- 172.21.15.29:0/2796824501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f97a80 0x5602e8fa7940 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f796800c6a0 tx=0x7f796809e850 comp rx=0 tx=0).stop 2024-03-11T19:12:46.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.207+0000 7f79765b9740 1 -- 172.21.15.29:0/2796824501 shutdown_connections 2024-03-11T19:12:46.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.207+0000 7f79765b9740 1 --2- 172.21.15.29:0/2796824501 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5602e8fadd50 0x5602e9007a60 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.207+0000 7f79765b9740 1 --2- 172.21.15.29:0/2796824501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5602e8f9d5e0 0x5602e8fa83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.207+0000 7f79765b9740 1 --2- 172.21.15.29:0/2796824501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5602e8f98470 0x5602e8fa7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.207+0000 7f79765b9740 1 --2- 172.21.15.29:0/2796824501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602e8f97a80 0x5602e8fa7940 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.207+0000 7f79765b9740 1 -- 172.21.15.29:0/2796824501 >> 172.21.15.29:0/2796824501 conn(0x5602e8f92ff0 msgr2=0x5602e8f892f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.207+0000 7f7954ff9640 0 client.16125 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:12:46.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.207+0000 7f79765b9740 1 -- 172.21.15.29:0/2796824501 shutdown_connections 2024-03-11T19:12:46.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.207+0000 7f79765b9740 1 -- 172.21.15.29:0/2796824501 wait complete. 2024-03-11T19:12:46.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.207+0000 7f79765b9740 20 client.16125 trim_cache size 0 max 16384 2024-03-11T19:12:46.214 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:12:46.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.208+0000 7f797390d640 1 -- 172.21.15.29:0/2551838483 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f79580a1a30 msgr2=0x7f79580a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.208+0000 7f797390d640 1 --2- 172.21.15.29:0/2551838483 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f79580a1a30 0x7f79580a3ef0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f795c0057c0 tx=0x7f795c02e040 comp rx=0 tx=0).stop 2024-03-11T19:12:46.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.208+0000 7f797390d640 1 -- 172.21.15.29:0/2551838483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f796c101cc0 msgr2=0x7f796c1971a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.208+0000 7f797390d640 1 --2- 172.21.15.29:0/2551838483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f796c101cc0 0x7f796c1971a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f796800e9f0 tx=0x7f796800eeb0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.208+0000 7f797390d640 1 -- 172.21.15.29:0/2551838483 shutdown_connections 2024-03-11T19:12:46.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.208+0000 7f797390d640 1 --2- 172.21.15.29:0/2551838483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f796c106f20 0x7f796c1a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.208+0000 7f797390d640 1 --2- 172.21.15.29:0/2551838483 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f79580a1a30 0x7f79580a3ef0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.208+0000 7f797390d640 1 --2- 172.21.15.29:0/2551838483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f796c1045f0 0x7f796c1976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.208+0000 7f797390d640 1 --2- 172.21.15.29:0/2551838483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f796c101cc0 0x7f796c1971a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.208+0000 7f797390d640 1 -- 172.21.15.29:0/2551838483 >> 172.21.15.29:0/2551838483 conn(0x7f796c0fb610 msgr2=0x7f796c1027c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.208+0000 7f797390d640 1 -- 172.21.15.29:0/2551838483 shutdown_connections 2024-03-11T19:12:46.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.208+0000 7f797390d640 1 -- 172.21.15.29:0/2551838483 wait complete. 2024-03-11T19:12:46.227 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:12:46.227 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:12:46.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.354+0000 7fd088d4f640 1 -- 172.21.15.29:0/1936574942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd084100ca0 msgr2=0x7fd084108540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.354+0000 7fd088d4f640 1 --2- 172.21.15.29:0/1936574942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd084100ca0 0x7fd084108540 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fd07c009ca0 tx=0x7fd07c02c140 comp rx=0 tx=0).stop 2024-03-11T19:12:46.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.354+0000 7fd088d4f640 1 -- 172.21.15.29:0/1936574942 shutdown_connections 2024-03-11T19:12:46.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.354+0000 7fd088d4f640 1 --2- 172.21.15.29:0/1936574942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd084100ca0 0x7fd084108540 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.354+0000 7fd088d4f640 1 --2- 172.21.15.29:0/1936574942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0841002e0 0x7fd084100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.354+0000 7fd088d4f640 1 --2- 172.21.15.29:0/1936574942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd084101720 0x7fd084101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.354+0000 7fd088d4f640 1 -- 172.21.15.29:0/1936574942 >> 172.21.15.29:0/1936574942 conn(0x7fd0840fadb0 msgr2=0x7fd0840fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.354+0000 7fd088d4f640 1 -- 172.21.15.29:0/1936574942 shutdown_connections 2024-03-11T19:12:46.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.354+0000 7fd088d4f640 1 -- 172.21.15.29:0/1936574942 wait complete. 2024-03-11T19:12:46.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.355+0000 7fd088d4f640 1 Processor -- start 2024-03-11T19:12:46.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.355+0000 7fd088d4f640 1 -- start start 2024-03-11T19:12:46.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.355+0000 7fd088d4f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0841002e0 0x7fd0841970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.355+0000 7fd088d4f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd084100ca0 0x7fd084197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.355+0000 7fd088d4f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd084101720 0x7fd08419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.355+0000 7fd088d4f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08419ebe0 con 0x7fd0841002e0 2024-03-11T19:12:46.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.355+0000 7fd088d4f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08419ed50 con 0x7fd084100ca0 2024-03-11T19:12:46.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.355+0000 7fd088d4f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08419ef10 con 0x7fd084101720 2024-03-11T19:12:46.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.355+0000 7fd08259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0841002e0 0x7fd0841970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.355+0000 7fd08259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0841002e0 0x7fd0841970f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52338/0 (socket says 172.21.15.29:52338) 2024-03-11T19:12:46.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.355+0000 7fd08259c640 1 -- 172.21.15.29:0/3732158691 learned_addr learned my addr 172.21.15.29:0/3732158691 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.355+0000 7fd081d9b640 1 --2- 172.21.15.29:0/3732158691 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd084100ca0 0x7fd084197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.355+0000 7fd082d9d640 1 --2- 172.21.15.29:0/3732158691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd084101720 0x7fd08419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.355+0000 7fd08259c640 1 -- 172.21.15.29:0/3732158691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd084101720 msgr2=0x7fd08419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.355+0000 7fd08259c640 1 --2- 172.21.15.29:0/3732158691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd084101720 0x7fd08419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.355+0000 7fd08259c640 1 -- 172.21.15.29:0/3732158691 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd084100ca0 msgr2=0x7fd084197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.355+0000 7fd08259c640 1 --2- 172.21.15.29:0/3732158691 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd084100ca0 0x7fd084197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.355+0000 7fd08259c640 1 -- 172.21.15.29:0/3732158691 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c009900 con 0x7fd0841002e0 2024-03-11T19:12:46.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.356+0000 7fd08259c640 1 --2- 172.21.15.29:0/3732158691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0841002e0 0x7fd0841970f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fd07400e9f0 tx=0x7fd07400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.356+0000 7fd0737fe640 1 -- 172.21.15.29:0/3732158691 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074009720 con 0x7fd0841002e0 2024-03-11T19:12:46.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.356+0000 7fd088d4f640 1 -- 172.21.15.29:0/3732158691 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd08419f1f0 con 0x7fd0841002e0 2024-03-11T19:12:46.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.356+0000 7fd0737fe640 1 -- 172.21.15.29:0/3732158691 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd07400cd80 con 0x7fd0841002e0 2024-03-11T19:12:46.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.356+0000 7fd0737fe640 1 -- 172.21.15.29:0/3732158691 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074010430 con 0x7fd0841002e0 2024-03-11T19:12:46.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.356+0000 7fd088d4f640 1 -- 172.21.15.29:0/3732158691 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd08419f700 con 0x7fd0841002e0 2024-03-11T19:12:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.358+0000 7fd0737fe640 1 -- 172.21.15.29:0/3732158691 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd074005020 con 0x7fd0841002e0 2024-03-11T19:12:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd08ba02740 1 Processor -- start 2024-03-11T19:12:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd0737fe640 1 --2- 172.21.15.29:0/3732158691 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd06c0a1a30 0x7fd06c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd08ba02740 1 -- start start 2024-03-11T19:12:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd082d9d640 1 --2- 172.21.15.29:0/3732158691 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd06c0a1a30 0x7fd06c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd08ba02740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565e5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd08ba02740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565e5c60 0x55a8565ea760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd08ba02740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 0x55a8565ed260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd08ba02740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8565a1960 con 0x55a8565eadd0 2024-03-11T19:12:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd08ba02740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a856555310 con 0x55a8565e5270 2024-03-11T19:12:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd0737fe640 1 -- 172.21.15.29:0/3732158691 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd074014070 con 0x7fd0841002e0 2024-03-11T19:12:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd08ba02740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a856552840 con 0x55a8565e5c60 2024-03-11T19:12:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd082d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 0x55a8565ed260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd082d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 0x55a8565ed260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52344/0 (socket says 172.21.15.29:52344) 2024-03-11T19:12:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd082d9d640 1 -- 172.21.15.29:0/637875822 learned_addr learned my addr 172.21.15.29:0/637875822 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd081d9b640 1 --2- 172.21.15.29:0/637875822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565e5c60 0x55a8565ea760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd08259c640 1 --2- 172.21.15.29:0/637875822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565e5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd081d9b640 1 -- 172.21.15.29:0/637875822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 msgr2=0x55a8565e5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd081d9b640 1 --2- 172.21.15.29:0/637875822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565e5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd081d9b640 1 -- 172.21.15.29:0/637875822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 msgr2=0x55a8565ed260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd081d9b640 1 --2- 172.21.15.29:0/637875822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 0x55a8565ed260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd082d9d640 1 --2- 172.21.15.29:0/3732158691 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd06c0a1a30 0x7fd06c0a3ef0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd07c032460 tx=0x7fd07c034540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd081d9b640 1 -- 172.21.15.29:0/637875822 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a856510060 con 0x55a8565e5c60 2024-03-11T19:12:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.359+0000 7fd08259c640 1 --2- 172.21.15.29:0/637875822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565e5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:12:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.360+0000 7fd081d9b640 1 --2- 172.21.15.29:0/637875822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565e5c60 0x55a8565ea760 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd0841013b0 tx=0x7fd07800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aace92d794dd7530 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.360+0000 7fd070ff9640 1 -- 172.21.15.29:0/637875822 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd078005220 con 0x55a8565e5c60 2024-03-11T19:12:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.360+0000 7fd070ff9640 1 -- 172.21.15.29:0/637875822 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd07800ae70 con 0x55a8565e5c60 2024-03-11T19:12:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.360+0000 7fd070ff9640 1 -- 172.21.15.29:0/637875822 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0780186a0 con 0x55a8565e5c60 2024-03-11T19:12:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.360+0000 7fd08ba02740 1 -- 172.21.15.29:0/637875822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565e5c60 msgr2=0x55a8565ea760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.360+0000 7fd08ba02740 1 --2- 172.21.15.29:0/637875822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565e5c60 0x55a8565ea760 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd0841013b0 tx=0x7fd07800f790 comp rx=0 tx=0).stop 2024-03-11T19:12:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd08ba02740 1 -- 172.21.15.29:0/637875822 shutdown_connections 2024-03-11T19:12:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd08ba02740 1 --2- 172.21.15.29:0/637875822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 0x55a8565ed260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd08ba02740 1 --2- 172.21.15.29:0/637875822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565e5c60 0x55a8565ea760 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd08ba02740 1 --2- 172.21.15.29:0/637875822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565e5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd08ba02740 1 -- 172.21.15.29:0/637875822 >> 172.21.15.29:0/637875822 conn(0x55a8565e07e0 msgr2=0x55a8565e0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd08ba02740 1 -- 172.21.15.29:0/637875822 shutdown_connections 2024-03-11T19:12:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd08ba02740 1 -- 172.21.15.29:0/637875822 wait complete. 2024-03-11T19:12:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd08ba02740 1 -- start start 2024-03-11T19:12:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd08ba02740 1 Processor -- start 2024-03-11T19:12:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd08ba02740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565ef550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd08ba02740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565e5c60 0x55a8565efa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd08ba02740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 0x55a8565f6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd08ba02740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a85654e020 con 0x55a8565eadd0 2024-03-11T19:12:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd08ba02740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a856221430 con 0x55a8565e5270 2024-03-11T19:12:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd08ba02740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8561dde00 con 0x55a8565e5c60 2024-03-11T19:12:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd08ba02740 10 client.0 _ll_register_callbacks cb 0x55a8564f5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd082d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 0x55a8565f6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd081d9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565e5c60 0x55a8565efa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd082d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 0x55a8565f6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52352/0 (socket says 172.21.15.29:52352) 2024-03-11T19:12:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd082d9d640 1 -- 172.21.15.29:0/2678183428 learned_addr learned my addr 172.21.15.29:0/2678183428 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.361+0000 7fd08259c640 1 --2- 172.21.15.29:0/2678183428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565ef550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.362+0000 7fd082d9d640 1 -- 172.21.15.29:0/2678183428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565e5c60 msgr2=0x55a8565efa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.362+0000 7fd082d9d640 1 --2- 172.21.15.29:0/2678183428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565e5c60 0x55a8565efa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.362+0000 7fd082d9d640 1 -- 172.21.15.29:0/2678183428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 msgr2=0x55a8565ef550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.362+0000 7fd082d9d640 1 --2- 172.21.15.29:0/2678183428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565ef550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.362+0000 7fd082d9d640 1 -- 172.21.15.29:0/2678183428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd07800a800 con 0x55a8565eadd0 2024-03-11T19:12:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.362+0000 7fd081d9b640 1 --2- 172.21.15.29:0/2678183428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565e5c60 0x55a8565efa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.362+0000 7fd082d9d640 1 --2- 172.21.15.29:0/2678183428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 0x55a8565f6a60 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd07c02c750 tx=0x7fd07c045040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2547ad76747c714b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.362+0000 7fd069ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:12:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.362+0000 7fd069ffb640 1 -- 172.21.15.29:0/2678183428 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c003b70 con 0x55a8565eadd0 2024-03-11T19:12:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.362+0000 7fd069ffb640 1 -- 172.21.15.29:0/2678183428 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd07c003cd0 con 0x55a8565eadd0 2024-03-11T19:12:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.362+0000 7fd08ba02740 1 -- 172.21.15.29:0/2678183428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a85624d7e0 con 0x55a8565eadd0 2024-03-11T19:12:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.362+0000 7fd069ffb640 1 -- 172.21.15.29:0/2678183428 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd07c003e90 con 0x55a8565eadd0 2024-03-11T19:12:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.362+0000 7fd08ba02740 10 client.16137 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.362+0000 7fd08ba02740 1 -- 172.21.15.29:0/2678183428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a85654c6a0 con 0x55a8565eadd0 2024-03-11T19:12:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.362+0000 7fd069ffb640 1 -- 172.21.15.29:0/2678183428 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55a85624d7e0 con 0x55a8565eadd0 2024-03-11T19:12:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.363+0000 7fd069ffb640 1 -- 172.21.15.29:0/2678183428 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd07c04d450 con 0x55a8565eadd0 2024-03-11T19:12:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.363+0000 7fd08ba02740 10 client.16137 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.363+0000 7fd08ba02740 10 client.16137 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:12:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.363+0000 7fd08ba02740 20 client.16137 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.363+0000 7fd08ba02740 1 --2- 172.21.15.29:0/2678183428 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55a8565fb690 0x55a8565fdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.363+0000 7fd08ba02740 4 client.16137 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:12:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.363+0000 7fd08ba02740 1 -- 172.21.15.29:0/2678183428 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a85628a720 con 0x55a8565fb690 2024-03-11T19:12:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.363+0000 7fd08259c640 1 --2- 172.21.15.29:0/2678183428 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55a8565fb690 0x55a8565fdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.363+0000 7fd08259c640 1 --2- 172.21.15.29:0/2678183428 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55a8565fb690 0x55a8565fdab0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=48cc0f5676d703c6 server_cookie=8fd40e49969e1cd2 in_seq=0 out_seq=0 2024-03-11T19:12:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.363+0000 7fd069ffb640 10 client.16137 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:12:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.364+0000 7fd069ffb640 1 -- 172.21.15.29:0/2678183428 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a85628a720 con 0x55a8565fb690 2024-03-11T19:12:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.364+0000 7fd069ffb640 10 client.16137 handle_command_reply: tid=0 2024-03-11T19:12:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.364+0000 7fd08ba02740 1 client.16137 shutdown 2024-03-11T19:12:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.364+0000 7fd08ba02740 1 -- 172.21.15.29:0/2678183428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 msgr2=0x55a8565f6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.364+0000 7fd08ba02740 1 --2- 172.21.15.29:0/2678183428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 0x55a8565f6a60 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd07c02c750 tx=0x7fd07c045040 comp rx=0 tx=0).stop 2024-03-11T19:12:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.364+0000 7fd08ba02740 1 -- 172.21.15.29:0/2678183428 shutdown_connections 2024-03-11T19:12:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.364+0000 7fd08ba02740 1 --2- 172.21.15.29:0/2678183428 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55a8565fb690 0x55a8565fdab0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.364+0000 7fd08ba02740 1 --2- 172.21.15.29:0/2678183428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 0x55a8565f6a60 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.364+0000 7fd069ffb640 0 client.16137 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:12:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.364+0000 7fd08ba02740 1 --2- 172.21.15.29:0/2678183428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565e5c60 0x55a8565efa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.364+0000 7fd08ba02740 1 --2- 172.21.15.29:0/2678183428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565ef550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.364+0000 7fd08ba02740 1 -- 172.21.15.29:0/2678183428 >> 172.21.15.29:0/2678183428 conn(0x55a8565e07e0 msgr2=0x55a8565dfb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.364+0000 7fd08ba02740 1 -- 172.21.15.29:0/2678183428 shutdown_connections 2024-03-11T19:12:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.364+0000 7fd08ba02740 1 -- 172.21.15.29:0/2678183428 wait complete. 2024-03-11T19:12:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.364+0000 7fd08ba02740 20 client.16137 trim_cache size 0 max 16384 2024-03-11T19:12:46.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd08ba02740 1 Processor -- start 2024-03-11T19:12:46.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd08ba02740 1 -- start start 2024-03-11T19:12:46.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd08ba02740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565e0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd08ba02740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565e5c60 0x55a8565f6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd08ba02740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 0x55a8565ef550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd08ba02740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8564d3130 con 0x55a8565eadd0 2024-03-11T19:12:46.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd08ba02740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8565e0570 con 0x55a8565e5270 2024-03-11T19:12:46.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd08ba02740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8565e6090 con 0x55a8565e5c60 2024-03-11T19:12:46.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd082d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 0x55a8565ef550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd08259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565e0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd082d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 0x55a8565ef550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52358/0 (socket says 172.21.15.29:52358) 2024-03-11T19:12:46.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd082d9d640 1 -- 172.21.15.29:0/1598850133 learned_addr learned my addr 172.21.15.29:0/1598850133 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd08259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565e0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:42834/0 (socket says 172.21.15.29:42834) 2024-03-11T19:12:46.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd081d9b640 1 --2- 172.21.15.29:0/1598850133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565e5c60 0x55a8565f6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd082d9d640 1 -- 172.21.15.29:0/1598850133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565e5c60 msgr2=0x55a8565f6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd082d9d640 1 --2- 172.21.15.29:0/1598850133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565e5c60 0x55a8565f6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd082d9d640 1 -- 172.21.15.29:0/1598850133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 msgr2=0x55a8565e0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd082d9d640 1 --2- 172.21.15.29:0/1598850133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565e0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd082d9d640 1 -- 172.21.15.29:0/1598850133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8564e8c20 con 0x55a8565eadd0 2024-03-11T19:12:46.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd081d9b640 1 --2- 172.21.15.29:0/1598850133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565e5c60 0x55a8565f6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:12:46.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.376+0000 7fd08259c640 1 --2- 172.21.15.29:0/1598850133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565e0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:46.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.377+0000 7fd082d9d640 1 --2- 172.21.15.29:0/1598850133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 0x55a8565ef550 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd07c009430 tx=0x7fd07c034880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4adf55bfeda6b3d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.377+0000 7fd06bfff640 1 -- 172.21.15.29:0/1598850133 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c032ea0 con 0x55a8565eadd0 2024-03-11T19:12:46.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.377+0000 7fd06bfff640 1 -- 172.21.15.29:0/1598850133 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd07c04d450 con 0x55a8565eadd0 2024-03-11T19:12:46.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.377+0000 7fd06bfff640 1 -- 172.21.15.29:0/1598850133 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c0042a0 con 0x55a8565eadd0 2024-03-11T19:12:46.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.377+0000 7fd08ba02740 1 -- 172.21.15.29:0/1598850133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 msgr2=0x55a8565ef550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.377+0000 7fd08ba02740 1 --2- 172.21.15.29:0/1598850133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 0x55a8565ef550 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd07c009430 tx=0x7fd07c034880 comp rx=0 tx=0).stop 2024-03-11T19:12:46.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.377+0000 7fd08ba02740 1 -- 172.21.15.29:0/1598850133 shutdown_connections 2024-03-11T19:12:46.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.377+0000 7fd08ba02740 1 --2- 172.21.15.29:0/1598850133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565eadd0 0x55a8565ef550 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.377+0000 7fd08ba02740 1 --2- 172.21.15.29:0/1598850133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565e5c60 0x55a8565f6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.377+0000 7fd08ba02740 1 --2- 172.21.15.29:0/1598850133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565e0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.377+0000 7fd08ba02740 1 -- 172.21.15.29:0/1598850133 >> 172.21.15.29:0/1598850133 conn(0x55a8565e07e0 msgr2=0x55a8565edfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.377+0000 7fd08ba02740 1 -- 172.21.15.29:0/1598850133 shutdown_connections 2024-03-11T19:12:46.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.377+0000 7fd08ba02740 1 -- 172.21.15.29:0/1598850133 wait complete. 2024-03-11T19:12:46.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd08ba02740 1 -- start start 2024-03-11T19:12:46.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd08ba02740 1 Processor -- start 2024-03-11T19:12:46.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd08ba02740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565f66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd08ba02740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565e5c60 0x55a8565f4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd08ba02740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565eadd0 0x55a8565f52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd08ba02740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8565f57e0 con 0x55a8565e5c60 2024-03-11T19:12:46.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd08ba02740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8565f5950 con 0x55a8565e5270 2024-03-11T19:12:46.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd08ba02740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8565f5b10 con 0x55a8565eadd0 2024-03-11T19:12:46.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd08ba02740 10 client.0 _ll_register_callbacks cb 0x7fd07800a630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:46.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd081d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565e5c60 0x55a8565f4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd081d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565e5c60 0x55a8565f4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52362/0 (socket says 172.21.15.29:52362) 2024-03-11T19:12:46.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd08259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565f66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd081d9b640 1 -- 172.21.15.29:0/3383098897 learned_addr learned my addr 172.21.15.29:0/3383098897 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd082d9d640 1 --2- 172.21.15.29:0/3383098897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565eadd0 0x55a8565f52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd08259c640 1 -- 172.21.15.29:0/3383098897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565eadd0 msgr2=0x55a8565f52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd08259c640 1 --2- 172.21.15.29:0/3383098897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565eadd0 0x55a8565f52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd08259c640 1 -- 172.21.15.29:0/3383098897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565e5c60 msgr2=0x55a8565f4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd08259c640 1 --2- 172.21.15.29:0/3383098897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565e5c60 0x55a8565f4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd08259c640 1 -- 172.21.15.29:0/3383098897 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd08419f700 con 0x55a8565e5270 2024-03-11T19:12:46.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd081d9b640 1 --2- 172.21.15.29:0/3383098897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565e5c60 0x55a8565f4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:46.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd08259c640 1 --2- 172.21.15.29:0/3383098897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565f66e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd0740cb250 tx=0x7fd07400e940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41ae2c3d1cfd4a9a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.378+0000 7fd069ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:12:46.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.379+0000 7fd069ffb640 1 -- 172.21.15.29:0/3383098897 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07409e7c0 con 0x55a8565e5270 2024-03-11T19:12:46.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.379+0000 7fd069ffb640 1 -- 172.21.15.29:0/3383098897 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd074083e40 con 0x55a8565e5270 2024-03-11T19:12:46.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.379+0000 7fd08ba02740 1 -- 172.21.15.29:0/3383098897 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd07800ae70 con 0x55a8565e5270 2024-03-11T19:12:46.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.379+0000 7fd069ffb640 1 -- 172.21.15.29:0/3383098897 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0740a2e10 con 0x55a8565e5270 2024-03-11T19:12:46.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.379+0000 7fd08ba02740 10 client.25673 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:46.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.379+0000 7fd08ba02740 1 -- 172.21.15.29:0/3383098897 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a85654c6a0 con 0x55a8565e5270 2024-03-11T19:12:46.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.379+0000 7fd069ffb640 1 -- 172.21.15.29:0/3383098897 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd07800ae70 con 0x55a8565e5270 2024-03-11T19:12:46.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.379+0000 7fd069ffb640 1 -- 172.21.15.29:0/3383098897 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd07401f060 con 0x55a8565e5270 2024-03-11T19:12:46.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.379+0000 7fd08ba02740 10 client.25673 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:46.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.379+0000 7fd08ba02740 10 client.25673 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:12:46.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.379+0000 7fd08ba02740 20 client.25673 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:46.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.379+0000 7fd08ba02740 1 --2- 172.21.15.29:0/3383098897 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55a856653360 0x55a856655780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.379+0000 7fd08ba02740 4 client.25673 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:12:46.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.379+0000 7fd08ba02740 1 -- 172.21.15.29:0/3383098897 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a8562d9890 con 0x55a856653360 2024-03-11T19:12:46.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.379+0000 7fd081d9b640 1 --2- 172.21.15.29:0/3383098897 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55a856653360 0x55a856655780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.380+0000 7fd081d9b640 1 --2- 172.21.15.29:0/3383098897 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55a856653360 0x55a856655780 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=386889d94cc2e288 server_cookie=2a1449638b6bcfb9 in_seq=0 out_seq=0 2024-03-11T19:12:46.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.380+0000 7fd069ffb640 10 client.25673 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:12:46.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.380+0000 7fd069ffb640 1 -- 172.21.15.29:0/3383098897 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a8562d9890 con 0x55a856653360 2024-03-11T19:12:46.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.380+0000 7fd069ffb640 10 client.25673 handle_command_reply: tid=0 2024-03-11T19:12:46.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.380+0000 7fd08ba02740 1 client.25673 shutdown 2024-03-11T19:12:46.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.380+0000 7fd08ba02740 1 -- 172.21.15.29:0/3383098897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 msgr2=0x55a8565f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.380+0000 7fd08ba02740 1 --2- 172.21.15.29:0/3383098897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565f66e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd0740cb250 tx=0x7fd07400e940 comp rx=0 tx=0).stop 2024-03-11T19:12:46.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.380+0000 7fd08ba02740 1 -- 172.21.15.29:0/3383098897 shutdown_connections 2024-03-11T19:12:46.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.380+0000 7fd08ba02740 1 --2- 172.21.15.29:0/3383098897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8565eadd0 0x55a8565f52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.380+0000 7fd08ba02740 1 --2- 172.21.15.29:0/3383098897 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55a856653360 0x55a856655780 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.380+0000 7fd08ba02740 1 --2- 172.21.15.29:0/3383098897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a8565e5c60 0x55a8565f4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.380+0000 7fd08ba02740 1 --2- 172.21.15.29:0/3383098897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8565e5270 0x55a8565f66e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.380+0000 7fd069ffb640 0 client.25673 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:12:46.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.380+0000 7fd08ba02740 1 -- 172.21.15.29:0/3383098897 >> 172.21.15.29:0/3383098897 conn(0x55a8565e07e0 msgr2=0x55a8565e0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.380+0000 7fd08ba02740 1 -- 172.21.15.29:0/3383098897 shutdown_connections 2024-03-11T19:12:46.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.380+0000 7fd08ba02740 1 -- 172.21.15.29:0/3383098897 wait complete. 2024-03-11T19:12:46.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.380+0000 7fd08ba02740 20 client.25673 trim_cache size 0 max 16384 2024-03-11T19:12:46.391 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:12:46.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.381+0000 7fd088d4f640 1 -- 172.21.15.29:0/3732158691 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd06c0a1a30 msgr2=0x7fd06c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.381+0000 7fd088d4f640 1 --2- 172.21.15.29:0/3732158691 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd06c0a1a30 0x7fd06c0a3ef0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd07c032460 tx=0x7fd07c034540 comp rx=0 tx=0).stop 2024-03-11T19:12:46.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.381+0000 7fd088d4f640 1 -- 172.21.15.29:0/3732158691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0841002e0 msgr2=0x7fd0841970f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.381+0000 7fd088d4f640 1 --2- 172.21.15.29:0/3732158691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0841002e0 0x7fd0841970f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fd07400e9f0 tx=0x7fd07400edf0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.381+0000 7fd088d4f640 1 -- 172.21.15.29:0/3732158691 shutdown_connections 2024-03-11T19:12:46.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.381+0000 7fd088d4f640 1 --2- 172.21.15.29:0/3732158691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd084101720 0x7fd08419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.381+0000 7fd088d4f640 1 --2- 172.21.15.29:0/3732158691 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd084100ca0 0x7fd084197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.381+0000 7fd088d4f640 1 --2- 172.21.15.29:0/3732158691 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd06c0a1a30 0x7fd06c0a3ef0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.381+0000 7fd088d4f640 1 --2- 172.21.15.29:0/3732158691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0841002e0 0x7fd0841970f0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.381+0000 7fd088d4f640 1 -- 172.21.15.29:0/3732158691 >> 172.21.15.29:0/3732158691 conn(0x7fd0840fadb0 msgr2=0x7fd084104520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.381+0000 7fd088d4f640 1 -- 172.21.15.29:0/3732158691 shutdown_connections 2024-03-11T19:12:46.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.381+0000 7fd088d4f640 1 -- 172.21.15.29:0/3732158691 wait complete. 2024-03-11T19:12:46.394 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:12:46.394 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:12:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.528+0000 7f21783f4640 1 -- 172.21.15.29:0/1464839322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2170102ec0 msgr2=0x7f2170109750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.528+0000 7f21783f4640 1 --2- 172.21.15.29:0/1464839322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2170102ec0 0x7f2170109750 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f2168009ca0 tx=0x7f216802c0f0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.528+0000 7f21783f4640 1 -- 172.21.15.29:0/1464839322 shutdown_connections 2024-03-11T19:12:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.528+0000 7f21783f4640 1 --2- 172.21.15.29:0/1464839322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2170102ec0 0x7f2170109750 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.528+0000 7f21783f4640 1 --2- 172.21.15.29:0/1464839322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2170102500 0x7f2170102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.528+0000 7f21783f4640 1 --2- 172.21.15.29:0/1464839322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21700ff490 0x7f21700ff8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.528+0000 7f21783f4640 1 -- 172.21.15.29:0/1464839322 >> 172.21.15.29:0/1464839322 conn(0x7f21700fad80 msgr2=0x7f21700fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.528+0000 7f21783f4640 1 -- 172.21.15.29:0/1464839322 shutdown_connections 2024-03-11T19:12:46.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.528+0000 7f21783f4640 1 -- 172.21.15.29:0/1464839322 wait complete. 2024-03-11T19:12:46.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.529+0000 7f21783f4640 1 Processor -- start 2024-03-11T19:12:46.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.529+0000 7f21783f4640 1 -- start start 2024-03-11T19:12:46.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.529+0000 7f21783f4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21700ff490 0x7f2170197110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.529+0000 7f21783f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2170102500 0x7f2170197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.529+0000 7f21783f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2170102ec0 0x7f217019e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.529+0000 7f21783f4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217019ec00 con 0x7f2170102500 2024-03-11T19:12:46.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.529+0000 7f21783f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217019ed70 con 0x7f2170102ec0 2024-03-11T19:12:46.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.529+0000 7f21783f4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217019ef30 con 0x7f21700ff490 2024-03-11T19:12:46.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.529+0000 7f217598f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2170102500 0x7f2170197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.529+0000 7f217598f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2170102500 0x7f2170197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52380/0 (socket says 172.21.15.29:52380) 2024-03-11T19:12:46.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.529+0000 7f217598f640 1 -- 172.21.15.29:0/3381270502 learned_addr learned my addr 172.21.15.29:0/3381270502 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.529+0000 7f2176991640 1 --2- 172.21.15.29:0/3381270502 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2170102ec0 0x7f217019e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.529+0000 7f2176190640 1 --2- 172.21.15.29:0/3381270502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21700ff490 0x7f2170197110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.530+0000 7f217598f640 1 -- 172.21.15.29:0/3381270502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21700ff490 msgr2=0x7f2170197110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.530+0000 7f217598f640 1 --2- 172.21.15.29:0/3381270502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21700ff490 0x7f2170197110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.530+0000 7f217598f640 1 -- 172.21.15.29:0/3381270502 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2170102ec0 msgr2=0x7f217019e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.530+0000 7f217598f640 1 --2- 172.21.15.29:0/3381270502 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2170102ec0 0x7f217019e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.530+0000 7f217598f640 1 -- 172.21.15.29:0/3381270502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2168009900 con 0x7f2170102500 2024-03-11T19:12:46.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.530+0000 7f217598f640 1 --2- 172.21.15.29:0/3381270502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2170102500 0x7f2170197650 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f216000e9f0 tx=0x7f216000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.530+0000 7f21677fe640 1 -- 172.21.15.29:0/3381270502 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2160009700 con 0x7f2170102500 2024-03-11T19:12:46.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.530+0000 7f21783f4640 1 -- 172.21.15.29:0/3381270502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f217019f210 con 0x7f2170102500 2024-03-11T19:12:46.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.530+0000 7f21677fe640 1 -- 172.21.15.29:0/3381270502 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f216000cd80 con 0x7f2170102500 2024-03-11T19:12:46.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.530+0000 7f21677fe640 1 -- 172.21.15.29:0/3381270502 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2160018840 con 0x7f2170102500 2024-03-11T19:12:46.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.530+0000 7f21783f4640 1 -- 172.21.15.29:0/3381270502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f217019f720 con 0x7f2170102500 2024-03-11T19:12:46.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.532+0000 7f21677fe640 1 -- 172.21.15.29:0/3381270502 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f21600189a0 con 0x7f2170102500 2024-03-11T19:12:46.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.533+0000 7f217afc7740 1 Processor -- start 2024-03-11T19:12:46.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.533+0000 7f217afc7740 1 -- start start 2024-03-11T19:12:46.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.533+0000 7f21677fe640 1 --2- 172.21.15.29:0/3381270502 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f215c0a1a30 0x7f215c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.533+0000 7f217afc7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55967334e270 0x55967334e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.533+0000 7f217afc7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x559673353760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.533+0000 7f217afc7740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559673353dd0 0x559673356260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.533+0000 7f217afc7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55967330a960 con 0x55967334ec60 2024-03-11T19:12:46.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.533+0000 7f217afc7740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596732be310 con 0x559673353dd0 2024-03-11T19:12:46.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.533+0000 7f217afc7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596732bb840 con 0x55967334e270 2024-03-11T19:12:46.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.533+0000 7f2176991640 1 --2- 172.21.15.29:0/3381270502 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f215c0a1a30 0x7f215c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.533+0000 7f21677fe640 1 -- 172.21.15.29:0/3381270502 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f2160014070 con 0x7f2170102500 2024-03-11T19:12:46.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.533+0000 7f217598f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x559673353760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.533+0000 7f2176190640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55967334e270 0x55967334e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.533+0000 7f2176190640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55967334e270 0x55967334e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:36982/0 (socket says 172.21.15.29:36982) 2024-03-11T19:12:46.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.533+0000 7f217598f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x559673353760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52394/0 (socket says 172.21.15.29:52394) 2024-03-11T19:12:46.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.533+0000 7f2176190640 1 -- 172.21.15.29:0/2485113677 learned_addr learned my addr 172.21.15.29:0/2485113677 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.534+0000 7f2176991640 1 --2- 172.21.15.29:0/2485113677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559673353dd0 0x559673356260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.534+0000 7f217598f640 1 -- 172.21.15.29:0/2485113677 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55967334e270 msgr2=0x55967334e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.534+0000 7f217598f640 1 --2- 172.21.15.29:0/2485113677 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55967334e270 0x55967334e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.534+0000 7f217598f640 1 -- 172.21.15.29:0/2485113677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559673353dd0 msgr2=0x559673356260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.534+0000 7f217598f640 1 --2- 172.21.15.29:0/2485113677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559673353dd0 0x559673356260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.534+0000 7f217598f640 1 -- 172.21.15.29:0/2485113677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559673279060 con 0x55967334ec60 2024-03-11T19:12:46.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.534+0000 7f2176190640 1 --2- 172.21.15.29:0/2485113677 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55967334e270 0x55967334e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:46.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.534+0000 7f2176991640 1 --2- 172.21.15.29:0/2485113677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559673353dd0 0x559673356260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:12:46.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.534+0000 7f2176991640 1 --2- 172.21.15.29:0/3381270502 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f215c0a1a30 0x7f215c0a3ef0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f216802c630 tx=0x7f2168032fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.534+0000 7f217598f640 1 --2- 172.21.15.29:0/2485113677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x559673353760 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f216000e9c0 tx=0x7f216009a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a781eaa266a1b37c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.534+0000 7f2164bfe640 1 -- 172.21.15.29:0/2485113677 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216009a750 con 0x55967334ec60 2024-03-11T19:12:46.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.534+0000 7f2164bfe640 1 -- 172.21.15.29:0/2485113677 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f216009e7f0 con 0x55967334ec60 2024-03-11T19:12:46.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.534+0000 7f2164bfe640 1 -- 172.21.15.29:0/2485113677 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2160018c50 con 0x55967334ec60 2024-03-11T19:12:46.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.535+0000 7f217afc7740 1 -- 172.21.15.29:0/2485113677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 msgr2=0x559673353760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.535+0000 7f217afc7740 1 --2- 172.21.15.29:0/2485113677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x559673353760 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f216000e9c0 tx=0x7f216009a650 comp rx=0 tx=0).stop 2024-03-11T19:12:46.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.535+0000 7f217afc7740 1 -- 172.21.15.29:0/2485113677 shutdown_connections 2024-03-11T19:12:46.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.535+0000 7f217afc7740 1 --2- 172.21.15.29:0/2485113677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559673353dd0 0x559673356260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.535+0000 7f217afc7740 1 --2- 172.21.15.29:0/2485113677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x559673353760 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.535+0000 7f217afc7740 1 --2- 172.21.15.29:0/2485113677 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55967334e270 0x55967334e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.535+0000 7f217afc7740 1 -- 172.21.15.29:0/2485113677 >> 172.21.15.29:0/2485113677 conn(0x5596733497e0 msgr2=0x559673349be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.535+0000 7f217afc7740 1 -- 172.21.15.29:0/2485113677 shutdown_connections 2024-03-11T19:12:46.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.535+0000 7f217afc7740 1 -- 172.21.15.29:0/2485113677 wait complete. 2024-03-11T19:12:46.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.535+0000 7f217afc7740 1 -- start start 2024-03-11T19:12:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.535+0000 7f217afc7740 1 Processor -- start 2024-03-11T19:12:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f217afc7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55967334e270 0x5596733584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f217afc7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x5596733589f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f217afc7740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559673353dd0 0x55967335fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f217afc7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596732b7020 con 0x55967334ec60 2024-03-11T19:12:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f217afc7740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559672f8a430 con 0x559673353dd0 2024-03-11T19:12:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f217afc7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559672f46e00 con 0x55967334e270 2024-03-11T19:12:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f217afc7740 10 client.0 _ll_register_callbacks cb 0x55967325ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f217598f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x5596733589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f217598f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x5596733589f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52408/0 (socket says 172.21.15.29:52408) 2024-03-11T19:12:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f217598f640 1 -- 172.21.15.29:0/2816707155 learned_addr learned my addr 172.21.15.29:0/2816707155 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f2176991640 1 --2- 172.21.15.29:0/2816707155 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559673353dd0 0x55967335fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f2176190640 1 --2- 172.21.15.29:0/2816707155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55967334e270 0x5596733584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f217598f640 1 -- 172.21.15.29:0/2816707155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55967334e270 msgr2=0x5596733584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f217598f640 1 --2- 172.21.15.29:0/2816707155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55967334e270 0x5596733584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f217598f640 1 -- 172.21.15.29:0/2816707155 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559673353dd0 msgr2=0x55967335fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f217598f640 1 --2- 172.21.15.29:0/2816707155 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559673353dd0 0x55967335fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f217598f640 1 -- 172.21.15.29:0/2816707155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f217019f720 con 0x55967334ec60 2024-03-11T19:12:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f217598f640 1 --2- 172.21.15.29:0/2816707155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x5596733589f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f2160097120 tx=0x7f2160010e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e57ec7ec3a6dcf server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f2159ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:12:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f2159ffb640 1 -- 172.21.15.29:0/2816707155 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216009a750 con 0x55967334ec60 2024-03-11T19:12:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f217afc7740 1 -- 172.21.15.29:0/2816707155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559672fb67e0 con 0x55967334ec60 2024-03-11T19:12:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f2159ffb640 1 -- 172.21.15.29:0/2816707155 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f21600a29b0 con 0x55967334ec60 2024-03-11T19:12:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f2159ffb640 1 -- 172.21.15.29:0/2816707155 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21600a2b70 con 0x55967334ec60 2024-03-11T19:12:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f217afc7740 10 client.16155 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.536+0000 7f217afc7740 1 -- 172.21.15.29:0/2816707155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5596732b56a0 con 0x55967334ec60 2024-03-11T19:12:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.537+0000 7f2159ffb640 1 -- 172.21.15.29:0/2816707155 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x559672fb67e0 con 0x55967334ec60 2024-03-11T19:12:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.537+0000 7f2159ffb640 1 -- 172.21.15.29:0/2816707155 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2160091020 con 0x55967334ec60 2024-03-11T19:12:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.537+0000 7f217afc7740 10 client.16155 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.537+0000 7f217afc7740 10 client.16155 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:12:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.537+0000 7f217afc7740 20 client.16155 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:46.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.537+0000 7f217afc7740 1 --2- 172.21.15.29:0/2816707155 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5596733642b0 0x5596733666d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.537+0000 7f217afc7740 4 client.16155 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:12:46.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.537+0000 7f217afc7740 1 -- 172.21.15.29:0/2816707155 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559672ff3720 con 0x5596733642b0 2024-03-11T19:12:46.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.537+0000 7f2176190640 1 --2- 172.21.15.29:0/2816707155 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5596733642b0 0x5596733666d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.537+0000 7f2159ffb640 10 client.16155 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:12:46.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.537+0000 7f2176190640 1 --2- 172.21.15.29:0/2816707155 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5596733642b0 0x5596733666d0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4476bfa58e66632a server_cookie=3f9d66b3ef0e9095 in_seq=0 out_seq=0 2024-03-11T19:12:46.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.538+0000 7f2159ffb640 1 -- 172.21.15.29:0/2816707155 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559672ff3720 con 0x5596733642b0 2024-03-11T19:12:46.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.538+0000 7f2159ffb640 10 client.16155 handle_command_reply: tid=0 2024-03-11T19:12:46.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.538+0000 7f217afc7740 1 client.16155 shutdown 2024-03-11T19:12:46.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.538+0000 7f217afc7740 1 -- 172.21.15.29:0/2816707155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 msgr2=0x5596733589f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.538+0000 7f217afc7740 1 --2- 172.21.15.29:0/2816707155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x5596733589f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f2160097120 tx=0x7f2160010e20 comp rx=0 tx=0).stop 2024-03-11T19:12:46.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.538+0000 7f217afc7740 1 -- 172.21.15.29:0/2816707155 shutdown_connections 2024-03-11T19:12:46.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.538+0000 7f217afc7740 1 --2- 172.21.15.29:0/2816707155 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5596733642b0 0x5596733666d0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.538+0000 7f217afc7740 1 --2- 172.21.15.29:0/2816707155 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559673353dd0 0x55967335fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.538+0000 7f217afc7740 1 --2- 172.21.15.29:0/2816707155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x5596733589f0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.538+0000 7f2159ffb640 0 client.16155 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:12:46.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.538+0000 7f217afc7740 1 --2- 172.21.15.29:0/2816707155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55967334e270 0x5596733584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.538+0000 7f217afc7740 1 -- 172.21.15.29:0/2816707155 >> 172.21.15.29:0/2816707155 conn(0x5596733497e0 msgr2=0x559673348b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.538+0000 7f217afc7740 1 -- 172.21.15.29:0/2816707155 shutdown_connections 2024-03-11T19:12:46.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.538+0000 7f217afc7740 1 -- 172.21.15.29:0/2816707155 wait complete. 2024-03-11T19:12:46.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.538+0000 7f217afc7740 20 client.16155 trim_cache size 0 max 16384 2024-03-11T19:12:46.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.555+0000 7f217afc7740 1 Processor -- start 2024-03-11T19:12:46.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.555+0000 7f217afc7740 1 -- start start 2024-03-11T19:12:46.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.555+0000 7f217afc7740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55967334e270 0x559673356fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.555+0000 7f217afc7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x5596733574e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.555+0000 7f217afc7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559673353dd0 0x5596733501b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.556+0000 7f217afc7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55967323c130 con 0x55967334ec60 2024-03-11T19:12:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.556+0000 7f217afc7740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55967333ba40 con 0x55967334e270 2024-03-11T19:12:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.556+0000 7f217afc7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559673356720 con 0x559673353dd0 2024-03-11T19:12:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.556+0000 7f217598f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x5596733574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.556+0000 7f2176190640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55967334e270 0x559673356fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.556+0000 7f2176190640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55967334e270 0x559673356fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:42868/0 (socket says 172.21.15.29:42868) 2024-03-11T19:12:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.556+0000 7f2176991640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559673353dd0 0x5596733501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.556+0000 7f2176190640 1 -- 172.21.15.29:0/249139093 learned_addr learned my addr 172.21.15.29:0/249139093 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.556+0000 7f2176991640 1 -- 172.21.15.29:0/249139093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55967334e270 msgr2=0x559673356fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.556+0000 7f2176991640 1 --2- 172.21.15.29:0/249139093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55967334e270 0x559673356fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.556+0000 7f2176991640 1 -- 172.21.15.29:0/249139093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 msgr2=0x5596733574e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.556+0000 7f2176991640 1 --2- 172.21.15.29:0/249139093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x5596733574e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.556+0000 7f2176991640 1 -- 172.21.15.29:0/249139093 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559673251c20 con 0x559673353dd0 2024-03-11T19:12:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.556+0000 7f2176991640 1 --2- 172.21.15.29:0/249139093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559673353dd0 0x5596733501b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2168037040 tx=0x7f2168046000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ebe85c2e186db26 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.557+0000 7f215bfff640 1 -- 172.21.15.29:0/249139093 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21680328d0 con 0x559673353dd0 2024-03-11T19:12:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.557+0000 7f215bfff640 1 -- 172.21.15.29:0/249139093 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2168032a30 con 0x559673353dd0 2024-03-11T19:12:46.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.557+0000 7f215bfff640 1 -- 172.21.15.29:0/249139093 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2168004280 con 0x559673353dd0 2024-03-11T19:12:46.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.558+0000 7f217afc7740 1 -- 172.21.15.29:0/249139093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559673353dd0 msgr2=0x5596733501b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.558+0000 7f217afc7740 1 --2- 172.21.15.29:0/249139093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559673353dd0 0x5596733501b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2168037040 tx=0x7f2168046000 comp rx=0 tx=0).stop 2024-03-11T19:12:46.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.558+0000 7f217afc7740 1 -- 172.21.15.29:0/249139093 shutdown_connections 2024-03-11T19:12:46.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.558+0000 7f217afc7740 1 --2- 172.21.15.29:0/249139093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559673353dd0 0x5596733501b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.558+0000 7f217afc7740 1 --2- 172.21.15.29:0/249139093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x5596733574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.558+0000 7f217afc7740 1 --2- 172.21.15.29:0/249139093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55967334e270 0x559673356fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.558+0000 7f217afc7740 1 -- 172.21.15.29:0/249139093 >> 172.21.15.29:0/249139093 conn(0x5596733497e0 msgr2=0x55967336b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.558+0000 7f217afc7740 1 -- 172.21.15.29:0/249139093 shutdown_connections 2024-03-11T19:12:46.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.558+0000 7f217afc7740 1 -- 172.21.15.29:0/249139093 wait complete. 2024-03-11T19:12:46.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.558+0000 7f217afc7740 1 -- start start 2024-03-11T19:12:46.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.558+0000 7f217afc7740 1 Processor -- start 2024-03-11T19:12:46.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.559+0000 7f217afc7740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55967334e270 0x5596733b6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.559+0000 7f217afc7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x5596733b7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.559+0000 7f217afc7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559673353dd0 0x5596733be1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.559+0000 7f217afc7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596733be6f0 con 0x55967334ec60 2024-03-11T19:12:46.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.559+0000 7f217afc7740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596733be830 con 0x55967334e270 2024-03-11T19:12:46.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.559+0000 7f217afc7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596733be9a0 con 0x559673353dd0 2024-03-11T19:12:46.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.559+0000 7f217afc7740 10 client.0 _ll_register_callbacks cb 0x7f2160096390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:46.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.559+0000 7f217598f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x5596733b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.559+0000 7f2176190640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55967334e270 0x5596733b6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.559+0000 7f2176190640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55967334e270 0x5596733b6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:42878/0 (socket says 172.21.15.29:42878) 2024-03-11T19:12:46.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.559+0000 7f2176190640 1 -- 172.21.15.29:0/2418995811 learned_addr learned my addr 172.21.15.29:0/2418995811 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.559+0000 7f2176991640 1 --2- 172.21.15.29:0/2418995811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559673353dd0 0x5596733be1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.559+0000 7f2176190640 1 -- 172.21.15.29:0/2418995811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559673353dd0 msgr2=0x5596733be1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.559+0000 7f2176190640 1 --2- 172.21.15.29:0/2418995811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559673353dd0 0x5596733be1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.559+0000 7f2176190640 1 -- 172.21.15.29:0/2418995811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 msgr2=0x5596733b7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.559+0000 7f2176190640 1 --2- 172.21.15.29:0/2418995811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x5596733b7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.559+0000 7f2176190640 1 -- 172.21.15.29:0/2418995811 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2168049070 con 0x55967334e270 2024-03-11T19:12:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.559+0000 7f217598f640 1 --2- 172.21.15.29:0/2418995811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x5596733b7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.560+0000 7f2176190640 1 --2- 172.21.15.29:0/2418995811 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55967334e270 0x5596733b6d10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x559673367020 tx=0x7f216c01f830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a73cad74645fa1d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.560+0000 7f2159ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:12:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.560+0000 7f2159ffb640 1 -- 172.21.15.29:0/2418995811 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c004030 con 0x55967334e270 2024-03-11T19:12:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.560+0000 7f2159ffb640 1 -- 172.21.15.29:0/2418995811 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f216c004190 con 0x55967334e270 2024-03-11T19:12:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.560+0000 7f217afc7740 1 -- 172.21.15.29:0/2418995811 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f216009e7f0 con 0x55967334e270 2024-03-11T19:12:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.560+0000 7f2159ffb640 1 -- 172.21.15.29:0/2418995811 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f216c004390 con 0x55967334e270 2024-03-11T19:12:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.560+0000 7f217afc7740 10 client.25691 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.560+0000 7f217afc7740 1 -- 172.21.15.29:0/2418995811 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5596732778c0 con 0x55967334e270 2024-03-11T19:12:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.560+0000 7f2159ffb640 1 -- 172.21.15.29:0/2418995811 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f216009e7f0 con 0x55967334e270 2024-03-11T19:12:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.560+0000 7f2159ffb640 1 -- 172.21.15.29:0/2418995811 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f216c0274a0 con 0x55967334e270 2024-03-11T19:12:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.561+0000 7f217afc7740 10 client.25691 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.561+0000 7f217afc7740 10 client.25691 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:12:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.561+0000 7f217afc7740 20 client.25691 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.561+0000 7f217afc7740 1 --2- 172.21.15.29:0/2418995811 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5596733c04a0 0x5596733c28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.561+0000 7f217afc7740 4 client.25691 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:12:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.561+0000 7f217afc7740 1 -- 172.21.15.29:0/2418995811 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559673042890 con 0x5596733c04a0 2024-03-11T19:12:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.561+0000 7f217598f640 1 --2- 172.21.15.29:0/2418995811 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5596733c04a0 0x5596733c28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.561+0000 7f2159ffb640 10 client.25691 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:12:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.561+0000 7f217598f640 1 --2- 172.21.15.29:0/2418995811 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5596733c04a0 0x5596733c28c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b7f2fdf78435d9eb server_cookie=28e20163f80b4024 in_seq=0 out_seq=0 2024-03-11T19:12:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.562+0000 7f2159ffb640 1 -- 172.21.15.29:0/2418995811 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559673042890 con 0x5596733c04a0 2024-03-11T19:12:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.562+0000 7f2159ffb640 10 client.25691 handle_command_reply: tid=0 2024-03-11T19:12:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.562+0000 7f217afc7740 1 client.25691 shutdown 2024-03-11T19:12:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.562+0000 7f217afc7740 1 -- 172.21.15.29:0/2418995811 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55967334e270 msgr2=0x5596733b6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.562+0000 7f217afc7740 1 --2- 172.21.15.29:0/2418995811 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55967334e270 0x5596733b6d10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x559673367020 tx=0x7f216c01f830 comp rx=0 tx=0).stop 2024-03-11T19:12:46.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.562+0000 7f217afc7740 1 -- 172.21.15.29:0/2418995811 shutdown_connections 2024-03-11T19:12:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.562+0000 7f217afc7740 1 --2- 172.21.15.29:0/2418995811 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5596733c04a0 0x5596733c28c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.562+0000 7f217afc7740 1 --2- 172.21.15.29:0/2418995811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559673353dd0 0x5596733be1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.562+0000 7f217afc7740 1 --2- 172.21.15.29:0/2418995811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55967334ec60 0x5596733b7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.562+0000 7f2159ffb640 0 client.25691 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:12:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.562+0000 7f217afc7740 1 --2- 172.21.15.29:0/2418995811 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55967334e270 0x5596733b6d10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.562+0000 7f217afc7740 1 -- 172.21.15.29:0/2418995811 >> 172.21.15.29:0/2418995811 conn(0x5596733497e0 msgr2=0x559673356fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.563+0000 7f217afc7740 1 -- 172.21.15.29:0/2418995811 shutdown_connections 2024-03-11T19:12:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.563+0000 7f217afc7740 1 -- 172.21.15.29:0/2418995811 wait complete. 2024-03-11T19:12:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.563+0000 7f217afc7740 20 client.25691 trim_cache size 0 max 16384 2024-03-11T19:12:46.571 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:12:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.564+0000 7f21783f4640 1 -- 172.21.15.29:0/3381270502 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f215c0a1a30 msgr2=0x7f215c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.564+0000 7f21783f4640 1 --2- 172.21.15.29:0/3381270502 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f215c0a1a30 0x7f215c0a3ef0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f216802c630 tx=0x7f2168032fb0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.564+0000 7f21783f4640 1 -- 172.21.15.29:0/3381270502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2170102500 msgr2=0x7f2170197650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.564+0000 7f21783f4640 1 --2- 172.21.15.29:0/3381270502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2170102500 0x7f2170197650 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f216000e9f0 tx=0x7f216000eeb0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.564+0000 7f21783f4640 1 -- 172.21.15.29:0/3381270502 shutdown_connections 2024-03-11T19:12:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.564+0000 7f21783f4640 1 --2- 172.21.15.29:0/3381270502 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2170102ec0 0x7f217019e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.564+0000 7f21783f4640 1 --2- 172.21.15.29:0/3381270502 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f215c0a1a30 0x7f215c0a3ef0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.564+0000 7f21783f4640 1 --2- 172.21.15.29:0/3381270502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2170102500 0x7f2170197650 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.564+0000 7f21783f4640 1 --2- 172.21.15.29:0/3381270502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21700ff490 0x7f2170197110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.564+0000 7f21783f4640 1 -- 172.21.15.29:0/3381270502 >> 172.21.15.29:0/3381270502 conn(0x7f21700fad80 msgr2=0x7f2170105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.565+0000 7f21783f4640 1 -- 172.21.15.29:0/3381270502 shutdown_connections 2024-03-11T19:12:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.565+0000 7f21783f4640 1 -- 172.21.15.29:0/3381270502 wait complete. 2024-03-11T19:12:46.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:46 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/218829801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:12:46.585 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:12:46.585 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:12:46.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.747+0000 7fecfef2c640 1 -- 172.21.15.29:0/3201844455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecf8108990 msgr2=0x7fecf8108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.747+0000 7fecfef2c640 1 --2- 172.21.15.29:0/3201844455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecf8108990 0x7fecf8108e10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fece4009900 tx=0x7fece402bc30 comp rx=0 tx=0).stop 2024-03-11T19:12:46.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.747+0000 7fecfef2c640 1 -- 172.21.15.29:0/3201844455 shutdown_connections 2024-03-11T19:12:46.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.747+0000 7fecfef2c640 1 --2- 172.21.15.29:0/3201844455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecf8109350 0x7fecf810fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.747+0000 7fecfef2c640 1 --2- 172.21.15.29:0/3201844455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecf8108990 0x7fecf8108e10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.747+0000 7fecfef2c640 1 --2- 172.21.15.29:0/3201844455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecf81077a0 0x7fecf8107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.747+0000 7fecfef2c640 1 -- 172.21.15.29:0/3201844455 >> 172.21.15.29:0/3201844455 conn(0x7fecf8076720 msgr2=0x7fecf8078b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.747+0000 7fecfef2c640 1 -- 172.21.15.29:0/3201844455 shutdown_connections 2024-03-11T19:12:46.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.747+0000 7fecfef2c640 1 -- 172.21.15.29:0/3201844455 wait complete. 2024-03-11T19:12:46.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.748+0000 7fecfef2c640 1 Processor -- start 2024-03-11T19:12:46.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.748+0000 7fecfef2c640 1 -- start start 2024-03-11T19:12:46.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.748+0000 7fecfef2c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecf81077a0 0x7fecf819d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.748+0000 7fecfef2c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecf8108990 0x7fecf819dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.748+0000 7fecfef2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecf8109350 0x7fecf81a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.748+0000 7fecfef2c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf81a5270 con 0x7fecf8109350 2024-03-11T19:12:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.748+0000 7fecfef2c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf81a53e0 con 0x7fecf8108990 2024-03-11T19:12:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.748+0000 7fecfef2c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf81a55a0 con 0x7fecf81077a0 2024-03-11T19:12:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.748+0000 7fecfccc8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecf81077a0 0x7fecf819d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.748+0000 7fecfccc8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecf81077a0 0x7fecf819d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37020/0 (socket says 172.21.15.29:37020) 2024-03-11T19:12:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.748+0000 7fecfccc8640 1 -- 172.21.15.29:0/1657512927 learned_addr learned my addr 172.21.15.29:0/1657512927 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.748+0000 7fecf7fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecf8108990 0x7fecf819dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.748+0000 7fecfd4c9640 1 --2- 172.21.15.29:0/1657512927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecf8109350 0x7fecf81a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.749+0000 7fecf7fff640 1 -- 172.21.15.29:0/1657512927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecf81077a0 msgr2=0x7fecf819d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.749+0000 7fecf7fff640 1 --2- 172.21.15.29:0/1657512927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecf81077a0 0x7fecf819d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.749+0000 7fecf7fff640 1 -- 172.21.15.29:0/1657512927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecf8109350 msgr2=0x7fecf81a4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.749+0000 7fecf7fff640 1 --2- 172.21.15.29:0/1657512927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecf8109350 0x7fecf81a4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.749+0000 7fecf7fff640 1 -- 172.21.15.29:0/1657512927 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece4009560 con 0x7fecf8108990 2024-03-11T19:12:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.749+0000 7fecfd4c9640 1 --2- 172.21.15.29:0/1657512927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecf8109350 0x7fecf81a4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.749+0000 7fecf7fff640 1 --2- 172.21.15.29:0/1657512927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecf8108990 0x7fecf819dcc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fece4000c00 tx=0x7fece40042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.749+0000 7fecf5ffb640 1 -- 172.21.15.29:0/1657512927 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4004aa0 con 0x7fecf8108990 2024-03-11T19:12:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.749+0000 7fecf5ffb640 1 -- 172.21.15.29:0/1657512927 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fece402de90 con 0x7fecf8108990 2024-03-11T19:12:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.749+0000 7fecf5ffb640 1 -- 172.21.15.29:0/1657512927 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece40053f0 con 0x7fecf8108990 2024-03-11T19:12:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.749+0000 7fecfef2c640 1 -- 172.21.15.29:0/1657512927 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf81a5820 con 0x7fecf8108990 2024-03-11T19:12:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.750+0000 7fecfef2c640 1 -- 172.21.15.29:0/1657512927 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf81a5bf0 con 0x7fecf8108990 2024-03-11T19:12:46.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.753+0000 7fed01bdc740 1 Processor -- start 2024-03-11T19:12:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.753+0000 7fecf5ffb640 1 -- 172.21.15.29:0/1657512927 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fece4005550 con 0x7fecf8108990 2024-03-11T19:12:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.753+0000 7fed01bdc740 1 -- start start 2024-03-11T19:12:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.753+0000 7fed01bdc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979270 0x561149979690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.753+0000 7fed01bdc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979c60 0x56114997e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.753+0000 7fed01bdc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56114997edd0 0x561149981260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.753+0000 7fed01bdc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x561149935960 con 0x56114997edd0 2024-03-11T19:12:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.753+0000 7fed01bdc740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611498e9310 con 0x561149979270 2024-03-11T19:12:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.753+0000 7fed01bdc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611498e6840 con 0x561149979c60 2024-03-11T19:12:46.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.754+0000 7fecfccc8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979c60 0x56114997e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.754+0000 7fecf7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56114997edd0 0x561149981260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.754+0000 7fecfccc8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979c60 0x56114997e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37024/0 (socket says 172.21.15.29:37024) 2024-03-11T19:12:46.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.754+0000 7fecfccc8640 1 -- 172.21.15.29:0/724266689 learned_addr learned my addr 172.21.15.29:0/724266689 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.754+0000 7fecfd4c9640 1 --2- 172.21.15.29:0/724266689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979270 0x561149979690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.754+0000 7fecfccc8640 1 -- 172.21.15.29:0/724266689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979270 msgr2=0x561149979690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.754+0000 7fecfccc8640 1 --2- 172.21.15.29:0/724266689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979270 0x561149979690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.754+0000 7fecfccc8640 1 -- 172.21.15.29:0/724266689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56114997edd0 msgr2=0x561149981260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.754+0000 7fecfccc8640 1 --2- 172.21.15.29:0/724266689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56114997edd0 0x561149981260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.754+0000 7fecfccc8640 1 -- 172.21.15.29:0/724266689 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611498a4060 con 0x561149979c60 2024-03-11T19:12:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.754+0000 7fecfd4c9640 1 --2- 172.21.15.29:0/724266689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979270 0x561149979690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:12:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.754+0000 7fecfccc8640 1 --2- 172.21.15.29:0/724266689 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979c60 0x56114997e760 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fecec00be50 tx=0x7fecec0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=489102aa2e49142c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.754+0000 7fece2ffd640 1 -- 172.21.15.29:0/724266689 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec00b950 con 0x561149979c60 2024-03-11T19:12:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.755+0000 7fece2ffd640 1 -- 172.21.15.29:0/724266689 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fecec015070 con 0x561149979c60 2024-03-11T19:12:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.755+0000 7fece2ffd640 1 -- 172.21.15.29:0/724266689 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec019960 con 0x561149979c60 2024-03-11T19:12:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.755+0000 7fecf5ffb640 1 --2- 172.21.15.29:0/1657512927 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fece80a1b30 0x7fece80a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.755+0000 7fecfd4c9640 1 --2- 172.21.15.29:0/1657512927 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fece80a1b30 0x7fece80a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.756+0000 7fed01bdc740 1 -- 172.21.15.29:0/724266689 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979c60 msgr2=0x56114997e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.756+0000 7fed01bdc740 1 --2- 172.21.15.29:0/724266689 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979c60 0x56114997e760 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fecec00be50 tx=0x7fecec0107a0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.756+0000 7fecf5ffb640 1 -- 172.21.15.29:0/1657512927 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fece40ef390 con 0x7fecf8108990 2024-03-11T19:12:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.756+0000 7fed01bdc740 1 -- 172.21.15.29:0/724266689 shutdown_connections 2024-03-11T19:12:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.756+0000 7fed01bdc740 1 --2- 172.21.15.29:0/724266689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56114997edd0 0x561149981260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.756+0000 7fed01bdc740 1 --2- 172.21.15.29:0/724266689 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979c60 0x56114997e760 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.756+0000 7fed01bdc740 1 --2- 172.21.15.29:0/724266689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979270 0x561149979690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.756+0000 7fed01bdc740 1 -- 172.21.15.29:0/724266689 >> 172.21.15.29:0/724266689 conn(0x5611499747e0 msgr2=0x561149974be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.756+0000 7fed01bdc740 1 -- 172.21.15.29:0/724266689 shutdown_connections 2024-03-11T19:12:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.756+0000 7fecfd4c9640 1 --2- 172.21.15.29:0/1657512927 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fece80a1b30 0x7fece80a3ff0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x56114997a4c0 tx=0x7fecf0009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.756+0000 7fed01bdc740 1 -- 172.21.15.29:0/724266689 wait complete. 2024-03-11T19:12:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.756+0000 7fed01bdc740 1 -- start start 2024-03-11T19:12:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.757+0000 7fed01bdc740 1 Processor -- start 2024-03-11T19:12:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.757+0000 7fed01bdc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979270 0x561149994470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.757+0000 7fed01bdc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979c60 0x5611499949b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.757+0000 7fecfccc8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979270 0x561149994470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.758+0000 7fecfccc8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979270 0x561149994470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:42902/0 (socket says 172.21.15.29:42902) 2024-03-11T19:12:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.758+0000 7fecfccc8640 1 -- 172.21.15.29:0/2041173381 learned_addr learned my addr 172.21.15.29:0/2041173381 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.758+0000 7fecf7fff640 1 --2- 172.21.15.29:0/2041173381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979c60 0x5611499949b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.758+0000 7fed01bdc740 1 --2- 172.21.15.29:0/2041173381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56114997edd0 0x56114999b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.758+0000 7fed01bdc740 1 -- 172.21.15.29:0/2041173381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611498e2020 con 0x56114997edd0 2024-03-11T19:12:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.758+0000 7fed01bdc740 1 -- 172.21.15.29:0/2041173381 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611495b5430 con 0x561149979270 2024-03-11T19:12:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.758+0000 7fed01bdc740 1 -- 172.21.15.29:0/2041173381 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561149571e00 con 0x561149979c60 2024-03-11T19:12:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.758+0000 7fed01bdc740 10 client.0 _ll_register_callbacks cb 0x561149889cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.758+0000 7fecfd4c9640 1 --2- 172.21.15.29:0/2041173381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56114997edd0 0x56114999b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.764 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:46 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/218829801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:12:46.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.759+0000 7fecfccc8640 1 -- 172.21.15.29:0/2041173381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979c60 msgr2=0x5611499949b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.759+0000 7fecfccc8640 1 --2- 172.21.15.29:0/2041173381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979c60 0x5611499949b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.759+0000 7fecfccc8640 1 -- 172.21.15.29:0/2041173381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56114997edd0 msgr2=0x56114999b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.759+0000 7fecfccc8640 1 --2- 172.21.15.29:0/2041173381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56114997edd0 0x56114999b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.759+0000 7fecfccc8640 1 -- 172.21.15.29:0/2041173381 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecec00bab0 con 0x561149979270 2024-03-11T19:12:46.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.759+0000 7fecfccc8640 1 --2- 172.21.15.29:0/2041173381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979270 0x561149994470 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fecec01a000 tx=0x7fecec005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d7c29c50f4690ad server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.759+0000 7fecd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:12:46.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.759+0000 7fecd7fff640 1 -- 172.21.15.29:0/2041173381 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec005260 con 0x561149979270 2024-03-11T19:12:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.759+0000 7fecd7fff640 1 -- 172.21.15.29:0/2041173381 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fecec0053c0 con 0x561149979270 2024-03-11T19:12:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.759+0000 7fed01bdc740 1 -- 172.21.15.29:0/2041173381 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5611495e17e0 con 0x561149979270 2024-03-11T19:12:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.760+0000 7fecd7fff640 1 -- 172.21.15.29:0/2041173381 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fecec005580 con 0x561149979270 2024-03-11T19:12:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.760+0000 7fed01bdc740 10 client.25700 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.760+0000 7fed01bdc740 1 -- 172.21.15.29:0/2041173381 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5611498e06a0 con 0x561149979270 2024-03-11T19:12:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.760+0000 7fecd7fff640 1 -- 172.21.15.29:0/2041173381 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5611495e17e0 con 0x561149979270 2024-03-11T19:12:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.760+0000 7fecd7fff640 1 -- 172.21.15.29:0/2041173381 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fecec0199a0 con 0x561149979270 2024-03-11T19:12:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.760+0000 7fed01bdc740 10 client.25700 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.760+0000 7fed01bdc740 10 client.25700 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:12:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.760+0000 7fed01bdc740 20 client.25700 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.760+0000 7fed01bdc740 1 --2- 172.21.15.29:0/2041173381 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5611499a0480 0x5611499a2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.760+0000 7fed01bdc740 4 client.25700 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:12:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.760+0000 7fed01bdc740 1 -- 172.21.15.29:0/2041173381 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56114961e720 con 0x5611499a0480 2024-03-11T19:12:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.760+0000 7fecf7fff640 1 --2- 172.21.15.29:0/2041173381 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5611499a0480 0x5611499a2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.761+0000 7fecd7fff640 10 client.25700 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:12:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.761+0000 7fecf7fff640 1 --2- 172.21.15.29:0/2041173381 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5611499a0480 0x5611499a2870 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=654a7dbe65e24310 server_cookie=245d96cf673a2d6f in_seq=0 out_seq=0 2024-03-11T19:12:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.762+0000 7fecd7fff640 1 -- 172.21.15.29:0/2041173381 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56114961e720 con 0x5611499a0480 2024-03-11T19:12:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.762+0000 7fecd7fff640 10 client.25700 handle_command_reply: tid=0 2024-03-11T19:12:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.762+0000 7fed01bdc740 1 client.25700 shutdown 2024-03-11T19:12:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.762+0000 7fed01bdc740 1 -- 172.21.15.29:0/2041173381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979270 msgr2=0x561149994470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.762+0000 7fed01bdc740 1 --2- 172.21.15.29:0/2041173381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979270 0x561149994470 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fecec01a000 tx=0x7fecec005040 comp rx=0 tx=0).stop 2024-03-11T19:12:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.762+0000 7fed01bdc740 1 -- 172.21.15.29:0/2041173381 shutdown_connections 2024-03-11T19:12:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.762+0000 7fed01bdc740 1 --2- 172.21.15.29:0/2041173381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56114997edd0 0x56114999b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.762+0000 7fed01bdc740 1 --2- 172.21.15.29:0/2041173381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979c60 0x5611499949b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.762+0000 7fed01bdc740 1 --2- 172.21.15.29:0/2041173381 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5611499a0480 0x5611499a2870 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.762+0000 7fed01bdc740 1 --2- 172.21.15.29:0/2041173381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979270 0x561149994470 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.762+0000 7fed01bdc740 1 -- 172.21.15.29:0/2041173381 >> 172.21.15.29:0/2041173381 conn(0x5611499747e0 msgr2=0x561149973b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.762+0000 7fecd7fff640 0 client.25700 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:12:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.762+0000 7fed01bdc740 1 -- 172.21.15.29:0/2041173381 shutdown_connections 2024-03-11T19:12:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.763+0000 7fed01bdc740 1 -- 172.21.15.29:0/2041173381 wait complete. 2024-03-11T19:12:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.763+0000 7fed01bdc740 20 client.25700 trim_cache size 0 max 16384 2024-03-11T19:12:46.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.781+0000 7fed01bdc740 1 Processor -- start 2024-03-11T19:12:46.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.781+0000 7fed01bdc740 1 -- start start 2024-03-11T19:12:46.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.782+0000 7fed01bdc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561149979270 0x56114997b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.782+0000 7fed01bdc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979c60 0x56114997b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.782+0000 7fecfccc8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561149979270 0x56114997b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.782+0000 7fed01bdc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56114997edd0 0x56114999b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.782+0000 7fecfccc8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561149979270 0x56114997b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52462/0 (socket says 172.21.15.29:52462) 2024-03-11T19:12:46.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.782+0000 7fecfccc8640 1 -- 172.21.15.29:0/1348524832 learned_addr learned my addr 172.21.15.29:0/1348524832 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.782+0000 7fecf7fff640 1 --2- 172.21.15.29:0/1348524832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979c60 0x56114997b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.782+0000 7fecfd4c9640 1 --2- 172.21.15.29:0/1348524832 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56114997edd0 0x56114999b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.782+0000 7fed01bdc740 1 -- 172.21.15.29:0/1348524832 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x561149867130 con 0x561149979270 2024-03-11T19:12:46.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.782+0000 7fed01bdc740 1 -- 172.21.15.29:0/1348524832 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56114997a0d0 con 0x56114997edd0 2024-03-11T19:12:46.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.782+0000 7fed01bdc740 1 -- 172.21.15.29:0/1348524832 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611499752d0 con 0x561149979c60 2024-03-11T19:12:46.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.783+0000 7fecfccc8640 1 -- 172.21.15.29:0/1348524832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979c60 msgr2=0x56114997b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.783+0000 7fecfccc8640 1 --2- 172.21.15.29:0/1348524832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979c60 0x56114997b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.783+0000 7fecfccc8640 1 -- 172.21.15.29:0/1348524832 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56114997edd0 msgr2=0x56114999b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.783+0000 7fecfccc8640 1 --2- 172.21.15.29:0/1348524832 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56114997edd0 0x56114999b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.783+0000 7fecfccc8640 1 -- 172.21.15.29:0/1348524832 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56114987cc20 con 0x561149979270 2024-03-11T19:12:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.783+0000 7fecfccc8640 1 --2- 172.21.15.29:0/1348524832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561149979270 0x56114997b1b0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fecec000c00 tx=0x7fecec010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705fbeec7a61f160 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.784+0000 7fece27fc640 1 -- 172.21.15.29:0/1348524832 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec011440 con 0x561149979270 2024-03-11T19:12:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.784+0000 7fece27fc640 1 -- 172.21.15.29:0/1348524832 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fecec005560 con 0x561149979270 2024-03-11T19:12:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.784+0000 7fece27fc640 1 -- 172.21.15.29:0/1348524832 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec0193d0 con 0x561149979270 2024-03-11T19:12:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.785+0000 7fed01bdc740 1 -- 172.21.15.29:0/1348524832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561149979270 msgr2=0x56114997b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.785+0000 7fed01bdc740 1 --2- 172.21.15.29:0/1348524832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561149979270 0x56114997b1b0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fecec000c00 tx=0x7fecec010c20 comp rx=0 tx=0).stop 2024-03-11T19:12:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.785+0000 7fed01bdc740 1 -- 172.21.15.29:0/1348524832 shutdown_connections 2024-03-11T19:12:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.785+0000 7fed01bdc740 1 --2- 172.21.15.29:0/1348524832 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56114997edd0 0x56114999b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.785+0000 7fed01bdc740 1 --2- 172.21.15.29:0/1348524832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979c60 0x56114997b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.785+0000 7fed01bdc740 1 --2- 172.21.15.29:0/1348524832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561149979270 0x56114997b1b0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.785+0000 7fed01bdc740 1 -- 172.21.15.29:0/1348524832 >> 172.21.15.29:0/1348524832 conn(0x5611499747e0 msgr2=0x561149974bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.785+0000 7fed01bdc740 1 -- 172.21.15.29:0/1348524832 shutdown_connections 2024-03-11T19:12:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.785+0000 7fed01bdc740 1 -- 172.21.15.29:0/1348524832 wait complete. 2024-03-11T19:12:46.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.786+0000 7fed01bdc740 1 -- start start 2024-03-11T19:12:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.786+0000 7fed01bdc740 1 Processor -- start 2024-03-11T19:12:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.787+0000 7fed01bdc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979270 0x56114999b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.787+0000 7fed01bdc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979c60 0x561149999c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.787+0000 7fed01bdc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56114997edd0 0x56114999a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.787+0000 7fed01bdc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56114999a6e0 con 0x56114997edd0 2024-03-11T19:12:46.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.787+0000 7fed01bdc740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56114999a850 con 0x561149979c60 2024-03-11T19:12:46.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.787+0000 7fed01bdc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56114999aa10 con 0x561149979270 2024-03-11T19:12:46.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.787+0000 7fed01bdc740 10 client.0 _ll_register_callbacks cb 0x7fecec00b880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:46.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.788+0000 7fecf7fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979c60 0x561149999c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.788+0000 7fecfccc8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979270 0x56114999b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.788+0000 7fecf7fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979c60 0x561149999c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:42918/0 (socket says 172.21.15.29:42918) 2024-03-11T19:12:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.788+0000 7fecf7fff640 1 -- 172.21.15.29:0/3260996349 learned_addr learned my addr 172.21.15.29:0/3260996349 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.788+0000 7fecfd4c9640 1 --2- 172.21.15.29:0/3260996349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56114997edd0 0x56114999a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.788+0000 7fecf7fff640 1 -- 172.21.15.29:0/3260996349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979270 msgr2=0x56114999b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.788+0000 7fecf7fff640 1 --2- 172.21.15.29:0/3260996349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979270 0x56114999b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.788+0000 7fecf7fff640 1 -- 172.21.15.29:0/3260996349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56114997edd0 msgr2=0x56114999a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.788+0000 7fecf7fff640 1 --2- 172.21.15.29:0/3260996349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56114997edd0 0x56114999a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.788+0000 7fecf7fff640 1 -- 172.21.15.29:0/3260996349 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5611498e06a0 con 0x561149979c60 2024-03-11T19:12:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.788+0000 7fecfd4c9640 1 --2- 172.21.15.29:0/3260996349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56114997edd0 0x56114999a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.789+0000 7fecf7fff640 1 --2- 172.21.15.29:0/3260996349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979c60 0x561149999c60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fece40bc000 tx=0x7fece40f2e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d1d3e2ea146c893 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.789+0000 7fecd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:12:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.789+0000 7fecd7fff640 1 -- 172.21.15.29:0/3260996349 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece40bc0d0 con 0x561149979c60 2024-03-11T19:12:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.789+0000 7fecd7fff640 1 -- 172.21.15.29:0/3260996349 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fece40bca00 con 0x561149979c60 2024-03-11T19:12:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.789+0000 7fed01bdc740 1 -- 172.21.15.29:0/3260996349 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fecec015070 con 0x561149979c60 2024-03-11T19:12:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.789+0000 7fecd7fff640 1 -- 172.21.15.29:0/3260996349 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fece40bbe40 con 0x561149979c60 2024-03-11T19:12:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.789+0000 7fed01bdc740 10 client.25706 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.789+0000 7fed01bdc740 1 -- 172.21.15.29:0/3260996349 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5611498a28c0 con 0x561149979c60 2024-03-11T19:12:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.790+0000 7fecd7fff640 1 -- 172.21.15.29:0/3260996349 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fecec015070 con 0x561149979c60 2024-03-11T19:12:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.790+0000 7fecd7fff640 1 -- 172.21.15.29:0/3260996349 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fece40ee070 con 0x561149979c60 2024-03-11T19:12:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.790+0000 7fed01bdc740 10 client.25706 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.790+0000 7fed01bdc740 10 client.25706 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:12:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.790+0000 7fed01bdc740 20 client.25706 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.790+0000 7fed01bdc740 1 --2- 172.21.15.29:0/3260996349 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5611499f7c40 0x5611499fa060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.790+0000 7fed01bdc740 4 client.25706 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:12:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.790+0000 7fed01bdc740 1 -- 172.21.15.29:0/3260996349 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56114966d890 con 0x5611499f7c40 2024-03-11T19:12:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.790+0000 7fecfccc8640 1 --2- 172.21.15.29:0/3260996349 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5611499f7c40 0x5611499fa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.791+0000 7fecd7fff640 10 client.25706 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:12:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.791+0000 7fecfccc8640 1 --2- 172.21.15.29:0/3260996349 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5611499f7c40 0x5611499fa060 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=999a70b66e57753 server_cookie=1ed2f6545d08d332 in_seq=0 out_seq=0 2024-03-11T19:12:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.791+0000 7fecd7fff640 1 -- 172.21.15.29:0/3260996349 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56114966d890 con 0x5611499f7c40 2024-03-11T19:12:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.791+0000 7fecd7fff640 10 client.25706 handle_command_reply: tid=0 2024-03-11T19:12:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.791+0000 7fed01bdc740 1 client.25706 shutdown 2024-03-11T19:12:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.791+0000 7fed01bdc740 1 -- 172.21.15.29:0/3260996349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979c60 msgr2=0x561149999c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.791+0000 7fed01bdc740 1 --2- 172.21.15.29:0/3260996349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979c60 0x561149999c60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fece40bc000 tx=0x7fece40f2e50 comp rx=0 tx=0).stop 2024-03-11T19:12:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.791+0000 7fed01bdc740 1 -- 172.21.15.29:0/3260996349 shutdown_connections 2024-03-11T19:12:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.791+0000 7fed01bdc740 1 --2- 172.21.15.29:0/3260996349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56114997edd0 0x56114999a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.791+0000 7fed01bdc740 1 --2- 172.21.15.29:0/3260996349 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5611499f7c40 0x5611499fa060 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.791+0000 7fed01bdc740 1 --2- 172.21.15.29:0/3260996349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561149979c60 0x561149999c60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.791+0000 7fed01bdc740 1 --2- 172.21.15.29:0/3260996349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561149979270 0x56114999b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.791+0000 7fecd7fff640 0 client.25706 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:12:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.791+0000 7fed01bdc740 1 -- 172.21.15.29:0/3260996349 >> 172.21.15.29:0/3260996349 conn(0x5611499747e0 msgr2=0x56114997b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.791+0000 7fed01bdc740 1 -- 172.21.15.29:0/3260996349 shutdown_connections 2024-03-11T19:12:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.792+0000 7fed01bdc740 1 -- 172.21.15.29:0/3260996349 wait complete. 2024-03-11T19:12:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.792+0000 7fed01bdc740 20 client.25706 trim_cache size 0 max 16384 2024-03-11T19:12:46.797 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:12:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.793+0000 7fecfef2c640 1 -- 172.21.15.29:0/1657512927 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fece80a1b30 msgr2=0x7fece80a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.793+0000 7fecfef2c640 1 --2- 172.21.15.29:0/1657512927 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fece80a1b30 0x7fece80a3ff0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x56114997a4c0 tx=0x7fecf0009500 comp rx=0 tx=0).stop 2024-03-11T19:12:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.793+0000 7fecfef2c640 1 -- 172.21.15.29:0/1657512927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecf8108990 msgr2=0x7fecf819dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.793+0000 7fecfef2c640 1 --2- 172.21.15.29:0/1657512927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecf8108990 0x7fecf819dcc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fece4000c00 tx=0x7fece40042f0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.793+0000 7fecfef2c640 1 -- 172.21.15.29:0/1657512927 shutdown_connections 2024-03-11T19:12:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.793+0000 7fecfef2c640 1 --2- 172.21.15.29:0/1657512927 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fece80a1b30 0x7fece80a3ff0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.793+0000 7fecfef2c640 1 --2- 172.21.15.29:0/1657512927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecf8109350 0x7fecf81a4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.793+0000 7fecfef2c640 1 --2- 172.21.15.29:0/1657512927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecf8108990 0x7fecf819dcc0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fece4000c00 tx=0x7fece40042f0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.793+0000 7fecfef2c640 1 --2- 172.21.15.29:0/1657512927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecf81077a0 0x7fecf819d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.793+0000 7fecfef2c640 1 -- 172.21.15.29:0/1657512927 >> 172.21.15.29:0/1657512927 conn(0x7fecf8076720 msgr2=0x7fecf810bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.793+0000 7fecfef2c640 1 -- 172.21.15.29:0/1657512927 shutdown_connections 2024-03-11T19:12:46.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.793+0000 7fecfef2c640 1 -- 172.21.15.29:0/1657512927 wait complete. 2024-03-11T19:12:46.813 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:12:46.813 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:12:46.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:46 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/218829801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:12:46.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.978+0000 7ff36ba2e640 1 -- 172.21.15.29:0/1129295294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff364108990 msgr2=0x7ff364108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.978+0000 7ff36ba2e640 1 --2- 172.21.15.29:0/1129295294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff364108990 0x7ff364108e10 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff354009900 tx=0x7ff35402bc30 comp rx=0 tx=0).stop 2024-03-11T19:12:46.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.978+0000 7ff36ba2e640 1 -- 172.21.15.29:0/1129295294 shutdown_connections 2024-03-11T19:12:46.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.978+0000 7ff36ba2e640 1 --2- 172.21.15.29:0/1129295294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff364109350 0x7ff36410fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.978+0000 7ff36ba2e640 1 --2- 172.21.15.29:0/1129295294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff364108990 0x7ff364108e10 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.978+0000 7ff36ba2e640 1 --2- 172.21.15.29:0/1129295294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3641077a0 0x7ff364107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.978+0000 7ff36ba2e640 1 -- 172.21.15.29:0/1129295294 >> 172.21.15.29:0/1129295294 conn(0x7ff364076730 msgr2=0x7ff364078b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.978+0000 7ff36ba2e640 1 -- 172.21.15.29:0/1129295294 shutdown_connections 2024-03-11T19:12:46.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.978+0000 7ff36ba2e640 1 -- 172.21.15.29:0/1129295294 wait complete. 2024-03-11T19:12:46.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.979+0000 7ff36ba2e640 1 Processor -- start 2024-03-11T19:12:46.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.979+0000 7ff36ba2e640 1 -- start start 2024-03-11T19:12:46.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.979+0000 7ff36ba2e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3641077a0 0x7ff36407fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.979+0000 7ff36ba2e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff364108990 0x7ff3640800a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.979+0000 7ff36ba2e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff364109350 0x7ff3640810e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.979+0000 7ff36ba2e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff364081620 con 0x7ff364108990 2024-03-11T19:12:46.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.979+0000 7ff36ba2e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff364081790 con 0x7ff364109350 2024-03-11T19:12:46.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.979+0000 7ff36ba2e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff364081950 con 0x7ff3641077a0 2024-03-11T19:12:46.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.980+0000 7ff3697ca640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3641077a0 0x7ff36407fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.980+0000 7ff368fc9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff364108990 0x7ff3640800a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.980+0000 7ff3697ca640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3641077a0 0x7ff36407fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37074/0 (socket says 172.21.15.29:37074) 2024-03-11T19:12:46.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.980+0000 7ff368fc9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff364108990 0x7ff3640800a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52494/0 (socket says 172.21.15.29:52494) 2024-03-11T19:12:46.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.980+0000 7ff3697ca640 1 -- 172.21.15.29:0/2500843995 learned_addr learned my addr 172.21.15.29:0/2500843995 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.980+0000 7ff369fcb640 1 --2- 172.21.15.29:0/2500843995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff364109350 0x7ff3640810e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.980+0000 7ff3697ca640 1 -- 172.21.15.29:0/2500843995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff364109350 msgr2=0x7ff3640810e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.980+0000 7ff3697ca640 1 --2- 172.21.15.29:0/2500843995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff364109350 0x7ff3640810e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.980+0000 7ff3697ca640 1 -- 172.21.15.29:0/2500843995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff364108990 msgr2=0x7ff3640800a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.980+0000 7ff3697ca640 1 --2- 172.21.15.29:0/2500843995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff364108990 0x7ff3640800a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.980+0000 7ff3697ca640 1 -- 172.21.15.29:0/2500843995 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff354009560 con 0x7ff3641077a0 2024-03-11T19:12:46.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.980+0000 7ff3697ca640 1 --2- 172.21.15.29:0/2500843995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3641077a0 0x7ff36407fb60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff36000e9c0 tx=0x7ff36000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.981+0000 7ff35a7fc640 1 -- 172.21.15.29:0/2500843995 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff360009700 con 0x7ff3641077a0 2024-03-11T19:12:46.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.981+0000 7ff35a7fc640 1 -- 172.21.15.29:0/2500843995 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff3600044d0 con 0x7ff3641077a0 2024-03-11T19:12:46.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.981+0000 7ff35a7fc640 1 -- 172.21.15.29:0/2500843995 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff360010430 con 0x7ff3641077a0 2024-03-11T19:12:46.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.981+0000 7ff36ba2e640 1 -- 172.21.15.29:0/2500843995 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff364081c30 con 0x7ff3641077a0 2024-03-11T19:12:46.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.981+0000 7ff36ba2e640 1 -- 172.21.15.29:0/2500843995 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3641ae650 con 0x7ff3641077a0 2024-03-11T19:12:46.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.984+0000 7ff35a7fc640 1 -- 172.21.15.29:0/2500843995 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff360003f30 con 0x7ff3641077a0 2024-03-11T19:12:46.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.985+0000 7ff35a7fc640 1 --2- 172.21.15.29:0/2500843995 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff3500a1b00 0x7ff3500a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.985+0000 7ff36e603740 1 Processor -- start 2024-03-11T19:12:46.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.985+0000 7ff35a7fc640 1 -- 172.21.15.29:0/2500843995 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff360014070 con 0x7ff3641077a0 2024-03-11T19:12:46.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.985+0000 7ff36e603740 1 -- start start 2024-03-11T19:12:46.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.985+0000 7ff36e603740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376b270 0x562b5376b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.985+0000 7ff36e603740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5376bc60 0x562b53770760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.985+0000 7ff36e603740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 0x562b53773260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.985+0000 7ff36e603740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b53727960 con 0x562b5376b270 2024-03-11T19:12:46.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.985+0000 7ff36e603740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b536db310 con 0x562b5376bc60 2024-03-11T19:12:46.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.985+0000 7ff36e603740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b536d8840 con 0x562b53770dd0 2024-03-11T19:12:46.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.985+0000 7ff368fc9640 1 --2- 172.21.15.29:0/2500843995 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff3500a1b00 0x7ff3500a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.986+0000 7ff3697ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376b270 0x562b5376b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.986+0000 7ff3697ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376b270 0x562b5376b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52504/0 (socket says 172.21.15.29:52504) 2024-03-11T19:12:46.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.986+0000 7ff369fcb640 1 --2- 172.21.15.29:0/2767901547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 0x562b53773260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.986+0000 7ff3697ca640 1 -- 172.21.15.29:0/2767901547 learned_addr learned my addr 172.21.15.29:0/2767901547 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.986+0000 7ff368fc9640 1 --2- 172.21.15.29:0/2767901547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5376bc60 0x562b53770760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.986+0000 7ff3697ca640 1 -- 172.21.15.29:0/2767901547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 msgr2=0x562b53773260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.986+0000 7ff3697ca640 1 --2- 172.21.15.29:0/2767901547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 0x562b53773260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.986+0000 7ff3697ca640 1 -- 172.21.15.29:0/2767901547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5376bc60 msgr2=0x562b53770760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.986+0000 7ff3697ca640 1 --2- 172.21.15.29:0/2767901547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5376bc60 0x562b53770760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.986+0000 7ff3697ca640 1 -- 172.21.15.29:0/2767901547 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b53696060 con 0x562b5376b270 2024-03-11T19:12:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.986+0000 7ff368fc9640 1 --2- 172.21.15.29:0/2767901547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5376bc60 0x562b53770760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:12:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.986+0000 7ff3697ca640 1 --2- 172.21.15.29:0/2767901547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376b270 0x562b5376b690 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7ff3600970a0 tx=0x7ff36009a030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c623d5c59eeff3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.986+0000 7ff368fc9640 1 --2- 172.21.15.29:0/2500843995 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff3500a1b00 0x7ff3500a3fc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff354000c00 tx=0x7ff35402cab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.986+0000 7ff34f7fe640 1 -- 172.21.15.29:0/2767901547 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff360083ea0 con 0x562b5376b270 2024-03-11T19:12:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.987+0000 7ff34f7fe640 1 -- 172.21.15.29:0/2767901547 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff3600b3210 con 0x562b5376b270 2024-03-11T19:12:46.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.987+0000 7ff34f7fe640 1 -- 172.21.15.29:0/2767901547 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3600041e0 con 0x562b5376b270 2024-03-11T19:12:46.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.987+0000 7ff36e603740 1 -- 172.21.15.29:0/2767901547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376b270 msgr2=0x562b5376b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.987+0000 7ff36e603740 1 --2- 172.21.15.29:0/2767901547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376b270 0x562b5376b690 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7ff3600970a0 tx=0x7ff36009a030 comp rx=0 tx=0).stop 2024-03-11T19:12:46.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.988+0000 7ff36e603740 1 -- 172.21.15.29:0/2767901547 shutdown_connections 2024-03-11T19:12:46.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.988+0000 7ff36e603740 1 --2- 172.21.15.29:0/2767901547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 0x562b53773260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.988+0000 7ff36e603740 1 --2- 172.21.15.29:0/2767901547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5376bc60 0x562b53770760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.988+0000 7ff36e603740 1 --2- 172.21.15.29:0/2767901547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376b270 0x562b5376b690 secure :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7ff3600970a0 tx=0x7ff36009a030 comp rx=0 tx=0).stop 2024-03-11T19:12:46.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.988+0000 7ff36e603740 1 -- 172.21.15.29:0/2767901547 >> 172.21.15.29:0/2767901547 conn(0x562b537667e0 msgr2=0x562b53766be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:46.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.988+0000 7ff36e603740 1 -- 172.21.15.29:0/2767901547 shutdown_connections 2024-03-11T19:12:46.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.988+0000 7ff36e603740 1 -- 172.21.15.29:0/2767901547 wait complete. 2024-03-11T19:12:46.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.988+0000 7ff36e603740 1 -- start start 2024-03-11T19:12:46.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.988+0000 7ff36e603740 1 Processor -- start 2024-03-11T19:12:46.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.989+0000 7ff36e603740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376bc60 0x562b537756f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.989+0000 7ff3697ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376bc60 0x562b537756f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.990+0000 7ff36e603740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 0x562b53775c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.990+0000 7ff36e603740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5377abf0 0x562b5377cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.990+0000 7ff36e603740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b536d4020 con 0x562b5376bc60 2024-03-11T19:12:46.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.990+0000 7ff36e603740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b533a7430 con 0x562b5377abf0 2024-03-11T19:12:46.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.990+0000 7ff36e603740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b53363e00 con 0x562b53770dd0 2024-03-11T19:12:46.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.990+0000 7ff3697ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376bc60 0x562b537756f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52514/0 (socket says 172.21.15.29:52514) 2024-03-11T19:12:46.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.990+0000 7ff3697ca640 1 -- 172.21.15.29:0/3536028101 learned_addr learned my addr 172.21.15.29:0/3536028101 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:46.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.990+0000 7ff36e603740 10 client.0 _ll_register_callbacks cb 0x562b5367bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:46.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.990+0000 7ff368fc9640 1 --2- 172.21.15.29:0/3536028101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 0x562b53775c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.990+0000 7ff369fcb640 1 --2- 172.21.15.29:0/3536028101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5377abf0 0x562b5377cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.990+0000 7ff3697ca640 1 -- 172.21.15.29:0/3536028101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 msgr2=0x562b53775c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.990+0000 7ff3697ca640 1 --2- 172.21.15.29:0/3536028101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 0x562b53775c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.990+0000 7ff3697ca640 1 -- 172.21.15.29:0/3536028101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5377abf0 msgr2=0x562b5377cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:46.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.990+0000 7ff3697ca640 1 --2- 172.21.15.29:0/3536028101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5377abf0 0x562b5377cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.990+0000 7ff3697ca640 1 -- 172.21.15.29:0/3536028101 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3641ae650 con 0x562b5376bc60 2024-03-11T19:12:46.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.990+0000 7ff369fcb640 1 --2- 172.21.15.29:0/3536028101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5377abf0 0x562b5377cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:12:46.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.990+0000 7ff368fc9640 1 --2- 172.21.15.29:0/3536028101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 0x562b53775c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:46.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.990+0000 7ff3697ca640 1 --2- 172.21.15.29:0/3536028101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376bc60 0x562b537756f0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff360003890 tx=0x7ff3600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da4683e6ade4c629 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:46.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.990+0000 7ff34cff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:12:46.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.991+0000 7ff34cff9640 1 -- 172.21.15.29:0/3536028101 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3600b8070 con 0x562b5376bc60 2024-03-11T19:12:46.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.991+0000 7ff36e603740 1 -- 172.21.15.29:0/3536028101 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562b533d37e0 con 0x562b5376bc60 2024-03-11T19:12:46.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.991+0000 7ff34cff9640 1 -- 172.21.15.29:0/3536028101 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff3600b8e40 con 0x562b5376bc60 2024-03-11T19:12:46.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.991+0000 7ff34cff9640 1 -- 172.21.15.29:0/3536028101 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff360083ea0 con 0x562b5376bc60 2024-03-11T19:12:46.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.991+0000 7ff36e603740 10 client.16191 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:46.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.991+0000 7ff36e603740 1 -- 172.21.15.29:0/3536028101 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x562b536d26a0 con 0x562b5376bc60 2024-03-11T19:12:46.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.991+0000 7ff34cff9640 1 -- 172.21.15.29:0/3536028101 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff36007f030 con 0x562b5376bc60 2024-03-11T19:12:46.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.991+0000 7ff34cff9640 1 -- 172.21.15.29:0/3536028101 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff36001f060 con 0x562b5376bc60 2024-03-11T19:12:46.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.992+0000 7ff36e603740 10 client.16191 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:46.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.992+0000 7ff36e603740 10 client.16191 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:12:46.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.992+0000 7ff36e603740 20 client.16191 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:46.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.992+0000 7ff36e603740 1 --2- 172.21.15.29:0/3536028101 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x562b537817d0 0x562b53783c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:46.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.992+0000 7ff36e603740 4 client.16191 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:12:46.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.992+0000 7ff36e603740 1 -- 172.21.15.29:0/3536028101 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562b533d37e0 con 0x562b537817d0 2024-03-11T19:12:46.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.992+0000 7ff368fc9640 1 --2- 172.21.15.29:0/3536028101 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x562b537817d0 0x562b53783c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:46.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.992+0000 7ff368fc9640 1 --2- 172.21.15.29:0/3536028101 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x562b537817d0 0x562b53783c20 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=150ead5d18f50b22 server_cookie=88c2e5b743e563b9 in_seq=0 out_seq=0 2024-03-11T19:12:46.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.993+0000 7ff34cff9640 10 client.16191 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:46.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.993+0000 7ff34cff9640 1 -- 172.21.15.29:0/3536028101 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562b533d37e0 con 0x562b537817d0 2024-03-11T19:12:46.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.993+0000 7ff34cff9640 10 client.16191 handle_command_reply: tid=0 2024-03-11T19:12:46.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.993+0000 7ff36e603740 1 client.16191 shutdown 2024-03-11T19:12:46.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.994+0000 7ff36e603740 1 -- 172.21.15.29:0/3536028101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376bc60 msgr2=0x562b537756f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:46.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.994+0000 7ff36e603740 1 --2- 172.21.15.29:0/3536028101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376bc60 0x562b537756f0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff360003890 tx=0x7ff3600046c0 comp rx=0 tx=0).stop 2024-03-11T19:12:46.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.994+0000 7ff36e603740 1 -- 172.21.15.29:0/3536028101 shutdown_connections 2024-03-11T19:12:46.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.994+0000 7ff36e603740 1 --2- 172.21.15.29:0/3536028101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5377abf0 0x562b5377cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.994+0000 7ff36e603740 1 --2- 172.21.15.29:0/3536028101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 0x562b53775c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.994+0000 7ff36e603740 1 --2- 172.21.15.29:0/3536028101 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x562b537817d0 0x562b53783c20 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.994+0000 7ff36e603740 1 --2- 172.21.15.29:0/3536028101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376bc60 0x562b537756f0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.994+0000 7ff36e603740 1 -- 172.21.15.29:0/3536028101 >> 172.21.15.29:0/3536028101 conn(0x562b537667e0 msgr2=0x562b53765b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:47.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.994+0000 7ff34cff9640 0 client.16191 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:47.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.994+0000 7ff36e603740 1 -- 172.21.15.29:0/3536028101 shutdown_connections 2024-03-11T19:12:47.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.994+0000 7ff36e603740 1 -- 172.21.15.29:0/3536028101 wait complete. 2024-03-11T19:12:47.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:46.994+0000 7ff36e603740 20 client.16191 trim_cache size 0 max 16384 2024-03-11T19:12:47.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.015+0000 7ff36e603740 1 Processor -- start 2024-03-11T19:12:47.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.015+0000 7ff36e603740 1 -- start start 2024-03-11T19:12:47.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.016+0000 7ff36e603740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 0x562b53791ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:47.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.016+0000 7ff36e603740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5377abf0 0x562b53792010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:47.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.016+0000 7ff36e603740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376d440 0x562b5376d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:47.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.016+0000 7ff36e603740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b53659130 con 0x562b5376d440 2024-03-11T19:12:47.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.016+0000 7ff36e603740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b5376bae0 con 0x562b5377abf0 2024-03-11T19:12:47.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.016+0000 7ff36e603740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b533d37e0 con 0x562b53770dd0 2024-03-11T19:12:47.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.016+0000 7ff369fcb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376d440 0x562b5376d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:47.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.016+0000 7ff368fc9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5377abf0 0x562b53792010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:47.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.016+0000 7ff369fcb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376d440 0x562b5376d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52518/0 (socket says 172.21.15.29:52518) 2024-03-11T19:12:47.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.016+0000 7ff369fcb640 1 -- 172.21.15.29:0/1077937834 learned_addr learned my addr 172.21.15.29:0/1077937834 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:47.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.016+0000 7ff368fc9640 1 --2- 172.21.15.29:0/1077937834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5377abf0 0x562b53792010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:42974/0 (socket says 172.21.15.29:42974) 2024-03-11T19:12:47.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.016+0000 7ff3697ca640 1 --2- 172.21.15.29:0/1077937834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 0x562b53791ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.017+0000 7ff368fc9640 1 -- 172.21.15.29:0/1077937834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 msgr2=0x562b53791ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.017+0000 7ff368fc9640 1 --2- 172.21.15.29:0/1077937834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 0x562b53791ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.017+0000 7ff368fc9640 1 -- 172.21.15.29:0/1077937834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376d440 msgr2=0x562b5376d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.017+0000 7ff368fc9640 1 --2- 172.21.15.29:0/1077937834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376d440 0x562b5376d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.017+0000 7ff368fc9640 1 -- 172.21.15.29:0/1077937834 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b5366ec20 con 0x562b5377abf0 2024-03-11T19:12:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.017+0000 7ff369fcb640 1 --2- 172.21.15.29:0/1077937834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376d440 0x562b5376d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.017+0000 7ff368fc9640 1 --2- 172.21.15.29:0/1077937834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5377abf0 0x562b53792010 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff354009850 tx=0x7ff354047000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbb7b93e86ec485 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.017+0000 7ff34effd640 1 -- 172.21.15.29:0/1077937834 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354004e80 con 0x562b5377abf0 2024-03-11T19:12:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.017+0000 7ff34effd640 1 -- 172.21.15.29:0/1077937834 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff354057070 con 0x562b5377abf0 2024-03-11T19:12:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.017+0000 7ff34effd640 1 -- 172.21.15.29:0/1077937834 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354003cd0 con 0x562b5377abf0 2024-03-11T19:12:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.018+0000 7ff36e603740 1 -- 172.21.15.29:0/1077937834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5377abf0 msgr2=0x562b53792010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.018+0000 7ff36e603740 1 --2- 172.21.15.29:0/1077937834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5377abf0 0x562b53792010 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff354009850 tx=0x7ff354047000 comp rx=0 tx=0).stop 2024-03-11T19:12:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.019+0000 7ff36e603740 1 -- 172.21.15.29:0/1077937834 shutdown_connections 2024-03-11T19:12:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.019+0000 7ff36e603740 1 --2- 172.21.15.29:0/1077937834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376d440 0x562b5376d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.019+0000 7ff36e603740 1 --2- 172.21.15.29:0/1077937834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5377abf0 0x562b53792010 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.019+0000 7ff36e603740 1 --2- 172.21.15.29:0/1077937834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 0x562b53791ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.019+0000 7ff36e603740 1 -- 172.21.15.29:0/1077937834 >> 172.21.15.29:0/1077937834 conn(0x562b537667e0 msgr2=0x562b53788590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.019+0000 7ff36e603740 1 -- 172.21.15.29:0/1077937834 shutdown_connections 2024-03-11T19:12:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.019+0000 7ff36e603740 1 -- 172.21.15.29:0/1077937834 wait complete. 2024-03-11T19:12:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.020+0000 7ff36e603740 1 -- start start 2024-03-11T19:12:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.020+0000 7ff36e603740 1 Processor -- start 2024-03-11T19:12:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.020+0000 7ff36e603740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376d440 0x562b537d2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.020+0000 7ff3697ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376d440 0x562b537d2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.020+0000 7ff3697ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376d440 0x562b537d2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52522/0 (socket says 172.21.15.29:52522) 2024-03-11T19:12:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.021+0000 7ff3697ca640 1 -- 172.21.15.29:0/2951269901 learned_addr learned my addr 172.21.15.29:0/2951269901 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.021+0000 7ff36e603740 1 --2- 172.21.15.29:0/2951269901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 0x562b537d4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.021+0000 7ff36e603740 1 --2- 172.21.15.29:0/2951269901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5377abf0 0x562b537d72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:47.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.021+0000 7ff368fc9640 1 --2- 172.21.15.29:0/2951269901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 0x562b537d4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.021+0000 7ff36e603740 1 -- 172.21.15.29:0/2951269901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b537d7a00 con 0x562b5376d440 2024-03-11T19:12:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.021+0000 7ff36e603740 1 -- 172.21.15.29:0/2951269901 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b537d7b70 con 0x562b5377abf0 2024-03-11T19:12:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.021+0000 7ff36e603740 1 -- 172.21.15.29:0/2951269901 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b537d7d30 con 0x562b53770dd0 2024-03-11T19:12:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.021+0000 7ff369fcb640 1 --2- 172.21.15.29:0/2951269901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5377abf0 0x562b537d72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.021+0000 7ff36e603740 10 client.0 _ll_register_callbacks cb 0x7ff3600037e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.021+0000 7ff3697ca640 1 -- 172.21.15.29:0/2951269901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 msgr2=0x562b537d4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.021+0000 7ff3697ca640 1 --2- 172.21.15.29:0/2951269901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 0x562b537d4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.021+0000 7ff3697ca640 1 -- 172.21.15.29:0/2951269901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5377abf0 msgr2=0x562b537d72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.021+0000 7ff3697ca640 1 --2- 172.21.15.29:0/2951269901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5377abf0 0x562b537d72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.021+0000 7ff3697ca640 1 -- 172.21.15.29:0/2951269901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35c009d10 con 0x562b5376d440 2024-03-11T19:12:47.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.021+0000 7ff368fc9640 1 --2- 172.21.15.29:0/2951269901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 0x562b537d4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:47.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.022+0000 7ff3697ca640 1 --2- 172.21.15.29:0/2951269901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376d440 0x562b537d2770 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff36000c6a0 tx=0x7ff360098be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f6edd6e9198539 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:47.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.022+0000 7ff34cff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:12:47.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.022+0000 7ff34cff9640 1 -- 172.21.15.29:0/2951269901 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3600b8070 con 0x562b5376d440 2024-03-11T19:12:47.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.022+0000 7ff34cff9640 1 -- 172.21.15.29:0/2951269901 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff3600b8e40 con 0x562b5376d440 2024-03-11T19:12:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.022+0000 7ff36e603740 1 -- 172.21.15.29:0/2951269901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff3600b3210 con 0x562b5376d440 2024-03-11T19:12:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.023+0000 7ff34cff9640 1 -- 172.21.15.29:0/2951269901 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff360083ea0 con 0x562b5376d440 2024-03-11T19:12:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.023+0000 7ff36e603740 10 client.16197 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.023+0000 7ff36e603740 1 -- 172.21.15.29:0/2951269901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7ff3641104e0 con 0x562b5376d440 2024-03-11T19:12:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.023+0000 7ff34cff9640 1 -- 172.21.15.29:0/2951269901 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff3600b3210 con 0x562b5376d440 2024-03-11T19:12:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.023+0000 7ff34cff9640 1 -- 172.21.15.29:0/2951269901 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff36001f040 con 0x562b5376d440 2024-03-11T19:12:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.023+0000 7ff36e603740 10 client.16197 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.023+0000 7ff36e603740 10 client.16197 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:12:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.023+0000 7ff36e603740 20 client.16197 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.023+0000 7ff36e603740 1 --2- 172.21.15.29:0/2951269901 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x562b537d92d0 0x562b537db6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.023+0000 7ff36e603740 4 client.16197 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:12:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.023+0000 7ff36e603740 1 -- 172.21.15.29:0/2951269901 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562b53410720 con 0x562b537d92d0 2024-03-11T19:12:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.023+0000 7ff368fc9640 1 --2- 172.21.15.29:0/2951269901 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x562b537d92d0 0x562b537db6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.024+0000 7ff34cff9640 10 client.16197 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:47.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.024+0000 7ff368fc9640 1 --2- 172.21.15.29:0/2951269901 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x562b537d92d0 0x562b537db6f0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c46d61e8c0c3108 server_cookie=ef35f4ae41d18bcd in_seq=0 out_seq=0 2024-03-11T19:12:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.024+0000 7ff34cff9640 1 -- 172.21.15.29:0/2951269901 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562b53410720 con 0x562b537d92d0 2024-03-11T19:12:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.024+0000 7ff34cff9640 10 client.16197 handle_command_reply: tid=0 2024-03-11T19:12:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.024+0000 7ff36e603740 1 client.16197 shutdown 2024-03-11T19:12:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.024+0000 7ff36e603740 1 -- 172.21.15.29:0/2951269901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376d440 msgr2=0x562b537d2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.024+0000 7ff36e603740 1 --2- 172.21.15.29:0/2951269901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376d440 0x562b537d2770 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff36000c6a0 tx=0x7ff360098be0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.024+0000 7ff36e603740 1 -- 172.21.15.29:0/2951269901 shutdown_connections 2024-03-11T19:12:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.024+0000 7ff36e603740 1 --2- 172.21.15.29:0/2951269901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b5377abf0 0x562b537d72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.024+0000 7ff36e603740 1 --2- 172.21.15.29:0/2951269901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b53770dd0 0x562b537d4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.024+0000 7ff36e603740 1 --2- 172.21.15.29:0/2951269901 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x562b537d92d0 0x562b537db6f0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.024+0000 7ff36e603740 1 --2- 172.21.15.29:0/2951269901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b5376d440 0x562b537d2770 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.024+0000 7ff36e603740 1 -- 172.21.15.29:0/2951269901 >> 172.21.15.29:0/2951269901 conn(0x562b537667e0 msgr2=0x562b53773fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.024+0000 7ff34cff9640 0 client.16197 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.025+0000 7ff36e603740 1 -- 172.21.15.29:0/2951269901 shutdown_connections 2024-03-11T19:12:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.025+0000 7ff36e603740 1 -- 172.21.15.29:0/2951269901 wait complete. 2024-03-11T19:12:47.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.025+0000 7ff36e603740 20 client.16197 trim_cache size 0 max 16384 2024-03-11T19:12:47.031 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:12:47.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.025+0000 7ff36ba2e640 1 -- 172.21.15.29:0/2500843995 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff3500a1b00 msgr2=0x7ff3500a3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:47.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.025+0000 7ff36ba2e640 1 --2- 172.21.15.29:0/2500843995 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff3500a1b00 0x7ff3500a3fc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff354000c00 tx=0x7ff35402cab0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.025+0000 7ff36ba2e640 1 -- 172.21.15.29:0/2500843995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3641077a0 msgr2=0x7ff36407fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:47.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.025+0000 7ff36ba2e640 1 --2- 172.21.15.29:0/2500843995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3641077a0 0x7ff36407fb60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff36000e9c0 tx=0x7ff36000ee80 comp rx=0 tx=0).stop 2024-03-11T19:12:47.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.025+0000 7ff36ba2e640 1 -- 172.21.15.29:0/2500843995 shutdown_connections 2024-03-11T19:12:47.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.025+0000 7ff36ba2e640 1 --2- 172.21.15.29:0/2500843995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff364109350 0x7ff3640810e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.025+0000 7ff36ba2e640 1 --2- 172.21.15.29:0/2500843995 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff3500a1b00 0x7ff3500a3fc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.025+0000 7ff36ba2e640 1 --2- 172.21.15.29:0/2500843995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff364108990 0x7ff3640800a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.025+0000 7ff36ba2e640 1 --2- 172.21.15.29:0/2500843995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3641077a0 0x7ff36407fb60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:47.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.025+0000 7ff36ba2e640 1 -- 172.21.15.29:0/2500843995 >> 172.21.15.29:0/2500843995 conn(0x7ff364076730 msgr2=0x7ff36410bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:47.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.026+0000 7ff36ba2e640 1 -- 172.21.15.29:0/2500843995 shutdown_connections 2024-03-11T19:12:47.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:47.026+0000 7ff36ba2e640 1 -- 172.21.15.29:0/2500843995 wait complete. 2024-03-11T19:12:47.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:47 smithi190 ceph-mon[36292]: pgmap v440: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-11T19:12:47.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:47 smithi170 ceph-mon[35690]: pgmap v440: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-11T19:12:47.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:47 smithi029 ceph-mon[32439]: pgmap v440: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-11T19:12:48.045 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:12:48.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:12:48.045 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:12:48.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.210+0000 7f7dae0fb640 1 -- 172.21.15.29:0/2555717320 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7da8101c20 msgr2=0x7f7da8104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:48.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.210+0000 7f7dae0fb640 1 --2- 172.21.15.29:0/2555717320 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7da8101c20 0x7f7da8104010 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7d90009900 tx=0x7f7d9002bc40 comp rx=0 tx=0).stop 2024-03-11T19:12:48.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.210+0000 7f7dae0fb640 1 -- 172.21.15.29:0/2555717320 shutdown_connections 2024-03-11T19:12:48.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.210+0000 7f7dae0fb640 1 --2- 172.21.15.29:0/2555717320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7da8106e80 0x7f7da81092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.210+0000 7f7dae0fb640 1 --2- 172.21.15.29:0/2555717320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7da8104550 0x7f7da8106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.210+0000 7f7dae0fb640 1 --2- 172.21.15.29:0/2555717320 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7da8101c20 0x7f7da8104010 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.210+0000 7f7dae0fb640 1 -- 172.21.15.29:0/2555717320 >> 172.21.15.29:0/2555717320 conn(0x7f7da80fb630 msgr2=0x7f7da80fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:48.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.210+0000 7f7dae0fb640 1 -- 172.21.15.29:0/2555717320 shutdown_connections 2024-03-11T19:12:48.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.211+0000 7f7dae0fb640 1 -- 172.21.15.29:0/2555717320 wait complete. 2024-03-11T19:12:48.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.211+0000 7f7dae0fb640 1 Processor -- start 2024-03-11T19:12:48.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.211+0000 7f7dae0fb640 1 -- start start 2024-03-11T19:12:48.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.212+0000 7f7dae0fb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7da8101c20 0x7f7da810e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.212+0000 7f7dae0fb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7da8104550 0x7f7da810ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.212+0000 7f7dae0fb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7da8106e80 0x7f7da810b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.212+0000 7f7dae0fb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da810bce0 con 0x7f7da8106e80 2024-03-11T19:12:48.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.212+0000 7f7dae0fb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da810be50 con 0x7f7da8101c20 2024-03-11T19:12:48.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.212+0000 7f7dae0fb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da810c010 con 0x7f7da8104550 2024-03-11T19:12:48.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.212+0000 7f7da7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7da8106e80 0x7f7da810b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.212+0000 7f7da77fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7da8101c20 0x7f7da810e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.212+0000 7f7da7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7da8106e80 0x7f7da810b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52534/0 (socket says 172.21.15.29:52534) 2024-03-11T19:12:48.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.212+0000 7f7da7fff640 1 -- 172.21.15.29:0/2041747934 learned_addr learned my addr 172.21.15.29:0/2041747934 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.212+0000 7f7da6ffd640 1 --2- 172.21.15.29:0/2041747934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7da8104550 0x7f7da810ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.212+0000 7f7da7fff640 1 -- 172.21.15.29:0/2041747934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7da8104550 msgr2=0x7f7da810ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.212+0000 7f7da7fff640 1 --2- 172.21.15.29:0/2041747934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7da8104550 0x7f7da810ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.212+0000 7f7da7fff640 1 -- 172.21.15.29:0/2041747934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7da8101c20 msgr2=0x7f7da810e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.212+0000 7f7da7fff640 1 --2- 172.21.15.29:0/2041747934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7da8101c20 0x7f7da810e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.212+0000 7f7da7fff640 1 -- 172.21.15.29:0/2041747934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d90009560 con 0x7f7da8106e80 2024-03-11T19:12:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.213+0000 7f7da77fe640 1 --2- 172.21.15.29:0/2041747934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7da8101c20 0x7f7da810e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.213+0000 7f7da7fff640 1 --2- 172.21.15.29:0/2041747934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7da8106e80 0x7f7da810b550 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7da000eb10 tx=0x7f7da000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.213+0000 7f7da4ff9640 1 -- 172.21.15.29:0/2041747934 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da0004b20 con 0x7f7da8106e80 2024-03-11T19:12:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.213+0000 7f7da4ff9640 1 -- 172.21.15.29:0/2041747934 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7da0004c80 con 0x7f7da8106e80 2024-03-11T19:12:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.213+0000 7f7da4ff9640 1 -- 172.21.15.29:0/2041747934 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da0010690 con 0x7f7da8106e80 2024-03-11T19:12:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.213+0000 7f7dae0fb640 1 -- 172.21.15.29:0/2041747934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7da81a5850 con 0x7f7da8106e80 2024-03-11T19:12:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.213+0000 7f7dae0fb640 1 -- 172.21.15.29:0/2041747934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da81a5d10 con 0x7f7da8106e80 2024-03-11T19:12:48.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.217+0000 7f7db0daf740 1 Processor -- start 2024-03-11T19:12:48.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.218+0000 7f7da4ff9640 1 -- 172.21.15.29:0/2041747934 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7da0005020 con 0x7f7da8106e80 2024-03-11T19:12:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.219+0000 7f7da4ff9640 1 --2- 172.21.15.29:0/2041747934 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7d940a1ab0 0x7f7d940a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.219+0000 7f7db0daf740 1 -- start start 2024-03-11T19:12:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.219+0000 7f7da6ffd640 1 --2- 172.21.15.29:0/2041747934 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7d940a1ab0 0x7f7d940a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.219+0000 7f7da4ff9640 1 -- 172.21.15.29:0/2041747934 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7da0014070 con 0x7f7da8106e80 2024-03-11T19:12:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.219+0000 7f7db0daf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f74c0 0x559ac01f78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.219+0000 7f7db0daf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f8690 0x559ac01f8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.219+0000 7f7db0daf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f9050 0x559ac01ff8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.219+0000 7f7db0daf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ac0166f90 con 0x559ac01f8690 2024-03-11T19:12:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.219+0000 7f7db0daf740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ac01644c0 con 0x559ac01f9050 2024-03-11T19:12:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.219+0000 7f7db0daf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ac015fca0 con 0x559ac01f74c0 2024-03-11T19:12:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.220+0000 7f7da7fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f9050 0x559ac01ff8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.220+0000 7f7da77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f74c0 0x559ac01f78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.220+0000 7f7da77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f74c0 0x559ac01f78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37146/0 (socket says 172.21.15.29:37146) 2024-03-11T19:12:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.220+0000 7f7da77fe640 1 -- 172.21.15.29:0/2841738663 learned_addr learned my addr 172.21.15.29:0/2841738663 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.220+0000 7f7da6ffd640 1 --2- 172.21.15.29:0/2041747934 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7d940a1ab0 0x7f7d940a3f70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7da810c960 tx=0x7f7d98005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.220+0000 7f7da6ffd640 1 --2- 172.21.15.29:0/2841738663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f8690 0x559ac01f8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.220+0000 7f7da7fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f9050 0x559ac01ff8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:43024/0 (socket says 172.21.15.29:43024) 2024-03-11T19:12:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.220+0000 7f7da6ffd640 1 -- 172.21.15.29:0/2841738663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f74c0 msgr2=0x559ac01f78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.220+0000 7f7da6ffd640 1 --2- 172.21.15.29:0/2841738663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f74c0 0x559ac01f78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.220+0000 7f7da6ffd640 1 -- 172.21.15.29:0/2841738663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f9050 msgr2=0x559ac01ff8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.220+0000 7f7da6ffd640 1 --2- 172.21.15.29:0/2841738663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f9050 0x559ac01ff8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.220+0000 7f7da6ffd640 1 -- 172.21.15.29:0/2841738663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ac01220b0 con 0x559ac01f8690 2024-03-11T19:12:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.221+0000 7f7da6ffd640 1 --2- 172.21.15.29:0/2841738663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f8690 0x559ac01f8b10 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f7d9800e560 tx=0x7f7d98014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb95954a44a61b3c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.221+0000 7f7d9dffb640 1 -- 172.21.15.29:0/2841738663 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d98004020 con 0x559ac01f8690 2024-03-11T19:12:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.221+0000 7f7d9dffb640 1 -- 172.21.15.29:0/2841738663 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7d98021070 con 0x559ac01f8690 2024-03-11T19:12:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.221+0000 7f7d9dffb640 1 -- 172.21.15.29:0/2841738663 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d98004a00 con 0x559ac01f8690 2024-03-11T19:12:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.222+0000 7f7db0daf740 1 -- 172.21.15.29:0/2841738663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f8690 msgr2=0x559ac01f8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.222+0000 7f7db0daf740 1 --2- 172.21.15.29:0/2841738663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f8690 0x559ac01f8b10 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f7d9800e560 tx=0x7f7d98014000 comp rx=0 tx=0).stop 2024-03-11T19:12:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.222+0000 7f7db0daf740 1 -- 172.21.15.29:0/2841738663 shutdown_connections 2024-03-11T19:12:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.222+0000 7f7db0daf740 1 --2- 172.21.15.29:0/2841738663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f9050 0x559ac01ff8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.222+0000 7f7db0daf740 1 --2- 172.21.15.29:0/2841738663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f8690 0x559ac01f8b10 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.222+0000 7f7db0daf740 1 --2- 172.21.15.29:0/2841738663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f74c0 0x559ac01f78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.222+0000 7f7db0daf740 1 -- 172.21.15.29:0/2841738663 >> 172.21.15.29:0/2841738663 conn(0x559ac01f26a0 msgr2=0x559ac01f2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.222+0000 7f7db0daf740 1 -- 172.21.15.29:0/2841738663 shutdown_connections 2024-03-11T19:12:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.223+0000 7f7db0daf740 1 -- 172.21.15.29:0/2841738663 wait complete. 2024-03-11T19:12:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.223+0000 7f7db0daf740 1 -- start start 2024-03-11T19:12:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.224+0000 7f7db0daf740 1 Processor -- start 2024-03-11T19:12:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.224+0000 7f7db0daf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f74c0 0x559ac0201760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.224+0000 7f7db0daf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f8690 0x559ac0201ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.224+0000 7f7db0daf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f9050 0x559ac0208ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.224+0000 7f7da6ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f8690 0x559ac0201ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.224+0000 7f7da6ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f8690 0x559ac0201ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:43026/0 (socket says 172.21.15.29:43026) 2024-03-11T19:12:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.225+0000 7f7da77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f74c0 0x559ac0201760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.225+0000 7f7da6ffd640 1 -- 172.21.15.29:0/3685335597 learned_addr learned my addr 172.21.15.29:0/3685335597 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.225+0000 7f7db0daf740 1 -- 172.21.15.29:0/3685335597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ac0104f40 con 0x559ac01f74c0 2024-03-11T19:12:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.225+0000 7f7db0daf740 1 -- 172.21.15.29:0/3685335597 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559abfe334b0 con 0x559ac01f8690 2024-03-11T19:12:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.225+0000 7f7db0daf740 1 -- 172.21.15.29:0/3685335597 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ac00e5670 con 0x559ac01f9050 2024-03-11T19:12:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.225+0000 7f7da7fff640 1 --2- 172.21.15.29:0/3685335597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f9050 0x559ac0208ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.225+0000 7f7db0daf740 10 client.0 _ll_register_callbacks cb 0x559ac0152f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.225+0000 7f7da7fff640 1 -- 172.21.15.29:0/3685335597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f8690 msgr2=0x559ac0201ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.225+0000 7f7da7fff640 1 --2- 172.21.15.29:0/3685335597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f8690 0x559ac0201ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.225+0000 7f7da7fff640 1 -- 172.21.15.29:0/3685335597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f74c0 msgr2=0x559ac0201760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.225+0000 7f7da7fff640 1 --2- 172.21.15.29:0/3685335597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f74c0 0x559ac0201760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.225+0000 7f7da7fff640 1 -- 172.21.15.29:0/3685335597 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d98017070 con 0x559ac01f9050 2024-03-11T19:12:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.225+0000 7f7da7fff640 1 --2- 172.21.15.29:0/3685335597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f9050 0x559ac0208ce0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7da00cb4c0 tx=0x7f7da009afe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4627b25897e8f8f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.225+0000 7f7d86ffd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:12:48.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.226+0000 7f7d86ffd640 1 -- 172.21.15.29:0/3685335597 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da00a3080 con 0x559ac01f9050 2024-03-11T19:12:48.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.226+0000 7f7d86ffd640 1 -- 172.21.15.29:0/3685335597 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7da00b32b0 con 0x559ac01f9050 2024-03-11T19:12:48.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.226+0000 7f7db0daf740 1 -- 172.21.15.29:0/3685335597 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559ac0139f40 con 0x559ac01f9050 2024-03-11T19:12:48.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.226+0000 7f7d86ffd640 1 -- 172.21.15.29:0/3685335597 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7da00a2bf0 con 0x559ac01f9050 2024-03-11T19:12:48.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.226+0000 7f7db0daf740 10 client.25768 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:48.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.226+0000 7f7db0daf740 1 -- 172.21.15.29:0/3685335597 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x559ac015e320 con 0x559ac01f9050 2024-03-11T19:12:48.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.226+0000 7f7d86ffd640 1 -- 172.21.15.29:0/3685335597 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7da007d020 con 0x559ac01f9050 2024-03-11T19:12:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.227+0000 7f7d86ffd640 1 -- 172.21.15.29:0/3685335597 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7da001f060 con 0x559ac01f9050 2024-03-11T19:12:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.227+0000 7f7db0daf740 10 client.25768 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.227+0000 7f7db0daf740 10 client.25768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:12:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.227+0000 7f7db0daf740 20 client.25768 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.227+0000 7f7db0daf740 1 --2- 172.21.15.29:0/3685335597 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x559ac020da50 0x559ac020fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.227+0000 7f7db0daf740 4 client.25768 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:12:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.227+0000 7f7db0daf740 1 -- 172.21.15.29:0/3685335597 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559ac0156310 con 0x559ac020da50 2024-03-11T19:12:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.227+0000 7f7da77fe640 1 --2- 172.21.15.29:0/3685335597 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x559ac020da50 0x559ac020fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.228+0000 7f7da77fe640 1 --2- 172.21.15.29:0/3685335597 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x559ac020da50 0x559ac020fe70 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a0839c71bf3e1a7 server_cookie=76353fd5eb10b75b in_seq=0 out_seq=0 2024-03-11T19:12:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.228+0000 7f7d86ffd640 10 client.25768 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.228+0000 7f7d86ffd640 1 -- 172.21.15.29:0/3685335597 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559ac0156310 con 0x559ac020da50 2024-03-11T19:12:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.228+0000 7f7d86ffd640 10 client.25768 handle_command_reply: tid=0 2024-03-11T19:12:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.228+0000 7f7db0daf740 1 client.25768 shutdown 2024-03-11T19:12:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.228+0000 7f7db0daf740 1 -- 172.21.15.29:0/3685335597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f9050 msgr2=0x559ac0208ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.228+0000 7f7db0daf740 1 --2- 172.21.15.29:0/3685335597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f9050 0x559ac0208ce0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7da00cb4c0 tx=0x7f7da009afe0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.229+0000 7f7db0daf740 1 -- 172.21.15.29:0/3685335597 shutdown_connections 2024-03-11T19:12:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.229+0000 7f7db0daf740 1 --2- 172.21.15.29:0/3685335597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f9050 0x559ac0208ce0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.229+0000 7f7db0daf740 1 --2- 172.21.15.29:0/3685335597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f8690 0x559ac0201ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.229+0000 7f7db0daf740 1 --2- 172.21.15.29:0/3685335597 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x559ac020da50 0x559ac020fe70 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.229+0000 7f7db0daf740 1 --2- 172.21.15.29:0/3685335597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f74c0 0x559ac0201760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.229+0000 7f7db0daf740 1 -- 172.21.15.29:0/3685335597 >> 172.21.15.29:0/3685335597 conn(0x559ac01f26a0 msgr2=0x559ac01f18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.229+0000 7f7d86ffd640 0 client.25768 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.229+0000 7f7db0daf740 1 -- 172.21.15.29:0/3685335597 shutdown_connections 2024-03-11T19:12:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.229+0000 7f7db0daf740 1 -- 172.21.15.29:0/3685335597 wait complete. 2024-03-11T19:12:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.229+0000 7f7db0daf740 20 client.25768 trim_cache size 0 max 16384 2024-03-11T19:12:48.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.257+0000 7f7db0daf740 1 Processor -- start 2024-03-11T19:12:48.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.257+0000 7f7db0daf740 1 -- start start 2024-03-11T19:12:48.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.257+0000 7f7db0daf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f74c0 0x559ac0201760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.257+0000 7f7db0daf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f8690 0x559ac0201ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.257+0000 7f7db0daf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f9050 0x559ac0208ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.257+0000 7f7db0daf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ac01f3440 con 0x559ac01f8690 2024-03-11T19:12:48.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.258+0000 7f7db0daf740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ac020bfc0 con 0x559ac01f9050 2024-03-11T19:12:48.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.258+0000 7f7db0daf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ac01f3640 con 0x559ac01f74c0 2024-03-11T19:12:48.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.258+0000 7f7da7fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f9050 0x559ac0208ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.258+0000 7f7da77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f74c0 0x559ac0201760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.258+0000 7f7da6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f8690 0x559ac0201ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.258+0000 7f7da77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f74c0 0x559ac0201760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37170/0 (socket says 172.21.15.29:37170) 2024-03-11T19:12:48.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.258+0000 7f7da77fe640 1 -- 172.21.15.29:0/264500303 learned_addr learned my addr 172.21.15.29:0/264500303 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:48.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.258+0000 7f7da6ffd640 1 -- 172.21.15.29:0/264500303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f74c0 msgr2=0x559ac0201760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:48.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.258+0000 7f7da6ffd640 1 --2- 172.21.15.29:0/264500303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f74c0 0x559ac0201760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.258+0000 7f7da6ffd640 1 -- 172.21.15.29:0/264500303 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f9050 msgr2=0x559ac0208ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:48.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.258+0000 7f7da6ffd640 1 --2- 172.21.15.29:0/264500303 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f9050 0x559ac0208ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.258+0000 7f7da6ffd640 1 -- 172.21.15.29:0/264500303 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ac00fac00 con 0x559ac01f8690 2024-03-11T19:12:48.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.258+0000 7f7da6ffd640 1 --2- 172.21.15.29:0/264500303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f8690 0x559ac0201ca0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x559ac0202c40 tx=0x7f7d9800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36d3d449beddf6ee server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:48.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.259+0000 7f7d9d7fa640 1 -- 172.21.15.29:0/264500303 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9802c070 con 0x559ac01f8690 2024-03-11T19:12:48.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.259+0000 7f7d9d7fa640 1 -- 172.21.15.29:0/264500303 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7d98021040 con 0x559ac01f8690 2024-03-11T19:12:48.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.259+0000 7f7d9d7fa640 1 -- 172.21.15.29:0/264500303 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d98003db0 con 0x559ac01f8690 2024-03-11T19:12:48.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.260+0000 7f7db0daf740 1 -- 172.21.15.29:0/264500303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f8690 msgr2=0x559ac0201ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:48.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.260+0000 7f7db0daf740 1 --2- 172.21.15.29:0/264500303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f8690 0x559ac0201ca0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x559ac0202c40 tx=0x7f7d9800ed40 comp rx=0 tx=0).stop 2024-03-11T19:12:48.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.260+0000 7f7db0daf740 1 -- 172.21.15.29:0/264500303 shutdown_connections 2024-03-11T19:12:48.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.260+0000 7f7db0daf740 1 --2- 172.21.15.29:0/264500303 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f9050 0x559ac0208ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.260+0000 7f7db0daf740 1 --2- 172.21.15.29:0/264500303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f8690 0x559ac0201ca0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.260+0000 7f7db0daf740 1 --2- 172.21.15.29:0/264500303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f74c0 0x559ac0201760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.260+0000 7f7db0daf740 1 -- 172.21.15.29:0/264500303 >> 172.21.15.29:0/264500303 conn(0x559ac01f26a0 msgr2=0x559ac02147d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:48.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.260+0000 7f7db0daf740 1 -- 172.21.15.29:0/264500303 shutdown_connections 2024-03-11T19:12:48.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.260+0000 7f7db0daf740 1 -- 172.21.15.29:0/264500303 wait complete. 2024-03-11T19:12:48.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.261+0000 7f7db0daf740 1 -- start start 2024-03-11T19:12:48.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.261+0000 7f7db0daf740 1 Processor -- start 2024-03-11T19:12:48.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.261+0000 7f7db0daf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f74c0 0x559ac0205a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.261+0000 7f7db0daf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f8690 0x559ac0205fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.261+0000 7f7db0daf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f9050 0x559ac0263f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.262+0000 7f7db0daf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ac0206590 con 0x559ac01f8690 2024-03-11T19:12:48.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.262+0000 7f7db0daf740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ac0264480 con 0x559ac01f9050 2024-03-11T19:12:48.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.262+0000 7f7db0daf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ac02645c0 con 0x559ac01f74c0 2024-03-11T19:12:48.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.262+0000 7f7db0daf740 10 client.0 _ll_register_callbacks cb 0x7f7d980099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:48.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.262+0000 7f7da6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f8690 0x559ac0205fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.262+0000 7f7da7fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f9050 0x559ac0263f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.262+0000 7f7da77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f74c0 0x559ac0205a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.262+0000 7f7da6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f8690 0x559ac0205fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52576/0 (socket says 172.21.15.29:52576) 2024-03-11T19:12:48.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.262+0000 7f7da6ffd640 1 -- 172.21.15.29:0/1961497815 learned_addr learned my addr 172.21.15.29:0/1961497815 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:48.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.262+0000 7f7da77fe640 1 -- 172.21.15.29:0/1961497815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f9050 msgr2=0x559ac0263f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:48.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.262+0000 7f7da77fe640 1 --2- 172.21.15.29:0/1961497815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f9050 0x559ac0263f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.262+0000 7f7da77fe640 1 -- 172.21.15.29:0/1961497815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f8690 msgr2=0x559ac0205fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:48.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.262+0000 7f7da77fe640 1 --2- 172.21.15.29:0/1961497815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f8690 0x559ac0205fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.262+0000 7f7da77fe640 1 -- 172.21.15.29:0/1961497815 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559ac01220b0 con 0x559ac01f74c0 2024-03-11T19:12:48.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.262+0000 7f7da7fff640 1 --2- 172.21.15.29:0/1961497815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f9050 0x559ac0263f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:48.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.262+0000 7f7da6ffd640 1 --2- 172.21.15.29:0/1961497815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f8690 0x559ac0205fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:48.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.262+0000 7f7da77fe640 1 --2- 172.21.15.29:0/1961497815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f74c0 0x559ac0205a70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7d90009230 tx=0x7f7d90005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62be92e61043d604 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:48.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.263+0000 7f7d86ffd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:12:48.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.263+0000 7f7d86ffd640 1 -- 172.21.15.29:0/1961497815 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9002dea0 con 0x559ac01f74c0 2024-03-11T19:12:48.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.263+0000 7f7d86ffd640 1 -- 172.21.15.29:0/1961497815 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7d90046040 con 0x559ac01f74c0 2024-03-11T19:12:48.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.263+0000 7f7db0daf740 1 -- 172.21.15.29:0/1961497815 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559ac0185900 con 0x559ac01f74c0 2024-03-11T19:12:48.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.263+0000 7f7d86ffd640 1 -- 172.21.15.29:0/1961497815 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d9002cb80 con 0x559ac01f74c0 2024-03-11T19:12:48.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.263+0000 7f7db0daf740 10 client.25774 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:48.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.263+0000 7f7db0daf740 1 -- 172.21.15.29:0/1961497815 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x559ac0120910 con 0x559ac01f74c0 2024-03-11T19:12:48.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.264+0000 7f7d86ffd640 1 -- 172.21.15.29:0/1961497815 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7d9004d070 con 0x559ac01f74c0 2024-03-11T19:12:48.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.264+0000 7f7d86ffd640 1 -- 172.21.15.29:0/1961497815 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7d90051030 con 0x559ac01f74c0 2024-03-11T19:12:48.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.264+0000 7f7db0daf740 10 client.25774 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:48.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.264+0000 7f7db0daf740 10 client.25774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:12:48.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.264+0000 7f7db0daf740 20 client.25774 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:48.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.264+0000 7f7db0daf740 1 --2- 172.21.15.29:0/1961497815 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x559ac0266070 0x559ac0268490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.264+0000 7f7db0daf740 4 client.25774 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:12:48.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.264+0000 7f7db0daf740 1 -- 172.21.15.29:0/1961497815 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x559ac0185900 con 0x559ac0266070 2024-03-11T19:12:48.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.264+0000 7f7da6ffd640 1 --2- 172.21.15.29:0/1961497815 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x559ac0266070 0x559ac0268490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.265+0000 7f7d86ffd640 10 client.25774 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.265+0000 7f7da6ffd640 1 --2- 172.21.15.29:0/1961497815 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x559ac0266070 0x559ac0268490 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f208fea590ec16be server_cookie=671da4257992555b in_seq=0 out_seq=0 2024-03-11T19:12:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.266+0000 7f7d86ffd640 1 -- 172.21.15.29:0/1961497815 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x559ac0185900 con 0x559ac0266070 2024-03-11T19:12:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.266+0000 7f7d86ffd640 10 client.25774 handle_command_reply: tid=0 2024-03-11T19:12:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.266+0000 7f7db0daf740 1 client.25774 shutdown 2024-03-11T19:12:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.267+0000 7f7db0daf740 1 -- 172.21.15.29:0/1961497815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f74c0 msgr2=0x559ac0205a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.267+0000 7f7db0daf740 1 --2- 172.21.15.29:0/1961497815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f74c0 0x559ac0205a70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7d90009230 tx=0x7f7d90005fd0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.267+0000 7f7db0daf740 1 -- 172.21.15.29:0/1961497815 shutdown_connections 2024-03-11T19:12:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.267+0000 7f7db0daf740 1 --2- 172.21.15.29:0/1961497815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ac01f9050 0x559ac0263f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.267+0000 7f7db0daf740 1 --2- 172.21.15.29:0/1961497815 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x559ac0266070 0x559ac0268490 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.267+0000 7f7db0daf740 1 --2- 172.21.15.29:0/1961497815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559ac01f8690 0x559ac0205fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.267+0000 7f7db0daf740 1 --2- 172.21.15.29:0/1961497815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559ac01f74c0 0x559ac0205a70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.267+0000 7f7db0daf740 1 -- 172.21.15.29:0/1961497815 >> 172.21.15.29:0/1961497815 conn(0x559ac01f26a0 msgr2=0x559ac01fc240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:48.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.267+0000 7f7d86ffd640 0 client.25774 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:48.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.267+0000 7f7db0daf740 1 -- 172.21.15.29:0/1961497815 shutdown_connections 2024-03-11T19:12:48.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.267+0000 7f7db0daf740 1 -- 172.21.15.29:0/1961497815 wait complete. 2024-03-11T19:12:48.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.268+0000 7f7db0daf740 20 client.25774 trim_cache size 0 max 16384 2024-03-11T19:12:48.274 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:12:48.274 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:12:48.274 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "0e25ef93-22d4-4164-afc0-db39db77fc55", 2024-03-11T19:12:48.274 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:12:48.274 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:12:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.269+0000 7f7dae0fb640 1 -- 172.21.15.29:0/2041747934 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7d940a1ab0 msgr2=0x7f7d940a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.269+0000 7f7dae0fb640 1 --2- 172.21.15.29:0/2041747934 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7d940a1ab0 0x7f7d940a3f70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7da810c960 tx=0x7f7d98005f90 comp rx=0 tx=0).stop 2024-03-11T19:12:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.269+0000 7f7dae0fb640 1 -- 172.21.15.29:0/2041747934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7da8106e80 msgr2=0x7f7da810b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.269+0000 7f7dae0fb640 1 --2- 172.21.15.29:0/2041747934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7da8106e80 0x7f7da810b550 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7da000eb10 tx=0x7f7da000c2d0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.269+0000 7f7dae0fb640 1 -- 172.21.15.29:0/2041747934 shutdown_connections 2024-03-11T19:12:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.269+0000 7f7dae0fb640 1 --2- 172.21.15.29:0/2041747934 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7d940a1ab0 0x7f7d940a3f70 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.269+0000 7f7dae0fb640 1 --2- 172.21.15.29:0/2041747934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7da8106e80 0x7f7da810b550 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.269+0000 7f7dae0fb640 1 --2- 172.21.15.29:0/2041747934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7da8104550 0x7f7da810ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.269+0000 7f7dae0fb640 1 --2- 172.21.15.29:0/2041747934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7da8101c20 0x7f7da810e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.269+0000 7f7dae0fb640 1 -- 172.21.15.29:0/2041747934 >> 172.21.15.29:0/2041747934 conn(0x7f7da80fb630 msgr2=0x7f7da81026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.270+0000 7f7dae0fb640 1 -- 172.21.15.29:0/2041747934 shutdown_connections 2024-03-11T19:12:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.270+0000 7f7dae0fb640 1 -- 172.21.15.29:0/2041747934 wait complete. 2024-03-11T19:12:48.289 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:12:48.289 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:12:48.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.459+0000 7f62e560e640 1 -- 172.21.15.29:0/2358928717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e0109350 msgr2=0x7f62e010fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:48.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.459+0000 7f62e560e640 1 --2- 172.21.15.29:0/2358928717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e0109350 0x7f62e010fbe0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f62d8009ca0 tx=0x7f62d802c0f0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.460+0000 7f62e560e640 1 -- 172.21.15.29:0/2358928717 shutdown_connections 2024-03-11T19:12:48.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.460+0000 7f62e560e640 1 --2- 172.21.15.29:0/2358928717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e0109350 0x7f62e010fbe0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.460+0000 7f62e560e640 1 --2- 172.21.15.29:0/2358928717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62e0108990 0x7f62e0108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.460+0000 7f62e560e640 1 --2- 172.21.15.29:0/2358928717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62e01077a0 0x7f62e0107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.460+0000 7f62e560e640 1 -- 172.21.15.29:0/2358928717 >> 172.21.15.29:0/2358928717 conn(0x7f62e0076740 msgr2=0x7f62e0078b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:48.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.460+0000 7f62e560e640 1 -- 172.21.15.29:0/2358928717 shutdown_connections 2024-03-11T19:12:48.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.460+0000 7f62e560e640 1 -- 172.21.15.29:0/2358928717 wait complete. 2024-03-11T19:12:48.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.461+0000 7f62e560e640 1 Processor -- start 2024-03-11T19:12:48.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.461+0000 7f62e560e640 1 -- start start 2024-03-11T19:12:48.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.462+0000 7f62e560e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62e01077a0 0x7f62e019d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.462+0000 7f62e560e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e0108990 0x7f62e019dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.462+0000 7f62e560e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62e0109350 0x7f62e01a4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.462+0000 7f62e560e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e01a5210 con 0x7f62e0108990 2024-03-11T19:12:48.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.462+0000 7f62e560e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e01a5380 con 0x7f62e01077a0 2024-03-11T19:12:48.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.462+0000 7f62e560e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e01a5540 con 0x7f62e0109350 2024-03-11T19:12:48.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.462+0000 7f62deffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62e01077a0 0x7f62e019d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.462+0000 7f62deffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62e01077a0 0x7f62e019d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:43050/0 (socket says 172.21.15.29:43050) 2024-03-11T19:12:48.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.462+0000 7f62deffd640 1 -- 172.21.15.29:0/1428993279 learned_addr learned my addr 172.21.15.29:0/1428993279 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:48.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.462+0000 7f62de7fc640 1 --2- 172.21.15.29:0/1428993279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e0108990 0x7f62e019dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.462+0000 7f62df7fe640 1 --2- 172.21.15.29:0/1428993279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62e0109350 0x7f62e01a4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.462+0000 7f62deffd640 1 -- 172.21.15.29:0/1428993279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62e0109350 msgr2=0x7f62e01a4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:48.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.463+0000 7f62deffd640 1 --2- 172.21.15.29:0/1428993279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62e0109350 0x7f62e01a4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.463+0000 7f62deffd640 1 -- 172.21.15.29:0/1428993279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e0108990 msgr2=0x7f62e019dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:48.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.463+0000 7f62deffd640 1 --2- 172.21.15.29:0/1428993279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e0108990 0x7f62e019dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.463+0000 7f62deffd640 1 -- 172.21.15.29:0/1428993279 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d8009900 con 0x7f62e01077a0 2024-03-11T19:12:48.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.463+0000 7f62de7fc640 1 --2- 172.21.15.29:0/1428993279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e0108990 0x7f62e019dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:12:48.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.463+0000 7f62df7fe640 1 --2- 172.21.15.29:0/1428993279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62e0109350 0x7f62e01a4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:12:48.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.463+0000 7f62deffd640 1 --2- 172.21.15.29:0/1428993279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62e01077a0 0x7f62e019d720 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f62d000e9c0 tx=0x7f62d000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:48.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.463+0000 7f62c7fff640 1 -- 172.21.15.29:0/1428993279 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d0009700 con 0x7f62e01077a0 2024-03-11T19:12:48.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.463+0000 7f62c7fff640 1 -- 172.21.15.29:0/1428993279 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f62d00044d0 con 0x7f62e01077a0 2024-03-11T19:12:48.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.463+0000 7f62c7fff640 1 -- 172.21.15.29:0/1428993279 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d0010430 con 0x7f62e01077a0 2024-03-11T19:12:48.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.463+0000 7f62e560e640 1 -- 172.21.15.29:0/1428993279 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e01a5820 con 0x7f62e01077a0 2024-03-11T19:12:48.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.464+0000 7f62e560e640 1 -- 172.21.15.29:0/1428993279 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e01a5c80 con 0x7f62e01077a0 2024-03-11T19:12:48.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.467+0000 7f62c7fff640 1 -- 172.21.15.29:0/1428993279 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f62d0004020 con 0x7f62e01077a0 2024-03-11T19:12:48.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.467+0000 7f62c7fff640 1 --2- 172.21.15.29:0/1428993279 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f62c80a1b00 0x7f62c80a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.467+0000 7f62e81de740 1 Processor -- start 2024-03-11T19:12:48.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.468+0000 7f62c7fff640 1 -- 172.21.15.29:0/1428993279 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f62d0014070 con 0x7f62e01077a0 2024-03-11T19:12:48.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.468+0000 7f62e81de740 1 -- start start 2024-03-11T19:12:48.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.468+0000 7f62de7fc640 1 --2- 172.21.15.29:0/1428993279 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f62c80a1b00 0x7f62c80a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.468+0000 7f62e81de740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb0b2d0 0x555a8cb0b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.468+0000 7f62e81de740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb0bcc0 0x555a8cb107c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.468+0000 7f62e81de740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb10e30 0x555a8cb132c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.468+0000 7f62e81de740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a8ca7b280 con 0x555a8cb10e30 2024-03-11T19:12:48.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.468+0000 7f62e81de740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a8ca787b0 con 0x555a8cb0b2d0 2024-03-11T19:12:48.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.468+0000 7f62e81de740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a8ca73f90 con 0x555a8cb0bcc0 2024-03-11T19:12:48.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.468+0000 7f62de7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb0bcc0 0x555a8cb107c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.468+0000 7f62deffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb0b2d0 0x555a8cb0b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.468+0000 7f62de7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb0bcc0 0x555a8cb107c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37184/0 (socket says 172.21.15.29:37184) 2024-03-11T19:12:48.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.468+0000 7f62de7fc640 1 -- 172.21.15.29:0/877732020 learned_addr learned my addr 172.21.15.29:0/877732020 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:48.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.468+0000 7f62df7fe640 1 --2- 172.21.15.29:0/877732020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb10e30 0x555a8cb132c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.469+0000 7f62de7fc640 1 --2- 172.21.15.29:0/1428993279 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f62c80a1b00 0x7f62c80a3fc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f62e019ec30 tx=0x7f62d40147e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.469+0000 7f62de7fc640 1 -- 172.21.15.29:0/877732020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb0b2d0 msgr2=0x555a8cb0b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.469+0000 7f62de7fc640 1 --2- 172.21.15.29:0/877732020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb0b2d0 0x555a8cb0b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.469+0000 7f62de7fc640 1 -- 172.21.15.29:0/877732020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb10e30 msgr2=0x555a8cb132c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.469+0000 7f62de7fc640 1 --2- 172.21.15.29:0/877732020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb10e30 0x555a8cb132c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.469+0000 7f62de7fc640 1 -- 172.21.15.29:0/877732020 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a8ca32570 con 0x555a8cb0bcc0 2024-03-11T19:12:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.469+0000 7f62de7fc640 1 --2- 172.21.15.29:0/877732020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb0bcc0 0x555a8cb107c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f62d4010370 tx=0x7f62d401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9039cbf3444cd5c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.469+0000 7f62c57fa640 1 -- 172.21.15.29:0/877732020 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d4010d30 con 0x555a8cb0bcc0 2024-03-11T19:12:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.469+0000 7f62c57fa640 1 -- 172.21.15.29:0/877732020 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f62d4022030 con 0x555a8cb0bcc0 2024-03-11T19:12:48.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.469+0000 7f62c57fa640 1 -- 172.21.15.29:0/877732020 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d401e680 con 0x555a8cb0bcc0 2024-03-11T19:12:48.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.470+0000 7f62e81de740 1 -- 172.21.15.29:0/877732020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb0bcc0 msgr2=0x555a8cb107c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:48.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.471+0000 7f62e81de740 1 --2- 172.21.15.29:0/877732020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb0bcc0 0x555a8cb107c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f62d4010370 tx=0x7f62d401a000 comp rx=0 tx=0).stop 2024-03-11T19:12:48.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.471+0000 7f62e81de740 1 -- 172.21.15.29:0/877732020 shutdown_connections 2024-03-11T19:12:48.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.471+0000 7f62e81de740 1 --2- 172.21.15.29:0/877732020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb10e30 0x555a8cb132c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.471+0000 7f62e81de740 1 --2- 172.21.15.29:0/877732020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb0bcc0 0x555a8cb107c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.471+0000 7f62e81de740 1 --2- 172.21.15.29:0/877732020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb0b2d0 0x555a8cb0b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.471+0000 7f62e81de740 1 -- 172.21.15.29:0/877732020 >> 172.21.15.29:0/877732020 conn(0x555a8cb067f0 msgr2=0x555a8cb06bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:48.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.471+0000 7f62e81de740 1 -- 172.21.15.29:0/877732020 shutdown_connections 2024-03-11T19:12:48.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.471+0000 7f62e81de740 1 -- 172.21.15.29:0/877732020 wait complete. 2024-03-11T19:12:48.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.471+0000 7f62e81de740 1 -- start start 2024-03-11T19:12:48.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.471+0000 7f62e81de740 1 Processor -- start 2024-03-11T19:12:48.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.472+0000 7f62e81de740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb0b2d0 0x555a8cb15280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.472+0000 7f62e81de740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb0bcc0 0x555a8cb157c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.472+0000 7f62deffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb0b2d0 0x555a8cb15280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.472+0000 7f62deffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb0b2d0 0x555a8cb15280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:43076/0 (socket says 172.21.15.29:43076) 2024-03-11T19:12:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.473+0000 7f62deffd640 1 -- 172.21.15.29:0/514033318 learned_addr learned my addr 172.21.15.29:0/514033318 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.473+0000 7f62de7fc640 1 --2- 172.21.15.29:0/514033318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb0bcc0 0x555a8cb157c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.473+0000 7f62e81de740 1 --2- 172.21.15.29:0/514033318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb10e30 0x555a8cb1c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.473+0000 7f62e81de740 1 -- 172.21.15.29:0/514033318 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a8c747430 con 0x555a8cb10e30 2024-03-11T19:12:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.473+0000 7f62e81de740 1 -- 172.21.15.29:0/514033318 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a8c703e00 con 0x555a8cb0b2d0 2024-03-11T19:12:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.473+0000 7f62e81de740 1 -- 172.21.15.29:0/514033318 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a8c9f8fd0 con 0x555a8cb0bcc0 2024-03-11T19:12:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.473+0000 7f62df7fe640 1 --2- 172.21.15.29:0/514033318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb10e30 0x555a8cb1c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.473+0000 7f62e81de740 10 client.0 _ll_register_callbacks cb 0x555a8ca4efd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.474+0000 7f62de7fc640 1 -- 172.21.15.29:0/514033318 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb0b2d0 msgr2=0x555a8cb15280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.474+0000 7f62de7fc640 1 --2- 172.21.15.29:0/514033318 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb0b2d0 0x555a8cb15280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.474+0000 7f62de7fc640 1 -- 172.21.15.29:0/514033318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb10e30 msgr2=0x555a8cb1c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:48.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.474+0000 7f62de7fc640 1 --2- 172.21.15.29:0/514033318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb10e30 0x555a8cb1c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.474+0000 7f62de7fc640 1 -- 172.21.15.29:0/514033318 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d401a070 con 0x555a8cb0bcc0 2024-03-11T19:12:48.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.474+0000 7f62df7fe640 1 --2- 172.21.15.29:0/514033318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb10e30 0x555a8cb1c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:12:48.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.474+0000 7f62de7fc640 1 --2- 172.21.15.29:0/514033318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb0bcc0 0x555a8cb157c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f62d4010b40 tx=0x7f62d4009810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bf2133d33d3cfa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:48.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.474+0000 7f62c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:12:48.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.474+0000 7f62c2ff5640 1 -- 172.21.15.29:0/514033318 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d4012850 con 0x555a8cb0bcc0 2024-03-11T19:12:48.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.475+0000 7f62c2ff5640 1 -- 172.21.15.29:0/514033318 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f62d402b070 con 0x555a8cb0bcc0 2024-03-11T19:12:48.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.475+0000 7f62e81de740 1 -- 172.21.15.29:0/514033318 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555a8c7737e0 con 0x555a8cb0bcc0 2024-03-11T19:12:48.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.475+0000 7f62c2ff5640 1 -- 172.21.15.29:0/514033318 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62d40129b0 con 0x555a8cb0bcc0 2024-03-11T19:12:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.475+0000 7f62e81de740 10 client.25783 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.475+0000 7f62e81de740 1 -- 172.21.15.29:0/514033318 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555a8ca72610 con 0x555a8cb0bcc0 2024-03-11T19:12:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.475+0000 7f62c2ff5640 1 -- 172.21.15.29:0/514033318 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x555a8c7737e0 con 0x555a8cb0bcc0 2024-03-11T19:12:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.475+0000 7f62c2ff5640 1 -- 172.21.15.29:0/514033318 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f62d4027410 con 0x555a8cb0bcc0 2024-03-11T19:12:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.475+0000 7f62e81de740 10 client.25783 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.475+0000 7f62e81de740 10 client.25783 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:12:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.475+0000 7f62e81de740 20 client.25783 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.475+0000 7f62e81de740 1 --2- 172.21.15.29:0/514033318 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555a8cb21300 0x555a8cb23720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.475+0000 7f62e81de740 4 client.25783 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:12:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.475+0000 7f62e81de740 1 -- 172.21.15.29:0/514033318 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555a8c7b0720 con 0x555a8cb21300 2024-03-11T19:12:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.476+0000 7f62deffd640 1 --2- 172.21.15.29:0/514033318 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555a8cb21300 0x555a8cb23720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.476+0000 7f62c2ff5640 10 client.25783 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.476+0000 7f62deffd640 1 --2- 172.21.15.29:0/514033318 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555a8cb21300 0x555a8cb23720 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9145df82bfcc27e server_cookie=78ed38a2588f4ac2 in_seq=0 out_seq=0 2024-03-11T19:12:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.477+0000 7f62c2ff5640 1 -- 172.21.15.29:0/514033318 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555a8c7b0720 con 0x555a8cb21300 2024-03-11T19:12:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.477+0000 7f62c2ff5640 10 client.25783 handle_command_reply: tid=0 2024-03-11T19:12:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.477+0000 7f62e81de740 1 client.25783 shutdown 2024-03-11T19:12:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.477+0000 7f62e81de740 1 -- 172.21.15.29:0/514033318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb0bcc0 msgr2=0x555a8cb157c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.477+0000 7f62e81de740 1 --2- 172.21.15.29:0/514033318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb0bcc0 0x555a8cb157c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f62d4010b40 tx=0x7f62d4009810 comp rx=0 tx=0).stop 2024-03-11T19:12:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.477+0000 7f62e81de740 1 -- 172.21.15.29:0/514033318 shutdown_connections 2024-03-11T19:12:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.477+0000 7f62e81de740 1 --2- 172.21.15.29:0/514033318 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555a8cb21300 0x555a8cb23720 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.477+0000 7f62e81de740 1 --2- 172.21.15.29:0/514033318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb10e30 0x555a8cb1c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.477+0000 7f62c2ff5640 0 client.25783 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.477+0000 7f62e81de740 1 --2- 172.21.15.29:0/514033318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb0bcc0 0x555a8cb157c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.477+0000 7f62e81de740 1 --2- 172.21.15.29:0/514033318 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb0b2d0 0x555a8cb15280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.477+0000 7f62e81de740 1 -- 172.21.15.29:0/514033318 >> 172.21.15.29:0/514033318 conn(0x555a8cb067f0 msgr2=0x555a8cb05a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.478+0000 7f62e81de740 1 -- 172.21.15.29:0/514033318 shutdown_connections 2024-03-11T19:12:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.478+0000 7f62e81de740 1 -- 172.21.15.29:0/514033318 wait complete. 2024-03-11T19:12:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.478+0000 7f62e81de740 20 client.25783 trim_cache size 0 max 16384 2024-03-11T19:12:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.500+0000 7f62e81de740 1 Processor -- start 2024-03-11T19:12:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.500+0000 7f62e81de740 1 -- start start 2024-03-11T19:12:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.500+0000 7f62e81de740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb0b2d0 0x555a8cb15280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.500+0000 7f62e81de740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb0bcc0 0x555a8cb1c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.500+0000 7f62e81de740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb10e30 0x555a8cb157c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.500+0000 7f62e81de740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a8c7b0720 con 0x555a8cb0bcc0 2024-03-11T19:12:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.500+0000 7f62e81de740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a8c7737e0 con 0x555a8cb0b2d0 2024-03-11T19:12:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.500+0000 7f62e81de740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a8cb020d0 con 0x555a8cb10e30 2024-03-11T19:12:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.501+0000 7f62de7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb0bcc0 0x555a8cb1c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.501+0000 7f62de7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb0bcc0 0x555a8cb1c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52628/0 (socket says 172.21.15.29:52628) 2024-03-11T19:12:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.501+0000 7f62de7fc640 1 -- 172.21.15.29:0/1287536721 learned_addr learned my addr 172.21.15.29:0/1287536721 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.501+0000 7f62deffd640 1 --2- 172.21.15.29:0/1287536721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb0b2d0 0x555a8cb15280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.501+0000 7f62df7fe640 1 --2- 172.21.15.29:0/1287536721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb10e30 0x555a8cb157c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.501+0000 7f62de7fc640 1 -- 172.21.15.29:0/1287536721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb10e30 msgr2=0x555a8cb157c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.501+0000 7f62de7fc640 1 --2- 172.21.15.29:0/1287536721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb10e30 0x555a8cb157c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.501+0000 7f62de7fc640 1 -- 172.21.15.29:0/1287536721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb0b2d0 msgr2=0x555a8cb15280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.501+0000 7f62de7fc640 1 --2- 172.21.15.29:0/1287536721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb0b2d0 0x555a8cb15280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.501+0000 7f62de7fc640 1 -- 172.21.15.29:0/1287536721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a8ca0eac0 con 0x555a8cb0bcc0 2024-03-11T19:12:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.501+0000 7f62de7fc640 1 --2- 172.21.15.29:0/1287536721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb0bcc0 0x555a8cb1c7c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f62d4027820 tx=0x7f62d4010d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c827216261686bc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.501+0000 7f62c4ff9640 1 -- 172.21.15.29:0/1287536721 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d4005a70 con 0x555a8cb0bcc0 2024-03-11T19:12:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.501+0000 7f62c4ff9640 1 -- 172.21.15.29:0/1287536721 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f62d40054e0 con 0x555a8cb0bcc0 2024-03-11T19:12:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.501+0000 7f62c4ff9640 1 -- 172.21.15.29:0/1287536721 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d40128b0 con 0x555a8cb0bcc0 2024-03-11T19:12:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.502+0000 7f62e81de740 1 -- 172.21.15.29:0/1287536721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb0bcc0 msgr2=0x555a8cb1c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.502+0000 7f62e81de740 1 --2- 172.21.15.29:0/1287536721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb0bcc0 0x555a8cb1c7c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f62d4027820 tx=0x7f62d4010d30 comp rx=0 tx=0).stop 2024-03-11T19:12:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.502+0000 7f62e81de740 1 -- 172.21.15.29:0/1287536721 shutdown_connections 2024-03-11T19:12:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.502+0000 7f62e81de740 1 --2- 172.21.15.29:0/1287536721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb10e30 0x555a8cb157c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.502+0000 7f62e81de740 1 --2- 172.21.15.29:0/1287536721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb0bcc0 0x555a8cb1c7c0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.502+0000 7f62e81de740 1 --2- 172.21.15.29:0/1287536721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb0b2d0 0x555a8cb15280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.502+0000 7f62e81de740 1 -- 172.21.15.29:0/1287536721 >> 172.21.15.29:0/1287536721 conn(0x555a8cb067f0 msgr2=0x555a8cb28080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.502+0000 7f62e81de740 1 -- 172.21.15.29:0/1287536721 shutdown_connections 2024-03-11T19:12:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.502+0000 7f62e81de740 1 -- 172.21.15.29:0/1287536721 wait complete. 2024-03-11T19:12:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.502+0000 7f62e81de740 1 -- start start 2024-03-11T19:12:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.502+0000 7f62e81de740 1 Processor -- start 2024-03-11T19:12:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62e81de740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb0b2d0 0x555a8cb1c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62e81de740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb0bcc0 0x555a8cb1a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62de7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb0bcc0 0x555a8cb1a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62e81de740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb10e30 0x555a8cb1af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62de7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb0bcc0 0x555a8cb1a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52630/0 (socket says 172.21.15.29:52630) 2024-03-11T19:12:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62deffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb0b2d0 0x555a8cb1c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62deffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb0b2d0 0x555a8cb1c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37218/0 (socket says 172.21.15.29:37218) 2024-03-11T19:12:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62de7fc640 1 -- 172.21.15.29:0/3969319468 learned_addr learned my addr 172.21.15.29:0/3969319468 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:12:48.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62e81de740 1 -- 172.21.15.29:0/3969319468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a8cb1cb40 con 0x555a8cb0bcc0 2024-03-11T19:12:48.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62e81de740 1 -- 172.21.15.29:0/3969319468 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a8cb1b450 con 0x555a8cb10e30 2024-03-11T19:12:48.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62e81de740 1 -- 172.21.15.29:0/3969319468 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a8cb1b5c0 con 0x555a8cb0b2d0 2024-03-11T19:12:48.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62df7fe640 1 --2- 172.21.15.29:0/3969319468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb10e30 0x555a8cb1af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62e81de740 10 client.0 _ll_register_callbacks cb 0x7f62d4009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:12:48.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62de7fc640 1 -- 172.21.15.29:0/3969319468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb0b2d0 msgr2=0x555a8cb1c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:48.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62de7fc640 1 --2- 172.21.15.29:0/3969319468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb0b2d0 0x555a8cb1c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62de7fc640 1 -- 172.21.15.29:0/3969319468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb10e30 msgr2=0x555a8cb1af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:12:48.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62de7fc640 1 --2- 172.21.15.29:0/3969319468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb10e30 0x555a8cb1af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62de7fc640 1 -- 172.21.15.29:0/3969319468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a8ca72610 con 0x555a8cb0bcc0 2024-03-11T19:12:48.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62de7fc640 1 --2- 172.21.15.29:0/3969319468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb0bcc0 0x555a8cb1a9d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f62d4011010 tx=0x7f62d4012980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f744831488e53132 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:12:48.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.503+0000 7f62c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:12:48.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.504+0000 7f62c2ff5640 1 -- 172.21.15.29:0/3969319468 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d4005a70 con 0x555a8cb0bcc0 2024-03-11T19:12:48.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.504+0000 7f62c2ff5640 1 -- 172.21.15.29:0/3969319468 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f62d402b040 con 0x555a8cb0bcc0 2024-03-11T19:12:48.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.504+0000 7f62e81de740 1 -- 172.21.15.29:0/3969319468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f62d4022030 con 0x555a8cb0bcc0 2024-03-11T19:12:48.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.504+0000 7f62c2ff5640 1 -- 172.21.15.29:0/3969319468 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62d4005650 con 0x555a8cb0bcc0 2024-03-11T19:12:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.504+0000 7f62e81de740 10 client.16230 fetch_fsmap learned FSMap version 27 2024-03-11T19:12:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.504+0000 7f62e81de740 1 -- 172.21.15.29:0/3969319468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555a8ca346a0 con 0x555a8cb0bcc0 2024-03-11T19:12:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.504+0000 7f62c2ff5640 1 -- 172.21.15.29:0/3969319468 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f62d4022030 con 0x555a8cb0bcc0 2024-03-11T19:12:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.504+0000 7f62c2ff5640 1 -- 172.21.15.29:0/3969319468 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f62d4040070 con 0x555a8cb0bcc0 2024-03-11T19:12:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.504+0000 7f62e81de740 10 client.16230 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:12:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.504+0000 7f62e81de740 10 client.16230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:12:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.504+0000 7f62e81de740 20 client.16230 populate_metadata read hostname 'smithi029' 2024-03-11T19:12:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.504+0000 7f62e81de740 1 --2- 172.21.15.29:0/3969319468 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555a8cb21300 0x555a8cb7aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:12:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.504+0000 7f62e81de740 4 client.16230 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:12:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.504+0000 7f62e81de740 1 -- 172.21.15.29:0/3969319468 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555a8c7ff890 con 0x555a8cb21300 2024-03-11T19:12:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.505+0000 7f62deffd640 1 --2- 172.21.15.29:0/3969319468 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555a8cb21300 0x555a8cb7aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:12:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.505+0000 7f62c2ff5640 10 client.16230 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.505+0000 7f62deffd640 1 --2- 172.21.15.29:0/3969319468 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555a8cb21300 0x555a8cb7aac0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6396fb290c66fcc server_cookie=5c7650c9458bdc30 in_seq=0 out_seq=0 2024-03-11T19:12:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.505+0000 7f62c2ff5640 1 -- 172.21.15.29:0/3969319468 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x555a8c7ff890 con 0x555a8cb21300 2024-03-11T19:12:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.505+0000 7f62c2ff5640 10 client.16230 handle_command_reply: tid=0 2024-03-11T19:12:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.505+0000 7f62e81de740 1 client.16230 shutdown 2024-03-11T19:12:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.505+0000 7f62e81de740 1 -- 172.21.15.29:0/3969319468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb0bcc0 msgr2=0x555a8cb1a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.505+0000 7f62e81de740 1 --2- 172.21.15.29:0/3969319468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb0bcc0 0x555a8cb1a9d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f62d4011010 tx=0x7f62d4012980 comp rx=0 tx=0).stop 2024-03-11T19:12:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.505+0000 7f62e81de740 1 -- 172.21.15.29:0/3969319468 shutdown_connections 2024-03-11T19:12:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.505+0000 7f62e81de740 1 --2- 172.21.15.29:0/3969319468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a8cb10e30 0x555a8cb1af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.505+0000 7f62e81de740 1 --2- 172.21.15.29:0/3969319468 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555a8cb21300 0x555a8cb7aac0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.505+0000 7f62e81de740 1 --2- 172.21.15.29:0/3969319468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a8cb0bcc0 0x555a8cb1a9d0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.505+0000 7f62c2ff5640 0 client.16230 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:12:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.505+0000 7f62e81de740 1 --2- 172.21.15.29:0/3969319468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cb0b2d0 0x555a8cb1c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.505+0000 7f62e81de740 1 -- 172.21.15.29:0/3969319468 >> 172.21.15.29:0/3969319468 conn(0x555a8cb067f0 msgr2=0x555a8cb15220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.505+0000 7f62e81de740 1 -- 172.21.15.29:0/3969319468 shutdown_connections 2024-03-11T19:12:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.506+0000 7f62e81de740 1 -- 172.21.15.29:0/3969319468 wait complete. 2024-03-11T19:12:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.506+0000 7f62e81de740 20 client.16230 trim_cache size 0 max 16384 2024-03-11T19:12:48.516 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:12:48.516 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:12:48.516 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:12:48.516 INFO:teuthology.orchestra.run.smithi029.stdout: "0e25ef93-22d4-4164-afc0-db39db77fc55": { 2024-03-11T19:12:48.516 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:12:48.516 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "0e25ef93-22d4-4164-afc0-db39db77fc55", 2024-03-11T19:12:48.516 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:12:48.516 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:12:48.516 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:12:48.516 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:12:48.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.506+0000 7f62e560e640 1 -- 172.21.15.29:0/1428993279 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f62c80a1b00 msgr2=0x7f62c80a3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:48.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.506+0000 7f62e560e640 1 --2- 172.21.15.29:0/1428993279 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f62c80a1b00 0x7f62c80a3fc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f62e019ec30 tx=0x7f62d40147e0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.506+0000 7f62e560e640 1 -- 172.21.15.29:0/1428993279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62e01077a0 msgr2=0x7f62e019d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:12:48.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.506+0000 7f62e560e640 1 --2- 172.21.15.29:0/1428993279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62e01077a0 0x7f62e019d720 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f62d000e9c0 tx=0x7f62d000ee80 comp rx=0 tx=0).stop 2024-03-11T19:12:48.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.506+0000 7f62e560e640 1 -- 172.21.15.29:0/1428993279 shutdown_connections 2024-03-11T19:12:48.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.506+0000 7f62e560e640 1 --2- 172.21.15.29:0/1428993279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62e0109350 0x7f62e01a4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.506+0000 7f62e560e640 1 --2- 172.21.15.29:0/1428993279 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f62c80a1b00 0x7f62c80a3fc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.506+0000 7f62e560e640 1 --2- 172.21.15.29:0/1428993279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e0108990 0x7f62e019dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.506+0000 7f62e560e640 1 --2- 172.21.15.29:0/1428993279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62e01077a0 0x7f62e019d720 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:12:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.506+0000 7f62e560e640 1 -- 172.21.15.29:0/1428993279 >> 172.21.15.29:0/1428993279 conn(0x7f62e0076740 msgr2=0x7f62e010bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:12:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.506+0000 7f62e560e640 1 -- 172.21.15.29:0/1428993279 shutdown_connections 2024-03-11T19:12:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:12:48.506+0000 7f62e560e640 1 -- 172.21.15.29:0/1428993279 wait complete. 2024-03-11T19:12:48.520 INFO:tasks.cephfs.filesystem:scrub status for tag:0e25ef93-22d4-4164-afc0-db39db77fc55 - {'path': '/', 'tag': '0e25ef93-22d4-4164-afc0-db39db77fc55', 'options': 'recursive,force'} 2024-03-11T19:12:49.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:49 smithi190 ceph-mon[36292]: pgmap v441: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-11T19:12:49.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:49 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:12:49.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:49 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:12:49.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:49 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:12:49.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:49 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:12:49.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:49 smithi170 ceph-mon[35690]: pgmap v441: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-11T19:12:49.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:49 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:12:49.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:49 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:12:49.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:49 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:12:49.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:49 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:12:49.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:49 smithi029 ceph-mon[32439]: pgmap v441: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-11T19:12:49.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:49 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:12:49.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:49 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:12:49.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:49 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:12:49.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:49 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:12:51.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:51 smithi190 ceph-mon[36292]: pgmap v442: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 81 MiB/s wr, 1.31k op/s 2024-03-11T19:12:51.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:51 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:12:51.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:51 smithi170 ceph-mon[35690]: pgmap v442: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 81 MiB/s wr, 1.31k op/s 2024-03-11T19:12:51.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:51 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:12:51.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:51 smithi029 ceph-mon[32439]: pgmap v442: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 81 MiB/s wr, 1.31k op/s 2024-03-11T19:12:51.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:51 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:12:53.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:53 smithi190 ceph-mon[36292]: pgmap v443: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T19:12:53.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:53 smithi170 ceph-mon[35690]: pgmap v443: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T19:12:53.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:53 smithi029 ceph-mon[32439]: pgmap v443: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T19:12:55.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:55 smithi190 ceph-mon[36292]: pgmap v444: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 76 MiB/s wr, 1.23k op/s 2024-03-11T19:12:55.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:55 smithi170 ceph-mon[35690]: pgmap v444: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 76 MiB/s wr, 1.23k op/s 2024-03-11T19:12:55.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:55 smithi029 ceph-mon[32439]: pgmap v444: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 76 MiB/s wr, 1.23k op/s 2024-03-11T19:12:57.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:57 smithi190 ceph-mon[36292]: pgmap v445: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-11T19:12:57.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:57 smithi170 ceph-mon[35690]: pgmap v445: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-11T19:12:57.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:57 smithi029 ceph-mon[32439]: pgmap v445: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-11T19:12:59.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:12:59 smithi190 ceph-mon[36292]: pgmap v446: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:12:59.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:12:59 smithi170 ceph-mon[35690]: pgmap v446: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:12:59.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:12:59 smithi029 ceph-mon[32439]: pgmap v446: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:13:01.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:01 smithi190 ceph-mon[36292]: pgmap v447: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-11T19:13:01.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:01 smithi170 ceph-mon[35690]: pgmap v447: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-11T19:13:01.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:01 smithi029 ceph-mon[32439]: pgmap v447: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-11T19:13:03.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:03 smithi190 ceph-mon[36292]: pgmap v448: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:13:03.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:03 smithi170 ceph-mon[35690]: pgmap v448: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:13:03.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:03 smithi029 ceph-mon[32439]: pgmap v448: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:13:05.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:05 smithi190 ceph-mon[36292]: pgmap v449: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T19:13:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:05 smithi170 ceph-mon[35690]: pgmap v449: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T19:13:05.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:05 smithi029 ceph-mon[32439]: pgmap v449: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T19:13:07.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:07 smithi190 ceph-mon[36292]: pgmap v450: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-11T19:13:07.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:07 smithi170 ceph-mon[35690]: pgmap v450: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-11T19:13:07.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:07 smithi029 ceph-mon[32439]: pgmap v450: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-11T19:13:09.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:09 smithi190 ceph-mon[36292]: pgmap v451: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-03-11T19:13:09.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:09 smithi170 ceph-mon[35690]: pgmap v451: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-03-11T19:13:09.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:09 smithi029 ceph-mon[32439]: pgmap v451: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-03-11T19:13:11.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:11 smithi190 ceph-mon[36292]: pgmap v452: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T19:13:11.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:11 smithi170 ceph-mon[35690]: pgmap v452: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T19:13:11.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:11 smithi029 ceph-mon[32439]: pgmap v452: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T19:13:13.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:13 smithi190 ceph-mon[36292]: pgmap v453: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T19:13:13.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:13 smithi170 ceph-mon[35690]: pgmap v453: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T19:13:13.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:13 smithi029 ceph-mon[32439]: pgmap v453: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T19:13:15.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:15 smithi190 ceph-mon[36292]: pgmap v454: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-11T19:13:15.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:15 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:13:15.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:15 smithi170 ceph-mon[35690]: pgmap v454: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-11T19:13:15.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:15 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:13:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:15 smithi029 ceph-mon[32439]: pgmap v454: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-11T19:13:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:15 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:13:16.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:13:16.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:13:16.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:16 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:13:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:13:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:13:16.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:16 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:13:16.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:13:16.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:13:16.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:16 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:13:17.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:17 smithi190 ceph-mon[36292]: pgmap v455: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T19:13:17.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:17 smithi170 ceph-mon[35690]: pgmap v455: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T19:13:17.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:17 smithi029 ceph-mon[32439]: pgmap v455: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T19:13:18.520 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:13:18.520 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:13:18.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.682+0000 7fb6d5a88640 1 -- 172.21.15.29:0/2831503247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6d0100b30 msgr2=0x7fb6d01013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:18.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.682+0000 7fb6d5a88640 1 --2- 172.21.15.29:0/2831503247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6d0100b30 0x7fb6d01013b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8009ca0 tx=0x7fb6c802c100 comp rx=0 tx=0).stop 2024-03-11T19:13:18.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.682+0000 7fb6d5a88640 1 -- 172.21.15.29:0/2831503247 shutdown_connections 2024-03-11T19:13:18.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.682+0000 7fb6d5a88640 1 --2- 172.21.15.29:0/2831503247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6d0100b30 0x7fb6d01013b0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.682+0000 7fb6d5a88640 1 --2- 172.21.15.29:0/2831503247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6d0100170 0x7fb6d01005f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.682+0000 7fb6d5a88640 1 --2- 172.21.15.29:0/2831503247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6d00ff0c0 0x7fb6d00ff4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.682+0000 7fb6d5a88640 1 -- 172.21.15.29:0/2831503247 >> 172.21.15.29:0/2831503247 conn(0x7fb6d00fa960 msgr2=0x7fb6d00fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:18.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.683+0000 7fb6d5a88640 1 -- 172.21.15.29:0/2831503247 shutdown_connections 2024-03-11T19:13:18.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.683+0000 7fb6d5a88640 1 -- 172.21.15.29:0/2831503247 wait complete. 2024-03-11T19:13:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.683+0000 7fb6d5a88640 1 Processor -- start 2024-03-11T19:13:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.684+0000 7fb6d5a88640 1 -- start start 2024-03-11T19:13:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.684+0000 7fb6d5a88640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6d00ff0c0 0x7fb6d019d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.684+0000 7fb6d5a88640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6d0100170 0x7fb6d019dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.684+0000 7fb6d5a88640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6d0100b30 0x7fb6d006cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.684+0000 7fb6d5a88640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d006d0f0 con 0x7fb6d0100170 2024-03-11T19:13:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.684+0000 7fb6d5a88640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d006d260 con 0x7fb6d0100b30 2024-03-11T19:13:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.684+0000 7fb6d5a88640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d006d420 con 0x7fb6d00ff0c0 2024-03-11T19:13:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.685+0000 7fb6cf7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6d00ff0c0 0x7fb6d019d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.685+0000 7fb6cf7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6d00ff0c0 0x7fb6d019d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:43054/0 (socket says 172.21.15.29:43054) 2024-03-11T19:13:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.685+0000 7fb6cf7fe640 1 -- 172.21.15.29:0/2510485055 learned_addr learned my addr 172.21.15.29:0/2510485055 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.685+0000 7fb6cffff640 1 --2- 172.21.15.29:0/2510485055 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6d0100b30 0x7fb6d006cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.685+0000 7fb6ceffd640 1 --2- 172.21.15.29:0/2510485055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6d0100170 0x7fb6d019dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.685+0000 7fb6cf7fe640 1 -- 172.21.15.29:0/2510485055 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6d0100b30 msgr2=0x7fb6d006cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.685+0000 7fb6cf7fe640 1 --2- 172.21.15.29:0/2510485055 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6d0100b30 0x7fb6d006cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.685+0000 7fb6cf7fe640 1 -- 172.21.15.29:0/2510485055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6d0100170 msgr2=0x7fb6d019dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.685+0000 7fb6cf7fe640 1 --2- 172.21.15.29:0/2510485055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6d0100170 0x7fb6d019dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.685+0000 7fb6cf7fe640 1 -- 172.21.15.29:0/2510485055 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c0009490 con 0x7fb6d00ff0c0 2024-03-11T19:13:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.685+0000 7fb6cf7fe640 1 --2- 172.21.15.29:0/2510485055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6d00ff0c0 0x7fb6d019d6e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb6c000e9c0 tx=0x7fb6c000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.686+0000 7fb6ccff9640 1 -- 172.21.15.29:0/2510485055 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c000cde0 con 0x7fb6d00ff0c0 2024-03-11T19:13:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.686+0000 7fb6ccff9640 1 -- 172.21.15.29:0/2510485055 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb6c00108e0 con 0x7fb6d00ff0c0 2024-03-11T19:13:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.686+0000 7fb6ccff9640 1 -- 172.21.15.29:0/2510485055 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c0017860 con 0x7fb6d00ff0c0 2024-03-11T19:13:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.686+0000 7fb6d5a88640 1 -- 172.21.15.29:0/2510485055 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c8009900 con 0x7fb6d00ff0c0 2024-03-11T19:13:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.686+0000 7fb6d5a88640 1 -- 172.21.15.29:0/2510485055 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d006d9d0 con 0x7fb6d00ff0c0 2024-03-11T19:13:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.689+0000 7fb6ccff9640 1 -- 172.21.15.29:0/2510485055 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb6c00179c0 con 0x7fb6d00ff0c0 2024-03-11T19:13:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.690+0000 7fb6d8667740 1 Processor -- start 2024-03-11T19:13:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.690+0000 7fb6d8667740 1 -- start start 2024-03-11T19:13:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.690+0000 7fb6d8667740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde482d0 0x5559fde486f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.690+0000 7fb6cffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde482d0 0x5559fde486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.690+0000 7fb6cffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde482d0 0x5559fde486f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55640/0 (socket says 172.21.15.29:55640) 2024-03-11T19:13:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.691+0000 7fb6d8667740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde48cc0 0x5559fde4d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.691+0000 7fb6d8667740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde4de30 0x5559fde502c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.691+0000 7fb6d8667740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559fddb8280 con 0x5559fde48cc0 2024-03-11T19:13:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.691+0000 7fb6d8667740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559fddb57b0 con 0x5559fde482d0 2024-03-11T19:13:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.691+0000 7fb6d8667740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559fddb0f90 con 0x5559fde4de30 2024-03-11T19:13:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.691+0000 7fb6cffff640 1 -- 172.21.15.29:0/2137679834 learned_addr learned my addr 172.21.15.29:0/2137679834 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.691+0000 7fb6ceffd640 1 --2- 172.21.15.29:0/2137679834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde4de30 0x5559fde502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.691+0000 7fb6cffff640 1 -- 172.21.15.29:0/2137679834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde4de30 msgr2=0x5559fde502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.691+0000 7fb6cffff640 1 --2- 172.21.15.29:0/2137679834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde4de30 0x5559fde502c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.691+0000 7fb6cffff640 1 -- 172.21.15.29:0/2137679834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde48cc0 msgr2=0x5559fde4d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.692+0000 7fb6ccff9640 1 --2- 172.21.15.29:0/2510485055 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb6b80a1b00 0x7fb6b80a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.692+0000 7fb6ceffd640 1 --2- 172.21.15.29:0/2510485055 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb6b80a1b00 0x7fb6b80a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.692+0000 7fb6ccff9640 1 -- 172.21.15.29:0/2510485055 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb6c0014070 con 0x7fb6d00ff0c0 2024-03-11T19:13:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.692+0000 7fb6cf7fe640 1 --2- 172.21.15.29:0/2137679834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde48cc0 0x5559fde4d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.692+0000 7fb6cffff640 1 --2- 172.21.15.29:0/2137679834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde48cc0 0x5559fde4d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.692+0000 7fb6cffff640 1 -- 172.21.15.29:0/2137679834 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559fdd6f570 con 0x5559fde482d0 2024-03-11T19:13:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.692+0000 7fb6cf7fe640 1 --2- 172.21.15.29:0/2137679834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde48cc0 0x5559fde4d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:13:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.693+0000 7fb6cffff640 1 --2- 172.21.15.29:0/2137679834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde482d0 0x5559fde486f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb6c802c710 tx=0x7fb6c8032340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f36a260fae501e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.693+0000 7fb6ceffd640 1 --2- 172.21.15.29:0/2510485055 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb6b80a1b00 0x7fb6b80a3fc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb6d019ebf0 tx=0x7fb6c400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.693+0000 7fb6b5ffb640 1 -- 172.21.15.29:0/2137679834 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8032cb0 con 0x5559fde482d0 2024-03-11T19:13:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.693+0000 7fb6b5ffb640 1 -- 172.21.15.29:0/2137679834 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb6c8032e10 con 0x5559fde482d0 2024-03-11T19:13:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.693+0000 7fb6b5ffb640 1 -- 172.21.15.29:0/2137679834 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8003970 con 0x5559fde482d0 2024-03-11T19:13:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.694+0000 7fb6d8667740 1 -- 172.21.15.29:0/2137679834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde482d0 msgr2=0x5559fde486f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.694+0000 7fb6d8667740 1 --2- 172.21.15.29:0/2137679834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde482d0 0x5559fde486f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb6c802c710 tx=0x7fb6c8032340 comp rx=0 tx=0).stop 2024-03-11T19:13:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.694+0000 7fb6d8667740 1 -- 172.21.15.29:0/2137679834 shutdown_connections 2024-03-11T19:13:18.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.694+0000 7fb6d8667740 1 --2- 172.21.15.29:0/2137679834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde4de30 0x5559fde502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.694+0000 7fb6d8667740 1 --2- 172.21.15.29:0/2137679834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde48cc0 0x5559fde4d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.694+0000 7fb6d8667740 1 --2- 172.21.15.29:0/2137679834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde482d0 0x5559fde486f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.694+0000 7fb6d8667740 1 -- 172.21.15.29:0/2137679834 >> 172.21.15.29:0/2137679834 conn(0x5559fde437f0 msgr2=0x5559fde43bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.694+0000 7fb6d8667740 1 -- 172.21.15.29:0/2137679834 shutdown_connections 2024-03-11T19:13:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.694+0000 7fb6d8667740 1 -- 172.21.15.29:0/2137679834 wait complete. 2024-03-11T19:13:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.695+0000 7fb6d8667740 1 -- start start 2024-03-11T19:13:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.695+0000 7fb6d8667740 1 Processor -- start 2024-03-11T19:13:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.695+0000 7fb6d8667740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde482d0 0x5559fde52250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.695+0000 7fb6d8667740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde48cc0 0x5559fde52790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.696+0000 7fb6d8667740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde4de30 0x5559fde597d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.696+0000 7fb6ceffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde48cc0 0x5559fde52790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.696+0000 7fb6ceffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde48cc0 0x5559fde52790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55642/0 (socket says 172.21.15.29:55642) 2024-03-11T19:13:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.696+0000 7fb6ceffd640 1 -- 172.21.15.29:0/875607684 learned_addr learned my addr 172.21.15.29:0/875607684 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.696+0000 7fb6d8667740 1 -- 172.21.15.29:0/875607684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559fda84430 con 0x5559fde482d0 2024-03-11T19:13:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.696+0000 7fb6cf7fe640 1 --2- 172.21.15.29:0/875607684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde482d0 0x5559fde52250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.696+0000 7fb6cffff640 1 --2- 172.21.15.29:0/875607684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde4de30 0x5559fde597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.696+0000 7fb6d8667740 1 -- 172.21.15.29:0/875607684 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559fda40e00 con 0x5559fde48cc0 2024-03-11T19:13:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.696+0000 7fb6d8667740 1 -- 172.21.15.29:0/875607684 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559fdd35fd0 con 0x5559fde4de30 2024-03-11T19:13:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.696+0000 7fb6d8667740 10 client.0 _ll_register_callbacks cb 0x5559fdd8bfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.696+0000 7fb6ceffd640 1 -- 172.21.15.29:0/875607684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde4de30 msgr2=0x5559fde597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.696+0000 7fb6ceffd640 1 --2- 172.21.15.29:0/875607684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde4de30 0x5559fde597d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.696+0000 7fb6ceffd640 1 -- 172.21.15.29:0/875607684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde482d0 msgr2=0x5559fde52250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.696+0000 7fb6ceffd640 1 --2- 172.21.15.29:0/875607684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde482d0 0x5559fde52250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.696+0000 7fb6ceffd640 1 -- 172.21.15.29:0/875607684 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d0101c70 con 0x5559fde48cc0 2024-03-11T19:13:18.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.697+0000 7fb6cf7fe640 1 --2- 172.21.15.29:0/875607684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde482d0 0x5559fde52250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:18.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.697+0000 7fb6ceffd640 1 --2- 172.21.15.29:0/875607684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde48cc0 0x5559fde52790 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4003e60 tx=0x7fb6c40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbffcc58dd85fe78 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:18.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.697+0000 7fb6aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:13:18.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.697+0000 7fb6aeffd640 1 -- 172.21.15.29:0/875607684 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c4005220 con 0x5559fde48cc0 2024-03-11T19:13:18.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.697+0000 7fb6d8667740 1 -- 172.21.15.29:0/875607684 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5559fdab07e0 con 0x5559fde48cc0 2024-03-11T19:13:18.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.697+0000 7fb6aeffd640 1 -- 172.21.15.29:0/875607684 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb6c4005380 con 0x5559fde48cc0 2024-03-11T19:13:18.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.698+0000 7fb6aeffd640 1 -- 172.21.15.29:0/875607684 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6c4005540 con 0x5559fde48cc0 2024-03-11T19:13:18.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.698+0000 7fb6d8667740 10 client.25766 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:18.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.698+0000 7fb6d8667740 1 -- 172.21.15.29:0/875607684 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5559fddaf610 con 0x5559fde48cc0 2024-03-11T19:13:18.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.698+0000 7fb6aeffd640 1 -- 172.21.15.29:0/875607684 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb6c4004e60 con 0x5559fde48cc0 2024-03-11T19:13:18.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.698+0000 7fb6aeffd640 1 -- 172.21.15.29:0/875607684 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb6c401bb90 con 0x5559fde48cc0 2024-03-11T19:13:18.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.698+0000 7fb6d8667740 10 client.25766 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:18.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.698+0000 7fb6d8667740 10 client.25766 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:13:18.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.698+0000 7fb6d8667740 20 client.25766 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:18.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.698+0000 7fb6d8667740 1 --2- 172.21.15.29:0/875607684 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5559fde5e3b0 0x5559fde607f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.698+0000 7fb6d8667740 4 client.25766 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:13:18.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.698+0000 7fb6d8667740 1 -- 172.21.15.29:0/875607684 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5559fdab07e0 con 0x5559fde5e3b0 2024-03-11T19:13:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.699+0000 7fb6cf7fe640 1 --2- 172.21.15.29:0/875607684 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5559fde5e3b0 0x5559fde607f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.699+0000 7fb6aeffd640 10 client.25766 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.699+0000 7fb6cf7fe640 1 --2- 172.21.15.29:0/875607684 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5559fde5e3b0 0x5559fde607f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66112afcc9f7536d server_cookie=f5e921146154dbce in_seq=0 out_seq=0 2024-03-11T19:13:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.700+0000 7fb6aeffd640 1 -- 172.21.15.29:0/875607684 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5559fdab07e0 con 0x5559fde5e3b0 2024-03-11T19:13:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.700+0000 7fb6aeffd640 10 client.25766 handle_command_reply: tid=0 2024-03-11T19:13:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.700+0000 7fb6d8667740 1 client.25766 shutdown 2024-03-11T19:13:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.700+0000 7fb6d8667740 1 -- 172.21.15.29:0/875607684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde48cc0 msgr2=0x5559fde52790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.700+0000 7fb6d8667740 1 --2- 172.21.15.29:0/875607684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde48cc0 0x5559fde52790 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4003e60 tx=0x7fb6c40046c0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.700+0000 7fb6d8667740 1 -- 172.21.15.29:0/875607684 shutdown_connections 2024-03-11T19:13:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.701+0000 7fb6d8667740 1 --2- 172.21.15.29:0/875607684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde4de30 0x5559fde597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.701+0000 7fb6d8667740 1 --2- 172.21.15.29:0/875607684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde48cc0 0x5559fde52790 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.701+0000 7fb6d8667740 1 --2- 172.21.15.29:0/875607684 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5559fde5e3b0 0x5559fde607f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.701+0000 7fb6d8667740 1 --2- 172.21.15.29:0/875607684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde482d0 0x5559fde52250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.701+0000 7fb6d8667740 1 -- 172.21.15.29:0/875607684 >> 172.21.15.29:0/875607684 conn(0x5559fde437f0 msgr2=0x5559fde42a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:18.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.701+0000 7fb6aeffd640 0 client.25766 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:18.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.701+0000 7fb6d8667740 1 -- 172.21.15.29:0/875607684 shutdown_connections 2024-03-11T19:13:18.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.701+0000 7fb6d8667740 1 -- 172.21.15.29:0/875607684 wait complete. 2024-03-11T19:13:18.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.701+0000 7fb6d8667740 20 client.25766 trim_cache size 0 max 16384 2024-03-11T19:13:18.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.724+0000 7fb6d8667740 1 Processor -- start 2024-03-11T19:13:18.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.724+0000 7fb6d8667740 1 -- start start 2024-03-11T19:13:18.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.724+0000 7fb6d8667740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde437f0 0x5559fde65190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.724+0000 7fb6d8667740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde482d0 0x5559fde59790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.724+0000 7fb6d8667740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde48cc0 0x5559fde52250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.724+0000 7fb6d8667740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559fde50450 con 0x5559fde482d0 2024-03-11T19:13:18.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.724+0000 7fb6ceffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde482d0 0x5559fde59790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.725+0000 7fb6ceffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde482d0 0x5559fde59790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54726/0 (socket says 172.21.15.29:54726) 2024-03-11T19:13:18.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.725+0000 7fb6ceffd640 1 -- 172.21.15.29:0/1447524664 learned_addr learned my addr 172.21.15.29:0/1447524664 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:18.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.725+0000 7fb6d8667740 1 -- 172.21.15.29:0/1447524664 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559fde52f80 con 0x5559fde48cc0 2024-03-11T19:13:18.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.725+0000 7fb6d8667740 1 -- 172.21.15.29:0/1447524664 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559fdab0810 con 0x5559fde437f0 2024-03-11T19:13:18.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.725+0000 7fb6cffff640 1 --2- 172.21.15.29:0/1447524664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde48cc0 0x5559fde52250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.725+0000 7fb6cf7fe640 1 --2- 172.21.15.29:0/1447524664 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde437f0 0x5559fde65190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.725+0000 7fb6cffff640 1 -- 172.21.15.29:0/1447524664 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde437f0 msgr2=0x5559fde65190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:18.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.725+0000 7fb6cffff640 1 --2- 172.21.15.29:0/1447524664 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde437f0 0x5559fde65190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.725+0000 7fb6cffff640 1 -- 172.21.15.29:0/1447524664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde482d0 msgr2=0x5559fde59790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:18.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.725+0000 7fb6cffff640 1 --2- 172.21.15.29:0/1447524664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde482d0 0x5559fde59790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.725+0000 7fb6cffff640 1 -- 172.21.15.29:0/1447524664 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559fdd4bac0 con 0x5559fde48cc0 2024-03-11T19:13:18.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.725+0000 7fb6cf7fe640 1 --2- 172.21.15.29:0/1447524664 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde437f0 0x5559fde65190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:13:18.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.725+0000 7fb6ceffd640 1 --2- 172.21.15.29:0/1447524664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde482d0 0x5559fde59790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:18.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.725+0000 7fb6cffff640 1 --2- 172.21.15.29:0/1447524664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde48cc0 0x5559fde52250 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8009ca0 tx=0x7fb6c802c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47a9a56906afdb75 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:18.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.726+0000 7fb6b57fa640 1 -- 172.21.15.29:0/1447524664 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c80326f0 con 0x5559fde48cc0 2024-03-11T19:13:18.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.726+0000 7fb6b57fa640 1 -- 172.21.15.29:0/1447524664 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb6c8032850 con 0x5559fde48cc0 2024-03-11T19:13:18.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.726+0000 7fb6b57fa640 1 -- 172.21.15.29:0/1447524664 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8004040 con 0x5559fde48cc0 2024-03-11T19:13:18.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.727+0000 7fb6d8667740 1 -- 172.21.15.29:0/1447524664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde48cc0 msgr2=0x5559fde52250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:18.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.727+0000 7fb6d8667740 1 --2- 172.21.15.29:0/1447524664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde48cc0 0x5559fde52250 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8009ca0 tx=0x7fb6c802c100 comp rx=0 tx=0).stop 2024-03-11T19:13:18.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.727+0000 7fb6d8667740 1 -- 172.21.15.29:0/1447524664 shutdown_connections 2024-03-11T19:13:18.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.727+0000 7fb6d8667740 1 --2- 172.21.15.29:0/1447524664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde48cc0 0x5559fde52250 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.727+0000 7fb6d8667740 1 --2- 172.21.15.29:0/1447524664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde482d0 0x5559fde59790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.727+0000 7fb6d8667740 1 --2- 172.21.15.29:0/1447524664 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde437f0 0x5559fde65190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.727+0000 7fb6d8667740 1 -- 172.21.15.29:0/1447524664 >> 172.21.15.29:0/1447524664 conn(0x5559fde498c0 msgr2=0x5559fde3e840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:18.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.727+0000 7fb6d8667740 1 -- 172.21.15.29:0/1447524664 shutdown_connections 2024-03-11T19:13:18.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.727+0000 7fb6d8667740 1 -- 172.21.15.29:0/1447524664 wait complete. 2024-03-11T19:13:18.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.728+0000 7fb6d8667740 1 -- start start 2024-03-11T19:13:18.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.728+0000 7fb6d8667740 1 Processor -- start 2024-03-11T19:13:18.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.728+0000 7fb6d8667740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde437f0 0x5559fde59120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.728+0000 7fb6d8667740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde482d0 0x5559fde59660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.729+0000 7fb6d8667740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde48cc0 0x5559fde57730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.729+0000 7fb6cf7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde437f0 0x5559fde59120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.729+0000 7fb6cf7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde437f0 0x5559fde59120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55652/0 (socket says 172.21.15.29:55652) 2024-03-11T19:13:18.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.729+0000 7fb6cf7fe640 1 -- 172.21.15.29:0/1274199295 learned_addr learned my addr 172.21.15.29:0/1274199295 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:18.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.729+0000 7fb6d8667740 1 -- 172.21.15.29:0/1274199295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559fde57e30 con 0x5559fde48cc0 2024-03-11T19:13:18.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.729+0000 7fb6d8667740 1 -- 172.21.15.29:0/1274199295 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559fde57fa0 con 0x5559fde437f0 2024-03-11T19:13:18.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.729+0000 7fb6d8667740 1 -- 172.21.15.29:0/1274199295 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559fde58160 con 0x5559fde482d0 2024-03-11T19:13:18.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.729+0000 7fb6cffff640 1 --2- 172.21.15.29:0/1274199295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde48cc0 0x5559fde57730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.729+0000 7fb6ceffd640 1 --2- 172.21.15.29:0/1274199295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde482d0 0x5559fde59660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.729+0000 7fb6d8667740 10 client.0 _ll_register_callbacks cb 0x7fb6c0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:18.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.729+0000 7fb6cf7fe640 1 -- 172.21.15.29:0/1274199295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde482d0 msgr2=0x5559fde59660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:18.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.729+0000 7fb6cf7fe640 1 --2- 172.21.15.29:0/1274199295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde482d0 0x5559fde59660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.729+0000 7fb6cf7fe640 1 -- 172.21.15.29:0/1274199295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde48cc0 msgr2=0x5559fde57730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:18.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.729+0000 7fb6cf7fe640 1 --2- 172.21.15.29:0/1274199295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde48cc0 0x5559fde57730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.729+0000 7fb6cf7fe640 1 -- 172.21.15.29:0/1274199295 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5559fddaf610 con 0x5559fde437f0 2024-03-11T19:13:18.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.729+0000 7fb6cf7fe640 1 --2- 172.21.15.29:0/1274199295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde437f0 0x5559fde59120 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb6c009a740 tx=0x7fb6c0096ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1859f5e212d413f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:18.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.729+0000 7fb6aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:13:18.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.730+0000 7fb6aeffd640 1 -- 172.21.15.29:0/1274199295 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c00ba040 con 0x5559fde437f0 2024-03-11T19:13:18.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.730+0000 7fb6aeffd640 1 -- 172.21.15.29:0/1274199295 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb6c00a3160 con 0x5559fde437f0 2024-03-11T19:13:18.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.730+0000 7fb6d8667740 1 -- 172.21.15.29:0/1274199295 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb6c8032e10 con 0x5559fde437f0 2024-03-11T19:13:18.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.730+0000 7fb6aeffd640 1 -- 172.21.15.29:0/1274199295 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6c0017e80 con 0x5559fde437f0 2024-03-11T19:13:18.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.730+0000 7fb6d8667740 10 client.25772 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:18.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.730+0000 7fb6d8667740 1 -- 172.21.15.29:0/1274199295 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5559fdd6f570 con 0x5559fde437f0 2024-03-11T19:13:18.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.730+0000 7fb6aeffd640 1 -- 172.21.15.29:0/1274199295 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb6c8032e10 con 0x5559fde437f0 2024-03-11T19:13:18.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.731+0000 7fb6aeffd640 1 -- 172.21.15.29:0/1274199295 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb6c001f060 con 0x5559fde437f0 2024-03-11T19:13:18.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.732+0000 7fb6d8667740 10 client.25772 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:18.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.732+0000 7fb6d8667740 10 client.25772 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:13:18.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.732+0000 7fb6d8667740 20 client.25772 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:18.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.732+0000 7fb6d8667740 1 --2- 172.21.15.29:0/1274199295 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5559fde4de30 0x5559fdeb7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.732+0000 7fb6d8667740 4 client.25772 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:13:18.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.732+0000 7fb6d8667740 1 -- 172.21.15.29:0/1274199295 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5559fdaed720 con 0x5559fde4de30 2024-03-11T19:13:18.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.733+0000 7fb6ceffd640 1 --2- 172.21.15.29:0/1274199295 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5559fde4de30 0x5559fdeb7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.734+0000 7fb6aeffd640 10 client.25772 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:18.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.734+0000 7fb6ceffd640 1 --2- 172.21.15.29:0/1274199295 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5559fde4de30 0x5559fdeb7d00 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9af3b5f0008ed1dc server_cookie=809c29651654b41 in_seq=0 out_seq=0 2024-03-11T19:13:18.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.734+0000 7fb6aeffd640 1 -- 172.21.15.29:0/1274199295 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5559fdaed720 con 0x5559fde4de30 2024-03-11T19:13:18.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.734+0000 7fb6aeffd640 10 client.25772 handle_command_reply: tid=0 2024-03-11T19:13:18.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.734+0000 7fb6d8667740 1 client.25772 shutdown 2024-03-11T19:13:18.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.734+0000 7fb6d8667740 1 -- 172.21.15.29:0/1274199295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde437f0 msgr2=0x5559fde59120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:18.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.734+0000 7fb6d8667740 1 --2- 172.21.15.29:0/1274199295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde437f0 0x5559fde59120 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb6c009a740 tx=0x7fb6c0096ea0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.735+0000 7fb6d8667740 1 -- 172.21.15.29:0/1274199295 shutdown_connections 2024-03-11T19:13:18.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.735+0000 7fb6d8667740 1 --2- 172.21.15.29:0/1274199295 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5559fde4de30 0x5559fdeb7d00 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.735+0000 7fb6d8667740 1 --2- 172.21.15.29:0/1274199295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5559fde48cc0 0x5559fde57730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.735+0000 7fb6aeffd640 0 client.25772 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:18.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.735+0000 7fb6d8667740 1 --2- 172.21.15.29:0/1274199295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5559fde482d0 0x5559fde59660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.735+0000 7fb6d8667740 1 --2- 172.21.15.29:0/1274199295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5559fde437f0 0x5559fde59120 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.735+0000 7fb6d8667740 1 -- 172.21.15.29:0/1274199295 >> 172.21.15.29:0/1274199295 conn(0x5559fde498c0 msgr2=0x5559fde65190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:18.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.735+0000 7fb6d8667740 1 -- 172.21.15.29:0/1274199295 shutdown_connections 2024-03-11T19:13:18.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.735+0000 7fb6d8667740 1 -- 172.21.15.29:0/1274199295 wait complete. 2024-03-11T19:13:18.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.735+0000 7fb6d8667740 20 client.25772 trim_cache size 0 max 16384 2024-03-11T19:13:18.742 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:13:18.742 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:13:18.743 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:13:18.743 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:13:18.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.736+0000 7fb6d5a88640 1 -- 172.21.15.29:0/2510485055 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb6b80a1b00 msgr2=0x7fb6b80a3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:18.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.736+0000 7fb6d5a88640 1 --2- 172.21.15.29:0/2510485055 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb6b80a1b00 0x7fb6b80a3fc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb6d019ebf0 tx=0x7fb6c400b390 comp rx=0 tx=0).stop 2024-03-11T19:13:18.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.736+0000 7fb6d5a88640 1 -- 172.21.15.29:0/2510485055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6d00ff0c0 msgr2=0x7fb6d019d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:18.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.736+0000 7fb6d5a88640 1 --2- 172.21.15.29:0/2510485055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6d00ff0c0 0x7fb6d019d6e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb6c000e9c0 tx=0x7fb6c000ee80 comp rx=0 tx=0).stop 2024-03-11T19:13:18.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.736+0000 7fb6d5a88640 1 -- 172.21.15.29:0/2510485055 shutdown_connections 2024-03-11T19:13:18.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.736+0000 7fb6d5a88640 1 --2- 172.21.15.29:0/2510485055 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6d0100b30 0x7fb6d006cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.736+0000 7fb6d5a88640 1 --2- 172.21.15.29:0/2510485055 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb6b80a1b00 0x7fb6b80a3fc0 secure :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb6d019ebf0 tx=0x7fb6c400b390 comp rx=0 tx=0).stop 2024-03-11T19:13:18.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.736+0000 7fb6d5a88640 1 --2- 172.21.15.29:0/2510485055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6d0100170 0x7fb6d019dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.736+0000 7fb6d5a88640 1 --2- 172.21.15.29:0/2510485055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6d00ff0c0 0x7fb6d019d6e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.736+0000 7fb6d5a88640 1 -- 172.21.15.29:0/2510485055 >> 172.21.15.29:0/2510485055 conn(0x7fb6d00fa960 msgr2=0x7fb6d00fc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:18.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.736+0000 7fb6d5a88640 1 -- 172.21.15.29:0/2510485055 shutdown_connections 2024-03-11T19:13:18.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.737+0000 7fb6d5a88640 1 -- 172.21.15.29:0/2510485055 wait complete. 2024-03-11T19:13:18.757 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:13:18.757 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:13:18.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.916+0000 7f9a4b1b5640 1 -- 172.21.15.29:0/1734125387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a44101c50 msgr2=0x7f9a441084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:18.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.916+0000 7f9a4b1b5640 1 --2- 172.21.15.29:0/1734125387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a44101c50 0x7f9a441084e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009ca0 tx=0x7f9a3c02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.916+0000 7f9a4b1b5640 1 -- 172.21.15.29:0/1734125387 shutdown_connections 2024-03-11T19:13:18.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.916+0000 7f9a4b1b5640 1 --2- 172.21.15.29:0/1734125387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a44101c50 0x7f9a441084e0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.916+0000 7f9a4b1b5640 1 --2- 172.21.15.29:0/1734125387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a44101290 0x7f9a44101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.916+0000 7f9a4b1b5640 1 --2- 172.21.15.29:0/1734125387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a441000a0 0x7f9a441004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.916+0000 7f9a4b1b5640 1 -- 172.21.15.29:0/1734125387 >> 172.21.15.29:0/1734125387 conn(0x7f9a440fb610 msgr2=0x7f9a440fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:18.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.916+0000 7f9a4b1b5640 1 -- 172.21.15.29:0/1734125387 shutdown_connections 2024-03-11T19:13:18.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.916+0000 7f9a4b1b5640 1 -- 172.21.15.29:0/1734125387 wait complete. 2024-03-11T19:13:18.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.917+0000 7f9a4b1b5640 1 Processor -- start 2024-03-11T19:13:18.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.917+0000 7f9a4b1b5640 1 -- start start 2024-03-11T19:13:18.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.917+0000 7f9a4b1b5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a441000a0 0x7f9a44195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.917+0000 7f9a4b1b5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a44101290 0x7f9a44196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.917+0000 7f9a4b1b5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a44101c50 0x7f9a4419d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.917+0000 7f9a4b1b5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4419dae0 con 0x7f9a44101c50 2024-03-11T19:13:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.917+0000 7f9a4b1b5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4419dc50 con 0x7f9a441000a0 2024-03-11T19:13:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.917+0000 7f9a4b1b5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4419de10 con 0x7f9a44101290 2024-03-11T19:13:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.917+0000 7f9a43fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a44101290 0x7f9a44196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.917+0000 7f9a43fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a44101290 0x7f9a44196530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:43094/0 (socket says 172.21.15.29:43094) 2024-03-11T19:13:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.917+0000 7f9a43fff640 1 -- 172.21.15.29:0/4230327861 learned_addr learned my addr 172.21.15.29:0/4230327861 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.917+0000 7f9a49752640 1 --2- 172.21.15.29:0/4230327861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a44101c50 0x7f9a4419d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.917+0000 7f9a48f51640 1 --2- 172.21.15.29:0/4230327861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a441000a0 0x7f9a44195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.918+0000 7f9a43fff640 1 -- 172.21.15.29:0/4230327861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a441000a0 msgr2=0x7f9a44195ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.918+0000 7f9a43fff640 1 --2- 172.21.15.29:0/4230327861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a441000a0 0x7f9a44195ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.918+0000 7f9a43fff640 1 -- 172.21.15.29:0/4230327861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a44101c50 msgr2=0x7f9a4419d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.918+0000 7f9a43fff640 1 --2- 172.21.15.29:0/4230327861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a44101c50 0x7f9a4419d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.918+0000 7f9a43fff640 1 -- 172.21.15.29:0/4230327861 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c009900 con 0x7f9a44101290 2024-03-11T19:13:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.918+0000 7f9a49752640 1 --2- 172.21.15.29:0/4230327861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a44101c50 0x7f9a4419d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.918+0000 7f9a48f51640 1 --2- 172.21.15.29:0/4230327861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a441000a0 0x7f9a44195ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:18.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.918+0000 7f9a43fff640 1 --2- 172.21.15.29:0/4230327861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a44101290 0x7f9a44196530 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9a3000e9f0 tx=0x7f9a3000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:18.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.918+0000 7f9a41ffb640 1 -- 172.21.15.29:0/4230327861 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a30009700 con 0x7f9a44101290 2024-03-11T19:13:18.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.918+0000 7f9a4b1b5640 1 -- 172.21.15.29:0/4230327861 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a4419e0f0 con 0x7f9a44101290 2024-03-11T19:13:18.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.918+0000 7f9a41ffb640 1 -- 172.21.15.29:0/4230327861 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9a3000cd80 con 0x7f9a44101290 2024-03-11T19:13:18.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.918+0000 7f9a41ffb640 1 -- 172.21.15.29:0/4230327861 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a30018840 con 0x7f9a44101290 2024-03-11T19:13:18.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.918+0000 7f9a4b1b5640 1 -- 172.21.15.29:0/4230327861 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4419e650 con 0x7f9a44101290 2024-03-11T19:13:18.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.920+0000 7f9a4b1b5640 1 -- 172.21.15.29:0/4230327861 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a44107440 con 0x7f9a44101290 2024-03-11T19:13:18.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.921+0000 7f9a41ffb640 1 -- 172.21.15.29:0/4230327861 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9a300189a0 con 0x7f9a44101290 2024-03-11T19:13:18.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.922+0000 7f9a41ffb640 1 --2- 172.21.15.29:0/4230327861 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a340a1a30 0x7f9a340a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:18.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.922+0000 7f9a41ffb640 1 -- 172.21.15.29:0/4230327861 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9a30014070 con 0x7f9a44101290 2024-03-11T19:13:18.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.922+0000 7f9a48f51640 1 --2- 172.21.15.29:0/4230327861 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a340a1a30 0x7f9a340a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:18.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.923+0000 7f9a48f51640 1 --2- 172.21.15.29:0/4230327861 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a340a1a30 0x7f9a340a3ef0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9a441010f0 tx=0x7f9a38005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:18.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:18.926+0000 7f9a41ffb640 1 -- 172.21.15.29:0/4230327861 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9a300969c0 con 0x7f9a44101290 2024-03-11T19:13:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.276+0000 7f9a4b1b5640 1 -- 172.21.15.29:0/4230327861 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a4410b960 con 0x7f9a44101290 2024-03-11T19:13:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.277+0000 7f9a41ffb640 1 -- 172.21.15.29:0/4230327861 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f9a3009a080 con 0x7f9a44101290 2024-03-11T19:13:19.281 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:13:19.282 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:13:19.282 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:13:19.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.279+0000 7f9a4b1b5640 1 -- 172.21.15.29:0/4230327861 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a340a1a30 msgr2=0x7f9a340a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.279+0000 7f9a4b1b5640 1 --2- 172.21.15.29:0/4230327861 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a340a1a30 0x7f9a340a3ef0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9a441010f0 tx=0x7f9a38005f90 comp rx=0 tx=0).stop 2024-03-11T19:13:19.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.279+0000 7f9a4b1b5640 1 -- 172.21.15.29:0/4230327861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a44101290 msgr2=0x7f9a44196530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.279+0000 7f9a4b1b5640 1 --2- 172.21.15.29:0/4230327861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a44101290 0x7f9a44196530 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9a3000e9f0 tx=0x7f9a3000eeb0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.279+0000 7f9a4b1b5640 1 -- 172.21.15.29:0/4230327861 shutdown_connections 2024-03-11T19:13:19.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.279+0000 7f9a4b1b5640 1 --2- 172.21.15.29:0/4230327861 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a340a1a30 0x7f9a340a3ef0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.279+0000 7f9a4b1b5640 1 --2- 172.21.15.29:0/4230327861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a44101c50 0x7f9a4419d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.279+0000 7f9a4b1b5640 1 --2- 172.21.15.29:0/4230327861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a44101290 0x7f9a44196530 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.279+0000 7f9a4b1b5640 1 --2- 172.21.15.29:0/4230327861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a441000a0 0x7f9a44195ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.279+0000 7f9a4b1b5640 1 -- 172.21.15.29:0/4230327861 >> 172.21.15.29:0/4230327861 conn(0x7f9a440fb610 msgr2=0x7f9a44104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.279+0000 7f9a4b1b5640 1 -- 172.21.15.29:0/4230327861 shutdown_connections 2024-03-11T19:13:19.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.279+0000 7f9a4b1b5640 1 -- 172.21.15.29:0/4230327861 wait complete. 2024-03-11T19:13:19.294 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:13:19.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:13:19.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.394+0000 7f23baaeb640 1 -- 172.21.15.29:0/940235818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b4101280 msgr2=0x7f23b4101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.394+0000 7f23baaeb640 1 --2- 172.21.15.29:0/940235818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b4101280 0x7f23b4101700 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f23a0009900 tx=0x7f23a002bc30 comp rx=0 tx=0).stop 2024-03-11T19:13:19.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.394+0000 7f23baaeb640 1 -- 172.21.15.29:0/940235818 shutdown_connections 2024-03-11T19:13:19.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.394+0000 7f23baaeb640 1 --2- 172.21.15.29:0/940235818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23b4101c40 0x7f23b41084d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.394+0000 7f23baaeb640 1 --2- 172.21.15.29:0/940235818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b4101280 0x7f23b4101700 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.394+0000 7f23baaeb640 1 --2- 172.21.15.29:0/940235818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23b4100090 0x7f23b41004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.394+0000 7f23baaeb640 1 -- 172.21.15.29:0/940235818 >> 172.21.15.29:0/940235818 conn(0x7f23b40fb620 msgr2=0x7f23b40fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.394+0000 7f23baaeb640 1 -- 172.21.15.29:0/940235818 shutdown_connections 2024-03-11T19:13:19.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.394+0000 7f23baaeb640 1 -- 172.21.15.29:0/940235818 wait complete. 2024-03-11T19:13:19.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.394+0000 7f23baaeb640 1 Processor -- start 2024-03-11T19:13:19.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23baaeb640 1 -- start start 2024-03-11T19:13:19.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23baaeb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23b4100090 0x7f23b406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23baaeb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b4101280 0x7f23b419bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23baaeb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23b4101c40 0x7f23b419e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23baaeb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b419ecc0 con 0x7f23b4101280 2024-03-11T19:13:19.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23baaeb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b419ee30 con 0x7f23b4101c40 2024-03-11T19:13:19.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23baaeb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b419eff0 con 0x7f23b4100090 2024-03-11T19:13:19.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23b3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b4101280 0x7f23b419bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23b3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b4101280 0x7f23b419bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54770/0 (socket says 172.21.15.29:54770) 2024-03-11T19:13:19.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23b3fff640 1 -- 172.21.15.29:0/3528632179 learned_addr learned my addr 172.21.15.29:0/3528632179 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:19.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23b9088640 1 --2- 172.21.15.29:0/3528632179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23b4101c40 0x7f23b419e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23b8887640 1 --2- 172.21.15.29:0/3528632179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23b4100090 0x7f23b406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23b3fff640 1 -- 172.21.15.29:0/3528632179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23b4100090 msgr2=0x7f23b406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23b3fff640 1 --2- 172.21.15.29:0/3528632179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23b4100090 0x7f23b406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23b3fff640 1 -- 172.21.15.29:0/3528632179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23b4101c40 msgr2=0x7f23b419e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23b3fff640 1 --2- 172.21.15.29:0/3528632179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23b4101c40 0x7f23b419e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23b3fff640 1 -- 172.21.15.29:0/3528632179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a0009560 con 0x7f23b4101280 2024-03-11T19:13:19.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23b8887640 1 --2- 172.21.15.29:0/3528632179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23b4100090 0x7f23b406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:13:19.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23b3fff640 1 --2- 172.21.15.29:0/3528632179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b4101280 0x7f23b419bec0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f23a0006010 tx=0x7f23a0004370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.395+0000 7f23b1ffb640 1 -- 172.21.15.29:0/3528632179 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a0004a50 con 0x7f23b4101280 2024-03-11T19:13:19.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.396+0000 7f23b1ffb640 1 -- 172.21.15.29:0/3528632179 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f23a002de90 con 0x7f23b4101280 2024-03-11T19:13:19.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.396+0000 7f23baaeb640 1 -- 172.21.15.29:0/3528632179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b419f210 con 0x7f23b4101280 2024-03-11T19:13:19.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.396+0000 7f23b1ffb640 1 -- 172.21.15.29:0/3528632179 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a0005410 con 0x7f23b4101280 2024-03-11T19:13:19.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.396+0000 7f23baaeb640 1 -- 172.21.15.29:0/3528632179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b419f720 con 0x7f23b4101280 2024-03-11T19:13:19.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.398+0000 7f23b1ffb640 1 -- 172.21.15.29:0/3528632179 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f23a0005570 con 0x7f23b4101280 2024-03-11T19:13:19.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.398+0000 7f23bd799740 1 Processor -- start 2024-03-11T19:13:19.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.398+0000 7f23bd799740 1 -- start start 2024-03-11T19:13:19.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23bd799740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 0x55f74f3b1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23bd799740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b1c60 0x55f74f3b6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23bd799740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b6dd0 0x55f74f3b9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23bd799740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f74f36d960 con 0x55f74f3b6dd0 2024-03-11T19:13:19.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23bd799740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f74f321310 con 0x55f74f3b1270 2024-03-11T19:13:19.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23bd799740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f74f31e840 con 0x55f74f3b1c60 2024-03-11T19:13:19.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23b1ffb640 1 --2- 172.21.15.29:0/3528632179 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23a40a1a30 0x7f23a40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23b3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b6dd0 0x55f74f3b9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23b3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b6dd0 0x55f74f3b9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54784/0 (socket says 172.21.15.29:54784) 2024-03-11T19:13:19.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23b3fff640 1 -- 172.21.15.29:0/2512494857 learned_addr learned my addr 172.21.15.29:0/2512494857 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:19.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23b9088640 1 --2- 172.21.15.29:0/3528632179 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23a40a1a30 0x7f23a40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23b9088640 1 --2- 172.21.15.29:0/2512494857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 0x55f74f3b1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23b8887640 1 --2- 172.21.15.29:0/2512494857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b1c60 0x55f74f3b6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23b1ffb640 1 -- 172.21.15.29:0/3528632179 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f23a00ef500 con 0x7f23b4101280 2024-03-11T19:13:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23b3fff640 1 -- 172.21.15.29:0/2512494857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b1c60 msgr2=0x55f74f3b6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23b3fff640 1 --2- 172.21.15.29:0/2512494857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b1c60 0x55f74f3b6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23b3fff640 1 -- 172.21.15.29:0/2512494857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 msgr2=0x55f74f3b1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23b3fff640 1 --2- 172.21.15.29:0/2512494857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 0x55f74f3b1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23b3fff640 1 -- 172.21.15.29:0/2512494857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f74f2dc060 con 0x55f74f3b6dd0 2024-03-11T19:13:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23b3fff640 1 --2- 172.21.15.29:0/2512494857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b6dd0 0x55f74f3b9260 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f23a00ed000 tx=0x7f23a002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=755bfb4b7e3006f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f23b9088640 1 --2- 172.21.15.29:0/3528632179 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23a40a1a30 0x7f23a40a3ef0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x55f74f3b24c0 tx=0x7f23ac004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.399+0000 7f239effd640 1 -- 172.21.15.29:0/2512494857 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a00a7110 con 0x55f74f3b6dd0 2024-03-11T19:13:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.400+0000 7f239effd640 1 -- 172.21.15.29:0/2512494857 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f23a00bbfb0 con 0x55f74f3b6dd0 2024-03-11T19:13:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.400+0000 7f239effd640 1 -- 172.21.15.29:0/2512494857 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a00ed110 con 0x55f74f3b6dd0 2024-03-11T19:13:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.400+0000 7f23bd799740 1 -- 172.21.15.29:0/2512494857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b6dd0 msgr2=0x55f74f3b9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.400+0000 7f23bd799740 1 --2- 172.21.15.29:0/2512494857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b6dd0 0x55f74f3b9260 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f23a00ed000 tx=0x7f23a002bc30 comp rx=0 tx=0).stop 2024-03-11T19:13:19.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.400+0000 7f23bd799740 1 -- 172.21.15.29:0/2512494857 shutdown_connections 2024-03-11T19:13:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.400+0000 7f23bd799740 1 --2- 172.21.15.29:0/2512494857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b6dd0 0x55f74f3b9260 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.400+0000 7f23bd799740 1 --2- 172.21.15.29:0/2512494857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b1c60 0x55f74f3b6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.400+0000 7f23bd799740 1 --2- 172.21.15.29:0/2512494857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 0x55f74f3b1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.400+0000 7f23bd799740 1 -- 172.21.15.29:0/2512494857 >> 172.21.15.29:0/2512494857 conn(0x55f74f3ac7e0 msgr2=0x55f74f3acbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.400+0000 7f23bd799740 1 -- 172.21.15.29:0/2512494857 shutdown_connections 2024-03-11T19:13:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.400+0000 7f23bd799740 1 -- 172.21.15.29:0/2512494857 wait complete. 2024-03-11T19:13:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.400+0000 7f23bd799740 1 -- start start 2024-03-11T19:13:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.401+0000 7f23bd799740 1 Processor -- start 2024-03-11T19:13:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.401+0000 7f23bd799740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 0x55f74f3bb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.401+0000 7f23bd799740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b1c60 0x55f74f3bb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.401+0000 7f23bd799740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b6dd0 0x55f74f3c26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.401+0000 7f23bd799740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f74f31a020 con 0x55f74f3b6dd0 2024-03-11T19:13:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.401+0000 7f23bd799740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f74efed430 con 0x55f74f3b1270 2024-03-11T19:13:19.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.401+0000 7f23bd799740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f74efa9e00 con 0x55f74f3b1c60 2024-03-11T19:13:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.401+0000 7f23bd799740 10 client.0 _ll_register_callbacks cb 0x55f74f2c1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.401+0000 7f23b9088640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b6dd0 0x55f74f3c26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.401+0000 7f23b9088640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b6dd0 0x55f74f3c26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54788/0 (socket says 172.21.15.29:54788) 2024-03-11T19:13:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.401+0000 7f23b9088640 1 -- 172.21.15.29:0/769603224 learned_addr learned my addr 172.21.15.29:0/769603224 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.401+0000 7f23b8887640 1 --2- 172.21.15.29:0/769603224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 0x55f74f3bb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.401+0000 7f23b3fff640 1 --2- 172.21.15.29:0/769603224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b1c60 0x55f74f3bb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.401+0000 7f23b9088640 1 -- 172.21.15.29:0/769603224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b1c60 msgr2=0x55f74f3bb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.401+0000 7f23b9088640 1 --2- 172.21.15.29:0/769603224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b1c60 0x55f74f3bb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.401+0000 7f23b9088640 1 -- 172.21.15.29:0/769603224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 msgr2=0x55f74f3bb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.401+0000 7f23b9088640 1 --2- 172.21.15.29:0/769603224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 0x55f74f3bb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.401+0000 7f23b9088640 1 -- 172.21.15.29:0/769603224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b419f720 con 0x55f74f3b6dd0 2024-03-11T19:13:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.401+0000 7f23b3fff640 1 --2- 172.21.15.29:0/769603224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b1c60 0x55f74f3bb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:13:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.402+0000 7f23b9088640 1 --2- 172.21.15.29:0/769603224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b6dd0 0x55f74f3c26d0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f23ac010f70 tx=0x7f23ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9dfc9301ea779e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.402+0000 7f2393fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:13:19.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.402+0000 7f2393fff640 1 -- 172.21.15.29:0/769603224 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac0215f0 con 0x55f74f3b6dd0 2024-03-11T19:13:19.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.402+0000 7f2393fff640 1 -- 172.21.15.29:0/769603224 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f23ac021750 con 0x55f74f3b6dd0 2024-03-11T19:13:19.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.402+0000 7f23bd799740 1 -- 172.21.15.29:0/769603224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f74f0197e0 con 0x55f74f3b6dd0 2024-03-11T19:13:19.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.402+0000 7f2393fff640 1 -- 172.21.15.29:0/769603224 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23ac021910 con 0x55f74f3b6dd0 2024-03-11T19:13:19.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.402+0000 7f23bd799740 10 client.16263 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:19.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.402+0000 7f23bd799740 1 -- 172.21.15.29:0/769603224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55f74f3186a0 con 0x55f74f3b6dd0 2024-03-11T19:13:19.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.402+0000 7f2393fff640 1 -- 172.21.15.29:0/769603224 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f74f0197e0 con 0x55f74f3b6dd0 2024-03-11T19:13:19.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.402+0000 7f2393fff640 1 -- 172.21.15.29:0/769603224 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f23ac022a30 con 0x55f74f3b6dd0 2024-03-11T19:13:19.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.402+0000 7f23bd799740 10 client.16263 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:19.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.402+0000 7f23bd799740 10 client.16263 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:13:19.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.402+0000 7f23bd799740 20 client.16263 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:19.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.402+0000 7f23bd799740 1 --2- 172.21.15.29:0/769603224 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55f74f3c7380 0x55f74f3c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.402+0000 7f23bd799740 4 client.16263 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:13:19.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.402+0000 7f23bd799740 1 -- 172.21.15.29:0/769603224 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f74f056720 con 0x55f74f3c7380 2024-03-11T19:13:19.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.403+0000 7f23b8887640 1 --2- 172.21.15.29:0/769603224 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55f74f3c7380 0x55f74f3c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.403+0000 7f2393fff640 10 client.16263 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:13:19.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.403+0000 7f23b8887640 1 --2- 172.21.15.29:0/769603224 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55f74f3c7380 0x55f74f3c97a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b6219db0d96bf02d server_cookie=4fd36dcc444ebb18 in_seq=0 out_seq=0 2024-03-11T19:13:19.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.403+0000 7f2393fff640 1 -- 172.21.15.29:0/769603224 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f74f056720 con 0x55f74f3c7380 2024-03-11T19:13:19.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.403+0000 7f2393fff640 10 client.16263 handle_command_reply: tid=0 2024-03-11T19:13:19.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.404+0000 7f23bd799740 1 client.16263 shutdown 2024-03-11T19:13:19.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.404+0000 7f23bd799740 1 -- 172.21.15.29:0/769603224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b6dd0 msgr2=0x55f74f3c26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.404+0000 7f23bd799740 1 --2- 172.21.15.29:0/769603224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b6dd0 0x55f74f3c26d0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f23ac010f70 tx=0x7f23ac004a40 comp rx=0 tx=0).stop 2024-03-11T19:13:19.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.404+0000 7f23bd799740 1 -- 172.21.15.29:0/769603224 shutdown_connections 2024-03-11T19:13:19.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.404+0000 7f23bd799740 1 --2- 172.21.15.29:0/769603224 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55f74f3c7380 0x55f74f3c97a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.404+0000 7f23bd799740 1 --2- 172.21.15.29:0/769603224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b6dd0 0x55f74f3c26d0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.404+0000 7f23bd799740 1 --2- 172.21.15.29:0/769603224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b1c60 0x55f74f3bb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.404+0000 7f23bd799740 1 --2- 172.21.15.29:0/769603224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 0x55f74f3bb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.404+0000 7f2393fff640 0 client.16263 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:13:19.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.404+0000 7f23bd799740 1 -- 172.21.15.29:0/769603224 >> 172.21.15.29:0/769603224 conn(0x55f74f3ac7e0 msgr2=0x55f74f3abb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.404+0000 7f23bd799740 1 -- 172.21.15.29:0/769603224 shutdown_connections 2024-03-11T19:13:19.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.404+0000 7f23bd799740 1 -- 172.21.15.29:0/769603224 wait complete. 2024-03-11T19:13:19.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.404+0000 7f23bd799740 20 client.16263 trim_cache size 0 max 16384 2024-03-11T19:13:19.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.415+0000 7f23bd799740 1 Processor -- start 2024-03-11T19:13:19.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.415+0000 7f23bd799740 1 -- start start 2024-03-11T19:13:19.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f23bd799740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 0x55f74f3bb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f23bd799740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b1c60 0x55f74f3bb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f23bd799740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b6dd0 0x55f74f3c2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f23bd799740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f74f29f130 con 0x55f74f3b1c60 2024-03-11T19:13:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f23bd799740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f74f3b0e30 con 0x55f74f3b1270 2024-03-11T19:13:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f23bd799740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f74f3ba5f0 con 0x55f74f3b6dd0 2024-03-11T19:13:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f23b3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b1c60 0x55f74f3bb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f23b3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b1c60 0x55f74f3bb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54790/0 (socket says 172.21.15.29:54790) 2024-03-11T19:13:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f23b3fff640 1 -- 172.21.15.29:0/3171410536 learned_addr learned my addr 172.21.15.29:0/3171410536 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f23b9088640 1 --2- 172.21.15.29:0/3171410536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b6dd0 0x55f74f3c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f23b8887640 1 --2- 172.21.15.29:0/3171410536 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 0x55f74f3bb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f23b3fff640 1 -- 172.21.15.29:0/3171410536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b6dd0 msgr2=0x55f74f3c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f23b3fff640 1 --2- 172.21.15.29:0/3171410536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b6dd0 0x55f74f3c2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f23b3fff640 1 -- 172.21.15.29:0/3171410536 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 msgr2=0x55f74f3bb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f23b3fff640 1 --2- 172.21.15.29:0/3171410536 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 0x55f74f3bb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f23b3fff640 1 -- 172.21.15.29:0/3171410536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f74f2b4c20 con 0x55f74f3b1c60 2024-03-11T19:13:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f23b8887640 1 --2- 172.21.15.29:0/3171410536 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 0x55f74f3bb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:13:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f23b3fff640 1 --2- 172.21.15.29:0/3171410536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b1c60 0x55f74f3bb690 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f23a00bc120 tx=0x7f23a00dc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96328ca748cc919 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f239e7fc640 1 -- 172.21.15.29:0/3171410536 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a00bcb70 con 0x55f74f3b1c60 2024-03-11T19:13:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f239e7fc640 1 -- 172.21.15.29:0/3171410536 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f23a00c05c0 con 0x55f74f3b1c60 2024-03-11T19:13:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.416+0000 7f239e7fc640 1 -- 172.21.15.29:0/3171410536 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a00ed1e0 con 0x55f74f3b1c60 2024-03-11T19:13:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.417+0000 7f23bd799740 1 -- 172.21.15.29:0/3171410536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b1c60 msgr2=0x55f74f3bb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.417+0000 7f23bd799740 1 --2- 172.21.15.29:0/3171410536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b1c60 0x55f74f3bb690 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f23a00bc120 tx=0x7f23a00dc040 comp rx=0 tx=0).stop 2024-03-11T19:13:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.417+0000 7f23bd799740 1 -- 172.21.15.29:0/3171410536 shutdown_connections 2024-03-11T19:13:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.417+0000 7f23bd799740 1 --2- 172.21.15.29:0/3171410536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b6dd0 0x55f74f3c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.417+0000 7f23bd799740 1 --2- 172.21.15.29:0/3171410536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b1c60 0x55f74f3bb690 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.417+0000 7f23bd799740 1 --2- 172.21.15.29:0/3171410536 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 0x55f74f3bb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.417+0000 7f23bd799740 1 -- 172.21.15.29:0/3171410536 >> 172.21.15.29:0/3171410536 conn(0x55f74f3ac7e0 msgr2=0x55f74f3ce100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.417+0000 7f23bd799740 1 -- 172.21.15.29:0/3171410536 shutdown_connections 2024-03-11T19:13:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.417+0000 7f23bd799740 1 -- 172.21.15.29:0/3171410536 wait complete. 2024-03-11T19:13:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.417+0000 7f23bd799740 1 -- start start 2024-03-11T19:13:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.417+0000 7f23bd799740 1 Processor -- start 2024-03-11T19:13:19.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23bd799740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 0x55f74f3c25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23bd799740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b1c60 0x55f74f3c0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23bd799740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b6dd0 0x55f74f3c10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23bd799740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f74f3c17f0 con 0x55f74f3b1c60 2024-03-11T19:13:19.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23bd799740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f74f3c1960 con 0x55f74f3b1270 2024-03-11T19:13:19.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23bd799740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f74f41d870 con 0x55f74f3b6dd0 2024-03-11T19:13:19.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23bd799740 10 client.0 _ll_register_callbacks cb 0x7f23ac010680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:19.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23b3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b1c60 0x55f74f3c0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23b3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b1c60 0x55f74f3c0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54796/0 (socket says 172.21.15.29:54796) 2024-03-11T19:13:19.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23b3fff640 1 -- 172.21.15.29:0/1144443846 learned_addr learned my addr 172.21.15.29:0/1144443846 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:19.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23b9088640 1 --2- 172.21.15.29:0/1144443846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b6dd0 0x55f74f3c10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23b8887640 1 --2- 172.21.15.29:0/1144443846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 0x55f74f3c25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23b3fff640 1 -- 172.21.15.29:0/1144443846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b6dd0 msgr2=0x55f74f3c10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23b3fff640 1 --2- 172.21.15.29:0/1144443846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b6dd0 0x55f74f3c10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23b3fff640 1 -- 172.21.15.29:0/1144443846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 msgr2=0x55f74f3c25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23b3fff640 1 --2- 172.21.15.29:0/1144443846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 0x55f74f3c25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23b3fff640 1 -- 172.21.15.29:0/1144443846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f74f2dc060 con 0x55f74f3b1c60 2024-03-11T19:13:19.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23b3fff640 1 --2- 172.21.15.29:0/1144443846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b1c60 0x55f74f3c0bb0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f23a00a7010 tx=0x7f23a00ef760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c2643e897d1f602 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f2393fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:13:19.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f2393fff640 1 -- 172.21.15.29:0/1144443846 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a00a70b0 con 0x55f74f3b1c60 2024-03-11T19:13:19.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f2393fff640 1 -- 172.21.15.29:0/1144443846 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f23a00bc6d0 con 0x55f74f3b1c60 2024-03-11T19:13:19.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23bd799740 1 -- 172.21.15.29:0/1144443846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f23a00bbfb0 con 0x55f74f3b1c60 2024-03-11T19:13:19.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f2393fff640 1 -- 172.21.15.29:0/1144443846 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23a00bc890 con 0x55f74f3b1c60 2024-03-11T19:13:19.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23bd799740 10 client.16269 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:19.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.418+0000 7f23bd799740 1 -- 172.21.15.29:0/1144443846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55f74f2da8c0 con 0x55f74f3b1c60 2024-03-11T19:13:19.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.419+0000 7f2393fff640 1 -- 172.21.15.29:0/1144443846 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f23a00a1080 con 0x55f74f3b1c60 2024-03-11T19:13:19.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.419+0000 7f2393fff640 1 -- 172.21.15.29:0/1144443846 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f23a003c070 con 0x55f74f3b1c60 2024-03-11T19:13:19.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.419+0000 7f23bd799740 10 client.16269 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:19.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.419+0000 7f23bd799740 10 client.16269 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:13:19.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.419+0000 7f23bd799740 20 client.16269 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:19.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.419+0000 7f23bd799740 1 --2- 172.21.15.29:0/1144443846 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55f74f41eed0 0x55f74f4212f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.419+0000 7f23bd799740 4 client.16269 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:13:19.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.419+0000 7f23bd799740 1 -- 172.21.15.29:0/1144443846 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f74f0a5890 con 0x55f74f41eed0 2024-03-11T19:13:19.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.419+0000 7f23b8887640 1 --2- 172.21.15.29:0/1144443846 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55f74f41eed0 0x55f74f4212f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.419+0000 7f2393fff640 10 client.16269 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:13:19.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.419+0000 7f23b8887640 1 --2- 172.21.15.29:0/1144443846 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55f74f41eed0 0x55f74f4212f0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5e75dab41cfcfd2b server_cookie=22ed707719300e86 in_seq=0 out_seq=0 2024-03-11T19:13:19.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f2393fff640 1 -- 172.21.15.29:0/1144443846 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f74f0a5890 con 0x55f74f41eed0 2024-03-11T19:13:19.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f2393fff640 10 client.16269 handle_command_reply: tid=0 2024-03-11T19:13:19.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f23bd799740 1 client.16269 shutdown 2024-03-11T19:13:19.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f23bd799740 1 -- 172.21.15.29:0/1144443846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b1c60 msgr2=0x55f74f3c0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f23bd799740 1 --2- 172.21.15.29:0/1144443846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b1c60 0x55f74f3c0bb0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f23a00a7010 tx=0x7f23a00ef760 comp rx=0 tx=0).stop 2024-03-11T19:13:19.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f23bd799740 1 -- 172.21.15.29:0/1144443846 shutdown_connections 2024-03-11T19:13:19.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f23bd799740 1 --2- 172.21.15.29:0/1144443846 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55f74f41eed0 0x55f74f4212f0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f23bd799740 1 --2- 172.21.15.29:0/1144443846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f74f3b6dd0 0x55f74f3c10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f23bd799740 1 --2- 172.21.15.29:0/1144443846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f74f3b1c60 0x55f74f3c0bb0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f2393fff640 0 client.16269 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:13:19.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f23bd799740 1 --2- 172.21.15.29:0/1144443846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f74f3b1270 0x55f74f3c25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f23bd799740 1 -- 172.21.15.29:0/1144443846 >> 172.21.15.29:0/1144443846 conn(0x55f74f3ac7e0 msgr2=0x55f74f3b3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f23bd799740 1 -- 172.21.15.29:0/1144443846 shutdown_connections 2024-03-11T19:13:19.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f23bd799740 1 -- 172.21.15.29:0/1144443846 wait complete. 2024-03-11T19:13:19.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f23bd799740 20 client.16269 trim_cache size 0 max 16384 2024-03-11T19:13:19.431 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:13:19.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f23baaeb640 1 -- 172.21.15.29:0/3528632179 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23a40a1a30 msgr2=0x7f23a40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f23baaeb640 1 --2- 172.21.15.29:0/3528632179 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23a40a1a30 0x7f23a40a3ef0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x55f74f3b24c0 tx=0x7f23ac004ab0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f23baaeb640 1 -- 172.21.15.29:0/3528632179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b4101280 msgr2=0x7f23b419bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f23baaeb640 1 --2- 172.21.15.29:0/3528632179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b4101280 0x7f23b419bec0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f23a0006010 tx=0x7f23a0004370 comp rx=0 tx=0).stop 2024-03-11T19:13:19.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f23baaeb640 1 -- 172.21.15.29:0/3528632179 shutdown_connections 2024-03-11T19:13:19.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f23baaeb640 1 --2- 172.21.15.29:0/3528632179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23b4101c40 0x7f23b419e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.420+0000 7f23baaeb640 1 --2- 172.21.15.29:0/3528632179 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23a40a1a30 0x7f23a40a3ef0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.421+0000 7f23baaeb640 1 --2- 172.21.15.29:0/3528632179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b4101280 0x7f23b419bec0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.421+0000 7f23baaeb640 1 --2- 172.21.15.29:0/3528632179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23b4100090 0x7f23b406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.421+0000 7f23baaeb640 1 -- 172.21.15.29:0/3528632179 >> 172.21.15.29:0/3528632179 conn(0x7f23b40fb620 msgr2=0x7f23b4104650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.421+0000 7f23baaeb640 1 -- 172.21.15.29:0/3528632179 shutdown_connections 2024-03-11T19:13:19.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.421+0000 7f23baaeb640 1 -- 172.21.15.29:0/3528632179 wait complete. 2024-03-11T19:13:19.436 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:13:19.436 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:13:19.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.601+0000 7f12f8cf4640 1 -- 172.21.15.29:0/2713927857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12f4104d10 msgr2=0x7f12f4105190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.601+0000 7f12f8cf4640 1 --2- 172.21.15.29:0/2713927857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12f4104d10 0x7f12f4105190 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f12ec009ca0 tx=0x7f12ec02c100 comp rx=0 tx=0).stop 2024-03-11T19:13:19.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.601+0000 7f12f8cf4640 1 -- 172.21.15.29:0/2713927857 shutdown_connections 2024-03-11T19:13:19.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.601+0000 7f12f8cf4640 1 --2- 172.21.15.29:0/2713927857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12f4104d10 0x7f12f4105190 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.601+0000 7f12f8cf4640 1 --2- 172.21.15.29:0/2713927857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12f4102360 0x7f12f41047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.601+0000 7f12f8cf4640 1 --2- 172.21.15.29:0/2713927857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12f40ff9f0 0x7f12f4101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.601+0000 7f12f8cf4640 1 -- 172.21.15.29:0/2713927857 >> 172.21.15.29:0/2713927857 conn(0x7f12f40fb610 msgr2=0x7f12f40fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.601+0000 7f12f8cf4640 1 -- 172.21.15.29:0/2713927857 shutdown_connections 2024-03-11T19:13:19.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.601+0000 7f12f8cf4640 1 -- 172.21.15.29:0/2713927857 wait complete. 2024-03-11T19:13:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.602+0000 7f12f8cf4640 1 Processor -- start 2024-03-11T19:13:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.602+0000 7f12f8cf4640 1 -- start start 2024-03-11T19:13:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.602+0000 7f12f8cf4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12f40ff9f0 0x7f12f419d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.602+0000 7f12f8cf4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12f4102360 0x7f12f419dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.602+0000 7f12f8cf4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12f4104d10 0x7f12f41a4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.602+0000 7f12f8cf4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f41a5220 con 0x7f12f40ff9f0 2024-03-11T19:13:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.603+0000 7f12f8cf4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f41a5390 con 0x7f12f4102360 2024-03-11T19:13:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.603+0000 7f12f8cf4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f41a5550 con 0x7f12f4104d10 2024-03-11T19:13:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.603+0000 7f12f259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12f40ff9f0 0x7f12f419d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.603+0000 7f12f259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12f40ff9f0 0x7f12f419d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54802/0 (socket says 172.21.15.29:54802) 2024-03-11T19:13:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.603+0000 7f12f259c640 1 -- 172.21.15.29:0/3960570086 learned_addr learned my addr 172.21.15.29:0/3960570086 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.603+0000 7f12f2d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12f4104d10 0x7f12f41a4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.603+0000 7f12f1d9b640 1 --2- 172.21.15.29:0/3960570086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12f4102360 0x7f12f419dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.603+0000 7f12f259c640 1 -- 172.21.15.29:0/3960570086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12f4104d10 msgr2=0x7f12f41a4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.603+0000 7f12f259c640 1 --2- 172.21.15.29:0/3960570086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12f4104d10 0x7f12f41a4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.603+0000 7f12f259c640 1 -- 172.21.15.29:0/3960570086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12f4102360 msgr2=0x7f12f419dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.603+0000 7f12f259c640 1 --2- 172.21.15.29:0/3960570086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12f4102360 0x7f12f419dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.603+0000 7f12f259c640 1 -- 172.21.15.29:0/3960570086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e4009490 con 0x7f12f40ff9f0 2024-03-11T19:13:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.604+0000 7f12f259c640 1 --2- 172.21.15.29:0/3960570086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12f40ff9f0 0x7f12f419d730 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f12e800e9f0 tx=0x7f12e800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.604+0000 7f12e37fe640 1 -- 172.21.15.29:0/3960570086 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e8009700 con 0x7f12f40ff9f0 2024-03-11T19:13:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.604+0000 7f12e37fe640 1 -- 172.21.15.29:0/3960570086 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f12e800cd80 con 0x7f12f40ff9f0 2024-03-11T19:13:19.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.604+0000 7f12e37fe640 1 -- 172.21.15.29:0/3960570086 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e8018840 con 0x7f12f40ff9f0 2024-03-11T19:13:19.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.604+0000 7f12f8cf4640 1 -- 172.21.15.29:0/3960570086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12ec009900 con 0x7f12f40ff9f0 2024-03-11T19:13:19.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.604+0000 7f12f8cf4640 1 -- 172.21.15.29:0/3960570086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f41a5b60 con 0x7f12f40ff9f0 2024-03-11T19:13:19.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.607+0000 7f12e37fe640 1 -- 172.21.15.29:0/3960570086 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f12e80189a0 con 0x7f12f40ff9f0 2024-03-11T19:13:19.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.608+0000 7f12fb9a6740 1 Processor -- start 2024-03-11T19:13:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.608+0000 7f12fb9a6740 1 -- start start 2024-03-11T19:13:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.608+0000 7f12e37fe640 1 --2- 172.21.15.29:0/3960570086 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f12dc0a1b80 0x7f12dc0a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.608+0000 7f12fb9a6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b56270 0x557402b56690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.608+0000 7f12fb9a6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b56c60 0x557402b5b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.608+0000 7f12fb9a6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b5bdd0 0x557402b5e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.609+0000 7f12fb9a6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557402b12960 con 0x557402b5bdd0 2024-03-11T19:13:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.609+0000 7f12fb9a6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557402ac6310 con 0x557402b56c60 2024-03-11T19:13:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.609+0000 7f12fb9a6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557402ac3840 con 0x557402b56270 2024-03-11T19:13:19.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.609+0000 7f12f2d9d640 1 --2- 172.21.15.29:0/3960570086 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f12dc0a1b80 0x7f12dc0a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.609+0000 7f12f259c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b56270 0x557402b56690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.609+0000 7f12f259c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b56270 0x557402b56690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:43166/0 (socket says 172.21.15.29:43166) 2024-03-11T19:13:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.609+0000 7f12f259c640 1 -- 172.21.15.29:0/1274436472 learned_addr learned my addr 172.21.15.29:0/1274436472 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.609+0000 7f12e37fe640 1 -- 172.21.15.29:0/3960570086 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f12e8014070 con 0x7f12f40ff9f0 2024-03-11T19:13:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.609+0000 7f12f259c640 1 -- 172.21.15.29:0/1274436472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b56c60 msgr2=0x557402b5b760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:13:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.609+0000 7f12f259c640 1 --2- 172.21.15.29:0/1274436472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b56c60 0x557402b5b760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.609+0000 7f12f259c640 1 -- 172.21.15.29:0/1274436472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b5bdd0 msgr2=0x557402b5e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.609+0000 7f12f259c640 1 --2- 172.21.15.29:0/1274436472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b5bdd0 0x557402b5e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.609+0000 7f12f259c640 1 -- 172.21.15.29:0/1274436472 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557402a81060 con 0x557402b56270 2024-03-11T19:13:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.610+0000 7f12f259c640 1 --2- 172.21.15.29:0/1274436472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b56270 0x557402b56690 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f12e8010ed0 tx=0x7f12e8096a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddc163018f941889 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.610+0000 7f12f2d9d640 1 --2- 172.21.15.29:0/3960570086 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f12dc0a1b80 0x7f12dc0a4040 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f12ec005e30 tx=0x7f12ec032380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.610+0000 7f12e0ff9640 1 -- 172.21.15.29:0/1274436472 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e809a6a0 con 0x557402b56270 2024-03-11T19:13:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.610+0000 7f12e0ff9640 1 -- 172.21.15.29:0/1274436472 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f12e809e740 con 0x557402b56270 2024-03-11T19:13:19.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.610+0000 7f12e0ff9640 1 -- 172.21.15.29:0/1274436472 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e8096d10 con 0x557402b56270 2024-03-11T19:13:19.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.611+0000 7f12fb9a6740 1 -- 172.21.15.29:0/1274436472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b56270 msgr2=0x557402b56690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.611+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/1274436472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b56270 0x557402b56690 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f12e8010ed0 tx=0x7f12e8096a70 comp rx=0 tx=0).stop 2024-03-11T19:13:19.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.611+0000 7f12fb9a6740 1 -- 172.21.15.29:0/1274436472 shutdown_connections 2024-03-11T19:13:19.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.611+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/1274436472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b5bdd0 0x557402b5e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.611+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/1274436472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b56c60 0x557402b5b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.611+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/1274436472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b56270 0x557402b56690 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.611+0000 7f12fb9a6740 1 -- 172.21.15.29:0/1274436472 >> 172.21.15.29:0/1274436472 conn(0x557402b517e0 msgr2=0x557402b51be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.611+0000 7f12fb9a6740 1 -- 172.21.15.29:0/1274436472 shutdown_connections 2024-03-11T19:13:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.611+0000 7f12fb9a6740 1 -- 172.21.15.29:0/1274436472 wait complete. 2024-03-11T19:13:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.612+0000 7f12fb9a6740 1 -- start start 2024-03-11T19:13:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.612+0000 7f12fb9a6740 1 Processor -- start 2024-03-11T19:13:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.612+0000 7f12fb9a6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b56270 0x557402b60450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.613+0000 7f12fb9a6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56c60 0x557402b60990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.613+0000 7f12fb9a6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b5bdd0 0x557402b679d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.613+0000 7f12f259c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b56270 0x557402b60450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.613+0000 7f12f1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56c60 0x557402b60990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.613+0000 7f12f259c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b56270 0x557402b60450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:43174/0 (socket says 172.21.15.29:43174) 2024-03-11T19:13:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.613+0000 7f12f1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56c60 0x557402b60990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54820/0 (socket says 172.21.15.29:54820) 2024-03-11T19:13:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.613+0000 7f12f259c640 1 -- 172.21.15.29:0/2763931980 learned_addr learned my addr 172.21.15.29:0/2763931980 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.613+0000 7f12fb9a6740 1 -- 172.21.15.29:0/2763931980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557402abf020 con 0x557402b56c60 2024-03-11T19:13:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.613+0000 7f12fb9a6740 1 -- 172.21.15.29:0/2763931980 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557402792430 con 0x557402b5bdd0 2024-03-11T19:13:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.613+0000 7f12fb9a6740 1 -- 172.21.15.29:0/2763931980 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55740274ee00 con 0x557402b56270 2024-03-11T19:13:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.613+0000 7f12f2d9d640 1 --2- 172.21.15.29:0/2763931980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b5bdd0 0x557402b679d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.613+0000 7f12f259c640 1 -- 172.21.15.29:0/2763931980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b5bdd0 msgr2=0x557402b679d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.613+0000 7f12f259c640 1 --2- 172.21.15.29:0/2763931980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b5bdd0 0x557402b679d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.613+0000 7f12f259c640 1 -- 172.21.15.29:0/2763931980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56c60 msgr2=0x557402b60990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.614+0000 7f12f259c640 1 --2- 172.21.15.29:0/2763931980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56c60 0x557402b60990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.614+0000 7f12f259c640 1 -- 172.21.15.29:0/2763931980 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f41a5b60 con 0x557402b56270 2024-03-11T19:13:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.614+0000 7f12f1d9b640 1 --2- 172.21.15.29:0/2763931980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56c60 0x557402b60990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.614+0000 7f12fb9a6740 10 client.0 _ll_register_callbacks cb 0x557402a66cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.614+0000 7f12fb9a6740 1 -- 172.21.15.29:0/2763931980 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5574027be7e0 con 0x557402b56270 2024-03-11T19:13:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.614+0000 7f12f259c640 1 --2- 172.21.15.29:0/2763931980 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b56270 0x557402b60450 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f12e80857b0 tx=0x7f12e8010d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4218134b6d03e48 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.614+0000 7f12d9ffb640 10 client.25840 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:13:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.614+0000 7f12d9ffb640 1 -- 172.21.15.29:0/2763931980 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e80852e0 con 0x557402b56270 2024-03-11T19:13:19.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.614+0000 7f12d9ffb640 1 -- 172.21.15.29:0/2763931980 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f12e809a6a0 con 0x557402b56270 2024-03-11T19:13:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.615+0000 7f12d9ffb640 1 -- 172.21.15.29:0/2763931980 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12e8085920 con 0x557402b56270 2024-03-11T19:13:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.615+0000 7f12fb9a6740 10 client.25840 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.615+0000 7f12fb9a6740 1 -- 172.21.15.29:0/2763931980 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x557402abd6a0 con 0x557402b56270 2024-03-11T19:13:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.615+0000 7f12d9ffb640 1 -- 172.21.15.29:0/2763931980 <== mon.1 v2:172.21.15.190:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f12e8099af0 con 0x557402b56270 2024-03-11T19:13:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.615+0000 7f12fb9a6740 10 client.25840 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.615+0000 7f12fb9a6740 10 client.25840 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:13:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.615+0000 7f12fb9a6740 20 client.25840 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.615+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/2763931980 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x557402b6c050 0x557402b6e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.615+0000 7f12fb9a6740 4 client.25840 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:13:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.615+0000 7f12fb9a6740 1 -- 172.21.15.29:0/2763931980 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5574027fb720 con 0x557402b6c050 2024-03-11T19:13:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.615+0000 7f12f1d9b640 1 --2- 172.21.15.29:0/2763931980 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x557402b6c050 0x557402b6e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.616+0000 7f12d9ffb640 10 client.25840 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:13:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.616+0000 7f12f1d9b640 1 --2- 172.21.15.29:0/2763931980 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x557402b6c050 0x557402b6e4e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6b89491cd61aafef server_cookie=9324d50ae1ac98a8 in_seq=0 out_seq=0 2024-03-11T19:13:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.617+0000 7f12d9ffb640 1 -- 172.21.15.29:0/2763931980 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5574027fb720 con 0x557402b6c050 2024-03-11T19:13:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.617+0000 7f12d9ffb640 10 client.25840 handle_command_reply: tid=0 2024-03-11T19:13:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.617+0000 7f12fb9a6740 1 client.25840 shutdown 2024-03-11T19:13:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.617+0000 7f12fb9a6740 1 -- 172.21.15.29:0/2763931980 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b56270 msgr2=0x557402b60450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.617+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/2763931980 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b56270 0x557402b60450 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f12e80857b0 tx=0x7f12e8010d80 comp rx=0 tx=0).stop 2024-03-11T19:13:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.617+0000 7f12fb9a6740 1 -- 172.21.15.29:0/2763931980 shutdown_connections 2024-03-11T19:13:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.617+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/2763931980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b5bdd0 0x557402b679d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.617+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/2763931980 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x557402b6c050 0x557402b6e4e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.617+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/2763931980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56c60 0x557402b60990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.617+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/2763931980 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b56270 0x557402b60450 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.617+0000 7f12fb9a6740 1 -- 172.21.15.29:0/2763931980 >> 172.21.15.29:0/2763931980 conn(0x557402b517e0 msgr2=0x557402b50b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.617+0000 7f12d9ffb640 0 client.25840 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:13:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.617+0000 7f12fb9a6740 1 -- 172.21.15.29:0/2763931980 shutdown_connections 2024-03-11T19:13:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.617+0000 7f12fb9a6740 1 -- 172.21.15.29:0/2763931980 wait complete. 2024-03-11T19:13:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.618+0000 7f12fb9a6740 20 client.25840 trim_cache size 0 max 16384 2024-03-11T19:13:19.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.638+0000 7f12fb9a6740 1 Processor -- start 2024-03-11T19:13:19.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.638+0000 7f12fb9a6740 1 -- start start 2024-03-11T19:13:19.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.638+0000 7f12fb9a6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56270 0x557402b608f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.638+0000 7f12fb9a6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b56c60 0x557402b67990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.638+0000 7f12fb9a6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b5bdd0 0x557402b55150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.638+0000 7f12fb9a6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557402a44130 con 0x557402b56270 2024-03-11T19:13:19.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.638+0000 7f12fb9a6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557402b43a40 con 0x557402b56c60 2024-03-11T19:13:19.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.638+0000 7f12fb9a6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557402b5e920 con 0x557402b5bdd0 2024-03-11T19:13:19.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.638+0000 7f12f259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56270 0x557402b608f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.639+0000 7f12f259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56270 0x557402b608f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54832/0 (socket says 172.21.15.29:54832) 2024-03-11T19:13:19.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.639+0000 7f12f259c640 1 -- 172.21.15.29:0/3687235687 learned_addr learned my addr 172.21.15.29:0/3687235687 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:19.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.639+0000 7f12f1d9b640 1 --2- 172.21.15.29:0/3687235687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b56c60 0x557402b67990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.639+0000 7f12f2d9d640 1 --2- 172.21.15.29:0/3687235687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b5bdd0 0x557402b55150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.639+0000 7f12f1d9b640 1 -- 172.21.15.29:0/3687235687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b5bdd0 msgr2=0x557402b55150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.639+0000 7f12f1d9b640 1 --2- 172.21.15.29:0/3687235687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b5bdd0 0x557402b55150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.639+0000 7f12f1d9b640 1 -- 172.21.15.29:0/3687235687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56270 msgr2=0x557402b608f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.639+0000 7f12f1d9b640 1 --2- 172.21.15.29:0/3687235687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56270 0x557402b608f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.639+0000 7f12f1d9b640 1 -- 172.21.15.29:0/3687235687 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557402a59c20 con 0x557402b56c60 2024-03-11T19:13:19.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.639+0000 7f12f259c640 1 --2- 172.21.15.29:0/3687235687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56270 0x557402b608f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:19.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.639+0000 7f12f1d9b640 1 --2- 172.21.15.29:0/3687235687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b56c60 0x557402b67990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x557402b61930 tx=0x7f12e4013a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cf6d1e20cc0a6df server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.640+0000 7f12dbfff640 1 -- 172.21.15.29:0/3687235687 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e4017040 con 0x557402b56c60 2024-03-11T19:13:19.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.640+0000 7f12dbfff640 1 -- 172.21.15.29:0/3687235687 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f12e4020070 con 0x557402b56c60 2024-03-11T19:13:19.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.640+0000 7f12dbfff640 1 -- 172.21.15.29:0/3687235687 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e4005550 con 0x557402b56c60 2024-03-11T19:13:19.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.641+0000 7f12fb9a6740 1 -- 172.21.15.29:0/3687235687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b56c60 msgr2=0x557402b67990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.641+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/3687235687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b56c60 0x557402b67990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x557402b61930 tx=0x7f12e4013a70 comp rx=0 tx=0).stop 2024-03-11T19:13:19.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.641+0000 7f12fb9a6740 1 -- 172.21.15.29:0/3687235687 shutdown_connections 2024-03-11T19:13:19.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.641+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/3687235687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b5bdd0 0x557402b55150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.641+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/3687235687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b56c60 0x557402b67990 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.641+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/3687235687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56270 0x557402b608f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.641+0000 7f12fb9a6740 1 -- 172.21.15.29:0/3687235687 >> 172.21.15.29:0/3687235687 conn(0x557402b517e0 msgr2=0x557402b54a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.641+0000 7f12fb9a6740 1 -- 172.21.15.29:0/3687235687 shutdown_connections 2024-03-11T19:13:19.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.641+0000 7f12fb9a6740 1 -- 172.21.15.29:0/3687235687 wait complete. 2024-03-11T19:13:19.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.641+0000 7f12fb9a6740 1 -- start start 2024-03-11T19:13:19.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.641+0000 7f12fb9a6740 1 Processor -- start 2024-03-11T19:13:19.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.642+0000 7f12fb9a6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b56270 0x557402b66340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.642+0000 7f12fb9a6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56c60 0x557402b66880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.642+0000 7f12fb9a6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b5bdd0 0x557402b66dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.642+0000 7f12fb9a6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557402bc2670 con 0x557402b56c60 2024-03-11T19:13:19.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.642+0000 7f12fb9a6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557402bc27e0 con 0x557402b56270 2024-03-11T19:13:19.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.642+0000 7f12fb9a6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557402bc29a0 con 0x557402b5bdd0 2024-03-11T19:13:19.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.642+0000 7f12fb9a6740 10 client.0 _ll_register_callbacks cb 0x7f12e8010e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:19.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.642+0000 7f12f259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b56270 0x557402b66340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.642+0000 7f12f1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56c60 0x557402b66880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.642+0000 7f12f2d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b5bdd0 0x557402b66dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.642+0000 7f12f1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56c60 0x557402b66880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54848/0 (socket says 172.21.15.29:54848) 2024-03-11T19:13:19.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.642+0000 7f12f2d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b5bdd0 0x557402b66dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:43194/0 (socket says 172.21.15.29:43194) 2024-03-11T19:13:19.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.642+0000 7f12f1d9b640 1 -- 172.21.15.29:0/466853981 learned_addr learned my addr 172.21.15.29:0/466853981 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:19.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.642+0000 7f12f2d9d640 1 -- 172.21.15.29:0/466853981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b56270 msgr2=0x557402b66340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.642+0000 7f12f2d9d640 1 --2- 172.21.15.29:0/466853981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b56270 0x557402b66340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.642+0000 7f12f2d9d640 1 -- 172.21.15.29:0/466853981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56c60 msgr2=0x557402b66880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.643+0000 7f12f2d9d640 1 --2- 172.21.15.29:0/466853981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56c60 0x557402b66880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.643+0000 7f12f2d9d640 1 -- 172.21.15.29:0/466853981 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557402abd6a0 con 0x557402b5bdd0 2024-03-11T19:13:19.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.643+0000 7f12f1d9b640 1 --2- 172.21.15.29:0/466853981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56c60 0x557402b66880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:19.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.643+0000 7f12f2d9d640 1 --2- 172.21.15.29:0/466853981 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b5bdd0 0x557402b66dc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f12ec02c640 tx=0x7f12ec02c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec40b68de8233385 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.643+0000 7f12d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:13:19.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.643+0000 7f12d9ffb640 1 -- 172.21.15.29:0/466853981 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec046040 con 0x557402b5bdd0 2024-03-11T19:13:19.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.643+0000 7f12d9ffb640 1 -- 172.21.15.29:0/466853981 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f12ec003b70 con 0x557402b5bdd0 2024-03-11T19:13:19.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.643+0000 7f12fb9a6740 1 -- 172.21.15.29:0/466853981 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557402b5bb40 con 0x557402b5bdd0 2024-03-11T19:13:19.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.644+0000 7f12d9ffb640 1 -- 172.21.15.29:0/466853981 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12ec003d30 con 0x557402b5bdd0 2024-03-11T19:13:19.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.644+0000 7f12fb9a6740 10 client.25846 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:19.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.644+0000 7f12fb9a6740 1 -- 172.21.15.29:0/466853981 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7f12e4012050 con 0x557402b5bdd0 2024-03-11T19:13:19.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.644+0000 7f12d9ffb640 1 -- 172.21.15.29:0/466853981 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x557402b5bb40 con 0x557402b5bdd0 2024-03-11T19:13:19.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.645+0000 7f12d9ffb640 1 -- 172.21.15.29:0/466853981 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f12ec04c040 con 0x557402b5bdd0 2024-03-11T19:13:19.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.645+0000 7f12fb9a6740 10 client.25846 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:19.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.645+0000 7f12fb9a6740 10 client.25846 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:13:19.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.645+0000 7f12fb9a6740 20 client.25846 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:19.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.645+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/466853981 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x557402bc4430 0x557402bc6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.645+0000 7f12fb9a6740 4 client.25846 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:13:19.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.645+0000 7f12fb9a6740 1 -- 172.21.15.29:0/466853981 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f12e809e740 con 0x557402bc4430 2024-03-11T19:13:19.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.645+0000 7f12f259c640 1 --2- 172.21.15.29:0/466853981 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x557402bc4430 0x557402bc6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.646+0000 7f12f259c640 1 --2- 172.21.15.29:0/466853981 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x557402bc4430 0x557402bc6850 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=94f605d7741d5b22 server_cookie=7d2cc71718d45bf0 in_seq=0 out_seq=0 2024-03-11T19:13:19.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.646+0000 7f12d9ffb640 10 client.25846 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:13:19.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.646+0000 7f12d9ffb640 1 -- 172.21.15.29:0/466853981 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f12e809e740 con 0x557402bc4430 2024-03-11T19:13:19.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.646+0000 7f12d9ffb640 10 client.25846 handle_command_reply: tid=0 2024-03-11T19:13:19.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.646+0000 7f12fb9a6740 1 client.25846 shutdown 2024-03-11T19:13:19.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.647+0000 7f12fb9a6740 1 -- 172.21.15.29:0/466853981 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b5bdd0 msgr2=0x557402b66dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.647+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/466853981 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b5bdd0 0x557402b66dc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f12ec02c640 tx=0x7f12ec02c100 comp rx=0 tx=0).stop 2024-03-11T19:13:19.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.647+0000 7f12fb9a6740 1 -- 172.21.15.29:0/466853981 shutdown_connections 2024-03-11T19:13:19.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.647+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/466853981 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557402b5bdd0 0x557402b66dc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.647+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/466853981 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x557402bc4430 0x557402bc6850 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.647+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/466853981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557402b56c60 0x557402b66880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.647+0000 7f12fb9a6740 1 --2- 172.21.15.29:0/466853981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557402b56270 0x557402b66340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.647+0000 7f12d9ffb640 0 client.25846 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:13:19.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.647+0000 7f12fb9a6740 1 -- 172.21.15.29:0/466853981 >> 172.21.15.29:0/466853981 conn(0x557402b517e0 msgr2=0x557402b552d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.647+0000 7f12fb9a6740 1 -- 172.21.15.29:0/466853981 shutdown_connections 2024-03-11T19:13:19.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.647+0000 7f12fb9a6740 1 -- 172.21.15.29:0/466853981 wait complete. 2024-03-11T19:13:19.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.648+0000 7f12fb9a6740 20 client.25846 trim_cache size 0 max 16384 2024-03-11T19:13:19.654 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:13:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.649+0000 7f12f8cf4640 1 -- 172.21.15.29:0/3960570086 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f12dc0a1b80 msgr2=0x7f12dc0a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.649+0000 7f12f8cf4640 1 --2- 172.21.15.29:0/3960570086 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f12dc0a1b80 0x7f12dc0a4040 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f12ec005e30 tx=0x7f12ec032380 comp rx=0 tx=0).stop 2024-03-11T19:13:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.649+0000 7f12f8cf4640 1 -- 172.21.15.29:0/3960570086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12f40ff9f0 msgr2=0x7f12f419d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.649+0000 7f12f8cf4640 1 --2- 172.21.15.29:0/3960570086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12f40ff9f0 0x7f12f419d730 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f12e800e9f0 tx=0x7f12e800eeb0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.649+0000 7f12f8cf4640 1 -- 172.21.15.29:0/3960570086 shutdown_connections 2024-03-11T19:13:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.649+0000 7f12f8cf4640 1 --2- 172.21.15.29:0/3960570086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12f4104d10 0x7f12f41a4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.649+0000 7f12f8cf4640 1 --2- 172.21.15.29:0/3960570086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12f4102360 0x7f12f419dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.649+0000 7f12f8cf4640 1 --2- 172.21.15.29:0/3960570086 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f12dc0a1b80 0x7f12dc0a4040 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.649+0000 7f12f8cf4640 1 --2- 172.21.15.29:0/3960570086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12f40ff9f0 0x7f12f419d730 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.649+0000 7f12f8cf4640 1 -- 172.21.15.29:0/3960570086 >> 172.21.15.29:0/3960570086 conn(0x7f12f40fb610 msgr2=0x7f12f4100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.649+0000 7f12f8cf4640 1 -- 172.21.15.29:0/3960570086 shutdown_connections 2024-03-11T19:13:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.649+0000 7f12f8cf4640 1 -- 172.21.15.29:0/3960570086 wait complete. 2024-03-11T19:13:19.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:19 smithi029 ceph-mon[32439]: pgmap v456: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-11T19:13:19.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:19 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/4230327861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:13:19.671 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:13:19.671 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:13:19.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:19 smithi190 ceph-mon[36292]: pgmap v456: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-11T19:13:19.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:19 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/4230327861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:13:19.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.829+0000 7f16dd2ce640 1 -- 172.21.15.29:0/801806236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8100090 msgr2=0x7f16d81004b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.829+0000 7f16dd2ce640 1 --2- 172.21.15.29:0/801806236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8100090 0x7f16d81004b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f16c8009900 tx=0x7f16c802bc30 comp rx=0 tx=0).stop 2024-03-11T19:13:19.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.829+0000 7f16dd2ce640 1 -- 172.21.15.29:0/801806236 shutdown_connections 2024-03-11T19:13:19.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.829+0000 7f16dd2ce640 1 --2- 172.21.15.29:0/801806236 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16d8101c40 0x7f16d81084d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.829+0000 7f16dd2ce640 1 --2- 172.21.15.29:0/801806236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16d8101280 0x7f16d8101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.829+0000 7f16dd2ce640 1 --2- 172.21.15.29:0/801806236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8100090 0x7f16d81004b0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.829+0000 7f16dd2ce640 1 -- 172.21.15.29:0/801806236 >> 172.21.15.29:0/801806236 conn(0x7f16d80fb620 msgr2=0x7f16d80fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.829+0000 7f16dd2ce640 1 -- 172.21.15.29:0/801806236 shutdown_connections 2024-03-11T19:13:19.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.829+0000 7f16dd2ce640 1 -- 172.21.15.29:0/801806236 wait complete. 2024-03-11T19:13:19.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16dd2ce640 1 Processor -- start 2024-03-11T19:13:19.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16dd2ce640 1 -- start start 2024-03-11T19:13:19.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16dd2ce640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16d8100090 0x7f16d806a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16dd2ce640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16d8101280 0x7f16d819e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16dd2ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8101c40 0x7f16d81a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16dd2ce640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d81a0ed0 con 0x7f16d8101c40 2024-03-11T19:13:19.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16dd2ce640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d81a1040 con 0x7f16d8100090 2024-03-11T19:13:19.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16dd2ce640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d81a1200 con 0x7f16d8101280 2024-03-11T19:13:19.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16d77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8101c40 0x7f16d81a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16d77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8101c40 0x7f16d81a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54860/0 (socket says 172.21.15.29:54860) 2024-03-11T19:13:19.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16d77fe640 1 -- 172.21.15.29:0/1577785508 learned_addr learned my addr 172.21.15.29:0/1577785508 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:19.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16d6ffd640 1 --2- 172.21.15.29:0/1577785508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16d8100090 0x7f16d806a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16cffff640 1 --2- 172.21.15.29:0/1577785508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16d8101280 0x7f16d819e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16d77fe640 1 -- 172.21.15.29:0/1577785508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16d8101280 msgr2=0x7f16d819e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16d77fe640 1 --2- 172.21.15.29:0/1577785508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16d8101280 0x7f16d819e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16d77fe640 1 -- 172.21.15.29:0/1577785508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16d8100090 msgr2=0x7f16d806a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16d77fe640 1 --2- 172.21.15.29:0/1577785508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16d8100090 0x7f16d806a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16d77fe640 1 -- 172.21.15.29:0/1577785508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c8009560 con 0x7f16d8101c40 2024-03-11T19:13:19.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16d77fe640 1 --2- 172.21.15.29:0/1577785508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8101c40 0x7f16d81a0810 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f16d000eb10 tx=0x7f16d000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16d4ff9640 1 -- 172.21.15.29:0/1577785508 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d0004b20 con 0x7f16d8101c40 2024-03-11T19:13:19.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.830+0000 7f16d4ff9640 1 -- 172.21.15.29:0/1577785508 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f16d0004c80 con 0x7f16d8101c40 2024-03-11T19:13:19.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.831+0000 7f16dd2ce640 1 -- 172.21.15.29:0/1577785508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16d81a1480 con 0x7f16d8101c40 2024-03-11T19:13:19.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.831+0000 7f16d4ff9640 1 -- 172.21.15.29:0/1577785508 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d0010690 con 0x7f16d8101c40 2024-03-11T19:13:19.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.831+0000 7f16dd2ce640 1 -- 172.21.15.29:0/1577785508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16d81a1940 con 0x7f16d8101c40 2024-03-11T19:13:19.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.833+0000 7f16d4ff9640 1 -- 172.21.15.29:0/1577785508 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f16d0005020 con 0x7f16d8101c40 2024-03-11T19:13:19.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.833+0000 7f16dff78740 1 Processor -- start 2024-03-11T19:13:19.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.833+0000 7f16dff78740 1 -- start start 2024-03-11T19:13:19.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.833+0000 7f16dff78740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7c8270 0x557bfd7c8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.833+0000 7f16dff78740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 0x557bfd7cd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.833+0000 7f16dff78740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7cddd0 0x557bfd7d0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.833+0000 7f16dff78740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bfd784960 con 0x557bfd7c8c60 2024-03-11T19:13:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.833+0000 7f16dff78740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bfd738310 con 0x557bfd7cddd0 2024-03-11T19:13:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.833+0000 7f16dff78740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bfd735840 con 0x557bfd7c8270 2024-03-11T19:13:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.833+0000 7f16d6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 0x557bfd7cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.833+0000 7f16cffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7c8270 0x557bfd7c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.833+0000 7f16d6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 0x557bfd7cd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54872/0 (socket says 172.21.15.29:54872) 2024-03-11T19:13:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.833+0000 7f16d6ffd640 1 -- 172.21.15.29:0/3907606007 learned_addr learned my addr 172.21.15.29:0/3907606007 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.834+0000 7f16d4ff9640 1 --2- 172.21.15.29:0/1577785508 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f16c40a1a60 0x7f16c40a3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.834+0000 7f16cffff640 1 --2- 172.21.15.29:0/3907606007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7cddd0 0x557bfd7d0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.834+0000 7f16d77fe640 1 --2- 172.21.15.29:0/1577785508 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f16c40a1a60 0x7f16c40a3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.834+0000 7f16cffff640 1 -- 172.21.15.29:0/3907606007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7cddd0 msgr2=0x557bfd7d0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.834+0000 7f16cffff640 1 --2- 172.21.15.29:0/3907606007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7cddd0 0x557bfd7d0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.834+0000 7f16cffff640 1 -- 172.21.15.29:0/3907606007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 msgr2=0x557bfd7cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.834+0000 7f16cffff640 1 --2- 172.21.15.29:0/3907606007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 0x557bfd7cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.834+0000 7f16d4ff9640 1 -- 172.21.15.29:0/1577785508 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f16d0014070 con 0x7f16d8101c40 2024-03-11T19:13:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.834+0000 7f16cffff640 1 -- 172.21.15.29:0/3907606007 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557bfd6f3060 con 0x557bfd7c8270 2024-03-11T19:13:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.834+0000 7f16d6ffd640 1 --2- 172.21.15.29:0/3907606007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 0x557bfd7cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.834+0000 7f16cffff640 1 --2- 172.21.15.29:0/3907606007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7c8270 0x557bfd7c8690 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f16c0014900 tx=0x7f16c000c430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa7d7059f8c34416 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.834+0000 7f16d77fe640 1 --2- 172.21.15.29:0/1577785508 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f16c40a1a60 0x7f16c40a3f20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f16d00a2c00 tx=0x7f16d006dc90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.834+0000 7f16cd7fa640 1 -- 172.21.15.29:0/3907606007 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c00142d0 con 0x557bfd7c8270 2024-03-11T19:13:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.834+0000 7f16cd7fa640 1 -- 172.21.15.29:0/3907606007 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f16c001a030 con 0x557bfd7c8270 2024-03-11T19:13:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.834+0000 7f16cd7fa640 1 -- 172.21.15.29:0/3907606007 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c0004c70 con 0x557bfd7c8270 2024-03-11T19:13:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.835+0000 7f16dff78740 1 -- 172.21.15.29:0/3907606007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7c8270 msgr2=0x557bfd7c8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.835+0000 7f16dff78740 1 --2- 172.21.15.29:0/3907606007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7c8270 0x557bfd7c8690 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f16c0014900 tx=0x7f16c000c430 comp rx=0 tx=0).stop 2024-03-11T19:13:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.835+0000 7f16dff78740 1 -- 172.21.15.29:0/3907606007 shutdown_connections 2024-03-11T19:13:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.835+0000 7f16dff78740 1 --2- 172.21.15.29:0/3907606007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7cddd0 0x557bfd7d0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.835+0000 7f16dff78740 1 --2- 172.21.15.29:0/3907606007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 0x557bfd7cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.835+0000 7f16dff78740 1 --2- 172.21.15.29:0/3907606007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7c8270 0x557bfd7c8690 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.835+0000 7f16dff78740 1 -- 172.21.15.29:0/3907606007 >> 172.21.15.29:0/3907606007 conn(0x557bfd7c37e0 msgr2=0x557bfd7c3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.835+0000 7f16dff78740 1 -- 172.21.15.29:0/3907606007 shutdown_connections 2024-03-11T19:13:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.835+0000 7f16dff78740 1 -- 172.21.15.29:0/3907606007 wait complete. 2024-03-11T19:13:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.835+0000 7f16dff78740 1 -- start start 2024-03-11T19:13:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.835+0000 7f16dff78740 1 Processor -- start 2024-03-11T19:13:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16dff78740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7c8270 0x557bfd7d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16dff78740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7c8c60 0x557bfd7d2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16dff78740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7cddd0 0x557bfd7d99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16dff78740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bfd731020 con 0x557bfd7cddd0 2024-03-11T19:13:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16dff78740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bfd404430 con 0x557bfd7c8270 2024-03-11T19:13:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16dff78740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bfd3c0e00 con 0x557bfd7c8c60 2024-03-11T19:13:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7cddd0 0x557bfd7d99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7cddd0 0x557bfd7d99d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54880/0 (socket says 172.21.15.29:54880) 2024-03-11T19:13:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16cffff640 1 -- 172.21.15.29:0/1328913822 learned_addr learned my addr 172.21.15.29:0/1328913822 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16dff78740 10 client.0 _ll_register_callbacks cb 0x557bfd6d8cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16cffff640 1 --2- 172.21.15.29:0/1328913822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7c8270 0x557bfd7d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16d6ffd640 1 --2- 172.21.15.29:0/1328913822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7c8c60 0x557bfd7d2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16cffff640 1 -- 172.21.15.29:0/1328913822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7c8c60 msgr2=0x557bfd7d2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16cffff640 1 --2- 172.21.15.29:0/1328913822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7c8c60 0x557bfd7d2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16cffff640 1 -- 172.21.15.29:0/1328913822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7c8270 msgr2=0x557bfd7d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16cffff640 1 --2- 172.21.15.29:0/1328913822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7c8270 0x557bfd7d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16cffff640 1 -- 172.21.15.29:0/1328913822 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c00145e0 con 0x557bfd7cddd0 2024-03-11T19:13:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16cffff640 1 --2- 172.21.15.29:0/1328913822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7cddd0 0x557bfd7d99d0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f16c00057c0 tx=0x7f16c0005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=977fdb2bb81f45aa server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16be7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:13:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16be7fc640 1 -- 172.21.15.29:0/1328913822 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c00051b0 con 0x557bfd7cddd0 2024-03-11T19:13:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16be7fc640 1 -- 172.21.15.29:0/1328913822 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f16c0005310 con 0x557bfd7cddd0 2024-03-11T19:13:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.836+0000 7f16dff78740 1 -- 172.21.15.29:0/1328913822 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557bfd4307e0 con 0x557bfd7cddd0 2024-03-11T19:13:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.837+0000 7f16be7fc640 1 -- 172.21.15.29:0/1328913822 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16c000f760 con 0x557bfd7cddd0 2024-03-11T19:13:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.837+0000 7f16dff78740 10 client.16296 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.837+0000 7f16dff78740 1 -- 172.21.15.29:0/1328913822 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x557bfd72f6a0 con 0x557bfd7cddd0 2024-03-11T19:13:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.837+0000 7f16be7fc640 1 -- 172.21.15.29:0/1328913822 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x557bfd4307e0 con 0x557bfd7cddd0 2024-03-11T19:13:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.837+0000 7f16be7fc640 1 -- 172.21.15.29:0/1328913822 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f16c0020570 con 0x557bfd7cddd0 2024-03-11T19:13:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.837+0000 7f16dff78740 10 client.16296 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.837+0000 7f16dff78740 10 client.16296 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:13:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.837+0000 7f16dff78740 20 client.16296 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.837+0000 7f16dff78740 1 --2- 172.21.15.29:0/1328913822 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x557bfd7de500 0x557bfd7e08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.837+0000 7f16dff78740 4 client.16296 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:13:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.837+0000 7f16dff78740 1 -- 172.21.15.29:0/1328913822 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557bfd46d720 con 0x557bfd7de500 2024-03-11T19:13:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.837+0000 7f16d6ffd640 1 --2- 172.21.15.29:0/1328913822 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x557bfd7de500 0x557bfd7e08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.838+0000 7f16d6ffd640 1 --2- 172.21.15.29:0/1328913822 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x557bfd7de500 0x557bfd7e08f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=88c02c365e2f0312 server_cookie=fe9a35f482338d78 in_seq=0 out_seq=0 2024-03-11T19:13:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.838+0000 7f16be7fc640 10 client.16296 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:13:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.838+0000 7f16be7fc640 1 -- 172.21.15.29:0/1328913822 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557bfd46d720 con 0x557bfd7de500 2024-03-11T19:13:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.838+0000 7f16be7fc640 10 client.16296 handle_command_reply: tid=0 2024-03-11T19:13:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.838+0000 7f16dff78740 1 client.16296 shutdown 2024-03-11T19:13:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.838+0000 7f16dff78740 1 -- 172.21.15.29:0/1328913822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7cddd0 msgr2=0x557bfd7d99d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.838+0000 7f16dff78740 1 --2- 172.21.15.29:0/1328913822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7cddd0 0x557bfd7d99d0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f16c00057c0 tx=0x7f16c0005f30 comp rx=0 tx=0).stop 2024-03-11T19:13:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.838+0000 7f16dff78740 1 -- 172.21.15.29:0/1328913822 shutdown_connections 2024-03-11T19:13:19.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.838+0000 7f16dff78740 1 --2- 172.21.15.29:0/1328913822 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x557bfd7de500 0x557bfd7e08f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.838+0000 7f16dff78740 1 --2- 172.21.15.29:0/1328913822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7cddd0 0x557bfd7d99d0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.838+0000 7f16be7fc640 0 client.16296 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:13:19.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.838+0000 7f16dff78740 1 --2- 172.21.15.29:0/1328913822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7c8c60 0x557bfd7d2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.838+0000 7f16dff78740 1 --2- 172.21.15.29:0/1328913822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7c8270 0x557bfd7d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.838+0000 7f16dff78740 1 -- 172.21.15.29:0/1328913822 >> 172.21.15.29:0/1328913822 conn(0x557bfd7c37e0 msgr2=0x557bfd7c2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.839+0000 7f16dff78740 1 -- 172.21.15.29:0/1328913822 shutdown_connections 2024-03-11T19:13:19.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.839+0000 7f16dff78740 1 -- 172.21.15.29:0/1328913822 wait complete. 2024-03-11T19:13:19.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.839+0000 7f16dff78740 20 client.16296 trim_cache size 0 max 16384 2024-03-11T19:13:19.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:19 smithi170 ceph-mon[35690]: pgmap v456: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-11T19:13:19.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:19 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/4230327861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:13:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.850+0000 7f16dff78740 1 Processor -- start 2024-03-11T19:13:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.850+0000 7f16dff78740 1 -- start start 2024-03-11T19:13:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.850+0000 7f16dff78740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7c8270 0x557bfd7ca120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.850+0000 7f16dff78740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 0x557bfd7d9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.850+0000 7f16dff78740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7cddd0 0x557bfd7ca660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.851+0000 7f16dff78740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bfd6b6130 con 0x557bfd7c8c60 2024-03-11T19:13:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.851+0000 7f16dff78740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bfd46d750 con 0x557bfd7cddd0 2024-03-11T19:13:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.851+0000 7f16dff78740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bfd7b5a40 con 0x557bfd7c8270 2024-03-11T19:13:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.851+0000 7f16d6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 0x557bfd7d9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.851+0000 7f16d6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 0x557bfd7d9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54884/0 (socket says 172.21.15.29:54884) 2024-03-11T19:13:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.851+0000 7f16d6ffd640 1 -- 172.21.15.29:0/2952633412 learned_addr learned my addr 172.21.15.29:0/2952633412 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.851+0000 7f16cffff640 1 --2- 172.21.15.29:0/2952633412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7c8270 0x557bfd7ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.851+0000 7f16d6ffd640 1 -- 172.21.15.29:0/2952633412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7c8270 msgr2=0x557bfd7ca120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.851+0000 7f16d6ffd640 1 --2- 172.21.15.29:0/2952633412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7c8270 0x557bfd7ca120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.851+0000 7f16d6ffd640 1 -- 172.21.15.29:0/2952633412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7cddd0 msgr2=0x557bfd7ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.851+0000 7f16d6ffd640 1 --2- 172.21.15.29:0/2952633412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7cddd0 0x557bfd7ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.851+0000 7f16d6ffd640 1 -- 172.21.15.29:0/2952633412 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557bfd6cbc20 con 0x557bfd7c8c60 2024-03-11T19:13:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.851+0000 7f16d6ffd640 1 --2- 172.21.15.29:0/2952633412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 0x557bfd7d9990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f16c8036040 tx=0x7f16c8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad78195c99b4695d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.851+0000 7f16ccff9640 1 -- 172.21.15.29:0/2952633412 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c802cbf0 con 0x557bfd7c8c60 2024-03-11T19:13:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.851+0000 7f16ccff9640 1 -- 172.21.15.29:0/2952633412 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f16c802de90 con 0x557bfd7c8c60 2024-03-11T19:13:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.851+0000 7f16ccff9640 1 -- 172.21.15.29:0/2952633412 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c8004500 con 0x557bfd7c8c60 2024-03-11T19:13:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.852+0000 7f16dff78740 1 -- 172.21.15.29:0/2952633412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 msgr2=0x557bfd7d9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.852+0000 7f16dff78740 1 --2- 172.21.15.29:0/2952633412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 0x557bfd7d9990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f16c8036040 tx=0x7f16c8005fd0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.852+0000 7f16dff78740 1 -- 172.21.15.29:0/2952633412 shutdown_connections 2024-03-11T19:13:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.852+0000 7f16dff78740 1 --2- 172.21.15.29:0/2952633412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7cddd0 0x557bfd7ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.852+0000 7f16dff78740 1 --2- 172.21.15.29:0/2952633412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 0x557bfd7d9990 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.852+0000 7f16dff78740 1 --2- 172.21.15.29:0/2952633412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7c8270 0x557bfd7ca120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.852+0000 7f16dff78740 1 -- 172.21.15.29:0/2952633412 >> 172.21.15.29:0/2952633412 conn(0x557bfd7c37e0 msgr2=0x557bfd7d2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.852+0000 7f16dff78740 1 -- 172.21.15.29:0/2952633412 shutdown_connections 2024-03-11T19:13:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.852+0000 7f16dff78740 1 -- 172.21.15.29:0/2952633412 wait complete. 2024-03-11T19:13:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.852+0000 7f16dff78740 1 -- start start 2024-03-11T19:13:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.852+0000 7f16dff78740 1 Processor -- start 2024-03-11T19:13:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.852+0000 7f16dff78740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7c8270 0x557bfd7d83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.852+0000 7f16dff78740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 0x557bfd7d8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.852+0000 7f16dff78740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7cddd0 0x557bfd7d8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.852+0000 7f16dff78740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bfd834b00 con 0x557bfd7c8c60 2024-03-11T19:13:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.852+0000 7f16dff78740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bfd834c70 con 0x557bfd7c8270 2024-03-11T19:13:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.852+0000 7f16dff78740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bfd834e30 con 0x557bfd7cddd0 2024-03-11T19:13:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.852+0000 7f16dff78740 10 client.0 _ll_register_callbacks cb 0x7f16c000c9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16d6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 0x557bfd7d8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16d6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 0x557bfd7d8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54896/0 (socket says 172.21.15.29:54896) 2024-03-11T19:13:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16d6ffd640 1 -- 172.21.15.29:0/1091770241 learned_addr learned my addr 172.21.15.29:0/1091770241 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16cffff640 1 --2- 172.21.15.29:0/1091770241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7cddd0 0x557bfd7d8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16cffff640 1 --2- 172.21.15.29:0/1091770241 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7c8270 0x557bfd7d83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16d6ffd640 1 -- 172.21.15.29:0/1091770241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7cddd0 msgr2=0x557bfd7d8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16d6ffd640 1 --2- 172.21.15.29:0/1091770241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7cddd0 0x557bfd7d8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16d6ffd640 1 -- 172.21.15.29:0/1091770241 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7c8270 msgr2=0x557bfd7d83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16d6ffd640 1 --2- 172.21.15.29:0/1091770241 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7c8270 0x557bfd7d83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16d6ffd640 1 -- 172.21.15.29:0/1091770241 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c8030070 con 0x557bfd7c8c60 2024-03-11T19:13:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16cffff640 1 --2- 172.21.15.29:0/1091770241 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7c8270 0x557bfd7d83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:13:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16d6ffd640 1 --2- 172.21.15.29:0/1091770241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 0x557bfd7d8930 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f16c8009850 tx=0x7f16c802cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ae90112c2ef381 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16be7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:13:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16be7fc640 1 -- 172.21.15.29:0/1091770241 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c8059070 con 0x557bfd7c8c60 2024-03-11T19:13:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16be7fc640 1 -- 172.21.15.29:0/1091770241 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f16c8045040 con 0x557bfd7c8c60 2024-03-11T19:13:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16dff78740 1 -- 172.21.15.29:0/1091770241 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f16c001a030 con 0x557bfd7c8c60 2024-03-11T19:13:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16be7fc640 1 -- 172.21.15.29:0/1091770241 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16c8048da0 con 0x557bfd7c8c60 2024-03-11T19:13:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16dff78740 10 client.16302 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16dff78740 1 -- 172.21.15.29:0/1091770241 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x557bfd6f18c0 con 0x557bfd7c8c60 2024-03-11T19:13:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.853+0000 7f16be7fc640 1 -- 172.21.15.29:0/1091770241 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f16c8064020 con 0x557bfd7c8c60 2024-03-11T19:13:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16be7fc640 1 -- 172.21.15.29:0/1091770241 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f16c805c020 con 0x557bfd7c8c60 2024-03-11T19:13:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16dff78740 10 client.16302 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16dff78740 10 client.16302 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:13:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16dff78740 20 client.16302 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16dff78740 1 --2- 172.21.15.29:0/1091770241 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x557bfd7de500 0x557bfd838690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16dff78740 4 client.16302 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:13:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16dff78740 1 -- 172.21.15.29:0/1091770241 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557bfd4bc890 con 0x557bfd7de500 2024-03-11T19:13:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16cffff640 1 --2- 172.21.15.29:0/1091770241 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x557bfd7de500 0x557bfd838690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16be7fc640 10 client.16302 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:13:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16cffff640 1 --2- 172.21.15.29:0/1091770241 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x557bfd7de500 0x557bfd838690 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=85bd08ade4179f60 server_cookie=b63efacee891c58 in_seq=0 out_seq=0 2024-03-11T19:13:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16be7fc640 1 -- 172.21.15.29:0/1091770241 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557bfd4bc890 con 0x557bfd7de500 2024-03-11T19:13:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16be7fc640 10 client.16302 handle_command_reply: tid=0 2024-03-11T19:13:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16dff78740 1 client.16302 shutdown 2024-03-11T19:13:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16dff78740 1 -- 172.21.15.29:0/1091770241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 msgr2=0x557bfd7d8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16dff78740 1 --2- 172.21.15.29:0/1091770241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 0x557bfd7d8930 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f16c8009850 tx=0x7f16c802cc80 comp rx=0 tx=0).stop 2024-03-11T19:13:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16dff78740 1 -- 172.21.15.29:0/1091770241 shutdown_connections 2024-03-11T19:13:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16dff78740 1 --2- 172.21.15.29:0/1091770241 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x557bfd7de500 0x557bfd838690 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16dff78740 1 --2- 172.21.15.29:0/1091770241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557bfd7cddd0 0x557bfd7d8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16dff78740 1 --2- 172.21.15.29:0/1091770241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bfd7c8c60 0x557bfd7d8930 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16be7fc640 0 client.16302 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:13:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16dff78740 1 --2- 172.21.15.29:0/1091770241 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557bfd7c8270 0x557bfd7d83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16dff78740 1 -- 172.21.15.29:0/1091770241 >> 172.21.15.29:0/1091770241 conn(0x557bfd7c37e0 msgr2=0x557bfd7d2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16dff78740 1 -- 172.21.15.29:0/1091770241 shutdown_connections 2024-03-11T19:13:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.854+0000 7f16dff78740 1 -- 172.21.15.29:0/1091770241 wait complete. 2024-03-11T19:13:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.855+0000 7f16dff78740 20 client.16302 trim_cache size 0 max 16384 2024-03-11T19:13:19.866 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:13:19.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.855+0000 7f16dd2ce640 1 -- 172.21.15.29:0/1577785508 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f16c40a1a60 msgr2=0x7f16c40a3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.855+0000 7f16dd2ce640 1 --2- 172.21.15.29:0/1577785508 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f16c40a1a60 0x7f16c40a3f20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f16d00a2c00 tx=0x7f16d006dc90 comp rx=0 tx=0).stop 2024-03-11T19:13:19.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.855+0000 7f16dd2ce640 1 -- 172.21.15.29:0/1577785508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8101c40 msgr2=0x7f16d81a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.855+0000 7f16dd2ce640 1 --2- 172.21.15.29:0/1577785508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8101c40 0x7f16d81a0810 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f16d000eb10 tx=0x7f16d000c2d0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.855+0000 7f16dd2ce640 1 -- 172.21.15.29:0/1577785508 shutdown_connections 2024-03-11T19:13:19.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.855+0000 7f16dd2ce640 1 --2- 172.21.15.29:0/1577785508 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f16c40a1a60 0x7f16c40a3f20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.855+0000 7f16dd2ce640 1 --2- 172.21.15.29:0/1577785508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16d8101c40 0x7f16d81a0810 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.855+0000 7f16dd2ce640 1 --2- 172.21.15.29:0/1577785508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16d8101280 0x7f16d819e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.855+0000 7f16dd2ce640 1 --2- 172.21.15.29:0/1577785508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16d8100090 0x7f16d806a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.855+0000 7f16dd2ce640 1 -- 172.21.15.29:0/1577785508 >> 172.21.15.29:0/1577785508 conn(0x7f16d80fb620 msgr2=0x7f16d81046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.855+0000 7f16dd2ce640 1 -- 172.21.15.29:0/1577785508 shutdown_connections 2024-03-11T19:13:19.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.855+0000 7f16dd2ce640 1 -- 172.21.15.29:0/1577785508 wait complete. 2024-03-11T19:13:19.870 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:13:19.870 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:13:19.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.970+0000 7feb10a4e640 1 -- 172.21.15.29:0/169953706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c101c20 msgr2=0x7feb0c104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.970+0000 7feb10a4e640 1 --2- 172.21.15.29:0/169953706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c101c20 0x7feb0c104010 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7feafc009900 tx=0x7feafc02bc30 comp rx=0 tx=0).stop 2024-03-11T19:13:19.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.970+0000 7feb10a4e640 1 -- 172.21.15.29:0/169953706 shutdown_connections 2024-03-11T19:13:19.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.970+0000 7feb10a4e640 1 --2- 172.21.15.29:0/169953706 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb0c106e80 0x7feb0c1092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.970+0000 7feb10a4e640 1 --2- 172.21.15.29:0/169953706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb0c104550 0x7feb0c106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.970+0000 7feb10a4e640 1 --2- 172.21.15.29:0/169953706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c101c20 0x7feb0c104010 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.970+0000 7feb10a4e640 1 -- 172.21.15.29:0/169953706 >> 172.21.15.29:0/169953706 conn(0x7feb0c0fb630 msgr2=0x7feb0c0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.970+0000 7feb10a4e640 1 -- 172.21.15.29:0/169953706 shutdown_connections 2024-03-11T19:13:19.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.970+0000 7feb10a4e640 1 -- 172.21.15.29:0/169953706 wait complete. 2024-03-11T19:13:19.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.971+0000 7feb10a4e640 1 Processor -- start 2024-03-11T19:13:19.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.971+0000 7feb10a4e640 1 -- start start 2024-03-11T19:13:19.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.971+0000 7feb10a4e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb0c101c20 0x7feb0c197190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.971+0000 7feb10a4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c104550 0x7feb0c1976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.971+0000 7feb10a4e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb0c106e80 0x7feb0c19e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.971+0000 7feb10a4e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c19ec80 con 0x7feb0c104550 2024-03-11T19:13:19.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.971+0000 7feb10a4e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c19edf0 con 0x7feb0c101c20 2024-03-11T19:13:19.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.971+0000 7feb10a4e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c19efb0 con 0x7feb0c106e80 2024-03-11T19:13:19.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.971+0000 7feb09d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c104550 0x7feb0c1976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.971+0000 7feb09d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c104550 0x7feb0c1976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54916/0 (socket says 172.21.15.29:54916) 2024-03-11T19:13:19.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.971+0000 7feb09d9b640 1 -- 172.21.15.29:0/2093450148 learned_addr learned my addr 172.21.15.29:0/2093450148 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:19.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.971+0000 7feb0ad9d640 1 --2- 172.21.15.29:0/2093450148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb0c106e80 0x7feb0c19e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.972+0000 7feb0a59c640 1 --2- 172.21.15.29:0/2093450148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb0c101c20 0x7feb0c197190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.972+0000 7feb09d9b640 1 -- 172.21.15.29:0/2093450148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb0c106e80 msgr2=0x7feb0c19e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.972+0000 7feb09d9b640 1 --2- 172.21.15.29:0/2093450148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb0c106e80 0x7feb0c19e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.972+0000 7feb09d9b640 1 -- 172.21.15.29:0/2093450148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb0c101c20 msgr2=0x7feb0c197190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.972+0000 7feb09d9b640 1 --2- 172.21.15.29:0/2093450148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb0c101c20 0x7feb0c197190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.972+0000 7feb09d9b640 1 -- 172.21.15.29:0/2093450148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feafc009560 con 0x7feb0c104550 2024-03-11T19:13:19.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.972+0000 7feb09d9b640 1 --2- 172.21.15.29:0/2093450148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c104550 0x7feb0c1976d0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7feb0000d740 tx=0x7feb0000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.972+0000 7feafb7fe640 1 -- 172.21.15.29:0/2093450148 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb00009c30 con 0x7feb0c104550 2024-03-11T19:13:19.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.972+0000 7feb10a4e640 1 -- 172.21.15.29:0/2093450148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb0c19f290 con 0x7feb0c104550 2024-03-11T19:13:19.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.972+0000 7feafb7fe640 1 -- 172.21.15.29:0/2093450148 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feb0000de50 con 0x7feb0c104550 2024-03-11T19:13:19.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.972+0000 7feafb7fe640 1 -- 172.21.15.29:0/2093450148 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb00004e40 con 0x7feb0c104550 2024-03-11T19:13:19.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.972+0000 7feb10a4e640 1 -- 172.21.15.29:0/2093450148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb0c19f750 con 0x7feb0c104550 2024-03-11T19:13:19.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.975+0000 7feafb7fe640 1 -- 172.21.15.29:0/2093450148 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7feb00004fa0 con 0x7feb0c104550 2024-03-11T19:13:19.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.975+0000 7feb1361e740 1 Processor -- start 2024-03-11T19:13:19.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.975+0000 7feb1361e740 1 -- start start 2024-03-11T19:13:19.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.975+0000 7feb1361e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25270 0x558498f25690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.975+0000 7feb1361e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f25c60 0x558498f2a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.975+0000 7feb1361e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f2add0 0x558498f2d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.975+0000 7feb1361e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558498ee1960 con 0x558498f25270 2024-03-11T19:13:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.975+0000 7feb1361e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558498e95310 con 0x558498f2add0 2024-03-11T19:13:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.975+0000 7feb1361e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558498e92840 con 0x558498f25c60 2024-03-11T19:13:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.975+0000 7feb0ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25270 0x558498f25690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.975+0000 7feafb7fe640 1 --2- 172.21.15.29:0/2093450148 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7feaf40a1a30 0x7feaf40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.975+0000 7feb0ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25270 0x558498f25690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54920/0 (socket says 172.21.15.29:54920) 2024-03-11T19:13:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.975+0000 7feb0ad9d640 1 -- 172.21.15.29:0/2129575867 learned_addr learned my addr 172.21.15.29:0/2129575867 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.976+0000 7feb0a59c640 1 --2- 172.21.15.29:0/2129575867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f25c60 0x558498f2a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.976+0000 7feb0ad9d640 1 --2- 172.21.15.29:0/2093450148 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7feaf40a1a30 0x7feaf40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.976+0000 7feafb7fe640 1 -- 172.21.15.29:0/2093450148 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7feb000ccd20 con 0x7feb0c104550 2024-03-11T19:13:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.976+0000 7feb09d9b640 1 --2- 172.21.15.29:0/2129575867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f2add0 0x558498f2d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.976+0000 7feb0ad9d640 1 -- 172.21.15.29:0/2129575867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f25c60 msgr2=0x558498f2a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.976+0000 7feb0ad9d640 1 --2- 172.21.15.29:0/2129575867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f25c60 0x558498f2a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.976+0000 7feb0ad9d640 1 -- 172.21.15.29:0/2129575867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f2add0 msgr2=0x558498f2d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.976+0000 7feb0ad9d640 1 --2- 172.21.15.29:0/2129575867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f2add0 0x558498f2d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.976+0000 7feb0ad9d640 1 -- 172.21.15.29:0/2129575867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558498e50060 con 0x558498f25270 2024-03-11T19:13:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.976+0000 7feb0ad9d640 1 --2- 172.21.15.29:0/2129575867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25270 0x558498f25690 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7feb04005c40 tx=0x7feb0400f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f846a4016a5bfdbd server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.976+0000 7feaf8ff9640 1 -- 172.21.15.29:0/2129575867 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb04003ec0 con 0x558498f25270 2024-03-11T19:13:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.976+0000 7feaf8ff9640 1 -- 172.21.15.29:0/2129575867 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feb0401e420 con 0x558498f25270 2024-03-11T19:13:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.976+0000 7feaf8ff9640 1 -- 172.21.15.29:0/2129575867 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb040263b0 con 0x558498f25270 2024-03-11T19:13:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.977+0000 7feb1361e740 1 -- 172.21.15.29:0/2129575867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25270 msgr2=0x558498f25690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.977+0000 7feb0ad9d640 1 --2- 172.21.15.29:0/2093450148 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7feaf40a1a30 0x7feaf40a3ef0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7feb04023000 tx=0x7feb04021000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.977+0000 7feb1361e740 1 --2- 172.21.15.29:0/2129575867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25270 0x558498f25690 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7feb04005c40 tx=0x7feb0400f3c0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.977+0000 7feb1361e740 1 -- 172.21.15.29:0/2129575867 shutdown_connections 2024-03-11T19:13:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.977+0000 7feb1361e740 1 --2- 172.21.15.29:0/2129575867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f2add0 0x558498f2d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.977+0000 7feb1361e740 1 --2- 172.21.15.29:0/2129575867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f25c60 0x558498f2a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.977+0000 7feb1361e740 1 --2- 172.21.15.29:0/2129575867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25270 0x558498f25690 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.977+0000 7feb1361e740 1 -- 172.21.15.29:0/2129575867 >> 172.21.15.29:0/2129575867 conn(0x558498f207e0 msgr2=0x558498f20be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.977+0000 7feb1361e740 1 -- 172.21.15.29:0/2129575867 shutdown_connections 2024-03-11T19:13:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.977+0000 7feb1361e740 1 -- 172.21.15.29:0/2129575867 wait complete. 2024-03-11T19:13:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.977+0000 7feb1361e740 1 -- start start 2024-03-11T19:13:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.977+0000 7feb1361e740 1 Processor -- start 2024-03-11T19:13:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb1361e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f25270 0x558498f2f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb1361e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f25c60 0x558498f2f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb1361e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f2add0 0x558498f369a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb1361e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558498e8e020 con 0x558498f2add0 2024-03-11T19:13:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb1361e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558498b61430 con 0x558498f25c60 2024-03-11T19:13:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb1361e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558498b1de00 con 0x558498f25270 2024-03-11T19:13:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb1361e740 10 client.0 _ll_register_callbacks cb 0x558498e35cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb0ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f2add0 0x558498f369a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb0ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f2add0 0x558498f369a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54934/0 (socket says 172.21.15.29:54934) 2024-03-11T19:13:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb0ad9d640 1 -- 172.21.15.29:0/1694842201 learned_addr learned my addr 172.21.15.29:0/1694842201 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb0a59c640 1 --2- 172.21.15.29:0/1694842201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f25270 0x558498f2f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb09d9b640 1 --2- 172.21.15.29:0/1694842201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f25c60 0x558498f2f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb0ad9d640 1 -- 172.21.15.29:0/1694842201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f25270 msgr2=0x558498f2f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb0ad9d640 1 --2- 172.21.15.29:0/1694842201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f25270 0x558498f2f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb0ad9d640 1 -- 172.21.15.29:0/1694842201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f25c60 msgr2=0x558498f2f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb0ad9d640 1 --2- 172.21.15.29:0/1694842201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f25c60 0x558498f2f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb0ad9d640 1 -- 172.21.15.29:0/1694842201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb04013700 con 0x558498f2add0 2024-03-11T19:13:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb0ad9d640 1 --2- 172.21.15.29:0/1694842201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f2add0 0x558498f369a0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7feb0400f8c0 tx=0x7feb0401eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7164db02633b52c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feaf1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:13:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feaf1ffb640 1 -- 172.21.15.29:0/1694842201 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb04013170 con 0x558498f2add0 2024-03-11T19:13:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feaf1ffb640 1 -- 172.21.15.29:0/1694842201 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7feb0401e590 con 0x558498f2add0 2024-03-11T19:13:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb1361e740 1 -- 172.21.15.29:0/1694842201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558498b8d7e0 con 0x558498f2add0 2024-03-11T19:13:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feaf1ffb640 1 -- 172.21.15.29:0/1694842201 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb040132d0 con 0x558498f2add0 2024-03-11T19:13:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb1361e740 10 client.16314 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.978+0000 7feb1361e740 1 -- 172.21.15.29:0/1694842201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x558498e8c6a0 con 0x558498f2add0 2024-03-11T19:13:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.979+0000 7feaf1ffb640 1 -- 172.21.15.29:0/1694842201 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x558498b8d7e0 con 0x558498f2add0 2024-03-11T19:13:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.979+0000 7feaf1ffb640 1 -- 172.21.15.29:0/1694842201 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7feb04033450 con 0x558498f2add0 2024-03-11T19:13:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.979+0000 7feb1361e740 10 client.16314 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.979+0000 7feb1361e740 10 client.16314 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:13:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.979+0000 7feb1361e740 20 client.16314 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.979+0000 7feb1361e740 1 --2- 172.21.15.29:0/1694842201 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x558498f3b460 0x558498f3d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.979+0000 7feb1361e740 4 client.16314 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:13:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.979+0000 7feb1361e740 1 -- 172.21.15.29:0/1694842201 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558498bca720 con 0x558498f3b460 2024-03-11T19:13:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.979+0000 7feb0a59c640 1 --2- 172.21.15.29:0/1694842201 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x558498f3b460 0x558498f3d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.979+0000 7feaf1ffb640 10 client.16314 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:13:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.979+0000 7feb0a59c640 1 --2- 172.21.15.29:0/1694842201 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x558498f3b460 0x558498f3d880 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4154ffa4e1645ca9 server_cookie=6c615e05a743b581 in_seq=0 out_seq=0 2024-03-11T19:13:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.980+0000 7feaf1ffb640 1 -- 172.21.15.29:0/1694842201 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558498bca720 con 0x558498f3b460 2024-03-11T19:13:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.980+0000 7feaf1ffb640 10 client.16314 handle_command_reply: tid=0 2024-03-11T19:13:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.980+0000 7feb1361e740 1 client.16314 shutdown 2024-03-11T19:13:19.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.980+0000 7feb1361e740 1 -- 172.21.15.29:0/1694842201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f2add0 msgr2=0x558498f369a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:19.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.980+0000 7feb1361e740 1 --2- 172.21.15.29:0/1694842201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f2add0 0x558498f369a0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7feb0400f8c0 tx=0x7feb0401eac0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.980+0000 7feb1361e740 1 -- 172.21.15.29:0/1694842201 shutdown_connections 2024-03-11T19:13:19.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.980+0000 7feb1361e740 1 --2- 172.21.15.29:0/1694842201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f2add0 0x558498f369a0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.980+0000 7feb1361e740 1 --2- 172.21.15.29:0/1694842201 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x558498f3b460 0x558498f3d880 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.980+0000 7feb1361e740 1 --2- 172.21.15.29:0/1694842201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f25c60 0x558498f2f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.980+0000 7feb1361e740 1 --2- 172.21.15.29:0/1694842201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f25270 0x558498f2f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.980+0000 7feb1361e740 1 -- 172.21.15.29:0/1694842201 >> 172.21.15.29:0/1694842201 conn(0x558498f207e0 msgr2=0x558498f1fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:19.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.980+0000 7feaf1ffb640 0 client.16314 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:13:19.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.980+0000 7feb1361e740 1 -- 172.21.15.29:0/1694842201 shutdown_connections 2024-03-11T19:13:19.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.980+0000 7feb1361e740 1 -- 172.21.15.29:0/1694842201 wait complete. 2024-03-11T19:13:19.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.980+0000 7feb1361e740 20 client.16314 trim_cache size 0 max 16384 2024-03-11T19:13:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb1361e740 1 Processor -- start 2024-03-11T19:13:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb1361e740 1 -- start start 2024-03-11T19:13:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb1361e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f25270 0x558498f421e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb1361e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25c60 0x558498f36960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb1361e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f2add0 0x558498f27180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb1361e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558498e13130 con 0x558498f25c60 2024-03-11T19:13:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb1361e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558498bca750 con 0x558498f2add0 2024-03-11T19:13:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb1361e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558498f12a40 con 0x558498f25270 2024-03-11T19:13:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb09d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25c60 0x558498f36960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb09d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25c60 0x558498f36960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54946/0 (socket says 172.21.15.29:54946) 2024-03-11T19:13:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb09d9b640 1 -- 172.21.15.29:0/1670462585 learned_addr learned my addr 172.21.15.29:0/1670462585 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb0ad9d640 1 --2- 172.21.15.29:0/1670462585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f2add0 0x558498f27180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb0a59c640 1 --2- 172.21.15.29:0/1670462585 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f25270 0x558498f421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb09d9b640 1 -- 172.21.15.29:0/1670462585 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f25270 msgr2=0x558498f421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb09d9b640 1 --2- 172.21.15.29:0/1670462585 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f25270 0x558498f421e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb09d9b640 1 -- 172.21.15.29:0/1670462585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f2add0 msgr2=0x558498f27180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb09d9b640 1 --2- 172.21.15.29:0/1670462585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f2add0 0x558498f27180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb09d9b640 1 -- 172.21.15.29:0/1670462585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558498e28c20 con 0x558498f25c60 2024-03-11T19:13:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb0a59c640 1 --2- 172.21.15.29:0/1670462585 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f25270 0x558498f421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:13:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.992+0000 7feb0ad9d640 1 --2- 172.21.15.29:0/1670462585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f2add0 0x558498f27180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:13:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.993+0000 7feb09d9b640 1 --2- 172.21.15.29:0/1670462585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25c60 0x558498f36960 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7feb00099cc0 tx=0x7feb00099b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41cd1ffb7caa3b4b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:20.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.993+0000 7feaf3fff640 1 -- 172.21.15.29:0/1670462585 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0009de80 con 0x558498f25c60 2024-03-11T19:13:20.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.993+0000 7feaf3fff640 1 -- 172.21.15.29:0/1670462585 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feb00009d90 con 0x558498f25c60 2024-03-11T19:13:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.994+0000 7feaf3fff640 1 -- 172.21.15.29:0/1670462585 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb00095a80 con 0x558498f25c60 2024-03-11T19:13:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.994+0000 7feb1361e740 1 -- 172.21.15.29:0/1670462585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25c60 msgr2=0x558498f36960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.994+0000 7feb1361e740 1 --2- 172.21.15.29:0/1670462585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25c60 0x558498f36960 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7feb00099cc0 tx=0x7feb00099b80 comp rx=0 tx=0).stop 2024-03-11T19:13:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.994+0000 7feb1361e740 1 -- 172.21.15.29:0/1670462585 shutdown_connections 2024-03-11T19:13:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.994+0000 7feb1361e740 1 --2- 172.21.15.29:0/1670462585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f2add0 0x558498f27180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.994+0000 7feb1361e740 1 --2- 172.21.15.29:0/1670462585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25c60 0x558498f36960 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.994+0000 7feb1361e740 1 --2- 172.21.15.29:0/1670462585 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f25270 0x558498f421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.994+0000 7feb1361e740 1 -- 172.21.15.29:0/1670462585 >> 172.21.15.29:0/1670462585 conn(0x558498f207e0 msgr2=0x558498f20bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.995+0000 7feb1361e740 1 -- 172.21.15.29:0/1670462585 shutdown_connections 2024-03-11T19:13:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.995+0000 7feb1361e740 1 -- 172.21.15.29:0/1670462585 wait complete. 2024-03-11T19:13:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.995+0000 7feb1361e740 1 -- start start 2024-03-11T19:13:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.995+0000 7feb1361e740 1 Processor -- start 2024-03-11T19:13:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.995+0000 7feb1361e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25270 0x558498f35030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.995+0000 7feb1361e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f2add0 0x558498f35570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.995+0000 7feb1361e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f36a70 0x558498f35ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.996+0000 7feb0a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25270 0x558498f35030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.996+0000 7feb1361e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558498f91ac0 con 0x558498f25270 2024-03-11T19:13:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.996+0000 7feb0a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25270 0x558498f35030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54954/0 (socket says 172.21.15.29:54954) 2024-03-11T19:13:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.996+0000 7feb0a59c640 1 -- 172.21.15.29:0/3037051877 learned_addr learned my addr 172.21.15.29:0/3037051877 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.996+0000 7feb1361e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558498f91c30 con 0x558498f36a70 2024-03-11T19:13:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.996+0000 7feb0ad9d640 1 --2- 172.21.15.29:0/3037051877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f36a70 0x558498f35ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.996+0000 7feb1361e740 1 -- 172.21.15.29:0/3037051877 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558498f91df0 con 0x558498f2add0 2024-03-11T19:13:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.996+0000 7feb1361e740 10 client.0 _ll_register_callbacks cb 0x7feb04009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.996+0000 7feb09d9b640 1 --2- 172.21.15.29:0/3037051877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f2add0 0x558498f35570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.996+0000 7feb0a59c640 1 -- 172.21.15.29:0/3037051877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f2add0 msgr2=0x558498f35570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.996+0000 7feb0a59c640 1 --2- 172.21.15.29:0/3037051877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f2add0 0x558498f35570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.996+0000 7feb0a59c640 1 -- 172.21.15.29:0/3037051877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f36a70 msgr2=0x558498f35ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.996+0000 7feb0a59c640 1 --2- 172.21.15.29:0/3037051877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f36a70 0x558498f35ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.996+0000 7feb0a59c640 1 -- 172.21.15.29:0/3037051877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb0c19f750 con 0x558498f25270 2024-03-11T19:13:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.996+0000 7feb0ad9d640 1 --2- 172.21.15.29:0/3037051877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f36a70 0x558498f35ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.997+0000 7feb0a59c640 1 --2- 172.21.15.29:0/3037051877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25270 0x558498f35030 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7feafc02c170 tx=0x7feafc02cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da9d049b8ac2582e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.997+0000 7feaf1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:13:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.997+0000 7feaf1ffb640 1 -- 172.21.15.29:0/3037051877 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc0049c0 con 0x558498f25270 2024-03-11T19:13:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.997+0000 7feaf1ffb640 1 -- 172.21.15.29:0/3037051877 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7feafc045040 con 0x558498f25270 2024-03-11T19:13:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.997+0000 7feb1361e740 1 -- 172.21.15.29:0/3037051877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558498f2e540 con 0x558498f25270 2024-03-11T19:13:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.997+0000 7feaf1ffb640 1 -- 172.21.15.29:0/3037051877 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7feafc004040 con 0x558498f25270 2024-03-11T19:13:20.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.997+0000 7feb1361e740 10 client.16320 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:20.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.997+0000 7feb1361e740 1 -- 172.21.15.29:0/3037051877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x558498e4e8c0 con 0x558498f25270 2024-03-11T19:13:20.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.998+0000 7feaf1ffb640 1 -- 172.21.15.29:0/3037051877 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feafc02de90 con 0x558498f25270 2024-03-11T19:13:20.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.998+0000 7feaf1ffb640 1 -- 172.21.15.29:0/3037051877 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7feafc03c040 con 0x558498f25270 2024-03-11T19:13:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.998+0000 7feb1361e740 10 client.16320 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.998+0000 7feb1361e740 10 client.16320 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:13:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.998+0000 7feb1361e740 20 client.16320 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.998+0000 7feb1361e740 1 --2- 172.21.15.29:0/3037051877 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x558498f93810 0x558498f95c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.998+0000 7feb1361e740 4 client.16320 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:13:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.998+0000 7feb1361e740 1 -- 172.21.15.29:0/3037051877 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558498f2e540 con 0x558498f93810 2024-03-11T19:13:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.998+0000 7feb09d9b640 1 --2- 172.21.15.29:0/3037051877 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x558498f93810 0x558498f95c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.999+0000 7feaf1ffb640 10 client.16320 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:13:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.999+0000 7feb09d9b640 1 --2- 172.21.15.29:0/3037051877 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x558498f93810 0x558498f95c30 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=384d33fd47ec7cd4 server_cookie=7cef4aa81061969b in_seq=0 out_seq=0 2024-03-11T19:13:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.999+0000 7feaf1ffb640 1 -- 172.21.15.29:0/3037051877 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558498f2e540 con 0x558498f93810 2024-03-11T19:13:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.999+0000 7feaf1ffb640 10 client.16320 handle_command_reply: tid=0 2024-03-11T19:13:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:19.999+0000 7feb1361e740 1 client.16320 shutdown 2024-03-11T19:13:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.000+0000 7feb1361e740 1 -- 172.21.15.29:0/3037051877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25270 msgr2=0x558498f35030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.000+0000 7feb1361e740 1 --2- 172.21.15.29:0/3037051877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25270 0x558498f35030 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7feafc02c170 tx=0x7feafc02cd00 comp rx=0 tx=0).stop 2024-03-11T19:13:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.000+0000 7feb1361e740 1 -- 172.21.15.29:0/3037051877 shutdown_connections 2024-03-11T19:13:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.000+0000 7feb1361e740 1 --2- 172.21.15.29:0/3037051877 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x558498f93810 0x558498f95c30 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.000+0000 7feb1361e740 1 --2- 172.21.15.29:0/3037051877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558498f36a70 0x558498f35ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.000+0000 7feb1361e740 1 --2- 172.21.15.29:0/3037051877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558498f2add0 0x558498f35570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.000+0000 7feb1361e740 1 --2- 172.21.15.29:0/3037051877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558498f25270 0x558498f35030 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.000+0000 7feb1361e740 1 -- 172.21.15.29:0/3037051877 >> 172.21.15.29:0/3037051877 conn(0x558498f207e0 msgr2=0x558498f421e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:20.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.000+0000 7feaf1ffb640 0 client.16320 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:13:20.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.000+0000 7feb1361e740 1 -- 172.21.15.29:0/3037051877 shutdown_connections 2024-03-11T19:13:20.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.000+0000 7feb1361e740 1 -- 172.21.15.29:0/3037051877 wait complete. 2024-03-11T19:13:20.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.000+0000 7feb1361e740 20 client.16320 trim_cache size 0 max 16384 2024-03-11T19:13:20.008 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:13:20.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.001+0000 7feb10a4e640 1 -- 172.21.15.29:0/2093450148 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7feaf40a1a30 msgr2=0x7feaf40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:20.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.001+0000 7feb10a4e640 1 --2- 172.21.15.29:0/2093450148 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7feaf40a1a30 0x7feaf40a3ef0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7feb04023000 tx=0x7feb04021000 comp rx=0 tx=0).stop 2024-03-11T19:13:20.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.001+0000 7feb10a4e640 1 -- 172.21.15.29:0/2093450148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c104550 msgr2=0x7feb0c1976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:20.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.001+0000 7feb10a4e640 1 --2- 172.21.15.29:0/2093450148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c104550 0x7feb0c1976d0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7feb0000d740 tx=0x7feb0000dc00 comp rx=0 tx=0).stop 2024-03-11T19:13:20.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.001+0000 7feb10a4e640 1 -- 172.21.15.29:0/2093450148 shutdown_connections 2024-03-11T19:13:20.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.001+0000 7feb10a4e640 1 --2- 172.21.15.29:0/2093450148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb0c106e80 0x7feb0c19e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.001+0000 7feb10a4e640 1 --2- 172.21.15.29:0/2093450148 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7feaf40a1a30 0x7feaf40a3ef0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.001+0000 7feb10a4e640 1 --2- 172.21.15.29:0/2093450148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c104550 0x7feb0c1976d0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.001+0000 7feb10a4e640 1 --2- 172.21.15.29:0/2093450148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb0c101c20 0x7feb0c197190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.001+0000 7feb10a4e640 1 -- 172.21.15.29:0/2093450148 >> 172.21.15.29:0/2093450148 conn(0x7feb0c0fb630 msgr2=0x7feb0c1026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:20.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.001+0000 7feb10a4e640 1 -- 172.21.15.29:0/2093450148 shutdown_connections 2024-03-11T19:13:20.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.001+0000 7feb10a4e640 1 -- 172.21.15.29:0/2093450148 wait complete. 2024-03-11T19:13:20.016 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:13:20.016 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:13:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.181+0000 7f6b503b9640 1 -- 172.21.15.29:0/1385384367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b481077a0 msgr2=0x7f6b48107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.181+0000 7f6b503b9640 1 --2- 172.21.15.29:0/1385384367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b481077a0 0x7f6b48107bc0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009900 tx=0x7f6b4402bc30 comp rx=0 tx=0).stop 2024-03-11T19:13:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.181+0000 7f6b503b9640 1 -- 172.21.15.29:0/1385384367 shutdown_connections 2024-03-11T19:13:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.181+0000 7f6b503b9640 1 --2- 172.21.15.29:0/1385384367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b48109350 0x7f6b4810fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.181+0000 7f6b503b9640 1 --2- 172.21.15.29:0/1385384367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b48108990 0x7f6b48108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.181+0000 7f6b503b9640 1 --2- 172.21.15.29:0/1385384367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b481077a0 0x7f6b48107bc0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.181+0000 7f6b503b9640 1 -- 172.21.15.29:0/1385384367 >> 172.21.15.29:0/1385384367 conn(0x7f6b48076730 msgr2=0x7f6b48078b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.181+0000 7f6b503b9640 1 -- 172.21.15.29:0/1385384367 shutdown_connections 2024-03-11T19:13:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.181+0000 7f6b503b9640 1 -- 172.21.15.29:0/1385384367 wait complete. 2024-03-11T19:13:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.182+0000 7f6b503b9640 1 Processor -- start 2024-03-11T19:13:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.182+0000 7f6b503b9640 1 -- start start 2024-03-11T19:13:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.183+0000 7f6b503b9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b481077a0 0x7f6b4819d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.183+0000 7f6b503b9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b48108990 0x7f6b4819dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.183+0000 7f6b503b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b48109350 0x7f6b481a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.183+0000 7f6b503b9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b481a5270 con 0x7f6b48109350 2024-03-11T19:13:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.183+0000 7f6b503b9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b481a53e0 con 0x7f6b481077a0 2024-03-11T19:13:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.183+0000 7f6b503b9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b481a55a0 con 0x7f6b48108990 2024-03-11T19:13:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.183+0000 7f6b4e155640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b481077a0 0x7f6b4819d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.183+0000 7f6b4e155640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b481077a0 0x7f6b4819d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55912/0 (socket says 172.21.15.29:55912) 2024-03-11T19:13:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.183+0000 7f6b4e155640 1 -- 172.21.15.29:0/3558202876 learned_addr learned my addr 172.21.15.29:0/3558202876 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.183+0000 7f6b4e956640 1 --2- 172.21.15.29:0/3558202876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b48109350 0x7f6b481a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.183+0000 7f6b4d954640 1 --2- 172.21.15.29:0/3558202876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b48108990 0x7f6b4819dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.183+0000 7f6b4e155640 1 -- 172.21.15.29:0/3558202876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b48108990 msgr2=0x7f6b4819dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.183+0000 7f6b4e155640 1 --2- 172.21.15.29:0/3558202876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b48108990 0x7f6b4819dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.183+0000 7f6b4e155640 1 -- 172.21.15.29:0/3558202876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b48109350 msgr2=0x7f6b481a4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.183+0000 7f6b4e155640 1 --2- 172.21.15.29:0/3558202876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b48109350 0x7f6b481a4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.183+0000 7f6b4e155640 1 -- 172.21.15.29:0/3558202876 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b44009560 con 0x7f6b481077a0 2024-03-11T19:13:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.183+0000 7f6b4d954640 1 --2- 172.21.15.29:0/3558202876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b48108990 0x7f6b4819dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:13:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.184+0000 7f6b4e155640 1 --2- 172.21.15.29:0/3558202876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b481077a0 0x7f6b4819d780 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6b4402c7d0 tx=0x7f6b44004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.184+0000 7f6b3f7fe640 1 -- 172.21.15.29:0/3558202876 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b44004500 con 0x7f6b481077a0 2024-03-11T19:13:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.184+0000 7f6b3f7fe640 1 -- 172.21.15.29:0/3558202876 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6b4402de90 con 0x7f6b481077a0 2024-03-11T19:13:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.184+0000 7f6b3f7fe640 1 -- 172.21.15.29:0/3558202876 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b44005350 con 0x7f6b481077a0 2024-03-11T19:13:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.184+0000 7f6b503b9640 1 -- 172.21.15.29:0/3558202876 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b481a5820 con 0x7f6b481077a0 2024-03-11T19:13:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.184+0000 7f6b503b9640 1 -- 172.21.15.29:0/3558202876 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b481a5c20 con 0x7f6b481077a0 2024-03-11T19:13:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.188+0000 7f6b3f7fe640 1 -- 172.21.15.29:0/3558202876 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6b440054b0 con 0x7f6b481077a0 2024-03-11T19:13:20.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.189+0000 7f6b52f83740 1 Processor -- start 2024-03-11T19:13:20.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.189+0000 7f6b3f7fe640 1 --2- 172.21.15.29:0/3558202876 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6b340a1b80 0x7f6b340a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.189+0000 7f6b52f83740 1 -- start start 2024-03-11T19:13:20.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.189+0000 7f6b3f7fe640 1 -- 172.21.15.29:0/3558202876 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f6b440ef430 con 0x7f6b481077a0 2024-03-11T19:13:20.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.189+0000 7f6b4e956640 1 --2- 172.21.15.29:0/3558202876 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6b340a1b80 0x7f6b340a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.189+0000 7f6b52f83740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee04270 0x55d71ee04690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.190+0000 7f6b52f83740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee04c60 0x55d71ee09760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.190+0000 7f6b52f83740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee09dd0 0x55d71ee0c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.190+0000 7f6b52f83740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d71edc0960 con 0x55d71ee04270 2024-03-11T19:13:20.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.190+0000 7f6b52f83740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d71ed74310 con 0x55d71ee04c60 2024-03-11T19:13:20.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.190+0000 7f6b52f83740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d71ed71840 con 0x55d71ee09dd0 2024-03-11T19:13:20.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.190+0000 7f6b4e155640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee04270 0x55d71ee04690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.190+0000 7f6b4e155640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee04270 0x55d71ee04690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54984/0 (socket says 172.21.15.29:54984) 2024-03-11T19:13:20.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.190+0000 7f6b4e155640 1 -- 172.21.15.29:0/2893591380 learned_addr learned my addr 172.21.15.29:0/2893591380 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:20.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.190+0000 7f6b4d954640 1 --2- 172.21.15.29:0/2893591380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee04c60 0x55d71ee09760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.190+0000 7f6b4e956640 1 --2- 172.21.15.29:0/3558202876 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6b340a1b80 0x7f6b340a4040 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6b40005fd0 tx=0x7f6b40009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:20.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.190+0000 7f6b4e155640 1 -- 172.21.15.29:0/2893591380 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee09dd0 msgr2=0x55d71ee0c260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:13:20.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.190+0000 7f6b4e155640 1 --2- 172.21.15.29:0/2893591380 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee09dd0 0x55d71ee0c260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.190+0000 7f6b4e155640 1 -- 172.21.15.29:0/2893591380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee04c60 msgr2=0x55d71ee09760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:20.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.190+0000 7f6b4e155640 1 --2- 172.21.15.29:0/2893591380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee04c60 0x55d71ee09760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.190+0000 7f6b4e155640 1 -- 172.21.15.29:0/2893591380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d71ed2f060 con 0x55d71ee04270 2024-03-11T19:13:20.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.190+0000 7f6b4d954640 1 --2- 172.21.15.29:0/2893591380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee04c60 0x55d71ee09760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:13:20.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.191+0000 7f6b4e155640 1 --2- 172.21.15.29:0/2893591380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee04270 0x55d71ee04690 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f6b440c46b0 tx=0x7f6b440bc340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db523180756085 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:20.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.191+0000 7f6b3cbfe640 1 -- 172.21.15.29:0/2893591380 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b440bc440 con 0x55d71ee04270 2024-03-11T19:13:20.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.191+0000 7f6b3cbfe640 1 -- 172.21.15.29:0/2893591380 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6b440c0590 con 0x55d71ee04270 2024-03-11T19:13:20.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.191+0000 7f6b3cbfe640 1 -- 172.21.15.29:0/2893591380 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b440ef0d0 con 0x55d71ee04270 2024-03-11T19:13:20.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.192+0000 7f6b52f83740 1 -- 172.21.15.29:0/2893591380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee04270 msgr2=0x55d71ee04690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:20.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.192+0000 7f6b52f83740 1 --2- 172.21.15.29:0/2893591380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee04270 0x55d71ee04690 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f6b440c46b0 tx=0x7f6b440bc340 comp rx=0 tx=0).stop 2024-03-11T19:13:20.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.192+0000 7f6b52f83740 1 -- 172.21.15.29:0/2893591380 shutdown_connections 2024-03-11T19:13:20.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.192+0000 7f6b52f83740 1 --2- 172.21.15.29:0/2893591380 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee09dd0 0x55d71ee0c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.192+0000 7f6b52f83740 1 --2- 172.21.15.29:0/2893591380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee04c60 0x55d71ee09760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.192+0000 7f6b52f83740 1 --2- 172.21.15.29:0/2893591380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee04270 0x55d71ee04690 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.192+0000 7f6b52f83740 1 -- 172.21.15.29:0/2893591380 >> 172.21.15.29:0/2893591380 conn(0x55d71edff7e0 msgr2=0x55d71edffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.192+0000 7f6b52f83740 1 -- 172.21.15.29:0/2893591380 shutdown_connections 2024-03-11T19:13:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.193+0000 7f6b52f83740 1 -- 172.21.15.29:0/2893591380 wait complete. 2024-03-11T19:13:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.193+0000 7f6b52f83740 1 -- start start 2024-03-11T19:13:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.193+0000 7f6b52f83740 1 Processor -- start 2024-03-11T19:13:20.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.194+0000 7f6b52f83740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee04270 0x55d71ee0e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.194+0000 7f6b52f83740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee04c60 0x55d71ee0e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.194+0000 7f6b52f83740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee09dd0 0x55d71ee157b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.194+0000 7f6b52f83740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d71ed6d020 con 0x55d71ee09dd0 2024-03-11T19:13:20.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.194+0000 7f6b4e155640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee04270 0x55d71ee0e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.194+0000 7f6b52f83740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d71ea40430 con 0x55d71ee04270 2024-03-11T19:13:20.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.194+0000 7f6b4d954640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee04c60 0x55d71ee0e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.194+0000 7f6b52f83740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d71e9fce00 con 0x55d71ee04c60 2024-03-11T19:13:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.194+0000 7f6b4d954640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee04c60 0x55d71ee0e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:43336/0 (socket says 172.21.15.29:43336) 2024-03-11T19:13:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.194+0000 7f6b4d954640 1 -- 172.21.15.29:0/1495579704 learned_addr learned my addr 172.21.15.29:0/1495579704 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.195+0000 7f6b4e956640 1 --2- 172.21.15.29:0/1495579704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee09dd0 0x55d71ee157b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.195+0000 7f6b52f83740 10 client.0 _ll_register_callbacks cb 0x55d71ed14cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.195+0000 7f6b4e956640 1 -- 172.21.15.29:0/1495579704 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee04c60 msgr2=0x55d71ee0e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.195+0000 7f6b4e956640 1 --2- 172.21.15.29:0/1495579704 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee04c60 0x55d71ee0e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.195+0000 7f6b4e956640 1 -- 172.21.15.29:0/1495579704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee04270 msgr2=0x55d71ee0e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.195+0000 7f6b4e956640 1 --2- 172.21.15.29:0/1495579704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee04270 0x55d71ee0e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.195+0000 7f6b4e956640 1 -- 172.21.15.29:0/1495579704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b481a5c20 con 0x55d71ee09dd0 2024-03-11T19:13:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.195+0000 7f6b4e155640 1 --2- 172.21.15.29:0/1495579704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee04270 0x55d71ee0e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.196+0000 7f6b4e956640 1 --2- 172.21.15.29:0/1495579704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee09dd0 0x55d71ee157b0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f6b40011bb0 tx=0x7f6b40004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c99e1a732f3fdd8a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.196+0000 7f6b31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:13:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.196+0000 7f6b31ffb640 1 -- 172.21.15.29:0/1495579704 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4000ee50 con 0x55d71ee09dd0 2024-03-11T19:13:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.196+0000 7f6b52f83740 1 -- 172.21.15.29:0/1495579704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d71ea6c7e0 con 0x55d71ee09dd0 2024-03-11T19:13:20.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.196+0000 7f6b31ffb640 1 -- 172.21.15.29:0/1495579704 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f6b40017070 con 0x55d71ee09dd0 2024-03-11T19:13:20.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.197+0000 7f6b31ffb640 1 -- 172.21.15.29:0/1495579704 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b40003970 con 0x55d71ee09dd0 2024-03-11T19:13:20.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.197+0000 7f6b52f83740 10 client.16332 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:20.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.197+0000 7f6b52f83740 1 -- 172.21.15.29:0/1495579704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d71ed6b6a0 con 0x55d71ee09dd0 2024-03-11T19:13:20.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.197+0000 7f6b31ffb640 1 -- 172.21.15.29:0/1495579704 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55d71ea6c7e0 con 0x55d71ee09dd0 2024-03-11T19:13:20.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.197+0000 7f6b31ffb640 1 -- 172.21.15.29:0/1495579704 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6b4001e420 con 0x55d71ee09dd0 2024-03-11T19:13:20.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.197+0000 7f6b52f83740 10 client.16332 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:20.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.197+0000 7f6b52f83740 10 client.16332 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:13:20.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.197+0000 7f6b52f83740 20 client.16332 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:20.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.197+0000 7f6b52f83740 1 --2- 172.21.15.29:0/1495579704 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d71ee1a500 0x55d71ee1c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.197+0000 7f6b52f83740 4 client.16332 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:13:20.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.197+0000 7f6b52f83740 1 -- 172.21.15.29:0/1495579704 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d71eaa9720 con 0x55d71ee1a500 2024-03-11T19:13:20.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.198+0000 7f6b4e155640 1 --2- 172.21.15.29:0/1495579704 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d71ee1a500 0x55d71ee1c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.199+0000 7f6b4e155640 1 --2- 172.21.15.29:0/1495579704 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d71ee1a500 0x55d71ee1c920 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e3b30bde70ea7c4 server_cookie=aeab6a4d34f4e19d in_seq=0 out_seq=0 2024-03-11T19:13:20.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.199+0000 7f6b31ffb640 10 client.16332 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:20.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.200+0000 7f6b31ffb640 1 -- 172.21.15.29:0/1495579704 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d71eaa9720 con 0x55d71ee1a500 2024-03-11T19:13:20.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.200+0000 7f6b31ffb640 10 client.16332 handle_command_reply: tid=0 2024-03-11T19:13:20.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.200+0000 7f6b52f83740 1 client.16332 shutdown 2024-03-11T19:13:20.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.200+0000 7f6b52f83740 1 -- 172.21.15.29:0/1495579704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee09dd0 msgr2=0x55d71ee157b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:20.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.200+0000 7f6b52f83740 1 --2- 172.21.15.29:0/1495579704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee09dd0 0x55d71ee157b0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f6b40011bb0 tx=0x7f6b40004ab0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.200+0000 7f6b52f83740 1 -- 172.21.15.29:0/1495579704 shutdown_connections 2024-03-11T19:13:20.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.200+0000 7f6b52f83740 1 --2- 172.21.15.29:0/1495579704 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d71ee1a500 0x55d71ee1c920 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.201+0000 7f6b52f83740 1 --2- 172.21.15.29:0/1495579704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee09dd0 0x55d71ee157b0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.201+0000 7f6b52f83740 1 --2- 172.21.15.29:0/1495579704 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee04c60 0x55d71ee0e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.201+0000 7f6b52f83740 1 --2- 172.21.15.29:0/1495579704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee04270 0x55d71ee0e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.201+0000 7f6b52f83740 1 -- 172.21.15.29:0/1495579704 >> 172.21.15.29:0/1495579704 conn(0x55d71edff7e0 msgr2=0x55d71edfeb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:20.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.201+0000 7f6b31ffb640 0 client.16332 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:20.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.201+0000 7f6b52f83740 1 -- 172.21.15.29:0/1495579704 shutdown_connections 2024-03-11T19:13:20.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.201+0000 7f6b52f83740 1 -- 172.21.15.29:0/1495579704 wait complete. 2024-03-11T19:13:20.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.201+0000 7f6b52f83740 20 client.16332 trim_cache size 0 max 16384 2024-03-11T19:13:20.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.223+0000 7f6b52f83740 1 Processor -- start 2024-03-11T19:13:20.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.223+0000 7f6b52f83740 1 -- start start 2024-03-11T19:13:20.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.224+0000 7f6b52f83740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee04270 0x55d71ee0e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.224+0000 7f6b52f83740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee04c60 0x55d71ee0e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.224+0000 7f6b52f83740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee09dd0 0x55d71ee15770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.224+0000 7f6b4e155640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee04270 0x55d71ee0e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.224+0000 7f6b4d954640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee04c60 0x55d71ee0e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.224+0000 7f6b4e155640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee04270 0x55d71ee0e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:43340/0 (socket says 172.21.15.29:43340) 2024-03-11T19:13:20.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.224+0000 7f6b4d954640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee04c60 0x55d71ee0e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55944/0 (socket says 172.21.15.29:55944) 2024-03-11T19:13:20.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.224+0000 7f6b4e956640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee09dd0 0x55d71ee15770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.224+0000 7f6b4e956640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee09dd0 0x55d71ee15770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55004/0 (socket says 172.21.15.29:55004) 2024-03-11T19:13:20.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.224+0000 7f6b4e155640 1 -- 172.21.15.29:0/3431811541 learned_addr learned my addr 172.21.15.29:0/3431811541 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:20.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.224+0000 7f6b52f83740 1 -- 172.21.15.29:0/3431811541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d71ecf2130 con 0x55d71ee09dd0 2024-03-11T19:13:20.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.224+0000 7f6b52f83740 1 -- 172.21.15.29:0/3431811541 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d71ee05350 con 0x55d71ee04c60 2024-03-11T19:13:20.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.224+0000 7f6b52f83740 1 -- 172.21.15.29:0/3431811541 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d71edf1a40 con 0x55d71ee04270 2024-03-11T19:13:20.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.225+0000 7f6b4d954640 1 -- 172.21.15.29:0/3431811541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee04270 msgr2=0x55d71ee0e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:20.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.225+0000 7f6b4d954640 1 --2- 172.21.15.29:0/3431811541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee04270 0x55d71ee0e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.225+0000 7f6b4d954640 1 -- 172.21.15.29:0/3431811541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee09dd0 msgr2=0x55d71ee15770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:20.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.225+0000 7f6b4d954640 1 --2- 172.21.15.29:0/3431811541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee09dd0 0x55d71ee15770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.225+0000 7f6b4d954640 1 -- 172.21.15.29:0/3431811541 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d71ed07c20 con 0x55d71ee04c60 2024-03-11T19:13:20.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.225+0000 7f6b4e956640 1 --2- 172.21.15.29:0/3431811541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee09dd0 0x55d71ee15770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:20.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.225+0000 7f6b4d954640 1 --2- 172.21.15.29:0/3431811541 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee04c60 0x55d71ee0e770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6b38009c20 tx=0x7f6b380127c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aabcb732ec4eca11 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:20.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.226+0000 7f6b33fff640 1 -- 172.21.15.29:0/3431811541 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b38017070 con 0x55d71ee04c60 2024-03-11T19:13:20.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.226+0000 7f6b33fff640 1 -- 172.21.15.29:0/3431811541 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6b380044e0 con 0x55d71ee04c60 2024-03-11T19:13:20.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.226+0000 7f6b33fff640 1 -- 172.21.15.29:0/3431811541 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b38003e80 con 0x55d71ee04c60 2024-03-11T19:13:20.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.227+0000 7f6b52f83740 1 -- 172.21.15.29:0/3431811541 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee04c60 msgr2=0x55d71ee0e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:20.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.227+0000 7f6b52f83740 1 --2- 172.21.15.29:0/3431811541 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee04c60 0x55d71ee0e770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6b38009c20 tx=0x7f6b380127c0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.227+0000 7f6b52f83740 1 -- 172.21.15.29:0/3431811541 shutdown_connections 2024-03-11T19:13:20.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.227+0000 7f6b52f83740 1 --2- 172.21.15.29:0/3431811541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee09dd0 0x55d71ee15770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.227+0000 7f6b52f83740 1 --2- 172.21.15.29:0/3431811541 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee04c60 0x55d71ee0e770 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.227+0000 7f6b52f83740 1 --2- 172.21.15.29:0/3431811541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee04270 0x55d71ee0e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.227+0000 7f6b52f83740 1 -- 172.21.15.29:0/3431811541 >> 172.21.15.29:0/3431811541 conn(0x55d71edff7e0 msgr2=0x55d71edffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:20.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.227+0000 7f6b52f83740 1 -- 172.21.15.29:0/3431811541 shutdown_connections 2024-03-11T19:13:20.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.227+0000 7f6b52f83740 1 -- 172.21.15.29:0/3431811541 wait complete. 2024-03-11T19:13:20.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.228+0000 7f6b52f83740 1 -- start start 2024-03-11T19:13:20.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.228+0000 7f6b52f83740 1 Processor -- start 2024-03-11T19:13:20.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.228+0000 7f6b52f83740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee04270 0x55d71ee15660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.228+0000 7f6b52f83740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee04c60 0x55d71ee13c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.228+0000 7f6b52f83740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee09dd0 0x55d71ee141b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.228+0000 7f6b52f83740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d71ee14890 con 0x55d71ee04270 2024-03-11T19:13:20.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.228+0000 7f6b52f83740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d71ee14a00 con 0x55d71ee09dd0 2024-03-11T19:13:20.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.228+0000 7f6b52f83740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d71ee709f0 con 0x55d71ee04c60 2024-03-11T19:13:20.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.228+0000 7f6b52f83740 10 client.0 _ll_register_callbacks cb 0x7f6b44005760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:20.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.228+0000 7f6b4e155640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee04270 0x55d71ee15660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.228+0000 7f6b4e956640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee09dd0 0x55d71ee141b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.228+0000 7f6b4e155640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee04270 0x55d71ee15660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55014/0 (socket says 172.21.15.29:55014) 2024-03-11T19:13:20.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.228+0000 7f6b4e956640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee09dd0 0x55d71ee141b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55956/0 (socket says 172.21.15.29:55956) 2024-03-11T19:13:20.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.228+0000 7f6b4e155640 1 -- 172.21.15.29:0/1371657588 learned_addr learned my addr 172.21.15.29:0/1371657588 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:20.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.229+0000 7f6b4e956640 1 -- 172.21.15.29:0/1371657588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee04c60 msgr2=0x55d71ee13c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:20.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.229+0000 7f6b4e956640 1 --2- 172.21.15.29:0/1371657588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee04c60 0x55d71ee13c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.229+0000 7f6b4e956640 1 -- 172.21.15.29:0/1371657588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee04270 msgr2=0x55d71ee15660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:20.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.229+0000 7f6b4e956640 1 --2- 172.21.15.29:0/1371657588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee04270 0x55d71ee15660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.229+0000 7f6b4e956640 1 -- 172.21.15.29:0/1371657588 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d71ed6b6a0 con 0x55d71ee09dd0 2024-03-11T19:13:20.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.229+0000 7f6b4e956640 1 --2- 172.21.15.29:0/1371657588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee09dd0 0x55d71ee141b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6b4001e7d0 tx=0x7f6b4001e650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=841a3bc51c2d68fe server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:20.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.229+0000 7f6b31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:13:20.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.229+0000 7f6b31ffb640 1 -- 172.21.15.29:0/1371657588 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b40036070 con 0x55d71ee09dd0 2024-03-11T19:13:20.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.230+0000 7f6b31ffb640 1 -- 172.21.15.29:0/1371657588 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f6b40017040 con 0x55d71ee09dd0 2024-03-11T19:13:20.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.230+0000 7f6b52f83740 1 -- 172.21.15.29:0/1371657588 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b440c0590 con 0x55d71ee09dd0 2024-03-11T19:13:20.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.230+0000 7f6b31ffb640 1 -- 172.21.15.29:0/1371657588 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b40009570 con 0x55d71ee09dd0 2024-03-11T19:13:20.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.230+0000 7f6b52f83740 10 client.25862 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:20.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.230+0000 7f6b52f83740 1 -- 172.21.15.29:0/1371657588 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7f6b38009900 con 0x55d71ee09dd0 2024-03-11T19:13:20.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.231+0000 7f6b31ffb640 1 -- 172.21.15.29:0/1371657588 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6b440c0590 con 0x55d71ee09dd0 2024-03-11T19:13:20.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.231+0000 7f6b31ffb640 1 -- 172.21.15.29:0/1371657588 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6b4001ed20 con 0x55d71ee09dd0 2024-03-11T19:13:20.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.231+0000 7f6b52f83740 10 client.25862 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:20.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.231+0000 7f6b52f83740 10 client.25862 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:13:20.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.231+0000 7f6b52f83740 20 client.25862 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:20.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.231+0000 7f6b52f83740 1 --2- 172.21.15.29:0/1371657588 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d71ee72400 0x55d71ee74820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:20.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.231+0000 7f6b52f83740 4 client.25862 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:13:20.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.231+0000 7f6b52f83740 1 -- 172.21.15.29:0/1371657588 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d71eaf8890 con 0x55d71ee72400 2024-03-11T19:13:20.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.232+0000 7f6b4e155640 1 --2- 172.21.15.29:0/1371657588 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d71ee72400 0x55d71ee74820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:20.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.232+0000 7f6b4e155640 1 --2- 172.21.15.29:0/1371657588 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d71ee72400 0x55d71ee74820 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12d56842260a3e0c server_cookie=1e9dbf561282f390 in_seq=0 out_seq=0 2024-03-11T19:13:20.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.232+0000 7f6b31ffb640 10 client.25862 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:20.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.232+0000 7f6b31ffb640 1 -- 172.21.15.29:0/1371657588 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d71eaf8890 con 0x55d71ee72400 2024-03-11T19:13:20.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.232+0000 7f6b31ffb640 10 client.25862 handle_command_reply: tid=0 2024-03-11T19:13:20.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.232+0000 7f6b52f83740 1 client.25862 shutdown 2024-03-11T19:13:20.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.232+0000 7f6b52f83740 1 -- 172.21.15.29:0/1371657588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee09dd0 msgr2=0x55d71ee141b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:20.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.232+0000 7f6b52f83740 1 --2- 172.21.15.29:0/1371657588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee09dd0 0x55d71ee141b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6b4001e7d0 tx=0x7f6b4001e650 comp rx=0 tx=0).stop 2024-03-11T19:13:20.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.232+0000 7f6b52f83740 1 -- 172.21.15.29:0/1371657588 shutdown_connections 2024-03-11T19:13:20.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.232+0000 7f6b52f83740 1 --2- 172.21.15.29:0/1371657588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d71ee09dd0 0x55d71ee141b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.232+0000 7f6b52f83740 1 --2- 172.21.15.29:0/1371657588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d71ee04c60 0x55d71ee13c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.232+0000 7f6b52f83740 1 --2- 172.21.15.29:0/1371657588 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d71ee72400 0x55d71ee74820 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.232+0000 7f6b52f83740 1 --2- 172.21.15.29:0/1371657588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d71ee04270 0x55d71ee15660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.232+0000 7f6b52f83740 1 -- 172.21.15.29:0/1371657588 >> 172.21.15.29:0/1371657588 conn(0x55d71edff7e0 msgr2=0x55d71ee0e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:20.239 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:13:20.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.233+0000 7f6b31ffb640 0 client.25862 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:20.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.233+0000 7f6b52f83740 1 -- 172.21.15.29:0/1371657588 shutdown_connections 2024-03-11T19:13:20.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.233+0000 7f6b52f83740 1 -- 172.21.15.29:0/1371657588 wait complete. 2024-03-11T19:13:20.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.233+0000 7f6b52f83740 20 client.25862 trim_cache size 0 max 16384 2024-03-11T19:13:20.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.235+0000 7f6b503b9640 1 -- 172.21.15.29:0/3558202876 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6b340a1b80 msgr2=0x7f6b340a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:20.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.235+0000 7f6b503b9640 1 --2- 172.21.15.29:0/3558202876 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6b340a1b80 0x7f6b340a4040 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6b40005fd0 tx=0x7f6b40009500 comp rx=0 tx=0).stop 2024-03-11T19:13:20.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.235+0000 7f6b503b9640 1 -- 172.21.15.29:0/3558202876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b481077a0 msgr2=0x7f6b4819d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:20.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.235+0000 7f6b503b9640 1 --2- 172.21.15.29:0/3558202876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b481077a0 0x7f6b4819d780 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6b4402c7d0 tx=0x7f6b44004040 comp rx=0 tx=0).stop 2024-03-11T19:13:20.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.235+0000 7f6b503b9640 1 -- 172.21.15.29:0/3558202876 shutdown_connections 2024-03-11T19:13:20.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.235+0000 7f6b503b9640 1 --2- 172.21.15.29:0/3558202876 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6b340a1b80 0x7f6b340a4040 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.235+0000 7f6b503b9640 1 --2- 172.21.15.29:0/3558202876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b48109350 0x7f6b481a4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.235+0000 7f6b503b9640 1 --2- 172.21.15.29:0/3558202876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b48108990 0x7f6b4819dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.235+0000 7f6b503b9640 1 --2- 172.21.15.29:0/3558202876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b481077a0 0x7f6b4819d780 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:20.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.235+0000 7f6b503b9640 1 -- 172.21.15.29:0/3558202876 >> 172.21.15.29:0/3558202876 conn(0x7f6b48076730 msgr2=0x7f6b4810be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:20.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.235+0000 7f6b503b9640 1 -- 172.21.15.29:0/3558202876 shutdown_connections 2024-03-11T19:13:20.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:20.235+0000 7f6b503b9640 1 -- 172.21.15.29:0/3558202876 wait complete. 2024-03-11T19:13:21.258 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:13:21.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:13:21.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:13:21.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.427+0000 7f4035d01640 1 -- 172.21.15.29:0/1213666855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40301000a0 msgr2=0x7f40301004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:21.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.427+0000 7f4035d01640 1 --2- 172.21.15.29:0/1213666855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40301000a0 0x7f40301004c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f4020009900 tx=0x7f402002bc30 comp rx=0 tx=0).stop 2024-03-11T19:13:21.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.427+0000 7f4035d01640 1 -- 172.21.15.29:0/1213666855 shutdown_connections 2024-03-11T19:13:21.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.427+0000 7f4035d01640 1 --2- 172.21.15.29:0/1213666855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4030101c50 0x7f40301084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.427+0000 7f4035d01640 1 --2- 172.21.15.29:0/1213666855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4030101290 0x7f4030101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.427+0000 7f4035d01640 1 --2- 172.21.15.29:0/1213666855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40301000a0 0x7f40301004c0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.427+0000 7f4035d01640 1 -- 172.21.15.29:0/1213666855 >> 172.21.15.29:0/1213666855 conn(0x7f40300fb630 msgr2=0x7f40300fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:21.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.428+0000 7f4035d01640 1 -- 172.21.15.29:0/1213666855 shutdown_connections 2024-03-11T19:13:21.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.428+0000 7f4035d01640 1 -- 172.21.15.29:0/1213666855 wait complete. 2024-03-11T19:13:21.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.428+0000 7f4035d01640 1 Processor -- start 2024-03-11T19:13:21.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.429+0000 7f4035d01640 1 -- start start 2024-03-11T19:13:21.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.429+0000 7f4035d01640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40301000a0 0x7f403019ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.429+0000 7f4035d01640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4030101290 0x7f40301a2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.429+0000 7f4035d01640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030101c50 0x7f40301a4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.429+0000 7f4035d01640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40301a52b0 con 0x7f4030101c50 2024-03-11T19:13:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.429+0000 7f4035d01640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40301a5420 con 0x7f40301000a0 2024-03-11T19:13:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.429+0000 7f402f7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40301000a0 0x7f403019ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.429+0000 7f402f7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40301000a0 0x7f403019ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55970/0 (socket says 172.21.15.29:55970) 2024-03-11T19:13:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.429+0000 7f402f7fe640 1 -- 172.21.15.29:0/449381947 learned_addr learned my addr 172.21.15.29:0/449381947 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.429+0000 7f4035d01640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40301a55e0 con 0x7f4030101290 2024-03-11T19:13:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.429+0000 7f402ffff640 1 --2- 172.21.15.29:0/449381947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030101c50 0x7f40301a4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.430+0000 7f402effd640 1 --2- 172.21.15.29:0/449381947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4030101290 0x7f40301a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.430+0000 7f402ffff640 1 -- 172.21.15.29:0/449381947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4030101290 msgr2=0x7f40301a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.430+0000 7f402ffff640 1 --2- 172.21.15.29:0/449381947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4030101290 0x7f40301a2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.430+0000 7f402ffff640 1 -- 172.21.15.29:0/449381947 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40301000a0 msgr2=0x7f403019ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.430+0000 7f402ffff640 1 --2- 172.21.15.29:0/449381947 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40301000a0 0x7f403019ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.430+0000 7f402ffff640 1 -- 172.21.15.29:0/449381947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4020009560 con 0x7f4030101c50 2024-03-11T19:13:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.430+0000 7f402f7fe640 1 --2- 172.21.15.29:0/449381947 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40301000a0 0x7f403019ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.430+0000 7f402ffff640 1 --2- 172.21.15.29:0/449381947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030101c50 0x7f40301a4b20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f4028009fd0 tx=0x7f402800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.431+0000 7f402cff9640 1 -- 172.21.15.29:0/449381947 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402800ce20 con 0x7f4030101c50 2024-03-11T19:13:21.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.431+0000 7f402cff9640 1 -- 172.21.15.29:0/449381947 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4028005020 con 0x7f4030101c50 2024-03-11T19:13:21.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.431+0000 7f4035d01640 1 -- 172.21.15.29:0/449381947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40301a58c0 con 0x7f4030101c50 2024-03-11T19:13:21.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.431+0000 7f402cff9640 1 -- 172.21.15.29:0/449381947 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4028010690 con 0x7f4030101c50 2024-03-11T19:13:21.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.431+0000 7f4035d01640 1 -- 172.21.15.29:0/449381947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40301a5d80 con 0x7f4030101c50 2024-03-11T19:13:21.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.434+0000 7f402cff9640 1 -- 172.21.15.29:0/449381947 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4028005190 con 0x7f4030101c50 2024-03-11T19:13:21.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.435+0000 7f40389a8740 1 Processor -- start 2024-03-11T19:13:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.435+0000 7f40389a8740 1 -- start start 2024-03-11T19:13:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.435+0000 7f40389a8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4828954c0 0x55f4828958e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.435+0000 7f40389a8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 0x55f482896b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.435+0000 7f40389a8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f482897050 0x55f48289d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.435+0000 7f40389a8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f482804f90 con 0x55f482897050 2024-03-11T19:13:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.435+0000 7f40389a8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4828024c0 con 0x55f4828954c0 2024-03-11T19:13:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.435+0000 7f40389a8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4827fdca0 con 0x55f482896690 2024-03-11T19:13:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.435+0000 7f402f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 0x55f482896b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.435+0000 7f402effd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4828954c0 0x55f4828958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.435+0000 7f402f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 0x55f482896b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:43376/0 (socket says 172.21.15.29:43376) 2024-03-11T19:13:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.435+0000 7f402f7fe640 1 -- 172.21.15.29:0/1353532985 learned_addr learned my addr 172.21.15.29:0/1353532985 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.435+0000 7f402effd640 1 --2- 172.21.15.29:0/1353532985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f482897050 0x55f48289d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.435+0000 7f402cff9640 1 --2- 172.21.15.29:0/449381947 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f40180a1a30 0x7f40180a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.435+0000 7f402effd640 1 -- 172.21.15.29:0/1353532985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 msgr2=0x55f482896b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.435+0000 7f402effd640 1 --2- 172.21.15.29:0/1353532985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 0x55f482896b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.435+0000 7f402effd640 1 -- 172.21.15.29:0/1353532985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f482897050 msgr2=0x55f48289d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.435+0000 7f402effd640 1 --2- 172.21.15.29:0/1353532985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f482897050 0x55f48289d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.435+0000 7f402effd640 1 -- 172.21.15.29:0/1353532985 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4827c00b0 con 0x55f4828954c0 2024-03-11T19:13:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.436+0000 7f402cff9640 1 -- 172.21.15.29:0/449381947 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4028014070 con 0x7f4030101c50 2024-03-11T19:13:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.436+0000 7f402effd640 1 --2- 172.21.15.29:0/1353532985 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4828954c0 0x55f4828958e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4024009730 tx=0x7f40240096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc9b89689c494202 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.436+0000 7f402ffff640 1 --2- 172.21.15.29:0/449381947 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f40180a1a30 0x7f40180a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.436+0000 7f4015ffb640 1 -- 172.21.15.29:0/1353532985 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4024004050 con 0x55f4828954c0 2024-03-11T19:13:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.436+0000 7f402ffff640 1 --2- 172.21.15.29:0/449381947 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f40180a1a30 0x7f40180a3ef0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f40280a2cf0 tx=0x7f402800c890 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.436+0000 7f4015ffb640 1 -- 172.21.15.29:0/1353532985 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4024005020 con 0x55f4828954c0 2024-03-11T19:13:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.436+0000 7f4015ffb640 1 -- 172.21.15.29:0/1353532985 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4024004bc0 con 0x55f4828954c0 2024-03-11T19:13:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.437+0000 7f40389a8740 1 -- 172.21.15.29:0/1353532985 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4828954c0 msgr2=0x55f4828958e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.437+0000 7f40389a8740 1 --2- 172.21.15.29:0/1353532985 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4828954c0 0x55f4828958e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4024009730 tx=0x7f40240096c0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.437+0000 7f40389a8740 1 -- 172.21.15.29:0/1353532985 shutdown_connections 2024-03-11T19:13:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.437+0000 7f40389a8740 1 --2- 172.21.15.29:0/1353532985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f482897050 0x55f48289d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.437+0000 7f40389a8740 1 --2- 172.21.15.29:0/1353532985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 0x55f482896b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.437+0000 7f40389a8740 1 --2- 172.21.15.29:0/1353532985 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f4828954c0 0x55f4828958e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.437+0000 7f40389a8740 1 -- 172.21.15.29:0/1353532985 >> 172.21.15.29:0/1353532985 conn(0x55f4828906a0 msgr2=0x55f482890aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:21.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.438+0000 7f40389a8740 1 -- 172.21.15.29:0/1353532985 shutdown_connections 2024-03-11T19:13:21.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.438+0000 7f40389a8740 1 -- 172.21.15.29:0/1353532985 wait complete. 2024-03-11T19:13:21.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.438+0000 7f40389a8740 1 -- start start 2024-03-11T19:13:21.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.438+0000 7f40389a8740 1 Processor -- start 2024-03-11T19:13:21.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.439+0000 7f40389a8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4828954c0 0x55f4828b0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.439+0000 7f40389a8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 0x55f4828b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.439+0000 7f40389a8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f482897050 0x55f4828b7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.439+0000 7f40389a8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4827a2f40 con 0x55f4828954c0 2024-03-11T19:13:21.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.439+0000 7f40389a8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4824d14b0 con 0x55f482897050 2024-03-11T19:13:21.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.439+0000 7f40389a8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f482783670 con 0x55f482896690 2024-03-11T19:13:21.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.439+0000 7f40389a8740 10 client.0 _ll_register_callbacks cb 0x55f4827f0f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:21.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.439+0000 7f402f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 0x55f4828b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.439+0000 7f402f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 0x55f4828b0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:43380/0 (socket says 172.21.15.29:43380) 2024-03-11T19:13:21.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.439+0000 7f402f7fe640 1 -- 172.21.15.29:0/3696505333 learned_addr learned my addr 172.21.15.29:0/3696505333 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:21.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.439+0000 7f402effd640 1 --2- 172.21.15.29:0/3696505333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4828954c0 0x55f4828b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.439+0000 7f402f7fe640 1 -- 172.21.15.29:0/3696505333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f482897050 msgr2=0x55f4828b7d10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:13:21.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.439+0000 7f402f7fe640 1 --2- 172.21.15.29:0/3696505333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f482897050 0x55f4828b7d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.439+0000 7f402f7fe640 1 -- 172.21.15.29:0/3696505333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4828954c0 msgr2=0x55f4828b0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:21.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.439+0000 7f402f7fe640 1 --2- 172.21.15.29:0/3696505333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4828954c0 0x55f4828b0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.439+0000 7f402f7fe640 1 -- 172.21.15.29:0/3696505333 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4024019040 con 0x55f482896690 2024-03-11T19:13:21.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.439+0000 7f402effd640 1 --2- 172.21.15.29:0/3696505333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4828954c0 0x55f4828b0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:13:21.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.440+0000 7f402f7fe640 1 --2- 172.21.15.29:0/3696505333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 0x55f4828b0cd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4020004270 tx=0x7f40200042a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1cf9b91a2a236bb server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:21.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.440+0000 7f400effd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:13:21.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.440+0000 7f400effd640 1 -- 172.21.15.29:0/3696505333 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402002cc20 con 0x55f482896690 2024-03-11T19:13:21.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.440+0000 7f40389a8740 1 -- 172.21.15.29:0/3696505333 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f4827d7f40 con 0x55f482896690 2024-03-11T19:13:21.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.440+0000 7f400effd640 1 -- 172.21.15.29:0/3696505333 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4020046070 con 0x55f482896690 2024-03-11T19:13:21.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.440+0000 7f400effd640 1 -- 172.21.15.29:0/3696505333 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40200044c0 con 0x55f482896690 2024-03-11T19:13:21.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.440+0000 7f40389a8740 10 client.25900 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:21.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.440+0000 7f40389a8740 1 -- 172.21.15.29:0/3696505333 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55f4827fc320 con 0x55f482896690 2024-03-11T19:13:21.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.441+0000 7f400effd640 1 -- 172.21.15.29:0/3696505333 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f4827d7f40 con 0x55f482896690 2024-03-11T19:13:21.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.441+0000 7f400effd640 1 -- 172.21.15.29:0/3696505333 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4020054080 con 0x55f482896690 2024-03-11T19:13:21.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.441+0000 7f40389a8740 10 client.25900 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:21.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.441+0000 7f40389a8740 10 client.25900 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:13:21.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.441+0000 7f40389a8740 20 client.25900 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:21.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.441+0000 7f40389a8740 1 --2- 172.21.15.29:0/3696505333 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f4828bcb20 0x55f4828bef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.441+0000 7f40389a8740 4 client.25900 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:13:21.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.441+0000 7f40389a8740 1 -- 172.21.15.29:0/3696505333 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f4827f4310 con 0x55f4828bcb20 2024-03-11T19:13:21.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.441+0000 7f402effd640 1 --2- 172.21.15.29:0/3696505333 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f4828bcb20 0x55f4828bef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.442+0000 7f400effd640 10 client.25900 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:21.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.442+0000 7f402effd640 1 --2- 172.21.15.29:0/3696505333 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f4828bcb20 0x55f4828bef40 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b9b0584aea3b9f3 server_cookie=1d5f6cb35235a62b in_seq=0 out_seq=0 2024-03-11T19:13:21.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.443+0000 7f400effd640 1 -- 172.21.15.29:0/3696505333 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f4827f4310 con 0x55f4828bcb20 2024-03-11T19:13:21.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.443+0000 7f400effd640 10 client.25900 handle_command_reply: tid=0 2024-03-11T19:13:21.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.443+0000 7f40389a8740 1 client.25900 shutdown 2024-03-11T19:13:21.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.443+0000 7f40389a8740 1 -- 172.21.15.29:0/3696505333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 msgr2=0x55f4828b0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:21.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.443+0000 7f40389a8740 1 --2- 172.21.15.29:0/3696505333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 0x55f4828b0cd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4020004270 tx=0x7f40200042a0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.444+0000 7f40389a8740 1 -- 172.21.15.29:0/3696505333 shutdown_connections 2024-03-11T19:13:21.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.444+0000 7f40389a8740 1 --2- 172.21.15.29:0/3696505333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f482897050 0x55f4828b7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.444+0000 7f40389a8740 1 --2- 172.21.15.29:0/3696505333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 0x55f4828b0cd0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.444+0000 7f40389a8740 1 --2- 172.21.15.29:0/3696505333 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f4828bcb20 0x55f4828bef40 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.444+0000 7f40389a8740 1 --2- 172.21.15.29:0/3696505333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4828954c0 0x55f4828b0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.444+0000 7f40389a8740 1 -- 172.21.15.29:0/3696505333 >> 172.21.15.29:0/3696505333 conn(0x55f4828906a0 msgr2=0x55f48288f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:21.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.444+0000 7f400effd640 0 client.25900 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:21.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.444+0000 7f40389a8740 1 -- 172.21.15.29:0/3696505333 shutdown_connections 2024-03-11T19:13:21.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.444+0000 7f40389a8740 1 -- 172.21.15.29:0/3696505333 wait complete. 2024-03-11T19:13:21.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.444+0000 7f40389a8740 20 client.25900 trim_cache size 0 max 16384 2024-03-11T19:13:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.469+0000 7f40389a8740 1 Processor -- start 2024-03-11T19:13:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.469+0000 7f40389a8740 1 -- start start 2024-03-11T19:13:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.469+0000 7f40389a8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4828954c0 0x55f48289a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.469+0000 7f40389a8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f482896690 0x55f48289a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.469+0000 7f40389a8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f482897050 0x55f4828b7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.469+0000 7f40389a8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f482894f00 con 0x55f482896690 2024-03-11T19:13:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.469+0000 7f402f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f482896690 0x55f48289a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.469+0000 7f402f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f482896690 0x55f48289a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55052/0 (socket says 172.21.15.29:55052) 2024-03-11T19:13:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.469+0000 7f402f7fe640 1 -- 172.21.15.29:0/1344137818 learned_addr learned my addr 172.21.15.29:0/1344137818 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.469+0000 7f40389a8740 1 -- 172.21.15.29:0/1344137818 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4828b05d0 con 0x55f482897050 2024-03-11T19:13:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.469+0000 7f40389a8740 1 -- 172.21.15.29:0/1344137818 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4827d7f40 con 0x55f4828954c0 2024-03-11T19:13:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.469+0000 7f402effd640 1 --2- 172.21.15.29:0/1344137818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4828954c0 0x55f48289a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.470+0000 7f402effd640 1 --2- 172.21.15.29:0/1344137818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f482897050 0x55f4828b7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.470+0000 7f402f7fe640 1 -- 172.21.15.29:0/1344137818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4828954c0 msgr2=0x55f48289a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.470+0000 7f402f7fe640 1 --2- 172.21.15.29:0/1344137818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4828954c0 0x55f48289a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.470+0000 7f402f7fe640 1 -- 172.21.15.29:0/1344137818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f482897050 msgr2=0x55f4828b7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.470+0000 7f402f7fe640 1 --2- 172.21.15.29:0/1344137818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f482897050 0x55f4828b7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.470+0000 7f402f7fe640 1 -- 172.21.15.29:0/1344137818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f482798c00 con 0x55f482896690 2024-03-11T19:13:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.470+0000 7f402f7fe640 1 --2- 172.21.15.29:0/1344137818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f482896690 0x55f48289a8b0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4020004310 tx=0x7f4020009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=532c810950e68529 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.470+0000 7f40157fa640 1 -- 172.21.15.29:0/1344137818 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4020058070 con 0x55f482896690 2024-03-11T19:13:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.470+0000 7f40157fa640 1 -- 172.21.15.29:0/1344137818 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4020044070 con 0x55f482896690 2024-03-11T19:13:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.470+0000 7f40157fa640 1 -- 172.21.15.29:0/1344137818 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4020003800 con 0x55f482896690 2024-03-11T19:13:21.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.471+0000 7f40389a8740 1 -- 172.21.15.29:0/1344137818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f482896690 msgr2=0x55f48289a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:21.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.471+0000 7f40389a8740 1 --2- 172.21.15.29:0/1344137818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f482896690 0x55f48289a8b0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4020004310 tx=0x7f4020009260 comp rx=0 tx=0).stop 2024-03-11T19:13:21.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.471+0000 7f40389a8740 1 -- 172.21.15.29:0/1344137818 shutdown_connections 2024-03-11T19:13:21.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.472+0000 7f40389a8740 1 --2- 172.21.15.29:0/1344137818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f482897050 0x55f4828b7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.472+0000 7f40389a8740 1 --2- 172.21.15.29:0/1344137818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f482896690 0x55f48289a8b0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.472+0000 7f40389a8740 1 --2- 172.21.15.29:0/1344137818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f4828954c0 0x55f48289a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.472+0000 7f40389a8740 1 -- 172.21.15.29:0/1344137818 >> 172.21.15.29:0/1344137818 conn(0x55f4828906a0 msgr2=0x55f4828c38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:21.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.472+0000 7f40389a8740 1 -- 172.21.15.29:0/1344137818 shutdown_connections 2024-03-11T19:13:21.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.472+0000 7f40389a8740 1 -- 172.21.15.29:0/1344137818 wait complete. 2024-03-11T19:13:21.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.472+0000 7f40389a8740 1 -- start start 2024-03-11T19:13:21.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.473+0000 7f40389a8740 1 Processor -- start 2024-03-11T19:13:21.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.474+0000 7f40389a8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4828954c0 0x55f4828b3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.474+0000 7f40389a8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 0x55f4828b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.474+0000 7f40389a8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f482897050 0x55f4828b6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.474+0000 7f40389a8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4828b74e0 con 0x55f4828954c0 2024-03-11T19:13:21.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.474+0000 7f40389a8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4828b7650 con 0x55f482897050 2024-03-11T19:13:21.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.474+0000 7f40389a8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4828b7810 con 0x55f482896690 2024-03-11T19:13:21.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.474+0000 7f402f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 0x55f4828b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.474+0000 7f402effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4828954c0 0x55f4828b3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.474+0000 7f402f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 0x55f4828b43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:43396/0 (socket says 172.21.15.29:43396) 2024-03-11T19:13:21.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.474+0000 7f402f7fe640 1 -- 172.21.15.29:0/4147964808 learned_addr learned my addr 172.21.15.29:0/4147964808 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:21.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.474+0000 7f40389a8740 10 client.0 _ll_register_callbacks cb 0x7f4024009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:21.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.475+0000 7f402f7fe640 1 -- 172.21.15.29:0/4147964808 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f482897050 msgr2=0x55f4828b6fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:13:21.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.475+0000 7f402f7fe640 1 --2- 172.21.15.29:0/4147964808 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f482897050 0x55f4828b6fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.475+0000 7f402f7fe640 1 -- 172.21.15.29:0/4147964808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4828954c0 msgr2=0x55f4828b3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:21.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.475+0000 7f402f7fe640 1 --2- 172.21.15.29:0/4147964808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4828954c0 0x55f4828b3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.475+0000 7f402f7fe640 1 -- 172.21.15.29:0/4147964808 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f4827fc320 con 0x55f482896690 2024-03-11T19:13:21.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.475+0000 7f402f7fe640 1 --2- 172.21.15.29:0/4147964808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 0x55f4828b43a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4020005600 tx=0x7f402002c770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db71abb319bbedf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:21.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.475+0000 7f400effd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:13:21.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.476+0000 7f400effd640 1 -- 172.21.15.29:0/4147964808 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402004a640 con 0x55f482896690 2024-03-11T19:13:21.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.476+0000 7f400effd640 1 -- 172.21.15.29:0/4147964808 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4020000e40 con 0x55f482896690 2024-03-11T19:13:21.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.476+0000 7f40389a8740 1 -- 172.21.15.29:0/4147964808 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f482823900 con 0x55f482896690 2024-03-11T19:13:21.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.476+0000 7f400effd640 1 -- 172.21.15.29:0/4147964808 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4020003800 con 0x55f482896690 2024-03-11T19:13:21.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.476+0000 7f40389a8740 10 client.25906 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:21.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.476+0000 7f40389a8740 1 -- 172.21.15.29:0/4147964808 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55f4827be910 con 0x55f482896690 2024-03-11T19:13:21.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.477+0000 7f400effd640 1 -- 172.21.15.29:0/4147964808 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4020003e10 con 0x55f482896690 2024-03-11T19:13:21.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.477+0000 7f400effd640 1 -- 172.21.15.29:0/4147964808 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4020066020 con 0x55f482896690 2024-03-11T19:13:21.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.477+0000 7f40389a8740 10 client.25906 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:21.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.477+0000 7f40389a8740 10 client.25906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:13:21.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.477+0000 7f40389a8740 20 client.25906 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:21.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.477+0000 7f40389a8740 1 --2- 172.21.15.29:0/4147964808 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f482913ba0 0x55f482915fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.477+0000 7f40389a8740 4 client.25906 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:13:21.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.477+0000 7f40389a8740 1 -- 172.21.15.29:0/4147964808 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f482823900 con 0x55f482913ba0 2024-03-11T19:13:21.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.477+0000 7f402effd640 1 --2- 172.21.15.29:0/4147964808 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f482913ba0 0x55f482915fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.478+0000 7f400effd640 10 client.25906 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:21.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.478+0000 7f402effd640 1 --2- 172.21.15.29:0/4147964808 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f482913ba0 0x55f482915fc0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=209d6f9ffda1c901 server_cookie=cdc9cc2c3fd2ec4b in_seq=0 out_seq=0 2024-03-11T19:13:21.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.480+0000 7f400effd640 1 -- 172.21.15.29:0/4147964808 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x7f4024032080 con 0x55f482913ba0 2024-03-11T19:13:21.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.480+0000 7f400effd640 10 client.25906 handle_command_reply: tid=0 2024-03-11T19:13:21.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.480+0000 7f40389a8740 1 client.25906 shutdown 2024-03-11T19:13:21.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.480+0000 7f40389a8740 1 -- 172.21.15.29:0/4147964808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 msgr2=0x55f4828b43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:21.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.480+0000 7f40389a8740 1 --2- 172.21.15.29:0/4147964808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 0x55f4828b43a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4020005600 tx=0x7f402002c770 comp rx=0 tx=0).stop 2024-03-11T19:13:21.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.480+0000 7f40389a8740 1 -- 172.21.15.29:0/4147964808 shutdown_connections 2024-03-11T19:13:21.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.480+0000 7f40389a8740 1 --2- 172.21.15.29:0/4147964808 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f482897050 0x55f4828b6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.480+0000 7f40389a8740 1 --2- 172.21.15.29:0/4147964808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f482896690 0x55f4828b43a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.480+0000 7f40389a8740 1 --2- 172.21.15.29:0/4147964808 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f482913ba0 0x55f482915fc0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.480+0000 7f40389a8740 1 --2- 172.21.15.29:0/4147964808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f4828954c0 0x55f4828b3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.481+0000 7f40389a8740 1 -- 172.21.15.29:0/4147964808 >> 172.21.15.29:0/4147964808 conn(0x55f4828906a0 msgr2=0x55f48289a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:21.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.481+0000 7f400effd640 0 client.25906 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:21.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.481+0000 7f40389a8740 1 -- 172.21.15.29:0/4147964808 shutdown_connections 2024-03-11T19:13:21.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.481+0000 7f40389a8740 1 -- 172.21.15.29:0/4147964808 wait complete. 2024-03-11T19:13:21.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.481+0000 7f40389a8740 20 client.25906 trim_cache size 0 max 16384 2024-03-11T19:13:21.487 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:13:21.487 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:13:21.487 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "54ffadb1-4447-43d3-a1a4-d4517d0c1aae", 2024-03-11T19:13:21.487 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:13:21.488 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:13:21.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.481+0000 7f4035d01640 1 -- 172.21.15.29:0/449381947 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f40180a1a30 msgr2=0x7f40180a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:21.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.481+0000 7f4035d01640 1 --2- 172.21.15.29:0/449381947 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f40180a1a30 0x7f40180a3ef0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f40280a2cf0 tx=0x7f402800c890 comp rx=0 tx=0).stop 2024-03-11T19:13:21.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.481+0000 7f4035d01640 1 -- 172.21.15.29:0/449381947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030101c50 msgr2=0x7f40301a4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:21.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.481+0000 7f4035d01640 1 --2- 172.21.15.29:0/449381947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030101c50 0x7f40301a4b20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f4028009fd0 tx=0x7f402800ef10 comp rx=0 tx=0).stop 2024-03-11T19:13:21.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.481+0000 7f4035d01640 1 -- 172.21.15.29:0/449381947 shutdown_connections 2024-03-11T19:13:21.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.481+0000 7f4035d01640 1 --2- 172.21.15.29:0/449381947 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f40180a1a30 0x7f40180a3ef0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.481+0000 7f4035d01640 1 --2- 172.21.15.29:0/449381947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030101c50 0x7f40301a4b20 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.481+0000 7f4035d01640 1 --2- 172.21.15.29:0/449381947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4030101290 0x7f40301a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.481+0000 7f4035d01640 1 --2- 172.21.15.29:0/449381947 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40301000a0 0x7f403019ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.482+0000 7f4035d01640 1 -- 172.21.15.29:0/449381947 >> 172.21.15.29:0/449381947 conn(0x7f40300fb630 msgr2=0x7f40300fd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:21.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.482+0000 7f4035d01640 1 -- 172.21.15.29:0/449381947 shutdown_connections 2024-03-11T19:13:21.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.482+0000 7f4035d01640 1 -- 172.21.15.29:0/449381947 wait complete. 2024-03-11T19:13:21.507 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:13:21.507 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:13:21.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:21 smithi029 ceph-mon[32439]: pgmap v457: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-11T19:13:21.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.676+0000 7fb263c42640 1 -- 172.21.15.29:0/636341328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c0ff9f0 msgr2=0x7fb25c101e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:21.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.676+0000 7fb263c42640 1 --2- 172.21.15.29:0/636341328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c0ff9f0 0x7fb25c101e20 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb250009900 tx=0x7fb25002bc30 comp rx=0 tx=0).stop 2024-03-11T19:13:21.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.676+0000 7fb263c42640 1 -- 172.21.15.29:0/636341328 shutdown_connections 2024-03-11T19:13:21.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.676+0000 7fb263c42640 1 --2- 172.21.15.29:0/636341328 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb25c104d10 0x7fb25c105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.676+0000 7fb263c42640 1 --2- 172.21.15.29:0/636341328 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb25c102360 0x7fb25c1047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.676+0000 7fb263c42640 1 --2- 172.21.15.29:0/636341328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c0ff9f0 0x7fb25c101e20 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.676+0000 7fb263c42640 1 -- 172.21.15.29:0/636341328 >> 172.21.15.29:0/636341328 conn(0x7fb25c0fb5f0 msgr2=0x7fb25c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:21.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.676+0000 7fb263c42640 1 -- 172.21.15.29:0/636341328 shutdown_connections 2024-03-11T19:13:21.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.676+0000 7fb263c42640 1 -- 172.21.15.29:0/636341328 wait complete. 2024-03-11T19:13:21.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.677+0000 7fb263c42640 1 Processor -- start 2024-03-11T19:13:21.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.677+0000 7fb263c42640 1 -- start start 2024-03-11T19:13:21.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.678+0000 7fb263c42640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c0ff9f0 0x7fb25c19d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.678+0000 7fb263c42640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb25c102360 0x7fb25c19dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.678+0000 7fb263c42640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb25c104d10 0x7fb25c1a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.678+0000 7fb263c42640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c1a5240 con 0x7fb25c0ff9f0 2024-03-11T19:13:21.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.678+0000 7fb2619de640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c0ff9f0 0x7fb25c19d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.678+0000 7fb263c42640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c1a53b0 con 0x7fb25c102360 2024-03-11T19:13:21.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.678+0000 7fb263c42640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c1a5570 con 0x7fb25c104d10 2024-03-11T19:13:21.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.678+0000 7fb2619de640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c0ff9f0 0x7fb25c19d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55086/0 (socket says 172.21.15.29:55086) 2024-03-11T19:13:21.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.678+0000 7fb2619de640 1 -- 172.21.15.29:0/2988070406 learned_addr learned my addr 172.21.15.29:0/2988070406 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:21.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.678+0000 7fb2611dd640 1 --2- 172.21.15.29:0/2988070406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb25c102360 0x7fb25c19dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.678+0000 7fb2621df640 1 --2- 172.21.15.29:0/2988070406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb25c104d10 0x7fb25c1a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.678+0000 7fb2611dd640 1 -- 172.21.15.29:0/2988070406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb25c104d10 msgr2=0x7fb25c1a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:21.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.678+0000 7fb2611dd640 1 --2- 172.21.15.29:0/2988070406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb25c104d10 0x7fb25c1a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.678+0000 7fb2611dd640 1 -- 172.21.15.29:0/2988070406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c0ff9f0 msgr2=0x7fb25c19d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.678+0000 7fb2611dd640 1 --2- 172.21.15.29:0/2988070406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c0ff9f0 0x7fb25c19d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.678+0000 7fb2611dd640 1 -- 172.21.15.29:0/2988070406 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb250009560 con 0x7fb25c102360 2024-03-11T19:13:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.679+0000 7fb2611dd640 1 --2- 172.21.15.29:0/2988070406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb25c102360 0x7fb25c19dc90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb25800b3f0 tx=0x7fb25800b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.679+0000 7fb24effd640 1 -- 172.21.15.29:0/2988070406 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb258011830 con 0x7fb25c102360 2024-03-11T19:13:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.679+0000 7fb24effd640 1 -- 172.21.15.29:0/2988070406 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb258011e40 con 0x7fb25c102360 2024-03-11T19:13:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.679+0000 7fb263c42640 1 -- 172.21.15.29:0/2988070406 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb25c106070 con 0x7fb25c102360 2024-03-11T19:13:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.679+0000 7fb24effd640 1 -- 172.21.15.29:0/2988070406 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25800f2e0 con 0x7fb25c102360 2024-03-11T19:13:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.679+0000 7fb263c42640 1 -- 172.21.15.29:0/2988070406 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25c106600 con 0x7fb25c102360 2024-03-11T19:13:21.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.683+0000 7fb26680c740 1 Processor -- start 2024-03-11T19:13:21.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.683+0000 7fb26680c740 1 -- start start 2024-03-11T19:13:21.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.683+0000 7fb26680c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdc92d0 0x55b96bdc96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.683+0000 7fb26680c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.683+0000 7fb26680c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdcee30 0x55b96bdd12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.683+0000 7fb26680c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b96bd39280 con 0x55b96bdc92d0 2024-03-11T19:13:21.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.683+0000 7fb26680c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b96bd367b0 con 0x55b96bdc9cc0 2024-03-11T19:13:21.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.683+0000 7fb26680c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b96bd31f90 con 0x55b96bdcee30 2024-03-11T19:13:21.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.683+0000 7fb2619de640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.683+0000 7fb2619de640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56014/0 (socket says 172.21.15.29:56014) 2024-03-11T19:13:21.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.684+0000 7fb2621df640 1 --2- 172.21.15.29:0/2149133842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdc92d0 0x55b96bdc96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.684+0000 7fb2619de640 1 -- 172.21.15.29:0/2149133842 learned_addr learned my addr 172.21.15.29:0/2149133842 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:21.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.684+0000 7fb2619de640 1 -- 172.21.15.29:0/2149133842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdcee30 msgr2=0x55b96bdd12c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:13:21.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.684+0000 7fb2619de640 1 --2- 172.21.15.29:0/2149133842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdcee30 0x55b96bdd12c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.684+0000 7fb2619de640 1 -- 172.21.15.29:0/2149133842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdc92d0 msgr2=0x55b96bdc96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:21.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.684+0000 7fb2619de640 1 --2- 172.21.15.29:0/2149133842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdc92d0 0x55b96bdc96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.684+0000 7fb2619de640 1 -- 172.21.15.29:0/2149133842 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b96bcf0570 con 0x55b96bdc9cc0 2024-03-11T19:13:21.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.684+0000 7fb2621df640 1 --2- 172.21.15.29:0/2149133842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdc92d0 0x55b96bdc96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:21.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.684+0000 7fb2619de640 1 --2- 172.21.15.29:0/2149133842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdce7c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb25002c9a0 tx=0x7fb250005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6247cb91262249f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:21.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.685+0000 7fb247fff640 1 -- 172.21.15.29:0/2149133842 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2500046c0 con 0x55b96bdc9cc0 2024-03-11T19:13:21.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.685+0000 7fb247fff640 1 -- 172.21.15.29:0/2149133842 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb25002de90 con 0x55b96bdc9cc0 2024-03-11T19:13:21.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.685+0000 7fb247fff640 1 -- 172.21.15.29:0/2149133842 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb250003b40 con 0x55b96bdc9cc0 2024-03-11T19:13:21.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.685+0000 7fb24effd640 1 -- 172.21.15.29:0/2988070406 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb258011990 con 0x7fb25c102360 2024-03-11T19:13:21.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.686+0000 7fb26680c740 1 -- 172.21.15.29:0/2149133842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 msgr2=0x55b96bdce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:21.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.686+0000 7fb26680c740 1 --2- 172.21.15.29:0/2149133842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdce7c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb25002c9a0 tx=0x7fb250005e80 comp rx=0 tx=0).stop 2024-03-11T19:13:21.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.686+0000 7fb26680c740 1 -- 172.21.15.29:0/2149133842 shutdown_connections 2024-03-11T19:13:21.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.686+0000 7fb26680c740 1 --2- 172.21.15.29:0/2149133842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdcee30 0x55b96bdd12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.686+0000 7fb26680c740 1 --2- 172.21.15.29:0/2149133842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdce7c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.686+0000 7fb26680c740 1 --2- 172.21.15.29:0/2149133842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdc92d0 0x55b96bdc96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.686+0000 7fb26680c740 1 -- 172.21.15.29:0/2149133842 >> 172.21.15.29:0/2149133842 conn(0x55b96bdc47f0 msgr2=0x55b96bdc4bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:21.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.686+0000 7fb26680c740 1 -- 172.21.15.29:0/2149133842 shutdown_connections 2024-03-11T19:13:21.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.686+0000 7fb26680c740 1 -- 172.21.15.29:0/2149133842 wait complete. 2024-03-11T19:13:21.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.687+0000 7fb26680c740 1 -- start start 2024-03-11T19:13:21.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.687+0000 7fb26680c740 1 Processor -- start 2024-03-11T19:13:21.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.687+0000 7fb24effd640 1 --2- 172.21.15.29:0/2988070406 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb2480a1b10 0x7fb2480a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.687+0000 7fb2621df640 1 --2- 172.21.15.29:0/2988070406 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb2480a1b10 0x7fb2480a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.687+0000 7fb24effd640 1 -- 172.21.15.29:0/2988070406 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb2580ccdc0 con 0x7fb25c102360 2024-03-11T19:13:21.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.688+0000 7fb26680c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdc92d0 0x55b96bdd32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.688+0000 7fb26680c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdd3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.688+0000 7fb26680c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdcee30 0x55b96bdda840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.688+0000 7fb26680c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b96ba05430 con 0x55b96bdc92d0 2024-03-11T19:13:21.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.688+0000 7fb26680c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b96b9c1e00 con 0x55b96bdc9cc0 2024-03-11T19:13:21.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.688+0000 7fb26680c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b96bcb6fd0 con 0x55b96bdcee30 2024-03-11T19:13:21.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.688+0000 7fb26680c740 10 client.0 _ll_register_callbacks cb 0x55b96bd0cfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:21.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.688+0000 7fb2611dd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdd3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.688+0000 7fb2611dd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdd3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56018/0 (socket says 172.21.15.29:56018) 2024-03-11T19:13:21.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.688+0000 7fb2611dd640 1 -- 172.21.15.29:0/3845240846 learned_addr learned my addr 172.21.15.29:0/3845240846 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:21.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.689+0000 7fb2619de640 1 --2- 172.21.15.29:0/3845240846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdc92d0 0x55b96bdd32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.689+0000 7fb2621df640 1 --2- 172.21.15.29:0/2988070406 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb2480a1b10 0x7fb2480a3fd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x55b96bdca520 tx=0x7fb254009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.689+0000 7fb2621df640 1 --2- 172.21.15.29:0/3845240846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdcee30 0x55b96bdda840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.689+0000 7fb2611dd640 1 -- 172.21.15.29:0/3845240846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdcee30 msgr2=0x55b96bdda840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.689+0000 7fb2611dd640 1 --2- 172.21.15.29:0/3845240846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdcee30 0x55b96bdda840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.689+0000 7fb2611dd640 1 -- 172.21.15.29:0/3845240846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdc92d0 msgr2=0x55b96bdd32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.689+0000 7fb2611dd640 1 --2- 172.21.15.29:0/3845240846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdc92d0 0x55b96bdd32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.689+0000 7fb2611dd640 1 -- 172.21.15.29:0/3845240846 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25c105c80 con 0x55b96bdc9cc0 2024-03-11T19:13:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.689+0000 7fb2619de640 1 --2- 172.21.15.29:0/3845240846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdc92d0 0x55b96bdd32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:13:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.689+0000 7fb2611dd640 1 --2- 172.21.15.29:0/3845240846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdd3800 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb258099cc0 tx=0x7fb2580106f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=110cdfae842fca7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.689+0000 7fb2457fa640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:13:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.689+0000 7fb2457fa640 1 -- 172.21.15.29:0/3845240846 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25809de80 con 0x55b96bdc9cc0 2024-03-11T19:13:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.689+0000 7fb26680c740 1 -- 172.21.15.29:0/3845240846 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b96ba317e0 con 0x55b96bdc9cc0 2024-03-11T19:13:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.689+0000 7fb2457fa640 1 -- 172.21.15.29:0/3845240846 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb2580a24d0 con 0x55b96bdc9cc0 2024-03-11T19:13:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.690+0000 7fb2457fa640 1 -- 172.21.15.29:0/3845240846 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2580b8250 con 0x55b96bdc9cc0 2024-03-11T19:13:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.690+0000 7fb26680c740 10 client.25886 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:21.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.690+0000 7fb26680c740 1 -- 172.21.15.29:0/3845240846 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55b96bd30610 con 0x55b96bdc9cc0 2024-03-11T19:13:21.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.690+0000 7fb2457fa640 1 -- 172.21.15.29:0/3845240846 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55b96ba317e0 con 0x55b96bdc9cc0 2024-03-11T19:13:21.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.691+0000 7fb2457fa640 1 -- 172.21.15.29:0/3845240846 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb258095c10 con 0x55b96bdc9cc0 2024-03-11T19:13:21.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.691+0000 7fb26680c740 10 client.25886 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:21.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.691+0000 7fb26680c740 10 client.25886 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:13:21.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.691+0000 7fb26680c740 20 client.25886 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:21.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.691+0000 7fb26680c740 1 --2- 172.21.15.29:0/3845240846 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55b96bddf470 0x55b96bde1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.691+0000 7fb26680c740 4 client.25886 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:13:21.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.691+0000 7fb26680c740 1 -- 172.21.15.29:0/3845240846 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b96ba6e720 con 0x55b96bddf470 2024-03-11T19:13:21.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.691+0000 7fb2619de640 1 --2- 172.21.15.29:0/3845240846 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55b96bddf470 0x55b96bde1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.691+0000 7fb2457fa640 10 client.25886 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:21.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.691+0000 7fb2619de640 1 --2- 172.21.15.29:0/3845240846 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55b96bddf470 0x55b96bde1890 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=965e0bef974d6783 server_cookie=77c4a711405802a8 in_seq=0 out_seq=0 2024-03-11T19:13:21.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.692+0000 7fb2457fa640 1 -- 172.21.15.29:0/3845240846 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b96ba6e720 con 0x55b96bddf470 2024-03-11T19:13:21.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.692+0000 7fb2457fa640 10 client.25886 handle_command_reply: tid=0 2024-03-11T19:13:21.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.692+0000 7fb26680c740 1 client.25886 shutdown 2024-03-11T19:13:21.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.693+0000 7fb26680c740 1 -- 172.21.15.29:0/3845240846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 msgr2=0x55b96bdd3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:21.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.693+0000 7fb26680c740 1 --2- 172.21.15.29:0/3845240846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdd3800 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb258099cc0 tx=0x7fb2580106f0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.693+0000 7fb26680c740 1 -- 172.21.15.29:0/3845240846 shutdown_connections 2024-03-11T19:13:21.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.693+0000 7fb26680c740 1 --2- 172.21.15.29:0/3845240846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdcee30 0x55b96bdda840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.693+0000 7fb26680c740 1 --2- 172.21.15.29:0/3845240846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdd3800 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.693+0000 7fb26680c740 1 --2- 172.21.15.29:0/3845240846 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55b96bddf470 0x55b96bde1890 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.693+0000 7fb26680c740 1 --2- 172.21.15.29:0/3845240846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdc92d0 0x55b96bdd32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.693+0000 7fb26680c740 1 -- 172.21.15.29:0/3845240846 >> 172.21.15.29:0/3845240846 conn(0x55b96bdc47f0 msgr2=0x55b96bdc3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:21.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.693+0000 7fb2457fa640 0 client.25886 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:21.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.693+0000 7fb26680c740 1 -- 172.21.15.29:0/3845240846 shutdown_connections 2024-03-11T19:13:21.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.693+0000 7fb26680c740 1 -- 172.21.15.29:0/3845240846 wait complete. 2024-03-11T19:13:21.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.693+0000 7fb26680c740 20 client.25886 trim_cache size 0 max 16384 2024-03-11T19:13:21.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.722+0000 7fb26680c740 1 Processor -- start 2024-03-11T19:13:21.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.722+0000 7fb26680c740 1 -- start start 2024-03-11T19:13:21.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.722+0000 7fb26680c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdc92d0 0x55b96bdd32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.722+0000 7fb26680c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdda800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.722+0000 7fb26680c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdcee30 0x55b96bdd37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.722+0000 7fb26680c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b96ba31810 con 0x55b96bdc92d0 2024-03-11T19:13:21.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.722+0000 7fb26680c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b96bdca580 con 0x55b96bdc9cc0 2024-03-11T19:13:21.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.722+0000 7fb26680c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b96bdc00f0 con 0x55b96bdcee30 2024-03-11T19:13:21.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.722+0000 7fb2619de640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdc92d0 0x55b96bdd32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.722+0000 7fb2611dd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdda800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.722+0000 7fb2619de640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdc92d0 0x55b96bdd32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55104/0 (socket says 172.21.15.29:55104) 2024-03-11T19:13:21.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.722+0000 7fb2619de640 1 -- 172.21.15.29:0/150527635 learned_addr learned my addr 172.21.15.29:0/150527635 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:21.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.722+0000 7fb2621df640 1 --2- 172.21.15.29:0/150527635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdcee30 0x55b96bdd37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.723+0000 7fb2619de640 1 -- 172.21.15.29:0/150527635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdcee30 msgr2=0x55b96bdd37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:21.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.723+0000 7fb2619de640 1 --2- 172.21.15.29:0/150527635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdcee30 0x55b96bdd37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.723+0000 7fb2619de640 1 -- 172.21.15.29:0/150527635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 msgr2=0x55b96bdda800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:21.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.723+0000 7fb2619de640 1 --2- 172.21.15.29:0/150527635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdda800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.723+0000 7fb2619de640 1 -- 172.21.15.29:0/150527635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b96bcccac0 con 0x55b96bdc92d0 2024-03-11T19:13:21.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.723+0000 7fb2611dd640 1 --2- 172.21.15.29:0/150527635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdda800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:21.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.723+0000 7fb2619de640 1 --2- 172.21.15.29:0/150527635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdc92d0 0x55b96bdd32a0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fb25002cd20 tx=0x7fb250041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d55bcebdaa3e8b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:21.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.723+0000 7fb2477fe640 1 -- 172.21.15.29:0/150527635 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb250054070 con 0x55b96bdc92d0 2024-03-11T19:13:21.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.723+0000 7fb2477fe640 1 -- 172.21.15.29:0/150527635 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb2500044f0 con 0x55b96bdc92d0 2024-03-11T19:13:21.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.723+0000 7fb2477fe640 1 -- 172.21.15.29:0/150527635 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb250050250 con 0x55b96bdc92d0 2024-03-11T19:13:21.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.724+0000 7fb26680c740 1 -- 172.21.15.29:0/150527635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdc92d0 msgr2=0x55b96bdd32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:21.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.724+0000 7fb26680c740 1 --2- 172.21.15.29:0/150527635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdc92d0 0x55b96bdd32a0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fb25002cd20 tx=0x7fb250041040 comp rx=0 tx=0).stop 2024-03-11T19:13:21.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.724+0000 7fb26680c740 1 -- 172.21.15.29:0/150527635 shutdown_connections 2024-03-11T19:13:21.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.724+0000 7fb26680c740 1 --2- 172.21.15.29:0/150527635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdcee30 0x55b96bdd37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.724+0000 7fb26680c740 1 --2- 172.21.15.29:0/150527635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdda800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.724+0000 7fb26680c740 1 --2- 172.21.15.29:0/150527635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdc92d0 0x55b96bdd32a0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.724+0000 7fb26680c740 1 -- 172.21.15.29:0/150527635 >> 172.21.15.29:0/150527635 conn(0x55b96bdc47f0 msgr2=0x55b96bdc4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:21.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.724+0000 7fb26680c740 1 -- 172.21.15.29:0/150527635 shutdown_connections 2024-03-11T19:13:21.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.724+0000 7fb26680c740 1 -- 172.21.15.29:0/150527635 wait complete. 2024-03-11T19:13:21.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.725+0000 7fb26680c740 1 -- start start 2024-03-11T19:13:21.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.725+0000 7fb26680c740 1 Processor -- start 2024-03-11T19:13:21.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.725+0000 7fb26680c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdc92d0 0x55b96bdda7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.725+0000 7fb26680c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdd8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.725+0000 7fb26680c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdcee30 0x55b96bdd93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.725+0000 7fb26680c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b96bdd9af0 con 0x55b96bdcee30 2024-03-11T19:13:21.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.725+0000 7fb26680c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b96bdd9c60 con 0x55b96bdc9cc0 2024-03-11T19:13:21.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.725+0000 7fb26680c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b96be359a0 con 0x55b96bdc92d0 2024-03-11T19:13:21.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.725+0000 7fb2619de640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdc92d0 0x55b96bdda7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.725+0000 7fb2611dd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdd8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.725+0000 7fb2619de640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdc92d0 0x55b96bdda7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:43434/0 (socket says 172.21.15.29:43434) 2024-03-11T19:13:21.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.725+0000 7fb2619de640 1 -- 172.21.15.29:0/2884035027 learned_addr learned my addr 172.21.15.29:0/2884035027 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:21.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.725+0000 7fb2621df640 1 --2- 172.21.15.29:0/2884035027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdcee30 0x55b96bdd93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.726+0000 7fb26680c740 10 client.0 _ll_register_callbacks cb 0x7fb250009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:21.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.726+0000 7fb2611dd640 1 -- 172.21.15.29:0/2884035027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdc92d0 msgr2=0x55b96bdda7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:21.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.726+0000 7fb2611dd640 1 --2- 172.21.15.29:0/2884035027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdc92d0 0x55b96bdda7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.726+0000 7fb2611dd640 1 -- 172.21.15.29:0/2884035027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdcee30 msgr2=0x55b96bdd93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:21.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.726+0000 7fb2611dd640 1 --2- 172.21.15.29:0/2884035027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdcee30 0x55b96bdd93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.726+0000 7fb2611dd640 1 -- 172.21.15.29:0/2884035027 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b96bd30610 con 0x55b96bdc9cc0 2024-03-11T19:13:21.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.726+0000 7fb2619de640 1 --2- 172.21.15.29:0/2884035027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdc92d0 0x55b96bdda7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:21.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.726+0000 7fb2611dd640 1 --2- 172.21.15.29:0/2884035027 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdd8e50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb258099a30 tx=0x7fb258098fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bdfc34aafa0d3e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:21.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.726+0000 7fb2457fa640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:13:21.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.726+0000 7fb2457fa640 1 -- 172.21.15.29:0/2884035027 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb258009200 con 0x55b96bdc9cc0 2024-03-11T19:13:21.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.726+0000 7fb2457fa640 1 -- 172.21.15.29:0/2884035027 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb2580b8250 con 0x55b96bdc9cc0 2024-03-11T19:13:21.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.726+0000 7fb26680c740 1 -- 172.21.15.29:0/2884035027 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb25002de90 con 0x55b96bdc9cc0 2024-03-11T19:13:21.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.727+0000 7fb2457fa640 1 -- 172.21.15.29:0/2884035027 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb25800d040 con 0x55b96bdc9cc0 2024-03-11T19:13:21.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.727+0000 7fb26680c740 10 client.25892 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:21.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.727+0000 7fb26680c740 1 -- 172.21.15.29:0/2884035027 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55b96bcf0570 con 0x55b96bdc9cc0 2024-03-11T19:13:21.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.727+0000 7fb2457fa640 1 -- 172.21.15.29:0/2884035027 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb25002de90 con 0x55b96bdc9cc0 2024-03-11T19:13:21.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.727+0000 7fb2457fa640 1 -- 172.21.15.29:0/2884035027 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb25809eda0 con 0x55b96bdc9cc0 2024-03-11T19:13:21.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.727+0000 7fb26680c740 10 client.25892 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:21.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.727+0000 7fb26680c740 10 client.25892 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:13:21.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.727+0000 7fb26680c740 20 client.25892 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:21.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.727+0000 7fb26680c740 1 --2- 172.21.15.29:0/2884035027 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55b96be37270 0x55b96be39690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:21.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.727+0000 7fb26680c740 4 client.25892 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:13:21.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.727+0000 7fb26680c740 1 -- 172.21.15.29:0/2884035027 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b96babd890 con 0x55b96be37270 2024-03-11T19:13:21.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.728+0000 7fb2619de640 1 --2- 172.21.15.29:0/2884035027 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55b96be37270 0x55b96be39690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:21.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.729+0000 7fb2457fa640 10 client.25892 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:21.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.729+0000 7fb2619de640 1 --2- 172.21.15.29:0/2884035027 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55b96be37270 0x55b96be39690 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1a3d3b50eb38450 server_cookie=2fbe6a24bd5c1052 in_seq=0 out_seq=0 2024-03-11T19:13:21.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.731+0000 7fb2457fa640 1 -- 172.21.15.29:0/2884035027 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55b96babd890 con 0x55b96be37270 2024-03-11T19:13:21.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.731+0000 7fb2457fa640 10 client.25892 handle_command_reply: tid=0 2024-03-11T19:13:21.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.731+0000 7fb26680c740 1 client.25892 shutdown 2024-03-11T19:13:21.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.731+0000 7fb26680c740 1 -- 172.21.15.29:0/2884035027 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 msgr2=0x55b96bdd8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:21.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.731+0000 7fb26680c740 1 --2- 172.21.15.29:0/2884035027 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdd8e50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb258099a30 tx=0x7fb258098fe0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.732+0000 7fb26680c740 1 -- 172.21.15.29:0/2884035027 shutdown_connections 2024-03-11T19:13:21.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.732+0000 7fb26680c740 1 --2- 172.21.15.29:0/2884035027 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55b96be37270 0x55b96be39690 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.732+0000 7fb26680c740 1 --2- 172.21.15.29:0/2884035027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b96bdcee30 0x55b96bdd93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.732+0000 7fb2457fa640 0 client.25892 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:21.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.732+0000 7fb26680c740 1 --2- 172.21.15.29:0/2884035027 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b96bdc9cc0 0x55b96bdd8e50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.732+0000 7fb26680c740 1 --2- 172.21.15.29:0/2884035027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b96bdc92d0 0x55b96bdda7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.732+0000 7fb26680c740 1 -- 172.21.15.29:0/2884035027 >> 172.21.15.29:0/2884035027 conn(0x55b96bdc47f0 msgr2=0x55b96bdd3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:21.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.732+0000 7fb26680c740 1 -- 172.21.15.29:0/2884035027 shutdown_connections 2024-03-11T19:13:21.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.732+0000 7fb26680c740 1 -- 172.21.15.29:0/2884035027 wait complete. 2024-03-11T19:13:21.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.732+0000 7fb26680c740 20 client.25892 trim_cache size 0 max 16384 2024-03-11T19:13:21.739 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:13:21.739 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:13:21.739 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:13:21.739 INFO:teuthology.orchestra.run.smithi029.stdout: "54ffadb1-4447-43d3-a1a4-d4517d0c1aae": { 2024-03-11T19:13:21.739 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:13:21.739 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "54ffadb1-4447-43d3-a1a4-d4517d0c1aae", 2024-03-11T19:13:21.739 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:13:21.739 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:13:21.739 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:13:21.739 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:13:21.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.733+0000 7fb263c42640 1 -- 172.21.15.29:0/2988070406 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb2480a1b10 msgr2=0x7fb2480a3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:21.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.733+0000 7fb263c42640 1 --2- 172.21.15.29:0/2988070406 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb2480a1b10 0x7fb2480a3fd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x55b96bdca520 tx=0x7fb254009500 comp rx=0 tx=0).stop 2024-03-11T19:13:21.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.733+0000 7fb263c42640 1 -- 172.21.15.29:0/2988070406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb25c102360 msgr2=0x7fb25c19dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:21.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.733+0000 7fb263c42640 1 --2- 172.21.15.29:0/2988070406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb25c102360 0x7fb25c19dc90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb25800b3f0 tx=0x7fb25800b8b0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.733+0000 7fb263c42640 1 -- 172.21.15.29:0/2988070406 shutdown_connections 2024-03-11T19:13:21.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.733+0000 7fb263c42640 1 --2- 172.21.15.29:0/2988070406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb25c104d10 0x7fb25c1a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.733+0000 7fb263c42640 1 --2- 172.21.15.29:0/2988070406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb25c102360 0x7fb25c19dc90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.733+0000 7fb263c42640 1 --2- 172.21.15.29:0/2988070406 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb2480a1b10 0x7fb2480a3fd0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.733+0000 7fb263c42640 1 --2- 172.21.15.29:0/2988070406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c0ff9f0 0x7fb25c19d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:21.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.733+0000 7fb263c42640 1 -- 172.21.15.29:0/2988070406 >> 172.21.15.29:0/2988070406 conn(0x7fb25c0fb5f0 msgr2=0x7fb25c102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:21.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.733+0000 7fb263c42640 1 -- 172.21.15.29:0/2988070406 shutdown_connections 2024-03-11T19:13:21.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:21.733+0000 7fb263c42640 1 -- 172.21.15.29:0/2988070406 wait complete. 2024-03-11T19:13:21.757 INFO:tasks.cephfs.filesystem:scrub status for tag:54ffadb1-4447-43d3-a1a4-d4517d0c1aae - {'path': '/', 'tag': '54ffadb1-4447-43d3-a1a4-d4517d0c1aae', 'options': 'recursive,force'} 2024-03-11T19:13:21.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:21 smithi190 ceph-mon[36292]: pgmap v457: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-11T19:13:21.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:21 smithi170 ceph-mon[35690]: pgmap v457: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-11T19:13:22.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:22 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:13:22.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:22 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:13:22.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:22 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:13:22.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:22 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:13:22.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:22 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:13:22.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:22 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:13:22.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:22 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:13:22.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:22 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:13:22.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:22 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:13:22.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:22 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:13:22.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:22 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:13:22.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:22 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:13:23.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:23 smithi190 ceph-mon[36292]: pgmap v458: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-11T19:13:23.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:23 smithi170 ceph-mon[35690]: pgmap v458: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-11T19:13:23.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:23 smithi029 ceph-mon[32439]: pgmap v458: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-11T19:13:25.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:25 smithi170 ceph-mon[35690]: pgmap v459: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-11T19:13:25.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:25 smithi029 ceph-mon[32439]: pgmap v459: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-11T19:13:26.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:25 smithi190 ceph-mon[36292]: pgmap v459: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-11T19:13:26.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:26 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:13:26.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:26 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:13:27.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:26 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:13:27.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:27 smithi029 ceph-mon[32439]: pgmap v460: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T19:13:27.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:27 smithi170 ceph-mon[35690]: pgmap v460: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T19:13:27.913 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:27 smithi190 ceph-mon[36292]: pgmap v460: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T19:13:29.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:29 smithi170 ceph-mon[35690]: pgmap v461: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:13:29.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:29 smithi029 ceph-mon[32439]: pgmap v461: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:13:30.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:29 smithi190 ceph-mon[36292]: pgmap v461: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:13:31.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:31 smithi170 ceph-mon[35690]: pgmap v462: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-11T19:13:31.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:31 smithi029 ceph-mon[32439]: pgmap v462: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-11T19:13:32.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:31 smithi190 ceph-mon[36292]: pgmap v462: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-11T19:13:33.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:33 smithi170 ceph-mon[35690]: pgmap v463: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 74 MiB/s wr, 1.20k op/s 2024-03-11T19:13:33.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:33 smithi029 ceph-mon[32439]: pgmap v463: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 74 MiB/s wr, 1.20k op/s 2024-03-11T19:13:34.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:33 smithi190 ceph-mon[36292]: pgmap v463: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 74 MiB/s wr, 1.20k op/s 2024-03-11T19:13:35.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:35 smithi170 ceph-mon[35690]: pgmap v464: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T19:13:35.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:35 smithi029 ceph-mon[32439]: pgmap v464: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T19:13:36.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:35 smithi190 ceph-mon[36292]: pgmap v464: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T19:13:37.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:37 smithi170 ceph-mon[35690]: pgmap v465: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T19:13:37.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:37 smithi029 ceph-mon[32439]: pgmap v465: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T19:13:38.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:37 smithi190 ceph-mon[36292]: pgmap v465: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T19:13:39.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:39 smithi170 ceph-mon[35690]: pgmap v466: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:13:39.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:39 smithi029 ceph-mon[32439]: pgmap v466: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:13:40.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:39 smithi190 ceph-mon[36292]: pgmap v466: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:13:41.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:41 smithi170 ceph-mon[35690]: pgmap v467: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-11T19:13:41.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:41 smithi029 ceph-mon[32439]: pgmap v467: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-11T19:13:42.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:41 smithi190 ceph-mon[36292]: pgmap v467: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-11T19:13:43.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:43 smithi170 ceph-mon[35690]: pgmap v468: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 76 MiB/s wr, 1.23k op/s 2024-03-11T19:13:43.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:43 smithi029 ceph-mon[32439]: pgmap v468: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 76 MiB/s wr, 1.23k op/s 2024-03-11T19:13:44.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:43 smithi190 ceph-mon[36292]: pgmap v468: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 76 MiB/s wr, 1.23k op/s 2024-03-11T19:13:45.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:45 smithi170 ceph-mon[35690]: pgmap v469: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T19:13:45.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:45 smithi029 ceph-mon[32439]: pgmap v469: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T19:13:46.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:45 smithi190 ceph-mon[36292]: pgmap v469: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T19:13:47.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:47 smithi170 ceph-mon[35690]: pgmap v470: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T19:13:47.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:47 smithi029 ceph-mon[32439]: pgmap v470: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T19:13:48.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:47 smithi190 ceph-mon[36292]: pgmap v470: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T19:13:49.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:49 smithi170 ceph-mon[35690]: pgmap v471: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T19:13:49.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:49 smithi029 ceph-mon[32439]: pgmap v471: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T19:13:50.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:49 smithi190 ceph-mon[36292]: pgmap v471: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T19:13:51.758 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:13:51.758 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:13:51.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:51 smithi170 ceph-mon[35690]: pgmap v472: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-11T19:13:51.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:51 smithi029 ceph-mon[32439]: pgmap v472: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-11T19:13:51.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.923+0000 7f8926d0c640 1 -- 172.21.15.29:0/349057635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8920102370 msgr2=0x7f89201027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:51.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.923+0000 7f8926d0c640 1 --2- 172.21.15.29:0/349057635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8920102370 0x7f89201027f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f890c009900 tx=0x7f890c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:13:51.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.923+0000 7f8926d0c640 1 -- 172.21.15.29:0/349057635 shutdown_connections 2024-03-11T19:13:51.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.923+0000 7f8926d0c640 1 --2- 172.21.15.29:0/349057635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8920102d30 0x7f89201095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.924+0000 7f8926d0c640 1 --2- 172.21.15.29:0/349057635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8920102370 0x7f89201027f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.924+0000 7f8926d0c640 1 --2- 172.21.15.29:0/349057635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8920101180 0x7f89201015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.924+0000 7f8926d0c640 1 -- 172.21.15.29:0/349057635 >> 172.21.15.29:0/349057635 conn(0x7f89200fc6f0 msgr2=0x7f89200feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:51.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.924+0000 7f8926d0c640 1 -- 172.21.15.29:0/349057635 shutdown_connections 2024-03-11T19:13:51.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.925+0000 7f8926d0c640 1 -- 172.21.15.29:0/349057635 wait complete. 2024-03-11T19:13:51.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.926+0000 7f8926d0c640 1 Processor -- start 2024-03-11T19:13:51.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.926+0000 7f8926d0c640 1 -- start start 2024-03-11T19:13:51.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.926+0000 7f8926d0c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8920101180 0x7f892019d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:51.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.926+0000 7f8926d0c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8920102370 0x7f892019dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:51.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.926+0000 7f8926d0c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8920102d30 0x7f89201a4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:51.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.926+0000 7f8926d0c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89201a51d0 con 0x7f8920101180 2024-03-11T19:13:51.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.926+0000 7f8926d0c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89201a5340 con 0x7f8920102370 2024-03-11T19:13:51.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.926+0000 7f8926d0c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89201a5500 con 0x7f8920102d30 2024-03-11T19:13:51.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.926+0000 7f8924aa8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8920101180 0x7f892019d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:51.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.926+0000 7f8924aa8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8920101180 0x7f892019d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41156/0 (socket says 172.21.15.29:41156) 2024-03-11T19:13:51.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.926+0000 7f8924aa8640 1 -- 172.21.15.29:0/2534753983 learned_addr learned my addr 172.21.15.29:0/2534753983 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:51.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.927+0000 7f89252a9640 1 --2- 172.21.15.29:0/2534753983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8920102d30 0x7f89201a4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:51.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.927+0000 7f891ffff640 1 --2- 172.21.15.29:0/2534753983 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8920102370 0x7f892019dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:51.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.927+0000 7f891ffff640 1 -- 172.21.15.29:0/2534753983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8920102d30 msgr2=0x7f89201a4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:51.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.927+0000 7f891ffff640 1 --2- 172.21.15.29:0/2534753983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8920102d30 0x7f89201a4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.927+0000 7f891ffff640 1 -- 172.21.15.29:0/2534753983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8920101180 msgr2=0x7f892019d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:51.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.927+0000 7f891ffff640 1 --2- 172.21.15.29:0/2534753983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8920101180 0x7f892019d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.927+0000 7f891ffff640 1 -- 172.21.15.29:0/2534753983 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8914009490 con 0x7f8920102370 2024-03-11T19:13:51.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.927+0000 7f89252a9640 1 --2- 172.21.15.29:0/2534753983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8920102d30 0x7f89201a4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:51.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.928+0000 7f891ffff640 1 --2- 172.21.15.29:0/2534753983 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8920102370 0x7f892019dc20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f890c02c170 tx=0x7f890c004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:51.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.928+0000 7f891dffb640 1 -- 172.21.15.29:0/2534753983 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c004500 con 0x7f8920102370 2024-03-11T19:13:51.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.929+0000 7f891dffb640 1 -- 172.21.15.29:0/2534753983 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f890c003aa0 con 0x7f8920102370 2024-03-11T19:13:51.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.929+0000 7f891dffb640 1 -- 172.21.15.29:0/2534753983 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c005400 con 0x7f8920102370 2024-03-11T19:13:51.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.929+0000 7f8926d0c640 1 -- 172.21.15.29:0/2534753983 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f890c009560 con 0x7f8920102370 2024-03-11T19:13:51.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.929+0000 7f8926d0c640 1 -- 172.21.15.29:0/2534753983 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89201a5a80 con 0x7f8920102370 2024-03-11T19:13:51.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.933+0000 7f89299c1740 1 Processor -- start 2024-03-11T19:13:51.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.933+0000 7f89299c1740 1 -- start start 2024-03-11T19:13:51.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.933+0000 7f89299c1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56de2d0 0x5643d56de6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:51.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.933+0000 7f89299c1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56decc0 0x5643d56e37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:51.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.933+0000 7f89299c1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56e3e30 0x5643d56e62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:51.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.933+0000 7f89299c1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d564e280 con 0x5643d56e3e30 2024-03-11T19:13:51.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.933+0000 7f89299c1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d564b7b0 con 0x5643d56de2d0 2024-03-11T19:13:51.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.933+0000 7f89299c1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d5646f90 con 0x5643d56decc0 2024-03-11T19:13:51.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.933+0000 7f8924aa8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56decc0 0x5643d56e37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:51.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.933+0000 7f8924aa8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56decc0 0x5643d56e37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:34594/0 (socket says 172.21.15.29:34594) 2024-03-11T19:13:51.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.933+0000 7f8924aa8640 1 -- 172.21.15.29:0/2696991913 learned_addr learned my addr 172.21.15.29:0/2696991913 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:51.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.933+0000 7f891ffff640 1 --2- 172.21.15.29:0/2696991913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56e3e30 0x5643d56e62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:51.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.933+0000 7f89252a9640 1 --2- 172.21.15.29:0/2696991913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56de2d0 0x5643d56de6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:51.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.934+0000 7f8924aa8640 1 -- 172.21.15.29:0/2696991913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56de2d0 msgr2=0x5643d56de6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:51.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.934+0000 7f8924aa8640 1 --2- 172.21.15.29:0/2696991913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56de2d0 0x5643d56de6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.934+0000 7f8924aa8640 1 -- 172.21.15.29:0/2696991913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56e3e30 msgr2=0x5643d56e62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:51.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.934+0000 7f8924aa8640 1 --2- 172.21.15.29:0/2696991913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56e3e30 0x5643d56e62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.934+0000 7f8924aa8640 1 -- 172.21.15.29:0/2696991913 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5643d5605570 con 0x5643d56decc0 2024-03-11T19:13:51.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.934+0000 7f8924aa8640 1 --2- 172.21.15.29:0/2696991913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56decc0 0x5643d56e37c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8914009ea0 tx=0x7f8914009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=462c6a2c832cc4f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:51.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.934+0000 7f890affd640 1 -- 172.21.15.29:0/2696991913 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8914009970 con 0x5643d56decc0 2024-03-11T19:13:51.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.935+0000 7f890affd640 1 -- 172.21.15.29:0/2696991913 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f89140047a0 con 0x5643d56decc0 2024-03-11T19:13:51.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.935+0000 7f890affd640 1 -- 172.21.15.29:0/2696991913 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89140127a0 con 0x5643d56decc0 2024-03-11T19:13:51.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.935+0000 7f891dffb640 1 -- 172.21.15.29:0/2534753983 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f890c003c10 con 0x7f8920102370 2024-03-11T19:13:51.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.935+0000 7f89299c1740 1 -- 172.21.15.29:0/2696991913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56decc0 msgr2=0x5643d56e37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:51.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.935+0000 7f89299c1740 1 --2- 172.21.15.29:0/2696991913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56decc0 0x5643d56e37c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8914009ea0 tx=0x7f8914009f90 comp rx=0 tx=0).stop 2024-03-11T19:13:51.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.935+0000 7f89299c1740 1 -- 172.21.15.29:0/2696991913 shutdown_connections 2024-03-11T19:13:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.935+0000 7f89299c1740 1 --2- 172.21.15.29:0/2696991913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56e3e30 0x5643d56e62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.935+0000 7f89299c1740 1 --2- 172.21.15.29:0/2696991913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56decc0 0x5643d56e37c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.935+0000 7f89299c1740 1 --2- 172.21.15.29:0/2696991913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56de2d0 0x5643d56de6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.935+0000 7f89299c1740 1 -- 172.21.15.29:0/2696991913 >> 172.21.15.29:0/2696991913 conn(0x5643d56d97f0 msgr2=0x5643d56d9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.936+0000 7f89299c1740 1 -- 172.21.15.29:0/2696991913 shutdown_connections 2024-03-11T19:13:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.936+0000 7f89299c1740 1 -- 172.21.15.29:0/2696991913 wait complete. 2024-03-11T19:13:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.936+0000 7f89299c1740 1 -- start start 2024-03-11T19:13:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.936+0000 7f89299c1740 1 Processor -- start 2024-03-11T19:13:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f891dffb640 1 --2- 172.21.15.29:0/2534753983 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f89100a19d0 0x7f89100a3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f89299c1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56de2d0 0x5643d56f92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f89299c1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56decc0 0x5643d56f9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f89299c1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56e3e30 0x5643d5700840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f89299c1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d531a430 con 0x5643d56de2d0 2024-03-11T19:13:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f89299c1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d52d6e00 con 0x5643d56e3e30 2024-03-11T19:13:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f89299c1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d55cbfd0 con 0x5643d56decc0 2024-03-11T19:13:51.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f89299c1740 10 client.0 _ll_register_callbacks cb 0x5643d5621fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:51.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f89252a9640 1 --2- 172.21.15.29:0/2534753983 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f89100a19d0 0x7f89100a3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:51.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f891ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56decc0 0x5643d56f9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:51.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f891ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56decc0 0x5643d56f9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:34604/0 (socket says 172.21.15.29:34604) 2024-03-11T19:13:51.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f891ffff640 1 -- 172.21.15.29:0/2245589711 learned_addr learned my addr 172.21.15.29:0/2245589711 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:51.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f89252a9640 1 --2- 172.21.15.29:0/2245589711 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56e3e30 0x5643d5700840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:51.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f8924aa8640 1 --2- 172.21.15.29:0/2245589711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56de2d0 0x5643d56f92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:51.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f891dffb640 1 -- 172.21.15.29:0/2534753983 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f890c02de40 con 0x7f8920102370 2024-03-11T19:13:51.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f891ffff640 1 -- 172.21.15.29:0/2245589711 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56e3e30 msgr2=0x5643d5700840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f891ffff640 1 --2- 172.21.15.29:0/2245589711 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56e3e30 0x5643d5700840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f891ffff640 1 -- 172.21.15.29:0/2245589711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56de2d0 msgr2=0x5643d56f92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f891ffff640 1 --2- 172.21.15.29:0/2245589711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56de2d0 0x5643d56f92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.937+0000 7f891ffff640 1 -- 172.21.15.29:0/2245589711 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8914009ad0 con 0x5643d56decc0 2024-03-11T19:13:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.938+0000 7f8924aa8640 1 --2- 172.21.15.29:0/2245589711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56de2d0 0x5643d56f92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:13:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.938+0000 7f89252a9640 1 --2- 172.21.15.29:0/2534753983 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f89100a19d0 0x7f89100a3e90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x5643d56df520 tx=0x7f89180128b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.938+0000 7f891ffff640 1 --2- 172.21.15.29:0/2245589711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56decc0 0x5643d56f9800 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f890c0c20e0 tx=0x7f890c0c1fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc00a19deae603d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.938+0000 7f88fffff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:13:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.938+0000 7f88fffff640 1 -- 172.21.15.29:0/2245589711 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c0782b0 con 0x5643d56decc0 2024-03-11T19:13:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.938+0000 7f89299c1740 1 -- 172.21.15.29:0/2245589711 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5643d53467e0 con 0x5643d56decc0 2024-03-11T19:13:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.938+0000 7f88fffff640 1 -- 172.21.15.29:0/2245589711 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f890c004dc0 con 0x5643d56decc0 2024-03-11T19:13:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.939+0000 7f88fffff640 1 -- 172.21.15.29:0/2245589711 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f890c0c0520 con 0x5643d56decc0 2024-03-11T19:13:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.939+0000 7f89299c1740 10 client.25927 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.939+0000 7f89299c1740 1 -- 172.21.15.29:0/2245589711 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5643d5645610 con 0x5643d56decc0 2024-03-11T19:13:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.940+0000 7f88fffff640 1 -- 172.21.15.29:0/2245589711 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5643d53467e0 con 0x5643d56decc0 2024-03-11T19:13:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.940+0000 7f88fffff640 1 -- 172.21.15.29:0/2245589711 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f890c03c070 con 0x5643d56decc0 2024-03-11T19:13:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.940+0000 7f89299c1740 10 client.25927 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.940+0000 7f89299c1740 10 client.25927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:13:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.940+0000 7f89299c1740 20 client.25927 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.940+0000 7f89299c1740 1 --2- 172.21.15.29:0/2245589711 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5643d5705070 0x5643d5707490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.940+0000 7f89299c1740 4 client.25927 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:13:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.940+0000 7f89299c1740 1 -- 172.21.15.29:0/2245589711 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5643d5383720 con 0x5643d5705070 2024-03-11T19:13:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.941+0000 7f8924aa8640 1 --2- 172.21.15.29:0/2245589711 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5643d5705070 0x5643d5707490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.942+0000 7f88fffff640 10 client.25927 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.942+0000 7f8924aa8640 1 --2- 172.21.15.29:0/2245589711 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5643d5705070 0x5643d5707490 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f40c14d2ba3f135 server_cookie=fc9afe12d4d4ffbd in_seq=0 out_seq=0 2024-03-11T19:13:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.942+0000 7f88fffff640 1 -- 172.21.15.29:0/2245589711 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5643d5383720 con 0x5643d5705070 2024-03-11T19:13:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.942+0000 7f88fffff640 10 client.25927 handle_command_reply: tid=0 2024-03-11T19:13:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.943+0000 7f89299c1740 1 client.25927 shutdown 2024-03-11T19:13:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.943+0000 7f89299c1740 1 -- 172.21.15.29:0/2245589711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56decc0 msgr2=0x5643d56f9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.943+0000 7f89299c1740 1 --2- 172.21.15.29:0/2245589711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56decc0 0x5643d56f9800 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f890c0c20e0 tx=0x7f890c0c1fe0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.943+0000 7f89299c1740 1 -- 172.21.15.29:0/2245589711 shutdown_connections 2024-03-11T19:13:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.943+0000 7f89299c1740 1 --2- 172.21.15.29:0/2245589711 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56e3e30 0x5643d5700840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.943+0000 7f89299c1740 1 --2- 172.21.15.29:0/2245589711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56decc0 0x5643d56f9800 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.943+0000 7f89299c1740 1 --2- 172.21.15.29:0/2245589711 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5643d5705070 0x5643d5707490 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.943+0000 7f89299c1740 1 --2- 172.21.15.29:0/2245589711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56de2d0 0x5643d56f92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.943+0000 7f89299c1740 1 -- 172.21.15.29:0/2245589711 >> 172.21.15.29:0/2245589711 conn(0x5643d56d97f0 msgr2=0x5643d56d8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.943+0000 7f88fffff640 0 client.25927 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.943+0000 7f89299c1740 1 -- 172.21.15.29:0/2245589711 shutdown_connections 2024-03-11T19:13:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.943+0000 7f89299c1740 1 -- 172.21.15.29:0/2245589711 wait complete. 2024-03-11T19:13:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.943+0000 7f89299c1740 20 client.25927 trim_cache size 0 max 16384 2024-03-11T19:13:51.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.968+0000 7f89299c1740 1 Processor -- start 2024-03-11T19:13:51.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.968+0000 7f89299c1740 1 -- start start 2024-03-11T19:13:51.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.968+0000 7f89299c1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56de2d0 0x5643d56f92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:51.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.968+0000 7f89299c1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56decc0 0x5643d5700800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:51.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.968+0000 7f89299c1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56e3e30 0x5643d56f97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:51.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.968+0000 7f89299c1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d53467e0 con 0x5643d56decc0 2024-03-11T19:13:51.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.968+0000 7f891ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56decc0 0x5643d5700800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.968+0000 7f8924aa8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56de2d0 0x5643d56f92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.968+0000 7f8924aa8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56de2d0 0x5643d56f92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:34620/0 (socket says 172.21.15.29:34620) 2024-03-11T19:13:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.968+0000 7f8924aa8640 1 -- 172.21.15.29:0/3263170213 learned_addr learned my addr 172.21.15.29:0/3263170213 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.968+0000 7f89299c1740 1 -- 172.21.15.29:0/3263170213 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d56df3b0 con 0x5643d56e3e30 2024-03-11T19:13:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.968+0000 7f89299c1740 1 -- 172.21.15.29:0/3263170213 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d56df550 con 0x5643d56de2d0 2024-03-11T19:13:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.968+0000 7f89252a9640 1 --2- 172.21.15.29:0/3263170213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56e3e30 0x5643d56f97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.969+0000 7f8924aa8640 1 -- 172.21.15.29:0/3263170213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56e3e30 msgr2=0x5643d56f97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.969+0000 7f8924aa8640 1 --2- 172.21.15.29:0/3263170213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56e3e30 0x5643d56f97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.969+0000 7f8924aa8640 1 -- 172.21.15.29:0/3263170213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56decc0 msgr2=0x5643d5700800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.969+0000 7f8924aa8640 1 --2- 172.21.15.29:0/3263170213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56decc0 0x5643d5700800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.969+0000 7f8924aa8640 1 -- 172.21.15.29:0/3263170213 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5643d55e1ac0 con 0x5643d56de2d0 2024-03-11T19:13:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.969+0000 7f8924aa8640 1 --2- 172.21.15.29:0/3263170213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56de2d0 0x5643d56f92a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8914009ea0 tx=0x7f8914005f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ead30b84dfea2674 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.969+0000 7f890a7fc640 1 -- 172.21.15.29:0/3263170213 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89140293e0 con 0x5643d56de2d0 2024-03-11T19:13:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.969+0000 7f890a7fc640 1 -- 172.21.15.29:0/3263170213 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8914012450 con 0x5643d56de2d0 2024-03-11T19:13:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.969+0000 7f890a7fc640 1 -- 172.21.15.29:0/3263170213 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8914004df0 con 0x5643d56de2d0 2024-03-11T19:13:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.970+0000 7f89299c1740 1 -- 172.21.15.29:0/3263170213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56de2d0 msgr2=0x5643d56f92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.970+0000 7f89299c1740 1 --2- 172.21.15.29:0/3263170213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56de2d0 0x5643d56f92a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8914009ea0 tx=0x7f8914005f50 comp rx=0 tx=0).stop 2024-03-11T19:13:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.970+0000 7f89299c1740 1 -- 172.21.15.29:0/3263170213 shutdown_connections 2024-03-11T19:13:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.970+0000 7f89299c1740 1 --2- 172.21.15.29:0/3263170213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56e3e30 0x5643d56f97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.970+0000 7f89299c1740 1 --2- 172.21.15.29:0/3263170213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56decc0 0x5643d5700800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.970+0000 7f89299c1740 1 --2- 172.21.15.29:0/3263170213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56de2d0 0x5643d56f92a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.970+0000 7f89299c1740 1 -- 172.21.15.29:0/3263170213 >> 172.21.15.29:0/3263170213 conn(0x5643d56d97f0 msgr2=0x5643d570be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.970+0000 7f89299c1740 1 -- 172.21.15.29:0/3263170213 shutdown_connections 2024-03-11T19:13:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.971+0000 7f89299c1740 1 -- 172.21.15.29:0/3263170213 wait complete. 2024-03-11T19:13:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.971+0000 7f89299c1740 1 -- start start 2024-03-11T19:13:51.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.971+0000 7f89299c1740 1 Processor -- start 2024-03-11T19:13:51.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.972+0000 7f89299c1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56de2d0 0x5643d5700520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:51.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.972+0000 7f89299c1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56decc0 0x5643d56feba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:51.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.972+0000 7f89299c1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56e3e30 0x5643d56ff0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:51.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.972+0000 7f89299c1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d56ff620 con 0x5643d56e3e30 2024-03-11T19:13:51.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.972+0000 7f89299c1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d56ff760 con 0x5643d56decc0 2024-03-11T19:13:51.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.972+0000 7f89299c1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d56ff920 con 0x5643d56de2d0 2024-03-11T19:13:51.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.972+0000 7f8924aa8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56de2d0 0x5643d5700520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:51.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.972+0000 7f8924aa8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56de2d0 0x5643d5700520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:34636/0 (socket says 172.21.15.29:34636) 2024-03-11T19:13:51.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.972+0000 7f8924aa8640 1 -- 172.21.15.29:0/4041269910 learned_addr learned my addr 172.21.15.29:0/4041269910 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:51.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.972+0000 7f89299c1740 10 client.0 _ll_register_callbacks cb 0x7f89140098c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:51.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.972+0000 7f891ffff640 1 --2- 172.21.15.29:0/4041269910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56decc0 0x5643d56feba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:51.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.972+0000 7f89252a9640 1 --2- 172.21.15.29:0/4041269910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56e3e30 0x5643d56ff0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:51.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.973+0000 7f8924aa8640 1 -- 172.21.15.29:0/4041269910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56decc0 msgr2=0x5643d56feba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:51.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.973+0000 7f8924aa8640 1 --2- 172.21.15.29:0/4041269910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56decc0 0x5643d56feba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.973+0000 7f8924aa8640 1 -- 172.21.15.29:0/4041269910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56e3e30 msgr2=0x5643d56ff0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.973+0000 7f8924aa8640 1 --2- 172.21.15.29:0/4041269910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56e3e30 0x5643d56ff0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.973+0000 7f8924aa8640 1 -- 172.21.15.29:0/4041269910 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f891401e040 con 0x5643d56de2d0 2024-03-11T19:13:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.973+0000 7f8924aa8640 1 --2- 172.21.15.29:0/4041269910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56de2d0 0x5643d5700520 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8914004730 tx=0x7f8914011c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b5eb2fb3ef78176 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.973+0000 7f88fffff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:13:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.973+0000 7f88fffff640 1 -- 172.21.15.29:0/4041269910 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8914038070 con 0x5643d56de2d0 2024-03-11T19:13:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.973+0000 7f89299c1740 1 -- 172.21.15.29:0/4041269910 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f89140047a0 con 0x5643d56de2d0 2024-03-11T19:13:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.973+0000 7f88fffff640 1 -- 172.21.15.29:0/4041269910 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f89140125c0 con 0x5643d56de2d0 2024-03-11T19:13:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.973+0000 7f88fffff640 1 -- 172.21.15.29:0/4041269910 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8914012780 con 0x5643d56de2d0 2024-03-11T19:13:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.973+0000 7f89299c1740 10 client.25933 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.973+0000 7f89299c1740 1 -- 172.21.15.29:0/4041269910 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5643d56076a0 con 0x5643d56de2d0 2024-03-11T19:13:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.974+0000 7f88fffff640 1 -- 172.21.15.29:0/4041269910 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f89140047a0 con 0x5643d56de2d0 2024-03-11T19:13:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.974+0000 7f88fffff640 1 -- 172.21.15.29:0/4041269910 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f891403b030 con 0x5643d56de2d0 2024-03-11T19:13:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.974+0000 7f89299c1740 10 client.25933 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.974+0000 7f89299c1740 10 client.25933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:13:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.974+0000 7f89299c1740 20 client.25933 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.974+0000 7f89299c1740 1 --2- 172.21.15.29:0/4041269910 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5643d575ca10 0x5643d575ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.975+0000 7f89299c1740 4 client.25933 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:13:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.975+0000 7f89299c1740 1 -- 172.21.15.29:0/4041269910 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5643d53d2890 con 0x5643d575ca10 2024-03-11T19:13:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.975+0000 7f891ffff640 1 --2- 172.21.15.29:0/4041269910 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5643d575ca10 0x5643d575ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.975+0000 7f891ffff640 1 --2- 172.21.15.29:0/4041269910 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5643d575ca10 0x5643d575ee30 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bb9d507bbbc2169 server_cookie=bd9d8064e83d9be2 in_seq=0 out_seq=0 2024-03-11T19:13:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.975+0000 7f88fffff640 10 client.25933 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.976+0000 7f88fffff640 1 -- 172.21.15.29:0/4041269910 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5643d53d2890 con 0x5643d575ca10 2024-03-11T19:13:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.976+0000 7f88fffff640 10 client.25933 handle_command_reply: tid=0 2024-03-11T19:13:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.976+0000 7f89299c1740 1 client.25933 shutdown 2024-03-11T19:13:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.976+0000 7f89299c1740 1 -- 172.21.15.29:0/4041269910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56de2d0 msgr2=0x5643d5700520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.976+0000 7f89299c1740 1 --2- 172.21.15.29:0/4041269910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56de2d0 0x5643d5700520 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8914004730 tx=0x7f8914011c40 comp rx=0 tx=0).stop 2024-03-11T19:13:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.976+0000 7f89299c1740 1 -- 172.21.15.29:0/4041269910 shutdown_connections 2024-03-11T19:13:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.976+0000 7f89299c1740 1 --2- 172.21.15.29:0/4041269910 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5643d575ca10 0x5643d575ee30 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.976+0000 7f89299c1740 1 --2- 172.21.15.29:0/4041269910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d56e3e30 0x5643d56ff0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.976+0000 7f88fffff640 0 client.25933 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:51.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.976+0000 7f89299c1740 1 --2- 172.21.15.29:0/4041269910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d56decc0 0x5643d56feba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.976+0000 7f89299c1740 1 --2- 172.21.15.29:0/4041269910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d56de2d0 0x5643d5700520 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.976+0000 7f89299c1740 1 -- 172.21.15.29:0/4041269910 >> 172.21.15.29:0/4041269910 conn(0x5643d56d97f0 msgr2=0x5643d56dffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:51.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.976+0000 7f89299c1740 1 -- 172.21.15.29:0/4041269910 shutdown_connections 2024-03-11T19:13:51.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.977+0000 7f89299c1740 1 -- 172.21.15.29:0/4041269910 wait complete. 2024-03-11T19:13:51.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.977+0000 7f89299c1740 20 client.25933 trim_cache size 0 max 16384 2024-03-11T19:13:51.984 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:13:51.984 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:13:51.984 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:13:51.984 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:13:51.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.978+0000 7f8926d0c640 1 -- 172.21.15.29:0/2534753983 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f89100a19d0 msgr2=0x7f89100a3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:51.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.978+0000 7f8926d0c640 1 --2- 172.21.15.29:0/2534753983 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f89100a19d0 0x7f89100a3e90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x5643d56df520 tx=0x7f89180128b0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.978+0000 7f8926d0c640 1 -- 172.21.15.29:0/2534753983 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8920102370 msgr2=0x7f892019dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:51.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.978+0000 7f8926d0c640 1 --2- 172.21.15.29:0/2534753983 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8920102370 0x7f892019dc20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f890c02c170 tx=0x7f890c004040 comp rx=0 tx=0).stop 2024-03-11T19:13:51.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.978+0000 7f8926d0c640 1 -- 172.21.15.29:0/2534753983 shutdown_connections 2024-03-11T19:13:51.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.978+0000 7f8926d0c640 1 --2- 172.21.15.29:0/2534753983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8920102d30 0x7f89201a4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.978+0000 7f8926d0c640 1 --2- 172.21.15.29:0/2534753983 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8920102370 0x7f892019dc20 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f890c02c170 tx=0x7f890c004040 comp rx=0 tx=0).stop 2024-03-11T19:13:51.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.978+0000 7f8926d0c640 1 --2- 172.21.15.29:0/2534753983 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f89100a19d0 0x7f89100a3e90 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.979+0000 7f8926d0c640 1 --2- 172.21.15.29:0/2534753983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8920101180 0x7f892019d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:51.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.979+0000 7f8926d0c640 1 -- 172.21.15.29:0/2534753983 >> 172.21.15.29:0/2534753983 conn(0x7f89200fc6f0 msgr2=0x7f8920103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:51.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.979+0000 7f8926d0c640 1 -- 172.21.15.29:0/2534753983 shutdown_connections 2024-03-11T19:13:51.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:51.979+0000 7f8926d0c640 1 -- 172.21.15.29:0/2534753983 wait complete. 2024-03-11T19:13:52.002 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:13:52.002 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:13:52.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:51 smithi190 ceph-mon[36292]: pgmap v472: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-11T19:13:52.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.163+0000 7f9972688640 1 -- 172.21.15.29:0/4111358362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c109480 msgr2=0x7f996c111d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:52.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.163+0000 7f9972688640 1 --2- 172.21.15.29:0/4111358362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c109480 0x7f996c111d60 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f9964009ca0 tx=0x7f996402c120 comp rx=0 tx=0).stop 2024-03-11T19:13:52.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.163+0000 7f9972688640 1 -- 172.21.15.29:0/4111358362 shutdown_connections 2024-03-11T19:13:52.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.163+0000 7f9972688640 1 --2- 172.21.15.29:0/4111358362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c109480 0x7f996c111d60 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.163+0000 7f9972688640 1 --2- 172.21.15.29:0/4111358362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f996c108ae0 0x7f996c108f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.163+0000 7f9972688640 1 --2- 172.21.15.29:0/4111358362 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f996c108180 0x7f996c1085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.163+0000 7f9972688640 1 -- 172.21.15.29:0/4111358362 >> 172.21.15.29:0/4111358362 conn(0x7f996c06e390 msgr2=0x7f996c0707b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:52.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.163+0000 7f9972688640 1 -- 172.21.15.29:0/4111358362 shutdown_connections 2024-03-11T19:13:52.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.163+0000 7f9972688640 1 -- 172.21.15.29:0/4111358362 wait complete. 2024-03-11T19:13:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.164+0000 7f9972688640 1 Processor -- start 2024-03-11T19:13:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.164+0000 7f9972688640 1 -- start start 2024-03-11T19:13:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.164+0000 7f9972688640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c108180 0x7f996c117520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.164+0000 7f9972688640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f996c108ae0 0x7f996c117a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.164+0000 7f9972688640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f996c109480 0x7f996c114090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.164+0000 7f9972688640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c114820 con 0x7f996c108180 2024-03-11T19:13:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.164+0000 7f9972688640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c114990 con 0x7f996c108ae0 2024-03-11T19:13:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.164+0000 7f9972688640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c114b50 con 0x7f996c109480 2024-03-11T19:13:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.164+0000 7f996bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c108180 0x7f996c117520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.164+0000 7f9970c25640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f996c109480 0x7f996c114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.164+0000 7f996bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c108180 0x7f996c117520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41228/0 (socket says 172.21.15.29:41228) 2024-03-11T19:13:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.164+0000 7f996bfff640 1 -- 172.21.15.29:0/1050155584 learned_addr learned my addr 172.21.15.29:0/1050155584 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.164+0000 7f9970c25640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f996c109480 0x7f996c114090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:34656/0 (socket says 172.21.15.29:34656) 2024-03-11T19:13:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.164+0000 7f99637fe640 1 --2- 172.21.15.29:0/1050155584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f996c108ae0 0x7f996c117a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.165+0000 7f996bfff640 1 -- 172.21.15.29:0/1050155584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f996c109480 msgr2=0x7f996c114090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.165+0000 7f996bfff640 1 --2- 172.21.15.29:0/1050155584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f996c109480 0x7f996c114090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.165+0000 7f996bfff640 1 -- 172.21.15.29:0/1050155584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f996c108ae0 msgr2=0x7f996c117a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.165+0000 7f996bfff640 1 --2- 172.21.15.29:0/1050155584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f996c108ae0 0x7f996c117a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.165+0000 7f996bfff640 1 -- 172.21.15.29:0/1050155584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9964009900 con 0x7f996c108180 2024-03-11T19:13:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.165+0000 7f9970c25640 1 --2- 172.21.15.29:0/1050155584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f996c109480 0x7f996c114090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.165+0000 7f996bfff640 1 --2- 172.21.15.29:0/1050155584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c108180 0x7f996c117520 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f995c00b450 tx=0x7f995c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.165+0000 7f9969ffb640 1 -- 172.21.15.29:0/1050155584 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995c011890 con 0x7f996c108180 2024-03-11T19:13:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.165+0000 7f9972688640 1 -- 172.21.15.29:0/1050155584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f996c1ae080 con 0x7f996c108180 2024-03-11T19:13:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.165+0000 7f9969ffb640 1 -- 172.21.15.29:0/1050155584 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f995c0103e0 con 0x7f996c108180 2024-03-11T19:13:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.165+0000 7f9969ffb640 1 -- 172.21.15.29:0/1050155584 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995c011ea0 con 0x7f996c108180 2024-03-11T19:13:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.165+0000 7f9972688640 1 -- 172.21.15.29:0/1050155584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f996c1ae660 con 0x7f996c108180 2024-03-11T19:13:52.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.167+0000 7f9972688640 1 -- 172.21.15.29:0/1050155584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f996c110b00 con 0x7f996c108180 2024-03-11T19:13:52.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.168+0000 7f9969ffb640 1 -- 172.21.15.29:0/1050155584 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f995c010550 con 0x7f996c108180 2024-03-11T19:13:52.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.169+0000 7f9969ffb640 1 --2- 172.21.15.29:0/1050155584 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f99580a1ab0 0x7f99580a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.169+0000 7f99637fe640 1 --2- 172.21.15.29:0/1050155584 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f99580a1ab0 0x7f99580a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.169+0000 7f9969ffb640 1 -- 172.21.15.29:0/1050155584 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f995c0cd050 con 0x7f996c108180 2024-03-11T19:13:52.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.169+0000 7f99637fe640 1 --2- 172.21.15.29:0/1050155584 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f99580a1ab0 0x7f99580a3f70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f996c1154a0 tx=0x7f9954005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:52.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.175+0000 7f9969ffb640 1 -- 172.21.15.29:0/1050155584 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f995c096300 con 0x7f996c108180 2024-03-11T19:13:52.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.598+0000 7f9972688640 1 -- 172.21.15.29:0/1050155584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f996c110570 con 0x7f996c108180 2024-03-11T19:13:52.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.599+0000 7f9969ffb640 1 -- 172.21.15.29:0/1050155584 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f995c0999c0 con 0x7f996c108180 2024-03-11T19:13:52.603 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:13:52.603 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:13:52.603 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:13:52.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.603+0000 7f9972688640 1 -- 172.21.15.29:0/1050155584 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f99580a1ab0 msgr2=0x7f99580a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:52.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.603+0000 7f9972688640 1 --2- 172.21.15.29:0/1050155584 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f99580a1ab0 0x7f99580a3f70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f996c1154a0 tx=0x7f9954005f90 comp rx=0 tx=0).stop 2024-03-11T19:13:52.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.603+0000 7f9972688640 1 -- 172.21.15.29:0/1050155584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c108180 msgr2=0x7f996c117520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:52.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.603+0000 7f9972688640 1 --2- 172.21.15.29:0/1050155584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c108180 0x7f996c117520 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f995c00b450 tx=0x7f995c00b910 comp rx=0 tx=0).stop 2024-03-11T19:13:52.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.603+0000 7f9972688640 1 -- 172.21.15.29:0/1050155584 shutdown_connections 2024-03-11T19:13:52.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.603+0000 7f9972688640 1 --2- 172.21.15.29:0/1050155584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f996c109480 0x7f996c114090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.603+0000 7f9972688640 1 --2- 172.21.15.29:0/1050155584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f996c108ae0 0x7f996c117a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.603+0000 7f9972688640 1 --2- 172.21.15.29:0/1050155584 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f99580a1ab0 0x7f99580a3f70 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.603+0000 7f9972688640 1 --2- 172.21.15.29:0/1050155584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c108180 0x7f996c117520 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.603+0000 7f9972688640 1 -- 172.21.15.29:0/1050155584 >> 172.21.15.29:0/1050155584 conn(0x7f996c06e390 msgr2=0x7f996c10dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:52.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.603+0000 7f9972688640 1 -- 172.21.15.29:0/1050155584 shutdown_connections 2024-03-11T19:13:52.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.603+0000 7f9972688640 1 -- 172.21.15.29:0/1050155584 wait complete. 2024-03-11T19:13:52.624 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:13:52.624 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:13:52.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.795+0000 7ffb19ffb640 1 -- 172.21.15.29:0/2558084502 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb140329f0 con 0x7ffb1c0ff9f0 2024-03-11T19:13:52.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.796+0000 7ffb212aa640 1 -- 172.21.15.29:0/2558084502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb1c0ff9f0 msgr2=0x7ffb1c101e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:52.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.796+0000 7ffb212aa640 1 --2- 172.21.15.29:0/2558084502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb1c0ff9f0 0x7ffb1c101e20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ffb14009ca0 tx=0x7ffb1402c100 comp rx=0 tx=0).stop 2024-03-11T19:13:52.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.797+0000 7ffb212aa640 1 -- 172.21.15.29:0/2558084502 shutdown_connections 2024-03-11T19:13:52.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.797+0000 7ffb212aa640 1 --2- 172.21.15.29:0/2558084502 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb1c104d10 0x7ffb1c105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.797+0000 7ffb212aa640 1 --2- 172.21.15.29:0/2558084502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb1c102360 0x7ffb1c1047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.797+0000 7ffb212aa640 1 --2- 172.21.15.29:0/2558084502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb1c0ff9f0 0x7ffb1c101e20 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.797+0000 7ffb212aa640 1 -- 172.21.15.29:0/2558084502 >> 172.21.15.29:0/2558084502 conn(0x7ffb1c0fb610 msgr2=0x7ffb1c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:52.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.798+0000 7ffb212aa640 1 -- 172.21.15.29:0/2558084502 shutdown_connections 2024-03-11T19:13:52.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.798+0000 7ffb212aa640 1 -- 172.21.15.29:0/2558084502 wait complete. 2024-03-11T19:13:52.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.798+0000 7ffb212aa640 1 Processor -- start 2024-03-11T19:13:52.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.799+0000 7ffb212aa640 1 -- start start 2024-03-11T19:13:52.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.799+0000 7ffb212aa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb1c0ff9f0 0x7ffb1c19d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.799+0000 7ffb212aa640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb1c102360 0x7ffb1c19dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.799+0000 7ffb212aa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb1c104d10 0x7ffb1c1a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.799+0000 7ffb212aa640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c1a52a0 con 0x7ffb1c104d10 2024-03-11T19:13:52.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.799+0000 7ffb212aa640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c1a5410 con 0x7ffb1c102360 2024-03-11T19:13:52.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.799+0000 7ffb212aa640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c1a55d0 con 0x7ffb1c0ff9f0 2024-03-11T19:13:52.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.799+0000 7ffb1b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb1c104d10 0x7ffb1c1a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.799+0000 7ffb1affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb1c0ff9f0 0x7ffb1c19d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.799+0000 7ffb1a7fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb1c102360 0x7ffb1c19dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.799+0000 7ffb1b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb1c104d10 0x7ffb1c1a4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41246/0 (socket says 172.21.15.29:41246) 2024-03-11T19:13:52.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.799+0000 7ffb1b7fe640 1 -- 172.21.15.29:0/2247543215 learned_addr learned my addr 172.21.15.29:0/2247543215 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:52.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.800+0000 7ffb1b7fe640 1 -- 172.21.15.29:0/2247543215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb1c0ff9f0 msgr2=0x7ffb1c19d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:52.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.800+0000 7ffb1b7fe640 1 --2- 172.21.15.29:0/2247543215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb1c0ff9f0 0x7ffb1c19d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.800+0000 7ffb1b7fe640 1 -- 172.21.15.29:0/2247543215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb1c102360 msgr2=0x7ffb1c19dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:52.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.800+0000 7ffb1b7fe640 1 --2- 172.21.15.29:0/2247543215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb1c102360 0x7ffb1c19dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.800+0000 7ffb1b7fe640 1 -- 172.21.15.29:0/2247543215 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb14009900 con 0x7ffb1c104d10 2024-03-11T19:13:52.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.800+0000 7ffb1b7fe640 1 --2- 172.21.15.29:0/2247543215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb1c104d10 0x7ffb1c1a4d60 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c00e970 tx=0x7ffb0c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:52.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.800+0000 7ffb03fff640 1 -- 172.21.15.29:0/2247543215 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c009720 con 0x7ffb1c104d10 2024-03-11T19:13:52.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.800+0000 7ffb212aa640 1 -- 172.21.15.29:0/2247543215 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1c1a58b0 con 0x7ffb1c104d10 2024-03-11T19:13:52.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.800+0000 7ffb03fff640 1 -- 172.21.15.29:0/2247543215 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ffb0c0054d0 con 0x7ffb1c104d10 2024-03-11T19:13:52.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.800+0000 7ffb212aa640 1 -- 172.21.15.29:0/2247543215 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1c1a5d70 con 0x7ffb1c104d10 2024-03-11T19:13:52.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.801+0000 7ffb03fff640 1 -- 172.21.15.29:0/2247543215 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c009720 con 0x7ffb1c104d10 2024-03-11T19:13:52.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.804+0000 7ffb03fff640 1 -- 172.21.15.29:0/2247543215 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ffb0c004d30 con 0x7ffb1c104d10 2024-03-11T19:13:52.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.804+0000 7ffb23e7d740 1 Processor -- start 2024-03-11T19:13:52.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.804+0000 7ffb23e7d740 1 -- start start 2024-03-11T19:13:52.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.805+0000 7ffb23e7d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9540270 0x55dfe9540690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.805+0000 7ffb23e7d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9540c60 0x55dfe9545760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.805+0000 7ffb1affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9540c60 0x55dfe9545760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.805+0000 7ffb23e7d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9545dd0 0x55dfe9548260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.805+0000 7ffb23e7d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe94fc960 con 0x55dfe9540c60 2024-03-11T19:13:52.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.805+0000 7ffb23e7d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe94b0310 con 0x55dfe9545dd0 2024-03-11T19:13:52.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.805+0000 7ffb23e7d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe94ad840 con 0x55dfe9540270 2024-03-11T19:13:52.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.805+0000 7ffb1affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9540c60 0x55dfe9545760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41260/0 (socket says 172.21.15.29:41260) 2024-03-11T19:13:52.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.805+0000 7ffb1affd640 1 -- 172.21.15.29:0/3862952021 learned_addr learned my addr 172.21.15.29:0/3862952021 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:52.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.805+0000 7ffb1a7fc640 1 --2- 172.21.15.29:0/3862952021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9540270 0x55dfe9540690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.805+0000 7ffb1a7fc640 1 --2- 172.21.15.29:0/3862952021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9545dd0 0x55dfe9548260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.805+0000 7ffb1affd640 1 -- 172.21.15.29:0/3862952021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9540270 msgr2=0x55dfe9540690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:52.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.805+0000 7ffb1affd640 1 --2- 172.21.15.29:0/3862952021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9540270 0x55dfe9540690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.805+0000 7ffb1affd640 1 -- 172.21.15.29:0/3862952021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9545dd0 msgr2=0x55dfe9548260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:52.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.805+0000 7ffb1affd640 1 --2- 172.21.15.29:0/3862952021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9545dd0 0x55dfe9548260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.805+0000 7ffb1affd640 1 -- 172.21.15.29:0/3862952021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dfe946b060 con 0x55dfe9540c60 2024-03-11T19:13:52.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.806+0000 7ffb1a7fc640 1 --2- 172.21.15.29:0/3862952021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9545dd0 0x55dfe9548260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:13:52.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.806+0000 7ffb03fff640 1 --2- 172.21.15.29:0/2247543215 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ffb040a1ac0 0x7ffb040a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.806+0000 7ffb1affd640 1 --2- 172.21.15.29:0/3862952021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9540c60 0x55dfe9545760 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ffb14009ca0 tx=0x7ffb14009890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88cdead66ab3fd60 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:52.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.806+0000 7ffb1a7fc640 1 --2- 172.21.15.29:0/2247543215 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ffb040a1ac0 0x7ffb040a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.806+0000 7ffb017fa640 1 -- 172.21.15.29:0/3862952021 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb14032340 con 0x55dfe9540c60 2024-03-11T19:13:52.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.806+0000 7ffb017fa640 1 -- 172.21.15.29:0/3862952021 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ffb14032cb0 con 0x55dfe9540c60 2024-03-11T19:13:52.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.806+0000 7ffb017fa640 1 -- 172.21.15.29:0/3862952021 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb14003970 con 0x55dfe9540c60 2024-03-11T19:13:52.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.806+0000 7ffb03fff640 1 -- 172.21.15.29:0/2247543215 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ffb0c014070 con 0x7ffb1c104d10 2024-03-11T19:13:52.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.807+0000 7ffb23e7d740 1 -- 172.21.15.29:0/3862952021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9540c60 msgr2=0x55dfe9545760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:52.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.807+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/3862952021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9540c60 0x55dfe9545760 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ffb14009ca0 tx=0x7ffb14009890 comp rx=0 tx=0).stop 2024-03-11T19:13:52.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.807+0000 7ffb1a7fc640 1 --2- 172.21.15.29:0/2247543215 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ffb040a1ac0 0x7ffb040a3f80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x55dfe95414c0 tx=0x7ffb10013040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:52.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.807+0000 7ffb23e7d740 1 -- 172.21.15.29:0/3862952021 shutdown_connections 2024-03-11T19:13:52.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.807+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/3862952021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9545dd0 0x55dfe9548260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.807+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/3862952021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9540c60 0x55dfe9545760 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.807+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/3862952021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9540270 0x55dfe9540690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.807+0000 7ffb23e7d740 1 -- 172.21.15.29:0/3862952021 >> 172.21.15.29:0/3862952021 conn(0x55dfe953b7e0 msgr2=0x55dfe953bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.807+0000 7ffb23e7d740 1 -- 172.21.15.29:0/3862952021 shutdown_connections 2024-03-11T19:13:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.807+0000 7ffb23e7d740 1 -- 172.21.15.29:0/3862952021 wait complete. 2024-03-11T19:13:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.808+0000 7ffb23e7d740 1 -- start start 2024-03-11T19:13:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.808+0000 7ffb23e7d740 1 Processor -- start 2024-03-11T19:13:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.808+0000 7ffb23e7d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9540270 0x55dfe954a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.808+0000 7ffb23e7d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9540c60 0x55dfe954a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.808+0000 7ffb23e7d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9545dd0 0x55dfe95519a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.808+0000 7ffb1affd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9540270 0x55dfe954a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.808+0000 7ffb1affd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9540270 0x55dfe954a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55264/0 (socket says 172.21.15.29:55264) 2024-03-11T19:13:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.808+0000 7ffb1affd640 1 -- 172.21.15.29:0/3069266250 learned_addr learned my addr 172.21.15.29:0/3069266250 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.808+0000 7ffb1a7fc640 1 --2- 172.21.15.29:0/3069266250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9540c60 0x55dfe954a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.809+0000 7ffb23e7d740 1 -- 172.21.15.29:0/3069266250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe94a9020 con 0x55dfe9545dd0 2024-03-11T19:13:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.809+0000 7ffb1b7fe640 1 --2- 172.21.15.29:0/3069266250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9545dd0 0x55dfe95519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.809+0000 7ffb23e7d740 1 -- 172.21.15.29:0/3069266250 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe917c430 con 0x55dfe9540270 2024-03-11T19:13:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.809+0000 7ffb23e7d740 1 -- 172.21.15.29:0/3069266250 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe9138e00 con 0x55dfe9540c60 2024-03-11T19:13:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.809+0000 7ffb23e7d740 10 client.0 _ll_register_callbacks cb 0x55dfe9450cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.809+0000 7ffb1a7fc640 1 -- 172.21.15.29:0/3069266250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9540270 msgr2=0x55dfe954a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.809+0000 7ffb1a7fc640 1 --2- 172.21.15.29:0/3069266250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9540270 0x55dfe954a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.809+0000 7ffb1a7fc640 1 -- 172.21.15.29:0/3069266250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9545dd0 msgr2=0x55dfe95519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.809+0000 7ffb1a7fc640 1 --2- 172.21.15.29:0/3069266250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9545dd0 0x55dfe95519a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.809+0000 7ffb1a7fc640 1 -- 172.21.15.29:0/3069266250 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1c105c80 con 0x55dfe9540c60 2024-03-11T19:13:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.809+0000 7ffb1b7fe640 1 --2- 172.21.15.29:0/3069266250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9545dd0 0x55dfe95519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.809+0000 7ffb1a7fc640 1 --2- 172.21.15.29:0/3069266250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9540c60 0x55dfe954a960 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c19ecc0 tx=0x7ffb10004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=920b2c54eeecb0eb server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.809+0000 7ffafeff5640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:13:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.810+0000 7ffafeff5640 1 -- 172.21.15.29:0/3069266250 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb10005020 con 0x55dfe9540c60 2024-03-11T19:13:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.810+0000 7ffafeff5640 1 -- 172.21.15.29:0/3069266250 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ffb10005180 con 0x55dfe9540c60 2024-03-11T19:13:52.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.810+0000 7ffb23e7d740 1 -- 172.21.15.29:0/3069266250 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dfe91a87e0 con 0x55dfe9540c60 2024-03-11T19:13:52.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.810+0000 7ffafeff5640 1 -- 172.21.15.29:0/3069266250 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb10005340 con 0x55dfe9540c60 2024-03-11T19:13:52.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.810+0000 7ffb23e7d740 10 client.25951 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:52.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.810+0000 7ffb23e7d740 1 -- 172.21.15.29:0/3069266250 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55dfe94a76a0 con 0x55dfe9540c60 2024-03-11T19:13:52.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.810+0000 7ffafeff5640 1 -- 172.21.15.29:0/3069266250 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55dfe91a87e0 con 0x55dfe9540c60 2024-03-11T19:13:52.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.810+0000 7ffafeff5640 1 -- 172.21.15.29:0/3069266250 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ffb10033400 con 0x55dfe9540c60 2024-03-11T19:13:52.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.811+0000 7ffb23e7d740 10 client.25951 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:52.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.811+0000 7ffb23e7d740 10 client.25951 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:13:52.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.811+0000 7ffb23e7d740 20 client.25951 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:52.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.811+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/3069266250 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55dfe95565e0 0x55dfe9558a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.811+0000 7ffb23e7d740 4 client.25951 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:13:52.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.811+0000 7ffb23e7d740 1 -- 172.21.15.29:0/3069266250 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dfe91e5720 con 0x55dfe95565e0 2024-03-11T19:13:52.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.811+0000 7ffb1affd640 1 --2- 172.21.15.29:0/3069266250 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55dfe95565e0 0x55dfe9558a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.812+0000 7ffb1affd640 1 --2- 172.21.15.29:0/3069266250 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55dfe95565e0 0x55dfe9558a20 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a771d32625f633ee server_cookie=b18c99a2454553c9 in_seq=0 out_seq=0 2024-03-11T19:13:52.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.812+0000 7ffafeff5640 10 client.25951 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:13:52.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.813+0000 7ffafeff5640 1 -- 172.21.15.29:0/3069266250 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55dfe91e5720 con 0x55dfe95565e0 2024-03-11T19:13:52.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.813+0000 7ffafeff5640 10 client.25951 handle_command_reply: tid=0 2024-03-11T19:13:52.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.813+0000 7ffb23e7d740 1 client.25951 shutdown 2024-03-11T19:13:52.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.813+0000 7ffb23e7d740 1 -- 172.21.15.29:0/3069266250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9540c60 msgr2=0x55dfe954a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:52.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.813+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/3069266250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9540c60 0x55dfe954a960 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c19ecc0 tx=0x7ffb10004730 comp rx=0 tx=0).stop 2024-03-11T19:13:52.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.813+0000 7ffb23e7d740 1 -- 172.21.15.29:0/3069266250 shutdown_connections 2024-03-11T19:13:52.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.813+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/3069266250 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55dfe95565e0 0x55dfe9558a20 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.814+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/3069266250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9545dd0 0x55dfe95519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.814+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/3069266250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9540c60 0x55dfe954a960 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.814+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/3069266250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9540270 0x55dfe954a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.814+0000 7ffb23e7d740 1 -- 172.21.15.29:0/3069266250 >> 172.21.15.29:0/3069266250 conn(0x55dfe953b7e0 msgr2=0x55dfe953ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:52.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.814+0000 7ffafeff5640 0 client.25951 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:13:52.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.814+0000 7ffb23e7d740 1 -- 172.21.15.29:0/3069266250 shutdown_connections 2024-03-11T19:13:52.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.814+0000 7ffb23e7d740 1 -- 172.21.15.29:0/3069266250 wait complete. 2024-03-11T19:13:52.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.814+0000 7ffb23e7d740 20 client.25951 trim_cache size 0 max 16384 2024-03-11T19:13:52.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.840+0000 7ffb23e7d740 1 Processor -- start 2024-03-11T19:13:52.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.840+0000 7ffb23e7d740 1 -- start start 2024-03-11T19:13:52.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.841+0000 7ffb23e7d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9540270 0x55dfe95421b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.841+0000 7ffb23e7d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9540c60 0x55dfe95426f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.841+0000 7ffb1affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9540270 0x55dfe95421b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.841+0000 7ffb1affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9540270 0x55dfe95421b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41274/0 (socket says 172.21.15.29:41274) 2024-03-11T19:13:52.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.841+0000 7ffb23e7d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9545dd0 0x55dfe9551960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.841+0000 7ffb23e7d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe942e130 con 0x55dfe9540270 2024-03-11T19:13:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.841+0000 7ffb23e7d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe954a260 con 0x55dfe9540c60 2024-03-11T19:13:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.841+0000 7ffb23e7d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe91a87e0 con 0x55dfe9545dd0 2024-03-11T19:13:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.841+0000 7ffb1affd640 1 -- 172.21.15.29:0/1002392619 learned_addr learned my addr 172.21.15.29:0/1002392619 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.841+0000 7ffb1a7fc640 1 --2- 172.21.15.29:0/1002392619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9540c60 0x55dfe95426f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.841+0000 7ffb1b7fe640 1 --2- 172.21.15.29:0/1002392619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9545dd0 0x55dfe9551960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.842+0000 7ffb1a7fc640 1 -- 172.21.15.29:0/1002392619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9545dd0 msgr2=0x55dfe9551960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.842+0000 7ffb1a7fc640 1 --2- 172.21.15.29:0/1002392619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9545dd0 0x55dfe9551960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.842+0000 7ffb1a7fc640 1 -- 172.21.15.29:0/1002392619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9540270 msgr2=0x55dfe95421b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.842+0000 7ffb1a7fc640 1 --2- 172.21.15.29:0/1002392619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9540270 0x55dfe95421b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.842+0000 7ffb1a7fc640 1 -- 172.21.15.29:0/1002392619 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dfe9443c20 con 0x55dfe9540c60 2024-03-11T19:13:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.842+0000 7ffb1a7fc640 1 --2- 172.21.15.29:0/1002392619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9540c60 0x55dfe95426f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffb10010d00 tx=0x7ffb10010db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f48b45f57512a671 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.842+0000 7ffb00ff9640 1 -- 172.21.15.29:0/1002392619 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1001d790 con 0x55dfe9540c60 2024-03-11T19:13:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.843+0000 7ffb00ff9640 1 -- 172.21.15.29:0/1002392619 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ffb100054d0 con 0x55dfe9540c60 2024-03-11T19:13:52.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.843+0000 7ffb00ff9640 1 -- 172.21.15.29:0/1002392619 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb100047a0 con 0x55dfe9540c60 2024-03-11T19:13:52.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.843+0000 7ffb23e7d740 1 -- 172.21.15.29:0/1002392619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9540c60 msgr2=0x55dfe95426f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:52.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.844+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/1002392619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9540c60 0x55dfe95426f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffb10010d00 tx=0x7ffb10010db0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.844+0000 7ffb23e7d740 1 -- 172.21.15.29:0/1002392619 shutdown_connections 2024-03-11T19:13:52.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.844+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/1002392619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9545dd0 0x55dfe9551960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.844+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/1002392619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9540c60 0x55dfe95426f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.844+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/1002392619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9540270 0x55dfe95421b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.844+0000 7ffb23e7d740 1 -- 172.21.15.29:0/1002392619 >> 172.21.15.29:0/1002392619 conn(0x55dfe953b7e0 msgr2=0x55dfe955d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:52.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.844+0000 7ffb23e7d740 1 -- 172.21.15.29:0/1002392619 shutdown_connections 2024-03-11T19:13:52.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.844+0000 7ffb23e7d740 1 -- 172.21.15.29:0/1002392619 wait complete. 2024-03-11T19:13:52.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.844+0000 7ffb23e7d740 1 -- start start 2024-03-11T19:13:52.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.845+0000 7ffb23e7d740 1 Processor -- start 2024-03-11T19:13:52.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.845+0000 7ffb23e7d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9540270 0x55dfe9551430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.845+0000 7ffb23e7d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9540c60 0x55dfe9551970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.845+0000 7ffb23e7d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9545dd0 0x55dfe954faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.845+0000 7ffb1affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9540270 0x55dfe9551430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.845+0000 7ffb23e7d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe95501a0 con 0x55dfe9540c60 2024-03-11T19:13:52.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.845+0000 7ffb1affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9540270 0x55dfe9551430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:34690/0 (socket says 172.21.15.29:34690) 2024-03-11T19:13:52.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.845+0000 7ffb1affd640 1 -- 172.21.15.29:0/2292205961 learned_addr learned my addr 172.21.15.29:0/2292205961 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:52.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.845+0000 7ffb23e7d740 1 -- 172.21.15.29:0/2292205961 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe9550310 con 0x55dfe9545dd0 2024-03-11T19:13:52.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.845+0000 7ffb23e7d740 1 -- 172.21.15.29:0/2292205961 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe95504d0 con 0x55dfe9540270 2024-03-11T19:13:52.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.846+0000 7ffb1a7fc640 1 --2- 172.21.15.29:0/2292205961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9540c60 0x55dfe9551970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.846+0000 7ffb23e7d740 10 client.0 _ll_register_callbacks cb 0x7ffb0c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:52.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.846+0000 7ffb1b7fe640 1 --2- 172.21.15.29:0/2292205961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9545dd0 0x55dfe954faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.846+0000 7ffb1a7fc640 1 -- 172.21.15.29:0/2292205961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9540270 msgr2=0x55dfe9551430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:52.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.846+0000 7ffb1a7fc640 1 --2- 172.21.15.29:0/2292205961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9540270 0x55dfe9551430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.846+0000 7ffb1a7fc640 1 -- 172.21.15.29:0/2292205961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9545dd0 msgr2=0x55dfe954faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:52.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.846+0000 7ffb1a7fc640 1 --2- 172.21.15.29:0/2292205961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9545dd0 0x55dfe954faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.846+0000 7ffb1a7fc640 1 -- 172.21.15.29:0/2292205961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dfe94a76a0 con 0x55dfe9540c60 2024-03-11T19:13:52.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.846+0000 7ffb1b7fe640 1 --2- 172.21.15.29:0/2292205961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9545dd0 0x55dfe954faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:13:52.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.846+0000 7ffb1a7fc640 1 --2- 172.21.15.29:0/2292205961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9540c60 0x55dfe9551970 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ffb10033730 tx=0x7ffb1001df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d3f9e67428ef2f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:52.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.846+0000 7ffafeff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:13:52.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.847+0000 7ffafeff5640 1 -- 172.21.15.29:0/2292205961 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb10005640 con 0x55dfe9540c60 2024-03-11T19:13:52.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.847+0000 7ffafeff5640 1 -- 172.21.15.29:0/2292205961 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ffb1002f030 con 0x55dfe9540c60 2024-03-11T19:13:52.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.847+0000 7ffb23e7d740 1 -- 172.21.15.29:0/2292205961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb14032cb0 con 0x55dfe9540c60 2024-03-11T19:13:52.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.847+0000 7ffafeff5640 1 -- 172.21.15.29:0/2292205961 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb10033400 con 0x55dfe9540c60 2024-03-11T19:13:52.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.847+0000 7ffb23e7d740 10 client.16416 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:52.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.847+0000 7ffb23e7d740 1 -- 172.21.15.29:0/2292205961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55dfe94698c0 con 0x55dfe9540c60 2024-03-11T19:13:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.848+0000 7ffafeff5640 1 -- 172.21.15.29:0/2292205961 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ffb1004b020 con 0x55dfe9540c60 2024-03-11T19:13:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.848+0000 7ffafeff5640 1 -- 172.21.15.29:0/2292205961 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ffb1003c750 con 0x55dfe9540c60 2024-03-11T19:13:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.848+0000 7ffb23e7d740 10 client.16416 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.848+0000 7ffb23e7d740 10 client.16416 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:13:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.848+0000 7ffb23e7d740 20 client.16416 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.848+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/2292205961 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55dfe95adbe0 0x55dfe95b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.848+0000 7ffb23e7d740 4 client.16416 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:13:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.848+0000 7ffb23e7d740 1 -- 172.21.15.29:0/2292205961 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7ffb14032cb0 con 0x55dfe95adbe0 2024-03-11T19:13:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.848+0000 7ffb1affd640 1 --2- 172.21.15.29:0/2292205961 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55dfe95adbe0 0x55dfe95b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.849+0000 7ffafeff5640 10 client.16416 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:13:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.849+0000 7ffb1affd640 1 --2- 172.21.15.29:0/2292205961 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55dfe95adbe0 0x55dfe95b0000 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=772b93c5bf38cbb8 server_cookie=c397d61fbc795cb9 in_seq=0 out_seq=0 2024-03-11T19:13:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.849+0000 7ffafeff5640 1 -- 172.21.15.29:0/2292205961 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7ffb14032cb0 con 0x55dfe95adbe0 2024-03-11T19:13:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.849+0000 7ffafeff5640 10 client.16416 handle_command_reply: tid=0 2024-03-11T19:13:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.849+0000 7ffb23e7d740 1 client.16416 shutdown 2024-03-11T19:13:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.849+0000 7ffb23e7d740 1 -- 172.21.15.29:0/2292205961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9540c60 msgr2=0x55dfe9551970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:52.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.849+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/2292205961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9540c60 0x55dfe9551970 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ffb10033730 tx=0x7ffb1001df90 comp rx=0 tx=0).stop 2024-03-11T19:13:52.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.849+0000 7ffb23e7d740 1 -- 172.21.15.29:0/2292205961 shutdown_connections 2024-03-11T19:13:52.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.849+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/2292205961 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55dfe95adbe0 0x55dfe95b0000 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.850+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/2292205961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dfe9545dd0 0x55dfe954faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.850+0000 7ffafeff5640 0 client.16416 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:13:52.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.850+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/2292205961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dfe9540c60 0x55dfe9551970 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.850+0000 7ffb23e7d740 1 --2- 172.21.15.29:0/2292205961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dfe9540270 0x55dfe9551430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.850+0000 7ffb23e7d740 1 -- 172.21.15.29:0/2292205961 >> 172.21.15.29:0/2292205961 conn(0x55dfe953b7e0 msgr2=0x55dfe95421b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:52.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.850+0000 7ffb23e7d740 1 -- 172.21.15.29:0/2292205961 shutdown_connections 2024-03-11T19:13:52.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.850+0000 7ffb23e7d740 1 -- 172.21.15.29:0/2292205961 wait complete. 2024-03-11T19:13:52.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.850+0000 7ffb23e7d740 20 client.16416 trim_cache size 0 max 16384 2024-03-11T19:13:52.856 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:13:52.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.851+0000 7ffb212aa640 1 -- 172.21.15.29:0/2247543215 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ffb040a1ac0 msgr2=0x7ffb040a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:52.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.851+0000 7ffb212aa640 1 --2- 172.21.15.29:0/2247543215 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ffb040a1ac0 0x7ffb040a3f80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x55dfe95414c0 tx=0x7ffb10013040 comp rx=0 tx=0).stop 2024-03-11T19:13:52.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.851+0000 7ffb212aa640 1 -- 172.21.15.29:0/2247543215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb1c104d10 msgr2=0x7ffb1c1a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:52.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.851+0000 7ffb212aa640 1 --2- 172.21.15.29:0/2247543215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb1c104d10 0x7ffb1c1a4d60 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c00e970 tx=0x7ffb0c00ed70 comp rx=0 tx=0).stop 2024-03-11T19:13:52.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.851+0000 7ffb212aa640 1 -- 172.21.15.29:0/2247543215 shutdown_connections 2024-03-11T19:13:52.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.851+0000 7ffb212aa640 1 --2- 172.21.15.29:0/2247543215 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ffb040a1ac0 0x7ffb040a3f80 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.851+0000 7ffb212aa640 1 --2- 172.21.15.29:0/2247543215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb1c104d10 0x7ffb1c1a4d60 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.851+0000 7ffb212aa640 1 --2- 172.21.15.29:0/2247543215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb1c102360 0x7ffb1c19dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.852+0000 7ffb212aa640 1 --2- 172.21.15.29:0/2247543215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb1c0ff9f0 0x7ffb1c19d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:52.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.852+0000 7ffb212aa640 1 -- 172.21.15.29:0/2247543215 >> 172.21.15.29:0/2247543215 conn(0x7ffb1c0fb610 msgr2=0x7ffb1c102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:52.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.852+0000 7ffb212aa640 1 -- 172.21.15.29:0/2247543215 shutdown_connections 2024-03-11T19:13:52.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:52.852+0000 7ffb212aa640 1 -- 172.21.15.29:0/2247543215 wait complete. 2024-03-11T19:13:52.873 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:13:52.873 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:13:53.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.038+0000 7f2f2573a640 1 -- 172.21.15.29:0/1709216174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f20100080 msgr2=0x7f2f201004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.038+0000 7f2f2573a640 1 --2- 172.21.15.29:0/1709216174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f20100080 0x7f2f201004a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2f10009900 tx=0x7f2f1002bc40 comp rx=0 tx=0).stop 2024-03-11T19:13:53.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.039+0000 7f2f2573a640 1 -- 172.21.15.29:0/1709216174 shutdown_connections 2024-03-11T19:13:53.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.039+0000 7f2f2573a640 1 --2- 172.21.15.29:0/1709216174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f20101c30 0x7f2f201084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.039+0000 7f2f2573a640 1 --2- 172.21.15.29:0/1709216174 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f20101270 0x7f2f201016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.039+0000 7f2f2573a640 1 --2- 172.21.15.29:0/1709216174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f20100080 0x7f2f201004a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.039+0000 7f2f2573a640 1 -- 172.21.15.29:0/1709216174 >> 172.21.15.29:0/1709216174 conn(0x7f2f200fb610 msgr2=0x7f2f200fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.039+0000 7f2f2573a640 1 -- 172.21.15.29:0/1709216174 shutdown_connections 2024-03-11T19:13:53.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.039+0000 7f2f2573a640 1 -- 172.21.15.29:0/1709216174 wait complete. 2024-03-11T19:13:53.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.040+0000 7f2f2573a640 1 Processor -- start 2024-03-11T19:13:53.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.040+0000 7f2f2573a640 1 -- start start 2024-03-11T19:13:53.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.040+0000 7f2f2573a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f20100080 0x7f2f20199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.040+0000 7f2f2573a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f20101270 0x7f2f20199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.040+0000 7f2f2573a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f20101c30 0x7f2f201a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.040+0000 7f2f2573a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f201a0f10 con 0x7f2f20100080 2024-03-11T19:13:53.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.040+0000 7f2f2573a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f201a1080 con 0x7f2f20101270 2024-03-11T19:13:53.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.040+0000 7f2f2573a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f201a1240 con 0x7f2f20101c30 2024-03-11T19:13:53.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.040+0000 7f2f1effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f20100080 0x7f2f20199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.041+0000 7f2f1effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f20100080 0x7f2f20199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41292/0 (socket says 172.21.15.29:41292) 2024-03-11T19:13:53.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.041+0000 7f2f1effd640 1 -- 172.21.15.29:0/2792112615 learned_addr learned my addr 172.21.15.29:0/2792112615 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.041+0000 7f2f1f7fe640 1 --2- 172.21.15.29:0/2792112615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f20101c30 0x7f2f201a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.041+0000 7f2f1e7fc640 1 --2- 172.21.15.29:0/2792112615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f20101270 0x7f2f20199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.041+0000 7f2f1effd640 1 -- 172.21.15.29:0/2792112615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f20101c30 msgr2=0x7f2f201a09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.041+0000 7f2f1effd640 1 --2- 172.21.15.29:0/2792112615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f20101c30 0x7f2f201a09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.041+0000 7f2f1effd640 1 -- 172.21.15.29:0/2792112615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f20101270 msgr2=0x7f2f20199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.041+0000 7f2f1effd640 1 --2- 172.21.15.29:0/2792112615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f20101270 0x7f2f20199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.041+0000 7f2f1effd640 1 -- 172.21.15.29:0/2792112615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f14009490 con 0x7f2f20100080 2024-03-11T19:13:53.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.041+0000 7f2f1f7fe640 1 --2- 172.21.15.29:0/2792112615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f20101c30 0x7f2f201a09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:53.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.041+0000 7f2f1effd640 1 --2- 172.21.15.29:0/2792112615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f20100080 0x7f2f20199420 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f2f1002c7e0 tx=0x7f2f100042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.042+0000 7f2f07fff640 1 -- 172.21.15.29:0/2792112615 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1002dea0 con 0x7f2f20100080 2024-03-11T19:13:53.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.042+0000 7f2f07fff640 1 -- 172.21.15.29:0/2792112615 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2f10004aa0 con 0x7f2f20100080 2024-03-11T19:13:53.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.042+0000 7f2f07fff640 1 -- 172.21.15.29:0/2792112615 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f10005440 con 0x7f2f20100080 2024-03-11T19:13:53.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.042+0000 7f2f2573a640 1 -- 172.21.15.29:0/2792112615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f10009560 con 0x7f2f20100080 2024-03-11T19:13:53.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.042+0000 7f2f2573a640 1 -- 172.21.15.29:0/2792112615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f201a17f0 con 0x7f2f20100080 2024-03-11T19:13:53.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.045+0000 7f2f07fff640 1 -- 172.21.15.29:0/2792112615 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2f100055a0 con 0x7f2f20100080 2024-03-11T19:13:53.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.045+0000 7f2f283e3740 1 Processor -- start 2024-03-11T19:13:53.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.045+0000 7f2f283e3740 1 -- start start 2024-03-11T19:13:53.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.046+0000 7f2f283e3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de5a80 0x55b3f9de5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.046+0000 7f2f283e3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9de6470 0x55b3f9deaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.046+0000 7f2f283e3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9deb5e0 0x55b3f9deda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.046+0000 7f2f283e3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3f9d99960 con 0x55b3f9deb5e0 2024-03-11T19:13:53.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.046+0000 7f2f283e3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3f9d4d310 con 0x55b3f9de6470 2024-03-11T19:13:53.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.046+0000 7f2f283e3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3f9d4a840 con 0x55b3f9de5a80 2024-03-11T19:13:53.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.046+0000 7f2f1e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9deb5e0 0x55b3f9deda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.046+0000 7f2f1effd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9de6470 0x55b3f9deaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.046+0000 7f2f1e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9deb5e0 0x55b3f9deda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41306/0 (socket says 172.21.15.29:41306) 2024-03-11T19:13:53.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.046+0000 7f2f1e7fc640 1 -- 172.21.15.29:0/1741497169 learned_addr learned my addr 172.21.15.29:0/1741497169 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.046+0000 7f2f1f7fe640 1 --2- 172.21.15.29:0/1741497169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de5a80 0x55b3f9de5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.046+0000 7f2f1e7fc640 1 -- 172.21.15.29:0/1741497169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de5a80 msgr2=0x55b3f9de5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.046+0000 7f2f1e7fc640 1 --2- 172.21.15.29:0/1741497169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de5a80 0x55b3f9de5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.046+0000 7f2f1e7fc640 1 -- 172.21.15.29:0/1741497169 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9de6470 msgr2=0x55b3f9deaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.046+0000 7f2f1e7fc640 1 --2- 172.21.15.29:0/1741497169 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9de6470 0x55b3f9deaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.046+0000 7f2f1e7fc640 1 -- 172.21.15.29:0/1741497169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b3f9d08060 con 0x55b3f9deb5e0 2024-03-11T19:13:53.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.046+0000 7f2f1f7fe640 1 --2- 172.21.15.29:0/1741497169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de5a80 0x55b3f9de5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:13:53.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.047+0000 7f2f1e7fc640 1 --2- 172.21.15.29:0/1741497169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9deb5e0 0x55b3f9deda70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f2f14009e70 tx=0x7f2f14011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a39045bc1b66114f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.047+0000 7f2f07fff640 1 --2- 172.21.15.29:0/2792112615 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2f080a1ab0 0x7f2f080a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.047+0000 7f2f057fa640 1 -- 172.21.15.29:0/1741497169 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1400e470 con 0x55b3f9deb5e0 2024-03-11T19:13:53.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.047+0000 7f2f057fa640 1 -- 172.21.15.29:0/1741497169 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2f1400ea80 con 0x55b3f9deb5e0 2024-03-11T19:13:53.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.047+0000 7f2f057fa640 1 -- 172.21.15.29:0/1741497169 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f14019ae0 con 0x55b3f9deb5e0 2024-03-11T19:13:53.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.048+0000 7f2f07fff640 1 -- 172.21.15.29:0/2792112615 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f2f100ef480 con 0x7f2f20100080 2024-03-11T19:13:53.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.048+0000 7f2f1f7fe640 1 --2- 172.21.15.29:0/2792112615 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2f080a1ab0 0x7f2f080a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.048+0000 7f2f1f7fe640 1 --2- 172.21.15.29:0/2792112615 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2f080a1ab0 0x7f2f080a3f70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x55b3f9de6cd0 tx=0x7f2f180096a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.048+0000 7f2f283e3740 1 -- 172.21.15.29:0/1741497169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9deb5e0 msgr2=0x55b3f9deda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.048+0000 7f2f283e3740 1 --2- 172.21.15.29:0/1741497169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9deb5e0 0x55b3f9deda70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f2f14009e70 tx=0x7f2f14011810 comp rx=0 tx=0).stop 2024-03-11T19:13:53.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.048+0000 7f2f283e3740 1 -- 172.21.15.29:0/1741497169 shutdown_connections 2024-03-11T19:13:53.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.048+0000 7f2f283e3740 1 --2- 172.21.15.29:0/1741497169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9deb5e0 0x55b3f9deda70 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.049+0000 7f2f283e3740 1 --2- 172.21.15.29:0/1741497169 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9de6470 0x55b3f9deaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.049+0000 7f2f283e3740 1 --2- 172.21.15.29:0/1741497169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de5a80 0x55b3f9de5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.049+0000 7f2f283e3740 1 -- 172.21.15.29:0/1741497169 >> 172.21.15.29:0/1741497169 conn(0x55b3f9de0ff0 msgr2=0x55b3f9de13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.049+0000 7f2f283e3740 1 -- 172.21.15.29:0/1741497169 shutdown_connections 2024-03-11T19:13:53.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.049+0000 7f2f283e3740 1 -- 172.21.15.29:0/1741497169 wait complete. 2024-03-11T19:13:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.049+0000 7f2f283e3740 1 -- start start 2024-03-11T19:13:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.049+0000 7f2f283e3740 1 Processor -- start 2024-03-11T19:13:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.050+0000 7f2f283e3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9de5a80 0x55b3f9defcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.050+0000 7f2f283e3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de6470 0x55b3f9df0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.050+0000 7f2f283e3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9deb5e0 0x55b3f9df7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.050+0000 7f2f283e3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3f9d46020 con 0x55b3f9deb5e0 2024-03-11T19:13:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.050+0000 7f2f283e3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3f9a19430 con 0x55b3f9de5a80 2024-03-11T19:13:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.050+0000 7f2f283e3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3f99d5e00 con 0x55b3f9de6470 2024-03-11T19:13:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.050+0000 7f2f283e3740 10 client.0 _ll_register_callbacks cb 0x55b3f9cedcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:53.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.050+0000 7f2f1effd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9de5a80 0x55b3f9defcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.050+0000 7f2f1effd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9de5a80 0x55b3f9defcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55308/0 (socket says 172.21.15.29:55308) 2024-03-11T19:13:53.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.050+0000 7f2f1effd640 1 -- 172.21.15.29:0/2040863873 learned_addr learned my addr 172.21.15.29:0/2040863873 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.050+0000 7f2f1f7fe640 1 --2- 172.21.15.29:0/2040863873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9deb5e0 0x55b3f9df7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.050+0000 7f2f1e7fc640 1 --2- 172.21.15.29:0/2040863873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de6470 0x55b3f9df0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.050+0000 7f2f1effd640 1 -- 172.21.15.29:0/2040863873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de6470 msgr2=0x55b3f9df0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.050+0000 7f2f1effd640 1 --2- 172.21.15.29:0/2040863873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de6470 0x55b3f9df0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.050+0000 7f2f1effd640 1 -- 172.21.15.29:0/2040863873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9deb5e0 msgr2=0x55b3f9df7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.050+0000 7f2f1effd640 1 --2- 172.21.15.29:0/2040863873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9deb5e0 0x55b3f9df7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.050+0000 7f2f1effd640 1 -- 172.21.15.29:0/2040863873 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f14009a80 con 0x55b3f9de5a80 2024-03-11T19:13:53.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.051+0000 7f2f1effd640 1 --2- 172.21.15.29:0/2040863873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9de5a80 0x55b3f9defcf0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2f100b8850 tx=0x7f2f10005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d7d25d2908a5515 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.051+0000 7f2f02ff5640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:13:53.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.051+0000 7f2f02ff5640 1 -- 172.21.15.29:0/2040863873 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f100c4900 con 0x55b3f9de5a80 2024-03-11T19:13:53.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.051+0000 7f2f283e3740 1 -- 172.21.15.29:0/2040863873 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b3f9a457e0 con 0x55b3f9de5a80 2024-03-11T19:13:53.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.051+0000 7f2f02ff5640 1 -- 172.21.15.29:0/2040863873 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f2f10004dc0 con 0x55b3f9de5a80 2024-03-11T19:13:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.051+0000 7f2f02ff5640 1 -- 172.21.15.29:0/2040863873 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f100bbf30 con 0x55b3f9de5a80 2024-03-11T19:13:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.051+0000 7f2f283e3740 10 client.25937 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.051+0000 7f2f283e3740 1 -- 172.21.15.29:0/2040863873 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55b3f9d446a0 con 0x55b3f9de5a80 2024-03-11T19:13:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.052+0000 7f2f02ff5640 1 -- 172.21.15.29:0/2040863873 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55b3f9a457e0 con 0x55b3f9de5a80 2024-03-11T19:13:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.052+0000 7f2f02ff5640 1 -- 172.21.15.29:0/2040863873 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2f1003c070 con 0x55b3f9de5a80 2024-03-11T19:13:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.052+0000 7f2f283e3740 10 client.25937 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.052+0000 7f2f283e3740 10 client.25937 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:13:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.052+0000 7f2f283e3740 20 client.25937 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.052+0000 7f2f283e3740 1 --2- 172.21.15.29:0/2040863873 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55b3f9dfbc80 0x55b3f9dfe0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.052+0000 7f2f283e3740 4 client.25937 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:13:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.052+0000 7f2f283e3740 1 -- 172.21.15.29:0/2040863873 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b3f9a82720 con 0x55b3f9dfbc80 2024-03-11T19:13:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.052+0000 7f2f1e7fc640 1 --2- 172.21.15.29:0/2040863873 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55b3f9dfbc80 0x55b3f9dfe0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.053+0000 7f2f02ff5640 10 client.25937 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:13:53.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.053+0000 7f2f1e7fc640 1 --2- 172.21.15.29:0/2040863873 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55b3f9dfbc80 0x55b3f9dfe0a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6bff0930b96e308a server_cookie=a2e57165ff5cadc4 in_seq=0 out_seq=0 2024-03-11T19:13:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.054+0000 7f2f02ff5640 1 -- 172.21.15.29:0/2040863873 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b3f9a82720 con 0x55b3f9dfbc80 2024-03-11T19:13:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.054+0000 7f2f02ff5640 10 client.25937 handle_command_reply: tid=0 2024-03-11T19:13:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.054+0000 7f2f283e3740 1 client.25937 shutdown 2024-03-11T19:13:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.054+0000 7f2f283e3740 1 -- 172.21.15.29:0/2040863873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9de5a80 msgr2=0x55b3f9defcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.054+0000 7f2f283e3740 1 --2- 172.21.15.29:0/2040863873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9de5a80 0x55b3f9defcf0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2f100b8850 tx=0x7f2f10005b50 comp rx=0 tx=0).stop 2024-03-11T19:13:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.054+0000 7f2f283e3740 1 -- 172.21.15.29:0/2040863873 shutdown_connections 2024-03-11T19:13:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.054+0000 7f2f283e3740 1 --2- 172.21.15.29:0/2040863873 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55b3f9dfbc80 0x55b3f9dfe0a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.054+0000 7f2f283e3740 1 --2- 172.21.15.29:0/2040863873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9deb5e0 0x55b3f9df7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.054+0000 7f2f283e3740 1 --2- 172.21.15.29:0/2040863873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de6470 0x55b3f9df0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.054+0000 7f2f283e3740 1 --2- 172.21.15.29:0/2040863873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9de5a80 0x55b3f9defcf0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.054+0000 7f2f283e3740 1 -- 172.21.15.29:0/2040863873 >> 172.21.15.29:0/2040863873 conn(0x55b3f9de0ff0 msgr2=0x55b3f9de2240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.054+0000 7f2f02ff5640 0 client.25937 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:13:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.054+0000 7f2f283e3740 1 -- 172.21.15.29:0/2040863873 shutdown_connections 2024-03-11T19:13:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.055+0000 7f2f283e3740 1 -- 172.21.15.29:0/2040863873 wait complete. 2024-03-11T19:13:53.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.055+0000 7f2f283e3740 20 client.25937 trim_cache size 0 max 16384 2024-03-11T19:13:53.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.074+0000 7f2f283e3740 1 Processor -- start 2024-03-11T19:13:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.074+0000 7f2f283e3740 1 -- start start 2024-03-11T19:13:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.074+0000 7f2f283e3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de5a80 0x55b3f9de79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.074+0000 7f2f283e3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9de6470 0x55b3f9de7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.074+0000 7f2f283e3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9deb5e0 0x55b3f9df7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.075+0000 7f2f283e3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3f9ccb130 con 0x55b3f9de6470 2024-03-11T19:13:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.075+0000 7f2f283e3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3f9a82720 con 0x55b3f9deb5e0 2024-03-11T19:13:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.075+0000 7f2f283e3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3f9a457e0 con 0x55b3f9de5a80 2024-03-11T19:13:53.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.075+0000 7f2f1effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de5a80 0x55b3f9de79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.075+0000 7f2f1f7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9deb5e0 0x55b3f9df7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.075+0000 7f2f1e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9de6470 0x55b3f9de7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.075+0000 7f2f1f7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9deb5e0 0x55b3f9df7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55322/0 (socket says 172.21.15.29:55322) 2024-03-11T19:13:53.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.075+0000 7f2f1f7fe640 1 -- 172.21.15.29:0/3193256001 learned_addr learned my addr 172.21.15.29:0/3193256001 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.075+0000 7f2f1effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de5a80 0x55b3f9de79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:34748/0 (socket says 172.21.15.29:34748) 2024-03-11T19:13:53.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.075+0000 7f2f1f7fe640 1 -- 172.21.15.29:0/3193256001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de5a80 msgr2=0x55b3f9de79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.075+0000 7f2f1f7fe640 1 --2- 172.21.15.29:0/3193256001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de5a80 0x55b3f9de79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.075+0000 7f2f1f7fe640 1 -- 172.21.15.29:0/3193256001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9de6470 msgr2=0x55b3f9de7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.075+0000 7f2f1f7fe640 1 --2- 172.21.15.29:0/3193256001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9de6470 0x55b3f9de7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.075+0000 7f2f1f7fe640 1 -- 172.21.15.29:0/3193256001 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b3f9ce0c20 con 0x55b3f9deb5e0 2024-03-11T19:13:53.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.076+0000 7f2f1f7fe640 1 --2- 172.21.15.29:0/3193256001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9deb5e0 0x55b3f9df7200 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2f18011c40 tx=0x7f2f18014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87c2cd076d3c7178 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.076+0000 7f2f04ff9640 1 -- 172.21.15.29:0/3193256001 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f18023070 con 0x55b3f9deb5e0 2024-03-11T19:13:53.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.076+0000 7f2f04ff9640 1 -- 172.21.15.29:0/3193256001 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2f18016070 con 0x55b3f9deb5e0 2024-03-11T19:13:53.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.076+0000 7f2f04ff9640 1 -- 172.21.15.29:0/3193256001 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1801eb30 con 0x55b3f9deb5e0 2024-03-11T19:13:53.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.077+0000 7f2f283e3740 1 -- 172.21.15.29:0/3193256001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9deb5e0 msgr2=0x55b3f9df7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.077+0000 7f2f283e3740 1 --2- 172.21.15.29:0/3193256001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9deb5e0 0x55b3f9df7200 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2f18011c40 tx=0x7f2f18014000 comp rx=0 tx=0).stop 2024-03-11T19:13:53.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.077+0000 7f2f283e3740 1 -- 172.21.15.29:0/3193256001 shutdown_connections 2024-03-11T19:13:53.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.077+0000 7f2f283e3740 1 --2- 172.21.15.29:0/3193256001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9deb5e0 0x55b3f9df7200 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.077+0000 7f2f283e3740 1 --2- 172.21.15.29:0/3193256001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9de6470 0x55b3f9de7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.077+0000 7f2f283e3740 1 --2- 172.21.15.29:0/3193256001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de5a80 0x55b3f9de79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.077+0000 7f2f283e3740 1 -- 172.21.15.29:0/3193256001 >> 172.21.15.29:0/3193256001 conn(0x55b3f9de0ff0 msgr2=0x55b3f9e02a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.077+0000 7f2f283e3740 1 -- 172.21.15.29:0/3193256001 shutdown_connections 2024-03-11T19:13:53.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.077+0000 7f2f283e3740 1 -- 172.21.15.29:0/3193256001 wait complete. 2024-03-11T19:13:53.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.077+0000 7f2f283e3740 1 -- start start 2024-03-11T19:13:53.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.077+0000 7f2f283e3740 1 Processor -- start 2024-03-11T19:13:53.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f283e3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de5a80 0x55b3f9df6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f283e3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9de6470 0x55b3f9df5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f283e3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9deb5e0 0x55b3f9df59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f283e3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3f9df75f0 con 0x55b3f9de6470 2024-03-11T19:13:53.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f283e3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3f9df5f30 con 0x55b3f9deb5e0 2024-03-11T19:13:53.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f283e3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3f9df60f0 con 0x55b3f9de5a80 2024-03-11T19:13:53.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f283e3740 10 client.0 _ll_register_callbacks cb 0x7f2f14009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:53.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f1effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de5a80 0x55b3f9df6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f1e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9de6470 0x55b3f9df5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f1effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de5a80 0x55b3f9df6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:34750/0 (socket says 172.21.15.29:34750) 2024-03-11T19:13:53.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f1effd640 1 -- 172.21.15.29:0/2046973713 learned_addr learned my addr 172.21.15.29:0/2046973713 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f1f7fe640 1 --2- 172.21.15.29:0/2046973713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9deb5e0 0x55b3f9df59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f1effd640 1 -- 172.21.15.29:0/2046973713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9deb5e0 msgr2=0x55b3f9df59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f1effd640 1 --2- 172.21.15.29:0/2046973713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9deb5e0 0x55b3f9df59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f1effd640 1 -- 172.21.15.29:0/2046973713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9de6470 msgr2=0x55b3f9df5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f1effd640 1 --2- 172.21.15.29:0/2046973713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9de6470 0x55b3f9df5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f1effd640 1 -- 172.21.15.29:0/2046973713 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f180118f0 con 0x55b3f9de5a80 2024-03-11T19:13:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f1e7fc640 1 --2- 172.21.15.29:0/2046973713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9de6470 0x55b3f9df5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f1effd640 1 --2- 172.21.15.29:0/2046973713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de5a80 0x55b3f9df6e00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2f100bc380 tx=0x7f2f100a5dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c1a8917b3d44e03 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.078+0000 7f2f02ff5640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:13:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.079+0000 7f2f02ff5640 1 -- 172.21.15.29:0/2046973713 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f100c0170 con 0x55b3f9de5a80 2024-03-11T19:13:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.079+0000 7f2f02ff5640 1 -- 172.21.15.29:0/2046973713 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f2f10003e20 con 0x55b3f9de5a80 2024-03-11T19:13:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.079+0000 7f2f283e3740 1 -- 172.21.15.29:0/2046973713 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2f1400ea80 con 0x55b3f9de5a80 2024-03-11T19:13:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.079+0000 7f2f02ff5640 1 -- 172.21.15.29:0/2046973713 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f100bbf30 con 0x55b3f9de5a80 2024-03-11T19:13:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.079+0000 7f2f283e3740 10 client.25972 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.079+0000 7f2f283e3740 1 -- 172.21.15.29:0/2046973713 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55b3f9d068c0 con 0x55b3f9de5a80 2024-03-11T19:13:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.079+0000 7f2f02ff5640 1 -- 172.21.15.29:0/2046973713 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2f100bbd50 con 0x55b3f9de5a80 2024-03-11T19:13:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.080+0000 7f2f02ff5640 1 -- 172.21.15.29:0/2046973713 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2f1003c040 con 0x55b3f9de5a80 2024-03-11T19:13:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.080+0000 7f2f283e3740 10 client.25972 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.080+0000 7f2f283e3740 10 client.25972 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:13:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.080+0000 7f2f283e3740 20 client.25972 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:53.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.080+0000 7f2f283e3740 1 --2- 172.21.15.29:0/2046973713 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55b3f9e53400 0x55b3f9e55820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.080+0000 7f2f283e3740 4 client.25972 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:13:53.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.080+0000 7f2f283e3740 1 -- 172.21.15.29:0/2046973713 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f2f1400ea80 con 0x55b3f9e53400 2024-03-11T19:13:53.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.080+0000 7f2f1e7fc640 1 --2- 172.21.15.29:0/2046973713 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55b3f9e53400 0x55b3f9e55820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.080+0000 7f2f1e7fc640 1 --2- 172.21.15.29:0/2046973713 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55b3f9e53400 0x55b3f9e55820 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3d354d91f81ae449 server_cookie=8a8c2fe6c90b25c9 in_seq=0 out_seq=0 2024-03-11T19:13:53.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.080+0000 7f2f02ff5640 10 client.25972 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:13:53.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.081+0000 7f2f02ff5640 1 -- 172.21.15.29:0/2046973713 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f2f1400ea80 con 0x55b3f9e53400 2024-03-11T19:13:53.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.081+0000 7f2f02ff5640 10 client.25972 handle_command_reply: tid=0 2024-03-11T19:13:53.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.081+0000 7f2f283e3740 1 client.25972 shutdown 2024-03-11T19:13:53.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.081+0000 7f2f283e3740 1 -- 172.21.15.29:0/2046973713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de5a80 msgr2=0x55b3f9df6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.081+0000 7f2f283e3740 1 --2- 172.21.15.29:0/2046973713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de5a80 0x55b3f9df6e00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2f100bc380 tx=0x7f2f100a5dd0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.082+0000 7f2f283e3740 1 -- 172.21.15.29:0/2046973713 shutdown_connections 2024-03-11T19:13:53.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.082+0000 7f2f283e3740 1 --2- 172.21.15.29:0/2046973713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b3f9deb5e0 0x55b3f9df59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.082+0000 7f2f283e3740 1 --2- 172.21.15.29:0/2046973713 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55b3f9e53400 0x55b3f9e55820 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.082+0000 7f2f283e3740 1 --2- 172.21.15.29:0/2046973713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b3f9de6470 0x55b3f9df5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.082+0000 7f2f02ff5640 0 client.25972 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:13:53.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.083+0000 7f2f283e3740 1 --2- 172.21.15.29:0/2046973713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b3f9de5a80 0x55b3f9df6e00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.083+0000 7f2f283e3740 1 -- 172.21.15.29:0/2046973713 >> 172.21.15.29:0/2046973713 conn(0x55b3f9de0ff0 msgr2=0x55b3f9de79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.083+0000 7f2f283e3740 1 -- 172.21.15.29:0/2046973713 shutdown_connections 2024-03-11T19:13:53.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.085+0000 7f2f283e3740 1 -- 172.21.15.29:0/2046973713 wait complete. 2024-03-11T19:13:53.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.085+0000 7f2f283e3740 20 client.25972 trim_cache size 0 max 16384 2024-03-11T19:13:53.089 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:13:53.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.086+0000 7f2f2573a640 1 -- 172.21.15.29:0/2792112615 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2f080a1ab0 msgr2=0x7f2f080a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.086+0000 7f2f2573a640 1 --2- 172.21.15.29:0/2792112615 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2f080a1ab0 0x7f2f080a3f70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x55b3f9de6cd0 tx=0x7f2f180096a0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.086+0000 7f2f2573a640 1 -- 172.21.15.29:0/2792112615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f20100080 msgr2=0x7f2f20199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.086+0000 7f2f2573a640 1 --2- 172.21.15.29:0/2792112615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f20100080 0x7f2f20199420 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f2f1002c7e0 tx=0x7f2f100042f0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.086+0000 7f2f2573a640 1 -- 172.21.15.29:0/2792112615 shutdown_connections 2024-03-11T19:13:53.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.086+0000 7f2f2573a640 1 --2- 172.21.15.29:0/2792112615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f20101c30 0x7f2f201a09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.086+0000 7f2f2573a640 1 --2- 172.21.15.29:0/2792112615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f20101270 0x7f2f20199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.086+0000 7f2f2573a640 1 --2- 172.21.15.29:0/2792112615 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f2f080a1ab0 0x7f2f080a3f70 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.086+0000 7f2f2573a640 1 --2- 172.21.15.29:0/2792112615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f20100080 0x7f2f20199420 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.086+0000 7f2f2573a640 1 -- 172.21.15.29:0/2792112615 >> 172.21.15.29:0/2792112615 conn(0x7f2f200fb610 msgr2=0x7f2f2010b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.087+0000 7f2f2573a640 1 -- 172.21.15.29:0/2792112615 shutdown_connections 2024-03-11T19:13:53.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.087+0000 7f2f2573a640 1 -- 172.21.15.29:0/2792112615 wait complete. 2024-03-11T19:13:53.107 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:13:53.108 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:13:53.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.281+0000 7f5c5294b640 1 -- 172.21.15.29:0/1690200279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c4c108180 msgr2=0x7f5c4c10a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.281+0000 7f5c5294b640 1 --2- 172.21.15.29:0/1690200279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c4c108180 0x7f5c4c10a5b0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009900 tx=0x7f5c4402bc30 comp rx=0 tx=0).stop 2024-03-11T19:13:53.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.281+0000 7f5c5294b640 1 -- 172.21.15.29:0/1690200279 shutdown_connections 2024-03-11T19:13:53.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.281+0000 7f5c5294b640 1 --2- 172.21.15.29:0/1690200279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c4c10d4a0 0x7f5c4c10d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.281+0000 7f5c5294b640 1 --2- 172.21.15.29:0/1690200279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c4c10aaf0 0x7f5c4c10cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.281+0000 7f5c5294b640 1 --2- 172.21.15.29:0/1690200279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c4c108180 0x7f5c4c10a5b0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.281+0000 7f5c5294b640 1 -- 172.21.15.29:0/1690200279 >> 172.21.15.29:0/1690200279 conn(0x7f5c4c06dfe0 msgr2=0x7f5c4c070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.281+0000 7f5c5294b640 1 -- 172.21.15.29:0/1690200279 shutdown_connections 2024-03-11T19:13:53.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.281+0000 7f5c5294b640 1 -- 172.21.15.29:0/1690200279 wait complete. 2024-03-11T19:13:53.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.282+0000 7f5c5294b640 1 Processor -- start 2024-03-11T19:13:53.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.282+0000 7f5c5294b640 1 -- start start 2024-03-11T19:13:53.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.283+0000 7f5c5294b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c4c108180 0x7f5c4c1a5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.283+0000 7f5c5294b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c4c10aaf0 0x7f5c4c1a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.283+0000 7f5c5294b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c4c10d4a0 0x7f5c4c1ad3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.283+0000 7f5c5294b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c1ad900 con 0x7f5c4c10aaf0 2024-03-11T19:13:53.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.283+0000 7f5c5294b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c1ada70 con 0x7f5c4c10d4a0 2024-03-11T19:13:53.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.283+0000 7f5c5294b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c1adc30 con 0x7f5c4c108180 2024-03-11T19:13:53.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.283+0000 7f5c4bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c4c108180 0x7f5c4c1a5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.283+0000 7f5c4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c4c10aaf0 0x7f5c4c1a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.283+0000 7f5c4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c4c10aaf0 0x7f5c4c1a63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41350/0 (socket says 172.21.15.29:41350) 2024-03-11T19:13:53.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.283+0000 7f5c4b7fe640 1 -- 172.21.15.29:0/1882740604 learned_addr learned my addr 172.21.15.29:0/1882740604 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.283+0000 7f5c4bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c4c108180 0x7f5c4c1a5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:34760/0 (socket says 172.21.15.29:34760) 2024-03-11T19:13:53.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.283+0000 7f5c50ee8640 1 --2- 172.21.15.29:0/1882740604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c4c10d4a0 0x7f5c4c1ad3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.283+0000 7f5c4b7fe640 1 -- 172.21.15.29:0/1882740604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c4c108180 msgr2=0x7f5c4c1a5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.283+0000 7f5c4b7fe640 1 --2- 172.21.15.29:0/1882740604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c4c108180 0x7f5c4c1a5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.283+0000 7f5c4b7fe640 1 -- 172.21.15.29:0/1882740604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c4c10d4a0 msgr2=0x7f5c4c1ad3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.283+0000 7f5c4b7fe640 1 --2- 172.21.15.29:0/1882740604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c4c10d4a0 0x7f5c4c1ad3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.283+0000 7f5c4b7fe640 1 -- 172.21.15.29:0/1882740604 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c44009560 con 0x7f5c4c10aaf0 2024-03-11T19:13:53.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.283+0000 7f5c4bfff640 1 --2- 172.21.15.29:0/1882740604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c4c108180 0x7f5c4c1a5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:53.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.284+0000 7f5c4b7fe640 1 --2- 172.21.15.29:0/1882740604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c4c10aaf0 0x7f5c4c1a63a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5c4000e9f0 tx=0x7f5c4000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.284+0000 7f5c497fa640 1 -- 172.21.15.29:0/1882740604 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c40009700 con 0x7f5c4c10aaf0 2024-03-11T19:13:53.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.284+0000 7f5c497fa640 1 -- 172.21.15.29:0/1882740604 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5c4000cd80 con 0x7f5c4c10aaf0 2024-03-11T19:13:53.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.284+0000 7f5c5294b640 1 -- 172.21.15.29:0/1882740604 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c4c1adf10 con 0x7f5c4c10aaf0 2024-03-11T19:13:53.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.284+0000 7f5c497fa640 1 -- 172.21.15.29:0/1882740604 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c40018840 con 0x7f5c4c10aaf0 2024-03-11T19:13:53.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.284+0000 7f5c5294b640 1 -- 172.21.15.29:0/1882740604 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c4c1ae350 con 0x7f5c4c10aaf0 2024-03-11T19:13:53.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.287+0000 7f5c497fa640 1 -- 172.21.15.29:0/1882740604 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5c400189a0 con 0x7f5c4c10aaf0 2024-03-11T19:13:53.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.288+0000 7f5c555f0740 1 Processor -- start 2024-03-11T19:13:53.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.288+0000 7f5c555f0740 1 -- start start 2024-03-11T19:13:53.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.288+0000 7f5c555f0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae3733d270 0x55ae3733d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.288+0000 7f5c555f0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae3733dc60 0x55ae37342760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.288+0000 7f5c555f0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae37342dd0 0x55ae37345260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.288+0000 7f5c555f0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae372f9960 con 0x55ae37342dd0 2024-03-11T19:13:53.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.288+0000 7f5c555f0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae372ad310 con 0x55ae3733dc60 2024-03-11T19:13:53.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.288+0000 7f5c555f0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae372aa840 con 0x55ae3733d270 2024-03-11T19:13:53.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.288+0000 7f5c4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae37342dd0 0x55ae37345260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.288+0000 7f5c4bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae3733dc60 0x55ae37342760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.288+0000 7f5c4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae37342dd0 0x55ae37345260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41362/0 (socket says 172.21.15.29:41362) 2024-03-11T19:13:53.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.288+0000 7f5c4bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae3733dc60 0x55ae37342760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55352/0 (socket says 172.21.15.29:55352) 2024-03-11T19:13:53.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.288+0000 7f5c4b7fe640 1 -- 172.21.15.29:0/2416083208 learned_addr learned my addr 172.21.15.29:0/2416083208 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.288+0000 7f5c50ee8640 1 --2- 172.21.15.29:0/2416083208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae3733d270 0x55ae3733d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.289+0000 7f5c4b7fe640 1 -- 172.21.15.29:0/2416083208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae3733d270 msgr2=0x55ae3733d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.289+0000 7f5c4b7fe640 1 --2- 172.21.15.29:0/2416083208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae3733d270 0x55ae3733d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.289+0000 7f5c4b7fe640 1 -- 172.21.15.29:0/2416083208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae3733dc60 msgr2=0x55ae37342760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.289+0000 7f5c4b7fe640 1 --2- 172.21.15.29:0/2416083208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae3733dc60 0x55ae37342760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.289+0000 7f5c4b7fe640 1 -- 172.21.15.29:0/2416083208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae37268060 con 0x55ae37342dd0 2024-03-11T19:13:53.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.289+0000 7f5c4bfff640 1 --2- 172.21.15.29:0/2416083208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae3733dc60 0x55ae37342760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.289+0000 7f5c4b7fe640 1 --2- 172.21.15.29:0/2416083208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae37342dd0 0x55ae37345260 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5c400d5400 tx=0x7f5c400d58c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e1bcac46ccb8ad6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.289+0000 7f5c497fa640 1 --2- 172.21.15.29:0/1882740604 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5c340a1a30 0x7f5c340a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.289+0000 7f5c50ee8640 1 --2- 172.21.15.29:0/1882740604 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5c340a1a30 0x7f5c340a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.289+0000 7f5c327fc640 1 -- 172.21.15.29:0/2416083208 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c40096930 con 0x55ae37342dd0 2024-03-11T19:13:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.290+0000 7f5c327fc640 1 -- 172.21.15.29:0/2416083208 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5c4009a460 con 0x55ae37342dd0 2024-03-11T19:13:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.290+0000 7f5c327fc640 1 -- 172.21.15.29:0/2416083208 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c40024120 con 0x55ae37342dd0 2024-03-11T19:13:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.290+0000 7f5c497fa640 1 -- 172.21.15.29:0/1882740604 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5c40014070 con 0x7f5c4c10aaf0 2024-03-11T19:13:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.290+0000 7f5c555f0740 1 -- 172.21.15.29:0/2416083208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae37342dd0 msgr2=0x55ae37345260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.290+0000 7f5c555f0740 1 --2- 172.21.15.29:0/2416083208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae37342dd0 0x55ae37345260 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5c400d5400 tx=0x7f5c400d58c0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.290+0000 7f5c50ee8640 1 --2- 172.21.15.29:0/1882740604 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5c340a1a30 0x7f5c340a3ef0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55ae3733e4c0 tx=0x7f5c3c00d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.291+0000 7f5c555f0740 1 -- 172.21.15.29:0/2416083208 shutdown_connections 2024-03-11T19:13:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.291+0000 7f5c555f0740 1 --2- 172.21.15.29:0/2416083208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae37342dd0 0x55ae37345260 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.291+0000 7f5c555f0740 1 --2- 172.21.15.29:0/2416083208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae3733dc60 0x55ae37342760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.291+0000 7f5c555f0740 1 --2- 172.21.15.29:0/2416083208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae3733d270 0x55ae3733d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.291+0000 7f5c555f0740 1 -- 172.21.15.29:0/2416083208 >> 172.21.15.29:0/2416083208 conn(0x55ae373387e0 msgr2=0x55ae37338be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.291+0000 7f5c555f0740 1 -- 172.21.15.29:0/2416083208 shutdown_connections 2024-03-11T19:13:53.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.291+0000 7f5c555f0740 1 -- 172.21.15.29:0/2416083208 wait complete. 2024-03-11T19:13:53.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.291+0000 7f5c555f0740 1 -- start start 2024-03-11T19:13:53.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.292+0000 7f5c555f0740 1 Processor -- start 2024-03-11T19:13:53.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.292+0000 7f5c555f0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae3733d270 0x55ae37347450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.292+0000 7f5c555f0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 0x55ae37347990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.292+0000 7f5c555f0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae37342dd0 0x55ae3734e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.292+0000 7f5c555f0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae372a6020 con 0x55ae3733dc60 2024-03-11T19:13:53.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.292+0000 7f5c555f0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae36f79430 con 0x55ae3733d270 2024-03-11T19:13:53.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.292+0000 7f5c555f0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae36f35e00 con 0x55ae37342dd0 2024-03-11T19:13:53.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.292+0000 7f5c4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 0x55ae37347990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.292+0000 7f5c555f0740 10 client.0 _ll_register_callbacks cb 0x55ae3724dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:53.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.292+0000 7f5c4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 0x55ae37347990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41378/0 (socket says 172.21.15.29:41378) 2024-03-11T19:13:53.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.292+0000 7f5c4b7fe640 1 -- 172.21.15.29:0/1984986580 learned_addr learned my addr 172.21.15.29:0/1984986580 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.292+0000 7f5c4bfff640 1 --2- 172.21.15.29:0/1984986580 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae3733d270 0x55ae37347450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.293+0000 7f5c4b7fe640 1 -- 172.21.15.29:0/1984986580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae37342dd0 msgr2=0x55ae3734e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.293+0000 7f5c50ee8640 1 --2- 172.21.15.29:0/1984986580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae37342dd0 0x55ae3734e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.293+0000 7f5c4b7fe640 1 --2- 172.21.15.29:0/1984986580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae37342dd0 0x55ae3734e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.293+0000 7f5c4b7fe640 1 -- 172.21.15.29:0/1984986580 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae3733d270 msgr2=0x55ae37347450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.293+0000 7f5c4b7fe640 1 --2- 172.21.15.29:0/1984986580 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae3733d270 0x55ae37347450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.293+0000 7f5c4b7fe640 1 -- 172.21.15.29:0/1984986580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c4c1ae350 con 0x55ae3733dc60 2024-03-11T19:13:53.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.293+0000 7f5c50ee8640 1 --2- 172.21.15.29:0/1984986580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae37342dd0 0x55ae3734e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:13:53.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.293+0000 7f5c4bfff640 1 --2- 172.21.15.29:0/1984986580 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae3733d270 0x55ae37347450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:53.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.293+0000 7f5c4b7fe640 1 --2- 172.21.15.29:0/1984986580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 0x55ae37347990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f5c40001eb0 tx=0x7f5c40024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5da8bd746f9654 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.293+0000 7f5c2b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:13:53.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.294+0000 7f5c2b7fe640 1 -- 172.21.15.29:0/1984986580 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c40096930 con 0x55ae3733dc60 2024-03-11T19:13:53.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.294+0000 7f5c2b7fe640 1 -- 172.21.15.29:0/1984986580 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5c4009a690 con 0x55ae3733dc60 2024-03-11T19:13:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.294+0000 7f5c555f0740 1 -- 172.21.15.29:0/1984986580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ae36fa57e0 con 0x55ae3733dc60 2024-03-11T19:13:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.294+0000 7f5c2b7fe640 1 -- 172.21.15.29:0/1984986580 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c400d09e0 con 0x55ae3733dc60 2024-03-11T19:13:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.294+0000 7f5c555f0740 10 client.16446 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.294+0000 7f5c555f0740 1 -- 172.21.15.29:0/1984986580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ae372a46a0 con 0x55ae3733dc60 2024-03-11T19:13:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.295+0000 7f5c2b7fe640 1 -- 172.21.15.29:0/1984986580 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55ae36fa57e0 con 0x55ae3733dc60 2024-03-11T19:13:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.295+0000 7f5c2b7fe640 1 -- 172.21.15.29:0/1984986580 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5c400850b0 con 0x55ae3733dc60 2024-03-11T19:13:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.295+0000 7f5c555f0740 10 client.16446 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.295+0000 7f5c555f0740 10 client.16446 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:13:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.295+0000 7f5c555f0740 20 client.16446 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.295+0000 7f5c555f0740 1 --2- 172.21.15.29:0/1984986580 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55ae373531e0 0x55ae37355600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.295+0000 7f5c555f0740 4 client.16446 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:13:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.295+0000 7f5c555f0740 1 -- 172.21.15.29:0/1984986580 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ae36fe2720 con 0x55ae373531e0 2024-03-11T19:13:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.296+0000 7f5c4bfff640 1 --2- 172.21.15.29:0/1984986580 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55ae373531e0 0x55ae37355600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.296+0000 7f5c2b7fe640 10 client.16446 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:13:53.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.296+0000 7f5c4bfff640 1 --2- 172.21.15.29:0/1984986580 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55ae373531e0 0x55ae37355600 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ed6ce6535b5a911 server_cookie=7a608812458a929f in_seq=0 out_seq=0 2024-03-11T19:13:53.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.298+0000 7f5c2b7fe640 1 -- 172.21.15.29:0/1984986580 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ae36fe2720 con 0x55ae373531e0 2024-03-11T19:13:53.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.298+0000 7f5c2b7fe640 10 client.16446 handle_command_reply: tid=0 2024-03-11T19:13:53.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.298+0000 7f5c555f0740 1 client.16446 shutdown 2024-03-11T19:13:53.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.298+0000 7f5c555f0740 1 -- 172.21.15.29:0/1984986580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 msgr2=0x55ae37347990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.298+0000 7f5c555f0740 1 --2- 172.21.15.29:0/1984986580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 0x55ae37347990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f5c40001eb0 tx=0x7f5c40024000 comp rx=0 tx=0).stop 2024-03-11T19:13:53.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.298+0000 7f5c555f0740 1 -- 172.21.15.29:0/1984986580 shutdown_connections 2024-03-11T19:13:53.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.299+0000 7f5c555f0740 1 --2- 172.21.15.29:0/1984986580 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55ae373531e0 0x55ae37355600 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.299+0000 7f5c555f0740 1 --2- 172.21.15.29:0/1984986580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae37342dd0 0x55ae3734e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.299+0000 7f5c555f0740 1 --2- 172.21.15.29:0/1984986580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 0x55ae37347990 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.299+0000 7f5c555f0740 1 --2- 172.21.15.29:0/1984986580 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae3733d270 0x55ae37347450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.299+0000 7f5c555f0740 1 -- 172.21.15.29:0/1984986580 >> 172.21.15.29:0/1984986580 conn(0x55ae373387e0 msgr2=0x55ae37337b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.299+0000 7f5c2b7fe640 0 client.16446 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:13:53.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.299+0000 7f5c555f0740 1 -- 172.21.15.29:0/1984986580 shutdown_connections 2024-03-11T19:13:53.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.299+0000 7f5c555f0740 1 -- 172.21.15.29:0/1984986580 wait complete. 2024-03-11T19:13:53.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.299+0000 7f5c555f0740 20 client.16446 trim_cache size 0 max 16384 2024-03-11T19:13:53.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.320+0000 7f5c555f0740 1 Processor -- start 2024-03-11T19:13:53.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.320+0000 7f5c555f0740 1 -- start start 2024-03-11T19:13:53.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.320+0000 7f5c555f0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae3733d270 0x55ae3733f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.320+0000 7f5c555f0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 0x55ae3734e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.320+0000 7f5c555f0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae37342dd0 0x55ae3733f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.320+0000 7f5c555f0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae3722b130 con 0x55ae3733dc60 2024-03-11T19:13:53.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.320+0000 7f5c555f0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae3733e350 con 0x55ae37342dd0 2024-03-11T19:13:53.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.320+0000 7f5c4bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae3733d270 0x55ae3733f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.320+0000 7f5c4bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae3733d270 0x55ae3733f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:34792/0 (socket says 172.21.15.29:34792) 2024-03-11T19:13:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.320+0000 7f5c4bfff640 1 -- 172.21.15.29:0/3865653472 learned_addr learned my addr 172.21.15.29:0/3865653472 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.321+0000 7f5c4b7fe640 1 --2- 172.21.15.29:0/3865653472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 0x55ae3734e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.321+0000 7f5c555f0740 1 -- 172.21.15.29:0/3865653472 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae3734fd50 con 0x55ae3733d270 2024-03-11T19:13:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.321+0000 7f5c50ee8640 1 --2- 172.21.15.29:0/3865653472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae37342dd0 0x55ae3733f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.321+0000 7f5c4b7fe640 1 -- 172.21.15.29:0/3865653472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae3733d270 msgr2=0x55ae3733f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.321+0000 7f5c4b7fe640 1 --2- 172.21.15.29:0/3865653472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae3733d270 0x55ae3733f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.321+0000 7f5c4b7fe640 1 -- 172.21.15.29:0/3865653472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae37342dd0 msgr2=0x55ae3733f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.321+0000 7f5c4b7fe640 1 --2- 172.21.15.29:0/3865653472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae37342dd0 0x55ae3733f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.321+0000 7f5c4b7fe640 1 -- 172.21.15.29:0/3865653472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae37240c20 con 0x55ae3733dc60 2024-03-11T19:13:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.321+0000 7f5c4b7fe640 1 --2- 172.21.15.29:0/3865653472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 0x55ae3734e990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5c4009a620 tx=0x7f5c400d0eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b26026a8c39f241 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.321+0000 7f5c4bfff640 1 --2- 172.21.15.29:0/3865653472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae3733d270 0x55ae3733f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:13:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.322+0000 7f5c31ffb640 1 -- 172.21.15.29:0/3865653472 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c400d4070 con 0x55ae3733dc60 2024-03-11T19:13:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.322+0000 7f5c31ffb640 1 -- 172.21.15.29:0/3865653472 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5c40096930 con 0x55ae3733dc60 2024-03-11T19:13:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.322+0000 7f5c31ffb640 1 -- 172.21.15.29:0/3865653472 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4009a720 con 0x55ae3733dc60 2024-03-11T19:13:53.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.323+0000 7f5c555f0740 1 -- 172.21.15.29:0/3865653472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 msgr2=0x55ae3734e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.323+0000 7f5c555f0740 1 --2- 172.21.15.29:0/3865653472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 0x55ae3734e990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5c4009a620 tx=0x7f5c400d0eb0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.323+0000 7f5c555f0740 1 -- 172.21.15.29:0/3865653472 shutdown_connections 2024-03-11T19:13:53.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.323+0000 7f5c555f0740 1 --2- 172.21.15.29:0/3865653472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae37342dd0 0x55ae3733f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.323+0000 7f5c555f0740 1 --2- 172.21.15.29:0/3865653472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 0x55ae3734e990 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.323+0000 7f5c555f0740 1 --2- 172.21.15.29:0/3865653472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae3733d270 0x55ae3733f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.323+0000 7f5c555f0740 1 -- 172.21.15.29:0/3865653472 >> 172.21.15.29:0/3865653472 conn(0x55ae373387e0 msgr2=0x55ae37347450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.323+0000 7f5c555f0740 1 -- 172.21.15.29:0/3865653472 shutdown_connections 2024-03-11T19:13:53.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.324+0000 7f5c555f0740 1 -- 172.21.15.29:0/3865653472 wait complete. 2024-03-11T19:13:53.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.324+0000 7f5c555f0740 1 -- start start 2024-03-11T19:13:53.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.324+0000 7f5c555f0740 1 Processor -- start 2024-03-11T19:13:53.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.325+0000 7f5c555f0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae3733d270 0x55ae3734c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.325+0000 7f5c555f0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 0x55ae3734cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.325+0000 7f5c555f0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae37342dd0 0x55ae3734dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.325+0000 7f5c555f0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae3734e160 con 0x55ae3733dc60 2024-03-11T19:13:53.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.325+0000 7f5c555f0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae3734e2d0 con 0x55ae37342dd0 2024-03-11T19:13:53.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.325+0000 7f5c555f0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae3734e490 con 0x55ae3733d270 2024-03-11T19:13:53.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.325+0000 7f5c555f0740 10 client.0 _ll_register_callbacks cb 0x7f5c400d0d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:53.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.325+0000 7f5c4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 0x55ae3734cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.325+0000 7f5c4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 0x55ae3734cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41384/0 (socket says 172.21.15.29:41384) 2024-03-11T19:13:53.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.325+0000 7f5c4b7fe640 1 -- 172.21.15.29:0/2394924666 learned_addr learned my addr 172.21.15.29:0/2394924666 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.325+0000 7f5c50ee8640 1 --2- 172.21.15.29:0/2394924666 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae37342dd0 0x55ae3734dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.325+0000 7f5c4bfff640 1 --2- 172.21.15.29:0/2394924666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae3733d270 0x55ae3734c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.325+0000 7f5c50ee8640 1 -- 172.21.15.29:0/2394924666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae3733d270 msgr2=0x55ae3734c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.325+0000 7f5c50ee8640 1 --2- 172.21.15.29:0/2394924666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae3733d270 0x55ae3734c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.325+0000 7f5c50ee8640 1 -- 172.21.15.29:0/2394924666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 msgr2=0x55ae3734cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.325+0000 7f5c50ee8640 1 --2- 172.21.15.29:0/2394924666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 0x55ae3734cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.325+0000 7f5c50ee8640 1 -- 172.21.15.29:0/2394924666 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ae372a46a0 con 0x55ae37342dd0 2024-03-11T19:13:53.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.325+0000 7f5c4b7fe640 1 --2- 172.21.15.29:0/2394924666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 0x55ae3734cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:53.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.326+0000 7f5c50ee8640 1 --2- 172.21.15.29:0/2394924666 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae37342dd0 0x55ae3734dc20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c009eb0 tx=0x7f5c3c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd8bf08c668f4453 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.326+0000 7f5c2b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:13:53.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.326+0000 7f5c2b7fe640 1 -- 172.21.15.29:0/2394924666 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c010040 con 0x55ae37342dd0 2024-03-11T19:13:53.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.326+0000 7f5c2b7fe640 1 -- 172.21.15.29:0/2394924666 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5c3c023820 con 0x55ae37342dd0 2024-03-11T19:13:53.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.326+0000 7f5c555f0740 1 -- 172.21.15.29:0/2394924666 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5c4009a460 con 0x55ae37342dd0 2024-03-11T19:13:53.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.326+0000 7f5c2b7fe640 1 -- 172.21.15.29:0/2394924666 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c3c0239e0 con 0x55ae37342dd0 2024-03-11T19:13:53.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.326+0000 7f5c555f0740 10 client.25958 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:53.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.326+0000 7f5c555f0740 1 -- 172.21.15.29:0/2394924666 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ae372668c0 con 0x55ae37342dd0 2024-03-11T19:13:53.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.327+0000 7f5c2b7fe640 1 -- 172.21.15.29:0/2394924666 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5c3c030030 con 0x55ae37342dd0 2024-03-11T19:13:53.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.327+0000 7f5c2b7fe640 1 -- 172.21.15.29:0/2394924666 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5c3c017c70 con 0x55ae37342dd0 2024-03-11T19:13:53.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.327+0000 7f5c555f0740 10 client.25958 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:53.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.327+0000 7f5c555f0740 10 client.25958 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:13:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.327+0000 7f5c555f0740 20 client.25958 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.327+0000 7f5c555f0740 1 --2- 172.21.15.29:0/2394924666 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55ae373aa6a0 0x55ae373acac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.327+0000 7f5c555f0740 4 client.25958 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:13:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.327+0000 7f5c555f0740 1 -- 172.21.15.29:0/2394924666 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ae37031890 con 0x55ae373aa6a0 2024-03-11T19:13:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.327+0000 7f5c4bfff640 1 --2- 172.21.15.29:0/2394924666 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55ae373aa6a0 0x55ae373acac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.328+0000 7f5c2b7fe640 10 client.25958 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:13:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.328+0000 7f5c4bfff640 1 --2- 172.21.15.29:0/2394924666 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55ae373aa6a0 0x55ae373acac0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=773476dd4dff770c server_cookie=783ee5baf1c024f in_seq=0 out_seq=0 2024-03-11T19:13:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.328+0000 7f5c2b7fe640 1 -- 172.21.15.29:0/2394924666 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ae37031890 con 0x55ae373aa6a0 2024-03-11T19:13:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.328+0000 7f5c2b7fe640 10 client.25958 handle_command_reply: tid=0 2024-03-11T19:13:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.328+0000 7f5c555f0740 1 client.25958 shutdown 2024-03-11T19:13:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.329+0000 7f5c555f0740 1 -- 172.21.15.29:0/2394924666 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae37342dd0 msgr2=0x55ae3734dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.329+0000 7f5c555f0740 1 --2- 172.21.15.29:0/2394924666 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae37342dd0 0x55ae3734dc20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c009eb0 tx=0x7f5c3c004a40 comp rx=0 tx=0).stop 2024-03-11T19:13:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.329+0000 7f5c555f0740 1 -- 172.21.15.29:0/2394924666 shutdown_connections 2024-03-11T19:13:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.329+0000 7f5c555f0740 1 --2- 172.21.15.29:0/2394924666 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55ae373aa6a0 0x55ae373acac0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.329+0000 7f5c555f0740 1 --2- 172.21.15.29:0/2394924666 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae37342dd0 0x55ae3734dc20 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.329+0000 7f5c2b7fe640 0 client.25958 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:13:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.329+0000 7f5c555f0740 1 --2- 172.21.15.29:0/2394924666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae3733dc60 0x55ae3734cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.329+0000 7f5c555f0740 1 --2- 172.21.15.29:0/2394924666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae3733d270 0x55ae3734c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.329+0000 7f5c555f0740 1 -- 172.21.15.29:0/2394924666 >> 172.21.15.29:0/2394924666 conn(0x55ae373387e0 msgr2=0x55ae37347410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.329+0000 7f5c555f0740 1 -- 172.21.15.29:0/2394924666 shutdown_connections 2024-03-11T19:13:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.330+0000 7f5c555f0740 1 -- 172.21.15.29:0/2394924666 wait complete. 2024-03-11T19:13:53.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.330+0000 7f5c555f0740 20 client.25958 trim_cache size 0 max 16384 2024-03-11T19:13:53.336 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:13:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.330+0000 7f5c5294b640 1 -- 172.21.15.29:0/1882740604 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5c340a1a30 msgr2=0x7f5c340a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.330+0000 7f5c5294b640 1 --2- 172.21.15.29:0/1882740604 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5c340a1a30 0x7f5c340a3ef0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55ae3733e4c0 tx=0x7f5c3c00d040 comp rx=0 tx=0).stop 2024-03-11T19:13:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.330+0000 7f5c5294b640 1 -- 172.21.15.29:0/1882740604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c4c10aaf0 msgr2=0x7f5c4c1a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.331+0000 7f5c5294b640 1 --2- 172.21.15.29:0/1882740604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c4c10aaf0 0x7f5c4c1a63a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5c4000e9f0 tx=0x7f5c4000eeb0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.331+0000 7f5c5294b640 1 -- 172.21.15.29:0/1882740604 shutdown_connections 2024-03-11T19:13:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.331+0000 7f5c5294b640 1 --2- 172.21.15.29:0/1882740604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c4c10d4a0 0x7f5c4c1ad3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.331+0000 7f5c5294b640 1 --2- 172.21.15.29:0/1882740604 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5c340a1a30 0x7f5c340a3ef0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.331+0000 7f5c5294b640 1 --2- 172.21.15.29:0/1882740604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c4c10aaf0 0x7f5c4c1a63a0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.331+0000 7f5c5294b640 1 --2- 172.21.15.29:0/1882740604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c4c108180 0x7f5c4c1a5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.331+0000 7f5c5294b640 1 -- 172.21.15.29:0/1882740604 >> 172.21.15.29:0/1882740604 conn(0x7f5c4c06dfe0 msgr2=0x7f5c4c10cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.331+0000 7f5c5294b640 1 -- 172.21.15.29:0/1882740604 shutdown_connections 2024-03-11T19:13:53.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.331+0000 7f5c5294b640 1 -- 172.21.15.29:0/1882740604 wait complete. 2024-03-11T19:13:53.350 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:13:53.351 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:13:53.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.520+0000 7fd4e49e2640 1 -- 172.21.15.29:0/2287259628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4e00ff9f0 msgr2=0x7fd4e0101e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.520+0000 7fd4e49e2640 1 --2- 172.21.15.29:0/2287259628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4e00ff9f0 0x7fd4e0101e20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009900 tx=0x7fd4d402bc30 comp rx=0 tx=0).stop 2024-03-11T19:13:53.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.520+0000 7fd4e49e2640 1 -- 172.21.15.29:0/2287259628 shutdown_connections 2024-03-11T19:13:53.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.520+0000 7fd4e49e2640 1 --2- 172.21.15.29:0/2287259628 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4e0104d10 0x7fd4e0105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.520+0000 7fd4e49e2640 1 --2- 172.21.15.29:0/2287259628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4e0102360 0x7fd4e01047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.520+0000 7fd4e49e2640 1 --2- 172.21.15.29:0/2287259628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4e00ff9f0 0x7fd4e0101e20 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.520+0000 7fd4e49e2640 1 -- 172.21.15.29:0/2287259628 >> 172.21.15.29:0/2287259628 conn(0x7fd4e00fb610 msgr2=0x7fd4e00fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.520+0000 7fd4e49e2640 1 -- 172.21.15.29:0/2287259628 shutdown_connections 2024-03-11T19:13:53.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.520+0000 7fd4e49e2640 1 -- 172.21.15.29:0/2287259628 wait complete. 2024-03-11T19:13:53.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.521+0000 7fd4e49e2640 1 Processor -- start 2024-03-11T19:13:53.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.521+0000 7fd4e49e2640 1 -- start start 2024-03-11T19:13:53.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.521+0000 7fd4e49e2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4e00ff9f0 0x7fd4e019d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.521+0000 7fd4e49e2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4e0102360 0x7fd4e019dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.521+0000 7fd4e49e2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4e0104d10 0x7fd4e01a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.521+0000 7fd4e49e2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e01a5240 con 0x7fd4e0102360 2024-03-11T19:13:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.521+0000 7fd4de59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4e00ff9f0 0x7fd4e019d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.521+0000 7fd4e49e2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e01a53b0 con 0x7fd4e00ff9f0 2024-03-11T19:13:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.521+0000 7fd4de59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4e00ff9f0 0x7fd4e019d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55402/0 (socket says 172.21.15.29:55402) 2024-03-11T19:13:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.521+0000 7fd4de59c640 1 -- 172.21.15.29:0/1915160017 learned_addr learned my addr 172.21.15.29:0/1915160017 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.521+0000 7fd4e49e2640 1 -- 172.21.15.29:0/1915160017 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e01a5570 con 0x7fd4e0104d10 2024-03-11T19:13:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.521+0000 7fd4ded9d640 1 --2- 172.21.15.29:0/1915160017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4e0104d10 0x7fd4e01a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.521+0000 7fd4ddd9b640 1 --2- 172.21.15.29:0/1915160017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4e0102360 0x7fd4e019dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.522+0000 7fd4de59c640 1 -- 172.21.15.29:0/1915160017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4e0104d10 msgr2=0x7fd4e01a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.522+0000 7fd4de59c640 1 --2- 172.21.15.29:0/1915160017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4e0104d10 0x7fd4e01a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.522+0000 7fd4de59c640 1 -- 172.21.15.29:0/1915160017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4e0102360 msgr2=0x7fd4e019dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.522+0000 7fd4de59c640 1 --2- 172.21.15.29:0/1915160017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4e0102360 0x7fd4e019dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.522+0000 7fd4de59c640 1 -- 172.21.15.29:0/1915160017 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d4009560 con 0x7fd4e00ff9f0 2024-03-11T19:13:53.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.522+0000 7fd4ded9d640 1 --2- 172.21.15.29:0/1915160017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4e0104d10 0x7fd4e01a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:53.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.522+0000 7fd4ddd9b640 1 --2- 172.21.15.29:0/1915160017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4e0102360 0x7fd4e019dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:53.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.522+0000 7fd4de59c640 1 --2- 172.21.15.29:0/1915160017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4e00ff9f0 0x7fd4e019d750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd4d402c7d0 tx=0x7fd4d4004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.522+0000 7fd4cf7fe640 1 -- 172.21.15.29:0/1915160017 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d40035f0 con 0x7fd4e00ff9f0 2024-03-11T19:13:53.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.522+0000 7fd4cf7fe640 1 -- 172.21.15.29:0/1915160017 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd4d4003c00 con 0x7fd4e00ff9f0 2024-03-11T19:13:53.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.522+0000 7fd4e49e2640 1 -- 172.21.15.29:0/1915160017 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e01a57f0 con 0x7fd4e00ff9f0 2024-03-11T19:13:53.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.523+0000 7fd4e49e2640 1 -- 172.21.15.29:0/1915160017 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e01a5d50 con 0x7fd4e00ff9f0 2024-03-11T19:13:53.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.523+0000 7fd4cf7fe640 1 -- 172.21.15.29:0/1915160017 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d40054e0 con 0x7fd4e00ff9f0 2024-03-11T19:13:53.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.526+0000 7fd4e75ba740 1 Processor -- start 2024-03-11T19:13:53.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.526+0000 7fd4e75ba740 1 -- start start 2024-03-11T19:13:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.526+0000 7fd4e75ba740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b4cb993270 0x55b4cb993690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.526+0000 7fd4e75ba740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb993c60 0x55b4cb998760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.526+0000 7fd4e75ba740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb998dd0 0x55b4cb99b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.526+0000 7fd4e75ba740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4cb94f960 con 0x55b4cb993270 2024-03-11T19:13:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.526+0000 7fd4e75ba740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4cb903310 con 0x55b4cb998dd0 2024-03-11T19:13:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.526+0000 7fd4e75ba740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4cb900840 con 0x55b4cb993c60 2024-03-11T19:13:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.527+0000 7fd4cf7fe640 1 -- 172.21.15.29:0/1915160017 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd4d4003750 con 0x7fd4e00ff9f0 2024-03-11T19:13:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.527+0000 7fd4ddd9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb998dd0 0x55b4cb99b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.527+0000 7fd4ddd9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb998dd0 0x55b4cb99b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55418/0 (socket says 172.21.15.29:55418) 2024-03-11T19:13:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.527+0000 7fd4ddd9b640 1 -- 172.21.15.29:0/1543986223 learned_addr learned my addr 172.21.15.29:0/1543986223 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.527+0000 7fd4ded9d640 1 --2- 172.21.15.29:0/1543986223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b4cb993270 0x55b4cb993690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.527+0000 7fd4de59c640 1 --2- 172.21.15.29:0/1543986223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb993c60 0x55b4cb998760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.527+0000 7fd4ddd9b640 1 -- 172.21.15.29:0/1543986223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb993c60 msgr2=0x55b4cb998760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.527+0000 7fd4ddd9b640 1 --2- 172.21.15.29:0/1543986223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb993c60 0x55b4cb998760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.527+0000 7fd4ddd9b640 1 -- 172.21.15.29:0/1543986223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b4cb993270 msgr2=0x55b4cb993690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.527+0000 7fd4ddd9b640 1 --2- 172.21.15.29:0/1543986223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b4cb993270 0x55b4cb993690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.527+0000 7fd4ddd9b640 1 -- 172.21.15.29:0/1543986223 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4cb8be060 con 0x55b4cb998dd0 2024-03-11T19:13:53.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.527+0000 7fd4de59c640 1 --2- 172.21.15.29:0/1543986223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb993c60 0x55b4cb998760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:13:53.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.528+0000 7fd4ddd9b640 1 --2- 172.21.15.29:0/1543986223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb998dd0 0x55b4cb99b260 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd4d00114f0 tx=0x7fd4d00119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef8b622f118c366f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.528+0000 7fd4ccff9640 1 -- 172.21.15.29:0/1543986223 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0004030 con 0x55b4cb998dd0 2024-03-11T19:13:53.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.528+0000 7fd4ccff9640 1 -- 172.21.15.29:0/1543986223 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd4d00035f0 con 0x55b4cb998dd0 2024-03-11T19:13:53.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.528+0000 7fd4cf7fe640 1 --2- 172.21.15.29:0/1915160017 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd4c80a1b80 0x7fd4c80a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.528+0000 7fd4cf7fe640 1 -- 172.21.15.29:0/1915160017 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd4d402de40 con 0x7fd4e00ff9f0 2024-03-11T19:13:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.528+0000 7fd4ded9d640 1 --2- 172.21.15.29:0/1915160017 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd4c80a1b80 0x7fd4c80a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.529+0000 7fd4ccff9640 1 -- 172.21.15.29:0/1543986223 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d000f860 con 0x55b4cb998dd0 2024-03-11T19:13:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.529+0000 7fd4e75ba740 1 -- 172.21.15.29:0/1543986223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb998dd0 msgr2=0x55b4cb99b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.529+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/1543986223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb998dd0 0x55b4cb99b260 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd4d00114f0 tx=0x7fd4d00119b0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.529+0000 7fd4ded9d640 1 --2- 172.21.15.29:0/1915160017 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd4c80a1b80 0x7fd4c80a4040 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x55b4cb9944c0 tx=0x7fd4d8009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.529+0000 7fd4e75ba740 1 -- 172.21.15.29:0/1543986223 shutdown_connections 2024-03-11T19:13:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.529+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/1543986223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb998dd0 0x55b4cb99b260 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.529+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/1543986223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb993c60 0x55b4cb998760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.529+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/1543986223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b4cb993270 0x55b4cb993690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.529+0000 7fd4e75ba740 1 -- 172.21.15.29:0/1543986223 >> 172.21.15.29:0/1543986223 conn(0x55b4cb98e7e0 msgr2=0x55b4cb98ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.529+0000 7fd4e75ba740 1 -- 172.21.15.29:0/1543986223 shutdown_connections 2024-03-11T19:13:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.529+0000 7fd4e75ba740 1 -- 172.21.15.29:0/1543986223 wait complete. 2024-03-11T19:13:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.530+0000 7fd4e75ba740 1 -- start start 2024-03-11T19:13:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.530+0000 7fd4e75ba740 1 Processor -- start 2024-03-11T19:13:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.530+0000 7fd4e75ba740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b4cb993270 0x55b4cb99d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.530+0000 7fd4e75ba740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993c60 0x55b4cb99d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.530+0000 7fd4e75ba740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb998dd0 0x55b4cb9a49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.530+0000 7fd4e75ba740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4cb8fc020 con 0x55b4cb993270 2024-03-11T19:13:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.530+0000 7fd4e75ba740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4cb5cf430 con 0x55b4cb993c60 2024-03-11T19:13:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.530+0000 7fd4e75ba740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4cb58be00 con 0x55b4cb998dd0 2024-03-11T19:13:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.530+0000 7fd4e75ba740 10 client.0 _ll_register_callbacks cb 0x55b4cb8a3cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.530+0000 7fd4ddd9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993c60 0x55b4cb99d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.530+0000 7fd4ddd9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993c60 0x55b4cb99d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55432/0 (socket says 172.21.15.29:55432) 2024-03-11T19:13:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.530+0000 7fd4ddd9b640 1 -- 172.21.15.29:0/649586049 learned_addr learned my addr 172.21.15.29:0/649586049 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.531+0000 7fd4de59c640 1 --2- 172.21.15.29:0/649586049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b4cb993270 0x55b4cb99d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.531+0000 7fd4ded9d640 1 --2- 172.21.15.29:0/649586049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb998dd0 0x55b4cb9a49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.531+0000 7fd4ddd9b640 1 -- 172.21.15.29:0/649586049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb998dd0 msgr2=0x55b4cb9a49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.531+0000 7fd4ddd9b640 1 --2- 172.21.15.29:0/649586049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb998dd0 0x55b4cb9a49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.531+0000 7fd4ddd9b640 1 -- 172.21.15.29:0/649586049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b4cb993270 msgr2=0x55b4cb99d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.531+0000 7fd4ddd9b640 1 --2- 172.21.15.29:0/649586049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b4cb993270 0x55b4cb99d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.531+0000 7fd4ddd9b640 1 -- 172.21.15.29:0/649586049 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d0009cc0 con 0x55b4cb993c60 2024-03-11T19:13:53.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.531+0000 7fd4ded9d640 1 --2- 172.21.15.29:0/649586049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb998dd0 0x55b4cb9a49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:13:53.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.531+0000 7fd4ddd9b640 1 --2- 172.21.15.29:0/649586049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993c60 0x55b4cb99d9f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0011eb0 tx=0x7fd4d0004210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1633bf7765efd778 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.531+0000 7fd4c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:13:53.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.532+0000 7fd4c5ffb640 1 -- 172.21.15.29:0/649586049 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0005040 con 0x55b4cb993c60 2024-03-11T19:13:53.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.532+0000 7fd4c5ffb640 1 -- 172.21.15.29:0/649586049 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd4d00051a0 con 0x55b4cb993c60 2024-03-11T19:13:53.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.532+0000 7fd4e75ba740 1 -- 172.21.15.29:0/649586049 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b4cb5fb7e0 con 0x55b4cb993c60 2024-03-11T19:13:53.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.532+0000 7fd4c5ffb640 1 -- 172.21.15.29:0/649586049 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4d0005360 con 0x55b4cb993c60 2024-03-11T19:13:53.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.532+0000 7fd4e75ba740 10 client.25970 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:53.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.533+0000 7fd4e75ba740 1 -- 172.21.15.29:0/649586049 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55b4cb8fa6a0 con 0x55b4cb993c60 2024-03-11T19:13:53.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.533+0000 7fd4c5ffb640 1 -- 172.21.15.29:0/649586049 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55b4cb5fb7e0 con 0x55b4cb993c60 2024-03-11T19:13:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.533+0000 7fd4c5ffb640 1 -- 172.21.15.29:0/649586049 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd4d000f650 con 0x55b4cb993c60 2024-03-11T19:13:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.533+0000 7fd4e75ba740 10 client.25970 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.533+0000 7fd4e75ba740 10 client.25970 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:13:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.533+0000 7fd4e75ba740 20 client.25970 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.533+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/649586049 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55b4cb9a93f0 0x55b4cb9ab810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.533+0000 7fd4e75ba740 4 client.25970 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:13:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.533+0000 7fd4e75ba740 1 -- 172.21.15.29:0/649586049 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b4cb638720 con 0x55b4cb9a93f0 2024-03-11T19:13:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.533+0000 7fd4de59c640 1 --2- 172.21.15.29:0/649586049 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55b4cb9a93f0 0x55b4cb9ab810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.534+0000 7fd4de59c640 1 --2- 172.21.15.29:0/649586049 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55b4cb9a93f0 0x55b4cb9ab810 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=15a702323866d1f6 server_cookie=e2897e865780df3d in_seq=0 out_seq=0 2024-03-11T19:13:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.534+0000 7fd4c5ffb640 10 client.25970 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:13:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.534+0000 7fd4c5ffb640 1 -- 172.21.15.29:0/649586049 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b4cb638720 con 0x55b4cb9a93f0 2024-03-11T19:13:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.534+0000 7fd4c5ffb640 10 client.25970 handle_command_reply: tid=0 2024-03-11T19:13:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.535+0000 7fd4e75ba740 1 client.25970 shutdown 2024-03-11T19:13:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.535+0000 7fd4e75ba740 1 -- 172.21.15.29:0/649586049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993c60 msgr2=0x55b4cb99d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.535+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/649586049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993c60 0x55b4cb99d9f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0011eb0 tx=0x7fd4d0004210 comp rx=0 tx=0).stop 2024-03-11T19:13:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.535+0000 7fd4e75ba740 1 -- 172.21.15.29:0/649586049 shutdown_connections 2024-03-11T19:13:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.535+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/649586049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb998dd0 0x55b4cb9a49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.535+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/649586049 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55b4cb9a93f0 0x55b4cb9ab810 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.535+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/649586049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993c60 0x55b4cb99d9f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.535+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/649586049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b4cb993270 0x55b4cb99d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.535+0000 7fd4e75ba740 1 -- 172.21.15.29:0/649586049 >> 172.21.15.29:0/649586049 conn(0x55b4cb98e7e0 msgr2=0x55b4cb98db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.535+0000 7fd4c5ffb640 0 client.25970 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:13:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.535+0000 7fd4e75ba740 1 -- 172.21.15.29:0/649586049 shutdown_connections 2024-03-11T19:13:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.535+0000 7fd4e75ba740 1 -- 172.21.15.29:0/649586049 wait complete. 2024-03-11T19:13:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.535+0000 7fd4e75ba740 20 client.25970 trim_cache size 0 max 16384 2024-03-11T19:13:53.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.564+0000 7fd4e75ba740 1 Processor -- start 2024-03-11T19:13:53.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.564+0000 7fd4e75ba740 1 -- start start 2024-03-11T19:13:53.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.564+0000 7fd4e75ba740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993270 0x55b4cb995120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.564+0000 7fd4e75ba740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b4cb993c60 0x55b4cb995660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.564+0000 7fd4e75ba740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb998dd0 0x55b4cb9a4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.564+0000 7fd4e75ba740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4cb881130 con 0x55b4cb993c60 2024-03-11T19:13:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.564+0000 7fd4de59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993270 0x55b4cb995120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.564+0000 7fd4e75ba740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4cb994380 con 0x55b4cb993270 2024-03-11T19:13:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.564+0000 7fd4e75ba740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4cb994090 con 0x55b4cb998dd0 2024-03-11T19:13:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.564+0000 7fd4ddd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b4cb993c60 0x55b4cb995660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.564+0000 7fd4de59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993270 0x55b4cb995120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55444/0 (socket says 172.21.15.29:55444) 2024-03-11T19:13:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.564+0000 7fd4ded9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb998dd0 0x55b4cb9a4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.564+0000 7fd4de59c640 1 -- 172.21.15.29:0/2870827011 learned_addr learned my addr 172.21.15.29:0/2870827011 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.565+0000 7fd4de59c640 1 -- 172.21.15.29:0/2870827011 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb998dd0 msgr2=0x55b4cb9a4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.565+0000 7fd4de59c640 1 --2- 172.21.15.29:0/2870827011 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb998dd0 0x55b4cb9a4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.565+0000 7fd4de59c640 1 -- 172.21.15.29:0/2870827011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b4cb993c60 msgr2=0x55b4cb995660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.565+0000 7fd4de59c640 1 --2- 172.21.15.29:0/2870827011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b4cb993c60 0x55b4cb995660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.565+0000 7fd4de59c640 1 -- 172.21.15.29:0/2870827011 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4cb896c20 con 0x55b4cb993270 2024-03-11T19:13:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.565+0000 7fd4de59c640 1 --2- 172.21.15.29:0/2870827011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993270 0x55b4cb995120 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd4d40bca90 tx=0x7fd4d40ef040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8848d7252b287708 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.565+0000 7fd4c7fff640 1 -- 172.21.15.29:0/2870827011 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d40d4070 con 0x55b4cb993270 2024-03-11T19:13:53.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.565+0000 7fd4c7fff640 1 -- 172.21.15.29:0/2870827011 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd4d40f2e90 con 0x55b4cb993270 2024-03-11T19:13:53.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.565+0000 7fd4c7fff640 1 -- 172.21.15.29:0/2870827011 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d4003d70 con 0x55b4cb993270 2024-03-11T19:13:53.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.566+0000 7fd4e75ba740 1 -- 172.21.15.29:0/2870827011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993270 msgr2=0x55b4cb995120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.566+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/2870827011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993270 0x55b4cb995120 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd4d40bca90 tx=0x7fd4d40ef040 comp rx=0 tx=0).stop 2024-03-11T19:13:53.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.566+0000 7fd4e75ba740 1 -- 172.21.15.29:0/2870827011 shutdown_connections 2024-03-11T19:13:53.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.566+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/2870827011 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb998dd0 0x55b4cb9a4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.566+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/2870827011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b4cb993c60 0x55b4cb995660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.566+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/2870827011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993270 0x55b4cb995120 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.566+0000 7fd4e75ba740 1 -- 172.21.15.29:0/2870827011 >> 172.21.15.29:0/2870827011 conn(0x55b4cb98e7e0 msgr2=0x55b4cb98ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.566+0000 7fd4e75ba740 1 -- 172.21.15.29:0/2870827011 shutdown_connections 2024-03-11T19:13:53.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.566+0000 7fd4e75ba740 1 -- 172.21.15.29:0/2870827011 wait complete. 2024-03-11T19:13:53.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.567+0000 7fd4e75ba740 1 -- start start 2024-03-11T19:13:53.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.567+0000 7fd4e75ba740 1 Processor -- start 2024-03-11T19:13:53.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.568+0000 7fd4e75ba740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993270 0x55b4cb9a4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.568+0000 7fd4e75ba740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb993c60 0x55b4cb9a4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.568+0000 7fd4e75ba740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b4cb998dd0 0x55b4cb9a2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.568+0000 7fd4e75ba740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4cb9a32e0 con 0x55b4cb998dd0 2024-03-11T19:13:53.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.568+0000 7fd4e75ba740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4cb9a3450 con 0x55b4cb993270 2024-03-11T19:13:53.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.568+0000 7fd4e75ba740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4cb9a3610 con 0x55b4cb993c60 2024-03-11T19:13:53.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.568+0000 7fd4e75ba740 10 client.0 _ll_register_callbacks cb 0x7fd4d0009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:53.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.568+0000 7fd4de59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993270 0x55b4cb9a4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.569+0000 7fd4de59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993270 0x55b4cb9a4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55446/0 (socket says 172.21.15.29:55446) 2024-03-11T19:13:53.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.569+0000 7fd4de59c640 1 -- 172.21.15.29:0/646823483 learned_addr learned my addr 172.21.15.29:0/646823483 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.569+0000 7fd4de59c640 1 -- 172.21.15.29:0/646823483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb993c60 msgr2=0x55b4cb9a4a40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:13:53.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.569+0000 7fd4de59c640 1 --2- 172.21.15.29:0/646823483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb993c60 0x55b4cb9a4a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.569+0000 7fd4de59c640 1 -- 172.21.15.29:0/646823483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b4cb998dd0 msgr2=0x55b4cb9a2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.569+0000 7fd4de59c640 1 --2- 172.21.15.29:0/646823483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b4cb998dd0 0x55b4cb9a2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.569+0000 7fd4de59c640 1 -- 172.21.15.29:0/646823483 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e01a5d50 con 0x55b4cb993270 2024-03-11T19:13:53.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.569+0000 7fd4de59c640 1 --2- 172.21.15.29:0/646823483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993270 0x55b4cb9a4500 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd4d40b8f60 tx=0x7fd4d4009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16ead00f48586c24 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.569+0000 7fd4c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:13:53.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.570+0000 7fd4c5ffb640 1 -- 172.21.15.29:0/646823483 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d40d4050 con 0x55b4cb993270 2024-03-11T19:13:53.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.570+0000 7fd4c5ffb640 1 -- 172.21.15.29:0/646823483 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd4d40bae00 con 0x55b4cb993270 2024-03-11T19:13:53.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.570+0000 7fd4e75ba740 1 -- 172.21.15.29:0/646823483 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd4d00035f0 con 0x55b4cb993270 2024-03-11T19:13:53.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.570+0000 7fd4c5ffb640 1 -- 172.21.15.29:0/646823483 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4d40c0cd0 con 0x55b4cb993270 2024-03-11T19:13:53.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.570+0000 7fd4e75ba740 10 client.25976 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:53.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.570+0000 7fd4e75ba740 1 -- 172.21.15.29:0/646823483 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55b4cb8bc8c0 con 0x55b4cb993270 2024-03-11T19:13:53.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.571+0000 7fd4c5ffb640 1 -- 172.21.15.29:0/646823483 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd4d4004dc0 con 0x55b4cb993270 2024-03-11T19:13:53.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.571+0000 7fd4c5ffb640 1 -- 172.21.15.29:0/646823483 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd4d403c070 con 0x55b4cb993270 2024-03-11T19:13:53.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.571+0000 7fd4e75ba740 10 client.25976 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:53.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.571+0000 7fd4e75ba740 10 client.25976 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:13:53.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.571+0000 7fd4e75ba740 20 client.25976 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:53.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.571+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/646823483 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55b4cba00a90 0x55b4cba02eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.571+0000 7fd4e75ba740 4 client.25976 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:13:53.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.571+0000 7fd4e75ba740 1 -- 172.21.15.29:0/646823483 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7fd4d00035f0 con 0x55b4cba00a90 2024-03-11T19:13:53.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.571+0000 7fd4ddd9b640 1 --2- 172.21.15.29:0/646823483 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55b4cba00a90 0x55b4cba02eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.572+0000 7fd4c5ffb640 10 client.25976 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:13:53.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.572+0000 7fd4ddd9b640 1 --2- 172.21.15.29:0/646823483 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55b4cba00a90 0x55b4cba02eb0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4b2647ffcfe9e209 server_cookie=1876afa44e304dcc in_seq=0 out_seq=0 2024-03-11T19:13:53.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.572+0000 7fd4c5ffb640 1 -- 172.21.15.29:0/646823483 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fd4d00035f0 con 0x55b4cba00a90 2024-03-11T19:13:53.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.572+0000 7fd4c5ffb640 10 client.25976 handle_command_reply: tid=0 2024-03-11T19:13:53.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.572+0000 7fd4e75ba740 1 client.25976 shutdown 2024-03-11T19:13:53.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.573+0000 7fd4e75ba740 1 -- 172.21.15.29:0/646823483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993270 msgr2=0x55b4cb9a4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.573+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/646823483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993270 0x55b4cb9a4500 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd4d40b8f60 tx=0x7fd4d4009260 comp rx=0 tx=0).stop 2024-03-11T19:13:53.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.573+0000 7fd4e75ba740 1 -- 172.21.15.29:0/646823483 shutdown_connections 2024-03-11T19:13:53.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.573+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/646823483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b4cb998dd0 0x55b4cb9a2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.573+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/646823483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b4cb993c60 0x55b4cb9a4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.573+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/646823483 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55b4cba00a90 0x55b4cba02eb0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.573+0000 7fd4e75ba740 1 --2- 172.21.15.29:0/646823483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b4cb993270 0x55b4cb9a4500 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.573+0000 7fd4c5ffb640 0 client.25976 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:13:53.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.573+0000 7fd4e75ba740 1 -- 172.21.15.29:0/646823483 >> 172.21.15.29:0/646823483 conn(0x55b4cb98e7e0 msgr2=0x55b4cb995120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.573+0000 7fd4e75ba740 1 -- 172.21.15.29:0/646823483 shutdown_connections 2024-03-11T19:13:53.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.573+0000 7fd4e75ba740 1 -- 172.21.15.29:0/646823483 wait complete. 2024-03-11T19:13:53.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.573+0000 7fd4e75ba740 20 client.25976 trim_cache size 0 max 16384 2024-03-11T19:13:53.578 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:13:53.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.574+0000 7fd4e49e2640 1 -- 172.21.15.29:0/1915160017 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd4c80a1b80 msgr2=0x7fd4c80a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.574+0000 7fd4e49e2640 1 --2- 172.21.15.29:0/1915160017 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd4c80a1b80 0x7fd4c80a4040 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x55b4cb9944c0 tx=0x7fd4d8009500 comp rx=0 tx=0).stop 2024-03-11T19:13:53.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.574+0000 7fd4e49e2640 1 -- 172.21.15.29:0/1915160017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4e00ff9f0 msgr2=0x7fd4e019d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.574+0000 7fd4e49e2640 1 --2- 172.21.15.29:0/1915160017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4e00ff9f0 0x7fd4e019d750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd4d402c7d0 tx=0x7fd4d4004270 comp rx=0 tx=0).stop 2024-03-11T19:13:53.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.574+0000 7fd4e49e2640 1 -- 172.21.15.29:0/1915160017 shutdown_connections 2024-03-11T19:13:53.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.574+0000 7fd4e49e2640 1 --2- 172.21.15.29:0/1915160017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4e0104d10 0x7fd4e01a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.574+0000 7fd4e49e2640 1 --2- 172.21.15.29:0/1915160017 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd4c80a1b80 0x7fd4c80a4040 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.574+0000 7fd4e49e2640 1 --2- 172.21.15.29:0/1915160017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4e0102360 0x7fd4e019dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.574+0000 7fd4e49e2640 1 --2- 172.21.15.29:0/1915160017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4e00ff9f0 0x7fd4e019d750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.574+0000 7fd4e49e2640 1 -- 172.21.15.29:0/1915160017 >> 172.21.15.29:0/1915160017 conn(0x7fd4e00fb610 msgr2=0x7fd4e0102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.575+0000 7fd4e49e2640 1 -- 172.21.15.29:0/1915160017 shutdown_connections 2024-03-11T19:13:53.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.575+0000 7fd4e49e2640 1 -- 172.21.15.29:0/1915160017 wait complete. 2024-03-11T19:13:53.596 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:13:53.596 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:13:53.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:53 smithi029 ceph-mon[32439]: pgmap v473: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-11T19:13:53.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:53 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1050155584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:13:53.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.764+0000 7f928c5f4640 1 -- 172.21.15.29:0/4162437750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92841005a0 msgr2=0x7f92841009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.764+0000 7f928c5f4640 1 --2- 172.21.15.29:0/4162437750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92841005a0 0x7f92841009c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9280009900 tx=0x7f928002bc30 comp rx=0 tx=0).stop 2024-03-11T19:13:53.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.764+0000 7f928c5f4640 1 -- 172.21.15.29:0/4162437750 shutdown_connections 2024-03-11T19:13:53.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.764+0000 7f928c5f4640 1 --2- 172.21.15.29:0/4162437750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9284101c70 0x7f9284108500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.764+0000 7f928c5f4640 1 --2- 172.21.15.29:0/4162437750 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92841012b0 0x7f9284101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.764+0000 7f928c5f4640 1 --2- 172.21.15.29:0/4162437750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92841005a0 0x7f92841009c0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.764+0000 7f928c5f4640 1 -- 172.21.15.29:0/4162437750 >> 172.21.15.29:0/4162437750 conn(0x7f92840fad70 msgr2=0x7f92840fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.765+0000 7f928c5f4640 1 -- 172.21.15.29:0/4162437750 shutdown_connections 2024-03-11T19:13:53.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.765+0000 7f928c5f4640 1 -- 172.21.15.29:0/4162437750 wait complete. 2024-03-11T19:13:53.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.765+0000 7f928c5f4640 1 Processor -- start 2024-03-11T19:13:53.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.765+0000 7f928c5f4640 1 -- start start 2024-03-11T19:13:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.766+0000 7f928c5f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92841005a0 0x7f9284194fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.766+0000 7f928c5f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92841012b0 0x7f92841954f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.766+0000 7f928c5f4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9284101c70 0x7f928419c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.766+0000 7f928c5f4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928419caa0 con 0x7f92841012b0 2024-03-11T19:13:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.766+0000 7f928c5f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928419cc10 con 0x7f92841005a0 2024-03-11T19:13:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.766+0000 7f928c5f4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928419cdd0 con 0x7f9284101c70 2024-03-11T19:13:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.766+0000 7f9289b8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92841012b0 0x7f92841954f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.766+0000 7f9289b8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92841012b0 0x7f92841954f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41468/0 (socket says 172.21.15.29:41468) 2024-03-11T19:13:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.766+0000 7f9289b8f640 1 -- 172.21.15.29:0/1796491461 learned_addr learned my addr 172.21.15.29:0/1796491461 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.766+0000 7f928a390640 1 --2- 172.21.15.29:0/1796491461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92841005a0 0x7f9284194fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.766+0000 7f928ab91640 1 --2- 172.21.15.29:0/1796491461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9284101c70 0x7f928419c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.766+0000 7f9289b8f640 1 -- 172.21.15.29:0/1796491461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9284101c70 msgr2=0x7f928419c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.766+0000 7f9289b8f640 1 --2- 172.21.15.29:0/1796491461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9284101c70 0x7f928419c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.766+0000 7f9289b8f640 1 -- 172.21.15.29:0/1796491461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92841005a0 msgr2=0x7f9284194fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.766+0000 7f9289b8f640 1 --2- 172.21.15.29:0/1796491461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92841005a0 0x7f9284194fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.766+0000 7f9289b8f640 1 -- 172.21.15.29:0/1796491461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927400b6a0 con 0x7f92841012b0 2024-03-11T19:13:53.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.766+0000 7f928ab91640 1 --2- 172.21.15.29:0/1796491461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9284101c70 0x7f928419c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:13:53.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.766+0000 7f928a390640 1 --2- 172.21.15.29:0/1796491461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92841005a0 0x7f9284194fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:53.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.767+0000 7f9289b8f640 1 --2- 172.21.15.29:0/1796491461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92841012b0 0x7f92841954f0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9274010b80 tx=0x7f927400e490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.767+0000 7f927b7fe640 1 -- 172.21.15.29:0/1796491461 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927400eb40 con 0x7f92841012b0 2024-03-11T19:13:53.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.767+0000 7f927b7fe640 1 -- 172.21.15.29:0/1796491461 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9274011040 con 0x7f92841012b0 2024-03-11T19:13:53.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.767+0000 7f928c5f4640 1 -- 172.21.15.29:0/1796491461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9280009560 con 0x7f92841012b0 2024-03-11T19:13:53.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.767+0000 7f927b7fe640 1 -- 172.21.15.29:0/1796491461 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9274003bb0 con 0x7f92841012b0 2024-03-11T19:13:53.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.767+0000 7f928c5f4640 1 -- 172.21.15.29:0/1796491461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f928419d430 con 0x7f92841012b0 2024-03-11T19:13:53.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.770+0000 7f927b7fe640 1 -- 172.21.15.29:0/1796491461 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9274016480 con 0x7f92841012b0 2024-03-11T19:13:53.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.771+0000 7f927b7fe640 1 --2- 172.21.15.29:0/1796491461 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f92700a1b80 0x7f92700a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.771+0000 7f927b7fe640 1 -- 172.21.15.29:0/1796491461 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f92740cf0e0 con 0x7f92841012b0 2024-03-11T19:13:53.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.772+0000 7f928a390640 1 --2- 172.21.15.29:0/1796491461 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f92700a1b80 0x7f92700a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.772+0000 7f928f1c9740 1 Processor -- start 2024-03-11T19:13:53.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.773+0000 7f928f1c9740 1 -- start start 2024-03-11T19:13:53.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.773+0000 7f928f1c9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a773060680 0x55a773060aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.773+0000 7f928f1c9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a773061070 0x55a773065b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.773+0000 7f928f1c9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7730661e0 0x55a773068670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.773+0000 7f9289b8f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a773061070 0x55a773065b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.773+0000 7f928ab91640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7730661e0 0x55a773068670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.773+0000 7f928a390640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a773060680 0x55a773060aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.773+0000 7f928ab91640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7730661e0 0x55a773068670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:34876/0 (socket says 172.21.15.29:34876) 2024-03-11T19:13:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.773+0000 7f928ab91640 1 -- 172.21.15.29:0/1788549315 learned_addr learned my addr 172.21.15.29:0/1788549315 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.773+0000 7f928a390640 1 --2- 172.21.15.29:0/1796491461 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f92700a1b80 0x7f92700a4040 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f928002c240 tx=0x7f928002c770 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.773+0000 7f928f1c9740 1 -- 172.21.15.29:0/1788549315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a773018960 con 0x55a773060680 2024-03-11T19:13:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.773+0000 7f928f1c9740 1 -- 172.21.15.29:0/1788549315 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a772fcc310 con 0x55a773061070 2024-03-11T19:13:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.773+0000 7f928f1c9740 1 -- 172.21.15.29:0/1788549315 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a772fc9840 con 0x55a7730661e0 2024-03-11T19:13:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.774+0000 7f928ab91640 1 -- 172.21.15.29:0/1788549315 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a773061070 msgr2=0x55a773065b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.774+0000 7f928ab91640 1 --2- 172.21.15.29:0/1788549315 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a773061070 0x55a773065b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.774+0000 7f928ab91640 1 -- 172.21.15.29:0/1788549315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a773060680 msgr2=0x55a773060aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.774+0000 7f928ab91640 1 --2- 172.21.15.29:0/1788549315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a773060680 0x55a773060aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.774+0000 7f928ab91640 1 -- 172.21.15.29:0/1788549315 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a772f87060 con 0x55a7730661e0 2024-03-11T19:13:53.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.774+0000 7f928a390640 1 --2- 172.21.15.29:0/1788549315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a773060680 0x55a773060aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:53.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.774+0000 7f928ab91640 1 --2- 172.21.15.29:0/1788549315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7730661e0 0x55a773068670 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f927c009fd0 tx=0x7f927c0108e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea37c57cbe318ddf server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.774+0000 7f9278bfe640 1 -- 172.21.15.29:0/1788549315 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927c011760 con 0x55a7730661e0 2024-03-11T19:13:53.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.774+0000 7f9278bfe640 1 -- 172.21.15.29:0/1788549315 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f927c015070 con 0x55a7730661e0 2024-03-11T19:13:53.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.774+0000 7f9278bfe640 1 -- 172.21.15.29:0/1788549315 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927c01bd30 con 0x55a7730661e0 2024-03-11T19:13:53.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.775+0000 7f928f1c9740 1 -- 172.21.15.29:0/1788549315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7730661e0 msgr2=0x55a773068670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.775+0000 7f928f1c9740 1 --2- 172.21.15.29:0/1788549315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7730661e0 0x55a773068670 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f927c009fd0 tx=0x7f927c0108e0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.775+0000 7f928f1c9740 1 -- 172.21.15.29:0/1788549315 shutdown_connections 2024-03-11T19:13:53.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.775+0000 7f928f1c9740 1 --2- 172.21.15.29:0/1788549315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7730661e0 0x55a773068670 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.775+0000 7f928f1c9740 1 --2- 172.21.15.29:0/1788549315 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a773061070 0x55a773065b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.775+0000 7f928f1c9740 1 --2- 172.21.15.29:0/1788549315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a773060680 0x55a773060aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.775+0000 7f928f1c9740 1 -- 172.21.15.29:0/1788549315 >> 172.21.15.29:0/1788549315 conn(0x55a7730577e0 msgr2=0x55a773057be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.776+0000 7f928f1c9740 1 -- 172.21.15.29:0/1788549315 shutdown_connections 2024-03-11T19:13:53.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.776+0000 7f928f1c9740 1 -- 172.21.15.29:0/1788549315 wait complete. 2024-03-11T19:13:53.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.776+0000 7f928f1c9740 1 -- start start 2024-03-11T19:13:53.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.777+0000 7f928f1c9740 1 Processor -- start 2024-03-11T19:13:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.777+0000 7f928f1c9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a773060680 0x55a77306a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.777+0000 7f928f1c9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a773061070 0x55a77306ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.777+0000 7f928f1c9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7730661e0 0x55a773071b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.777+0000 7f928f1c9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a772fc5020 con 0x55a773061070 2024-03-11T19:13:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.777+0000 7f928f1c9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a772c98430 con 0x55a7730661e0 2024-03-11T19:13:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.777+0000 7f928f1c9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a772c54e00 con 0x55a773060680 2024-03-11T19:13:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.777+0000 7f928f1c9740 10 client.0 _ll_register_callbacks cb 0x55a772f6ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.777+0000 7f9289b8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a773061070 0x55a77306ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.777+0000 7f928a390640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a773060680 0x55a77306a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.777+0000 7f9289b8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a773061070 0x55a77306ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41486/0 (socket says 172.21.15.29:41486) 2024-03-11T19:13:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.777+0000 7f928ab91640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7730661e0 0x55a773071b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.777+0000 7f9289b8f640 1 -- 172.21.15.29:0/1908359292 learned_addr learned my addr 172.21.15.29:0/1908359292 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.777+0000 7f9289b8f640 1 -- 172.21.15.29:0/1908359292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a773060680 msgr2=0x55a77306a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.777+0000 7f9289b8f640 1 --2- 172.21.15.29:0/1908359292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a773060680 0x55a77306a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.777+0000 7f9289b8f640 1 -- 172.21.15.29:0/1908359292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7730661e0 msgr2=0x55a773071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.778+0000 7f9289b8f640 1 --2- 172.21.15.29:0/1908359292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7730661e0 0x55a773071b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.778+0000 7f9289b8f640 1 -- 172.21.15.29:0/1908359292 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f927c009cb0 con 0x55a773061070 2024-03-11T19:13:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.778+0000 7f9289b8f640 1 --2- 172.21.15.29:0/1908359292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a773061070 0x55a77306ab20 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f92740ccbe0 tx=0x7f927409c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da38879f01c1014a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.778+0000 7f926dffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:13:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.778+0000 7f926dffb640 1 -- 172.21.15.29:0/1908359292 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92740a47f0 con 0x55a773061070 2024-03-11T19:13:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.778+0000 7f926dffb640 1 -- 172.21.15.29:0/1908359292 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9274096bc0 con 0x55a773061070 2024-03-11T19:13:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.778+0000 7f928f1c9740 1 -- 172.21.15.29:0/1908359292 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a772cc47e0 con 0x55a773061070 2024-03-11T19:13:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.778+0000 7f926dffb640 1 -- 172.21.15.29:0/1908359292 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9274096d80 con 0x55a773061070 2024-03-11T19:13:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.778+0000 7f928f1c9740 10 client.16479 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.778+0000 7f928f1c9740 1 -- 172.21.15.29:0/1908359292 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a772fc36a0 con 0x55a773061070 2024-03-11T19:13:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.779+0000 7f926dffb640 1 -- 172.21.15.29:0/1908359292 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55a772cc47e0 con 0x55a773061070 2024-03-11T19:13:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.779+0000 7f926dffb640 1 -- 172.21.15.29:0/1908359292 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f927409af10 con 0x55a773061070 2024-03-11T19:13:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.779+0000 7f928f1c9740 10 client.16479 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.779+0000 7f928f1c9740 10 client.16479 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:13:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.779+0000 7f928f1c9740 20 client.16479 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.779+0000 7f928f1c9740 1 --2- 172.21.15.29:0/1908359292 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55a7730767d0 0x55a773078bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.779+0000 7f928f1c9740 4 client.16479 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:13:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.779+0000 7f928f1c9740 1 -- 172.21.15.29:0/1908359292 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a772d01720 con 0x55a7730767d0 2024-03-11T19:13:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.779+0000 7f928a390640 1 --2- 172.21.15.29:0/1908359292 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55a7730767d0 0x55a773078bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.780+0000 7f926dffb640 10 client.16479 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:53.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.780+0000 7f928a390640 1 --2- 172.21.15.29:0/1908359292 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55a7730767d0 0x55a773078bf0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3d6c7f1e469679d server_cookie=c71d9c8535aaaa2d in_seq=0 out_seq=0 2024-03-11T19:13:53.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.781+0000 7f926dffb640 1 -- 172.21.15.29:0/1908359292 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a772d01720 con 0x55a7730767d0 2024-03-11T19:13:53.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.781+0000 7f926dffb640 10 client.16479 handle_command_reply: tid=0 2024-03-11T19:13:53.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.781+0000 7f928f1c9740 1 client.16479 shutdown 2024-03-11T19:13:53.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.781+0000 7f928f1c9740 1 -- 172.21.15.29:0/1908359292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a773061070 msgr2=0x55a77306ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.781+0000 7f928f1c9740 1 --2- 172.21.15.29:0/1908359292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a773061070 0x55a77306ab20 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f92740ccbe0 tx=0x7f927409c810 comp rx=0 tx=0).stop 2024-03-11T19:13:53.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.781+0000 7f928f1c9740 1 -- 172.21.15.29:0/1908359292 shutdown_connections 2024-03-11T19:13:53.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.781+0000 7f928f1c9740 1 --2- 172.21.15.29:0/1908359292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7730661e0 0x55a773071b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.781+0000 7f928f1c9740 1 --2- 172.21.15.29:0/1908359292 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55a7730767d0 0x55a773078bf0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.781+0000 7f926dffb640 0 client.16479 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:53.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.781+0000 7f928f1c9740 1 --2- 172.21.15.29:0/1908359292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a773061070 0x55a77306ab20 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.781+0000 7f928f1c9740 1 --2- 172.21.15.29:0/1908359292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a773060680 0x55a77306a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.781+0000 7f928f1c9740 1 -- 172.21.15.29:0/1908359292 >> 172.21.15.29:0/1908359292 conn(0x55a7730577e0 msgr2=0x55a773056b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.781+0000 7f928f1c9740 1 -- 172.21.15.29:0/1908359292 shutdown_connections 2024-03-11T19:13:53.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.782+0000 7f928f1c9740 1 -- 172.21.15.29:0/1908359292 wait complete. 2024-03-11T19:13:53.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.782+0000 7f928f1c9740 20 client.16479 trim_cache size 0 max 16384 2024-03-11T19:13:53.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.804+0000 7f928f1c9740 1 Processor -- start 2024-03-11T19:13:53.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.804+0000 7f928f1c9740 1 -- start start 2024-03-11T19:13:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.804+0000 7f928f1c9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a773060680 0x55a77306a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.805+0000 7f928f1c9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a773061070 0x55a77306ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.805+0000 7f928f1c9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7730661e0 0x55a773071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.805+0000 7f928f1c9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a772f4a130 con 0x55a7730661e0 2024-03-11T19:13:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.805+0000 7f928ab91640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7730661e0 0x55a773071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.805+0000 7f928ab91640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7730661e0 0x55a773071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41492/0 (socket says 172.21.15.29:41492) 2024-03-11T19:13:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.805+0000 7f928ab91640 1 -- 172.21.15.29:0/872540303 learned_addr learned my addr 172.21.15.29:0/872540303 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.805+0000 7f9289b8f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a773061070 0x55a77306ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.805+0000 7f928f1c9740 1 -- 172.21.15.29:0/872540303 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a773061560 con 0x55a773061070 2024-03-11T19:13:53.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.805+0000 7f928f1c9740 1 -- 172.21.15.29:0/872540303 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a773049a40 con 0x55a773060680 2024-03-11T19:13:53.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.805+0000 7f928ab91640 1 -- 172.21.15.29:0/872540303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a773060680 msgr2=0x55a77306a5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:13:53.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.805+0000 7f928ab91640 1 --2- 172.21.15.29:0/872540303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a773060680 0x55a77306a5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.805+0000 7f928ab91640 1 -- 172.21.15.29:0/872540303 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a773061070 msgr2=0x55a77306ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.805+0000 7f928ab91640 1 --2- 172.21.15.29:0/872540303 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a773061070 0x55a77306ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.805+0000 7f928ab91640 1 -- 172.21.15.29:0/872540303 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a772f5fc20 con 0x55a7730661e0 2024-03-11T19:13:53.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.806+0000 7f9289b8f640 1 --2- 172.21.15.29:0/872540303 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a773061070 0x55a77306ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:53.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.806+0000 7f928ab91640 1 --2- 172.21.15.29:0/872540303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7730661e0 0x55a773071b20 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x55a773058750 tx=0x7f927c009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3612795f159a67d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.806+0000 7f926ffff640 1 -- 172.21.15.29:0/872540303 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927c021050 con 0x55a7730661e0 2024-03-11T19:13:53.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.806+0000 7f926ffff640 1 -- 172.21.15.29:0/872540303 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f927c022040 con 0x55a7730661e0 2024-03-11T19:13:53.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.806+0000 7f926ffff640 1 -- 172.21.15.29:0/872540303 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927c01b6b0 con 0x55a7730661e0 2024-03-11T19:13:53.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.807+0000 7f928f1c9740 1 -- 172.21.15.29:0/872540303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7730661e0 msgr2=0x55a773071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.807+0000 7f928f1c9740 1 --2- 172.21.15.29:0/872540303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7730661e0 0x55a773071b20 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x55a773058750 tx=0x7f927c009a20 comp rx=0 tx=0).stop 2024-03-11T19:13:53.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.807+0000 7f928f1c9740 1 -- 172.21.15.29:0/872540303 shutdown_connections 2024-03-11T19:13:53.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.807+0000 7f928f1c9740 1 --2- 172.21.15.29:0/872540303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7730661e0 0x55a773071b20 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.807+0000 7f928f1c9740 1 --2- 172.21.15.29:0/872540303 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a773061070 0x55a77306ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.807+0000 7f928f1c9740 1 --2- 172.21.15.29:0/872540303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a773060680 0x55a77306a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.807+0000 7f928f1c9740 1 -- 172.21.15.29:0/872540303 >> 172.21.15.29:0/872540303 conn(0x55a7730577e0 msgr2=0x55a773057bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.808+0000 7f928f1c9740 1 -- 172.21.15.29:0/872540303 shutdown_connections 2024-03-11T19:13:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.808+0000 7f928f1c9740 1 -- 172.21.15.29:0/872540303 wait complete. 2024-03-11T19:13:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.808+0000 7f928f1c9740 1 -- start start 2024-03-11T19:13:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.808+0000 7f928f1c9740 1 Processor -- start 2024-03-11T19:13:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.809+0000 7f928f1c9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a773060680 0x55a7730701c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.809+0000 7f928f1c9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a773061070 0x55a773070700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.809+0000 7f928f1c9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7730661e0 0x55a773070c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.809+0000 7f928a390640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a773060680 0x55a7730701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.809+0000 7f928a390640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a773060680 0x55a7730701c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55508/0 (socket says 172.21.15.29:55508) 2024-03-11T19:13:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.809+0000 7f9289b8f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a773061070 0x55a773070700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.809+0000 7f9289b8f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a773061070 0x55a773070700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:34902/0 (socket says 172.21.15.29:34902) 2024-03-11T19:13:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.809+0000 7f928ab91640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7730661e0 0x55a773070c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.809+0000 7f928ab91640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7730661e0 0x55a773070c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41498/0 (socket says 172.21.15.29:41498) 2024-03-11T19:13:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.809+0000 7f928a390640 1 -- 172.21.15.29:0/569633510 learned_addr learned my addr 172.21.15.29:0/569633510 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.809+0000 7f928f1c9740 1 -- 172.21.15.29:0/569633510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a773071e10 con 0x55a7730661e0 2024-03-11T19:13:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.809+0000 7f928f1c9740 1 -- 172.21.15.29:0/569633510 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7730ccce0 con 0x55a773060680 2024-03-11T19:13:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.809+0000 7f928f1c9740 1 -- 172.21.15.29:0/569633510 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7730cce20 con 0x55a773061070 2024-03-11T19:13:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.809+0000 7f928f1c9740 10 client.0 _ll_register_callbacks cb 0x7f927c009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.809+0000 7f928a390640 1 -- 172.21.15.29:0/569633510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a773061070 msgr2=0x55a773070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.809+0000 7f928a390640 1 --2- 172.21.15.29:0/569633510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a773061070 0x55a773070700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.809+0000 7f928a390640 1 -- 172.21.15.29:0/569633510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7730661e0 msgr2=0x55a773070c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.809+0000 7f928a390640 1 --2- 172.21.15.29:0/569633510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7730661e0 0x55a773070c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.809+0000 7f928a390640 1 -- 172.21.15.29:0/569633510 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a772fc36a0 con 0x55a773060680 2024-03-11T19:13:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.810+0000 7f928a390640 1 --2- 172.21.15.29:0/569633510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a773060680 0x55a7730701c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9280009900 tx=0x7f928002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a538515d13ecd28c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.810+0000 7f926dffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:13:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.810+0000 7f926dffb640 1 -- 172.21.15.29:0/569633510 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92800044b0 con 0x55a773060680 2024-03-11T19:13:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.810+0000 7f926dffb640 1 -- 172.21.15.29:0/569633510 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9280053040 con 0x55a773060680 2024-03-11T19:13:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.810+0000 7f928f1c9740 1 -- 172.21.15.29:0/569633510 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f927c015070 con 0x55a773060680 2024-03-11T19:13:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.810+0000 7f926dffb640 1 -- 172.21.15.29:0/569633510 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92800563a0 con 0x55a773060680 2024-03-11T19:13:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.810+0000 7f928f1c9740 10 client.25994 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.810+0000 7f928f1c9740 1 -- 172.21.15.29:0/569633510 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a772f87060 con 0x55a773060680 2024-03-11T19:13:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.811+0000 7f926dffb640 1 -- 172.21.15.29:0/569633510 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f927c015070 con 0x55a773060680 2024-03-11T19:13:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.811+0000 7f926dffb640 1 -- 172.21.15.29:0/569633510 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9280005590 con 0x55a773060680 2024-03-11T19:13:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.812+0000 7f928f1c9740 10 client.25994 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.812+0000 7f928f1c9740 10 client.25994 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:13:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.812+0000 7f928f1c9740 20 client.25994 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.812+0000 7f928f1c9740 1 --2- 172.21.15.29:0/569633510 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55a7730767d0 0x55a7730d0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.812+0000 7f928f1c9740 4 client.25994 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:13:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.812+0000 7f928f1c9740 1 -- 172.21.15.29:0/569633510 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a772d50890 con 0x55a7730767d0 2024-03-11T19:13:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.812+0000 7f9289b8f640 1 --2- 172.21.15.29:0/569633510 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55a7730767d0 0x55a7730d0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.812+0000 7f9289b8f640 1 --2- 172.21.15.29:0/569633510 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55a7730767d0 0x55a7730d0990 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1e21324bb288d6a server_cookie=906f360628cd0f6e in_seq=0 out_seq=0 2024-03-11T19:13:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.812+0000 7f926dffb640 10 client.25994 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.813+0000 7f926dffb640 1 -- 172.21.15.29:0/569633510 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a772d50890 con 0x55a7730767d0 2024-03-11T19:13:53.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.813+0000 7f926dffb640 10 client.25994 handle_command_reply: tid=0 2024-03-11T19:13:53.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.813+0000 7f928f1c9740 1 client.25994 shutdown 2024-03-11T19:13:53.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.813+0000 7f928f1c9740 1 -- 172.21.15.29:0/569633510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a773060680 msgr2=0x55a7730701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.813+0000 7f928f1c9740 1 --2- 172.21.15.29:0/569633510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a773060680 0x55a7730701c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9280009900 tx=0x7f928002bc30 comp rx=0 tx=0).stop 2024-03-11T19:13:53.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.813+0000 7f928f1c9740 1 -- 172.21.15.29:0/569633510 shutdown_connections 2024-03-11T19:13:53.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.813+0000 7f928f1c9740 1 --2- 172.21.15.29:0/569633510 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55a7730767d0 0x55a7730d0990 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.813+0000 7f928f1c9740 1 --2- 172.21.15.29:0/569633510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7730661e0 0x55a773070c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.813+0000 7f928f1c9740 1 --2- 172.21.15.29:0/569633510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a773061070 0x55a773070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.813+0000 7f928f1c9740 1 --2- 172.21.15.29:0/569633510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a773060680 0x55a7730701c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.813+0000 7f928f1c9740 1 -- 172.21.15.29:0/569633510 >> 172.21.15.29:0/569633510 conn(0x55a7730577e0 msgr2=0x55a7730654e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.813+0000 7f926dffb640 0 client.25994 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:53.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.813+0000 7f928f1c9740 1 -- 172.21.15.29:0/569633510 shutdown_connections 2024-03-11T19:13:53.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.813+0000 7f928f1c9740 1 -- 172.21.15.29:0/569633510 wait complete. 2024-03-11T19:13:53.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.813+0000 7f928f1c9740 20 client.25994 trim_cache size 0 max 16384 2024-03-11T19:13:53.820 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:13:53.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.814+0000 7f928c5f4640 1 -- 172.21.15.29:0/1796491461 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f92700a1b80 msgr2=0x7f92700a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.814+0000 7f928c5f4640 1 --2- 172.21.15.29:0/1796491461 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f92700a1b80 0x7f92700a4040 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f928002c240 tx=0x7f928002c770 comp rx=0 tx=0).stop 2024-03-11T19:13:53.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.814+0000 7f928c5f4640 1 -- 172.21.15.29:0/1796491461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92841012b0 msgr2=0x7f92841954f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:53.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.814+0000 7f928c5f4640 1 --2- 172.21.15.29:0/1796491461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92841012b0 0x7f92841954f0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9274010b80 tx=0x7f927400e490 comp rx=0 tx=0).stop 2024-03-11T19:13:53.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.814+0000 7f928c5f4640 1 -- 172.21.15.29:0/1796491461 shutdown_connections 2024-03-11T19:13:53.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.814+0000 7f928c5f4640 1 --2- 172.21.15.29:0/1796491461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9284101c70 0x7f928419c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.814+0000 7f928c5f4640 1 --2- 172.21.15.29:0/1796491461 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f92700a1b80 0x7f92700a4040 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.814+0000 7f928c5f4640 1 --2- 172.21.15.29:0/1796491461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92841012b0 0x7f92841954f0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.814+0000 7f928c5f4640 1 --2- 172.21.15.29:0/1796491461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92841005a0 0x7f9284194fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:53.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.814+0000 7f928c5f4640 1 -- 172.21.15.29:0/1796491461 >> 172.21.15.29:0/1796491461 conn(0x7f92840fad70 msgr2=0x7f9284104680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:53.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.814+0000 7f928c5f4640 1 -- 172.21.15.29:0/1796491461 shutdown_connections 2024-03-11T19:13:53.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:53.814+0000 7f928c5f4640 1 -- 172.21.15.29:0/1796491461 wait complete. 2024-03-11T19:13:53.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:53 smithi170 ceph-mon[35690]: pgmap v473: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-11T19:13:53.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:53 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1050155584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:13:54.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:53 smithi190 ceph-mon[36292]: pgmap v473: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-11T19:13:54.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:53 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1050155584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:13:54.837 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:13:54.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:13:54.837 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:13:55.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.012+0000 7fd24159a640 1 -- 172.21.15.29:0/2149542360 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23802cb70 con 0x7fd244101c30 2024-03-11T19:13:55.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.012+0000 7fd248965640 1 -- 172.21.15.29:0/2149542360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd244101c30 msgr2=0x7fd2441084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:55.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.012+0000 7fd248965640 1 --2- 172.21.15.29:0/2149542360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd244101c30 0x7fd2441084c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd238009900 tx=0x7fd23802bc30 comp rx=0 tx=0).stop 2024-03-11T19:13:55.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.013+0000 7fd248965640 1 -- 172.21.15.29:0/2149542360 shutdown_connections 2024-03-11T19:13:55.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.013+0000 7fd248965640 1 --2- 172.21.15.29:0/2149542360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd244101c30 0x7fd2441084c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.013+0000 7fd248965640 1 --2- 172.21.15.29:0/2149542360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd244101270 0x7fd2441016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.013+0000 7fd248965640 1 --2- 172.21.15.29:0/2149542360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd244100080 0x7fd2441004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.013+0000 7fd248965640 1 -- 172.21.15.29:0/2149542360 >> 172.21.15.29:0/2149542360 conn(0x7fd2440fb610 msgr2=0x7fd2440fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:55.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.013+0000 7fd248965640 1 -- 172.21.15.29:0/2149542360 shutdown_connections 2024-03-11T19:13:55.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.013+0000 7fd248965640 1 -- 172.21.15.29:0/2149542360 wait complete. 2024-03-11T19:13:55.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.014+0000 7fd248965640 1 Processor -- start 2024-03-11T19:13:55.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.014+0000 7fd248965640 1 -- start start 2024-03-11T19:13:55.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.014+0000 7fd248965640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd244100080 0x7fd244104010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.014+0000 7fd248965640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd244101270 0x7fd244102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.014+0000 7fd248965640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd244101c30 0x7fd244102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.014+0000 7fd248965640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2441032f0 con 0x7fd244101270 2024-03-11T19:13:55.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.014+0000 7fd248965640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd244103460 con 0x7fd244101c30 2024-03-11T19:13:55.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.014+0000 7fd248965640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2441a11b0 con 0x7fd244100080 2024-03-11T19:13:55.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.015+0000 7fd242d9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd244101c30 0x7fd244102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.015+0000 7fd241d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd244101270 0x7fd244102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.015+0000 7fd241d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd244101270 0x7fd244102650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41516/0 (socket says 172.21.15.29:41516) 2024-03-11T19:13:55.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.015+0000 7fd241d9b640 1 -- 172.21.15.29:0/2379978512 learned_addr learned my addr 172.21.15.29:0/2379978512 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:55.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.015+0000 7fd24259c640 1 --2- 172.21.15.29:0/2379978512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd244100080 0x7fd244104010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.015+0000 7fd242d9d640 1 -- 172.21.15.29:0/2379978512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd244100080 msgr2=0x7fd244104010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:55.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.015+0000 7fd242d9d640 1 --2- 172.21.15.29:0/2379978512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd244100080 0x7fd244104010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.015+0000 7fd242d9d640 1 -- 172.21.15.29:0/2379978512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd244101270 msgr2=0x7fd244102650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:55.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.015+0000 7fd242d9d640 1 --2- 172.21.15.29:0/2379978512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd244101270 0x7fd244102650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.015+0000 7fd242d9d640 1 -- 172.21.15.29:0/2379978512 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd23c009840 con 0x7fd244101c30 2024-03-11T19:13:55.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.016+0000 7fd242d9d640 1 --2- 172.21.15.29:0/2379978512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd244101c30 0x7fd244102b90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd238036040 tx=0x7fd238005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:55.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.016+0000 7fd2337fe640 1 -- 172.21.15.29:0/2379978512 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd238004490 con 0x7fd244101c30 2024-03-11T19:13:55.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.016+0000 7fd2337fe640 1 -- 172.21.15.29:0/2379978512 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd23802de90 con 0x7fd244101c30 2024-03-11T19:13:55.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.016+0000 7fd2337fe640 1 -- 172.21.15.29:0/2379978512 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd238003b40 con 0x7fd244101c30 2024-03-11T19:13:55.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.016+0000 7fd248965640 1 -- 172.21.15.29:0/2379978512 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd238009560 con 0x7fd244101c30 2024-03-11T19:13:55.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.016+0000 7fd248965640 1 -- 172.21.15.29:0/2379978512 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2441a1730 con 0x7fd244101c30 2024-03-11T19:13:55.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.019+0000 7fd2337fe640 1 -- 172.21.15.29:0/2379978512 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd2380054e0 con 0x7fd244101c30 2024-03-11T19:13:55.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.020+0000 7fd24b612740 1 Processor -- start 2024-03-11T19:13:55.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.020+0000 7fd24b612740 1 -- start start 2024-03-11T19:13:55.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.020+0000 7fd24b612740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e4fecd0 0x55601e4ff0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.020+0000 7fd24b612740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e4ffea0 0x55601e500320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.020+0000 7fd24b612740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e500860 0x55601e5070f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.020+0000 7fd24b612740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55601e465f90 con 0x55601e4fecd0 2024-03-11T19:13:55.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.020+0000 7fd24b612740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55601e4634c0 con 0x55601e4ffea0 2024-03-11T19:13:55.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.020+0000 7fd24b612740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55601e45eca0 con 0x55601e500860 2024-03-11T19:13:55.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.020+0000 7fd2337fe640 1 --2- 172.21.15.29:0/2379978512 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd22c0a1b30 0x7fd22c0a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.021+0000 7fd24259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e4ffea0 0x55601e500320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.021+0000 7fd24259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e4ffea0 0x55601e500320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55530/0 (socket says 172.21.15.29:55530) 2024-03-11T19:13:55.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.021+0000 7fd24259c640 1 -- 172.21.15.29:0/685999384 learned_addr learned my addr 172.21.15.29:0/685999384 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:55.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.021+0000 7fd241d9b640 1 --2- 172.21.15.29:0/685999384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e4fecd0 0x55601e4ff0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.021+0000 7fd2337fe640 1 -- 172.21.15.29:0/2379978512 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd2380ef3c0 con 0x7fd244101c30 2024-03-11T19:13:55.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.021+0000 7fd241d9b640 1 --2- 172.21.15.29:0/685999384 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e500860 0x55601e5070f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.021+0000 7fd24259c640 1 -- 172.21.15.29:0/685999384 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e500860 msgr2=0x55601e5070f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:55.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.021+0000 7fd242d9d640 1 --2- 172.21.15.29:0/2379978512 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd22c0a1b30 0x7fd22c0a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.022+0000 7fd24259c640 1 --2- 172.21.15.29:0/685999384 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e500860 0x55601e5070f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.022+0000 7fd24259c640 1 -- 172.21.15.29:0/685999384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e4fecd0 msgr2=0x55601e4ff0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:55.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.022+0000 7fd24259c640 1 --2- 172.21.15.29:0/685999384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e4fecd0 0x55601e4ff0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.022+0000 7fd24259c640 1 -- 172.21.15.29:0/685999384 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55601e4210b0 con 0x55601e4ffea0 2024-03-11T19:13:55.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.022+0000 7fd24259c640 1 --2- 172.21.15.29:0/685999384 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e4ffea0 0x55601e500320 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd23c010580 tx=0x7fd23c010a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3108eaa7bcde5c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:55.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.022+0000 7fd242d9d640 1 --2- 172.21.15.29:0/2379978512 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd22c0a1b30 0x7fd22c0a3ff0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd238004650 tx=0x7fd2380dc040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:55.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.022+0000 7fd230ff9640 1 -- 172.21.15.29:0/685999384 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c004d20 con 0x55601e4ffea0 2024-03-11T19:13:55.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.022+0000 7fd230ff9640 1 -- 172.21.15.29:0/685999384 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd23c004e80 con 0x55601e4ffea0 2024-03-11T19:13:55.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.022+0000 7fd230ff9640 1 -- 172.21.15.29:0/685999384 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c005590 con 0x55601e4ffea0 2024-03-11T19:13:55.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.023+0000 7fd24b612740 1 -- 172.21.15.29:0/685999384 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e4ffea0 msgr2=0x55601e500320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:55.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.023+0000 7fd24b612740 1 --2- 172.21.15.29:0/685999384 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e4ffea0 0x55601e500320 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd23c010580 tx=0x7fd23c010a40 comp rx=0 tx=0).stop 2024-03-11T19:13:55.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.023+0000 7fd24b612740 1 -- 172.21.15.29:0/685999384 shutdown_connections 2024-03-11T19:13:55.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.023+0000 7fd24b612740 1 --2- 172.21.15.29:0/685999384 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e500860 0x55601e5070f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.023+0000 7fd24b612740 1 --2- 172.21.15.29:0/685999384 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e4ffea0 0x55601e500320 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.023+0000 7fd24b612740 1 --2- 172.21.15.29:0/685999384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e4fecd0 0x55601e4ff0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.023+0000 7fd24b612740 1 -- 172.21.15.29:0/685999384 >> 172.21.15.29:0/685999384 conn(0x55601e4f9eb0 msgr2=0x55601e4fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:55.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.024+0000 7fd24b612740 1 -- 172.21.15.29:0/685999384 shutdown_connections 2024-03-11T19:13:55.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.024+0000 7fd24b612740 1 -- 172.21.15.29:0/685999384 wait complete. 2024-03-11T19:13:55.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.024+0000 7fd24b612740 1 -- start start 2024-03-11T19:13:55.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.024+0000 7fd24b612740 1 Processor -- start 2024-03-11T19:13:55.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.024+0000 7fd24b612740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e4fecd0 0x55601e5030d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.025+0000 7fd24b612740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e4ffea0 0x55601e505620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.025+0000 7fd24b612740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e500860 0x55601e507b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.025+0000 7fd24b612740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55601e403f40 con 0x55601e4ffea0 2024-03-11T19:13:55.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.025+0000 7fd24259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e4ffea0 0x55601e505620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.025+0000 7fd24259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e4ffea0 0x55601e505620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41530/0 (socket says 172.21.15.29:41530) 2024-03-11T19:13:55.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.025+0000 7fd24259c640 1 -- 172.21.15.29:0/132507648 learned_addr learned my addr 172.21.15.29:0/132507648 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:55.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.025+0000 7fd241d9b640 1 --2- 172.21.15.29:0/132507648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e4fecd0 0x55601e5030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.025+0000 7fd24b612740 1 -- 172.21.15.29:0/132507648 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55601e1324b0 con 0x55601e500860 2024-03-11T19:13:55.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.025+0000 7fd24b612740 1 -- 172.21.15.29:0/132507648 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55601e3e4670 con 0x55601e4fecd0 2024-03-11T19:13:55.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.025+0000 7fd24b612740 10 client.0 _ll_register_callbacks cb 0x55601e451f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:55.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.025+0000 7fd241d9b640 1 --2- 172.21.15.29:0/132507648 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e500860 0x55601e507b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.025+0000 7fd241d9b640 1 -- 172.21.15.29:0/132507648 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e500860 msgr2=0x55601e507b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:55.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.025+0000 7fd241d9b640 1 --2- 172.21.15.29:0/132507648 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e500860 0x55601e507b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.025+0000 7fd241d9b640 1 -- 172.21.15.29:0/132507648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e4ffea0 msgr2=0x55601e505620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:55.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.025+0000 7fd241d9b640 1 --2- 172.21.15.29:0/132507648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e4ffea0 0x55601e505620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.025+0000 7fd241d9b640 1 -- 172.21.15.29:0/132507648 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd23c009dc0 con 0x55601e4fecd0 2024-03-11T19:13:55.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.026+0000 7fd241d9b640 1 --2- 172.21.15.29:0/132507648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e4fecd0 0x55601e5030d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd234009620 tx=0x7fd234004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=921a77fbc70f0441 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:55.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.026+0000 7fd229ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:13:55.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.026+0000 7fd229ffb640 1 -- 172.21.15.29:0/132507648 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd234027040 con 0x55601e4fecd0 2024-03-11T19:13:55.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.026+0000 7fd24b612740 1 -- 172.21.15.29:0/132507648 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55601e438f40 con 0x55601e4fecd0 2024-03-11T19:13:55.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.026+0000 7fd229ffb640 1 -- 172.21.15.29:0/132507648 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd2340230f0 con 0x55601e4fecd0 2024-03-11T19:13:55.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.026+0000 7fd229ffb640 1 -- 172.21.15.29:0/132507648 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2340232b0 con 0x55601e4fecd0 2024-03-11T19:13:55.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.026+0000 7fd24b612740 10 client.26023 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:55.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.026+0000 7fd24b612740 1 -- 172.21.15.29:0/132507648 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55601e45d320 con 0x55601e4fecd0 2024-03-11T19:13:55.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.027+0000 7fd229ffb640 1 -- 172.21.15.29:0/132507648 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55601e438f40 con 0x55601e4fecd0 2024-03-11T19:13:55.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.027+0000 7fd229ffb640 1 -- 172.21.15.29:0/132507648 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd234021440 con 0x55601e4fecd0 2024-03-11T19:13:55.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.027+0000 7fd24b612740 10 client.26023 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:55.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.027+0000 7fd24b612740 10 client.26023 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:13:55.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.027+0000 7fd24b612740 20 client.26023 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:55.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.027+0000 7fd24b612740 1 --2- 172.21.15.29:0/132507648 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55601e50c840 0x55601e50eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.027+0000 7fd24b612740 4 client.26023 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:13:55.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.027+0000 7fd24b612740 1 -- 172.21.15.29:0/132507648 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55601e455310 con 0x55601e50c840 2024-03-11T19:13:55.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.027+0000 7fd24259c640 1 --2- 172.21.15.29:0/132507648 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55601e50c840 0x55601e50eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.027+0000 7fd229ffb640 10 client.26023 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:55.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.027+0000 7fd24259c640 1 --2- 172.21.15.29:0/132507648 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55601e50c840 0x55601e50eca0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6c4d8d60fdef886 server_cookie=4805a9e7b799396 in_seq=0 out_seq=0 2024-03-11T19:13:55.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.028+0000 7fd229ffb640 1 -- 172.21.15.29:0/132507648 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55601e455310 con 0x55601e50c840 2024-03-11T19:13:55.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.028+0000 7fd229ffb640 10 client.26023 handle_command_reply: tid=0 2024-03-11T19:13:55.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.028+0000 7fd24b612740 1 client.26023 shutdown 2024-03-11T19:13:55.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.028+0000 7fd24b612740 1 -- 172.21.15.29:0/132507648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e4fecd0 msgr2=0x55601e5030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:55.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.028+0000 7fd24b612740 1 --2- 172.21.15.29:0/132507648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e4fecd0 0x55601e5030d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd234009620 tx=0x7fd234004730 comp rx=0 tx=0).stop 2024-03-11T19:13:55.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.028+0000 7fd24b612740 1 -- 172.21.15.29:0/132507648 shutdown_connections 2024-03-11T19:13:55.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.028+0000 7fd24b612740 1 --2- 172.21.15.29:0/132507648 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e500860 0x55601e507b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.028+0000 7fd24b612740 1 --2- 172.21.15.29:0/132507648 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55601e50c840 0x55601e50eca0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.028+0000 7fd24b612740 1 --2- 172.21.15.29:0/132507648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e4ffea0 0x55601e505620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.028+0000 7fd24b612740 1 --2- 172.21.15.29:0/132507648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e4fecd0 0x55601e5030d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.028+0000 7fd229ffb640 0 client.26023 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:55.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.028+0000 7fd24b612740 1 -- 172.21.15.29:0/132507648 >> 172.21.15.29:0/132507648 conn(0x55601e4f9eb0 msgr2=0x55601e4f08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:55.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.029+0000 7fd24b612740 1 -- 172.21.15.29:0/132507648 shutdown_connections 2024-03-11T19:13:55.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.029+0000 7fd24b612740 1 -- 172.21.15.29:0/132507648 wait complete. 2024-03-11T19:13:55.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.029+0000 7fd24b612740 20 client.26023 trim_cache size 0 max 16384 2024-03-11T19:13:55.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.059+0000 7fd24b612740 1 Processor -- start 2024-03-11T19:13:55.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.059+0000 7fd24b612740 1 -- start start 2024-03-11T19:13:55.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.059+0000 7fd24b612740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e4fecd0 0x55601e5050e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.059+0000 7fd24b612740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e4ffea0 0x55601e505620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.059+0000 7fd24b612740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e500860 0x55601e4f7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.059+0000 7fd24b612740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55601e4fa6f0 con 0x55601e500860 2024-03-11T19:13:55.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.059+0000 7fd24b612740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55601e4e6350 con 0x55601e4ffea0 2024-03-11T19:13:55.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.059+0000 7fd24b612740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55601e50ae90 con 0x55601e4fecd0 2024-03-11T19:13:55.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.059+0000 7fd24259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e4ffea0 0x55601e505620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.059+0000 7fd24259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e4ffea0 0x55601e505620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55550/0 (socket says 172.21.15.29:55550) 2024-03-11T19:13:55.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.059+0000 7fd24259c640 1 -- 172.21.15.29:0/1082960605 learned_addr learned my addr 172.21.15.29:0/1082960605 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:55.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.059+0000 7fd241d9b640 1 --2- 172.21.15.29:0/1082960605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e500860 0x55601e4f7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.060+0000 7fd241d9b640 1 --2- 172.21.15.29:0/1082960605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e4fecd0 0x55601e5050e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.060+0000 7fd24259c640 1 -- 172.21.15.29:0/1082960605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e4fecd0 msgr2=0x55601e5050e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:55.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.060+0000 7fd24259c640 1 --2- 172.21.15.29:0/1082960605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e4fecd0 0x55601e5050e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.060+0000 7fd24259c640 1 -- 172.21.15.29:0/1082960605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e500860 msgr2=0x55601e4f7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:55.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.060+0000 7fd24259c640 1 --2- 172.21.15.29:0/1082960605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e500860 0x55601e4f7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.060+0000 7fd24259c640 1 -- 172.21.15.29:0/1082960605 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55601e4f3b50 con 0x55601e4ffea0 2024-03-11T19:13:55.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.060+0000 7fd241d9b640 1 --2- 172.21.15.29:0/1082960605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e500860 0x55601e4f7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:13:55.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.060+0000 7fd24259c640 1 --2- 172.21.15.29:0/1082960605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e4ffea0 0x55601e505620 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009d90 tx=0x7fd23c013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f93e75b10267e0f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:55.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.060+0000 7fd22bfff640 1 -- 172.21.15.29:0/1082960605 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c003b70 con 0x55601e4ffea0 2024-03-11T19:13:55.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.061+0000 7fd22bfff640 1 -- 172.21.15.29:0/1082960605 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd23c020040 con 0x55601e4ffea0 2024-03-11T19:13:55.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.061+0000 7fd22bfff640 1 -- 172.21.15.29:0/1082960605 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c02b0f0 con 0x55601e4ffea0 2024-03-11T19:13:55.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.062+0000 7fd24b612740 1 -- 172.21.15.29:0/1082960605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e4ffea0 msgr2=0x55601e505620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.062+0000 7fd24b612740 1 --2- 172.21.15.29:0/1082960605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e4ffea0 0x55601e505620 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009d90 tx=0x7fd23c013040 comp rx=0 tx=0).stop 2024-03-11T19:13:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.062+0000 7fd24b612740 1 -- 172.21.15.29:0/1082960605 shutdown_connections 2024-03-11T19:13:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.062+0000 7fd24b612740 1 --2- 172.21.15.29:0/1082960605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e500860 0x55601e4f7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.062+0000 7fd24b612740 1 --2- 172.21.15.29:0/1082960605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e4ffea0 0x55601e505620 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.062+0000 7fd24b612740 1 --2- 172.21.15.29:0/1082960605 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e4fecd0 0x55601e5050e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.062+0000 7fd24b612740 1 -- 172.21.15.29:0/1082960605 >> 172.21.15.29:0/1082960605 conn(0x55601e4f9eb0 msgr2=0x55601e4f3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.062+0000 7fd24b612740 1 -- 172.21.15.29:0/1082960605 shutdown_connections 2024-03-11T19:13:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.062+0000 7fd24b612740 1 -- 172.21.15.29:0/1082960605 wait complete. 2024-03-11T19:13:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.062+0000 7fd24b612740 1 -- start start 2024-03-11T19:13:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.063+0000 7fd24b612740 1 Processor -- start 2024-03-11T19:13:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.063+0000 7fd24b612740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e4fecd0 0x55601e504750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.063+0000 7fd24b612740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e4ffea0 0x55601e504c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.063+0000 7fd24b612740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e500860 0x55601e562e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.063+0000 7fd24b612740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55601e563390 con 0x55601e4ffea0 2024-03-11T19:13:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.063+0000 7fd24b612740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55601e5634d0 con 0x55601e500860 2024-03-11T19:13:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.063+0000 7fd24b612740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55601e563690 con 0x55601e4fecd0 2024-03-11T19:13:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.063+0000 7fd24b612740 10 client.0 _ll_register_callbacks cb 0x7fd23c009bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.063+0000 7fd24259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e4ffea0 0x55601e504c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.063+0000 7fd241d9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e4fecd0 0x55601e504750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.063+0000 7fd24259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e4ffea0 0x55601e504c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41556/0 (socket says 172.21.15.29:41556) 2024-03-11T19:13:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.063+0000 7fd24259c640 1 -- 172.21.15.29:0/1285778230 learned_addr learned my addr 172.21.15.29:0/1285778230 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.063+0000 7fd241d9b640 1 --2- 172.21.15.29:0/1285778230 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e500860 0x55601e562e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.064+0000 7fd241d9b640 1 -- 172.21.15.29:0/1285778230 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e500860 msgr2=0x55601e562e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.064+0000 7fd241d9b640 1 --2- 172.21.15.29:0/1285778230 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e500860 0x55601e562e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.064+0000 7fd241d9b640 1 -- 172.21.15.29:0/1285778230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e4ffea0 msgr2=0x55601e504c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.064+0000 7fd241d9b640 1 --2- 172.21.15.29:0/1285778230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e4ffea0 0x55601e504c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.064+0000 7fd241d9b640 1 -- 172.21.15.29:0/1285778230 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55601e4210b0 con 0x55601e4fecd0 2024-03-11T19:13:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.064+0000 7fd24259c640 1 --2- 172.21.15.29:0/1285778230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e4ffea0 0x55601e504c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:13:55.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.064+0000 7fd241d9b640 1 --2- 172.21.15.29:0/1285778230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e4fecd0 0x55601e504750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd234025010 tx=0x7fd23400fc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82dc971b3518d094 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:55.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.064+0000 7fd229ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:13:55.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.064+0000 7fd229ffb640 1 -- 172.21.15.29:0/1285778230 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd234027030 con 0x55601e4fecd0 2024-03-11T19:13:55.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.065+0000 7fd229ffb640 1 -- 172.21.15.29:0/1285778230 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd234000e40 con 0x55601e4fecd0 2024-03-11T19:13:55.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.065+0000 7fd24b612740 1 -- 172.21.15.29:0/1285778230 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55601e484900 con 0x55601e4fecd0 2024-03-11T19:13:55.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.065+0000 7fd229ffb640 1 -- 172.21.15.29:0/1285778230 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd234022e80 con 0x55601e4fecd0 2024-03-11T19:13:55.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.065+0000 7fd24b612740 10 client.26029 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:55.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.065+0000 7fd24b612740 1 -- 172.21.15.29:0/1285778230 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55601e3f9c00 con 0x55601e4fecd0 2024-03-11T19:13:55.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.065+0000 7fd229ffb640 1 -- 172.21.15.29:0/1285778230 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd234051080 con 0x55601e4fecd0 2024-03-11T19:13:55.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.065+0000 7fd229ffb640 1 -- 172.21.15.29:0/1285778230 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd234005390 con 0x55601e4fecd0 2024-03-11T19:13:55.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.065+0000 7fd24b612740 10 client.26029 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:55.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.066+0000 7fd24b612740 10 client.26029 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:13:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.066+0000 7fd24b612740 20 client.26029 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.066+0000 7fd24b612740 1 --2- 172.21.15.29:0/1285778230 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55601e50c840 0x55601e5671c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.066+0000 7fd24b612740 4 client.26029 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:13:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.066+0000 7fd24b612740 1 -- 172.21.15.29:0/1285778230 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55601e15e860 con 0x55601e50c840 2024-03-11T19:13:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.066+0000 7fd24259c640 1 --2- 172.21.15.29:0/1285778230 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55601e50c840 0x55601e5671c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.066+0000 7fd229ffb640 10 client.26029 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.066+0000 7fd24259c640 1 --2- 172.21.15.29:0/1285778230 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55601e50c840 0x55601e5671c0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50910a2782207f28 server_cookie=38bf5e52549638db in_seq=0 out_seq=0 2024-03-11T19:13:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.067+0000 7fd229ffb640 1 -- 172.21.15.29:0/1285778230 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55601e15e860 con 0x55601e50c840 2024-03-11T19:13:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.067+0000 7fd229ffb640 10 client.26029 handle_command_reply: tid=0 2024-03-11T19:13:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.067+0000 7fd24b612740 1 client.26029 shutdown 2024-03-11T19:13:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.068+0000 7fd24b612740 1 -- 172.21.15.29:0/1285778230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e4fecd0 msgr2=0x55601e504750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.068+0000 7fd24b612740 1 --2- 172.21.15.29:0/1285778230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e4fecd0 0x55601e504750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd234025010 tx=0x7fd23400fc20 comp rx=0 tx=0).stop 2024-03-11T19:13:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.068+0000 7fd24b612740 1 -- 172.21.15.29:0/1285778230 shutdown_connections 2024-03-11T19:13:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.068+0000 7fd24b612740 1 --2- 172.21.15.29:0/1285778230 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55601e500860 0x55601e562e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.068+0000 7fd24b612740 1 --2- 172.21.15.29:0/1285778230 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55601e50c840 0x55601e5671c0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.068+0000 7fd24b612740 1 --2- 172.21.15.29:0/1285778230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55601e4ffea0 0x55601e504c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.068+0000 7fd24b612740 1 --2- 172.21.15.29:0/1285778230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55601e4fecd0 0x55601e504750 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.068+0000 7fd229ffb640 0 client.26029 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:55.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.068+0000 7fd24b612740 1 -- 172.21.15.29:0/1285778230 >> 172.21.15.29:0/1285778230 conn(0x55601e4f9eb0 msgr2=0x55601e4fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:55.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.069+0000 7fd24b612740 1 -- 172.21.15.29:0/1285778230 shutdown_connections 2024-03-11T19:13:55.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.069+0000 7fd24b612740 1 -- 172.21.15.29:0/1285778230 wait complete. 2024-03-11T19:13:55.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.069+0000 7fd24b612740 20 client.26029 trim_cache size 0 max 16384 2024-03-11T19:13:55.075 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:13:55.075 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:13:55.075 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "6d3044f5-45cb-4909-9ea5-9506e09c3bf4", 2024-03-11T19:13:55.076 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:13:55.076 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:13:55.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.070+0000 7fd248965640 1 -- 172.21.15.29:0/2379978512 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd22c0a1b30 msgr2=0x7fd22c0a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:55.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.070+0000 7fd248965640 1 --2- 172.21.15.29:0/2379978512 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd22c0a1b30 0x7fd22c0a3ff0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd238004650 tx=0x7fd2380dc040 comp rx=0 tx=0).stop 2024-03-11T19:13:55.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.070+0000 7fd248965640 1 -- 172.21.15.29:0/2379978512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd244101c30 msgr2=0x7fd244102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:55.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.070+0000 7fd248965640 1 --2- 172.21.15.29:0/2379978512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd244101c30 0x7fd244102b90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd238036040 tx=0x7fd238005e40 comp rx=0 tx=0).stop 2024-03-11T19:13:55.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.070+0000 7fd248965640 1 -- 172.21.15.29:0/2379978512 shutdown_connections 2024-03-11T19:13:55.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.070+0000 7fd248965640 1 --2- 172.21.15.29:0/2379978512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd244101c30 0x7fd244102b90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.070+0000 7fd248965640 1 --2- 172.21.15.29:0/2379978512 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd22c0a1b30 0x7fd22c0a3ff0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.070+0000 7fd248965640 1 --2- 172.21.15.29:0/2379978512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd244101270 0x7fd244102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.070+0000 7fd248965640 1 --2- 172.21.15.29:0/2379978512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd244100080 0x7fd244104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.071+0000 7fd248965640 1 -- 172.21.15.29:0/2379978512 >> 172.21.15.29:0/2379978512 conn(0x7fd2440fb610 msgr2=0x7fd244068310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:55.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.071+0000 7fd248965640 1 -- 172.21.15.29:0/2379978512 shutdown_connections 2024-03-11T19:13:55.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.071+0000 7fd248965640 1 -- 172.21.15.29:0/2379978512 wait complete. 2024-03-11T19:13:55.091 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:13:55.091 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:13:55.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.257+0000 7f29f61e0640 1 -- 172.21.15.29:0/2035319465 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29e8096f80 msgr2=0x7f29e8097400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:55.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.257+0000 7f29f61e0640 1 --2- 172.21.15.29:0/2035319465 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29e8096f80 0x7f29e8097400 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f29e0009860 tx=0x7f29e002bb70 comp rx=0 tx=0).stop 2024-03-11T19:13:55.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.257+0000 7f29f61e0640 1 -- 172.21.15.29:0/2035319465 shutdown_connections 2024-03-11T19:13:55.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.258+0000 7f29f61e0640 1 --2- 172.21.15.29:0/2035319465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29e8097940 0x7f29e809e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.258+0000 7f29f61e0640 1 --2- 172.21.15.29:0/2035319465 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29e8096f80 0x7f29e8097400 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.258+0000 7f29f61e0640 1 --2- 172.21.15.29:0/2035319465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29e8095d90 0x7f29e80961b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.258+0000 7f29f61e0640 1 -- 172.21.15.29:0/2035319465 >> 172.21.15.29:0/2035319465 conn(0x7f29e80912e0 msgr2=0x7f29e8093740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:55.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.258+0000 7f29f61e0640 1 -- 172.21.15.29:0/2035319465 shutdown_connections 2024-03-11T19:13:55.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.258+0000 7f29f61e0640 1 -- 172.21.15.29:0/2035319465 wait complete. 2024-03-11T19:13:55.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.259+0000 7f29f61e0640 1 Processor -- start 2024-03-11T19:13:55.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.259+0000 7f29f61e0640 1 -- start start 2024-03-11T19:13:55.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.259+0000 7f29f61e0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29e8095d90 0x7f29e8008980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.259+0000 7f29f61e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29e8096f80 0x7f29e8006fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.259+0000 7f29f61e0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29e8097940 0x7f29e8007500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.259+0000 7f29f61e0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e8007c60 con 0x7f29e8096f80 2024-03-11T19:13:55.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.259+0000 7f29f61e0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e8007dd0 con 0x7f29e8095d90 2024-03-11T19:13:55.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.259+0000 7f29f61e0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e8135bb0 con 0x7f29e8097940 2024-03-11T19:13:55.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.259+0000 7f29f51de640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29e8095d90 0x7f29e8008980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.259+0000 7f29f51de640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29e8095d90 0x7f29e8008980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55578/0 (socket says 172.21.15.29:55578) 2024-03-11T19:13:55.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.259+0000 7f29f51de640 1 -- 172.21.15.29:0/1213531056 learned_addr learned my addr 172.21.15.29:0/1213531056 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:55.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.259+0000 7f29f49dd640 1 --2- 172.21.15.29:0/1213531056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29e8096f80 0x7f29e8006fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.259+0000 7f29f59df640 1 --2- 172.21.15.29:0/1213531056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29e8097940 0x7f29e8007500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.260+0000 7f29f51de640 1 -- 172.21.15.29:0/1213531056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29e8097940 msgr2=0x7f29e8007500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:55.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.260+0000 7f29f51de640 1 --2- 172.21.15.29:0/1213531056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29e8097940 0x7f29e8007500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.260+0000 7f29f51de640 1 -- 172.21.15.29:0/1213531056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29e8096f80 msgr2=0x7f29e8006fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:55.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.260+0000 7f29f51de640 1 --2- 172.21.15.29:0/1213531056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29e8096f80 0x7f29e8006fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.260+0000 7f29f51de640 1 -- 172.21.15.29:0/1213531056 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e0009490 con 0x7f29e8095d90 2024-03-11T19:13:55.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.260+0000 7f29f51de640 1 --2- 172.21.15.29:0/1213531056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29e8095d90 0x7f29e8008980 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f29ec00ea20 tx=0x7f29ec00ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:55.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.260+0000 7f29e67fc640 1 -- 172.21.15.29:0/1213531056 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec00ccc0 con 0x7f29e8095d90 2024-03-11T19:13:55.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.260+0000 7f29f61e0640 1 -- 172.21.15.29:0/1213531056 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e8135e60 con 0x7f29e8095d90 2024-03-11T19:13:55.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.260+0000 7f29e67fc640 1 -- 172.21.15.29:0/1213531056 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f29ec0044d0 con 0x7f29e8095d90 2024-03-11T19:13:55.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.260+0000 7f29e67fc640 1 -- 172.21.15.29:0/1213531056 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec003dc0 con 0x7f29e8095d90 2024-03-11T19:13:55.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.261+0000 7f29f61e0640 1 -- 172.21.15.29:0/1213531056 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e8136230 con 0x7f29e8095d90 2024-03-11T19:13:55.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.263+0000 7f29e67fc640 1 -- 172.21.15.29:0/1213531056 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f29ec003f20 con 0x7f29e8095d90 2024-03-11T19:13:55.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.264+0000 7f29e67fc640 1 --2- 172.21.15.29:0/1213531056 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f29dc0a1ab0 0x7f29dc0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.264+0000 7f29fa00c740 1 Processor -- start 2024-03-11T19:13:55.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.265+0000 7f29fa00c740 1 -- start start 2024-03-11T19:13:55.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.265+0000 7f29e67fc640 1 -- 172.21.15.29:0/1213531056 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f29ec014070 con 0x7f29e8095d90 2024-03-11T19:13:55.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.265+0000 7f29f49dd640 1 --2- 172.21.15.29:0/1213531056 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f29dc0a1ab0 0x7f29dc0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.265+0000 7f29fa00c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f34b0 0x56376b5f38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.265+0000 7f29fa00c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f3ea0 0x56376b5f89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.265+0000 7f29fa00c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f9010 0x56376b5fb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.265+0000 7f29fa00c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56376b563280 con 0x56376b5f9010 2024-03-11T19:13:55.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.265+0000 7f29fa00c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56376b5607b0 con 0x56376b5f34b0 2024-03-11T19:13:55.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.265+0000 7f29fa00c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56376b55bf90 con 0x56376b5f3ea0 2024-03-11T19:13:55.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.265+0000 7f29f59df640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f9010 0x56376b5fb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.265+0000 7f29f51de640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f34b0 0x56376b5f38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.265+0000 7f29f59df640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f9010 0x56376b5fb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41588/0 (socket says 172.21.15.29:41588) 2024-03-11T19:13:55.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.265+0000 7f29f59df640 1 -- 172.21.15.29:0/562797390 learned_addr learned my addr 172.21.15.29:0/562797390 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:55.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.266+0000 7f29f59df640 1 -- 172.21.15.29:0/562797390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f3ea0 msgr2=0x56376b5f89a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:13:55.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.266+0000 7f29f59df640 1 --2- 172.21.15.29:0/562797390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f3ea0 0x56376b5f89a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.266+0000 7f29f49dd640 1 --2- 172.21.15.29:0/1213531056 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f29dc0a1ab0 0x7f29dc0a3f70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f29e00057c0 tx=0x7f29e0005710 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:55.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.266+0000 7f29f59df640 1 -- 172.21.15.29:0/562797390 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f34b0 msgr2=0x56376b5f38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:55.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.266+0000 7f29f59df640 1 --2- 172.21.15.29:0/562797390 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f34b0 0x56376b5f38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.266+0000 7f29f59df640 1 -- 172.21.15.29:0/562797390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56376b51a570 con 0x56376b5f9010 2024-03-11T19:13:55.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.266+0000 7f29f51de640 1 --2- 172.21.15.29:0/562797390 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f34b0 0x56376b5f38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:13:55.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.266+0000 7f29f59df640 1 --2- 172.21.15.29:0/562797390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f9010 0x56376b5fb4a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f29f00507e0 tx=0x7f29f004ff00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1bba690e67ee80 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:55.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.266+0000 7f29db7fe640 1 -- 172.21.15.29:0/562797390 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f0086070 con 0x56376b5f9010 2024-03-11T19:13:55.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.266+0000 7f29db7fe640 1 -- 172.21.15.29:0/562797390 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f29f007c040 con 0x56376b5f9010 2024-03-11T19:13:55.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.266+0000 7f29db7fe640 1 -- 172.21.15.29:0/562797390 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f007b070 con 0x56376b5f9010 2024-03-11T19:13:55.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.267+0000 7f29fa00c740 1 -- 172.21.15.29:0/562797390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f9010 msgr2=0x56376b5fb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:55.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.267+0000 7f29fa00c740 1 --2- 172.21.15.29:0/562797390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f9010 0x56376b5fb4a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f29f00507e0 tx=0x7f29f004ff00 comp rx=0 tx=0).stop 2024-03-11T19:13:55.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.267+0000 7f29fa00c740 1 -- 172.21.15.29:0/562797390 shutdown_connections 2024-03-11T19:13:55.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.267+0000 7f29fa00c740 1 --2- 172.21.15.29:0/562797390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f9010 0x56376b5fb4a0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.268+0000 7f29fa00c740 1 --2- 172.21.15.29:0/562797390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f3ea0 0x56376b5f89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.268+0000 7f29fa00c740 1 --2- 172.21.15.29:0/562797390 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f34b0 0x56376b5f38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.268+0000 7f29fa00c740 1 -- 172.21.15.29:0/562797390 >> 172.21.15.29:0/562797390 conn(0x56376b5eea20 msgr2=0x56376b5eee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:55.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.268+0000 7f29fa00c740 1 -- 172.21.15.29:0/562797390 shutdown_connections 2024-03-11T19:13:55.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.268+0000 7f29fa00c740 1 -- 172.21.15.29:0/562797390 wait complete. 2024-03-11T19:13:55.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.268+0000 7f29fa00c740 1 -- start start 2024-03-11T19:13:55.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.269+0000 7f29fa00c740 1 Processor -- start 2024-03-11T19:13:55.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.269+0000 7f29fa00c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f34b0 0x56376b5fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.269+0000 7f29fa00c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f3ea0 0x56376b5fd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.269+0000 7f29fa00c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f9010 0x56376b604920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.269+0000 7f29fa00c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56376b22f430 con 0x56376b5f9010 2024-03-11T19:13:55.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.269+0000 7f29fa00c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56376b1ebe00 con 0x56376b5f3ea0 2024-03-11T19:13:55.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.269+0000 7f29fa00c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56376b4e0fd0 con 0x56376b5f34b0 2024-03-11T19:13:55.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.269+0000 7f29fa00c740 10 client.0 _ll_register_callbacks cb 0x56376b536fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:55.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.270+0000 7f29f49dd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f3ea0 0x56376b5fd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.270+0000 7f29f49dd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f3ea0 0x56376b5fd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55596/0 (socket says 172.21.15.29:55596) 2024-03-11T19:13:55.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.270+0000 7f29f49dd640 1 -- 172.21.15.29:0/813047164 learned_addr learned my addr 172.21.15.29:0/813047164 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:55.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.270+0000 7f29f51de640 1 --2- 172.21.15.29:0/813047164 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f34b0 0x56376b5fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.270+0000 7f29f49dd640 1 -- 172.21.15.29:0/813047164 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f34b0 msgr2=0x56376b5fd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:55.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.270+0000 7f29f49dd640 1 --2- 172.21.15.29:0/813047164 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f34b0 0x56376b5fd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.270+0000 7f29f49dd640 1 -- 172.21.15.29:0/813047164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f9010 msgr2=0x56376b604920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:55.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.270+0000 7f29f59df640 1 --2- 172.21.15.29:0/813047164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f9010 0x56376b604920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.270+0000 7f29f49dd640 1 --2- 172.21.15.29:0/813047164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f9010 0x56376b604920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.271+0000 7f29f49dd640 1 -- 172.21.15.29:0/813047164 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f00687c0 con 0x56376b5f3ea0 2024-03-11T19:13:55.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.271+0000 7f29f59df640 1 --2- 172.21.15.29:0/813047164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f9010 0x56376b604920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:13:55.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.271+0000 7f29f49dd640 1 --2- 172.21.15.29:0/813047164 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f3ea0 0x56376b5fd8e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f29e0009830 tx=0x7f29e0000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe0a4c24b3f4c86a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:55.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.271+0000 7f29d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:13:55.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.271+0000 7f29d8ff9640 1 -- 172.21.15.29:0/813047164 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e002ca60 con 0x56376b5f3ea0 2024-03-11T19:13:55.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.271+0000 7f29d8ff9640 1 -- 172.21.15.29:0/813047164 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f29e00035f0 con 0x56376b5f3ea0 2024-03-11T19:13:55.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.271+0000 7f29fa00c740 1 -- 172.21.15.29:0/813047164 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56376b25b7e0 con 0x56376b5f3ea0 2024-03-11T19:13:55.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.272+0000 7f29d8ff9640 1 -- 172.21.15.29:0/813047164 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29e002cbc0 con 0x56376b5f3ea0 2024-03-11T19:13:55.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.272+0000 7f29fa00c740 10 client.26024 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:55.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.272+0000 7f29fa00c740 1 -- 172.21.15.29:0/813047164 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56376b55a610 con 0x56376b5f3ea0 2024-03-11T19:13:55.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.272+0000 7f29d8ff9640 1 -- 172.21.15.29:0/813047164 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x56376b25b7e0 con 0x56376b5f3ea0 2024-03-11T19:13:55.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.272+0000 7f29d8ff9640 1 -- 172.21.15.29:0/813047164 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f29e0032a90 con 0x56376b5f3ea0 2024-03-11T19:13:55.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.272+0000 7f29fa00c740 10 client.26024 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:55.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.272+0000 7f29fa00c740 10 client.26024 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:13:55.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.273+0000 7f29fa00c740 20 client.26024 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:55.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.273+0000 7f29fa00c740 1 --2- 172.21.15.29:0/813047164 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56376b611e10 0x56376b614230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.273+0000 7f29fa00c740 4 client.26024 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:13:55.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.273+0000 7f29fa00c740 1 -- 172.21.15.29:0/813047164 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56376b298720 con 0x56376b611e10 2024-03-11T19:13:55.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.273+0000 7f29f51de640 1 --2- 172.21.15.29:0/813047164 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56376b611e10 0x56376b614230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.274+0000 7f29f51de640 1 --2- 172.21.15.29:0/813047164 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56376b611e10 0x56376b614230 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ab08202698e2f6f server_cookie=962883bc2321ade9 in_seq=0 out_seq=0 2024-03-11T19:13:55.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.274+0000 7f29d8ff9640 10 client.26024 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.274+0000 7f29d8ff9640 1 -- 172.21.15.29:0/813047164 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56376b298720 con 0x56376b611e10 2024-03-11T19:13:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.274+0000 7f29d8ff9640 10 client.26024 handle_command_reply: tid=0 2024-03-11T19:13:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.274+0000 7f29fa00c740 1 client.26024 shutdown 2024-03-11T19:13:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.275+0000 7f29fa00c740 1 -- 172.21.15.29:0/813047164 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f3ea0 msgr2=0x56376b5fd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.275+0000 7f29fa00c740 1 --2- 172.21.15.29:0/813047164 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f3ea0 0x56376b5fd8e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f29e0009830 tx=0x7f29e0000bc0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.275+0000 7f29fa00c740 1 -- 172.21.15.29:0/813047164 shutdown_connections 2024-03-11T19:13:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.275+0000 7f29fa00c740 1 --2- 172.21.15.29:0/813047164 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56376b611e10 0x56376b614230 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.275+0000 7f29d8ff9640 0 client.26024 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.275+0000 7f29fa00c740 1 --2- 172.21.15.29:0/813047164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f9010 0x56376b604920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.275+0000 7f29fa00c740 1 --2- 172.21.15.29:0/813047164 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f3ea0 0x56376b5fd8e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.275+0000 7f29fa00c740 1 --2- 172.21.15.29:0/813047164 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f34b0 0x56376b5fd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.275+0000 7f29fa00c740 1 -- 172.21.15.29:0/813047164 >> 172.21.15.29:0/813047164 conn(0x56376b5eea20 msgr2=0x56376b5edc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.275+0000 7f29fa00c740 1 -- 172.21.15.29:0/813047164 shutdown_connections 2024-03-11T19:13:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.275+0000 7f29fa00c740 1 -- 172.21.15.29:0/813047164 wait complete. 2024-03-11T19:13:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.276+0000 7f29fa00c740 20 client.26024 trim_cache size 0 max 16384 2024-03-11T19:13:55.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.302+0000 7f29fa00c740 1 Processor -- start 2024-03-11T19:13:55.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.303+0000 7f29fa00c740 1 -- start start 2024-03-11T19:13:55.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.303+0000 7f29fa00c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f34b0 0x56376b5fcda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.303+0000 7f29fa00c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f3ea0 0x56376b6048e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.303+0000 7f29fa00c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f9010 0x56376b5ed3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.303+0000 7f29fa00c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56376b25b7e0 con 0x56376b5f3ea0 2024-03-11T19:13:55.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.303+0000 7f29f51de640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f34b0 0x56376b5fcda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.303+0000 7f29fa00c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56376b5fe590 con 0x56376b5f34b0 2024-03-11T19:13:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.303+0000 7f29f49dd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f3ea0 0x56376b6048e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.303+0000 7f29fa00c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56376b5f3140 con 0x56376b5f9010 2024-03-11T19:13:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.303+0000 7f29f51de640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f34b0 0x56376b5fcda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55608/0 (socket says 172.21.15.29:55608) 2024-03-11T19:13:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.303+0000 7f29f51de640 1 -- 172.21.15.29:0/2439609585 learned_addr learned my addr 172.21.15.29:0/2439609585 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.303+0000 7f29f59df640 1 --2- 172.21.15.29:0/2439609585 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f9010 0x56376b5ed3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.304+0000 7f29f59df640 1 -- 172.21.15.29:0/2439609585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f34b0 msgr2=0x56376b5fcda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.304+0000 7f29f59df640 1 --2- 172.21.15.29:0/2439609585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f34b0 0x56376b5fcda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.304+0000 7f29f59df640 1 -- 172.21.15.29:0/2439609585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f3ea0 msgr2=0x56376b6048e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.304+0000 7f29f59df640 1 --2- 172.21.15.29:0/2439609585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f3ea0 0x56376b6048e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.304+0000 7f29f59df640 1 -- 172.21.15.29:0/2439609585 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56376b4f6ac0 con 0x56376b5f9010 2024-03-11T19:13:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.304+0000 7f29f59df640 1 --2- 172.21.15.29:0/2439609585 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f9010 0x56376b5ed3b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f29f004f1b0 tx=0x7f29f0066a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9d8951b2ceb4a56 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.304+0000 7f29daffd640 1 -- 172.21.15.29:0/2439609585 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f008e070 con 0x56376b5f9010 2024-03-11T19:13:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.304+0000 7f29daffd640 1 -- 172.21.15.29:0/2439609585 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f29f0062500 con 0x56376b5f9010 2024-03-11T19:13:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.304+0000 7f29daffd640 1 -- 172.21.15.29:0/2439609585 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f0086040 con 0x56376b5f9010 2024-03-11T19:13:55.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.305+0000 7f29fa00c740 1 -- 172.21.15.29:0/2439609585 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f9010 msgr2=0x56376b5ed3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:55.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.305+0000 7f29fa00c740 1 --2- 172.21.15.29:0/2439609585 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f9010 0x56376b5ed3b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f29f004f1b0 tx=0x7f29f0066a00 comp rx=0 tx=0).stop 2024-03-11T19:13:55.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.305+0000 7f29fa00c740 1 -- 172.21.15.29:0/2439609585 shutdown_connections 2024-03-11T19:13:55.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.305+0000 7f29fa00c740 1 --2- 172.21.15.29:0/2439609585 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f9010 0x56376b5ed3b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.305+0000 7f29fa00c740 1 --2- 172.21.15.29:0/2439609585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f3ea0 0x56376b6048e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.305+0000 7f29fa00c740 1 --2- 172.21.15.29:0/2439609585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f34b0 0x56376b5fcda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.305+0000 7f29fa00c740 1 -- 172.21.15.29:0/2439609585 >> 172.21.15.29:0/2439609585 conn(0x56376b5eea20 msgr2=0x56376b5fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:55.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.305+0000 7f29fa00c740 1 -- 172.21.15.29:0/2439609585 shutdown_connections 2024-03-11T19:13:55.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.305+0000 7f29fa00c740 1 -- 172.21.15.29:0/2439609585 wait complete. 2024-03-11T19:13:55.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.306+0000 7f29fa00c740 1 -- start start 2024-03-11T19:13:55.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.306+0000 7f29fa00c740 1 Processor -- start 2024-03-11T19:13:55.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.306+0000 7f29fa00c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f34b0 0x56376b603a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.307+0000 7f29fa00c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f3ea0 0x56376b603f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.307+0000 7f29f51de640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f34b0 0x56376b603a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.307+0000 7f29fa00c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f9010 0x56376b6683b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.307+0000 7f29fa00c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56376b6688f0 con 0x56376b5f34b0 2024-03-11T19:13:55.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.307+0000 7f29fa00c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56376b668a30 con 0x56376b5f9010 2024-03-11T19:13:55.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.307+0000 7f29f51de640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f34b0 0x56376b603a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41614/0 (socket says 172.21.15.29:41614) 2024-03-11T19:13:55.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.307+0000 7f29fa00c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56376b668bc0 con 0x56376b5f3ea0 2024-03-11T19:13:55.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.307+0000 7f29f51de640 1 -- 172.21.15.29:0/364935790 learned_addr learned my addr 172.21.15.29:0/364935790 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:13:55.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.307+0000 7f29fa00c740 10 client.0 _ll_register_callbacks cb 0x7f29f00686f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:13:55.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.307+0000 7f29f59df640 1 --2- 172.21.15.29:0/364935790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f9010 0x56376b6683b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.307+0000 7f29f49dd640 1 --2- 172.21.15.29:0/364935790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f3ea0 0x56376b603f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.307+0000 7f29f51de640 1 -- 172.21.15.29:0/364935790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f3ea0 msgr2=0x56376b603f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:55.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.307+0000 7f29f51de640 1 --2- 172.21.15.29:0/364935790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f3ea0 0x56376b603f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.307+0000 7f29f51de640 1 -- 172.21.15.29:0/364935790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f9010 msgr2=0x56376b6683b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:13:55.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.307+0000 7f29f51de640 1 --2- 172.21.15.29:0/364935790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f9010 0x56376b6683b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.307+0000 7f29f51de640 1 -- 172.21.15.29:0/364935790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56376b51a570 con 0x56376b5f34b0 2024-03-11T19:13:55.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.307+0000 7f29f51de640 1 --2- 172.21.15.29:0/364935790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f34b0 0x56376b603a30 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f29ec009fd0 tx=0x7f29ec0caf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5412d8b0f2569300 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:13:55.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.307+0000 7f29d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:13:55.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.307+0000 7f29d8ff9640 1 -- 172.21.15.29:0/364935790 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec0ba0b0 con 0x56376b5f34b0 2024-03-11T19:13:55.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.308+0000 7f29d8ff9640 1 -- 172.21.15.29:0/364935790 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f29ec09a4c0 con 0x56376b5f34b0 2024-03-11T19:13:55.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.308+0000 7f29fa00c740 1 -- 172.21.15.29:0/364935790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f29f007c040 con 0x56376b5f34b0 2024-03-11T19:13:55.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.308+0000 7f29d8ff9640 1 -- 172.21.15.29:0/364935790 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29ec09e610 con 0x56376b5f34b0 2024-03-11T19:13:55.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.308+0000 7f29fa00c740 10 client.16518 fetch_fsmap learned FSMap version 27 2024-03-11T19:13:55.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.308+0000 7f29fa00c740 1 -- 172.21.15.29:0/364935790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56376b51c6a0 con 0x56376b5f34b0 2024-03-11T19:13:55.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.308+0000 7f29d8ff9640 1 -- 172.21.15.29:0/364935790 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f29f007c040 con 0x56376b5f34b0 2024-03-11T19:13:55.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.308+0000 7f29d8ff9640 1 -- 172.21.15.29:0/364935790 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f29ec01f060 con 0x56376b5f34b0 2024-03-11T19:13:55.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.308+0000 7f29fa00c740 10 client.16518 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:13:55.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.308+0000 7f29fa00c740 10 client.16518 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:13:55.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.308+0000 7f29fa00c740 20 client.16518 populate_metadata read hostname 'smithi029' 2024-03-11T19:13:55.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.308+0000 7f29fa00c740 1 --2- 172.21.15.29:0/364935790 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56376b66a6f0 0x56376b66cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:13:55.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.308+0000 7f29fa00c740 4 client.16518 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:13:55.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.308+0000 7f29fa00c740 1 -- 172.21.15.29:0/364935790 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56376b2e7890 con 0x56376b66a6f0 2024-03-11T19:13:55.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.308+0000 7f29f49dd640 1 --2- 172.21.15.29:0/364935790 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56376b66a6f0 0x56376b66cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:13:55.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.309+0000 7f29d8ff9640 10 client.16518 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:55.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.309+0000 7f29f49dd640 1 --2- 172.21.15.29:0/364935790 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56376b66a6f0 0x56376b66cb10 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2096cd2596922211 server_cookie=5c68248380cb3de5 in_seq=0 out_seq=0 2024-03-11T19:13:55.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.309+0000 7f29d8ff9640 1 -- 172.21.15.29:0/364935790 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56376b2e7890 con 0x56376b66a6f0 2024-03-11T19:13:55.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.309+0000 7f29d8ff9640 10 client.16518 handle_command_reply: tid=0 2024-03-11T19:13:55.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.309+0000 7f29fa00c740 1 client.16518 shutdown 2024-03-11T19:13:55.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.309+0000 7f29fa00c740 1 -- 172.21.15.29:0/364935790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f34b0 msgr2=0x56376b603a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:55.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.309+0000 7f29fa00c740 1 --2- 172.21.15.29:0/364935790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f34b0 0x56376b603a30 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f29ec009fd0 tx=0x7f29ec0caf20 comp rx=0 tx=0).stop 2024-03-11T19:13:55.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.309+0000 7f29fa00c740 1 -- 172.21.15.29:0/364935790 shutdown_connections 2024-03-11T19:13:55.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.309+0000 7f29fa00c740 1 --2- 172.21.15.29:0/364935790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56376b5f9010 0x56376b6683b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.309+0000 7f29fa00c740 1 --2- 172.21.15.29:0/364935790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56376b5f3ea0 0x56376b603f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.309+0000 7f29fa00c740 1 --2- 172.21.15.29:0/364935790 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56376b66a6f0 0x56376b66cb10 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.309+0000 7f29fa00c740 1 --2- 172.21.15.29:0/364935790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56376b5f34b0 0x56376b603a30 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.309+0000 7f29d8ff9640 0 client.16518 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:13:55.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.309+0000 7f29fa00c740 1 -- 172.21.15.29:0/364935790 >> 172.21.15.29:0/364935790 conn(0x56376b5eea20 msgr2=0x56376b5f1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:55.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.310+0000 7f29fa00c740 1 -- 172.21.15.29:0/364935790 shutdown_connections 2024-03-11T19:13:55.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.310+0000 7f29fa00c740 1 -- 172.21.15.29:0/364935790 wait complete. 2024-03-11T19:13:55.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.310+0000 7f29fa00c740 20 client.16518 trim_cache size 0 max 16384 2024-03-11T19:13:55.318 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:13:55.318 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:13:55.318 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:13:55.318 INFO:teuthology.orchestra.run.smithi029.stdout: "6d3044f5-45cb-4909-9ea5-9506e09c3bf4": { 2024-03-11T19:13:55.318 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:13:55.318 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "6d3044f5-45cb-4909-9ea5-9506e09c3bf4", 2024-03-11T19:13:55.318 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:13:55.318 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:13:55.318 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:13:55.318 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:13:55.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.310+0000 7f29f61e0640 1 -- 172.21.15.29:0/1213531056 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f29dc0a1ab0 msgr2=0x7f29dc0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:55.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.310+0000 7f29f61e0640 1 --2- 172.21.15.29:0/1213531056 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f29dc0a1ab0 0x7f29dc0a3f70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f29e00057c0 tx=0x7f29e0005710 comp rx=0 tx=0).stop 2024-03-11T19:13:55.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.310+0000 7f29f61e0640 1 -- 172.21.15.29:0/1213531056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29e8095d90 msgr2=0x7f29e8008980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:13:55.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.310+0000 7f29f61e0640 1 --2- 172.21.15.29:0/1213531056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29e8095d90 0x7f29e8008980 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f29ec00ea20 tx=0x7f29ec00ee20 comp rx=0 tx=0).stop 2024-03-11T19:13:55.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.311+0000 7f29f61e0640 1 -- 172.21.15.29:0/1213531056 shutdown_connections 2024-03-11T19:13:55.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.311+0000 7f29f61e0640 1 --2- 172.21.15.29:0/1213531056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29e8097940 0x7f29e8007500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.311+0000 7f29f61e0640 1 --2- 172.21.15.29:0/1213531056 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f29dc0a1ab0 0x7f29dc0a3f70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.311+0000 7f29f61e0640 1 --2- 172.21.15.29:0/1213531056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29e8096f80 0x7f29e8006fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.311+0000 7f29f61e0640 1 --2- 172.21.15.29:0/1213531056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29e8095d90 0x7f29e8008980 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:13:55.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.311+0000 7f29f61e0640 1 -- 172.21.15.29:0/1213531056 >> 172.21.15.29:0/1213531056 conn(0x7f29e80912e0 msgr2=0x7f29e809a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:13:55.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.311+0000 7f29f61e0640 1 -- 172.21.15.29:0/1213531056 shutdown_connections 2024-03-11T19:13:55.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:13:55.311+0000 7f29f61e0640 1 -- 172.21.15.29:0/1213531056 wait complete. 2024-03-11T19:13:55.332 INFO:tasks.cephfs.filesystem:scrub status for tag:6d3044f5-45cb-4909-9ea5-9506e09c3bf4 - {'path': '/', 'tag': '6d3044f5-45cb-4909-9ea5-9506e09c3bf4', 'options': 'recursive,force'} 2024-03-11T19:13:55.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:55 smithi170 ceph-mon[35690]: pgmap v474: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-11T19:13:55.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:55 smithi029 ceph-mon[32439]: pgmap v474: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-11T19:13:56.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:55 smithi190 ceph-mon[36292]: pgmap v474: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-11T19:13:56.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:56 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:13:56.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:56 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:13:56.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:56 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:13:56.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:56 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:13:56.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:56 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:13:56.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:56 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:13:56.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:56 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:13:56.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:56 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:13:56.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:56 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:13:56.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:56 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:13:57.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:56 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:13:57.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:56 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:13:57.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:56 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:13:57.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:56 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:13:57.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:56 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:13:57.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:57 smithi170 ceph-mon[35690]: pgmap v475: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T19:13:57.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:57 smithi029 ceph-mon[32439]: pgmap v475: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T19:13:57.913 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:57 smithi190 ceph-mon[36292]: pgmap v475: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T19:13:59.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:13:59 smithi170 ceph-mon[35690]: pgmap v476: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:13:59.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:13:59 smithi029 ceph-mon[32439]: pgmap v476: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:14:00.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:13:59 smithi190 ceph-mon[36292]: pgmap v476: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:14:01.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:01 smithi170 ceph-mon[35690]: pgmap v477: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-11T19:14:01.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:01 smithi029 ceph-mon[32439]: pgmap v477: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-11T19:14:02.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:01 smithi190 ceph-mon[36292]: pgmap v477: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-11T19:14:03.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:03 smithi170 ceph-mon[35690]: pgmap v478: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-03-11T19:14:03.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:03 smithi029 ceph-mon[32439]: pgmap v478: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-03-11T19:14:04.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:03 smithi190 ceph-mon[36292]: pgmap v478: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-03-11T19:14:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:05 smithi170 ceph-mon[35690]: pgmap v479: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-11T19:14:05.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:05 smithi029 ceph-mon[32439]: pgmap v479: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-11T19:14:06.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:05 smithi190 ceph-mon[36292]: pgmap v479: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-11T19:14:07.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:07 smithi170 ceph-mon[35690]: pgmap v480: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-11T19:14:07.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:07 smithi029 ceph-mon[32439]: pgmap v480: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-11T19:14:08.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:07 smithi190 ceph-mon[36292]: pgmap v480: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-11T19:14:09.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:09 smithi170 ceph-mon[35690]: pgmap v481: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T19:14:09.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:09 smithi029 ceph-mon[32439]: pgmap v481: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T19:14:10.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:09 smithi190 ceph-mon[36292]: pgmap v481: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T19:14:11.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:11 smithi170 ceph-mon[35690]: pgmap v482: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-11T19:14:11.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:11 smithi029 ceph-mon[32439]: pgmap v482: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-11T19:14:12.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:11 smithi190 ceph-mon[36292]: pgmap v482: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-11T19:14:13.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:13 smithi170 ceph-mon[35690]: pgmap v483: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-11T19:14:13.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:13 smithi029 ceph-mon[32439]: pgmap v483: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-11T19:14:14.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:13 smithi190 ceph-mon[36292]: pgmap v483: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-11T19:14:15.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:15 smithi170 ceph-mon[35690]: pgmap v484: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 76 MiB/s wr, 1.23k op/s 2024-03-11T19:14:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:15 smithi029 ceph-mon[32439]: pgmap v484: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 76 MiB/s wr, 1.23k op/s 2024-03-11T19:14:16.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:15 smithi190 ceph-mon[36292]: pgmap v484: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 76 MiB/s wr, 1.23k op/s 2024-03-11T19:14:17.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:17 smithi170 ceph-mon[35690]: pgmap v485: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-11T19:14:17.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:17 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:14:17.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:17 smithi029 ceph-mon[32439]: pgmap v485: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-11T19:14:17.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:17 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:14:18.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:17 smithi190 ceph-mon[36292]: pgmap v485: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-11T19:14:18.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:17 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:14:18.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:18 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:14:18.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:18 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:14:18.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:18 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:14:18.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:18 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:14:18.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:18 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:14:18.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:18 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:14:19.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:18 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:14:19.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:18 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:14:19.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:18 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:14:19.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:19 smithi170 ceph-mon[35690]: pgmap v486: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T19:14:19.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:19 smithi029 ceph-mon[32439]: pgmap v486: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T19:14:20.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:19 smithi190 ceph-mon[36292]: pgmap v486: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T19:14:21.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:21 smithi170 ceph-mon[35690]: pgmap v487: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T19:14:21.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:21 smithi029 ceph-mon[32439]: pgmap v487: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T19:14:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:21 smithi190 ceph-mon[36292]: pgmap v487: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T19:14:23.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:23 smithi170 ceph-mon[35690]: pgmap v488: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:14:23.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:23 smithi029 ceph-mon[32439]: pgmap v488: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:14:24.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:23 smithi190 ceph-mon[36292]: pgmap v488: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:14:25.333 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:14:25.334 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:14:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.502+0000 7ff0ee713640 1 -- 172.21.15.29:0/111075229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0e81009b0 msgr2=0x7ff0e8100e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.502+0000 7ff0ee713640 1 --2- 172.21.15.29:0/111075229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0e81009b0 0x7ff0e8100e30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc009900 tx=0x7ff0dc02bc40 comp rx=0 tx=0).stop 2024-03-11T19:14:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.502+0000 7ff0ee713640 1 -- 172.21.15.29:0/111075229 shutdown_connections 2024-03-11T19:14:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.502+0000 7ff0ee713640 1 --2- 172.21.15.29:0/111075229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e8101370 0x7ff0e8107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.502+0000 7ff0ee713640 1 --2- 172.21.15.29:0/111075229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0e81009b0 0x7ff0e8100e30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.502+0000 7ff0ee713640 1 --2- 172.21.15.29:0/111075229 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0e80ff9c0 0x7ff0e80ffde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.502+0000 7ff0ee713640 1 -- 172.21.15.29:0/111075229 >> 172.21.15.29:0/111075229 conn(0x7ff0e80fa960 msgr2=0x7ff0e80fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.502+0000 7ff0ee713640 1 -- 172.21.15.29:0/111075229 shutdown_connections 2024-03-11T19:14:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.502+0000 7ff0ee713640 1 -- 172.21.15.29:0/111075229 wait complete. 2024-03-11T19:14:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.503+0000 7ff0ee713640 1 Processor -- start 2024-03-11T19:14:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.503+0000 7ff0ee713640 1 -- start start 2024-03-11T19:14:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.503+0000 7ff0ee713640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0e80ff9c0 0x7ff0e8103750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.503+0000 7ff0ee713640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e81009b0 0x7ff0e8101d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.503+0000 7ff0ee713640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0e8101370 0x7ff0e81022d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.503+0000 7ff0ee713640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e819af20 con 0x7ff0e81009b0 2024-03-11T19:14:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.503+0000 7ff0ee713640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e8102810 con 0x7ff0e8101370 2024-03-11T19:14:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.503+0000 7ff0ee713640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e81029d0 con 0x7ff0e80ff9c0 2024-03-11T19:14:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.504+0000 7ff0eccb0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0e8101370 0x7ff0e81022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.504+0000 7ff0e7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0e80ff9c0 0x7ff0e8103750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.504+0000 7ff0eccb0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0e8101370 0x7ff0e81022d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37456/0 (socket says 172.21.15.29:37456) 2024-03-11T19:14:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.504+0000 7ff0e7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0e80ff9c0 0x7ff0e8103750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41310/0 (socket says 172.21.15.29:41310) 2024-03-11T19:14:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.504+0000 7ff0eccb0640 1 -- 172.21.15.29:0/2949703902 learned_addr learned my addr 172.21.15.29:0/2949703902 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.504+0000 7ff0e77fe640 1 --2- 172.21.15.29:0/2949703902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e81009b0 0x7ff0e8101d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.504+0000 7ff0eccb0640 1 -- 172.21.15.29:0/2949703902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0e80ff9c0 msgr2=0x7ff0e8103750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:25.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.504+0000 7ff0eccb0640 1 --2- 172.21.15.29:0/2949703902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0e80ff9c0 0x7ff0e8103750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.504+0000 7ff0eccb0640 1 -- 172.21.15.29:0/2949703902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e81009b0 msgr2=0x7ff0e8101d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:25.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.504+0000 7ff0eccb0640 1 --2- 172.21.15.29:0/2949703902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e81009b0 0x7ff0e8101d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.504+0000 7ff0eccb0640 1 -- 172.21.15.29:0/2949703902 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0dc009560 con 0x7ff0e8101370 2024-03-11T19:14:25.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.505+0000 7ff0eccb0640 1 --2- 172.21.15.29:0/2949703902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0e8101370 0x7ff0e81022d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff0e000ec60 tx=0x7ff0e000c7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:25.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.505+0000 7ff0e57fa640 1 -- 172.21.15.29:0/2949703902 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0004d20 con 0x7ff0e8101370 2024-03-11T19:14:25.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.505+0000 7ff0e57fa640 1 -- 172.21.15.29:0/2949703902 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff0e0004e80 con 0x7ff0e8101370 2024-03-11T19:14:25.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.505+0000 7ff0e57fa640 1 -- 172.21.15.29:0/2949703902 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e00055b0 con 0x7ff0e8101370 2024-03-11T19:14:25.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.506+0000 7ff0ee713640 1 -- 172.21.15.29:0/2949703902 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e81a5910 con 0x7ff0e8101370 2024-03-11T19:14:25.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.506+0000 7ff0ee713640 1 -- 172.21.15.29:0/2949703902 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e81a5c80 con 0x7ff0e8101370 2024-03-11T19:14:25.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.509+0000 7ff0e57fa640 1 -- 172.21.15.29:0/2949703902 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff0e000ed50 con 0x7ff0e8101370 2024-03-11T19:14:25.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.510+0000 7ff0e57fa640 1 --2- 172.21.15.29:0/2949703902 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff0d00a1ab0 0x7ff0d00a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.510+0000 7ff0e57fa640 1 -- 172.21.15.29:0/2949703902 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff0e00cd5a0 con 0x7ff0e8101370 2024-03-11T19:14:25.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.511+0000 7ff0e7fff640 1 --2- 172.21.15.29:0/2949703902 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff0d00a1ab0 0x7ff0d00a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.511+0000 7ff0f13c7740 1 Processor -- start 2024-03-11T19:14:25.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.512+0000 7ff0f13c7740 1 -- start start 2024-03-11T19:14:25.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.512+0000 7ff0e7fff640 1 --2- 172.21.15.29:0/2949703902 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff0d00a1ab0 0x7ff0d00a3f70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8100880 tx=0x7ff0d800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:25.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.512+0000 7ff0f13c7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557815812d0 0x5557815816f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.512+0000 7ff0f13c7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555781581cc0 0x5557815867c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.512+0000 7ff0e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557815812d0 0x5557815816f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.512+0000 7ff0e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557815812d0 0x5557815816f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54446/0 (socket says 172.21.15.29:54446) 2024-03-11T19:14:25.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.512+0000 7ff0f13c7740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555781586e30 0x5557815892c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.512+0000 7ff0e7fff640 1 -- 172.21.15.29:0/790081622 learned_addr learned my addr 172.21.15.29:0/790081622 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:25.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.512+0000 7ff0f13c7740 1 -- 172.21.15.29:0/790081622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557814f1280 con 0x5557815812d0 2024-03-11T19:14:25.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.512+0000 7ff0f13c7740 1 -- 172.21.15.29:0/790081622 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557814ee7b0 con 0x555781586e30 2024-03-11T19:14:25.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.512+0000 7ff0f13c7740 1 -- 172.21.15.29:0/790081622 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557814e9f90 con 0x555781581cc0 2024-03-11T19:14:25.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.513+0000 7ff0e77fe640 1 --2- 172.21.15.29:0/790081622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555781581cc0 0x5557815867c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.513+0000 7ff0eccb0640 1 --2- 172.21.15.29:0/790081622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555781586e30 0x5557815892c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.513+0000 7ff0e7fff640 1 -- 172.21.15.29:0/790081622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555781581cc0 msgr2=0x5557815867c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.513+0000 7ff0e7fff640 1 --2- 172.21.15.29:0/790081622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555781581cc0 0x5557815867c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.513+0000 7ff0e7fff640 1 -- 172.21.15.29:0/790081622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555781586e30 msgr2=0x5557815892c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.513+0000 7ff0e7fff640 1 --2- 172.21.15.29:0/790081622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555781586e30 0x5557815892c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.513+0000 7ff0e7fff640 1 -- 172.21.15.29:0/790081622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557814a8570 con 0x5557815812d0 2024-03-11T19:14:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.513+0000 7ff0e77fe640 1 --2- 172.21.15.29:0/790081622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555781581cc0 0x5557815867c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:14:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.513+0000 7ff0e7fff640 1 --2- 172.21.15.29:0/790081622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557815812d0 0x5557815816f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff0d80058a0 tx=0x7ff0d8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9a4ecd28edef7f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.514+0000 7ff0ce7fc640 1 -- 172.21.15.29:0/790081622 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8004220 con 0x5557815812d0 2024-03-11T19:14:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.514+0000 7ff0ce7fc640 1 -- 172.21.15.29:0/790081622 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff0d8015070 con 0x5557815812d0 2024-03-11T19:14:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.514+0000 7ff0ce7fc640 1 -- 172.21.15.29:0/790081622 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d801e680 con 0x5557815812d0 2024-03-11T19:14:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.514+0000 7ff0f13c7740 1 -- 172.21.15.29:0/790081622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557815812d0 msgr2=0x5557815816f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:25.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.514+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/790081622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557815812d0 0x5557815816f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff0d80058a0 tx=0x7ff0d8013000 comp rx=0 tx=0).stop 2024-03-11T19:14:25.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.514+0000 7ff0f13c7740 1 -- 172.21.15.29:0/790081622 shutdown_connections 2024-03-11T19:14:25.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.514+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/790081622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555781586e30 0x5557815892c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.515+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/790081622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555781581cc0 0x5557815867c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.515+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/790081622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557815812d0 0x5557815816f0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.515+0000 7ff0f13c7740 1 -- 172.21.15.29:0/790081622 >> 172.21.15.29:0/790081622 conn(0x55578157c7f0 msgr2=0x55578157cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:25.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.515+0000 7ff0f13c7740 1 -- 172.21.15.29:0/790081622 shutdown_connections 2024-03-11T19:14:25.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.515+0000 7ff0f13c7740 1 -- 172.21.15.29:0/790081622 wait complete. 2024-03-11T19:14:25.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.515+0000 7ff0f13c7740 1 -- start start 2024-03-11T19:14:25.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.515+0000 7ff0f13c7740 1 Processor -- start 2024-03-11T19:14:25.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.516+0000 7ff0f13c7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557815812d0 0x55578158b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.516+0000 7ff0f13c7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555781581cc0 0x55578158b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.516+0000 7ff0f13c7740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555781586e30 0x5557815927f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.516+0000 7ff0f13c7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557811bd430 con 0x5557815812d0 2024-03-11T19:14:25.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.516+0000 7ff0f13c7740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555781179e00 con 0x555781586e30 2024-03-11T19:14:25.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.516+0000 7ff0f13c7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55578146efd0 con 0x555781581cc0 2024-03-11T19:14:25.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.516+0000 7ff0e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557815812d0 0x55578158b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.516+0000 7ff0eccb0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555781586e30 0x5557815927f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.516+0000 7ff0e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557815812d0 0x55578158b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54458/0 (socket says 172.21.15.29:54458) 2024-03-11T19:14:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.516+0000 7ff0e7fff640 1 -- 172.21.15.29:0/1661417570 learned_addr learned my addr 172.21.15.29:0/1661417570 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.516+0000 7ff0e77fe640 1 --2- 172.21.15.29:0/1661417570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555781581cc0 0x55578158b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.516+0000 7ff0f13c7740 10 client.0 _ll_register_callbacks cb 0x5557814c4fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:14:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.517+0000 7ff0eccb0640 1 -- 172.21.15.29:0/1661417570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555781581cc0 msgr2=0x55578158b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.517+0000 7ff0eccb0640 1 --2- 172.21.15.29:0/1661417570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555781581cc0 0x55578158b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.517+0000 7ff0eccb0640 1 -- 172.21.15.29:0/1661417570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557815812d0 msgr2=0x55578158b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.517+0000 7ff0eccb0640 1 --2- 172.21.15.29:0/1661417570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557815812d0 0x55578158b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.517+0000 7ff0eccb0640 1 -- 172.21.15.29:0/1661417570 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d8010cc0 con 0x555781586e30 2024-03-11T19:14:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.517+0000 7ff0e7fff640 1 --2- 172.21.15.29:0/1661417570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557815812d0 0x55578158b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:14:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.517+0000 7ff0eccb0640 1 --2- 172.21.15.29:0/1661417570 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555781586e30 0x5557815927f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff0e009a210 tx=0x7ff0e0099f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81d66798beff4c4e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.517+0000 7ff0c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:14:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.517+0000 7ff0c77fe640 1 -- 172.21.15.29:0/1661417570 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e00a2cb0 con 0x555781586e30 2024-03-11T19:14:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.517+0000 7ff0c77fe640 1 -- 172.21.15.29:0/1661417570 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff0e00850c0 con 0x555781586e30 2024-03-11T19:14:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.517+0000 7ff0f13c7740 1 -- 172.21.15.29:0/1661417570 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5557811e97e0 con 0x555781586e30 2024-03-11T19:14:25.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.518+0000 7ff0c77fe640 1 -- 172.21.15.29:0/1661417570 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0e000bcb0 con 0x555781586e30 2024-03-11T19:14:25.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.518+0000 7ff0f13c7740 10 client.26042 fetch_fsmap learned FSMap version 27 2024-03-11T19:14:25.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.518+0000 7ff0f13c7740 1 -- 172.21.15.29:0/1661417570 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5557814e8610 con 0x555781586e30 2024-03-11T19:14:25.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.519+0000 7ff0c77fe640 1 -- 172.21.15.29:0/1661417570 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5557811e97e0 con 0x555781586e30 2024-03-11T19:14:25.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.519+0000 7ff0c77fe640 1 -- 172.21.15.29:0/1661417570 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff0e001d060 con 0x555781586e30 2024-03-11T19:14:25.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.519+0000 7ff0f13c7740 10 client.26042 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:14:25.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.519+0000 7ff0f13c7740 10 client.26042 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:14:25.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.519+0000 7ff0f13c7740 20 client.26042 populate_metadata read hostname 'smithi029' 2024-03-11T19:14:25.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.519+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/1661417570 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555781597350 0x5557815997a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.519+0000 7ff0f13c7740 4 client.26042 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:14:25.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.519+0000 7ff0f13c7740 1 -- 172.21.15.29:0/1661417570 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555781226720 con 0x555781597350 2024-03-11T19:14:25.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.519+0000 7ff0e7fff640 1 --2- 172.21.15.29:0/1661417570 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555781597350 0x5557815997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.520+0000 7ff0e7fff640 1 --2- 172.21.15.29:0/1661417570 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555781597350 0x5557815997a0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a634b56804135fbd server_cookie=3e11dc3914dbd817 in_seq=0 out_seq=0 2024-03-11T19:14:25.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.520+0000 7ff0c77fe640 10 client.26042 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:14:25.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.521+0000 7ff0c77fe640 1 -- 172.21.15.29:0/1661417570 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555781226720 con 0x555781597350 2024-03-11T19:14:25.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.521+0000 7ff0c77fe640 10 client.26042 handle_command_reply: tid=0 2024-03-11T19:14:25.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.521+0000 7ff0f13c7740 1 client.26042 shutdown 2024-03-11T19:14:25.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.521+0000 7ff0f13c7740 1 -- 172.21.15.29:0/1661417570 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555781586e30 msgr2=0x5557815927f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:25.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.521+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/1661417570 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555781586e30 0x5557815927f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff0e009a210 tx=0x7ff0e0099f70 comp rx=0 tx=0).stop 2024-03-11T19:14:25.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.521+0000 7ff0f13c7740 1 -- 172.21.15.29:0/1661417570 shutdown_connections 2024-03-11T19:14:25.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.521+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/1661417570 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555781586e30 0x5557815927f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.521+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/1661417570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555781581cc0 0x55578158b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.521+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/1661417570 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555781597350 0x5557815997a0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.521+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/1661417570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5557815812d0 0x55578158b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.521+0000 7ff0f13c7740 1 -- 172.21.15.29:0/1661417570 >> 172.21.15.29:0/1661417570 conn(0x55578157c7f0 msgr2=0x55578157ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:25.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.521+0000 7ff0c77fe640 0 client.26042 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:14:25.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.521+0000 7ff0f13c7740 1 -- 172.21.15.29:0/1661417570 shutdown_connections 2024-03-11T19:14:25.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.521+0000 7ff0f13c7740 1 -- 172.21.15.29:0/1661417570 wait complete. 2024-03-11T19:14:25.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.521+0000 7ff0f13c7740 20 client.26042 trim_cache size 0 max 16384 2024-03-11T19:14:25.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.551+0000 7ff0f13c7740 1 Processor -- start 2024-03-11T19:14:25.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.551+0000 7ff0f13c7740 1 -- start start 2024-03-11T19:14:25.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.552+0000 7ff0f13c7740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55578157c7f0 0x55578158b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.552+0000 7ff0f13c7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557815812d0 0x55578158b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.552+0000 7ff0f13c7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555781581cc0 0x5557815927b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.552+0000 7ff0f13c7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555781578120 con 0x555781581cc0 2024-03-11T19:14:25.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.552+0000 7ff0f13c7740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555781226720 con 0x55578157c7f0 2024-03-11T19:14:25.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.552+0000 7ff0f13c7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557811e97e0 con 0x5557815812d0 2024-03-11T19:14:25.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.552+0000 7ff0e7fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55578157c7f0 0x55578158b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.552+0000 7ff0e7fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55578157c7f0 0x55578158b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37488/0 (socket says 172.21.15.29:37488) 2024-03-11T19:14:25.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.552+0000 7ff0e7fff640 1 -- 172.21.15.29:0/2128375170 learned_addr learned my addr 172.21.15.29:0/2128375170 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:25.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.552+0000 7ff0e77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557815812d0 0x55578158b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.552+0000 7ff0eccb0640 1 --2- 172.21.15.29:0/2128375170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555781581cc0 0x5557815927b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.552+0000 7ff0e7fff640 1 -- 172.21.15.29:0/2128375170 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557815812d0 msgr2=0x55578158b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:25.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.552+0000 7ff0e7fff640 1 --2- 172.21.15.29:0/2128375170 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557815812d0 0x55578158b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.552+0000 7ff0e7fff640 1 -- 172.21.15.29:0/2128375170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555781581cc0 msgr2=0x5557815927b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:25.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.552+0000 7ff0e7fff640 1 --2- 172.21.15.29:0/2128375170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555781581cc0 0x5557815927b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.552+0000 7ff0e7fff640 1 -- 172.21.15.29:0/2128375170 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555781484ac0 con 0x55578157c7f0 2024-03-11T19:14:25.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.553+0000 7ff0e7fff640 1 --2- 172.21.15.29:0/2128375170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55578157c7f0 0x55578158b270 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8010c90 tx=0x7ff0d8003780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7098229a65b0cbe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:25.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.553+0000 7ff0cdffb640 1 -- 172.21.15.29:0/2128375170 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8003870 con 0x55578157c7f0 2024-03-11T19:14:25.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.553+0000 7ff0cdffb640 1 -- 172.21.15.29:0/2128375170 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff0d8039070 con 0x55578157c7f0 2024-03-11T19:14:25.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.554+0000 7ff0cdffb640 1 -- 172.21.15.29:0/2128375170 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d80354e0 con 0x55578157c7f0 2024-03-11T19:14:25.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.554+0000 7ff0f13c7740 1 -- 172.21.15.29:0/2128375170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55578157c7f0 msgr2=0x55578158b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:25.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.554+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/2128375170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55578157c7f0 0x55578158b270 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8010c90 tx=0x7ff0d8003780 comp rx=0 tx=0).stop 2024-03-11T19:14:25.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.555+0000 7ff0f13c7740 1 -- 172.21.15.29:0/2128375170 shutdown_connections 2024-03-11T19:14:25.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.555+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/2128375170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555781581cc0 0x5557815927b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.555+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/2128375170 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5557815812d0 0x55578158b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.555+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/2128375170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55578157c7f0 0x55578158b270 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.555+0000 7ff0f13c7740 1 -- 172.21.15.29:0/2128375170 >> 172.21.15.29:0/2128375170 conn(0x555781586900 msgr2=0x55578159e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:25.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.555+0000 7ff0f13c7740 1 -- 172.21.15.29:0/2128375170 shutdown_connections 2024-03-11T19:14:25.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.555+0000 7ff0f13c7740 1 -- 172.21.15.29:0/2128375170 wait complete. 2024-03-11T19:14:25.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.555+0000 7ff0f13c7740 1 -- start start 2024-03-11T19:14:25.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.556+0000 7ff0f13c7740 1 Processor -- start 2024-03-11T19:14:25.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.556+0000 7ff0f13c7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55578157c7f0 0x5557815925c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.556+0000 7ff0f13c7740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557815812d0 0x555781590c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.556+0000 7ff0f13c7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555781581cc0 0x555781591160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.556+0000 7ff0f13c7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555781591720 con 0x555781581cc0 2024-03-11T19:14:25.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.556+0000 7ff0f13c7740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555781591890 con 0x5557815812d0 2024-03-11T19:14:25.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.556+0000 7ff0f13c7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555781591a50 con 0x55578157c7f0 2024-03-11T19:14:25.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.556+0000 7ff0f13c7740 10 client.0 _ll_register_callbacks cb 0x7ff0d800b8f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:14:25.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.557+0000 7ff0e77fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557815812d0 0x555781590c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.557+0000 7ff0e77fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557815812d0 0x555781590c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37496/0 (socket says 172.21.15.29:37496) 2024-03-11T19:14:25.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.557+0000 7ff0e7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55578157c7f0 0x5557815925c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.557+0000 7ff0e77fe640 1 -- 172.21.15.29:0/3748816647 learned_addr learned my addr 172.21.15.29:0/3748816647 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:25.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.557+0000 7ff0e77fe640 1 -- 172.21.15.29:0/3748816647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55578157c7f0 msgr2=0x5557815925c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:25.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.557+0000 7ff0eccb0640 1 --2- 172.21.15.29:0/3748816647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555781581cc0 0x555781591160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.557+0000 7ff0e77fe640 1 --2- 172.21.15.29:0/3748816647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55578157c7f0 0x5557815925c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.557+0000 7ff0e77fe640 1 -- 172.21.15.29:0/3748816647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555781581cc0 msgr2=0x555781591160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:25.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.557+0000 7ff0e77fe640 1 --2- 172.21.15.29:0/3748816647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555781581cc0 0x555781591160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.557+0000 7ff0e77fe640 1 -- 172.21.15.29:0/3748816647 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5557814a8570 con 0x5557815812d0 2024-03-11T19:14:25.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.557+0000 7ff0eccb0640 1 --2- 172.21.15.29:0/3748816647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555781581cc0 0x555781591160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:14:25.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.557+0000 7ff0e7fff640 1 --2- 172.21.15.29:0/3748816647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55578157c7f0 0x5557815925c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:14:25.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.557+0000 7ff0e77fe640 1 --2- 172.21.15.29:0/3748816647 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557815812d0 0x555781590c20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc035000 tx=0x7ff0dc004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a64efef3ff4f4650 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:25.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.558+0000 7ff0c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:14:25.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.558+0000 7ff0c77fe640 1 -- 172.21.15.29:0/3748816647 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc02dea0 con 0x5557815812d0 2024-03-11T19:14:25.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.558+0000 7ff0c77fe640 1 -- 172.21.15.29:0/3748816647 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff0dc0044b0 con 0x5557815812d0 2024-03-11T19:14:25.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.558+0000 7ff0f13c7740 1 -- 172.21.15.29:0/3748816647 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff0d8015070 con 0x5557815812d0 2024-03-11T19:14:25.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.558+0000 7ff0c77fe640 1 -- 172.21.15.29:0/3748816647 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0dc004dc0 con 0x5557815812d0 2024-03-11T19:14:25.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.558+0000 7ff0f13c7740 10 client.26048 fetch_fsmap learned FSMap version 27 2024-03-11T19:14:25.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.558+0000 7ff0f13c7740 1 -- 172.21.15.29:0/3748816647 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5557814aa6a0 con 0x5557815812d0 2024-03-11T19:14:25.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.559+0000 7ff0c77fe640 1 -- 172.21.15.29:0/3748816647 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff0d8015070 con 0x5557815812d0 2024-03-11T19:14:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.559+0000 7ff0c77fe640 1 -- 172.21.15.29:0/3748816647 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff0dc03c040 con 0x5557815812d0 2024-03-11T19:14:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.559+0000 7ff0f13c7740 10 client.26048 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:14:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.559+0000 7ff0f13c7740 10 client.26048 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:14:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.559+0000 7ff0f13c7740 20 client.26048 populate_metadata read hostname 'smithi029' 2024-03-11T19:14:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.559+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/3748816647 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555781586e30 0x5557815f11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.559+0000 7ff0f13c7740 4 client.26048 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:14:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.559+0000 7ff0f13c7740 1 -- 172.21.15.29:0/3748816647 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555781275890 con 0x555781586e30 2024-03-11T19:14:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.559+0000 7ff0e7fff640 1 --2- 172.21.15.29:0/3748816647 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555781586e30 0x5557815f11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.560+0000 7ff0e7fff640 1 --2- 172.21.15.29:0/3748816647 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555781586e30 0x5557815f11f0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b857e97445e7c2d server_cookie=29d2f21b88debca6 in_seq=0 out_seq=0 2024-03-11T19:14:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.560+0000 7ff0c77fe640 10 client.26048 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:14:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.560+0000 7ff0c77fe640 1 -- 172.21.15.29:0/3748816647 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x555781275890 con 0x555781586e30 2024-03-11T19:14:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.560+0000 7ff0c77fe640 10 client.26048 handle_command_reply: tid=0 2024-03-11T19:14:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.560+0000 7ff0f13c7740 1 client.26048 shutdown 2024-03-11T19:14:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.561+0000 7ff0f13c7740 1 -- 172.21.15.29:0/3748816647 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557815812d0 msgr2=0x555781590c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.561+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/3748816647 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557815812d0 0x555781590c20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc035000 tx=0x7ff0dc004040 comp rx=0 tx=0).stop 2024-03-11T19:14:25.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.561+0000 7ff0f13c7740 1 -- 172.21.15.29:0/3748816647 shutdown_connections 2024-03-11T19:14:25.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.561+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/3748816647 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x555781586e30 0x5557815f11f0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.561+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/3748816647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555781581cc0 0x555781591160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.561+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/3748816647 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5557815812d0 0x555781590c20 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.561+0000 7ff0f13c7740 1 --2- 172.21.15.29:0/3748816647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55578157c7f0 0x5557815925c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.561+0000 7ff0f13c7740 1 -- 172.21.15.29:0/3748816647 >> 172.21.15.29:0/3748816647 conn(0x555781586900 msgr2=0x555781582f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:25.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.561+0000 7ff0c77fe640 0 client.26048 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:14:25.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.561+0000 7ff0f13c7740 1 -- 172.21.15.29:0/3748816647 shutdown_connections 2024-03-11T19:14:25.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.561+0000 7ff0f13c7740 1 -- 172.21.15.29:0/3748816647 wait complete. 2024-03-11T19:14:25.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.561+0000 7ff0f13c7740 20 client.26048 trim_cache size 0 max 16384 2024-03-11T19:14:25.566 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:14:25.567 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:14:25.567 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:14:25.567 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:14:25.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.562+0000 7ff0ee713640 1 -- 172.21.15.29:0/2949703902 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff0d00a1ab0 msgr2=0x7ff0d00a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:25.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.562+0000 7ff0ee713640 1 --2- 172.21.15.29:0/2949703902 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff0d00a1ab0 0x7ff0d00a3f70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8100880 tx=0x7ff0d800b390 comp rx=0 tx=0).stop 2024-03-11T19:14:25.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.562+0000 7ff0ee713640 1 -- 172.21.15.29:0/2949703902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0e8101370 msgr2=0x7ff0e81022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:25.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.562+0000 7ff0ee713640 1 --2- 172.21.15.29:0/2949703902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0e8101370 0x7ff0e81022d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff0e000ec60 tx=0x7ff0e000c7a0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.562+0000 7ff0ee713640 1 -- 172.21.15.29:0/2949703902 shutdown_connections 2024-03-11T19:14:25.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.562+0000 7ff0ee713640 1 --2- 172.21.15.29:0/2949703902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0e8101370 0x7ff0e81022d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.562+0000 7ff0ee713640 1 --2- 172.21.15.29:0/2949703902 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff0d00a1ab0 0x7ff0d00a3f70 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.562+0000 7ff0ee713640 1 --2- 172.21.15.29:0/2949703902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e81009b0 0x7ff0e8101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.562+0000 7ff0ee713640 1 --2- 172.21.15.29:0/2949703902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff0e80ff9c0 0x7ff0e8103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.562+0000 7ff0ee713640 1 -- 172.21.15.29:0/2949703902 >> 172.21.15.29:0/2949703902 conn(0x7ff0e80fa960 msgr2=0x7ff0e8105ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:25.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.562+0000 7ff0ee713640 1 -- 172.21.15.29:0/2949703902 shutdown_connections 2024-03-11T19:14:25.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.562+0000 7ff0ee713640 1 -- 172.21.15.29:0/2949703902 wait complete. 2024-03-11T19:14:25.583 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:14:25.583 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:14:25.622 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:25 smithi029 ceph-mon[32439]: pgmap v489: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 77 MiB/s wr, 1.25k op/s 2024-03-11T19:14:25.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.733+0000 7f51d136c640 1 -- 172.21.15.29:0/3254684281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51cc0ff470 msgr2=0x7f51cc0ff890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:25.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.733+0000 7f51d136c640 1 --2- 172.21.15.29:0/3254684281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51cc0ff470 0x7f51cc0ff890 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f51bc009900 tx=0x7f51bc02bc30 comp rx=0 tx=0).stop 2024-03-11T19:14:25.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.733+0000 7f51d136c640 1 -- 172.21.15.29:0/3254684281 shutdown_connections 2024-03-11T19:14:25.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.733+0000 7f51d136c640 1 --2- 172.21.15.29:0/3254684281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51cc102ea0 0x7f51cc109730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.733+0000 7f51d136c640 1 --2- 172.21.15.29:0/3254684281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc1024e0 0x7f51cc102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.733+0000 7f51d136c640 1 --2- 172.21.15.29:0/3254684281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51cc0ff470 0x7f51cc0ff890 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.733+0000 7f51d136c640 1 -- 172.21.15.29:0/3254684281 >> 172.21.15.29:0/3254684281 conn(0x7f51cc0fad60 msgr2=0x7f51cc0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:25.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.733+0000 7f51d136c640 1 -- 172.21.15.29:0/3254684281 shutdown_connections 2024-03-11T19:14:25.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.733+0000 7f51d136c640 1 -- 172.21.15.29:0/3254684281 wait complete. 2024-03-11T19:14:25.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.734+0000 7f51d136c640 1 Processor -- start 2024-03-11T19:14:25.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.734+0000 7f51d136c640 1 -- start start 2024-03-11T19:14:25.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.734+0000 7f51d136c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc0ff470 0x7f51cc196f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.734+0000 7f51d136c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51cc1024e0 0x7f51cc1974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.735+0000 7f51d136c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51cc102ea0 0x7f51cc19e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.735+0000 7f51d136c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc19ea80 con 0x7f51cc0ff470 2024-03-11T19:14:25.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.735+0000 7f51d136c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc19ebf0 con 0x7f51cc1024e0 2024-03-11T19:14:25.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.735+0000 7f51d136c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc19edb0 con 0x7f51cc102ea0 2024-03-11T19:14:25.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.735+0000 7f51cb7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51cc102ea0 0x7f51cc19e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.735+0000 7f51caffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc0ff470 0x7f51cc196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.735+0000 7f51cb7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51cc102ea0 0x7f51cc19e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41360/0 (socket says 172.21.15.29:41360) 2024-03-11T19:14:25.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.735+0000 7f51cb7fe640 1 -- 172.21.15.29:0/3266964930 learned_addr learned my addr 172.21.15.29:0/3266964930 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:25.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.735+0000 7f51cb7fe640 1 -- 172.21.15.29:0/3266964930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51cc1024e0 msgr2=0x7f51cc1974d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:14:25.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.735+0000 7f51cb7fe640 1 --2- 172.21.15.29:0/3266964930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51cc1024e0 0x7f51cc1974d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.735+0000 7f51cb7fe640 1 -- 172.21.15.29:0/3266964930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc0ff470 msgr2=0x7f51cc196f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:25.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.735+0000 7f51cb7fe640 1 --2- 172.21.15.29:0/3266964930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc0ff470 0x7f51cc196f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:25.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.735+0000 7f51cb7fe640 1 -- 172.21.15.29:0/3266964930 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51bc009560 con 0x7f51cc102ea0 2024-03-11T19:14:25.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.735+0000 7f51caffd640 1 --2- 172.21.15.29:0/3266964930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc0ff470 0x7f51cc196f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:14:25.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.736+0000 7f51cb7fe640 1 --2- 172.21.15.29:0/3266964930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51cc102ea0 0x7f51cc19e540 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f51c400eaa0 tx=0x7f51c400ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:25.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.736+0000 7f51b3fff640 1 -- 172.21.15.29:0/3266964930 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c400cdb0 con 0x7f51cc102ea0 2024-03-11T19:14:25.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.736+0000 7f51b3fff640 1 -- 172.21.15.29:0/3266964930 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f51c4005020 con 0x7f51cc102ea0 2024-03-11T19:14:25.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.736+0000 7f51d136c640 1 -- 172.21.15.29:0/3266964930 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51cc1002d0 con 0x7f51cc102ea0 2024-03-11T19:14:25.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.736+0000 7f51b3fff640 1 -- 172.21.15.29:0/3266964930 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c40116b0 con 0x7f51cc102ea0 2024-03-11T19:14:25.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.737+0000 7f51d136c640 1 -- 172.21.15.29:0/3266964930 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51cc100780 con 0x7f51cc102ea0 2024-03-11T19:14:25.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.738+0000 7f51d136c640 1 -- 172.21.15.29:0/3266964930 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51cc108540 con 0x7f51cc102ea0 2024-03-11T19:14:25.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.739+0000 7f51b3fff640 1 -- 172.21.15.29:0/3266964930 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f51c4011810 con 0x7f51cc102ea0 2024-03-11T19:14:25.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.740+0000 7f51b3fff640 1 --2- 172.21.15.29:0/3266964930 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51b40a1b80 0x7f51b40a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:25.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.741+0000 7f51caffd640 1 --2- 172.21.15.29:0/3266964930 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51b40a1b80 0x7f51b40a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:25.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.741+0000 7f51b3fff640 1 -- 172.21.15.29:0/3266964930 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f51c40cd4e0 con 0x7f51cc102ea0 2024-03-11T19:14:25.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.741+0000 7f51caffd640 1 --2- 172.21.15.29:0/3266964930 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51b40a1b80 0x7f51b40a4040 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f51bc009230 tx=0x7f51bc000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:25.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:25.745+0000 7f51b3fff640 1 -- 172.21.15.29:0/3266964930 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f51c4096770 con 0x7f51cc102ea0 2024-03-11T19:14:26.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:25 smithi190 ceph-mon[36292]: pgmap v489: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 77 MiB/s wr, 1.25k op/s 2024-03-11T19:14:26.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:25 smithi170 ceph-mon[35690]: pgmap v489: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 77 MiB/s wr, 1.25k op/s 2024-03-11T19:14:26.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.165+0000 7f51d136c640 1 -- 172.21.15.29:0/3266964930 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51cc10cc10 con 0x7f51cc102ea0 2024-03-11T19:14:26.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.166+0000 7f51b3fff640 1 -- 172.21.15.29:0/3266964930 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f51c401f090 con 0x7f51cc102ea0 2024-03-11T19:14:26.169 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:14:26.169 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:14:26.170 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:14:26.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.169+0000 7f51d136c640 1 -- 172.21.15.29:0/3266964930 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51b40a1b80 msgr2=0x7f51b40a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.169+0000 7f51d136c640 1 --2- 172.21.15.29:0/3266964930 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51b40a1b80 0x7f51b40a4040 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f51bc009230 tx=0x7f51bc000bc0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.169+0000 7f51d136c640 1 -- 172.21.15.29:0/3266964930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51cc102ea0 msgr2=0x7f51cc19e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.169+0000 7f51d136c640 1 --2- 172.21.15.29:0/3266964930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51cc102ea0 0x7f51cc19e540 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f51c400eaa0 tx=0x7f51c400ef60 comp rx=0 tx=0).stop 2024-03-11T19:14:26.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.169+0000 7f51d136c640 1 -- 172.21.15.29:0/3266964930 shutdown_connections 2024-03-11T19:14:26.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.169+0000 7f51d136c640 1 --2- 172.21.15.29:0/3266964930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51cc102ea0 0x7f51cc19e540 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.169+0000 7f51d136c640 1 --2- 172.21.15.29:0/3266964930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51cc1024e0 0x7f51cc1974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.169+0000 7f51d136c640 1 --2- 172.21.15.29:0/3266964930 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51b40a1b80 0x7f51b40a4040 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.169+0000 7f51d136c640 1 --2- 172.21.15.29:0/3266964930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51cc0ff470 0x7f51cc196f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.169+0000 7f51d136c640 1 -- 172.21.15.29:0/3266964930 >> 172.21.15.29:0/3266964930 conn(0x7f51cc0fad60 msgr2=0x7f51cc105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:26.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.169+0000 7f51d136c640 1 -- 172.21.15.29:0/3266964930 shutdown_connections 2024-03-11T19:14:26.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.169+0000 7f51d136c640 1 -- 172.21.15.29:0/3266964930 wait complete. 2024-03-11T19:14:26.189 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:14:26.189 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:14:26.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.398+0000 7f0e993d0640 1 -- 172.21.15.29:0/1249675696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e94101190 msgr2=0x7f0e941015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.398+0000 7f0e993d0640 1 --2- 172.21.15.29:0/1249675696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e94101190 0x7f0e941015b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0e84009860 tx=0x7f0e8402bb70 comp rx=0 tx=0).stop 2024-03-11T19:14:26.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.398+0000 7f0e993d0640 1 -- 172.21.15.29:0/1249675696 shutdown_connections 2024-03-11T19:14:26.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.398+0000 7f0e993d0640 1 --2- 172.21.15.29:0/1249675696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e94102d40 0x7f0e941095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.398+0000 7f0e993d0640 1 --2- 172.21.15.29:0/1249675696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e94102380 0x7f0e94102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.398+0000 7f0e993d0640 1 --2- 172.21.15.29:0/1249675696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e94101190 0x7f0e941015b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.398+0000 7f0e993d0640 1 -- 172.21.15.29:0/1249675696 >> 172.21.15.29:0/1249675696 conn(0x7f0e940fc720 msgr2=0x7f0e940feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:26.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.399+0000 7f0e993d0640 1 -- 172.21.15.29:0/1249675696 shutdown_connections 2024-03-11T19:14:26.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.399+0000 7f0e993d0640 1 -- 172.21.15.29:0/1249675696 wait complete. 2024-03-11T19:14:26.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.400+0000 7f0e993d0640 1 Processor -- start 2024-03-11T19:14:26.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.400+0000 7f0e993d0640 1 -- start start 2024-03-11T19:14:26.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.401+0000 7f0e993d0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e94101190 0x7f0e94199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.401+0000 7f0e993d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e94102380 0x7f0e941998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.401+0000 7f0e993d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e94102d40 0x7f0e941a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.401+0000 7f0e92ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e94101190 0x7f0e94199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.401+0000 7f0e92ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e94101190 0x7f0e94199390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41382/0 (socket says 172.21.15.29:41382) 2024-03-11T19:14:26.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.401+0000 7f0e927fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e94102380 0x7f0e941998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.401+0000 7f0e927fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e94102380 0x7f0e941998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37520/0 (socket says 172.21.15.29:37520) 2024-03-11T19:14:26.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.401+0000 7f0e993d0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e941a0e80 con 0x7f0e94102d40 2024-03-11T19:14:26.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.401+0000 7f0e993d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e941a0ff0 con 0x7f0e94102380 2024-03-11T19:14:26.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.401+0000 7f0e993d0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e941a11b0 con 0x7f0e94101190 2024-03-11T19:14:26.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.401+0000 7f0e92ffd640 1 -- 172.21.15.29:0/2967286538 learned_addr learned my addr 172.21.15.29:0/2967286538 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:26.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.401+0000 7f0e937fe640 1 --2- 172.21.15.29:0/2967286538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e94102d40 0x7f0e941a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.402+0000 7f0e927fc640 1 -- 172.21.15.29:0/2967286538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e94101190 msgr2=0x7f0e94199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.402+0000 7f0e927fc640 1 --2- 172.21.15.29:0/2967286538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e94101190 0x7f0e94199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.402+0000 7f0e927fc640 1 -- 172.21.15.29:0/2967286538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e94102d40 msgr2=0x7f0e941a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.402+0000 7f0e927fc640 1 --2- 172.21.15.29:0/2967286538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e94102d40 0x7f0e941a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.402+0000 7f0e927fc640 1 -- 172.21.15.29:0/2967286538 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e84009490 con 0x7f0e94102380 2024-03-11T19:14:26.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.402+0000 7f0e927fc640 1 --2- 172.21.15.29:0/2967286538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e94102380 0x7f0e941998d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0e8800e790 tx=0x7f0e8800ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:26.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.402+0000 7f0e7bfff640 1 -- 172.21.15.29:0/2967286538 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8800ccc0 con 0x7f0e94102380 2024-03-11T19:14:26.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.403+0000 7f0e7bfff640 1 -- 172.21.15.29:0/2967286538 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0e8800eea0 con 0x7f0e94102380 2024-03-11T19:14:26.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.403+0000 7f0e993d0640 1 -- 172.21.15.29:0/2967286538 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e941a1490 con 0x7f0e94102380 2024-03-11T19:14:26.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.403+0000 7f0e7bfff640 1 -- 172.21.15.29:0/2967286538 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e88018660 con 0x7f0e94102380 2024-03-11T19:14:26.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.403+0000 7f0e993d0640 1 -- 172.21.15.29:0/2967286538 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e941a19f0 con 0x7f0e94102380 2024-03-11T19:14:26.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.406+0000 7f0e9bf9b740 1 Processor -- start 2024-03-11T19:14:26.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.406+0000 7f0e7bfff640 1 -- 172.21.15.29:0/2967286538 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0e88010430 con 0x7f0e94102380 2024-03-11T19:14:26.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.408+0000 7f0e9bf9b740 1 -- start start 2024-03-11T19:14:26.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.408+0000 7f0e9bf9b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954b6a80 0x5558954b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.408+0000 7f0e9bf9b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954b7470 0x5558954bbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.408+0000 7f0e9bf9b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954bc5e0 0x5558954bea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.408+0000 7f0e9bf9b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55589546a960 con 0x5558954b7470 2024-03-11T19:14:26.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.408+0000 7f0e9bf9b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55589541e310 con 0x5558954b6a80 2024-03-11T19:14:26.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.408+0000 7f0e9bf9b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55589541b840 con 0x5558954bc5e0 2024-03-11T19:14:26.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.408+0000 7f0e937fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954b6a80 0x5558954b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.408+0000 7f0e92ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954b7470 0x5558954bbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.408+0000 7f0e92ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954b7470 0x5558954bbf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54526/0 (socket says 172.21.15.29:54526) 2024-03-11T19:14:26.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.408+0000 7f0e92ffd640 1 -- 172.21.15.29:0/429793214 learned_addr learned my addr 172.21.15.29:0/429793214 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:26.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.409+0000 7f0e937fe640 1 -- 172.21.15.29:0/429793214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954bc5e0 msgr2=0x5558954bea70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:14:26.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.409+0000 7f0e937fe640 1 --2- 172.21.15.29:0/429793214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954bc5e0 0x5558954bea70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.409+0000 7f0e937fe640 1 -- 172.21.15.29:0/429793214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954b7470 msgr2=0x5558954bbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.409+0000 7f0e937fe640 1 --2- 172.21.15.29:0/429793214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954b7470 0x5558954bbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.409+0000 7f0e937fe640 1 -- 172.21.15.29:0/429793214 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558953d9060 con 0x5558954b6a80 2024-03-11T19:14:26.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.409+0000 7f0e92ffd640 1 --2- 172.21.15.29:0/429793214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954b7470 0x5558954bbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:14:26.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.409+0000 7f0e937fe640 1 --2- 172.21.15.29:0/429793214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954b6a80 0x5558954b6ea0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009f00 tx=0x7f0e8c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7549a257155d32b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:26.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.409+0000 7f0e797fa640 1 -- 172.21.15.29:0/429793214 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c004d20 con 0x5558954b6a80 2024-03-11T19:14:26.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.409+0000 7f0e797fa640 1 -- 172.21.15.29:0/429793214 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0e8c004e80 con 0x5558954b6a80 2024-03-11T19:14:26.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.409+0000 7f0e797fa640 1 -- 172.21.15.29:0/429793214 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c005020 con 0x5558954b6a80 2024-03-11T19:14:26.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.410+0000 7f0e7bfff640 1 --2- 172.21.15.29:0/2967286538 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0e7c0a1b30 0x7f0e7c0a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.410+0000 7f0e92ffd640 1 --2- 172.21.15.29:0/2967286538 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0e7c0a1b30 0x7f0e7c0a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.410+0000 7f0e7bfff640 1 -- 172.21.15.29:0/2967286538 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0e88014070 con 0x7f0e94102380 2024-03-11T19:14:26.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.410+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/429793214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954b6a80 msgr2=0x5558954b6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.410+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/429793214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954b6a80 0x5558954b6ea0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009f00 tx=0x7f0e8c011810 comp rx=0 tx=0).stop 2024-03-11T19:14:26.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.410+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/429793214 shutdown_connections 2024-03-11T19:14:26.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.411+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/429793214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954bc5e0 0x5558954bea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.411+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/429793214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954b7470 0x5558954bbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.411+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/429793214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954b6a80 0x5558954b6ea0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.411+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/429793214 >> 172.21.15.29:0/429793214 conn(0x5558954b1ff0 msgr2=0x5558954b23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:26.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.411+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/429793214 shutdown_connections 2024-03-11T19:14:26.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.411+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/429793214 wait complete. 2024-03-11T19:14:26.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.411+0000 7f0e92ffd640 1 --2- 172.21.15.29:0/2967286538 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0e7c0a1b30 0x7f0e7c0a3ff0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x5558954b7870 tx=0x7f0e84034ad0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:26.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.411+0000 7f0e9bf9b740 1 -- start start 2024-03-11T19:14:26.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.412+0000 7f0e9bf9b740 1 Processor -- start 2024-03-11T19:14:26.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.412+0000 7f0e9bf9b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954b6a80 0x5558954c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.412+0000 7f0e9bf9b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954b7470 0x5558954c0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.412+0000 7f0e9bf9b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954bc5e0 0x5558954c8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.412+0000 7f0e9bf9b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555895417020 con 0x5558954bc5e0 2024-03-11T19:14:26.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.412+0000 7f0e9bf9b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558950ea430 con 0x5558954b6a80 2024-03-11T19:14:26.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.412+0000 7f0e9bf9b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558950a6e00 con 0x5558954b7470 2024-03-11T19:14:26.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.412+0000 7f0e9bf9b740 10 client.0 _ll_register_callbacks cb 0x5558953becb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:14:26.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.412+0000 7f0e92ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954b6a80 0x5558954c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.412+0000 7f0e92ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954b6a80 0x5558954c0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37540/0 (socket says 172.21.15.29:37540) 2024-03-11T19:14:26.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.412+0000 7f0e92ffd640 1 -- 172.21.15.29:0/3473428252 learned_addr learned my addr 172.21.15.29:0/3473428252 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:26.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.412+0000 7f0e937fe640 1 --2- 172.21.15.29:0/3473428252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954bc5e0 0x5558954c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.412+0000 7f0e927fc640 1 --2- 172.21.15.29:0/3473428252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954b7470 0x5558954c0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.413+0000 7f0e92ffd640 1 -- 172.21.15.29:0/3473428252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954b7470 msgr2=0x5558954c0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.413+0000 7f0e92ffd640 1 --2- 172.21.15.29:0/3473428252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954b7470 0x5558954c0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.413+0000 7f0e92ffd640 1 -- 172.21.15.29:0/3473428252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954bc5e0 msgr2=0x5558954c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.413+0000 7f0e92ffd640 1 --2- 172.21.15.29:0/3473428252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954bc5e0 0x5558954c8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.413+0000 7f0e92ffd640 1 -- 172.21.15.29:0/3473428252 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e8c009be0 con 0x5558954b6a80 2024-03-11T19:14:26.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.413+0000 7f0e92ffd640 1 --2- 172.21.15.29:0/3473428252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954b6a80 0x5558954c0ab0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0e8402c180 tx=0x7f0e8402bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=694ee93352263601 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:26.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.413+0000 7f0e76ff5640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:14:26.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.413+0000 7f0e76ff5640 1 -- 172.21.15.29:0/3473428252 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e84004380 con 0x5558954b6a80 2024-03-11T19:14:26.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.413+0000 7f0e76ff5640 1 -- 172.21.15.29:0/3473428252 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0e840035f0 con 0x5558954b6a80 2024-03-11T19:14:26.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.414+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/3473428252 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555895153720 con 0x5558954b6a80 2024-03-11T19:14:26.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.414+0000 7f0e76ff5640 1 -- 172.21.15.29:0/3473428252 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e840044e0 con 0x5558954b6a80 2024-03-11T19:14:26.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.414+0000 7f0e9bf9b740 10 client.26063 fetch_fsmap learned FSMap version 27 2024-03-11T19:14:26.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.414+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/3473428252 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5558954156a0 con 0x5558954b6a80 2024-03-11T19:14:26.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.414+0000 7f0e76ff5640 1 -- 172.21.15.29:0/3473428252 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x555895153720 con 0x5558954b6a80 2024-03-11T19:14:26.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.414+0000 7f0e76ff5640 1 -- 172.21.15.29:0/3473428252 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0e840328f0 con 0x5558954b6a80 2024-03-11T19:14:26.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.414+0000 7f0e9bf9b740 10 client.26063 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:14:26.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.414+0000 7f0e9bf9b740 10 client.26063 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:14:26.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.414+0000 7f0e9bf9b740 20 client.26063 populate_metadata read hostname 'smithi029' 2024-03-11T19:14:26.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.414+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/3473428252 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5558954ccc40 0x5558954cf060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.414+0000 7f0e9bf9b740 4 client.26063 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:14:26.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.414+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/3473428252 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5558951a2890 con 0x5558954ccc40 2024-03-11T19:14:26.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.415+0000 7f0e927fc640 1 --2- 172.21.15.29:0/3473428252 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5558954ccc40 0x5558954cf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.415+0000 7f0e76ff5640 10 client.26063 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:14:26.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.416+0000 7f0e927fc640 1 --2- 172.21.15.29:0/3473428252 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5558954ccc40 0x5558954cf060 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d08cc63e5226affa server_cookie=e77716b9d9113dae in_seq=0 out_seq=0 2024-03-11T19:14:26.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.417+0000 7f0e76ff5640 1 -- 172.21.15.29:0/3473428252 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5558951a2890 con 0x5558954ccc40 2024-03-11T19:14:26.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.417+0000 7f0e76ff5640 10 client.26063 handle_command_reply: tid=0 2024-03-11T19:14:26.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.417+0000 7f0e9bf9b740 1 client.26063 shutdown 2024-03-11T19:14:26.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.417+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/3473428252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954b6a80 msgr2=0x5558954c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.417+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/3473428252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954b6a80 0x5558954c0ab0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0e8402c180 tx=0x7f0e8402bb70 comp rx=0 tx=0).stop 2024-03-11T19:14:26.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.417+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/3473428252 shutdown_connections 2024-03-11T19:14:26.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.417+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/3473428252 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5558954ccc40 0x5558954cf060 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.417+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/3473428252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954bc5e0 0x5558954c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.417+0000 7f0e76ff5640 0 client.26063 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:14:26.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.417+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/3473428252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954b7470 0x5558954c0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.417+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/3473428252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954b6a80 0x5558954c0ab0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.417+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/3473428252 >> 172.21.15.29:0/3473428252 conn(0x5558954b1ff0 msgr2=0x5558954b3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:26.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.418+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/3473428252 shutdown_connections 2024-03-11T19:14:26.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.418+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/3473428252 wait complete. 2024-03-11T19:14:26.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.418+0000 7f0e9bf9b740 20 client.26063 trim_cache size 0 max 16384 2024-03-11T19:14:26.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.436+0000 7f0e9bf9b740 1 Processor -- start 2024-03-11T19:14:26.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.436+0000 7f0e9bf9b740 1 -- start start 2024-03-11T19:14:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.436+0000 7f0e9bf9b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954b6a80 0x5558954b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.436+0000 7f0e9bf9b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954b7470 0x5558954c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.436+0000 7f0e92ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954b6a80 0x5558954b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.436+0000 7f0e92ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954b6a80 0x5558954b23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41406/0 (socket says 172.21.15.29:41406) 2024-03-11T19:14:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.436+0000 7f0e9bf9b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954bc5e0 0x5558954c06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.436+0000 7f0e927fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954b7470 0x5558954c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.436+0000 7f0e92ffd640 1 -- 172.21.15.29:0/2447511238 learned_addr learned my addr 172.21.15.29:0/2447511238 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.436+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/2447511238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55589539c130 con 0x5558954b7470 2024-03-11T19:14:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.436+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/2447511238 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555895153720 con 0x5558954bc5e0 2024-03-11T19:14:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.436+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/2447511238 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558954b7b60 con 0x5558954b6a80 2024-03-11T19:14:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.436+0000 7f0e937fe640 1 --2- 172.21.15.29:0/2447511238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954bc5e0 0x5558954c06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.437+0000 7f0e937fe640 1 -- 172.21.15.29:0/2447511238 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954b6a80 msgr2=0x5558954b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.437+0000 7f0e937fe640 1 --2- 172.21.15.29:0/2447511238 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954b6a80 0x5558954b23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.437+0000 7f0e937fe640 1 -- 172.21.15.29:0/2447511238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954b7470 msgr2=0x5558954c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.437+0000 7f0e937fe640 1 --2- 172.21.15.29:0/2447511238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954b7470 0x5558954c7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.437+0000 7f0e937fe640 1 -- 172.21.15.29:0/2447511238 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558953b1c20 con 0x5558954bc5e0 2024-03-11T19:14:26.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.437+0000 7f0e937fe640 1 --2- 172.21.15.29:0/2447511238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954bc5e0 0x5558954c06c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009270 tx=0x7f0e8c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fca8d6dbeec484e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:26.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.437+0000 7f0e78ff9640 1 -- 172.21.15.29:0/2447511238 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c0043e0 con 0x5558954bc5e0 2024-03-11T19:14:26.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.437+0000 7f0e78ff9640 1 -- 172.21.15.29:0/2447511238 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0e8c004540 con 0x5558954bc5e0 2024-03-11T19:14:26.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.437+0000 7f0e78ff9640 1 -- 172.21.15.29:0/2447511238 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c005260 con 0x5558954bc5e0 2024-03-11T19:14:26.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.439+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/2447511238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954bc5e0 msgr2=0x5558954c06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.439+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/2447511238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954bc5e0 0x5558954c06c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009270 tx=0x7f0e8c004ab0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.440+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/2447511238 shutdown_connections 2024-03-11T19:14:26.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.440+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/2447511238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954bc5e0 0x5558954c06c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.440+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/2447511238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954b7470 0x5558954c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.440+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/2447511238 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954b6a80 0x5558954b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.440+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/2447511238 >> 172.21.15.29:0/2447511238 conn(0x5558954b1ff0 msgr2=0x5558954a4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:26.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.440+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/2447511238 shutdown_connections 2024-03-11T19:14:26.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.441+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/2447511238 wait complete. 2024-03-11T19:14:26.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.441+0000 7f0e9bf9b740 1 -- start start 2024-03-11T19:14:26.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.441+0000 7f0e9bf9b740 1 Processor -- start 2024-03-11T19:14:26.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.442+0000 7f0e9bf9b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954b6a80 0x5558954c6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.442+0000 7f0e9bf9b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954b7470 0x5558954c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.442+0000 7f0e9bf9b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954bc5e0 0x5558954c72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.442+0000 7f0e9bf9b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555895523130 con 0x5558954b6a80 2024-03-11T19:14:26.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.442+0000 7f0e927fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954b7470 0x5558954c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.442+0000 7f0e927fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954b7470 0x5558954c6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37548/0 (socket says 172.21.15.29:37548) 2024-03-11T19:14:26.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.442+0000 7f0e927fc640 1 -- 172.21.15.29:0/1165081530 learned_addr learned my addr 172.21.15.29:0/1165081530 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:26.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.442+0000 7f0e92ffd640 1 --2- 172.21.15.29:0/1165081530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954b6a80 0x5558954c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.442+0000 7f0e937fe640 1 --2- 172.21.15.29:0/1165081530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954bc5e0 0x5558954c72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.442+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/1165081530 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555895523270 con 0x5558954b7470 2024-03-11T19:14:26.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.442+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/1165081530 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558955233b0 con 0x5558954bc5e0 2024-03-11T19:14:26.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.442+0000 7f0e9bf9b740 10 client.0 _ll_register_callbacks cb 0x7f0e8c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:14:26.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.443+0000 7f0e937fe640 1 -- 172.21.15.29:0/1165081530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954b7470 msgr2=0x5558954c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.443+0000 7f0e937fe640 1 --2- 172.21.15.29:0/1165081530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954b7470 0x5558954c6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.443+0000 7f0e937fe640 1 -- 172.21.15.29:0/1165081530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954b6a80 msgr2=0x5558954c6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.443+0000 7f0e937fe640 1 --2- 172.21.15.29:0/1165081530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954b6a80 0x5558954c6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.443+0000 7f0e937fe640 1 -- 172.21.15.29:0/1165081530 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558953d9060 con 0x5558954bc5e0 2024-03-11T19:14:26.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.444+0000 7f0e937fe640 1 --2- 172.21.15.29:0/1165081530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954bc5e0 0x5558954c72b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x5558954b6860 tx=0x7f0e8c003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83502b62fb6285db server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:26.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.444+0000 7f0e76ff5640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:14:26.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.444+0000 7f0e76ff5640 1 -- 172.21.15.29:0/1165081530 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c0043a0 con 0x5558954bc5e0 2024-03-11T19:14:26.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.444+0000 7f0e76ff5640 1 -- 172.21.15.29:0/1165081530 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0e8c005000 con 0x5558954bc5e0 2024-03-11T19:14:26.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.444+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/1165081530 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5558951167e0 con 0x5558954bc5e0 2024-03-11T19:14:26.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.444+0000 7f0e76ff5640 1 -- 172.21.15.29:0/1165081530 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e8c0051c0 con 0x5558954bc5e0 2024-03-11T19:14:26.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.445+0000 7f0e9bf9b740 10 client.26077 fetch_fsmap learned FSMap version 27 2024-03-11T19:14:26.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.445+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/1165081530 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5558953d78c0 con 0x5558954bc5e0 2024-03-11T19:14:26.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.445+0000 7f0e76ff5640 1 -- 172.21.15.29:0/1165081530 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5558951167e0 con 0x5558954bc5e0 2024-03-11T19:14:26.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.445+0000 7f0e76ff5640 1 -- 172.21.15.29:0/1165081530 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0e8c0377f0 con 0x5558954bc5e0 2024-03-11T19:14:26.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.445+0000 7f0e9bf9b740 10 client.26077 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:14:26.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.445+0000 7f0e9bf9b740 10 client.26077 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:14:26.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.445+0000 7f0e9bf9b740 20 client.26077 populate_metadata read hostname 'smithi029' 2024-03-11T19:14:26.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.445+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/1165081530 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5558954ccc40 0x555895526b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.445+0000 7f0e9bf9b740 4 client.26077 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:14:26.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.445+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/1165081530 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5558950c2700 con 0x5558954ccc40 2024-03-11T19:14:26.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.446+0000 7f0e92ffd640 1 --2- 172.21.15.29:0/1165081530 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5558954ccc40 0x555895526b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.447+0000 7f0e92ffd640 1 --2- 172.21.15.29:0/1165081530 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5558954ccc40 0x555895526b00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=da7eb7f400c67924 server_cookie=7d9c417ec4b0b4cd in_seq=0 out_seq=0 2024-03-11T19:14:26.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.447+0000 7f0e76ff5640 10 client.26077 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:14:26.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.447+0000 7f0e76ff5640 1 -- 172.21.15.29:0/1165081530 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5558950c2700 con 0x5558954ccc40 2024-03-11T19:14:26.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.447+0000 7f0e76ff5640 10 client.26077 handle_command_reply: tid=0 2024-03-11T19:14:26.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.447+0000 7f0e9bf9b740 1 client.26077 shutdown 2024-03-11T19:14:26.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.448+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/1165081530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954bc5e0 msgr2=0x5558954c72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.448+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/1165081530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954bc5e0 0x5558954c72b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x5558954b6860 tx=0x7f0e8c003ba0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.448+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/1165081530 shutdown_connections 2024-03-11T19:14:26.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.449+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/1165081530 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5558954ccc40 0x555895526b00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.449+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/1165081530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558954bc5e0 0x5558954c72b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.449+0000 7f0e76ff5640 0 client.26077 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:14:26.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.449+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/1165081530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5558954b7470 0x5558954c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.449+0000 7f0e9bf9b740 1 --2- 172.21.15.29:0/1165081530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5558954b6a80 0x5558954c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.449+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/1165081530 >> 172.21.15.29:0/1165081530 conn(0x5558954b1ff0 msgr2=0x5558954b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:26.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.449+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/1165081530 shutdown_connections 2024-03-11T19:14:26.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.449+0000 7f0e9bf9b740 1 -- 172.21.15.29:0/1165081530 wait complete. 2024-03-11T19:14:26.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.449+0000 7f0e9bf9b740 20 client.26077 trim_cache size 0 max 16384 2024-03-11T19:14:26.454 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:14:26.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.450+0000 7f0e993d0640 1 -- 172.21.15.29:0/2967286538 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0e7c0a1b30 msgr2=0x7f0e7c0a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.450+0000 7f0e993d0640 1 --2- 172.21.15.29:0/2967286538 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0e7c0a1b30 0x7f0e7c0a3ff0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x5558954b7870 tx=0x7f0e84034ad0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.450+0000 7f0e993d0640 1 -- 172.21.15.29:0/2967286538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e94102380 msgr2=0x7f0e941998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.450+0000 7f0e993d0640 1 --2- 172.21.15.29:0/2967286538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e94102380 0x7f0e941998d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0e8800e790 tx=0x7f0e8800ec50 comp rx=0 tx=0).stop 2024-03-11T19:14:26.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.451+0000 7f0e993d0640 1 -- 172.21.15.29:0/2967286538 shutdown_connections 2024-03-11T19:14:26.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.451+0000 7f0e993d0640 1 --2- 172.21.15.29:0/2967286538 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0e7c0a1b30 0x7f0e7c0a3ff0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.451+0000 7f0e993d0640 1 --2- 172.21.15.29:0/2967286538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e94102d40 0x7f0e941a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.451+0000 7f0e993d0640 1 --2- 172.21.15.29:0/2967286538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e94102380 0x7f0e941998d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.451+0000 7f0e993d0640 1 --2- 172.21.15.29:0/2967286538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e94101190 0x7f0e94199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.451+0000 7f0e993d0640 1 -- 172.21.15.29:0/2967286538 >> 172.21.15.29:0/2967286538 conn(0x7f0e940fc720 msgr2=0x7f0e94107b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:26.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.451+0000 7f0e993d0640 1 -- 172.21.15.29:0/2967286538 shutdown_connections 2024-03-11T19:14:26.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.451+0000 7f0e993d0640 1 -- 172.21.15.29:0/2967286538 wait complete. 2024-03-11T19:14:26.472 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:14:26.472 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:14:26.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.640+0000 7f07a80ae640 1 -- 172.21.15.29:0/2488177082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07a01070b0 msgr2=0x7f07a01094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.640+0000 7f07a80ae640 1 --2- 172.21.15.29:0/2488177082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07a01070b0 0x7f07a01094a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0798009ca0 tx=0x7f079802c0f0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.640+0000 7f07a80ae640 1 -- 172.21.15.29:0/2488177082 shutdown_connections 2024-03-11T19:14:26.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.640+0000 7f07a80ae640 1 --2- 172.21.15.29:0/2488177082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07a01070b0 0x7f07a01094a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.640+0000 7f07a80ae640 1 --2- 172.21.15.29:0/2488177082 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07a0104780 0x7f07a0106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.640+0000 7f07a80ae640 1 --2- 172.21.15.29:0/2488177082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07a01016a0 0x7f07a0101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.640+0000 7f07a80ae640 1 -- 172.21.15.29:0/2488177082 >> 172.21.15.29:0/2488177082 conn(0x7f07a00fad70 msgr2=0x7f07a00fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:26.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.640+0000 7f07a80ae640 1 -- 172.21.15.29:0/2488177082 shutdown_connections 2024-03-11T19:14:26.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.640+0000 7f07a80ae640 1 -- 172.21.15.29:0/2488177082 wait complete. 2024-03-11T19:14:26.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.641+0000 7f07a80ae640 1 Processor -- start 2024-03-11T19:14:26.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.642+0000 7f07a80ae640 1 -- start start 2024-03-11T19:14:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.642+0000 7f07a80ae640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07a01016a0 0x7f07a019d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.642+0000 7f07a80ae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07a0104780 0x7f07a019dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.642+0000 7f07a80ae640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07a01070b0 0x7f07a01a4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.642+0000 7f07a80ae640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a01a51d0 con 0x7f07a0104780 2024-03-11T19:14:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.642+0000 7f07a80ae640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a01a5340 con 0x7f07a01070b0 2024-03-11T19:14:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.642+0000 7f07a80ae640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a01a5500 con 0x7f07a01016a0 2024-03-11T19:14:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.642+0000 7f07a5649640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07a0104780 0x7f07a019dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.643+0000 7f07a664b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07a01070b0 0x7f07a01a4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.643+0000 7f07a664b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07a01070b0 0x7f07a01a4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37564/0 (socket says 172.21.15.29:37564) 2024-03-11T19:14:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.643+0000 7f07a664b640 1 -- 172.21.15.29:0/2089091460 learned_addr learned my addr 172.21.15.29:0/2089091460 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.643+0000 7f07a5e4a640 1 --2- 172.21.15.29:0/2089091460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07a01016a0 0x7f07a019d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.643+0000 7f07a664b640 1 -- 172.21.15.29:0/2089091460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07a01016a0 msgr2=0x7f07a019d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.643+0000 7f07a664b640 1 --2- 172.21.15.29:0/2089091460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07a01016a0 0x7f07a019d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.643+0000 7f07a664b640 1 -- 172.21.15.29:0/2089091460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07a0104780 msgr2=0x7f07a019dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.643+0000 7f07a664b640 1 --2- 172.21.15.29:0/2089091460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07a0104780 0x7f07a019dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.643+0000 7f07a664b640 1 -- 172.21.15.29:0/2089091460 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0798009900 con 0x7f07a01070b0 2024-03-11T19:14:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.643+0000 7f07a5649640 1 --2- 172.21.15.29:0/2089091460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07a0104780 0x7f07a019dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:14:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.643+0000 7f07a664b640 1 --2- 172.21.15.29:0/2089091460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07a01070b0 0x7f07a01a4c90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0798037040 tx=0x7f0798032980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.644+0000 7f0796ffd640 1 -- 172.21.15.29:0/2089091460 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0798034e70 con 0x7f07a01070b0 2024-03-11T19:14:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.644+0000 7f0796ffd640 1 -- 172.21.15.29:0/2089091460 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0798003e20 con 0x7f07a01070b0 2024-03-11T19:14:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.644+0000 7f0796ffd640 1 -- 172.21.15.29:0/2089091460 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07980047d0 con 0x7f07a01070b0 2024-03-11T19:14:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.644+0000 7f07a80ae640 1 -- 172.21.15.29:0/2089091460 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a01a5780 con 0x7f07a01070b0 2024-03-11T19:14:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.644+0000 7f07a80ae640 1 -- 172.21.15.29:0/2089091460 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a01a5b20 con 0x7f07a01070b0 2024-03-11T19:14:26.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.647+0000 7f0796ffd640 1 -- 172.21.15.29:0/2089091460 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f079803b040 con 0x7f07a01070b0 2024-03-11T19:14:26.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.647+0000 7f07aa922740 1 Processor -- start 2024-03-11T19:14:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.647+0000 7f07aa922740 1 -- start start 2024-03-11T19:14:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.648+0000 7f0796ffd640 1 --2- 172.21.15.29:0/2089091460 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f078c0a1ab0 0x7f078c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.648+0000 7f07aa922740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102270 0x5569aa102690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.648+0000 7f07aa922740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa102c60 0x5569aa107760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.648+0000 7f07aa922740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa107dd0 0x5569aa10a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.648+0000 7f07aa922740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569aa0be960 con 0x5569aa107dd0 2024-03-11T19:14:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.648+0000 7f07aa922740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569aa072310 con 0x5569aa102270 2024-03-11T19:14:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.648+0000 7f07aa922740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569aa06f840 con 0x5569aa102c60 2024-03-11T19:14:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.648+0000 7f0796ffd640 1 -- 172.21.15.29:0/2089091460 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f07980efb10 con 0x7f07a01070b0 2024-03-11T19:14:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.648+0000 7f07a664b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa107dd0 0x5569aa10a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.648+0000 7f07a5649640 1 --2- 172.21.15.29:0/2089091460 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f078c0a1ab0 0x7f078c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.648+0000 7f07a5e4a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102270 0x5569aa102690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.648+0000 7f07a664b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa107dd0 0x5569aa10a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54566/0 (socket says 172.21.15.29:54566) 2024-03-11T19:14:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.648+0000 7f07a664b640 1 -- 172.21.15.29:0/3364410531 learned_addr learned my addr 172.21.15.29:0/3364410531 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.648+0000 7f07a5649640 1 --2- 172.21.15.29:0/3364410531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa102c60 0x5569aa107760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.648+0000 7f07a664b640 1 -- 172.21.15.29:0/3364410531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa102c60 msgr2=0x5569aa107760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.648+0000 7f07a664b640 1 --2- 172.21.15.29:0/3364410531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa102c60 0x5569aa107760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.648+0000 7f07a664b640 1 -- 172.21.15.29:0/3364410531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102270 msgr2=0x5569aa102690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.648+0000 7f07a664b640 1 --2- 172.21.15.29:0/3364410531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102270 0x5569aa102690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.648+0000 7f07a664b640 1 -- 172.21.15.29:0/3364410531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569aa02d060 con 0x5569aa107dd0 2024-03-11T19:14:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.649+0000 7f07a5649640 1 --2- 172.21.15.29:0/2089091460 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f078c0a1ab0 0x7f078c0a3f70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f07a019ebf0 tx=0x7f079c00bc30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.649+0000 7f07a664b640 1 --2- 172.21.15.29:0/3364410531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa107dd0 0x5569aa10a260 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f07980c4cf0 tx=0x7f0798003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5de764be8d9ddb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.649+0000 7f078bfff640 1 -- 172.21.15.29:0/3364410531 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07980bd180 con 0x5569aa107dd0 2024-03-11T19:14:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.649+0000 7f078bfff640 1 -- 172.21.15.29:0/3364410531 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f07980bca80 con 0x5569aa107dd0 2024-03-11T19:14:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.649+0000 7f078bfff640 1 -- 172.21.15.29:0/3364410531 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07980b90f0 con 0x5569aa107dd0 2024-03-11T19:14:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.650+0000 7f07aa922740 1 -- 172.21.15.29:0/3364410531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa107dd0 msgr2=0x5569aa10a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.650+0000 7f07aa922740 1 --2- 172.21.15.29:0/3364410531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa107dd0 0x5569aa10a260 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f07980c4cf0 tx=0x7f0798003f90 comp rx=0 tx=0).stop 2024-03-11T19:14:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.651+0000 7f07aa922740 1 -- 172.21.15.29:0/3364410531 shutdown_connections 2024-03-11T19:14:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.651+0000 7f07aa922740 1 --2- 172.21.15.29:0/3364410531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa107dd0 0x5569aa10a260 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.651+0000 7f07aa922740 1 --2- 172.21.15.29:0/3364410531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa102c60 0x5569aa107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.651+0000 7f07aa922740 1 --2- 172.21.15.29:0/3364410531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102270 0x5569aa102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.651+0000 7f07aa922740 1 -- 172.21.15.29:0/3364410531 >> 172.21.15.29:0/3364410531 conn(0x5569aa0fd7e0 msgr2=0x5569aa0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.651+0000 7f07aa922740 1 -- 172.21.15.29:0/3364410531 shutdown_connections 2024-03-11T19:14:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.651+0000 7f07aa922740 1 -- 172.21.15.29:0/3364410531 wait complete. 2024-03-11T19:14:26.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.652+0000 7f07aa922740 1 -- start start 2024-03-11T19:14:26.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.652+0000 7f07aa922740 1 Processor -- start 2024-03-11T19:14:26.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.652+0000 7f07aa922740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102270 0x5569aa10c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.652+0000 7f07aa922740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa102c60 0x5569aa10c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.652+0000 7f07aa922740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa107dd0 0x5569aa1135b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.652+0000 7f07aa922740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569aa06b020 con 0x5569aa107dd0 2024-03-11T19:14:26.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.652+0000 7f07aa922740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569a9d3e430 con 0x5569aa102270 2024-03-11T19:14:26.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.652+0000 7f07aa922740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569a9cfae00 con 0x5569aa102c60 2024-03-11T19:14:26.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.653+0000 7f07a5e4a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102270 0x5569aa10c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.653+0000 7f07a5e4a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102270 0x5569aa10c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37576/0 (socket says 172.21.15.29:37576) 2024-03-11T19:14:26.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.653+0000 7f07a5e4a640 1 -- 172.21.15.29:0/1123085851 learned_addr learned my addr 172.21.15.29:0/1123085851 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:26.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.653+0000 7f07aa922740 10 client.0 _ll_register_callbacks cb 0x5569aa012cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:14:26.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.653+0000 7f07a5649640 1 --2- 172.21.15.29:0/1123085851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa102c60 0x5569aa10c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.653+0000 7f07a664b640 1 --2- 172.21.15.29:0/1123085851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa107dd0 0x5569aa1135b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.653+0000 7f07a5e4a640 1 -- 172.21.15.29:0/1123085851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa102c60 msgr2=0x5569aa10c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.653+0000 7f07a5e4a640 1 --2- 172.21.15.29:0/1123085851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa102c60 0x5569aa10c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.653+0000 7f07a5e4a640 1 -- 172.21.15.29:0/1123085851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa107dd0 msgr2=0x5569aa1135b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.653+0000 7f07a5e4a640 1 --2- 172.21.15.29:0/1123085851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa107dd0 0x5569aa1135b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.653+0000 7f07a5e4a640 1 -- 172.21.15.29:0/1123085851 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a01a5b20 con 0x5569aa102270 2024-03-11T19:14:26.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.653+0000 7f07a664b640 1 --2- 172.21.15.29:0/1123085851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa107dd0 0x5569aa1135b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:14:26.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.653+0000 7f07a5e4a640 1 --2- 172.21.15.29:0/1123085851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102270 0x5569aa10c030 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0790012700 tx=0x7f0790012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5ec5e8deab9bd07 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:26.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.653+0000 7f07897fa640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:14:26.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.653+0000 7f07897fa640 1 -- 172.21.15.29:0/1123085851 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0790017070 con 0x5569aa102270 2024-03-11T19:14:26.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.654+0000 7f07aa922740 1 -- 172.21.15.29:0/1123085851 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5569a9d6a7e0 con 0x5569aa102270 2024-03-11T19:14:26.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.654+0000 7f07897fa640 1 -- 172.21.15.29:0/1123085851 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0790004260 con 0x5569aa102270 2024-03-11T19:14:26.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.654+0000 7f07897fa640 1 -- 172.21.15.29:0/1123085851 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0790004420 con 0x5569aa102270 2024-03-11T19:14:26.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.654+0000 7f07aa922740 10 client.26081 fetch_fsmap learned FSMap version 27 2024-03-11T19:14:26.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.654+0000 7f07aa922740 1 -- 172.21.15.29:0/1123085851 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5569aa0696a0 con 0x5569aa102270 2024-03-11T19:14:26.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.654+0000 7f07897fa640 1 -- 172.21.15.29:0/1123085851 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5569a9d6a7e0 con 0x5569aa102270 2024-03-11T19:14:26.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.655+0000 7f07897fa640 1 -- 172.21.15.29:0/1123085851 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0790003d10 con 0x5569aa102270 2024-03-11T19:14:26.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.655+0000 7f07aa922740 10 client.26081 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:14:26.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.655+0000 7f07aa922740 10 client.26081 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:14:26.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.655+0000 7f07aa922740 20 client.26081 populate_metadata read hostname 'smithi029' 2024-03-11T19:14:26.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.655+0000 7f07aa922740 1 --2- 172.21.15.29:0/1123085851 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5569aa118220 0x5569aa11a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.655+0000 7f07aa922740 4 client.26081 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:14:26.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.655+0000 7f07aa922740 1 -- 172.21.15.29:0/1123085851 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5569a9da7720 con 0x5569aa118220 2024-03-11T19:14:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.655+0000 7f07a5649640 1 --2- 172.21.15.29:0/1123085851 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5569aa118220 0x5569aa11a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.656+0000 7f07a5649640 1 --2- 172.21.15.29:0/1123085851 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5569aa118220 0x5569aa11a660 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=74e533a413af998e server_cookie=c9fbc8c7ba897fd7 in_seq=0 out_seq=0 2024-03-11T19:14:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.656+0000 7f07897fa640 10 client.26081 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:14:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.656+0000 7f07897fa640 1 -- 172.21.15.29:0/1123085851 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5569a9da7720 con 0x5569aa118220 2024-03-11T19:14:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.656+0000 7f07897fa640 10 client.26081 handle_command_reply: tid=0 2024-03-11T19:14:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.656+0000 7f07aa922740 1 client.26081 shutdown 2024-03-11T19:14:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.657+0000 7f07aa922740 1 -- 172.21.15.29:0/1123085851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102270 msgr2=0x5569aa10c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.657+0000 7f07aa922740 1 --2- 172.21.15.29:0/1123085851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102270 0x5569aa10c030 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0790012700 tx=0x7f0790012bc0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.657+0000 7f07aa922740 1 -- 172.21.15.29:0/1123085851 shutdown_connections 2024-03-11T19:14:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.657+0000 7f07aa922740 1 --2- 172.21.15.29:0/1123085851 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5569aa118220 0x5569aa11a660 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.657+0000 7f07aa922740 1 --2- 172.21.15.29:0/1123085851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa107dd0 0x5569aa1135b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.657+0000 7f07aa922740 1 --2- 172.21.15.29:0/1123085851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa102c60 0x5569aa10c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.657+0000 7f07aa922740 1 --2- 172.21.15.29:0/1123085851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102270 0x5569aa10c030 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.657+0000 7f07aa922740 1 -- 172.21.15.29:0/1123085851 >> 172.21.15.29:0/1123085851 conn(0x5569aa0fd7e0 msgr2=0x5569aa0fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.657+0000 7f07897fa640 0 client.26081 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:14:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.657+0000 7f07aa922740 1 -- 172.21.15.29:0/1123085851 shutdown_connections 2024-03-11T19:14:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.658+0000 7f07aa922740 1 -- 172.21.15.29:0/1123085851 wait complete. 2024-03-11T19:14:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.658+0000 7f07aa922740 20 client.26081 trim_cache size 0 max 16384 2024-03-11T19:14:26.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.680+0000 7f07aa922740 1 Processor -- start 2024-03-11T19:14:26.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.680+0000 7f07aa922740 1 -- start start 2024-03-11T19:14:26.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.681+0000 7f07aa922740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102270 0x5569aa0fc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.681+0000 7f07aa922740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa102c60 0x5569aa1134d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.681+0000 7f07aa922740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa107dd0 0x5569aa0fc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.681+0000 7f07aa922740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569a9ff0130 con 0x5569aa102c60 2024-03-11T19:14:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.681+0000 7f07aa922740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569aa10a620 con 0x5569aa102270 2024-03-11T19:14:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.681+0000 7f07aa922740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569aa103090 con 0x5569aa107dd0 2024-03-11T19:14:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.681+0000 7f07a5e4a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102270 0x5569aa0fc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.681+0000 7f07a5649640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa102c60 0x5569aa1134d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.681+0000 7f07a5e4a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102270 0x5569aa0fc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37590/0 (socket says 172.21.15.29:37590) 2024-03-11T19:14:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.681+0000 7f07a5e4a640 1 -- 172.21.15.29:0/3263126043 learned_addr learned my addr 172.21.15.29:0/3263126043 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.681+0000 7f07a664b640 1 --2- 172.21.15.29:0/3263126043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa107dd0 0x5569aa0fc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.682+0000 7f07a5e4a640 1 -- 172.21.15.29:0/3263126043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa107dd0 msgr2=0x5569aa0fc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.682+0000 7f07a5e4a640 1 --2- 172.21.15.29:0/3263126043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa107dd0 0x5569aa0fc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.682+0000 7f07a5e4a640 1 -- 172.21.15.29:0/3263126043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa102c60 msgr2=0x5569aa1134d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.682+0000 7f07a5e4a640 1 --2- 172.21.15.29:0/3263126043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa102c60 0x5569aa1134d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.682+0000 7f07a5e4a640 1 -- 172.21.15.29:0/3263126043 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569aa005c20 con 0x5569aa102270 2024-03-11T19:14:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.682+0000 7f07a5e4a640 1 --2- 172.21.15.29:0/3263126043 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102270 0x5569aa0fc2f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f07900126d0 tx=0x7f0790009dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17791ede43cf218c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.682+0000 7f078b7fe640 1 -- 172.21.15.29:0/3263126043 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0790017070 con 0x5569aa102270 2024-03-11T19:14:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.682+0000 7f078b7fe640 1 -- 172.21.15.29:0/3263126043 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0790018040 con 0x5569aa102270 2024-03-11T19:14:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.682+0000 7f078b7fe640 1 -- 172.21.15.29:0/3263126043 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0790011c00 con 0x5569aa102270 2024-03-11T19:14:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.683+0000 7f07aa922740 1 -- 172.21.15.29:0/3263126043 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102270 msgr2=0x5569aa0fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.683+0000 7f07aa922740 1 --2- 172.21.15.29:0/3263126043 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102270 0x5569aa0fc2f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f07900126d0 tx=0x7f0790009dd0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.683+0000 7f07aa922740 1 -- 172.21.15.29:0/3263126043 shutdown_connections 2024-03-11T19:14:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.683+0000 7f07aa922740 1 --2- 172.21.15.29:0/3263126043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa107dd0 0x5569aa0fc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.683+0000 7f07aa922740 1 --2- 172.21.15.29:0/3263126043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa102c60 0x5569aa1134d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.683+0000 7f07aa922740 1 --2- 172.21.15.29:0/3263126043 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102270 0x5569aa0fc2f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.683+0000 7f07aa922740 1 -- 172.21.15.29:0/3263126043 >> 172.21.15.29:0/3263126043 conn(0x5569aa0fd7e0 msgr2=0x5569aa104080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.683+0000 7f07aa922740 1 -- 172.21.15.29:0/3263126043 shutdown_connections 2024-03-11T19:14:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.683+0000 7f07aa922740 1 -- 172.21.15.29:0/3263126043 wait complete. 2024-03-11T19:14:26.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.683+0000 7f07aa922740 1 -- start start 2024-03-11T19:14:26.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.684+0000 7f07aa922740 1 Processor -- start 2024-03-11T19:14:26.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.684+0000 7f07aa922740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa102270 0x5569aa113170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.684+0000 7f07aa922740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102c60 0x5569aa1117d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.684+0000 7f07aa922740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa107dd0 0x5569aa111d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.684+0000 7f07aa922740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569aa113960 con 0x5569aa107dd0 2024-03-11T19:14:26.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.684+0000 7f07aa922740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569aa112250 con 0x5569aa102c60 2024-03-11T19:14:26.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.684+0000 7f07a5e4a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa102270 0x5569aa113170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.684+0000 7f07a5e4a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa102270 0x5569aa113170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41474/0 (socket says 172.21.15.29:41474) 2024-03-11T19:14:26.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.684+0000 7f07a5e4a640 1 -- 172.21.15.29:0/2183464704 learned_addr learned my addr 172.21.15.29:0/2183464704 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:26.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.684+0000 7f07aa922740 1 -- 172.21.15.29:0/2183464704 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569aa112410 con 0x5569aa102270 2024-03-11T19:14:26.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.684+0000 7f07a5649640 1 --2- 172.21.15.29:0/2183464704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102c60 0x5569aa1117d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.684+0000 7f07a664b640 1 --2- 172.21.15.29:0/2183464704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa107dd0 0x5569aa111d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.684+0000 7f07aa922740 10 client.0 _ll_register_callbacks cb 0x7f07900096c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:14:26.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.685+0000 7f07a5e4a640 1 -- 172.21.15.29:0/2183464704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102c60 msgr2=0x5569aa1117d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.685+0000 7f07a5e4a640 1 --2- 172.21.15.29:0/2183464704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102c60 0x5569aa1117d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.685+0000 7f07a5e4a640 1 -- 172.21.15.29:0/2183464704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa107dd0 msgr2=0x5569aa111d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.685+0000 7f07a5e4a640 1 --2- 172.21.15.29:0/2183464704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa107dd0 0x5569aa111d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.685+0000 7f07a5e4a640 1 -- 172.21.15.29:0/2183464704 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5569aa0696a0 con 0x5569aa102270 2024-03-11T19:14:26.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.685+0000 7f07a5e4a640 1 --2- 172.21.15.29:0/2183464704 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa102270 0x5569aa113170 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f079000f440 tx=0x7f0790011bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd47c4be306fce09 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:26.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.685+0000 7f07897fa640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:14:26.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.685+0000 7f07897fa640 1 -- 172.21.15.29:0/2183464704 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0790036070 con 0x5569aa102270 2024-03-11T19:14:26.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.685+0000 7f07897fa640 1 -- 172.21.15.29:0/2183464704 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0790000e40 con 0x5569aa102270 2024-03-11T19:14:26.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.685+0000 7f07aa922740 1 -- 172.21.15.29:0/2183464704 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f07980bca80 con 0x5569aa102270 2024-03-11T19:14:26.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.686+0000 7f07897fa640 1 -- 172.21.15.29:0/2183464704 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0790017040 con 0x5569aa102270 2024-03-11T19:14:26.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.686+0000 7f07aa922740 10 client.26095 fetch_fsmap learned FSMap version 27 2024-03-11T19:14:26.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.686+0000 7f07aa922740 1 -- 172.21.15.29:0/2183464704 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5569aa02b8c0 con 0x5569aa102270 2024-03-11T19:14:26.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.686+0000 7f07897fa640 1 -- 172.21.15.29:0/2183464704 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0790041020 con 0x5569aa102270 2024-03-11T19:14:26.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.686+0000 7f07897fa640 1 -- 172.21.15.29:0/2183464704 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0790039020 con 0x5569aa102270 2024-03-11T19:14:26.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.687+0000 7f07aa922740 10 client.26095 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:14:26.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.687+0000 7f07aa922740 10 client.26095 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:14:26.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.687+0000 7f07aa922740 20 client.26095 populate_metadata read hostname 'smithi029' 2024-03-11T19:14:26.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.687+0000 7f07aa922740 1 --2- 172.21.15.29:0/2183464704 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5569aa16f970 0x5569aa171d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.687+0000 7f07aa922740 4 client.26095 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:14:26.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.687+0000 7f07aa922740 1 -- 172.21.15.29:0/2183464704 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f07980bca80 con 0x5569aa16f970 2024-03-11T19:14:26.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.688+0000 7f07a5649640 1 --2- 172.21.15.29:0/2183464704 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5569aa16f970 0x5569aa171d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.689+0000 7f07a5649640 1 --2- 172.21.15.29:0/2183464704 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5569aa16f970 0x5569aa171d90 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fa64e75a839202a3 server_cookie=f424c8a565527ce8 in_seq=0 out_seq=0 2024-03-11T19:14:26.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.689+0000 7f07897fa640 10 client.26095 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:14:26.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.689+0000 7f07897fa640 1 -- 172.21.15.29:0/2183464704 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f07980bca80 con 0x5569aa16f970 2024-03-11T19:14:26.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.689+0000 7f07897fa640 10 client.26095 handle_command_reply: tid=0 2024-03-11T19:14:26.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.689+0000 7f07aa922740 1 client.26095 shutdown 2024-03-11T19:14:26.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.689+0000 7f07aa922740 1 -- 172.21.15.29:0/2183464704 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa102270 msgr2=0x5569aa113170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.689+0000 7f07aa922740 1 --2- 172.21.15.29:0/2183464704 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa102270 0x5569aa113170 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f079000f440 tx=0x7f0790011bd0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.689+0000 7f07aa922740 1 -- 172.21.15.29:0/2183464704 shutdown_connections 2024-03-11T19:14:26.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.689+0000 7f07aa922740 1 --2- 172.21.15.29:0/2183464704 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5569aa16f970 0x5569aa171d90 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.689+0000 7f07897fa640 0 client.26095 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:14:26.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.689+0000 7f07aa922740 1 --2- 172.21.15.29:0/2183464704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569aa107dd0 0x5569aa111d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.689+0000 7f07aa922740 1 --2- 172.21.15.29:0/2183464704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5569aa102c60 0x5569aa1117d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.689+0000 7f07aa922740 1 --2- 172.21.15.29:0/2183464704 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569aa102270 0x5569aa113170 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.689+0000 7f07aa922740 1 -- 172.21.15.29:0/2183464704 >> 172.21.15.29:0/2183464704 conn(0x5569aa0fd7e0 msgr2=0x5569aa100a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:26.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.689+0000 7f07aa922740 1 -- 172.21.15.29:0/2183464704 shutdown_connections 2024-03-11T19:14:26.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.689+0000 7f07aa922740 1 -- 172.21.15.29:0/2183464704 wait complete. 2024-03-11T19:14:26.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.690+0000 7f07aa922740 20 client.26095 trim_cache size 0 max 16384 2024-03-11T19:14:26.695 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:14:26.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.690+0000 7f07a80ae640 1 -- 172.21.15.29:0/2089091460 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f078c0a1ab0 msgr2=0x7f078c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.690+0000 7f07a80ae640 1 --2- 172.21.15.29:0/2089091460 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f078c0a1ab0 0x7f078c0a3f70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f07a019ebf0 tx=0x7f079c00bc30 comp rx=0 tx=0).stop 2024-03-11T19:14:26.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.690+0000 7f07a80ae640 1 -- 172.21.15.29:0/2089091460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07a01070b0 msgr2=0x7f07a01a4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.690+0000 7f07a80ae640 1 --2- 172.21.15.29:0/2089091460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07a01070b0 0x7f07a01a4c90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0798037040 tx=0x7f0798032980 comp rx=0 tx=0).stop 2024-03-11T19:14:26.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.690+0000 7f07a80ae640 1 -- 172.21.15.29:0/2089091460 shutdown_connections 2024-03-11T19:14:26.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.690+0000 7f07a80ae640 1 --2- 172.21.15.29:0/2089091460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07a01070b0 0x7f07a01a4c90 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.690+0000 7f07a80ae640 1 --2- 172.21.15.29:0/2089091460 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f078c0a1ab0 0x7f078c0a3f70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.690+0000 7f07a80ae640 1 --2- 172.21.15.29:0/2089091460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07a0104780 0x7f07a019dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.690+0000 7f07a80ae640 1 --2- 172.21.15.29:0/2089091460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07a01016a0 0x7f07a019d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.690+0000 7f07a80ae640 1 -- 172.21.15.29:0/2089091460 >> 172.21.15.29:0/2089091460 conn(0x7f07a00fad70 msgr2=0x7f07a01050c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:26.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.691+0000 7f07a80ae640 1 -- 172.21.15.29:0/2089091460 shutdown_connections 2024-03-11T19:14:26.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.691+0000 7f07a80ae640 1 -- 172.21.15.29:0/2089091460 wait complete. 2024-03-11T19:14:26.710 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:14:26.710 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:14:26.745 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:26 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3266964930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:14:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.879+0000 7fabb8bbc640 1 -- 172.21.15.29:0/2620292673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabb4106e60 msgr2=0x7fabb41092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.879+0000 7fabb8bbc640 1 --2- 172.21.15.29:0/2620292673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabb4106e60 0x7fabb41092b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7faba4009860 tx=0x7faba402bb70 comp rx=0 tx=0).stop 2024-03-11T19:14:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.879+0000 7fabb8bbc640 1 -- 172.21.15.29:0/2620292673 shutdown_connections 2024-03-11T19:14:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.879+0000 7fabb8bbc640 1 --2- 172.21.15.29:0/2620292673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabb4106e60 0x7fabb41092b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.879+0000 7fabb8bbc640 1 --2- 172.21.15.29:0/2620292673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabb4104530 0x7fabb4106920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.879+0000 7fabb8bbc640 1 --2- 172.21.15.29:0/2620292673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fabb4101c00 0x7fabb4103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.879+0000 7fabb8bbc640 1 -- 172.21.15.29:0/2620292673 >> 172.21.15.29:0/2620292673 conn(0x7fabb40fb610 msgr2=0x7fabb40fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.879+0000 7fabb8bbc640 1 -- 172.21.15.29:0/2620292673 shutdown_connections 2024-03-11T19:14:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.879+0000 7fabb8bbc640 1 -- 172.21.15.29:0/2620292673 wait complete. 2024-03-11T19:14:26.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.880+0000 7fabb8bbc640 1 Processor -- start 2024-03-11T19:14:26.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.880+0000 7fabb8bbc640 1 -- start start 2024-03-11T19:14:26.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.880+0000 7fabb8bbc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fabb4101c00 0x7fabb4197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.880+0000 7fabb8bbc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabb4104530 0x7fabb4197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.880+0000 7fabb8bbc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabb4106e60 0x7fabb419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.880+0000 7fabb8bbc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb419ec40 con 0x7fabb4106e60 2024-03-11T19:14:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.880+0000 7fabb8bbc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb419edb0 con 0x7fabb4101c00 2024-03-11T19:14:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.880+0000 7fabb8bbc640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb419ef70 con 0x7fabb4104530 2024-03-11T19:14:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.880+0000 7fabb1d9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabb4104530 0x7fabb4197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.880+0000 7fabb259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fabb4101c00 0x7fabb4197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.880+0000 7fabb1d9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabb4104530 0x7fabb4197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41486/0 (socket says 172.21.15.29:41486) 2024-03-11T19:14:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.880+0000 7fabb1d9b640 1 -- 172.21.15.29:0/2347994229 learned_addr learned my addr 172.21.15.29:0/2347994229 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.880+0000 7fabb2d9d640 1 --2- 172.21.15.29:0/2347994229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabb4106e60 0x7fabb419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.881+0000 7fabb259c640 1 -- 172.21.15.29:0/2347994229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabb4104530 msgr2=0x7fabb4197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.881+0000 7fabb259c640 1 --2- 172.21.15.29:0/2347994229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabb4104530 0x7fabb4197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.881+0000 7fabb259c640 1 -- 172.21.15.29:0/2347994229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabb4106e60 msgr2=0x7fabb419e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.881+0000 7fabb259c640 1 --2- 172.21.15.29:0/2347994229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabb4106e60 0x7fabb419e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.881+0000 7fabb259c640 1 -- 172.21.15.29:0/2347994229 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba4009490 con 0x7fabb4101c00 2024-03-11T19:14:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.881+0000 7fabb259c640 1 --2- 172.21.15.29:0/2347994229 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fabb4101c00 0x7fabb4197150 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00e9f0 tx=0x7fab9c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.881+0000 7fabab7fe640 1 -- 172.21.15.29:0/2347994229 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c00ccc0 con 0x7fabb4101c00 2024-03-11T19:14:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.881+0000 7fabab7fe640 1 -- 172.21.15.29:0/2347994229 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fab9c0044d0 con 0x7fabb4101c00 2024-03-11T19:14:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.881+0000 7fabab7fe640 1 -- 172.21.15.29:0/2347994229 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c004a00 con 0x7fabb4101c00 2024-03-11T19:14:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.881+0000 7fabb8bbc640 1 -- 172.21.15.29:0/2347994229 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb419f250 con 0x7fabb4101c00 2024-03-11T19:14:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.881+0000 7fabb8bbc640 1 -- 172.21.15.29:0/2347994229 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb419f6b0 con 0x7fabb4101c00 2024-03-11T19:14:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.884+0000 7fabab7fe640 1 -- 172.21.15.29:0/2347994229 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fab9c004b60 con 0x7fabb4101c00 2024-03-11T19:14:26.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.885+0000 7fabab7fe640 1 --2- 172.21.15.29:0/2347994229 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faba00a1a30 0x7faba00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.885+0000 7fabab7fe640 1 -- 172.21.15.29:0/2347994229 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fab9c014070 con 0x7fabb4101c00 2024-03-11T19:14:26.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.885+0000 7fabbb793740 1 Processor -- start 2024-03-11T19:14:26.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.885+0000 7fabb1d9b640 1 --2- 172.21.15.29:0/2347994229 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faba00a1a30 0x7faba00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.886+0000 7fabbb793740 1 -- start start 2024-03-11T19:14:26.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.886+0000 7fabbb793740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de7f0a0 0x55c01de7f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.886+0000 7fabbb793740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de81f70 0x55c01de84360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.886+0000 7fabbb793740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de848a0 0x55c01de86c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.886+0000 7fabbb793740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c01de32960 con 0x55c01de7f0a0 2024-03-11T19:14:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.886+0000 7fabbb793740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c01dde6310 con 0x55c01de848a0 2024-03-11T19:14:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.886+0000 7fabbb793740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c01dde3840 con 0x55c01de81f70 2024-03-11T19:14:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.886+0000 7fabb2d9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de848a0 0x55c01de86c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.886+0000 7fabb2d9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de848a0 0x55c01de86c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37634/0 (socket says 172.21.15.29:37634) 2024-03-11T19:14:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.886+0000 7fabb2d9d640 1 -- 172.21.15.29:0/2600561221 learned_addr learned my addr 172.21.15.29:0/2600561221 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.886+0000 7fabb1d9b640 1 --2- 172.21.15.29:0/2347994229 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faba00a1a30 0x7faba00a3ef0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fabb4198660 tx=0x7fabac010040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.886+0000 7fabb1d9b640 1 --2- 172.21.15.29:0/2600561221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de81f70 0x55c01de84360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.886+0000 7fabb259c640 1 --2- 172.21.15.29:0/2600561221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de7f0a0 0x55c01de7f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.887+0000 7fabb2d9d640 1 -- 172.21.15.29:0/2600561221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de81f70 msgr2=0x55c01de84360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.887+0000 7fabb2d9d640 1 --2- 172.21.15.29:0/2600561221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de81f70 0x55c01de84360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.887+0000 7fabb2d9d640 1 -- 172.21.15.29:0/2600561221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de7f0a0 msgr2=0x55c01de7f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.887+0000 7fabb2d9d640 1 --2- 172.21.15.29:0/2600561221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de7f0a0 0x55c01de7f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.887+0000 7fabb2d9d640 1 -- 172.21.15.29:0/2600561221 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c01dda1060 con 0x55c01de848a0 2024-03-11T19:14:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.887+0000 7fabb2d9d640 1 --2- 172.21.15.29:0/2600561221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de848a0 0x55c01de86c90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faba4034b10 tx=0x7faba4005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd1727b522067c2e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.887+0000 7faba8ff9640 1 -- 172.21.15.29:0/2600561221 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba4034e70 con 0x55c01de848a0 2024-03-11T19:14:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.887+0000 7faba8ff9640 1 -- 172.21.15.29:0/2600561221 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faba402ce90 con 0x55c01de848a0 2024-03-11T19:14:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.887+0000 7faba8ff9640 1 -- 172.21.15.29:0/2600561221 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba402c8e0 con 0x55c01de848a0 2024-03-11T19:14:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.888+0000 7fabbb793740 1 -- 172.21.15.29:0/2600561221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de848a0 msgr2=0x55c01de86c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.888+0000 7fabbb793740 1 --2- 172.21.15.29:0/2600561221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de848a0 0x55c01de86c90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faba4034b10 tx=0x7faba4005b20 comp rx=0 tx=0).stop 2024-03-11T19:14:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.888+0000 7fabbb793740 1 -- 172.21.15.29:0/2600561221 shutdown_connections 2024-03-11T19:14:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.888+0000 7fabbb793740 1 --2- 172.21.15.29:0/2600561221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de848a0 0x55c01de86c90 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.888+0000 7fabbb793740 1 --2- 172.21.15.29:0/2600561221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de81f70 0x55c01de84360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.888+0000 7fabbb793740 1 --2- 172.21.15.29:0/2600561221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de7f0a0 0x55c01de7f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.888+0000 7fabbb793740 1 -- 172.21.15.29:0/2600561221 >> 172.21.15.29:0/2600561221 conn(0x55c01de708e0 msgr2=0x55c01de70ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:26.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.888+0000 7fabbb793740 1 -- 172.21.15.29:0/2600561221 shutdown_connections 2024-03-11T19:14:26.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.888+0000 7fabbb793740 1 -- 172.21.15.29:0/2600561221 wait complete. 2024-03-11T19:14:26.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.889+0000 7fabbb793740 1 -- start start 2024-03-11T19:14:26.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.889+0000 7fabbb793740 1 Processor -- start 2024-03-11T19:14:26.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.889+0000 7fabbb793740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de7f0a0 0x55c01de88c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.889+0000 7fabbb793740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de81f70 0x55c01de89140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.889+0000 7fabbb793740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de848a0 0x55c01de90180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.889+0000 7fabbb793740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c01dddf020 con 0x55c01de7f0a0 2024-03-11T19:14:26.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.889+0000 7fabbb793740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c01dab2430 con 0x55c01de81f70 2024-03-11T19:14:26.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.889+0000 7fabbb793740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c01da6ee00 con 0x55c01de848a0 2024-03-11T19:14:26.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.889+0000 7fabbb793740 10 client.0 _ll_register_callbacks cb 0x55c01dd86cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:14:26.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.889+0000 7fabb259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de7f0a0 0x55c01de88c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.889+0000 7fabb1d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de81f70 0x55c01de89140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.889+0000 7fabb1d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de81f70 0x55c01de89140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37640/0 (socket says 172.21.15.29:37640) 2024-03-11T19:14:26.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.889+0000 7fabb1d9b640 1 -- 172.21.15.29:0/1606279291 learned_addr learned my addr 172.21.15.29:0/1606279291 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:26.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.889+0000 7fabb2d9d640 1 --2- 172.21.15.29:0/1606279291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de848a0 0x55c01de90180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.890+0000 7fabb1d9b640 1 -- 172.21.15.29:0/1606279291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de848a0 msgr2=0x55c01de90180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.890+0000 7fabb1d9b640 1 --2- 172.21.15.29:0/1606279291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de848a0 0x55c01de90180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.890+0000 7fabb1d9b640 1 -- 172.21.15.29:0/1606279291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de7f0a0 msgr2=0x55c01de88c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.890+0000 7fabb1d9b640 1 --2- 172.21.15.29:0/1606279291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de7f0a0 0x55c01de88c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.890+0000 7fabb1d9b640 1 -- 172.21.15.29:0/1606279291 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb4109e90 con 0x55c01de81f70 2024-03-11T19:14:26.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.890+0000 7fabb1d9b640 1 --2- 172.21.15.29:0/1606279291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de81f70 0x55c01de89140 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fabac0045f0 tx=0x7fabac004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14c6a998aa36668b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:26.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.890+0000 7fab99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:14:26.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.890+0000 7fab99ffb640 1 -- 172.21.15.29:0/1606279291 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac029070 con 0x55c01de81f70 2024-03-11T19:14:26.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.890+0000 7fab99ffb640 1 -- 172.21.15.29:0/1606279291 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fabac00dd50 con 0x55c01de81f70 2024-03-11T19:14:26.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.890+0000 7fabbb793740 1 -- 172.21.15.29:0/1606279291 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c01dade7e0 con 0x55c01de81f70 2024-03-11T19:14:26.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.890+0000 7fab99ffb640 1 -- 172.21.15.29:0/1606279291 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabac013040 con 0x55c01de81f70 2024-03-11T19:14:26.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.890+0000 7fabbb793740 10 client.26099 fetch_fsmap learned FSMap version 27 2024-03-11T19:14:26.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.890+0000 7fabbb793740 1 -- 172.21.15.29:0/1606279291 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55c01dddd6a0 con 0x55c01de81f70 2024-03-11T19:14:26.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.891+0000 7fab99ffb640 1 -- 172.21.15.29:0/1606279291 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55c01dade7e0 con 0x55c01de81f70 2024-03-11T19:14:26.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.891+0000 7fab99ffb640 1 -- 172.21.15.29:0/1606279291 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fabac02c030 con 0x55c01de81f70 2024-03-11T19:14:26.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.891+0000 7fabbb793740 10 client.26099 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:14:26.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.891+0000 7fabbb793740 10 client.26099 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:14:26.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.891+0000 7fabbb793740 20 client.26099 populate_metadata read hostname 'smithi029' 2024-03-11T19:14:26.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.891+0000 7fabbb793740 1 --2- 172.21.15.29:0/1606279291 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55c01de94ee0 0x55c01de97300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.891+0000 7fabbb793740 4 client.26099 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:14:26.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.891+0000 7fabbb793740 1 -- 172.21.15.29:0/1606279291 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c01db1b720 con 0x55c01de94ee0 2024-03-11T19:14:26.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.891+0000 7fabb259c640 1 --2- 172.21.15.29:0/1606279291 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55c01de94ee0 0x55c01de97300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.892+0000 7fab99ffb640 10 client.26099 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:14:26.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.892+0000 7fabb259c640 1 --2- 172.21.15.29:0/1606279291 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55c01de94ee0 0x55c01de97300 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=99727a466fabd4d2 server_cookie=73ffd09cefb4639d in_seq=0 out_seq=0 2024-03-11T19:14:26.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.893+0000 7fab99ffb640 1 -- 172.21.15.29:0/1606279291 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c01db1b720 con 0x55c01de94ee0 2024-03-11T19:14:26.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.893+0000 7fab99ffb640 10 client.26099 handle_command_reply: tid=0 2024-03-11T19:14:26.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.893+0000 7fabbb793740 1 client.26099 shutdown 2024-03-11T19:14:26.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.894+0000 7fabbb793740 1 -- 172.21.15.29:0/1606279291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de81f70 msgr2=0x55c01de89140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.894+0000 7fabbb793740 1 --2- 172.21.15.29:0/1606279291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de81f70 0x55c01de89140 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fabac0045f0 tx=0x7fabac004a40 comp rx=0 tx=0).stop 2024-03-11T19:14:26.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.894+0000 7fabbb793740 1 -- 172.21.15.29:0/1606279291 shutdown_connections 2024-03-11T19:14:26.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.894+0000 7fabbb793740 1 --2- 172.21.15.29:0/1606279291 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55c01de94ee0 0x55c01de97300 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.894+0000 7fabbb793740 1 --2- 172.21.15.29:0/1606279291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de848a0 0x55c01de90180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.894+0000 7fabbb793740 1 --2- 172.21.15.29:0/1606279291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de81f70 0x55c01de89140 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.894+0000 7fabbb793740 1 --2- 172.21.15.29:0/1606279291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de7f0a0 0x55c01de88c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.894+0000 7fabbb793740 1 -- 172.21.15.29:0/1606279291 >> 172.21.15.29:0/1606279291 conn(0x55c01de708e0 msgr2=0x55c01de74ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:26.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.894+0000 7fab99ffb640 0 client.26099 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:14:26.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.894+0000 7fabbb793740 1 -- 172.21.15.29:0/1606279291 shutdown_connections 2024-03-11T19:14:26.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.894+0000 7fabbb793740 1 -- 172.21.15.29:0/1606279291 wait complete. 2024-03-11T19:14:26.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.894+0000 7fabbb793740 20 client.26099 trim_cache size 0 max 16384 2024-03-11T19:14:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.917+0000 7fabbb793740 1 Processor -- start 2024-03-11T19:14:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.917+0000 7fabbb793740 1 -- start start 2024-03-11T19:14:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.917+0000 7fabbb793740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de7f0a0 0x55c01de83140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.917+0000 7fabbb793740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de81f70 0x55c01de83680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.917+0000 7fabbb793740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de848a0 0x55c01de90140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.917+0000 7fabbb793740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c01dd64130 con 0x55c01de848a0 2024-03-11T19:14:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.917+0000 7fabbb793740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c01db1b720 con 0x55c01de7f0a0 2024-03-11T19:14:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.917+0000 7fabbb793740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c01dade7e0 con 0x55c01de81f70 2024-03-11T19:14:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.917+0000 7fabb2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de848a0 0x55c01de90140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.917+0000 7fabb2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de848a0 0x55c01de90140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54630/0 (socket says 172.21.15.29:54630) 2024-03-11T19:14:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.917+0000 7fabb2d9d640 1 -- 172.21.15.29:0/3506389889 learned_addr learned my addr 172.21.15.29:0/3506389889 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.917+0000 7fabb259c640 1 --2- 172.21.15.29:0/3506389889 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de7f0a0 0x55c01de83140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.917+0000 7fabb1d9b640 1 --2- 172.21.15.29:0/3506389889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de81f70 0x55c01de83680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.918+0000 7fabb2d9d640 1 -- 172.21.15.29:0/3506389889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de81f70 msgr2=0x55c01de83680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.918+0000 7fabb2d9d640 1 --2- 172.21.15.29:0/3506389889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de81f70 0x55c01de83680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.918+0000 7fabb2d9d640 1 -- 172.21.15.29:0/3506389889 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de7f0a0 msgr2=0x55c01de83140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.918+0000 7fabb2d9d640 1 --2- 172.21.15.29:0/3506389889 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de7f0a0 0x55c01de83140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.918+0000 7fabb2d9d640 1 -- 172.21.15.29:0/3506389889 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c01dd79c20 con 0x55c01de848a0 2024-03-11T19:14:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.918+0000 7fabb259c640 1 --2- 172.21.15.29:0/3506389889 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de7f0a0 0x55c01de83140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:14:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.918+0000 7fabb2d9d640 1 --2- 172.21.15.29:0/3506389889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de848a0 0x55c01de90140 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7faba4032530 tx=0x7faba404a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa39c00a41f95cb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.918+0000 7fab9bfff640 1 -- 172.21.15.29:0/3506389889 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba4034db0 con 0x55c01de848a0 2024-03-11T19:14:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.918+0000 7fab9bfff640 1 -- 172.21.15.29:0/3506389889 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faba402c270 con 0x55c01de848a0 2024-03-11T19:14:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.918+0000 7fab9bfff640 1 -- 172.21.15.29:0/3506389889 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba4032860 con 0x55c01de848a0 2024-03-11T19:14:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.919+0000 7fabbb793740 1 -- 172.21.15.29:0/3506389889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de848a0 msgr2=0x55c01de90140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.920+0000 7fabbb793740 1 --2- 172.21.15.29:0/3506389889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de848a0 0x55c01de90140 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7faba4032530 tx=0x7faba404a040 comp rx=0 tx=0).stop 2024-03-11T19:14:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.920+0000 7fabbb793740 1 -- 172.21.15.29:0/3506389889 shutdown_connections 2024-03-11T19:14:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.920+0000 7fabbb793740 1 --2- 172.21.15.29:0/3506389889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de848a0 0x55c01de90140 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.920+0000 7fabbb793740 1 --2- 172.21.15.29:0/3506389889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de81f70 0x55c01de83680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.920+0000 7fabbb793740 1 --2- 172.21.15.29:0/3506389889 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de7f0a0 0x55c01de83140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.920+0000 7fabbb793740 1 -- 172.21.15.29:0/3506389889 >> 172.21.15.29:0/3506389889 conn(0x55c01de708e0 msgr2=0x55c01de82350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.920+0000 7fabbb793740 1 -- 172.21.15.29:0/3506389889 shutdown_connections 2024-03-11T19:14:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.920+0000 7fabbb793740 1 -- 172.21.15.29:0/3506389889 wait complete. 2024-03-11T19:14:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.920+0000 7fabbb793740 1 -- start start 2024-03-11T19:14:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.920+0000 7fabbb793740 1 Processor -- start 2024-03-11T19:14:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.921+0000 7fabbb793740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de7f0a0 0x55c01de8df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.921+0000 7fabbb793740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de81f70 0x55c01de8e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.921+0000 7fabbb793740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de848a0 0x55c01de8f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.921+0000 7fabb259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de7f0a0 0x55c01de8df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.921+0000 7fabb259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de7f0a0 0x55c01de8df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54644/0 (socket says 172.21.15.29:54644) 2024-03-11T19:14:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.921+0000 7fabb259c640 1 -- 172.21.15.29:0/1271424290 learned_addr learned my addr 172.21.15.29:0/1271424290 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.921+0000 7fabbb793740 1 -- 172.21.15.29:0/1271424290 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c01de8f990 con 0x55c01de7f0a0 2024-03-11T19:14:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.921+0000 7fabbb793740 1 -- 172.21.15.29:0/1271424290 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c01de8fb00 con 0x55c01de81f70 2024-03-11T19:14:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.921+0000 7fabbb793740 1 -- 172.21.15.29:0/1271424290 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c01de8fcc0 con 0x55c01de848a0 2024-03-11T19:14:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.921+0000 7fabb1d9b640 1 --2- 172.21.15.29:0/1271424290 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de81f70 0x55c01de8e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.921+0000 7fabb2d9d640 1 --2- 172.21.15.29:0/1271424290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de848a0 0x55c01de8f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.921+0000 7fabbb793740 10 client.0 _ll_register_callbacks cb 0x7fabac009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:14:26.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.922+0000 7fabb1d9b640 1 -- 172.21.15.29:0/1271424290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de848a0 msgr2=0x55c01de8f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.922+0000 7fabb1d9b640 1 --2- 172.21.15.29:0/1271424290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de848a0 0x55c01de8f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.922+0000 7fabb1d9b640 1 -- 172.21.15.29:0/1271424290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de7f0a0 msgr2=0x55c01de8df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:26.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.922+0000 7fabb1d9b640 1 --2- 172.21.15.29:0/1271424290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de7f0a0 0x55c01de8df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.922+0000 7fabb1d9b640 1 -- 172.21.15.29:0/1271424290 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c01dda1060 con 0x55c01de81f70 2024-03-11T19:14:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.922+0000 7fabb1d9b640 1 --2- 172.21.15.29:0/1271424290 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de81f70 0x55c01de8e490 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fabac000f80 tx=0x7fabac009f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=830a4dc726c9a27b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.922+0000 7fab99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:14:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.922+0000 7fab99ffb640 1 -- 172.21.15.29:0/1271424290 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac029070 con 0x55c01de81f70 2024-03-11T19:14:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.922+0000 7fab99ffb640 1 -- 172.21.15.29:0/1271424290 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fabac025bc0 con 0x55c01de81f70 2024-03-11T19:14:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.922+0000 7fabbb793740 1 -- 172.21.15.29:0/1271424290 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c01db6a890 con 0x55c01de81f70 2024-03-11T19:14:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.923+0000 7fab99ffb640 1 -- 172.21.15.29:0/1271424290 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabac013070 con 0x55c01de81f70 2024-03-11T19:14:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.923+0000 7fabbb793740 10 client.26105 fetch_fsmap learned FSMap version 27 2024-03-11T19:14:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.923+0000 7fabbb793740 1 -- 172.21.15.29:0/1271424290 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55c01dd9f8c0 con 0x55c01de81f70 2024-03-11T19:14:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.923+0000 7fab99ffb640 1 -- 172.21.15.29:0/1271424290 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fabac02a040 con 0x55c01de81f70 2024-03-11T19:14:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.923+0000 7fab99ffb640 1 -- 172.21.15.29:0/1271424290 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fabac037030 con 0x55c01de81f70 2024-03-11T19:14:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.923+0000 7fabbb793740 10 client.26105 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:14:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.923+0000 7fabbb793740 10 client.26105 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:14:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.923+0000 7fabbb793740 20 client.26105 populate_metadata read hostname 'smithi029' 2024-03-11T19:14:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.923+0000 7fabbb793740 1 --2- 172.21.15.29:0/1271424290 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55c01deec430 0x55c01deee850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.923+0000 7fabbb793740 4 client.26105 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:14:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.923+0000 7fabbb793740 1 -- 172.21.15.29:0/1271424290 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c01db6a890 con 0x55c01deec430 2024-03-11T19:14:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.924+0000 7fabb259c640 1 --2- 172.21.15.29:0/1271424290 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55c01deec430 0x55c01deee850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.924+0000 7fabb259c640 1 --2- 172.21.15.29:0/1271424290 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55c01deec430 0x55c01deee850 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b4e7d02f6a8d42fc server_cookie=7dadd208c6bb1ac3 in_seq=0 out_seq=0 2024-03-11T19:14:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.924+0000 7fab99ffb640 10 client.26105 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:14:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.924+0000 7fab99ffb640 1 -- 172.21.15.29:0/1271424290 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c01db6a890 con 0x55c01deec430 2024-03-11T19:14:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.924+0000 7fab99ffb640 10 client.26105 handle_command_reply: tid=0 2024-03-11T19:14:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.924+0000 7fabbb793740 1 client.26105 shutdown 2024-03-11T19:14:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.924+0000 7fabbb793740 1 -- 172.21.15.29:0/1271424290 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de81f70 msgr2=0x55c01de8e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.924+0000 7fabbb793740 1 --2- 172.21.15.29:0/1271424290 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de81f70 0x55c01de8e490 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fabac000f80 tx=0x7fabac009f10 comp rx=0 tx=0).stop 2024-03-11T19:14:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.924+0000 7fabbb793740 1 -- 172.21.15.29:0/1271424290 shutdown_connections 2024-03-11T19:14:26.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.924+0000 7fabbb793740 1 --2- 172.21.15.29:0/1271424290 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55c01deec430 0x55c01deee850 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.924+0000 7fabbb793740 1 --2- 172.21.15.29:0/1271424290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c01de848a0 0x55c01de8f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.924+0000 7fabbb793740 1 --2- 172.21.15.29:0/1271424290 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c01de81f70 0x55c01de8e490 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.924+0000 7fabbb793740 1 --2- 172.21.15.29:0/1271424290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c01de7f0a0 0x55c01de8df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.924+0000 7fab99ffb640 0 client.26105 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:14:26.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.925+0000 7fabbb793740 1 -- 172.21.15.29:0/1271424290 >> 172.21.15.29:0/1271424290 conn(0x55c01de708e0 msgr2=0x55c01de7de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:26.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.925+0000 7fabbb793740 1 -- 172.21.15.29:0/1271424290 shutdown_connections 2024-03-11T19:14:26.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.925+0000 7fabbb793740 1 -- 172.21.15.29:0/1271424290 wait complete. 2024-03-11T19:14:26.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.925+0000 7fabbb793740 20 client.26105 trim_cache size 0 max 16384 2024-03-11T19:14:26.931 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:14:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.925+0000 7fabb8bbc640 1 -- 172.21.15.29:0/2347994229 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faba00a1a30 msgr2=0x7faba00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.925+0000 7fabb8bbc640 1 --2- 172.21.15.29:0/2347994229 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faba00a1a30 0x7faba00a3ef0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fabb4198660 tx=0x7fabac010040 comp rx=0 tx=0).stop 2024-03-11T19:14:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.925+0000 7fabb8bbc640 1 -- 172.21.15.29:0/2347994229 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fabb4101c00 msgr2=0x7fabb4197150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.925+0000 7fabb8bbc640 1 --2- 172.21.15.29:0/2347994229 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fabb4101c00 0x7fabb4197150 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00e9f0 tx=0x7fab9c00eeb0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.925+0000 7fabb8bbc640 1 -- 172.21.15.29:0/2347994229 shutdown_connections 2024-03-11T19:14:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.925+0000 7fabb8bbc640 1 --2- 172.21.15.29:0/2347994229 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faba00a1a30 0x7faba00a3ef0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.925+0000 7fabb8bbc640 1 --2- 172.21.15.29:0/2347994229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabb4106e60 0x7fabb419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.925+0000 7fabb8bbc640 1 --2- 172.21.15.29:0/2347994229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabb4104530 0x7fabb4197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.925+0000 7fabb8bbc640 1 --2- 172.21.15.29:0/2347994229 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fabb4101c00 0x7fabb4197150 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.926+0000 7fabb8bbc640 1 -- 172.21.15.29:0/2347994229 >> 172.21.15.29:0/2347994229 conn(0x7fabb40fb610 msgr2=0x7fabb4102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.926+0000 7fabb8bbc640 1 -- 172.21.15.29:0/2347994229 shutdown_connections 2024-03-11T19:14:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:26.926+0000 7fabb8bbc640 1 -- 172.21.15.29:0/2347994229 wait complete. 2024-03-11T19:14:26.945 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:14:26.946 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:14:27.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:26 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3266964930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:14:27.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:26 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3266964930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:14:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.157+0000 7f51b0b13640 1 -- 172.21.15.29:0/132415706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51ac1008d0 msgr2=0x7f51ac102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.157+0000 7f51b0b13640 1 --2- 172.21.15.29:0/132415706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51ac1008d0 0x7f51ac102cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f519c009ca0 tx=0x7f519c02c100 comp rx=0 tx=0).stop 2024-03-11T19:14:27.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.157+0000 7f51b0b13640 1 -- 172.21.15.29:0/132415706 shutdown_connections 2024-03-11T19:14:27.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.157+0000 7f51b0b13640 1 --2- 172.21.15.29:0/132415706 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ac105d10 0x7f51ac1081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.157+0000 7f51b0b13640 1 --2- 172.21.15.29:0/132415706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51ac103290 0x7f51ac105680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.157+0000 7f51b0b13640 1 --2- 172.21.15.29:0/132415706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51ac1008d0 0x7f51ac102cc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.157+0000 7f51b0b13640 1 -- 172.21.15.29:0/132415706 >> 172.21.15.29:0/132415706 conn(0x7f51ac0fa940 msgr2=0x7f51ac0fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:27.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.157+0000 7f51b0b13640 1 -- 172.21.15.29:0/132415706 shutdown_connections 2024-03-11T19:14:27.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.157+0000 7f51b0b13640 1 -- 172.21.15.29:0/132415706 wait complete. 2024-03-11T19:14:27.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.158+0000 7f51b0b13640 1 Processor -- start 2024-03-11T19:14:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.158+0000 7f51b0b13640 1 -- start start 2024-03-11T19:14:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.158+0000 7f51b0b13640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ac1008d0 0x7f51ac195020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.158+0000 7f51b0b13640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51ac103290 0x7f51ac195560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.158+0000 7f51b0b13640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51ac105d10 0x7f51ac19c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.159+0000 7f51aa59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ac1008d0 0x7f51ac195020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.159+0000 7f51b0b13640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac19cb10 con 0x7f51ac103290 2024-03-11T19:14:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.159+0000 7f51b0b13640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac19cc80 con 0x7f51ac1008d0 2024-03-11T19:14:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.159+0000 7f51b0b13640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac19ce40 con 0x7f51ac105d10 2024-03-11T19:14:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.159+0000 7f51aa59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ac1008d0 0x7f51ac195020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37672/0 (socket says 172.21.15.29:37672) 2024-03-11T19:14:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.159+0000 7f51aa59c640 1 -- 172.21.15.29:0/1422022586 learned_addr learned my addr 172.21.15.29:0/1422022586 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.159+0000 7f51aad9d640 1 --2- 172.21.15.29:0/1422022586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51ac105d10 0x7f51ac19c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.159+0000 7f51a1d9b640 1 --2- 172.21.15.29:0/1422022586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51ac103290 0x7f51ac195560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.159+0000 7f51aa59c640 1 -- 172.21.15.29:0/1422022586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51ac105d10 msgr2=0x7f51ac19c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:27.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.159+0000 7f51aa59c640 1 --2- 172.21.15.29:0/1422022586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51ac105d10 0x7f51ac19c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.159+0000 7f51aa59c640 1 -- 172.21.15.29:0/1422022586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51ac103290 msgr2=0x7f51ac195560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:27.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.159+0000 7f51aa59c640 1 --2- 172.21.15.29:0/1422022586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51ac103290 0x7f51ac195560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.159+0000 7f51aa59c640 1 -- 172.21.15.29:0/1422022586 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519c009900 con 0x7f51ac1008d0 2024-03-11T19:14:27.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.159+0000 7f51a1d9b640 1 --2- 172.21.15.29:0/1422022586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51ac103290 0x7f51ac195560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:14:27.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.159+0000 7f51aa59c640 1 --2- 172.21.15.29:0/1422022586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ac1008d0 0x7f51ac195020 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f519c009bf0 tx=0x7f519c032920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:27.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.160+0000 7f51a3fff640 1 -- 172.21.15.29:0/1422022586 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519c034850 con 0x7f51ac1008d0 2024-03-11T19:14:27.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.160+0000 7f51a3fff640 1 -- 172.21.15.29:0/1422022586 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f519c032a90 con 0x7f51ac1008d0 2024-03-11T19:14:27.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.160+0000 7f51a3fff640 1 -- 172.21.15.29:0/1422022586 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519c004720 con 0x7f51ac1008d0 2024-03-11T19:14:27.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.161+0000 7f51b0b13640 1 -- 172.21.15.29:0/1422022586 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51ac19d0c0 con 0x7f51ac1008d0 2024-03-11T19:14:27.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.161+0000 7f51b0b13640 1 -- 172.21.15.29:0/1422022586 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51ac19d520 con 0x7f51ac1008d0 2024-03-11T19:14:27.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.164+0000 7f51a3fff640 1 -- 172.21.15.29:0/1422022586 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f519c03b040 con 0x7f51ac1008d0 2024-03-11T19:14:27.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.165+0000 7f51b37c7740 1 Processor -- start 2024-03-11T19:14:27.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.165+0000 7f51a3fff640 1 --2- 172.21.15.29:0/1422022586 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51980a1b00 0x7f51980a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.166+0000 7f51a3fff640 1 -- 172.21.15.29:0/1422022586 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f519c0ef7a0 con 0x7f51ac1008d0 2024-03-11T19:14:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.166+0000 7f51b37c7740 1 -- start start 2024-03-11T19:14:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.166+0000 7f51b37c7740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32070 0x557ca4b32490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.166+0000 7f51b37c7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32a60 0x557ca4b36ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.166+0000 7f51b37c7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b37660 0x557ca4b39ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.166+0000 7f51b37c7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca4ae5960 con 0x557ca4b37660 2024-03-11T19:14:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.166+0000 7f51b37c7740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca4a99310 con 0x557ca4b32070 2024-03-11T19:14:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.166+0000 7f51b37c7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca4a96840 con 0x557ca4b32a60 2024-03-11T19:14:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.166+0000 7f51a1d9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32a60 0x557ca4b36ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.166+0000 7f51aad9d640 1 --2- 172.21.15.29:0/1422022586 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51980a1b00 0x7f51980a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.166+0000 7f51a1d9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32a60 0x557ca4b36ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41544/0 (socket says 172.21.15.29:41544) 2024-03-11T19:14:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.166+0000 7f51a1d9b640 1 -- 172.21.15.29:0/3019488515 learned_addr learned my addr 172.21.15.29:0/3019488515 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:27.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.166+0000 7f51aa59c640 1 --2- 172.21.15.29:0/3019488515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32070 0x557ca4b32490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.167+0000 7f51a1d9b640 1 -- 172.21.15.29:0/3019488515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32070 msgr2=0x557ca4b32490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:27.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.167+0000 7f51a1d9b640 1 --2- 172.21.15.29:0/3019488515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32070 0x557ca4b32490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.167+0000 7f51a1d9b640 1 -- 172.21.15.29:0/3019488515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b37660 msgr2=0x557ca4b39ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:14:27.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.167+0000 7f51a1d9b640 1 --2- 172.21.15.29:0/3019488515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b37660 0x557ca4b39ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.167+0000 7f51a1d9b640 1 -- 172.21.15.29:0/3019488515 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ca4a54060 con 0x557ca4b32a60 2024-03-11T19:14:27.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.167+0000 7f51aad9d640 1 --2- 172.21.15.29:0/1422022586 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51980a1b00 0x7f51980a3fc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f51a4009bb0 tx=0x7f51a4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:27.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.167+0000 7f51a1d9b640 1 --2- 172.21.15.29:0/3019488515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32a60 0x557ca4b36ff0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f519400dbf0 tx=0x7f5194005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20170ebd13359dd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:27.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.168+0000 7f51a0d99640 1 -- 172.21.15.29:0/3019488515 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5194012040 con 0x557ca4b32a60 2024-03-11T19:14:27.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.168+0000 7f51a0d99640 1 -- 172.21.15.29:0/3019488515 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f51940044e0 con 0x557ca4b32a60 2024-03-11T19:14:27.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.168+0000 7f51a0d99640 1 -- 172.21.15.29:0/3019488515 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5194016420 con 0x557ca4b32a60 2024-03-11T19:14:27.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.169+0000 7f51b37c7740 1 -- 172.21.15.29:0/3019488515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32a60 msgr2=0x557ca4b36ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:27.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.169+0000 7f51b37c7740 1 --2- 172.21.15.29:0/3019488515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32a60 0x557ca4b36ff0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f519400dbf0 tx=0x7f5194005f90 comp rx=0 tx=0).stop 2024-03-11T19:14:27.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.169+0000 7f51b37c7740 1 -- 172.21.15.29:0/3019488515 shutdown_connections 2024-03-11T19:14:27.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.169+0000 7f51b37c7740 1 --2- 172.21.15.29:0/3019488515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b37660 0x557ca4b39ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.169+0000 7f51b37c7740 1 --2- 172.21.15.29:0/3019488515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32a60 0x557ca4b36ff0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.169+0000 7f51b37c7740 1 --2- 172.21.15.29:0/3019488515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32070 0x557ca4b32490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.169+0000 7f51b37c7740 1 -- 172.21.15.29:0/3019488515 >> 172.21.15.29:0/3019488515 conn(0x557ca4b24760 msgr2=0x557ca4b24b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:27.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.169+0000 7f51b37c7740 1 -- 172.21.15.29:0/3019488515 shutdown_connections 2024-03-11T19:14:27.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.169+0000 7f51b37c7740 1 -- 172.21.15.29:0/3019488515 wait complete. 2024-03-11T19:14:27.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.169+0000 7f51b37c7740 1 -- start start 2024-03-11T19:14:27.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.169+0000 7f51b37c7740 1 Processor -- start 2024-03-11T19:14:27.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.170+0000 7f51b37c7740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32070 0x557ca4b3bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.170+0000 7f51b37c7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32a60 0x557ca4b3c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.170+0000 7f51b37c7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b37660 0x557ca4b43210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.170+0000 7f51b37c7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca4a92020 con 0x557ca4b37660 2024-03-11T19:14:27.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.170+0000 7f51b37c7740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca4765430 con 0x557ca4b32070 2024-03-11T19:14:27.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.170+0000 7f51b37c7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca4721e00 con 0x557ca4b32a60 2024-03-11T19:14:27.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.170+0000 7f51b37c7740 10 client.0 _ll_register_callbacks cb 0x557ca4a39cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:14:27.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.170+0000 7f51a1d9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32a60 0x557ca4b3c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.170+0000 7f51aa59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32070 0x557ca4b3bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.170+0000 7f51a1d9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32a60 0x557ca4b3c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41550/0 (socket says 172.21.15.29:41550) 2024-03-11T19:14:27.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.170+0000 7f51a1d9b640 1 -- 172.21.15.29:0/4142751254 learned_addr learned my addr 172.21.15.29:0/4142751254 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:27.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.170+0000 7f51aad9d640 1 --2- 172.21.15.29:0/4142751254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b37660 0x557ca4b43210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.170+0000 7f51a1d9b640 1 -- 172.21.15.29:0/4142751254 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32070 msgr2=0x557ca4b3bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:27.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.170+0000 7f51a1d9b640 1 --2- 172.21.15.29:0/4142751254 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32070 0x557ca4b3bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.170+0000 7f51a1d9b640 1 -- 172.21.15.29:0/4142751254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b37660 msgr2=0x557ca4b43210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:27.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.170+0000 7f51a1d9b640 1 --2- 172.21.15.29:0/4142751254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b37660 0x557ca4b43210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.170+0000 7f51a1d9b640 1 -- 172.21.15.29:0/4142751254 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f519400d850 con 0x557ca4b32a60 2024-03-11T19:14:27.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.171+0000 7f51a1d9b640 1 --2- 172.21.15.29:0/4142751254 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32a60 0x557ca4b3c1d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5194003920 tx=0x7f5194009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c0c1392c9844d1a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:27.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.171+0000 7f5191ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:14:27.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.171+0000 7f5191ffb640 1 -- 172.21.15.29:0/4142751254 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5194004150 con 0x557ca4b32a60 2024-03-11T19:14:27.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.171+0000 7f5191ffb640 1 -- 172.21.15.29:0/4142751254 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f51940042b0 con 0x557ca4b32a60 2024-03-11T19:14:27.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.171+0000 7f51b37c7740 1 -- 172.21.15.29:0/4142751254 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557ca47ce720 con 0x557ca4b32a60 2024-03-11T19:14:27.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.171+0000 7f5191ffb640 1 -- 172.21.15.29:0/4142751254 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5194016420 con 0x557ca4b32a60 2024-03-11T19:14:27.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.172+0000 7f51b37c7740 10 client.26125 fetch_fsmap learned FSMap version 27 2024-03-11T19:14:27.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.172+0000 7f51b37c7740 1 -- 172.21.15.29:0/4142751254 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x557ca4a906a0 con 0x557ca4b32a60 2024-03-11T19:14:27.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.172+0000 7f5191ffb640 1 -- 172.21.15.29:0/4142751254 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x557ca47ce720 con 0x557ca4b32a60 2024-03-11T19:14:27.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.172+0000 7f5191ffb640 1 -- 172.21.15.29:0/4142751254 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5194027020 con 0x557ca4b32a60 2024-03-11T19:14:27.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.172+0000 7f51b37c7740 10 client.26125 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:14:27.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.172+0000 7f51b37c7740 10 client.26125 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:14:27.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.172+0000 7f51b37c7740 20 client.26125 populate_metadata read hostname 'smithi029' 2024-03-11T19:14:27.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.172+0000 7f51b37c7740 1 --2- 172.21.15.29:0/4142751254 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557ca4b47cd0 0x557ca4b4a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.172+0000 7f51b37c7740 4 client.26125 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:14:27.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.172+0000 7f51b37c7740 1 -- 172.21.15.29:0/4142751254 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557ca481d890 con 0x557ca4b47cd0 2024-03-11T19:14:27.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.173+0000 7f51aa59c640 1 --2- 172.21.15.29:0/4142751254 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557ca4b47cd0 0x557ca4b4a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.173+0000 7f5191ffb640 10 client.26125 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:14:27.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.173+0000 7f51aa59c640 1 --2- 172.21.15.29:0/4142751254 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557ca4b47cd0 0x557ca4b4a0f0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=433951b743638f73 server_cookie=7c0cdd4d70f86864 in_seq=0 out_seq=0 2024-03-11T19:14:27.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.174+0000 7f5191ffb640 1 -- 172.21.15.29:0/4142751254 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557ca481d890 con 0x557ca4b47cd0 2024-03-11T19:14:27.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.174+0000 7f5191ffb640 10 client.26125 handle_command_reply: tid=0 2024-03-11T19:14:27.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.174+0000 7f51b37c7740 1 client.26125 shutdown 2024-03-11T19:14:27.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.176+0000 7f51b37c7740 1 -- 172.21.15.29:0/4142751254 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32a60 msgr2=0x557ca4b3c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:27.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.176+0000 7f51b37c7740 1 --2- 172.21.15.29:0/4142751254 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32a60 0x557ca4b3c1d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5194003920 tx=0x7f5194009f70 comp rx=0 tx=0).stop 2024-03-11T19:14:27.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.178+0000 7f51b37c7740 1 -- 172.21.15.29:0/4142751254 shutdown_connections 2024-03-11T19:14:27.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.178+0000 7f51b37c7740 1 --2- 172.21.15.29:0/4142751254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b37660 0x557ca4b43210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.178+0000 7f51b37c7740 1 --2- 172.21.15.29:0/4142751254 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32a60 0x557ca4b3c1d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.178+0000 7f51b37c7740 1 --2- 172.21.15.29:0/4142751254 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557ca4b47cd0 0x557ca4b4a0f0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.178+0000 7f51b37c7740 1 --2- 172.21.15.29:0/4142751254 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32070 0x557ca4b3bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.178+0000 7f51b37c7740 1 -- 172.21.15.29:0/4142751254 >> 172.21.15.29:0/4142751254 conn(0x557ca4b24760 msgr2=0x557ca4b30e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:27.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.178+0000 7f5191ffb640 0 client.26125 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:14:27.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.178+0000 7f51b37c7740 1 -- 172.21.15.29:0/4142751254 shutdown_connections 2024-03-11T19:14:27.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.178+0000 7f51b37c7740 1 -- 172.21.15.29:0/4142751254 wait complete. 2024-03-11T19:14:27.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.178+0000 7f51b37c7740 20 client.26125 trim_cache size 0 max 16384 2024-03-11T19:14:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.199+0000 7f51b37c7740 1 Processor -- start 2024-03-11T19:14:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.199+0000 7f51b37c7740 1 -- start start 2024-03-11T19:14:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.199+0000 7f51b37c7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b32070 0x557ca4b1fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.199+0000 7f51b37c7740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32a60 0x557ca4b431d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.199+0000 7f51b37c7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b37660 0x557ca4b3bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.199+0000 7f51b37c7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca4a17130 con 0x557ca4b32070 2024-03-11T19:14:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.199+0000 7f51b37c7740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca4b16a70 con 0x557ca4b32a60 2024-03-11T19:14:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.199+0000 7f51b37c7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca4b254d0 con 0x557ca4b37660 2024-03-11T19:14:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.200+0000 7f51aa59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b32070 0x557ca4b1fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.200+0000 7f51a1d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32a60 0x557ca4b431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.200+0000 7f51aa59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b32070 0x557ca4b1fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54680/0 (socket says 172.21.15.29:54680) 2024-03-11T19:14:27.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.200+0000 7f51aa59c640 1 -- 172.21.15.29:0/2067732087 learned_addr learned my addr 172.21.15.29:0/2067732087 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.200+0000 7f51aad9d640 1 --2- 172.21.15.29:0/2067732087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b37660 0x557ca4b3bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.200+0000 7f51a1d9b640 1 -- 172.21.15.29:0/2067732087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b37660 msgr2=0x557ca4b3bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.200+0000 7f51a1d9b640 1 --2- 172.21.15.29:0/2067732087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b37660 0x557ca4b3bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.200+0000 7f51a1d9b640 1 -- 172.21.15.29:0/2067732087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b32070 msgr2=0x557ca4b1fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:27.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.200+0000 7f51a1d9b640 1 --2- 172.21.15.29:0/2067732087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b32070 0x557ca4b1fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.200+0000 7f51a1d9b640 1 -- 172.21.15.29:0/2067732087 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ca4a2cc20 con 0x557ca4b32a60 2024-03-11T19:14:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.200+0000 7f51aad9d640 1 --2- 172.21.15.29:0/2067732087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b37660 0x557ca4b3bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:14:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.200+0000 7f51aa59c640 1 --2- 172.21.15.29:0/2067732087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b32070 0x557ca4b1fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:14:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.200+0000 7f51a1d9b640 1 --2- 172.21.15.29:0/2067732087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32a60 0x557ca4b431d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f51940059d0 tx=0x7f5194003800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffcf761e9b6dda7f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.201+0000 7f5193fff640 1 -- 172.21.15.29:0/2067732087 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5194021ab0 con 0x557ca4b32a60 2024-03-11T19:14:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.201+0000 7f5193fff640 1 -- 172.21.15.29:0/2067732087 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f519402f030 con 0x557ca4b32a60 2024-03-11T19:14:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.201+0000 7f5193fff640 1 -- 172.21.15.29:0/2067732087 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51940053e0 con 0x557ca4b32a60 2024-03-11T19:14:27.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.202+0000 7f51b37c7740 1 -- 172.21.15.29:0/2067732087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32a60 msgr2=0x557ca4b431d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:27.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.202+0000 7f51b37c7740 1 --2- 172.21.15.29:0/2067732087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32a60 0x557ca4b431d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f51940059d0 tx=0x7f5194003800 comp rx=0 tx=0).stop 2024-03-11T19:14:27.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.202+0000 7f51b37c7740 1 -- 172.21.15.29:0/2067732087 shutdown_connections 2024-03-11T19:14:27.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.202+0000 7f51b37c7740 1 --2- 172.21.15.29:0/2067732087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b37660 0x557ca4b3bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.202+0000 7f51b37c7740 1 --2- 172.21.15.29:0/2067732087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32a60 0x557ca4b431d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.202+0000 7f51b37c7740 1 --2- 172.21.15.29:0/2067732087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b32070 0x557ca4b1fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.202+0000 7f51b37c7740 1 -- 172.21.15.29:0/2067732087 >> 172.21.15.29:0/2067732087 conn(0x557ca4b24760 msgr2=0x557ca4b4ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:27.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.202+0000 7f51b37c7740 1 -- 172.21.15.29:0/2067732087 shutdown_connections 2024-03-11T19:14:27.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.202+0000 7f51b37c7740 1 -- 172.21.15.29:0/2067732087 wait complete. 2024-03-11T19:14:27.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.202+0000 7f51b37c7740 1 -- start start 2024-03-11T19:14:27.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.202+0000 7f51b37c7740 1 Processor -- start 2024-03-11T19:14:27.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.202+0000 7f51b37c7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32070 0x557ca4b41960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.202+0000 7f51b37c7740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32a60 0x557ca4b41ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.203+0000 7f51b37c7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b37660 0x557ca4b423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.203+0000 7f51aa59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32070 0x557ca4b41960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.203+0000 7f51aa59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32070 0x557ca4b41960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41570/0 (socket says 172.21.15.29:41570) 2024-03-11T19:14:27.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.203+0000 7f51a1d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32a60 0x557ca4b41ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.203+0000 7f51b37c7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca4b435b0 con 0x557ca4b37660 2024-03-11T19:14:27.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.203+0000 7f51b37c7740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca4b9e1c0 con 0x557ca4b32a60 2024-03-11T19:14:27.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.203+0000 7f51b37c7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca4b9e300 con 0x557ca4b32070 2024-03-11T19:14:27.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.203+0000 7f51a1d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32a60 0x557ca4b41ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37700/0 (socket says 172.21.15.29:37700) 2024-03-11T19:14:27.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.203+0000 7f51a1d9b640 1 -- 172.21.15.29:0/2646062711 learned_addr learned my addr 172.21.15.29:0/2646062711 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:27.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.203+0000 7f51b37c7740 10 client.0 _ll_register_callbacks cb 0x7f519400d630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:14:27.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.203+0000 7f51aad9d640 1 --2- 172.21.15.29:0/2646062711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b37660 0x557ca4b423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.203+0000 7f51a1d9b640 1 -- 172.21.15.29:0/2646062711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32070 msgr2=0x557ca4b41960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:27.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.203+0000 7f51a1d9b640 1 --2- 172.21.15.29:0/2646062711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32070 0x557ca4b41960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.203+0000 7f51a1d9b640 1 -- 172.21.15.29:0/2646062711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b37660 msgr2=0x557ca4b423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:27.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.203+0000 7f51a1d9b640 1 --2- 172.21.15.29:0/2646062711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b37660 0x557ca4b423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.203+0000 7f51a1d9b640 1 -- 172.21.15.29:0/2646062711 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51ac19d520 con 0x557ca4b32a60 2024-03-11T19:14:27.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.203+0000 7f51aad9d640 1 --2- 172.21.15.29:0/2646062711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b37660 0x557ca4b423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:14:27.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.203+0000 7f51a1d9b640 1 --2- 172.21.15.29:0/2646062711 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32a60 0x557ca4b41ea0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f51940059d0 tx=0x7f5194003b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41758fe4149e8da6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:27.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.203+0000 7f5191ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:14:27.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.204+0000 7f5191ffb640 1 -- 172.21.15.29:0/2646062711 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5194004360 con 0x557ca4b32a60 2024-03-11T19:14:27.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.204+0000 7f5191ffb640 1 -- 172.21.15.29:0/2646062711 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5194005120 con 0x557ca4b32a60 2024-03-11T19:14:27.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.204+0000 7f51b37c7740 1 -- 172.21.15.29:0/2646062711 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557ca47917e0 con 0x557ca4b32a60 2024-03-11T19:14:27.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.204+0000 7f5191ffb640 1 -- 172.21.15.29:0/2646062711 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51940052e0 con 0x557ca4b32a60 2024-03-11T19:14:27.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.204+0000 7f51b37c7740 10 client.26123 fetch_fsmap learned FSMap version 27 2024-03-11T19:14:27.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.204+0000 7f51b37c7740 1 -- 172.21.15.29:0/2646062711 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7f51a4011d40 con 0x557ca4b32a60 2024-03-11T19:14:27.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.204+0000 7f5191ffb640 1 -- 172.21.15.29:0/2646062711 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f519404a020 con 0x557ca4b32a60 2024-03-11T19:14:27.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.204+0000 7f5191ffb640 1 -- 172.21.15.29:0/2646062711 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f519403a070 con 0x557ca4b32a60 2024-03-11T19:14:27.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.204+0000 7f51b37c7740 10 client.26123 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:14:27.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.204+0000 7f51b37c7740 10 client.26123 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:14:27.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.204+0000 7f51b37c7740 20 client.26123 populate_metadata read hostname 'smithi029' 2024-03-11T19:14:27.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.204+0000 7f51b37c7740 1 --2- 172.21.15.29:0/2646062711 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557ca4b9f940 0x557ca4ba1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.204+0000 7f51b37c7740 4 client.26123 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:14:27.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.204+0000 7f51b37c7740 1 -- 172.21.15.29:0/2646062711 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557ca473d700 con 0x557ca4b9f940 2024-03-11T19:14:27.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.205+0000 7f51aa59c640 1 --2- 172.21.15.29:0/2646062711 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557ca4b9f940 0x557ca4ba1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.205+0000 7f5191ffb640 10 client.26123 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:14:27.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.205+0000 7f51aa59c640 1 --2- 172.21.15.29:0/2646062711 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557ca4b9f940 0x557ca4ba1d60 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=844694f2daa178d2 server_cookie=1d8500b397ceaff7 in_seq=0 out_seq=0 2024-03-11T19:14:27.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.205+0000 7f5191ffb640 1 -- 172.21.15.29:0/2646062711 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557ca473d700 con 0x557ca4b9f940 2024-03-11T19:14:27.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.206+0000 7f5191ffb640 10 client.26123 handle_command_reply: tid=0 2024-03-11T19:14:27.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.206+0000 7f51b37c7740 1 client.26123 shutdown 2024-03-11T19:14:27.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.206+0000 7f51b37c7740 1 -- 172.21.15.29:0/2646062711 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32a60 msgr2=0x557ca4b41ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:27.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.206+0000 7f51b37c7740 1 --2- 172.21.15.29:0/2646062711 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32a60 0x557ca4b41ea0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f51940059d0 tx=0x7f5194003b80 comp rx=0 tx=0).stop 2024-03-11T19:14:27.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.206+0000 7f51b37c7740 1 -- 172.21.15.29:0/2646062711 shutdown_connections 2024-03-11T19:14:27.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.206+0000 7f51b37c7740 1 --2- 172.21.15.29:0/2646062711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ca4b37660 0x557ca4b423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.206+0000 7f51b37c7740 1 --2- 172.21.15.29:0/2646062711 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557ca4b9f940 0x557ca4ba1d60 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.206+0000 7f51b37c7740 1 --2- 172.21.15.29:0/2646062711 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557ca4b32a60 0x557ca4b41ea0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.206+0000 7f51b37c7740 1 --2- 172.21.15.29:0/2646062711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557ca4b32070 0x557ca4b41960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.206+0000 7f51b37c7740 1 -- 172.21.15.29:0/2646062711 >> 172.21.15.29:0/2646062711 conn(0x557ca4b24760 msgr2=0x557ca4b33940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:27.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.206+0000 7f5191ffb640 0 client.26123 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:14:27.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.206+0000 7f51b37c7740 1 -- 172.21.15.29:0/2646062711 shutdown_connections 2024-03-11T19:14:27.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.206+0000 7f51b37c7740 1 -- 172.21.15.29:0/2646062711 wait complete. 2024-03-11T19:14:27.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.207+0000 7f51b37c7740 20 client.26123 trim_cache size 0 max 16384 2024-03-11T19:14:27.215 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:14:27.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.208+0000 7f51b0b13640 1 -- 172.21.15.29:0/1422022586 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51980a1b00 msgr2=0x7f51980a3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:27.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.208+0000 7f51b0b13640 1 --2- 172.21.15.29:0/1422022586 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51980a1b00 0x7f51980a3fc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f51a4009bb0 tx=0x7f51a4005f90 comp rx=0 tx=0).stop 2024-03-11T19:14:27.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.208+0000 7f51b0b13640 1 -- 172.21.15.29:0/1422022586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ac1008d0 msgr2=0x7f51ac195020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:27.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.208+0000 7f51b0b13640 1 --2- 172.21.15.29:0/1422022586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ac1008d0 0x7f51ac195020 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f519c009bf0 tx=0x7f519c032920 comp rx=0 tx=0).stop 2024-03-11T19:14:27.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.209+0000 7f51b0b13640 1 -- 172.21.15.29:0/1422022586 shutdown_connections 2024-03-11T19:14:27.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.209+0000 7f51b0b13640 1 --2- 172.21.15.29:0/1422022586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51ac105d10 0x7f51ac19c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.209+0000 7f51b0b13640 1 --2- 172.21.15.29:0/1422022586 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f51980a1b00 0x7f51980a3fc0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.209+0000 7f51b0b13640 1 --2- 172.21.15.29:0/1422022586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51ac103290 0x7f51ac195560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.209+0000 7f51b0b13640 1 --2- 172.21.15.29:0/1422022586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ac1008d0 0x7f51ac195020 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.209+0000 7f51b0b13640 1 -- 172.21.15.29:0/1422022586 >> 172.21.15.29:0/1422022586 conn(0x7f51ac0fa940 msgr2=0x7f51ac102790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:27.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.209+0000 7f51b0b13640 1 -- 172.21.15.29:0/1422022586 shutdown_connections 2024-03-11T19:14:27.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.209+0000 7f51b0b13640 1 -- 172.21.15.29:0/1422022586 wait complete. 2024-03-11T19:14:27.233 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:14:27.233 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:14:27.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.403+0000 7f09f75f4640 1 -- 172.21.15.29:0/2349938536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09f01077a0 msgr2=0x7f09f0107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:27.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.403+0000 7f09f75f4640 1 --2- 172.21.15.29:0/2349938536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09f01077a0 0x7f09f0107bc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009900 tx=0x7f09ec02bc30 comp rx=0 tx=0).stop 2024-03-11T19:14:27.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.403+0000 7f09f75f4640 1 -- 172.21.15.29:0/2349938536 shutdown_connections 2024-03-11T19:14:27.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.403+0000 7f09f75f4640 1 --2- 172.21.15.29:0/2349938536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09f0109350 0x7f09f010fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.403+0000 7f09f75f4640 1 --2- 172.21.15.29:0/2349938536 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09f0108990 0x7f09f0108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.403+0000 7f09f75f4640 1 --2- 172.21.15.29:0/2349938536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09f01077a0 0x7f09f0107bc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.403+0000 7f09f75f4640 1 -- 172.21.15.29:0/2349938536 >> 172.21.15.29:0/2349938536 conn(0x7f09f0076730 msgr2=0x7f09f0078b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:27.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.403+0000 7f09f75f4640 1 -- 172.21.15.29:0/2349938536 shutdown_connections 2024-03-11T19:14:27.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.403+0000 7f09f75f4640 1 -- 172.21.15.29:0/2349938536 wait complete. 2024-03-11T19:14:27.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.404+0000 7f09f75f4640 1 Processor -- start 2024-03-11T19:14:27.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.404+0000 7f09f75f4640 1 -- start start 2024-03-11T19:14:27.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.404+0000 7f09f75f4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09f01077a0 0x7f09f019d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.405+0000 7f09f75f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09f0108990 0x7f09f019db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.405+0000 7f09f75f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09f0109350 0x7f09f01a4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.405+0000 7f09f75f4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f01a50d0 con 0x7f09f0108990 2024-03-11T19:14:27.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.405+0000 7f09f75f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f01a5240 con 0x7f09f0109350 2024-03-11T19:14:27.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.405+0000 7f09f75f4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f01a5400 con 0x7f09f01077a0 2024-03-11T19:14:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.405+0000 7f09f5b91640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09f0109350 0x7f09f01a4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.405+0000 7f09f5390640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09f01077a0 0x7f09f019d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.405+0000 7f09f5b91640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09f0109350 0x7f09f01a4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37712/0 (socket says 172.21.15.29:37712) 2024-03-11T19:14:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.405+0000 7f09f5390640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09f01077a0 0x7f09f019d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41594/0 (socket says 172.21.15.29:41594) 2024-03-11T19:14:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.405+0000 7f09f5b91640 1 -- 172.21.15.29:0/641506751 learned_addr learned my addr 172.21.15.29:0/641506751 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.405+0000 7f09f4b8f640 1 --2- 172.21.15.29:0/641506751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09f0108990 0x7f09f019db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.406+0000 7f09f5390640 1 -- 172.21.15.29:0/641506751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09f0109350 msgr2=0x7f09f01a4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.406+0000 7f09f5390640 1 --2- 172.21.15.29:0/641506751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09f0109350 0x7f09f01a4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.406+0000 7f09f5390640 1 -- 172.21.15.29:0/641506751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09f0108990 msgr2=0x7f09f019db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.406+0000 7f09f5390640 1 --2- 172.21.15.29:0/641506751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09f0108990 0x7f09f019db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.406+0000 7f09f5390640 1 -- 172.21.15.29:0/641506751 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ec009560 con 0x7f09f01077a0 2024-03-11T19:14:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.406+0000 7f09f4b8f640 1 --2- 172.21.15.29:0/641506751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09f0108990 0x7f09f019db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:14:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.406+0000 7f09f5390640 1 --2- 172.21.15.29:0/641506751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09f01077a0 0x7f09f019d5e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009230 tx=0x7f09ec0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.406+0000 7f09e67fc640 1 -- 172.21.15.29:0/641506751 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec0044e0 con 0x7f09f01077a0 2024-03-11T19:14:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.407+0000 7f09e67fc640 1 -- 172.21.15.29:0/641506751 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f09ec003cb0 con 0x7f09f01077a0 2024-03-11T19:14:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.407+0000 7f09f75f4640 1 -- 172.21.15.29:0/641506751 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f01a5680 con 0x7f09f01077a0 2024-03-11T19:14:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.407+0000 7f09f75f4640 1 -- 172.21.15.29:0/641506751 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f01a5bb0 con 0x7f09f01077a0 2024-03-11T19:14:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.407+0000 7f09e67fc640 1 -- 172.21.15.29:0/641506751 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec03e450 con 0x7f09f01077a0 2024-03-11T19:14:27.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.410+0000 7f09e67fc640 1 -- 172.21.15.29:0/641506751 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f09ec004ee0 con 0x7f09f01077a0 2024-03-11T19:14:27.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.410+0000 7f09fa1c8740 1 Processor -- start 2024-03-11T19:14:27.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.410+0000 7f09fa1c8740 1 -- start start 2024-03-11T19:14:27.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.411+0000 7f09fa1c8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161a2270 0x55e3161a2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.411+0000 7f09fa1c8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161a2c60 0x55e3161a7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.411+0000 7f09fa1c8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161a7dd0 0x55e3161aa260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.411+0000 7f09fa1c8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31615e960 con 0x55e3161a7dd0 2024-03-11T19:14:27.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.411+0000 7f09fa1c8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e316112310 con 0x55e3161a2270 2024-03-11T19:14:27.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.411+0000 7f09fa1c8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31610f840 con 0x55e3161a2c60 2024-03-11T19:14:27.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.411+0000 7f09e67fc640 1 --2- 172.21.15.29:0/641506751 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f09dc0a1b30 0x7f09dc0a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.411+0000 7f09f5b91640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161a2270 0x55e3161a2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.411+0000 7f09f5b91640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161a2270 0x55e3161a2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37714/0 (socket says 172.21.15.29:37714) 2024-03-11T19:14:27.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.411+0000 7f09f5b91640 1 -- 172.21.15.29:0/4110837487 learned_addr learned my addr 172.21.15.29:0/4110837487 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:27.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.411+0000 7f09f5390640 1 --2- 172.21.15.29:0/4110837487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161a2c60 0x55e3161a7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.411+0000 7f09f4b8f640 1 --2- 172.21.15.29:0/4110837487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161a7dd0 0x55e3161aa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.411+0000 7f09f5b91640 1 --2- 172.21.15.29:0/641506751 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f09dc0a1b30 0x7f09dc0a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.411+0000 7f09f5390640 1 -- 172.21.15.29:0/4110837487 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161a2270 msgr2=0x55e3161a2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:27.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.411+0000 7f09f5390640 1 --2- 172.21.15.29:0/4110837487 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161a2270 0x55e3161a2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.411+0000 7f09f5390640 1 -- 172.21.15.29:0/4110837487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161a7dd0 msgr2=0x55e3161aa260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:27.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.411+0000 7f09f5390640 1 --2- 172.21.15.29:0/4110837487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161a7dd0 0x55e3161aa260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.411+0000 7f09f5390640 1 -- 172.21.15.29:0/4110837487 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3160cd060 con 0x55e3161a2c60 2024-03-11T19:14:27.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.412+0000 7f09f5390640 1 --2- 172.21.15.29:0/4110837487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161a2c60 0x55e3161a7760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f09ec0edd10 tx=0x7f09ec088210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4639cf6fe2180ce server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:27.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.412+0000 7f09e67fc640 1 -- 172.21.15.29:0/641506751 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f09ec02de40 con 0x7f09f01077a0 2024-03-11T19:14:27.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.412+0000 7f09db7fe640 1 -- 172.21.15.29:0/4110837487 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec0f3050 con 0x55e3161a2c60 2024-03-11T19:14:27.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.412+0000 7f09db7fe640 1 -- 172.21.15.29:0/4110837487 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f09ec09f330 con 0x55e3161a2c60 2024-03-11T19:14:27.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.413+0000 7f09db7fe640 1 -- 172.21.15.29:0/4110837487 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec088720 con 0x55e3161a2c60 2024-03-11T19:14:27.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.413+0000 7f09f5b91640 1 --2- 172.21.15.29:0/641506751 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f09dc0a1b30 0x7f09dc0a3ff0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f09e8009fd0 tx=0x7f09e8015040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:27.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.413+0000 7f09fa1c8740 1 -- 172.21.15.29:0/4110837487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161a2c60 msgr2=0x55e3161a7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:27.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.413+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/4110837487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161a2c60 0x55e3161a7760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f09ec0edd10 tx=0x7f09ec088210 comp rx=0 tx=0).stop 2024-03-11T19:14:27.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.413+0000 7f09fa1c8740 1 -- 172.21.15.29:0/4110837487 shutdown_connections 2024-03-11T19:14:27.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.413+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/4110837487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161a7dd0 0x55e3161aa260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.413+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/4110837487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161a2c60 0x55e3161a7760 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f09ec0edd10 tx=0x7f09ec088210 comp rx=0 tx=0).stop 2024-03-11T19:14:27.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.413+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/4110837487 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161a2270 0x55e3161a2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.413+0000 7f09fa1c8740 1 -- 172.21.15.29:0/4110837487 >> 172.21.15.29:0/4110837487 conn(0x55e31619d7e0 msgr2=0x55e31619dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:27.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.414+0000 7f09fa1c8740 1 -- 172.21.15.29:0/4110837487 shutdown_connections 2024-03-11T19:14:27.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.414+0000 7f09fa1c8740 1 -- 172.21.15.29:0/4110837487 wait complete. 2024-03-11T19:14:27.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.414+0000 7f09fa1c8740 1 -- start start 2024-03-11T19:14:27.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.414+0000 7f09fa1c8740 1 Processor -- start 2024-03-11T19:14:27.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.415+0000 7f09fa1c8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161a2270 0x55e3161ac200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.415+0000 7f09fa1c8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161a7dd0 0x55e3161ac740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.415+0000 7f09fa1c8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161b1770 0x55e3161b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.415+0000 7f09f5390640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161a2270 0x55e3161ac200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.415+0000 7f09f5390640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161a2270 0x55e3161ac200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54742/0 (socket says 172.21.15.29:54742) 2024-03-11T19:14:27.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.415+0000 7f09fa1c8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31610b020 con 0x55e3161a2270 2024-03-11T19:14:27.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.415+0000 7f09fa1c8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e315dde430 con 0x55e3161b1770 2024-03-11T19:14:27.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.415+0000 7f09fa1c8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e315d9ae00 con 0x55e3161a7dd0 2024-03-11T19:14:27.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.415+0000 7f09f4b8f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161a7dd0 0x55e3161ac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.415+0000 7f09fa1c8740 10 client.0 _ll_register_callbacks cb 0x55e3160b2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:14:27.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.415+0000 7f09f5b91640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161b1770 0x55e3161b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.415+0000 7f09f5390640 1 -- 172.21.15.29:0/1790216310 learned_addr learned my addr 172.21.15.29:0/1790216310 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:27.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.415+0000 7f09f5390640 1 -- 172.21.15.29:0/1790216310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161a7dd0 msgr2=0x55e3161ac740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:27.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.415+0000 7f09f5390640 1 --2- 172.21.15.29:0/1790216310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161a7dd0 0x55e3161ac740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.415+0000 7f09f5390640 1 -- 172.21.15.29:0/1790216310 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161b1770 msgr2=0x55e3161b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:27.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.415+0000 7f09f5390640 1 --2- 172.21.15.29:0/1790216310 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161b1770 0x55e3161b3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.415+0000 7f09f5390640 1 -- 172.21.15.29:0/1790216310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f01a5bb0 con 0x55e3161a2270 2024-03-11T19:14:27.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.416+0000 7f09f5390640 1 --2- 172.21.15.29:0/1790216310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161a2270 0x55e3161ac200 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009900 tx=0x7f09ec0bb320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffc9a1729dea5cc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:27.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.416+0000 7f09d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:14:27.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.416+0000 7f09d8ff9640 1 -- 172.21.15.29:0/1790216310 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec09f8c0 con 0x55e3161a2270 2024-03-11T19:14:27.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.416+0000 7f09d8ff9640 1 -- 172.21.15.29:0/1790216310 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f09ec0bd510 con 0x55e3161a2270 2024-03-11T19:14:27.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.416+0000 7f09fa1c8740 1 -- 172.21.15.29:0/1790216310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e315e0a7e0 con 0x55e3161a2270 2024-03-11T19:14:27.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.416+0000 7f09d8ff9640 1 -- 172.21.15.29:0/1790216310 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09ec003e20 con 0x55e3161a2270 2024-03-11T19:14:27.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.416+0000 7f09fa1c8740 10 client.16614 fetch_fsmap learned FSMap version 27 2024-03-11T19:14:27.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.416+0000 7f09fa1c8740 1 -- 172.21.15.29:0/1790216310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e3161096a0 con 0x55e3161a2270 2024-03-11T19:14:27.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.417+0000 7f09d8ff9640 1 -- 172.21.15.29:0/1790216310 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55e315e0a7e0 con 0x55e3161a2270 2024-03-11T19:14:27.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.417+0000 7f09d8ff9640 1 -- 172.21.15.29:0/1790216310 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f09ec0f6020 con 0x55e3161a2270 2024-03-11T19:14:27.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.417+0000 7f09fa1c8740 10 client.16614 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:14:27.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.417+0000 7f09fa1c8740 10 client.16614 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:14:27.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.417+0000 7f09fa1c8740 20 client.16614 populate_metadata read hostname 'smithi029' 2024-03-11T19:14:27.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.417+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/1790216310 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e3161b8340 0x55e3161ba730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.417+0000 7f09fa1c8740 4 client.16614 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:14:27.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.417+0000 7f09fa1c8740 1 -- 172.21.15.29:0/1790216310 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e315e47720 con 0x55e3161b8340 2024-03-11T19:14:27.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.417+0000 7f09f4b8f640 1 --2- 172.21.15.29:0/1790216310 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e3161b8340 0x55e3161ba730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.418+0000 7f09f4b8f640 1 --2- 172.21.15.29:0/1790216310 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e3161b8340 0x55e3161ba730 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e7e27be3ecf0e08 server_cookie=907419ca2ac4321b in_seq=0 out_seq=0 2024-03-11T19:14:27.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.418+0000 7f09d8ff9640 10 client.16614 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:14:27.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.418+0000 7f09d8ff9640 1 -- 172.21.15.29:0/1790216310 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e315e47720 con 0x55e3161b8340 2024-03-11T19:14:27.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.418+0000 7f09d8ff9640 10 client.16614 handle_command_reply: tid=0 2024-03-11T19:14:27.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.418+0000 7f09fa1c8740 1 client.16614 shutdown 2024-03-11T19:14:27.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.418+0000 7f09fa1c8740 1 -- 172.21.15.29:0/1790216310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161a2270 msgr2=0x55e3161ac200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:27.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.419+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/1790216310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161a2270 0x55e3161ac200 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009900 tx=0x7f09ec0bb320 comp rx=0 tx=0).stop 2024-03-11T19:14:27.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.419+0000 7f09fa1c8740 1 -- 172.21.15.29:0/1790216310 shutdown_connections 2024-03-11T19:14:27.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.419+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/1790216310 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161b1770 0x55e3161b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.419+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/1790216310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161a7dd0 0x55e3161ac740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.419+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/1790216310 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e3161b8340 0x55e3161ba730 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.419+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/1790216310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161a2270 0x55e3161ac200 secure :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009900 tx=0x7f09ec0bb320 comp rx=0 tx=0).stop 2024-03-11T19:14:27.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.419+0000 7f09fa1c8740 1 -- 172.21.15.29:0/1790216310 >> 172.21.15.29:0/1790216310 conn(0x55e31619d7e0 msgr2=0x55e31619cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:27.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.419+0000 7f09d8ff9640 0 client.16614 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:14:27.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.419+0000 7f09fa1c8740 1 -- 172.21.15.29:0/1790216310 shutdown_connections 2024-03-11T19:14:27.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.419+0000 7f09fa1c8740 1 -- 172.21.15.29:0/1790216310 wait complete. 2024-03-11T19:14:27.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.419+0000 7f09fa1c8740 20 client.16614 trim_cache size 0 max 16384 2024-03-11T19:14:27.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.442+0000 7f09fa1c8740 1 Processor -- start 2024-03-11T19:14:27.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.442+0000 7f09fa1c8740 1 -- start start 2024-03-11T19:14:27.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.442+0000 7f09fa1c8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161a7dd0 0x55e3161bf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.442+0000 7f09fa1c8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161b1770 0x55e3161c85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.442+0000 7f09fa1c8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161bf5d0 0x55e3161c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.442+0000 7f09fa1c8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e316090130 con 0x55e3161b1770 2024-03-11T19:14:27.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.442+0000 7f09fa1c8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e315e47720 con 0x55e3161bf5d0 2024-03-11T19:14:27.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.442+0000 7f09fa1c8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31618fa40 con 0x55e3161a7dd0 2024-03-11T19:14:27.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.443+0000 7f09f5b91640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161bf5d0 0x55e3161c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.443+0000 7f09f4b8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161b1770 0x55e3161c85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.443+0000 7f09f5b91640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161bf5d0 0x55e3161c8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37732/0 (socket says 172.21.15.29:37732) 2024-03-11T19:14:27.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.443+0000 7f09f5b91640 1 -- 172.21.15.29:0/2122718699 learned_addr learned my addr 172.21.15.29:0/2122718699 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:27.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.443+0000 7f09f5390640 1 --2- 172.21.15.29:0/2122718699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161a7dd0 0x55e3161bf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.443+0000 7f09f5b91640 1 -- 172.21.15.29:0/2122718699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161a7dd0 msgr2=0x55e3161bf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:27.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.443+0000 7f09f5b91640 1 --2- 172.21.15.29:0/2122718699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161a7dd0 0x55e3161bf090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.443+0000 7f09f5b91640 1 -- 172.21.15.29:0/2122718699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161b1770 msgr2=0x55e3161c85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:27.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.443+0000 7f09f5b91640 1 --2- 172.21.15.29:0/2122718699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161b1770 0x55e3161c85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.443+0000 7f09f5b91640 1 -- 172.21.15.29:0/2122718699 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3160a5c20 con 0x55e3161bf5d0 2024-03-11T19:14:27.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.443+0000 7f09f5b91640 1 --2- 172.21.15.29:0/2122718699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161bf5d0 0x55e3161c8b10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f09e800f5e0 tx=0x7f09e8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f54f542c811f939 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:27.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.444+0000 7f09daffd640 1 -- 172.21.15.29:0/2122718699 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8024040 con 0x55e3161bf5d0 2024-03-11T19:14:27.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.444+0000 7f09daffd640 1 -- 172.21.15.29:0/2122718699 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f09e801d930 con 0x55e3161bf5d0 2024-03-11T19:14:27.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.444+0000 7f09daffd640 1 -- 172.21.15.29:0/2122718699 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8010980 con 0x55e3161bf5d0 2024-03-11T19:14:27.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.445+0000 7f09fa1c8740 1 -- 172.21.15.29:0/2122718699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161bf5d0 msgr2=0x55e3161c8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:27.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.445+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/2122718699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161bf5d0 0x55e3161c8b10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f09e800f5e0 tx=0x7f09e8017000 comp rx=0 tx=0).stop 2024-03-11T19:14:27.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.445+0000 7f09fa1c8740 1 -- 172.21.15.29:0/2122718699 shutdown_connections 2024-03-11T19:14:27.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.445+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/2122718699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161bf5d0 0x55e3161c8b10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.445+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/2122718699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161b1770 0x55e3161c85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.445+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/2122718699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161a7dd0 0x55e3161bf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.445+0000 7f09fa1c8740 1 -- 172.21.15.29:0/2122718699 >> 172.21.15.29:0/2122718699 conn(0x55e31619d7e0 msgr2=0x55e31619dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:27.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.448+0000 7f09fa1c8740 1 -- 172.21.15.29:0/2122718699 shutdown_connections 2024-03-11T19:14:27.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.448+0000 7f09fa1c8740 1 -- 172.21.15.29:0/2122718699 wait complete. 2024-03-11T19:14:27.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.449+0000 7f09fa1c8740 1 -- start start 2024-03-11T19:14:27.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.449+0000 7f09fa1c8740 1 Processor -- start 2024-03-11T19:14:27.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.449+0000 7f09fa1c8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161a7dd0 0x55e316209270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.450+0000 7f09fa1c8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161b1770 0x55e31620b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.450+0000 7f09fa1c8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161bf5d0 0x55e31620de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.450+0000 7f09fa1c8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31620e500 con 0x55e3161a7dd0 2024-03-11T19:14:27.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.450+0000 7f09fa1c8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31620e670 con 0x55e3161bf5d0 2024-03-11T19:14:27.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.450+0000 7f09fa1c8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31620e830 con 0x55e3161b1770 2024-03-11T19:14:27.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.450+0000 7f09fa1c8740 10 client.0 _ll_register_callbacks cb 0x7f09ec0b9130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:14:27.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.450+0000 7f09f5390640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161a7dd0 0x55e316209270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.450+0000 7f09f5b91640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161bf5d0 0x55e31620de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.450+0000 7f09f5390640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161a7dd0 0x55e316209270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54772/0 (socket says 172.21.15.29:54772) 2024-03-11T19:14:27.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.450+0000 7f09f5b91640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161bf5d0 0x55e31620de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37748/0 (socket says 172.21.15.29:37748) 2024-03-11T19:14:27.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.450+0000 7f09f5390640 1 -- 172.21.15.29:0/2048022576 learned_addr learned my addr 172.21.15.29:0/2048022576 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:27.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.450+0000 7f09f4b8f640 1 --2- 172.21.15.29:0/2048022576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161b1770 0x55e31620b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.450+0000 7f09f5390640 1 -- 172.21.15.29:0/2048022576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161b1770 msgr2=0x55e31620b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:27.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.450+0000 7f09f5390640 1 --2- 172.21.15.29:0/2048022576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161b1770 0x55e31620b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.450+0000 7f09f5390640 1 -- 172.21.15.29:0/2048022576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161bf5d0 msgr2=0x55e31620de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:27.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.450+0000 7f09f5390640 1 --2- 172.21.15.29:0/2048022576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161bf5d0 0x55e31620de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.450+0000 7f09f5390640 1 -- 172.21.15.29:0/2048022576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e80131f0 con 0x55e3161a7dd0 2024-03-11T19:14:27.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.451+0000 7f09f4b8f640 1 --2- 172.21.15.29:0/2048022576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161b1770 0x55e31620b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:14:27.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.451+0000 7f09f5b91640 1 --2- 172.21.15.29:0/2048022576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161bf5d0 0x55e31620de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:14:27.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.451+0000 7f09f5390640 1 --2- 172.21.15.29:0/2048022576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161a7dd0 0x55e316209270 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f09ec004c20 tx=0x7f09ec004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5093f2897fe1d17f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:27.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.451+0000 7f09d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:14:27.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.451+0000 7f09d8ff9640 1 -- 172.21.15.29:0/2048022576 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec0bd300 con 0x55e3161a7dd0 2024-03-11T19:14:27.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.451+0000 7f09d8ff9640 1 -- 172.21.15.29:0/2048022576 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f09ec0b7710 con 0x55e3161a7dd0 2024-03-11T19:14:27.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.451+0000 7f09fa1c8740 1 -- 172.21.15.29:0/2048022576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f09ec09f330 con 0x55e3161a7dd0 2024-03-11T19:14:27.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.452+0000 7f09d8ff9640 1 -- 172.21.15.29:0/2048022576 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09ec0bd510 con 0x55e3161a7dd0 2024-03-11T19:14:27.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.452+0000 7f09fa1c8740 10 client.16620 fetch_fsmap learned FSMap version 27 2024-03-11T19:14:27.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.452+0000 7f09fa1c8740 1 -- 172.21.15.29:0/2048022576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e3160cb8c0 con 0x55e3161a7dd0 2024-03-11T19:14:27.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.452+0000 7f09d8ff9640 1 -- 172.21.15.29:0/2048022576 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f09ec09f330 con 0x55e3161a7dd0 2024-03-11T19:14:27.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.452+0000 7f09d8ff9640 1 -- 172.21.15.29:0/2048022576 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f09ec0f6040 con 0x55e3161a7dd0 2024-03-11T19:14:27.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.452+0000 7f09fa1c8740 10 client.16620 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:14:27.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.452+0000 7f09fa1c8740 10 client.16620 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:14:27.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.452+0000 7f09fa1c8740 20 client.16620 populate_metadata read hostname 'smithi029' 2024-03-11T19:14:27.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.452+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/2048022576 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e31620fda0 0x55e316212190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:27.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.452+0000 7f09fa1c8740 4 client.16620 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:14:27.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.452+0000 7f09fa1c8740 1 -- 172.21.15.29:0/2048022576 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e315e96890 con 0x55e31620fda0 2024-03-11T19:14:27.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.452+0000 7f09f4b8f640 1 --2- 172.21.15.29:0/2048022576 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e31620fda0 0x55e316212190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:27.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.453+0000 7f09d8ff9640 10 client.16620 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:14:27.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.453+0000 7f09f4b8f640 1 --2- 172.21.15.29:0/2048022576 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e31620fda0 0x55e316212190 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dc2c8f28722f03f server_cookie=22cccd995c696038 in_seq=0 out_seq=0 2024-03-11T19:14:27.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.453+0000 7f09d8ff9640 1 -- 172.21.15.29:0/2048022576 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e315e96890 con 0x55e31620fda0 2024-03-11T19:14:27.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.453+0000 7f09d8ff9640 10 client.16620 handle_command_reply: tid=0 2024-03-11T19:14:27.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.453+0000 7f09fa1c8740 1 client.16620 shutdown 2024-03-11T19:14:27.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.453+0000 7f09fa1c8740 1 -- 172.21.15.29:0/2048022576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161a7dd0 msgr2=0x55e316209270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:27.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.453+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/2048022576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161a7dd0 0x55e316209270 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f09ec004c20 tx=0x7f09ec004e40 comp rx=0 tx=0).stop 2024-03-11T19:14:27.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.453+0000 7f09fa1c8740 1 -- 172.21.15.29:0/2048022576 shutdown_connections 2024-03-11T19:14:27.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.453+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/2048022576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3161bf5d0 0x55e31620de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.453+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/2048022576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e3161b1770 0x55e31620b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.453+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/2048022576 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55e31620fda0 0x55e316212190 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.453+0000 7f09d8ff9640 0 client.16620 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:14:27.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.453+0000 7f09fa1c8740 1 --2- 172.21.15.29:0/2048022576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3161a7dd0 0x55e316209270 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.454+0000 7f09fa1c8740 1 -- 172.21.15.29:0/2048022576 >> 172.21.15.29:0/2048022576 conn(0x55e31619d7e0 msgr2=0x55e3161a41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:27.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.454+0000 7f09fa1c8740 1 -- 172.21.15.29:0/2048022576 shutdown_connections 2024-03-11T19:14:27.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.454+0000 7f09fa1c8740 1 -- 172.21.15.29:0/2048022576 wait complete. 2024-03-11T19:14:27.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.454+0000 7f09fa1c8740 20 client.16620 trim_cache size 0 max 16384 2024-03-11T19:14:27.460 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:14:27.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.455+0000 7f09f75f4640 1 -- 172.21.15.29:0/641506751 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f09dc0a1b30 msgr2=0x7f09dc0a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:27.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.455+0000 7f09f75f4640 1 --2- 172.21.15.29:0/641506751 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f09dc0a1b30 0x7f09dc0a3ff0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f09e8009fd0 tx=0x7f09e8015040 comp rx=0 tx=0).stop 2024-03-11T19:14:27.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.455+0000 7f09f75f4640 1 -- 172.21.15.29:0/641506751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09f01077a0 msgr2=0x7f09f019d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:27.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.455+0000 7f09f75f4640 1 --2- 172.21.15.29:0/641506751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09f01077a0 0x7f09f019d5e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009230 tx=0x7f09ec0042f0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.455+0000 7f09f75f4640 1 -- 172.21.15.29:0/641506751 shutdown_connections 2024-03-11T19:14:27.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.455+0000 7f09f75f4640 1 --2- 172.21.15.29:0/641506751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09f0109350 0x7f09f01a4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.455+0000 7f09f75f4640 1 --2- 172.21.15.29:0/641506751 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f09dc0a1b30 0x7f09dc0a3ff0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.455+0000 7f09f75f4640 1 --2- 172.21.15.29:0/641506751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09f0108990 0x7f09f019db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.455+0000 7f09f75f4640 1 --2- 172.21.15.29:0/641506751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09f01077a0 0x7f09f019d5e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:27.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.455+0000 7f09f75f4640 1 -- 172.21.15.29:0/641506751 >> 172.21.15.29:0/641506751 conn(0x7f09f0076730 msgr2=0x7f09f010bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:27.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.456+0000 7f09f75f4640 1 -- 172.21.15.29:0/641506751 shutdown_connections 2024-03-11T19:14:27.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:27.456+0000 7f09f75f4640 1 -- 172.21.15.29:0/641506751 wait complete. 2024-03-11T19:14:27.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:27 smithi029 ceph-mon[32439]: pgmap v490: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-11T19:14:27.913 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:27 smithi190 ceph-mon[36292]: pgmap v490: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-11T19:14:27.997 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:27 smithi170 ceph-mon[35690]: pgmap v490: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-11T19:14:28.478 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:14:28.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:14:28.478 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:14:28.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.665+0000 7f1f52a8f640 1 -- 172.21.15.29:0/396641112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f4c10d4a0 msgr2=0x7f1f4c10d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:28.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.665+0000 7f1f52a8f640 1 --2- 172.21.15.29:0/396641112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f4c10d4a0 0x7f1f4c10d920 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f1f44009be0 tx=0x7f1f4402bf30 comp rx=0 tx=0).stop 2024-03-11T19:14:28.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.666+0000 7f1f52a8f640 1 -- 172.21.15.29:0/396641112 shutdown_connections 2024-03-11T19:14:28.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.666+0000 7f1f52a8f640 1 --2- 172.21.15.29:0/396641112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f4c10d4a0 0x7f1f4c10d920 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.666+0000 7f1f52a8f640 1 --2- 172.21.15.29:0/396641112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f4c10aaf0 0x7f1f4c10cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.666+0000 7f1f52a8f640 1 --2- 172.21.15.29:0/396641112 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1f4c108180 0x7f1f4c10a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.666+0000 7f1f52a8f640 1 -- 172.21.15.29:0/396641112 >> 172.21.15.29:0/396641112 conn(0x7f1f4c06dfe0 msgr2=0x7f1f4c070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:28.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.667+0000 7f1f52a8f640 1 -- 172.21.15.29:0/396641112 shutdown_connections 2024-03-11T19:14:28.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.667+0000 7f1f52a8f640 1 -- 172.21.15.29:0/396641112 wait complete. 2024-03-11T19:14:28.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.668+0000 7f1f52a8f640 1 Processor -- start 2024-03-11T19:14:28.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.668+0000 7f1f52a8f640 1 -- start start 2024-03-11T19:14:28.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.668+0000 7f1f52a8f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1f4c108180 0x7f1f4c1147a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.668+0000 7f1f52a8f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f4c10aaf0 0x7f1f4c114ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.668+0000 7f1f52a8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f4c10d4a0 0x7f1f4c1b6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.668+0000 7f1f52a8f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c117d30 con 0x7f1f4c10d4a0 2024-03-11T19:14:28.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.668+0000 7f1f52a8f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c117ea0 con 0x7f1f4c108180 2024-03-11T19:14:28.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.668+0000 7f1f52a8f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c1b65d0 con 0x7f1f4c10aaf0 2024-03-11T19:14:28.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.668+0000 7f1f5082b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1f4c108180 0x7f1f4c1147a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.668+0000 7f1f4bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f4c10aaf0 0x7f1f4c114ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.669+0000 7f1f4bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f4c10aaf0 0x7f1f4c114ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41654/0 (socket says 172.21.15.29:41654) 2024-03-11T19:14:28.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.669+0000 7f1f4bfff640 1 -- 172.21.15.29:0/3066839361 learned_addr learned my addr 172.21.15.29:0/3066839361 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:28.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.669+0000 7f1f4bfff640 1 -- 172.21.15.29:0/3066839361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1f4c108180 msgr2=0x7f1f4c1147a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:28.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.669+0000 7f1f4bfff640 1 --2- 172.21.15.29:0/3066839361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1f4c108180 0x7f1f4c1147a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.669+0000 7f1f4bfff640 1 -- 172.21.15.29:0/3066839361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f4c10d4a0 msgr2=0x7f1f4c1b6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:28.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.669+0000 7f1f4bfff640 1 --2- 172.21.15.29:0/3066839361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f4c10d4a0 0x7f1f4c1b6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.669+0000 7f1f4bfff640 1 -- 172.21.15.29:0/3066839361 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f44009840 con 0x7f1f4c10aaf0 2024-03-11T19:14:28.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.669+0000 7f1f4bfff640 1 --2- 172.21.15.29:0/3066839361 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f4c10aaf0 0x7f1f4c114ce0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1f4000e9c0 tx=0x7f1f4000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:28.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.670+0000 7f1f49ffb640 1 -- 172.21.15.29:0/3066839361 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4000ccc0 con 0x7f1f4c10aaf0 2024-03-11T19:14:28.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.670+0000 7f1f49ffb640 1 -- 172.21.15.29:0/3066839361 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1f4000ce20 con 0x7f1f4c10aaf0 2024-03-11T19:14:28.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.670+0000 7f1f49ffb640 1 -- 172.21.15.29:0/3066839361 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40004e70 con 0x7f1f4c10aaf0 2024-03-11T19:14:28.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.671+0000 7f1f52a8f640 1 -- 172.21.15.29:0/3066839361 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f4c1b6800 con 0x7f1f4c10aaf0 2024-03-11T19:14:28.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.672+0000 7f1f52a8f640 1 -- 172.21.15.29:0/3066839361 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f4c1b6c90 con 0x7f1f4c10aaf0 2024-03-11T19:14:28.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.676+0000 7f1f5565d740 1 Processor -- start 2024-03-11T19:14:28.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.677+0000 7f1f49ffb640 1 -- 172.21.15.29:0/3066839361 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1f40004fd0 con 0x7f1f4c10aaf0 2024-03-11T19:14:28.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.679+0000 7f1f49ffb640 1 --2- 172.21.15.29:0/3066839361 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1f340a1bb0 0x7f1f340a4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.679+0000 7f1f5565d740 1 -- start start 2024-03-11T19:14:28.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.680+0000 7f1f49ffb640 1 -- 172.21.15.29:0/3066839361 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f1f40014070 con 0x7f1f4c10aaf0 2024-03-11T19:14:28.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.680+0000 7f1f5565d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 0x560ecfe388e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.680+0000 7f1f5565d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ecfe39690 0x560ecfe39b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.680+0000 7f1f5565d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe3a050 0x560ecfe408e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.680+0000 7f1f5565d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecfda7f90 con 0x560ecfe39690 2024-03-11T19:14:28.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.680+0000 7f1f5565d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecfda54c0 con 0x560ecfe3a050 2024-03-11T19:14:28.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.680+0000 7f1f5565d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecfda0ca0 con 0x560ecfe384c0 2024-03-11T19:14:28.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.680+0000 7f1f5082b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 0x560ecfe388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.680+0000 7f1f5082b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 0x560ecfe388e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41660/0 (socket says 172.21.15.29:41660) 2024-03-11T19:14:28.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.680+0000 7f1f5082b640 1 -- 172.21.15.29:0/4149410298 learned_addr learned my addr 172.21.15.29:0/4149410298 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.680+0000 7f1f5102c640 1 --2- 172.21.15.29:0/3066839361 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1f340a1bb0 0x7f1f340a4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.680+0000 7f1f5102c640 1 --2- 172.21.15.29:0/4149410298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe3a050 0x560ecfe408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.681+0000 7f1f4bfff640 1 --2- 172.21.15.29:0/4149410298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ecfe39690 0x560ecfe39b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.681+0000 7f1f5102c640 1 --2- 172.21.15.29:0/3066839361 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1f340a1bb0 0x7f1f340a4070 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1f44005ee0 tx=0x7f1f44004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.681+0000 7f1f5102c640 1 -- 172.21.15.29:0/4149410298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 msgr2=0x560ecfe388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.681+0000 7f1f5102c640 1 --2- 172.21.15.29:0/4149410298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 0x560ecfe388e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.681+0000 7f1f5102c640 1 -- 172.21.15.29:0/4149410298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ecfe39690 msgr2=0x560ecfe39b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.681+0000 7f1f5102c640 1 --2- 172.21.15.29:0/4149410298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ecfe39690 0x560ecfe39b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.681+0000 7f1f5102c640 1 -- 172.21.15.29:0/4149410298 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ecfd630b0 con 0x560ecfe3a050 2024-03-11T19:14:28.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.682+0000 7f1f5102c640 1 --2- 172.21.15.29:0/4149410298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe3a050 0x560ecfe408e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1f44009b30 tx=0x7f1f44048000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5b8c54ad461d465 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:28.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.682+0000 7f1f32ffd640 1 -- 172.21.15.29:0/4149410298 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f44034880 con 0x560ecfe3a050 2024-03-11T19:14:28.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.682+0000 7f1f32ffd640 1 -- 172.21.15.29:0/4149410298 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1f440349e0 con 0x560ecfe3a050 2024-03-11T19:14:28.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.682+0000 7f1f32ffd640 1 -- 172.21.15.29:0/4149410298 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f440423e0 con 0x560ecfe3a050 2024-03-11T19:14:28.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.683+0000 7f1f5565d740 1 -- 172.21.15.29:0/4149410298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe3a050 msgr2=0x560ecfe408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:28.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.683+0000 7f1f5565d740 1 --2- 172.21.15.29:0/4149410298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe3a050 0x560ecfe408e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1f44009b30 tx=0x7f1f44048000 comp rx=0 tx=0).stop 2024-03-11T19:14:28.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.684+0000 7f1f5565d740 1 -- 172.21.15.29:0/4149410298 shutdown_connections 2024-03-11T19:14:28.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.684+0000 7f1f5565d740 1 --2- 172.21.15.29:0/4149410298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe3a050 0x560ecfe408e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.684+0000 7f1f5565d740 1 --2- 172.21.15.29:0/4149410298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ecfe39690 0x560ecfe39b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.684+0000 7f1f5565d740 1 --2- 172.21.15.29:0/4149410298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 0x560ecfe388e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.684+0000 7f1f5565d740 1 -- 172.21.15.29:0/4149410298 >> 172.21.15.29:0/4149410298 conn(0x560ecfe336a0 msgr2=0x560ecfe33aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:28.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.684+0000 7f1f5565d740 1 -- 172.21.15.29:0/4149410298 shutdown_connections 2024-03-11T19:14:28.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.684+0000 7f1f5565d740 1 -- 172.21.15.29:0/4149410298 wait complete. 2024-03-11T19:14:28.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.685+0000 7f1f5565d740 1 -- start start 2024-03-11T19:14:28.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.685+0000 7f1f5565d740 1 Processor -- start 2024-03-11T19:14:28.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.685+0000 7f1f5565d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 0x560ecfe42730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.685+0000 7f1f5565d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ecfe39690 0x560ecfe42c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.686+0000 7f1f5565d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe3a050 0x560ecfe49cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.686+0000 7f1f5565d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecfd45f40 con 0x560ecfe39690 2024-03-11T19:14:28.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.686+0000 7f1f5565d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecfa744b0 con 0x560ecfe3a050 2024-03-11T19:14:28.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.686+0000 7f1f5565d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecfd26670 con 0x560ecfe384c0 2024-03-11T19:14:28.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.686+0000 7f1f5565d740 10 client.0 _ll_register_callbacks cb 0x560ecfd93f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:14:28.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.686+0000 7f1f5102c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe3a050 0x560ecfe49cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.686+0000 7f1f5102c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe3a050 0x560ecfe49cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37782/0 (socket says 172.21.15.29:37782) 2024-03-11T19:14:28.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.686+0000 7f1f5102c640 1 -- 172.21.15.29:0/2749159744 learned_addr learned my addr 172.21.15.29:0/2749159744 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:28.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.686+0000 7f1f4bfff640 1 --2- 172.21.15.29:0/2749159744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ecfe39690 0x560ecfe42c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.686+0000 7f1f5082b640 1 --2- 172.21.15.29:0/2749159744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 0x560ecfe42730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.686+0000 7f1f5102c640 1 -- 172.21.15.29:0/2749159744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 msgr2=0x560ecfe42730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:28.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.686+0000 7f1f5102c640 1 --2- 172.21.15.29:0/2749159744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 0x560ecfe42730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.686+0000 7f1f5102c640 1 -- 172.21.15.29:0/2749159744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ecfe39690 msgr2=0x560ecfe42c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:28.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.686+0000 7f1f5102c640 1 --2- 172.21.15.29:0/2749159744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ecfe39690 0x560ecfe42c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.686+0000 7f1f5102c640 1 -- 172.21.15.29:0/2749159744 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f4c10e4b0 con 0x560ecfe3a050 2024-03-11T19:14:28.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.687+0000 7f1f5102c640 1 --2- 172.21.15.29:0/2749159744 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe3a050 0x560ecfe49cb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1f44005bd0 tx=0x7f1f44005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f0c239cafbd0a7a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:28.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.687+0000 7f1f2bfff640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:14:28.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.687+0000 7f1f2bfff640 1 -- 172.21.15.29:0/2749159744 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f44043e50 con 0x560ecfe3a050 2024-03-11T19:14:28.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.687+0000 7f1f2bfff640 1 -- 172.21.15.29:0/2749159744 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f1f44034880 con 0x560ecfe3a050 2024-03-11T19:14:28.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.687+0000 7f1f5565d740 1 -- 172.21.15.29:0/2749159744 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560ecfd7af40 con 0x560ecfe3a050 2024-03-11T19:14:28.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.687+0000 7f1f2bfff640 1 -- 172.21.15.29:0/2749159744 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f44034a40 con 0x560ecfe3a050 2024-03-11T19:14:28.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.687+0000 7f1f5565d740 10 client.26153 fetch_fsmap learned FSMap version 27 2024-03-11T19:14:28.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.687+0000 7f1f5565d740 1 -- 172.21.15.29:0/2749159744 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x560ecfd9f320 con 0x560ecfe3a050 2024-03-11T19:14:28.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.688+0000 7f1f2bfff640 1 -- 172.21.15.29:0/2749159744 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x560ecfd7af40 con 0x560ecfe3a050 2024-03-11T19:14:28.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.688+0000 7f1f2bfff640 1 -- 172.21.15.29:0/2749159744 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1f44063880 con 0x560ecfe3a050 2024-03-11T19:14:28.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.688+0000 7f1f5565d740 10 client.26153 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:14:28.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.688+0000 7f1f5565d740 10 client.26153 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:14:28.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.688+0000 7f1f5565d740 20 client.26153 populate_metadata read hostname 'smithi029' 2024-03-11T19:14:28.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.689+0000 7f1f5565d740 1 --2- 172.21.15.29:0/2749159744 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x560ecfe4e580 0x560ecfe50970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.689+0000 7f1f5565d740 4 client.26153 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:14:28.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.689+0000 7f1f5565d740 1 -- 172.21.15.29:0/2749159744 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560ecfd97310 con 0x560ecfe4e580 2024-03-11T19:14:28.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.689+0000 7f1f5082b640 1 --2- 172.21.15.29:0/2749159744 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x560ecfe4e580 0x560ecfe50970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.690+0000 7f1f2bfff640 10 client.26153 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:14:28.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.690+0000 7f1f5082b640 1 --2- 172.21.15.29:0/2749159744 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x560ecfe4e580 0x560ecfe50970 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c412939eaeec20ed server_cookie=ed1f49828daccc96 in_seq=0 out_seq=0 2024-03-11T19:14:28.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.691+0000 7f1f2bfff640 1 -- 172.21.15.29:0/2749159744 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560ecfd97310 con 0x560ecfe4e580 2024-03-11T19:14:28.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.691+0000 7f1f2bfff640 10 client.26153 handle_command_reply: tid=0 2024-03-11T19:14:28.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.691+0000 7f1f5565d740 1 client.26153 shutdown 2024-03-11T19:14:28.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.691+0000 7f1f5565d740 1 -- 172.21.15.29:0/2749159744 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe3a050 msgr2=0x560ecfe49cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:28.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.691+0000 7f1f5565d740 1 --2- 172.21.15.29:0/2749159744 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe3a050 0x560ecfe49cb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1f44005bd0 tx=0x7f1f44005fd0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.691+0000 7f1f5565d740 1 -- 172.21.15.29:0/2749159744 shutdown_connections 2024-03-11T19:14:28.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.691+0000 7f1f5565d740 1 --2- 172.21.15.29:0/2749159744 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe3a050 0x560ecfe49cb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.691+0000 7f1f5565d740 1 --2- 172.21.15.29:0/2749159744 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x560ecfe4e580 0x560ecfe50970 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.691+0000 7f1f5565d740 1 --2- 172.21.15.29:0/2749159744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ecfe39690 0x560ecfe42c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.691+0000 7f1f5565d740 1 --2- 172.21.15.29:0/2749159744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 0x560ecfe42730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.691+0000 7f1f5565d740 1 -- 172.21.15.29:0/2749159744 >> 172.21.15.29:0/2749159744 conn(0x560ecfe336a0 msgr2=0x560ecfe328e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:28.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.691+0000 7f1f2bfff640 0 client.26153 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:14:28.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.691+0000 7f1f5565d740 1 -- 172.21.15.29:0/2749159744 shutdown_connections 2024-03-11T19:14:28.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.691+0000 7f1f5565d740 1 -- 172.21.15.29:0/2749159744 wait complete. 2024-03-11T19:14:28.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.692+0000 7f1f5565d740 20 client.26153 trim_cache size 0 max 16384 2024-03-11T19:14:28.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.739+0000 7f1f5565d740 1 Processor -- start 2024-03-11T19:14:28.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.739+0000 7f1f5565d740 1 -- start start 2024-03-11T19:14:28.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.739+0000 7f1f5565d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 0x560ecfe552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.740+0000 7f1f5565d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe39690 0x560ecfe49bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.740+0000 7f1f5565d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ecfe3a050 0x560ecfe42730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.740+0000 7f1f4bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe39690 0x560ecfe49bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.740+0000 7f1f4bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe39690 0x560ecfe49bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37786/0 (socket says 172.21.15.29:37786) 2024-03-11T19:14:28.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.740+0000 7f1f5082b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 0x560ecfe552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.740+0000 7f1f5082b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 0x560ecfe552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41690/0 (socket says 172.21.15.29:41690) 2024-03-11T19:14:28.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.740+0000 7f1f4bfff640 1 -- 172.21.15.29:0/1855252141 learned_addr learned my addr 172.21.15.29:0/1855252141 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:28.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.740+0000 7f1f5565d740 1 -- 172.21.15.29:0/1855252141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecfe3ce40 con 0x560ecfe3a050 2024-03-11T19:14:28.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.740+0000 7f1f5565d740 1 -- 172.21.15.29:0/1855252141 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecfd97310 con 0x560ecfe39690 2024-03-11T19:14:28.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.740+0000 7f1f5565d740 1 -- 172.21.15.29:0/1855252141 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecfd7af40 con 0x560ecfe384c0 2024-03-11T19:14:28.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.740+0000 7f1f5102c640 1 --2- 172.21.15.29:0/1855252141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ecfe3a050 0x560ecfe42730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.741+0000 7f1f4bfff640 1 -- 172.21.15.29:0/1855252141 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 msgr2=0x560ecfe552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:28.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.741+0000 7f1f4bfff640 1 --2- 172.21.15.29:0/1855252141 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 0x560ecfe552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.741+0000 7f1f4bfff640 1 -- 172.21.15.29:0/1855252141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ecfe3a050 msgr2=0x560ecfe42730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:28.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.741+0000 7f1f4bfff640 1 --2- 172.21.15.29:0/1855252141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ecfe3a050 0x560ecfe42730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.741+0000 7f1f4bfff640 1 -- 172.21.15.29:0/1855252141 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ecfd3bc00 con 0x560ecfe39690 2024-03-11T19:14:28.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.741+0000 7f1f4bfff640 1 --2- 172.21.15.29:0/1855252141 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe39690 0x560ecfe49bd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1f40096340 tx=0x7f1f400ba040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ee693ce6c4acd22 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:28.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.743+0000 7f1f327fc640 1 -- 172.21.15.29:0/1855252141 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f400b31a0 con 0x560ecfe39690 2024-03-11T19:14:28.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.743+0000 7f1f327fc640 1 -- 172.21.15.29:0/1855252141 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1f4009a660 con 0x560ecfe39690 2024-03-11T19:14:28.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.743+0000 7f1f327fc640 1 -- 172.21.15.29:0/1855252141 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f400ba0b0 con 0x560ecfe39690 2024-03-11T19:14:28.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.744+0000 7f1f5565d740 1 -- 172.21.15.29:0/1855252141 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe39690 msgr2=0x560ecfe49bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:28.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.744+0000 7f1f5565d740 1 --2- 172.21.15.29:0/1855252141 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe39690 0x560ecfe49bd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1f40096340 tx=0x7f1f400ba040 comp rx=0 tx=0).stop 2024-03-11T19:14:28.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.744+0000 7f1f5565d740 1 -- 172.21.15.29:0/1855252141 shutdown_connections 2024-03-11T19:14:28.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.744+0000 7f1f5565d740 1 --2- 172.21.15.29:0/1855252141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ecfe3a050 0x560ecfe42730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.744+0000 7f1f5565d740 1 --2- 172.21.15.29:0/1855252141 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe39690 0x560ecfe49bd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.744+0000 7f1f5565d740 1 --2- 172.21.15.29:0/1855252141 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 0x560ecfe552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.744+0000 7f1f5565d740 1 -- 172.21.15.29:0/1855252141 >> 172.21.15.29:0/1855252141 conn(0x560ecfe336a0 msgr2=0x560ecfe3c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:28.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.747+0000 7f1f5565d740 1 -- 172.21.15.29:0/1855252141 shutdown_connections 2024-03-11T19:14:28.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.747+0000 7f1f5565d740 1 -- 172.21.15.29:0/1855252141 wait complete. 2024-03-11T19:14:28.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.748+0000 7f1f5565d740 1 -- start start 2024-03-11T19:14:28.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.748+0000 7f1f5565d740 1 Processor -- start 2024-03-11T19:14:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.749+0000 7f1f5565d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 0x560ecfe495d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.749+0000 7f1f5565d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe39690 0x560ecfe49b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.749+0000 7f1f5565d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ecfe3a050 0x560ecfe460b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.750+0000 7f1f5565d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecfe467b0 con 0x560ecfe3a050 2024-03-11T19:14:28.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.750+0000 7f1f4bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe39690 0x560ecfe49b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.750+0000 7f1f5082b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 0x560ecfe495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.750+0000 7f1f5565d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecfe46920 con 0x560ecfe39690 2024-03-11T19:14:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.750+0000 7f1f5565d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecfe46ae0 con 0x560ecfe384c0 2024-03-11T19:14:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.750+0000 7f1f4bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe39690 0x560ecfe49b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37794/0 (socket says 172.21.15.29:37794) 2024-03-11T19:14:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.750+0000 7f1f4bfff640 1 -- 172.21.15.29:0/2094448554 learned_addr learned my addr 172.21.15.29:0/2094448554 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.750+0000 7f1f5565d740 10 client.0 _ll_register_callbacks cb 0x7f1f3c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:14:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.750+0000 7f1f4bfff640 1 -- 172.21.15.29:0/2094448554 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 msgr2=0x560ecfe495d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.750+0000 7f1f4bfff640 1 --2- 172.21.15.29:0/2094448554 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 0x560ecfe495d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.750+0000 7f1f4bfff640 1 -- 172.21.15.29:0/2094448554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ecfe3a050 msgr2=0x560ecfe460b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.750+0000 7f1f4bfff640 1 --2- 172.21.15.29:0/2094448554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ecfe3a050 0x560ecfe460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.750+0000 7f1f4bfff640 1 -- 172.21.15.29:0/2094448554 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f4c1b6c90 con 0x560ecfe39690 2024-03-11T19:14:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.751+0000 7f1f4bfff640 1 --2- 172.21.15.29:0/2094448554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe39690 0x560ecfe49b10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1f400cb240 tx=0x7f1f400094c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e8389be8187be2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.751+0000 7f1f2bfff640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:14:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.752+0000 7f1f2bfff640 1 -- 172.21.15.29:0/2094448554 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f400b31a0 con 0x560ecfe39690 2024-03-11T19:14:28.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.752+0000 7f1f2bfff640 1 -- 172.21.15.29:0/2094448554 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f1f4009e7b0 con 0x560ecfe39690 2024-03-11T19:14:28.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.755+0000 7f1f5565d740 1 -- 172.21.15.29:0/2094448554 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560ecfdc6900 con 0x560ecfe39690 2024-03-11T19:14:28.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.756+0000 7f1f2bfff640 1 -- 172.21.15.29:0/2094448554 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f400ba2c0 con 0x560ecfe39690 2024-03-11T19:14:28.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.758+0000 7f1f5565d740 10 client.26159 fetch_fsmap learned FSMap version 27 2024-03-11T19:14:28.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.758+0000 7f1f5565d740 1 -- 172.21.15.29:0/2094448554 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x560ecfd61910 con 0x560ecfe39690 2024-03-11T19:14:28.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.759+0000 7f1f2bfff640 1 -- 172.21.15.29:0/2094448554 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1f400ae410 con 0x560ecfe39690 2024-03-11T19:14:28.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.759+0000 7f1f2bfff640 1 -- 172.21.15.29:0/2094448554 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1f400996c0 con 0x560ecfe39690 2024-03-11T19:14:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.759+0000 7f1f5565d740 10 client.26159 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:14:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.760+0000 7f1f5565d740 10 client.26159 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:14:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.760+0000 7f1f5565d740 20 client.26159 populate_metadata read hostname 'smithi029' 2024-03-11T19:14:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.760+0000 7f1f5565d740 1 --2- 172.21.15.29:0/2094448554 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x560ecfea5ff0 0x560ecfea8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.760+0000 7f1f5565d740 4 client.26159 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:14:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.760+0000 7f1f5565d740 1 -- 172.21.15.29:0/2094448554 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x560ecfdc6900 con 0x560ecfea5ff0 2024-03-11T19:14:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.760+0000 7f1f5082b640 1 --2- 172.21.15.29:0/2094448554 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x560ecfea5ff0 0x560ecfea8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.766+0000 7f1f5082b640 1 --2- 172.21.15.29:0/2094448554 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x560ecfea5ff0 0x560ecfea8410 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=796f7685a3e0998b server_cookie=c8506e580fba4b66 in_seq=0 out_seq=0 2024-03-11T19:14:28.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.766+0000 7f1f2bfff640 10 client.26159 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:14:28.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.769+0000 7f1f2bfff640 1 -- 172.21.15.29:0/2094448554 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x560ecfdc6900 con 0x560ecfea5ff0 2024-03-11T19:14:28.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.769+0000 7f1f2bfff640 10 client.26159 handle_command_reply: tid=0 2024-03-11T19:14:28.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.769+0000 7f1f5565d740 1 client.26159 shutdown 2024-03-11T19:14:28.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.773+0000 7f1f5565d740 1 -- 172.21.15.29:0/2094448554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe39690 msgr2=0x560ecfe49b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:28.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.773+0000 7f1f5565d740 1 --2- 172.21.15.29:0/2094448554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe39690 0x560ecfe49b10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1f400cb240 tx=0x7f1f400094c0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.778+0000 7f1f5565d740 1 -- 172.21.15.29:0/2094448554 shutdown_connections 2024-03-11T19:14:28.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.778+0000 7f1f5565d740 1 --2- 172.21.15.29:0/2094448554 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x560ecfea5ff0 0x560ecfea8410 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.778+0000 7f1f5565d740 1 --2- 172.21.15.29:0/2094448554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560ecfe3a050 0x560ecfe460b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.778+0000 7f1f5565d740 1 --2- 172.21.15.29:0/2094448554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecfe39690 0x560ecfe49b10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.778+0000 7f1f5565d740 1 --2- 172.21.15.29:0/2094448554 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560ecfe384c0 0x560ecfe495d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.778+0000 7f1f5565d740 1 -- 172.21.15.29:0/2094448554 >> 172.21.15.29:0/2094448554 conn(0x560ecfe336a0 msgr2=0x560ecfe32050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:28.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.778+0000 7f1f2bfff640 0 client.26159 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:14:28.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.778+0000 7f1f5565d740 1 -- 172.21.15.29:0/2094448554 shutdown_connections 2024-03-11T19:14:28.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.778+0000 7f1f5565d740 1 -- 172.21.15.29:0/2094448554 wait complete. 2024-03-11T19:14:28.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.779+0000 7f1f5565d740 20 client.26159 trim_cache size 0 max 16384 2024-03-11T19:14:28.783 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:14:28.783 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:14:28.783 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "cdf4988e-c7d7-4a00-9d1c-6a125e257ccb", 2024-03-11T19:14:28.783 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:14:28.783 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:14:28.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.780+0000 7f1f52a8f640 1 -- 172.21.15.29:0/3066839361 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1f340a1bb0 msgr2=0x7f1f340a4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:28.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.780+0000 7f1f52a8f640 1 --2- 172.21.15.29:0/3066839361 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1f340a1bb0 0x7f1f340a4070 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1f44005ee0 tx=0x7f1f44004ab0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.780+0000 7f1f52a8f640 1 -- 172.21.15.29:0/3066839361 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f4c10aaf0 msgr2=0x7f1f4c114ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:28.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.780+0000 7f1f52a8f640 1 --2- 172.21.15.29:0/3066839361 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f4c10aaf0 0x7f1f4c114ce0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1f4000e9c0 tx=0x7f1f4000ee80 comp rx=0 tx=0).stop 2024-03-11T19:14:28.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.782+0000 7f1f52a8f640 1 -- 172.21.15.29:0/3066839361 shutdown_connections 2024-03-11T19:14:28.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.782+0000 7f1f52a8f640 1 --2- 172.21.15.29:0/3066839361 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1f340a1bb0 0x7f1f340a4070 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.782+0000 7f1f52a8f640 1 --2- 172.21.15.29:0/3066839361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f4c10d4a0 0x7f1f4c1b6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.782+0000 7f1f52a8f640 1 --2- 172.21.15.29:0/3066839361 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f4c10aaf0 0x7f1f4c114ce0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.782+0000 7f1f52a8f640 1 --2- 172.21.15.29:0/3066839361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1f4c108180 0x7f1f4c1147a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.782+0000 7f1f52a8f640 1 -- 172.21.15.29:0/3066839361 >> 172.21.15.29:0/3066839361 conn(0x7f1f4c06dfe0 msgr2=0x7f1f4c1097f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:28.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.782+0000 7f1f52a8f640 1 -- 172.21.15.29:0/3066839361 shutdown_connections 2024-03-11T19:14:28.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.782+0000 7f1f52a8f640 1 -- 172.21.15.29:0/3066839361 wait complete. 2024-03-11T19:14:28.803 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:14:28.803 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:14:28.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.976+0000 7f6f2e1b9640 1 -- 172.21.15.29:0/1563910974 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f28109370 msgr2=0x7f6f2810fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:28.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.976+0000 7f6f2e1b9640 1 --2- 172.21.15.29:0/1563910974 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f28109370 0x7f6f2810fc00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6f20009be0 tx=0x7f6f2002bef0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.976+0000 7f6f2e1b9640 1 -- 172.21.15.29:0/1563910974 shutdown_connections 2024-03-11T19:14:28.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.976+0000 7f6f2e1b9640 1 --2- 172.21.15.29:0/1563910974 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f28109370 0x7f6f2810fc00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.976+0000 7f6f2e1b9640 1 --2- 172.21.15.29:0/1563910974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f281089b0 0x7f6f28108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.976+0000 7f6f2e1b9640 1 --2- 172.21.15.29:0/1563910974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f281077c0 0x7f6f28107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.976+0000 7f6f2e1b9640 1 -- 172.21.15.29:0/1563910974 >> 172.21.15.29:0/1563910974 conn(0x7f6f28076770 msgr2=0x7f6f28078b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:28.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.976+0000 7f6f2e1b9640 1 -- 172.21.15.29:0/1563910974 shutdown_connections 2024-03-11T19:14:28.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.976+0000 7f6f2e1b9640 1 -- 172.21.15.29:0/1563910974 wait complete. 2024-03-11T19:14:28.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.977+0000 7f6f2e1b9640 1 Processor -- start 2024-03-11T19:14:28.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.977+0000 7f6f2e1b9640 1 -- start start 2024-03-11T19:14:28.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.977+0000 7f6f2e1b9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f281077c0 0x7f6f2819d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.977+0000 7f6f2e1b9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f281089b0 0x7f6f2819dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.978+0000 7f6f2e1b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f28109370 0x7f6f281a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.978+0000 7f6f2e1b9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f281a5270 con 0x7f6f28109370 2024-03-11T19:14:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.978+0000 7f6f2e1b9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f281a53e0 con 0x7f6f281077c0 2024-03-11T19:14:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.978+0000 7f6f2e1b9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f281a55a0 con 0x7f6f281089b0 2024-03-11T19:14:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.978+0000 7f6f26ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f281089b0 0x7f6f2819dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.978+0000 7f6f277fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f281077c0 0x7f6f2819d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.978+0000 7f6f26ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f281089b0 0x7f6f2819dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41724/0 (socket says 172.21.15.29:41724) 2024-03-11T19:14:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.978+0000 7f6f26ffd640 1 -- 172.21.15.29:0/3946140826 learned_addr learned my addr 172.21.15.29:0/3946140826 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.978+0000 7f6f277fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f281077c0 0x7f6f2819d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37812/0 (socket says 172.21.15.29:37812) 2024-03-11T19:14:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.978+0000 7f6f27fff640 1 --2- 172.21.15.29:0/3946140826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f28109370 0x7f6f281a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.978+0000 7f6f26ffd640 1 -- 172.21.15.29:0/3946140826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f281077c0 msgr2=0x7f6f2819d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.978+0000 7f6f26ffd640 1 --2- 172.21.15.29:0/3946140826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f281077c0 0x7f6f2819d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.978+0000 7f6f26ffd640 1 -- 172.21.15.29:0/3946140826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f28109370 msgr2=0x7f6f281a4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:28.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.978+0000 7f6f26ffd640 1 --2- 172.21.15.29:0/3946140826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f28109370 0x7f6f281a4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.978+0000 7f6f26ffd640 1 -- 172.21.15.29:0/3946140826 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f1c009560 con 0x7f6f281089b0 2024-03-11T19:14:28.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.978+0000 7f6f27fff640 1 --2- 172.21.15.29:0/3946140826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f28109370 0x7f6f281a4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:14:28.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.979+0000 7f6f26ffd640 1 --2- 172.21.15.29:0/3946140826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f281089b0 0x7f6f2819dcc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c00e9c0 tx=0x7f6f1c00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:28.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.979+0000 7f6f24ff9640 1 -- 172.21.15.29:0/3946140826 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c00cd80 con 0x7f6f281089b0 2024-03-11T19:14:28.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.979+0000 7f6f2e1b9640 1 -- 172.21.15.29:0/3946140826 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f20009840 con 0x7f6f281089b0 2024-03-11T19:14:28.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.979+0000 7f6f24ff9640 1 -- 172.21.15.29:0/3946140826 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6f1c0044d0 con 0x7f6f281089b0 2024-03-11T19:14:28.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.979+0000 7f6f24ff9640 1 -- 172.21.15.29:0/3946140826 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c010430 con 0x7f6f281089b0 2024-03-11T19:14:28.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.980+0000 7f6f2e1b9640 1 -- 172.21.15.29:0/3946140826 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f281a5cd0 con 0x7f6f281089b0 2024-03-11T19:14:28.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.982+0000 7f6f24ff9640 1 -- 172.21.15.29:0/3946140826 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6f1c003f60 con 0x7f6f281089b0 2024-03-11T19:14:28.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.983+0000 7f6f24ff9640 1 --2- 172.21.15.29:0/3946140826 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6f100a1ab0 0x7f6f100a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.983+0000 7f6f30d83740 1 Processor -- start 2024-03-11T19:14:28.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.983+0000 7f6f30d83740 1 -- start start 2024-03-11T19:14:28.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.983+0000 7f6f277fe640 1 --2- 172.21.15.29:0/3946140826 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6f100a1ab0 0x7f6f100a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.983+0000 7f6f24ff9640 1 -- 172.21.15.29:0/3946140826 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f6f1c014070 con 0x7f6f281089b0 2024-03-11T19:14:28.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.984+0000 7f6f30d83740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928deb2d0 0x564928deb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.984+0000 7f6f30d83740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928debcc0 0x564928df07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.984+0000 7f6f30d83740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928df0e30 0x564928df32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.984+0000 7f6f30d83740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564928d5b280 con 0x564928df0e30 2024-03-11T19:14:28.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.984+0000 7f6f30d83740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564928d587b0 con 0x564928deb2d0 2024-03-11T19:14:28.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.984+0000 7f6f30d83740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x564928d53f90 con 0x564928debcc0 2024-03-11T19:14:28.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.984+0000 7f6f27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928df0e30 0x564928df32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.984+0000 7f6f27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928df0e30 0x564928df32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54842/0 (socket says 172.21.15.29:54842) 2024-03-11T19:14:28.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.984+0000 7f6f27fff640 1 -- 172.21.15.29:0/2163878424 learned_addr learned my addr 172.21.15.29:0/2163878424 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:28.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.984+0000 7f6f277fe640 1 --2- 172.21.15.29:0/3946140826 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6f100a1ab0 0x7f6f100a3f70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6f28108810 tx=0x7f6f18009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:28.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.984+0000 7f6f26ffd640 1 --2- 172.21.15.29:0/2163878424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928debcc0 0x564928df07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.984+0000 7f6f27fff640 1 -- 172.21.15.29:0/2163878424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928debcc0 msgr2=0x564928df07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:28.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.984+0000 7f6f27fff640 1 --2- 172.21.15.29:0/2163878424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928debcc0 0x564928df07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.985+0000 7f6f27fff640 1 -- 172.21.15.29:0/2163878424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928deb2d0 msgr2=0x564928deb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:28.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.985+0000 7f6f277fe640 1 --2- 172.21.15.29:0/2163878424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928deb2d0 0x564928deb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.985+0000 7f6f27fff640 1 --2- 172.21.15.29:0/2163878424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928deb2d0 0x564928deb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.985+0000 7f6f27fff640 1 -- 172.21.15.29:0/2163878424 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564928d12570 con 0x564928df0e30 2024-03-11T19:14:28.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.985+0000 7f6f277fe640 1 --2- 172.21.15.29:0/2163878424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928deb2d0 0x564928deb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:14:28.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.985+0000 7f6f27fff640 1 --2- 172.21.15.29:0/2163878424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928df0e30 0x564928df32c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f6f20034e90 tx=0x7f6f2002cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5632ee08fac2bd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:28.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.985+0000 7f6f0dffb640 1 -- 172.21.15.29:0/2163878424 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2002ca90 con 0x564928df0e30 2024-03-11T19:14:28.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.986+0000 7f6f0dffb640 1 -- 172.21.15.29:0/2163878424 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6f20003970 con 0x564928df0e30 2024-03-11T19:14:28.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.986+0000 7f6f0dffb640 1 -- 172.21.15.29:0/2163878424 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f20004590 con 0x564928df0e30 2024-03-11T19:14:28.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.986+0000 7f6f30d83740 1 -- 172.21.15.29:0/2163878424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928df0e30 msgr2=0x564928df32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:28.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.987+0000 7f6f30d83740 1 --2- 172.21.15.29:0/2163878424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928df0e30 0x564928df32c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f6f20034e90 tx=0x7f6f2002cd60 comp rx=0 tx=0).stop 2024-03-11T19:14:28.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.987+0000 7f6f30d83740 1 -- 172.21.15.29:0/2163878424 shutdown_connections 2024-03-11T19:14:28.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.987+0000 7f6f30d83740 1 --2- 172.21.15.29:0/2163878424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928df0e30 0x564928df32c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.987+0000 7f6f30d83740 1 --2- 172.21.15.29:0/2163878424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928debcc0 0x564928df07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.987+0000 7f6f30d83740 1 --2- 172.21.15.29:0/2163878424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928deb2d0 0x564928deb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.987+0000 7f6f30d83740 1 -- 172.21.15.29:0/2163878424 >> 172.21.15.29:0/2163878424 conn(0x564928de67f0 msgr2=0x564928de6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:28.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.987+0000 7f6f30d83740 1 -- 172.21.15.29:0/2163878424 shutdown_connections 2024-03-11T19:14:28.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.987+0000 7f6f30d83740 1 -- 172.21.15.29:0/2163878424 wait complete. 2024-03-11T19:14:28.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.987+0000 7f6f30d83740 1 -- start start 2024-03-11T19:14:28.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.988+0000 7f6f30d83740 1 Processor -- start 2024-03-11T19:14:28.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.988+0000 7f6f30d83740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928deb2d0 0x564928df5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.988+0000 7f6f30d83740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928debcc0 0x564928df5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.988+0000 7f6f277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928deb2d0 0x564928df5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.988+0000 7f6f277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928deb2d0 0x564928df5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54856/0 (socket says 172.21.15.29:54856) 2024-03-11T19:14:28.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.989+0000 7f6f277fe640 1 -- 172.21.15.29:0/2579874118 learned_addr learned my addr 172.21.15.29:0/2579874118 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:28.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.989+0000 7f6f30d83740 1 --2- 172.21.15.29:0/2579874118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928df0e30 0x564928dfc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.989+0000 7f6f26ffd640 1 --2- 172.21.15.29:0/2579874118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928debcc0 0x564928df5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.989+0000 7f6f30d83740 1 -- 172.21.15.29:0/2579874118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564928a27430 con 0x564928deb2d0 2024-03-11T19:14:28.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.989+0000 7f6f27fff640 1 --2- 172.21.15.29:0/2579874118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928df0e30 0x564928dfc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.989+0000 7f6f30d83740 1 -- 172.21.15.29:0/2579874118 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649289e3e00 con 0x564928debcc0 2024-03-11T19:14:28.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.989+0000 7f6f30d83740 1 -- 172.21.15.29:0/2579874118 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x564928cd8fd0 con 0x564928df0e30 2024-03-11T19:14:28.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.990+0000 7f6f30d83740 10 client.0 _ll_register_callbacks cb 0x564928d2efd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:14:28.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.990+0000 7f6f277fe640 1 -- 172.21.15.29:0/2579874118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928df0e30 msgr2=0x564928dfc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:28.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.990+0000 7f6f277fe640 1 --2- 172.21.15.29:0/2579874118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928df0e30 0x564928dfc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.990+0000 7f6f277fe640 1 -- 172.21.15.29:0/2579874118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928debcc0 msgr2=0x564928df5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:28.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.990+0000 7f6f277fe640 1 --2- 172.21.15.29:0/2579874118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928debcc0 0x564928df5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:28.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.990+0000 7f6f277fe640 1 -- 172.21.15.29:0/2579874118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f28110500 con 0x564928deb2d0 2024-03-11T19:14:28.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.990+0000 7f6f27fff640 1 --2- 172.21.15.29:0/2579874118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928df0e30 0x564928dfc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:14:28.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.991+0000 7f6f277fe640 1 --2- 172.21.15.29:0/2579874118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928deb2d0 0x564928df5220 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800efe0 tx=0x7f6f180047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902f19584797fc86 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:28.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.991+0000 7f6f06ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:14:28.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.991+0000 7f6f06ffd640 1 -- 172.21.15.29:0/2579874118 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f18020670 con 0x564928deb2d0 2024-03-11T19:14:28.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.991+0000 7f6f06ffd640 1 -- 172.21.15.29:0/2579874118 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f6f180207d0 con 0x564928deb2d0 2024-03-11T19:14:28.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.991+0000 7f6f26ffd640 1 --2- 172.21.15.29:0/2579874118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928debcc0 0x564928df5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:14:28.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.992+0000 7f6f30d83740 1 -- 172.21.15.29:0/2579874118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564928a537e0 con 0x564928deb2d0 2024-03-11T19:14:28.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.992+0000 7f6f06ffd640 1 -- 172.21.15.29:0/2579874118 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f18020990 con 0x564928deb2d0 2024-03-11T19:14:28.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.992+0000 7f6f30d83740 10 client.16644 fetch_fsmap learned FSMap version 27 2024-03-11T19:14:28.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.992+0000 7f6f30d83740 1 -- 172.21.15.29:0/2579874118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x564928d52610 con 0x564928deb2d0 2024-03-11T19:14:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.993+0000 7f6f06ffd640 1 -- 172.21.15.29:0/2579874118 <== mon.0 v2:172.21.15.29:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6f18011070 con 0x564928deb2d0 2024-03-11T19:14:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.993+0000 7f6f30d83740 10 client.16644 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:14:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.993+0000 7f6f30d83740 10 client.16644 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:14:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.993+0000 7f6f30d83740 20 client.16644 populate_metadata read hostname 'smithi029' 2024-03-11T19:14:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.993+0000 7f6f30d83740 1 --2- 172.21.15.29:0/2579874118 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564928e011d0 0x564928e03630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.993+0000 7f6f30d83740 4 client.16644 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:14:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.993+0000 7f6f30d83740 1 -- 172.21.15.29:0/2579874118 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564928a90720 con 0x564928e011d0 2024-03-11T19:14:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.993+0000 7f6f26ffd640 1 --2- 172.21.15.29:0/2579874118 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564928e011d0 0x564928e03630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:28.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.994+0000 7f6f26ffd640 1 --2- 172.21.15.29:0/2579874118 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564928e011d0 0x564928e03630 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b7d3d20e87de92a server_cookie=76420ed17caf3e9b in_seq=0 out_seq=0 2024-03-11T19:14:28.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.994+0000 7f6f06ffd640 10 client.16644 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:14:28.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.994+0000 7f6f06ffd640 1 -- 172.21.15.29:0/2579874118 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564928a90720 con 0x564928e011d0 2024-03-11T19:14:28.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.994+0000 7f6f06ffd640 10 client.16644 handle_command_reply: tid=0 2024-03-11T19:14:28.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.994+0000 7f6f30d83740 1 client.16644 shutdown 2024-03-11T19:14:28.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.995+0000 7f6f30d83740 1 -- 172.21.15.29:0/2579874118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928deb2d0 msgr2=0x564928df5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:28.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.995+0000 7f6f30d83740 1 --2- 172.21.15.29:0/2579874118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928deb2d0 0x564928df5220 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800efe0 tx=0x7f6f180047d0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.995+0000 7f6f30d83740 1 -- 172.21.15.29:0/2579874118 shutdown_connections 2024-03-11T19:14:29.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.995+0000 7f6f30d83740 1 --2- 172.21.15.29:0/2579874118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928df0e30 0x564928dfc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.995+0000 7f6f30d83740 1 --2- 172.21.15.29:0/2579874118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928debcc0 0x564928df5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.995+0000 7f6f30d83740 1 --2- 172.21.15.29:0/2579874118 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564928e011d0 0x564928e03630 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.995+0000 7f6f30d83740 1 --2- 172.21.15.29:0/2579874118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928deb2d0 0x564928df5220 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.995+0000 7f6f30d83740 1 -- 172.21.15.29:0/2579874118 >> 172.21.15.29:0/2579874118 conn(0x564928de67f0 msgr2=0x564928de5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:29.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.995+0000 7f6f06ffd640 0 client.16644 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:14:29.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.996+0000 7f6f30d83740 1 -- 172.21.15.29:0/2579874118 shutdown_connections 2024-03-11T19:14:29.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.996+0000 7f6f30d83740 1 -- 172.21.15.29:0/2579874118 wait complete. 2024-03-11T19:14:29.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:28.996+0000 7f6f30d83740 20 client.16644 trim_cache size 0 max 16384 2024-03-11T19:14:29.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.021+0000 7f6f30d83740 1 Processor -- start 2024-03-11T19:14:29.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.021+0000 7f6f30d83740 1 -- start start 2024-03-11T19:14:29.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.021+0000 7f6f30d83740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928deb2d0 0x564928dea1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:29.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.021+0000 7f6f30d83740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928debcc0 0x564928dfc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:29.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.021+0000 7f6f30d83740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928df0e30 0x564928df4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:29.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.021+0000 7f6f30d83740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564928a90720 con 0x564928df0e30 2024-03-11T19:14:29.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.021+0000 7f6f30d83740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564928df6410 con 0x564928debcc0 2024-03-11T19:14:29.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.021+0000 7f6f30d83740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x564928df6250 con 0x564928deb2d0 2024-03-11T19:14:29.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.021+0000 7f6f27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928df0e30 0x564928df4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:29.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.021+0000 7f6f27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928df0e30 0x564928df4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54872/0 (socket says 172.21.15.29:54872) 2024-03-11T19:14:29.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.021+0000 7f6f27fff640 1 -- 172.21.15.29:0/3956435034 learned_addr learned my addr 172.21.15.29:0/3956435034 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:29.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.022+0000 7f6f26ffd640 1 --2- 172.21.15.29:0/3956435034 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928debcc0 0x564928dfc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:29.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.022+0000 7f6f27fff640 1 -- 172.21.15.29:0/3956435034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928deb2d0 msgr2=0x564928dea1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:14:29.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.022+0000 7f6f27fff640 1 --2- 172.21.15.29:0/3956435034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928deb2d0 0x564928dea1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.022+0000 7f6f27fff640 1 -- 172.21.15.29:0/3956435034 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928debcc0 msgr2=0x564928dfc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:29.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.022+0000 7f6f27fff640 1 --2- 172.21.15.29:0/3956435034 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928debcc0 0x564928dfc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.022+0000 7f6f27fff640 1 -- 172.21.15.29:0/3956435034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564928ceeac0 con 0x564928df0e30 2024-03-11T19:14:29.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.022+0000 7f6f27fff640 1 --2- 172.21.15.29:0/3956435034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928df0e30 0x564928df4840 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6f20006010 tx=0x7f6f20005c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df0a7aa555cd999d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:29.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.022+0000 7f6f0d7fa640 1 -- 172.21.15.29:0/3956435034 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f20003ae0 con 0x564928df0e30 2024-03-11T19:14:29.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.022+0000 7f6f0d7fa640 1 -- 172.21.15.29:0/3956435034 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6f20003c40 con 0x564928df0e30 2024-03-11T19:14:29.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.022+0000 7f6f0d7fa640 1 -- 172.21.15.29:0/3956435034 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f20034c30 con 0x564928df0e30 2024-03-11T19:14:29.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.023+0000 7f6f30d83740 1 -- 172.21.15.29:0/3956435034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928df0e30 msgr2=0x564928df4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:29.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.023+0000 7f6f30d83740 1 --2- 172.21.15.29:0/3956435034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928df0e30 0x564928df4840 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6f20006010 tx=0x7f6f20005c80 comp rx=0 tx=0).stop 2024-03-11T19:14:29.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.023+0000 7f6f30d83740 1 -- 172.21.15.29:0/3956435034 shutdown_connections 2024-03-11T19:14:29.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.023+0000 7f6f30d83740 1 --2- 172.21.15.29:0/3956435034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928df0e30 0x564928df4840 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.024+0000 7f6f30d83740 1 --2- 172.21.15.29:0/3956435034 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928debcc0 0x564928dfc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.024+0000 7f6f30d83740 1 --2- 172.21.15.29:0/3956435034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928deb2d0 0x564928dea1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.024+0000 7f6f30d83740 1 -- 172.21.15.29:0/3956435034 >> 172.21.15.29:0/3956435034 conn(0x564928de67f0 msgr2=0x564928de9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:29.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.024+0000 7f6f30d83740 1 -- 172.21.15.29:0/3956435034 shutdown_connections 2024-03-11T19:14:29.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.024+0000 7f6f30d83740 1 -- 172.21.15.29:0/3956435034 wait complete. 2024-03-11T19:14:29.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.024+0000 7f6f30d83740 1 -- start start 2024-03-11T19:14:29.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.024+0000 7f6f30d83740 1 Processor -- start 2024-03-11T19:14:29.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.025+0000 7f6f30d83740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928deb2d0 0x564928dfb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:29.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.025+0000 7f6f30d83740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928debcc0 0x564928dfbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:29.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.025+0000 7f6f277fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928deb2d0 0x564928dfb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:29.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.025+0000 7f6f30d83740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928df0e30 0x564928e57a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:29.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.025+0000 7f6f277fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928deb2d0 0x564928dfb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:41770/0 (socket says 172.21.15.29:41770) 2024-03-11T19:14:29.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.025+0000 7f6f30d83740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564928e57f40 con 0x564928debcc0 2024-03-11T19:14:29.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.025+0000 7f6f30d83740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564928e58080 con 0x564928df0e30 2024-03-11T19:14:29.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.025+0000 7f6f30d83740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x564928e581f0 con 0x564928deb2d0 2024-03-11T19:14:29.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.025+0000 7f6f30d83740 10 client.0 _ll_register_callbacks cb 0x7f6f20009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:14:29.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.025+0000 7f6f26ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928debcc0 0x564928dfbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:29.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.025+0000 7f6f26ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928debcc0 0x564928dfbf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54884/0 (socket says 172.21.15.29:54884) 2024-03-11T19:14:29.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.025+0000 7f6f26ffd640 1 -- 172.21.15.29:0/2786636413 learned_addr learned my addr 172.21.15.29:0/2786636413 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:14:29.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.025+0000 7f6f27fff640 1 --2- 172.21.15.29:0/2786636413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928df0e30 0x564928e57a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:29.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.026+0000 7f6f27fff640 1 -- 172.21.15.29:0/2786636413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928deb2d0 msgr2=0x564928dfb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:29.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.026+0000 7f6f27fff640 1 --2- 172.21.15.29:0/2786636413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928deb2d0 0x564928dfb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.026+0000 7f6f27fff640 1 -- 172.21.15.29:0/2786636413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928debcc0 msgr2=0x564928dfbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:14:29.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.026+0000 7f6f27fff640 1 --2- 172.21.15.29:0/2786636413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928debcc0 0x564928dfbf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.026+0000 7f6f27fff640 1 -- 172.21.15.29:0/2786636413 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564928d12570 con 0x564928df0e30 2024-03-11T19:14:29.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.026+0000 7f6f27fff640 1 --2- 172.21.15.29:0/2786636413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928df0e30 0x564928e57a00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6f20009be0 tx=0x7f6f20034be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3a4583926aa04a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:14:29.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.026+0000 7f6f06ffd640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:14:29.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.026+0000 7f6f06ffd640 1 -- 172.21.15.29:0/2786636413 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f20003ae0 con 0x564928df0e30 2024-03-11T19:14:29.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.026+0000 7f6f06ffd640 1 -- 172.21.15.29:0/2786636413 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f6f20004000 con 0x564928df0e30 2024-03-11T19:14:29.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.027+0000 7f6f30d83740 1 -- 172.21.15.29:0/2786636413 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6f20003970 con 0x564928df0e30 2024-03-11T19:14:29.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.027+0000 7f6f06ffd640 1 -- 172.21.15.29:0/2786636413 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f20033e20 con 0x564928df0e30 2024-03-11T19:14:29.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.027+0000 7f6f30d83740 10 client.26168 fetch_fsmap learned FSMap version 27 2024-03-11T19:14:29.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.027+0000 7f6f30d83740 1 -- 172.21.15.29:0/2786636413 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x564928d146a0 con 0x564928df0e30 2024-03-11T19:14:29.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.027+0000 7f6f06ffd640 1 -- 172.21.15.29:0/2786636413 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6f20003970 con 0x564928df0e30 2024-03-11T19:14:29.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.028+0000 7f6f06ffd640 1 -- 172.21.15.29:0/2786636413 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6f20045020 con 0x564928df0e30 2024-03-11T19:14:29.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.028+0000 7f6f30d83740 10 client.26168 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:14:29.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.028+0000 7f6f30d83740 10 client.26168 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:14:29.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.028+0000 7f6f30d83740 20 client.26168 populate_metadata read hostname 'smithi029' 2024-03-11T19:14:29.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.028+0000 7f6f30d83740 1 --2- 172.21.15.29:0/2786636413 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564928e59770 0x564928e5bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:14:29.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.028+0000 7f6f30d83740 4 client.26168 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:14:29.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.028+0000 7f6f30d83740 1 -- 172.21.15.29:0/2786636413 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564928adf890 con 0x564928e59770 2024-03-11T19:14:29.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.028+0000 7f6f277fe640 1 --2- 172.21.15.29:0/2786636413 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564928e59770 0x564928e5bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:14:29.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.028+0000 7f6f277fe640 1 --2- 172.21.15.29:0/2786636413 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564928e59770 0x564928e5bb60 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=750936330b678409 server_cookie=12dc3ef5a7e22876 in_seq=0 out_seq=0 2024-03-11T19:14:29.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.029+0000 7f6f06ffd640 10 client.26168 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:14:29.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.029+0000 7f6f06ffd640 1 -- 172.21.15.29:0/2786636413 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x564928adf890 con 0x564928e59770 2024-03-11T19:14:29.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.029+0000 7f6f06ffd640 10 client.26168 handle_command_reply: tid=0 2024-03-11T19:14:29.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.029+0000 7f6f30d83740 1 client.26168 shutdown 2024-03-11T19:14:29.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.029+0000 7f6f30d83740 1 -- 172.21.15.29:0/2786636413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928df0e30 msgr2=0x564928e57a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:29.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.029+0000 7f6f30d83740 1 --2- 172.21.15.29:0/2786636413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928df0e30 0x564928e57a00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6f20009be0 tx=0x7f6f20034be0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.029+0000 7f6f30d83740 1 -- 172.21.15.29:0/2786636413 shutdown_connections 2024-03-11T19:14:29.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.029+0000 7f6f30d83740 1 --2- 172.21.15.29:0/2786636413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564928df0e30 0x564928e57a00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.029+0000 7f6f30d83740 1 --2- 172.21.15.29:0/2786636413 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564928e59770 0x564928e5bb60 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.029+0000 7f6f30d83740 1 --2- 172.21.15.29:0/2786636413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564928debcc0 0x564928dfbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.029+0000 7f6f30d83740 1 --2- 172.21.15.29:0/2786636413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564928deb2d0 0x564928dfb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.029+0000 7f6f30d83740 1 -- 172.21.15.29:0/2786636413 >> 172.21.15.29:0/2786636413 conn(0x564928de67f0 msgr2=0x564928dea330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:29.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.029+0000 7f6f06ffd640 0 client.26168 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:14:29.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.029+0000 7f6f30d83740 1 -- 172.21.15.29:0/2786636413 shutdown_connections 2024-03-11T19:14:29.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.029+0000 7f6f30d83740 1 -- 172.21.15.29:0/2786636413 wait complete. 2024-03-11T19:14:29.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.029+0000 7f6f30d83740 20 client.26168 trim_cache size 0 max 16384 2024-03-11T19:14:29.036 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:14:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:14:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:14:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "cdf4988e-c7d7-4a00-9d1c-6a125e257ccb": { 2024-03-11T19:14:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:14:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "cdf4988e-c7d7-4a00-9d1c-6a125e257ccb", 2024-03-11T19:14:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:14:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:14:29.037 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:14:29.037 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:14:29.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.030+0000 7f6f2e1b9640 1 -- 172.21.15.29:0/3946140826 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6f100a1ab0 msgr2=0x7f6f100a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.030+0000 7f6f2e1b9640 1 --2- 172.21.15.29:0/3946140826 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6f100a1ab0 0x7f6f100a3f70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6f28108810 tx=0x7f6f18009110 comp rx=0 tx=0).stop 2024-03-11T19:14:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.030+0000 7f6f2e1b9640 1 -- 172.21.15.29:0/3946140826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f281089b0 msgr2=0x7f6f2819dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:14:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.030+0000 7f6f2e1b9640 1 --2- 172.21.15.29:0/3946140826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f281089b0 0x7f6f2819dcc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c00e9c0 tx=0x7f6f1c00edc0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.030+0000 7f6f2e1b9640 1 -- 172.21.15.29:0/3946140826 shutdown_connections 2024-03-11T19:14:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.030+0000 7f6f2e1b9640 1 --2- 172.21.15.29:0/3946140826 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6f100a1ab0 0x7f6f100a3f70 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.030+0000 7f6f2e1b9640 1 --2- 172.21.15.29:0/3946140826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f28109370 0x7f6f281a4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.030+0000 7f6f2e1b9640 1 --2- 172.21.15.29:0/3946140826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f281089b0 0x7f6f2819dcc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.030+0000 7f6f2e1b9640 1 --2- 172.21.15.29:0/3946140826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f281077c0 0x7f6f2819d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:14:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.030+0000 7f6f2e1b9640 1 -- 172.21.15.29:0/3946140826 >> 172.21.15.29:0/3946140826 conn(0x7f6f28076770 msgr2=0x7f6f2810bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:14:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.030+0000 7f6f2e1b9640 1 -- 172.21.15.29:0/3946140826 shutdown_connections 2024-03-11T19:14:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:14:29.030+0000 7f6f2e1b9640 1 -- 172.21.15.29:0/3946140826 wait complete. 2024-03-11T19:14:29.051 INFO:tasks.cephfs.filesystem:scrub status for tag:cdf4988e-c7d7-4a00-9d1c-6a125e257ccb - {'path': '/', 'tag': 'cdf4988e-c7d7-4a00-9d1c-6a125e257ccb', 'options': 'recursive,force'} 2024-03-11T19:14:29.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:29 smithi029 ceph-mon[32439]: pgmap v491: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T19:14:30.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:29 smithi190 ceph-mon[36292]: pgmap v491: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T19:14:30.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:29 smithi170 ceph-mon[35690]: pgmap v491: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T19:14:30.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:30 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:14:30.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:30 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:14:30.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:30 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:14:30.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:30 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:14:31.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:30 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:14:31.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:30 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:14:31.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:30 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:14:31.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:30 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:14:31.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:30 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:14:31.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:30 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:14:31.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:30 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:14:31.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:30 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:14:31.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:31 smithi029 ceph-mon[32439]: pgmap v492: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:14:31.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:31 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:14:32.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:31 smithi190 ceph-mon[36292]: pgmap v492: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:14:32.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:31 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:14:32.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:31 smithi170 ceph-mon[35690]: pgmap v492: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:14:32.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:31 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:14:33.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:33 smithi029 ceph-mon[32439]: pgmap v493: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:14:34.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:33 smithi190 ceph-mon[36292]: pgmap v493: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:14:34.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:33 smithi170 ceph-mon[35690]: pgmap v493: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:14:35.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:35 smithi029 ceph-mon[32439]: pgmap v494: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-03-11T19:14:36.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:35 smithi190 ceph-mon[36292]: pgmap v494: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-03-11T19:14:36.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:35 smithi170 ceph-mon[35690]: pgmap v494: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-03-11T19:14:37.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:37 smithi029 ceph-mon[32439]: pgmap v495: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:14:38.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:37 smithi190 ceph-mon[36292]: pgmap v495: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:14:38.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:37 smithi170 ceph-mon[35690]: pgmap v495: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:14:39.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:39 smithi029 ceph-mon[32439]: pgmap v496: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-11T19:14:40.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:39 smithi190 ceph-mon[36292]: pgmap v496: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-11T19:14:40.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:39 smithi170 ceph-mon[35690]: pgmap v496: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-11T19:14:41.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:41 smithi029 ceph-mon[32439]: pgmap v497: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-11T19:14:42.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:41 smithi190 ceph-mon[36292]: pgmap v497: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-11T19:14:42.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:41 smithi170 ceph-mon[35690]: pgmap v497: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-11T19:14:43.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:43 smithi029 ceph-mon[32439]: pgmap v498: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-11T19:14:44.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:43 smithi190 ceph-mon[36292]: pgmap v498: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-11T19:14:44.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:43 smithi170 ceph-mon[35690]: pgmap v498: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-11T19:14:45.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:45 smithi029 ceph-mon[32439]: pgmap v499: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-11T19:14:46.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:45 smithi190 ceph-mon[36292]: pgmap v499: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-11T19:14:46.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:45 smithi170 ceph-mon[35690]: pgmap v499: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-11T19:14:47.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:47 smithi029 ceph-mon[32439]: pgmap v500: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-11T19:14:48.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:47 smithi190 ceph-mon[36292]: pgmap v500: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-11T19:14:48.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:47 smithi170 ceph-mon[35690]: pgmap v500: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-11T19:14:49.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:49 smithi029 ceph-mon[32439]: pgmap v501: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:14:50.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:49 smithi190 ceph-mon[36292]: pgmap v501: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:14:50.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:49 smithi170 ceph-mon[35690]: pgmap v501: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:14:51.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:51 smithi029 ceph-mon[32439]: pgmap v502: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T19:14:52.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:51 smithi190 ceph-mon[36292]: pgmap v502: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T19:14:52.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:51 smithi170 ceph-mon[35690]: pgmap v502: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T19:14:53.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:53 smithi029 ceph-mon[32439]: pgmap v503: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:14:54.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:53 smithi190 ceph-mon[36292]: pgmap v503: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:14:54.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:53 smithi170 ceph-mon[35690]: pgmap v503: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:14:55.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:54 smithi190 ceph-mon[36292]: pgmap v504: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-03-11T19:14:55.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:54 smithi170 ceph-mon[35690]: pgmap v504: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-03-11T19:14:55.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:54 smithi029 ceph-mon[32439]: pgmap v504: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-03-11T19:14:57.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:56 smithi170 ceph-mon[35690]: pgmap v505: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-11T19:14:57.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:56 smithi029 ceph-mon[32439]: pgmap v505: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-11T19:14:57.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:56 smithi190 ceph-mon[36292]: pgmap v505: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-11T19:14:59.051 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:14:59.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:14:59.064 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:14:58 smithi190 ceph-mon[36292]: pgmap v506: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T19:14:59.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:14:58 smithi170 ceph-mon[35690]: pgmap v506: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T19:14:59.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:14:58 smithi029 ceph-mon[32439]: pgmap v506: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T19:15:01.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:00 smithi170 ceph-mon[35690]: pgmap v507: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-11T19:15:01.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:00 smithi029 ceph-mon[32439]: pgmap v507: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-11T19:15:01.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:00 smithi190 ceph-mon[36292]: pgmap v507: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-11T19:15:03.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:02 smithi170 ceph-mon[35690]: pgmap v508: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-11T19:15:03.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:02 smithi029 ceph-mon[32439]: pgmap v508: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-11T19:15:03.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:02 smithi190 ceph-mon[36292]: pgmap v508: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-11T19:15:05.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:04 smithi170 ceph-mon[35690]: pgmap v509: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:15:05.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:04 smithi029 ceph-mon[32439]: pgmap v509: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:15:05.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:04 smithi190 ceph-mon[36292]: pgmap v509: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:15:07.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:06 smithi170 ceph-mon[35690]: pgmap v510: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-11T19:15:07.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:06 smithi029 ceph-mon[32439]: pgmap v510: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-11T19:15:07.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:06 smithi190 ceph-mon[36292]: pgmap v510: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-11T19:15:07.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.943+0000 7f5898901640 1 -- 172.21.15.29:0/264791895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f589410e3d0 msgr2=0x7f58941107e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:07.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.943+0000 7f5898901640 1 --2- 172.21.15.29:0/264791895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f589410e3d0 0x7f58941107e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f588c009be0 tx=0x7f588c02bef0 comp rx=0 tx=0).stop 2024-03-11T19:15:07.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.943+0000 7f5898901640 1 -- 172.21.15.29:0/264791895 shutdown_connections 2024-03-11T19:15:07.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.943+0000 7f5898901640 1 --2- 172.21.15.29:0/264791895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f589410e3d0 0x7f58941107e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:07.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.943+0000 7f5898901640 1 --2- 172.21.15.29:0/264791895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5894072a30 0x7f5894109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:07.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.943+0000 7f5898901640 1 --2- 172.21.15.29:0/264791895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5894072580 0x7f5894109500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:07.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.943+0000 7f5898901640 1 -- 172.21.15.29:0/264791895 >> 172.21.15.29:0/264791895 conn(0x7f589406dfa0 msgr2=0x7f58940703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:07.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.943+0000 7f5898901640 1 -- 172.21.15.29:0/264791895 shutdown_connections 2024-03-11T19:15:07.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.943+0000 7f5898901640 1 -- 172.21.15.29:0/264791895 wait complete. 2024-03-11T19:15:07.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.947+0000 7f5898901640 1 Processor -- start 2024-03-11T19:15:07.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.947+0000 7f5898901640 1 -- start start 2024-03-11T19:15:07.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.947+0000 7f5898901640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5894072580 0x7f589419e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:07.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.947+0000 7f5898901640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5894072a30 0x7f589419eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:07.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.947+0000 7f5898901640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f589410e3d0 0x7f58941a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:07.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.947+0000 7f5898901640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58941a60b0 con 0x7f5894072a30 2024-03-11T19:15:07.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.947+0000 7f5898901640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58941a6220 con 0x7f5894072580 2024-03-11T19:15:07.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.947+0000 7f5898901640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58941a63e0 con 0x7f589410e3d0 2024-03-11T19:15:07.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.947+0000 7f589259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5894072a30 0x7f589419eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:07.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.948+0000 7f589259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5894072a30 0x7f589419eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33460/0 (socket says 172.21.15.29:33460) 2024-03-11T19:15:07.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.948+0000 7f589359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f589410e3d0 0x7f58941a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:07.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.948+0000 7f589259c640 1 -- 172.21.15.29:0/1245659032 learned_addr learned my addr 172.21.15.29:0/1245659032 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:07.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.948+0000 7f5892d9d640 1 --2- 172.21.15.29:0/1245659032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5894072580 0x7f589419e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:07.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.948+0000 7f589359e640 1 -- 172.21.15.29:0/1245659032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5894072580 msgr2=0x7f589419e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:07.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.948+0000 7f589359e640 1 --2- 172.21.15.29:0/1245659032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5894072580 0x7f589419e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:07.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.948+0000 7f589359e640 1 -- 172.21.15.29:0/1245659032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5894072a30 msgr2=0x7f589419eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:07.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.948+0000 7f589359e640 1 --2- 172.21.15.29:0/1245659032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5894072a30 0x7f589419eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:07.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.948+0000 7f589359e640 1 -- 172.21.15.29:0/1245659032 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5884009490 con 0x7f589410e3d0 2024-03-11T19:15:07.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.948+0000 7f589259c640 1 --2- 172.21.15.29:0/1245659032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5894072a30 0x7f589419eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:15:07.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:07.948+0000 7f589359e640 1 --2- 172.21.15.29:0/1245659032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f589410e3d0 0x7f58941a5b70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f588c036040 tx=0x7f588c033cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:08.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.079+0000 7f587bfff640 1 -- 172.21.15.29:0/1245659032 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588c033e80 con 0x7f589410e3d0 2024-03-11T19:15:08.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.079+0000 7f587bfff640 1 -- 172.21.15.29:0/1245659032 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f588c003b80 con 0x7f589410e3d0 2024-03-11T19:15:08.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.079+0000 7f587bfff640 1 -- 172.21.15.29:0/1245659032 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588c034720 con 0x7f589410e3d0 2024-03-11T19:15:08.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.079+0000 7f5898901640 1 -- 172.21.15.29:0/1245659032 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588c009840 con 0x7f589410e3d0 2024-03-11T19:15:08.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.080+0000 7f5898901640 1 -- 172.21.15.29:0/1245659032 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5894108720 con 0x7f589410e3d0 2024-03-11T19:15:08.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.132+0000 7f587bfff640 1 -- 172.21.15.29:0/1245659032 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f588c034880 con 0x7f589410e3d0 2024-03-11T19:15:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.151+0000 7f587bfff640 1 --2- 172.21.15.29:0/1245659032 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f587c0a1bb0 0x7f587c0a4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.152+0000 7f587bfff640 1 -- 172.21.15.29:0/1245659032 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f588c0eff30 con 0x7f589410e3d0 2024-03-11T19:15:08.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.152+0000 7f5892d9d640 1 --2- 172.21.15.29:0/1245659032 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f587c0a1bb0 0x7f587c0a4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:08.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.153+0000 7f5892d9d640 1 --2- 172.21.15.29:0/1245659032 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f587c0a1bb0 0x7f587c0a4070 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5894073380 tx=0x7f5888009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.917+0000 7f589bdb4740 1 Processor -- start 2024-03-11T19:15:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.917+0000 7f589bdb4740 1 -- start start 2024-03-11T19:15:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.918+0000 7f589bdb4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dc4f0 0x55b7093dc910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.918+0000 7f589bdb4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dcee0 0x55b7093e19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.918+0000 7f589bdb4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 0x55b7093e44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.918+0000 7f589bdb4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b70934c280 con 0x55b7093dcee0 2024-03-11T19:15:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.918+0000 7f589bdb4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7093497b0 con 0x55b7093e2050 2024-03-11T19:15:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.918+0000 7f589bdb4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b709344f90 con 0x55b7093dc4f0 2024-03-11T19:15:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.918+0000 7f5892d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dc4f0 0x55b7093dc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:08.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.918+0000 7f589259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dcee0 0x55b7093e19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:08.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.918+0000 7f5892d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dc4f0 0x55b7093dc910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37354/0 (socket says 172.21.15.29:37354) 2024-03-11T19:15:08.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.918+0000 7f5892d9d640 1 -- 172.21.15.29:0/808754246 learned_addr learned my addr 172.21.15.29:0/808754246 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:08.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.918+0000 7f589359e640 1 --2- 172.21.15.29:0/808754246 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 0x55b7093e44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:08.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.918+0000 7f589259c640 1 -- 172.21.15.29:0/808754246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dc4f0 msgr2=0x55b7093dc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:08.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.918+0000 7f589259c640 1 --2- 172.21.15.29:0/808754246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dc4f0 0x55b7093dc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:08.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.918+0000 7f589259c640 1 -- 172.21.15.29:0/808754246 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 msgr2=0x55b7093e44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:08.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.918+0000 7f589259c640 1 --2- 172.21.15.29:0/808754246 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 0x55b7093e44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:08.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.918+0000 7f589259c640 1 -- 172.21.15.29:0/808754246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b709303570 con 0x55b7093dcee0 2024-03-11T19:15:08.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.919+0000 7f589259c640 1 --2- 172.21.15.29:0/808754246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dcee0 0x55b7093e19e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f5884009e30 tx=0x7f5884005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff9d9ad109cbcd65 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:08.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.919+0000 7f58797fa640 1 -- 172.21.15.29:0/808754246 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5884013040 con 0x55b7093dcee0 2024-03-11T19:15:08.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.919+0000 7f58797fa640 1 -- 172.21.15.29:0/808754246 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5884017ad0 con 0x55b7093dcee0 2024-03-11T19:15:08.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.919+0000 7f58797fa640 1 -- 172.21.15.29:0/808754246 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5884016b10 con 0x55b7093dcee0 2024-03-11T19:15:08.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.920+0000 7f589bdb4740 1 -- 172.21.15.29:0/808754246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dcee0 msgr2=0x55b7093e19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:08.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.920+0000 7f589bdb4740 1 --2- 172.21.15.29:0/808754246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dcee0 0x55b7093e19e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f5884009e30 tx=0x7f5884005f90 comp rx=0 tx=0).stop 2024-03-11T19:15:08.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.921+0000 7f589bdb4740 1 -- 172.21.15.29:0/808754246 shutdown_connections 2024-03-11T19:15:08.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.921+0000 7f589bdb4740 1 --2- 172.21.15.29:0/808754246 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 0x55b7093e44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:08.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.921+0000 7f589bdb4740 1 --2- 172.21.15.29:0/808754246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dcee0 0x55b7093e19e0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:08.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.921+0000 7f589bdb4740 1 --2- 172.21.15.29:0/808754246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dc4f0 0x55b7093dc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:08.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.921+0000 7f589bdb4740 1 -- 172.21.15.29:0/808754246 >> 172.21.15.29:0/808754246 conn(0x55b7093d7a60 msgr2=0x55b7093d7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:08.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.921+0000 7f589bdb4740 1 -- 172.21.15.29:0/808754246 shutdown_connections 2024-03-11T19:15:08.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:08.921+0000 7f589bdb4740 1 -- 172.21.15.29:0/808754246 wait complete. 2024-03-11T19:15:09.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:08 smithi170 ceph-mon[35690]: pgmap v511: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T19:15:09.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.101+0000 7f589bdb4740 1 -- start start 2024-03-11T19:15:09.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.101+0000 7f589bdb4740 1 Processor -- start 2024-03-11T19:15:09.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.102+0000 7f589bdb4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dc4f0 0x55b7093ded80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:09.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.102+0000 7f589bdb4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dcee0 0x55b7093df2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:09.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.102+0000 7f589bdb4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 0x55b7093e0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:09.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.102+0000 7f589bdb4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b709018430 con 0x55b7093dc4f0 2024-03-11T19:15:09.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.102+0000 7f589bdb4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b708fd4e00 con 0x55b7093e2050 2024-03-11T19:15:09.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.102+0000 7f589bdb4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7092c9fd0 con 0x55b7093dcee0 2024-03-11T19:15:09.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.102+0000 7f589259c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dcee0 0x55b7093df2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:09.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.102+0000 7f5892d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dc4f0 0x55b7093ded80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:09.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.102+0000 7f5892d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dc4f0 0x55b7093ded80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33478/0 (socket says 172.21.15.29:33478) 2024-03-11T19:15:09.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.102+0000 7f5892d9d640 1 -- 172.21.15.29:0/1819787431 learned_addr learned my addr 172.21.15.29:0/1819787431 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:09.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.102+0000 7f589359e640 1 --2- 172.21.15.29:0/1819787431 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 0x55b7093e0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:09.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.103+0000 7f5892d9d640 1 -- 172.21.15.29:0/1819787431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dcee0 msgr2=0x55b7093df2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:09.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.103+0000 7f5892d9d640 1 --2- 172.21.15.29:0/1819787431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dcee0 0x55b7093df2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.103+0000 7f5892d9d640 1 -- 172.21.15.29:0/1819787431 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 msgr2=0x55b7093e0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:09.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.103+0000 7f5892d9d640 1 --2- 172.21.15.29:0/1819787431 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 0x55b7093e0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.117+0000 7f589bdb4740 10 client.0 _ll_register_callbacks cb 0x55b70931ffd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:09.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.118+0000 7f5892d9d640 1 -- 172.21.15.29:0/1819787431 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5884009a90 con 0x55b7093dc4f0 2024-03-11T19:15:09.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.118+0000 7f5892d9d640 1 --2- 172.21.15.29:0/1819787431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dc4f0 0x55b7093ded80 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f5888012c20 tx=0x7f5888004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6572de031af9a90 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:09.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.118+0000 7f5876ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:15:09.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.119+0000 7f5876ff5640 1 -- 172.21.15.29:0/1819787431 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5888029640 con 0x55b7093dc4f0 2024-03-11T19:15:09.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.119+0000 7f5876ff5640 1 -- 172.21.15.29:0/1819787431 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5888018070 con 0x55b7093dc4f0 2024-03-11T19:15:09.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:08 smithi029 ceph-mon[32439]: pgmap v511: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T19:15:09.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:08 smithi190 ceph-mon[36292]: pgmap v511: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T19:15:09.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.260+0000 7f589bdb4740 1 -- 172.21.15.29:0/1819787431 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b7090447e0 con 0x55b7093dc4f0 2024-03-11T19:15:09.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.260+0000 7f5876ff5640 1 -- 172.21.15.29:0/1819787431 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5888012e60 con 0x55b7093dc4f0 2024-03-11T19:15:09.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.261+0000 7f589bdb4740 10 client.16662 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:09.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.261+0000 7f589bdb4740 1 -- 172.21.15.29:0/1819787431 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x55b709343610 con 0x55b7093dc4f0 2024-03-11T19:15:09.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.261+0000 7f5876ff5640 1 -- 172.21.15.29:0/1819787431 <== mon.0 v2:172.21.15.29:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f58880297a0 con 0x55b7093dc4f0 2024-03-11T19:15:09.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.261+0000 7f589bdb4740 10 client.16662 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:09.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.261+0000 7f589bdb4740 10 client.16662 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:15:09.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.261+0000 7f589bdb4740 20 client.16662 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:09.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.261+0000 7f589bdb4740 1 --2- 172.21.15.29:0/1819787431 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55b7093fac80 0x55b7093fd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:09.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.261+0000 7f589bdb4740 4 client.16662 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:15:09.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.261+0000 7f589bdb4740 1 -- 172.21.15.29:0/1819787431 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b709081720 con 0x55b7093fac80 2024-03-11T19:15:09.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.262+0000 7f589259c640 1 --2- 172.21.15.29:0/1819787431 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55b7093fac80 0x55b7093fd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:09.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.262+0000 7f5876ff5640 10 client.16662 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:09.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.262+0000 7f589259c640 1 --2- 172.21.15.29:0/1819787431 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55b7093fac80 0x55b7093fd140 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2882a5e690db133 server_cookie=c718e6124af113ea in_seq=0 out_seq=0 2024-03-11T19:15:09.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.263+0000 7f5876ff5640 1 -- 172.21.15.29:0/1819787431 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b709081720 con 0x55b7093fac80 2024-03-11T19:15:09.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.263+0000 7f5876ff5640 10 client.16662 handle_command_reply: tid=0 2024-03-11T19:15:09.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.263+0000 7f589bdb4740 1 client.16662 shutdown 2024-03-11T19:15:09.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.263+0000 7f589bdb4740 1 -- 172.21.15.29:0/1819787431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dc4f0 msgr2=0x55b7093ded80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:09.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.263+0000 7f589bdb4740 1 --2- 172.21.15.29:0/1819787431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dc4f0 0x55b7093ded80 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f5888012c20 tx=0x7f5888004730 comp rx=0 tx=0).stop 2024-03-11T19:15:09.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.263+0000 7f589bdb4740 1 -- 172.21.15.29:0/1819787431 shutdown_connections 2024-03-11T19:15:09.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.263+0000 7f589bdb4740 1 --2- 172.21.15.29:0/1819787431 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 0x55b7093e0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.263+0000 7f589bdb4740 1 --2- 172.21.15.29:0/1819787431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dcee0 0x55b7093df2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.263+0000 7f589bdb4740 1 --2- 172.21.15.29:0/1819787431 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55b7093fac80 0x55b7093fd140 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.263+0000 7f589bdb4740 1 --2- 172.21.15.29:0/1819787431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dc4f0 0x55b7093ded80 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.263+0000 7f5876ff5640 0 client.16662 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:09.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.263+0000 7f589bdb4740 1 -- 172.21.15.29:0/1819787431 >> 172.21.15.29:0/1819787431 conn(0x55b7093d7a60 msgr2=0x55b7093d6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:09.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.263+0000 7f589bdb4740 1 -- 172.21.15.29:0/1819787431 shutdown_connections 2024-03-11T19:15:09.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.263+0000 7f589bdb4740 1 -- 172.21.15.29:0/1819787431 wait complete. 2024-03-11T19:15:09.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.263+0000 7f589bdb4740 20 client.16662 trim_cache size 0 max 16384 2024-03-11T19:15:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.293+0000 7f589bdb4740 1 Processor -- start 2024-03-11T19:15:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.293+0000 7f589bdb4740 1 -- start start 2024-03-11T19:15:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.293+0000 7f589bdb4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dc4f0 0x55b7093e54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.293+0000 7f589bdb4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dcee0 0x55b7093e5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.293+0000 7f589bdb4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 0x55b7093d64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.293+0000 7f589bdb4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7093dd310 con 0x55b7093dc4f0 2024-03-11T19:15:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.293+0000 7f589bdb4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7093d87d0 con 0x55b7093e2050 2024-03-11T19:15:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.293+0000 7f589bdb4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7093d3260 con 0x55b7093dcee0 2024-03-11T19:15:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.293+0000 7f5892d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dc4f0 0x55b7093e54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.293+0000 7f5892d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dc4f0 0x55b7093e54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33480/0 (socket says 172.21.15.29:33480) 2024-03-11T19:15:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.293+0000 7f5892d9d640 1 -- 172.21.15.29:0/3325920223 learned_addr learned my addr 172.21.15.29:0/3325920223 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.293+0000 7f589259c640 1 --2- 172.21.15.29:0/3325920223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dcee0 0x55b7093e5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.293+0000 7f589359e640 1 --2- 172.21.15.29:0/3325920223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 0x55b7093d64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.294+0000 7f5892d9d640 1 -- 172.21.15.29:0/3325920223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dcee0 msgr2=0x55b7093e5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.294+0000 7f5892d9d640 1 --2- 172.21.15.29:0/3325920223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dcee0 0x55b7093e5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.294+0000 7f5892d9d640 1 -- 172.21.15.29:0/3325920223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 msgr2=0x55b7093d64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.294+0000 7f5892d9d640 1 --2- 172.21.15.29:0/3325920223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 0x55b7093d64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.294+0000 7f5892d9d640 1 -- 172.21.15.29:0/3325920223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b7092dfac0 con 0x55b7093dc4f0 2024-03-11T19:15:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.294+0000 7f5892d9d640 1 --2- 172.21.15.29:0/3325920223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dc4f0 0x55b7093e54c0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f5888005a30 tx=0x7f5888022040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a27a8f83aee07a5d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.294+0000 7f5878ff9640 1 -- 172.21.15.29:0/3325920223 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5888011cd0 con 0x55b7093dc4f0 2024-03-11T19:15:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.295+0000 7f5878ff9640 1 -- 172.21.15.29:0/3325920223 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55b7090447e0 con 0x55b7093dc4f0 2024-03-11T19:15:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.295+0000 7f5878ff9640 1 -- 172.21.15.29:0/3325920223 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5888005530 con 0x55b7093dc4f0 2024-03-11T19:15:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.296+0000 7f589bdb4740 1 -- 172.21.15.29:0/3325920223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dc4f0 msgr2=0x55b7093e54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.296+0000 7f589bdb4740 1 --2- 172.21.15.29:0/3325920223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dc4f0 0x55b7093e54c0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f5888005a30 tx=0x7f5888022040 comp rx=0 tx=0).stop 2024-03-11T19:15:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.296+0000 7f589bdb4740 1 -- 172.21.15.29:0/3325920223 shutdown_connections 2024-03-11T19:15:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.296+0000 7f589bdb4740 1 --2- 172.21.15.29:0/3325920223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 0x55b7093d64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.296+0000 7f589bdb4740 1 --2- 172.21.15.29:0/3325920223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dcee0 0x55b7093e5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.297+0000 7f589bdb4740 1 --2- 172.21.15.29:0/3325920223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dc4f0 0x55b7093e54c0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.297+0000 7f589bdb4740 1 -- 172.21.15.29:0/3325920223 >> 172.21.15.29:0/3325920223 conn(0x55b7093d7a60 msgr2=0x55b7093d7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.297+0000 7f589bdb4740 1 -- 172.21.15.29:0/3325920223 shutdown_connections 2024-03-11T19:15:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.297+0000 7f589bdb4740 1 -- 172.21.15.29:0/3325920223 wait complete. 2024-03-11T19:15:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.298+0000 7f589bdb4740 1 -- start start 2024-03-11T19:15:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.298+0000 7f589bdb4740 1 Processor -- start 2024-03-11T19:15:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.298+0000 7f589bdb4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dc4f0 0x55b7093f6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.298+0000 7f589bdb4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dcee0 0x55b7093f6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.298+0000 7f589bdb4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 0x55b7093f6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.298+0000 7f589bdb4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b709451250 con 0x55b7093dcee0 2024-03-11T19:15:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.298+0000 7f589bdb4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b709451390 con 0x55b7093e2050 2024-03-11T19:15:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.298+0000 7f589bdb4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7094514d0 con 0x55b7093dc4f0 2024-03-11T19:15:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.298+0000 7f589359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 0x55b7093f6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.299+0000 7f589359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 0x55b7093f6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:53904/0 (socket says 172.21.15.29:53904) 2024-03-11T19:15:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.299+0000 7f589359e640 1 -- 172.21.15.29:0/2397487801 learned_addr learned my addr 172.21.15.29:0/2397487801 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.299+0000 7f5892d9d640 1 --2- 172.21.15.29:0/2397487801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dc4f0 0x55b7093f6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.299+0000 7f589359e640 1 -- 172.21.15.29:0/2397487801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dc4f0 msgr2=0x55b7093f6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.299+0000 7f589359e640 1 --2- 172.21.15.29:0/2397487801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dc4f0 0x55b7093f6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.299+0000 7f589359e640 1 -- 172.21.15.29:0/2397487801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dcee0 msgr2=0x55b7093f6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.299+0000 7f589359e640 1 --2- 172.21.15.29:0/2397487801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dcee0 0x55b7093f6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.299+0000 7f589359e640 1 -- 172.21.15.29:0/2397487801 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b709343610 con 0x55b7093e2050 2024-03-11T19:15:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.299+0000 7f589bdb4740 10 client.0 _ll_register_callbacks cb 0x7f5884009800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.299+0000 7f589bdb4740 1 -- 172.21.15.29:0/2397487801 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5884017ad0 con 0x55b7093e2050 2024-03-11T19:15:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.299+0000 7f589359e640 1 --2- 172.21.15.29:0/2397487801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 0x55b7093f6cb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f588c0b92c0 tx=0x7f588c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e418620e693d3e26 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.299+0000 7f5876ff5640 10 client.26186 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:15:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.300+0000 7f5876ff5640 1 -- 172.21.15.29:0/2397487801 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588c0d9070 con 0x55b7093e2050 2024-03-11T19:15:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.300+0000 7f5876ff5640 1 -- 172.21.15.29:0/2397487801 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f588c0bcf40 con 0x55b7093e2050 2024-03-11T19:15:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.300+0000 7f5876ff5640 1 -- 172.21.15.29:0/2397487801 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f588c0c1090 con 0x55b7093e2050 2024-03-11T19:15:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.300+0000 7f589bdb4740 10 client.26186 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.300+0000 7f589bdb4740 1 -- 172.21.15.29:0/2397487801 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x55b7093056a0 con 0x55b7093e2050 2024-03-11T19:15:09.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.300+0000 7f5876ff5640 1 -- 172.21.15.29:0/2397487801 <== mon.2 v2:172.21.15.170:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f588c050060 con 0x55b7093e2050 2024-03-11T19:15:09.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.300+0000 7f589bdb4740 10 client.26186 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:09.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.300+0000 7f589bdb4740 10 client.26186 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:15:09.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.300+0000 7f589bdb4740 20 client.26186 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:09.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.300+0000 7f589bdb4740 1 --2- 172.21.15.29:0/2397487801 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55b709452b00 0x55b709454f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:09.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.301+0000 7f589bdb4740 4 client.26186 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:15:09.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.301+0000 7f589bdb4740 1 -- 172.21.15.29:0/2397487801 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b7090d0890 con 0x55b709452b00 2024-03-11T19:15:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.301+0000 7f5892d9d640 1 --2- 172.21.15.29:0/2397487801 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55b709452b00 0x55b709454f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.301+0000 7f5876ff5640 10 client.26186 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.301+0000 7f5892d9d640 1 --2- 172.21.15.29:0/2397487801 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55b709452b00 0x55b709454f40 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36f44621e280677f server_cookie=7798d7df48a78807 in_seq=0 out_seq=0 2024-03-11T19:15:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.302+0000 7f5876ff5640 1 -- 172.21.15.29:0/2397487801 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55b7090d0890 con 0x55b709452b00 2024-03-11T19:15:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.302+0000 7f5876ff5640 10 client.26186 handle_command_reply: tid=0 2024-03-11T19:15:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.302+0000 7f589bdb4740 1 client.26186 shutdown 2024-03-11T19:15:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.302+0000 7f589bdb4740 1 -- 172.21.15.29:0/2397487801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 msgr2=0x55b7093f6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.302+0000 7f589bdb4740 1 --2- 172.21.15.29:0/2397487801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 0x55b7093f6cb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f588c0b92c0 tx=0x7f588c004a40 comp rx=0 tx=0).stop 2024-03-11T19:15:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.302+0000 7f589bdb4740 1 -- 172.21.15.29:0/2397487801 shutdown_connections 2024-03-11T19:15:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.302+0000 7f589bdb4740 1 --2- 172.21.15.29:0/2397487801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b7093e2050 0x55b7093f6cb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.302+0000 7f589bdb4740 1 --2- 172.21.15.29:0/2397487801 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55b709452b00 0x55b709454f40 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.302+0000 7f589bdb4740 1 --2- 172.21.15.29:0/2397487801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b7093dcee0 0x55b7093f6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.302+0000 7f5876ff5640 0 client.26186 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:09.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.302+0000 7f589bdb4740 1 --2- 172.21.15.29:0/2397487801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b7093dc4f0 0x55b7093f6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.302+0000 7f589bdb4740 1 -- 172.21.15.29:0/2397487801 >> 172.21.15.29:0/2397487801 conn(0x55b7093d7a60 msgr2=0x55b7093e54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:09.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.302+0000 7f589bdb4740 1 -- 172.21.15.29:0/2397487801 shutdown_connections 2024-03-11T19:15:09.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.302+0000 7f589bdb4740 1 -- 172.21.15.29:0/2397487801 wait complete. 2024-03-11T19:15:09.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.302+0000 7f589bdb4740 20 client.26186 trim_cache size 0 max 16384 2024-03-11T19:15:09.308 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:15:09.308 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:15:09.308 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:15:09.308 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:15:09.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.303+0000 7f5898901640 1 -- 172.21.15.29:0/1245659032 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f587c0a1bb0 msgr2=0x7f587c0a4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:09.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.303+0000 7f5898901640 1 --2- 172.21.15.29:0/1245659032 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f587c0a1bb0 0x7f587c0a4070 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5894073380 tx=0x7f5888009260 comp rx=0 tx=0).stop 2024-03-11T19:15:09.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.303+0000 7f5898901640 1 -- 172.21.15.29:0/1245659032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f589410e3d0 msgr2=0x7f58941a5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:09.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.303+0000 7f5898901640 1 --2- 172.21.15.29:0/1245659032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f589410e3d0 0x7f58941a5b70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f588c036040 tx=0x7f588c033cd0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.303+0000 7f5898901640 1 -- 172.21.15.29:0/1245659032 shutdown_connections 2024-03-11T19:15:09.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.303+0000 7f5898901640 1 --2- 172.21.15.29:0/1245659032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f589410e3d0 0x7f58941a5b70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.303+0000 7f5898901640 1 --2- 172.21.15.29:0/1245659032 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f587c0a1bb0 0x7f587c0a4070 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.303+0000 7f5898901640 1 --2- 172.21.15.29:0/1245659032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5894072a30 0x7f589419eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.303+0000 7f5898901640 1 --2- 172.21.15.29:0/1245659032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5894072580 0x7f589419e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:09.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.303+0000 7f5898901640 1 -- 172.21.15.29:0/1245659032 >> 172.21.15.29:0/1245659032 conn(0x7f589406dfa0 msgr2=0x7f589410a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:09.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.304+0000 7f5898901640 1 -- 172.21.15.29:0/1245659032 shutdown_connections 2024-03-11T19:15:09.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:09.304+0000 7f5898901640 1 -- 172.21.15.29:0/1245659032 wait complete. 2024-03-11T19:15:09.664 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:15:09.665 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:15:11.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:10 smithi170 ceph-mon[35690]: pgmap v512: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:15:11.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:10 smithi029 ceph-mon[32439]: pgmap v512: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:15:11.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:10 smithi190 ceph-mon[36292]: pgmap v512: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T19:15:11.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.405+0000 7f999afe2640 1 -- 172.21.15.29:0/728592353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9994106ee0 msgr2=0x7f99941092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:11.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.405+0000 7f999afe2640 1 --2- 172.21.15.29:0/728592353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9994106ee0 0x7f99941092d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f998c009ca0 tx=0x7f998c02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:15:11.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.405+0000 7f999afe2640 1 -- 172.21.15.29:0/728592353 shutdown_connections 2024-03-11T19:15:11.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.405+0000 7f999afe2640 1 --2- 172.21.15.29:0/728592353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9994106ee0 0x7f99941092d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:11.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.405+0000 7f999afe2640 1 --2- 172.21.15.29:0/728592353 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99941045b0 0x7f99941069a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:11.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.405+0000 7f999afe2640 1 --2- 172.21.15.29:0/728592353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9994101c80 0x7f9994104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:11.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.405+0000 7f999afe2640 1 -- 172.21.15.29:0/728592353 >> 172.21.15.29:0/728592353 conn(0x7f99940fb610 msgr2=0x7f99940fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:11.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.405+0000 7f999afe2640 1 -- 172.21.15.29:0/728592353 shutdown_connections 2024-03-11T19:15:11.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.406+0000 7f999afe2640 1 -- 172.21.15.29:0/728592353 wait complete. 2024-03-11T19:15:11.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.406+0000 7f999afe2640 1 Processor -- start 2024-03-11T19:15:11.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.406+0000 7f999afe2640 1 -- start start 2024-03-11T19:15:11.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.406+0000 7f999afe2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9994101c80 0x7f999419d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:11.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.407+0000 7f999afe2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99941045b0 0x7f999419dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.407+0000 7f999afe2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9994106ee0 0x7f99941a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.407+0000 7f999afe2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99941a5230 con 0x7f9994101c80 2024-03-11T19:15:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.407+0000 7f999afe2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99941a53a0 con 0x7f9994106ee0 2024-03-11T19:15:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.407+0000 7f999afe2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99941a5560 con 0x7f99941045b0 2024-03-11T19:15:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.407+0000 7f999957f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9994106ee0 0x7f99941a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.407+0000 7f999957f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9994106ee0 0x7f99941a4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:53916/0 (socket says 172.21.15.29:53916) 2024-03-11T19:15:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.407+0000 7f9998d7e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9994101c80 0x7f999419d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.407+0000 7f999957f640 1 -- 172.21.15.29:0/2880691922 learned_addr learned my addr 172.21.15.29:0/2880691922 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.407+0000 7f9993fff640 1 --2- 172.21.15.29:0/2880691922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99941045b0 0x7f999419dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.407+0000 7f9993fff640 1 -- 172.21.15.29:0/2880691922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9994106ee0 msgr2=0x7f99941a4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.407+0000 7f9993fff640 1 --2- 172.21.15.29:0/2880691922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9994106ee0 0x7f99941a4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.408+0000 7f9993fff640 1 -- 172.21.15.29:0/2880691922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9994101c80 msgr2=0x7f999419d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.408+0000 7f9993fff640 1 --2- 172.21.15.29:0/2880691922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9994101c80 0x7f999419d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.408+0000 7f9993fff640 1 -- 172.21.15.29:0/2880691922 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998c009900 con 0x7f99941045b0 2024-03-11T19:15:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.408+0000 7f9993fff640 1 --2- 172.21.15.29:0/2880691922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99941045b0 0x7f999419dca0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f998800e9c0 tx=0x7f998800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.408+0000 7f9991ffb640 1 -- 172.21.15.29:0/2880691922 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998800ccc0 con 0x7f99941045b0 2024-03-11T19:15:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.408+0000 7f999afe2640 1 -- 172.21.15.29:0/2880691922 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99941a5840 con 0x7f99941045b0 2024-03-11T19:15:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.408+0000 7f9991ffb640 1 -- 172.21.15.29:0/2880691922 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f998800ce20 con 0x7f99941045b0 2024-03-11T19:15:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.408+0000 7f9991ffb640 1 -- 172.21.15.29:0/2880691922 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9988018710 con 0x7f99941045b0 2024-03-11T19:15:11.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.409+0000 7f999afe2640 1 -- 172.21.15.29:0/2880691922 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99941a5da0 con 0x7f99941045b0 2024-03-11T19:15:11.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.410+0000 7f999afe2640 1 -- 172.21.15.29:0/2880691922 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9994100bf0 con 0x7f99941045b0 2024-03-11T19:15:11.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.412+0000 7f9991ffb640 1 -- 172.21.15.29:0/2880691922 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9988010430 con 0x7f99941045b0 2024-03-11T19:15:11.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.412+0000 7f9991ffb640 1 --2- 172.21.15.29:0/2880691922 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f997c0a1b30 0x7f997c0a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:11.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.413+0000 7f9991ffb640 1 -- 172.21.15.29:0/2880691922 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9988014070 con 0x7f99941045b0 2024-03-11T19:15:11.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.413+0000 7f9998d7e640 1 --2- 172.21.15.29:0/2880691922 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f997c0a1b30 0x7f997c0a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:11.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.414+0000 7f9998d7e640 1 --2- 172.21.15.29:0/2880691922 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f997c0a1b30 0x7f997c0a3ff0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f999419ec20 tx=0x7f9984005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:11.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.420+0000 7f9991ffb640 1 -- 172.21.15.29:0/2880691922 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9988052d20 con 0x7f99941045b0 2024-03-11T19:15:11.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.836+0000 7f999afe2640 1 -- 172.21.15.29:0/2880691922 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9994100d80 con 0x7f99941045b0 2024-03-11T19:15:11.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.837+0000 7f9991ffb640 1 -- 172.21.15.29:0/2880691922 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f998809a530 con 0x7f99941045b0 2024-03-11T19:15:11.841 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:15:11.842 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:15:11.842 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:15:11.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.842+0000 7f999afe2640 1 -- 172.21.15.29:0/2880691922 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f997c0a1b30 msgr2=0x7f997c0a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:11.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.842+0000 7f999afe2640 1 --2- 172.21.15.29:0/2880691922 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f997c0a1b30 0x7f997c0a3ff0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f999419ec20 tx=0x7f9984005f90 comp rx=0 tx=0).stop 2024-03-11T19:15:11.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.842+0000 7f999afe2640 1 -- 172.21.15.29:0/2880691922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99941045b0 msgr2=0x7f999419dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:11.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.842+0000 7f999afe2640 1 --2- 172.21.15.29:0/2880691922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99941045b0 0x7f999419dca0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f998800e9c0 tx=0x7f998800ee80 comp rx=0 tx=0).stop 2024-03-11T19:15:11.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.842+0000 7f999afe2640 1 -- 172.21.15.29:0/2880691922 shutdown_connections 2024-03-11T19:15:11.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.842+0000 7f999afe2640 1 --2- 172.21.15.29:0/2880691922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9994106ee0 0x7f99941a4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:11.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.842+0000 7f999afe2640 1 --2- 172.21.15.29:0/2880691922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99941045b0 0x7f999419dca0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:11.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.842+0000 7f999afe2640 1 --2- 172.21.15.29:0/2880691922 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f997c0a1b30 0x7f997c0a3ff0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:11.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.842+0000 7f999afe2640 1 --2- 172.21.15.29:0/2880691922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9994101c80 0x7f999419d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:11.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.842+0000 7f999afe2640 1 -- 172.21.15.29:0/2880691922 >> 172.21.15.29:0/2880691922 conn(0x7f99940fb610 msgr2=0x7f99941072c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:11.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.842+0000 7f999afe2640 1 -- 172.21.15.29:0/2880691922 shutdown_connections 2024-03-11T19:15:11.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:11.842+0000 7f999afe2640 1 -- 172.21.15.29:0/2880691922 wait complete. 2024-03-11T19:15:11.921 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:15:11.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:15:12.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.389+0000 7f3c497d0640 1 -- 172.21.15.29:0/333134527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c44108990 msgr2=0x7f3c44108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:12.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.389+0000 7f3c497d0640 1 --2- 172.21.15.29:0/333134527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c44108990 0x7f3c44108e10 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f3c38009900 tx=0x7f3c3802bc30 comp rx=0 tx=0).stop 2024-03-11T19:15:12.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.389+0000 7f3c497d0640 1 -- 172.21.15.29:0/333134527 shutdown_connections 2024-03-11T19:15:12.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.389+0000 7f3c497d0640 1 --2- 172.21.15.29:0/333134527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c44109350 0x7f3c4410fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.389+0000 7f3c497d0640 1 --2- 172.21.15.29:0/333134527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c44108990 0x7f3c44108e10 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.389+0000 7f3c497d0640 1 --2- 172.21.15.29:0/333134527 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c441077a0 0x7f3c44107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.389+0000 7f3c497d0640 1 -- 172.21.15.29:0/333134527 >> 172.21.15.29:0/333134527 conn(0x7f3c44076730 msgr2=0x7f3c44078b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:12.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.389+0000 7f3c497d0640 1 -- 172.21.15.29:0/333134527 shutdown_connections 2024-03-11T19:15:12.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.390+0000 7f3c497d0640 1 -- 172.21.15.29:0/333134527 wait complete. 2024-03-11T19:15:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.390+0000 7f3c497d0640 1 Processor -- start 2024-03-11T19:15:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.391+0000 7f3c497d0640 1 -- start start 2024-03-11T19:15:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.391+0000 7f3c497d0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c441077a0 0x7f3c4419d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.391+0000 7f3c497d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c44108990 0x7f3c4419dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.391+0000 7f3c497d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c44109350 0x7f3c441a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.391+0000 7f3c497d0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c441a52a0 con 0x7f3c44108990 2024-03-11T19:15:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.391+0000 7f3c497d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c441a5410 con 0x7f3c44109350 2024-03-11T19:15:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.391+0000 7f3c497d0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c441a55d0 con 0x7f3c441077a0 2024-03-11T19:15:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.391+0000 7f3c427fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c44108990 0x7f3c4419dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.391+0000 7f3c42ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c441077a0 0x7f3c4419d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.391+0000 7f3c42ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c441077a0 0x7f3c4419d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37406/0 (socket says 172.21.15.29:37406) 2024-03-11T19:15:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.391+0000 7f3c427fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c44108990 0x7f3c4419dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33532/0 (socket says 172.21.15.29:33532) 2024-03-11T19:15:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.391+0000 7f3c437fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c44109350 0x7f3c441a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.391+0000 7f3c427fc640 1 -- 172.21.15.29:0/2211157594 learned_addr learned my addr 172.21.15.29:0/2211157594 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:12.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.392+0000 7f3c427fc640 1 -- 172.21.15.29:0/2211157594 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c441077a0 msgr2=0x7f3c4419d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:12.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.392+0000 7f3c427fc640 1 --2- 172.21.15.29:0/2211157594 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c441077a0 0x7f3c4419d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.392+0000 7f3c427fc640 1 -- 172.21.15.29:0/2211157594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c44109350 msgr2=0x7f3c441a4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:12.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.392+0000 7f3c427fc640 1 --2- 172.21.15.29:0/2211157594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c44109350 0x7f3c441a4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.392+0000 7f3c427fc640 1 -- 172.21.15.29:0/2211157594 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c38009560 con 0x7f3c44108990 2024-03-11T19:15:12.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.392+0000 7f3c427fc640 1 --2- 172.21.15.29:0/2211157594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c44108990 0x7f3c4419dcf0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f3c38005ab0 tx=0x7f3c380040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:12.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.392+0000 7f3c2bfff640 1 -- 172.21.15.29:0/2211157594 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c380044c0 con 0x7f3c44108990 2024-03-11T19:15:12.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.392+0000 7f3c2bfff640 1 -- 172.21.15.29:0/2211157594 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3c38003aa0 con 0x7f3c44108990 2024-03-11T19:15:12.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.392+0000 7f3c2bfff640 1 -- 172.21.15.29:0/2211157594 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c38005580 con 0x7f3c44108990 2024-03-11T19:15:12.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.393+0000 7f3c497d0640 1 -- 172.21.15.29:0/2211157594 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c441a5850 con 0x7f3c44108990 2024-03-11T19:15:12.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.393+0000 7f3c497d0640 1 -- 172.21.15.29:0/2211157594 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c441a5d10 con 0x7f3c44108990 2024-03-11T19:15:12.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.397+0000 7f3c4c3a2740 1 Processor -- start 2024-03-11T19:15:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.397+0000 7f3c4c3a2740 1 -- start start 2024-03-11T19:15:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.398+0000 7f3c2bfff640 1 -- 172.21.15.29:0/2211157594 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3c38004dc0 con 0x7f3c44108990 2024-03-11T19:15:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.398+0000 7f3c4c3a2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3270 0x56207b8d3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.398+0000 7f3c4c3a2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d3c60 0x56207b8d8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.398+0000 7f3c4c3a2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d8dd0 0x56207b8db260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.398+0000 7f3c4c3a2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56207b88f960 con 0x56207b8d8dd0 2024-03-11T19:15:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.398+0000 7f3c4c3a2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56207b843310 con 0x56207b8d3c60 2024-03-11T19:15:12.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.398+0000 7f3c4c3a2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56207b840840 con 0x56207b8d3270 2024-03-11T19:15:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.398+0000 7f3c42ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d3c60 0x56207b8d8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.398+0000 7f3c42ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d3c60 0x56207b8d8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:53932/0 (socket says 172.21.15.29:53932) 2024-03-11T19:15:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.398+0000 7f3c42ffd640 1 -- 172.21.15.29:0/2835609275 learned_addr learned my addr 172.21.15.29:0/2835609275 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.398+0000 7f3c437fe640 1 --2- 172.21.15.29:0/2835609275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3270 0x56207b8d3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.398+0000 7f3c427fc640 1 --2- 172.21.15.29:0/2835609275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d8dd0 0x56207b8db260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.399+0000 7f3c2bfff640 1 --2- 172.21.15.29:0/2211157594 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3c2c0a1b80 0x7f3c2c0a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.399+0000 7f3c42ffd640 1 -- 172.21.15.29:0/2835609275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3270 msgr2=0x56207b8d3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.399+0000 7f3c42ffd640 1 --2- 172.21.15.29:0/2835609275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3270 0x56207b8d3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.399+0000 7f3c42ffd640 1 -- 172.21.15.29:0/2835609275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d8dd0 msgr2=0x56207b8db260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:12.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.399+0000 7f3c42ffd640 1 --2- 172.21.15.29:0/2835609275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d8dd0 0x56207b8db260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.399+0000 7f3c42ffd640 1 -- 172.21.15.29:0/2835609275 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56207b7fe060 con 0x56207b8d3c60 2024-03-11T19:15:12.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.399+0000 7f3c427fc640 1 --2- 172.21.15.29:0/2835609275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d8dd0 0x56207b8db260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:15:12.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.399+0000 7f3c437fe640 1 --2- 172.21.15.29:0/2211157594 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3c2c0a1b80 0x7f3c2c0a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.399+0000 7f3c2bfff640 1 -- 172.21.15.29:0/2211157594 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f3c3802de40 con 0x7f3c44108990 2024-03-11T19:15:12.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.399+0000 7f3c42ffd640 1 --2- 172.21.15.29:0/2835609275 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d3c60 0x56207b8d8760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3c34009ca0 tx=0x7f3c34011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb3ec88320c4c24e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:12.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.399+0000 7f3c297fa640 1 -- 172.21.15.29:0/2835609275 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c34004220 con 0x56207b8d3c60 2024-03-11T19:15:12.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.399+0000 7f3c297fa640 1 -- 172.21.15.29:0/2835609275 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3c34004380 con 0x56207b8d3c60 2024-03-11T19:15:12.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.399+0000 7f3c297fa640 1 -- 172.21.15.29:0/2835609275 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c340197a0 con 0x56207b8d3c60 2024-03-11T19:15:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.400+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/2835609275 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d3c60 msgr2=0x56207b8d8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.400+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/2835609275 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d3c60 0x56207b8d8760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3c34009ca0 tx=0x7f3c34011810 comp rx=0 tx=0).stop 2024-03-11T19:15:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.400+0000 7f3c437fe640 1 --2- 172.21.15.29:0/2211157594 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3c2c0a1b80 0x7f3c2c0a4040 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009fd0 tx=0x7f3c3c004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.400+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/2835609275 shutdown_connections 2024-03-11T19:15:12.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.400+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/2835609275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d8dd0 0x56207b8db260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.400+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/2835609275 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d3c60 0x56207b8d8760 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.400+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/2835609275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3270 0x56207b8d3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.400+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/2835609275 >> 172.21.15.29:0/2835609275 conn(0x56207b8ce7e0 msgr2=0x56207b8cebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:12.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.401+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/2835609275 shutdown_connections 2024-03-11T19:15:12.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.401+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/2835609275 wait complete. 2024-03-11T19:15:12.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.401+0000 7f3c4c3a2740 1 -- start start 2024-03-11T19:15:12.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.401+0000 7f3c4c3a2740 1 Processor -- start 2024-03-11T19:15:12.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.402+0000 7f3c4c3a2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3270 0x56207b8dd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.402+0000 7f3c4c3a2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d3c60 0x56207b8dda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.402+0000 7f3c4c3a2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d8dd0 0x56207b8e4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.402+0000 7f3c42ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3270 0x56207b8dd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.402+0000 7f3c42ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3270 0x56207b8dd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37422/0 (socket says 172.21.15.29:37422) 2024-03-11T19:15:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.402+0000 7f3c437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d8dd0 0x56207b8e4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.402+0000 7f3c437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d8dd0 0x56207b8e4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33550/0 (socket says 172.21.15.29:33550) 2024-03-11T19:15:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.402+0000 7f3c42ffd640 1 -- 172.21.15.29:0/3267438779 learned_addr learned my addr 172.21.15.29:0/3267438779 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.402+0000 7f3c427fc640 1 --2- 172.21.15.29:0/3267438779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d3c60 0x56207b8dda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.402+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/3267438779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56207b83c020 con 0x56207b8d8dd0 2024-03-11T19:15:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.402+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/3267438779 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56207b50f430 con 0x56207b8d3c60 2024-03-11T19:15:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.402+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/3267438779 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56207b4cbe00 con 0x56207b8d3270 2024-03-11T19:15:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.402+0000 7f3c4c3a2740 10 client.0 _ll_register_callbacks cb 0x56207b7e3cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.403+0000 7f3c437fe640 1 -- 172.21.15.29:0/3267438779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3270 msgr2=0x56207b8dd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.403+0000 7f3c437fe640 1 --2- 172.21.15.29:0/3267438779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3270 0x56207b8dd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.403+0000 7f3c437fe640 1 -- 172.21.15.29:0/3267438779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d3c60 msgr2=0x56207b8dda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.403+0000 7f3c437fe640 1 --2- 172.21.15.29:0/3267438779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d3c60 0x56207b8dda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.403+0000 7f3c437fe640 1 -- 172.21.15.29:0/3267438779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c340098d0 con 0x56207b8d8dd0 2024-03-11T19:15:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.403+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/3267438779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56207b53b7e0 con 0x56207b8d8dd0 2024-03-11T19:15:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.403+0000 7f3c437fe640 1 --2- 172.21.15.29:0/3267438779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d8dd0 0x56207b8e4a60 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x56207b8d44c0 tx=0x7f3c3c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d768fd17018bfa server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.403+0000 7f3c26ff5640 10 client.16680 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:15:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.403+0000 7f3c26ff5640 1 -- 172.21.15.29:0/3267438779 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c023070 con 0x56207b8d8dd0 2024-03-11T19:15:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.403+0000 7f3c26ff5640 1 -- 172.21.15.29:0/3267438779 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f3c3c01e910 con 0x56207b8d8dd0 2024-03-11T19:15:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.404+0000 7f3c26ff5640 1 -- 172.21.15.29:0/3267438779 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c3c01ead0 con 0x56207b8d8dd0 2024-03-11T19:15:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.404+0000 7f3c4c3a2740 10 client.16680 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.404+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/3267438779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x56207b83a6a0 con 0x56207b8d8dd0 2024-03-11T19:15:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.404+0000 7f3c26ff5640 1 -- 172.21.15.29:0/3267438779 <== mon.0 v2:172.21.15.29:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3c3c01ec30 con 0x56207b8d8dd0 2024-03-11T19:15:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.404+0000 7f3c4c3a2740 10 client.16680 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.404+0000 7f3c4c3a2740 10 client.16680 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:15:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.404+0000 7f3c4c3a2740 20 client.16680 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.404+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/3267438779 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x56207b8e94d0 0x56207b8eb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.404+0000 7f3c4c3a2740 4 client.16680 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:15:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.404+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/3267438779 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56207b578720 con 0x56207b8e94d0 2024-03-11T19:15:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.405+0000 7f3c42ffd640 1 --2- 172.21.15.29:0/3267438779 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x56207b8e94d0 0x56207b8eb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.405+0000 7f3c42ffd640 1 --2- 172.21.15.29:0/3267438779 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x56207b8e94d0 0x56207b8eb970 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c3d841d1463452ca server_cookie=77f278e83ff7affd in_seq=0 out_seq=0 2024-03-11T19:15:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.405+0000 7f3c26ff5640 10 client.16680 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:15:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.406+0000 7f3c26ff5640 1 -- 172.21.15.29:0/3267438779 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56207b578720 con 0x56207b8e94d0 2024-03-11T19:15:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.406+0000 7f3c26ff5640 10 client.16680 handle_command_reply: tid=0 2024-03-11T19:15:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.406+0000 7f3c4c3a2740 1 client.16680 shutdown 2024-03-11T19:15:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.406+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/3267438779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d8dd0 msgr2=0x56207b8e4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.406+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/3267438779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d8dd0 0x56207b8e4a60 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x56207b8d44c0 tx=0x7f3c3c004ab0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.406+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/3267438779 shutdown_connections 2024-03-11T19:15:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.407+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/3267438779 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x56207b8e94d0 0x56207b8eb970 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.407+0000 7f3c26ff5640 0 client.16680 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:15:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.407+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/3267438779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d8dd0 0x56207b8e4a60 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.407+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/3267438779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d3c60 0x56207b8dda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.407+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/3267438779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3270 0x56207b8dd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.407+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/3267438779 >> 172.21.15.29:0/3267438779 conn(0x56207b8ce7e0 msgr2=0x56207b8cdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:12.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.408+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/3267438779 shutdown_connections 2024-03-11T19:15:12.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.408+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/3267438779 wait complete. 2024-03-11T19:15:12.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.408+0000 7f3c4c3a2740 20 client.16680 trim_cache size 0 max 16384 2024-03-11T19:15:12.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.430+0000 7f3c4c3a2740 1 Processor -- start 2024-03-11T19:15:12.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.430+0000 7f3c4c3a2740 1 -- start start 2024-03-11T19:15:12.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.430+0000 7f3c4c3a2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d3270 0x56207b8cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.430+0000 7f3c4c3a2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3c60 0x56207b8e4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.430+0000 7f3c4c3a2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d8dd0 0x56207b8dd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.431+0000 7f3c4c3a2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56207b7c1130 con 0x56207b8d8dd0 2024-03-11T19:15:12.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.431+0000 7f3c4c3a2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56207b578720 con 0x56207b8d3270 2024-03-11T19:15:12.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.431+0000 7f3c4c3a2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56207b8d4350 con 0x56207b8d3c60 2024-03-11T19:15:12.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.431+0000 7f3c437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d8dd0 0x56207b8dd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.431+0000 7f3c427fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3c60 0x56207b8e4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.431+0000 7f3c42ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d3270 0x56207b8cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.431+0000 7f3c437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d8dd0 0x56207b8dd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33564/0 (socket says 172.21.15.29:33564) 2024-03-11T19:15:12.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.431+0000 7f3c437fe640 1 -- 172.21.15.29:0/674306421 learned_addr learned my addr 172.21.15.29:0/674306421 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:12.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.431+0000 7f3c427fc640 1 -- 172.21.15.29:0/674306421 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d3270 msgr2=0x56207b8cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:12.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.431+0000 7f3c427fc640 1 --2- 172.21.15.29:0/674306421 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d3270 0x56207b8cebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.431+0000 7f3c427fc640 1 -- 172.21.15.29:0/674306421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d8dd0 msgr2=0x56207b8dd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:12.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.431+0000 7f3c427fc640 1 --2- 172.21.15.29:0/674306421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d8dd0 0x56207b8dd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.431+0000 7f3c427fc640 1 -- 172.21.15.29:0/674306421 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56207b7d6c20 con 0x56207b8d3c60 2024-03-11T19:15:12.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.432+0000 7f3c427fc640 1 --2- 172.21.15.29:0/674306421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3c60 0x56207b8e4a20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3c380bc7c0 tx=0x7f3c3802bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0f8099bf4a11258 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:12.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.432+0000 7f3c28ff9640 1 -- 172.21.15.29:0/674306421 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c380bcac0 con 0x56207b8d3c60 2024-03-11T19:15:12.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.432+0000 7f3c28ff9640 1 -- 172.21.15.29:0/674306421 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3c380c0c10 con 0x56207b8d3c60 2024-03-11T19:15:12.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.432+0000 7f3c28ff9640 1 -- 172.21.15.29:0/674306421 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c38003c10 con 0x56207b8d3c60 2024-03-11T19:15:12.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.433+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/674306421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3c60 msgr2=0x56207b8e4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:12.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.433+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/674306421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3c60 0x56207b8e4a20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3c380bc7c0 tx=0x7f3c3802bc30 comp rx=0 tx=0).stop 2024-03-11T19:15:12.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.433+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/674306421 shutdown_connections 2024-03-11T19:15:12.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.433+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/674306421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d8dd0 0x56207b8dd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.433+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/674306421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3c60 0x56207b8e4a20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.433+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/674306421 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d3270 0x56207b8cebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.433+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/674306421 >> 172.21.15.29:0/674306421 conn(0x56207b8ce7e0 msgr2=0x56207b8d46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:12.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.435+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/674306421 shutdown_connections 2024-03-11T19:15:12.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.435+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/674306421 wait complete. 2024-03-11T19:15:12.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.435+0000 7f3c4c3a2740 1 -- start start 2024-03-11T19:15:12.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.435+0000 7f3c4c3a2740 1 Processor -- start 2024-03-11T19:15:12.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.436+0000 7f3c4c3a2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3270 0x56207b8e3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.436+0000 7f3c4c3a2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d3c60 0x56207b8e3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.436+0000 7f3c4c3a2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d8dd0 0x56207b8e3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.436+0000 7f3c42ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3270 0x56207b8e3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.436+0000 7f3c42ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3270 0x56207b8e3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37444/0 (socket says 172.21.15.29:37444) 2024-03-11T19:15:12.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.436+0000 7f3c42ffd640 1 -- 172.21.15.29:0/1449584821 learned_addr learned my addr 172.21.15.29:0/1449584821 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:12.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.436+0000 7f3c427fc640 1 --2- 172.21.15.29:0/1449584821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d3c60 0x56207b8e3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.436+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/1449584821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56207b8e4da0 con 0x56207b8d3c60 2024-03-11T19:15:12.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.436+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/1449584821 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56207b8e4f10 con 0x56207b8d8dd0 2024-03-11T19:15:12.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.436+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/1449584821 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56207b93fb00 con 0x56207b8d3270 2024-03-11T19:15:12.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.436+0000 7f3c4c3a2740 10 client.0 _ll_register_callbacks cb 0x7f3c340097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:12.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.436+0000 7f3c437fe640 1 --2- 172.21.15.29:0/1449584821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d8dd0 0x56207b8e3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.437+0000 7f3c427fc640 1 -- 172.21.15.29:0/1449584821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3270 msgr2=0x56207b8e3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.437+0000 7f3c427fc640 1 --2- 172.21.15.29:0/1449584821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3270 0x56207b8e3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.437+0000 7f3c427fc640 1 -- 172.21.15.29:0/1449584821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d8dd0 msgr2=0x56207b8e3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.437+0000 7f3c427fc640 1 --2- 172.21.15.29:0/1449584821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d8dd0 0x56207b8e3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.437+0000 7f3c427fc640 1 -- 172.21.15.29:0/1449584821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c441a5d10 con 0x56207b8d3c60 2024-03-11T19:15:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.437+0000 7f3c427fc640 1 --2- 172.21.15.29:0/1449584821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d3c60 0x56207b8e3690 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x56207b8d4490 tx=0x7f3c380ed760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75732dbc63b46eb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.437+0000 7f3c26ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:15:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.437+0000 7f3c26ff5640 1 -- 172.21.15.29:0/1449584821 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c380c4fd0 con 0x56207b8d3c60 2024-03-11T19:15:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.437+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/1449584821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3c34004380 con 0x56207b8d3c60 2024-03-11T19:15:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.438+0000 7f3c26ff5640 1 -- 172.21.15.29:0/1449584821 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f3c380c5260 con 0x56207b8d3c60 2024-03-11T19:15:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.438+0000 7f3c26ff5640 1 -- 172.21.15.29:0/1449584821 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c380035f0 con 0x56207b8d3c60 2024-03-11T19:15:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.438+0000 7f3c4c3a2740 10 client.16686 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.438+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/1449584821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56207b7fc8c0 con 0x56207b8d3c60 2024-03-11T19:15:12.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.439+0000 7f3c26ff5640 1 -- 172.21.15.29:0/1449584821 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3c34004380 con 0x56207b8d3c60 2024-03-11T19:15:12.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.439+0000 7f3c26ff5640 1 -- 172.21.15.29:0/1449584821 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3c3803c070 con 0x56207b8d3c60 2024-03-11T19:15:12.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.439+0000 7f3c4c3a2740 10 client.16686 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:12.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.439+0000 7f3c4c3a2740 10 client.16686 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:15:12.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.439+0000 7f3c4c3a2740 20 client.16686 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:12.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.439+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/1449584821 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x56207b8e94d0 0x56207b942f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.439+0000 7f3c4c3a2740 4 client.16686 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:15:12.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.439+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/1449584821 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56207b5c7890 con 0x56207b8e94d0 2024-03-11T19:15:12.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.439+0000 7f3c42ffd640 1 --2- 172.21.15.29:0/1449584821 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x56207b8e94d0 0x56207b942f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.440+0000 7f3c42ffd640 1 --2- 172.21.15.29:0/1449584821 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x56207b8e94d0 0x56207b942f50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d8fd82e273452c0a server_cookie=c0655c7ffba55131 in_seq=0 out_seq=0 2024-03-11T19:15:12.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.440+0000 7f3c26ff5640 10 client.16686 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:15:12.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.440+0000 7f3c26ff5640 1 -- 172.21.15.29:0/1449584821 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56207b5c7890 con 0x56207b8e94d0 2024-03-11T19:15:12.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.440+0000 7f3c26ff5640 10 client.16686 handle_command_reply: tid=0 2024-03-11T19:15:12.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.440+0000 7f3c4c3a2740 1 client.16686 shutdown 2024-03-11T19:15:12.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.440+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/1449584821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d3c60 msgr2=0x56207b8e3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:12.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.440+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/1449584821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d3c60 0x56207b8e3690 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x56207b8d4490 tx=0x7f3c380ed760 comp rx=0 tx=0).stop 2024-03-11T19:15:12.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.441+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/1449584821 shutdown_connections 2024-03-11T19:15:12.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.441+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/1449584821 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x56207b8e94d0 0x56207b942f50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.441+0000 7f3c26ff5640 0 client.16686 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:15:12.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.441+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/1449584821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56207b8d8dd0 0x56207b8e3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.441+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/1449584821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56207b8d3c60 0x56207b8e3690 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.441+0000 7f3c4c3a2740 1 --2- 172.21.15.29:0/1449584821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56207b8d3270 0x56207b8e3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.441+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/1449584821 >> 172.21.15.29:0/1449584821 conn(0x56207b8ce7e0 msgr2=0x56207b8cebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:12.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.441+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/1449584821 shutdown_connections 2024-03-11T19:15:12.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.441+0000 7f3c4c3a2740 1 -- 172.21.15.29:0/1449584821 wait complete. 2024-03-11T19:15:12.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.441+0000 7f3c4c3a2740 20 client.16686 trim_cache size 0 max 16384 2024-03-11T19:15:12.447 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:15:12.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.442+0000 7f3c497d0640 1 -- 172.21.15.29:0/2211157594 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3c2c0a1b80 msgr2=0x7f3c2c0a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:12.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.442+0000 7f3c497d0640 1 --2- 172.21.15.29:0/2211157594 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3c2c0a1b80 0x7f3c2c0a4040 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009fd0 tx=0x7f3c3c004a40 comp rx=0 tx=0).stop 2024-03-11T19:15:12.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.442+0000 7f3c497d0640 1 -- 172.21.15.29:0/2211157594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c44108990 msgr2=0x7f3c4419dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:12.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.442+0000 7f3c497d0640 1 --2- 172.21.15.29:0/2211157594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c44108990 0x7f3c4419dcf0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f3c38005ab0 tx=0x7f3c380040c0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.442+0000 7f3c497d0640 1 -- 172.21.15.29:0/2211157594 shutdown_connections 2024-03-11T19:15:12.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.442+0000 7f3c497d0640 1 --2- 172.21.15.29:0/2211157594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c44109350 0x7f3c441a4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.442+0000 7f3c497d0640 1 --2- 172.21.15.29:0/2211157594 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3c2c0a1b80 0x7f3c2c0a4040 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.442+0000 7f3c497d0640 1 --2- 172.21.15.29:0/2211157594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c44108990 0x7f3c4419dcf0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.442+0000 7f3c497d0640 1 --2- 172.21.15.29:0/2211157594 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c441077a0 0x7f3c4419d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.442+0000 7f3c497d0640 1 -- 172.21.15.29:0/2211157594 >> 172.21.15.29:0/2211157594 conn(0x7f3c44076730 msgr2=0x7f3c4410bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:12.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.442+0000 7f3c497d0640 1 -- 172.21.15.29:0/2211157594 shutdown_connections 2024-03-11T19:15:12.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.443+0000 7f3c497d0640 1 -- 172.21.15.29:0/2211157594 wait complete. 2024-03-11T19:15:12.464 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:15:12.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:15:12.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.640+0000 7f0745d6c640 1 -- 172.21.15.29:0/4010058743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f074010aaf0 msgr2=0x7f074010af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:12.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.640+0000 7f0745d6c640 1 --2- 172.21.15.29:0/4010058743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f074010aaf0 0x7f074010af50 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f0734009900 tx=0x7f073402bc30 comp rx=0 tx=0).stop 2024-03-11T19:15:12.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.640+0000 7f0745d6c640 1 -- 172.21.15.29:0/4010058743 shutdown_connections 2024-03-11T19:15:12.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.640+0000 7f0745d6c640 1 --2- 172.21.15.29:0/4010058743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07401118c0 0x7f0740113cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.640+0000 7f0745d6c640 1 --2- 172.21.15.29:0/4010058743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f074010aaf0 0x7f074010af50 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.640+0000 7f0745d6c640 1 --2- 172.21.15.29:0/4010058743 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0740108180 0x7f074010a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.640+0000 7f0745d6c640 1 -- 172.21.15.29:0/4010058743 >> 172.21.15.29:0/4010058743 conn(0x7f074006dfe0 msgr2=0x7f0740070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:12.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.641+0000 7f0745d6c640 1 -- 172.21.15.29:0/4010058743 shutdown_connections 2024-03-11T19:15:12.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.642+0000 7f0745d6c640 1 -- 172.21.15.29:0/4010058743 wait complete. 2024-03-11T19:15:12.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.643+0000 7f0745d6c640 1 Processor -- start 2024-03-11T19:15:12.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.643+0000 7f0745d6c640 1 -- start start 2024-03-11T19:15:12.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.643+0000 7f0745d6c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0740108180 0x7f07401a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.643+0000 7f0745d6c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f074010aaf0 0x7f07401a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.643+0000 7f0745d6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07401118c0 0x7f07401a8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.643+0000 7f0745d6c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07401a94e0 con 0x7f07401118c0 2024-03-11T19:15:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.643+0000 7f0745d6c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07401a9650 con 0x7f0740108180 2024-03-11T19:15:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.643+0000 7f0745d6c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07401a9810 con 0x7f074010aaf0 2024-03-11T19:15:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.643+0000 7f073ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07401118c0 0x7f07401a8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.644+0000 7f073ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07401118c0 0x7f07401a8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33596/0 (socket says 172.21.15.29:33596) 2024-03-11T19:15:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.644+0000 7f073ffff640 1 -- 172.21.15.29:0/2558608895 learned_addr learned my addr 172.21.15.29:0/2558608895 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.644+0000 7f073effd640 1 --2- 172.21.15.29:0/2558608895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f074010aaf0 0x7f07401a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.644+0000 7f073f7fe640 1 --2- 172.21.15.29:0/2558608895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0740108180 0x7f07401a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.644+0000 7f073ffff640 1 -- 172.21.15.29:0/2558608895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f074010aaf0 msgr2=0x7f07401a1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.644+0000 7f073ffff640 1 --2- 172.21.15.29:0/2558608895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f074010aaf0 0x7f07401a1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.644+0000 7f073ffff640 1 -- 172.21.15.29:0/2558608895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0740108180 msgr2=0x7f07401a1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:12.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.644+0000 7f073ffff640 1 --2- 172.21.15.29:0/2558608895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0740108180 0x7f07401a1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.644+0000 7f073ffff640 1 -- 172.21.15.29:0/2558608895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0730009490 con 0x7f07401118c0 2024-03-11T19:15:12.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.644+0000 7f073f7fe640 1 --2- 172.21.15.29:0/2558608895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0740108180 0x7f07401a1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:12.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.645+0000 7f073ffff640 1 --2- 172.21.15.29:0/2558608895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07401118c0 0x7f07401a8fa0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f073800ec50 tx=0x7f073800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:12.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.645+0000 7f073cff9640 1 -- 172.21.15.29:0/2558608895 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738003b70 con 0x7f07401118c0 2024-03-11T19:15:12.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.645+0000 7f0745d6c640 1 -- 172.21.15.29:0/2558608895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0738009840 con 0x7f07401118c0 2024-03-11T19:15:12.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.645+0000 7f0745d6c640 1 -- 172.21.15.29:0/2558608895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0734009560 con 0x7f07401118c0 2024-03-11T19:15:12.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.645+0000 7f073cff9640 1 -- 172.21.15.29:0/2558608895 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0738003cd0 con 0x7f07401118c0 2024-03-11T19:15:12.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.645+0000 7f073cff9640 1 -- 172.21.15.29:0/2558608895 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738010830 con 0x7f07401118c0 2024-03-11T19:15:12.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.649+0000 7f073cff9640 1 -- 172.21.15.29:0/2558608895 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0738010990 con 0x7f07401118c0 2024-03-11T19:15:12.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.649+0000 7f0748a19740 1 Processor -- start 2024-03-11T19:15:12.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.649+0000 7f0748a19740 1 -- start start 2024-03-11T19:15:12.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.649+0000 7f0748a19740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e503a80 0x55ae9e503ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.649+0000 7f0748a19740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e504470 0x55ae9e508f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.649+0000 7f0748a19740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e5095e0 0x55ae9e50ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.649+0000 7f0748a19740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae9e4b7960 con 0x55ae9e503a80 2024-03-11T19:15:12.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.649+0000 7f0748a19740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae9e46b310 con 0x55ae9e504470 2024-03-11T19:15:12.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.649+0000 7f0748a19740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae9e468840 con 0x55ae9e5095e0 2024-03-11T19:15:12.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.650+0000 7f073effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e5095e0 0x55ae9e50ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.650+0000 7f073effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e503a80 0x55ae9e503ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.650+0000 7f073effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e503a80 0x55ae9e503ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33606/0 (socket says 172.21.15.29:33606) 2024-03-11T19:15:12.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.650+0000 7f073effd640 1 -- 172.21.15.29:0/3742039107 learned_addr learned my addr 172.21.15.29:0/3742039107 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:12.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.650+0000 7f073f7fe640 1 --2- 172.21.15.29:0/3742039107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e504470 0x55ae9e508f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.651+0000 7f073effd640 1 -- 172.21.15.29:0/3742039107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e5095e0 msgr2=0x55ae9e50ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:12.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.651+0000 7f073effd640 1 --2- 172.21.15.29:0/3742039107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e5095e0 0x55ae9e50ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.651+0000 7f073effd640 1 -- 172.21.15.29:0/3742039107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e504470 msgr2=0x55ae9e508f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:12.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.651+0000 7f073effd640 1 --2- 172.21.15.29:0/3742039107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e504470 0x55ae9e508f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.651+0000 7f073cff9640 1 --2- 172.21.15.29:0/2558608895 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f07280a1b30 0x7f07280a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.651+0000 7f073effd640 1 -- 172.21.15.29:0/3742039107 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae9e426060 con 0x55ae9e503a80 2024-03-11T19:15:12.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.651+0000 7f073cff9640 1 -- 172.21.15.29:0/2558608895 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0738014070 con 0x7f07401118c0 2024-03-11T19:15:12.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.652+0000 7f073effd640 1 --2- 172.21.15.29:0/3742039107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e503a80 0x55ae9e503ea0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0734009900 tx=0x7f0734000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1144d18a351e8341 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:12.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.652+0000 7f073effd640 1 --2- 172.21.15.29:0/2558608895 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f07280a1b30 0x7f07280a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.652+0000 7f0725ffb640 1 -- 172.21.15.29:0/3742039107 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0734004040 con 0x55ae9e503a80 2024-03-11T19:15:12.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.652+0000 7f0725ffb640 1 -- 172.21.15.29:0/3742039107 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f07340035f0 con 0x55ae9e503a80 2024-03-11T19:15:12.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.652+0000 7f0725ffb640 1 -- 172.21.15.29:0/3742039107 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0734005060 con 0x55ae9e503a80 2024-03-11T19:15:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.653+0000 7f0748a19740 1 -- 172.21.15.29:0/3742039107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e503a80 msgr2=0x55ae9e503ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.653+0000 7f0748a19740 1 --2- 172.21.15.29:0/3742039107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e503a80 0x55ae9e503ea0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0734009900 tx=0x7f0734000bc0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.653+0000 7f073effd640 1 --2- 172.21.15.29:0/2558608895 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f07280a1b30 0x7f07280a3ff0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0734004c20 tx=0x7f0734005280 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.654+0000 7f0748a19740 1 -- 172.21.15.29:0/3742039107 shutdown_connections 2024-03-11T19:15:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.654+0000 7f0748a19740 1 --2- 172.21.15.29:0/3742039107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e5095e0 0x55ae9e50ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.654+0000 7f0748a19740 1 --2- 172.21.15.29:0/3742039107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e504470 0x55ae9e508f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.654+0000 7f0748a19740 1 --2- 172.21.15.29:0/3742039107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e503a80 0x55ae9e503ea0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.654+0000 7f0748a19740 1 -- 172.21.15.29:0/3742039107 >> 172.21.15.29:0/3742039107 conn(0x55ae9e4feff0 msgr2=0x55ae9e4ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.654+0000 7f0748a19740 1 -- 172.21.15.29:0/3742039107 shutdown_connections 2024-03-11T19:15:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.654+0000 7f0748a19740 1 -- 172.21.15.29:0/3742039107 wait complete. 2024-03-11T19:15:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.654+0000 7f0748a19740 1 -- start start 2024-03-11T19:15:12.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.655+0000 7f0748a19740 1 Processor -- start 2024-03-11T19:15:12.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.655+0000 7f0748a19740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e503a80 0x55ae9e50dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.655+0000 7f0748a19740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e504470 0x55ae9e50e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.655+0000 7f0748a19740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e5095e0 0x55ae9e515210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.655+0000 7f0748a19740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae9e464020 con 0x55ae9e504470 2024-03-11T19:15:12.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.655+0000 7f0748a19740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae9e137430 con 0x55ae9e5095e0 2024-03-11T19:15:12.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.655+0000 7f0748a19740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae9e0f3e00 con 0x55ae9e503a80 2024-03-11T19:15:12.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.655+0000 7f0748a19740 10 client.0 _ll_register_callbacks cb 0x55ae9e40bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:12.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.655+0000 7f073ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e5095e0 0x55ae9e515210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.656+0000 7f073ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e5095e0 0x55ae9e515210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:54002/0 (socket says 172.21.15.29:54002) 2024-03-11T19:15:12.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.656+0000 7f073ffff640 1 -- 172.21.15.29:0/1765042929 learned_addr learned my addr 172.21.15.29:0/1765042929 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:12.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.656+0000 7f073effd640 1 --2- 172.21.15.29:0/1765042929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e504470 0x55ae9e50e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.656+0000 7f073f7fe640 1 --2- 172.21.15.29:0/1765042929 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e503a80 0x55ae9e50dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.656+0000 7f073ffff640 1 -- 172.21.15.29:0/1765042929 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e503a80 msgr2=0x55ae9e50dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:12.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.656+0000 7f073ffff640 1 --2- 172.21.15.29:0/1765042929 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e503a80 0x55ae9e50dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.656+0000 7f073ffff640 1 -- 172.21.15.29:0/1765042929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e504470 msgr2=0x55ae9e50e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:12.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.656+0000 7f073ffff640 1 --2- 172.21.15.29:0/1765042929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e504470 0x55ae9e50e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.656+0000 7f073ffff640 1 -- 172.21.15.29:0/1765042929 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07401148b0 con 0x55ae9e5095e0 2024-03-11T19:15:12.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.656+0000 7f073ffff640 1 --2- 172.21.15.29:0/1765042929 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e5095e0 0x55ae9e515210 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0738097650 tx=0x7f073809a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11f67a5bd1dcc6ac server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:12.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.656+0000 7f071effd640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:15:12.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.657+0000 7f071effd640 1 -- 172.21.15.29:0/1765042929 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07380aed10 con 0x55ae9e5095e0 2024-03-11T19:15:12.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.657+0000 7f071effd640 1 -- 172.21.15.29:0/1765042929 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f07380cb820 con 0x55ae9e5095e0 2024-03-11T19:15:12.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.657+0000 7f0748a19740 1 -- 172.21.15.29:0/1765042929 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ae9e1637e0 con 0x55ae9e5095e0 2024-03-11T19:15:12.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.657+0000 7f071effd640 1 -- 172.21.15.29:0/1765042929 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0738010e50 con 0x55ae9e5095e0 2024-03-11T19:15:12.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.657+0000 7f0748a19740 10 client.26222 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:12.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.657+0000 7f0748a19740 1 -- 172.21.15.29:0/1765042929 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ae9e4626a0 con 0x55ae9e5095e0 2024-03-11T19:15:12.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.658+0000 7f071effd640 1 -- 172.21.15.29:0/1765042929 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55ae9e1637e0 con 0x55ae9e5095e0 2024-03-11T19:15:12.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.658+0000 7f071effd640 1 -- 172.21.15.29:0/1765042929 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f073801f060 con 0x55ae9e5095e0 2024-03-11T19:15:12.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.658+0000 7f0748a19740 10 client.26222 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:12.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.658+0000 7f0748a19740 10 client.26222 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:15:12.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.658+0000 7f0748a19740 20 client.26222 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:12.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.658+0000 7f0748a19740 1 --2- 172.21.15.29:0/1765042929 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55ae9e519cf0 0x55ae9e51c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.658+0000 7f0748a19740 4 client.26222 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:15:12.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.658+0000 7f0748a19740 1 -- 172.21.15.29:0/1765042929 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ae9e1a0720 con 0x55ae9e519cf0 2024-03-11T19:15:12.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.659+0000 7f073f7fe640 1 --2- 172.21.15.29:0/1765042929 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55ae9e519cf0 0x55ae9e51c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.660+0000 7f071effd640 10 client.26222 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:15:12.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.660+0000 7f073f7fe640 1 --2- 172.21.15.29:0/1765042929 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55ae9e519cf0 0x55ae9e51c110 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=15edd22b894b5b38 server_cookie=5ced82886674fae0 in_seq=0 out_seq=0 2024-03-11T19:15:12.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.660+0000 7f071effd640 1 -- 172.21.15.29:0/1765042929 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ae9e1a0720 con 0x55ae9e519cf0 2024-03-11T19:15:12.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.660+0000 7f071effd640 10 client.26222 handle_command_reply: tid=0 2024-03-11T19:15:12.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.660+0000 7f0748a19740 1 client.26222 shutdown 2024-03-11T19:15:12.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.660+0000 7f0748a19740 1 -- 172.21.15.29:0/1765042929 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e5095e0 msgr2=0x55ae9e515210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:12.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.660+0000 7f0748a19740 1 --2- 172.21.15.29:0/1765042929 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e5095e0 0x55ae9e515210 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0738097650 tx=0x7f073809a6d0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.660+0000 7f0748a19740 1 -- 172.21.15.29:0/1765042929 shutdown_connections 2024-03-11T19:15:12.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.661+0000 7f0748a19740 1 --2- 172.21.15.29:0/1765042929 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e5095e0 0x55ae9e515210 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.661+0000 7f0748a19740 1 --2- 172.21.15.29:0/1765042929 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55ae9e519cf0 0x55ae9e51c110 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.661+0000 7f0748a19740 1 --2- 172.21.15.29:0/1765042929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e504470 0x55ae9e50e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.661+0000 7f071effd640 0 client.26222 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:15:12.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.661+0000 7f0748a19740 1 --2- 172.21.15.29:0/1765042929 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e503a80 0x55ae9e50dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.661+0000 7f0748a19740 1 -- 172.21.15.29:0/1765042929 >> 172.21.15.29:0/1765042929 conn(0x55ae9e4feff0 msgr2=0x55ae9e500240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:12.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.661+0000 7f0748a19740 1 -- 172.21.15.29:0/1765042929 shutdown_connections 2024-03-11T19:15:12.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.661+0000 7f0748a19740 1 -- 172.21.15.29:0/1765042929 wait complete. 2024-03-11T19:15:12.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.661+0000 7f0748a19740 20 client.26222 trim_cache size 0 max 16384 2024-03-11T19:15:12.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.683+0000 7f0748a19740 1 Processor -- start 2024-03-11T19:15:12.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.683+0000 7f0748a19740 1 -- start start 2024-03-11T19:15:12.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.683+0000 7f0748a19740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e503a80 0x55ae9e505990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.684+0000 7f0748a19740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e504470 0x55ae9e5151d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.684+0000 7f0748a19740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e5095e0 0x55ae9e505ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.684+0000 7f073f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e503a80 0x55ae9e505990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.684+0000 7f073f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e503a80 0x55ae9e505990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37502/0 (socket says 172.21.15.29:37502) 2024-03-11T19:15:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.684+0000 7f073effd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e504470 0x55ae9e5151d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.684+0000 7f073effd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e504470 0x55ae9e5151d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:54004/0 (socket says 172.21.15.29:54004) 2024-03-11T19:15:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.684+0000 7f073ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e5095e0 0x55ae9e505ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.684+0000 7f073ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e5095e0 0x55ae9e505ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33624/0 (socket says 172.21.15.29:33624) 2024-03-11T19:15:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.684+0000 7f073f7fe640 1 -- 172.21.15.29:0/1168015311 learned_addr learned my addr 172.21.15.29:0/1168015311 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.685+0000 7f0748a19740 1 -- 172.21.15.29:0/1168015311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae9e3e9130 con 0x55ae9e5095e0 2024-03-11T19:15:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.685+0000 7f0748a19740 1 -- 172.21.15.29:0/1168015311 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae9e503640 con 0x55ae9e504470 2024-03-11T19:15:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.685+0000 7f0748a19740 1 -- 172.21.15.29:0/1168015311 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae9e519b00 con 0x55ae9e503a80 2024-03-11T19:15:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.685+0000 7f073ffff640 1 -- 172.21.15.29:0/1168015311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e503a80 msgr2=0x55ae9e505990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.685+0000 7f073ffff640 1 --2- 172.21.15.29:0/1168015311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e503a80 0x55ae9e505990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.685+0000 7f073ffff640 1 -- 172.21.15.29:0/1168015311 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e504470 msgr2=0x55ae9e5151d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.685+0000 7f073ffff640 1 --2- 172.21.15.29:0/1168015311 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e504470 0x55ae9e5151d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.685+0000 7f073ffff640 1 -- 172.21.15.29:0/1168015311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae9e3fec20 con 0x55ae9e5095e0 2024-03-11T19:15:12.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.686+0000 7f073ffff640 1 --2- 172.21.15.29:0/1168015311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e5095e0 0x55ae9e505ed0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f073809ac10 tx=0x7f0738004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cdc3e46ab89cbc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:12.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.686+0000 7f07257fa640 1 -- 172.21.15.29:0/1168015311 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738010e50 con 0x55ae9e5095e0 2024-03-11T19:15:12.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.686+0000 7f07257fa640 1 -- 172.21.15.29:0/1168015311 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0738096f20 con 0x55ae9e5095e0 2024-03-11T19:15:12.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.686+0000 7f07257fa640 1 -- 172.21.15.29:0/1168015311 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738099da0 con 0x55ae9e5095e0 2024-03-11T19:15:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.687+0000 7f0748a19740 1 -- 172.21.15.29:0/1168015311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e5095e0 msgr2=0x55ae9e505ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.687+0000 7f0748a19740 1 --2- 172.21.15.29:0/1168015311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e5095e0 0x55ae9e505ed0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f073809ac10 tx=0x7f0738004790 comp rx=0 tx=0).stop 2024-03-11T19:15:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.688+0000 7f0748a19740 1 -- 172.21.15.29:0/1168015311 shutdown_connections 2024-03-11T19:15:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.688+0000 7f0748a19740 1 --2- 172.21.15.29:0/1168015311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e5095e0 0x55ae9e505ed0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.688+0000 7f0748a19740 1 --2- 172.21.15.29:0/1168015311 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e504470 0x55ae9e5151d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.688+0000 7f0748a19740 1 --2- 172.21.15.29:0/1168015311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e503a80 0x55ae9e505990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.688+0000 7f0748a19740 1 -- 172.21.15.29:0/1168015311 >> 172.21.15.29:0/1168015311 conn(0x55ae9e4feff0 msgr2=0x55ae9e50dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:12.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.688+0000 7f0748a19740 1 -- 172.21.15.29:0/1168015311 shutdown_connections 2024-03-11T19:15:12.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.688+0000 7f0748a19740 1 -- 172.21.15.29:0/1168015311 wait complete. 2024-03-11T19:15:12.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.689+0000 7f0748a19740 1 -- start start 2024-03-11T19:15:12.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.689+0000 7f0748a19740 1 Processor -- start 2024-03-11T19:15:12.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.690+0000 7f0748a19740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e503a80 0x55ae9e5138b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.690+0000 7f0748a19740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e504470 0x55ae9e513df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.690+0000 7f0748a19740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e5095e0 0x55ae9e514330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.690+0000 7f0748a19740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae9e5154b0 con 0x55ae9e504470 2024-03-11T19:15:12.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.690+0000 7f0748a19740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae9e570200 con 0x55ae9e5095e0 2024-03-11T19:15:12.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.690+0000 7f0748a19740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae9e570340 con 0x55ae9e503a80 2024-03-11T19:15:12.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.690+0000 7f0748a19740 10 client.0 _ll_register_callbacks cb 0x7f07380093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:12.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.690+0000 7f073f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e503a80 0x55ae9e5138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.690+0000 7f073f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e503a80 0x55ae9e5138b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37508/0 (socket says 172.21.15.29:37508) 2024-03-11T19:15:12.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.690+0000 7f073f7fe640 1 -- 172.21.15.29:0/2426808945 learned_addr learned my addr 172.21.15.29:0/2426808945 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:12.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.690+0000 7f073effd640 1 --2- 172.21.15.29:0/2426808945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e504470 0x55ae9e513df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.690+0000 7f073ffff640 1 --2- 172.21.15.29:0/2426808945 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e5095e0 0x55ae9e514330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.691+0000 7f073f7fe640 1 -- 172.21.15.29:0/2426808945 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e5095e0 msgr2=0x55ae9e514330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.691+0000 7f073f7fe640 1 --2- 172.21.15.29:0/2426808945 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e5095e0 0x55ae9e514330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.691+0000 7f073f7fe640 1 -- 172.21.15.29:0/2426808945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e504470 msgr2=0x55ae9e513df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.691+0000 7f073f7fe640 1 --2- 172.21.15.29:0/2426808945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e504470 0x55ae9e513df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.691+0000 7f073f7fe640 1 -- 172.21.15.29:0/2426808945 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07380b9020 con 0x55ae9e503a80 2024-03-11T19:15:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.691+0000 7f073f7fe640 1 --2- 172.21.15.29:0/2426808945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e503a80 0x55ae9e5138b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x55ae9e504870 tx=0x7f0730004410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d376847ec612b31c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:12.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.691+0000 7f071effd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:15:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.691+0000 7f071effd640 1 -- 172.21.15.29:0/2426808945 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0730017040 con 0x55ae9e503a80 2024-03-11T19:15:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.691+0000 7f071effd640 1 -- 172.21.15.29:0/2426808945 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0730021610 con 0x55ae9e503a80 2024-03-11T19:15:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.691+0000 7f0748a19740 1 -- 172.21.15.29:0/2426808945 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f07340035f0 con 0x55ae9e503a80 2024-03-11T19:15:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.692+0000 7f071effd640 1 -- 172.21.15.29:0/2426808945 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f07300217f0 con 0x55ae9e503a80 2024-03-11T19:15:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.692+0000 7f0748a19740 10 client.26236 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.692+0000 7f0748a19740 1 -- 172.21.15.29:0/2426808945 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ae9e4248c0 con 0x55ae9e503a80 2024-03-11T19:15:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.692+0000 7f071effd640 1 -- 172.21.15.29:0/2426808945 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f07340035f0 con 0x55ae9e503a80 2024-03-11T19:15:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.692+0000 7f071effd640 1 -- 172.21.15.29:0/2426808945 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0730027030 con 0x55ae9e503a80 2024-03-11T19:15:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.692+0000 7f0748a19740 10 client.26236 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.692+0000 7f0748a19740 10 client.26236 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:15:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.692+0000 7f0748a19740 20 client.26236 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.692+0000 7f0748a19740 1 --2- 172.21.15.29:0/2426808945 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55ae9e519cf0 0x55ae9e573eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.692+0000 7f0748a19740 4 client.26236 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:15:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.692+0000 7f0748a19740 1 -- 172.21.15.29:0/2426808945 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ae9e1ef890 con 0x55ae9e519cf0 2024-03-11T19:15:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.693+0000 7f073effd640 1 --2- 172.21.15.29:0/2426808945 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55ae9e519cf0 0x55ae9e573eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.693+0000 7f073effd640 1 --2- 172.21.15.29:0/2426808945 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55ae9e519cf0 0x55ae9e573eb0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=da0447cfb5ffa23f server_cookie=a041b057703c2fb8 in_seq=0 out_seq=0 2024-03-11T19:15:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.693+0000 7f071effd640 10 client.26236 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:15:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.694+0000 7f071effd640 1 -- 172.21.15.29:0/2426808945 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ae9e1ef890 con 0x55ae9e519cf0 2024-03-11T19:15:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.694+0000 7f071effd640 10 client.26236 handle_command_reply: tid=0 2024-03-11T19:15:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.694+0000 7f0748a19740 1 client.26236 shutdown 2024-03-11T19:15:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.694+0000 7f0748a19740 1 -- 172.21.15.29:0/2426808945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e503a80 msgr2=0x55ae9e5138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.694+0000 7f0748a19740 1 --2- 172.21.15.29:0/2426808945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e503a80 0x55ae9e5138b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x55ae9e504870 tx=0x7f0730004410 comp rx=0 tx=0).stop 2024-03-11T19:15:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.694+0000 7f0748a19740 1 -- 172.21.15.29:0/2426808945 shutdown_connections 2024-03-11T19:15:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.694+0000 7f0748a19740 1 --2- 172.21.15.29:0/2426808945 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ae9e5095e0 0x55ae9e514330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.694+0000 7f0748a19740 1 --2- 172.21.15.29:0/2426808945 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55ae9e519cf0 0x55ae9e573eb0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.694+0000 7f0748a19740 1 --2- 172.21.15.29:0/2426808945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ae9e504470 0x55ae9e513df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.694+0000 7f0748a19740 1 --2- 172.21.15.29:0/2426808945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ae9e503a80 0x55ae9e5138b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.694+0000 7f0748a19740 1 -- 172.21.15.29:0/2426808945 >> 172.21.15.29:0/2426808945 conn(0x55ae9e4feff0 msgr2=0x55ae9e50dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:12.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.694+0000 7f071effd640 0 client.26236 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:15:12.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.694+0000 7f0748a19740 1 -- 172.21.15.29:0/2426808945 shutdown_connections 2024-03-11T19:15:12.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.694+0000 7f0748a19740 1 -- 172.21.15.29:0/2426808945 wait complete. 2024-03-11T19:15:12.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.694+0000 7f0748a19740 20 client.26236 trim_cache size 0 max 16384 2024-03-11T19:15:12.700 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:15:12.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.695+0000 7f0745d6c640 1 -- 172.21.15.29:0/2558608895 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f07280a1b30 msgr2=0x7f07280a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:12.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.695+0000 7f0745d6c640 1 --2- 172.21.15.29:0/2558608895 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f07280a1b30 0x7f07280a3ff0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0734004c20 tx=0x7f0734005280 comp rx=0 tx=0).stop 2024-03-11T19:15:12.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.695+0000 7f0745d6c640 1 -- 172.21.15.29:0/2558608895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07401118c0 msgr2=0x7f07401a8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.695+0000 7f0745d6c640 1 --2- 172.21.15.29:0/2558608895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07401118c0 0x7f07401a8fa0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f073800ec50 tx=0x7f073800c6a0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.696+0000 7f0745d6c640 1 -- 172.21.15.29:0/2558608895 shutdown_connections 2024-03-11T19:15:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.696+0000 7f0745d6c640 1 --2- 172.21.15.29:0/2558608895 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f07280a1b30 0x7f07280a3ff0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.696+0000 7f0745d6c640 1 --2- 172.21.15.29:0/2558608895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07401118c0 0x7f07401a8fa0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.696+0000 7f0745d6c640 1 --2- 172.21.15.29:0/2558608895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f074010aaf0 0x7f07401a1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.696+0000 7f0745d6c640 1 --2- 172.21.15.29:0/2558608895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0740108180 0x7f07401a1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.696+0000 7f0745d6c640 1 -- 172.21.15.29:0/2558608895 >> 172.21.15.29:0/2558608895 conn(0x7f074006dfe0 msgr2=0x7f0740108fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.696+0000 7f0745d6c640 1 -- 172.21.15.29:0/2558608895 shutdown_connections 2024-03-11T19:15:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:12.697+0000 7f0745d6c640 1 -- 172.21.15.29:0/2558608895 wait complete. 2024-03-11T19:15:12.719 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:15:12.720 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:15:13.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:12 smithi170 ceph-mon[35690]: pgmap v513: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T19:15:13.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:12 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2880691922' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:15:13.112 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:12 smithi029 ceph-mon[32439]: pgmap v513: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T19:15:13.112 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:13 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2880691922' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:15:13.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.114+0000 7f8507fac640 1 -- 172.21.15.29:0/4110054168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8500101c30 msgr2=0x7f85001084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.114+0000 7f8507fac640 1 --2- 172.21.15.29:0/4110054168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8500101c30 0x7f85001084c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009ca0 tx=0x7f84f802c0f0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.114+0000 7f8507fac640 1 -- 172.21.15.29:0/4110054168 shutdown_connections 2024-03-11T19:15:13.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.114+0000 7f8507fac640 1 --2- 172.21.15.29:0/4110054168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8500101c30 0x7f85001084c0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.114+0000 7f8507fac640 1 --2- 172.21.15.29:0/4110054168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8500101270 0x7f85001016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.115+0000 7f8507fac640 1 --2- 172.21.15.29:0/4110054168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8500100080 0x7f85001004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.115+0000 7f8507fac640 1 -- 172.21.15.29:0/4110054168 >> 172.21.15.29:0/4110054168 conn(0x7f85000fb610 msgr2=0x7f85000fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:13.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.115+0000 7f8507fac640 1 -- 172.21.15.29:0/4110054168 shutdown_connections 2024-03-11T19:15:13.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.115+0000 7f8507fac640 1 -- 172.21.15.29:0/4110054168 wait complete. 2024-03-11T19:15:13.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.116+0000 7f8507fac640 1 Processor -- start 2024-03-11T19:15:13.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.116+0000 7f8507fac640 1 -- start start 2024-03-11T19:15:13.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.116+0000 7f8507fac640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8500100080 0x7f850019d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.116+0000 7f8507fac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8500101270 0x7f850019dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.116+0000 7f8507fac640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8500101c30 0x7f85001a4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.116+0000 7f8507fac640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85001a5220 con 0x7f8500101270 2024-03-11T19:15:13.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.116+0000 7f8507fac640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85001a5390 con 0x7f8500101c30 2024-03-11T19:15:13.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.116+0000 7f8507fac640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85001a5550 con 0x7f8500100080 2024-03-11T19:15:13.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.116+0000 7f8505547640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8500101270 0x7f850019dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.116+0000 7f8505547640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8500101270 0x7f850019dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33644/0 (socket says 172.21.15.29:33644) 2024-03-11T19:15:13.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.116+0000 7f8505547640 1 -- 172.21.15.29:0/680853886 learned_addr learned my addr 172.21.15.29:0/680853886 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:13.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.117+0000 7f8505d48640 1 --2- 172.21.15.29:0/680853886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8500100080 0x7f850019d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.117+0000 7f8505547640 1 -- 172.21.15.29:0/680853886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8500100080 msgr2=0x7f850019d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.117+0000 7f8505547640 1 --2- 172.21.15.29:0/680853886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8500100080 0x7f850019d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.117+0000 7f8505547640 1 -- 172.21.15.29:0/680853886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8500101c30 msgr2=0x7f85001a4ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-11T19:15:13.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.117+0000 7f8505547640 1 --2- 172.21.15.29:0/680853886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8500101c30 0x7f85001a4ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.117+0000 7f8505547640 1 -- 172.21.15.29:0/680853886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f8009900 con 0x7f8500101270 2024-03-11T19:15:13.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.117+0000 7f8505547640 1 --2- 172.21.15.29:0/680853886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8500101270 0x7f850019dc80 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f84f000e9f0 tx=0x7f84f000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:13.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.118+0000 7f84f6ffd640 1 -- 172.21.15.29:0/680853886 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f000cd80 con 0x7f8500101270 2024-03-11T19:15:13.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.118+0000 7f8507fac640 1 -- 172.21.15.29:0/680853886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85001a5830 con 0x7f8500101270 2024-03-11T19:15:13.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.118+0000 7f8507fac640 1 -- 172.21.15.29:0/680853886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85001a5d40 con 0x7f8500101270 2024-03-11T19:15:13.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.118+0000 7f84f6ffd640 1 -- 172.21.15.29:0/680853886 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f84f00054d0 con 0x7f8500101270 2024-03-11T19:15:13.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.118+0000 7f84f6ffd640 1 -- 172.21.15.29:0/680853886 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f0010690 con 0x7f8500101270 2024-03-11T19:15:13.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.122+0000 7f850ab80740 1 Processor -- start 2024-03-11T19:15:13.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.122+0000 7f84f6ffd640 1 -- 172.21.15.29:0/680853886 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f84f00046c0 con 0x7f8500101270 2024-03-11T19:15:13.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.122+0000 7f850ab80740 1 -- start start 2024-03-11T19:15:13.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.123+0000 7f850ab80740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684270 0x556dfe684690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.123+0000 7f850ab80740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe684c60 0x556dfe689760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.123+0000 7f850ab80740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe689dd0 0x556dfe68c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.123+0000 7f850ab80740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dfe640960 con 0x556dfe684270 2024-03-11T19:15:13.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.123+0000 7f850ab80740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dfe5f4310 con 0x556dfe684c60 2024-03-11T19:15:13.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.123+0000 7f850ab80740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dfe5f1840 con 0x556dfe689dd0 2024-03-11T19:15:13.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.123+0000 7f8506549640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684270 0x556dfe684690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.123+0000 7f8506549640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684270 0x556dfe684690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33656/0 (socket says 172.21.15.29:33656) 2024-03-11T19:15:13.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.123+0000 7f8506549640 1 -- 172.21.15.29:0/1397082596 learned_addr learned my addr 172.21.15.29:0/1397082596 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:13.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.123+0000 7f84f6ffd640 1 --2- 172.21.15.29:0/680853886 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f84ec0a1b80 0x7f84ec0a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.123+0000 7f84f6ffd640 1 -- 172.21.15.29:0/680853886 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f84f0014070 con 0x7f8500101270 2024-03-11T19:15:13.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.124+0000 7f8506549640 1 -- 172.21.15.29:0/1397082596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe689dd0 msgr2=0x556dfe68c260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:15:13.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.124+0000 7f8506549640 1 --2- 172.21.15.29:0/1397082596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe689dd0 0x556dfe68c260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.124+0000 7f8506549640 1 -- 172.21.15.29:0/1397082596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe684c60 msgr2=0x556dfe689760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.124+0000 7f8505d48640 1 --2- 172.21.15.29:0/1397082596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe684c60 0x556dfe689760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.124+0000 7f8506549640 1 --2- 172.21.15.29:0/1397082596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe684c60 0x556dfe689760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.124+0000 7f8506549640 1 -- 172.21.15.29:0/1397082596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556dfe5af060 con 0x556dfe684270 2024-03-11T19:15:13.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.124+0000 7f8505d48640 1 --2- 172.21.15.29:0/1397082596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe684c60 0x556dfe689760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:15:13.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.125+0000 7f8506549640 1 --2- 172.21.15.29:0/680853886 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f84ec0a1b80 0x7f84ec0a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.125+0000 7f8506549640 1 --2- 172.21.15.29:0/1397082596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684270 0x556dfe684690 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f84f802c630 tx=0x7f84f80093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8742f36a5c1433c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:13.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.125+0000 7f8506549640 1 --2- 172.21.15.29:0/680853886 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f84ec0a1b80 0x7f84ec0a4040 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f84f8048000 tx=0x7f84f8049000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:13.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.125+0000 7f84ebfff640 1 -- 172.21.15.29:0/1397082596 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f8032a10 con 0x556dfe684270 2024-03-11T19:15:13.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.126+0000 7f84ebfff640 1 -- 172.21.15.29:0/1397082596 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f84f8003970 con 0x556dfe684270 2024-03-11T19:15:13.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.126+0000 7f84ebfff640 1 -- 172.21.15.29:0/1397082596 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f8004260 con 0x556dfe684270 2024-03-11T19:15:13.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.127+0000 7f850ab80740 1 -- 172.21.15.29:0/1397082596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684270 msgr2=0x556dfe684690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.127+0000 7f850ab80740 1 --2- 172.21.15.29:0/1397082596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684270 0x556dfe684690 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f84f802c630 tx=0x7f84f80093d0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.127+0000 7f850ab80740 1 -- 172.21.15.29:0/1397082596 shutdown_connections 2024-03-11T19:15:13.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.127+0000 7f850ab80740 1 --2- 172.21.15.29:0/1397082596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe689dd0 0x556dfe68c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.127+0000 7f850ab80740 1 --2- 172.21.15.29:0/1397082596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe684c60 0x556dfe689760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.127+0000 7f850ab80740 1 --2- 172.21.15.29:0/1397082596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684270 0x556dfe684690 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.127+0000 7f850ab80740 1 -- 172.21.15.29:0/1397082596 >> 172.21.15.29:0/1397082596 conn(0x556dfe67f7e0 msgr2=0x556dfe67fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:13.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.127+0000 7f850ab80740 1 -- 172.21.15.29:0/1397082596 shutdown_connections 2024-03-11T19:15:13.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.127+0000 7f850ab80740 1 -- 172.21.15.29:0/1397082596 wait complete. 2024-03-11T19:15:13.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.128+0000 7f850ab80740 1 -- start start 2024-03-11T19:15:13.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.128+0000 7f850ab80740 1 Processor -- start 2024-03-11T19:15:13.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.128+0000 7f850ab80740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684270 0x556dfe68e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.128+0000 7f850ab80740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe684c60 0x556dfe68eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.128+0000 7f850ab80740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe689dd0 0x556dfe695b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.128+0000 7f850ab80740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dfe5ed020 con 0x556dfe684270 2024-03-11T19:15:13.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.128+0000 7f850ab80740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dfe2c0430 con 0x556dfe689dd0 2024-03-11T19:15:13.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.128+0000 7f850ab80740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dfe27ce00 con 0x556dfe684c60 2024-03-11T19:15:13.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.128+0000 7f850ab80740 10 client.0 _ll_register_callbacks cb 0x556dfe594cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:13.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.129+0000 7f8505547640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe684c60 0x556dfe68eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.129+0000 7f8505d48640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684270 0x556dfe68e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.129+0000 7f8506549640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe689dd0 0x556dfe695b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.129+0000 7f8505547640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe684c60 0x556dfe68eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37522/0 (socket says 172.21.15.29:37522) 2024-03-11T19:15:13.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.129+0000 7f8505547640 1 -- 172.21.15.29:0/1705995775 learned_addr learned my addr 172.21.15.29:0/1705995775 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:13.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.129+0000 7f8505547640 1 -- 172.21.15.29:0/1705995775 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe689dd0 msgr2=0x556dfe695b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.129+0000 7f8505547640 1 --2- 172.21.15.29:0/1705995775 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe689dd0 0x556dfe695b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.129+0000 7f8505547640 1 -- 172.21.15.29:0/1705995775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684270 msgr2=0x556dfe68e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.129+0000 7f8505547640 1 --2- 172.21.15.29:0/1705995775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684270 0x556dfe68e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.129+0000 7f8505547640 1 -- 172.21.15.29:0/1705995775 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85000682e0 con 0x556dfe684c60 2024-03-11T19:15:13.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.129+0000 7f8505d48640 1 --2- 172.21.15.29:0/1705995775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684270 0x556dfe68e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:13.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.130+0000 7f8505547640 1 --2- 172.21.15.29:0/1705995775 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe684c60 0x556dfe68eaf0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f84f009a360 tx=0x7f84f0005700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd136a59970ca1bf server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:13.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.130+0000 7f84e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:15:13.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.130+0000 7f84e97fa640 1 -- 172.21.15.29:0/1705995775 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f009e7b0 con 0x556dfe684c60 2024-03-11T19:15:13.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.130+0000 7f84e97fa640 1 -- 172.21.15.29:0/1705995775 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f84f00a2e00 con 0x556dfe684c60 2024-03-11T19:15:13.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.130+0000 7f850ab80740 1 -- 172.21.15.29:0/1705995775 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556dfe2ec7e0 con 0x556dfe684c60 2024-03-11T19:15:13.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.130+0000 7f84e97fa640 1 -- 172.21.15.29:0/1705995775 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84f00b3120 con 0x556dfe684c60 2024-03-11T19:15:13.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.130+0000 7f850ab80740 10 client.26242 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:13.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.130+0000 7f850ab80740 1 -- 172.21.15.29:0/1705995775 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x556dfe5eb6a0 con 0x556dfe684c60 2024-03-11T19:15:13.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.130+0000 7f84e97fa640 1 -- 172.21.15.29:0/1705995775 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x556dfe2ec7e0 con 0x556dfe684c60 2024-03-11T19:15:13.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.132+0000 7f84e97fa640 1 -- 172.21.15.29:0/1705995775 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f84f001f060 con 0x556dfe684c60 2024-03-11T19:15:13.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.132+0000 7f850ab80740 10 client.26242 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:13.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.132+0000 7f850ab80740 10 client.26242 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:15:13.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.132+0000 7f850ab80740 20 client.26242 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:13.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.132+0000 7f850ab80740 1 --2- 172.21.15.29:0/1705995775 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x556dfe69a780 0x556dfe69cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.132+0000 7f850ab80740 4 client.26242 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:15:13.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.132+0000 7f850ab80740 1 -- 172.21.15.29:0/1705995775 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556dfe329720 con 0x556dfe69a780 2024-03-11T19:15:13.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.133+0000 7f8505d48640 1 --2- 172.21.15.29:0/1705995775 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x556dfe69a780 0x556dfe69cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.133+0000 7f84e97fa640 10 client.26242 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:15:13.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.133+0000 7f8505d48640 1 --2- 172.21.15.29:0/1705995775 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x556dfe69a780 0x556dfe69cba0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=154ae493fa9e2646 server_cookie=66a5fb7e9bae1f9a in_seq=0 out_seq=0 2024-03-11T19:15:13.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.134+0000 7f84e97fa640 1 -- 172.21.15.29:0/1705995775 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556dfe329720 con 0x556dfe69a780 2024-03-11T19:15:13.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.134+0000 7f84e97fa640 10 client.26242 handle_command_reply: tid=0 2024-03-11T19:15:13.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.134+0000 7f850ab80740 1 client.26242 shutdown 2024-03-11T19:15:13.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.134+0000 7f850ab80740 1 -- 172.21.15.29:0/1705995775 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe684c60 msgr2=0x556dfe68eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.134+0000 7f850ab80740 1 --2- 172.21.15.29:0/1705995775 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe684c60 0x556dfe68eaf0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f84f009a360 tx=0x7f84f0005700 comp rx=0 tx=0).stop 2024-03-11T19:15:13.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.135+0000 7f850ab80740 1 -- 172.21.15.29:0/1705995775 shutdown_connections 2024-03-11T19:15:13.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.135+0000 7f850ab80740 1 --2- 172.21.15.29:0/1705995775 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x556dfe69a780 0x556dfe69cba0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.135+0000 7f850ab80740 1 --2- 172.21.15.29:0/1705995775 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe689dd0 0x556dfe695b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.135+0000 7f850ab80740 1 --2- 172.21.15.29:0/1705995775 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe684c60 0x556dfe68eaf0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.135+0000 7f850ab80740 1 --2- 172.21.15.29:0/1705995775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684270 0x556dfe68e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.135+0000 7f850ab80740 1 -- 172.21.15.29:0/1705995775 >> 172.21.15.29:0/1705995775 conn(0x556dfe67f7e0 msgr2=0x556dfe67eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:13.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.135+0000 7f84e97fa640 0 client.26242 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:15:13.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.135+0000 7f850ab80740 1 -- 172.21.15.29:0/1705995775 shutdown_connections 2024-03-11T19:15:13.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.135+0000 7f850ab80740 1 -- 172.21.15.29:0/1705995775 wait complete. 2024-03-11T19:15:13.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.135+0000 7f850ab80740 20 client.26242 trim_cache size 0 max 16384 2024-03-11T19:15:13.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.161+0000 7f850ab80740 1 Processor -- start 2024-03-11T19:15:13.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.161+0000 7f850ab80740 1 -- start start 2024-03-11T19:15:13.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.161+0000 7f850ab80740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe684270 0x556dfe68d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.161+0000 7f850ab80740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684c60 0x556dfe695af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.161+0000 7f850ab80740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe689dd0 0x556dfe68d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.161+0000 7f850ab80740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dfe572130 con 0x556dfe684c60 2024-03-11T19:15:13.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.161+0000 7f850ab80740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dfe2ec810 con 0x556dfe684270 2024-03-11T19:15:13.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.161+0000 7f850ab80740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dfe671a40 con 0x556dfe689dd0 2024-03-11T19:15:13.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.162+0000 7f8505547640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684c60 0x556dfe695af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.162+0000 7f8505547640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684c60 0x556dfe695af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33674/0 (socket says 172.21.15.29:33674) 2024-03-11T19:15:13.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.162+0000 7f8505547640 1 -- 172.21.15.29:0/1976934513 learned_addr learned my addr 172.21.15.29:0/1976934513 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:13.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.162+0000 7f8506549640 1 --2- 172.21.15.29:0/1976934513 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe689dd0 0x556dfe68d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.162+0000 7f8505547640 1 -- 172.21.15.29:0/1976934513 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe689dd0 msgr2=0x556dfe68d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.162+0000 7f8505547640 1 --2- 172.21.15.29:0/1976934513 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe689dd0 0x556dfe68d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.162+0000 7f8505547640 1 -- 172.21.15.29:0/1976934513 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe684270 msgr2=0x556dfe68d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.162+0000 7f8505547640 1 --2- 172.21.15.29:0/1976934513 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe684270 0x556dfe68d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.162+0000 7f8505547640 1 -- 172.21.15.29:0/1976934513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556dfe587c20 con 0x556dfe684c60 2024-03-11T19:15:13.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.163+0000 7f8505547640 1 --2- 172.21.15.29:0/1976934513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684c60 0x556dfe695af0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f84f0096ac0 tx=0x7f84f0096240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7673a55b6b3561e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:13.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.163+0000 7f84eb7fe640 1 -- 172.21.15.29:0/1976934513 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f00b3120 con 0x556dfe684c60 2024-03-11T19:15:13.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.163+0000 7f84eb7fe640 1 -- 172.21.15.29:0/1976934513 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f84f0016d90 con 0x556dfe684c60 2024-03-11T19:15:13.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.163+0000 7f84eb7fe640 1 -- 172.21.15.29:0/1976934513 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f009e7b0 con 0x556dfe684c60 2024-03-11T19:15:13.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.164+0000 7f850ab80740 1 -- 172.21.15.29:0/1976934513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684c60 msgr2=0x556dfe695af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.164+0000 7f850ab80740 1 --2- 172.21.15.29:0/1976934513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684c60 0x556dfe695af0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f84f0096ac0 tx=0x7f84f0096240 comp rx=0 tx=0).stop 2024-03-11T19:15:13.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.165+0000 7f850ab80740 1 -- 172.21.15.29:0/1976934513 shutdown_connections 2024-03-11T19:15:13.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.165+0000 7f850ab80740 1 --2- 172.21.15.29:0/1976934513 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe689dd0 0x556dfe68d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.165+0000 7f850ab80740 1 --2- 172.21.15.29:0/1976934513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe684c60 0x556dfe695af0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.165+0000 7f850ab80740 1 --2- 172.21.15.29:0/1976934513 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe684270 0x556dfe68d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.165+0000 7f850ab80740 1 -- 172.21.15.29:0/1976934513 >> 172.21.15.29:0/1976934513 conn(0x556dfe67f7e0 msgr2=0x556dfe686230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:13.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.166+0000 7f850ab80740 1 -- 172.21.15.29:0/1976934513 shutdown_connections 2024-03-11T19:15:13.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.166+0000 7f850ab80740 1 -- 172.21.15.29:0/1976934513 wait complete. 2024-03-11T19:15:13.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.167+0000 7f850ab80740 1 -- start start 2024-03-11T19:15:13.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.167+0000 7f850ab80740 1 Processor -- start 2024-03-11T19:15:13.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.168+0000 7f850ab80740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe684270 0x556dfe6958e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.168+0000 7f850ab80740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe684c60 0x556dfe693ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.168+0000 7f850ab80740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe689dd0 0x556dfe694430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.168+0000 7f8505547640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe684c60 0x556dfe693ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.168+0000 7f8505547640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe684c60 0x556dfe693ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37548/0 (socket says 172.21.15.29:37548) 2024-03-11T19:15:13.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.168+0000 7f8505547640 1 -- 172.21.15.29:0/782991549 learned_addr learned my addr 172.21.15.29:0/782991549 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:13.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.168+0000 7f8505d48640 1 --2- 172.21.15.29:0/782991549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe684270 0x556dfe6958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.168+0000 7f850ab80740 1 -- 172.21.15.29:0/782991549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dfe694a10 con 0x556dfe689dd0 2024-03-11T19:15:13.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.168+0000 7f850ab80740 1 -- 172.21.15.29:0/782991549 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dfe694b80 con 0x556dfe684270 2024-03-11T19:15:13.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.168+0000 7f850ab80740 1 -- 172.21.15.29:0/782991549 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dfe694d40 con 0x556dfe684c60 2024-03-11T19:15:13.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.168+0000 7f850ab80740 10 client.0 _ll_register_callbacks cb 0x7f84f8037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:13.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.169+0000 7f8506549640 1 --2- 172.21.15.29:0/782991549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe689dd0 0x556dfe694430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.169+0000 7f8505547640 1 -- 172.21.15.29:0/782991549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe684270 msgr2=0x556dfe6958e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.169+0000 7f8505547640 1 --2- 172.21.15.29:0/782991549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe684270 0x556dfe6958e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.169+0000 7f8505547640 1 -- 172.21.15.29:0/782991549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe689dd0 msgr2=0x556dfe694430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.169+0000 7f8505547640 1 --2- 172.21.15.29:0/782991549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe689dd0 0x556dfe694430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.169+0000 7f8505547640 1 -- 172.21.15.29:0/782991549 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556dfe5eb6a0 con 0x556dfe684c60 2024-03-11T19:15:13.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.169+0000 7f8506549640 1 --2- 172.21.15.29:0/782991549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe689dd0 0x556dfe694430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:15:13.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.169+0000 7f8505547640 1 --2- 172.21.15.29:0/782991549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe684c60 0x556dfe693ef0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f84f00a28d0 tx=0x7f84f00965e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d1658a417d3959b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:13.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.169+0000 7f84e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:15:13.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.170+0000 7f84e97fa640 1 -- 172.21.15.29:0/782991549 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f009e7b0 con 0x556dfe684c60 2024-03-11T19:15:13.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.170+0000 7f84e97fa640 1 -- 172.21.15.29:0/782991549 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f84f00a2e00 con 0x556dfe684c60 2024-03-11T19:15:13.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.170+0000 7f850ab80740 1 -- 172.21.15.29:0/782991549 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f84f8003970 con 0x556dfe684c60 2024-03-11T19:15:13.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.171+0000 7f84e97fa640 1 -- 172.21.15.29:0/782991549 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84f00b3120 con 0x556dfe684c60 2024-03-11T19:15:13.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.171+0000 7f850ab80740 10 client.26248 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:13.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.171+0000 7f850ab80740 1 -- 172.21.15.29:0/782991549 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x556dfe5ad8c0 con 0x556dfe684c60 2024-03-11T19:15:13.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.172+0000 7f84e97fa640 1 -- 172.21.15.29:0/782991549 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f84f0089020 con 0x556dfe684c60 2024-03-11T19:15:13.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.172+0000 7f84e97fa640 1 -- 172.21.15.29:0/782991549 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f84f001f070 con 0x556dfe684c60 2024-03-11T19:15:13.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.172+0000 7f850ab80740 10 client.26248 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:13.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.172+0000 7f850ab80740 10 client.26248 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:15:13.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.172+0000 7f850ab80740 20 client.26248 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:13.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.172+0000 7f850ab80740 1 --2- 172.21.15.29:0/782991549 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x556dfe6f1f00 0x556dfe6f42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.172+0000 7f850ab80740 4 client.26248 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:15:13.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.172+0000 7f850ab80740 1 -- 172.21.15.29:0/782991549 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556dfe378890 con 0x556dfe6f1f00 2024-03-11T19:15:13.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.173+0000 7f8505d48640 1 --2- 172.21.15.29:0/782991549 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x556dfe6f1f00 0x556dfe6f42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.175+0000 7f84e97fa640 10 client.26248 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:15:13.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.175+0000 7f8505d48640 1 --2- 172.21.15.29:0/782991549 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x556dfe6f1f00 0x556dfe6f42f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=474d8ee9cc8e1bbb server_cookie=7e73a4e13aced7a3 in_seq=0 out_seq=0 2024-03-11T19:15:13.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.175+0000 7f84e97fa640 1 -- 172.21.15.29:0/782991549 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556dfe378890 con 0x556dfe6f1f00 2024-03-11T19:15:13.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.175+0000 7f84e97fa640 10 client.26248 handle_command_reply: tid=0 2024-03-11T19:15:13.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.175+0000 7f850ab80740 1 client.26248 shutdown 2024-03-11T19:15:13.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.176+0000 7f850ab80740 1 -- 172.21.15.29:0/782991549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe684c60 msgr2=0x556dfe693ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.176+0000 7f850ab80740 1 --2- 172.21.15.29:0/782991549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe684c60 0x556dfe693ef0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f84f00a28d0 tx=0x7f84f00965e0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.176+0000 7f850ab80740 1 -- 172.21.15.29:0/782991549 shutdown_connections 2024-03-11T19:15:13.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.176+0000 7f850ab80740 1 --2- 172.21.15.29:0/782991549 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x556dfe6f1f00 0x556dfe6f42f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.176+0000 7f850ab80740 1 --2- 172.21.15.29:0/782991549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556dfe689dd0 0x556dfe694430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.176+0000 7f850ab80740 1 --2- 172.21.15.29:0/782991549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556dfe684c60 0x556dfe693ef0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.176+0000 7f850ab80740 1 --2- 172.21.15.29:0/782991549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556dfe684270 0x556dfe6958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.176+0000 7f850ab80740 1 -- 172.21.15.29:0/782991549 >> 172.21.15.29:0/782991549 conn(0x556dfe67f7e0 msgr2=0x556dfe68e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:13.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.176+0000 7f84e97fa640 0 client.26248 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:15:13.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.176+0000 7f850ab80740 1 -- 172.21.15.29:0/782991549 shutdown_connections 2024-03-11T19:15:13.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.176+0000 7f850ab80740 1 -- 172.21.15.29:0/782991549 wait complete. 2024-03-11T19:15:13.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.176+0000 7f850ab80740 20 client.26248 trim_cache size 0 max 16384 2024-03-11T19:15:13.181 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:15:13.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.177+0000 7f8507fac640 1 -- 172.21.15.29:0/680853886 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f84ec0a1b80 msgr2=0x7f84ec0a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.177+0000 7f8507fac640 1 --2- 172.21.15.29:0/680853886 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f84ec0a1b80 0x7f84ec0a4040 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f84f8048000 tx=0x7f84f8049000 comp rx=0 tx=0).stop 2024-03-11T19:15:13.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.177+0000 7f8507fac640 1 -- 172.21.15.29:0/680853886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8500101270 msgr2=0x7f850019dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.177+0000 7f8507fac640 1 --2- 172.21.15.29:0/680853886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8500101270 0x7f850019dc80 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f84f000e9f0 tx=0x7f84f000edf0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.177+0000 7f8507fac640 1 -- 172.21.15.29:0/680853886 shutdown_connections 2024-03-11T19:15:13.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.177+0000 7f8507fac640 1 --2- 172.21.15.29:0/680853886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8500101c30 0x7f85001a4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.177+0000 7f8507fac640 1 --2- 172.21.15.29:0/680853886 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f84ec0a1b80 0x7f84ec0a4040 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.177+0000 7f8507fac640 1 --2- 172.21.15.29:0/680853886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8500101270 0x7f850019dc80 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.177+0000 7f8507fac640 1 --2- 172.21.15.29:0/680853886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8500100080 0x7f850019d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.177+0000 7f8507fac640 1 -- 172.21.15.29:0/680853886 >> 172.21.15.29:0/680853886 conn(0x7f85000fb610 msgr2=0x7f85000fcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:13.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.178+0000 7f8507fac640 1 -- 172.21.15.29:0/680853886 shutdown_connections 2024-03-11T19:15:13.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.178+0000 7f8507fac640 1 -- 172.21.15.29:0/680853886 wait complete. 2024-03-11T19:15:13.199 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:15:13.199 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:15:13.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:12 smithi190 ceph-mon[36292]: pgmap v513: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T19:15:13.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:12 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2880691922' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:15:13.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.382+0000 7ff5247a7640 1 -- 172.21.15.29:0/165315547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff51c101c00 msgr2=0x7ff51c103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.382+0000 7ff5247a7640 1 --2- 172.21.15.29:0/165315547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff51c101c00 0x7ff51c103ff0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff518009900 tx=0x7ff51802bc30 comp rx=0 tx=0).stop 2024-03-11T19:15:13.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.382+0000 7ff5247a7640 1 -- 172.21.15.29:0/165315547 shutdown_connections 2024-03-11T19:15:13.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.382+0000 7ff5247a7640 1 --2- 172.21.15.29:0/165315547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff51c106e60 0x7ff51c1092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.382+0000 7ff5247a7640 1 --2- 172.21.15.29:0/165315547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51c104530 0x7ff51c106920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.382+0000 7ff5247a7640 1 --2- 172.21.15.29:0/165315547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff51c101c00 0x7ff51c103ff0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.382+0000 7ff5247a7640 1 -- 172.21.15.29:0/165315547 >> 172.21.15.29:0/165315547 conn(0x7ff51c0fb610 msgr2=0x7ff51c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:13.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.382+0000 7ff5247a7640 1 -- 172.21.15.29:0/165315547 shutdown_connections 2024-03-11T19:15:13.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.382+0000 7ff5247a7640 1 -- 172.21.15.29:0/165315547 wait complete. 2024-03-11T19:15:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.383+0000 7ff5247a7640 1 Processor -- start 2024-03-11T19:15:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.383+0000 7ff5247a7640 1 -- start start 2024-03-11T19:15:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.384+0000 7ff5247a7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51c101c00 0x7ff51c195110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.384+0000 7ff5247a7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff51c104530 0x7ff51c195650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.384+0000 7ff5247a7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff51c106e60 0x7ff51c1a5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.384+0000 7ff5247a7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c1a5560 con 0x7ff51c104530 2024-03-11T19:15:13.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.384+0000 7ff5247a7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c1a56a0 con 0x7ff51c106e60 2024-03-11T19:15:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.384+0000 7ff5247a7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c1a57e0 con 0x7ff51c101c00 2024-03-11T19:15:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.384+0000 7ff522543640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51c101c00 0x7ff51c195110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.384+0000 7ff522543640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51c101c00 0x7ff51c195110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37570/0 (socket says 172.21.15.29:37570) 2024-03-11T19:15:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.384+0000 7ff522543640 1 -- 172.21.15.29:0/2860263473 learned_addr learned my addr 172.21.15.29:0/2860263473 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.384+0000 7ff522d44640 1 --2- 172.21.15.29:0/2860263473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff51c106e60 0x7ff51c1a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.384+0000 7ff521d42640 1 --2- 172.21.15.29:0/2860263473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff51c104530 0x7ff51c195650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.385+0000 7ff522543640 1 -- 172.21.15.29:0/2860263473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff51c106e60 msgr2=0x7ff51c1a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.385+0000 7ff522543640 1 --2- 172.21.15.29:0/2860263473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff51c106e60 0x7ff51c1a5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.385+0000 7ff522543640 1 -- 172.21.15.29:0/2860263473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff51c104530 msgr2=0x7ff51c195650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.385+0000 7ff522543640 1 --2- 172.21.15.29:0/2860263473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff51c104530 0x7ff51c195650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.385+0000 7ff522543640 1 -- 172.21.15.29:0/2860263473 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff518009560 con 0x7ff51c101c00 2024-03-11T19:15:13.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.385+0000 7ff522543640 1 --2- 172.21.15.29:0/2860263473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51c101c00 0x7ff51c195110 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff518005ab0 tx=0x7ff5180042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:13.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.385+0000 7ff5137fe640 1 -- 172.21.15.29:0/2860263473 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5180044e0 con 0x7ff51c101c00 2024-03-11T19:15:13.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.385+0000 7ff5247a7640 1 -- 172.21.15.29:0/2860263473 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff51c1a59e0 con 0x7ff51c101c00 2024-03-11T19:15:13.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.385+0000 7ff5247a7640 1 -- 172.21.15.29:0/2860263473 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff51c1a5fc0 con 0x7ff51c101c00 2024-03-11T19:15:13.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.386+0000 7ff5137fe640 1 -- 172.21.15.29:0/2860263473 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff518003cb0 con 0x7ff51c101c00 2024-03-11T19:15:13.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.386+0000 7ff5137fe640 1 -- 172.21.15.29:0/2860263473 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5180044e0 con 0x7ff51c101c00 2024-03-11T19:15:13.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.389+0000 7ff527377740 1 Processor -- start 2024-03-11T19:15:13.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.389+0000 7ff527377740 1 -- start start 2024-03-11T19:15:13.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.389+0000 7ff527377740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e620787270 0x55e620787690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.389+0000 7ff527377740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e620787c60 0x55e62078c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.389+0000 7ff527377740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e62078cdd0 0x55e62078f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.389+0000 7ff527377740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e620743960 con 0x55e620787c60 2024-03-11T19:15:13.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.389+0000 7ff527377740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6206f7310 con 0x55e62078cdd0 2024-03-11T19:15:13.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.389+0000 7ff527377740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6206f4840 con 0x55e620787270 2024-03-11T19:15:13.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.390+0000 7ff522d44640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e620787270 0x55e620787690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.390+0000 7ff522d44640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e620787270 0x55e620787690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37572/0 (socket says 172.21.15.29:37572) 2024-03-11T19:15:13.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.390+0000 7ff522d44640 1 -- 172.21.15.29:0/4153552796 learned_addr learned my addr 172.21.15.29:0/4153552796 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:13.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.390+0000 7ff521d42640 1 --2- 172.21.15.29:0/4153552796 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e62078cdd0 0x55e62078f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.390+0000 7ff521d42640 1 -- 172.21.15.29:0/4153552796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e620787270 msgr2=0x55e620787690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.390+0000 7ff521d42640 1 --2- 172.21.15.29:0/4153552796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e620787270 0x55e620787690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.390+0000 7ff521d42640 1 -- 172.21.15.29:0/4153552796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e620787c60 msgr2=0x55e62078c760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:15:13.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.390+0000 7ff521d42640 1 --2- 172.21.15.29:0/4153552796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e620787c60 0x55e62078c760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.390+0000 7ff521d42640 1 -- 172.21.15.29:0/4153552796 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6206b2060 con 0x55e62078cdd0 2024-03-11T19:15:13.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.391+0000 7ff521d42640 1 --2- 172.21.15.29:0/4153552796 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e62078cdd0 0x55e62078f260 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff51c068e40 tx=0x7ff50c009e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fa4a115c4062aa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:13.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.391+0000 7ff510ff9640 1 -- 172.21.15.29:0/4153552796 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c00ea80 con 0x55e62078cdd0 2024-03-11T19:15:13.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.391+0000 7ff510ff9640 1 -- 172.21.15.29:0/4153552796 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff50c00ebe0 con 0x55e62078cdd0 2024-03-11T19:15:13.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.392+0000 7ff5137fe640 1 -- 172.21.15.29:0/2860263473 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff518005480 con 0x7ff51c101c00 2024-03-11T19:15:13.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.392+0000 7ff510ff9640 1 -- 172.21.15.29:0/4153552796 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c012630 con 0x55e62078cdd0 2024-03-11T19:15:13.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.393+0000 7ff527377740 1 -- 172.21.15.29:0/4153552796 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e62078cdd0 msgr2=0x55e62078f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.393+0000 7ff527377740 1 --2- 172.21.15.29:0/4153552796 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e62078cdd0 0x55e62078f260 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff51c068e40 tx=0x7ff50c009e00 comp rx=0 tx=0).stop 2024-03-11T19:15:13.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.393+0000 7ff5137fe640 1 --2- 172.21.15.29:0/2860263473 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff5080a1b60 0x7ff5080a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.393+0000 7ff5137fe640 1 -- 172.21.15.29:0/2860263473 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff51802de40 con 0x7ff51c101c00 2024-03-11T19:15:13.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.394+0000 7ff527377740 1 -- 172.21.15.29:0/4153552796 shutdown_connections 2024-03-11T19:15:13.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.394+0000 7ff527377740 1 --2- 172.21.15.29:0/4153552796 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e62078cdd0 0x55e62078f260 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.394+0000 7ff527377740 1 --2- 172.21.15.29:0/4153552796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e620787c60 0x55e62078c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.394+0000 7ff527377740 1 --2- 172.21.15.29:0/4153552796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e620787270 0x55e620787690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.394+0000 7ff527377740 1 -- 172.21.15.29:0/4153552796 >> 172.21.15.29:0/4153552796 conn(0x55e6207827e0 msgr2=0x55e620782be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:13.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.394+0000 7ff522d44640 1 --2- 172.21.15.29:0/2860263473 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff5080a1b60 0x7ff5080a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.394+0000 7ff527377740 1 -- 172.21.15.29:0/4153552796 shutdown_connections 2024-03-11T19:15:13.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.394+0000 7ff527377740 1 -- 172.21.15.29:0/4153552796 wait complete. 2024-03-11T19:15:13.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.395+0000 7ff527377740 1 -- start start 2024-03-11T19:15:13.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.395+0000 7ff527377740 1 Processor -- start 2024-03-11T19:15:13.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.396+0000 7ff522d44640 1 --2- 172.21.15.29:0/2860263473 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff5080a1b60 0x7ff5080a4020 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x55e6207884c0 tx=0x7ff51400d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:13.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.396+0000 7ff527377740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e620787270 0x55e6207914e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.396+0000 7ff527377740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e620787c60 0x55e620791a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.396+0000 7ff527377740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e62078cdd0 0x55e620798a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.396+0000 7ff527377740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6206f0020 con 0x55e62078cdd0 2024-03-11T19:15:13.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.396+0000 7ff527377740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6203c3430 con 0x55e620787c60 2024-03-11T19:15:13.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.396+0000 7ff527377740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e62037fe00 con 0x55e620787270 2024-03-11T19:15:13.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.396+0000 7ff527377740 10 client.0 _ll_register_callbacks cb 0x55e620697cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:13.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.397+0000 7ff522d44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e62078cdd0 0x55e620798a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.397+0000 7ff522d44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e62078cdd0 0x55e620798a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33702/0 (socket says 172.21.15.29:33702) 2024-03-11T19:15:13.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.397+0000 7ff522d44640 1 -- 172.21.15.29:0/1258373801 learned_addr learned my addr 172.21.15.29:0/1258373801 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:13.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.397+0000 7ff522d44640 1 -- 172.21.15.29:0/1258373801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e620787270 msgr2=0x55e6207914e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.397+0000 7ff522d44640 1 --2- 172.21.15.29:0/1258373801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e620787270 0x55e6207914e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.397+0000 7ff522d44640 1 -- 172.21.15.29:0/1258373801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e620787c60 msgr2=0x55e620791a20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:15:13.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.397+0000 7ff522d44640 1 --2- 172.21.15.29:0/1258373801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e620787c60 0x55e620791a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.397+0000 7ff522d44640 1 -- 172.21.15.29:0/1258373801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff50c0095d0 con 0x55e62078cdd0 2024-03-11T19:15:13.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.397+0000 7ff522d44640 1 --2- 172.21.15.29:0/1258373801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e62078cdd0 0x55e620798a60 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7ff514018e30 tx=0x7ff514004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae5c66d3af8831e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:13.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.398+0000 7ff505ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:15:13.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.398+0000 7ff505ffb640 1 -- 172.21.15.29:0/1258373801 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff514010040 con 0x55e62078cdd0 2024-03-11T19:15:13.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.398+0000 7ff527377740 1 -- 172.21.15.29:0/1258373801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e6203ef7e0 con 0x55e62078cdd0 2024-03-11T19:15:13.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.398+0000 7ff505ffb640 1 -- 172.21.15.29:0/1258373801 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff514016e40 con 0x55e62078cdd0 2024-03-11T19:15:13.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.398+0000 7ff505ffb640 1 -- 172.21.15.29:0/1258373801 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff514009dc0 con 0x55e62078cdd0 2024-03-11T19:15:13.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.398+0000 7ff527377740 10 client.16728 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:13.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.398+0000 7ff527377740 1 -- 172.21.15.29:0/1258373801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e6206ee6a0 con 0x55e62078cdd0 2024-03-11T19:15:13.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.399+0000 7ff505ffb640 1 -- 172.21.15.29:0/1258373801 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55e6203ef7e0 con 0x55e62078cdd0 2024-03-11T19:15:13.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.399+0000 7ff505ffb640 1 -- 172.21.15.29:0/1258373801 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff51402a870 con 0x55e62078cdd0 2024-03-11T19:15:13.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.399+0000 7ff527377740 10 client.16728 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:13.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.399+0000 7ff527377740 10 client.16728 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:15:13.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.399+0000 7ff527377740 20 client.16728 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:13.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.399+0000 7ff527377740 1 --2- 172.21.15.29:0/1258373801 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55e62079d8d0 0x55e62079fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.399+0000 7ff527377740 4 client.16728 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:15:13.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.399+0000 7ff527377740 1 -- 172.21.15.29:0/1258373801 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e62042c720 con 0x55e62079d8d0 2024-03-11T19:15:13.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.399+0000 7ff522543640 1 --2- 172.21.15.29:0/1258373801 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55e62079d8d0 0x55e62079fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.400+0000 7ff522543640 1 --2- 172.21.15.29:0/1258373801 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55e62079d8d0 0x55e62079fd30 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a3682b300a38c0ae server_cookie=698c7b8865d44378 in_seq=0 out_seq=0 2024-03-11T19:15:13.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.400+0000 7ff505ffb640 10 client.16728 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:15:13.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.401+0000 7ff505ffb640 1 -- 172.21.15.29:0/1258373801 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e62042c720 con 0x55e62079d8d0 2024-03-11T19:15:13.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.401+0000 7ff505ffb640 10 client.16728 handle_command_reply: tid=0 2024-03-11T19:15:13.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.401+0000 7ff527377740 1 client.16728 shutdown 2024-03-11T19:15:13.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.402+0000 7ff527377740 1 -- 172.21.15.29:0/1258373801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e62078cdd0 msgr2=0x55e620798a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.402+0000 7ff527377740 1 --2- 172.21.15.29:0/1258373801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e62078cdd0 0x55e620798a60 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7ff514018e30 tx=0x7ff514004a40 comp rx=0 tx=0).stop 2024-03-11T19:15:13.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.403+0000 7ff527377740 1 -- 172.21.15.29:0/1258373801 shutdown_connections 2024-03-11T19:15:13.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.403+0000 7ff527377740 1 --2- 172.21.15.29:0/1258373801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e62078cdd0 0x55e620798a60 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.403+0000 7ff527377740 1 --2- 172.21.15.29:0/1258373801 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55e62079d8d0 0x55e62079fd30 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.403+0000 7ff527377740 1 --2- 172.21.15.29:0/1258373801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e620787c60 0x55e620791a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.403+0000 7ff527377740 1 --2- 172.21.15.29:0/1258373801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e620787270 0x55e6207914e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.403+0000 7ff527377740 1 -- 172.21.15.29:0/1258373801 >> 172.21.15.29:0/1258373801 conn(0x55e6207827e0 msgr2=0x55e620781b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:13.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.403+0000 7ff505ffb640 0 client.16728 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:15:13.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.404+0000 7ff527377740 1 -- 172.21.15.29:0/1258373801 shutdown_connections 2024-03-11T19:15:13.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.404+0000 7ff527377740 1 -- 172.21.15.29:0/1258373801 wait complete. 2024-03-11T19:15:13.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.404+0000 7ff527377740 20 client.16728 trim_cache size 0 max 16384 2024-03-11T19:15:13.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.426+0000 7ff527377740 1 Processor -- start 2024-03-11T19:15:13.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.426+0000 7ff527377740 1 -- start start 2024-03-11T19:15:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.427+0000 7ff527377740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e620787270 0x55e62078ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.427+0000 7ff527377740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e620787c60 0x55e620790510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.427+0000 7ff527377740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e62078cdd0 0x55e620798a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.427+0000 7ff527377740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e620675130 con 0x55e620787c60 2024-03-11T19:15:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.427+0000 7ff527377740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e620782df0 con 0x55e620787270 2024-03-11T19:15:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.427+0000 7ff527377740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e620799910 con 0x55e62078cdd0 2024-03-11T19:15:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.427+0000 7ff522543640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e620787270 0x55e62078ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.427+0000 7ff522543640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e620787270 0x55e62078ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:54112/0 (socket says 172.21.15.29:54112) 2024-03-11T19:15:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.427+0000 7ff522543640 1 -- 172.21.15.29:0/170891730 learned_addr learned my addr 172.21.15.29:0/170891730 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.427+0000 7ff521d42640 1 --2- 172.21.15.29:0/170891730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e620787c60 0x55e620790510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.427+0000 7ff522d44640 1 --2- 172.21.15.29:0/170891730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e62078cdd0 0x55e620798a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.427+0000 7ff522543640 1 -- 172.21.15.29:0/170891730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e62078cdd0 msgr2=0x55e620798a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.427+0000 7ff522543640 1 --2- 172.21.15.29:0/170891730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e62078cdd0 0x55e620798a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.427+0000 7ff522543640 1 -- 172.21.15.29:0/170891730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e620787c60 msgr2=0x55e620790510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.427+0000 7ff522543640 1 --2- 172.21.15.29:0/170891730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e620787c60 0x55e620790510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.427+0000 7ff522543640 1 -- 172.21.15.29:0/170891730 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e62068ac20 con 0x55e620787270 2024-03-11T19:15:13.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.427+0000 7ff522543640 1 --2- 172.21.15.29:0/170891730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e620787270 0x55e62078ffd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff5180ed4e0 tx=0x7ff51802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b52b923d2a7043a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:13.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.428+0000 7ff507fff640 1 -- 172.21.15.29:0/170891730 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5180044e0 con 0x55e620787270 2024-03-11T19:15:13.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.428+0000 7ff507fff640 1 -- 172.21.15.29:0/170891730 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff5180bc950 con 0x55e620787270 2024-03-11T19:15:13.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.428+0000 7ff507fff640 1 -- 172.21.15.29:0/170891730 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5180b8fc0 con 0x55e620787270 2024-03-11T19:15:13.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.429+0000 7ff527377740 1 -- 172.21.15.29:0/170891730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e620787270 msgr2=0x55e62078ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.429+0000 7ff527377740 1 --2- 172.21.15.29:0/170891730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e620787270 0x55e62078ffd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff5180ed4e0 tx=0x7ff51802bc30 comp rx=0 tx=0).stop 2024-03-11T19:15:13.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.429+0000 7ff527377740 1 -- 172.21.15.29:0/170891730 shutdown_connections 2024-03-11T19:15:13.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.429+0000 7ff527377740 1 --2- 172.21.15.29:0/170891730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e62078cdd0 0x55e620798a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.429+0000 7ff527377740 1 --2- 172.21.15.29:0/170891730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e620787c60 0x55e620790510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.429+0000 7ff527377740 1 --2- 172.21.15.29:0/170891730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e620787270 0x55e62078ffd0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.429+0000 7ff527377740 1 -- 172.21.15.29:0/170891730 >> 172.21.15.29:0/170891730 conn(0x55e6207827e0 msgr2=0x55e6207914e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:13.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.429+0000 7ff527377740 1 -- 172.21.15.29:0/170891730 shutdown_connections 2024-03-11T19:15:13.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.429+0000 7ff527377740 1 -- 172.21.15.29:0/170891730 wait complete. 2024-03-11T19:15:13.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.429+0000 7ff527377740 1 -- start start 2024-03-11T19:15:13.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.429+0000 7ff527377740 1 Processor -- start 2024-03-11T19:15:13.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.430+0000 7ff527377740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e620787270 0x55e6207969a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.430+0000 7ff527377740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e620787c60 0x55e620796ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.430+0000 7ff527377740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e62078cdd0 0x55e620797ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.430+0000 7ff527377740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e620798430 con 0x55e620787270 2024-03-11T19:15:13.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.430+0000 7ff527377740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6207985a0 con 0x55e620787c60 2024-03-11T19:15:13.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.430+0000 7ff527377740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e620798760 con 0x55e62078cdd0 2024-03-11T19:15:13.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.430+0000 7ff527377740 10 client.0 _ll_register_callbacks cb 0x7ff50c0093c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:13.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.430+0000 7ff522543640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e620787270 0x55e6207969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.430+0000 7ff522d44640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e62078cdd0 0x55e620797ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.430+0000 7ff521d42640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e620787c60 0x55e620796ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.430+0000 7ff522543640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e620787270 0x55e6207969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33724/0 (socket says 172.21.15.29:33724) 2024-03-11T19:15:13.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.430+0000 7ff522543640 1 -- 172.21.15.29:0/4094539917 learned_addr learned my addr 172.21.15.29:0/4094539917 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:13.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.430+0000 7ff521d42640 1 -- 172.21.15.29:0/4094539917 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e62078cdd0 msgr2=0x55e620797ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.430+0000 7ff521d42640 1 --2- 172.21.15.29:0/4094539917 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e62078cdd0 0x55e620797ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.430+0000 7ff521d42640 1 -- 172.21.15.29:0/4094539917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e620787270 msgr2=0x55e6207969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.431+0000 7ff521d42640 1 --2- 172.21.15.29:0/4094539917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e620787270 0x55e6207969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.431+0000 7ff521d42640 1 -- 172.21.15.29:0/4094539917 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff51c1a5fc0 con 0x55e620787c60 2024-03-11T19:15:13.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.431+0000 7ff522d44640 1 --2- 172.21.15.29:0/4094539917 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e62078cdd0 0x55e620797ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:13.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.431+0000 7ff522543640 1 --2- 172.21.15.29:0/4094539917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e620787270 0x55e6207969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:13.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.431+0000 7ff521d42640 1 --2- 172.21.15.29:0/4094539917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e620787c60 0x55e620796ee0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x55e620783720 tx=0x7ff50c013b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d94207a917052131 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:13.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.431+0000 7ff505ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:15:13.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.431+0000 7ff505ffb640 1 -- 172.21.15.29:0/4094539917 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c022070 con 0x55e620787c60 2024-03-11T19:15:13.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.431+0000 7ff505ffb640 1 -- 172.21.15.29:0/4094539917 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff50c013c80 con 0x55e620787c60 2024-03-11T19:15:13.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.432+0000 7ff527377740 1 -- 172.21.15.29:0/4094539917 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e62078cb40 con 0x55e620787c60 2024-03-11T19:15:13.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.432+0000 7ff505ffb640 1 -- 172.21.15.29:0/4094539917 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff50c00ea80 con 0x55e620787c60 2024-03-11T19:15:13.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.432+0000 7ff527377740 10 client.26252 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:13.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.432+0000 7ff527377740 1 -- 172.21.15.29:0/4094539917 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e6206b08c0 con 0x55e620787c60 2024-03-11T19:15:13.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.432+0000 7ff505ffb640 1 -- 172.21.15.29:0/4094539917 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff50c013e40 con 0x55e620787c60 2024-03-11T19:15:13.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.432+0000 7ff505ffb640 1 -- 172.21.15.29:0/4094539917 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff50c011070 con 0x55e620787c60 2024-03-11T19:15:13.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.433+0000 7ff527377740 10 client.26252 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:13.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.433+0000 7ff527377740 10 client.26252 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:15:13.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.433+0000 7ff527377740 20 client.26252 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:13.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.433+0000 7ff527377740 1 --2- 172.21.15.29:0/4094539917 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55e6207f4f80 0x55e6207f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.433+0000 7ff527377740 4 client.26252 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:15:13.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.433+0000 7ff527377740 1 -- 172.21.15.29:0/4094539917 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e62078cb40 con 0x55e6207f4f80 2024-03-11T19:15:13.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.433+0000 7ff522543640 1 --2- 172.21.15.29:0/4094539917 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55e6207f4f80 0x55e6207f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.434+0000 7ff522543640 1 --2- 172.21.15.29:0/4094539917 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55e6207f4f80 0x55e6207f73a0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=15c6d87a0c415941 server_cookie=cf7dba4ad0afa001 in_seq=0 out_seq=0 2024-03-11T19:15:13.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.434+0000 7ff505ffb640 10 client.26252 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:15:13.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.434+0000 7ff505ffb640 1 -- 172.21.15.29:0/4094539917 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e62078cb40 con 0x55e6207f4f80 2024-03-11T19:15:13.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.434+0000 7ff505ffb640 10 client.26252 handle_command_reply: tid=0 2024-03-11T19:15:13.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.434+0000 7ff527377740 1 client.26252 shutdown 2024-03-11T19:15:13.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.434+0000 7ff527377740 1 -- 172.21.15.29:0/4094539917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e620787c60 msgr2=0x55e620796ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.434+0000 7ff527377740 1 --2- 172.21.15.29:0/4094539917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e620787c60 0x55e620796ee0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x55e620783720 tx=0x7ff50c013b90 comp rx=0 tx=0).stop 2024-03-11T19:15:13.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.434+0000 7ff527377740 1 -- 172.21.15.29:0/4094539917 shutdown_connections 2024-03-11T19:15:13.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.434+0000 7ff527377740 1 --2- 172.21.15.29:0/4094539917 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e62078cdd0 0x55e620797ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.434+0000 7ff527377740 1 --2- 172.21.15.29:0/4094539917 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55e6207f4f80 0x55e6207f73a0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.434+0000 7ff527377740 1 --2- 172.21.15.29:0/4094539917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e620787c60 0x55e620796ee0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.435+0000 7ff505ffb640 0 client.26252 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:15:13.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.435+0000 7ff527377740 1 --2- 172.21.15.29:0/4094539917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e620787270 0x55e6207969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.435+0000 7ff527377740 1 -- 172.21.15.29:0/4094539917 >> 172.21.15.29:0/4094539917 conn(0x55e6207827e0 msgr2=0x55e6207891b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:13.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.435+0000 7ff527377740 1 -- 172.21.15.29:0/4094539917 shutdown_connections 2024-03-11T19:15:13.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.435+0000 7ff527377740 1 -- 172.21.15.29:0/4094539917 wait complete. 2024-03-11T19:15:13.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.435+0000 7ff527377740 20 client.26252 trim_cache size 0 max 16384 2024-03-11T19:15:13.442 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:15:13.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.436+0000 7ff5247a7640 1 -- 172.21.15.29:0/2860263473 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff5080a1b60 msgr2=0x7ff5080a4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.436+0000 7ff5247a7640 1 --2- 172.21.15.29:0/2860263473 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff5080a1b60 0x7ff5080a4020 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x55e6207884c0 tx=0x7ff51400d040 comp rx=0 tx=0).stop 2024-03-11T19:15:13.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.436+0000 7ff5247a7640 1 -- 172.21.15.29:0/2860263473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51c101c00 msgr2=0x7ff51c195110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.436+0000 7ff5247a7640 1 --2- 172.21.15.29:0/2860263473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51c101c00 0x7ff51c195110 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff518005ab0 tx=0x7ff5180042f0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.436+0000 7ff5247a7640 1 -- 172.21.15.29:0/2860263473 shutdown_connections 2024-03-11T19:15:13.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.436+0000 7ff5247a7640 1 --2- 172.21.15.29:0/2860263473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff51c106e60 0x7ff51c1a5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.436+0000 7ff5247a7640 1 --2- 172.21.15.29:0/2860263473 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff5080a1b60 0x7ff5080a4020 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.436+0000 7ff5247a7640 1 --2- 172.21.15.29:0/2860263473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff51c104530 0x7ff51c195650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.436+0000 7ff5247a7640 1 --2- 172.21.15.29:0/2860263473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51c101c00 0x7ff51c195110 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.436+0000 7ff5247a7640 1 -- 172.21.15.29:0/2860263473 >> 172.21.15.29:0/2860263473 conn(0x7ff51c0fb610 msgr2=0x7ff51c1012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:13.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.436+0000 7ff5247a7640 1 -- 172.21.15.29:0/2860263473 shutdown_connections 2024-03-11T19:15:13.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.436+0000 7ff5247a7640 1 -- 172.21.15.29:0/2860263473 wait complete. 2024-03-11T19:15:13.461 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:15:13.461 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:15:13.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.633+0000 7f248f0c9640 1 -- 172.21.15.29:0/2224550342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24881077c0 msgr2=0x7f2488107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.633+0000 7f248f0c9640 1 --2- 172.21.15.29:0/2224550342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24881077c0 0x7f2488107be0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f247c009900 tx=0x7f247c02bc40 comp rx=0 tx=0).stop 2024-03-11T19:15:13.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.633+0000 7f248f0c9640 1 -- 172.21.15.29:0/2224550342 shutdown_connections 2024-03-11T19:15:13.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.633+0000 7f248f0c9640 1 --2- 172.21.15.29:0/2224550342 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2488109370 0x7f248810fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.633+0000 7f248f0c9640 1 --2- 172.21.15.29:0/2224550342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24881089b0 0x7f2488108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.633+0000 7f248f0c9640 1 --2- 172.21.15.29:0/2224550342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24881077c0 0x7f2488107be0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.633+0000 7f248f0c9640 1 -- 172.21.15.29:0/2224550342 >> 172.21.15.29:0/2224550342 conn(0x7f2488076750 msgr2=0x7f2488078b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:13.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.633+0000 7f248f0c9640 1 -- 172.21.15.29:0/2224550342 shutdown_connections 2024-03-11T19:15:13.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.633+0000 7f248f0c9640 1 -- 172.21.15.29:0/2224550342 wait complete. 2024-03-11T19:15:13.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.634+0000 7f248f0c9640 1 Processor -- start 2024-03-11T19:15:13.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.634+0000 7f248f0c9640 1 -- start start 2024-03-11T19:15:13.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.634+0000 7f248f0c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24881077c0 0x7f248819d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.634+0000 7f248f0c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24881089b0 0x7f248819dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.634+0000 7f248f0c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2488109370 0x7f24881a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.634+0000 7f248f0c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24881a5310 con 0x7f24881077c0 2024-03-11T19:15:13.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.634+0000 7f248f0c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24881a5480 con 0x7f2488109370 2024-03-11T19:15:13.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.634+0000 7f248f0c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24881a5640 con 0x7f24881089b0 2024-03-11T19:15:13.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.634+0000 7f248d666640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2488109370 0x7f24881a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.634+0000 7f248ce65640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24881077c0 0x7f248819d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.634+0000 7f248d666640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2488109370 0x7f24881a4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:54138/0 (socket says 172.21.15.29:54138) 2024-03-11T19:15:13.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.635+0000 7f248d666640 1 -- 172.21.15.29:0/1372597224 learned_addr learned my addr 172.21.15.29:0/1372597224 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:13.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.635+0000 7f248ce65640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24881077c0 0x7f248819d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33734/0 (socket says 172.21.15.29:33734) 2024-03-11T19:15:13.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.635+0000 7f2487fff640 1 --2- 172.21.15.29:0/1372597224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24881089b0 0x7f248819dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.635+0000 7f248d666640 1 -- 172.21.15.29:0/1372597224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24881089b0 msgr2=0x7f248819dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.635+0000 7f248d666640 1 --2- 172.21.15.29:0/1372597224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24881089b0 0x7f248819dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.635+0000 7f248d666640 1 -- 172.21.15.29:0/1372597224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24881077c0 msgr2=0x7f248819d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.635+0000 7f248d666640 1 --2- 172.21.15.29:0/1372597224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24881077c0 0x7f248819d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.635+0000 7f248d666640 1 -- 172.21.15.29:0/1372597224 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247c009560 con 0x7f2488109370 2024-03-11T19:15:13.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.635+0000 7f248d666640 1 --2- 172.21.15.29:0/1372597224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2488109370 0x7f24881a4dd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f248000b970 tx=0x7f248000be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:13.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.635+0000 7f2485ffb640 1 -- 172.21.15.29:0/1372597224 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248000cb40 con 0x7f2488109370 2024-03-11T19:15:13.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.636+0000 7f2485ffb640 1 -- 172.21.15.29:0/1372597224 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f248000cca0 con 0x7f2488109370 2024-03-11T19:15:13.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.636+0000 7f2485ffb640 1 -- 172.21.15.29:0/1372597224 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24800076e0 con 0x7f2488109370 2024-03-11T19:15:13.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.636+0000 7f248f0c9640 1 -- 172.21.15.29:0/1372597224 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24881a5920 con 0x7f2488109370 2024-03-11T19:15:13.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.636+0000 7f248f0c9640 1 -- 172.21.15.29:0/1372597224 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24881a5e00 con 0x7f2488109370 2024-03-11T19:15:13.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.639+0000 7f2485ffb640 1 -- 172.21.15.29:0/1372597224 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2480014440 con 0x7f2488109370 2024-03-11T19:15:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.639+0000 7f2491d70740 1 Processor -- start 2024-03-11T19:15:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f2491d70740 1 -- start start 2024-03-11T19:15:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f2485ffb640 1 --2- 172.21.15.29:0/1372597224 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f24780a1b30 0x7f24780a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f2491d70740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192130270 0x564192130690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f2491d70740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192130c60 0x564192135760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f2491d70740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192135dd0 0x564192138260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f2491d70740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641920ec960 con 0x564192130c60 2024-03-11T19:15:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f2491d70740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641920a0310 con 0x564192135dd0 2024-03-11T19:15:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f2491d70740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56419209d840 con 0x564192130270 2024-03-11T19:15:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f248d666640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192135dd0 0x564192138260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f248d666640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192135dd0 0x564192138260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:54140/0 (socket says 172.21.15.29:54140) 2024-03-11T19:15:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f248d666640 1 -- 172.21.15.29:0/2590076762 learned_addr learned my addr 172.21.15.29:0/2590076762 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f2487fff640 1 --2- 172.21.15.29:0/1372597224 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f24780a1b30 0x7f24780a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f248ce65640 1 --2- 172.21.15.29:0/2590076762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192130270 0x564192130690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f2487fff640 1 --2- 172.21.15.29:0/2590076762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192130c60 0x564192135760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f248d666640 1 -- 172.21.15.29:0/2590076762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192130270 msgr2=0x564192130690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f248d666640 1 --2- 172.21.15.29:0/2590076762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192130270 0x564192130690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f248d666640 1 -- 172.21.15.29:0/2590076762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192130c60 msgr2=0x564192135760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f248d666640 1 --2- 172.21.15.29:0/2590076762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192130c60 0x564192135760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f2485ffb640 1 -- 172.21.15.29:0/1372597224 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f24800ccc50 con 0x7f2488109370 2024-03-11T19:15:13.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.640+0000 7f248d666640 1 -- 172.21.15.29:0/2590076762 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56419205b060 con 0x564192135dd0 2024-03-11T19:15:13.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.641+0000 7f2487fff640 1 --2- 172.21.15.29:0/2590076762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192130c60 0x564192135760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:15:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.641+0000 7f248d666640 1 --2- 172.21.15.29:0/2590076762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192135dd0 0x564192138260 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2480009a00 tx=0x7f2480009970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a568c6026d9f3b15 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.641+0000 7f2472ffd640 1 -- 172.21.15.29:0/2590076762 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24800995d0 con 0x564192135dd0 2024-03-11T19:15:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.641+0000 7f2472ffd640 1 -- 172.21.15.29:0/2590076762 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f24800a1cc0 con 0x564192135dd0 2024-03-11T19:15:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.641+0000 7f2472ffd640 1 -- 172.21.15.29:0/2590076762 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24800a0040 con 0x564192135dd0 2024-03-11T19:15:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.642+0000 7f2491d70740 1 -- 172.21.15.29:0/2590076762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192135dd0 msgr2=0x564192138260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.642+0000 7f2491d70740 1 --2- 172.21.15.29:0/2590076762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192135dd0 0x564192138260 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2480009a00 tx=0x7f2480009970 comp rx=0 tx=0).stop 2024-03-11T19:15:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.642+0000 7f2487fff640 1 --2- 172.21.15.29:0/1372597224 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f24780a1b30 0x7f24780a3ff0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x564192131060 tx=0x7f2474009710 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.642+0000 7f2491d70740 1 -- 172.21.15.29:0/2590076762 shutdown_connections 2024-03-11T19:15:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.642+0000 7f2491d70740 1 --2- 172.21.15.29:0/2590076762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192135dd0 0x564192138260 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.642+0000 7f2491d70740 1 --2- 172.21.15.29:0/2590076762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192130c60 0x564192135760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.642+0000 7f2491d70740 1 --2- 172.21.15.29:0/2590076762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192130270 0x564192130690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.642+0000 7f2491d70740 1 -- 172.21.15.29:0/2590076762 >> 172.21.15.29:0/2590076762 conn(0x56419212b7e0 msgr2=0x56419212bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.642+0000 7f2491d70740 1 -- 172.21.15.29:0/2590076762 shutdown_connections 2024-03-11T19:15:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.643+0000 7f2491d70740 1 -- 172.21.15.29:0/2590076762 wait complete. 2024-03-11T19:15:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.643+0000 7f2491d70740 1 -- start start 2024-03-11T19:15:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.643+0000 7f2491d70740 1 Processor -- start 2024-03-11T19:15:13.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.644+0000 7f2491d70740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130270 0x56419213a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.644+0000 7f2491d70740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192130c60 0x56419213a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.644+0000 7f2491d70740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192135dd0 0x564192141940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.644+0000 7f2491d70740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564192099020 con 0x564192135dd0 2024-03-11T19:15:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.644+0000 7f2491d70740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564191d6c430 con 0x564192130270 2024-03-11T19:15:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.644+0000 7f2491d70740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x564191d28e00 con 0x564192130c60 2024-03-11T19:15:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.644+0000 7f2491d70740 10 client.0 _ll_register_callbacks cb 0x564192040cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.644+0000 7f248ce65640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130270 0x56419213a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.644+0000 7f248ce65640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130270 0x56419213a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:54144/0 (socket says 172.21.15.29:54144) 2024-03-11T19:15:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.644+0000 7f248ce65640 1 -- 172.21.15.29:0/1561749887 learned_addr learned my addr 172.21.15.29:0/1561749887 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.644+0000 7f248d666640 1 --2- 172.21.15.29:0/1561749887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192135dd0 0x564192141940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.644+0000 7f2487fff640 1 --2- 172.21.15.29:0/1561749887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192130c60 0x56419213a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.645+0000 7f248ce65640 1 -- 172.21.15.29:0/1561749887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192130c60 msgr2=0x56419213a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.645+0000 7f248ce65640 1 --2- 172.21.15.29:0/1561749887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192130c60 0x56419213a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.645+0000 7f248ce65640 1 -- 172.21.15.29:0/1561749887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192135dd0 msgr2=0x564192141940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.645+0000 7f248ce65640 1 --2- 172.21.15.29:0/1561749887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192135dd0 0x564192141940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.645+0000 7f248ce65640 1 -- 172.21.15.29:0/1561749887 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24881a5e00 con 0x564192130270 2024-03-11T19:15:13.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.645+0000 7f2487fff640 1 --2- 172.21.15.29:0/1561749887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192130c60 0x56419213a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:15:13.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.645+0000 7f248d666640 1 --2- 172.21.15.29:0/1561749887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192135dd0 0x564192141940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:15:13.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.645+0000 7f248ce65640 1 --2- 172.21.15.29:0/1561749887 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130270 0x56419213a3c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f247c006010 tx=0x7f247c004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c22e0cc0b1d988d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:13.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.645+0000 7f2467fff640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:15:13.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.645+0000 7f2467fff640 1 -- 172.21.15.29:0/1561749887 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c004250 con 0x564192130270 2024-03-11T19:15:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.646+0000 7f2467fff640 1 -- 172.21.15.29:0/1561749887 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f247c045070 con 0x564192130270 2024-03-11T19:15:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.646+0000 7f2491d70740 1 -- 172.21.15.29:0/1561749887 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564191d987e0 con 0x564192130270 2024-03-11T19:15:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.646+0000 7f2467fff640 1 -- 172.21.15.29:0/1561749887 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f247c0043b0 con 0x564192130270 2024-03-11T19:15:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.646+0000 7f2491d70740 10 client.26261 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.646+0000 7f2491d70740 1 -- 172.21.15.29:0/1561749887 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5641920976a0 con 0x564192130270 2024-03-11T19:15:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.646+0000 7f2467fff640 1 -- 172.21.15.29:0/1561749887 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x564191d987e0 con 0x564192130270 2024-03-11T19:15:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.646+0000 7f2467fff640 1 -- 172.21.15.29:0/1561749887 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f247c03c040 con 0x564192130270 2024-03-11T19:15:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.647+0000 7f2491d70740 10 client.26261 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.647+0000 7f2491d70740 10 client.26261 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:15:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.647+0000 7f2491d70740 20 client.26261 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.647+0000 7f2491d70740 1 --2- 172.21.15.29:0/1561749887 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564192157530 0x564192159950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.647+0000 7f2491d70740 4 client.26261 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:15:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.647+0000 7f2491d70740 1 -- 172.21.15.29:0/1561749887 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564191dd5720 con 0x564192157530 2024-03-11T19:15:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.647+0000 7f2487fff640 1 --2- 172.21.15.29:0/1561749887 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564192157530 0x564192159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.648+0000 7f2487fff640 1 --2- 172.21.15.29:0/1561749887 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564192157530 0x564192159950 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc813a771e48db98 server_cookie=be87167b339082d3 in_seq=0 out_seq=0 2024-03-11T19:15:13.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.648+0000 7f2467fff640 10 client.26261 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:13.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.649+0000 7f2467fff640 1 -- 172.21.15.29:0/1561749887 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564191dd5720 con 0x564192157530 2024-03-11T19:15:13.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.649+0000 7f2467fff640 10 client.26261 handle_command_reply: tid=0 2024-03-11T19:15:13.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.649+0000 7f2491d70740 1 client.26261 shutdown 2024-03-11T19:15:13.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.649+0000 7f2491d70740 1 -- 172.21.15.29:0/1561749887 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130270 msgr2=0x56419213a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.649+0000 7f2491d70740 1 --2- 172.21.15.29:0/1561749887 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130270 0x56419213a3c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f247c006010 tx=0x7f247c004960 comp rx=0 tx=0).stop 2024-03-11T19:15:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.650+0000 7f2491d70740 1 -- 172.21.15.29:0/1561749887 shutdown_connections 2024-03-11T19:15:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.650+0000 7f2491d70740 1 --2- 172.21.15.29:0/1561749887 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564192157530 0x564192159950 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.650+0000 7f2491d70740 1 --2- 172.21.15.29:0/1561749887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192135dd0 0x564192141940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.650+0000 7f2491d70740 1 --2- 172.21.15.29:0/1561749887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192130c60 0x56419213a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.650+0000 7f2491d70740 1 --2- 172.21.15.29:0/1561749887 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130270 0x56419213a3c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.650+0000 7f2491d70740 1 -- 172.21.15.29:0/1561749887 >> 172.21.15.29:0/1561749887 conn(0x56419212b7e0 msgr2=0x56419212ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.650+0000 7f2467fff640 0 client.26261 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.650+0000 7f2491d70740 1 -- 172.21.15.29:0/1561749887 shutdown_connections 2024-03-11T19:15:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.650+0000 7f2491d70740 1 -- 172.21.15.29:0/1561749887 wait complete. 2024-03-11T19:15:13.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.650+0000 7f2491d70740 20 client.26261 trim_cache size 0 max 16384 2024-03-11T19:15:13.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.668+0000 7f2491d70740 1 Processor -- start 2024-03-11T19:15:13.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.668+0000 7f2491d70740 1 -- start start 2024-03-11T19:15:13.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.668+0000 7f2491d70740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192130270 0x564192132120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.668+0000 7f2491d70740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130c60 0x564192141900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.668+0000 7f2491d70740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192135dd0 0x564192132660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.668+0000 7f2491d70740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56419201e130 con 0x564192130270 2024-03-11T19:15:13.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.668+0000 7f2487fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130c60 0x564192141900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.669+0000 7f2487fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130c60 0x564192141900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:54146/0 (socket says 172.21.15.29:54146) 2024-03-11T19:15:13.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.669+0000 7f2487fff640 1 -- 172.21.15.29:0/3498581641 learned_addr learned my addr 172.21.15.29:0/3498581641 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:13.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.669+0000 7f248d666640 1 --2- 172.21.15.29:0/3498581641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192135dd0 0x564192132660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.669+0000 7f2491d70740 1 -- 172.21.15.29:0/3498581641 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564191dd5720 con 0x564192130c60 2024-03-11T19:15:13.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.669+0000 7f2491d70740 1 -- 172.21.15.29:0/3498581641 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56419211da40 con 0x564192135dd0 2024-03-11T19:15:13.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.669+0000 7f248ce65640 1 --2- 172.21.15.29:0/3498581641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192130270 0x564192132120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.669+0000 7f248d666640 1 -- 172.21.15.29:0/3498581641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130c60 msgr2=0x564192141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.669+0000 7f248d666640 1 --2- 172.21.15.29:0/3498581641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130c60 0x564192141900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.669+0000 7f248d666640 1 -- 172.21.15.29:0/3498581641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192130270 msgr2=0x564192132120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.669+0000 7f248d666640 1 --2- 172.21.15.29:0/3498581641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192130270 0x564192132120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.669+0000 7f248d666640 1 -- 172.21.15.29:0/3498581641 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564192033c20 con 0x564192135dd0 2024-03-11T19:15:13.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.670+0000 7f248d666640 1 --2- 172.21.15.29:0/3498581641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192135dd0 0x564192132660 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2480009a00 tx=0x7f24800a1060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fe105d75d0b6655 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:13.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.670+0000 7f24727fc640 1 -- 172.21.15.29:0/3498581641 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24800995d0 con 0x564192135dd0 2024-03-11T19:15:13.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.670+0000 7f24727fc640 1 -- 172.21.15.29:0/3498581641 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f24800a0e20 con 0x564192135dd0 2024-03-11T19:15:13.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.670+0000 7f24727fc640 1 -- 172.21.15.29:0/3498581641 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24800962e0 con 0x564192135dd0 2024-03-11T19:15:13.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.671+0000 7f2491d70740 1 -- 172.21.15.29:0/3498581641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192135dd0 msgr2=0x564192132660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.671+0000 7f2491d70740 1 --2- 172.21.15.29:0/3498581641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192135dd0 0x564192132660 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2480009a00 tx=0x7f24800a1060 comp rx=0 tx=0).stop 2024-03-11T19:15:13.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.671+0000 7f2491d70740 1 -- 172.21.15.29:0/3498581641 shutdown_connections 2024-03-11T19:15:13.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.671+0000 7f2491d70740 1 --2- 172.21.15.29:0/3498581641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192135dd0 0x564192132660 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.671+0000 7f2491d70740 1 --2- 172.21.15.29:0/3498581641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130c60 0x564192141900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.671+0000 7f2491d70740 1 --2- 172.21.15.29:0/3498581641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192130270 0x564192132120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.671+0000 7f2491d70740 1 -- 172.21.15.29:0/3498581641 >> 172.21.15.29:0/3498581641 conn(0x56419212b7e0 msgr2=0x564192138f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:13.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.671+0000 7f2491d70740 1 -- 172.21.15.29:0/3498581641 shutdown_connections 2024-03-11T19:15:13.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.672+0000 7f2491d70740 1 -- 172.21.15.29:0/3498581641 wait complete. 2024-03-11T19:15:13.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.672+0000 7f2491d70740 1 -- start start 2024-03-11T19:15:13.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.672+0000 7f2491d70740 1 Processor -- start 2024-03-11T19:15:13.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.673+0000 7f2491d70740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130270 0x56419213f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.673+0000 7f2491d70740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192130c60 0x56419213fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.673+0000 7f2491d70740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192135dd0 0x564192140af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.673+0000 7f2491d70740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564192141030 con 0x564192135dd0 2024-03-11T19:15:13.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.673+0000 7f2491d70740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641921411a0 con 0x564192130270 2024-03-11T19:15:13.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.673+0000 7f2491d70740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x564192141360 con 0x564192130c60 2024-03-11T19:15:13.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.673+0000 7f2491d70740 10 client.0 _ll_register_callbacks cb 0x7f2480095f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:13.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.673+0000 7f248ce65640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130270 0x56419213f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.674+0000 7f248ce65640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130270 0x56419213f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:54158/0 (socket says 172.21.15.29:54158) 2024-03-11T19:15:13.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.674+0000 7f248ce65640 1 -- 172.21.15.29:0/3831531350 learned_addr learned my addr 172.21.15.29:0/3831531350 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:13.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.674+0000 7f2487fff640 1 --2- 172.21.15.29:0/3831531350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192130c60 0x56419213fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.674+0000 7f248ce65640 1 -- 172.21.15.29:0/3831531350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192130c60 msgr2=0x56419213fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.674+0000 7f248ce65640 1 --2- 172.21.15.29:0/3831531350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192130c60 0x56419213fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.674+0000 7f248ce65640 1 -- 172.21.15.29:0/3831531350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192135dd0 msgr2=0x564192140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.674+0000 7f248ce65640 1 --2- 172.21.15.29:0/3831531350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192135dd0 0x564192140af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.674+0000 7f248ce65640 1 -- 172.21.15.29:0/3831531350 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56419205b060 con 0x564192130270 2024-03-11T19:15:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.674+0000 7f2487fff640 1 --2- 172.21.15.29:0/3831531350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192130c60 0x56419213fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.674+0000 7f248ce65640 1 --2- 172.21.15.29:0/3831531350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130270 0x56419213f5a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f247c005ec0 tx=0x7f247c005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42c5f5e29a0d4610 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.674+0000 7f2467fff640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:15:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.675+0000 7f2467fff640 1 -- 172.21.15.29:0/3831531350 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c059070 con 0x564192130270 2024-03-11T19:15:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.675+0000 7f2467fff640 1 -- 172.21.15.29:0/3831531350 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f247c045040 con 0x564192130270 2024-03-11T19:15:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.675+0000 7f2491d70740 1 -- 172.21.15.29:0/3831531350 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f24800a1cc0 con 0x564192130270 2024-03-11T19:15:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.675+0000 7f2467fff640 1 -- 172.21.15.29:0/3831531350 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f247c0049d0 con 0x564192130270 2024-03-11T19:15:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.675+0000 7f2491d70740 10 client.26267 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.675+0000 7f2491d70740 1 -- 172.21.15.29:0/3831531350 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5641920598c0 con 0x564192130270 2024-03-11T19:15:13.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.676+0000 7f2467fff640 1 -- 172.21.15.29:0/3831531350 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f247c064020 con 0x564192130270 2024-03-11T19:15:13.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.676+0000 7f2467fff640 1 -- 172.21.15.29:0/3831531350 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f247c05c020 con 0x564192130270 2024-03-11T19:15:13.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.676+0000 7f2491d70740 10 client.26267 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:13.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.676+0000 7f2491d70740 10 client.26267 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:15:13.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.676+0000 7f2491d70740 20 client.26267 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.676+0000 7f2491d70740 1 --2- 172.21.15.29:0/3831531350 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5641921ae810 0x5641921b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.676+0000 7f2491d70740 4 client.26267 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:15:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.676+0000 7f2491d70740 1 -- 172.21.15.29:0/3831531350 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f24800a1cc0 con 0x5641921ae810 2024-03-11T19:15:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.676+0000 7f2487fff640 1 --2- 172.21.15.29:0/3831531350 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5641921ae810 0x5641921b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.677+0000 7f2467fff640 10 client.26267 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.677+0000 7f2487fff640 1 --2- 172.21.15.29:0/3831531350 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5641921ae810 0x5641921b0c30 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19dbcca67342b7ce server_cookie=78baf35b9093ef08 in_seq=0 out_seq=0 2024-03-11T19:15:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.677+0000 7f2467fff640 1 -- 172.21.15.29:0/3831531350 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f24800a1cc0 con 0x5641921ae810 2024-03-11T19:15:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.677+0000 7f2467fff640 10 client.26267 handle_command_reply: tid=0 2024-03-11T19:15:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.677+0000 7f2491d70740 1 client.26267 shutdown 2024-03-11T19:15:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.678+0000 7f2491d70740 1 -- 172.21.15.29:0/3831531350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130270 msgr2=0x56419213f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.678+0000 7f2491d70740 1 --2- 172.21.15.29:0/3831531350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130270 0x56419213f5a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f247c005ec0 tx=0x7f247c005850 comp rx=0 tx=0).stop 2024-03-11T19:15:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.678+0000 7f2491d70740 1 -- 172.21.15.29:0/3831531350 shutdown_connections 2024-03-11T19:15:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.678+0000 7f2491d70740 1 --2- 172.21.15.29:0/3831531350 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5641921ae810 0x5641921b0c30 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.678+0000 7f2491d70740 1 --2- 172.21.15.29:0/3831531350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564192135dd0 0x564192140af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.678+0000 7f2491d70740 1 --2- 172.21.15.29:0/3831531350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564192130c60 0x56419213fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.678+0000 7f2491d70740 1 --2- 172.21.15.29:0/3831531350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564192130270 0x56419213f5a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.678+0000 7f2491d70740 1 -- 172.21.15.29:0/3831531350 >> 172.21.15.29:0/3831531350 conn(0x56419212b7e0 msgr2=0x56419212ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.678+0000 7f2467fff640 0 client.26267 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.678+0000 7f2491d70740 1 -- 172.21.15.29:0/3831531350 shutdown_connections 2024-03-11T19:15:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.678+0000 7f2491d70740 1 -- 172.21.15.29:0/3831531350 wait complete. 2024-03-11T19:15:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.678+0000 7f2491d70740 20 client.26267 trim_cache size 0 max 16384 2024-03-11T19:15:13.683 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:15:13.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.679+0000 7f248f0c9640 1 -- 172.21.15.29:0/1372597224 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f24780a1b30 msgr2=0x7f24780a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.679+0000 7f248f0c9640 1 --2- 172.21.15.29:0/1372597224 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f24780a1b30 0x7f24780a3ff0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x564192131060 tx=0x7f2474009710 comp rx=0 tx=0).stop 2024-03-11T19:15:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.679+0000 7f248f0c9640 1 -- 172.21.15.29:0/1372597224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2488109370 msgr2=0x7f24881a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.679+0000 7f248f0c9640 1 --2- 172.21.15.29:0/1372597224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2488109370 0x7f24881a4dd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f248000b970 tx=0x7f248000be30 comp rx=0 tx=0).stop 2024-03-11T19:15:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.679+0000 7f248f0c9640 1 -- 172.21.15.29:0/1372597224 shutdown_connections 2024-03-11T19:15:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.679+0000 7f248f0c9640 1 --2- 172.21.15.29:0/1372597224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2488109370 0x7f24881a4dd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.679+0000 7f248f0c9640 1 --2- 172.21.15.29:0/1372597224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24881089b0 0x7f248819dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.679+0000 7f248f0c9640 1 --2- 172.21.15.29:0/1372597224 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f24780a1b30 0x7f24780a3ff0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.679+0000 7f248f0c9640 1 --2- 172.21.15.29:0/1372597224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24881077c0 0x7f248819d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.679+0000 7f248f0c9640 1 -- 172.21.15.29:0/1372597224 >> 172.21.15.29:0/1372597224 conn(0x7f2488076750 msgr2=0x7f248810bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.679+0000 7f248f0c9640 1 -- 172.21.15.29:0/1372597224 shutdown_connections 2024-03-11T19:15:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:13.679+0000 7f248f0c9640 1 -- 172.21.15.29:0/1372597224 wait complete. 2024-03-11T19:15:14.700 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:15:14.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:15:14.700 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:15:15.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:14 smithi170 ceph-mon[35690]: pgmap v514: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-11T19:15:15.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.116+0000 7f7220908640 1 -- 172.21.15.29:0/2823471680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f721c109980 msgr2=0x7f721c109da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:15.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.116+0000 7f7220908640 1 --2- 172.21.15.29:0/2823471680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f721c109980 0x7f721c109da0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f720c009900 tx=0x7f720c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:15:15.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.117+0000 7f7220908640 1 -- 172.21.15.29:0/2823471680 shutdown_connections 2024-03-11T19:15:15.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.117+0000 7f7220908640 1 --2- 172.21.15.29:0/2823471680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f721c10b530 0x7f721c111dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.117+0000 7f7220908640 1 --2- 172.21.15.29:0/2823471680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f721c10ab70 0x7f721c10aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.117+0000 7f7220908640 1 --2- 172.21.15.29:0/2823471680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f721c109980 0x7f721c109da0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.117+0000 7f7220908640 1 -- 172.21.15.29:0/2823471680 >> 172.21.15.29:0/2823471680 conn(0x7f721c070140 msgr2=0x7f721c072560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:15.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.118+0000 7f7220908640 1 -- 172.21.15.29:0/2823471680 shutdown_connections 2024-03-11T19:15:15.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.118+0000 7f7220908640 1 -- 172.21.15.29:0/2823471680 wait complete. 2024-03-11T19:15:15.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.119+0000 7f7220908640 1 Processor -- start 2024-03-11T19:15:15.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.120+0000 7f7220908640 1 -- start start 2024-03-11T19:15:15.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.120+0000 7f7220908640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f721c109980 0x7f721c1a5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.120+0000 7f7220908640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f721c10ab70 0x7f721c1a64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.120+0000 7f7220908640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f721c10b530 0x7f721c1ad550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.120+0000 7f7220908640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c1ada90 con 0x7f721c109980 2024-03-11T19:15:15.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.120+0000 7f7220908640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c1adc00 con 0x7f721c10b530 2024-03-11T19:15:15.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.120+0000 7f7220908640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c1addc0 con 0x7f721c10ab70 2024-03-11T19:15:15.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.120+0000 7f721ad9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f721c10b530 0x7f721c1ad550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.120+0000 7f721ad9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f721c10b530 0x7f721c1ad550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:54186/0 (socket says 172.21.15.29:54186) 2024-03-11T19:15:15.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.120+0000 7f721ad9d640 1 -- 172.21.15.29:0/1925170225 learned_addr learned my addr 172.21.15.29:0/1925170225 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:15.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.120+0000 7f7213fff640 1 --2- 172.21.15.29:0/1925170225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f721c10ab70 0x7f721c1a64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.120+0000 7f721a59c640 1 --2- 172.21.15.29:0/1925170225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f721c109980 0x7f721c1a5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.121+0000 7f721ad9d640 1 -- 172.21.15.29:0/1925170225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f721c10ab70 msgr2=0x7f721c1a64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:15.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.121+0000 7f721ad9d640 1 --2- 172.21.15.29:0/1925170225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f721c10ab70 0x7f721c1a64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.121+0000 7f721ad9d640 1 -- 172.21.15.29:0/1925170225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f721c109980 msgr2=0x7f721c1a5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:15.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.121+0000 7f721ad9d640 1 --2- 172.21.15.29:0/1925170225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f721c109980 0x7f721c1a5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.121+0000 7f721ad9d640 1 -- 172.21.15.29:0/1925170225 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720c009560 con 0x7f721c10b530 2024-03-11T19:15:15.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.121+0000 7f721ad9d640 1 --2- 172.21.15.29:0/1925170225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f721c10b530 0x7f721c1ad550 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f721400eb10 tx=0x7f721400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:15.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.122+0000 7f72137fe640 1 -- 172.21.15.29:0/1925170225 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7214004b20 con 0x7f721c10b530 2024-03-11T19:15:15.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.122+0000 7f72137fe640 1 -- 172.21.15.29:0/1925170225 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7214004c80 con 0x7f721c10b530 2024-03-11T19:15:15.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.122+0000 7f72137fe640 1 -- 172.21.15.29:0/1925170225 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7214010690 con 0x7f721c10b530 2024-03-11T19:15:15.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.123+0000 7f7220908640 1 -- 172.21.15.29:0/1925170225 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f721c1ae0a0 con 0x7f721c10b530 2024-03-11T19:15:15.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.123+0000 7f7220908640 1 -- 172.21.15.29:0/1925170225 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721c1ae560 con 0x7f721c10b530 2024-03-11T19:15:15.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.126+0000 7f72137fe640 1 -- 172.21.15.29:0/1925170225 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f72140107f0 con 0x7f721c10b530 2024-03-11T19:15:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.126+0000 7f72235b0740 1 Processor -- start 2024-03-11T19:15:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.127+0000 7f72235b0740 1 -- start start 2024-03-11T19:15:15.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.128+0000 7f72137fe640 1 --2- 172.21.15.29:0/1925170225 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f72080a1be0 0x7f72080a40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.129+0000 7f72235b0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764b7d0 0x55f90764bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.129+0000 7f72235b0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764c9a0 0x55f90764ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.129+0000 7f72235b0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764d360 0x55f907653bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.129+0000 7f72235b0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9075baf90 con 0x55f90764c9a0 2024-03-11T19:15:15.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.129+0000 7f72235b0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9075b84c0 con 0x55f90764b7d0 2024-03-11T19:15:15.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.129+0000 7f72235b0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9075b3ca0 con 0x55f90764d360 2024-03-11T19:15:15.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.129+0000 7f72137fe640 1 -- 172.21.15.29:0/1925170225 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7214014070 con 0x7f721c10b530 2024-03-11T19:15:15.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.129+0000 7f721ad9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764d360 0x55f907653bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.129+0000 7f721ad9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764d360 0x55f907653bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37656/0 (socket says 172.21.15.29:37656) 2024-03-11T19:15:15.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.129+0000 7f721ad9d640 1 -- 172.21.15.29:0/1380262800 learned_addr learned my addr 172.21.15.29:0/1380262800 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:15.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.130+0000 7f7213fff640 1 --2- 172.21.15.29:0/1925170225 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f72080a1be0 0x7f72080a40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.130+0000 7f721ad9d640 1 -- 172.21.15.29:0/1380262800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764b7d0 msgr2=0x55f90764bbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:15:15.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.130+0000 7f721ad9d640 1 --2- 172.21.15.29:0/1380262800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764b7d0 0x55f90764bbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.130+0000 7f721ad9d640 1 -- 172.21.15.29:0/1380262800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764c9a0 msgr2=0x55f90764ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:15.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.130+0000 7f721ad9d640 1 --2- 172.21.15.29:0/1380262800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764c9a0 0x55f90764ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.130+0000 7f721ad9d640 1 -- 172.21.15.29:0/1380262800 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9075760b0 con 0x55f90764d360 2024-03-11T19:15:15.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.130+0000 7f721ad9d640 1 --2- 172.21.15.29:0/1380262800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764d360 0x55f907653bf0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f721400efd0 tx=0x7f721409b3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=936fc3d14f8531d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:15.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.131+0000 7f7203fff640 1 -- 172.21.15.29:0/1380262800 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721409b7f0 con 0x55f90764d360 2024-03-11T19:15:15.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.131+0000 7f7203fff640 1 -- 172.21.15.29:0/1380262800 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f721409f940 con 0x55f90764d360 2024-03-11T19:15:15.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.131+0000 7f7203fff640 1 -- 172.21.15.29:0/1380262800 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7214097e60 con 0x55f90764d360 2024-03-11T19:15:15.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.132+0000 7f72235b0740 1 -- 172.21.15.29:0/1380262800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764d360 msgr2=0x55f907653bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:15.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.132+0000 7f72235b0740 1 --2- 172.21.15.29:0/1380262800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764d360 0x55f907653bf0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f721400efd0 tx=0x7f721409b3f0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.132+0000 7f7213fff640 1 --2- 172.21.15.29:0/1925170225 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f72080a1be0 0x7f72080a40a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f721c1a74b0 tx=0x7f72040099c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:15.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.133+0000 7f72235b0740 1 -- 172.21.15.29:0/1380262800 shutdown_connections 2024-03-11T19:15:15.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.133+0000 7f72235b0740 1 --2- 172.21.15.29:0/1380262800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764d360 0x55f907653bf0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.133+0000 7f72235b0740 1 --2- 172.21.15.29:0/1380262800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764c9a0 0x55f90764ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.133+0000 7f72235b0740 1 --2- 172.21.15.29:0/1380262800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764b7d0 0x55f90764bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.133+0000 7f72235b0740 1 -- 172.21.15.29:0/1380262800 >> 172.21.15.29:0/1380262800 conn(0x55f9076469b0 msgr2=0x55f907646db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:15.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.133+0000 7f72235b0740 1 -- 172.21.15.29:0/1380262800 shutdown_connections 2024-03-11T19:15:15.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.134+0000 7f72235b0740 1 -- 172.21.15.29:0/1380262800 wait complete. 2024-03-11T19:15:15.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.134+0000 7f72235b0740 1 -- start start 2024-03-11T19:15:15.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.134+0000 7f72235b0740 1 Processor -- start 2024-03-11T19:15:15.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.135+0000 7f72235b0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764b7d0 0x55f907655720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.135+0000 7f72235b0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764c9a0 0x55f907655c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.135+0000 7f72235b0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764d360 0x55f90765cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.135+0000 7f7213fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764c9a0 0x55f907655c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.135+0000 7f7213fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764c9a0 0x55f907655c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:54200/0 (socket says 172.21.15.29:54200) 2024-03-11T19:15:15.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.135+0000 7f7213fff640 1 -- 172.21.15.29:0/374063252 learned_addr learned my addr 172.21.15.29:0/374063252 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:15.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.135+0000 7f72235b0740 1 -- 172.21.15.29:0/374063252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f907558f40 con 0x55f90764d360 2024-03-11T19:15:15.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.135+0000 7f72235b0740 1 -- 172.21.15.29:0/374063252 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9072874b0 con 0x55f90764c9a0 2024-03-11T19:15:15.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.135+0000 7f721ad9d640 1 --2- 172.21.15.29:0/374063252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764d360 0x55f90765cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.135+0000 7f721a59c640 1 --2- 172.21.15.29:0/374063252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764b7d0 0x55f907655720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.136+0000 7f72235b0740 1 -- 172.21.15.29:0/374063252 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f907539670 con 0x55f90764b7d0 2024-03-11T19:15:15.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.137+0000 7f721a59c640 1 -- 172.21.15.29:0/374063252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764c9a0 msgr2=0x55f907655c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:15.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.137+0000 7f721a59c640 1 --2- 172.21.15.29:0/374063252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764c9a0 0x55f907655c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.137+0000 7f721a59c640 1 -- 172.21.15.29:0/374063252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764d360 msgr2=0x55f90765cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:15.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.137+0000 7f721a59c640 1 --2- 172.21.15.29:0/374063252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764d360 0x55f90765cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.137+0000 7f721a59c640 1 -- 172.21.15.29:0/374063252 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721c1ae560 con 0x55f90764b7d0 2024-03-11T19:15:15.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.137+0000 7f7201ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:15:15.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.137+0000 7f721a59c640 1 --2- 172.21.15.29:0/374063252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764b7d0 0x55f907655720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f720c036040 tx=0x7f720c0048d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6a52b5590eeb3e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:15.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.136+0000 7f72235b0740 10 client.0 _ll_register_callbacks cb 0x55f9075a6f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:15.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.138+0000 7f72235b0740 1 -- 172.21.15.29:0/374063252 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f90758df40 con 0x55f90764b7d0 2024-03-11T19:15:15.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.138+0000 7f7201ffb640 1 -- 172.21.15.29:0/374063252 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c004240 con 0x55f90764b7d0 2024-03-11T19:15:15.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.138+0000 7f7201ffb640 1 -- 172.21.15.29:0/374063252 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f720c02de40 con 0x55f90764b7d0 2024-03-11T19:15:15.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.138+0000 7f7201ffb640 1 -- 172.21.15.29:0/374063252 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f720c005d50 con 0x55f90764b7d0 2024-03-11T19:15:15.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.138+0000 7f72235b0740 10 client.26284 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:15.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.138+0000 7f72235b0740 1 -- 172.21.15.29:0/374063252 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x55f9075b2320 con 0x55f90764b7d0 2024-03-11T19:15:15.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.139+0000 7f7201ffb640 1 -- 172.21.15.29:0/374063252 <== mon.1 v2:172.21.15.190:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f720c03c040 con 0x55f90764b7d0 2024-03-11T19:15:15.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.139+0000 7f72235b0740 10 client.26284 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:15.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.140+0000 7f72235b0740 10 client.26284 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:15:15.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.140+0000 7f72235b0740 20 client.26284 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:15.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.140+0000 7f72235b0740 1 --2- 172.21.15.29:0/374063252 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f907661700 0x55f907663b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.141+0000 7f7213fff640 1 --2- 172.21.15.29:0/374063252 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f907661700 0x55f907663b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.141+0000 7f72235b0740 4 client.26284 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:15:15.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.141+0000 7f7213fff640 1 --2- 172.21.15.29:0/374063252 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f907661700 0x55f907663b90 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59ea2b0dd1b9fee0 server_cookie=8566e780b8380106 in_seq=0 out_seq=0 2024-03-11T19:15:15.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.141+0000 7f7201ffb640 10 client.26284 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:15.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.141+0000 7f72235b0740 1 -- 172.21.15.29:0/374063252 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f9075aa310 con 0x55f907661700 2024-03-11T19:15:15.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.142+0000 7f7201ffb640 1 -- 172.21.15.29:0/374063252 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f9075aa310 con 0x55f907661700 2024-03-11T19:15:15.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.142+0000 7f7201ffb640 10 client.26284 handle_command_reply: tid=0 2024-03-11T19:15:15.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.142+0000 7f72235b0740 1 client.26284 shutdown 2024-03-11T19:15:15.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.143+0000 7f72235b0740 1 -- 172.21.15.29:0/374063252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764b7d0 msgr2=0x55f907655720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:15.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.143+0000 7f72235b0740 1 --2- 172.21.15.29:0/374063252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764b7d0 0x55f907655720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f720c036040 tx=0x7f720c0048d0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.143+0000 7f72235b0740 1 -- 172.21.15.29:0/374063252 shutdown_connections 2024-03-11T19:15:15.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.143+0000 7f72235b0740 1 --2- 172.21.15.29:0/374063252 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f907661700 0x55f907663b90 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.143+0000 7f72235b0740 1 --2- 172.21.15.29:0/374063252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764d360 0x55f90765cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.143+0000 7f72235b0740 1 --2- 172.21.15.29:0/374063252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764c9a0 0x55f907655c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.143+0000 7f72235b0740 1 --2- 172.21.15.29:0/374063252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764b7d0 0x55f907655720 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.143+0000 7f72235b0740 1 -- 172.21.15.29:0/374063252 >> 172.21.15.29:0/374063252 conn(0x55f9076469b0 msgr2=0x55f907645c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:15.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.143+0000 7f7201ffb640 0 client.26284 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:15.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.143+0000 7f72235b0740 1 -- 172.21.15.29:0/374063252 shutdown_connections 2024-03-11T19:15:15.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.143+0000 7f72235b0740 1 -- 172.21.15.29:0/374063252 wait complete. 2024-03-11T19:15:15.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.143+0000 7f72235b0740 20 client.26284 trim_cache size 0 max 16384 2024-03-11T19:15:15.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:14 smithi029 ceph-mon[32439]: pgmap v514: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-11T19:15:15.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.167+0000 7f72235b0740 1 Processor -- start 2024-03-11T19:15:15.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.167+0000 7f72235b0740 1 -- start start 2024-03-11T19:15:15.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.167+0000 7f72235b0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764b7d0 0x55f90764a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.167+0000 7f72235b0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764c9a0 0x55f90765cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.168+0000 7f72235b0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764d360 0x55f907654a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.168+0000 7f72235b0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9075aa310 con 0x55f90764c9a0 2024-03-11T19:15:15.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.168+0000 7f72235b0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f90764c830 con 0x55f90764b7d0 2024-03-11T19:15:15.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.168+0000 7f72235b0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f907656910 con 0x55f90764d360 2024-03-11T19:15:15.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.168+0000 7f721ad9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764d360 0x55f907654a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.168+0000 7f721a59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764b7d0 0x55f90764a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.168+0000 7f721ad9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764d360 0x55f907654a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37678/0 (socket says 172.21.15.29:37678) 2024-03-11T19:15:15.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.168+0000 7f721ad9d640 1 -- 172.21.15.29:0/2777075575 learned_addr learned my addr 172.21.15.29:0/2777075575 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:15.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.168+0000 7f7213fff640 1 --2- 172.21.15.29:0/2777075575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764c9a0 0x55f90765cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.169+0000 7f721ad9d640 1 -- 172.21.15.29:0/2777075575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764b7d0 msgr2=0x55f90764a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:15.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.169+0000 7f721ad9d640 1 --2- 172.21.15.29:0/2777075575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764b7d0 0x55f90764a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.169+0000 7f721ad9d640 1 -- 172.21.15.29:0/2777075575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764c9a0 msgr2=0x55f90765cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:15.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.169+0000 7f721ad9d640 1 --2- 172.21.15.29:0/2777075575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764c9a0 0x55f90765cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.169+0000 7f721ad9d640 1 -- 172.21.15.29:0/2777075575 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f90754ec00 con 0x55f90764d360 2024-03-11T19:15:15.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.169+0000 7f7213fff640 1 --2- 172.21.15.29:0/2777075575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764c9a0 0x55f90765cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:15.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.169+0000 7f721ad9d640 1 --2- 172.21.15.29:0/2777075575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764d360 0x55f907654a10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7214099aa0 tx=0x7f721409f940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8f00f428d36dfbc server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:15.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.169+0000 7f72037fe640 1 -- 172.21.15.29:0/2777075575 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721409b7f0 con 0x55f90764d360 2024-03-11T19:15:15.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.169+0000 7f72037fe640 1 -- 172.21.15.29:0/2777075575 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f72140a3f90 con 0x55f90764d360 2024-03-11T19:15:15.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.170+0000 7f72037fe640 1 -- 172.21.15.29:0/2777075575 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72140a3d60 con 0x55f90764d360 2024-03-11T19:15:15.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.171+0000 7f72235b0740 1 -- 172.21.15.29:0/2777075575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764d360 msgr2=0x55f907654a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:15.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.171+0000 7f72235b0740 1 --2- 172.21.15.29:0/2777075575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764d360 0x55f907654a10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7214099aa0 tx=0x7f721409f940 comp rx=0 tx=0).stop 2024-03-11T19:15:15.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.171+0000 7f72235b0740 1 -- 172.21.15.29:0/2777075575 shutdown_connections 2024-03-11T19:15:15.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.171+0000 7f72235b0740 1 --2- 172.21.15.29:0/2777075575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764d360 0x55f907654a10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.171+0000 7f72235b0740 1 --2- 172.21.15.29:0/2777075575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764c9a0 0x55f90765cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.171+0000 7f72235b0740 1 --2- 172.21.15.29:0/2777075575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764b7d0 0x55f90764a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.171+0000 7f72235b0740 1 -- 172.21.15.29:0/2777075575 >> 172.21.15.29:0/2777075575 conn(0x55f9076469b0 msgr2=0x55f907649c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:15.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.171+0000 7f72235b0740 1 -- 172.21.15.29:0/2777075575 shutdown_connections 2024-03-11T19:15:15.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.171+0000 7f72235b0740 1 -- 172.21.15.29:0/2777075575 wait complete. 2024-03-11T19:15:15.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.172+0000 7f72235b0740 1 -- start start 2024-03-11T19:15:15.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.172+0000 7f72235b0740 1 Processor -- start 2024-03-11T19:15:15.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.173+0000 7f72235b0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764b7d0 0x55f90765b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.173+0000 7f72235b0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764c9a0 0x55f90765b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.173+0000 7f72235b0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764d360 0x55f90765be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.173+0000 7f72235b0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f90765d000 con 0x55f90764d360 2024-03-11T19:15:15.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.173+0000 7f72235b0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9076b7c80 con 0x55f90764c9a0 2024-03-11T19:15:15.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.173+0000 7f72235b0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9076b7e40 con 0x55f90764b7d0 2024-03-11T19:15:15.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.173+0000 7f72235b0740 10 client.0 _ll_register_callbacks cb 0x7f7214005ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:15.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.173+0000 7f721a59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764b7d0 0x55f90765b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.173+0000 7f721a59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764b7d0 0x55f90765b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37684/0 (socket says 172.21.15.29:37684) 2024-03-11T19:15:15.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.173+0000 7f721a59c640 1 -- 172.21.15.29:0/2306827438 learned_addr learned my addr 172.21.15.29:0/2306827438 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:15.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.173+0000 7f7213fff640 1 --2- 172.21.15.29:0/2306827438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764c9a0 0x55f90765b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.173+0000 7f721ad9d640 1 --2- 172.21.15.29:0/2306827438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764d360 0x55f90765be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.174+0000 7f7213fff640 1 -- 172.21.15.29:0/2306827438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764b7d0 msgr2=0x55f90765b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:15.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.174+0000 7f7213fff640 1 --2- 172.21.15.29:0/2306827438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764b7d0 0x55f90765b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.174+0000 7f7213fff640 1 -- 172.21.15.29:0/2306827438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764d360 msgr2=0x55f90765be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:15.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.174+0000 7f7213fff640 1 --2- 172.21.15.29:0/2306827438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764d360 0x55f90765be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.174+0000 7f7213fff640 1 -- 172.21.15.29:0/2306827438 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7204017070 con 0x55f90764c9a0 2024-03-11T19:15:15.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.174+0000 7f7213fff640 1 --2- 172.21.15.29:0/2306827438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764c9a0 0x55f90765b8f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x55f907664430 tx=0x7f7204014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99a3f2edceb66bf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:15.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.174+0000 7f72017fa640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:15:15.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.175+0000 7f72017fa640 1 -- 172.21.15.29:0/2306827438 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720402f480 con 0x55f90764c9a0 2024-03-11T19:15:15.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.175+0000 7f72017fa640 1 -- 172.21.15.29:0/2306827438 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f720402f5e0 con 0x55f90764c9a0 2024-03-11T19:15:15.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.175+0000 7f72235b0740 1 -- 172.21.15.29:0/2306827438 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f9075d9900 con 0x55f90764c9a0 2024-03-11T19:15:15.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.175+0000 7f72017fa640 1 -- 172.21.15.29:0/2306827438 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f720402f7a0 con 0x55f90764c9a0 2024-03-11T19:15:15.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.175+0000 7f72235b0740 10 client.26282 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:15.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.175+0000 7f72235b0740 1 -- 172.21.15.29:0/2306827438 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55f9075760b0 con 0x55f90764c9a0 2024-03-11T19:15:15.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.176+0000 7f72017fa640 1 -- 172.21.15.29:0/2306827438 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f9075d9900 con 0x55f90764c9a0 2024-03-11T19:15:15.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.176+0000 7f72017fa640 1 -- 172.21.15.29:0/2306827438 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f720401a040 con 0x55f90764c9a0 2024-03-11T19:15:15.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.176+0000 7f72235b0740 10 client.26282 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:15.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.176+0000 7f72235b0740 10 client.26282 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:15:15.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.176+0000 7f72235b0740 20 client.26282 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:15.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.177+0000 7f72235b0740 1 --2- 172.21.15.29:0/2306827438 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f9076b9970 0x55f9076bbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.177+0000 7f72235b0740 4 client.26282 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:15:15.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.177+0000 7f72235b0740 1 -- 172.21.15.29:0/2306827438 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f9072b3860 con 0x55f9076b9970 2024-03-11T19:15:15.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.177+0000 7f721a59c640 1 --2- 172.21.15.29:0/2306827438 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f9076b9970 0x55f9076bbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.178+0000 7f721a59c640 1 --2- 172.21.15.29:0/2306827438 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f9076b9970 0x55f9076bbd90 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30df6e6af617615c server_cookie=1bef89ca26c0d258 in_seq=0 out_seq=0 2024-03-11T19:15:15.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.178+0000 7f72017fa640 10 client.26282 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:15.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.179+0000 7f72017fa640 1 -- 172.21.15.29:0/2306827438 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f9072b3860 con 0x55f9076b9970 2024-03-11T19:15:15.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.179+0000 7f72017fa640 10 client.26282 handle_command_reply: tid=0 2024-03-11T19:15:15.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.179+0000 7f72235b0740 1 client.26282 shutdown 2024-03-11T19:15:15.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.179+0000 7f72235b0740 1 -- 172.21.15.29:0/2306827438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764c9a0 msgr2=0x55f90765b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:15.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.179+0000 7f72235b0740 1 --2- 172.21.15.29:0/2306827438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764c9a0 0x55f90765b8f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x55f907664430 tx=0x7f7204014000 comp rx=0 tx=0).stop 2024-03-11T19:15:15.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.180+0000 7f72235b0740 1 -- 172.21.15.29:0/2306827438 shutdown_connections 2024-03-11T19:15:15.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.180+0000 7f72235b0740 1 --2- 172.21.15.29:0/2306827438 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f9076b9970 0x55f9076bbd90 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.180+0000 7f72235b0740 1 --2- 172.21.15.29:0/2306827438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f90764d360 0x55f90765be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.180+0000 7f72235b0740 1 --2- 172.21.15.29:0/2306827438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f90764c9a0 0x55f90765b8f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.180+0000 7f72235b0740 1 --2- 172.21.15.29:0/2306827438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f90764b7d0 0x55f90765b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.180+0000 7f72235b0740 1 -- 172.21.15.29:0/2306827438 >> 172.21.15.29:0/2306827438 conn(0x55f9076469b0 msgr2=0x55f90764a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:15.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.180+0000 7f72017fa640 0 client.26282 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:15.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.180+0000 7f72235b0740 1 -- 172.21.15.29:0/2306827438 shutdown_connections 2024-03-11T19:15:15.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.180+0000 7f72235b0740 1 -- 172.21.15.29:0/2306827438 wait complete. 2024-03-11T19:15:15.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.180+0000 7f72235b0740 20 client.26282 trim_cache size 0 max 16384 2024-03-11T19:15:15.187 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:15:15.187 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:15:15.187 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "6f0bac3b-fbf7-4bbb-95e6-c92e31f1e14d", 2024-03-11T19:15:15.187 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:15:15.188 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:15:15.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.181+0000 7f72117fa640 1 -- 172.21.15.29:0/1925170225 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f72080a1be0 msgr2=0x7f72080a40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:15.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.181+0000 7f72117fa640 1 --2- 172.21.15.29:0/1925170225 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f72080a1be0 0x7f72080a40a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f721c1a74b0 tx=0x7f72040099c0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.181+0000 7f72117fa640 1 -- 172.21.15.29:0/1925170225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f721c10b530 msgr2=0x7f721c1ad550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:15.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.181+0000 7f72117fa640 1 --2- 172.21.15.29:0/1925170225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f721c10b530 0x7f721c1ad550 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f721400eb10 tx=0x7f721400c2d0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.182+0000 7f72117fa640 1 -- 172.21.15.29:0/1925170225 shutdown_connections 2024-03-11T19:15:15.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.182+0000 7f72117fa640 1 --2- 172.21.15.29:0/1925170225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f721c10b530 0x7f721c1ad550 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.182+0000 7f72117fa640 1 --2- 172.21.15.29:0/1925170225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f721c10ab70 0x7f721c1a64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.182+0000 7f72117fa640 1 --2- 172.21.15.29:0/1925170225 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f72080a1be0 0x7f72080a40a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.182+0000 7f72117fa640 1 --2- 172.21.15.29:0/1925170225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f721c109980 0x7f721c1a5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.182+0000 7f72117fa640 1 -- 172.21.15.29:0/1925170225 >> 172.21.15.29:0/1925170225 conn(0x7f721c070140 msgr2=0x7f721c10dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:15.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.182+0000 7f72117fa640 1 -- 172.21.15.29:0/1925170225 shutdown_connections 2024-03-11T19:15:15.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.182+0000 7f72117fa640 1 -- 172.21.15.29:0/1925170225 wait complete. 2024-03-11T19:15:15.206 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:15:15.206 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:15:15.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:14 smithi190 ceph-mon[36292]: pgmap v514: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-11T19:15:15.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.388+0000 7fecf90f4640 1 -- 172.21.15.29:0/2591601187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecf40751c0 msgr2=0x7fecf4073610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:15.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.388+0000 7fecf90f4640 1 --2- 172.21.15.29:0/2591601187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecf40751c0 0x7fecf4073610 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fece8009930 tx=0x7fece802bc30 comp rx=0 tx=0).stop 2024-03-11T19:15:15.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.388+0000 7fecf90f4640 1 -- 172.21.15.29:0/2591601187 shutdown_connections 2024-03-11T19:15:15.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.388+0000 7fecf90f4640 1 --2- 172.21.15.29:0/2591601187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecf410d790 0x7fecf410fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.388+0000 7fecf90f4640 1 --2- 172.21.15.29:0/2591601187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecf4073be0 0x7fecf4074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.388+0000 7fecf90f4640 1 --2- 172.21.15.29:0/2591601187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecf40751c0 0x7fecf4073610 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.388+0000 7fecf90f4640 1 -- 172.21.15.29:0/2591601187 >> 172.21.15.29:0/2591601187 conn(0x7fecf40fc710 msgr2=0x7fecf40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:15.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.388+0000 7fecf90f4640 1 -- 172.21.15.29:0/2591601187 shutdown_connections 2024-03-11T19:15:15.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.389+0000 7fecf90f4640 1 -- 172.21.15.29:0/2591601187 wait complete. 2024-03-11T19:15:15.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.389+0000 7fecf90f4640 1 Processor -- start 2024-03-11T19:15:15.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.390+0000 7fecf90f4640 1 -- start start 2024-03-11T19:15:15.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.390+0000 7fecf90f4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecf4073be0 0x7fecf419d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.390+0000 7fecf90f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecf40751c0 0x7fecf419dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.390+0000 7fecf90f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecf410d790 0x7fecf41a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.390+0000 7fecf90f4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf41a51f0 con 0x7fecf40751c0 2024-03-11T19:15:15.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.390+0000 7fecf90f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf41a5360 con 0x7fecf410d790 2024-03-11T19:15:15.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.390+0000 7fecf90f4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf41a5520 con 0x7fecf4073be0 2024-03-11T19:15:15.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.390+0000 7fecf259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecf40751c0 0x7fecf419dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.390+0000 7fecf359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecf410d790 0x7fecf41a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.390+0000 7fecf259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecf40751c0 0x7fecf419dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33822/0 (socket says 172.21.15.29:33822) 2024-03-11T19:15:15.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.390+0000 7fecf259c640 1 -- 172.21.15.29:0/2991347907 learned_addr learned my addr 172.21.15.29:0/2991347907 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:15.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.390+0000 7fecf2d9d640 1 --2- 172.21.15.29:0/2991347907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecf4073be0 0x7fecf419d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.391+0000 7fecf259c640 1 -- 172.21.15.29:0/2991347907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecf4073be0 msgr2=0x7fecf419d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:15.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.391+0000 7fecf259c640 1 --2- 172.21.15.29:0/2991347907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecf4073be0 0x7fecf419d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.391+0000 7fecf259c640 1 -- 172.21.15.29:0/2991347907 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecf410d790 msgr2=0x7fecf41a4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:15.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.391+0000 7fecf259c640 1 --2- 172.21.15.29:0/2991347907 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecf410d790 0x7fecf41a4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.391+0000 7fecf259c640 1 -- 172.21.15.29:0/2991347907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece8009560 con 0x7fecf40751c0 2024-03-11T19:15:15.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.391+0000 7fecf2d9d640 1 --2- 172.21.15.29:0/2991347907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecf4073be0 0x7fecf419d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:15:15.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.391+0000 7fecf259c640 1 --2- 172.21.15.29:0/2991347907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecf40751c0 0x7fecf419dc40 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fece400c920 tx=0x7fece400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:15.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.391+0000 7fecdbfff640 1 -- 172.21.15.29:0/2991347907 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4007c70 con 0x7fecf40751c0 2024-03-11T19:15:15.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.392+0000 7fecdbfff640 1 -- 172.21.15.29:0/2991347907 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fece4007dd0 con 0x7fecf40751c0 2024-03-11T19:15:15.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.392+0000 7fecf90f4640 1 -- 172.21.15.29:0/2991347907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf41a5800 con 0x7fecf40751c0 2024-03-11T19:15:15.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.392+0000 7fecdbfff640 1 -- 172.21.15.29:0/2991347907 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece40187c0 con 0x7fecf40751c0 2024-03-11T19:15:15.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.392+0000 7fecf90f4640 1 -- 172.21.15.29:0/2991347907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf41a5d60 con 0x7fecf40751c0 2024-03-11T19:15:15.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.395+0000 7fecdbfff640 1 -- 172.21.15.29:0/2991347907 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fece400f450 con 0x7fecf40751c0 2024-03-11T19:15:15.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.395+0000 7fecdbfff640 1 --2- 172.21.15.29:0/2991347907 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fecdc0a1b30 0x7fecdc0a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.395+0000 7fecfbd9c740 1 Processor -- start 2024-03-11T19:15:15.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.395+0000 7fecdbfff640 1 -- 172.21.15.29:0/2991347907 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fece40cd4e0 con 0x7fecf40751c0 2024-03-11T19:15:15.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.395+0000 7fecf2d9d640 1 --2- 172.21.15.29:0/2991347907 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fecdc0a1b30 0x7fecdc0a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.396+0000 7fecfbd9c740 1 -- start start 2024-03-11T19:15:15.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.396+0000 7fecfbd9c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 0x558ed60236f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.396+0000 7fecfbd9c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6023cc0 0x558ed60287c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.396+0000 7fecf2d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 0x558ed60236f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.396+0000 7fecf2d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 0x558ed60236f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:37714/0 (socket says 172.21.15.29:37714) 2024-03-11T19:15:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.396+0000 7fecf2d9d640 1 -- 172.21.15.29:0/3084170855 learned_addr learned my addr 172.21.15.29:0/3084170855 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.396+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/3084170855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6028e30 0x558ed602b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.397+0000 7fecfbd9c740 1 -- 172.21.15.29:0/3084170855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ed5f93280 con 0x558ed6023cc0 2024-03-11T19:15:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.397+0000 7fecf359e640 1 --2- 172.21.15.29:0/3084170855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6028e30 0x558ed602b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.397+0000 7fecf259c640 1 --2- 172.21.15.29:0/3084170855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6023cc0 0x558ed60287c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.397+0000 7fecfbd9c740 1 -- 172.21.15.29:0/3084170855 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ed5f907b0 con 0x558ed6028e30 2024-03-11T19:15:15.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.397+0000 7fecfbd9c740 1 -- 172.21.15.29:0/3084170855 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ed5f8bf90 con 0x558ed60232d0 2024-03-11T19:15:15.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.397+0000 7fecf359e640 1 -- 172.21.15.29:0/3084170855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 msgr2=0x558ed60236f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:15.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.397+0000 7fecf359e640 1 --2- 172.21.15.29:0/3084170855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 0x558ed60236f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.397+0000 7fecf359e640 1 -- 172.21.15.29:0/3084170855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6023cc0 msgr2=0x558ed60287c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:15.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.397+0000 7fecf359e640 1 --2- 172.21.15.29:0/3084170855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6023cc0 0x558ed60287c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.397+0000 7fecf359e640 1 -- 172.21.15.29:0/3084170855 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ed5f4a570 con 0x558ed6028e30 2024-03-11T19:15:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.398+0000 7fecf359e640 1 --2- 172.21.15.29:0/3084170855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6028e30 0x558ed602b2c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fecec009f30 tx=0x7fecec010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e1228ae7c137ba0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.399+0000 7fecd97fa640 1 -- 172.21.15.29:0/3084170855 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec004d20 con 0x558ed6028e30 2024-03-11T19:15:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.399+0000 7fecd97fa640 1 -- 172.21.15.29:0/3084170855 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fecec004e80 con 0x558ed6028e30 2024-03-11T19:15:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.399+0000 7fecd97fa640 1 -- 172.21.15.29:0/3084170855 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec012870 con 0x558ed6028e30 2024-03-11T19:15:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.400+0000 7fecf2d9d640 1 --2- 172.21.15.29:0/2991347907 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fecdc0a1b30 0x7fecdc0a3ff0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x558ed6024520 tx=0x7fece8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.400+0000 7fecfbd9c740 1 -- 172.21.15.29:0/3084170855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6028e30 msgr2=0x558ed602b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.400+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/3084170855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6028e30 0x558ed602b2c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fecec009f30 tx=0x7fecec010800 comp rx=0 tx=0).stop 2024-03-11T19:15:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.400+0000 7fecfbd9c740 1 -- 172.21.15.29:0/3084170855 shutdown_connections 2024-03-11T19:15:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.400+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/3084170855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6028e30 0x558ed602b2c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.400+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/3084170855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6023cc0 0x558ed60287c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.400+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/3084170855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 0x558ed60236f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.400+0000 7fecfbd9c740 1 -- 172.21.15.29:0/3084170855 >> 172.21.15.29:0/3084170855 conn(0x558ed601e7f0 msgr2=0x558ed601ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:15.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.403+0000 7fecfbd9c740 1 -- 172.21.15.29:0/3084170855 shutdown_connections 2024-03-11T19:15:15.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.403+0000 7fecfbd9c740 1 -- 172.21.15.29:0/3084170855 wait complete. 2024-03-11T19:15:15.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.403+0000 7fecfbd9c740 1 -- start start 2024-03-11T19:15:15.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.404+0000 7fecfbd9c740 1 Processor -- start 2024-03-11T19:15:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.404+0000 7fecfbd9c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 0x558ed602d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.404+0000 7fecfbd9c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6023cc0 0x558ed602d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.404+0000 7fecfbd9c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6028e30 0x558ed6034640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.404+0000 7fecfbd9c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ed5c5f430 con 0x558ed6028e30 2024-03-11T19:15:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.404+0000 7fecfbd9c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ed5c1be00 con 0x558ed6023cc0 2024-03-11T19:15:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.404+0000 7fecfbd9c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ed5f10fd0 con 0x558ed60232d0 2024-03-11T19:15:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.404+0000 7fecf259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6023cc0 0x558ed602d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.404+0000 7fecf259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6023cc0 0x558ed602d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:54260/0 (socket says 172.21.15.29:54260) 2024-03-11T19:15:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.404+0000 7fecf259c640 1 -- 172.21.15.29:0/2759833378 learned_addr learned my addr 172.21.15.29:0/2759833378 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.404+0000 7fecf2d9d640 1 --2- 172.21.15.29:0/2759833378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 0x558ed602d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.404+0000 7fecf359e640 1 --2- 172.21.15.29:0/2759833378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6028e30 0x558ed6034640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.404+0000 7fecfbd9c740 10 client.0 _ll_register_callbacks cb 0x558ed5f66fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.405+0000 7fecf259c640 1 -- 172.21.15.29:0/2759833378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 msgr2=0x558ed602d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.405+0000 7fecf259c640 1 --2- 172.21.15.29:0/2759833378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 0x558ed602d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.405+0000 7fecf259c640 1 -- 172.21.15.29:0/2759833378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6028e30 msgr2=0x558ed6034640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.405+0000 7fecf259c640 1 --2- 172.21.15.29:0/2759833378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6028e30 0x558ed6034640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.405+0000 7fecf259c640 1 -- 172.21.15.29:0/2759833378 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecec009b90 con 0x558ed6023cc0 2024-03-11T19:15:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.405+0000 7fecf359e640 1 --2- 172.21.15.29:0/2759833378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6028e30 0x558ed6034640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:15.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.405+0000 7fecf259c640 1 --2- 172.21.15.29:0/2759833378 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6023cc0 0x558ed602d600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fece400ffd0 tx=0x7fece4099e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1101f668ebddda7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:15.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.405+0000 7fecd6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:15:15.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.406+0000 7fecd6ff5640 1 -- 172.21.15.29:0/2759833378 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece40b2070 con 0x558ed6023cc0 2024-03-11T19:15:15.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.406+0000 7fecd6ff5640 1 -- 172.21.15.29:0/2759833378 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fece409a450 con 0x558ed6023cc0 2024-03-11T19:15:15.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.406+0000 7fecfbd9c740 1 -- 172.21.15.29:0/2759833378 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558ed5cc8720 con 0x558ed6023cc0 2024-03-11T19:15:15.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.406+0000 7fecd6ff5640 1 -- 172.21.15.29:0/2759833378 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fece409e5a0 con 0x558ed6023cc0 2024-03-11T19:15:15.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.406+0000 7fecfbd9c740 10 client.26294 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:15.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.406+0000 7fecfbd9c740 1 -- 172.21.15.29:0/2759833378 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x558ed5f8a610 con 0x558ed6023cc0 2024-03-11T19:15:15.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.407+0000 7fecd6ff5640 1 -- 172.21.15.29:0/2759833378 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fece407d020 con 0x558ed6023cc0 2024-03-11T19:15:15.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.407+0000 7fecd6ff5640 1 -- 172.21.15.29:0/2759833378 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fece4016020 con 0x558ed6023cc0 2024-03-11T19:15:15.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.407+0000 7fecfbd9c740 10 client.26294 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:15.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.407+0000 7fecfbd9c740 10 client.26294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:15:15.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.407+0000 7fecfbd9c740 20 client.26294 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:15.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.407+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/2759833378 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558ed6039280 0x558ed603b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.407+0000 7fecfbd9c740 4 client.26294 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:15:15.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.407+0000 7fecfbd9c740 1 -- 172.21.15.29:0/2759833378 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558ed5cc8720 con 0x558ed6039280 2024-03-11T19:15:15.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.407+0000 7fecf2d9d640 1 --2- 172.21.15.29:0/2759833378 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558ed6039280 0x558ed603b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.408+0000 7fecf2d9d640 1 --2- 172.21.15.29:0/2759833378 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558ed6039280 0x558ed603b700 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89e1df30f082f024 server_cookie=5e13108ef300f49 in_seq=0 out_seq=0 2024-03-11T19:15:15.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.408+0000 7fecd6ff5640 10 client.26294 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.409+0000 7fecd6ff5640 1 -- 172.21.15.29:0/2759833378 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558ed5cc8720 con 0x558ed6039280 2024-03-11T19:15:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.409+0000 7fecd6ff5640 10 client.26294 handle_command_reply: tid=0 2024-03-11T19:15:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.409+0000 7fecfbd9c740 1 client.26294 shutdown 2024-03-11T19:15:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.410+0000 7fecfbd9c740 1 -- 172.21.15.29:0/2759833378 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6023cc0 msgr2=0x558ed602d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.410+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/2759833378 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6023cc0 0x558ed602d600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fece400ffd0 tx=0x7fece4099e70 comp rx=0 tx=0).stop 2024-03-11T19:15:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.410+0000 7fecfbd9c740 1 -- 172.21.15.29:0/2759833378 shutdown_connections 2024-03-11T19:15:15.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.410+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/2759833378 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558ed6039280 0x558ed603b700 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.410+0000 7fecd6ff5640 0 client.26294 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:15.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.410+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/2759833378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6028e30 0x558ed6034640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.410+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/2759833378 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6023cc0 0x558ed602d600 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.410+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/2759833378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 0x558ed602d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.410+0000 7fecfbd9c740 1 -- 172.21.15.29:0/2759833378 >> 172.21.15.29:0/2759833378 conn(0x558ed601e7f0 msgr2=0x558ed601da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:15.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.410+0000 7fecfbd9c740 1 -- 172.21.15.29:0/2759833378 shutdown_connections 2024-03-11T19:15:15.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.411+0000 7fecfbd9c740 1 -- 172.21.15.29:0/2759833378 wait complete. 2024-03-11T19:15:15.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.411+0000 7fecfbd9c740 20 client.26294 trim_cache size 0 max 16384 2024-03-11T19:15:15.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.434+0000 7fecfbd9c740 1 Processor -- start 2024-03-11T19:15:15.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.434+0000 7fecfbd9c740 1 -- start start 2024-03-11T19:15:15.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.434+0000 7fecfbd9c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 0x558ed602c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.435+0000 7fecfbd9c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6023cc0 0x558ed6034600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.435+0000 7fecfbd9c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6028e30 0x558ed60400a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.435+0000 7fecfbd9c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ed5cc8720 con 0x558ed6028e30 2024-03-11T19:15:15.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.435+0000 7fecfbd9c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ed601f820 con 0x558ed6023cc0 2024-03-11T19:15:15.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.435+0000 7fecfbd9c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ed601e220 con 0x558ed60232d0 2024-03-11T19:15:15.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.435+0000 7fecf359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6028e30 0x558ed60400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.435+0000 7fecf359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6028e30 0x558ed60400a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33852/0 (socket says 172.21.15.29:33852) 2024-03-11T19:15:15.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.435+0000 7fecf359e640 1 -- 172.21.15.29:0/1071722491 learned_addr learned my addr 172.21.15.29:0/1071722491 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:15.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.435+0000 7fecf2d9d640 1 --2- 172.21.15.29:0/1071722491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 0x558ed602c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.435+0000 7fecf259c640 1 --2- 172.21.15.29:0/1071722491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6023cc0 0x558ed6034600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.435+0000 7fecf359e640 1 -- 172.21.15.29:0/1071722491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 msgr2=0x558ed602c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:15.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.435+0000 7fecf359e640 1 --2- 172.21.15.29:0/1071722491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 0x558ed602c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.435+0000 7fecf359e640 1 -- 172.21.15.29:0/1071722491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6023cc0 msgr2=0x558ed6034600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:15.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.435+0000 7fecf359e640 1 --2- 172.21.15.29:0/1071722491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6023cc0 0x558ed6034600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.436+0000 7fecf359e640 1 -- 172.21.15.29:0/1071722491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ed5f26ac0 con 0x558ed6028e30 2024-03-11T19:15:15.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.436+0000 7fecf359e640 1 --2- 172.21.15.29:0/1071722491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6028e30 0x558ed60400a0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fecec019000 tx=0x7fecec010fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c8fd8c2bd8184da server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:15.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.436+0000 7fecd8ff9640 1 -- 172.21.15.29:0/1071722491 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec005200 con 0x558ed6028e30 2024-03-11T19:15:15.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.436+0000 7fecd8ff9640 1 -- 172.21.15.29:0/1071722491 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fecec01f040 con 0x558ed6028e30 2024-03-11T19:15:15.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.436+0000 7fecd8ff9640 1 -- 172.21.15.29:0/1071722491 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec025a30 con 0x558ed6028e30 2024-03-11T19:15:15.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.437+0000 7fecfbd9c740 1 -- 172.21.15.29:0/1071722491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6028e30 msgr2=0x558ed60400a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:15.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.437+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/1071722491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6028e30 0x558ed60400a0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fecec019000 tx=0x7fecec010fd0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.437+0000 7fecfbd9c740 1 -- 172.21.15.29:0/1071722491 shutdown_connections 2024-03-11T19:15:15.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.437+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/1071722491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6028e30 0x558ed60400a0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.437+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/1071722491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6023cc0 0x558ed6034600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.437+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/1071722491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 0x558ed602c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.437+0000 7fecfbd9c740 1 -- 172.21.15.29:0/1071722491 >> 172.21.15.29:0/1071722491 conn(0x558ed601e7f0 msgr2=0x558ed6021aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:15.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.438+0000 7fecfbd9c740 1 -- 172.21.15.29:0/1071722491 shutdown_connections 2024-03-11T19:15:15.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.438+0000 7fecfbd9c740 1 -- 172.21.15.29:0/1071722491 wait complete. 2024-03-11T19:15:15.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.438+0000 7fecfbd9c740 1 -- start start 2024-03-11T19:15:15.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.438+0000 7fecfbd9c740 1 Processor -- start 2024-03-11T19:15:15.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.439+0000 7fecfbd9c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 0x558ed6032e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.439+0000 7fecfbd9c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6023cc0 0x558ed60333a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.439+0000 7fecfbd9c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6028e30 0x558ed60338e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.439+0000 7fecfbd9c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ed608f810 con 0x558ed6023cc0 2024-03-11T19:15:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.439+0000 7fecfbd9c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ed608f950 con 0x558ed6028e30 2024-03-11T19:15:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.439+0000 7fecfbd9c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ed608fa90 con 0x558ed60232d0 2024-03-11T19:15:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.439+0000 7fecf359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6028e30 0x558ed60338e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.439+0000 7fecf2d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 0x558ed6032e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.439+0000 7fecf359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6028e30 0x558ed60338e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:54268/0 (socket says 172.21.15.29:54268) 2024-03-11T19:15:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.439+0000 7fecf359e640 1 -- 172.21.15.29:0/677038670 learned_addr learned my addr 172.21.15.29:0/677038670 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.439+0000 7fecfbd9c740 10 client.0 _ll_register_callbacks cb 0x7fecec009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.439+0000 7fecf259c640 1 --2- 172.21.15.29:0/677038670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6023cc0 0x558ed60333a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.439+0000 7fecf359e640 1 -- 172.21.15.29:0/677038670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 msgr2=0x558ed6032e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.439+0000 7fecf359e640 1 --2- 172.21.15.29:0/677038670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 0x558ed6032e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.439+0000 7fecf359e640 1 -- 172.21.15.29:0/677038670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6023cc0 msgr2=0x558ed60333a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.439+0000 7fecf359e640 1 --2- 172.21.15.29:0/677038670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6023cc0 0x558ed60333a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.439+0000 7fecf359e640 1 -- 172.21.15.29:0/677038670 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558ed5f4a570 con 0x558ed6028e30 2024-03-11T19:15:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.440+0000 7fecf359e640 1 --2- 172.21.15.29:0/677038670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6028e30 0x558ed60338e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fecec0098b0 tx=0x7fecec025f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72573f86b2bbede8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.440+0000 7fecd6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:15:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.440+0000 7fecd6ff5640 1 -- 172.21.15.29:0/677038670 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec005200 con 0x558ed6028e30 2024-03-11T19:15:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.440+0000 7fecd6ff5640 1 -- 172.21.15.29:0/677038670 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fecec005360 con 0x558ed6028e30 2024-03-11T19:15:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.440+0000 7fecfbd9c740 1 -- 172.21.15.29:0/677038670 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558ed5c8b7e0 con 0x558ed6028e30 2024-03-11T19:15:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.440+0000 7fecd6ff5640 1 -- 172.21.15.29:0/677038670 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fecec005540 con 0x558ed6028e30 2024-03-11T19:15:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.440+0000 7fecfbd9c740 10 client.26300 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.441+0000 7fecfbd9c740 1 -- 172.21.15.29:0/677038670 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x558ed5f4c6a0 con 0x558ed6028e30 2024-03-11T19:15:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.441+0000 7fecd6ff5640 1 -- 172.21.15.29:0/677038670 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x558ed5c8b7e0 con 0x558ed6028e30 2024-03-11T19:15:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.441+0000 7fecd6ff5640 1 -- 172.21.15.29:0/677038670 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fecec02c050 con 0x558ed6028e30 2024-03-11T19:15:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.441+0000 7fecfbd9c740 10 client.26300 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.441+0000 7fecfbd9c740 10 client.26300 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:15:15.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.441+0000 7fecfbd9c740 20 client.26300 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:15.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.441+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/677038670 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558ed6039280 0x558ed60931e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:15.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.441+0000 7fecfbd9c740 4 client.26300 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:15:15.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.441+0000 7fecfbd9c740 1 -- 172.21.15.29:0/677038670 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558ed5d17890 con 0x558ed6039280 2024-03-11T19:15:15.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.441+0000 7fecf2d9d640 1 --2- 172.21.15.29:0/677038670 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558ed6039280 0x558ed60931e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:15.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.442+0000 7fecd6ff5640 10 client.26300 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:15.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.442+0000 7fecf2d9d640 1 --2- 172.21.15.29:0/677038670 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558ed6039280 0x558ed60931e0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7f4e1135bf9888e server_cookie=ab10c7b4090d92b1 in_seq=0 out_seq=0 2024-03-11T19:15:15.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.442+0000 7fecd6ff5640 1 -- 172.21.15.29:0/677038670 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x558ed5d17890 con 0x558ed6039280 2024-03-11T19:15:15.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.442+0000 7fecd6ff5640 10 client.26300 handle_command_reply: tid=0 2024-03-11T19:15:15.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.442+0000 7fecfbd9c740 1 client.26300 shutdown 2024-03-11T19:15:15.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.442+0000 7fecfbd9c740 1 -- 172.21.15.29:0/677038670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6028e30 msgr2=0x558ed60338e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:15.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.442+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/677038670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6028e30 0x558ed60338e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fecec0098b0 tx=0x7fecec025f60 comp rx=0 tx=0).stop 2024-03-11T19:15:15.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.443+0000 7fecfbd9c740 1 -- 172.21.15.29:0/677038670 shutdown_connections 2024-03-11T19:15:15.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.443+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/677038670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558ed6028e30 0x558ed60338e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.443+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/677038670 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558ed6039280 0x558ed60931e0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.443+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/677038670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ed6023cc0 0x558ed60333a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.443+0000 7fecfbd9c740 1 --2- 172.21.15.29:0/677038670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558ed60232d0 0x558ed6032e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.443+0000 7fecd6ff5640 0 client.26300 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:15.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.443+0000 7fecfbd9c740 1 -- 172.21.15.29:0/677038670 >> 172.21.15.29:0/677038670 conn(0x558ed601e7f0 msgr2=0x558ed6021aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:15.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.443+0000 7fecfbd9c740 1 -- 172.21.15.29:0/677038670 shutdown_connections 2024-03-11T19:15:15.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.443+0000 7fecfbd9c740 1 -- 172.21.15.29:0/677038670 wait complete. 2024-03-11T19:15:15.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.443+0000 7fecfbd9c740 20 client.26300 trim_cache size 0 max 16384 2024-03-11T19:15:15.451 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:15:15.451 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:15:15.451 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:15:15.451 INFO:teuthology.orchestra.run.smithi029.stdout: "6f0bac3b-fbf7-4bbb-95e6-c92e31f1e14d": { 2024-03-11T19:15:15.451 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:15:15.451 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "6f0bac3b-fbf7-4bbb-95e6-c92e31f1e14d", 2024-03-11T19:15:15.451 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:15:15.451 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:15:15.451 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:15:15.451 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:15:15.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.444+0000 7fecf90f4640 1 -- 172.21.15.29:0/2991347907 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fecdc0a1b30 msgr2=0x7fecdc0a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:15.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.444+0000 7fecf90f4640 1 --2- 172.21.15.29:0/2991347907 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fecdc0a1b30 0x7fecdc0a3ff0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x558ed6024520 tx=0x7fece8000bc0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.444+0000 7fecf90f4640 1 -- 172.21.15.29:0/2991347907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecf40751c0 msgr2=0x7fecf419dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:15.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.444+0000 7fecf90f4640 1 --2- 172.21.15.29:0/2991347907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecf40751c0 0x7fecf419dc40 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fece400c920 tx=0x7fece400cde0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.444+0000 7fecf90f4640 1 -- 172.21.15.29:0/2991347907 shutdown_connections 2024-03-11T19:15:15.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.444+0000 7fecf90f4640 1 --2- 172.21.15.29:0/2991347907 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecf410d790 0x7fecf41a4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.444+0000 7fecf90f4640 1 --2- 172.21.15.29:0/2991347907 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fecdc0a1b30 0x7fecdc0a3ff0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.444+0000 7fecf90f4640 1 --2- 172.21.15.29:0/2991347907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fecf40751c0 0x7fecf419dc40 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.444+0000 7fecf90f4640 1 --2- 172.21.15.29:0/2991347907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecf4073be0 0x7fecf419d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:15.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.444+0000 7fecf90f4640 1 -- 172.21.15.29:0/2991347907 >> 172.21.15.29:0/2991347907 conn(0x7fecf40fc710 msgr2=0x7fecf4100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:15.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.444+0000 7fecf90f4640 1 -- 172.21.15.29:0/2991347907 shutdown_connections 2024-03-11T19:15:15.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:15.444+0000 7fecf90f4640 1 -- 172.21.15.29:0/2991347907 wait complete. 2024-03-11T19:15:15.471 INFO:tasks.cephfs.filesystem:scrub status for tag:6f0bac3b-fbf7-4bbb-95e6-c92e31f1e14d - {'path': '/', 'tag': '6f0bac3b-fbf7-4bbb-95e6-c92e31f1e14d', 'options': 'recursive,force'} 2024-03-11T19:15:16.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:15 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:15:16.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:15 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:15:16.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:15 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:15:16.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:15 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:15:16.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:15 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:15:16.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:15 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:15:16.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:15 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:15:16.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:15 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:15:16.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:15 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:15:16.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:15 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:15:16.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:15 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:15:16.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:15 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:15:16.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:15 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:15:16.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:15 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:15:16.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:15 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:15:17.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:16 smithi170 ceph-mon[35690]: pgmap v515: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-11T19:15:17.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:16 smithi029 ceph-mon[32439]: pgmap v515: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-11T19:15:17.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:16 smithi190 ceph-mon[36292]: pgmap v515: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-11T19:15:19.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:18 smithi170 ceph-mon[35690]: pgmap v516: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 64 MiB/s wr, 1.04k op/s 2024-03-11T19:15:19.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:18 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:15:19.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:18 smithi029 ceph-mon[32439]: pgmap v516: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 64 MiB/s wr, 1.04k op/s 2024-03-11T19:15:19.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:18 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:15:19.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:18 smithi190 ceph-mon[36292]: pgmap v516: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 64 MiB/s wr, 1.04k op/s 2024-03-11T19:15:19.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:18 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:15:21.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:20 smithi170 ceph-mon[35690]: pgmap v517: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:15:21.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:20 smithi029 ceph-mon[32439]: pgmap v517: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:15:21.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:20 smithi190 ceph-mon[36292]: pgmap v517: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:15:23.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:22 smithi170 ceph-mon[35690]: pgmap v518: 97 pgs: 97 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-11T19:15:23.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:22 smithi029 ceph-mon[32439]: pgmap v518: 97 pgs: 97 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-11T19:15:23.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:22 smithi190 ceph-mon[36292]: pgmap v518: 97 pgs: 97 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-11T19:15:25.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:24 smithi170 ceph-mon[35690]: pgmap v519: 97 pgs: 97 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 65 MiB/s wr, 1.06k op/s 2024-03-11T19:15:25.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:24 smithi029 ceph-mon[32439]: pgmap v519: 97 pgs: 97 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 65 MiB/s wr, 1.06k op/s 2024-03-11T19:15:25.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:24 smithi190 ceph-mon[36292]: pgmap v519: 97 pgs: 97 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 65 MiB/s wr, 1.06k op/s 2024-03-11T19:15:27.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:26 smithi170 ceph-mon[35690]: pgmap v520: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:15:27.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:26 smithi029 ceph-mon[32439]: pgmap v520: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:15:28.098 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:26 smithi190 ceph-mon[36292]: pgmap v520: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:15:29.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:28 smithi170 ceph-mon[35690]: pgmap v521: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 61 MiB/s wr, 993 op/s 2024-03-11T19:15:29.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:28 smithi029 ceph-mon[32439]: pgmap v521: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 61 MiB/s wr, 993 op/s 2024-03-11T19:15:29.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:28 smithi190 ceph-mon[36292]: pgmap v521: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 61 MiB/s wr, 993 op/s 2024-03-11T19:15:31.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:30 smithi170 ceph-mon[35690]: pgmap v522: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-11T19:15:31.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:30 smithi029 ceph-mon[32439]: pgmap v522: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-11T19:15:31.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:30 smithi190 ceph-mon[36292]: pgmap v522: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-11T19:15:33.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:32 smithi170 ceph-mon[35690]: pgmap v523: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-11T19:15:33.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:32 smithi029 ceph-mon[32439]: pgmap v523: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-11T19:15:33.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:32 smithi190 ceph-mon[36292]: pgmap v523: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-11T19:15:35.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:34 smithi170 ceph-mon[35690]: pgmap v524: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-11T19:15:35.159 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:34 smithi029 ceph-mon[32439]: pgmap v524: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-11T19:15:35.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:34 smithi190 ceph-mon[36292]: pgmap v524: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-11T19:15:37.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:36 smithi170 ceph-mon[35690]: pgmap v525: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-03-11T19:15:37.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:36 smithi029 ceph-mon[32439]: pgmap v525: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-03-11T19:15:37.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:36 smithi190 ceph-mon[36292]: pgmap v525: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-03-11T19:15:38.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:37 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:15:38.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:37 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:15:38.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:37 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:15:38.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:37 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:15:38.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:37 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:15:38.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:37 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:15:38.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:37 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:15:38.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:37 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:15:38.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:37 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:15:39.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:38 smithi029 ceph-mon[32439]: pgmap v526: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-11T19:15:39.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:38 smithi190 ceph-mon[36292]: pgmap v526: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-11T19:15:39.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:38 smithi170 ceph-mon[35690]: pgmap v526: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-11T19:15:41.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:40 smithi029 ceph-mon[32439]: pgmap v527: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-11T19:15:41.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:40 smithi190 ceph-mon[36292]: pgmap v527: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-11T19:15:41.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:40 smithi170 ceph-mon[35690]: pgmap v527: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-11T19:15:43.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:42 smithi029 ceph-mon[32439]: pgmap v528: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-11T19:15:43.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:42 smithi190 ceph-mon[36292]: pgmap v528: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-11T19:15:43.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:42 smithi170 ceph-mon[35690]: pgmap v528: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-11T19:15:45.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:44 smithi029 ceph-mon[32439]: pgmap v529: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-03-11T19:15:45.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:44 smithi190 ceph-mon[36292]: pgmap v529: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-03-11T19:15:45.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:44 smithi170 ceph-mon[35690]: pgmap v529: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-03-11T19:15:45.472 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:15:45.472 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:15:46.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.546+0000 7fc1c759e640 1 -- 172.21.15.29:0/4114578255 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c81021b0 msgr2=0x7fc1c8102630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:46.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.546+0000 7fc1c759e640 1 --2- 172.21.15.29:0/4114578255 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c81021b0 0x7fc1c8102630 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc009ca0 tx=0x7fc1bc02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.546+0000 7fc1c759e640 1 -- 172.21.15.29:0/4114578255 shutdown_connections 2024-03-11T19:15:46.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.546+0000 7fc1c759e640 1 --2- 172.21.15.29:0/4114578255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1c8102b70 0x7fc1c8109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.546+0000 7fc1c759e640 1 --2- 172.21.15.29:0/4114578255 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c81021b0 0x7fc1c8102630 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.546+0000 7fc1c759e640 1 --2- 172.21.15.29:0/4114578255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1c8100fc0 0x7fc1c81013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.546+0000 7fc1c759e640 1 -- 172.21.15.29:0/4114578255 >> 172.21.15.29:0/4114578255 conn(0x7fc1c80fc530 msgr2=0x7fc1c80fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:46.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.546+0000 7fc1c759e640 1 -- 172.21.15.29:0/4114578255 shutdown_connections 2024-03-11T19:15:46.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.546+0000 7fc1c759e640 1 -- 172.21.15.29:0/4114578255 wait complete. 2024-03-11T19:15:46.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.547+0000 7fc1c759e640 1 Processor -- start 2024-03-11T19:15:46.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.547+0000 7fc1c759e640 1 -- start start 2024-03-11T19:15:46.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.547+0000 7fc1c759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1c8100fc0 0x7fc1c8196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.547+0000 7fc1c759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1c81021b0 0x7fc1c8197400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.547+0000 7fc1c759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c8102b70 0x7fc1c81a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.547+0000 7fc1c759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c81a0cd0 con 0x7fc1c8100fc0 2024-03-11T19:15:46.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.547+0000 7fc1c759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c81a0e10 con 0x7fc1c8102b70 2024-03-11T19:15:46.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.547+0000 7fc1c759e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c81a0f50 con 0x7fc1c81021b0 2024-03-11T19:15:46.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.547+0000 7fc1c5d9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1c81021b0 0x7fc1c8197400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.547+0000 7fc1c6d9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c8102b70 0x7fc1c81a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.547+0000 7fc1c659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1c8100fc0 0x7fc1c8196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.547+0000 7fc1c6d9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c8102b70 0x7fc1c81a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:38122/0 (socket says 172.21.15.29:38122) 2024-03-11T19:15:46.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.547+0000 7fc1c5d9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1c81021b0 0x7fc1c8197400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48926/0 (socket says 172.21.15.29:48926) 2024-03-11T19:15:46.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.547+0000 7fc1c6d9d640 1 -- 172.21.15.29:0/3815467132 learned_addr learned my addr 172.21.15.29:0/3815467132 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:46.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.548+0000 7fc1c6d9d640 1 -- 172.21.15.29:0/3815467132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1c81021b0 msgr2=0x7fc1c8197400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:46.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.548+0000 7fc1c6d9d640 1 --2- 172.21.15.29:0/3815467132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1c81021b0 0x7fc1c8197400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.548+0000 7fc1c6d9d640 1 -- 172.21.15.29:0/3815467132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1c8100fc0 msgr2=0x7fc1c8196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:46.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.548+0000 7fc1c6d9d640 1 --2- 172.21.15.29:0/3815467132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1c8100fc0 0x7fc1c8196ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.548+0000 7fc1c6d9d640 1 -- 172.21.15.29:0/3815467132 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1bc009900 con 0x7fc1c8102b70 2024-03-11T19:15:46.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.548+0000 7fc1c6d9d640 1 --2- 172.21.15.29:0/3815467132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c8102b70 0x7fc1c81a0790 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800e970 tx=0x7fc1b800ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:46.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.548+0000 7fc1b77fe640 1 -- 172.21.15.29:0/3815467132 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8009720 con 0x7fc1c8102b70 2024-03-11T19:15:46.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.548+0000 7fc1c759e640 1 -- 172.21.15.29:0/3815467132 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c81a11e0 con 0x7fc1c8102b70 2024-03-11T19:15:46.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.548+0000 7fc1b77fe640 1 -- 172.21.15.29:0/3815467132 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc1b80054d0 con 0x7fc1c8102b70 2024-03-11T19:15:46.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.548+0000 7fc1b77fe640 1 -- 172.21.15.29:0/3815467132 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8010430 con 0x7fc1c8102b70 2024-03-11T19:15:46.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.548+0000 7fc1c759e640 1 -- 172.21.15.29:0/3815467132 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c81a16c0 con 0x7fc1c8102b70 2024-03-11T19:15:46.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.552+0000 7fc1cf790740 1 Processor -- start 2024-03-11T19:15:46.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.552+0000 7fc1b77fe640 1 -- 172.21.15.29:0/3815467132 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc1b8004f30 con 0x7fc1c8102b70 2024-03-11T19:15:46.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.552+0000 7fc1cf790740 1 -- start start 2024-03-11T19:15:46.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.552+0000 7fc1cf790740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f6574f0 0x56223f657910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.552+0000 7fc1cf790740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f657ee0 0x56223f65c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.552+0000 7fc1cf790740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f65d050 0x56223f65f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.552+0000 7fc1cf790740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56223f5c7280 con 0x56223f6574f0 2024-03-11T19:15:46.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.552+0000 7fc1cf790740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56223f5c47b0 con 0x56223f65d050 2024-03-11T19:15:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.552+0000 7fc1cf790740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56223f5bff90 con 0x56223f657ee0 2024-03-11T19:15:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.553+0000 7fc1c5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f6574f0 0x56223f657910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.553+0000 7fc1c5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f6574f0 0x56223f657910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54530/0 (socket says 172.21.15.29:54530) 2024-03-11T19:15:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.553+0000 7fc1c5d9b640 1 -- 172.21.15.29:0/36699550 learned_addr learned my addr 172.21.15.29:0/36699550 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.553+0000 7fc1c5d9b640 1 --2- 172.21.15.29:0/36699550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f65d050 0x56223f65f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.553+0000 7fc1c659c640 1 --2- 172.21.15.29:0/36699550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f657ee0 0x56223f65c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.553+0000 7fc1c5d9b640 1 -- 172.21.15.29:0/36699550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f657ee0 msgr2=0x56223f65c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:46.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.553+0000 7fc1c5d9b640 1 --2- 172.21.15.29:0/36699550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f657ee0 0x56223f65c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.553+0000 7fc1c5d9b640 1 -- 172.21.15.29:0/36699550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f6574f0 msgr2=0x56223f657910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.553+0000 7fc1c5d9b640 1 --2- 172.21.15.29:0/36699550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f6574f0 0x56223f657910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.553+0000 7fc1c5d9b640 1 -- 172.21.15.29:0/36699550 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56223f57e570 con 0x56223f65d050 2024-03-11T19:15:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.554+0000 7fc1b77fe640 1 --2- 172.21.15.29:0/3815467132 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc1b00a1b30 0x7fc1b00a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.554+0000 7fc1c5d9b640 1 --2- 172.21.15.29:0/36699550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f65d050 0x56223f65f4e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x56223f658740 tx=0x7fc1bc0325d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54792801ea7ff3d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.554+0000 7fc1c659c640 1 --2- 172.21.15.29:0/3815467132 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc1b00a1b30 0x7fc1b00a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.554+0000 7fc1b4ff9640 1 -- 172.21.15.29:0/36699550 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc034640 con 0x56223f65d050 2024-03-11T19:15:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.554+0000 7fc1b4ff9640 1 -- 172.21.15.29:0/36699550 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc1bc069030 con 0x56223f65d050 2024-03-11T19:15:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.554+0000 7fc1b4ff9640 1 -- 172.21.15.29:0/36699550 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc003bd0 con 0x56223f65d050 2024-03-11T19:15:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.554+0000 7fc1b77fe640 1 -- 172.21.15.29:0/3815467132 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fc1b8014070 con 0x7fc1c8102b70 2024-03-11T19:15:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.555+0000 7fc1cf790740 1 -- 172.21.15.29:0/36699550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f65d050 msgr2=0x56223f65f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:46.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.555+0000 7fc1cf790740 1 --2- 172.21.15.29:0/36699550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f65d050 0x56223f65f4e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x56223f658740 tx=0x7fc1bc0325d0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.555+0000 7fc1cf790740 1 -- 172.21.15.29:0/36699550 shutdown_connections 2024-03-11T19:15:46.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.555+0000 7fc1cf790740 1 --2- 172.21.15.29:0/36699550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f65d050 0x56223f65f4e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.555+0000 7fc1cf790740 1 --2- 172.21.15.29:0/36699550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f657ee0 0x56223f65c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.555+0000 7fc1cf790740 1 --2- 172.21.15.29:0/36699550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f6574f0 0x56223f657910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.555+0000 7fc1cf790740 1 -- 172.21.15.29:0/36699550 >> 172.21.15.29:0/36699550 conn(0x56223f652a60 msgr2=0x56223f652e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:46.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.556+0000 7fc1cf790740 1 -- 172.21.15.29:0/36699550 shutdown_connections 2024-03-11T19:15:46.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.556+0000 7fc1c659c640 1 --2- 172.21.15.29:0/3815467132 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc1b00a1b30 0x7fc1b00a3ff0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x56223f6582e0 tx=0x7fc1c0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:46.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.556+0000 7fc1cf790740 1 -- 172.21.15.29:0/36699550 wait complete. 2024-03-11T19:15:46.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.556+0000 7fc1cf790740 1 -- start start 2024-03-11T19:15:46.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.556+0000 7fc1cf790740 1 Processor -- start 2024-03-11T19:15:46.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.557+0000 7fc1cf790740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f6574f0 0x56223f669b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.557+0000 7fc1cf790740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f657ee0 0x56223f66a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.557+0000 7fc1cf790740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f65d050 0x56223f66f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.557+0000 7fc1cf790740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56223f293430 con 0x56223f65d050 2024-03-11T19:15:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.557+0000 7fc1cf790740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56223f24fe00 con 0x56223f657ee0 2024-03-11T19:15:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.557+0000 7fc1cf790740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56223f544fd0 con 0x56223f6574f0 2024-03-11T19:15:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.557+0000 7fc1cf790740 10 client.0 _ll_register_callbacks cb 0x56223f59afd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.557+0000 7fc1c5d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f657ee0 0x56223f66a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.557+0000 7fc1c5d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f657ee0 0x56223f66a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:38138/0 (socket says 172.21.15.29:38138) 2024-03-11T19:15:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.557+0000 7fc1c5d9b640 1 -- 172.21.15.29:0/3176758700 learned_addr learned my addr 172.21.15.29:0/3176758700 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.557+0000 7fc1c6d9d640 1 --2- 172.21.15.29:0/3176758700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f65d050 0x56223f66f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.557+0000 7fc1c659c640 1 --2- 172.21.15.29:0/3176758700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f6574f0 0x56223f669b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.557+0000 7fc1c6d9d640 1 -- 172.21.15.29:0/3176758700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f6574f0 msgr2=0x56223f669b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.557+0000 7fc1c6d9d640 1 --2- 172.21.15.29:0/3176758700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f6574f0 0x56223f669b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.558+0000 7fc1c6d9d640 1 -- 172.21.15.29:0/3176758700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f657ee0 msgr2=0x56223f66a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.558+0000 7fc1c6d9d640 1 --2- 172.21.15.29:0/3176758700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f657ee0 0x56223f66a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.558+0000 7fc1c6d9d640 1 -- 172.21.15.29:0/3176758700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c8109d00 con 0x56223f65d050 2024-03-11T19:15:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.558+0000 7fc1c5d9b640 1 --2- 172.21.15.29:0/3176758700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f657ee0 0x56223f66a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.558+0000 7fc1c6d9d640 1 --2- 172.21.15.29:0/3176758700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f65d050 0x56223f66f090 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fc1b809a3a0 tx=0x7fc1b809a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b1d2521006d5e79 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.558+0000 7fc1adffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:15:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.558+0000 7fc1adffb640 1 -- 172.21.15.29:0/3176758700 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b80b3200 con 0x56223f65d050 2024-03-11T19:15:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.558+0000 7fc1cf790740 1 -- 172.21.15.29:0/3176758700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56223f2bf7e0 con 0x56223f65d050 2024-03-11T19:15:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.558+0000 7fc1adffb640 1 -- 172.21.15.29:0/3176758700 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fc1b809a6a0 con 0x56223f65d050 2024-03-11T19:15:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.559+0000 7fc1adffb640 1 -- 172.21.15.29:0/3176758700 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1b809e7f0 con 0x56223f65d050 2024-03-11T19:15:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.559+0000 7fc1cf790740 10 client.16788 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.559+0000 7fc1cf790740 1 -- 172.21.15.29:0/3176758700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56223f5be610 con 0x56223f65d050 2024-03-11T19:15:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.559+0000 7fc1adffb640 1 -- 172.21.15.29:0/3176758700 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x56223f2bf7e0 con 0x56223f65d050 2024-03-11T19:15:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.559+0000 7fc1adffb640 1 -- 172.21.15.29:0/3176758700 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc1b801f060 con 0x56223f65d050 2024-03-11T19:15:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.560+0000 7fc1cf790740 10 client.16788 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.560+0000 7fc1cf790740 10 client.16788 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:15:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.560+0000 7fc1cf790740 20 client.16788 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.560+0000 7fc1cf790740 1 --2- 172.21.15.29:0/3176758700 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56223f675ff0 0x56223f678410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.560+0000 7fc1cf790740 4 client.16788 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:15:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.560+0000 7fc1cf790740 1 -- 172.21.15.29:0/3176758700 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56223f2fc720 con 0x56223f675ff0 2024-03-11T19:15:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.560+0000 7fc1c659c640 1 --2- 172.21.15.29:0/3176758700 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56223f675ff0 0x56223f678410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.560+0000 7fc1c659c640 1 --2- 172.21.15.29:0/3176758700 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56223f675ff0 0x56223f678410 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe6246685273aac7 server_cookie=8d32f9bc668a025e in_seq=0 out_seq=0 2024-03-11T19:15:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.561+0000 7fc1adffb640 10 client.16788 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.561+0000 7fc1adffb640 1 -- 172.21.15.29:0/3176758700 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56223f2fc720 con 0x56223f675ff0 2024-03-11T19:15:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.561+0000 7fc1adffb640 10 client.16788 handle_command_reply: tid=0 2024-03-11T19:15:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.561+0000 7fc1cf790740 1 client.16788 shutdown 2024-03-11T19:15:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.562+0000 7fc1cf790740 1 -- 172.21.15.29:0/3176758700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f65d050 msgr2=0x56223f66f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.562+0000 7fc1cf790740 1 --2- 172.21.15.29:0/3176758700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f65d050 0x56223f66f090 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fc1b809a3a0 tx=0x7fc1b809a000 comp rx=0 tx=0).stop 2024-03-11T19:15:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.562+0000 7fc1cf790740 1 -- 172.21.15.29:0/3176758700 shutdown_connections 2024-03-11T19:15:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.562+0000 7fc1cf790740 1 --2- 172.21.15.29:0/3176758700 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56223f675ff0 0x56223f678410 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.562+0000 7fc1cf790740 1 --2- 172.21.15.29:0/3176758700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f65d050 0x56223f66f090 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.562+0000 7fc1cf790740 1 --2- 172.21.15.29:0/3176758700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f657ee0 0x56223f66a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.562+0000 7fc1cf790740 1 --2- 172.21.15.29:0/3176758700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f6574f0 0x56223f669b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.562+0000 7fc1cf790740 1 -- 172.21.15.29:0/3176758700 >> 172.21.15.29:0/3176758700 conn(0x56223f652a60 msgr2=0x56223f651d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.563+0000 7fc1adffb640 0 client.16788 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.563+0000 7fc1cf790740 1 -- 172.21.15.29:0/3176758700 shutdown_connections 2024-03-11T19:15:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.564+0000 7fc1cf790740 1 -- 172.21.15.29:0/3176758700 wait complete. 2024-03-11T19:15:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.564+0000 7fc1cf790740 20 client.16788 trim_cache size 0 max 16384 2024-03-11T19:15:46.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.591+0000 7fc1cf790740 1 Processor -- start 2024-03-11T19:15:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.591+0000 7fc1cf790740 1 -- start start 2024-03-11T19:15:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.592+0000 7fc1cf790740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f6574f0 0x56223f686270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.592+0000 7fc1cf790740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f657ee0 0x56223f6867b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.592+0000 7fc1cf790740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f65d050 0x56223f669b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.592+0000 7fc1cf790740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56223f2fc720 con 0x56223f6574f0 2024-03-11T19:15:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.592+0000 7fc1cf790740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56223f2bf7e0 con 0x56223f657ee0 2024-03-11T19:15:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.592+0000 7fc1cf790740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56223f658720 con 0x56223f65d050 2024-03-11T19:15:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.592+0000 7fc1c659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f6574f0 0x56223f686270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.592+0000 7fc1c5d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f657ee0 0x56223f6867b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.592+0000 7fc1c659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f6574f0 0x56223f686270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54548/0 (socket says 172.21.15.29:54548) 2024-03-11T19:15:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.592+0000 7fc1c659c640 1 -- 172.21.15.29:0/3779866814 learned_addr learned my addr 172.21.15.29:0/3779866814 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.592+0000 7fc1c5d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f657ee0 0x56223f6867b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:38142/0 (socket says 172.21.15.29:38142) 2024-03-11T19:15:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.592+0000 7fc1c6d9d640 1 --2- 172.21.15.29:0/3779866814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f65d050 0x56223f669b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.592+0000 7fc1c659c640 1 -- 172.21.15.29:0/3779866814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f65d050 msgr2=0x56223f669b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.592+0000 7fc1c659c640 1 --2- 172.21.15.29:0/3779866814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f65d050 0x56223f669b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.592+0000 7fc1c659c640 1 -- 172.21.15.29:0/3779866814 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f657ee0 msgr2=0x56223f6867b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.592+0000 7fc1c659c640 1 --2- 172.21.15.29:0/3779866814 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f657ee0 0x56223f6867b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.592+0000 7fc1c659c640 1 -- 172.21.15.29:0/3779866814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56223f658cf0 con 0x56223f6574f0 2024-03-11T19:15:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.592+0000 7fc1c659c640 1 --2- 172.21.15.29:0/3779866814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f6574f0 0x56223f686270 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0011010 tx=0x7fc1c0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d307d0ba0014ecb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:46.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.593+0000 7fc1affff640 1 -- 172.21.15.29:0/3779866814 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0015070 con 0x56223f6574f0 2024-03-11T19:15:46.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.593+0000 7fc1affff640 1 -- 172.21.15.29:0/3779866814 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc1c0022040 con 0x56223f6574f0 2024-03-11T19:15:46.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.593+0000 7fc1affff640 1 -- 172.21.15.29:0/3779866814 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c00053c0 con 0x56223f6574f0 2024-03-11T19:15:46.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.594+0000 7fc1cf790740 1 -- 172.21.15.29:0/3779866814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f6574f0 msgr2=0x56223f686270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:46.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.594+0000 7fc1cf790740 1 --2- 172.21.15.29:0/3779866814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f6574f0 0x56223f686270 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0011010 tx=0x7fc1c0019000 comp rx=0 tx=0).stop 2024-03-11T19:15:46.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.594+0000 7fc1cf790740 1 -- 172.21.15.29:0/3779866814 shutdown_connections 2024-03-11T19:15:46.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.594+0000 7fc1cf790740 1 --2- 172.21.15.29:0/3779866814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f65d050 0x56223f669b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.594+0000 7fc1cf790740 1 --2- 172.21.15.29:0/3779866814 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f657ee0 0x56223f6867b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.594+0000 7fc1cf790740 1 --2- 172.21.15.29:0/3779866814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f6574f0 0x56223f686270 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.594+0000 7fc1cf790740 1 -- 172.21.15.29:0/3779866814 >> 172.21.15.29:0/3779866814 conn(0x56223f652a60 msgr2=0x56223f64dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.594+0000 7fc1cf790740 1 -- 172.21.15.29:0/3779866814 shutdown_connections 2024-03-11T19:15:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.594+0000 7fc1cf790740 1 -- 172.21.15.29:0/3779866814 wait complete. 2024-03-11T19:15:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.594+0000 7fc1cf790740 1 -- start start 2024-03-11T19:15:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.594+0000 7fc1cf790740 1 Processor -- start 2024-03-11T19:15:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.595+0000 7fc1cf790740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f6574f0 0x56223f6c6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.595+0000 7fc1cf790740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f657ee0 0x56223f6c9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.595+0000 7fc1cf790740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f65d050 0x56223f6cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.595+0000 7fc1cf790740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56223f6cbfc0 con 0x56223f65d050 2024-03-11T19:15:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.595+0000 7fc1cf790740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56223f6cc130 con 0x56223f657ee0 2024-03-11T19:15:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.595+0000 7fc1cf790740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56223f6cc2f0 con 0x56223f6574f0 2024-03-11T19:15:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.595+0000 7fc1cf790740 10 client.0 _ll_register_callbacks cb 0x7fc1b8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:46.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.595+0000 7fc1c6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f65d050 0x56223f6cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.595+0000 7fc1c6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f65d050 0x56223f6cb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54560/0 (socket says 172.21.15.29:54560) 2024-03-11T19:15:46.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.595+0000 7fc1c6d9d640 1 -- 172.21.15.29:0/3392954944 learned_addr learned my addr 172.21.15.29:0/3392954944 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:46.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.595+0000 7fc1c5d9b640 1 --2- 172.21.15.29:0/3392954944 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f657ee0 0x56223f6c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.595+0000 7fc1c659c640 1 --2- 172.21.15.29:0/3392954944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f6574f0 0x56223f6c6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.595+0000 7fc1c6d9d640 1 -- 172.21.15.29:0/3392954944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f6574f0 msgr2=0x56223f6c6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:46.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.595+0000 7fc1c6d9d640 1 --2- 172.21.15.29:0/3392954944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f6574f0 0x56223f6c6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.595+0000 7fc1c6d9d640 1 -- 172.21.15.29:0/3392954944 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f657ee0 msgr2=0x56223f6c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:46.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.595+0000 7fc1c6d9d640 1 --2- 172.21.15.29:0/3392954944 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f657ee0 0x56223f6c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.595+0000 7fc1c6d9d640 1 -- 172.21.15.29:0/3392954944 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c000ad10 con 0x56223f65d050 2024-03-11T19:15:46.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.595+0000 7fc1c659c640 1 --2- 172.21.15.29:0/3392954944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f6574f0 0x56223f6c6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:15:46.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.596+0000 7fc1c6d9d640 1 --2- 172.21.15.29:0/3392954944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f65d050 0x56223f6cb9b0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8096b00 tx=0x7fc1b80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30f1d3e0e554178b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:46.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.596+0000 7fc1adffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:15:46.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.596+0000 7fc1adffb640 1 -- 172.21.15.29:0/3392954944 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8099cf0 con 0x56223f65d050 2024-03-11T19:15:46.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.596+0000 7fc1cf790740 1 -- 172.21.15.29:0/3392954944 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc1bc069030 con 0x56223f65d050 2024-03-11T19:15:46.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.596+0000 7fc1adffb640 1 -- 172.21.15.29:0/3392954944 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fc1b800ce30 con 0x56223f65d050 2024-03-11T19:15:46.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.596+0000 7fc1adffb640 1 -- 172.21.15.29:0/3392954944 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1b80b3200 con 0x56223f65d050 2024-03-11T19:15:46.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.596+0000 7fc1cf790740 10 client.16794 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:46.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.596+0000 7fc1cf790740 1 -- 172.21.15.29:0/3392954944 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56223f55aac0 con 0x56223f65d050 2024-03-11T19:15:46.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.597+0000 7fc1adffb640 1 -- 172.21.15.29:0/3392954944 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc1bc069030 con 0x56223f65d050 2024-03-11T19:15:46.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.597+0000 7fc1adffb640 1 -- 172.21.15.29:0/3392954944 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc1b80a9070 con 0x56223f65d050 2024-03-11T19:15:46.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.597+0000 7fc1cf790740 10 client.16794 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:46.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.597+0000 7fc1cf790740 10 client.16794 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:15:46.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.597+0000 7fc1cf790740 20 client.16794 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:46.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.597+0000 7fc1cf790740 1 --2- 172.21.15.29:0/3392954944 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56223f6cd9c0 0x56223f6cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.597+0000 7fc1cf790740 4 client.16794 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:15:46.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.597+0000 7fc1cf790740 1 -- 172.21.15.29:0/3392954944 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56223f34b890 con 0x56223f6cd9c0 2024-03-11T19:15:46.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.597+0000 7fc1c659c640 1 --2- 172.21.15.29:0/3392954944 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56223f6cd9c0 0x56223f6cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.598+0000 7fc1c659c640 1 --2- 172.21.15.29:0/3392954944 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56223f6cd9c0 0x56223f6cfde0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8905e1fb9e595d4b server_cookie=62f38c9abb5b61e7 in_seq=0 out_seq=0 2024-03-11T19:15:46.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.598+0000 7fc1adffb640 10 client.16794 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:46.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.598+0000 7fc1adffb640 1 -- 172.21.15.29:0/3392954944 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56223f34b890 con 0x56223f6cd9c0 2024-03-11T19:15:46.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.598+0000 7fc1adffb640 10 client.16794 handle_command_reply: tid=0 2024-03-11T19:15:46.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.598+0000 7fc1cf790740 1 client.16794 shutdown 2024-03-11T19:15:46.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.598+0000 7fc1cf790740 1 -- 172.21.15.29:0/3392954944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f65d050 msgr2=0x56223f6cb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:46.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.598+0000 7fc1cf790740 1 --2- 172.21.15.29:0/3392954944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f65d050 0x56223f6cb9b0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8096b00 tx=0x7fc1b80057b0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.598+0000 7fc1cf790740 1 -- 172.21.15.29:0/3392954944 shutdown_connections 2024-03-11T19:15:46.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.598+0000 7fc1cf790740 1 --2- 172.21.15.29:0/3392954944 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56223f6cd9c0 0x56223f6cfde0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.598+0000 7fc1cf790740 1 --2- 172.21.15.29:0/3392954944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56223f65d050 0x56223f6cb9b0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.598+0000 7fc1cf790740 1 --2- 172.21.15.29:0/3392954944 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56223f657ee0 0x56223f6c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.598+0000 7fc1cf790740 1 --2- 172.21.15.29:0/3392954944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56223f6574f0 0x56223f6c6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.598+0000 7fc1cf790740 1 -- 172.21.15.29:0/3392954944 >> 172.21.15.29:0/3392954944 conn(0x56223f652a60 msgr2=0x56223f659180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:46.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.598+0000 7fc1adffb640 0 client.16794 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:46.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.598+0000 7fc1cf790740 1 -- 172.21.15.29:0/3392954944 shutdown_connections 2024-03-11T19:15:46.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.598+0000 7fc1cf790740 1 -- 172.21.15.29:0/3392954944 wait complete. 2024-03-11T19:15:46.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.599+0000 7fc1cf790740 20 client.16794 trim_cache size 0 max 16384 2024-03-11T19:15:46.606 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:15:46.606 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:15:46.606 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:15:46.606 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:15:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.599+0000 7fc1c759e640 1 -- 172.21.15.29:0/3815467132 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc1b00a1b30 msgr2=0x7fc1b00a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.599+0000 7fc1c759e640 1 --2- 172.21.15.29:0/3815467132 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc1b00a1b30 0x7fc1b00a3ff0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x56223f6582e0 tx=0x7fc1c0005f90 comp rx=0 tx=0).stop 2024-03-11T19:15:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.599+0000 7fc1c759e640 1 -- 172.21.15.29:0/3815467132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c8102b70 msgr2=0x7fc1c81a0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.599+0000 7fc1c759e640 1 --2- 172.21.15.29:0/3815467132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c8102b70 0x7fc1c81a0790 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800e970 tx=0x7fc1b800ed70 comp rx=0 tx=0).stop 2024-03-11T19:15:46.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.600+0000 7fc1c759e640 1 -- 172.21.15.29:0/3815467132 shutdown_connections 2024-03-11T19:15:46.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.600+0000 7fc1c759e640 1 --2- 172.21.15.29:0/3815467132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c8102b70 0x7fc1c81a0790 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.600+0000 7fc1c759e640 1 --2- 172.21.15.29:0/3815467132 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1c81021b0 0x7fc1c8197400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.600+0000 7fc1c759e640 1 --2- 172.21.15.29:0/3815467132 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc1b00a1b30 0x7fc1b00a3ff0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.600+0000 7fc1c759e640 1 --2- 172.21.15.29:0/3815467132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1c8100fc0 0x7fc1c8196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.600+0000 7fc1c759e640 1 -- 172.21.15.29:0/3815467132 >> 172.21.15.29:0/3815467132 conn(0x7fc1c80fc530 msgr2=0x7fc1c8105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:46.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.600+0000 7fc1c759e640 1 -- 172.21.15.29:0/3815467132 shutdown_connections 2024-03-11T19:15:46.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.600+0000 7fc1c759e640 1 -- 172.21.15.29:0/3815467132 wait complete. 2024-03-11T19:15:46.621 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:15:46.622 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:15:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.786+0000 7f18682fb640 1 -- 172.21.15.29:0/2023551801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f186010bbc0 msgr2=0x7f186010dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:46.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.786+0000 7f18682fb640 1 --2- 172.21.15.29:0/2023551801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f186010bbc0 0x7f186010dfb0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f1850009900 tx=0x7f185002bc30 comp rx=0 tx=0).stop 2024-03-11T19:15:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.787+0000 7f18682fb640 1 -- 172.21.15.29:0/2023551801 shutdown_connections 2024-03-11T19:15:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.787+0000 7f18682fb640 1 --2- 172.21.15.29:0/2023551801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f186010e540 0x7f18601109e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.787+0000 7f18682fb640 1 --2- 172.21.15.29:0/2023551801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f186010bbc0 0x7f186010dfb0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.787+0000 7f18682fb640 1 --2- 172.21.15.29:0/2023551801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1860109290 0x7f186010b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.787+0000 7f18682fb640 1 -- 172.21.15.29:0/2023551801 >> 172.21.15.29:0/2023551801 conn(0x7f186006dff0 msgr2=0x7f1860070430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.787+0000 7f18682fb640 1 -- 172.21.15.29:0/2023551801 shutdown_connections 2024-03-11T19:15:46.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.787+0000 7f18682fb640 1 -- 172.21.15.29:0/2023551801 wait complete. 2024-03-11T19:15:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.788+0000 7f18682fb640 1 Processor -- start 2024-03-11T19:15:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.788+0000 7f18682fb640 1 -- start start 2024-03-11T19:15:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.788+0000 7f18682fb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1860109290 0x7f186019d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.788+0000 7f18682fb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f186010bbc0 0x7f186019db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.788+0000 7f1866097640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1860109290 0x7f186019d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.788+0000 7f1866097640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1860109290 0x7f186019d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54576/0 (socket says 172.21.15.29:54576) 2024-03-11T19:15:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.788+0000 7f18682fb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f186010e540 0x7f18601a4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.788+0000 7f18682fb640 1 -- 172.21.15.29:0/561107891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18601a50d0 con 0x7f1860109290 2024-03-11T19:15:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.788+0000 7f1866097640 1 -- 172.21.15.29:0/561107891 learned_addr learned my addr 172.21.15.29:0/561107891 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.788+0000 7f18682fb640 1 -- 172.21.15.29:0/561107891 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18601a5240 con 0x7f186010bbc0 2024-03-11T19:15:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.789+0000 7f18682fb640 1 -- 172.21.15.29:0/561107891 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18601a5400 con 0x7f186010e540 2024-03-11T19:15:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.789+0000 7f1865896640 1 --2- 172.21.15.29:0/561107891 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f186010bbc0 0x7f186019db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.789+0000 7f1866898640 1 --2- 172.21.15.29:0/561107891 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f186010e540 0x7f18601a4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.789+0000 7f1866097640 1 -- 172.21.15.29:0/561107891 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f186010e540 msgr2=0x7f18601a4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.789+0000 7f1866097640 1 --2- 172.21.15.29:0/561107891 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f186010e540 0x7f18601a4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.789+0000 7f1866097640 1 -- 172.21.15.29:0/561107891 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f186010bbc0 msgr2=0x7f186019db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.789+0000 7f1866097640 1 --2- 172.21.15.29:0/561107891 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f186010bbc0 0x7f186019db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.789+0000 7f1866097640 1 -- 172.21.15.29:0/561107891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1850009560 con 0x7f1860109290 2024-03-11T19:15:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.789+0000 7f1866898640 1 --2- 172.21.15.29:0/561107891 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f186010e540 0x7f18601a4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:15:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.789+0000 7f1866097640 1 --2- 172.21.15.29:0/561107891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1860109290 0x7f186019d5e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f185c010ad0 tx=0x7f185c010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.789+0000 7f18577fe640 1 -- 172.21.15.29:0/561107891 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c004230 con 0x7f1860109290 2024-03-11T19:15:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.789+0000 7f18577fe640 1 -- 172.21.15.29:0/561107891 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f185c011040 con 0x7f1860109290 2024-03-11T19:15:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.789+0000 7f18577fe640 1 -- 172.21.15.29:0/561107891 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c003cd0 con 0x7f1860109290 2024-03-11T19:15:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.790+0000 7f18682fb640 1 -- 172.21.15.29:0/561107891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18601a56e0 con 0x7f1860109290 2024-03-11T19:15:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.790+0000 7f18682fb640 1 -- 172.21.15.29:0/561107891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18601a5b70 con 0x7f1860109290 2024-03-11T19:15:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.791+0000 7f18682fb640 1 -- 172.21.15.29:0/561107891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1860067f70 con 0x7f1860109290 2024-03-11T19:15:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.793+0000 7f18577fe640 1 -- 172.21.15.29:0/561107891 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f185c00e970 con 0x7f1860109290 2024-03-11T19:15:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.794+0000 7f18577fe640 1 --2- 172.21.15.29:0/561107891 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f184c0a1b30 0x7f184c0a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.794+0000 7f18577fe640 1 -- 172.21.15.29:0/561107891 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f185c0d0180 con 0x7f1860109290 2024-03-11T19:15:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.794+0000 7f1865896640 1 --2- 172.21.15.29:0/561107891 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f184c0a1b30 0x7f184c0a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.795+0000 7f1865896640 1 --2- 172.21.15.29:0/561107891 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f184c0a1b30 0x7f184c0a3ff0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f185002c6a0 tx=0x7f1850000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:46.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:46.800+0000 7f18577fe640 1 -- 172.21.15.29:0/561107891 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f185c099390 con 0x7f1860109290 2024-03-11T19:15:46.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:46 smithi029 ceph-mon[32439]: pgmap v530: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-11T19:15:47.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.203+0000 7f18682fb640 1 -- 172.21.15.29:0/561107891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1860113070 con 0x7f1860109290 2024-03-11T19:15:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.205+0000 7f18577fe640 1 -- 172.21.15.29:0/561107891 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f185c09ca50 con 0x7f1860109290 2024-03-11T19:15:47.209 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:15:47.209 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:15:47.209 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.209+0000 7f18682fb640 1 -- 172.21.15.29:0/561107891 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f184c0a1b30 msgr2=0x7f184c0a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.209+0000 7f18682fb640 1 --2- 172.21.15.29:0/561107891 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f184c0a1b30 0x7f184c0a3ff0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f185002c6a0 tx=0x7f1850000bc0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.209+0000 7f18682fb640 1 -- 172.21.15.29:0/561107891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1860109290 msgr2=0x7f186019d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.209+0000 7f18682fb640 1 --2- 172.21.15.29:0/561107891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1860109290 0x7f186019d5e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f185c010ad0 tx=0x7f185c010f90 comp rx=0 tx=0).stop 2024-03-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.209+0000 7f18682fb640 1 -- 172.21.15.29:0/561107891 shutdown_connections 2024-03-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.209+0000 7f18682fb640 1 --2- 172.21.15.29:0/561107891 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f186010e540 0x7f18601a4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.209+0000 7f18682fb640 1 --2- 172.21.15.29:0/561107891 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f186010bbc0 0x7f186019db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.209+0000 7f18682fb640 1 --2- 172.21.15.29:0/561107891 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f184c0a1b30 0x7f184c0a3ff0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.209+0000 7f18682fb640 1 --2- 172.21.15.29:0/561107891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1860109290 0x7f186019d5e0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.209+0000 7f18682fb640 1 -- 172.21.15.29:0/561107891 >> 172.21.15.29:0/561107891 conn(0x7f186006dff0 msgr2=0x7f186010dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.209+0000 7f18682fb640 1 -- 172.21.15.29:0/561107891 shutdown_connections 2024-03-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.209+0000 7f18682fb640 1 -- 172.21.15.29:0/561107891 wait complete. 2024-03-11T19:15:47.231 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:15:47.231 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:15:47.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:46 smithi190 ceph-mon[36292]: pgmap v530: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-11T19:15:47.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:46 smithi170 ceph-mon[35690]: pgmap v530: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-11T19:15:47.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.408+0000 7f98a449d640 1 -- 172.21.15.29:0/1904400361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c108180 msgr2=0x7f989c10a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.408+0000 7f98a449d640 1 --2- 172.21.15.29:0/1904400361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c108180 0x7f989c10a5b0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f9894009ca0 tx=0x7f989402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.409+0000 7f98a449d640 1 -- 172.21.15.29:0/1904400361 shutdown_connections 2024-03-11T19:15:47.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.409+0000 7f98a449d640 1 --2- 172.21.15.29:0/1904400361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f989c1118c0 0x7f989c113cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.409+0000 7f98a449d640 1 --2- 172.21.15.29:0/1904400361 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f989c10aaf0 0x7f989c10af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.409+0000 7f98a449d640 1 --2- 172.21.15.29:0/1904400361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c108180 0x7f989c10a5b0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.409+0000 7f98a449d640 1 -- 172.21.15.29:0/1904400361 >> 172.21.15.29:0/1904400361 conn(0x7f989c06dfe0 msgr2=0x7f989c070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:47.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.409+0000 7f98a449d640 1 -- 172.21.15.29:0/1904400361 shutdown_connections 2024-03-11T19:15:47.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.409+0000 7f98a449d640 1 -- 172.21.15.29:0/1904400361 wait complete. 2024-03-11T19:15:47.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.410+0000 7f98a449d640 1 Processor -- start 2024-03-11T19:15:47.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.410+0000 7f98a449d640 1 -- start start 2024-03-11T19:15:47.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.410+0000 7f98a449d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f989c108180 0x7f989c10ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.410+0000 7f98a449d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f989c10aaf0 0x7f989c10f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.410+0000 7f98a449d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c1118c0 0x7f989c10b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.410+0000 7f98a449d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989c10c110 con 0x7f989c1118c0 2024-03-11T19:15:47.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.410+0000 7f98a449d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989c10c280 con 0x7f989c108180 2024-03-11T19:15:47.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.410+0000 7f98a449d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989c10c440 con 0x7f989c10aaf0 2024-03-11T19:15:47.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.410+0000 7f98a2239640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f989c108180 0x7f989c10ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.410+0000 7f98a1a38640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f989c10aaf0 0x7f989c10f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.410+0000 7f98a1a38640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f989c10aaf0 0x7f989c10f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48986/0 (socket says 172.21.15.29:48986) 2024-03-11T19:15:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.410+0000 7f98a1a38640 1 -- 172.21.15.29:0/3877688397 learned_addr learned my addr 172.21.15.29:0/3877688397 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.410+0000 7f98a2a3a640 1 --2- 172.21.15.29:0/3877688397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c1118c0 0x7f989c10b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.411+0000 7f98a1a38640 1 -- 172.21.15.29:0/3877688397 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f989c108180 msgr2=0x7f989c10ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.411+0000 7f98a1a38640 1 --2- 172.21.15.29:0/3877688397 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f989c108180 0x7f989c10ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.411+0000 7f98a1a38640 1 -- 172.21.15.29:0/3877688397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c1118c0 msgr2=0x7f989c10b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.411+0000 7f98a1a38640 1 --2- 172.21.15.29:0/3877688397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c1118c0 0x7f989c10b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.411+0000 7f98a1a38640 1 -- 172.21.15.29:0/3877688397 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9894009900 con 0x7f989c10aaf0 2024-03-11T19:15:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.411+0000 7f98a2a3a640 1 --2- 172.21.15.29:0/3877688397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c1118c0 0x7f989c10b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:15:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.411+0000 7f98a2239640 1 --2- 172.21.15.29:0/3877688397 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f989c108180 0x7f989c10ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:15:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.411+0000 7f98a1a38640 1 --2- 172.21.15.29:0/3877688397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f989c10aaf0 0x7f989c10f350 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f989800e9f0 tx=0x7f989800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.411+0000 7f98937fe640 1 -- 172.21.15.29:0/3877688397 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9898009720 con 0x7f989c10aaf0 2024-03-11T19:15:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.411+0000 7f98937fe640 1 -- 172.21.15.29:0/3877688397 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f989800cd80 con 0x7f989c10aaf0 2024-03-11T19:15:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.411+0000 7f98a449d640 1 -- 172.21.15.29:0/3877688397 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f989c1ae110 con 0x7f989c10aaf0 2024-03-11T19:15:47.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.411+0000 7f98a449d640 1 -- 172.21.15.29:0/3877688397 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f989c1ae620 con 0x7f989c10aaf0 2024-03-11T19:15:47.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.412+0000 7f98937fe640 1 -- 172.21.15.29:0/3877688397 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9898010430 con 0x7f989c10aaf0 2024-03-11T19:15:47.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.415+0000 7f98a7074740 1 Processor -- start 2024-03-11T19:15:47.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.415+0000 7f98a7074740 1 -- start start 2024-03-11T19:15:47.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.415+0000 7f98a7074740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 0x55860a166690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.415+0000 7f98a7074740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a166c60 0x55860a16b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.415+0000 7f98a7074740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a16bdd0 0x55860a16e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.415+0000 7f98a7074740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55860a122960 con 0x55860a166270 2024-03-11T19:15:47.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.415+0000 7f98a7074740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55860a0d6310 con 0x55860a16bdd0 2024-03-11T19:15:47.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.415+0000 7f98a7074740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55860a0d3840 con 0x55860a166c60 2024-03-11T19:15:47.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.416+0000 7f98a2a3a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 0x55860a166690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.416+0000 7f98a2239640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a166c60 0x55860a16b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.416+0000 7f98a2a3a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 0x55860a166690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54606/0 (socket says 172.21.15.29:54606) 2024-03-11T19:15:47.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.416+0000 7f98a2a3a640 1 -- 172.21.15.29:0/3032312151 learned_addr learned my addr 172.21.15.29:0/3032312151 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:47.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.416+0000 7f98937fe640 1 -- 172.21.15.29:0/3877688397 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9898005020 con 0x7f989c10aaf0 2024-03-11T19:15:47.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.416+0000 7f98a2a3a640 1 -- 172.21.15.29:0/3032312151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a166c60 msgr2=0x55860a16b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:47.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.416+0000 7f98a2a3a640 1 --2- 172.21.15.29:0/3032312151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a166c60 0x55860a16b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.416+0000 7f98a2a3a640 1 -- 172.21.15.29:0/3032312151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a16bdd0 msgr2=0x55860a16e260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:15:47.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.416+0000 7f98a2a3a640 1 --2- 172.21.15.29:0/3032312151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a16bdd0 0x55860a16e260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.416+0000 7f98a2a3a640 1 -- 172.21.15.29:0/3032312151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55860a091060 con 0x55860a166270 2024-03-11T19:15:47.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.417+0000 7f98a2239640 1 --2- 172.21.15.29:0/3032312151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a166c60 0x55860a16b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:47.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.417+0000 7f98937fe640 1 --2- 172.21.15.29:0/3877688397 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f98880a1ab0 0x7f98880a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.417+0000 7f98a2239640 1 --2- 172.21.15.29:0/3877688397 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f98880a1ab0 0x7f98880a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.417+0000 7f98a2a3a640 1 --2- 172.21.15.29:0/3032312151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 0x55860a166690 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f988c009c30 tx=0x7f988c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809c228292cd10b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:47.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.418+0000 7f98937fe640 1 -- 172.21.15.29:0/3877688397 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9898014070 con 0x7f989c10aaf0 2024-03-11T19:15:47.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.418+0000 7f9890bfe640 1 -- 172.21.15.29:0/3032312151 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988c012450 con 0x55860a166270 2024-03-11T19:15:47.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.418+0000 7f9890bfe640 1 -- 172.21.15.29:0/3032312151 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f988c012a60 con 0x55860a166270 2024-03-11T19:15:47.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.418+0000 7f9890bfe640 1 -- 172.21.15.29:0/3032312151 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988c00fa10 con 0x55860a166270 2024-03-11T19:15:47.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.419+0000 7f98a7074740 1 -- 172.21.15.29:0/3032312151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 msgr2=0x55860a166690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.419+0000 7f98a7074740 1 --2- 172.21.15.29:0/3032312151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 0x55860a166690 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f988c009c30 tx=0x7f988c0117b0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.419+0000 7f98a2239640 1 --2- 172.21.15.29:0/3877688397 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f98880a1ab0 0x7f98880a3f70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9894032520 tx=0x7f9894009800 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:47.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.419+0000 7f98a7074740 1 -- 172.21.15.29:0/3032312151 shutdown_connections 2024-03-11T19:15:47.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.419+0000 7f98a7074740 1 --2- 172.21.15.29:0/3032312151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a16bdd0 0x55860a16e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.420+0000 7f98a7074740 1 --2- 172.21.15.29:0/3032312151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a166c60 0x55860a16b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.420+0000 7f98a7074740 1 --2- 172.21.15.29:0/3032312151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 0x55860a166690 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.420+0000 7f98a7074740 1 -- 172.21.15.29:0/3032312151 >> 172.21.15.29:0/3032312151 conn(0x55860a1617e0 msgr2=0x55860a161be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:47.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.420+0000 7f98a7074740 1 -- 172.21.15.29:0/3032312151 shutdown_connections 2024-03-11T19:15:47.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.420+0000 7f98a7074740 1 -- 172.21.15.29:0/3032312151 wait complete. 2024-03-11T19:15:47.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.420+0000 7f98a7074740 1 -- start start 2024-03-11T19:15:47.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.420+0000 7f98a7074740 1 Processor -- start 2024-03-11T19:15:47.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.421+0000 7f98a7074740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a166270 0x55860a1704e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.421+0000 7f98a7074740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166c60 0x55860a170a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.421+0000 7f98a7074740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a16bdd0 0x55860a177a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.421+0000 7f98a1a38640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166c60 0x55860a170a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.421+0000 7f98a1a38640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166c60 0x55860a170a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54612/0 (socket says 172.21.15.29:54612) 2024-03-11T19:15:47.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.421+0000 7f98a1a38640 1 -- 172.21.15.29:0/712857530 learned_addr learned my addr 172.21.15.29:0/712857530 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:47.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.421+0000 7f98a7074740 1 -- 172.21.15.29:0/712857530 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55860a0cf020 con 0x55860a166c60 2024-03-11T19:15:47.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.422+0000 7f98a7074740 1 -- 172.21.15.29:0/712857530 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558609da2430 con 0x55860a166270 2024-03-11T19:15:47.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.422+0000 7f98a7074740 1 -- 172.21.15.29:0/712857530 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558609d5ee00 con 0x55860a16bdd0 2024-03-11T19:15:47.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.422+0000 7f98a2a3a640 1 --2- 172.21.15.29:0/712857530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a16bdd0 0x55860a177a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.422+0000 7f98a7074740 10 client.0 _ll_register_callbacks cb 0x55860a076cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:47.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.422+0000 7f98a2239640 1 --2- 172.21.15.29:0/712857530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a166270 0x55860a1704e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.422+0000 7f98a2a3a640 1 -- 172.21.15.29:0/712857530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a166270 msgr2=0x55860a1704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:47.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.422+0000 7f98a2a3a640 1 --2- 172.21.15.29:0/712857530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a166270 0x55860a1704e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.422+0000 7f98a2a3a640 1 -- 172.21.15.29:0/712857530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166c60 msgr2=0x55860a170a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:47.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.422+0000 7f98a2a3a640 1 --2- 172.21.15.29:0/712857530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166c60 0x55860a170a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.422+0000 7f98a2a3a640 1 -- 172.21.15.29:0/712857530 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f988c009860 con 0x55860a16bdd0 2024-03-11T19:15:47.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.423+0000 7f98a2a3a640 1 --2- 172.21.15.29:0/712857530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a16bdd0 0x55860a177a60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f988c01c040 tx=0x7f988c012450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93a43426b4296b43 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:47.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.423+0000 7f9885ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:15:47.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.423+0000 7f9885ffb640 1 -- 172.21.15.29:0/712857530 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988c005040 con 0x55860a16bdd0 2024-03-11T19:15:47.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.423+0000 7f9885ffb640 1 -- 172.21.15.29:0/712857530 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f988c0051a0 con 0x55860a16bdd0 2024-03-11T19:15:47.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.423+0000 7f98a7074740 1 -- 172.21.15.29:0/712857530 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558609dce7e0 con 0x55860a16bdd0 2024-03-11T19:15:47.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.424+0000 7f9885ffb640 1 -- 172.21.15.29:0/712857530 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f988c005360 con 0x55860a16bdd0 2024-03-11T19:15:47.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.424+0000 7f98a7074740 10 client.26335 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:47.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.424+0000 7f98a7074740 1 -- 172.21.15.29:0/712857530 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55860a0cd6a0 con 0x55860a16bdd0 2024-03-11T19:15:47.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.424+0000 7f9885ffb640 1 -- 172.21.15.29:0/712857530 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x558609dce7e0 con 0x55860a16bdd0 2024-03-11T19:15:47.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.424+0000 7f9885ffb640 1 -- 172.21.15.29:0/712857530 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f988c022040 con 0x55860a16bdd0 2024-03-11T19:15:47.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.425+0000 7f98a7074740 10 client.26335 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:47.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.425+0000 7f98a7074740 10 client.26335 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:15:47.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.425+0000 7f98a7074740 20 client.26335 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:47.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.425+0000 7f98a7074740 1 --2- 172.21.15.29:0/712857530 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55860a17c6b0 0x55860a17ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.425+0000 7f98a7074740 4 client.26335 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:15:47.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.425+0000 7f98a7074740 1 -- 172.21.15.29:0/712857530 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558609e0b720 con 0x55860a17c6b0 2024-03-11T19:15:47.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.426+0000 7f98a2239640 1 --2- 172.21.15.29:0/712857530 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55860a17c6b0 0x55860a17ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.427+0000 7f98a2239640 1 --2- 172.21.15.29:0/712857530 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55860a17c6b0 0x55860a17ead0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c23fef545df68205 server_cookie=542c0c6adf17b4de in_seq=0 out_seq=0 2024-03-11T19:15:47.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.427+0000 7f9885ffb640 10 client.26335 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:15:47.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.428+0000 7f9885ffb640 1 -- 172.21.15.29:0/712857530 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558609e0b720 con 0x55860a17c6b0 2024-03-11T19:15:47.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.428+0000 7f9885ffb640 10 client.26335 handle_command_reply: tid=0 2024-03-11T19:15:47.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.428+0000 7f98a7074740 1 client.26335 shutdown 2024-03-11T19:15:47.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.429+0000 7f98a7074740 1 -- 172.21.15.29:0/712857530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a16bdd0 msgr2=0x55860a177a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.429+0000 7f98a7074740 1 --2- 172.21.15.29:0/712857530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a16bdd0 0x55860a177a60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f988c01c040 tx=0x7f988c012450 comp rx=0 tx=0).stop 2024-03-11T19:15:47.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.429+0000 7f98a7074740 1 -- 172.21.15.29:0/712857530 shutdown_connections 2024-03-11T19:15:47.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.429+0000 7f98a7074740 1 --2- 172.21.15.29:0/712857530 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55860a17c6b0 0x55860a17ead0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.429+0000 7f98a7074740 1 --2- 172.21.15.29:0/712857530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a16bdd0 0x55860a177a60 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.429+0000 7f98a7074740 1 --2- 172.21.15.29:0/712857530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166c60 0x55860a170a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.429+0000 7f98a7074740 1 --2- 172.21.15.29:0/712857530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a166270 0x55860a1704e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.429+0000 7f98a7074740 1 -- 172.21.15.29:0/712857530 >> 172.21.15.29:0/712857530 conn(0x55860a1617e0 msgr2=0x55860a160b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:47.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.429+0000 7f9885ffb640 0 client.26335 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:15:47.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.429+0000 7f98a7074740 1 -- 172.21.15.29:0/712857530 shutdown_connections 2024-03-11T19:15:47.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.429+0000 7f98a7074740 1 -- 172.21.15.29:0/712857530 wait complete. 2024-03-11T19:15:47.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.431+0000 7f98a7074740 20 client.26335 trim_cache size 0 max 16384 2024-03-11T19:15:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.455+0000 7f98a7074740 1 Processor -- start 2024-03-11T19:15:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.455+0000 7f98a7074740 1 -- start start 2024-03-11T19:15:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.456+0000 7f98a7074740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 0x55860a167350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.456+0000 7f98a7074740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a166c60 0x55860a167890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.456+0000 7f98a7074740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a16bdd0 0x55860a177a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.456+0000 7f98a7074740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55860a054130 con 0x55860a166270 2024-03-11T19:15:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.456+0000 7f98a7074740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558609dce7e0 con 0x55860a16bdd0 2024-03-11T19:15:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.456+0000 7f98a7074740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55860a165e30 con 0x55860a166c60 2024-03-11T19:15:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.456+0000 7f98a2239640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 0x55860a167350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.456+0000 7f98a2239640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 0x55860a167350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54622/0 (socket says 172.21.15.29:54622) 2024-03-11T19:15:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.456+0000 7f98a2239640 1 -- 172.21.15.29:0/120970031 learned_addr learned my addr 172.21.15.29:0/120970031 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.456+0000 7f98a1a38640 1 --2- 172.21.15.29:0/120970031 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a166c60 0x55860a167890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.456+0000 7f98a2a3a640 1 --2- 172.21.15.29:0/120970031 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a16bdd0 0x55860a177a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.456+0000 7f98a2239640 1 -- 172.21.15.29:0/120970031 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a166c60 msgr2=0x55860a167890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:47.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.456+0000 7f98a2239640 1 --2- 172.21.15.29:0/120970031 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a166c60 0x55860a167890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.456+0000 7f98a2239640 1 -- 172.21.15.29:0/120970031 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a16bdd0 msgr2=0x55860a177a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:47.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.456+0000 7f98a2239640 1 --2- 172.21.15.29:0/120970031 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a16bdd0 0x55860a177a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.456+0000 7f98a2239640 1 -- 172.21.15.29:0/120970031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55860a069c20 con 0x55860a166270 2024-03-11T19:15:47.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.456+0000 7f98a1a38640 1 --2- 172.21.15.29:0/120970031 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a166c60 0x55860a167890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:15:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.457+0000 7f98a2239640 1 --2- 172.21.15.29:0/120970031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 0x55860a167350 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f9894032880 tx=0x7f9894045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5525a360e363ce server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.457+0000 7f9887fff640 1 -- 172.21.15.29:0/120970031 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9894050040 con 0x55860a166270 2024-03-11T19:15:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.458+0000 7f9887fff640 1 -- 172.21.15.29:0/120970031 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9894043070 con 0x55860a166270 2024-03-11T19:15:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.458+0000 7f9887fff640 1 -- 172.21.15.29:0/120970031 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9894032cb0 con 0x55860a166270 2024-03-11T19:15:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.458+0000 7f98a7074740 1 -- 172.21.15.29:0/120970031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 msgr2=0x55860a167350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.458+0000 7f98a7074740 1 --2- 172.21.15.29:0/120970031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 0x55860a167350 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f9894032880 tx=0x7f9894045000 comp rx=0 tx=0).stop 2024-03-11T19:15:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.458+0000 7f98a7074740 1 -- 172.21.15.29:0/120970031 shutdown_connections 2024-03-11T19:15:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.458+0000 7f98a7074740 1 --2- 172.21.15.29:0/120970031 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a16bdd0 0x55860a177a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.458+0000 7f98a7074740 1 --2- 172.21.15.29:0/120970031 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a166c60 0x55860a167890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.458+0000 7f98a7074740 1 --2- 172.21.15.29:0/120970031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 0x55860a167350 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.458+0000 7f98a7074740 1 -- 172.21.15.29:0/120970031 >> 172.21.15.29:0/120970031 conn(0x55860a1617e0 msgr2=0x55860a1681b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.459+0000 7f98a7074740 1 -- 172.21.15.29:0/120970031 shutdown_connections 2024-03-11T19:15:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.459+0000 7f98a7074740 1 -- 172.21.15.29:0/120970031 wait complete. 2024-03-11T19:15:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.459+0000 7f98a7074740 1 -- start start 2024-03-11T19:15:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.459+0000 7f98a7074740 1 Processor -- start 2024-03-11T19:15:47.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.460+0000 7f98a7074740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 0x55860a1777a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.460+0000 7f98a2239640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 0x55860a1777a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.460+0000 7f98a2239640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 0x55860a1777a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54626/0 (socket says 172.21.15.29:54626) 2024-03-11T19:15:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.460+0000 7f98a2239640 1 -- 172.21.15.29:0/3588925468 learned_addr learned my addr 172.21.15.29:0/3588925468 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.460+0000 7f98a7074740 1 --2- 172.21.15.29:0/3588925468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a166c60 0x55860a175db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.460+0000 7f98a7074740 1 --2- 172.21.15.29:0/3588925468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a16bdd0 0x55860a1762f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.460+0000 7f98a7074740 1 -- 172.21.15.29:0/3588925468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55860a176830 con 0x55860a166270 2024-03-11T19:15:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.460+0000 7f98a2a3a640 1 --2- 172.21.15.29:0/3588925468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a16bdd0 0x55860a1762f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.460+0000 7f98a1a38640 1 --2- 172.21.15.29:0/3588925468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a166c60 0x55860a175db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.460+0000 7f98a7074740 1 -- 172.21.15.29:0/3588925468 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55860a1769a0 con 0x55860a166c60 2024-03-11T19:15:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.461+0000 7f98a7074740 1 -- 172.21.15.29:0/3588925468 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55860a176b10 con 0x55860a16bdd0 2024-03-11T19:15:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.461+0000 7f98a7074740 10 client.0 _ll_register_callbacks cb 0x7f988c009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.461+0000 7f98a2239640 1 -- 172.21.15.29:0/3588925468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a16bdd0 msgr2=0x55860a1762f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.461+0000 7f98a2239640 1 --2- 172.21.15.29:0/3588925468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a16bdd0 0x55860a1762f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.461+0000 7f98a2239640 1 -- 172.21.15.29:0/3588925468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a166c60 msgr2=0x55860a175db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.461+0000 7f98a2239640 1 --2- 172.21.15.29:0/3588925468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a166c60 0x55860a175db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.461+0000 7f98a2239640 1 -- 172.21.15.29:0/3588925468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9894030070 con 0x55860a166270 2024-03-11T19:15:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.461+0000 7f98a2a3a640 1 --2- 172.21.15.29:0/3588925468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a16bdd0 0x55860a1762f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.461+0000 7f98a1a38640 1 --2- 172.21.15.29:0/3588925468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a166c60 0x55860a175db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.462+0000 7f98a7074740 1 -- 172.21.15.29:0/3588925468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f988c012a60 con 0x55860a166270 2024-03-11T19:15:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.462+0000 7f98a2239640 1 --2- 172.21.15.29:0/3588925468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 0x55860a1777a0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9894032480 tx=0x7f9894032770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d275a2a88840a693 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.462+0000 7f9885ffb640 10 client.16818 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:15:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.462+0000 7f9885ffb640 1 -- 172.21.15.29:0/3588925468 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9894050040 con 0x55860a166270 2024-03-11T19:15:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.462+0000 7f9885ffb640 1 -- 172.21.15.29:0/3588925468 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9894003b80 con 0x55860a166270 2024-03-11T19:15:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.462+0000 7f9885ffb640 1 -- 172.21.15.29:0/3588925468 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9894032cb0 con 0x55860a166270 2024-03-11T19:15:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.463+0000 7f98a7074740 10 client.16818 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.463+0000 7f98a7074740 1 -- 172.21.15.29:0/3588925468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x55860a08f8c0 con 0x55860a166270 2024-03-11T19:15:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.463+0000 7f9885ffb640 1 -- 172.21.15.29:0/3588925468 <== mon.0 v2:172.21.15.29:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9894047040 con 0x55860a166270 2024-03-11T19:15:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.463+0000 7f98a7074740 10 client.16818 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.463+0000 7f98a7074740 10 client.16818 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:15:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.463+0000 7f98a7074740 20 client.16818 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.463+0000 7f98a7074740 1 --2- 172.21.15.29:0/3588925468 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55860a1d3dc0 0x55860a1d61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.463+0000 7f98a7074740 4 client.16818 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:15:47.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.463+0000 7f98a7074740 1 -- 172.21.15.29:0/3588925468 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558609e5a890 con 0x55860a1d3dc0 2024-03-11T19:15:47.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.463+0000 7f98a1a38640 1 --2- 172.21.15.29:0/3588925468 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55860a1d3dc0 0x55860a1d61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.464+0000 7f98a1a38640 1 --2- 172.21.15.29:0/3588925468 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55860a1d3dc0 0x55860a1d61e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=37605358cc7f5ed4 server_cookie=3cf5527ba8200dac in_seq=0 out_seq=0 2024-03-11T19:15:47.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.464+0000 7f9885ffb640 10 client.16818 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:15:47.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.464+0000 7f9885ffb640 1 -- 172.21.15.29:0/3588925468 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558609e5a890 con 0x55860a1d3dc0 2024-03-11T19:15:47.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.464+0000 7f9885ffb640 10 client.16818 handle_command_reply: tid=0 2024-03-11T19:15:47.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.464+0000 7f98a7074740 1 client.16818 shutdown 2024-03-11T19:15:47.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.464+0000 7f98a7074740 1 -- 172.21.15.29:0/3588925468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 msgr2=0x55860a1777a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.464+0000 7f98a7074740 1 --2- 172.21.15.29:0/3588925468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 0x55860a1777a0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9894032480 tx=0x7f9894032770 comp rx=0 tx=0).stop 2024-03-11T19:15:47.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.464+0000 7f98a7074740 1 -- 172.21.15.29:0/3588925468 shutdown_connections 2024-03-11T19:15:47.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.464+0000 7f98a7074740 1 --2- 172.21.15.29:0/3588925468 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55860a1d3dc0 0x55860a1d61e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.465+0000 7f98a7074740 1 --2- 172.21.15.29:0/3588925468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55860a16bdd0 0x55860a1762f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.465+0000 7f9885ffb640 0 client.16818 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:15:47.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.465+0000 7f98a7074740 1 --2- 172.21.15.29:0/3588925468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55860a166c60 0x55860a175db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.465+0000 7f98a7074740 1 --2- 172.21.15.29:0/3588925468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55860a166270 0x55860a1777a0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.465+0000 7f98a7074740 1 -- 172.21.15.29:0/3588925468 >> 172.21.15.29:0/3588925468 conn(0x55860a1617e0 msgr2=0x55860a167350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:47.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.465+0000 7f98a7074740 1 -- 172.21.15.29:0/3588925468 shutdown_connections 2024-03-11T19:15:47.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.465+0000 7f98a7074740 1 -- 172.21.15.29:0/3588925468 wait complete. 2024-03-11T19:15:47.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.465+0000 7f98a7074740 20 client.16818 trim_cache size 0 max 16384 2024-03-11T19:15:47.472 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:15:47.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.467+0000 7f98a449d640 1 -- 172.21.15.29:0/3877688397 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f98880a1ab0 msgr2=0x7f98880a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.467+0000 7f98a449d640 1 --2- 172.21.15.29:0/3877688397 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f98880a1ab0 0x7f98880a3f70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9894032520 tx=0x7f9894009800 comp rx=0 tx=0).stop 2024-03-11T19:15:47.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.467+0000 7f98a449d640 1 -- 172.21.15.29:0/3877688397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f989c10aaf0 msgr2=0x7f989c10f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.467+0000 7f98a449d640 1 --2- 172.21.15.29:0/3877688397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f989c10aaf0 0x7f989c10f350 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f989800e9f0 tx=0x7f989800edf0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.467+0000 7f98a449d640 1 -- 172.21.15.29:0/3877688397 shutdown_connections 2024-03-11T19:15:47.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.467+0000 7f98a449d640 1 --2- 172.21.15.29:0/3877688397 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f98880a1ab0 0x7f98880a3f70 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.467+0000 7f98a449d640 1 --2- 172.21.15.29:0/3877688397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f989c1118c0 0x7f989c10b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.467+0000 7f98a449d640 1 --2- 172.21.15.29:0/3877688397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f989c10aaf0 0x7f989c10f350 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.467+0000 7f98a449d640 1 --2- 172.21.15.29:0/3877688397 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f989c108180 0x7f989c10ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.467+0000 7f98a449d640 1 -- 172.21.15.29:0/3877688397 >> 172.21.15.29:0/3877688397 conn(0x7f989c06dfe0 msgr2=0x7f989c1095c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:47.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.468+0000 7f98a449d640 1 -- 172.21.15.29:0/3877688397 shutdown_connections 2024-03-11T19:15:47.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.468+0000 7f98a449d640 1 -- 172.21.15.29:0/3877688397 wait complete. 2024-03-11T19:15:47.495 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:15:47.496 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:15:47.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.661+0000 7fbe5224e640 1 -- 172.21.15.29:0/2913361334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe4c10d760 msgr2=0x7fbe4c10fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.661+0000 7fbe5224e640 1 --2- 172.21.15.29:0/2913361334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe4c10d760 0x7fbe4c10fb50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009c10 tx=0x7fbe4402bee0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.662+0000 7fbe5224e640 1 -- 172.21.15.29:0/2913361334 shutdown_connections 2024-03-11T19:15:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.662+0000 7fbe5224e640 1 --2- 172.21.15.29:0/2913361334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe4c10d760 0x7fbe4c10fb50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.662+0000 7fbe5224e640 1 --2- 172.21.15.29:0/2913361334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe4c073be0 0x7fbe4c074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.662+0000 7fbe5224e640 1 --2- 172.21.15.29:0/2913361334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe4c0751c0 0x7fbe4c073610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.662+0000 7fbe5224e640 1 -- 172.21.15.29:0/2913361334 >> 172.21.15.29:0/2913361334 conn(0x7fbe4c0fc720 msgr2=0x7fbe4c0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.662+0000 7fbe5224e640 1 -- 172.21.15.29:0/2913361334 shutdown_connections 2024-03-11T19:15:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.662+0000 7fbe5224e640 1 -- 172.21.15.29:0/2913361334 wait complete. 2024-03-11T19:15:47.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.663+0000 7fbe5224e640 1 Processor -- start 2024-03-11T19:15:47.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.664+0000 7fbe5224e640 1 -- start start 2024-03-11T19:15:47.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.664+0000 7fbe5224e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe4c073be0 0x7fbe4c070e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.664+0000 7fbe5224e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe4c0751c0 0x7fbe4c071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.664+0000 7fbe5224e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe4c10d760 0x7fbe4c0723c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.665+0000 7fbe5224e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c072900 con 0x7fbe4c10d760 2024-03-11T19:15:47.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.665+0000 7fbe5224e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c072a70 con 0x7fbe4c0751c0 2024-03-11T19:15:47.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.665+0000 7fbe5224e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c072c30 con 0x7fbe4c073be0 2024-03-11T19:15:47.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.666+0000 7fbe4b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe4c073be0 0x7fbe4c070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.666+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe4c10d760 0x7fbe4c0723c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.666+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe4c10d760 0x7fbe4c0723c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54648/0 (socket says 172.21.15.29:54648) 2024-03-11T19:15:47.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.666+0000 7fbe4bfff640 1 -- 172.21.15.29:0/768663125 learned_addr learned my addr 172.21.15.29:0/768663125 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:47.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.666+0000 7fbe4bfff640 1 -- 172.21.15.29:0/768663125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe4c073be0 msgr2=0x7fbe4c070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.666+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/768663125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe4c073be0 0x7fbe4c070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.666+0000 7fbe4bfff640 1 -- 172.21.15.29:0/768663125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe4c0751c0 msgr2=0x7fbe4c071380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-11T19:15:47.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.666+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/768663125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe4c0751c0 0x7fbe4c071380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.666+0000 7fbe4bfff640 1 -- 172.21.15.29:0/768663125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe40009560 con 0x7fbe4c10d760 2024-03-11T19:15:47.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.666+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/768663125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe4c10d760 0x7fbe4c0723c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fbe44036040 tx=0x7fbe44033d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:47.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.667+0000 7fbe48ff9640 1 -- 172.21.15.29:0/768663125 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe44033e70 con 0x7fbe4c10d760 2024-03-11T19:15:47.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.667+0000 7fbe48ff9640 1 -- 172.21.15.29:0/768663125 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbe44003b80 con 0x7fbe4c10d760 2024-03-11T19:15:47.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.667+0000 7fbe5224e640 1 -- 172.21.15.29:0/768663125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe44009840 con 0x7fbe4c10d760 2024-03-11T19:15:47.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.667+0000 7fbe5224e640 1 -- 172.21.15.29:0/768663125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe4c0732b0 con 0x7fbe4c10d760 2024-03-11T19:15:47.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.668+0000 7fbe48ff9640 1 -- 172.21.15.29:0/768663125 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe44034720 con 0x7fbe4c10d760 2024-03-11T19:15:47.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.671+0000 7fbe54e1e740 1 Processor -- start 2024-03-11T19:15:47.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.673+0000 7fbe48ff9640 1 -- 172.21.15.29:0/768663125 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fbe44003cf0 con 0x7fbe4c10d760 2024-03-11T19:15:47.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.673+0000 7fbe54e1e740 1 -- start start 2024-03-11T19:15:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.673+0000 7fbe54e1e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2270 0x55f5f28b2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.673+0000 7fbe54e1e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b2c60 0x55f5f28b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.673+0000 7fbe54e1e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b7dd0 0x55f5f28ba260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.673+0000 7fbe54e1e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f286e960 con 0x55f5f28b7dd0 2024-03-11T19:15:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.673+0000 7fbe54e1e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f2822310 con 0x55f5f28b2c60 2024-03-11T19:15:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.673+0000 7fbe54e1e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f281f840 con 0x55f5f28b2270 2024-03-11T19:15:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.674+0000 7fbe4b7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b2c60 0x55f5f28b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.674+0000 7fbe4b7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b2c60 0x55f5f28b7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:38236/0 (socket says 172.21.15.29:38236) 2024-03-11T19:15:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.674+0000 7fbe4b7fe640 1 -- 172.21.15.29:0/4195987393 learned_addr learned my addr 172.21.15.29:0/4195987393 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.674+0000 7fbe4affd640 1 --2- 172.21.15.29:0/4195987393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2270 0x55f5f28b2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.674+0000 7fbe4b7fe640 1 -- 172.21.15.29:0/4195987393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2270 msgr2=0x55f5f28b2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:47.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.674+0000 7fbe4b7fe640 1 --2- 172.21.15.29:0/4195987393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2270 0x55f5f28b2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.674+0000 7fbe4b7fe640 1 -- 172.21.15.29:0/4195987393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b7dd0 msgr2=0x55f5f28ba260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:47.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.674+0000 7fbe4b7fe640 1 --2- 172.21.15.29:0/4195987393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b7dd0 0x55f5f28ba260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.674+0000 7fbe4b7fe640 1 -- 172.21.15.29:0/4195987393 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5f27dd060 con 0x55f5f28b2c60 2024-03-11T19:15:47.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.675+0000 7fbe48ff9640 1 --2- 172.21.15.29:0/768663125 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbe340a1aa0 0x7fbe340a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.675+0000 7fbe48ff9640 1 -- 172.21.15.29:0/768663125 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fbe440f02a0 con 0x7fbe4c10d760 2024-03-11T19:15:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.675+0000 7fbe4affd640 1 --2- 172.21.15.29:0/768663125 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbe340a1aa0 0x7fbe340a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.676+0000 7fbe4b7fe640 1 --2- 172.21.15.29:0/4195987393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b2c60 0x55f5f28b7760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c009a70 tx=0x7fbe3c009f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c21a5d341953e49f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.676+0000 7fbe31ffb640 1 -- 172.21.15.29:0/4195987393 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c009570 con 0x55f5f28b2c60 2024-03-11T19:15:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.676+0000 7fbe31ffb640 1 -- 172.21.15.29:0/4195987393 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbe3c013470 con 0x55f5f28b2c60 2024-03-11T19:15:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.676+0000 7fbe31ffb640 1 -- 172.21.15.29:0/4195987393 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c012610 con 0x55f5f28b2c60 2024-03-11T19:15:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.677+0000 7fbe54e1e740 1 -- 172.21.15.29:0/4195987393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b2c60 msgr2=0x55f5f28b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.677+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/4195987393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b2c60 0x55f5f28b7760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c009a70 tx=0x7fbe3c009f30 comp rx=0 tx=0).stop 2024-03-11T19:15:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.677+0000 7fbe4affd640 1 --2- 172.21.15.29:0/768663125 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbe340a1aa0 0x7fbe340a3f60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x55f5f28b34c0 tx=0x7fbe40009ae0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.678+0000 7fbe54e1e740 1 -- 172.21.15.29:0/4195987393 shutdown_connections 2024-03-11T19:15:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.678+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/4195987393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b7dd0 0x55f5f28ba260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.678+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/4195987393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b2c60 0x55f5f28b7760 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.678+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/4195987393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2270 0x55f5f28b2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.678+0000 7fbe54e1e740 1 -- 172.21.15.29:0/4195987393 >> 172.21.15.29:0/4195987393 conn(0x55f5f28ad7e0 msgr2=0x55f5f28adbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.678+0000 7fbe54e1e740 1 -- 172.21.15.29:0/4195987393 shutdown_connections 2024-03-11T19:15:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.678+0000 7fbe54e1e740 1 -- 172.21.15.29:0/4195987393 wait complete. 2024-03-11T19:15:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.679+0000 7fbe54e1e740 1 -- start start 2024-03-11T19:15:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.679+0000 7fbe54e1e740 1 Processor -- start 2024-03-11T19:15:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.679+0000 7fbe54e1e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b2270 0x55f5f28bc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.679+0000 7fbe54e1e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2c60 0x55f5f28bca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.679+0000 7fbe54e1e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b7dd0 0x55f5f28c3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.679+0000 7fbe54e1e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f281b020 con 0x55f5f28b7dd0 2024-03-11T19:15:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.679+0000 7fbe54e1e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f24ee430 con 0x55f5f28b2270 2024-03-11T19:15:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.679+0000 7fbe54e1e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f24aae00 con 0x55f5f28b2c60 2024-03-11T19:15:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.679+0000 7fbe54e1e740 10 client.0 _ll_register_callbacks cb 0x55f5f27c2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.680+0000 7fbe4b7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b2270 0x55f5f28bc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.680+0000 7fbe4affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2c60 0x55f5f28bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.680+0000 7fbe4b7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b2270 0x55f5f28bc4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:38252/0 (socket says 172.21.15.29:38252) 2024-03-11T19:15:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.680+0000 7fbe4b7fe640 1 -- 172.21.15.29:0/4277016953 learned_addr learned my addr 172.21.15.29:0/4277016953 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.680+0000 7fbe4affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2c60 0x55f5f28bca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49056/0 (socket says 172.21.15.29:49056) 2024-03-11T19:15:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.680+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/4277016953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b7dd0 0x55f5f28c3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.680+0000 7fbe4affd640 1 -- 172.21.15.29:0/4277016953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b2270 msgr2=0x55f5f28bc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.680+0000 7fbe4affd640 1 --2- 172.21.15.29:0/4277016953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b2270 0x55f5f28bc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.680+0000 7fbe4affd640 1 -- 172.21.15.29:0/4277016953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b7dd0 msgr2=0x55f5f28c3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.680+0000 7fbe4affd640 1 --2- 172.21.15.29:0/4277016953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b7dd0 0x55f5f28c3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.680+0000 7fbe4affd640 1 -- 172.21.15.29:0/4277016953 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe3c0096d0 con 0x55f5f28b2c60 2024-03-11T19:15:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.680+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/4277016953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b7dd0 0x55f5f28c3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.681+0000 7fbe4affd640 1 --2- 172.21.15.29:0/4277016953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2c60 0x55f5f28bca30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c072350 tx=0x7fbe40004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82c643cf4204d885 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:47.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.681+0000 7fbe2affd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:15:47.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.681+0000 7fbe2affd640 1 -- 172.21.15.29:0/4277016953 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4001dce0 con 0x55f5f28b2c60 2024-03-11T19:15:47.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.681+0000 7fbe2affd640 1 -- 172.21.15.29:0/4277016953 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fbe4001de40 con 0x55f5f28b2c60 2024-03-11T19:15:47.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.681+0000 7fbe54e1e740 1 -- 172.21.15.29:0/4277016953 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f5f251a7e0 con 0x55f5f28b2c60 2024-03-11T19:15:47.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.681+0000 7fbe2affd640 1 -- 172.21.15.29:0/4277016953 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe4001b460 con 0x55f5f28b2c60 2024-03-11T19:15:47.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.682+0000 7fbe54e1e740 10 client.26350 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:47.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.682+0000 7fbe54e1e740 1 -- 172.21.15.29:0/4277016953 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55f5f28196a0 con 0x55f5f28b2c60 2024-03-11T19:15:47.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.684+0000 7fbe2affd640 1 -- 172.21.15.29:0/4277016953 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f5f251a7e0 con 0x55f5f28b2c60 2024-03-11T19:15:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.684+0000 7fbe2affd640 1 -- 172.21.15.29:0/4277016953 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbe400306e0 con 0x55f5f28b2c60 2024-03-11T19:15:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.685+0000 7fbe54e1e740 10 client.26350 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.685+0000 7fbe54e1e740 10 client.26350 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:15:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.685+0000 7fbe54e1e740 20 client.26350 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.685+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/4277016953 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55f5f28c87f0 0x55f5f28cac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.685+0000 7fbe54e1e740 4 client.26350 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:15:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.685+0000 7fbe54e1e740 1 -- 172.21.15.29:0/4277016953 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f5f2557720 con 0x55f5f28c87f0 2024-03-11T19:15:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.685+0000 7fbe4b7fe640 1 --2- 172.21.15.29:0/4277016953 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55f5f28c87f0 0x55f5f28cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.686+0000 7fbe4b7fe640 1 --2- 172.21.15.29:0/4277016953 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55f5f28c87f0 0x55f5f28cac10 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6dd187e9f71fc088 server_cookie=e44fdbb35c85d711 in_seq=0 out_seq=0 2024-03-11T19:15:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.686+0000 7fbe2affd640 10 client.26350 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:15:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.687+0000 7fbe2affd640 1 -- 172.21.15.29:0/4277016953 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f5f2557720 con 0x55f5f28c87f0 2024-03-11T19:15:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.687+0000 7fbe2affd640 10 client.26350 handle_command_reply: tid=0 2024-03-11T19:15:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.687+0000 7fbe54e1e740 1 client.26350 shutdown 2024-03-11T19:15:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.687+0000 7fbe54e1e740 1 -- 172.21.15.29:0/4277016953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2c60 msgr2=0x55f5f28bca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.687+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/4277016953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2c60 0x55f5f28bca30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c072350 tx=0x7fbe40004730 comp rx=0 tx=0).stop 2024-03-11T19:15:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.688+0000 7fbe54e1e740 1 -- 172.21.15.29:0/4277016953 shutdown_connections 2024-03-11T19:15:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.688+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/4277016953 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55f5f28c87f0 0x55f5f28cac10 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.688+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/4277016953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b7dd0 0x55f5f28c3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.688+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/4277016953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2c60 0x55f5f28bca30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.688+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/4277016953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b2270 0x55f5f28bc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.688+0000 7fbe54e1e740 1 -- 172.21.15.29:0/4277016953 >> 172.21.15.29:0/4277016953 conn(0x55f5f28ad7e0 msgr2=0x55f5f28acb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.688+0000 7fbe2affd640 0 client.26350 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:15:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.688+0000 7fbe54e1e740 1 -- 172.21.15.29:0/4277016953 shutdown_connections 2024-03-11T19:15:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.688+0000 7fbe54e1e740 1 -- 172.21.15.29:0/4277016953 wait complete. 2024-03-11T19:15:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.688+0000 7fbe54e1e740 20 client.26350 trim_cache size 0 max 16384 2024-03-11T19:15:47.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.701+0000 7fbe54e1e740 1 Processor -- start 2024-03-11T19:15:47.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.702+0000 7fbe54e1e740 1 -- start start 2024-03-11T19:15:47.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.702+0000 7fbe54e1e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2270 0x55f5f28b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.702+0000 7fbe54e1e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b2c60 0x55f5f28b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.702+0000 7fbe4b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2270 0x55f5f28b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.702+0000 7fbe54e1e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b7dd0 0x55f5f28c3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.702+0000 7fbe54e1e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f27a0130 con 0x55f5f28b7dd0 2024-03-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.702+0000 7fbe54e1e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f251a7e0 con 0x55f5f28b2c60 2024-03-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.702+0000 7fbe54e1e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f289fa40 con 0x55f5f28b2270 2024-03-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.703+0000 7fbe4b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2270 0x55f5f28b42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49058/0 (socket says 172.21.15.29:49058) 2024-03-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.703+0000 7fbe4b7fe640 1 -- 172.21.15.29:0/1156137170 learned_addr learned my addr 172.21.15.29:0/1156137170 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.703+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/1156137170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b7dd0 0x55f5f28c3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.703+0000 7fbe4affd640 1 --2- 172.21.15.29:0/1156137170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b2c60 0x55f5f28b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.703+0000 7fbe4b7fe640 1 -- 172.21.15.29:0/1156137170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b2c60 msgr2=0x55f5f28b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.703+0000 7fbe4b7fe640 1 --2- 172.21.15.29:0/1156137170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b2c60 0x55f5f28b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.703+0000 7fbe4b7fe640 1 -- 172.21.15.29:0/1156137170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b7dd0 msgr2=0x55f5f28c3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.703+0000 7fbe4b7fe640 1 --2- 172.21.15.29:0/1156137170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b7dd0 0x55f5f28c3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.704+0000 7fbe4b7fe640 1 -- 172.21.15.29:0/1156137170 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5f27b5c20 con 0x55f5f28b2270 2024-03-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.704+0000 7fbe4affd640 1 --2- 172.21.15.29:0/1156137170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b2c60 0x55f5f28b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.704+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/1156137170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b7dd0 0x55f5f28c3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.704+0000 7fbe4b7fe640 1 --2- 172.21.15.29:0/1156137170 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2270 0x55f5f28b42d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c008010 tx=0x7fbe3c0047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f084d01b7d0b740 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.704+0000 7fbe317fa640 1 -- 172.21.15.29:0/1156137170 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c021040 con 0x55f5f28b2270 2024-03-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.704+0000 7fbe317fa640 1 -- 172.21.15.29:0/1156137170 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbe3c023040 con 0x55f5f28b2270 2024-03-11T19:15:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.704+0000 7fbe317fa640 1 -- 172.21.15.29:0/1156137170 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c005e60 con 0x55f5f28b2270 2024-03-11T19:15:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.705+0000 7fbe54e1e740 1 -- 172.21.15.29:0/1156137170 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2270 msgr2=0x55f5f28b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.705+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/1156137170 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2270 0x55f5f28b42d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c008010 tx=0x7fbe3c0047d0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.705+0000 7fbe54e1e740 1 -- 172.21.15.29:0/1156137170 shutdown_connections 2024-03-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.705+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/1156137170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b7dd0 0x55f5f28c3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.705+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/1156137170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b2c60 0x55f5f28b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.705+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/1156137170 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2270 0x55f5f28b42d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.705+0000 7fbe54e1e740 1 -- 172.21.15.29:0/1156137170 >> 172.21.15.29:0/1156137170 conn(0x55f5f28ad7e0 msgr2=0x55f5f28bc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.705+0000 7fbe54e1e740 1 -- 172.21.15.29:0/1156137170 shutdown_connections 2024-03-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.705+0000 7fbe54e1e740 1 -- 172.21.15.29:0/1156137170 wait complete. 2024-03-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.706+0000 7fbe54e1e740 1 -- start start 2024-03-11T19:15:47.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.708+0000 7fbe54e1e740 1 Processor -- start 2024-03-11T19:15:47.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.708+0000 7fbe54e1e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2270 0x55f5f28c3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.708+0000 7fbe54e1e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b2c60 0x55f5f28c1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.708+0000 7fbe4b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2270 0x55f5f28c3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.708+0000 7fbe4b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2270 0x55f5f28c3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49064/0 (socket says 172.21.15.29:49064) 2024-03-11T19:15:47.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.709+0000 7fbe4b7fe640 1 -- 172.21.15.29:0/608705708 learned_addr learned my addr 172.21.15.29:0/608705708 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:47.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.709+0000 7fbe4affd640 1 --2- 172.21.15.29:0/608705708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b2c60 0x55f5f28c1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.709+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/608705708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b7dd0 0x55f5f28c2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.709+0000 7fbe54e1e740 1 -- 172.21.15.29:0/608705708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f28c28a0 con 0x55f5f28b2c60 2024-03-11T19:15:47.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.709+0000 7fbe4bfff640 1 --2- 172.21.15.29:0/608705708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b7dd0 0x55f5f28c2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.709+0000 7fbe54e1e740 1 -- 172.21.15.29:0/608705708 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f28c2a10 con 0x55f5f28b7dd0 2024-03-11T19:15:47.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.709+0000 7fbe54e1e740 1 -- 172.21.15.29:0/608705708 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f28c2bd0 con 0x55f5f28b2270 2024-03-11T19:15:47.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.710+0000 7fbe54e1e740 10 client.0 _ll_register_callbacks cb 0x7fbe3c0094a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:47.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.710+0000 7fbe4b7fe640 1 -- 172.21.15.29:0/608705708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b7dd0 msgr2=0x55f5f28c2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:47.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.710+0000 7fbe4b7fe640 1 --2- 172.21.15.29:0/608705708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b7dd0 0x55f5f28c2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.710+0000 7fbe4b7fe640 1 -- 172.21.15.29:0/608705708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b2c60 msgr2=0x55f5f28c1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:47.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.710+0000 7fbe4b7fe640 1 --2- 172.21.15.29:0/608705708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b2c60 0x55f5f28c1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.710+0000 7fbe4b7fe640 1 -- 172.21.15.29:0/608705708 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5f28196a0 con 0x55f5f28b2270 2024-03-11T19:15:47.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.713+0000 7fbe4b7fe640 1 --2- 172.21.15.29:0/608705708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2270 0x55f5f28c3810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c00e660 tx=0x7fbe3c013ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=315ec03cc1c37e93 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:47.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.713+0000 7fbe2affd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:15:47.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.713+0000 7fbe2affd640 1 -- 172.21.15.29:0/608705708 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c021040 con 0x55f5f28b2270 2024-03-11T19:15:47.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.713+0000 7fbe2affd640 1 -- 172.21.15.29:0/608705708 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fbe3c00ed40 con 0x55f5f28b2270 2024-03-11T19:15:47.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.714+0000 7fbe54e1e740 1 -- 172.21.15.29:0/608705708 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbe3c013470 con 0x55f5f28b2270 2024-03-11T19:15:47.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.714+0000 7fbe2affd640 1 -- 172.21.15.29:0/608705708 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe3c0135e0 con 0x55f5f28b2270 2024-03-11T19:15:47.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.714+0000 7fbe54e1e740 10 client.26356 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:47.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.714+0000 7fbe54e1e740 1 -- 172.21.15.29:0/608705708 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7fbe4c0732b0 con 0x55f5f28b2270 2024-03-11T19:15:47.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.714+0000 7fbe2affd640 1 -- 172.21.15.29:0/608705708 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbe3c013470 con 0x55f5f28b2270 2024-03-11T19:15:47.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.715+0000 7fbe2affd640 1 -- 172.21.15.29:0/608705708 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbe3c011070 con 0x55f5f28b2270 2024-03-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.715+0000 7fbe54e1e740 10 client.26356 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.715+0000 7fbe54e1e740 10 client.26356 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.715+0000 7fbe54e1e740 20 client.26356 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.715+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/608705708 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55f5f2920000 0x55f5f29223f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.715+0000 7fbe54e1e740 4 client.26356 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.715+0000 7fbe54e1e740 1 -- 172.21.15.29:0/608705708 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f5f25a6890 con 0x55f5f2920000 2024-03-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.715+0000 7fbe4affd640 1 --2- 172.21.15.29:0/608705708 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55f5f2920000 0x55f5f29223f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.716+0000 7fbe2affd640 10 client.26356 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:15:47.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.716+0000 7fbe4affd640 1 --2- 172.21.15.29:0/608705708 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55f5f2920000 0x55f5f29223f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2e55ab483b6a0da3 server_cookie=5cb838a7923fbe45 in_seq=0 out_seq=0 2024-03-11T19:15:47.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.718+0000 7fbe2affd640 1 -- 172.21.15.29:0/608705708 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f5f25a6890 con 0x55f5f2920000 2024-03-11T19:15:47.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.718+0000 7fbe2affd640 10 client.26356 handle_command_reply: tid=0 2024-03-11T19:15:47.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.718+0000 7fbe54e1e740 1 client.26356 shutdown 2024-03-11T19:15:47.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.718+0000 7fbe54e1e740 1 -- 172.21.15.29:0/608705708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2270 msgr2=0x55f5f28c3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.718+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/608705708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2270 0x55f5f28c3810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c00e660 tx=0x7fbe3c013ed0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.720+0000 7fbe54e1e740 1 -- 172.21.15.29:0/608705708 shutdown_connections 2024-03-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.720+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/608705708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5f28b7dd0 0x55f5f28c2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.720+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/608705708 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55f5f2920000 0x55f5f29223f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.720+0000 7fbe2affd640 0 client.26356 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.720+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/608705708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5f28b2c60 0x55f5f28c1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.720+0000 7fbe54e1e740 1 --2- 172.21.15.29:0/608705708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5f28b2270 0x55f5f28c3810 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.720+0000 7fbe54e1e740 1 -- 172.21.15.29:0/608705708 >> 172.21.15.29:0/608705708 conn(0x55f5f28ad7e0 msgr2=0x55f5f28bc490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:47.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.722+0000 7fbe54e1e740 1 -- 172.21.15.29:0/608705708 shutdown_connections 2024-03-11T19:15:47.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.723+0000 7fbe54e1e740 1 -- 172.21.15.29:0/608705708 wait complete. 2024-03-11T19:15:47.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.723+0000 7fbe54e1e740 20 client.26356 trim_cache size 0 max 16384 2024-03-11T19:15:47.727 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.725+0000 7fbe5224e640 1 -- 172.21.15.29:0/768663125 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbe340a1aa0 msgr2=0x7fbe340a3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.725+0000 7fbe5224e640 1 --2- 172.21.15.29:0/768663125 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbe340a1aa0 0x7fbe340a3f60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x55f5f28b34c0 tx=0x7fbe40009ae0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.725+0000 7fbe5224e640 1 -- 172.21.15.29:0/768663125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe4c10d760 msgr2=0x7fbe4c0723c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.725+0000 7fbe5224e640 1 --2- 172.21.15.29:0/768663125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe4c10d760 0x7fbe4c0723c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fbe44036040 tx=0x7fbe44033d80 comp rx=0 tx=0).stop 2024-03-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.725+0000 7fbe5224e640 1 -- 172.21.15.29:0/768663125 shutdown_connections 2024-03-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.725+0000 7fbe5224e640 1 --2- 172.21.15.29:0/768663125 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbe340a1aa0 0x7fbe340a3f60 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.725+0000 7fbe5224e640 1 --2- 172.21.15.29:0/768663125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe4c10d760 0x7fbe4c0723c0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.725+0000 7fbe5224e640 1 --2- 172.21.15.29:0/768663125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe4c0751c0 0x7fbe4c071380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.725+0000 7fbe5224e640 1 --2- 172.21.15.29:0/768663125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe4c073be0 0x7fbe4c070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.725+0000 7fbe5224e640 1 -- 172.21.15.29:0/768663125 >> 172.21.15.29:0/768663125 conn(0x7fbe4c0fc720 msgr2=0x7fbe4c100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:47.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.725+0000 7fbe5224e640 1 -- 172.21.15.29:0/768663125 shutdown_connections 2024-03-11T19:15:47.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.726+0000 7fbe5224e640 1 -- 172.21.15.29:0/768663125 wait complete. 2024-03-11T19:15:47.751 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:15:47.751 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:15:47.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.980+0000 7f743e649640 1 -- 172.21.15.29:0/2686726184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7438100d80 msgr2=0x7f7438101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.980+0000 7f743e649640 1 --2- 172.21.15.29:0/2686726184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7438100d80 0x7f7438101200 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7430009ca0 tx=0x7f743002c180 comp rx=0 tx=0).stop 2024-03-11T19:15:47.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.981+0000 7f743e649640 1 -- 172.21.15.29:0/2686726184 shutdown_connections 2024-03-11T19:15:47.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.981+0000 7f743e649640 1 --2- 172.21.15.29:0/2686726184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7438100d80 0x7f7438101200 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.981+0000 7f743e649640 1 --2- 172.21.15.29:0/2686726184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74381003e0 0x7f7438100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.981+0000 7f743e649640 1 --2- 172.21.15.29:0/2686726184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74380ffa80 0x7f74380ffea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.981+0000 7f743e649640 1 -- 172.21.15.29:0/2686726184 >> 172.21.15.29:0/2686726184 conn(0x7f74380fb620 msgr2=0x7f74380fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.981+0000 7f743e649640 1 -- 172.21.15.29:0/2686726184 shutdown_connections 2024-03-11T19:15:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.981+0000 7f743e649640 1 -- 172.21.15.29:0/2686726184 wait complete. 2024-03-11T19:15:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.981+0000 7f743e649640 1 Processor -- start 2024-03-11T19:15:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.981+0000 7f743e649640 1 -- start start 2024-03-11T19:15:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.982+0000 7f743e649640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74380ffa80 0x7f7438199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.982+0000 7f743e649640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74381003e0 0x7f7438199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.982+0000 7f743e649640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7438100d80 0x7f74381a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.982+0000 7f743e649640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74381a0e10 con 0x7f74381003e0 2024-03-11T19:15:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.982+0000 7f743e649640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74381a0f80 con 0x7f7438100d80 2024-03-11T19:15:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.982+0000 7f743e649640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74381a1140 con 0x7f74380ffa80 2024-03-11T19:15:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.982+0000 7f74377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74381003e0 0x7f7438199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.982+0000 7f74377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74381003e0 0x7f7438199860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54704/0 (socket says 172.21.15.29:54704) 2024-03-11T19:15:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.982+0000 7f74377fe640 1 -- 172.21.15.29:0/869617222 learned_addr learned my addr 172.21.15.29:0/869617222 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.982+0000 7f7437fff640 1 --2- 172.21.15.29:0/869617222 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74380ffa80 0x7f7438199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.982+0000 7f743cbe6640 1 --2- 172.21.15.29:0/869617222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7438100d80 0x7f74381a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.982+0000 7f74377fe640 1 -- 172.21.15.29:0/869617222 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74380ffa80 msgr2=0x7f7438199320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.982+0000 7f74377fe640 1 --2- 172.21.15.29:0/869617222 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74380ffa80 0x7f7438199320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.982+0000 7f74377fe640 1 -- 172.21.15.29:0/869617222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7438100d80 msgr2=0x7f74381a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.982+0000 7f74377fe640 1 --2- 172.21.15.29:0/869617222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7438100d80 0x7f74381a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.982+0000 7f74377fe640 1 -- 172.21.15.29:0/869617222 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7430009900 con 0x7f74381003e0 2024-03-11T19:15:47.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.983+0000 7f74377fe640 1 --2- 172.21.15.29:0/869617222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74381003e0 0x7f7438199860 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f742c00d950 tx=0x7f742c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:47.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.983+0000 7f74357fa640 1 -- 172.21.15.29:0/869617222 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c009c30 con 0x7f74381003e0 2024-03-11T19:15:47.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.983+0000 7f743e649640 1 -- 172.21.15.29:0/869617222 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74381a1420 con 0x7f74381003e0 2024-03-11T19:15:47.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.983+0000 7f74357fa640 1 -- 172.21.15.29:0/869617222 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f742c009d90 con 0x7f74381003e0 2024-03-11T19:15:47.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.983+0000 7f74357fa640 1 -- 172.21.15.29:0/869617222 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c00f6b0 con 0x7f74381003e0 2024-03-11T19:15:47.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.983+0000 7f743e649640 1 -- 172.21.15.29:0/869617222 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74381a1980 con 0x7f74381003e0 2024-03-11T19:15:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.985+0000 7f74357fa640 1 -- 172.21.15.29:0/869617222 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f742c00f810 con 0x7f74381003e0 2024-03-11T19:15:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.985+0000 7f7441217740 1 Processor -- start 2024-03-11T19:15:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.985+0000 7f7441217740 1 -- start start 2024-03-11T19:15:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f7441217740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ab47c7270 0x561ab47c7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f7441217740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7c60 0x561ab47cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f7441217740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47ccdd0 0x561ab47cf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f7441217740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab4783960 con 0x561ab47c7c60 2024-03-11T19:15:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f7441217740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab4737310 con 0x561ab47ccdd0 2024-03-11T19:15:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f7441217740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab4734840 con 0x561ab47c7270 2024-03-11T19:15:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f7437fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7c60 0x561ab47cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f7437fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7c60 0x561ab47cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54712/0 (socket says 172.21.15.29:54712) 2024-03-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f7437fff640 1 -- 172.21.15.29:0/908545245 learned_addr learned my addr 172.21.15.29:0/908545245 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f743cbe6640 1 --2- 172.21.15.29:0/908545245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ab47c7270 0x561ab47c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f74377fe640 1 --2- 172.21.15.29:0/908545245 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47ccdd0 0x561ab47cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f7437fff640 1 -- 172.21.15.29:0/908545245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ab47c7270 msgr2=0x561ab47c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f7437fff640 1 --2- 172.21.15.29:0/908545245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ab47c7270 0x561ab47c7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f74357fa640 1 --2- 172.21.15.29:0/869617222 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f74200a1a30 0x7f74200a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f7437fff640 1 -- 172.21.15.29:0/908545245 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47ccdd0 msgr2=0x561ab47cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f7437fff640 1 --2- 172.21.15.29:0/908545245 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47ccdd0 0x561ab47cf260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f7437fff640 1 -- 172.21.15.29:0/908545245 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ab46f2060 con 0x561ab47c7c60 2024-03-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f74377fe640 1 --2- 172.21.15.29:0/908545245 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47ccdd0 0x561ab47cf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f743cbe6640 1 --2- 172.21.15.29:0/869617222 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f74200a1a30 0x7f74200a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f74357fa640 1 -- 172.21.15.29:0/869617222 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f742c0ccfa0 con 0x7f74381003e0 2024-03-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f7437fff640 1 --2- 172.21.15.29:0/908545245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7c60 0x561ab47cc760 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f742800bd10 tx=0x7f74280107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce8035e74ccb0e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.986+0000 7f741e7fc640 1 -- 172.21.15.29:0/908545245 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7428004030 con 0x561ab47c7c60 2024-03-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.987+0000 7f741e7fc640 1 -- 172.21.15.29:0/908545245 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7428015070 con 0x561ab47c7c60 2024-03-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.987+0000 7f741e7fc640 1 -- 172.21.15.29:0/908545245 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74280116a0 con 0x561ab47c7c60 2024-03-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.987+0000 7f7441217740 1 -- 172.21.15.29:0/908545245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7c60 msgr2=0x561ab47cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.987+0000 7f7441217740 1 --2- 172.21.15.29:0/908545245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7c60 0x561ab47cc760 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f742800bd10 tx=0x7f74280107a0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.987+0000 7f743cbe6640 1 --2- 172.21.15.29:0/869617222 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f74200a1a30 0x7f74200a3ef0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7430005cc0 tx=0x7f7430005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.987+0000 7f7441217740 1 -- 172.21.15.29:0/908545245 shutdown_connections 2024-03-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.987+0000 7f7441217740 1 --2- 172.21.15.29:0/908545245 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47ccdd0 0x561ab47cf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.987+0000 7f7441217740 1 --2- 172.21.15.29:0/908545245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7c60 0x561ab47cc760 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.987+0000 7f7441217740 1 --2- 172.21.15.29:0/908545245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ab47c7270 0x561ab47c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.987+0000 7f7441217740 1 -- 172.21.15.29:0/908545245 >> 172.21.15.29:0/908545245 conn(0x561ab47c27e0 msgr2=0x561ab47c2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.987+0000 7f7441217740 1 -- 172.21.15.29:0/908545245 shutdown_connections 2024-03-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.987+0000 7f7441217740 1 -- 172.21.15.29:0/908545245 wait complete. 2024-03-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.988+0000 7f7441217740 1 -- start start 2024-03-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.988+0000 7f7441217740 1 Processor -- start 2024-03-11T19:15:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.988+0000 7f7441217740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47c7270 0x561ab47d1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.988+0000 7f7441217740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ab47c7c60 0x561ab47d19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.988+0000 7f7441217740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47ccdd0 0x561ab47d8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.988+0000 7f7441217740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab4730020 con 0x561ab47ccdd0 2024-03-11T19:15:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.988+0000 7f7441217740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab4403430 con 0x561ab47c7270 2024-03-11T19:15:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.988+0000 7f7441217740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab43bfe00 con 0x561ab47c7c60 2024-03-11T19:15:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.988+0000 7f7441217740 10 client.0 _ll_register_callbacks cb 0x561ab46d7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.988+0000 7f743cbe6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47ccdd0 0x561ab47d8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.988+0000 7f743cbe6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47ccdd0 0x561ab47d8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54716/0 (socket says 172.21.15.29:54716) 2024-03-11T19:15:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.988+0000 7f743cbe6640 1 -- 172.21.15.29:0/2246287281 learned_addr learned my addr 172.21.15.29:0/2246287281 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.988+0000 7f74377fe640 1 --2- 172.21.15.29:0/2246287281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ab47c7c60 0x561ab47d19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.988+0000 7f7437fff640 1 --2- 172.21.15.29:0/2246287281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47c7270 0x561ab47d1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f743cbe6640 1 -- 172.21.15.29:0/2246287281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ab47c7c60 msgr2=0x561ab47d19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f743cbe6640 1 --2- 172.21.15.29:0/2246287281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ab47c7c60 0x561ab47d19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f743cbe6640 1 -- 172.21.15.29:0/2246287281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47c7270 msgr2=0x561ab47d1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f743cbe6640 1 --2- 172.21.15.29:0/2246287281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47c7270 0x561ab47d1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f743cbe6640 1 -- 172.21.15.29:0/2246287281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f742800b940 con 0x561ab47ccdd0 2024-03-11T19:15:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f7437fff640 1 --2- 172.21.15.29:0/2246287281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47c7270 0x561ab47d1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:15:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f743cbe6640 1 --2- 172.21.15.29:0/2246287281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47ccdd0 0x561ab47d8a00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f7430006010 tx=0x7f74300095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c45883f40d3461a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f74177fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:15:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f74177fe640 1 -- 172.21.15.29:0/2246287281 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7430032eb0 con 0x561ab47ccdd0 2024-03-11T19:15:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f74177fe640 1 -- 172.21.15.29:0/2246287281 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7430003f80 con 0x561ab47ccdd0 2024-03-11T19:15:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f7441217740 1 -- 172.21.15.29:0/2246287281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561ab442f7e0 con 0x561ab47ccdd0 2024-03-11T19:15:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f74177fe640 1 -- 172.21.15.29:0/2246287281 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7430004140 con 0x561ab47ccdd0 2024-03-11T19:15:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f7441217740 10 client.16842 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f7441217740 1 -- 172.21.15.29:0/2246287281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x561ab472e6a0 con 0x561ab47ccdd0 2024-03-11T19:15:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f74177fe640 1 -- 172.21.15.29:0/2246287281 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x561ab442f7e0 con 0x561ab47ccdd0 2024-03-11T19:15:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f74177fe640 1 -- 172.21.15.29:0/2246287281 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7430053ab0 con 0x561ab47ccdd0 2024-03-11T19:15:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f7441217740 10 client.16842 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f7441217740 10 client.16842 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:15:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f7441217740 20 client.16842 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f7441217740 1 --2- 172.21.15.29:0/2246287281 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x561ab47dd6e0 0x561ab47dfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f7441217740 4 client.16842 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:15:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.989+0000 7f7441217740 1 -- 172.21.15.29:0/2246287281 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561ab446c720 con 0x561ab47dd6e0 2024-03-11T19:15:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.990+0000 7f7437fff640 1 --2- 172.21.15.29:0/2246287281 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x561ab47dd6e0 0x561ab47dfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.990+0000 7f7437fff640 1 --2- 172.21.15.29:0/2246287281 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x561ab47dd6e0 0x561ab47dfb20 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6110c261a251c07e server_cookie=a15825023e6d6b44 in_seq=0 out_seq=0 2024-03-11T19:15:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.990+0000 7f74177fe640 10 client.16842 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:15:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.991+0000 7f74177fe640 1 -- 172.21.15.29:0/2246287281 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561ab446c720 con 0x561ab47dd6e0 2024-03-11T19:15:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.991+0000 7f74177fe640 10 client.16842 handle_command_reply: tid=0 2024-03-11T19:15:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.991+0000 7f7441217740 1 client.16842 shutdown 2024-03-11T19:15:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.991+0000 7f7441217740 1 -- 172.21.15.29:0/2246287281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47ccdd0 msgr2=0x561ab47d8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.991+0000 7f7441217740 1 --2- 172.21.15.29:0/2246287281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47ccdd0 0x561ab47d8a00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f7430006010 tx=0x7f74300095c0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.991+0000 7f7441217740 1 -- 172.21.15.29:0/2246287281 shutdown_connections 2024-03-11T19:15:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.991+0000 7f7441217740 1 --2- 172.21.15.29:0/2246287281 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x561ab47dd6e0 0x561ab47dfb20 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.991+0000 7f7441217740 1 --2- 172.21.15.29:0/2246287281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47ccdd0 0x561ab47d8a00 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.991+0000 7f7441217740 1 --2- 172.21.15.29:0/2246287281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ab47c7c60 0x561ab47d19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.991+0000 7f74177fe640 0 client.16842 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:15:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.991+0000 7f7441217740 1 --2- 172.21.15.29:0/2246287281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47c7270 0x561ab47d1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.991+0000 7f7441217740 1 -- 172.21.15.29:0/2246287281 >> 172.21.15.29:0/2246287281 conn(0x561ab47c27e0 msgr2=0x561ab47c1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.991+0000 7f7441217740 1 -- 172.21.15.29:0/2246287281 shutdown_connections 2024-03-11T19:15:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.991+0000 7f7441217740 1 -- 172.21.15.29:0/2246287281 wait complete. 2024-03-11T19:15:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:47.991+0000 7f7441217740 20 client.16842 trim_cache size 0 max 16384 2024-03-11T19:15:48.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.003+0000 7f7441217740 1 Processor -- start 2024-03-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.003+0000 7f7441217740 1 -- start start 2024-03-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.003+0000 7f7441217740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7270 0x561ab47c9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.003+0000 7f7441217740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47c7c60 0x561ab47d89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.003+0000 7f7441217740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ab47ccdd0 0x561ab47c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.003+0000 7f7441217740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab46b5130 con 0x561ab47c7270 2024-03-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.003+0000 7f7441217740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab446c720 con 0x561ab47c7c60 2024-03-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.003+0000 7f7441217740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab442f7e0 con 0x561ab47ccdd0 2024-03-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.003+0000 7f7437fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7270 0x561ab47c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.003+0000 7f7437fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7270 0x561ab47c9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54722/0 (socket says 172.21.15.29:54722) 2024-03-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.003+0000 7f7437fff640 1 -- 172.21.15.29:0/935860862 learned_addr learned my addr 172.21.15.29:0/935860862 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.003+0000 7f74377fe640 1 --2- 172.21.15.29:0/935860862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47c7c60 0x561ab47d89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.003+0000 7f7437fff640 1 -- 172.21.15.29:0/935860862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ab47ccdd0 msgr2=0x561ab47c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.003+0000 7f7437fff640 1 --2- 172.21.15.29:0/935860862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ab47ccdd0 0x561ab47c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.003+0000 7f7437fff640 1 -- 172.21.15.29:0/935860862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47c7c60 msgr2=0x561ab47d89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.003+0000 7f7437fff640 1 --2- 172.21.15.29:0/935860862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47c7c60 0x561ab47d89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.003+0000 7f7437fff640 1 -- 172.21.15.29:0/935860862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ab46cac20 con 0x561ab47c7270 2024-03-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.003+0000 7f74377fe640 1 --2- 172.21.15.29:0/935860862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47c7c60 0x561ab47d89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.003+0000 7f7437fff640 1 --2- 172.21.15.29:0/935860862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7270 0x561ab47c9160 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f742800bce0 tx=0x7f7428005d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93ddfd84d6e8baaa server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.004+0000 7f741dffb640 1 -- 172.21.15.29:0/935860862 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7428005240 con 0x561ab47c7270 2024-03-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.004+0000 7f741dffb640 1 -- 172.21.15.29:0/935860862 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7428021040 con 0x561ab47c7270 2024-03-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.004+0000 7f741dffb640 1 -- 172.21.15.29:0/935860862 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74280283c0 con 0x561ab47c7270 2024-03-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.004+0000 7f7441217740 1 -- 172.21.15.29:0/935860862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7270 msgr2=0x561ab47c9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.004+0000 7f7441217740 1 --2- 172.21.15.29:0/935860862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7270 0x561ab47c9160 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f742800bce0 tx=0x7f7428005d10 comp rx=0 tx=0).stop 2024-03-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.004+0000 7f7441217740 1 -- 172.21.15.29:0/935860862 shutdown_connections 2024-03-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.004+0000 7f7441217740 1 --2- 172.21.15.29:0/935860862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ab47ccdd0 0x561ab47c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.004+0000 7f7441217740 1 --2- 172.21.15.29:0/935860862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47c7c60 0x561ab47d89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.004+0000 7f7441217740 1 --2- 172.21.15.29:0/935860862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7270 0x561ab47c9160 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.004+0000 7f7441217740 1 -- 172.21.15.29:0/935860862 >> 172.21.15.29:0/935860862 conn(0x561ab47c27e0 msgr2=0x561ab47c2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.004+0000 7f7441217740 1 -- 172.21.15.29:0/935860862 shutdown_connections 2024-03-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.004+0000 7f7441217740 1 -- 172.21.15.29:0/935860862 wait complete. 2024-03-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.004+0000 7f7441217740 1 -- start start 2024-03-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.005+0000 7f7441217740 1 Processor -- start 2024-03-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.005+0000 7f7441217740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47c7270 0x561ab47d7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.005+0000 7f7441217740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7c60 0x561ab47d7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.005+0000 7f7441217740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ab47ccdd0 0x561ab4833c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.005+0000 7f7441217740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab47d8410 con 0x561ab47c7c60 2024-03-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.005+0000 7f7441217740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab4834170 con 0x561ab47c7270 2024-03-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.005+0000 7f7441217740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab48342b0 con 0x561ab47ccdd0 2024-03-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.005+0000 7f7441217740 10 client.0 _ll_register_callbacks cb 0x7f742800b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.005+0000 7f74377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7c60 0x561ab47d7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.005+0000 7f74377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7c60 0x561ab47d7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54734/0 (socket says 172.21.15.29:54734) 2024-03-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.005+0000 7f74377fe640 1 -- 172.21.15.29:0/3543392881 learned_addr learned my addr 172.21.15.29:0/3543392881 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.005+0000 7f7437fff640 1 --2- 172.21.15.29:0/3543392881 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47c7270 0x561ab47d7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.005+0000 7f743cbe6640 1 --2- 172.21.15.29:0/3543392881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ab47ccdd0 0x561ab4833c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.005+0000 7f74377fe640 1 -- 172.21.15.29:0/3543392881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ab47ccdd0 msgr2=0x561ab4833c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.005+0000 7f74377fe640 1 --2- 172.21.15.29:0/3543392881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ab47ccdd0 0x561ab4833c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.005+0000 7f74377fe640 1 -- 172.21.15.29:0/3543392881 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47c7270 msgr2=0x561ab47d7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.005+0000 7f74377fe640 1 --2- 172.21.15.29:0/3543392881 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47c7270 0x561ab47d7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.005+0000 7f74377fe640 1 -- 172.21.15.29:0/3543392881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ab46f2060 con 0x561ab47c7c60 2024-03-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.005+0000 7f7437fff640 1 --2- 172.21.15.29:0/3543392881 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47c7270 0x561ab47d7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.006+0000 7f74377fe640 1 --2- 172.21.15.29:0/3543392881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7c60 0x561ab47d7db0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f742c099d60 tx=0x7f742c09a120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d80ef7291cd0daf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.006+0000 7f74177fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.006+0000 7f74177fe640 1 -- 172.21.15.29:0/3543392881 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c0966d0 con 0x561ab47c7c60 2024-03-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.006+0000 7f7441217740 1 -- 172.21.15.29:0/3543392881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7428015070 con 0x561ab47c7c60 2024-03-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.006+0000 7f74177fe640 1 -- 172.21.15.29:0/3543392881 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f742c096830 con 0x561ab47c7c60 2024-03-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.006+0000 7f74177fe640 1 -- 172.21.15.29:0/3543392881 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f742c0cf070 con 0x561ab47c7c60 2024-03-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.006+0000 7f7441217740 10 client.16848 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.006+0000 7f7441217740 1 -- 172.21.15.29:0/3543392881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x561ab46f08c0 con 0x561ab47c7c60 2024-03-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.006+0000 7f74177fe640 1 -- 172.21.15.29:0/3543392881 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7428015070 con 0x561ab47c7c60 2024-03-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.006+0000 7f74177fe640 1 -- 172.21.15.29:0/3543392881 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f742c0b8240 con 0x561ab47c7c60 2024-03-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.006+0000 7f7441217740 10 client.16848 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.006+0000 7f7441217740 10 client.16848 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.006+0000 7f7441217740 20 client.16848 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.006+0000 7f7441217740 1 --2- 172.21.15.29:0/3543392881 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x561ab4835ce0 0x561ab4838100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.006+0000 7f7441217740 4 client.16848 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.006+0000 7f7441217740 1 -- 172.21.15.29:0/3543392881 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561ab44bb890 con 0x561ab4835ce0 2024-03-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.007+0000 7f7437fff640 1 --2- 172.21.15.29:0/3543392881 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x561ab4835ce0 0x561ab4838100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.007+0000 7f74177fe640 10 client.16848 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.007+0000 7f7437fff640 1 --2- 172.21.15.29:0/3543392881 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x561ab4835ce0 0x561ab4838100 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=19d1cbe851fd46d8 server_cookie=2dcb0bbf23b4d47b in_seq=0 out_seq=0 2024-03-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.007+0000 7f74177fe640 1 -- 172.21.15.29:0/3543392881 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561ab44bb890 con 0x561ab4835ce0 2024-03-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.007+0000 7f74177fe640 10 client.16848 handle_command_reply: tid=0 2024-03-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.007+0000 7f7441217740 1 client.16848 shutdown 2024-03-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.007+0000 7f7441217740 1 -- 172.21.15.29:0/3543392881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7c60 msgr2=0x561ab47d7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.007+0000 7f7441217740 1 --2- 172.21.15.29:0/3543392881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7c60 0x561ab47d7db0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f742c099d60 tx=0x7f742c09a120 comp rx=0 tx=0).stop 2024-03-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f7441217740 1 -- 172.21.15.29:0/3543392881 shutdown_connections 2024-03-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f7441217740 1 --2- 172.21.15.29:0/3543392881 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x561ab4835ce0 0x561ab4838100 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f7441217740 1 --2- 172.21.15.29:0/3543392881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561ab47ccdd0 0x561ab4833c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f7441217740 1 --2- 172.21.15.29:0/3543392881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ab47c7c60 0x561ab47d7db0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f7441217740 1 --2- 172.21.15.29:0/3543392881 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ab47c7270 0x561ab47d7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f7441217740 1 -- 172.21.15.29:0/3543392881 >> 172.21.15.29:0/3543392881 conn(0x561ab47c27e0 msgr2=0x561ab47d1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f74177fe640 0 client.16848 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f7441217740 1 -- 172.21.15.29:0/3543392881 shutdown_connections 2024-03-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f7441217740 1 -- 172.21.15.29:0/3543392881 wait complete. 2024-03-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f7441217740 20 client.16848 trim_cache size 0 max 16384 2024-03-11T19:15:48.018 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:15:48.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f743e649640 1 -- 172.21.15.29:0/869617222 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f74200a1a30 msgr2=0x7f74200a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f743e649640 1 --2- 172.21.15.29:0/869617222 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f74200a1a30 0x7f74200a3ef0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7430005cc0 tx=0x7f7430005c50 comp rx=0 tx=0).stop 2024-03-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f743e649640 1 -- 172.21.15.29:0/869617222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74381003e0 msgr2=0x7f7438199860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f743e649640 1 --2- 172.21.15.29:0/869617222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74381003e0 0x7f7438199860 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f742c00d950 tx=0x7f742c00de10 comp rx=0 tx=0).stop 2024-03-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f743e649640 1 -- 172.21.15.29:0/869617222 shutdown_connections 2024-03-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f743e649640 1 --2- 172.21.15.29:0/869617222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7438100d80 0x7f74381a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f743e649640 1 --2- 172.21.15.29:0/869617222 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f74200a1a30 0x7f74200a3ef0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f743e649640 1 --2- 172.21.15.29:0/869617222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74381003e0 0x7f7438199860 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f743e649640 1 --2- 172.21.15.29:0/869617222 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74380ffa80 0x7f7438199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f743e649640 1 -- 172.21.15.29:0/869617222 >> 172.21.15.29:0/869617222 conn(0x7f74380fb620 msgr2=0x7f74380fcdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f743e649640 1 -- 172.21.15.29:0/869617222 shutdown_connections 2024-03-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.008+0000 7f743e649640 1 -- 172.21.15.29:0/869617222 wait complete. 2024-03-11T19:15:48.021 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:47 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/561107891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:15:48.022 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:15:48.023 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:15:48.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.174+0000 7fe6fab08640 1 -- 172.21.15.29:0/2416467565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6f410d4a0 msgr2=0x7fe6f410d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.174+0000 7fe6fab08640 1 --2- 172.21.15.29:0/2416467565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6f410d4a0 0x7fe6f410d920 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec009ca0 tx=0x7fe6ec02c100 comp rx=0 tx=0).stop 2024-03-11T19:15:48.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.174+0000 7fe6fab08640 1 -- 172.21.15.29:0/2416467565 shutdown_connections 2024-03-11T19:15:48.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.174+0000 7fe6fab08640 1 --2- 172.21.15.29:0/2416467565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6f410d4a0 0x7fe6f410d920 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.174+0000 7fe6fab08640 1 --2- 172.21.15.29:0/2416467565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6f410aaf0 0x7fe6f410cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.174+0000 7fe6fab08640 1 --2- 172.21.15.29:0/2416467565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6f4108180 0x7fe6f410a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.174+0000 7fe6fab08640 1 -- 172.21.15.29:0/2416467565 >> 172.21.15.29:0/2416467565 conn(0x7fe6f406dfe0 msgr2=0x7fe6f4070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:48.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.174+0000 7fe6fab08640 1 -- 172.21.15.29:0/2416467565 shutdown_connections 2024-03-11T19:15:48.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.174+0000 7fe6fab08640 1 -- 172.21.15.29:0/2416467565 wait complete. 2024-03-11T19:15:48.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.174+0000 7fe6fab08640 1 Processor -- start 2024-03-11T19:15:48.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.174+0000 7fe6fab08640 1 -- start start 2024-03-11T19:15:48.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.175+0000 7fe6fab08640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6f4108180 0x7fe6f41a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.175+0000 7fe6fab08640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6f410aaf0 0x7fe6f41a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.175+0000 7fe6fab08640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6f410d4a0 0x7fe6f41ad520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.175+0000 7fe6fab08640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f41ada60 con 0x7fe6f4108180 2024-03-11T19:15:48.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.175+0000 7fe6fab08640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f41adbd0 con 0x7fe6f410aaf0 2024-03-11T19:15:48.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.175+0000 7fe6fab08640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f41add90 con 0x7fe6f410d4a0 2024-03-11T19:15:48.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.175+0000 7fe6f88a4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6f4108180 0x7fe6f41a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.175+0000 7fe6f88a4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6f4108180 0x7fe6f41a5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54746/0 (socket says 172.21.15.29:54746) 2024-03-11T19:15:48.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.175+0000 7fe6f88a4640 1 -- 172.21.15.29:0/960350152 learned_addr learned my addr 172.21.15.29:0/960350152 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.175+0000 7fe6f90a5640 1 --2- 172.21.15.29:0/960350152 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6f410d4a0 0x7fe6f41ad520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.175+0000 7fe6f3fff640 1 --2- 172.21.15.29:0/960350152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6f410aaf0 0x7fe6f41a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.175+0000 7fe6f90a5640 1 -- 172.21.15.29:0/960350152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6f410aaf0 msgr2=0x7fe6f41a64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.176+0000 7fe6f90a5640 1 --2- 172.21.15.29:0/960350152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6f410aaf0 0x7fe6f41a64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.176+0000 7fe6f90a5640 1 -- 172.21.15.29:0/960350152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6f4108180 msgr2=0x7fe6f41a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.176+0000 7fe6f90a5640 1 --2- 172.21.15.29:0/960350152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6f4108180 0x7fe6f41a5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.176+0000 7fe6f90a5640 1 -- 172.21.15.29:0/960350152 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e0009490 con 0x7fe6f410d4a0 2024-03-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.176+0000 7fe6f90a5640 1 --2- 172.21.15.29:0/960350152 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6f410d4a0 0x7fe6f41ad520 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec032520 tx=0x7fe6ec032920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.176+0000 7fe6f1ffb640 1 -- 172.21.15.29:0/960350152 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec034650 con 0x7fe6f410d4a0 2024-03-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.176+0000 7fe6f1ffb640 1 -- 172.21.15.29:0/960350152 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe6ec003e20 con 0x7fe6f410d4a0 2024-03-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.176+0000 7fe6f1ffb640 1 -- 172.21.15.29:0/960350152 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec004840 con 0x7fe6f410d4a0 2024-03-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.176+0000 7fe6fab08640 1 -- 172.21.15.29:0/960350152 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6ec009900 con 0x7fe6f410d4a0 2024-03-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.176+0000 7fe6fab08640 1 -- 172.21.15.29:0/960350152 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f41ae2b0 con 0x7fe6f410d4a0 2024-03-11T19:15:48.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.180+0000 7fe6f1ffb640 1 -- 172.21.15.29:0/960350152 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe6ec03b040 con 0x7fe6f410d4a0 2024-03-11T19:15:48.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.180+0000 7fe6fd7b0740 1 Processor -- start 2024-03-11T19:15:48.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.180+0000 7fe6fd7b0740 1 -- start start 2024-03-11T19:15:48.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.180+0000 7fe6fd7b0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7de270 0x55cdbf7de690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.181+0000 7fe6fd7b0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7dec60 0x55cdbf7e3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.181+0000 7fe6fd7b0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7e3dd0 0x55cdbf7e6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.181+0000 7fe6fd7b0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdbf79a960 con 0x55cdbf7dec60 2024-03-11T19:15:48.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.181+0000 7fe6f1ffb640 1 --2- 172.21.15.29:0/960350152 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe6e40a1b00 0x7fe6e40a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.181+0000 7fe6fd7b0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdbf74e310 con 0x55cdbf7de270 2024-03-11T19:15:48.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.181+0000 7fe6fd7b0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdbf74b840 con 0x55cdbf7e3dd0 2024-03-11T19:15:48.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.181+0000 7fe6f3fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7de270 0x55cdbf7de690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.181+0000 7fe6f1ffb640 1 -- 172.21.15.29:0/960350152 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe6ec0efde0 con 0x7fe6f410d4a0 2024-03-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.181+0000 7fe6f88a4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7dec60 0x55cdbf7e3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.181+0000 7fe6f3fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7de270 0x55cdbf7de690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:38364/0 (socket says 172.21.15.29:38364) 2024-03-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.181+0000 7fe6f88a4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7dec60 0x55cdbf7e3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54760/0 (socket says 172.21.15.29:54760) 2024-03-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.181+0000 7fe6f3fff640 1 -- 172.21.15.29:0/370052596 learned_addr learned my addr 172.21.15.29:0/370052596 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.181+0000 7fe6f3fff640 1 --2- 172.21.15.29:0/370052596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7e3dd0 0x55cdbf7e6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.181+0000 7fe6f90a5640 1 --2- 172.21.15.29:0/960350152 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe6e40a1b00 0x7fe6e40a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.182+0000 7fe6f88a4640 1 -- 172.21.15.29:0/370052596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7e3dd0 msgr2=0x55cdbf7e6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.182+0000 7fe6f88a4640 1 --2- 172.21.15.29:0/370052596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7e3dd0 0x55cdbf7e6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.182+0000 7fe6f88a4640 1 -- 172.21.15.29:0/370052596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7de270 msgr2=0x55cdbf7de690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.182+0000 7fe6f88a4640 1 --2- 172.21.15.29:0/370052596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7de270 0x55cdbf7de690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.182+0000 7fe6f88a4640 1 -- 172.21.15.29:0/370052596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cdbf709060 con 0x55cdbf7dec60 2024-03-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.182+0000 7fe6f90a5640 1 --2- 172.21.15.29:0/960350152 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe6e40a1b00 0x7fe6e40a3fc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec0ed8e0 tx=0x7fe6ec003f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.182+0000 7fe6f3fff640 1 --2- 172.21.15.29:0/370052596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7e3dd0 0x55cdbf7e6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.182+0000 7fe6f88a4640 1 --2- 172.21.15.29:0/370052596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7dec60 0x55cdbf7e3760 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fe6e80058c0 tx=0x7fe6e80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7fee16b1d09b3da server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.182+0000 7fe6deffd640 1 -- 172.21.15.29:0/370052596 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8004230 con 0x55cdbf7dec60 2024-03-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.182+0000 7fe6deffd640 1 -- 172.21.15.29:0/370052596 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe6e8015070 con 0x55cdbf7dec60 2024-03-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.182+0000 7fe6deffd640 1 -- 172.21.15.29:0/370052596 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e80116a0 con 0x55cdbf7dec60 2024-03-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.183+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/370052596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7dec60 msgr2=0x55cdbf7e3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.183+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/370052596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7dec60 0x55cdbf7e3760 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fe6e80058c0 tx=0x7fe6e80107a0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.184+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/370052596 shutdown_connections 2024-03-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.184+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/370052596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7e3dd0 0x55cdbf7e6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.184+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/370052596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7dec60 0x55cdbf7e3760 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.184+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/370052596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7de270 0x55cdbf7de690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.184+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/370052596 >> 172.21.15.29:0/370052596 conn(0x55cdbf7d97e0 msgr2=0x55cdbf7d9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.184+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/370052596 shutdown_connections 2024-03-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.184+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/370052596 wait complete. 2024-03-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.184+0000 7fe6fd7b0740 1 -- start start 2024-03-11T19:15:48.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.185+0000 7fe6fd7b0740 1 Processor -- start 2024-03-11T19:15:48.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.185+0000 7fe6fd7b0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7de270 0x55cdbf7e8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.185+0000 7fe6fd7b0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7dec60 0x55cdbf7e8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.185+0000 7fe6fd7b0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7e3dd0 0x55cdbf7ef9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.185+0000 7fe6fd7b0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdbf747020 con 0x55cdbf7de270 2024-03-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.185+0000 7fe6fd7b0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdbf41a430 con 0x55cdbf7e3dd0 2024-03-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.185+0000 7fe6fd7b0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdbf3d6e00 con 0x55cdbf7dec60 2024-03-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.185+0000 7fe6fd7b0740 10 client.0 _ll_register_callbacks cb 0x55cdbf6eecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.185+0000 7fe6f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7de270 0x55cdbf7e8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.185+0000 7fe6f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7de270 0x55cdbf7e8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54766/0 (socket says 172.21.15.29:54766) 2024-03-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.185+0000 7fe6f3fff640 1 -- 172.21.15.29:0/540439719 learned_addr learned my addr 172.21.15.29:0/540439719 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.185+0000 7fe6f88a4640 1 --2- 172.21.15.29:0/540439719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7dec60 0x55cdbf7e8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.185+0000 7fe6f3fff640 1 --2- 172.21.15.29:0/540439719 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7e3dd0 0x55cdbf7ef9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.186+0000 7fe6f3fff640 1 -- 172.21.15.29:0/540439719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7dec60 msgr2=0x55cdbf7e8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.186+0000 7fe6f3fff640 1 --2- 172.21.15.29:0/540439719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7dec60 0x55cdbf7e8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.186+0000 7fe6f3fff640 1 -- 172.21.15.29:0/540439719 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7e3dd0 msgr2=0x55cdbf7ef9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.186+0000 7fe6f3fff640 1 --2- 172.21.15.29:0/540439719 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7e3dd0 0x55cdbf7ef9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.186+0000 7fe6f3fff640 1 -- 172.21.15.29:0/540439719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e800b970 con 0x55cdbf7de270 2024-03-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.186+0000 7fe6f88a4640 1 --2- 172.21.15.29:0/540439719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7dec60 0x55cdbf7e8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.186+0000 7fe6f3fff640 1 --2- 172.21.15.29:0/540439719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7de270 0x55cdbf7e8450 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0013f50 tx=0x7fe6e0009370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e48b032732ff90 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.186+0000 7fe6d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.187+0000 7fe6d3fff640 1 -- 172.21.15.29:0/540439719 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e00122e0 con 0x55cdbf7de270 2024-03-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.187+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/540439719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cdbf4467e0 con 0x55cdbf7de270 2024-03-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.187+0000 7fe6d3fff640 1 -- 172.21.15.29:0/540439719 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe6e0012e60 con 0x55cdbf7de270 2024-03-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.187+0000 7fe6d3fff640 1 -- 172.21.15.29:0/540439719 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6e0005020 con 0x55cdbf7de270 2024-03-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.187+0000 7fe6fd7b0740 10 client.16860 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.187+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/540439719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55cdbf7456a0 con 0x55cdbf7de270 2024-03-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.188+0000 7fe6d3fff640 1 -- 172.21.15.29:0/540439719 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55cdbf4467e0 con 0x55cdbf7de270 2024-03-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.188+0000 7fe6d3fff640 1 -- 172.21.15.29:0/540439719 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe6e0004b80 con 0x55cdbf7de270 2024-03-11T19:15:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.188+0000 7fe6fd7b0740 10 client.16860 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.188+0000 7fe6fd7b0740 10 client.16860 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:15:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.188+0000 7fe6fd7b0740 20 client.16860 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.188+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/540439719 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55cdbf7f46e0 0x55cdbf7f6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.188+0000 7fe6fd7b0740 4 client.16860 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:15:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.188+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/540439719 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cdbf483720 con 0x55cdbf7f46e0 2024-03-11T19:15:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.189+0000 7fe6f88a4640 1 --2- 172.21.15.29:0/540439719 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55cdbf7f46e0 0x55cdbf7f6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.190+0000 7fe6f88a4640 1 --2- 172.21.15.29:0/540439719 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55cdbf7f46e0 0x55cdbf7f6b00 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dffc4071d3e86d32 server_cookie=e6b8b777374aac32 in_seq=0 out_seq=0 2024-03-11T19:15:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.190+0000 7fe6d3fff640 10 client.16860 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:15:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.190+0000 7fe6d3fff640 1 -- 172.21.15.29:0/540439719 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cdbf483720 con 0x55cdbf7f46e0 2024-03-11T19:15:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.190+0000 7fe6d3fff640 10 client.16860 handle_command_reply: tid=0 2024-03-11T19:15:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.190+0000 7fe6fd7b0740 1 client.16860 shutdown 2024-03-11T19:15:48.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.191+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/540439719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7de270 msgr2=0x55cdbf7e8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.191+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/540439719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7de270 0x55cdbf7e8450 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0013f50 tx=0x7fe6e0009370 comp rx=0 tx=0).stop 2024-03-11T19:15:48.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.193+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/540439719 shutdown_connections 2024-03-11T19:15:48.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.193+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/540439719 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55cdbf7f46e0 0x55cdbf7f6b00 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.193+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/540439719 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7e3dd0 0x55cdbf7ef9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.193+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/540439719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7dec60 0x55cdbf7e8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.193+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/540439719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7de270 0x55cdbf7e8450 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.193+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/540439719 >> 172.21.15.29:0/540439719 conn(0x55cdbf7d97e0 msgr2=0x55cdbf7d8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:48.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.193+0000 7fe6d3fff640 0 client.16860 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:15:48.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.193+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/540439719 shutdown_connections 2024-03-11T19:15:48.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.193+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/540439719 wait complete. 2024-03-11T19:15:48.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.194+0000 7fe6fd7b0740 20 client.16860 trim_cache size 0 max 16384 2024-03-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.215+0000 7fe6fd7b0740 1 Processor -- start 2024-03-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.215+0000 7fe6fd7b0740 1 -- start start 2024-03-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.215+0000 7fe6fd7b0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7de270 0x55cdbf7d82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.215+0000 7fe6fd7b0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7dec60 0x55cdbf7ef990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.215+0000 7fe6fd7b0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7e3dd0 0x55cdbf7d8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.215+0000 7fe6fd7b0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdbf6cc130 con 0x55cdbf7dec60 2024-03-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.216+0000 7fe6fd7b0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdbf4467e0 con 0x55cdbf7de270 2024-03-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.216+0000 7fe6fd7b0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdbf7cba40 con 0x55cdbf7e3dd0 2024-03-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.216+0000 7fe6f3fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7de270 0x55cdbf7d82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.216+0000 7fe6f88a4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7dec60 0x55cdbf7ef990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.216+0000 7fe6f88a4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7dec60 0x55cdbf7ef990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54768/0 (socket says 172.21.15.29:54768) 2024-03-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.216+0000 7fe6f88a4640 1 -- 172.21.15.29:0/2628437552 learned_addr learned my addr 172.21.15.29:0/2628437552 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.216+0000 7fe6f3fff640 1 --2- 172.21.15.29:0/2628437552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7e3dd0 0x55cdbf7d8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.217+0000 7fe6f88a4640 1 -- 172.21.15.29:0/2628437552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7e3dd0 msgr2=0x55cdbf7d8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.217+0000 7fe6f88a4640 1 --2- 172.21.15.29:0/2628437552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7e3dd0 0x55cdbf7d8830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.217+0000 7fe6f88a4640 1 -- 172.21.15.29:0/2628437552 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7de270 msgr2=0x55cdbf7d82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.217+0000 7fe6f88a4640 1 --2- 172.21.15.29:0/2628437552 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7de270 0x55cdbf7d82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.217+0000 7fe6f88a4640 1 -- 172.21.15.29:0/2628437552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cdbf6e1c20 con 0x55cdbf7dec60 2024-03-11T19:15:48.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.217+0000 7fe6f88a4640 1 --2- 172.21.15.29:0/2628437552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7dec60 0x55cdbf7ef990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe6e80058c0 tx=0x7fe6e8013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe78c66cdc76f9ab server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:48.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.218+0000 7fe6de7fc640 1 -- 172.21.15.29:0/2628437552 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8022070 con 0x55cdbf7dec60 2024-03-11T19:15:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.218+0000 7fe6de7fc640 1 -- 172.21.15.29:0/2628437552 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe6e8023040 con 0x55cdbf7dec60 2024-03-11T19:15:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.218+0000 7fe6de7fc640 1 -- 172.21.15.29:0/2628437552 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e80039f0 con 0x55cdbf7dec60 2024-03-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.220+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/2628437552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7dec60 msgr2=0x55cdbf7ef990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.220+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/2628437552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7dec60 0x55cdbf7ef990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe6e80058c0 tx=0x7fe6e8013040 comp rx=0 tx=0).stop 2024-03-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.220+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/2628437552 shutdown_connections 2024-03-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.220+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/2628437552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7e3dd0 0x55cdbf7d8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.220+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/2628437552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7dec60 0x55cdbf7ef990 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.220+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/2628437552 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7de270 0x55cdbf7d82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.220+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/2628437552 >> 172.21.15.29:0/2628437552 conn(0x55cdbf7d97e0 msgr2=0x55cdbf7e01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.220+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/2628437552 shutdown_connections 2024-03-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.220+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/2628437552 wait complete. 2024-03-11T19:15:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.221+0000 7fe6fd7b0740 1 -- start start 2024-03-11T19:15:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.221+0000 7fe6fd7b0740 1 Processor -- start 2024-03-11T19:15:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.222+0000 7fe6fd7b0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7de270 0x55cdbf7ef350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.222+0000 7fe6fd7b0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7dec60 0x55cdbf7ef890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.222+0000 7fe6fd7b0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7e3dd0 0x55cdbf7ed960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.222+0000 7fe6fd7b0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdbf7ee060 con 0x55cdbf7e3dd0 2024-03-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.222+0000 7fe6fd7b0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdbf7ee1d0 con 0x55cdbf7dec60 2024-03-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.222+0000 7fe6fd7b0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdbf7ee390 con 0x55cdbf7de270 2024-03-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.222+0000 7fe6fd7b0740 10 client.0 _ll_register_callbacks cb 0x7fe6e800b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.222+0000 7fe6f3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7de270 0x55cdbf7ef350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.222+0000 7fe6f3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7de270 0x55cdbf7ef350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49174/0 (socket says 172.21.15.29:49174) 2024-03-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.222+0000 7fe6f3fff640 1 -- 172.21.15.29:0/2226566578 learned_addr learned my addr 172.21.15.29:0/2226566578 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.222+0000 7fe6f88a4640 1 --2- 172.21.15.29:0/2226566578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7dec60 0x55cdbf7ef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.222+0000 7fe6f3fff640 1 --2- 172.21.15.29:0/2226566578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7e3dd0 0x55cdbf7ed960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.223+0000 7fe6f88a4640 1 -- 172.21.15.29:0/2226566578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7de270 msgr2=0x55cdbf7ef350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.223+0000 7fe6f88a4640 1 --2- 172.21.15.29:0/2226566578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7de270 0x55cdbf7ef350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.223+0000 7fe6f88a4640 1 -- 172.21.15.29:0/2226566578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7e3dd0 msgr2=0x55cdbf7ed960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.223+0000 7fe6f88a4640 1 --2- 172.21.15.29:0/2226566578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7e3dd0 0x55cdbf7ed960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.223+0000 7fe6f88a4640 1 -- 172.21.15.29:0/2226566578 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cdbf709060 con 0x55cdbf7dec60 2024-03-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.223+0000 7fe6f3fff640 1 --2- 172.21.15.29:0/2226566578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7e3dd0 0x55cdbf7ed960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.223+0000 7fe6f88a4640 1 --2- 172.21.15.29:0/2226566578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7dec60 0x55cdbf7ef890 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55cdbf7d9770 tx=0x7fe6e80107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62fbe95e853e7e98 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.223+0000 7fe6d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.223+0000 7fe6d3fff640 1 -- 172.21.15.29:0/2226566578 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8022050 con 0x55cdbf7dec60 2024-03-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.224+0000 7fe6d3fff640 1 -- 172.21.15.29:0/2226566578 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe6e8017070 con 0x55cdbf7dec60 2024-03-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.224+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/2226566578 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe6e8015070 con 0x55cdbf7dec60 2024-03-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.224+0000 7fe6d3fff640 1 -- 172.21.15.29:0/2226566578 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6e80052c0 con 0x55cdbf7dec60 2024-03-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.224+0000 7fe6fd7b0740 10 client.26375 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.224+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/2226566578 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55cdbf7078c0 con 0x55cdbf7dec60 2024-03-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.224+0000 7fe6d3fff640 1 -- 172.21.15.29:0/2226566578 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe6e8005420 con 0x55cdbf7dec60 2024-03-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.225+0000 7fe6d3fff640 1 -- 172.21.15.29:0/2226566578 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe6e801b7a0 con 0x55cdbf7dec60 2024-03-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.225+0000 7fe6fd7b0740 10 client.26375 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.225+0000 7fe6fd7b0740 10 client.26375 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:15:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.225+0000 7fe6fd7b0740 20 client.26375 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.225+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/2226566578 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55cdbf84bca0 0x55cdbf84e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.225+0000 7fe6fd7b0740 4 client.26375 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:15:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.225+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/2226566578 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7fe6e8015070 con 0x55cdbf84bca0 2024-03-11T19:15:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.225+0000 7fe6f3fff640 1 --2- 172.21.15.29:0/2226566578 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55cdbf84bca0 0x55cdbf84e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.226+0000 7fe6f3fff640 1 --2- 172.21.15.29:0/2226566578 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55cdbf84bca0 0x55cdbf84e0c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=148ced8a0e86e3 server_cookie=76304bc960bfc639 in_seq=0 out_seq=0 2024-03-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.226+0000 7fe6d3fff640 10 client.26375 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.227+0000 7fe6d3fff640 1 -- 172.21.15.29:0/2226566578 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fe6e8015070 con 0x55cdbf84bca0 2024-03-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.227+0000 7fe6d3fff640 10 client.26375 handle_command_reply: tid=0 2024-03-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.227+0000 7fe6fd7b0740 1 client.26375 shutdown 2024-03-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.227+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/2226566578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7dec60 msgr2=0x55cdbf7ef890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.227+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/2226566578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7dec60 0x55cdbf7ef890 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55cdbf7d9770 tx=0x7fe6e80107a0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.228+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/2226566578 shutdown_connections 2024-03-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.228+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/2226566578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cdbf7e3dd0 0x55cdbf7ed960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.228+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/2226566578 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55cdbf84bca0 0x55cdbf84e0c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.228+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/2226566578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cdbf7dec60 0x55cdbf7ef890 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.228+0000 7fe6d3fff640 0 client.26375 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.228+0000 7fe6fd7b0740 1 --2- 172.21.15.29:0/2226566578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cdbf7de270 0x55cdbf7ef350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.228+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/2226566578 >> 172.21.15.29:0/2226566578 conn(0x55cdbf7d97e0 msgr2=0x55cdbf7dca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.229+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/2226566578 shutdown_connections 2024-03-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.229+0000 7fe6fd7b0740 1 -- 172.21.15.29:0/2226566578 wait complete. 2024-03-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.229+0000 7fe6fd7b0740 20 client.26375 trim_cache size 0 max 16384 2024-03-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:15:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.230+0000 7fe6fab08640 1 -- 172.21.15.29:0/960350152 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe6e40a1b00 msgr2=0x7fe6e40a3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.230+0000 7fe6fab08640 1 --2- 172.21.15.29:0/960350152 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe6e40a1b00 0x7fe6e40a3fc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec0ed8e0 tx=0x7fe6ec003f90 comp rx=0 tx=0).stop 2024-03-11T19:15:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.230+0000 7fe6fab08640 1 -- 172.21.15.29:0/960350152 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6f410d4a0 msgr2=0x7fe6f41ad520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.230+0000 7fe6fab08640 1 --2- 172.21.15.29:0/960350152 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6f410d4a0 0x7fe6f41ad520 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec032520 tx=0x7fe6ec032920 comp rx=0 tx=0).stop 2024-03-11T19:15:48.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.230+0000 7fe6fab08640 1 -- 172.21.15.29:0/960350152 shutdown_connections 2024-03-11T19:15:48.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.230+0000 7fe6fab08640 1 --2- 172.21.15.29:0/960350152 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6f410d4a0 0x7fe6f41ad520 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.230+0000 7fe6fab08640 1 --2- 172.21.15.29:0/960350152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6f410aaf0 0x7fe6f41a64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.230+0000 7fe6fab08640 1 --2- 172.21.15.29:0/960350152 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe6e40a1b00 0x7fe6e40a3fc0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.230+0000 7fe6fab08640 1 --2- 172.21.15.29:0/960350152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6f4108180 0x7fe6f41a5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.230+0000 7fe6fab08640 1 -- 172.21.15.29:0/960350152 >> 172.21.15.29:0/960350152 conn(0x7fe6f406dfe0 msgr2=0x7fe6f41085a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:48.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.230+0000 7fe6fab08640 1 -- 172.21.15.29:0/960350152 shutdown_connections 2024-03-11T19:15:48.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.230+0000 7fe6fab08640 1 -- 172.21.15.29:0/960350152 wait complete. 2024-03-11T19:15:48.252 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:15:48.252 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:15:48.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:47 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/561107891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:15:48.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:47 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/561107891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:15:48.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.425+0000 7f9def264640 1 -- 172.21.15.29:0/1422188090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9de81077a0 msgr2=0x7f9de8107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.425+0000 7f9def264640 1 --2- 172.21.15.29:0/1422188090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9de81077a0 0x7f9de8107bc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4009900 tx=0x7f9dd402bc30 comp rx=0 tx=0).stop 2024-03-11T19:15:48.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.426+0000 7f9def264640 1 -- 172.21.15.29:0/1422188090 shutdown_connections 2024-03-11T19:15:48.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.426+0000 7f9def264640 1 --2- 172.21.15.29:0/1422188090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9de8109350 0x7f9de810fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.426+0000 7f9def264640 1 --2- 172.21.15.29:0/1422188090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de8108990 0x7f9de8108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.426+0000 7f9def264640 1 --2- 172.21.15.29:0/1422188090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9de81077a0 0x7f9de8107bc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.426+0000 7f9def264640 1 -- 172.21.15.29:0/1422188090 >> 172.21.15.29:0/1422188090 conn(0x7f9de8076740 msgr2=0x7f9de8078b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:48.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.426+0000 7f9def264640 1 -- 172.21.15.29:0/1422188090 shutdown_connections 2024-03-11T19:15:48.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.426+0000 7f9def264640 1 -- 172.21.15.29:0/1422188090 wait complete. 2024-03-11T19:15:48.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.427+0000 7f9def264640 1 Processor -- start 2024-03-11T19:15:48.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.427+0000 7f9def264640 1 -- start start 2024-03-11T19:15:48.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.427+0000 7f9def264640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de81077a0 0x7f9de819d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.427+0000 7f9def264640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9de8108990 0x7f9de819dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.427+0000 7f9def264640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9de8109350 0x7f9de81a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.427+0000 7f9def264640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de81a5270 con 0x7f9de81077a0 2024-03-11T19:15:48.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.427+0000 7f9def264640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de81a53e0 con 0x7f9de8109350 2024-03-11T19:15:48.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.427+0000 7f9def264640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de81a55a0 con 0x7f9de8108990 2024-03-11T19:15:48.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.427+0000 7f9ded000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de81077a0 0x7f9de819d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.427+0000 7f9ded000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de81077a0 0x7f9de819d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54806/0 (socket says 172.21.15.29:54806) 2024-03-11T19:15:48.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.427+0000 7f9ded000640 1 -- 172.21.15.29:0/833039844 learned_addr learned my addr 172.21.15.29:0/833039844 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:48.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.428+0000 7f9de7fff640 1 --2- 172.21.15.29:0/833039844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9de8108990 0x7f9de819dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.428+0000 7f9ded801640 1 --2- 172.21.15.29:0/833039844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9de8109350 0x7f9de81a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.428+0000 7f9ded801640 1 -- 172.21.15.29:0/833039844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9de8108990 msgr2=0x7f9de819dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.428+0000 7f9ded801640 1 --2- 172.21.15.29:0/833039844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9de8108990 0x7f9de819dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.428+0000 7f9ded801640 1 -- 172.21.15.29:0/833039844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de81077a0 msgr2=0x7f9de819d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.428+0000 7f9ded801640 1 --2- 172.21.15.29:0/833039844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de81077a0 0x7f9de819d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.428+0000 7f9ded801640 1 -- 172.21.15.29:0/833039844 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ddc009490 con 0x7f9de8109350 2024-03-11T19:15:48.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.428+0000 7f9ded801640 1 --2- 172.21.15.29:0/833039844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9de8109350 0x7f9de81a4d30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9de0009fd0 tx=0x7f9de000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:48.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.428+0000 7f9de5ffb640 1 -- 172.21.15.29:0/833039844 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de000ce20 con 0x7f9de8109350 2024-03-11T19:15:48.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.429+0000 7f9de5ffb640 1 -- 172.21.15.29:0/833039844 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9de0005020 con 0x7f9de8109350 2024-03-11T19:15:48.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.429+0000 7f9def264640 1 -- 172.21.15.29:0/833039844 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd4009560 con 0x7f9de8109350 2024-03-11T19:15:48.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.429+0000 7f9de5ffb640 1 -- 172.21.15.29:0/833039844 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de0010690 con 0x7f9de8109350 2024-03-11T19:15:48.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.429+0000 7f9def264640 1 -- 172.21.15.29:0/833039844 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de81a5c50 con 0x7f9de8109350 2024-03-11T19:15:48.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.432+0000 7f9de5ffb640 1 -- 172.21.15.29:0/833039844 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9de0005190 con 0x7f9de8109350 2024-03-11T19:15:48.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.432+0000 7f9df1e38740 1 Processor -- start 2024-03-11T19:15:48.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.432+0000 7f9df1e38740 1 -- start start 2024-03-11T19:15:48.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.432+0000 7f9df1e38740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17270 0x55cb36f17690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.432+0000 7f9df1e38740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17c60 0x55cb36f1c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.432+0000 7f9df1e38740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f1cdd0 0x55cb36f1f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.432+0000 7f9df1e38740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb36ed3960 con 0x55cb36f17c60 2024-03-11T19:15:48.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.432+0000 7f9df1e38740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb36e87310 con 0x55cb36f1cdd0 2024-03-11T19:15:48.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.432+0000 7f9df1e38740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb36e84840 con 0x55cb36f17270 2024-03-11T19:15:48.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.433+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17270 0x55cb36f17690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.433+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17270 0x55cb36f17690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49196/0 (socket says 172.21.15.29:49196) 2024-03-11T19:15:48.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.433+0000 7f9de7fff640 1 -- 172.21.15.29:0/808514531 learned_addr learned my addr 172.21.15.29:0/808514531 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:48.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.433+0000 7f9de7fff640 1 --2- 172.21.15.29:0/808514531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f1cdd0 0x55cb36f1f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.433+0000 7f9de7fff640 1 -- 172.21.15.29:0/808514531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f1cdd0 msgr2=0x55cb36f1f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.433+0000 7f9de7fff640 1 --2- 172.21.15.29:0/808514531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f1cdd0 0x55cb36f1f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.433+0000 7f9de7fff640 1 -- 172.21.15.29:0/808514531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17c60 msgr2=0x55cb36f1c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.433+0000 7f9ded000640 1 --2- 172.21.15.29:0/808514531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17c60 0x55cb36f1c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.433+0000 7f9de7fff640 1 --2- 172.21.15.29:0/808514531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17c60 0x55cb36f1c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.433+0000 7f9de7fff640 1 -- 172.21.15.29:0/808514531 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cb36e42060 con 0x55cb36f17270 2024-03-11T19:15:48.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.433+0000 7f9de7fff640 1 --2- 172.21.15.29:0/808514531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17270 0x55cb36f17690 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc011990 tx=0x7f9ddc009370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90a44eda63dafa92 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:48.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.433+0000 7f9de5ffb640 1 --2- 172.21.15.29:0/833039844 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9dd80a1b30 0x7f9dd80a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.434+0000 7f9dd2ffd640 1 -- 172.21.15.29:0/808514531 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc011220 con 0x55cb36f17270 2024-03-11T19:15:48.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.434+0000 7f9dd2ffd640 1 -- 172.21.15.29:0/808514531 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9ddc00f7c0 con 0x55cb36f17270 2024-03-11T19:15:48.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.434+0000 7f9dd2ffd640 1 -- 172.21.15.29:0/808514531 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc003830 con 0x55cb36f17270 2024-03-11T19:15:48.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.434+0000 7f9de5ffb640 1 -- 172.21.15.29:0/833039844 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9de0014070 con 0x7f9de8109350 2024-03-11T19:15:48.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.434+0000 7f9ded000640 1 --2- 172.21.15.29:0/808514531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17c60 0x55cb36f1c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:15:48.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.435+0000 7f9ded000640 1 --2- 172.21.15.29:0/833039844 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9dd80a1b30 0x7f9dd80a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.435+0000 7f9df1e38740 1 -- 172.21.15.29:0/808514531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17270 msgr2=0x55cb36f17690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.435+0000 7f9df1e38740 1 --2- 172.21.15.29:0/808514531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17270 0x55cb36f17690 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc011990 tx=0x7f9ddc009370 comp rx=0 tx=0).stop 2024-03-11T19:15:48.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.436+0000 7f9df1e38740 1 -- 172.21.15.29:0/808514531 shutdown_connections 2024-03-11T19:15:48.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.436+0000 7f9df1e38740 1 --2- 172.21.15.29:0/808514531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f1cdd0 0x55cb36f1f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.436+0000 7f9df1e38740 1 --2- 172.21.15.29:0/808514531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17c60 0x55cb36f1c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.436+0000 7f9df1e38740 1 --2- 172.21.15.29:0/808514531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17270 0x55cb36f17690 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.436+0000 7f9df1e38740 1 -- 172.21.15.29:0/808514531 >> 172.21.15.29:0/808514531 conn(0x55cb36f127e0 msgr2=0x55cb36f12be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:48.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.436+0000 7f9ded000640 1 --2- 172.21.15.29:0/833039844 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9dd80a1b30 0x7f9dd80a3ff0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x55cb36f18060 tx=0x7f9dd402c770 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:48.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.436+0000 7f9df1e38740 1 -- 172.21.15.29:0/808514531 shutdown_connections 2024-03-11T19:15:48.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.436+0000 7f9df1e38740 1 -- 172.21.15.29:0/808514531 wait complete. 2024-03-11T19:15:48.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.436+0000 7f9df1e38740 1 -- start start 2024-03-11T19:15:48.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.437+0000 7f9df1e38740 1 Processor -- start 2024-03-11T19:15:48.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.438+0000 7f9df1e38740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f17270 0x55cb36f21510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.438+0000 7f9df1e38740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17c60 0x55cb36f21a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.438+0000 7f9ded000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f17270 0x55cb36f21510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.438+0000 7f9ded000640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f17270 0x55cb36f21510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:38428/0 (socket says 172.21.15.29:38428) 2024-03-11T19:15:48.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.438+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17c60 0x55cb36f21a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.438+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17c60 0x55cb36f21a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54830/0 (socket says 172.21.15.29:54830) 2024-03-11T19:15:48.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.438+0000 7f9ded000640 1 -- 172.21.15.29:0/2447203569 learned_addr learned my addr 172.21.15.29:0/2447203569 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:48.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.439+0000 7f9df1e38740 1 --2- 172.21.15.29:0/2447203569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f1cdd0 0x55cb36f28a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.439+0000 7f9df1e38740 1 -- 172.21.15.29:0/2447203569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb36e80020 con 0x55cb36f17c60 2024-03-11T19:15:48.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.439+0000 7f9df1e38740 1 -- 172.21.15.29:0/2447203569 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb36b53430 con 0x55cb36f17270 2024-03-11T19:15:48.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.439+0000 7f9ded801640 1 --2- 172.21.15.29:0/2447203569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f1cdd0 0x55cb36f28a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.439+0000 7f9df1e38740 1 -- 172.21.15.29:0/2447203569 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb36b0fe00 con 0x55cb36f1cdd0 2024-03-11T19:15:48.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.439+0000 7f9df1e38740 10 client.0 _ll_register_callbacks cb 0x55cb36e27cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:48.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.440+0000 7f9de7fff640 1 -- 172.21.15.29:0/2447203569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f1cdd0 msgr2=0x55cb36f28a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.440+0000 7f9de7fff640 1 --2- 172.21.15.29:0/2447203569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f1cdd0 0x55cb36f28a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.440+0000 7f9de7fff640 1 -- 172.21.15.29:0/2447203569 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f17270 msgr2=0x55cb36f21510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.440+0000 7f9de7fff640 1 --2- 172.21.15.29:0/2447203569 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f17270 0x55cb36f21510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.440+0000 7f9de7fff640 1 -- 172.21.15.29:0/2447203569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ddc019040 con 0x55cb36f17c60 2024-03-11T19:15:48.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.440+0000 7f9de7fff640 1 --2- 172.21.15.29:0/2447203569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17c60 0x55cb36f21a50 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc009870 tx=0x7f9ddc009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c66599d3571ef39 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:48.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.440+0000 7f9dc7fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:15:48.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.440+0000 7f9dc7fff640 1 -- 172.21.15.29:0/2447203569 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc037070 con 0x55cb36f17c60 2024-03-11T19:15:48.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.440+0000 7f9dc7fff640 1 -- 172.21.15.29:0/2447203569 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9ddc00f930 con 0x55cb36f17c60 2024-03-11T19:15:48.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.440+0000 7f9df1e38740 1 -- 172.21.15.29:0/2447203569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cb36b7f7e0 con 0x55cb36f17c60 2024-03-11T19:15:48.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.441+0000 7f9dc7fff640 1 -- 172.21.15.29:0/2447203569 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ddc004400 con 0x55cb36f17c60 2024-03-11T19:15:48.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.441+0000 7f9df1e38740 10 client.16875 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:48.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.441+0000 7f9df1e38740 1 -- 172.21.15.29:0/2447203569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55cb36e7e6a0 con 0x55cb36f17c60 2024-03-11T19:15:48.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.442+0000 7f9dc7fff640 1 -- 172.21.15.29:0/2447203569 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55cb36b7f7e0 con 0x55cb36f17c60 2024-03-11T19:15:48.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.443+0000 7f9dc7fff640 1 -- 172.21.15.29:0/2447203569 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9ddc004970 con 0x55cb36f17c60 2024-03-11T19:15:48.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.443+0000 7f9df1e38740 10 client.16875 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:48.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.443+0000 7f9df1e38740 10 client.16875 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:15:48.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.443+0000 7f9df1e38740 20 client.16875 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:48.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.443+0000 7f9df1e38740 1 --2- 172.21.15.29:0/2447203569 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cb36f2d480 0x55cb36f2f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.443+0000 7f9df1e38740 4 client.16875 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:15:48.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.443+0000 7f9df1e38740 1 -- 172.21.15.29:0/2447203569 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cb36bbc720 con 0x55cb36f2d480 2024-03-11T19:15:48.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.443+0000 7f9ded000640 1 --2- 172.21.15.29:0/2447203569 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cb36f2d480 0x55cb36f2f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.444+0000 7f9ded000640 1 --2- 172.21.15.29:0/2447203569 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cb36f2d480 0x55cb36f2f8a0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=466089653e37b4fb server_cookie=6bf67482d2de6ddb in_seq=0 out_seq=0 2024-03-11T19:15:48.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.444+0000 7f9dc7fff640 10 client.16875 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:48.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.445+0000 7f9dc7fff640 1 -- 172.21.15.29:0/2447203569 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cb36bbc720 con 0x55cb36f2d480 2024-03-11T19:15:48.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.445+0000 7f9dc7fff640 10 client.16875 handle_command_reply: tid=0 2024-03-11T19:15:48.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.445+0000 7f9df1e38740 1 client.16875 shutdown 2024-03-11T19:15:48.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.445+0000 7f9df1e38740 1 -- 172.21.15.29:0/2447203569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17c60 msgr2=0x55cb36f21a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.445+0000 7f9df1e38740 1 --2- 172.21.15.29:0/2447203569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17c60 0x55cb36f21a50 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc009870 tx=0x7f9ddc009e00 comp rx=0 tx=0).stop 2024-03-11T19:15:48.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.445+0000 7f9df1e38740 1 -- 172.21.15.29:0/2447203569 shutdown_connections 2024-03-11T19:15:48.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.445+0000 7f9df1e38740 1 --2- 172.21.15.29:0/2447203569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f1cdd0 0x55cb36f28a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.445+0000 7f9df1e38740 1 --2- 172.21.15.29:0/2447203569 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cb36f2d480 0x55cb36f2f8a0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.445+0000 7f9df1e38740 1 --2- 172.21.15.29:0/2447203569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17c60 0x55cb36f21a50 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.445+0000 7f9df1e38740 1 --2- 172.21.15.29:0/2447203569 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f17270 0x55cb36f21510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.445+0000 7f9df1e38740 1 -- 172.21.15.29:0/2447203569 >> 172.21.15.29:0/2447203569 conn(0x55cb36f127e0 msgr2=0x55cb36f11b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:48.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.446+0000 7f9dc7fff640 0 client.16875 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:48.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.446+0000 7f9df1e38740 1 -- 172.21.15.29:0/2447203569 shutdown_connections 2024-03-11T19:15:48.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.446+0000 7f9df1e38740 1 -- 172.21.15.29:0/2447203569 wait complete. 2024-03-11T19:15:48.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.446+0000 7f9df1e38740 20 client.16875 trim_cache size 0 max 16384 2024-03-11T19:15:48.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.467+0000 7f9df1e38740 1 Processor -- start 2024-03-11T19:15:48.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.467+0000 7f9df1e38740 1 -- start start 2024-03-11T19:15:48.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.468+0000 7f9df1e38740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17270 0x55cb36f341c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.468+0000 7f9df1e38740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17c60 0x55cb36f28a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.468+0000 7f9df1e38740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f1cdd0 0x55cb36f21510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.468+0000 7f9df1e38740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb36e05130 con 0x55cb36f17270 2024-03-11T19:15:48.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.468+0000 7f9df1e38740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb36f20630 con 0x55cb36f1cdd0 2024-03-11T19:15:48.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.468+0000 7f9df1e38740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb36bbc720 con 0x55cb36f17c60 2024-03-11T19:15:48.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.468+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17c60 0x55cb36f28a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.468+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17c60 0x55cb36f28a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49208/0 (socket says 172.21.15.29:49208) 2024-03-11T19:15:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.468+0000 7f9de7fff640 1 -- 172.21.15.29:0/1401460115 learned_addr learned my addr 172.21.15.29:0/1401460115 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.468+0000 7f9ded000640 1 --2- 172.21.15.29:0/1401460115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17270 0x55cb36f341c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.468+0000 7f9ded801640 1 --2- 172.21.15.29:0/1401460115 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f1cdd0 0x55cb36f21510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.469+0000 7f9de7fff640 1 -- 172.21.15.29:0/1401460115 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f1cdd0 msgr2=0x55cb36f21510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.469+0000 7f9de7fff640 1 --2- 172.21.15.29:0/1401460115 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f1cdd0 0x55cb36f21510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.469+0000 7f9de7fff640 1 -- 172.21.15.29:0/1401460115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17270 msgr2=0x55cb36f341c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.469+0000 7f9de7fff640 1 --2- 172.21.15.29:0/1401460115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17270 0x55cb36f341c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.469+0000 7f9de7fff640 1 -- 172.21.15.29:0/1401460115 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cb36e1ac20 con 0x55cb36f17c60 2024-03-11T19:15:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.469+0000 7f9ded801640 1 --2- 172.21.15.29:0/1401460115 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f1cdd0 0x55cb36f21510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:15:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.469+0000 7f9ded000640 1 --2- 172.21.15.29:0/1401460115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17270 0x55cb36f341c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:15:48.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.469+0000 7f9de7fff640 1 --2- 172.21.15.29:0/1401460115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17c60 0x55cb36f28a50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc03d040 tx=0x7f9ddc005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3b353b455aea742 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:48.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.469+0000 7f9dd27fc640 1 -- 172.21.15.29:0/1401460115 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc037070 con 0x55cb36f17c60 2024-03-11T19:15:48.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.469+0000 7f9dd27fc640 1 -- 172.21.15.29:0/1401460115 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9ddc038040 con 0x55cb36f17c60 2024-03-11T19:15:48.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.469+0000 7f9dd27fc640 1 -- 172.21.15.29:0/1401460115 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc003870 con 0x55cb36f17c60 2024-03-11T19:15:48.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.470+0000 7f9df1e38740 1 -- 172.21.15.29:0/1401460115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17c60 msgr2=0x55cb36f28a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.470+0000 7f9df1e38740 1 --2- 172.21.15.29:0/1401460115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17c60 0x55cb36f28a50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc03d040 tx=0x7f9ddc005850 comp rx=0 tx=0).stop 2024-03-11T19:15:48.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.470+0000 7f9df1e38740 1 -- 172.21.15.29:0/1401460115 shutdown_connections 2024-03-11T19:15:48.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.470+0000 7f9df1e38740 1 --2- 172.21.15.29:0/1401460115 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f1cdd0 0x55cb36f21510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.470+0000 7f9df1e38740 1 --2- 172.21.15.29:0/1401460115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17c60 0x55cb36f28a50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.470+0000 7f9df1e38740 1 --2- 172.21.15.29:0/1401460115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17270 0x55cb36f341c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.470+0000 7f9df1e38740 1 -- 172.21.15.29:0/1401460115 >> 172.21.15.29:0/1401460115 conn(0x55cb36f127e0 msgr2=0x55cb36f20b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:48.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.470+0000 7f9df1e38740 1 -- 172.21.15.29:0/1401460115 shutdown_connections 2024-03-11T19:15:48.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.470+0000 7f9df1e38740 1 -- 172.21.15.29:0/1401460115 wait complete. 2024-03-11T19:15:48.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.471+0000 7f9df1e38740 1 -- start start 2024-03-11T19:15:48.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.471+0000 7f9df1e38740 1 Processor -- start 2024-03-11T19:15:48.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.471+0000 7f9df1e38740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17270 0x55cb36f26360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.471+0000 7f9df1e38740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17c60 0x55cb36f268a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.471+0000 7f9df1e38740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f1cdd0 0x55cb36f27840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.471+0000 7f9ded000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17270 0x55cb36f26360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.471+0000 7f9ded000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17270 0x55cb36f26360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54844/0 (socket says 172.21.15.29:54844) 2024-03-11T19:15:48.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.471+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17c60 0x55cb36f268a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.471+0000 7f9ded000640 1 -- 172.21.15.29:0/2538347360 learned_addr learned my addr 172.21.15.29:0/2538347360 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.471+0000 7f9df1e38740 1 -- 172.21.15.29:0/2538347360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb36f27d80 con 0x55cb36f17270 2024-03-11T19:15:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.471+0000 7f9df1e38740 1 -- 172.21.15.29:0/2538347360 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb36f27ef0 con 0x55cb36f1cdd0 2024-03-11T19:15:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.471+0000 7f9df1e38740 1 -- 172.21.15.29:0/2538347360 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb36f280b0 con 0x55cb36f17c60 2024-03-11T19:15:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.472+0000 7f9ded801640 1 --2- 172.21.15.29:0/2538347360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f1cdd0 0x55cb36f27840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.472+0000 7f9df1e38740 10 client.0 _ll_register_callbacks cb 0x7f9ddc009c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.472+0000 7f9de7fff640 1 -- 172.21.15.29:0/2538347360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f1cdd0 msgr2=0x55cb36f27840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.472+0000 7f9de7fff640 1 --2- 172.21.15.29:0/2538347360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f1cdd0 0x55cb36f27840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.472+0000 7f9de7fff640 1 -- 172.21.15.29:0/2538347360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17270 msgr2=0x55cb36f26360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.472+0000 7f9de7fff640 1 --2- 172.21.15.29:0/2538347360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17270 0x55cb36f26360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.472+0000 7f9de7fff640 1 -- 172.21.15.29:0/2538347360 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cb36e7e6a0 con 0x55cb36f17c60 2024-03-11T19:15:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.472+0000 7f9de7fff640 1 --2- 172.21.15.29:0/2538347360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17c60 0x55cb36f268a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc0059f0 tx=0x7f9ddc004840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b694a7ba58cfb3e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.472+0000 7f9dc7fff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:15:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.472+0000 7f9dc7fff640 1 -- 172.21.15.29:0/2538347360 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc037040 con 0x55cb36f17c60 2024-03-11T19:15:48.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.472+0000 7f9dc7fff640 1 -- 172.21.15.29:0/2538347360 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9ddc003640 con 0x55cb36f17c60 2024-03-11T19:15:48.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.472+0000 7f9df1e38740 1 -- 172.21.15.29:0/2538347360 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9ddc00f7c0 con 0x55cb36f17c60 2024-03-11T19:15:48.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.473+0000 7f9dc7fff640 1 -- 172.21.15.29:0/2538347360 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ddc009cf0 con 0x55cb36f17c60 2024-03-11T19:15:48.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.473+0000 7f9df1e38740 10 client.26401 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:48.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.473+0000 7f9df1e38740 1 -- 172.21.15.29:0/2538347360 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55cb36e408c0 con 0x55cb36f17c60 2024-03-11T19:15:48.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.473+0000 7f9dc7fff640 1 -- 172.21.15.29:0/2538347360 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9ddc00f7c0 con 0x55cb36f17c60 2024-03-11T19:15:48.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.473+0000 7f9dc7fff640 1 -- 172.21.15.29:0/2538347360 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9ddc004e50 con 0x55cb36f17c60 2024-03-11T19:15:48.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.473+0000 7f9df1e38740 10 client.26401 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:48.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.473+0000 7f9df1e38740 10 client.26401 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:15:48.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.473+0000 7f9df1e38740 20 client.26401 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:48.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.473+0000 7f9df1e38740 1 --2- 172.21.15.29:0/2538347360 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cb36f83ff0 0x55cb36f86410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:48.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.473+0000 7f9df1e38740 4 client.26401 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:15:48.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.473+0000 7f9df1e38740 1 -- 172.21.15.29:0/2538347360 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cb36c0b890 con 0x55cb36f83ff0 2024-03-11T19:15:48.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.474+0000 7f9ded000640 1 --2- 172.21.15.29:0/2538347360 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cb36f83ff0 0x55cb36f86410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:48.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.474+0000 7f9dc7fff640 10 client.26401 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:48.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.474+0000 7f9ded000640 1 --2- 172.21.15.29:0/2538347360 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cb36f83ff0 0x55cb36f86410 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b86bdb68807ea33 server_cookie=734bab290c79824 in_seq=0 out_seq=0 2024-03-11T19:15:48.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.474+0000 7f9dc7fff640 1 -- 172.21.15.29:0/2538347360 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cb36c0b890 con 0x55cb36f83ff0 2024-03-11T19:15:48.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.474+0000 7f9dc7fff640 10 client.26401 handle_command_reply: tid=0 2024-03-11T19:15:48.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.475+0000 7f9df1e38740 1 client.26401 shutdown 2024-03-11T19:15:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.475+0000 7f9df1e38740 1 -- 172.21.15.29:0/2538347360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17c60 msgr2=0x55cb36f268a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.475+0000 7f9df1e38740 1 --2- 172.21.15.29:0/2538347360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17c60 0x55cb36f268a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc0059f0 tx=0x7f9ddc004840 comp rx=0 tx=0).stop 2024-03-11T19:15:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.475+0000 7f9df1e38740 1 -- 172.21.15.29:0/2538347360 shutdown_connections 2024-03-11T19:15:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.475+0000 7f9df1e38740 1 --2- 172.21.15.29:0/2538347360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb36f1cdd0 0x55cb36f27840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.475+0000 7f9df1e38740 1 --2- 172.21.15.29:0/2538347360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb36f17c60 0x55cb36f268a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.475+0000 7f9df1e38740 1 --2- 172.21.15.29:0/2538347360 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cb36f83ff0 0x55cb36f86410 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.475+0000 7f9dc7fff640 0 client.26401 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.475+0000 7f9df1e38740 1 --2- 172.21.15.29:0/2538347360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb36f17270 0x55cb36f26360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.475+0000 7f9df1e38740 1 -- 172.21.15.29:0/2538347360 >> 172.21.15.29:0/2538347360 conn(0x55cb36f127e0 msgr2=0x55cb36f191f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.475+0000 7f9df1e38740 1 -- 172.21.15.29:0/2538347360 shutdown_connections 2024-03-11T19:15:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.475+0000 7f9df1e38740 1 -- 172.21.15.29:0/2538347360 wait complete. 2024-03-11T19:15:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.475+0000 7f9df1e38740 20 client.26401 trim_cache size 0 max 16384 2024-03-11T19:15:48.483 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:15:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.476+0000 7f9def264640 1 -- 172.21.15.29:0/833039844 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9dd80a1b30 msgr2=0x7f9dd80a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.476+0000 7f9def264640 1 --2- 172.21.15.29:0/833039844 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9dd80a1b30 0x7f9dd80a3ff0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x55cb36f18060 tx=0x7f9dd402c770 comp rx=0 tx=0).stop 2024-03-11T19:15:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.476+0000 7f9def264640 1 -- 172.21.15.29:0/833039844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9de8109350 msgr2=0x7f9de81a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.476+0000 7f9def264640 1 --2- 172.21.15.29:0/833039844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9de8109350 0x7f9de81a4d30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9de0009fd0 tx=0x7f9de000ef10 comp rx=0 tx=0).stop 2024-03-11T19:15:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.476+0000 7f9def264640 1 -- 172.21.15.29:0/833039844 shutdown_connections 2024-03-11T19:15:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.476+0000 7f9def264640 1 --2- 172.21.15.29:0/833039844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9de8109350 0x7f9de81a4d30 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.476+0000 7f9def264640 1 --2- 172.21.15.29:0/833039844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9de8108990 0x7f9de819dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.476+0000 7f9def264640 1 --2- 172.21.15.29:0/833039844 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9dd80a1b30 0x7f9dd80a3ff0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.476+0000 7f9def264640 1 --2- 172.21.15.29:0/833039844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9de81077a0 0x7f9de819d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.476+0000 7f9def264640 1 -- 172.21.15.29:0/833039844 >> 172.21.15.29:0/833039844 conn(0x7f9de8076740 msgr2=0x7f9de810bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.476+0000 7f9def264640 1 -- 172.21.15.29:0/833039844 shutdown_connections 2024-03-11T19:15:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:48.477+0000 7f9def264640 1 -- 172.21.15.29:0/833039844 wait complete. 2024-03-11T19:15:49.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:48 smithi029 ceph-mon[32439]: pgmap v531: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-11T19:15:49.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:48 smithi190 ceph-mon[36292]: pgmap v531: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-11T19:15:49.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:48 smithi170 ceph-mon[35690]: pgmap v531: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-11T19:15:49.500 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:15:49.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:15:49.501 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:15:49.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.673+0000 7f4441064640 1 -- 172.21.15.29:0/88525992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f443c101190 msgr2=0x7f443c1015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:49.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.673+0000 7f4441064640 1 --2- 172.21.15.29:0/88525992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f443c101190 0x7f443c1015b0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f442c009900 tx=0x7f442c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:15:49.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.674+0000 7f4441064640 1 -- 172.21.15.29:0/88525992 shutdown_connections 2024-03-11T19:15:49.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.674+0000 7f4441064640 1 --2- 172.21.15.29:0/88525992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f443c102d40 0x7f443c1095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.674+0000 7f4441064640 1 --2- 172.21.15.29:0/88525992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f443c102380 0x7f443c102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.674+0000 7f4441064640 1 --2- 172.21.15.29:0/88525992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f443c101190 0x7f443c1015b0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.674+0000 7f4441064640 1 -- 172.21.15.29:0/88525992 >> 172.21.15.29:0/88525992 conn(0x7f443c0fc6e0 msgr2=0x7f443c0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:49.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.674+0000 7f4441064640 1 -- 172.21.15.29:0/88525992 shutdown_connections 2024-03-11T19:15:49.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.674+0000 7f4441064640 1 -- 172.21.15.29:0/88525992 wait complete. 2024-03-11T19:15:49.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.675+0000 7f4441064640 1 Processor -- start 2024-03-11T19:15:49.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.675+0000 7f4441064640 1 -- start start 2024-03-11T19:15:49.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.676+0000 7f4441064640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f443c101190 0x7f443c1993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.676+0000 7f4441064640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f443c102380 0x7f443c199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.676+0000 7f443ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f443c101190 0x7f443c1993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.676+0000 7f443ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f443c101190 0x7f443c1993c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54862/0 (socket says 172.21.15.29:54862) 2024-03-11T19:15:49.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.676+0000 7f443ad9d640 1 -- 172.21.15.29:0/1948512932 learned_addr learned my addr 172.21.15.29:0/1948512932 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:49.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.676+0000 7f4441064640 1 --2- 172.21.15.29:0/1948512932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f443c102d40 0x7f443c1a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.676+0000 7f4441064640 1 -- 172.21.15.29:0/1948512932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c1a0eb0 con 0x7f443c101190 2024-03-11T19:15:49.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.676+0000 7f4441064640 1 -- 172.21.15.29:0/1948512932 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c1a1020 con 0x7f443c102380 2024-03-11T19:15:49.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.676+0000 7f4441064640 1 -- 172.21.15.29:0/1948512932 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c1a11e0 con 0x7f443c102d40 2024-03-11T19:15:49.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.676+0000 7f443a59c640 1 --2- 172.21.15.29:0/1948512932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f443c102380 0x7f443c199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.676+0000 7f443b59e640 1 --2- 172.21.15.29:0/1948512932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f443c102d40 0x7f443c1a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.677+0000 7f443a59c640 1 -- 172.21.15.29:0/1948512932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f443c102d40 msgr2=0x7f443c1a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:49.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.677+0000 7f443a59c640 1 --2- 172.21.15.29:0/1948512932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f443c102d40 0x7f443c1a0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.677+0000 7f443a59c640 1 -- 172.21.15.29:0/1948512932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f443c101190 msgr2=0x7f443c1993c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:49.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.677+0000 7f443a59c640 1 --2- 172.21.15.29:0/1948512932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f443c101190 0x7f443c1993c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.677+0000 7f443a59c640 1 -- 172.21.15.29:0/1948512932 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c009560 con 0x7f443c102380 2024-03-11T19:15:49.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.677+0000 7f443ad9d640 1 --2- 172.21.15.29:0/1948512932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f443c101190 0x7f443c1993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:49.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.677+0000 7f443a59c640 1 --2- 172.21.15.29:0/1948512932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f443c102380 0x7f443c199900 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f443000e9f0 tx=0x7f443000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:49.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.677+0000 7f4423fff640 1 -- 172.21.15.29:0/1948512932 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443000ccc0 con 0x7f443c102380 2024-03-11T19:15:49.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.677+0000 7f4423fff640 1 -- 172.21.15.29:0/1948512932 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f443000ce20 con 0x7f443c102380 2024-03-11T19:15:49.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.677+0000 7f4441064640 1 -- 172.21.15.29:0/1948512932 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f443c1a14c0 con 0x7f443c102380 2024-03-11T19:15:49.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.678+0000 7f4423fff640 1 -- 172.21.15.29:0/1948512932 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44300187c0 con 0x7f443c102380 2024-03-11T19:15:49.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.678+0000 7f4441064640 1 -- 172.21.15.29:0/1948512932 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f443c1a1960 con 0x7f443c102380 2024-03-11T19:15:49.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.681+0000 7f4423fff640 1 -- 172.21.15.29:0/1948512932 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4430010430 con 0x7f443c102380 2024-03-11T19:15:49.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.681+0000 7f4443c35740 1 Processor -- start 2024-03-11T19:15:49.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.681+0000 7f4443c35740 1 -- start start 2024-03-11T19:15:49.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.681+0000 7f4443c35740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fb20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.681+0000 7f4443c35740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fb3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.681+0000 7f4443c35740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb3860 0x55f5c6fba0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.681+0000 7f4443c35740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5c6f18f90 con 0x55f5c6fb2ea0 2024-03-11T19:15:49.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.681+0000 7f4443c35740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5c6f164c0 con 0x55f5c6fb1cd0 2024-03-11T19:15:49.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.681+0000 7f4443c35740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5c6f11ca0 con 0x55f5c6fb3860 2024-03-11T19:15:49.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.681+0000 7f4423fff640 1 --2- 172.21.15.29:0/1948512932 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f44240a1b30 0x7f44240a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.682+0000 7f4423fff640 1 -- 172.21.15.29:0/1948512932 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4430014070 con 0x7f443c102380 2024-03-11T19:15:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.682+0000 7f443b59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fb20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.682+0000 7f443b59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fb20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:38468/0 (socket says 172.21.15.29:38468) 2024-03-11T19:15:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.682+0000 7f443b59e640 1 -- 172.21.15.29:0/4192793944 learned_addr learned my addr 172.21.15.29:0/4192793944 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.682+0000 7f443b59e640 1 --2- 172.21.15.29:0/1948512932 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f44240a1b30 0x7f44240a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.682+0000 7f443ad9d640 1 --2- 172.21.15.29:0/4192793944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fb3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.682+0000 7f443a59c640 1 --2- 172.21.15.29:0/4192793944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb3860 0x55f5c6fba0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.682+0000 7f443b59e640 1 -- 172.21.15.29:0/4192793944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb3860 msgr2=0x55f5c6fba0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.682+0000 7f443b59e640 1 --2- 172.21.15.29:0/4192793944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb3860 0x55f5c6fba0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.682+0000 7f443b59e640 1 -- 172.21.15.29:0/4192793944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb2ea0 msgr2=0x55f5c6fb3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.682+0000 7f443b59e640 1 --2- 172.21.15.29:0/4192793944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fb3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.682+0000 7f443b59e640 1 -- 172.21.15.29:0/4192793944 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5c6ed40b0 con 0x55f5c6fb1cd0 2024-03-11T19:15:49.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.682+0000 7f443ad9d640 1 --2- 172.21.15.29:0/4192793944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fb3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:49.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.683+0000 7f443b59e640 1 --2- 172.21.15.29:0/1948512932 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f44240a1b30 0x7f44240a3ff0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4434011cb0 tx=0x7f443400e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:49.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.683+0000 7f443b59e640 1 --2- 172.21.15.29:0/4192793944 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fb20f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4434009270 tx=0x7f44340117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e9758952a153148 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:49.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.683+0000 7f44217fa640 1 -- 172.21.15.29:0/4192793944 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443400dd50 con 0x55f5c6fb1cd0 2024-03-11T19:15:49.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.683+0000 7f44217fa640 1 -- 172.21.15.29:0/4192793944 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f443401a040 con 0x55f5c6fb1cd0 2024-03-11T19:15:49.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.683+0000 7f44217fa640 1 -- 172.21.15.29:0/4192793944 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4434004290 con 0x55f5c6fb1cd0 2024-03-11T19:15:49.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.684+0000 7f4443c35740 1 -- 172.21.15.29:0/4192793944 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb1cd0 msgr2=0x55f5c6fb20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:49.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.684+0000 7f4443c35740 1 --2- 172.21.15.29:0/4192793944 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fb20f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4434009270 tx=0x7f44340117b0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.684+0000 7f4443c35740 1 -- 172.21.15.29:0/4192793944 shutdown_connections 2024-03-11T19:15:49.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.685+0000 7f4443c35740 1 --2- 172.21.15.29:0/4192793944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb3860 0x55f5c6fba0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.685+0000 7f4443c35740 1 --2- 172.21.15.29:0/4192793944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fb3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.685+0000 7f4443c35740 1 --2- 172.21.15.29:0/4192793944 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fb20f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.685+0000 7f4443c35740 1 -- 172.21.15.29:0/4192793944 >> 172.21.15.29:0/4192793944 conn(0x55f5c6faceb0 msgr2=0x55f5c6fad2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:49.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.685+0000 7f4443c35740 1 -- 172.21.15.29:0/4192793944 shutdown_connections 2024-03-11T19:15:49.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.685+0000 7f4443c35740 1 -- 172.21.15.29:0/4192793944 wait complete. 2024-03-11T19:15:49.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.685+0000 7f4443c35740 1 -- start start 2024-03-11T19:15:49.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.685+0000 7f4443c35740 1 Processor -- start 2024-03-11T19:15:49.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.686+0000 7f4443c35740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fbbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.686+0000 7f4443c35740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fbc490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.686+0000 7f4443c35740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb3860 0x55f5c6fc34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.686+0000 7f4443c35740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5c6eb6f40 con 0x55f5c6fb1cd0 2024-03-11T19:15:49.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.686+0000 7f4443c35740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5c6be54b0 con 0x55f5c6fb2ea0 2024-03-11T19:15:49.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.686+0000 7f4443c35740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5c6e97670 con 0x55f5c6fb3860 2024-03-11T19:15:49.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.686+0000 7f4443c35740 10 client.0 _ll_register_callbacks cb 0x55f5c6f04f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:49.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.686+0000 7f443ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fbbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.686+0000 7f443a59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fbc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.686+0000 7f443ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fbbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54886/0 (socket says 172.21.15.29:54886) 2024-03-11T19:15:49.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.686+0000 7f443ad9d640 1 -- 172.21.15.29:0/3622135199 learned_addr learned my addr 172.21.15.29:0/3622135199 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:49.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.686+0000 7f443b59e640 1 --2- 172.21.15.29:0/3622135199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb3860 0x55f5c6fc34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.686+0000 7f443a59c640 1 -- 172.21.15.29:0/3622135199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb3860 msgr2=0x55f5c6fc34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:49.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.686+0000 7f443a59c640 1 --2- 172.21.15.29:0/3622135199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb3860 0x55f5c6fc34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.686+0000 7f443a59c640 1 -- 172.21.15.29:0/3622135199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb1cd0 msgr2=0x55f5c6fbbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:49.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.686+0000 7f443a59c640 1 --2- 172.21.15.29:0/3622135199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fbbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.687+0000 7f443a59c640 1 -- 172.21.15.29:0/3622135199 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4434014a70 con 0x55f5c6fb2ea0 2024-03-11T19:15:49.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.687+0000 7f443a59c640 1 --2- 172.21.15.29:0/3622135199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fbc490 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f443000c6a0 tx=0x7f443009a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc34c7b58976fceb server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.687+0000 7f441eff5640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:15:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.687+0000 7f441eff5640 1 -- 172.21.15.29:0/3622135199 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44300a2e20 con 0x55f5c6fb2ea0 2024-03-11T19:15:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.687+0000 7f441eff5640 1 -- 172.21.15.29:0/3622135199 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f44300b3190 con 0x55f5c6fb2ea0 2024-03-11T19:15:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.688+0000 7f4443c35740 1 -- 172.21.15.29:0/3622135199 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f5c6eebf40 con 0x55f5c6fb2ea0 2024-03-11T19:15:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.688+0000 7f441eff5640 1 -- 172.21.15.29:0/3622135199 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4430010b00 con 0x55f5c6fb2ea0 2024-03-11T19:15:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.688+0000 7f4443c35740 10 client.26405 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.688+0000 7f4443c35740 1 -- 172.21.15.29:0/3622135199 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55f5c6f10320 con 0x55f5c6fb2ea0 2024-03-11T19:15:49.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.688+0000 7f441eff5640 1 -- 172.21.15.29:0/3622135199 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f44300850c0 con 0x55f5c6fb2ea0 2024-03-11T19:15:49.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.689+0000 7f441eff5640 1 -- 172.21.15.29:0/3622135199 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f44300a2990 con 0x55f5c6fb2ea0 2024-03-11T19:15:49.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.689+0000 7f4443c35740 10 client.26405 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:49.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.689+0000 7f4443c35740 10 client.26405 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:15:49.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.689+0000 7f4443c35740 20 client.26405 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:49.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.689+0000 7f4443c35740 1 --2- 172.21.15.29:0/3622135199 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f5c6fc8070 0x55f5c6fca490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.689+0000 7f4443c35740 4 client.26405 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:15:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.690+0000 7f4443c35740 1 -- 172.21.15.29:0/3622135199 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f5c6eebf40 con 0x55f5c6fc8070 2024-03-11T19:15:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.690+0000 7f443ad9d640 1 --2- 172.21.15.29:0/3622135199 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f5c6fc8070 0x55f5c6fca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.691+0000 7f443ad9d640 1 --2- 172.21.15.29:0/3622135199 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f5c6fc8070 0x55f5c6fca490 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd95d43678eb1181 server_cookie=efe1d80568d70a49 in_seq=0 out_seq=0 2024-03-11T19:15:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.691+0000 7f441eff5640 10 client.26405 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.692+0000 7f441eff5640 1 -- 172.21.15.29:0/3622135199 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f5c6eebf40 con 0x55f5c6fc8070 2024-03-11T19:15:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.692+0000 7f441eff5640 10 client.26405 handle_command_reply: tid=0 2024-03-11T19:15:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.692+0000 7f4443c35740 1 client.26405 shutdown 2024-03-11T19:15:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.692+0000 7f4443c35740 1 -- 172.21.15.29:0/3622135199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb2ea0 msgr2=0x55f5c6fbc490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.692+0000 7f4443c35740 1 --2- 172.21.15.29:0/3622135199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fbc490 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f443000c6a0 tx=0x7f443009a680 comp rx=0 tx=0).stop 2024-03-11T19:15:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.693+0000 7f4443c35740 1 -- 172.21.15.29:0/3622135199 shutdown_connections 2024-03-11T19:15:49.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.693+0000 7f4443c35740 1 --2- 172.21.15.29:0/3622135199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb3860 0x55f5c6fc34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.693+0000 7f4443c35740 1 --2- 172.21.15.29:0/3622135199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fbc490 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.693+0000 7f4443c35740 1 --2- 172.21.15.29:0/3622135199 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f5c6fc8070 0x55f5c6fca490 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.693+0000 7f4443c35740 1 --2- 172.21.15.29:0/3622135199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fbbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.693+0000 7f4443c35740 1 -- 172.21.15.29:0/3622135199 >> 172.21.15.29:0/3622135199 conn(0x55f5c6faceb0 msgr2=0x55f5c6fa38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:49.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.693+0000 7f441eff5640 0 client.26405 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:49.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.693+0000 7f4443c35740 1 -- 172.21.15.29:0/3622135199 shutdown_connections 2024-03-11T19:15:49.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.694+0000 7f4443c35740 1 -- 172.21.15.29:0/3622135199 wait complete. 2024-03-11T19:15:49.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.694+0000 7f4443c35740 20 client.26405 trim_cache size 0 max 16384 2024-03-11T19:15:49.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.724+0000 7f4443c35740 1 Processor -- start 2024-03-11T19:15:49.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.724+0000 7f4443c35740 1 -- start start 2024-03-11T19:15:49.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.725+0000 7f4443c35740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fcee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.725+0000 7f4443c35740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fc3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.725+0000 7f4443c35740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb3860 0x55f5c6fbbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.725+0000 7f443a59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fc3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.725+0000 7f443a59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fc3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49260/0 (socket says 172.21.15.29:49260) 2024-03-11T19:15:49.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.725+0000 7f443ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fcee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.725+0000 7f443a59c640 1 -- 172.21.15.29:0/1637901363 learned_addr learned my addr 172.21.15.29:0/1637901363 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:49.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.725+0000 7f4443c35740 1 -- 172.21.15.29:0/1637901363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5c6f9fe10 con 0x55f5c6fb1cd0 2024-03-11T19:15:49.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.725+0000 7f4443c35740 1 -- 172.21.15.29:0/1637901363 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5c6fad6f0 con 0x55f5c6fb3860 2024-03-11T19:15:49.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.726+0000 7f4443c35740 1 -- 172.21.15.29:0/1637901363 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5c6fb2c30 con 0x55f5c6fb2ea0 2024-03-11T19:15:49.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.726+0000 7f443b59e640 1 --2- 172.21.15.29:0/1637901363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb3860 0x55f5c6fbbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.726+0000 7f443a59c640 1 -- 172.21.15.29:0/1637901363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb3860 msgr2=0x55f5c6fbbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:49.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.726+0000 7f443a59c640 1 --2- 172.21.15.29:0/1637901363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb3860 0x55f5c6fbbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.726+0000 7f443a59c640 1 -- 172.21.15.29:0/1637901363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb1cd0 msgr2=0x55f5c6fcee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:49.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.726+0000 7f443a59c640 1 --2- 172.21.15.29:0/1637901363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fcee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.726+0000 7f443a59c640 1 -- 172.21.15.29:0/1637901363 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5c6eacc00 con 0x55f5c6fb2ea0 2024-03-11T19:15:49.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.726+0000 7f443ad9d640 1 --2- 172.21.15.29:0/1637901363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fcee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:49.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.726+0000 7f443a59c640 1 --2- 172.21.15.29:0/1637901363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fc3490 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f44300ba420 tx=0x7f443000cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc6655e1d38a47c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:49.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.727+0000 7f4420ff9640 1 -- 172.21.15.29:0/1637901363 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4430096920 con 0x55f5c6fb2ea0 2024-03-11T19:15:49.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.727+0000 7f4420ff9640 1 -- 172.21.15.29:0/1637901363 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4430096720 con 0x55f5c6fb2ea0 2024-03-11T19:15:49.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.727+0000 7f4420ff9640 1 -- 172.21.15.29:0/1637901363 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44300850b0 con 0x55f5c6fb2ea0 2024-03-11T19:15:49.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.728+0000 7f4443c35740 1 -- 172.21.15.29:0/1637901363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb2ea0 msgr2=0x55f5c6fc3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:49.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.728+0000 7f4443c35740 1 --2- 172.21.15.29:0/1637901363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fc3490 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f44300ba420 tx=0x7f443000cf90 comp rx=0 tx=0).stop 2024-03-11T19:15:49.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.728+0000 7f4443c35740 1 -- 172.21.15.29:0/1637901363 shutdown_connections 2024-03-11T19:15:49.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.728+0000 7f4443c35740 1 --2- 172.21.15.29:0/1637901363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb3860 0x55f5c6fbbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.728+0000 7f4443c35740 1 --2- 172.21.15.29:0/1637901363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fc3490 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.728+0000 7f4443c35740 1 --2- 172.21.15.29:0/1637901363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fcee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.728+0000 7f4443c35740 1 -- 172.21.15.29:0/1637901363 >> 172.21.15.29:0/1637901363 conn(0x55f5c6faceb0 msgr2=0x55f5c6fbb570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:49.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.728+0000 7f4443c35740 1 -- 172.21.15.29:0/1637901363 shutdown_connections 2024-03-11T19:15:49.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.728+0000 7f4443c35740 1 -- 172.21.15.29:0/1637901363 wait complete. 2024-03-11T19:15:49.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.728+0000 7f4443c35740 1 -- start start 2024-03-11T19:15:49.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.729+0000 7f4443c35740 1 Processor -- start 2024-03-11T19:15:49.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.729+0000 7f4443c35740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fc3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.729+0000 7f4443c35740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fbfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.729+0000 7f4443c35740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb3860 0x55f5c6fc0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.729+0000 7f4443c35740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5c6fc06a0 con 0x55f5c6fb3860 2024-03-11T19:15:49.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.729+0000 7f4443c35740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5c6fc0810 con 0x55f5c6fb2ea0 2024-03-11T19:15:49.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.729+0000 7f4443c35740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5c701e580 con 0x55f5c6fb1cd0 2024-03-11T19:15:49.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.729+0000 7f4443c35740 10 client.0 _ll_register_callbacks cb 0x7f4434009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:49.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.729+0000 7f443ad9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fc3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.729+0000 7f443a59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fbfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.729+0000 7f443b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb3860 0x55f5c6fc0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.729+0000 7f443ad9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fc3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49272/0 (socket says 172.21.15.29:49272) 2024-03-11T19:15:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.729+0000 7f443ad9d640 1 -- 172.21.15.29:0/3995168357 learned_addr learned my addr 172.21.15.29:0/3995168357 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.730+0000 7f443ad9d640 1 -- 172.21.15.29:0/3995168357 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb2ea0 msgr2=0x55f5c6fbfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.730+0000 7f443ad9d640 1 --2- 172.21.15.29:0/3995168357 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fbfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.730+0000 7f443ad9d640 1 -- 172.21.15.29:0/3995168357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb3860 msgr2=0x55f5c6fc0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.730+0000 7f443ad9d640 1 --2- 172.21.15.29:0/3995168357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb3860 0x55f5c6fc0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.730+0000 7f443ad9d640 1 -- 172.21.15.29:0/3995168357 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f442c030070 con 0x55f5c6fb1cd0 2024-03-11T19:15:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.730+0000 7f443a59c640 1 --2- 172.21.15.29:0/3995168357 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fbfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.730+0000 7f443ad9d640 1 --2- 172.21.15.29:0/3995168357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fc3180 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f442c009850 tx=0x7f442c004b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5777b2f1892caea0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.730+0000 7f441eff5640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:15:49.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.730+0000 7f441eff5640 1 -- 172.21.15.29:0/3995168357 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c045040 con 0x55f5c6fb1cd0 2024-03-11T19:15:49.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.730+0000 7f441eff5640 1 -- 172.21.15.29:0/3995168357 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f442c02de40 con 0x55f5c6fb1cd0 2024-03-11T19:15:49.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.730+0000 7f4443c35740 1 -- 172.21.15.29:0/3995168357 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f5c6f37900 con 0x55f5c6fb1cd0 2024-03-11T19:15:49.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.731+0000 7f441eff5640 1 -- 172.21.15.29:0/3995168357 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f442c004320 con 0x55f5c6fb1cd0 2024-03-11T19:15:49.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.731+0000 7f4443c35740 10 client.26416 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:49.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.731+0000 7f4443c35740 1 -- 172.21.15.29:0/3995168357 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55f5c6f10320 con 0x55f5c6fb1cd0 2024-03-11T19:15:49.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.731+0000 7f441eff5640 1 -- 172.21.15.29:0/3995168357 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f5c6f37900 con 0x55f5c6fb1cd0 2024-03-11T19:15:49.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.732+0000 7f441eff5640 1 -- 172.21.15.29:0/3995168357 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f442c05e080 con 0x55f5c6fb1cd0 2024-03-11T19:15:49.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.732+0000 7f4443c35740 10 client.26416 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:49.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.732+0000 7f4443c35740 10 client.26416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:15:49.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.732+0000 7f4443c35740 20 client.26416 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:49.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.732+0000 7f4443c35740 1 --2- 172.21.15.29:0/3995168357 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f5c7020010 0x55f5c7022430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.732+0000 7f4443c35740 4 client.26416 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:15:49.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.732+0000 7f4443c35740 1 -- 172.21.15.29:0/3995168357 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f5c6f08310 con 0x55f5c7020010 2024-03-11T19:15:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.734+0000 7f443a59c640 1 --2- 172.21.15.29:0/3995168357 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f5c7020010 0x55f5c7022430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.734+0000 7f443a59c640 1 --2- 172.21.15.29:0/3995168357 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f5c7020010 0x55f5c7022430 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29bd714a01bc4b81 server_cookie=d8b62026d71caf0e in_seq=0 out_seq=0 2024-03-11T19:15:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.734+0000 7f441eff5640 10 client.26416 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:49.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.736+0000 7f441eff5640 1 -- 172.21.15.29:0/3995168357 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f5c6f08310 con 0x55f5c7020010 2024-03-11T19:15:49.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.736+0000 7f441eff5640 10 client.26416 handle_command_reply: tid=0 2024-03-11T19:15:49.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.736+0000 7f4443c35740 1 client.26416 shutdown 2024-03-11T19:15:49.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.737+0000 7f4443c35740 1 -- 172.21.15.29:0/3995168357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb1cd0 msgr2=0x55f5c6fc3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:49.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.737+0000 7f4443c35740 1 --2- 172.21.15.29:0/3995168357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fc3180 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f442c009850 tx=0x7f442c004b90 comp rx=0 tx=0).stop 2024-03-11T19:15:49.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.738+0000 7f4443c35740 1 -- 172.21.15.29:0/3995168357 shutdown_connections 2024-03-11T19:15:49.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.738+0000 7f4443c35740 1 --2- 172.21.15.29:0/3995168357 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f5c7020010 0x55f5c7022430 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.738+0000 7f4443c35740 1 --2- 172.21.15.29:0/3995168357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f5c6fb3860 0x55f5c6fc0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.738+0000 7f441eff5640 0 client.26416 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:49.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.738+0000 7f4443c35740 1 --2- 172.21.15.29:0/3995168357 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f5c6fb2ea0 0x55f5c6fbfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.738+0000 7f4443c35740 1 --2- 172.21.15.29:0/3995168357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f5c6fb1cd0 0x55f5c6fc3180 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.738+0000 7f4443c35740 1 -- 172.21.15.29:0/3995168357 >> 172.21.15.29:0/3995168357 conn(0x55f5c6faceb0 msgr2=0x55f5c6fb0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:49.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.738+0000 7f4443c35740 1 -- 172.21.15.29:0/3995168357 shutdown_connections 2024-03-11T19:15:49.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.738+0000 7f4443c35740 1 -- 172.21.15.29:0/3995168357 wait complete. 2024-03-11T19:15:49.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.739+0000 7f4443c35740 20 client.26416 trim_cache size 0 max 16384 2024-03-11T19:15:49.744 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:15:49.744 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:15:49.745 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "86c5ee58-6dab-4905-8e39-546bd7820164", 2024-03-11T19:15:49.745 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:15:49.745 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:15:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.740+0000 7f4441064640 1 -- 172.21.15.29:0/1948512932 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f44240a1b30 msgr2=0x7f44240a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.740+0000 7f4441064640 1 --2- 172.21.15.29:0/1948512932 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f44240a1b30 0x7f44240a3ff0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4434011cb0 tx=0x7f443400e000 comp rx=0 tx=0).stop 2024-03-11T19:15:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.740+0000 7f4441064640 1 -- 172.21.15.29:0/1948512932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f443c102380 msgr2=0x7f443c199900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.740+0000 7f4441064640 1 --2- 172.21.15.29:0/1948512932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f443c102380 0x7f443c199900 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f443000e9f0 tx=0x7f443000eeb0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.741+0000 7f4441064640 1 -- 172.21.15.29:0/1948512932 shutdown_connections 2024-03-11T19:15:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.741+0000 7f4441064640 1 --2- 172.21.15.29:0/1948512932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f443c102d40 0x7f443c1a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.741+0000 7f4441064640 1 --2- 172.21.15.29:0/1948512932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f443c102380 0x7f443c199900 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.741+0000 7f4441064640 1 --2- 172.21.15.29:0/1948512932 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f44240a1b30 0x7f44240a3ff0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.741+0000 7f4441064640 1 --2- 172.21.15.29:0/1948512932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f443c101190 0x7f443c1993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.741+0000 7f4441064640 1 -- 172.21.15.29:0/1948512932 >> 172.21.15.29:0/1948512932 conn(0x7f443c0fc6e0 msgr2=0x7f443c107bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:49.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.743+0000 7f4441064640 1 -- 172.21.15.29:0/1948512932 shutdown_connections 2024-03-11T19:15:49.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.744+0000 7f4441064640 1 -- 172.21.15.29:0/1948512932 wait complete. 2024-03-11T19:15:49.768 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:15:49.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:15:49.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.946+0000 7f0fc2dd7640 1 -- 172.21.15.29:0/3921031592 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fbc1077c0 msgr2=0x7f0fbc107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:49.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.946+0000 7f0fc2dd7640 1 --2- 172.21.15.29:0/3921031592 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fbc1077c0 0x7f0fbc107be0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4009be0 tx=0x7f0fb402bef0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.946+0000 7f0fc2dd7640 1 -- 172.21.15.29:0/3921031592 shutdown_connections 2024-03-11T19:15:49.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.946+0000 7f0fc2dd7640 1 --2- 172.21.15.29:0/3921031592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fbc109370 0x7f0fbc10fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.946+0000 7f0fc2dd7640 1 --2- 172.21.15.29:0/3921031592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fbc1089b0 0x7f0fbc108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.946+0000 7f0fc2dd7640 1 --2- 172.21.15.29:0/3921031592 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fbc1077c0 0x7f0fbc107be0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.946+0000 7f0fc2dd7640 1 -- 172.21.15.29:0/3921031592 >> 172.21.15.29:0/3921031592 conn(0x7f0fbc076770 msgr2=0x7f0fbc078b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:49.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.946+0000 7f0fc2dd7640 1 -- 172.21.15.29:0/3921031592 shutdown_connections 2024-03-11T19:15:49.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.946+0000 7f0fc2dd7640 1 -- 172.21.15.29:0/3921031592 wait complete. 2024-03-11T19:15:49.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.947+0000 7f0fc2dd7640 1 Processor -- start 2024-03-11T19:15:49.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.947+0000 7f0fc2dd7640 1 -- start start 2024-03-11T19:15:49.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.947+0000 7f0fc2dd7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fbc1077c0 0x7f0fbc19d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.947+0000 7f0fc2dd7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fbc1089b0 0x7f0fbc19dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.947+0000 7f0fc2dd7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fbc109370 0x7f0fbc1a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.947+0000 7f0fc2dd7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fbc1a5230 con 0x7f0fbc1077c0 2024-03-11T19:15:49.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.947+0000 7f0fc2dd7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fbc1a53a0 con 0x7f0fbc1089b0 2024-03-11T19:15:49.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.947+0000 7f0fc2dd7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fbc1a5560 con 0x7f0fbc109370 2024-03-11T19:15:49.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.947+0000 7f0fc1374640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fbc109370 0x7f0fbc1a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.947+0000 7f0fbbfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fbc1089b0 0x7f0fbc19dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.947+0000 7f0fbbfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fbc1089b0 0x7f0fbc19dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:38508/0 (socket says 172.21.15.29:38508) 2024-03-11T19:15:49.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.947+0000 7f0fbbfff640 1 -- 172.21.15.29:0/784307187 learned_addr learned my addr 172.21.15.29:0/784307187 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:49.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.948+0000 7f0fc0b73640 1 --2- 172.21.15.29:0/784307187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fbc1077c0 0x7f0fbc19d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.948+0000 7f0fc1374640 1 -- 172.21.15.29:0/784307187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fbc1089b0 msgr2=0x7f0fbc19dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:49.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.948+0000 7f0fc1374640 1 --2- 172.21.15.29:0/784307187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fbc1089b0 0x7f0fbc19dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.948+0000 7f0fc1374640 1 -- 172.21.15.29:0/784307187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fbc1077c0 msgr2=0x7f0fbc19d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:49.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.948+0000 7f0fc1374640 1 --2- 172.21.15.29:0/784307187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fbc1077c0 0x7f0fbc19d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.948+0000 7f0fc1374640 1 -- 172.21.15.29:0/784307187 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb4009840 con 0x7f0fbc109370 2024-03-11T19:15:49.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.948+0000 7f0fc1374640 1 --2- 172.21.15.29:0/784307187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fbc109370 0x7f0fbc1a4cf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0fac009fd0 tx=0x7f0fac00eba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:49.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.949+0000 7f0fb9ffb640 1 -- 172.21.15.29:0/784307187 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac004220 con 0x7f0fbc109370 2024-03-11T19:15:49.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.949+0000 7f0fb9ffb640 1 -- 172.21.15.29:0/784307187 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0fac00cd60 con 0x7f0fbc109370 2024-03-11T19:15:49.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.949+0000 7f0fc2dd7640 1 -- 172.21.15.29:0/784307187 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fbc1a5840 con 0x7f0fbc109370 2024-03-11T19:15:49.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.949+0000 7f0fb9ffb640 1 -- 172.21.15.29:0/784307187 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac0047a0 con 0x7f0fbc109370 2024-03-11T19:15:49.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.950+0000 7f0fc2dd7640 1 -- 172.21.15.29:0/784307187 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fbc1a5cf0 con 0x7f0fbc109370 2024-03-11T19:15:49.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.953+0000 7f0fb9ffb640 1 -- 172.21.15.29:0/784307187 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0fac00c8b0 con 0x7f0fbc109370 2024-03-11T19:15:49.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.953+0000 7f0fc59ab740 1 Processor -- start 2024-03-11T19:15:49.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.953+0000 7f0fc59ab740 1 -- start start 2024-03-11T19:15:49.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.954+0000 7f0fc59ab740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0e2d0 0x5637eeb0e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.954+0000 7f0fc59ab740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0ecc0 0x5637eeb137c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.954+0000 7f0fc59ab740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb13e30 0x5637eeb162c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.954+0000 7f0fc59ab740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637eea7e280 con 0x5637eeb13e30 2024-03-11T19:15:49.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.954+0000 7f0fc59ab740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637eea7b7b0 con 0x5637eeb0e2d0 2024-03-11T19:15:49.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.954+0000 7f0fc59ab740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637eea76f90 con 0x5637eeb0ecc0 2024-03-11T19:15:49.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.954+0000 7f0fbbfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0e2d0 0x5637eeb0e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.954+0000 7f0fbbfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0e2d0 0x5637eeb0e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:38516/0 (socket says 172.21.15.29:38516) 2024-03-11T19:15:49.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.954+0000 7f0fbbfff640 1 -- 172.21.15.29:0/3578783861 learned_addr learned my addr 172.21.15.29:0/3578783861 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:49.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.954+0000 7f0fc0b73640 1 --2- 172.21.15.29:0/3578783861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0ecc0 0x5637eeb137c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.954+0000 7f0fbbfff640 1 --2- 172.21.15.29:0/3578783861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb13e30 0x5637eeb162c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.954+0000 7f0fc0b73640 1 -- 172.21.15.29:0/3578783861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0e2d0 msgr2=0x5637eeb0e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:49.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.954+0000 7f0fc0b73640 1 --2- 172.21.15.29:0/3578783861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0e2d0 0x5637eeb0e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.954+0000 7f0fc0b73640 1 -- 172.21.15.29:0/3578783861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb13e30 msgr2=0x5637eeb162c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:49.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.954+0000 7f0fc0b73640 1 --2- 172.21.15.29:0/3578783861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb13e30 0x5637eeb162c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.954+0000 7f0fc0b73640 1 -- 172.21.15.29:0/3578783861 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5637eea35570 con 0x5637eeb0ecc0 2024-03-11T19:15:49.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.955+0000 7f0fc0b73640 1 --2- 172.21.15.29:0/3578783861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0ecc0 0x5637eeb137c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4036040 tx=0x7f0fb402c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb3510aae4e1b35c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:49.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.955+0000 7f0fb9ffb640 1 --2- 172.21.15.29:0/784307187 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0fa40a1b30 0x7f0fa40a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.955+0000 7f0fa2ffd640 1 -- 172.21.15.29:0/3578783861 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4033450 con 0x5637eeb0ecc0 2024-03-11T19:15:49.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.955+0000 7f0fbbfff640 1 --2- 172.21.15.29:0/784307187 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0fa40a1b30 0x7f0fa40a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.955+0000 7f0fa2ffd640 1 -- 172.21.15.29:0/3578783861 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0fb40335b0 con 0x5637eeb0ecc0 2024-03-11T19:15:49.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.955+0000 7f0fa2ffd640 1 -- 172.21.15.29:0/3578783861 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4004040 con 0x5637eeb0ecc0 2024-03-11T19:15:49.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.956+0000 7f0fb9ffb640 1 -- 172.21.15.29:0/784307187 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0fac014070 con 0x7f0fbc109370 2024-03-11T19:15:49.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.956+0000 7f0fbbfff640 1 --2- 172.21.15.29:0/784307187 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0fa40a1b30 0x7f0fa40a3ff0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x5637eeb0f520 tx=0x7f0fb0004730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:49.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.956+0000 7f0fc59ab740 1 -- 172.21.15.29:0/3578783861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0ecc0 msgr2=0x5637eeb137c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:49.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.956+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/3578783861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0ecc0 0x5637eeb137c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4036040 tx=0x7f0fb402c890 comp rx=0 tx=0).stop 2024-03-11T19:15:49.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.956+0000 7f0fc59ab740 1 -- 172.21.15.29:0/3578783861 shutdown_connections 2024-03-11T19:15:49.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.956+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/3578783861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb13e30 0x5637eeb162c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.956+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/3578783861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0ecc0 0x5637eeb137c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.956+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/3578783861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0e2d0 0x5637eeb0e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.956+0000 7f0fc59ab740 1 -- 172.21.15.29:0/3578783861 >> 172.21.15.29:0/3578783861 conn(0x5637eeb097f0 msgr2=0x5637eeb09bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:49.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.956+0000 7f0fc59ab740 1 -- 172.21.15.29:0/3578783861 shutdown_connections 2024-03-11T19:15:49.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.956+0000 7f0fc59ab740 1 -- 172.21.15.29:0/3578783861 wait complete. 2024-03-11T19:15:49.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.957+0000 7f0fc59ab740 1 -- start start 2024-03-11T19:15:49.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.957+0000 7f0fc59ab740 1 Processor -- start 2024-03-11T19:15:49.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.957+0000 7f0fc59ab740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb0e2d0 0x5637eeb18200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.958+0000 7f0fc0b73640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb0e2d0 0x5637eeb18200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.958+0000 7f0fc0b73640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb0e2d0 0x5637eeb18200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54946/0 (socket says 172.21.15.29:54946) 2024-03-11T19:15:49.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.958+0000 7f0fc0b73640 1 -- 172.21.15.29:0/2293625835 learned_addr learned my addr 172.21.15.29:0/2293625835 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:49.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.958+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/2293625835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0ecc0 0x5637eeb18740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.958+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/2293625835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb13e30 0x5637eeb1f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.958+0000 7f0fbbfff640 1 --2- 172.21.15.29:0/2293625835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0ecc0 0x5637eeb18740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.958+0000 7f0fc1374640 1 --2- 172.21.15.29:0/2293625835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb13e30 0x5637eeb1f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.958+0000 7f0fc59ab740 1 -- 172.21.15.29:0/2293625835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637ee74a430 con 0x5637eeb0e2d0 2024-03-11T19:15:49.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.958+0000 7f0fc59ab740 1 -- 172.21.15.29:0/2293625835 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637ee706e00 con 0x5637eeb0ecc0 2024-03-11T19:15:49.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.958+0000 7f0fc59ab740 1 -- 172.21.15.29:0/2293625835 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637ee9fbfd0 con 0x5637eeb13e30 2024-03-11T19:15:49.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.958+0000 7f0fbbfff640 1 -- 172.21.15.29:0/2293625835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb13e30 msgr2=0x5637eeb1f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:49.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.958+0000 7f0fbbfff640 1 --2- 172.21.15.29:0/2293625835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb13e30 0x5637eeb1f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.958+0000 7f0fc59ab740 10 client.0 _ll_register_callbacks cb 0x5637eea51fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:49.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.958+0000 7f0fbbfff640 1 -- 172.21.15.29:0/2293625835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb0e2d0 msgr2=0x5637eeb18200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:49.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.958+0000 7f0fbbfff640 1 --2- 172.21.15.29:0/2293625835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb0e2d0 0x5637eeb18200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.959+0000 7f0fbbfff640 1 -- 172.21.15.29:0/2293625835 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fbc110500 con 0x5637eeb0ecc0 2024-03-11T19:15:49.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.959+0000 7f0fc0b73640 1 --2- 172.21.15.29:0/2293625835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb0e2d0 0x5637eeb18200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:49.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.959+0000 7f0fc59ab740 1 -- 172.21.15.29:0/2293625835 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5637ee7767e0 con 0x5637eeb0ecc0 2024-03-11T19:15:49.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.959+0000 7f0fbbfff640 1 --2- 172.21.15.29:0/2293625835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0ecc0 0x5637eeb18740 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0004600 tx=0x7f0fb00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91ab413fbde6fd77 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:49.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.959+0000 7f0f9bfff640 10 client.26423 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:15:49.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.959+0000 7f0f9bfff640 1 -- 172.21.15.29:0/2293625835 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0026070 con 0x5637eeb0ecc0 2024-03-11T19:15:49.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.959+0000 7f0f9bfff640 1 -- 172.21.15.29:0/2293625835 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0fb00217f0 con 0x5637eeb0ecc0 2024-03-11T19:15:49.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.959+0000 7f0f9bfff640 1 -- 172.21.15.29:0/2293625835 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fb0016070 con 0x5637eeb0ecc0 2024-03-11T19:15:49.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.959+0000 7f0fc59ab740 10 client.26423 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:49.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.959+0000 7f0fc59ab740 1 -- 172.21.15.29:0/2293625835 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x5637eea75610 con 0x5637eeb0ecc0 2024-03-11T19:15:49.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.960+0000 7f0f9bfff640 1 -- 172.21.15.29:0/2293625835 <== mon.2 v2:172.21.15.170:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0fb001d070 con 0x5637eeb0ecc0 2024-03-11T19:15:49.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.960+0000 7f0fc59ab740 10 client.26423 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:49.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.960+0000 7f0fc59ab740 10 client.26423 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:15:49.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.960+0000 7f0fc59ab740 20 client.26423 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:49.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.960+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/2293625835 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5637eeb24250 0x5637eeb26680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:49.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.961+0000 7f0fc59ab740 4 client.26423 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:15:49.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.961+0000 7f0fc59ab740 1 -- 172.21.15.29:0/2293625835 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5637ee7b3720 con 0x5637eeb24250 2024-03-11T19:15:49.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.961+0000 7f0fc0b73640 1 --2- 172.21.15.29:0/2293625835 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5637eeb24250 0x5637eeb26680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:49.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.961+0000 7f0fc0b73640 1 --2- 172.21.15.29:0/2293625835 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5637eeb24250 0x5637eeb26680 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93a0029f8d9ac393 server_cookie=3332667316ff2e0e in_seq=0 out_seq=0 2024-03-11T19:15:49.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.961+0000 7f0f9bfff640 10 client.26423 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:49.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.962+0000 7f0f9bfff640 1 -- 172.21.15.29:0/2293625835 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5637ee7b3720 con 0x5637eeb24250 2024-03-11T19:15:49.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.962+0000 7f0f9bfff640 10 client.26423 handle_command_reply: tid=0 2024-03-11T19:15:49.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.963+0000 7f0fc59ab740 1 client.26423 shutdown 2024-03-11T19:15:49.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.963+0000 7f0fc59ab740 1 -- 172.21.15.29:0/2293625835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0ecc0 msgr2=0x5637eeb18740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:49.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.963+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/2293625835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0ecc0 0x5637eeb18740 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0004600 tx=0x7f0fb00046c0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.963+0000 7f0fc59ab740 1 -- 172.21.15.29:0/2293625835 shutdown_connections 2024-03-11T19:15:49.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.963+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/2293625835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb13e30 0x5637eeb1f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.963+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/2293625835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0ecc0 0x5637eeb18740 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.963+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/2293625835 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5637eeb24250 0x5637eeb26680 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.963+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/2293625835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb0e2d0 0x5637eeb18200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:49.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.963+0000 7f0f9bfff640 0 client.26423 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:49.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.963+0000 7f0fc59ab740 1 -- 172.21.15.29:0/2293625835 >> 172.21.15.29:0/2293625835 conn(0x5637eeb097f0 msgr2=0x5637eeb08a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:49.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.964+0000 7f0fc59ab740 1 -- 172.21.15.29:0/2293625835 shutdown_connections 2024-03-11T19:15:49.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.964+0000 7f0fc59ab740 1 -- 172.21.15.29:0/2293625835 wait complete. 2024-03-11T19:15:49.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.964+0000 7f0fc59ab740 20 client.26423 trim_cache size 0 max 16384 2024-03-11T19:15:49.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.996+0000 7f0fc59ab740 1 Processor -- start 2024-03-11T19:15:50.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.996+0000 7f0fc59ab740 1 -- start start 2024-03-11T19:15:50.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.996+0000 7f0fc59ab740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0e2d0 0x5637eeb0e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:50.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.996+0000 7f0fc59ab740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0ecc0 0x5637eeb1f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:50.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.996+0000 7f0fc59ab740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb13e30 0x5637eeb181c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:50.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.996+0000 7f0fc59ab740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637eeb18f30 con 0x5637eeb13e30 2024-03-11T19:15:50.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.996+0000 7f0fc59ab740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637eeb16980 con 0x5637eeb0e2d0 2024-03-11T19:15:50.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.996+0000 7f0fc59ab740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637ee7b3720 con 0x5637eeb0ecc0 2024-03-11T19:15:50.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.996+0000 7f0fbbfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0ecc0 0x5637eeb1f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:50.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.996+0000 7f0fbbfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0ecc0 0x5637eeb1f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49318/0 (socket says 172.21.15.29:49318) 2024-03-11T19:15:50.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.996+0000 7f0fbbfff640 1 -- 172.21.15.29:0/3095675068 learned_addr learned my addr 172.21.15.29:0/3095675068 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:50.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.996+0000 7f0fc1374640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb13e30 0x5637eeb181c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:50.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.996+0000 7f0fc0b73640 1 --2- 172.21.15.29:0/3095675068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0e2d0 0x5637eeb0e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:50.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.997+0000 7f0fc1374640 1 -- 172.21.15.29:0/3095675068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0ecc0 msgr2=0x5637eeb1f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:50.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.997+0000 7f0fc1374640 1 --2- 172.21.15.29:0/3095675068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0ecc0 0x5637eeb1f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:50.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.997+0000 7f0fc1374640 1 -- 172.21.15.29:0/3095675068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0e2d0 msgr2=0x5637eeb0e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:50.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.997+0000 7f0fc1374640 1 --2- 172.21.15.29:0/3095675068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0e2d0 0x5637eeb0e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:50.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.997+0000 7f0fc1374640 1 -- 172.21.15.29:0/3095675068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5637eea11ac0 con 0x5637eeb13e30 2024-03-11T19:15:50.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.997+0000 7f0fbbfff640 1 --2- 172.21.15.29:0/3095675068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0ecc0 0x5637eeb1f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:50.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.997+0000 7f0fc0b73640 1 --2- 172.21.15.29:0/3095675068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0e2d0 0x5637eeb0e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:15:50.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.997+0000 7f0fc1374640 1 --2- 172.21.15.29:0/3095675068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb13e30 0x5637eeb181c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f0fac0cac40 tx=0x7f0fac099fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d81f1fee666c6d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:50.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.997+0000 7f0fa27fc640 1 -- 172.21.15.29:0/3095675068 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac09a7b0 con 0x5637eeb13e30 2024-03-11T19:15:50.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.997+0000 7f0fa27fc640 1 -- 172.21.15.29:0/3095675068 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0fac09a0b0 con 0x5637eeb13e30 2024-03-11T19:15:50.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.997+0000 7f0fa27fc640 1 -- 172.21.15.29:0/3095675068 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac096720 con 0x5637eeb13e30 2024-03-11T19:15:50.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.998+0000 7f0fc59ab740 1 -- 172.21.15.29:0/3095675068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb13e30 msgr2=0x5637eeb181c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:50.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.998+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/3095675068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb13e30 0x5637eeb181c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f0fac0cac40 tx=0x7f0fac099fa0 comp rx=0 tx=0).stop 2024-03-11T19:15:50.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.998+0000 7f0fc59ab740 1 -- 172.21.15.29:0/3095675068 shutdown_connections 2024-03-11T19:15:50.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.998+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/3095675068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb13e30 0x5637eeb181c0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:50.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.998+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/3095675068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0ecc0 0x5637eeb1f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:50.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.998+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/3095675068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0e2d0 0x5637eeb0e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.998+0000 7f0fc59ab740 1 -- 172.21.15.29:0/3095675068 >> 172.21.15.29:0/3095675068 conn(0x5637eeb097f0 msgr2=0x5637eeb0ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.998+0000 7f0fc59ab740 1 -- 172.21.15.29:0/3095675068 shutdown_connections 2024-03-11T19:15:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.998+0000 7f0fc59ab740 1 -- 172.21.15.29:0/3095675068 wait complete. 2024-03-11T19:15:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.999+0000 7f0fc59ab740 1 -- start start 2024-03-11T19:15:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.999+0000 7f0fc59ab740 1 Processor -- start 2024-03-11T19:15:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.999+0000 7f0fc59ab740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0e2d0 0x5637eeb1f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.999+0000 7f0fc59ab740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0ecc0 0x5637eeb1c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.999+0000 7f0fc59ab740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb13e30 0x5637eeb1c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.999+0000 7f0fc59ab740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637eeb8b7a0 con 0x5637eeb13e30 2024-03-11T19:15:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.999+0000 7f0fc59ab740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637eeb8b8e0 con 0x5637eeb0ecc0 2024-03-11T19:15:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.999+0000 7f0fc59ab740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637eeb8ba20 con 0x5637eeb0e2d0 2024-03-11T19:15:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.999+0000 7f0fc59ab740 10 client.0 _ll_register_callbacks cb 0x7f0fac005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:15:50.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:49.999+0000 7f0fc0b73640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0e2d0 0x5637eeb1f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:50.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.000+0000 7f0fc0b73640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0e2d0 0x5637eeb1f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:49334/0 (socket says 172.21.15.29:49334) 2024-03-11T19:15:50.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.000+0000 7f0fc0b73640 1 -- 172.21.15.29:0/4081196969 learned_addr learned my addr 172.21.15.29:0/4081196969 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:15:50.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.000+0000 7f0fbbfff640 1 --2- 172.21.15.29:0/4081196969 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0ecc0 0x5637eeb1c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:50.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.000+0000 7f0fc1374640 1 --2- 172.21.15.29:0/4081196969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb13e30 0x5637eeb1c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:50.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.000+0000 7f0fc0b73640 1 -- 172.21.15.29:0/4081196969 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0ecc0 msgr2=0x5637eeb1c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:50.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.000+0000 7f0fc0b73640 1 --2- 172.21.15.29:0/4081196969 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0ecc0 0x5637eeb1c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:50.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.000+0000 7f0fc0b73640 1 -- 172.21.15.29:0/4081196969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb13e30 msgr2=0x5637eeb1c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:15:50.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.000+0000 7f0fc0b73640 1 --2- 172.21.15.29:0/4081196969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb13e30 0x5637eeb1c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:50.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.000+0000 7f0fc0b73640 1 -- 172.21.15.29:0/4081196969 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5637eea35570 con 0x5637eeb0e2d0 2024-03-11T19:15:50.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.000+0000 7f0fc0b73640 1 --2- 172.21.15.29:0/4081196969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0e2d0 0x5637eeb1f810 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4005f00 tx=0x7f0fb4009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a993c0f4a0759f4f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:15:50.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.000+0000 7f0f9bfff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:15:50.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.000+0000 7f0f9bfff640 1 -- 172.21.15.29:0/4081196969 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4033450 con 0x5637eeb0e2d0 2024-03-11T19:15:50.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.001+0000 7f0f9bfff640 1 -- 172.21.15.29:0/4081196969 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0fb404f040 con 0x5637eeb0e2d0 2024-03-11T19:15:50.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.001+0000 7f0fc59ab740 1 -- 172.21.15.29:0/4081196969 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0fb40335b0 con 0x5637eeb0e2d0 2024-03-11T19:15:50.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.001+0000 7f0f9bfff640 1 -- 172.21.15.29:0/4081196969 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fb4003b70 con 0x5637eeb0e2d0 2024-03-11T19:15:50.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.001+0000 7f0fc59ab740 10 client.26434 fetch_fsmap learned FSMap version 27 2024-03-11T19:15:50.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.001+0000 7f0fc59ab740 1 -- 172.21.15.29:0/4081196969 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5637eea75610 con 0x5637eeb0e2d0 2024-03-11T19:15:50.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.001+0000 7f0f9bfff640 1 -- 172.21.15.29:0/4081196969 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0fb40335b0 con 0x5637eeb0e2d0 2024-03-11T19:15:50.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.001+0000 7f0f9bfff640 1 -- 172.21.15.29:0/4081196969 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0fb403c050 con 0x5637eeb0e2d0 2024-03-11T19:15:50.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.001+0000 7f0fc59ab740 10 client.26434 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:15:50.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.001+0000 7f0fc59ab740 10 client.26434 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:15:50.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.001+0000 7f0fc59ab740 20 client.26434 populate_metadata read hostname 'smithi029' 2024-03-11T19:15:50.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.001+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/4081196969 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5637eeb8d0e0 0x5637eeb8f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:15:50.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.002+0000 7f0fc59ab740 4 client.26434 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:15:50.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.002+0000 7f0fc59ab740 1 -- 172.21.15.29:0/4081196969 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5637ee802890 con 0x5637eeb8d0e0 2024-03-11T19:15:50.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.002+0000 7f0fbbfff640 1 --2- 172.21.15.29:0/4081196969 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5637eeb8d0e0 0x5637eeb8f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:15:50.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.003+0000 7f0fbbfff640 1 --2- 172.21.15.29:0/4081196969 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5637eeb8d0e0 0x5637eeb8f500 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55cfad7d13d07bc0 server_cookie=262a0c5b0c7acdc0 in_seq=0 out_seq=0 2024-03-11T19:15:50.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.003+0000 7f0f9bfff640 10 client.26434 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:50.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.003+0000 7f0f9bfff640 1 -- 172.21.15.29:0/4081196969 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5637ee802890 con 0x5637eeb8d0e0 2024-03-11T19:15:50.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.003+0000 7f0f9bfff640 10 client.26434 handle_command_reply: tid=0 2024-03-11T19:15:50.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.003+0000 7f0fc59ab740 1 client.26434 shutdown 2024-03-11T19:15:50.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.003+0000 7f0fc59ab740 1 -- 172.21.15.29:0/4081196969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0e2d0 msgr2=0x5637eeb1f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:50.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.003+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/4081196969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0e2d0 0x5637eeb1f810 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4005f00 tx=0x7f0fb4009500 comp rx=0 tx=0).stop 2024-03-11T19:15:50.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.004+0000 7f0fc59ab740 1 -- 172.21.15.29:0/4081196969 shutdown_connections 2024-03-11T19:15:50.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.004+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/4081196969 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5637eeb8d0e0 0x5637eeb8f500 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:50.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.004+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/4081196969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5637eeb13e30 0x5637eeb1c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:50.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.004+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/4081196969 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5637eeb0ecc0 0x5637eeb1c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:50.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.004+0000 7f0fc59ab740 1 --2- 172.21.15.29:0/4081196969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5637eeb0e2d0 0x5637eeb1f810 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:50.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.004+0000 7f0f9bfff640 0 client.26434 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:15:50.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.004+0000 7f0fc59ab740 1 -- 172.21.15.29:0/4081196969 >> 172.21.15.29:0/4081196969 conn(0x5637eeb097f0 msgr2=0x5637eeb18170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:50.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.004+0000 7f0fc59ab740 1 -- 172.21.15.29:0/4081196969 shutdown_connections 2024-03-11T19:15:50.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.004+0000 7f0fc59ab740 1 -- 172.21.15.29:0/4081196969 wait complete. 2024-03-11T19:15:50.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.004+0000 7f0fc59ab740 20 client.26434 trim_cache size 0 max 16384 2024-03-11T19:15:50.010 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:15:50.010 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:15:50.011 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:15:50.011 INFO:teuthology.orchestra.run.smithi029.stdout: "86c5ee58-6dab-4905-8e39-546bd7820164": { 2024-03-11T19:15:50.011 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:15:50.011 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "86c5ee58-6dab-4905-8e39-546bd7820164", 2024-03-11T19:15:50.011 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:15:50.011 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:15:50.011 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:15:50.011 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:15:50.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.005+0000 7f0fc2dd7640 1 -- 172.21.15.29:0/784307187 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0fa40a1b30 msgr2=0x7f0fa40a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:50.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.005+0000 7f0fc2dd7640 1 --2- 172.21.15.29:0/784307187 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0fa40a1b30 0x7f0fa40a3ff0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x5637eeb0f520 tx=0x7f0fb0004730 comp rx=0 tx=0).stop 2024-03-11T19:15:50.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.005+0000 7f0fc2dd7640 1 -- 172.21.15.29:0/784307187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fbc109370 msgr2=0x7f0fbc1a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:15:50.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.005+0000 7f0fc2dd7640 1 --2- 172.21.15.29:0/784307187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fbc109370 0x7f0fbc1a4cf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0fac009fd0 tx=0x7f0fac00eba0 comp rx=0 tx=0).stop 2024-03-11T19:15:50.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.005+0000 7f0fc2dd7640 1 -- 172.21.15.29:0/784307187 shutdown_connections 2024-03-11T19:15:50.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.005+0000 7f0fc2dd7640 1 --2- 172.21.15.29:0/784307187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fbc109370 0x7f0fbc1a4cf0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:50.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.005+0000 7f0fc2dd7640 1 --2- 172.21.15.29:0/784307187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fbc1089b0 0x7f0fbc19dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:50.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.005+0000 7f0fc2dd7640 1 --2- 172.21.15.29:0/784307187 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0fa40a1b30 0x7f0fa40a3ff0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:50.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.005+0000 7f0fc2dd7640 1 --2- 172.21.15.29:0/784307187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fbc1077c0 0x7f0fbc19d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:15:50.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.005+0000 7f0fc2dd7640 1 -- 172.21.15.29:0/784307187 >> 172.21.15.29:0/784307187 conn(0x7f0fbc076770 msgr2=0x7f0fbc10bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:15:50.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.005+0000 7f0fc2dd7640 1 -- 172.21.15.29:0/784307187 shutdown_connections 2024-03-11T19:15:50.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:15:50.005+0000 7f0fc2dd7640 1 -- 172.21.15.29:0/784307187 wait complete. 2024-03-11T19:15:50.025 INFO:tasks.cephfs.filesystem:scrub status for tag:86c5ee58-6dab-4905-8e39-546bd7820164 - {'path': '/', 'tag': '86c5ee58-6dab-4905-8e39-546bd7820164', 'options': 'recursive,force'} 2024-03-11T19:15:51.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:50 smithi029 ceph-mon[32439]: pgmap v532: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T19:15:51.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:50 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:15:51.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:50 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:15:51.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:50 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:15:51.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:50 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:15:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:50 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:15:51.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:50 smithi190 ceph-mon[36292]: pgmap v532: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T19:15:51.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:50 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:15:51.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:50 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:15:51.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:50 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:15:51.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:50 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:15:51.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:50 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:15:51.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:50 smithi170 ceph-mon[35690]: pgmap v532: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T19:15:51.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:50 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:15:51.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:50 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:15:51.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:50 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:15:51.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:50 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:15:51.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:50 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:15:53.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:52 smithi029 ceph-mon[32439]: pgmap v533: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-11T19:15:53.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:52 smithi190 ceph-mon[36292]: pgmap v533: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-11T19:15:53.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:52 smithi170 ceph-mon[35690]: pgmap v533: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-11T19:15:55.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:54 smithi029 ceph-mon[32439]: pgmap v534: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:15:55.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:54 smithi190 ceph-mon[36292]: pgmap v534: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:15:55.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:54 smithi170 ceph-mon[35690]: pgmap v534: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:15:57.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:56 smithi029 ceph-mon[32439]: pgmap v535: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-03-11T19:15:57.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:56 smithi190 ceph-mon[36292]: pgmap v535: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-03-11T19:15:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:56 smithi170 ceph-mon[35690]: pgmap v535: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-03-11T19:15:59.147 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:15:58 smithi170 ceph-mon[35690]: pgmap v536: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-11T19:15:59.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:15:58 smithi029 ceph-mon[32439]: pgmap v536: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-11T19:15:59.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:15:58 smithi190 ceph-mon[36292]: pgmap v536: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-11T19:16:01.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:00 smithi029 ceph-mon[32439]: pgmap v537: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T19:16:01.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:00 smithi190 ceph-mon[36292]: pgmap v537: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T19:16:01.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:00 smithi170 ceph-mon[35690]: pgmap v537: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T19:16:03.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:02 smithi029 ceph-mon[32439]: pgmap v538: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-11T19:16:03.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:02 smithi190 ceph-mon[36292]: pgmap v538: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-11T19:16:03.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:02 smithi170 ceph-mon[35690]: pgmap v538: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-11T19:16:05.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:04 smithi029 ceph-mon[32439]: pgmap v539: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:16:05.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:04 smithi190 ceph-mon[36292]: pgmap v539: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:16:05.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:04 smithi170 ceph-mon[35690]: pgmap v539: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:16:07.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:06 smithi029 ceph-mon[32439]: pgmap v540: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-11T19:16:07.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:06 smithi190 ceph-mon[36292]: pgmap v540: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-11T19:16:07.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:06 smithi170 ceph-mon[35690]: pgmap v540: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-11T19:16:09.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:08 smithi029 ceph-mon[32439]: pgmap v541: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-11T19:16:09.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:08 smithi190 ceph-mon[36292]: pgmap v541: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-11T19:16:09.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:08 smithi170 ceph-mon[35690]: pgmap v541: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-11T19:16:11.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:10 smithi029 ceph-mon[32439]: pgmap v542: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-11T19:16:11.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:10 smithi190 ceph-mon[36292]: pgmap v542: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-11T19:16:11.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:10 smithi170 ceph-mon[35690]: pgmap v542: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-11T19:16:13.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:12 smithi029 ceph-mon[32439]: pgmap v543: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T19:16:13.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:12 smithi190 ceph-mon[36292]: pgmap v543: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T19:16:13.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:12 smithi170 ceph-mon[35690]: pgmap v543: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T19:16:15.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:14 smithi029 ceph-mon[32439]: pgmap v544: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:16:15.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:14 smithi190 ceph-mon[36292]: pgmap v544: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:16:15.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:14 smithi170 ceph-mon[35690]: pgmap v544: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:16:17.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:16 smithi190 ceph-mon[36292]: pgmap v545: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-03-11T19:16:17.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:16 smithi170 ceph-mon[35690]: pgmap v545: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-03-11T19:16:17.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:16 smithi029 ceph-mon[32439]: pgmap v545: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-03-11T19:16:19.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:18 smithi190 ceph-mon[36292]: pgmap v546: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 64 MiB/s wr, 1.04k op/s 2024-03-11T19:16:19.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:18 smithi170 ceph-mon[35690]: pgmap v546: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 64 MiB/s wr, 1.04k op/s 2024-03-11T19:16:19.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:18 smithi029 ceph-mon[32439]: pgmap v546: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 64 MiB/s wr, 1.04k op/s 2024-03-11T19:16:20.026 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:16:20.026 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:16:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.192+0000 7fb703b0c640 1 -- 172.21.15.29:0/1390993575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6fc101270 msgr2=0x7fb6fc1016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.192+0000 7fb703b0c640 1 --2- 172.21.15.29:0/1390993575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6fc101270 0x7fb6fc1016f0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec009900 tx=0x7fb6ec02bc30 comp rx=0 tx=0).stop 2024-03-11T19:16:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.192+0000 7fb703b0c640 1 -- 172.21.15.29:0/1390993575 shutdown_connections 2024-03-11T19:16:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.192+0000 7fb703b0c640 1 --2- 172.21.15.29:0/1390993575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6fc101c30 0x7fb6fc1084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.192+0000 7fb703b0c640 1 --2- 172.21.15.29:0/1390993575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6fc101270 0x7fb6fc1016f0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.192+0000 7fb703b0c640 1 --2- 172.21.15.29:0/1390993575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6fc100080 0x7fb6fc1004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.192+0000 7fb703b0c640 1 -- 172.21.15.29:0/1390993575 >> 172.21.15.29:0/1390993575 conn(0x7fb6fc0fb5f0 msgr2=0x7fb6fc0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:20.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.192+0000 7fb703b0c640 1 -- 172.21.15.29:0/1390993575 shutdown_connections 2024-03-11T19:16:20.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.192+0000 7fb703b0c640 1 -- 172.21.15.29:0/1390993575 wait complete. 2024-03-11T19:16:20.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.193+0000 7fb703b0c640 1 Processor -- start 2024-03-11T19:16:20.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.193+0000 7fb703b0c640 1 -- start start 2024-03-11T19:16:20.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.193+0000 7fb703b0c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6fc100080 0x7fb6fc199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.193+0000 7fb703b0c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6fc101270 0x7fb6fc199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.193+0000 7fb703b0c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6fc101c30 0x7fb6fc1a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.193+0000 7fb703b0c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc1a0c40 con 0x7fb6fc100080 2024-03-11T19:16:20.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.193+0000 7fb703b0c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc1a0db0 con 0x7fb6fc101270 2024-03-11T19:16:20.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.193+0000 7fb703b0c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc1a0f70 con 0x7fb6fc101c30 2024-03-11T19:16:20.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.194+0000 7fb7010a7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6fc101270 0x7fb6fc199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:20.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.194+0000 7fb7010a7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6fc101270 0x7fb6fc199690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44312/0 (socket says 172.21.15.29:44312) 2024-03-11T19:16:20.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.194+0000 7fb7020a9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6fc101c30 0x7fb6fc1a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:20.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.194+0000 7fb7010a7640 1 -- 172.21.15.29:0/2308579903 learned_addr learned my addr 172.21.15.29:0/2308579903 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.194+0000 7fb7020a9640 1 -- 172.21.15.29:0/2308579903 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6fc101270 msgr2=0x7fb6fc199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.194+0000 7fb7020a9640 1 --2- 172.21.15.29:0/2308579903 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6fc101270 0x7fb6fc199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.194+0000 7fb7020a9640 1 -- 172.21.15.29:0/2308579903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6fc100080 msgr2=0x7fb6fc199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.194+0000 7fb7020a9640 1 --2- 172.21.15.29:0/2308579903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6fc100080 0x7fb6fc199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.194+0000 7fb7020a9640 1 -- 172.21.15.29:0/2308579903 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f4009840 con 0x7fb6fc101c30 2024-03-11T19:16:20.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.194+0000 7fb7020a9640 1 --2- 172.21.15.29:0/2308579903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6fc101c30 0x7fb6fc1a0700 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400ec50 tx=0x7fb6f400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:20.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.195+0000 7fb6f2ffd640 1 -- 172.21.15.29:0/2308579903 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4003b70 con 0x7fb6fc101c30 2024-03-11T19:16:20.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.195+0000 7fb703b0c640 1 -- 172.21.15.29:0/2308579903 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ec009560 con 0x7fb6fc101c30 2024-03-11T19:16:20.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.195+0000 7fb703b0c640 1 -- 172.21.15.29:0/2308579903 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6fc1a15d0 con 0x7fb6fc101c30 2024-03-11T19:16:20.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.196+0000 7fb6f2ffd640 1 -- 172.21.15.29:0/2308579903 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb6f400ce70 con 0x7fb6fc101c30 2024-03-11T19:16:20.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.197+0000 7fb6f2ffd640 1 -- 172.21.15.29:0/2308579903 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4010670 con 0x7fb6fc101c30 2024-03-11T19:16:20.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.198+0000 7fb7067c3740 1 Processor -- start 2024-03-11T19:16:20.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.199+0000 7fb7067c3740 1 -- start start 2024-03-11T19:16:20.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.200+0000 7fb7067c3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee0352e0 0x55feee035700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.200+0000 7fb7067c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee035cd0 0x55feee03a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.200+0000 7fb6f2ffd640 1 -- 172.21.15.29:0/2308579903 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb6f4003cd0 con 0x7fb6fc101c30 2024-03-11T19:16:20.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.200+0000 7fb7067c3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee03ae40 0x55feee03d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.200+0000 7fb7067c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feedf94280 con 0x55feee035cd0 2024-03-11T19:16:20.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.200+0000 7fb7067c3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feedf917b0 con 0x55feee03ae40 2024-03-11T19:16:20.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.200+0000 7fb7067c3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feedf8cf90 con 0x55feee0352e0 2024-03-11T19:16:20.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.200+0000 7fb7010a7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee0352e0 0x55feee035700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:20.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.200+0000 7fb7010a7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee0352e0 0x55feee035700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:57838/0 (socket says 172.21.15.29:57838) 2024-03-11T19:16:20.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.200+0000 7fb7018a8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee035cd0 0x55feee03a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:20.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.200+0000 7fb7010a7640 1 -- 172.21.15.29:0/1320646218 learned_addr learned my addr 172.21.15.29:0/1320646218 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:20.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.200+0000 7fb7010a7640 1 --2- 172.21.15.29:0/1320646218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee03ae40 0x55feee03d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:20.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.201+0000 7fb7018a8640 1 -- 172.21.15.29:0/1320646218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee0352e0 msgr2=0x55feee035700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:20.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.201+0000 7fb7018a8640 1 --2- 172.21.15.29:0/1320646218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee0352e0 0x55feee035700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.201+0000 7fb7018a8640 1 -- 172.21.15.29:0/1320646218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee03ae40 msgr2=0x55feee03d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:20.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.201+0000 7fb7018a8640 1 --2- 172.21.15.29:0/1320646218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee03ae40 0x55feee03d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.201+0000 7fb7018a8640 1 -- 172.21.15.29:0/1320646218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55feedf4b570 con 0x55feee035cd0 2024-03-11T19:16:20.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.201+0000 7fb7010a7640 1 --2- 172.21.15.29:0/1320646218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee03ae40 0x55feee03d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:16:20.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.201+0000 7fb7018a8640 1 --2- 172.21.15.29:0/1320646218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee035cd0 0x55feee03a7d0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8009b30 tx=0x7fb6f800e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d37142ce13f9e19 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:20.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.202+0000 7fb6f2ffd640 1 --2- 172.21.15.29:0/2308579903 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb6e80a1b80 0x7fb6e80a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.202+0000 7fb6e7fff640 1 -- 172.21.15.29:0/1320646218 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f800ed80 con 0x55feee035cd0 2024-03-11T19:16:20.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.202+0000 7fb6e7fff640 1 -- 172.21.15.29:0/1320646218 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb6f80054d0 con 0x55feee035cd0 2024-03-11T19:16:20.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.202+0000 7fb6e7fff640 1 -- 172.21.15.29:0/1320646218 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8010690 con 0x55feee035cd0 2024-03-11T19:16:20.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.203+0000 7fb7010a7640 1 --2- 172.21.15.29:0/2308579903 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb6e80a1b80 0x7fb6e80a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:20.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.203+0000 7fb6f2ffd640 1 -- 172.21.15.29:0/2308579903 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb6f4014070 con 0x7fb6fc101c30 2024-03-11T19:16:20.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.203+0000 7fb7010a7640 1 --2- 172.21.15.29:0/2308579903 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb6e80a1b80 0x7fb6e80a4040 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x55feee036530 tx=0x7fb6ec005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:20.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.203+0000 7fb7067c3740 1 -- 172.21.15.29:0/1320646218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee035cd0 msgr2=0x55feee03a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:20.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.203+0000 7fb7067c3740 1 --2- 172.21.15.29:0/1320646218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee035cd0 0x55feee03a7d0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8009b30 tx=0x7fb6f800e350 comp rx=0 tx=0).stop 2024-03-11T19:16:20.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.203+0000 7fb7067c3740 1 -- 172.21.15.29:0/1320646218 shutdown_connections 2024-03-11T19:16:20.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.203+0000 7fb7067c3740 1 --2- 172.21.15.29:0/1320646218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee03ae40 0x55feee03d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.203+0000 7fb7067c3740 1 --2- 172.21.15.29:0/1320646218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee035cd0 0x55feee03a7d0 secure :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8009b30 tx=0x7fb6f800e350 comp rx=0 tx=0).stop 2024-03-11T19:16:20.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.203+0000 7fb7067c3740 1 --2- 172.21.15.29:0/1320646218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee0352e0 0x55feee035700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.203+0000 7fb7067c3740 1 -- 172.21.15.29:0/1320646218 >> 172.21.15.29:0/1320646218 conn(0x55feee01f7f0 msgr2=0x55feee01fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:20.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.204+0000 7fb7067c3740 1 -- 172.21.15.29:0/1320646218 shutdown_connections 2024-03-11T19:16:20.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.204+0000 7fb7067c3740 1 -- 172.21.15.29:0/1320646218 wait complete. 2024-03-11T19:16:20.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.204+0000 7fb7067c3740 1 -- start start 2024-03-11T19:16:20.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.205+0000 7fb7067c3740 1 Processor -- start 2024-03-11T19:16:20.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.205+0000 7fb7067c3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee0352e0 0x55feee047d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.205+0000 7fb7067c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee03ae40 0x55feee048250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.205+0000 7fb7067c3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee04d280 0x55feee04f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.206+0000 7fb7067c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feedc60430 con 0x55feee03ae40 2024-03-11T19:16:20.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.206+0000 7fb7018a8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee0352e0 0x55feee047d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:20.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.206+0000 7fb7010a7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee03ae40 0x55feee048250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:20.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.206+0000 7fb7010a7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee03ae40 0x55feee048250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47086/0 (socket says 172.21.15.29:47086) 2024-03-11T19:16:20.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.206+0000 7fb7010a7640 1 -- 172.21.15.29:0/3845555073 learned_addr learned my addr 172.21.15.29:0/3845555073 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:20.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.206+0000 7fb7020a9640 1 --2- 172.21.15.29:0/3845555073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee04d280 0x55feee04f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:20.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.206+0000 7fb7067c3740 1 -- 172.21.15.29:0/3845555073 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feedc1ce00 con 0x55feee0352e0 2024-03-11T19:16:20.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.206+0000 7fb7067c3740 1 -- 172.21.15.29:0/3845555073 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feedf11fd0 con 0x55feee04d280 2024-03-11T19:16:20.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.206+0000 7fb7067c3740 10 client.0 _ll_register_callbacks cb 0x55feedf67fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:20.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.207+0000 7fb7018a8640 1 -- 172.21.15.29:0/3845555073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee04d280 msgr2=0x55feee04f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:20.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.207+0000 7fb7018a8640 1 --2- 172.21.15.29:0/3845555073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee04d280 0x55feee04f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.207+0000 7fb7018a8640 1 -- 172.21.15.29:0/3845555073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee03ae40 msgr2=0x55feee048250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:20.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.207+0000 7fb7018a8640 1 --2- 172.21.15.29:0/3845555073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee03ae40 0x55feee048250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.207+0000 7fb7018a8640 1 -- 172.21.15.29:0/3845555073 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f8009810 con 0x55feee0352e0 2024-03-11T19:16:20.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.207+0000 7fb7010a7640 1 --2- 172.21.15.29:0/3845555073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee03ae40 0x55feee048250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:16:20.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.207+0000 7fb7018a8640 1 --2- 172.21.15.29:0/3845555073 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee0352e0 0x55feee047d10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55feee036190 tx=0x7fb6f8004ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8b2438a3c1a823e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:20.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.207+0000 7fb6e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:16:20.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.208+0000 7fb6e57fa640 1 -- 172.21.15.29:0/3845555073 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8026070 con 0x55feee0352e0 2024-03-11T19:16:20.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.208+0000 7fb6e57fa640 1 -- 172.21.15.29:0/3845555073 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb6f8014070 con 0x55feee0352e0 2024-03-11T19:16:20.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.208+0000 7fb7067c3740 1 -- 172.21.15.29:0/3845555073 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55feedc8c7e0 con 0x55feee0352e0 2024-03-11T19:16:20.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.208+0000 7fb6e57fa640 1 -- 172.21.15.29:0/3845555073 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6f8004e70 con 0x55feee0352e0 2024-03-11T19:16:20.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.209+0000 7fb7067c3740 10 client.26441 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:20.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.209+0000 7fb7067c3740 1 -- 172.21.15.29:0/3845555073 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55feedf8b610 con 0x55feee0352e0 2024-03-11T19:16:20.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.210+0000 7fb6e57fa640 1 -- 172.21.15.29:0/3845555073 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb6f8005640 con 0x55feee0352e0 2024-03-11T19:16:20.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.210+0000 7fb6e57fa640 1 -- 172.21.15.29:0/3845555073 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb6f8029020 con 0x55feee0352e0 2024-03-11T19:16:20.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.210+0000 7fb7067c3740 10 client.26441 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:20.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.210+0000 7fb7067c3740 10 client.26441 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:16:20.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.210+0000 7fb7067c3740 20 client.26441 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:20.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.210+0000 7fb7067c3740 1 --2- 172.21.15.29:0/3845555073 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55feee053fb0 0x55feee056450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.210+0000 7fb7067c3740 4 client.26441 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:16:20.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.210+0000 7fb7067c3740 1 -- 172.21.15.29:0/3845555073 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55feedc8c7e0 con 0x55feee053fb0 2024-03-11T19:16:20.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.211+0000 7fb7010a7640 1 --2- 172.21.15.29:0/3845555073 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55feee053fb0 0x55feee056450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:20.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.212+0000 7fb7010a7640 1 --2- 172.21.15.29:0/3845555073 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55feee053fb0 0x55feee056450 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e171a04e50071bd server_cookie=4c8c96e591dc63f5 in_seq=0 out_seq=0 2024-03-11T19:16:20.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.212+0000 7fb6e57fa640 10 client.26441 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:20.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.213+0000 7fb6e57fa640 1 -- 172.21.15.29:0/3845555073 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x7fb6ec004040 con 0x55feee053fb0 2024-03-11T19:16:20.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.213+0000 7fb6e57fa640 10 client.26441 handle_command_reply: tid=0 2024-03-11T19:16:20.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.213+0000 7fb7067c3740 1 client.26441 shutdown 2024-03-11T19:16:20.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.214+0000 7fb7067c3740 1 -- 172.21.15.29:0/3845555073 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee0352e0 msgr2=0x55feee047d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:20.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.214+0000 7fb7067c3740 1 --2- 172.21.15.29:0/3845555073 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee0352e0 0x55feee047d10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55feee036190 tx=0x7fb6f8004ba0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.214+0000 7fb7067c3740 1 -- 172.21.15.29:0/3845555073 shutdown_connections 2024-03-11T19:16:20.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.214+0000 7fb7067c3740 1 --2- 172.21.15.29:0/3845555073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee04d280 0x55feee04f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.214+0000 7fb7067c3740 1 --2- 172.21.15.29:0/3845555073 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55feee053fb0 0x55feee056450 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.214+0000 7fb7067c3740 1 --2- 172.21.15.29:0/3845555073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee03ae40 0x55feee048250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.214+0000 7fb7067c3740 1 --2- 172.21.15.29:0/3845555073 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee0352e0 0x55feee047d10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.214+0000 7fb7067c3740 1 -- 172.21.15.29:0/3845555073 >> 172.21.15.29:0/3845555073 conn(0x55feee01f7f0 msgr2=0x55feee01ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:20.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.214+0000 7fb6e57fa640 0 client.26441 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:20.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.214+0000 7fb7067c3740 1 -- 172.21.15.29:0/3845555073 shutdown_connections 2024-03-11T19:16:20.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.214+0000 7fb7067c3740 1 -- 172.21.15.29:0/3845555073 wait complete. 2024-03-11T19:16:20.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.214+0000 7fb7067c3740 20 client.26441 trim_cache size 0 max 16384 2024-03-11T19:16:20.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.244+0000 7fb7067c3740 1 Processor -- start 2024-03-11T19:16:20.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.244+0000 7fb7067c3740 1 -- start start 2024-03-11T19:16:20.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.244+0000 7fb7067c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee0352e0 0x55feee064330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.244+0000 7fb7067c3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee03ae40 0x55feee064870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.244+0000 7fb7067c3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee04d280 0x55feee01e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.244+0000 7fb7067c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feee01aeb0 con 0x55feee0352e0 2024-03-11T19:16:20.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.244+0000 7fb7067c3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feee034f80 con 0x55feee03ae40 2024-03-11T19:16:20.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.244+0000 7fb7067c3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feee048f00 con 0x55feee04d280 2024-03-11T19:16:20.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.244+0000 7fb7018a8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee0352e0 0x55feee064330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:20.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.244+0000 7fb7018a8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee0352e0 0x55feee064330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47096/0 (socket says 172.21.15.29:47096) 2024-03-11T19:16:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.244+0000 7fb7018a8640 1 -- 172.21.15.29:0/1425225343 learned_addr learned my addr 172.21.15.29:0/1425225343 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.244+0000 7fb7020a9640 1 --2- 172.21.15.29:0/1425225343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee04d280 0x55feee01e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.245+0000 7fb7018a8640 1 -- 172.21.15.29:0/1425225343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee04d280 msgr2=0x55feee01e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.245+0000 7fb7018a8640 1 --2- 172.21.15.29:0/1425225343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee04d280 0x55feee01e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.245+0000 7fb7018a8640 1 -- 172.21.15.29:0/1425225343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee03ae40 msgr2=0x55feee064870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:16:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.245+0000 7fb7018a8640 1 --2- 172.21.15.29:0/1425225343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee03ae40 0x55feee064870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.245+0000 7fb7018a8640 1 -- 172.21.15.29:0/1425225343 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55feedf27ac0 con 0x55feee0352e0 2024-03-11T19:16:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.245+0000 7fb7018a8640 1 --2- 172.21.15.29:0/1425225343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee0352e0 0x55feee064330 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8010540 tx=0x7fb6f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe184bdc61db9a03 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.245+0000 7fb6e77fe640 1 -- 172.21.15.29:0/1425225343 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8026070 con 0x55feee0352e0 2024-03-11T19:16:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.245+0000 7fb6e77fe640 1 -- 172.21.15.29:0/1425225343 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb6f80049b0 con 0x55feee0352e0 2024-03-11T19:16:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.245+0000 7fb6e77fe640 1 -- 172.21.15.29:0/1425225343 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8004e70 con 0x55feee0352e0 2024-03-11T19:16:20.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.246+0000 7fb7067c3740 1 -- 172.21.15.29:0/1425225343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee0352e0 msgr2=0x55feee064330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:20.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.246+0000 7fb7067c3740 1 --2- 172.21.15.29:0/1425225343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee0352e0 0x55feee064330 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8010540 tx=0x7fb6f80046c0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.246+0000 7fb7067c3740 1 -- 172.21.15.29:0/1425225343 shutdown_connections 2024-03-11T19:16:20.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.246+0000 7fb7067c3740 1 --2- 172.21.15.29:0/1425225343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee04d280 0x55feee01e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.246+0000 7fb7067c3740 1 --2- 172.21.15.29:0/1425225343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee03ae40 0x55feee064870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.246+0000 7fb7067c3740 1 --2- 172.21.15.29:0/1425225343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee0352e0 0x55feee064330 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.246+0000 7fb7067c3740 1 -- 172.21.15.29:0/1425225343 >> 172.21.15.29:0/1425225343 conn(0x55feee01f7f0 msgr2=0x55feee037150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:20.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.247+0000 7fb7067c3740 1 -- 172.21.15.29:0/1425225343 shutdown_connections 2024-03-11T19:16:20.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.247+0000 7fb7067c3740 1 -- 172.21.15.29:0/1425225343 wait complete. 2024-03-11T19:16:20.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.248+0000 7fb7067c3740 1 -- start start 2024-03-11T19:16:20.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.248+0000 7fb7067c3740 1 Processor -- start 2024-03-11T19:16:20.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.248+0000 7fb7067c3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee0352e0 0x55feee0a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.248+0000 7fb7067c3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee03ae40 0x55feee0a7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.248+0000 7fb7067c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee04d280 0x55feee0a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.248+0000 7fb7067c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feee0a9fb0 con 0x55feee04d280 2024-03-11T19:16:20.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.248+0000 7fb7067c3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feee0aa0f0 con 0x55feee0352e0 2024-03-11T19:16:20.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.248+0000 7fb7067c3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feee0aa2b0 con 0x55feee03ae40 2024-03-11T19:16:20.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.248+0000 7fb7067c3740 10 client.0 _ll_register_callbacks cb 0x7fb6f8009590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:20.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.249+0000 7fb7010a7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee03ae40 0x55feee0a7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:20.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.249+0000 7fb7020a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee04d280 0x55feee0a9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:20.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.249+0000 7fb7010a7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee03ae40 0x55feee0a7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:57860/0 (socket says 172.21.15.29:57860) 2024-03-11T19:16:20.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.249+0000 7fb7010a7640 1 -- 172.21.15.29:0/2644037230 learned_addr learned my addr 172.21.15.29:0/2644037230 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.249+0000 7fb7020a9640 1 -- 172.21.15.29:0/2644037230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee03ae40 msgr2=0x55feee0a7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.249+0000 7fb7020a9640 1 --2- 172.21.15.29:0/2644037230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee03ae40 0x55feee0a7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.249+0000 7fb7020a9640 1 -- 172.21.15.29:0/2644037230 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee0352e0 msgr2=0x55feee0a4fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:16:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.249+0000 7fb7020a9640 1 --2- 172.21.15.29:0/2644037230 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee0352e0 0x55feee0a4fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.249+0000 7fb7020a9640 1 -- 172.21.15.29:0/2644037230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55feedf8b610 con 0x55feee04d280 2024-03-11T19:16:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.250+0000 7fb7020a9640 1 --2- 172.21.15.29:0/2644037230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee04d280 0x55feee0a9a70 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400cba0 tx=0x7fb6f409b220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac46d1ac89c69f44 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.250+0000 7fb6e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:16:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.251+0000 7fb6e57fa640 1 -- 172.21.15.29:0/2644037230 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f40b3450 con 0x55feee04d280 2024-03-11T19:16:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.251+0000 7fb6e57fa640 1 -- 172.21.15.29:0/2644037230 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb6f4004b40 con 0x55feee04d280 2024-03-11T19:16:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.251+0000 7fb7067c3740 1 -- 172.21.15.29:0/2644037230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55feedcc9720 con 0x55feee04d280 2024-03-11T19:16:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.251+0000 7fb6e57fa640 1 -- 172.21.15.29:0/2644037230 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6f4004d00 con 0x55feee04d280 2024-03-11T19:16:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.252+0000 7fb7067c3740 10 client.16932 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.252+0000 7fb7067c3740 1 -- 172.21.15.29:0/2644037230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55feedf4d6a0 con 0x55feee04d280 2024-03-11T19:16:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.252+0000 7fb6e57fa640 1 -- 172.21.15.29:0/2644037230 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb6f407f080 con 0x55feee04d280 2024-03-11T19:16:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.253+0000 7fb6e57fa640 1 -- 172.21.15.29:0/2644037230 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb6f401f060 con 0x55feee04d280 2024-03-11T19:16:20.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.254+0000 7fb7067c3740 10 client.16932 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:20.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.254+0000 7fb7067c3740 10 client.16932 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:16:20.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.254+0000 7fb7067c3740 20 client.16932 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:20.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.254+0000 7fb7067c3740 1 --2- 172.21.15.29:0/2644037230 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55feee0abd90 0x55feee0ae1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.254+0000 7fb7067c3740 4 client.16932 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:16:20.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.254+0000 7fb7067c3740 1 -- 172.21.15.29:0/2644037230 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55feedd18890 con 0x55feee0abd90 2024-03-11T19:16:20.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.256+0000 7fb7018a8640 1 --2- 172.21.15.29:0/2644037230 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55feee0abd90 0x55feee0ae1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:20.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.257+0000 7fb7018a8640 1 --2- 172.21.15.29:0/2644037230 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55feee0abd90 0x55feee0ae1b0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4210151753ae29c server_cookie=41e33dc5d44d35c5 in_seq=0 out_seq=0 2024-03-11T19:16:20.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.257+0000 7fb6e57fa640 10 client.16932 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:20.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.258+0000 7fb6e57fa640 1 -- 172.21.15.29:0/2644037230 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55feedd18890 con 0x55feee0abd90 2024-03-11T19:16:20.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.258+0000 7fb6e57fa640 10 client.16932 handle_command_reply: tid=0 2024-03-11T19:16:20.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.258+0000 7fb7067c3740 1 client.16932 shutdown 2024-03-11T19:16:20.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.258+0000 7fb7067c3740 1 -- 172.21.15.29:0/2644037230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee04d280 msgr2=0x55feee0a9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:20.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.258+0000 7fb7067c3740 1 --2- 172.21.15.29:0/2644037230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee04d280 0x55feee0a9a70 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400cba0 tx=0x7fb6f409b220 comp rx=0 tx=0).stop 2024-03-11T19:16:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.258+0000 7fb7067c3740 1 -- 172.21.15.29:0/2644037230 shutdown_connections 2024-03-11T19:16:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.258+0000 7fb7067c3740 1 --2- 172.21.15.29:0/2644037230 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55feee0abd90 0x55feee0ae1b0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.258+0000 7fb7067c3740 1 --2- 172.21.15.29:0/2644037230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55feee04d280 0x55feee0a9a70 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.258+0000 7fb7067c3740 1 --2- 172.21.15.29:0/2644037230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55feee03ae40 0x55feee0a7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.258+0000 7fb7067c3740 1 --2- 172.21.15.29:0/2644037230 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55feee0352e0 0x55feee0a4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.258+0000 7fb7067c3740 1 -- 172.21.15.29:0/2644037230 >> 172.21.15.29:0/2644037230 conn(0x55feee01f7f0 msgr2=0x55feee022aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:20.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.258+0000 7fb6e57fa640 0 client.16932 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:20.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.258+0000 7fb7067c3740 1 -- 172.21.15.29:0/2644037230 shutdown_connections 2024-03-11T19:16:20.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.258+0000 7fb7067c3740 1 -- 172.21.15.29:0/2644037230 wait complete. 2024-03-11T19:16:20.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.258+0000 7fb7067c3740 20 client.16932 trim_cache size 0 max 16384 2024-03-11T19:16:20.263 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:16:20.263 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:16:20.263 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:16:20.263 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:16:20.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.260+0000 7fb703b0c640 1 -- 172.21.15.29:0/2308579903 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb6e80a1b80 msgr2=0x7fb6e80a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:20.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.260+0000 7fb703b0c640 1 --2- 172.21.15.29:0/2308579903 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb6e80a1b80 0x7fb6e80a4040 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x55feee036530 tx=0x7fb6ec005fd0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.260+0000 7fb703b0c640 1 -- 172.21.15.29:0/2308579903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6fc101c30 msgr2=0x7fb6fc1a0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:20.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.260+0000 7fb703b0c640 1 --2- 172.21.15.29:0/2308579903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6fc101c30 0x7fb6fc1a0700 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400ec50 tx=0x7fb6f400c6a0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.261+0000 7fb703b0c640 1 -- 172.21.15.29:0/2308579903 shutdown_connections 2024-03-11T19:16:20.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.261+0000 7fb703b0c640 1 --2- 172.21.15.29:0/2308579903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6fc101c30 0x7fb6fc1a0700 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.261+0000 7fb703b0c640 1 --2- 172.21.15.29:0/2308579903 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6fc101270 0x7fb6fc199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.261+0000 7fb703b0c640 1 --2- 172.21.15.29:0/2308579903 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb6e80a1b80 0x7fb6e80a4040 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.261+0000 7fb703b0c640 1 --2- 172.21.15.29:0/2308579903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6fc100080 0x7fb6fc199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.261+0000 7fb703b0c640 1 -- 172.21.15.29:0/2308579903 >> 172.21.15.29:0/2308579903 conn(0x7fb6fc0fb5f0 msgr2=0x7fb6fc0682e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:20.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.261+0000 7fb703b0c640 1 -- 172.21.15.29:0/2308579903 shutdown_connections 2024-03-11T19:16:20.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.261+0000 7fb703b0c640 1 -- 172.21.15.29:0/2308579903 wait complete. 2024-03-11T19:16:20.282 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:16:20.283 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:16:20.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.455+0000 7fac4b8eb640 1 -- 172.21.15.29:0/2698729838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac44101c50 msgr2=0x7fac441084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:20.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.455+0000 7fac4b8eb640 1 --2- 172.21.15.29:0/2698729838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac44101c50 0x7fac441084e0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fac40009860 tx=0x7fac4002bb70 comp rx=0 tx=0).stop 2024-03-11T19:16:20.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.455+0000 7fac4b8eb640 1 -- 172.21.15.29:0/2698729838 shutdown_connections 2024-03-11T19:16:20.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.455+0000 7fac4b8eb640 1 --2- 172.21.15.29:0/2698729838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac44101c50 0x7fac441084e0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.455+0000 7fac4b8eb640 1 --2- 172.21.15.29:0/2698729838 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac44101290 0x7fac44101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.455+0000 7fac4b8eb640 1 --2- 172.21.15.29:0/2698729838 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac441000a0 0x7fac441004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.455+0000 7fac4b8eb640 1 -- 172.21.15.29:0/2698729838 >> 172.21.15.29:0/2698729838 conn(0x7fac440fb610 msgr2=0x7fac440fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:20.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.455+0000 7fac4b8eb640 1 -- 172.21.15.29:0/2698729838 shutdown_connections 2024-03-11T19:16:20.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.455+0000 7fac4b8eb640 1 -- 172.21.15.29:0/2698729838 wait complete. 2024-03-11T19:16:20.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.456+0000 7fac4b8eb640 1 Processor -- start 2024-03-11T19:16:20.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.456+0000 7fac4b8eb640 1 -- start start 2024-03-11T19:16:20.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.456+0000 7fac4b8eb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac441000a0 0x7fac4410e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.456+0000 7fac4b8eb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac44101290 0x7fac4410eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.456+0000 7fac4b8eb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac44101c50 0x7fac4410b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.456+0000 7fac4b8eb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4410b8a0 con 0x7fac441000a0 2024-03-11T19:16:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.456+0000 7fac4b8eb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4410ba10 con 0x7fac44101290 2024-03-11T19:16:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.456+0000 7fac4b8eb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4410bbd0 con 0x7fac44101c50 2024-03-11T19:16:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.457+0000 7fac49e88640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac44101c50 0x7fac4410b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.457+0000 7fac49e88640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac44101c50 0x7fac4410b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:57884/0 (socket says 172.21.15.29:57884) 2024-03-11T19:16:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.457+0000 7fac49e88640 1 -- 172.21.15.29:0/2116084448 learned_addr learned my addr 172.21.15.29:0/2116084448 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.457+0000 7fac48e86640 1 --2- 172.21.15.29:0/2116084448 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac44101290 0x7fac4410eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.457+0000 7fac49687640 1 --2- 172.21.15.29:0/2116084448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac441000a0 0x7fac4410e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.457+0000 7fac49e88640 1 -- 172.21.15.29:0/2116084448 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac44101290 msgr2=0x7fac4410eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.457+0000 7fac49e88640 1 --2- 172.21.15.29:0/2116084448 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac44101290 0x7fac4410eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.457+0000 7fac49e88640 1 -- 172.21.15.29:0/2116084448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac441000a0 msgr2=0x7fac4410e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.457+0000 7fac49e88640 1 --2- 172.21.15.29:0/2116084448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac441000a0 0x7fac4410e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.457+0000 7fac49e88640 1 -- 172.21.15.29:0/2116084448 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac40009490 con 0x7fac44101c50 2024-03-11T19:16:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.458+0000 7fac49e88640 1 --2- 172.21.15.29:0/2116084448 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac44101c50 0x7fac4410b110 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fac40009780 tx=0x7fac400324a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:20.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.458+0000 7fac3a7fc640 1 -- 172.21.15.29:0/2116084448 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4002c510 con 0x7fac44101c50 2024-03-11T19:16:20.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.458+0000 7fac4b8eb640 1 -- 172.21.15.29:0/2116084448 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac441a57d0 con 0x7fac44101c50 2024-03-11T19:16:20.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.458+0000 7fac3a7fc640 1 -- 172.21.15.29:0/2116084448 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fac4002ce90 con 0x7fac44101c50 2024-03-11T19:16:20.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.458+0000 7fac4b8eb640 1 -- 172.21.15.29:0/2116084448 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac441a5d30 con 0x7fac44101c50 2024-03-11T19:16:20.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.458+0000 7fac3a7fc640 1 -- 172.21.15.29:0/2116084448 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac400035f0 con 0x7fac44101c50 2024-03-11T19:16:20.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.460+0000 7fac4b8eb640 1 -- 172.21.15.29:0/2116084448 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac44067f70 con 0x7fac44101c50 2024-03-11T19:16:20.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.461+0000 7fac3a7fc640 1 -- 172.21.15.29:0/2116084448 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fac4003b040 con 0x7fac44101c50 2024-03-11T19:16:20.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.462+0000 7fac3a7fc640 1 --2- 172.21.15.29:0/2116084448 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fac300a1b00 0x7fac300a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:20.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.462+0000 7fac49687640 1 --2- 172.21.15.29:0/2116084448 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fac300a1b00 0x7fac300a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:20.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.463+0000 7fac3a7fc640 1 -- 172.21.15.29:0/2116084448 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fac400eef10 con 0x7fac44101c50 2024-03-11T19:16:20.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.463+0000 7fac49687640 1 --2- 172.21.15.29:0/2116084448 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fac300a1b00 0x7fac300a3fc0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fac441010f0 tx=0x7fac34005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:20.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.467+0000 7fac3a7fc640 1 -- 172.21.15.29:0/2116084448 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fac400b8120 con 0x7fac44101c50 2024-03-11T19:16:20.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.895+0000 7fac4b8eb640 1 -- 172.21.15.29:0/2116084448 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac441073c0 con 0x7fac44101c50 2024-03-11T19:16:20.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.897+0000 7fac3a7fc640 1 -- 172.21.15.29:0/2116084448 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7fac40041090 con 0x7fac44101c50 2024-03-11T19:16:20.900 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:16:20.901 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:16:20.901 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:16:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.900+0000 7fac4b8eb640 1 -- 172.21.15.29:0/2116084448 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fac300a1b00 msgr2=0x7fac300a3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.901+0000 7fac4b8eb640 1 --2- 172.21.15.29:0/2116084448 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fac300a1b00 0x7fac300a3fc0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fac441010f0 tx=0x7fac34005f90 comp rx=0 tx=0).stop 2024-03-11T19:16:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.901+0000 7fac4b8eb640 1 -- 172.21.15.29:0/2116084448 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac44101c50 msgr2=0x7fac4410b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.901+0000 7fac4b8eb640 1 --2- 172.21.15.29:0/2116084448 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac44101c50 0x7fac4410b110 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fac40009780 tx=0x7fac400324a0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.902+0000 7fac4b8eb640 1 -- 172.21.15.29:0/2116084448 shutdown_connections 2024-03-11T19:16:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.902+0000 7fac4b8eb640 1 --2- 172.21.15.29:0/2116084448 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac44101c50 0x7fac4410b110 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.902+0000 7fac4b8eb640 1 --2- 172.21.15.29:0/2116084448 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac44101290 0x7fac4410eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.902+0000 7fac4b8eb640 1 --2- 172.21.15.29:0/2116084448 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fac300a1b00 0x7fac300a3fc0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.902+0000 7fac4b8eb640 1 --2- 172.21.15.29:0/2116084448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac441000a0 0x7fac4410e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.902+0000 7fac4b8eb640 1 -- 172.21.15.29:0/2116084448 >> 172.21.15.29:0/2116084448 conn(0x7fac440fb610 msgr2=0x7fac44104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.903+0000 7fac4b8eb640 1 -- 172.21.15.29:0/2116084448 shutdown_connections 2024-03-11T19:16:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:20.903+0000 7fac4b8eb640 1 -- 172.21.15.29:0/2116084448 wait complete. 2024-03-11T19:16:20.925 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:16:20.925 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:16:21.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.044+0000 7fc4e13c9640 1 -- 172.21.15.29:0/4098981048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4dc102d40 msgr2=0x7fc4dc1095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.044+0000 7fc4e13c9640 1 --2- 172.21.15.29:0/4098981048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4dc102d40 0x7fc4dc1095d0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4009ca0 tx=0x7fc4d402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.044+0000 7fc4e13c9640 1 -- 172.21.15.29:0/4098981048 shutdown_connections 2024-03-11T19:16:21.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.044+0000 7fc4e13c9640 1 --2- 172.21.15.29:0/4098981048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4dc102d40 0x7fc4dc1095d0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.044+0000 7fc4e13c9640 1 --2- 172.21.15.29:0/4098981048 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4dc102380 0x7fc4dc102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.044+0000 7fc4e13c9640 1 --2- 172.21.15.29:0/4098981048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4dc101190 0x7fc4dc1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.044+0000 7fc4e13c9640 1 -- 172.21.15.29:0/4098981048 >> 172.21.15.29:0/4098981048 conn(0x7fc4dc0fc720 msgr2=0x7fc4dc0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.044+0000 7fc4e13c9640 1 -- 172.21.15.29:0/4098981048 shutdown_connections 2024-03-11T19:16:21.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.044+0000 7fc4e13c9640 1 -- 172.21.15.29:0/4098981048 wait complete. 2024-03-11T19:16:21.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.044+0000 7fc4e13c9640 1 Processor -- start 2024-03-11T19:16:21.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.044+0000 7fc4e13c9640 1 -- start start 2024-03-11T19:16:21.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4e13c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4dc101190 0x7fc4dc1970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4e13c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4dc102380 0x7fc4dc197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4e13c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4dc102d40 0x7fc4dc19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4e13c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc19ebd0 con 0x7fc4dc101190 2024-03-11T19:16:21.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4e13c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc19ed40 con 0x7fc4dc102380 2024-03-11T19:16:21.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4e13c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc19ef00 con 0x7fc4dc102d40 2024-03-11T19:16:21.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4daffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4dc101190 0x7fc4dc1970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4daffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4dc101190 0x7fc4dc1970e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47146/0 (socket says 172.21.15.29:47146) 2024-03-11T19:16:21.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4daffd640 1 -- 172.21.15.29:0/3486487817 learned_addr learned my addr 172.21.15.29:0/3486487817 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4db7fe640 1 --2- 172.21.15.29:0/3486487817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4dc102d40 0x7fc4dc19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4da7fc640 1 --2- 172.21.15.29:0/3486487817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4dc102380 0x7fc4dc197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4daffd640 1 -- 172.21.15.29:0/3486487817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4dc102d40 msgr2=0x7fc4dc19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4daffd640 1 --2- 172.21.15.29:0/3486487817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4dc102d40 0x7fc4dc19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4daffd640 1 -- 172.21.15.29:0/3486487817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4dc102380 msgr2=0x7fc4dc197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4daffd640 1 --2- 172.21.15.29:0/3486487817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4dc102380 0x7fc4dc197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4daffd640 1 -- 172.21.15.29:0/3486487817 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d4009900 con 0x7fc4dc101190 2024-03-11T19:16:21.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4daffd640 1 --2- 172.21.15.29:0/3486487817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4dc101190 0x7fc4dc1970e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc00e9c0 tx=0x7fc4cc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4c3fff640 1 -- 172.21.15.29:0/3486487817 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc009700 con 0x7fc4dc101190 2024-03-11T19:16:21.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4c3fff640 1 -- 172.21.15.29:0/3486487817 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc4cc0044d0 con 0x7fc4dc101190 2024-03-11T19:16:21.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4c3fff640 1 -- 172.21.15.29:0/3486487817 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc010430 con 0x7fc4dc101190 2024-03-11T19:16:21.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4e13c9640 1 -- 172.21.15.29:0/3486487817 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4dc19f1e0 con 0x7fc4dc101190 2024-03-11T19:16:21.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.045+0000 7fc4e13c9640 1 -- 172.21.15.29:0/3486487817 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4dc19f6f0 con 0x7fc4dc101190 2024-03-11T19:16:21.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.047+0000 7fc4c3fff640 1 -- 172.21.15.29:0/3486487817 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc4cc003f30 con 0x7fc4dc101190 2024-03-11T19:16:21.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.048+0000 7fc4e3fa0740 1 Processor -- start 2024-03-11T19:16:21.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.048+0000 7fc4e3fa0740 1 -- start start 2024-03-11T19:16:21.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.048+0000 7fc4e3fa0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6e270 0x5566fbb6e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.048+0000 7fc4e3fa0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb6ec60 0x5566fbb73760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.048+0000 7fc4e3fa0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb73dd0 0x5566fbb76260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.048+0000 7fc4e3fa0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566fbb2a960 con 0x5566fbb73dd0 2024-03-11T19:16:21.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.048+0000 7fc4e3fa0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566fbade310 con 0x5566fbb6e270 2024-03-11T19:16:21.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.048+0000 7fc4e3fa0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566fbadb840 con 0x5566fbb6ec60 2024-03-11T19:16:21.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.048+0000 7fc4da7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb73dd0 0x5566fbb76260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.048+0000 7fc4daffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb6ec60 0x5566fbb73760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.048+0000 7fc4c3fff640 1 --2- 172.21.15.29:0/3486487817 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc4c40a1a30 0x7fc4c40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.048+0000 7fc4db7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6e270 0x5566fbb6e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.048+0000 7fc4da7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb73dd0 0x5566fbb76260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47150/0 (socket says 172.21.15.29:47150) 2024-03-11T19:16:21.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.048+0000 7fc4da7fc640 1 -- 172.21.15.29:0/3503943265 learned_addr learned my addr 172.21.15.29:0/3503943265 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.048+0000 7fc4daffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb6ec60 0x5566fbb73760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:57910/0 (socket says 172.21.15.29:57910) 2024-03-11T19:16:21.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.048+0000 7fc4c3fff640 1 -- 172.21.15.29:0/3486487817 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fc4cc014070 con 0x7fc4dc101190 2024-03-11T19:16:21.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.048+0000 7fc4da7fc640 1 -- 172.21.15.29:0/3503943265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb6ec60 msgr2=0x5566fbb73760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.048+0000 7fc4da7fc640 1 --2- 172.21.15.29:0/3503943265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb6ec60 0x5566fbb73760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.048+0000 7fc4da7fc640 1 -- 172.21.15.29:0/3503943265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6e270 msgr2=0x5566fbb6e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.049+0000 7fc4da7fc640 1 --2- 172.21.15.29:0/3503943265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6e270 0x5566fbb6e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.049+0000 7fc4db7fe640 1 --2- 172.21.15.29:0/3486487817 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc4c40a1a30 0x7fc4c40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.049+0000 7fc4da7fc640 1 -- 172.21.15.29:0/3503943265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566fba99060 con 0x5566fbb73dd0 2024-03-11T19:16:21.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.049+0000 7fc4da7fc640 1 --2- 172.21.15.29:0/3503943265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb73dd0 0x5566fbb76260 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009c30 tx=0x7fc4d00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b29b127afb50b14 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.049+0000 7fc4db7fe640 1 --2- 172.21.15.29:0/3486487817 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc4c40a1a30 0x7fc4c40a3ef0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4005e30 tx=0x7fc4d402c0f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.049+0000 7fc4c17fa640 1 -- 172.21.15.29:0/3503943265 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0009700 con 0x5566fbb73dd0 2024-03-11T19:16:21.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.049+0000 7fc4c17fa640 1 -- 172.21.15.29:0/3503943265 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc4d0012900 con 0x5566fbb73dd0 2024-03-11T19:16:21.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.049+0000 7fc4c17fa640 1 -- 172.21.15.29:0/3503943265 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d000fa40 con 0x5566fbb73dd0 2024-03-11T19:16:21.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/3503943265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb73dd0 msgr2=0x5566fbb76260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/3503943265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb73dd0 0x5566fbb76260 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009c30 tx=0x7fc4d00117b0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/3503943265 shutdown_connections 2024-03-11T19:16:21.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/3503943265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb73dd0 0x5566fbb76260 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/3503943265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb6ec60 0x5566fbb73760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/3503943265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6e270 0x5566fbb6e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/3503943265 >> 172.21.15.29:0/3503943265 conn(0x5566fbb697e0 msgr2=0x5566fbb69be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/3503943265 shutdown_connections 2024-03-11T19:16:21.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/3503943265 wait complete. 2024-03-11T19:16:21.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4e3fa0740 1 -- start start 2024-03-11T19:16:21.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4e3fa0740 1 Processor -- start 2024-03-11T19:16:21.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4e3fa0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6e270 0x5566fbb78450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4e3fa0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb6ec60 0x5566fbb78990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4e3fa0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb73dd0 0x5566fbb7f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4e3fa0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566fbad7020 con 0x5566fbb6ec60 2024-03-11T19:16:21.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4e3fa0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566fb7aa430 con 0x5566fbb6e270 2024-03-11T19:16:21.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4e3fa0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566fb766e00 con 0x5566fbb73dd0 2024-03-11T19:16:21.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4e3fa0740 10 client.0 _ll_register_callbacks cb 0x5566fba7ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:21.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4da7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb6ec60 0x5566fbb78990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4db7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb73dd0 0x5566fbb7f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4da7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb6ec60 0x5566fbb78990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47164/0 (socket says 172.21.15.29:47164) 2024-03-11T19:16:21.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4db7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb73dd0 0x5566fbb7f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:57926/0 (socket says 172.21.15.29:57926) 2024-03-11T19:16:21.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.050+0000 7fc4da7fc640 1 -- 172.21.15.29:0/1036347184 learned_addr learned my addr 172.21.15.29:0/1036347184 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.051+0000 7fc4daffd640 1 --2- 172.21.15.29:0/1036347184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6e270 0x5566fbb78450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.051+0000 7fc4da7fc640 1 -- 172.21.15.29:0/1036347184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb73dd0 msgr2=0x5566fbb7f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.051+0000 7fc4da7fc640 1 --2- 172.21.15.29:0/1036347184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb73dd0 0x5566fbb7f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.051+0000 7fc4da7fc640 1 -- 172.21.15.29:0/1036347184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6e270 msgr2=0x5566fbb78450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.051+0000 7fc4da7fc640 1 --2- 172.21.15.29:0/1036347184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6e270 0x5566fbb78450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.051+0000 7fc4da7fc640 1 -- 172.21.15.29:0/1036347184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d0009860 con 0x5566fbb6ec60 2024-03-11T19:16:21.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.051+0000 7fc4db7fe640 1 --2- 172.21.15.29:0/1036347184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb73dd0 0x5566fbb7f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:16:21.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.051+0000 7fc4da7fc640 1 --2- 172.21.15.29:0/1036347184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb6ec60 0x5566fbb78990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009c00 tx=0x7fc4d00127a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=545477c419b547c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.051+0000 7fc4beff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:16:21.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.051+0000 7fc4beff5640 1 -- 172.21.15.29:0/1036347184 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d000f780 con 0x5566fbb6ec60 2024-03-11T19:16:21.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.051+0000 7fc4beff5640 1 -- 172.21.15.29:0/1036347184 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fc4d000f8e0 con 0x5566fbb6ec60 2024-03-11T19:16:21.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.051+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/1036347184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5566fb7d67e0 con 0x5566fbb6ec60 2024-03-11T19:16:21.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.051+0000 7fc4beff5640 1 -- 172.21.15.29:0/1036347184 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4d000faa0 con 0x5566fbb6ec60 2024-03-11T19:16:21.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.051+0000 7fc4e3fa0740 10 client.16950 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:21.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.051+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/1036347184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5566fbad56a0 con 0x5566fbb6ec60 2024-03-11T19:16:21.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.051+0000 7fc4beff5640 1 -- 172.21.15.29:0/1036347184 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5566fb7d67e0 con 0x5566fbb6ec60 2024-03-11T19:16:21.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.052+0000 7fc4beff5640 1 -- 172.21.15.29:0/1036347184 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc4d0022040 con 0x5566fbb6ec60 2024-03-11T19:16:21.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.052+0000 7fc4e3fa0740 10 client.16950 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:21.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.052+0000 7fc4e3fa0740 10 client.16950 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:16:21.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.052+0000 7fc4e3fa0740 20 client.16950 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:21.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.052+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/1036347184 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5566fbb84570 0x5566fbb86990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.052+0000 7fc4e3fa0740 4 client.16950 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:16:21.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.052+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/1036347184 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5566fb813720 con 0x5566fbb84570 2024-03-11T19:16:21.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.052+0000 7fc4daffd640 1 --2- 172.21.15.29:0/1036347184 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5566fbb84570 0x5566fbb86990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.052+0000 7fc4beff5640 10 client.16950 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:16:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.052+0000 7fc4daffd640 1 --2- 172.21.15.29:0/1036347184 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5566fbb84570 0x5566fbb86990 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=75b1c477fcedf3a8 server_cookie=4c95edf6dc211ab in_seq=0 out_seq=0 2024-03-11T19:16:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.053+0000 7fc4beff5640 1 -- 172.21.15.29:0/1036347184 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5566fb813720 con 0x5566fbb84570 2024-03-11T19:16:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.053+0000 7fc4beff5640 10 client.16950 handle_command_reply: tid=0 2024-03-11T19:16:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.053+0000 7fc4e3fa0740 1 client.16950 shutdown 2024-03-11T19:16:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.053+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/1036347184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb6ec60 msgr2=0x5566fbb78990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.053+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/1036347184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb6ec60 0x5566fbb78990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009c00 tx=0x7fc4d00127a0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.053+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/1036347184 shutdown_connections 2024-03-11T19:16:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.053+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/1036347184 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5566fbb84570 0x5566fbb86990 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.053+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/1036347184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb73dd0 0x5566fbb7f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.053+0000 7fc4beff5640 0 client.16950 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:16:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.053+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/1036347184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb6ec60 0x5566fbb78990 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.053+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/1036347184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6e270 0x5566fbb78450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.053+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/1036347184 >> 172.21.15.29:0/1036347184 conn(0x5566fbb697e0 msgr2=0x5566fbb68b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.053+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/1036347184 shutdown_connections 2024-03-11T19:16:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.053+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/1036347184 wait complete. 2024-03-11T19:16:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.053+0000 7fc4e3fa0740 20 client.16950 trim_cache size 0 max 16384 2024-03-11T19:16:21.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.064+0000 7fc4e3fa0740 1 Processor -- start 2024-03-11T19:16:21.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4e3fa0740 1 -- start start 2024-03-11T19:16:21.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4e3fa0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb6e270 0x5566fbb78450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4e3fa0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6ec60 0x5566fbb78990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4e3fa0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb73dd0 0x5566fbb7f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4e3fa0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566fba5c130 con 0x5566fbb6e270 2024-03-11T19:16:21.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4e3fa0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566fbb5ba40 con 0x5566fbb6ec60 2024-03-11T19:16:21.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4e3fa0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566fbb64a10 con 0x5566fbb73dd0 2024-03-11T19:16:21.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4daffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb6e270 0x5566fbb78450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4daffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb6e270 0x5566fbb78450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47174/0 (socket says 172.21.15.29:47174) 2024-03-11T19:16:21.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4daffd640 1 -- 172.21.15.29:0/106013107 learned_addr learned my addr 172.21.15.29:0/106013107 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4db7fe640 1 --2- 172.21.15.29:0/106013107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb73dd0 0x5566fbb7f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4da7fc640 1 --2- 172.21.15.29:0/106013107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6ec60 0x5566fbb78990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4daffd640 1 -- 172.21.15.29:0/106013107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb73dd0 msgr2=0x5566fbb7f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4daffd640 1 --2- 172.21.15.29:0/106013107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb73dd0 0x5566fbb7f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4daffd640 1 -- 172.21.15.29:0/106013107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6ec60 msgr2=0x5566fbb78990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4daffd640 1 --2- 172.21.15.29:0/106013107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6ec60 0x5566fbb78990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4daffd640 1 -- 172.21.15.29:0/106013107 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566fba71c20 con 0x5566fbb6e270 2024-03-11T19:16:21.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4da7fc640 1 --2- 172.21.15.29:0/106013107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6ec60 0x5566fbb78990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:16:21.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4daffd640 1 --2- 172.21.15.29:0/106013107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb6e270 0x5566fbb78450 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc0971d0 tx=0x7fc4cc085040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec3c941d497c3f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4c0ff9640 1 -- 172.21.15.29:0/106013107 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc0035f0 con 0x5566fbb6e270 2024-03-11T19:16:21.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4c0ff9640 1 -- 172.21.15.29:0/106013107 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc4cc09af00 con 0x5566fbb6e270 2024-03-11T19:16:21.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.065+0000 7fc4c0ff9640 1 -- 172.21.15.29:0/106013107 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc0041e0 con 0x5566fbb6e270 2024-03-11T19:16:21.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.066+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/106013107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb6e270 msgr2=0x5566fbb78450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.066+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/106013107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb6e270 0x5566fbb78450 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc0971d0 tx=0x7fc4cc085040 comp rx=0 tx=0).stop 2024-03-11T19:16:21.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.066+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/106013107 shutdown_connections 2024-03-11T19:16:21.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.066+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/106013107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb73dd0 0x5566fbb7f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.066+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/106013107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6ec60 0x5566fbb78990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.066+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/106013107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb6e270 0x5566fbb78450 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.066+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/106013107 >> 172.21.15.29:0/106013107 conn(0x5566fbb697e0 msgr2=0x5566fbb69bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.066+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/106013107 shutdown_connections 2024-03-11T19:16:21.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.066+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/106013107 wait complete. 2024-03-11T19:16:21.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.066+0000 7fc4e3fa0740 1 -- start start 2024-03-11T19:16:21.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.066+0000 7fc4e3fa0740 1 Processor -- start 2024-03-11T19:16:21.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.066+0000 7fc4e3fa0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb6e270 0x5566fbb7e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.066+0000 7fc4e3fa0740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6ec60 0x5566fbb7ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.066+0000 7fc4e3fa0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb73dd0 0x5566fbbdaa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.066+0000 7fc4e3fa0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566fbbdafa0 con 0x5566fbb73dd0 2024-03-11T19:16:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.066+0000 7fc4e3fa0740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566fbbdb0e0 con 0x5566fbb6ec60 2024-03-11T19:16:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.066+0000 7fc4e3fa0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566fbbdb250 con 0x5566fbb6e270 2024-03-11T19:16:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.066+0000 7fc4e3fa0740 10 client.0 _ll_register_callbacks cb 0x7fc4d0009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4db7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb73dd0 0x5566fbbdaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4db7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb73dd0 0x5566fbbdaa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47184/0 (socket says 172.21.15.29:47184) 2024-03-11T19:16:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4db7fe640 1 -- 172.21.15.29:0/2825182911 learned_addr learned my addr 172.21.15.29:0/2825182911 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4daffd640 1 --2- 172.21.15.29:0/2825182911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb6e270 0x5566fbb7e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4da7fc640 1 --2- 172.21.15.29:0/2825182911 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6ec60 0x5566fbb7ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4db7fe640 1 -- 172.21.15.29:0/2825182911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb6e270 msgr2=0x5566fbb7e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4db7fe640 1 --2- 172.21.15.29:0/2825182911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb6e270 0x5566fbb7e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4db7fe640 1 -- 172.21.15.29:0/2825182911 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6ec60 msgr2=0x5566fbb7ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4db7fe640 1 --2- 172.21.15.29:0/2825182911 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6ec60 0x5566fbb7ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4db7fe640 1 -- 172.21.15.29:0/2825182911 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4dc19f6f0 con 0x5566fbb73dd0 2024-03-11T19:16:21.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4da7fc640 1 --2- 172.21.15.29:0/2825182911 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6ec60 0x5566fbb7ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:16:21.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4db7fe640 1 --2- 172.21.15.29:0/2825182911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb73dd0 0x5566fbbdaa60 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4009ca0 tx=0x7fc4d40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1955a41525b8ca5b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4beff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:16:21.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4beff5640 1 -- 172.21.15.29:0/2825182911 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d4003c30 con 0x5566fbb73dd0 2024-03-11T19:16:21.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4beff5640 1 -- 172.21.15.29:0/2825182911 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fc4d4003d90 con 0x5566fbb73dd0 2024-03-11T19:16:21.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/2825182911 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc4d0012900 con 0x5566fbb73dd0 2024-03-11T19:16:21.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4beff5640 1 -- 172.21.15.29:0/2825182911 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4d4004040 con 0x5566fbb73dd0 2024-03-11T19:16:21.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4e3fa0740 10 client.16956 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:21.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/2825182911 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5566fba978c0 con 0x5566fbb73dd0 2024-03-11T19:16:21.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.067+0000 7fc4beff5640 1 -- 172.21.15.29:0/2825182911 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc4d0012900 con 0x5566fbb73dd0 2024-03-11T19:16:21.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.068+0000 7fc4beff5640 1 -- 172.21.15.29:0/2825182911 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc4d4042860 con 0x5566fbb73dd0 2024-03-11T19:16:21.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.068+0000 7fc4e3fa0740 10 client.16956 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:21.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.068+0000 7fc4e3fa0740 10 client.16956 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:16:21.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.068+0000 7fc4e3fa0740 20 client.16956 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:21.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.068+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/2825182911 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5566fbbdcd00 0x5566fbbdf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.068+0000 7fc4e3fa0740 4 client.16956 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:16:21.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.068+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/2825182911 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5566fb862890 con 0x5566fbbdcd00 2024-03-11T19:16:21.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.068+0000 7fc4daffd640 1 --2- 172.21.15.29:0/2825182911 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5566fbbdcd00 0x5566fbbdf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.068+0000 7fc4daffd640 1 --2- 172.21.15.29:0/2825182911 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5566fbbdcd00 0x5566fbbdf120 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4bc3e5f2254ca244 server_cookie=f02dd15cdf64e462 in_seq=0 out_seq=0 2024-03-11T19:16:21.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.068+0000 7fc4beff5640 10 client.16956 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:16:21.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.068+0000 7fc4beff5640 1 -- 172.21.15.29:0/2825182911 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5566fb862890 con 0x5566fbbdcd00 2024-03-11T19:16:21.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.068+0000 7fc4beff5640 10 client.16956 handle_command_reply: tid=0 2024-03-11T19:16:21.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.068+0000 7fc4e3fa0740 1 client.16956 shutdown 2024-03-11T19:16:21.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.068+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/2825182911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb73dd0 msgr2=0x5566fbbdaa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.068+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/2825182911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb73dd0 0x5566fbbdaa60 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4009ca0 tx=0x7fc4d40095c0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/2825182911 shutdown_connections 2024-03-11T19:16:21.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/2825182911 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5566fbbdcd00 0x5566fbbdf120 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/2825182911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566fbb73dd0 0x5566fbbdaa60 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/2825182911 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566fbb6ec60 0x5566fbb7ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e3fa0740 1 --2- 172.21.15.29:0/2825182911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566fbb6e270 0x5566fbb7e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4beff5640 0 client.16956 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:16:21.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/2825182911 >> 172.21.15.29:0/2825182911 conn(0x5566fbb697e0 msgr2=0x5566fbb701b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/2825182911 shutdown_connections 2024-03-11T19:16:21.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e3fa0740 1 -- 172.21.15.29:0/2825182911 wait complete. 2024-03-11T19:16:21.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e3fa0740 20 client.16956 trim_cache size 0 max 16384 2024-03-11T19:16:21.081 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:16:21.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e13c9640 1 -- 172.21.15.29:0/3486487817 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc4c40a1a30 msgr2=0x7fc4c40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e13c9640 1 --2- 172.21.15.29:0/3486487817 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc4c40a1a30 0x7fc4c40a3ef0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4005e30 tx=0x7fc4d402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e13c9640 1 -- 172.21.15.29:0/3486487817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4dc101190 msgr2=0x7fc4dc1970e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e13c9640 1 --2- 172.21.15.29:0/3486487817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4dc101190 0x7fc4dc1970e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc00e9c0 tx=0x7fc4cc00ee80 comp rx=0 tx=0).stop 2024-03-11T19:16:21.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e13c9640 1 -- 172.21.15.29:0/3486487817 shutdown_connections 2024-03-11T19:16:21.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e13c9640 1 --2- 172.21.15.29:0/3486487817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4dc102d40 0x7fc4dc19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e13c9640 1 --2- 172.21.15.29:0/3486487817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4dc102380 0x7fc4dc197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e13c9640 1 --2- 172.21.15.29:0/3486487817 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc4c40a1a30 0x7fc4c40a3ef0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e13c9640 1 --2- 172.21.15.29:0/3486487817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4dc101190 0x7fc4dc1970e0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e13c9640 1 -- 172.21.15.29:0/3486487817 >> 172.21.15.29:0/3486487817 conn(0x7fc4dc0fc720 msgr2=0x7fc4dc105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e13c9640 1 -- 172.21.15.29:0/3486487817 shutdown_connections 2024-03-11T19:16:21.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.069+0000 7fc4e13c9640 1 -- 172.21.15.29:0/3486487817 wait complete. 2024-03-11T19:16:21.083 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:16:21.084 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:16:21.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.196+0000 7f88d6b7a640 1 -- 172.21.15.29:0/578110368 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d010d4a0 msgr2=0x7f88d010d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.196+0000 7f88d6b7a640 1 --2- 172.21.15.29:0/578110368 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d010d4a0 0x7f88d010d920 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009900 tx=0x7f88c002bc60 comp rx=0 tx=0).stop 2024-03-11T19:16:21.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.197+0000 7f88d6b7a640 1 -- 172.21.15.29:0/578110368 shutdown_connections 2024-03-11T19:16:21.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.197+0000 7f88d6b7a640 1 --2- 172.21.15.29:0/578110368 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d010d4a0 0x7f88d010d920 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.197+0000 7f88d6b7a640 1 --2- 172.21.15.29:0/578110368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88d010aaf0 0x7f88d010cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.197+0000 7f88d6b7a640 1 --2- 172.21.15.29:0/578110368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88d0108180 0x7f88d010a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.197+0000 7f88d6b7a640 1 -- 172.21.15.29:0/578110368 >> 172.21.15.29:0/578110368 conn(0x7f88d006dfe0 msgr2=0x7f88d0070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.197+0000 7f88d6b7a640 1 -- 172.21.15.29:0/578110368 shutdown_connections 2024-03-11T19:16:21.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.197+0000 7f88d6b7a640 1 -- 172.21.15.29:0/578110368 wait complete. 2024-03-11T19:16:21.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.198+0000 7f88d6b7a640 1 Processor -- start 2024-03-11T19:16:21.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.198+0000 7f88d6b7a640 1 -- start start 2024-03-11T19:16:21.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.198+0000 7f88d6b7a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88d0108180 0x7f88d01a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.198+0000 7f88d6b7a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88d010aaf0 0x7f88d01a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.198+0000 7f88d6b7a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d010d4a0 0x7f88d01ad490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.198+0000 7f88d6b7a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d01ad9d0 con 0x7f88d010aaf0 2024-03-11T19:16:21.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.198+0000 7f88d6b7a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d01adb40 con 0x7f88d010d4a0 2024-03-11T19:16:21.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.198+0000 7f88d6b7a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d01add00 con 0x7f88d0108180 2024-03-11T19:16:21.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.198+0000 7f88d5117640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d010d4a0 0x7f88d01ad490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.198+0000 7f88d5117640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d010d4a0 0x7f88d01ad490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44440/0 (socket says 172.21.15.29:44440) 2024-03-11T19:16:21.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.198+0000 7f88d5117640 1 -- 172.21.15.29:0/741162601 learned_addr learned my addr 172.21.15.29:0/741162601 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.198+0000 7f88d4916640 1 --2- 172.21.15.29:0/741162601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88d0108180 0x7f88d01a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.198+0000 7f88cffff640 1 --2- 172.21.15.29:0/741162601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88d010aaf0 0x7f88d01a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.198+0000 7f88d5117640 1 -- 172.21.15.29:0/741162601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88d0108180 msgr2=0x7f88d01a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.198+0000 7f88d5117640 1 --2- 172.21.15.29:0/741162601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88d0108180 0x7f88d01a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.198+0000 7f88d5117640 1 -- 172.21.15.29:0/741162601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88d010aaf0 msgr2=0x7f88d01a6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.198+0000 7f88d5117640 1 --2- 172.21.15.29:0/741162601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88d010aaf0 0x7f88d01a6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.198+0000 7f88d5117640 1 -- 172.21.15.29:0/741162601 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c0009560 con 0x7f88d010d4a0 2024-03-11T19:16:21.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.199+0000 7f88cffff640 1 --2- 172.21.15.29:0/741162601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88d010aaf0 0x7f88d01a6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:16:21.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.199+0000 7f88d5117640 1 --2- 172.21.15.29:0/741162601 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d010d4a0 0x7f88d01ad490 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009850 tx=0x7f88c0004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.199+0000 7f88cdffb640 1 -- 172.21.15.29:0/741162601 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c002dea0 con 0x7f88d010d4a0 2024-03-11T19:16:21.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.199+0000 7f88cdffb640 1 -- 172.21.15.29:0/741162601 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f88c0003aa0 con 0x7f88d010d4a0 2024-03-11T19:16:21.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.199+0000 7f88cdffb640 1 -- 172.21.15.29:0/741162601 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c00052d0 con 0x7f88d010d4a0 2024-03-11T19:16:21.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.199+0000 7f88d6b7a640 1 -- 172.21.15.29:0/741162601 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d01adf80 con 0x7f88d010d4a0 2024-03-11T19:16:21.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.199+0000 7f88d6b7a640 1 -- 172.21.15.29:0/741162601 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d01ae390 con 0x7f88d010d4a0 2024-03-11T19:16:21.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.203+0000 7f88cdffb640 1 -- 172.21.15.29:0/741162601 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f88c0005430 con 0x7f88d010d4a0 2024-03-11T19:16:21.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.204+0000 7f88d982d740 1 Processor -- start 2024-03-11T19:16:21.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.204+0000 7f88d982d740 1 -- start start 2024-03-11T19:16:21.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.204+0000 7f88cdffb640 1 --2- 172.21.15.29:0/741162601 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f88b80a1b00 0x7f88b80a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.204+0000 7f88d982d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9bc270 0x55741d9bc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.204+0000 7f88d982d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55741d9bcc60 0x55741d9c1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.204+0000 7f88d982d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9c1dd0 0x55741d9c4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.204+0000 7f88d982d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741d978960 con 0x55741d9bc270 2024-03-11T19:16:21.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.204+0000 7f88d982d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741d92c310 con 0x55741d9bcc60 2024-03-11T19:16:21.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.204+0000 7f88d982d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741d929840 con 0x55741d9c1dd0 2024-03-11T19:16:21.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.204+0000 7f88cffff640 1 --2- 172.21.15.29:0/741162601 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f88b80a1b00 0x7f88b80a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.204+0000 7f88d4916640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9bc270 0x55741d9bc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.204+0000 7f88cdffb640 1 -- 172.21.15.29:0/741162601 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f88c00efba0 con 0x7f88d010d4a0 2024-03-11T19:16:21.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.204+0000 7f88d4916640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9bc270 0x55741d9bc690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47212/0 (socket says 172.21.15.29:47212) 2024-03-11T19:16:21.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.204+0000 7f88d4916640 1 -- 172.21.15.29:0/2144132187 learned_addr learned my addr 172.21.15.29:0/2144132187 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.204+0000 7f88d5117640 1 --2- 172.21.15.29:0/2144132187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9c1dd0 0x55741d9c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.205+0000 7f88d4916640 1 -- 172.21.15.29:0/2144132187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9c1dd0 msgr2=0x55741d9c4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.205+0000 7f88d4916640 1 --2- 172.21.15.29:0/2144132187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9c1dd0 0x55741d9c4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.205+0000 7f88d4916640 1 -- 172.21.15.29:0/2144132187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55741d9bcc60 msgr2=0x55741d9c1760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:16:21.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.205+0000 7f88d4916640 1 --2- 172.21.15.29:0/2144132187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55741d9bcc60 0x55741d9c1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.205+0000 7f88d4916640 1 -- 172.21.15.29:0/2144132187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55741d8e7060 con 0x55741d9bc270 2024-03-11T19:16:21.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.205+0000 7f88d4916640 1 --2- 172.21.15.29:0/2144132187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9bc270 0x55741d9bc690 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f88c80039f0 tx=0x7f88c8003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b1f8f2e9d65e511 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.205+0000 7f88cffff640 1 --2- 172.21.15.29:0/741162601 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f88b80a1b00 0x7f88b80a3fc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f88d01a73f0 tx=0x7f88c4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.205+0000 7f88b6ffd640 1 -- 172.21.15.29:0/2144132187 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c8004b20 con 0x55741d9bc270 2024-03-11T19:16:21.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.205+0000 7f88b6ffd640 1 -- 172.21.15.29:0/2144132187 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f88c8004c80 con 0x55741d9bc270 2024-03-11T19:16:21.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.205+0000 7f88b6ffd640 1 -- 172.21.15.29:0/2144132187 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c80126b0 con 0x55741d9bc270 2024-03-11T19:16:21.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.206+0000 7f88d982d740 1 -- 172.21.15.29:0/2144132187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9bc270 msgr2=0x55741d9bc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.206+0000 7f88d982d740 1 --2- 172.21.15.29:0/2144132187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9bc270 0x55741d9bc690 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f88c80039f0 tx=0x7f88c8003eb0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.206+0000 7f88d982d740 1 -- 172.21.15.29:0/2144132187 shutdown_connections 2024-03-11T19:16:21.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.206+0000 7f88d982d740 1 --2- 172.21.15.29:0/2144132187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9c1dd0 0x55741d9c4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.206+0000 7f88d982d740 1 --2- 172.21.15.29:0/2144132187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55741d9bcc60 0x55741d9c1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.206+0000 7f88d982d740 1 --2- 172.21.15.29:0/2144132187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9bc270 0x55741d9bc690 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.206+0000 7f88d982d740 1 -- 172.21.15.29:0/2144132187 >> 172.21.15.29:0/2144132187 conn(0x55741d9b77e0 msgr2=0x55741d9b7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.207+0000 7f88d982d740 1 -- 172.21.15.29:0/2144132187 shutdown_connections 2024-03-11T19:16:21.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.207+0000 7f88d982d740 1 -- 172.21.15.29:0/2144132187 wait complete. 2024-03-11T19:16:21.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.207+0000 7f88d982d740 1 -- start start 2024-03-11T19:16:21.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.208+0000 7f88d982d740 1 Processor -- start 2024-03-11T19:16:21.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.208+0000 7f88d982d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9bc270 0x55741d9c6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.209+0000 7f88d982d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55741d9bcc60 0x55741d9c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.209+0000 7f88d4916640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9bc270 0x55741d9c6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.209+0000 7f88d4916640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9bc270 0x55741d9c6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47224/0 (socket says 172.21.15.29:47224) 2024-03-11T19:16:21.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.209+0000 7f88d4916640 1 -- 172.21.15.29:0/3645651956 learned_addr learned my addr 172.21.15.29:0/3645651956 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.209+0000 7f88d982d740 1 --2- 172.21.15.29:0/3645651956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9c1dd0 0x55741d9cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.209+0000 7f88d982d740 1 -- 172.21.15.29:0/3645651956 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741d925020 con 0x55741d9bc270 2024-03-11T19:16:21.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.209+0000 7f88d5117640 1 --2- 172.21.15.29:0/3645651956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9c1dd0 0x55741d9cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.209+0000 7f88d982d740 1 -- 172.21.15.29:0/3645651956 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741d5f8430 con 0x55741d9bcc60 2024-03-11T19:16:21.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.209+0000 7f88d982d740 1 -- 172.21.15.29:0/3645651956 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741d5b4e00 con 0x55741d9c1dd0 2024-03-11T19:16:21.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.210+0000 7f88d982d740 10 client.0 _ll_register_callbacks cb 0x55741d8cccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:21.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.210+0000 7f88d5117640 1 -- 172.21.15.29:0/3645651956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55741d9bcc60 msgr2=0x55741d9c6a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:16:21.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.210+0000 7f88d5117640 1 --2- 172.21.15.29:0/3645651956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55741d9bcc60 0x55741d9c6a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.210+0000 7f88d5117640 1 -- 172.21.15.29:0/3645651956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9bc270 msgr2=0x55741d9c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.210+0000 7f88d5117640 1 --2- 172.21.15.29:0/3645651956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9bc270 0x55741d9c6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.210+0000 7f88d5117640 1 -- 172.21.15.29:0/3645651956 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c8009cd0 con 0x55741d9c1dd0 2024-03-11T19:16:21.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.210+0000 7f88d4916640 1 --2- 172.21.15.29:0/3645651956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9bc270 0x55741d9c6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:16:21.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.210+0000 7f88d982d740 1 -- 172.21.15.29:0/3645651956 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55741d6247e0 con 0x55741d9c1dd0 2024-03-11T19:16:21.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.211+0000 7f88d5117640 1 --2- 172.21.15.29:0/3645651956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9c1dd0 0x55741d9cda90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f88c00ed790 tx=0x7f88c00bd370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8d0973249ada8ed server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.211+0000 7f88affff640 10 client.26488 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:16:21.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.211+0000 7f88affff640 1 -- 172.21.15.29:0/3645651956 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c00bd010 con 0x55741d9c1dd0 2024-03-11T19:16:21.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.212+0000 7f88affff640 1 -- 172.21.15.29:0/3645651956 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f88c00c5350 con 0x55741d9c1dd0 2024-03-11T19:16:21.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.212+0000 7f88affff640 1 -- 172.21.15.29:0/3645651956 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88c00c4ec0 con 0x55741d9c1dd0 2024-03-11T19:16:21.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.212+0000 7f88d982d740 10 client.26488 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:21.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.212+0000 7f88d982d740 1 -- 172.21.15.29:0/3645651956 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x55741d9236a0 con 0x55741d9c1dd0 2024-03-11T19:16:21.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.212+0000 7f88affff640 1 -- 172.21.15.29:0/3645651956 <== mon.1 v2:172.21.15.190:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f88c003c070 con 0x55741d9c1dd0 2024-03-11T19:16:21.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.212+0000 7f88d982d740 10 client.26488 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:21.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.212+0000 7f88d982d740 10 client.26488 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:16:21.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.213+0000 7f88d982d740 20 client.26488 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:21.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.213+0000 7f88d982d740 1 --2- 172.21.15.29:0/3645651956 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55741d9d2310 0x55741d9d4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.213+0000 7f88d982d740 4 client.26488 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:16:21.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.213+0000 7f88d982d740 1 -- 172.21.15.29:0/3645651956 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55741d661720 con 0x55741d9d2310 2024-03-11T19:16:21.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.213+0000 7f88d4916640 1 --2- 172.21.15.29:0/3645651956 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55741d9d2310 0x55741d9d4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.213+0000 7f88d4916640 1 --2- 172.21.15.29:0/3645651956 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55741d9d2310 0x55741d9d4700 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9f323c29b95150fb server_cookie=65cf772c97f5e90a in_seq=0 out_seq=0 2024-03-11T19:16:21.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.214+0000 7f88affff640 10 client.26488 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:16:21.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.214+0000 7f88affff640 1 -- 172.21.15.29:0/3645651956 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55741d661720 con 0x55741d9d2310 2024-03-11T19:16:21.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.214+0000 7f88affff640 10 client.26488 handle_command_reply: tid=0 2024-03-11T19:16:21.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.214+0000 7f88d982d740 1 client.26488 shutdown 2024-03-11T19:16:21.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.215+0000 7f88d982d740 1 -- 172.21.15.29:0/3645651956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9c1dd0 msgr2=0x55741d9cda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.215+0000 7f88d982d740 1 --2- 172.21.15.29:0/3645651956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9c1dd0 0x55741d9cda90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f88c00ed790 tx=0x7f88c00bd370 comp rx=0 tx=0).stop 2024-03-11T19:16:21.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.215+0000 7f88d982d740 1 -- 172.21.15.29:0/3645651956 shutdown_connections 2024-03-11T19:16:21.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.215+0000 7f88d982d740 1 --2- 172.21.15.29:0/3645651956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9c1dd0 0x55741d9cda90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.215+0000 7f88d982d740 1 --2- 172.21.15.29:0/3645651956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55741d9bcc60 0x55741d9c6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.215+0000 7f88d982d740 1 --2- 172.21.15.29:0/3645651956 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55741d9d2310 0x55741d9d4700 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.215+0000 7f88d982d740 1 --2- 172.21.15.29:0/3645651956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9bc270 0x55741d9c6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.215+0000 7f88affff640 0 client.26488 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:16:21.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.215+0000 7f88d982d740 1 -- 172.21.15.29:0/3645651956 >> 172.21.15.29:0/3645651956 conn(0x55741d9b77e0 msgr2=0x55741d9b6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.215+0000 7f88d982d740 1 -- 172.21.15.29:0/3645651956 shutdown_connections 2024-03-11T19:16:21.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.215+0000 7f88d982d740 1 -- 172.21.15.29:0/3645651956 wait complete. 2024-03-11T19:16:21.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.215+0000 7f88d982d740 20 client.26488 trim_cache size 0 max 16384 2024-03-11T19:16:21.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.239+0000 7f88d982d740 1 Processor -- start 2024-03-11T19:16:21.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.239+0000 7f88d982d740 1 -- start start 2024-03-11T19:16:21.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.239+0000 7f88d982d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55741d9bc270 0x55741d9c6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.239+0000 7f88d982d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9bcc60 0x55741d9cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.239+0000 7f88d982d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9c1dd0 0x55741d9c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.239+0000 7f88d982d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741d8aa130 con 0x55741d9c1dd0 2024-03-11T19:16:21.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.239+0000 7f88d982d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741d9c7700 con 0x55741d9bc270 2024-03-11T19:16:21.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.239+0000 7f88d982d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741d9b30f0 con 0x55741d9bcc60 2024-03-11T19:16:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.240+0000 7f88d5117640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9c1dd0 0x55741d9c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.240+0000 7f88d4916640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55741d9bc270 0x55741d9c6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.240+0000 7f88d5117640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9c1dd0 0x55741d9c6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47232/0 (socket says 172.21.15.29:47232) 2024-03-11T19:16:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.240+0000 7f88d4916640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55741d9bc270 0x55741d9c6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44448/0 (socket says 172.21.15.29:44448) 2024-03-11T19:16:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.240+0000 7f88d5117640 1 -- 172.21.15.29:0/2800214508 learned_addr learned my addr 172.21.15.29:0/2800214508 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.240+0000 7f88cffff640 1 --2- 172.21.15.29:0/2800214508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9bcc60 0x55741d9cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.240+0000 7f88d5117640 1 -- 172.21.15.29:0/2800214508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9bcc60 msgr2=0x55741d9cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.240+0000 7f88d5117640 1 --2- 172.21.15.29:0/2800214508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9bcc60 0x55741d9cda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.240+0000 7f88d5117640 1 -- 172.21.15.29:0/2800214508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55741d9bc270 msgr2=0x55741d9c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.240+0000 7f88d5117640 1 --2- 172.21.15.29:0/2800214508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55741d9bc270 0x55741d9c6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.240+0000 7f88d5117640 1 -- 172.21.15.29:0/2800214508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55741d8bfc20 con 0x55741d9c1dd0 2024-03-11T19:16:21.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.241+0000 7f88d5117640 1 --2- 172.21.15.29:0/2800214508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9c1dd0 0x55741d9c6a50 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f88c0006010 tx=0x7f88c00be000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7efd5b0cdad5c343 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.241+0000 7f88b67fc640 1 -- 172.21.15.29:0/2800214508 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c00bc6f0 con 0x55741d9c1dd0 2024-03-11T19:16:21.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.241+0000 7f88b67fc640 1 -- 172.21.15.29:0/2800214508 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55741d6247e0 con 0x55741d9c1dd0 2024-03-11T19:16:21.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.241+0000 7f88b67fc640 1 -- 172.21.15.29:0/2800214508 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c00edb60 con 0x55741d9c1dd0 2024-03-11T19:16:21.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.242+0000 7f88d982d740 1 -- 172.21.15.29:0/2800214508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9c1dd0 msgr2=0x55741d9c6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.242+0000 7f88d982d740 1 --2- 172.21.15.29:0/2800214508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9c1dd0 0x55741d9c6a50 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f88c0006010 tx=0x7f88c00be000 comp rx=0 tx=0).stop 2024-03-11T19:16:21.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.242+0000 7f88d982d740 1 -- 172.21.15.29:0/2800214508 shutdown_connections 2024-03-11T19:16:21.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.242+0000 7f88d982d740 1 --2- 172.21.15.29:0/2800214508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9c1dd0 0x55741d9c6a50 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.242+0000 7f88d982d740 1 --2- 172.21.15.29:0/2800214508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9bcc60 0x55741d9cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.242+0000 7f88d982d740 1 --2- 172.21.15.29:0/2800214508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55741d9bc270 0x55741d9c6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.242+0000 7f88d982d740 1 -- 172.21.15.29:0/2800214508 >> 172.21.15.29:0/2800214508 conn(0x55741d9b77e0 msgr2=0x55741d9c4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.242+0000 7f88d982d740 1 -- 172.21.15.29:0/2800214508 shutdown_connections 2024-03-11T19:16:21.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.242+0000 7f88d982d740 1 -- 172.21.15.29:0/2800214508 wait complete. 2024-03-11T19:16:21.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.242+0000 7f88d982d740 1 -- start start 2024-03-11T19:16:21.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.242+0000 7f88d982d740 1 Processor -- start 2024-03-11T19:16:21.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.243+0000 7f88d982d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9bc270 0x55741d9ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.243+0000 7f88d982d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55741d9bcc60 0x55741d9cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.243+0000 7f88d982d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9c1dd0 0x55741da28b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.243+0000 7f88d982d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741da29060 con 0x55741d9c1dd0 2024-03-11T19:16:21.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.243+0000 7f88d982d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741da291a0 con 0x55741d9bcc60 2024-03-11T19:16:21.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.243+0000 7f88d982d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741da29310 con 0x55741d9bc270 2024-03-11T19:16:21.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.243+0000 7f88d982d740 10 client.0 _ll_register_callbacks cb 0x7f88c8009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:21.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.243+0000 7f88d4916640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9bc270 0x55741d9ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.243+0000 7f88cffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55741d9bcc60 0x55741d9cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.243+0000 7f88d4916640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9bc270 0x55741d9ccdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:58004/0 (socket says 172.21.15.29:58004) 2024-03-11T19:16:21.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.243+0000 7f88d4916640 1 -- 172.21.15.29:0/138213836 learned_addr learned my addr 172.21.15.29:0/138213836 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.243+0000 7f88d5117640 1 --2- 172.21.15.29:0/138213836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9c1dd0 0x55741da28b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.244+0000 7f88d4916640 1 -- 172.21.15.29:0/138213836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55741d9bcc60 msgr2=0x55741d9cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.244+0000 7f88d4916640 1 --2- 172.21.15.29:0/138213836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55741d9bcc60 0x55741d9cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.244+0000 7f88d4916640 1 -- 172.21.15.29:0/138213836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9c1dd0 msgr2=0x55741da28b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.244+0000 7f88d4916640 1 --2- 172.21.15.29:0/138213836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9c1dd0 0x55741da28b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.244+0000 7f88d4916640 1 -- 172.21.15.29:0/138213836 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55741d9236a0 con 0x55741d9bc270 2024-03-11T19:16:21.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.244+0000 7f88d4916640 1 --2- 172.21.15.29:0/138213836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9bc270 0x55741d9ccdc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f88c8009ca0 tx=0x7f88c80093c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6287026511420d81 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.244+0000 7f88affff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:16:21.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.245+0000 7f88affff640 1 -- 172.21.15.29:0/138213836 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c8004920 con 0x55741d9bc270 2024-03-11T19:16:21.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.245+0000 7f88affff640 1 -- 172.21.15.29:0/138213836 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f88c8029640 con 0x55741d9bc270 2024-03-11T19:16:21.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.245+0000 7f88d982d740 1 -- 172.21.15.29:0/138213836 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f88c8004c80 con 0x55741d9bc270 2024-03-11T19:16:21.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.245+0000 7f88affff640 1 -- 172.21.15.29:0/138213836 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88c8029800 con 0x55741d9bc270 2024-03-11T19:16:21.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.245+0000 7f88d982d740 10 client.26494 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:21.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.245+0000 7f88d982d740 1 -- 172.21.15.29:0/138213836 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55741d8e58c0 con 0x55741d9bc270 2024-03-11T19:16:21.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.246+0000 7f88affff640 1 -- 172.21.15.29:0/138213836 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f88c8004c80 con 0x55741d9bc270 2024-03-11T19:16:21.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.246+0000 7f88affff640 1 -- 172.21.15.29:0/138213836 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f88c8025030 con 0x55741d9bc270 2024-03-11T19:16:21.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.246+0000 7f88d982d740 10 client.26494 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:21.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.246+0000 7f88d982d740 10 client.26494 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:16:21.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.246+0000 7f88d982d740 20 client.26494 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:21.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.246+0000 7f88d982d740 1 --2- 172.21.15.29:0/138213836 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55741da2ad80 0x55741da2d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.246+0000 7f88d982d740 4 client.26494 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:16:21.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.246+0000 7f88d982d740 1 -- 172.21.15.29:0/138213836 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55741d6b0890 con 0x55741da2ad80 2024-03-11T19:16:21.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.247+0000 7f88cffff640 1 --2- 172.21.15.29:0/138213836 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55741da2ad80 0x55741da2d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.248+0000 7f88cffff640 1 --2- 172.21.15.29:0/138213836 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55741da2ad80 0x55741da2d1a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=35f3010cf1c8af65 server_cookie=3c9e190e6c86ace in_seq=0 out_seq=0 2024-03-11T19:16:21.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.248+0000 7f88affff640 10 client.26494 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:16:21.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.249+0000 7f88affff640 1 -- 172.21.15.29:0/138213836 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55741d6b0890 con 0x55741da2ad80 2024-03-11T19:16:21.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.249+0000 7f88affff640 10 client.26494 handle_command_reply: tid=0 2024-03-11T19:16:21.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.249+0000 7f88d982d740 1 client.26494 shutdown 2024-03-11T19:16:21.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.249+0000 7f88d982d740 1 -- 172.21.15.29:0/138213836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9bc270 msgr2=0x55741d9ccdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.249+0000 7f88d982d740 1 --2- 172.21.15.29:0/138213836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9bc270 0x55741d9ccdc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f88c8009ca0 tx=0x7f88c80093c0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.250+0000 7f88d982d740 1 -- 172.21.15.29:0/138213836 shutdown_connections 2024-03-11T19:16:21.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.250+0000 7f88d982d740 1 --2- 172.21.15.29:0/138213836 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55741da2ad80 0x55741da2d1a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.250+0000 7f88d982d740 1 --2- 172.21.15.29:0/138213836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741d9c1dd0 0x55741da28b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.250+0000 7f88d982d740 1 --2- 172.21.15.29:0/138213836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55741d9bcc60 0x55741d9cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.250+0000 7f88d982d740 1 --2- 172.21.15.29:0/138213836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55741d9bc270 0x55741d9ccdc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.250+0000 7f88d982d740 1 -- 172.21.15.29:0/138213836 >> 172.21.15.29:0/138213836 conn(0x55741d9b77e0 msgr2=0x55741d9c4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.250+0000 7f88affff640 0 client.26494 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:16:21.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.251+0000 7f88d982d740 1 -- 172.21.15.29:0/138213836 shutdown_connections 2024-03-11T19:16:21.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.251+0000 7f88d982d740 1 -- 172.21.15.29:0/138213836 wait complete. 2024-03-11T19:16:21.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.251+0000 7f88d982d740 20 client.26494 trim_cache size 0 max 16384 2024-03-11T19:16:21.256 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:16:21.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.254+0000 7f88d6b7a640 1 -- 172.21.15.29:0/741162601 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f88b80a1b00 msgr2=0x7f88b80a3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.254+0000 7f88d6b7a640 1 --2- 172.21.15.29:0/741162601 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f88b80a1b00 0x7f88b80a3fc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f88d01a73f0 tx=0x7f88c4005f90 comp rx=0 tx=0).stop 2024-03-11T19:16:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.254+0000 7f88d6b7a640 1 -- 172.21.15.29:0/741162601 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d010d4a0 msgr2=0x7f88d01ad490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.254+0000 7f88d6b7a640 1 --2- 172.21.15.29:0/741162601 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d010d4a0 0x7f88d01ad490 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009850 tx=0x7f88c0004270 comp rx=0 tx=0).stop 2024-03-11T19:16:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.255+0000 7f88d6b7a640 1 -- 172.21.15.29:0/741162601 shutdown_connections 2024-03-11T19:16:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.255+0000 7f88d6b7a640 1 --2- 172.21.15.29:0/741162601 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d010d4a0 0x7f88d01ad490 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.255+0000 7f88d6b7a640 1 --2- 172.21.15.29:0/741162601 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f88b80a1b00 0x7f88b80a3fc0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.255+0000 7f88d6b7a640 1 --2- 172.21.15.29:0/741162601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88d010aaf0 0x7f88d01a6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.255+0000 7f88d6b7a640 1 --2- 172.21.15.29:0/741162601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88d0108180 0x7f88d01a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.255+0000 7f88d6b7a640 1 -- 172.21.15.29:0/741162601 >> 172.21.15.29:0/741162601 conn(0x7f88d006dfe0 msgr2=0x7f88d01097d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.255+0000 7f88d6b7a640 1 -- 172.21.15.29:0/741162601 shutdown_connections 2024-03-11T19:16:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.255+0000 7f88d6b7a640 1 -- 172.21.15.29:0/741162601 wait complete. 2024-03-11T19:16:21.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:20 smithi190 ceph-mon[36292]: pgmap v547: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T19:16:21.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:20 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2116084448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:16:21.276 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:20 smithi029 ceph-mon[32439]: pgmap v547: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T19:16:21.276 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:20 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2116084448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:16:21.278 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:16:21.278 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:16:21.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:20 smithi170 ceph-mon[35690]: pgmap v547: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T19:16:21.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:20 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2116084448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:16:21.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.449+0000 7f7e2e913640 1 -- 172.21.15.29:0/982487756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e281045b0 msgr2=0x7f7e28104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.449+0000 7f7e2e913640 1 --2- 172.21.15.29:0/982487756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e281045b0 0x7f7e28104a30 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c009860 tx=0x7f7e1c02bb70 comp rx=0 tx=0).stop 2024-03-11T19:16:21.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.450+0000 7f7e2e913640 1 -- 172.21.15.29:0/982487756 shutdown_connections 2024-03-11T19:16:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.450+0000 7f7e2e913640 1 --2- 172.21.15.29:0/982487756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e28104f70 0x7f7e2810b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.450+0000 7f7e2e913640 1 --2- 172.21.15.29:0/982487756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e281045b0 0x7f7e28104a30 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.450+0000 7f7e2e913640 1 --2- 172.21.15.29:0/982487756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e281033c0 0x7f7e281037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.450+0000 7f7e2e913640 1 -- 172.21.15.29:0/982487756 >> 172.21.15.29:0/982487756 conn(0x7f7e280fe930 msgr2=0x7f7e28100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.450+0000 7f7e2e913640 1 -- 172.21.15.29:0/982487756 shutdown_connections 2024-03-11T19:16:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.450+0000 7f7e2e913640 1 -- 172.21.15.29:0/982487756 wait complete. 2024-03-11T19:16:21.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.451+0000 7f7e2e913640 1 Processor -- start 2024-03-11T19:16:21.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.451+0000 7f7e2e913640 1 -- start start 2024-03-11T19:16:21.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.452+0000 7f7e2e913640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e281033c0 0x7f7e28199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.452+0000 7f7e2e913640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e281045b0 0x7f7e28199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.452+0000 7f7e2e913640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e28104f70 0x7f7e281a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.452+0000 7f7e2e913640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e281a0d10 con 0x7f7e281033c0 2024-03-11T19:16:21.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.452+0000 7f7e2e913640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e281a0e80 con 0x7f7e28104f70 2024-03-11T19:16:21.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.452+0000 7f7e2e913640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e281a1040 con 0x7f7e281045b0 2024-03-11T19:16:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.452+0000 7f7e277fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e281045b0 0x7f7e28199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.452+0000 7f7e277fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e281045b0 0x7f7e28199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:58016/0 (socket says 172.21.15.29:58016) 2024-03-11T19:16:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.452+0000 7f7e277fe640 1 -- 172.21.15.29:0/2342851741 learned_addr learned my addr 172.21.15.29:0/2342851741 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.452+0000 7f7e2ceb0640 1 --2- 172.21.15.29:0/2342851741 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e28104f70 0x7f7e281a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.452+0000 7f7e277fe640 1 -- 172.21.15.29:0/2342851741 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e28104f70 msgr2=0x7f7e281a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.452+0000 7f7e277fe640 1 --2- 172.21.15.29:0/2342851741 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e28104f70 0x7f7e281a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.453+0000 7f7e277fe640 1 -- 172.21.15.29:0/2342851741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e281033c0 msgr2=0x7f7e28199220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.453+0000 7f7e277fe640 1 --2- 172.21.15.29:0/2342851741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e281033c0 0x7f7e28199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.453+0000 7f7e277fe640 1 -- 172.21.15.29:0/2342851741 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c009490 con 0x7f7e281045b0 2024-03-11T19:16:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.453+0000 7f7e277fe640 1 --2- 172.21.15.29:0/2342851741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e281045b0 0x7f7e28199760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c02c180 tx=0x7f7e1c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.453+0000 7f7e257fa640 1 -- 172.21.15.29:0/2342851741 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c02c710 con 0x7f7e281045b0 2024-03-11T19:16:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.454+0000 7f7e257fa640 1 -- 172.21.15.29:0/2342851741 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7e1c02ce90 con 0x7f7e281045b0 2024-03-11T19:16:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.454+0000 7f7e257fa640 1 -- 172.21.15.29:0/2342851741 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c0035f0 con 0x7f7e281045b0 2024-03-11T19:16:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.454+0000 7f7e2e913640 1 -- 172.21.15.29:0/2342851741 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e281a12c0 con 0x7f7e281045b0 2024-03-11T19:16:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.454+0000 7f7e2e913640 1 -- 172.21.15.29:0/2342851741 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e281a1720 con 0x7f7e281045b0 2024-03-11T19:16:21.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.457+0000 7f7e315ca740 1 Processor -- start 2024-03-11T19:16:21.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.457+0000 7f7e257fa640 1 -- 172.21.15.29:0/2342851741 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7e1c03b040 con 0x7f7e281045b0 2024-03-11T19:16:21.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.457+0000 7f7e315ca740 1 -- start start 2024-03-11T19:16:21.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.457+0000 7f7e315ca740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55715192d270 0x55715192d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.457+0000 7f7e315ca740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192dc60 0x557151932760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.457+0000 7f7e315ca740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557151932dd0 0x557151935260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.457+0000 7f7e315ca740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571518e9960 con 0x557151932dd0 2024-03-11T19:16:21.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.457+0000 7f7e315ca740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55715189d310 con 0x55715192d270 2024-03-11T19:16:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.457+0000 7f7e315ca740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55715189a840 con 0x55715192dc60 2024-03-11T19:16:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.458+0000 7f7e2ceb0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55715192d270 0x55715192d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.458+0000 7f7e27fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192dc60 0x557151932760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.458+0000 7f7e27fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192dc60 0x557151932760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:58020/0 (socket says 172.21.15.29:58020) 2024-03-11T19:16:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.458+0000 7f7e277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557151932dd0 0x557151935260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.458+0000 7f7e27fff640 1 -- 172.21.15.29:0/1712518621 learned_addr learned my addr 172.21.15.29:0/1712518621 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.458+0000 7f7e257fa640 1 --2- 172.21.15.29:0/2342851741 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7e100a1b00 0x7f7e100a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.458+0000 7f7e27fff640 1 -- 172.21.15.29:0/1712518621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55715192d270 msgr2=0x55715192d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.458+0000 7f7e27fff640 1 --2- 172.21.15.29:0/1712518621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55715192d270 0x55715192d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.458+0000 7f7e27fff640 1 -- 172.21.15.29:0/1712518621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557151932dd0 msgr2=0x557151935260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.458+0000 7f7e27fff640 1 --2- 172.21.15.29:0/1712518621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557151932dd0 0x557151935260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.458+0000 7f7e27fff640 1 -- 172.21.15.29:0/1712518621 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557151858060 con 0x55715192dc60 2024-03-11T19:16:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.458+0000 7f7e257fa640 1 -- 172.21.15.29:0/2342851741 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7e1c0ef0e0 con 0x7f7e281045b0 2024-03-11T19:16:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.458+0000 7f7e2ceb0640 1 --2- 172.21.15.29:0/2342851741 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7e100a1b00 0x7f7e100a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.459+0000 7f7e27fff640 1 --2- 172.21.15.29:0/1712518621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192dc60 0x557151932760 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7e18009b20 tx=0x7f7e1800e350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6ee25e0be3d68d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.459+0000 7f7e0e7fc640 1 -- 172.21.15.29:0/1712518621 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e18009670 con 0x55715192dc60 2024-03-11T19:16:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.459+0000 7f7e0e7fc640 1 -- 172.21.15.29:0/1712518621 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7e1800ed80 con 0x55715192dc60 2024-03-11T19:16:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.459+0000 7f7e0e7fc640 1 -- 172.21.15.29:0/1712518621 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e18010430 con 0x55715192dc60 2024-03-11T19:16:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.460+0000 7f7e315ca740 1 -- 172.21.15.29:0/1712518621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192dc60 msgr2=0x557151932760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.460+0000 7f7e315ca740 1 --2- 172.21.15.29:0/1712518621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192dc60 0x557151932760 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7e18009b20 tx=0x7f7e1800e350 comp rx=0 tx=0).stop 2024-03-11T19:16:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.460+0000 7f7e2ceb0640 1 --2- 172.21.15.29:0/2342851741 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7e100a1b00 0x7f7e100a3fc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x55715192e4c0 tx=0x7f7e20004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.460+0000 7f7e315ca740 1 -- 172.21.15.29:0/1712518621 shutdown_connections 2024-03-11T19:16:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.460+0000 7f7e315ca740 1 --2- 172.21.15.29:0/1712518621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557151932dd0 0x557151935260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.460+0000 7f7e315ca740 1 --2- 172.21.15.29:0/1712518621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192dc60 0x557151932760 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.460+0000 7f7e315ca740 1 --2- 172.21.15.29:0/1712518621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55715192d270 0x55715192d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.460+0000 7f7e315ca740 1 -- 172.21.15.29:0/1712518621 >> 172.21.15.29:0/1712518621 conn(0x5571519287e0 msgr2=0x557151928be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.460+0000 7f7e315ca740 1 -- 172.21.15.29:0/1712518621 shutdown_connections 2024-03-11T19:16:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.460+0000 7f7e315ca740 1 -- 172.21.15.29:0/1712518621 wait complete. 2024-03-11T19:16:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.461+0000 7f7e315ca740 1 -- start start 2024-03-11T19:16:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.461+0000 7f7e315ca740 1 Processor -- start 2024-03-11T19:16:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.461+0000 7f7e315ca740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192d270 0x5571519373c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.461+0000 7f7e315ca740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192dc60 0x557151937900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.461+0000 7f7e27fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192d270 0x5571519373c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.461+0000 7f7e315ca740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557151932dd0 0x55715193e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.461+0000 7f7e27fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192d270 0x5571519373c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:58036/0 (socket says 172.21.15.29:58036) 2024-03-11T19:16:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.461+0000 7f7e27fff640 1 -- 172.21.15.29:0/4025433095 learned_addr learned my addr 172.21.15.29:0/4025433095 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.461+0000 7f7e315ca740 1 -- 172.21.15.29:0/4025433095 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557151896020 con 0x55715192dc60 2024-03-11T19:16:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.461+0000 7f7e315ca740 1 -- 172.21.15.29:0/4025433095 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557151569430 con 0x557151932dd0 2024-03-11T19:16:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.462+0000 7f7e277fe640 1 --2- 172.21.15.29:0/4025433095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192dc60 0x557151937900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.462+0000 7f7e315ca740 1 -- 172.21.15.29:0/4025433095 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557151525e00 con 0x55715192d270 2024-03-11T19:16:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.462+0000 7f7e315ca740 10 client.0 _ll_register_callbacks cb 0x55715183dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.462+0000 7f7e27fff640 1 -- 172.21.15.29:0/4025433095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557151932dd0 msgr2=0x55715193e940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:16:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.462+0000 7f7e27fff640 1 --2- 172.21.15.29:0/4025433095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557151932dd0 0x55715193e940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.462+0000 7f7e27fff640 1 -- 172.21.15.29:0/4025433095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192dc60 msgr2=0x557151937900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.462+0000 7f7e27fff640 1 --2- 172.21.15.29:0/4025433095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192dc60 0x557151937900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.462+0000 7f7e27fff640 1 -- 172.21.15.29:0/4025433095 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e180097d0 con 0x55715192d270 2024-03-11T19:16:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.462+0000 7f7e277fe640 1 --2- 172.21.15.29:0/4025433095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192dc60 0x557151937900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:16:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.463+0000 7f7e27fff640 1 --2- 172.21.15.29:0/4025433095 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192d270 0x5571519373c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7e18018000 tx=0x7f7e180052d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21d31158c8e7489 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.463+0000 7f7e077fe640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:16:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.463+0000 7f7e077fe640 1 -- 172.21.15.29:0/4025433095 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e18009670 con 0x55715192d270 2024-03-11T19:16:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.463+0000 7f7e315ca740 1 -- 172.21.15.29:0/4025433095 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5571515957e0 con 0x55715192d270 2024-03-11T19:16:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.463+0000 7f7e077fe640 1 -- 172.21.15.29:0/4025433095 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7e18014070 con 0x55715192d270 2024-03-11T19:16:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.463+0000 7f7e077fe640 1 -- 172.21.15.29:0/4025433095 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e180054f0 con 0x55715192d270 2024-03-11T19:16:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.463+0000 7f7e315ca740 10 client.26506 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.463+0000 7f7e315ca740 1 -- 172.21.15.29:0/4025433095 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5571518946a0 con 0x55715192d270 2024-03-11T19:16:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.463+0000 7f7e077fe640 1 -- 172.21.15.29:0/4025433095 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5571515957e0 con 0x55715192d270 2024-03-11T19:16:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.463+0000 7f7e077fe640 1 -- 172.21.15.29:0/4025433095 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7e18010430 con 0x55715192d270 2024-03-11T19:16:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.464+0000 7f7e315ca740 10 client.26506 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.464+0000 7f7e315ca740 10 client.26506 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:16:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.464+0000 7f7e315ca740 20 client.26506 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.464+0000 7f7e315ca740 1 --2- 172.21.15.29:0/4025433095 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x557151943490 0x5571519458d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.464+0000 7f7e315ca740 4 client.26506 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:16:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.464+0000 7f7e315ca740 1 -- 172.21.15.29:0/4025433095 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5571515d2720 con 0x557151943490 2024-03-11T19:16:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.464+0000 7f7e277fe640 1 --2- 172.21.15.29:0/4025433095 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x557151943490 0x5571519458d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.464+0000 7f7e077fe640 10 client.26506 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:16:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.464+0000 7f7e277fe640 1 --2- 172.21.15.29:0/4025433095 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x557151943490 0x5571519458d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5f392859462cdea5 server_cookie=fea7df1803b39c49 in_seq=0 out_seq=0 2024-03-11T19:16:21.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.465+0000 7f7e077fe640 1 -- 172.21.15.29:0/4025433095 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5571515d2720 con 0x557151943490 2024-03-11T19:16:21.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.465+0000 7f7e077fe640 10 client.26506 handle_command_reply: tid=0 2024-03-11T19:16:21.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.465+0000 7f7e315ca740 1 client.26506 shutdown 2024-03-11T19:16:21.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.465+0000 7f7e315ca740 1 -- 172.21.15.29:0/4025433095 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192d270 msgr2=0x5571519373c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.465+0000 7f7e315ca740 1 --2- 172.21.15.29:0/4025433095 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192d270 0x5571519373c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7e18018000 tx=0x7f7e180052d0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.465+0000 7f7e315ca740 1 -- 172.21.15.29:0/4025433095 shutdown_connections 2024-03-11T19:16:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.465+0000 7f7e315ca740 1 --2- 172.21.15.29:0/4025433095 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x557151943490 0x5571519458d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.465+0000 7f7e315ca740 1 --2- 172.21.15.29:0/4025433095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557151932dd0 0x55715193e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.465+0000 7f7e315ca740 1 --2- 172.21.15.29:0/4025433095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192dc60 0x557151937900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.465+0000 7f7e077fe640 0 client.26506 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:16:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.465+0000 7f7e315ca740 1 --2- 172.21.15.29:0/4025433095 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192d270 0x5571519373c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.465+0000 7f7e315ca740 1 -- 172.21.15.29:0/4025433095 >> 172.21.15.29:0/4025433095 conn(0x5571519287e0 msgr2=0x557151927b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.465+0000 7f7e315ca740 1 -- 172.21.15.29:0/4025433095 shutdown_connections 2024-03-11T19:16:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.465+0000 7f7e315ca740 1 -- 172.21.15.29:0/4025433095 wait complete. 2024-03-11T19:16:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.465+0000 7f7e315ca740 20 client.26506 trim_cache size 0 max 16384 2024-03-11T19:16:21.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.486+0000 7f7e315ca740 1 Processor -- start 2024-03-11T19:16:21.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.486+0000 7f7e315ca740 1 -- start start 2024-03-11T19:16:21.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.487+0000 7f7e315ca740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192d270 0x55715193f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.487+0000 7f7e315ca740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192dc60 0x55715193e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.487+0000 7f7e315ca740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557151932dd0 0x55715194a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.487+0000 7f7e315ca740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55715181b130 con 0x55715192dc60 2024-03-11T19:16:21.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.487+0000 7f7e277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192dc60 0x55715193e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.487+0000 7f7e277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192dc60 0x55715193e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47276/0 (socket says 172.21.15.29:47276) 2024-03-11T19:16:21.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.487+0000 7f7e277fe640 1 -- 172.21.15.29:0/2980589120 learned_addr learned my addr 172.21.15.29:0/2980589120 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.487+0000 7f7e315ca740 1 -- 172.21.15.29:0/2980589120 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571519292d0 con 0x557151932dd0 2024-03-11T19:16:21.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.487+0000 7f7e315ca740 1 -- 172.21.15.29:0/2980589120 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571519245c0 con 0x55715192d270 2024-03-11T19:16:21.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.488+0000 7f7e277fe640 1 -- 172.21.15.29:0/2980589120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192d270 msgr2=0x55715193f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.488+0000 7f7e277fe640 1 --2- 172.21.15.29:0/2980589120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192d270 0x55715193f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.488+0000 7f7e277fe640 1 -- 172.21.15.29:0/2980589120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557151932dd0 msgr2=0x55715194a270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:16:21.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.488+0000 7f7e277fe640 1 --2- 172.21.15.29:0/2980589120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557151932dd0 0x55715194a270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.488+0000 7f7e277fe640 1 -- 172.21.15.29:0/2980589120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557151830c20 con 0x55715192dc60 2024-03-11T19:16:21.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.488+0000 7f7e277fe640 1 --2- 172.21.15.29:0/2980589120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192dc60 0x55715193e900 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c0bbdf0 tx=0x7f7e1c0f2dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da040cb997b20a3d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.488+0000 7f7e0dffb640 1 -- 172.21.15.29:0/2980589120 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c0bba50 con 0x55715192dc60 2024-03-11T19:16:21.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.489+0000 7f7e0dffb640 1 -- 172.21.15.29:0/2980589120 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7e1c032600 con 0x55715192dc60 2024-03-11T19:16:21.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.489+0000 7f7e0dffb640 1 -- 172.21.15.29:0/2980589120 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c0c0240 con 0x55715192dc60 2024-03-11T19:16:21.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.490+0000 7f7e315ca740 1 -- 172.21.15.29:0/2980589120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192dc60 msgr2=0x55715193e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.490+0000 7f7e315ca740 1 --2- 172.21.15.29:0/2980589120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192dc60 0x55715193e900 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c0bbdf0 tx=0x7f7e1c0f2dd0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.490+0000 7f7e315ca740 1 -- 172.21.15.29:0/2980589120 shutdown_connections 2024-03-11T19:16:21.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.490+0000 7f7e315ca740 1 --2- 172.21.15.29:0/2980589120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557151932dd0 0x55715194a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.490+0000 7f7e315ca740 1 --2- 172.21.15.29:0/2980589120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192dc60 0x55715193e900 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.490+0000 7f7e315ca740 1 --2- 172.21.15.29:0/2980589120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192d270 0x55715193f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.490+0000 7f7e315ca740 1 -- 172.21.15.29:0/2980589120 >> 172.21.15.29:0/2980589120 conn(0x5571519287e0 msgr2=0x557151928bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.490+0000 7f7e315ca740 1 -- 172.21.15.29:0/2980589120 shutdown_connections 2024-03-11T19:16:21.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.490+0000 7f7e315ca740 1 -- 172.21.15.29:0/2980589120 wait complete. 2024-03-11T19:16:21.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.491+0000 7f7e315ca740 1 -- start start 2024-03-11T19:16:21.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.491+0000 7f7e315ca740 1 Processor -- start 2024-03-11T19:16:21.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.491+0000 7f7e315ca740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192d270 0x55715193e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.491+0000 7f7e315ca740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192dc60 0x55715193cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.492+0000 7f7e315ca740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557151932dd0 0x55715193d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.492+0000 7f7e315ca740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55715193db90 con 0x55715192d270 2024-03-11T19:16:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.492+0000 7f7e315ca740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55715193dd00 con 0x557151932dd0 2024-03-11T19:16:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.492+0000 7f7e315ca740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571519999e0 con 0x55715192dc60 2024-03-11T19:16:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.492+0000 7f7e27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192d270 0x55715193e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.492+0000 7f7e27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192d270 0x55715193e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47292/0 (socket says 172.21.15.29:47292) 2024-03-11T19:16:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.492+0000 7f7e27fff640 1 -- 172.21.15.29:0/3897636926 learned_addr learned my addr 172.21.15.29:0/3897636926 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.492+0000 7f7e315ca740 10 client.0 _ll_register_callbacks cb 0x7f7e1c0f2ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.492+0000 7f7e2ceb0640 1 --2- 172.21.15.29:0/3897636926 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557151932dd0 0x55715193d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.492+0000 7f7e277fe640 1 --2- 172.21.15.29:0/3897636926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192dc60 0x55715193cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.492+0000 7f7e27fff640 1 -- 172.21.15.29:0/3897636926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192dc60 msgr2=0x55715193cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.492+0000 7f7e27fff640 1 --2- 172.21.15.29:0/3897636926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192dc60 0x55715193cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.492+0000 7f7e27fff640 1 -- 172.21.15.29:0/3897636926 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557151932dd0 msgr2=0x55715193d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.492+0000 7f7e27fff640 1 --2- 172.21.15.29:0/3897636926 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557151932dd0 0x55715193d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.492+0000 7f7e27fff640 1 -- 172.21.15.29:0/3897636926 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e281a1720 con 0x55715192d270 2024-03-11T19:16:21.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.492+0000 7f7e277fe640 1 --2- 172.21.15.29:0/3897636926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192dc60 0x55715193cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:16:21.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.492+0000 7f7e27fff640 1 --2- 172.21.15.29:0/3897636926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192d270 0x55715193e8e0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f7e1802f040 tx=0x7f7e18004c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d19594e1a06fd527 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.493+0000 7f7e077fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:16:21.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.493+0000 7f7e077fe640 1 -- 172.21.15.29:0/3897636926 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e180047a0 con 0x55715192d270 2024-03-11T19:16:21.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.493+0000 7f7e077fe640 1 -- 172.21.15.29:0/3897636926 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7e18014040 con 0x55715192d270 2024-03-11T19:16:21.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.493+0000 7f7e315ca740 1 -- 172.21.15.29:0/3897636926 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5571519240f0 con 0x55715192d270 2024-03-11T19:16:21.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.494+0000 7f7e077fe640 1 -- 172.21.15.29:0/3897636926 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e18009600 con 0x55715192d270 2024-03-11T19:16:21.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.494+0000 7f7e315ca740 10 client.16989 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:21.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.494+0000 7f7e315ca740 1 -- 172.21.15.29:0/3897636926 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5571518568c0 con 0x55715192d270 2024-03-11T19:16:21.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.495+0000 7f7e077fe640 1 -- 172.21.15.29:0/3897636926 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7e1802d430 con 0x55715192d270 2024-03-11T19:16:21.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.495+0000 7f7e077fe640 1 -- 172.21.15.29:0/3897636926 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7e18018430 con 0x55715192d270 2024-03-11T19:16:21.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.495+0000 7f7e315ca740 10 client.16989 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:21.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.495+0000 7f7e315ca740 10 client.16989 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:16:21.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.495+0000 7f7e315ca740 20 client.16989 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:21.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.495+0000 7f7e315ca740 1 --2- 172.21.15.29:0/3897636926 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x557151943490 0x55715199d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.495+0000 7f7e315ca740 4 client.16989 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:16:21.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.495+0000 7f7e315ca740 1 -- 172.21.15.29:0/3897636926 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5571519240f0 con 0x557151943490 2024-03-11T19:16:21.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.495+0000 7f7e277fe640 1 --2- 172.21.15.29:0/3897636926 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x557151943490 0x55715199d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.496+0000 7f7e277fe640 1 --2- 172.21.15.29:0/3897636926 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x557151943490 0x55715199d170 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=76e82d382e42be0c server_cookie=34b5c539c0786a22 in_seq=0 out_seq=0 2024-03-11T19:16:21.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.496+0000 7f7e077fe640 10 client.16989 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:16:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.497+0000 7f7e077fe640 1 -- 172.21.15.29:0/3897636926 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5571519240f0 con 0x557151943490 2024-03-11T19:16:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.497+0000 7f7e077fe640 10 client.16989 handle_command_reply: tid=0 2024-03-11T19:16:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.497+0000 7f7e315ca740 1 client.16989 shutdown 2024-03-11T19:16:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.497+0000 7f7e315ca740 1 -- 172.21.15.29:0/3897636926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192d270 msgr2=0x55715193e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.497+0000 7f7e315ca740 1 --2- 172.21.15.29:0/3897636926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192d270 0x55715193e8e0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f7e1802f040 tx=0x7f7e18004c40 comp rx=0 tx=0).stop 2024-03-11T19:16:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.497+0000 7f7e315ca740 1 -- 172.21.15.29:0/3897636926 shutdown_connections 2024-03-11T19:16:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.497+0000 7f7e315ca740 1 --2- 172.21.15.29:0/3897636926 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x557151943490 0x55715199d170 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.497+0000 7f7e315ca740 1 --2- 172.21.15.29:0/3897636926 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557151932dd0 0x55715193d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.497+0000 7f7e315ca740 1 --2- 172.21.15.29:0/3897636926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55715192dc60 0x55715193cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.498+0000 7f7e077fe640 0 client.16989 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:16:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.498+0000 7f7e315ca740 1 --2- 172.21.15.29:0/3897636926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55715192d270 0x55715193e8e0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.498+0000 7f7e315ca740 1 -- 172.21.15.29:0/3897636926 >> 172.21.15.29:0/3897636926 conn(0x5571519287e0 msgr2=0x55715194a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.498+0000 7f7e315ca740 1 -- 172.21.15.29:0/3897636926 shutdown_connections 2024-03-11T19:16:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.498+0000 7f7e315ca740 1 -- 172.21.15.29:0/3897636926 wait complete. 2024-03-11T19:16:21.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.498+0000 7f7e315ca740 20 client.16989 trim_cache size 0 max 16384 2024-03-11T19:16:21.504 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:16:21.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.499+0000 7f7e2e913640 1 -- 172.21.15.29:0/2342851741 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7e100a1b00 msgr2=0x7f7e100a3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.499+0000 7f7e2e913640 1 --2- 172.21.15.29:0/2342851741 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7e100a1b00 0x7f7e100a3fc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x55715192e4c0 tx=0x7f7e20004a40 comp rx=0 tx=0).stop 2024-03-11T19:16:21.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.499+0000 7f7e2e913640 1 -- 172.21.15.29:0/2342851741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e281045b0 msgr2=0x7f7e28199760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.499+0000 7f7e2e913640 1 --2- 172.21.15.29:0/2342851741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e281045b0 0x7f7e28199760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c02c180 tx=0x7f7e1c004270 comp rx=0 tx=0).stop 2024-03-11T19:16:21.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.500+0000 7f7e2e913640 1 -- 172.21.15.29:0/2342851741 shutdown_connections 2024-03-11T19:16:21.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.500+0000 7f7e2e913640 1 --2- 172.21.15.29:0/2342851741 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e28104f70 0x7f7e281a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.500+0000 7f7e2e913640 1 --2- 172.21.15.29:0/2342851741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e281045b0 0x7f7e28199760 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.500+0000 7f7e2e913640 1 --2- 172.21.15.29:0/2342851741 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7e100a1b00 0x7f7e100a3fc0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.500+0000 7f7e2e913640 1 --2- 172.21.15.29:0/2342851741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e281033c0 0x7f7e28199220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.500+0000 7f7e2e913640 1 -- 172.21.15.29:0/2342851741 >> 172.21.15.29:0/2342851741 conn(0x7f7e280fe930 msgr2=0x7f7e28107850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.500+0000 7f7e2e913640 1 -- 172.21.15.29:0/2342851741 shutdown_connections 2024-03-11T19:16:21.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.500+0000 7f7e2e913640 1 -- 172.21.15.29:0/2342851741 wait complete. 2024-03-11T19:16:21.522 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:16:21.522 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:16:21.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.702+0000 7ffa1ffe9640 1 -- 172.21.15.29:0/916255333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa1810bbc0 msgr2=0x7ffa1810dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.702+0000 7ffa1ffe9640 1 --2- 172.21.15.29:0/916255333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa1810bbc0 0x7ffa1810dfb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ffa10009ca0 tx=0x7ffa1002c100 comp rx=0 tx=0).stop 2024-03-11T19:16:21.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.702+0000 7ffa1ffe9640 1 -- 172.21.15.29:0/916255333 shutdown_connections 2024-03-11T19:16:21.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.702+0000 7ffa1ffe9640 1 --2- 172.21.15.29:0/916255333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa1810e540 0x7ffa181109d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.702+0000 7ffa1ffe9640 1 --2- 172.21.15.29:0/916255333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa1810bbc0 0x7ffa1810dfb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.702+0000 7ffa1ffe9640 1 --2- 172.21.15.29:0/916255333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa18109290 0x7ffa1810b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.703+0000 7ffa1ffe9640 1 -- 172.21.15.29:0/916255333 >> 172.21.15.29:0/916255333 conn(0x7ffa1806dfe0 msgr2=0x7ffa18070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.703+0000 7ffa1ffe9640 1 -- 172.21.15.29:0/916255333 shutdown_connections 2024-03-11T19:16:21.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.703+0000 7ffa1ffe9640 1 -- 172.21.15.29:0/916255333 wait complete. 2024-03-11T19:16:21.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.703+0000 7ffa1ffe9640 1 Processor -- start 2024-03-11T19:16:21.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.704+0000 7ffa1ffe9640 1 -- start start 2024-03-11T19:16:21.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.704+0000 7ffa1ffe9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa18109290 0x7ffa181a1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.704+0000 7ffa1ffe9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa1810bbc0 0x7ffa181a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.704+0000 7ffa1ffe9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa1810e540 0x7ffa181a91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.704+0000 7ffa1ffe9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa181a96e0 con 0x7ffa18109290 2024-03-11T19:16:21.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.704+0000 7ffa1ffe9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa181a9850 con 0x7ffa1810bbc0 2024-03-11T19:16:21.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.704+0000 7ffa1ffe9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa181a9a10 con 0x7ffa1810e540 2024-03-11T19:16:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.704+0000 7ffa1dd85640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa18109290 0x7ffa181a1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.704+0000 7ffa1dd85640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa18109290 0x7ffa181a1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47316/0 (socket says 172.21.15.29:47316) 2024-03-11T19:16:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.704+0000 7ffa1dd85640 1 -- 172.21.15.29:0/3421310433 learned_addr learned my addr 172.21.15.29:0/3421310433 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.705+0000 7ffa1e586640 1 --2- 172.21.15.29:0/3421310433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa1810e540 0x7ffa181a91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.705+0000 7ffa1d584640 1 --2- 172.21.15.29:0/3421310433 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa1810bbc0 0x7ffa181a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.705+0000 7ffa1e586640 1 -- 172.21.15.29:0/3421310433 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa1810bbc0 msgr2=0x7ffa181a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.705+0000 7ffa1e586640 1 --2- 172.21.15.29:0/3421310433 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa1810bbc0 0x7ffa181a2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.705+0000 7ffa1e586640 1 -- 172.21.15.29:0/3421310433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa18109290 msgr2=0x7ffa181a1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.705+0000 7ffa1e586640 1 --2- 172.21.15.29:0/3421310433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa18109290 0x7ffa181a1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.705+0000 7ffa1e586640 1 -- 172.21.15.29:0/3421310433 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa10009900 con 0x7ffa1810e540 2024-03-11T19:16:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.705+0000 7ffa1e586640 1 --2- 172.21.15.29:0/3421310433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa1810e540 0x7ffa181a91a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ffa0800e940 tx=0x7ffa0800ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.706+0000 7ffa0effd640 1 -- 172.21.15.29:0/3421310433 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa08009620 con 0x7ffa1810e540 2024-03-11T19:16:21.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.706+0000 7ffa0effd640 1 -- 172.21.15.29:0/3421310433 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ffa080044d0 con 0x7ffa1810e540 2024-03-11T19:16:21.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.706+0000 7ffa0effd640 1 -- 172.21.15.29:0/3421310433 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa08004800 con 0x7ffa1810e540 2024-03-11T19:16:21.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.706+0000 7ffa1ffe9640 1 -- 172.21.15.29:0/3421310433 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa181a9cf0 con 0x7ffa1810e540 2024-03-11T19:16:21.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.706+0000 7ffa1ffe9640 1 -- 172.21.15.29:0/3421310433 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa181aa180 con 0x7ffa1810e540 2024-03-11T19:16:21.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.710+0000 7ffa0effd640 1 -- 172.21.15.29:0/3421310433 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ffa080049a0 con 0x7ffa1810e540 2024-03-11T19:16:21.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.711+0000 7ffa0effd640 1 --2- 172.21.15.29:0/3421310433 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ffa040a1c00 0x7ffa040a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.711+0000 7ffa0effd640 1 -- 172.21.15.29:0/3421310433 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ffa08014070 con 0x7ffa1810e540 2024-03-11T19:16:21.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.711+0000 7ffa22bc4740 1 Processor -- start 2024-03-11T19:16:21.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.711+0000 7ffa1dd85640 1 --2- 172.21.15.29:0/3421310433 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ffa040a1c00 0x7ffa040a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.712+0000 7ffa22bc4740 1 -- start start 2024-03-11T19:16:21.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.712+0000 7ffa22bc4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567050ffa80 0x5567050ffea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.712+0000 7ffa22bc4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556705100470 0x556705104f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.712+0000 7ffa22bc4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5567051055e0 0x556705107a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.712+0000 7ffa22bc4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567050b3960 con 0x5567051055e0 2024-03-11T19:16:21.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.712+0000 7ffa22bc4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556705067310 con 0x5567050ffa80 2024-03-11T19:16:21.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.712+0000 7ffa22bc4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556705064840 con 0x556705100470 2024-03-11T19:16:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.712+0000 7ffa1dd85640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567050ffa80 0x5567050ffea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.712+0000 7ffa1dd85640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567050ffa80 0x5567050ffea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44510/0 (socket says 172.21.15.29:44510) 2024-03-11T19:16:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.712+0000 7ffa1dd85640 1 -- 172.21.15.29:0/2625370723 learned_addr learned my addr 172.21.15.29:0/2625370723 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.712+0000 7ffa1d584640 1 --2- 172.21.15.29:0/2625370723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556705100470 0x556705104f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.712+0000 7ffa1e586640 1 --2- 172.21.15.29:0/2625370723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5567051055e0 0x556705107a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.712+0000 7ffa1dd85640 1 --2- 172.21.15.29:0/3421310433 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ffa040a1c00 0x7ffa040a40c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ffa14012590 tx=0x7ffa14012990 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.712+0000 7ffa1dd85640 1 -- 172.21.15.29:0/2625370723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556705100470 msgr2=0x556705104f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.713+0000 7ffa1dd85640 1 --2- 172.21.15.29:0/2625370723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556705100470 0x556705104f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.713+0000 7ffa1dd85640 1 -- 172.21.15.29:0/2625370723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5567051055e0 msgr2=0x556705107a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.713+0000 7ffa1dd85640 1 --2- 172.21.15.29:0/2625370723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5567051055e0 0x556705107a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.713+0000 7ffa1dd85640 1 -- 172.21.15.29:0/2625370723 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556705022060 con 0x5567050ffa80 2024-03-11T19:16:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.713+0000 7ffa1d584640 1 --2- 172.21.15.29:0/2625370723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556705100470 0x556705104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:16:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.713+0000 7ffa1dd85640 1 --2- 172.21.15.29:0/2625370723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567050ffa80 0x5567050ffea0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ffa1400e8d0 tx=0x7ffa1401e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b95241dca97dec2b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.713+0000 7ffa03fff640 1 -- 172.21.15.29:0/2625370723 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa14005220 con 0x5567050ffa80 2024-03-11T19:16:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.713+0000 7ffa03fff640 1 -- 172.21.15.29:0/2625370723 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ffa1401e830 con 0x5567050ffa80 2024-03-11T19:16:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.713+0000 7ffa03fff640 1 -- 172.21.15.29:0/2625370723 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa14004a00 con 0x5567050ffa80 2024-03-11T19:16:21.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.714+0000 7ffa22bc4740 1 -- 172.21.15.29:0/2625370723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567050ffa80 msgr2=0x5567050ffea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.714+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/2625370723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567050ffa80 0x5567050ffea0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ffa1400e8d0 tx=0x7ffa1401e000 comp rx=0 tx=0).stop 2024-03-11T19:16:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.714+0000 7ffa22bc4740 1 -- 172.21.15.29:0/2625370723 shutdown_connections 2024-03-11T19:16:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.714+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/2625370723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5567051055e0 0x556705107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.714+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/2625370723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556705100470 0x556705104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.714+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/2625370723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567050ffa80 0x5567050ffea0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.714+0000 7ffa22bc4740 1 -- 172.21.15.29:0/2625370723 >> 172.21.15.29:0/2625370723 conn(0x5567050faff0 msgr2=0x5567050fb3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.715+0000 7ffa22bc4740 1 -- 172.21.15.29:0/2625370723 shutdown_connections 2024-03-11T19:16:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.715+0000 7ffa22bc4740 1 -- 172.21.15.29:0/2625370723 wait complete. 2024-03-11T19:16:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.715+0000 7ffa22bc4740 1 -- start start 2024-03-11T19:16:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.715+0000 7ffa22bc4740 1 Processor -- start 2024-03-11T19:16:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.715+0000 7ffa22bc4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5567050ffa80 0x556705109c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.716+0000 7ffa22bc4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556705100470 0x55670510a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.716+0000 7ffa22bc4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567051055e0 0x556705111190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.716+0000 7ffa22bc4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556705060020 con 0x556705100470 2024-03-11T19:16:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.716+0000 7ffa22bc4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556704d33430 con 0x5567051055e0 2024-03-11T19:16:21.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.716+0000 7ffa22bc4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556704cefe00 con 0x5567050ffa80 2024-03-11T19:16:21.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.716+0000 7ffa22bc4740 10 client.0 _ll_register_callbacks cb 0x556705007cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:21.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.716+0000 7ffa1dd85640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5567050ffa80 0x556705109c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.716+0000 7ffa1dd85640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5567050ffa80 0x556705109c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:58110/0 (socket says 172.21.15.29:58110) 2024-03-11T19:16:21.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.716+0000 7ffa1dd85640 1 -- 172.21.15.29:0/1122300065 learned_addr learned my addr 172.21.15.29:0/1122300065 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.716+0000 7ffa1e586640 1 --2- 172.21.15.29:0/1122300065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567051055e0 0x556705111190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.716+0000 7ffa1d584640 1 --2- 172.21.15.29:0/1122300065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556705100470 0x55670510a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.716+0000 7ffa1dd85640 1 -- 172.21.15.29:0/1122300065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567051055e0 msgr2=0x556705111190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.716+0000 7ffa1dd85640 1 --2- 172.21.15.29:0/1122300065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567051055e0 0x556705111190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.716+0000 7ffa1dd85640 1 -- 172.21.15.29:0/1122300065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556705100470 msgr2=0x55670510a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.716+0000 7ffa1dd85640 1 --2- 172.21.15.29:0/1122300065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556705100470 0x55670510a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.716+0000 7ffa1dd85640 1 -- 172.21.15.29:0/1122300065 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1400e690 con 0x5567050ffa80 2024-03-11T19:16:21.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.717+0000 7ffa1dd85640 1 --2- 172.21.15.29:0/1122300065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5567050ffa80 0x556705109c80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffa14012e90 tx=0x7ffa1401ea20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e436fcc71b42a2c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.717+0000 7ffa017fa640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:16:21.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.717+0000 7ffa017fa640 1 -- 172.21.15.29:0/1122300065 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa14005240 con 0x5567050ffa80 2024-03-11T19:16:21.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.717+0000 7ffa22bc4740 1 -- 172.21.15.29:0/1122300065 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556704d5f7e0 con 0x5567050ffa80 2024-03-11T19:16:21.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.717+0000 7ffa017fa640 1 -- 172.21.15.29:0/1122300065 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ffa140053a0 con 0x5567050ffa80 2024-03-11T19:16:21.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.717+0000 7ffa017fa640 1 -- 172.21.15.29:0/1122300065 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa14005560 con 0x5567050ffa80 2024-03-11T19:16:21.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.717+0000 7ffa22bc4740 10 client.26521 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:21.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.717+0000 7ffa22bc4740 1 -- 172.21.15.29:0/1122300065 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55670505e6a0 con 0x5567050ffa80 2024-03-11T19:16:21.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.717+0000 7ffa017fa640 1 -- 172.21.15.29:0/1122300065 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x556704d5f7e0 con 0x5567050ffa80 2024-03-11T19:16:21.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.717+0000 7ffa017fa640 1 -- 172.21.15.29:0/1122300065 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ffa140286c0 con 0x5567050ffa80 2024-03-11T19:16:21.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.718+0000 7ffa22bc4740 10 client.26521 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:21.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.718+0000 7ffa22bc4740 10 client.26521 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:16:21.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.718+0000 7ffa22bc4740 20 client.26521 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:21.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.718+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/1122300065 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x556705115cf0 0x556705118110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.718+0000 7ffa22bc4740 4 client.26521 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:16:21.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.718+0000 7ffa22bc4740 1 -- 172.21.15.29:0/1122300065 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556704d9c720 con 0x556705115cf0 2024-03-11T19:16:21.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.718+0000 7ffa1d584640 1 --2- 172.21.15.29:0/1122300065 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x556705115cf0 0x556705118110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.719+0000 7ffa1d584640 1 --2- 172.21.15.29:0/1122300065 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x556705115cf0 0x556705118110 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5b6043cfac886e74 server_cookie=a9ff216631d9950c in_seq=0 out_seq=0 2024-03-11T19:16:21.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.719+0000 7ffa017fa640 10 client.26521 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:16:21.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.719+0000 7ffa017fa640 1 -- 172.21.15.29:0/1122300065 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556704d9c720 con 0x556705115cf0 2024-03-11T19:16:21.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.719+0000 7ffa017fa640 10 client.26521 handle_command_reply: tid=0 2024-03-11T19:16:21.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.719+0000 7ffa22bc4740 1 client.26521 shutdown 2024-03-11T19:16:21.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.719+0000 7ffa22bc4740 1 -- 172.21.15.29:0/1122300065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5567050ffa80 msgr2=0x556705109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.719+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/1122300065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5567050ffa80 0x556705109c80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffa14012e90 tx=0x7ffa1401ea20 comp rx=0 tx=0).stop 2024-03-11T19:16:21.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.720+0000 7ffa22bc4740 1 -- 172.21.15.29:0/1122300065 shutdown_connections 2024-03-11T19:16:21.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.720+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/1122300065 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x556705115cf0 0x556705118110 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.720+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/1122300065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567051055e0 0x556705111190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.720+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/1122300065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556705100470 0x55670510a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.720+0000 7ffa017fa640 0 client.26521 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:16:21.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.720+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/1122300065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5567050ffa80 0x556705109c80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.720+0000 7ffa22bc4740 1 -- 172.21.15.29:0/1122300065 >> 172.21.15.29:0/1122300065 conn(0x5567050faff0 msgr2=0x5567050fc240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.720+0000 7ffa22bc4740 1 -- 172.21.15.29:0/1122300065 shutdown_connections 2024-03-11T19:16:21.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.720+0000 7ffa22bc4740 1 -- 172.21.15.29:0/1122300065 wait complete. 2024-03-11T19:16:21.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.720+0000 7ffa22bc4740 20 client.26521 trim_cache size 0 max 16384 2024-03-11T19:16:21.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.741+0000 7ffa22bc4740 1 Processor -- start 2024-03-11T19:16:21.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.741+0000 7ffa22bc4740 1 -- start start 2024-03-11T19:16:21.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.742+0000 7ffa22bc4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5567050ffa80 0x5567050eda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.742+0000 7ffa22bc4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556705100470 0x556705111150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.742+0000 7ffa22bc4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567051055e0 0x5567050fb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.742+0000 7ffa22bc4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556704fe5130 con 0x556705100470 2024-03-11T19:16:21.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.742+0000 7ffa22bc4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567050ee320 con 0x5567051055e0 2024-03-11T19:16:21.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.742+0000 7ffa22bc4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567050fad60 con 0x5567050ffa80 2024-03-11T19:16:21.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.742+0000 7ffa1dd85640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5567050ffa80 0x5567050eda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.742+0000 7ffa1dd85640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5567050ffa80 0x5567050eda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:58114/0 (socket says 172.21.15.29:58114) 2024-03-11T19:16:21.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.742+0000 7ffa1dd85640 1 -- 172.21.15.29:0/4036736475 learned_addr learned my addr 172.21.15.29:0/4036736475 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.743+0000 7ffa1dd85640 1 -- 172.21.15.29:0/4036736475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567051055e0 msgr2=0x5567050fb3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:16:21.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.743+0000 7ffa1dd85640 1 --2- 172.21.15.29:0/4036736475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567051055e0 0x5567050fb3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.743+0000 7ffa1dd85640 1 -- 172.21.15.29:0/4036736475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556705100470 msgr2=0x556705111150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:16:21.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.743+0000 7ffa1dd85640 1 --2- 172.21.15.29:0/4036736475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556705100470 0x556705111150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.743+0000 7ffa1dd85640 1 -- 172.21.15.29:0/4036736475 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556704ffac20 con 0x5567050ffa80 2024-03-11T19:16:21.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.744+0000 7ffa1dd85640 1 --2- 172.21.15.29:0/4036736475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5567050ffa80 0x5567050eda10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffa1400bbd0 tx=0x7ffa1400eed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9afee4b23cb7dda1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.744+0000 7ffa037fe640 1 -- 172.21.15.29:0/4036736475 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa14040070 con 0x5567050ffa80 2024-03-11T19:16:21.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.744+0000 7ffa037fe640 1 -- 172.21.15.29:0/4036736475 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ffa1400be30 con 0x5567050ffa80 2024-03-11T19:16:21.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.744+0000 7ffa037fe640 1 -- 172.21.15.29:0/4036736475 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa14005610 con 0x5567050ffa80 2024-03-11T19:16:21.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.745+0000 7ffa22bc4740 1 -- 172.21.15.29:0/4036736475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5567050ffa80 msgr2=0x5567050eda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.745+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/4036736475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5567050ffa80 0x5567050eda10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffa1400bbd0 tx=0x7ffa1400eed0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.746+0000 7ffa22bc4740 1 -- 172.21.15.29:0/4036736475 shutdown_connections 2024-03-11T19:16:21.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.746+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/4036736475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567051055e0 0x5567050fb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.746+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/4036736475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556705100470 0x556705111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.746+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/4036736475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5567050ffa80 0x5567050eda10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.746+0000 7ffa22bc4740 1 -- 172.21.15.29:0/4036736475 >> 172.21.15.29:0/4036736475 conn(0x5567050faff0 msgr2=0x5567051079d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.746+0000 7ffa22bc4740 1 -- 172.21.15.29:0/4036736475 shutdown_connections 2024-03-11T19:16:21.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.746+0000 7ffa22bc4740 1 -- 172.21.15.29:0/4036736475 wait complete. 2024-03-11T19:16:21.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.747+0000 7ffa22bc4740 1 -- start start 2024-03-11T19:16:21.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.747+0000 7ffa22bc4740 1 Processor -- start 2024-03-11T19:16:21.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.748+0000 7ffa22bc4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5567050ffa80 0x556705110f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.748+0000 7ffa22bc4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556705100470 0x55670510f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.748+0000 7ffa22bc4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567051055e0 0x55670510faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.748+0000 7ffa22bc4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567051100b0 con 0x556705100470 2024-03-11T19:16:21.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.748+0000 7ffa22bc4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556705110220 con 0x5567051055e0 2024-03-11T19:16:21.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.748+0000 7ffa1e586640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567051055e0 0x55670510faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.748+0000 7ffa22bc4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567051103e0 con 0x5567050ffa80 2024-03-11T19:16:21.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.748+0000 7ffa1e586640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567051055e0 0x55670510faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44528/0 (socket says 172.21.15.29:44528) 2024-03-11T19:16:21.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.748+0000 7ffa1e586640 1 -- 172.21.15.29:0/3788107722 learned_addr learned my addr 172.21.15.29:0/3788107722 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.748+0000 7ffa22bc4740 10 client.0 _ll_register_callbacks cb 0x7ffa1400bb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:21.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.749+0000 7ffa1e586640 1 -- 172.21.15.29:0/3788107722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5567050ffa80 msgr2=0x556705110f30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:16:21.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.749+0000 7ffa1e586640 1 --2- 172.21.15.29:0/3788107722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5567050ffa80 0x556705110f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.749+0000 7ffa1e586640 1 -- 172.21.15.29:0/3788107722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556705100470 msgr2=0x55670510f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.749+0000 7ffa1e586640 1 --2- 172.21.15.29:0/3788107722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556705100470 0x55670510f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.749+0000 7ffa1e586640 1 -- 172.21.15.29:0/3788107722 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55670505e6a0 con 0x5567051055e0 2024-03-11T19:16:21.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.749+0000 7ffa1e586640 1 --2- 172.21.15.29:0/3788107722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567051055e0 0x55670510faf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ffa080965c0 tx=0x7ffa08099dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32207ee51e5493e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.750+0000 7ffa017fa640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:16:21.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.750+0000 7ffa017fa640 1 -- 172.21.15.29:0/3788107722 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0809a160 con 0x5567051055e0 2024-03-11T19:16:21.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.750+0000 7ffa22bc4740 1 -- 172.21.15.29:0/3788107722 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556704deb890 con 0x5567051055e0 2024-03-11T19:16:21.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.750+0000 7ffa017fa640 1 -- 172.21.15.29:0/3788107722 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ffa0809e2b0 con 0x5567051055e0 2024-03-11T19:16:21.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.750+0000 7ffa017fa640 1 -- 172.21.15.29:0/3788107722 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa080a2900 con 0x5567051055e0 2024-03-11T19:16:21.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.751+0000 7ffa22bc4740 10 client.26498 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:21.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.751+0000 7ffa22bc4740 1 -- 172.21.15.29:0/3788107722 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5567050208c0 con 0x5567051055e0 2024-03-11T19:16:21.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.751+0000 7ffa017fa640 1 -- 172.21.15.29:0/3788107722 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x556704deb890 con 0x5567051055e0 2024-03-11T19:16:21.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.751+0000 7ffa017fa640 1 -- 172.21.15.29:0/3788107722 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ffa080995b0 con 0x5567051055e0 2024-03-11T19:16:21.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.751+0000 7ffa22bc4740 10 client.26498 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:21.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.751+0000 7ffa22bc4740 10 client.26498 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:16:21.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.751+0000 7ffa22bc4740 20 client.26498 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:21.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.751+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/3788107722 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55670516da50 0x55670516fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.751+0000 7ffa22bc4740 4 client.26498 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:16:21.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.751+0000 7ffa22bc4740 1 -- 172.21.15.29:0/3788107722 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556704d0b700 con 0x55670516da50 2024-03-11T19:16:21.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.752+0000 7ffa1dd85640 1 --2- 172.21.15.29:0/3788107722 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55670516da50 0x55670516fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.753+0000 7ffa1dd85640 1 --2- 172.21.15.29:0/3788107722 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55670516da50 0x55670516fe70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=56f1bb84d4d6a33a server_cookie=1c4d4ef63dc15570 in_seq=0 out_seq=0 2024-03-11T19:16:21.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.753+0000 7ffa017fa640 10 client.26498 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:16:21.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.753+0000 7ffa017fa640 1 -- 172.21.15.29:0/3788107722 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556704d0b700 con 0x55670516da50 2024-03-11T19:16:21.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.753+0000 7ffa017fa640 10 client.26498 handle_command_reply: tid=0 2024-03-11T19:16:21.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.753+0000 7ffa22bc4740 1 client.26498 shutdown 2024-03-11T19:16:21.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.754+0000 7ffa22bc4740 1 -- 172.21.15.29:0/3788107722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567051055e0 msgr2=0x55670510faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.754+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/3788107722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567051055e0 0x55670510faf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ffa080965c0 tx=0x7ffa08099dd0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.754+0000 7ffa22bc4740 1 -- 172.21.15.29:0/3788107722 shutdown_connections 2024-03-11T19:16:21.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.754+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/3788107722 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55670516da50 0x55670516fe70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.754+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/3788107722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5567051055e0 0x55670510faf0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.754+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/3788107722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556705100470 0x55670510f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.754+0000 7ffa22bc4740 1 --2- 172.21.15.29:0/3788107722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5567050ffa80 0x556705110f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.754+0000 7ffa22bc4740 1 -- 172.21.15.29:0/3788107722 >> 172.21.15.29:0/3788107722 conn(0x5567050faff0 msgr2=0x5567050eda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.754+0000 7ffa017fa640 0 client.26498 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:16:21.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.754+0000 7ffa22bc4740 1 -- 172.21.15.29:0/3788107722 shutdown_connections 2024-03-11T19:16:21.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.754+0000 7ffa22bc4740 1 -- 172.21.15.29:0/3788107722 wait complete. 2024-03-11T19:16:21.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.755+0000 7ffa22bc4740 20 client.26498 trim_cache size 0 max 16384 2024-03-11T19:16:21.760 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:16:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.756+0000 7ffa1ffe9640 1 -- 172.21.15.29:0/3421310433 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ffa040a1c00 msgr2=0x7ffa040a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.756+0000 7ffa1ffe9640 1 --2- 172.21.15.29:0/3421310433 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ffa040a1c00 0x7ffa040a40c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ffa14012590 tx=0x7ffa14012990 comp rx=0 tx=0).stop 2024-03-11T19:16:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.756+0000 7ffa1ffe9640 1 -- 172.21.15.29:0/3421310433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa1810e540 msgr2=0x7ffa181a91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.756+0000 7ffa1ffe9640 1 --2- 172.21.15.29:0/3421310433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa1810e540 0x7ffa181a91a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ffa0800e940 tx=0x7ffa0800ee00 comp rx=0 tx=0).stop 2024-03-11T19:16:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.756+0000 7ffa1ffe9640 1 -- 172.21.15.29:0/3421310433 shutdown_connections 2024-03-11T19:16:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.756+0000 7ffa1ffe9640 1 --2- 172.21.15.29:0/3421310433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa1810e540 0x7ffa181a91a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.756+0000 7ffa1ffe9640 1 --2- 172.21.15.29:0/3421310433 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa1810bbc0 0x7ffa181a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.756+0000 7ffa1ffe9640 1 --2- 172.21.15.29:0/3421310433 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ffa040a1c00 0x7ffa040a40c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.756+0000 7ffa1ffe9640 1 --2- 172.21.15.29:0/3421310433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa18109290 0x7ffa181a1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.756+0000 7ffa1ffe9640 1 -- 172.21.15.29:0/3421310433 >> 172.21.15.29:0/3421310433 conn(0x7ffa1806dfe0 msgr2=0x7ffa1810ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.757+0000 7ffa1ffe9640 1 -- 172.21.15.29:0/3421310433 shutdown_connections 2024-03-11T19:16:21.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.757+0000 7ffa1ffe9640 1 -- 172.21.15.29:0/3421310433 wait complete. 2024-03-11T19:16:21.778 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:16:21.779 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:16:21.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.959+0000 7f698241f640 1 -- 172.21.15.29:0/4225235826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f697c0724d0 msgr2=0x7f697c0728f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.959+0000 7f698241f640 1 --2- 172.21.15.29:0/4225235826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f697c0724d0 0x7f697c0728f0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f6978009900 tx=0x7f697802bc30 comp rx=0 tx=0).stop 2024-03-11T19:16:21.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.959+0000 7f698241f640 1 -- 172.21.15.29:0/4225235826 shutdown_connections 2024-03-11T19:16:21.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.959+0000 7f698241f640 1 --2- 172.21.15.29:0/4225235826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f697c109120 0x7f697c10d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.959+0000 7f698241f640 1 --2- 172.21.15.29:0/4225235826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f697c108780 0x7f697c108be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.959+0000 7f698241f640 1 --2- 172.21.15.29:0/4225235826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f697c0724d0 0x7f697c0728f0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.959+0000 7f698241f640 1 -- 172.21.15.29:0/4225235826 >> 172.21.15.29:0/4225235826 conn(0x7f697c06dfa0 msgr2=0x7f697c0703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.961+0000 7f698241f640 1 -- 172.21.15.29:0/4225235826 shutdown_connections 2024-03-11T19:16:21.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.961+0000 7f698241f640 1 -- 172.21.15.29:0/4225235826 wait complete. 2024-03-11T19:16:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.962+0000 7f698241f640 1 Processor -- start 2024-03-11T19:16:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.962+0000 7f698241f640 1 -- start start 2024-03-11T19:16:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.962+0000 7f698241f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f697c0724d0 0x7f697c1a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.963+0000 7f698241f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f697c108780 0x7f697c1a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.963+0000 7f698241f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f697c109120 0x7f697c1ad310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.963+0000 7f698241f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697c1ad850 con 0x7f697c0724d0 2024-03-11T19:16:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.963+0000 7f698241f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697c1ad9c0 con 0x7f697c109120 2024-03-11T19:16:21.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.963+0000 7f698241f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697c1adb80 con 0x7f697c108780 2024-03-11T19:16:21.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.963+0000 7f6980c1c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f697c108780 0x7f697c1a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.963+0000 7f6980c1c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f697c108780 0x7f697c1a62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:58142/0 (socket says 172.21.15.29:58142) 2024-03-11T19:16:21.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.963+0000 7f6980c1c640 1 -- 172.21.15.29:0/3939866287 learned_addr learned my addr 172.21.15.29:0/3939866287 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.963+0000 7f6981c1e640 1 --2- 172.21.15.29:0/3939866287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f697c109120 0x7f697c1ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.963+0000 7f6980c1c640 1 -- 172.21.15.29:0/3939866287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f697c109120 msgr2=0x7f697c1ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.963+0000 7f6980c1c640 1 --2- 172.21.15.29:0/3939866287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f697c109120 0x7f697c1ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.963+0000 7f6980c1c640 1 -- 172.21.15.29:0/3939866287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f697c0724d0 msgr2=0x7f697c1a5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.963+0000 7f6980c1c640 1 --2- 172.21.15.29:0/3939866287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f697c0724d0 0x7f697c1a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.963+0000 7f6980c1c640 1 -- 172.21.15.29:0/3939866287 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6978009560 con 0x7f697c108780 2024-03-11T19:16:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.964+0000 7f6980c1c640 1 --2- 172.21.15.29:0/3939866287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f697c108780 0x7f697c1a62a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f696c00d950 tx=0x7f696c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.964+0000 7f69727fc640 1 -- 172.21.15.29:0/3939866287 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696c009c30 con 0x7f697c108780 2024-03-11T19:16:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.964+0000 7f69727fc640 1 -- 172.21.15.29:0/3939866287 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f696c009d90 con 0x7f697c108780 2024-03-11T19:16:21.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.964+0000 7f698241f640 1 -- 172.21.15.29:0/3939866287 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f697c1ade60 con 0x7f697c108780 2024-03-11T19:16:21.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.964+0000 7f698241f640 1 -- 172.21.15.29:0/3939866287 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f697c1ae2e0 con 0x7f697c108780 2024-03-11T19:16:21.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.966+0000 7f69727fc640 1 -- 172.21.15.29:0/3939866287 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696c00f6b0 con 0x7f697c108780 2024-03-11T19:16:21.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.968+0000 7f69727fc640 1 -- 172.21.15.29:0/3939866287 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f696c00f850 con 0x7f697c108780 2024-03-11T19:16:21.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.970+0000 7f69727fc640 1 --2- 172.21.15.29:0/3939866287 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f69680a1c30 0x7f69680a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.971+0000 7f6986254740 1 Processor -- start 2024-03-11T19:16:21.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.971+0000 7f6986254740 1 -- start start 2024-03-11T19:16:21.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.971+0000 7f6986254740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e66480 0x55dca8e668a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.971+0000 7f6986254740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dca8e66e70 0x55dca8e6b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.971+0000 7f6986254740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e6bfe0 0x55dca8e6e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.971+0000 7f6986254740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8e22960 con 0x55dca8e66e70 2024-03-11T19:16:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.971+0000 7f6986254740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8dd6310 con 0x55dca8e6bfe0 2024-03-11T19:16:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.971+0000 7f6986254740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8dd3840 con 0x55dca8e66480 2024-03-11T19:16:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.971+0000 7f69727fc640 1 -- 172.21.15.29:0/3939866287 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f696c0cdee0 con 0x7f697c108780 2024-03-11T19:16:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.971+0000 7f698141d640 1 --2- 172.21.15.29:0/3939866287 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f69680a1c30 0x7f69680a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.971+0000 7f6981c1e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e6bfe0 0x55dca8e6e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.971+0000 7f6981c1e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e6bfe0 0x55dca8e6e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44546/0 (socket says 172.21.15.29:44546) 2024-03-11T19:16:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.971+0000 7f6981c1e640 1 -- 172.21.15.29:0/965361281 learned_addr learned my addr 172.21.15.29:0/965361281 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.971+0000 7f6980c1c640 1 --2- 172.21.15.29:0/965361281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dca8e66e70 0x55dca8e6b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.972+0000 7f698141d640 1 --2- 172.21.15.29:0/965361281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e66480 0x55dca8e668a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.972+0000 7f6981c1e640 1 -- 172.21.15.29:0/965361281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e66480 msgr2=0x55dca8e668a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.972+0000 7f6981c1e640 1 --2- 172.21.15.29:0/965361281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e66480 0x55dca8e668a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.972+0000 7f6981c1e640 1 -- 172.21.15.29:0/965361281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dca8e66e70 msgr2=0x55dca8e6b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.972+0000 7f6981c1e640 1 --2- 172.21.15.29:0/965361281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dca8e66e70 0x55dca8e6b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.972+0000 7f6981c1e640 1 -- 172.21.15.29:0/965361281 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dca8d91060 con 0x55dca8e6bfe0 2024-03-11T19:16:21.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.972+0000 7f698141d640 1 --2- 172.21.15.29:0/3939866287 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f69680a1c30 0x7f69680a40f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6978005ab0 tx=0x7f697802ccc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.972+0000 7f6981c1e640 1 --2- 172.21.15.29:0/965361281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e6bfe0 0x55dca8e6e470 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f69740039f0 tx=0x7f6974003eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4e74d2ba75c57d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.972+0000 7f69677fe640 1 -- 172.21.15.29:0/965361281 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697401a470 con 0x55dca8e6bfe0 2024-03-11T19:16:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.973+0000 7f69677fe640 1 -- 172.21.15.29:0/965361281 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f697401aa80 con 0x55dca8e6bfe0 2024-03-11T19:16:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.973+0000 7f69677fe640 1 -- 172.21.15.29:0/965361281 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6974011ae0 con 0x55dca8e6bfe0 2024-03-11T19:16:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.973+0000 7f6986254740 1 -- 172.21.15.29:0/965361281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e6bfe0 msgr2=0x55dca8e6e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.973+0000 7f6986254740 1 --2- 172.21.15.29:0/965361281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e6bfe0 0x55dca8e6e470 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f69740039f0 tx=0x7f6974003eb0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.974+0000 7f6986254740 1 -- 172.21.15.29:0/965361281 shutdown_connections 2024-03-11T19:16:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.974+0000 7f6986254740 1 --2- 172.21.15.29:0/965361281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e6bfe0 0x55dca8e6e470 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.974+0000 7f6986254740 1 --2- 172.21.15.29:0/965361281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dca8e66e70 0x55dca8e6b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.974+0000 7f6986254740 1 --2- 172.21.15.29:0/965361281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e66480 0x55dca8e668a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.974+0000 7f6986254740 1 -- 172.21.15.29:0/965361281 >> 172.21.15.29:0/965361281 conn(0x55dca8e619f0 msgr2=0x55dca8e61df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.974+0000 7f6986254740 1 -- 172.21.15.29:0/965361281 shutdown_connections 2024-03-11T19:16:21.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.974+0000 7f6986254740 1 -- 172.21.15.29:0/965361281 wait complete. 2024-03-11T19:16:21.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.974+0000 7f6986254740 1 -- start start 2024-03-11T19:16:21.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.974+0000 7f6986254740 1 Processor -- start 2024-03-11T19:16:21.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.975+0000 7f6986254740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e66480 0x55dca8e70420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.975+0000 7f6986254740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dca8e66e70 0x55dca8e70960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.975+0000 7f6986254740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e6bfe0 0x55dca8e779a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.975+0000 7f6986254740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8dcf020 con 0x55dca8e66e70 2024-03-11T19:16:21.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.975+0000 7f698141d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e66480 0x55dca8e70420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.975+0000 7f698141d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e66480 0x55dca8e70420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:58160/0 (socket says 172.21.15.29:58160) 2024-03-11T19:16:21.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.975+0000 7f698141d640 1 -- 172.21.15.29:0/2547217335 learned_addr learned my addr 172.21.15.29:0/2547217335 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:21.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.975+0000 7f6980c1c640 1 --2- 172.21.15.29:0/2547217335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dca8e66e70 0x55dca8e70960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.975+0000 7f6981c1e640 1 --2- 172.21.15.29:0/2547217335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e6bfe0 0x55dca8e779a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.976+0000 7f6986254740 1 -- 172.21.15.29:0/2547217335 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8aa2430 con 0x55dca8e6bfe0 2024-03-11T19:16:21.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.976+0000 7f6986254740 1 -- 172.21.15.29:0/2547217335 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8a5ee00 con 0x55dca8e66480 2024-03-11T19:16:21.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.976+0000 7f6986254740 10 client.0 _ll_register_callbacks cb 0x55dca8d76cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:21.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.976+0000 7f698141d640 1 -- 172.21.15.29:0/2547217335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e6bfe0 msgr2=0x55dca8e779a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.976+0000 7f698141d640 1 --2- 172.21.15.29:0/2547217335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e6bfe0 0x55dca8e779a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.976+0000 7f698141d640 1 -- 172.21.15.29:0/2547217335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dca8e66e70 msgr2=0x55dca8e70960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:21.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.976+0000 7f698141d640 1 --2- 172.21.15.29:0/2547217335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dca8e66e70 0x55dca8e70960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.976+0000 7f698141d640 1 -- 172.21.15.29:0/2547217335 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6974009c90 con 0x55dca8e66480 2024-03-11T19:16:21.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.976+0000 7f698141d640 1 --2- 172.21.15.29:0/2547217335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e66480 0x55dca8e70420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6978035000 tx=0x7f6978005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0d3e161c997b306 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:21.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.976+0000 7f6964ff9640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:16:21.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.977+0000 7f6964ff9640 1 -- 172.21.15.29:0/2547217335 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6978003ac0 con 0x55dca8e66480 2024-03-11T19:16:21.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.977+0000 7f6964ff9640 1 -- 172.21.15.29:0/2547217335 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f697802de40 con 0x55dca8e66480 2024-03-11T19:16:21.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.977+0000 7f6986254740 1 -- 172.21.15.29:0/2547217335 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dca8ace7e0 con 0x55dca8e66480 2024-03-11T19:16:21.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.977+0000 7f6964ff9640 1 -- 172.21.15.29:0/2547217335 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6978003c20 con 0x55dca8e66480 2024-03-11T19:16:21.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.977+0000 7f6986254740 10 client.26536 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:21.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.977+0000 7f6986254740 1 -- 172.21.15.29:0/2547217335 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55dca8dcd6a0 con 0x55dca8e66480 2024-03-11T19:16:21.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.977+0000 7f6964ff9640 1 -- 172.21.15.29:0/2547217335 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55dca8ace7e0 con 0x55dca8e66480 2024-03-11T19:16:21.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.977+0000 7f6964ff9640 1 -- 172.21.15.29:0/2547217335 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6978056070 con 0x55dca8e66480 2024-03-11T19:16:21.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.978+0000 7f6986254740 10 client.26536 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.978+0000 7f6986254740 10 client.26536 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:16:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.978+0000 7f6986254740 20 client.26536 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.978+0000 7f6986254740 1 --2- 172.21.15.29:0/2547217335 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55dca8e7c620 0x55dca8e7ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.978+0000 7f6986254740 4 client.26536 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:16:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.978+0000 7f6986254740 1 -- 172.21.15.29:0/2547217335 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dca8b0b720 con 0x55dca8e7c620 2024-03-11T19:16:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.978+0000 7f6980c1c640 1 --2- 172.21.15.29:0/2547217335 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55dca8e7c620 0x55dca8e7ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.978+0000 7f6980c1c640 1 --2- 172.21.15.29:0/2547217335 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55dca8e7c620 0x55dca8e7ea80 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c47d860107e5c78 server_cookie=f286e245a0cac37f in_seq=0 out_seq=0 2024-03-11T19:16:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.978+0000 7f6964ff9640 10 client.26536 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.979+0000 7f6964ff9640 1 -- 172.21.15.29:0/2547217335 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55dca8b0b720 con 0x55dca8e7c620 2024-03-11T19:16:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.979+0000 7f6964ff9640 10 client.26536 handle_command_reply: tid=0 2024-03-11T19:16:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.979+0000 7f6986254740 1 client.26536 shutdown 2024-03-11T19:16:21.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.980+0000 7f6986254740 1 -- 172.21.15.29:0/2547217335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e66480 msgr2=0x55dca8e70420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:21.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.980+0000 7f6986254740 1 --2- 172.21.15.29:0/2547217335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e66480 0x55dca8e70420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6978035000 tx=0x7f6978005fd0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.980+0000 7f6986254740 1 -- 172.21.15.29:0/2547217335 shutdown_connections 2024-03-11T19:16:21.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.980+0000 7f6986254740 1 --2- 172.21.15.29:0/2547217335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e6bfe0 0x55dca8e779a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.980+0000 7f6986254740 1 --2- 172.21.15.29:0/2547217335 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55dca8e7c620 0x55dca8e7ea80 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.980+0000 7f6986254740 1 --2- 172.21.15.29:0/2547217335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dca8e66e70 0x55dca8e70960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.980+0000 7f6986254740 1 --2- 172.21.15.29:0/2547217335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e66480 0x55dca8e70420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:21.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.980+0000 7f6964ff9640 0 client.26536 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:21.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.980+0000 7f6986254740 1 -- 172.21.15.29:0/2547217335 >> 172.21.15.29:0/2547217335 conn(0x55dca8e619f0 msgr2=0x55dca8e60cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:21.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.980+0000 7f6986254740 1 -- 172.21.15.29:0/2547217335 shutdown_connections 2024-03-11T19:16:21.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.980+0000 7f6986254740 1 -- 172.21.15.29:0/2547217335 wait complete. 2024-03-11T19:16:21.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:21.980+0000 7f6986254740 20 client.26536 trim_cache size 0 max 16384 2024-03-11T19:16:22.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.005+0000 7f6986254740 1 Processor -- start 2024-03-11T19:16:22.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.005+0000 7f6986254740 1 -- start start 2024-03-11T19:16:22.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.005+0000 7f6986254740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e66480 0x55dca8e64c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:22.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.005+0000 7f6986254740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dca8e66e70 0x55dca8e77930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:22.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.005+0000 7f6986254740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e6bfe0 0x55dca8e65190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:22.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.005+0000 7f6986254740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8d54130 con 0x55dca8e66e70 2024-03-11T19:16:22.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.005+0000 7f6986254740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8e71490 con 0x55dca8e66480 2024-03-11T19:16:22.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.005+0000 7f6986254740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8b0b720 con 0x55dca8e6bfe0 2024-03-11T19:16:22.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.006+0000 7f6981c1e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e6bfe0 0x55dca8e65190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:22.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.006+0000 7f6981c1e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e6bfe0 0x55dca8e65190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:58168/0 (socket says 172.21.15.29:58168) 2024-03-11T19:16:22.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.006+0000 7f6981c1e640 1 -- 172.21.15.29:0/3296332388 learned_addr learned my addr 172.21.15.29:0/3296332388 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:22.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.006+0000 7f698141d640 1 --2- 172.21.15.29:0/3296332388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e66480 0x55dca8e64c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.006+0000 7f6981c1e640 1 -- 172.21.15.29:0/3296332388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e66480 msgr2=0x55dca8e64c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.006+0000 7f6981c1e640 1 --2- 172.21.15.29:0/3296332388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e66480 0x55dca8e64c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.006+0000 7f6981c1e640 1 -- 172.21.15.29:0/3296332388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dca8e66e70 msgr2=0x55dca8e77930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.006+0000 7f6981c1e640 1 --2- 172.21.15.29:0/3296332388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dca8e66e70 0x55dca8e77930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.006+0000 7f6981c1e640 1 -- 172.21.15.29:0/3296332388 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dca8e6f520 con 0x55dca8e6bfe0 2024-03-11T19:16:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.006+0000 7f6981c1e640 1 --2- 172.21.15.29:0/3296332388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e6bfe0 0x55dca8e65190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6974019000 tx=0x7f6974011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d802329590928e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.007+0000 7f6966ffd640 1 -- 172.21.15.29:0/3296332388 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697401e040 con 0x55dca8e6bfe0 2024-03-11T19:16:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.007+0000 7f6966ffd640 1 -- 172.21.15.29:0/3296332388 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f697401a670 con 0x55dca8e6bfe0 2024-03-11T19:16:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.007+0000 7f6966ffd640 1 -- 172.21.15.29:0/3296332388 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69740246a0 con 0x55dca8e6bfe0 2024-03-11T19:16:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.008+0000 7f6986254740 1 -- 172.21.15.29:0/3296332388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e6bfe0 msgr2=0x55dca8e65190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.008+0000 7f6986254740 1 --2- 172.21.15.29:0/3296332388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e6bfe0 0x55dca8e65190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6974019000 tx=0x7f6974011440 comp rx=0 tx=0).stop 2024-03-11T19:16:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.008+0000 7f6986254740 1 -- 172.21.15.29:0/3296332388 shutdown_connections 2024-03-11T19:16:22.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.008+0000 7f6986254740 1 --2- 172.21.15.29:0/3296332388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e6bfe0 0x55dca8e65190 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:22.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.008+0000 7f6986254740 1 --2- 172.21.15.29:0/3296332388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dca8e66e70 0x55dca8e77930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:22.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.008+0000 7f6986254740 1 --2- 172.21.15.29:0/3296332388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e66480 0x55dca8e64c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:22.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.008+0000 7f6986254740 1 -- 172.21.15.29:0/3296332388 >> 172.21.15.29:0/3296332388 conn(0x55dca8e619f0 msgr2=0x55dca8e680f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:22.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.008+0000 7f6986254740 1 -- 172.21.15.29:0/3296332388 shutdown_connections 2024-03-11T19:16:22.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.008+0000 7f6986254740 1 -- 172.21.15.29:0/3296332388 wait complete. 2024-03-11T19:16:22.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.008+0000 7f6986254740 1 -- start start 2024-03-11T19:16:22.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.009+0000 7f6986254740 1 Processor -- start 2024-03-11T19:16:22.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.009+0000 7f6986254740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e66480 0x55dca8e76a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:22.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.009+0000 7f6986254740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dca8e66e70 0x55dca8e76fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:22.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.009+0000 7f6986254740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e6bfe0 0x55dca8ee3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:22.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.009+0000 7f698141d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e66480 0x55dca8e76a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.009+0000 7f6986254740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8ee4170 con 0x55dca8e66e70 2024-03-11T19:16:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.009+0000 7f6986254740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8ee42b0 con 0x55dca8e6bfe0 2024-03-11T19:16:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.009+0000 7f6986254740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8ee4420 con 0x55dca8e66480 2024-03-11T19:16:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.009+0000 7f698141d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e66480 0x55dca8e76a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:58178/0 (socket says 172.21.15.29:58178) 2024-03-11T19:16:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.009+0000 7f698141d640 1 -- 172.21.15.29:0/2162469065 learned_addr learned my addr 172.21.15.29:0/2162469065 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.010+0000 7f6980c1c640 1 --2- 172.21.15.29:0/2162469065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dca8e66e70 0x55dca8e76fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.010+0000 7f6981c1e640 1 --2- 172.21.15.29:0/2162469065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e6bfe0 0x55dca8ee3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.010+0000 7f6986254740 10 client.0 _ll_register_callbacks cb 0x7f6974009ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.010+0000 7f698141d640 1 -- 172.21.15.29:0/2162469065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e6bfe0 msgr2=0x55dca8ee3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.010+0000 7f698141d640 1 --2- 172.21.15.29:0/2162469065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e6bfe0 0x55dca8ee3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.010+0000 7f698141d640 1 -- 172.21.15.29:0/2162469065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dca8e66e70 msgr2=0x55dca8e76fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.010+0000 7f698141d640 1 --2- 172.21.15.29:0/2162469065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dca8e66e70 0x55dca8e76fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.010+0000 7f698141d640 1 -- 172.21.15.29:0/2162469065 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dca8d91060 con 0x55dca8e66480 2024-03-11T19:16:22.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.011+0000 7f698141d640 1 --2- 172.21.15.29:0/2162469065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e66480 0x55dca8e76a80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6978009900 tx=0x7f697802c2b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd9d569df2d14712 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.011+0000 7f6964ff9640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:16:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.012+0000 7f6964ff9640 1 -- 172.21.15.29:0/2162469065 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6978003940 con 0x55dca8e66480 2024-03-11T19:16:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.012+0000 7f6964ff9640 1 -- 172.21.15.29:0/2162469065 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f6978003aa0 con 0x55dca8e66480 2024-03-11T19:16:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.012+0000 7f6986254740 1 -- 172.21.15.29:0/2162469065 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f697401aa80 con 0x55dca8e66480 2024-03-11T19:16:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.012+0000 7f6964ff9640 1 -- 172.21.15.29:0/2162469065 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6978003c60 con 0x55dca8e66480 2024-03-11T19:16:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.012+0000 7f6986254740 10 client.26542 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.012+0000 7f6986254740 1 -- 172.21.15.29:0/2162469065 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55dca8d69c20 con 0x55dca8e66480 2024-03-11T19:16:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.013+0000 7f6964ff9640 1 -- 172.21.15.29:0/2162469065 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6978003dc0 con 0x55dca8e66480 2024-03-11T19:16:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.014+0000 7f6964ff9640 1 -- 172.21.15.29:0/2162469065 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6978056070 con 0x55dca8e66480 2024-03-11T19:16:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.015+0000 7f6986254740 10 client.26542 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.015+0000 7f6986254740 10 client.26542 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:16:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.015+0000 7f6986254740 20 client.26542 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.015+0000 7f6986254740 1 --2- 172.21.15.29:0/2162469065 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55dca8ee5ca0 0x55dca8ee80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.016+0000 7f6986254740 4 client.26542 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:16:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.016+0000 7f6986254740 1 -- 172.21.15.29:0/2162469065 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f697401aa80 con 0x55dca8ee5ca0 2024-03-11T19:16:22.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.016+0000 7f6980c1c640 1 --2- 172.21.15.29:0/2162469065 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55dca8ee5ca0 0x55dca8ee80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.016+0000 7f6980c1c640 1 --2- 172.21.15.29:0/2162469065 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55dca8ee5ca0 0x55dca8ee80c0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be5bbc2577ce63ca server_cookie=8f3e639fd5060aba in_seq=0 out_seq=0 2024-03-11T19:16:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.016+0000 7f6964ff9640 10 client.26542 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.017+0000 7f6964ff9640 1 -- 172.21.15.29:0/2162469065 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f697401aa80 con 0x55dca8ee5ca0 2024-03-11T19:16:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.017+0000 7f6964ff9640 10 client.26542 handle_command_reply: tid=0 2024-03-11T19:16:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.017+0000 7f6986254740 1 client.26542 shutdown 2024-03-11T19:16:22.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.017+0000 7f6986254740 1 -- 172.21.15.29:0/2162469065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e66480 msgr2=0x55dca8e76a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:22.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.017+0000 7f6986254740 1 --2- 172.21.15.29:0/2162469065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e66480 0x55dca8e76a80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6978009900 tx=0x7f697802c2b0 comp rx=0 tx=0).stop 2024-03-11T19:16:22.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.017+0000 7f6986254740 1 -- 172.21.15.29:0/2162469065 shutdown_connections 2024-03-11T19:16:22.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.017+0000 7f6986254740 1 --2- 172.21.15.29:0/2162469065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55dca8e6bfe0 0x55dca8ee3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:22.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.018+0000 7f6986254740 1 --2- 172.21.15.29:0/2162469065 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55dca8ee5ca0 0x55dca8ee80c0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:22.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.018+0000 7f6986254740 1 --2- 172.21.15.29:0/2162469065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dca8e66e70 0x55dca8e76fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:22.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.018+0000 7f6986254740 1 --2- 172.21.15.29:0/2162469065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55dca8e66480 0x55dca8e76a80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:22.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.018+0000 7f6986254740 1 -- 172.21.15.29:0/2162469065 >> 172.21.15.29:0/2162469065 conn(0x55dca8e619f0 msgr2=0x55dca8e680f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:22.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.018+0000 7f6964ff9640 0 client.26542 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:22.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.018+0000 7f6986254740 1 -- 172.21.15.29:0/2162469065 shutdown_connections 2024-03-11T19:16:22.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.018+0000 7f6986254740 1 -- 172.21.15.29:0/2162469065 wait complete. 2024-03-11T19:16:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.018+0000 7f6986254740 20 client.26542 trim_cache size 0 max 16384 2024-03-11T19:16:22.024 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:16:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.021+0000 7f698241f640 1 -- 172.21.15.29:0/3939866287 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f69680a1c30 msgr2=0x7f69680a40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.021+0000 7f698241f640 1 --2- 172.21.15.29:0/3939866287 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f69680a1c30 0x7f69680a40f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6978005ab0 tx=0x7f697802ccc0 comp rx=0 tx=0).stop 2024-03-11T19:16:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.021+0000 7f698241f640 1 -- 172.21.15.29:0/3939866287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f697c108780 msgr2=0x7f697c1a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.021+0000 7f698241f640 1 --2- 172.21.15.29:0/3939866287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f697c108780 0x7f697c1a62a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f696c00d950 tx=0x7f696c00de10 comp rx=0 tx=0).stop 2024-03-11T19:16:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.021+0000 7f698241f640 1 -- 172.21.15.29:0/3939866287 shutdown_connections 2024-03-11T19:16:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.021+0000 7f698241f640 1 --2- 172.21.15.29:0/3939866287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f697c109120 0x7f697c1ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.021+0000 7f698241f640 1 --2- 172.21.15.29:0/3939866287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f697c108780 0x7f697c1a62a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.021+0000 7f698241f640 1 --2- 172.21.15.29:0/3939866287 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f69680a1c30 0x7f69680a40f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.021+0000 7f698241f640 1 --2- 172.21.15.29:0/3939866287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f697c0724d0 0x7f697c1a5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.021+0000 7f698241f640 1 -- 172.21.15.29:0/3939866287 >> 172.21.15.29:0/3939866287 conn(0x7f697c06dfa0 msgr2=0x7f697c06fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.021+0000 7f698241f640 1 -- 172.21.15.29:0/3939866287 shutdown_connections 2024-03-11T19:16:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:22.021+0000 7f698241f640 1 -- 172.21.15.29:0/3939866287 wait complete. 2024-03-11T19:16:23.043 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:16:23.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:16:23.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:16:23.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.232+0000 7f6f58640640 1 -- 172.21.15.29:0/3522102669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f5010adb0 msgr2=0x7f6f5010d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:23.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.233+0000 7f6f58640640 1 --2- 172.21.15.29:0/3522102669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f5010adb0 0x7f6f5010d1c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000b9f0 tx=0x7f6f4002d8f0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.234+0000 7f6f58640640 1 -- 172.21.15.29:0/3522102669 shutdown_connections 2024-03-11T19:16:23.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.234+0000 7f6f58640640 1 --2- 172.21.15.29:0/3522102669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f5010d850 0x7f6f5010fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.234+0000 7f6f58640640 1 --2- 172.21.15.29:0/3522102669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f5010adb0 0x7f6f5010d1c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.234+0000 7f6f58640640 1 --2- 172.21.15.29:0/3522102669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f501083f0 0x7f6f5010a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.234+0000 7f6f58640640 1 -- 172.21.15.29:0/3522102669 >> 172.21.15.29:0/3522102669 conn(0x7f6f5006dfe0 msgr2=0x7f6f50070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:23.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.235+0000 7f6f58640640 1 -- 172.21.15.29:0/3522102669 shutdown_connections 2024-03-11T19:16:23.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.235+0000 7f6f58640640 1 -- 172.21.15.29:0/3522102669 wait complete. 2024-03-11T19:16:23.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.236+0000 7f6f58640640 1 Processor -- start 2024-03-11T19:16:23.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.236+0000 7f6f58640640 1 -- start start 2024-03-11T19:16:23.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.236+0000 7f6f58640640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f501083f0 0x7f6f501a1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.236+0000 7f6f58640640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f5010adb0 0x7f6f501a2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.237+0000 7f6f563dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f501083f0 0x7f6f501a1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.237+0000 7f6f563dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f501083f0 0x7f6f501a1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47388/0 (socket says 172.21.15.29:47388) 2024-03-11T19:16:23.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.237+0000 7f6f55bdb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f5010adb0 0x7f6f501a2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.237+0000 7f6f55bdb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f5010adb0 0x7f6f501a2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44574/0 (socket says 172.21.15.29:44574) 2024-03-11T19:16:23.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.237+0000 7f6f58640640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f5010d850 0x7f6f501a91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.237+0000 7f6f58640640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f501a9720 con 0x7f6f501083f0 2024-03-11T19:16:23.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.237+0000 7f6f58640640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f501a9890 con 0x7f6f5010adb0 2024-03-11T19:16:23.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.237+0000 7f6f58640640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f501a9a50 con 0x7f6f5010d850 2024-03-11T19:16:23.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.237+0000 7f6f56bdd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f5010d850 0x7f6f501a91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.237+0000 7f6f56bdd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f5010d850 0x7f6f501a91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:58202/0 (socket says 172.21.15.29:58202) 2024-03-11T19:16:23.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.237+0000 7f6f56bdd640 1 -- 172.21.15.29:0/1507690133 learned_addr learned my addr 172.21.15.29:0/1507690133 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:23.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.238+0000 7f6f563dc640 1 -- 172.21.15.29:0/1507690133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f5010d850 msgr2=0x7f6f501a91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:23.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.238+0000 7f6f563dc640 1 --2- 172.21.15.29:0/1507690133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f5010d850 0x7f6f501a91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.238+0000 7f6f563dc640 1 -- 172.21.15.29:0/1507690133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f5010adb0 msgr2=0x7f6f501a2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:23.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.238+0000 7f6f563dc640 1 --2- 172.21.15.29:0/1507690133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f5010adb0 0x7f6f501a2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.238+0000 7f6f563dc640 1 -- 172.21.15.29:0/1507690133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4000b6a0 con 0x7f6f501083f0 2024-03-11T19:16:23.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.238+0000 7f6f55bdb640 1 --2- 172.21.15.29:0/1507690133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f5010adb0 0x7f6f501a2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:16:23.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.239+0000 7f6f563dc640 1 --2- 172.21.15.29:0/1507690133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f501083f0 0x7f6f501a1c40 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f6f48005fd0 tx=0x7f6f48009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:23.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.239+0000 7f6f477fe640 1 -- 172.21.15.29:0/1507690133 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4800ecb0 con 0x7f6f501083f0 2024-03-11T19:16:23.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.239+0000 7f6f477fe640 1 -- 172.21.15.29:0/1507690133 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6f4800f040 con 0x7f6f501083f0 2024-03-11T19:16:23.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.239+0000 7f6f477fe640 1 -- 172.21.15.29:0/1507690133 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4800ecb0 con 0x7f6f501083f0 2024-03-11T19:16:23.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.239+0000 7f6f58640640 1 -- 172.21.15.29:0/1507690133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f501a9d30 con 0x7f6f501083f0 2024-03-11T19:16:23.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.239+0000 7f6f58640640 1 -- 172.21.15.29:0/1507690133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f501aa190 con 0x7f6f501083f0 2024-03-11T19:16:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.242+0000 7f6f477fe640 1 -- 172.21.15.29:0/1507690133 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6f48015410 con 0x7f6f501083f0 2024-03-11T19:16:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.243+0000 7f6f477fe640 1 --2- 172.21.15.29:0/1507690133 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6f3c0a1ae0 0x7f6f3c0a3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.243+0000 7f6f5b20d740 1 Processor -- start 2024-03-11T19:16:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.243+0000 7f6f477fe640 1 -- 172.21.15.29:0/1507690133 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f6f480cccb0 con 0x7f6f501083f0 2024-03-11T19:16:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.243+0000 7f6f5b20d740 1 -- start start 2024-03-11T19:16:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.244+0000 7f6f55bdb640 1 --2- 172.21.15.29:0/1507690133 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6f3c0a1ae0 0x7f6f3c0a3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.244+0000 7f6f5b20d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991b4c0 0x559be991b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.244+0000 7f6f5b20d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991c690 0x559be991cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.244+0000 7f6f5b20d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991d050 0x559be99238e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.244+0000 7f6f5b20d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559be988af90 con 0x559be991b4c0 2024-03-11T19:16:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.244+0000 7f6f5b20d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559be98884c0 con 0x559be991c690 2024-03-11T19:16:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.244+0000 7f6f5b20d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559be9883ca0 con 0x559be991d050 2024-03-11T19:16:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.244+0000 7f6f563dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991b4c0 0x559be991b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.244+0000 7f6f55bdb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991c690 0x559be991cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.244+0000 7f6f56bdd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991d050 0x559be99238e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.244+0000 7f6f55bdb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991c690 0x559be991cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44588/0 (socket says 172.21.15.29:44588) 2024-03-11T19:16:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.244+0000 7f6f55bdb640 1 -- 172.21.15.29:0/2422638902 learned_addr learned my addr 172.21.15.29:0/2422638902 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.244+0000 7f6f563dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991b4c0 0x559be991b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47398/0 (socket says 172.21.15.29:47398) 2024-03-11T19:16:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.244+0000 7f6f56bdd640 1 -- 172.21.15.29:0/2422638902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991c690 msgr2=0x559be991cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.244+0000 7f6f56bdd640 1 --2- 172.21.15.29:0/2422638902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991c690 0x559be991cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.244+0000 7f6f56bdd640 1 -- 172.21.15.29:0/2422638902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991b4c0 msgr2=0x559be991b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.244+0000 7f6f56bdd640 1 --2- 172.21.15.29:0/2422638902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991b4c0 0x559be991b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.244+0000 7f6f56bdd640 1 -- 172.21.15.29:0/2422638902 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559be98460b0 con 0x559be991d050 2024-03-11T19:16:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.245+0000 7f6f55bdb640 1 --2- 172.21.15.29:0/1507690133 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6f3c0a1ae0 0x7f6f3c0a3fa0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6f40000c00 tx=0x7f6f40009280 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:23.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.245+0000 7f6f56bdd640 1 --2- 172.21.15.29:0/2422638902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991d050 0x559be99238e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009c00 tx=0x7f6f4c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd76cd588f6eb643 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:23.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.245+0000 7f6f44bfe640 1 -- 172.21.15.29:0/2422638902 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c019430 con 0x559be991d050 2024-03-11T19:16:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.245+0000 7f6f44bfe640 1 -- 172.21.15.29:0/2422638902 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6f4c019a40 con 0x559be991d050 2024-03-11T19:16:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.245+0000 7f6f44bfe640 1 -- 172.21.15.29:0/2422638902 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c00e990 con 0x559be991d050 2024-03-11T19:16:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.246+0000 7f6f5b20d740 1 -- 172.21.15.29:0/2422638902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991d050 msgr2=0x559be99238e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.246+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/2422638902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991d050 0x559be99238e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009c00 tx=0x7f6f4c011810 comp rx=0 tx=0).stop 2024-03-11T19:16:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.246+0000 7f6f5b20d740 1 -- 172.21.15.29:0/2422638902 shutdown_connections 2024-03-11T19:16:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.246+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/2422638902 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991d050 0x559be99238e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.246+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/2422638902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991c690 0x559be991cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.246+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/2422638902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991b4c0 0x559be991b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.246+0000 7f6f5b20d740 1 -- 172.21.15.29:0/2422638902 >> 172.21.15.29:0/2422638902 conn(0x559be99166a0 msgr2=0x559be9916aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.246+0000 7f6f5b20d740 1 -- 172.21.15.29:0/2422638902 shutdown_connections 2024-03-11T19:16:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.246+0000 7f6f5b20d740 1 -- 172.21.15.29:0/2422638902 wait complete. 2024-03-11T19:16:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.247+0000 7f6f5b20d740 1 -- start start 2024-03-11T19:16:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.247+0000 7f6f5b20d740 1 Processor -- start 2024-03-11T19:16:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.248+0000 7f6f5b20d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991b4c0 0x559be9925700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.248+0000 7f6f5b20d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991c690 0x559be9925c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.248+0000 7f6f5b20d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991d050 0x559be992cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.248+0000 7f6f56bdd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991d050 0x559be992cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.248+0000 7f6f5b20d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559be9828f40 con 0x559be991d050 2024-03-11T19:16:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.248+0000 7f6f5b20d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559be95574b0 con 0x559be991c690 2024-03-11T19:16:23.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.248+0000 7f6f5b20d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559be9809670 con 0x559be991b4c0 2024-03-11T19:16:23.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.249+0000 7f6f56bdd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991d050 0x559be992cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47402/0 (socket says 172.21.15.29:47402) 2024-03-11T19:16:23.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.249+0000 7f6f56bdd640 1 -- 172.21.15.29:0/3451112830 learned_addr learned my addr 172.21.15.29:0/3451112830 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:23.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.249+0000 7f6f5b20d740 10 client.0 _ll_register_callbacks cb 0x559be9876f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:23.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.249+0000 7f6f56bdd640 1 -- 172.21.15.29:0/3451112830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991b4c0 msgr2=0x559be9925700 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:16:23.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.249+0000 7f6f56bdd640 1 --2- 172.21.15.29:0/3451112830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991b4c0 0x559be9925700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.249+0000 7f6f56bdd640 1 -- 172.21.15.29:0/3451112830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991c690 msgr2=0x559be9925c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:16:23.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.249+0000 7f6f56bdd640 1 --2- 172.21.15.29:0/3451112830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991c690 0x559be9925c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.249+0000 7f6f56bdd640 1 -- 172.21.15.29:0/3451112830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f4c0098e0 con 0x559be991d050 2024-03-11T19:16:23.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.250+0000 7f6f56bdd640 1 --2- 172.21.15.29:0/3451112830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991d050 0x559be992cc80 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c000c00 tx=0x7f6f4c0056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155bc4fc22543c67 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:23.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.250+0000 7f6f39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:16:23.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.250+0000 7f6f39ffb640 1 -- 172.21.15.29:0/3451112830 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c00eae0 con 0x559be991d050 2024-03-11T19:16:23.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.250+0000 7f6f5b20d740 1 -- 172.21.15.29:0/3451112830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559be985df40 con 0x559be991d050 2024-03-11T19:16:23.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.250+0000 7f6f39ffb640 1 -- 172.21.15.29:0/3451112830 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f6f4c00ec40 con 0x559be991d050 2024-03-11T19:16:23.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.250+0000 7f6f39ffb640 1 -- 172.21.15.29:0/3451112830 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f4c00ee40 con 0x559be991d050 2024-03-11T19:16:23.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.251+0000 7f6f5b20d740 10 client.17025 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:23.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.251+0000 7f6f5b20d740 1 -- 172.21.15.29:0/3451112830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x559be9882320 con 0x559be991d050 2024-03-11T19:16:23.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.251+0000 7f6f39ffb640 1 -- 172.21.15.29:0/3451112830 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6f4c004b30 con 0x559be991d050 2024-03-11T19:16:23.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.252+0000 7f6f39ffb640 1 -- 172.21.15.29:0/3451112830 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6f4c019830 con 0x559be991d050 2024-03-11T19:16:23.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.253+0000 7f6f5b20d740 10 client.17025 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:23.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.253+0000 7f6f5b20d740 10 client.17025 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:16:23.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.253+0000 7f6f5b20d740 20 client.17025 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:23.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.253+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/3451112830 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x559be99317a0 0x559be9933c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.253+0000 7f6f5b20d740 4 client.17025 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:16:23.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.253+0000 7f6f5b20d740 1 -- 172.21.15.29:0/3451112830 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559be985df40 con 0x559be99317a0 2024-03-11T19:16:23.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.254+0000 7f6f563dc640 1 --2- 172.21.15.29:0/3451112830 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x559be99317a0 0x559be9933c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.255+0000 7f6f563dc640 1 --2- 172.21.15.29:0/3451112830 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x559be99317a0 0x559be9933c20 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb84342a5aeca7ea server_cookie=4cf126f1a4a42bd4 in_seq=0 out_seq=0 2024-03-11T19:16:23.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.255+0000 7f6f39ffb640 10 client.17025 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:23.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.256+0000 7f6f39ffb640 1 -- 172.21.15.29:0/3451112830 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559be985df40 con 0x559be99317a0 2024-03-11T19:16:23.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.256+0000 7f6f39ffb640 10 client.17025 handle_command_reply: tid=0 2024-03-11T19:16:23.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.256+0000 7f6f5b20d740 1 client.17025 shutdown 2024-03-11T19:16:23.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.256+0000 7f6f5b20d740 1 -- 172.21.15.29:0/3451112830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991d050 msgr2=0x559be992cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:23.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.257+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/3451112830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991d050 0x559be992cc80 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c000c00 tx=0x7f6f4c0056a0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.257+0000 7f6f5b20d740 1 -- 172.21.15.29:0/3451112830 shutdown_connections 2024-03-11T19:16:23.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.257+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/3451112830 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x559be99317a0 0x559be9933c20 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.257+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/3451112830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991d050 0x559be992cc80 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.257+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/3451112830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991c690 0x559be9925c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.257+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/3451112830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991b4c0 0x559be9925700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.257+0000 7f6f5b20d740 1 -- 172.21.15.29:0/3451112830 >> 172.21.15.29:0/3451112830 conn(0x559be99166a0 msgr2=0x559be99158e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:23.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.257+0000 7f6f39ffb640 0 client.17025 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:23.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.257+0000 7f6f5b20d740 1 -- 172.21.15.29:0/3451112830 shutdown_connections 2024-03-11T19:16:23.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.257+0000 7f6f5b20d740 1 -- 172.21.15.29:0/3451112830 wait complete. 2024-03-11T19:16:23.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.257+0000 7f6f5b20d740 20 client.17025 trim_cache size 0 max 16384 2024-03-11T19:16:23.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:22 smithi190 ceph-mon[36292]: pgmap v548: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T19:16:23.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.287+0000 7f6f5b20d740 1 Processor -- start 2024-03-11T19:16:23.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.287+0000 7f6f5b20d740 1 -- start start 2024-03-11T19:16:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.288+0000 7f6f5b20d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991b4c0 0x559be9925700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.288+0000 7f6f5b20d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991c690 0x559be9925c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.288+0000 7f6f5b20d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991d050 0x559be992cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.288+0000 7f6f5b20d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559be991bd30 con 0x559be991c690 2024-03-11T19:16:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.288+0000 7f6f5b20d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559be985df40 con 0x559be991b4c0 2024-03-11T19:16:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.288+0000 7f6f5b20d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559be9916470 con 0x559be991d050 2024-03-11T19:16:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.288+0000 7f6f56bdd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991d050 0x559be992cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.288+0000 7f6f56bdd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991d050 0x559be992cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:58238/0 (socket says 172.21.15.29:58238) 2024-03-11T19:16:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.288+0000 7f6f56bdd640 1 -- 172.21.15.29:0/2241084623 learned_addr learned my addr 172.21.15.29:0/2241084623 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:23.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.288+0000 7f6f563dc640 1 --2- 172.21.15.29:0/2241084623 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991b4c0 0x559be9925700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.288+0000 7f6f55bdb640 1 --2- 172.21.15.29:0/2241084623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991c690 0x559be9925c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.288+0000 7f6f56bdd640 1 -- 172.21.15.29:0/2241084623 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991b4c0 msgr2=0x559be9925700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:23.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.288+0000 7f6f56bdd640 1 --2- 172.21.15.29:0/2241084623 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991b4c0 0x559be9925700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.288+0000 7f6f56bdd640 1 -- 172.21.15.29:0/2241084623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991c690 msgr2=0x559be9925c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:23.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.288+0000 7f6f56bdd640 1 --2- 172.21.15.29:0/2241084623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991c690 0x559be9925c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.288+0000 7f6f56bdd640 1 -- 172.21.15.29:0/2241084623 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559be981ec00 con 0x559be991d050 2024-03-11T19:16:23.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.289+0000 7f6f56bdd640 1 --2- 172.21.15.29:0/2241084623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991d050 0x559be992cc40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c006000 tx=0x7f6f4c009820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=160922298b1f74c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:23.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.289+0000 7f6f3bfff640 1 -- 172.21.15.29:0/2241084623 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c00eae0 con 0x559be991d050 2024-03-11T19:16:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.289+0000 7f6f3bfff640 1 -- 172.21.15.29:0/2241084623 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6f4c005850 con 0x559be991d050 2024-03-11T19:16:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.289+0000 7f6f3bfff640 1 -- 172.21.15.29:0/2241084623 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c0048c0 con 0x559be991d050 2024-03-11T19:16:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.290+0000 7f6f5b20d740 1 -- 172.21.15.29:0/2241084623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991d050 msgr2=0x559be992cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.290+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/2241084623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991d050 0x559be992cc40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c006000 tx=0x7f6f4c009820 comp rx=0 tx=0).stop 2024-03-11T19:16:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.290+0000 7f6f5b20d740 1 -- 172.21.15.29:0/2241084623 shutdown_connections 2024-03-11T19:16:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.290+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/2241084623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991d050 0x559be992cc40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.290+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/2241084623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991c690 0x559be9925c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.290+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/2241084623 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991b4c0 0x559be9925700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.290+0000 7f6f5b20d740 1 -- 172.21.15.29:0/2241084623 >> 172.21.15.29:0/2241084623 conn(0x559be99166a0 msgr2=0x559be99385c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.290+0000 7f6f5b20d740 1 -- 172.21.15.29:0/2241084623 shutdown_connections 2024-03-11T19:16:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.290+0000 7f6f5b20d740 1 -- 172.21.15.29:0/2241084623 wait complete. 2024-03-11T19:16:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.291+0000 7f6f5b20d740 1 -- start start 2024-03-11T19:16:23.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.291+0000 7f6f5b20d740 1 Processor -- start 2024-03-11T19:16:23.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.291+0000 7f6f5b20d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991b4c0 0x559be992ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.292+0000 7f6f5b20d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991c690 0x559be992b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.292+0000 7f6f5b20d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991d050 0x559be992b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.292+0000 7f6f5b20d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559be992bbe0 con 0x559be991c690 2024-03-11T19:16:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.292+0000 7f6f5b20d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559be992bd50 con 0x559be991b4c0 2024-03-11T19:16:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.292+0000 7f6f5b20d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559be992bf10 con 0x559be991d050 2024-03-11T19:16:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.292+0000 7f6f5b20d740 10 client.0 _ll_register_callbacks cb 0x7f6f4c009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.292+0000 7f6f563dc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991b4c0 0x559be992ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.292+0000 7f6f563dc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991b4c0 0x559be992ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44610/0 (socket says 172.21.15.29:44610) 2024-03-11T19:16:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.292+0000 7f6f563dc640 1 -- 172.21.15.29:0/155040488 learned_addr learned my addr 172.21.15.29:0/155040488 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.292+0000 7f6f55bdb640 1 --2- 172.21.15.29:0/155040488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991c690 0x559be992b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.292+0000 7f6f563dc640 1 -- 172.21.15.29:0/155040488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991d050 msgr2=0x559be992b5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:16:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.292+0000 7f6f563dc640 1 --2- 172.21.15.29:0/155040488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991d050 0x559be992b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.292+0000 7f6f563dc640 1 -- 172.21.15.29:0/155040488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991c690 msgr2=0x559be992b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.292+0000 7f6f563dc640 1 --2- 172.21.15.29:0/155040488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991c690 0x559be992b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.292+0000 7f6f563dc640 1 -- 172.21.15.29:0/155040488 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559be9882320 con 0x559be991b4c0 2024-03-11T19:16:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.292+0000 7f6f55bdb640 1 --2- 172.21.15.29:0/155040488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991c690 0x559be992b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:16:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.293+0000 7f6f563dc640 1 --2- 172.21.15.29:0/155040488 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991b4c0 0x559be992ca80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6f480ca850 tx=0x7f6f4800ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe6847f166f93767 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.293+0000 7f6f39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:16:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.293+0000 7f6f39ffb640 1 -- 172.21.15.29:0/155040488 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4800ecb0 con 0x559be991b4c0 2024-03-11T19:16:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.293+0000 7f6f5b20d740 1 -- 172.21.15.29:0/155040488 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559be98a9900 con 0x559be991b4c0 2024-03-11T19:16:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.293+0000 7f6f39ffb640 1 -- 172.21.15.29:0/155040488 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f6f480a2460 con 0x559be991b4c0 2024-03-11T19:16:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.293+0000 7f6f39ffb640 1 -- 172.21.15.29:0/155040488 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f480b81b0 con 0x559be991b4c0 2024-03-11T19:16:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.293+0000 7f6f5b20d740 10 client.26528 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:23.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.293+0000 7f6f5b20d740 1 -- 172.21.15.29:0/155040488 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x559be9844910 con 0x559be991b4c0 2024-03-11T19:16:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.294+0000 7f6f39ffb640 1 -- 172.21.15.29:0/155040488 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x559be98a9900 con 0x559be991b4c0 2024-03-11T19:16:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.294+0000 7f6f39ffb640 1 -- 172.21.15.29:0/155040488 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6f48095b80 con 0x559be991b4c0 2024-03-11T19:16:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.294+0000 7f6f5b20d740 10 client.26528 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.294+0000 7f6f5b20d740 10 client.26528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:16:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.294+0000 7f6f5b20d740 20 client.26528 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.294+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/155040488 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x559be99896f0 0x559be998bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.294+0000 7f6f5b20d740 4 client.26528 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:16:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.294+0000 7f6f5b20d740 1 -- 172.21.15.29:0/155040488 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x559be987a310 con 0x559be99896f0 2024-03-11T19:16:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.294+0000 7f6f55bdb640 1 --2- 172.21.15.29:0/155040488 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x559be99896f0 0x559be998bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.295+0000 7f6f55bdb640 1 --2- 172.21.15.29:0/155040488 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x559be99896f0 0x559be998bb10 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8eda9695a64065ca server_cookie=a0c9644588893eb5 in_seq=0 out_seq=0 2024-03-11T19:16:23.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.295+0000 7f6f39ffb640 10 client.26528 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:23.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.297+0000 7f6f39ffb640 1 -- 172.21.15.29:0/155040488 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x559be987a310 con 0x559be99896f0 2024-03-11T19:16:23.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.297+0000 7f6f39ffb640 10 client.26528 handle_command_reply: tid=0 2024-03-11T19:16:23.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.297+0000 7f6f5b20d740 1 client.26528 shutdown 2024-03-11T19:16:23.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.297+0000 7f6f5b20d740 1 -- 172.21.15.29:0/155040488 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991b4c0 msgr2=0x559be992ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:23.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.297+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/155040488 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991b4c0 0x559be992ca80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6f480ca850 tx=0x7f6f4800ee90 comp rx=0 tx=0).stop 2024-03-11T19:16:23.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.297+0000 7f6f5b20d740 1 -- 172.21.15.29:0/155040488 shutdown_connections 2024-03-11T19:16:23.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.297+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/155040488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559be991d050 0x559be992b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.297+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/155040488 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x559be99896f0 0x559be998bb10 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.297+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/155040488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559be991c690 0x559be992b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.297+0000 7f6f5b20d740 1 --2- 172.21.15.29:0/155040488 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559be991b4c0 0x559be992ca80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.297+0000 7f6f5b20d740 1 -- 172.21.15.29:0/155040488 >> 172.21.15.29:0/155040488 conn(0x559be99166a0 msgr2=0x559be99256a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:23.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.297+0000 7f6f39ffb640 0 client.26528 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:23.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.297+0000 7f6f5b20d740 1 -- 172.21.15.29:0/155040488 shutdown_connections 2024-03-11T19:16:23.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.298+0000 7f6f5b20d740 1 -- 172.21.15.29:0/155040488 wait complete. 2024-03-11T19:16:23.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.298+0000 7f6f5b20d740 20 client.26528 trim_cache size 0 max 16384 2024-03-11T19:16:23.304 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:16:23.304 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:16:23.304 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "8266d22c-3712-41ad-8813-67a13d0d223d", 2024-03-11T19:16:23.304 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:16:23.304 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:16:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.299+0000 7f6f58640640 1 -- 172.21.15.29:0/1507690133 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6f3c0a1ae0 msgr2=0x7f6f3c0a3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.299+0000 7f6f58640640 1 --2- 172.21.15.29:0/1507690133 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6f3c0a1ae0 0x7f6f3c0a3fa0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6f40000c00 tx=0x7f6f40009280 comp rx=0 tx=0).stop 2024-03-11T19:16:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.300+0000 7f6f58640640 1 -- 172.21.15.29:0/1507690133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f501083f0 msgr2=0x7f6f501a1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.300+0000 7f6f58640640 1 --2- 172.21.15.29:0/1507690133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f501083f0 0x7f6f501a1c40 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f6f48005fd0 tx=0x7f6f48009b50 comp rx=0 tx=0).stop 2024-03-11T19:16:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.300+0000 7f6f58640640 1 -- 172.21.15.29:0/1507690133 shutdown_connections 2024-03-11T19:16:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.300+0000 7f6f58640640 1 --2- 172.21.15.29:0/1507690133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6f5010d850 0x7f6f501a91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.300+0000 7f6f58640640 1 --2- 172.21.15.29:0/1507690133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f5010adb0 0x7f6f501a2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.300+0000 7f6f58640640 1 --2- 172.21.15.29:0/1507690133 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6f3c0a1ae0 0x7f6f3c0a3fa0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.300+0000 7f6f58640640 1 --2- 172.21.15.29:0/1507690133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f501083f0 0x7f6f501a1c40 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.300+0000 7f6f58640640 1 -- 172.21.15.29:0/1507690133 >> 172.21.15.29:0/1507690133 conn(0x7f6f5006dfe0 msgr2=0x7f6f501097d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.300+0000 7f6f58640640 1 -- 172.21.15.29:0/1507690133 shutdown_connections 2024-03-11T19:16:23.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.301+0000 7f6f58640640 1 -- 172.21.15.29:0/1507690133 wait complete. 2024-03-11T19:16:23.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:22 smithi029 ceph-mon[32439]: pgmap v548: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T19:16:23.324 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:16:23.324 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:16:23.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:22 smithi170 ceph-mon[35690]: pgmap v548: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T19:16:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.510+0000 7fd90dec9640 1 -- 172.21.15.29:0/1498600013 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd908102360 msgr2=0x7fd9081047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.510+0000 7fd90dec9640 1 --2- 172.21.15.29:0/1498600013 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd908102360 0x7fd9081047d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc009860 tx=0x7fd8fc02bb70 comp rx=0 tx=0).stop 2024-03-11T19:16:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.510+0000 7fd90dec9640 1 -- 172.21.15.29:0/1498600013 shutdown_connections 2024-03-11T19:16:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.510+0000 7fd90dec9640 1 --2- 172.21.15.29:0/1498600013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd908104d10 0x7fd908105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.510+0000 7fd90dec9640 1 --2- 172.21.15.29:0/1498600013 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd908102360 0x7fd9081047d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.510+0000 7fd90dec9640 1 --2- 172.21.15.29:0/1498600013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9080ff9f0 0x7fd908101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.510+0000 7fd90dec9640 1 -- 172.21.15.29:0/1498600013 >> 172.21.15.29:0/1498600013 conn(0x7fd9080fb5f0 msgr2=0x7fd9080fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.510+0000 7fd90dec9640 1 -- 172.21.15.29:0/1498600013 shutdown_connections 2024-03-11T19:16:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.510+0000 7fd90dec9640 1 -- 172.21.15.29:0/1498600013 wait complete. 2024-03-11T19:16:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.511+0000 7fd90dec9640 1 Processor -- start 2024-03-11T19:16:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.511+0000 7fd90dec9640 1 -- start start 2024-03-11T19:16:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.511+0000 7fd90dec9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9080ff9f0 0x7fd90819d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.511+0000 7fd90dec9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd908102360 0x7fd90819dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.511+0000 7fd906ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd908102360 0x7fd90819dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.511+0000 7fd906ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd908102360 0x7fd90819dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44628/0 (socket says 172.21.15.29:44628) 2024-03-11T19:16:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.511+0000 7fd9077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9080ff9f0 0x7fd90819d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.512+0000 7fd9077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9080ff9f0 0x7fd90819d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47432/0 (socket says 172.21.15.29:47432) 2024-03-11T19:16:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.511+0000 7fd90dec9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd908104d10 0x7fd9081a4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.512+0000 7fd906ffd640 1 -- 172.21.15.29:0/2539356940 learned_addr learned my addr 172.21.15.29:0/2539356940 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.512+0000 7fd90dec9640 1 -- 172.21.15.29:0/2539356940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9081a5210 con 0x7fd9080ff9f0 2024-03-11T19:16:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.512+0000 7fd90dec9640 1 -- 172.21.15.29:0/2539356940 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9081a5380 con 0x7fd908102360 2024-03-11T19:16:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.512+0000 7fd90dec9640 1 -- 172.21.15.29:0/2539356940 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9081a5540 con 0x7fd908104d10 2024-03-11T19:16:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.512+0000 7fd907fff640 1 --2- 172.21.15.29:0/2539356940 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd908104d10 0x7fd9081a4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.512+0000 7fd906ffd640 1 -- 172.21.15.29:0/2539356940 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd908104d10 msgr2=0x7fd9081a4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.512+0000 7fd906ffd640 1 --2- 172.21.15.29:0/2539356940 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd908104d10 0x7fd9081a4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.512+0000 7fd906ffd640 1 -- 172.21.15.29:0/2539356940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9080ff9f0 msgr2=0x7fd90819d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.512+0000 7fd906ffd640 1 --2- 172.21.15.29:0/2539356940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9080ff9f0 0x7fd90819d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.512+0000 7fd906ffd640 1 -- 172.21.15.29:0/2539356940 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc009490 con 0x7fd908102360 2024-03-11T19:16:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.513+0000 7fd906ffd640 1 --2- 172.21.15.29:0/2539356940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd908102360 0x7fd90819dc60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc0349e0 tx=0x7fd8fc032450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.513+0000 7fd904ff9640 1 -- 172.21.15.29:0/2539356940 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc005020 con 0x7fd908102360 2024-03-11T19:16:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.513+0000 7fd90dec9640 1 -- 172.21.15.29:0/2539356940 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9081a57c0 con 0x7fd908102360 2024-03-11T19:16:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.513+0000 7fd904ff9640 1 -- 172.21.15.29:0/2539356940 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd8fc02ce90 con 0x7fd908102360 2024-03-11T19:16:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.513+0000 7fd904ff9640 1 -- 172.21.15.29:0/2539356940 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc004720 con 0x7fd908102360 2024-03-11T19:16:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.513+0000 7fd90dec9640 1 -- 172.21.15.29:0/2539356940 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9081a5cc0 con 0x7fd908102360 2024-03-11T19:16:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.515+0000 7fd904ff9640 1 -- 172.21.15.29:0/2539356940 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd8fc03b040 con 0x7fd908102360 2024-03-11T19:16:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.516+0000 7fd904ff9640 1 --2- 172.21.15.29:0/2539356940 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8f00a1ab0 0x7fd8f00a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.516+0000 7fd910b70740 1 Processor -- start 2024-03-11T19:16:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.516+0000 7fd904ff9640 1 -- 172.21.15.29:0/2539356940 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd8fc0eeb40 con 0x7fd908102360 2024-03-11T19:16:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.517+0000 7fd910b70740 1 -- start start 2024-03-11T19:16:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.517+0000 7fd910b70740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28ba2d0 0x5616f28ba6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.517+0000 7fd910b70740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28bacc0 0x5616f28bf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.517+0000 7fd910b70740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 0x5616f28c22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.517+0000 7fd910b70740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f282a280 con 0x5616f28bacc0 2024-03-11T19:16:23.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.517+0000 7fd910b70740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f28277b0 con 0x5616f28ba2d0 2024-03-11T19:16:23.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.517+0000 7fd910b70740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f2822f90 con 0x5616f28bfe30 2024-03-11T19:16:23.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.517+0000 7fd907fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 0x5616f28c22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.517+0000 7fd907fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 0x5616f28c22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:58288/0 (socket says 172.21.15.29:58288) 2024-03-11T19:16:23.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.517+0000 7fd907fff640 1 -- 172.21.15.29:0/3424198643 learned_addr learned my addr 172.21.15.29:0/3424198643 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:23.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.517+0000 7fd906ffd640 1 --2- 172.21.15.29:0/3424198643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28bacc0 0x5616f28bf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.517+0000 7fd907fff640 1 -- 172.21.15.29:0/3424198643 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28ba2d0 msgr2=0x5616f28ba6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:16:23.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.517+0000 7fd907fff640 1 --2- 172.21.15.29:0/3424198643 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28ba2d0 0x5616f28ba6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.517+0000 7fd907fff640 1 -- 172.21.15.29:0/3424198643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28bacc0 msgr2=0x5616f28bf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:23.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.517+0000 7fd907fff640 1 --2- 172.21.15.29:0/3424198643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28bacc0 0x5616f28bf7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.517+0000 7fd907fff640 1 -- 172.21.15.29:0/3424198643 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616f27e1570 con 0x5616f28bfe30 2024-03-11T19:16:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.518+0000 7fd9077fe640 1 --2- 172.21.15.29:0/2539356940 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8f00a1ab0 0x7fd8f00a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.518+0000 7fd907fff640 1 --2- 172.21.15.29:0/3424198643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 0x5616f28c22c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd900009fd0 tx=0x7fd9000118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf5b4b0ca7ce7c4c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.518+0000 7fd8edffb640 1 -- 172.21.15.29:0/3424198643 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd900004d20 con 0x5616f28bfe30 2024-03-11T19:16:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.518+0000 7fd8edffb640 1 -- 172.21.15.29:0/3424198643 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd900004e80 con 0x5616f28bfe30 2024-03-11T19:16:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.519+0000 7fd8edffb640 1 -- 172.21.15.29:0/3424198643 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd900005280 con 0x5616f28bfe30 2024-03-11T19:16:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.519+0000 7fd910b70740 1 -- 172.21.15.29:0/3424198643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 msgr2=0x5616f28c22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.519+0000 7fd910b70740 1 --2- 172.21.15.29:0/3424198643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 0x5616f28c22c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd900009fd0 tx=0x7fd9000118d0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.519+0000 7fd9077fe640 1 --2- 172.21.15.29:0/2539356940 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8f00a1ab0 0x7fd8f00a3f70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x5616f28bb520 tx=0x7fd8f8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.519+0000 7fd910b70740 1 -- 172.21.15.29:0/3424198643 shutdown_connections 2024-03-11T19:16:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.519+0000 7fd910b70740 1 --2- 172.21.15.29:0/3424198643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 0x5616f28c22c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.519+0000 7fd910b70740 1 --2- 172.21.15.29:0/3424198643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28bacc0 0x5616f28bf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.519+0000 7fd910b70740 1 --2- 172.21.15.29:0/3424198643 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28ba2d0 0x5616f28ba6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.519+0000 7fd910b70740 1 -- 172.21.15.29:0/3424198643 >> 172.21.15.29:0/3424198643 conn(0x5616f28b57f0 msgr2=0x5616f28b5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:23.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.520+0000 7fd910b70740 1 -- 172.21.15.29:0/3424198643 shutdown_connections 2024-03-11T19:16:23.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.520+0000 7fd910b70740 1 -- 172.21.15.29:0/3424198643 wait complete. 2024-03-11T19:16:23.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.520+0000 7fd910b70740 1 -- start start 2024-03-11T19:16:23.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.521+0000 7fd910b70740 1 Processor -- start 2024-03-11T19:16:23.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.521+0000 7fd910b70740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28ba2d0 0x5616f28c4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.521+0000 7fd910b70740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28bacc0 0x5616f28c47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.521+0000 7fd910b70740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 0x5616f28cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.521+0000 7fd910b70740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f24f6430 con 0x5616f28bacc0 2024-03-11T19:16:23.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.522+0000 7fd910b70740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f24b2e00 con 0x5616f28ba2d0 2024-03-11T19:16:23.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.522+0000 7fd910b70740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f27a7fd0 con 0x5616f28bfe30 2024-03-11T19:16:23.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.522+0000 7fd9077fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28ba2d0 0x5616f28c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.522+0000 7fd9077fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28ba2d0 0x5616f28c4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44634/0 (socket says 172.21.15.29:44634) 2024-03-11T19:16:23.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.522+0000 7fd9077fe640 1 -- 172.21.15.29:0/2942436205 learned_addr learned my addr 172.21.15.29:0/2942436205 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:23.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.522+0000 7fd910b70740 10 client.0 _ll_register_callbacks cb 0x5616f27fdfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:23.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.522+0000 7fd906ffd640 1 --2- 172.21.15.29:0/2942436205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28bacc0 0x5616f28c47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.522+0000 7fd907fff640 1 --2- 172.21.15.29:0/2942436205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 0x5616f28cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.522+0000 7fd9077fe640 1 -- 172.21.15.29:0/2942436205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 msgr2=0x5616f28cb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:23.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.522+0000 7fd9077fe640 1 --2- 172.21.15.29:0/2942436205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 0x5616f28cb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.522+0000 7fd9077fe640 1 -- 172.21.15.29:0/2942436205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28bacc0 msgr2=0x5616f28c47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:23.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.522+0000 7fd9077fe640 1 --2- 172.21.15.29:0/2942436205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28bacc0 0x5616f28c47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.522+0000 7fd9077fe640 1 -- 172.21.15.29:0/2942436205 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd900009c80 con 0x5616f28ba2d0 2024-03-11T19:16:23.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.522+0000 7fd906ffd640 1 --2- 172.21.15.29:0/2942436205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28bacc0 0x5616f28c47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:16:23.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.522+0000 7fd9077fe640 1 --2- 172.21.15.29:0/2942436205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28ba2d0 0x5616f28c4260 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd8f801c960 tx=0x7fd8f8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20196d024a222ccd server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:23.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.522+0000 7fd8e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:16:23.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.523+0000 7fd8e6ffd640 1 -- 172.21.15.29:0/2942436205 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8009ea0 con 0x5616f28ba2d0 2024-03-11T19:16:23.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.523+0000 7fd910b70740 1 -- 172.21.15.29:0/2942436205 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5616f255f720 con 0x5616f28ba2d0 2024-03-11T19:16:23.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.523+0000 7fd8e6ffd640 1 -- 172.21.15.29:0/2942436205 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd8f8023070 con 0x5616f28ba2d0 2024-03-11T19:16:23.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.523+0000 7fd8e6ffd640 1 -- 172.21.15.29:0/2942436205 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8f800e930 con 0x5616f28ba2d0 2024-03-11T19:16:23.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.523+0000 7fd910b70740 10 client.26537 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:23.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.523+0000 7fd910b70740 1 -- 172.21.15.29:0/2942436205 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5616f2821610 con 0x5616f28ba2d0 2024-03-11T19:16:23.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.523+0000 7fd8e6ffd640 1 -- 172.21.15.29:0/2942436205 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd8f802d030 con 0x5616f28ba2d0 2024-03-11T19:16:23.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.524+0000 7fd8e6ffd640 1 -- 172.21.15.29:0/2942436205 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd8f80048a0 con 0x5616f28ba2d0 2024-03-11T19:16:23.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.524+0000 7fd910b70740 10 client.26537 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:23.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.524+0000 7fd910b70740 10 client.26537 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:16:23.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.524+0000 7fd910b70740 20 client.26537 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:23.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.524+0000 7fd910b70740 1 --2- 172.21.15.29:0/2942436205 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5616f28d0410 0x5616f28d28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.524+0000 7fd910b70740 4 client.26537 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:16:23.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.524+0000 7fd910b70740 1 -- 172.21.15.29:0/2942436205 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5616f255f720 con 0x5616f28d0410 2024-03-11T19:16:23.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.524+0000 7fd906ffd640 1 --2- 172.21.15.29:0/2942436205 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5616f28d0410 0x5616f28d28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.526+0000 7fd906ffd640 1 --2- 172.21.15.29:0/2942436205 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5616f28d0410 0x5616f28d28d0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a62f80b258badac server_cookie=141fa891846984da in_seq=0 out_seq=0 2024-03-11T19:16:23.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.528+0000 7fd8e6ffd640 10 client.26537 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:23.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.528+0000 7fd8e6ffd640 1 -- 172.21.15.29:0/2942436205 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5616f255f720 con 0x5616f28d0410 2024-03-11T19:16:23.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.528+0000 7fd8e6ffd640 10 client.26537 handle_command_reply: tid=0 2024-03-11T19:16:23.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.528+0000 7fd910b70740 1 client.26537 shutdown 2024-03-11T19:16:23.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.528+0000 7fd910b70740 1 -- 172.21.15.29:0/2942436205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28ba2d0 msgr2=0x5616f28c4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:23.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.528+0000 7fd910b70740 1 --2- 172.21.15.29:0/2942436205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28ba2d0 0x5616f28c4260 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd8f801c960 tx=0x7fd8f8004730 comp rx=0 tx=0).stop 2024-03-11T19:16:23.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.528+0000 7fd910b70740 1 -- 172.21.15.29:0/2942436205 shutdown_connections 2024-03-11T19:16:23.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.529+0000 7fd910b70740 1 --2- 172.21.15.29:0/2942436205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 0x5616f28cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.529+0000 7fd910b70740 1 --2- 172.21.15.29:0/2942436205 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5616f28d0410 0x5616f28d28d0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.529+0000 7fd910b70740 1 --2- 172.21.15.29:0/2942436205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28bacc0 0x5616f28c47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.529+0000 7fd910b70740 1 --2- 172.21.15.29:0/2942436205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28ba2d0 0x5616f28c4260 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.529+0000 7fd910b70740 1 -- 172.21.15.29:0/2942436205 >> 172.21.15.29:0/2942436205 conn(0x5616f28b57f0 msgr2=0x5616f28b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:23.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.529+0000 7fd8e6ffd640 0 client.26537 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:23.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.530+0000 7fd910b70740 1 -- 172.21.15.29:0/2942436205 shutdown_connections 2024-03-11T19:16:23.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.531+0000 7fd910b70740 1 -- 172.21.15.29:0/2942436205 wait complete. 2024-03-11T19:16:23.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.531+0000 7fd910b70740 20 client.26537 trim_cache size 0 max 16384 2024-03-11T19:16:23.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.568+0000 7fd910b70740 1 Processor -- start 2024-03-11T19:16:23.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.568+0000 7fd910b70740 1 -- start start 2024-03-11T19:16:23.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.568+0000 7fd910b70740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28ba2d0 0x5616f28c3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.568+0000 7fd910b70740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28bacc0 0x5616f28cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.568+0000 7fd9077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28ba2d0 0x5616f28c3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.568+0000 7fd9077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28ba2d0 0x5616f28c3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47448/0 (socket says 172.21.15.29:47448) 2024-03-11T19:16:23.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.569+0000 7fd9077fe640 1 -- 172.21.15.29:0/446437505 learned_addr learned my addr 172.21.15.29:0/446437505 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:23.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.569+0000 7fd910b70740 1 --2- 172.21.15.29:0/446437505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 0x5616f28c43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.569+0000 7fd906ffd640 1 --2- 172.21.15.29:0/446437505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28bacc0 0x5616f28cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.569+0000 7fd910b70740 1 -- 172.21.15.29:0/446437505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f255f720 con 0x5616f28ba2d0 2024-03-11T19:16:23.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.569+0000 7fd910b70740 1 -- 172.21.15.29:0/446437505 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f28bb3b0 con 0x5616f28bacc0 2024-03-11T19:16:23.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.569+0000 7fd907fff640 1 --2- 172.21.15.29:0/446437505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 0x5616f28c43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.569+0000 7fd910b70740 1 -- 172.21.15.29:0/446437505 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f28b67e0 con 0x5616f28bfe30 2024-03-11T19:16:23.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.570+0000 7fd9077fe640 1 -- 172.21.15.29:0/446437505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 msgr2=0x5616f28c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:23.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.570+0000 7fd9077fe640 1 --2- 172.21.15.29:0/446437505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 0x5616f28c43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.570+0000 7fd9077fe640 1 -- 172.21.15.29:0/446437505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28bacc0 msgr2=0x5616f28cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:23.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.570+0000 7fd9077fe640 1 --2- 172.21.15.29:0/446437505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28bacc0 0x5616f28cb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.570+0000 7fd9077fe640 1 -- 172.21.15.29:0/446437505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616f27bdac0 con 0x5616f28ba2d0 2024-03-11T19:16:23.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.570+0000 7fd9077fe640 1 --2- 172.21.15.29:0/446437505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28ba2d0 0x5616f28c3e70 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8012fd0 tx=0x7fd8f8009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9988c931e081277c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:23.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.571+0000 7fd8ed7fa640 1 -- 172.21.15.29:0/446437505 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8027430 con 0x5616f28ba2d0 2024-03-11T19:16:23.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.571+0000 7fd8ed7fa640 1 -- 172.21.15.29:0/446437505 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd8f802d040 con 0x5616f28ba2d0 2024-03-11T19:16:23.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.571+0000 7fd8ed7fa640 1 -- 172.21.15.29:0/446437505 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f800ec10 con 0x5616f28ba2d0 2024-03-11T19:16:23.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.572+0000 7fd910b70740 1 -- 172.21.15.29:0/446437505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28ba2d0 msgr2=0x5616f28c3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:23.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.572+0000 7fd910b70740 1 --2- 172.21.15.29:0/446437505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28ba2d0 0x5616f28c3e70 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8012fd0 tx=0x7fd8f8009fd0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.572+0000 7fd910b70740 1 -- 172.21.15.29:0/446437505 shutdown_connections 2024-03-11T19:16:23.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.572+0000 7fd910b70740 1 --2- 172.21.15.29:0/446437505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 0x5616f28c43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.572+0000 7fd910b70740 1 --2- 172.21.15.29:0/446437505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28bacc0 0x5616f28cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.572+0000 7fd910b70740 1 --2- 172.21.15.29:0/446437505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28ba2d0 0x5616f28c3e70 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.572+0000 7fd910b70740 1 -- 172.21.15.29:0/446437505 >> 172.21.15.29:0/446437505 conn(0x5616f28b57f0 msgr2=0x5616f28b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:23.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.572+0000 7fd910b70740 1 -- 172.21.15.29:0/446437505 shutdown_connections 2024-03-11T19:16:23.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.572+0000 7fd910b70740 1 -- 172.21.15.29:0/446437505 wait complete. 2024-03-11T19:16:23.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.573+0000 7fd910b70740 1 -- start start 2024-03-11T19:16:23.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.573+0000 7fd910b70740 1 Processor -- start 2024-03-11T19:16:23.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.574+0000 7fd910b70740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28ba2d0 0x5616f28c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.574+0000 7fd910b70740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28bacc0 0x5616f28ca520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.574+0000 7fd910b70740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 0x5616f28caa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.574+0000 7fd906ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28bacc0 0x5616f28ca520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.574+0000 7fd906ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28bacc0 0x5616f28ca520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44652/0 (socket says 172.21.15.29:44652) 2024-03-11T19:16:23.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.574+0000 7fd906ffd640 1 -- 172.21.15.29:0/2206421657 learned_addr learned my addr 172.21.15.29:0/2206421657 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:23.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.575+0000 7fd907fff640 1 --2- 172.21.15.29:0/2206421657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 0x5616f28caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.574+0000 7fd910b70740 1 -- 172.21.15.29:0/2206421657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f29269e0 con 0x5616f28ba2d0 2024-03-11T19:16:23.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.575+0000 7fd9077fe640 1 --2- 172.21.15.29:0/2206421657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28ba2d0 0x5616f28c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.575+0000 7fd910b70740 1 -- 172.21.15.29:0/2206421657 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f2926b20 con 0x5616f28bacc0 2024-03-11T19:16:23.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.575+0000 7fd910b70740 1 -- 172.21.15.29:0/2206421657 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f2926c60 con 0x5616f28bfe30 2024-03-11T19:16:23.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.575+0000 7fd910b70740 10 client.0 _ll_register_callbacks cb 0x7fd900009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:23.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.576+0000 7fd907fff640 1 -- 172.21.15.29:0/2206421657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28bacc0 msgr2=0x5616f28ca520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:23.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.576+0000 7fd907fff640 1 --2- 172.21.15.29:0/2206421657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28bacc0 0x5616f28ca520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.576+0000 7fd907fff640 1 -- 172.21.15.29:0/2206421657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28ba2d0 msgr2=0x5616f28c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:23.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.576+0000 7fd907fff640 1 --2- 172.21.15.29:0/2206421657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28ba2d0 0x5616f28c9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.576+0000 7fd907fff640 1 -- 172.21.15.29:0/2206421657 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5616f2821610 con 0x5616f28bfe30 2024-03-11T19:16:23.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.576+0000 7fd9077fe640 1 --2- 172.21.15.29:0/2206421657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28ba2d0 0x5616f28c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:16:23.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.576+0000 7fd906ffd640 1 --2- 172.21.15.29:0/2206421657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28bacc0 0x5616f28ca520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:16:23.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.576+0000 7fd910b70740 1 -- 172.21.15.29:0/2206421657 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5616f25227e0 con 0x5616f28bfe30 2024-03-11T19:16:23.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.576+0000 7fd907fff640 1 --2- 172.21.15.29:0/2206421657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 0x5616f28caa60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd900006010 tx=0x7fd90000f710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3724a66431be426a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:23.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.576+0000 7fd8e6ffd640 10 client.26572 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:16:23.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.577+0000 7fd8e6ffd640 1 -- 172.21.15.29:0/2206421657 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90000fcd0 con 0x5616f28bfe30 2024-03-11T19:16:23.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.577+0000 7fd8e6ffd640 1 -- 172.21.15.29:0/2206421657 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd90000fe30 con 0x5616f28bfe30 2024-03-11T19:16:23.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.577+0000 7fd8e6ffd640 1 -- 172.21.15.29:0/2206421657 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd900005000 con 0x5616f28bfe30 2024-03-11T19:16:23.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.577+0000 7fd910b70740 10 client.26572 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:23.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.577+0000 7fd910b70740 1 -- 172.21.15.29:0/2206421657 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x5616f27e36a0 con 0x5616f28bfe30 2024-03-11T19:16:23.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.578+0000 7fd8e6ffd640 1 -- 172.21.15.29:0/2206421657 <== mon.1 v2:172.21.15.190:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd900005220 con 0x5616f28bfe30 2024-03-11T19:16:23.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.578+0000 7fd910b70740 10 client.26572 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:23.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.578+0000 7fd910b70740 10 client.26572 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:16:23.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.578+0000 7fd910b70740 20 client.26572 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:23.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.578+0000 7fd910b70740 1 --2- 172.21.15.29:0/2206421657 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5616f2928470 0x5616f292a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:23.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.578+0000 7fd910b70740 4 client.26572 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:16:23.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.578+0000 7fd910b70740 1 -- 172.21.15.29:0/2206421657 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5616f25ae890 con 0x5616f2928470 2024-03-11T19:16:23.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.578+0000 7fd9077fe640 1 --2- 172.21.15.29:0/2206421657 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5616f2928470 0x5616f292a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:23.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.579+0000 7fd9077fe640 1 --2- 172.21.15.29:0/2206421657 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5616f2928470 0x5616f292a890 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2113e5a53e4475ec server_cookie=e3aa9a981c70339a in_seq=0 out_seq=0 2024-03-11T19:16:23.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.579+0000 7fd8e6ffd640 10 client.26572 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:23.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.579+0000 7fd8e6ffd640 1 -- 172.21.15.29:0/2206421657 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5616f25ae890 con 0x5616f2928470 2024-03-11T19:16:23.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.579+0000 7fd8e6ffd640 10 client.26572 handle_command_reply: tid=0 2024-03-11T19:16:23.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.579+0000 7fd910b70740 1 client.26572 shutdown 2024-03-11T19:16:23.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.580+0000 7fd910b70740 1 -- 172.21.15.29:0/2206421657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 msgr2=0x5616f28caa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:23.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.580+0000 7fd910b70740 1 --2- 172.21.15.29:0/2206421657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 0x5616f28caa60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd900006010 tx=0x7fd90000f710 comp rx=0 tx=0).stop 2024-03-11T19:16:23.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.580+0000 7fd910b70740 1 -- 172.21.15.29:0/2206421657 shutdown_connections 2024-03-11T19:16:23.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.580+0000 7fd910b70740 1 --2- 172.21.15.29:0/2206421657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5616f28bfe30 0x5616f28caa60 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.580+0000 7fd910b70740 1 --2- 172.21.15.29:0/2206421657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5616f28bacc0 0x5616f28ca520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.580+0000 7fd910b70740 1 --2- 172.21.15.29:0/2206421657 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5616f2928470 0x5616f292a890 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.580+0000 7fd910b70740 1 --2- 172.21.15.29:0/2206421657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5616f28ba2d0 0x5616f28c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.580+0000 7fd8e6ffd640 0 client.26572 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:23.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.580+0000 7fd910b70740 1 -- 172.21.15.29:0/2206421657 >> 172.21.15.29:0/2206421657 conn(0x5616f28b57f0 msgr2=0x5616f28c3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:23.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.580+0000 7fd910b70740 1 -- 172.21.15.29:0/2206421657 shutdown_connections 2024-03-11T19:16:23.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.580+0000 7fd910b70740 1 -- 172.21.15.29:0/2206421657 wait complete. 2024-03-11T19:16:23.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.581+0000 7fd910b70740 20 client.26572 trim_cache size 0 max 16384 2024-03-11T19:16:23.588 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:16:23.588 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:16:23.588 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:16:23.588 INFO:teuthology.orchestra.run.smithi029.stdout: "8266d22c-3712-41ad-8813-67a13d0d223d": { 2024-03-11T19:16:23.588 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:16:23.589 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "8266d22c-3712-41ad-8813-67a13d0d223d", 2024-03-11T19:16:23.589 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:16:23.589 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:16:23.589 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:16:23.589 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:16:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.582+0000 7fd90dec9640 1 -- 172.21.15.29:0/2539356940 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8f00a1ab0 msgr2=0x7fd8f00a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.582+0000 7fd90dec9640 1 --2- 172.21.15.29:0/2539356940 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8f00a1ab0 0x7fd8f00a3f70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x5616f28bb520 tx=0x7fd8f8009260 comp rx=0 tx=0).stop 2024-03-11T19:16:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.582+0000 7fd90dec9640 1 -- 172.21.15.29:0/2539356940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd908102360 msgr2=0x7fd90819dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.582+0000 7fd90dec9640 1 --2- 172.21.15.29:0/2539356940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd908102360 0x7fd90819dc60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc0349e0 tx=0x7fd8fc032450 comp rx=0 tx=0).stop 2024-03-11T19:16:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.582+0000 7fd90dec9640 1 -- 172.21.15.29:0/2539356940 shutdown_connections 2024-03-11T19:16:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.582+0000 7fd90dec9640 1 --2- 172.21.15.29:0/2539356940 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd908104d10 0x7fd9081a4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.582+0000 7fd90dec9640 1 --2- 172.21.15.29:0/2539356940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd908102360 0x7fd90819dc60 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.582+0000 7fd90dec9640 1 --2- 172.21.15.29:0/2539356940 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8f00a1ab0 0x7fd8f00a3f70 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.582+0000 7fd90dec9640 1 --2- 172.21.15.29:0/2539356940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9080ff9f0 0x7fd90819d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.582+0000 7fd90dec9640 1 -- 172.21.15.29:0/2539356940 >> 172.21.15.29:0/2539356940 conn(0x7fd9080fb5f0 msgr2=0x7fd9081002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.583+0000 7fd90dec9640 1 -- 172.21.15.29:0/2539356940 shutdown_connections 2024-03-11T19:16:23.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:23.583+0000 7fd90dec9640 1 -- 172.21.15.29:0/2539356940 wait complete. 2024-03-11T19:16:23.615 INFO:tasks.cephfs.filesystem:scrub status for tag:8266d22c-3712-41ad-8813-67a13d0d223d - {'path': '/', 'tag': '8266d22c-3712-41ad-8813-67a13d0d223d', 'options': 'recursive,force'} 2024-03-11T19:16:24.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:23 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:16:24.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:23 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:16:24.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:23 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:16:24.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:23 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:16:24.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:23 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:16:24.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:23 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:16:24.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:23 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:16:24.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:23 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:16:24.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:23 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:16:24.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:23 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:16:24.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:23 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:16:24.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:23 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:16:25.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:24 smithi190 ceph-mon[36292]: pgmap v549: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 69 MiB/s wr, 1.12k op/s 2024-03-11T19:16:25.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:24 smithi170 ceph-mon[35690]: pgmap v549: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 69 MiB/s wr, 1.12k op/s 2024-03-11T19:16:25.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:24 smithi029 ceph-mon[32439]: pgmap v549: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 69 MiB/s wr, 1.12k op/s 2024-03-11T19:16:26.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:25 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:16:26.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:25 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:16:26.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:25 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:16:27.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:26 smithi190 ceph-mon[36292]: pgmap v550: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T19:16:27.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:26 smithi170 ceph-mon[35690]: pgmap v550: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T19:16:27.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:26 smithi029 ceph-mon[32439]: pgmap v550: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T19:16:29.208 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:28 smithi029 ceph-mon[32439]: pgmap v551: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 62 MiB/s wr, 1.01k op/s 2024-03-11T19:16:29.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:28 smithi190 ceph-mon[36292]: pgmap v551: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 62 MiB/s wr, 1.01k op/s 2024-03-11T19:16:29.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:28 smithi170 ceph-mon[35690]: pgmap v551: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 62 MiB/s wr, 1.01k op/s 2024-03-11T19:16:31.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:30 smithi190 ceph-mon[36292]: pgmap v552: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:16:31.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:30 smithi170 ceph-mon[35690]: pgmap v552: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:16:31.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:30 smithi029 ceph-mon[32439]: pgmap v552: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:16:33.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:32 smithi190 ceph-mon[36292]: pgmap v553: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-11T19:16:33.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:32 smithi170 ceph-mon[35690]: pgmap v553: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-11T19:16:33.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:32 smithi029 ceph-mon[32439]: pgmap v553: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-11T19:16:35.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:34 smithi190 ceph-mon[36292]: pgmap v554: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T19:16:35.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:34 smithi170 ceph-mon[35690]: pgmap v554: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T19:16:35.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:34 smithi029 ceph-mon[32439]: pgmap v554: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T19:16:37.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:36 smithi190 ceph-mon[36292]: pgmap v555: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T19:16:37.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:36 smithi170 ceph-mon[35690]: pgmap v555: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T19:16:37.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:36 smithi029 ceph-mon[32439]: pgmap v555: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T19:16:39.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:38 smithi190 ceph-mon[36292]: pgmap v556: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-11T19:16:39.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:38 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:16:39.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:38 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:39.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:38 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:39.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:38 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:39.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:38 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:39.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:38 smithi170 ceph-mon[35690]: pgmap v556: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-11T19:16:39.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:38 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:16:39.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:38 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:39.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:38 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:39.347 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:38 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:39.347 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:38 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:39.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:38 smithi029 ceph-mon[32439]: pgmap v556: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-11T19:16:39.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:38 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:16:39.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:38 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:39.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:38 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:39.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:38 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:39.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:38 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:40.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:39 smithi190 ceph-mon[36292]: Adjusting osd_memory_target on smithi170 to 176.2M 2024-03-11T19:16:40.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:39 smithi190 ceph-mon[36292]: Unable to set osd_memory_target on smithi170 to 184790323: error parsing value: Value '184790323' is below minimum 939524096 2024-03-11T19:16:40.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:39 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:40.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:39 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:40.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:39 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:40.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:39 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:40.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:39 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:16:40.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:39 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:16:40.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:39 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:16:40.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:39 smithi170 ceph-mon[35690]: Adjusting osd_memory_target on smithi170 to 176.2M 2024-03-11T19:16:40.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:39 smithi170 ceph-mon[35690]: Unable to set osd_memory_target on smithi170 to 184790323: error parsing value: Value '184790323' is below minimum 939524096 2024-03-11T19:16:40.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:39 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:40.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:39 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:40.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:39 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:40.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:39 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:40.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:39 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:16:40.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:39 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:16:40.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:39 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:16:40.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:39 smithi029 ceph-mon[32439]: Adjusting osd_memory_target on smithi170 to 176.2M 2024-03-11T19:16:40.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:40 smithi029 ceph-mon[32439]: Unable to set osd_memory_target on smithi170 to 184790323: error parsing value: Value '184790323' is below minimum 939524096 2024-03-11T19:16:40.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:40.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:40.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:40.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:16:40.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:16:40.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:16:40.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:16:41.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:40 smithi190 ceph-mon[36292]: pgmap v557: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:16:41.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:40 smithi190 ceph-mon[36292]: Adjusting osd_memory_target on smithi029 to 177.2M 2024-03-11T19:16:41.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:40 smithi190 ceph-mon[36292]: Unable to set osd_memory_target on smithi029 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-03-11T19:16:41.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:40 smithi170 ceph-mon[35690]: pgmap v557: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:16:41.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:40 smithi170 ceph-mon[35690]: Adjusting osd_memory_target on smithi029 to 177.2M 2024-03-11T19:16:41.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:40 smithi170 ceph-mon[35690]: Unable to set osd_memory_target on smithi029 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-03-11T19:16:41.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:40 smithi029 ceph-mon[32439]: pgmap v557: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T19:16:41.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:40 smithi029 ceph-mon[32439]: Adjusting osd_memory_target on smithi029 to 177.2M 2024-03-11T19:16:41.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:40 smithi029 ceph-mon[32439]: Unable to set osd_memory_target on smithi029 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-03-11T19:16:43.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:42 smithi190 ceph-mon[36292]: pgmap v558: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-11T19:16:43.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:42 smithi170 ceph-mon[35690]: pgmap v558: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-11T19:16:43.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:42 smithi029 ceph-mon[32439]: pgmap v558: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-03-11T19:16:45.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:44 smithi190 ceph-mon[36292]: pgmap v559: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T19:16:45.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:44 smithi170 ceph-mon[35690]: pgmap v559: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T19:16:45.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:44 smithi029 ceph-mon[32439]: pgmap v559: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T19:16:47.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:46 smithi190 ceph-mon[36292]: pgmap v560: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 70 MiB/s wr, 1.14k op/s 2024-03-11T19:16:47.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:46 smithi170 ceph-mon[35690]: pgmap v560: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 70 MiB/s wr, 1.14k op/s 2024-03-11T19:16:47.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:46 smithi029 ceph-mon[32439]: pgmap v560: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 70 MiB/s wr, 1.14k op/s 2024-03-11T19:16:49.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:48 smithi190 ceph-mon[36292]: pgmap v561: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 63 MiB/s wr, 1.02k op/s 2024-03-11T19:16:49.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:48 smithi170 ceph-mon[35690]: pgmap v561: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 63 MiB/s wr, 1.02k op/s 2024-03-11T19:16:49.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:48 smithi029 ceph-mon[32439]: pgmap v561: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 63 MiB/s wr, 1.02k op/s 2024-03-11T19:16:51.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:50 smithi190 ceph-mon[36292]: pgmap v562: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:16:51.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:50 smithi170 ceph-mon[35690]: pgmap v562: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:16:51.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:50 smithi029 ceph-mon[32439]: pgmap v562: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-11T19:16:53.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:52 smithi190 ceph-mon[36292]: pgmap v563: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-11T19:16:53.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:52 smithi170 ceph-mon[35690]: pgmap v563: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-11T19:16:53.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:52 smithi029 ceph-mon[32439]: pgmap v563: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-11T19:16:53.616 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:16:53.616 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:16:53.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.796+0000 7fdc4cc9d640 1 -- 172.21.15.29:0/2248449321 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc480fff10 msgr2=0x7fdc48100350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:53.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.796+0000 7fdc4cc9d640 1 --2- 172.21.15.29:0/2248449321 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc480fff10 0x7fdc48100350 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009930 tx=0x7fdc3802bc30 comp rx=0 tx=0).stop 2024-03-11T19:16:53.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.796+0000 7fdc4cc9d640 1 -- 172.21.15.29:0/2248449321 shutdown_connections 2024-03-11T19:16:53.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.796+0000 7fdc4cc9d640 1 --2- 172.21.15.29:0/2248449321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc48101aa0 0x7fdc481083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.796+0000 7fdc4cc9d640 1 --2- 172.21.15.29:0/2248449321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc481010e0 0x7fdc48101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.796+0000 7fdc4cc9d640 1 --2- 172.21.15.29:0/2248449321 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc480fff10 0x7fdc48100350 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.796+0000 7fdc4cc9d640 1 -- 172.21.15.29:0/2248449321 >> 172.21.15.29:0/2248449321 conn(0x7fdc480fb4f0 msgr2=0x7fdc480fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:53.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.797+0000 7fdc4cc9d640 1 -- 172.21.15.29:0/2248449321 shutdown_connections 2024-03-11T19:16:53.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.797+0000 7fdc4cc9d640 1 -- 172.21.15.29:0/2248449321 wait complete. 2024-03-11T19:16:53.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.798+0000 7fdc4cc9d640 1 Processor -- start 2024-03-11T19:16:53.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.798+0000 7fdc4cc9d640 1 -- start start 2024-03-11T19:16:53.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.798+0000 7fdc4cc9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc480fff10 0x7fdc48110210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:53.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.798+0000 7fdc4cc9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc481010e0 0x7fdc48110750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:53.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.798+0000 7fdc4cc9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc48101aa0 0x7fdc481ad870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:53.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.798+0000 7fdc4cc9d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc48115710 con 0x7fdc481010e0 2024-03-11T19:16:53.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.798+0000 7fdc4cc9d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc48115880 con 0x7fdc48101aa0 2024-03-11T19:16:53.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.798+0000 7fdc4cc9d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc481addb0 con 0x7fdc480fff10 2024-03-11T19:16:53.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.798+0000 7fdc477fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc480fff10 0x7fdc48110210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:53.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.798+0000 7fdc477fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc480fff10 0x7fdc48110210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:54058/0 (socket says 172.21.15.29:54058) 2024-03-11T19:16:53.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.798+0000 7fdc477fe640 1 -- 172.21.15.29:0/2790287482 learned_addr learned my addr 172.21.15.29:0/2790287482 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:53.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.798+0000 7fdc3ffff640 1 --2- 172.21.15.29:0/2790287482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc481010e0 0x7fdc48110750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:53.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.799+0000 7fdc47fff640 1 --2- 172.21.15.29:0/2790287482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc48101aa0 0x7fdc481ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.799+0000 7fdc3ffff640 1 -- 172.21.15.29:0/2790287482 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc480fff10 msgr2=0x7fdc48110210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.799+0000 7fdc3ffff640 1 --2- 172.21.15.29:0/2790287482 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc480fff10 0x7fdc48110210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.799+0000 7fdc3ffff640 1 -- 172.21.15.29:0/2790287482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc48101aa0 msgr2=0x7fdc481ad870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.799+0000 7fdc3ffff640 1 --2- 172.21.15.29:0/2790287482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc48101aa0 0x7fdc481ad870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.799+0000 7fdc3ffff640 1 -- 172.21.15.29:0/2790287482 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc38009560 con 0x7fdc481010e0 2024-03-11T19:16:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.800+0000 7fdc3ffff640 1 --2- 172.21.15.29:0/2790287482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc481010e0 0x7fdc48110750 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fdc3000d6d0 tx=0x7fdc3000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.800+0000 7fdc457fa640 1 -- 172.21.15.29:0/2790287482 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc30005220 con 0x7fdc481010e0 2024-03-11T19:16:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.800+0000 7fdc457fa640 1 -- 172.21.15.29:0/2790287482 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdc30009cd0 con 0x7fdc481010e0 2024-03-11T19:16:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.800+0000 7fdc4cc9d640 1 -- 172.21.15.29:0/2790287482 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc481adfb0 con 0x7fdc481010e0 2024-03-11T19:16:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.800+0000 7fdc457fa640 1 -- 172.21.15.29:0/2790287482 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc30005220 con 0x7fdc481010e0 2024-03-11T19:16:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.800+0000 7fdc4cc9d640 1 -- 172.21.15.29:0/2790287482 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc481ae430 con 0x7fdc481010e0 2024-03-11T19:16:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.804+0000 7fdc50baa740 1 Processor -- start 2024-03-11T19:16:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.804+0000 7fdc50baa740 1 -- start start 2024-03-11T19:16:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.804+0000 7fdc50baa740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d34b0 0x55c13d1d38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.804+0000 7fdc50baa740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d3ea0 0x55c13d1d89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.804+0000 7fdc50baa740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d9010 0x55c13d1db4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.804+0000 7fdc50baa740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c13d143280 con 0x55c13d1d34b0 2024-03-11T19:16:53.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.804+0000 7fdc50baa740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c13d1407b0 con 0x55c13d1d3ea0 2024-03-11T19:16:53.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.804+0000 7fdc50baa740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c13d13bf90 con 0x55c13d1d9010 2024-03-11T19:16:53.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.805+0000 7fdc47fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d34b0 0x55c13d1d38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:53.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.805+0000 7fdc47fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d34b0 0x55c13d1d38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49448/0 (socket says 172.21.15.29:49448) 2024-03-11T19:16:53.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.805+0000 7fdc47fff640 1 -- 172.21.15.29:0/3390732463 learned_addr learned my addr 172.21.15.29:0/3390732463 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:53.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.805+0000 7fdc477fe640 1 --2- 172.21.15.29:0/3390732463 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d3ea0 0x55c13d1d89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:53.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.805+0000 7fdc47fff640 1 -- 172.21.15.29:0/3390732463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d9010 msgr2=0x55c13d1db4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:16:53.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.805+0000 7fdc47fff640 1 --2- 172.21.15.29:0/3390732463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d9010 0x55c13d1db4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.805+0000 7fdc47fff640 1 -- 172.21.15.29:0/3390732463 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d3ea0 msgr2=0x55c13d1d89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:53.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.805+0000 7fdc47fff640 1 --2- 172.21.15.29:0/3390732463 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d3ea0 0x55c13d1d89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.805+0000 7fdc47fff640 1 -- 172.21.15.29:0/3390732463 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c13d0fa570 con 0x55c13d1d34b0 2024-03-11T19:16:53.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.806+0000 7fdc457fa640 1 -- 172.21.15.29:0/2790287482 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fdc30009820 con 0x7fdc481010e0 2024-03-11T19:16:53.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.806+0000 7fdc47fff640 1 --2- 172.21.15.29:0/3390732463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d34b0 0x55c13d1d38d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fdc40009fd0 tx=0x7fdc400118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a57b0af1cc34e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:53.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.806+0000 7fdc457fa640 1 --2- 172.21.15.29:0/2790287482 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdc340a1a90 0x7fdc340a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:53.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.807+0000 7fdc457fa640 1 -- 172.21.15.29:0/2790287482 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fdc30009e40 con 0x7fdc481010e0 2024-03-11T19:16:53.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.807+0000 7fdc3dffb640 1 -- 172.21.15.29:0/3390732463 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc40004d20 con 0x55c13d1d34b0 2024-03-11T19:16:53.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.807+0000 7fdc3dffb640 1 -- 172.21.15.29:0/3390732463 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdc40004e80 con 0x55c13d1d34b0 2024-03-11T19:16:53.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.807+0000 7fdc3dffb640 1 -- 172.21.15.29:0/3390732463 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc40005280 con 0x55c13d1d34b0 2024-03-11T19:16:53.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.807+0000 7fdc477fe640 1 --2- 172.21.15.29:0/2790287482 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdc340a1a90 0x7fdc340a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:53.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.808+0000 7fdc50baa740 1 -- 172.21.15.29:0/3390732463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d34b0 msgr2=0x55c13d1d38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:53.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.808+0000 7fdc50baa740 1 --2- 172.21.15.29:0/3390732463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d34b0 0x55c13d1d38d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fdc40009fd0 tx=0x7fdc400118d0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.808+0000 7fdc50baa740 1 -- 172.21.15.29:0/3390732463 shutdown_connections 2024-03-11T19:16:53.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.808+0000 7fdc50baa740 1 --2- 172.21.15.29:0/3390732463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d9010 0x55c13d1db4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.808+0000 7fdc50baa740 1 --2- 172.21.15.29:0/3390732463 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d3ea0 0x55c13d1d89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.808+0000 7fdc50baa740 1 --2- 172.21.15.29:0/3390732463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d34b0 0x55c13d1d38d0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.808+0000 7fdc50baa740 1 -- 172.21.15.29:0/3390732463 >> 172.21.15.29:0/3390732463 conn(0x55c13d1cea20 msgr2=0x55c13d1cee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:53.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.808+0000 7fdc477fe640 1 --2- 172.21.15.29:0/2790287482 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdc340a1a90 0x7fdc340a3f50 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x55c13d1d4910 tx=0x7fdc3802e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:53.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.808+0000 7fdc50baa740 1 -- 172.21.15.29:0/3390732463 shutdown_connections 2024-03-11T19:16:53.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.808+0000 7fdc50baa740 1 -- 172.21.15.29:0/3390732463 wait complete. 2024-03-11T19:16:53.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.809+0000 7fdc50baa740 1 -- start start 2024-03-11T19:16:53.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.809+0000 7fdc50baa740 1 Processor -- start 2024-03-11T19:16:53.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.810+0000 7fdc50baa740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d34b0 0x55c13d1dd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.810+0000 7fdc50baa740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d3ea0 0x55c13d1dda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.810+0000 7fdc50baa740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d9010 0x55c13d1e4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.810+0000 7fdc50baa740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c13ce0f430 con 0x55c13d1d3ea0 2024-03-11T19:16:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.810+0000 7fdc50baa740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c13cdcbe00 con 0x55c13d1d34b0 2024-03-11T19:16:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.810+0000 7fdc3ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d3ea0 0x55c13d1dda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.810+0000 7fdc50baa740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c13d0c0fd0 con 0x55c13d1d9010 2024-03-11T19:16:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.810+0000 7fdc3ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d3ea0 0x55c13d1dda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49458/0 (socket says 172.21.15.29:49458) 2024-03-11T19:16:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.810+0000 7fdc3ffff640 1 -- 172.21.15.29:0/2442614494 learned_addr learned my addr 172.21.15.29:0/2442614494 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.810+0000 7fdc477fe640 1 --2- 172.21.15.29:0/2442614494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d34b0 0x55c13d1dd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.810+0000 7fdc50baa740 10 client.0 _ll_register_callbacks cb 0x55c13d116fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.810+0000 7fdc47fff640 1 --2- 172.21.15.29:0/2442614494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d9010 0x55c13d1e4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.810+0000 7fdc3ffff640 1 -- 172.21.15.29:0/2442614494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d9010 msgr2=0x55c13d1e4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.810+0000 7fdc3ffff640 1 --2- 172.21.15.29:0/2442614494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d9010 0x55c13d1e4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.810+0000 7fdc3ffff640 1 -- 172.21.15.29:0/2442614494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d34b0 msgr2=0x55c13d1dd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.810+0000 7fdc3ffff640 1 --2- 172.21.15.29:0/2442614494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d34b0 0x55c13d1dd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.810+0000 7fdc3ffff640 1 -- 172.21.15.29:0/2442614494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc40009c80 con 0x55c13d1d3ea0 2024-03-11T19:16:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.810+0000 7fdc47fff640 1 --2- 172.21.15.29:0/2442614494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d9010 0x55c13d1e4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:16:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.810+0000 7fdc477fe640 1 --2- 172.21.15.29:0/2442614494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d34b0 0x55c13d1dd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:16:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.811+0000 7fdc26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:16:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.811+0000 7fdc3ffff640 1 --2- 172.21.15.29:0/2442614494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d3ea0 0x55c13d1dda60 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fdc30009610 tx=0x7fdc300945b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a27ca4f74074c842 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.811+0000 7fdc26ffd640 1 -- 172.21.15.29:0/2442614494 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc30099410 con 0x55c13d1d3ea0 2024-03-11T19:16:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.811+0000 7fdc26ffd640 1 -- 172.21.15.29:0/2442614494 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fdc30009e40 con 0x55c13d1d3ea0 2024-03-11T19:16:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.811+0000 7fdc50baa740 1 -- 172.21.15.29:0/2442614494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c13ce78720 con 0x55c13d1d3ea0 2024-03-11T19:16:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.812+0000 7fdc26ffd640 1 -- 172.21.15.29:0/2442614494 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc3009a1b0 con 0x55c13d1d3ea0 2024-03-11T19:16:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.812+0000 7fdc50baa740 10 client.17061 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.812+0000 7fdc50baa740 1 -- 172.21.15.29:0/2442614494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55c13d13a610 con 0x55c13d1d3ea0 2024-03-11T19:16:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.812+0000 7fdc26ffd640 1 -- 172.21.15.29:0/2442614494 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55c13ce78720 con 0x55c13d1d3ea0 2024-03-11T19:16:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.812+0000 7fdc26ffd640 1 -- 172.21.15.29:0/2442614494 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fdc30095800 con 0x55c13d1d3ea0 2024-03-11T19:16:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.812+0000 7fdc50baa740 10 client.17061 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.813+0000 7fdc50baa740 10 client.17061 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:16:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.813+0000 7fdc50baa740 20 client.17061 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.813+0000 7fdc50baa740 1 --2- 172.21.15.29:0/2442614494 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c13d1e9610 0x55c13d1eba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.813+0000 7fdc50baa740 4 client.17061 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:16:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.813+0000 7fdc50baa740 1 -- 172.21.15.29:0/2442614494 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c13cec7890 con 0x55c13d1e9610 2024-03-11T19:16:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.813+0000 7fdc477fe640 1 --2- 172.21.15.29:0/2442614494 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c13d1e9610 0x55c13d1eba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.813+0000 7fdc477fe640 1 --2- 172.21.15.29:0/2442614494 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c13d1e9610 0x55c13d1eba00 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbb2c3eb5bf18602 server_cookie=de1c06ca0b970a5b in_seq=0 out_seq=0 2024-03-11T19:16:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.813+0000 7fdc26ffd640 10 client.17061 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:53.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.814+0000 7fdc26ffd640 1 -- 172.21.15.29:0/2442614494 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c13cec7890 con 0x55c13d1e9610 2024-03-11T19:16:53.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.814+0000 7fdc26ffd640 10 client.17061 handle_command_reply: tid=0 2024-03-11T19:16:53.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.814+0000 7fdc50baa740 1 client.17061 shutdown 2024-03-11T19:16:53.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.815+0000 7fdc50baa740 1 -- 172.21.15.29:0/2442614494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d3ea0 msgr2=0x55c13d1dda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:53.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.815+0000 7fdc50baa740 1 --2- 172.21.15.29:0/2442614494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d3ea0 0x55c13d1dda60 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fdc30009610 tx=0x7fdc300945b0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.815+0000 7fdc50baa740 1 -- 172.21.15.29:0/2442614494 shutdown_connections 2024-03-11T19:16:53.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.815+0000 7fdc50baa740 1 --2- 172.21.15.29:0/2442614494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d9010 0x55c13d1e4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.815+0000 7fdc50baa740 1 --2- 172.21.15.29:0/2442614494 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c13d1e9610 0x55c13d1eba00 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.815+0000 7fdc50baa740 1 --2- 172.21.15.29:0/2442614494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d3ea0 0x55c13d1dda60 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.815+0000 7fdc26ffd640 0 client.17061 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:53.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.815+0000 7fdc50baa740 1 --2- 172.21.15.29:0/2442614494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d34b0 0x55c13d1dd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.815+0000 7fdc50baa740 1 -- 172.21.15.29:0/2442614494 >> 172.21.15.29:0/2442614494 conn(0x55c13d1cea20 msgr2=0x55c13d1cdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:53.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.816+0000 7fdc50baa740 1 -- 172.21.15.29:0/2442614494 shutdown_connections 2024-03-11T19:16:53.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.816+0000 7fdc50baa740 1 -- 172.21.15.29:0/2442614494 wait complete. 2024-03-11T19:16:53.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.816+0000 7fdc50baa740 20 client.17061 trim_cache size 0 max 16384 2024-03-11T19:16:53.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.846+0000 7fdc50baa740 1 Processor -- start 2024-03-11T19:16:53.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.846+0000 7fdc50baa740 1 -- start start 2024-03-11T19:16:53.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.846+0000 7fdc50baa740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d34b0 0x55c13d1c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:53.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.846+0000 7fdc50baa740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d3ea0 0x55c13d1c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:53.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.846+0000 7fdc50baa740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d9010 0x55c13d1e4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:53.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.847+0000 7fdc50baa740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c13d1cf510 con 0x55c13d1d9010 2024-03-11T19:16:53.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.847+0000 7fdc50baa740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c13cec78c0 con 0x55c13d1d3ea0 2024-03-11T19:16:53.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.847+0000 7fdc50baa740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c13d1ce560 con 0x55c13d1d34b0 2024-03-11T19:16:53.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.847+0000 7fdc47fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d9010 0x55c13d1e4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:53.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.847+0000 7fdc47fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d9010 0x55c13d1e4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49470/0 (socket says 172.21.15.29:49470) 2024-03-11T19:16:53.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.847+0000 7fdc47fff640 1 -- 172.21.15.29:0/85459185 learned_addr learned my addr 172.21.15.29:0/85459185 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:53.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.847+0000 7fdc477fe640 1 --2- 172.21.15.29:0/85459185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d34b0 0x55c13d1c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:53.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.847+0000 7fdc477fe640 1 -- 172.21.15.29:0/85459185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d3ea0 msgr2=0x55c13d1c9ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:16:53.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.847+0000 7fdc477fe640 1 --2- 172.21.15.29:0/85459185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d3ea0 0x55c13d1c9ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.847+0000 7fdc477fe640 1 -- 172.21.15.29:0/85459185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d9010 msgr2=0x55c13d1e4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:53.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.847+0000 7fdc477fe640 1 --2- 172.21.15.29:0/85459185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d9010 0x55c13d1e4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.847+0000 7fdc477fe640 1 -- 172.21.15.29:0/85459185 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c13d0d6ac0 con 0x55c13d1d34b0 2024-03-11T19:16:53.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.848+0000 7fdc477fe640 1 --2- 172.21.15.29:0/85459185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d34b0 0x55c13d1c99a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdc38006010 tx=0x7fdc38041000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30c0e44b4c9555a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:53.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.848+0000 7fdc3d7fa640 1 -- 172.21.15.29:0/85459185 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc380042c0 con 0x55c13d1d34b0 2024-03-11T19:16:53.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.848+0000 7fdc3d7fa640 1 -- 172.21.15.29:0/85459185 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdc38004420 con 0x55c13d1d34b0 2024-03-11T19:16:53.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.848+0000 7fdc3d7fa640 1 -- 172.21.15.29:0/85459185 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc38003af0 con 0x55c13d1d34b0 2024-03-11T19:16:53.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.849+0000 7fdc50baa740 1 -- 172.21.15.29:0/85459185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d34b0 msgr2=0x55c13d1c99a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:53.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.849+0000 7fdc50baa740 1 --2- 172.21.15.29:0/85459185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d34b0 0x55c13d1c99a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdc38006010 tx=0x7fdc38041000 comp rx=0 tx=0).stop 2024-03-11T19:16:53.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.850+0000 7fdc50baa740 1 -- 172.21.15.29:0/85459185 shutdown_connections 2024-03-11T19:16:53.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.850+0000 7fdc50baa740 1 --2- 172.21.15.29:0/85459185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d9010 0x55c13d1e4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.850+0000 7fdc50baa740 1 --2- 172.21.15.29:0/85459185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d3ea0 0x55c13d1c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.850+0000 7fdc50baa740 1 --2- 172.21.15.29:0/85459185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d34b0 0x55c13d1c99a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.850+0000 7fdc50baa740 1 -- 172.21.15.29:0/85459185 >> 172.21.15.29:0/85459185 conn(0x55c13d1cea20 msgr2=0x55c13d1cee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:53.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.850+0000 7fdc50baa740 1 -- 172.21.15.29:0/85459185 shutdown_connections 2024-03-11T19:16:53.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.850+0000 7fdc50baa740 1 -- 172.21.15.29:0/85459185 wait complete. 2024-03-11T19:16:53.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.850+0000 7fdc50baa740 1 -- start start 2024-03-11T19:16:53.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.851+0000 7fdc50baa740 1 Processor -- start 2024-03-11T19:16:53.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.851+0000 7fdc50baa740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d34b0 0x55c13d1e4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:53.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.851+0000 7fdc50baa740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d3ea0 0x55c13d1e2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:53.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.851+0000 7fdc50baa740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d9010 0x55c13d1e33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:53.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.851+0000 7fdc50baa740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c13d1e38e0 con 0x55c13d1d9010 2024-03-11T19:16:53.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.851+0000 7fdc50baa740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c13d1e3a50 con 0x55c13d1d34b0 2024-03-11T19:16:53.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.851+0000 7fdc50baa740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c13d1e3bc0 con 0x55c13d1d3ea0 2024-03-11T19:16:53.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.851+0000 7fdc50baa740 10 client.0 _ll_register_callbacks cb 0x7fdc40009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:53.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.851+0000 7fdc477fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d34b0 0x55c13d1e4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:53.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.851+0000 7fdc477fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d34b0 0x55c13d1e4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:58950/0 (socket says 172.21.15.29:58950) 2024-03-11T19:16:53.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.851+0000 7fdc477fe640 1 -- 172.21.15.29:0/93428174 learned_addr learned my addr 172.21.15.29:0/93428174 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:53.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.851+0000 7fdc3ffff640 1 --2- 172.21.15.29:0/93428174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d3ea0 0x55c13d1e2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:53.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.852+0000 7fdc47fff640 1 --2- 172.21.15.29:0/93428174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d9010 0x55c13d1e33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:53.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.852+0000 7fdc3ffff640 1 -- 172.21.15.29:0/93428174 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d34b0 msgr2=0x55c13d1e4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:53.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.852+0000 7fdc3ffff640 1 --2- 172.21.15.29:0/93428174 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d34b0 0x55c13d1e4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.852+0000 7fdc3ffff640 1 -- 172.21.15.29:0/93428174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d9010 msgr2=0x55c13d1e33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:53.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.852+0000 7fdc3ffff640 1 --2- 172.21.15.29:0/93428174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d9010 0x55c13d1e33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.852+0000 7fdc3ffff640 1 -- 172.21.15.29:0/93428174 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc48108c80 con 0x55c13d1d3ea0 2024-03-11T19:16:53.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.852+0000 7fdc3ffff640 1 --2- 172.21.15.29:0/93428174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d3ea0 0x55c13d1e2e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55c13d1d4c80 tx=0x7fdc300057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae4e028c4ddd6219 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:53.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.852+0000 7fdc26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:16:53.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.853+0000 7fdc26ffd640 1 -- 172.21.15.29:0/93428174 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc30014060 con 0x55c13d1d3ea0 2024-03-11T19:16:53.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.853+0000 7fdc26ffd640 1 -- 172.21.15.29:0/93428174 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fdc30009e40 con 0x55c13d1d3ea0 2024-03-11T19:16:53.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.853+0000 7fdc50baa740 1 -- 172.21.15.29:0/93428174 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c13ce3b7e0 con 0x55c13d1d3ea0 2024-03-11T19:16:53.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.853+0000 7fdc26ffd640 1 -- 172.21.15.29:0/93428174 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc30099410 con 0x55c13d1d3ea0 2024-03-11T19:16:53.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.853+0000 7fdc50baa740 10 client.26587 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:53.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.853+0000 7fdc50baa740 1 -- 172.21.15.29:0/93428174 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55c13d0fc6a0 con 0x55c13d1d3ea0 2024-03-11T19:16:53.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.853+0000 7fdc26ffd640 1 -- 172.21.15.29:0/93428174 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55c13ce3b7e0 con 0x55c13d1d3ea0 2024-03-11T19:16:53.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.853+0000 7fdc26ffd640 1 -- 172.21.15.29:0/93428174 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fdc3009eb20 con 0x55c13d1d3ea0 2024-03-11T19:16:53.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.853+0000 7fdc50baa740 10 client.26587 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:53.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.853+0000 7fdc50baa740 10 client.26587 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:16:53.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.853+0000 7fdc50baa740 20 client.26587 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:53.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.853+0000 7fdc50baa740 1 --2- 172.21.15.29:0/93428174 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c13d1e9610 0x55c13d242e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:53.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.854+0000 7fdc50baa740 4 client.26587 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:16:53.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.854+0000 7fdc50baa740 1 -- 172.21.15.29:0/93428174 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c13ce2a4e0 con 0x55c13d1e9610 2024-03-11T19:16:53.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.854+0000 7fdc477fe640 1 --2- 172.21.15.29:0/93428174 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c13d1e9610 0x55c13d242e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:53.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.855+0000 7fdc477fe640 1 --2- 172.21.15.29:0/93428174 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c13d1e9610 0x55c13d242e20 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3f08aac1a80df7b server_cookie=fced4d548b2e6d80 in_seq=0 out_seq=0 2024-03-11T19:16:53.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.855+0000 7fdc26ffd640 10 client.26587 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:53.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.855+0000 7fdc26ffd640 1 -- 172.21.15.29:0/93428174 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55c13ce2a4e0 con 0x55c13d1e9610 2024-03-11T19:16:53.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.855+0000 7fdc26ffd640 10 client.26587 handle_command_reply: tid=0 2024-03-11T19:16:53.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.855+0000 7fdc50baa740 1 client.26587 shutdown 2024-03-11T19:16:53.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.856+0000 7fdc50baa740 1 -- 172.21.15.29:0/93428174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d3ea0 msgr2=0x55c13d1e2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:53.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.856+0000 7fdc50baa740 1 --2- 172.21.15.29:0/93428174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d3ea0 0x55c13d1e2e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55c13d1d4c80 tx=0x7fdc300057f0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.856+0000 7fdc50baa740 1 -- 172.21.15.29:0/93428174 shutdown_connections 2024-03-11T19:16:53.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.856+0000 7fdc50baa740 1 --2- 172.21.15.29:0/93428174 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c13d1e9610 0x55c13d242e20 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.856+0000 7fdc50baa740 1 --2- 172.21.15.29:0/93428174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c13d1d9010 0x55c13d1e33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.856+0000 7fdc26ffd640 0 client.26587 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:53.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.856+0000 7fdc50baa740 1 --2- 172.21.15.29:0/93428174 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c13d1d3ea0 0x55c13d1e2e60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.856+0000 7fdc50baa740 1 --2- 172.21.15.29:0/93428174 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c13d1d34b0 0x55c13d1e4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.856+0000 7fdc50baa740 1 -- 172.21.15.29:0/93428174 >> 172.21.15.29:0/93428174 conn(0x55c13d1cea20 msgr2=0x55c13d1c99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:53.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.856+0000 7fdc50baa740 1 -- 172.21.15.29:0/93428174 shutdown_connections 2024-03-11T19:16:53.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.856+0000 7fdc50baa740 1 -- 172.21.15.29:0/93428174 wait complete. 2024-03-11T19:16:53.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.856+0000 7fdc50baa740 20 client.26587 trim_cache size 0 max 16384 2024-03-11T19:16:53.861 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:16:53.861 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:16:53.861 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:16:53.861 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:16:53.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.857+0000 7fdc4cc9d640 1 -- 172.21.15.29:0/2790287482 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdc340a1a90 msgr2=0x7fdc340a3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:53.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.857+0000 7fdc4cc9d640 1 --2- 172.21.15.29:0/2790287482 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdc340a1a90 0x7fdc340a3f50 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x55c13d1d4910 tx=0x7fdc3802e040 comp rx=0 tx=0).stop 2024-03-11T19:16:53.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.857+0000 7fdc4cc9d640 1 -- 172.21.15.29:0/2790287482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc481010e0 msgr2=0x7fdc48110750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:53.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.858+0000 7fdc4cc9d640 1 --2- 172.21.15.29:0/2790287482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc481010e0 0x7fdc48110750 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fdc3000d6d0 tx=0x7fdc3000db90 comp rx=0 tx=0).stop 2024-03-11T19:16:53.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.858+0000 7fdc4cc9d640 1 -- 172.21.15.29:0/2790287482 shutdown_connections 2024-03-11T19:16:53.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.858+0000 7fdc4cc9d640 1 --2- 172.21.15.29:0/2790287482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc48101aa0 0x7fdc481ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.858+0000 7fdc4cc9d640 1 --2- 172.21.15.29:0/2790287482 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fdc340a1a90 0x7fdc340a3f50 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.858+0000 7fdc4cc9d640 1 --2- 172.21.15.29:0/2790287482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc481010e0 0x7fdc48110750 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.858+0000 7fdc4cc9d640 1 --2- 172.21.15.29:0/2790287482 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc480fff10 0x7fdc48110210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:53.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.858+0000 7fdc4cc9d640 1 -- 172.21.15.29:0/2790287482 >> 172.21.15.29:0/2790287482 conn(0x7fdc480fb4f0 msgr2=0x7fdc48104650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:53.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.858+0000 7fdc4cc9d640 1 -- 172.21.15.29:0/2790287482 shutdown_connections 2024-03-11T19:16:53.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.858+0000 7fdc4cc9d640 1 -- 172.21.15.29:0/2790287482 wait complete. 2024-03-11T19:16:53.878 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:16:53.878 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:16:54.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.995+0000 7f614fc8d640 1 -- 172.21.15.29:0/3480642300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61481016b0 msgr2=0x7f6148101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.995+0000 7f614fc8d640 1 --2- 172.21.15.29:0/3480642300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61481016b0 0x7f6148101ad0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f6144009900 tx=0x7f614402bc30 comp rx=0 tx=0).stop 2024-03-11T19:16:54.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.995+0000 7f614fc8d640 1 -- 172.21.15.29:0/3480642300 shutdown_connections 2024-03-11T19:16:54.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.995+0000 7f614fc8d640 1 --2- 172.21.15.29:0/3480642300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6148100d10 0x7f61481085b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.995+0000 7f614fc8d640 1 --2- 172.21.15.29:0/3480642300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6148100350 0x7f61481007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.995+0000 7f614fc8d640 1 --2- 172.21.15.29:0/3480642300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61481016b0 0x7f6148101ad0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.995+0000 7f614fc8d640 1 -- 172.21.15.29:0/3480642300 >> 172.21.15.29:0/3480642300 conn(0x7f61480fada0 msgr2=0x7f61480fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.995+0000 7f614fc8d640 1 -- 172.21.15.29:0/3480642300 shutdown_connections 2024-03-11T19:16:54.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.995+0000 7f614fc8d640 1 -- 172.21.15.29:0/3480642300 wait complete. 2024-03-11T19:16:54.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614fc8d640 1 Processor -- start 2024-03-11T19:16:54.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614fc8d640 1 -- start start 2024-03-11T19:16:54.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614fc8d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6148100350 0x7f61480ffe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614fc8d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6148100d10 0x7f614819be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614fc8d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61481016b0 0x7f614819e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614fc8d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614819ec30 con 0x7f61481016b0 2024-03-11T19:16:54.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614fc8d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614819eda0 con 0x7f6148100d10 2024-03-11T19:16:54.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614fc8d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614819ef60 con 0x7f6148100350 2024-03-11T19:16:54.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614e22a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61481016b0 0x7f614819e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614da29640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6148100350 0x7f61480ffe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614da29640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6148100350 0x7f61480ffe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:54110/0 (socket says 172.21.15.29:54110) 2024-03-11T19:16:54.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614da29640 1 -- 172.21.15.29:0/3252879837 learned_addr learned my addr 172.21.15.29:0/3252879837 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:54.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614e22a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61481016b0 0x7f614819e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49496/0 (socket says 172.21.15.29:49496) 2024-03-11T19:16:54.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614d228640 1 --2- 172.21.15.29:0/3252879837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6148100d10 0x7f614819be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614e22a640 1 -- 172.21.15.29:0/3252879837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6148100350 msgr2=0x7f61480ffe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614e22a640 1 --2- 172.21.15.29:0/3252879837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6148100350 0x7f61480ffe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614e22a640 1 -- 172.21.15.29:0/3252879837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6148100d10 msgr2=0x7f614819be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614e22a640 1 --2- 172.21.15.29:0/3252879837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6148100d10 0x7f614819be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614e22a640 1 -- 172.21.15.29:0/3252879837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6144009560 con 0x7f61481016b0 2024-03-11T19:16:54.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614da29640 1 --2- 172.21.15.29:0/3252879837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6148100350 0x7f61480ffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:16:54.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.996+0000 7f614e22a640 1 --2- 172.21.15.29:0/3252879837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61481016b0 0x7f614819e4a0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f614000e940 tx=0x7f614000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.997+0000 7f613effd640 1 -- 172.21.15.29:0/3252879837 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6140004d20 con 0x7f61481016b0 2024-03-11T19:16:54.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.997+0000 7f613effd640 1 -- 172.21.15.29:0/3252879837 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6140004e80 con 0x7f61481016b0 2024-03-11T19:16:54.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.997+0000 7f614fc8d640 1 -- 172.21.15.29:0/3252879837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f614819f240 con 0x7f61481016b0 2024-03-11T19:16:54.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.997+0000 7f613effd640 1 -- 172.21.15.29:0/3252879837 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6140011470 con 0x7f61481016b0 2024-03-11T19:16:54.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.997+0000 7f614fc8d640 1 -- 172.21.15.29:0/3252879837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f614819f750 con 0x7f61481016b0 2024-03-11T19:16:54.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.998+0000 7f614fc8d640 1 -- 172.21.15.29:0/3252879837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61481074b0 con 0x7f61481016b0 2024-03-11T19:16:54.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.999+0000 7f613effd640 1 -- 172.21.15.29:0/3252879837 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f61400052b0 con 0x7f61481016b0 2024-03-11T19:16:54.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.999+0000 7f613effd640 1 --2- 172.21.15.29:0/3252879837 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f61340a1a30 0x7f61340a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.999+0000 7f613effd640 1 -- 172.21.15.29:0/3252879837 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f61400cd7b0 con 0x7f61481016b0 2024-03-11T19:16:54.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:53.999+0000 7f614da29640 1 --2- 172.21.15.29:0/3252879837 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f61340a1a30 0x7f61340a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.000+0000 7f614da29640 1 --2- 172.21.15.29:0/3252879837 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f61340a1a30 0x7f61340a3ef0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6144005940 tx=0x7f6144005890 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.003+0000 7f613effd640 1 -- 172.21.15.29:0/3252879837 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6140096a60 con 0x7f61481016b0 2024-03-11T19:16:54.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.369+0000 7f614fc8d640 1 -- 172.21.15.29:0/3252879837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f614810b9a0 con 0x7f61481016b0 2024-03-11T19:16:54.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.371+0000 7f613effd640 1 -- 172.21.15.29:0/3252879837 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f614001f090 con 0x7f61481016b0 2024-03-11T19:16:54.375 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:16:54.375 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:16:54.375 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:16:54.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.374+0000 7f614fc8d640 1 -- 172.21.15.29:0/3252879837 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f61340a1a30 msgr2=0x7f61340a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.374+0000 7f614fc8d640 1 --2- 172.21.15.29:0/3252879837 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f61340a1a30 0x7f61340a3ef0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6144005940 tx=0x7f6144005890 comp rx=0 tx=0).stop 2024-03-11T19:16:54.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.374+0000 7f614fc8d640 1 -- 172.21.15.29:0/3252879837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61481016b0 msgr2=0x7f614819e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.374+0000 7f614fc8d640 1 --2- 172.21.15.29:0/3252879837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61481016b0 0x7f614819e4a0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f614000e940 tx=0x7f614000ee00 comp rx=0 tx=0).stop 2024-03-11T19:16:54.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.374+0000 7f614fc8d640 1 -- 172.21.15.29:0/3252879837 shutdown_connections 2024-03-11T19:16:54.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.374+0000 7f614fc8d640 1 --2- 172.21.15.29:0/3252879837 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f61340a1a30 0x7f61340a3ef0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.374+0000 7f614fc8d640 1 --2- 172.21.15.29:0/3252879837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61481016b0 0x7f614819e4a0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.374+0000 7f614fc8d640 1 --2- 172.21.15.29:0/3252879837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6148100d10 0x7f614819be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.374+0000 7f614fc8d640 1 --2- 172.21.15.29:0/3252879837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6148100350 0x7f61480ffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.374+0000 7f614fc8d640 1 -- 172.21.15.29:0/3252879837 >> 172.21.15.29:0/3252879837 conn(0x7f61480fada0 msgr2=0x7f6148104680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.374+0000 7f614fc8d640 1 -- 172.21.15.29:0/3252879837 shutdown_connections 2024-03-11T19:16:54.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.374+0000 7f614fc8d640 1 -- 172.21.15.29:0/3252879837 wait complete. 2024-03-11T19:16:54.394 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:16:54.395 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:16:54.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.526+0000 7fc3af42e640 1 -- 172.21.15.29:0/1693717346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a8101270 msgr2=0x7fc3a81016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.526+0000 7fc3af42e640 1 --2- 172.21.15.29:0/1693717346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a8101270 0x7fc3a81016f0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fc39c009900 tx=0x7fc39c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:16:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.526+0000 7fc3af42e640 1 -- 172.21.15.29:0/1693717346 shutdown_connections 2024-03-11T19:16:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.526+0000 7fc3af42e640 1 --2- 172.21.15.29:0/1693717346 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3a8101c30 0x7fc3a81084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.526+0000 7fc3af42e640 1 --2- 172.21.15.29:0/1693717346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a8101270 0x7fc3a81016f0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.526+0000 7fc3af42e640 1 --2- 172.21.15.29:0/1693717346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3a8100080 0x7fc3a81004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.526+0000 7fc3af42e640 1 -- 172.21.15.29:0/1693717346 >> 172.21.15.29:0/1693717346 conn(0x7fc3a80fb610 msgr2=0x7fc3a80fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.527+0000 7fc3af42e640 1 -- 172.21.15.29:0/1693717346 shutdown_connections 2024-03-11T19:16:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.527+0000 7fc3af42e640 1 -- 172.21.15.29:0/1693717346 wait complete. 2024-03-11T19:16:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.527+0000 7fc3af42e640 1 Processor -- start 2024-03-11T19:16:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.527+0000 7fc3af42e640 1 -- start start 2024-03-11T19:16:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.527+0000 7fc3af42e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a8100080 0x7fc3a8195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.527+0000 7fc3af42e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3a8101270 0x7fc3a8196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.527+0000 7fc3af42e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3a8101c30 0x7fc3a819d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.527+0000 7fc3af42e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a819dad0 con 0x7fc3a8100080 2024-03-11T19:16:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.527+0000 7fc3af42e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a819dc40 con 0x7fc3a8101270 2024-03-11T19:16:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.527+0000 7fc3af42e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a819de00 con 0x7fc3a8101c30 2024-03-11T19:16:54.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.527+0000 7fc3ad1ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a8100080 0x7fc3a8195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.527+0000 7fc3ad1ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a8100080 0x7fc3a8195fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49512/0 (socket says 172.21.15.29:49512) 2024-03-11T19:16:54.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.527+0000 7fc3ad1ca640 1 -- 172.21.15.29:0/687101751 learned_addr learned my addr 172.21.15.29:0/687101751 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:54.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.528+0000 7fc3ad9cb640 1 --2- 172.21.15.29:0/687101751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3a8101c30 0x7fc3a819d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.528+0000 7fc3ad1ca640 1 -- 172.21.15.29:0/687101751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3a8101c30 msgr2=0x7fc3a819d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.528+0000 7fc3ad1ca640 1 --2- 172.21.15.29:0/687101751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3a8101c30 0x7fc3a819d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.528+0000 7fc3ad1ca640 1 -- 172.21.15.29:0/687101751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3a8101270 msgr2=0x7fc3a8196520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.528+0000 7fc3ac9c9640 1 --2- 172.21.15.29:0/687101751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3a8101270 0x7fc3a8196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.528+0000 7fc3ad1ca640 1 --2- 172.21.15.29:0/687101751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3a8101270 0x7fc3a8196520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.528+0000 7fc3ad1ca640 1 -- 172.21.15.29:0/687101751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39c009560 con 0x7fc3a8100080 2024-03-11T19:16:54.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.528+0000 7fc3ad1ca640 1 --2- 172.21.15.29:0/687101751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a8100080 0x7fc3a8195fe0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fc3a000c970 tx=0x7fc3a000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.528+0000 7fc3a67fc640 1 -- 172.21.15.29:0/687101751 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0004220 con 0x7fc3a8100080 2024-03-11T19:16:54.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.528+0000 7fc3a67fc640 1 -- 172.21.15.29:0/687101751 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc3a0007e60 con 0x7fc3a8100080 2024-03-11T19:16:54.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.528+0000 7fc3a67fc640 1 -- 172.21.15.29:0/687101751 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0003e40 con 0x7fc3a8100080 2024-03-11T19:16:54.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.528+0000 7fc3af42e640 1 -- 172.21.15.29:0/687101751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a819e0e0 con 0x7fc3a8100080 2024-03-11T19:16:54.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.528+0000 7fc3af42e640 1 -- 172.21.15.29:0/687101751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a819e5f0 con 0x7fc3a8100080 2024-03-11T19:16:54.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.530+0000 7fc3a67fc640 1 -- 172.21.15.29:0/687101751 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc3a0003610 con 0x7fc3a8100080 2024-03-11T19:16:54.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3b2002740 1 Processor -- start 2024-03-11T19:16:54.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3b2002740 1 -- start start 2024-03-11T19:16:54.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3b2002740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbbb270 0x55e72fbbb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3b2002740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbbbc60 0x55e72fbc0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3b2002740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbc0dd0 0x55e72fbc3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3b2002740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e72fb77960 con 0x55e72fbc0dd0 2024-03-11T19:16:54.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3b2002740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e72fb2b310 con 0x55e72fbbbc60 2024-03-11T19:16:54.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3b2002740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e72fb28840 con 0x55e72fbbb270 2024-03-11T19:16:54.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3a67fc640 1 --2- 172.21.15.29:0/687101751 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc3900a1a30 0x7fc3900a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3ac9c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbc0dd0 0x55e72fbc3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3ac9c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbc0dd0 0x55e72fbc3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49528/0 (socket says 172.21.15.29:49528) 2024-03-11T19:16:54.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3ac9c9640 1 -- 172.21.15.29:0/4039964575 learned_addr learned my addr 172.21.15.29:0/4039964575 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:54.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3ad1ca640 1 --2- 172.21.15.29:0/4039964575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbbbc60 0x55e72fbc0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3ad9cb640 1 --2- 172.21.15.29:0/687101751 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc3900a1a30 0x7fc3900a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3ad9cb640 1 --2- 172.21.15.29:0/4039964575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbbb270 0x55e72fbbb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3a67fc640 1 -- 172.21.15.29:0/687101751 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fc3a00cd0f0 con 0x7fc3a8100080 2024-03-11T19:16:54.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3ac9c9640 1 -- 172.21.15.29:0/4039964575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbbb270 msgr2=0x55e72fbbb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3ac9c9640 1 --2- 172.21.15.29:0/4039964575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbbb270 0x55e72fbbb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3ac9c9640 1 -- 172.21.15.29:0/4039964575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbbbc60 msgr2=0x55e72fbc0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3ac9c9640 1 --2- 172.21.15.29:0/4039964575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbbbc60 0x55e72fbc0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.531+0000 7fc3ac9c9640 1 -- 172.21.15.29:0/4039964575 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e72fae6060 con 0x55e72fbc0dd0 2024-03-11T19:16:54.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.532+0000 7fc3ac9c9640 1 --2- 172.21.15.29:0/4039964575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbc0dd0 0x55e72fbc3260 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fc39c000c00 tx=0x7fc39c02c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ac1d12eea4ea86 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.532+0000 7fc3ad9cb640 1 --2- 172.21.15.29:0/687101751 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc3900a1a30 0x7fc3900a3ef0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x55e72fbbc4c0 tx=0x7fc398007410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.532+0000 7fc38f7fe640 1 -- 172.21.15.29:0/4039964575 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c02cbf0 con 0x55e72fbc0dd0 2024-03-11T19:16:54.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.532+0000 7fc38f7fe640 1 -- 172.21.15.29:0/4039964575 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc39c0044e0 con 0x55e72fbc0dd0 2024-03-11T19:16:54.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.532+0000 7fc38f7fe640 1 -- 172.21.15.29:0/4039964575 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c003d10 con 0x55e72fbc0dd0 2024-03-11T19:16:54.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.532+0000 7fc3b2002740 1 -- 172.21.15.29:0/4039964575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbc0dd0 msgr2=0x55e72fbc3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.532+0000 7fc3b2002740 1 --2- 172.21.15.29:0/4039964575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbc0dd0 0x55e72fbc3260 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fc39c000c00 tx=0x7fc39c02c940 comp rx=0 tx=0).stop 2024-03-11T19:16:54.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.533+0000 7fc3b2002740 1 -- 172.21.15.29:0/4039964575 shutdown_connections 2024-03-11T19:16:54.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.533+0000 7fc3b2002740 1 --2- 172.21.15.29:0/4039964575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbc0dd0 0x55e72fbc3260 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.533+0000 7fc3b2002740 1 --2- 172.21.15.29:0/4039964575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbbbc60 0x55e72fbc0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.533+0000 7fc3b2002740 1 --2- 172.21.15.29:0/4039964575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbbb270 0x55e72fbbb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.533+0000 7fc3b2002740 1 -- 172.21.15.29:0/4039964575 >> 172.21.15.29:0/4039964575 conn(0x55e72fbb67e0 msgr2=0x55e72fbb6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.533+0000 7fc3b2002740 1 -- 172.21.15.29:0/4039964575 shutdown_connections 2024-03-11T19:16:54.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.533+0000 7fc3b2002740 1 -- 172.21.15.29:0/4039964575 wait complete. 2024-03-11T19:16:54.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.533+0000 7fc3b2002740 1 -- start start 2024-03-11T19:16:54.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.533+0000 7fc3b2002740 1 Processor -- start 2024-03-11T19:16:54.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.533+0000 7fc3b2002740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbbb270 0x55e72fbc53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.533+0000 7fc3b2002740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbbc60 0x55e72fbc5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.533+0000 7fc3b2002740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbc0dd0 0x55e72fbcc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.533+0000 7fc3b2002740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e72fb24020 con 0x55e72fbbbc60 2024-03-11T19:16:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.533+0000 7fc3b2002740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e72f7f7430 con 0x55e72fbc0dd0 2024-03-11T19:16:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.533+0000 7fc3b2002740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e72f7b3e00 con 0x55e72fbbb270 2024-03-11T19:16:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.533+0000 7fc3b2002740 10 client.0 _ll_register_callbacks cb 0x55e72facbcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.533+0000 7fc3ac9c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbbc60 0x55e72fbc5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.533+0000 7fc3ac9c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbbc60 0x55e72fbc5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49534/0 (socket says 172.21.15.29:49534) 2024-03-11T19:16:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.533+0000 7fc3ac9c9640 1 -- 172.21.15.29:0/2276672458 learned_addr learned my addr 172.21.15.29:0/2276672458 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.534+0000 7fc3ad1ca640 1 --2- 172.21.15.29:0/2276672458 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbbb270 0x55e72fbc53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.534+0000 7fc3ad9cb640 1 --2- 172.21.15.29:0/2276672458 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbc0dd0 0x55e72fbcc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.534+0000 7fc3ac9c9640 1 -- 172.21.15.29:0/2276672458 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbbb270 msgr2=0x55e72fbc53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.534+0000 7fc3ac9c9640 1 --2- 172.21.15.29:0/2276672458 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbbb270 0x55e72fbc53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.534+0000 7fc3ac9c9640 1 -- 172.21.15.29:0/2276672458 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbc0dd0 msgr2=0x55e72fbcc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.534+0000 7fc3ac9c9640 1 --2- 172.21.15.29:0/2276672458 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbc0dd0 0x55e72fbcc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.534+0000 7fc3ac9c9640 1 -- 172.21.15.29:0/2276672458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a8108dc0 con 0x55e72fbbbc60 2024-03-11T19:16:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.534+0000 7fc3ac9c9640 1 --2- 172.21.15.29:0/2276672458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbbc60 0x55e72fbc5930 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fc39c02f020 tx=0x7fc39c009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc54f22be899052 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.534+0000 7fc38cff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:16:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.534+0000 7fc38cff9640 1 -- 172.21.15.29:0/2276672458 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c0051d0 con 0x55e72fbbbc60 2024-03-11T19:16:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.534+0000 7fc38cff9640 1 -- 172.21.15.29:0/2276672458 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fc39c005330 con 0x55e72fbbbc60 2024-03-11T19:16:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.534+0000 7fc3b2002740 1 -- 172.21.15.29:0/2276672458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e72f8237e0 con 0x55e72fbbbc60 2024-03-11T19:16:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.534+0000 7fc38cff9640 1 -- 172.21.15.29:0/2276672458 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc39c0054f0 con 0x55e72fbbbc60 2024-03-11T19:16:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.534+0000 7fc3b2002740 10 client.17085 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.534+0000 7fc3b2002740 1 -- 172.21.15.29:0/2276672458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e72fb226a0 con 0x55e72fbbbc60 2024-03-11T19:16:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.534+0000 7fc38cff9640 1 -- 172.21.15.29:0/2276672458 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55e72f8237e0 con 0x55e72fbbbc60 2024-03-11T19:16:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.535+0000 7fc38cff9640 1 -- 172.21.15.29:0/2276672458 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc39c03c040 con 0x55e72fbbbc60 2024-03-11T19:16:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.535+0000 7fc3b2002740 10 client.17085 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.535+0000 7fc3b2002740 10 client.17085 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:16:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.535+0000 7fc3b2002740 20 client.17085 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:54.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.535+0000 7fc3b2002740 1 --2- 172.21.15.29:0/2276672458 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55e72fbd11f0 0x55e72fbd3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.535+0000 7fc3b2002740 4 client.17085 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:16:54.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.535+0000 7fc3b2002740 1 -- 172.21.15.29:0/2276672458 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e72f860720 con 0x55e72fbd11f0 2024-03-11T19:16:54.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.535+0000 7fc3ad1ca640 1 --2- 172.21.15.29:0/2276672458 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55e72fbd11f0 0x55e72fbd3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.535+0000 7fc38cff9640 10 client.17085 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:16:54.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.535+0000 7fc3ad1ca640 1 --2- 172.21.15.29:0/2276672458 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55e72fbd11f0 0x55e72fbd3610 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=244d1097edc4fad3 server_cookie=32692d5e649f2fd3 in_seq=0 out_seq=0 2024-03-11T19:16:54.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.536+0000 7fc38cff9640 1 -- 172.21.15.29:0/2276672458 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e72f860720 con 0x55e72fbd11f0 2024-03-11T19:16:54.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.536+0000 7fc38cff9640 10 client.17085 handle_command_reply: tid=0 2024-03-11T19:16:54.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.536+0000 7fc3b2002740 1 client.17085 shutdown 2024-03-11T19:16:54.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.536+0000 7fc3b2002740 1 -- 172.21.15.29:0/2276672458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbbc60 msgr2=0x55e72fbc5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.536+0000 7fc3b2002740 1 --2- 172.21.15.29:0/2276672458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbbc60 0x55e72fbc5930 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fc39c02f020 tx=0x7fc39c009230 comp rx=0 tx=0).stop 2024-03-11T19:16:54.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.536+0000 7fc3b2002740 1 -- 172.21.15.29:0/2276672458 shutdown_connections 2024-03-11T19:16:54.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.536+0000 7fc3b2002740 1 --2- 172.21.15.29:0/2276672458 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55e72fbd11f0 0x55e72fbd3610 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.536+0000 7fc3b2002740 1 --2- 172.21.15.29:0/2276672458 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbc0dd0 0x55e72fbcc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.536+0000 7fc38cff9640 0 client.17085 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:16:54.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.536+0000 7fc3b2002740 1 --2- 172.21.15.29:0/2276672458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbbc60 0x55e72fbc5930 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.536+0000 7fc3b2002740 1 --2- 172.21.15.29:0/2276672458 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbbb270 0x55e72fbc53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.536+0000 7fc3b2002740 1 -- 172.21.15.29:0/2276672458 >> 172.21.15.29:0/2276672458 conn(0x55e72fbb67e0 msgr2=0x55e72fbb5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.536+0000 7fc3b2002740 1 -- 172.21.15.29:0/2276672458 shutdown_connections 2024-03-11T19:16:54.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.536+0000 7fc3b2002740 1 -- 172.21.15.29:0/2276672458 wait complete. 2024-03-11T19:16:54.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.536+0000 7fc3b2002740 20 client.17085 trim_cache size 0 max 16384 2024-03-11T19:16:54.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.548+0000 7fc3b2002740 1 Processor -- start 2024-03-11T19:16:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.548+0000 7fc3b2002740 1 -- start start 2024-03-11T19:16:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.548+0000 7fc3b2002740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbb270 0x55e72fbc53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.548+0000 7fc3b2002740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbbbc60 0x55e72fbc5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.548+0000 7fc3b2002740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbc0dd0 0x55e72fbbd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.548+0000 7fc3b2002740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e72faa9130 con 0x55e72fbbb270 2024-03-11T19:16:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.548+0000 7fc3b2002740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e72fbb6650 con 0x55e72fbbbc60 2024-03-11T19:16:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.548+0000 7fc3b2002740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e72fbb7550 con 0x55e72fbc0dd0 2024-03-11T19:16:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.548+0000 7fc3ad1ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbb270 0x55e72fbc53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.548+0000 7fc3ad1ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbb270 0x55e72fbc53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49538/0 (socket says 172.21.15.29:49538) 2024-03-11T19:16:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.548+0000 7fc3ad1ca640 1 -- 172.21.15.29:0/1317181615 learned_addr learned my addr 172.21.15.29:0/1317181615 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.548+0000 7fc3ad9cb640 1 --2- 172.21.15.29:0/1317181615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbc0dd0 0x55e72fbbd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.548+0000 7fc3ac9c9640 1 --2- 172.21.15.29:0/1317181615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbbbc60 0x55e72fbc5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.548+0000 7fc3ad1ca640 1 -- 172.21.15.29:0/1317181615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbc0dd0 msgr2=0x55e72fbbd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.548+0000 7fc3ad1ca640 1 --2- 172.21.15.29:0/1317181615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbc0dd0 0x55e72fbbd120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.548+0000 7fc3ad1ca640 1 -- 172.21.15.29:0/1317181615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbbbc60 msgr2=0x55e72fbc5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.548+0000 7fc3ad1ca640 1 --2- 172.21.15.29:0/1317181615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbbbc60 0x55e72fbc5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.548+0000 7fc3ad1ca640 1 -- 172.21.15.29:0/1317181615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e72fabec20 con 0x55e72fbbb270 2024-03-11T19:16:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.548+0000 7fc3ad1ca640 1 --2- 172.21.15.29:0/1317181615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbb270 0x55e72fbc53f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fc3a006c320 tx=0x7fc3a006d400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15dca16d38620a3b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.549+0000 7fc38effd640 1 -- 172.21.15.29:0/1317181615 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a00a28a0 con 0x55e72fbbb270 2024-03-11T19:16:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.549+0000 7fc38effd640 1 -- 172.21.15.29:0/1317181615 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc3a00981a0 con 0x55e72fbbb270 2024-03-11T19:16:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.549+0000 7fc38effd640 1 -- 172.21.15.29:0/1317181615 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0099550 con 0x55e72fbbb270 2024-03-11T19:16:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.549+0000 7fc3b2002740 1 -- 172.21.15.29:0/1317181615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbb270 msgr2=0x55e72fbc53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.549+0000 7fc3b2002740 1 --2- 172.21.15.29:0/1317181615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbb270 0x55e72fbc53f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fc3a006c320 tx=0x7fc3a006d400 comp rx=0 tx=0).stop 2024-03-11T19:16:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.549+0000 7fc3b2002740 1 -- 172.21.15.29:0/1317181615 shutdown_connections 2024-03-11T19:16:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.549+0000 7fc3b2002740 1 --2- 172.21.15.29:0/1317181615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbc0dd0 0x55e72fbbd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.549+0000 7fc3b2002740 1 --2- 172.21.15.29:0/1317181615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbbbc60 0x55e72fbc5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.549+0000 7fc3b2002740 1 --2- 172.21.15.29:0/1317181615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbb270 0x55e72fbc53f0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.549+0000 7fc3b2002740 1 -- 172.21.15.29:0/1317181615 >> 172.21.15.29:0/1317181615 conn(0x55e72fbb67e0 msgr2=0x55e72fbd7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.549+0000 7fc3b2002740 1 -- 172.21.15.29:0/1317181615 shutdown_connections 2024-03-11T19:16:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.549+0000 7fc3b2002740 1 -- 172.21.15.29:0/1317181615 wait complete. 2024-03-11T19:16:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.550+0000 7fc3b2002740 1 -- start start 2024-03-11T19:16:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.550+0000 7fc3b2002740 1 Processor -- start 2024-03-11T19:16:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.550+0000 7fc3b2002740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbbb270 0x55e72fc21fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.550+0000 7fc3b2002740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbbc60 0x55e72fc22500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.550+0000 7fc3b2002740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbc0dd0 0x55e72fc29540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.550+0000 7fc3b2002740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e72fc29a80 con 0x55e72fbbbc60 2024-03-11T19:16:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.550+0000 7fc3b2002740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e72fc29bc0 con 0x55e72fbc0dd0 2024-03-11T19:16:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.550+0000 7fc3b2002740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e72fc29d30 con 0x55e72fbbb270 2024-03-11T19:16:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.550+0000 7fc3b2002740 10 client.0 _ll_register_callbacks cb 0x7fc39c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.550+0000 7fc3ac9c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbbc60 0x55e72fc22500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.550+0000 7fc3ac9c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbbc60 0x55e72fc22500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49554/0 (socket says 172.21.15.29:49554) 2024-03-11T19:16:54.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.550+0000 7fc3ac9c9640 1 -- 172.21.15.29:0/1746811738 learned_addr learned my addr 172.21.15.29:0/1746811738 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:54.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.550+0000 7fc3ad1ca640 1 --2- 172.21.15.29:0/1746811738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbbb270 0x55e72fc21fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.550+0000 7fc3ad9cb640 1 --2- 172.21.15.29:0/1746811738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbc0dd0 0x55e72fc29540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.550+0000 7fc3ac9c9640 1 -- 172.21.15.29:0/1746811738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbbb270 msgr2=0x55e72fc21fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.550+0000 7fc3ac9c9640 1 --2- 172.21.15.29:0/1746811738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbbb270 0x55e72fc21fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.550+0000 7fc3ac9c9640 1 -- 172.21.15.29:0/1746811738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbc0dd0 msgr2=0x55e72fc29540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.550+0000 7fc3ac9c9640 1 --2- 172.21.15.29:0/1746811738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbc0dd0 0x55e72fc29540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.550+0000 7fc3ac9c9640 1 -- 172.21.15.29:0/1746811738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a819e5f0 con 0x55e72fbbbc60 2024-03-11T19:16:54.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.551+0000 7fc3ac9c9640 1 --2- 172.21.15.29:0/1746811738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbbc60 0x55e72fc22500 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fc39c006010 tx=0x7fc39c02cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7868e6907c52eb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.551+0000 7fc38cff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:16:54.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.551+0000 7fc38cff9640 1 -- 172.21.15.29:0/1746811738 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c0348b0 con 0x55e72fbbbc60 2024-03-11T19:16:54.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.551+0000 7fc38cff9640 1 -- 172.21.15.29:0/1746811738 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fc39c000e40 con 0x55e72fbbbc60 2024-03-11T19:16:54.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.551+0000 7fc3b2002740 1 -- 172.21.15.29:0/1746811738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc39c0044e0 con 0x55e72fbbbc60 2024-03-11T19:16:54.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.551+0000 7fc38cff9640 1 -- 172.21.15.29:0/1746811738 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc39c02c2b0 con 0x55e72fbbbc60 2024-03-11T19:16:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.551+0000 7fc3b2002740 10 client.17091 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.551+0000 7fc3b2002740 1 -- 172.21.15.29:0/1746811738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e72fae48c0 con 0x55e72fbbbc60 2024-03-11T19:16:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.551+0000 7fc38cff9640 1 -- 172.21.15.29:0/1746811738 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc39c068020 con 0x55e72fbbbc60 2024-03-11T19:16:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.551+0000 7fc38cff9640 1 -- 172.21.15.29:0/1746811738 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc39c060020 con 0x55e72fbbbc60 2024-03-11T19:16:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.551+0000 7fc3b2002740 10 client.17091 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.551+0000 7fc3b2002740 10 client.17091 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:16:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.551+0000 7fc3b2002740 20 client.17091 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.551+0000 7fc3b2002740 1 --2- 172.21.15.29:0/1746811738 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55e72fc2b360 0x55e72fc2d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.551+0000 7fc3b2002740 4 client.17091 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:16:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.551+0000 7fc3b2002740 1 -- 172.21.15.29:0/1746811738 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e72f8af890 con 0x55e72fc2b360 2024-03-11T19:16:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.551+0000 7fc3ad1ca640 1 --2- 172.21.15.29:0/1746811738 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55e72fc2b360 0x55e72fc2d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.552+0000 7fc38cff9640 10 client.17091 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:16:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.552+0000 7fc3ad1ca640 1 --2- 172.21.15.29:0/1746811738 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55e72fc2b360 0x55e72fc2d780 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5cd26404fe00de41 server_cookie=30fb43a13c3de6ba in_seq=0 out_seq=0 2024-03-11T19:16:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.552+0000 7fc38cff9640 1 -- 172.21.15.29:0/1746811738 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e72f8af890 con 0x55e72fc2b360 2024-03-11T19:16:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.552+0000 7fc38cff9640 10 client.17091 handle_command_reply: tid=0 2024-03-11T19:16:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.552+0000 7fc3b2002740 1 client.17091 shutdown 2024-03-11T19:16:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.552+0000 7fc3b2002740 1 -- 172.21.15.29:0/1746811738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbbc60 msgr2=0x55e72fc22500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.552+0000 7fc3b2002740 1 --2- 172.21.15.29:0/1746811738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbbc60 0x55e72fc22500 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fc39c006010 tx=0x7fc39c02cb30 comp rx=0 tx=0).stop 2024-03-11T19:16:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.552+0000 7fc3b2002740 1 -- 172.21.15.29:0/1746811738 shutdown_connections 2024-03-11T19:16:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.552+0000 7fc3b2002740 1 --2- 172.21.15.29:0/1746811738 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55e72fc2b360 0x55e72fc2d780 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.552+0000 7fc3b2002740 1 --2- 172.21.15.29:0/1746811738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e72fbc0dd0 0x55e72fc29540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.552+0000 7fc3b2002740 1 --2- 172.21.15.29:0/1746811738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e72fbbbc60 0x55e72fc22500 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.552+0000 7fc38cff9640 0 client.17091 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:16:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.552+0000 7fc3b2002740 1 --2- 172.21.15.29:0/1746811738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e72fbbb270 0x55e72fc21fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.552+0000 7fc3b2002740 1 -- 172.21.15.29:0/1746811738 >> 172.21.15.29:0/1746811738 conn(0x55e72fbb67e0 msgr2=0x55e72fbb9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.552+0000 7fc3b2002740 1 -- 172.21.15.29:0/1746811738 shutdown_connections 2024-03-11T19:16:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.552+0000 7fc3b2002740 1 -- 172.21.15.29:0/1746811738 wait complete. 2024-03-11T19:16:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.552+0000 7fc3b2002740 20 client.17091 trim_cache size 0 max 16384 2024-03-11T19:16:54.563 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:16:54.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.553+0000 7fc3af42e640 1 -- 172.21.15.29:0/687101751 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc3900a1a30 msgr2=0x7fc3900a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.553+0000 7fc3af42e640 1 --2- 172.21.15.29:0/687101751 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc3900a1a30 0x7fc3900a3ef0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x55e72fbbc4c0 tx=0x7fc398007410 comp rx=0 tx=0).stop 2024-03-11T19:16:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.553+0000 7fc3af42e640 1 -- 172.21.15.29:0/687101751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a8100080 msgr2=0x7fc3a8195fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.553+0000 7fc3af42e640 1 --2- 172.21.15.29:0/687101751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a8100080 0x7fc3a8195fe0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fc3a000c970 tx=0x7fc3a000ce30 comp rx=0 tx=0).stop 2024-03-11T19:16:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.553+0000 7fc3af42e640 1 -- 172.21.15.29:0/687101751 shutdown_connections 2024-03-11T19:16:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.553+0000 7fc3af42e640 1 --2- 172.21.15.29:0/687101751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3a8101c30 0x7fc3a819d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.553+0000 7fc3af42e640 1 --2- 172.21.15.29:0/687101751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3a8101270 0x7fc3a8196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.553+0000 7fc3af42e640 1 --2- 172.21.15.29:0/687101751 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc3900a1a30 0x7fc3900a3ef0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.553+0000 7fc3af42e640 1 --2- 172.21.15.29:0/687101751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a8100080 0x7fc3a8195fe0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.553+0000 7fc3af42e640 1 -- 172.21.15.29:0/687101751 >> 172.21.15.29:0/687101751 conn(0x7fc3a80fb610 msgr2=0x7fc3a8104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.553+0000 7fc3af42e640 1 -- 172.21.15.29:0/687101751 shutdown_connections 2024-03-11T19:16:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.553+0000 7fc3af42e640 1 -- 172.21.15.29:0/687101751 wait complete. 2024-03-11T19:16:54.567 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:16:54.567 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:16:54.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.679+0000 7fb7320d4640 1 -- 172.21.15.29:0/925806503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb72c100080 msgr2=0x7fb72c1004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.679+0000 7fb7320d4640 1 --2- 172.21.15.29:0/925806503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb72c100080 0x7fb72c1004a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb71c009860 tx=0x7fb71c02bb70 comp rx=0 tx=0).stop 2024-03-11T19:16:54.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.679+0000 7fb7320d4640 1 -- 172.21.15.29:0/925806503 shutdown_connections 2024-03-11T19:16:54.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.679+0000 7fb7320d4640 1 --2- 172.21.15.29:0/925806503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb72c101c30 0x7fb72c1084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.679+0000 7fb7320d4640 1 --2- 172.21.15.29:0/925806503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb72c101270 0x7fb72c1016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.679+0000 7fb7320d4640 1 --2- 172.21.15.29:0/925806503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb72c100080 0x7fb72c1004a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.679+0000 7fb7320d4640 1 -- 172.21.15.29:0/925806503 >> 172.21.15.29:0/925806503 conn(0x7fb72c0fb610 msgr2=0x7fb72c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.679+0000 7fb7320d4640 1 -- 172.21.15.29:0/925806503 shutdown_connections 2024-03-11T19:16:54.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.679+0000 7fb7320d4640 1 -- 172.21.15.29:0/925806503 wait complete. 2024-03-11T19:16:54.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.680+0000 7fb7320d4640 1 Processor -- start 2024-03-11T19:16:54.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.680+0000 7fb7320d4640 1 -- start start 2024-03-11T19:16:54.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.680+0000 7fb7320d4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb72c100080 0x7fb72c196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.680+0000 7fb7320d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb72c101270 0x7fb72c1965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.680+0000 7fb7320d4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb72c101c30 0x7fb72c19d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.680+0000 7fb7320d4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c19db80 con 0x7fb72c101270 2024-03-11T19:16:54.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.680+0000 7fb7320d4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c19dcf0 con 0x7fb72c100080 2024-03-11T19:16:54.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.680+0000 7fb7320d4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c19deb0 con 0x7fb72c101c30 2024-03-11T19:16:54.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.680+0000 7fb72affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb72c101270 0x7fb72c1965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.680+0000 7fb72affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb72c101270 0x7fb72c1965d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49562/0 (socket says 172.21.15.29:49562) 2024-03-11T19:16:54.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.680+0000 7fb72affd640 1 -- 172.21.15.29:0/979123835 learned_addr learned my addr 172.21.15.29:0/979123835 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:54.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.680+0000 7fb72b7fe640 1 --2- 172.21.15.29:0/979123835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb72c100080 0x7fb72c196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.680+0000 7fb72bfff640 1 --2- 172.21.15.29:0/979123835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb72c101c30 0x7fb72c19d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.681+0000 7fb72affd640 1 -- 172.21.15.29:0/979123835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb72c101c30 msgr2=0x7fb72c19d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.681+0000 7fb72affd640 1 --2- 172.21.15.29:0/979123835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb72c101c30 0x7fb72c19d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.681+0000 7fb72affd640 1 -- 172.21.15.29:0/979123835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb72c100080 msgr2=0x7fb72c196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.681+0000 7fb72affd640 1 --2- 172.21.15.29:0/979123835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb72c100080 0x7fb72c196090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.681+0000 7fb72affd640 1 -- 172.21.15.29:0/979123835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71c009490 con 0x7fb72c101270 2024-03-11T19:16:54.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.681+0000 7fb72affd640 1 --2- 172.21.15.29:0/979123835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb72c101270 0x7fb72c1965d0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fb72000d950 tx=0x7fb72000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.681+0000 7fb728ff9640 1 -- 172.21.15.29:0/979123835 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb720009c30 con 0x7fb72c101270 2024-03-11T19:16:54.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.681+0000 7fb7320d4640 1 -- 172.21.15.29:0/979123835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72c19e190 con 0x7fb72c101270 2024-03-11T19:16:54.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.681+0000 7fb728ff9640 1 -- 172.21.15.29:0/979123835 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb720009d90 con 0x7fb72c101270 2024-03-11T19:16:54.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.681+0000 7fb728ff9640 1 -- 172.21.15.29:0/979123835 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72000f6b0 con 0x7fb72c101270 2024-03-11T19:16:54.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.681+0000 7fb7320d4640 1 -- 172.21.15.29:0/979123835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72c19e6a0 con 0x7fb72c101270 2024-03-11T19:16:54.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.684+0000 7fb728ff9640 1 -- 172.21.15.29:0/979123835 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb720010aa0 con 0x7fb72c101270 2024-03-11T19:16:54.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.684+0000 7fb734d84740 1 Processor -- start 2024-03-11T19:16:54.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.684+0000 7fb734d84740 1 -- start start 2024-03-11T19:16:54.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.684+0000 7fb734d84740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f4c30 0x56216e5f3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.684+0000 7fb734d84740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e5f38c0 0x56216e5f3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.684+0000 7fb734d84740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e602550 0x56216e604940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.684+0000 7fb734d84740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216e5b4960 con 0x56216e5f38c0 2024-03-11T19:16:54.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.684+0000 7fb734d84740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216e568310 con 0x56216e602550 2024-03-11T19:16:54.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.684+0000 7fb734d84740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216e565840 con 0x56216e5f4c30 2024-03-11T19:16:54.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.684+0000 7fb72b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e5f38c0 0x56216e5f3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.684+0000 7fb72b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e5f38c0 0x56216e5f3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49576/0 (socket says 172.21.15.29:49576) 2024-03-11T19:16:54.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.684+0000 7fb72b7fe640 1 -- 172.21.15.29:0/2038161515 learned_addr learned my addr 172.21.15.29:0/2038161515 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:54.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.684+0000 7fb72affd640 1 --2- 172.21.15.29:0/2038161515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e602550 0x56216e604940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.684+0000 7fb728ff9640 1 --2- 172.21.15.29:0/979123835 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7140a1a30 0x7fb7140a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.684+0000 7fb72bfff640 1 --2- 172.21.15.29:0/2038161515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f4c30 0x56216e5f3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.685+0000 7fb72bfff640 1 --2- 172.21.15.29:0/979123835 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7140a1a30 0x7fb7140a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.685+0000 7fb72b7fe640 1 -- 172.21.15.29:0/2038161515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f4c30 msgr2=0x56216e5f3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.685+0000 7fb72b7fe640 1 --2- 172.21.15.29:0/2038161515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f4c30 0x56216e5f3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.685+0000 7fb72b7fe640 1 -- 172.21.15.29:0/2038161515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e602550 msgr2=0x56216e604940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.685+0000 7fb72b7fe640 1 --2- 172.21.15.29:0/2038161515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e602550 0x56216e604940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.685+0000 7fb72b7fe640 1 -- 172.21.15.29:0/2038161515 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56216e523060 con 0x56216e5f38c0 2024-03-11T19:16:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.685+0000 7fb72bfff640 1 --2- 172.21.15.29:0/2038161515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f4c30 0x56216e5f3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:16:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.685+0000 7fb728ff9640 1 -- 172.21.15.29:0/979123835 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb7200ccfd0 con 0x7fb72c101270 2024-03-11T19:16:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.685+0000 7fb72b7fe640 1 --2- 172.21.15.29:0/2038161515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e5f38c0 0x56216e5f3d40 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb71c0057c0 tx=0x7fb71c005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70bac406284d779c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.685+0000 7fb72bfff640 1 --2- 172.21.15.29:0/979123835 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7140a1a30 0x7fb7140a3ef0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x56216e5f47c0 tx=0x7fb7240095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.685+0000 7fb711ffb640 1 -- 172.21.15.29:0/2038161515 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71c034db0 con 0x56216e5f38c0 2024-03-11T19:16:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.685+0000 7fb711ffb640 1 -- 172.21.15.29:0/2038161515 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb71c02ce90 con 0x56216e5f38c0 2024-03-11T19:16:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.685+0000 7fb711ffb640 1 -- 172.21.15.29:0/2038161515 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71c02c750 con 0x56216e5f38c0 2024-03-11T19:16:54.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.686+0000 7fb734d84740 1 -- 172.21.15.29:0/2038161515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e5f38c0 msgr2=0x56216e5f3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.686+0000 7fb734d84740 1 --2- 172.21.15.29:0/2038161515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e5f38c0 0x56216e5f3d40 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb71c0057c0 tx=0x7fb71c005b20 comp rx=0 tx=0).stop 2024-03-11T19:16:54.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.686+0000 7fb734d84740 1 -- 172.21.15.29:0/2038161515 shutdown_connections 2024-03-11T19:16:54.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.686+0000 7fb734d84740 1 --2- 172.21.15.29:0/2038161515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e602550 0x56216e604940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.686+0000 7fb734d84740 1 --2- 172.21.15.29:0/2038161515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e5f38c0 0x56216e5f3d40 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.686+0000 7fb734d84740 1 --2- 172.21.15.29:0/2038161515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f4c30 0x56216e5f3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.686+0000 7fb734d84740 1 -- 172.21.15.29:0/2038161515 >> 172.21.15.29:0/2038161515 conn(0x56216e5f59f0 msgr2=0x56216e5f5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.686+0000 7fb734d84740 1 -- 172.21.15.29:0/2038161515 shutdown_connections 2024-03-11T19:16:54.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.686+0000 7fb734d84740 1 -- 172.21.15.29:0/2038161515 wait complete. 2024-03-11T19:16:54.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.686+0000 7fb734d84740 1 -- start start 2024-03-11T19:16:54.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.686+0000 7fb734d84740 1 Processor -- start 2024-03-11T19:16:54.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb734d84740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f38c0 0x56216e606730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb734d84740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e5f4c30 0x56216e606c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb734d84740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e602550 0x56216e60dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb734d84740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216e561020 con 0x56216e602550 2024-03-11T19:16:54.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb734d84740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216e234430 con 0x56216e5f4c30 2024-03-11T19:16:54.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb734d84740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216e1f0e00 con 0x56216e5f38c0 2024-03-11T19:16:54.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb734d84740 10 client.0 _ll_register_callbacks cb 0x56216e508cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:54.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb72bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e602550 0x56216e60dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb72bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e602550 0x56216e60dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49592/0 (socket says 172.21.15.29:49592) 2024-03-11T19:16:54.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb72b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f38c0 0x56216e606730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb72bfff640 1 -- 172.21.15.29:0/4161511484 learned_addr learned my addr 172.21.15.29:0/4161511484 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:54.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb72affd640 1 --2- 172.21.15.29:0/4161511484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e5f4c30 0x56216e606c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb72bfff640 1 -- 172.21.15.29:0/4161511484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f38c0 msgr2=0x56216e606730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb72bfff640 1 --2- 172.21.15.29:0/4161511484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f38c0 0x56216e606730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb72bfff640 1 -- 172.21.15.29:0/4161511484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e5f4c30 msgr2=0x56216e606c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb72bfff640 1 --2- 172.21.15.29:0/4161511484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e5f4c30 0x56216e606c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb72bfff640 1 -- 172.21.15.29:0/4161511484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72c108dc0 con 0x56216e602550 2024-03-11T19:16:54.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb72b7fe640 1 --2- 172.21.15.29:0/4161511484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f38c0 0x56216e606730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:16:54.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb72bfff640 1 --2- 172.21.15.29:0/4161511484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e602550 0x56216e60dcb0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb72400cc10 tx=0x7fb724004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8f0c2c32945a54 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb70affd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:16:54.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb70affd640 1 -- 172.21.15.29:0/4161511484 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724013070 con 0x56216e602550 2024-03-11T19:16:54.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb70affd640 1 -- 172.21.15.29:0/4161511484 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb72401bd60 con 0x56216e602550 2024-03-11T19:16:54.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.687+0000 7fb734d84740 1 -- 172.21.15.29:0/4161511484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56216e2607e0 con 0x56216e602550 2024-03-11T19:16:54.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.688+0000 7fb70affd640 1 -- 172.21.15.29:0/4161511484 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb724004b20 con 0x56216e602550 2024-03-11T19:16:54.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.688+0000 7fb734d84740 10 client.17103 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:54.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.688+0000 7fb734d84740 1 -- 172.21.15.29:0/4161511484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56216e55f6a0 con 0x56216e602550 2024-03-11T19:16:54.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.688+0000 7fb70affd640 1 -- 172.21.15.29:0/4161511484 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x56216e2607e0 con 0x56216e602550 2024-03-11T19:16:54.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.688+0000 7fb70affd640 1 -- 172.21.15.29:0/4161511484 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb724026400 con 0x56216e602550 2024-03-11T19:16:54.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.688+0000 7fb734d84740 10 client.17103 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:54.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.688+0000 7fb734d84740 10 client.17103 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:16:54.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.688+0000 7fb734d84740 20 client.17103 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:54.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.688+0000 7fb734d84740 1 --2- 172.21.15.29:0/4161511484 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x56216e612a70 0x56216e614e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.688+0000 7fb734d84740 4 client.17103 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:16:54.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.688+0000 7fb734d84740 1 -- 172.21.15.29:0/4161511484 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56216e29d720 con 0x56216e612a70 2024-03-11T19:16:54.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.688+0000 7fb72b7fe640 1 --2- 172.21.15.29:0/4161511484 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x56216e612a70 0x56216e614e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.689+0000 7fb70affd640 10 client.17103 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:16:54.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.689+0000 7fb72b7fe640 1 --2- 172.21.15.29:0/4161511484 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x56216e612a70 0x56216e614e90 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3bba47f396ab6919 server_cookie=fb3178cc6d216469 in_seq=0 out_seq=0 2024-03-11T19:16:54.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.689+0000 7fb70affd640 1 -- 172.21.15.29:0/4161511484 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56216e29d720 con 0x56216e612a70 2024-03-11T19:16:54.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.689+0000 7fb70affd640 10 client.17103 handle_command_reply: tid=0 2024-03-11T19:16:54.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.689+0000 7fb734d84740 1 client.17103 shutdown 2024-03-11T19:16:54.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.689+0000 7fb734d84740 1 -- 172.21.15.29:0/4161511484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e602550 msgr2=0x56216e60dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.689+0000 7fb734d84740 1 --2- 172.21.15.29:0/4161511484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e602550 0x56216e60dcb0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb72400cc10 tx=0x7fb724004a40 comp rx=0 tx=0).stop 2024-03-11T19:16:54.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.689+0000 7fb734d84740 1 -- 172.21.15.29:0/4161511484 shutdown_connections 2024-03-11T19:16:54.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.689+0000 7fb734d84740 1 --2- 172.21.15.29:0/4161511484 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x56216e612a70 0x56216e614e90 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.690+0000 7fb734d84740 1 --2- 172.21.15.29:0/4161511484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e602550 0x56216e60dcb0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.690+0000 7fb734d84740 1 --2- 172.21.15.29:0/4161511484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e5f4c30 0x56216e606c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.690+0000 7fb734d84740 1 --2- 172.21.15.29:0/4161511484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f38c0 0x56216e606730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.690+0000 7fb734d84740 1 -- 172.21.15.29:0/4161511484 >> 172.21.15.29:0/4161511484 conn(0x56216e5f59f0 msgr2=0x56216e5f2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.690+0000 7fb70affd640 0 client.17103 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:16:54.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.690+0000 7fb734d84740 1 -- 172.21.15.29:0/4161511484 shutdown_connections 2024-03-11T19:16:54.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.690+0000 7fb734d84740 1 -- 172.21.15.29:0/4161511484 wait complete. 2024-03-11T19:16:54.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.690+0000 7fb734d84740 20 client.17103 trim_cache size 0 max 16384 2024-03-11T19:16:54.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.701+0000 7fb734d84740 1 Processor -- start 2024-03-11T19:16:54.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.701+0000 7fb734d84740 1 -- start start 2024-03-11T19:16:54.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.701+0000 7fb734d84740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f38c0 0x56216e606710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.702+0000 7fb734d84740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e5f4c30 0x56216e5fe140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.702+0000 7fb734d84740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e602550 0x56216e606c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.702+0000 7fb734d84740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216e4e6130 con 0x56216e5f4c30 2024-03-11T19:16:54.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.702+0000 7fb734d84740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216e606570 con 0x56216e602550 2024-03-11T19:16:54.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.702+0000 7fb734d84740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216e2607e0 con 0x56216e5f38c0 2024-03-11T19:16:54.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.702+0000 7fb72affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e5f4c30 0x56216e5fe140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.702+0000 7fb72affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e5f4c30 0x56216e5fe140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49606/0 (socket says 172.21.15.29:49606) 2024-03-11T19:16:54.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.702+0000 7fb72affd640 1 -- 172.21.15.29:0/852365487 learned_addr learned my addr 172.21.15.29:0/852365487 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:54.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.702+0000 7fb72bfff640 1 --2- 172.21.15.29:0/852365487 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e602550 0x56216e606c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.702+0000 7fb72b7fe640 1 --2- 172.21.15.29:0/852365487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f38c0 0x56216e606710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.702+0000 7fb72affd640 1 -- 172.21.15.29:0/852365487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f38c0 msgr2=0x56216e606710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.702+0000 7fb72affd640 1 --2- 172.21.15.29:0/852365487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f38c0 0x56216e606710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.702+0000 7fb72affd640 1 -- 172.21.15.29:0/852365487 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e602550 msgr2=0x56216e606c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.702+0000 7fb72affd640 1 --2- 172.21.15.29:0/852365487 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e602550 0x56216e606c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.702+0000 7fb72affd640 1 -- 172.21.15.29:0/852365487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56216e4fbc20 con 0x56216e5f4c30 2024-03-11T19:16:54.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.702+0000 7fb72affd640 1 --2- 172.21.15.29:0/852365487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e5f4c30 0x56216e5fe140 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fb7200058c0 tx=0x7fb7200d2040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1933e44056cd57a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.702+0000 7fb7117fa640 1 -- 172.21.15.29:0/852365487 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72009e130 con 0x56216e5f4c30 2024-03-11T19:16:54.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.702+0000 7fb7117fa640 1 -- 172.21.15.29:0/852365487 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb7200a2780 con 0x56216e5f4c30 2024-03-11T19:16:54.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.702+0000 7fb7117fa640 1 -- 172.21.15.29:0/852365487 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb720096890 con 0x56216e5f4c30 2024-03-11T19:16:54.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.703+0000 7fb734d84740 1 -- 172.21.15.29:0/852365487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e5f4c30 msgr2=0x56216e5fe140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.703+0000 7fb734d84740 1 --2- 172.21.15.29:0/852365487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e5f4c30 0x56216e5fe140 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fb7200058c0 tx=0x7fb7200d2040 comp rx=0 tx=0).stop 2024-03-11T19:16:54.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.703+0000 7fb734d84740 1 -- 172.21.15.29:0/852365487 shutdown_connections 2024-03-11T19:16:54.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.703+0000 7fb734d84740 1 --2- 172.21.15.29:0/852365487 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e602550 0x56216e606c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.703+0000 7fb734d84740 1 --2- 172.21.15.29:0/852365487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e5f4c30 0x56216e5fe140 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.703+0000 7fb734d84740 1 --2- 172.21.15.29:0/852365487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f38c0 0x56216e606710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.703+0000 7fb734d84740 1 -- 172.21.15.29:0/852365487 >> 172.21.15.29:0/852365487 conn(0x56216e5f59f0 msgr2=0x56216e5f5dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.703+0000 7fb734d84740 1 -- 172.21.15.29:0/852365487 shutdown_connections 2024-03-11T19:16:54.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.703+0000 7fb734d84740 1 -- 172.21.15.29:0/852365487 wait complete. 2024-03-11T19:16:54.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.703+0000 7fb734d84740 1 -- start start 2024-03-11T19:16:54.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.703+0000 7fb734d84740 1 Processor -- start 2024-03-11T19:16:54.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.703+0000 7fb734d84740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e5f38c0 0x56216e607e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.703+0000 7fb734d84740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f4c30 0x56216e608350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.703+0000 7fb734d84740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e602550 0x56216e60d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.703+0000 7fb734d84740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216e60d870 con 0x56216e602550 2024-03-11T19:16:54.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.703+0000 7fb734d84740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216e60d9e0 con 0x56216e5f38c0 2024-03-11T19:16:54.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.703+0000 7fb734d84740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216e60dba0 con 0x56216e5f4c30 2024-03-11T19:16:54.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.703+0000 7fb734d84740 10 client.0 _ll_register_callbacks cb 0x7fb724009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:54.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.704+0000 7fb72affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f4c30 0x56216e608350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.704+0000 7fb72affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f4c30 0x56216e608350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:54214/0 (socket says 172.21.15.29:54214) 2024-03-11T19:16:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.704+0000 7fb72affd640 1 -- 172.21.15.29:0/1930436349 learned_addr learned my addr 172.21.15.29:0/1930436349 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.704+0000 7fb72b7fe640 1 --2- 172.21.15.29:0/1930436349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e5f38c0 0x56216e607e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.704+0000 7fb72affd640 1 -- 172.21.15.29:0/1930436349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e5f38c0 msgr2=0x56216e607e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.704+0000 7fb72affd640 1 --2- 172.21.15.29:0/1930436349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e5f38c0 0x56216e607e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.704+0000 7fb72affd640 1 -- 172.21.15.29:0/1930436349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e602550 msgr2=0x56216e60d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.704+0000 7fb72affd640 1 --2- 172.21.15.29:0/1930436349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e602550 0x56216e60d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.704+0000 7fb72affd640 1 -- 172.21.15.29:0/1930436349 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72c19e6a0 con 0x56216e5f4c30 2024-03-11T19:16:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.704+0000 7fb72affd640 1 --2- 172.21.15.29:0/1930436349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f4c30 0x56216e608350 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb7200985e0 tx=0x7fb720005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bf46033a1e0d68c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.704+0000 7fb70affd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:16:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.704+0000 7fb70affd640 1 -- 172.21.15.29:0/1930436349 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7200996e0 con 0x56216e5f4c30 2024-03-11T19:16:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.704+0000 7fb70affd640 1 -- 172.21.15.29:0/1930436349 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb720098390 con 0x56216e5f4c30 2024-03-11T19:16:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.704+0000 7fb734d84740 1 -- 172.21.15.29:0/1930436349 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb71c02ce90 con 0x56216e5f4c30 2024-03-11T19:16:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.704+0000 7fb70affd640 1 -- 172.21.15.29:0/1930436349 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb720099840 con 0x56216e5f4c30 2024-03-11T19:16:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.705+0000 7fb734d84740 10 client.26626 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:54.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.705+0000 7fb734d84740 1 -- 172.21.15.29:0/1930436349 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56216e5218c0 con 0x56216e5f4c30 2024-03-11T19:16:54.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.705+0000 7fb70affd640 1 -- 172.21.15.29:0/1930436349 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb71c02ce90 con 0x56216e5f4c30 2024-03-11T19:16:54.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.705+0000 7fb70affd640 1 -- 172.21.15.29:0/1930436349 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb72007d070 con 0x56216e5f4c30 2024-03-11T19:16:54.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.705+0000 7fb734d84740 10 client.26626 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:54.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.705+0000 7fb734d84740 10 client.26626 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:16:54.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.705+0000 7fb734d84740 20 client.26626 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:54.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.705+0000 7fb734d84740 1 --2- 172.21.15.29:0/1930436349 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x56216e66e6f0 0x56216e670b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.705+0000 7fb734d84740 4 client.26626 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:16:54.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.705+0000 7fb734d84740 1 -- 172.21.15.29:0/1930436349 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56216e2ec890 con 0x56216e66e6f0 2024-03-11T19:16:54.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.705+0000 7fb72b7fe640 1 --2- 172.21.15.29:0/1930436349 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x56216e66e6f0 0x56216e670b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.706+0000 7fb70affd640 10 client.26626 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:16:54.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.706+0000 7fb72b7fe640 1 --2- 172.21.15.29:0/1930436349 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x56216e66e6f0 0x56216e670b10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a4100a5821f0baea server_cookie=c6081089dece7fdd in_seq=0 out_seq=0 2024-03-11T19:16:54.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.706+0000 7fb70affd640 1 -- 172.21.15.29:0/1930436349 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56216e2ec890 con 0x56216e66e6f0 2024-03-11T19:16:54.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.706+0000 7fb70affd640 10 client.26626 handle_command_reply: tid=0 2024-03-11T19:16:54.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.706+0000 7fb734d84740 1 client.26626 shutdown 2024-03-11T19:16:54.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.706+0000 7fb734d84740 1 -- 172.21.15.29:0/1930436349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f4c30 msgr2=0x56216e608350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.706+0000 7fb734d84740 1 --2- 172.21.15.29:0/1930436349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f4c30 0x56216e608350 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb7200985e0 tx=0x7fb720005f90 comp rx=0 tx=0).stop 2024-03-11T19:16:54.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.706+0000 7fb734d84740 1 -- 172.21.15.29:0/1930436349 shutdown_connections 2024-03-11T19:16:54.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.706+0000 7fb734d84740 1 --2- 172.21.15.29:0/1930436349 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x56216e66e6f0 0x56216e670b10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.706+0000 7fb734d84740 1 --2- 172.21.15.29:0/1930436349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216e602550 0x56216e60d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.706+0000 7fb734d84740 1 --2- 172.21.15.29:0/1930436349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56216e5f4c30 0x56216e608350 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.706+0000 7fb70affd640 0 client.26626 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:16:54.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.706+0000 7fb734d84740 1 --2- 172.21.15.29:0/1930436349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56216e5f38c0 0x56216e607e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.706+0000 7fb734d84740 1 -- 172.21.15.29:0/1930436349 >> 172.21.15.29:0/1930436349 conn(0x56216e5f59f0 msgr2=0x56216e6066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.706+0000 7fb734d84740 1 -- 172.21.15.29:0/1930436349 shutdown_connections 2024-03-11T19:16:54.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.706+0000 7fb734d84740 1 -- 172.21.15.29:0/1930436349 wait complete. 2024-03-11T19:16:54.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.706+0000 7fb734d84740 20 client.26626 trim_cache size 0 max 16384 2024-03-11T19:16:54.715 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:16:54.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.707+0000 7fb7320d4640 1 -- 172.21.15.29:0/979123835 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7140a1a30 msgr2=0x7fb7140a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.707+0000 7fb7320d4640 1 --2- 172.21.15.29:0/979123835 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7140a1a30 0x7fb7140a3ef0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x56216e5f47c0 tx=0x7fb7240095c0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.707+0000 7fb7320d4640 1 -- 172.21.15.29:0/979123835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb72c101270 msgr2=0x7fb72c1965d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.707+0000 7fb7320d4640 1 --2- 172.21.15.29:0/979123835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb72c101270 0x7fb72c1965d0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fb72000d950 tx=0x7fb72000de10 comp rx=0 tx=0).stop 2024-03-11T19:16:54.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.707+0000 7fb7320d4640 1 -- 172.21.15.29:0/979123835 shutdown_connections 2024-03-11T19:16:54.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.707+0000 7fb7320d4640 1 --2- 172.21.15.29:0/979123835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb72c101c30 0x7fb72c19d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.707+0000 7fb7320d4640 1 --2- 172.21.15.29:0/979123835 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7140a1a30 0x7fb7140a3ef0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.707+0000 7fb7320d4640 1 --2- 172.21.15.29:0/979123835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb72c101270 0x7fb72c1965d0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.707+0000 7fb7320d4640 1 --2- 172.21.15.29:0/979123835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb72c100080 0x7fb72c196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.707+0000 7fb7320d4640 1 -- 172.21.15.29:0/979123835 >> 172.21.15.29:0/979123835 conn(0x7fb72c0fb610 msgr2=0x7fb72c104680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.707+0000 7fb7320d4640 1 -- 172.21.15.29:0/979123835 shutdown_connections 2024-03-11T19:16:54.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.707+0000 7fb7320d4640 1 -- 172.21.15.29:0/979123835 wait complete. 2024-03-11T19:16:54.725 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:16:54.725 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:16:54.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.894+0000 7faae61b6640 1 -- 172.21.15.29:0/2783306257 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faae0101c30 msgr2=0x7faae00682e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.894+0000 7faae61b6640 1 --2- 172.21.15.29:0/2783306257 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faae0101c30 0x7faae00682e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7faad4009900 tx=0x7faad402bc40 comp rx=0 tx=0).stop 2024-03-11T19:16:54.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.894+0000 7faae61b6640 1 -- 172.21.15.29:0/2783306257 shutdown_connections 2024-03-11T19:16:54.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.894+0000 7faae61b6640 1 --2- 172.21.15.29:0/2783306257 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faae0101c30 0x7faae00682e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.894+0000 7faae61b6640 1 --2- 172.21.15.29:0/2783306257 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faae0101270 0x7faae01016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.894+0000 7faae61b6640 1 --2- 172.21.15.29:0/2783306257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faae0100080 0x7faae01004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.894+0000 7faae61b6640 1 -- 172.21.15.29:0/2783306257 >> 172.21.15.29:0/2783306257 conn(0x7faae00fb610 msgr2=0x7faae00fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.895+0000 7faae61b6640 1 -- 172.21.15.29:0/2783306257 shutdown_connections 2024-03-11T19:16:54.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.895+0000 7faae61b6640 1 -- 172.21.15.29:0/2783306257 wait complete. 2024-03-11T19:16:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.895+0000 7faae61b6640 1 Processor -- start 2024-03-11T19:16:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.896+0000 7faae61b6640 1 -- start start 2024-03-11T19:16:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.896+0000 7faae61b6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faae0100080 0x7faae01993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.896+0000 7faae61b6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faae0101270 0x7faae0199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.896+0000 7faae61b6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faae0101c30 0x7faae01a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.896+0000 7faae61b6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae01a0eb0 con 0x7faae0100080 2024-03-11T19:16:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.896+0000 7faae61b6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae01a1020 con 0x7faae0101c30 2024-03-11T19:16:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.896+0000 7faae61b6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae01a11e0 con 0x7faae0101270 2024-03-11T19:16:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.896+0000 7faadeffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faae0101270 0x7faae0199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.896+0000 7faadffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faae0101c30 0x7faae01a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.896+0000 7faadeffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faae0101270 0x7faae0199900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:54232/0 (socket says 172.21.15.29:54232) 2024-03-11T19:16:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.896+0000 7faadffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faae0101c30 0x7faae01a0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59084/0 (socket says 172.21.15.29:59084) 2024-03-11T19:16:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.896+0000 7faadeffd640 1 -- 172.21.15.29:0/1142211635 learned_addr learned my addr 172.21.15.29:0/1142211635 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.896+0000 7faadf7fe640 1 --2- 172.21.15.29:0/1142211635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faae0100080 0x7faae01993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.896+0000 7faadeffd640 1 -- 172.21.15.29:0/1142211635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faae0101c30 msgr2=0x7faae01a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.896+0000 7faadeffd640 1 --2- 172.21.15.29:0/1142211635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faae0101c30 0x7faae01a0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.896+0000 7faadeffd640 1 -- 172.21.15.29:0/1142211635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faae0100080 msgr2=0x7faae01993c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.896+0000 7faadeffd640 1 --2- 172.21.15.29:0/1142211635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faae0100080 0x7faae01993c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.896+0000 7faadeffd640 1 -- 172.21.15.29:0/1142211635 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad4009560 con 0x7faae0101270 2024-03-11T19:16:54.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.897+0000 7faadeffd640 1 --2- 172.21.15.29:0/1142211635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faae0101270 0x7faae0199900 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faad000c890 tx=0x7faad000cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.897+0000 7faadcff9640 1 -- 172.21.15.29:0/1142211635 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad0004220 con 0x7faae0101270 2024-03-11T19:16:54.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.898+0000 7faadcff9640 1 -- 172.21.15.29:0/1142211635 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faad0007e60 con 0x7faae0101270 2024-03-11T19:16:54.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.899+0000 7faae61b6640 1 -- 172.21.15.29:0/1142211635 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae01a14c0 con 0x7faae0101270 2024-03-11T19:16:54.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.899+0000 7faae61b6640 1 -- 172.21.15.29:0/1142211635 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae01a1a50 con 0x7faae0101270 2024-03-11T19:16:54.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.899+0000 7faadcff9640 1 -- 172.21.15.29:0/1142211635 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad0004a00 con 0x7faae0101270 2024-03-11T19:16:54.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.902+0000 7faadcff9640 1 -- 172.21.15.29:0/1142211635 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7faad00079b0 con 0x7faae0101270 2024-03-11T19:16:54.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.903+0000 7faae8d84740 1 Processor -- start 2024-03-11T19:16:54.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.903+0000 7faae8d84740 1 -- start start 2024-03-11T19:16:54.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.903+0000 7faae8d84740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23660270 0x55cd23660690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.903+0000 7faae8d84740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd23660c60 0x55cd23665760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.903+0000 7faae8d84740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23665dd0 0x55cd23668260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.903+0000 7faae8d84740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd2361c960 con 0x55cd23665dd0 2024-03-11T19:16:54.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.903+0000 7faae8d84740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd235d0310 con 0x55cd23660270 2024-03-11T19:16:54.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.903+0000 7faae8d84740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd235cd840 con 0x55cd23660c60 2024-03-11T19:16:54.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.903+0000 7faadeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23665dd0 0x55cd23668260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.903+0000 7faadeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23665dd0 0x55cd23668260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49662/0 (socket says 172.21.15.29:49662) 2024-03-11T19:16:54.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.903+0000 7faadeffd640 1 -- 172.21.15.29:0/1049357077 learned_addr learned my addr 172.21.15.29:0/1049357077 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:54.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.903+0000 7faadffff640 1 --2- 172.21.15.29:0/1049357077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23660270 0x55cd23660690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.903+0000 7faadf7fe640 1 --2- 172.21.15.29:0/1049357077 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd23660c60 0x55cd23665760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.904+0000 7faadeffd640 1 -- 172.21.15.29:0/1049357077 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd23660c60 msgr2=0x55cd23665760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.904+0000 7faadeffd640 1 --2- 172.21.15.29:0/1049357077 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd23660c60 0x55cd23665760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.904+0000 7faadeffd640 1 -- 172.21.15.29:0/1049357077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23660270 msgr2=0x55cd23660690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.904+0000 7faadeffd640 1 --2- 172.21.15.29:0/1049357077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23660270 0x55cd23660690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.904+0000 7faadeffd640 1 -- 172.21.15.29:0/1049357077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd2358b060 con 0x55cd23665dd0 2024-03-11T19:16:54.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.904+0000 7faadeffd640 1 --2- 172.21.15.29:0/1049357077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23665dd0 0x55cd23668260 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7faad00d0e10 tx=0x7faad00d54b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9130d2e3f446c91 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.904+0000 7faac5ffb640 1 -- 172.21.15.29:0/1049357077 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad00db600 con 0x55cd23665dd0 2024-03-11T19:16:54.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.904+0000 7faac5ffb640 1 -- 172.21.15.29:0/1049357077 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faad00d5e90 con 0x55cd23665dd0 2024-03-11T19:16:54.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.904+0000 7faac5ffb640 1 -- 172.21.15.29:0/1049357077 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad00daa50 con 0x55cd23665dd0 2024-03-11T19:16:54.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.904+0000 7faadcff9640 1 --2- 172.21.15.29:0/1142211635 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faac80a1b20 0x7faac80a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.905+0000 7faadffff640 1 --2- 172.21.15.29:0/1142211635 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faac80a1b20 0x7faac80a3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.905+0000 7faae8d84740 1 -- 172.21.15.29:0/1049357077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23665dd0 msgr2=0x55cd23668260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.905+0000 7faae8d84740 1 --2- 172.21.15.29:0/1049357077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23665dd0 0x55cd23668260 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7faad00d0e10 tx=0x7faad00d54b0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.906+0000 7faadcff9640 1 -- 172.21.15.29:0/1142211635 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7faad00ccfc0 con 0x7faae0101270 2024-03-11T19:16:54.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.906+0000 7faadffff640 1 --2- 172.21.15.29:0/1142211635 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faac80a1b20 0x7faac80a3fe0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7faad4006010 tx=0x7faad402c670 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.906+0000 7faae8d84740 1 -- 172.21.15.29:0/1049357077 shutdown_connections 2024-03-11T19:16:54.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.906+0000 7faae8d84740 1 --2- 172.21.15.29:0/1049357077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23665dd0 0x55cd23668260 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.906+0000 7faae8d84740 1 --2- 172.21.15.29:0/1049357077 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd23660c60 0x55cd23665760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.906+0000 7faae8d84740 1 --2- 172.21.15.29:0/1049357077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23660270 0x55cd23660690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.906+0000 7faae8d84740 1 -- 172.21.15.29:0/1049357077 >> 172.21.15.29:0/1049357077 conn(0x55cd2365b7e0 msgr2=0x55cd2365bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.906+0000 7faae8d84740 1 -- 172.21.15.29:0/1049357077 shutdown_connections 2024-03-11T19:16:54.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.906+0000 7faae8d84740 1 -- 172.21.15.29:0/1049357077 wait complete. 2024-03-11T19:16:54.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.906+0000 7faae8d84740 1 -- start start 2024-03-11T19:16:54.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.907+0000 7faae8d84740 1 Processor -- start 2024-03-11T19:16:54.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.907+0000 7faae8d84740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd23660270 0x55cd23672d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.907+0000 7faae8d84740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23660c60 0x55cd23673280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.907+0000 7faae8d84740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23665dd0 0x55cd2367a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.907+0000 7faadeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23660c60 0x55cd23673280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.907+0000 7faadeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23660c60 0x55cd23673280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49678/0 (socket says 172.21.15.29:49678) 2024-03-11T19:16:54.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.907+0000 7faadeffd640 1 -- 172.21.15.29:0/2045617223 learned_addr learned my addr 172.21.15.29:0/2045617223 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:54.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.907+0000 7faadf7fe640 1 --2- 172.21.15.29:0/2045617223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd23660270 0x55cd23672d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.907+0000 7faae8d84740 1 -- 172.21.15.29:0/2045617223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd235c9020 con 0x55cd23660c60 2024-03-11T19:16:54.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.907+0000 7faae8d84740 1 -- 172.21.15.29:0/2045617223 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd2329c430 con 0x55cd23665dd0 2024-03-11T19:16:54.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.907+0000 7faae8d84740 1 -- 172.21.15.29:0/2045617223 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd23258e00 con 0x55cd23660270 2024-03-11T19:16:54.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.908+0000 7faadffff640 1 --2- 172.21.15.29:0/2045617223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23665dd0 0x55cd2367a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.907+0000 7faae8d84740 10 client.0 _ll_register_callbacks cb 0x55cd23570cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.908+0000 7faadeffd640 1 -- 172.21.15.29:0/2045617223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd23660270 msgr2=0x55cd23672d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.908+0000 7faadeffd640 1 --2- 172.21.15.29:0/2045617223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd23660270 0x55cd23672d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.908+0000 7faadeffd640 1 -- 172.21.15.29:0/2045617223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23665dd0 msgr2=0x55cd2367a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.908+0000 7faadeffd640 1 --2- 172.21.15.29:0/2045617223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23665dd0 0x55cd2367a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.908+0000 7faadeffd640 1 -- 172.21.15.29:0/2045617223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae01a1a50 con 0x55cd23660c60 2024-03-11T19:16:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.908+0000 7faadf7fe640 1 --2- 172.21.15.29:0/2045617223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd23660270 0x55cd23672d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:16:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.908+0000 7faadeffd640 1 --2- 172.21.15.29:0/2045617223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23660c60 0x55cd23673280 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7faad0005fd0 tx=0x7faad0005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410ea7b14e0d557e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.908+0000 7faabeffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:16:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.908+0000 7faabeffd640 1 -- 172.21.15.29:0/2045617223 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad00da3a0 con 0x55cd23660c60 2024-03-11T19:16:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.908+0000 7faae8d84740 1 -- 172.21.15.29:0/2045617223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cd232c87e0 con 0x55cd23660c60 2024-03-11T19:16:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.908+0000 7faabeffd640 1 -- 172.21.15.29:0/2045617223 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7faad009a630 con 0x55cd23660c60 2024-03-11T19:16:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.909+0000 7faabeffd640 1 -- 172.21.15.29:0/2045617223 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7faad00961d0 con 0x55cd23660c60 2024-03-11T19:16:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.909+0000 7faae8d84740 10 client.17115 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.909+0000 7faae8d84740 1 -- 172.21.15.29:0/2045617223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55cd235c76a0 con 0x55cd23660c60 2024-03-11T19:16:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.909+0000 7faabeffd640 1 -- 172.21.15.29:0/2045617223 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faad00a6080 con 0x55cd23660c60 2024-03-11T19:16:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.909+0000 7faabeffd640 1 -- 172.21.15.29:0/2045617223 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7faad0016030 con 0x55cd23660c60 2024-03-11T19:16:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.909+0000 7faae8d84740 10 client.17115 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.909+0000 7faae8d84740 10 client.17115 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:16:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.909+0000 7faae8d84740 20 client.17115 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.909+0000 7faae8d84740 1 --2- 172.21.15.29:0/2045617223 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55cd2367e8a0 0x55cd23680cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.909+0000 7faae8d84740 4 client.17115 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:16:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.909+0000 7faae8d84740 1 -- 172.21.15.29:0/2045617223 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cd23305720 con 0x55cd2367e8a0 2024-03-11T19:16:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.910+0000 7faadf7fe640 1 --2- 172.21.15.29:0/2045617223 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55cd2367e8a0 0x55cd23680cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.910+0000 7faadf7fe640 1 --2- 172.21.15.29:0/2045617223 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55cd2367e8a0 0x55cd23680cc0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=19006b1cf0ef28ae server_cookie=9398c9f85986ee46 in_seq=0 out_seq=0 2024-03-11T19:16:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.910+0000 7faabeffd640 10 client.17115 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:16:54.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.911+0000 7faabeffd640 1 -- 172.21.15.29:0/2045617223 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cd23305720 con 0x55cd2367e8a0 2024-03-11T19:16:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.911+0000 7faabeffd640 10 client.17115 handle_command_reply: tid=0 2024-03-11T19:16:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.911+0000 7faae8d84740 1 client.17115 shutdown 2024-03-11T19:16:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.912+0000 7faae8d84740 1 -- 172.21.15.29:0/2045617223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23660c60 msgr2=0x55cd23673280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.912+0000 7faae8d84740 1 --2- 172.21.15.29:0/2045617223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23660c60 0x55cd23673280 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7faad0005fd0 tx=0x7faad0005dc0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.912+0000 7faae8d84740 1 -- 172.21.15.29:0/2045617223 shutdown_connections 2024-03-11T19:16:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.912+0000 7faae8d84740 1 --2- 172.21.15.29:0/2045617223 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55cd2367e8a0 0x55cd23680cc0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.912+0000 7faae8d84740 1 --2- 172.21.15.29:0/2045617223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23665dd0 0x55cd2367a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.912+0000 7faabeffd640 0 client.17115 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:16:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.912+0000 7faae8d84740 1 --2- 172.21.15.29:0/2045617223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23660c60 0x55cd23673280 secure :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7faad0005fd0 tx=0x7faad0005dc0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.912+0000 7faae8d84740 1 --2- 172.21.15.29:0/2045617223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd23660270 0x55cd23672d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.912+0000 7faae8d84740 1 -- 172.21.15.29:0/2045617223 >> 172.21.15.29:0/2045617223 conn(0x55cd2365b7e0 msgr2=0x55cd2365ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.913+0000 7faae8d84740 1 -- 172.21.15.29:0/2045617223 shutdown_connections 2024-03-11T19:16:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.913+0000 7faae8d84740 1 -- 172.21.15.29:0/2045617223 wait complete. 2024-03-11T19:16:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.914+0000 7faae8d84740 20 client.17115 trim_cache size 0 max 16384 2024-03-11T19:16:54.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.934+0000 7faae8d84740 1 Processor -- start 2024-03-11T19:16:54.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.934+0000 7faae8d84740 1 -- start start 2024-03-11T19:16:54.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.935+0000 7faae8d84740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23660270 0x55cd23672d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.935+0000 7faae8d84740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23665dd0 0x55cd23673280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.935+0000 7faae8d84740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd2367a210 0x55cd23662320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.935+0000 7faae8d84740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd2354e130 con 0x55cd23665dd0 2024-03-11T19:16:54.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.935+0000 7faae8d84740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd236610c0 con 0x55cd23660270 2024-03-11T19:16:54.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.935+0000 7faae8d84740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd236575c0 con 0x55cd2367a210 2024-03-11T19:16:54.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.935+0000 7faadf7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23660270 0x55cd23672d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.935+0000 7faadf7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23660270 0x55cd23672d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59126/0 (socket says 172.21.15.29:59126) 2024-03-11T19:16:54.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.935+0000 7faadf7fe640 1 -- 172.21.15.29:0/3229514517 learned_addr learned my addr 172.21.15.29:0/3229514517 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:54.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.936+0000 7faadffff640 1 --2- 172.21.15.29:0/3229514517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd2367a210 0x55cd23662320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.936+0000 7faadeffd640 1 --2- 172.21.15.29:0/3229514517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23665dd0 0x55cd23673280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.936+0000 7faadf7fe640 1 -- 172.21.15.29:0/3229514517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd2367a210 msgr2=0x55cd23662320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.936+0000 7faadf7fe640 1 --2- 172.21.15.29:0/3229514517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd2367a210 0x55cd23662320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.936+0000 7faadf7fe640 1 -- 172.21.15.29:0/3229514517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23665dd0 msgr2=0x55cd23673280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.936+0000 7faadf7fe640 1 --2- 172.21.15.29:0/3229514517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23665dd0 0x55cd23673280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.936+0000 7faadf7fe640 1 -- 172.21.15.29:0/3229514517 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd23563c20 con 0x55cd23660270 2024-03-11T19:16:54.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.936+0000 7faadeffd640 1 --2- 172.21.15.29:0/3229514517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23665dd0 0x55cd23673280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:16:54.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.936+0000 7faadf7fe640 1 --2- 172.21.15.29:0/3229514517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23660270 0x55cd23672d40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7faad800a870 tx=0x7faad80197d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5b53c362435c0c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.937+0000 7faac57fa640 1 -- 172.21.15.29:0/3229514517 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad8010040 con 0x55cd23660270 2024-03-11T19:16:54.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.937+0000 7faac57fa640 1 -- 172.21.15.29:0/3229514517 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faad80178a0 con 0x55cd23660270 2024-03-11T19:16:54.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.937+0000 7faac57fa640 1 -- 172.21.15.29:0/3229514517 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad8016810 con 0x55cd23660270 2024-03-11T19:16:54.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.939+0000 7faae8d84740 1 -- 172.21.15.29:0/3229514517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23660270 msgr2=0x55cd23672d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.939+0000 7faae8d84740 1 --2- 172.21.15.29:0/3229514517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23660270 0x55cd23672d40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7faad800a870 tx=0x7faad80197d0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.939+0000 7faae8d84740 1 -- 172.21.15.29:0/3229514517 shutdown_connections 2024-03-11T19:16:54.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.939+0000 7faae8d84740 1 --2- 172.21.15.29:0/3229514517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd2367a210 0x55cd23662320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.939+0000 7faae8d84740 1 --2- 172.21.15.29:0/3229514517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23665dd0 0x55cd23673280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.939+0000 7faae8d84740 1 --2- 172.21.15.29:0/3229514517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23660270 0x55cd23672d40 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.939+0000 7faae8d84740 1 -- 172.21.15.29:0/3229514517 >> 172.21.15.29:0/3229514517 conn(0x55cd2365b7e0 msgr2=0x55cd23685640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.939+0000 7faae8d84740 1 -- 172.21.15.29:0/3229514517 shutdown_connections 2024-03-11T19:16:54.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.940+0000 7faae8d84740 1 -- 172.21.15.29:0/3229514517 wait complete. 2024-03-11T19:16:54.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.940+0000 7faae8d84740 1 -- start start 2024-03-11T19:16:54.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.940+0000 7faae8d84740 1 Processor -- start 2024-03-11T19:16:54.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.941+0000 7faae8d84740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23660270 0x55cd236785f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.941+0000 7faae8d84740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23665dd0 0x55cd23678b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.941+0000 7faae8d84740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd2367a210 0x55cd23679070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.941+0000 7faae8d84740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd236d4db0 con 0x55cd23665dd0 2024-03-11T19:16:54.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.941+0000 7faae8d84740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd236d4f20 con 0x55cd23660270 2024-03-11T19:16:54.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.941+0000 7faae8d84740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd236d50e0 con 0x55cd2367a210 2024-03-11T19:16:54.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.941+0000 7faae8d84740 10 client.0 _ll_register_callbacks cb 0x7faad00d0810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:54.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.941+0000 7faadf7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23660270 0x55cd236785f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.941+0000 7faadf7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23660270 0x55cd236785f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59138/0 (socket says 172.21.15.29:59138) 2024-03-11T19:16:54.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.941+0000 7faadf7fe640 1 -- 172.21.15.29:0/1803163789 learned_addr learned my addr 172.21.15.29:0/1803163789 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:54.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.941+0000 7faadffff640 1 --2- 172.21.15.29:0/1803163789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd2367a210 0x55cd23679070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.941+0000 7faadeffd640 1 --2- 172.21.15.29:0/1803163789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23665dd0 0x55cd23678b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.941+0000 7faadf7fe640 1 -- 172.21.15.29:0/1803163789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd2367a210 msgr2=0x55cd23679070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.941+0000 7faadf7fe640 1 --2- 172.21.15.29:0/1803163789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd2367a210 0x55cd23679070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.941+0000 7faadf7fe640 1 -- 172.21.15.29:0/1803163789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23665dd0 msgr2=0x55cd23678b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:54.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.941+0000 7faadf7fe640 1 --2- 172.21.15.29:0/1803163789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23665dd0 0x55cd23678b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.941+0000 7faadf7fe640 1 -- 172.21.15.29:0/1803163789 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad800aa00 con 0x55cd23660270 2024-03-11T19:16:54.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.942+0000 7faadeffd640 1 --2- 172.21.15.29:0/1803163789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23665dd0 0x55cd23678b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:16:54.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.942+0000 7faadf7fe640 1 --2- 172.21.15.29:0/1803163789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23660270 0x55cd236785f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faad8004b90 tx=0x7faad8017620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d550132aa2f7b514 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:54.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.942+0000 7faabeffd640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:16:54.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.942+0000 7faabeffd640 1 -- 172.21.15.29:0/1803163789 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad8010050 con 0x55cd23660270 2024-03-11T19:16:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.942+0000 7faabeffd640 1 -- 172.21.15.29:0/1803163789 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7faad8016590 con 0x55cd23660270 2024-03-11T19:16:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.942+0000 7faae8d84740 1 -- 172.21.15.29:0/1803163789 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faad00d5e90 con 0x55cd23660270 2024-03-11T19:16:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.943+0000 7faabeffd640 1 -- 172.21.15.29:0/1803163789 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7faad8017e20 con 0x55cd23660270 2024-03-11T19:16:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.943+0000 7faae8d84740 10 client.26612 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.943+0000 7faae8d84740 1 -- 172.21.15.29:0/1803163789 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55cd235898c0 con 0x55cd23660270 2024-03-11T19:16:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.943+0000 7faabeffd640 1 -- 172.21.15.29:0/1803163789 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faad00d5e90 con 0x55cd23660270 2024-03-11T19:16:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.944+0000 7faabeffd640 1 -- 172.21.15.29:0/1803163789 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7faad8026070 con 0x55cd23660270 2024-03-11T19:16:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.944+0000 7faae8d84740 10 client.26612 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.944+0000 7faae8d84740 10 client.26612 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:16:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.944+0000 7faae8d84740 20 client.26612 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.944+0000 7faae8d84740 1 --2- 172.21.15.29:0/1803163789 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55cd236d6ac0 0x55cd236d8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.944+0000 7faae8d84740 4 client.26612 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:16:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.944+0000 7faae8d84740 1 -- 172.21.15.29:0/1803163789 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cd23354890 con 0x55cd236d6ac0 2024-03-11T19:16:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.944+0000 7faadeffd640 1 --2- 172.21.15.29:0/1803163789 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55cd236d6ac0 0x55cd236d8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:54.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.945+0000 7faadeffd640 1 --2- 172.21.15.29:0/1803163789 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55cd236d6ac0 0x55cd236d8ee0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8b3e2bda106b9563 server_cookie=13d8355d5fce9b89 in_seq=0 out_seq=0 2024-03-11T19:16:54.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.945+0000 7faabeffd640 10 client.26612 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:16:54.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.945+0000 7faabeffd640 1 -- 172.21.15.29:0/1803163789 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cd23354890 con 0x55cd236d6ac0 2024-03-11T19:16:54.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.945+0000 7faabeffd640 10 client.26612 handle_command_reply: tid=0 2024-03-11T19:16:54.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.945+0000 7faae8d84740 1 client.26612 shutdown 2024-03-11T19:16:54.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.946+0000 7faae8d84740 1 -- 172.21.15.29:0/1803163789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23660270 msgr2=0x55cd236785f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.946+0000 7faae8d84740 1 --2- 172.21.15.29:0/1803163789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23660270 0x55cd236785f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faad8004b90 tx=0x7faad8017620 comp rx=0 tx=0).stop 2024-03-11T19:16:54.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.946+0000 7faae8d84740 1 -- 172.21.15.29:0/1803163789 shutdown_connections 2024-03-11T19:16:54.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.946+0000 7faae8d84740 1 --2- 172.21.15.29:0/1803163789 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55cd236d6ac0 0x55cd236d8ee0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.946+0000 7faae8d84740 1 --2- 172.21.15.29:0/1803163789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd2367a210 0x55cd23679070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.946+0000 7faabeffd640 0 client.26612 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:16:54.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.946+0000 7faae8d84740 1 --2- 172.21.15.29:0/1803163789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd23665dd0 0x55cd23678b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.947+0000 7faae8d84740 1 --2- 172.21.15.29:0/1803163789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd23660270 0x55cd236785f0 secure :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faad8004b90 tx=0x7faad8017620 comp rx=0 tx=0).stop 2024-03-11T19:16:54.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.947+0000 7faae8d84740 1 -- 172.21.15.29:0/1803163789 >> 172.21.15.29:0/1803163789 conn(0x55cd2365b7e0 msgr2=0x55cd23662320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.947+0000 7faae8d84740 1 -- 172.21.15.29:0/1803163789 shutdown_connections 2024-03-11T19:16:54.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.947+0000 7faae8d84740 1 -- 172.21.15.29:0/1803163789 wait complete. 2024-03-11T19:16:54.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.948+0000 7faae8d84740 20 client.26612 trim_cache size 0 max 16384 2024-03-11T19:16:54.952 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:16:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.949+0000 7faae61b6640 1 -- 172.21.15.29:0/1142211635 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faac80a1b20 msgr2=0x7faac80a3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.949+0000 7faae61b6640 1 --2- 172.21.15.29:0/1142211635 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faac80a1b20 0x7faac80a3fe0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7faad4006010 tx=0x7faad402c670 comp rx=0 tx=0).stop 2024-03-11T19:16:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.949+0000 7faae61b6640 1 -- 172.21.15.29:0/1142211635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faae0101270 msgr2=0x7faae0199900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.949+0000 7faae61b6640 1 --2- 172.21.15.29:0/1142211635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faae0101270 0x7faae0199900 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faad000c890 tx=0x7faad000cd50 comp rx=0 tx=0).stop 2024-03-11T19:16:54.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.949+0000 7faae61b6640 1 -- 172.21.15.29:0/1142211635 shutdown_connections 2024-03-11T19:16:54.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.950+0000 7faae61b6640 1 --2- 172.21.15.29:0/1142211635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faae0101c30 0x7faae01a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.950+0000 7faae61b6640 1 --2- 172.21.15.29:0/1142211635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faae0101270 0x7faae0199900 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.950+0000 7faae61b6640 1 --2- 172.21.15.29:0/1142211635 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faac80a1b20 0x7faac80a3fe0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.950+0000 7faae61b6640 1 --2- 172.21.15.29:0/1142211635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faae0100080 0x7faae01993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:54.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.950+0000 7faae61b6640 1 -- 172.21.15.29:0/1142211635 >> 172.21.15.29:0/1142211635 conn(0x7faae00fb610 msgr2=0x7faae00fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:54.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.950+0000 7faae61b6640 1 -- 172.21.15.29:0/1142211635 shutdown_connections 2024-03-11T19:16:54.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:54.950+0000 7faae61b6640 1 -- 172.21.15.29:0/1142211635 wait complete. 2024-03-11T19:16:54.974 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:16:54.975 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:16:54.990 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:54 smithi029 ceph-mon[32439]: pgmap v564: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 70 MiB/s wr, 1.14k op/s 2024-03-11T19:16:54.990 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:54 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3252879837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:16:55.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.145+0000 7f5938c02640 1 -- 172.21.15.29:0/1593577025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5934104e30 msgr2=0x7f593410b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:55.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.145+0000 7f5938c02640 1 --2- 172.21.15.29:0/1593577025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5934104e30 0x7f593410b6c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f592c009be0 tx=0x7f592c02bef0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.145+0000 7f5938c02640 1 -- 172.21.15.29:0/1593577025 shutdown_connections 2024-03-11T19:16:55.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.145+0000 7f5938c02640 1 --2- 172.21.15.29:0/1593577025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5934104e30 0x7f593410b6c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.145+0000 7f5938c02640 1 --2- 172.21.15.29:0/1593577025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5934104470 0x7f59341048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.145+0000 7f5938c02640 1 --2- 172.21.15.29:0/1593577025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5934103280 0x7f59341036a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.145+0000 7f5938c02640 1 -- 172.21.15.29:0/1593577025 >> 172.21.15.29:0/1593577025 conn(0x7f59340fe7f0 msgr2=0x7f5934100c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:55.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.146+0000 7f5938c02640 1 -- 172.21.15.29:0/1593577025 shutdown_connections 2024-03-11T19:16:55.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.146+0000 7f5938c02640 1 -- 172.21.15.29:0/1593577025 wait complete. 2024-03-11T19:16:55.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.147+0000 7f5938c02640 1 Processor -- start 2024-03-11T19:16:55.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.147+0000 7f5938c02640 1 -- start start 2024-03-11T19:16:55.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.148+0000 7f5938c02640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5934103280 0x7f593419d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.148+0000 7f5938c02640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5934104470 0x7f593419daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.148+0000 7f5938c02640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5934104e30 0x7f59341a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.148+0000 7f5938c02640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59341a50a0 con 0x7f5934104470 2024-03-11T19:16:55.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.148+0000 7f5938c02640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59341a5210 con 0x7f5934104e30 2024-03-11T19:16:55.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.148+0000 7f5938c02640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59341a53d0 con 0x7f5934103280 2024-03-11T19:16:55.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.148+0000 7f5932ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5934104470 0x7f593419daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.148+0000 7f5932ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5934104470 0x7f593419daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49712/0 (socket says 172.21.15.29:49712) 2024-03-11T19:16:55.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.148+0000 7f5932ffd640 1 -- 172.21.15.29:0/3424932202 learned_addr learned my addr 172.21.15.29:0/3424932202 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:55.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.148+0000 7f5933fff640 1 --2- 172.21.15.29:0/3424932202 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5934104e30 0x7f59341a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.148+0000 7f59337fe640 1 --2- 172.21.15.29:0/3424932202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5934103280 0x7f593419d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.148+0000 7f5932ffd640 1 -- 172.21.15.29:0/3424932202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5934103280 msgr2=0x7f593419d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:55.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.148+0000 7f5932ffd640 1 --2- 172.21.15.29:0/3424932202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5934103280 0x7f593419d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.148+0000 7f5932ffd640 1 -- 172.21.15.29:0/3424932202 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5934104e30 msgr2=0x7f59341a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:55.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.148+0000 7f5932ffd640 1 --2- 172.21.15.29:0/3424932202 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5934104e30 0x7f59341a4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.148+0000 7f5932ffd640 1 -- 172.21.15.29:0/3424932202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592c009840 con 0x7f5934104470 2024-03-11T19:16:55.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.149+0000 7f5932ffd640 1 --2- 172.21.15.29:0/3424932202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5934104470 0x7f593419daf0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f592400e9f0 tx=0x7f592400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:55.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.149+0000 7f5930ff9640 1 -- 172.21.15.29:0/3424932202 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5924009700 con 0x7f5934104470 2024-03-11T19:16:55.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.149+0000 7f5938c02640 1 -- 172.21.15.29:0/3424932202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59341a56b0 con 0x7f5934104470 2024-03-11T19:16:55.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.149+0000 7f5938c02640 1 -- 172.21.15.29:0/3424932202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59341a5c10 con 0x7f5934104470 2024-03-11T19:16:55.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.149+0000 7f5930ff9640 1 -- 172.21.15.29:0/3424932202 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f592400ccc0 con 0x7f5934104470 2024-03-11T19:16:55.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.149+0000 7f5930ff9640 1 -- 172.21.15.29:0/3424932202 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5924018790 con 0x7f5934104470 2024-03-11T19:16:55.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.152+0000 7f5930ff9640 1 -- 172.21.15.29:0/3424932202 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f59240188f0 con 0x7f5934104470 2024-03-11T19:16:55.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.153+0000 7f593ca34740 1 Processor -- start 2024-03-11T19:16:55.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.153+0000 7f593ca34740 1 -- start start 2024-03-11T19:16:55.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.153+0000 7f593ca34740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6121460 0x556ed6121880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.153+0000 7f593ca34740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6121e50 0x556ed6126950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.153+0000 7f593ca34740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6126fc0 0x556ed6129450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.153+0000 7f593ca34740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ed60dd960 con 0x556ed6121460 2024-03-11T19:16:55.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.153+0000 7f593ca34740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ed6091310 con 0x556ed6121e50 2024-03-11T19:16:55.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.153+0000 7f593ca34740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ed608e840 con 0x556ed6126fc0 2024-03-11T19:16:55.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.153+0000 7f5933fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6121460 0x556ed6121880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.153+0000 7f5933fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6121460 0x556ed6121880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49718/0 (socket says 172.21.15.29:49718) 2024-03-11T19:16:55.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.153+0000 7f5932ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6126fc0 0x556ed6129450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.153+0000 7f5933fff640 1 -- 172.21.15.29:0/3735057647 learned_addr learned my addr 172.21.15.29:0/3735057647 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:55.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.154+0000 7f5930ff9640 1 --2- 172.21.15.29:0/3424932202 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f591c0a1b30 0x7f591c0a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.154+0000 7f5930ff9640 1 -- 172.21.15.29:0/3424932202 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5924014070 con 0x7f5934104470 2024-03-11T19:16:55.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.154+0000 7f5932ffd640 1 -- 172.21.15.29:0/3735057647 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6121e50 msgr2=0x556ed6126950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:16:55.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.154+0000 7f5932ffd640 1 --2- 172.21.15.29:0/3735057647 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6121e50 0x556ed6126950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.154+0000 7f5932ffd640 1 -- 172.21.15.29:0/3735057647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6121460 msgr2=0x556ed6121880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:55.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.154+0000 7f5932ffd640 1 --2- 172.21.15.29:0/3735057647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6121460 0x556ed6121880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.154+0000 7f5932ffd640 1 -- 172.21.15.29:0/3735057647 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ed604c060 con 0x556ed6126fc0 2024-03-11T19:16:55.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.154+0000 7f5933fff640 1 --2- 172.21.15.29:0/3735057647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6121460 0x556ed6121880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:16:55.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.154+0000 7f5933fff640 1 --2- 172.21.15.29:0/3424932202 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f591c0a1b30 0x7f591c0a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.154+0000 7f5932ffd640 1 --2- 172.21.15.29:0/3735057647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6126fc0 0x556ed6129450 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f59240cb160 tx=0x7f5924096950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fa4b3600d67b483 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:55.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.154+0000 7f5919ffb640 1 -- 172.21.15.29:0/3735057647 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5924099f90 con 0x556ed6126fc0 2024-03-11T19:16:55.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.154+0000 7f5919ffb640 1 -- 172.21.15.29:0/3735057647 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f592409ad30 con 0x556ed6126fc0 2024-03-11T19:16:55.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.155+0000 7f5919ffb640 1 -- 172.21.15.29:0/3735057647 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59240989c0 con 0x556ed6126fc0 2024-03-11T19:16:55.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.155+0000 7f5933fff640 1 --2- 172.21.15.29:0/3424932202 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f591c0a1b30 0x7f591c0a3ff0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x556ed61226b0 tx=0x7f592c009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:55.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.155+0000 7f593ca34740 1 -- 172.21.15.29:0/3735057647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6126fc0 msgr2=0x556ed6129450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:55.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.155+0000 7f593ca34740 1 --2- 172.21.15.29:0/3735057647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6126fc0 0x556ed6129450 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f59240cb160 tx=0x7f5924096950 comp rx=0 tx=0).stop 2024-03-11T19:16:55.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.156+0000 7f593ca34740 1 -- 172.21.15.29:0/3735057647 shutdown_connections 2024-03-11T19:16:55.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.156+0000 7f593ca34740 1 --2- 172.21.15.29:0/3735057647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6126fc0 0x556ed6129450 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.156+0000 7f593ca34740 1 --2- 172.21.15.29:0/3735057647 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6121e50 0x556ed6126950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.156+0000 7f593ca34740 1 --2- 172.21.15.29:0/3735057647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6121460 0x556ed6121880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.156+0000 7f593ca34740 1 -- 172.21.15.29:0/3735057647 >> 172.21.15.29:0/3735057647 conn(0x556ed611c9d0 msgr2=0x556ed611cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:55.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.156+0000 7f593ca34740 1 -- 172.21.15.29:0/3735057647 shutdown_connections 2024-03-11T19:16:55.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.156+0000 7f593ca34740 1 -- 172.21.15.29:0/3735057647 wait complete. 2024-03-11T19:16:55.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.156+0000 7f593ca34740 1 -- start start 2024-03-11T19:16:55.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.157+0000 7f593ca34740 1 Processor -- start 2024-03-11T19:16:55.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.157+0000 7f593ca34740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6121460 0x556ed612b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.157+0000 7f593ca34740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6121e50 0x556ed612bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.158+0000 7f593ca34740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6130d20 0x556ed6133110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.158+0000 7f593ca34740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ed608a020 con 0x556ed6130d20 2024-03-11T19:16:55.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.158+0000 7f593ca34740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ed5d5d430 con 0x556ed6121e50 2024-03-11T19:16:55.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.158+0000 7f593ca34740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ed5d19e00 con 0x556ed6121460 2024-03-11T19:16:55.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.158+0000 7f593ca34740 10 client.0 _ll_register_callbacks cb 0x556ed6031cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:55.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.158+0000 7f59337fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6121460 0x556ed612b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.158+0000 7f5932ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6121e50 0x556ed612bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.158+0000 7f5932ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6121e50 0x556ed612bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59174/0 (socket says 172.21.15.29:59174) 2024-03-11T19:16:55.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.158+0000 7f5932ffd640 1 -- 172.21.15.29:0/525759302 learned_addr learned my addr 172.21.15.29:0/525759302 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:55.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.158+0000 7f59337fe640 1 --2- 172.21.15.29:0/525759302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6121460 0x556ed612b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:54316/0 (socket says 172.21.15.29:54316) 2024-03-11T19:16:55.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.158+0000 7f5933fff640 1 --2- 172.21.15.29:0/525759302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6130d20 0x556ed6133110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.158+0000 7f59337fe640 1 -- 172.21.15.29:0/525759302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6121e50 msgr2=0x556ed612bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:55.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.158+0000 7f59337fe640 1 --2- 172.21.15.29:0/525759302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6121e50 0x556ed612bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.158+0000 7f59337fe640 1 -- 172.21.15.29:0/525759302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6130d20 msgr2=0x556ed6133110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:55.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.158+0000 7f59337fe640 1 --2- 172.21.15.29:0/525759302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6130d20 0x556ed6133110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.158+0000 7f59337fe640 1 -- 172.21.15.29:0/525759302 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59341a5c10 con 0x556ed6121460 2024-03-11T19:16:55.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.159+0000 7f59337fe640 1 --2- 172.21.15.29:0/525759302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6121460 0x556ed612b7b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x556ed6122250 tx=0x7f5928011b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36d06bf97f809e25 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:55.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.159+0000 7f5912ffd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:16:55.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.159+0000 7f5912ffd640 1 -- 172.21.15.29:0/525759302 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592800fd60 con 0x556ed6121460 2024-03-11T19:16:55.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.159+0000 7f5912ffd640 1 -- 172.21.15.29:0/525759302 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5928004020 con 0x556ed6121460 2024-03-11T19:16:55.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.160+0000 7f593ca34740 1 -- 172.21.15.29:0/525759302 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556ed5d897e0 con 0x556ed6121460 2024-03-11T19:16:55.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.161+0000 7f5912ffd640 1 -- 172.21.15.29:0/525759302 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f592800fec0 con 0x556ed6121460 2024-03-11T19:16:55.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.161+0000 7f593ca34740 10 client.26653 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:55.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.161+0000 7f593ca34740 1 -- 172.21.15.29:0/525759302 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x556ed60886a0 con 0x556ed6121460 2024-03-11T19:16:55.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.161+0000 7f5912ffd640 1 -- 172.21.15.29:0/525759302 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x556ed5d897e0 con 0x556ed6121460 2024-03-11T19:16:55.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.162+0000 7f5912ffd640 1 -- 172.21.15.29:0/525759302 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5928004920 con 0x556ed6121460 2024-03-11T19:16:55.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.163+0000 7f593ca34740 10 client.26653 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:55.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.163+0000 7f593ca34740 10 client.26653 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:16:55.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.163+0000 7f593ca34740 20 client.26653 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:55.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.163+0000 7f593ca34740 1 --2- 172.21.15.29:0/525759302 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x556ed6137de0 0x556ed613a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.163+0000 7f593ca34740 4 client.26653 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:16:55.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.163+0000 7f593ca34740 1 -- 172.21.15.29:0/525759302 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556ed5dc6720 con 0x556ed6137de0 2024-03-11T19:16:55.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.163+0000 7f5932ffd640 1 --2- 172.21.15.29:0/525759302 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x556ed6137de0 0x556ed613a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.164+0000 7f5932ffd640 1 --2- 172.21.15.29:0/525759302 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x556ed6137de0 0x556ed613a200 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cc2f07b6968e002f server_cookie=96897b525ddb434e in_seq=0 out_seq=0 2024-03-11T19:16:55.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.164+0000 7f5912ffd640 10 client.26653 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:16:55.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.165+0000 7f5912ffd640 1 -- 172.21.15.29:0/525759302 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556ed5dc6720 con 0x556ed6137de0 2024-03-11T19:16:55.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.165+0000 7f5912ffd640 10 client.26653 handle_command_reply: tid=0 2024-03-11T19:16:55.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.165+0000 7f593ca34740 1 client.26653 shutdown 2024-03-11T19:16:55.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.166+0000 7f593ca34740 1 -- 172.21.15.29:0/525759302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6121460 msgr2=0x556ed612b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:55.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.166+0000 7f593ca34740 1 --2- 172.21.15.29:0/525759302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6121460 0x556ed612b7b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x556ed6122250 tx=0x7f5928011b50 comp rx=0 tx=0).stop 2024-03-11T19:16:55.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.166+0000 7f593ca34740 1 -- 172.21.15.29:0/525759302 shutdown_connections 2024-03-11T19:16:55.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.166+0000 7f593ca34740 1 --2- 172.21.15.29:0/525759302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6130d20 0x556ed6133110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.166+0000 7f593ca34740 1 --2- 172.21.15.29:0/525759302 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x556ed6137de0 0x556ed613a200 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.166+0000 7f593ca34740 1 --2- 172.21.15.29:0/525759302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6121e50 0x556ed612bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.166+0000 7f593ca34740 1 --2- 172.21.15.29:0/525759302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6121460 0x556ed612b7b0 secure :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0x556ed6122250 tx=0x7f5928011b50 comp rx=0 tx=0).stop 2024-03-11T19:16:55.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.166+0000 7f593ca34740 1 -- 172.21.15.29:0/525759302 >> 172.21.15.29:0/525759302 conn(0x556ed611c9d0 msgr2=0x556ed611bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:55.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.166+0000 7f5912ffd640 0 client.26653 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:16:55.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.167+0000 7f593ca34740 1 -- 172.21.15.29:0/525759302 shutdown_connections 2024-03-11T19:16:55.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.167+0000 7f593ca34740 1 -- 172.21.15.29:0/525759302 wait complete. 2024-03-11T19:16:55.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.167+0000 7f593ca34740 20 client.26653 trim_cache size 0 max 16384 2024-03-11T19:16:55.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.188+0000 7f593ca34740 1 Processor -- start 2024-03-11T19:16:55.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.188+0000 7f593ca34740 1 -- start start 2024-03-11T19:16:55.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.189+0000 7f593ca34740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6121e50 0x556ed61480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.189+0000 7f593ca34740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6130d20 0x556ed6148600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.189+0000 7f593ca34740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6123420 0x556ed61238a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.189+0000 7f593ca34740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ed600f130 con 0x556ed6121e50 2024-03-11T19:16:55.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.189+0000 7f593ca34740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ed6129940 con 0x556ed6130d20 2024-03-11T19:16:55.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.189+0000 7f593ca34740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ed612c930 con 0x556ed6123420 2024-03-11T19:16:55.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.189+0000 7f5933fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6123420 0x556ed61238a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.189+0000 7f5933fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6123420 0x556ed61238a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:54326/0 (socket says 172.21.15.29:54326) 2024-03-11T19:16:55.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.189+0000 7f5933fff640 1 -- 172.21.15.29:0/4175387452 learned_addr learned my addr 172.21.15.29:0/4175387452 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:55.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.189+0000 7f5932ffd640 1 --2- 172.21.15.29:0/4175387452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6130d20 0x556ed6148600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.190+0000 7f5933fff640 1 -- 172.21.15.29:0/4175387452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6130d20 msgr2=0x556ed6148600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:55.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.190+0000 7f5933fff640 1 --2- 172.21.15.29:0/4175387452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6130d20 0x556ed6148600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.190+0000 7f5933fff640 1 -- 172.21.15.29:0/4175387452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6121e50 msgr2=0x556ed61480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:55.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.190+0000 7f5933fff640 1 --2- 172.21.15.29:0/4175387452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6121e50 0x556ed61480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.190+0000 7f5933fff640 1 -- 172.21.15.29:0/4175387452 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ed6024c20 con 0x556ed6123420 2024-03-11T19:16:55.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.190+0000 7f5933fff640 1 --2- 172.21.15.29:0/4175387452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6123420 0x556ed61238a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f592c009be0 tx=0x7f592c043000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7efc377bc57607e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:55.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.190+0000 7f59197fa640 1 -- 172.21.15.29:0/4175387452 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c0414f0 con 0x556ed6123420 2024-03-11T19:16:55.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.190+0000 7f59197fa640 1 -- 172.21.15.29:0/4175387452 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f592c041b00 con 0x556ed6123420 2024-03-11T19:16:55.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.190+0000 7f59197fa640 1 -- 172.21.15.29:0/4175387452 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c033c10 con 0x556ed6123420 2024-03-11T19:16:55.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.191+0000 7f593ca34740 1 -- 172.21.15.29:0/4175387452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6123420 msgr2=0x556ed61238a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:55.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.191+0000 7f593ca34740 1 --2- 172.21.15.29:0/4175387452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6123420 0x556ed61238a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f592c009be0 tx=0x7f592c043000 comp rx=0 tx=0).stop 2024-03-11T19:16:55.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.191+0000 7f593ca34740 1 -- 172.21.15.29:0/4175387452 shutdown_connections 2024-03-11T19:16:55.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.191+0000 7f593ca34740 1 --2- 172.21.15.29:0/4175387452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6123420 0x556ed61238a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.191+0000 7f593ca34740 1 --2- 172.21.15.29:0/4175387452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6130d20 0x556ed6148600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.191+0000 7f593ca34740 1 --2- 172.21.15.29:0/4175387452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6121e50 0x556ed61480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.191+0000 7f593ca34740 1 -- 172.21.15.29:0/4175387452 >> 172.21.15.29:0/4175387452 conn(0x556ed611c9d0 msgr2=0x556ed613eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:55.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.192+0000 7f593ca34740 1 -- 172.21.15.29:0/4175387452 shutdown_connections 2024-03-11T19:16:55.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.192+0000 7f593ca34740 1 -- 172.21.15.29:0/4175387452 wait complete. 2024-03-11T19:16:55.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.192+0000 7f593ca34740 1 -- start start 2024-03-11T19:16:55.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.192+0000 7f593ca34740 1 Processor -- start 2024-03-11T19:16:55.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.193+0000 7f593ca34740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6121e50 0x556ed6148540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.193+0000 7f593ca34740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6123420 0x556ed618ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.193+0000 7f593ca34740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6130d20 0x556ed618d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.193+0000 7f5932ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6123420 0x556ed618ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.193+0000 7f5932ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6123420 0x556ed618ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:54330/0 (socket says 172.21.15.29:54330) 2024-03-11T19:16:55.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.193+0000 7f59337fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6121e50 0x556ed6148540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.193+0000 7f59337fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6121e50 0x556ed6148540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59198/0 (socket says 172.21.15.29:59198) 2024-03-11T19:16:55.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.193+0000 7f5932ffd640 1 -- 172.21.15.29:0/4011228194 learned_addr learned my addr 172.21.15.29:0/4011228194 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:55.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.193+0000 7f593ca34740 1 -- 172.21.15.29:0/4011228194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ed618da00 con 0x556ed6130d20 2024-03-11T19:16:55.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.193+0000 7f593ca34740 1 -- 172.21.15.29:0/4011228194 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ed618db70 con 0x556ed6121e50 2024-03-11T19:16:55.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.193+0000 7f593ca34740 1 -- 172.21.15.29:0/4011228194 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ed618dd30 con 0x556ed6123420 2024-03-11T19:16:55.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.193+0000 7f593ca34740 10 client.0 _ll_register_callbacks cb 0x7f59240d0d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:55.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.194+0000 7f5933fff640 1 --2- 172.21.15.29:0/4011228194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6130d20 0x556ed618d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.194+0000 7f5933fff640 1 -- 172.21.15.29:0/4011228194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6123420 msgr2=0x556ed618ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:55.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.194+0000 7f5933fff640 1 --2- 172.21.15.29:0/4011228194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6123420 0x556ed618ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.194+0000 7f5933fff640 1 -- 172.21.15.29:0/4011228194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6121e50 msgr2=0x556ed6148540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:55.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.194+0000 7f5933fff640 1 --2- 172.21.15.29:0/4011228194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6121e50 0x556ed6148540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.194+0000 7f5933fff640 1 -- 172.21.15.29:0/4011228194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f592c030070 con 0x556ed6130d20 2024-03-11T19:16:55.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.195+0000 7f5933fff640 1 --2- 172.21.15.29:0/4011228194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6130d20 0x556ed618d300 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f592c02c500 tx=0x7f592c0417a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da3e636a02eb6afe server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:55.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.195+0000 7f5912ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:16:55.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.195+0000 7f5912ffd640 1 -- 172.21.15.29:0/4011228194 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c033da0 con 0x556ed6130d20 2024-03-11T19:16:55.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.196+0000 7f593ca34740 1 -- 172.21.15.29:0/4011228194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f592409ad30 con 0x556ed6130d20 2024-03-11T19:16:55.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.196+0000 7f5912ffd640 1 -- 172.21.15.29:0/4011228194 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f592c056430 con 0x556ed6130d20 2024-03-11T19:16:55.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.196+0000 7f5912ffd640 1 -- 172.21.15.29:0/4011228194 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f592c041990 con 0x556ed6130d20 2024-03-11T19:16:55.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.196+0000 7f593ca34740 10 client.17133 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:55.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.196+0000 7f593ca34740 1 -- 172.21.15.29:0/4011228194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x556ed604a8c0 con 0x556ed6130d20 2024-03-11T19:16:55.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.197+0000 7f5912ffd640 1 -- 172.21.15.29:0/4011228194 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f592409ad30 con 0x556ed6130d20 2024-03-11T19:16:55.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.197+0000 7f5912ffd640 1 -- 172.21.15.29:0/4011228194 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f592c049040 con 0x556ed6130d20 2024-03-11T19:16:55.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.197+0000 7f593ca34740 10 client.17133 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:55.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.197+0000 7f593ca34740 10 client.17133 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:16:55.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.197+0000 7f593ca34740 20 client.17133 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:55.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.197+0000 7f593ca34740 1 --2- 172.21.15.29:0/4011228194 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x556ed618f530 0x556ed6191950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.197+0000 7f593ca34740 4 client.17133 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:16:55.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.197+0000 7f593ca34740 1 -- 172.21.15.29:0/4011228194 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556ed5e15890 con 0x556ed618f530 2024-03-11T19:16:55.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.197+0000 7f59337fe640 1 --2- 172.21.15.29:0/4011228194 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x556ed618f530 0x556ed6191950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.198+0000 7f59337fe640 1 --2- 172.21.15.29:0/4011228194 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x556ed618f530 0x556ed6191950 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=88a733c40fd49a5f server_cookie=d5a07436dbdd2884 in_seq=0 out_seq=0 2024-03-11T19:16:55.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.198+0000 7f5912ffd640 10 client.17133 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:16:55.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.198+0000 7f5912ffd640 1 -- 172.21.15.29:0/4011228194 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556ed5e15890 con 0x556ed618f530 2024-03-11T19:16:55.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.198+0000 7f5912ffd640 10 client.17133 handle_command_reply: tid=0 2024-03-11T19:16:55.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.198+0000 7f593ca34740 1 client.17133 shutdown 2024-03-11T19:16:55.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.199+0000 7f593ca34740 1 -- 172.21.15.29:0/4011228194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6130d20 msgr2=0x556ed618d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:55.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.199+0000 7f593ca34740 1 --2- 172.21.15.29:0/4011228194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6130d20 0x556ed618d300 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f592c02c500 tx=0x7f592c0417a0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.199+0000 7f593ca34740 1 -- 172.21.15.29:0/4011228194 shutdown_connections 2024-03-11T19:16:55.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.199+0000 7f593ca34740 1 --2- 172.21.15.29:0/4011228194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556ed6130d20 0x556ed618d300 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.199+0000 7f593ca34740 1 --2- 172.21.15.29:0/4011228194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ed6123420 0x556ed618ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.199+0000 7f593ca34740 1 --2- 172.21.15.29:0/4011228194 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x556ed618f530 0x556ed6191950 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.199+0000 7f593ca34740 1 --2- 172.21.15.29:0/4011228194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556ed6121e50 0x556ed6148540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.199+0000 7f593ca34740 1 -- 172.21.15.29:0/4011228194 >> 172.21.15.29:0/4011228194 conn(0x556ed611c9d0 msgr2=0x556ed612a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:55.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.199+0000 7f5912ffd640 0 client.17133 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:16:55.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.200+0000 7f593ca34740 1 -- 172.21.15.29:0/4011228194 shutdown_connections 2024-03-11T19:16:55.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.201+0000 7f593ca34740 1 -- 172.21.15.29:0/4011228194 wait complete. 2024-03-11T19:16:55.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.202+0000 7f593ca34740 20 client.17133 trim_cache size 0 max 16384 2024-03-11T19:16:55.205 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:16:55.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.203+0000 7f5938c02640 1 -- 172.21.15.29:0/3424932202 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f591c0a1b30 msgr2=0x7f591c0a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:55.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.203+0000 7f5938c02640 1 --2- 172.21.15.29:0/3424932202 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f591c0a1b30 0x7f591c0a3ff0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x556ed61226b0 tx=0x7f592c009500 comp rx=0 tx=0).stop 2024-03-11T19:16:55.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.203+0000 7f5938c02640 1 -- 172.21.15.29:0/3424932202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5934104470 msgr2=0x7f593419daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:55.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.203+0000 7f5938c02640 1 --2- 172.21.15.29:0/3424932202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5934104470 0x7f593419daf0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f592400e9f0 tx=0x7f592400edf0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.203+0000 7f5938c02640 1 -- 172.21.15.29:0/3424932202 shutdown_connections 2024-03-11T19:16:55.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.203+0000 7f5938c02640 1 --2- 172.21.15.29:0/3424932202 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5934104e30 0x7f59341a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.203+0000 7f5938c02640 1 --2- 172.21.15.29:0/3424932202 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f591c0a1b30 0x7f591c0a3ff0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.203+0000 7f5938c02640 1 --2- 172.21.15.29:0/3424932202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5934104470 0x7f593419daf0 secure :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f592400e9f0 tx=0x7f592400edf0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.203+0000 7f5938c02640 1 --2- 172.21.15.29:0/3424932202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5934103280 0x7f593419d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.203+0000 7f5938c02640 1 -- 172.21.15.29:0/3424932202 >> 172.21.15.29:0/3424932202 conn(0x7f59340fe7f0 msgr2=0x7f59340791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:55.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.203+0000 7f5938c02640 1 -- 172.21.15.29:0/3424932202 shutdown_connections 2024-03-11T19:16:55.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.203+0000 7f5938c02640 1 -- 172.21.15.29:0/3424932202 wait complete. 2024-03-11T19:16:55.227 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:16:55.227 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:16:55.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:54 smithi190 ceph-mon[36292]: pgmap v564: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 70 MiB/s wr, 1.14k op/s 2024-03-11T19:16:55.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:54 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3252879837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:16:55.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:54 smithi170 ceph-mon[35690]: pgmap v564: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 70 MiB/s wr, 1.14k op/s 2024-03-11T19:16:55.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:54 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3252879837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:16:55.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.448+0000 7f269b1bc640 1 -- 172.21.15.29:0/1985733595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2694108180 msgr2=0x7f269410a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:55.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.448+0000 7f269b1bc640 1 --2- 172.21.15.29:0/1985733595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2694108180 0x7f269410a5b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f2684009900 tx=0x7f268402bc80 comp rx=0 tx=0).stop 2024-03-11T19:16:55.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.449+0000 7f269b1bc640 1 -- 172.21.15.29:0/1985733595 shutdown_connections 2024-03-11T19:16:55.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.449+0000 7f269b1bc640 1 --2- 172.21.15.29:0/1985733595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f269410d4a0 0x7f269410d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.449+0000 7f269b1bc640 1 --2- 172.21.15.29:0/1985733595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f269410aaf0 0x7f269410cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.449+0000 7f269b1bc640 1 --2- 172.21.15.29:0/1985733595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2694108180 0x7f269410a5b0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.449+0000 7f269b1bc640 1 -- 172.21.15.29:0/1985733595 >> 172.21.15.29:0/1985733595 conn(0x7f269406dfe0 msgr2=0x7f2694070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:55.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.449+0000 7f269b1bc640 1 -- 172.21.15.29:0/1985733595 shutdown_connections 2024-03-11T19:16:55.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.449+0000 7f269b1bc640 1 -- 172.21.15.29:0/1985733595 wait complete. 2024-03-11T19:16:55.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.450+0000 7f269b1bc640 1 Processor -- start 2024-03-11T19:16:55.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.450+0000 7f269b1bc640 1 -- start start 2024-03-11T19:16:55.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.450+0000 7f269b1bc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2694108180 0x7f26941a5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.451+0000 7f269b1bc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f269410aaf0 0x7f26941a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.451+0000 7f269b1bc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f269410d4a0 0x7f26941ad430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.451+0000 7f269b1bc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26941ad970 con 0x7f2694108180 2024-03-11T19:16:55.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.451+0000 7f269b1bc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26941adae0 con 0x7f269410aaf0 2024-03-11T19:16:55.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.451+0000 7f269b1bc640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26941adca0 con 0x7f269410d4a0 2024-03-11T19:16:55.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.451+0000 7f2698f58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2694108180 0x7f26941a5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.451+0000 7f2698f58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2694108180 0x7f26941a5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49752/0 (socket says 172.21.15.29:49752) 2024-03-11T19:16:55.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.451+0000 7f2698f58640 1 -- 172.21.15.29:0/3241404927 learned_addr learned my addr 172.21.15.29:0/3241404927 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:55.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.451+0000 7f2699759640 1 --2- 172.21.15.29:0/3241404927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f269410d4a0 0x7f26941ad430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.451+0000 7f2693fff640 1 --2- 172.21.15.29:0/3241404927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f269410aaf0 0x7f26941a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.451+0000 7f2698f58640 1 -- 172.21.15.29:0/3241404927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f269410d4a0 msgr2=0x7f26941ad430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:55.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.451+0000 7f2698f58640 1 --2- 172.21.15.29:0/3241404927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f269410d4a0 0x7f26941ad430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.451+0000 7f2698f58640 1 -- 172.21.15.29:0/3241404927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f269410aaf0 msgr2=0x7f26941a6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:55.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.451+0000 7f2698f58640 1 --2- 172.21.15.29:0/3241404927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f269410aaf0 0x7f26941a6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.451+0000 7f2698f58640 1 -- 172.21.15.29:0/3241404927 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2684009560 con 0x7f2694108180 2024-03-11T19:16:55.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.451+0000 7f2698f58640 1 --2- 172.21.15.29:0/3241404927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2694108180 0x7f26941a5ed0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2684036040 tx=0x7f26840045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:55.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.452+0000 7f2691ffb640 1 -- 172.21.15.29:0/3241404927 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26840038c0 con 0x7f2694108180 2024-03-11T19:16:55.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.452+0000 7f2691ffb640 1 -- 172.21.15.29:0/3241404927 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f268402de90 con 0x7f2694108180 2024-03-11T19:16:55.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.452+0000 7f269b1bc640 1 -- 172.21.15.29:0/3241404927 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26941adf20 con 0x7f2694108180 2024-03-11T19:16:55.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.452+0000 7f2691ffb640 1 -- 172.21.15.29:0/3241404927 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268403e450 con 0x7f2694108180 2024-03-11T19:16:55.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.452+0000 7f269b1bc640 1 -- 172.21.15.29:0/3241404927 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26941ae3c0 con 0x7f2694108180 2024-03-11T19:16:55.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.456+0000 7f269dd8f740 1 Processor -- start 2024-03-11T19:16:55.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.456+0000 7f269dd8f740 1 -- start start 2024-03-11T19:16:55.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.456+0000 7f269dd8f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac09270 0x563cbac09690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.456+0000 7f269dd8f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cbac09c60 0x563cbac0e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.456+0000 7f269dd8f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac0edd0 0x563cbac11260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.456+0000 7f269dd8f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cbabc5960 con 0x563cbac0edd0 2024-03-11T19:16:55.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.456+0000 7f269dd8f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cbab79310 con 0x563cbac09270 2024-03-11T19:16:55.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.456+0000 7f269dd8f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cbab76840 con 0x563cbac09c60 2024-03-11T19:16:55.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.456+0000 7f2699759640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac09270 0x563cbac09690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.456+0000 7f2699759640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac09270 0x563cbac09690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:43836/0 (socket says 172.21.15.29:43836) 2024-03-11T19:16:55.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.456+0000 7f2699759640 1 -- 172.21.15.29:0/1530239669 learned_addr learned my addr 172.21.15.29:0/1530239669 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:55.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.457+0000 7f2699759640 1 -- 172.21.15.29:0/1530239669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cbac09c60 msgr2=0x563cbac0e760 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:16:55.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.457+0000 7f2699759640 1 --2- 172.21.15.29:0/1530239669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cbac09c60 0x563cbac0e760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.457+0000 7f2699759640 1 -- 172.21.15.29:0/1530239669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac0edd0 msgr2=0x563cbac11260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:55.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.457+0000 7f2699759640 1 --2- 172.21.15.29:0/1530239669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac0edd0 0x563cbac11260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.457+0000 7f2699759640 1 -- 172.21.15.29:0/1530239669 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cbab34060 con 0x563cbac09270 2024-03-11T19:16:55.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.457+0000 7f2699759640 1 --2- 172.21.15.29:0/1530239669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac09270 0x563cbac09690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f268c009990 tx=0x7f268c009e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d63046089dcf4c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:55.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.457+0000 7f268affd640 1 -- 172.21.15.29:0/1530239669 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c00ea60 con 0x563cbac09270 2024-03-11T19:16:55.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.457+0000 7f268affd640 1 -- 172.21.15.29:0/1530239669 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f268c00ebc0 con 0x563cbac09270 2024-03-11T19:16:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.458+0000 7f268affd640 1 -- 172.21.15.29:0/1530239669 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c012630 con 0x563cbac09270 2024-03-11T19:16:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.458+0000 7f269dd8f740 1 -- 172.21.15.29:0/1530239669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac09270 msgr2=0x563cbac09690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.458+0000 7f269dd8f740 1 --2- 172.21.15.29:0/1530239669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac09270 0x563cbac09690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f268c009990 tx=0x7f268c009e50 comp rx=0 tx=0).stop 2024-03-11T19:16:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.459+0000 7f269dd8f740 1 -- 172.21.15.29:0/1530239669 shutdown_connections 2024-03-11T19:16:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.459+0000 7f269dd8f740 1 --2- 172.21.15.29:0/1530239669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac0edd0 0x563cbac11260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.459+0000 7f269dd8f740 1 --2- 172.21.15.29:0/1530239669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cbac09c60 0x563cbac0e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.459+0000 7f269dd8f740 1 --2- 172.21.15.29:0/1530239669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac09270 0x563cbac09690 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.459+0000 7f269dd8f740 1 -- 172.21.15.29:0/1530239669 >> 172.21.15.29:0/1530239669 conn(0x563cbac047e0 msgr2=0x563cbac04be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.459+0000 7f269dd8f740 1 -- 172.21.15.29:0/1530239669 shutdown_connections 2024-03-11T19:16:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.459+0000 7f269dd8f740 1 -- 172.21.15.29:0/1530239669 wait complete. 2024-03-11T19:16:55.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.459+0000 7f2691ffb640 1 -- 172.21.15.29:0/3241404927 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2684004f50 con 0x7f2694108180 2024-03-11T19:16:55.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.459+0000 7f269dd8f740 1 -- start start 2024-03-11T19:16:55.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.459+0000 7f269dd8f740 1 Processor -- start 2024-03-11T19:16:55.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.460+0000 7f269dd8f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac09270 0x563cbac13530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.460+0000 7f269dd8f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac09c60 0x563cbac13a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.460+0000 7f269dd8f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cbac0edd0 0x563cbac1aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.460+0000 7f269dd8f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cbab72020 con 0x563cbac09c60 2024-03-11T19:16:55.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.460+0000 7f269dd8f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cba845430 con 0x563cbac09270 2024-03-11T19:16:55.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.460+0000 7f269dd8f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cba801e00 con 0x563cbac0edd0 2024-03-11T19:16:55.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.460+0000 7f2699759640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac09270 0x563cbac13530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.460+0000 7f2699759640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac09270 0x563cbac13530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:43852/0 (socket says 172.21.15.29:43852) 2024-03-11T19:16:55.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.460+0000 7f2699759640 1 -- 172.21.15.29:0/3905644530 learned_addr learned my addr 172.21.15.29:0/3905644530 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:55.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.460+0000 7f269dd8f740 10 client.0 _ll_register_callbacks cb 0x563cbab19cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:55.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.461+0000 7f2699759640 1 -- 172.21.15.29:0/3905644530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cbac0edd0 msgr2=0x563cbac1aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:16:55.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.461+0000 7f2699759640 1 --2- 172.21.15.29:0/3905644530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cbac0edd0 0x563cbac1aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.461+0000 7f2699759640 1 -- 172.21.15.29:0/3905644530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac09c60 msgr2=0x563cbac13a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:55.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.461+0000 7f2699759640 1 --2- 172.21.15.29:0/3905644530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac09c60 0x563cbac13a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.461+0000 7f2699759640 1 -- 172.21.15.29:0/3905644530 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f268c0095c0 con 0x563cbac09270 2024-03-11T19:16:55.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.461+0000 7f2699759640 1 --2- 172.21.15.29:0/3905644530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac09270 0x563cbac13530 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f268c01c6b0 tx=0x7f268c013a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d413c5e36c0e303 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:55.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.461+0000 7f2673fff640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:16:55.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.462+0000 7f2673fff640 1 -- 172.21.15.29:0/3905644530 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c00ea60 con 0x563cbac09270 2024-03-11T19:16:55.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.462+0000 7f2673fff640 1 -- 172.21.15.29:0/3905644530 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f268c01caa0 con 0x563cbac09270 2024-03-11T19:16:55.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.462+0000 7f269dd8f740 1 -- 172.21.15.29:0/3905644530 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563cba8717e0 con 0x563cbac09270 2024-03-11T19:16:55.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.462+0000 7f2673fff640 1 -- 172.21.15.29:0/3905644530 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f268c01cc60 con 0x563cbac09270 2024-03-11T19:16:55.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.462+0000 7f2691ffb640 1 --2- 172.21.15.29:0/3241404927 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f26800a1a90 0x7f26800a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.462+0000 7f269dd8f740 10 client.26636 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:55.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.462+0000 7f269dd8f740 1 -- 172.21.15.29:0/3905644530 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563cbab706a0 con 0x563cbac09270 2024-03-11T19:16:55.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.462+0000 7f2673fff640 1 -- 172.21.15.29:0/3905644530 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x563cba8717e0 con 0x563cbac09270 2024-03-11T19:16:55.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.462+0000 7f2691ffb640 1 -- 172.21.15.29:0/3241404927 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f26840effe0 con 0x7f2694108180 2024-03-11T19:16:55.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.463+0000 7f2673fff640 1 -- 172.21.15.29:0/3905644530 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f268c011070 con 0x563cbac09270 2024-03-11T19:16:55.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.463+0000 7f269dd8f740 10 client.26636 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:55.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.463+0000 7f269dd8f740 10 client.26636 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:16:55.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.463+0000 7f269dd8f740 20 client.26636 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:55.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.463+0000 7f269dd8f740 1 --2- 172.21.15.29:0/3905644530 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x563cbac30660 0x563cbac32aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.463+0000 7f269dd8f740 4 client.26636 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:16:55.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.463+0000 7f269dd8f740 1 -- 172.21.15.29:0/3905644530 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563cba8ae720 con 0x563cbac30660 2024-03-11T19:16:55.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.463+0000 7f2693fff640 1 --2- 172.21.15.29:0/3905644530 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x563cbac30660 0x563cbac32aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.463+0000 7f2673fff640 10 client.26636 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:55.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.464+0000 7f2693fff640 1 --2- 172.21.15.29:0/3905644530 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x563cbac30660 0x563cbac32aa0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a2b37ceae0f7885 server_cookie=1f91f41e501c8e5a in_seq=0 out_seq=0 2024-03-11T19:16:55.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.465+0000 7f2698f58640 1 --2- 172.21.15.29:0/3241404927 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f26800a1a90 0x7f26800a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.465+0000 7f2673fff640 1 -- 172.21.15.29:0/3905644530 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563cba8ae720 con 0x563cbac30660 2024-03-11T19:16:55.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.465+0000 7f2673fff640 10 client.26636 handle_command_reply: tid=0 2024-03-11T19:16:55.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.465+0000 7f269dd8f740 1 client.26636 shutdown 2024-03-11T19:16:55.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.465+0000 7f269dd8f740 1 -- 172.21.15.29:0/3905644530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac09270 msgr2=0x563cbac13530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:55.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.465+0000 7f2698f58640 1 --2- 172.21.15.29:0/3241404927 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f26800a1a90 0x7f26800a3f50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x563cbac14a10 tx=0x7f2684005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:55.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.465+0000 7f269dd8f740 1 --2- 172.21.15.29:0/3905644530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac09270 0x563cbac13530 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f268c01c6b0 tx=0x7f268c013a00 comp rx=0 tx=0).stop 2024-03-11T19:16:55.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.465+0000 7f269dd8f740 1 -- 172.21.15.29:0/3905644530 shutdown_connections 2024-03-11T19:16:55.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.465+0000 7f269dd8f740 1 --2- 172.21.15.29:0/3905644530 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cbac0edd0 0x563cbac1aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.465+0000 7f269dd8f740 1 --2- 172.21.15.29:0/3905644530 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x563cbac30660 0x563cbac32aa0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.465+0000 7f269dd8f740 1 --2- 172.21.15.29:0/3905644530 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac09c60 0x563cbac13a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.465+0000 7f269dd8f740 1 --2- 172.21.15.29:0/3905644530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac09270 0x563cbac13530 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.465+0000 7f2673fff640 0 client.26636 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:55.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.465+0000 7f269dd8f740 1 -- 172.21.15.29:0/3905644530 >> 172.21.15.29:0/3905644530 conn(0x563cbac047e0 msgr2=0x563cbac03b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:55.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.466+0000 7f269dd8f740 1 -- 172.21.15.29:0/3905644530 shutdown_connections 2024-03-11T19:16:55.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.466+0000 7f269dd8f740 1 -- 172.21.15.29:0/3905644530 wait complete. 2024-03-11T19:16:55.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.466+0000 7f269dd8f740 20 client.26636 trim_cache size 0 max 16384 2024-03-11T19:16:55.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.489+0000 7f269dd8f740 1 Processor -- start 2024-03-11T19:16:55.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.489+0000 7f269dd8f740 1 -- start start 2024-03-11T19:16:55.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.489+0000 7f269dd8f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cbac09270 0x563cbac13140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.489+0000 7f269dd8f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac09c60 0x563cbac1aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.489+0000 7f269dd8f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac0edd0 0x563cbac13680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.489+0000 7f269dd8f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cbaaf7130 con 0x563cbac09c60 2024-03-11T19:16:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.489+0000 7f269dd8f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cba871810 con 0x563cbac0edd0 2024-03-11T19:16:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.489+0000 7f269dd8f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cbabf6a40 con 0x563cbac09270 2024-03-11T19:16:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.489+0000 7f2693fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac09c60 0x563cbac1aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.489+0000 7f2693fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac09c60 0x563cbac1aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59948/0 (socket says 172.21.15.29:59948) 2024-03-11T19:16:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.489+0000 7f2693fff640 1 -- 172.21.15.29:0/864695580 learned_addr learned my addr 172.21.15.29:0/864695580 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.490+0000 7f2699759640 1 --2- 172.21.15.29:0/864695580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cbac09270 0x563cbac13140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.490+0000 7f2693fff640 1 -- 172.21.15.29:0/864695580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cbac09270 msgr2=0x563cbac13140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.490+0000 7f2693fff640 1 --2- 172.21.15.29:0/864695580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cbac09270 0x563cbac13140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.490+0000 7f2693fff640 1 -- 172.21.15.29:0/864695580 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac0edd0 msgr2=0x563cbac13680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:16:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.490+0000 7f2693fff640 1 --2- 172.21.15.29:0/864695580 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac0edd0 0x563cbac13680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.490+0000 7f2693fff640 1 -- 172.21.15.29:0/864695580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cbab0cc20 con 0x563cbac09c60 2024-03-11T19:16:55.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.490+0000 7f2693fff640 1 --2- 172.21.15.29:0/864695580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac09c60 0x563cbac1aa70 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x563cbac333f0 tx=0x7f267c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5fd932ddd9fdd2a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:55.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.491+0000 7f268a7fc640 1 -- 172.21.15.29:0/864695580 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c016040 con 0x563cbac09c60 2024-03-11T19:16:55.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.491+0000 7f268a7fc640 1 -- 172.21.15.29:0/864695580 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x563cba8ae720 con 0x563cbac09c60 2024-03-11T19:16:55.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.491+0000 7f268a7fc640 1 -- 172.21.15.29:0/864695580 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c0174c0 con 0x563cbac09c60 2024-03-11T19:16:55.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.492+0000 7f269dd8f740 1 -- 172.21.15.29:0/864695580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac09c60 msgr2=0x563cbac1aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:55.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.492+0000 7f269dd8f740 1 --2- 172.21.15.29:0/864695580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac09c60 0x563cbac1aa70 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x563cbac333f0 tx=0x7f267c004ab0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.492+0000 7f269dd8f740 1 -- 172.21.15.29:0/864695580 shutdown_connections 2024-03-11T19:16:55.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.492+0000 7f269dd8f740 1 --2- 172.21.15.29:0/864695580 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac0edd0 0x563cbac13680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.492+0000 7f269dd8f740 1 --2- 172.21.15.29:0/864695580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac09c60 0x563cbac1aa70 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.492+0000 7f269dd8f740 1 --2- 172.21.15.29:0/864695580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cbac09270 0x563cbac13140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.492+0000 7f269dd8f740 1 -- 172.21.15.29:0/864695580 >> 172.21.15.29:0/864695580 conn(0x563cbac047e0 msgr2=0x563cbac04bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:55.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.493+0000 7f269dd8f740 1 -- 172.21.15.29:0/864695580 shutdown_connections 2024-03-11T19:16:55.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.494+0000 7f269dd8f740 1 -- 172.21.15.29:0/864695580 wait complete. 2024-03-11T19:16:55.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.494+0000 7f269dd8f740 1 -- start start 2024-03-11T19:16:55.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.494+0000 7f269dd8f740 1 Processor -- start 2024-03-11T19:16:55.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.495+0000 7f269dd8f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac09270 0x563cbac1a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.495+0000 7f269dd8f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cbac09c60 0x563cbac1a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.495+0000 7f269dd8f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac0edd0 0x563cbac87170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.495+0000 7f269dd8f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cbac876b0 con 0x563cbac09270 2024-03-11T19:16:55.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.495+0000 7f269dd8f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cbac87820 con 0x563cbac0edd0 2024-03-11T19:16:55.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.495+0000 7f269dd8f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cbac879e0 con 0x563cbac09c60 2024-03-11T19:16:55.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.495+0000 7f269dd8f740 10 client.0 _ll_register_callbacks cb 0x7f268c0093b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:55.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.495+0000 7f2699759640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac09270 0x563cbac1a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.495+0000 7f2699759640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac09270 0x563cbac1a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59962/0 (socket says 172.21.15.29:59962) 2024-03-11T19:16:55.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.495+0000 7f2693fff640 1 --2- 172.21.15.29:0/1606417883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cbac09c60 0x563cbac1a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.495+0000 7f2699759640 1 -- 172.21.15.29:0/1606417883 learned_addr learned my addr 172.21.15.29:0/1606417883 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:55.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.495+0000 7f2699759640 1 --2- 172.21.15.29:0/1606417883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac0edd0 0x563cbac87170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.496+0000 7f2699759640 1 -- 172.21.15.29:0/1606417883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cbac09c60 msgr2=0x563cbac1a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:55.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.496+0000 7f2699759640 1 --2- 172.21.15.29:0/1606417883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cbac09c60 0x563cbac1a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.496+0000 7f2699759640 1 -- 172.21.15.29:0/1606417883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac0edd0 msgr2=0x563cbac87170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:55.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.496+0000 7f2699759640 1 --2- 172.21.15.29:0/1606417883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac0edd0 0x563cbac87170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.496+0000 7f2699759640 1 -- 172.21.15.29:0/1606417883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f267c00ac90 con 0x563cbac09270 2024-03-11T19:16:55.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.496+0000 7f2693fff640 1 --2- 172.21.15.29:0/1606417883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cbac09c60 0x563cbac1a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:16:55.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.496+0000 7f2699759640 1 --2- 172.21.15.29:0/1606417883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac09270 0x563cbac1a0a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f268c00e620 tx=0x7f268c02f750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c3080857807a2f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:55.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.496+0000 7f2673fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:16:55.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.497+0000 7f2673fff640 1 -- 172.21.15.29:0/1606417883 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c02fe10 con 0x563cbac09270 2024-03-11T19:16:55.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.497+0000 7f2673fff640 1 -- 172.21.15.29:0/1606417883 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f268c0097e0 con 0x563cbac09270 2024-03-11T19:16:55.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.497+0000 7f269dd8f740 1 -- 172.21.15.29:0/1606417883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f268c00ebc0 con 0x563cbac09270 2024-03-11T19:16:55.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.497+0000 7f2673fff640 1 -- 172.21.15.29:0/1606417883 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f268c004040 con 0x563cbac09270 2024-03-11T19:16:55.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.497+0000 7f269dd8f740 10 client.17145 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:55.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.497+0000 7f269dd8f740 1 -- 172.21.15.29:0/1606417883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563cbab328c0 con 0x563cbac09270 2024-03-11T19:16:55.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.497+0000 7f2673fff640 1 -- 172.21.15.29:0/1606417883 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f268c03e080 con 0x563cbac09270 2024-03-11T19:16:55.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.497+0000 7f2673fff640 1 -- 172.21.15.29:0/1606417883 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f268c011040 con 0x563cbac09270 2024-03-11T19:16:55.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.498+0000 7f269dd8f740 10 client.17145 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:55.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.498+0000 7f269dd8f740 10 client.17145 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:16:55.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.498+0000 7f269dd8f740 20 client.17145 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:55.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.498+0000 7f269dd8f740 1 --2- 172.21.15.29:0/1606417883 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x563cbac88f70 0x563cbac8b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:55.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.498+0000 7f269dd8f740 4 client.17145 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:16:55.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.498+0000 7f269dd8f740 1 -- 172.21.15.29:0/1606417883 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f268c00ebc0 con 0x563cbac88f70 2024-03-11T19:16:55.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.498+0000 7f2693fff640 1 --2- 172.21.15.29:0/1606417883 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x563cbac88f70 0x563cbac8b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:55.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.498+0000 7f2673fff640 10 client.17145 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:55.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.498+0000 7f2693fff640 1 --2- 172.21.15.29:0/1606417883 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x563cbac88f70 0x563cbac8b390 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3078ca34d2644757 server_cookie=4cc086a8b4907b88 in_seq=0 out_seq=0 2024-03-11T19:16:55.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.499+0000 7f2673fff640 1 -- 172.21.15.29:0/1606417883 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f268c00ebc0 con 0x563cbac88f70 2024-03-11T19:16:55.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.499+0000 7f2673fff640 10 client.17145 handle_command_reply: tid=0 2024-03-11T19:16:55.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.499+0000 7f269dd8f740 1 client.17145 shutdown 2024-03-11T19:16:55.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.500+0000 7f269dd8f740 1 -- 172.21.15.29:0/1606417883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac09270 msgr2=0x563cbac1a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:55.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.500+0000 7f269dd8f740 1 --2- 172.21.15.29:0/1606417883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac09270 0x563cbac1a0a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f268c00e620 tx=0x7f268c02f750 comp rx=0 tx=0).stop 2024-03-11T19:16:55.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.500+0000 7f269dd8f740 1 -- 172.21.15.29:0/1606417883 shutdown_connections 2024-03-11T19:16:55.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.500+0000 7f269dd8f740 1 --2- 172.21.15.29:0/1606417883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563cbac0edd0 0x563cbac87170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.500+0000 7f269dd8f740 1 --2- 172.21.15.29:0/1606417883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563cbac09c60 0x563cbac1a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.500+0000 7f269dd8f740 1 --2- 172.21.15.29:0/1606417883 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x563cbac88f70 0x563cbac8b390 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.500+0000 7f269dd8f740 1 --2- 172.21.15.29:0/1606417883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563cbac09270 0x563cbac1a0a0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.501+0000 7f269dd8f740 1 -- 172.21.15.29:0/1606417883 >> 172.21.15.29:0/1606417883 conn(0x563cbac047e0 msgr2=0x563cbac13140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.501+0000 7f2673fff640 0 client.17145 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.501+0000 7f269dd8f740 1 -- 172.21.15.29:0/1606417883 shutdown_connections 2024-03-11T19:16:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.501+0000 7f269dd8f740 1 -- 172.21.15.29:0/1606417883 wait complete. 2024-03-11T19:16:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.502+0000 7f269dd8f740 20 client.17145 trim_cache size 0 max 16384 2024-03-11T19:16:55.507 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:16:55.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.504+0000 7f269b1bc640 1 -- 172.21.15.29:0/3241404927 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f26800a1a90 msgr2=0x7f26800a3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:55.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.504+0000 7f269b1bc640 1 --2- 172.21.15.29:0/3241404927 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f26800a1a90 0x7f26800a3f50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x563cbac14a10 tx=0x7f2684005fd0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.504+0000 7f269b1bc640 1 -- 172.21.15.29:0/3241404927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2694108180 msgr2=0x7f26941a5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:55.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.504+0000 7f269b1bc640 1 --2- 172.21.15.29:0/3241404927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2694108180 0x7f26941a5ed0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2684036040 tx=0x7f26840045e0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.504+0000 7f269b1bc640 1 -- 172.21.15.29:0/3241404927 shutdown_connections 2024-03-11T19:16:55.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.504+0000 7f269b1bc640 1 --2- 172.21.15.29:0/3241404927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f269410d4a0 0x7f26941ad430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.504+0000 7f269b1bc640 1 --2- 172.21.15.29:0/3241404927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f269410aaf0 0x7f26941a6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.504+0000 7f269b1bc640 1 --2- 172.21.15.29:0/3241404927 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f26800a1a90 0x7f26800a3f50 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.504+0000 7f269b1bc640 1 --2- 172.21.15.29:0/3241404927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2694108180 0x7f26941a5ed0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:55.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.504+0000 7f269b1bc640 1 -- 172.21.15.29:0/3241404927 >> 172.21.15.29:0/3241404927 conn(0x7f269406dfe0 msgr2=0x7f269410a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:55.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.504+0000 7f269b1bc640 1 -- 172.21.15.29:0/3241404927 shutdown_connections 2024-03-11T19:16:55.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:55.504+0000 7f269b1bc640 1 -- 172.21.15.29:0/3241404927 wait complete. 2024-03-11T19:16:56.527 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:16:56.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:16:56.528 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:16:56.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.716+0000 7fd98569d640 1 -- 172.21.15.29:0/2352379387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd98010a760 msgr2=0x7fd98010abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:56.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.716+0000 7fd98569d640 1 --2- 172.21.15.29:0/2352379387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd98010a760 0x7fd98010abc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd970009ca0 tx=0x7fd97002c0f0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.716+0000 7fd98569d640 1 -- 172.21.15.29:0/2352379387 shutdown_connections 2024-03-11T19:16:56.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.716+0000 7fd98569d640 1 --2- 172.21.15.29:0/2352379387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9801118c0 0x7fd980113cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.716+0000 7fd98569d640 1 --2- 172.21.15.29:0/2352379387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd98010a760 0x7fd98010abc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.716+0000 7fd98569d640 1 --2- 172.21.15.29:0/2352379387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd980072f90 0x7fd98010a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.716+0000 7fd98569d640 1 -- 172.21.15.29:0/2352379387 >> 172.21.15.29:0/2352379387 conn(0x7fd98006e1a0 msgr2=0x7fd9800705e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:56.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.716+0000 7fd98569d640 1 -- 172.21.15.29:0/2352379387 shutdown_connections 2024-03-11T19:16:56.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.716+0000 7fd98569d640 1 -- 172.21.15.29:0/2352379387 wait complete. 2024-03-11T19:16:56.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.716+0000 7fd98569d640 1 Processor -- start 2024-03-11T19:16:56.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.717+0000 7fd98569d640 1 -- start start 2024-03-11T19:16:56.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.717+0000 7fd98569d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd980072f90 0x7fd9801a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.717+0000 7fd98569d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9801118c0 0x7fd9801a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.717+0000 7fd97effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd980072f90 0x7fd9801a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.717+0000 7fd97effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd980072f90 0x7fd9801a5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59984/0 (socket says 172.21.15.29:59984) 2024-03-11T19:16:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.717+0000 7fd98569d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9801ab510 0x7fd9801ad900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.717+0000 7fd97effd640 1 -- 172.21.15.29:0/751302542 learned_addr learned my addr 172.21.15.29:0/751302542 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.717+0000 7fd977fff640 1 --2- 172.21.15.29:0/751302542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9801118c0 0x7fd9801a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.717+0000 7fd97f7fe640 1 --2- 172.21.15.29:0/751302542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9801ab510 0x7fd9801ad900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.717+0000 7fd98569d640 1 -- 172.21.15.29:0/751302542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9801ade40 con 0x7fd980072f90 2024-03-11T19:16:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.717+0000 7fd98569d640 1 -- 172.21.15.29:0/751302542 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9801adfb0 con 0x7fd9801ab510 2024-03-11T19:16:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.717+0000 7fd98569d640 1 -- 172.21.15.29:0/751302542 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9801ae170 con 0x7fd9801118c0 2024-03-11T19:16:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.718+0000 7fd97effd640 1 -- 172.21.15.29:0/751302542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9801118c0 msgr2=0x7fd9801a64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.718+0000 7fd97effd640 1 --2- 172.21.15.29:0/751302542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9801118c0 0x7fd9801a64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.718+0000 7fd97effd640 1 -- 172.21.15.29:0/751302542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9801ab510 msgr2=0x7fd9801ad900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.718+0000 7fd97effd640 1 --2- 172.21.15.29:0/751302542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9801ab510 0x7fd9801ad900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.718+0000 7fd97effd640 1 -- 172.21.15.29:0/751302542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd970009900 con 0x7fd980072f90 2024-03-11T19:16:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.718+0000 7fd97effd640 1 --2- 172.21.15.29:0/751302542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd980072f90 0x7fd9801a5f70 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fd96800d950 tx=0x7fd96800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:56.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.719+0000 7fd97cff9640 1 -- 172.21.15.29:0/751302542 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd968009c30 con 0x7fd980072f90 2024-03-11T19:16:56.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.719+0000 7fd97cff9640 1 -- 172.21.15.29:0/751302542 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd968009d90 con 0x7fd980072f90 2024-03-11T19:16:56.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.719+0000 7fd97cff9640 1 -- 172.21.15.29:0/751302542 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96800f6b0 con 0x7fd980072f90 2024-03-11T19:16:56.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.719+0000 7fd98569d640 1 -- 172.21.15.29:0/751302542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9801ae450 con 0x7fd980072f90 2024-03-11T19:16:56.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.719+0000 7fd98569d640 1 -- 172.21.15.29:0/751302542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9801ae8e0 con 0x7fd980072f90 2024-03-11T19:16:56.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.723+0000 7fd98827a740 1 Processor -- start 2024-03-11T19:16:56.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.723+0000 7fd98827a740 1 -- start start 2024-03-11T19:16:56.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.723+0000 7fd97cff9640 1 -- 172.21.15.29:0/751302542 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd968010aa0 con 0x7fd980072f90 2024-03-11T19:16:56.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.723+0000 7fd98827a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0c690 0x55d111a0cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:56.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.723+0000 7fd98827a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0d860 0x55d111a0dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:56.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.723+0000 7fd98827a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0e220 0x55d111a14ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:56.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.723+0000 7fd98827a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d11197bf90 con 0x55d111a0c690 2024-03-11T19:16:56.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.723+0000 7fd98827a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1119794c0 con 0x55d111a0e220 2024-03-11T19:16:56.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.723+0000 7fd98827a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d111974ca0 con 0x55d111a0d860 2024-03-11T19:16:56.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.724+0000 7fd977fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0e220 0x55d111a14ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:56.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.724+0000 7fd977fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0e220 0x55d111a14ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:43876/0 (socket says 172.21.15.29:43876) 2024-03-11T19:16:56.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.724+0000 7fd977fff640 1 -- 172.21.15.29:0/235083258 learned_addr learned my addr 172.21.15.29:0/235083258 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:56.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.724+0000 7fd97f7fe640 1 --2- 172.21.15.29:0/235083258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0c690 0x55d111a0cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:56.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.724+0000 7fd97effd640 1 --2- 172.21.15.29:0/235083258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0d860 0x55d111a0dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:56.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.724+0000 7fd97cff9640 1 --2- 172.21.15.29:0/751302542 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd96c0a1b30 0x7fd96c0a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:56.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.724+0000 7fd97cff9640 1 -- 172.21.15.29:0/751302542 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd9680ccf80 con 0x7fd980072f90 2024-03-11T19:16:56.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.724+0000 7fd977fff640 1 -- 172.21.15.29:0/235083258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0d860 msgr2=0x55d111a0dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:56.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.724+0000 7fd977fff640 1 --2- 172.21.15.29:0/235083258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0d860 0x55d111a0dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.724+0000 7fd977fff640 1 -- 172.21.15.29:0/235083258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0c690 msgr2=0x55d111a0cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:56.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.724+0000 7fd977fff640 1 --2- 172.21.15.29:0/235083258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0c690 0x55d111a0cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.724+0000 7fd977fff640 1 -- 172.21.15.29:0/235083258 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1119370b0 con 0x55d111a0e220 2024-03-11T19:16:56.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.725+0000 7fd97f7fe640 1 --2- 172.21.15.29:0/235083258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0c690 0x55d111a0cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:16:56.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.725+0000 7fd977fff640 1 --2- 172.21.15.29:0/235083258 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0e220 0x55d111a14ab0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd97002c700 tx=0x7fd970032340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78b27622360b9186 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:56.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.725+0000 7fd97f7fe640 1 --2- 172.21.15.29:0/751302542 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd96c0a1b30 0x7fd96c0a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:56.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.725+0000 7fd9757fa640 1 -- 172.21.15.29:0/235083258 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970032c90 con 0x55d111a0e220 2024-03-11T19:16:56.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.725+0000 7fd9757fa640 1 -- 172.21.15.29:0/235083258 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd970032df0 con 0x55d111a0e220 2024-03-11T19:16:56.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.725+0000 7fd9757fa640 1 -- 172.21.15.29:0/235083258 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970003970 con 0x55d111a0e220 2024-03-11T19:16:56.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.726+0000 7fd98827a740 1 -- 172.21.15.29:0/235083258 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0e220 msgr2=0x55d111a14ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:56.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.726+0000 7fd98827a740 1 --2- 172.21.15.29:0/235083258 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0e220 0x55d111a14ab0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd97002c700 tx=0x7fd970032340 comp rx=0 tx=0).stop 2024-03-11T19:16:56.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.726+0000 7fd97f7fe640 1 --2- 172.21.15.29:0/751302542 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd96c0a1b30 0x7fd96c0a3ff0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd978009710 tx=0x7fd978014040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:56.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.726+0000 7fd98827a740 1 -- 172.21.15.29:0/235083258 shutdown_connections 2024-03-11T19:16:56.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.726+0000 7fd98827a740 1 --2- 172.21.15.29:0/235083258 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0e220 0x55d111a14ab0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.726+0000 7fd98827a740 1 --2- 172.21.15.29:0/235083258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0d860 0x55d111a0dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.727+0000 7fd98827a740 1 --2- 172.21.15.29:0/235083258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0c690 0x55d111a0cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.727+0000 7fd98827a740 1 -- 172.21.15.29:0/235083258 >> 172.21.15.29:0/235083258 conn(0x55d111a07870 msgr2=0x55d111a07c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:56.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.727+0000 7fd98827a740 1 -- 172.21.15.29:0/235083258 shutdown_connections 2024-03-11T19:16:56.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.727+0000 7fd98827a740 1 -- 172.21.15.29:0/235083258 wait complete. 2024-03-11T19:16:56.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.727+0000 7fd98827a740 1 -- start start 2024-03-11T19:16:56.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.728+0000 7fd98827a740 1 Processor -- start 2024-03-11T19:16:56.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.728+0000 7fd98827a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0c690 0x55d111a16850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:56.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.728+0000 7fd98827a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0d860 0x55d111a16d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.729+0000 7fd97effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0c690 0x55d111a16850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.729+0000 7fd97effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0c690 0x55d111a16850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:36618/0 (socket says 172.21.15.29:36618) 2024-03-11T19:16:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.729+0000 7fd97effd640 1 -- 172.21.15.29:0/2894152771 learned_addr learned my addr 172.21.15.29:0/2894152771 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.729+0000 7fd977fff640 1 --2- 172.21.15.29:0/2894152771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0d860 0x55d111a16d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.729+0000 7fd98827a740 1 --2- 172.21.15.29:0/2894152771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0e220 0x55d111a1ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.729+0000 7fd98827a740 1 -- 172.21.15.29:0/2894152771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d111919f40 con 0x55d111a0e220 2024-03-11T19:16:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.729+0000 7fd98827a740 1 -- 172.21.15.29:0/2894152771 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1116484b0 con 0x55d111a0d860 2024-03-11T19:16:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.729+0000 7fd97f7fe640 1 --2- 172.21.15.29:0/2894152771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0e220 0x55d111a1ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.729+0000 7fd98827a740 1 -- 172.21.15.29:0/2894152771 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1118fa670 con 0x55d111a0c690 2024-03-11T19:16:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.729+0000 7fd98827a740 10 client.0 _ll_register_callbacks cb 0x55d111967f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.730+0000 7fd97f7fe640 1 -- 172.21.15.29:0/2894152771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0c690 msgr2=0x55d111a16850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.730+0000 7fd97f7fe640 1 --2- 172.21.15.29:0/2894152771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0c690 0x55d111a16850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.730+0000 7fd97f7fe640 1 -- 172.21.15.29:0/2894152771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0d860 msgr2=0x55d111a16d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.730+0000 7fd97f7fe640 1 --2- 172.21.15.29:0/2894152771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0d860 0x55d111a16d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.730+0000 7fd97f7fe640 1 -- 172.21.15.29:0/2894152771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9801146e0 con 0x55d111a0e220 2024-03-11T19:16:56.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.730+0000 7fd97f7fe640 1 --2- 172.21.15.29:0/2894152771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0e220 0x55d111a1ddd0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x55d111a0d760 tx=0x7fd978005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa1fa59adee6eb80 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:56.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.730+0000 7fd9667fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:16:56.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.730+0000 7fd9667fc640 1 -- 172.21.15.29:0/2894152771 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97801c680 con 0x55d111a0e220 2024-03-11T19:16:56.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.730+0000 7fd9667fc640 1 -- 172.21.15.29:0/2894152771 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd978010e10 con 0x55d111a0e220 2024-03-11T19:16:56.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.731+0000 7fd98827a740 1 -- 172.21.15.29:0/2894152771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d11194ef40 con 0x55d111a0e220 2024-03-11T19:16:56.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.731+0000 7fd9667fc640 1 -- 172.21.15.29:0/2894152771 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd97801c7e0 con 0x55d111a0e220 2024-03-11T19:16:56.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.731+0000 7fd98827a740 10 client.17157 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:56.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.731+0000 7fd98827a740 1 -- 172.21.15.29:0/2894152771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d111973320 con 0x55d111a0e220 2024-03-11T19:16:56.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.731+0000 7fd9667fc640 1 -- 172.21.15.29:0/2894152771 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd97801cdf0 con 0x55d111a0e220 2024-03-11T19:16:56.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.732+0000 7fd9667fc640 1 -- 172.21.15.29:0/2894152771 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd978027c90 con 0x55d111a0e220 2024-03-11T19:16:56.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.732+0000 7fd98827a740 10 client.17157 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:56.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.732+0000 7fd98827a740 10 client.17157 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:16:56.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.732+0000 7fd98827a740 20 client.17157 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:56.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.732+0000 7fd98827a740 1 --2- 172.21.15.29:0/2894152771 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d111a22a50 0x55d111a24ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:56.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.732+0000 7fd98827a740 4 client.17157 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:16:56.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.732+0000 7fd98827a740 1 -- 172.21.15.29:0/2894152771 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d11194ef40 con 0x55d111a22a50 2024-03-11T19:16:56.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.732+0000 7fd97effd640 1 --2- 172.21.15.29:0/2894152771 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d111a22a50 0x55d111a24ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:56.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.732+0000 7fd9667fc640 10 client.17157 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:56.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.733+0000 7fd97effd640 1 --2- 172.21.15.29:0/2894152771 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d111a22a50 0x55d111a24ed0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a418bf10190196e server_cookie=978cbc675319d36a in_seq=0 out_seq=0 2024-03-11T19:16:56.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.734+0000 7fd9667fc640 1 -- 172.21.15.29:0/2894152771 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d11194ef40 con 0x55d111a22a50 2024-03-11T19:16:56.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.734+0000 7fd9667fc640 10 client.17157 handle_command_reply: tid=0 2024-03-11T19:16:56.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.734+0000 7fd98827a740 1 client.17157 shutdown 2024-03-11T19:16:56.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.734+0000 7fd98827a740 1 -- 172.21.15.29:0/2894152771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0e220 msgr2=0x55d111a1ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:56.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.734+0000 7fd98827a740 1 --2- 172.21.15.29:0/2894152771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0e220 0x55d111a1ddd0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x55d111a0d760 tx=0x7fd978005f90 comp rx=0 tx=0).stop 2024-03-11T19:16:56.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.734+0000 7fd98827a740 1 -- 172.21.15.29:0/2894152771 shutdown_connections 2024-03-11T19:16:56.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.734+0000 7fd98827a740 1 --2- 172.21.15.29:0/2894152771 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d111a22a50 0x55d111a24ed0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.734+0000 7fd9667fc640 0 client.17157 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:56.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.734+0000 7fd98827a740 1 --2- 172.21.15.29:0/2894152771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0e220 0x55d111a1ddd0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.735+0000 7fd98827a740 1 --2- 172.21.15.29:0/2894152771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0d860 0x55d111a16d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.735+0000 7fd98827a740 1 --2- 172.21.15.29:0/2894152771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0c690 0x55d111a16850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.735+0000 7fd98827a740 1 -- 172.21.15.29:0/2894152771 >> 172.21.15.29:0/2894152771 conn(0x55d111a07870 msgr2=0x55d111a0aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:56.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.735+0000 7fd98827a740 1 -- 172.21.15.29:0/2894152771 shutdown_connections 2024-03-11T19:16:56.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.735+0000 7fd98827a740 1 -- 172.21.15.29:0/2894152771 wait complete. 2024-03-11T19:16:56.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.735+0000 7fd98827a740 20 client.17157 trim_cache size 0 max 16384 2024-03-11T19:16:56.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.766+0000 7fd98827a740 1 Processor -- start 2024-03-11T19:16:56.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.766+0000 7fd98827a740 1 -- start start 2024-03-11T19:16:56.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.767+0000 7fd98827a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0c690 0x55d111a16850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:56.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.767+0000 7fd98827a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0d860 0x55d111a16d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:56.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.767+0000 7fd98827a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0e220 0x55d111a1dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:56.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.767+0000 7fd98827a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d111a082c0 con 0x55d111a0c690 2024-03-11T19:16:56.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.767+0000 7fd98827a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1119fc350 con 0x55d111a0d860 2024-03-11T19:16:56.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.767+0000 7fd98827a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d111a21050 con 0x55d111a0e220 2024-03-11T19:16:56.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.767+0000 7fd97effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0c690 0x55d111a16850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:56.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.767+0000 7fd97effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0c690 0x55d111a16850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60024/0 (socket says 172.21.15.29:60024) 2024-03-11T19:16:56.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.767+0000 7fd97effd640 1 -- 172.21.15.29:0/3653824615 learned_addr learned my addr 172.21.15.29:0/3653824615 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:56.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.767+0000 7fd977fff640 1 --2- 172.21.15.29:0/3653824615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0d860 0x55d111a16d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:56.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.767+0000 7fd97f7fe640 1 --2- 172.21.15.29:0/3653824615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0e220 0x55d111a1dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:56.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.767+0000 7fd97effd640 1 -- 172.21.15.29:0/3653824615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0e220 msgr2=0x55d111a1dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:56.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.767+0000 7fd97effd640 1 --2- 172.21.15.29:0/3653824615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0e220 0x55d111a1dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.767+0000 7fd97effd640 1 -- 172.21.15.29:0/3653824615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0d860 msgr2=0x55d111a16d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:56.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.767+0000 7fd97effd640 1 --2- 172.21.15.29:0/3653824615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0d860 0x55d111a16d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.767+0000 7fd97effd640 1 -- 172.21.15.29:0/3653824615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d11190fc00 con 0x55d111a0c690 2024-03-11T19:16:56.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.767+0000 7fd977fff640 1 --2- 172.21.15.29:0/3653824615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0d860 0x55d111a16d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:16:56.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.768+0000 7fd97effd640 1 --2- 172.21.15.29:0/3653824615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0c690 0x55d111a16850 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fd9680ca9c0 tx=0x7fd968096240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdfafaa7d8d33ca0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:56.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.768+0000 7fd974ff9640 1 -- 172.21.15.29:0/3653824615 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9680caea0 con 0x55d111a0c690 2024-03-11T19:16:56.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.768+0000 7fd974ff9640 1 -- 172.21.15.29:0/3653824615 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd968099fa0 con 0x55d111a0c690 2024-03-11T19:16:56.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.768+0000 7fd974ff9640 1 -- 172.21.15.29:0/3653824615 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd968094b10 con 0x55d111a0c690 2024-03-11T19:16:56.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.769+0000 7fd98827a740 1 -- 172.21.15.29:0/3653824615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0c690 msgr2=0x55d111a16850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:56.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.769+0000 7fd98827a740 1 --2- 172.21.15.29:0/3653824615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0c690 0x55d111a16850 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fd9680ca9c0 tx=0x7fd968096240 comp rx=0 tx=0).stop 2024-03-11T19:16:56.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.769+0000 7fd98827a740 1 -- 172.21.15.29:0/3653824615 shutdown_connections 2024-03-11T19:16:56.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.769+0000 7fd98827a740 1 --2- 172.21.15.29:0/3653824615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0e220 0x55d111a1dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.769+0000 7fd98827a740 1 --2- 172.21.15.29:0/3653824615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0d860 0x55d111a16d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.769+0000 7fd98827a740 1 --2- 172.21.15.29:0/3653824615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0c690 0x55d111a16850 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.769+0000 7fd98827a740 1 -- 172.21.15.29:0/3653824615 >> 172.21.15.29:0/3653824615 conn(0x55d111a07870 msgr2=0x55d111a11460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:56.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.769+0000 7fd98827a740 1 -- 172.21.15.29:0/3653824615 shutdown_connections 2024-03-11T19:16:56.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.769+0000 7fd98827a740 1 -- 172.21.15.29:0/3653824615 wait complete. 2024-03-11T19:16:56.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.769+0000 7fd98827a740 1 -- start start 2024-03-11T19:16:56.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.769+0000 7fd98827a740 1 Processor -- start 2024-03-11T19:16:56.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.770+0000 7fd98827a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0c690 0x55d111a1d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:56.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.770+0000 7fd98827a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0d860 0x55d111a1da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:56.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.770+0000 7fd98827a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0e220 0x55d111a79630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:56.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.770+0000 7fd98827a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d111a79b70 con 0x55d111a0d860 2024-03-11T19:16:56.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.770+0000 7fd98827a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d111a79cb0 con 0x55d111a0e220 2024-03-11T19:16:56.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.770+0000 7fd977fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0d860 0x55d111a1da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:56.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.770+0000 7fd97effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0c690 0x55d111a1d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:56.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.770+0000 7fd977fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0d860 0x55d111a1da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60040/0 (socket says 172.21.15.29:60040) 2024-03-11T19:16:56.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.770+0000 7fd977fff640 1 -- 172.21.15.29:0/3089792927 learned_addr learned my addr 172.21.15.29:0/3089792927 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:56.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.770+0000 7fd98827a740 1 -- 172.21.15.29:0/3089792927 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d111a79e20 con 0x55d111a0c690 2024-03-11T19:16:56.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.770+0000 7fd97f7fe640 1 --2- 172.21.15.29:0/3089792927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0e220 0x55d111a79630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:56.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.770+0000 7fd98827a740 10 client.0 _ll_register_callbacks cb 0x7fd978005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:56.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.770+0000 7fd97effd640 1 -- 172.21.15.29:0/3089792927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0e220 msgr2=0x55d111a79630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:56.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.770+0000 7fd97effd640 1 --2- 172.21.15.29:0/3089792927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0e220 0x55d111a79630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.770+0000 7fd97effd640 1 -- 172.21.15.29:0/3089792927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0d860 msgr2=0x55d111a1da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:56.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.770+0000 7fd97effd640 1 --2- 172.21.15.29:0/3089792927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0d860 0x55d111a1da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.770+0000 7fd97effd640 1 -- 172.21.15.29:0/3089792927 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d1119370b0 con 0x55d111a0c690 2024-03-11T19:16:56.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.771+0000 7fd97effd640 1 --2- 172.21.15.29:0/3089792927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0c690 0x55d111a1d4c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd9680a2600 tx=0x7fd9680a2710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff2b61aa758852a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:56.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.771+0000 7fd9667fc640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:16:56.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.771+0000 7fd9667fc640 1 -- 172.21.15.29:0/3089792927 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd968014030 con 0x55d111a0c690 2024-03-11T19:16:56.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.771+0000 7fd9667fc640 1 -- 172.21.15.29:0/3089792927 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd968015070 con 0x55d111a0c690 2024-03-11T19:16:56.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.771+0000 7fd98827a740 1 -- 172.21.15.29:0/3089792927 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d11199a900 con 0x55d111a0c690 2024-03-11T19:16:56.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.771+0000 7fd9667fc640 1 -- 172.21.15.29:0/3089792927 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9680caea0 con 0x55d111a0c690 2024-03-11T19:16:56.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.771+0000 7fd98827a740 10 client.26683 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:56.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.771+0000 7fd98827a740 1 -- 172.21.15.29:0/3089792927 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7fd9801ae8e0 con 0x55d111a0c690 2024-03-11T19:16:56.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.771+0000 7fd9667fc640 1 -- 172.21.15.29:0/3089792927 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd96807c050 con 0x55d111a0c690 2024-03-11T19:16:56.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.772+0000 7fd9667fc640 1 -- 172.21.15.29:0/3089792927 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd96807f080 con 0x55d111a0c690 2024-03-11T19:16:56.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.772+0000 7fd98827a740 10 client.26683 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:56.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.772+0000 7fd98827a740 10 client.26683 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:16:56.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.772+0000 7fd98827a740 20 client.26683 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:56.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.772+0000 7fd98827a740 1 --2- 172.21.15.29:0/3089792927 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d111a22a50 0x55d111a7d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:56.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.772+0000 7fd98827a740 4 client.26683 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:16:56.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.772+0000 7fd98827a740 1 -- 172.21.15.29:0/3089792927 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55d11199a900 con 0x55d111a22a50 2024-03-11T19:16:56.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.772+0000 7fd977fff640 1 --2- 172.21.15.29:0/3089792927 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d111a22a50 0x55d111a7d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:56.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.773+0000 7fd9667fc640 10 client.26683 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:56.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.773+0000 7fd977fff640 1 --2- 172.21.15.29:0/3089792927 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d111a22a50 0x55d111a7d5b0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8662d3df6b38e71 server_cookie=47e39fdd429b9d38 in_seq=0 out_seq=0 2024-03-11T19:16:56.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.774+0000 7fd9667fc640 1 -- 172.21.15.29:0/3089792927 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d11199a900 con 0x55d111a22a50 2024-03-11T19:16:56.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.774+0000 7fd9667fc640 10 client.26683 handle_command_reply: tid=0 2024-03-11T19:16:56.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.775+0000 7fd98827a740 1 client.26683 shutdown 2024-03-11T19:16:56.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.775+0000 7fd98827a740 1 -- 172.21.15.29:0/3089792927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0c690 msgr2=0x55d111a1d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:56.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.775+0000 7fd98827a740 1 --2- 172.21.15.29:0/3089792927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0c690 0x55d111a1d4c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd9680a2600 tx=0x7fd9680a2710 comp rx=0 tx=0).stop 2024-03-11T19:16:56.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.775+0000 7fd98827a740 1 -- 172.21.15.29:0/3089792927 shutdown_connections 2024-03-11T19:16:56.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.775+0000 7fd98827a740 1 --2- 172.21.15.29:0/3089792927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d111a0e220 0x55d111a79630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.775+0000 7fd98827a740 1 --2- 172.21.15.29:0/3089792927 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d111a22a50 0x55d111a7d5b0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.775+0000 7fd98827a740 1 --2- 172.21.15.29:0/3089792927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d111a0d860 0x55d111a1da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.775+0000 7fd9667fc640 0 client.26683 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:56.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.775+0000 7fd98827a740 1 --2- 172.21.15.29:0/3089792927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d111a0c690 0x55d111a1d4c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.775+0000 7fd98827a740 1 -- 172.21.15.29:0/3089792927 >> 172.21.15.29:0/3089792927 conn(0x55d111a07870 msgr2=0x55d111a16720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:56.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.776+0000 7fd98827a740 1 -- 172.21.15.29:0/3089792927 shutdown_connections 2024-03-11T19:16:56.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.776+0000 7fd98827a740 1 -- 172.21.15.29:0/3089792927 wait complete. 2024-03-11T19:16:56.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.776+0000 7fd98827a740 20 client.26683 trim_cache size 0 max 16384 2024-03-11T19:16:56.782 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:16:56.782 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:16:56.782 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "2c4bcea3-55c1-47d6-8ef9-819876b946cb", 2024-03-11T19:16:56.782 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:16:56.782 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:16:56.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.780+0000 7fd98569d640 1 -- 172.21.15.29:0/751302542 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd96c0a1b30 msgr2=0x7fd96c0a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:56.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.780+0000 7fd98569d640 1 --2- 172.21.15.29:0/751302542 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd96c0a1b30 0x7fd96c0a3ff0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd978009710 tx=0x7fd978014040 comp rx=0 tx=0).stop 2024-03-11T19:16:56.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.780+0000 7fd98569d640 1 -- 172.21.15.29:0/751302542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd980072f90 msgr2=0x7fd9801a5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:56.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.780+0000 7fd98569d640 1 --2- 172.21.15.29:0/751302542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd980072f90 0x7fd9801a5f70 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fd96800d950 tx=0x7fd96800de10 comp rx=0 tx=0).stop 2024-03-11T19:16:56.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.780+0000 7fd98569d640 1 -- 172.21.15.29:0/751302542 shutdown_connections 2024-03-11T19:16:56.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.780+0000 7fd98569d640 1 --2- 172.21.15.29:0/751302542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9801ab510 0x7fd9801ad900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.780+0000 7fd98569d640 1 --2- 172.21.15.29:0/751302542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9801118c0 0x7fd9801a64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.780+0000 7fd98569d640 1 --2- 172.21.15.29:0/751302542 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd96c0a1b30 0x7fd96c0a3ff0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.780+0000 7fd98569d640 1 --2- 172.21.15.29:0/751302542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd980072f90 0x7fd9801a5f70 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:56.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.780+0000 7fd98569d640 1 -- 172.21.15.29:0/751302542 >> 172.21.15.29:0/751302542 conn(0x7fd98006e1a0 msgr2=0x7fd98006e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:56.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.781+0000 7fd98569d640 1 -- 172.21.15.29:0/751302542 shutdown_connections 2024-03-11T19:16:56.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:56.781+0000 7fd98569d640 1 -- 172.21.15.29:0/751302542 wait complete. 2024-03-11T19:16:56.814 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:16:56.814 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:16:57.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.107+0000 7f7a5a977640 1 -- 172.21.15.29:0/766254007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a5410e580 msgr2=0x7f7a54110a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:57.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.107+0000 7f7a5a977640 1 --2- 172.21.15.29:0/766254007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a5410e580 0x7f7a54110a20 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f7a48009900 tx=0x7f7a4802bc30 comp rx=0 tx=0).stop 2024-03-11T19:16:57.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.107+0000 7f7a5a977640 1 -- 172.21.15.29:0/766254007 shutdown_connections 2024-03-11T19:16:57.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.107+0000 7f7a5a977640 1 --2- 172.21.15.29:0/766254007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a5410e580 0x7f7a54110a20 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.107+0000 7f7a5a977640 1 --2- 172.21.15.29:0/766254007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a5410bbc0 0x7f7a5410dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.107+0000 7f7a5a977640 1 --2- 172.21.15.29:0/766254007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a54109290 0x7f7a5410b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.107+0000 7f7a5a977640 1 -- 172.21.15.29:0/766254007 >> 172.21.15.29:0/766254007 conn(0x7f7a5406e360 msgr2=0x7f7a540707a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:57.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.107+0000 7f7a5a977640 1 -- 172.21.15.29:0/766254007 shutdown_connections 2024-03-11T19:16:57.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.107+0000 7f7a5a977640 1 -- 172.21.15.29:0/766254007 wait complete. 2024-03-11T19:16:57.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.108+0000 7f7a5a977640 1 Processor -- start 2024-03-11T19:16:57.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.108+0000 7f7a5a977640 1 -- start start 2024-03-11T19:16:57.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.109+0000 7f7a5a977640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a54109290 0x7f7a5419d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:57.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.109+0000 7f7a5a977640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a5410bbc0 0x7f7a5419daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:57.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.109+0000 7f7a5a977640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a5410e580 0x7f7a541ad730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:57.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.109+0000 7f7a5a977640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5419e3a0 con 0x7f7a5410e580 2024-03-11T19:16:57.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.109+0000 7f7a5a977640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a541adc70 con 0x7f7a5410bbc0 2024-03-11T19:16:57.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.109+0000 7f7a5a977640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a541adde0 con 0x7f7a54109290 2024-03-11T19:16:57.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.109+0000 7f7a58f14640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a5410e580 0x7f7a541ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:57.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.109+0000 7f7a58f14640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a5410e580 0x7f7a541ad730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60058/0 (socket says 172.21.15.29:60058) 2024-03-11T19:16:57.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.109+0000 7f7a537fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a5410bbc0 0x7f7a5419daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:57.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.109+0000 7f7a58f14640 1 -- 172.21.15.29:0/2810708889 learned_addr learned my addr 172.21.15.29:0/2810708889 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:57.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.109+0000 7f7a53fff640 1 --2- 172.21.15.29:0/2810708889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a54109290 0x7f7a5419d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:57.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.109+0000 7f7a58f14640 1 -- 172.21.15.29:0/2810708889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a54109290 msgr2=0x7f7a5419d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:57.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.109+0000 7f7a58f14640 1 --2- 172.21.15.29:0/2810708889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a54109290 0x7f7a5419d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.109+0000 7f7a58f14640 1 -- 172.21.15.29:0/2810708889 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a5410bbc0 msgr2=0x7f7a5419daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:57.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.109+0000 7f7a58f14640 1 --2- 172.21.15.29:0/2810708889 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a5410bbc0 0x7f7a5419daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.109+0000 7f7a58f14640 1 -- 172.21.15.29:0/2810708889 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a48009560 con 0x7f7a5410e580 2024-03-11T19:16:57.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.110+0000 7f7a537fe640 1 --2- 172.21.15.29:0/2810708889 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a5410bbc0 0x7f7a5419daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:16:57.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.110+0000 7f7a58f14640 1 --2- 172.21.15.29:0/2810708889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a5410e580 0x7f7a541ad730 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f7a48036040 tx=0x7f7a48004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:57.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.110+0000 7f7a517fa640 1 -- 172.21.15.29:0/2810708889 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a480044b0 con 0x7f7a5410e580 2024-03-11T19:16:57.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.111+0000 7f7a517fa640 1 -- 172.21.15.29:0/2810708889 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7a48003aa0 con 0x7f7a5410e580 2024-03-11T19:16:57.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.111+0000 7f7a5a977640 1 -- 172.21.15.29:0/2810708889 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a541ae060 con 0x7f7a5410e580 2024-03-11T19:16:57.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.111+0000 7f7a517fa640 1 -- 172.21.15.29:0/2810708889 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a48005430 con 0x7f7a5410e580 2024-03-11T19:16:57.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.111+0000 7f7a5a977640 1 -- 172.21.15.29:0/2810708889 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a541ae560 con 0x7f7a5410e580 2024-03-11T19:16:57.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.115+0000 7f7a517fa640 1 -- 172.21.15.29:0/2810708889 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7a48003c10 con 0x7f7a5410e580 2024-03-11T19:16:57.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.115+0000 7f7a5d627740 1 Processor -- start 2024-03-11T19:16:57.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.116+0000 7f7a5d627740 1 -- start start 2024-03-11T19:16:57.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.116+0000 7f7a517fa640 1 --2- 172.21.15.29:0/2810708889 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7a3c0a1b80 0x7f7a3c0a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:57.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.116+0000 7f7a5d627740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3a2d0 0x55ea64b3a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:57.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.116+0000 7f7a5d627740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3acc0 0x55ea64b3f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:57.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.116+0000 7f7a5d627740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b3fe30 0x55ea64b422c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:57.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.116+0000 7f7a5d627740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea64aaa280 con 0x55ea64b3acc0 2024-03-11T19:16:57.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.116+0000 7f7a5d627740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea64aa77b0 con 0x55ea64b3a2d0 2024-03-11T19:16:57.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.116+0000 7f7a5d627740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea64aa2f90 con 0x55ea64b3fe30 2024-03-11T19:16:57.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.116+0000 7f7a517fa640 1 -- 172.21.15.29:0/2810708889 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7a4802de40 con 0x7f7a5410e580 2024-03-11T19:16:57.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.116+0000 7f7a537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3acc0 0x55ea64b3f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:57.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.116+0000 7f7a53fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3a2d0 0x55ea64b3a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:57.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.116+0000 7f7a537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3acc0 0x55ea64b3f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60074/0 (socket says 172.21.15.29:60074) 2024-03-11T19:16:57.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.116+0000 7f7a537fe640 1 -- 172.21.15.29:0/2993903550 learned_addr learned my addr 172.21.15.29:0/2993903550 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:57.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.116+0000 7f7a537fe640 1 --2- 172.21.15.29:0/2810708889 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7a3c0a1b80 0x7f7a3c0a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:57.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.116+0000 7f7a58f14640 1 --2- 172.21.15.29:0/2993903550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b3fe30 0x55ea64b422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:57.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.117+0000 7f7a58f14640 1 -- 172.21.15.29:0/2993903550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3a2d0 msgr2=0x55ea64b3a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:57.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.117+0000 7f7a58f14640 1 --2- 172.21.15.29:0/2993903550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3a2d0 0x55ea64b3a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.117+0000 7f7a58f14640 1 -- 172.21.15.29:0/2993903550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3acc0 msgr2=0x55ea64b3f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:57.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.117+0000 7f7a58f14640 1 --2- 172.21.15.29:0/2993903550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3acc0 0x55ea64b3f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.117+0000 7f7a58f14640 1 -- 172.21.15.29:0/2993903550 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea64a61570 con 0x55ea64b3fe30 2024-03-11T19:16:57.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.117+0000 7f7a53fff640 1 --2- 172.21.15.29:0/2993903550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3a2d0 0x55ea64b3a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:16:57.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.117+0000 7f7a58f14640 1 --2- 172.21.15.29:0/2993903550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b3fe30 0x55ea64b422c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7a480bc880 tx=0x7f7a48004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db030a925d4d76fd server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:57.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.117+0000 7f7a3a7fc640 1 -- 172.21.15.29:0/2993903550 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a480c0cd0 con 0x55ea64b3fe30 2024-03-11T19:16:57.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.117+0000 7f7a3a7fc640 1 -- 172.21.15.29:0/2993903550 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7a480c5320 con 0x55ea64b3fe30 2024-03-11T19:16:57.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.117+0000 7f7a3a7fc640 1 -- 172.21.15.29:0/2993903550 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a480c4e90 con 0x55ea64b3fe30 2024-03-11T19:16:57.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.117+0000 7f7a537fe640 1 --2- 172.21.15.29:0/2810708889 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7a3c0a1b80 0x7f7a3c0a4040 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009fd0 tx=0x7f7a44011440 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:57.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.118+0000 7f7a5d627740 1 -- 172.21.15.29:0/2993903550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b3fe30 msgr2=0x55ea64b422c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:57.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.118+0000 7f7a5d627740 1 --2- 172.21.15.29:0/2993903550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b3fe30 0x55ea64b422c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7a480bc880 tx=0x7f7a48004610 comp rx=0 tx=0).stop 2024-03-11T19:16:57.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.119+0000 7f7a5d627740 1 -- 172.21.15.29:0/2993903550 shutdown_connections 2024-03-11T19:16:57.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.119+0000 7f7a5d627740 1 --2- 172.21.15.29:0/2993903550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b3fe30 0x55ea64b422c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.119+0000 7f7a5d627740 1 --2- 172.21.15.29:0/2993903550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3acc0 0x55ea64b3f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.119+0000 7f7a5d627740 1 --2- 172.21.15.29:0/2993903550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3a2d0 0x55ea64b3a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.119+0000 7f7a5d627740 1 -- 172.21.15.29:0/2993903550 >> 172.21.15.29:0/2993903550 conn(0x55ea64b357f0 msgr2=0x55ea64b35bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:57.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.119+0000 7f7a5d627740 1 -- 172.21.15.29:0/2993903550 shutdown_connections 2024-03-11T19:16:57.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.119+0000 7f7a5d627740 1 -- 172.21.15.29:0/2993903550 wait complete. 2024-03-11T19:16:57.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.120+0000 7f7a5d627740 1 -- start start 2024-03-11T19:16:57.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.120+0000 7f7a5d627740 1 Processor -- start 2024-03-11T19:16:57.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.121+0000 7f7a5d627740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3a2d0 0x55ea64b44060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:57.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.121+0000 7f7a5d627740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3acc0 0x55ea64b445a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:57.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.121+0000 7f7a5d627740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b3fe30 0x55ea64b4b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:57.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.121+0000 7f7a5d627740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea64776430 con 0x55ea64b3acc0 2024-03-11T19:16:57.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.121+0000 7f7a5d627740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea64732e00 con 0x55ea64b3a2d0 2024-03-11T19:16:57.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.121+0000 7f7a5d627740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea64a27fd0 con 0x55ea64b3fe30 2024-03-11T19:16:57.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.122+0000 7f7a53fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3a2d0 0x55ea64b44060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:57.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.122+0000 7f7a537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3acc0 0x55ea64b445a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:57.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.122+0000 7f7a53fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3a2d0 0x55ea64b44060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:43916/0 (socket says 172.21.15.29:43916) 2024-03-11T19:16:57.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.122+0000 7f7a53fff640 1 -- 172.21.15.29:0/2640044496 learned_addr learned my addr 172.21.15.29:0/2640044496 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:57.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.122+0000 7f7a58f14640 1 --2- 172.21.15.29:0/2640044496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b3fe30 0x55ea64b4b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:57.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.122+0000 7f7a5d627740 10 client.0 _ll_register_callbacks cb 0x55ea64a7dfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:57.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.122+0000 7f7a53fff640 1 -- 172.21.15.29:0/2640044496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b3fe30 msgr2=0x55ea64b4b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:57.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.122+0000 7f7a53fff640 1 --2- 172.21.15.29:0/2640044496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b3fe30 0x55ea64b4b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.122+0000 7f7a53fff640 1 -- 172.21.15.29:0/2640044496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3acc0 msgr2=0x55ea64b445a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:57.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.122+0000 7f7a53fff640 1 --2- 172.21.15.29:0/2640044496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3acc0 0x55ea64b445a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.122+0000 7f7a53fff640 1 -- 172.21.15.29:0/2640044496 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a541ae560 con 0x55ea64b3a2d0 2024-03-11T19:16:57.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.123+0000 7f7a53fff640 1 --2- 172.21.15.29:0/2640044496 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3a2d0 0x55ea64b44060 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c012700 tx=0x7f7a4c012b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=214c3a8d21ffb936 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:57.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.123+0000 7f7a337fe640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:16:57.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.123+0000 7f7a337fe640 1 -- 172.21.15.29:0/2640044496 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c017070 con 0x55ea64b3a2d0 2024-03-11T19:16:57.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.123+0000 7f7a5d627740 1 -- 172.21.15.29:0/2640044496 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ea647a27e0 con 0x55ea64b3a2d0 2024-03-11T19:16:57.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.123+0000 7f7a337fe640 1 -- 172.21.15.29:0/2640044496 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7a4c004240 con 0x55ea64b3a2d0 2024-03-11T19:16:57.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.123+0000 7f7a337fe640 1 -- 172.21.15.29:0/2640044496 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a4c004400 con 0x55ea64b3a2d0 2024-03-11T19:16:57.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.123+0000 7f7a5d627740 10 client.26669 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:57.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.123+0000 7f7a5d627740 1 -- 172.21.15.29:0/2640044496 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ea64aa1610 con 0x55ea64b3a2d0 2024-03-11T19:16:57.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.123+0000 7f7a337fe640 1 -- 172.21.15.29:0/2640044496 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55ea647a27e0 con 0x55ea64b3a2d0 2024-03-11T19:16:57.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.123+0000 7f7a337fe640 1 -- 172.21.15.29:0/2640044496 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7a4c011990 con 0x55ea64b3a2d0 2024-03-11T19:16:57.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.124+0000 7f7a5d627740 10 client.26669 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:57.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.124+0000 7f7a5d627740 10 client.26669 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:16:57.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.124+0000 7f7a5d627740 20 client.26669 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:57.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.124+0000 7f7a5d627740 1 --2- 172.21.15.29:0/2640044496 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ea64b50270 0x55ea64b52690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:57.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.125+0000 7f7a5d627740 4 client.26669 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:16:57.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.125+0000 7f7a5d627740 1 -- 172.21.15.29:0/2640044496 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ea647df720 con 0x55ea64b50270 2024-03-11T19:16:57.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.125+0000 7f7a537fe640 1 --2- 172.21.15.29:0/2640044496 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ea64b50270 0x55ea64b52690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:57.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.126+0000 7f7a337fe640 10 client.26669 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:57.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.126+0000 7f7a537fe640 1 --2- 172.21.15.29:0/2640044496 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ea64b50270 0x55ea64b52690 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91e94cff0fd91fec server_cookie=be17386b549821c0 in_seq=0 out_seq=0 2024-03-11T19:16:57.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.126+0000 7f7a337fe640 1 -- 172.21.15.29:0/2640044496 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ea647df720 con 0x55ea64b50270 2024-03-11T19:16:57.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.126+0000 7f7a337fe640 10 client.26669 handle_command_reply: tid=0 2024-03-11T19:16:57.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.126+0000 7f7a5d627740 1 client.26669 shutdown 2024-03-11T19:16:57.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.127+0000 7f7a5d627740 1 -- 172.21.15.29:0/2640044496 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3a2d0 msgr2=0x55ea64b44060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:57.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.127+0000 7f7a5d627740 1 --2- 172.21.15.29:0/2640044496 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3a2d0 0x55ea64b44060 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c012700 tx=0x7f7a4c012b00 comp rx=0 tx=0).stop 2024-03-11T19:16:57.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.127+0000 7f7a5d627740 1 -- 172.21.15.29:0/2640044496 shutdown_connections 2024-03-11T19:16:57.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.127+0000 7f7a5d627740 1 --2- 172.21.15.29:0/2640044496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b3fe30 0x55ea64b4b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.127+0000 7f7a5d627740 1 --2- 172.21.15.29:0/2640044496 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ea64b50270 0x55ea64b52690 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.127+0000 7f7a5d627740 1 --2- 172.21.15.29:0/2640044496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3acc0 0x55ea64b445a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.127+0000 7f7a5d627740 1 --2- 172.21.15.29:0/2640044496 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3a2d0 0x55ea64b44060 secure :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c012700 tx=0x7f7a4c012b00 comp rx=0 tx=0).stop 2024-03-11T19:16:57.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.127+0000 7f7a5d627740 1 -- 172.21.15.29:0/2640044496 >> 172.21.15.29:0/2640044496 conn(0x55ea64b357f0 msgr2=0x55ea64b34a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:57.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.127+0000 7f7a337fe640 0 client.26669 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:57.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.127+0000 7f7a5d627740 1 -- 172.21.15.29:0/2640044496 shutdown_connections 2024-03-11T19:16:57.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.127+0000 7f7a5d627740 1 -- 172.21.15.29:0/2640044496 wait complete. 2024-03-11T19:16:57.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.127+0000 7f7a5d627740 20 client.26669 trim_cache size 0 max 16384 2024-03-11T19:16:57.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:56 smithi029 ceph-mon[32439]: pgmap v565: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 72 MiB/s wr, 1.18k op/s 2024-03-11T19:16:57.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.156+0000 7f7a5d627740 1 Processor -- start 2024-03-11T19:16:57.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.156+0000 7f7a5d627740 1 -- start start 2024-03-11T19:16:57.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.156+0000 7f7a5d627740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3acc0 0x55ea64b391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:57.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.156+0000 7f7a5d627740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3fe30 0x55ea64b4b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:57.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.156+0000 7f7a5d627740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b436b0 0x55ea64b43b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:57.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.156+0000 7f7a5d627740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea647df720 con 0x55ea64b3fe30 2024-03-11T19:16:57.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.156+0000 7f7a5d627740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea647a27e0 con 0x55ea64b3acc0 2024-03-11T19:16:57.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.156+0000 7f7a5d627740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea64b44d90 con 0x55ea64b436b0 2024-03-11T19:16:57.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.157+0000 7f7a53fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3acc0 0x55ea64b391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:57.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.157+0000 7f7a53fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3acc0 0x55ea64b391b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:43932/0 (socket says 172.21.15.29:43932) 2024-03-11T19:16:57.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.157+0000 7f7a53fff640 1 -- 172.21.15.29:0/3693815457 learned_addr learned my addr 172.21.15.29:0/3693815457 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:57.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.157+0000 7f7a58f14640 1 --2- 172.21.15.29:0/3693815457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b436b0 0x55ea64b43b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:57.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.157+0000 7f7a53fff640 1 -- 172.21.15.29:0/3693815457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b436b0 msgr2=0x55ea64b43b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:57.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.157+0000 7f7a53fff640 1 --2- 172.21.15.29:0/3693815457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b436b0 0x55ea64b43b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.157+0000 7f7a53fff640 1 -- 172.21.15.29:0/3693815457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3fe30 msgr2=0x55ea64b4b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:57.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.157+0000 7f7a53fff640 1 --2- 172.21.15.29:0/3693815457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3fe30 0x55ea64b4b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.157+0000 7f7a53fff640 1 -- 172.21.15.29:0/3693815457 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea64a3dac0 con 0x55ea64b3acc0 2024-03-11T19:16:57.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.158+0000 7f7a53fff640 1 --2- 172.21.15.29:0/3693815457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3acc0 0x55ea64b391b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c0126d0 tx=0x7f7a4c009f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a351e3309cf0323c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:57.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.158+0000 7f7a39ffb640 1 -- 172.21.15.29:0/3693815457 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c017070 con 0x55ea64b3acc0 2024-03-11T19:16:57.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.158+0000 7f7a39ffb640 1 -- 172.21.15.29:0/3693815457 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7a4c018040 con 0x55ea64b3acc0 2024-03-11T19:16:57.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.158+0000 7f7a39ffb640 1 -- 172.21.15.29:0/3693815457 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c003b50 con 0x55ea64b3acc0 2024-03-11T19:16:57.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.159+0000 7f7a5d627740 1 -- 172.21.15.29:0/3693815457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3acc0 msgr2=0x55ea64b391b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:57.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.159+0000 7f7a5d627740 1 --2- 172.21.15.29:0/3693815457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3acc0 0x55ea64b391b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c0126d0 tx=0x7f7a4c009f00 comp rx=0 tx=0).stop 2024-03-11T19:16:57.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.160+0000 7f7a5d627740 1 -- 172.21.15.29:0/3693815457 shutdown_connections 2024-03-11T19:16:57.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.160+0000 7f7a5d627740 1 --2- 172.21.15.29:0/3693815457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b436b0 0x55ea64b43b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.160+0000 7f7a5d627740 1 --2- 172.21.15.29:0/3693815457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3fe30 0x55ea64b4b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.160+0000 7f7a5d627740 1 --2- 172.21.15.29:0/3693815457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3acc0 0x55ea64b391b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.160+0000 7f7a5d627740 1 -- 172.21.15.29:0/3693815457 >> 172.21.15.29:0/3693815457 conn(0x55ea64b357f0 msgr2=0x55ea64b38aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:57.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.160+0000 7f7a5d627740 1 -- 172.21.15.29:0/3693815457 shutdown_connections 2024-03-11T19:16:57.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.160+0000 7f7a5d627740 1 -- 172.21.15.29:0/3693815457 wait complete. 2024-03-11T19:16:57.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.161+0000 7f7a5d627740 1 -- start start 2024-03-11T19:16:57.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.161+0000 7f7a5d627740 1 Processor -- start 2024-03-11T19:16:57.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.161+0000 7f7a5d627740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3acc0 0x55ea64b4a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:57.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.162+0000 7f7a5d627740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3fe30 0x55ea64b4ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:57.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.162+0000 7f7a5d627740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b436b0 0x55ea64ba6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:57.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.162+0000 7f7a5d627740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea64ba7080 con 0x55ea64b3acc0 2024-03-11T19:16:57.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.162+0000 7f7a5d627740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea64ba71c0 con 0x55ea64b3fe30 2024-03-11T19:16:57.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.162+0000 7f7a5d627740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea64ba7330 con 0x55ea64b436b0 2024-03-11T19:16:57.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.162+0000 7f7a5d627740 10 client.0 _ll_register_callbacks cb 0x7f7a480b8e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:16:57.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.162+0000 7f7a53fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3acc0 0x55ea64b4a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:57.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.162+0000 7f7a53fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3acc0 0x55ea64b4a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60108/0 (socket says 172.21.15.29:60108) 2024-03-11T19:16:57.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.162+0000 7f7a53fff640 1 -- 172.21.15.29:0/389551548 learned_addr learned my addr 172.21.15.29:0/389551548 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:16:57.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.162+0000 7f7a537fe640 1 --2- 172.21.15.29:0/389551548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3fe30 0x55ea64b4ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:57.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.162+0000 7f7a58f14640 1 --2- 172.21.15.29:0/389551548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b436b0 0x55ea64ba6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:57.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.162+0000 7f7a537fe640 1 -- 172.21.15.29:0/389551548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b436b0 msgr2=0x55ea64ba6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:57.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.162+0000 7f7a537fe640 1 --2- 172.21.15.29:0/389551548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b436b0 0x55ea64ba6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.162+0000 7f7a537fe640 1 -- 172.21.15.29:0/389551548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3acc0 msgr2=0x55ea64b4a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:16:57.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.163+0000 7f7a537fe640 1 --2- 172.21.15.29:0/389551548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3acc0 0x55ea64b4a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.163+0000 7f7a537fe640 1 -- 172.21.15.29:0/389551548 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ea64aa1610 con 0x55ea64b3fe30 2024-03-11T19:16:57.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.163+0000 7f7a53fff640 1 --2- 172.21.15.29:0/389551548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3acc0 0x55ea64b4a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:16:57.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.163+0000 7f7a537fe640 1 --2- 172.21.15.29:0/389551548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3fe30 0x55ea64b4ae90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7a44005fd0 tx=0x7f7a44004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5dda241ec18bb4d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:16:57.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.163+0000 7f7a337fe640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:16:57.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.163+0000 7f7a337fe640 1 -- 172.21.15.29:0/389551548 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a44024070 con 0x55ea64b3fe30 2024-03-11T19:16:57.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.163+0000 7f7a337fe640 1 -- 172.21.15.29:0/389551548 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7a440058d0 con 0x55ea64b3fe30 2024-03-11T19:16:57.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.164+0000 7f7a5d627740 1 -- 172.21.15.29:0/389551548 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a480c5320 con 0x55ea64b3fe30 2024-03-11T19:16:57.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.164+0000 7f7a337fe640 1 -- 172.21.15.29:0/389551548 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a4402b420 con 0x55ea64b3fe30 2024-03-11T19:16:57.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.164+0000 7f7a5d627740 10 client.26675 fetch_fsmap learned FSMap version 27 2024-03-11T19:16:57.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.164+0000 7f7a5d627740 1 -- 172.21.15.29:0/389551548 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ea64a636a0 con 0x55ea64b3fe30 2024-03-11T19:16:57.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.164+0000 7f7a337fe640 1 -- 172.21.15.29:0/389551548 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7a44025040 con 0x55ea64b3fe30 2024-03-11T19:16:57.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.165+0000 7f7a337fe640 1 -- 172.21.15.29:0/389551548 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7a4400eb30 con 0x55ea64b3fe30 2024-03-11T19:16:57.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.165+0000 7f7a5d627740 10 client.26675 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:16:57.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.165+0000 7f7a5d627740 10 client.26675 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:16:57.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.165+0000 7f7a5d627740 20 client.26675 populate_metadata read hostname 'smithi029' 2024-03-11T19:16:57.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.165+0000 7f7a5d627740 1 --2- 172.21.15.29:0/389551548 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ea64ba8dd0 0x55ea64bab1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:16:57.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.165+0000 7f7a5d627740 4 client.26675 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:16:57.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.165+0000 7f7a5d627740 1 -- 172.21.15.29:0/389551548 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f7a480c5320 con 0x55ea64ba8dd0 2024-03-11T19:16:57.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.165+0000 7f7a53fff640 1 --2- 172.21.15.29:0/389551548 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ea64ba8dd0 0x55ea64bab1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:16:57.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.167+0000 7f7a53fff640 1 --2- 172.21.15.29:0/389551548 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ea64ba8dd0 0x55ea64bab1f0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc120877978faf63 server_cookie=31c00883a5de76d7 in_seq=0 out_seq=0 2024-03-11T19:16:57.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.167+0000 7f7a337fe640 10 client.26675 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:57.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.169+0000 7f7a337fe640 1 -- 172.21.15.29:0/389551548 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7f7a480c5320 con 0x55ea64ba8dd0 2024-03-11T19:16:57.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.169+0000 7f7a337fe640 10 client.26675 handle_command_reply: tid=0 2024-03-11T19:16:57.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.170+0000 7f7a5d627740 1 client.26675 shutdown 2024-03-11T19:16:57.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.170+0000 7f7a5d627740 1 -- 172.21.15.29:0/389551548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3fe30 msgr2=0x55ea64b4ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:57.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.170+0000 7f7a5d627740 1 --2- 172.21.15.29:0/389551548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3fe30 0x55ea64b4ae90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7a44005fd0 tx=0x7f7a44004a40 comp rx=0 tx=0).stop 2024-03-11T19:16:57.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.170+0000 7f7a5d627740 1 -- 172.21.15.29:0/389551548 shutdown_connections 2024-03-11T19:16:57.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.170+0000 7f7a5d627740 1 --2- 172.21.15.29:0/389551548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea64b436b0 0x55ea64ba6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.170+0000 7f7a5d627740 1 --2- 172.21.15.29:0/389551548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea64b3fe30 0x55ea64b4ae90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.170+0000 7f7a5d627740 1 --2- 172.21.15.29:0/389551548 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ea64ba8dd0 0x55ea64bab1f0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.170+0000 7f7a5d627740 1 --2- 172.21.15.29:0/389551548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea64b3acc0 0x55ea64b4a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.170+0000 7f7a5d627740 1 -- 172.21.15.29:0/389551548 >> 172.21.15.29:0/389551548 conn(0x55ea64b357f0 msgr2=0x55ea64b443d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:57.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.170+0000 7f7a337fe640 0 client.26675 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:16:57.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.170+0000 7f7a5d627740 1 -- 172.21.15.29:0/389551548 shutdown_connections 2024-03-11T19:16:57.175 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:16:57.175 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:16:57.175 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:16:57.175 INFO:teuthology.orchestra.run.smithi029.stdout: "2c4bcea3-55c1-47d6-8ef9-819876b946cb": { 2024-03-11T19:16:57.175 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:16:57.175 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "2c4bcea3-55c1-47d6-8ef9-819876b946cb", 2024-03-11T19:16:57.175 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:16:57.175 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:16:57.176 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:16:57.176 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:16:57.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.170+0000 7f7a5d627740 1 -- 172.21.15.29:0/389551548 wait complete. 2024-03-11T19:16:57.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.170+0000 7f7a5d627740 20 client.26675 trim_cache size 0 max 16384 2024-03-11T19:16:57.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.171+0000 7f7a5a977640 1 -- 172.21.15.29:0/2810708889 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7a3c0a1b80 msgr2=0x7f7a3c0a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:57.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.171+0000 7f7a5a977640 1 --2- 172.21.15.29:0/2810708889 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7a3c0a1b80 0x7f7a3c0a4040 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009fd0 tx=0x7f7a44011440 comp rx=0 tx=0).stop 2024-03-11T19:16:57.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.171+0000 7f7a5a977640 1 -- 172.21.15.29:0/2810708889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a5410e580 msgr2=0x7f7a541ad730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:16:57.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.171+0000 7f7a5a977640 1 --2- 172.21.15.29:0/2810708889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a5410e580 0x7f7a541ad730 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f7a48036040 tx=0x7f7a48004040 comp rx=0 tx=0).stop 2024-03-11T19:16:57.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.171+0000 7f7a5a977640 1 -- 172.21.15.29:0/2810708889 shutdown_connections 2024-03-11T19:16:57.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.172+0000 7f7a5a977640 1 --2- 172.21.15.29:0/2810708889 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7a3c0a1b80 0x7f7a3c0a4040 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.172+0000 7f7a5a977640 1 --2- 172.21.15.29:0/2810708889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a5410e580 0x7f7a541ad730 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.172+0000 7f7a5a977640 1 --2- 172.21.15.29:0/2810708889 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a5410bbc0 0x7f7a5419daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.172+0000 7f7a5a977640 1 --2- 172.21.15.29:0/2810708889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a54109290 0x7f7a5419d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:16:57.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.172+0000 7f7a5a977640 1 -- 172.21.15.29:0/2810708889 >> 172.21.15.29:0/2810708889 conn(0x7f7a5406e360 msgr2=0x7f7a54113b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:16:57.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.173+0000 7f7a5a977640 1 -- 172.21.15.29:0/2810708889 shutdown_connections 2024-03-11T19:16:57.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:16:57.173+0000 7f7a5a977640 1 -- 172.21.15.29:0/2810708889 wait complete. 2024-03-11T19:16:57.196 INFO:tasks.cephfs.filesystem:scrub status for tag:2c4bcea3-55c1-47d6-8ef9-819876b946cb - {'path': '/', 'tag': '2c4bcea3-55c1-47d6-8ef9-819876b946cb', 'options': 'recursive,force'} 2024-03-11T19:16:57.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:56 smithi190 ceph-mon[36292]: pgmap v565: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 72 MiB/s wr, 1.18k op/s 2024-03-11T19:16:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:56 smithi170 ceph-mon[35690]: pgmap v565: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 72 MiB/s wr, 1.18k op/s 2024-03-11T19:16:58.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:57 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:16:58.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:57 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:16:58.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:57 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:16:58.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:57 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:16:58.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:57 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:16:58.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:57 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:16:58.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:57 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:16:58.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:57 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:16:58.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:57 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:16:58.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:57 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:16:58.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:57 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:16:58.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:57 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:16:59.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:16:58 smithi190 ceph-mon[36292]: pgmap v566: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 63 MiB/s wr, 1.04k op/s 2024-03-11T19:16:59.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:16:58 smithi170 ceph-mon[35690]: pgmap v566: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 63 MiB/s wr, 1.04k op/s 2024-03-11T19:16:59.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:16:58 smithi029 ceph-mon[32439]: pgmap v566: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 63 MiB/s wr, 1.04k op/s 2024-03-11T19:17:01.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:01 smithi190 ceph-mon[36292]: pgmap v567: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 68 MiB/s wr, 1.11k op/s 2024-03-11T19:17:01.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:01 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:17:01.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:01 smithi170 ceph-mon[35690]: pgmap v567: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 68 MiB/s wr, 1.11k op/s 2024-03-11T19:17:01.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:01 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:17:01.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:01 smithi029 ceph-mon[32439]: pgmap v567: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 68 MiB/s wr, 1.11k op/s 2024-03-11T19:17:01.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:01 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:17:03.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:03 smithi190 ceph-mon[36292]: pgmap v568: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 64 MiB/s wr, 1.05k op/s 2024-03-11T19:17:03.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:03 smithi170 ceph-mon[35690]: pgmap v568: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 64 MiB/s wr, 1.05k op/s 2024-03-11T19:17:03.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:03 smithi029 ceph-mon[32439]: pgmap v568: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 64 MiB/s wr, 1.05k op/s 2024-03-11T19:17:05.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:05 smithi190 ceph-mon[36292]: pgmap v569: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 65 MiB/s wr, 1.08k op/s 2024-03-11T19:17:05.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:05 smithi170 ceph-mon[35690]: pgmap v569: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 65 MiB/s wr, 1.08k op/s 2024-03-11T19:17:05.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:05 smithi029 ceph-mon[32439]: pgmap v569: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 65 MiB/s wr, 1.08k op/s 2024-03-11T19:17:07.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:07 smithi190 ceph-mon[36292]: pgmap v570: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 67 MiB/s wr, 1.12k op/s 2024-03-11T19:17:07.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:07 smithi170 ceph-mon[35690]: pgmap v570: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 67 MiB/s wr, 1.12k op/s 2024-03-11T19:17:07.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:07 smithi029 ceph-mon[32439]: pgmap v570: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 67 MiB/s wr, 1.12k op/s 2024-03-11T19:17:09.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:09 smithi170 ceph-mon[35690]: pgmap v571: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 60 MiB/s wr, 996 op/s 2024-03-11T19:17:09.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:09 smithi029 ceph-mon[32439]: pgmap v571: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 60 MiB/s wr, 996 op/s 2024-03-11T19:17:09.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:09 smithi190 ceph-mon[36292]: pgmap v571: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 60 MiB/s wr, 996 op/s 2024-03-11T19:17:11.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:11 smithi170 ceph-mon[35690]: pgmap v572: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 66 MiB/s wr, 1.11k op/s 2024-03-11T19:17:11.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:11 smithi029 ceph-mon[32439]: pgmap v572: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 66 MiB/s wr, 1.11k op/s 2024-03-11T19:17:11.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:11 smithi190 ceph-mon[36292]: pgmap v572: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 66 MiB/s wr, 1.11k op/s 2024-03-11T19:17:13.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:13 smithi170 ceph-mon[35690]: pgmap v573: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 63 MiB/s wr, 1.06k op/s 2024-03-11T19:17:13.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:13 smithi029 ceph-mon[32439]: pgmap v573: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 63 MiB/s wr, 1.06k op/s 2024-03-11T19:17:13.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:13 smithi190 ceph-mon[36292]: pgmap v573: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 63 MiB/s wr, 1.06k op/s 2024-03-11T19:17:15.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:15 smithi170 ceph-mon[35690]: pgmap v574: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 65 MiB/s wr, 1.10k op/s 2024-03-11T19:17:15.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:15 smithi029 ceph-mon[32439]: pgmap v574: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 65 MiB/s wr, 1.10k op/s 2024-03-11T19:17:15.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:15 smithi190 ceph-mon[36292]: pgmap v574: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 65 MiB/s wr, 1.10k op/s 2024-03-11T19:17:17.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:17 smithi170 ceph-mon[35690]: pgmap v575: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 67 MiB/s wr, 1.14k op/s 2024-03-11T19:17:17.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:17 smithi029 ceph-mon[32439]: pgmap v575: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 67 MiB/s wr, 1.14k op/s 2024-03-11T19:17:17.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:17 smithi190 ceph-mon[36292]: pgmap v575: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 67 MiB/s wr, 1.14k op/s 2024-03-11T19:17:19.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:19 smithi170 ceph-mon[35690]: pgmap v576: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 59 MiB/s wr, 1.02k op/s 2024-03-11T19:17:19.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:19 smithi029 ceph-mon[32439]: pgmap v576: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 59 MiB/s wr, 1.02k op/s 2024-03-11T19:17:19.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:19 smithi190 ceph-mon[36292]: pgmap v576: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 59 MiB/s wr, 1.02k op/s 2024-03-11T19:17:21.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:21 smithi170 ceph-mon[35690]: pgmap v577: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 65 MiB/s wr, 1.12k op/s 2024-03-11T19:17:21.409 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:21 smithi029 ceph-mon[32439]: pgmap v577: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 65 MiB/s wr, 1.12k op/s 2024-03-11T19:17:21.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:21 smithi190 ceph-mon[36292]: pgmap v577: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 65 MiB/s wr, 1.12k op/s 2024-03-11T19:17:23.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:23 smithi170 ceph-mon[35690]: pgmap v578: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 65 MiB/s wr, 1.11k op/s 2024-03-11T19:17:23.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:23 smithi029 ceph-mon[32439]: pgmap v578: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 65 MiB/s wr, 1.11k op/s 2024-03-11T19:17:23.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:23 smithi190 ceph-mon[36292]: pgmap v578: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 65 MiB/s wr, 1.11k op/s 2024-03-11T19:17:25.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:25 smithi170 ceph-mon[35690]: pgmap v579: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 64 MiB/s wr, 1.10k op/s 2024-03-11T19:17:25.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:25 smithi029 ceph-mon[32439]: pgmap v579: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 64 MiB/s wr, 1.10k op/s 2024-03-11T19:17:25.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:25 smithi190 ceph-mon[36292]: pgmap v579: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 64 MiB/s wr, 1.10k op/s 2024-03-11T19:17:27.196 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:17:27.197 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:17:27.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:27 smithi170 ceph-mon[35690]: pgmap v580: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 67 MiB/s wr, 1.15k op/s 2024-03-11T19:17:27.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.368+0000 7ff93d50c640 1 -- 172.21.15.29:0/1408948847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9381083f0 msgr2=0x7ff93810a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:27.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.368+0000 7ff93d50c640 1 --2- 172.21.15.29:0/1408948847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9381083f0 0x7ff93810a7e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff928009ca0 tx=0x7ff92802c100 comp rx=0 tx=0).stop 2024-03-11T19:17:27.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.369+0000 7ff93d50c640 1 -- 172.21.15.29:0/1408948847 shutdown_connections 2024-03-11T19:17:27.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.369+0000 7ff93d50c640 1 --2- 172.21.15.29:0/1408948847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff93810d830 0x7ff93810fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.369+0000 7ff93d50c640 1 --2- 172.21.15.29:0/1408948847 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff93810adb0 0x7ff93810d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.369+0000 7ff93d50c640 1 --2- 172.21.15.29:0/1408948847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9381083f0 0x7ff93810a7e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.369+0000 7ff93d50c640 1 -- 172.21.15.29:0/1408948847 >> 172.21.15.29:0/1408948847 conn(0x7ff93806e360 msgr2=0x7ff9380707a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:27.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.369+0000 7ff93d50c640 1 -- 172.21.15.29:0/1408948847 shutdown_connections 2024-03-11T19:17:27.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.370+0000 7ff93d50c640 1 -- 172.21.15.29:0/1408948847 wait complete. 2024-03-11T19:17:27.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.371+0000 7ff93d50c640 1 Processor -- start 2024-03-11T19:17:27.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.371+0000 7ff93d50c640 1 -- start start 2024-03-11T19:17:27.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.372+0000 7ff93d50c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9381083f0 0x7ff93819d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.372+0000 7ff93d50c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff93810adb0 0x7ff93819dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.372+0000 7ff93d50c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff93810d830 0x7ff9381a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.372+0000 7ff93d50c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9381a5200 con 0x7ff93810d830 2024-03-11T19:17:27.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.372+0000 7ff9377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff93810d830 0x7ff9381a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.372+0000 7ff9377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff93810d830 0x7ff9381a4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42134/0 (socket says 172.21.15.29:42134) 2024-03-11T19:17:27.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.373+0000 7ff9367fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff93810adb0 0x7ff93819dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.373+0000 7ff9367fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff93810adb0 0x7ff93819dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49856/0 (socket says 172.21.15.29:49856) 2024-03-11T19:17:27.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.373+0000 7ff9367fc640 1 -- 172.21.15.29:0/2109999173 learned_addr learned my addr 172.21.15.29:0/2109999173 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:27.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.373+0000 7ff936ffd640 1 --2- 172.21.15.29:0/2109999173 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9381083f0 0x7ff93819d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.373+0000 7ff93d50c640 1 -- 172.21.15.29:0/2109999173 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9381a5370 con 0x7ff93810adb0 2024-03-11T19:17:27.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.373+0000 7ff93d50c640 1 -- 172.21.15.29:0/2109999173 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9381a5530 con 0x7ff9381083f0 2024-03-11T19:17:27.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.374+0000 7ff936ffd640 1 -- 172.21.15.29:0/2109999173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff93810adb0 msgr2=0x7ff93819dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:27.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.374+0000 7ff936ffd640 1 --2- 172.21.15.29:0/2109999173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff93810adb0 0x7ff93819dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.374+0000 7ff936ffd640 1 -- 172.21.15.29:0/2109999173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff93810d830 msgr2=0x7ff9381a4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:27.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.374+0000 7ff936ffd640 1 --2- 172.21.15.29:0/2109999173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff93810d830 0x7ff9381a4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.374+0000 7ff936ffd640 1 -- 172.21.15.29:0/2109999173 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff928009900 con 0x7ff9381083f0 2024-03-11T19:17:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.375+0000 7ff936ffd640 1 --2- 172.21.15.29:0/2109999173 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9381083f0 0x7ff93819d710 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff928009bf0 tx=0x7ff928032920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.375+0000 7ff91ffff640 1 -- 172.21.15.29:0/2109999173 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff928034850 con 0x7ff9381083f0 2024-03-11T19:17:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.375+0000 7ff91ffff640 1 -- 172.21.15.29:0/2109999173 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff928032a90 con 0x7ff9381083f0 2024-03-11T19:17:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.376+0000 7ff91ffff640 1 -- 172.21.15.29:0/2109999173 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9280047d0 con 0x7ff9381083f0 2024-03-11T19:17:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.376+0000 7ff93d50c640 1 -- 172.21.15.29:0/2109999173 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9381a57b0 con 0x7ff9381083f0 2024-03-11T19:17:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.376+0000 7ff93d50c640 1 -- 172.21.15.29:0/2109999173 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9381a5c60 con 0x7ff9381083f0 2024-03-11T19:17:27.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.382+0000 7ff9401c3740 1 Processor -- start 2024-03-11T19:17:27.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.382+0000 7ff9401c3740 1 -- start start 2024-03-11T19:17:27.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.382+0000 7ff9401c3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601777972d0 0x5601777976f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.382+0000 7ff9401c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560177797cc0 0x56017779c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.382+0000 7ff9401c3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56017779ce30 0x56017779f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.382+0000 7ff9401c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x560177707280 con 0x560177797cc0 2024-03-11T19:17:27.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.382+0000 7ff9401c3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601777047b0 con 0x56017779ce30 2024-03-11T19:17:27.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.382+0000 7ff9401c3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601776fff90 con 0x5601777972d0 2024-03-11T19:17:27.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.383+0000 7ff9367fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56017779ce30 0x56017779f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.383+0000 7ff9377fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601777972d0 0x5601777976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.383+0000 7ff9367fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56017779ce30 0x56017779f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49870/0 (socket says 172.21.15.29:49870) 2024-03-11T19:17:27.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.383+0000 7ff9367fc640 1 -- 172.21.15.29:0/4150550639 learned_addr learned my addr 172.21.15.29:0/4150550639 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:27.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.383+0000 7ff91ffff640 1 -- 172.21.15.29:0/2109999173 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff92803b040 con 0x7ff9381083f0 2024-03-11T19:17:27.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.384+0000 7ff9367fc640 1 -- 172.21.15.29:0/4150550639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601777972d0 msgr2=0x5601777976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:27.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.384+0000 7ff9367fc640 1 --2- 172.21.15.29:0/4150550639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601777972d0 0x5601777976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.384+0000 7ff9367fc640 1 -- 172.21.15.29:0/4150550639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560177797cc0 msgr2=0x56017779c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:27.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.384+0000 7ff9367fc640 1 --2- 172.21.15.29:0/4150550639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560177797cc0 0x56017779c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.384+0000 7ff9367fc640 1 -- 172.21.15.29:0/4150550639 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601776be570 con 0x56017779ce30 2024-03-11T19:17:27.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.384+0000 7ff91ffff640 1 --2- 172.21.15.29:0/2109999173 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff9200a1c30 0x7ff9200a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.386+0000 7ff91ffff640 1 -- 172.21.15.29:0/2109999173 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff9280efc20 con 0x7ff9381083f0 2024-03-11T19:17:27.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.387+0000 7ff9367fc640 1 --2- 172.21.15.29:0/4150550639 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56017779ce30 0x56017779f2c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff92c00ad10 tx=0x7ff92c00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93acbeaa15827e29 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:27.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.387+0000 7ff9377fe640 1 --2- 172.21.15.29:0/2109999173 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff9200a1c30 0x7ff9200a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.388+0000 7ff91d7fa640 1 -- 172.21.15.29:0/4150550639 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c004020 con 0x56017779ce30 2024-03-11T19:17:27.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.388+0000 7ff91d7fa640 1 -- 172.21.15.29:0/4150550639 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff92c010640 con 0x56017779ce30 2024-03-11T19:17:27.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.388+0000 7ff91d7fa640 1 -- 172.21.15.29:0/4150550639 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c003830 con 0x56017779ce30 2024-03-11T19:17:27.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.389+0000 7ff9401c3740 1 -- 172.21.15.29:0/4150550639 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56017779ce30 msgr2=0x56017779f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:27.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.389+0000 7ff9377fe640 1 --2- 172.21.15.29:0/2109999173 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff9200a1c30 0x7ff9200a40f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x560177798520 tx=0x7ff930005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:27.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.389+0000 7ff9401c3740 1 --2- 172.21.15.29:0/4150550639 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56017779ce30 0x56017779f2c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff92c00ad10 tx=0x7ff92c00f790 comp rx=0 tx=0).stop 2024-03-11T19:17:27.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.389+0000 7ff9401c3740 1 -- 172.21.15.29:0/4150550639 shutdown_connections 2024-03-11T19:17:27.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.389+0000 7ff9401c3740 1 --2- 172.21.15.29:0/4150550639 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56017779ce30 0x56017779f2c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.389+0000 7ff9401c3740 1 --2- 172.21.15.29:0/4150550639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560177797cc0 0x56017779c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.389+0000 7ff9401c3740 1 --2- 172.21.15.29:0/4150550639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601777972d0 0x5601777976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.389+0000 7ff9401c3740 1 -- 172.21.15.29:0/4150550639 >> 172.21.15.29:0/4150550639 conn(0x5601777927f0 msgr2=0x560177792bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:27.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.389+0000 7ff9401c3740 1 -- 172.21.15.29:0/4150550639 shutdown_connections 2024-03-11T19:17:27.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.389+0000 7ff9401c3740 1 -- 172.21.15.29:0/4150550639 wait complete. 2024-03-11T19:17:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.390+0000 7ff9401c3740 1 -- start start 2024-03-11T19:17:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.390+0000 7ff9401c3740 1 Processor -- start 2024-03-11T19:17:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.390+0000 7ff9401c3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601777972d0 0x5601777a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.390+0000 7ff9401c3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560177797cc0 0x5601777a3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.390+0000 7ff9401c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56017779ce30 0x5601777aab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.390+0000 7ff9401c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601773d3430 con 0x56017779ce30 2024-03-11T19:17:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.390+0000 7ff9401c3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56017738fe00 con 0x560177797cc0 2024-03-11T19:17:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.390+0000 7ff9401c3740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560177684fd0 con 0x5601777972d0 2024-03-11T19:17:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.390+0000 7ff9401c3740 10 client.0 _ll_register_callbacks cb 0x5601776dafd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:17:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.391+0000 7ff9367fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560177797cc0 0x5601777a3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.391+0000 7ff9377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56017779ce30 0x5601777aab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.391+0000 7ff9367fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560177797cc0 0x5601777a3b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49886/0 (socket says 172.21.15.29:49886) 2024-03-11T19:17:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.391+0000 7ff9377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56017779ce30 0x5601777aab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42148/0 (socket says 172.21.15.29:42148) 2024-03-11T19:17:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.391+0000 7ff9367fc640 1 -- 172.21.15.29:0/3819752558 learned_addr learned my addr 172.21.15.29:0/3819752558 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.391+0000 7ff936ffd640 1 --2- 172.21.15.29:0/3819752558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601777972d0 0x5601777a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.391+0000 7ff9367fc640 1 -- 172.21.15.29:0/3819752558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601777972d0 msgr2=0x5601777a35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.391+0000 7ff9367fc640 1 --2- 172.21.15.29:0/3819752558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601777972d0 0x5601777a35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.391+0000 7ff9367fc640 1 -- 172.21.15.29:0/3819752558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56017779ce30 msgr2=0x5601777aab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.391+0000 7ff9367fc640 1 --2- 172.21.15.29:0/3819752558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56017779ce30 0x5601777aab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.391+0000 7ff9367fc640 1 -- 172.21.15.29:0/3819752558 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92c00a970 con 0x560177797cc0 2024-03-11T19:17:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.391+0000 7ff9367fc640 1 --2- 172.21.15.29:0/3819752558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560177797cc0 0x5601777a3b10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff92c01a040 tx=0x7ff92c005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=359de3becb2f4c82 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.391+0000 7ff91aff5640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:17:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.392+0000 7ff91aff5640 1 -- 172.21.15.29:0/3819752558 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c005560 con 0x560177797cc0 2024-03-11T19:17:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.392+0000 7ff9401c3740 1 -- 172.21.15.29:0/3819752558 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5601773ff7e0 con 0x560177797cc0 2024-03-11T19:17:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.392+0000 7ff91aff5640 1 -- 172.21.15.29:0/3819752558 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff92c0046c0 con 0x560177797cc0 2024-03-11T19:17:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.392+0000 7ff91aff5640 1 -- 172.21.15.29:0/3819752558 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff92c0048a0 con 0x560177797cc0 2024-03-11T19:17:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.392+0000 7ff9401c3740 10 client.26687 fetch_fsmap learned FSMap version 27 2024-03-11T19:17:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.392+0000 7ff9401c3740 1 -- 172.21.15.29:0/3819752558 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5601776fe610 con 0x560177797cc0 2024-03-11T19:17:27.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.392+0000 7ff91aff5640 1 -- 172.21.15.29:0/3819752558 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5601773ff7e0 con 0x560177797cc0 2024-03-11T19:17:27.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.392+0000 7ff91aff5640 1 -- 172.21.15.29:0/3819752558 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff92c020040 con 0x560177797cc0 2024-03-11T19:17:27.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.392+0000 7ff9401c3740 10 client.26687 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:17:27.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.392+0000 7ff9401c3740 10 client.26687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:17:27.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.392+0000 7ff9401c3740 20 client.26687 populate_metadata read hostname 'smithi029' 2024-03-11T19:17:27.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.392+0000 7ff9401c3740 1 --2- 172.21.15.29:0/3819752558 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5601777af670 0x5601777b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.393+0000 7ff9401c3740 4 client.26687 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:17:27.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.393+0000 7ff9401c3740 1 -- 172.21.15.29:0/3819752558 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56017743c720 con 0x5601777af670 2024-03-11T19:17:27.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.393+0000 7ff936ffd640 1 --2- 172.21.15.29:0/3819752558 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5601777af670 0x5601777b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.394+0000 7ff936ffd640 1 --2- 172.21.15.29:0/3819752558 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5601777af670 0x5601777b1a90 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab6caadaa8d9efb0 server_cookie=1dcf286d9f925a07 in_seq=0 out_seq=0 2024-03-11T19:17:27.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.394+0000 7ff91aff5640 10 client.26687 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:17:27.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.394+0000 7ff91aff5640 1 -- 172.21.15.29:0/3819752558 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56017743c720 con 0x5601777af670 2024-03-11T19:17:27.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.394+0000 7ff91aff5640 10 client.26687 handle_command_reply: tid=0 2024-03-11T19:17:27.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.394+0000 7ff9401c3740 1 client.26687 shutdown 2024-03-11T19:17:27.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.394+0000 7ff9401c3740 1 -- 172.21.15.29:0/3819752558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560177797cc0 msgr2=0x5601777a3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:27.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.394+0000 7ff9401c3740 1 --2- 172.21.15.29:0/3819752558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560177797cc0 0x5601777a3b10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff92c01a040 tx=0x7ff92c005270 comp rx=0 tx=0).stop 2024-03-11T19:17:27.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.395+0000 7ff9401c3740 1 -- 172.21.15.29:0/3819752558 shutdown_connections 2024-03-11T19:17:27.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.395+0000 7ff9401c3740 1 --2- 172.21.15.29:0/3819752558 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5601777af670 0x5601777b1a90 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.395+0000 7ff9401c3740 1 --2- 172.21.15.29:0/3819752558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56017779ce30 0x5601777aab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.395+0000 7ff9401c3740 1 --2- 172.21.15.29:0/3819752558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560177797cc0 0x5601777a3b10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.395+0000 7ff9401c3740 1 --2- 172.21.15.29:0/3819752558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601777972d0 0x5601777a35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.395+0000 7ff9401c3740 1 -- 172.21.15.29:0/3819752558 >> 172.21.15.29:0/3819752558 conn(0x5601777927f0 msgr2=0x560177791a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:27.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.395+0000 7ff91aff5640 0 client.26687 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:17:27.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.395+0000 7ff9401c3740 1 -- 172.21.15.29:0/3819752558 shutdown_connections 2024-03-11T19:17:27.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.395+0000 7ff9401c3740 1 -- 172.21.15.29:0/3819752558 wait complete. 2024-03-11T19:17:27.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.395+0000 7ff9401c3740 20 client.26687 trim_cache size 0 max 16384 2024-03-11T19:17:27.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:27 smithi029 ceph-mon[32439]: pgmap v580: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 67 MiB/s wr, 1.15k op/s 2024-03-11T19:17:27.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.424+0000 7ff9401c3740 1 Processor -- start 2024-03-11T19:17:27.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.426+0000 7ff9401c3740 1 -- start start 2024-03-11T19:17:27.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.426+0000 7ff9401c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5601777972d0 0x56017779f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.426+0000 7ff9401c3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560177797cc0 0x5601777aab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.426+0000 7ff9401c3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56017779ce30 0x5601777a3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.426+0000 7ff9401c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56017743c720 con 0x5601777972d0 2024-03-11T19:17:27.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.426+0000 7ff936ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5601777972d0 0x56017779f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.426+0000 7ff936ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5601777972d0 0x56017779f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42154/0 (socket says 172.21.15.29:42154) 2024-03-11T19:17:27.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.426+0000 7ff936ffd640 1 -- 172.21.15.29:0/484533468 learned_addr learned my addr 172.21.15.29:0/484533468 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:27.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.426+0000 7ff9401c3740 1 -- 172.21.15.29:0/484533468 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601773ff7e0 con 0x56017779ce30 2024-03-11T19:17:27.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.426+0000 7ff9401c3740 1 -- 172.21.15.29:0/484533468 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601777980f0 con 0x560177797cc0 2024-03-11T19:17:27.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.427+0000 7ff9377fe640 1 --2- 172.21.15.29:0/484533468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56017779ce30 0x5601777a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.427+0000 7ff9367fc640 1 --2- 172.21.15.29:0/484533468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560177797cc0 0x5601777aab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.427+0000 7ff936ffd640 1 -- 172.21.15.29:0/484533468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560177797cc0 msgr2=0x5601777aab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:27.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.427+0000 7ff936ffd640 1 --2- 172.21.15.29:0/484533468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560177797cc0 0x5601777aab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.427+0000 7ff936ffd640 1 -- 172.21.15.29:0/484533468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56017779ce30 msgr2=0x5601777a3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:27.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.427+0000 7ff936ffd640 1 --2- 172.21.15.29:0/484533468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56017779ce30 0x5601777a3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.427+0000 7ff936ffd640 1 -- 172.21.15.29:0/484533468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56017769aac0 con 0x5601777972d0 2024-03-11T19:17:27.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.427+0000 7ff936ffd640 1 --2- 172.21.15.29:0/484533468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5601777972d0 0x56017779f220 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff928006010 tx=0x7ff92802c100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b326ea48b3259769 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:27.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.428+0000 7ff91cff9640 1 -- 172.21.15.29:0/484533468 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9280bd290 con 0x5601777972d0 2024-03-11T19:17:27.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.428+0000 7ff91cff9640 1 -- 172.21.15.29:0/484533468 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff9280bcb90 con 0x5601777972d0 2024-03-11T19:17:27.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.428+0000 7ff91cff9640 1 -- 172.21.15.29:0/484533468 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff928004040 con 0x5601777972d0 2024-03-11T19:17:27.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.429+0000 7ff9401c3740 1 -- 172.21.15.29:0/484533468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5601777972d0 msgr2=0x56017779f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:27.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.429+0000 7ff9401c3740 1 --2- 172.21.15.29:0/484533468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5601777972d0 0x56017779f220 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff928006010 tx=0x7ff92802c100 comp rx=0 tx=0).stop 2024-03-11T19:17:27.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.429+0000 7ff9401c3740 1 -- 172.21.15.29:0/484533468 shutdown_connections 2024-03-11T19:17:27.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.429+0000 7ff9401c3740 1 --2- 172.21.15.29:0/484533468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56017779ce30 0x5601777a3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.429+0000 7ff9401c3740 1 --2- 172.21.15.29:0/484533468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560177797cc0 0x5601777aab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.429+0000 7ff9401c3740 1 --2- 172.21.15.29:0/484533468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5601777972d0 0x56017779f220 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.429+0000 7ff9401c3740 1 -- 172.21.15.29:0/484533468 >> 172.21.15.29:0/484533468 conn(0x5601777927f0 msgr2=0x5601777b63f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:27.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.429+0000 7ff9401c3740 1 -- 172.21.15.29:0/484533468 shutdown_connections 2024-03-11T19:17:27.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.429+0000 7ff9401c3740 1 -- 172.21.15.29:0/484533468 wait complete. 2024-03-11T19:17:27.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.430+0000 7ff9401c3740 1 -- start start 2024-03-11T19:17:27.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.430+0000 7ff9401c3740 1 Processor -- start 2024-03-11T19:17:27.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.430+0000 7ff9401c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5601777972d0 0x5601777a6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.431+0000 7ff9401c3740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560177797cc0 0x5601777a7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.431+0000 7ff936ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5601777972d0 0x5601777a6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.431+0000 7ff9401c3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56017779ce30 0x5601777aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.431+0000 7ff9401c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601777aa710 con 0x5601777972d0 2024-03-11T19:17:27.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.431+0000 7ff936ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5601777972d0 0x5601777a6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42160/0 (socket says 172.21.15.29:42160) 2024-03-11T19:17:27.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.431+0000 7ff936ffd640 1 -- 172.21.15.29:0/959520399 learned_addr learned my addr 172.21.15.29:0/959520399 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:27.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.431+0000 7ff9401c3740 1 -- 172.21.15.29:0/959520399 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601777aa880 con 0x56017779ce30 2024-03-11T19:17:27.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.431+0000 7ff9401c3740 1 -- 172.21.15.29:0/959520399 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601777aaa40 con 0x560177797cc0 2024-03-11T19:17:27.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.431+0000 7ff9367fc640 1 --2- 172.21.15.29:0/959520399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560177797cc0 0x5601777a7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.431+0000 7ff9401c3740 10 client.0 _ll_register_callbacks cb 0x7ff92c00a740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:17:27.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.431+0000 7ff936ffd640 1 -- 172.21.15.29:0/959520399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560177797cc0 msgr2=0x5601777a7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:27.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.431+0000 7ff936ffd640 1 --2- 172.21.15.29:0/959520399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560177797cc0 0x5601777a7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.431+0000 7ff936ffd640 1 -- 172.21.15.29:0/959520399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56017779ce30 msgr2=0x5601777aa1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:17:27.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.431+0000 7ff936ffd640 1 --2- 172.21.15.29:0/959520399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56017779ce30 0x5601777aa1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.431+0000 7ff936ffd640 1 -- 172.21.15.29:0/959520399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9381a5c60 con 0x5601777972d0 2024-03-11T19:17:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.432+0000 7ff936ffd640 1 --2- 172.21.15.29:0/959520399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5601777972d0 0x5601777a6ce0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff9280098d0 tx=0x7ff9280eff40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e65f69af6d4441 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.432+0000 7ff91aff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:17:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.432+0000 7ff91aff5640 1 -- 172.21.15.29:0/959520399 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9280bd290 con 0x5601777972d0 2024-03-11T19:17:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.432+0000 7ff91aff5640 1 -- 172.21.15.29:0/959520399 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff9280b9200 con 0x5601777972d0 2024-03-11T19:17:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.432+0000 7ff9401c3740 1 -- 172.21.15.29:0/959520399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff92c010640 con 0x5601777972d0 2024-03-11T19:17:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.432+0000 7ff91aff5640 1 -- 172.21.15.29:0/959520399 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9280b93c0 con 0x5601777972d0 2024-03-11T19:17:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.432+0000 7ff9401c3740 10 client.17193 fetch_fsmap learned FSMap version 27 2024-03-11T19:17:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.433+0000 7ff9401c3740 1 -- 172.21.15.29:0/959520399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5601776c06a0 con 0x5601777972d0 2024-03-11T19:17:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.433+0000 7ff91aff5640 1 -- 172.21.15.29:0/959520399 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff9280bcdf0 con 0x5601777972d0 2024-03-11T19:17:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.433+0000 7ff91aff5640 1 -- 172.21.15.29:0/959520399 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff9280bbfe0 con 0x5601777972d0 2024-03-11T19:17:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.433+0000 7ff9401c3740 10 client.17193 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:17:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.433+0000 7ff9401c3740 10 client.17193 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:17:27.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.433+0000 7ff9401c3740 20 client.17193 populate_metadata read hostname 'smithi029' 2024-03-11T19:17:27.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.433+0000 7ff9401c3740 1 --2- 172.21.15.29:0/959520399 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x560177804ab0 0x560177806ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.433+0000 7ff9401c3740 4 client.17193 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:17:27.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.433+0000 7ff9401c3740 1 -- 172.21.15.29:0/959520399 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7ff92c010640 con 0x560177804ab0 2024-03-11T19:17:27.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.434+0000 7ff9367fc640 1 --2- 172.21.15.29:0/959520399 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x560177804ab0 0x560177806ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.434+0000 7ff9367fc640 1 --2- 172.21.15.29:0/959520399 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x560177804ab0 0x560177806ed0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3988d6dba7816de9 server_cookie=2021a72082409527 in_seq=0 out_seq=0 2024-03-11T19:17:27.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.434+0000 7ff91aff5640 10 client.17193 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:17:27.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.435+0000 7ff91aff5640 1 -- 172.21.15.29:0/959520399 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7ff92c010640 con 0x560177804ab0 2024-03-11T19:17:27.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.435+0000 7ff91aff5640 10 client.17193 handle_command_reply: tid=0 2024-03-11T19:17:27.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.435+0000 7ff9401c3740 1 client.17193 shutdown 2024-03-11T19:17:27.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.435+0000 7ff9401c3740 1 -- 172.21.15.29:0/959520399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5601777972d0 msgr2=0x5601777a6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:27.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.435+0000 7ff9401c3740 1 --2- 172.21.15.29:0/959520399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5601777972d0 0x5601777a6ce0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff9280098d0 tx=0x7ff9280eff40 comp rx=0 tx=0).stop 2024-03-11T19:17:27.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.435+0000 7ff9401c3740 1 -- 172.21.15.29:0/959520399 shutdown_connections 2024-03-11T19:17:27.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.435+0000 7ff9401c3740 1 --2- 172.21.15.29:0/959520399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56017779ce30 0x5601777aa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.435+0000 7ff9401c3740 1 --2- 172.21.15.29:0/959520399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560177797cc0 0x5601777a7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.435+0000 7ff9401c3740 1 --2- 172.21.15.29:0/959520399 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x560177804ab0 0x560177806ed0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.435+0000 7ff91aff5640 0 client.17193 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:17:27.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.435+0000 7ff9401c3740 1 --2- 172.21.15.29:0/959520399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5601777972d0 0x5601777a6ce0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.435+0000 7ff9401c3740 1 -- 172.21.15.29:0/959520399 >> 172.21.15.29:0/959520399 conn(0x5601777927f0 msgr2=0x560177799040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:27.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.436+0000 7ff9401c3740 1 -- 172.21.15.29:0/959520399 shutdown_connections 2024-03-11T19:17:27.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.436+0000 7ff9401c3740 1 -- 172.21.15.29:0/959520399 wait complete. 2024-03-11T19:17:27.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.436+0000 7ff9401c3740 20 client.17193 trim_cache size 0 max 16384 2024-03-11T19:17:27.443 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:17:27.443 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:17:27.443 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:17:27.443 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:17:27.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.437+0000 7ff93d50c640 1 -- 172.21.15.29:0/2109999173 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff9200a1c30 msgr2=0x7ff9200a40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:27.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.437+0000 7ff93d50c640 1 --2- 172.21.15.29:0/2109999173 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff9200a1c30 0x7ff9200a40f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x560177798520 tx=0x7ff930005f90 comp rx=0 tx=0).stop 2024-03-11T19:17:27.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.437+0000 7ff93d50c640 1 -- 172.21.15.29:0/2109999173 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9381083f0 msgr2=0x7ff93819d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:27.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.437+0000 7ff93d50c640 1 --2- 172.21.15.29:0/2109999173 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9381083f0 0x7ff93819d710 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff928009bf0 tx=0x7ff928032920 comp rx=0 tx=0).stop 2024-03-11T19:17:27.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.437+0000 7ff93d50c640 1 -- 172.21.15.29:0/2109999173 shutdown_connections 2024-03-11T19:17:27.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.437+0000 7ff93d50c640 1 --2- 172.21.15.29:0/2109999173 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff9200a1c30 0x7ff9200a40f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.437+0000 7ff93d50c640 1 --2- 172.21.15.29:0/2109999173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff93810d830 0x7ff9381a4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.437+0000 7ff93d50c640 1 --2- 172.21.15.29:0/2109999173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff93810adb0 0x7ff93819dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.437+0000 7ff93d50c640 1 --2- 172.21.15.29:0/2109999173 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9381083f0 0x7ff93819d710 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.437+0000 7ff93d50c640 1 -- 172.21.15.29:0/2109999173 >> 172.21.15.29:0/2109999173 conn(0x7ff93806e360 msgr2=0x7ff938108c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:27.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.437+0000 7ff93d50c640 1 -- 172.21.15.29:0/2109999173 shutdown_connections 2024-03-11T19:17:27.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.437+0000 7ff93d50c640 1 -- 172.21.15.29:0/2109999173 wait complete. 2024-03-11T19:17:27.460 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:17:27.460 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:17:27.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:27 smithi190 ceph-mon[36292]: pgmap v580: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 67 MiB/s wr, 1.15k op/s 2024-03-11T19:17:27.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.641+0000 7fd63d927640 1 -- 172.21.15.29:0/2975820476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6380ff3d0 msgr2=0x7fd6380ff7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:27.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.641+0000 7fd63d927640 1 --2- 172.21.15.29:0/2975820476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6380ff3d0 0x7fd6380ff7f0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fd62c009900 tx=0x7fd62c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:17:27.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.642+0000 7fd63d927640 1 -- 172.21.15.29:0/2975820476 shutdown_connections 2024-03-11T19:17:27.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.642+0000 7fd63d927640 1 --2- 172.21.15.29:0/2975820476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd638100f80 0x7fd638107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.642+0000 7fd63d927640 1 --2- 172.21.15.29:0/2975820476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6381005c0 0x7fd638100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.642+0000 7fd63d927640 1 --2- 172.21.15.29:0/2975820476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6380ff3d0 0x7fd6380ff7f0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.642+0000 7fd63d927640 1 -- 172.21.15.29:0/2975820476 >> 172.21.15.29:0/2975820476 conn(0x7fd6380fa980 msgr2=0x7fd6380fcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:27.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.642+0000 7fd63d927640 1 -- 172.21.15.29:0/2975820476 shutdown_connections 2024-03-11T19:17:27.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.642+0000 7fd63d927640 1 -- 172.21.15.29:0/2975820476 wait complete. 2024-03-11T19:17:27.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.643+0000 7fd63d927640 1 Processor -- start 2024-03-11T19:17:27.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.643+0000 7fd63d927640 1 -- start start 2024-03-11T19:17:27.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.643+0000 7fd63d927640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6380ff3d0 0x7fd6381a0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.643+0000 7fd63d927640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6381005c0 0x7fd6381a27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.643+0000 7fd63d927640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd638100f80 0x7fd6381a2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.643+0000 7fd63d927640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6381a3260 con 0x7fd6381005c0 2024-03-11T19:17:27.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.643+0000 7fd63d927640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6381a33d0 con 0x7fd6380ff3d0 2024-03-11T19:17:27.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.643+0000 7fd63d927640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6381a3590 con 0x7fd638100f80 2024-03-11T19:17:27.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.644+0000 7fd636ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6380ff3d0 0x7fd6381a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.644+0000 7fd636ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6380ff3d0 0x7fd6381a0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49926/0 (socket says 172.21.15.29:49926) 2024-03-11T19:17:27.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.644+0000 7fd636ffd640 1 -- 172.21.15.29:0/1015484871 learned_addr learned my addr 172.21.15.29:0/1015484871 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:27.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.644+0000 7fd6367fc640 1 --2- 172.21.15.29:0/1015484871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6381005c0 0x7fd6381a27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.644+0000 7fd6377fe640 1 --2- 172.21.15.29:0/1015484871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd638100f80 0x7fd6381a2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.644+0000 7fd636ffd640 1 -- 172.21.15.29:0/1015484871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd638100f80 msgr2=0x7fd6381a2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:27.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.644+0000 7fd636ffd640 1 --2- 172.21.15.29:0/1015484871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd638100f80 0x7fd6381a2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.644+0000 7fd636ffd640 1 -- 172.21.15.29:0/1015484871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6381005c0 msgr2=0x7fd6381a27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:27.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.644+0000 7fd636ffd640 1 --2- 172.21.15.29:0/1015484871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6381005c0 0x7fd6381a27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:27.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.644+0000 7fd636ffd640 1 -- 172.21.15.29:0/1015484871 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd62c009560 con 0x7fd6380ff3d0 2024-03-11T19:17:27.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.644+0000 7fd636ffd640 1 --2- 172.21.15.29:0/1015484871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6380ff3d0 0x7fd6381a0290 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd62c02c240 tx=0x7fd62c0045c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:27.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.645+0000 7fd63c925640 1 -- 172.21.15.29:0/1015484871 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c003a00 con 0x7fd6380ff3d0 2024-03-11T19:17:27.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.645+0000 7fd63d927640 1 -- 172.21.15.29:0/1015484871 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6381a3810 con 0x7fd6380ff3d0 2024-03-11T19:17:27.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.645+0000 7fd63d927640 1 -- 172.21.15.29:0/1015484871 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6381a3d10 con 0x7fd6380ff3d0 2024-03-11T19:17:27.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.645+0000 7fd63c925640 1 -- 172.21.15.29:0/1015484871 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd62c003b60 con 0x7fd6380ff3d0 2024-03-11T19:17:27.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.645+0000 7fd63c925640 1 -- 172.21.15.29:0/1015484871 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c003a00 con 0x7fd6380ff3d0 2024-03-11T19:17:27.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.647+0000 7fd63d927640 1 -- 172.21.15.29:0/1015484871 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6240052c0 con 0x7fd6380ff3d0 2024-03-11T19:17:27.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.648+0000 7fd63c925640 1 -- 172.21.15.29:0/1015484871 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd62c003cd0 con 0x7fd6380ff3d0 2024-03-11T19:17:27.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.649+0000 7fd63c925640 1 --2- 172.21.15.29:0/1015484871 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd6200a1bd0 0x7fd6200a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:27.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.649+0000 7fd63c925640 1 -- 172.21.15.29:0/1015484871 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd62c0ef760 con 0x7fd6380ff3d0 2024-03-11T19:17:27.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.649+0000 7fd6367fc640 1 --2- 172.21.15.29:0/1015484871 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd6200a1bd0 0x7fd6200a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:27.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.649+0000 7fd6367fc640 1 --2- 172.21.15.29:0/1015484871 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd6200a1bd0 0x7fd6200a4090 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd638072eb0 tx=0x7fd62800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:27.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:27.656+0000 7fd63c925640 1 -- 172.21.15.29:0/1015484871 <== mon.2 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd62c0b8970 con 0x7fd6380ff3d0 2024-03-11T19:17:28.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.072+0000 7fd63d927640 1 -- 172.21.15.29:0/1015484871 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd624005af0 con 0x7fd6380ff3d0 2024-03-11T19:17:28.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.073+0000 7fd63c925640 1 -- 172.21.15.29:0/1015484871 <== mon.2 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7fd62c02cb30 con 0x7fd6380ff3d0 2024-03-11T19:17:28.078 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:17:28.078 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:17:28.078 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:17:28.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.076+0000 7fd63d927640 1 -- 172.21.15.29:0/1015484871 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd6200a1bd0 msgr2=0x7fd6200a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.076+0000 7fd63d927640 1 --2- 172.21.15.29:0/1015484871 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd6200a1bd0 0x7fd6200a4090 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd638072eb0 tx=0x7fd62800a2d0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.076+0000 7fd63d927640 1 -- 172.21.15.29:0/1015484871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6380ff3d0 msgr2=0x7fd6381a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.076+0000 7fd63d927640 1 --2- 172.21.15.29:0/1015484871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6380ff3d0 0x7fd6381a0290 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd62c02c240 tx=0x7fd62c0045c0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.076+0000 7fd63d927640 1 -- 172.21.15.29:0/1015484871 shutdown_connections 2024-03-11T19:17:28.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.076+0000 7fd63d927640 1 --2- 172.21.15.29:0/1015484871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd638100f80 0x7fd6381a2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.076+0000 7fd63d927640 1 --2- 172.21.15.29:0/1015484871 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd6200a1bd0 0x7fd6200a4090 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.076+0000 7fd63d927640 1 --2- 172.21.15.29:0/1015484871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6381005c0 0x7fd6381a27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.076+0000 7fd63d927640 1 --2- 172.21.15.29:0/1015484871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6380ff3d0 0x7fd6381a0290 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.076+0000 7fd63d927640 1 -- 172.21.15.29:0/1015484871 >> 172.21.15.29:0/1015484871 conn(0x7fd6380fa980 msgr2=0x7fd6381038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:28.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.077+0000 7fd63d927640 1 -- 172.21.15.29:0/1015484871 shutdown_connections 2024-03-11T19:17:28.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.077+0000 7fd63d927640 1 -- 172.21.15.29:0/1015484871 wait complete. 2024-03-11T19:17:28.102 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:17:28.102 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:17:28.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.277+0000 7f5ccc655640 1 -- 172.21.15.29:0/2796283867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cc410b870 msgr2=0x7f5cc410dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.277+0000 7f5ccc655640 1 --2- 172.21.15.29:0/2796283867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cc410b870 0x7f5cc410dca0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0009900 tx=0x7f5cc002bc30 comp rx=0 tx=0).stop 2024-03-11T19:17:28.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.277+0000 7f5ccc655640 1 -- 172.21.15.29:0/2796283867 shutdown_connections 2024-03-11T19:17:28.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.277+0000 7f5ccc655640 1 --2- 172.21.15.29:0/2796283867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cc410e330 0x7f5cc41107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.277+0000 7f5ccc655640 1 --2- 172.21.15.29:0/2796283867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cc410b870 0x7f5cc410dca0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.277+0000 7f5ccc655640 1 --2- 172.21.15.29:0/2796283867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cc4072f90 0x7f5cc410b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.277+0000 7f5ccc655640 1 -- 172.21.15.29:0/2796283867 >> 172.21.15.29:0/2796283867 conn(0x7f5cc406dfa0 msgr2=0x7f5cc40703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:28.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.277+0000 7f5ccc655640 1 -- 172.21.15.29:0/2796283867 shutdown_connections 2024-03-11T19:17:28.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.278+0000 7f5ccc655640 1 -- 172.21.15.29:0/2796283867 wait complete. 2024-03-11T19:17:28.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.279+0000 7f5ccc655640 1 Processor -- start 2024-03-11T19:17:28.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.279+0000 7f5ccc655640 1 -- start start 2024-03-11T19:17:28.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.279+0000 7f5ccc655640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cc4072f90 0x7f5cc419d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.279+0000 7f5ccc655640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cc410b870 0x7f5cc419db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.279+0000 7f5ccc655640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cc410e330 0x7f5cc41a4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.280+0000 7f5ccc655640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc41a5080 con 0x7f5cc410b870 2024-03-11T19:17:28.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.280+0000 7f5ccc655640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc41a51f0 con 0x7f5cc4072f90 2024-03-11T19:17:28.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.280+0000 7f5ccabf2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cc410e330 0x7f5cc41a4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.280+0000 7f5ccabf2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cc410e330 0x7f5cc41a4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:53826/0 (socket says 172.21.15.29:53826) 2024-03-11T19:17:28.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.280+0000 7f5ccabf2640 1 -- 172.21.15.29:0/1282177259 learned_addr learned my addr 172.21.15.29:0/1282177259 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:28.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.280+0000 7f5ccc655640 1 -- 172.21.15.29:0/1282177259 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc41a53b0 con 0x7f5cc410e330 2024-03-11T19:17:28.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.282+0000 7f5cc9bf0640 1 --2- 172.21.15.29:0/1282177259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cc410b870 0x7f5cc419db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.282+0000 7f5cc9bf0640 1 -- 172.21.15.29:0/1282177259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cc410e330 msgr2=0x7f5cc41a4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.282+0000 7f5cc9bf0640 1 --2- 172.21.15.29:0/1282177259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cc410e330 0x7f5cc41a4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.282+0000 7f5cc9bf0640 1 -- 172.21.15.29:0/1282177259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cc4072f90 msgr2=0x7f5cc419d5e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-11T19:17:28.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.282+0000 7f5cc9bf0640 1 --2- 172.21.15.29:0/1282177259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cc4072f90 0x7f5cc419d5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.282+0000 7f5cc9bf0640 1 -- 172.21.15.29:0/1282177259 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc0009560 con 0x7f5cc410b870 2024-03-11T19:17:28.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.283+0000 7f5cc9bf0640 1 --2- 172.21.15.29:0/1282177259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cc410b870 0x7f5cc419db20 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0036040 tx=0x7f5cc00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:28.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.283+0000 7f5cbb7fe640 1 -- 172.21.15.29:0/1282177259 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0004aa0 con 0x7f5cc410b870 2024-03-11T19:17:28.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.283+0000 7f5cbb7fe640 1 -- 172.21.15.29:0/1282177259 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5cc002de90 con 0x7f5cc410b870 2024-03-11T19:17:28.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.283+0000 7f5ccc655640 1 -- 172.21.15.29:0/1282177259 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc41a5630 con 0x7f5cc410b870 2024-03-11T19:17:28.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.283+0000 7f5cbb7fe640 1 -- 172.21.15.29:0/1282177259 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0005430 con 0x7f5cc410b870 2024-03-11T19:17:28.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.283+0000 7f5ccc655640 1 -- 172.21.15.29:0/1282177259 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc41a5b10 con 0x7f5cc410b870 2024-03-11T19:17:28.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.287+0000 7f5cbb7fe640 1 -- 172.21.15.29:0/1282177259 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5cc0005590 con 0x7f5cc410b870 2024-03-11T19:17:28.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.288+0000 7f5cbb7fe640 1 --2- 172.21.15.29:0/1282177259 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5cb00a1ab0 0x7f5cb00a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.288+0000 7f5cbb7fe640 1 -- 172.21.15.29:0/1282177259 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5cc00ef5a0 con 0x7f5cc410b870 2024-03-11T19:17:28.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.290+0000 7f5cca3f1640 1 --2- 172.21.15.29:0/1282177259 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5cb00a1ab0 0x7f5cb00a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.290+0000 7f5ccf221740 1 Processor -- start 2024-03-11T19:17:28.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.291+0000 7f5cca3f1640 1 --2- 172.21.15.29:0/1282177259 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5cb00a1ab0 0x7f5cb00a3f70 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4005fd0 tx=0x7f5cb4005c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:28.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.291+0000 7f5ccf221740 1 -- start start 2024-03-11T19:17:28.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.291+0000 7f5ccf221740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa4c260 0x55d55fa4c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.291+0000 7f5ccf221740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4cc50 0x55d55fa511e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.291+0000 7f5ccf221740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa51850 0x55d55fa53ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.291+0000 7f5ccf221740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d55f9ff960 con 0x55d55fa4cc50 2024-03-11T19:17:28.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.291+0000 7f5ccf221740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d55f9b3310 con 0x55d55fa51850 2024-03-11T19:17:28.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.291+0000 7f5ccf221740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d55f9b0840 con 0x55d55fa4c260 2024-03-11T19:17:28.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.291+0000 7f5ccabf2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa51850 0x55d55fa53ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.291+0000 7f5ccabf2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa51850 0x55d55fa53ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49954/0 (socket says 172.21.15.29:49954) 2024-03-11T19:17:28.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.291+0000 7f5ccabf2640 1 -- 172.21.15.29:0/3068660900 learned_addr learned my addr 172.21.15.29:0/3068660900 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:28.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.292+0000 7f5cca3f1640 1 --2- 172.21.15.29:0/3068660900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa4c260 0x55d55fa4c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.292+0000 7f5cc9bf0640 1 --2- 172.21.15.29:0/3068660900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4cc50 0x55d55fa511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.292+0000 7f5ccabf2640 1 -- 172.21.15.29:0/3068660900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa4c260 msgr2=0x55d55fa4c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.292+0000 7f5ccabf2640 1 --2- 172.21.15.29:0/3068660900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa4c260 0x55d55fa4c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.292+0000 7f5ccabf2640 1 -- 172.21.15.29:0/3068660900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4cc50 msgr2=0x55d55fa511e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.292+0000 7f5ccabf2640 1 --2- 172.21.15.29:0/3068660900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4cc50 0x55d55fa511e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.292+0000 7f5ccabf2640 1 -- 172.21.15.29:0/3068660900 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d55f96e060 con 0x55d55fa51850 2024-03-11T19:17:28.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.292+0000 7f5cc9bf0640 1 --2- 172.21.15.29:0/3068660900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4cc50 0x55d55fa511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:17:28.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.292+0000 7f5ccabf2640 1 --2- 172.21.15.29:0/3068660900 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa51850 0x55d55fa53ce0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00a350 tx=0x7f5cbc00b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d2fe28e30c83159 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:28.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.293+0000 7f5cca3f1640 1 --2- 172.21.15.29:0/3068660900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa4c260 0x55d55fa4c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:17:28.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.293+0000 7f5cb8bfe640 1 -- 172.21.15.29:0/3068660900 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc010050 con 0x55d55fa51850 2024-03-11T19:17:28.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.293+0000 7f5cb8bfe640 1 -- 172.21.15.29:0/3068660900 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5cbc011040 con 0x55d55fa51850 2024-03-11T19:17:28.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.293+0000 7f5cb8bfe640 1 -- 172.21.15.29:0/3068660900 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc014ad0 con 0x55d55fa51850 2024-03-11T19:17:28.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.294+0000 7f5ccf221740 1 -- 172.21.15.29:0/3068660900 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa51850 msgr2=0x55d55fa53ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.294+0000 7f5ccf221740 1 --2- 172.21.15.29:0/3068660900 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa51850 0x55d55fa53ce0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00a350 tx=0x7f5cbc00b710 comp rx=0 tx=0).stop 2024-03-11T19:17:28.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.294+0000 7f5ccf221740 1 -- 172.21.15.29:0/3068660900 shutdown_connections 2024-03-11T19:17:28.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.294+0000 7f5ccf221740 1 --2- 172.21.15.29:0/3068660900 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa51850 0x55d55fa53ce0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.294+0000 7f5ccf221740 1 --2- 172.21.15.29:0/3068660900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4cc50 0x55d55fa511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.294+0000 7f5ccf221740 1 --2- 172.21.15.29:0/3068660900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa4c260 0x55d55fa4c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.294+0000 7f5ccf221740 1 -- 172.21.15.29:0/3068660900 >> 172.21.15.29:0/3068660900 conn(0x55d55fa3e950 msgr2=0x55d55fa3ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:28.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.295+0000 7f5ccf221740 1 -- 172.21.15.29:0/3068660900 shutdown_connections 2024-03-11T19:17:28.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.295+0000 7f5ccf221740 1 -- 172.21.15.29:0/3068660900 wait complete. 2024-03-11T19:17:28.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.295+0000 7f5ccf221740 1 -- start start 2024-03-11T19:17:28.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.295+0000 7f5ccf221740 1 Processor -- start 2024-03-11T19:17:28.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.296+0000 7f5ccf221740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4c260 0x55d55fa55cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.296+0000 7f5ccf221740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa4cc50 0x55d55fa56230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.296+0000 7f5ccf221740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa51850 0x55d55fa5d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.296+0000 7f5cca3f1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4c260 0x55d55fa55cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.296+0000 7f5cca3f1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4c260 0x55d55fa55cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42204/0 (socket says 172.21.15.29:42204) 2024-03-11T19:17:28.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.296+0000 7f5cc9bf0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa4cc50 0x55d55fa56230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.296+0000 7f5cc9bf0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa4cc50 0x55d55fa56230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:53834/0 (socket says 172.21.15.29:53834) 2024-03-11T19:17:28.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.296+0000 7f5cca3f1640 1 -- 172.21.15.29:0/3833225405 learned_addr learned my addr 172.21.15.29:0/3833225405 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:28.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.296+0000 7f5ccf221740 1 -- 172.21.15.29:0/3833225405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d55f9ac020 con 0x55d55fa4c260 2024-03-11T19:17:28.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.297+0000 7f5ccf221740 1 -- 172.21.15.29:0/3833225405 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d55f67f430 con 0x55d55fa51850 2024-03-11T19:17:28.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.297+0000 7f5ccf221740 1 -- 172.21.15.29:0/3833225405 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d55f63be00 con 0x55d55fa4cc50 2024-03-11T19:17:28.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.297+0000 7f5ccf221740 10 client.0 _ll_register_callbacks cb 0x55d55f953cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:17:28.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.297+0000 7f5ccabf2640 1 --2- 172.21.15.29:0/3833225405 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa51850 0x55d55fa5d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.297+0000 7f5cc9bf0640 1 -- 172.21.15.29:0/3833225405 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa51850 msgr2=0x55d55fa5d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.297+0000 7f5cc9bf0640 1 --2- 172.21.15.29:0/3833225405 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa51850 0x55d55fa5d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.297+0000 7f5cc9bf0640 1 -- 172.21.15.29:0/3833225405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4c260 msgr2=0x55d55fa55cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.297+0000 7f5cc9bf0640 1 --2- 172.21.15.29:0/3833225405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4c260 0x55d55fa55cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.297+0000 7f5cc9bf0640 1 -- 172.21.15.29:0/3833225405 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cbc00bde0 con 0x55d55fa4cc50 2024-03-11T19:17:28.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.297+0000 7f5cc9bf0640 1 --2- 172.21.15.29:0/3833225405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa4cc50 0x55d55fa56230 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5cc00ed0a0 tx=0x7f5cc00bbe70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df071f311d09931 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:28.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.298+0000 7f5cadffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:17:28.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.298+0000 7f5cadffb640 1 -- 172.21.15.29:0/3833225405 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc00bcc10 con 0x55d55fa4cc50 2024-03-11T19:17:28.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.298+0000 7f5ccf221740 1 -- 172.21.15.29:0/3833225405 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d55f6ab7e0 con 0x55d55fa4cc50 2024-03-11T19:17:28.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.298+0000 7f5cadffb640 1 -- 172.21.15.29:0/3833225405 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5cc00bc510 con 0x55d55fa4cc50 2024-03-11T19:17:28.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.298+0000 7f5cadffb640 1 -- 172.21.15.29:0/3833225405 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5cc00c0660 con 0x55d55fa4cc50 2024-03-11T19:17:28.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.298+0000 7f5ccf221740 10 client.26731 fetch_fsmap learned FSMap version 27 2024-03-11T19:17:28.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.298+0000 7f5ccf221740 1 -- 172.21.15.29:0/3833225405 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d55f9aa6a0 con 0x55d55fa4cc50 2024-03-11T19:17:28.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.298+0000 7f5cadffb640 1 -- 172.21.15.29:0/3833225405 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55d55f6ab7e0 con 0x55d55fa4cc50 2024-03-11T19:17:28.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.299+0000 7f5cadffb640 1 -- 172.21.15.29:0/3833225405 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5cc003c070 con 0x55d55fa4cc50 2024-03-11T19:17:28.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.299+0000 7f5ccf221740 10 client.26731 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:17:28.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.299+0000 7f5ccf221740 10 client.26731 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:17:28.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.299+0000 7f5ccf221740 20 client.26731 populate_metadata read hostname 'smithi029' 2024-03-11T19:17:28.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.299+0000 7f5ccf221740 1 --2- 172.21.15.29:0/3833225405 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55d55fa61c50 0x55d55fa64040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.299+0000 7f5ccf221740 4 client.26731 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:17:28.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.299+0000 7f5ccf221740 1 -- 172.21.15.29:0/3833225405 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d55f6e8720 con 0x55d55fa61c50 2024-03-11T19:17:28.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.300+0000 7f5cca3f1640 1 --2- 172.21.15.29:0/3833225405 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55d55fa61c50 0x55d55fa64040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.300+0000 7f5cca3f1640 1 --2- 172.21.15.29:0/3833225405 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55d55fa61c50 0x55d55fa64040 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=77bc24ef634afecb server_cookie=499bed78119b72a in_seq=0 out_seq=0 2024-03-11T19:17:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.300+0000 7f5cadffb640 10 client.26731 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:17:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.301+0000 7f5cadffb640 1 -- 172.21.15.29:0/3833225405 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d55f6e8720 con 0x55d55fa61c50 2024-03-11T19:17:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.301+0000 7f5cadffb640 10 client.26731 handle_command_reply: tid=0 2024-03-11T19:17:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.301+0000 7f5ccf221740 1 client.26731 shutdown 2024-03-11T19:17:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.301+0000 7f5ccf221740 1 -- 172.21.15.29:0/3833225405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa4cc50 msgr2=0x55d55fa56230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.301+0000 7f5ccf221740 1 --2- 172.21.15.29:0/3833225405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa4cc50 0x55d55fa56230 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5cc00ed0a0 tx=0x7f5cc00bbe70 comp rx=0 tx=0).stop 2024-03-11T19:17:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.302+0000 7f5ccf221740 1 -- 172.21.15.29:0/3833225405 shutdown_connections 2024-03-11T19:17:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.302+0000 7f5ccf221740 1 --2- 172.21.15.29:0/3833225405 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55d55fa61c50 0x55d55fa64040 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.302+0000 7f5ccf221740 1 --2- 172.21.15.29:0/3833225405 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa51850 0x55d55fa5d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.302+0000 7f5ccf221740 1 --2- 172.21.15.29:0/3833225405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa4cc50 0x55d55fa56230 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.302+0000 7f5ccf221740 1 --2- 172.21.15.29:0/3833225405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4c260 0x55d55fa55cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.302+0000 7f5cadffb640 0 client.26731 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:17:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.302+0000 7f5ccf221740 1 -- 172.21.15.29:0/3833225405 >> 172.21.15.29:0/3833225405 conn(0x55d55fa3e950 msgr2=0x55d55fa4b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:28.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.302+0000 7f5ccf221740 1 -- 172.21.15.29:0/3833225405 shutdown_connections 2024-03-11T19:17:28.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.302+0000 7f5ccf221740 1 -- 172.21.15.29:0/3833225405 wait complete. 2024-03-11T19:17:28.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.302+0000 7f5ccf221740 20 client.26731 trim_cache size 0 max 16384 2024-03-11T19:17:28.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.321+0000 7f5ccf221740 1 Processor -- start 2024-03-11T19:17:28.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.321+0000 7f5ccf221740 1 -- start start 2024-03-11T19:17:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.321+0000 7f5ccf221740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa4c260 0x55d55fa68960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.321+0000 7f5ccf221740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4cc50 0x55d55fa5d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.323+0000 7f5cca3f1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa4c260 0x55d55fa68960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.323+0000 7f5cca3f1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa4c260 0x55d55fa68960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49972/0 (socket says 172.21.15.29:49972) 2024-03-11T19:17:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.323+0000 7f5cc9bf0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4cc50 0x55d55fa5d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.323+0000 7f5cc9bf0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4cc50 0x55d55fa5d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42220/0 (socket says 172.21.15.29:42220) 2024-03-11T19:17:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.323+0000 7f5cca3f1640 1 -- 172.21.15.29:0/2162057908 learned_addr learned my addr 172.21.15.29:0/2162057908 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.323+0000 7f5ccf221740 1 --2- 172.21.15.29:0/2162057908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa51850 0x55d55fa4da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.323+0000 7f5ccf221740 1 -- 172.21.15.29:0/2162057908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d55f931130 con 0x55d55fa4cc50 2024-03-11T19:17:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.323+0000 7f5ccf221740 1 -- 172.21.15.29:0/2162057908 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d55fa4d260 con 0x55d55fa4c260 2024-03-11T19:17:28.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.323+0000 7f5ccf221740 1 -- 172.21.15.29:0/2162057908 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d55fa3f3a0 con 0x55d55fa51850 2024-03-11T19:17:28.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.323+0000 7f5ccabf2640 1 --2- 172.21.15.29:0/2162057908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa51850 0x55d55fa4da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.323+0000 7f5cc9bf0640 1 -- 172.21.15.29:0/2162057908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa51850 msgr2=0x55d55fa4da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.323+0000 7f5cc9bf0640 1 --2- 172.21.15.29:0/2162057908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa51850 0x55d55fa4da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.323+0000 7f5cc9bf0640 1 -- 172.21.15.29:0/2162057908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa4c260 msgr2=0x55d55fa68960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.323+0000 7f5cc9bf0640 1 --2- 172.21.15.29:0/2162057908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa4c260 0x55d55fa68960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.323+0000 7f5cc9bf0640 1 -- 172.21.15.29:0/2162057908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d55f946c20 con 0x55d55fa4cc50 2024-03-11T19:17:28.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.324+0000 7f5cc9bf0640 1 --2- 172.21.15.29:0/2162057908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4cc50 0x55d55fa5d1c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f5cc00bc210 tx=0x7f5cc002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dda9ce85f7dab2a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:28.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.325+0000 7f5caffff640 1 -- 172.21.15.29:0/2162057908 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc00c4820 con 0x55d55fa4cc50 2024-03-11T19:17:28.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.325+0000 7f5caffff640 1 -- 172.21.15.29:0/2162057908 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5cc00bc9d0 con 0x55d55fa4cc50 2024-03-11T19:17:28.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.325+0000 7f5caffff640 1 -- 172.21.15.29:0/2162057908 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc00c4cb0 con 0x55d55fa4cc50 2024-03-11T19:17:28.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.326+0000 7f5ccf221740 1 -- 172.21.15.29:0/2162057908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4cc50 msgr2=0x55d55fa5d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.326+0000 7f5ccf221740 1 --2- 172.21.15.29:0/2162057908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4cc50 0x55d55fa5d1c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f5cc00bc210 tx=0x7f5cc002bc30 comp rx=0 tx=0).stop 2024-03-11T19:17:28.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.327+0000 7f5ccf221740 1 -- 172.21.15.29:0/2162057908 shutdown_connections 2024-03-11T19:17:28.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.327+0000 7f5ccf221740 1 --2- 172.21.15.29:0/2162057908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa51850 0x55d55fa4da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.327+0000 7f5ccf221740 1 --2- 172.21.15.29:0/2162057908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4cc50 0x55d55fa5d1c0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.327+0000 7f5ccf221740 1 --2- 172.21.15.29:0/2162057908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa4c260 0x55d55fa68960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.327+0000 7f5ccf221740 1 -- 172.21.15.29:0/2162057908 >> 172.21.15.29:0/2162057908 conn(0x55d55fa3e950 msgr2=0x55d55fa63c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:28.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.327+0000 7f5ccf221740 1 -- 172.21.15.29:0/2162057908 shutdown_connections 2024-03-11T19:17:28.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.327+0000 7f5ccf221740 1 -- 172.21.15.29:0/2162057908 wait complete. 2024-03-11T19:17:28.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.328+0000 7f5ccf221740 1 -- start start 2024-03-11T19:17:28.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.328+0000 7f5ccf221740 1 Processor -- start 2024-03-11T19:17:28.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.328+0000 7f5ccf221740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4c260 0x55d55fa5af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.328+0000 7f5ccf221740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa4cc50 0x55d55fa5b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.329+0000 7f5ccf221740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa51850 0x55d55fa5c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.329+0000 7f5ccf221740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d55fa5c980 con 0x55d55fa4c260 2024-03-11T19:17:28.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.329+0000 7f5ccf221740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d55fa5caf0 con 0x55d55fa51850 2024-03-11T19:17:28.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.329+0000 7f5ccf221740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d55fa5ccb0 con 0x55d55fa4cc50 2024-03-11T19:17:28.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.329+0000 7f5cca3f1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4c260 0x55d55fa5af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.329+0000 7f5ccf221740 10 client.0 _ll_register_callbacks cb 0x7f5cbc00bbf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:17:28.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.329+0000 7f5cca3f1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4c260 0x55d55fa5af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42234/0 (socket says 172.21.15.29:42234) 2024-03-11T19:17:28.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.329+0000 7f5cca3f1640 1 -- 172.21.15.29:0/2386703967 learned_addr learned my addr 172.21.15.29:0/2386703967 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.329+0000 7f5cc9bf0640 1 --2- 172.21.15.29:0/2386703967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa4cc50 0x55d55fa5b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.329+0000 7f5ccabf2640 1 --2- 172.21.15.29:0/2386703967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa51850 0x55d55fa5c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.329+0000 7f5cca3f1640 1 -- 172.21.15.29:0/2386703967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa4cc50 msgr2=0x55d55fa5b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.329+0000 7f5cca3f1640 1 --2- 172.21.15.29:0/2386703967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa4cc50 0x55d55fa5b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.329+0000 7f5cca3f1640 1 -- 172.21.15.29:0/2386703967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa51850 msgr2=0x55d55fa5c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.329+0000 7f5cca3f1640 1 --2- 172.21.15.29:0/2386703967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa51850 0x55d55fa5c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.329+0000 7f5cca3f1640 1 -- 172.21.15.29:0/2386703967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d55f9aa6a0 con 0x55d55fa4c260 2024-03-11T19:17:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.329+0000 7f5cc9bf0640 1 --2- 172.21.15.29:0/2386703967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa4cc50 0x55d55fa5b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:17:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.329+0000 7f5cca3f1640 1 --2- 172.21.15.29:0/2386703967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4c260 0x55d55fa5af60 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f5cb400dd00 tx=0x7f5cb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5011fd8a59f28244 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.330+0000 7f5cadffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:17:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.330+0000 7f5cadffb640 1 -- 172.21.15.29:0/2386703967 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb4025050 con 0x55d55fa4c260 2024-03-11T19:17:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.330+0000 7f5cadffb640 1 -- 172.21.15.29:0/2386703967 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5cb4027e40 con 0x55d55fa4c260 2024-03-11T19:17:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.330+0000 7f5ccf221740 1 -- 172.21.15.29:0/2386703967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5cbc011040 con 0x55d55fa4c260 2024-03-11T19:17:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.330+0000 7f5cadffb640 1 -- 172.21.15.29:0/2386703967 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5cb4018040 con 0x55d55fa4c260 2024-03-11T19:17:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.330+0000 7f5ccf221740 10 client.17217 fetch_fsmap learned FSMap version 27 2024-03-11T19:17:28.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.330+0000 7f5ccf221740 1 -- 172.21.15.29:0/2386703967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d55f96c8c0 con 0x55d55fa4c260 2024-03-11T19:17:28.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.330+0000 7f5cadffb640 1 -- 172.21.15.29:0/2386703967 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5cbc011040 con 0x55d55fa4c260 2024-03-11T19:17:28.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.330+0000 7f5cadffb640 1 -- 172.21.15.29:0/2386703967 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5cb4003710 con 0x55d55fa4c260 2024-03-11T19:17:28.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.331+0000 7f5ccf221740 10 client.17217 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:17:28.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.331+0000 7f5ccf221740 10 client.17217 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:17:28.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.331+0000 7f5ccf221740 20 client.17217 populate_metadata read hostname 'smithi029' 2024-03-11T19:17:28.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.331+0000 7f5ccf221740 1 --2- 172.21.15.29:0/2386703967 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55d55fab9070 0x55d55fabb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.331+0000 7f5ccf221740 4 client.17217 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:17:28.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.331+0000 7f5ccf221740 1 -- 172.21.15.29:0/2386703967 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d55f737890 con 0x55d55fab9070 2024-03-11T19:17:28.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.331+0000 7f5cc9bf0640 1 --2- 172.21.15.29:0/2386703967 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55d55fab9070 0x55d55fabb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.332+0000 7f5cadffb640 10 client.17217 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:17:28.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.332+0000 7f5cc9bf0640 1 --2- 172.21.15.29:0/2386703967 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55d55fab9070 0x55d55fabb490 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=20f545d789a41be6 server_cookie=e26dce382447fe95 in_seq=0 out_seq=0 2024-03-11T19:17:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.332+0000 7f5cadffb640 1 -- 172.21.15.29:0/2386703967 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d55f737890 con 0x55d55fab9070 2024-03-11T19:17:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.332+0000 7f5cadffb640 10 client.17217 handle_command_reply: tid=0 2024-03-11T19:17:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.332+0000 7f5ccf221740 1 client.17217 shutdown 2024-03-11T19:17:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.332+0000 7f5ccf221740 1 -- 172.21.15.29:0/2386703967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4c260 msgr2=0x55d55fa5af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.332+0000 7f5ccf221740 1 --2- 172.21.15.29:0/2386703967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4c260 0x55d55fa5af60 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f5cb400dd00 tx=0x7f5cb40046c0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.333+0000 7f5ccf221740 1 -- 172.21.15.29:0/2386703967 shutdown_connections 2024-03-11T19:17:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.333+0000 7f5ccf221740 1 --2- 172.21.15.29:0/2386703967 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55d55fab9070 0x55d55fabb490 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.333+0000 7f5ccf221740 1 --2- 172.21.15.29:0/2386703967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d55fa51850 0x55d55fa5c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.333+0000 7f5cadffb640 0 client.17217 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:17:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.333+0000 7f5ccf221740 1 --2- 172.21.15.29:0/2386703967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d55fa4cc50 0x55d55fa5b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.333+0000 7f5ccf221740 1 --2- 172.21.15.29:0/2386703967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d55fa4c260 0x55d55fa5af60 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.333+0000 7f5ccf221740 1 -- 172.21.15.29:0/2386703967 >> 172.21.15.29:0/2386703967 conn(0x55d55fa3e950 msgr2=0x55d55fa4da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.333+0000 7f5ccf221740 1 -- 172.21.15.29:0/2386703967 shutdown_connections 2024-03-11T19:17:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.333+0000 7f5ccf221740 1 -- 172.21.15.29:0/2386703967 wait complete. 2024-03-11T19:17:28.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.333+0000 7f5ccf221740 20 client.17217 trim_cache size 0 max 16384 2024-03-11T19:17:28.341 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:17:28.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.334+0000 7f5ccc655640 1 -- 172.21.15.29:0/1282177259 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5cb00a1ab0 msgr2=0x7f5cb00a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.334+0000 7f5ccc655640 1 --2- 172.21.15.29:0/1282177259 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5cb00a1ab0 0x7f5cb00a3f70 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4005fd0 tx=0x7f5cb4005c70 comp rx=0 tx=0).stop 2024-03-11T19:17:28.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.335+0000 7f5ccc655640 1 -- 172.21.15.29:0/1282177259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cc410b870 msgr2=0x7f5cc419db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.335+0000 7f5ccc655640 1 --2- 172.21.15.29:0/1282177259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cc410b870 0x7f5cc419db20 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0036040 tx=0x7f5cc00042f0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.335+0000 7f5ccc655640 1 -- 172.21.15.29:0/1282177259 shutdown_connections 2024-03-11T19:17:28.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.335+0000 7f5ccc655640 1 --2- 172.21.15.29:0/1282177259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5cc410e330 0x7f5cc41a4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.335+0000 7f5ccc655640 1 --2- 172.21.15.29:0/1282177259 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5cb00a1ab0 0x7f5cb00a3f70 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.335+0000 7f5ccc655640 1 --2- 172.21.15.29:0/1282177259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cc410b870 0x7f5cc419db20 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.335+0000 7f5ccc655640 1 --2- 172.21.15.29:0/1282177259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cc4072f90 0x7f5cc419d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.335+0000 7f5ccc655640 1 -- 172.21.15.29:0/1282177259 >> 172.21.15.29:0/1282177259 conn(0x7f5cc406dfa0 msgr2=0x7f5cc4109290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:28.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.335+0000 7f5ccc655640 1 -- 172.21.15.29:0/1282177259 shutdown_connections 2024-03-11T19:17:28.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.335+0000 7f5ccc655640 1 -- 172.21.15.29:0/1282177259 wait complete. 2024-03-11T19:17:28.361 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:17:28.361 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:17:28.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.538+0000 7f9ce9d9b640 1 -- 172.21.15.29:0/2345536728 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce40329f0 con 0x7f9cec10aaf0 2024-03-11T19:17:28.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.539+0000 7f9cf1023640 1 -- 172.21.15.29:0/2345536728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec10aaf0 msgr2=0x7f9cec10cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.539+0000 7f9cf1023640 1 --2- 172.21.15.29:0/2345536728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec10aaf0 0x7f9cec10cf60 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4009ca0 tx=0x7f9ce402c100 comp rx=0 tx=0).stop 2024-03-11T19:17:28.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.540+0000 7f9cf1023640 1 -- 172.21.15.29:0/2345536728 shutdown_connections 2024-03-11T19:17:28.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.540+0000 7f9cf1023640 1 --2- 172.21.15.29:0/2345536728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cec10d4a0 0x7f9cec10d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.540+0000 7f9cf1023640 1 --2- 172.21.15.29:0/2345536728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec10aaf0 0x7f9cec10cf60 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.540+0000 7f9cf1023640 1 --2- 172.21.15.29:0/2345536728 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cec108180 0x7f9cec10a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.540+0000 7f9cf1023640 1 -- 172.21.15.29:0/2345536728 >> 172.21.15.29:0/2345536728 conn(0x7f9cec06dfe0 msgr2=0x7f9cec070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:28.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.540+0000 7f9cf1023640 1 -- 172.21.15.29:0/2345536728 shutdown_connections 2024-03-11T19:17:28.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.540+0000 7f9cf1023640 1 -- 172.21.15.29:0/2345536728 wait complete. 2024-03-11T19:17:28.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.541+0000 7f9cf1023640 1 Processor -- start 2024-03-11T19:17:28.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.541+0000 7f9cf1023640 1 -- start start 2024-03-11T19:17:28.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.541+0000 7f9cf1023640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cec108180 0x7f9cec1a5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.541+0000 7f9cf1023640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cec10aaf0 0x7f9cec1a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.541+0000 7f9cf1023640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec10d4a0 0x7f9cec1ad550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.542+0000 7f9cf1023640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec1ada90 con 0x7f9cec10d4a0 2024-03-11T19:17:28.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.542+0000 7f9cf1023640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec1adc00 con 0x7f9cec10aaf0 2024-03-11T19:17:28.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.542+0000 7f9cead9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cec108180 0x7f9cec1a5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.542+0000 7f9cead9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cec108180 0x7f9cec1a5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:53880/0 (socket says 172.21.15.29:53880) 2024-03-11T19:17:28.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.542+0000 7f9cead9d640 1 -- 172.21.15.29:0/2934745265 learned_addr learned my addr 172.21.15.29:0/2934745265 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:28.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.542+0000 7f9cea59c640 1 --2- 172.21.15.29:0/2934745265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cec10aaf0 0x7f9cec1a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.542+0000 7f9cf1023640 1 -- 172.21.15.29:0/2934745265 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec1addc0 con 0x7f9cec108180 2024-03-11T19:17:28.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.542+0000 7f9ceb59e640 1 --2- 172.21.15.29:0/2934745265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec10d4a0 0x7f9cec1ad550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.543+0000 7f9cea59c640 1 -- 172.21.15.29:0/2934745265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cec108180 msgr2=0x7f9cec1a5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.543+0000 7f9cea59c640 1 --2- 172.21.15.29:0/2934745265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cec108180 0x7f9cec1a5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.543+0000 7f9cea59c640 1 -- 172.21.15.29:0/2934745265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec10d4a0 msgr2=0x7f9cec1ad550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.543+0000 7f9cea59c640 1 --2- 172.21.15.29:0/2934745265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec10d4a0 0x7f9cec1ad550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.543+0000 7f9cea59c640 1 -- 172.21.15.29:0/2934745265 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce4009900 con 0x7f9cec10aaf0 2024-03-11T19:17:28.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.544+0000 7f9cea59c640 1 --2- 172.21.15.29:0/2934745265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cec10aaf0 0x7f9cec1a6530 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9ce402c640 tx=0x7f9ce40329a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:28.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.545+0000 7f9cd3fff640 1 -- 172.21.15.29:0/2934745265 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce4034850 con 0x7f9cec10aaf0 2024-03-11T19:17:28.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.545+0000 7f9cd3fff640 1 -- 172.21.15.29:0/2934745265 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9ce4003e20 con 0x7f9cec10aaf0 2024-03-11T19:17:28.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.545+0000 7f9cd3fff640 1 -- 172.21.15.29:0/2934745265 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce4004870 con 0x7f9cec10aaf0 2024-03-11T19:17:28.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.546+0000 7f9cf1023640 1 -- 172.21.15.29:0/2934745265 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cec1ae040 con 0x7f9cec10aaf0 2024-03-11T19:17:28.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.546+0000 7f9cf1023640 1 -- 172.21.15.29:0/2934745265 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cec1ae4a0 con 0x7f9cec10aaf0 2024-03-11T19:17:28.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.549+0000 7f9cf3bf9740 1 Processor -- start 2024-03-11T19:17:28.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.549+0000 7f9cf3bf9740 1 -- start start 2024-03-11T19:17:28.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.550+0000 7f9cf3bf9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac056280 0x55c3ac0566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.550+0000 7f9cf3bf9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056c70 0x55c3ac05b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.550+0000 7f9cf3bf9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c3ac05bde0 0x55c3ac05e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.550+0000 7f9cf3bf9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3ac001960 con 0x55c3ac056280 2024-03-11T19:17:28.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.550+0000 7f9cf3bf9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3abfb5310 con 0x55c3ac05bde0 2024-03-11T19:17:28.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.550+0000 7f9cf3bf9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3abfb2840 con 0x55c3ac056c70 2024-03-11T19:17:28.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.550+0000 7f9ceb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac056280 0x55c3ac0566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.550+0000 7f9ceb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac056280 0x55c3ac0566a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42270/0 (socket says 172.21.15.29:42270) 2024-03-11T19:17:28.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.550+0000 7f9ceb59e640 1 -- 172.21.15.29:0/2046564093 learned_addr learned my addr 172.21.15.29:0/2046564093 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:28.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.551+0000 7f9cead9d640 1 --2- 172.21.15.29:0/2046564093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056c70 0x55c3ac05b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.552+0000 7f9cead9d640 1 -- 172.21.15.29:0/2046564093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c3ac05bde0 msgr2=0x55c3ac05e270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:17:28.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.552+0000 7f9cead9d640 1 --2- 172.21.15.29:0/2046564093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c3ac05bde0 0x55c3ac05e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.552+0000 7f9cead9d640 1 -- 172.21.15.29:0/2046564093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac056280 msgr2=0x55c3ac0566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.552+0000 7f9cead9d640 1 --2- 172.21.15.29:0/2046564093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac056280 0x55c3ac0566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.552+0000 7f9cead9d640 1 -- 172.21.15.29:0/2046564093 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3abf70060 con 0x55c3ac056c70 2024-03-11T19:17:28.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.552+0000 7f9cd3fff640 1 -- 172.21.15.29:0/2934745265 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9ce403b040 con 0x7f9cec10aaf0 2024-03-11T19:17:28.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.553+0000 7f9cd3fff640 1 --2- 172.21.15.29:0/2934745265 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9cd40a1c00 0x7f9cd40a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.553+0000 7f9cead9d640 1 --2- 172.21.15.29:0/2046564093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056c70 0x55c3ac05b770 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0009c80 tx=0x7f9ce0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98b4ea37382882ba server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:28.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.553+0000 7f9cd3fff640 1 -- 172.21.15.29:0/2934745265 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9ce40efcc0 con 0x7f9cec10aaf0 2024-03-11T19:17:28.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.553+0000 7f9cd17fa640 1 -- 172.21.15.29:0/2046564093 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0004230 con 0x55c3ac056c70 2024-03-11T19:17:28.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.553+0000 7f9cd17fa640 1 -- 172.21.15.29:0/2046564093 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9ce0004390 con 0x55c3ac056c70 2024-03-11T19:17:28.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.553+0000 7f9cd17fa640 1 -- 172.21.15.29:0/2046564093 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0004ce0 con 0x55c3ac056c70 2024-03-11T19:17:28.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.554+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/2046564093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056c70 msgr2=0x55c3ac05b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.554+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/2046564093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056c70 0x55c3ac05b770 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0009c80 tx=0x7f9ce0011810 comp rx=0 tx=0).stop 2024-03-11T19:17:28.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.555+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/2046564093 shutdown_connections 2024-03-11T19:17:28.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.555+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/2046564093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c3ac05bde0 0x55c3ac05e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.555+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/2046564093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056c70 0x55c3ac05b770 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.555+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/2046564093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac056280 0x55c3ac0566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.555+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/2046564093 >> 172.21.15.29:0/2046564093 conn(0x55c3ac0407e0 msgr2=0x55c3ac040be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.557+0000 7f9ceb59e640 1 --2- 172.21.15.29:0/2934745265 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9cd40a1c00 0x7f9cd40a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.557+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/2046564093 shutdown_connections 2024-03-11T19:17:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.558+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/2046564093 wait complete. 2024-03-11T19:17:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.558+0000 7f9cf3bf9740 1 -- start start 2024-03-11T19:17:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.558+0000 7f9ceb59e640 1 --2- 172.21.15.29:0/2934745265 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9cd40a1c00 0x7f9cd40a40c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x55c3ac0574d0 tx=0x7f9cdc00b2e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.558+0000 7f9cf3bf9740 1 Processor -- start 2024-03-11T19:17:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.558+0000 7f9cf3bf9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056280 0x55c3ac046940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.559+0000 7f9cf3bf9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c3ac056c70 0x55c3ac046e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.559+0000 7f9cf3bf9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac05bde0 0x55c3ac04dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.559+0000 7f9cf3bf9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3abfae020 con 0x55c3ac05bde0 2024-03-11T19:17:28.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.559+0000 7f9cf3bf9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3abc81430 con 0x55c3ac056c70 2024-03-11T19:17:28.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.559+0000 7f9cf3bf9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3abc3de00 con 0x55c3ac056280 2024-03-11T19:17:28.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.559+0000 7f9cf3bf9740 10 client.0 _ll_register_callbacks cb 0x55c3abf55cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:17:28.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.559+0000 7f9cead9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056280 0x55c3ac046940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.559+0000 7f9cead9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056280 0x55c3ac046940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:53894/0 (socket says 172.21.15.29:53894) 2024-03-11T19:17:28.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.559+0000 7f9cead9d640 1 -- 172.21.15.29:0/2900002754 learned_addr learned my addr 172.21.15.29:0/2900002754 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:28.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.559+0000 7f9ceb59e640 1 --2- 172.21.15.29:0/2900002754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac05bde0 0x55c3ac04dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.559+0000 7f9cea59c640 1 --2- 172.21.15.29:0/2900002754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c3ac056c70 0x55c3ac046e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.559+0000 7f9cead9d640 1 -- 172.21.15.29:0/2900002754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c3ac056c70 msgr2=0x55c3ac046e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.559+0000 7f9cead9d640 1 --2- 172.21.15.29:0/2900002754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c3ac056c70 0x55c3ac046e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.559+0000 7f9cead9d640 1 -- 172.21.15.29:0/2900002754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac05bde0 msgr2=0x55c3ac04dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.559+0000 7f9cead9d640 1 --2- 172.21.15.29:0/2900002754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac05bde0 0x55c3ac04dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.559+0000 7f9cead9d640 1 -- 172.21.15.29:0/2900002754 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce00098e0 con 0x55c3ac056280 2024-03-11T19:17:28.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.560+0000 7f9cead9d640 1 --2- 172.21.15.29:0/2900002754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056280 0x55c3ac046940 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0000c00 tx=0x7f9ce0019bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef9efccb9f54fd40 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:28.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.560+0000 7f9cceff5640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:17:28.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.560+0000 7f9cceff5640 1 -- 172.21.15.29:0/2900002754 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0024620 con 0x55c3ac056280 2024-03-11T19:17:28.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.560+0000 7f9cceff5640 1 -- 172.21.15.29:0/2900002754 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9ce0024780 con 0x55c3ac056280 2024-03-11T19:17:28.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.560+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/2900002754 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c3abcad7e0 con 0x55c3ac056280 2024-03-11T19:17:28.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.562+0000 7f9cceff5640 1 -- 172.21.15.29:0/2900002754 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ce0004500 con 0x55c3ac056280 2024-03-11T19:17:28.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.562+0000 7f9cf3bf9740 10 client.26749 fetch_fsmap learned FSMap version 27 2024-03-11T19:17:28.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.562+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/2900002754 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55c3abfac6a0 con 0x55c3ac056280 2024-03-11T19:17:28.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.562+0000 7f9cceff5640 1 -- 172.21.15.29:0/2900002754 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55c3abcad7e0 con 0x55c3ac056280 2024-03-11T19:17:28.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.562+0000 7f9cceff5640 1 -- 172.21.15.29:0/2900002754 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9ce0005060 con 0x55c3ac056280 2024-03-11T19:17:28.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.562+0000 7f9cf3bf9740 10 client.26749 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:17:28.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.562+0000 7f9cf3bf9740 10 client.26749 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:17:28.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.562+0000 7f9cf3bf9740 20 client.26749 populate_metadata read hostname 'smithi029' 2024-03-11T19:17:28.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.562+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/2900002754 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55c3ac0529b0 0x55c3ac054dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.562+0000 7f9cf3bf9740 4 client.26749 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:17:28.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.562+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/2900002754 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c3abcea720 con 0x55c3ac0529b0 2024-03-11T19:17:28.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.564+0000 7f9cea59c640 1 --2- 172.21.15.29:0/2900002754 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55c3ac0529b0 0x55c3ac054dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.564+0000 7f9cea59c640 1 --2- 172.21.15.29:0/2900002754 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55c3ac0529b0 0x55c3ac054dd0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=deb74e1f2dfa288c server_cookie=a3757e23df85559e in_seq=0 out_seq=0 2024-03-11T19:17:28.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.564+0000 7f9cceff5640 10 client.26749 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:17:28.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.565+0000 7f9cceff5640 1 -- 172.21.15.29:0/2900002754 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c3abcea720 con 0x55c3ac0529b0 2024-03-11T19:17:28.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.565+0000 7f9cceff5640 10 client.26749 handle_command_reply: tid=0 2024-03-11T19:17:28.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.565+0000 7f9cf3bf9740 1 client.26749 shutdown 2024-03-11T19:17:28.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.566+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/2900002754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056280 msgr2=0x55c3ac046940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.566+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/2900002754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056280 0x55c3ac046940 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0000c00 tx=0x7f9ce0019bb0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.566+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/2900002754 shutdown_connections 2024-03-11T19:17:28.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.566+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/2900002754 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55c3ac0529b0 0x55c3ac054dd0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.566+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/2900002754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac05bde0 0x55c3ac04dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.566+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/2900002754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c3ac056c70 0x55c3ac046e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.566+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/2900002754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056280 0x55c3ac046940 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.566+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/2900002754 >> 172.21.15.29:0/2900002754 conn(0x55c3ac0407e0 msgr2=0x55c3ac03fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:28.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.566+0000 7f9cceff5640 0 client.26749 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:17:28.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.566+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/2900002754 shutdown_connections 2024-03-11T19:17:28.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.567+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/2900002754 wait complete. 2024-03-11T19:17:28.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.567+0000 7f9cf3bf9740 20 client.26749 trim_cache size 0 max 16384 2024-03-11T19:17:28.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.589+0000 7f9cf3bf9740 1 Processor -- start 2024-03-11T19:17:28.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.589+0000 7f9cf3bf9740 1 -- start start 2024-03-11T19:17:28.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.589+0000 7f9cf3bf9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056280 0x55c3ac062b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.589+0000 7f9cf3bf9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac056c70 0x55c3ac063050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.589+0000 7f9cead9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056280 0x55c3ac062b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.589+0000 7f9cead9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056280 0x55c3ac062b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:53908/0 (socket says 172.21.15.29:53908) 2024-03-11T19:17:28.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.590+0000 7f9cea59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac056c70 0x55c3ac063050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.590+0000 7f9cf3bf9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c3ac05bde0 0x55c3ac03f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.590+0000 7f9cf3bf9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3abf33130 con 0x55c3ac056c70 2024-03-11T19:17:28.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.590+0000 7f9cf3bf9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3ac032a40 con 0x55c3ac05bde0 2024-03-11T19:17:28.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.590+0000 7f9cf3bf9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3ac05e930 con 0x55c3ac056280 2024-03-11T19:17:28.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.590+0000 7f9cea59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac056c70 0x55c3ac063050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42290/0 (socket says 172.21.15.29:42290) 2024-03-11T19:17:28.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.590+0000 7f9cea59c640 1 -- 172.21.15.29:0/553645137 learned_addr learned my addr 172.21.15.29:0/553645137 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:28.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.590+0000 7f9ceb59e640 1 --2- 172.21.15.29:0/553645137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c3ac05bde0 0x55c3ac03f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.590+0000 7f9cead9d640 1 -- 172.21.15.29:0/553645137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c3ac05bde0 msgr2=0x55c3ac03f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.590+0000 7f9cead9d640 1 --2- 172.21.15.29:0/553645137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c3ac05bde0 0x55c3ac03f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.590+0000 7f9cead9d640 1 -- 172.21.15.29:0/553645137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac056c70 msgr2=0x55c3ac063050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.590+0000 7f9cead9d640 1 --2- 172.21.15.29:0/553645137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac056c70 0x55c3ac063050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.590+0000 7f9cead9d640 1 -- 172.21.15.29:0/553645137 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3abf48c20 con 0x55c3ac056280 2024-03-11T19:17:28.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.590+0000 7f9cea59c640 1 --2- 172.21.15.29:0/553645137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac056c70 0x55c3ac063050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:17:28.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.590+0000 7f9cead9d640 1 --2- 172.21.15.29:0/553645137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056280 0x55c3ac062b10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9ce00053f0 tx=0x7f9ce002c600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26bd7ab72891c8ef server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:28.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.591+0000 7f9cd0ff9640 1 -- 172.21.15.29:0/553645137 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce002c8d0 con 0x55c3ac056280 2024-03-11T19:17:28.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.591+0000 7f9cd0ff9640 1 -- 172.21.15.29:0/553645137 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9ce00055e0 con 0x55c3ac056280 2024-03-11T19:17:28.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.591+0000 7f9cd0ff9640 1 -- 172.21.15.29:0/553645137 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce00247a0 con 0x55c3ac056280 2024-03-11T19:17:28.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.592+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/553645137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056280 msgr2=0x55c3ac062b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.592+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/553645137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056280 0x55c3ac062b10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9ce00053f0 tx=0x7f9ce002c600 comp rx=0 tx=0).stop 2024-03-11T19:17:28.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.592+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/553645137 shutdown_connections 2024-03-11T19:17:28.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.592+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/553645137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c3ac05bde0 0x55c3ac03f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.592+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/553645137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac056c70 0x55c3ac063050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.592+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/553645137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac056280 0x55c3ac062b10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.592+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/553645137 >> 172.21.15.29:0/553645137 conn(0x55c3ac0407e0 msgr2=0x55c3ac03ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:28.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.592+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/553645137 shutdown_connections 2024-03-11T19:17:28.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.593+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/553645137 wait complete. 2024-03-11T19:17:28.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.593+0000 7f9cf3bf9740 1 -- start start 2024-03-11T19:17:28.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.593+0000 7f9cf3bf9740 1 Processor -- start 2024-03-11T19:17:28.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.594+0000 7f9cf3bf9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c3ac056280 0x55c3ac0ac2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.594+0000 7f9cf3bf9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac056c70 0x55c3ac0ae800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.594+0000 7f9cf3bf9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac05bde0 0x55c3ac0b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.594+0000 7f9cea59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac056c70 0x55c3ac0ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.594+0000 7f9cea59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac056c70 0x55c3ac0ae800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42300/0 (socket says 172.21.15.29:42300) 2024-03-11T19:17:28.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.594+0000 7f9cea59c640 1 -- 172.21.15.29:0/173167673 learned_addr learned my addr 172.21.15.29:0/173167673 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:28.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.594+0000 7f9ceb59e640 1 --2- 172.21.15.29:0/173167673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac05bde0 0x55c3ac0b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.594+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/173167673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3ac0b12c0 con 0x55c3ac056c70 2024-03-11T19:17:28.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.595+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/173167673 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3ac0b1430 con 0x55c3ac056280 2024-03-11T19:17:28.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.595+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/173167673 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3ac0b15f0 con 0x55c3ac05bde0 2024-03-11T19:17:28.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.595+0000 7f9cf3bf9740 10 client.0 _ll_register_callbacks cb 0x7f9ce0009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:17:28.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.595+0000 7f9cead9d640 1 --2- 172.21.15.29:0/173167673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c3ac056280 0x55c3ac0ac2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.595+0000 7f9ceb59e640 1 -- 172.21.15.29:0/173167673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c3ac056280 msgr2=0x55c3ac0ac2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.595+0000 7f9ceb59e640 1 --2- 172.21.15.29:0/173167673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c3ac056280 0x55c3ac0ac2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.595+0000 7f9ceb59e640 1 -- 172.21.15.29:0/173167673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac056c70 msgr2=0x55c3ac0ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.595+0000 7f9ceb59e640 1 --2- 172.21.15.29:0/173167673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac056c70 0x55c3ac0ae800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.595+0000 7f9ceb59e640 1 -- 172.21.15.29:0/173167673 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c3abfac6a0 con 0x55c3ac05bde0 2024-03-11T19:17:28.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.596+0000 7f9ceb59e640 1 --2- 172.21.15.29:0/173167673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac05bde0 0x55c3ac0b0d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00bf70 tx=0x7f9cdc036040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cf41e70ea2657ab server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:28.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.596+0000 7f9cceff5640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:17:28.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.596+0000 7f9cceff5640 1 -- 172.21.15.29:0/173167673 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc044040 con 0x55c3ac05bde0 2024-03-11T19:17:28.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.596+0000 7f9cceff5640 1 -- 172.21.15.29:0/173167673 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9cdc03d660 con 0x55c3ac05bde0 2024-03-11T19:17:28.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.596+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/173167673 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9ce0004390 con 0x55c3ac05bde0 2024-03-11T19:17:28.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.597+0000 7f9cceff5640 1 -- 172.21.15.29:0/173167673 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9cdc03d820 con 0x55c3ac05bde0 2024-03-11T19:17:28.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.597+0000 7f9cf3bf9740 10 client.26755 fetch_fsmap learned FSMap version 27 2024-03-11T19:17:28.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.597+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/173167673 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55c3abf6e8c0 con 0x55c3ac05bde0 2024-03-11T19:17:28.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.597+0000 7f9cceff5640 1 -- 172.21.15.29:0/173167673 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9cdc03de30 con 0x55c3ac05bde0 2024-03-11T19:17:28.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.597+0000 7f9cceff5640 1 -- 172.21.15.29:0/173167673 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9cdc050070 con 0x55c3ac05bde0 2024-03-11T19:17:28.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.597+0000 7f9cf3bf9740 10 client.26755 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:17:28.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.597+0000 7f9cf3bf9740 10 client.26755 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:17:28.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.597+0000 7f9cf3bf9740 20 client.26755 populate_metadata read hostname 'smithi029' 2024-03-11T19:17:28.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.597+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/173167673 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55c3ac0b30e0 0x55c3ac0b5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.597+0000 7f9cf3bf9740 4 client.26755 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:17:28.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.597+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/173167673 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f9ce0004390 con 0x55c3ac0b30e0 2024-03-11T19:17:28.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.598+0000 7f9cead9d640 1 --2- 172.21.15.29:0/173167673 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55c3ac0b30e0 0x55c3ac0b5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.598+0000 7f9cceff5640 10 client.26755 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:17:28.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.598+0000 7f9cead9d640 1 --2- 172.21.15.29:0/173167673 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55c3ac0b30e0 0x55c3ac0b5500 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3bde593615ed6df7 server_cookie=3919a6cda1c65f6b in_seq=0 out_seq=0 2024-03-11T19:17:28.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.599+0000 7f9cceff5640 1 -- 172.21.15.29:0/173167673 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f9ce0004390 con 0x55c3ac0b30e0 2024-03-11T19:17:28.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.599+0000 7f9cceff5640 10 client.26755 handle_command_reply: tid=0 2024-03-11T19:17:28.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.599+0000 7f9cf3bf9740 1 client.26755 shutdown 2024-03-11T19:17:28.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.599+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/173167673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac05bde0 msgr2=0x55c3ac0b0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.599+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/173167673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac05bde0 0x55c3ac0b0d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00bf70 tx=0x7f9cdc036040 comp rx=0 tx=0).stop 2024-03-11T19:17:28.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.599+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/173167673 shutdown_connections 2024-03-11T19:17:28.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.599+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/173167673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c3ac05bde0 0x55c3ac0b0d50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.599+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/173167673 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55c3ac0b30e0 0x55c3ac0b5500 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.599+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/173167673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3ac056c70 0x55c3ac0ae800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.599+0000 7f9cf3bf9740 1 --2- 172.21.15.29:0/173167673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c3ac056280 0x55c3ac0ac2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.599+0000 7f9cceff5640 0 client.26755 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:17:28.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.599+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/173167673 >> 172.21.15.29:0/173167673 conn(0x55c3ac0407e0 msgr2=0x55c3ac043a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:28.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.599+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/173167673 shutdown_connections 2024-03-11T19:17:28.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.599+0000 7f9cf3bf9740 1 -- 172.21.15.29:0/173167673 wait complete. 2024-03-11T19:17:28.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.599+0000 7f9cf3bf9740 20 client.26755 trim_cache size 0 max 16384 2024-03-11T19:17:28.606 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:17:28.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.601+0000 7f9cf1023640 1 -- 172.21.15.29:0/2934745265 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9cd40a1c00 msgr2=0x7f9cd40a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.601+0000 7f9cf1023640 1 --2- 172.21.15.29:0/2934745265 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9cd40a1c00 0x7f9cd40a40c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x55c3ac0574d0 tx=0x7f9cdc00b2e0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.601+0000 7f9cf1023640 1 -- 172.21.15.29:0/2934745265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cec10aaf0 msgr2=0x7f9cec1a6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.601+0000 7f9cf1023640 1 --2- 172.21.15.29:0/2934745265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cec10aaf0 0x7f9cec1a6530 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9ce402c640 tx=0x7f9ce40329a0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.601+0000 7f9cf1023640 1 -- 172.21.15.29:0/2934745265 shutdown_connections 2024-03-11T19:17:28.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.601+0000 7f9cf1023640 1 --2- 172.21.15.29:0/2934745265 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9cd40a1c00 0x7f9cd40a40c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.601+0000 7f9cf1023640 1 --2- 172.21.15.29:0/2934745265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec10d4a0 0x7f9cec1ad550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.601+0000 7f9cf1023640 1 --2- 172.21.15.29:0/2934745265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cec10aaf0 0x7f9cec1a6530 secure :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9ce402c640 tx=0x7f9ce40329a0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.601+0000 7f9cf1023640 1 --2- 172.21.15.29:0/2934745265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cec108180 0x7f9cec1a5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.601+0000 7f9cf1023640 1 -- 172.21.15.29:0/2934745265 >> 172.21.15.29:0/2934745265 conn(0x7f9cec06dfe0 msgr2=0x7f9cec1085a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:28.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.601+0000 7f9cf1023640 1 -- 172.21.15.29:0/2934745265 shutdown_connections 2024-03-11T19:17:28.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.601+0000 7f9cf1023640 1 -- 172.21.15.29:0/2934745265 wait complete. 2024-03-11T19:17:28.628 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:17:28.628 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:17:28.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.818+0000 7f4f8558c640 1 -- 172.21.15.29:0/4159804402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f8010aaf0 msgr2=0x7f4f8010cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.818+0000 7f4f8558c640 1 --2- 172.21.15.29:0/4159804402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f8010aaf0 0x7f4f8010cf60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4f74009900 tx=0x7f4f7402bc40 comp rx=0 tx=0).stop 2024-03-11T19:17:28.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.818+0000 7f4f8558c640 1 -- 172.21.15.29:0/4159804402 shutdown_connections 2024-03-11T19:17:28.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.818+0000 7f4f8558c640 1 --2- 172.21.15.29:0/4159804402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f8010d4a0 0x7f4f8010d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.818+0000 7f4f8558c640 1 --2- 172.21.15.29:0/4159804402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f8010aaf0 0x7f4f8010cf60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.818+0000 7f4f8558c640 1 --2- 172.21.15.29:0/4159804402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f80108180 0x7f4f8010a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.818+0000 7f4f8558c640 1 -- 172.21.15.29:0/4159804402 >> 172.21.15.29:0/4159804402 conn(0x7f4f8006dfe0 msgr2=0x7f4f80070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:28.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.818+0000 7f4f8558c640 1 -- 172.21.15.29:0/4159804402 shutdown_connections 2024-03-11T19:17:28.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.818+0000 7f4f8558c640 1 -- 172.21.15.29:0/4159804402 wait complete. 2024-03-11T19:17:28.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.819+0000 7f4f8558c640 1 Processor -- start 2024-03-11T19:17:28.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.819+0000 7f4f8558c640 1 -- start start 2024-03-11T19:17:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.819+0000 7f4f8558c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f80108180 0x7f4f801a5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.819+0000 7f4f8558c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f8010aaf0 0x7f4f801a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.819+0000 7f4f8558c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f8010d4a0 0x7f4f801ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.819+0000 7f4f8558c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f801ada50 con 0x7f4f80108180 2024-03-11T19:17:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.819+0000 7f4f8558c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f801adbc0 con 0x7f4f8010d4a0 2024-03-11T19:17:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.819+0000 7f4f8558c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f801add80 con 0x7f4f8010aaf0 2024-03-11T19:17:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.820+0000 7f4f7e7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f8010aaf0 0x7f4f801a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.820+0000 7f4f7e7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f8010aaf0 0x7f4f801a64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:53936/0 (socket says 172.21.15.29:53936) 2024-03-11T19:17:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.820+0000 7f4f7e7fc640 1 -- 172.21.15.29:0/1219944673 learned_addr learned my addr 172.21.15.29:0/1219944673 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.820+0000 7f4f7effd640 1 --2- 172.21.15.29:0/1219944673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f80108180 0x7f4f801a5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.820+0000 7f4f7f7fe640 1 --2- 172.21.15.29:0/1219944673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f8010d4a0 0x7f4f801ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.820+0000 7f4f7e7fc640 1 -- 172.21.15.29:0/1219944673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f8010d4a0 msgr2=0x7f4f801ad510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.820+0000 7f4f7e7fc640 1 --2- 172.21.15.29:0/1219944673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f8010d4a0 0x7f4f801ad510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.820+0000 7f4f7e7fc640 1 -- 172.21.15.29:0/1219944673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f80108180 msgr2=0x7f4f801a5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.820+0000 7f4f7e7fc640 1 --2- 172.21.15.29:0/1219944673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f80108180 0x7f4f801a5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.820+0000 7f4f7e7fc640 1 -- 172.21.15.29:0/1219944673 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f74009560 con 0x7f4f8010aaf0 2024-03-11T19:17:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.820+0000 7f4f7f7fe640 1 --2- 172.21.15.29:0/1219944673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f8010d4a0 0x7f4f801ad510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:17:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.820+0000 7f4f7e7fc640 1 --2- 172.21.15.29:0/1219944673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f8010aaf0 0x7f4f801a64a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4f74005ab0 tx=0x7f4f740042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.820+0000 7f4f67fff640 1 -- 172.21.15.29:0/1219944673 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f740044e0 con 0x7f4f8010aaf0 2024-03-11T19:17:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.820+0000 7f4f67fff640 1 -- 172.21.15.29:0/1219944673 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4f74003cb0 con 0x7f4f8010aaf0 2024-03-11T19:17:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.820+0000 7f4f8558c640 1 -- 172.21.15.29:0/1219944673 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f801ae000 con 0x7f4f8010aaf0 2024-03-11T19:17:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.821+0000 7f4f67fff640 1 -- 172.21.15.29:0/1219944673 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f74005680 con 0x7f4f8010aaf0 2024-03-11T19:17:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.821+0000 7f4f8558c640 1 -- 172.21.15.29:0/1219944673 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f801ae560 con 0x7f4f8010aaf0 2024-03-11T19:17:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.823+0000 7f4f67fff640 1 -- 172.21.15.29:0/1219944673 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4f74004e50 con 0x7f4f8010aaf0 2024-03-11T19:17:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.823+0000 7f4f67fff640 1 --2- 172.21.15.29:0/1219944673 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4f680a1ab0 0x7f4f680a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.823+0000 7f4f67fff640 1 -- 172.21.15.29:0/1219944673 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4f740efca0 con 0x7f4f8010aaf0 2024-03-11T19:17:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.824+0000 7f4f7effd640 1 --2- 172.21.15.29:0/1219944673 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4f680a1ab0 0x7f4f680a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.824+0000 7f4f88163740 1 Processor -- start 2024-03-11T19:17:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.824+0000 7f4f88163740 1 -- start start 2024-03-11T19:17:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.824+0000 7f4f88163740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e2270 0x5617685e2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.824+0000 7f4f88163740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e2c60 0x5617685e7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.824+0000 7f4f88163740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e7dd0 0x5617685ea260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.824+0000 7f4f88163740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56176859e960 con 0x5617685e2c60 2024-03-11T19:17:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.824+0000 7f4f88163740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x561768552310 con 0x5617685e2270 2024-03-11T19:17:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.824+0000 7f4f88163740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56176854f840 con 0x5617685e7dd0 2024-03-11T19:17:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.825+0000 7f4f7effd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e2270 0x5617685e2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.825+0000 7f4f7effd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e2270 0x5617685e2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:50064/0 (socket says 172.21.15.29:50064) 2024-03-11T19:17:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.825+0000 7f4f7effd640 1 -- 172.21.15.29:0/2603184331 learned_addr learned my addr 172.21.15.29:0/2603184331 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.825+0000 7f4f7e7fc640 1 --2- 172.21.15.29:0/2603184331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e2c60 0x5617685e7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.825+0000 7f4f7f7fe640 1 --2- 172.21.15.29:0/2603184331 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e7dd0 0x5617685ea260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.825+0000 7f4f7e7fc640 1 -- 172.21.15.29:0/2603184331 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e7dd0 msgr2=0x5617685ea260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.825+0000 7f4f7e7fc640 1 --2- 172.21.15.29:0/2603184331 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e7dd0 0x5617685ea260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.825+0000 7f4f7e7fc640 1 -- 172.21.15.29:0/2603184331 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e2270 msgr2=0x5617685e2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.825+0000 7f4f7e7fc640 1 --2- 172.21.15.29:0/2603184331 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e2270 0x5617685e2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.825+0000 7f4f7e7fc640 1 -- 172.21.15.29:0/2603184331 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56176850d060 con 0x5617685e2c60 2024-03-11T19:17:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.825+0000 7f4f7e7fc640 1 --2- 172.21.15.29:0/2603184331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e2c60 0x5617685e7760 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f4f740ed890 tx=0x7f4f7402bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf43649ce5fad179 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.825+0000 7f4f7effd640 1 --2- 172.21.15.29:0/1219944673 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4f680a1ab0 0x7f4f680a3f70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4f78003f20 tx=0x7f4f78004410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.826+0000 7f4f657fa640 1 -- 172.21.15.29:0/2603184331 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f740bd3b0 con 0x5617685e2c60 2024-03-11T19:17:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.826+0000 7f4f657fa640 1 -- 172.21.15.29:0/2603184331 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4f740bccb0 con 0x5617685e2c60 2024-03-11T19:17:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.826+0000 7f4f657fa640 1 -- 172.21.15.29:0/2603184331 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f740c4fc0 con 0x5617685e2c60 2024-03-11T19:17:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.826+0000 7f4f88163740 1 -- 172.21.15.29:0/2603184331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e2c60 msgr2=0x5617685e7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.826+0000 7f4f88163740 1 --2- 172.21.15.29:0/2603184331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e2c60 0x5617685e7760 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f4f740ed890 tx=0x7f4f7402bc40 comp rx=0 tx=0).stop 2024-03-11T19:17:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.827+0000 7f4f88163740 1 -- 172.21.15.29:0/2603184331 shutdown_connections 2024-03-11T19:17:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.827+0000 7f4f88163740 1 --2- 172.21.15.29:0/2603184331 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e7dd0 0x5617685ea260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.827+0000 7f4f88163740 1 --2- 172.21.15.29:0/2603184331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e2c60 0x5617685e7760 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.827+0000 7f4f88163740 1 --2- 172.21.15.29:0/2603184331 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e2270 0x5617685e2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.827+0000 7f4f88163740 1 -- 172.21.15.29:0/2603184331 >> 172.21.15.29:0/2603184331 conn(0x5617685dd7e0 msgr2=0x5617685ddbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.827+0000 7f4f88163740 1 -- 172.21.15.29:0/2603184331 shutdown_connections 2024-03-11T19:17:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.827+0000 7f4f88163740 1 -- 172.21.15.29:0/2603184331 wait complete. 2024-03-11T19:17:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.827+0000 7f4f88163740 1 -- start start 2024-03-11T19:17:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.827+0000 7f4f88163740 1 Processor -- start 2024-03-11T19:17:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.828+0000 7f4f88163740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e2270 0x5617685ec1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.828+0000 7f4f88163740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e2c60 0x5617685ec720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.828+0000 7f4f88163740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e7dd0 0x5617685f3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.828+0000 7f4f7effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e2270 0x5617685ec1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.828+0000 7f4f7effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e2270 0x5617685ec1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42342/0 (socket says 172.21.15.29:42342) 2024-03-11T19:17:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.828+0000 7f4f7effd640 1 -- 172.21.15.29:0/3917519537 learned_addr learned my addr 172.21.15.29:0/3917519537 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.828+0000 7f4f7f7fe640 1 --2- 172.21.15.29:0/3917519537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e7dd0 0x5617685f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.828+0000 7f4f88163740 1 -- 172.21.15.29:0/3917519537 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56176854b020 con 0x5617685e2270 2024-03-11T19:17:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.828+0000 7f4f88163740 1 -- 172.21.15.29:0/3917519537 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56176821e430 con 0x5617685e7dd0 2024-03-11T19:17:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.828+0000 7f4f88163740 1 -- 172.21.15.29:0/3917519537 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617681dae00 con 0x5617685e2c60 2024-03-11T19:17:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.828+0000 7f4f7e7fc640 1 --2- 172.21.15.29:0/3917519537 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e2c60 0x5617685ec720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.828+0000 7f4f7e7fc640 1 -- 172.21.15.29:0/3917519537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e7dd0 msgr2=0x5617685f3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.829+0000 7f4f7e7fc640 1 --2- 172.21.15.29:0/3917519537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e7dd0 0x5617685f3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.829+0000 7f4f7e7fc640 1 -- 172.21.15.29:0/3917519537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e2270 msgr2=0x5617685ec1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.829+0000 7f4f7e7fc640 1 --2- 172.21.15.29:0/3917519537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e2270 0x5617685ec1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.829+0000 7f4f7e7fc640 1 -- 172.21.15.29:0/3917519537 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f801ae560 con 0x5617685e2c60 2024-03-11T19:17:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.829+0000 7f4f7f7fe640 1 --2- 172.21.15.29:0/3917519537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e7dd0 0x5617685f3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:17:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.829+0000 7f4f7effd640 1 --2- 172.21.15.29:0/3917519537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e2270 0x5617685ec1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:17:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.829+0000 7f4f88163740 10 client.0 _ll_register_callbacks cb 0x5617684f2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:17:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.829+0000 7f4f88163740 1 -- 172.21.15.29:0/3917519537 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56176824a7e0 con 0x5617685e2c60 2024-03-11T19:17:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.829+0000 7f4f7e7fc640 1 --2- 172.21.15.29:0/3917519537 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e2c60 0x5617685ec720 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4f740c0fc0 tx=0x7f4f74005100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4105b117b5458096 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.829+0000 7f4f62ff5640 10 client.26767 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:17:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.829+0000 7f4f62ff5640 1 -- 172.21.15.29:0/3917519537 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f740c0e00 con 0x5617685e2c60 2024-03-11T19:17:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.829+0000 7f4f62ff5640 1 -- 172.21.15.29:0/3917519537 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4f740c5450 con 0x5617685e2c60 2024-03-11T19:17:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.829+0000 7f4f62ff5640 1 -- 172.21.15.29:0/3917519537 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f74003a10 con 0x5617685e2c60 2024-03-11T19:17:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.829+0000 7f4f88163740 10 client.26767 fetch_fsmap learned FSMap version 27 2024-03-11T19:17:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.829+0000 7f4f88163740 1 -- 172.21.15.29:0/3917519537 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x5617685496a0 con 0x5617685e2c60 2024-03-11T19:17:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.830+0000 7f4f62ff5640 1 -- 172.21.15.29:0/3917519537 <== mon.1 v2:172.21.15.190:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4f7403c070 con 0x5617685e2c60 2024-03-11T19:17:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.830+0000 7f4f88163740 10 client.26767 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:17:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.830+0000 7f4f88163740 10 client.26767 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:17:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.830+0000 7f4f88163740 20 client.26767 populate_metadata read hostname 'smithi029' 2024-03-11T19:17:28.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.830+0000 7f4f88163740 1 --2- 172.21.15.29:0/3917519537 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5617685f7c60 0x5617685fa090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.830+0000 7f4f88163740 4 client.26767 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:17:28.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.830+0000 7f4f88163740 1 -- 172.21.15.29:0/3917519537 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561768287720 con 0x5617685f7c60 2024-03-11T19:17:28.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.830+0000 7f4f7effd640 1 --2- 172.21.15.29:0/3917519537 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5617685f7c60 0x5617685fa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.831+0000 7f4f7effd640 1 --2- 172.21.15.29:0/3917519537 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5617685f7c60 0x5617685fa090 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=10ea396dbf1836cd server_cookie=cb510696bbce4c0b in_seq=0 out_seq=0 2024-03-11T19:17:28.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.831+0000 7f4f62ff5640 10 client.26767 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:17:28.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.831+0000 7f4f62ff5640 1 -- 172.21.15.29:0/3917519537 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561768287720 con 0x5617685f7c60 2024-03-11T19:17:28.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.831+0000 7f4f62ff5640 10 client.26767 handle_command_reply: tid=0 2024-03-11T19:17:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.831+0000 7f4f88163740 1 client.26767 shutdown 2024-03-11T19:17:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.832+0000 7f4f88163740 1 -- 172.21.15.29:0/3917519537 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e2c60 msgr2=0x5617685ec720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.832+0000 7f4f88163740 1 --2- 172.21.15.29:0/3917519537 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e2c60 0x5617685ec720 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4f740c0fc0 tx=0x7f4f74005100 comp rx=0 tx=0).stop 2024-03-11T19:17:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.832+0000 7f4f88163740 1 -- 172.21.15.29:0/3917519537 shutdown_connections 2024-03-11T19:17:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.832+0000 7f4f88163740 1 --2- 172.21.15.29:0/3917519537 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x5617685f7c60 0x5617685fa090 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.832+0000 7f4f88163740 1 --2- 172.21.15.29:0/3917519537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e7dd0 0x5617685f3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.832+0000 7f4f88163740 1 --2- 172.21.15.29:0/3917519537 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e2c60 0x5617685ec720 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.832+0000 7f4f88163740 1 --2- 172.21.15.29:0/3917519537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e2270 0x5617685ec1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.832+0000 7f4f88163740 1 -- 172.21.15.29:0/3917519537 >> 172.21.15.29:0/3917519537 conn(0x5617685dd7e0 msgr2=0x5617685dcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.832+0000 7f4f62ff5640 0 client.26767 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:17:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.832+0000 7f4f88163740 1 -- 172.21.15.29:0/3917519537 shutdown_connections 2024-03-11T19:17:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.832+0000 7f4f88163740 1 -- 172.21.15.29:0/3917519537 wait complete. 2024-03-11T19:17:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.832+0000 7f4f88163740 20 client.26767 trim_cache size 0 max 16384 2024-03-11T19:17:28.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.857+0000 7f4f88163740 1 Processor -- start 2024-03-11T19:17:28.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.857+0000 7f4f88163740 1 -- start start 2024-03-11T19:17:28.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.857+0000 7f4f88163740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e2270 0x5617685ec1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.857+0000 7f4f88163740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e2c60 0x5617685f3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.857+0000 7f4f88163740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e7dd0 0x5617685ec720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.857+0000 7f4f88163740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617684d0130 con 0x5617685e7dd0 2024-03-11T19:17:28.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.857+0000 7f4f88163740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617685d95c0 con 0x5617685e2c60 2024-03-11T19:17:28.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.857+0000 7f4f88163740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561768287720 con 0x5617685e2270 2024-03-11T19:17:28.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.858+0000 7f4f7e7fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e2c60 0x5617685f3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.858+0000 7f4f7e7fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e2c60 0x5617685f3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:50086/0 (socket says 172.21.15.29:50086) 2024-03-11T19:17:28.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.858+0000 7f4f7e7fc640 1 -- 172.21.15.29:0/3636562617 learned_addr learned my addr 172.21.15.29:0/3636562617 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:28.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.858+0000 7f4f7f7fe640 1 --2- 172.21.15.29:0/3636562617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e7dd0 0x5617685ec720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.858+0000 7f4f7effd640 1 --2- 172.21.15.29:0/3636562617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e2270 0x5617685ec1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.858+0000 7f4f7e7fc640 1 -- 172.21.15.29:0/3636562617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e2270 msgr2=0x5617685ec1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.858+0000 7f4f7e7fc640 1 --2- 172.21.15.29:0/3636562617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e2270 0x5617685ec1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.858+0000 7f4f7e7fc640 1 -- 172.21.15.29:0/3636562617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e7dd0 msgr2=0x5617685ec720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.858+0000 7f4f7e7fc640 1 --2- 172.21.15.29:0/3636562617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e7dd0 0x5617685ec720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.858+0000 7f4f7e7fc640 1 -- 172.21.15.29:0/3636562617 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617684e5c20 con 0x5617685e2c60 2024-03-11T19:17:28.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.859+0000 7f4f7e7fc640 1 --2- 172.21.15.29:0/3636562617 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e2c60 0x5617685f3720 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4f740b8720 tx=0x7f4f740d8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a417cd0c1cd11efa server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:28.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.859+0000 7f4f64ff9640 1 -- 172.21.15.29:0/3636562617 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f74003a10 con 0x5617685e2c60 2024-03-11T19:17:28.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.859+0000 7f4f64ff9640 1 -- 172.21.15.29:0/3636562617 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x56176824a7e0 con 0x5617685e2c60 2024-03-11T19:17:28.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.859+0000 7f4f64ff9640 1 -- 172.21.15.29:0/3636562617 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f740bd3b0 con 0x5617685e2c60 2024-03-11T19:17:28.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.861+0000 7f4f88163740 1 -- 172.21.15.29:0/3636562617 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e2c60 msgr2=0x5617685f3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.861+0000 7f4f88163740 1 --2- 172.21.15.29:0/3636562617 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e2c60 0x5617685f3720 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4f740b8720 tx=0x7f4f740d8000 comp rx=0 tx=0).stop 2024-03-11T19:17:28.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.861+0000 7f4f88163740 1 -- 172.21.15.29:0/3636562617 shutdown_connections 2024-03-11T19:17:28.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.861+0000 7f4f88163740 1 --2- 172.21.15.29:0/3636562617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e7dd0 0x5617685ec720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.861+0000 7f4f88163740 1 --2- 172.21.15.29:0/3636562617 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e2c60 0x5617685f3720 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.861+0000 7f4f88163740 1 --2- 172.21.15.29:0/3636562617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e2270 0x5617685ec1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.861+0000 7f4f88163740 1 -- 172.21.15.29:0/3636562617 >> 172.21.15.29:0/3636562617 conn(0x5617685dd7e0 msgr2=0x5617685e4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:28.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.863+0000 7f4f88163740 1 -- 172.21.15.29:0/3636562617 shutdown_connections 2024-03-11T19:17:28.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.863+0000 7f4f88163740 1 -- 172.21.15.29:0/3636562617 wait complete. 2024-03-11T19:17:28.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.863+0000 7f4f88163740 1 -- start start 2024-03-11T19:17:28.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.863+0000 7f4f88163740 1 Processor -- start 2024-03-11T19:17:28.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.864+0000 7f4f88163740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e2270 0x5617685f3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.864+0000 7f4f88163740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e2c60 0x5617685f3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.864+0000 7f4f88163740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e7dd0 0x5617685f1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.864+0000 7f4f7effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e2270 0x5617685f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.864+0000 7f4f7effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e2270 0x5617685f3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:53952/0 (socket says 172.21.15.29:53952) 2024-03-11T19:17:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.864+0000 7f4f7effd640 1 -- 172.21.15.29:0/3412436715 learned_addr learned my addr 172.21.15.29:0/3412436715 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.864+0000 7f4f88163740 1 -- 172.21.15.29:0/3412436715 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617685f2040 con 0x5617685e2c60 2024-03-11T19:17:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.864+0000 7f4f88163740 1 -- 172.21.15.29:0/3412436715 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617685f21b0 con 0x5617685e7dd0 2024-03-11T19:17:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.864+0000 7f4f7f7fe640 1 --2- 172.21.15.29:0/3412436715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e7dd0 0x5617685f1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.864+0000 7f4f88163740 1 -- 172.21.15.29:0/3412436715 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617685f2370 con 0x5617685e2270 2024-03-11T19:17:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.864+0000 7f4f88163740 10 client.0 _ll_register_callbacks cb 0x7f4f74005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:17:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.865+0000 7f4f7e7fc640 1 --2- 172.21.15.29:0/3412436715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e2c60 0x5617685f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.865+0000 7f4f7f7fe640 1 -- 172.21.15.29:0/3412436715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e2270 msgr2=0x5617685f3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.865+0000 7f4f7f7fe640 1 --2- 172.21.15.29:0/3412436715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e2270 0x5617685f3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.865+0000 7f4f7f7fe640 1 -- 172.21.15.29:0/3412436715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e2c60 msgr2=0x5617685f3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.865+0000 7f4f7f7fe640 1 --2- 172.21.15.29:0/3412436715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e2c60 0x5617685f3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.865+0000 7f4f7f7fe640 1 -- 172.21.15.29:0/3412436715 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5617685496a0 con 0x5617685e7dd0 2024-03-11T19:17:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.865+0000 7f4f7f7fe640 1 --2- 172.21.15.29:0/3412436715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e7dd0 0x5617685f1940 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x5617685e30e0 tx=0x7f4f7000fea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc8248a148d6c3cd server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:28.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.865+0000 7f4f62ff5640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:17:28.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.866+0000 7f4f62ff5640 1 -- 172.21.15.29:0/3412436715 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7001e070 con 0x5617685e7dd0 2024-03-11T19:17:28.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.866+0000 7f4f62ff5640 1 -- 172.21.15.29:0/3412436715 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4f7000ea70 con 0x5617685e7dd0 2024-03-11T19:17:28.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.866+0000 7f4f88163740 1 -- 172.21.15.29:0/3412436715 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f740bccb0 con 0x5617685e7dd0 2024-03-11T19:17:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.866+0000 7f4f62ff5640 1 -- 172.21.15.29:0/3412436715 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f7000ec30 con 0x5617685e7dd0 2024-03-11T19:17:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.866+0000 7f4f88163740 10 client.26732 fetch_fsmap learned FSMap version 27 2024-03-11T19:17:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.866+0000 7f4f88163740 1 -- 172.21.15.29:0/3412436715 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56176850b8c0 con 0x5617685e7dd0 2024-03-11T19:17:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.867+0000 7f4f62ff5640 1 -- 172.21.15.29:0/3412436715 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4f70029020 con 0x5617685e7dd0 2024-03-11T19:17:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.867+0000 7f4f62ff5640 1 -- 172.21.15.29:0/3412436715 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4f70021020 con 0x5617685e7dd0 2024-03-11T19:17:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.867+0000 7f4f88163740 10 client.26732 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:17:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.867+0000 7f4f88163740 10 client.26732 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:17:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.867+0000 7f4f88163740 20 client.26732 populate_metadata read hostname 'smithi029' 2024-03-11T19:17:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.867+0000 7f4f88163740 1 --2- 172.21.15.29:0/3412436715 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56176864f750 0x561768651b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.867+0000 7f4f88163740 4 client.26732 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:17:28.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.867+0000 7f4f88163740 1 -- 172.21.15.29:0/3412436715 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f4f740bccb0 con 0x56176864f750 2024-03-11T19:17:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.868+0000 7f4f7effd640 1 --2- 172.21.15.29:0/3412436715 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56176864f750 0x561768651b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.869+0000 7f4f62ff5640 10 client.26732 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:17:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.869+0000 7f4f7effd640 1 --2- 172.21.15.29:0/3412436715 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56176864f750 0x561768651b70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5c3952e6fdff90f8 server_cookie=aae819252bf7f031 in_seq=0 out_seq=0 2024-03-11T19:17:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.869+0000 7f4f62ff5640 1 -- 172.21.15.29:0/3412436715 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f4f740bccb0 con 0x56176864f750 2024-03-11T19:17:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.869+0000 7f4f62ff5640 10 client.26732 handle_command_reply: tid=0 2024-03-11T19:17:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.869+0000 7f4f88163740 1 client.26732 shutdown 2024-03-11T19:17:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.869+0000 7f4f88163740 1 -- 172.21.15.29:0/3412436715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e7dd0 msgr2=0x5617685f1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.869+0000 7f4f88163740 1 --2- 172.21.15.29:0/3412436715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e7dd0 0x5617685f1940 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x5617685e30e0 tx=0x7f4f7000fea0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.869+0000 7f4f88163740 1 -- 172.21.15.29:0/3412436715 shutdown_connections 2024-03-11T19:17:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.869+0000 7f4f88163740 1 --2- 172.21.15.29:0/3412436715 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56176864f750 0x561768651b70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.869+0000 7f4f88163740 1 --2- 172.21.15.29:0/3412436715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5617685e7dd0 0x5617685f1940 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.870+0000 7f4f62ff5640 0 client.26732 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:17:28.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.870+0000 7f4f88163740 1 --2- 172.21.15.29:0/3412436715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5617685e2c60 0x5617685f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.870+0000 7f4f88163740 1 --2- 172.21.15.29:0/3412436715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617685e2270 0x5617685f3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.870+0000 7f4f88163740 1 -- 172.21.15.29:0/3412436715 >> 172.21.15.29:0/3412436715 conn(0x5617685dd7e0 msgr2=0x5617685ec1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:28.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.870+0000 7f4f88163740 1 -- 172.21.15.29:0/3412436715 shutdown_connections 2024-03-11T19:17:28.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.870+0000 7f4f88163740 1 -- 172.21.15.29:0/3412436715 wait complete. 2024-03-11T19:17:28.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.870+0000 7f4f88163740 20 client.26732 trim_cache size 0 max 16384 2024-03-11T19:17:28.876 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:17:28.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.871+0000 7f4f8558c640 1 -- 172.21.15.29:0/1219944673 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4f680a1ab0 msgr2=0x7f4f680a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.871+0000 7f4f8558c640 1 --2- 172.21.15.29:0/1219944673 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4f680a1ab0 0x7f4f680a3f70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4f78003f20 tx=0x7f4f78004410 comp rx=0 tx=0).stop 2024-03-11T19:17:28.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.871+0000 7f4f8558c640 1 -- 172.21.15.29:0/1219944673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f8010aaf0 msgr2=0x7f4f801a64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:28.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.871+0000 7f4f8558c640 1 --2- 172.21.15.29:0/1219944673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f8010aaf0 0x7f4f801a64a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4f74005ab0 tx=0x7f4f740042f0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.871+0000 7f4f8558c640 1 -- 172.21.15.29:0/1219944673 shutdown_connections 2024-03-11T19:17:28.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.872+0000 7f4f8558c640 1 --2- 172.21.15.29:0/1219944673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f8010d4a0 0x7f4f801ad510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.872+0000 7f4f8558c640 1 --2- 172.21.15.29:0/1219944673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f8010aaf0 0x7f4f801a64a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.872+0000 7f4f8558c640 1 --2- 172.21.15.29:0/1219944673 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4f680a1ab0 0x7f4f680a3f70 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.872+0000 7f4f8558c640 1 --2- 172.21.15.29:0/1219944673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f80108180 0x7f4f801a5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:28.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.872+0000 7f4f8558c640 1 -- 172.21.15.29:0/1219944673 >> 172.21.15.29:0/1219944673 conn(0x7f4f8006dfe0 msgr2=0x7f4f801097f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:28.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.872+0000 7f4f8558c640 1 -- 172.21.15.29:0/1219944673 shutdown_connections 2024-03-11T19:17:28.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:28.872+0000 7f4f8558c640 1 -- 172.21.15.29:0/1219944673 wait complete. 2024-03-11T19:17:28.895 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:17:28.895 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:17:29.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.068+0000 7f7718876640 1 -- 172.21.15.29:0/3478946809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7714100080 msgr2=0x7f77141004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:29.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.068+0000 7f7718876640 1 --2- 172.21.15.29:0/3478946809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7714100080 0x7f77141004a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f7704009900 tx=0x7f770402bc30 comp rx=0 tx=0).stop 2024-03-11T19:17:29.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.069+0000 7f7718876640 1 -- 172.21.15.29:0/3478946809 shutdown_connections 2024-03-11T19:17:29.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.069+0000 7f7718876640 1 --2- 172.21.15.29:0/3478946809 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7714101c30 0x7f77141084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.069+0000 7f7718876640 1 --2- 172.21.15.29:0/3478946809 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7714101270 0x7f77141016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.069+0000 7f7718876640 1 --2- 172.21.15.29:0/3478946809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7714100080 0x7f77141004a0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.069+0000 7f7718876640 1 -- 172.21.15.29:0/3478946809 >> 172.21.15.29:0/3478946809 conn(0x7f77140fb610 msgr2=0x7f77140fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:29.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.069+0000 7f7718876640 1 -- 172.21.15.29:0/3478946809 shutdown_connections 2024-03-11T19:17:29.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.069+0000 7f7718876640 1 -- 172.21.15.29:0/3478946809 wait complete. 2024-03-11T19:17:29.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.070+0000 7f7718876640 1 Processor -- start 2024-03-11T19:17:29.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.070+0000 7f7718876640 1 -- start start 2024-03-11T19:17:29.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.070+0000 7f7718876640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7714100080 0x7f771419d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.070+0000 7f7718876640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7714101270 0x7f771419dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.070+0000 7f771259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7714100080 0x7f771419d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.070+0000 7f7718876640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7714101c30 0x7f77141a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.071+0000 7f771259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7714100080 0x7f771419d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:50112/0 (socket says 172.21.15.29:50112) 2024-03-11T19:17:29.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.071+0000 7f771259c640 1 -- 172.21.15.29:0/2533489309 learned_addr learned my addr 172.21.15.29:0/2533489309 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:29.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.071+0000 7f7718876640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77141a52a0 con 0x7f7714101c30 2024-03-11T19:17:29.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.071+0000 7f7718876640 1 -- 172.21.15.29:0/2533489309 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77141a5410 con 0x7f7714100080 2024-03-11T19:17:29.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.071+0000 7f7718876640 1 -- 172.21.15.29:0/2533489309 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77141a55d0 con 0x7f7714101270 2024-03-11T19:17:29.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.071+0000 7f770bfff640 1 --2- 172.21.15.29:0/2533489309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7714101270 0x7f771419dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.071+0000 7f7712d9d640 1 --2- 172.21.15.29:0/2533489309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7714101c30 0x7f77141a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.071+0000 7f771259c640 1 -- 172.21.15.29:0/2533489309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7714101270 msgr2=0x7f771419dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:29.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.071+0000 7f771259c640 1 --2- 172.21.15.29:0/2533489309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7714101270 0x7f771419dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.071+0000 7f771259c640 1 -- 172.21.15.29:0/2533489309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7714101c30 msgr2=0x7f77141a4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:29.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.071+0000 7f771259c640 1 --2- 172.21.15.29:0/2533489309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7714101c30 0x7f77141a4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.071+0000 7f771259c640 1 -- 172.21.15.29:0/2533489309 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7704009560 con 0x7f7714100080 2024-03-11T19:17:29.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.071+0000 7f7712d9d640 1 --2- 172.21.15.29:0/2533489309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7714101c30 0x7f77141a4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:17:29.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.071+0000 7f771259c640 1 --2- 172.21.15.29:0/2533489309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7714100080 0x7f771419d7b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f770402c7d0 tx=0x7f7704004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:29.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.072+0000 7f770b7fe640 1 -- 172.21.15.29:0/2533489309 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7704004500 con 0x7f7714100080 2024-03-11T19:17:29.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.072+0000 7f7718876640 1 -- 172.21.15.29:0/2533489309 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77141a5850 con 0x7f7714100080 2024-03-11T19:17:29.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.072+0000 7f7718876640 1 -- 172.21.15.29:0/2533489309 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77141a5db0 con 0x7f7714100080 2024-03-11T19:17:29.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.072+0000 7f770b7fe640 1 -- 172.21.15.29:0/2533489309 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f770402de90 con 0x7f7714100080 2024-03-11T19:17:29.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.072+0000 7f770b7fe640 1 -- 172.21.15.29:0/2533489309 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7704005350 con 0x7f7714100080 2024-03-11T19:17:29.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.075+0000 7f770b7fe640 1 -- 172.21.15.29:0/2533489309 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f77040054b0 con 0x7f7714100080 2024-03-11T19:17:29.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.075+0000 7f771b446740 1 Processor -- start 2024-03-11T19:17:29.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.076+0000 7f771b446740 1 -- start start 2024-03-11T19:17:29.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.076+0000 7f771b446740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7270 0x557751ed7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.076+0000 7f771b446740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751ed7c60 0x557751edc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.076+0000 7f771b446740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751edcdd0 0x557751edf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.076+0000 7f771b446740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557751e93960 con 0x557751ed7270 2024-03-11T19:17:29.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.076+0000 7f771b446740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557751e47310 con 0x557751edcdd0 2024-03-11T19:17:29.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.076+0000 7f771b446740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557751e44840 con 0x557751ed7c60 2024-03-11T19:17:29.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.076+0000 7f7712d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7270 0x557751ed7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.076+0000 7f770bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751edcdd0 0x557751edf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.076+0000 7f771259c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751ed7c60 0x557751edc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.076+0000 7f7712d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7270 0x557751ed7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42382/0 (socket says 172.21.15.29:42382) 2024-03-11T19:17:29.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.076+0000 7f7712d9d640 1 -- 172.21.15.29:0/3316396092 learned_addr learned my addr 172.21.15.29:0/3316396092 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:29.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.076+0000 7f770b7fe640 1 --2- 172.21.15.29:0/2533489309 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f77000a1b30 0x7f77000a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.077+0000 7f770bfff640 1 -- 172.21.15.29:0/3316396092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751ed7c60 msgr2=0x557751edc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:29.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.077+0000 7f770bfff640 1 --2- 172.21.15.29:0/3316396092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751ed7c60 0x557751edc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.077+0000 7f770bfff640 1 -- 172.21.15.29:0/3316396092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7270 msgr2=0x557751ed7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:29.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.077+0000 7f770bfff640 1 --2- 172.21.15.29:0/3316396092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7270 0x557751ed7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.077+0000 7f770bfff640 1 -- 172.21.15.29:0/3316396092 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557751e02060 con 0x557751edcdd0 2024-03-11T19:17:29.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.077+0000 7f7712d9d640 1 --2- 172.21.15.29:0/3316396092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7270 0x557751ed7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:17:29.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.077+0000 7f770b7fe640 1 -- 172.21.15.29:0/2533489309 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f77040ef3e0 con 0x7f7714100080 2024-03-11T19:17:29.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.077+0000 7f7712d9d640 1 --2- 172.21.15.29:0/2533489309 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f77000a1b30 0x7f77000a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.077+0000 7f770bfff640 1 --2- 172.21.15.29:0/3316396092 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751edcdd0 0x557751edf260 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009bf0 tx=0x7f76fc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1454a13d2efa717 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:29.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.077+0000 7f7708ff9640 1 -- 172.21.15.29:0/3316396092 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc004050 con 0x557751edcdd0 2024-03-11T19:17:29.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.077+0000 7f7708ff9640 1 -- 172.21.15.29:0/3316396092 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f76fc005020 con 0x557751edcdd0 2024-03-11T19:17:29.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.077+0000 7f7708ff9640 1 -- 172.21.15.29:0/3316396092 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc004c40 con 0x557751edcdd0 2024-03-11T19:17:29.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.078+0000 7f771b446740 1 -- 172.21.15.29:0/3316396092 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751edcdd0 msgr2=0x557751edf260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:29.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.078+0000 7f7712d9d640 1 --2- 172.21.15.29:0/2533489309 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f77000a1b30 0x7f77000a3ff0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f770c009fd0 tx=0x7f770c004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:29.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.078+0000 7f771b446740 1 --2- 172.21.15.29:0/3316396092 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751edcdd0 0x557751edf260 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009bf0 tx=0x7f76fc011810 comp rx=0 tx=0).stop 2024-03-11T19:17:29.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.078+0000 7f771b446740 1 -- 172.21.15.29:0/3316396092 shutdown_connections 2024-03-11T19:17:29.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.078+0000 7f771b446740 1 --2- 172.21.15.29:0/3316396092 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751edcdd0 0x557751edf260 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.078+0000 7f771b446740 1 --2- 172.21.15.29:0/3316396092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751ed7c60 0x557751edc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.078+0000 7f771b446740 1 --2- 172.21.15.29:0/3316396092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7270 0x557751ed7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.078+0000 7f771b446740 1 -- 172.21.15.29:0/3316396092 >> 172.21.15.29:0/3316396092 conn(0x557751ed27e0 msgr2=0x557751ed2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:29.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.079+0000 7f771b446740 1 -- 172.21.15.29:0/3316396092 shutdown_connections 2024-03-11T19:17:29.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.079+0000 7f771b446740 1 -- 172.21.15.29:0/3316396092 wait complete. 2024-03-11T19:17:29.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.079+0000 7f771b446740 1 -- start start 2024-03-11T19:17:29.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.079+0000 7f771b446740 1 Processor -- start 2024-03-11T19:17:29.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.079+0000 7f771b446740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751ed7270 0x557751ee1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.080+0000 7f771b446740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751ed7c60 0x557751ee19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.080+0000 7f771b446740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751edcdd0 0x557751ee8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.080+0000 7f771259c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751ed7270 0x557751ee1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.080+0000 7f771259c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751ed7270 0x557751ee1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:53998/0 (socket says 172.21.15.29:53998) 2024-03-11T19:17:29.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.080+0000 7f771259c640 1 -- 172.21.15.29:0/2910255006 learned_addr learned my addr 172.21.15.29:0/2910255006 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:29.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.080+0000 7f771b446740 1 -- 172.21.15.29:0/2910255006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557751e40020 con 0x557751edcdd0 2024-03-11T19:17:29.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.080+0000 7f771b446740 1 -- 172.21.15.29:0/2910255006 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557751b13430 con 0x557751ed7c60 2024-03-11T19:17:29.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.080+0000 7f771b446740 1 -- 172.21.15.29:0/2910255006 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557751acfe00 con 0x557751ed7270 2024-03-11T19:17:29.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.080+0000 7f7712d9d640 1 --2- 172.21.15.29:0/2910255006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751edcdd0 0x557751ee8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.080+0000 7f770bfff640 1 --2- 172.21.15.29:0/2910255006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751ed7c60 0x557751ee19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.081+0000 7f771259c640 1 -- 172.21.15.29:0/2910255006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751ed7c60 msgr2=0x557751ee19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:29.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.081+0000 7f771259c640 1 --2- 172.21.15.29:0/2910255006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751ed7c60 0x557751ee19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.081+0000 7f771259c640 1 -- 172.21.15.29:0/2910255006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751edcdd0 msgr2=0x557751ee8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:29.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.081+0000 7f771259c640 1 --2- 172.21.15.29:0/2910255006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751edcdd0 0x557751ee8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.081+0000 7f771259c640 1 -- 172.21.15.29:0/2910255006 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76fc009850 con 0x557751ed7270 2024-03-11T19:17:29.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.081+0000 7f771259c640 1 --2- 172.21.15.29:0/2910255006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751ed7270 0x557751ee1480 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f77040bc050 tx=0x7f7704004dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d077f864a6cca0e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:29.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.081+0000 7f771b446740 10 client.0 _ll_register_callbacks cb 0x557751de7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:17:29.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.081+0000 7f76f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:17:29.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.081+0000 7f76f9ffb640 1 -- 172.21.15.29:0/2910255006 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77040c4860 con 0x557751ed7270 2024-03-11T19:17:29.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.081+0000 7f76f9ffb640 1 -- 172.21.15.29:0/2910255006 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f77040bca50 con 0x557751ed7270 2024-03-11T19:17:29.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.081+0000 7f771b446740 1 -- 172.21.15.29:0/2910255006 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557751b3f7e0 con 0x557751ed7270 2024-03-11T19:17:29.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.082+0000 7f76f9ffb640 1 -- 172.21.15.29:0/2910255006 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77040bbe90 con 0x557751ed7270 2024-03-11T19:17:29.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.082+0000 7f771b446740 10 client.26785 fetch_fsmap learned FSMap version 27 2024-03-11T19:17:29.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.082+0000 7f771b446740 1 -- 172.21.15.29:0/2910255006 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x557751e3e6a0 con 0x557751ed7270 2024-03-11T19:17:29.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.082+0000 7f76f9ffb640 1 -- 172.21.15.29:0/2910255006 <== mon.1 v2:172.21.15.190:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f770403c070 con 0x557751ed7270 2024-03-11T19:17:29.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.082+0000 7f771b446740 10 client.26785 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:17:29.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.082+0000 7f771b446740 10 client.26785 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:17:29.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.082+0000 7f771b446740 20 client.26785 populate_metadata read hostname 'smithi029' 2024-03-11T19:17:29.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.082+0000 7f771b446740 1 --2- 172.21.15.29:0/2910255006 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557751eed050 0x557751eef510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.083+0000 7f771b446740 4 client.26785 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:17:29.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.083+0000 7f771b446740 1 -- 172.21.15.29:0/2910255006 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557751b3f7e0 con 0x557751eed050 2024-03-11T19:17:29.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.083+0000 7f770bfff640 1 --2- 172.21.15.29:0/2910255006 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557751eed050 0x557751eef510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.083+0000 7f770bfff640 1 --2- 172.21.15.29:0/2910255006 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557751eed050 0x557751eef510 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ebf62138a8294801 server_cookie=371426bc3ba663c9 in_seq=0 out_seq=0 2024-03-11T19:17:29.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.083+0000 7f76f9ffb640 10 client.26785 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:17:29.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.084+0000 7f76f9ffb640 1 -- 172.21.15.29:0/2910255006 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557751b3f7e0 con 0x557751eed050 2024-03-11T19:17:29.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.084+0000 7f76f9ffb640 10 client.26785 handle_command_reply: tid=0 2024-03-11T19:17:29.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.084+0000 7f771b446740 1 client.26785 shutdown 2024-03-11T19:17:29.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.085+0000 7f771b446740 1 -- 172.21.15.29:0/2910255006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751ed7270 msgr2=0x557751ee1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:29.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.085+0000 7f771b446740 1 --2- 172.21.15.29:0/2910255006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751ed7270 0x557751ee1480 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f77040bc050 tx=0x7f7704004dc0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.085+0000 7f771b446740 1 -- 172.21.15.29:0/2910255006 shutdown_connections 2024-03-11T19:17:29.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.085+0000 7f771b446740 1 --2- 172.21.15.29:0/2910255006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751edcdd0 0x557751ee8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.085+0000 7f771b446740 1 --2- 172.21.15.29:0/2910255006 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557751eed050 0x557751eef510 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.085+0000 7f771b446740 1 --2- 172.21.15.29:0/2910255006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751ed7c60 0x557751ee19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.085+0000 7f771b446740 1 --2- 172.21.15.29:0/2910255006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751ed7270 0x557751ee1480 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.085+0000 7f771b446740 1 -- 172.21.15.29:0/2910255006 >> 172.21.15.29:0/2910255006 conn(0x557751ed27e0 msgr2=0x557751ed1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:29.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.086+0000 7f76f9ffb640 0 client.26785 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:17:29.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.086+0000 7f771b446740 1 -- 172.21.15.29:0/2910255006 shutdown_connections 2024-03-11T19:17:29.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.087+0000 7f771b446740 1 -- 172.21.15.29:0/2910255006 wait complete. 2024-03-11T19:17:29.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.087+0000 7f771b446740 20 client.26785 trim_cache size 0 max 16384 2024-03-11T19:17:29.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.111+0000 7f771b446740 1 Processor -- start 2024-03-11T19:17:29.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.111+0000 7f771b446740 1 -- start start 2024-03-11T19:17:29.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.111+0000 7f771b446740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751ed7270 0x557751ed9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.111+0000 7f771b446740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7c60 0x557751ee89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.111+0000 7f771b446740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751edcdd0 0x557751ed9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.111+0000 7f771b446740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557751dc5130 con 0x557751ed7c60 2024-03-11T19:17:29.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.111+0000 7f771b446740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557751edf920 con 0x557751ed7270 2024-03-11T19:17:29.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.111+0000 7f771b446740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557751ed37d0 con 0x557751edcdd0 2024-03-11T19:17:29.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.111+0000 7f770bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7c60 0x557751ee89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.112+0000 7f771259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751ed7270 0x557751ed9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.112+0000 7f771259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751ed7270 0x557751ed9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:50152/0 (socket says 172.21.15.29:50152) 2024-03-11T19:17:29.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.112+0000 7f770bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7c60 0x557751ee89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42404/0 (socket says 172.21.15.29:42404) 2024-03-11T19:17:29.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.112+0000 7f771259c640 1 -- 172.21.15.29:0/2553575187 learned_addr learned my addr 172.21.15.29:0/2553575187 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:29.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.112+0000 7f7712d9d640 1 --2- 172.21.15.29:0/2553575187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751edcdd0 0x557751ed9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.112+0000 7f771259c640 1 -- 172.21.15.29:0/2553575187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751edcdd0 msgr2=0x557751ed9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:29.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.112+0000 7f771259c640 1 --2- 172.21.15.29:0/2553575187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751edcdd0 0x557751ed9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.112+0000 7f771259c640 1 -- 172.21.15.29:0/2553575187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7c60 msgr2=0x557751ee89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:29.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.112+0000 7f771259c640 1 --2- 172.21.15.29:0/2553575187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7c60 0x557751ee89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.112+0000 7f771259c640 1 -- 172.21.15.29:0/2553575187 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557751ddac20 con 0x557751ed7270 2024-03-11T19:17:29.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.112+0000 7f771259c640 1 --2- 172.21.15.29:0/2553575187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751ed7270 0x557751ed9120 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f77040c45c0 tx=0x7f77040a6000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=916251b0563b4f31 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:29.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.112+0000 7f76fbfff640 1 -- 172.21.15.29:0/2553575187 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77040bbe90 con 0x557751ed7270 2024-03-11T19:17:29.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.113+0000 7f76fbfff640 1 -- 172.21.15.29:0/2553575187 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f77040f2d60 con 0x557751ed7270 2024-03-11T19:17:29.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.113+0000 7f76fbfff640 1 -- 172.21.15.29:0/2553575187 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77040c4af0 con 0x557751ed7270 2024-03-11T19:17:29.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.114+0000 7f771b446740 1 -- 172.21.15.29:0/2553575187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751ed7270 msgr2=0x557751ed9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:29.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.114+0000 7f771b446740 1 --2- 172.21.15.29:0/2553575187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751ed7270 0x557751ed9120 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f77040c45c0 tx=0x7f77040a6000 comp rx=0 tx=0).stop 2024-03-11T19:17:29.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.114+0000 7f771b446740 1 -- 172.21.15.29:0/2553575187 shutdown_connections 2024-03-11T19:17:29.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.114+0000 7f771b446740 1 --2- 172.21.15.29:0/2553575187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751edcdd0 0x557751ed9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.114+0000 7f771b446740 1 --2- 172.21.15.29:0/2553575187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7c60 0x557751ee89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.114+0000 7f771b446740 1 --2- 172.21.15.29:0/2553575187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751ed7270 0x557751ed9120 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.114+0000 7f771b446740 1 -- 172.21.15.29:0/2553575187 >> 172.21.15.29:0/2553575187 conn(0x557751ed27e0 msgr2=0x557751ef3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:29.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.114+0000 7f771b446740 1 -- 172.21.15.29:0/2553575187 shutdown_connections 2024-03-11T19:17:29.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.114+0000 7f771b446740 1 -- 172.21.15.29:0/2553575187 wait complete. 2024-03-11T19:17:29.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.114+0000 7f771b446740 1 -- start start 2024-03-11T19:17:29.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.114+0000 7f771b446740 1 Processor -- start 2024-03-11T19:17:29.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.115+0000 7f771b446740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7270 0x557751ee6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.115+0000 7f771b446740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751ed7c60 0x557751ee6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.115+0000 7f771b446740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751edcdd0 0x557751ee7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.115+0000 7f771b446740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557751ee80f0 con 0x557751ed7270 2024-03-11T19:17:29.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.115+0000 7f771b446740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557751ee8260 con 0x557751edcdd0 2024-03-11T19:17:29.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.115+0000 7f771b446740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557751ee8420 con 0x557751ed7c60 2024-03-11T19:17:29.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.115+0000 7f771259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7270 0x557751ee6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.115+0000 7f771b446740 10 client.0 _ll_register_callbacks cb 0x7f76fc0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:17:29.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.115+0000 7f771259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7270 0x557751ee6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42416/0 (socket says 172.21.15.29:42416) 2024-03-11T19:17:29.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.115+0000 7f771259c640 1 -- 172.21.15.29:0/1263359035 learned_addr learned my addr 172.21.15.29:0/1263359035 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:29.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.115+0000 7f770bfff640 1 --2- 172.21.15.29:0/1263359035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751ed7c60 0x557751ee6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.115+0000 7f7712d9d640 1 --2- 172.21.15.29:0/1263359035 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751edcdd0 0x557751ee7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.115+0000 7f770bfff640 1 -- 172.21.15.29:0/1263359035 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751edcdd0 msgr2=0x557751ee7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:29.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.115+0000 7f770bfff640 1 --2- 172.21.15.29:0/1263359035 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751edcdd0 0x557751ee7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.115+0000 7f770bfff640 1 -- 172.21.15.29:0/1263359035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7270 msgr2=0x557751ee6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:29.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.115+0000 7f770bfff640 1 --2- 172.21.15.29:0/1263359035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7270 0x557751ee6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.115+0000 7f770bfff640 1 -- 172.21.15.29:0/1263359035 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557751e3e6a0 con 0x557751ed7c60 2024-03-11T19:17:29.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.115+0000 7f771259c640 1 --2- 172.21.15.29:0/1263359035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7270 0x557751ee6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:17:29.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.117+0000 7f770bfff640 1 --2- 172.21.15.29:0/1263359035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751ed7c60 0x557751ee6ba0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009730 tx=0x7f76fc005d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11b26d23eb6d4c36 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:29.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.117+0000 7f76f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:17:29.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.118+0000 7f76f9ffb640 1 -- 172.21.15.29:0/1263359035 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc02dd80 con 0x557751ed7c60 2024-03-11T19:17:29.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.118+0000 7f76f9ffb640 1 -- 172.21.15.29:0/1263359035 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f76fc032030 con 0x557751ed7c60 2024-03-11T19:17:29.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.118+0000 7f771b446740 1 -- 172.21.15.29:0/1263359035 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f76fc005020 con 0x557751ed7c60 2024-03-11T19:17:29.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.119+0000 7f76f9ffb640 1 -- 172.21.15.29:0/1263359035 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76fc0055c0 con 0x557751ed7c60 2024-03-11T19:17:29.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.119+0000 7f771b446740 10 client.26791 fetch_fsmap learned FSMap version 27 2024-03-11T19:17:29.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.119+0000 7f771b446740 1 -- 172.21.15.29:0/1263359035 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x557751e008c0 con 0x557751ed7c60 2024-03-11T19:17:29.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.120+0000 7f76f9ffb640 1 -- 172.21.15.29:0/1263359035 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f76fc03b030 con 0x557751ed7c60 2024-03-11T19:17:29.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.120+0000 7f76f9ffb640 1 -- 172.21.15.29:0/1263359035 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f76fc021040 con 0x557751ed7c60 2024-03-11T19:17:29.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.120+0000 7f771b446740 10 client.26791 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:17:29.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.120+0000 7f771b446740 10 client.26791 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:17:29.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.120+0000 7f771b446740 20 client.26791 populate_metadata read hostname 'smithi029' 2024-03-11T19:17:29.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.120+0000 7f771b446740 1 --2- 172.21.15.29:0/1263359035 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557751f444e0 0x557751f46900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.120+0000 7f771b446740 4 client.26791 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:17:29.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.120+0000 7f771b446740 1 -- 172.21.15.29:0/1263359035 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557751b7c720 con 0x557751f444e0 2024-03-11T19:17:29.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.121+0000 7f771259c640 1 --2- 172.21.15.29:0/1263359035 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557751f444e0 0x557751f46900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.121+0000 7f771259c640 1 --2- 172.21.15.29:0/1263359035 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557751f444e0 0x557751f46900 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=20c28dbe06c5d7d3 server_cookie=e530ced8b9f8514f in_seq=0 out_seq=0 2024-03-11T19:17:29.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.121+0000 7f76f9ffb640 10 client.26791 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:17:29.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.122+0000 7f76f9ffb640 1 -- 172.21.15.29:0/1263359035 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557751b7c720 con 0x557751f444e0 2024-03-11T19:17:29.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.122+0000 7f76f9ffb640 10 client.26791 handle_command_reply: tid=0 2024-03-11T19:17:29.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.122+0000 7f771b446740 1 client.26791 shutdown 2024-03-11T19:17:29.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.123+0000 7f771b446740 1 -- 172.21.15.29:0/1263359035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751ed7c60 msgr2=0x557751ee6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:29.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.123+0000 7f771b446740 1 --2- 172.21.15.29:0/1263359035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751ed7c60 0x557751ee6ba0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009730 tx=0x7f76fc005d30 comp rx=0 tx=0).stop 2024-03-11T19:17:29.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.123+0000 7f771b446740 1 -- 172.21.15.29:0/1263359035 shutdown_connections 2024-03-11T19:17:29.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.123+0000 7f771b446740 1 --2- 172.21.15.29:0/1263359035 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x557751f444e0 0x557751f46900 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.123+0000 7f771b446740 1 --2- 172.21.15.29:0/1263359035 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557751edcdd0 0x557751ee7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.123+0000 7f76f9ffb640 0 client.26791 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:17:29.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.123+0000 7f771b446740 1 --2- 172.21.15.29:0/1263359035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557751ed7c60 0x557751ee6ba0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.123+0000 7f771b446740 1 --2- 172.21.15.29:0/1263359035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557751ed7270 0x557751ee6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.123+0000 7f771b446740 1 -- 172.21.15.29:0/1263359035 >> 172.21.15.29:0/1263359035 conn(0x557751ed27e0 msgr2=0x557751ed9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:29.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.123+0000 7f771b446740 1 -- 172.21.15.29:0/1263359035 shutdown_connections 2024-03-11T19:17:29.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.124+0000 7f771b446740 1 -- 172.21.15.29:0/1263359035 wait complete. 2024-03-11T19:17:29.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.124+0000 7f771b446740 20 client.26791 trim_cache size 0 max 16384 2024-03-11T19:17:29.129 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:17:29.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.125+0000 7f7718876640 1 -- 172.21.15.29:0/2533489309 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f77000a1b30 msgr2=0x7f77000a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:29.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.125+0000 7f7718876640 1 --2- 172.21.15.29:0/2533489309 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f77000a1b30 0x7f77000a3ff0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f770c009fd0 tx=0x7f770c004a40 comp rx=0 tx=0).stop 2024-03-11T19:17:29.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.125+0000 7f7718876640 1 -- 172.21.15.29:0/2533489309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7714100080 msgr2=0x7f771419d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:29.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.125+0000 7f7718876640 1 --2- 172.21.15.29:0/2533489309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7714100080 0x7f771419d7b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f770402c7d0 tx=0x7f7704004040 comp rx=0 tx=0).stop 2024-03-11T19:17:29.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.125+0000 7f7718876640 1 -- 172.21.15.29:0/2533489309 shutdown_connections 2024-03-11T19:17:29.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.125+0000 7f7718876640 1 --2- 172.21.15.29:0/2533489309 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f77000a1b30 0x7f77000a3ff0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.125+0000 7f7718876640 1 --2- 172.21.15.29:0/2533489309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7714101c30 0x7f77141a4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.125+0000 7f7718876640 1 --2- 172.21.15.29:0/2533489309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7714101270 0x7f771419dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.125+0000 7f7718876640 1 --2- 172.21.15.29:0/2533489309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7714100080 0x7f771419d7b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.125+0000 7f7718876640 1 -- 172.21.15.29:0/2533489309 >> 172.21.15.29:0/2533489309 conn(0x7f77140fb610 msgr2=0x7f77141026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:29.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.126+0000 7f7718876640 1 -- 172.21.15.29:0/2533489309 shutdown_connections 2024-03-11T19:17:29.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.126+0000 7f7718876640 1 -- 172.21.15.29:0/2533489309 wait complete. 2024-03-11T19:17:29.147 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:29 smithi029 ceph-mon[32439]: pgmap v581: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 59 MiB/s wr, 1.03k op/s 2024-03-11T19:17:29.147 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:29 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1015484871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:17:29.148 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:17:29.148 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:17:29.314 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:29 smithi190 ceph-mon[36292]: pgmap v581: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 59 MiB/s wr, 1.03k op/s 2024-03-11T19:17:29.315 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:29 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1015484871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:17:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.329+0000 7f8cdf59e640 1 -- 172.21.15.29:0/1668913093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ce0101170 msgr2=0x7f8ce01015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.329+0000 7f8cdf59e640 1 --2- 172.21.15.29:0/1668913093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ce0101170 0x7f8ce01015f0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8009900 tx=0x7f8cc802bc30 comp rx=0 tx=0).stop 2024-03-11T19:17:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.329+0000 7f8cdf59e640 1 -- 172.21.15.29:0/1668913093 shutdown_connections 2024-03-11T19:17:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.329+0000 7f8cdf59e640 1 --2- 172.21.15.29:0/1668913093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ce0101b30 0x7f8ce01083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.329+0000 7f8cdf59e640 1 --2- 172.21.15.29:0/1668913093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ce0101170 0x7f8ce01015f0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.329+0000 7f8cdf59e640 1 --2- 172.21.15.29:0/1668913093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ce00fff80 0x7f8ce01003a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.329+0000 7f8cdf59e640 1 -- 172.21.15.29:0/1668913093 >> 172.21.15.29:0/1668913093 conn(0x7f8ce00fb4f0 msgr2=0x7f8ce00fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.329+0000 7f8cdf59e640 1 -- 172.21.15.29:0/1668913093 shutdown_connections 2024-03-11T19:17:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.329+0000 7f8cdf59e640 1 -- 172.21.15.29:0/1668913093 wait complete. 2024-03-11T19:17:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.330+0000 7f8cdf59e640 1 Processor -- start 2024-03-11T19:17:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.330+0000 7f8cdf59e640 1 -- start start 2024-03-11T19:17:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.330+0000 7f8cdf59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ce00fff80 0x7f8ce0197000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.331+0000 7f8cdf59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ce0101170 0x7f8ce0197540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.331+0000 7f8cdf59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ce0101b30 0x7f8ce019e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.331+0000 7f8cdf59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce019eaf0 con 0x7f8ce0101170 2024-03-11T19:17:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.331+0000 7f8cdf59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce019ec60 con 0x7f8ce0101b30 2024-03-11T19:17:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.331+0000 7f8cdf59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce019ee20 con 0x7f8ce00fff80 2024-03-11T19:17:29.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.331+0000 7f8cd5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ce0101170 0x7f8ce0197540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.331+0000 7f8cd5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ce0101170 0x7f8ce0197540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42426/0 (socket says 172.21.15.29:42426) 2024-03-11T19:17:29.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.331+0000 7f8cd5d9b640 1 -- 172.21.15.29:0/2185482567 learned_addr learned my addr 172.21.15.29:0/2185482567 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:29.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.331+0000 7f8cde59c640 1 --2- 172.21.15.29:0/2185482567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ce00fff80 0x7f8ce0197000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.331+0000 7f8cded9d640 1 --2- 172.21.15.29:0/2185482567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ce0101b30 0x7f8ce019e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.331+0000 7f8cd5d9b640 1 -- 172.21.15.29:0/2185482567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ce00fff80 msgr2=0x7f8ce0197000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:29.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.331+0000 7f8cd5d9b640 1 --2- 172.21.15.29:0/2185482567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ce00fff80 0x7f8ce0197000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.331+0000 7f8cd5d9b640 1 -- 172.21.15.29:0/2185482567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ce0101b30 msgr2=0x7f8ce019e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:29.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.331+0000 7f8cd5d9b640 1 --2- 172.21.15.29:0/2185482567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ce0101b30 0x7f8ce019e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.331+0000 7f8cd5d9b640 1 -- 172.21.15.29:0/2185482567 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc8009560 con 0x7f8ce0101170 2024-03-11T19:17:29.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.331+0000 7f8cd5d9b640 1 --2- 172.21.15.29:0/2185482567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ce0101170 0x7f8ce0197540 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8036040 tx=0x7f8cc8004140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:29.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.332+0000 7f8cd7fff640 1 -- 172.21.15.29:0/2185482567 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc80035f0 con 0x7f8ce0101170 2024-03-11T19:17:29.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.332+0000 7f8cdf59e640 1 -- 172.21.15.29:0/2185482567 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce019f0a0 con 0x7f8ce0101170 2024-03-11T19:17:29.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.332+0000 7f8cd7fff640 1 -- 172.21.15.29:0/2185482567 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8cc802de90 con 0x7f8ce0101170 2024-03-11T19:17:29.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.332+0000 7f8cdf59e640 1 -- 172.21.15.29:0/2185482567 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce019f600 con 0x7f8ce0101170 2024-03-11T19:17:29.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.332+0000 7f8cd7fff640 1 -- 172.21.15.29:0/2185482567 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc8005490 con 0x7f8ce0101170 2024-03-11T19:17:29.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.334+0000 7f8cd7fff640 1 -- 172.21.15.29:0/2185482567 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8cc8003750 con 0x7f8ce0101170 2024-03-11T19:17:29.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.334+0000 7f8ce7796740 1 Processor -- start 2024-03-11T19:17:29.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.334+0000 7f8ce7796740 1 -- start start 2024-03-11T19:17:29.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8ce7796740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029ad480 0x55c7029ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8ce7796740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ade70 0x55c7029b2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8ce7796740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 0x55c7029b5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8ce7796740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c702969960 con 0x55c7029b2fe0 2024-03-11T19:17:29.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8ce7796740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c70291d310 con 0x55c7029ad480 2024-03-11T19:17:29.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8ce7796740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c70291a840 con 0x55c7029ade70 2024-03-11T19:17:29.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8cd7fff640 1 --2- 172.21.15.29:0/2185482567 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8ccc0a1ab0 0x7f8ccc0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8cd5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 0x55c7029b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8cded9d640 1 --2- 172.21.15.29:0/2185482567 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8ccc0a1ab0 0x7f8ccc0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8cd5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 0x55c7029b5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42442/0 (socket says 172.21.15.29:42442) 2024-03-11T19:17:29.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8cd5d9b640 1 -- 172.21.15.29:0/4088849806 learned_addr learned my addr 172.21.15.29:0/4088849806 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:29.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8cde59c640 1 --2- 172.21.15.29:0/4088849806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ade70 0x55c7029b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8cd7fff640 1 -- 172.21.15.29:0/2185482567 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f8cc80ef4e0 con 0x7f8ce0101170 2024-03-11T19:17:29.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8cd5d9b640 1 -- 172.21.15.29:0/4088849806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ade70 msgr2=0x55c7029b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:29.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8cd5d9b640 1 --2- 172.21.15.29:0/4088849806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ade70 0x55c7029b2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8cd5d9b640 1 -- 172.21.15.29:0/4088849806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029ad480 msgr2=0x55c7029ad8a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:17:29.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8cd5d9b640 1 --2- 172.21.15.29:0/4088849806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029ad480 0x55c7029ad8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8cd5d9b640 1 -- 172.21.15.29:0/4088849806 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c7028d8060 con 0x55c7029b2fe0 2024-03-11T19:17:29.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8cd5d9b640 1 --2- 172.21.15.29:0/4088849806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 0x55c7029b5470 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8005c20 tx=0x7f8cc8087d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab262f5e034987eb server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:29.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8cded9d640 1 --2- 172.21.15.29:0/2185482567 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8ccc0a1ab0 0x7f8ccc0a3f70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8005fd0 tx=0x7f8cd8009600 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:29.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.335+0000 7f8cd4d99640 1 -- 172.21.15.29:0/4088849806 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc8003e20 con 0x55c7029b2fe0 2024-03-11T19:17:29.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.336+0000 7f8cd4d99640 1 -- 172.21.15.29:0/4088849806 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8cc8004dc0 con 0x55c7029b2fe0 2024-03-11T19:17:29.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.336+0000 7f8cd4d99640 1 -- 172.21.15.29:0/4088849806 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc80c4d20 con 0x55c7029b2fe0 2024-03-11T19:17:29.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.336+0000 7f8ce7796740 1 -- 172.21.15.29:0/4088849806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 msgr2=0x55c7029b5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:29.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.336+0000 7f8ce7796740 1 --2- 172.21.15.29:0/4088849806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 0x55c7029b5470 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8005c20 tx=0x7f8cc8087d20 comp rx=0 tx=0).stop 2024-03-11T19:17:29.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.336+0000 7f8ce7796740 1 -- 172.21.15.29:0/4088849806 shutdown_connections 2024-03-11T19:17:29.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.336+0000 7f8ce7796740 1 --2- 172.21.15.29:0/4088849806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 0x55c7029b5470 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.336+0000 7f8ce7796740 1 --2- 172.21.15.29:0/4088849806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ade70 0x55c7029b2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.336+0000 7f8ce7796740 1 --2- 172.21.15.29:0/4088849806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029ad480 0x55c7029ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.336+0000 7f8ce7796740 1 -- 172.21.15.29:0/4088849806 >> 172.21.15.29:0/4088849806 conn(0x55c7029a89f0 msgr2=0x55c7029a8df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:29.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.336+0000 7f8ce7796740 1 -- 172.21.15.29:0/4088849806 shutdown_connections 2024-03-11T19:17:29.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.336+0000 7f8ce7796740 1 -- 172.21.15.29:0/4088849806 wait complete. 2024-03-11T19:17:29.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8ce7796740 1 -- start start 2024-03-11T19:17:29.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8ce7796740 1 Processor -- start 2024-03-11T19:17:29.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8ce7796740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ad480 0x55c7029bb920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8ce7796740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029ade70 0x55c7029bbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8ce7796740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 0x55c7029c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8ce7796740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c702916020 con 0x55c7029b2fe0 2024-03-11T19:17:29.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8ce7796740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7025e9430 con 0x55c7029ade70 2024-03-11T19:17:29.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8ce7796740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7025a5e00 con 0x55c7029ad480 2024-03-11T19:17:29.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8ce7796740 10 client.0 _ll_register_callbacks cb 0x55c7028bdcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:17:29.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8cded9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 0x55c7029c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8cded9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 0x55c7029c2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42458/0 (socket says 172.21.15.29:42458) 2024-03-11T19:17:29.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8cded9d640 1 -- 172.21.15.29:0/739583703 learned_addr learned my addr 172.21.15.29:0/739583703 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:29.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8cde59c640 1 --2- 172.21.15.29:0/739583703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ad480 0x55c7029bb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8cd5d9b640 1 --2- 172.21.15.29:0/739583703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029ade70 0x55c7029bbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8cded9d640 1 -- 172.21.15.29:0/739583703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ad480 msgr2=0x55c7029bb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:29.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8cded9d640 1 --2- 172.21.15.29:0/739583703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ad480 0x55c7029bb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8cded9d640 1 -- 172.21.15.29:0/739583703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029ade70 msgr2=0x55c7029bbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:29.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8cded9d640 1 --2- 172.21.15.29:0/739583703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029ade70 0x55c7029bbe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8cded9d640 1 -- 172.21.15.29:0/739583703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce019f600 con 0x55c7029b2fe0 2024-03-11T19:17:29.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.337+0000 7f8cd5d9b640 1 --2- 172.21.15.29:0/739583703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029ade70 0x55c7029bbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:17:29.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.338+0000 7f8cded9d640 1 --2- 172.21.15.29:0/739583703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 0x55c7029c2e30 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8012bc0 tx=0x7f8cd8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e7b53186b2cf618 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:29.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.338+0000 7f8cc5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:17:29.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.338+0000 7f8cc5ffb640 1 -- 172.21.15.29:0/739583703 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd8011e30 con 0x55c7029b2fe0 2024-03-11T19:17:29.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.338+0000 7f8cc5ffb640 1 -- 172.21.15.29:0/739583703 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f8cd801f450 con 0x55c7029b2fe0 2024-03-11T19:17:29.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.338+0000 7f8ce7796740 1 -- 172.21.15.29:0/739583703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c7026157e0 con 0x55c7029b2fe0 2024-03-11T19:17:29.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.338+0000 7f8cc5ffb640 1 -- 172.21.15.29:0/739583703 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8cd801f610 con 0x55c7029b2fe0 2024-03-11T19:17:29.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.338+0000 7f8ce7796740 10 client.17283 fetch_fsmap learned FSMap version 27 2024-03-11T19:17:29.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.338+0000 7f8ce7796740 1 -- 172.21.15.29:0/739583703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55c7029146a0 con 0x55c7029b2fe0 2024-03-11T19:17:29.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.338+0000 7f8cc5ffb640 1 -- 172.21.15.29:0/739583703 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55c7026157e0 con 0x55c7029b2fe0 2024-03-11T19:17:29.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.338+0000 7f8cc5ffb640 1 -- 172.21.15.29:0/739583703 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8cd80277a0 con 0x55c7029b2fe0 2024-03-11T19:17:29.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.338+0000 7f8ce7796740 10 client.17283 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:17:29.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.338+0000 7f8ce7796740 10 client.17283 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:17:29.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.338+0000 7f8ce7796740 20 client.17283 populate_metadata read hostname 'smithi029' 2024-03-11T19:17:29.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.338+0000 7f8ce7796740 1 --2- 172.21.15.29:0/739583703 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c7029c7c20 0x55c7029ca040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.338+0000 7f8ce7796740 4 client.17283 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:17:29.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.338+0000 7f8ce7796740 1 -- 172.21.15.29:0/739583703 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c702652720 con 0x55c7029c7c20 2024-03-11T19:17:29.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.339+0000 7f8cde59c640 1 --2- 172.21.15.29:0/739583703 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c7029c7c20 0x55c7029ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.339+0000 7f8cde59c640 1 --2- 172.21.15.29:0/739583703 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c7029c7c20 0x55c7029ca040 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fc1feecf3886054 server_cookie=fa0b0637eb6246ce in_seq=0 out_seq=0 2024-03-11T19:17:29.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.339+0000 7f8cc5ffb640 10 client.17283 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:17:29.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.340+0000 7f8cc5ffb640 1 -- 172.21.15.29:0/739583703 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c702652720 con 0x55c7029c7c20 2024-03-11T19:17:29.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.340+0000 7f8cc5ffb640 10 client.17283 handle_command_reply: tid=0 2024-03-11T19:17:29.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.340+0000 7f8ce7796740 1 client.17283 shutdown 2024-03-11T19:17:29.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.340+0000 7f8ce7796740 1 -- 172.21.15.29:0/739583703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 msgr2=0x55c7029c2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:29.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.340+0000 7f8ce7796740 1 --2- 172.21.15.29:0/739583703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 0x55c7029c2e30 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8012bc0 tx=0x7f8cd8004a40 comp rx=0 tx=0).stop 2024-03-11T19:17:29.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.340+0000 7f8ce7796740 1 -- 172.21.15.29:0/739583703 shutdown_connections 2024-03-11T19:17:29.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.340+0000 7f8ce7796740 1 --2- 172.21.15.29:0/739583703 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c7029c7c20 0x55c7029ca040 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.340+0000 7f8ce7796740 1 --2- 172.21.15.29:0/739583703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 0x55c7029c2e30 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.340+0000 7f8ce7796740 1 --2- 172.21.15.29:0/739583703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029ade70 0x55c7029bbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.340+0000 7f8cc5ffb640 0 client.17283 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:17:29.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.340+0000 7f8ce7796740 1 --2- 172.21.15.29:0/739583703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ad480 0x55c7029bb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.340+0000 7f8ce7796740 1 -- 172.21.15.29:0/739583703 >> 172.21.15.29:0/739583703 conn(0x55c7029a89f0 msgr2=0x55c7029a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:29.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.340+0000 7f8ce7796740 1 -- 172.21.15.29:0/739583703 shutdown_connections 2024-03-11T19:17:29.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.340+0000 7f8ce7796740 1 -- 172.21.15.29:0/739583703 wait complete. 2024-03-11T19:17:29.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.340+0000 7f8ce7796740 20 client.17283 trim_cache size 0 max 16384 2024-03-11T19:17:29.351 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:29 smithi170 ceph-mon[35690]: pgmap v581: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 59 MiB/s wr, 1.03k op/s 2024-03-11T19:17:29.351 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:29 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1015484871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:17:29.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8ce7796740 1 Processor -- start 2024-03-11T19:17:29.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8ce7796740 1 -- start start 2024-03-11T19:17:29.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8ce7796740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029ad480 0x55c7029ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8ce7796740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ade70 0x55c7029c2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8ce7796740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 0x55c7029bb920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8ce7796740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c70289b130 con 0x55c7029b2fe0 2024-03-11T19:17:29.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8ce7796740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7029aed70 con 0x55c7029ad480 2024-03-11T19:17:29.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8ce7796740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7029ad040 con 0x55c7029ade70 2024-03-11T19:17:29.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8cded9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 0x55c7029bb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8cded9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 0x55c7029bb920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42470/0 (socket says 172.21.15.29:42470) 2024-03-11T19:17:29.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8cded9d640 1 -- 172.21.15.29:0/3617006970 learned_addr learned my addr 172.21.15.29:0/3617006970 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:29.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8cde59c640 1 --2- 172.21.15.29:0/3617006970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029ad480 0x55c7029ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8cd5d9b640 1 --2- 172.21.15.29:0/3617006970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ade70 0x55c7029c2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8cded9d640 1 -- 172.21.15.29:0/3617006970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ade70 msgr2=0x55c7029c2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:29.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8cded9d640 1 --2- 172.21.15.29:0/3617006970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ade70 0x55c7029c2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8cded9d640 1 -- 172.21.15.29:0/3617006970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029ad480 msgr2=0x55c7029ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:29.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8cded9d640 1 --2- 172.21.15.29:0/3617006970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029ad480 0x55c7029ce9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8cded9d640 1 -- 172.21.15.29:0/3617006970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c7028b0c20 con 0x55c7029b2fe0 2024-03-11T19:17:29.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8cde59c640 1 --2- 172.21.15.29:0/3617006970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029ad480 0x55c7029ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:17:29.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.352+0000 7f8cded9d640 1 --2- 172.21.15.29:0/3617006970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 0x55c7029bb920 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8027bb0 tx=0x7f8cd80127f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c87273c884a69 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:29.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.353+0000 7f8cc7fff640 1 -- 172.21.15.29:0/3617006970 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd8027d20 con 0x55c7029b2fe0 2024-03-11T19:17:29.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.353+0000 7f8cc7fff640 1 -- 172.21.15.29:0/3617006970 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8cd8027e80 con 0x55c7029b2fe0 2024-03-11T19:17:29.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.353+0000 7f8cc7fff640 1 -- 172.21.15.29:0/3617006970 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd8027850 con 0x55c7029b2fe0 2024-03-11T19:17:29.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.353+0000 7f8ce7796740 1 -- 172.21.15.29:0/3617006970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 msgr2=0x55c7029bb920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:29.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.353+0000 7f8ce7796740 1 --2- 172.21.15.29:0/3617006970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 0x55c7029bb920 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8027bb0 tx=0x7f8cd80127f0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.353+0000 7f8ce7796740 1 -- 172.21.15.29:0/3617006970 shutdown_connections 2024-03-11T19:17:29.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.353+0000 7f8ce7796740 1 --2- 172.21.15.29:0/3617006970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029b2fe0 0x55c7029bb920 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.353+0000 7f8ce7796740 1 --2- 172.21.15.29:0/3617006970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ade70 0x55c7029c2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.353+0000 7f8ce7796740 1 --2- 172.21.15.29:0/3617006970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029ad480 0x55c7029ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.353+0000 7f8ce7796740 1 -- 172.21.15.29:0/3617006970 >> 172.21.15.29:0/3617006970 conn(0x55c7029a89f0 msgr2=0x55c7029a8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:29.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.353+0000 7f8ce7796740 1 -- 172.21.15.29:0/3617006970 shutdown_connections 2024-03-11T19:17:29.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.353+0000 7f8ce7796740 1 -- 172.21.15.29:0/3617006970 wait complete. 2024-03-11T19:17:29.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.353+0000 7f8ce7796740 1 -- start start 2024-03-11T19:17:29.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8ce7796740 1 Processor -- start 2024-03-11T19:17:29.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8ce7796740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029ad480 0x55c7029c0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8ce7796740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ade70 0x55c7029c1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8ce7796740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029b2fe0 0x55c7029c2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8ce7796740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7029c2670 con 0x55c7029ad480 2024-03-11T19:17:29.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8ce7796740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7029c27e0 con 0x55c7029b2fe0 2024-03-11T19:17:29.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8ce7796740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7029c29a0 con 0x55c7029ade70 2024-03-11T19:17:29.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8ce7796740 10 client.0 _ll_register_callbacks cb 0x7f8cc8005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:17:29.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8cde59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029ad480 0x55c7029c0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8cde59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029ad480 0x55c7029c0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42484/0 (socket says 172.21.15.29:42484) 2024-03-11T19:17:29.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8cde59c640 1 -- 172.21.15.29:0/2990323014 learned_addr learned my addr 172.21.15.29:0/2990323014 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:29.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8cd5d9b640 1 --2- 172.21.15.29:0/2990323014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ade70 0x55c7029c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8cded9d640 1 --2- 172.21.15.29:0/2990323014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029b2fe0 0x55c7029c2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8cde59c640 1 -- 172.21.15.29:0/2990323014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ade70 msgr2=0x55c7029c1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:29.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8cde59c640 1 --2- 172.21.15.29:0/2990323014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ade70 0x55c7029c1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8cde59c640 1 -- 172.21.15.29:0/2990323014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029b2fe0 msgr2=0x55c7029c2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:29.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8cde59c640 1 --2- 172.21.15.29:0/2990323014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029b2fe0 0x55c7029c2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8cde59c640 1 -- 172.21.15.29:0/2990323014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd0004040 con 0x55c7029ad480 2024-03-11T19:17:29.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8cded9d640 1 --2- 172.21.15.29:0/2990323014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029b2fe0 0x55c7029c2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:17:29.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8cde59c640 1 --2- 172.21.15.29:0/2990323014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029ad480 0x55c7029c0be0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x55c7029ae270 tx=0x7f8cd001e700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f57cde73dbf1777a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:29.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8cc5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:17:29.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8cc5ffb640 1 -- 172.21.15.29:0/2990323014 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0017070 con 0x55c7029ad480 2024-03-11T19:17:29.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8cc5ffb640 1 -- 172.21.15.29:0/2990323014 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f8cd0005020 con 0x55c7029ad480 2024-03-11T19:17:29.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.354+0000 7f8ce7796740 1 -- 172.21.15.29:0/2990323014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c7026a1890 con 0x55c7029ad480 2024-03-11T19:17:29.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.355+0000 7f8cc5ffb640 1 -- 172.21.15.29:0/2990323014 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8cd001e8f0 con 0x55c7029ad480 2024-03-11T19:17:29.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.355+0000 7f8ce7796740 10 client.17289 fetch_fsmap learned FSMap version 27 2024-03-11T19:17:29.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.355+0000 7f8ce7796740 1 -- 172.21.15.29:0/2990323014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55c7029146a0 con 0x55c7029ad480 2024-03-11T19:17:29.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.355+0000 7f8cc5ffb640 1 -- 172.21.15.29:0/2990323014 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55c7026a1890 con 0x55c7029ad480 2024-03-11T19:17:29.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.355+0000 7f8cc5ffb640 1 -- 172.21.15.29:0/2990323014 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8cd001d040 con 0x55c7029ad480 2024-03-11T19:17:29.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.355+0000 7f8ce7796740 10 client.17289 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:17:29.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.355+0000 7f8ce7796740 10 client.17289 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:17:29.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.355+0000 7f8ce7796740 20 client.17289 populate_metadata read hostname 'smithi029' 2024-03-11T19:17:29.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.355+0000 7f8ce7796740 1 --2- 172.21.15.29:0/2990323014 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c7029c7c20 0x55c702a21230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:29.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.355+0000 7f8ce7796740 4 client.17289 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:17:29.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.355+0000 7f8ce7796740 1 -- 172.21.15.29:0/2990323014 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c7025c1700 con 0x55c7029c7c20 2024-03-11T19:17:29.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.355+0000 7f8cd5d9b640 1 --2- 172.21.15.29:0/2990323014 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c7029c7c20 0x55c702a21230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:29.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8cc5ffb640 10 client.17289 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:17:29.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8cd5d9b640 1 --2- 172.21.15.29:0/2990323014 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c7029c7c20 0x55c702a21230 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7fa3625180c55c8 server_cookie=5e6f2e37b0cb6ba7 in_seq=0 out_seq=0 2024-03-11T19:17:29.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8cc5ffb640 1 -- 172.21.15.29:0/2990323014 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c7025c1700 con 0x55c7029c7c20 2024-03-11T19:17:29.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8cc5ffb640 10 client.17289 handle_command_reply: tid=0 2024-03-11T19:17:29.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8ce7796740 1 client.17289 shutdown 2024-03-11T19:17:29.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8ce7796740 1 -- 172.21.15.29:0/2990323014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029ad480 msgr2=0x55c7029c0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:29.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8ce7796740 1 --2- 172.21.15.29:0/2990323014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029ad480 0x55c7029c0be0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x55c7029ae270 tx=0x7f8cd001e700 comp rx=0 tx=0).stop 2024-03-11T19:17:29.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8ce7796740 1 -- 172.21.15.29:0/2990323014 shutdown_connections 2024-03-11T19:17:29.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8ce7796740 1 --2- 172.21.15.29:0/2990323014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c7029b2fe0 0x55c7029c2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8ce7796740 1 --2- 172.21.15.29:0/2990323014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c7029ade70 0x55c7029c1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8ce7796740 1 --2- 172.21.15.29:0/2990323014 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c7029c7c20 0x55c702a21230 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8ce7796740 1 --2- 172.21.15.29:0/2990323014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c7029ad480 0x55c7029c0be0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8cc5ffb640 0 client.17289 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:17:29.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8ce7796740 1 -- 172.21.15.29:0/2990323014 >> 172.21.15.29:0/2990323014 conn(0x55c7029a89f0 msgr2=0x55c7029af3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:29.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8ce7796740 1 -- 172.21.15.29:0/2990323014 shutdown_connections 2024-03-11T19:17:29.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8ce7796740 1 -- 172.21.15.29:0/2990323014 wait complete. 2024-03-11T19:17:29.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8ce7796740 20 client.17289 trim_cache size 0 max 16384 2024-03-11T19:17:29.368 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:17:29.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8cdf59e640 1 -- 172.21.15.29:0/2185482567 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8ccc0a1ab0 msgr2=0x7f8ccc0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:29.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8cdf59e640 1 --2- 172.21.15.29:0/2185482567 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8ccc0a1ab0 0x7f8ccc0a3f70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8005fd0 tx=0x7f8cd8009600 comp rx=0 tx=0).stop 2024-03-11T19:17:29.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8cdf59e640 1 -- 172.21.15.29:0/2185482567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ce0101170 msgr2=0x7f8ce0197540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:29.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8cdf59e640 1 --2- 172.21.15.29:0/2185482567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ce0101170 0x7f8ce0197540 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8036040 tx=0x7f8cc8004140 comp rx=0 tx=0).stop 2024-03-11T19:17:29.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8cdf59e640 1 -- 172.21.15.29:0/2185482567 shutdown_connections 2024-03-11T19:17:29.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8cdf59e640 1 --2- 172.21.15.29:0/2185482567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ce0101b30 0x7f8ce019e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8cdf59e640 1 --2- 172.21.15.29:0/2185482567 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8ccc0a1ab0 0x7f8ccc0a3f70 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8cdf59e640 1 --2- 172.21.15.29:0/2185482567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ce0101170 0x7f8ce0197540 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8cdf59e640 1 --2- 172.21.15.29:0/2185482567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ce00fff80 0x7f8ce0197000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:29.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.356+0000 7f8cdf59e640 1 -- 172.21.15.29:0/2185482567 >> 172.21.15.29:0/2185482567 conn(0x7f8ce00fb4f0 msgr2=0x7f8ce0069670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:29.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.357+0000 7f8cdf59e640 1 -- 172.21.15.29:0/2185482567 shutdown_connections 2024-03-11T19:17:29.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:29.357+0000 7f8cdf59e640 1 -- 172.21.15.29:0/2185482567 wait complete. 2024-03-11T19:17:30.372 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:17:30.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:17:30.372 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:17:30.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.563+0000 7f413667d640 1 -- 172.21.15.29:0/1815855560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4130108780 msgr2=0x7f4130108be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:30.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.563+0000 7f413667d640 1 --2- 172.21.15.29:0/1815855560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4130108780 0x7f4130108be0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4128009ca0 tx=0x7f412802c100 comp rx=0 tx=0).stop 2024-03-11T19:17:30.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.563+0000 7f413667d640 1 -- 172.21.15.29:0/1815855560 shutdown_connections 2024-03-11T19:17:30.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.563+0000 7f413667d640 1 --2- 172.21.15.29:0/1815855560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4130109120 0x7f413010d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.563+0000 7f413667d640 1 --2- 172.21.15.29:0/1815855560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4130108780 0x7f4130108be0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.563+0000 7f413667d640 1 --2- 172.21.15.29:0/1815855560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41300724d0 0x7f41300728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.563+0000 7f413667d640 1 -- 172.21.15.29:0/1815855560 >> 172.21.15.29:0/1815855560 conn(0x7f413006dfa0 msgr2=0x7f41300703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:30.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.563+0000 7f413667d640 1 -- 172.21.15.29:0/1815855560 shutdown_connections 2024-03-11T19:17:30.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.564+0000 7f413667d640 1 -- 172.21.15.29:0/1815855560 wait complete. 2024-03-11T19:17:30.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.564+0000 7f413667d640 1 Processor -- start 2024-03-11T19:17:30.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.566+0000 7f413667d640 1 -- start start 2024-03-11T19:17:30.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.567+0000 7f413667d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41300724d0 0x7f41301a5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.567+0000 7f413667d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4130108780 0x7f41301a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.567+0000 7f413667d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4130109120 0x7f41301ad300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.567+0000 7f413667d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41301ad840 con 0x7f4130108780 2024-03-11T19:17:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.567+0000 7f413667d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41301ad9b0 con 0x7f4130109120 2024-03-11T19:17:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.567+0000 7f413667d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41301adb70 con 0x7f41300724d0 2024-03-11T19:17:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.567+0000 7f4135e7c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4130109120 0x7f41301ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.567+0000 7f4135e7c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4130109120 0x7f41301ad300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:50198/0 (socket says 172.21.15.29:50198) 2024-03-11T19:17:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.567+0000 7f4135e7c640 1 -- 172.21.15.29:0/4069412762 learned_addr learned my addr 172.21.15.29:0/4069412762 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.568+0000 7f413567b640 1 --2- 172.21.15.29:0/4069412762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41300724d0 0x7f41301a5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.568+0000 7f4134e7a640 1 --2- 172.21.15.29:0/4069412762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4130108780 0x7f41301a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.568+0000 7f4135e7c640 1 -- 172.21.15.29:0/4069412762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41300724d0 msgr2=0x7f41301a5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.568+0000 7f4135e7c640 1 --2- 172.21.15.29:0/4069412762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41300724d0 0x7f41301a5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.568+0000 7f4135e7c640 1 -- 172.21.15.29:0/4069412762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4130108780 msgr2=0x7f41301a6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.568+0000 7f4135e7c640 1 --2- 172.21.15.29:0/4069412762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4130108780 0x7f41301a6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.568+0000 7f4135e7c640 1 -- 172.21.15.29:0/4069412762 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4128009900 con 0x7f4130109120 2024-03-11T19:17:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.568+0000 7f4135e7c640 1 --2- 172.21.15.29:0/4069412762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4130109120 0x7f41301ad300 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f412000e970 tx=0x7f412000ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:30.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.569+0000 7f41267fc640 1 -- 172.21.15.29:0/4069412762 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412000ccc0 con 0x7f4130109120 2024-03-11T19:17:30.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.569+0000 7f41267fc640 1 -- 172.21.15.29:0/4069412762 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f412000ce20 con 0x7f4130109120 2024-03-11T19:17:30.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.569+0000 7f41267fc640 1 -- 172.21.15.29:0/4069412762 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4120010430 con 0x7f4130109120 2024-03-11T19:17:30.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.570+0000 7f413667d640 1 -- 172.21.15.29:0/4069412762 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41301ade50 con 0x7f4130109120 2024-03-11T19:17:30.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.570+0000 7f413667d640 1 -- 172.21.15.29:0/4069412762 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41301ae280 con 0x7f4130109120 2024-03-11T19:17:30.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.573+0000 7f41267fc640 1 -- 172.21.15.29:0/4069412762 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4120004f10 con 0x7f4130109120 2024-03-11T19:17:30.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.573+0000 7f413a58e740 1 Processor -- start 2024-03-11T19:17:30.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.574+0000 7f41267fc640 1 --2- 172.21.15.29:0/4069412762 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f411c0a1b30 0x7f411c0a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.574+0000 7f413a58e740 1 -- start start 2024-03-11T19:17:30.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.574+0000 7f41267fc640 1 -- 172.21.15.29:0/4069412762 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4120014070 con 0x7f4130109120 2024-03-11T19:17:30.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.574+0000 7f413a58e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c82b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.574+0000 7f413a58e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c838e0 0x5643d9c83d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.574+0000 7f413a58e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c842a0 0x5643d9c8ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.574+0000 7f413a58e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d9bf1f90 con 0x5643d9c838e0 2024-03-11T19:17:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.574+0000 7f413a58e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d9bef4c0 con 0x5643d9c842a0 2024-03-11T19:17:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.574+0000 7f413a58e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d9beaca0 con 0x5643d9c82710 2024-03-11T19:17:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.574+0000 7f4134e7a640 1 --2- 172.21.15.29:0/4069412762 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f411c0a1b30 0x7f411c0a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.575+0000 7f413567b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c82b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.575+0000 7f413567b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c82b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:54088/0 (socket says 172.21.15.29:54088) 2024-03-11T19:17:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.575+0000 7f413567b640 1 -- 172.21.15.29:0/2949562412 learned_addr learned my addr 172.21.15.29:0/2949562412 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.575+0000 7f4135e7c640 1 --2- 172.21.15.29:0/2949562412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c842a0 0x5643d9c8ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.575+0000 7f4134e7a640 1 --2- 172.21.15.29:0/2949562412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c838e0 0x5643d9c83d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.575+0000 7f413567b640 1 -- 172.21.15.29:0/2949562412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c842a0 msgr2=0x5643d9c8ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.575+0000 7f413567b640 1 --2- 172.21.15.29:0/2949562412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c842a0 0x5643d9c8ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.575+0000 7f413567b640 1 -- 172.21.15.29:0/2949562412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c838e0 msgr2=0x5643d9c83d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.575+0000 7f413567b640 1 --2- 172.21.15.29:0/2949562412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c838e0 0x5643d9c83d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.575+0000 7f413567b640 1 -- 172.21.15.29:0/2949562412 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5643d9bad0b0 con 0x5643d9c82710 2024-03-11T19:17:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.575+0000 7f4134e7a640 1 --2- 172.21.15.29:0/2949562412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c838e0 0x5643d9c83d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:17:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.575+0000 7f413567b640 1 --2- 172.21.15.29:0/2949562412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c82b30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f412c009cf0 tx=0x7f412c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fde3845ed084b706 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.576+0000 7f411b7fe640 1 -- 172.21.15.29:0/2949562412 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c009ea0 con 0x5643d9c82710 2024-03-11T19:17:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.576+0000 7f4134e7a640 1 --2- 172.21.15.29:0/4069412762 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f411c0a1b30 0x7f411c0a3ff0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f412802c710 tx=0x7f4128009360 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.576+0000 7f411b7fe640 1 -- 172.21.15.29:0/2949562412 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f412c004230 con 0x5643d9c82710 2024-03-11T19:17:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.576+0000 7f411b7fe640 1 -- 172.21.15.29:0/2949562412 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c019800 con 0x5643d9c82710 2024-03-11T19:17:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.577+0000 7f413a58e740 1 -- 172.21.15.29:0/2949562412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 msgr2=0x5643d9c82b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.577+0000 7f413a58e740 1 --2- 172.21.15.29:0/2949562412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c82b30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f412c009cf0 tx=0x7f412c011810 comp rx=0 tx=0).stop 2024-03-11T19:17:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.577+0000 7f413a58e740 1 -- 172.21.15.29:0/2949562412 shutdown_connections 2024-03-11T19:17:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.577+0000 7f413a58e740 1 --2- 172.21.15.29:0/2949562412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c842a0 0x5643d9c8ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.577+0000 7f413a58e740 1 --2- 172.21.15.29:0/2949562412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c838e0 0x5643d9c83d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.577+0000 7f413a58e740 1 --2- 172.21.15.29:0/2949562412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c82b30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.577+0000 7f413a58e740 1 -- 172.21.15.29:0/2949562412 >> 172.21.15.29:0/2949562412 conn(0x5643d9c7d8f0 msgr2=0x5643d9c7dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.577+0000 7f413a58e740 1 -- 172.21.15.29:0/2949562412 shutdown_connections 2024-03-11T19:17:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.577+0000 7f413a58e740 1 -- 172.21.15.29:0/2949562412 wait complete. 2024-03-11T19:17:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.577+0000 7f413a58e740 1 -- start start 2024-03-11T19:17:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.577+0000 7f413a58e740 1 Processor -- start 2024-03-11T19:17:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.578+0000 7f413a58e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c8ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.578+0000 7f413a58e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c838e0 0x5643d9c8cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.578+0000 7f413a58e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c842a0 0x5643d9c93f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.578+0000 7f413a58e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d9b8ff40 con 0x5643d9c842a0 2024-03-11T19:17:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.578+0000 7f413a58e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d98be4b0 con 0x5643d9c838e0 2024-03-11T19:17:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.578+0000 7f413a58e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d9b70670 con 0x5643d9c82710 2024-03-11T19:17:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.578+0000 7f413a58e740 10 client.0 _ll_register_callbacks cb 0x5643d9bddf70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:17:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.578+0000 7f413567b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c8ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.578+0000 7f413567b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c8ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:54102/0 (socket says 172.21.15.29:54102) 2024-03-11T19:17:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.578+0000 7f413567b640 1 -- 172.21.15.29:0/2453303995 learned_addr learned my addr 172.21.15.29:0/2453303995 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.578+0000 7f4135e7c640 1 --2- 172.21.15.29:0/2453303995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c842a0 0x5643d9c93f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.579+0000 7f4134e7a640 1 --2- 172.21.15.29:0/2453303995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c838e0 0x5643d9c8cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.579+0000 7f4135e7c640 1 -- 172.21.15.29:0/2453303995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 msgr2=0x5643d9c8ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.579+0000 7f4135e7c640 1 --2- 172.21.15.29:0/2453303995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c8ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.579+0000 7f4135e7c640 1 -- 172.21.15.29:0/2453303995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c838e0 msgr2=0x5643d9c8cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.579+0000 7f4135e7c640 1 --2- 172.21.15.29:0/2453303995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c838e0 0x5643d9c8cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.579+0000 7f4135e7c640 1 -- 172.21.15.29:0/2453303995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f412c009950 con 0x5643d9c842a0 2024-03-11T19:17:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.579+0000 7f4134e7a640 1 --2- 172.21.15.29:0/2453303995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c838e0 0x5643d9c8cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:17:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.579+0000 7f4135e7c640 1 --2- 172.21.15.29:0/2453303995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c842a0 0x5643d9c93f20 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f412000e940 tx=0x7f412009a070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e46f41e0f74b4a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.579+0000 7f4118ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:17:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.579+0000 7f4118ff9640 1 -- 172.21.15.29:0/2453303995 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41200b2070 con 0x5643d9c842a0 2024-03-11T19:17:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.579+0000 7f4118ff9640 1 -- 172.21.15.29:0/2453303995 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f412009e860 con 0x5643d9c842a0 2024-03-11T19:17:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.579+0000 7f413a58e740 1 -- 172.21.15.29:0/2453303995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5643d9bc4f40 con 0x5643d9c842a0 2024-03-11T19:17:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.580+0000 7f4118ff9640 1 -- 172.21.15.29:0/2453303995 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41200b3260 con 0x5643d9c842a0 2024-03-11T19:17:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.580+0000 7f413a58e740 10 client.17301 fetch_fsmap learned FSMap version 27 2024-03-11T19:17:30.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.580+0000 7f413a58e740 1 -- 172.21.15.29:0/2453303995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5643d9be9320 con 0x5643d9c842a0 2024-03-11T19:17:30.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.580+0000 7f4118ff9640 1 -- 172.21.15.29:0/2453303995 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f412007d020 con 0x5643d9c842a0 2024-03-11T19:17:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.580+0000 7f4118ff9640 1 -- 172.21.15.29:0/2453303995 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f412001f060 con 0x5643d9c842a0 2024-03-11T19:17:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.581+0000 7f413a58e740 10 client.17301 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:17:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.581+0000 7f413a58e740 10 client.17301 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:17:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.581+0000 7f413a58e740 20 client.17301 populate_metadata read hostname 'smithi029' 2024-03-11T19:17:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.581+0000 7f413a58e740 1 --2- 172.21.15.29:0/2453303995 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5643d9c98c10 0x5643d9c9b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.581+0000 7f413a58e740 4 client.17301 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:17:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.581+0000 7f413a58e740 1 -- 172.21.15.29:0/2453303995 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5643d9be1310 con 0x5643d9c98c10 2024-03-11T19:17:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.581+0000 7f413567b640 1 --2- 172.21.15.29:0/2453303995 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5643d9c98c10 0x5643d9c9b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.581+0000 7f4118ff9640 10 client.17301 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:17:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.581+0000 7f413567b640 1 --2- 172.21.15.29:0/2453303995 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5643d9c98c10 0x5643d9c9b030 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a86f6113aee1c419 server_cookie=d21e0c491cfc501e in_seq=0 out_seq=0 2024-03-11T19:17:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.582+0000 7f4118ff9640 1 -- 172.21.15.29:0/2453303995 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5643d9be1310 con 0x5643d9c98c10 2024-03-11T19:17:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.582+0000 7f4118ff9640 10 client.17301 handle_command_reply: tid=0 2024-03-11T19:17:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.582+0000 7f413a58e740 1 client.17301 shutdown 2024-03-11T19:17:30.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.583+0000 7f413a58e740 1 -- 172.21.15.29:0/2453303995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c842a0 msgr2=0x5643d9c93f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:30.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.583+0000 7f413a58e740 1 --2- 172.21.15.29:0/2453303995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c842a0 0x5643d9c93f20 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f412000e940 tx=0x7f412009a070 comp rx=0 tx=0).stop 2024-03-11T19:17:30.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.583+0000 7f413a58e740 1 -- 172.21.15.29:0/2453303995 shutdown_connections 2024-03-11T19:17:30.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.583+0000 7f413a58e740 1 --2- 172.21.15.29:0/2453303995 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5643d9c98c10 0x5643d9c9b030 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.583+0000 7f413a58e740 1 --2- 172.21.15.29:0/2453303995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c842a0 0x5643d9c93f20 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.583+0000 7f413a58e740 1 --2- 172.21.15.29:0/2453303995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c838e0 0x5643d9c8cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.583+0000 7f4118ff9640 0 client.17301 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:17:30.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.583+0000 7f413a58e740 1 --2- 172.21.15.29:0/2453303995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c8ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.583+0000 7f413a58e740 1 -- 172.21.15.29:0/2453303995 >> 172.21.15.29:0/2453303995 conn(0x5643d9c7d8f0 msgr2=0x5643d9c7cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:30.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.583+0000 7f413a58e740 1 -- 172.21.15.29:0/2453303995 shutdown_connections 2024-03-11T19:17:30.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.583+0000 7f413a58e740 1 -- 172.21.15.29:0/2453303995 wait complete. 2024-03-11T19:17:30.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.583+0000 7f413a58e740 20 client.17301 trim_cache size 0 max 16384 2024-03-11T19:17:30.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.617+0000 7f413a58e740 1 Processor -- start 2024-03-11T19:17:30.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.618+0000 7f413a58e740 1 -- start start 2024-03-11T19:17:30.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.618+0000 7f413a58e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c875c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.618+0000 7f413a58e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c838e0 0x5643d9c87b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.618+0000 7f413a58e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c842a0 0x5643d9c93ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.618+0000 7f413a58e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d9be1310 con 0x5643d9c838e0 2024-03-11T19:17:30.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.618+0000 7f413a58e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d9c83770 con 0x5643d9c842a0 2024-03-11T19:17:30.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.618+0000 7f413a58e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d9c8db90 con 0x5643d9c82710 2024-03-11T19:17:30.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.618+0000 7f413567b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c875c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.618+0000 7f413567b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c875c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:54110/0 (socket says 172.21.15.29:54110) 2024-03-11T19:17:30.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.618+0000 7f413567b640 1 -- 172.21.15.29:0/2829355279 learned_addr learned my addr 172.21.15.29:0/2829355279 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:30.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.618+0000 7f4135e7c640 1 --2- 172.21.15.29:0/2829355279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c842a0 0x5643d9c93ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.618+0000 7f413567b640 1 -- 172.21.15.29:0/2829355279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c842a0 msgr2=0x5643d9c93ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:30.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.619+0000 7f413567b640 1 --2- 172.21.15.29:0/2829355279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c842a0 0x5643d9c93ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.619+0000 7f413567b640 1 -- 172.21.15.29:0/2829355279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c838e0 msgr2=0x5643d9c87b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:30.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.619+0000 7f4134e7a640 1 --2- 172.21.15.29:0/2829355279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c838e0 0x5643d9c87b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.619+0000 7f413567b640 1 --2- 172.21.15.29:0/2829355279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c838e0 0x5643d9c87b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.619+0000 7f413567b640 1 -- 172.21.15.29:0/2829355279 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5643d9b85c00 con 0x5643d9c82710 2024-03-11T19:17:30.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.619+0000 7f4134e7a640 1 --2- 172.21.15.29:0/2829355279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c838e0 0x5643d9c87b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:17:30.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.619+0000 7f413567b640 1 --2- 172.21.15.29:0/2829355279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c875c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f412c005fd0 tx=0x7f412c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d203fb38becfc91 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:30.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.619+0000 7f411affd640 1 -- 172.21.15.29:0/2829355279 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c02e070 con 0x5643d9c82710 2024-03-11T19:17:30.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.619+0000 7f411affd640 1 -- 172.21.15.29:0/2829355279 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f412c009310 con 0x5643d9c82710 2024-03-11T19:17:30.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.619+0000 7f411affd640 1 -- 172.21.15.29:0/2829355279 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c02a520 con 0x5643d9c82710 2024-03-11T19:17:30.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.620+0000 7f413a58e740 1 -- 172.21.15.29:0/2829355279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 msgr2=0x5643d9c875c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:30.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.621+0000 7f413a58e740 1 --2- 172.21.15.29:0/2829355279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c875c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f412c005fd0 tx=0x7f412c011810 comp rx=0 tx=0).stop 2024-03-11T19:17:30.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.621+0000 7f413a58e740 1 -- 172.21.15.29:0/2829355279 shutdown_connections 2024-03-11T19:17:30.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.621+0000 7f413a58e740 1 --2- 172.21.15.29:0/2829355279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c842a0 0x5643d9c93ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.621+0000 7f413a58e740 1 --2- 172.21.15.29:0/2829355279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c838e0 0x5643d9c87b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.621+0000 7f413a58e740 1 --2- 172.21.15.29:0/2829355279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c875c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.621+0000 7f413a58e740 1 -- 172.21.15.29:0/2829355279 >> 172.21.15.29:0/2829355279 conn(0x5643d9c7d8f0 msgr2=0x5643d9c9f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:30.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.621+0000 7f413a58e740 1 -- 172.21.15.29:0/2829355279 shutdown_connections 2024-03-11T19:17:30.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.621+0000 7f413a58e740 1 -- 172.21.15.29:0/2829355279 wait complete. 2024-03-11T19:17:30.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.622+0000 7f413a58e740 1 -- start start 2024-03-11T19:17:30.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.622+0000 7f413a58e740 1 Processor -- start 2024-03-11T19:17:30.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.623+0000 7f413a58e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c90e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.623+0000 7f413a58e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c838e0 0x5643d9c913c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.623+0000 7f413a58e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c842a0 0x5643d9cef100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.623+0000 7f413a58e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d9c91a00 con 0x5643d9c838e0 2024-03-11T19:17:30.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.623+0000 7f413a58e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d9cef640 con 0x5643d9c842a0 2024-03-11T19:17:30.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.623+0000 7f413a58e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643d9cef780 con 0x5643d9c82710 2024-03-11T19:17:30.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.623+0000 7f413a58e740 10 client.0 _ll_register_callbacks cb 0x7f412c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:17:30.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.624+0000 7f413567b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c90e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.624+0000 7f413567b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c90e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:54122/0 (socket says 172.21.15.29:54122) 2024-03-11T19:17:30.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.624+0000 7f413567b640 1 -- 172.21.15.29:0/1977825194 learned_addr learned my addr 172.21.15.29:0/1977825194 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:30.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.624+0000 7f4135e7c640 1 --2- 172.21.15.29:0/1977825194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c842a0 0x5643d9cef100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.625+0000 7f413567b640 1 -- 172.21.15.29:0/1977825194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c842a0 msgr2=0x5643d9cef100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:30.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.625+0000 7f413567b640 1 --2- 172.21.15.29:0/1977825194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c842a0 0x5643d9cef100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.625+0000 7f413567b640 1 -- 172.21.15.29:0/1977825194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c838e0 msgr2=0x5643d9c913c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:17:30.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.625+0000 7f413567b640 1 --2- 172.21.15.29:0/1977825194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c838e0 0x5643d9c913c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.625+0000 7f413567b640 1 -- 172.21.15.29:0/1977825194 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5643d9bad0b0 con 0x5643d9c82710 2024-03-11T19:17:30.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.626+0000 7f413567b640 1 --2- 172.21.15.29:0/1977825194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c90e80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f412c009250 tx=0x7f412c0054f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a449d1f143c7b85f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:30.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.629+0000 7f4118ff9640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:17:30.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.629+0000 7f4118ff9640 1 -- 172.21.15.29:0/1977825194 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c02e050 con 0x5643d9c82710 2024-03-11T19:17:30.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.629+0000 7f4118ff9640 1 -- 172.21.15.29:0/1977825194 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f412c003b00 con 0x5643d9c82710 2024-03-11T19:17:30.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.629+0000 7f413a58e740 1 -- 172.21.15.29:0/1977825194 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5643d98ea860 con 0x5643d9c82710 2024-03-11T19:17:30.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.630+0000 7f4118ff9640 1 -- 172.21.15.29:0/1977825194 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f412c003cc0 con 0x5643d9c82710 2024-03-11T19:17:30.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.630+0000 7f413a58e740 10 client.26827 fetch_fsmap learned FSMap version 27 2024-03-11T19:17:30.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.630+0000 7f413a58e740 1 -- 172.21.15.29:0/1977825194 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5643d9bab910 con 0x5643d9c82710 2024-03-11T19:17:30.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.631+0000 7f4118ff9640 1 -- 172.21.15.29:0/1977825194 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f412c044020 con 0x5643d9c82710 2024-03-11T19:17:30.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.631+0000 7f4118ff9640 1 -- 172.21.15.29:0/1977825194 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f412c021040 con 0x5643d9c82710 2024-03-11T19:17:30.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.631+0000 7f413a58e740 10 client.26827 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:17:30.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.631+0000 7f413a58e740 10 client.26827 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:17:30.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.631+0000 7f413a58e740 20 client.26827 populate_metadata read hostname 'smithi029' 2024-03-11T19:17:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.631+0000 7f413a58e740 1 --2- 172.21.15.29:0/1977825194 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5643d9cf0da0 0x5643d9cf31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.631+0000 7f413a58e740 4 client.26827 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:17:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.631+0000 7f413a58e740 1 -- 172.21.15.29:0/1977825194 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5643d98ea860 con 0x5643d9cf0da0 2024-03-11T19:17:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.631+0000 7f4134e7a640 1 --2- 172.21.15.29:0/1977825194 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5643d9cf0da0 0x5643d9cf31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.632+0000 7f4118ff9640 10 client.26827 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:17:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.632+0000 7f4134e7a640 1 --2- 172.21.15.29:0/1977825194 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5643d9cf0da0 0x5643d9cf31c0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6406ca2ce36ce535 server_cookie=b642e20669017186 in_seq=0 out_seq=0 2024-03-11T19:17:30.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.633+0000 7f4118ff9640 1 -- 172.21.15.29:0/1977825194 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5643d98ea860 con 0x5643d9cf0da0 2024-03-11T19:17:30.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.633+0000 7f4118ff9640 10 client.26827 handle_command_reply: tid=0 2024-03-11T19:17:30.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.633+0000 7f413a58e740 1 client.26827 shutdown 2024-03-11T19:17:30.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.633+0000 7f413a58e740 1 -- 172.21.15.29:0/1977825194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 msgr2=0x5643d9c90e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:30.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.633+0000 7f413a58e740 1 --2- 172.21.15.29:0/1977825194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c90e80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f412c009250 tx=0x7f412c0054f0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.634+0000 7f413a58e740 1 -- 172.21.15.29:0/1977825194 shutdown_connections 2024-03-11T19:17:30.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.634+0000 7f413a58e740 1 --2- 172.21.15.29:0/1977825194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5643d9c842a0 0x5643d9cef100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.634+0000 7f413a58e740 1 --2- 172.21.15.29:0/1977825194 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5643d9cf0da0 0x5643d9cf31c0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.634+0000 7f4118ff9640 0 client.26827 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:17:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.634+0000 7f413a58e740 1 --2- 172.21.15.29:0/1977825194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5643d9c838e0 0x5643d9c913c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.634+0000 7f413a58e740 1 --2- 172.21.15.29:0/1977825194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643d9c82710 0x5643d9c90e80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.634+0000 7f413a58e740 1 -- 172.21.15.29:0/1977825194 >> 172.21.15.29:0/1977825194 conn(0x5643d9c7d8f0 msgr2=0x5643d9c86b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.634+0000 7f413a58e740 1 -- 172.21.15.29:0/1977825194 shutdown_connections 2024-03-11T19:17:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.634+0000 7f413a58e740 1 -- 172.21.15.29:0/1977825194 wait complete. 2024-03-11T19:17:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.635+0000 7f413a58e740 20 client.26827 trim_cache size 0 max 16384 2024-03-11T19:17:30.641 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:17:30.641 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:17:30.641 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "36700e6f-c84e-4e18-8647-5366c68307c4", 2024-03-11T19:17:30.641 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:17:30.642 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:17:30.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.638+0000 7f413667d640 1 -- 172.21.15.29:0/4069412762 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f411c0a1b30 msgr2=0x7f411c0a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.638+0000 7f413667d640 1 --2- 172.21.15.29:0/4069412762 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f411c0a1b30 0x7f411c0a3ff0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f412802c710 tx=0x7f4128009360 comp rx=0 tx=0).stop 2024-03-11T19:17:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.638+0000 7f413667d640 1 -- 172.21.15.29:0/4069412762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4130109120 msgr2=0x7f41301ad300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.638+0000 7f413667d640 1 --2- 172.21.15.29:0/4069412762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4130109120 0x7f41301ad300 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f412000e970 tx=0x7f412000ed70 comp rx=0 tx=0).stop 2024-03-11T19:17:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.638+0000 7f413667d640 1 -- 172.21.15.29:0/4069412762 shutdown_connections 2024-03-11T19:17:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.638+0000 7f413667d640 1 --2- 172.21.15.29:0/4069412762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4130109120 0x7f41301ad300 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.638+0000 7f413667d640 1 --2- 172.21.15.29:0/4069412762 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f411c0a1b30 0x7f411c0a3ff0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.638+0000 7f413667d640 1 --2- 172.21.15.29:0/4069412762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4130108780 0x7f41301a6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.638+0000 7f413667d640 1 --2- 172.21.15.29:0/4069412762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41300724d0 0x7f41301a5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.638+0000 7f413667d640 1 -- 172.21.15.29:0/4069412762 >> 172.21.15.29:0/4069412762 conn(0x7f413006dfa0 msgr2=0x7f413006e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.638+0000 7f413667d640 1 -- 172.21.15.29:0/4069412762 shutdown_connections 2024-03-11T19:17:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.638+0000 7f413667d640 1 -- 172.21.15.29:0/4069412762 wait complete. 2024-03-11T19:17:30.665 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:17:30.665 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:17:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.844+0000 7f4c5e0a7640 1 -- 172.21.15.29:0/481784757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c58100ae0 msgr2=0x7f4c58100f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.844+0000 7f4c5e0a7640 1 --2- 172.21.15.29:0/481784757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c58100ae0 0x7f4c58100f60 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f4c50009ca0 tx=0x7f4c5002c100 comp rx=0 tx=0).stop 2024-03-11T19:17:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.844+0000 7f4c5e0a7640 1 -- 172.21.15.29:0/481784757 shutdown_connections 2024-03-11T19:17:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.844+0000 7f4c5e0a7640 1 --2- 172.21.15.29:0/481784757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c58100ae0 0x7f4c58100f60 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.845+0000 7f4c5e0a7640 1 --2- 172.21.15.29:0/481784757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c580676e0 0x7f4c581005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.845+0000 7f4c5e0a7640 1 --2- 172.21.15.29:0/481784757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c580ff3b0 0x7f4c580ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.845+0000 7f4c5e0a7640 1 -- 172.21.15.29:0/481784757 >> 172.21.15.29:0/481784757 conn(0x7f4c580fa960 msgr2=0x7f4c580fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.845+0000 7f4c5e0a7640 1 -- 172.21.15.29:0/481784757 shutdown_connections 2024-03-11T19:17:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.845+0000 7f4c5e0a7640 1 -- 172.21.15.29:0/481784757 wait complete. 2024-03-11T19:17:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.845+0000 7f4c5e0a7640 1 Processor -- start 2024-03-11T19:17:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.846+0000 7f4c5e0a7640 1 -- start start 2024-03-11T19:17:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.846+0000 7f4c5e0a7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c580676e0 0x7f4c58198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.846+0000 7f4c5e0a7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c580ff3b0 0x7f4c58194db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.846+0000 7f4c5e0a7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c58100ae0 0x7f4c58195370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.846+0000 7f4c5e0a7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c581ad960 con 0x7f4c58100ae0 2024-03-11T19:17:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.846+0000 7f4c5e0a7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c581adaa0 con 0x7f4c580676e0 2024-03-11T19:17:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.846+0000 7f4c5e0a7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c581adc10 con 0x7f4c580ff3b0 2024-03-11T19:17:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.846+0000 7f4c56ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c580ff3b0 0x7f4c58194db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.847+0000 7f4c56ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c580ff3b0 0x7f4c58194db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:54140/0 (socket says 172.21.15.29:54140) 2024-03-11T19:17:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.847+0000 7f4c56ffd640 1 -- 172.21.15.29:0/2037207749 learned_addr learned my addr 172.21.15.29:0/2037207749 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.847+0000 7f4c577fe640 1 --2- 172.21.15.29:0/2037207749 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c580676e0 0x7f4c58198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.847+0000 7f4c577fe640 1 -- 172.21.15.29:0/2037207749 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c580ff3b0 msgr2=0x7f4c58194db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.847+0000 7f4c577fe640 1 --2- 172.21.15.29:0/2037207749 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c580ff3b0 0x7f4c58194db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.847+0000 7f4c577fe640 1 -- 172.21.15.29:0/2037207749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c58100ae0 msgr2=0x7f4c58195370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.847+0000 7f4c577fe640 1 --2- 172.21.15.29:0/2037207749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c58100ae0 0x7f4c58195370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.847+0000 7f4c577fe640 1 -- 172.21.15.29:0/2037207749 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c50009900 con 0x7f4c580676e0 2024-03-11T19:17:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.847+0000 7f4c577fe640 1 --2- 172.21.15.29:0/2037207749 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c580676e0 0x7f4c58198340 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4c58100400 tx=0x7f4c4c00dbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.848+0000 7f4c54ff9640 1 -- 172.21.15.29:0/2037207749 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c009950 con 0x7f4c580676e0 2024-03-11T19:17:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.848+0000 7f4c5e0a7640 1 -- 172.21.15.29:0/2037207749 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c581adef0 con 0x7f4c580676e0 2024-03-11T19:17:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.848+0000 7f4c5e0a7640 1 -- 172.21.15.29:0/2037207749 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c581ae3a0 con 0x7f4c580676e0 2024-03-11T19:17:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.848+0000 7f4c54ff9640 1 -- 172.21.15.29:0/2037207749 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4c4c010460 con 0x7f4c580676e0 2024-03-11T19:17:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.849+0000 7f4c54ff9640 1 -- 172.21.15.29:0/2037207749 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c00f620 con 0x7f4c580676e0 2024-03-11T19:17:30.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.851+0000 7f4c54ff9640 1 -- 172.21.15.29:0/2037207749 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4c4c009ab0 con 0x7f4c580676e0 2024-03-11T19:17:30.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.852+0000 7f4c54ff9640 1 --2- 172.21.15.29:0/2037207749 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4c400a1a90 0x7f4c400a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.853+0000 7f4c56ffd640 1 --2- 172.21.15.29:0/2037207749 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4c400a1a90 0x7f4c400a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.853+0000 7f4c54ff9640 1 -- 172.21.15.29:0/2037207749 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4c4c0ccee0 con 0x7f4c580676e0 2024-03-11T19:17:30.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.854+0000 7f4c56ffd640 1 --2- 172.21.15.29:0/2037207749 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4c400a1a90 0x7f4c400a3f50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4c581962c0 tx=0x7f4c48009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:30.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.855+0000 7f4c60c7f740 1 Processor -- start 2024-03-11T19:17:30.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.855+0000 7f4c60c7f740 1 -- start start 2024-03-11T19:17:30.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.855+0000 7f4c60c7f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff2e2d0 0x564d0ff2e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.855+0000 7f4c60c7f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff2ecc0 0x564d0ff337c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.855+0000 7f4c60c7f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff33e30 0x564d0ff362c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.855+0000 7f4c577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff2e2d0 0x564d0ff2e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.856+0000 7f4c577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff2e2d0 0x564d0ff2e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42560/0 (socket says 172.21.15.29:42560) 2024-03-11T19:17:30.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.856+0000 7f4c577fe640 1 -- 172.21.15.29:0/952448412 learned_addr learned my addr 172.21.15.29:0/952448412 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:30.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.856+0000 7f4c60c7f740 1 -- 172.21.15.29:0/952448412 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0fe9e280 con 0x564d0ff2e2d0 2024-03-11T19:17:30.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.856+0000 7f4c60c7f740 1 -- 172.21.15.29:0/952448412 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0fe9b7b0 con 0x564d0ff2ecc0 2024-03-11T19:17:30.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.856+0000 7f4c60c7f740 1 -- 172.21.15.29:0/952448412 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0fe96f90 con 0x564d0ff33e30 2024-03-11T19:17:30.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.856+0000 7f4c57fff640 1 --2- 172.21.15.29:0/952448412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff33e30 0x564d0ff362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.856+0000 7f4c56ffd640 1 --2- 172.21.15.29:0/952448412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff2ecc0 0x564d0ff337c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.856+0000 7f4c577fe640 1 -- 172.21.15.29:0/952448412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff33e30 msgr2=0x564d0ff362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.856+0000 7f4c577fe640 1 --2- 172.21.15.29:0/952448412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff33e30 0x564d0ff362c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.856+0000 7f4c577fe640 1 -- 172.21.15.29:0/952448412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff2ecc0 msgr2=0x564d0ff337c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.856+0000 7f4c577fe640 1 --2- 172.21.15.29:0/952448412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff2ecc0 0x564d0ff337c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.856+0000 7f4c577fe640 1 -- 172.21.15.29:0/952448412 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d0fe55570 con 0x564d0ff2e2d0 2024-03-11T19:17:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.856+0000 7f4c577fe640 1 --2- 172.21.15.29:0/952448412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff2e2d0 0x564d0ff2e6f0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c099b50 tx=0x7f4c4c099e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3a276140f9c875 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.857+0000 7f4c3dffb640 1 -- 172.21.15.29:0/952448412 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c0adea0 con 0x564d0ff2e2d0 2024-03-11T19:17:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.857+0000 7f4c3dffb640 1 -- 172.21.15.29:0/952448412 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4c4c098150 con 0x564d0ff2e2d0 2024-03-11T19:17:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.857+0000 7f4c3dffb640 1 -- 172.21.15.29:0/952448412 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c095f20 con 0x564d0ff2e2d0 2024-03-11T19:17:30.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.857+0000 7f4c60c7f740 1 -- 172.21.15.29:0/952448412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff2e2d0 msgr2=0x564d0ff2e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:30.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.857+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/952448412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff2e2d0 0x564d0ff2e6f0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c099b50 tx=0x7f4c4c099e50 comp rx=0 tx=0).stop 2024-03-11T19:17:30.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.858+0000 7f4c60c7f740 1 -- 172.21.15.29:0/952448412 shutdown_connections 2024-03-11T19:17:30.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.858+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/952448412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff33e30 0x564d0ff362c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.858+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/952448412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff2ecc0 0x564d0ff337c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.858+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/952448412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff2e2d0 0x564d0ff2e6f0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.858+0000 7f4c60c7f740 1 -- 172.21.15.29:0/952448412 >> 172.21.15.29:0/952448412 conn(0x564d0ff297f0 msgr2=0x564d0ff29bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:30.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.858+0000 7f4c60c7f740 1 -- 172.21.15.29:0/952448412 shutdown_connections 2024-03-11T19:17:30.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.858+0000 7f4c60c7f740 1 -- 172.21.15.29:0/952448412 wait complete. 2024-03-11T19:17:30.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.858+0000 7f4c60c7f740 1 -- start start 2024-03-11T19:17:30.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.858+0000 7f4c60c7f740 1 Processor -- start 2024-03-11T19:17:30.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.859+0000 7f4c60c7f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff2e2d0 0x564d0ff381e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.859+0000 7f4c60c7f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff2ecc0 0x564d0ff38720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.859+0000 7f4c60c7f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff33e30 0x564d0ff3f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.859+0000 7f4c577fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff2e2d0 0x564d0ff381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.859+0000 7f4c577fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff2e2d0 0x564d0ff381e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:50250/0 (socket says 172.21.15.29:50250) 2024-03-11T19:17:30.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.859+0000 7f4c56ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff2ecc0 0x564d0ff38720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.859+0000 7f4c577fe640 1 -- 172.21.15.29:0/956448722 learned_addr learned my addr 172.21.15.29:0/956448722 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:30.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.859+0000 7f4c60c7f740 1 -- 172.21.15.29:0/956448722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0fb6a430 con 0x564d0ff33e30 2024-03-11T19:17:30.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.859+0000 7f4c60c7f740 1 -- 172.21.15.29:0/956448722 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0fb26e00 con 0x564d0ff2e2d0 2024-03-11T19:17:30.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.859+0000 7f4c60c7f740 1 -- 172.21.15.29:0/956448722 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0fe1bfd0 con 0x564d0ff2ecc0 2024-03-11T19:17:30.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.859+0000 7f4c57fff640 1 --2- 172.21.15.29:0/956448722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff33e30 0x564d0ff3f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.859+0000 7f4c60c7f740 10 client.0 _ll_register_callbacks cb 0x564d0fe71fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:17:30.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.859+0000 7f4c577fe640 1 -- 172.21.15.29:0/956448722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff2ecc0 msgr2=0x564d0ff38720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:30.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.859+0000 7f4c577fe640 1 --2- 172.21.15.29:0/956448722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff2ecc0 0x564d0ff38720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.859+0000 7f4c577fe640 1 -- 172.21.15.29:0/956448722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff33e30 msgr2=0x564d0ff3f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:30.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.859+0000 7f4c577fe640 1 --2- 172.21.15.29:0/956448722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff33e30 0x564d0ff3f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.859+0000 7f4c577fe640 1 -- 172.21.15.29:0/956448722 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c581ae3a0 con 0x564d0ff2e2d0 2024-03-11T19:17:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.860+0000 7f4c577fe640 1 --2- 172.21.15.29:0/956448722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff2e2d0 0x564d0ff381e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0a20c0 tx=0x7f4c4c098330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20e4627a60698a4d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.860+0000 7f4c36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:17:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.860+0000 7f4c36ffd640 1 -- 172.21.15.29:0/956448722 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c0adea0 con 0x564d0ff2e2d0 2024-03-11T19:17:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.860+0000 7f4c36ffd640 1 -- 172.21.15.29:0/956448722 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4c4c0cae40 con 0x564d0ff2e2d0 2024-03-11T19:17:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.860+0000 7f4c60c7f740 1 -- 172.21.15.29:0/956448722 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564d0fb967e0 con 0x564d0ff2e2d0 2024-03-11T19:17:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.860+0000 7f4c36ffd640 1 -- 172.21.15.29:0/956448722 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c4c0964c0 con 0x564d0ff2e2d0 2024-03-11T19:17:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.860+0000 7f4c60c7f740 10 client.26783 fetch_fsmap learned FSMap version 27 2024-03-11T19:17:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.860+0000 7f4c60c7f740 1 -- 172.21.15.29:0/956448722 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x564d0fe95610 con 0x564d0ff2e2d0 2024-03-11T19:17:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.860+0000 7f4c36ffd640 1 -- 172.21.15.29:0/956448722 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x564d0fb967e0 con 0x564d0ff2e2d0 2024-03-11T19:17:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.861+0000 7f4c36ffd640 1 -- 172.21.15.29:0/956448722 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4c4c098d10 con 0x564d0ff2e2d0 2024-03-11T19:17:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.861+0000 7f4c60c7f740 10 client.26783 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:17:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.861+0000 7f4c60c7f740 10 client.26783 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:17:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.861+0000 7f4c60c7f740 20 client.26783 populate_metadata read hostname 'smithi029' 2024-03-11T19:17:30.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.861+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/956448722 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564d0ff44080 0x564d0ff464a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.861+0000 7f4c60c7f740 4 client.26783 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:17:30.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.861+0000 7f4c60c7f740 1 -- 172.21.15.29:0/956448722 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564d0fbd3720 con 0x564d0ff44080 2024-03-11T19:17:30.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.861+0000 7f4c56ffd640 1 --2- 172.21.15.29:0/956448722 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564d0ff44080 0x564d0ff464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.862+0000 7f4c36ffd640 10 client.26783 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:17:30.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.862+0000 7f4c56ffd640 1 --2- 172.21.15.29:0/956448722 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564d0ff44080 0x564d0ff464a0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2509a902264789de server_cookie=fdc9dc25a804eb72 in_seq=0 out_seq=0 2024-03-11T19:17:30.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.862+0000 7f4c36ffd640 1 -- 172.21.15.29:0/956448722 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564d0fbd3720 con 0x564d0ff44080 2024-03-11T19:17:30.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.862+0000 7f4c36ffd640 10 client.26783 handle_command_reply: tid=0 2024-03-11T19:17:30.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.862+0000 7f4c60c7f740 1 client.26783 shutdown 2024-03-11T19:17:30.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.862+0000 7f4c60c7f740 1 -- 172.21.15.29:0/956448722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff2e2d0 msgr2=0x564d0ff381e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:30.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.862+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/956448722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff2e2d0 0x564d0ff381e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0a20c0 tx=0x7f4c4c098330 comp rx=0 tx=0).stop 2024-03-11T19:17:30.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.862+0000 7f4c60c7f740 1 -- 172.21.15.29:0/956448722 shutdown_connections 2024-03-11T19:17:30.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.862+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/956448722 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564d0ff44080 0x564d0ff464a0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.863+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/956448722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff33e30 0x564d0ff3f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.863+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/956448722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff2ecc0 0x564d0ff38720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.863+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/956448722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff2e2d0 0x564d0ff381e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.863+0000 7f4c60c7f740 1 -- 172.21.15.29:0/956448722 >> 172.21.15.29:0/956448722 conn(0x564d0ff297f0 msgr2=0x564d0ff28a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.863+0000 7f4c36ffd640 0 client.26783 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:17:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.863+0000 7f4c60c7f740 1 -- 172.21.15.29:0/956448722 shutdown_connections 2024-03-11T19:17:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.863+0000 7f4c60c7f740 1 -- 172.21.15.29:0/956448722 wait complete. 2024-03-11T19:17:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.863+0000 7f4c60c7f740 20 client.26783 trim_cache size 0 max 16384 2024-03-11T19:17:30.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.902+0000 7f4c60c7f740 1 Processor -- start 2024-03-11T19:17:30.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.905+0000 7f4c60c7f740 1 -- start start 2024-03-11T19:17:30.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.905+0000 7f4c60c7f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff2e2d0 0x564d0ff2ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.905+0000 7f4c60c7f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff2ecc0 0x564d0ff30440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.905+0000 7f4c60c7f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff33e30 0x564d0ff3f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.905+0000 7f4c60c7f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0ff2f0f0 con 0x564d0ff2e2d0 2024-03-11T19:17:30.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.905+0000 7f4c60c7f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0ff25320 con 0x564d0ff33e30 2024-03-11T19:17:30.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.905+0000 7f4c60c7f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0ff29660 con 0x564d0ff2ecc0 2024-03-11T19:17:30.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.905+0000 7f4c577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff2e2d0 0x564d0ff2ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.905+0000 7f4c577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff2e2d0 0x564d0ff2ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42580/0 (socket says 172.21.15.29:42580) 2024-03-11T19:17:30.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.905+0000 7f4c577fe640 1 -- 172.21.15.29:0/2469040149 learned_addr learned my addr 172.21.15.29:0/2469040149 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:30.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.905+0000 7f4c56ffd640 1 --2- 172.21.15.29:0/2469040149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff2ecc0 0x564d0ff30440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.905+0000 7f4c57fff640 1 --2- 172.21.15.29:0/2469040149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff33e30 0x564d0ff3f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.906+0000 7f4c56ffd640 1 -- 172.21.15.29:0/2469040149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff33e30 msgr2=0x564d0ff3f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:30.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.906+0000 7f4c56ffd640 1 --2- 172.21.15.29:0/2469040149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff33e30 0x564d0ff3f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.906+0000 7f4c56ffd640 1 -- 172.21.15.29:0/2469040149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff2e2d0 msgr2=0x564d0ff2ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:30.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.906+0000 7f4c56ffd640 1 --2- 172.21.15.29:0/2469040149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff2e2d0 0x564d0ff2ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.906+0000 7f4c56ffd640 1 -- 172.21.15.29:0/2469040149 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d0fe31ac0 con 0x564d0ff2ecc0 2024-03-11T19:17:30.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.906+0000 7f4c57fff640 1 --2- 172.21.15.29:0/2469040149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff33e30 0x564d0ff3f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:17:30.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.906+0000 7f4c56ffd640 1 --2- 172.21.15.29:0/2469040149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff2ecc0 0x564d0ff30440 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x564d0ff46dd0 tx=0x7f4c48015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fba6b10b8acbe417 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:30.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.906+0000 7f4c3d7fa640 1 -- 172.21.15.29:0/2469040149 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c48028290 con 0x564d0ff2ecc0 2024-03-11T19:17:30.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.906+0000 7f4c3d7fa640 1 -- 172.21.15.29:0/2469040149 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4c480288a0 con 0x564d0ff2ecc0 2024-03-11T19:17:30.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.906+0000 7f4c3d7fa640 1 -- 172.21.15.29:0/2469040149 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c48022430 con 0x564d0ff2ecc0 2024-03-11T19:17:30.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.907+0000 7f4c60c7f740 1 -- 172.21.15.29:0/2469040149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff2ecc0 msgr2=0x564d0ff30440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:30.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.907+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/2469040149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff2ecc0 0x564d0ff30440 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x564d0ff46dd0 tx=0x7f4c48015000 comp rx=0 tx=0).stop 2024-03-11T19:17:30.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.907+0000 7f4c60c7f740 1 -- 172.21.15.29:0/2469040149 shutdown_connections 2024-03-11T19:17:30.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.907+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/2469040149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff33e30 0x564d0ff3f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.907+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/2469040149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff2ecc0 0x564d0ff30440 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x564d0ff46dd0 tx=0x7f4c48015000 comp rx=0 tx=0).stop 2024-03-11T19:17:30.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.907+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/2469040149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff2e2d0 0x564d0ff2ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.907+0000 7f4c60c7f740 1 -- 172.21.15.29:0/2469040149 >> 172.21.15.29:0/2469040149 conn(0x564d0ff297f0 msgr2=0x564d0ff4ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:30.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.907+0000 7f4c60c7f740 1 -- 172.21.15.29:0/2469040149 shutdown_connections 2024-03-11T19:17:30.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.908+0000 7f4c60c7f740 1 -- 172.21.15.29:0/2469040149 wait complete. 2024-03-11T19:17:30.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.908+0000 7f4c60c7f740 1 -- start start 2024-03-11T19:17:30.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.908+0000 7f4c60c7f740 1 Processor -- start 2024-03-11T19:17:30.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.909+0000 7f4c60c7f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff2e2d0 0x564d0ff3e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.909+0000 7f4c60c7f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff33e30 0x564d0ff3e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.909+0000 7f4c60c7f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff3eac0 0x564d0ff9a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.909+0000 7f4c60c7f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0ff3fae0 con 0x564d0ff33e30 2024-03-11T19:17:30.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.909+0000 7f4c60c7f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0ff9ab00 con 0x564d0ff2e2d0 2024-03-11T19:17:30.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.909+0000 7f4c60c7f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0ff9acc0 con 0x564d0ff3eac0 2024-03-11T19:17:30.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.909+0000 7f4c60c7f740 10 client.0 _ll_register_callbacks cb 0x7f4c4c099c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:17:30.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.909+0000 7f4c56ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff33e30 0x564d0ff3e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.909+0000 7f4c56ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff33e30 0x564d0ff3e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42586/0 (socket says 172.21.15.29:42586) 2024-03-11T19:17:30.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.909+0000 7f4c56ffd640 1 -- 172.21.15.29:0/3836280260 learned_addr learned my addr 172.21.15.29:0/3836280260 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:17:30.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.909+0000 7f4c57fff640 1 --2- 172.21.15.29:0/3836280260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff3eac0 0x564d0ff9a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.909+0000 7f4c577fe640 1 --2- 172.21.15.29:0/3836280260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff2e2d0 0x564d0ff3e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.909+0000 7f4c577fe640 1 -- 172.21.15.29:0/3836280260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff3eac0 msgr2=0x564d0ff9a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:30.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.909+0000 7f4c577fe640 1 --2- 172.21.15.29:0/3836280260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff3eac0 0x564d0ff9a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.909+0000 7f4c577fe640 1 -- 172.21.15.29:0/3836280260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff33e30 msgr2=0x564d0ff3e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:17:30.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.909+0000 7f4c577fe640 1 --2- 172.21.15.29:0/3836280260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff33e30 0x564d0ff3e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.909+0000 7f4c577fe640 1 -- 172.21.15.29:0/3836280260 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c581017c0 con 0x564d0ff2e2d0 2024-03-11T19:17:30.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.909+0000 7f4c56ffd640 1 --2- 172.21.15.29:0/3836280260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff33e30 0x564d0ff3e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:17:30.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.910+0000 7f4c577fe640 1 --2- 172.21.15.29:0/3836280260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff2e2d0 0x564d0ff3e040 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c096820 tx=0x7f4c4c0a2770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8c6043ffd6d6958 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:17:30.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.910+0000 7f4c36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:17:30.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.910+0000 7f4c36ffd640 1 -- 172.21.15.29:0/3836280260 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c0adea0 con 0x564d0ff2e2d0 2024-03-11T19:17:30.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.910+0000 7f4c36ffd640 1 -- 172.21.15.29:0/3836280260 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4c4c0a8030 con 0x564d0ff2e2d0 2024-03-11T19:17:30.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.910+0000 7f4c60c7f740 1 -- 172.21.15.29:0/3836280260 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c4c098150 con 0x564d0ff2e2d0 2024-03-11T19:17:30.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.910+0000 7f4c36ffd640 1 -- 172.21.15.29:0/3836280260 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c4c0960f0 con 0x564d0ff2e2d0 2024-03-11T19:17:30.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.910+0000 7f4c60c7f740 10 client.26789 fetch_fsmap learned FSMap version 27 2024-03-11T19:17:30.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.910+0000 7f4c60c7f740 1 -- 172.21.15.29:0/3836280260 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7f4c4801a040 con 0x564d0ff2e2d0 2024-03-11T19:17:30.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.910+0000 7f4c36ffd640 1 -- 172.21.15.29:0/3836280260 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4c4c098150 con 0x564d0ff2e2d0 2024-03-11T19:17:30.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.911+0000 7f4c36ffd640 1 -- 172.21.15.29:0/3836280260 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4c4c0b2b80 con 0x564d0ff2e2d0 2024-03-11T19:17:30.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.911+0000 7f4c60c7f740 10 client.26789 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:17:30.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.911+0000 7f4c60c7f740 10 client.26789 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:17:30.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.911+0000 7f4c60c7f740 20 client.26789 populate_metadata read hostname 'smithi029' 2024-03-11T19:17:30.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.911+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/3836280260 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564d0ff9c4d0 0x564d0ff9e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:17:30.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.911+0000 7f4c60c7f740 4 client.26789 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:17:30.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.911+0000 7f4c60c7f740 1 -- 172.21.15.29:0/3836280260 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564d0fc22890 con 0x564d0ff9c4d0 2024-03-11T19:17:30.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.911+0000 7f4c56ffd640 1 --2- 172.21.15.29:0/3836280260 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564d0ff9c4d0 0x564d0ff9e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:17:30.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.912+0000 7f4c36ffd640 10 client.26789 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:17:30.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.912+0000 7f4c56ffd640 1 --2- 172.21.15.29:0/3836280260 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564d0ff9c4d0 0x564d0ff9e8f0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8682bff14a06242e server_cookie=32e4ed4e5b9c7bb2 in_seq=0 out_seq=0 2024-03-11T19:17:30.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.912+0000 7f4c36ffd640 1 -- 172.21.15.29:0/3836280260 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x564d0fc22890 con 0x564d0ff9c4d0 2024-03-11T19:17:30.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.912+0000 7f4c36ffd640 10 client.26789 handle_command_reply: tid=0 2024-03-11T19:17:30.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.912+0000 7f4c60c7f740 1 client.26789 shutdown 2024-03-11T19:17:30.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.912+0000 7f4c60c7f740 1 -- 172.21.15.29:0/3836280260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff2e2d0 msgr2=0x564d0ff3e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:30.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.912+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/3836280260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff2e2d0 0x564d0ff3e040 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c096820 tx=0x7f4c4c0a2770 comp rx=0 tx=0).stop 2024-03-11T19:17:30.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.912+0000 7f4c60c7f740 1 -- 172.21.15.29:0/3836280260 shutdown_connections 2024-03-11T19:17:30.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.912+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/3836280260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d0ff3eac0 0x564d0ff9a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.912+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/3836280260 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x564d0ff9c4d0 0x564d0ff9e8f0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.912+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/3836280260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564d0ff33e30 0x564d0ff3e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.912+0000 7f4c60c7f740 1 --2- 172.21.15.29:0/3836280260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564d0ff2e2d0 0x564d0ff3e040 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.912+0000 7f4c36ffd640 0 client.26789 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:17:30.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.912+0000 7f4c60c7f740 1 -- 172.21.15.29:0/3836280260 >> 172.21.15.29:0/3836280260 conn(0x564d0ff297f0 msgr2=0x564d0ff2ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:30.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.912+0000 7f4c60c7f740 1 -- 172.21.15.29:0/3836280260 shutdown_connections 2024-03-11T19:17:30.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.913+0000 7f4c60c7f740 1 -- 172.21.15.29:0/3836280260 wait complete. 2024-03-11T19:17:30.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.913+0000 7f4c60c7f740 20 client.26789 trim_cache size 0 max 16384 2024-03-11T19:17:30.920 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:17:30.920 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:17:30.921 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:17:30.921 INFO:teuthology.orchestra.run.smithi029.stdout: "36700e6f-c84e-4e18-8647-5366c68307c4": { 2024-03-11T19:17:30.921 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:17:30.921 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "36700e6f-c84e-4e18-8647-5366c68307c4", 2024-03-11T19:17:30.921 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:17:30.921 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:17:30.921 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:17:30.921 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:17:30.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.913+0000 7f4c5e0a7640 1 -- 172.21.15.29:0/2037207749 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4c400a1a90 msgr2=0x7f4c400a3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:30.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.913+0000 7f4c5e0a7640 1 --2- 172.21.15.29:0/2037207749 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4c400a1a90 0x7f4c400a3f50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4c581962c0 tx=0x7f4c48009190 comp rx=0 tx=0).stop 2024-03-11T19:17:30.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.914+0000 7f4c5e0a7640 1 -- 172.21.15.29:0/2037207749 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c580676e0 msgr2=0x7f4c58198340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:17:30.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.914+0000 7f4c5e0a7640 1 --2- 172.21.15.29:0/2037207749 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c580676e0 0x7f4c58198340 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4c58100400 tx=0x7f4c4c00dbb0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.914+0000 7f4c5e0a7640 1 -- 172.21.15.29:0/2037207749 shutdown_connections 2024-03-11T19:17:30.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.914+0000 7f4c5e0a7640 1 --2- 172.21.15.29:0/2037207749 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4c400a1a90 0x7f4c400a3f50 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.914+0000 7f4c5e0a7640 1 --2- 172.21.15.29:0/2037207749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c58100ae0 0x7f4c58195370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.914+0000 7f4c5e0a7640 1 --2- 172.21.15.29:0/2037207749 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c580ff3b0 0x7f4c58194db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.914+0000 7f4c5e0a7640 1 --2- 172.21.15.29:0/2037207749 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c580676e0 0x7f4c58198340 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:17:30.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.914+0000 7f4c5e0a7640 1 -- 172.21.15.29:0/2037207749 >> 172.21.15.29:0/2037207749 conn(0x7f4c580fa960 msgr2=0x7f4c580fbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:17:30.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.914+0000 7f4c5e0a7640 1 -- 172.21.15.29:0/2037207749 shutdown_connections 2024-03-11T19:17:30.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:17:30.914+0000 7f4c5e0a7640 1 -- 172.21.15.29:0/2037207749 wait complete. 2024-03-11T19:17:30.936 INFO:tasks.cephfs.filesystem:scrub status for tag:36700e6f-c84e-4e18-8647-5366c68307c4 - {'path': '/', 'tag': '36700e6f-c84e-4e18-8647-5366c68307c4', 'options': 'recursive,force'} 2024-03-11T19:17:31.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:31 smithi170 ceph-mon[35690]: pgmap v582: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 65 MiB/s wr, 1.14k op/s 2024-03-11T19:17:31.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:31 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:17:31.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:31 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:17:31.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:31 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:17:31.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:31 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:17:31.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:31 smithi029 ceph-mon[32439]: pgmap v582: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 65 MiB/s wr, 1.14k op/s 2024-03-11T19:17:31.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:31 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:17:31.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:31 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:17:31.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:31 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:17:31.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:31 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:17:31.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:31 smithi190 ceph-mon[36292]: pgmap v582: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 65 MiB/s wr, 1.14k op/s 2024-03-11T19:17:31.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:31 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:17:31.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:31 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:17:31.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:31 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:17:31.513 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:31 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:17:33.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:33 smithi170 ceph-mon[35690]: pgmap v583: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 65 MiB/s wr, 1.14k op/s 2024-03-11T19:17:33.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:33 smithi029 ceph-mon[32439]: pgmap v583: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 65 MiB/s wr, 1.14k op/s 2024-03-11T19:17:33.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:33 smithi190 ceph-mon[36292]: pgmap v583: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 65 MiB/s wr, 1.14k op/s 2024-03-11T19:17:35.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:35 smithi170 ceph-mon[35690]: pgmap v584: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 61 MiB/s wr, 1.08k op/s 2024-03-11T19:17:35.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:35 smithi029 ceph-mon[32439]: pgmap v584: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 61 MiB/s wr, 1.08k op/s 2024-03-11T19:17:35.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:35 smithi190 ceph-mon[36292]: pgmap v584: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 61 MiB/s wr, 1.08k op/s 2024-03-11T19:17:36.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:36 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:17:36.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:36 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:17:36.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:36 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:17:37.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:37 smithi170 ceph-mon[35690]: pgmap v585: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 67 MiB/s wr, 1.17k op/s 2024-03-11T19:17:37.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:37 smithi029 ceph-mon[32439]: pgmap v585: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 67 MiB/s wr, 1.17k op/s 2024-03-11T19:17:37.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:37 smithi190 ceph-mon[36292]: pgmap v585: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 67 MiB/s wr, 1.17k op/s 2024-03-11T19:17:39.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:39 smithi170 ceph-mon[35690]: pgmap v586: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 59 MiB/s wr, 1.04k op/s 2024-03-11T19:17:39.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:39 smithi029 ceph-mon[32439]: pgmap v586: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 59 MiB/s wr, 1.04k op/s 2024-03-11T19:17:39.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:39 smithi190 ceph-mon[36292]: pgmap v586: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 59 MiB/s wr, 1.04k op/s 2024-03-11T19:17:40.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:40 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:17:40.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:40 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:17:40.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:40 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:17:41.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:41 smithi170 ceph-mon[35690]: pgmap v587: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 64 MiB/s wr, 1.14k op/s 2024-03-11T19:17:41.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:41 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:17:41.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:41 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:17:41.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:41 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:17:41.347 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:41 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:17:41.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:41 smithi029 ceph-mon[32439]: pgmap v587: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 64 MiB/s wr, 1.14k op/s 2024-03-11T19:17:41.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:41 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:17:41.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:41 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:17:41.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:41 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:17:41.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:41 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:17:41.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:41 smithi190 ceph-mon[36292]: pgmap v587: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 64 MiB/s wr, 1.14k op/s 2024-03-11T19:17:41.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:41 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:17:41.513 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:41 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:17:41.513 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:41 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:17:41.513 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:41 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:17:42.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:42 smithi170 ceph-mon[35690]: Adjusting osd_memory_target on smithi190 to 176.2M 2024-03-11T19:17:42.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:42 smithi170 ceph-mon[35690]: Unable to set osd_memory_target on smithi190 to 184793190: error parsing value: Value '184793190' is below minimum 939524096 2024-03-11T19:17:42.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:42 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:17:42.347 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:42 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:17:42.347 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:42 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:17:42.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:42 smithi029 ceph-mon[32439]: Adjusting osd_memory_target on smithi190 to 176.2M 2024-03-11T19:17:42.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:42 smithi029 ceph-mon[32439]: Unable to set osd_memory_target on smithi190 to 184793190: error parsing value: Value '184793190' is below minimum 939524096 2024-03-11T19:17:42.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:42 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:17:42.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:42 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:17:42.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:42 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:17:42.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:42 smithi190 ceph-mon[36292]: Adjusting osd_memory_target on smithi190 to 176.2M 2024-03-11T19:17:42.513 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:42 smithi190 ceph-mon[36292]: Unable to set osd_memory_target on smithi190 to 184793190: error parsing value: Value '184793190' is below minimum 939524096 2024-03-11T19:17:42.513 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:42 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:17:42.513 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:42 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:17:42.513 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:42 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:17:43.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:43 smithi190 ceph-mon[36292]: pgmap v588: 97 pgs: 97 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 64 MiB/s wr, 1.13k op/s 2024-03-11T19:17:43.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:43 smithi170 ceph-mon[35690]: pgmap v588: 97 pgs: 97 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 64 MiB/s wr, 1.13k op/s 2024-03-11T19:17:43.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:43 smithi029 ceph-mon[32439]: pgmap v588: 97 pgs: 97 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 64 MiB/s wr, 1.13k op/s 2024-03-11T19:17:45.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:45 smithi190 ceph-mon[36292]: pgmap v589: 97 pgs: 97 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 60 MiB/s wr, 1.08k op/s 2024-03-11T19:17:45.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:45 smithi170 ceph-mon[35690]: pgmap v589: 97 pgs: 97 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 60 MiB/s wr, 1.08k op/s 2024-03-11T19:17:45.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:45 smithi029 ceph-mon[32439]: pgmap v589: 97 pgs: 97 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 60 MiB/s wr, 1.08k op/s 2024-03-11T19:17:47.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:47 smithi190 ceph-mon[36292]: pgmap v590: 97 pgs: 97 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 66 MiB/s wr, 1.18k op/s 2024-03-11T19:17:47.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:47 smithi170 ceph-mon[35690]: pgmap v590: 97 pgs: 97 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 66 MiB/s wr, 1.18k op/s 2024-03-11T19:17:47.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:47 smithi029 ceph-mon[32439]: pgmap v590: 97 pgs: 97 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 66 MiB/s wr, 1.18k op/s 2024-03-11T19:17:49.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:49 smithi190 ceph-mon[36292]: pgmap v591: 97 pgs: 97 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 59 MiB/s wr, 1.07k op/s 2024-03-11T19:17:49.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:49 smithi170 ceph-mon[35690]: pgmap v591: 97 pgs: 97 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 59 MiB/s wr, 1.07k op/s 2024-03-11T19:17:49.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:49 smithi029 ceph-mon[32439]: pgmap v591: 97 pgs: 97 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 59 MiB/s wr, 1.07k op/s 2024-03-11T19:17:51.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:51 smithi190 ceph-mon[36292]: pgmap v592: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 65 MiB/s wr, 1.19k op/s 2024-03-11T19:17:51.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:51 smithi170 ceph-mon[35690]: pgmap v592: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 65 MiB/s wr, 1.19k op/s 2024-03-11T19:17:51.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:51 smithi029 ceph-mon[32439]: pgmap v592: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 65 MiB/s wr, 1.19k op/s 2024-03-11T19:17:53.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:53 smithi190 ceph-mon[36292]: pgmap v593: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 66 MiB/s wr, 1.21k op/s 2024-03-11T19:17:53.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:53 smithi170 ceph-mon[35690]: pgmap v593: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 66 MiB/s wr, 1.21k op/s 2024-03-11T19:17:53.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:53 smithi029 ceph-mon[32439]: pgmap v593: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 66 MiB/s wr, 1.21k op/s 2024-03-11T19:17:55.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:55 smithi190 ceph-mon[36292]: pgmap v594: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 62 MiB/s wr, 1.14k op/s 2024-03-11T19:17:55.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:55 smithi170 ceph-mon[35690]: pgmap v594: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 62 MiB/s wr, 1.14k op/s 2024-03-11T19:17:55.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:55 smithi029 ceph-mon[32439]: pgmap v594: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 62 MiB/s wr, 1.14k op/s 2024-03-11T19:17:57.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:57 smithi190 ceph-mon[36292]: pgmap v595: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 67 MiB/s wr, 1.24k op/s 2024-03-11T19:17:57.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:57 smithi170 ceph-mon[35690]: pgmap v595: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 67 MiB/s wr, 1.24k op/s 2024-03-11T19:17:57.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:57 smithi029 ceph-mon[32439]: pgmap v595: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 67 MiB/s wr, 1.24k op/s 2024-03-11T19:17:59.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:17:59 smithi190 ceph-mon[36292]: pgmap v596: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 60 MiB/s wr, 1.11k op/s 2024-03-11T19:17:59.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:17:59 smithi170 ceph-mon[35690]: pgmap v596: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 60 MiB/s wr, 1.11k op/s 2024-03-11T19:17:59.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:17:59 smithi029 ceph-mon[32439]: pgmap v596: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 60 MiB/s wr, 1.11k op/s 2024-03-11T19:18:00.937 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:18:00.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:18:01.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.266+0000 7ff372c26640 1 -- 172.21.15.29:0/2846544753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff36c100290 msgr2=0x7ff36c1006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:01.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.266+0000 7ff372c26640 1 --2- 172.21.15.29:0/2846544753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff36c100290 0x7ff36c1006f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff35c009900 tx=0x7ff35c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:18:01.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.266+0000 7ff372c26640 1 -- 172.21.15.29:0/2846544753 shutdown_connections 2024-03-11T19:18:01.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.266+0000 7ff372c26640 1 --2- 172.21.15.29:0/2846544753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff36c100c30 0x7ff36c1010b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.266+0000 7ff372c26640 1 --2- 172.21.15.29:0/2846544753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff36c100290 0x7ff36c1006f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.266+0000 7ff372c26640 1 --2- 172.21.15.29:0/2846544753 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff36c0ff930 0x7ff36c0ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.266+0000 7ff372c26640 1 -- 172.21.15.29:0/2846544753 >> 172.21.15.29:0/2846544753 conn(0x7ff36c0fb4f0 msgr2=0x7ff36c0fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:01.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.266+0000 7ff372c26640 1 -- 172.21.15.29:0/2846544753 shutdown_connections 2024-03-11T19:18:01.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.266+0000 7ff372c26640 1 -- 172.21.15.29:0/2846544753 wait complete. 2024-03-11T19:18:01.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.267+0000 7ff372c26640 1 Processor -- start 2024-03-11T19:18:01.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.267+0000 7ff372c26640 1 -- start start 2024-03-11T19:18:01.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.268+0000 7ff372c26640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff36c0ff930 0x7ff36c199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.268+0000 7ff372c26640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff36c100290 0x7ff36c199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.268+0000 7ff372c26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff36c100c30 0x7ff36c1a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.268+0000 7ff372c26640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c1a0d00 con 0x7ff36c100c30 2024-03-11T19:18:01.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.268+0000 7ff372c26640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c1a0e70 con 0x7ff36c100290 2024-03-11T19:18:01.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.268+0000 7ff372c26640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c1a1030 con 0x7ff36c0ff930 2024-03-11T19:18:01.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.268+0000 7ff371423640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff36c100290 0x7ff36c199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.268+0000 7ff371c24640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff36c0ff930 0x7ff36c199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.268+0000 7ff371423640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff36c100290 0x7ff36c199750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56540/0 (socket says 172.21.15.29:56540) 2024-03-11T19:18:01.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.268+0000 7ff371423640 1 -- 172.21.15.29:0/4174512847 learned_addr learned my addr 172.21.15.29:0/4174512847 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:01.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.269+0000 7ff372425640 1 --2- 172.21.15.29:0/4174512847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff36c100c30 0x7ff36c1a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.269+0000 7ff371423640 1 -- 172.21.15.29:0/4174512847 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff36c0ff930 msgr2=0x7ff36c199210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:01.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.269+0000 7ff371423640 1 --2- 172.21.15.29:0/4174512847 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff36c0ff930 0x7ff36c199210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.269+0000 7ff371423640 1 -- 172.21.15.29:0/4174512847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff36c100c30 msgr2=0x7ff36c1a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:01.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.269+0000 7ff371423640 1 --2- 172.21.15.29:0/4174512847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff36c100c30 0x7ff36c1a07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.269+0000 7ff371423640 1 -- 172.21.15.29:0/4174512847 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c009560 con 0x7ff36c100290 2024-03-11T19:18:01.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.269+0000 7ff372425640 1 --2- 172.21.15.29:0/4174512847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff36c100c30 0x7ff36c1a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:18:01.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.269+0000 7ff371423640 1 --2- 172.21.15.29:0/4174512847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff36c100290 0x7ff36c199750 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff35c005ab0 tx=0x7ff35c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:01.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.269+0000 7ff362ffd640 1 -- 172.21.15.29:0/4174512847 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c004ac0 con 0x7ff36c100290 2024-03-11T19:18:01.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.270+0000 7ff372c26640 1 -- 172.21.15.29:0/4174512847 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff36c1a12b0 con 0x7ff36c100290 2024-03-11T19:18:01.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.270+0000 7ff372c26640 1 -- 172.21.15.29:0/4174512847 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff36c1a1810 con 0x7ff36c100290 2024-03-11T19:18:01.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.270+0000 7ff362ffd640 1 -- 172.21.15.29:0/4174512847 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff35c0044e0 con 0x7ff36c100290 2024-03-11T19:18:01.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.270+0000 7ff362ffd640 1 -- 172.21.15.29:0/4174512847 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c0055c0 con 0x7ff36c100290 2024-03-11T19:18:01.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.273+0000 7ff362ffd640 1 -- 172.21.15.29:0/4174512847 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff35c03e450 con 0x7ff36c100290 2024-03-11T19:18:01.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.273+0000 7ff376a69740 1 Processor -- start 2024-03-11T19:18:01.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.274+0000 7ff376a69740 1 -- start start 2024-03-11T19:18:01.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.274+0000 7ff376a69740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c0124b0 0x55cc9c0128d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.274+0000 7ff376a69740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c012ea0 0x55cc9c0179a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.274+0000 7ff376a69740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c018010 0x55cc9c01a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.274+0000 7ff376a69740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9bf82280 con 0x55cc9c0124b0 2024-03-11T19:18:01.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.274+0000 7ff376a69740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9bf7f7b0 con 0x55cc9c012ea0 2024-03-11T19:18:01.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.274+0000 7ff376a69740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9bf7af90 con 0x55cc9c018010 2024-03-11T19:18:01.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.275+0000 7ff371c24640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c012ea0 0x55cc9c0179a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.275+0000 7ff371c24640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c012ea0 0x55cc9c0179a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56544/0 (socket says 172.21.15.29:56544) 2024-03-11T19:18:01.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.275+0000 7ff371c24640 1 -- 172.21.15.29:0/3777706256 learned_addr learned my addr 172.21.15.29:0/3777706256 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:01.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.275+0000 7ff372425640 1 --2- 172.21.15.29:0/3777706256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c0124b0 0x55cc9c0128d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.275+0000 7ff371423640 1 --2- 172.21.15.29:0/3777706256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c018010 0x55cc9c01a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.276+0000 7ff372425640 1 -- 172.21.15.29:0/3777706256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c018010 msgr2=0x55cc9c01a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.276+0000 7ff372425640 1 --2- 172.21.15.29:0/3777706256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c018010 0x55cc9c01a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.276+0000 7ff372425640 1 -- 172.21.15.29:0/3777706256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c012ea0 msgr2=0x55cc9c0179a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.276+0000 7ff372425640 1 --2- 172.21.15.29:0/3777706256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c012ea0 0x55cc9c0179a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.276+0000 7ff372425640 1 -- 172.21.15.29:0/3777706256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc9bf39570 con 0x55cc9c0124b0 2024-03-11T19:18:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.276+0000 7ff372425640 1 --2- 172.21.15.29:0/3777706256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c0124b0 0x55cc9c0128d0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ff36400bc20 tx=0x7ff36400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=add7e87bbc6325b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.276+0000 7ff362ffd640 1 --2- 172.21.15.29:0/4174512847 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff3580a1ab0 0x7ff3580a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.277+0000 7ff371c24640 1 --2- 172.21.15.29:0/4174512847 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff3580a1ab0 0x7ff3580a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.277+0000 7ff362ffd640 1 -- 172.21.15.29:0/4174512847 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff35c02de40 con 0x7ff36c100290 2024-03-11T19:18:01.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.277+0000 7ff357fff640 1 -- 172.21.15.29:0/3777706256 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36401a460 con 0x55cc9c0124b0 2024-03-11T19:18:01.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.277+0000 7ff371c24640 1 --2- 172.21.15.29:0/4174512847 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff3580a1ab0 0x7ff3580a3f70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x55cc9c0132a0 tx=0x7ff368009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:01.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.277+0000 7ff357fff640 1 -- 172.21.15.29:0/3777706256 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff36401aa70 con 0x55cc9c0124b0 2024-03-11T19:18:01.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.278+0000 7ff357fff640 1 -- 172.21.15.29:0/3777706256 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3640119d0 con 0x55cc9c0124b0 2024-03-11T19:18:01.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.279+0000 7ff376a69740 1 -- 172.21.15.29:0/3777706256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c0124b0 msgr2=0x55cc9c0128d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:01.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.279+0000 7ff376a69740 1 --2- 172.21.15.29:0/3777706256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c0124b0 0x55cc9c0128d0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ff36400bc20 tx=0x7ff36400f730 comp rx=0 tx=0).stop 2024-03-11T19:18:01.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.279+0000 7ff376a69740 1 -- 172.21.15.29:0/3777706256 shutdown_connections 2024-03-11T19:18:01.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.279+0000 7ff376a69740 1 --2- 172.21.15.29:0/3777706256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c018010 0x55cc9c01a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.279+0000 7ff376a69740 1 --2- 172.21.15.29:0/3777706256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c012ea0 0x55cc9c0179a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.279+0000 7ff376a69740 1 --2- 172.21.15.29:0/3777706256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c0124b0 0x55cc9c0128d0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.279+0000 7ff376a69740 1 -- 172.21.15.29:0/3777706256 >> 172.21.15.29:0/3777706256 conn(0x55cc9c00da20 msgr2=0x55cc9c00de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:01.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.279+0000 7ff376a69740 1 -- 172.21.15.29:0/3777706256 shutdown_connections 2024-03-11T19:18:01.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.279+0000 7ff376a69740 1 -- 172.21.15.29:0/3777706256 wait complete. 2024-03-11T19:18:01.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.280+0000 7ff376a69740 1 -- start start 2024-03-11T19:18:01.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.280+0000 7ff376a69740 1 Processor -- start 2024-03-11T19:18:01.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.280+0000 7ff376a69740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c0124b0 0x55cc9c02d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.281+0000 7ff376a69740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c012ea0 0x55cc9c02dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.281+0000 7ff376a69740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c018010 0x55cc9c034b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.281+0000 7ff376a69740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9bc4e430 con 0x55cc9c018010 2024-03-11T19:18:01.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.281+0000 7ff376a69740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9bc0ae00 con 0x55cc9c012ea0 2024-03-11T19:18:01.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.281+0000 7ff376a69740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9befffd0 con 0x55cc9c0124b0 2024-03-11T19:18:01.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.281+0000 7ff376a69740 10 client.0 _ll_register_callbacks cb 0x55cc9bf55fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:01.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.281+0000 7ff371c24640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c0124b0 0x55cc9c02d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.281+0000 7ff371c24640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c0124b0 0x55cc9c02d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:44156/0 (socket says 172.21.15.29:44156) 2024-03-11T19:18:01.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.281+0000 7ff371c24640 1 -- 172.21.15.29:0/1408338694 learned_addr learned my addr 172.21.15.29:0/1408338694 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:01.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.281+0000 7ff371423640 1 --2- 172.21.15.29:0/1408338694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c012ea0 0x55cc9c02dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.281+0000 7ff371c24640 1 -- 172.21.15.29:0/1408338694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c012ea0 msgr2=0x55cc9c02dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:01.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.282+0000 7ff371c24640 1 --2- 172.21.15.29:0/1408338694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c012ea0 0x55cc9c02dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.282+0000 7ff371c24640 1 -- 172.21.15.29:0/1408338694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c018010 msgr2=0x55cc9c034b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:01.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.282+0000 7ff371c24640 1 --2- 172.21.15.29:0/1408338694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c018010 0x55cc9c034b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.282+0000 7ff371c24640 1 -- 172.21.15.29:0/1408338694 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff36400b850 con 0x55cc9c0124b0 2024-03-11T19:18:01.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.282+0000 7ff371c24640 1 --2- 172.21.15.29:0/1408338694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c0124b0 0x55cc9c02d590 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff36800fae0 tx=0x7ff368004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43a5a33150671bb server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:01.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.283+0000 7ff3557fa640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:18:01.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.283+0000 7ff3557fa640 1 -- 172.21.15.29:0/1408338694 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff368023070 con 0x55cc9c0124b0 2024-03-11T19:18:01.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.283+0000 7ff3557fa640 1 -- 172.21.15.29:0/1408338694 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff368004b20 con 0x55cc9c0124b0 2024-03-11T19:18:01.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.284+0000 7ff376a69740 1 -- 172.21.15.29:0/1408338694 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cc9bc7a7e0 con 0x55cc9c0124b0 2024-03-11T19:18:01.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.284+0000 7ff3557fa640 1 -- 172.21.15.29:0/1408338694 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff368017070 con 0x55cc9c0124b0 2024-03-11T19:18:01.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.284+0000 7ff376a69740 10 client.26857 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:01.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.284+0000 7ff376a69740 1 -- 172.21.15.29:0/1408338694 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55cc9bf79610 con 0x55cc9c0124b0 2024-03-11T19:18:01.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.285+0000 7ff3557fa640 1 -- 172.21.15.29:0/1408338694 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55cc9bc7a7e0 con 0x55cc9c0124b0 2024-03-11T19:18:01.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.285+0000 7ff3557fa640 1 -- 172.21.15.29:0/1408338694 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff368029750 con 0x55cc9c0124b0 2024-03-11T19:18:01.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.285+0000 7ff376a69740 10 client.26857 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:01.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.285+0000 7ff376a69740 10 client.26857 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:18:01.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.285+0000 7ff376a69740 20 client.26857 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:01.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.285+0000 7ff376a69740 1 --2- 172.21.15.29:0/1408338694 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cc9c039790 0x55cc9c03bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.285+0000 7ff376a69740 4 client.26857 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:18:01.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.285+0000 7ff376a69740 1 -- 172.21.15.29:0/1408338694 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cc9bcb7720 con 0x55cc9c039790 2024-03-11T19:18:01.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.286+0000 7ff371423640 1 --2- 172.21.15.29:0/1408338694 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cc9c039790 0x55cc9c03bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.287+0000 7ff3557fa640 10 client.26857 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:01.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.287+0000 7ff371423640 1 --2- 172.21.15.29:0/1408338694 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cc9c039790 0x55cc9c03bbb0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c040925a1d91b1b5 server_cookie=cdc7cc5e395890bd in_seq=0 out_seq=0 2024-03-11T19:18:01.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.288+0000 7ff3557fa640 1 -- 172.21.15.29:0/1408338694 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cc9bcb7720 con 0x55cc9c039790 2024-03-11T19:18:01.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.288+0000 7ff3557fa640 10 client.26857 handle_command_reply: tid=0 2024-03-11T19:18:01.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.288+0000 7ff376a69740 1 client.26857 shutdown 2024-03-11T19:18:01.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.288+0000 7ff376a69740 1 -- 172.21.15.29:0/1408338694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c0124b0 msgr2=0x55cc9c02d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:01.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.288+0000 7ff376a69740 1 --2- 172.21.15.29:0/1408338694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c0124b0 0x55cc9c02d590 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff36800fae0 tx=0x7ff368004ab0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.288+0000 7ff376a69740 1 -- 172.21.15.29:0/1408338694 shutdown_connections 2024-03-11T19:18:01.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.288+0000 7ff376a69740 1 --2- 172.21.15.29:0/1408338694 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cc9c039790 0x55cc9c03bbb0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.288+0000 7ff376a69740 1 --2- 172.21.15.29:0/1408338694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c018010 0x55cc9c034b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.288+0000 7ff376a69740 1 --2- 172.21.15.29:0/1408338694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c012ea0 0x55cc9c02dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.288+0000 7ff3557fa640 0 client.26857 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:01.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.288+0000 7ff376a69740 1 --2- 172.21.15.29:0/1408338694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c0124b0 0x55cc9c02d590 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.288+0000 7ff376a69740 1 -- 172.21.15.29:0/1408338694 >> 172.21.15.29:0/1408338694 conn(0x55cc9c00da20 msgr2=0x55cc9c00cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:01.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.288+0000 7ff376a69740 1 -- 172.21.15.29:0/1408338694 shutdown_connections 2024-03-11T19:18:01.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.288+0000 7ff376a69740 1 -- 172.21.15.29:0/1408338694 wait complete. 2024-03-11T19:18:01.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.288+0000 7ff376a69740 20 client.26857 trim_cache size 0 max 16384 2024-03-11T19:18:01.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.321+0000 7ff376a69740 1 Processor -- start 2024-03-11T19:18:01.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.321+0000 7ff376a69740 1 -- start start 2024-03-11T19:18:01.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.321+0000 7ff376a69740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c0124b0 0x55cc9c02d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.321+0000 7ff376a69740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c012ea0 0x55cc9c034ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.321+0000 7ff376a69740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c018010 0x55cc9c02d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.322+0000 7ff371423640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c012ea0 0x55cc9c034ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.322+0000 7ff371423640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c012ea0 0x55cc9c034ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56562/0 (socket says 172.21.15.29:56562) 2024-03-11T19:18:01.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.322+0000 7ff372425640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c018010 0x55cc9c02d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.322+0000 7ff372425640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c018010 0x55cc9c02d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:44162/0 (socket says 172.21.15.29:44162) 2024-03-11T19:18:01.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.322+0000 7ff371423640 1 -- 172.21.15.29:0/800552587 learned_addr learned my addr 172.21.15.29:0/800552587 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:01.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.322+0000 7ff371c24640 1 --2- 172.21.15.29:0/800552587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c0124b0 0x55cc9c02d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.322+0000 7ff376a69740 1 -- 172.21.15.29:0/800552587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9c02d010 con 0x55cc9c0124b0 2024-03-11T19:18:01.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.322+0000 7ff376a69740 1 -- 172.21.15.29:0/800552587 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9c01a960 con 0x55cc9c012ea0 2024-03-11T19:18:01.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.322+0000 7ff376a69740 1 -- 172.21.15.29:0/800552587 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9c00d5a0 con 0x55cc9c018010 2024-03-11T19:18:01.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.323+0000 7ff372425640 1 -- 172.21.15.29:0/800552587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c012ea0 msgr2=0x55cc9c034ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:01.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.323+0000 7ff372425640 1 --2- 172.21.15.29:0/800552587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c012ea0 0x55cc9c034ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.323+0000 7ff372425640 1 -- 172.21.15.29:0/800552587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c0124b0 msgr2=0x55cc9c02d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:01.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.323+0000 7ff372425640 1 --2- 172.21.15.29:0/800552587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c0124b0 0x55cc9c02d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.323+0000 7ff372425640 1 -- 172.21.15.29:0/800552587 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc9bf15ac0 con 0x55cc9c018010 2024-03-11T19:18:01.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.323+0000 7ff372425640 1 --2- 172.21.15.29:0/800552587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c018010 0x55cc9c02d6e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x55cc9c008160 tx=0x7ff36400b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a3c234e7b751568 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:01.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.323+0000 7ff3577fe640 1 -- 172.21.15.29:0/800552587 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff364020070 con 0x55cc9c018010 2024-03-11T19:18:01.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.323+0000 7ff3577fe640 1 -- 172.21.15.29:0/800552587 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff36400bdc0 con 0x55cc9c018010 2024-03-11T19:18:01.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.323+0000 7ff3577fe640 1 -- 172.21.15.29:0/800552587 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff364004900 con 0x55cc9c018010 2024-03-11T19:18:01.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.324+0000 7ff376a69740 1 -- 172.21.15.29:0/800552587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c018010 msgr2=0x55cc9c02d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:01.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.325+0000 7ff376a69740 1 --2- 172.21.15.29:0/800552587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c018010 0x55cc9c02d6e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x55cc9c008160 tx=0x7ff36400b7e0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.325+0000 7ff376a69740 1 -- 172.21.15.29:0/800552587 shutdown_connections 2024-03-11T19:18:01.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.325+0000 7ff376a69740 1 --2- 172.21.15.29:0/800552587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c018010 0x55cc9c02d6e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.325+0000 7ff376a69740 1 --2- 172.21.15.29:0/800552587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c012ea0 0x55cc9c034ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.325+0000 7ff376a69740 1 --2- 172.21.15.29:0/800552587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c0124b0 0x55cc9c02d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.325+0000 7ff376a69740 1 -- 172.21.15.29:0/800552587 >> 172.21.15.29:0/800552587 conn(0x55cc9c00da20 msgr2=0x55cc9c014260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:01.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.325+0000 7ff376a69740 1 -- 172.21.15.29:0/800552587 shutdown_connections 2024-03-11T19:18:01.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.325+0000 7ff376a69740 1 -- 172.21.15.29:0/800552587 wait complete. 2024-03-11T19:18:01.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.326+0000 7ff376a69740 1 -- start start 2024-03-11T19:18:01.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.326+0000 7ff376a69740 1 Processor -- start 2024-03-11T19:18:01.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.327+0000 7ff376a69740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c0124b0 0x55cc9c033240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.327+0000 7ff376a69740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c012ea0 0x55cc9c033780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.327+0000 7ff371c24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c0124b0 0x55cc9c033240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.327+0000 7ff371c24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c0124b0 0x55cc9c033240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49406/0 (socket says 172.21.15.29:49406) 2024-03-11T19:18:01.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.327+0000 7ff371423640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c012ea0 0x55cc9c033780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.327+0000 7ff371423640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c012ea0 0x55cc9c033780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:44168/0 (socket says 172.21.15.29:44168) 2024-03-11T19:18:01.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.327+0000 7ff376a69740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c018010 0x55cc9c033cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.327+0000 7ff371c24640 1 -- 172.21.15.29:0/3985145240 learned_addr learned my addr 172.21.15.29:0/3985145240 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:01.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.328+0000 7ff376a69740 1 -- 172.21.15.29:0/3985145240 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9c034e90 con 0x55cc9c0124b0 2024-03-11T19:18:01.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.328+0000 7ff376a69740 1 -- 172.21.15.29:0/3985145240 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9c08fca0 con 0x55cc9c018010 2024-03-11T19:18:01.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.328+0000 7ff376a69740 1 -- 172.21.15.29:0/3985145240 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9c08fde0 con 0x55cc9c012ea0 2024-03-11T19:18:01.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.328+0000 7ff372425640 1 --2- 172.21.15.29:0/3985145240 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c018010 0x55cc9c033cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.328+0000 7ff376a69740 10 client.0 _ll_register_callbacks cb 0x7ff36400b420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:01.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.328+0000 7ff371423640 1 -- 172.21.15.29:0/3985145240 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c018010 msgr2=0x55cc9c033cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:01.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.328+0000 7ff371423640 1 --2- 172.21.15.29:0/3985145240 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c018010 0x55cc9c033cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.328+0000 7ff371423640 1 -- 172.21.15.29:0/3985145240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c0124b0 msgr2=0x55cc9c033240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:01.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.328+0000 7ff371423640 1 --2- 172.21.15.29:0/3985145240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c0124b0 0x55cc9c033240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.328+0000 7ff371423640 1 -- 172.21.15.29:0/3985145240 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc9bf39570 con 0x55cc9c012ea0 2024-03-11T19:18:01.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.329+0000 7ff371423640 1 --2- 172.21.15.29:0/3985145240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c012ea0 0x55cc9c033780 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff35c005ec0 tx=0x7ff35c02bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=888b63c3cc5a389d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:01.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.329+0000 7ff3557fa640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:18:01.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.329+0000 7ff3557fa640 1 -- 172.21.15.29:0/3985145240 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c0c53b0 con 0x55cc9c012ea0 2024-03-11T19:18:01.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.329+0000 7ff3557fa640 1 -- 172.21.15.29:0/3985145240 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff35c0b9280 con 0x55cc9c012ea0 2024-03-11T19:18:01.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.329+0000 7ff376a69740 1 -- 172.21.15.29:0/3985145240 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff36401aa70 con 0x55cc9c012ea0 2024-03-11T19:18:01.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.329+0000 7ff3557fa640 1 -- 172.21.15.29:0/3985145240 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff35c0b9440 con 0x55cc9c012ea0 2024-03-11T19:18:01.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.329+0000 7ff376a69740 10 client.26863 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:01.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.329+0000 7ff376a69740 1 -- 172.21.15.29:0/3985145240 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55cc9bf3b6a0 con 0x55cc9c012ea0 2024-03-11T19:18:01.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.330+0000 7ff3557fa640 1 -- 172.21.15.29:0/3985145240 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff36401aa70 con 0x55cc9c012ea0 2024-03-11T19:18:01.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.330+0000 7ff3557fa640 1 -- 172.21.15.29:0/3985145240 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff35c03c070 con 0x55cc9c012ea0 2024-03-11T19:18:01.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.330+0000 7ff376a69740 10 client.26863 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:01.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.330+0000 7ff376a69740 10 client.26863 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:18:01.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.330+0000 7ff376a69740 20 client.26863 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:01.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.330+0000 7ff376a69740 1 --2- 172.21.15.29:0/3985145240 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cc9c0914e0 0x55cc9c093900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.330+0000 7ff376a69740 4 client.26863 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:18:01.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.330+0000 7ff376a69740 1 -- 172.21.15.29:0/3985145240 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55cc9bd06890 con 0x55cc9c0914e0 2024-03-11T19:18:01.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.330+0000 7ff371c24640 1 --2- 172.21.15.29:0/3985145240 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cc9c0914e0 0x55cc9c093900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.331+0000 7ff3557fa640 10 client.26863 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:01.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.331+0000 7ff371c24640 1 --2- 172.21.15.29:0/3985145240 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cc9c0914e0 0x55cc9c093900 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e8850e9d4d0ebad server_cookie=81f0e6b9d5b5eef3 in_seq=0 out_seq=0 2024-03-11T19:18:01.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.331+0000 7ff3557fa640 1 -- 172.21.15.29:0/3985145240 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55cc9bd06890 con 0x55cc9c0914e0 2024-03-11T19:18:01.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.331+0000 7ff3557fa640 10 client.26863 handle_command_reply: tid=0 2024-03-11T19:18:01.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.331+0000 7ff376a69740 1 client.26863 shutdown 2024-03-11T19:18:01.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.332+0000 7ff376a69740 1 -- 172.21.15.29:0/3985145240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c012ea0 msgr2=0x55cc9c033780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:01.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.332+0000 7ff376a69740 1 --2- 172.21.15.29:0/3985145240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c012ea0 0x55cc9c033780 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff35c005ec0 tx=0x7ff35c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:18:01.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.332+0000 7ff376a69740 1 -- 172.21.15.29:0/3985145240 shutdown_connections 2024-03-11T19:18:01.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.332+0000 7ff376a69740 1 --2- 172.21.15.29:0/3985145240 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc9c018010 0x55cc9c033cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.332+0000 7ff376a69740 1 --2- 172.21.15.29:0/3985145240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9c012ea0 0x55cc9c033780 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.332+0000 7ff376a69740 1 --2- 172.21.15.29:0/3985145240 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cc9c0914e0 0x55cc9c093900 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.332+0000 7ff3557fa640 0 client.26863 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:01.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.332+0000 7ff376a69740 1 --2- 172.21.15.29:0/3985145240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc9c0124b0 0x55cc9c033240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.332+0000 7ff376a69740 1 -- 172.21.15.29:0/3985145240 >> 172.21.15.29:0/3985145240 conn(0x55cc9c00da20 msgr2=0x55cc9c02d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:01.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.332+0000 7ff376a69740 1 -- 172.21.15.29:0/3985145240 shutdown_connections 2024-03-11T19:18:01.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.333+0000 7ff376a69740 1 -- 172.21.15.29:0/3985145240 wait complete. 2024-03-11T19:18:01.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.333+0000 7ff376a69740 20 client.26863 trim_cache size 0 max 16384 2024-03-11T19:18:01.339 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:18:01.339 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:18:01.339 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:18:01.339 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:18:01.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.334+0000 7ff372c26640 1 -- 172.21.15.29:0/4174512847 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff3580a1ab0 msgr2=0x7ff3580a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:01.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.334+0000 7ff372c26640 1 --2- 172.21.15.29:0/4174512847 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff3580a1ab0 0x7ff3580a3f70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x55cc9c0132a0 tx=0x7ff368009500 comp rx=0 tx=0).stop 2024-03-11T19:18:01.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.334+0000 7ff372c26640 1 -- 172.21.15.29:0/4174512847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff36c100290 msgr2=0x7ff36c199750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:01.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.334+0000 7ff372c26640 1 --2- 172.21.15.29:0/4174512847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff36c100290 0x7ff36c199750 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff35c005ab0 tx=0x7ff35c0042f0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.334+0000 7ff372c26640 1 -- 172.21.15.29:0/4174512847 shutdown_connections 2024-03-11T19:18:01.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.334+0000 7ff372c26640 1 --2- 172.21.15.29:0/4174512847 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff3580a1ab0 0x7ff3580a3f70 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.334+0000 7ff372c26640 1 --2- 172.21.15.29:0/4174512847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff36c100c30 0x7ff36c1a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.334+0000 7ff372c26640 1 --2- 172.21.15.29:0/4174512847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff36c100290 0x7ff36c199750 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.334+0000 7ff372c26640 1 --2- 172.21.15.29:0/4174512847 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff36c0ff930 0x7ff36c199210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.334+0000 7ff372c26640 1 -- 172.21.15.29:0/4174512847 >> 172.21.15.29:0/4174512847 conn(0x7ff36c0fb4f0 msgr2=0x7ff36c0fcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:01.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.334+0000 7ff372c26640 1 -- 172.21.15.29:0/4174512847 shutdown_connections 2024-03-11T19:18:01.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.334+0000 7ff372c26640 1 -- 172.21.15.29:0/4174512847 wait complete. 2024-03-11T19:18:01.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:01 smithi029 ceph-mon[32439]: pgmap v597: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 65 MiB/s wr, 1.21k op/s 2024-03-11T19:18:01.359 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:18:01.359 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:18:01.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:01 smithi190 ceph-mon[36292]: pgmap v597: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 65 MiB/s wr, 1.21k op/s 2024-03-11T19:18:01.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.538+0000 7f50737fe640 1 -- 172.21.15.29:0/1421333183 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5068005220 con 0x7f50740751c0 2024-03-11T19:18:01.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.538+0000 7f5079b09640 1 -- 172.21.15.29:0/1421333183 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50740751c0 msgr2=0x7f5074073610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:01.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.538+0000 7f5079b09640 1 --2- 172.21.15.29:0/1421333183 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50740751c0 0x7f5074073610 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5068009930 tx=0x7f506802bc50 comp rx=0 tx=0).stop 2024-03-11T19:18:01.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.538+0000 7f5079b09640 1 -- 172.21.15.29:0/1421333183 shutdown_connections 2024-03-11T19:18:01.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.538+0000 7f5079b09640 1 --2- 172.21.15.29:0/1421333183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f507410d590 0x7f507410f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.538+0000 7f5079b09640 1 --2- 172.21.15.29:0/1421333183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5074073be0 0x7f5074074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.538+0000 7f5079b09640 1 --2- 172.21.15.29:0/1421333183 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50740751c0 0x7f5074073610 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.538+0000 7f5079b09640 1 -- 172.21.15.29:0/1421333183 >> 172.21.15.29:0/1421333183 conn(0x7f50740fc510 msgr2=0x7f50740fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.538+0000 7f5079b09640 1 -- 172.21.15.29:0/1421333183 shutdown_connections 2024-03-11T19:18:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.538+0000 7f5079b09640 1 -- 172.21.15.29:0/1421333183 wait complete. 2024-03-11T19:18:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.539+0000 7f5079b09640 1 Processor -- start 2024-03-11T19:18:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.539+0000 7f5079b09640 1 -- start start 2024-03-11T19:18:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.539+0000 7f5079b09640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5074073be0 0x7f507419d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.539+0000 7f5079b09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50740751c0 0x7f507419db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.539+0000 7f5079b09640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f507410d590 0x7f50741a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.539+0000 7f5079b09640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50741a5070 con 0x7f50740751c0 2024-03-11T19:18:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.539+0000 7f5079b09640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50741a51e0 con 0x7f507410d590 2024-03-11T19:18:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.539+0000 7f5079b09640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50741a53a0 con 0x7f5074073be0 2024-03-11T19:18:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.540+0000 7f5079308640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f507410d590 0x7f50741a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.540+0000 7f5079308640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f507410d590 0x7f50741a4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56590/0 (socket says 172.21.15.29:56590) 2024-03-11T19:18:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.540+0000 7f5079308640 1 -- 172.21.15.29:0/806484443 learned_addr learned my addr 172.21.15.29:0/806484443 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.540+0000 7f5078b07640 1 --2- 172.21.15.29:0/806484443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5074073be0 0x7f507419d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.540+0000 7f5073fff640 1 --2- 172.21.15.29:0/806484443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50740751c0 0x7f507419db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.540+0000 7f5079308640 1 -- 172.21.15.29:0/806484443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5074073be0 msgr2=0x7f507419d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:01.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.540+0000 7f5079308640 1 --2- 172.21.15.29:0/806484443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5074073be0 0x7f507419d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.540+0000 7f5079308640 1 -- 172.21.15.29:0/806484443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50740751c0 msgr2=0x7f507419db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:01.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.540+0000 7f5079308640 1 --2- 172.21.15.29:0/806484443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50740751c0 0x7f507419db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.540+0000 7f5079308640 1 -- 172.21.15.29:0/806484443 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5068009560 con 0x7f507410d590 2024-03-11T19:18:01.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.540+0000 7f5079308640 1 --2- 172.21.15.29:0/806484443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f507410d590 0x7f50741a4b30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f506c00eb10 tx=0x7f506c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:01.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.541+0000 7f5071ffb640 1 -- 172.21.15.29:0/806484443 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c004b20 con 0x7f507410d590 2024-03-11T19:18:01.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.541+0000 7f5079b09640 1 -- 172.21.15.29:0/806484443 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50741a5680 con 0x7f507410d590 2024-03-11T19:18:01.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.541+0000 7f5071ffb640 1 -- 172.21.15.29:0/806484443 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f506c004c80 con 0x7f507410d590 2024-03-11T19:18:01.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.541+0000 7f5071ffb640 1 -- 172.21.15.29:0/806484443 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c010690 con 0x7f507410d590 2024-03-11T19:18:01.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.541+0000 7f5079b09640 1 -- 172.21.15.29:0/806484443 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50741a5b90 con 0x7f507410d590 2024-03-11T19:18:01.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.542+0000 7f5079b09640 1 -- 172.21.15.29:0/806484443 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5074103650 con 0x7f507410d590 2024-03-11T19:18:01.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.543+0000 7f5071ffb640 1 -- 172.21.15.29:0/806484443 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f506c005020 con 0x7f507410d590 2024-03-11T19:18:01.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.544+0000 7f5071ffb640 1 --2- 172.21.15.29:0/806484443 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f50640a1ab0 0x7f50640a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:01.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.544+0000 7f5071ffb640 1 -- 172.21.15.29:0/806484443 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f506c014070 con 0x7f507410d590 2024-03-11T19:18:01.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.544+0000 7f5078b07640 1 --2- 172.21.15.29:0/806484443 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f50640a1ab0 0x7f50640a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:01.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.545+0000 7f5078b07640 1 --2- 172.21.15.29:0/806484443 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f50640a1ab0 0x7f50640a3f70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f5068006010 tx=0x7f5068000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:01.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.548+0000 7f5071ffb640 1 -- 172.21.15.29:0/806484443 <== mon.2 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f506c096bc0 con 0x7f507410d590 2024-03-11T19:18:01.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:01 smithi170 ceph-mon[35690]: pgmap v597: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 65 MiB/s wr, 1.21k op/s 2024-03-11T19:18:01.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.901+0000 7f5079b09640 1 -- 172.21.15.29:0/806484443 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f507402ce40 con 0x7f507410d590 2024-03-11T19:18:01.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.902+0000 7f5071ffb640 1 -- 172.21.15.29:0/806484443 <== mon.2 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f506c09a280 con 0x7f507410d590 2024-03-11T19:18:01.906 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:18:01.906 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:18:01.907 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:18:01.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.906+0000 7f5079b09640 1 -- 172.21.15.29:0/806484443 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f50640a1ab0 msgr2=0x7f50640a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:01.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.906+0000 7f5079b09640 1 --2- 172.21.15.29:0/806484443 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f50640a1ab0 0x7f50640a3f70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f5068006010 tx=0x7f5068000bc0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.906+0000 7f5079b09640 1 -- 172.21.15.29:0/806484443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f507410d590 msgr2=0x7f50741a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:01.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.906+0000 7f5079b09640 1 --2- 172.21.15.29:0/806484443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f507410d590 0x7f50741a4b30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f506c00eb10 tx=0x7f506c00c2d0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.906+0000 7f5079b09640 1 -- 172.21.15.29:0/806484443 shutdown_connections 2024-03-11T19:18:01.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.906+0000 7f5079b09640 1 --2- 172.21.15.29:0/806484443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f507410d590 0x7f50741a4b30 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.906+0000 7f5079b09640 1 --2- 172.21.15.29:0/806484443 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f50640a1ab0 0x7f50640a3f70 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.906+0000 7f5079b09640 1 --2- 172.21.15.29:0/806484443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50740751c0 0x7f507419db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.906+0000 7f5079b09640 1 --2- 172.21.15.29:0/806484443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5074073be0 0x7f507419d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:01.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.906+0000 7f5079b09640 1 -- 172.21.15.29:0/806484443 >> 172.21.15.29:0/806484443 conn(0x7f50740fc510 msgr2=0x7f5074100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:01.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.906+0000 7f5079b09640 1 -- 172.21.15.29:0/806484443 shutdown_connections 2024-03-11T19:18:01.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:01.906+0000 7f5079b09640 1 -- 172.21.15.29:0/806484443 wait complete. 2024-03-11T19:18:01.928 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:18:01.929 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:18:02.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.114+0000 7f7b35b48640 1 -- 172.21.15.29:0/1814595607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b30076d80 msgr2=0x7f7b300771e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.114+0000 7f7b35b48640 1 --2- 172.21.15.29:0/1814595607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b30076d80 0x7f7b300771e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7b20009860 tx=0x7f7b2002bb70 comp rx=0 tx=0).stop 2024-03-11T19:18:02.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.115+0000 7f7b35b48640 1 -- 172.21.15.29:0/1814595607 shutdown_connections 2024-03-11T19:18:02.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.115+0000 7f7b35b48640 1 --2- 172.21.15.29:0/1814595607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b30077720 0x7f7b30077fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.115+0000 7f7b35b48640 1 --2- 172.21.15.29:0/1814595607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b30076d80 0x7f7b300771e0 secure :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7b20009860 tx=0x7f7b2002bb70 comp rx=0 tx=0).stop 2024-03-11T19:18:02.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.115+0000 7f7b35b48640 1 --2- 172.21.15.29:0/1814595607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b30075b40 0x7f7b30075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.115+0000 7f7b35b48640 1 -- 172.21.15.29:0/1814595607 >> 172.21.15.29:0/1814595607 conn(0x7f7b300fe930 msgr2=0x7f7b30100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.115+0000 7f7b35b48640 1 -- 172.21.15.29:0/1814595607 shutdown_connections 2024-03-11T19:18:02.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.115+0000 7f7b35b48640 1 -- 172.21.15.29:0/1814595607 wait complete. 2024-03-11T19:18:02.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.116+0000 7f7b35b48640 1 Processor -- start 2024-03-11T19:18:02.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.116+0000 7f7b35b48640 1 -- start start 2024-03-11T19:18:02.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.116+0000 7f7b35b48640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b30075b40 0x7f7b3019d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.116+0000 7f7b35b48640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b30077720 0x7f7b3019def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.116+0000 7f7b35b48640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b301a2f50 0x7f7b301a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.116+0000 7f7b35b48640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b301a5880 con 0x7f7b30075b40 2024-03-11T19:18:02.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.116+0000 7f7b35b48640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b301a59f0 con 0x7f7b30077720 2024-03-11T19:18:02.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.116+0000 7f7b35b48640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b301a5bb0 con 0x7f7b301a2f50 2024-03-11T19:18:02.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.116+0000 7f7b27fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b30077720 0x7f7b3019def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.116+0000 7f7b27fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b30077720 0x7f7b3019def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56600/0 (socket says 172.21.15.29:56600) 2024-03-11T19:18:02.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.116+0000 7f7b27fff640 1 -- 172.21.15.29:0/1604803717 learned_addr learned my addr 172.21.15.29:0/1604803717 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:02.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.117+0000 7f7b2ffff640 1 --2- 172.21.15.29:0/1604803717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b301a2f50 0x7f7b301a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.117+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/1604803717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b30075b40 0x7f7b3019d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.117+0000 7f7b27fff640 1 -- 172.21.15.29:0/1604803717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b301a2f50 msgr2=0x7f7b301a5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.117+0000 7f7b27fff640 1 --2- 172.21.15.29:0/1604803717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b301a2f50 0x7f7b301a5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.117+0000 7f7b27fff640 1 -- 172.21.15.29:0/1604803717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b30075b40 msgr2=0x7f7b3019d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.117+0000 7f7b27fff640 1 --2- 172.21.15.29:0/1604803717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b30075b40 0x7f7b3019d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.117+0000 7f7b27fff640 1 -- 172.21.15.29:0/1604803717 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b20009490 con 0x7f7b30077720 2024-03-11T19:18:02.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.117+0000 7f7b27fff640 1 --2- 172.21.15.29:0/1604803717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b30077720 0x7f7b3019def0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7b20000c00 tx=0x7f7b2002c7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.118+0000 7f7b2d7fa640 1 -- 172.21.15.29:0/1604803717 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2002dea0 con 0x7f7b30077720 2024-03-11T19:18:02.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.118+0000 7f7b35b48640 1 -- 172.21.15.29:0/1604803717 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b301a5e30 con 0x7f7b30077720 2024-03-11T19:18:02.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.118+0000 7f7b35b48640 1 -- 172.21.15.29:0/1604803717 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b301a6230 con 0x7f7b30077720 2024-03-11T19:18:02.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.118+0000 7f7b2d7fa640 1 -- 172.21.15.29:0/1604803717 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7b20004230 con 0x7f7b30077720 2024-03-11T19:18:02.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.118+0000 7f7b2d7fa640 1 -- 172.21.15.29:0/1604803717 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b200406b0 con 0x7f7b30077720 2024-03-11T19:18:02.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.121+0000 7f7b2d7fa640 1 -- 172.21.15.29:0/1604803717 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7b20040810 con 0x7f7b30077720 2024-03-11T19:18:02.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.122+0000 7f7b387fc740 1 Processor -- start 2024-03-11T19:18:02.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.122+0000 7f7b2d7fa640 1 --2- 172.21.15.29:0/1604803717 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7b1c0a1c00 0x7f7b1c0a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.122+0000 7f7b2d7fa640 1 -- 172.21.15.29:0/1604803717 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7b200ef4f0 con 0x7f7b30077720 2024-03-11T19:18:02.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.123+0000 7f7b387fc740 1 -- start start 2024-03-11T19:18:02.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.123+0000 7f7b387fc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3270 0x5624167a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.123+0000 7f7b387fc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a3c60 0x5624167a8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.123+0000 7f7b387fc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a8dd0 0x5624167ab260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.123+0000 7f7b2ffff640 1 --2- 172.21.15.29:0/1604803717 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7b1c0a1c00 0x7f7b1c0a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.123+0000 7f7b387fc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56241675f960 con 0x5624167a3270 2024-03-11T19:18:02.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.123+0000 7f7b387fc740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562416713310 con 0x5624167a8dd0 2024-03-11T19:18:02.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.123+0000 7f7b387fc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562416710840 con 0x5624167a3c60 2024-03-11T19:18:02.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.124+0000 7f7b2ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a8dd0 0x5624167ab260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.124+0000 7f7b2ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a8dd0 0x5624167ab260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56610/0 (socket says 172.21.15.29:56610) 2024-03-11T19:18:02.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.124+0000 7f7b2ffff640 1 -- 172.21.15.29:0/158218852 learned_addr learned my addr 172.21.15.29:0/158218852 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:02.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.126+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/158218852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3270 0x5624167a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.126+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/158218852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a3c60 msgr2=0x5624167a8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.126+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/158218852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a3c60 0x5624167a8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.126+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/158218852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a8dd0 msgr2=0x5624167ab260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.126+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/158218852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a8dd0 0x5624167ab260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.126+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/158218852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624166ce060 con 0x5624167a3270 2024-03-11T19:18:02.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.127+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/158218852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3270 0x5624167a3690 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7b18009c50 tx=0x7f7b180117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab1c989b1863ced server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.128+0000 7f7b25ffb640 1 -- 172.21.15.29:0/158218852 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b18012450 con 0x5624167a3270 2024-03-11T19:18:02.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.128+0000 7f7b25ffb640 1 -- 172.21.15.29:0/158218852 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7b18012a60 con 0x5624167a3270 2024-03-11T19:18:02.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.128+0000 7f7b25ffb640 1 -- 172.21.15.29:0/158218852 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1800faa0 con 0x5624167a3270 2024-03-11T19:18:02.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.131+0000 7f7b387fc740 1 -- 172.21.15.29:0/158218852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3270 msgr2=0x5624167a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.131+0000 7f7b387fc740 1 --2- 172.21.15.29:0/158218852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3270 0x5624167a3690 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7b18009c50 tx=0x7f7b180117b0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.134+0000 7f7b2ffff640 1 --2- 172.21.15.29:0/1604803717 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7b1c0a1c00 0x7f7b1c0a40c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7b28013830 tx=0x7f7b28013c30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.137+0000 7f7b387fc740 1 -- 172.21.15.29:0/158218852 shutdown_connections 2024-03-11T19:18:02.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.137+0000 7f7b387fc740 1 --2- 172.21.15.29:0/158218852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a8dd0 0x5624167ab260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.137+0000 7f7b387fc740 1 --2- 172.21.15.29:0/158218852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a3c60 0x5624167a8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.137+0000 7f7b387fc740 1 --2- 172.21.15.29:0/158218852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3270 0x5624167a3690 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.137+0000 7f7b387fc740 1 -- 172.21.15.29:0/158218852 >> 172.21.15.29:0/158218852 conn(0x56241679e7e0 msgr2=0x56241679ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.140+0000 7f7b387fc740 1 -- 172.21.15.29:0/158218852 shutdown_connections 2024-03-11T19:18:02.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.141+0000 7f7b387fc740 1 -- 172.21.15.29:0/158218852 wait complete. 2024-03-11T19:18:02.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.142+0000 7f7b387fc740 1 -- start start 2024-03-11T19:18:02.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.142+0000 7f7b387fc740 1 Processor -- start 2024-03-11T19:18:02.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.142+0000 7f7b387fc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3270 0x5624167be4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.142+0000 7f7b387fc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a3c60 0x5624167bea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.142+0000 7f7b387fc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a8dd0 0x5624167c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.142+0000 7f7b387fc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56241670c020 con 0x5624167a3270 2024-03-11T19:18:02.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.142+0000 7f7b387fc740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624163df430 con 0x5624167a3c60 2024-03-11T19:18:02.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.142+0000 7f7b387fc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56241639be00 con 0x5624167a8dd0 2024-03-11T19:18:02.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.142+0000 7f7b387fc740 10 client.0 _ll_register_callbacks cb 0x5624166b3cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:02.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.143+0000 7f7b2ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a8dd0 0x5624167c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.143+0000 7f7b2ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a8dd0 0x5624167c5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:44210/0 (socket says 172.21.15.29:44210) 2024-03-11T19:18:02.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.143+0000 7f7b2ffff640 1 -- 172.21.15.29:0/2196293121 learned_addr learned my addr 172.21.15.29:0/2196293121 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:02.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.143+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/2196293121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3270 0x5624167be4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.143+0000 7f7b2ffff640 1 -- 172.21.15.29:0/2196293121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a3c60 msgr2=0x5624167bea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.143+0000 7f7b2ffff640 1 --2- 172.21.15.29:0/2196293121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a3c60 0x5624167bea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.144+0000 7f7b2ffff640 1 -- 172.21.15.29:0/2196293121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3270 msgr2=0x5624167be4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.144+0000 7f7b2ffff640 1 --2- 172.21.15.29:0/2196293121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3270 0x5624167be4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.144+0000 7f7b2ffff640 1 -- 172.21.15.29:0/2196293121 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b180098b0 con 0x5624167a8dd0 2024-03-11T19:18:02.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.148+0000 7f7b2ffff640 1 --2- 172.21.15.29:0/2196293121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a8dd0 0x5624167c5a70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7b2800fdf0 tx=0x7f7b28004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb0d3fb5d9746d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.148+0000 7f7b0effd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:18:02.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.148+0000 7f7b0effd640 1 -- 172.21.15.29:0/2196293121 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b28020430 con 0x5624167a8dd0 2024-03-11T19:18:02.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.148+0000 7f7b0effd640 1 -- 172.21.15.29:0/2196293121 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7b28020590 con 0x5624167a8dd0 2024-03-11T19:18:02.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.148+0000 7f7b387fc740 1 -- 172.21.15.29:0/2196293121 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56241640b7e0 con 0x5624167a8dd0 2024-03-11T19:18:02.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.148+0000 7f7b0effd640 1 -- 172.21.15.29:0/2196293121 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b28020750 con 0x5624167a8dd0 2024-03-11T19:18:02.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.149+0000 7f7b387fc740 10 client.26878 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:02.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.149+0000 7f7b387fc740 1 -- 172.21.15.29:0/2196293121 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56241670a6a0 con 0x5624167a8dd0 2024-03-11T19:18:02.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.149+0000 7f7b0effd640 1 -- 172.21.15.29:0/2196293121 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x56241640b7e0 con 0x5624167a8dd0 2024-03-11T19:18:02.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.150+0000 7f7b0effd640 1 -- 172.21.15.29:0/2196293121 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7b28033900 con 0x5624167a8dd0 2024-03-11T19:18:02.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.150+0000 7f7b387fc740 10 client.26878 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:02.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.150+0000 7f7b387fc740 10 client.26878 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:18:02.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.150+0000 7f7b387fc740 20 client.26878 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:02.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.150+0000 7f7b387fc740 1 --2- 172.21.15.29:0/2196293121 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5624167ca8c0 0x5624167ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.150+0000 7f7b387fc740 4 client.26878 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:18:02.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.150+0000 7f7b387fc740 1 -- 172.21.15.29:0/2196293121 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562416448720 con 0x5624167ca8c0 2024-03-11T19:18:02.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.151+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/2196293121 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5624167ca8c0 0x5624167ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.154+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/2196293121 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5624167ca8c0 0x5624167ccce0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6a4a09c59c6898dc server_cookie=deb2e98a6d3cd675 in_seq=0 out_seq=0 2024-03-11T19:18:02.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.154+0000 7f7b0effd640 10 client.26878 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:18:02.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.155+0000 7f7b0effd640 1 -- 172.21.15.29:0/2196293121 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562416448720 con 0x5624167ca8c0 2024-03-11T19:18:02.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.155+0000 7f7b0effd640 10 client.26878 handle_command_reply: tid=0 2024-03-11T19:18:02.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.155+0000 7f7b387fc740 1 client.26878 shutdown 2024-03-11T19:18:02.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.156+0000 7f7b387fc740 1 -- 172.21.15.29:0/2196293121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a8dd0 msgr2=0x5624167c5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.156+0000 7f7b387fc740 1 --2- 172.21.15.29:0/2196293121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a8dd0 0x5624167c5a70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7b2800fdf0 tx=0x7f7b28004ab0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.156+0000 7f7b387fc740 1 -- 172.21.15.29:0/2196293121 shutdown_connections 2024-03-11T19:18:02.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.156+0000 7f7b387fc740 1 --2- 172.21.15.29:0/2196293121 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5624167ca8c0 0x5624167ccce0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.156+0000 7f7b387fc740 1 --2- 172.21.15.29:0/2196293121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a8dd0 0x5624167c5a70 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.156+0000 7f7b387fc740 1 --2- 172.21.15.29:0/2196293121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a3c60 0x5624167bea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.156+0000 7f7b387fc740 1 --2- 172.21.15.29:0/2196293121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3270 0x5624167be4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.156+0000 7f7b387fc740 1 -- 172.21.15.29:0/2196293121 >> 172.21.15.29:0/2196293121 conn(0x56241679e7e0 msgr2=0x56241679db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.156+0000 7f7b0effd640 0 client.26878 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:18:02.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.156+0000 7f7b387fc740 1 -- 172.21.15.29:0/2196293121 shutdown_connections 2024-03-11T19:18:02.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.156+0000 7f7b387fc740 1 -- 172.21.15.29:0/2196293121 wait complete. 2024-03-11T19:18:02.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.156+0000 7f7b387fc740 20 client.26878 trim_cache size 0 max 16384 2024-03-11T19:18:02.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.180+0000 7f7b387fc740 1 Processor -- start 2024-03-11T19:18:02.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.180+0000 7f7b387fc740 1 -- start start 2024-03-11T19:18:02.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.180+0000 7f7b387fc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a3270 0x5624167a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.180+0000 7f7b387fc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3c60 0x5624167a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.180+0000 7f7b387fc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a8dd0 0x5624167c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.180+0000 7f7b2f7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a3270 0x5624167a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.180+0000 7f7b27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3c60 0x5624167a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.180+0000 7f7b2f7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a3270 0x5624167a51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56618/0 (socket says 172.21.15.29:56618) 2024-03-11T19:18:02.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.180+0000 7f7b27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3c60 0x5624167a56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49498/0 (socket says 172.21.15.29:49498) 2024-03-11T19:18:02.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.180+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/1657614453 learned_addr learned my addr 172.21.15.29:0/1657614453 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:02.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.181+0000 7f7b387fc740 1 -- 172.21.15.29:0/1657614453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562416691130 con 0x5624167a3c60 2024-03-11T19:18:02.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.181+0000 7f7b387fc740 1 -- 172.21.15.29:0/1657614453 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624167a4b60 con 0x5624167a3270 2024-03-11T19:18:02.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.181+0000 7f7b387fc740 1 -- 172.21.15.29:0/1657614453 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624167c8d20 con 0x5624167a8dd0 2024-03-11T19:18:02.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.181+0000 7f7b2ffff640 1 --2- 172.21.15.29:0/1657614453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a8dd0 0x5624167c5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.181+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/1657614453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a8dd0 msgr2=0x5624167c5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.181+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/1657614453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a8dd0 0x5624167c5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.181+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/1657614453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3c60 msgr2=0x5624167a56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.181+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/1657614453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3c60 0x5624167a56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.181+0000 7f7b2f7fe640 1 -- 172.21.15.29:0/1657614453 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624166a6c20 con 0x5624167a3270 2024-03-11T19:18:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.182+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/1657614453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a3270 0x5624167a51b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7b18014010 tx=0x7f7b180057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8821d7842b132f56 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.182+0000 7f7b257fa640 1 -- 172.21.15.29:0/1657614453 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b18012650 con 0x5624167a3270 2024-03-11T19:18:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.182+0000 7f7b257fa640 1 -- 172.21.15.29:0/1657614453 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7b180127b0 con 0x5624167a3270 2024-03-11T19:18:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.182+0000 7f7b257fa640 1 -- 172.21.15.29:0/1657614453 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b18004b50 con 0x5624167a3270 2024-03-11T19:18:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.183+0000 7f7b387fc740 1 -- 172.21.15.29:0/1657614453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a3270 msgr2=0x5624167a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.183+0000 7f7b387fc740 1 --2- 172.21.15.29:0/1657614453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a3270 0x5624167a51b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7b18014010 tx=0x7f7b180057f0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.183+0000 7f7b387fc740 1 -- 172.21.15.29:0/1657614453 shutdown_connections 2024-03-11T19:18:02.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.183+0000 7f7b387fc740 1 --2- 172.21.15.29:0/1657614453 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a8dd0 0x5624167c5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.183+0000 7f7b387fc740 1 --2- 172.21.15.29:0/1657614453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3c60 0x5624167a56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.183+0000 7f7b387fc740 1 --2- 172.21.15.29:0/1657614453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a3270 0x5624167a51b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.183+0000 7f7b387fc740 1 -- 172.21.15.29:0/1657614453 >> 172.21.15.29:0/1657614453 conn(0x56241679e7e0 msgr2=0x5624167d1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.184+0000 7f7b387fc740 1 -- 172.21.15.29:0/1657614453 shutdown_connections 2024-03-11T19:18:02.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.184+0000 7f7b387fc740 1 -- 172.21.15.29:0/1657614453 wait complete. 2024-03-11T19:18:02.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.184+0000 7f7b387fc740 1 -- start start 2024-03-11T19:18:02.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.185+0000 7f7b387fc740 1 Processor -- start 2024-03-11T19:18:02.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.185+0000 7f7b387fc740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a3270 0x5624167c58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.185+0000 7f7b387fc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3c60 0x5624167c3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.185+0000 7f7b387fc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a8dd0 0x5624167c4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.185+0000 7f7b387fc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624167c4a30 con 0x5624167a3c60 2024-03-11T19:18:02.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.185+0000 7f7b387fc740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624167c4ba0 con 0x5624167a3270 2024-03-11T19:18:02.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.185+0000 7f7b387fc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624167c4d60 con 0x5624167a8dd0 2024-03-11T19:18:02.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.185+0000 7f7b2ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a8dd0 0x5624167c4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.185+0000 7f7b2ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a8dd0 0x5624167c4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:44236/0 (socket says 172.21.15.29:44236) 2024-03-11T19:18:02.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.185+0000 7f7b2ffff640 1 -- 172.21.15.29:0/3269651698 learned_addr learned my addr 172.21.15.29:0/3269651698 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:02.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.185+0000 7f7b27fff640 1 --2- 172.21.15.29:0/3269651698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3c60 0x5624167c3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.185+0000 7f7b387fc740 10 client.0 _ll_register_callbacks cb 0x7f7b180096a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:02.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.186+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/3269651698 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a3270 0x5624167c58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.186+0000 7f7b27fff640 1 -- 172.21.15.29:0/3269651698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a8dd0 msgr2=0x5624167c4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.186+0000 7f7b27fff640 1 --2- 172.21.15.29:0/3269651698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a8dd0 0x5624167c4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.186+0000 7f7b27fff640 1 -- 172.21.15.29:0/3269651698 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a3270 msgr2=0x5624167c58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.186+0000 7f7b27fff640 1 --2- 172.21.15.29:0/3269651698 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a3270 0x5624167c58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.186+0000 7f7b27fff640 1 -- 172.21.15.29:0/3269651698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5624166ce060 con 0x5624167a3c60 2024-03-11T19:18:02.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.187+0000 7f7b27fff640 1 --2- 172.21.15.29:0/3269651698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3c60 0x5624167c3f30 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x5624167a46d0 tx=0x7f7b200bca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8fa8f692fd44286 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.188+0000 7f7b0effd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:18:02.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.188+0000 7f7b0effd640 1 -- 172.21.15.29:0/3269651698 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b200b8040 con 0x5624167a3c60 2024-03-11T19:18:02.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.188+0000 7f7b0effd640 1 -- 172.21.15.29:0/3269651698 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7b200bc060 con 0x5624167a3c60 2024-03-11T19:18:02.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.188+0000 7f7b387fc740 1 -- 172.21.15.29:0/3269651698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b18012a60 con 0x5624167a3c60 2024-03-11T19:18:02.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.189+0000 7f7b0effd640 1 -- 172.21.15.29:0/3269651698 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b200ef320 con 0x5624167a3c60 2024-03-11T19:18:02.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.189+0000 7f7b387fc740 10 client.17352 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:02.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.189+0000 7f7b387fc740 1 -- 172.21.15.29:0/3269651698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5624166cc8c0 con 0x5624167a3c60 2024-03-11T19:18:02.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.190+0000 7f7b0effd640 1 -- 172.21.15.29:0/3269651698 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7b200d4070 con 0x5624167a3c60 2024-03-11T19:18:02.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.190+0000 7f7b0effd640 1 -- 172.21.15.29:0/3269651698 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7b200dabd0 con 0x5624167a3c60 2024-03-11T19:18:02.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.190+0000 7f7b387fc740 10 client.17352 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:02.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.190+0000 7f7b387fc740 10 client.17352 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:18:02.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.190+0000 7f7b387fc740 20 client.17352 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:02.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.190+0000 7f7b387fc740 1 --2- 172.21.15.29:0/3269651698 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x562416822280 0x5624168246a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.190+0000 7f7b387fc740 4 client.17352 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:18:02.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.190+0000 7f7b387fc740 1 -- 172.21.15.29:0/3269651698 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562416497890 con 0x562416822280 2024-03-11T19:18:02.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.193+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/3269651698 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x562416822280 0x5624168246a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.195+0000 7f7b2f7fe640 1 --2- 172.21.15.29:0/3269651698 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x562416822280 0x5624168246a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2a37f5b756ea5a80 server_cookie=286ea4534e12a5c3 in_seq=0 out_seq=0 2024-03-11T19:18:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.195+0000 7f7b0effd640 10 client.17352 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:18:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.195+0000 7f7b0effd640 1 -- 172.21.15.29:0/3269651698 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562416497890 con 0x562416822280 2024-03-11T19:18:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.195+0000 7f7b0effd640 10 client.17352 handle_command_reply: tid=0 2024-03-11T19:18:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.196+0000 7f7b387fc740 1 client.17352 shutdown 2024-03-11T19:18:02.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.196+0000 7f7b387fc740 1 -- 172.21.15.29:0/3269651698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3c60 msgr2=0x5624167c3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.196+0000 7f7b387fc740 1 --2- 172.21.15.29:0/3269651698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3c60 0x5624167c3f30 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x5624167a46d0 tx=0x7f7b200bca10 comp rx=0 tx=0).stop 2024-03-11T19:18:02.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.196+0000 7f7b387fc740 1 -- 172.21.15.29:0/3269651698 shutdown_connections 2024-03-11T19:18:02.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.196+0000 7f7b387fc740 1 --2- 172.21.15.29:0/3269651698 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x562416822280 0x5624168246a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.196+0000 7f7b387fc740 1 --2- 172.21.15.29:0/3269651698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5624167a8dd0 0x5624167c4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.196+0000 7f7b387fc740 1 --2- 172.21.15.29:0/3269651698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5624167a3c60 0x5624167c3f30 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.197+0000 7f7b387fc740 1 --2- 172.21.15.29:0/3269651698 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624167a3270 0x5624167c58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.197+0000 7f7b0effd640 0 client.17352 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:18:02.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.197+0000 7f7b387fc740 1 -- 172.21.15.29:0/3269651698 >> 172.21.15.29:0/3269651698 conn(0x56241679e7e0 msgr2=0x5624167a51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.197+0000 7f7b387fc740 1 -- 172.21.15.29:0/3269651698 shutdown_connections 2024-03-11T19:18:02.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.197+0000 7f7b387fc740 1 -- 172.21.15.29:0/3269651698 wait complete. 2024-03-11T19:18:02.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.197+0000 7f7b387fc740 20 client.17352 trim_cache size 0 max 16384 2024-03-11T19:18:02.203 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:18:02.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.199+0000 7f7b35b48640 1 -- 172.21.15.29:0/1604803717 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7b1c0a1c00 msgr2=0x7f7b1c0a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.199+0000 7f7b35b48640 1 --2- 172.21.15.29:0/1604803717 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7b1c0a1c00 0x7f7b1c0a40c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7b28013830 tx=0x7f7b28013c30 comp rx=0 tx=0).stop 2024-03-11T19:18:02.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.199+0000 7f7b35b48640 1 -- 172.21.15.29:0/1604803717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b30077720 msgr2=0x7f7b3019def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.199+0000 7f7b35b48640 1 --2- 172.21.15.29:0/1604803717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b30077720 0x7f7b3019def0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7b20000c00 tx=0x7f7b2002c7c0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.199+0000 7f7b35b48640 1 -- 172.21.15.29:0/1604803717 shutdown_connections 2024-03-11T19:18:02.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.199+0000 7f7b35b48640 1 --2- 172.21.15.29:0/1604803717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b301a2f50 0x7f7b301a5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.199+0000 7f7b35b48640 1 --2- 172.21.15.29:0/1604803717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b30077720 0x7f7b3019def0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.199+0000 7f7b35b48640 1 --2- 172.21.15.29:0/1604803717 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7b1c0a1c00 0x7f7b1c0a40c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.199+0000 7f7b35b48640 1 --2- 172.21.15.29:0/1604803717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b30075b40 0x7f7b3019d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.199+0000 7f7b35b48640 1 -- 172.21.15.29:0/1604803717 >> 172.21.15.29:0/1604803717 conn(0x7f7b300fe930 msgr2=0x7f7b30100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.199+0000 7f7b35b48640 1 -- 172.21.15.29:0/1604803717 shutdown_connections 2024-03-11T19:18:02.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.199+0000 7f7b35b48640 1 -- 172.21.15.29:0/1604803717 wait complete. 2024-03-11T19:18:02.223 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:18:02.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:18:02.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.415+0000 7f40127fc640 1 -- 172.21.15.29:0/1831802832 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4004034440 con 0x7f40140ff970 2024-03-11T19:18:02.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.416+0000 7f401a1b6640 1 -- 172.21.15.29:0/1831802832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40140ff970 msgr2=0x7f40140ffd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.416+0000 7f401a1b6640 1 --2- 172.21.15.29:0/1831802832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40140ff970 0x7f40140ffd90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f400400aa10 tx=0x7f400402be60 comp rx=0 tx=0).stop 2024-03-11T19:18:02.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.417+0000 7f401a1b6640 1 -- 172.21.15.29:0/1831802832 shutdown_connections 2024-03-11T19:18:02.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.417+0000 7f401a1b6640 1 --2- 172.21.15.29:0/1831802832 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40141013c0 0x7f4014107c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.417+0000 7f401a1b6640 1 --2- 172.21.15.29:0/1831802832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4014100a20 0x7f4014100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.417+0000 7f401a1b6640 1 --2- 172.21.15.29:0/1831802832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40140ff970 0x7f40140ffd90 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.417+0000 7f401a1b6640 1 -- 172.21.15.29:0/1831802832 >> 172.21.15.29:0/1831802832 conn(0x7f40140fa940 msgr2=0x7f40140fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.419+0000 7f401a1b6640 1 -- 172.21.15.29:0/1831802832 shutdown_connections 2024-03-11T19:18:02.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.419+0000 7f401a1b6640 1 -- 172.21.15.29:0/1831802832 wait complete. 2024-03-11T19:18:02.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.419+0000 7f401a1b6640 1 Processor -- start 2024-03-11T19:18:02.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.420+0000 7f401a1b6640 1 -- start start 2024-03-11T19:18:02.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.420+0000 7f401a1b6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40140ff970 0x7f4014198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.420+0000 7f401a1b6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4014100a20 0x7f4014198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.420+0000 7f401a1b6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40141013c0 0x7f4014195070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.420+0000 7f401a1b6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4014195800 con 0x7f40140ff970 2024-03-11T19:18:02.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.420+0000 7f401a1b6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4014195970 con 0x7f4014100a20 2024-03-11T19:18:02.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.420+0000 7f401a1b6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4014195b30 con 0x7f40141013c0 2024-03-11T19:18:02.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.420+0000 7f40137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40140ff970 0x7f4014198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.420+0000 7f40137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40140ff970 0x7f4014198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49534/0 (socket says 172.21.15.29:49534) 2024-03-11T19:18:02.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.420+0000 7f40137fe640 1 -- 172.21.15.29:0/3874744165 learned_addr learned my addr 172.21.15.29:0/3874744165 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:02.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.420+0000 7f4012ffd640 1 --2- 172.21.15.29:0/3874744165 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4014100a20 0x7f4014198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.420+0000 7f4013fff640 1 --2- 172.21.15.29:0/3874744165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40141013c0 0x7f4014195070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.421+0000 7f40137fe640 1 -- 172.21.15.29:0/3874744165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40141013c0 msgr2=0x7f4014195070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.421+0000 7f40137fe640 1 --2- 172.21.15.29:0/3874744165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40141013c0 0x7f4014195070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.421+0000 7f40137fe640 1 -- 172.21.15.29:0/3874744165 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4014100a20 msgr2=0x7f4014198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.421+0000 7f40137fe640 1 --2- 172.21.15.29:0/3874744165 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4014100a20 0x7f4014198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.421+0000 7f40137fe640 1 -- 172.21.15.29:0/3874744165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400400a670 con 0x7f40140ff970 2024-03-11T19:18:02.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.421+0000 7f4013fff640 1 --2- 172.21.15.29:0/3874744165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40141013c0 0x7f4014195070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:18:02.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.421+0000 7f40137fe640 1 --2- 172.21.15.29:0/3874744165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40140ff970 0x7f4014198500 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f400402c470 tx=0x7f4004004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.424+0000 7f4010ff9640 1 -- 172.21.15.29:0/3874744165 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4004034910 con 0x7f40140ff970 2024-03-11T19:18:02.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.424+0000 7f401a1b6640 1 -- 172.21.15.29:0/3874744165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4014199060 con 0x7f40140ff970 2024-03-11T19:18:02.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.424+0000 7f401a1b6640 1 -- 172.21.15.29:0/3874744165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40141995c0 con 0x7f40140ff970 2024-03-11T19:18:02.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.424+0000 7f4010ff9640 1 -- 172.21.15.29:0/3874744165 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f40040035f0 con 0x7f40140ff970 2024-03-11T19:18:02.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.424+0000 7f4010ff9640 1 -- 172.21.15.29:0/3874744165 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4004004eb0 con 0x7f40140ff970 2024-03-11T19:18:02.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.428+0000 7f401cd84740 1 Processor -- start 2024-03-11T19:18:02.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.429+0000 7f4010ff9640 1 -- 172.21.15.29:0/3874744165 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4004005010 con 0x7f40140ff970 2024-03-11T19:18:02.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.429+0000 7f4010ff9640 1 --2- 172.21.15.29:0/3874744165 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3ffc0a1ab0 0x7f3ffc0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.430+0000 7f4010ff9640 1 -- 172.21.15.29:0/3874744165 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f40040f0140 con 0x7f40140ff970 2024-03-11T19:18:02.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.430+0000 7f401cd84740 1 -- start start 2024-03-11T19:18:02.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.430+0000 7f4013fff640 1 --2- 172.21.15.29:0/3874744165 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3ffc0a1ab0 0x7f3ffc0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.430+0000 7f401cd84740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9a5270 0x55567a9a5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.430+0000 7f401cd84740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9a5c60 0x55567a9aa760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.430+0000 7f401cd84740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9aadd0 0x55567a9ad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.430+0000 7f401cd84740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55567a961960 con 0x55567a9a5270 2024-03-11T19:18:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.430+0000 7f401cd84740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55567a915310 con 0x55567a9a5c60 2024-03-11T19:18:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.430+0000 7f401cd84740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55567a912840 con 0x55567a9aadd0 2024-03-11T19:18:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.431+0000 7f40137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9a5270 0x55567a9a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.431+0000 7f4013fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9aadd0 0x55567a9ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.431+0000 7f4013fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9aadd0 0x55567a9ad260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:44266/0 (socket says 172.21.15.29:44266) 2024-03-11T19:18:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.431+0000 7f4013fff640 1 -- 172.21.15.29:0/494235393 learned_addr learned my addr 172.21.15.29:0/494235393 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:02.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.431+0000 7f4013fff640 1 --2- 172.21.15.29:0/3874744165 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3ffc0a1ab0 0x7f3ffc0a3f70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f400c005fd0 tx=0x7f400c00f730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.431+0000 7f4012ffd640 1 --2- 172.21.15.29:0/494235393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9a5c60 0x55567a9aa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.431+0000 7f4013fff640 1 -- 172.21.15.29:0/494235393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9a5c60 msgr2=0x55567a9aa760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.431+0000 7f4013fff640 1 --2- 172.21.15.29:0/494235393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9a5c60 0x55567a9aa760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.431+0000 7f4013fff640 1 -- 172.21.15.29:0/494235393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9a5270 msgr2=0x55567a9a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.431+0000 7f4013fff640 1 --2- 172.21.15.29:0/494235393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9a5270 0x55567a9a5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.431+0000 7f4013fff640 1 -- 172.21.15.29:0/494235393 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55567a8d0060 con 0x55567a9aadd0 2024-03-11T19:18:02.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.431+0000 7f4012ffd640 1 --2- 172.21.15.29:0/494235393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9a5c60 0x55567a9aa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:18:02.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.432+0000 7f4013fff640 1 --2- 172.21.15.29:0/494235393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9aadd0 0x55567a9ad260 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f400c0126f0 tx=0x7f400c01a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a2b2a63272d94f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.432+0000 7f3ff9ffb640 1 -- 172.21.15.29:0/494235393 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c00d790 con 0x55567a9aadd0 2024-03-11T19:18:02.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.432+0000 7f3ff9ffb640 1 -- 172.21.15.29:0/494235393 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f400c00dda0 con 0x55567a9aadd0 2024-03-11T19:18:02.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.432+0000 7f3ff9ffb640 1 -- 172.21.15.29:0/494235393 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c01de90 con 0x55567a9aadd0 2024-03-11T19:18:02.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.433+0000 7f401cd84740 1 -- 172.21.15.29:0/494235393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9aadd0 msgr2=0x55567a9ad260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.433+0000 7f401cd84740 1 --2- 172.21.15.29:0/494235393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9aadd0 0x55567a9ad260 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f400c0126f0 tx=0x7f400c01a000 comp rx=0 tx=0).stop 2024-03-11T19:18:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.435+0000 7f401cd84740 1 -- 172.21.15.29:0/494235393 shutdown_connections 2024-03-11T19:18:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.435+0000 7f401cd84740 1 --2- 172.21.15.29:0/494235393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9aadd0 0x55567a9ad260 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.435+0000 7f401cd84740 1 --2- 172.21.15.29:0/494235393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9a5c60 0x55567a9aa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.435+0000 7f401cd84740 1 --2- 172.21.15.29:0/494235393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9a5270 0x55567a9a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.435+0000 7f401cd84740 1 -- 172.21.15.29:0/494235393 >> 172.21.15.29:0/494235393 conn(0x55567a9a07e0 msgr2=0x55567a9a0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.435+0000 7f401cd84740 1 -- 172.21.15.29:0/494235393 shutdown_connections 2024-03-11T19:18:02.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.435+0000 7f401cd84740 1 -- 172.21.15.29:0/494235393 wait complete. 2024-03-11T19:18:02.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.435+0000 7f401cd84740 1 -- start start 2024-03-11T19:18:02.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.436+0000 7f401cd84740 1 Processor -- start 2024-03-11T19:18:02.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.436+0000 7f401cd84740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9a5270 0x55567a9af320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.436+0000 7f401cd84740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9a5c60 0x55567a9af860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.436+0000 7f401cd84740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9aadd0 0x55567a9b6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.436+0000 7f401cd84740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55567a90e020 con 0x55567a9aadd0 2024-03-11T19:18:02.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.436+0000 7f401cd84740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55567a5e1430 con 0x55567a9a5c60 2024-03-11T19:18:02.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.436+0000 7f401cd84740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55567a59de00 con 0x55567a9a5270 2024-03-11T19:18:02.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.436+0000 7f401cd84740 10 client.0 _ll_register_callbacks cb 0x55567a8b5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:02.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.437+0000 7f4013fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9aadd0 0x55567a9b6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.437+0000 7f4013fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9aadd0 0x55567a9b6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49540/0 (socket says 172.21.15.29:49540) 2024-03-11T19:18:02.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.437+0000 7f4013fff640 1 -- 172.21.15.29:0/1170560441 learned_addr learned my addr 172.21.15.29:0/1170560441 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:02.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.437+0000 7f4013fff640 1 -- 172.21.15.29:0/1170560441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9a5270 msgr2=0x55567a9af320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.437+0000 7f4013fff640 1 --2- 172.21.15.29:0/1170560441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9a5270 0x55567a9af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.437+0000 7f4013fff640 1 -- 172.21.15.29:0/1170560441 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9a5c60 msgr2=0x55567a9af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.437+0000 7f4013fff640 1 --2- 172.21.15.29:0/1170560441 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9a5c60 0x55567a9af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.437+0000 7f4013fff640 1 -- 172.21.15.29:0/1170560441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f400c0123f0 con 0x55567a9aadd0 2024-03-11T19:18:02.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.438+0000 7f4013fff640 1 --2- 172.21.15.29:0/1170560441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9aadd0 0x55567a9b6830 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f400c01ddf0 tx=0x7f400c009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d800894036671c33 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.438+0000 7f3ff2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:18:02.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.438+0000 7f3ff2ffd640 1 -- 172.21.15.29:0/1170560441 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c01d850 con 0x55567a9aadd0 2024-03-11T19:18:02.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.438+0000 7f401cd84740 1 -- 172.21.15.29:0/1170560441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55567a60d7e0 con 0x55567a9aadd0 2024-03-11T19:18:02.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.438+0000 7f3ff2ffd640 1 -- 172.21.15.29:0/1170560441 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f400c018050 con 0x55567a9aadd0 2024-03-11T19:18:02.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.438+0000 7f3ff2ffd640 1 -- 172.21.15.29:0/1170560441 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f400c004c50 con 0x55567a9aadd0 2024-03-11T19:18:02.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.438+0000 7f401cd84740 10 client.17364 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:02.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.438+0000 7f401cd84740 1 -- 172.21.15.29:0/1170560441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55567a90c6a0 con 0x55567a9aadd0 2024-03-11T19:18:02.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.439+0000 7f3ff2ffd640 1 -- 172.21.15.29:0/1170560441 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f400c004e30 con 0x55567a9aadd0 2024-03-11T19:18:02.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.439+0000 7f3ff2ffd640 1 -- 172.21.15.29:0/1170560441 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f400c023070 con 0x55567a9aadd0 2024-03-11T19:18:02.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.440+0000 7f401cd84740 10 client.17364 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:02.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.440+0000 7f401cd84740 10 client.17364 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:18:02.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.440+0000 7f401cd84740 20 client.17364 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:02.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.440+0000 7f401cd84740 1 --2- 172.21.15.29:0/1170560441 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55567a9bb370 0x55567a9bd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.440+0000 7f401cd84740 4 client.17364 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:18:02.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.440+0000 7f401cd84740 1 -- 172.21.15.29:0/1170560441 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55567a60d7e0 con 0x55567a9bb370 2024-03-11T19:18:02.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.440+0000 7f40137fe640 1 --2- 172.21.15.29:0/1170560441 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55567a9bb370 0x55567a9bd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.441+0000 7f40137fe640 1 --2- 172.21.15.29:0/1170560441 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55567a9bb370 0x55567a9bd830 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3f6378f5b9a79d6e server_cookie=1d4f7c69a5a4945c in_seq=0 out_seq=0 2024-03-11T19:18:02.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.441+0000 7f3ff2ffd640 10 client.17364 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:18:02.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.442+0000 7f3ff2ffd640 1 -- 172.21.15.29:0/1170560441 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55567a60d7e0 con 0x55567a9bb370 2024-03-11T19:18:02.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.442+0000 7f3ff2ffd640 10 client.17364 handle_command_reply: tid=0 2024-03-11T19:18:02.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.442+0000 7f401cd84740 1 client.17364 shutdown 2024-03-11T19:18:02.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.442+0000 7f401cd84740 1 -- 172.21.15.29:0/1170560441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9aadd0 msgr2=0x55567a9b6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.442+0000 7f401cd84740 1 --2- 172.21.15.29:0/1170560441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9aadd0 0x55567a9b6830 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f400c01ddf0 tx=0x7f400c009500 comp rx=0 tx=0).stop 2024-03-11T19:18:02.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.443+0000 7f401cd84740 1 -- 172.21.15.29:0/1170560441 shutdown_connections 2024-03-11T19:18:02.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.443+0000 7f401cd84740 1 --2- 172.21.15.29:0/1170560441 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55567a9bb370 0x55567a9bd830 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.443+0000 7f401cd84740 1 --2- 172.21.15.29:0/1170560441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9aadd0 0x55567a9b6830 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.443+0000 7f401cd84740 1 --2- 172.21.15.29:0/1170560441 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9a5c60 0x55567a9af860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.443+0000 7f401cd84740 1 --2- 172.21.15.29:0/1170560441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9a5270 0x55567a9af320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.443+0000 7f401cd84740 1 -- 172.21.15.29:0/1170560441 >> 172.21.15.29:0/1170560441 conn(0x55567a9a07e0 msgr2=0x55567a99fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.443+0000 7f3ff2ffd640 0 client.17364 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:18:02.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.443+0000 7f401cd84740 1 -- 172.21.15.29:0/1170560441 shutdown_connections 2024-03-11T19:18:02.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.443+0000 7f401cd84740 1 -- 172.21.15.29:0/1170560441 wait complete. 2024-03-11T19:18:02.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.443+0000 7f401cd84740 20 client.17364 trim_cache size 0 max 16384 2024-03-11T19:18:02.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.465+0000 7f401cd84740 1 Processor -- start 2024-03-11T19:18:02.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.465+0000 7f401cd84740 1 -- start start 2024-03-11T19:18:02.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.465+0000 7f401cd84740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9a5270 0x55567a9a6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.465+0000 7f401cd84740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9a5c60 0x55567a9a7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.465+0000 7f401cd84740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9aadd0 0x55567a9b67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.466+0000 7f4013fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9aadd0 0x55567a9b67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.466+0000 7f401cd84740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55567a9aabb0 con 0x55567a9aadd0 2024-03-11T19:18:02.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.466+0000 7f401cd84740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55567a893130 con 0x55567a9a5c60 2024-03-11T19:18:02.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.466+0000 7f401cd84740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55567a9a12d0 con 0x55567a9a5270 2024-03-11T19:18:02.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.466+0000 7f4012ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9a5c60 0x55567a9a7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.466+0000 7f4012ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9a5c60 0x55567a9a7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56676/0 (socket says 172.21.15.29:56676) 2024-03-11T19:18:02.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.466+0000 7f4012ffd640 1 -- 172.21.15.29:0/1415953090 learned_addr learned my addr 172.21.15.29:0/1415953090 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:02.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.467+0000 7f4013fff640 1 -- 172.21.15.29:0/1415953090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9a5270 msgr2=0x55567a9a6fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:18:02.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.467+0000 7f4013fff640 1 --2- 172.21.15.29:0/1415953090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9a5270 0x55567a9a6fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.467+0000 7f4013fff640 1 -- 172.21.15.29:0/1415953090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9a5c60 msgr2=0x55567a9a7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.467+0000 7f4013fff640 1 --2- 172.21.15.29:0/1415953090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9a5c60 0x55567a9a7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.467+0000 7f4013fff640 1 -- 172.21.15.29:0/1415953090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55567a8a8c20 con 0x55567a9aadd0 2024-03-11T19:18:02.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.467+0000 7f4013fff640 1 --2- 172.21.15.29:0/1415953090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9aadd0 0x55567a9b67f0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f400c01d710 tx=0x7f400c02e4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=363d1c8af3461c1d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.467+0000 7f3ff97fa640 1 -- 172.21.15.29:0/1415953090 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c01da20 con 0x55567a9aadd0 2024-03-11T19:18:02.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.467+0000 7f3ff97fa640 1 -- 172.21.15.29:0/1415953090 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f400c00da40 con 0x55567a9aadd0 2024-03-11T19:18:02.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.467+0000 7f3ff97fa640 1 -- 172.21.15.29:0/1415953090 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c02ece0 con 0x55567a9aadd0 2024-03-11T19:18:02.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.469+0000 7f401cd84740 1 -- 172.21.15.29:0/1415953090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9aadd0 msgr2=0x55567a9b67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.469+0000 7f401cd84740 1 --2- 172.21.15.29:0/1415953090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9aadd0 0x55567a9b67f0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f400c01d710 tx=0x7f400c02e4e0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.469+0000 7f401cd84740 1 -- 172.21.15.29:0/1415953090 shutdown_connections 2024-03-11T19:18:02.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.469+0000 7f401cd84740 1 --2- 172.21.15.29:0/1415953090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9aadd0 0x55567a9b67f0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.469+0000 7f401cd84740 1 --2- 172.21.15.29:0/1415953090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9a5c60 0x55567a9a7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.469+0000 7f401cd84740 1 --2- 172.21.15.29:0/1415953090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9a5270 0x55567a9a6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.469+0000 7f401cd84740 1 -- 172.21.15.29:0/1415953090 >> 172.21.15.29:0/1415953090 conn(0x55567a9a07e0 msgr2=0x55567a9a0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.472+0000 7f401cd84740 1 -- 172.21.15.29:0/1415953090 shutdown_connections 2024-03-11T19:18:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.473+0000 7f401cd84740 1 -- 172.21.15.29:0/1415953090 wait complete. 2024-03-11T19:18:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.474+0000 7f401cd84740 1 -- start start 2024-03-11T19:18:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.474+0000 7f401cd84740 1 Processor -- start 2024-03-11T19:18:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.474+0000 7f401cd84740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9a5270 0x55567a9b51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.474+0000 7f401cd84740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9a5c60 0x55567a9b5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.474+0000 7f401cd84740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9aadd0 0x55567a9b5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.474+0000 7f401cd84740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55567aa11900 con 0x55567a9a5c60 2024-03-11T19:18:02.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.474+0000 7f401cd84740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55567aa11a40 con 0x55567a9aadd0 2024-03-11T19:18:02.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.474+0000 7f401cd84740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55567aa11bd0 con 0x55567a9a5270 2024-03-11T19:18:02.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.474+0000 7f401cd84740 10 client.0 _ll_register_callbacks cb 0x7f400c00fed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:02.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.475+0000 7f4013fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9aadd0 0x55567a9b5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.475+0000 7f4013fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9aadd0 0x55567a9b5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56682/0 (socket says 172.21.15.29:56682) 2024-03-11T19:18:02.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.475+0000 7f4013fff640 1 -- 172.21.15.29:0/582595651 learned_addr learned my addr 172.21.15.29:0/582595651 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:02.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.475+0000 7f4012ffd640 1 --2- 172.21.15.29:0/582595651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9a5c60 0x55567a9b5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.476+0000 7f40137fe640 1 --2- 172.21.15.29:0/582595651 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9a5270 0x55567a9b51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.477+0000 7f4013fff640 1 -- 172.21.15.29:0/582595651 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9a5270 msgr2=0x55567a9b51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.477+0000 7f4013fff640 1 --2- 172.21.15.29:0/582595651 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9a5270 0x55567a9b51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.477+0000 7f4013fff640 1 -- 172.21.15.29:0/582595651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9a5c60 msgr2=0x55567a9b5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.477+0000 7f4013fff640 1 --2- 172.21.15.29:0/582595651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9a5c60 0x55567a9b5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.477+0000 7f4013fff640 1 -- 172.21.15.29:0/582595651 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55567a90c6a0 con 0x55567a9aadd0 2024-03-11T19:18:02.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.477+0000 7f4013fff640 1 --2- 172.21.15.29:0/582595651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9aadd0 0x55567a9b5c60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f400c0058c0 tx=0x7f400c009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cb15ba62d2b2120 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.477+0000 7f3ff2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:18:02.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.478+0000 7f3ff2ffd640 1 -- 172.21.15.29:0/582595651 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c00dbb0 con 0x55567a9aadd0 2024-03-11T19:18:02.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.478+0000 7f3ff2ffd640 1 -- 172.21.15.29:0/582595651 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f400c018070 con 0x55567a9aadd0 2024-03-11T19:18:02.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.478+0000 7f401cd84740 1 -- 172.21.15.29:0/582595651 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f400c00dda0 con 0x55567a9aadd0 2024-03-11T19:18:02.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.478+0000 7f3ff2ffd640 1 -- 172.21.15.29:0/582595651 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f400c00d7a0 con 0x55567a9aadd0 2024-03-11T19:18:02.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.479+0000 7f401cd84740 10 client.26840 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:02.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.479+0000 7f401cd84740 1 -- 172.21.15.29:0/582595651 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55567a8ce8c0 con 0x55567a9aadd0 2024-03-11T19:18:02.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.483+0000 7f3ff2ffd640 1 -- 172.21.15.29:0/582595651 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f400c00dda0 con 0x55567a9aadd0 2024-03-11T19:18:02.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.483+0000 7f3ff2ffd640 1 -- 172.21.15.29:0/582595651 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f400c04a020 con 0x55567a9aadd0 2024-03-11T19:18:02.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.484+0000 7f401cd84740 10 client.26840 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:02.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.484+0000 7f401cd84740 10 client.26840 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:18:02.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.484+0000 7f401cd84740 20 client.26840 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:02.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.484+0000 7f401cd84740 1 --2- 172.21.15.29:0/582595651 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55567aa132d0 0x55567aa156f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.484+0000 7f401cd84740 4 client.26840 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:18:02.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.484+0000 7f401cd84740 1 -- 172.21.15.29:0/582595651 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55567a64a720 con 0x55567aa132d0 2024-03-11T19:18:02.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.485+0000 7f40137fe640 1 --2- 172.21.15.29:0/582595651 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55567aa132d0 0x55567aa156f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.485+0000 7f40137fe640 1 --2- 172.21.15.29:0/582595651 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55567aa132d0 0x55567aa156f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e263fe691c1affb5 server_cookie=363f30cd843dfed8 in_seq=0 out_seq=0 2024-03-11T19:18:02.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.485+0000 7f3ff2ffd640 10 client.26840 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:18:02.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.486+0000 7f3ff2ffd640 1 -- 172.21.15.29:0/582595651 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55567a64a720 con 0x55567aa132d0 2024-03-11T19:18:02.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.486+0000 7f3ff2ffd640 10 client.26840 handle_command_reply: tid=0 2024-03-11T19:18:02.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.486+0000 7f401cd84740 1 client.26840 shutdown 2024-03-11T19:18:02.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.486+0000 7f401cd84740 1 -- 172.21.15.29:0/582595651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9aadd0 msgr2=0x55567a9b5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.486+0000 7f401cd84740 1 --2- 172.21.15.29:0/582595651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9aadd0 0x55567a9b5c60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f400c0058c0 tx=0x7f400c009500 comp rx=0 tx=0).stop 2024-03-11T19:18:02.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.486+0000 7f401cd84740 1 -- 172.21.15.29:0/582595651 shutdown_connections 2024-03-11T19:18:02.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.486+0000 7f401cd84740 1 --2- 172.21.15.29:0/582595651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55567a9aadd0 0x55567a9b5c60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.486+0000 7f401cd84740 1 --2- 172.21.15.29:0/582595651 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55567aa132d0 0x55567aa156f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.486+0000 7f401cd84740 1 --2- 172.21.15.29:0/582595651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55567a9a5c60 0x55567a9b5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.486+0000 7f401cd84740 1 --2- 172.21.15.29:0/582595651 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55567a9a5270 0x55567a9b51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.486+0000 7f3ff2ffd640 0 client.26840 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:18:02.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.486+0000 7f401cd84740 1 -- 172.21.15.29:0/582595651 >> 172.21.15.29:0/582595651 conn(0x55567a9a07e0 msgr2=0x55567a9a3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.487+0000 7f401cd84740 1 -- 172.21.15.29:0/582595651 shutdown_connections 2024-03-11T19:18:02.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.487+0000 7f401cd84740 1 -- 172.21.15.29:0/582595651 wait complete. 2024-03-11T19:18:02.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.487+0000 7f401cd84740 20 client.26840 trim_cache size 0 max 16384 2024-03-11T19:18:02.492 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:18:02.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.488+0000 7f401a1b6640 1 -- 172.21.15.29:0/3874744165 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3ffc0a1ab0 msgr2=0x7f3ffc0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.488+0000 7f401a1b6640 1 --2- 172.21.15.29:0/3874744165 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3ffc0a1ab0 0x7f3ffc0a3f70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f400c005fd0 tx=0x7f400c00f730 comp rx=0 tx=0).stop 2024-03-11T19:18:02.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.488+0000 7f401a1b6640 1 -- 172.21.15.29:0/3874744165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40140ff970 msgr2=0x7f4014198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.488+0000 7f401a1b6640 1 --2- 172.21.15.29:0/3874744165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40140ff970 0x7f4014198500 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f400402c470 tx=0x7f4004004610 comp rx=0 tx=0).stop 2024-03-11T19:18:02.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.488+0000 7f401a1b6640 1 -- 172.21.15.29:0/3874744165 shutdown_connections 2024-03-11T19:18:02.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.488+0000 7f401a1b6640 1 --2- 172.21.15.29:0/3874744165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40141013c0 0x7f4014195070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.488+0000 7f401a1b6640 1 --2- 172.21.15.29:0/3874744165 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4014100a20 0x7f4014198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.488+0000 7f401a1b6640 1 --2- 172.21.15.29:0/3874744165 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3ffc0a1ab0 0x7f3ffc0a3f70 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.488+0000 7f401a1b6640 1 --2- 172.21.15.29:0/3874744165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40140ff970 0x7f4014198500 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.488+0000 7f401a1b6640 1 -- 172.21.15.29:0/3874744165 >> 172.21.15.29:0/3874744165 conn(0x7f40140fa940 msgr2=0x7f4014103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.488+0000 7f401a1b6640 1 -- 172.21.15.29:0/3874744165 shutdown_connections 2024-03-11T19:18:02.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.490+0000 7f401a1b6640 1 -- 172.21.15.29:0/3874744165 wait complete. 2024-03-11T19:18:02.508 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:02 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/806484443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:18:02.509 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:18:02.509 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:18:02.513 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:02 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/806484443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:18:02.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:02 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/806484443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:18:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.684+0000 7fe1cea26640 1 -- 172.21.15.29:0/1451492694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1c0096810 msgr2=0x7fe1c009d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.684+0000 7fe1cea26640 1 --2- 172.21.15.29:0/1451492694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1c0096810 0x7fe1c009d0a0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8009900 tx=0x7fe1b802bc60 comp rx=0 tx=0).stop 2024-03-11T19:18:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.684+0000 7fe1cea26640 1 -- 172.21.15.29:0/1451492694 shutdown_connections 2024-03-11T19:18:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.684+0000 7fe1cea26640 1 --2- 172.21.15.29:0/1451492694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1c0096810 0x7fe1c009d0a0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.684+0000 7fe1cea26640 1 --2- 172.21.15.29:0/1451492694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1c0095e50 0x7fe1c00962d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.684+0000 7fe1cea26640 1 --2- 172.21.15.29:0/1451492694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1c0094c60 0x7fe1c0095080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.684+0000 7fe1cea26640 1 -- 172.21.15.29:0/1451492694 >> 172.21.15.29:0/1451492694 conn(0x7fe1c00901d0 msgr2=0x7fe1c0092610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.684+0000 7fe1cea26640 1 -- 172.21.15.29:0/1451492694 shutdown_connections 2024-03-11T19:18:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.684+0000 7fe1cea26640 1 -- 172.21.15.29:0/1451492694 wait complete. 2024-03-11T19:18:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.685+0000 7fe1cea26640 1 Processor -- start 2024-03-11T19:18:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.685+0000 7fe1cea26640 1 -- start start 2024-03-11T19:18:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.686+0000 7fe1cea26640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1c0094c60 0x7fe1c012bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.686+0000 7fe1cea26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1c0095e50 0x7fe1c012c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.686+0000 7fe1cea26640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1c0096810 0x7fe1c01332b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.686+0000 7fe1cea26640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c0133820 con 0x7fe1c0095e50 2024-03-11T19:18:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.686+0000 7fe1cea26640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c0133990 con 0x7fe1c0096810 2024-03-11T19:18:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.686+0000 7fe1cea26640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c0133b50 con 0x7fe1c0094c60 2024-03-11T19:18:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.686+0000 7fe1ce225640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1c0096810 0x7fe1c01332b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.686+0000 7fe1ce225640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1c0096810 0x7fe1c01332b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56702/0 (socket says 172.21.15.29:56702) 2024-03-11T19:18:02.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.686+0000 7fe1ce225640 1 -- 172.21.15.29:0/911326600 learned_addr learned my addr 172.21.15.29:0/911326600 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:02.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.687+0000 7fe1ce225640 1 -- 172.21.15.29:0/911326600 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1c0094c60 msgr2=0x7fe1c012bd40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:18:02.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.687+0000 7fe1ce225640 1 --2- 172.21.15.29:0/911326600 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1c0094c60 0x7fe1c012bd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.687+0000 7fe1ce225640 1 -- 172.21.15.29:0/911326600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1c0095e50 msgr2=0x7fe1c012c280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:18:02.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.687+0000 7fe1ce225640 1 --2- 172.21.15.29:0/911326600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1c0095e50 0x7fe1c012c280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.687+0000 7fe1ce225640 1 -- 172.21.15.29:0/911326600 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b8009560 con 0x7fe1c0096810 2024-03-11T19:18:02.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.687+0000 7fe1ce225640 1 --2- 172.21.15.29:0/911326600 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1c0096810 0x7fe1c01332b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe1b802c660 tx=0x7fe1b8004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.688+0000 7fe1beffd640 1 -- 172.21.15.29:0/911326600 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b80035f0 con 0x7fe1c0096810 2024-03-11T19:18:02.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.688+0000 7fe1cea26640 1 -- 172.21.15.29:0/911326600 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c0133dd0 con 0x7fe1c0096810 2024-03-11T19:18:02.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.688+0000 7fe1cea26640 1 -- 172.21.15.29:0/911326600 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c0134260 con 0x7fe1c0096810 2024-03-11T19:18:02.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.688+0000 7fe1beffd640 1 -- 172.21.15.29:0/911326600 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe1b8003c00 con 0x7fe1c0096810 2024-03-11T19:18:02.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.689+0000 7fe1beffd640 1 -- 172.21.15.29:0/911326600 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b80035f0 con 0x7fe1c0096810 2024-03-11T19:18:02.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.692+0000 7fe1d2866740 1 Processor -- start 2024-03-11T19:18:02.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.693+0000 7fe1d2866740 1 -- start start 2024-03-11T19:18:02.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.693+0000 7fe1d2866740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a32459e400 0x55a32459e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.693+0000 7fe1d2866740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459edf0 0x55a3245a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.693+0000 7fe1d2866740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3245a1f50 0x55a3245aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.693+0000 7fe1d2866740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32455a960 con 0x55a3245a1f50 2024-03-11T19:18:02.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.693+0000 7fe1d2866740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32450e310 con 0x55a32459edf0 2024-03-11T19:18:02.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.693+0000 7fe1d2866740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32450b840 con 0x55a32459e400 2024-03-11T19:18:02.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.694+0000 7fe1beffd640 1 -- 172.21.15.29:0/911326600 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe1b8003750 con 0x7fe1c0096810 2024-03-11T19:18:02.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.695+0000 7fe1cd223640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a32459e400 0x55a32459e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.695+0000 7fe1cd223640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a32459e400 0x55a32459e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:44296/0 (socket says 172.21.15.29:44296) 2024-03-11T19:18:02.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.695+0000 7fe1cd223640 1 -- 172.21.15.29:0/2373671268 learned_addr learned my addr 172.21.15.29:0/2373671268 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:02.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.695+0000 7fe1cd223640 1 --2- 172.21.15.29:0/2373671268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3245a1f50 0x55a3245aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.695+0000 7fe1cd223640 1 -- 172.21.15.29:0/2373671268 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459edf0 msgr2=0x55a3245a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.695+0000 7fe1cd223640 1 --2- 172.21.15.29:0/2373671268 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459edf0 0x55a3245a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.695+0000 7fe1cd223640 1 -- 172.21.15.29:0/2373671268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3245a1f50 msgr2=0x55a3245aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.695+0000 7fe1cd223640 1 --2- 172.21.15.29:0/2373671268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3245a1f50 0x55a3245aedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.695+0000 7fe1cd223640 1 -- 172.21.15.29:0/2373671268 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a3244c9060 con 0x55a32459e400 2024-03-11T19:18:02.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.695+0000 7fe1beffd640 1 --2- 172.21.15.29:0/911326600 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe1b40a1ae0 0x7fe1b40a3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.695+0000 7fe1beffd640 1 -- 172.21.15.29:0/911326600 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe1b80ef130 con 0x7fe1c0096810 2024-03-11T19:18:02.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.697+0000 7fe1cda24640 1 --2- 172.21.15.29:0/911326600 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe1b40a1ae0 0x7fe1b40a3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.697+0000 7fe1cd223640 1 --2- 172.21.15.29:0/2373671268 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a32459e400 0x55a32459e820 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe1c40137a0 tx=0x7fe1c400c2b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2a464ade30d3acb server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.698+0000 7fe1b3fff640 1 -- 172.21.15.29:0/2373671268 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c4013250 con 0x55a32459e400 2024-03-11T19:18:02.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.698+0000 7fe1b3fff640 1 -- 172.21.15.29:0/2373671268 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe1c40054d0 con 0x55a32459e400 2024-03-11T19:18:02.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.699+0000 7fe1d2866740 1 -- 172.21.15.29:0/2373671268 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a32459e400 msgr2=0x55a32459e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.699+0000 7fe1d2866740 1 --2- 172.21.15.29:0/2373671268 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a32459e400 0x55a32459e820 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe1c40137a0 tx=0x7fe1c400c2b0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.700+0000 7fe1d2866740 1 -- 172.21.15.29:0/2373671268 shutdown_connections 2024-03-11T19:18:02.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.700+0000 7fe1d2866740 1 --2- 172.21.15.29:0/2373671268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3245a1f50 0x55a3245aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.700+0000 7fe1d2866740 1 --2- 172.21.15.29:0/2373671268 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459edf0 0x55a3245a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.700+0000 7fe1d2866740 1 --2- 172.21.15.29:0/2373671268 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a32459e400 0x55a32459e820 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.700+0000 7fe1d2866740 1 -- 172.21.15.29:0/2373671268 >> 172.21.15.29:0/2373671268 conn(0x55a3245999f0 msgr2=0x55a324599df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.701+0000 7fe1cda24640 1 --2- 172.21.15.29:0/911326600 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe1b40a1ae0 0x7fe1b40a3fa0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8066920 tx=0x7fe1c8062680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.702+0000 7fe1d2866740 1 -- 172.21.15.29:0/2373671268 shutdown_connections 2024-03-11T19:18:02.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.703+0000 7fe1d2866740 1 -- 172.21.15.29:0/2373671268 wait complete. 2024-03-11T19:18:02.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.703+0000 7fe1d2866740 1 -- start start 2024-03-11T19:18:02.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.704+0000 7fe1d2866740 1 Processor -- start 2024-03-11T19:18:02.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.704+0000 7fe1d2866740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a32459e400 0x55a3245a85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.704+0000 7fe1d2866740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459edf0 0x55a3245a8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.704+0000 7fe1d2866740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3245a1f50 0x55a3245adb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.705+0000 7fe1d2866740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a324507020 con 0x55a32459e400 2024-03-11T19:18:02.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.705+0000 7fe1d2866740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3241da430 con 0x55a32459edf0 2024-03-11T19:18:02.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.705+0000 7fe1cda24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a32459e400 0x55a3245a85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.705+0000 7fe1cd223640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459edf0 0x55a3245a8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.705+0000 7fe1cda24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a32459e400 0x55a3245a85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49568/0 (socket says 172.21.15.29:49568) 2024-03-11T19:18:02.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.705+0000 7fe1cda24640 1 -- 172.21.15.29:0/1950244219 learned_addr learned my addr 172.21.15.29:0/1950244219 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:02.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.705+0000 7fe1d2866740 1 -- 172.21.15.29:0/1950244219 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a324196e00 con 0x55a3245a1f50 2024-03-11T19:18:02.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.705+0000 7fe1d2866740 10 client.0 _ll_register_callbacks cb 0x55a3244aecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:02.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.705+0000 7fe1ce225640 1 --2- 172.21.15.29:0/1950244219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3245a1f50 0x55a3245adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.706+0000 7fe1cd223640 1 -- 172.21.15.29:0/1950244219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3245a1f50 msgr2=0x55a3245adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.706+0000 7fe1cd223640 1 --2- 172.21.15.29:0/1950244219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3245a1f50 0x55a3245adb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.706+0000 7fe1cd223640 1 -- 172.21.15.29:0/1950244219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a32459e400 msgr2=0x55a3245a85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.706+0000 7fe1cd223640 1 --2- 172.21.15.29:0/1950244219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a32459e400 0x55a3245a85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.706+0000 7fe1cd223640 1 -- 172.21.15.29:0/1950244219 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c4013480 con 0x55a32459edf0 2024-03-11T19:18:02.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.706+0000 7fe1cda24640 1 --2- 172.21.15.29:0/1950244219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a32459e400 0x55a3245a85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:18:02.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.706+0000 7fe1cd223640 1 --2- 172.21.15.29:0/1950244219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459edf0 0x55a3245a8b00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x55a32459f650 tx=0x7fe1c4004910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5756c8c8a800d6f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.706+0000 7fe1b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:18:02.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.706+0000 7fe1b17fa640 1 -- 172.21.15.29:0/1950244219 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c4005640 con 0x55a32459edf0 2024-03-11T19:18:02.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.707+0000 7fe1b17fa640 1 -- 172.21.15.29:0/1950244219 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe1c4004020 con 0x55a32459edf0 2024-03-11T19:18:02.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.707+0000 7fe1d2866740 1 -- 172.21.15.29:0/1950244219 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a3242067e0 con 0x55a32459edf0 2024-03-11T19:18:02.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.707+0000 7fe1b17fa640 1 -- 172.21.15.29:0/1950244219 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1c40041e0 con 0x55a32459edf0 2024-03-11T19:18:02.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.707+0000 7fe1d2866740 10 client.26846 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:02.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.707+0000 7fe1d2866740 1 -- 172.21.15.29:0/1950244219 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a3245056a0 con 0x55a32459edf0 2024-03-11T19:18:02.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.707+0000 7fe1b17fa640 1 -- 172.21.15.29:0/1950244219 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55a3242067e0 con 0x55a32459edf0 2024-03-11T19:18:02.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.708+0000 7fe1b17fa640 1 -- 172.21.15.29:0/1950244219 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe1c401b070 con 0x55a32459edf0 2024-03-11T19:18:02.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.708+0000 7fe1d2866740 10 client.26846 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:02.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.708+0000 7fe1d2866740 10 client.26846 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:18:02.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.708+0000 7fe1d2866740 20 client.26846 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:02.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.708+0000 7fe1d2866740 1 --2- 172.21.15.29:0/1950244219 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55a3245c5870 0x55a3245c7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.708+0000 7fe1d2866740 4 client.26846 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:18:02.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.708+0000 7fe1d2866740 1 -- 172.21.15.29:0/1950244219 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a324243720 con 0x55a3245c5870 2024-03-11T19:18:02.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.709+0000 7fe1cda24640 1 --2- 172.21.15.29:0/1950244219 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55a3245c5870 0x55a3245c7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.709+0000 7fe1b17fa640 10 client.26846 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:18:02.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.709+0000 7fe1cda24640 1 --2- 172.21.15.29:0/1950244219 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55a3245c5870 0x55a3245c7c90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2e5d5c902d5b247b server_cookie=ed173136e6e7243b in_seq=0 out_seq=0 2024-03-11T19:18:02.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.710+0000 7fe1b17fa640 1 -- 172.21.15.29:0/1950244219 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a324243720 con 0x55a3245c5870 2024-03-11T19:18:02.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.710+0000 7fe1b17fa640 10 client.26846 handle_command_reply: tid=0 2024-03-11T19:18:02.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.710+0000 7fe1d2866740 1 client.26846 shutdown 2024-03-11T19:18:02.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.710+0000 7fe1d2866740 1 -- 172.21.15.29:0/1950244219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459edf0 msgr2=0x55a3245a8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.710+0000 7fe1d2866740 1 --2- 172.21.15.29:0/1950244219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459edf0 0x55a3245a8b00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x55a32459f650 tx=0x7fe1c4004910 comp rx=0 tx=0).stop 2024-03-11T19:18:02.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.711+0000 7fe1d2866740 1 -- 172.21.15.29:0/1950244219 shutdown_connections 2024-03-11T19:18:02.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.711+0000 7fe1d2866740 1 --2- 172.21.15.29:0/1950244219 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55a3245c5870 0x55a3245c7c90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.711+0000 7fe1d2866740 1 --2- 172.21.15.29:0/1950244219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3245a1f50 0x55a3245adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.711+0000 7fe1d2866740 1 --2- 172.21.15.29:0/1950244219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459edf0 0x55a3245a8b00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.711+0000 7fe1d2866740 1 --2- 172.21.15.29:0/1950244219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a32459e400 0x55a3245a85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.711+0000 7fe1d2866740 1 -- 172.21.15.29:0/1950244219 >> 172.21.15.29:0/1950244219 conn(0x55a3245999f0 msgr2=0x55a324598cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.711+0000 7fe1b17fa640 0 client.26846 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:18:02.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.712+0000 7fe1d2866740 1 -- 172.21.15.29:0/1950244219 shutdown_connections 2024-03-11T19:18:02.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.712+0000 7fe1d2866740 1 -- 172.21.15.29:0/1950244219 wait complete. 2024-03-11T19:18:02.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.712+0000 7fe1d2866740 20 client.26846 trim_cache size 0 max 16384 2024-03-11T19:18:02.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.738+0000 7fe1d2866740 1 Processor -- start 2024-03-11T19:18:02.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.738+0000 7fe1d2866740 1 -- start start 2024-03-11T19:18:02.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.738+0000 7fe1d2866740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459e400 0x55a3245a85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.738+0000 7fe1d2866740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a32459edf0 0x55a3245a8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.738+0000 7fe1d2866740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3245a1f50 0x55a3245d5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.738+0000 7fe1d2866740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32448c130 con 0x55a3245a1f50 2024-03-11T19:18:02.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.738+0000 7fe1d2866740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a324243720 con 0x55a32459e400 2024-03-11T19:18:02.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.738+0000 7fe1d2866740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3242067e0 con 0x55a32459edf0 2024-03-11T19:18:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.739+0000 7fe1ce225640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3245a1f50 0x55a3245d5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.739+0000 7fe1ce225640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3245a1f50 0x55a3245d5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49578/0 (socket says 172.21.15.29:49578) 2024-03-11T19:18:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.739+0000 7fe1cd223640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a32459edf0 0x55a3245a8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.739+0000 7fe1ce225640 1 -- 172.21.15.29:0/359185304 learned_addr learned my addr 172.21.15.29:0/359185304 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.739+0000 7fe1ce225640 1 -- 172.21.15.29:0/359185304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a32459edf0 msgr2=0x55a3245a8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.739+0000 7fe1ce225640 1 --2- 172.21.15.29:0/359185304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a32459edf0 0x55a3245a8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.739+0000 7fe1ce225640 1 -- 172.21.15.29:0/359185304 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459e400 msgr2=0x55a3245a85c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:18:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.739+0000 7fe1ce225640 1 --2- 172.21.15.29:0/359185304 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459e400 0x55a3245a85c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.739+0000 7fe1ce225640 1 -- 172.21.15.29:0/359185304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a3244a1c20 con 0x55a3245a1f50 2024-03-11T19:18:02.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.740+0000 7fe1ce225640 1 --2- 172.21.15.29:0/359185304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3245a1f50 0x55a3245d5b30 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8005ab0 tx=0x7fe1b80bc280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23d778a19f06fd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.741+0000 7fe1b37fe640 1 -- 172.21.15.29:0/359185304 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b80035f0 con 0x55a3245a1f50 2024-03-11T19:18:02.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.741+0000 7fe1b37fe640 1 -- 172.21.15.29:0/359185304 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe1b8003c00 con 0x55a3245a1f50 2024-03-11T19:18:02.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.741+0000 7fe1b37fe640 1 -- 172.21.15.29:0/359185304 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b80b87b0 con 0x55a3245a1f50 2024-03-11T19:18:02.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.742+0000 7fe1d2866740 1 -- 172.21.15.29:0/359185304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3245a1f50 msgr2=0x55a3245d5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.742+0000 7fe1d2866740 1 --2- 172.21.15.29:0/359185304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3245a1f50 0x55a3245d5b30 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8005ab0 tx=0x7fe1b80bc280 comp rx=0 tx=0).stop 2024-03-11T19:18:02.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.742+0000 7fe1d2866740 1 -- 172.21.15.29:0/359185304 shutdown_connections 2024-03-11T19:18:02.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.742+0000 7fe1d2866740 1 --2- 172.21.15.29:0/359185304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a3245a1f50 0x55a3245d5b30 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.742+0000 7fe1d2866740 1 --2- 172.21.15.29:0/359185304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a32459edf0 0x55a3245a8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.742+0000 7fe1d2866740 1 --2- 172.21.15.29:0/359185304 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459e400 0x55a3245a85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.742+0000 7fe1d2866740 1 -- 172.21.15.29:0/359185304 >> 172.21.15.29:0/359185304 conn(0x55a3245999f0 msgr2=0x55a3245cc5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.743+0000 7fe1d2866740 1 -- 172.21.15.29:0/359185304 shutdown_connections 2024-03-11T19:18:02.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.743+0000 7fe1d2866740 1 -- 172.21.15.29:0/359185304 wait complete. 2024-03-11T19:18:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.743+0000 7fe1d2866740 1 -- start start 2024-03-11T19:18:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.744+0000 7fe1d2866740 1 Processor -- start 2024-03-11T19:18:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.744+0000 7fe1d2866740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459e400 0x55a3246167d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.745+0000 7fe1d2866740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a32459edf0 0x55a324618d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.745+0000 7fe1cda24640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459e400 0x55a3246167d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.745+0000 7fe1cda24640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459e400 0x55a3246167d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56738/0 (socket says 172.21.15.29:56738) 2024-03-11T19:18:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.745+0000 7fe1cda24640 1 -- 172.21.15.29:0/3866732732 learned_addr learned my addr 172.21.15.29:0/3866732732 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.745+0000 7fe1d2866740 1 --2- 172.21.15.29:0/3866732732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3245a1f50 0x55a32461b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.745+0000 7fe1d2866740 1 -- 172.21.15.29:0/3866732732 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3245d63e0 con 0x55a32459edf0 2024-03-11T19:18:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.745+0000 7fe1ce225640 1 --2- 172.21.15.29:0/3866732732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3245a1f50 0x55a32461b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.745+0000 7fe1d2866740 1 -- 172.21.15.29:0/3866732732 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32461b7b0 con 0x55a32459e400 2024-03-11T19:18:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.745+0000 7fe1d2866740 1 -- 172.21.15.29:0/3866732732 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32461b8f0 con 0x55a3245a1f50 2024-03-11T19:18:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.746+0000 7fe1d2866740 10 client.0 _ll_register_callbacks cb 0x7fe1c400c850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:02.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.746+0000 7fe1cda24640 1 -- 172.21.15.29:0/3866732732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3245a1f50 msgr2=0x55a32461b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.746+0000 7fe1cda24640 1 --2- 172.21.15.29:0/3866732732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3245a1f50 0x55a32461b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.746+0000 7fe1cda24640 1 -- 172.21.15.29:0/3866732732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a32459edf0 msgr2=0x55a324618d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.746+0000 7fe1cda24640 1 --2- 172.21.15.29:0/3866732732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a32459edf0 0x55a324618d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.746+0000 7fe1cda24640 1 -- 172.21.15.29:0/3866732732 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c0134260 con 0x55a32459e400 2024-03-11T19:18:02.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.746+0000 7fe1d2866740 1 -- 172.21.15.29:0/3866732732 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe1c40054d0 con 0x55a32459e400 2024-03-11T19:18:02.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.746+0000 7fe1cda24640 1 --2- 172.21.15.29:0/3866732732 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459e400 0x55a3246167d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe1c80668b0 tx=0x7fe1c8066df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b50b32311010b90 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.746+0000 7fe1b17fa640 10 client.26849 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:18:02.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.747+0000 7fe1b17fa640 1 -- 172.21.15.29:0/3866732732 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8087050 con 0x55a32459e400 2024-03-11T19:18:02.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.748+0000 7fe1b17fa640 1 -- 172.21.15.29:0/3866732732 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe1c808e020 con 0x55a32459e400 2024-03-11T19:18:02.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.748+0000 7fe1b17fa640 1 -- 172.21.15.29:0/3866732732 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1c807b070 con 0x55a32459e400 2024-03-11T19:18:02.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.748+0000 7fe1d2866740 10 client.26849 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:02.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.748+0000 7fe1d2866740 1 -- 172.21.15.29:0/3866732732 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x55a3244c78c0 con 0x55a32459e400 2024-03-11T19:18:02.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.749+0000 7fe1b17fa640 1 -- 172.21.15.29:0/3866732732 <== mon.2 v2:172.21.15.170:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe1c808e1e0 con 0x55a32459e400 2024-03-11T19:18:02.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.749+0000 7fe1d2866740 10 client.26849 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:02.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.749+0000 7fe1d2866740 10 client.26849 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:18:02.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.749+0000 7fe1d2866740 20 client.26849 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:02.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.750+0000 7fe1d2866740 1 --2- 172.21.15.29:0/3866732732 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55a32461d250 0x55a32461f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.750+0000 7fe1d2866740 4 client.26849 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:18:02.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.750+0000 7fe1d2866740 1 -- 172.21.15.29:0/3866732732 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7fe1c40054d0 con 0x55a32461d250 2024-03-11T19:18:02.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.750+0000 7fe1cd223640 1 --2- 172.21.15.29:0/3866732732 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55a32461d250 0x55a32461f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.751+0000 7fe1cd223640 1 --2- 172.21.15.29:0/3866732732 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55a32461d250 0x55a32461f670 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=834e45480216310b server_cookie=23227e87ceb89101 in_seq=0 out_seq=0 2024-03-11T19:18:02.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.751+0000 7fe1b17fa640 10 client.26849 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:18:02.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.751+0000 7fe1b17fa640 1 -- 172.21.15.29:0/3866732732 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fe1c40054d0 con 0x55a32461d250 2024-03-11T19:18:02.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.751+0000 7fe1b17fa640 10 client.26849 handle_command_reply: tid=0 2024-03-11T19:18:02.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.751+0000 7fe1d2866740 1 client.26849 shutdown 2024-03-11T19:18:02.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.753+0000 7fe1d2866740 1 -- 172.21.15.29:0/3866732732 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459e400 msgr2=0x55a3246167d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.753+0000 7fe1d2866740 1 --2- 172.21.15.29:0/3866732732 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459e400 0x55a3246167d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe1c80668b0 tx=0x7fe1c8066df0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.753+0000 7fe1d2866740 1 -- 172.21.15.29:0/3866732732 shutdown_connections 2024-03-11T19:18:02.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.753+0000 7fe1d2866740 1 --2- 172.21.15.29:0/3866732732 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55a32461d250 0x55a32461f670 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.753+0000 7fe1d2866740 1 --2- 172.21.15.29:0/3866732732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a3245a1f50 0x55a32461b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.753+0000 7fe1d2866740 1 --2- 172.21.15.29:0/3866732732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a32459edf0 0x55a324618d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.753+0000 7fe1d2866740 1 --2- 172.21.15.29:0/3866732732 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a32459e400 0x55a3246167d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.753+0000 7fe1b17fa640 0 client.26849 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:18:02.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.753+0000 7fe1d2866740 1 -- 172.21.15.29:0/3866732732 >> 172.21.15.29:0/3866732732 conn(0x55a3245999f0 msgr2=0x55a3245a35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.754+0000 7fe1d2866740 1 -- 172.21.15.29:0/3866732732 shutdown_connections 2024-03-11T19:18:02.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.754+0000 7fe1d2866740 1 -- 172.21.15.29:0/3866732732 wait complete. 2024-03-11T19:18:02.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.755+0000 7fe1d2866740 20 client.26849 trim_cache size 0 max 16384 2024-03-11T19:18:02.759 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:18:02.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.755+0000 7fe1cea26640 1 -- 172.21.15.29:0/911326600 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe1b40a1ae0 msgr2=0x7fe1b40a3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.755+0000 7fe1cea26640 1 --2- 172.21.15.29:0/911326600 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe1b40a1ae0 0x7fe1b40a3fa0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8066920 tx=0x7fe1c8062680 comp rx=0 tx=0).stop 2024-03-11T19:18:02.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.755+0000 7fe1cea26640 1 -- 172.21.15.29:0/911326600 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1c0096810 msgr2=0x7fe1c01332b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.755+0000 7fe1cea26640 1 --2- 172.21.15.29:0/911326600 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1c0096810 0x7fe1c01332b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe1b802c660 tx=0x7fe1b8004270 comp rx=0 tx=0).stop 2024-03-11T19:18:02.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.756+0000 7fe1cea26640 1 -- 172.21.15.29:0/911326600 shutdown_connections 2024-03-11T19:18:02.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.756+0000 7fe1cea26640 1 --2- 172.21.15.29:0/911326600 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1c0096810 0x7fe1c01332b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.756+0000 7fe1cea26640 1 --2- 172.21.15.29:0/911326600 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe1b40a1ae0 0x7fe1b40a3fa0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.756+0000 7fe1cea26640 1 --2- 172.21.15.29:0/911326600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1c0095e50 0x7fe1c012c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.756+0000 7fe1cea26640 1 --2- 172.21.15.29:0/911326600 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1c0094c60 0x7fe1c012bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.756+0000 7fe1cea26640 1 -- 172.21.15.29:0/911326600 >> 172.21.15.29:0/911326600 conn(0x7fe1c00901d0 msgr2=0x7fe1c0099080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.756+0000 7fe1cea26640 1 -- 172.21.15.29:0/911326600 shutdown_connections 2024-03-11T19:18:02.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.756+0000 7fe1cea26640 1 -- 172.21.15.29:0/911326600 wait complete. 2024-03-11T19:18:02.776 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:18:02.777 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:18:02.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.959+0000 7f3e919c1640 1 -- 172.21.15.29:0/1529256808 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e8c0724d0 msgr2=0x7f3e8c0728f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.959+0000 7f3e919c1640 1 --2- 172.21.15.29:0/1529256808 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e8c0724d0 0x7f3e8c0728f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c01a910 tx=0x7f3e7c03cc90 comp rx=0 tx=0).stop 2024-03-11T19:18:02.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.961+0000 7f3e919c1640 1 -- 172.21.15.29:0/1529256808 shutdown_connections 2024-03-11T19:18:02.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.961+0000 7f3e919c1640 1 --2- 172.21.15.29:0/1529256808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e8c109120 0x7f3e8c10d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.961+0000 7f3e919c1640 1 --2- 172.21.15.29:0/1529256808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8c108780 0x7f3e8c108be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.961+0000 7f3e919c1640 1 --2- 172.21.15.29:0/1529256808 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e8c0724d0 0x7f3e8c0728f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.961+0000 7f3e919c1640 1 -- 172.21.15.29:0/1529256808 >> 172.21.15.29:0/1529256808 conn(0x7f3e8c06dfa0 msgr2=0x7f3e8c0703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.963+0000 7f3e919c1640 1 -- 172.21.15.29:0/1529256808 shutdown_connections 2024-03-11T19:18:02.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.963+0000 7f3e919c1640 1 -- 172.21.15.29:0/1529256808 wait complete. 2024-03-11T19:18:02.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.964+0000 7f3e919c1640 1 Processor -- start 2024-03-11T19:18:02.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.964+0000 7f3e919c1640 1 -- start start 2024-03-11T19:18:02.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.964+0000 7f3e919c1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e8c0724d0 0x7f3e8c19d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.964+0000 7f3e919c1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e8c108780 0x7f3e8c19db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.964+0000 7f3e919c1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8c109120 0x7f3e8c1a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.964+0000 7f3e919c1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c1a50b0 con 0x7f3e8c109120 2024-03-11T19:18:02.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.964+0000 7f3e919c1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c1a5220 con 0x7f3e8c0724d0 2024-03-11T19:18:02.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.964+0000 7f3e919c1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c1a53e0 con 0x7f3e8c108780 2024-03-11T19:18:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.964+0000 7f3e911c0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8c109120 0x7f3e8c1a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.964+0000 7f3e911c0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8c109120 0x7f3e8c1a4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49598/0 (socket says 172.21.15.29:49598) 2024-03-11T19:18:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.964+0000 7f3e911c0640 1 -- 172.21.15.29:0/1179641887 learned_addr learned my addr 172.21.15.29:0/1179641887 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.964+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/1179641887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e8c108780 0x7f3e8c19db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.965+0000 7f3e909bf640 1 --2- 172.21.15.29:0/1179641887 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e8c0724d0 0x7f3e8c19d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.965+0000 7f3e911c0640 1 -- 172.21.15.29:0/1179641887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e8c108780 msgr2=0x7f3e8c19db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.965+0000 7f3e911c0640 1 --2- 172.21.15.29:0/1179641887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e8c108780 0x7f3e8c19db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.965+0000 7f3e911c0640 1 -- 172.21.15.29:0/1179641887 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e8c0724d0 msgr2=0x7f3e8c19d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.965+0000 7f3e911c0640 1 --2- 172.21.15.29:0/1179641887 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e8c0724d0 0x7f3e8c19d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.965+0000 7f3e911c0640 1 -- 172.21.15.29:0/1179641887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e7c01a570 con 0x7f3e8c109120 2024-03-11T19:18:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.965+0000 7f3e911c0640 1 --2- 172.21.15.29:0/1179641887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8c109120 0x7f3e8c1a4b70 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f3e8400b990 tx=0x7f3e8400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.966+0000 7f3e89ffb640 1 -- 172.21.15.29:0/1179641887 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8400cba0 con 0x7f3e8c109120 2024-03-11T19:18:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.966+0000 7f3e89ffb640 1 -- 172.21.15.29:0/1179641887 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3e8400cd00 con 0x7f3e8c109120 2024-03-11T19:18:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.966+0000 7f3e89ffb640 1 -- 172.21.15.29:0/1179641887 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e840076c0 con 0x7f3e8c109120 2024-03-11T19:18:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.966+0000 7f3e919c1640 1 -- 172.21.15.29:0/1179641887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e8c1a56c0 con 0x7f3e8c109120 2024-03-11T19:18:02.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.966+0000 7f3e919c1640 1 -- 172.21.15.29:0/1179641887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e8c1a5c00 con 0x7f3e8c109120 2024-03-11T19:18:02.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.969+0000 7f3e89ffb640 1 -- 172.21.15.29:0/1179641887 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3e84014440 con 0x7f3e8c109120 2024-03-11T19:18:02.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.970+0000 7f3e957fb740 1 Processor -- start 2024-03-11T19:18:02.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.970+0000 7f3e957fb740 1 -- start start 2024-03-11T19:18:02.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.970+0000 7f3e957fb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210480 0x555b242108a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.970+0000 7f3e957fb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210e70 0x555b24215970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.970+0000 7f3e957fb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 0x555b24218470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.970+0000 7f3e957fb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b241cc960 con 0x555b24215fe0 2024-03-11T19:18:02.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.970+0000 7f3e957fb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b24180310 con 0x555b24210480 2024-03-11T19:18:02.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.970+0000 7f3e957fb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b2417d840 con 0x555b24210e70 2024-03-11T19:18:02.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.970+0000 7f3e89ffb640 1 --2- 172.21.15.29:0/1179641887 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3e740a1c00 0x7f3e740a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.970+0000 7f3e89ffb640 1 -- 172.21.15.29:0/1179641887 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f3e840ccc10 con 0x7f3e8c109120 2024-03-11T19:18:02.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.970+0000 7f3e911c0640 1 --2- 172.21.15.29:0/1179641887 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3e740a1c00 0x7f3e740a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.971+0000 7f3e909bf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210e70 0x555b24215970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.971+0000 7f3e8bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 0x555b24218470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.971+0000 7f3e8bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 0x555b24218470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49614/0 (socket says 172.21.15.29:49614) 2024-03-11T19:18:02.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.971+0000 7f3e8bfff640 1 -- 172.21.15.29:0/977318568 learned_addr learned my addr 172.21.15.29:0/977318568 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:02.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.971+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/977318568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210480 0x555b242108a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.971+0000 7f3e8bfff640 1 -- 172.21.15.29:0/977318568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210e70 msgr2=0x555b24215970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.971+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/977318568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210e70 0x555b24215970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.971+0000 7f3e8bfff640 1 -- 172.21.15.29:0/977318568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210480 msgr2=0x555b242108a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.971+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/977318568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210480 0x555b242108a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.971+0000 7f3e8bfff640 1 -- 172.21.15.29:0/977318568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b2413b060 con 0x555b24215fe0 2024-03-11T19:18:02.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.972+0000 7f3e911c0640 1 --2- 172.21.15.29:0/1179641887 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3e740a1c00 0x7f3e740a40c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3e84096600 tx=0x7f3e84014d10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.972+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/977318568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 0x555b24218470 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f3e80012790 tx=0x7f3e8000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f709985951c0294 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.972+0000 7f3e72ffd640 1 -- 172.21.15.29:0/977318568 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e80012dd0 con 0x555b24215fe0 2024-03-11T19:18:02.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.972+0000 7f3e72ffd640 1 -- 172.21.15.29:0/977318568 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3e80010c20 con 0x555b24215fe0 2024-03-11T19:18:02.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.972+0000 7f3e72ffd640 1 -- 172.21.15.29:0/977318568 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8001ea60 con 0x555b24215fe0 2024-03-11T19:18:02.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.973+0000 7f3e957fb740 1 -- 172.21.15.29:0/977318568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 msgr2=0x555b24218470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.973+0000 7f3e957fb740 1 --2- 172.21.15.29:0/977318568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 0x555b24218470 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f3e80012790 tx=0x7f3e8000b7c0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.974+0000 7f3e957fb740 1 -- 172.21.15.29:0/977318568 shutdown_connections 2024-03-11T19:18:02.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.974+0000 7f3e957fb740 1 --2- 172.21.15.29:0/977318568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 0x555b24218470 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.974+0000 7f3e957fb740 1 --2- 172.21.15.29:0/977318568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210e70 0x555b24215970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.974+0000 7f3e957fb740 1 --2- 172.21.15.29:0/977318568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210480 0x555b242108a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.974+0000 7f3e957fb740 1 -- 172.21.15.29:0/977318568 >> 172.21.15.29:0/977318568 conn(0x555b2420b9f0 msgr2=0x555b2420bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.974+0000 7f3e957fb740 1 -- 172.21.15.29:0/977318568 shutdown_connections 2024-03-11T19:18:02.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.974+0000 7f3e957fb740 1 -- 172.21.15.29:0/977318568 wait complete. 2024-03-11T19:18:02.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.974+0000 7f3e957fb740 1 -- start start 2024-03-11T19:18:02.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.974+0000 7f3e957fb740 1 Processor -- start 2024-03-11T19:18:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.975+0000 7f3e957fb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210480 0x555b2421a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.975+0000 7f3e957fb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210e70 0x555b2421abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.975+0000 7f3e957fb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 0x555b24221c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.975+0000 7f3e957fb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b24179020 con 0x555b24215fe0 2024-03-11T19:18:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.975+0000 7f3e957fb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b23e4c430 con 0x555b24210480 2024-03-11T19:18:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.975+0000 7f3e957fb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b23e08e00 con 0x555b24210e70 2024-03-11T19:18:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.975+0000 7f3e909bf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210e70 0x555b2421abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.975+0000 7f3e909bf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210e70 0x555b2421abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:44372/0 (socket says 172.21.15.29:44372) 2024-03-11T19:18:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.975+0000 7f3e909bf640 1 -- 172.21.15.29:0/320340022 learned_addr learned my addr 172.21.15.29:0/320340022 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.975+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/320340022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210480 0x555b2421a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.975+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/320340022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 0x555b24221c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.975+0000 7f3e957fb740 10 client.0 _ll_register_callbacks cb 0x555b24120cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.975+0000 7f3e8bfff640 1 -- 172.21.15.29:0/320340022 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210e70 msgr2=0x555b2421abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.975+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/320340022 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210e70 0x555b2421abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.975+0000 7f3e8bfff640 1 -- 172.21.15.29:0/320340022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 msgr2=0x555b24221c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.975+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/320340022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 0x555b24221c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.975+0000 7f3e8bfff640 1 -- 172.21.15.29:0/320340022 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e8001a040 con 0x555b24210480 2024-03-11T19:18:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.976+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/320340022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210480 0x555b2421a6a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3e800058c0 tx=0x7f3e80014040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff56c4a735ba66f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.976+0000 7f3e6bfff640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:18:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.976+0000 7f3e6bfff640 1 -- 172.21.15.29:0/320340022 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e80012e40 con 0x555b24210480 2024-03-11T19:18:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.976+0000 7f3e6bfff640 1 -- 172.21.15.29:0/320340022 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f3e80000e40 con 0x555b24210480 2024-03-11T19:18:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.976+0000 7f3e957fb740 1 -- 172.21.15.29:0/320340022 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555b23e787e0 con 0x555b24210480 2024-03-11T19:18:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.977+0000 7f3e6bfff640 1 -- 172.21.15.29:0/320340022 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e80010950 con 0x555b24210480 2024-03-11T19:18:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.977+0000 7f3e957fb740 10 client.26861 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.977+0000 7f3e957fb740 1 -- 172.21.15.29:0/320340022 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555b241776a0 con 0x555b24210480 2024-03-11T19:18:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.977+0000 7f3e6bfff640 1 -- 172.21.15.29:0/320340022 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x555b23e787e0 con 0x555b24210480 2024-03-11T19:18:02.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.977+0000 7f3e6bfff640 1 -- 172.21.15.29:0/320340022 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3e8002d630 con 0x555b24210480 2024-03-11T19:18:02.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.977+0000 7f3e957fb740 10 client.26861 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:02.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.977+0000 7f3e957fb740 10 client.26861 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:18:02.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.977+0000 7f3e957fb740 20 client.26861 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:02.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.977+0000 7f3e957fb740 1 --2- 172.21.15.29:0/320340022 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x555b242266b0 0x555b24228ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:02.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.977+0000 7f3e957fb740 4 client.26861 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:18:02.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.977+0000 7f3e957fb740 1 -- 172.21.15.29:0/320340022 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555b23eb5720 con 0x555b242266b0 2024-03-11T19:18:02.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.978+0000 7f3e909bf640 1 --2- 172.21.15.29:0/320340022 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x555b242266b0 0x555b24228ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:02.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.979+0000 7f3e6bfff640 10 client.26861 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:18:02.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.979+0000 7f3e909bf640 1 --2- 172.21.15.29:0/320340022 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x555b242266b0 0x555b24228ad0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f51b0fe0b2f68798 server_cookie=9142b1e21e5071df in_seq=0 out_seq=0 2024-03-11T19:18:02.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.980+0000 7f3e6bfff640 1 -- 172.21.15.29:0/320340022 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555b23eb5720 con 0x555b242266b0 2024-03-11T19:18:02.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.980+0000 7f3e6bfff640 10 client.26861 handle_command_reply: tid=0 2024-03-11T19:18:02.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.980+0000 7f3e957fb740 1 client.26861 shutdown 2024-03-11T19:18:02.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.980+0000 7f3e957fb740 1 -- 172.21.15.29:0/320340022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210480 msgr2=0x555b2421a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:02.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.980+0000 7f3e957fb740 1 --2- 172.21.15.29:0/320340022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210480 0x555b2421a6a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3e800058c0 tx=0x7f3e80014040 comp rx=0 tx=0).stop 2024-03-11T19:18:02.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.980+0000 7f3e957fb740 1 -- 172.21.15.29:0/320340022 shutdown_connections 2024-03-11T19:18:02.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.980+0000 7f3e957fb740 1 --2- 172.21.15.29:0/320340022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 0x555b24221c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.980+0000 7f3e957fb740 1 --2- 172.21.15.29:0/320340022 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210e70 0x555b2421abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.981+0000 7f3e957fb740 1 --2- 172.21.15.29:0/320340022 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x555b242266b0 0x555b24228ad0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.981+0000 7f3e957fb740 1 --2- 172.21.15.29:0/320340022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210480 0x555b2421a6a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.981+0000 7f3e6bfff640 0 client.26861 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:18:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.981+0000 7f3e957fb740 1 -- 172.21.15.29:0/320340022 >> 172.21.15.29:0/320340022 conn(0x555b2420b9f0 msgr2=0x555b2420acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.981+0000 7f3e957fb740 1 -- 172.21.15.29:0/320340022 shutdown_connections 2024-03-11T19:18:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.981+0000 7f3e957fb740 1 -- 172.21.15.29:0/320340022 wait complete. 2024-03-11T19:18:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:02.981+0000 7f3e957fb740 20 client.26861 trim_cache size 0 max 16384 2024-03-11T19:18:03.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.008+0000 7f3e957fb740 1 Processor -- start 2024-03-11T19:18:03.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.009+0000 7f3e957fb740 1 -- start start 2024-03-11T19:18:03.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.009+0000 7f3e957fb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210480 0x555b24211890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.009+0000 7f3e957fb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210e70 0x555b24221be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.009+0000 7f3e957fb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 0x555b2421a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.009+0000 7f3e957fb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b24207340 con 0x555b24215fe0 2024-03-11T19:18:03.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.009+0000 7f3e957fb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b240fe130 con 0x555b24210e70 2024-03-11T19:18:03.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.009+0000 7f3e957fb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b24218aa0 con 0x555b24210480 2024-03-11T19:18:03.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.010+0000 7f3e8bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 0x555b2421a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.010+0000 7f3e8bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 0x555b2421a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49640/0 (socket says 172.21.15.29:49640) 2024-03-11T19:18:03.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.010+0000 7f3e8bfff640 1 -- 172.21.15.29:0/51675745 learned_addr learned my addr 172.21.15.29:0/51675745 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:03.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.010+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/51675745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210480 0x555b24211890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.010+0000 7f3e909bf640 1 --2- 172.21.15.29:0/51675745 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210e70 0x555b24221be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.010+0000 7f3e8bfff640 1 -- 172.21.15.29:0/51675745 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210e70 msgr2=0x555b24221be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:03.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.010+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/51675745 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210e70 0x555b24221be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.010+0000 7f3e8bfff640 1 -- 172.21.15.29:0/51675745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 msgr2=0x555b2421a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:03.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.010+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/51675745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 0x555b2421a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.010+0000 7f3e8bfff640 1 -- 172.21.15.29:0/51675745 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b24113c20 con 0x555b24210480 2024-03-11T19:18:03.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.011+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/51675745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210480 0x555b24211890 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3e8002dc10 tx=0x7f3e80000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd7ba5678b64d79e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:03.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.011+0000 7f3e727fc640 1 -- 172.21.15.29:0/51675745 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8000bb80 con 0x555b24210480 2024-03-11T19:18:03.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.011+0000 7f3e727fc640 1 -- 172.21.15.29:0/51675745 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3e80033030 con 0x555b24210480 2024-03-11T19:18:03.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.012+0000 7f3e727fc640 1 -- 172.21.15.29:0/51675745 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8001f420 con 0x555b24210480 2024-03-11T19:18:03.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.012+0000 7f3e957fb740 1 -- 172.21.15.29:0/51675745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210480 msgr2=0x555b24211890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:03.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.012+0000 7f3e957fb740 1 --2- 172.21.15.29:0/51675745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210480 0x555b24211890 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3e8002dc10 tx=0x7f3e80000bc0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.012+0000 7f3e957fb740 1 -- 172.21.15.29:0/51675745 shutdown_connections 2024-03-11T19:18:03.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.012+0000 7f3e957fb740 1 --2- 172.21.15.29:0/51675745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 0x555b2421a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.012+0000 7f3e957fb740 1 --2- 172.21.15.29:0/51675745 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210e70 0x555b24221be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.012+0000 7f3e957fb740 1 --2- 172.21.15.29:0/51675745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210480 0x555b24211890 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.012+0000 7f3e957fb740 1 -- 172.21.15.29:0/51675745 >> 172.21.15.29:0/51675745 conn(0x555b2420b9f0 msgr2=0x555b2420bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:03.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.012+0000 7f3e957fb740 1 -- 172.21.15.29:0/51675745 shutdown_connections 2024-03-11T19:18:03.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.013+0000 7f3e957fb740 1 -- 172.21.15.29:0/51675745 wait complete. 2024-03-11T19:18:03.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.013+0000 7f3e957fb740 1 -- start start 2024-03-11T19:18:03.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.013+0000 7f3e957fb740 1 Processor -- start 2024-03-11T19:18:03.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.014+0000 7f3e957fb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210480 0x555b2421f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.014+0000 7f3e957fb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210e70 0x555b2421ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.014+0000 7f3e957fb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 0x555b24220f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.014+0000 7f3e957fb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b24221460 con 0x555b24215fe0 2024-03-11T19:18:03.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.014+0000 7f3e957fb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b242215d0 con 0x555b24210e70 2024-03-11T19:18:03.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.014+0000 7f3e957fb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b24221790 con 0x555b24210480 2024-03-11T19:18:03.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.014+0000 7f3e957fb740 10 client.0 _ll_register_callbacks cb 0x7f3e8000bad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:03.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.014+0000 7f3e8bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210480 0x555b2421f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.014+0000 7f3e8bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210480 0x555b2421f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:44398/0 (socket says 172.21.15.29:44398) 2024-03-11T19:18:03.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.014+0000 7f3e8bfff640 1 -- 172.21.15.29:0/273856120 learned_addr learned my addr 172.21.15.29:0/273856120 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:03.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.014+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/273856120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 0x555b24220f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.014+0000 7f3e909bf640 1 --2- 172.21.15.29:0/273856120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210e70 0x555b2421ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.014+0000 7f3e8bfff640 1 -- 172.21.15.29:0/273856120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210e70 msgr2=0x555b2421ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:03.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.014+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/273856120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210e70 0x555b2421ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.014+0000 7f3e8bfff640 1 -- 172.21.15.29:0/273856120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 msgr2=0x555b24220f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:03.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.014+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/273856120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 0x555b24220f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.014+0000 7f3e8bfff640 1 -- 172.21.15.29:0/273856120 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555b241776a0 con 0x555b24210480 2024-03-11T19:18:03.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.015+0000 7f3e6bfff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:18:03.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.015+0000 7f3e8bfff640 1 --2- 172.21.15.29:0/273856120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210480 0x555b2421f9d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3e80012e30 tx=0x7f3e80000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98f63117324b9fbd server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:03.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.015+0000 7f3e6bfff640 1 -- 172.21.15.29:0/273856120 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8001f420 con 0x555b24210480 2024-03-11T19:18:03.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.015+0000 7f3e6bfff640 1 -- 172.21.15.29:0/273856120 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f3e80000e40 con 0x555b24210480 2024-03-11T19:18:03.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.015+0000 7f3e957fb740 1 -- 172.21.15.29:0/273856120 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3e80010c20 con 0x555b24210480 2024-03-11T19:18:03.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.016+0000 7f3e6bfff640 1 -- 172.21.15.29:0/273856120 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e8000bb80 con 0x555b24210480 2024-03-11T19:18:03.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.016+0000 7f3e957fb740 10 client.26920 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:03.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.016+0000 7f3e957fb740 1 -- 172.21.15.29:0/273856120 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555b241398c0 con 0x555b24210480 2024-03-11T19:18:03.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.017+0000 7f3e6bfff640 1 -- 172.21.15.29:0/273856120 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3e80056070 con 0x555b24210480 2024-03-11T19:18:03.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.017+0000 7f3e6bfff640 1 -- 172.21.15.29:0/273856120 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3e80050480 con 0x555b24210480 2024-03-11T19:18:03.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.017+0000 7f3e957fb740 10 client.26920 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:03.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.017+0000 7f3e957fb740 10 client.26920 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:18:03.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.017+0000 7f3e957fb740 20 client.26920 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:03.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.017+0000 7f3e957fb740 1 --2- 172.21.15.29:0/273856120 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x555b2427d6c0 0x555b2427fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.017+0000 7f3e957fb740 4 client.26920 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:18:03.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.017+0000 7f3e957fb740 1 -- 172.21.15.29:0/273856120 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555b23f04890 con 0x555b2427d6c0 2024-03-11T19:18:03.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.018+0000 7f3e909bf640 1 --2- 172.21.15.29:0/273856120 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x555b2427d6c0 0x555b2427fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.019+0000 7f3e909bf640 1 --2- 172.21.15.29:0/273856120 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x555b2427d6c0 0x555b2427fae0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=53c0c0c1d85c5a96 server_cookie=f04455a96b9a4816 in_seq=0 out_seq=0 2024-03-11T19:18:03.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.019+0000 7f3e6bfff640 10 client.26920 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:18:03.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.019+0000 7f3e6bfff640 1 -- 172.21.15.29:0/273856120 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555b23f04890 con 0x555b2427d6c0 2024-03-11T19:18:03.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.019+0000 7f3e6bfff640 10 client.26920 handle_command_reply: tid=0 2024-03-11T19:18:03.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.019+0000 7f3e957fb740 1 client.26920 shutdown 2024-03-11T19:18:03.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.019+0000 7f3e957fb740 1 -- 172.21.15.29:0/273856120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210480 msgr2=0x555b2421f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:03.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.019+0000 7f3e957fb740 1 --2- 172.21.15.29:0/273856120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210480 0x555b2421f9d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3e80012e30 tx=0x7f3e80000bc0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.020+0000 7f3e957fb740 1 -- 172.21.15.29:0/273856120 shutdown_connections 2024-03-11T19:18:03.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.020+0000 7f3e957fb740 1 --2- 172.21.15.29:0/273856120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555b24215fe0 0x555b24220f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.020+0000 7f3e957fb740 1 --2- 172.21.15.29:0/273856120 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x555b2427d6c0 0x555b2427fae0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.020+0000 7f3e957fb740 1 --2- 172.21.15.29:0/273856120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555b24210e70 0x555b2421ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.020+0000 7f3e957fb740 1 --2- 172.21.15.29:0/273856120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b24210480 0x555b2421f9d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.020+0000 7f3e957fb740 1 -- 172.21.15.29:0/273856120 >> 172.21.15.29:0/273856120 conn(0x555b2420b9f0 msgr2=0x555b24211870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:03.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.020+0000 7f3e6bfff640 0 client.26920 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:18:03.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.020+0000 7f3e957fb740 1 -- 172.21.15.29:0/273856120 shutdown_connections 2024-03-11T19:18:03.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.020+0000 7f3e957fb740 1 -- 172.21.15.29:0/273856120 wait complete. 2024-03-11T19:18:03.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.020+0000 7f3e957fb740 20 client.26920 trim_cache size 0 max 16384 2024-03-11T19:18:03.028 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:18:03.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.026+0000 7f3e919c1640 1 -- 172.21.15.29:0/1179641887 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3e740a1c00 msgr2=0x7f3e740a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:03.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.026+0000 7f3e919c1640 1 --2- 172.21.15.29:0/1179641887 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3e740a1c00 0x7f3e740a40c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3e84096600 tx=0x7f3e84014d10 comp rx=0 tx=0).stop 2024-03-11T19:18:03.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.026+0000 7f3e919c1640 1 -- 172.21.15.29:0/1179641887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8c109120 msgr2=0x7f3e8c1a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:03.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.026+0000 7f3e919c1640 1 --2- 172.21.15.29:0/1179641887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8c109120 0x7f3e8c1a4b70 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f3e8400b990 tx=0x7f3e8400be50 comp rx=0 tx=0).stop 2024-03-11T19:18:03.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.026+0000 7f3e919c1640 1 -- 172.21.15.29:0/1179641887 shutdown_connections 2024-03-11T19:18:03.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.026+0000 7f3e919c1640 1 --2- 172.21.15.29:0/1179641887 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3e740a1c00 0x7f3e740a40c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.026+0000 7f3e919c1640 1 --2- 172.21.15.29:0/1179641887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8c109120 0x7f3e8c1a4b70 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.026+0000 7f3e919c1640 1 --2- 172.21.15.29:0/1179641887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e8c108780 0x7f3e8c19db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.026+0000 7f3e919c1640 1 --2- 172.21.15.29:0/1179641887 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e8c0724d0 0x7f3e8c19d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.026+0000 7f3e919c1640 1 -- 172.21.15.29:0/1179641887 >> 172.21.15.29:0/1179641887 conn(0x7f3e8c06dfa0 msgr2=0x7f3e8c10ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:03.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.026+0000 7f3e919c1640 1 -- 172.21.15.29:0/1179641887 shutdown_connections 2024-03-11T19:18:03.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.026+0000 7f3e919c1640 1 -- 172.21.15.29:0/1179641887 wait complete. 2024-03-11T19:18:03.048 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:18:03.048 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:18:03.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.191+0000 7f024bdd0640 1 -- 172.21.15.29:0/3414442748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0244101190 msgr2=0x7f02441015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:03.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.191+0000 7f024bdd0640 1 --2- 172.21.15.29:0/3414442748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0244101190 0x7f02441015b0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f0234009900 tx=0x7f023402bc30 comp rx=0 tx=0).stop 2024-03-11T19:18:03.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.191+0000 7f024bdd0640 1 -- 172.21.15.29:0/3414442748 shutdown_connections 2024-03-11T19:18:03.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.191+0000 7f024bdd0640 1 --2- 172.21.15.29:0/3414442748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0244102d40 0x7f02441095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.191+0000 7f024bdd0640 1 --2- 172.21.15.29:0/3414442748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0244102380 0x7f0244102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.191+0000 7f024bdd0640 1 --2- 172.21.15.29:0/3414442748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0244101190 0x7f02441015b0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.191+0000 7f024bdd0640 1 -- 172.21.15.29:0/3414442748 >> 172.21.15.29:0/3414442748 conn(0x7f02440fc720 msgr2=0x7f02440feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:03.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.192+0000 7f024bdd0640 1 -- 172.21.15.29:0/3414442748 shutdown_connections 2024-03-11T19:18:03.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.192+0000 7f024bdd0640 1 -- 172.21.15.29:0/3414442748 wait complete. 2024-03-11T19:18:03.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.192+0000 7f024bdd0640 1 Processor -- start 2024-03-11T19:18:03.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.192+0000 7f024bdd0640 1 -- start start 2024-03-11T19:18:03.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.192+0000 7f024bdd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0244101190 0x7f0244074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.192+0000 7f024bdd0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0244102380 0x7f0244073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.192+0000 7f024bdd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0244102d40 0x7f0244073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.192+0000 7f024bdd0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0244074210 con 0x7f0244102d40 2024-03-11T19:18:03.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.192+0000 7f024bdd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0244074380 con 0x7f0244101190 2024-03-11T19:18:03.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.192+0000 7f024bdd0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02441a1270 con 0x7f0244102380 2024-03-11T19:18:03.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.192+0000 7f024a36d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0244102d40 0x7f0244073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.192+0000 7f024a36d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0244102d40 0x7f0244073ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49666/0 (socket says 172.21.15.29:49666) 2024-03-11T19:18:03.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.192+0000 7f024a36d640 1 -- 172.21.15.29:0/3356566382 learned_addr learned my addr 172.21.15.29:0/3356566382 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:03.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.192+0000 7f024936b640 1 --2- 172.21.15.29:0/3356566382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0244102380 0x7f0244073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.193+0000 7f0249b6c640 1 --2- 172.21.15.29:0/3356566382 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0244101190 0x7f0244074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.193+0000 7f024a36d640 1 -- 172.21.15.29:0/3356566382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0244102380 msgr2=0x7f0244073570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:03.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.193+0000 7f024a36d640 1 --2- 172.21.15.29:0/3356566382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0244102380 0x7f0244073570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.193+0000 7f024a36d640 1 -- 172.21.15.29:0/3356566382 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0244101190 msgr2=0x7f0244074f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:03.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.193+0000 7f024a36d640 1 --2- 172.21.15.29:0/3356566382 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0244101190 0x7f0244074f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.193+0000 7f024a36d640 1 -- 172.21.15.29:0/3356566382 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0234009560 con 0x7f0244102d40 2024-03-11T19:18:03.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.193+0000 7f024a36d640 1 --2- 172.21.15.29:0/3356566382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0244102d40 0x7f0244073ab0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f023c00ead0 tx=0x7f023c00eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:03.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.193+0000 7f023affd640 1 -- 172.21.15.29:0/3356566382 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023c004d20 con 0x7f0244102d40 2024-03-11T19:18:03.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.193+0000 7f023affd640 1 -- 172.21.15.29:0/3356566382 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f023c004e80 con 0x7f0244102d40 2024-03-11T19:18:03.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.193+0000 7f024bdd0640 1 -- 172.21.15.29:0/3356566382 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02441a1520 con 0x7f0244102d40 2024-03-11T19:18:03.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.193+0000 7f023affd640 1 -- 172.21.15.29:0/3356566382 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023c011470 con 0x7f0244102d40 2024-03-11T19:18:03.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.193+0000 7f024bdd0640 1 -- 172.21.15.29:0/3356566382 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02441a1ab0 con 0x7f0244102d40 2024-03-11T19:18:03.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.195+0000 7f023affd640 1 -- 172.21.15.29:0/3356566382 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f023c005020 con 0x7f0244102d40 2024-03-11T19:18:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.196+0000 7f024e9a2740 1 Processor -- start 2024-03-11T19:18:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.196+0000 7f024e9a2740 1 -- start start 2024-03-11T19:18:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.196+0000 7f023affd640 1 --2- 172.21.15.29:0/3356566382 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f02300a1a30 0x7f02300a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.196+0000 7f024e9a2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2270 0x55fe941b2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.196+0000 7f024e9a2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b2c60 0x55fe941b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.196+0000 7f024e9a2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b7dd0 0x55fe941ba260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.196+0000 7f024e9a2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe9416e960 con 0x55fe941b2c60 2024-03-11T19:18:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.196+0000 7f024e9a2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe94122310 con 0x55fe941b7dd0 2024-03-11T19:18:03.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.196+0000 7f024e9a2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe9411f840 con 0x55fe941b2270 2024-03-11T19:18:03.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.196+0000 7f024936b640 1 --2- 172.21.15.29:0/3356566382 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f02300a1a30 0x7f02300a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.196+0000 7f023affd640 1 -- 172.21.15.29:0/3356566382 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f023c0cd5a0 con 0x7f0244102d40 2024-03-11T19:18:03.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.196+0000 7f024936b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b2c60 0x55fe941b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.196+0000 7f024936b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b2c60 0x55fe941b7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49674/0 (socket says 172.21.15.29:49674) 2024-03-11T19:18:03.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.196+0000 7f024936b640 1 -- 172.21.15.29:0/4144482667 learned_addr learned my addr 172.21.15.29:0/4144482667 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:03.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.196+0000 7f024a36d640 1 --2- 172.21.15.29:0/4144482667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b7dd0 0x55fe941ba260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.196+0000 7f0249b6c640 1 --2- 172.21.15.29:0/4144482667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2270 0x55fe941b2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.197+0000 7f024a36d640 1 -- 172.21.15.29:0/4144482667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2270 msgr2=0x55fe941b2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:03.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.197+0000 7f024a36d640 1 --2- 172.21.15.29:0/4144482667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2270 0x55fe941b2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.197+0000 7f024a36d640 1 -- 172.21.15.29:0/4144482667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b2c60 msgr2=0x55fe941b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:03.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.197+0000 7f024a36d640 1 --2- 172.21.15.29:0/4144482667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b2c60 0x55fe941b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.197+0000 7f024a36d640 1 -- 172.21.15.29:0/4144482667 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe940dd060 con 0x55fe941b7dd0 2024-03-11T19:18:03.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.197+0000 7f024936b640 1 --2- 172.21.15.29:0/4144482667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b2c60 0x55fe941b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:18:03.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.197+0000 7f024936b640 1 --2- 172.21.15.29:0/3356566382 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f02300a1a30 0x7f02300a3ef0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0240009fd0 tx=0x7f02400099b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:03.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.197+0000 7f024a36d640 1 --2- 172.21.15.29:0/4144482667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b7dd0 0x55fe941ba260 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f023c09a4a0 tx=0x7f023c0968e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecb26223b3cbca7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:03.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.197+0000 7f022ffff640 1 -- 172.21.15.29:0/4144482667 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023c0138c0 con 0x55fe941b7dd0 2024-03-11T19:18:03.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.197+0000 7f022ffff640 1 -- 172.21.15.29:0/4144482667 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f023c09ac10 con 0x55fe941b7dd0 2024-03-11T19:18:03.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.197+0000 7f022ffff640 1 -- 172.21.15.29:0/4144482667 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023c0a2d20 con 0x55fe941b7dd0 2024-03-11T19:18:03.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.198+0000 7f024e9a2740 1 -- 172.21.15.29:0/4144482667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b7dd0 msgr2=0x55fe941ba260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:03.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.198+0000 7f024e9a2740 1 --2- 172.21.15.29:0/4144482667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b7dd0 0x55fe941ba260 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f023c09a4a0 tx=0x7f023c0968e0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.198+0000 7f024e9a2740 1 -- 172.21.15.29:0/4144482667 shutdown_connections 2024-03-11T19:18:03.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.198+0000 7f024e9a2740 1 --2- 172.21.15.29:0/4144482667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b7dd0 0x55fe941ba260 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.198+0000 7f024e9a2740 1 --2- 172.21.15.29:0/4144482667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b2c60 0x55fe941b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.198+0000 7f024e9a2740 1 --2- 172.21.15.29:0/4144482667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2270 0x55fe941b2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.198+0000 7f024e9a2740 1 -- 172.21.15.29:0/4144482667 >> 172.21.15.29:0/4144482667 conn(0x55fe941ad7e0 msgr2=0x55fe941adbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:03.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.198+0000 7f024e9a2740 1 -- 172.21.15.29:0/4144482667 shutdown_connections 2024-03-11T19:18:03.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.198+0000 7f024e9a2740 1 -- 172.21.15.29:0/4144482667 wait complete. 2024-03-11T19:18:03.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.198+0000 7f024e9a2740 1 -- start start 2024-03-11T19:18:03.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.198+0000 7f024e9a2740 1 Processor -- start 2024-03-11T19:18:03.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.198+0000 7f024e9a2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b2270 0x55fe941bc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.198+0000 7f024e9a2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2c60 0x55fe941bc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f024e9a2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 0x55fe941c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f024e9a2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe9411b020 con 0x55fe941b7dd0 2024-03-11T19:18:03.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f024e9a2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe93dee430 con 0x55fe941b2270 2024-03-11T19:18:03.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f024e9a2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe93daae00 con 0x55fe941b2c60 2024-03-11T19:18:03.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f024e9a2740 10 client.0 _ll_register_callbacks cb 0x55fe940c2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:03.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f024a36d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 0x55fe941c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f024a36d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 0x55fe941c3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49690/0 (socket says 172.21.15.29:49690) 2024-03-11T19:18:03.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f024a36d640 1 -- 172.21.15.29:0/2166168286 learned_addr learned my addr 172.21.15.29:0/2166168286 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:03.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f0249b6c640 1 --2- 172.21.15.29:0/2166168286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b2270 0x55fe941bc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f024936b640 1 --2- 172.21.15.29:0/2166168286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2c60 0x55fe941bc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f024a36d640 1 -- 172.21.15.29:0/2166168286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2c60 msgr2=0x55fe941bc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:03.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f024a36d640 1 --2- 172.21.15.29:0/2166168286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2c60 0x55fe941bc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f024a36d640 1 -- 172.21.15.29:0/2166168286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b2270 msgr2=0x55fe941bc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:03.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f024a36d640 1 --2- 172.21.15.29:0/2166168286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b2270 0x55fe941bc480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f024a36d640 1 -- 172.21.15.29:0/2166168286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02441a1ab0 con 0x55fe941b7dd0 2024-03-11T19:18:03.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f024936b640 1 --2- 172.21.15.29:0/2166168286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2c60 0x55fe941bc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:18:03.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f024a36d640 1 --2- 172.21.15.29:0/2166168286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 0x55fe941c3a00 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f023c0a2780 tx=0x7f023c0cb090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc8dae0f0121ed8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:03.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f022d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:18:03.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f022d7fa640 1 -- 172.21.15.29:0/2166168286 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023c09a510 con 0x55fe941b7dd0 2024-03-11T19:18:03.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f022d7fa640 1 -- 172.21.15.29:0/2166168286 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f023c0850e0 con 0x55fe941b7dd0 2024-03-11T19:18:03.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f024e9a2740 1 -- 172.21.15.29:0/2166168286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fe93e1a7e0 con 0x55fe941b7dd0 2024-03-11T19:18:03.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f022d7fa640 1 -- 172.21.15.29:0/2166168286 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f023c098810 con 0x55fe941b7dd0 2024-03-11T19:18:03.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f024e9a2740 10 client.17412 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:03.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.199+0000 7f024e9a2740 1 -- 172.21.15.29:0/2166168286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55fe941196a0 con 0x55fe941b7dd0 2024-03-11T19:18:03.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.200+0000 7f022d7fa640 1 -- 172.21.15.29:0/2166168286 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55fe93e1a7e0 con 0x55fe941b7dd0 2024-03-11T19:18:03.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.200+0000 7f022d7fa640 1 -- 172.21.15.29:0/2166168286 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f023c018030 con 0x55fe941b7dd0 2024-03-11T19:18:03.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.200+0000 7f024e9a2740 10 client.17412 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:03.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.200+0000 7f024e9a2740 10 client.17412 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:18:03.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.200+0000 7f024e9a2740 20 client.17412 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:03.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.200+0000 7f024e9a2740 1 --2- 172.21.15.29:0/2166168286 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55fe941c8280 0x55fe941ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.200+0000 7f024e9a2740 4 client.17412 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:18:03.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.200+0000 7f024e9a2740 1 -- 172.21.15.29:0/2166168286 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe93e57720 con 0x55fe941c8280 2024-03-11T19:18:03.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.200+0000 7f0249b6c640 1 --2- 172.21.15.29:0/2166168286 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55fe941c8280 0x55fe941ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.200+0000 7f022d7fa640 10 client.17412 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:03.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.200+0000 7f0249b6c640 1 --2- 172.21.15.29:0/2166168286 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55fe941c8280 0x55fe941ca6a0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d21ffa869a74a667 server_cookie=f384b51dcd0e8099 in_seq=0 out_seq=0 2024-03-11T19:18:03.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.201+0000 7f022d7fa640 1 -- 172.21.15.29:0/2166168286 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fe93e57720 con 0x55fe941c8280 2024-03-11T19:18:03.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.201+0000 7f022d7fa640 10 client.17412 handle_command_reply: tid=0 2024-03-11T19:18:03.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.201+0000 7f024e9a2740 1 client.17412 shutdown 2024-03-11T19:18:03.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.201+0000 7f024e9a2740 1 -- 172.21.15.29:0/2166168286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 msgr2=0x55fe941c3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:03.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.201+0000 7f024e9a2740 1 --2- 172.21.15.29:0/2166168286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 0x55fe941c3a00 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f023c0a2780 tx=0x7f023c0cb090 comp rx=0 tx=0).stop 2024-03-11T19:18:03.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.201+0000 7f024e9a2740 1 -- 172.21.15.29:0/2166168286 shutdown_connections 2024-03-11T19:18:03.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.201+0000 7f024e9a2740 1 --2- 172.21.15.29:0/2166168286 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55fe941c8280 0x55fe941ca6a0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.201+0000 7f024e9a2740 1 --2- 172.21.15.29:0/2166168286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 0x55fe941c3a00 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.201+0000 7f024e9a2740 1 --2- 172.21.15.29:0/2166168286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2c60 0x55fe941bc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.201+0000 7f022d7fa640 0 client.17412 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:03.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.201+0000 7f024e9a2740 1 --2- 172.21.15.29:0/2166168286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b2270 0x55fe941bc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.201+0000 7f024e9a2740 1 -- 172.21.15.29:0/2166168286 >> 172.21.15.29:0/2166168286 conn(0x55fe941ad7e0 msgr2=0x55fe941acb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:03.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.201+0000 7f024e9a2740 1 -- 172.21.15.29:0/2166168286 shutdown_connections 2024-03-11T19:18:03.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.201+0000 7f024e9a2740 1 -- 172.21.15.29:0/2166168286 wait complete. 2024-03-11T19:18:03.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.201+0000 7f024e9a2740 20 client.17412 trim_cache size 0 max 16384 2024-03-11T19:18:03.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.213+0000 7f024e9a2740 1 Processor -- start 2024-03-11T19:18:03.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.213+0000 7f024e9a2740 1 -- start start 2024-03-11T19:18:03.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.213+0000 7f024e9a2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2270 0x55fe941adbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.213+0000 7f024e9a2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b2c60 0x55fe941c39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.213+0000 7f024e9a2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 0x55fe941bc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.213+0000 7f024e9a2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe940a0130 con 0x55fe941b7dd0 2024-03-11T19:18:03.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.213+0000 7f024e9a2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe941bb5b0 con 0x55fe941b2c60 2024-03-11T19:18:03.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.213+0000 7f024e9a2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe93e57720 con 0x55fe941b2270 2024-03-11T19:18:03.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.213+0000 7f024a36d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 0x55fe941bc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.213+0000 7f024a36d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 0x55fe941bc480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49702/0 (socket says 172.21.15.29:49702) 2024-03-11T19:18:03.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.213+0000 7f024a36d640 1 -- 172.21.15.29:0/2395817932 learned_addr learned my addr 172.21.15.29:0/2395817932 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:03.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.213+0000 7f024936b640 1 --2- 172.21.15.29:0/2395817932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b2c60 0x55fe941c39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.213+0000 7f0249b6c640 1 --2- 172.21.15.29:0/2395817932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2270 0x55fe941adbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.213+0000 7f024a36d640 1 -- 172.21.15.29:0/2395817932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2270 msgr2=0x55fe941adbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:03.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.214+0000 7f024a36d640 1 --2- 172.21.15.29:0/2395817932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2270 0x55fe941adbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.214+0000 7f024a36d640 1 -- 172.21.15.29:0/2395817932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b2c60 msgr2=0x55fe941c39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:03.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.214+0000 7f024a36d640 1 --2- 172.21.15.29:0/2395817932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b2c60 0x55fe941c39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.214+0000 7f024a36d640 1 -- 172.21.15.29:0/2395817932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe940b5c20 con 0x55fe941b7dd0 2024-03-11T19:18:03.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.214+0000 7f024936b640 1 --2- 172.21.15.29:0/2395817932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b2c60 0x55fe941c39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:18:03.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.214+0000 7f0249b6c640 1 --2- 172.21.15.29:0/2395817932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2270 0x55fe941adbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:18:03.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.214+0000 7f024a36d640 1 --2- 172.21.15.29:0/2395817932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 0x55fe941bc480 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f023c096e90 tx=0x7f023c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dd23ce1d3eef839 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:03.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.214+0000 7f022f7fe640 1 -- 172.21.15.29:0/2395817932 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023c0cd110 con 0x55fe941b7dd0 2024-03-11T19:18:03.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.214+0000 7f022f7fe640 1 -- 172.21.15.29:0/2395817932 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f023c09e6d0 con 0x55fe941b7dd0 2024-03-11T19:18:03.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.214+0000 7f022f7fe640 1 -- 172.21.15.29:0/2395817932 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023c005340 con 0x55fe941b7dd0 2024-03-11T19:18:03.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.214+0000 7f024e9a2740 1 -- 172.21.15.29:0/2395817932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 msgr2=0x55fe941bc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:03.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.214+0000 7f024e9a2740 1 --2- 172.21.15.29:0/2395817932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 0x55fe941bc480 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f023c096e90 tx=0x7f023c004a40 comp rx=0 tx=0).stop 2024-03-11T19:18:03.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.214+0000 7f024e9a2740 1 -- 172.21.15.29:0/2395817932 shutdown_connections 2024-03-11T19:18:03.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.214+0000 7f024e9a2740 1 --2- 172.21.15.29:0/2395817932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 0x55fe941bc480 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.214+0000 7f024e9a2740 1 --2- 172.21.15.29:0/2395817932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b2c60 0x55fe941c39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.214+0000 7f024e9a2740 1 --2- 172.21.15.29:0/2395817932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2270 0x55fe941adbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.214+0000 7f024e9a2740 1 -- 172.21.15.29:0/2395817932 >> 172.21.15.29:0/2395817932 conn(0x55fe941ad7e0 msgr2=0x55fe941a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:03.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.214+0000 7f024e9a2740 1 -- 172.21.15.29:0/2395817932 shutdown_connections 2024-03-11T19:18:03.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.214+0000 7f024e9a2740 1 -- 172.21.15.29:0/2395817932 wait complete. 2024-03-11T19:18:03.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.215+0000 7f024e9a2740 1 -- start start 2024-03-11T19:18:03.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.215+0000 7f024e9a2740 1 Processor -- start 2024-03-11T19:18:03.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.215+0000 7f024e9a2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2270 0x55fe941c34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.215+0000 7f024e9a2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b2c60 0x55fe941c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.215+0000 7f024e9a2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 0x55fe941c20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.215+0000 7f024e9a2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe941c3cd0 con 0x55fe941b7dd0 2024-03-11T19:18:03.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.215+0000 7f024e9a2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe941c25e0 con 0x55fe941b2c60 2024-03-11T19:18:03.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.215+0000 7f024e9a2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe941c27a0 con 0x55fe941b2270 2024-03-11T19:18:03.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.215+0000 7f024e9a2740 10 client.0 _ll_register_callbacks cb 0x7f023c096040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.215+0000 7f024a36d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 0x55fe941c20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.215+0000 7f024a36d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 0x55fe941c20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49704/0 (socket says 172.21.15.29:49704) 2024-03-11T19:18:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.215+0000 7f024a36d640 1 -- 172.21.15.29:0/349357457 learned_addr learned my addr 172.21.15.29:0/349357457 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.215+0000 7f0249b6c640 1 --2- 172.21.15.29:0/349357457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2270 0x55fe941c34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.215+0000 7f024936b640 1 --2- 172.21.15.29:0/349357457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b2c60 0x55fe941c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.215+0000 7f024a36d640 1 -- 172.21.15.29:0/349357457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2270 msgr2=0x55fe941c34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.215+0000 7f024a36d640 1 --2- 172.21.15.29:0/349357457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2270 0x55fe941c34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.215+0000 7f024a36d640 1 -- 172.21.15.29:0/349357457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b2c60 msgr2=0x55fe941c1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.215+0000 7f024a36d640 1 --2- 172.21.15.29:0/349357457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b2c60 0x55fe941c1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.215+0000 7f024a36d640 1 -- 172.21.15.29:0/349357457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe941196a0 con 0x55fe941b7dd0 2024-03-11T19:18:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.216+0000 7f024a36d640 1 --2- 172.21.15.29:0/349357457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 0x55fe941c20a0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f023c09a190 tx=0x7f023c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97810661d1926547 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.216+0000 7f022d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:18:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.216+0000 7f022d7fa640 1 -- 172.21.15.29:0/349357457 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023c0cd110 con 0x55fe941b7dd0 2024-03-11T19:18:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.216+0000 7f022d7fa640 1 -- 172.21.15.29:0/349357457 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f023c09a510 con 0x55fe941b7dd0 2024-03-11T19:18:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.216+0000 7f024e9a2740 1 -- 172.21.15.29:0/349357457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fe93ea6890 con 0x55fe941b7dd0 2024-03-11T19:18:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.216+0000 7f022d7fa640 1 -- 172.21.15.29:0/349357457 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f023c005340 con 0x55fe941b7dd0 2024-03-11T19:18:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.216+0000 7f024e9a2740 10 client.17418 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.216+0000 7f024e9a2740 1 -- 172.21.15.29:0/349357457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55fe940db8c0 con 0x55fe941b7dd0 2024-03-11T19:18:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.216+0000 7f022d7fa640 1 -- 172.21.15.29:0/349357457 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55fe93ea6890 con 0x55fe941b7dd0 2024-03-11T19:18:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.216+0000 7f022d7fa640 1 -- 172.21.15.29:0/349357457 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f023c018070 con 0x55fe941b7dd0 2024-03-11T19:18:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.216+0000 7f024e9a2740 10 client.17418 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.216+0000 7f024e9a2740 10 client.17418 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:18:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.216+0000 7f024e9a2740 20 client.17418 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.216+0000 7f024e9a2740 1 --2- 172.21.15.29:0/349357457 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55fe9421f820 0x55fe94221c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.216+0000 7f024e9a2740 4 client.17418 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:18:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.216+0000 7f024e9a2740 1 -- 172.21.15.29:0/349357457 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fe93dc6700 con 0x55fe9421f820 2024-03-11T19:18:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.217+0000 7f0249b6c640 1 --2- 172.21.15.29:0/349357457 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55fe9421f820 0x55fe94221c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.217+0000 7f0249b6c640 1 --2- 172.21.15.29:0/349357457 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55fe9421f820 0x55fe94221c40 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=339dc77028aaae9e server_cookie=5992dd830bcf7358 in_seq=0 out_seq=0 2024-03-11T19:18:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.217+0000 7f022d7fa640 10 client.17418 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:03.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.217+0000 7f022d7fa640 1 -- 172.21.15.29:0/349357457 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fe93dc6700 con 0x55fe9421f820 2024-03-11T19:18:03.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.217+0000 7f022d7fa640 10 client.17418 handle_command_reply: tid=0 2024-03-11T19:18:03.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.217+0000 7f024e9a2740 1 client.17418 shutdown 2024-03-11T19:18:03.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.217+0000 7f024e9a2740 1 -- 172.21.15.29:0/349357457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 msgr2=0x55fe941c20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:03.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.217+0000 7f024e9a2740 1 --2- 172.21.15.29:0/349357457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 0x55fe941c20a0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f023c09a190 tx=0x7f023c004ab0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.217+0000 7f024e9a2740 1 -- 172.21.15.29:0/349357457 shutdown_connections 2024-03-11T19:18:03.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.217+0000 7f024e9a2740 1 --2- 172.21.15.29:0/349357457 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55fe9421f820 0x55fe94221c40 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.217+0000 7f024e9a2740 1 --2- 172.21.15.29:0/349357457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fe941b7dd0 0x55fe941c20a0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.217+0000 7f024e9a2740 1 --2- 172.21.15.29:0/349357457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fe941b2c60 0x55fe941c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.217+0000 7f022d7fa640 0 client.17418 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:03.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.217+0000 7f024e9a2740 1 --2- 172.21.15.29:0/349357457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe941b2270 0x55fe941c34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.217+0000 7f024e9a2740 1 -- 172.21.15.29:0/349357457 >> 172.21.15.29:0/349357457 conn(0x55fe941ad7e0 msgr2=0x55fe941adbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:03.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.217+0000 7f024e9a2740 1 -- 172.21.15.29:0/349357457 shutdown_connections 2024-03-11T19:18:03.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.217+0000 7f024e9a2740 1 -- 172.21.15.29:0/349357457 wait complete. 2024-03-11T19:18:03.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.218+0000 7f024e9a2740 20 client.17418 trim_cache size 0 max 16384 2024-03-11T19:18:03.229 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:18:03.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.218+0000 7f024bdd0640 1 -- 172.21.15.29:0/3356566382 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f02300a1a30 msgr2=0x7f02300a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:03.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.218+0000 7f024bdd0640 1 --2- 172.21.15.29:0/3356566382 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f02300a1a30 0x7f02300a3ef0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0240009fd0 tx=0x7f02400099b0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.218+0000 7f024bdd0640 1 -- 172.21.15.29:0/3356566382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0244102d40 msgr2=0x7f0244073ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:03.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.218+0000 7f024bdd0640 1 --2- 172.21.15.29:0/3356566382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0244102d40 0x7f0244073ab0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f023c00ead0 tx=0x7f023c00eed0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.218+0000 7f024bdd0640 1 -- 172.21.15.29:0/3356566382 shutdown_connections 2024-03-11T19:18:03.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.218+0000 7f024bdd0640 1 --2- 172.21.15.29:0/3356566382 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f02300a1a30 0x7f02300a3ef0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.218+0000 7f024bdd0640 1 --2- 172.21.15.29:0/3356566382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0244102d40 0x7f0244073ab0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.218+0000 7f024bdd0640 1 --2- 172.21.15.29:0/3356566382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0244102380 0x7f0244073570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.218+0000 7f024bdd0640 1 --2- 172.21.15.29:0/3356566382 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0244101190 0x7f0244074f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:03.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.218+0000 7f024bdd0640 1 -- 172.21.15.29:0/3356566382 >> 172.21.15.29:0/3356566382 conn(0x7f02440fc720 msgr2=0x7f0244105750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:03.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.218+0000 7f024bdd0640 1 -- 172.21.15.29:0/3356566382 shutdown_connections 2024-03-11T19:18:03.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:03.218+0000 7f024bdd0640 1 -- 172.21.15.29:0/3356566382 wait complete. 2024-03-11T19:18:03.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:03 smithi029 ceph-mon[32439]: pgmap v598: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 64 MiB/s wr, 1.19k op/s 2024-03-11T19:18:03.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:03 smithi190 ceph-mon[36292]: pgmap v598: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 64 MiB/s wr, 1.19k op/s 2024-03-11T19:18:03.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:03 smithi170 ceph-mon[35690]: pgmap v598: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 64 MiB/s wr, 1.19k op/s 2024-03-11T19:18:04.233 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:18:04.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:18:04.234 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:18:04.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.421+0000 7f1bbce89640 1 -- 172.21.15.29:0/2303277367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bb8100990 msgr2=0x7f1bb8105230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:04.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.421+0000 7f1bbce89640 1 --2- 172.21.15.29:0/2303277367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bb8100990 0x7f1bb8105230 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0009be0 tx=0x7f1bb002bf10 comp rx=0 tx=0).stop 2024-03-11T19:18:04.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.421+0000 7f1bbce89640 1 -- 172.21.15.29:0/2303277367 shutdown_connections 2024-03-11T19:18:04.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.421+0000 7f1bbce89640 1 --2- 172.21.15.29:0/2303277367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bb8100990 0x7f1bb8105230 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.421+0000 7f1bbce89640 1 --2- 172.21.15.29:0/2303277367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb80fffd0 0x7f1bb8100450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.421+0000 7f1bbce89640 1 --2- 172.21.15.29:0/2303277367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bb80ff270 0x7f1bb80ff690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.421+0000 7f1bbce89640 1 -- 172.21.15.29:0/2303277367 >> 172.21.15.29:0/2303277367 conn(0x7f1bb80faba0 msgr2=0x7f1bb80fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:04.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.423+0000 7f1bbce89640 1 -- 172.21.15.29:0/2303277367 shutdown_connections 2024-03-11T19:18:04.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.423+0000 7f1bbce89640 1 -- 172.21.15.29:0/2303277367 wait complete. 2024-03-11T19:18:04.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.424+0000 7f1bbce89640 1 Processor -- start 2024-03-11T19:18:04.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.425+0000 7f1bbce89640 1 -- start start 2024-03-11T19:18:04.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.425+0000 7f1bbce89640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb80ff270 0x7f1bb8070720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.425+0000 7f1bbce89640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bb80fffd0 0x7f1bb8070c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.425+0000 7f1bbce89640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bb8100990 0x7f1bb8071c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.425+0000 7f1bbce89640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb8072190 con 0x7f1bb8100990 2024-03-11T19:18:04.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.425+0000 7f1bbce89640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb8072300 con 0x7f1bb80fffd0 2024-03-11T19:18:04.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.425+0000 7f1bbce89640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb80724c0 con 0x7f1bb80ff270 2024-03-11T19:18:04.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.425+0000 7f1bb7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bb8100990 0x7f1bb8071c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.425+0000 7f1bb7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bb8100990 0x7f1bb8071c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49716/0 (socket says 172.21.15.29:49716) 2024-03-11T19:18:04.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.425+0000 7f1bb77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb80ff270 0x7f1bb8070720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.425+0000 7f1bb7fff640 1 -- 172.21.15.29:0/3570154979 learned_addr learned my addr 172.21.15.29:0/3570154979 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:04.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.425+0000 7f1bb6ffd640 1 --2- 172.21.15.29:0/3570154979 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bb80fffd0 0x7f1bb8070c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.426+0000 7f1bb7fff640 1 -- 172.21.15.29:0/3570154979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb80ff270 msgr2=0x7f1bb8070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:04.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.426+0000 7f1bb7fff640 1 --2- 172.21.15.29:0/3570154979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb80ff270 0x7f1bb8070720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.426+0000 7f1bb7fff640 1 -- 172.21.15.29:0/3570154979 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bb80fffd0 msgr2=0x7f1bb8070c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:04.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.426+0000 7f1bb7fff640 1 --2- 172.21.15.29:0/3570154979 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bb80fffd0 0x7f1bb8070c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.426+0000 7f1bb7fff640 1 -- 172.21.15.29:0/3570154979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb0009840 con 0x7f1bb8100990 2024-03-11T19:18:04.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.426+0000 7f1bb7fff640 1 --2- 172.21.15.29:0/3570154979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bb8100990 0x7f1bb8071c50 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f1bb002c520 tx=0x7f1bb0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:04.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.428+0000 7f1bb4ff9640 1 -- 172.21.15.29:0/3570154979 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0004040 con 0x7f1bb8100990 2024-03-11T19:18:04.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.428+0000 7f1bbce89640 1 -- 172.21.15.29:0/3570154979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb8072740 con 0x7f1bb8100990 2024-03-11T19:18:04.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.428+0000 7f1bbce89640 1 -- 172.21.15.29:0/3570154979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb8072c50 con 0x7f1bb8100990 2024-03-11T19:18:04.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.428+0000 7f1bb4ff9640 1 -- 172.21.15.29:0/3570154979 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1bb0004650 con 0x7f1bb8100990 2024-03-11T19:18:04.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.429+0000 7f1bb4ff9640 1 -- 172.21.15.29:0/3570154979 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb003e450 con 0x7f1bb8100990 2024-03-11T19:18:04.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.432+0000 7f1bc0d98740 1 Processor -- start 2024-03-11T19:18:04.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.434+0000 7f1bb4ff9640 1 -- 172.21.15.29:0/3570154979 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1bb00041a0 con 0x7f1bb8100990 2024-03-11T19:18:04.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.435+0000 7f1bc0d98740 1 -- start start 2024-03-11T19:18:04.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.435+0000 7f1bb4ff9640 1 --2- 172.21.15.29:0/3570154979 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1ba00a1c40 0x7f1ba00a4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.435+0000 7f1bc0d98740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e2e720 0x55f1a2e2eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.435+0000 7f1bc0d98740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e2f8f0 0x55f1a2e2fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.435+0000 7f1bc0d98740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e302b0 0x55f1a2e36b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.435+0000 7f1bb4ff9640 1 -- 172.21.15.29:0/3570154979 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f1bb00f00f0 con 0x7f1bb8100990 2024-03-11T19:18:04.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.435+0000 7f1bc0d98740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1a2d8cf90 con 0x55f1a2e2e720 2024-03-11T19:18:04.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.435+0000 7f1bc0d98740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1a2d8a4c0 con 0x55f1a2e2f8f0 2024-03-11T19:18:04.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.435+0000 7f1bc0d98740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1a2d85ca0 con 0x55f1a2e302b0 2024-03-11T19:18:04.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.436+0000 7f1bb7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e302b0 0x55f1a2e36b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.436+0000 7f1bb7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e302b0 0x55f1a2e36b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:44488/0 (socket says 172.21.15.29:44488) 2024-03-11T19:18:04.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.436+0000 7f1bb7fff640 1 -- 172.21.15.29:0/3149250039 learned_addr learned my addr 172.21.15.29:0/3149250039 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:04.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.436+0000 7f1bb6ffd640 1 --2- 172.21.15.29:0/3149250039 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e2f8f0 0x55f1a2e2fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.436+0000 7f1bb6ffd640 1 --2- 172.21.15.29:0/3570154979 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1ba00a1c40 0x7f1ba00a4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.436+0000 7f1bb7fff640 1 -- 172.21.15.29:0/3149250039 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e2f8f0 msgr2=0x55f1a2e2fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:04.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.436+0000 7f1bb7fff640 1 --2- 172.21.15.29:0/3149250039 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e2f8f0 0x55f1a2e2fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.436+0000 7f1bb7fff640 1 -- 172.21.15.29:0/3149250039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e2e720 msgr2=0x55f1a2e2eb40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:18:04.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.436+0000 7f1bb7fff640 1 --2- 172.21.15.29:0/3149250039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e2e720 0x55f1a2e2eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.436+0000 7f1bb7fff640 1 -- 172.21.15.29:0/3149250039 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1a2d480b0 con 0x55f1a2e302b0 2024-03-11T19:18:04.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.437+0000 7f1bb7fff640 1 --2- 172.21.15.29:0/3149250039 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e302b0 0x55f1a2e36b40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1bb00b94c0 tx=0x7f1bb00bcba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b39a84bcfc99db0d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:04.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.437+0000 7f1b9dffb640 1 -- 172.21.15.29:0/3149250039 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb00bd760 con 0x55f1a2e302b0 2024-03-11T19:18:04.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.437+0000 7f1b9dffb640 1 -- 172.21.15.29:0/3149250039 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1bb00bd060 con 0x55f1a2e302b0 2024-03-11T19:18:04.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.437+0000 7f1b9dffb640 1 -- 172.21.15.29:0/3149250039 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0033450 con 0x55f1a2e302b0 2024-03-11T19:18:04.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.438+0000 7f1bc0d98740 1 -- 172.21.15.29:0/3149250039 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e302b0 msgr2=0x55f1a2e36b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:04.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.438+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/3149250039 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e302b0 0x55f1a2e36b40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1bb00b94c0 tx=0x7f1bb00bcba0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.438+0000 7f1bb6ffd640 1 --2- 172.21.15.29:0/3570154979 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1ba00a1c40 0x7f1ba00a4100 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8071be0 tx=0x7f1ba8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:04.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.439+0000 7f1bc0d98740 1 -- 172.21.15.29:0/3149250039 shutdown_connections 2024-03-11T19:18:04.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.439+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/3149250039 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e302b0 0x55f1a2e36b40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.439+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/3149250039 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e2f8f0 0x55f1a2e2fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.439+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/3149250039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e2e720 0x55f1a2e2eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.439+0000 7f1bc0d98740 1 -- 172.21.15.29:0/3149250039 >> 172.21.15.29:0/3149250039 conn(0x55f1a2e188f0 msgr2=0x55f1a2e18cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:04.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.441+0000 7f1bc0d98740 1 -- 172.21.15.29:0/3149250039 shutdown_connections 2024-03-11T19:18:04.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.441+0000 7f1bc0d98740 1 -- 172.21.15.29:0/3149250039 wait complete. 2024-03-11T19:18:04.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.442+0000 7f1bc0d98740 1 -- start start 2024-03-11T19:18:04.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.442+0000 7f1bc0d98740 1 Processor -- start 2024-03-11T19:18:04.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.443+0000 7f1bc0d98740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e2e720 0x55f1a2e1e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.443+0000 7f1bc0d98740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e2f8f0 0x55f1a2e1ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.443+0000 7f1bc0d98740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e302b0 0x55f1a2e25f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.443+0000 7f1bc0d98740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1a2d2af40 con 0x55f1a2e2f8f0 2024-03-11T19:18:04.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.443+0000 7f1bc0d98740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1a2a594b0 con 0x55f1a2e2e720 2024-03-11T19:18:04.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.443+0000 7f1bc0d98740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1a2d0b670 con 0x55f1a2e302b0 2024-03-11T19:18:04.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.443+0000 7f1bc0d98740 10 client.0 _ll_register_callbacks cb 0x55f1a2d78f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:04.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.444+0000 7f1bb6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e2f8f0 0x55f1a2e1ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.444+0000 7f1bb7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e302b0 0x55f1a2e25f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.444+0000 7f1bb6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e2f8f0 0x55f1a2e1ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49722/0 (socket says 172.21.15.29:49722) 2024-03-11T19:18:04.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.444+0000 7f1bb6ffd640 1 -- 172.21.15.29:0/2806925661 learned_addr learned my addr 172.21.15.29:0/2806925661 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:04.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.444+0000 7f1bb6ffd640 1 -- 172.21.15.29:0/2806925661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e302b0 msgr2=0x55f1a2e25f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:04.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.444+0000 7f1bb6ffd640 1 --2- 172.21.15.29:0/2806925661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e302b0 0x55f1a2e25f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.444+0000 7f1bb6ffd640 1 -- 172.21.15.29:0/2806925661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e2e720 msgr2=0x55f1a2e1e9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:18:04.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.444+0000 7f1bb6ffd640 1 --2- 172.21.15.29:0/2806925661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e2e720 0x55f1a2e1e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.444+0000 7f1bb6ffd640 1 -- 172.21.15.29:0/2806925661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb8072c50 con 0x55f1a2e2f8f0 2024-03-11T19:18:04.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.445+0000 7f1bb6ffd640 1 --2- 172.21.15.29:0/2806925661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e2f8f0 0x55f1a2e1ef20 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800fc30 tx=0x7f1ba8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42f2e548258b6a15 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:04.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.445+0000 7f1b96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:18:04.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.445+0000 7f1b96ffd640 1 -- 172.21.15.29:0/2806925661 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba800dcd0 con 0x55f1a2e2f8f0 2024-03-11T19:18:04.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.445+0000 7f1b96ffd640 1 -- 172.21.15.29:0/2806925661 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f1ba800de30 con 0x55f1a2e2f8f0 2024-03-11T19:18:04.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.446+0000 7f1bc0d98740 1 -- 172.21.15.29:0/2806925661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f1a2d5ff40 con 0x55f1a2e2f8f0 2024-03-11T19:18:04.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.446+0000 7f1b96ffd640 1 -- 172.21.15.29:0/2806925661 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ba80047a0 con 0x55f1a2e2f8f0 2024-03-11T19:18:04.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.446+0000 7f1bc0d98740 10 client.17427 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:04.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.446+0000 7f1bc0d98740 1 -- 172.21.15.29:0/2806925661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55f1a2d84320 con 0x55f1a2e2f8f0 2024-03-11T19:18:04.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.447+0000 7f1b96ffd640 1 -- 172.21.15.29:0/2806925661 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f1a2d5ff40 con 0x55f1a2e2f8f0 2024-03-11T19:18:04.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.447+0000 7f1b96ffd640 1 -- 172.21.15.29:0/2806925661 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1ba8021470 con 0x55f1a2e2f8f0 2024-03-11T19:18:04.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.447+0000 7f1bc0d98740 10 client.17427 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:04.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.447+0000 7f1bc0d98740 10 client.17427 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:18:04.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.447+0000 7f1bc0d98740 20 client.17427 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:04.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.447+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/2806925661 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f1a2e2acd0 0x55f1a2e2d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.447+0000 7f1bc0d98740 4 client.17427 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:18:04.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.447+0000 7f1bc0d98740 1 -- 172.21.15.29:0/2806925661 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f1a2d7c310 con 0x55f1a2e2acd0 2024-03-11T19:18:04.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.448+0000 7f1bb77fe640 1 --2- 172.21.15.29:0/2806925661 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f1a2e2acd0 0x55f1a2e2d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.450+0000 7f1bb77fe640 1 --2- 172.21.15.29:0/2806925661 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f1a2e2acd0 0x55f1a2e2d0f0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c06a4edb851873cf server_cookie=9814d0ca9cde6193 in_seq=0 out_seq=0 2024-03-11T19:18:04.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.450+0000 7f1b96ffd640 10 client.17427 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:04.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.451+0000 7f1b96ffd640 1 -- 172.21.15.29:0/2806925661 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f1a2d7c310 con 0x55f1a2e2acd0 2024-03-11T19:18:04.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.451+0000 7f1b96ffd640 10 client.17427 handle_command_reply: tid=0 2024-03-11T19:18:04.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.451+0000 7f1bc0d98740 1 client.17427 shutdown 2024-03-11T19:18:04.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.451+0000 7f1bc0d98740 1 -- 172.21.15.29:0/2806925661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e2f8f0 msgr2=0x55f1a2e1ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:04.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.451+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/2806925661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e2f8f0 0x55f1a2e1ef20 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800fc30 tx=0x7f1ba8004730 comp rx=0 tx=0).stop 2024-03-11T19:18:04.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.451+0000 7f1bc0d98740 1 -- 172.21.15.29:0/2806925661 shutdown_connections 2024-03-11T19:18:04.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.451+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/2806925661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e302b0 0x55f1a2e25f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.451+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/2806925661 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f1a2e2acd0 0x55f1a2e2d0f0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.451+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/2806925661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e2f8f0 0x55f1a2e1ef20 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.451+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/2806925661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e2e720 0x55f1a2e1e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.451+0000 7f1bc0d98740 1 -- 172.21.15.29:0/2806925661 >> 172.21.15.29:0/2806925661 conn(0x55f1a2e188f0 msgr2=0x55f1a2e17c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:04.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.451+0000 7f1b96ffd640 0 client.17427 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:04.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.451+0000 7f1bc0d98740 1 -- 172.21.15.29:0/2806925661 shutdown_connections 2024-03-11T19:18:04.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.451+0000 7f1bc0d98740 1 -- 172.21.15.29:0/2806925661 wait complete. 2024-03-11T19:18:04.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.452+0000 7f1bc0d98740 20 client.17427 trim_cache size 0 max 16384 2024-03-11T19:18:04.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.485+0000 7f1bc0d98740 1 Processor -- start 2024-03-11T19:18:04.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.485+0000 7f1bc0d98740 1 -- start start 2024-03-11T19:18:04.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.486+0000 7f1bc0d98740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e2e720 0x55f1a2e5d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.486+0000 7f1bc0d98740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e302b0 0x55f1a2e25f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.486+0000 7f1bc0d98740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e5d5f0 0x55f1a2e1e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.486+0000 7f1bc0d98740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1a2e198e0 con 0x55f1a2e302b0 2024-03-11T19:18:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.486+0000 7f1bc0d98740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1a2e291e0 con 0x55f1a2e2e720 2024-03-11T19:18:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.486+0000 7f1bb77fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e2e720 0x55f1a2e5d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.486+0000 7f1bb77fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e2e720 0x55f1a2e5d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56886/0 (socket says 172.21.15.29:56886) 2024-03-11T19:18:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.486+0000 7f1bb77fe640 1 -- 172.21.15.29:0/2556784690 learned_addr learned my addr 172.21.15.29:0/2556784690 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.486+0000 7f1bb7fff640 1 --2- 172.21.15.29:0/2556784690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e5d5f0 0x55f1a2e1e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.486+0000 7f1bc0d98740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1a2e33100 con 0x55f1a2e5d5f0 2024-03-11T19:18:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.486+0000 7f1bb6ffd640 1 --2- 172.21.15.29:0/2556784690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e302b0 0x55f1a2e25f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.486+0000 7f1bb7fff640 1 -- 172.21.15.29:0/2556784690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e2e720 msgr2=0x55f1a2e5d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.486+0000 7f1bb7fff640 1 --2- 172.21.15.29:0/2556784690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e2e720 0x55f1a2e5d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.486+0000 7f1bb7fff640 1 -- 172.21.15.29:0/2556784690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e302b0 msgr2=0x55f1a2e25f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.486+0000 7f1bb7fff640 1 --2- 172.21.15.29:0/2556784690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e302b0 0x55f1a2e25f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.486+0000 7f1bb7fff640 1 -- 172.21.15.29:0/2556784690 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1a2d20c00 con 0x55f1a2e5d5f0 2024-03-11T19:18:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.487+0000 7f1bb6ffd640 1 --2- 172.21.15.29:0/2556784690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e302b0 0x55f1a2e25f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:18:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.487+0000 7f1bb7fff640 1 --2- 172.21.15.29:0/2556784690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e5d5f0 0x55f1a2e1e9e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1bb00c52d0 tx=0x7f1bb00bccd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8108461188418575 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:04.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.487+0000 7f1b9d7fa640 1 -- 172.21.15.29:0/2556784690 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb00bd760 con 0x55f1a2e5d5f0 2024-03-11T19:18:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.487+0000 7f1b9d7fa640 1 -- 172.21.15.29:0/2556784690 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1bb00bd060 con 0x55f1a2e5d5f0 2024-03-11T19:18:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.487+0000 7f1b9d7fa640 1 -- 172.21.15.29:0/2556784690 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0033710 con 0x55f1a2e5d5f0 2024-03-11T19:18:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.488+0000 7f1bc0d98740 1 -- 172.21.15.29:0/2556784690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e5d5f0 msgr2=0x55f1a2e1e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.488+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/2556784690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e5d5f0 0x55f1a2e1e9e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1bb00c52d0 tx=0x7f1bb00bccd0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.488+0000 7f1bc0d98740 1 -- 172.21.15.29:0/2556784690 shutdown_connections 2024-03-11T19:18:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.488+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/2556784690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e5d5f0 0x55f1a2e1e9e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.489+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/2556784690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e302b0 0x55f1a2e25f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.489+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/2556784690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e2e720 0x55f1a2e5d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.489+0000 7f1bc0d98740 1 -- 172.21.15.29:0/2556784690 >> 172.21.15.29:0/2556784690 conn(0x55f1a2e188f0 msgr2=0x55f1a2e329f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:04.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.489+0000 7f1bc0d98740 1 -- 172.21.15.29:0/2556784690 shutdown_connections 2024-03-11T19:18:04.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.489+0000 7f1bc0d98740 1 -- 172.21.15.29:0/2556784690 wait complete. 2024-03-11T19:18:04.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.490+0000 7f1bc0d98740 1 -- start start 2024-03-11T19:18:04.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.491+0000 7f1bc0d98740 1 Processor -- start 2024-03-11T19:18:04.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.491+0000 7f1bc0d98740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e2e720 0x55f1a2e22000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.491+0000 7f1bc0d98740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e302b0 0x55f1a2e22540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.491+0000 7f1bc0d98740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e5d5f0 0x55f1a2e25560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.491+0000 7f1bc0d98740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1a2e25aa0 con 0x55f1a2e2e720 2024-03-11T19:18:04.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.491+0000 7f1bc0d98740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1a2e25c10 con 0x55f1a2e302b0 2024-03-11T19:18:04.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.491+0000 7f1bc0d98740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1a2e25dd0 con 0x55f1a2e5d5f0 2024-03-11T19:18:04.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.491+0000 7f1bc0d98740 10 client.0 _ll_register_callbacks cb 0x7f1bb0009480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:04.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.492+0000 7f1bb6ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e302b0 0x55f1a2e22540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.492+0000 7f1bb6ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e302b0 0x55f1a2e22540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56896/0 (socket says 172.21.15.29:56896) 2024-03-11T19:18:04.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.492+0000 7f1bb6ffd640 1 -- 172.21.15.29:0/341706756 learned_addr learned my addr 172.21.15.29:0/341706756 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:04.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.492+0000 7f1bb6ffd640 1 -- 172.21.15.29:0/341706756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e5d5f0 msgr2=0x55f1a2e25560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:18:04.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.492+0000 7f1bb6ffd640 1 --2- 172.21.15.29:0/341706756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e5d5f0 0x55f1a2e25560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.492+0000 7f1bb6ffd640 1 -- 172.21.15.29:0/341706756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e2e720 msgr2=0x55f1a2e22000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:04.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.492+0000 7f1bb6ffd640 1 --2- 172.21.15.29:0/341706756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e2e720 0x55f1a2e22000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.492+0000 7f1bb6ffd640 1 -- 172.21.15.29:0/341706756 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f1a2d84320 con 0x55f1a2e302b0 2024-03-11T19:18:04.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.492+0000 7f1bb6ffd640 1 --2- 172.21.15.29:0/341706756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e302b0 0x55f1a2e22540 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8021850 tx=0x7f1ba8012f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c62fc3f3bb382c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:04.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.492+0000 7f1b96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:18:04.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.493+0000 7f1b96ffd640 1 -- 172.21.15.29:0/341706756 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba8021c40 con 0x55f1a2e302b0 2024-03-11T19:18:04.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.493+0000 7f1b96ffd640 1 -- 172.21.15.29:0/341706756 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f1ba80047a0 con 0x55f1a2e302b0 2024-03-11T19:18:04.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.493+0000 7f1bc0d98740 1 -- 172.21.15.29:0/341706756 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f1a2dab900 con 0x55f1a2e302b0 2024-03-11T19:18:04.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.493+0000 7f1b96ffd640 1 -- 172.21.15.29:0/341706756 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ba800f810 con 0x55f1a2e302b0 2024-03-11T19:18:04.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.493+0000 7f1bc0d98740 10 client.26897 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:04.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.493+0000 7f1bc0d98740 1 -- 172.21.15.29:0/341706756 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55f1a2d480b0 con 0x55f1a2e302b0 2024-03-11T19:18:04.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.494+0000 7f1b96ffd640 1 -- 172.21.15.29:0/341706756 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1ba8004e10 con 0x55f1a2e302b0 2024-03-11T19:18:04.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.494+0000 7f1b96ffd640 1 -- 172.21.15.29:0/341706756 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1ba8035440 con 0x55f1a2e302b0 2024-03-11T19:18:04.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.494+0000 7f1bc0d98740 10 client.26897 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:04.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.494+0000 7f1bc0d98740 10 client.26897 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:18:04.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.494+0000 7f1bc0d98740 20 client.26897 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:04.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.494+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/341706756 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f1a2eadc10 0x55f1a2eb0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.494+0000 7f1bc0d98740 4 client.26897 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:18:04.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.494+0000 7f1bb77fe640 1 --2- 172.21.15.29:0/341706756 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f1a2eadc10 0x55f1a2eb0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.494+0000 7f1bc0d98740 1 -- 172.21.15.29:0/341706756 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f1a2dab900 con 0x55f1a2eadc10 2024-03-11T19:18:04.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.495+0000 7f1b96ffd640 10 client.26897 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:04.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.495+0000 7f1bb77fe640 1 --2- 172.21.15.29:0/341706756 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f1a2eadc10 0x55f1a2eb0030 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2afb094662507144 server_cookie=9c1b3e5490c7ef3e in_seq=0 out_seq=0 2024-03-11T19:18:04.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.497+0000 7f1b96ffd640 1 -- 172.21.15.29:0/341706756 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f1a2dab900 con 0x55f1a2eadc10 2024-03-11T19:18:04.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.497+0000 7f1b96ffd640 10 client.26897 handle_command_reply: tid=0 2024-03-11T19:18:04.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.497+0000 7f1bc0d98740 1 client.26897 shutdown 2024-03-11T19:18:04.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.498+0000 7f1bc0d98740 1 -- 172.21.15.29:0/341706756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e302b0 msgr2=0x55f1a2e22540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:04.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.498+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/341706756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e302b0 0x55f1a2e22540 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8021850 tx=0x7f1ba8012f80 comp rx=0 tx=0).stop 2024-03-11T19:18:04.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.498+0000 7f1bc0d98740 1 -- 172.21.15.29:0/341706756 shutdown_connections 2024-03-11T19:18:04.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.498+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/341706756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1a2e5d5f0 0x55f1a2e25560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.498+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/341706756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f1a2e302b0 0x55f1a2e22540 secure :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8021850 tx=0x7f1ba8012f80 comp rx=0 tx=0).stop 2024-03-11T19:18:04.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.498+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/341706756 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55f1a2eadc10 0x55f1a2eb0030 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.498+0000 7f1b96ffd640 0 client.26897 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:04.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.498+0000 7f1bc0d98740 1 --2- 172.21.15.29:0/341706756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1a2e2e720 0x55f1a2e22000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.498+0000 7f1bc0d98740 1 -- 172.21.15.29:0/341706756 >> 172.21.15.29:0/341706756 conn(0x55f1a2e188f0 msgr2=0x55f1a2e1e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:04.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.498+0000 7f1bc0d98740 1 -- 172.21.15.29:0/341706756 shutdown_connections 2024-03-11T19:18:04.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.498+0000 7f1bc0d98740 1 -- 172.21.15.29:0/341706756 wait complete. 2024-03-11T19:18:04.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.498+0000 7f1bc0d98740 20 client.26897 trim_cache size 0 max 16384 2024-03-11T19:18:04.504 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:18:04.504 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:18:04.504 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "ffd16b2d-43c1-4241-a9a7-d9c9206b49cc", 2024-03-11T19:18:04.504 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:18:04.504 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:18:04.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.501+0000 7f1bbce89640 1 -- 172.21.15.29:0/3570154979 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1ba00a1c40 msgr2=0x7f1ba00a4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:04.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.501+0000 7f1bbce89640 1 --2- 172.21.15.29:0/3570154979 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1ba00a1c40 0x7f1ba00a4100 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8071be0 tx=0x7f1ba8009260 comp rx=0 tx=0).stop 2024-03-11T19:18:04.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.502+0000 7f1bbce89640 1 -- 172.21.15.29:0/3570154979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bb8100990 msgr2=0x7f1bb8071c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:04.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.502+0000 7f1bbce89640 1 --2- 172.21.15.29:0/3570154979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bb8100990 0x7f1bb8071c50 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f1bb002c520 tx=0x7f1bb0003c10 comp rx=0 tx=0).stop 2024-03-11T19:18:04.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.502+0000 7f1bbce89640 1 -- 172.21.15.29:0/3570154979 shutdown_connections 2024-03-11T19:18:04.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.502+0000 7f1bbce89640 1 --2- 172.21.15.29:0/3570154979 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1ba00a1c40 0x7f1ba00a4100 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.502+0000 7f1bbce89640 1 --2- 172.21.15.29:0/3570154979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1bb8100990 0x7f1bb8071c50 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.502+0000 7f1bbce89640 1 --2- 172.21.15.29:0/3570154979 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bb80fffd0 0x7f1bb8070c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.502+0000 7f1bbce89640 1 --2- 172.21.15.29:0/3570154979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb80ff270 0x7f1bb8070720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.502+0000 7f1bbce89640 1 -- 172.21.15.29:0/3570154979 >> 172.21.15.29:0/3570154979 conn(0x7f1bb80faba0 msgr2=0x7f1bb81011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:04.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.502+0000 7f1bbce89640 1 -- 172.21.15.29:0/3570154979 shutdown_connections 2024-03-11T19:18:04.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.502+0000 7f1bbce89640 1 -- 172.21.15.29:0/3570154979 wait complete. 2024-03-11T19:18:04.523 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:18:04.523 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:18:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.724+0000 7fd8c6b8c640 1 -- 172.21.15.29:0/226047908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c010e580 msgr2=0x7fd8c0110a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.724+0000 7fd8c6b8c640 1 --2- 172.21.15.29:0/226047908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c010e580 0x7fd8c0110a10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8009be0 tx=0x7fd8b802bf10 comp rx=0 tx=0).stop 2024-03-11T19:18:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.724+0000 7fd8c6b8c640 1 -- 172.21.15.29:0/226047908 shutdown_connections 2024-03-11T19:18:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.724+0000 7fd8c6b8c640 1 --2- 172.21.15.29:0/226047908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c010e580 0x7fd8c0110a10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.724+0000 7fd8c6b8c640 1 --2- 172.21.15.29:0/226047908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c010bbc0 0x7fd8c010dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.724+0000 7fd8c6b8c640 1 --2- 172.21.15.29:0/226047908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8c0109290 0x7fd8c010b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.724+0000 7fd8c6b8c640 1 -- 172.21.15.29:0/226047908 >> 172.21.15.29:0/226047908 conn(0x7fd8c006e200 msgr2=0x7fd8c0070640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.725+0000 7fd8c6b8c640 1 -- 172.21.15.29:0/226047908 shutdown_connections 2024-03-11T19:18:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.727+0000 7fd8c6b8c640 1 -- 172.21.15.29:0/226047908 wait complete. 2024-03-11T19:18:04.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.729+0000 7fd8c6b8c640 1 Processor -- start 2024-03-11T19:18:04.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.729+0000 7fd8c6b8c640 1 -- start start 2024-03-11T19:18:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.729+0000 7fd8c6b8c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c0109290 0x7fd8c019d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.730+0000 7fd8c6b8c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8c010bbc0 0x7fd8c019dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.730+0000 7fd8c6b8c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c010e580 0x7fd8c01a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.730+0000 7fd8c6b8c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c01a5280 con 0x7fd8c010bbc0 2024-03-11T19:18:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.730+0000 7fd8c6b8c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c01a53f0 con 0x7fd8c0109290 2024-03-11T19:18:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.730+0000 7fd8c6b8c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c01a55b0 con 0x7fd8c010e580 2024-03-11T19:18:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.730+0000 7fd8c4928640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c0109290 0x7fd8c019d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.730+0000 7fd8c4928640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c0109290 0x7fd8c019d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56918/0 (socket says 172.21.15.29:56918) 2024-03-11T19:18:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.730+0000 7fd8c4928640 1 -- 172.21.15.29:0/1144574525 learned_addr learned my addr 172.21.15.29:0/1144574525 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.730+0000 7fd8c5129640 1 --2- 172.21.15.29:0/1144574525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c010e580 0x7fd8c01a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.730+0000 7fd8c4928640 1 -- 172.21.15.29:0/1144574525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c010e580 msgr2=0x7fd8c01a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:04.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.730+0000 7fd8c4928640 1 --2- 172.21.15.29:0/1144574525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c010e580 0x7fd8c01a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.730+0000 7fd8c4928640 1 -- 172.21.15.29:0/1144574525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8c010bbc0 msgr2=0x7fd8c019dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:04.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.730+0000 7fd8c4928640 1 --2- 172.21.15.29:0/1144574525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8c010bbc0 0x7fd8c019dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.730+0000 7fd8c4928640 1 -- 172.21.15.29:0/1144574525 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b8009840 con 0x7fd8c0109290 2024-03-11T19:18:04.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.731+0000 7fd8c4928640 1 --2- 172.21.15.29:0/1144574525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c0109290 0x7fd8c019d790 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400e9c0 tx=0x7fd8b400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:04.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.731+0000 7fd8bdffb640 1 -- 172.21.15.29:0/1144574525 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b400ccc0 con 0x7fd8c0109290 2024-03-11T19:18:04.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.731+0000 7fd8bdffb640 1 -- 172.21.15.29:0/1144574525 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd8b400ce20 con 0x7fd8c0109290 2024-03-11T19:18:04.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.731+0000 7fd8bdffb640 1 -- 172.21.15.29:0/1144574525 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4010430 con 0x7fd8c0109290 2024-03-11T19:18:04.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.731+0000 7fd8c6b8c640 1 -- 172.21.15.29:0/1144574525 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c01a5890 con 0x7fd8c0109290 2024-03-11T19:18:04.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.732+0000 7fd8c6b8c640 1 -- 172.21.15.29:0/1144574525 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c01a5dd0 con 0x7fd8c0109290 2024-03-11T19:18:04.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.735+0000 7fd8bdffb640 1 -- 172.21.15.29:0/1144574525 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd8b40046c0 con 0x7fd8c0109290 2024-03-11T19:18:04.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.736+0000 7fd8c93fd740 1 Processor -- start 2024-03-11T19:18:04.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.736+0000 7fd8c93fd740 1 -- start start 2024-03-11T19:18:04.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.736+0000 7fd8c93fd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c2122f2d0 0x562c2122f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.736+0000 7fd8c93fd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c2122fcc0 0x562c212347c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.736+0000 7fd8c93fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c21234e30 0x562c212372c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.736+0000 7fd8c93fd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c2119f280 con 0x562c2122f2d0 2024-03-11T19:18:04.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.736+0000 7fd8c93fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c2119c7b0 con 0x562c21234e30 2024-03-11T19:18:04.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.736+0000 7fd8c93fd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c21197f90 con 0x562c2122fcc0 2024-03-11T19:18:04.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.737+0000 7fd8c5129640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c2122f2d0 0x562c2122f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.737+0000 7fd8c4928640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c2122fcc0 0x562c212347c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.737+0000 7fd8c4928640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c2122fcc0 0x562c212347c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:44520/0 (socket says 172.21.15.29:44520) 2024-03-11T19:18:04.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.737+0000 7fd8c4928640 1 -- 172.21.15.29:0/767028688 learned_addr learned my addr 172.21.15.29:0/767028688 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:04.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.737+0000 7fd8bffff640 1 --2- 172.21.15.29:0/767028688 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c21234e30 0x562c212372c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.737+0000 7fd8c4928640 1 -- 172.21.15.29:0/767028688 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c21234e30 msgr2=0x562c212372c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:04.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.737+0000 7fd8c4928640 1 --2- 172.21.15.29:0/767028688 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c21234e30 0x562c212372c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.737+0000 7fd8c4928640 1 -- 172.21.15.29:0/767028688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c2122f2d0 msgr2=0x562c2122f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:04.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.737+0000 7fd8c4928640 1 --2- 172.21.15.29:0/767028688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c2122f2d0 0x562c2122f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.737+0000 7fd8c4928640 1 -- 172.21.15.29:0/767028688 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c21156570 con 0x562c2122fcc0 2024-03-11T19:18:04.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.738+0000 7fd8bdffb640 1 --2- 172.21.15.29:0/1144574525 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8b00a6010 0x7fd8b00a84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.738+0000 7fd8c4928640 1 --2- 172.21.15.29:0/767028688 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c2122fcc0 0x562c212347c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd8b40d53c0 tx=0x7fd8b40d54b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=550f04b745c5f6e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:04.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.738+0000 7fd8bffff640 1 --2- 172.21.15.29:0/767028688 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c21234e30 0x562c212372c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:18:04.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.738+0000 7fd8bdffb640 1 -- 172.21.15.29:0/1144574525 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd8b4014070 con 0x7fd8c0109290 2024-03-11T19:18:04.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.738+0000 7fd8aaffd640 1 -- 172.21.15.29:0/767028688 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4096850 con 0x562c2122fcc0 2024-03-11T19:18:04.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.738+0000 7fd8aaffd640 1 -- 172.21.15.29:0/767028688 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd8b409a380 con 0x562c2122fcc0 2024-03-11T19:18:04.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.738+0000 7fd8aaffd640 1 -- 172.21.15.29:0/767028688 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b409ad20 con 0x562c2122fcc0 2024-03-11T19:18:04.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.738+0000 7fd8c5129640 1 --2- 172.21.15.29:0/1144574525 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8b00a6010 0x7fd8b00a84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.739+0000 7fd8c93fd740 1 -- 172.21.15.29:0/767028688 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c2122fcc0 msgr2=0x562c212347c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:04.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.739+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/767028688 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c2122fcc0 0x562c212347c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd8b40d53c0 tx=0x7fd8b40d54b0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.739+0000 7fd8c5129640 1 --2- 172.21.15.29:0/1144574525 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8b00a6010 0x7fd8b00a84d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x562c21230520 tx=0x7fd8b8034f30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:04.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.739+0000 7fd8c93fd740 1 -- 172.21.15.29:0/767028688 shutdown_connections 2024-03-11T19:18:04.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.739+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/767028688 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c21234e30 0x562c212372c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.739+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/767028688 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c2122fcc0 0x562c212347c0 secure :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd8b40d53c0 tx=0x7fd8b40d54b0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.739+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/767028688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c2122f2d0 0x562c2122f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.739+0000 7fd8c93fd740 1 -- 172.21.15.29:0/767028688 >> 172.21.15.29:0/767028688 conn(0x562c2122a7f0 msgr2=0x562c2122abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:04.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.740+0000 7fd8c93fd740 1 -- 172.21.15.29:0/767028688 shutdown_connections 2024-03-11T19:18:04.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.740+0000 7fd8c93fd740 1 -- 172.21.15.29:0/767028688 wait complete. 2024-03-11T19:18:04.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.740+0000 7fd8c93fd740 1 -- start start 2024-03-11T19:18:04.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.740+0000 7fd8c93fd740 1 Processor -- start 2024-03-11T19:18:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.741+0000 7fd8c93fd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c2122f2d0 0x562c21239450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.741+0000 7fd8c93fd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c21234e30 0x562c21239990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.741+0000 7fd8c93fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c2123e9c0 0x562c21240db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.741+0000 7fd8c93fd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c20e6b430 con 0x562c21234e30 2024-03-11T19:18:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.741+0000 7fd8c93fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c20e27e00 con 0x562c2123e9c0 2024-03-11T19:18:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.741+0000 7fd8c93fd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c2111cfd0 con 0x562c2122f2d0 2024-03-11T19:18:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.741+0000 7fd8c93fd740 10 client.0 _ll_register_callbacks cb 0x562c21172fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.741+0000 7fd8c4928640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c2122f2d0 0x562c21239450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.741+0000 7fd8c4928640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c2122f2d0 0x562c21239450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:44528/0 (socket says 172.21.15.29:44528) 2024-03-11T19:18:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.741+0000 7fd8c5129640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c2123e9c0 0x562c21240db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.741+0000 7fd8c4928640 1 -- 172.21.15.29:0/324370988 learned_addr learned my addr 172.21.15.29:0/324370988 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.741+0000 7fd8bffff640 1 --2- 172.21.15.29:0/324370988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c21234e30 0x562c21239990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.742+0000 7fd8bffff640 1 -- 172.21.15.29:0/324370988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c2122f2d0 msgr2=0x562c21239450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.742+0000 7fd8bffff640 1 --2- 172.21.15.29:0/324370988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c2122f2d0 0x562c21239450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.742+0000 7fd8bffff640 1 -- 172.21.15.29:0/324370988 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c2123e9c0 msgr2=0x562c21240db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:04.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.742+0000 7fd8bffff640 1 --2- 172.21.15.29:0/324370988 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c2123e9c0 0x562c21240db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.742+0000 7fd8bffff640 1 -- 172.21.15.29:0/324370988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c01a5dd0 con 0x562c21234e30 2024-03-11T19:18:04.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.742+0000 7fd8c4928640 1 --2- 172.21.15.29:0/324370988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c2122f2d0 0x562c21239450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:18:04.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.742+0000 7fd8bffff640 1 --2- 172.21.15.29:0/324370988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c21234e30 0x562c21239990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd8c019eca0 tx=0x7fd8ac00c5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f2bce406d370ef5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:04.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.742+0000 7fd89ffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:18:04.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.743+0000 7fd89ffff640 1 -- 172.21.15.29:0/324370988 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac0115f0 con 0x562c21234e30 2024-03-11T19:18:04.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.743+0000 7fd89ffff640 1 -- 172.21.15.29:0/324370988 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd8ac011750 con 0x562c21234e30 2024-03-11T19:18:04.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.743+0000 7fd8c93fd740 1 -- 172.21.15.29:0/324370988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562c20e977e0 con 0x562c21234e30 2024-03-11T19:18:04.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.743+0000 7fd89ffff640 1 -- 172.21.15.29:0/324370988 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8ac011910 con 0x562c21234e30 2024-03-11T19:18:04.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.743+0000 7fd8c93fd740 10 client.17439 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:04.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.743+0000 7fd8c93fd740 1 -- 172.21.15.29:0/324370988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x562c21196610 con 0x562c21234e30 2024-03-11T19:18:04.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.744+0000 7fd89ffff640 1 -- 172.21.15.29:0/324370988 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x562c20e977e0 con 0x562c21234e30 2024-03-11T19:18:04.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.744+0000 7fd89ffff640 1 -- 172.21.15.29:0/324370988 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd8ac019bb0 con 0x562c21234e30 2024-03-11T19:18:04.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.744+0000 7fd8c93fd740 10 client.17439 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:04.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.744+0000 7fd8c93fd740 10 client.17439 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:18:04.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.744+0000 7fd8c93fd740 20 client.17439 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:04.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.744+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/324370988 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x562c2124e260 0x562c21250680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.744+0000 7fd8c93fd740 4 client.17439 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:18:04.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.744+0000 7fd8c93fd740 1 -- 172.21.15.29:0/324370988 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562c20ed4720 con 0x562c2124e260 2024-03-11T19:18:04.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.744+0000 7fd8c4928640 1 --2- 172.21.15.29:0/324370988 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x562c2124e260 0x562c21250680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.745+0000 7fd8c4928640 1 --2- 172.21.15.29:0/324370988 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x562c2124e260 0x562c21250680 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e58fc0e4eb6b5a9 server_cookie=98333f8338530e2d in_seq=0 out_seq=0 2024-03-11T19:18:04.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.745+0000 7fd89ffff640 10 client.17439 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:04.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.746+0000 7fd89ffff640 1 -- 172.21.15.29:0/324370988 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562c20ed4720 con 0x562c2124e260 2024-03-11T19:18:04.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.746+0000 7fd89ffff640 10 client.17439 handle_command_reply: tid=0 2024-03-11T19:18:04.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.746+0000 7fd8c93fd740 1 client.17439 shutdown 2024-03-11T19:18:04.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.746+0000 7fd8c93fd740 1 -- 172.21.15.29:0/324370988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c21234e30 msgr2=0x562c21239990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:04.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.746+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/324370988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c21234e30 0x562c21239990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd8c019eca0 tx=0x7fd8ac00c5e0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.746+0000 7fd8c93fd740 1 -- 172.21.15.29:0/324370988 shutdown_connections 2024-03-11T19:18:04.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.746+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/324370988 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c2123e9c0 0x562c21240db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.746+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/324370988 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x562c2124e260 0x562c21250680 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.746+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/324370988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c21234e30 0x562c21239990 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.746+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/324370988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c2122f2d0 0x562c21239450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.746+0000 7fd8c93fd740 1 -- 172.21.15.29:0/324370988 >> 172.21.15.29:0/324370988 conn(0x562c2122a7f0 msgr2=0x562c21229a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:04.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.746+0000 7fd89ffff640 0 client.17439 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:04.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.746+0000 7fd8c93fd740 1 -- 172.21.15.29:0/324370988 shutdown_connections 2024-03-11T19:18:04.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.746+0000 7fd8c93fd740 1 -- 172.21.15.29:0/324370988 wait complete. 2024-03-11T19:18:04.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.746+0000 7fd8c93fd740 20 client.17439 trim_cache size 0 max 16384 2024-03-11T19:18:04.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.775+0000 7fd8c93fd740 1 Processor -- start 2024-03-11T19:18:04.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.775+0000 7fd8c93fd740 1 -- start start 2024-03-11T19:18:04.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.775+0000 7fd8c93fd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c2122f2d0 0x562c21254fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.775+0000 7fd8c93fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c21234e30 0x562c2125e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.775+0000 7fd8c93fd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c2123e9c0 0x562c2125ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.775+0000 7fd8c93fd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c20e97810 con 0x562c2122f2d0 2024-03-11T19:18:04.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.775+0000 7fd8c93fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c2122a3a0 con 0x562c21234e30 2024-03-11T19:18:04.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.775+0000 7fd8c93fd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c212300a0 con 0x562c2123e9c0 2024-03-11T19:18:04.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.775+0000 7fd8bffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c21234e30 0x562c2125e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.775+0000 7fd8bffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c21234e30 0x562c2125e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56946/0 (socket says 172.21.15.29:56946) 2024-03-11T19:18:04.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.775+0000 7fd8bffff640 1 -- 172.21.15.29:0/2834649966 learned_addr learned my addr 172.21.15.29:0/2834649966 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:04.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.776+0000 7fd8c5129640 1 --2- 172.21.15.29:0/2834649966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c2123e9c0 0x562c2125ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.776+0000 7fd8bffff640 1 -- 172.21.15.29:0/2834649966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c2123e9c0 msgr2=0x562c2125ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:04.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.776+0000 7fd8bffff640 1 --2- 172.21.15.29:0/2834649966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c2123e9c0 0x562c2125ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.776+0000 7fd8bffff640 1 -- 172.21.15.29:0/2834649966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c2122f2d0 msgr2=0x562c21254fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:18:04.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.776+0000 7fd8bffff640 1 --2- 172.21.15.29:0/2834649966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c2122f2d0 0x562c21254fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.776+0000 7fd8bffff640 1 -- 172.21.15.29:0/2834649966 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c21132ac0 con 0x562c21234e30 2024-03-11T19:18:04.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.777+0000 7fd8bffff640 1 --2- 172.21.15.29:0/2834649966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c21234e30 0x562c2125e520 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac008010 tx=0x7fd8ac00cb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1051285b3f1ccf57 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:04.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.777+0000 7fd8aa7fc640 1 -- 172.21.15.29:0/2834649966 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac010490 con 0x562c21234e30 2024-03-11T19:18:04.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.777+0000 7fd8aa7fc640 1 -- 172.21.15.29:0/2834649966 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd8ac0105f0 con 0x562c21234e30 2024-03-11T19:18:04.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.777+0000 7fd8aa7fc640 1 -- 172.21.15.29:0/2834649966 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac0199e0 con 0x562c21234e30 2024-03-11T19:18:04.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.778+0000 7fd8c93fd740 1 -- 172.21.15.29:0/2834649966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c21234e30 msgr2=0x562c2125e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:04.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.778+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/2834649966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c21234e30 0x562c2125e520 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac008010 tx=0x7fd8ac00cb60 comp rx=0 tx=0).stop 2024-03-11T19:18:04.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.778+0000 7fd8c93fd740 1 -- 172.21.15.29:0/2834649966 shutdown_connections 2024-03-11T19:18:04.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.778+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/2834649966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c2123e9c0 0x562c2125ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.778+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/2834649966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c21234e30 0x562c2125e520 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.778+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/2834649966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c2122f2d0 0x562c21254fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.779+0000 7fd8c93fd740 1 -- 172.21.15.29:0/2834649966 >> 172.21.15.29:0/2834649966 conn(0x562c2122a7f0 msgr2=0x562c21225840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:04.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.779+0000 7fd8c93fd740 1 -- 172.21.15.29:0/2834649966 shutdown_connections 2024-03-11T19:18:04.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.779+0000 7fd8c93fd740 1 -- 172.21.15.29:0/2834649966 wait complete. 2024-03-11T19:18:04.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.779+0000 7fd8c93fd740 1 -- start start 2024-03-11T19:18:04.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.779+0000 7fd8c93fd740 1 Processor -- start 2024-03-11T19:18:04.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.780+0000 7fd8c93fd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c2122f2d0 0x562c2125e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.780+0000 7fd8c93fd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c21234e30 0x562c21245430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.780+0000 7fd8c93fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c2123e9c0 0x562c212479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.780+0000 7fd8c93fd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c212480c0 con 0x562c2122f2d0 2024-03-11T19:18:04.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.780+0000 7fd8c93fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c21248230 con 0x562c2123e9c0 2024-03-11T19:18:04.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.780+0000 7fd8c93fd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c212483f0 con 0x562c21234e30 2024-03-11T19:18:04.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.780+0000 7fd8c93fd740 10 client.0 _ll_register_callbacks cb 0x7fd8b40d0cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:04.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.780+0000 7fd8c5129640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c2123e9c0 0x562c212479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.780+0000 7fd8c5129640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c2123e9c0 0x562c212479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56956/0 (socket says 172.21.15.29:56956) 2024-03-11T19:18:04.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.780+0000 7fd8c5129640 1 -- 172.21.15.29:0/3297440003 learned_addr learned my addr 172.21.15.29:0/3297440003 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:04.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.781+0000 7fd8c4928640 1 --2- 172.21.15.29:0/3297440003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c2122f2d0 0x562c2125e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.781+0000 7fd8c5129640 1 -- 172.21.15.29:0/3297440003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c21234e30 msgr2=0x562c21245430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:04.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.781+0000 7fd8c5129640 1 --2- 172.21.15.29:0/3297440003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c21234e30 0x562c21245430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.781+0000 7fd8c5129640 1 -- 172.21.15.29:0/3297440003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c2122f2d0 msgr2=0x562c2125e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:04.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.781+0000 7fd8c5129640 1 --2- 172.21.15.29:0/3297440003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c2122f2d0 0x562c2125e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.781+0000 7fd8c5129640 1 -- 172.21.15.29:0/3297440003 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562c21196610 con 0x562c2123e9c0 2024-03-11T19:18:04.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.781+0000 7fd8c5129640 1 --2- 172.21.15.29:0/3297440003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c2123e9c0 0x562c212479c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd8b802c520 tx=0x7fd8b8009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bfa07f9c85d03ae server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:04.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.782+0000 7fd89ffff640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:18:04.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.782+0000 7fd89ffff640 1 -- 172.21.15.29:0/3297440003 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8051070 con 0x562c2123e9c0 2024-03-11T19:18:04.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.782+0000 7fd89ffff640 1 -- 172.21.15.29:0/3297440003 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd8b8034430 con 0x562c2123e9c0 2024-03-11T19:18:04.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.782+0000 7fd8c93fd740 1 -- 172.21.15.29:0/3297440003 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd8b409a380 con 0x562c2123e9c0 2024-03-11T19:18:04.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.782+0000 7fd89ffff640 1 -- 172.21.15.29:0/3297440003 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8b80345f0 con 0x562c2123e9c0 2024-03-11T19:18:04.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.782+0000 7fd8c93fd740 10 client.26912 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:04.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.782+0000 7fd8c93fd740 1 -- 172.21.15.29:0/3297440003 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x562c211586a0 con 0x562c2123e9c0 2024-03-11T19:18:04.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.783+0000 7fd89ffff640 1 -- 172.21.15.29:0/3297440003 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd8b409a380 con 0x562c2123e9c0 2024-03-11T19:18:04.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.783+0000 7fd89ffff640 1 -- 172.21.15.29:0/3297440003 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd8b805d070 con 0x562c2123e9c0 2024-03-11T19:18:04.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.783+0000 7fd8c93fd740 10 client.26912 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:04.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.783+0000 7fd8c93fd740 10 client.26912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:18:04.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.783+0000 7fd8c93fd740 20 client.26912 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:04.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.783+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/3297440003 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x562c212ae500 0x562c212b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:04.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.783+0000 7fd8c93fd740 4 client.26912 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:18:04.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.783+0000 7fd8c93fd740 1 -- 172.21.15.29:0/3297440003 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562c20f23890 con 0x562c212ae500 2024-03-11T19:18:04.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.785+0000 7fd8c4928640 1 --2- 172.21.15.29:0/3297440003 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x562c212ae500 0x562c212b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:04.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.785+0000 7fd89ffff640 10 client.26912 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:04.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.785+0000 7fd8c4928640 1 --2- 172.21.15.29:0/3297440003 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x562c212ae500 0x562c212b08f0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf44340130569e51 server_cookie=2ea163df90c2f03c in_seq=0 out_seq=0 2024-03-11T19:18:04.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.786+0000 7fd89ffff640 1 -- 172.21.15.29:0/3297440003 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x562c20f23890 con 0x562c212ae500 2024-03-11T19:18:04.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.786+0000 7fd89ffff640 10 client.26912 handle_command_reply: tid=0 2024-03-11T19:18:04.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.786+0000 7fd8c93fd740 1 client.26912 shutdown 2024-03-11T19:18:04.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.786+0000 7fd8c93fd740 1 -- 172.21.15.29:0/3297440003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c2123e9c0 msgr2=0x562c212479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:04.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.786+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/3297440003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c2123e9c0 0x562c212479c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd8b802c520 tx=0x7fd8b8009500 comp rx=0 tx=0).stop 2024-03-11T19:18:04.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.787+0000 7fd8c93fd740 1 -- 172.21.15.29:0/3297440003 shutdown_connections 2024-03-11T19:18:04.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.787+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/3297440003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562c2123e9c0 0x562c212479c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.787+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/3297440003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c21234e30 0x562c21245430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.787+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/3297440003 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x562c212ae500 0x562c212b08f0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.787+0000 7fd8c93fd740 1 --2- 172.21.15.29:0/3297440003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c2122f2d0 0x562c2125e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.787+0000 7fd8c93fd740 1 -- 172.21.15.29:0/3297440003 >> 172.21.15.29:0/3297440003 conn(0x562c2122a7f0 msgr2=0x562c21254fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:04.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.787+0000 7fd89ffff640 0 client.26912 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:04.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.787+0000 7fd8c93fd740 1 -- 172.21.15.29:0/3297440003 shutdown_connections 2024-03-11T19:18:04.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.787+0000 7fd8c93fd740 1 -- 172.21.15.29:0/3297440003 wait complete. 2024-03-11T19:18:04.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.788+0000 7fd8c93fd740 20 client.26912 trim_cache size 0 max 16384 2024-03-11T19:18:04.792 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:18:04.792 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:18:04.792 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:18:04.792 INFO:teuthology.orchestra.run.smithi029.stdout: "ffd16b2d-43c1-4241-a9a7-d9c9206b49cc": { 2024-03-11T19:18:04.792 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:18:04.792 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "ffd16b2d-43c1-4241-a9a7-d9c9206b49cc", 2024-03-11T19:18:04.792 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:18:04.792 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:18:04.792 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:18:04.792 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:18:04.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.789+0000 7fd8c6b8c640 1 -- 172.21.15.29:0/1144574525 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8b00a6010 msgr2=0x7fd8b00a84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:04.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.789+0000 7fd8c6b8c640 1 --2- 172.21.15.29:0/1144574525 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8b00a6010 0x7fd8b00a84d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x562c21230520 tx=0x7fd8b8034f30 comp rx=0 tx=0).stop 2024-03-11T19:18:04.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.789+0000 7fd8c6b8c640 1 -- 172.21.15.29:0/1144574525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c0109290 msgr2=0x7fd8c019d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:04.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.789+0000 7fd8c6b8c640 1 --2- 172.21.15.29:0/1144574525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c0109290 0x7fd8c019d790 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400e9c0 tx=0x7fd8b400ee80 comp rx=0 tx=0).stop 2024-03-11T19:18:04.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.789+0000 7fd8c6b8c640 1 -- 172.21.15.29:0/1144574525 shutdown_connections 2024-03-11T19:18:04.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.789+0000 7fd8c6b8c640 1 --2- 172.21.15.29:0/1144574525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c010e580 0x7fd8c01a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.789+0000 7fd8c6b8c640 1 --2- 172.21.15.29:0/1144574525 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8b00a6010 0x7fd8b00a84d0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.789+0000 7fd8c6b8c640 1 --2- 172.21.15.29:0/1144574525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8c010bbc0 0x7fd8c019dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.789+0000 7fd8c6b8c640 1 --2- 172.21.15.29:0/1144574525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c0109290 0x7fd8c019d790 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:04.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.789+0000 7fd8c6b8c640 1 -- 172.21.15.29:0/1144574525 >> 172.21.15.29:0/1144574525 conn(0x7fd8c006e200 msgr2=0x7fd8c010a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:04.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.790+0000 7fd8c6b8c640 1 -- 172.21.15.29:0/1144574525 shutdown_connections 2024-03-11T19:18:04.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:04.791+0000 7fd8c6b8c640 1 -- 172.21.15.29:0/1144574525 wait complete. 2024-03-11T19:18:04.814 INFO:tasks.cephfs.filesystem:scrub status for tag:ffd16b2d-43c1-4241-a9a7-d9c9206b49cc - {'path': '/', 'tag': 'ffd16b2d-43c1-4241-a9a7-d9c9206b49cc', 'options': 'recursive,force'} 2024-03-11T19:18:05.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:05 smithi190 ceph-mon[36292]: pgmap v599: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 59 MiB/s wr, 1.10k op/s 2024-03-11T19:18:05.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:05 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:18:05.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:05 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:18:05.513 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:05 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:18:05.513 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:05 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:18:05.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:05 smithi170 ceph-mon[35690]: pgmap v599: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 59 MiB/s wr, 1.10k op/s 2024-03-11T19:18:05.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:05 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:18:05.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:05 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:18:05.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:05 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:18:05.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:05 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:18:05.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:05 smithi029 ceph-mon[32439]: pgmap v599: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 59 MiB/s wr, 1.10k op/s 2024-03-11T19:18:05.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:05 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:18:05.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:05 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:18:05.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:05 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:18:05.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:05 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:18:06.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:06 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:18:06.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:06 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:18:06.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:06 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:18:07.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:07 smithi190 ceph-mon[36292]: pgmap v600: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 63 MiB/s wr, 1.18k op/s 2024-03-11T19:18:07.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:07 smithi170 ceph-mon[35690]: pgmap v600: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 63 MiB/s wr, 1.18k op/s 2024-03-11T19:18:07.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:07 smithi029 ceph-mon[32439]: pgmap v600: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 63 MiB/s wr, 1.18k op/s 2024-03-11T19:18:09.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:09 smithi190 ceph-mon[36292]: pgmap v601: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 55 MiB/s wr, 1.04k op/s 2024-03-11T19:18:09.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:09 smithi170 ceph-mon[35690]: pgmap v601: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 55 MiB/s wr, 1.04k op/s 2024-03-11T19:18:09.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:09 smithi029 ceph-mon[32439]: pgmap v601: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 55 MiB/s wr, 1.04k op/s 2024-03-11T19:18:11.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:11 smithi170 ceph-mon[35690]: pgmap v602: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 61 MiB/s wr, 1.14k op/s 2024-03-11T19:18:11.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:11 smithi029 ceph-mon[32439]: pgmap v602: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 61 MiB/s wr, 1.14k op/s 2024-03-11T19:18:11.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:11 smithi190 ceph-mon[36292]: pgmap v602: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 61 MiB/s wr, 1.14k op/s 2024-03-11T19:18:13.599 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:13 smithi170 ceph-mon[35690]: pgmap v603: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 60 MiB/s wr, 1.13k op/s 2024-03-11T19:18:14.511 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:13 smithi190 ceph-mon[36292]: pgmap v603: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 60 MiB/s wr, 1.13k op/s 2024-03-11T19:18:14.511 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:13 smithi029 ceph-mon[32439]: pgmap v603: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 60 MiB/s wr, 1.13k op/s 2024-03-11T19:18:15.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:15 smithi170 ceph-mon[35690]: pgmap v604: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 56 MiB/s wr, 1.05k op/s 2024-03-11T19:18:15.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:15 smithi029 ceph-mon[32439]: pgmap v604: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 56 MiB/s wr, 1.05k op/s 2024-03-11T19:18:15.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:15 smithi190 ceph-mon[36292]: pgmap v604: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 56 MiB/s wr, 1.05k op/s 2024-03-11T19:18:17.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:17 smithi170 ceph-mon[35690]: pgmap v605: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 62 MiB/s wr, 1.16k op/s 2024-03-11T19:18:17.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:17 smithi029 ceph-mon[32439]: pgmap v605: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 62 MiB/s wr, 1.16k op/s 2024-03-11T19:18:17.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:17 smithi190 ceph-mon[36292]: pgmap v605: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 62 MiB/s wr, 1.16k op/s 2024-03-11T19:18:19.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:19 smithi170 ceph-mon[35690]: pgmap v606: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 55 MiB/s wr, 1.03k op/s 2024-03-11T19:18:19.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:19 smithi029 ceph-mon[32439]: pgmap v606: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 55 MiB/s wr, 1.03k op/s 2024-03-11T19:18:19.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:19 smithi190 ceph-mon[36292]: pgmap v606: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 55 MiB/s wr, 1.03k op/s 2024-03-11T19:18:21.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:21 smithi170 ceph-mon[35690]: pgmap v607: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 61 MiB/s wr, 1.14k op/s 2024-03-11T19:18:21.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:21 smithi029 ceph-mon[32439]: pgmap v607: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 61 MiB/s wr, 1.14k op/s 2024-03-11T19:18:21.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:21 smithi190 ceph-mon[36292]: pgmap v607: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 61 MiB/s wr, 1.14k op/s 2024-03-11T19:18:23.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:23 smithi170 ceph-mon[35690]: pgmap v608: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 60 MiB/s wr, 1.12k op/s 2024-03-11T19:18:23.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:23 smithi029 ceph-mon[32439]: pgmap v608: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 60 MiB/s wr, 1.12k op/s 2024-03-11T19:18:23.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:23 smithi190 ceph-mon[36292]: pgmap v608: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 60 MiB/s wr, 1.12k op/s 2024-03-11T19:18:25.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:25 smithi170 ceph-mon[35690]: pgmap v609: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 56 MiB/s wr, 1.05k op/s 2024-03-11T19:18:25.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:25 smithi029 ceph-mon[32439]: pgmap v609: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 56 MiB/s wr, 1.05k op/s 2024-03-11T19:18:25.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:25 smithi190 ceph-mon[36292]: pgmap v609: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 56 MiB/s wr, 1.05k op/s 2024-03-11T19:18:27.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:27 smithi170 ceph-mon[35690]: pgmap v610: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 62 MiB/s wr, 1.17k op/s 2024-03-11T19:18:27.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:27 smithi029 ceph-mon[32439]: pgmap v610: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 62 MiB/s wr, 1.17k op/s 2024-03-11T19:18:27.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:27 smithi190 ceph-mon[36292]: pgmap v610: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 62 MiB/s wr, 1.17k op/s 2024-03-11T19:18:29.564 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:29 smithi190 ceph-mon[36292]: pgmap v611: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 56 MiB/s wr, 1.04k op/s 2024-03-11T19:18:29.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:29 smithi170 ceph-mon[35690]: pgmap v611: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 56 MiB/s wr, 1.04k op/s 2024-03-11T19:18:29.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:29 smithi029 ceph-mon[32439]: pgmap v611: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 56 MiB/s wr, 1.04k op/s 2024-03-11T19:18:31.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:31 smithi170 ceph-mon[35690]: pgmap v612: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 62 MiB/s wr, 1.17k op/s 2024-03-11T19:18:31.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:31 smithi029 ceph-mon[32439]: pgmap v612: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 62 MiB/s wr, 1.17k op/s 2024-03-11T19:18:31.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:31 smithi190 ceph-mon[36292]: pgmap v612: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 62 MiB/s wr, 1.17k op/s 2024-03-11T19:18:33.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:33 smithi170 ceph-mon[35690]: pgmap v613: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 63 MiB/s wr, 1.19k op/s 2024-03-11T19:18:33.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:33 smithi029 ceph-mon[32439]: pgmap v613: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 63 MiB/s wr, 1.19k op/s 2024-03-11T19:18:33.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:33 smithi190 ceph-mon[36292]: pgmap v613: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 63 MiB/s wr, 1.19k op/s 2024-03-11T19:18:34.815 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:18:34.816 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:18:34.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.993+0000 7f93ed6c5640 1 -- 172.21.15.29:0/1967093126 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93e009d550 msgr2=0x7f93e009d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:34.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.993+0000 7f93ed6c5640 1 --2- 172.21.15.29:0/1967093126 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93e009d550 0x7f93e009d9d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f93d4009900 tx=0x7f93d402bc30 comp rx=0 tx=0).stop 2024-03-11T19:18:34.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.994+0000 7f93ed6c5640 1 -- 172.21.15.29:0/1967093126 shutdown_connections 2024-03-11T19:18:34.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.994+0000 7f93ed6c5640 1 --2- 172.21.15.29:0/1967093126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93e009df10 0x7f93e00a47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:34.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.994+0000 7f93ed6c5640 1 --2- 172.21.15.29:0/1967093126 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93e009d550 0x7f93e009d9d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:34.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.994+0000 7f93ed6c5640 1 --2- 172.21.15.29:0/1967093126 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93e009c360 0x7f93e009c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:34.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.994+0000 7f93ed6c5640 1 -- 172.21.15.29:0/1967093126 >> 172.21.15.29:0/1967093126 conn(0x7f93e000b4c0 msgr2=0x7f93e009a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:34.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.994+0000 7f93ed6c5640 1 -- 172.21.15.29:0/1967093126 shutdown_connections 2024-03-11T19:18:34.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.994+0000 7f93ed6c5640 1 -- 172.21.15.29:0/1967093126 wait complete. 2024-03-11T19:18:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.995+0000 7f93ed6c5640 1 Processor -- start 2024-03-11T19:18:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.995+0000 7f93ed6c5640 1 -- start start 2024-03-11T19:18:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.995+0000 7f93ed6c5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93e009c360 0x7f93e01322f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.995+0000 7f93ed6c5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93e009d550 0x7f93e0132830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.995+0000 7f93ed6c5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93e009df10 0x7f93e01398a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.996+0000 7f93ed6c5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e0139de0 con 0x7f93e009c360 2024-03-11T19:18:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.996+0000 7f93ed6c5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e0139f50 con 0x7f93e009d550 2024-03-11T19:18:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.996+0000 7f93e77fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93e009d550 0x7f93e0132830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.996+0000 7f93e77fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93e009d550 0x7f93e0132830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44910/0 (socket says 172.21.15.29:44910) 2024-03-11T19:18:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.996+0000 7f93e77fe640 1 -- 172.21.15.29:0/4133444877 learned_addr learned my addr 172.21.15.29:0/4133444877 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.996+0000 7f93ecec4640 1 --2- 172.21.15.29:0/4133444877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93e009df10 0x7f93e01398a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.996+0000 7f93ed6c5640 1 -- 172.21.15.29:0/4133444877 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e013a110 con 0x7f93e009df10 2024-03-11T19:18:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.996+0000 7f93e7fff640 1 --2- 172.21.15.29:0/4133444877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93e009c360 0x7f93e01322f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.996+0000 7f93e77fe640 1 -- 172.21.15.29:0/4133444877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93e009df10 msgr2=0x7f93e01398a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.996+0000 7f93e77fe640 1 --2- 172.21.15.29:0/4133444877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93e009df10 0x7f93e01398a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.996+0000 7f93e77fe640 1 -- 172.21.15.29:0/4133444877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93e009c360 msgr2=0x7f93e01322f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.996+0000 7f93e77fe640 1 --2- 172.21.15.29:0/4133444877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93e009c360 0x7f93e01322f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.996+0000 7f93e77fe640 1 -- 172.21.15.29:0/4133444877 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d4009560 con 0x7f93e009d550 2024-03-11T19:18:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.996+0000 7f93ecec4640 1 --2- 172.21.15.29:0/4133444877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93e009df10 0x7f93e01398a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:18:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.996+0000 7f93e7fff640 1 --2- 172.21.15.29:0/4133444877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93e009c360 0x7f93e01322f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:18:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.996+0000 7f93e77fe640 1 --2- 172.21.15.29:0/4133444877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93e009d550 0x7f93e0132830 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f93d4006010 tx=0x7f93d4004370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.997+0000 7f93e57fa640 1 -- 172.21.15.29:0/4133444877 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d4004a50 con 0x7f93e009d550 2024-03-11T19:18:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.997+0000 7f93e57fa640 1 -- 172.21.15.29:0/4133444877 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f93d402de90 con 0x7f93e009d550 2024-03-11T19:18:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.997+0000 7f93ed6c5640 1 -- 172.21.15.29:0/4133444877 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e013a390 con 0x7f93e009d550 2024-03-11T19:18:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.997+0000 7f93e57fa640 1 -- 172.21.15.29:0/4133444877 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d40052f0 con 0x7f93e009d550 2024-03-11T19:18:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:34.997+0000 7f93ed6c5640 1 -- 172.21.15.29:0/4133444877 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e013a830 con 0x7f93e009d550 2024-03-11T19:18:35.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.001+0000 7f93e57fa640 1 -- 172.21.15.29:0/4133444877 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f93d4005450 con 0x7f93e009d550 2024-03-11T19:18:35.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.001+0000 7f93f15cf740 1 Processor -- start 2024-03-11T19:18:35.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.001+0000 7f93f15cf740 1 -- start start 2024-03-11T19:18:35.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.002+0000 7f93f15cf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df1914f0 0x5618df191910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.002+0000 7f93f15cf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df191ee0 0x5618df1969e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.002+0000 7f93f15cf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df197050 0x5618df1994e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.002+0000 7f93f15cf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618df101280 con 0x5618df191ee0 2024-03-11T19:18:35.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.002+0000 7f93f15cf740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618df0fe7b0 con 0x5618df197050 2024-03-11T19:18:35.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.002+0000 7f93f15cf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618df0f9f90 con 0x5618df1914f0 2024-03-11T19:18:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.002+0000 7f93e77fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df197050 0x5618df1994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.002+0000 7f93e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df191ee0 0x5618df1969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.002+0000 7f93e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df191ee0 0x5618df1969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48352/0 (socket says 172.21.15.29:48352) 2024-03-11T19:18:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.002+0000 7f93e7fff640 1 -- 172.21.15.29:0/2356013763 learned_addr learned my addr 172.21.15.29:0/2356013763 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.002+0000 7f93ecec4640 1 --2- 172.21.15.29:0/2356013763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df1914f0 0x5618df191910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.003+0000 7f93e77fe640 1 -- 172.21.15.29:0/2356013763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df1914f0 msgr2=0x5618df191910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.003+0000 7f93e77fe640 1 --2- 172.21.15.29:0/2356013763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df1914f0 0x5618df191910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.003+0000 7f93e77fe640 1 -- 172.21.15.29:0/2356013763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df191ee0 msgr2=0x5618df1969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.003+0000 7f93e77fe640 1 --2- 172.21.15.29:0/2356013763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df191ee0 0x5618df1969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.003+0000 7f93e77fe640 1 -- 172.21.15.29:0/2356013763 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5618df0b8570 con 0x5618df197050 2024-03-11T19:18:35.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.003+0000 7f93e57fa640 1 --2- 172.21.15.29:0/4133444877 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f93d80a1b30 0x7f93d80a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.003+0000 7f93e77fe640 1 --2- 172.21.15.29:0/2356013763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df197050 0x5618df1994e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f93d40eced0 tx=0x7f93d40f73d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb5ebe9f24f185fe server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:35.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.003+0000 7f93ecec4640 1 --2- 172.21.15.29:0/4133444877 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f93d80a1b30 0x7f93d80a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.003+0000 7f93d27fc640 1 -- 172.21.15.29:0/2356013763 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d40f10d0 con 0x5618df197050 2024-03-11T19:18:35.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.004+0000 7f93d27fc640 1 -- 172.21.15.29:0/2356013763 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f93d40b8610 con 0x5618df197050 2024-03-11T19:18:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.004+0000 7f93e57fa640 1 -- 172.21.15.29:0/4133444877 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f93d40ef400 con 0x7f93e009d550 2024-03-11T19:18:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.004+0000 7f93d27fc640 1 -- 172.21.15.29:0/2356013763 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d40bb7c0 con 0x5618df197050 2024-03-11T19:18:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.005+0000 7f93ecec4640 1 --2- 172.21.15.29:0/4133444877 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f93d80a1b30 0x7f93d80a3ff0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f93e80668b0 tx=0x7f93e8078040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.005+0000 7f93f15cf740 1 -- 172.21.15.29:0/2356013763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df197050 msgr2=0x5618df1994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.005+0000 7f93f15cf740 1 --2- 172.21.15.29:0/2356013763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df197050 0x5618df1994e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f93d40eced0 tx=0x7f93d40f73d0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.005+0000 7f93f15cf740 1 -- 172.21.15.29:0/2356013763 shutdown_connections 2024-03-11T19:18:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.005+0000 7f93f15cf740 1 --2- 172.21.15.29:0/2356013763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df197050 0x5618df1994e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.005+0000 7f93f15cf740 1 --2- 172.21.15.29:0/2356013763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df191ee0 0x5618df1969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.005+0000 7f93f15cf740 1 --2- 172.21.15.29:0/2356013763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df1914f0 0x5618df191910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.005+0000 7f93f15cf740 1 -- 172.21.15.29:0/2356013763 >> 172.21.15.29:0/2356013763 conn(0x5618df18ca60 msgr2=0x5618df18ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.005+0000 7f93f15cf740 1 -- 172.21.15.29:0/2356013763 shutdown_connections 2024-03-11T19:18:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.005+0000 7f93f15cf740 1 -- 172.21.15.29:0/2356013763 wait complete. 2024-03-11T19:18:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.005+0000 7f93f15cf740 1 -- start start 2024-03-11T19:18:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.006+0000 7f93f15cf740 1 Processor -- start 2024-03-11T19:18:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.006+0000 7f93f15cf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df1914f0 0x5618df19b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.006+0000 7f93f15cf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df191ee0 0x5618df19b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.006+0000 7f93f15cf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df197050 0x5618df1a27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.006+0000 7f93f15cf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618dedcd430 con 0x5618df191ee0 2024-03-11T19:18:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.006+0000 7f93f15cf740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618ded89e00 con 0x5618df197050 2024-03-11T19:18:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.006+0000 7f93f15cf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618df07efd0 con 0x5618df1914f0 2024-03-11T19:18:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.006+0000 7f93f15cf740 10 client.0 _ll_register_callbacks cb 0x5618df0d4fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.007+0000 7f93e7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df1914f0 0x5618df19b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.007+0000 7f93e77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df191ee0 0x5618df19b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.007+0000 7f93e7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df1914f0 0x5618df19b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45516/0 (socket says 172.21.15.29:45516) 2024-03-11T19:18:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.007+0000 7f93e7fff640 1 -- 172.21.15.29:0/377718996 learned_addr learned my addr 172.21.15.29:0/377718996 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.007+0000 7f93ecec4640 1 --2- 172.21.15.29:0/377718996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df197050 0x5618df1a27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.007+0000 7f93e77fe640 1 -- 172.21.15.29:0/377718996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df1914f0 msgr2=0x5618df19b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.007+0000 7f93e77fe640 1 --2- 172.21.15.29:0/377718996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df1914f0 0x5618df19b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.007+0000 7f93e77fe640 1 -- 172.21.15.29:0/377718996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df197050 msgr2=0x5618df1a27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.007+0000 7f93e77fe640 1 --2- 172.21.15.29:0/377718996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df197050 0x5618df1a27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.007+0000 7f93e77fe640 1 -- 172.21.15.29:0/377718996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e013a830 con 0x5618df191ee0 2024-03-11T19:18:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.008+0000 7f93e77fe640 1 --2- 172.21.15.29:0/377718996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df191ee0 0x5618df19b790 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f93d40eced0 tx=0x7f93d40ecf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d337ed1cac8388e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.008+0000 7f93c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:18:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.008+0000 7f93c77fe640 1 -- 172.21.15.29:0/377718996 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d40bc140 con 0x5618df191ee0 2024-03-11T19:18:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.008+0000 7f93c77fe640 1 -- 172.21.15.29:0/377718996 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f93d40bca70 con 0x5618df191ee0 2024-03-11T19:18:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.009+0000 7f93f15cf740 1 -- 172.21.15.29:0/377718996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5618dedf97e0 con 0x5618df191ee0 2024-03-11T19:18:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.009+0000 7f93c77fe640 1 -- 172.21.15.29:0/377718996 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93d40bbeb0 con 0x5618df191ee0 2024-03-11T19:18:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.009+0000 7f93f15cf740 10 client.17451 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.009+0000 7f93f15cf740 1 -- 172.21.15.29:0/377718996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5618df0f8610 con 0x5618df191ee0 2024-03-11T19:18:35.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.010+0000 7f93c77fe640 1 -- 172.21.15.29:0/377718996 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5618dedf97e0 con 0x5618df191ee0 2024-03-11T19:18:35.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.010+0000 7f93c77fe640 1 -- 172.21.15.29:0/377718996 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f93d403c070 con 0x5618df191ee0 2024-03-11T19:18:35.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.010+0000 7f93f15cf740 10 client.17451 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:35.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.010+0000 7f93f15cf740 10 client.17451 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:18:35.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.010+0000 7f93f15cf740 20 client.17451 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:35.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.010+0000 7f93f15cf740 1 --2- 172.21.15.29:0/377718996 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5618df1a6e20 0x5618df1a9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.010+0000 7f93f15cf740 4 client.17451 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:18:35.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.010+0000 7f93f15cf740 1 -- 172.21.15.29:0/377718996 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5618dee36720 con 0x5618df1a6e20 2024-03-11T19:18:35.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.011+0000 7f93e7fff640 1 --2- 172.21.15.29:0/377718996 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5618df1a6e20 0x5618df1a9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.011+0000 7f93e7fff640 1 --2- 172.21.15.29:0/377718996 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5618df1a6e20 0x5618df1a9210 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ee42777ce268416 server_cookie=2e9ff3e8540d2dd5 in_seq=0 out_seq=0 2024-03-11T19:18:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.011+0000 7f93c77fe640 10 client.17451 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.012+0000 7f93c77fe640 1 -- 172.21.15.29:0/377718996 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5618dee36720 con 0x5618df1a6e20 2024-03-11T19:18:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.012+0000 7f93c77fe640 10 client.17451 handle_command_reply: tid=0 2024-03-11T19:18:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.012+0000 7f93f15cf740 1 client.17451 shutdown 2024-03-11T19:18:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.012+0000 7f93f15cf740 1 -- 172.21.15.29:0/377718996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df191ee0 msgr2=0x5618df19b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.012+0000 7f93f15cf740 1 --2- 172.21.15.29:0/377718996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df191ee0 0x5618df19b790 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f93d40eced0 tx=0x7f93d40ecf00 comp rx=0 tx=0).stop 2024-03-11T19:18:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.012+0000 7f93f15cf740 1 -- 172.21.15.29:0/377718996 shutdown_connections 2024-03-11T19:18:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.012+0000 7f93f15cf740 1 --2- 172.21.15.29:0/377718996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df197050 0x5618df1a27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.012+0000 7f93f15cf740 1 --2- 172.21.15.29:0/377718996 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5618df1a6e20 0x5618df1a9210 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.012+0000 7f93f15cf740 1 --2- 172.21.15.29:0/377718996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df191ee0 0x5618df19b790 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.012+0000 7f93f15cf740 1 --2- 172.21.15.29:0/377718996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df1914f0 0x5618df19b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.012+0000 7f93f15cf740 1 -- 172.21.15.29:0/377718996 >> 172.21.15.29:0/377718996 conn(0x5618df18ca60 msgr2=0x5618df18bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.012+0000 7f93c77fe640 0 client.17451 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.012+0000 7f93f15cf740 1 -- 172.21.15.29:0/377718996 shutdown_connections 2024-03-11T19:18:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.013+0000 7f93f15cf740 1 -- 172.21.15.29:0/377718996 wait complete. 2024-03-11T19:18:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.013+0000 7f93f15cf740 20 client.17451 trim_cache size 0 max 16384 2024-03-11T19:18:35.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.044+0000 7f93f15cf740 1 Processor -- start 2024-03-11T19:18:35.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.044+0000 7f93f15cf740 1 -- start start 2024-03-11T19:18:35.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.044+0000 7f93f15cf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df1914f0 0x5618df19b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.044+0000 7f93f15cf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df191ee0 0x5618df19b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.044+0000 7f93f15cf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df197050 0x5618df1a2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.044+0000 7f93f15cf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618df18d070 con 0x5618df197050 2024-03-11T19:18:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.044+0000 7f93f15cf740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618df19b090 con 0x5618df1914f0 2024-03-11T19:18:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.044+0000 7f93f15cf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618dedf97e0 con 0x5618df191ee0 2024-03-11T19:18:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.044+0000 7f93e77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df191ee0 0x5618df19b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.044+0000 7f93ecec4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df197050 0x5618df1a2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.044+0000 7f93e77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df191ee0 0x5618df19b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45532/0 (socket says 172.21.15.29:45532) 2024-03-11T19:18:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.044+0000 7f93ecec4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df197050 0x5618df1a2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48368/0 (socket says 172.21.15.29:48368) 2024-03-11T19:18:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.044+0000 7f93e77fe640 1 -- 172.21.15.29:0/2395971668 learned_addr learned my addr 172.21.15.29:0/2395971668 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.044+0000 7f93e7fff640 1 --2- 172.21.15.29:0/2395971668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df1914f0 0x5618df19b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.045+0000 7f93e77fe640 1 -- 172.21.15.29:0/2395971668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df1914f0 msgr2=0x5618df19b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.045+0000 7f93e77fe640 1 --2- 172.21.15.29:0/2395971668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df1914f0 0x5618df19b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.045+0000 7f93e77fe640 1 -- 172.21.15.29:0/2395971668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df197050 msgr2=0x5618df1a2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.045+0000 7f93e77fe640 1 --2- 172.21.15.29:0/2395971668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df197050 0x5618df1a2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.045+0000 7f93e77fe640 1 -- 172.21.15.29:0/2395971668 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5618df094ac0 con 0x5618df191ee0 2024-03-11T19:18:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.045+0000 7f93e77fe640 1 --2- 172.21.15.29:0/2395971668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df191ee0 0x5618df19b770 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f93d40b8d40 tx=0x7f93d40f73d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f82e7a94862d3d22 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:35.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.045+0000 7f93d1ffb640 1 -- 172.21.15.29:0/2395971668 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d40bc140 con 0x5618df191ee0 2024-03-11T19:18:35.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.045+0000 7f93d1ffb640 1 -- 172.21.15.29:0/2395971668 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f93d40bbeb0 con 0x5618df191ee0 2024-03-11T19:18:35.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.045+0000 7f93d1ffb640 1 -- 172.21.15.29:0/2395971668 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d40bca70 con 0x5618df191ee0 2024-03-11T19:18:35.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.046+0000 7f93f15cf740 1 -- 172.21.15.29:0/2395971668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df191ee0 msgr2=0x5618df19b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:35.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.046+0000 7f93f15cf740 1 --2- 172.21.15.29:0/2395971668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df191ee0 0x5618df19b770 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f93d40b8d40 tx=0x7f93d40f73d0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.046+0000 7f93f15cf740 1 -- 172.21.15.29:0/2395971668 shutdown_connections 2024-03-11T19:18:35.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.046+0000 7f93f15cf740 1 --2- 172.21.15.29:0/2395971668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df197050 0x5618df1a2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.046+0000 7f93f15cf740 1 --2- 172.21.15.29:0/2395971668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df191ee0 0x5618df19b770 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.046+0000 7f93f15cf740 1 --2- 172.21.15.29:0/2395971668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df1914f0 0x5618df19b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.046+0000 7f93f15cf740 1 -- 172.21.15.29:0/2395971668 >> 172.21.15.29:0/2395971668 conn(0x5618df18ca60 msgr2=0x5618df1adb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:35.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.046+0000 7f93f15cf740 1 -- 172.21.15.29:0/2395971668 shutdown_connections 2024-03-11T19:18:35.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.046+0000 7f93f15cf740 1 -- 172.21.15.29:0/2395971668 wait complete. 2024-03-11T19:18:35.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.047+0000 7f93f15cf740 1 -- start start 2024-03-11T19:18:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.047+0000 7f93f15cf740 1 Processor -- start 2024-03-11T19:18:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.047+0000 7f93f15cf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df1914f0 0x5618df1a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.047+0000 7f93f15cf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df191ee0 0x5618df1a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.047+0000 7f93f15cf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df197050 0x5618df1a1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.048+0000 7f93f15cf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618df1a21f0 con 0x5618df197050 2024-03-11T19:18:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.048+0000 7f93e7fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df1914f0 0x5618df1a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.048+0000 7f93e77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df191ee0 0x5618df1a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.048+0000 7f93f15cf740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618df1a2360 con 0x5618df1914f0 2024-03-11T19:18:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.048+0000 7f93f15cf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618df1a2520 con 0x5618df191ee0 2024-03-11T19:18:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.048+0000 7f93e7fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df1914f0 0x5618df1a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44962/0 (socket says 172.21.15.29:44962) 2024-03-11T19:18:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.048+0000 7f93e7fff640 1 -- 172.21.15.29:0/1045134189 learned_addr learned my addr 172.21.15.29:0/1045134189 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.048+0000 7f93f15cf740 10 client.0 _ll_register_callbacks cb 0x7f93d40f2c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.048+0000 7f93ecec4640 1 --2- 172.21.15.29:0/1045134189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df197050 0x5618df1a1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.048+0000 7f93e77fe640 1 -- 172.21.15.29:0/1045134189 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df1914f0 msgr2=0x5618df1a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:35.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.048+0000 7f93e77fe640 1 --2- 172.21.15.29:0/1045134189 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df1914f0 0x5618df1a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.048+0000 7f93e77fe640 1 -- 172.21.15.29:0/1045134189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df197050 msgr2=0x5618df1a1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:35.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.048+0000 7f93e77fe640 1 --2- 172.21.15.29:0/1045134189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df197050 0x5618df1a1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.048+0000 7f93e77fe640 1 -- 172.21.15.29:0/1045134189 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e013a830 con 0x5618df191ee0 2024-03-11T19:18:35.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.048+0000 7f93ecec4640 1 --2- 172.21.15.29:0/1045134189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df197050 0x5618df1a1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:18:35.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.048+0000 7f93e77fe640 1 --2- 172.21.15.29:0/1045134189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df191ee0 0x5618df1a0de0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x5618dedf9920 tx=0x7f93d40b8c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6aa617ddd261eec server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:35.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.048+0000 7f93c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:18:35.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.049+0000 7f93c77fe640 1 -- 172.21.15.29:0/1045134189 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d40c0540 con 0x5618df191ee0 2024-03-11T19:18:35.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.049+0000 7f93c77fe640 1 -- 172.21.15.29:0/1045134189 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f93d40bbad0 con 0x5618df191ee0 2024-03-11T19:18:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.049+0000 7f93f15cf740 1 -- 172.21.15.29:0/1045134189 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f93d40b8610 con 0x5618df191ee0 2024-03-11T19:18:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.049+0000 7f93c77fe640 1 -- 172.21.15.29:0/1045134189 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93d40bc140 con 0x5618df191ee0 2024-03-11T19:18:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.049+0000 7f93f15cf740 10 client.26977 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.049+0000 7f93f15cf740 1 -- 172.21.15.29:0/1045134189 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5618df0ba6a0 con 0x5618df191ee0 2024-03-11T19:18:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.049+0000 7f93c77fe640 1 -- 172.21.15.29:0/1045134189 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f93d40b8610 con 0x5618df191ee0 2024-03-11T19:18:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.049+0000 7f93c77fe640 1 -- 172.21.15.29:0/1045134189 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f93d403c070 con 0x5618df191ee0 2024-03-11T19:18:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.049+0000 7f93f15cf740 10 client.26977 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.049+0000 7f93f15cf740 10 client.26977 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:18:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.050+0000 7f93f15cf740 20 client.26977 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.050+0000 7f93f15cf740 1 --2- 172.21.15.29:0/1045134189 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5618df1fdfd0 0x5618df2003f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.050+0000 7f93f15cf740 4 client.26977 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:18:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.050+0000 7f93f15cf740 1 -- 172.21.15.29:0/1045134189 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5618dee85890 con 0x5618df1fdfd0 2024-03-11T19:18:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.050+0000 7f93e7fff640 1 --2- 172.21.15.29:0/1045134189 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5618df1fdfd0 0x5618df2003f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.051+0000 7f93e7fff640 1 --2- 172.21.15.29:0/1045134189 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5618df1fdfd0 0x5618df2003f0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9589094e598d561 server_cookie=8c9c017cf9ddc0e8 in_seq=0 out_seq=0 2024-03-11T19:18:35.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.051+0000 7f93c77fe640 10 client.26977 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.052+0000 7f93c77fe640 1 -- 172.21.15.29:0/1045134189 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5618dee85890 con 0x5618df1fdfd0 2024-03-11T19:18:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.052+0000 7f93c77fe640 10 client.26977 handle_command_reply: tid=0 2024-03-11T19:18:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.052+0000 7f93f15cf740 1 client.26977 shutdown 2024-03-11T19:18:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.052+0000 7f93f15cf740 1 -- 172.21.15.29:0/1045134189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df191ee0 msgr2=0x5618df1a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.052+0000 7f93f15cf740 1 --2- 172.21.15.29:0/1045134189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df191ee0 0x5618df1a0de0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x5618dedf9920 tx=0x7f93d40b8c10 comp rx=0 tx=0).stop 2024-03-11T19:18:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.053+0000 7f93f15cf740 1 -- 172.21.15.29:0/1045134189 shutdown_connections 2024-03-11T19:18:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.053+0000 7f93f15cf740 1 --2- 172.21.15.29:0/1045134189 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5618df1fdfd0 0x5618df2003f0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.053+0000 7f93f15cf740 1 --2- 172.21.15.29:0/1045134189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5618df197050 0x5618df1a1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.053+0000 7f93f15cf740 1 --2- 172.21.15.29:0/1045134189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5618df191ee0 0x5618df1a0de0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.053+0000 7f93f15cf740 1 --2- 172.21.15.29:0/1045134189 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5618df1914f0 0x5618df1a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.053+0000 7f93f15cf740 1 -- 172.21.15.29:0/1045134189 >> 172.21.15.29:0/1045134189 conn(0x5618df18ca60 msgr2=0x5618df1930c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:35.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.053+0000 7f93c77fe640 0 client.26977 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:35.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.053+0000 7f93f15cf740 1 -- 172.21.15.29:0/1045134189 shutdown_connections 2024-03-11T19:18:35.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.053+0000 7f93f15cf740 1 -- 172.21.15.29:0/1045134189 wait complete. 2024-03-11T19:18:35.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.053+0000 7f93f15cf740 20 client.26977 trim_cache size 0 max 16384 2024-03-11T19:18:35.059 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:18:35.059 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:18:35.059 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:18:35.059 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:18:35.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.055+0000 7f93ed6c5640 1 -- 172.21.15.29:0/4133444877 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f93d80a1b30 msgr2=0x7f93d80a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:35.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.055+0000 7f93ed6c5640 1 --2- 172.21.15.29:0/4133444877 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f93d80a1b30 0x7f93d80a3ff0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f93e80668b0 tx=0x7f93e8078040 comp rx=0 tx=0).stop 2024-03-11T19:18:35.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.055+0000 7f93ed6c5640 1 -- 172.21.15.29:0/4133444877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93e009d550 msgr2=0x7f93e0132830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:35.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.055+0000 7f93ed6c5640 1 --2- 172.21.15.29:0/4133444877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93e009d550 0x7f93e0132830 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f93d4006010 tx=0x7f93d4004370 comp rx=0 tx=0).stop 2024-03-11T19:18:35.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.055+0000 7f93ed6c5640 1 -- 172.21.15.29:0/4133444877 shutdown_connections 2024-03-11T19:18:35.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.055+0000 7f93ed6c5640 1 --2- 172.21.15.29:0/4133444877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93e009df10 0x7f93e01398a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.055+0000 7f93ed6c5640 1 --2- 172.21.15.29:0/4133444877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93e009d550 0x7f93e0132830 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.055+0000 7f93ed6c5640 1 --2- 172.21.15.29:0/4133444877 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f93d80a1b30 0x7f93d80a3ff0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.055+0000 7f93ed6c5640 1 --2- 172.21.15.29:0/4133444877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93e009c360 0x7f93e01322f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.055+0000 7f93ed6c5640 1 -- 172.21.15.29:0/4133444877 >> 172.21.15.29:0/4133444877 conn(0x7f93e000b4c0 msgr2=0x7f93e00a08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:35.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.055+0000 7f93ed6c5640 1 -- 172.21.15.29:0/4133444877 shutdown_connections 2024-03-11T19:18:35.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.055+0000 7f93ed6c5640 1 -- 172.21.15.29:0/4133444877 wait complete. 2024-03-11T19:18:35.079 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:18:35.079 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:18:35.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.259+0000 7f0dc8447640 1 -- 172.21.15.29:0/2098471478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0db8094650 msgr2=0x7f0db8094a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:35.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.259+0000 7f0dc8447640 1 --2- 172.21.15.29:0/2098471478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0db8094650 0x7f0db8094a70 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009860 tx=0x7f0dbc02bb70 comp rx=0 tx=0).stop 2024-03-11T19:18:35.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.260+0000 7f0dc8447640 1 -- 172.21.15.29:0/2098471478 shutdown_connections 2024-03-11T19:18:35.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.260+0000 7f0dc8447640 1 --2- 172.21.15.29:0/2098471478 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0db8095950 0x7f0db8095dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.260+0000 7f0dc8447640 1 --2- 172.21.15.29:0/2098471478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0db8094fb0 0x7f0db8095410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.260+0000 7f0dc8447640 1 --2- 172.21.15.29:0/2098471478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0db8094650 0x7f0db8094a70 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.260+0000 7f0dc8447640 1 -- 172.21.15.29:0/2098471478 >> 172.21.15.29:0/2098471478 conn(0x7f0db80901d0 msgr2=0x7f0db8092630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:35.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.260+0000 7f0dc8447640 1 -- 172.21.15.29:0/2098471478 shutdown_connections 2024-03-11T19:18:35.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.260+0000 7f0dc8447640 1 -- 172.21.15.29:0/2098471478 wait complete. 2024-03-11T19:18:35.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.261+0000 7f0dc8447640 1 Processor -- start 2024-03-11T19:18:35.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.261+0000 7f0dc8447640 1 -- start start 2024-03-11T19:18:35.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.261+0000 7f0dc8447640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0db8094650 0x7f0db8132350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.261+0000 7f0dc8447640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0db8094fb0 0x7f0db8132890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.261+0000 7f0dc8447640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0db8095950 0x7f0db814a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.261+0000 7f0dc8447640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db814ae50 con 0x7f0db8094650 2024-03-11T19:18:35.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.261+0000 7f0dc8447640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db814afc0 con 0x7f0db8094fb0 2024-03-11T19:18:35.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.261+0000 7f0dc8447640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db814b180 con 0x7f0db8095950 2024-03-11T19:18:35.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.261+0000 7f0dc69e4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0db8095950 0x7f0db814a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.262+0000 7f0dc69e4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0db8095950 0x7f0db814a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45562/0 (socket says 172.21.15.29:45562) 2024-03-11T19:18:35.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.262+0000 7f0dc69e4640 1 -- 172.21.15.29:0/2065965877 learned_addr learned my addr 172.21.15.29:0/2065965877 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.262+0000 7f0dc69e4640 1 -- 172.21.15.29:0/2065965877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0db8094fb0 msgr2=0x7f0db8132890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:18:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.262+0000 7f0dc69e4640 1 --2- 172.21.15.29:0/2065965877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0db8094fb0 0x7f0db8132890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.262+0000 7f0dc69e4640 1 -- 172.21.15.29:0/2065965877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0db8094650 msgr2=0x7f0db8132350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:18:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.262+0000 7f0dc69e4640 1 --2- 172.21.15.29:0/2065965877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0db8094650 0x7f0db8132350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.262+0000 7f0dc69e4640 1 -- 172.21.15.29:0/2065965877 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc009490 con 0x7f0db8095950 2024-03-11T19:18:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.262+0000 7f0dc69e4640 1 --2- 172.21.15.29:0/2065965877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0db8095950 0x7f0db814a910 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0064670 tx=0x7f0dc0076f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.262+0000 7f0db77fe640 1 -- 172.21.15.29:0/2065965877 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc006e040 con 0x7f0db8095950 2024-03-11T19:18:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.263+0000 7f0db77fe640 1 -- 172.21.15.29:0/2065965877 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0dc00763e0 con 0x7f0db8095950 2024-03-11T19:18:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.263+0000 7f0db77fe640 1 -- 172.21.15.29:0/2065965877 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0067860 con 0x7f0db8095950 2024-03-11T19:18:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.263+0000 7f0dc8447640 1 -- 172.21.15.29:0/2065965877 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db814b460 con 0x7f0db8095950 2024-03-11T19:18:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.263+0000 7f0dc8447640 1 -- 172.21.15.29:0/2065965877 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db814b890 con 0x7f0db8095950 2024-03-11T19:18:35.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.266+0000 7f0dc8447640 1 -- 172.21.15.29:0/2065965877 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db809cc60 con 0x7f0db8095950 2024-03-11T19:18:35.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.266+0000 7f0db77fe640 1 -- 172.21.15.29:0/2065965877 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0dc006ca60 con 0x7f0db8095950 2024-03-11T19:18:35.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.267+0000 7f0db77fe640 1 --2- 172.21.15.29:0/2065965877 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0dac0a1b30 0x7f0dac0a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.267+0000 7f0db77fe640 1 -- 172.21.15.29:0/2065965877 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0dc012f820 con 0x7f0db8095950 2024-03-11T19:18:35.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.268+0000 7f0dc61e3640 1 --2- 172.21.15.29:0/2065965877 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0dac0a1b30 0x7f0dac0a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.269+0000 7f0dc61e3640 1 --2- 172.21.15.29:0/2065965877 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0dac0a1b30 0x7f0dac0a3ff0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc005e00 tx=0x7f0dbc005c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:35.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.272+0000 7f0db77fe640 1 -- 172.21.15.29:0/2065965877 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0dc00f8a30 con 0x7f0db8095950 2024-03-11T19:18:35.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:35 smithi029 ceph-mon[32439]: pgmap v614: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 60 MiB/s wr, 1.13k op/s 2024-03-11T19:18:35.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:35 smithi170 ceph-mon[35690]: pgmap v614: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 60 MiB/s wr, 1.13k op/s 2024-03-11T19:18:35.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.718+0000 7f0dc8447640 1 -- 172.21.15.29:0/2065965877 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0db8097790 con 0x7f0db8095950 2024-03-11T19:18:35.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.719+0000 7f0db77fe640 1 -- 172.21.15.29:0/2065965877 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f0dc00fc0f0 con 0x7f0db8095950 2024-03-11T19:18:35.723 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:18:35.723 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:18:35.723 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:18:35.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.723+0000 7f0dc8447640 1 -- 172.21.15.29:0/2065965877 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0dac0a1b30 msgr2=0x7f0dac0a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:35.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.723+0000 7f0dc8447640 1 --2- 172.21.15.29:0/2065965877 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0dac0a1b30 0x7f0dac0a3ff0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc005e00 tx=0x7f0dbc005c90 comp rx=0 tx=0).stop 2024-03-11T19:18:35.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.723+0000 7f0dc8447640 1 -- 172.21.15.29:0/2065965877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0db8095950 msgr2=0x7f0db814a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:35.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.723+0000 7f0dc8447640 1 --2- 172.21.15.29:0/2065965877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0db8095950 0x7f0db814a910 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0064670 tx=0x7f0dc0076f80 comp rx=0 tx=0).stop 2024-03-11T19:18:35.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.723+0000 7f0dc8447640 1 -- 172.21.15.29:0/2065965877 shutdown_connections 2024-03-11T19:18:35.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.723+0000 7f0dc8447640 1 --2- 172.21.15.29:0/2065965877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0db8095950 0x7f0db814a910 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.723+0000 7f0dc8447640 1 --2- 172.21.15.29:0/2065965877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0db8094fb0 0x7f0db8132890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.723+0000 7f0dc8447640 1 --2- 172.21.15.29:0/2065965877 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0dac0a1b30 0x7f0dac0a3ff0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.723+0000 7f0dc8447640 1 --2- 172.21.15.29:0/2065965877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0db8094650 0x7f0db8132350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.723+0000 7f0dc8447640 1 -- 172.21.15.29:0/2065965877 >> 172.21.15.29:0/2065965877 conn(0x7f0db80901d0 msgr2=0x7f0db80919d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:35.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.724+0000 7f0dc8447640 1 -- 172.21.15.29:0/2065965877 shutdown_connections 2024-03-11T19:18:35.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.724+0000 7f0dc8447640 1 -- 172.21.15.29:0/2065965877 wait complete. 2024-03-11T19:18:35.747 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:18:35.748 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:18:35.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:35 smithi190 ceph-mon[36292]: pgmap v614: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 60 MiB/s wr, 1.13k op/s 2024-03-11T19:18:35.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.932+0000 7fd71f064640 1 -- 172.21.15.29:0/2684696178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd718073030 msgr2=0x7fd718073450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:35.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.932+0000 7fd71f064640 1 --2- 172.21.15.29:0/2684696178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd718073030 0x7fd718073450 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fd71000b2e0 tx=0x7fd71002d070 comp rx=0 tx=0).stop 2024-03-11T19:18:35.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.933+0000 7fd71f064640 1 -- 172.21.15.29:0/2684696178 shutdown_connections 2024-03-11T19:18:35.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.933+0000 7fd71f064640 1 --2- 172.21.15.29:0/2684696178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7180788c0 0x7fd71807ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.933+0000 7fd71f064640 1 --2- 172.21.15.29:0/2684696178 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd718075f10 0x7fd718078380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.933+0000 7fd71f064640 1 --2- 172.21.15.29:0/2684696178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd718073030 0x7fd718073450 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.933+0000 7fd71f064640 1 -- 172.21.15.29:0/2684696178 >> 172.21.15.29:0/2684696178 conn(0x7fd71806dfe0 msgr2=0x7fd718070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:35.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.933+0000 7fd71f064640 1 -- 172.21.15.29:0/2684696178 shutdown_connections 2024-03-11T19:18:35.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.934+0000 7fd71f064640 1 -- 172.21.15.29:0/2684696178 wait complete. 2024-03-11T19:18:35.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.934+0000 7fd71f064640 1 Processor -- start 2024-03-11T19:18:35.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.934+0000 7fd71f064640 1 -- start start 2024-03-11T19:18:35.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.935+0000 7fd71f064640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd718073030 0x7fd7181b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.935+0000 7fd71f064640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd718075f10 0x7fd7181bbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.935+0000 7fd71ce00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd718073030 0x7fd7181b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.935+0000 7fd71ce00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd718073030 0x7fd7181b98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51632/0 (socket says 172.21.15.29:51632) 2024-03-11T19:18:35.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.935+0000 7fd717fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd718075f10 0x7fd7181bbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.935+0000 7fd717fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd718075f10 0x7fd7181bbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:47300/0 (socket says 172.21.15.29:47300) 2024-03-11T19:18:35.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.936+0000 7fd71f064640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7180788c0 0x7fd7181be340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.936+0000 7fd71f064640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7181beab0 con 0x7fd718073030 2024-03-11T19:18:35.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.936+0000 7fd71f064640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7181bec20 con 0x7fd718075f10 2024-03-11T19:18:35.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.936+0000 7fd71f064640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7181bede0 con 0x7fd7180788c0 2024-03-11T19:18:35.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.936+0000 7fd71ce00640 1 -- 172.21.15.29:0/562223542 learned_addr learned my addr 172.21.15.29:0/562223542 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:35.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.936+0000 7fd717fff640 1 -- 172.21.15.29:0/562223542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7180788c0 msgr2=0x7fd7181be340 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-11T19:18:35.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.936+0000 7fd717fff640 1 --2- 172.21.15.29:0/562223542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7180788c0 0x7fd7181be340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.936+0000 7fd717fff640 1 -- 172.21.15.29:0/562223542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd718073030 msgr2=0x7fd7181b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:35.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.937+0000 7fd717fff640 1 --2- 172.21.15.29:0/562223542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd718073030 0x7fd7181b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.937+0000 7fd717fff640 1 -- 172.21.15.29:0/562223542 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd710009840 con 0x7fd718075f10 2024-03-11T19:18:35.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.937+0000 7fd717fff640 1 --2- 172.21.15.29:0/562223542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd718075f10 0x7fd7181bbdf0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00e790 tx=0x7fd70c00ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:35.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.937+0000 7fd715ffb640 1 -- 172.21.15.29:0/562223542 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c00ccc0 con 0x7fd718075f10 2024-03-11T19:18:35.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.937+0000 7fd715ffb640 1 -- 172.21.15.29:0/562223542 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd70c00eea0 con 0x7fd718075f10 2024-03-11T19:18:35.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.938+0000 7fd715ffb640 1 -- 172.21.15.29:0/562223542 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c018710 con 0x7fd718075f10 2024-03-11T19:18:35.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.938+0000 7fd71f064640 1 -- 172.21.15.29:0/562223542 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7181bf0c0 con 0x7fd718075f10 2024-03-11T19:18:35.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.938+0000 7fd71f064640 1 -- 172.21.15.29:0/562223542 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7181bf600 con 0x7fd718075f10 2024-03-11T19:18:35.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.941+0000 7fd715ffb640 1 -- 172.21.15.29:0/562223542 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd70c018870 con 0x7fd718075f10 2024-03-11T19:18:35.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.942+0000 7fd721c3a740 1 Processor -- start 2024-03-11T19:18:35.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.942+0000 7fd721c3a740 1 -- start start 2024-03-11T19:18:35.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.942+0000 7fd721c3a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56504358b270 0x56504358b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.942+0000 7fd721c3a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56504358bc60 0x565043590760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.942+0000 7fd721c3a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565043590dd0 0x565043593260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.942+0000 7fd721c3a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x565043547960 con 0x56504358b270 2024-03-11T19:18:35.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.942+0000 7fd721c3a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650434fb310 con 0x56504358bc60 2024-03-11T19:18:35.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.942+0000 7fd721c3a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650434f8840 con 0x565043590dd0 2024-03-11T19:18:35.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.942+0000 7fd71d601640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56504358b270 0x56504358b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.942+0000 7fd717fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565043590dd0 0x565043593260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.943+0000 7fd71ce00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56504358bc60 0x565043590760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.943+0000 7fd71d601640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56504358b270 0x56504358b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51636/0 (socket says 172.21.15.29:51636) 2024-03-11T19:18:35.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.943+0000 7fd71d601640 1 -- 172.21.15.29:0/3076114661 learned_addr learned my addr 172.21.15.29:0/3076114661 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:35.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.943+0000 7fd715ffb640 1 --2- 172.21.15.29:0/562223542 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd7000a1c30 0x7fd7000a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.943+0000 7fd717fff640 1 -- 172.21.15.29:0/3076114661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56504358bc60 msgr2=0x565043590760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:35.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.943+0000 7fd717fff640 1 --2- 172.21.15.29:0/3076114661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56504358bc60 0x565043590760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.943+0000 7fd71d601640 1 --2- 172.21.15.29:0/562223542 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd7000a1c30 0x7fd7000a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.943+0000 7fd717fff640 1 -- 172.21.15.29:0/3076114661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56504358b270 msgr2=0x56504358b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:35.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.943+0000 7fd717fff640 1 --2- 172.21.15.29:0/3076114661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56504358b270 0x56504358b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.943+0000 7fd717fff640 1 -- 172.21.15.29:0/3076114661 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650434b6060 con 0x565043590dd0 2024-03-11T19:18:35.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.943+0000 7fd715ffb640 1 -- 172.21.15.29:0/562223542 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd70c014070 con 0x7fd718075f10 2024-03-11T19:18:35.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.944+0000 7fd717fff640 1 --2- 172.21.15.29:0/3076114661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565043590dd0 0x565043593260 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00f010 tx=0x7fd70c013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9452e7af4cc3261 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:35.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.944+0000 7fd6feffd640 1 -- 172.21.15.29:0/3076114661 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c09af90 con 0x565043590dd0 2024-03-11T19:18:35.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.944+0000 7fd6feffd640 1 -- 172.21.15.29:0/3076114661 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd70c09adb0 con 0x565043590dd0 2024-03-11T19:18:35.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.944+0000 7fd6feffd640 1 -- 172.21.15.29:0/3076114661 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c09f5a0 con 0x565043590dd0 2024-03-11T19:18:35.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.944+0000 7fd71d601640 1 --2- 172.21.15.29:0/562223542 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd7000a1c30 0x7fd7000a40f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x56504358c4c0 tx=0x7fd708008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:35.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.945+0000 7fd721c3a740 1 -- 172.21.15.29:0/3076114661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565043590dd0 msgr2=0x565043593260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:35.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.945+0000 7fd721c3a740 1 --2- 172.21.15.29:0/3076114661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565043590dd0 0x565043593260 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00f010 tx=0x7fd70c013000 comp rx=0 tx=0).stop 2024-03-11T19:18:35.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.945+0000 7fd721c3a740 1 -- 172.21.15.29:0/3076114661 shutdown_connections 2024-03-11T19:18:35.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.945+0000 7fd721c3a740 1 --2- 172.21.15.29:0/3076114661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565043590dd0 0x565043593260 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.945+0000 7fd721c3a740 1 --2- 172.21.15.29:0/3076114661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56504358bc60 0x565043590760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.945+0000 7fd721c3a740 1 --2- 172.21.15.29:0/3076114661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56504358b270 0x56504358b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.945+0000 7fd721c3a740 1 -- 172.21.15.29:0/3076114661 >> 172.21.15.29:0/3076114661 conn(0x5650435867e0 msgr2=0x565043586be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:35.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.945+0000 7fd721c3a740 1 -- 172.21.15.29:0/3076114661 shutdown_connections 2024-03-11T19:18:35.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.945+0000 7fd721c3a740 1 -- 172.21.15.29:0/3076114661 wait complete. 2024-03-11T19:18:35.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.946+0000 7fd721c3a740 1 -- start start 2024-03-11T19:18:35.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.946+0000 7fd721c3a740 1 Processor -- start 2024-03-11T19:18:35.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.947+0000 7fd721c3a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56504358b270 0x565043595430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.947+0000 7fd721c3a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56504358bc60 0x565043595970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.947+0000 7fd71ce00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56504358b270 0x565043595430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.947+0000 7fd71ce00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56504358b270 0x565043595430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51640/0 (socket says 172.21.15.29:51640) 2024-03-11T19:18:35.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.947+0000 7fd721c3a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565043590dd0 0x56504359c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.947+0000 7fd71ce00640 1 -- 172.21.15.29:0/1010005859 learned_addr learned my addr 172.21.15.29:0/1010005859 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:35.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.947+0000 7fd721c3a740 1 -- 172.21.15.29:0/1010005859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650434f4020 con 0x56504358b270 2024-03-11T19:18:35.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.947+0000 7fd717fff640 1 --2- 172.21.15.29:0/1010005859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56504358bc60 0x565043595970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.947+0000 7fd721c3a740 1 -- 172.21.15.29:0/1010005859 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650431c7430 con 0x56504358bc60 2024-03-11T19:18:35.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.947+0000 7fd721c3a740 1 -- 172.21.15.29:0/1010005859 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x565043183e00 con 0x565043590dd0 2024-03-11T19:18:35.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.947+0000 7fd721c3a740 10 client.0 _ll_register_callbacks cb 0x56504349bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:35.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.947+0000 7fd71d601640 1 --2- 172.21.15.29:0/1010005859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565043590dd0 0x56504359c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.948+0000 7fd71d601640 1 -- 172.21.15.29:0/1010005859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56504358bc60 msgr2=0x565043595970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:35.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.948+0000 7fd71d601640 1 --2- 172.21.15.29:0/1010005859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56504358bc60 0x565043595970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.948+0000 7fd71d601640 1 -- 172.21.15.29:0/1010005859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56504358b270 msgr2=0x565043595430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:35.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.948+0000 7fd71d601640 1 --2- 172.21.15.29:0/1010005859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56504358b270 0x565043595430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.948+0000 7fd71d601640 1 -- 172.21.15.29:0/1010005859 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7181bf600 con 0x565043590dd0 2024-03-11T19:18:35.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.948+0000 7fd717fff640 1 --2- 172.21.15.29:0/1010005859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56504358bc60 0x565043595970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:18:35.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.948+0000 7fd71ce00640 1 --2- 172.21.15.29:0/1010005859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56504358b270 0x565043595430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:18:35.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.948+0000 7fd71d601640 1 --2- 172.21.15.29:0/1010005859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565043590dd0 0x56504359c9b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd7080166c0 tx=0x7fd708004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de7452cd54028c93 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:35.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.948+0000 7fd6f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:18:35.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.948+0000 7fd6f7fff640 1 -- 172.21.15.29:0/1010005859 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd708023620 con 0x565043590dd0 2024-03-11T19:18:35.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.948+0000 7fd6f7fff640 1 -- 172.21.15.29:0/1010005859 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd708014420 con 0x565043590dd0 2024-03-11T19:18:35.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.948+0000 7fd721c3a740 1 -- 172.21.15.29:0/1010005859 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5650431f37e0 con 0x565043590dd0 2024-03-11T19:18:35.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.949+0000 7fd6f7fff640 1 -- 172.21.15.29:0/1010005859 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7080145e0 con 0x565043590dd0 2024-03-11T19:18:35.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.949+0000 7fd721c3a740 10 client.26989 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:35.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.949+0000 7fd721c3a740 1 -- 172.21.15.29:0/1010005859 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5650434f26a0 con 0x565043590dd0 2024-03-11T19:18:35.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.949+0000 7fd6f7fff640 1 -- 172.21.15.29:0/1010005859 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5650431f37e0 con 0x565043590dd0 2024-03-11T19:18:35.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.949+0000 7fd6f7fff640 1 -- 172.21.15.29:0/1010005859 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd70802d730 con 0x565043590dd0 2024-03-11T19:18:35.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.949+0000 7fd721c3a740 10 client.26989 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:35.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.949+0000 7fd721c3a740 10 client.26989 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:18:35.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.949+0000 7fd721c3a740 20 client.26989 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:35.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.949+0000 7fd721c3a740 1 --2- 172.21.15.29:0/1010005859 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5650435a16e0 0x5650435a3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.949+0000 7fd721c3a740 4 client.26989 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:18:35.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.949+0000 7fd721c3a740 1 -- 172.21.15.29:0/1010005859 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565043230720 con 0x5650435a16e0 2024-03-11T19:18:35.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.949+0000 7fd71ce00640 1 --2- 172.21.15.29:0/1010005859 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5650435a16e0 0x5650435a3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.950+0000 7fd6f7fff640 10 client.26989 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:18:35.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.950+0000 7fd71ce00640 1 --2- 172.21.15.29:0/1010005859 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5650435a16e0 0x5650435a3b40 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=44149e4cbaf8340b server_cookie=32cabb39ec69842a in_seq=0 out_seq=0 2024-03-11T19:18:35.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.950+0000 7fd6f7fff640 1 -- 172.21.15.29:0/1010005859 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x565043230720 con 0x5650435a16e0 2024-03-11T19:18:35.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.950+0000 7fd6f7fff640 10 client.26989 handle_command_reply: tid=0 2024-03-11T19:18:35.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.951+0000 7fd721c3a740 1 client.26989 shutdown 2024-03-11T19:18:35.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.951+0000 7fd721c3a740 1 -- 172.21.15.29:0/1010005859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565043590dd0 msgr2=0x56504359c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:35.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.951+0000 7fd721c3a740 1 --2- 172.21.15.29:0/1010005859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565043590dd0 0x56504359c9b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd7080166c0 tx=0x7fd708004750 comp rx=0 tx=0).stop 2024-03-11T19:18:35.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.951+0000 7fd721c3a740 1 -- 172.21.15.29:0/1010005859 shutdown_connections 2024-03-11T19:18:35.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.951+0000 7fd721c3a740 1 --2- 172.21.15.29:0/1010005859 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5650435a16e0 0x5650435a3b40 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.951+0000 7fd721c3a740 1 --2- 172.21.15.29:0/1010005859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565043590dd0 0x56504359c9b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.951+0000 7fd6f7fff640 0 client.26989 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:18:35.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.951+0000 7fd721c3a740 1 --2- 172.21.15.29:0/1010005859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56504358bc60 0x565043595970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.951+0000 7fd721c3a740 1 --2- 172.21.15.29:0/1010005859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56504358b270 0x565043595430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.951+0000 7fd721c3a740 1 -- 172.21.15.29:0/1010005859 >> 172.21.15.29:0/1010005859 conn(0x5650435867e0 msgr2=0x565043585b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:35.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.951+0000 7fd721c3a740 1 -- 172.21.15.29:0/1010005859 shutdown_connections 2024-03-11T19:18:35.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.951+0000 7fd721c3a740 1 -- 172.21.15.29:0/1010005859 wait complete. 2024-03-11T19:18:35.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.951+0000 7fd721c3a740 20 client.26989 trim_cache size 0 max 16384 2024-03-11T19:18:35.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.974+0000 7fd721c3a740 1 Processor -- start 2024-03-11T19:18:35.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.974+0000 7fd721c3a740 1 -- start start 2024-03-11T19:18:35.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.974+0000 7fd721c3a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56504358b270 0x565043595410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.975+0000 7fd721c3a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56504358bc60 0x565043595950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.975+0000 7fd721c3a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565043590dd0 0x56504359c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.975+0000 7fd721c3a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x565043479130 con 0x565043590dd0 2024-03-11T19:18:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.975+0000 7fd721c3a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56504358c350 con 0x56504358b270 2024-03-11T19:18:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.975+0000 7fd721c3a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x565043594eb0 con 0x56504358bc60 2024-03-11T19:18:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.975+0000 7fd71d601640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565043590dd0 0x56504359c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.975+0000 7fd71d601640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565043590dd0 0x56504359c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51644/0 (socket says 172.21.15.29:51644) 2024-03-11T19:18:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.975+0000 7fd71d601640 1 -- 172.21.15.29:0/2141804964 learned_addr learned my addr 172.21.15.29:0/2141804964 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.975+0000 7fd71ce00640 1 --2- 172.21.15.29:0/2141804964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56504358b270 0x565043595410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.976+0000 7fd71d601640 1 -- 172.21.15.29:0/2141804964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56504358bc60 msgr2=0x565043595950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:18:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.976+0000 7fd71d601640 1 --2- 172.21.15.29:0/2141804964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56504358bc60 0x565043595950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.976+0000 7fd71d601640 1 -- 172.21.15.29:0/2141804964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56504358b270 msgr2=0x565043595410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.976+0000 7fd71d601640 1 --2- 172.21.15.29:0/2141804964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56504358b270 0x565043595410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.976+0000 7fd71d601640 1 -- 172.21.15.29:0/2141804964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56504348ec20 con 0x565043590dd0 2024-03-11T19:18:35.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.976+0000 7fd71d601640 1 --2- 172.21.15.29:0/2141804964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565043590dd0 0x56504359c8d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd70802db90 tx=0x7fd70800f5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410c513d9b233d99 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:35.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.977+0000 7fd6fe7fc640 1 -- 172.21.15.29:0/2141804964 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70802d400 con 0x565043590dd0 2024-03-11T19:18:35.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.977+0000 7fd6fe7fc640 1 -- 172.21.15.29:0/2141804964 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd708023420 con 0x565043590dd0 2024-03-11T19:18:35.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.977+0000 7fd6fe7fc640 1 -- 172.21.15.29:0/2141804964 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd708014680 con 0x565043590dd0 2024-03-11T19:18:35.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.978+0000 7fd721c3a740 1 -- 172.21.15.29:0/2141804964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565043590dd0 msgr2=0x56504359c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:35.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.978+0000 7fd721c3a740 1 --2- 172.21.15.29:0/2141804964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565043590dd0 0x56504359c8d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd70802db90 tx=0x7fd70800f5a0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.979+0000 7fd721c3a740 1 -- 172.21.15.29:0/2141804964 shutdown_connections 2024-03-11T19:18:35.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.979+0000 7fd721c3a740 1 --2- 172.21.15.29:0/2141804964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x565043590dd0 0x56504359c8d0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.979+0000 7fd721c3a740 1 --2- 172.21.15.29:0/2141804964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56504358bc60 0x565043595950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.979+0000 7fd721c3a740 1 --2- 172.21.15.29:0/2141804964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56504358b270 0x565043595410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.979+0000 7fd721c3a740 1 -- 172.21.15.29:0/2141804964 >> 172.21.15.29:0/2141804964 conn(0x5650435867e0 msgr2=0x565043581a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:35.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.980+0000 7fd721c3a740 1 -- 172.21.15.29:0/2141804964 shutdown_connections 2024-03-11T19:18:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.980+0000 7fd721c3a740 1 -- 172.21.15.29:0/2141804964 wait complete. 2024-03-11T19:18:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.980+0000 7fd721c3a740 1 -- start start 2024-03-11T19:18:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.980+0000 7fd721c3a740 1 Processor -- start 2024-03-11T19:18:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.981+0000 7fd721c3a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56504358b270 0x56504359c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.981+0000 7fd721c3a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56504358bc60 0x56504359adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.981+0000 7fd721c3a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x565043590dd0 0x56504359b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.981+0000 7fd721c3a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56504359b950 con 0x56504358bc60 2024-03-11T19:18:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.981+0000 7fd721c3a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56504359bac0 con 0x565043590dd0 2024-03-11T19:18:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.981+0000 7fd721c3a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56504359bc80 con 0x56504358b270 2024-03-11T19:18:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.981+0000 7fd721c3a740 10 client.0 _ll_register_callbacks cb 0x7fd70c010550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.981+0000 7fd71d601640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x565043590dd0 0x56504359b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.981+0000 7fd71d601640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x565043590dd0 0x56504359b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:47312/0 (socket says 172.21.15.29:47312) 2024-03-11T19:18:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.981+0000 7fd71d601640 1 -- 172.21.15.29:0/317073479 learned_addr learned my addr 172.21.15.29:0/317073479 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:35.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.982+0000 7fd71d601640 1 -- 172.21.15.29:0/317073479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56504358b270 msgr2=0x56504359c730 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:18:35.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.982+0000 7fd71d601640 1 --2- 172.21.15.29:0/317073479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56504358b270 0x56504359c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.982+0000 7fd71d601640 1 -- 172.21.15.29:0/317073479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56504358bc60 msgr2=0x56504359adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:35.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.982+0000 7fd71d601640 1 --2- 172.21.15.29:0/317073479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56504358bc60 0x56504359adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.982+0000 7fd71d601640 1 -- 172.21.15.29:0/317073479 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5650434f26a0 con 0x565043590dd0 2024-03-11T19:18:35.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.982+0000 7fd71d601640 1 --2- 172.21.15.29:0/317073479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x565043590dd0 0x56504359b2f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd70802dcd0 tx=0x7fd70802d630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b68aba1ec15a7265 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:35.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.983+0000 7fd6f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:18:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.984+0000 7fd6f7fff640 1 -- 172.21.15.29:0/317073479 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7080239a0 con 0x565043590dd0 2024-03-11T19:18:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.984+0000 7fd721c3a740 1 -- 172.21.15.29:0/317073479 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd70c09adb0 con 0x565043590dd0 2024-03-11T19:18:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.984+0000 7fd6f7fff640 1 -- 172.21.15.29:0/317073479 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd708023e40 con 0x565043590dd0 2024-03-11T19:18:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.984+0000 7fd6f7fff640 1 -- 172.21.15.29:0/317073479 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd70802d820 con 0x565043590dd0 2024-03-11T19:18:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.984+0000 7fd721c3a740 10 client.26951 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.984+0000 7fd721c3a740 1 -- 172.21.15.29:0/317073479 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5650434b48c0 con 0x565043590dd0 2024-03-11T19:18:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.984+0000 7fd6f7fff640 1 -- 172.21.15.29:0/317073479 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd70c09adb0 con 0x565043590dd0 2024-03-11T19:18:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.984+0000 7fd6f7fff640 1 -- 172.21.15.29:0/317073479 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd708034030 con 0x565043590dd0 2024-03-11T19:18:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.984+0000 7fd721c3a740 10 client.26951 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.984+0000 7fd721c3a740 10 client.26951 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:18:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.984+0000 7fd721c3a740 20 client.26951 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.984+0000 7fd721c3a740 1 --2- 172.21.15.29:0/317073479 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5650435f92b0 0x5650435fb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.984+0000 7fd721c3a740 4 client.26951 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:18:35.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.984+0000 7fd721c3a740 1 -- 172.21.15.29:0/317073479 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56504327f890 con 0x5650435f92b0 2024-03-11T19:18:35.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.985+0000 7fd71ce00640 1 --2- 172.21.15.29:0/317073479 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5650435f92b0 0x5650435fb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:35.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.985+0000 7fd6f7fff640 10 client.26951 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:18:35.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.987+0000 7fd71ce00640 1 --2- 172.21.15.29:0/317073479 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5650435f92b0 0x5650435fb6d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1d6596ae7b7939bf server_cookie=d424aed418146f60 in_seq=0 out_seq=0 2024-03-11T19:18:35.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.987+0000 7fd6f7fff640 1 -- 172.21.15.29:0/317073479 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56504327f890 con 0x5650435f92b0 2024-03-11T19:18:35.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.987+0000 7fd6f7fff640 10 client.26951 handle_command_reply: tid=0 2024-03-11T19:18:35.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.987+0000 7fd721c3a740 1 client.26951 shutdown 2024-03-11T19:18:35.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.988+0000 7fd721c3a740 1 -- 172.21.15.29:0/317073479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x565043590dd0 msgr2=0x56504359b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:35.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.988+0000 7fd721c3a740 1 --2- 172.21.15.29:0/317073479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x565043590dd0 0x56504359b2f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd70802dcd0 tx=0x7fd70802d630 comp rx=0 tx=0).stop 2024-03-11T19:18:35.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.988+0000 7fd721c3a740 1 -- 172.21.15.29:0/317073479 shutdown_connections 2024-03-11T19:18:35.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.988+0000 7fd721c3a740 1 --2- 172.21.15.29:0/317073479 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5650435f92b0 0x5650435fb6d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.988+0000 7fd721c3a740 1 --2- 172.21.15.29:0/317073479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x565043590dd0 0x56504359b2f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.988+0000 7fd721c3a740 1 --2- 172.21.15.29:0/317073479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56504358bc60 0x56504359adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.988+0000 7fd721c3a740 1 --2- 172.21.15.29:0/317073479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56504358b270 0x56504359c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.988+0000 7fd721c3a740 1 -- 172.21.15.29:0/317073479 >> 172.21.15.29:0/317073479 conn(0x5650435867e0 msgr2=0x5650435953d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:35.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.988+0000 7fd6f7fff640 0 client.26951 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:18:35.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.988+0000 7fd721c3a740 1 -- 172.21.15.29:0/317073479 shutdown_connections 2024-03-11T19:18:35.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.988+0000 7fd721c3a740 1 -- 172.21.15.29:0/317073479 wait complete. 2024-03-11T19:18:35.993 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:18:35.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.989+0000 7fd721c3a740 20 client.26951 trim_cache size 0 max 16384 2024-03-11T19:18:35.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.992+0000 7fd71f064640 1 -- 172.21.15.29:0/562223542 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd7000a1c30 msgr2=0x7fd7000a40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:35.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.992+0000 7fd71f064640 1 --2- 172.21.15.29:0/562223542 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd7000a1c30 0x7fd7000a40f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x56504358c4c0 tx=0x7fd708008040 comp rx=0 tx=0).stop 2024-03-11T19:18:35.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.992+0000 7fd71f064640 1 -- 172.21.15.29:0/562223542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd718075f10 msgr2=0x7fd7181bbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:35.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.992+0000 7fd71f064640 1 --2- 172.21.15.29:0/562223542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd718075f10 0x7fd7181bbdf0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00e790 tx=0x7fd70c00ec50 comp rx=0 tx=0).stop 2024-03-11T19:18:35.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.992+0000 7fd71f064640 1 -- 172.21.15.29:0/562223542 shutdown_connections 2024-03-11T19:18:35.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.992+0000 7fd71f064640 1 --2- 172.21.15.29:0/562223542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7180788c0 0x7fd7181be340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.992+0000 7fd71f064640 1 --2- 172.21.15.29:0/562223542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd718075f10 0x7fd7181bbdf0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.992+0000 7fd71f064640 1 --2- 172.21.15.29:0/562223542 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd7000a1c30 0x7fd7000a40f0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.992+0000 7fd71f064640 1 --2- 172.21.15.29:0/562223542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd718073030 0x7fd7181b98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:35.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.992+0000 7fd71f064640 1 -- 172.21.15.29:0/562223542 >> 172.21.15.29:0/562223542 conn(0x7fd71806dfe0 msgr2=0x7fd718076be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:35.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.995+0000 7fd71f064640 1 -- 172.21.15.29:0/562223542 shutdown_connections 2024-03-11T19:18:35.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:35.995+0000 7fd71f064640 1 -- 172.21.15.29:0/562223542 wait complete. 2024-03-11T19:18:36.019 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:18:36.019 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:18:36.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.202+0000 7fe391379640 1 -- 172.21.15.29:0/3081616316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe38c104f70 msgr2=0x7fe38c10b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.202+0000 7fe391379640 1 --2- 172.21.15.29:0/3081616316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe38c104f70 0x7fe38c10b800 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fe384009ca0 tx=0x7fe38402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.202+0000 7fe391379640 1 -- 172.21.15.29:0/3081616316 shutdown_connections 2024-03-11T19:18:36.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.202+0000 7fe391379640 1 --2- 172.21.15.29:0/3081616316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe38c104f70 0x7fe38c10b800 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.202+0000 7fe391379640 1 --2- 172.21.15.29:0/3081616316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe38c1045b0 0x7fe38c104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.202+0000 7fe391379640 1 --2- 172.21.15.29:0/3081616316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe38c1033c0 0x7fe38c1037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.202+0000 7fe391379640 1 -- 172.21.15.29:0/3081616316 >> 172.21.15.29:0/3081616316 conn(0x7fe38c0fe930 msgr2=0x7fe38c100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:36.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.202+0000 7fe391379640 1 -- 172.21.15.29:0/3081616316 shutdown_connections 2024-03-11T19:18:36.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.202+0000 7fe391379640 1 -- 172.21.15.29:0/3081616316 wait complete. 2024-03-11T19:18:36.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.203+0000 7fe391379640 1 Processor -- start 2024-03-11T19:18:36.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.203+0000 7fe391379640 1 -- start start 2024-03-11T19:18:36.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.204+0000 7fe391379640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe38c1033c0 0x7fe38c199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.204+0000 7fe391379640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe38c1045b0 0x7fe38c199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.204+0000 7fe391379640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe38c104f70 0x7fe38c1a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.204+0000 7fe391379640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c1a0e30 con 0x7fe38c104f70 2024-03-11T19:18:36.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.204+0000 7fe391379640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c1a0fa0 con 0x7fe38c1033c0 2024-03-11T19:18:36.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.204+0000 7fe391379640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c1a1160 con 0x7fe38c1045b0 2024-03-11T19:18:36.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.204+0000 7fe38b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe38c104f70 0x7fe38c1a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.204+0000 7fe38b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe38c104f70 0x7fe38c1a08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51678/0 (socket says 172.21.15.29:51678) 2024-03-11T19:18:36.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.204+0000 7fe38affd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe38c1033c0 0x7fe38c199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.204+0000 7fe38b7fe640 1 -- 172.21.15.29:0/2288864021 learned_addr learned my addr 172.21.15.29:0/2288864021 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.204+0000 7fe38a7fc640 1 --2- 172.21.15.29:0/2288864021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe38c1045b0 0x7fe38c199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.204+0000 7fe38b7fe640 1 -- 172.21.15.29:0/2288864021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe38c1045b0 msgr2=0x7fe38c199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.204+0000 7fe38b7fe640 1 --2- 172.21.15.29:0/2288864021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe38c1045b0 0x7fe38c199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.204+0000 7fe38b7fe640 1 -- 172.21.15.29:0/2288864021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe38c1033c0 msgr2=0x7fe38c199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.204+0000 7fe38b7fe640 1 --2- 172.21.15.29:0/2288864021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe38c1033c0 0x7fe38c199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.204+0000 7fe38b7fe640 1 -- 172.21.15.29:0/2288864021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe384009900 con 0x7fe38c104f70 2024-03-11T19:18:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.204+0000 7fe38a7fc640 1 --2- 172.21.15.29:0/2288864021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe38c1045b0 0x7fe38c199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:18:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.204+0000 7fe38affd640 1 --2- 172.21.15.29:0/2288864021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe38c1033c0 0x7fe38c199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:18:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.205+0000 7fe38b7fe640 1 --2- 172.21.15.29:0/2288864021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe38c104f70 0x7fe38c1a08f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fe38402c630 tx=0x7fe384032900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.205+0000 7fe373fff640 1 -- 172.21.15.29:0/2288864021 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe384034e70 con 0x7fe38c104f70 2024-03-11T19:18:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.205+0000 7fe391379640 1 -- 172.21.15.29:0/2288864021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe38c1a13e0 con 0x7fe38c104f70 2024-03-11T19:18:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.205+0000 7fe391379640 1 -- 172.21.15.29:0/2288864021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe38c1a18e0 con 0x7fe38c104f70 2024-03-11T19:18:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.205+0000 7fe373fff640 1 -- 172.21.15.29:0/2288864021 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe384003e20 con 0x7fe38c104f70 2024-03-11T19:18:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.205+0000 7fe373fff640 1 -- 172.21.15.29:0/2288864021 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe384004700 con 0x7fe38c104f70 2024-03-11T19:18:36.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.208+0000 7fe394020740 1 Processor -- start 2024-03-11T19:18:36.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.208+0000 7fe373fff640 1 -- 172.21.15.29:0/2288864021 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe38403b040 con 0x7fe38c104f70 2024-03-11T19:18:36.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.208+0000 7fe394020740 1 -- start start 2024-03-11T19:18:36.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.208+0000 7fe394020740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a666ca80 0x5579a666cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.208+0000 7fe394020740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a666d470 0x5579a6671f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.208+0000 7fe394020740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a66725e0 0x5579a6674a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.208+0000 7fe394020740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579a6620960 con 0x5579a66725e0 2024-03-11T19:18:36.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.208+0000 7fe394020740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579a65d4310 con 0x5579a666d470 2024-03-11T19:18:36.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.208+0000 7fe394020740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579a65d1840 con 0x5579a666ca80 2024-03-11T19:18:36.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.209+0000 7fe38a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a66725e0 0x5579a6674a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.209+0000 7fe38affd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a666d470 0x5579a6671f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.209+0000 7fe38affd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a666d470 0x5579a6671f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:47344/0 (socket says 172.21.15.29:47344) 2024-03-11T19:18:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.209+0000 7fe38affd640 1 -- 172.21.15.29:0/1409806144 learned_addr learned my addr 172.21.15.29:0/1409806144 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.209+0000 7fe38a7fc640 1 --2- 172.21.15.29:0/1409806144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a666ca80 0x5579a666cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.209+0000 7fe38a7fc640 1 -- 172.21.15.29:0/1409806144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a666ca80 msgr2=0x5579a666cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.209+0000 7fe38a7fc640 1 --2- 172.21.15.29:0/1409806144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a666ca80 0x5579a666cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.209+0000 7fe38a7fc640 1 -- 172.21.15.29:0/1409806144 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a666d470 msgr2=0x5579a6671f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.209+0000 7fe38a7fc640 1 --2- 172.21.15.29:0/1409806144 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a666d470 0x5579a6671f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.209+0000 7fe38a7fc640 1 -- 172.21.15.29:0/1409806144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5579a658f060 con 0x5579a66725e0 2024-03-11T19:18:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.209+0000 7fe38a7fc640 1 --2- 172.21.15.29:0/1409806144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a66725e0 0x5579a6674a70 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fe3800058c0 tx=0x7fe380005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1005633c6356f713 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.209+0000 7fe373fff640 1 --2- 172.21.15.29:0/2288864021 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3740a1a30 0x7fe3740a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.209+0000 7fe3717fa640 1 -- 172.21.15.29:0/1409806144 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380011050 con 0x5579a66725e0 2024-03-11T19:18:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.209+0000 7fe38affd640 1 --2- 172.21.15.29:0/2288864021 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3740a1a30 0x7fe3740a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.210+0000 7fe3717fa640 1 -- 172.21.15.29:0/1409806144 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe3800054d0 con 0x5579a66725e0 2024-03-11T19:18:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.210+0000 7fe3717fa640 1 -- 172.21.15.29:0/1409806144 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3800047a0 con 0x5579a66725e0 2024-03-11T19:18:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.210+0000 7fe373fff640 1 -- 172.21.15.29:0/2288864021 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe3840efbd0 con 0x7fe38c104f70 2024-03-11T19:18:36.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.210+0000 7fe394020740 1 -- 172.21.15.29:0/1409806144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a66725e0 msgr2=0x5579a6674a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.210+0000 7fe394020740 1 --2- 172.21.15.29:0/1409806144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a66725e0 0x5579a6674a70 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fe3800058c0 tx=0x7fe380005850 comp rx=0 tx=0).stop 2024-03-11T19:18:36.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.210+0000 7fe38affd640 1 --2- 172.21.15.29:0/2288864021 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3740a1a30 0x7fe3740a3ef0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x5579a666d870 tx=0x7fe37c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:36.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.210+0000 7fe394020740 1 -- 172.21.15.29:0/1409806144 shutdown_connections 2024-03-11T19:18:36.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.210+0000 7fe394020740 1 --2- 172.21.15.29:0/1409806144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a66725e0 0x5579a6674a70 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.210+0000 7fe394020740 1 --2- 172.21.15.29:0/1409806144 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a666d470 0x5579a6671f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.210+0000 7fe394020740 1 --2- 172.21.15.29:0/1409806144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a666ca80 0x5579a666cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.211+0000 7fe394020740 1 -- 172.21.15.29:0/1409806144 >> 172.21.15.29:0/1409806144 conn(0x5579a6667ff0 msgr2=0x5579a66683f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:36.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.211+0000 7fe394020740 1 -- 172.21.15.29:0/1409806144 shutdown_connections 2024-03-11T19:18:36.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.211+0000 7fe394020740 1 -- 172.21.15.29:0/1409806144 wait complete. 2024-03-11T19:18:36.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.211+0000 7fe394020740 1 -- start start 2024-03-11T19:18:36.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.211+0000 7fe394020740 1 Processor -- start 2024-03-11T19:18:36.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.211+0000 7fe394020740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a666ca80 0x5579a6676ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.211+0000 7fe394020740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a666d470 0x5579a6677220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.211+0000 7fe394020740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a66725e0 0x5579a667e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.211+0000 7fe394020740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579a65cd020 con 0x5579a666d470 2024-03-11T19:18:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.211+0000 7fe394020740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579a62a0430 con 0x5579a666ca80 2024-03-11T19:18:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.211+0000 7fe394020740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579a625ce00 con 0x5579a66725e0 2024-03-11T19:18:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.211+0000 7fe394020740 10 client.0 _ll_register_callbacks cb 0x5579a6574cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.212+0000 7fe38a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a666d470 0x5579a6677220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.212+0000 7fe38a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a666d470 0x5579a6677220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51696/0 (socket says 172.21.15.29:51696) 2024-03-11T19:18:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.212+0000 7fe38a7fc640 1 -- 172.21.15.29:0/1866613212 learned_addr learned my addr 172.21.15.29:0/1866613212 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.212+0000 7fe38affd640 1 --2- 172.21.15.29:0/1866613212 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a666ca80 0x5579a6676ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.212+0000 7fe38b7fe640 1 --2- 172.21.15.29:0/1866613212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a66725e0 0x5579a667e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.212+0000 7fe38a7fc640 1 -- 172.21.15.29:0/1866613212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a66725e0 msgr2=0x5579a667e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.212+0000 7fe38a7fc640 1 --2- 172.21.15.29:0/1866613212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a66725e0 0x5579a667e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.212+0000 7fe38a7fc640 1 -- 172.21.15.29:0/1866613212 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a666ca80 msgr2=0x5579a6676ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.212+0000 7fe38a7fc640 1 --2- 172.21.15.29:0/1866613212 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a666ca80 0x5579a6676ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.212+0000 7fe38a7fc640 1 -- 172.21.15.29:0/1866613212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe380019040 con 0x5579a666d470 2024-03-11T19:18:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.212+0000 7fe38b7fe640 1 --2- 172.21.15.29:0/1866613212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a66725e0 0x5579a667e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:18:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.212+0000 7fe38affd640 1 --2- 172.21.15.29:0/1866613212 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a666ca80 0x5579a6676ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:18:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.212+0000 7fe38a7fc640 1 --2- 172.21.15.29:0/1866613212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a666d470 0x5579a6677220 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe380005f90 tx=0x7fe380005fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a446e511c3b8e141 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.212+0000 7fe36eff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:18:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.212+0000 7fe36eff5640 1 -- 172.21.15.29:0/1866613212 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380024070 con 0x5579a666d470 2024-03-11T19:18:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.212+0000 7fe394020740 1 -- 172.21.15.29:0/1866613212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5579a62cc7e0 con 0x5579a666d470 2024-03-11T19:18:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.212+0000 7fe36eff5640 1 -- 172.21.15.29:0/1866613212 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe380000e40 con 0x5579a666d470 2024-03-11T19:18:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.212+0000 7fe36eff5640 1 -- 172.21.15.29:0/1866613212 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe380005640 con 0x5579a666d470 2024-03-11T19:18:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.213+0000 7fe394020740 10 client.17487 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.213+0000 7fe394020740 1 -- 172.21.15.29:0/1866613212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5579a65cb6a0 con 0x5579a666d470 2024-03-11T19:18:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.213+0000 7fe36eff5640 1 -- 172.21.15.29:0/1866613212 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5579a62cc7e0 con 0x5579a666d470 2024-03-11T19:18:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.213+0000 7fe36eff5640 1 -- 172.21.15.29:0/1866613212 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe38001d070 con 0x5579a666d470 2024-03-11T19:18:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.213+0000 7fe394020740 10 client.17487 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.213+0000 7fe394020740 10 client.17487 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:18:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.213+0000 7fe394020740 20 client.17487 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.213+0000 7fe394020740 1 --2- 172.21.15.29:0/1866613212 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5579a6682c70 0x5579a6685090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.213+0000 7fe394020740 4 client.17487 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:18:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.213+0000 7fe394020740 1 -- 172.21.15.29:0/1866613212 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5579a6309720 con 0x5579a6682c70 2024-03-11T19:18:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.213+0000 7fe38affd640 1 --2- 172.21.15.29:0/1866613212 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5579a6682c70 0x5579a6685090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.214+0000 7fe36eff5640 10 client.17487 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:18:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.214+0000 7fe38affd640 1 --2- 172.21.15.29:0/1866613212 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5579a6682c70 0x5579a6685090 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8c038eab4b0b1997 server_cookie=acf5d938cd3bb415 in_seq=0 out_seq=0 2024-03-11T19:18:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.215+0000 7fe36eff5640 1 -- 172.21.15.29:0/1866613212 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5579a6309720 con 0x5579a6682c70 2024-03-11T19:18:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.215+0000 7fe36eff5640 10 client.17487 handle_command_reply: tid=0 2024-03-11T19:18:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.215+0000 7fe394020740 1 client.17487 shutdown 2024-03-11T19:18:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.215+0000 7fe394020740 1 -- 172.21.15.29:0/1866613212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a666d470 msgr2=0x5579a6677220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.215+0000 7fe394020740 1 --2- 172.21.15.29:0/1866613212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a666d470 0x5579a6677220 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe380005f90 tx=0x7fe380005fc0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.215+0000 7fe394020740 1 -- 172.21.15.29:0/1866613212 shutdown_connections 2024-03-11T19:18:36.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.215+0000 7fe394020740 1 --2- 172.21.15.29:0/1866613212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a66725e0 0x5579a667e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.215+0000 7fe394020740 1 --2- 172.21.15.29:0/1866613212 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5579a6682c70 0x5579a6685090 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.215+0000 7fe394020740 1 --2- 172.21.15.29:0/1866613212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a666d470 0x5579a6677220 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.215+0000 7fe394020740 1 --2- 172.21.15.29:0/1866613212 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a666ca80 0x5579a6676ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.215+0000 7fe394020740 1 -- 172.21.15.29:0/1866613212 >> 172.21.15.29:0/1866613212 conn(0x5579a6667ff0 msgr2=0x5579a6669240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:36.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.215+0000 7fe36eff5640 0 client.17487 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:18:36.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.215+0000 7fe394020740 1 -- 172.21.15.29:0/1866613212 shutdown_connections 2024-03-11T19:18:36.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.215+0000 7fe394020740 1 -- 172.21.15.29:0/1866613212 wait complete. 2024-03-11T19:18:36.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.216+0000 7fe394020740 20 client.17487 trim_cache size 0 max 16384 2024-03-11T19:18:36.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.234+0000 7fe394020740 1 Processor -- start 2024-03-11T19:18:36.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.234+0000 7fe394020740 1 -- start start 2024-03-11T19:18:36.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.234+0000 7fe394020740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a666ca80 0x5579a6676ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.234+0000 7fe394020740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a666d470 0x5579a6677220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.234+0000 7fe394020740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a66725e0 0x5579a667e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.234+0000 7fe394020740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579a6552130 con 0x5579a66725e0 2024-03-11T19:18:36.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.234+0000 7fe394020740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579a62cc7e0 con 0x5579a666ca80 2024-03-11T19:18:36.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.234+0000 7fe394020740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579a66750a0 con 0x5579a666d470 2024-03-11T19:18:36.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.234+0000 7fe38b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a66725e0 0x5579a667e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.234+0000 7fe38b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a66725e0 0x5579a667e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51710/0 (socket says 172.21.15.29:51710) 2024-03-11T19:18:36.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.234+0000 7fe38b7fe640 1 -- 172.21.15.29:0/429389138 learned_addr learned my addr 172.21.15.29:0/429389138 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:36.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.234+0000 7fe38affd640 1 --2- 172.21.15.29:0/429389138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a666ca80 0x5579a6676ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.234+0000 7fe38a7fc640 1 --2- 172.21.15.29:0/429389138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a666d470 0x5579a6677220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.234+0000 7fe38b7fe640 1 -- 172.21.15.29:0/429389138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a666d470 msgr2=0x5579a6677220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.234+0000 7fe38b7fe640 1 --2- 172.21.15.29:0/429389138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a666d470 0x5579a6677220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.235+0000 7fe38b7fe640 1 -- 172.21.15.29:0/429389138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a666ca80 msgr2=0x5579a6676ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.235+0000 7fe38b7fe640 1 --2- 172.21.15.29:0/429389138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a666ca80 0x5579a6676ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.235+0000 7fe38b7fe640 1 -- 172.21.15.29:0/429389138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5579a6567c20 con 0x5579a66725e0 2024-03-11T19:18:36.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.235+0000 7fe38b7fe640 1 --2- 172.21.15.29:0/429389138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a66725e0 0x5579a667e220 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fe3840bc990 tx=0x7fe384003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a2a83c011b71b0e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:36.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.235+0000 7fe370ff9640 1 -- 172.21.15.29:0/429389138 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3840bcc90 con 0x5579a66725e0 2024-03-11T19:18:36.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.235+0000 7fe370ff9640 1 -- 172.21.15.29:0/429389138 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe3840c0d30 con 0x5579a66725e0 2024-03-11T19:18:36.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.235+0000 7fe370ff9640 1 -- 172.21.15.29:0/429389138 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3840bcef0 con 0x5579a66725e0 2024-03-11T19:18:36.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.236+0000 7fe394020740 1 -- 172.21.15.29:0/429389138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a66725e0 msgr2=0x5579a667e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.236+0000 7fe394020740 1 --2- 172.21.15.29:0/429389138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a66725e0 0x5579a667e220 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fe3840bc990 tx=0x7fe384003f90 comp rx=0 tx=0).stop 2024-03-11T19:18:36.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.236+0000 7fe394020740 1 -- 172.21.15.29:0/429389138 shutdown_connections 2024-03-11T19:18:36.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.236+0000 7fe394020740 1 --2- 172.21.15.29:0/429389138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a66725e0 0x5579a667e220 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.236+0000 7fe394020740 1 --2- 172.21.15.29:0/429389138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a666d470 0x5579a6677220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.236+0000 7fe394020740 1 --2- 172.21.15.29:0/429389138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a666ca80 0x5579a6676ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.236+0000 7fe394020740 1 -- 172.21.15.29:0/429389138 >> 172.21.15.29:0/429389138 conn(0x5579a6667ff0 msgr2=0x5579a66683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:36.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.236+0000 7fe394020740 1 -- 172.21.15.29:0/429389138 shutdown_connections 2024-03-11T19:18:36.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.236+0000 7fe394020740 1 -- 172.21.15.29:0/429389138 wait complete. 2024-03-11T19:18:36.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.236+0000 7fe394020740 1 -- start start 2024-03-11T19:18:36.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.236+0000 7fe394020740 1 Processor -- start 2024-03-11T19:18:36.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe394020740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a666ca80 0x5579a667c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe394020740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a666d470 0x5579a667c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe394020740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a66725e0 0x5579a667d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe394020740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579a667dc50 con 0x5579a666d470 2024-03-11T19:18:36.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe394020740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579a667ddc0 con 0x5579a66725e0 2024-03-11T19:18:36.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe394020740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579a667df80 con 0x5579a666ca80 2024-03-11T19:18:36.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe38affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a666ca80 0x5579a667c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe38affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a666ca80 0x5579a667c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45884/0 (socket says 172.21.15.29:45884) 2024-03-11T19:18:36.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe38a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a666d470 0x5579a667c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe38affd640 1 -- 172.21.15.29:0/3074993976 learned_addr learned my addr 172.21.15.29:0/3074993976 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:36.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe394020740 10 client.0 _ll_register_callbacks cb 0x7fe380009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:36.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe38b7fe640 1 --2- 172.21.15.29:0/3074993976 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a66725e0 0x5579a667d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe38affd640 1 -- 172.21.15.29:0/3074993976 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a66725e0 msgr2=0x5579a667d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe38affd640 1 --2- 172.21.15.29:0/3074993976 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a66725e0 0x5579a667d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe38affd640 1 -- 172.21.15.29:0/3074993976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a666d470 msgr2=0x5579a667c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe38affd640 1 --2- 172.21.15.29:0/3074993976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a666d470 0x5579a667c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe38affd640 1 -- 172.21.15.29:0/3074993976 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe38c1a18e0 con 0x5579a666ca80 2024-03-11T19:18:36.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe38a7fc640 1 --2- 172.21.15.29:0/3074993976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a666d470 0x5579a667c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:18:36.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe38affd640 1 --2- 172.21.15.29:0/3074993976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a666ca80 0x5579a667c230 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x5579a666dee0 tx=0x7fe37c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9e047757f699ae9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:36.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.237+0000 7fe36eff5640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:18:36.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.238+0000 7fe36eff5640 1 -- 172.21.15.29:0/3074993976 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c025040 con 0x5579a666ca80 2024-03-11T19:18:36.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.238+0000 7fe394020740 1 -- 172.21.15.29:0/3074993976 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe3800054d0 con 0x5579a666ca80 2024-03-11T19:18:36.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.238+0000 7fe36eff5640 1 -- 172.21.15.29:0/3074993976 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe37c00f920 con 0x5579a666ca80 2024-03-11T19:18:36.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.238+0000 7fe36eff5640 1 -- 172.21.15.29:0/3074993976 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe37c017070 con 0x5579a666ca80 2024-03-11T19:18:36.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.238+0000 7fe394020740 10 client.27004 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:36.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.238+0000 7fe394020740 1 -- 172.21.15.29:0/3074993976 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5579a658d8c0 con 0x5579a666ca80 2024-03-11T19:18:36.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.238+0000 7fe36eff5640 1 -- 172.21.15.29:0/3074993976 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe3800054d0 con 0x5579a666ca80 2024-03-11T19:18:36.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.238+0000 7fe36eff5640 1 -- 172.21.15.29:0/3074993976 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe37c01fce0 con 0x5579a666ca80 2024-03-11T19:18:36.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.238+0000 7fe394020740 10 client.27004 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:36.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.238+0000 7fe394020740 10 client.27004 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:18:36.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.238+0000 7fe394020740 20 client.27004 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:36.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.238+0000 7fe394020740 1 --2- 172.21.15.29:0/3074993976 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5579a66da430 0x5579a66dc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.238+0000 7fe394020740 4 client.27004 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:18:36.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.238+0000 7fe394020740 1 -- 172.21.15.29:0/3074993976 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5579a6358890 con 0x5579a66da430 2024-03-11T19:18:36.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.238+0000 7fe38a7fc640 1 --2- 172.21.15.29:0/3074993976 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5579a66da430 0x5579a66dc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.239+0000 7fe36eff5640 10 client.27004 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:18:36.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.239+0000 7fe38a7fc640 1 --2- 172.21.15.29:0/3074993976 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5579a66da430 0x5579a66dc850 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5e9d2447c2febc76 server_cookie=37a5b4ae615c4252 in_seq=0 out_seq=0 2024-03-11T19:18:36.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.239+0000 7fe36eff5640 1 -- 172.21.15.29:0/3074993976 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5579a6358890 con 0x5579a66da430 2024-03-11T19:18:36.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.239+0000 7fe36eff5640 10 client.27004 handle_command_reply: tid=0 2024-03-11T19:18:36.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.239+0000 7fe394020740 1 client.27004 shutdown 2024-03-11T19:18:36.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.239+0000 7fe394020740 1 -- 172.21.15.29:0/3074993976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a666ca80 msgr2=0x5579a667c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.239+0000 7fe394020740 1 --2- 172.21.15.29:0/3074993976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a666ca80 0x5579a667c230 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x5579a666dee0 tx=0x7fe37c0046c0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.239+0000 7fe394020740 1 -- 172.21.15.29:0/3074993976 shutdown_connections 2024-03-11T19:18:36.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.239+0000 7fe394020740 1 --2- 172.21.15.29:0/3074993976 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579a66725e0 0x5579a667d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.239+0000 7fe394020740 1 --2- 172.21.15.29:0/3074993976 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5579a66da430 0x5579a66dc850 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.239+0000 7fe36eff5640 0 client.27004 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:18:36.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.239+0000 7fe394020740 1 --2- 172.21.15.29:0/3074993976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5579a666d470 0x5579a667c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.239+0000 7fe394020740 1 --2- 172.21.15.29:0/3074993976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5579a666ca80 0x5579a667c230 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.239+0000 7fe394020740 1 -- 172.21.15.29:0/3074993976 >> 172.21.15.29:0/3074993976 conn(0x5579a6667ff0 msgr2=0x5579a666ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:36.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.240+0000 7fe394020740 1 -- 172.21.15.29:0/3074993976 shutdown_connections 2024-03-11T19:18:36.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.240+0000 7fe394020740 1 -- 172.21.15.29:0/3074993976 wait complete. 2024-03-11T19:18:36.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.240+0000 7fe394020740 20 client.27004 trim_cache size 0 max 16384 2024-03-11T19:18:36.249 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:18:36.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.240+0000 7fe391379640 1 -- 172.21.15.29:0/2288864021 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3740a1a30 msgr2=0x7fe3740a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.240+0000 7fe391379640 1 --2- 172.21.15.29:0/2288864021 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3740a1a30 0x7fe3740a3ef0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x5579a666d870 tx=0x7fe37c005f90 comp rx=0 tx=0).stop 2024-03-11T19:18:36.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.240+0000 7fe391379640 1 -- 172.21.15.29:0/2288864021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe38c104f70 msgr2=0x7fe38c1a08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.240+0000 7fe391379640 1 --2- 172.21.15.29:0/2288864021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe38c104f70 0x7fe38c1a08f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fe38402c630 tx=0x7fe384032900 comp rx=0 tx=0).stop 2024-03-11T19:18:36.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.240+0000 7fe391379640 1 -- 172.21.15.29:0/2288864021 shutdown_connections 2024-03-11T19:18:36.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.240+0000 7fe391379640 1 --2- 172.21.15.29:0/2288864021 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3740a1a30 0x7fe3740a3ef0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.240+0000 7fe391379640 1 --2- 172.21.15.29:0/2288864021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe38c104f70 0x7fe38c1a08f0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.240+0000 7fe391379640 1 --2- 172.21.15.29:0/2288864021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe38c1045b0 0x7fe38c199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.240+0000 7fe391379640 1 --2- 172.21.15.29:0/2288864021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe38c1033c0 0x7fe38c199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.240+0000 7fe391379640 1 -- 172.21.15.29:0/2288864021 >> 172.21.15.29:0/2288864021 conn(0x7fe38c0fe930 msgr2=0x7fe38c107980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:36.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.241+0000 7fe391379640 1 -- 172.21.15.29:0/2288864021 shutdown_connections 2024-03-11T19:18:36.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.241+0000 7fe391379640 1 -- 172.21.15.29:0/2288864021 wait complete. 2024-03-11T19:18:36.258 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:18:36.258 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:18:36.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.407+0000 7f786b881640 1 -- 172.21.15.29:0/3720132988 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7864075f10 msgr2=0x7f7864078380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.407+0000 7f786b881640 1 --2- 172.21.15.29:0/3720132988 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7864075f10 0x7f7864078380 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f785c009be0 tx=0x7f785c02bf30 comp rx=0 tx=0).stop 2024-03-11T19:18:36.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.410+0000 7f786b881640 1 -- 172.21.15.29:0/3720132988 shutdown_connections 2024-03-11T19:18:36.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.410+0000 7f786b881640 1 --2- 172.21.15.29:0/3720132988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78640788c0 0x7f786407ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.410+0000 7f786b881640 1 --2- 172.21.15.29:0/3720132988 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7864075f10 0x7f7864078380 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.410+0000 7f786b881640 1 --2- 172.21.15.29:0/3720132988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7864073030 0x7f7864073450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.410+0000 7f786b881640 1 -- 172.21.15.29:0/3720132988 >> 172.21.15.29:0/3720132988 conn(0x7f786406dfe0 msgr2=0x7f7864070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:36.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.414+0000 7f786b881640 1 -- 172.21.15.29:0/3720132988 shutdown_connections 2024-03-11T19:18:36.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.415+0000 7f786b881640 1 -- 172.21.15.29:0/3720132988 wait complete. 2024-03-11T19:18:36.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.415+0000 7f786b881640 1 Processor -- start 2024-03-11T19:18:36.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.416+0000 7f786b881640 1 -- start start 2024-03-11T19:18:36.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.416+0000 7f786b881640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7864073030 0x7f78641b10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.416+0000 7f786b881640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7864075f10 0x7f78641b3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.416+0000 7f786b881640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78640788c0 0x7f78641b5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.416+0000 7f786b881640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78641b6300 con 0x7f7864073030 2024-03-11T19:18:36.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.416+0000 7f786b881640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78641b6470 con 0x7f7864075f10 2024-03-11T19:18:36.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.416+0000 7f786b881640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78641b6630 con 0x7f78640788c0 2024-03-11T19:18:36.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.417+0000 7f7869e1e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78640788c0 0x7f78641b5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.417+0000 7f7869e1e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78640788c0 0x7f78641b5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45894/0 (socket says 172.21.15.29:45894) 2024-03-11T19:18:36.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.417+0000 7f7869e1e640 1 -- 172.21.15.29:0/3735109101 learned_addr learned my addr 172.21.15.29:0/3735109101 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:36.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.419+0000 7f7868e1c640 1 --2- 172.21.15.29:0/3735109101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7864075f10 0x7f78641b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.421+0000 7f7869e1e640 1 -- 172.21.15.29:0/3735109101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7864075f10 msgr2=0x7f78641b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.421+0000 7f7869e1e640 1 --2- 172.21.15.29:0/3735109101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7864075f10 0x7f78641b3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.421+0000 7f7869e1e640 1 -- 172.21.15.29:0/3735109101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7864073030 msgr2=0x7f78641b10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.421+0000 7f7869e1e640 1 --2- 172.21.15.29:0/3735109101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7864073030 0x7f78641b10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.421+0000 7f7869e1e640 1 -- 172.21.15.29:0/3735109101 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7854009560 con 0x7f78640788c0 2024-03-11T19:18:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.421+0000 7f7869e1e640 1 --2- 172.21.15.29:0/3735109101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78640788c0 0x7f78641b5b70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f786000c7a0 tx=0x7f786000cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.421+0000 7f785a7fc640 1 -- 172.21.15.29:0/3735109101 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7860007bb0 con 0x7f78640788c0 2024-03-11T19:18:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.422+0000 7f785a7fc640 1 -- 172.21.15.29:0/3735109101 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7860007d10 con 0x7f78640788c0 2024-03-11T19:18:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.422+0000 7f785a7fc640 1 -- 172.21.15.29:0/3735109101 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7860004e10 con 0x7f78640788c0 2024-03-11T19:18:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.422+0000 7f786b881640 1 -- 172.21.15.29:0/3735109101 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f785c009840 con 0x7f78640788c0 2024-03-11T19:18:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.422+0000 7f786b881640 1 -- 172.21.15.29:0/3735109101 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78641b6c40 con 0x7f78640788c0 2024-03-11T19:18:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.426+0000 7f785a7fc640 1 -- 172.21.15.29:0/3735109101 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7860004f70 con 0x7f78640788c0 2024-03-11T19:18:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.427+0000 7f786e454740 1 Processor -- start 2024-03-11T19:18:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.427+0000 7f785a7fc640 1 --2- 172.21.15.29:0/3735109101 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f78500a1b60 0x7f78500a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.428+0000 7f785a7fc640 1 -- 172.21.15.29:0/3735109101 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f78600cd040 con 0x7f78640788c0 2024-03-11T19:18:36.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.428+0000 7f786e454740 1 -- start start 2024-03-11T19:18:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.428+0000 7f786e454740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e49e270 0x55877e49e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.428+0000 7f786e454740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49ec60 0x55877e4a3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.428+0000 7f786e454740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55877e4a3dd0 0x55877e4a6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.428+0000 7f786e454740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877e45a960 con 0x55877e4a3dd0 2024-03-11T19:18:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.428+0000 7f786e454740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877e40e310 con 0x55877e49e270 2024-03-11T19:18:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.428+0000 7f786e454740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877e40b840 con 0x55877e49ec60 2024-03-11T19:18:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.429+0000 7f7868e1c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49ec60 0x55877e4a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.429+0000 7f7868e1c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49ec60 0x55877e4a3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45896/0 (socket says 172.21.15.29:45896) 2024-03-11T19:18:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.429+0000 7f7868e1c640 1 -- 172.21.15.29:0/3551402471 learned_addr learned my addr 172.21.15.29:0/3551402471 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.429+0000 7f7868e1c640 1 -- 172.21.15.29:0/3551402471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e49e270 msgr2=0x55877e49e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.429+0000 7f7868e1c640 1 --2- 172.21.15.29:0/3551402471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e49e270 0x55877e49e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.429+0000 7f7868e1c640 1 -- 172.21.15.29:0/3551402471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55877e4a3dd0 msgr2=0x55877e4a6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.429+0000 7f7868e1c640 1 --2- 172.21.15.29:0/3551402471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55877e4a3dd0 0x55877e4a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.429+0000 7f7868e1c640 1 -- 172.21.15.29:0/3551402471 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55877e3c9060 con 0x55877e49ec60 2024-03-11T19:18:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.430+0000 7f7868e1c640 1 --2- 172.21.15.29:0/3551402471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49ec60 0x55877e4a3760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f785c035000 tx=0x7f785c02c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d464991d5f71c561 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:36.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.431+0000 7f784f7fe640 1 -- 172.21.15.29:0/3551402471 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c0335e0 con 0x55877e49ec60 2024-03-11T19:18:36.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.431+0000 7f7868e1c640 1 --2- 172.21.15.29:0/3735109101 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f78500a1b60 0x7f78500a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.431+0000 7f784f7fe640 1 -- 172.21.15.29:0/3551402471 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f785c003e20 con 0x55877e49ec60 2024-03-11T19:18:36.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.431+0000 7f784f7fe640 1 -- 172.21.15.29:0/3551402471 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c034690 con 0x55877e49ec60 2024-03-11T19:18:36.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.432+0000 7f786e454740 1 -- 172.21.15.29:0/3551402471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49ec60 msgr2=0x55877e4a3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.432+0000 7f786e454740 1 --2- 172.21.15.29:0/3551402471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49ec60 0x55877e4a3760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f785c035000 tx=0x7f785c02c920 comp rx=0 tx=0).stop 2024-03-11T19:18:36.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.433+0000 7f786e454740 1 -- 172.21.15.29:0/3551402471 shutdown_connections 2024-03-11T19:18:36.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.433+0000 7f786e454740 1 --2- 172.21.15.29:0/3551402471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55877e4a3dd0 0x55877e4a6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.433+0000 7f786e454740 1 --2- 172.21.15.29:0/3551402471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49ec60 0x55877e4a3760 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.433+0000 7f786e454740 1 --2- 172.21.15.29:0/3551402471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e49e270 0x55877e49e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.433+0000 7f786e454740 1 -- 172.21.15.29:0/3551402471 >> 172.21.15.29:0/3551402471 conn(0x55877e4997e0 msgr2=0x55877e499be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:36.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.433+0000 7f786e454740 1 -- 172.21.15.29:0/3551402471 shutdown_connections 2024-03-11T19:18:36.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.433+0000 7f786e454740 1 -- 172.21.15.29:0/3551402471 wait complete. 2024-03-11T19:18:36.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.433+0000 7f786e454740 1 -- start start 2024-03-11T19:18:36.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.433+0000 7f786e454740 1 Processor -- start 2024-03-11T19:18:36.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.434+0000 7f786e454740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49e270 0x55877e4a8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.434+0000 7f786e454740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55877e49ec60 0x55877e4a8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.434+0000 7f786e454740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e4a3dd0 0x55877e4afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.434+0000 7f786e454740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877e407020 con 0x55877e49ec60 2024-03-11T19:18:36.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.434+0000 7f786e454740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877e0da430 con 0x55877e4a3dd0 2024-03-11T19:18:36.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.434+0000 7f786e454740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877e096e00 con 0x55877e49e270 2024-03-11T19:18:36.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.434+0000 7f786e454740 10 client.0 _ll_register_callbacks cb 0x55877e3aecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:36.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.435+0000 7f786961d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49e270 0x55877e4a8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.435+0000 7f786961d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49e270 0x55877e4a8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45910/0 (socket says 172.21.15.29:45910) 2024-03-11T19:18:36.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.435+0000 7f786961d640 1 -- 172.21.15.29:0/118014736 learned_addr learned my addr 172.21.15.29:0/118014736 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.435+0000 7f786961d640 1 -- 172.21.15.29:0/118014736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e4a3dd0 msgr2=0x55877e4afa90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:18:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.435+0000 7f786961d640 1 --2- 172.21.15.29:0/118014736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e4a3dd0 0x55877e4afa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.435+0000 7f786961d640 1 -- 172.21.15.29:0/118014736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55877e49ec60 msgr2=0x55877e4a8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.435+0000 7f786961d640 1 --2- 172.21.15.29:0/118014736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55877e49ec60 0x55877e4a8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.435+0000 7f786961d640 1 -- 172.21.15.29:0/118014736 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f786407b780 con 0x55877e49e270 2024-03-11T19:18:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.435+0000 7f786961d640 1 --2- 172.21.15.29:0/118014736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49e270 0x55877e4a8510 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f785400e5d0 tx=0x7f785400ea90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d612cd02005ab4a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.436+0000 7f784cff9640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:18:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.436+0000 7f784cff9640 1 -- 172.21.15.29:0/118014736 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7854010900 con 0x55877e49e270 2024-03-11T19:18:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.436+0000 7f784cff9640 1 -- 172.21.15.29:0/118014736 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7854014070 con 0x55877e49e270 2024-03-11T19:18:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.436+0000 7f786e454740 1 -- 172.21.15.29:0/118014736 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55877e1067e0 con 0x55877e49e270 2024-03-11T19:18:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.437+0000 7f7868e1c640 1 --2- 172.21.15.29:0/3735109101 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f78500a1b60 0x7f78500a4020 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x55877e49f060 tx=0x7f785c02c920 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.437+0000 7f784cff9640 1 -- 172.21.15.29:0/118014736 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7854010a60 con 0x55877e49e270 2024-03-11T19:18:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.437+0000 7f786e454740 10 client.27016 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.437+0000 7f786e454740 1 -- 172.21.15.29:0/118014736 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55877e4056a0 con 0x55877e49e270 2024-03-11T19:18:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.441+0000 7f784cff9640 1 -- 172.21.15.29:0/118014736 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55877e1067e0 con 0x55877e49e270 2024-03-11T19:18:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.441+0000 7f784cff9640 1 -- 172.21.15.29:0/118014736 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f785401f020 con 0x55877e49e270 2024-03-11T19:18:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.441+0000 7f786e454740 10 client.27016 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.441+0000 7f786e454740 10 client.27016 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:18:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.441+0000 7f786e454740 20 client.27016 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.441+0000 7f786e454740 1 --2- 172.21.15.29:0/118014736 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55877e4b48f0 0x55877e4b6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.442+0000 7f786e454740 4 client.27016 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:18:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.442+0000 7f786e454740 1 -- 172.21.15.29:0/118014736 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55877e143720 con 0x55877e4b48f0 2024-03-11T19:18:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.442+0000 7f7868e1c640 1 --2- 172.21.15.29:0/118014736 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55877e4b48f0 0x55877e4b6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.442+0000 7f784cff9640 10 client.27016 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:18:36.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.443+0000 7f7868e1c640 1 --2- 172.21.15.29:0/118014736 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55877e4b48f0 0x55877e4b6d10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=284eb5ba6dbfa40e server_cookie=4cd04d9acbedd38c in_seq=0 out_seq=0 2024-03-11T19:18:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.446+0000 7f784cff9640 1 -- 172.21.15.29:0/118014736 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55877e143720 con 0x55877e4b48f0 2024-03-11T19:18:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.446+0000 7f784cff9640 10 client.27016 handle_command_reply: tid=0 2024-03-11T19:18:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.446+0000 7f786e454740 1 client.27016 shutdown 2024-03-11T19:18:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.447+0000 7f786e454740 1 -- 172.21.15.29:0/118014736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49e270 msgr2=0x55877e4a8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.447+0000 7f786e454740 1 --2- 172.21.15.29:0/118014736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49e270 0x55877e4a8510 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f785400e5d0 tx=0x7f785400ea90 comp rx=0 tx=0).stop 2024-03-11T19:18:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.447+0000 7f786e454740 1 -- 172.21.15.29:0/118014736 shutdown_connections 2024-03-11T19:18:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.447+0000 7f786e454740 1 --2- 172.21.15.29:0/118014736 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55877e4b48f0 0x55877e4b6d10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.447+0000 7f786e454740 1 --2- 172.21.15.29:0/118014736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e4a3dd0 0x55877e4afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.447+0000 7f786e454740 1 --2- 172.21.15.29:0/118014736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55877e49ec60 0x55877e4a8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.447+0000 7f786e454740 1 --2- 172.21.15.29:0/118014736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49e270 0x55877e4a8510 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.447+0000 7f786e454740 1 -- 172.21.15.29:0/118014736 >> 172.21.15.29:0/118014736 conn(0x55877e4997e0 msgr2=0x55877e498b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.447+0000 7f784cff9640 0 client.27016 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:18:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.451+0000 7f786e454740 1 -- 172.21.15.29:0/118014736 shutdown_connections 2024-03-11T19:18:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.451+0000 7f786e454740 1 -- 172.21.15.29:0/118014736 wait complete. 2024-03-11T19:18:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.451+0000 7f786e454740 20 client.27016 trim_cache size 0 max 16384 2024-03-11T19:18:36.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.480+0000 7f786e454740 1 Processor -- start 2024-03-11T19:18:36.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.482+0000 7f786e454740 1 -- start start 2024-03-11T19:18:36.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.482+0000 7f786e454740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55877e49e270 0x55877e4a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.483+0000 7f786e454740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e49ec60 0x55877e4a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.483+0000 7f7868e1c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e49ec60 0x55877e4a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.483+0000 7f7868e1c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e49ec60 0x55877e4a06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:47424/0 (socket says 172.21.15.29:47424) 2024-03-11T19:18:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.483+0000 7f786961d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55877e49e270 0x55877e4a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.484+0000 7f786961d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55877e49e270 0x55877e4a01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51760/0 (socket says 172.21.15.29:51760) 2024-03-11T19:18:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.484+0000 7f786e454740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e4a3dd0 0x55877e4afa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.484+0000 7f7868e1c640 1 -- 172.21.15.29:0/1806066347 learned_addr learned my addr 172.21.15.29:0/1806066347 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.484+0000 7f786e454740 1 -- 172.21.15.29:0/1806066347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877e38c130 con 0x55877e49e270 2024-03-11T19:18:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.484+0000 7f786e454740 1 -- 172.21.15.29:0/1806066347 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877e143720 con 0x55877e49ec60 2024-03-11T19:18:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.484+0000 7f786e454740 1 -- 172.21.15.29:0/1806066347 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877e1067e0 con 0x55877e4a3dd0 2024-03-11T19:18:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.484+0000 7f7869e1e640 1 --2- 172.21.15.29:0/1806066347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e4a3dd0 0x55877e4afa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.485+0000 7f7868e1c640 1 -- 172.21.15.29:0/1806066347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e4a3dd0 msgr2=0x55877e4afa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.485+0000 7f7868e1c640 1 --2- 172.21.15.29:0/1806066347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e4a3dd0 0x55877e4afa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.485+0000 7f7868e1c640 1 -- 172.21.15.29:0/1806066347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55877e49e270 msgr2=0x55877e4a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.485+0000 7f7868e1c640 1 --2- 172.21.15.29:0/1806066347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55877e49e270 0x55877e4a01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.485+0000 7f7868e1c640 1 -- 172.21.15.29:0/1806066347 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55877e3a1c20 con 0x55877e49ec60 2024-03-11T19:18:36.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.485+0000 7f7868e1c640 1 --2- 172.21.15.29:0/1806066347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e49ec60 0x55877e4a06f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f785c009b30 tx=0x7f785c0348b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d17805cbf7ac7852 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:36.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.485+0000 7f7869e1e640 1 --2- 172.21.15.29:0/1806066347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e4a3dd0 0x55877e4afa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:18:36.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.485+0000 7f784effd640 1 -- 172.21.15.29:0/1806066347 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c059040 con 0x55877e49ec60 2024-03-11T19:18:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.486+0000 7f784effd640 1 -- 172.21.15.29:0/1806066347 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f785c05b040 con 0x55877e49ec60 2024-03-11T19:18:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.486+0000 7f784effd640 1 -- 172.21.15.29:0/1806066347 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c04e070 con 0x55877e49ec60 2024-03-11T19:18:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.487+0000 7f786e454740 1 -- 172.21.15.29:0/1806066347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e49ec60 msgr2=0x55877e4a06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.487+0000 7f786e454740 1 --2- 172.21.15.29:0/1806066347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e49ec60 0x55877e4a06f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f785c009b30 tx=0x7f785c0348b0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.487+0000 7f786e454740 1 -- 172.21.15.29:0/1806066347 shutdown_connections 2024-03-11T19:18:36.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.487+0000 7f786e454740 1 --2- 172.21.15.29:0/1806066347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e4a3dd0 0x55877e4afa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.487+0000 7f786e454740 1 --2- 172.21.15.29:0/1806066347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e49ec60 0x55877e4a06f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.487+0000 7f786e454740 1 --2- 172.21.15.29:0/1806066347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55877e49e270 0x55877e4a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.487+0000 7f786e454740 1 -- 172.21.15.29:0/1806066347 >> 172.21.15.29:0/1806066347 conn(0x55877e4997e0 msgr2=0x55877e499bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:36.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.487+0000 7f786e454740 1 -- 172.21.15.29:0/1806066347 shutdown_connections 2024-03-11T19:18:36.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.488+0000 7f786e454740 1 -- 172.21.15.29:0/1806066347 wait complete. 2024-03-11T19:18:36.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.488+0000 7f786e454740 1 -- start start 2024-03-11T19:18:36.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.488+0000 7f786e454740 1 Processor -- start 2024-03-11T19:18:36.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.489+0000 7f786e454740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49e270 0x55877e4ad840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.489+0000 7f786e454740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55877e49ec60 0x55877e4add80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.489+0000 7f786961d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49e270 0x55877e4ad840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.489+0000 7f786961d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49e270 0x55877e4ad840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45920/0 (socket says 172.21.15.29:45920) 2024-03-11T19:18:36.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.489+0000 7f786961d640 1 -- 172.21.15.29:0/3166439871 learned_addr learned my addr 172.21.15.29:0/3166439871 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:36.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.489+0000 7f786e454740 1 --2- 172.21.15.29:0/3166439871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e4a3dd0 0x55877e4aed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.489+0000 7f786e454740 1 -- 172.21.15.29:0/3166439871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877e4af2d0 con 0x55877e49ec60 2024-03-11T19:18:36.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.489+0000 7f786e454740 1 -- 172.21.15.29:0/3166439871 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877e4af440 con 0x55877e4a3dd0 2024-03-11T19:18:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.489+0000 7f786e454740 1 -- 172.21.15.29:0/3166439871 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877e4af600 con 0x55877e49e270 2024-03-11T19:18:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.489+0000 7f7868e1c640 1 --2- 172.21.15.29:0/3166439871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55877e49ec60 0x55877e4add80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.489+0000 7f786e454740 10 client.0 _ll_register_callbacks cb 0x7f7854005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.489+0000 7f7869e1e640 1 --2- 172.21.15.29:0/3166439871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e4a3dd0 0x55877e4aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.489+0000 7f786961d640 1 -- 172.21.15.29:0/3166439871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e4a3dd0 msgr2=0x55877e4aed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.489+0000 7f786961d640 1 --2- 172.21.15.29:0/3166439871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e4a3dd0 0x55877e4aed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.489+0000 7f786961d640 1 -- 172.21.15.29:0/3166439871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55877e49ec60 msgr2=0x55877e4add80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.489+0000 7f786961d640 1 --2- 172.21.15.29:0/3166439871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55877e49ec60 0x55877e4add80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.489+0000 7f786961d640 1 -- 172.21.15.29:0/3166439871 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55877e3c9060 con 0x55877e49e270 2024-03-11T19:18:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.489+0000 7f7869e1e640 1 --2- 172.21.15.29:0/3166439871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e4a3dd0 0x55877e4aed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:18:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.490+0000 7f786961d640 1 --2- 172.21.15.29:0/3166439871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49e270 0x55877e4ad840 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f78540092a0 tx=0x7f7854011940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c98e1125f5edc04 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.490+0000 7f784cff9640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:18:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.490+0000 7f784cff9640 1 -- 172.21.15.29:0/3166439871 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78540047c0 con 0x55877e49e270 2024-03-11T19:18:36.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.490+0000 7f786e454740 1 -- 172.21.15.29:0/3166439871 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f785c003e20 con 0x55877e49e270 2024-03-11T19:18:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.490+0000 7f784cff9640 1 -- 172.21.15.29:0/3166439871 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7854014040 con 0x55877e49e270 2024-03-11T19:18:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.490+0000 7f784cff9640 1 -- 172.21.15.29:0/3166439871 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7854005890 con 0x55877e49e270 2024-03-11T19:18:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.490+0000 7f786e454740 10 client.27022 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.490+0000 7f786e454740 1 -- 172.21.15.29:0/3166439871 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55877e3c78c0 con 0x55877e49e270 2024-03-11T19:18:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.490+0000 7f784cff9640 1 -- 172.21.15.29:0/3166439871 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f785403a020 con 0x55877e49e270 2024-03-11T19:18:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.490+0000 7f784cff9640 1 -- 172.21.15.29:0/3166439871 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f785402b440 con 0x55877e49e270 2024-03-11T19:18:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.490+0000 7f786e454740 10 client.27022 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.490+0000 7f786e454740 10 client.27022 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:18:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.491+0000 7f786e454740 20 client.27022 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.491+0000 7f786e454740 1 --2- 172.21.15.29:0/3166439871 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55877e50be40 0x55877e50e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.491+0000 7f786e454740 4 client.27022 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:18:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.491+0000 7f786e454740 1 -- 172.21.15.29:0/3166439871 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55877e192890 con 0x55877e50be40 2024-03-11T19:18:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.491+0000 7f7868e1c640 1 --2- 172.21.15.29:0/3166439871 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55877e50be40 0x55877e50e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.491+0000 7f784cff9640 10 client.27022 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:18:36.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.491+0000 7f7868e1c640 1 --2- 172.21.15.29:0/3166439871 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55877e50be40 0x55877e50e260 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d952f2f4c451093b server_cookie=7f23480e4e140a59 in_seq=0 out_seq=0 2024-03-11T19:18:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.491+0000 7f784cff9640 1 -- 172.21.15.29:0/3166439871 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55877e192890 con 0x55877e50be40 2024-03-11T19:18:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.491+0000 7f784cff9640 10 client.27022 handle_command_reply: tid=0 2024-03-11T19:18:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.492+0000 7f786e454740 1 client.27022 shutdown 2024-03-11T19:18:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.492+0000 7f786e454740 1 -- 172.21.15.29:0/3166439871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49e270 msgr2=0x55877e4ad840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.492+0000 7f786e454740 1 --2- 172.21.15.29:0/3166439871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49e270 0x55877e4ad840 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f78540092a0 tx=0x7f7854011940 comp rx=0 tx=0).stop 2024-03-11T19:18:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.492+0000 7f786e454740 1 -- 172.21.15.29:0/3166439871 shutdown_connections 2024-03-11T19:18:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.492+0000 7f786e454740 1 --2- 172.21.15.29:0/3166439871 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55877e50be40 0x55877e50e260 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.492+0000 7f784cff9640 0 client.27022 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:18:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.492+0000 7f786e454740 1 --2- 172.21.15.29:0/3166439871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55877e4a3dd0 0x55877e4aed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.492+0000 7f786e454740 1 --2- 172.21.15.29:0/3166439871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55877e49ec60 0x55877e4add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.492+0000 7f786e454740 1 --2- 172.21.15.29:0/3166439871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55877e49e270 0x55877e4ad840 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.492+0000 7f786e454740 1 -- 172.21.15.29:0/3166439871 >> 172.21.15.29:0/3166439871 conn(0x55877e4997e0 msgr2=0x55877e4a01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.492+0000 7f786e454740 1 -- 172.21.15.29:0/3166439871 shutdown_connections 2024-03-11T19:18:36.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.492+0000 7f786e454740 1 -- 172.21.15.29:0/3166439871 wait complete. 2024-03-11T19:18:36.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.492+0000 7f786e454740 20 client.27022 trim_cache size 0 max 16384 2024-03-11T19:18:36.499 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:18:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.493+0000 7f786b881640 1 -- 172.21.15.29:0/3735109101 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f78500a1b60 msgr2=0x7f78500a4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.493+0000 7f786b881640 1 --2- 172.21.15.29:0/3735109101 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f78500a1b60 0x7f78500a4020 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x55877e49f060 tx=0x7f785c02c920 comp rx=0 tx=0).stop 2024-03-11T19:18:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.493+0000 7f786b881640 1 -- 172.21.15.29:0/3735109101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78640788c0 msgr2=0x7f78641b5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.493+0000 7f786b881640 1 --2- 172.21.15.29:0/3735109101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78640788c0 0x7f78641b5b70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f786000c7a0 tx=0x7f786000cc60 comp rx=0 tx=0).stop 2024-03-11T19:18:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.493+0000 7f786b881640 1 -- 172.21.15.29:0/3735109101 shutdown_connections 2024-03-11T19:18:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.493+0000 7f786b881640 1 --2- 172.21.15.29:0/3735109101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78640788c0 0x7f78641b5b70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.493+0000 7f786b881640 1 --2- 172.21.15.29:0/3735109101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7864075f10 0x7f78641b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.493+0000 7f786b881640 1 --2- 172.21.15.29:0/3735109101 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f78500a1b60 0x7f78500a4020 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.493+0000 7f786b881640 1 --2- 172.21.15.29:0/3735109101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7864073030 0x7f78641b10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.493+0000 7f786b881640 1 -- 172.21.15.29:0/3735109101 >> 172.21.15.29:0/3735109101 conn(0x7f786406dfe0 msgr2=0x7f786407aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:36.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.493+0000 7f786b881640 1 -- 172.21.15.29:0/3735109101 shutdown_connections 2024-03-11T19:18:36.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.493+0000 7f786b881640 1 -- 172.21.15.29:0/3735109101 wait complete. 2024-03-11T19:18:36.522 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:18:36.522 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:18:36.558 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:36 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2065965877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:18:36.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:36 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2065965877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:18:36.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.699+0000 7f0a9e78c640 1 -- 172.21.15.29:0/1181446670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a981045d0 msgr2=0x7f0a981069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.699+0000 7f0a9e78c640 1 --2- 172.21.15.29:0/1181446670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a981045d0 0x7f0a981069c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009860 tx=0x7f0a8c02bb70 comp rx=0 tx=0).stop 2024-03-11T19:18:36.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.700+0000 7f0a9e78c640 1 -- 172.21.15.29:0/1181446670 shutdown_connections 2024-03-11T19:18:36.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.700+0000 7f0a9e78c640 1 --2- 172.21.15.29:0/1181446670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a98106f00 0x7f0a981092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.700+0000 7f0a9e78c640 1 --2- 172.21.15.29:0/1181446670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a981045d0 0x7f0a981069c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.700+0000 7f0a9e78c640 1 --2- 172.21.15.29:0/1181446670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a98101ca0 0x7f0a98104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.700+0000 7f0a9e78c640 1 -- 172.21.15.29:0/1181446670 >> 172.21.15.29:0/1181446670 conn(0x7f0a980fb610 msgr2=0x7f0a980fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:36.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.700+0000 7f0a9e78c640 1 -- 172.21.15.29:0/1181446670 shutdown_connections 2024-03-11T19:18:36.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.700+0000 7f0a9e78c640 1 -- 172.21.15.29:0/1181446670 wait complete. 2024-03-11T19:18:36.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.701+0000 7f0a9e78c640 1 Processor -- start 2024-03-11T19:18:36.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.701+0000 7f0a9e78c640 1 -- start start 2024-03-11T19:18:36.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.701+0000 7f0a9e78c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a98101ca0 0x7f0a981971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.701+0000 7f0a9e78c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a981045d0 0x7f0a981976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.701+0000 7f0a9e78c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a98106f00 0x7f0a9819e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.701+0000 7f0a9e78c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9819ec90 con 0x7f0a98106f00 2024-03-11T19:18:36.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.701+0000 7f0a9e78c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9819ee00 con 0x7f0a98101ca0 2024-03-11T19:18:36.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.701+0000 7f0a9e78c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9819efc0 con 0x7f0a981045d0 2024-03-11T19:18:36.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.701+0000 7f0a9cd29640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a98106f00 0x7f0a9819e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.701+0000 7f0a977fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a981045d0 0x7f0a981976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.701+0000 7f0a9cd29640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a98106f00 0x7f0a9819e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51790/0 (socket says 172.21.15.29:51790) 2024-03-11T19:18:36.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.701+0000 7f0a9cd29640 1 -- 172.21.15.29:0/3022624613 learned_addr learned my addr 172.21.15.29:0/3022624613 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:36.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.701+0000 7f0a97fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a98101ca0 0x7f0a981971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.702+0000 7f0a977fe640 1 -- 172.21.15.29:0/3022624613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a98101ca0 msgr2=0x7f0a981971a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.702+0000 7f0a977fe640 1 --2- 172.21.15.29:0/3022624613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a98101ca0 0x7f0a981971a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.702+0000 7f0a977fe640 1 -- 172.21.15.29:0/3022624613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a98106f00 msgr2=0x7f0a9819e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.702+0000 7f0a977fe640 1 --2- 172.21.15.29:0/3022624613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a98106f00 0x7f0a9819e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.702+0000 7f0a977fe640 1 -- 172.21.15.29:0/3022624613 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8c009490 con 0x7f0a981045d0 2024-03-11T19:18:36.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.702+0000 7f0a977fe640 1 --2- 172.21.15.29:0/3022624613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a981045d0 0x7f0a981976e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009780 tx=0x7f0a8c032430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:36.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.703+0000 7f0a957fa640 1 -- 172.21.15.29:0/3022624613 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c004340 con 0x7f0a981045d0 2024-03-11T19:18:36.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.703+0000 7f0a9e78c640 1 -- 172.21.15.29:0/3022624613 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a980ffa90 con 0x7f0a981045d0 2024-03-11T19:18:36.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.703+0000 7f0a9e78c640 1 -- 172.21.15.29:0/3022624613 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a980fff60 con 0x7f0a981045d0 2024-03-11T19:18:36.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.703+0000 7f0a957fa640 1 -- 172.21.15.29:0/3022624613 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0a8c042070 con 0x7f0a981045d0 2024-03-11T19:18:36.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.703+0000 7f0a957fa640 1 -- 172.21.15.29:0/3022624613 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c03ec00 con 0x7f0a981045d0 2024-03-11T19:18:36.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.706+0000 7f0a957fa640 1 -- 172.21.15.29:0/3022624613 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0a8c044030 con 0x7f0a981045d0 2024-03-11T19:18:36.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.706+0000 7f0aa0ffd740 1 Processor -- start 2024-03-11T19:18:36.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0aa0ffd740 1 -- start start 2024-03-11T19:18:36.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0aa0ffd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da6a80 0x555a85da6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0aa0ffd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85da7470 0x555a85dabf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0aa0ffd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85dac5e0 0x555a85daea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0aa0ffd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a85d5a960 con 0x555a85da6a80 2024-03-11T19:18:36.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0aa0ffd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a85d0e310 con 0x555a85dac5e0 2024-03-11T19:18:36.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0aa0ffd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a85d0b840 con 0x555a85da7470 2024-03-11T19:18:36.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0a9cd29640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da6a80 0x555a85da6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0a97fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85da7470 0x555a85dabf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0a957fa640 1 --2- 172.21.15.29:0/3022624613 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0a800a1ab0 0x7f0a800a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0a9cd29640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da6a80 0x555a85da6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51798/0 (socket says 172.21.15.29:51798) 2024-03-11T19:18:36.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0a9cd29640 1 -- 172.21.15.29:0/924707755 learned_addr learned my addr 172.21.15.29:0/924707755 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:36.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0a977fe640 1 --2- 172.21.15.29:0/924707755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85dac5e0 0x555a85daea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0a957fa640 1 -- 172.21.15.29:0/3022624613 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0a8c0f2ab0 con 0x7f0a981045d0 2024-03-11T19:18:36.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0a97fff640 1 -- 172.21.15.29:0/924707755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85dac5e0 msgr2=0x555a85daea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0a97fff640 1 --2- 172.21.15.29:0/924707755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85dac5e0 0x555a85daea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0a97fff640 1 -- 172.21.15.29:0/924707755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da6a80 msgr2=0x555a85da6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0a97fff640 1 --2- 172.21.15.29:0/924707755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da6a80 0x555a85da6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0a97fff640 1 -- 172.21.15.29:0/924707755 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a85cc9060 con 0x555a85da7470 2024-03-11T19:18:36.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.707+0000 7f0a9cd29640 1 --2- 172.21.15.29:0/924707755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da6a80 0x555a85da6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:18:36.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.708+0000 7f0a9cd29640 1 --2- 172.21.15.29:0/3022624613 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0a800a1ab0 0x7f0a800a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.708+0000 7f0a97fff640 1 --2- 172.21.15.29:0/924707755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85da7470 0x555a85dabf70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009c50 tx=0x7f0a880117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aab1cb9cd298ad04 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:36.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.708+0000 7f0a7e7fc640 1 -- 172.21.15.29:0/924707755 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a88012450 con 0x555a85da7470 2024-03-11T19:18:36.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.708+0000 7f0a7e7fc640 1 -- 172.21.15.29:0/924707755 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0a88012a60 con 0x555a85da7470 2024-03-11T19:18:36.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.708+0000 7f0a7e7fc640 1 -- 172.21.15.29:0/924707755 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8800fb20 con 0x555a85da7470 2024-03-11T19:18:36.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.709+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/924707755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85da7470 msgr2=0x555a85dabf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.709+0000 7f0a9cd29640 1 --2- 172.21.15.29:0/3022624613 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0a800a1ab0 0x7f0a800a3f70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x555a85da7cd0 tx=0x7f0a90004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:36.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.709+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/924707755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85da7470 0x555a85dabf70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009c50 tx=0x7f0a880117b0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.709+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/924707755 shutdown_connections 2024-03-11T19:18:36.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.709+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/924707755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85dac5e0 0x555a85daea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.709+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/924707755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85da7470 0x555a85dabf70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.709+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/924707755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da6a80 0x555a85da6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.709+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/924707755 >> 172.21.15.29:0/924707755 conn(0x555a85d997e0 msgr2=0x555a85d99be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:36.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.709+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/924707755 shutdown_connections 2024-03-11T19:18:36.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.709+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/924707755 wait complete. 2024-03-11T19:18:36.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.710+0000 7f0aa0ffd740 1 -- start start 2024-03-11T19:18:36.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.710+0000 7f0aa0ffd740 1 Processor -- start 2024-03-11T19:18:36.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.710+0000 7f0aa0ffd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85da6a80 0x555a85db0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.710+0000 7f0aa0ffd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da7470 0x555a85db11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.710+0000 7f0aa0ffd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85dac5e0 0x555a85db8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.710+0000 7f0aa0ffd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a85d07020 con 0x555a85da7470 2024-03-11T19:18:36.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.710+0000 7f0aa0ffd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a859da430 con 0x555a85dac5e0 2024-03-11T19:18:36.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.710+0000 7f0aa0ffd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a85996e00 con 0x555a85da6a80 2024-03-11T19:18:36.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.710+0000 7f0aa0ffd740 10 client.0 _ll_register_callbacks cb 0x555a85caecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:36.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.710+0000 7f0a9cd29640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85dac5e0 0x555a85db8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.710+0000 7f0a9cd29640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85dac5e0 0x555a85db8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:47460/0 (socket says 172.21.15.29:47460) 2024-03-11T19:18:36.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.710+0000 7f0a9cd29640 1 -- 172.21.15.29:0/1330127576 learned_addr learned my addr 172.21.15.29:0/1330127576 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:36.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.710+0000 7f0a977fe640 1 --2- 172.21.15.29:0/1330127576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da7470 0x555a85db11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.710+0000 7f0a97fff640 1 --2- 172.21.15.29:0/1330127576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85da6a80 0x555a85db0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.711+0000 7f0a9cd29640 1 -- 172.21.15.29:0/1330127576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85da6a80 msgr2=0x555a85db0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.711+0000 7f0a9cd29640 1 --2- 172.21.15.29:0/1330127576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85da6a80 0x555a85db0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.711+0000 7f0a9cd29640 1 -- 172.21.15.29:0/1330127576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da7470 msgr2=0x555a85db11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.711+0000 7f0a9cd29640 1 --2- 172.21.15.29:0/1330127576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da7470 0x555a85db11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.711+0000 7f0a9cd29640 1 -- 172.21.15.29:0/1330127576 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a880098b0 con 0x555a85dac5e0 2024-03-11T19:18:36.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.711+0000 7f0a9cd29640 1 --2- 172.21.15.29:0/1330127576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85dac5e0 0x555a85db8230 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0a90009fd0 tx=0x7f0a90009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9483ae54504805cf server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:36.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.711+0000 7f0a777fe640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:18:36.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.711+0000 7f0a777fe640 1 -- 172.21.15.29:0/1330127576 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a90018070 con 0x555a85dac5e0 2024-03-11T19:18:36.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.711+0000 7f0a777fe640 1 -- 172.21.15.29:0/1330127576 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0a90020bc0 con 0x555a85dac5e0 2024-03-11T19:18:36.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.711+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/1330127576 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555a85a067e0 con 0x555a85dac5e0 2024-03-11T19:18:36.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.711+0000 7f0a777fe640 1 -- 172.21.15.29:0/1330127576 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a90020d80 con 0x555a85dac5e0 2024-03-11T19:18:36.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.712+0000 7f0aa0ffd740 10 client.26993 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:36.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.712+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/1330127576 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555a85d056a0 con 0x555a85dac5e0 2024-03-11T19:18:36.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.712+0000 7f0a777fe640 1 -- 172.21.15.29:0/1330127576 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x555a85a067e0 con 0x555a85dac5e0 2024-03-11T19:18:36.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.712+0000 7f0a777fe640 1 -- 172.21.15.29:0/1330127576 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0a90010550 con 0x555a85dac5e0 2024-03-11T19:18:36.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.712+0000 7f0aa0ffd740 10 client.26993 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:36.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.712+0000 7f0aa0ffd740 10 client.26993 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:18:36.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.713+0000 7f0aa0ffd740 20 client.26993 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:36.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.713+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/1330127576 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x555a85dbcf10 0x555a85dbf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.713+0000 7f0aa0ffd740 4 client.26993 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:18:36.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.713+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/1330127576 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555a85a43720 con 0x555a85dbcf10 2024-03-11T19:18:36.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.713+0000 7f0a97fff640 1 --2- 172.21.15.29:0/1330127576 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x555a85dbcf10 0x555a85dbf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.713+0000 7f0a97fff640 1 --2- 172.21.15.29:0/1330127576 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x555a85dbcf10 0x555a85dbf330 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e20411f26a5fbfec server_cookie=a9233bbbebc33307 in_seq=0 out_seq=0 2024-03-11T19:18:36.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.713+0000 7f0a777fe640 10 client.26993 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:18:36.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.714+0000 7f0a777fe640 1 -- 172.21.15.29:0/1330127576 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555a85a43720 con 0x555a85dbcf10 2024-03-11T19:18:36.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.714+0000 7f0a777fe640 10 client.26993 handle_command_reply: tid=0 2024-03-11T19:18:36.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.714+0000 7f0aa0ffd740 1 client.26993 shutdown 2024-03-11T19:18:36.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.715+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/1330127576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85dac5e0 msgr2=0x555a85db8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.715+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/1330127576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85dac5e0 0x555a85db8230 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0a90009fd0 tx=0x7f0a90009500 comp rx=0 tx=0).stop 2024-03-11T19:18:36.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.715+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/1330127576 shutdown_connections 2024-03-11T19:18:36.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.715+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/1330127576 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x555a85dbcf10 0x555a85dbf330 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.715+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/1330127576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85dac5e0 0x555a85db8230 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.715+0000 7f0a777fe640 0 client.26993 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:18:36.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.715+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/1330127576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da7470 0x555a85db11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.715+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/1330127576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85da6a80 0x555a85db0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.715+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/1330127576 >> 172.21.15.29:0/1330127576 conn(0x555a85d997e0 msgr2=0x555a85d98b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:36.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.715+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/1330127576 shutdown_connections 2024-03-11T19:18:36.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.715+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/1330127576 wait complete. 2024-03-11T19:18:36.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.716+0000 7f0aa0ffd740 20 client.26993 trim_cache size 0 max 16384 2024-03-11T19:18:36.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.749+0000 7f0aa0ffd740 1 Processor -- start 2024-03-11T19:18:36.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.749+0000 7f0aa0ffd740 1 -- start start 2024-03-11T19:18:36.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.750+0000 7f0aa0ffd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da6a80 0x555a85daffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.750+0000 7f0aa0ffd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85da7470 0x555a85db81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.750+0000 7f0aa0ffd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85dac5e0 0x555a85db04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.750+0000 7f0aa0ffd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a85c8c130 con 0x555a85da6a80 2024-03-11T19:18:36.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.750+0000 7f0aa0ffd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a85daec00 con 0x555a85da7470 2024-03-11T19:18:36.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.750+0000 7f0a9cd29640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85dac5e0 0x555a85db04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.750+0000 7f0a9cd29640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85dac5e0 0x555a85db04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45958/0 (socket says 172.21.15.29:45958) 2024-03-11T19:18:36.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.750+0000 7f0a9cd29640 1 -- 172.21.15.29:0/2818185754 learned_addr learned my addr 172.21.15.29:0/2818185754 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:36.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.755+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/2818185754 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a85da8140 con 0x555a85dac5e0 2024-03-11T19:18:36.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.755+0000 7f0a977fe640 1 --2- 172.21.15.29:0/2818185754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85da7470 0x555a85db81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.756+0000 7f0a97fff640 1 --2- 172.21.15.29:0/2818185754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da6a80 0x555a85daffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.756+0000 7f0a97fff640 1 -- 172.21.15.29:0/2818185754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85dac5e0 msgr2=0x555a85db04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.756+0000 7f0a97fff640 1 --2- 172.21.15.29:0/2818185754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85dac5e0 0x555a85db04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.756+0000 7f0a97fff640 1 -- 172.21.15.29:0/2818185754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85da7470 msgr2=0x555a85db81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.756+0000 7f0a97fff640 1 --2- 172.21.15.29:0/2818185754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85da7470 0x555a85db81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.756+0000 7f0a97fff640 1 -- 172.21.15.29:0/2818185754 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a85ca1c20 con 0x555a85da6a80 2024-03-11T19:18:36.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.757+0000 7f0a9cd29640 1 --2- 172.21.15.29:0/2818185754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85dac5e0 0x555a85db04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:18:36.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.757+0000 7f0a977fe640 1 --2- 172.21.15.29:0/2818185754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85da7470 0x555a85db81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:18:36.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.757+0000 7f0a97fff640 1 --2- 172.21.15.29:0/2818185754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da6a80 0x555a85daffa0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x555a85d9a750 tx=0x7f0a88009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79c4cac7e6a8baf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:36.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:36 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2065965877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:18:36.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.757+0000 7f0a7dffb640 1 -- 172.21.15.29:0/2818185754 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a88009ee0 con 0x555a85da6a80 2024-03-11T19:18:36.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.757+0000 7f0a7dffb640 1 -- 172.21.15.29:0/2818185754 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0a880054f0 con 0x555a85da6a80 2024-03-11T19:18:36.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.757+0000 7f0a7dffb640 1 -- 172.21.15.29:0/2818185754 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a88003b70 con 0x555a85da6a80 2024-03-11T19:18:36.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.758+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/2818185754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da6a80 msgr2=0x555a85daffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.758+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/2818185754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da6a80 0x555a85daffa0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x555a85d9a750 tx=0x7f0a88009260 comp rx=0 tx=0).stop 2024-03-11T19:18:36.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.759+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/2818185754 shutdown_connections 2024-03-11T19:18:36.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.759+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/2818185754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85dac5e0 0x555a85db04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.759+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/2818185754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85da7470 0x555a85db81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.759+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/2818185754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da6a80 0x555a85daffa0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.759+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/2818185754 >> 172.21.15.29:0/2818185754 conn(0x555a85d997e0 msgr2=0x555a85da8890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:36.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.759+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/2818185754 shutdown_connections 2024-03-11T19:18:36.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.759+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/2818185754 wait complete. 2024-03-11T19:18:36.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.760+0000 7f0aa0ffd740 1 -- start start 2024-03-11T19:18:36.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.760+0000 7f0aa0ffd740 1 Processor -- start 2024-03-11T19:18:36.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.761+0000 7f0aa0ffd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85da6a80 0x555a85db6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.761+0000 7f0aa0ffd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da7470 0x555a85db70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.761+0000 7f0aa0ffd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85dac5e0 0x555a85db7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.761+0000 7f0aa0ffd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a85dafb10 con 0x555a85da7470 2024-03-11T19:18:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.761+0000 7f0aa0ffd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a85e13460 con 0x555a85dac5e0 2024-03-11T19:18:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.761+0000 7f0aa0ffd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a85e13620 con 0x555a85da6a80 2024-03-11T19:18:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.761+0000 7f0a97fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85da6a80 0x555a85db6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.761+0000 7f0a97fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85da6a80 0x555a85db6b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45964/0 (socket says 172.21.15.29:45964) 2024-03-11T19:18:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.761+0000 7f0a97fff640 1 -- 172.21.15.29:0/1744769297 learned_addr learned my addr 172.21.15.29:0/1744769297 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.761+0000 7f0a9cd29640 1 --2- 172.21.15.29:0/1744769297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85dac5e0 0x555a85db7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.761+0000 7f0aa0ffd740 10 client.0 _ll_register_callbacks cb 0x7f0a880096a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.761+0000 7f0a977fe640 1 --2- 172.21.15.29:0/1744769297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da7470 0x555a85db70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.761+0000 7f0a9cd29640 1 -- 172.21.15.29:0/1744769297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85da6a80 msgr2=0x555a85db6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.761+0000 7f0a9cd29640 1 --2- 172.21.15.29:0/1744769297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85da6a80 0x555a85db6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.761+0000 7f0a9cd29640 1 -- 172.21.15.29:0/1744769297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da7470 msgr2=0x555a85db70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.761+0000 7f0a9cd29640 1 --2- 172.21.15.29:0/1744769297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da7470 0x555a85db70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.761+0000 7f0a9cd29640 1 -- 172.21.15.29:0/1744769297 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a980fff60 con 0x555a85dac5e0 2024-03-11T19:18:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.761+0000 7f0a97fff640 1 --2- 172.21.15.29:0/1744769297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85da6a80 0x555a85db6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:18:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.762+0000 7f0a9cd29640 1 --2- 172.21.15.29:0/1744769297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85dac5e0 0x555a85db7600 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0a90010a00 tx=0x7f0a900049d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65524b9ee67da77d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.762+0000 7f0a777fe640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:18:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.762+0000 7f0a777fe640 1 -- 172.21.15.29:0/1744769297 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a90018070 con 0x555a85dac5e0 2024-03-11T19:18:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.762+0000 7f0a777fe640 1 -- 172.21.15.29:0/1744769297 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0a9002f030 con 0x555a85dac5e0 2024-03-11T19:18:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.762+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/1744769297 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0a88012a60 con 0x555a85dac5e0 2024-03-11T19:18:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.762+0000 7f0a777fe640 1 -- 172.21.15.29:0/1744769297 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a90010cf0 con 0x555a85dac5e0 2024-03-11T19:18:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.762+0000 7f0aa0ffd740 10 client.26999 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.762+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/1744769297 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555a85cc9060 con 0x555a85dac5e0 2024-03-11T19:18:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.762+0000 7f0a777fe640 1 -- 172.21.15.29:0/1744769297 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0a90041030 con 0x555a85dac5e0 2024-03-11T19:18:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.763+0000 7f0a777fe640 1 -- 172.21.15.29:0/1744769297 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0a90029a20 con 0x555a85dac5e0 2024-03-11T19:18:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.763+0000 7f0aa0ffd740 10 client.26999 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.763+0000 7f0aa0ffd740 10 client.26999 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:18:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.763+0000 7f0aa0ffd740 20 client.26999 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.763+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/1744769297 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x555a85e150c0 0x555a85e174e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:36.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.763+0000 7f0aa0ffd740 4 client.26999 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:18:36.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.763+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/1744769297 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555a85a92890 con 0x555a85e150c0 2024-03-11T19:18:36.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.763+0000 7f0a97fff640 1 --2- 172.21.15.29:0/1744769297 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x555a85e150c0 0x555a85e174e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:36.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.763+0000 7f0a97fff640 1 --2- 172.21.15.29:0/1744769297 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x555a85e150c0 0x555a85e174e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bd30ef3a3a92e5aa server_cookie=dd770130ea05ded2 in_seq=0 out_seq=0 2024-03-11T19:18:36.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.764+0000 7f0a777fe640 10 client.26999 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:18:36.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.764+0000 7f0a777fe640 1 -- 172.21.15.29:0/1744769297 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555a85a92890 con 0x555a85e150c0 2024-03-11T19:18:36.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.764+0000 7f0a777fe640 10 client.26999 handle_command_reply: tid=0 2024-03-11T19:18:36.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.764+0000 7f0aa0ffd740 1 client.26999 shutdown 2024-03-11T19:18:36.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.764+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/1744769297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85dac5e0 msgr2=0x555a85db7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.764+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/1744769297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85dac5e0 0x555a85db7600 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0a90010a00 tx=0x7f0a900049d0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.764+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/1744769297 shutdown_connections 2024-03-11T19:18:36.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.764+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/1744769297 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x555a85e150c0 0x555a85e174e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.764+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/1744769297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555a85dac5e0 0x555a85db7600 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.764+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/1744769297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555a85da7470 0x555a85db70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.764+0000 7f0aa0ffd740 1 --2- 172.21.15.29:0/1744769297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a85da6a80 0x555a85db6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.764+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/1744769297 >> 172.21.15.29:0/1744769297 conn(0x555a85d997e0 msgr2=0x555a85daffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:36.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.764+0000 7f0a777fe640 0 client.26999 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:18:36.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.765+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/1744769297 shutdown_connections 2024-03-11T19:18:36.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.765+0000 7f0aa0ffd740 1 -- 172.21.15.29:0/1744769297 wait complete. 2024-03-11T19:18:36.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.765+0000 7f0aa0ffd740 20 client.26999 trim_cache size 0 max 16384 2024-03-11T19:18:36.771 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:18:36.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.766+0000 7f0a9e78c640 1 -- 172.21.15.29:0/3022624613 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0a800a1ab0 msgr2=0x7f0a800a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.766+0000 7f0a9e78c640 1 --2- 172.21.15.29:0/3022624613 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0a800a1ab0 0x7f0a800a3f70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x555a85da7cd0 tx=0x7f0a90004a40 comp rx=0 tx=0).stop 2024-03-11T19:18:36.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.766+0000 7f0a9e78c640 1 -- 172.21.15.29:0/3022624613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a981045d0 msgr2=0x7f0a981976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:36.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.766+0000 7f0a9e78c640 1 --2- 172.21.15.29:0/3022624613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a981045d0 0x7f0a981976e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009780 tx=0x7f0a8c032430 comp rx=0 tx=0).stop 2024-03-11T19:18:36.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.766+0000 7f0a9e78c640 1 -- 172.21.15.29:0/3022624613 shutdown_connections 2024-03-11T19:18:36.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.766+0000 7f0a9e78c640 1 --2- 172.21.15.29:0/3022624613 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0a800a1ab0 0x7f0a800a3f70 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.766+0000 7f0a9e78c640 1 --2- 172.21.15.29:0/3022624613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a98106f00 0x7f0a9819e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.766+0000 7f0a9e78c640 1 --2- 172.21.15.29:0/3022624613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a981045d0 0x7f0a981976e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.766+0000 7f0a9e78c640 1 --2- 172.21.15.29:0/3022624613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a98101ca0 0x7f0a981971a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:36.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.766+0000 7f0a9e78c640 1 -- 172.21.15.29:0/3022624613 >> 172.21.15.29:0/3022624613 conn(0x7f0a980fb610 msgr2=0x7f0a98103b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:36.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.766+0000 7f0a9e78c640 1 -- 172.21.15.29:0/3022624613 shutdown_connections 2024-03-11T19:18:36.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:36.766+0000 7f0a9e78c640 1 -- 172.21.15.29:0/3022624613 wait complete. 2024-03-11T19:18:36.785 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:18:36.785 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:18:37.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.017+0000 7facc82d4640 1 -- 172.21.15.29:0/3618988213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facc010aaf0 msgr2=0x7facc010cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:37.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.017+0000 7facc82d4640 1 --2- 172.21.15.29:0/3618988213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facc010aaf0 0x7facc010cf60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7facb0009860 tx=0x7facb002bb70 comp rx=0 tx=0).stop 2024-03-11T19:18:37.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.018+0000 7facc82d4640 1 -- 172.21.15.29:0/3618988213 shutdown_connections 2024-03-11T19:18:37.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.018+0000 7facc82d4640 1 --2- 172.21.15.29:0/3618988213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc010d4a0 0x7facc010d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.018+0000 7facc82d4640 1 --2- 172.21.15.29:0/3618988213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facc010aaf0 0x7facc010cf60 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.018+0000 7facc82d4640 1 --2- 172.21.15.29:0/3618988213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facc0108180 0x7facc010a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.018+0000 7facc82d4640 1 -- 172.21.15.29:0/3618988213 >> 172.21.15.29:0/3618988213 conn(0x7facc006dfe0 msgr2=0x7facc0070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:37.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.018+0000 7facc82d4640 1 -- 172.21.15.29:0/3618988213 shutdown_connections 2024-03-11T19:18:37.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.018+0000 7facc82d4640 1 -- 172.21.15.29:0/3618988213 wait complete. 2024-03-11T19:18:37.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.019+0000 7facc82d4640 1 Processor -- start 2024-03-11T19:18:37.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.019+0000 7facc82d4640 1 -- start start 2024-03-11T19:18:37.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.019+0000 7facc82d4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc0108180 0x7facc01a6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:37.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.019+0000 7facc82d4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facc010aaf0 0x7facc01a6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.019+0000 7facc82d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facc010d4a0 0x7facc01ad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.019+0000 7facc82d4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc01adb00 con 0x7facc010d4a0 2024-03-11T19:18:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.019+0000 7facc82d4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc01adc70 con 0x7facc010aaf0 2024-03-11T19:18:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.019+0000 7facc82d4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc01ade30 con 0x7facc0108180 2024-03-11T19:18:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.019+0000 7facc6070640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc0108180 0x7facc01a6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.019+0000 7facc6070640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc0108180 0x7facc01a6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45986/0 (socket says 172.21.15.29:45986) 2024-03-11T19:18:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.019+0000 7facc6070640 1 -- 172.21.15.29:0/1367149667 learned_addr learned my addr 172.21.15.29:0/1367149667 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.019+0000 7facc586f640 1 --2- 172.21.15.29:0/1367149667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facc010aaf0 0x7facc01a6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.019+0000 7facc6871640 1 --2- 172.21.15.29:0/1367149667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facc010d4a0 0x7facc01ad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.020+0000 7facc586f640 1 -- 172.21.15.29:0/1367149667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc0108180 msgr2=0x7facc01a6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.020+0000 7facc586f640 1 --2- 172.21.15.29:0/1367149667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc0108180 0x7facc01a6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.020+0000 7facc586f640 1 -- 172.21.15.29:0/1367149667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facc010d4a0 msgr2=0x7facc01ad5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.020+0000 7facc586f640 1 --2- 172.21.15.29:0/1367149667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facc010d4a0 0x7facc01ad5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.020+0000 7facc586f640 1 -- 172.21.15.29:0/1367149667 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb0009490 con 0x7facc010aaf0 2024-03-11T19:18:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.020+0000 7facc586f640 1 --2- 172.21.15.29:0/1367149667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facc010aaf0 0x7facc01a6550 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7facb002c0b0 tx=0x7facb00042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.020+0000 7facb77fe640 1 -- 172.21.15.29:0/1367149667 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb002c920 con 0x7facc010aaf0 2024-03-11T19:18:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.020+0000 7facb77fe640 1 -- 172.21.15.29:0/1367149667 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7facb002ce90 con 0x7facc010aaf0 2024-03-11T19:18:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.020+0000 7facb77fe640 1 -- 172.21.15.29:0/1367149667 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb00035f0 con 0x7facc010aaf0 2024-03-11T19:18:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.020+0000 7facc82d4640 1 -- 172.21.15.29:0/1367149667 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc01ae0b0 con 0x7facc010aaf0 2024-03-11T19:18:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.020+0000 7facc82d4640 1 -- 172.21.15.29:0/1367149667 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc01ae5b0 con 0x7facc010aaf0 2024-03-11T19:18:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.023+0000 7facb77fe640 1 -- 172.21.15.29:0/1367149667 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7facb003b040 con 0x7facc010aaf0 2024-03-11T19:18:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.024+0000 7facb77fe640 1 --2- 172.21.15.29:0/1367149667 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7facac0a1ab0 0x7facac0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.025+0000 7facb77fe640 1 -- 172.21.15.29:0/1367149667 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7facb00eee60 con 0x7facc010aaf0 2024-03-11T19:18:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.025+0000 7facc6070640 1 --2- 172.21.15.29:0/1367149667 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7facac0a1ab0 0x7facac0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:37.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.025+0000 7faccaf7d740 1 Processor -- start 2024-03-11T19:18:37.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.025+0000 7facc6070640 1 --2- 172.21.15.29:0/1367149667 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7facac0a1ab0 0x7facac0a3f70 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7facbc009a40 tx=0x7facbc0091c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:37.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.026+0000 7faccaf7d740 1 -- start start 2024-03-11T19:18:37.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.026+0000 7faccaf7d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d4270 0x55d85a0d4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:37.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.026+0000 7faccaf7d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4c60 0x55d85a0d9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:37.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.026+0000 7faccaf7d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d9dd0 0x55d85a0dc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.026+0000 7faccaf7d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d85a090960 con 0x55d85a0d4c60 2024-03-11T19:18:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.026+0000 7faccaf7d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d85a044310 con 0x55d85a0d9dd0 2024-03-11T19:18:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.026+0000 7faccaf7d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d85a041840 con 0x55d85a0d4270 2024-03-11T19:18:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.026+0000 7facc6070640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d4270 0x55d85a0d4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.026+0000 7facc6070640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d4270 0x55d85a0d4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:45998/0 (socket says 172.21.15.29:45998) 2024-03-11T19:18:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.026+0000 7facc6070640 1 -- 172.21.15.29:0/3013185336 learned_addr learned my addr 172.21.15.29:0/3013185336 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.026+0000 7facc586f640 1 --2- 172.21.15.29:0/3013185336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4c60 0x55d85a0d9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.026+0000 7facc6871640 1 --2- 172.21.15.29:0/3013185336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d9dd0 0x55d85a0dc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.026+0000 7facc6070640 1 -- 172.21.15.29:0/3013185336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d9dd0 msgr2=0x55d85a0dc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.026+0000 7facc6070640 1 --2- 172.21.15.29:0/3013185336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d9dd0 0x55d85a0dc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.026+0000 7facc6070640 1 -- 172.21.15.29:0/3013185336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4c60 msgr2=0x55d85a0d9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.026+0000 7facc6070640 1 --2- 172.21.15.29:0/3013185336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4c60 0x55d85a0d9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.026+0000 7facc6070640 1 -- 172.21.15.29:0/3013185336 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d859fff060 con 0x55d85a0d4270 2024-03-11T19:18:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.027+0000 7facc6070640 1 --2- 172.21.15.29:0/3013185336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d4270 0x55d85a0d4690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7facbc009360 tx=0x7facbc011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59618b48c07a0062 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.027+0000 7facb4bfe640 1 -- 172.21.15.29:0/3013185336 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc015b80 con 0x55d85a0d4270 2024-03-11T19:18:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.027+0000 7facb4bfe640 1 -- 172.21.15.29:0/3013185336 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7facbc015ce0 con 0x55d85a0d4270 2024-03-11T19:18:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.027+0000 7facb4bfe640 1 -- 172.21.15.29:0/3013185336 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc01f720 con 0x55d85a0d4270 2024-03-11T19:18:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.028+0000 7faccaf7d740 1 -- 172.21.15.29:0/3013185336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d4270 msgr2=0x55d85a0d4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.028+0000 7faccaf7d740 1 --2- 172.21.15.29:0/3013185336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d4270 0x55d85a0d4690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7facbc009360 tx=0x7facbc011000 comp rx=0 tx=0).stop 2024-03-11T19:18:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.028+0000 7faccaf7d740 1 -- 172.21.15.29:0/3013185336 shutdown_connections 2024-03-11T19:18:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.028+0000 7faccaf7d740 1 --2- 172.21.15.29:0/3013185336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d9dd0 0x55d85a0dc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.028+0000 7faccaf7d740 1 --2- 172.21.15.29:0/3013185336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4c60 0x55d85a0d9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.028+0000 7faccaf7d740 1 --2- 172.21.15.29:0/3013185336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d4270 0x55d85a0d4690 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.028+0000 7faccaf7d740 1 -- 172.21.15.29:0/3013185336 >> 172.21.15.29:0/3013185336 conn(0x55d85a0cf7e0 msgr2=0x55d85a0cfbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.029+0000 7faccaf7d740 1 -- 172.21.15.29:0/3013185336 shutdown_connections 2024-03-11T19:18:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.029+0000 7faccaf7d740 1 -- 172.21.15.29:0/3013185336 wait complete. 2024-03-11T19:18:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.030+0000 7faccaf7d740 1 -- start start 2024-03-11T19:18:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.030+0000 7faccaf7d740 1 Processor -- start 2024-03-11T19:18:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.031+0000 7faccaf7d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4270 0x55d85a0de2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.031+0000 7faccaf7d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d4c60 0x55d85a0de7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.031+0000 7faccaf7d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d9dd0 0x55d85a0e5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.031+0000 7faccaf7d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d85a03d020 con 0x55d85a0d4270 2024-03-11T19:18:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.031+0000 7faccaf7d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d859d10430 con 0x55d85a0d4c60 2024-03-11T19:18:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.031+0000 7faccaf7d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d859ccce00 con 0x55d85a0d9dd0 2024-03-11T19:18:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.032+0000 7facc6871640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d9dd0 0x55d85a0e5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.032+0000 7facc586f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d4c60 0x55d85a0de7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.032+0000 7facc6871640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d9dd0 0x55d85a0e5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:46008/0 (socket says 172.21.15.29:46008) 2024-03-11T19:18:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.032+0000 7facc6871640 1 -- 172.21.15.29:0/3477223054 learned_addr learned my addr 172.21.15.29:0/3477223054 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.032+0000 7faccaf7d740 10 client.0 _ll_register_callbacks cb 0x55d859fe4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.032+0000 7facc586f640 1 -- 172.21.15.29:0/3477223054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d9dd0 msgr2=0x55d85a0e5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.032+0000 7facc586f640 1 --2- 172.21.15.29:0/3477223054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d9dd0 0x55d85a0e5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.032+0000 7facc586f640 1 -- 172.21.15.29:0/3477223054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4270 msgr2=0x55d85a0de2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:18:37.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.032+0000 7facc586f640 1 --2- 172.21.15.29:0/3477223054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4270 0x55d85a0de2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.032+0000 7facc586f640 1 -- 172.21.15.29:0/3477223054 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facbc00e910 con 0x55d85a0d4c60 2024-03-11T19:18:37.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.033+0000 7facc586f640 1 --2- 172.21.15.29:0/3477223054 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d4c60 0x55d85a0de7e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7facb0009780 tx=0x7facb00bb8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eba9113522b0dc2f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.033+0000 7faca9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:18:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.034+0000 7faca9ffb640 1 -- 172.21.15.29:0/3477223054 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb00c40e0 con 0x55d85a0d4c60 2024-03-11T19:18:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.034+0000 7faccaf7d740 1 -- 172.21.15.29:0/3477223054 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d859d3c7e0 con 0x55d85a0d4c60 2024-03-11T19:18:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.034+0000 7faca9ffb640 1 -- 172.21.15.29:0/3477223054 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7facb00c4240 con 0x55d85a0d4c60 2024-03-11T19:18:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.034+0000 7faca9ffb640 1 -- 172.21.15.29:0/3477223054 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7facb00da170 con 0x55d85a0d4c60 2024-03-11T19:18:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.034+0000 7faccaf7d740 10 client.27011 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.034+0000 7faccaf7d740 1 -- 172.21.15.29:0/3477223054 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d85a03b6a0 con 0x55d85a0d4c60 2024-03-11T19:18:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.035+0000 7faca9ffb640 1 -- 172.21.15.29:0/3477223054 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7facb009e080 con 0x55d85a0d4c60 2024-03-11T19:18:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.035+0000 7faca9ffb640 1 -- 172.21.15.29:0/3477223054 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7facb00d4b00 con 0x55d85a0d4c60 2024-03-11T19:18:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.035+0000 7faccaf7d740 10 client.27011 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.035+0000 7faccaf7d740 10 client.27011 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:18:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.035+0000 7faccaf7d740 20 client.27011 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.035+0000 7faccaf7d740 1 --2- 172.21.15.29:0/3477223054 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d85a0e9fc0 0x55d85a0ec3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.035+0000 7faccaf7d740 4 client.27011 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:18:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.035+0000 7faccaf7d740 1 -- 172.21.15.29:0/3477223054 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d859d79720 con 0x55d85a0e9fc0 2024-03-11T19:18:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.037+0000 7facc6070640 1 --2- 172.21.15.29:0/3477223054 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d85a0e9fc0 0x55d85a0ec3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:37.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.037+0000 7facc6070640 1 --2- 172.21.15.29:0/3477223054 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d85a0e9fc0 0x55d85a0ec3b0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c36be12d316c76f3 server_cookie=6e59e1f4649d279d in_seq=0 out_seq=0 2024-03-11T19:18:37.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.037+0000 7faca9ffb640 10 client.27011 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:37.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.039+0000 7faca9ffb640 1 -- 172.21.15.29:0/3477223054 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d859d79720 con 0x55d85a0e9fc0 2024-03-11T19:18:37.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.039+0000 7faca9ffb640 10 client.27011 handle_command_reply: tid=0 2024-03-11T19:18:37.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.039+0000 7faccaf7d740 1 client.27011 shutdown 2024-03-11T19:18:37.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.039+0000 7faccaf7d740 1 -- 172.21.15.29:0/3477223054 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d4c60 msgr2=0x55d85a0de7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:37.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.039+0000 7faccaf7d740 1 --2- 172.21.15.29:0/3477223054 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d4c60 0x55d85a0de7e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7facb0009780 tx=0x7facb00bb8a0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.039+0000 7faccaf7d740 1 -- 172.21.15.29:0/3477223054 shutdown_connections 2024-03-11T19:18:37.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.039+0000 7faccaf7d740 1 --2- 172.21.15.29:0/3477223054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d9dd0 0x55d85a0e5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.039+0000 7faccaf7d740 1 --2- 172.21.15.29:0/3477223054 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d4c60 0x55d85a0de7e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.039+0000 7faccaf7d740 1 --2- 172.21.15.29:0/3477223054 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d85a0e9fc0 0x55d85a0ec3b0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.039+0000 7faccaf7d740 1 --2- 172.21.15.29:0/3477223054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4270 0x55d85a0de2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.039+0000 7faca9ffb640 0 client.27011 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:37.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.039+0000 7faccaf7d740 1 -- 172.21.15.29:0/3477223054 >> 172.21.15.29:0/3477223054 conn(0x55d85a0cf7e0 msgr2=0x55d85a0ceb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:37.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.039+0000 7faccaf7d740 1 -- 172.21.15.29:0/3477223054 shutdown_connections 2024-03-11T19:18:37.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.039+0000 7faccaf7d740 1 -- 172.21.15.29:0/3477223054 wait complete. 2024-03-11T19:18:37.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.040+0000 7faccaf7d740 20 client.27011 trim_cache size 0 max 16384 2024-03-11T19:18:37.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.063+0000 7faccaf7d740 1 Processor -- start 2024-03-11T19:18:37.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.064+0000 7faccaf7d740 1 -- start start 2024-03-11T19:18:37.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.064+0000 7faccaf7d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4270 0x55d85a0d5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:37.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.064+0000 7faccaf7d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d4c60 0x55d85a0e57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:37.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.064+0000 7facc6070640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4270 0x55d85a0d5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:37.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.064+0000 7facc6070640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4270 0x55d85a0d5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51850/0 (socket says 172.21.15.29:51850) 2024-03-11T19:18:37.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.065+0000 7facc6070640 1 -- 172.21.15.29:0/3329580582 learned_addr learned my addr 172.21.15.29:0/3329580582 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:37.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.065+0000 7faccaf7d740 1 --2- 172.21.15.29:0/3329580582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d9dd0 0x55d85a0d6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:37.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.065+0000 7facc586f640 1 --2- 172.21.15.29:0/3329580582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d4c60 0x55d85a0e57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:37.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.065+0000 7faccaf7d740 1 -- 172.21.15.29:0/3329580582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d859fc2130 con 0x55d85a0d4270 2024-03-11T19:18:37.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.065+0000 7faccaf7d740 1 -- 172.21.15.29:0/3329580582 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d85a0ec5e0 con 0x55d85a0d4c60 2024-03-11T19:18:37.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.065+0000 7facc6871640 1 --2- 172.21.15.29:0/3329580582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d9dd0 0x55d85a0d6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:37.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.065+0000 7faccaf7d740 1 -- 172.21.15.29:0/3329580582 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d85a0d02d0 con 0x55d85a0d9dd0 2024-03-11T19:18:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.066+0000 7facc6871640 1 -- 172.21.15.29:0/3329580582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d4c60 msgr2=0x55d85a0e57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.066+0000 7facc6871640 1 --2- 172.21.15.29:0/3329580582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d4c60 0x55d85a0e57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.066+0000 7facc6871640 1 -- 172.21.15.29:0/3329580582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4270 msgr2=0x55d85a0d5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.066+0000 7facc6871640 1 --2- 172.21.15.29:0/3329580582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4270 0x55d85a0d5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.066+0000 7facc6871640 1 -- 172.21.15.29:0/3329580582 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d859fd7c20 con 0x55d85a0d9dd0 2024-03-11T19:18:37.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.067+0000 7facc6871640 1 --2- 172.21.15.29:0/3329580582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d9dd0 0x55d85a0d6520 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x55d85a0d5580 tx=0x7facb80095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6688e890b296dcb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:37.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.068+0000 7facabfff640 1 -- 172.21.15.29:0/3329580582 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb8013040 con 0x55d85a0d9dd0 2024-03-11T19:18:37.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.068+0000 7facabfff640 1 -- 172.21.15.29:0/3329580582 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7facb8003e20 con 0x55d85a0d9dd0 2024-03-11T19:18:37.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.068+0000 7facabfff640 1 -- 172.21.15.29:0/3329580582 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb8004020 con 0x55d85a0d9dd0 2024-03-11T19:18:37.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.070+0000 7faccaf7d740 1 -- 172.21.15.29:0/3329580582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d9dd0 msgr2=0x55d85a0d6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:37.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.070+0000 7faccaf7d740 1 --2- 172.21.15.29:0/3329580582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d9dd0 0x55d85a0d6520 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x55d85a0d5580 tx=0x7facb80095c0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.070+0000 7faccaf7d740 1 -- 172.21.15.29:0/3329580582 shutdown_connections 2024-03-11T19:18:37.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.071+0000 7faccaf7d740 1 --2- 172.21.15.29:0/3329580582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d9dd0 0x55d85a0d6520 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.071+0000 7faccaf7d740 1 --2- 172.21.15.29:0/3329580582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d4c60 0x55d85a0e57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.071+0000 7faccaf7d740 1 --2- 172.21.15.29:0/3329580582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4270 0x55d85a0d5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.071+0000 7faccaf7d740 1 -- 172.21.15.29:0/3329580582 >> 172.21.15.29:0/3329580582 conn(0x55d85a0cf7e0 msgr2=0x55d85a0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:37.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.072+0000 7faccaf7d740 1 -- 172.21.15.29:0/3329580582 shutdown_connections 2024-03-11T19:18:37.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.072+0000 7faccaf7d740 1 -- 172.21.15.29:0/3329580582 wait complete. 2024-03-11T19:18:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.072+0000 7faccaf7d740 1 -- start start 2024-03-11T19:18:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.073+0000 7faccaf7d740 1 Processor -- start 2024-03-11T19:18:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.073+0000 7faccaf7d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4270 0x55d85a0e42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.073+0000 7faccaf7d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d4c60 0x55d85a0e4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.074+0000 7faccaf7d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d9dd0 0x55d85a140480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.074+0000 7facc586f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d4c60 0x55d85a0e4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.074+0000 7facc586f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d4c60 0x55d85a0e4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:46028/0 (socket says 172.21.15.29:46028) 2024-03-11T19:18:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.074+0000 7facc6070640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4270 0x55d85a0e42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.074+0000 7facc6070640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4270 0x55d85a0e42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51852/0 (socket says 172.21.15.29:51852) 2024-03-11T19:18:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.074+0000 7facc586f640 1 -- 172.21.15.29:0/2463591091 learned_addr learned my addr 172.21.15.29:0/2463591091 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.074+0000 7facc6871640 1 --2- 172.21.15.29:0/2463591091 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d9dd0 0x55d85a140480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.074+0000 7faccaf7d740 1 -- 172.21.15.29:0/2463591091 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d85a0e4f80 con 0x55d85a0d4270 2024-03-11T19:18:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.074+0000 7faccaf7d740 1 -- 172.21.15.29:0/2463591091 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d85a0e50f0 con 0x55d85a0d9dd0 2024-03-11T19:18:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.074+0000 7faccaf7d740 1 -- 172.21.15.29:0/2463591091 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d85a1409c0 con 0x55d85a0d4c60 2024-03-11T19:18:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.075+0000 7faccaf7d740 10 client.0 _ll_register_callbacks cb 0x7facbc009690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.075+0000 7facc586f640 1 -- 172.21.15.29:0/2463591091 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d9dd0 msgr2=0x55d85a140480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.075+0000 7facc586f640 1 --2- 172.21.15.29:0/2463591091 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d9dd0 0x55d85a140480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.075+0000 7facc586f640 1 -- 172.21.15.29:0/2463591091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4270 msgr2=0x55d85a0e42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.075+0000 7facc586f640 1 --2- 172.21.15.29:0/2463591091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4270 0x55d85a0e42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.075+0000 7facc586f640 1 -- 172.21.15.29:0/2463591091 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb8009d70 con 0x55d85a0d4c60 2024-03-11T19:18:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.075+0000 7faccaf7d740 1 -- 172.21.15.29:0/2463591091 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7facbc015ce0 con 0x55d85a0d4c60 2024-03-11T19:18:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.075+0000 7facc586f640 1 --2- 172.21.15.29:0/2463591091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d4c60 0x55d85a0e4820 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7facb00ba130 tx=0x7facb002bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9684494fb3c38710 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:37.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.076+0000 7faca9ffb640 10 client.27055 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:18:37.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.076+0000 7faca9ffb640 1 -- 172.21.15.29:0/2463591091 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb00bb4e0 con 0x55d85a0d4c60 2024-03-11T19:18:37.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.076+0000 7faca9ffb640 1 -- 172.21.15.29:0/2463591091 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7facb00c40e0 con 0x55d85a0d4c60 2024-03-11T19:18:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.077+0000 7faca9ffb640 1 -- 172.21.15.29:0/2463591091 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7facb00c42e0 con 0x55d85a0d4c60 2024-03-11T19:18:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.077+0000 7faccaf7d740 10 client.27055 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.077+0000 7faccaf7d740 1 -- 172.21.15.29:0/2463591091 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x55d859ffd8c0 con 0x55d85a0d4c60 2024-03-11T19:18:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.077+0000 7faca9ffb640 1 -- 172.21.15.29:0/2463591091 <== mon.1 v2:172.21.15.190:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7facb00a5210 con 0x55d85a0d4c60 2024-03-11T19:18:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.077+0000 7faccaf7d740 10 client.27055 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.077+0000 7faccaf7d740 10 client.27055 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:18:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.077+0000 7faccaf7d740 20 client.27055 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.077+0000 7faccaf7d740 1 --2- 172.21.15.29:0/2463591091 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d85a141cd0 0x55d85a1440c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.077+0000 7faccaf7d740 4 client.27055 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:18:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.077+0000 7faccaf7d740 1 -- 172.21.15.29:0/2463591091 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7facbc015ce0 con 0x55d85a141cd0 2024-03-11T19:18:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.078+0000 7facc6070640 1 --2- 172.21.15.29:0/2463591091 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d85a141cd0 0x55d85a1440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.079+0000 7facc6070640 1 --2- 172.21.15.29:0/2463591091 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d85a141cd0 0x55d85a1440c0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6c9e053c428ef08 server_cookie=a8f54158d757e794 in_seq=0 out_seq=0 2024-03-11T19:18:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.079+0000 7faca9ffb640 10 client.27055 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.079+0000 7faca9ffb640 1 -- 172.21.15.29:0/2463591091 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7facbc015ce0 con 0x55d85a141cd0 2024-03-11T19:18:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.079+0000 7faca9ffb640 10 client.27055 handle_command_reply: tid=0 2024-03-11T19:18:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.079+0000 7faccaf7d740 1 client.27055 shutdown 2024-03-11T19:18:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.079+0000 7faccaf7d740 1 -- 172.21.15.29:0/2463591091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d4c60 msgr2=0x55d85a0e4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.079+0000 7faccaf7d740 1 --2- 172.21.15.29:0/2463591091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d4c60 0x55d85a0e4820 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7facb00ba130 tx=0x7facb002bb70 comp rx=0 tx=0).stop 2024-03-11T19:18:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.080+0000 7faccaf7d740 1 -- 172.21.15.29:0/2463591091 shutdown_connections 2024-03-11T19:18:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.080+0000 7faccaf7d740 1 --2- 172.21.15.29:0/2463591091 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d85a0d9dd0 0x55d85a140480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.080+0000 7faccaf7d740 1 --2- 172.21.15.29:0/2463591091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d85a0d4c60 0x55d85a0e4820 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.080+0000 7faccaf7d740 1 --2- 172.21.15.29:0/2463591091 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d85a141cd0 0x55d85a1440c0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.080+0000 7faccaf7d740 1 --2- 172.21.15.29:0/2463591091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d85a0d4270 0x55d85a0e42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.080+0000 7faca9ffb640 0 client.27055 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:37.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.080+0000 7faccaf7d740 1 -- 172.21.15.29:0/2463591091 >> 172.21.15.29:0/2463591091 conn(0x55d85a0cf7e0 msgr2=0x55d85a0d5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:37.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.080+0000 7faccaf7d740 1 -- 172.21.15.29:0/2463591091 shutdown_connections 2024-03-11T19:18:37.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.080+0000 7faccaf7d740 1 -- 172.21.15.29:0/2463591091 wait complete. 2024-03-11T19:18:37.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.080+0000 7faccaf7d740 20 client.27055 trim_cache size 0 max 16384 2024-03-11T19:18:37.085 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:18:37.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.081+0000 7facc82d4640 1 -- 172.21.15.29:0/1367149667 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7facac0a1ab0 msgr2=0x7facac0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:37.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.081+0000 7facc82d4640 1 --2- 172.21.15.29:0/1367149667 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7facac0a1ab0 0x7facac0a3f70 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7facbc009a40 tx=0x7facbc0091c0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.081+0000 7facc82d4640 1 -- 172.21.15.29:0/1367149667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facc010aaf0 msgr2=0x7facc01a6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:37.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.081+0000 7facc82d4640 1 --2- 172.21.15.29:0/1367149667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facc010aaf0 0x7facc01a6550 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7facb002c0b0 tx=0x7facb00042f0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.081+0000 7facc82d4640 1 -- 172.21.15.29:0/1367149667 shutdown_connections 2024-03-11T19:18:37.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.081+0000 7facc82d4640 1 --2- 172.21.15.29:0/1367149667 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7facac0a1ab0 0x7facac0a3f70 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.081+0000 7facc82d4640 1 --2- 172.21.15.29:0/1367149667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facc010d4a0 0x7facc01ad5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.081+0000 7facc82d4640 1 --2- 172.21.15.29:0/1367149667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facc010aaf0 0x7facc01a6550 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.081+0000 7facc82d4640 1 --2- 172.21.15.29:0/1367149667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc0108180 0x7facc01a6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:37.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.081+0000 7facc82d4640 1 -- 172.21.15.29:0/1367149667 >> 172.21.15.29:0/1367149667 conn(0x7facc006dfe0 msgr2=0x7facc0109560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:37.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.081+0000 7facc82d4640 1 -- 172.21.15.29:0/1367149667 shutdown_connections 2024-03-11T19:18:37.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:37.081+0000 7facc82d4640 1 -- 172.21.15.29:0/1367149667 wait complete. 2024-03-11T19:18:37.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:37 smithi029 ceph-mon[32439]: pgmap v615: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 66 MiB/s wr, 1.24k op/s 2024-03-11T19:18:37.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:37 smithi190 ceph-mon[36292]: pgmap v615: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 66 MiB/s wr, 1.24k op/s 2024-03-11T19:18:37.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:37 smithi170 ceph-mon[35690]: pgmap v615: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 66 MiB/s wr, 1.24k op/s 2024-03-11T19:18:38.107 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:18:38.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:18:38.107 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:18:38.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.285+0000 7fe6d528a640 1 -- 172.21.15.29:0/3590422896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d010adb0 msgr2=0x7fe6d010d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:38.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.285+0000 7fe6d528a640 1 --2- 172.21.15.29:0/3590422896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d010adb0 0x7fe6d010d1c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe6c40074e0 tx=0x7fe6c402c8d0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.285+0000 7fe6d528a640 1 -- 172.21.15.29:0/3590422896 shutdown_connections 2024-03-11T19:18:38.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.285+0000 7fe6d528a640 1 --2- 172.21.15.29:0/3590422896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d010d850 0x7fe6d010fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.285+0000 7fe6d528a640 1 --2- 172.21.15.29:0/3590422896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d010adb0 0x7fe6d010d1c0 secure :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe6c40074e0 tx=0x7fe6c402c8d0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.285+0000 7fe6d528a640 1 --2- 172.21.15.29:0/3590422896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d01083f0 0x7fe6d010a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.285+0000 7fe6d528a640 1 -- 172.21.15.29:0/3590422896 >> 172.21.15.29:0/3590422896 conn(0x7fe6d006dfe0 msgr2=0x7fe6d0070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.286+0000 7fe6d528a640 1 -- 172.21.15.29:0/3590422896 shutdown_connections 2024-03-11T19:18:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.286+0000 7fe6d528a640 1 -- 172.21.15.29:0/3590422896 wait complete. 2024-03-11T19:18:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.287+0000 7fe6d528a640 1 Processor -- start 2024-03-11T19:18:38.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.287+0000 7fe6d528a640 1 -- start start 2024-03-11T19:18:38.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.288+0000 7fe6d528a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d01083f0 0x7fe6d019da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.288+0000 7fe6d528a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d010d850 0x7fe6d019dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.288+0000 7fe6d528a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d01a3000 0x7fe6d01a34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.288+0000 7fe6d528a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d01a39f0 con 0x7fe6d010d850 2024-03-11T19:18:38.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.288+0000 7fe6d528a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d01a3b60 con 0x7fe6d01083f0 2024-03-11T19:18:38.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.288+0000 7fe6d528a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d01a3d20 con 0x7fe6d01a3000 2024-03-11T19:18:38.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.288+0000 7fe6cf7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d01a3000 0x7fe6d01a34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.288+0000 7fe6cf7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d01a3000 0x7fe6d01a34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:46052/0 (socket says 172.21.15.29:46052) 2024-03-11T19:18:38.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.288+0000 7fe6cf7fe640 1 -- 172.21.15.29:0/3992745531 learned_addr learned my addr 172.21.15.29:0/3992745531 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:38.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.288+0000 7fe6ce7fc640 1 --2- 172.21.15.29:0/3992745531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d010d850 0x7fe6d019dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.288+0000 7fe6cf7fe640 1 -- 172.21.15.29:0/3992745531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d01083f0 msgr2=0x7fe6d019da60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-11T19:18:38.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.288+0000 7fe6cf7fe640 1 --2- 172.21.15.29:0/3992745531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d01083f0 0x7fe6d019da60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.288+0000 7fe6cf7fe640 1 -- 172.21.15.29:0/3992745531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d010d850 msgr2=0x7fe6d019dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:38.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.288+0000 7fe6cf7fe640 1 --2- 172.21.15.29:0/3992745531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d010d850 0x7fe6d019dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.288+0000 7fe6cf7fe640 1 -- 172.21.15.29:0/3992745531 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c40071c0 con 0x7fe6d01a3000 2024-03-11T19:18:38.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.289+0000 7fe6cf7fe640 1 --2- 172.21.15.29:0/3992745531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d01a3000 0x7fe6d01a34b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe6c800eb40 tx=0x7fe6c800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:38.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.289+0000 7fe6b7fff640 1 -- 172.21.15.29:0/3992745531 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c8004b20 con 0x7fe6d01a3000 2024-03-11T19:18:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.290+0000 7fe6d528a640 1 -- 172.21.15.29:0/3992745531 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d01a4000 con 0x7fe6d01a3000 2024-03-11T19:18:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.290+0000 7fe6d528a640 1 -- 172.21.15.29:0/3992745531 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d01a4560 con 0x7fe6d01a3000 2024-03-11T19:18:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.290+0000 7fe6b7fff640 1 -- 172.21.15.29:0/3992745531 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe6c8004c80 con 0x7fe6d01a3000 2024-03-11T19:18:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.290+0000 7fe6b7fff640 1 -- 172.21.15.29:0/3992745531 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c80186d0 con 0x7fe6d01a3000 2024-03-11T19:18:38.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.295+0000 7fe6b7fff640 1 -- 172.21.15.29:0/3992745531 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe6c8018830 con 0x7fe6d01a3000 2024-03-11T19:18:38.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.295+0000 7fe6d7e57740 1 Processor -- start 2024-03-11T19:18:38.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.296+0000 7fe6d7e57740 1 -- start start 2024-03-11T19:18:38.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.296+0000 7fe6d7e57740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf1a4c0 0x558c5bf1a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.296+0000 7fe6d7e57740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf1b690 0x558c5bf1bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.296+0000 7fe6d7e57740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf1c050 0x558c5bf228e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.296+0000 7fe6d7e57740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c5be89f90 con 0x558c5bf1b690 2024-03-11T19:18:38.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.296+0000 7fe6d7e57740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c5be874c0 con 0x558c5bf1a4c0 2024-03-11T19:18:38.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.296+0000 7fe6d7e57740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c5be82ca0 con 0x558c5bf1c050 2024-03-11T19:18:38.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.296+0000 7fe6b7fff640 1 --2- 172.21.15.29:0/3992745531 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe6b80a1be0 0x7fe6b80a40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.297+0000 7fe6b7fff640 1 -- 172.21.15.29:0/3992745531 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe6c8014070 con 0x7fe6d01a3000 2024-03-11T19:18:38.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.297+0000 7fe6cf7fe640 1 --2- 172.21.15.29:0/3992745531 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe6b80a1be0 0x7fe6b80a40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.298+0000 7fe6ce7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf1c050 0x558c5bf228e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.298+0000 7fe6ce7fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf1a4c0 0x558c5bf1a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.298+0000 7fe6ce7fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf1a4c0 0x558c5bf1a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:47546/0 (socket says 172.21.15.29:47546) 2024-03-11T19:18:38.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.298+0000 7fe6ce7fc640 1 -- 172.21.15.29:0/2454028538 learned_addr learned my addr 172.21.15.29:0/2454028538 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:38.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.299+0000 7fe6ce7fc640 1 -- 172.21.15.29:0/2454028538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf1c050 msgr2=0x558c5bf228e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:38.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.299+0000 7fe6ce7fc640 1 --2- 172.21.15.29:0/2454028538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf1c050 0x558c5bf228e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.299+0000 7fe6ce7fc640 1 -- 172.21.15.29:0/2454028538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf1b690 msgr2=0x558c5bf1bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:38.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.299+0000 7fe6ce7fc640 1 --2- 172.21.15.29:0/2454028538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf1b690 0x558c5bf1bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.299+0000 7fe6ce7fc640 1 -- 172.21.15.29:0/2454028538 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c5be450b0 con 0x558c5bf1a4c0 2024-03-11T19:18:38.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.299+0000 7fe6cf7fe640 1 --2- 172.21.15.29:0/3992745531 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe6b80a1be0 0x7fe6b80a40a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe6c80ceb40 tx=0x7fe6c80ceab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:38.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.299+0000 7fe6ce7fc640 1 --2- 172.21.15.29:0/2454028538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf1a4c0 0x558c5bf1a8e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4005ec0 tx=0x7fe6c4000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60bf514a5de808df server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:38.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.300+0000 7fe6b57fa640 1 -- 172.21.15.29:0/2454028538 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4036040 con 0x558c5bf1a4c0 2024-03-11T19:18:38.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.300+0000 7fe6b57fa640 1 -- 172.21.15.29:0/2454028538 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe6c4045030 con 0x558c5bf1a4c0 2024-03-11T19:18:38.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.300+0000 7fe6b57fa640 1 -- 172.21.15.29:0/2454028538 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4005500 con 0x558c5bf1a4c0 2024-03-11T19:18:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.301+0000 7fe6d7e57740 1 -- 172.21.15.29:0/2454028538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf1a4c0 msgr2=0x558c5bf1a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.301+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/2454028538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf1a4c0 0x558c5bf1a8e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4005ec0 tx=0x7fe6c4000bc0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.302+0000 7fe6d7e57740 1 -- 172.21.15.29:0/2454028538 shutdown_connections 2024-03-11T19:18:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.302+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/2454028538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf1c050 0x558c5bf228e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.302+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/2454028538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf1b690 0x558c5bf1bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.302+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/2454028538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf1a4c0 0x558c5bf1a8e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.302+0000 7fe6d7e57740 1 -- 172.21.15.29:0/2454028538 >> 172.21.15.29:0/2454028538 conn(0x558c5bf156a0 msgr2=0x558c5bf15aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:38.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.302+0000 7fe6d7e57740 1 -- 172.21.15.29:0/2454028538 shutdown_connections 2024-03-11T19:18:38.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.305+0000 7fe6d7e57740 1 -- 172.21.15.29:0/2454028538 wait complete. 2024-03-11T19:18:38.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.306+0000 7fe6d7e57740 1 -- start start 2024-03-11T19:18:38.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.306+0000 7fe6d7e57740 1 Processor -- start 2024-03-11T19:18:38.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.306+0000 7fe6d7e57740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf1c050 0x558c5bf249c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.306+0000 7fe6d7e57740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf24f00 0x558c5bf29f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.306+0000 7fe6d7e57740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf25380 0x558c5bf2c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.306+0000 7fe6d7e57740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c5be27f40 con 0x558c5bf1c050 2024-03-11T19:18:38.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.306+0000 7fe6d7e57740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c5bb564b0 con 0x558c5bf25380 2024-03-11T19:18:38.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.306+0000 7fe6d7e57740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c5be08670 con 0x558c5bf24f00 2024-03-11T19:18:38.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.306+0000 7fe6d7e57740 10 client.0 _ll_register_callbacks cb 0x558c5be75f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:38.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.307+0000 7fe6ce7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf1c050 0x558c5bf249c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.307+0000 7fe6ce7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf1c050 0x558c5bf249c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51888/0 (socket says 172.21.15.29:51888) 2024-03-11T19:18:38.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.307+0000 7fe6ce7fc640 1 -- 172.21.15.29:0/3382808850 learned_addr learned my addr 172.21.15.29:0/3382808850 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:38.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.307+0000 7fe6ce7fc640 1 --2- 172.21.15.29:0/3382808850 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf25380 0x558c5bf2c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.307+0000 7fe6ceffd640 1 --2- 172.21.15.29:0/3382808850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf24f00 0x558c5bf29f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.307+0000 7fe6ce7fc640 1 -- 172.21.15.29:0/3382808850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf24f00 msgr2=0x558c5bf29f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:38.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.307+0000 7fe6ce7fc640 1 --2- 172.21.15.29:0/3382808850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf24f00 0x558c5bf29f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.307+0000 7fe6ce7fc640 1 -- 172.21.15.29:0/3382808850 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf25380 msgr2=0x558c5bf2c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:38.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.308+0000 7fe6ce7fc640 1 --2- 172.21.15.29:0/3382808850 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf25380 0x558c5bf2c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.308+0000 7fe6ce7fc640 1 -- 172.21.15.29:0/3382808850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d01108f0 con 0x558c5bf1c050 2024-03-11T19:18:38.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.309+0000 7fe6ce7fc640 1 --2- 172.21.15.29:0/3382808850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf1c050 0x558c5bf249c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4004bb0 tx=0x7fe6c4005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=500b02c0c598e77e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:38.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.309+0000 7fe6b2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:18:38.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.310+0000 7fe6b2ff5640 1 -- 172.21.15.29:0/3382808850 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4036040 con 0x558c5bf1c050 2024-03-11T19:18:38.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.310+0000 7fe6b2ff5640 1 -- 172.21.15.29:0/3382808850 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe6c4004230 con 0x558c5bf1c050 2024-03-11T19:18:38.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.310+0000 7fe6d7e57740 1 -- 172.21.15.29:0/3382808850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558c5be5cf40 con 0x558c5bf1c050 2024-03-11T19:18:38.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.310+0000 7fe6b2ff5640 1 -- 172.21.15.29:0/3382808850 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6c40043f0 con 0x558c5bf1c050 2024-03-11T19:18:38.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.310+0000 7fe6d7e57740 10 client.17535 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:38.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.310+0000 7fe6d7e57740 1 -- 172.21.15.29:0/3382808850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x558c5be81320 con 0x558c5bf1c050 2024-03-11T19:18:38.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.311+0000 7fe6b2ff5640 1 -- 172.21.15.29:0/3382808850 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x558c5be5cf40 con 0x558c5bf1c050 2024-03-11T19:18:38.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.312+0000 7fe6b2ff5640 1 -- 172.21.15.29:0/3382808850 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe6c4057070 con 0x558c5bf1c050 2024-03-11T19:18:38.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.312+0000 7fe6d7e57740 10 client.17535 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:38.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.312+0000 7fe6d7e57740 10 client.17535 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:18:38.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.312+0000 7fe6d7e57740 20 client.17535 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:38.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.312+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/3382808850 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558c5bf30f10 0x558c5bf33330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.312+0000 7fe6d7e57740 4 client.17535 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:18:38.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.312+0000 7fe6d7e57740 1 -- 172.21.15.29:0/3382808850 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558c5be79310 con 0x558c5bf30f10 2024-03-11T19:18:38.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.313+0000 7fe6ceffd640 1 --2- 172.21.15.29:0/3382808850 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558c5bf30f10 0x558c5bf33330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.314+0000 7fe6b2ff5640 10 client.17535 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:38.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.314+0000 7fe6ceffd640 1 --2- 172.21.15.29:0/3382808850 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558c5bf30f10 0x558c5bf33330 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1263dbe4ea6413b6 server_cookie=a89871353e019272 in_seq=0 out_seq=0 2024-03-11T19:18:38.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.315+0000 7fe6b2ff5640 1 -- 172.21.15.29:0/3382808850 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558c5be79310 con 0x558c5bf30f10 2024-03-11T19:18:38.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.315+0000 7fe6b2ff5640 10 client.17535 handle_command_reply: tid=0 2024-03-11T19:18:38.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.315+0000 7fe6d7e57740 1 client.17535 shutdown 2024-03-11T19:18:38.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.317+0000 7fe6d7e57740 1 -- 172.21.15.29:0/3382808850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf1c050 msgr2=0x558c5bf249c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:38.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.317+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/3382808850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf1c050 0x558c5bf249c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4004bb0 tx=0x7fe6c4005000 comp rx=0 tx=0).stop 2024-03-11T19:18:38.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.317+0000 7fe6d7e57740 1 -- 172.21.15.29:0/3382808850 shutdown_connections 2024-03-11T19:18:38.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.317+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/3382808850 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf25380 0x558c5bf2c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.317+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/3382808850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf24f00 0x558c5bf29f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.317+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/3382808850 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558c5bf30f10 0x558c5bf33330 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.317+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/3382808850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf1c050 0x558c5bf249c0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.317+0000 7fe6d7e57740 1 -- 172.21.15.29:0/3382808850 >> 172.21.15.29:0/3382808850 conn(0x558c5bf156a0 msgr2=0x558c5bf148e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:38.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.317+0000 7fe6b2ff5640 0 client.17535 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:38.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.318+0000 7fe6d7e57740 1 -- 172.21.15.29:0/3382808850 shutdown_connections 2024-03-11T19:18:38.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.318+0000 7fe6d7e57740 1 -- 172.21.15.29:0/3382808850 wait complete. 2024-03-11T19:18:38.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.318+0000 7fe6d7e57740 20 client.17535 trim_cache size 0 max 16384 2024-03-11T19:18:38.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.362+0000 7fe6d7e57740 1 Processor -- start 2024-03-11T19:18:38.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.362+0000 7fe6d7e57740 1 -- start start 2024-03-11T19:18:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.363+0000 7fe6d7e57740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf1c050 0x558c5bf249a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.363+0000 7fe6d7e57740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf24f00 0x558c5bf29ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.363+0000 7fe6ce7fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf1c050 0x558c5bf249a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.363+0000 7fe6ce7fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf1c050 0x558c5bf249a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:47564/0 (socket says 172.21.15.29:47564) 2024-03-11T19:18:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.363+0000 7fe6ce7fc640 1 -- 172.21.15.29:0/1539319043 learned_addr learned my addr 172.21.15.29:0/1539319043 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.363+0000 7fe6ceffd640 1 --2- 172.21.15.29:0/1539319043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf24f00 0x558c5bf29ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.363+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/1539319043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf25380 0x558c5bf2a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.364+0000 7fe6d7e57740 1 -- 172.21.15.29:0/1539319043 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c5be5cf40 con 0x558c5bf24f00 2024-03-11T19:18:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.364+0000 7fe6d7e57740 1 -- 172.21.15.29:0/1539319043 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c5bf24530 con 0x558c5bf1c050 2024-03-11T19:18:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.364+0000 7fe6d7e57740 1 -- 172.21.15.29:0/1539319043 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c5bf1c460 con 0x558c5bf25380 2024-03-11T19:18:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.364+0000 7fe6ce7fc640 1 --2- 172.21.15.29:0/1539319043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf25380 0x558c5bf2a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.364+0000 7fe6ce7fc640 1 -- 172.21.15.29:0/1539319043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf25380 msgr2=0x558c5bf2a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.364+0000 7fe6ce7fc640 1 --2- 172.21.15.29:0/1539319043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf25380 0x558c5bf2a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.364+0000 7fe6ce7fc640 1 -- 172.21.15.29:0/1539319043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf24f00 msgr2=0x558c5bf29ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.364+0000 7fe6ce7fc640 1 --2- 172.21.15.29:0/1539319043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf24f00 0x558c5bf29ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.364+0000 7fe6ce7fc640 1 -- 172.21.15.29:0/1539319043 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c5be1dc00 con 0x558c5bf1c050 2024-03-11T19:18:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.364+0000 7fe6ceffd640 1 --2- 172.21.15.29:0/1539319043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf24f00 0x558c5bf29ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:18:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.364+0000 7fe6ce7fc640 1 --2- 172.21.15.29:0/1539319043 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf1c050 0x558c5bf249a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4005c30 tx=0x7fe6c4052000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f0ce845949757bd server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.367+0000 7fe6b4ff9640 1 -- 172.21.15.29:0/1539319043 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4036050 con 0x558c5bf1c050 2024-03-11T19:18:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.367+0000 7fe6b4ff9640 1 -- 172.21.15.29:0/1539319043 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe6c4045070 con 0x558c5bf1c050 2024-03-11T19:18:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.368+0000 7fe6b4ff9640 1 -- 172.21.15.29:0/1539319043 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4004850 con 0x558c5bf1c050 2024-03-11T19:18:38.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.369+0000 7fe6d7e57740 1 -- 172.21.15.29:0/1539319043 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf1c050 msgr2=0x558c5bf249a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:38.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.369+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/1539319043 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf1c050 0x558c5bf249a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4005c30 tx=0x7fe6c4052000 comp rx=0 tx=0).stop 2024-03-11T19:18:38.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.369+0000 7fe6d7e57740 1 -- 172.21.15.29:0/1539319043 shutdown_connections 2024-03-11T19:18:38.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.369+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/1539319043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf25380 0x558c5bf2a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.369+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/1539319043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf24f00 0x558c5bf29ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.369+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/1539319043 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf1c050 0x558c5bf249a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.369+0000 7fe6d7e57740 1 -- 172.21.15.29:0/1539319043 >> 172.21.15.29:0/1539319043 conn(0x558c5bf156a0 msgr2=0x558c5bf1f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:38.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.369+0000 7fe6d7e57740 1 -- 172.21.15.29:0/1539319043 shutdown_connections 2024-03-11T19:18:38.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.369+0000 7fe6d7e57740 1 -- 172.21.15.29:0/1539319043 wait complete. 2024-03-11T19:18:38.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.370+0000 7fe6d7e57740 1 -- start start 2024-03-11T19:18:38.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.370+0000 7fe6d7e57740 1 Processor -- start 2024-03-11T19:18:38.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.370+0000 7fe6d7e57740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf1c050 0x558c5bf28350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.370+0000 7fe6d7e57740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf24f00 0x558c5bf28890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.371+0000 7fe6d7e57740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf25380 0x558c5bf29830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.371+0000 7fe6d7e57740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c5bf29d70 con 0x558c5bf1c050 2024-03-11T19:18:38.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.371+0000 7fe6d7e57740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c5bf29ee0 con 0x558c5bf24f00 2024-03-11T19:18:38.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.371+0000 7fe6ceffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf24f00 0x558c5bf28890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.371+0000 7fe6ce7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf25380 0x558c5bf29830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.371+0000 7fe6ceffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf24f00 0x558c5bf28890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:47576/0 (socket says 172.21.15.29:47576) 2024-03-11T19:18:38.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.371+0000 7fe6ceffd640 1 -- 172.21.15.29:0/4060702577 learned_addr learned my addr 172.21.15.29:0/4060702577 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:38.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.371+0000 7fe6d7e57740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c5bf2a0a0 con 0x558c5bf25380 2024-03-11T19:18:38.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.371+0000 7fe6ce7fc640 1 --2- 172.21.15.29:0/4060702577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf1c050 0x558c5bf28350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.371+0000 7fe6d7e57740 10 client.0 _ll_register_callbacks cb 0x7fe6c4004f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:38.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.371+0000 7fe6ce7fc640 1 -- 172.21.15.29:0/4060702577 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf24f00 msgr2=0x558c5bf28890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:38.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.371+0000 7fe6ce7fc640 1 --2- 172.21.15.29:0/4060702577 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf24f00 0x558c5bf28890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.371+0000 7fe6ce7fc640 1 -- 172.21.15.29:0/4060702577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf1c050 msgr2=0x558c5bf28350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:38.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.371+0000 7fe6ce7fc640 1 --2- 172.21.15.29:0/4060702577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf1c050 0x558c5bf28350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.371+0000 7fe6ce7fc640 1 -- 172.21.15.29:0/4060702577 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558c5be81320 con 0x558c5bf25380 2024-03-11T19:18:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.372+0000 7fe6d7e57740 1 -- 172.21.15.29:0/4060702577 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558c5bea8900 con 0x558c5bf25380 2024-03-11T19:18:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.372+0000 7fe6ce7fc640 1 --2- 172.21.15.29:0/4060702577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf25380 0x558c5bf29830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4003730 tx=0x7fe6c4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9500704bcb99be5a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.372+0000 7fe6b2ff5640 10 client.27073 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:18:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.372+0000 7fe6b2ff5640 1 -- 172.21.15.29:0/4060702577 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4036070 con 0x558c5bf25380 2024-03-11T19:18:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.372+0000 7fe6b2ff5640 1 -- 172.21.15.29:0/4060702577 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe6c4000e40 con 0x558c5bf25380 2024-03-11T19:18:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.372+0000 7fe6b2ff5640 1 -- 172.21.15.29:0/4060702577 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6c4004830 con 0x558c5bf25380 2024-03-11T19:18:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.372+0000 7fe6d7e57740 10 client.27073 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.372+0000 7fe6d7e57740 1 -- 172.21.15.29:0/4060702577 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x558c5be43910 con 0x558c5bf25380 2024-03-11T19:18:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.373+0000 7fe6b2ff5640 1 -- 172.21.15.29:0/4060702577 <== mon.1 v2:172.21.15.190:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe6c4057070 con 0x558c5bf25380 2024-03-11T19:18:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.373+0000 7fe6d7e57740 10 client.27073 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.373+0000 7fe6d7e57740 10 client.27073 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:18:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.373+0000 7fe6d7e57740 20 client.27073 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.373+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/4060702577 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558c5bf87f80 0x558c5bf8a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.373+0000 7fe6d7e57740 4 client.27073 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:18:38.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.373+0000 7fe6d7e57740 1 -- 172.21.15.29:0/4060702577 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x558c5bb82860 con 0x558c5bf87f80 2024-03-11T19:18:38.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.373+0000 7fe6ceffd640 1 --2- 172.21.15.29:0/4060702577 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558c5bf87f80 0x558c5bf8a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.374+0000 7fe6b2ff5640 10 client.27073 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:38.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.374+0000 7fe6ceffd640 1 --2- 172.21.15.29:0/4060702577 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558c5bf87f80 0x558c5bf8a3a0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85d837a854fd6365 server_cookie=646d87d6142d751e in_seq=0 out_seq=0 2024-03-11T19:18:38.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.375+0000 7fe6b2ff5640 1 -- 172.21.15.29:0/4060702577 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x558c5bb82860 con 0x558c5bf87f80 2024-03-11T19:18:38.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.375+0000 7fe6b2ff5640 10 client.27073 handle_command_reply: tid=0 2024-03-11T19:18:38.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.375+0000 7fe6d7e57740 1 client.27073 shutdown 2024-03-11T19:18:38.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.375+0000 7fe6d7e57740 1 -- 172.21.15.29:0/4060702577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf25380 msgr2=0x558c5bf29830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.375+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/4060702577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf25380 0x558c5bf29830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4003730 tx=0x7fe6c4000bc0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.375+0000 7fe6d7e57740 1 -- 172.21.15.29:0/4060702577 shutdown_connections 2024-03-11T19:18:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.375+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/4060702577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558c5bf25380 0x558c5bf29830 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.375+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/4060702577 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558c5bf24f00 0x558c5bf28890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.375+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/4060702577 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558c5bf87f80 0x558c5bf8a3a0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.375+0000 7fe6d7e57740 1 --2- 172.21.15.29:0/4060702577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558c5bf1c050 0x558c5bf28350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.375+0000 7fe6b2ff5640 0 client.27073 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.375+0000 7fe6d7e57740 1 -- 172.21.15.29:0/4060702577 >> 172.21.15.29:0/4060702577 conn(0x558c5bf156a0 msgr2=0x558c5bf1e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.376+0000 7fe6d7e57740 1 -- 172.21.15.29:0/4060702577 shutdown_connections 2024-03-11T19:18:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.376+0000 7fe6d7e57740 1 -- 172.21.15.29:0/4060702577 wait complete. 2024-03-11T19:18:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.376+0000 7fe6d7e57740 20 client.27073 trim_cache size 0 max 16384 2024-03-11T19:18:38.382 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:18:38.382 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:18:38.382 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "3ae2879e-5eac-4a4d-abd7-c9a8e8eee06d", 2024-03-11T19:18:38.382 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:18:38.382 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:18:38.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.378+0000 7fe6d528a640 1 -- 172.21.15.29:0/3992745531 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe6b80a1be0 msgr2=0x7fe6b80a40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:38.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.378+0000 7fe6d528a640 1 --2- 172.21.15.29:0/3992745531 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe6b80a1be0 0x7fe6b80a40a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe6c80ceb40 tx=0x7fe6c80ceab0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.378+0000 7fe6d528a640 1 -- 172.21.15.29:0/3992745531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d01a3000 msgr2=0x7fe6d01a34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:38.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.378+0000 7fe6d528a640 1 --2- 172.21.15.29:0/3992745531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d01a3000 0x7fe6d01a34b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe6c800eb40 tx=0x7fe6c800c2d0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.378+0000 7fe6d528a640 1 -- 172.21.15.29:0/3992745531 shutdown_connections 2024-03-11T19:18:38.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.378+0000 7fe6d528a640 1 --2- 172.21.15.29:0/3992745531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d01a3000 0x7fe6d01a34b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.378+0000 7fe6d528a640 1 --2- 172.21.15.29:0/3992745531 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe6b80a1be0 0x7fe6b80a40a0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.378+0000 7fe6d528a640 1 --2- 172.21.15.29:0/3992745531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d010d850 0x7fe6d019dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.378+0000 7fe6d528a640 1 --2- 172.21.15.29:0/3992745531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d01083f0 0x7fe6d019da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.378+0000 7fe6d528a640 1 -- 172.21.15.29:0/3992745531 >> 172.21.15.29:0/3992745531 conn(0x7fe6d006dfe0 msgr2=0x7fe6d010c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:38.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.379+0000 7fe6d528a640 1 -- 172.21.15.29:0/3992745531 shutdown_connections 2024-03-11T19:18:38.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.379+0000 7fe6d528a640 1 -- 172.21.15.29:0/3992745531 wait complete. 2024-03-11T19:18:38.403 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:18:38.403 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:18:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.590+0000 7fbc7837a640 1 -- 172.21.15.29:0/4027652466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc70101180 msgr2=0x7fbc701015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.590+0000 7fbc7837a640 1 --2- 172.21.15.29:0/4027652466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc70101180 0x7fbc701015a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c009900 tx=0x7fbc6c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:18:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.590+0000 7fbc7837a640 1 -- 172.21.15.29:0/4027652466 shutdown_connections 2024-03-11T19:18:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.590+0000 7fbc7837a640 1 --2- 172.21.15.29:0/4027652466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc70102d30 0x7fbc701095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.590+0000 7fbc7837a640 1 --2- 172.21.15.29:0/4027652466 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc70102370 0x7fbc701027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.590+0000 7fbc7837a640 1 --2- 172.21.15.29:0/4027652466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc70101180 0x7fbc701015a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.590+0000 7fbc7837a640 1 -- 172.21.15.29:0/4027652466 >> 172.21.15.29:0/4027652466 conn(0x7fbc700fc6f0 msgr2=0x7fbc700feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.590+0000 7fbc7837a640 1 -- 172.21.15.29:0/4027652466 shutdown_connections 2024-03-11T19:18:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.590+0000 7fbc7837a640 1 -- 172.21.15.29:0/4027652466 wait complete. 2024-03-11T19:18:38.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.591+0000 7fbc7837a640 1 Processor -- start 2024-03-11T19:18:38.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.591+0000 7fbc7837a640 1 -- start start 2024-03-11T19:18:38.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.591+0000 7fbc7837a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc70101180 0x7fbc7019d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.591+0000 7fbc7837a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc70102370 0x7fbc7019dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.591+0000 7fbc7837a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc70102d30 0x7fbc701a4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.591+0000 7fbc7837a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc701a5240 con 0x7fbc70102370 2024-03-11T19:18:38.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.591+0000 7fbc7837a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc701a53b0 con 0x7fbc70102d30 2024-03-11T19:18:38.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.591+0000 7fbc7837a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc701a5570 con 0x7fbc70101180 2024-03-11T19:18:38.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.591+0000 7fbc76116640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc70101180 0x7fbc7019d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.591+0000 7fbc76917640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc70102d30 0x7fbc701a4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.591+0000 7fbc76917640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc70102d30 0x7fbc701a4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:47594/0 (socket says 172.21.15.29:47594) 2024-03-11T19:18:38.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.591+0000 7fbc76116640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc70101180 0x7fbc7019d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:46112/0 (socket says 172.21.15.29:46112) 2024-03-11T19:18:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.591+0000 7fbc76917640 1 -- 172.21.15.29:0/2088025967 learned_addr learned my addr 172.21.15.29:0/2088025967 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.592+0000 7fbc76917640 1 -- 172.21.15.29:0/2088025967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc70101180 msgr2=0x7fbc7019d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.592+0000 7fbc76917640 1 --2- 172.21.15.29:0/2088025967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc70101180 0x7fbc7019d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.592+0000 7fbc76917640 1 -- 172.21.15.29:0/2088025967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc70102370 msgr2=0x7fbc7019dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.592+0000 7fbc75915640 1 --2- 172.21.15.29:0/2088025967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc70102370 0x7fbc7019dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.592+0000 7fbc76917640 1 --2- 172.21.15.29:0/2088025967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc70102370 0x7fbc7019dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.592+0000 7fbc76917640 1 -- 172.21.15.29:0/2088025967 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6c009560 con 0x7fbc70102d30 2024-03-11T19:18:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.592+0000 7fbc75915640 1 --2- 172.21.15.29:0/2088025967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc70102370 0x7fbc7019dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:18:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.592+0000 7fbc76917640 1 --2- 172.21.15.29:0/2088025967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc70102d30 0x7fbc701a4cd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbc6800eb10 tx=0x7fbc6800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.592+0000 7fbc677fe640 1 -- 172.21.15.29:0/2088025967 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc68004b20 con 0x7fbc70102d30 2024-03-11T19:18:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.592+0000 7fbc677fe640 1 -- 172.21.15.29:0/2088025967 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbc68004c80 con 0x7fbc70102d30 2024-03-11T19:18:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.592+0000 7fbc7837a640 1 -- 172.21.15.29:0/2088025967 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc701a5850 con 0x7fbc70102d30 2024-03-11T19:18:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.592+0000 7fbc677fe640 1 -- 172.21.15.29:0/2088025967 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc68010690 con 0x7fbc70102d30 2024-03-11T19:18:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.592+0000 7fbc7837a640 1 -- 172.21.15.29:0/2088025967 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc701a5d30 con 0x7fbc70102d30 2024-03-11T19:18:38.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.596+0000 7fbc7b02d740 1 Processor -- start 2024-03-11T19:18:38.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.596+0000 7fbc677fe640 1 -- 172.21.15.29:0/2088025967 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fbc68005020 con 0x7fbc70102d30 2024-03-11T19:18:38.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.597+0000 7fbc7b02d740 1 -- start start 2024-03-11T19:18:38.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.597+0000 7fbc7b02d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e6f2d0 0x5577e2e6f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.597+0000 7fbc7b02d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e6fcc0 0x5577e2e747c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.597+0000 7fbc7b02d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e74e30 0x5577e2e772c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.597+0000 7fbc7b02d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e2ddf280 con 0x5577e2e74e30 2024-03-11T19:18:38.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.597+0000 7fbc7b02d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e2ddc7b0 con 0x5577e2e6fcc0 2024-03-11T19:18:38.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.597+0000 7fbc7b02d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e2dd7f90 con 0x5577e2e6f2d0 2024-03-11T19:18:38.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.597+0000 7fbc76116640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e6fcc0 0x5577e2e747c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.597+0000 7fbc76116640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e6fcc0 0x5577e2e747c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:47610/0 (socket says 172.21.15.29:47610) 2024-03-11T19:18:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.597+0000 7fbc76116640 1 -- 172.21.15.29:0/3512290542 learned_addr learned my addr 172.21.15.29:0/3512290542 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.597+0000 7fbc75915640 1 --2- 172.21.15.29:0/3512290542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e74e30 0x5577e2e772c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.597+0000 7fbc75915640 1 --2- 172.21.15.29:0/3512290542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e6f2d0 0x5577e2e6f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.598+0000 7fbc677fe640 1 --2- 172.21.15.29:0/2088025967 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbc5c0a1c30 0x7fbc5c0a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.598+0000 7fbc75915640 1 -- 172.21.15.29:0/3512290542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e6f2d0 msgr2=0x5577e2e6f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.598+0000 7fbc75915640 1 --2- 172.21.15.29:0/3512290542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e6f2d0 0x5577e2e6f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.598+0000 7fbc75915640 1 -- 172.21.15.29:0/3512290542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e6fcc0 msgr2=0x5577e2e747c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.598+0000 7fbc75915640 1 --2- 172.21.15.29:0/3512290542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e6fcc0 0x5577e2e747c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.598+0000 7fbc75915640 1 -- 172.21.15.29:0/3512290542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5577e2d96570 con 0x5577e2e74e30 2024-03-11T19:18:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.598+0000 7fbc677fe640 1 -- 172.21.15.29:0/2088025967 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fbc68014070 con 0x7fbc70102d30 2024-03-11T19:18:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.598+0000 7fbc75915640 1 --2- 172.21.15.29:0/3512290542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e74e30 0x5577e2e772c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fbc7019e5f0 tx=0x7fbc600095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee32b7d4cebee90a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.599+0000 7fbc76917640 1 --2- 172.21.15.29:0/2088025967 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbc5c0a1c30 0x7fbc5c0a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.599+0000 7fbc76917640 1 --2- 172.21.15.29:0/2088025967 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbc5c0a1c30 0x7fbc5c0a40f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbc68096d40 tx=0x7fbc6800efd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.600+0000 7fbc64bfe640 1 -- 172.21.15.29:0/3512290542 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc60014310 con 0x5577e2e74e30 2024-03-11T19:18:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.600+0000 7fbc64bfe640 1 -- 172.21.15.29:0/3512290542 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbc6001a070 con 0x5577e2e74e30 2024-03-11T19:18:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.600+0000 7fbc64bfe640 1 -- 172.21.15.29:0/3512290542 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6000fcc0 con 0x5577e2e74e30 2024-03-11T19:18:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.601+0000 7fbc7b02d740 1 -- 172.21.15.29:0/3512290542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e74e30 msgr2=0x5577e2e772c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.601+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/3512290542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e74e30 0x5577e2e772c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fbc7019e5f0 tx=0x7fbc600095d0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.603+0000 7fbc7b02d740 1 -- 172.21.15.29:0/3512290542 shutdown_connections 2024-03-11T19:18:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.603+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/3512290542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e74e30 0x5577e2e772c0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.603+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/3512290542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e6fcc0 0x5577e2e747c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.603+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/3512290542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e6f2d0 0x5577e2e6f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.603+0000 7fbc7b02d740 1 -- 172.21.15.29:0/3512290542 >> 172.21.15.29:0/3512290542 conn(0x5577e2e6a7f0 msgr2=0x5577e2e6abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.604+0000 7fbc7b02d740 1 -- 172.21.15.29:0/3512290542 shutdown_connections 2024-03-11T19:18:38.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.604+0000 7fbc7b02d740 1 -- 172.21.15.29:0/3512290542 wait complete. 2024-03-11T19:18:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.605+0000 7fbc7b02d740 1 -- start start 2024-03-11T19:18:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.605+0000 7fbc7b02d740 1 Processor -- start 2024-03-11T19:18:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.605+0000 7fbc7b02d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e6f2d0 0x5577e2e792e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.605+0000 7fbc7b02d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e6fcc0 0x5577e2e79820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.606+0000 7fbc7b02d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e74e30 0x5577e2e80860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.606+0000 7fbc7b02d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e2aab430 con 0x5577e2e6fcc0 2024-03-11T19:18:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.606+0000 7fbc7b02d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e2a67e00 con 0x5577e2e6f2d0 2024-03-11T19:18:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.606+0000 7fbc7b02d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e2d5cfd0 con 0x5577e2e74e30 2024-03-11T19:18:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.606+0000 7fbc7b02d740 10 client.0 _ll_register_callbacks cb 0x5577e2db2fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.606+0000 7fbc75915640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e6f2d0 0x5577e2e792e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.606+0000 7fbc75915640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e6f2d0 0x5577e2e792e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:47622/0 (socket says 172.21.15.29:47622) 2024-03-11T19:18:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.606+0000 7fbc75915640 1 -- 172.21.15.29:0/1257642251 learned_addr learned my addr 172.21.15.29:0/1257642251 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.606+0000 7fbc76116640 1 --2- 172.21.15.29:0/1257642251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e6fcc0 0x5577e2e79820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.606+0000 7fbc75915640 1 --2- 172.21.15.29:0/1257642251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e74e30 0x5577e2e80860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.607+0000 7fbc75915640 1 -- 172.21.15.29:0/1257642251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e74e30 msgr2=0x5577e2e80860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.607+0000 7fbc75915640 1 --2- 172.21.15.29:0/1257642251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e74e30 0x5577e2e80860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.607+0000 7fbc75915640 1 -- 172.21.15.29:0/1257642251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e6fcc0 msgr2=0x5577e2e79820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.607+0000 7fbc75915640 1 --2- 172.21.15.29:0/1257642251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e6fcc0 0x5577e2e79820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.607+0000 7fbc75915640 1 -- 172.21.15.29:0/1257642251 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc60014550 con 0x5577e2e6f2d0 2024-03-11T19:18:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.607+0000 7fbc76116640 1 --2- 172.21.15.29:0/1257642251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e6fcc0 0x5577e2e79820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:18:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.607+0000 7fbc75915640 1 --2- 172.21.15.29:0/1257642251 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e6f2d0 0x5577e2e792e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbc600144b0 tx=0x7fbc60009ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9df1e62370b31e51 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.607+0000 7fbc59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:18:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.607+0000 7fbc59ffb640 1 -- 172.21.15.29:0/1257642251 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc60005250 con 0x5577e2e6f2d0 2024-03-11T19:18:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.607+0000 7fbc59ffb640 1 -- 172.21.15.29:0/1257642251 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fbc6000fe60 con 0x5577e2e6f2d0 2024-03-11T19:18:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.608+0000 7fbc7b02d740 1 -- 172.21.15.29:0/1257642251 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5577e2ad77e0 con 0x5577e2e6f2d0 2024-03-11T19:18:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.608+0000 7fbc59ffb640 1 -- 172.21.15.29:0/1257642251 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbc600053b0 con 0x5577e2e6f2d0 2024-03-11T19:18:38.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.609+0000 7fbc7b02d740 10 client.27044 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:38.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.609+0000 7fbc7b02d740 1 -- 172.21.15.29:0/1257642251 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5577e2dd6610 con 0x5577e2e6f2d0 2024-03-11T19:18:38.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.610+0000 7fbc59ffb640 1 -- 172.21.15.29:0/1257642251 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5577e2ad77e0 con 0x5577e2e6f2d0 2024-03-11T19:18:38.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.610+0000 7fbc59ffb640 1 -- 172.21.15.29:0/1257642251 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbc60012450 con 0x5577e2e6f2d0 2024-03-11T19:18:38.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.610+0000 7fbc7b02d740 10 client.27044 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:38.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.610+0000 7fbc7b02d740 10 client.27044 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:18:38.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.610+0000 7fbc7b02d740 20 client.27044 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:38.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.610+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/1257642251 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5577e2e852c0 0x5577e2e876b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.610+0000 7fbc7b02d740 4 client.27044 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:18:38.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.610+0000 7fbc7b02d740 1 -- 172.21.15.29:0/1257642251 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5577e2b14720 con 0x5577e2e852c0 2024-03-11T19:18:38.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.611+0000 7fbc76116640 1 --2- 172.21.15.29:0/1257642251 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5577e2e852c0 0x5577e2e876b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.612+0000 7fbc59ffb640 10 client.27044 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:38.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.612+0000 7fbc76116640 1 --2- 172.21.15.29:0/1257642251 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5577e2e852c0 0x5577e2e876b0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57c6070e34dbd653 server_cookie=fc35e9c87e3c08f in_seq=0 out_seq=0 2024-03-11T19:18:38.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.613+0000 7fbc59ffb640 1 -- 172.21.15.29:0/1257642251 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5577e2b14720 con 0x5577e2e852c0 2024-03-11T19:18:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.613+0000 7fbc59ffb640 10 client.27044 handle_command_reply: tid=0 2024-03-11T19:18:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.613+0000 7fbc7b02d740 1 client.27044 shutdown 2024-03-11T19:18:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.613+0000 7fbc7b02d740 1 -- 172.21.15.29:0/1257642251 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e6f2d0 msgr2=0x5577e2e792e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.613+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/1257642251 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e6f2d0 0x5577e2e792e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbc600144b0 tx=0x7fbc60009ba0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.613+0000 7fbc7b02d740 1 -- 172.21.15.29:0/1257642251 shutdown_connections 2024-03-11T19:18:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.614+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/1257642251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e74e30 0x5577e2e80860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.614+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/1257642251 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5577e2e852c0 0x5577e2e876b0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.614+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/1257642251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e6fcc0 0x5577e2e79820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.614+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/1257642251 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e6f2d0 0x5577e2e792e0 secure :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbc600144b0 tx=0x7fbc60009ba0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.614+0000 7fbc7b02d740 1 -- 172.21.15.29:0/1257642251 >> 172.21.15.29:0/1257642251 conn(0x5577e2e6a7f0 msgr2=0x5577e2e69a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.614+0000 7fbc59ffb640 0 client.27044 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:38.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.615+0000 7fbc7b02d740 1 -- 172.21.15.29:0/1257642251 shutdown_connections 2024-03-11T19:18:38.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.615+0000 7fbc7b02d740 1 -- 172.21.15.29:0/1257642251 wait complete. 2024-03-11T19:18:38.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.615+0000 7fbc7b02d740 20 client.27044 trim_cache size 0 max 16384 2024-03-11T19:18:38.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.644+0000 7fbc7b02d740 1 Processor -- start 2024-03-11T19:18:38.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.644+0000 7fbc7b02d740 1 -- start start 2024-03-11T19:18:38.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.644+0000 7fbc7b02d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e6fcc0 0x5577e2e70fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.644+0000 7fbc7b02d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e74e30 0x5577e2e80820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.644+0000 7fbc7b02d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e714e0 0x5577e2e78ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.644+0000 7fbc7b02d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e2b14720 con 0x5577e2e6fcc0 2024-03-11T19:18:38.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.644+0000 7fbc75915640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e6fcc0 0x5577e2e70fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.644+0000 7fbc76116640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e74e30 0x5577e2e80820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.644+0000 7fbc75915640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e6fcc0 0x5577e2e70fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51972/0 (socket says 172.21.15.29:51972) 2024-03-11T19:18:38.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.644+0000 7fbc75915640 1 -- 172.21.15.29:0/1014673040 learned_addr learned my addr 172.21.15.29:0/1014673040 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:38.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.644+0000 7fbc7b02d740 1 -- 172.21.15.29:0/1014673040 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e2ad77e0 con 0x5577e2e74e30 2024-03-11T19:18:38.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.644+0000 7fbc7b02d740 1 -- 172.21.15.29:0/1014673040 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e2e6fac0 con 0x5577e2e714e0 2024-03-11T19:18:38.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.645+0000 7fbc75915640 1 --2- 172.21.15.29:0/1014673040 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e714e0 0x5577e2e78ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.645+0000 7fbc76116640 1 -- 172.21.15.29:0/1014673040 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e714e0 msgr2=0x5577e2e78ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:38.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.645+0000 7fbc76116640 1 --2- 172.21.15.29:0/1014673040 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e714e0 0x5577e2e78ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.645+0000 7fbc76116640 1 -- 172.21.15.29:0/1014673040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e6fcc0 msgr2=0x5577e2e70fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:38.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.645+0000 7fbc76116640 1 --2- 172.21.15.29:0/1014673040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e6fcc0 0x5577e2e70fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.645+0000 7fbc76116640 1 -- 172.21.15.29:0/1014673040 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5577e2d72ac0 con 0x5577e2e74e30 2024-03-11T19:18:38.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.645+0000 7fbc76116640 1 --2- 172.21.15.29:0/1014673040 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e74e30 0x5577e2e80820 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x5577e2e70730 tx=0x7fbc6c02cd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e51332ba0a8594e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:38.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.645+0000 7fbc5bfff640 1 -- 172.21.15.29:0/1014673040 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c045040 con 0x5577e2e74e30 2024-03-11T19:18:38.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.645+0000 7fbc5bfff640 1 -- 172.21.15.29:0/1014673040 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbc6c050070 con 0x5577e2e74e30 2024-03-11T19:18:38.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.645+0000 7fbc5bfff640 1 -- 172.21.15.29:0/1014673040 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c04c910 con 0x5577e2e74e30 2024-03-11T19:18:38.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.646+0000 7fbc7b02d740 1 -- 172.21.15.29:0/1014673040 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e74e30 msgr2=0x5577e2e80820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:38.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.646+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/1014673040 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e74e30 0x5577e2e80820 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x5577e2e70730 tx=0x7fbc6c02cd10 comp rx=0 tx=0).stop 2024-03-11T19:18:38.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.646+0000 7fbc7b02d740 1 -- 172.21.15.29:0/1014673040 shutdown_connections 2024-03-11T19:18:38.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.646+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/1014673040 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e714e0 0x5577e2e78ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.646+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/1014673040 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e74e30 0x5577e2e80820 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.646+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/1014673040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e6fcc0 0x5577e2e70fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.646+0000 7fbc7b02d740 1 -- 172.21.15.29:0/1014673040 >> 172.21.15.29:0/1014673040 conn(0x5577e2e6a7f0 msgr2=0x5577e2e8c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:38.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.646+0000 7fbc7b02d740 1 -- 172.21.15.29:0/1014673040 shutdown_connections 2024-03-11T19:18:38.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.647+0000 7fbc7b02d740 1 -- 172.21.15.29:0/1014673040 wait complete. 2024-03-11T19:18:38.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.647+0000 7fbc7b02d740 1 -- start start 2024-03-11T19:18:38.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.647+0000 7fbc7b02d740 1 Processor -- start 2024-03-11T19:18:38.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.648+0000 7fbc7b02d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e6fcc0 0x5577e2e80430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.648+0000 7fbc7b02d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e714e0 0x5577e2e7ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.648+0000 7fbc7b02d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e74e30 0x5577e2e7ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.648+0000 7fbc7b02d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e2e80c20 con 0x5577e2e6fcc0 2024-03-11T19:18:38.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.648+0000 7fbc7b02d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e2e7f4c0 con 0x5577e2e714e0 2024-03-11T19:18:38.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.648+0000 7fbc7b02d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e2e7f680 con 0x5577e2e74e30 2024-03-11T19:18:38.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.648+0000 7fbc75915640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e6fcc0 0x5577e2e80430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.648+0000 7fbc75915640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e6fcc0 0x5577e2e80430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51984/0 (socket says 172.21.15.29:51984) 2024-03-11T19:18:38.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.648+0000 7fbc7b02d740 10 client.0 _ll_register_callbacks cb 0x7fbc60009990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:18:38.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.648+0000 7fbc75915640 1 -- 172.21.15.29:0/2147817148 learned_addr learned my addr 172.21.15.29:0/2147817148 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:18:38.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.648+0000 7fbc76116640 1 --2- 172.21.15.29:0/2147817148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e714e0 0x5577e2e7ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.648+0000 7fbc75915640 1 --2- 172.21.15.29:0/2147817148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e74e30 0x5577e2e7ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.648+0000 7fbc76116640 1 -- 172.21.15.29:0/2147817148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e74e30 msgr2=0x5577e2e7ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:38.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.648+0000 7fbc76116640 1 --2- 172.21.15.29:0/2147817148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e74e30 0x5577e2e7ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.648+0000 7fbc76116640 1 -- 172.21.15.29:0/2147817148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e6fcc0 msgr2=0x5577e2e80430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:18:38.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.648+0000 7fbc76116640 1 --2- 172.21.15.29:0/2147817148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e6fcc0 0x5577e2e80430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.648+0000 7fbc76116640 1 -- 172.21.15.29:0/2147817148 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc70109ec0 con 0x5577e2e714e0 2024-03-11T19:18:38.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.649+0000 7fbc76116640 1 --2- 172.21.15.29:0/2147817148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e714e0 0x5577e2e7ea40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x5577e2e745a0 tx=0x7fbc6c04ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4dc3022544a061f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:18:38.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.649+0000 7fbc59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:18:38.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.649+0000 7fbc59ffb640 1 -- 172.21.15.29:0/2147817148 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c045040 con 0x5577e2e714e0 2024-03-11T19:18:38.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.649+0000 7fbc7b02d740 1 -- 172.21.15.29:0/2147817148 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbc6001a070 con 0x5577e2e714e0 2024-03-11T19:18:38.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.649+0000 7fbc59ffb640 1 -- 172.21.15.29:0/2147817148 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fbc6c02de40 con 0x5577e2e714e0 2024-03-11T19:18:38.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.649+0000 7fbc59ffb640 1 -- 172.21.15.29:0/2147817148 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbc6c004a20 con 0x5577e2e714e0 2024-03-11T19:18:38.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.650+0000 7fbc7b02d740 10 client.27050 fetch_fsmap learned FSMap version 27 2024-03-11T19:18:38.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.650+0000 7fbc7b02d740 1 -- 172.21.15.29:0/2147817148 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5577e2d986a0 con 0x5577e2e714e0 2024-03-11T19:18:38.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.650+0000 7fbc59ffb640 1 -- 172.21.15.29:0/2147817148 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbc6001a070 con 0x5577e2e714e0 2024-03-11T19:18:38.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.650+0000 7fbc59ffb640 1 -- 172.21.15.29:0/2147817148 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbc6c05f070 con 0x5577e2e714e0 2024-03-11T19:18:38.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.650+0000 7fbc7b02d740 10 client.27050 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:18:38.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.650+0000 7fbc7b02d740 10 client.27050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:18:38.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.650+0000 7fbc7b02d740 20 client.27050 populate_metadata read hostname 'smithi029' 2024-03-11T19:18:38.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.651+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/2147817148 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5577e2edca10 0x5577e2edee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:18:38.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.651+0000 7fbc7b02d740 4 client.27050 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:18:38.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.651+0000 7fbc7b02d740 1 -- 172.21.15.29:0/2147817148 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5577e2b63890 con 0x5577e2edca10 2024-03-11T19:18:38.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.654+0000 7fbc75915640 1 --2- 172.21.15.29:0/2147817148 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5577e2edca10 0x5577e2edee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:18:38.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.654+0000 7fbc75915640 1 --2- 172.21.15.29:0/2147817148 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5577e2edca10 0x5577e2edee30 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7feee2d139209088 server_cookie=2e50d4a4402e1636 in_seq=0 out_seq=0 2024-03-11T19:18:38.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.654+0000 7fbc59ffb640 10 client.27050 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:38.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.655+0000 7fbc59ffb640 1 -- 172.21.15.29:0/2147817148 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5577e2b63890 con 0x5577e2edca10 2024-03-11T19:18:38.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.655+0000 7fbc59ffb640 10 client.27050 handle_command_reply: tid=0 2024-03-11T19:18:38.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.656+0000 7fbc7b02d740 1 client.27050 shutdown 2024-03-11T19:18:38.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.656+0000 7fbc7b02d740 1 -- 172.21.15.29:0/2147817148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e714e0 msgr2=0x5577e2e7ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:38.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.656+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/2147817148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e714e0 0x5577e2e7ea40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x5577e2e745a0 tx=0x7fbc6c04ca10 comp rx=0 tx=0).stop 2024-03-11T19:18:38.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.658+0000 7fbc7b02d740 1 -- 172.21.15.29:0/2147817148 shutdown_connections 2024-03-11T19:18:38.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.658+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/2147817148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e2e74e30 0x5577e2e7ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.658+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/2147817148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5577e2e714e0 0x5577e2e7ea40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.658+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/2147817148 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5577e2edca10 0x5577e2edee30 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.658+0000 7fbc7b02d740 1 --2- 172.21.15.29:0/2147817148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577e2e6fcc0 0x5577e2e80430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.658+0000 7fbc7b02d740 1 -- 172.21.15.29:0/2147817148 >> 172.21.15.29:0/2147817148 conn(0x5577e2e6a7f0 msgr2=0x5577e2e78ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:38.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.658+0000 7fbc59ffb640 0 client.27050 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:18:38.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.659+0000 7fbc7b02d740 1 -- 172.21.15.29:0/2147817148 shutdown_connections 2024-03-11T19:18:38.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.659+0000 7fbc7b02d740 1 -- 172.21.15.29:0/2147817148 wait complete. 2024-03-11T19:18:38.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.659+0000 7fbc7b02d740 20 client.27050 trim_cache size 0 max 16384 2024-03-11T19:18:38.663 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:18:38.663 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:18:38.663 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:18:38.664 INFO:teuthology.orchestra.run.smithi029.stdout: "3ae2879e-5eac-4a4d-abd7-c9a8e8eee06d": { 2024-03-11T19:18:38.664 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:18:38.664 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "3ae2879e-5eac-4a4d-abd7-c9a8e8eee06d", 2024-03-11T19:18:38.664 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:18:38.664 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:18:38.664 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:18:38.664 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:18:38.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.660+0000 7fbc7837a640 1 -- 172.21.15.29:0/2088025967 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbc5c0a1c30 msgr2=0x7fbc5c0a40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:38.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.660+0000 7fbc7837a640 1 --2- 172.21.15.29:0/2088025967 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbc5c0a1c30 0x7fbc5c0a40f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbc68096d40 tx=0x7fbc6800efd0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.661+0000 7fbc7837a640 1 -- 172.21.15.29:0/2088025967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc70102d30 msgr2=0x7fbc701a4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:18:38.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.661+0000 7fbc7837a640 1 --2- 172.21.15.29:0/2088025967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc70102d30 0x7fbc701a4cd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbc6800eb10 tx=0x7fbc6800c2d0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.661+0000 7fbc7837a640 1 -- 172.21.15.29:0/2088025967 shutdown_connections 2024-03-11T19:18:38.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.661+0000 7fbc7837a640 1 --2- 172.21.15.29:0/2088025967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc70102d30 0x7fbc701a4cd0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.661+0000 7fbc7837a640 1 --2- 172.21.15.29:0/2088025967 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbc5c0a1c30 0x7fbc5c0a40f0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.661+0000 7fbc7837a640 1 --2- 172.21.15.29:0/2088025967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc70102370 0x7fbc7019dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.661+0000 7fbc7837a640 1 --2- 172.21.15.29:0/2088025967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc70101180 0x7fbc7019d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:18:38.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.661+0000 7fbc7837a640 1 -- 172.21.15.29:0/2088025967 >> 172.21.15.29:0/2088025967 conn(0x7fbc700fc6f0 msgr2=0x7fbc700fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:18:38.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.661+0000 7fbc7837a640 1 -- 172.21.15.29:0/2088025967 shutdown_connections 2024-03-11T19:18:38.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:18:38.661+0000 7fbc7837a640 1 -- 172.21.15.29:0/2088025967 wait complete. 2024-03-11T19:18:38.691 INFO:tasks.cephfs.filesystem:scrub status for tag:3ae2879e-5eac-4a4d-abd7-c9a8e8eee06d - {'path': '/', 'tag': '3ae2879e-5eac-4a4d-abd7-c9a8e8eee06d', 'options': 'recursive,force'} 2024-03-11T19:18:39.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:39 smithi029 ceph-mon[32439]: pgmap v616: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 58 MiB/s wr, 1.10k op/s 2024-03-11T19:18:39.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:39 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:18:39.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:39 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:18:39.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:39 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:18:39.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:39 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:18:39.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:39 smithi190 ceph-mon[36292]: pgmap v616: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 58 MiB/s wr, 1.10k op/s 2024-03-11T19:18:39.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:39 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:18:39.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:39 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:18:39.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:39 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:18:39.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:39 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:18:39.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:39 smithi170 ceph-mon[35690]: pgmap v616: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 58 MiB/s wr, 1.10k op/s 2024-03-11T19:18:39.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:39 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:18:39.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:39 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:18:39.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:39 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:18:39.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:39 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:18:41.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:41 smithi029 ceph-mon[32439]: pgmap v617: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 63 MiB/s wr, 1.20k op/s 2024-03-11T19:18:41.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:41 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:18:41.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:41 smithi190 ceph-mon[36292]: pgmap v617: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 63 MiB/s wr, 1.20k op/s 2024-03-11T19:18:41.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:41 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:18:41.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:41 smithi170 ceph-mon[35690]: pgmap v617: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 63 MiB/s wr, 1.20k op/s 2024-03-11T19:18:41.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:41 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:18:42.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:42 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:18:42.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:42 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:18:42.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:42 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:18:43.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:43 smithi029 ceph-mon[32439]: pgmap v618: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 62 MiB/s wr, 1.19k op/s 2024-03-11T19:18:43.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:18:43.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:18:43.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:18:43.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:43 smithi190 ceph-mon[36292]: pgmap v618: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 62 MiB/s wr, 1.19k op/s 2024-03-11T19:18:43.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:43 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:18:43.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:43 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:18:43.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:43 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:18:43.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:43 smithi170 ceph-mon[35690]: pgmap v618: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 62 MiB/s wr, 1.19k op/s 2024-03-11T19:18:43.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:43 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:18:43.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:43 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:18:43.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:43 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:18:45.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:45 smithi029 ceph-mon[32439]: pgmap v619: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 58 MiB/s wr, 1.12k op/s 2024-03-11T19:18:45.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:45 smithi190 ceph-mon[36292]: pgmap v619: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 58 MiB/s wr, 1.12k op/s 2024-03-11T19:18:45.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:45 smithi170 ceph-mon[35690]: pgmap v619: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 58 MiB/s wr, 1.12k op/s 2024-03-11T19:18:47.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:47 smithi029 ceph-mon[32439]: pgmap v620: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 63 MiB/s wr, 1.23k op/s 2024-03-11T19:18:47.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:47 smithi190 ceph-mon[36292]: pgmap v620: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 63 MiB/s wr, 1.23k op/s 2024-03-11T19:18:47.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:47 smithi170 ceph-mon[35690]: pgmap v620: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 63 MiB/s wr, 1.23k op/s 2024-03-11T19:18:49.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:49 smithi029 ceph-mon[32439]: pgmap v621: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 56 MiB/s wr, 1.09k op/s 2024-03-11T19:18:49.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:49 smithi190 ceph-mon[36292]: pgmap v621: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 56 MiB/s wr, 1.09k op/s 2024-03-11T19:18:49.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:49 smithi170 ceph-mon[35690]: pgmap v621: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 56 MiB/s wr, 1.09k op/s 2024-03-11T19:18:51.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:51 smithi029 ceph-mon[32439]: pgmap v622: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 62 MiB/s wr, 1.22k op/s 2024-03-11T19:18:51.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:51 smithi190 ceph-mon[36292]: pgmap v622: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 62 MiB/s wr, 1.22k op/s 2024-03-11T19:18:51.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:51 smithi170 ceph-mon[35690]: pgmap v622: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 62 MiB/s wr, 1.22k op/s 2024-03-11T19:18:53.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:53 smithi029 ceph-mon[32439]: pgmap v623: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 62 MiB/s wr, 1.22k op/s 2024-03-11T19:18:53.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:53 smithi190 ceph-mon[36292]: pgmap v623: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 62 MiB/s wr, 1.22k op/s 2024-03-11T19:18:53.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:53 smithi170 ceph-mon[35690]: pgmap v623: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 62 MiB/s wr, 1.22k op/s 2024-03-11T19:18:55.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:55 smithi029 ceph-mon[32439]: pgmap v624: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 59 MiB/s wr, 1.16k op/s 2024-03-11T19:18:55.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:55 smithi190 ceph-mon[36292]: pgmap v624: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 59 MiB/s wr, 1.16k op/s 2024-03-11T19:18:55.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:55 smithi170 ceph-mon[35690]: pgmap v624: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 59 MiB/s wr, 1.16k op/s 2024-03-11T19:18:57.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:57 smithi190 ceph-mon[36292]: pgmap v625: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 64 MiB/s wr, 1.27k op/s 2024-03-11T19:18:57.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:57 smithi170 ceph-mon[35690]: pgmap v625: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 64 MiB/s wr, 1.27k op/s 2024-03-11T19:18:57.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:57 smithi029 ceph-mon[32439]: pgmap v625: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 64 MiB/s wr, 1.27k op/s 2024-03-11T19:18:59.708 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:18:59 smithi029 ceph-mon[32439]: pgmap v626: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 57 MiB/s wr, 1.12k op/s 2024-03-11T19:18:59.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:18:59 smithi190 ceph-mon[36292]: pgmap v626: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 57 MiB/s wr, 1.12k op/s 2024-03-11T19:18:59.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:18:59 smithi170 ceph-mon[35690]: pgmap v626: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 57 MiB/s wr, 1.12k op/s 2024-03-11T19:19:01.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:01 smithi190 ceph-mon[36292]: pgmap v627: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 63 MiB/s wr, 1.23k op/s 2024-03-11T19:19:01.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:01 smithi170 ceph-mon[35690]: pgmap v627: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 63 MiB/s wr, 1.23k op/s 2024-03-11T19:19:01.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:01 smithi029 ceph-mon[32439]: pgmap v627: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 63 MiB/s wr, 1.23k op/s 2024-03-11T19:19:03.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:03 smithi190 ceph-mon[36292]: pgmap v628: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 62 MiB/s wr, 1.22k op/s 2024-03-11T19:19:03.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:03 smithi170 ceph-mon[35690]: pgmap v628: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 62 MiB/s wr, 1.22k op/s 2024-03-11T19:19:03.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:03 smithi029 ceph-mon[32439]: pgmap v628: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 62 MiB/s wr, 1.22k op/s 2024-03-11T19:19:05.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:05 smithi190 ceph-mon[36292]: pgmap v629: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 59 MiB/s wr, 1.15k op/s 2024-03-11T19:19:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:05 smithi170 ceph-mon[35690]: pgmap v629: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 59 MiB/s wr, 1.15k op/s 2024-03-11T19:19:05.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:05 smithi029 ceph-mon[32439]: pgmap v629: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 59 MiB/s wr, 1.15k op/s 2024-03-11T19:19:07.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:07 smithi190 ceph-mon[36292]: pgmap v630: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 63 MiB/s wr, 1.25k op/s 2024-03-11T19:19:07.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:07 smithi170 ceph-mon[35690]: pgmap v630: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 63 MiB/s wr, 1.25k op/s 2024-03-11T19:19:07.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:07 smithi029 ceph-mon[32439]: pgmap v630: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 63 MiB/s wr, 1.25k op/s 2024-03-11T19:19:08.692 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:19:08.692 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:19:08.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.878+0000 7f40c8577640 1 -- 172.21.15.29:0/2744919950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40c0102360 msgr2=0x7f40c01047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:08.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.878+0000 7f40c8577640 1 --2- 172.21.15.29:0/2744919950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40c0102360 0x7f40c01047d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f40b0009860 tx=0x7f40b002bb70 comp rx=0 tx=0).stop 2024-03-11T19:19:08.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.878+0000 7f40c8577640 1 -- 172.21.15.29:0/2744919950 shutdown_connections 2024-03-11T19:19:08.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.878+0000 7f40c8577640 1 --2- 172.21.15.29:0/2744919950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c0104d10 0x7f40c0105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.878+0000 7f40c8577640 1 --2- 172.21.15.29:0/2744919950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40c0102360 0x7f40c01047d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.878+0000 7f40c8577640 1 --2- 172.21.15.29:0/2744919950 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40c00ff9f0 0x7f40c0101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.878+0000 7f40c8577640 1 -- 172.21.15.29:0/2744919950 >> 172.21.15.29:0/2744919950 conn(0x7f40c00fb5f0 msgr2=0x7f40c00fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:08.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.879+0000 7f40c8577640 1 -- 172.21.15.29:0/2744919950 shutdown_connections 2024-03-11T19:19:08.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.879+0000 7f40c8577640 1 -- 172.21.15.29:0/2744919950 wait complete. 2024-03-11T19:19:08.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.880+0000 7f40c8577640 1 Processor -- start 2024-03-11T19:19:08.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.880+0000 7f40c8577640 1 -- start start 2024-03-11T19:19:08.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.880+0000 7f40c8577640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c00ff9f0 0x7f40c006fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:08.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.880+0000 7f40c8577640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40c0102360 0x7f40c006ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:08.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.880+0000 7f40c8577640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40c0104d10 0x7f40c01ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:08.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.880+0000 7f40c5b12640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40c0102360 0x7f40c006ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:08.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.880+0000 7f40c5b12640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40c0102360 0x7f40c006ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:47832/0 (socket says 172.21.15.29:47832) 2024-03-11T19:19:08.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.880+0000 7f40c8577640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c0072ff0 con 0x7f40c00ff9f0 2024-03-11T19:19:08.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.881+0000 7f40c8577640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c0073160 con 0x7f40c0104d10 2024-03-11T19:19:08.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.881+0000 7f40c8577640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c01addc0 con 0x7f40c0102360 2024-03-11T19:19:08.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.881+0000 7f40c5b12640 1 -- 172.21.15.29:0/3905435361 learned_addr learned my addr 172.21.15.29:0/3905435361 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:08.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.881+0000 7f40c6b14640 1 --2- 172.21.15.29:0/3905435361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40c0104d10 0x7f40c01ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:08.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.881+0000 7f40c5b12640 1 -- 172.21.15.29:0/3905435361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40c0104d10 msgr2=0x7f40c01ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:08.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.881+0000 7f40c5b12640 1 --2- 172.21.15.29:0/3905435361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40c0104d10 0x7f40c01ad880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.881+0000 7f40c5b12640 1 -- 172.21.15.29:0/3905435361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c00ff9f0 msgr2=0x7f40c006fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:08.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.881+0000 7f40c5b12640 1 --2- 172.21.15.29:0/3905435361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c00ff9f0 0x7f40c006fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.881+0000 7f40c5b12640 1 -- 172.21.15.29:0/3905435361 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b0009490 con 0x7f40c0102360 2024-03-11T19:19:08.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.882+0000 7f40c5b12640 1 --2- 172.21.15.29:0/3905435361 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40c0102360 0x7f40c006ffa0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f40b0005940 tx=0x7f40b0032470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:08.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.882+0000 7f40b77fe640 1 -- 172.21.15.29:0/3905435361 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b002c510 con 0x7f40c0102360 2024-03-11T19:19:08.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.882+0000 7f40c8577640 1 -- 172.21.15.29:0/3905435361 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c01adfc0 con 0x7f40c0102360 2024-03-11T19:19:08.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.882+0000 7f40c8577640 1 -- 172.21.15.29:0/3905435361 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c01ae440 con 0x7f40c0102360 2024-03-11T19:19:08.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.882+0000 7f40b77fe640 1 -- 172.21.15.29:0/3905435361 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f40b002ce90 con 0x7f40c0102360 2024-03-11T19:19:08.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.882+0000 7f40b77fe640 1 -- 172.21.15.29:0/3905435361 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0003d00 con 0x7f40c0102360 2024-03-11T19:19:08.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.891+0000 7f40b77fe640 1 -- 172.21.15.29:0/3905435361 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f40b003b040 con 0x7f40c0102360 2024-03-11T19:19:08.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.892+0000 7f40b77fe640 1 --2- 172.21.15.29:0/3905435361 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f40ac0a1b10 0x7f40ac0a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:08.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.893+0000 7f40b77fe640 1 -- 172.21.15.29:0/3905435361 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f40b00eef10 con 0x7f40c0102360 2024-03-11T19:19:08.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.893+0000 7f40c6313640 1 --2- 172.21.15.29:0/3905435361 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f40ac0a1b10 0x7f40ac0a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:08.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.894+0000 7f40c6313640 1 --2- 172.21.15.29:0/3905435361 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f40ac0a1b10 0x7f40ac0a3fd0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f40bc0058c0 tx=0x7f40bc005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:08.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.896+0000 7f40cb229740 1 Processor -- start 2024-03-11T19:19:08.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.897+0000 7f40cb229740 1 -- start start 2024-03-11T19:19:08.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.897+0000 7f40cb229740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac02d0 0x55c8b4ac06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:08.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.897+0000 7f40cb229740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac0cc0 0x55c8b4ac57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:08.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.897+0000 7f40cb229740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac5e30 0x55c8b4ac82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:08.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.897+0000 7f40cb229740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b4a30280 con 0x55c8b4ac0cc0 2024-03-11T19:19:08.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.897+0000 7f40cb229740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b4a2d7b0 con 0x55c8b4ac5e30 2024-03-11T19:19:08.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.897+0000 7f40cb229740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b4a28f90 con 0x55c8b4ac02d0 2024-03-11T19:19:08.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.897+0000 7f40c6b14640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac5e30 0x55c8b4ac82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:08.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.897+0000 7f40c6b14640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac5e30 0x55c8b4ac82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55740/0 (socket says 172.21.15.29:55740) 2024-03-11T19:19:08.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.897+0000 7f40c6b14640 1 -- 172.21.15.29:0/3345427403 learned_addr learned my addr 172.21.15.29:0/3345427403 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:08.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.898+0000 7f40c6313640 1 --2- 172.21.15.29:0/3345427403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac02d0 0x55c8b4ac06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:08.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.898+0000 7f40c6313640 1 -- 172.21.15.29:0/3345427403 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac5e30 msgr2=0x55c8b4ac82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:08.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.898+0000 7f40c6313640 1 --2- 172.21.15.29:0/3345427403 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac5e30 0x55c8b4ac82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.898+0000 7f40c6313640 1 -- 172.21.15.29:0/3345427403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac0cc0 msgr2=0x55c8b4ac57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:08.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.898+0000 7f40c6313640 1 --2- 172.21.15.29:0/3345427403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac0cc0 0x55c8b4ac57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.898+0000 7f40c6313640 1 -- 172.21.15.29:0/3345427403 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8b49e7570 con 0x55c8b4ac02d0 2024-03-11T19:19:08.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.898+0000 7f40c6313640 1 --2- 172.21.15.29:0/3345427403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac02d0 0x55c8b4ac06f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00ed60 tx=0x7f40bc011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96666756e07dc9f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:08.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.899+0000 7f40b4bfe640 1 -- 172.21.15.29:0/3345427403 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc005020 con 0x55c8b4ac02d0 2024-03-11T19:19:08.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.899+0000 7f40b4bfe640 1 -- 172.21.15.29:0/3345427403 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f40bc005630 con 0x55c8b4ac02d0 2024-03-11T19:19:08.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.899+0000 7f40b4bfe640 1 -- 172.21.15.29:0/3345427403 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc004ea0 con 0x55c8b4ac02d0 2024-03-11T19:19:08.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.900+0000 7f40cb229740 1 -- 172.21.15.29:0/3345427403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac02d0 msgr2=0x55c8b4ac06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:08.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.900+0000 7f40cb229740 1 --2- 172.21.15.29:0/3345427403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac02d0 0x55c8b4ac06f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00ed60 tx=0x7f40bc011000 comp rx=0 tx=0).stop 2024-03-11T19:19:08.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.900+0000 7f40cb229740 1 -- 172.21.15.29:0/3345427403 shutdown_connections 2024-03-11T19:19:08.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.900+0000 7f40cb229740 1 --2- 172.21.15.29:0/3345427403 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac5e30 0x55c8b4ac82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.900+0000 7f40cb229740 1 --2- 172.21.15.29:0/3345427403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac0cc0 0x55c8b4ac57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.900+0000 7f40cb229740 1 --2- 172.21.15.29:0/3345427403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac02d0 0x55c8b4ac06f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.900+0000 7f40cb229740 1 -- 172.21.15.29:0/3345427403 >> 172.21.15.29:0/3345427403 conn(0x55c8b4abb7f0 msgr2=0x55c8b4abbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:08.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.900+0000 7f40cb229740 1 -- 172.21.15.29:0/3345427403 shutdown_connections 2024-03-11T19:19:08.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.900+0000 7f40cb229740 1 -- 172.21.15.29:0/3345427403 wait complete. 2024-03-11T19:19:08.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.901+0000 7f40cb229740 1 -- start start 2024-03-11T19:19:08.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.901+0000 7f40cb229740 1 Processor -- start 2024-03-11T19:19:08.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.901+0000 7f40cb229740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac02d0 0x55c8b4aca370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:08.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.901+0000 7f40cb229740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac0cc0 0x55c8b4aca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:08.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.901+0000 7f40cb229740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac5e30 0x55c8b4ad18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.901+0000 7f40cb229740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b46fc430 con 0x55c8b4ac0cc0 2024-03-11T19:19:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.901+0000 7f40cb229740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b46b8e00 con 0x55c8b4ac02d0 2024-03-11T19:19:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.901+0000 7f40cb229740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b49adfd0 con 0x55c8b4ac5e30 2024-03-11T19:19:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.901+0000 7f40cb229740 10 client.0 _ll_register_callbacks cb 0x55c8b4a03fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.902+0000 7f40c6b14640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac5e30 0x55c8b4ad18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.902+0000 7f40c6b14640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac5e30 0x55c8b4ad18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:47838/0 (socket says 172.21.15.29:47838) 2024-03-11T19:19:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.902+0000 7f40c6b14640 1 -- 172.21.15.29:0/1361983778 learned_addr learned my addr 172.21.15.29:0/1361983778 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.902+0000 7f40c6313640 1 --2- 172.21.15.29:0/1361983778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac02d0 0x55c8b4aca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.902+0000 7f40c5b12640 1 --2- 172.21.15.29:0/1361983778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac0cc0 0x55c8b4aca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.902+0000 7f40c6b14640 1 -- 172.21.15.29:0/1361983778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac02d0 msgr2=0x55c8b4aca370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.902+0000 7f40c6b14640 1 --2- 172.21.15.29:0/1361983778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac02d0 0x55c8b4aca370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.902+0000 7f40c6b14640 1 -- 172.21.15.29:0/1361983778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac0cc0 msgr2=0x55c8b4aca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.902+0000 7f40c6b14640 1 --2- 172.21.15.29:0/1361983778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac0cc0 0x55c8b4aca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.902+0000 7f40c6b14640 1 -- 172.21.15.29:0/1361983778 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40bc00ea40 con 0x55c8b4ac5e30 2024-03-11T19:19:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.903+0000 7f40c6b14640 1 --2- 172.21.15.29:0/1361983778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac5e30 0x55c8b4ad18f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f40b8011d10 tx=0x7f40b800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e3527348e6de55f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.903+0000 7f40a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:19:08.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.903+0000 7f40a9ffb640 1 -- 172.21.15.29:0/1361983778 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b8016070 con 0x55c8b4ac5e30 2024-03-11T19:19:08.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.903+0000 7f40cb229740 1 -- 172.21.15.29:0/1361983778 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c8b47287e0 con 0x55c8b4ac5e30 2024-03-11T19:19:08.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.903+0000 7f40a9ffb640 1 -- 172.21.15.29:0/1361983778 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f40b801a3f0 con 0x55c8b4ac5e30 2024-03-11T19:19:08.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.904+0000 7f40a9ffb640 1 -- 172.21.15.29:0/1361983778 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40b801a5b0 con 0x55c8b4ac5e30 2024-03-11T19:19:08.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.905+0000 7f40cb229740 10 client.27103 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:08.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.905+0000 7f40cb229740 1 -- 172.21.15.29:0/1361983778 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55c8b4a27610 con 0x55c8b4ac5e30 2024-03-11T19:19:08.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.906+0000 7f40a9ffb640 1 -- 172.21.15.29:0/1361983778 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55c8b47287e0 con 0x55c8b4ac5e30 2024-03-11T19:19:08.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.907+0000 7f40a9ffb640 1 -- 172.21.15.29:0/1361983778 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f40b80245d0 con 0x55c8b4ac5e30 2024-03-11T19:19:08.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.907+0000 7f40cb229740 10 client.27103 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:08.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.907+0000 7f40cb229740 10 client.27103 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:19:08.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.907+0000 7f40cb229740 20 client.27103 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:08.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.907+0000 7f40cb229740 1 --2- 172.21.15.29:0/1361983778 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c8b4ad66c0 0x55c8b4ad8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:08.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.907+0000 7f40cb229740 4 client.27103 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:19:08.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.907+0000 7f40cb229740 1 -- 172.21.15.29:0/1361983778 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c8b4765720 con 0x55c8b4ad66c0 2024-03-11T19:19:08.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.907+0000 7f40c6313640 1 --2- 172.21.15.29:0/1361983778 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c8b4ad66c0 0x55c8b4ad8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:08.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.909+0000 7f40c6313640 1 --2- 172.21.15.29:0/1361983778 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c8b4ad66c0 0x55c8b4ad8ae0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=944b26aa5284d071 server_cookie=165531738ceebe72 in_seq=0 out_seq=0 2024-03-11T19:19:08.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.909+0000 7f40a9ffb640 10 client.27103 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:08.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.910+0000 7f40a9ffb640 1 -- 172.21.15.29:0/1361983778 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c8b4765720 con 0x55c8b4ad66c0 2024-03-11T19:19:08.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.910+0000 7f40a9ffb640 10 client.27103 handle_command_reply: tid=0 2024-03-11T19:19:08.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.911+0000 7f40cb229740 1 client.27103 shutdown 2024-03-11T19:19:08.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.913+0000 7f40cb229740 1 -- 172.21.15.29:0/1361983778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac5e30 msgr2=0x55c8b4ad18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:08.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.913+0000 7f40cb229740 1 --2- 172.21.15.29:0/1361983778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac5e30 0x55c8b4ad18f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f40b8011d10 tx=0x7f40b800f790 comp rx=0 tx=0).stop 2024-03-11T19:19:08.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.913+0000 7f40cb229740 1 -- 172.21.15.29:0/1361983778 shutdown_connections 2024-03-11T19:19:08.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.913+0000 7f40cb229740 1 --2- 172.21.15.29:0/1361983778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac5e30 0x55c8b4ad18f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.913+0000 7f40cb229740 1 --2- 172.21.15.29:0/1361983778 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c8b4ad66c0 0x55c8b4ad8ae0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.913+0000 7f40cb229740 1 --2- 172.21.15.29:0/1361983778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac0cc0 0x55c8b4aca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.913+0000 7f40cb229740 1 --2- 172.21.15.29:0/1361983778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac02d0 0x55c8b4aca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.913+0000 7f40cb229740 1 -- 172.21.15.29:0/1361983778 >> 172.21.15.29:0/1361983778 conn(0x55c8b4abb7f0 msgr2=0x55c8b4abaa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:08.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.913+0000 7f40a9ffb640 0 client.27103 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:08.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.914+0000 7f40cb229740 1 -- 172.21.15.29:0/1361983778 shutdown_connections 2024-03-11T19:19:08.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.914+0000 7f40cb229740 1 -- 172.21.15.29:0/1361983778 wait complete. 2024-03-11T19:19:08.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.914+0000 7f40cb229740 20 client.27103 trim_cache size 0 max 16384 2024-03-11T19:19:08.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.943+0000 7f40cb229740 1 Processor -- start 2024-03-11T19:19:08.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.943+0000 7f40cb229740 1 -- start start 2024-03-11T19:19:08.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.944+0000 7f40cb229740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac02d0 0x55c8b4aca370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:08.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.944+0000 7f40cb229740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac0cc0 0x55c8b4aca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:08.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.944+0000 7f40cb229740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac5e30 0x55c8b4ad1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:08.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.944+0000 7f40cb229740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b4abc030 con 0x55c8b4ac5e30 2024-03-11T19:19:08.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.944+0000 7f40cb229740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b4aca1b0 con 0x55c8b4ac02d0 2024-03-11T19:19:08.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.944+0000 7f40cb229740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b4765720 con 0x55c8b4ac0cc0 2024-03-11T19:19:08.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.944+0000 7f40c6313640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac02d0 0x55c8b4aca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:08.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.944+0000 7f40c6313640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac02d0 0x55c8b4aca370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55764/0 (socket says 172.21.15.29:55764) 2024-03-11T19:19:08.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.944+0000 7f40c6313640 1 -- 172.21.15.29:0/3387074388 learned_addr learned my addr 172.21.15.29:0/3387074388 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:08.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.944+0000 7f40c6b14640 1 --2- 172.21.15.29:0/3387074388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac5e30 0x55c8b4ad1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:08.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.945+0000 7f40c6313640 1 -- 172.21.15.29:0/3387074388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac0cc0 msgr2=0x55c8b4aca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:08.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.945+0000 7f40c5b12640 1 --2- 172.21.15.29:0/3387074388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac0cc0 0x55c8b4aca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:08.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.945+0000 7f40c6313640 1 --2- 172.21.15.29:0/3387074388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac0cc0 0x55c8b4aca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.945+0000 7f40c6313640 1 -- 172.21.15.29:0/3387074388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac5e30 msgr2=0x55c8b4ad1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:08.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.945+0000 7f40c6313640 1 --2- 172.21.15.29:0/3387074388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac5e30 0x55c8b4ad1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.945+0000 7f40c6313640 1 -- 172.21.15.29:0/3387074388 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8b49c3ac0 con 0x55c8b4ac02d0 2024-03-11T19:19:08.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.945+0000 7f40c5b12640 1 --2- 172.21.15.29:0/3387074388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac0cc0 0x55c8b4aca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:19:08.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.945+0000 7f40c6313640 1 --2- 172.21.15.29:0/3387074388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac02d0 0x55c8b4aca370 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f40bc013040 tx=0x7f40bc005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f93e56a0f25d3f56 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:08.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.945+0000 7f40abfff640 1 -- 172.21.15.29:0/3387074388 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc010e40 con 0x55c8b4ac02d0 2024-03-11T19:19:08.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.946+0000 7f40abfff640 1 -- 172.21.15.29:0/3387074388 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f40bc029040 con 0x55c8b4ac02d0 2024-03-11T19:19:08.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.947+0000 7f40cb229740 1 -- 172.21.15.29:0/3387074388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac02d0 msgr2=0x55c8b4aca370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:08.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.947+0000 7f40cb229740 1 --2- 172.21.15.29:0/3387074388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac02d0 0x55c8b4aca370 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f40bc013040 tx=0x7f40bc005fd0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.947+0000 7f40cb229740 1 -- 172.21.15.29:0/3387074388 shutdown_connections 2024-03-11T19:19:08.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.947+0000 7f40cb229740 1 --2- 172.21.15.29:0/3387074388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac5e30 0x55c8b4ad1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.947+0000 7f40cb229740 1 --2- 172.21.15.29:0/3387074388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac0cc0 0x55c8b4aca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.947+0000 7f40cb229740 1 --2- 172.21.15.29:0/3387074388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac02d0 0x55c8b4aca370 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.947+0000 7f40cb229740 1 -- 172.21.15.29:0/3387074388 >> 172.21.15.29:0/3387074388 conn(0x55c8b4abb7f0 msgr2=0x55c8b4add440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:08.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.947+0000 7f40cb229740 1 -- 172.21.15.29:0/3387074388 shutdown_connections 2024-03-11T19:19:08.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.947+0000 7f40cb229740 1 -- 172.21.15.29:0/3387074388 wait complete. 2024-03-11T19:19:08.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.948+0000 7f40cb229740 1 -- start start 2024-03-11T19:19:08.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.948+0000 7f40cb229740 1 Processor -- start 2024-03-11T19:19:08.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.948+0000 7f40cb229740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac02d0 0x55c8b4ad1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:08.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.948+0000 7f40cb229740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac0cc0 0x55c8b4acfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:08.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.948+0000 7f40cb229740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac5e30 0x55c8b4ad02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:08.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.948+0000 7f40cb229740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b4ad08c0 con 0x55c8b4ac02d0 2024-03-11T19:19:08.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.948+0000 7f40cb229740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b4ad0a30 con 0x55c8b4ac5e30 2024-03-11T19:19:08.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.948+0000 7f40cb229740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b4ad0bf0 con 0x55c8b4ac0cc0 2024-03-11T19:19:08.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.948+0000 7f40cb229740 10 client.0 _ll_register_callbacks cb 0x7f40bc014050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:08.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.949+0000 7f40c6313640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac02d0 0x55c8b4ad1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:08.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.949+0000 7f40c6313640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac02d0 0x55c8b4ad1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53546/0 (socket says 172.21.15.29:53546) 2024-03-11T19:19:08.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.949+0000 7f40c6313640 1 -- 172.21.15.29:0/4244434910 learned_addr learned my addr 172.21.15.29:0/4244434910 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:08.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.949+0000 7f40c5b12640 1 --2- 172.21.15.29:0/4244434910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac0cc0 0x55c8b4acfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:08.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.949+0000 7f40c6313640 1 -- 172.21.15.29:0/4244434910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac0cc0 msgr2=0x55c8b4acfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:08.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.949+0000 7f40c6313640 1 --2- 172.21.15.29:0/4244434910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac0cc0 0x55c8b4acfda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.949+0000 7f40c6313640 1 -- 172.21.15.29:0/4244434910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac5e30 msgr2=0x55c8b4ad02e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:19:08.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.949+0000 7f40c6313640 1 --2- 172.21.15.29:0/4244434910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac5e30 0x55c8b4ad02e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.949+0000 7f40c6313640 1 -- 172.21.15.29:0/4244434910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c8b49e7570 con 0x55c8b4ac02d0 2024-03-11T19:19:08.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.950+0000 7f40c6313640 1 --2- 172.21.15.29:0/4244434910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac02d0 0x55c8b4ad1790 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f40bc0057c0 tx=0x7f40bc0090d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23bfee3d8c65c34 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:08.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.951+0000 7f40a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:19:08.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.953+0000 7f40a9ffb640 1 -- 172.21.15.29:0/4244434910 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc034350 con 0x55c8b4ac02d0 2024-03-11T19:19:08.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.953+0000 7f40a9ffb640 1 -- 172.21.15.29:0/4244434910 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f40bc0344b0 con 0x55c8b4ac02d0 2024-03-11T19:19:08.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.953+0000 7f40cb229740 1 -- 172.21.15.29:0/4244434910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f40bc005630 con 0x55c8b4ac02d0 2024-03-11T19:19:08.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.954+0000 7f40a9ffb640 1 -- 172.21.15.29:0/4244434910 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40bc034670 con 0x55c8b4ac02d0 2024-03-11T19:19:08.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.954+0000 7f40cb229740 10 client.17568 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:08.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.954+0000 7f40cb229740 1 -- 172.21.15.29:0/4244434910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55c8b49e96a0 con 0x55c8b4ac02d0 2024-03-11T19:19:08.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.955+0000 7f40a9ffb640 1 -- 172.21.15.29:0/4244434910 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f40bc005630 con 0x55c8b4ac02d0 2024-03-11T19:19:08.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.956+0000 7f40a9ffb640 1 -- 172.21.15.29:0/4244434910 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f40bc02e040 con 0x55c8b4ac02d0 2024-03-11T19:19:08.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.956+0000 7f40cb229740 10 client.17568 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:08.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.956+0000 7f40cb229740 10 client.17568 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:19:08.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.956+0000 7f40cb229740 20 client.17568 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:08.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.956+0000 7f40cb229740 1 --2- 172.21.15.29:0/4244434910 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c8b4b3f030 0x55c8b4b41450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:08.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.956+0000 7f40cb229740 4 client.17568 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:19:08.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.956+0000 7f40cb229740 1 -- 172.21.15.29:0/4244434910 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c8b47b4890 con 0x55c8b4b3f030 2024-03-11T19:19:08.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.956+0000 7f40c5b12640 1 --2- 172.21.15.29:0/4244434910 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c8b4b3f030 0x55c8b4b41450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:08.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.958+0000 7f40c5b12640 1 --2- 172.21.15.29:0/4244434910 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c8b4b3f030 0x55c8b4b41450 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69f7c89bed498609 server_cookie=d63ccc598a284a9d in_seq=0 out_seq=0 2024-03-11T19:19:08.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.958+0000 7f40a9ffb640 10 client.17568 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:08.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.958+0000 7f40a9ffb640 1 -- 172.21.15.29:0/4244434910 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55c8b47b4890 con 0x55c8b4b3f030 2024-03-11T19:19:08.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.958+0000 7f40a9ffb640 10 client.17568 handle_command_reply: tid=0 2024-03-11T19:19:08.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.958+0000 7f40cb229740 1 client.17568 shutdown 2024-03-11T19:19:08.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.960+0000 7f40cb229740 1 -- 172.21.15.29:0/4244434910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac02d0 msgr2=0x55c8b4ad1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:08.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.960+0000 7f40cb229740 1 --2- 172.21.15.29:0/4244434910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac02d0 0x55c8b4ad1790 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f40bc0057c0 tx=0x7f40bc0090d0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.960+0000 7f40cb229740 1 -- 172.21.15.29:0/4244434910 shutdown_connections 2024-03-11T19:19:08.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.960+0000 7f40cb229740 1 --2- 172.21.15.29:0/4244434910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8b4ac5e30 0x55c8b4ad02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.960+0000 7f40cb229740 1 --2- 172.21.15.29:0/4244434910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c8b4ac0cc0 0x55c8b4acfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.960+0000 7f40cb229740 1 --2- 172.21.15.29:0/4244434910 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55c8b4b3f030 0x55c8b4b41450 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.960+0000 7f40cb229740 1 --2- 172.21.15.29:0/4244434910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c8b4ac02d0 0x55c8b4ad1790 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.960+0000 7f40cb229740 1 -- 172.21.15.29:0/4244434910 >> 172.21.15.29:0/4244434910 conn(0x55c8b4abb7f0 msgr2=0x55c8b4aca370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:08.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.961+0000 7f40a9ffb640 0 client.17568 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:08.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.961+0000 7f40cb229740 1 -- 172.21.15.29:0/4244434910 shutdown_connections 2024-03-11T19:19:08.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.961+0000 7f40cb229740 1 -- 172.21.15.29:0/4244434910 wait complete. 2024-03-11T19:19:08.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.961+0000 7f40cb229740 20 client.17568 trim_cache size 0 max 16384 2024-03-11T19:19:08.965 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:19:08.965 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:19:08.966 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:19:08.966 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:19:08.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.963+0000 7f40c8577640 1 -- 172.21.15.29:0/3905435361 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f40ac0a1b10 msgr2=0x7f40ac0a3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:08.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.963+0000 7f40c8577640 1 --2- 172.21.15.29:0/3905435361 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f40ac0a1b10 0x7f40ac0a3fd0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f40bc0058c0 tx=0x7f40bc005850 comp rx=0 tx=0).stop 2024-03-11T19:19:08.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.963+0000 7f40c8577640 1 -- 172.21.15.29:0/3905435361 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40c0102360 msgr2=0x7f40c006ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:08.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.963+0000 7f40c8577640 1 --2- 172.21.15.29:0/3905435361 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40c0102360 0x7f40c006ffa0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f40b0005940 tx=0x7f40b0032470 comp rx=0 tx=0).stop 2024-03-11T19:19:08.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.963+0000 7f40c8577640 1 -- 172.21.15.29:0/3905435361 shutdown_connections 2024-03-11T19:19:08.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.963+0000 7f40c8577640 1 --2- 172.21.15.29:0/3905435361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40c0104d10 0x7f40c01ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.963+0000 7f40c8577640 1 --2- 172.21.15.29:0/3905435361 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40c0102360 0x7f40c006ffa0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.963+0000 7f40c8577640 1 --2- 172.21.15.29:0/3905435361 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f40ac0a1b10 0x7f40ac0a3fd0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.963+0000 7f40c8577640 1 --2- 172.21.15.29:0/3905435361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c00ff9f0 0x7f40c006fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:08.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.963+0000 7f40c8577640 1 -- 172.21.15.29:0/3905435361 >> 172.21.15.29:0/3905435361 conn(0x7f40c00fb5f0 msgr2=0x7f40c01002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:08.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.963+0000 7f40c8577640 1 -- 172.21.15.29:0/3905435361 shutdown_connections 2024-03-11T19:19:08.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:08.964+0000 7f40c8577640 1 -- 172.21.15.29:0/3905435361 wait complete. 2024-03-11T19:19:08.986 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:19:08.986 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:19:09.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.177+0000 7fd9589b9640 1 -- 172.21.15.29:0/3940899287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd95410d4c0 msgr2=0x7fd95410d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:09.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.177+0000 7fd9589b9640 1 --2- 172.21.15.29:0/3940899287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd95410d4c0 0x7fd95410d970 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fd94c009900 tx=0x7fd94c02bc50 comp rx=0 tx=0).stop 2024-03-11T19:19:09.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.178+0000 7fd9589b9640 1 -- 172.21.15.29:0/3940899287 shutdown_connections 2024-03-11T19:19:09.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.178+0000 7fd9589b9640 1 --2- 172.21.15.29:0/3940899287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd95410d4c0 0x7fd95410d970 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.178+0000 7fd9589b9640 1 --2- 172.21.15.29:0/3940899287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd95410aaf0 0x7fd95410cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.178+0000 7fd9589b9640 1 --2- 172.21.15.29:0/3940899287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd954108180 0x7fd95410a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.178+0000 7fd9589b9640 1 -- 172.21.15.29:0/3940899287 >> 172.21.15.29:0/3940899287 conn(0x7fd95406dff0 msgr2=0x7fd954070430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:09.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.178+0000 7fd9589b9640 1 -- 172.21.15.29:0/3940899287 shutdown_connections 2024-03-11T19:19:09.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.178+0000 7fd9589b9640 1 -- 172.21.15.29:0/3940899287 wait complete. 2024-03-11T19:19:09.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.179+0000 7fd9589b9640 1 Processor -- start 2024-03-11T19:19:09.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.179+0000 7fd9589b9640 1 -- start start 2024-03-11T19:19:09.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.179+0000 7fd9589b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd954108180 0x7fd9541a5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.179+0000 7fd9589b9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd95410aaf0 0x7fd9541a6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.179+0000 7fd9589b9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd95410d4c0 0x7fd9541ad590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.179+0000 7fd9589b9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9541adad0 con 0x7fd954108180 2024-03-11T19:19:09.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.179+0000 7fd9589b9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95410f8e0 con 0x7fd95410aaf0 2024-03-11T19:19:09.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.179+0000 7fd9589b9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95410faa0 con 0x7fd95410d4c0 2024-03-11T19:19:09.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.179+0000 7fd952d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd95410d4c0 0x7fd9541ad590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:09.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.180+0000 7fd952d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd95410d4c0 0x7fd9541ad590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:47860/0 (socket says 172.21.15.29:47860) 2024-03-11T19:19:09.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.180+0000 7fd952d9d640 1 -- 172.21.15.29:0/3155491930 learned_addr learned my addr 172.21.15.29:0/3155491930 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:09.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.180+0000 7fd951d9b640 1 --2- 172.21.15.29:0/3155491930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd95410aaf0 0x7fd9541a6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:09.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.180+0000 7fd95259c640 1 --2- 172.21.15.29:0/3155491930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd954108180 0x7fd9541a5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:09.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.180+0000 7fd951d9b640 1 -- 172.21.15.29:0/3155491930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd95410d4c0 msgr2=0x7fd9541ad590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:09.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.180+0000 7fd951d9b640 1 --2- 172.21.15.29:0/3155491930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd95410d4c0 0x7fd9541ad590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.180+0000 7fd951d9b640 1 -- 172.21.15.29:0/3155491930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd954108180 msgr2=0x7fd9541a5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:09.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.180+0000 7fd951d9b640 1 --2- 172.21.15.29:0/3155491930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd954108180 0x7fd9541a5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.180+0000 7fd951d9b640 1 -- 172.21.15.29:0/3155491930 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd94c009560 con 0x7fd95410aaf0 2024-03-11T19:19:09.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.181+0000 7fd951d9b640 1 --2- 172.21.15.29:0/3155491930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd95410aaf0 0x7fd9541a6520 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd94400be20 tx=0x7fd944009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:09.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.181+0000 7fd94b7fe640 1 -- 172.21.15.29:0/3155491930 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd944014440 con 0x7fd95410aaf0 2024-03-11T19:19:09.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.181+0000 7fd94b7fe640 1 -- 172.21.15.29:0/3155491930 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd944014a50 con 0x7fd95410aaf0 2024-03-11T19:19:09.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.181+0000 7fd94b7fe640 1 -- 172.21.15.29:0/3155491930 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd944012a90 con 0x7fd95410aaf0 2024-03-11T19:19:09.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.181+0000 7fd9589b9640 1 -- 172.21.15.29:0/3155491930 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95410fd80 con 0x7fd95410aaf0 2024-03-11T19:19:09.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.181+0000 7fd9589b9640 1 -- 172.21.15.29:0/3155491930 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9541102c0 con 0x7fd95410aaf0 2024-03-11T19:19:09.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.184+0000 7fd94b7fe640 1 -- 172.21.15.29:0/3155491930 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd944014bc0 con 0x7fd95410aaf0 2024-03-11T19:19:09.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.185+0000 7fd94b7fe640 1 --2- 172.21.15.29:0/3155491930 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd9400a1ab0 0x7fd9400a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.186+0000 7fd94b7fe640 1 -- 172.21.15.29:0/3155491930 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd9440cce50 con 0x7fd95410aaf0 2024-03-11T19:19:09.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.186+0000 7fd95259c640 1 --2- 172.21.15.29:0/3155491930 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd9400a1ab0 0x7fd9400a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:09.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.186+0000 7fd9589b9640 1 -- 172.21.15.29:0/3155491930 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95411c270 con 0x7fd95410aaf0 2024-03-11T19:19:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.187+0000 7fd95259c640 1 --2- 172.21.15.29:0/3155491930 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd9400a1ab0 0x7fd9400a3f70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd93c005fd0 tx=0x7fd93c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:09.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.192+0000 7fd94b7fe640 1 -- 172.21.15.29:0/3155491930 <== mon.2 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd94404fd20 con 0x7fd95410aaf0 2024-03-11T19:19:09.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.635+0000 7fd9589b9640 1 -- 172.21.15.29:0/3155491930 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd954070af0 con 0x7fd95410aaf0 2024-03-11T19:19:09.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.636+0000 7fd94b7fe640 1 -- 172.21.15.29:0/3155491930 <== mon.2 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7fd944019030 con 0x7fd95410aaf0 2024-03-11T19:19:09.640 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:19:09.640 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:19:09.640 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:19:09.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.639+0000 7fd9589b9640 1 -- 172.21.15.29:0/3155491930 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd9400a1ab0 msgr2=0x7fd9400a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:09.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.639+0000 7fd9589b9640 1 --2- 172.21.15.29:0/3155491930 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd9400a1ab0 0x7fd9400a3f70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd93c005fd0 tx=0x7fd93c005850 comp rx=0 tx=0).stop 2024-03-11T19:19:09.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.639+0000 7fd9589b9640 1 -- 172.21.15.29:0/3155491930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd95410aaf0 msgr2=0x7fd9541a6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:09.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.639+0000 7fd9589b9640 1 --2- 172.21.15.29:0/3155491930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd95410aaf0 0x7fd9541a6520 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd94400be20 tx=0x7fd944009f70 comp rx=0 tx=0).stop 2024-03-11T19:19:09.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.640+0000 7fd9589b9640 1 -- 172.21.15.29:0/3155491930 shutdown_connections 2024-03-11T19:19:09.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.640+0000 7fd9589b9640 1 --2- 172.21.15.29:0/3155491930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd95410d4c0 0x7fd9541ad590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.640+0000 7fd9589b9640 1 --2- 172.21.15.29:0/3155491930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd95410aaf0 0x7fd9541a6520 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.640+0000 7fd9589b9640 1 --2- 172.21.15.29:0/3155491930 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd9400a1ab0 0x7fd9400a3f70 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.640+0000 7fd9589b9640 1 --2- 172.21.15.29:0/3155491930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd954108180 0x7fd9541a5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.640+0000 7fd9589b9640 1 -- 172.21.15.29:0/3155491930 >> 172.21.15.29:0/3155491930 conn(0x7fd95406dff0 msgr2=0x7fd95410af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:09.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.641+0000 7fd9589b9640 1 -- 172.21.15.29:0/3155491930 shutdown_connections 2024-03-11T19:19:09.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.641+0000 7fd9589b9640 1 -- 172.21.15.29:0/3155491930 wait complete. 2024-03-11T19:19:09.668 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:19:09.668 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:19:09.717 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:09 smithi029 ceph-mon[32439]: pgmap v631: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 56 MiB/s wr, 1.10k op/s 2024-03-11T19:19:09.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:09 smithi190 ceph-mon[36292]: pgmap v631: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 56 MiB/s wr, 1.10k op/s 2024-03-11T19:19:09.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:09 smithi170 ceph-mon[35690]: pgmap v631: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 56 MiB/s wr, 1.10k op/s 2024-03-11T19:19:09.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.852+0000 7f323a713640 1 -- 172.21.15.29:0/3855021767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3234108180 msgr2=0x7f323410a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:09.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.852+0000 7f323a713640 1 --2- 172.21.15.29:0/3855021767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3234108180 0x7f323410a5b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f322c009ca0 tx=0x7f322c02c120 comp rx=0 tx=0).stop 2024-03-11T19:19:09.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.853+0000 7f323a713640 1 -- 172.21.15.29:0/3855021767 shutdown_connections 2024-03-11T19:19:09.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.853+0000 7f323a713640 1 --2- 172.21.15.29:0/3855021767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f323410d4a0 0x7f323410d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.853+0000 7f323a713640 1 --2- 172.21.15.29:0/3855021767 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f323410aaf0 0x7f323410cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.853+0000 7f323a713640 1 --2- 172.21.15.29:0/3855021767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3234108180 0x7f323410a5b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.853+0000 7f323a713640 1 -- 172.21.15.29:0/3855021767 >> 172.21.15.29:0/3855021767 conn(0x7f323406dfe0 msgr2=0x7f3234070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:09.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.853+0000 7f323a713640 1 -- 172.21.15.29:0/3855021767 shutdown_connections 2024-03-11T19:19:09.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.853+0000 7f323a713640 1 -- 172.21.15.29:0/3855021767 wait complete. 2024-03-11T19:19:09.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.854+0000 7f323a713640 1 Processor -- start 2024-03-11T19:19:09.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.854+0000 7f323a713640 1 -- start start 2024-03-11T19:19:09.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.854+0000 7f323a713640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3234108180 0x7f32341a5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.854+0000 7f323a713640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f323410aaf0 0x7f32341a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.854+0000 7f323a713640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f323410d4a0 0x7f32341ad360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.854+0000 7f323a713640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32341ad8a0 con 0x7f323410d4a0 2024-03-11T19:19:09.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.854+0000 7f323a713640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32341ada10 con 0x7f323410aaf0 2024-03-11T19:19:09.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.854+0000 7f323a713640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32341adbd0 con 0x7f3234108180 2024-03-11T19:19:09.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.855+0000 7f32337fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f323410aaf0 0x7f32341a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:09.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.855+0000 7f32337fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f323410aaf0 0x7f32341a62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55802/0 (socket says 172.21.15.29:55802) 2024-03-11T19:19:09.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.855+0000 7f32337fe640 1 -- 172.21.15.29:0/3746128041 learned_addr learned my addr 172.21.15.29:0/3746128041 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:09.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.855+0000 7f3238cb0640 1 --2- 172.21.15.29:0/3746128041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f323410d4a0 0x7f32341ad360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:09.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.855+0000 7f3238cb0640 1 -- 172.21.15.29:0/3746128041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3234108180 msgr2=0x7f32341a5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:09.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.855+0000 7f3238cb0640 1 --2- 172.21.15.29:0/3746128041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3234108180 0x7f32341a5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.855+0000 7f3238cb0640 1 -- 172.21.15.29:0/3746128041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f323410aaf0 msgr2=0x7f32341a62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:09.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.855+0000 7f3238cb0640 1 --2- 172.21.15.29:0/3746128041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f323410aaf0 0x7f32341a62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.855+0000 7f3238cb0640 1 -- 172.21.15.29:0/3746128041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c009900 con 0x7f323410d4a0 2024-03-11T19:19:09.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.856+0000 7f3238cb0640 1 --2- 172.21.15.29:0/3746128041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f323410d4a0 0x7f32341ad360 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f322400d650 tx=0x7f322400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:09.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.856+0000 7f32317fa640 1 -- 172.21.15.29:0/3746128041 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3224005220 con 0x7f323410d4a0 2024-03-11T19:19:09.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.856+0000 7f32317fa640 1 -- 172.21.15.29:0/3746128041 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3224009cd0 con 0x7f323410d4a0 2024-03-11T19:19:09.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.856+0000 7f323a713640 1 -- 172.21.15.29:0/3746128041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32341adeb0 con 0x7f323410d4a0 2024-03-11T19:19:09.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.856+0000 7f323a713640 1 -- 172.21.15.29:0/3746128041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32341ae2e0 con 0x7f323410d4a0 2024-03-11T19:19:09.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.857+0000 7f32317fa640 1 -- 172.21.15.29:0/3746128041 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3224004d40 con 0x7f323410d4a0 2024-03-11T19:19:09.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.860+0000 7f323d3c8740 1 Processor -- start 2024-03-11T19:19:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.861+0000 7f32317fa640 1 -- 172.21.15.29:0/3746128041 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3224009820 con 0x7f323410d4a0 2024-03-11T19:19:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.862+0000 7f32317fa640 1 --2- 172.21.15.29:0/3746128041 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f321c0a1a30 0x7f321c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.862+0000 7f32317fa640 1 -- 172.21.15.29:0/3746128041 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f32240ccca0 con 0x7f323410d4a0 2024-03-11T19:19:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.862+0000 7f32337fe640 1 --2- 172.21.15.29:0/3746128041 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f321c0a1a30 0x7f321c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.863+0000 7f323d3c8740 1 -- start start 2024-03-11T19:19:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.863+0000 7f323d3c8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d1506270 0x5654d1506690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.863+0000 7f323d3c8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5654d1506c60 0x5654d151c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.863+0000 7f323d3c8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d151cde0 0x5654d151f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.863+0000 7f323d3c8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654d14c2960 con 0x5654d151cde0 2024-03-11T19:19:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.863+0000 7f323d3c8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654d1476310 con 0x5654d1506270 2024-03-11T19:19:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.863+0000 7f323d3c8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654d1473840 con 0x5654d1506c60 2024-03-11T19:19:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.863+0000 7f32337fe640 1 --2- 172.21.15.29:0/3746128041 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f321c0a1a30 0x7f321c0a3ef0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f32341a72c0 tx=0x7f3228005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.864+0000 7f3238cb0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d151cde0 0x5654d151f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.864+0000 7f32337fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5654d1506c60 0x5654d151c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.864+0000 7f3238cb0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d151cde0 0x5654d151f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53592/0 (socket says 172.21.15.29:53592) 2024-03-11T19:19:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.864+0000 7f3238cb0640 1 -- 172.21.15.29:0/3785187694 learned_addr learned my addr 172.21.15.29:0/3785187694 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.864+0000 7f3238cb0640 1 -- 172.21.15.29:0/3785187694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5654d1506c60 msgr2=0x5654d151c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.864+0000 7f3238cb0640 1 --2- 172.21.15.29:0/3785187694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5654d1506c60 0x5654d151c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.864+0000 7f3238cb0640 1 -- 172.21.15.29:0/3785187694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d1506270 msgr2=0x5654d1506690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.864+0000 7f3238cb0640 1 --2- 172.21.15.29:0/3785187694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d1506270 0x5654d1506690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.864+0000 7f3238cb0640 1 -- 172.21.15.29:0/3785187694 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654d1431060 con 0x5654d151cde0 2024-03-11T19:19:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.865+0000 7f3238cb0640 1 --2- 172.21.15.29:0/3785187694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d151cde0 0x5654d151f270 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f3224009f80 tx=0x7f32240998b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c1d4ea05fbf19e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.866+0000 7f321a7fc640 1 -- 172.21.15.29:0/3785187694 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322409a3b0 con 0x5654d151cde0 2024-03-11T19:19:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.866+0000 7f321a7fc640 1 -- 172.21.15.29:0/3785187694 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3224099cb0 con 0x5654d151cde0 2024-03-11T19:19:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.867+0000 7f323d3c8740 1 -- 172.21.15.29:0/3785187694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d151cde0 msgr2=0x5654d151f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:09.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.867+0000 7f323d3c8740 1 --2- 172.21.15.29:0/3785187694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d151cde0 0x5654d151f270 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f3224009f80 tx=0x7f32240998b0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.867+0000 7f323d3c8740 1 -- 172.21.15.29:0/3785187694 shutdown_connections 2024-03-11T19:19:09.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.867+0000 7f323d3c8740 1 --2- 172.21.15.29:0/3785187694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d151cde0 0x5654d151f270 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.867+0000 7f323d3c8740 1 --2- 172.21.15.29:0/3785187694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5654d1506c60 0x5654d151c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.867+0000 7f323d3c8740 1 --2- 172.21.15.29:0/3785187694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d1506270 0x5654d1506690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.867+0000 7f323d3c8740 1 -- 172.21.15.29:0/3785187694 >> 172.21.15.29:0/3785187694 conn(0x5654d15017e0 msgr2=0x5654d1501be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:09.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.868+0000 7f323d3c8740 1 -- 172.21.15.29:0/3785187694 shutdown_connections 2024-03-11T19:19:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.868+0000 7f323d3c8740 1 -- 172.21.15.29:0/3785187694 wait complete. 2024-03-11T19:19:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.868+0000 7f323d3c8740 1 -- start start 2024-03-11T19:19:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.868+0000 7f323d3c8740 1 Processor -- start 2024-03-11T19:19:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.869+0000 7f323d3c8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d1506270 0x5654d150d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.869+0000 7f323d3c8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d1506c60 0x5654d150dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.869+0000 7f323d3c8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5654d151cde0 0x5654d1514be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.869+0000 7f323d3c8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654d146f020 con 0x5654d1506270 2024-03-11T19:19:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.869+0000 7f323d3c8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654d1142430 con 0x5654d1506c60 2024-03-11T19:19:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.869+0000 7f323d3c8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654d10fee00 con 0x5654d151cde0 2024-03-11T19:19:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.869+0000 7f323d3c8740 10 client.0 _ll_register_callbacks cb 0x5654d1416cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.869+0000 7f32337fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d1506c60 0x5654d150dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.869+0000 7f32337fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d1506c60 0x5654d150dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55824/0 (socket says 172.21.15.29:55824) 2024-03-11T19:19:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.869+0000 7f32337fe640 1 -- 172.21.15.29:0/152736664 learned_addr learned my addr 172.21.15.29:0/152736664 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.870+0000 7f32337fe640 1 -- 172.21.15.29:0/152736664 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5654d151cde0 msgr2=0x5654d1514be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:09.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.870+0000 7f32337fe640 1 --2- 172.21.15.29:0/152736664 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5654d151cde0 0x5654d1514be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.870+0000 7f32337fe640 1 -- 172.21.15.29:0/152736664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d1506270 msgr2=0x5654d150d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:09.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.870+0000 7f32337fe640 1 --2- 172.21.15.29:0/152736664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d1506270 0x5654d150d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.870+0000 7f32337fe640 1 -- 172.21.15.29:0/152736664 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32341ae2e0 con 0x5654d1506c60 2024-03-11T19:19:09.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.870+0000 7f32337fe640 1 --2- 172.21.15.29:0/152736664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d1506c60 0x5654d150dba0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f322801b9e0 tx=0x7f3228013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af3904fbd40b25c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:09.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.870+0000 7f32137fe640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:19:09.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.870+0000 7f32137fe640 1 -- 172.21.15.29:0/152736664 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228020040 con 0x5654d1506c60 2024-03-11T19:19:09.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.871+0000 7f323d3c8740 1 -- 172.21.15.29:0/152736664 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5654d116e7e0 con 0x5654d1506c60 2024-03-11T19:19:09.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.871+0000 7f32137fe640 1 -- 172.21.15.29:0/152736664 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f3228009e10 con 0x5654d1506c60 2024-03-11T19:19:09.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.871+0000 7f32137fe640 1 -- 172.21.15.29:0/152736664 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f322800ed40 con 0x5654d1506c60 2024-03-11T19:19:09.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.875+0000 7f323d3c8740 10 client.27077 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:09.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.875+0000 7f323d3c8740 1 -- 172.21.15.29:0/152736664 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5654d146d6a0 con 0x5654d1506c60 2024-03-11T19:19:09.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.875+0000 7f32137fe640 1 -- 172.21.15.29:0/152736664 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5654d116e7e0 con 0x5654d1506c60 2024-03-11T19:19:09.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.875+0000 7f32137fe640 1 -- 172.21.15.29:0/152736664 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3228004bd0 con 0x5654d1506c60 2024-03-11T19:19:09.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.875+0000 7f323d3c8740 10 client.27077 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:09.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.875+0000 7f323d3c8740 10 client.27077 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:19:09.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.875+0000 7f323d3c8740 20 client.27077 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:09.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.875+0000 7f323d3c8740 1 --2- 172.21.15.29:0/152736664 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5654d1519890 0x5654d151bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.875+0000 7f323d3c8740 4 client.27077 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:19:09.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.875+0000 7f323d3c8740 1 -- 172.21.15.29:0/152736664 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5654d11ab720 con 0x5654d1519890 2024-03-11T19:19:09.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.876+0000 7f3233fff640 1 --2- 172.21.15.29:0/152736664 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5654d1519890 0x5654d151bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.877+0000 7f3233fff640 1 --2- 172.21.15.29:0/152736664 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5654d1519890 0x5654d151bcd0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a2050e296bb7c63b server_cookie=b5fceffa89ed55c6 in_seq=0 out_seq=0 2024-03-11T19:19:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.877+0000 7f32137fe640 10 client.27077 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:19:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.878+0000 7f32137fe640 1 -- 172.21.15.29:0/152736664 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5654d11ab720 con 0x5654d1519890 2024-03-11T19:19:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.878+0000 7f32137fe640 10 client.27077 handle_command_reply: tid=0 2024-03-11T19:19:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.878+0000 7f323d3c8740 1 client.27077 shutdown 2024-03-11T19:19:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.878+0000 7f323d3c8740 1 -- 172.21.15.29:0/152736664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d1506c60 msgr2=0x5654d150dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.878+0000 7f323d3c8740 1 --2- 172.21.15.29:0/152736664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d1506c60 0x5654d150dba0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f322801b9e0 tx=0x7f3228013040 comp rx=0 tx=0).stop 2024-03-11T19:19:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.878+0000 7f323d3c8740 1 -- 172.21.15.29:0/152736664 shutdown_connections 2024-03-11T19:19:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.878+0000 7f323d3c8740 1 --2- 172.21.15.29:0/152736664 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5654d1519890 0x5654d151bcd0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.878+0000 7f323d3c8740 1 --2- 172.21.15.29:0/152736664 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5654d151cde0 0x5654d1514be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.878+0000 7f323d3c8740 1 --2- 172.21.15.29:0/152736664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d1506c60 0x5654d150dba0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.878+0000 7f32137fe640 0 client.27077 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:19:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.878+0000 7f323d3c8740 1 --2- 172.21.15.29:0/152736664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d1506270 0x5654d150d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.878+0000 7f323d3c8740 1 -- 172.21.15.29:0/152736664 >> 172.21.15.29:0/152736664 conn(0x5654d15017e0 msgr2=0x5654d1500b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:09.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.878+0000 7f323d3c8740 1 -- 172.21.15.29:0/152736664 shutdown_connections 2024-03-11T19:19:09.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.878+0000 7f323d3c8740 1 -- 172.21.15.29:0/152736664 wait complete. 2024-03-11T19:19:09.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.878+0000 7f323d3c8740 20 client.27077 trim_cache size 0 max 16384 2024-03-11T19:19:09.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.901+0000 7f323d3c8740 1 Processor -- start 2024-03-11T19:19:09.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.901+0000 7f323d3c8740 1 -- start start 2024-03-11T19:19:09.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.902+0000 7f323d3c8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5654d1506270 0x5654d1523a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.902+0000 7f323d3c8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d1506c60 0x5654d1523fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.902+0000 7f323d3c8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d151cde0 0x5654d150c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.902+0000 7f323d3c8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654d13f4130 con 0x5654d1506c60 2024-03-11T19:19:09.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.902+0000 7f323d3c8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654d1501650 con 0x5654d151cde0 2024-03-11T19:19:09.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.902+0000 7f323d3c8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654d1507040 con 0x5654d1506270 2024-03-11T19:19:09.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.902+0000 7f32337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d1506c60 0x5654d1523fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:09.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.902+0000 7f32337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d1506c60 0x5654d1523fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53614/0 (socket says 172.21.15.29:53614) 2024-03-11T19:19:09.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.902+0000 7f32337fe640 1 -- 172.21.15.29:0/3653951782 learned_addr learned my addr 172.21.15.29:0/3653951782 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:09.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.902+0000 7f3233fff640 1 --2- 172.21.15.29:0/3653951782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5654d1506270 0x5654d1523a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:09.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.902+0000 7f3238cb0640 1 --2- 172.21.15.29:0/3653951782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d151cde0 0x5654d150c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:09.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.902+0000 7f32337fe640 1 -- 172.21.15.29:0/3653951782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5654d1506270 msgr2=0x5654d1523a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:09.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.902+0000 7f32337fe640 1 --2- 172.21.15.29:0/3653951782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5654d1506270 0x5654d1523a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.902+0000 7f32337fe640 1 -- 172.21.15.29:0/3653951782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d151cde0 msgr2=0x5654d150c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.902+0000 7f32337fe640 1 --2- 172.21.15.29:0/3653951782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d151cde0 0x5654d150c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.902+0000 7f32337fe640 1 -- 172.21.15.29:0/3653951782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654d1409c20 con 0x5654d1506c60 2024-03-11T19:19:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.903+0000 7f32337fe640 1 --2- 172.21.15.29:0/3653951782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d1506c60 0x5654d1523fc0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f3228011ca0 tx=0x7f3228009e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=772bc33fd791f2fc server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.903+0000 7f3219ffb640 1 -- 172.21.15.29:0/3653951782 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322800eb40 con 0x5654d1506c60 2024-03-11T19:19:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.903+0000 7f3219ffb640 1 -- 172.21.15.29:0/3653951782 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3228026e00 con 0x5654d1506c60 2024-03-11T19:19:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.903+0000 7f3219ffb640 1 -- 172.21.15.29:0/3653951782 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228005530 con 0x5654d1506c60 2024-03-11T19:19:09.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.904+0000 7f323d3c8740 1 -- 172.21.15.29:0/3653951782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d1506c60 msgr2=0x5654d1523fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:09.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.904+0000 7f323d3c8740 1 --2- 172.21.15.29:0/3653951782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d1506c60 0x5654d1523fc0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f3228011ca0 tx=0x7f3228009e10 comp rx=0 tx=0).stop 2024-03-11T19:19:09.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.904+0000 7f323d3c8740 1 -- 172.21.15.29:0/3653951782 shutdown_connections 2024-03-11T19:19:09.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.904+0000 7f323d3c8740 1 --2- 172.21.15.29:0/3653951782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d151cde0 0x5654d150c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.904+0000 7f323d3c8740 1 --2- 172.21.15.29:0/3653951782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d1506c60 0x5654d1523fc0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.904+0000 7f323d3c8740 1 --2- 172.21.15.29:0/3653951782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5654d1506270 0x5654d1523a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.904+0000 7f323d3c8740 1 -- 172.21.15.29:0/3653951782 >> 172.21.15.29:0/3653951782 conn(0x5654d15017e0 msgr2=0x5654d150bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.904+0000 7f323d3c8740 1 -- 172.21.15.29:0/3653951782 shutdown_connections 2024-03-11T19:19:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.904+0000 7f323d3c8740 1 -- 172.21.15.29:0/3653951782 wait complete. 2024-03-11T19:19:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.905+0000 7f323d3c8740 1 -- start start 2024-03-11T19:19:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.905+0000 7f323d3c8740 1 Processor -- start 2024-03-11T19:19:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.906+0000 7f323d3c8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5654d1506270 0x5654d156b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.906+0000 7f323d3c8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d1506c60 0x5654d156ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.907+0000 7f323d3c8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d151cde0 0x5654d1572890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.907+0000 7f323d3c8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654d1572dd0 con 0x5654d151cde0 2024-03-11T19:19:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.907+0000 7f32337fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d1506c60 0x5654d156ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.907+0000 7f32337fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d1506c60 0x5654d156ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55846/0 (socket says 172.21.15.29:55846) 2024-03-11T19:19:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.907+0000 7f323d3c8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654d1572f10 con 0x5654d1506c60 2024-03-11T19:19:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.907+0000 7f3233fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5654d1506270 0x5654d156b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.907+0000 7f32337fe640 1 -- 172.21.15.29:0/502486606 learned_addr learned my addr 172.21.15.29:0/502486606 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.907+0000 7f323d3c8740 1 -- 172.21.15.29:0/502486606 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654d1573050 con 0x5654d1506270 2024-03-11T19:19:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.907+0000 7f3238cb0640 1 --2- 172.21.15.29:0/502486606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d151cde0 0x5654d1572890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.907+0000 7f323d3c8740 10 client.0 _ll_register_callbacks cb 0x7f3224005380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.907+0000 7f32337fe640 1 -- 172.21.15.29:0/502486606 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5654d1506270 msgr2=0x5654d156b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.907+0000 7f32337fe640 1 --2- 172.21.15.29:0/502486606 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5654d1506270 0x5654d156b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.907+0000 7f32337fe640 1 -- 172.21.15.29:0/502486606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d151cde0 msgr2=0x5654d1572890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.907+0000 7f32337fe640 1 --2- 172.21.15.29:0/502486606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d151cde0 0x5654d1572890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.907+0000 7f32337fe640 1 -- 172.21.15.29:0/502486606 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5654d146d6a0 con 0x5654d1506c60 2024-03-11T19:19:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.907+0000 7f3238cb0640 1 --2- 172.21.15.29:0/502486606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d151cde0 0x5654d1572890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.908+0000 7f32337fe640 1 --2- 172.21.15.29:0/502486606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d1506c60 0x5654d156ba30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f322800e900 tx=0x7f3228004fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=521e216627ce3450 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.908+0000 7f32137fe640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:19:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.908+0000 7f32137fe640 1 -- 172.21.15.29:0/502486606 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228020030 con 0x5654d1506c60 2024-03-11T19:19:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.908+0000 7f32137fe640 1 -- 172.21.15.29:0/502486606 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f32280265a0 con 0x5654d1506c60 2024-03-11T19:19:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.909+0000 7f323d3c8740 1 -- 172.21.15.29:0/502486606 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3224099cb0 con 0x5654d1506c60 2024-03-11T19:19:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.909+0000 7f32137fe640 1 -- 172.21.15.29:0/502486606 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3228026760 con 0x5654d1506c60 2024-03-11T19:19:09.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.911+0000 7f323d3c8740 10 client.27083 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:09.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.911+0000 7f323d3c8740 1 -- 172.21.15.29:0/502486606 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5654d142f8c0 con 0x5654d1506c60 2024-03-11T19:19:09.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.911+0000 7f32137fe640 1 -- 172.21.15.29:0/502486606 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3228004620 con 0x5654d1506c60 2024-03-11T19:19:09.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.911+0000 7f32137fe640 1 -- 172.21.15.29:0/502486606 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f322804b070 con 0x5654d1506c60 2024-03-11T19:19:09.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.911+0000 7f323d3c8740 10 client.27083 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:09.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.911+0000 7f323d3c8740 10 client.27083 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:19:09.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.911+0000 7f323d3c8740 20 client.27083 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:09.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.911+0000 7f323d3c8740 1 --2- 172.21.15.29:0/502486606 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5654d15746f0 0x5654d1576b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:09.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.912+0000 7f323d3c8740 4 client.27083 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:19:09.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.912+0000 7f323d3c8740 1 -- 172.21.15.29:0/502486606 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f3224099cb0 con 0x5654d15746f0 2024-03-11T19:19:09.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.912+0000 7f3233fff640 1 --2- 172.21.15.29:0/502486606 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5654d15746f0 0x5654d1576b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:09.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.912+0000 7f32137fe640 10 client.27083 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:19:09.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.913+0000 7f3233fff640 1 --2- 172.21.15.29:0/502486606 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5654d15746f0 0x5654d1576b10 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d5b32fb2f737ce6f server_cookie=3d16c6184ff48e8b in_seq=0 out_seq=0 2024-03-11T19:19:09.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.913+0000 7f32137fe640 1 -- 172.21.15.29:0/502486606 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f3224099cb0 con 0x5654d15746f0 2024-03-11T19:19:09.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.913+0000 7f32137fe640 10 client.27083 handle_command_reply: tid=0 2024-03-11T19:19:09.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.913+0000 7f323d3c8740 1 client.27083 shutdown 2024-03-11T19:19:09.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.914+0000 7f323d3c8740 1 -- 172.21.15.29:0/502486606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d1506c60 msgr2=0x5654d156ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:09.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.914+0000 7f323d3c8740 1 --2- 172.21.15.29:0/502486606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d1506c60 0x5654d156ba30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f322800e900 tx=0x7f3228004fd0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.914+0000 7f323d3c8740 1 -- 172.21.15.29:0/502486606 shutdown_connections 2024-03-11T19:19:09.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.914+0000 7f323d3c8740 1 --2- 172.21.15.29:0/502486606 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5654d15746f0 0x5654d1576b10 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.914+0000 7f32137fe640 0 client.27083 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:19:09.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.914+0000 7f323d3c8740 1 --2- 172.21.15.29:0/502486606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5654d151cde0 0x5654d1572890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.914+0000 7f323d3c8740 1 --2- 172.21.15.29:0/502486606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5654d1506c60 0x5654d156ba30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.914+0000 7f323d3c8740 1 --2- 172.21.15.29:0/502486606 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5654d1506270 0x5654d156b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.914+0000 7f323d3c8740 1 -- 172.21.15.29:0/502486606 >> 172.21.15.29:0/502486606 conn(0x5654d15017e0 msgr2=0x5654d150bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:09.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.915+0000 7f323d3c8740 1 -- 172.21.15.29:0/502486606 shutdown_connections 2024-03-11T19:19:09.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.915+0000 7f323d3c8740 1 -- 172.21.15.29:0/502486606 wait complete. 2024-03-11T19:19:09.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.915+0000 7f323d3c8740 20 client.27083 trim_cache size 0 max 16384 2024-03-11T19:19:09.921 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:19:09.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.922+0000 7f323a713640 1 -- 172.21.15.29:0/3746128041 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f321c0a1a30 msgr2=0x7f321c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:09.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.922+0000 7f323a713640 1 --2- 172.21.15.29:0/3746128041 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f321c0a1a30 0x7f321c0a3ef0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f32341a72c0 tx=0x7f3228005f90 comp rx=0 tx=0).stop 2024-03-11T19:19:09.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.922+0000 7f323a713640 1 -- 172.21.15.29:0/3746128041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f323410d4a0 msgr2=0x7f32341ad360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:09.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.922+0000 7f323a713640 1 --2- 172.21.15.29:0/3746128041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f323410d4a0 0x7f32341ad360 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f322400d650 tx=0x7f322400db10 comp rx=0 tx=0).stop 2024-03-11T19:19:09.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.925+0000 7f323a713640 1 -- 172.21.15.29:0/3746128041 shutdown_connections 2024-03-11T19:19:09.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.925+0000 7f323a713640 1 --2- 172.21.15.29:0/3746128041 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f321c0a1a30 0x7f321c0a3ef0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.925+0000 7f323a713640 1 --2- 172.21.15.29:0/3746128041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f323410d4a0 0x7f32341ad360 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.925+0000 7f323a713640 1 --2- 172.21.15.29:0/3746128041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f323410aaf0 0x7f32341a62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.925+0000 7f323a713640 1 --2- 172.21.15.29:0/3746128041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3234108180 0x7f32341a5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:09.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.925+0000 7f323a713640 1 -- 172.21.15.29:0/3746128041 >> 172.21.15.29:0/3746128041 conn(0x7f323406dfe0 msgr2=0x7f323410cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:09.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.926+0000 7f323a713640 1 -- 172.21.15.29:0/3746128041 shutdown_connections 2024-03-11T19:19:09.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:09.926+0000 7f323a713640 1 -- 172.21.15.29:0/3746128041 wait complete. 2024-03-11T19:19:09.949 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:19:09.949 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:19:10.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.134+0000 7fed0f426640 1 -- 172.21.15.29:0/2660484899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed08108890 msgr2=0x7fed08108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.134+0000 7fed0f426640 1 --2- 172.21.15.29:0/2660484899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed08108890 0x7fed08108d10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fecf8009900 tx=0x7fecf802bc40 comp rx=0 tx=0).stop 2024-03-11T19:19:10.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.135+0000 7fed0f426640 1 -- 172.21.15.29:0/2660484899 shutdown_connections 2024-03-11T19:19:10.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.135+0000 7fed0f426640 1 --2- 172.21.15.29:0/2660484899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed08109250 0x7fed0810fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.135+0000 7fed0f426640 1 --2- 172.21.15.29:0/2660484899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed08108890 0x7fed08108d10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.135+0000 7fed0f426640 1 --2- 172.21.15.29:0/2660484899 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed081076a0 0x7fed08107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.135+0000 7fed0f426640 1 -- 172.21.15.29:0/2660484899 >> 172.21.15.29:0/2660484899 conn(0x7fed08076650 msgr2=0x7fed08078a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.135+0000 7fed0f426640 1 -- 172.21.15.29:0/2660484899 shutdown_connections 2024-03-11T19:19:10.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.135+0000 7fed0f426640 1 -- 172.21.15.29:0/2660484899 wait complete. 2024-03-11T19:19:10.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.136+0000 7fed0f426640 1 Processor -- start 2024-03-11T19:19:10.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.136+0000 7fed0f426640 1 -- start start 2024-03-11T19:19:10.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.136+0000 7fed0f426640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed081076a0 0x7fed0819d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.136+0000 7fed0f426640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed08108890 0x7fed0819dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.136+0000 7fed0f426640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed08109250 0x7fed081a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.136+0000 7fed0f426640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed080799d0 con 0x7fed08108890 2024-03-11T19:19:10.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.136+0000 7fed0f426640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed08079b40 con 0x7fed08109250 2024-03-11T19:19:10.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.136+0000 7fed0f426640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed08079d00 con 0x7fed081076a0 2024-03-11T19:19:10.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.137+0000 7fed0e424640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed081076a0 0x7fed0819d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.137+0000 7fed0e424640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed081076a0 0x7fed0819d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:47938/0 (socket says 172.21.15.29:47938) 2024-03-11T19:19:10.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.137+0000 7fed0e424640 1 -- 172.21.15.29:0/3479826805 learned_addr learned my addr 172.21.15.29:0/3479826805 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.137+0000 7fed0dc23640 1 --2- 172.21.15.29:0/3479826805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed08108890 0x7fed0819dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.137+0000 7fed0ec25640 1 --2- 172.21.15.29:0/3479826805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed08109250 0x7fed081a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.137+0000 7fed0e424640 1 -- 172.21.15.29:0/3479826805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed08109250 msgr2=0x7fed081a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.137+0000 7fed0e424640 1 --2- 172.21.15.29:0/3479826805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed08109250 0x7fed081a4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.137+0000 7fed0e424640 1 -- 172.21.15.29:0/3479826805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed08108890 msgr2=0x7fed0819dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.137+0000 7fed0e424640 1 --2- 172.21.15.29:0/3479826805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed08108890 0x7fed0819dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.137+0000 7fed0e424640 1 -- 172.21.15.29:0/3479826805 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf8009560 con 0x7fed081076a0 2024-03-11T19:19:10.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.138+0000 7fed0ec25640 1 --2- 172.21.15.29:0/3479826805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed08109250 0x7fed081a4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:19:10.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.138+0000 7fed0e424640 1 --2- 172.21.15.29:0/3479826805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed081076a0 0x7fed0819d6b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fed0000ebc0 tx=0x7fed0000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.141+0000 7fecff7fe640 1 -- 172.21.15.29:0/3479826805 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00003b70 con 0x7fed081076a0 2024-03-11T19:19:10.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.141+0000 7fecff7fe640 1 -- 172.21.15.29:0/3479826805 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fed0000eec0 con 0x7fed081076a0 2024-03-11T19:19:10.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.141+0000 7fecff7fe640 1 -- 172.21.15.29:0/3479826805 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00019770 con 0x7fed081076a0 2024-03-11T19:19:10.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.141+0000 7fed0f426640 1 -- 172.21.15.29:0/3479826805 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed08079fe0 con 0x7fed081076a0 2024-03-11T19:19:10.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.141+0000 7fed0f426640 1 -- 172.21.15.29:0/3479826805 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0807a410 con 0x7fed081076a0 2024-03-11T19:19:10.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.145+0000 7fecff7fe640 1 -- 172.21.15.29:0/3479826805 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fed000198d0 con 0x7fed081076a0 2024-03-11T19:19:10.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.146+0000 7fed13267740 1 Processor -- start 2024-03-11T19:19:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.146+0000 7fecff7fe640 1 --2- 172.21.15.29:0/3479826805 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fecf40a1c00 0x7fecf40a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.146+0000 7fecff7fe640 1 -- 172.21.15.29:0/3479826805 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fed00014070 con 0x7fed081076a0 2024-03-11T19:19:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.147+0000 7fed13267740 1 -- start start 2024-03-11T19:19:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.147+0000 7fed13267740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aa7480 0x5648c7aa78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.147+0000 7fed13267740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa7e70 0x5648c7aac970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.147+0000 7fed13267740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aacfe0 0x5648c7aaf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.147+0000 7fed13267740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c7a63960 con 0x5648c7aa7480 2024-03-11T19:19:10.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.147+0000 7fed13267740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c7a17310 con 0x5648c7aa7e70 2024-03-11T19:19:10.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.147+0000 7fed13267740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c7a14840 con 0x5648c7aacfe0 2024-03-11T19:19:10.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.147+0000 7fed0ec25640 1 --2- 172.21.15.29:0/3479826805 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fecf40a1c00 0x7fecf40a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.147+0000 7fed0e424640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aa7480 0x5648c7aa78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.147+0000 7fed0e424640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aa7480 0x5648c7aa78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53658/0 (socket says 172.21.15.29:53658) 2024-03-11T19:19:10.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.147+0000 7fed0e424640 1 -- 172.21.15.29:0/2684837393 learned_addr learned my addr 172.21.15.29:0/2684837393 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.148+0000 7fed0ec25640 1 --2- 172.21.15.29:0/2684837393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aacfe0 0x5648c7aaf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.148+0000 7fed0e424640 1 -- 172.21.15.29:0/2684837393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aacfe0 msgr2=0x5648c7aaf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.148+0000 7fed0e424640 1 --2- 172.21.15.29:0/2684837393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aacfe0 0x5648c7aaf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.148+0000 7fed0e424640 1 -- 172.21.15.29:0/2684837393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa7e70 msgr2=0x5648c7aac970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:19:10.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.148+0000 7fed0e424640 1 --2- 172.21.15.29:0/2684837393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa7e70 0x5648c7aac970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.148+0000 7fed0e424640 1 -- 172.21.15.29:0/2684837393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648c79d2060 con 0x5648c7aa7480 2024-03-11T19:19:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.148+0000 7fed0e424640 1 --2- 172.21.15.29:0/2684837393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aa7480 0x5648c7aa78a0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fed00009270 tx=0x7fed00096e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daf005481c191186 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.148+0000 7fecfcff9640 1 -- 172.21.15.29:0/2684837393 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed000b2070 con 0x5648c7aa7480 2024-03-11T19:19:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.148+0000 7fed0ec25640 1 --2- 172.21.15.29:0/3479826805 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fecf40a1c00 0x7fecf40a40c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fed04014520 tx=0x7fed04014920 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.149+0000 7fecfcff9640 1 -- 172.21.15.29:0/2684837393 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fed0009b2a0 con 0x5648c7aa7480 2024-03-11T19:19:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.149+0000 7fecfcff9640 1 -- 172.21.15.29:0/2684837393 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00097210 con 0x5648c7aa7480 2024-03-11T19:19:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.150+0000 7fed13267740 1 -- 172.21.15.29:0/2684837393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aa7480 msgr2=0x5648c7aa78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.150+0000 7fed13267740 1 --2- 172.21.15.29:0/2684837393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aa7480 0x5648c7aa78a0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fed00009270 tx=0x7fed00096e60 comp rx=0 tx=0).stop 2024-03-11T19:19:10.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.151+0000 7fed13267740 1 -- 172.21.15.29:0/2684837393 shutdown_connections 2024-03-11T19:19:10.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.151+0000 7fed13267740 1 --2- 172.21.15.29:0/2684837393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aacfe0 0x5648c7aaf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.151+0000 7fed13267740 1 --2- 172.21.15.29:0/2684837393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa7e70 0x5648c7aac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.151+0000 7fed13267740 1 --2- 172.21.15.29:0/2684837393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aa7480 0x5648c7aa78a0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.151+0000 7fed13267740 1 -- 172.21.15.29:0/2684837393 >> 172.21.15.29:0/2684837393 conn(0x5648c7aa29f0 msgr2=0x5648c7aa2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.151+0000 7fed13267740 1 -- 172.21.15.29:0/2684837393 shutdown_connections 2024-03-11T19:19:10.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.151+0000 7fed13267740 1 -- 172.21.15.29:0/2684837393 wait complete. 2024-03-11T19:19:10.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.151+0000 7fed13267740 1 -- start start 2024-03-11T19:19:10.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.151+0000 7fed13267740 1 Processor -- start 2024-03-11T19:19:10.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.152+0000 7fed13267740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa7480 0x5648c7ab1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.152+0000 7fed13267740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aa7e70 0x5648c7ab1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.152+0000 7fed0e424640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa7480 0x5648c7ab1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.152+0000 7fed13267740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aacfe0 0x5648c7ab89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.152+0000 7fed13267740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c7a10020 con 0x5648c7aacfe0 2024-03-11T19:19:10.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.152+0000 7fed13267740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c76e3430 con 0x5648c7aa7480 2024-03-11T19:19:10.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.152+0000 7fed13267740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c769fe00 con 0x5648c7aa7e70 2024-03-11T19:19:10.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.152+0000 7fed0dc23640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aa7e70 0x5648c7ab1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.152+0000 7fed0dc23640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aa7e70 0x5648c7ab1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:47958/0 (socket says 172.21.15.29:47958) 2024-03-11T19:19:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.152+0000 7fed13267740 10 client.0 _ll_register_callbacks cb 0x5648c79b7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.152+0000 7fed0dc23640 1 -- 172.21.15.29:0/1719197106 learned_addr learned my addr 172.21.15.29:0/1719197106 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.152+0000 7fed0ec25640 1 --2- 172.21.15.29:0/1719197106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aacfe0 0x5648c7ab89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.153+0000 7fed0dc23640 1 -- 172.21.15.29:0/1719197106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa7480 msgr2=0x5648c7ab1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.153+0000 7fed0dc23640 1 --2- 172.21.15.29:0/1719197106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa7480 0x5648c7ab1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.153+0000 7fed0dc23640 1 -- 172.21.15.29:0/1719197106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aacfe0 msgr2=0x5648c7ab89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.153+0000 7fed0dc23640 1 --2- 172.21.15.29:0/1719197106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aacfe0 0x5648c7ab89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.153+0000 7fed0dc23640 1 -- 172.21.15.29:0/1719197106 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0807a410 con 0x5648c7aa7e70 2024-03-11T19:19:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.153+0000 7fed0dc23640 1 --2- 172.21.15.29:0/1719197106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aa7e70 0x5648c7ab1990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fecf802c7e0 tx=0x7fecf80048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e415b1e108500f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.153+0000 7fecf1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:19:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.153+0000 7fecf1ffb640 1 -- 172.21.15.29:0/1719197106 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf8004450 con 0x5648c7aa7e70 2024-03-11T19:19:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.153+0000 7fed13267740 1 -- 172.21.15.29:0/1719197106 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5648c770f7e0 con 0x5648c7aa7e70 2024-03-11T19:19:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.153+0000 7fecf1ffb640 1 -- 172.21.15.29:0/1719197106 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fecf8004a60 con 0x5648c7aa7e70 2024-03-11T19:19:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.153+0000 7fecf1ffb640 1 -- 172.21.15.29:0/1719197106 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fecf8005d50 con 0x5648c7aa7e70 2024-03-11T19:19:10.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.153+0000 7fed13267740 10 client.27136 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:10.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.153+0000 7fed13267740 1 -- 172.21.15.29:0/1719197106 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5648c7a0e6a0 con 0x5648c7aa7e70 2024-03-11T19:19:10.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.154+0000 7fecf1ffb640 1 -- 172.21.15.29:0/1719197106 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5648c770f7e0 con 0x5648c7aa7e70 2024-03-11T19:19:10.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.154+0000 7fecf1ffb640 1 -- 172.21.15.29:0/1719197106 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fecf803c040 con 0x5648c7aa7e70 2024-03-11T19:19:10.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.155+0000 7fed13267740 10 client.27136 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:10.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.155+0000 7fed13267740 10 client.27136 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:19:10.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.155+0000 7fed13267740 20 client.27136 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:10.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.155+0000 7fed13267740 1 --2- 172.21.15.29:0/1719197106 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5648c7abd710 0x5648c7abfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.155+0000 7fed13267740 4 client.27136 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:19:10.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.155+0000 7fed13267740 1 -- 172.21.15.29:0/1719197106 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5648c774c720 con 0x5648c7abd710 2024-03-11T19:19:10.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.156+0000 7fed0e424640 1 --2- 172.21.15.29:0/1719197106 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5648c7abd710 0x5648c7abfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.159+0000 7fed0e424640 1 --2- 172.21.15.29:0/1719197106 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5648c7abd710 0x5648c7abfb30 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=79c95e28bf65f805 server_cookie=78c5ad2810c41be in_seq=0 out_seq=0 2024-03-11T19:19:10.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.159+0000 7fecf1ffb640 10 client.27136 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:19:10.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.160+0000 7fecf1ffb640 1 -- 172.21.15.29:0/1719197106 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5648c774c720 con 0x5648c7abd710 2024-03-11T19:19:10.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.160+0000 7fecf1ffb640 10 client.27136 handle_command_reply: tid=0 2024-03-11T19:19:10.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.160+0000 7fed13267740 1 client.27136 shutdown 2024-03-11T19:19:10.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.160+0000 7fed13267740 1 -- 172.21.15.29:0/1719197106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aa7e70 msgr2=0x5648c7ab1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.160+0000 7fed13267740 1 --2- 172.21.15.29:0/1719197106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aa7e70 0x5648c7ab1990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fecf802c7e0 tx=0x7fecf80048f0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.160+0000 7fed13267740 1 -- 172.21.15.29:0/1719197106 shutdown_connections 2024-03-11T19:19:10.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.160+0000 7fed13267740 1 --2- 172.21.15.29:0/1719197106 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5648c7abd710 0x5648c7abfb30 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.160+0000 7fed13267740 1 --2- 172.21.15.29:0/1719197106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aacfe0 0x5648c7ab89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.160+0000 7fed13267740 1 --2- 172.21.15.29:0/1719197106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aa7e70 0x5648c7ab1990 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.160+0000 7fed13267740 1 --2- 172.21.15.29:0/1719197106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa7480 0x5648c7ab1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.161+0000 7fecf1ffb640 0 client.27136 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:19:10.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.161+0000 7fed13267740 1 -- 172.21.15.29:0/1719197106 >> 172.21.15.29:0/1719197106 conn(0x5648c7aa29f0 msgr2=0x5648c7aa1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.161+0000 7fed13267740 1 -- 172.21.15.29:0/1719197106 shutdown_connections 2024-03-11T19:19:10.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.161+0000 7fed13267740 1 -- 172.21.15.29:0/1719197106 wait complete. 2024-03-11T19:19:10.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.161+0000 7fed13267740 20 client.27136 trim_cache size 0 max 16384 2024-03-11T19:19:10.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.183+0000 7fed13267740 1 Processor -- start 2024-03-11T19:19:10.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.183+0000 7fed13267740 1 -- start start 2024-03-11T19:19:10.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.184+0000 7fed13267740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aa29f0 0x5648c7aa9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.184+0000 7fed13267740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa7480 0x5648c7aa9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.184+0000 7fed13267740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aa7e70 0x5648c7ab8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.184+0000 7fed0e424640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aa29f0 0x5648c7aa9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.184+0000 7fed0e424640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aa29f0 0x5648c7aa9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:47972/0 (socket says 172.21.15.29:47972) 2024-03-11T19:19:10.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.184+0000 7fed0e424640 1 -- 172.21.15.29:0/1995189966 learned_addr learned my addr 172.21.15.29:0/1995189966 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.184+0000 7fed0dc23640 1 --2- 172.21.15.29:0/1995189966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa7480 0x5648c7aa9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.184+0000 7fed0ec25640 1 --2- 172.21.15.29:0/1995189966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aa7e70 0x5648c7ab8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.184+0000 7fed13267740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c7995130 con 0x5648c7aa7e70 2024-03-11T19:19:10.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.184+0000 7fed13267740 1 -- 172.21.15.29:0/1995189966 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c7aa2860 con 0x5648c7aa7480 2024-03-11T19:19:10.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.184+0000 7fed13267740 1 -- 172.21.15.29:0/1995189966 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c7aa7880 con 0x5648c7aa29f0 2024-03-11T19:19:10.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.185+0000 7fed0e424640 1 -- 172.21.15.29:0/1995189966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa7480 msgr2=0x5648c7aa9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.185+0000 7fed0e424640 1 --2- 172.21.15.29:0/1995189966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa7480 0x5648c7aa9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.185+0000 7fed0e424640 1 -- 172.21.15.29:0/1995189966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aa7e70 msgr2=0x5648c7ab8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.185+0000 7fed0e424640 1 --2- 172.21.15.29:0/1995189966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aa7e70 0x5648c7ab8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.185+0000 7fed0e424640 1 -- 172.21.15.29:0/1995189966 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648c79aac20 con 0x5648c7aa29f0 2024-03-11T19:19:10.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.185+0000 7fed0e424640 1 --2- 172.21.15.29:0/1995189966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aa29f0 0x5648c7aa9150 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fed00098e50 tx=0x7fed00000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1322f29e6a6e317 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.185+0000 7fecf3fff640 1 -- 172.21.15.29:0/1995189966 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed000b2040 con 0x5648c7aa29f0 2024-03-11T19:19:10.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.185+0000 7fecf3fff640 1 -- 172.21.15.29:0/1995189966 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fed000a3340 con 0x5648c7aa29f0 2024-03-11T19:19:10.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.185+0000 7fecf3fff640 1 -- 172.21.15.29:0/1995189966 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed000a2eb0 con 0x5648c7aa29f0 2024-03-11T19:19:10.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.186+0000 7fed13267740 1 -- 172.21.15.29:0/1995189966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aa29f0 msgr2=0x5648c7aa9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.186+0000 7fed13267740 1 --2- 172.21.15.29:0/1995189966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aa29f0 0x5648c7aa9150 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fed00098e50 tx=0x7fed00000f40 comp rx=0 tx=0).stop 2024-03-11T19:19:10.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.186+0000 7fed13267740 1 -- 172.21.15.29:0/1995189966 shutdown_connections 2024-03-11T19:19:10.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.186+0000 7fed13267740 1 --2- 172.21.15.29:0/1995189966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aa7e70 0x5648c7ab8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.186+0000 7fed13267740 1 --2- 172.21.15.29:0/1995189966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa7480 0x5648c7aa9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.186+0000 7fed13267740 1 --2- 172.21.15.29:0/1995189966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aa29f0 0x5648c7aa9150 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.186+0000 7fed13267740 1 -- 172.21.15.29:0/1995189966 >> 172.21.15.29:0/1995189966 conn(0x5648c7aa8a70 msgr2=0x5648c7ab1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.186+0000 7fed13267740 1 -- 172.21.15.29:0/1995189966 shutdown_connections 2024-03-11T19:19:10.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.186+0000 7fed13267740 1 -- 172.21.15.29:0/1995189966 wait complete. 2024-03-11T19:19:10.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.187+0000 7fed13267740 1 -- start start 2024-03-11T19:19:10.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.187+0000 7fed13267740 1 Processor -- start 2024-03-11T19:19:10.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.187+0000 7fed13267740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa29f0 0x5648c7ab84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.187+0000 7fed13267740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aa7480 0x5648c7ab6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.187+0000 7fed13267740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aa7e70 0x5648c7ab6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.187+0000 7fed13267740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c7ab8c90 con 0x5648c7aa7480 2024-03-11T19:19:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.187+0000 7fed13267740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c7ab7530 con 0x5648c7aa29f0 2024-03-11T19:19:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.188+0000 7fed13267740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648c7ab76f0 con 0x5648c7aa7e70 2024-03-11T19:19:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.188+0000 7fed0e424640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa29f0 0x5648c7ab84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.188+0000 7fed0ec25640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aa7e70 0x5648c7ab6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.188+0000 7fed0e424640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa29f0 0x5648c7ab84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55898/0 (socket says 172.21.15.29:55898) 2024-03-11T19:19:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.188+0000 7fed0e424640 1 -- 172.21.15.29:0/3887997960 learned_addr learned my addr 172.21.15.29:0/3887997960 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.188+0000 7fed13267740 10 client.0 _ll_register_callbacks cb 0x7fed00004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.188+0000 7fed0e424640 1 -- 172.21.15.29:0/3887997960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aa7e70 msgr2=0x5648c7ab6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.188+0000 7fed0e424640 1 --2- 172.21.15.29:0/3887997960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aa7e70 0x5648c7ab6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.188+0000 7fed0e424640 1 -- 172.21.15.29:0/3887997960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aa7480 msgr2=0x5648c7ab6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.188+0000 7fed0e424640 1 --2- 172.21.15.29:0/3887997960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aa7480 0x5648c7ab6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.188+0000 7fed0e424640 1 -- 172.21.15.29:0/3887997960 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5648c79d2060 con 0x5648c7aa29f0 2024-03-11T19:19:10.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.188+0000 7fed0e424640 1 --2- 172.21.15.29:0/3887997960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa29f0 0x5648c7ab84a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fed000a2e10 tx=0x7fed0009b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3e436efc8177717 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.188+0000 7fecf1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:19:10.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.189+0000 7fecf1ffb640 1 -- 172.21.15.29:0/3887997960 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed000b2040 con 0x5648c7aa29f0 2024-03-11T19:19:10.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.189+0000 7fecf1ffb640 1 -- 172.21.15.29:0/3887997960 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fed000a2eb0 con 0x5648c7aa29f0 2024-03-11T19:19:10.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.189+0000 7fed13267740 1 -- 172.21.15.29:0/3887997960 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fed0009b2a0 con 0x5648c7aa29f0 2024-03-11T19:19:10.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.189+0000 7fecf1ffb640 1 -- 172.21.15.29:0/3887997960 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed000a3070 con 0x5648c7aa29f0 2024-03-11T19:19:10.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.189+0000 7fed13267740 10 client.27095 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:10.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.189+0000 7fed13267740 1 -- 172.21.15.29:0/3887997960 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5648c79d08c0 con 0x5648c7aa29f0 2024-03-11T19:19:10.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.189+0000 7fecf1ffb640 1 -- 172.21.15.29:0/3887997960 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fed0009b2a0 con 0x5648c7aa29f0 2024-03-11T19:19:10.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.190+0000 7fecf1ffb640 1 -- 172.21.15.29:0/3887997960 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fed00004c80 con 0x5648c7aa29f0 2024-03-11T19:19:10.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.190+0000 7fed13267740 10 client.27095 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:10.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.190+0000 7fed13267740 10 client.27095 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:19:10.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.190+0000 7fed13267740 20 client.27095 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:10.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.190+0000 7fed13267740 1 --2- 172.21.15.29:0/3887997960 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5648c7aacfe0 0x5648c7b16d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.190+0000 7fed13267740 4 client.27095 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:19:10.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.190+0000 7fed13267740 1 -- 172.21.15.29:0/3887997960 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5648c779b890 con 0x5648c7aacfe0 2024-03-11T19:19:10.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.191+0000 7fed0dc23640 1 --2- 172.21.15.29:0/3887997960 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5648c7aacfe0 0x5648c7b16d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.191+0000 7fed0dc23640 1 --2- 172.21.15.29:0/3887997960 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5648c7aacfe0 0x5648c7b16d70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=67d382ff3f3162f server_cookie=587d9d41dbcba9b in_seq=0 out_seq=0 2024-03-11T19:19:10.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.191+0000 7fecf1ffb640 10 client.27095 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:19:10.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.191+0000 7fecf1ffb640 1 -- 172.21.15.29:0/3887997960 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5648c779b890 con 0x5648c7aacfe0 2024-03-11T19:19:10.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.191+0000 7fecf1ffb640 10 client.27095 handle_command_reply: tid=0 2024-03-11T19:19:10.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.191+0000 7fed13267740 1 client.27095 shutdown 2024-03-11T19:19:10.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.192+0000 7fed13267740 1 -- 172.21.15.29:0/3887997960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa29f0 msgr2=0x5648c7ab84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.192+0000 7fed13267740 1 --2- 172.21.15.29:0/3887997960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa29f0 0x5648c7ab84a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fed000a2e10 tx=0x7fed0009b410 comp rx=0 tx=0).stop 2024-03-11T19:19:10.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.192+0000 7fed13267740 1 -- 172.21.15.29:0/3887997960 shutdown_connections 2024-03-11T19:19:10.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.192+0000 7fed13267740 1 --2- 172.21.15.29:0/3887997960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c7aa7e70 0x5648c7ab6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.192+0000 7fed13267740 1 --2- 172.21.15.29:0/3887997960 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5648c7aacfe0 0x5648c7b16d70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.192+0000 7fed13267740 1 --2- 172.21.15.29:0/3887997960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5648c7aa7480 0x5648c7ab6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.192+0000 7fed13267740 1 --2- 172.21.15.29:0/3887997960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5648c7aa29f0 0x5648c7ab84a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.192+0000 7fed13267740 1 -- 172.21.15.29:0/3887997960 >> 172.21.15.29:0/3887997960 conn(0x5648c7aa8a70 msgr2=0x5648c7aa9150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.192+0000 7fecf1ffb640 0 client.27095 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:19:10.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.193+0000 7fed13267740 1 -- 172.21.15.29:0/3887997960 shutdown_connections 2024-03-11T19:19:10.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.193+0000 7fed13267740 1 -- 172.21.15.29:0/3887997960 wait complete. 2024-03-11T19:19:10.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.193+0000 7fed13267740 20 client.27095 trim_cache size 0 max 16384 2024-03-11T19:19:10.199 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:19:10.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.194+0000 7fed0f426640 1 -- 172.21.15.29:0/3479826805 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fecf40a1c00 msgr2=0x7fecf40a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.194+0000 7fed0f426640 1 --2- 172.21.15.29:0/3479826805 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fecf40a1c00 0x7fecf40a40c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fed04014520 tx=0x7fed04014920 comp rx=0 tx=0).stop 2024-03-11T19:19:10.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.194+0000 7fed0f426640 1 -- 172.21.15.29:0/3479826805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed081076a0 msgr2=0x7fed0819d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.194+0000 7fed0f426640 1 --2- 172.21.15.29:0/3479826805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed081076a0 0x7fed0819d6b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fed0000ebc0 tx=0x7fed0000c6a0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.195+0000 7fed0f426640 1 -- 172.21.15.29:0/3479826805 shutdown_connections 2024-03-11T19:19:10.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.195+0000 7fed0f426640 1 --2- 172.21.15.29:0/3479826805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed08109250 0x7fed081a4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.195+0000 7fed0f426640 1 --2- 172.21.15.29:0/3479826805 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fecf40a1c00 0x7fecf40a40c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.195+0000 7fed0f426640 1 --2- 172.21.15.29:0/3479826805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed08108890 0x7fed0819dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.195+0000 7fed0f426640 1 --2- 172.21.15.29:0/3479826805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed081076a0 0x7fed0819d6b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.195+0000 7fed0f426640 1 -- 172.21.15.29:0/3479826805 >> 172.21.15.29:0/3479826805 conn(0x7fed08076650 msgr2=0x7fed0810bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.195+0000 7fed0f426640 1 -- 172.21.15.29:0/3479826805 shutdown_connections 2024-03-11T19:19:10.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.195+0000 7fed0f426640 1 -- 172.21.15.29:0/3479826805 wait complete. 2024-03-11T19:19:10.222 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:19:10.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:19:10.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.385+0000 7f9c8b96c640 1 -- 172.21.15.29:0/2465965706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c84102380 msgr2=0x7f9c84102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.385+0000 7f9c8b96c640 1 --2- 172.21.15.29:0/2465965706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c84102380 0x7f9c84102800 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f9c74009900 tx=0x7f9c7402bc30 comp rx=0 tx=0).stop 2024-03-11T19:19:10.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.385+0000 7f9c8b96c640 1 -- 172.21.15.29:0/2465965706 shutdown_connections 2024-03-11T19:19:10.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.385+0000 7f9c8b96c640 1 --2- 172.21.15.29:0/2465965706 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c84102d40 0x7f9c841095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.385+0000 7f9c8b96c640 1 --2- 172.21.15.29:0/2465965706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c84102380 0x7f9c84102800 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.385+0000 7f9c8b96c640 1 --2- 172.21.15.29:0/2465965706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c84101190 0x7f9c841015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.385+0000 7f9c8b96c640 1 -- 172.21.15.29:0/2465965706 >> 172.21.15.29:0/2465965706 conn(0x7f9c840fc720 msgr2=0x7f9c840feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.385+0000 7f9c8b96c640 1 -- 172.21.15.29:0/2465965706 shutdown_connections 2024-03-11T19:19:10.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.385+0000 7f9c8b96c640 1 -- 172.21.15.29:0/2465965706 wait complete. 2024-03-11T19:19:10.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.386+0000 7f9c8b96c640 1 Processor -- start 2024-03-11T19:19:10.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.386+0000 7f9c8b96c640 1 -- start start 2024-03-11T19:19:10.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.386+0000 7f9c8b96c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c84101190 0x7f9c84197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.386+0000 7f9c8b96c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c84102380 0x7f9c84197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.386+0000 7f9c8b96c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c84102d40 0x7f9c8419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.386+0000 7f9c8b96c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8419ec40 con 0x7f9c84102380 2024-03-11T19:19:10.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.386+0000 7f9c8b96c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8419edb0 con 0x7f9c84101190 2024-03-11T19:19:10.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.386+0000 7f9c8b96c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8419ef70 con 0x7f9c84102d40 2024-03-11T19:19:10.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.386+0000 7f9c88f07640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c84102380 0x7f9c84197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.386+0000 7f9c88f07640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c84102380 0x7f9c84197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53704/0 (socket says 172.21.15.29:53704) 2024-03-11T19:19:10.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.386+0000 7f9c88f07640 1 -- 172.21.15.29:0/1153663906 learned_addr learned my addr 172.21.15.29:0/1153663906 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.386+0000 7f9c89708640 1 --2- 172.21.15.29:0/1153663906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c84101190 0x7f9c84197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.386+0000 7f9c89f09640 1 --2- 172.21.15.29:0/1153663906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c84102d40 0x7f9c8419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.386+0000 7f9c88f07640 1 -- 172.21.15.29:0/1153663906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c84102d40 msgr2=0x7f9c8419e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.386+0000 7f9c88f07640 1 --2- 172.21.15.29:0/1153663906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c84102d40 0x7f9c8419e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.386+0000 7f9c88f07640 1 -- 172.21.15.29:0/1153663906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c84101190 msgr2=0x7f9c84197150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.386+0000 7f9c88f07640 1 --2- 172.21.15.29:0/1153663906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c84101190 0x7f9c84197150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.386+0000 7f9c88f07640 1 -- 172.21.15.29:0/1153663906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c74009560 con 0x7f9c84102380 2024-03-11T19:19:10.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.386+0000 7f9c89f09640 1 --2- 172.21.15.29:0/1153663906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c84102d40 0x7f9c8419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:19:10.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.387+0000 7f9c88f07640 1 --2- 172.21.15.29:0/1153663906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c84102380 0x7f9c84197690 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f9c74036040 tx=0x7f9c740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.387+0000 7f9c7a7fc640 1 -- 172.21.15.29:0/1153663906 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c740044e0 con 0x7f9c84102380 2024-03-11T19:19:10.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.387+0000 7f9c7a7fc640 1 -- 172.21.15.29:0/1153663906 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9c7402de90 con 0x7f9c84102380 2024-03-11T19:19:10.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.387+0000 7f9c7a7fc640 1 -- 172.21.15.29:0/1153663906 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c74005470 con 0x7f9c84102380 2024-03-11T19:19:10.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.387+0000 7f9c8b96c640 1 -- 172.21.15.29:0/1153663906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c8419f1f0 con 0x7f9c84102380 2024-03-11T19:19:10.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.387+0000 7f9c8b96c640 1 -- 172.21.15.29:0/1153663906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c8419f750 con 0x7f9c84102380 2024-03-11T19:19:10.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.390+0000 7f9c8e61c740 1 Processor -- start 2024-03-11T19:19:10.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.390+0000 7f9c7a7fc640 1 -- 172.21.15.29:0/1153663906 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9c74003800 con 0x7f9c84102380 2024-03-11T19:19:10.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.390+0000 7f9c8e61c740 1 -- start start 2024-03-11T19:19:10.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.390+0000 7f9c8e61c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5270 0x56084a0e5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.390+0000 7f9c8e61c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0e5c60 0x56084a0ea760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.390+0000 7f9c8e61c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0eadd0 0x56084a0ed260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.390+0000 7f9c8e61c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56084a0a1960 con 0x56084a0eadd0 2024-03-11T19:19:10.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.390+0000 7f9c8e61c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56084a055310 con 0x56084a0e5c60 2024-03-11T19:19:10.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.390+0000 7f9c8e61c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56084a052840 con 0x56084a0e5270 2024-03-11T19:19:10.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.391+0000 7f9c88f07640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0eadd0 0x56084a0ed260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.391+0000 7f9c88f07640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0eadd0 0x56084a0ed260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53706/0 (socket says 172.21.15.29:53706) 2024-03-11T19:19:10.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.391+0000 7f9c88f07640 1 -- 172.21.15.29:0/1067738514 learned_addr learned my addr 172.21.15.29:0/1067738514 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.391+0000 7f9c89708640 1 --2- 172.21.15.29:0/1067738514 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0e5c60 0x56084a0ea760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.391+0000 7f9c89f09640 1 --2- 172.21.15.29:0/1067738514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5270 0x56084a0e5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.391+0000 7f9c89708640 1 -- 172.21.15.29:0/1067738514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5270 msgr2=0x56084a0e5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.391+0000 7f9c89708640 1 --2- 172.21.15.29:0/1067738514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5270 0x56084a0e5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.391+0000 7f9c89708640 1 -- 172.21.15.29:0/1067738514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0eadd0 msgr2=0x56084a0ed260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.391+0000 7f9c89708640 1 --2- 172.21.15.29:0/1067738514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0eadd0 0x56084a0ed260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.391+0000 7f9c89708640 1 -- 172.21.15.29:0/1067738514 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56084a010060 con 0x56084a0e5c60 2024-03-11T19:19:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.391+0000 7f9c88f07640 1 --2- 172.21.15.29:0/1067738514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0eadd0 0x56084a0ed260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.391+0000 7f9c7a7fc640 1 --2- 172.21.15.29:0/1153663906 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9c700a1ab0 0x7f9c700a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.391+0000 7f9c89f09640 1 --2- 172.21.15.29:0/1153663906 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9c700a1ab0 0x7f9c700a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.391+0000 7f9c89708640 1 --2- 172.21.15.29:0/1067738514 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0e5c60 0x56084a0ea760 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9c841021e0 tx=0x7f9c7c0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92ae081674743c01 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.391+0000 7f9c7a7fc640 1 -- 172.21.15.29:0/1153663906 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9c740ef540 con 0x7f9c84102380 2024-03-11T19:19:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.391+0000 7f9c6f7fe640 1 -- 172.21.15.29:0/1067738514 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c01c480 con 0x56084a0e5c60 2024-03-11T19:19:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.391+0000 7f9c6f7fe640 1 -- 172.21.15.29:0/1067738514 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9c7c015070 con 0x56084a0e5c60 2024-03-11T19:19:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.391+0000 7f9c6f7fe640 1 -- 172.21.15.29:0/1067738514 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c01a5a0 con 0x56084a0e5c60 2024-03-11T19:19:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.392+0000 7f9c8e61c740 1 -- 172.21.15.29:0/1067738514 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0e5c60 msgr2=0x56084a0ea760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.392+0000 7f9c89f09640 1 --2- 172.21.15.29:0/1153663906 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9c700a1ab0 0x7f9c700a3f70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x56084a0e64c0 tx=0x7f9c80009700 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.392+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/1067738514 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0e5c60 0x56084a0ea760 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9c841021e0 tx=0x7f9c7c0107a0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.392+0000 7f9c8e61c740 1 -- 172.21.15.29:0/1067738514 shutdown_connections 2024-03-11T19:19:10.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.392+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/1067738514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0eadd0 0x56084a0ed260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.392+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/1067738514 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0e5c60 0x56084a0ea760 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.392+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/1067738514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5270 0x56084a0e5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.392+0000 7f9c8e61c740 1 -- 172.21.15.29:0/1067738514 >> 172.21.15.29:0/1067738514 conn(0x56084a0e07e0 msgr2=0x56084a0e0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.392+0000 7f9c8e61c740 1 -- 172.21.15.29:0/1067738514 shutdown_connections 2024-03-11T19:19:10.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.392+0000 7f9c8e61c740 1 -- 172.21.15.29:0/1067738514 wait complete. 2024-03-11T19:19:10.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.392+0000 7f9c8e61c740 1 -- start start 2024-03-11T19:19:10.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.392+0000 7f9c8e61c740 1 Processor -- start 2024-03-11T19:19:10.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.393+0000 7f9c8e61c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5270 0x56084a0f3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.393+0000 7f9c8e61c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5c60 0x56084a0f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.393+0000 7f9c8e61c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0eadd0 0x56084a0fad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.393+0000 7f9c8e61c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56084a04e020 con 0x56084a0e5270 2024-03-11T19:19:10.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.393+0000 7f9c8e61c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x560849d21430 con 0x56084a0eadd0 2024-03-11T19:19:10.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.393+0000 7f9c8e61c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560849cdde00 con 0x56084a0e5c60 2024-03-11T19:19:10.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.393+0000 7f9c8e61c740 10 client.0 _ll_register_callbacks cb 0x560849ff5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:10.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.393+0000 7f9c89708640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5270 0x56084a0f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.393+0000 7f9c89708640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5270 0x56084a0f3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53714/0 (socket says 172.21.15.29:53714) 2024-03-11T19:19:10.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.393+0000 7f9c89708640 1 -- 172.21.15.29:0/2293827808 learned_addr learned my addr 172.21.15.29:0/2293827808 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.393+0000 7f9c89f09640 1 --2- 172.21.15.29:0/2293827808 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0eadd0 0x56084a0fad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.393+0000 7f9c88f07640 1 --2- 172.21.15.29:0/2293827808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5c60 0x56084a0f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.393+0000 7f9c89708640 1 -- 172.21.15.29:0/2293827808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5c60 msgr2=0x56084a0f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.393+0000 7f9c89708640 1 --2- 172.21.15.29:0/2293827808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5c60 0x56084a0f3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.393+0000 7f9c89708640 1 -- 172.21.15.29:0/2293827808 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0eadd0 msgr2=0x56084a0fad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.393+0000 7f9c89708640 1 --2- 172.21.15.29:0/2293827808 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0eadd0 0x56084a0fad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.393+0000 7f9c89708640 1 -- 172.21.15.29:0/2293827808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7c00b860 con 0x56084a0e5270 2024-03-11T19:19:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.393+0000 7f9c89708640 1 --2- 172.21.15.29:0/2293827808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5270 0x56084a0f3810 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c011010 tx=0x7f9c7c0052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45ca5809b78b9d6e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.393+0000 7f9c6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:19:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.394+0000 7f9c6cff9640 1 -- 172.21.15.29:0/2293827808 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c0048c0 con 0x56084a0e5270 2024-03-11T19:19:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.394+0000 7f9c6cff9640 1 -- 172.21.15.29:0/2293827808 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9c7c017070 con 0x56084a0e5270 2024-03-11T19:19:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.394+0000 7f9c8e61c740 1 -- 172.21.15.29:0/2293827808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560849d4d7e0 con 0x56084a0e5270 2024-03-11T19:19:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.394+0000 7f9c6cff9640 1 -- 172.21.15.29:0/2293827808 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c7c004a20 con 0x56084a0e5270 2024-03-11T19:19:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.394+0000 7f9c8e61c740 10 client.17613 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.394+0000 7f9c8e61c740 1 -- 172.21.15.29:0/2293827808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56084a04c6a0 con 0x56084a0e5270 2024-03-11T19:19:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.394+0000 7f9c6cff9640 1 -- 172.21.15.29:0/2293827808 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9c7c034020 con 0x56084a0e5270 2024-03-11T19:19:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.394+0000 7f9c6cff9640 1 -- 172.21.15.29:0/2293827808 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9c7c024070 con 0x56084a0e5270 2024-03-11T19:19:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.394+0000 7f9c8e61c740 10 client.17613 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.394+0000 7f9c8e61c740 10 client.17613 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:19:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.394+0000 7f9c8e61c740 20 client.17613 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:10.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.394+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/2293827808 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56084a0ff930 0x56084a101d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.394+0000 7f9c8e61c740 4 client.17613 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:19:10.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.394+0000 7f9c8e61c740 1 -- 172.21.15.29:0/2293827808 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560849d8a720 con 0x56084a0ff930 2024-03-11T19:19:10.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.395+0000 7f9c88f07640 1 --2- 172.21.15.29:0/2293827808 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56084a0ff930 0x56084a101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.395+0000 7f9c6cff9640 10 client.17613 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:19:10.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.395+0000 7f9c88f07640 1 --2- 172.21.15.29:0/2293827808 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56084a0ff930 0x56084a101d20 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=aa737356d967982e server_cookie=7d8b6892a0d13c84 in_seq=0 out_seq=0 2024-03-11T19:19:10.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.395+0000 7f9c6cff9640 1 -- 172.21.15.29:0/2293827808 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560849d8a720 con 0x56084a0ff930 2024-03-11T19:19:10.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.395+0000 7f9c6cff9640 10 client.17613 handle_command_reply: tid=0 2024-03-11T19:19:10.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.395+0000 7f9c8e61c740 1 client.17613 shutdown 2024-03-11T19:19:10.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.396+0000 7f9c8e61c740 1 -- 172.21.15.29:0/2293827808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5270 msgr2=0x56084a0f3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.396+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/2293827808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5270 0x56084a0f3810 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c011010 tx=0x7f9c7c0052f0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.396+0000 7f9c8e61c740 1 -- 172.21.15.29:0/2293827808 shutdown_connections 2024-03-11T19:19:10.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.396+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/2293827808 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56084a0ff930 0x56084a101d20 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.396+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/2293827808 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0eadd0 0x56084a0fad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.396+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/2293827808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5c60 0x56084a0f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.396+0000 7f9c6cff9640 0 client.17613 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:19:10.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.396+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/2293827808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5270 0x56084a0f3810 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.396+0000 7f9c8e61c740 1 -- 172.21.15.29:0/2293827808 >> 172.21.15.29:0/2293827808 conn(0x56084a0e07e0 msgr2=0x56084a0dfb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.396+0000 7f9c8e61c740 1 -- 172.21.15.29:0/2293827808 shutdown_connections 2024-03-11T19:19:10.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.396+0000 7f9c8e61c740 1 -- 172.21.15.29:0/2293827808 wait complete. 2024-03-11T19:19:10.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.396+0000 7f9c8e61c740 20 client.17613 trim_cache size 0 max 16384 2024-03-11T19:19:10.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.407+0000 7f9c8e61c740 1 Processor -- start 2024-03-11T19:19:10.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c8e61c740 1 -- start start 2024-03-11T19:19:10.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c8e61c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5270 0x56084a0df2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c8e61c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5c60 0x56084a0fad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c8e61c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0eadd0 0x56084a0df830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c8e61c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x560849fd3130 con 0x56084a0e5c60 2024-03-11T19:19:10.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c8e61c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56084a0d2a40 con 0x56084a0eadd0 2024-03-11T19:19:10.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c8e61c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56084a0dc370 con 0x56084a0e5270 2024-03-11T19:19:10.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c88f07640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5c60 0x56084a0fad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c88f07640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5c60 0x56084a0fad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53724/0 (socket says 172.21.15.29:53724) 2024-03-11T19:19:10.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c88f07640 1 -- 172.21.15.29:0/2985291931 learned_addr learned my addr 172.21.15.29:0/2985291931 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c89708640 1 --2- 172.21.15.29:0/2985291931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5270 0x56084a0df2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c89f09640 1 --2- 172.21.15.29:0/2985291931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0eadd0 0x56084a0df830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c88f07640 1 -- 172.21.15.29:0/2985291931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5270 msgr2=0x56084a0df2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c88f07640 1 --2- 172.21.15.29:0/2985291931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5270 0x56084a0df2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c88f07640 1 -- 172.21.15.29:0/2985291931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0eadd0 msgr2=0x56084a0df830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c88f07640 1 --2- 172.21.15.29:0/2985291931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0eadd0 0x56084a0df830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c88f07640 1 -- 172.21.15.29:0/2985291931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560849fe8c20 con 0x56084a0e5c60 2024-03-11T19:19:10.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c88f07640 1 --2- 172.21.15.29:0/2985291931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5c60 0x56084a0fad50 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f9c74000c00 tx=0x7f9c740a7040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e1ff0decdb4191 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c6effd640 1 -- 172.21.15.29:0/2985291931 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c740bc320 con 0x56084a0e5c60 2024-03-11T19:19:10.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c6effd640 1 -- 172.21.15.29:0/2985291931 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9c740bc090 con 0x56084a0e5c60 2024-03-11T19:19:10.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.408+0000 7f9c6effd640 1 -- 172.21.15.29:0/2985291931 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c740c06a0 con 0x56084a0e5c60 2024-03-11T19:19:10.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.409+0000 7f9c8e61c740 1 -- 172.21.15.29:0/2985291931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5c60 msgr2=0x56084a0fad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.409+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/2985291931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5c60 0x56084a0fad50 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f9c74000c00 tx=0x7f9c740a7040 comp rx=0 tx=0).stop 2024-03-11T19:19:10.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.409+0000 7f9c8e61c740 1 -- 172.21.15.29:0/2985291931 shutdown_connections 2024-03-11T19:19:10.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.409+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/2985291931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0eadd0 0x56084a0df830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.409+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/2985291931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5c60 0x56084a0fad50 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.409+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/2985291931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5270 0x56084a0df2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.409+0000 7f9c8e61c740 1 -- 172.21.15.29:0/2985291931 >> 172.21.15.29:0/2985291931 conn(0x56084a0e07e0 msgr2=0x56084a0e0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.409+0000 7f9c8e61c740 1 -- 172.21.15.29:0/2985291931 shutdown_connections 2024-03-11T19:19:10.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.409+0000 7f9c8e61c740 1 -- 172.21.15.29:0/2985291931 wait complete. 2024-03-11T19:19:10.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.409+0000 7f9c8e61c740 1 -- start start 2024-03-11T19:19:10.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.409+0000 7f9c8e61c740 1 Processor -- start 2024-03-11T19:19:10.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c8e61c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5270 0x56084a0f8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c8e61c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5c60 0x56084a0f9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c8e61c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0eadd0 0x56084a0fa280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c8e61c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56084a0fa7c0 con 0x56084a0e5270 2024-03-11T19:19:10.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c8e61c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56084a0fa930 con 0x56084a0eadd0 2024-03-11T19:19:10.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c8e61c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56084a0faaf0 con 0x56084a0e5c60 2024-03-11T19:19:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c8e61c740 10 client.0 _ll_register_callbacks cb 0x7f9c7c00b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c89708640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5270 0x56084a0f8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c89708640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5270 0x56084a0f8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53736/0 (socket says 172.21.15.29:53736) 2024-03-11T19:19:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c89708640 1 -- 172.21.15.29:0/750348794 learned_addr learned my addr 172.21.15.29:0/750348794 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c89f09640 1 --2- 172.21.15.29:0/750348794 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0eadd0 0x56084a0fa280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c88f07640 1 --2- 172.21.15.29:0/750348794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5c60 0x56084a0f9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c89708640 1 -- 172.21.15.29:0/750348794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5c60 msgr2=0x56084a0f9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c89708640 1 --2- 172.21.15.29:0/750348794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5c60 0x56084a0f9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c89708640 1 -- 172.21.15.29:0/750348794 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0eadd0 msgr2=0x56084a0fa280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c89708640 1 --2- 172.21.15.29:0/750348794 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0eadd0 0x56084a0fa280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c89708640 1 -- 172.21.15.29:0/750348794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c8419f1f0 con 0x56084a0e5270 2024-03-11T19:19:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c88f07640 1 --2- 172.21.15.29:0/750348794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5c60 0x56084a0f9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:19:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c89708640 1 --2- 172.21.15.29:0/750348794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5270 0x56084a0f8d30 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c0107e0 tx=0x7f9c7c00ba80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c972328c2ae39f97 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:19:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c6cff9640 1 -- 172.21.15.29:0/750348794 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c01cd10 con 0x56084a0e5270 2024-03-11T19:19:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c6cff9640 1 -- 172.21.15.29:0/750348794 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9c7c017040 con 0x56084a0e5270 2024-03-11T19:19:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.410+0000 7f9c8e61c740 1 -- 172.21.15.29:0/750348794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c7c015070 con 0x56084a0e5270 2024-03-11T19:19:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.411+0000 7f9c6cff9640 1 -- 172.21.15.29:0/750348794 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c7c005ed0 con 0x56084a0e5270 2024-03-11T19:19:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.411+0000 7f9c8e61c740 10 client.17619 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.411+0000 7f9c8e61c740 1 -- 172.21.15.29:0/750348794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56084a00e8c0 con 0x56084a0e5270 2024-03-11T19:19:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.411+0000 7f9c6cff9640 1 -- 172.21.15.29:0/750348794 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9c7c015070 con 0x56084a0e5270 2024-03-11T19:19:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.411+0000 7f9c6cff9640 1 -- 172.21.15.29:0/750348794 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9c7c024040 con 0x56084a0e5270 2024-03-11T19:19:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.411+0000 7f9c8e61c740 10 client.17619 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.411+0000 7f9c8e61c740 10 client.17619 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:19:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.411+0000 7f9c8e61c740 20 client.17619 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.411+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/750348794 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56084a156e40 0x56084a159260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.412+0000 7f9c8e61c740 4 client.17619 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:19:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.412+0000 7f9c8e61c740 1 -- 172.21.15.29:0/750348794 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560849dd9890 con 0x56084a156e40 2024-03-11T19:19:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.412+0000 7f9c88f07640 1 --2- 172.21.15.29:0/750348794 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56084a156e40 0x56084a159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.412+0000 7f9c6cff9640 10 client.17619 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:19:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.412+0000 7f9c88f07640 1 --2- 172.21.15.29:0/750348794 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56084a156e40 0x56084a159260 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=938f4d6afc0c84d5 server_cookie=872b4dfdbb6f99f8 in_seq=0 out_seq=0 2024-03-11T19:19:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.412+0000 7f9c6cff9640 1 -- 172.21.15.29:0/750348794 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560849dd9890 con 0x56084a156e40 2024-03-11T19:19:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.412+0000 7f9c6cff9640 10 client.17619 handle_command_reply: tid=0 2024-03-11T19:19:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.412+0000 7f9c8e61c740 1 client.17619 shutdown 2024-03-11T19:19:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8e61c740 1 -- 172.21.15.29:0/750348794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5270 msgr2=0x56084a0f8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/750348794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5270 0x56084a0f8d30 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c0107e0 tx=0x7f9c7c00ba80 comp rx=0 tx=0).stop 2024-03-11T19:19:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8e61c740 1 -- 172.21.15.29:0/750348794 shutdown_connections 2024-03-11T19:19:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/750348794 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56084a156e40 0x56084a159260 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/750348794 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56084a0eadd0 0x56084a0fa280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c6cff9640 0 client.17619 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:19:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/750348794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56084a0e5c60 0x56084a0f9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8e61c740 1 --2- 172.21.15.29:0/750348794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56084a0e5270 0x56084a0f8d30 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8e61c740 1 -- 172.21.15.29:0/750348794 >> 172.21.15.29:0/750348794 conn(0x56084a0e07e0 msgr2=0x56084a0e3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8e61c740 1 -- 172.21.15.29:0/750348794 shutdown_connections 2024-03-11T19:19:10.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8e61c740 1 -- 172.21.15.29:0/750348794 wait complete. 2024-03-11T19:19:10.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8e61c740 20 client.17619 trim_cache size 0 max 16384 2024-03-11T19:19:10.423 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:19:10.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8b96c640 1 -- 172.21.15.29:0/1153663906 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9c700a1ab0 msgr2=0x7f9c700a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8b96c640 1 --2- 172.21.15.29:0/1153663906 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9c700a1ab0 0x7f9c700a3f70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x56084a0e64c0 tx=0x7f9c80009700 comp rx=0 tx=0).stop 2024-03-11T19:19:10.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8b96c640 1 -- 172.21.15.29:0/1153663906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c84102380 msgr2=0x7f9c84197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8b96c640 1 --2- 172.21.15.29:0/1153663906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c84102380 0x7f9c84197690 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f9c74036040 tx=0x7f9c740042f0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8b96c640 1 -- 172.21.15.29:0/1153663906 shutdown_connections 2024-03-11T19:19:10.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8b96c640 1 --2- 172.21.15.29:0/1153663906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c84102d40 0x7f9c8419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8b96c640 1 --2- 172.21.15.29:0/1153663906 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9c700a1ab0 0x7f9c700a3f70 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8b96c640 1 --2- 172.21.15.29:0/1153663906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c84102380 0x7f9c84197690 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8b96c640 1 --2- 172.21.15.29:0/1153663906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c84101190 0x7f9c84197150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.413+0000 7f9c8b96c640 1 -- 172.21.15.29:0/1153663906 >> 172.21.15.29:0/1153663906 conn(0x7f9c840fc720 msgr2=0x7f9c84105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.414+0000 7f9c8b96c640 1 -- 172.21.15.29:0/1153663906 shutdown_connections 2024-03-11T19:19:10.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.414+0000 7f9c8b96c640 1 -- 172.21.15.29:0/1153663906 wait complete. 2024-03-11T19:19:10.427 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:19:10.427 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:19:10.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.591+0000 7fab18815640 1 -- 172.21.15.29:0/4174621125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab14101c40 msgr2=0x7fab141084d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.591+0000 7fab18815640 1 --2- 172.21.15.29:0/4174621125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab14101c40 0x7fab141084d0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fab04009ca0 tx=0x7fab0402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.591+0000 7fab18815640 1 -- 172.21.15.29:0/4174621125 shutdown_connections 2024-03-11T19:19:10.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.591+0000 7fab18815640 1 --2- 172.21.15.29:0/4174621125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab14101c40 0x7fab141084d0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.591+0000 7fab18815640 1 --2- 172.21.15.29:0/4174621125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab14101280 0x7fab14101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.591+0000 7fab18815640 1 --2- 172.21.15.29:0/4174621125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab14100090 0x7fab141004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.591+0000 7fab18815640 1 -- 172.21.15.29:0/4174621125 >> 172.21.15.29:0/4174621125 conn(0x7fab140fb620 msgr2=0x7fab140fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.591+0000 7fab18815640 1 -- 172.21.15.29:0/4174621125 shutdown_connections 2024-03-11T19:19:10.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.591+0000 7fab18815640 1 -- 172.21.15.29:0/4174621125 wait complete. 2024-03-11T19:19:10.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.592+0000 7fab18815640 1 Processor -- start 2024-03-11T19:19:10.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.593+0000 7fab18815640 1 -- start start 2024-03-11T19:19:10.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.593+0000 7fab18815640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab14100090 0x7fab14194f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.593+0000 7fab18815640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab14101280 0x7fab141954a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.593+0000 7fab18815640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab14101c40 0x7fab1419c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.593+0000 7fab18815640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1419ca50 con 0x7fab14100090 2024-03-11T19:19:10.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.593+0000 7fab18815640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1419cbc0 con 0x7fab14101280 2024-03-11T19:19:10.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.593+0000 7fab18815640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1419cd80 con 0x7fab14101c40 2024-03-11T19:19:10.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.593+0000 7fab11d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab14101280 0x7fab141954a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.593+0000 7fab12d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab14101c40 0x7fab1419c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.593+0000 7fab11d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab14101280 0x7fab141954a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55948/0 (socket says 172.21.15.29:55948) 2024-03-11T19:19:10.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.593+0000 7fab12d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab14101c40 0x7fab1419c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48058/0 (socket says 172.21.15.29:48058) 2024-03-11T19:19:10.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.593+0000 7fab11d9b640 1 -- 172.21.15.29:0/232211547 learned_addr learned my addr 172.21.15.29:0/232211547 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.593+0000 7fab1259c640 1 --2- 172.21.15.29:0/232211547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab14100090 0x7fab14194f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.594+0000 7fab11d9b640 1 -- 172.21.15.29:0/232211547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab14101c40 msgr2=0x7fab1419c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.594+0000 7fab11d9b640 1 --2- 172.21.15.29:0/232211547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab14101c40 0x7fab1419c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.594+0000 7fab11d9b640 1 -- 172.21.15.29:0/232211547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab14100090 msgr2=0x7fab14194f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.594+0000 7fab11d9b640 1 --2- 172.21.15.29:0/232211547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab14100090 0x7fab14194f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.594+0000 7fab11d9b640 1 -- 172.21.15.29:0/232211547 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab04009900 con 0x7fab14101280 2024-03-11T19:19:10.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.594+0000 7fab1259c640 1 --2- 172.21.15.29:0/232211547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab14100090 0x7fab14194f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:10.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.594+0000 7fab11d9b640 1 --2- 172.21.15.29:0/232211547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab14101280 0x7fab141954a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fab0800d950 tx=0x7fab0800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.594+0000 7fab037fe640 1 -- 172.21.15.29:0/232211547 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab08004020 con 0x7fab14101280 2024-03-11T19:19:10.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.594+0000 7fab037fe640 1 -- 172.21.15.29:0/232211547 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fab080099b0 con 0x7fab14101280 2024-03-11T19:19:10.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.594+0000 7fab18815640 1 -- 172.21.15.29:0/232211547 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1419d060 con 0x7fab14101280 2024-03-11T19:19:10.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.594+0000 7fab037fe640 1 -- 172.21.15.29:0/232211547 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab080047a0 con 0x7fab14101280 2024-03-11T19:19:10.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.594+0000 7fab18815640 1 -- 172.21.15.29:0/232211547 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab1419d570 con 0x7fab14101280 2024-03-11T19:19:10.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.597+0000 7fab037fe640 1 -- 172.21.15.29:0/232211547 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fab08004900 con 0x7fab14101280 2024-03-11T19:19:10.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.597+0000 7fab1b3ee740 1 Processor -- start 2024-03-11T19:19:10.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.597+0000 7fab1b3ee740 1 -- start start 2024-03-11T19:19:10.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.597+0000 7fab1b3ee740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9bc270 0x5581dd9bc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.597+0000 7fab1b3ee740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9bcc60 0x5581dd9c1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.597+0000 7fab1b3ee740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9c1dd0 0x5581dd9c4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.597+0000 7fab1b3ee740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581dd978960 con 0x5581dd9c1dd0 2024-03-11T19:19:10.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.597+0000 7fab1b3ee740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581dd92c310 con 0x5581dd9bcc60 2024-03-11T19:19:10.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.597+0000 7fab1b3ee740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581dd929840 con 0x5581dd9bc270 2024-03-11T19:19:10.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.598+0000 7fab037fe640 1 --2- 172.21.15.29:0/232211547 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faafc0a5e80 0x7faafc0a8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.598+0000 7fab12d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9bc270 0x5581dd9bc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.598+0000 7fab11d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9c1dd0 0x5581dd9c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.598+0000 7fab12d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9bc270 0x5581dd9bc690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48068/0 (socket says 172.21.15.29:48068) 2024-03-11T19:19:10.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.598+0000 7fab1259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9bcc60 0x5581dd9c1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.598+0000 7fab12d9d640 1 -- 172.21.15.29:0/3162626264 learned_addr learned my addr 172.21.15.29:0/3162626264 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.598+0000 7fab12d9d640 1 --2- 172.21.15.29:0/232211547 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faafc0a5e80 0x7faafc0a8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.598+0000 7fab12d9d640 1 -- 172.21.15.29:0/3162626264 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9bcc60 msgr2=0x5581dd9c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.598+0000 7fab12d9d640 1 --2- 172.21.15.29:0/3162626264 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9bcc60 0x5581dd9c1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.598+0000 7fab12d9d640 1 -- 172.21.15.29:0/3162626264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9c1dd0 msgr2=0x5581dd9c4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.598+0000 7fab12d9d640 1 --2- 172.21.15.29:0/3162626264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9c1dd0 0x5581dd9c4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.598+0000 7fab12d9d640 1 -- 172.21.15.29:0/3162626264 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581dd8e7060 con 0x5581dd9bc270 2024-03-11T19:19:10.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.598+0000 7fab1259c640 1 --2- 172.21.15.29:0/3162626264 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9bcc60 0x5581dd9c1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:10.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.598+0000 7fab12d9d640 1 --2- 172.21.15.29:0/3162626264 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9bc270 0x5581dd9bc690 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fab040098d0 tx=0x7fab040345f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf93e6620fa7d1ad server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.599+0000 7fab12d9d640 1 --2- 172.21.15.29:0/232211547 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faafc0a5e80 0x7faafc0a8340 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fab04048000 tx=0x7fab04049000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.599+0000 7fab037fe640 1 -- 172.21.15.29:0/232211547 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fab0809a3b0 con 0x7fab14101280 2024-03-11T19:19:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.599+0000 7fab00ff9640 1 -- 172.21.15.29:0/3162626264 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab04032df0 con 0x5581dd9bc270 2024-03-11T19:19:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.599+0000 7fab00ff9640 1 -- 172.21.15.29:0/3162626264 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fab04003e20 con 0x5581dd9bc270 2024-03-11T19:19:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.599+0000 7fab00ff9640 1 -- 172.21.15.29:0/3162626264 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0403d3f0 con 0x5581dd9bc270 2024-03-11T19:19:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.599+0000 7fab1b3ee740 1 -- 172.21.15.29:0/3162626264 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9bc270 msgr2=0x5581dd9bc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.599+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/3162626264 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9bc270 0x5581dd9bc690 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fab040098d0 tx=0x7fab040345f0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.599+0000 7fab1b3ee740 1 -- 172.21.15.29:0/3162626264 shutdown_connections 2024-03-11T19:19:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.599+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/3162626264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9c1dd0 0x5581dd9c4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.599+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/3162626264 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9bcc60 0x5581dd9c1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.599+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/3162626264 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9bc270 0x5581dd9bc690 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.599+0000 7fab1b3ee740 1 -- 172.21.15.29:0/3162626264 >> 172.21.15.29:0/3162626264 conn(0x5581dd9b77e0 msgr2=0x5581dd9b7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.600+0000 7fab1b3ee740 1 -- 172.21.15.29:0/3162626264 shutdown_connections 2024-03-11T19:19:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.600+0000 7fab1b3ee740 1 -- 172.21.15.29:0/3162626264 wait complete. 2024-03-11T19:19:10.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.600+0000 7fab1b3ee740 1 -- start start 2024-03-11T19:19:10.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.600+0000 7fab1b3ee740 1 Processor -- start 2024-03-11T19:19:10.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.600+0000 7fab1b3ee740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9bc270 0x5581dd9c6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7fab1b3ee740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bcc60 0x5581dd9c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7fab1b3ee740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9c1dd0 0x5581dd9cda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7fab11d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bcc60 0x5581dd9c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7fab1b3ee740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581dd925020 con 0x5581dd9bcc60 2024-03-11T19:19:10.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7fab1b3ee740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581dd5f8430 con 0x5581dd9bc270 2024-03-11T19:19:10.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7fab1b3ee740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581dd5b4e00 con 0x5581dd9c1dd0 2024-03-11T19:19:10.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7fab11d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bcc60 0x5581dd9c69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53764/0 (socket says 172.21.15.29:53764) 2024-03-11T19:19:10.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7fab11d9b640 1 -- 172.21.15.29:0/2367620179 learned_addr learned my addr 172.21.15.29:0/2367620179 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7fab1259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9bc270 0x5581dd9c6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7fab1b3ee740 10 client.0 _ll_register_callbacks cb 0x5581dd8cccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:10.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7fab12d9d640 1 --2- 172.21.15.29:0/2367620179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9c1dd0 0x5581dd9cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7fab11d9b640 1 -- 172.21.15.29:0/2367620179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9c1dd0 msgr2=0x5581dd9cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7fab11d9b640 1 --2- 172.21.15.29:0/2367620179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9c1dd0 0x5581dd9cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7fab11d9b640 1 -- 172.21.15.29:0/2367620179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9bc270 msgr2=0x5581dd9c6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7fab11d9b640 1 --2- 172.21.15.29:0/2367620179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9bc270 0x5581dd9c6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7fab11d9b640 1 -- 172.21.15.29:0/2367620179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab14108dd0 con 0x5581dd9bcc60 2024-03-11T19:19:10.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7fab11d9b640 1 --2- 172.21.15.29:0/2367620179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bcc60 0x5581dd9c69c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fab080a1f20 tx=0x7fab08099610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3871f720a2a38e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7faaf9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:19:10.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7faaf9ffb640 1 -- 172.21.15.29:0/2367620179 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab080a2450 con 0x5581dd9bcc60 2024-03-11T19:19:10.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7faaf9ffb640 1 -- 172.21.15.29:0/2367620179 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fab08096320 con 0x5581dd9bcc60 2024-03-11T19:19:10.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.601+0000 7fab1b3ee740 1 -- 172.21.15.29:0/2367620179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5581dd6247e0 con 0x5581dd9bcc60 2024-03-11T19:19:10.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.602+0000 7faaf9ffb640 1 -- 172.21.15.29:0/2367620179 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab080964e0 con 0x5581dd9bcc60 2024-03-11T19:19:10.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.602+0000 7fab1b3ee740 10 client.17631 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:10.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.602+0000 7fab1b3ee740 1 -- 172.21.15.29:0/2367620179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5581dd9236a0 con 0x5581dd9bcc60 2024-03-11T19:19:10.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.602+0000 7faaf9ffb640 1 -- 172.21.15.29:0/2367620179 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5581dd6247e0 con 0x5581dd9bcc60 2024-03-11T19:19:10.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.602+0000 7faaf9ffb640 1 -- 172.21.15.29:0/2367620179 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fab080992c0 con 0x5581dd9bcc60 2024-03-11T19:19:10.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.602+0000 7fab1b3ee740 10 client.17631 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:10.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.602+0000 7fab1b3ee740 10 client.17631 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:19:10.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.602+0000 7fab1b3ee740 20 client.17631 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:10.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.602+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/2367620179 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5581dd9d25d0 0x5581dd9d49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.602+0000 7fab1b3ee740 4 client.17631 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:19:10.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.602+0000 7fab1b3ee740 1 -- 172.21.15.29:0/2367620179 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5581dd661720 con 0x5581dd9d25d0 2024-03-11T19:19:10.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.602+0000 7fab1259c640 1 --2- 172.21.15.29:0/2367620179 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5581dd9d25d0 0x5581dd9d49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.603+0000 7fab1259c640 1 --2- 172.21.15.29:0/2367620179 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5581dd9d25d0 0x5581dd9d49f0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8f5c2fb81d1351c8 server_cookie=919f751cb107d0a7 in_seq=0 out_seq=0 2024-03-11T19:19:10.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.603+0000 7faaf9ffb640 10 client.17631 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:19:10.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.603+0000 7faaf9ffb640 1 -- 172.21.15.29:0/2367620179 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5581dd661720 con 0x5581dd9d25d0 2024-03-11T19:19:10.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.603+0000 7faaf9ffb640 10 client.17631 handle_command_reply: tid=0 2024-03-11T19:19:10.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.603+0000 7fab1b3ee740 1 client.17631 shutdown 2024-03-11T19:19:10.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.603+0000 7fab1b3ee740 1 -- 172.21.15.29:0/2367620179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bcc60 msgr2=0x5581dd9c69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.603+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/2367620179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bcc60 0x5581dd9c69c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fab080a1f20 tx=0x7fab08099610 comp rx=0 tx=0).stop 2024-03-11T19:19:10.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.604+0000 7fab1b3ee740 1 -- 172.21.15.29:0/2367620179 shutdown_connections 2024-03-11T19:19:10.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.604+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/2367620179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9c1dd0 0x5581dd9cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.604+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/2367620179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bcc60 0x5581dd9c69c0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.604+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/2367620179 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5581dd9d25d0 0x5581dd9d49f0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.604+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/2367620179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9bc270 0x5581dd9c6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.604+0000 7faaf9ffb640 0 client.17631 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:19:10.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.604+0000 7fab1b3ee740 1 -- 172.21.15.29:0/2367620179 >> 172.21.15.29:0/2367620179 conn(0x5581dd9b77e0 msgr2=0x5581dd9b6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.604+0000 7fab1b3ee740 1 -- 172.21.15.29:0/2367620179 shutdown_connections 2024-03-11T19:19:10.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.604+0000 7fab1b3ee740 1 -- 172.21.15.29:0/2367620179 wait complete. 2024-03-11T19:19:10.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.604+0000 7fab1b3ee740 20 client.17631 trim_cache size 0 max 16384 2024-03-11T19:19:10.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab1b3ee740 1 Processor -- start 2024-03-11T19:19:10.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab1b3ee740 1 -- start start 2024-03-11T19:19:10.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab1b3ee740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bc270 0x5581dd9c4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab1b3ee740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9bcc60 0x5581dd9cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab1b3ee740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9c1dd0 0x5581dd9b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab1b3ee740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581dd8aa130 con 0x5581dd9bc270 2024-03-11T19:19:10.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab1b3ee740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581dd9a9a40 con 0x5581dd9c1dd0 2024-03-11T19:19:10.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab1b3ee740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581dd9c5f00 con 0x5581dd9bcc60 2024-03-11T19:19:10.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab1259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bc270 0x5581dd9c4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab1259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bc270 0x5581dd9c4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53778/0 (socket says 172.21.15.29:53778) 2024-03-11T19:19:10.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab1259c640 1 -- 172.21.15.29:0/1979853824 learned_addr learned my addr 172.21.15.29:0/1979853824 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab11d9b640 1 --2- 172.21.15.29:0/1979853824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9bcc60 0x5581dd9cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab12d9d640 1 --2- 172.21.15.29:0/1979853824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9c1dd0 0x5581dd9b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab1259c640 1 -- 172.21.15.29:0/1979853824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9bcc60 msgr2=0x5581dd9cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab1259c640 1 --2- 172.21.15.29:0/1979853824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9bcc60 0x5581dd9cd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab1259c640 1 -- 172.21.15.29:0/1979853824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9c1dd0 msgr2=0x5581dd9b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab1259c640 1 --2- 172.21.15.29:0/1979853824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9c1dd0 0x5581dd9b2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab1259c640 1 -- 172.21.15.29:0/1979853824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581dd8bfc20 con 0x5581dd9bc270 2024-03-11T19:19:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab11d9b640 1 --2- 172.21.15.29:0/1979853824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9bcc60 0x5581dd9cd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab12d9d640 1 --2- 172.21.15.29:0/1979853824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9c1dd0 0x5581dd9b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:19:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.616+0000 7fab1259c640 1 --2- 172.21.15.29:0/1979853824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bc270 0x5581dd9c4f40 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x5581dd9bd6d0 tx=0x7fab0c00b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c073cd42638570f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.617+0000 7faafbfff640 1 -- 172.21.15.29:0/1979853824 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0c016030 con 0x5581dd9bc270 2024-03-11T19:19:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.617+0000 7faafbfff640 1 -- 172.21.15.29:0/1979853824 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fab0c017040 con 0x5581dd9bc270 2024-03-11T19:19:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.617+0000 7faafbfff640 1 -- 172.21.15.29:0/1979853824 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0c003c20 con 0x5581dd9bc270 2024-03-11T19:19:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.617+0000 7fab1b3ee740 1 -- 172.21.15.29:0/1979853824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bc270 msgr2=0x5581dd9c4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.617+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/1979853824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bc270 0x5581dd9c4f40 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x5581dd9bd6d0 tx=0x7fab0c00b390 comp rx=0 tx=0).stop 2024-03-11T19:19:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.617+0000 7fab1b3ee740 1 -- 172.21.15.29:0/1979853824 shutdown_connections 2024-03-11T19:19:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.617+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/1979853824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9c1dd0 0x5581dd9b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.617+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/1979853824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9bcc60 0x5581dd9cd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.617+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/1979853824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bc270 0x5581dd9c4f40 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.617+0000 7fab1b3ee740 1 -- 172.21.15.29:0/1979853824 >> 172.21.15.29:0/1979853824 conn(0x5581dd9b77e0 msgr2=0x5581dd9bd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.617+0000 7fab1b3ee740 1 -- 172.21.15.29:0/1979853824 shutdown_connections 2024-03-11T19:19:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.617+0000 7fab1b3ee740 1 -- 172.21.15.29:0/1979853824 wait complete. 2024-03-11T19:19:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab1b3ee740 1 -- start start 2024-03-11T19:19:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab1b3ee740 1 Processor -- start 2024-03-11T19:19:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab1b3ee740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bc270 0x5581dd9cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab1b3ee740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9bcc60 0x5581dd9cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab1b3ee740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9c1dd0 0x5581dd9cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab1b3ee740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581dd9ccac0 con 0x5581dd9bc270 2024-03-11T19:19:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab1b3ee740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581dd9ccc30 con 0x5581dd9bcc60 2024-03-11T19:19:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab1b3ee740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581dda28b00 con 0x5581dd9c1dd0 2024-03-11T19:19:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab1b3ee740 10 client.0 _ll_register_callbacks cb 0x7fab04009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab1259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bc270 0x5581dd9cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab1259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bc270 0x5581dd9cd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53792/0 (socket says 172.21.15.29:53792) 2024-03-11T19:19:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab1259c640 1 -- 172.21.15.29:0/927625943 learned_addr learned my addr 172.21.15.29:0/927625943 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab11d9b640 1 --2- 172.21.15.29:0/927625943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9bcc60 0x5581dd9cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab12d9d640 1 --2- 172.21.15.29:0/927625943 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9c1dd0 0x5581dd9cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab1259c640 1 -- 172.21.15.29:0/927625943 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9c1dd0 msgr2=0x5581dd9cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab1259c640 1 --2- 172.21.15.29:0/927625943 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9c1dd0 0x5581dd9cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab1259c640 1 -- 172.21.15.29:0/927625943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9bcc60 msgr2=0x5581dd9cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab1259c640 1 --2- 172.21.15.29:0/927625943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9bcc60 0x5581dd9cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab1259c640 1 -- 172.21.15.29:0/927625943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5581dd9236a0 con 0x5581dd9bc270 2024-03-11T19:19:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7fab1259c640 1 --2- 172.21.15.29:0/927625943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bc270 0x5581dd9cd870 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fab0c0042d0 tx=0x7fab0c005220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=751478bbd1b4daaf server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.618+0000 7faaf9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:19:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.619+0000 7faaf9ffb640 1 -- 172.21.15.29:0/927625943 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0c016070 con 0x5581dd9bc270 2024-03-11T19:19:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.619+0000 7faaf9ffb640 1 -- 172.21.15.29:0/927625943 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fab0c005620 con 0x5581dd9bc270 2024-03-11T19:19:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.619+0000 7fab1b3ee740 1 -- 172.21.15.29:0/927625943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fab04003e20 con 0x5581dd9bc270 2024-03-11T19:19:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.619+0000 7faaf9ffb640 1 -- 172.21.15.29:0/927625943 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab0c003b60 con 0x5581dd9bc270 2024-03-11T19:19:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.619+0000 7fab1b3ee740 10 client.17637 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.619+0000 7fab1b3ee740 1 -- 172.21.15.29:0/927625943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7fab0c0043e0 con 0x5581dd9bc270 2024-03-11T19:19:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.619+0000 7faaf9ffb640 1 -- 172.21.15.29:0/927625943 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fab0c037020 con 0x5581dd9bc270 2024-03-11T19:19:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.619+0000 7faaf9ffb640 1 -- 172.21.15.29:0/927625943 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fab0c01f180 con 0x5581dd9bc270 2024-03-11T19:19:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.619+0000 7fab1b3ee740 10 client.17637 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.619+0000 7fab1b3ee740 10 client.17637 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:19:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.619+0000 7fab1b3ee740 20 client.17637 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.619+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/927625943 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5581dda2a3d0 0x5581dda2c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.619+0000 7fab1b3ee740 4 client.17637 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:19:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.619+0000 7fab1b3ee740 1 -- 172.21.15.29:0/927625943 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5581dd6b0890 con 0x5581dda2a3d0 2024-03-11T19:19:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.619+0000 7fab11d9b640 1 --2- 172.21.15.29:0/927625943 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5581dda2a3d0 0x5581dda2c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.620+0000 7fab11d9b640 1 --2- 172.21.15.29:0/927625943 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5581dda2a3d0 0x5581dda2c7f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=74002259999f799a server_cookie=e868a04338445ce5 in_seq=0 out_seq=0 2024-03-11T19:19:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.620+0000 7faaf9ffb640 10 client.17637 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:19:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.620+0000 7faaf9ffb640 1 -- 172.21.15.29:0/927625943 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5581dd6b0890 con 0x5581dda2a3d0 2024-03-11T19:19:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.620+0000 7faaf9ffb640 10 client.17637 handle_command_reply: tid=0 2024-03-11T19:19:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.620+0000 7fab1b3ee740 1 client.17637 shutdown 2024-03-11T19:19:10.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.620+0000 7fab1b3ee740 1 -- 172.21.15.29:0/927625943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bc270 msgr2=0x5581dd9cd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.620+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/927625943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bc270 0x5581dd9cd870 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fab0c0042d0 tx=0x7fab0c005220 comp rx=0 tx=0).stop 2024-03-11T19:19:10.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.620+0000 7fab1b3ee740 1 -- 172.21.15.29:0/927625943 shutdown_connections 2024-03-11T19:19:10.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.620+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/927625943 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5581dd9c1dd0 0x5581dd9cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.620+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/927625943 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5581dda2a3d0 0x5581dda2c7f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.620+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/927625943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5581dd9bcc60 0x5581dd9cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.620+0000 7fab1b3ee740 1 --2- 172.21.15.29:0/927625943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5581dd9bc270 0x5581dd9cd870 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.620+0000 7faaf9ffb640 0 client.17637 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:19:10.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.620+0000 7fab1b3ee740 1 -- 172.21.15.29:0/927625943 >> 172.21.15.29:0/927625943 conn(0x5581dd9b77e0 msgr2=0x5581dd9c4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.620+0000 7fab1b3ee740 1 -- 172.21.15.29:0/927625943 shutdown_connections 2024-03-11T19:19:10.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.620+0000 7fab1b3ee740 1 -- 172.21.15.29:0/927625943 wait complete. 2024-03-11T19:19:10.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.620+0000 7fab1b3ee740 20 client.17637 trim_cache size 0 max 16384 2024-03-11T19:19:10.631 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:19:10.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.621+0000 7fab18815640 1 -- 172.21.15.29:0/232211547 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faafc0a5e80 msgr2=0x7faafc0a8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.621+0000 7fab18815640 1 --2- 172.21.15.29:0/232211547 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faafc0a5e80 0x7faafc0a8340 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fab04048000 tx=0x7fab04049000 comp rx=0 tx=0).stop 2024-03-11T19:19:10.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.621+0000 7fab18815640 1 -- 172.21.15.29:0/232211547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab14101280 msgr2=0x7fab141954a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.621+0000 7fab18815640 1 --2- 172.21.15.29:0/232211547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab14101280 0x7fab141954a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fab0800d950 tx=0x7fab0800de10 comp rx=0 tx=0).stop 2024-03-11T19:19:10.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.621+0000 7fab18815640 1 -- 172.21.15.29:0/232211547 shutdown_connections 2024-03-11T19:19:10.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.621+0000 7fab18815640 1 --2- 172.21.15.29:0/232211547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab14101c40 0x7fab1419c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.621+0000 7fab18815640 1 --2- 172.21.15.29:0/232211547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab14101280 0x7fab141954a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.621+0000 7fab18815640 1 --2- 172.21.15.29:0/232211547 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7faafc0a5e80 0x7faafc0a8340 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.621+0000 7fab18815640 1 --2- 172.21.15.29:0/232211547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab14100090 0x7fab14194f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.621+0000 7fab18815640 1 -- 172.21.15.29:0/232211547 >> 172.21.15.29:0/232211547 conn(0x7fab140fb620 msgr2=0x7fab14104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.621+0000 7fab18815640 1 -- 172.21.15.29:0/232211547 shutdown_connections 2024-03-11T19:19:10.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.621+0000 7fab18815640 1 -- 172.21.15.29:0/232211547 wait complete. 2024-03-11T19:19:10.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:10 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3155491930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:19:10.635 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:19:10.635 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:19:10.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:10 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3155491930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:19:10.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.799+0000 7f17ed3c7640 1 -- 172.21.15.29:0/3740443472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e8101ca0 msgr2=0x7f17e8104090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.799+0000 7f17ed3c7640 1 --2- 172.21.15.29:0/3740443472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e8101ca0 0x7f17e8104090 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009900 tx=0x7f17d002bc30 comp rx=0 tx=0).stop 2024-03-11T19:19:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.799+0000 7f17ed3c7640 1 -- 172.21.15.29:0/3740443472 shutdown_connections 2024-03-11T19:19:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.799+0000 7f17ed3c7640 1 --2- 172.21.15.29:0/3740443472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17e8106f00 0x7f17e81092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.799+0000 7f17ed3c7640 1 --2- 172.21.15.29:0/3740443472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17e81045d0 0x7f17e81069c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.799+0000 7f17ed3c7640 1 --2- 172.21.15.29:0/3740443472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e8101ca0 0x7f17e8104090 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.799+0000 7f17ed3c7640 1 -- 172.21.15.29:0/3740443472 >> 172.21.15.29:0/3740443472 conn(0x7f17e80fb610 msgr2=0x7f17e80fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.799+0000 7f17ed3c7640 1 -- 172.21.15.29:0/3740443472 shutdown_connections 2024-03-11T19:19:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.799+0000 7f17ed3c7640 1 -- 172.21.15.29:0/3740443472 wait complete. 2024-03-11T19:19:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.800+0000 7f17ed3c7640 1 Processor -- start 2024-03-11T19:19:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.800+0000 7f17ed3c7640 1 -- start start 2024-03-11T19:19:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.800+0000 7f17ed3c7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17e8101ca0 0x7f17e8199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.800+0000 7f17ed3c7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e81045d0 0x7f17e81998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.800+0000 7f17ed3c7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17e8106f00 0x7f17e81a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.800+0000 7f17ed3c7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e81a0e70 con 0x7f17e8101ca0 2024-03-11T19:19:10.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.800+0000 7f17ed3c7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e81a0fe0 con 0x7f17e8106f00 2024-03-11T19:19:10.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.800+0000 7f17ed3c7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e81a11a0 con 0x7f17e81045d0 2024-03-11T19:19:10.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.801+0000 7f17e77fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17e8106f00 0x7f17e81a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.801+0000 7f17e6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17e8101ca0 0x7f17e8199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.801+0000 7f17e77fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17e8106f00 0x7f17e81a0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55978/0 (socket says 172.21.15.29:55978) 2024-03-11T19:19:10.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.801+0000 7f17e77fe640 1 -- 172.21.15.29:0/2747140150 learned_addr learned my addr 172.21.15.29:0/2747140150 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.801+0000 7f17dffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e81045d0 0x7f17e81998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.801+0000 7f17e6ffd640 1 -- 172.21.15.29:0/2747140150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e81045d0 msgr2=0x7f17e81998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.801+0000 7f17e6ffd640 1 --2- 172.21.15.29:0/2747140150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e81045d0 0x7f17e81998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.801+0000 7f17e6ffd640 1 -- 172.21.15.29:0/2747140150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17e8106f00 msgr2=0x7f17e81a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.801+0000 7f17e6ffd640 1 --2- 172.21.15.29:0/2747140150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17e8106f00 0x7f17e81a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.801+0000 7f17e6ffd640 1 -- 172.21.15.29:0/2747140150 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e0009840 con 0x7f17e8101ca0 2024-03-11T19:19:10.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.801+0000 7f17dffff640 1 --2- 172.21.15.29:0/2747140150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e81045d0 0x7f17e81998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:19:10.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.801+0000 7f17e6ffd640 1 --2- 172.21.15.29:0/2747140150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17e8101ca0 0x7f17e8199380 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f17d0000c00 tx=0x7f17d0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.802+0000 7f17e4ff9640 1 -- 172.21.15.29:0/2747140150 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0004500 con 0x7f17e8101ca0 2024-03-11T19:19:10.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.802+0000 7f17e4ff9640 1 -- 172.21.15.29:0/2747140150 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f17d002de90 con 0x7f17e8101ca0 2024-03-11T19:19:10.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.802+0000 7f17e4ff9640 1 -- 172.21.15.29:0/2747140150 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0005310 con 0x7f17e8101ca0 2024-03-11T19:19:10.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.802+0000 7f17ed3c7640 1 -- 172.21.15.29:0/2747140150 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d0009560 con 0x7f17e8101ca0 2024-03-11T19:19:10.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.802+0000 7f17ed3c7640 1 -- 172.21.15.29:0/2747140150 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e81a1720 con 0x7f17e8101ca0 2024-03-11T19:19:10.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.804+0000 7f17e4ff9640 1 -- 172.21.15.29:0/2747140150 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f17d0003ca0 con 0x7f17e8101ca0 2024-03-11T19:19:10.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.805+0000 7f17eff96740 1 Processor -- start 2024-03-11T19:19:10.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.805+0000 7f17eff96740 1 -- start start 2024-03-11T19:19:10.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.805+0000 7f17eff96740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125e270 0x563a5125e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.805+0000 7f17eff96740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a5125ec60 0x563a51263760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.805+0000 7f17eff96740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a51263dd0 0x563a51266260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.805+0000 7f17eff96740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a5121a960 con 0x563a5125e270 2024-03-11T19:19:10.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.805+0000 7f17eff96740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a511ce310 con 0x563a51263dd0 2024-03-11T19:19:10.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.805+0000 7f17eff96740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a511cb840 con 0x563a5125ec60 2024-03-11T19:19:10.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.805+0000 7f17e4ff9640 1 --2- 172.21.15.29:0/2747140150 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f17d40a1ab0 0x7f17d40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.806+0000 7f17dffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a51263dd0 0x563a51266260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.806+0000 7f17e77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125e270 0x563a5125e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.806+0000 7f17e77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125e270 0x563a5125e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53820/0 (socket says 172.21.15.29:53820) 2024-03-11T19:19:10.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.806+0000 7f17e77fe640 1 -- 172.21.15.29:0/2876059774 learned_addr learned my addr 172.21.15.29:0/2876059774 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.806+0000 7f17e6ffd640 1 --2- 172.21.15.29:0/2876059774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a5125ec60 0x563a51263760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.806+0000 7f17e77fe640 1 --2- 172.21.15.29:0/2747140150 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f17d40a1ab0 0x7f17d40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.806+0000 7f17e6ffd640 1 -- 172.21.15.29:0/2876059774 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a51263dd0 msgr2=0x563a51266260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.806+0000 7f17e6ffd640 1 --2- 172.21.15.29:0/2876059774 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a51263dd0 0x563a51266260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.806+0000 7f17e6ffd640 1 -- 172.21.15.29:0/2876059774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125e270 msgr2=0x563a5125e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.806+0000 7f17e6ffd640 1 --2- 172.21.15.29:0/2876059774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125e270 0x563a5125e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.806+0000 7f17e6ffd640 1 -- 172.21.15.29:0/2876059774 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a51189060 con 0x563a5125ec60 2024-03-11T19:19:10.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.806+0000 7f17e4ff9640 1 -- 172.21.15.29:0/2747140150 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f17d00ef340 con 0x7f17e8101ca0 2024-03-11T19:19:10.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.807+0000 7f17e6ffd640 1 --2- 172.21.15.29:0/2876059774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a5125ec60 0x563a51263760 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f17d00ecf30 tx=0x7f17d0047f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc265dfc253280bd server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.807+0000 7f17e77fe640 1 --2- 172.21.15.29:0/2747140150 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f17d40a1ab0 0x7f17d40a3f70 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009270 tx=0x7f17e0009f50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.807+0000 7f17dd7fa640 1 -- 172.21.15.29:0/2876059774 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d00d5ea0 con 0x563a5125ec60 2024-03-11T19:19:10.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.807+0000 7f17dd7fa640 1 -- 172.21.15.29:0/2876059774 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f17d004d250 con 0x563a5125ec60 2024-03-11T19:19:10.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.807+0000 7f17dd7fa640 1 -- 172.21.15.29:0/2876059774 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d00c4af0 con 0x563a5125ec60 2024-03-11T19:19:10.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.808+0000 7f17eff96740 1 -- 172.21.15.29:0/2876059774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a5125ec60 msgr2=0x563a51263760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.808+0000 7f17eff96740 1 --2- 172.21.15.29:0/2876059774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a5125ec60 0x563a51263760 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f17d00ecf30 tx=0x7f17d0047f90 comp rx=0 tx=0).stop 2024-03-11T19:19:10.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.809+0000 7f17eff96740 1 -- 172.21.15.29:0/2876059774 shutdown_connections 2024-03-11T19:19:10.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.809+0000 7f17eff96740 1 --2- 172.21.15.29:0/2876059774 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a51263dd0 0x563a51266260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.809+0000 7f17eff96740 1 --2- 172.21.15.29:0/2876059774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a5125ec60 0x563a51263760 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.809+0000 7f17eff96740 1 --2- 172.21.15.29:0/2876059774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125e270 0x563a5125e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.809+0000 7f17eff96740 1 -- 172.21.15.29:0/2876059774 >> 172.21.15.29:0/2876059774 conn(0x563a512597e0 msgr2=0x563a51259be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.809+0000 7f17eff96740 1 -- 172.21.15.29:0/2876059774 shutdown_connections 2024-03-11T19:19:10.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.809+0000 7f17eff96740 1 -- 172.21.15.29:0/2876059774 wait complete. 2024-03-11T19:19:10.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.809+0000 7f17eff96740 1 -- start start 2024-03-11T19:19:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.810+0000 7f17eff96740 1 Processor -- start 2024-03-11T19:19:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.810+0000 7f17eff96740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125e270 0x563a51268240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.810+0000 7f17eff96740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a5125ec60 0x563a51268780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.810+0000 7f17eff96740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a51263dd0 0x563a5126f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.810+0000 7f17e6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125e270 0x563a51268240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.811+0000 7f17e6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125e270 0x563a51268240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53834/0 (socket says 172.21.15.29:53834) 2024-03-11T19:19:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.811+0000 7f17e77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a51263dd0 0x563a5126f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.811+0000 7f17dffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a5125ec60 0x563a51268780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.811+0000 7f17dffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a5125ec60 0x563a51268780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:55996/0 (socket says 172.21.15.29:55996) 2024-03-11T19:19:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.811+0000 7f17e77fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a51263dd0 0x563a5126f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48116/0 (socket says 172.21.15.29:48116) 2024-03-11T19:19:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.811+0000 7f17e6ffd640 1 -- 172.21.15.29:0/2640641090 learned_addr learned my addr 172.21.15.29:0/2640641090 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.811+0000 7f17eff96740 1 -- 172.21.15.29:0/2640641090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a511c7020 con 0x563a5125e270 2024-03-11T19:19:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.811+0000 7f17eff96740 1 -- 172.21.15.29:0/2640641090 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a50e9a430 con 0x563a5125ec60 2024-03-11T19:19:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.811+0000 7f17eff96740 1 -- 172.21.15.29:0/2640641090 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a50e56e00 con 0x563a51263dd0 2024-03-11T19:19:10.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.811+0000 7f17eff96740 10 client.0 _ll_register_callbacks cb 0x563a5116ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:10.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.812+0000 7f17e6ffd640 1 -- 172.21.15.29:0/2640641090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a51263dd0 msgr2=0x563a5126f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.812+0000 7f17e6ffd640 1 --2- 172.21.15.29:0/2640641090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a51263dd0 0x563a5126f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.812+0000 7f17e6ffd640 1 -- 172.21.15.29:0/2640641090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a5125ec60 msgr2=0x563a51268780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.812+0000 7f17e6ffd640 1 --2- 172.21.15.29:0/2640641090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a5125ec60 0x563a51268780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.812+0000 7f17e6ffd640 1 -- 172.21.15.29:0/2640641090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e81a1720 con 0x563a5125e270 2024-03-11T19:19:10.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.812+0000 7f17e6ffd640 1 --2- 172.21.15.29:0/2640641090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125e270 0x563a51268240 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f17d00b8d20 tx=0x7f17d0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26bf9e14414898d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.813+0000 7f17ce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:19:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.813+0000 7f17ce7fc640 1 -- 172.21.15.29:0/2640641090 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d00d5ea0 con 0x563a5125e270 2024-03-11T19:19:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.813+0000 7f17ce7fc640 1 -- 172.21.15.29:0/2640641090 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f17d00bc350 con 0x563a5125e270 2024-03-11T19:19:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.813+0000 7f17eff96740 1 -- 172.21.15.29:0/2640641090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563a50ec67e0 con 0x563a5125e270 2024-03-11T19:19:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.813+0000 7f17ce7fc640 1 -- 172.21.15.29:0/2640641090 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17d00bc120 con 0x563a5125e270 2024-03-11T19:19:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.813+0000 7f17eff96740 10 client.17649 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.813+0000 7f17eff96740 1 -- 172.21.15.29:0/2640641090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563a511c56a0 con 0x563a5125e270 2024-03-11T19:19:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.813+0000 7f17ce7fc640 1 -- 172.21.15.29:0/2640641090 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f17d00c04a0 con 0x563a5125e270 2024-03-11T19:19:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.814+0000 7f17ce7fc640 1 -- 172.21.15.29:0/2640641090 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f17d003c070 con 0x563a5125e270 2024-03-11T19:19:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.814+0000 7f17eff96740 10 client.17649 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.814+0000 7f17eff96740 10 client.17649 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:19:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.814+0000 7f17eff96740 20 client.17649 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.814+0000 7f17eff96740 1 --2- 172.21.15.29:0/2640641090 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x563a51273fa0 0x563a51276410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.814+0000 7f17eff96740 4 client.17649 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:19:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.814+0000 7f17eff96740 1 -- 172.21.15.29:0/2640641090 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563a50ec67e0 con 0x563a51273fa0 2024-03-11T19:19:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.814+0000 7f17dffff640 1 --2- 172.21.15.29:0/2640641090 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x563a51273fa0 0x563a51276410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.816+0000 7f17dffff640 1 --2- 172.21.15.29:0/2640641090 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x563a51273fa0 0x563a51276410 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67f5537d96a50da8 server_cookie=5c473f98a225127b in_seq=0 out_seq=0 2024-03-11T19:19:10.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.816+0000 7f17ce7fc640 10 client.17649 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:10.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.818+0000 7f17ce7fc640 1 -- 172.21.15.29:0/2640641090 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563a50ec67e0 con 0x563a51273fa0 2024-03-11T19:19:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.818+0000 7f17ce7fc640 10 client.17649 handle_command_reply: tid=0 2024-03-11T19:19:10.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.820+0000 7f17eff96740 1 client.17649 shutdown 2024-03-11T19:19:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.820+0000 7f17eff96740 1 -- 172.21.15.29:0/2640641090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125e270 msgr2=0x563a51268240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.820+0000 7f17eff96740 1 --2- 172.21.15.29:0/2640641090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125e270 0x563a51268240 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f17d00b8d20 tx=0x7f17d0005fd0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.820+0000 7f17eff96740 1 -- 172.21.15.29:0/2640641090 shutdown_connections 2024-03-11T19:19:10.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.820+0000 7f17eff96740 1 --2- 172.21.15.29:0/2640641090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a51263dd0 0x563a5126f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.820+0000 7f17eff96740 1 --2- 172.21.15.29:0/2640641090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a5125ec60 0x563a51268780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.820+0000 7f17eff96740 1 --2- 172.21.15.29:0/2640641090 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x563a51273fa0 0x563a51276410 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.820+0000 7f17eff96740 1 --2- 172.21.15.29:0/2640641090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125e270 0x563a51268240 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.820+0000 7f17eff96740 1 -- 172.21.15.29:0/2640641090 >> 172.21.15.29:0/2640641090 conn(0x563a512597e0 msgr2=0x563a51258b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.821+0000 7f17ce7fc640 0 client.17649 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:10.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.821+0000 7f17eff96740 1 -- 172.21.15.29:0/2640641090 shutdown_connections 2024-03-11T19:19:10.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.821+0000 7f17eff96740 1 -- 172.21.15.29:0/2640641090 wait complete. 2024-03-11T19:19:10.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.821+0000 7f17eff96740 20 client.17649 trim_cache size 0 max 16384 2024-03-11T19:19:10.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:10 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3155491930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:19:10.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.845+0000 7f17eff96740 1 Processor -- start 2024-03-11T19:19:10.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.845+0000 7f17eff96740 1 -- start start 2024-03-11T19:19:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.845+0000 7f17eff96740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a5125e270 0x563a51267e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.845+0000 7f17eff96740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125ec60 0x563a5126f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.846+0000 7f17eff96740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a51263dd0 0x563a51268390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.846+0000 7f17eff96740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a5114c130 con 0x563a5125ec60 2024-03-11T19:19:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.846+0000 7f17eff96740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a5124ba40 con 0x563a51263dd0 2024-03-11T19:19:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.846+0000 7f17eff96740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a51266720 con 0x563a5125e270 2024-03-11T19:19:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.846+0000 7f17dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125ec60 0x563a5126f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.846+0000 7f17e6ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a5125e270 0x563a51267e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.846+0000 7f17e6ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a5125e270 0x563a51267e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48120/0 (socket says 172.21.15.29:48120) 2024-03-11T19:19:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.846+0000 7f17e6ffd640 1 -- 172.21.15.29:0/214067817 learned_addr learned my addr 172.21.15.29:0/214067817 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.846+0000 7f17dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125ec60 0x563a5126f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53846/0 (socket says 172.21.15.29:53846) 2024-03-11T19:19:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.846+0000 7f17e77fe640 1 --2- 172.21.15.29:0/214067817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a51263dd0 0x563a51268390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.846+0000 7f17e6ffd640 1 -- 172.21.15.29:0/214067817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a51263dd0 msgr2=0x563a51268390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.846+0000 7f17e6ffd640 1 --2- 172.21.15.29:0/214067817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a51263dd0 0x563a51268390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.846+0000 7f17e6ffd640 1 -- 172.21.15.29:0/214067817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125ec60 msgr2=0x563a5126f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.846+0000 7f17e6ffd640 1 --2- 172.21.15.29:0/214067817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125ec60 0x563a5126f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.846+0000 7f17e6ffd640 1 -- 172.21.15.29:0/214067817 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a51161c20 con 0x563a5125e270 2024-03-11T19:19:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.846+0000 7f17dffff640 1 --2- 172.21.15.29:0/214067817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125ec60 0x563a5126f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:10.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.847+0000 7f17e6ffd640 1 --2- 172.21.15.29:0/214067817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a5125e270 0x563a51267e50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f17d00b8cd0 tx=0x7f17d00ba860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b4b602f3603e2f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.847+0000 7f17dcff9640 1 -- 172.21.15.29:0/214067817 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d00d5ea0 con 0x563a5125e270 2024-03-11T19:19:10.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.847+0000 7f17dcff9640 1 -- 172.21.15.29:0/214067817 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f17d00c4820 con 0x563a5125e270 2024-03-11T19:19:10.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.847+0000 7f17dcff9640 1 -- 172.21.15.29:0/214067817 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d00056d0 con 0x563a5125e270 2024-03-11T19:19:10.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.848+0000 7f17eff96740 1 -- 172.21.15.29:0/214067817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a5125e270 msgr2=0x563a51267e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.848+0000 7f17eff96740 1 --2- 172.21.15.29:0/214067817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a5125e270 0x563a51267e50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f17d00b8cd0 tx=0x7f17d00ba860 comp rx=0 tx=0).stop 2024-03-11T19:19:10.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.848+0000 7f17eff96740 1 -- 172.21.15.29:0/214067817 shutdown_connections 2024-03-11T19:19:10.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.848+0000 7f17eff96740 1 --2- 172.21.15.29:0/214067817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a51263dd0 0x563a51268390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.848+0000 7f17eff96740 1 --2- 172.21.15.29:0/214067817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a5125ec60 0x563a5126f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.848+0000 7f17eff96740 1 --2- 172.21.15.29:0/214067817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a5125e270 0x563a51267e50 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.848+0000 7f17eff96740 1 -- 172.21.15.29:0/214067817 >> 172.21.15.29:0/214067817 conn(0x563a512597e0 msgr2=0x563a51260180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.848+0000 7f17eff96740 1 -- 172.21.15.29:0/214067817 shutdown_connections 2024-03-11T19:19:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.848+0000 7f17eff96740 1 -- 172.21.15.29:0/214067817 wait complete. 2024-03-11T19:19:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.849+0000 7f17eff96740 1 -- start start 2024-03-11T19:19:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.849+0000 7f17eff96740 1 Processor -- start 2024-03-11T19:19:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.849+0000 7f17eff96740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a5125e270 0x563a5126cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.849+0000 7f17eff96740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a5125ec60 0x563a5126d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.849+0000 7f17eff96740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a51263dd0 0x563a5126e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.849+0000 7f17e6ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a5125e270 0x563a5126cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.849+0000 7f17e6ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a5125e270 0x563a5126cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56018/0 (socket says 172.21.15.29:56018) 2024-03-11T19:19:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.850+0000 7f17dffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a5125ec60 0x563a5126d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.850+0000 7f17e77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a51263dd0 0x563a5126e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.850+0000 7f17dffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a5125ec60 0x563a5126d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48134/0 (socket says 172.21.15.29:48134) 2024-03-11T19:19:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.850+0000 7f17e77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a51263dd0 0x563a5126e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53858/0 (socket says 172.21.15.29:53858) 2024-03-11T19:19:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.850+0000 7f17e6ffd640 1 -- 172.21.15.29:0/3341419517 learned_addr learned my addr 172.21.15.29:0/3341419517 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.850+0000 7f17eff96740 1 -- 172.21.15.29:0/3341419517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a5126e970 con 0x563a51263dd0 2024-03-11T19:19:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.850+0000 7f17eff96740 1 -- 172.21.15.29:0/3341419517 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a5126eae0 con 0x563a5125e270 2024-03-11T19:19:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.850+0000 7f17eff96740 1 -- 172.21.15.29:0/3341419517 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a5126eca0 con 0x563a5125ec60 2024-03-11T19:19:10.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.850+0000 7f17e6ffd640 1 -- 172.21.15.29:0/3341419517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a5125ec60 msgr2=0x563a5126d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.850+0000 7f17e6ffd640 1 --2- 172.21.15.29:0/3341419517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a5125ec60 0x563a5126d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.850+0000 7f17e6ffd640 1 -- 172.21.15.29:0/3341419517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a51263dd0 msgr2=0x563a5126e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:10.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.850+0000 7f17e6ffd640 1 --2- 172.21.15.29:0/3341419517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a51263dd0 0x563a5126e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.850+0000 7f17eff96740 10 client.0 _ll_register_callbacks cb 0x7f17d00bbe90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:10.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.850+0000 7f17e6ffd640 1 -- 172.21.15.29:0/3341419517 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563a511c56a0 con 0x563a5125e270 2024-03-11T19:19:10.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.850+0000 7f17eff96740 1 -- 172.21.15.29:0/3341419517 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f17d004d250 con 0x563a5125e270 2024-03-11T19:19:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.850+0000 7f17e77fe640 1 --2- 172.21.15.29:0/3341419517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a51263dd0 0x563a5126e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.850+0000 7f17e6ffd640 1 --2- 172.21.15.29:0/3341419517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a5125e270 0x563a5126cf30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f17d00b87b0 tx=0x7f17d002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e79876b0f6d2a1dc server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.850+0000 7f17ce7fc640 10 client.27149 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:19:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.851+0000 7f17ce7fc640 1 -- 172.21.15.29:0/3341419517 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d00d5ea0 con 0x563a5125e270 2024-03-11T19:19:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.851+0000 7f17ce7fc640 1 -- 172.21.15.29:0/3341419517 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f17d00c4af0 con 0x563a5125e270 2024-03-11T19:19:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.851+0000 7f17ce7fc640 1 -- 172.21.15.29:0/3341419517 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17d00c0510 con 0x563a5125e270 2024-03-11T19:19:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.851+0000 7f17eff96740 10 client.27149 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.851+0000 7f17eff96740 1 -- 172.21.15.29:0/3341419517 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x563a511878c0 con 0x563a5125e270 2024-03-11T19:19:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.851+0000 7f17ce7fc640 1 -- 172.21.15.29:0/3341419517 <== mon.2 v2:172.21.15.170:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f17d003c070 con 0x563a5125e270 2024-03-11T19:19:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.851+0000 7f17eff96740 10 client.27149 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.851+0000 7f17eff96740 10 client.27149 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:19:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.851+0000 7f17eff96740 20 client.27149 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.851+0000 7f17eff96740 1 --2- 172.21.15.29:0/3341419517 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x563a512ca560 0x563a512cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.851+0000 7f17eff96740 4 client.27149 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:19:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.851+0000 7f17eff96740 1 -- 172.21.15.29:0/3341419517 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563a50f03720 con 0x563a512ca560 2024-03-11T19:19:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.851+0000 7f17dffff640 1 --2- 172.21.15.29:0/3341419517 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x563a512ca560 0x563a512cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.852+0000 7f17ce7fc640 10 client.27149 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.852+0000 7f17dffff640 1 --2- 172.21.15.29:0/3341419517 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x563a512ca560 0x563a512cc950 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5eb6a6f2d86ec97 server_cookie=f4a7e889a48b6156 in_seq=0 out_seq=0 2024-03-11T19:19:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.852+0000 7f17ce7fc640 1 -- 172.21.15.29:0/3341419517 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563a50f03720 con 0x563a512ca560 2024-03-11T19:19:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.852+0000 7f17ce7fc640 10 client.27149 handle_command_reply: tid=0 2024-03-11T19:19:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.852+0000 7f17eff96740 1 client.27149 shutdown 2024-03-11T19:19:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.852+0000 7f17eff96740 1 -- 172.21.15.29:0/3341419517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a5125e270 msgr2=0x563a5126cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.852+0000 7f17eff96740 1 --2- 172.21.15.29:0/3341419517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a5125e270 0x563a5126cf30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f17d00b87b0 tx=0x7f17d002bc30 comp rx=0 tx=0).stop 2024-03-11T19:19:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.853+0000 7f17eff96740 1 -- 172.21.15.29:0/3341419517 shutdown_connections 2024-03-11T19:19:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.853+0000 7f17eff96740 1 --2- 172.21.15.29:0/3341419517 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x563a512ca560 0x563a512cc950 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.853+0000 7f17eff96740 1 --2- 172.21.15.29:0/3341419517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563a51263dd0 0x563a5126e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.853+0000 7f17eff96740 1 --2- 172.21.15.29:0/3341419517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a5125ec60 0x563a5126d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.853+0000 7f17ce7fc640 0 client.27149 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.853+0000 7f17eff96740 1 --2- 172.21.15.29:0/3341419517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563a5125e270 0x563a5126cf30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.853+0000 7f17eff96740 1 -- 172.21.15.29:0/3341419517 >> 172.21.15.29:0/3341419517 conn(0x563a512597e0 msgr2=0x563a5125ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.853+0000 7f17eff96740 1 -- 172.21.15.29:0/3341419517 shutdown_connections 2024-03-11T19:19:10.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.853+0000 7f17eff96740 1 -- 172.21.15.29:0/3341419517 wait complete. 2024-03-11T19:19:10.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.853+0000 7f17eff96740 20 client.27149 trim_cache size 0 max 16384 2024-03-11T19:19:10.860 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:19:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.854+0000 7f17ed3c7640 1 -- 172.21.15.29:0/2747140150 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f17d40a1ab0 msgr2=0x7f17d40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.854+0000 7f17ed3c7640 1 --2- 172.21.15.29:0/2747140150 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f17d40a1ab0 0x7f17d40a3f70 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009270 tx=0x7f17e0009f50 comp rx=0 tx=0).stop 2024-03-11T19:19:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.854+0000 7f17ed3c7640 1 -- 172.21.15.29:0/2747140150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17e8101ca0 msgr2=0x7f17e8199380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.854+0000 7f17ed3c7640 1 --2- 172.21.15.29:0/2747140150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17e8101ca0 0x7f17e8199380 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f17d0000c00 tx=0x7f17d0004040 comp rx=0 tx=0).stop 2024-03-11T19:19:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.854+0000 7f17ed3c7640 1 -- 172.21.15.29:0/2747140150 shutdown_connections 2024-03-11T19:19:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.854+0000 7f17ed3c7640 1 --2- 172.21.15.29:0/2747140150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17e8106f00 0x7f17e81a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.854+0000 7f17ed3c7640 1 --2- 172.21.15.29:0/2747140150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e81045d0 0x7f17e81998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.854+0000 7f17ed3c7640 1 --2- 172.21.15.29:0/2747140150 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f17d40a1ab0 0x7f17d40a3f70 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.854+0000 7f17ed3c7640 1 --2- 172.21.15.29:0/2747140150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17e8101ca0 0x7f17e8199380 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.854+0000 7f17ed3c7640 1 -- 172.21.15.29:0/2747140150 >> 172.21.15.29:0/2747140150 conn(0x7f17e80fb610 msgr2=0x7f17e81072e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.854+0000 7f17ed3c7640 1 -- 172.21.15.29:0/2747140150 shutdown_connections 2024-03-11T19:19:10.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:10.854+0000 7f17ed3c7640 1 -- 172.21.15.29:0/2747140150 wait complete. 2024-03-11T19:19:11.879 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:19:11.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:19:11.879 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:19:11.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:11 smithi029 ceph-mon[32439]: pgmap v632: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 62 MiB/s wr, 1.21k op/s 2024-03-11T19:19:12.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:11 smithi190 ceph-mon[36292]: pgmap v632: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 62 MiB/s wr, 1.21k op/s 2024-03-11T19:19:12.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.068+0000 7f1777fff640 1 -- 172.21.15.29:0/1621465334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1778108780 msgr2=0x7f1778108be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:12.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.068+0000 7f1777fff640 1 --2- 172.21.15.29:0/1621465334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1778108780 0x7f1778108be0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f176c009900 tx=0x7f176c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:19:12.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.071+0000 7f1777fff640 1 -- 172.21.15.29:0/1621465334 shutdown_connections 2024-03-11T19:19:12.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.071+0000 7f1777fff640 1 --2- 172.21.15.29:0/1621465334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1778109120 0x7f177810d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.071+0000 7f1777fff640 1 --2- 172.21.15.29:0/1621465334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1778108780 0x7f1778108be0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.071+0000 7f1777fff640 1 --2- 172.21.15.29:0/1621465334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17780724d0 0x7f17780728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.071+0000 7f1777fff640 1 -- 172.21.15.29:0/1621465334 >> 172.21.15.29:0/1621465334 conn(0x7f177806dfa0 msgr2=0x7f17780703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:12.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.071+0000 7f1777fff640 1 -- 172.21.15.29:0/1621465334 shutdown_connections 2024-03-11T19:19:12.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.071+0000 7f1777fff640 1 -- 172.21.15.29:0/1621465334 wait complete. 2024-03-11T19:19:12.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.072+0000 7f1777fff640 1 Processor -- start 2024-03-11T19:19:12.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.073+0000 7f1777fff640 1 -- start start 2024-03-11T19:19:12.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.073+0000 7f1777fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17780724d0 0x7f17781a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.073+0000 7f1777fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1778108780 0x7f17781a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.073+0000 7f1777fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1778109120 0x7f17781ad310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.073+0000 7f1777fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17781ad850 con 0x7f1778109120 2024-03-11T19:19:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.073+0000 7f1777fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177810f8e0 con 0x7f1778108780 2024-03-11T19:19:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.073+0000 7f1777fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177810faa0 con 0x7f17780724d0 2024-03-11T19:19:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.073+0000 7f17767fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1778108780 0x7f17781a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.073+0000 7f17777fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1778109120 0x7f17781ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.073+0000 7f17767fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1778108780 0x7f17781a62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56036/0 (socket says 172.21.15.29:56036) 2024-03-11T19:19:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.073+0000 7f17767fc640 1 -- 172.21.15.29:0/3942586523 learned_addr learned my addr 172.21.15.29:0/3942586523 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.074+0000 7f17777fe640 1 -- 172.21.15.29:0/3942586523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17780724d0 msgr2=0x7f17781a5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.074+0000 7f17777fe640 1 --2- 172.21.15.29:0/3942586523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17780724d0 0x7f17781a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.074+0000 7f17777fe640 1 -- 172.21.15.29:0/3942586523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1778108780 msgr2=0x7f17781a62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.074+0000 7f17777fe640 1 --2- 172.21.15.29:0/3942586523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1778108780 0x7f17781a62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.074+0000 7f17777fe640 1 -- 172.21.15.29:0/3942586523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c009560 con 0x7f1778109120 2024-03-11T19:19:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.075+0000 7f17777fe640 1 --2- 172.21.15.29:0/3942586523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1778109120 0x7f17781ad310 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f176800e970 tx=0x7f176800ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:12.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.075+0000 7f175ffff640 1 -- 172.21.15.29:0/3942586523 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1768009620 con 0x7f1778109120 2024-03-11T19:19:12.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.075+0000 7f1777fff640 1 -- 172.21.15.29:0/3942586523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f177810fd80 con 0x7f1778109120 2024-03-11T19:19:12.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.075+0000 7f1777fff640 1 -- 172.21.15.29:0/3942586523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1778110290 con 0x7f1778109120 2024-03-11T19:19:12.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.076+0000 7f175ffff640 1 -- 172.21.15.29:0/3942586523 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f17680044d0 con 0x7f1778109120 2024-03-11T19:19:12.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.076+0000 7f175ffff640 1 -- 172.21.15.29:0/3942586523 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1768004800 con 0x7f1778109120 2024-03-11T19:19:12.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.079+0000 7f177fff2740 1 Processor -- start 2024-03-11T19:19:12.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.079+0000 7f175ffff640 1 -- 172.21.15.29:0/3942586523 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1768004aa0 con 0x7f1778109120 2024-03-11T19:19:12.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.079+0000 7f177fff2740 1 -- start start 2024-03-11T19:19:12.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.080+0000 7f177fff2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14de8710 0x55bb14de8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.080+0000 7f177fff2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14de98e0 0x55bb14de9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.080+0000 7f177fff2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14dea2a0 0x55bb14df0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.080+0000 7f177fff2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb14d57f90 con 0x55bb14dea2a0 2024-03-11T19:19:12.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.080+0000 7f177fff2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb14d554c0 con 0x55bb14de8710 2024-03-11T19:19:12.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.080+0000 7f177fff2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb14d50ca0 con 0x55bb14de98e0 2024-03-11T19:19:12.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.080+0000 7f17767fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14de8710 0x55bb14de8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.080+0000 7f1776ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14de98e0 0x55bb14de9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.080+0000 7f17767fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14de8710 0x55bb14de8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56038/0 (socket says 172.21.15.29:56038) 2024-03-11T19:19:12.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.080+0000 7f1776ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14de98e0 0x55bb14de9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48174/0 (socket says 172.21.15.29:48174) 2024-03-11T19:19:12.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.080+0000 7f17767fc640 1 -- 172.21.15.29:0/4225264153 learned_addr learned my addr 172.21.15.29:0/4225264153 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:12.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.080+0000 7f17767fc640 1 --2- 172.21.15.29:0/4225264153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14dea2a0 0x55bb14df0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.080+0000 7f175ffff640 1 --2- 172.21.15.29:0/3942586523 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f17600a1bb0 0x7f17600a4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.080+0000 7f1776ffd640 1 -- 172.21.15.29:0/4225264153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14de8710 msgr2=0x55bb14de8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:12.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.080+0000 7f1776ffd640 1 --2- 172.21.15.29:0/4225264153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14de8710 0x55bb14de8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.080+0000 7f1776ffd640 1 -- 172.21.15.29:0/4225264153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14dea2a0 msgr2=0x55bb14df0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:12.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.080+0000 7f1776ffd640 1 --2- 172.21.15.29:0/4225264153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14dea2a0 0x55bb14df0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.081+0000 7f1776ffd640 1 -- 172.21.15.29:0/4225264153 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb14d130b0 con 0x55bb14de98e0 2024-03-11T19:19:12.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.081+0000 7f1776ffd640 1 --2- 172.21.15.29:0/4225264153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14de98e0 0x55bb14de9d60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1770007c30 tx=0x7f177000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af16a6233393334 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:12.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.081+0000 7f175ffff640 1 -- 172.21.15.29:0/3942586523 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f1768014070 con 0x7f1778109120 2024-03-11T19:19:12.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.081+0000 7f17777fe640 1 --2- 172.21.15.29:0/3942586523 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f17600a1bb0 0x7f17600a4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.081+0000 7f175d7fa640 1 -- 172.21.15.29:0/4225264153 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177000f650 con 0x55bb14de98e0 2024-03-11T19:19:12.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.081+0000 7f175d7fa640 1 -- 172.21.15.29:0/4225264153 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f177000fc60 con 0x55bb14de98e0 2024-03-11T19:19:12.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.081+0000 7f175d7fa640 1 -- 172.21.15.29:0/4225264153 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177000eac0 con 0x55bb14de98e0 2024-03-11T19:19:12.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.082+0000 7f17777fe640 1 --2- 172.21.15.29:0/3942586523 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f17600a1bb0 0x7f17600a4070 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f176809a190 tx=0x7f17680cd2f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:12.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.082+0000 7f177fff2740 1 -- 172.21.15.29:0/4225264153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14de98e0 msgr2=0x55bb14de9d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:12.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.082+0000 7f177fff2740 1 --2- 172.21.15.29:0/4225264153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14de98e0 0x55bb14de9d60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1770007c30 tx=0x7f177000c6a0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.082+0000 7f177fff2740 1 -- 172.21.15.29:0/4225264153 shutdown_connections 2024-03-11T19:19:12.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.082+0000 7f177fff2740 1 --2- 172.21.15.29:0/4225264153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14dea2a0 0x55bb14df0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.082+0000 7f177fff2740 1 --2- 172.21.15.29:0/4225264153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14de98e0 0x55bb14de9d60 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.082+0000 7f177fff2740 1 --2- 172.21.15.29:0/4225264153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14de8710 0x55bb14de8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.082+0000 7f177fff2740 1 -- 172.21.15.29:0/4225264153 >> 172.21.15.29:0/4225264153 conn(0x55bb14de38f0 msgr2=0x55bb14de3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:12.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.083+0000 7f177fff2740 1 -- 172.21.15.29:0/4225264153 shutdown_connections 2024-03-11T19:19:12.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.083+0000 7f177fff2740 1 -- 172.21.15.29:0/4225264153 wait complete. 2024-03-11T19:19:12.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.084+0000 7f177fff2740 1 -- start start 2024-03-11T19:19:12.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.084+0000 7f177fff2740 1 Processor -- start 2024-03-11T19:19:12.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.084+0000 7f177fff2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14de8710 0x55bb14df2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.084+0000 7f177fff2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14de98e0 0x55bb14df2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.084+0000 7f177fff2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14dea2a0 0x55bb14df9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.084+0000 7f177fff2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb14cf5f40 con 0x55bb14dea2a0 2024-03-11T19:19:12.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.084+0000 7f177fff2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb14a244b0 con 0x55bb14de8710 2024-03-11T19:19:12.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.084+0000 7f177fff2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb14cd6670 con 0x55bb14de98e0 2024-03-11T19:19:12.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.084+0000 7f177fff2740 10 client.0 _ll_register_callbacks cb 0x55bb14d43f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:12.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.084+0000 7f17767fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14dea2a0 0x55bb14df9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.084+0000 7f17767fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14dea2a0 0x55bb14df9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53908/0 (socket says 172.21.15.29:53908) 2024-03-11T19:19:12.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.084+0000 7f17767fc640 1 -- 172.21.15.29:0/1330205724 learned_addr learned my addr 172.21.15.29:0/1330205724 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:12.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.084+0000 7f1776ffd640 1 --2- 172.21.15.29:0/1330205724 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14de98e0 0x55bb14df2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.085+0000 7f17767fc640 1 --2- 172.21.15.29:0/1330205724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14de8710 0x55bb14df2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.085+0000 7f17767fc640 1 -- 172.21.15.29:0/1330205724 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14de98e0 msgr2=0x55bb14df2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:12.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.085+0000 7f17767fc640 1 --2- 172.21.15.29:0/1330205724 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14de98e0 0x55bb14df2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.085+0000 7f17767fc640 1 -- 172.21.15.29:0/1330205724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14de8710 msgr2=0x55bb14df2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:12.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.085+0000 7f17767fc640 1 --2- 172.21.15.29:0/1330205724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14de8710 0x55bb14df2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.085+0000 7f17767fc640 1 -- 172.21.15.29:0/1330205724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1770007860 con 0x55bb14dea2a0 2024-03-11T19:19:12.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.085+0000 7f17767fc640 1 --2- 172.21.15.29:0/1330205724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14dea2a0 0x55bb14df9ea0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f176c02c7d0 tx=0x7f176c02c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7573c9a233a83132 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:12.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.085+0000 7f175aff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:19:12.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.086+0000 7f175aff5640 1 -- 172.21.15.29:0/1330205724 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176c004440 con 0x55bb14dea2a0 2024-03-11T19:19:12.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.086+0000 7f177fff2740 1 -- 172.21.15.29:0/1330205724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bb14d2af40 con 0x55bb14dea2a0 2024-03-11T19:19:12.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.086+0000 7f175aff5640 1 -- 172.21.15.29:0/1330205724 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f176c02de40 con 0x55bb14dea2a0 2024-03-11T19:19:12.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.086+0000 7f175aff5640 1 -- 172.21.15.29:0/1330205724 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f176c0035f0 con 0x55bb14dea2a0 2024-03-11T19:19:12.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.086+0000 7f177fff2740 10 client.17667 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:12.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.086+0000 7f177fff2740 1 -- 172.21.15.29:0/1330205724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55bb14d4f320 con 0x55bb14dea2a0 2024-03-11T19:19:12.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.087+0000 7f175aff5640 1 -- 172.21.15.29:0/1330205724 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55bb14d2af40 con 0x55bb14dea2a0 2024-03-11T19:19:12.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.087+0000 7f175aff5640 1 -- 172.21.15.29:0/1330205724 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f176c03c070 con 0x55bb14dea2a0 2024-03-11T19:19:12.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.087+0000 7f177fff2740 10 client.17667 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:12.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.087+0000 7f177fff2740 10 client.17667 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:19:12.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.087+0000 7f177fff2740 20 client.17667 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:12.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.087+0000 7f177fff2740 1 --2- 172.21.15.29:0/1330205724 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55bb14dfeb10 0x55bb14e00f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.088+0000 7f177fff2740 4 client.17667 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:19:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.088+0000 7f177fff2740 1 -- 172.21.15.29:0/1330205724 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bb14d47310 con 0x55bb14dfeb10 2024-03-11T19:19:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.088+0000 7f1776ffd640 1 --2- 172.21.15.29:0/1330205724 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55bb14dfeb10 0x55bb14e00f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.088+0000 7f1776ffd640 1 --2- 172.21.15.29:0/1330205724 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55bb14dfeb10 0x55bb14e00f30 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7e7633770abd48e server_cookie=49aa360fe4df22c9 in_seq=0 out_seq=0 2024-03-11T19:19:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.088+0000 7f175aff5640 10 client.17667 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.089+0000 7f175aff5640 1 -- 172.21.15.29:0/1330205724 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55bb14d47310 con 0x55bb14dfeb10 2024-03-11T19:19:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.089+0000 7f175aff5640 10 client.17667 handle_command_reply: tid=0 2024-03-11T19:19:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.089+0000 7f177fff2740 1 client.17667 shutdown 2024-03-11T19:19:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.089+0000 7f177fff2740 1 -- 172.21.15.29:0/1330205724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14dea2a0 msgr2=0x55bb14df9ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.089+0000 7f177fff2740 1 --2- 172.21.15.29:0/1330205724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14dea2a0 0x55bb14df9ea0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f176c02c7d0 tx=0x7f176c02c9f0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.090+0000 7f177fff2740 1 -- 172.21.15.29:0/1330205724 shutdown_connections 2024-03-11T19:19:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.090+0000 7f177fff2740 1 --2- 172.21.15.29:0/1330205724 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55bb14dfeb10 0x55bb14e00f30 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.090+0000 7f177fff2740 1 --2- 172.21.15.29:0/1330205724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14dea2a0 0x55bb14df9ea0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.090+0000 7f177fff2740 1 --2- 172.21.15.29:0/1330205724 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14de98e0 0x55bb14df2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.090+0000 7f175aff5640 0 client.17667 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.090+0000 7f177fff2740 1 --2- 172.21.15.29:0/1330205724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14de8710 0x55bb14df2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.090+0000 7f177fff2740 1 -- 172.21.15.29:0/1330205724 >> 172.21.15.29:0/1330205724 conn(0x55bb14de38f0 msgr2=0x55bb14de2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:12.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.090+0000 7f177fff2740 1 -- 172.21.15.29:0/1330205724 shutdown_connections 2024-03-11T19:19:12.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.090+0000 7f177fff2740 1 -- 172.21.15.29:0/1330205724 wait complete. 2024-03-11T19:19:12.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.090+0000 7f177fff2740 20 client.17667 trim_cache size 0 max 16384 2024-03-11T19:19:12.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:11 smithi170 ceph-mon[35690]: pgmap v632: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 62 MiB/s wr, 1.21k op/s 2024-03-11T19:19:12.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.129+0000 7f177fff2740 1 Processor -- start 2024-03-11T19:19:12.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.129+0000 7f177fff2740 1 -- start start 2024-03-11T19:19:12.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.129+0000 7f177fff2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14de8710 0x55bb14de7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.129+0000 7f177fff2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14de98e0 0x55bb14df9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.129+0000 7f177fff2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14dea2a0 0x55bb14de2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.129+0000 7f177fff2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb14d2af70 con 0x55bb14de8710 2024-03-11T19:19:12.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.130+0000 7f177fff2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb14df3b40 con 0x55bb14de98e0 2024-03-11T19:19:12.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.130+0000 7f177fff2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb14dfa300 con 0x55bb14dea2a0 2024-03-11T19:19:12.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.130+0000 7f17767fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14de8710 0x55bb14de7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.130+0000 7f17767fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14de8710 0x55bb14de7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53912/0 (socket says 172.21.15.29:53912) 2024-03-11T19:19:12.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.130+0000 7f17767fc640 1 -- 172.21.15.29:0/3809262092 learned_addr learned my addr 172.21.15.29:0/3809262092 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:12.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.130+0000 7f17767fc640 1 --2- 172.21.15.29:0/3809262092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14dea2a0 0x55bb14de2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.130+0000 7f17767fc640 1 -- 172.21.15.29:0/3809262092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14dea2a0 msgr2=0x55bb14de2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:12.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.130+0000 7f17767fc640 1 --2- 172.21.15.29:0/3809262092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14dea2a0 0x55bb14de2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.130+0000 7f17767fc640 1 -- 172.21.15.29:0/3809262092 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14de98e0 msgr2=0x55bb14df9dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:19:12.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.130+0000 7f17767fc640 1 --2- 172.21.15.29:0/3809262092 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14de98e0 0x55bb14df9dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.130+0000 7f17767fc640 1 -- 172.21.15.29:0/3809262092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb14cebc00 con 0x55bb14de8710 2024-03-11T19:19:12.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.131+0000 7f17767fc640 1 --2- 172.21.15.29:0/3809262092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14de8710 0x55bb14de7260 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f176c009430 tx=0x7f176c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2b2c10e6079a24e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:12.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.132+0000 7f175cff9640 1 -- 172.21.15.29:0/3809262092 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176c004dc0 con 0x55bb14de8710 2024-03-11T19:19:12.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.132+0000 7f175cff9640 1 -- 172.21.15.29:0/3809262092 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f176c049030 con 0x55bb14de8710 2024-03-11T19:19:12.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.132+0000 7f175cff9640 1 -- 172.21.15.29:0/3809262092 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176c04f470 con 0x55bb14de8710 2024-03-11T19:19:12.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.133+0000 7f177fff2740 1 -- 172.21.15.29:0/3809262092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14de8710 msgr2=0x55bb14de7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:12.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.133+0000 7f177fff2740 1 --2- 172.21.15.29:0/3809262092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14de8710 0x55bb14de7260 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f176c009430 tx=0x7f176c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:19:12.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.135+0000 7f177fff2740 1 -- 172.21.15.29:0/3809262092 shutdown_connections 2024-03-11T19:19:12.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.135+0000 7f177fff2740 1 --2- 172.21.15.29:0/3809262092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14dea2a0 0x55bb14de2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.135+0000 7f177fff2740 1 --2- 172.21.15.29:0/3809262092 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14de98e0 0x55bb14df9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.135+0000 7f177fff2740 1 --2- 172.21.15.29:0/3809262092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14de8710 0x55bb14de7260 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.135+0000 7f177fff2740 1 -- 172.21.15.29:0/3809262092 >> 172.21.15.29:0/3809262092 conn(0x55bb14de38f0 msgr2=0x55bb14de6b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:12.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.135+0000 7f177fff2740 1 -- 172.21.15.29:0/3809262092 shutdown_connections 2024-03-11T19:19:12.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.135+0000 7f177fff2740 1 -- 172.21.15.29:0/3809262092 wait complete. 2024-03-11T19:19:12.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.136+0000 7f177fff2740 1 -- start start 2024-03-11T19:19:12.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.136+0000 7f177fff2740 1 Processor -- start 2024-03-11T19:19:12.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.136+0000 7f177fff2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14de8710 0x55bb14df8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.136+0000 7f177fff2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14de98e0 0x55bb14df8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.136+0000 7f177fff2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14dea2a0 0x55bb14df9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.136+0000 7f177fff2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb14e550e0 con 0x55bb14de8710 2024-03-11T19:19:12.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.136+0000 7f177fff2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb14e55250 con 0x55bb14dea2a0 2024-03-11T19:19:12.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.136+0000 7f177fff2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb14e55410 con 0x55bb14de98e0 2024-03-11T19:19:12.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.136+0000 7f177fff2740 10 client.0 _ll_register_callbacks cb 0x7f17700073e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:12.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.137+0000 7f17767fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14de8710 0x55bb14df8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.137+0000 7f17767fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14de8710 0x55bb14df8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53914/0 (socket says 172.21.15.29:53914) 2024-03-11T19:19:12.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.137+0000 7f17767fc640 1 -- 172.21.15.29:0/36306369 learned_addr learned my addr 172.21.15.29:0/36306369 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:12.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.137+0000 7f17767fc640 1 --2- 172.21.15.29:0/36306369 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14dea2a0 0x55bb14df9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.137+0000 7f1776ffd640 1 --2- 172.21.15.29:0/36306369 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14de98e0 0x55bb14df8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.137+0000 7f17767fc640 1 -- 172.21.15.29:0/36306369 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14de98e0 msgr2=0x55bb14df8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:12.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.137+0000 7f17767fc640 1 --2- 172.21.15.29:0/36306369 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14de98e0 0x55bb14df8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.137+0000 7f17767fc640 1 -- 172.21.15.29:0/36306369 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14dea2a0 msgr2=0x55bb14df9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:12.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.137+0000 7f17767fc640 1 --2- 172.21.15.29:0/36306369 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14dea2a0 0x55bb14df9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.137+0000 7f17767fc640 1 -- 172.21.15.29:0/36306369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bb14d4f320 con 0x55bb14de8710 2024-03-11T19:19:12.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.137+0000 7f17767fc640 1 --2- 172.21.15.29:0/36306369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14de8710 0x55bb14df8780 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f176c0059d0 tx=0x7f176c052040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7de037481f50b17 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:12.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.137+0000 7f175aff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:19:12.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.138+0000 7f175aff5640 1 -- 172.21.15.29:0/36306369 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176c004510 con 0x55bb14de8710 2024-03-11T19:19:12.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.138+0000 7f177fff2740 1 -- 172.21.15.29:0/36306369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bb14d76900 con 0x55bb14de8710 2024-03-11T19:19:12.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.138+0000 7f175aff5640 1 -- 172.21.15.29:0/36306369 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f176c02de40 con 0x55bb14de8710 2024-03-11T19:19:12.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.138+0000 7f175aff5640 1 -- 172.21.15.29:0/36306369 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f176c04f470 con 0x55bb14de8710 2024-03-11T19:19:12.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.138+0000 7f177fff2740 10 client.17673 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:12.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.138+0000 7f177fff2740 1 -- 172.21.15.29:0/36306369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55bb14d11910 con 0x55bb14de8710 2024-03-11T19:19:12.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.138+0000 7f175aff5640 1 -- 172.21.15.29:0/36306369 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55bb14d76900 con 0x55bb14de8710 2024-03-11T19:19:12.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.139+0000 7f175aff5640 1 -- 172.21.15.29:0/36306369 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f176c06b050 con 0x55bb14de8710 2024-03-11T19:19:12.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.139+0000 7f177fff2740 10 client.17673 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:12.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.139+0000 7f177fff2740 10 client.17673 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:19:12.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.139+0000 7f177fff2740 20 client.17673 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:12.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.139+0000 7f177fff2740 1 --2- 172.21.15.29:0/36306369 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55bb14dfeb10 0x55bb14e58b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.139+0000 7f177fff2740 4 client.17673 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:19:12.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.139+0000 7f177fff2740 1 -- 172.21.15.29:0/36306369 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55bb14a50860 con 0x55bb14dfeb10 2024-03-11T19:19:12.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.139+0000 7f1776ffd640 1 --2- 172.21.15.29:0/36306369 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55bb14dfeb10 0x55bb14e58b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.139+0000 7f175aff5640 10 client.17673 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:12.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.139+0000 7f1776ffd640 1 --2- 172.21.15.29:0/36306369 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55bb14dfeb10 0x55bb14e58b90 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a97c8291b4d0ed99 server_cookie=2c234aa15164e7c2 in_seq=0 out_seq=0 2024-03-11T19:19:12.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.140+0000 7f175aff5640 1 -- 172.21.15.29:0/36306369 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55bb14a50860 con 0x55bb14dfeb10 2024-03-11T19:19:12.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.140+0000 7f175aff5640 10 client.17673 handle_command_reply: tid=0 2024-03-11T19:19:12.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.140+0000 7f177fff2740 1 client.17673 shutdown 2024-03-11T19:19:12.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.141+0000 7f177fff2740 1 -- 172.21.15.29:0/36306369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14de8710 msgr2=0x55bb14df8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:12.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.141+0000 7f177fff2740 1 --2- 172.21.15.29:0/36306369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14de8710 0x55bb14df8780 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f176c0059d0 tx=0x7f176c052040 comp rx=0 tx=0).stop 2024-03-11T19:19:12.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.141+0000 7f177fff2740 1 -- 172.21.15.29:0/36306369 shutdown_connections 2024-03-11T19:19:12.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.141+0000 7f177fff2740 1 --2- 172.21.15.29:0/36306369 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb14dea2a0 0x55bb14df9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.141+0000 7f177fff2740 1 --2- 172.21.15.29:0/36306369 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb14de98e0 0x55bb14df8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.141+0000 7f177fff2740 1 --2- 172.21.15.29:0/36306369 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55bb14dfeb10 0x55bb14e58b90 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.141+0000 7f177fff2740 1 --2- 172.21.15.29:0/36306369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb14de8710 0x55bb14df8780 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.141+0000 7f177fff2740 1 -- 172.21.15.29:0/36306369 >> 172.21.15.29:0/36306369 conn(0x55bb14de38f0 msgr2=0x55bb14de73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:12.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.141+0000 7f175aff5640 0 client.17673 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:12.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.141+0000 7f177fff2740 1 -- 172.21.15.29:0/36306369 shutdown_connections 2024-03-11T19:19:12.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.141+0000 7f177fff2740 1 -- 172.21.15.29:0/36306369 wait complete. 2024-03-11T19:19:12.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.141+0000 7f177fff2740 20 client.17673 trim_cache size 0 max 16384 2024-03-11T19:19:12.148 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:19:12.148 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:19:12.148 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "f86ca28a-38a6-413c-bfa7-26cb453f57c2", 2024-03-11T19:19:12.148 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:19:12.148 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:19:12.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.142+0000 7f1777fff640 1 -- 172.21.15.29:0/3942586523 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f17600a1bb0 msgr2=0x7f17600a4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:12.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.142+0000 7f1777fff640 1 --2- 172.21.15.29:0/3942586523 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f17600a1bb0 0x7f17600a4070 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f176809a190 tx=0x7f17680cd2f0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.142+0000 7f1777fff640 1 -- 172.21.15.29:0/3942586523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1778109120 msgr2=0x7f17781ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:12.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.142+0000 7f1777fff640 1 --2- 172.21.15.29:0/3942586523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1778109120 0x7f17781ad310 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f176800e970 tx=0x7f176800ee30 comp rx=0 tx=0).stop 2024-03-11T19:19:12.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.142+0000 7f1777fff640 1 -- 172.21.15.29:0/3942586523 shutdown_connections 2024-03-11T19:19:12.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.142+0000 7f1777fff640 1 --2- 172.21.15.29:0/3942586523 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f17600a1bb0 0x7f17600a4070 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.142+0000 7f1777fff640 1 --2- 172.21.15.29:0/3942586523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1778109120 0x7f17781ad310 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.142+0000 7f1777fff640 1 --2- 172.21.15.29:0/3942586523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1778108780 0x7f17781a62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.142+0000 7f1777fff640 1 --2- 172.21.15.29:0/3942586523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17780724d0 0x7f17781a5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.142+0000 7f1777fff640 1 -- 172.21.15.29:0/3942586523 >> 172.21.15.29:0/3942586523 conn(0x7f177806dfa0 msgr2=0x7f177806fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:12.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.143+0000 7f1777fff640 1 -- 172.21.15.29:0/3942586523 shutdown_connections 2024-03-11T19:19:12.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.143+0000 7f1777fff640 1 -- 172.21.15.29:0/3942586523 wait complete. 2024-03-11T19:19:12.168 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:19:12.168 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:19:12.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.394+0000 7fb7f4455640 1 -- 172.21.15.29:0/1926449138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7ec0788a0 msgr2=0x7fb7ec07ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:12.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.394+0000 7fb7f4455640 1 --2- 172.21.15.29:0/1926449138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7ec0788a0 0x7fb7ec07ad30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4008570 tx=0x7fb7e400ac00 comp rx=0 tx=0).stop 2024-03-11T19:19:12.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.395+0000 7fb7f4455640 1 -- 172.21.15.29:0/1926449138 shutdown_connections 2024-03-11T19:19:12.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.395+0000 7fb7f4455640 1 --2- 172.21.15.29:0/1926449138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7ec0788a0 0x7fb7ec07ad30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.395+0000 7fb7f4455640 1 --2- 172.21.15.29:0/1926449138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7ec075ef0 0x7fb7ec078360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.395+0000 7fb7f4455640 1 --2- 172.21.15.29:0/1926449138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7ec073010 0x7fb7ec073430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.395+0000 7fb7f4455640 1 -- 172.21.15.29:0/1926449138 >> 172.21.15.29:0/1926449138 conn(0x7fb7ec06e200 msgr2=0x7fb7ec070640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:12.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.395+0000 7fb7f4455640 1 -- 172.21.15.29:0/1926449138 shutdown_connections 2024-03-11T19:19:12.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.395+0000 7fb7f4455640 1 -- 172.21.15.29:0/1926449138 wait complete. 2024-03-11T19:19:12.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.395+0000 7fb7f4455640 1 Processor -- start 2024-03-11T19:19:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.395+0000 7fb7f4455640 1 -- start start 2024-03-11T19:19:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.396+0000 7fb7f4455640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7ec073010 0x7fb7ec1b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.396+0000 7fb7f4455640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7ec075ef0 0x7fb7ec1b3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.396+0000 7fb7f4455640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7ec0788a0 0x7fb7ec1b59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.396+0000 7fb7f4455640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec1b6140 con 0x7fb7ec075ef0 2024-03-11T19:19:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.396+0000 7fb7f4455640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec1b62b0 con 0x7fb7ec073010 2024-03-11T19:19:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.396+0000 7fb7f4455640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec1b6470 con 0x7fb7ec0788a0 2024-03-11T19:19:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.396+0000 7fb7f29f2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7ec0788a0 0x7fb7ec1b59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.396+0000 7fb7f29f2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7ec0788a0 0x7fb7ec1b59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48214/0 (socket says 172.21.15.29:48214) 2024-03-11T19:19:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.396+0000 7fb7f29f2640 1 -- 172.21.15.29:0/2789958109 learned_addr learned my addr 172.21.15.29:0/2789958109 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.396+0000 7fb7f21f1640 1 --2- 172.21.15.29:0/2789958109 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7ec073010 0x7fb7ec1b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.396+0000 7fb7f19f0640 1 --2- 172.21.15.29:0/2789958109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7ec075ef0 0x7fb7ec1b3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.396+0000 7fb7f29f2640 1 -- 172.21.15.29:0/2789958109 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7ec073010 msgr2=0x7fb7ec1b0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.396+0000 7fb7f29f2640 1 --2- 172.21.15.29:0/2789958109 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7ec073010 0x7fb7ec1b0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.396+0000 7fb7f29f2640 1 -- 172.21.15.29:0/2789958109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7ec075ef0 msgr2=0x7fb7ec1b3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.396+0000 7fb7f29f2640 1 --2- 172.21.15.29:0/2789958109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7ec075ef0 0x7fb7ec1b3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.396+0000 7fb7f29f2640 1 -- 172.21.15.29:0/2789958109 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e40081d0 con 0x7fb7ec0788a0 2024-03-11T19:19:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.396+0000 7fb7f21f1640 1 --2- 172.21.15.29:0/2789958109 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7ec073010 0x7fb7ec1b0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.397+0000 7fb7f29f2640 1 --2- 172.21.15.29:0/2789958109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7ec0788a0 0x7fb7ec1b59b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4005f30 tx=0x7fb7e4004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.397+0000 7fb7e37fe640 1 -- 172.21.15.29:0/2789958109 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e400a800 con 0x7fb7ec0788a0 2024-03-11T19:19:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.397+0000 7fb7e37fe640 1 -- 172.21.15.29:0/2789958109 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb7e4005360 con 0x7fb7ec0788a0 2024-03-11T19:19:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.397+0000 7fb7e37fe640 1 -- 172.21.15.29:0/2789958109 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4003850 con 0x7fb7ec0788a0 2024-03-11T19:19:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.397+0000 7fb7f4455640 1 -- 172.21.15.29:0/2789958109 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ec1b66f0 con 0x7fb7ec0788a0 2024-03-11T19:19:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.397+0000 7fb7f4455640 1 -- 172.21.15.29:0/2789958109 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ec1b6bb0 con 0x7fb7ec0788a0 2024-03-11T19:19:12.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.401+0000 7fb7e37fe640 1 -- 172.21.15.29:0/2789958109 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb7e4004040 con 0x7fb7ec0788a0 2024-03-11T19:19:12.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.402+0000 7fb7f702b740 1 Processor -- start 2024-03-11T19:19:12.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.402+0000 7fb7f702b740 1 -- start start 2024-03-11T19:19:12.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.402+0000 7fb7e37fe640 1 --2- 172.21.15.29:0/2789958109 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7d80a1a30 0x7fb7d80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.402+0000 7fb7f702b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd760f02d0 0x55cd760f06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.402+0000 7fb7f702b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f0cc0 0x55cd760f57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.402+0000 7fb7f702b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f5e30 0x55cd760f82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.402+0000 7fb7f702b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd76060280 con 0x55cd760f5e30 2024-03-11T19:19:12.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.402+0000 7fb7f702b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd7605d7b0 con 0x55cd760f02d0 2024-03-11T19:19:12.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.402+0000 7fb7f702b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd76058f90 con 0x55cd760f0cc0 2024-03-11T19:19:12.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.402+0000 7fb7e37fe640 1 -- 172.21.15.29:0/2789958109 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb7e40ef110 con 0x7fb7ec0788a0 2024-03-11T19:19:12.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.402+0000 7fb7f21f1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd760f02d0 0x55cd760f06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.402+0000 7fb7f19f0640 1 --2- 172.21.15.29:0/2789958109 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7d80a1a30 0x7fb7d80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.402+0000 7fb7f19f0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f0cc0 0x55cd760f57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.402+0000 7fb7f29f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f5e30 0x55cd760f82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.403+0000 7fb7f19f0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f0cc0 0x55cd760f57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48230/0 (socket says 172.21.15.29:48230) 2024-03-11T19:19:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.403+0000 7fb7f19f0640 1 -- 172.21.15.29:0/3253134218 learned_addr learned my addr 172.21.15.29:0/3253134218 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.403+0000 7fb7f29f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f5e30 0x55cd760f82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53940/0 (socket says 172.21.15.29:53940) 2024-03-11T19:19:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.403+0000 7fb7f21f1640 1 --2- 172.21.15.29:0/3253134218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd760f02d0 0x55cd760f06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56108/0 (socket says 172.21.15.29:56108) 2024-03-11T19:19:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.403+0000 7fb7f19f0640 1 -- 172.21.15.29:0/3253134218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd760f02d0 msgr2=0x55cd760f06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.403+0000 7fb7f19f0640 1 --2- 172.21.15.29:0/3253134218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd760f02d0 0x55cd760f06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.403+0000 7fb7f19f0640 1 -- 172.21.15.29:0/3253134218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f5e30 msgr2=0x55cd760f82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.403+0000 7fb7f19f0640 1 --2- 172.21.15.29:0/3253134218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f5e30 0x55cd760f82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.403+0000 7fb7f19f0640 1 -- 172.21.15.29:0/3253134218 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd76017570 con 0x55cd760f0cc0 2024-03-11T19:19:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.403+0000 7fb7f29f2640 1 --2- 172.21.15.29:0/3253134218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f5e30 0x55cd760f82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.403+0000 7fb7f19f0640 1 --2- 172.21.15.29:0/3253134218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f0cc0 0x55cd760f57c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800bef0 tx=0x7fb7e801d740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e1a410722de0333 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.403+0000 7fb7f19f0640 1 --2- 172.21.15.29:0/2789958109 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7d80a1a30 0x7fb7d80a3ef0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb7e801dc40 tx=0x7fb7e8013000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.404+0000 7fb7e0bfe640 1 -- 172.21.15.29:0/3253134218 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e801fe60 con 0x55cd760f0cc0 2024-03-11T19:19:12.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.404+0000 7fb7e0bfe640 1 -- 172.21.15.29:0/3253134218 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb7e8013070 con 0x55cd760f0cc0 2024-03-11T19:19:12.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.404+0000 7fb7e0bfe640 1 -- 172.21.15.29:0/3253134218 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8004630 con 0x55cd760f0cc0 2024-03-11T19:19:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.404+0000 7fb7f702b740 1 -- 172.21.15.29:0/3253134218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f0cc0 msgr2=0x55cd760f57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.404+0000 7fb7f702b740 1 --2- 172.21.15.29:0/3253134218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f0cc0 0x55cd760f57c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800bef0 tx=0x7fb7e801d740 comp rx=0 tx=0).stop 2024-03-11T19:19:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.404+0000 7fb7f702b740 1 -- 172.21.15.29:0/3253134218 shutdown_connections 2024-03-11T19:19:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.404+0000 7fb7f702b740 1 --2- 172.21.15.29:0/3253134218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f5e30 0x55cd760f82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.404+0000 7fb7f702b740 1 --2- 172.21.15.29:0/3253134218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f0cc0 0x55cd760f57c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.404+0000 7fb7f702b740 1 --2- 172.21.15.29:0/3253134218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd760f02d0 0x55cd760f06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.404+0000 7fb7f702b740 1 -- 172.21.15.29:0/3253134218 >> 172.21.15.29:0/3253134218 conn(0x55cd760eb7f0 msgr2=0x55cd760ebbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.405+0000 7fb7f702b740 1 -- 172.21.15.29:0/3253134218 shutdown_connections 2024-03-11T19:19:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.405+0000 7fb7f702b740 1 -- 172.21.15.29:0/3253134218 wait complete. 2024-03-11T19:19:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.405+0000 7fb7f702b740 1 -- start start 2024-03-11T19:19:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.405+0000 7fb7f702b740 1 Processor -- start 2024-03-11T19:19:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.405+0000 7fb7f702b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd760f02d0 0x55cd76102ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.405+0000 7fb7f702b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f0cc0 0x55cd76103000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.405+0000 7fb7f702b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f5e30 0x55cd7610a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.405+0000 7fb7f702b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd75d2c430 con 0x55cd760f0cc0 2024-03-11T19:19:12.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.406+0000 7fb7f702b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd75ce8e00 con 0x55cd760f02d0 2024-03-11T19:19:12.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.406+0000 7fb7f702b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd75fddfd0 con 0x55cd760f5e30 2024-03-11T19:19:12.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.406+0000 7fb7f702b740 10 client.0 _ll_register_callbacks cb 0x55cd76033fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:12.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.406+0000 7fb7f19f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f0cc0 0x55cd76103000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.406+0000 7fb7f21f1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd760f02d0 0x55cd76102ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.406+0000 7fb7f21f1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd760f02d0 0x55cd76102ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56122/0 (socket says 172.21.15.29:56122) 2024-03-11T19:19:12.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.406+0000 7fb7f19f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f0cc0 0x55cd76103000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53954/0 (socket says 172.21.15.29:53954) 2024-03-11T19:19:12.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.406+0000 7fb7f21f1640 1 -- 172.21.15.29:0/847005896 learned_addr learned my addr 172.21.15.29:0/847005896 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:12.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.406+0000 7fb7f29f2640 1 --2- 172.21.15.29:0/847005896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f5e30 0x55cd7610a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.406+0000 7fb7f29f2640 1 -- 172.21.15.29:0/847005896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd760f02d0 msgr2=0x55cd76102ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:12.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.406+0000 7fb7f29f2640 1 --2- 172.21.15.29:0/847005896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd760f02d0 0x55cd76102ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.406+0000 7fb7f29f2640 1 -- 172.21.15.29:0/847005896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f0cc0 msgr2=0x55cd76103000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.406+0000 7fb7f29f2640 1 --2- 172.21.15.29:0/847005896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f0cc0 0x55cd76103000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.406+0000 7fb7f29f2640 1 -- 172.21.15.29:0/847005896 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e800baa0 con 0x55cd760f5e30 2024-03-11T19:19:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.406+0000 7fb7f29f2640 1 --2- 172.21.15.29:0/847005896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f5e30 0x55cd7610a040 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb7e40ecd00 tx=0x7fb7e40bbd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6658237f2fa5f0ef server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.406+0000 7fb7d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:19:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.407+0000 7fb7d5ffb640 1 -- 172.21.15.29:0/847005896 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e402f020 con 0x55cd760f5e30 2024-03-11T19:19:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.407+0000 7fb7d5ffb640 1 -- 172.21.15.29:0/847005896 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb7e40c0270 con 0x55cd760f5e30 2024-03-11T19:19:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.407+0000 7fb7f702b740 1 -- 172.21.15.29:0/847005896 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cd75d587e0 con 0x55cd760f5e30 2024-03-11T19:19:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.407+0000 7fb7d5ffb640 1 -- 172.21.15.29:0/847005896 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7e40c48c0 con 0x55cd760f5e30 2024-03-11T19:19:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.407+0000 7fb7f702b740 10 client.27214 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:12.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.407+0000 7fb7f702b740 1 -- 172.21.15.29:0/847005896 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55cd76057610 con 0x55cd760f5e30 2024-03-11T19:19:12.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.407+0000 7fb7d5ffb640 1 -- 172.21.15.29:0/847005896 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55cd75d587e0 con 0x55cd760f5e30 2024-03-11T19:19:12.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.407+0000 7fb7d5ffb640 1 -- 172.21.15.29:0/847005896 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb7e403b070 con 0x55cd760f5e30 2024-03-11T19:19:12.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.407+0000 7fb7f702b740 10 client.27214 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:12.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.407+0000 7fb7f702b740 10 client.27214 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:19:12.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.407+0000 7fb7f702b740 20 client.27214 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:12.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.407+0000 7fb7f702b740 1 --2- 172.21.15.29:0/847005896 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cd7610eb20 0x55cd76110f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.407+0000 7fb7f702b740 4 client.27214 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:19:12.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.407+0000 7fb7f702b740 1 -- 172.21.15.29:0/847005896 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cd75d95720 con 0x55cd7610eb20 2024-03-11T19:19:12.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.408+0000 7fb7f21f1640 1 --2- 172.21.15.29:0/847005896 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cd7610eb20 0x55cd76110f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.408+0000 7fb7d5ffb640 10 client.27214 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:12.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.408+0000 7fb7f21f1640 1 --2- 172.21.15.29:0/847005896 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cd7610eb20 0x55cd76110f40 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=516f71361126b5bd server_cookie=8b92bfc2f08e9214 in_seq=0 out_seq=0 2024-03-11T19:19:12.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.409+0000 7fb7d5ffb640 1 -- 172.21.15.29:0/847005896 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cd75d95720 con 0x55cd7610eb20 2024-03-11T19:19:12.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.409+0000 7fb7d5ffb640 10 client.27214 handle_command_reply: tid=0 2024-03-11T19:19:12.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.409+0000 7fb7f702b740 1 client.27214 shutdown 2024-03-11T19:19:12.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.410+0000 7fb7f702b740 1 -- 172.21.15.29:0/847005896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f5e30 msgr2=0x55cd7610a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:12.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.410+0000 7fb7f702b740 1 --2- 172.21.15.29:0/847005896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f5e30 0x55cd7610a040 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb7e40ecd00 tx=0x7fb7e40bbd20 comp rx=0 tx=0).stop 2024-03-11T19:19:12.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.410+0000 7fb7f702b740 1 -- 172.21.15.29:0/847005896 shutdown_connections 2024-03-11T19:19:12.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.410+0000 7fb7f702b740 1 --2- 172.21.15.29:0/847005896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f5e30 0x55cd7610a040 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.410+0000 7fb7f702b740 1 --2- 172.21.15.29:0/847005896 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cd7610eb20 0x55cd76110f40 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.410+0000 7fb7f702b740 1 --2- 172.21.15.29:0/847005896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f0cc0 0x55cd76103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.410+0000 7fb7f702b740 1 --2- 172.21.15.29:0/847005896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd760f02d0 0x55cd76102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.410+0000 7fb7f702b740 1 -- 172.21.15.29:0/847005896 >> 172.21.15.29:0/847005896 conn(0x55cd760eb7f0 msgr2=0x55cd760eaa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:12.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.410+0000 7fb7d5ffb640 0 client.27214 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:12.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.410+0000 7fb7f702b740 1 -- 172.21.15.29:0/847005896 shutdown_connections 2024-03-11T19:19:12.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.410+0000 7fb7f702b740 1 -- 172.21.15.29:0/847005896 wait complete. 2024-03-11T19:19:12.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.410+0000 7fb7f702b740 20 client.27214 trim_cache size 0 max 16384 2024-03-11T19:19:12.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.440+0000 7fb7f702b740 1 Processor -- start 2024-03-11T19:19:12.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.440+0000 7fb7f702b740 1 -- start start 2024-03-11T19:19:12.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.440+0000 7fb7f702b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f02d0 0x55cd76102ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.440+0000 7fb7f702b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd760f0cc0 0x55cd7610a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.441+0000 7fb7f702b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f5e30 0x55cd76103000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.441+0000 7fb7f702b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd760f5c10 con 0x55cd760f02d0 2024-03-11T19:19:12.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.441+0000 7fb7f702b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd75d95720 con 0x55cd760f0cc0 2024-03-11T19:19:12.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.441+0000 7fb7f702b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd75d587e0 con 0x55cd760f5e30 2024-03-11T19:19:12.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.441+0000 7fb7f19f0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd760f0cc0 0x55cd7610a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.441+0000 7fb7f19f0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd760f0cc0 0x55cd7610a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56138/0 (socket says 172.21.15.29:56138) 2024-03-11T19:19:12.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.441+0000 7fb7f19f0640 1 -- 172.21.15.29:0/793921044 learned_addr learned my addr 172.21.15.29:0/793921044 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:12.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.441+0000 7fb7f29f2640 1 --2- 172.21.15.29:0/793921044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f5e30 0x55cd76103000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.441+0000 7fb7f21f1640 1 --2- 172.21.15.29:0/793921044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f02d0 0x55cd76102ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.441+0000 7fb7f19f0640 1 -- 172.21.15.29:0/793921044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f5e30 msgr2=0x55cd76103000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:12.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.441+0000 7fb7f19f0640 1 --2- 172.21.15.29:0/793921044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f5e30 0x55cd76103000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.441+0000 7fb7f19f0640 1 -- 172.21.15.29:0/793921044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f02d0 msgr2=0x55cd76102ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:12.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.441+0000 7fb7f19f0640 1 --2- 172.21.15.29:0/793921044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f02d0 0x55cd76102ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.441+0000 7fb7f19f0640 1 -- 172.21.15.29:0/793921044 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd75ff3ac0 con 0x55cd760f0cc0 2024-03-11T19:19:12.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.441+0000 7fb7f19f0640 1 --2- 172.21.15.29:0/793921044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd760f0cc0 0x55cd7610a000 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800dfd0 tx=0x7fb7e8005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17d9c1a9d30c5342 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:12.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.442+0000 7fb7d7fff640 1 -- 172.21.15.29:0/793921044 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e800b3c0 con 0x55cd760f0cc0 2024-03-11T19:19:12.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.442+0000 7fb7d7fff640 1 -- 172.21.15.29:0/793921044 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb7e8003e20 con 0x55cd760f0cc0 2024-03-11T19:19:12.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.442+0000 7fb7d7fff640 1 -- 172.21.15.29:0/793921044 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8027450 con 0x55cd760f0cc0 2024-03-11T19:19:12.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.443+0000 7fb7f702b740 1 -- 172.21.15.29:0/793921044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd760f0cc0 msgr2=0x55cd7610a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:12.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.443+0000 7fb7f702b740 1 --2- 172.21.15.29:0/793921044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd760f0cc0 0x55cd7610a000 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800dfd0 tx=0x7fb7e8005fd0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.443+0000 7fb7f702b740 1 -- 172.21.15.29:0/793921044 shutdown_connections 2024-03-11T19:19:12.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.443+0000 7fb7f702b740 1 --2- 172.21.15.29:0/793921044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f5e30 0x55cd76103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.443+0000 7fb7f702b740 1 --2- 172.21.15.29:0/793921044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd760f0cc0 0x55cd7610a000 secure :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800dfd0 tx=0x7fb7e8005fd0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.443+0000 7fb7f702b740 1 --2- 172.21.15.29:0/793921044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f02d0 0x55cd76102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.443+0000 7fb7f702b740 1 -- 172.21.15.29:0/793921044 >> 172.21.15.29:0/793921044 conn(0x55cd760eb7f0 msgr2=0x55cd761158a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:12.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.444+0000 7fb7f702b740 1 -- 172.21.15.29:0/793921044 shutdown_connections 2024-03-11T19:19:12.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.444+0000 7fb7f702b740 1 -- 172.21.15.29:0/793921044 wait complete. 2024-03-11T19:19:12.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.444+0000 7fb7f702b740 1 -- start start 2024-03-11T19:19:12.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.445+0000 7fb7f702b740 1 Processor -- start 2024-03-11T19:19:12.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.445+0000 7fb7f702b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f02d0 0x55cd76108a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.445+0000 7fb7f702b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f5e30 0x55cd76108f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.445+0000 7fb7f21f1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f02d0 0x55cd76108a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.446+0000 7fb7f21f1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f02d0 0x55cd76108a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:48250/0 (socket says 172.21.15.29:48250) 2024-03-11T19:19:12.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.446+0000 7fb7f702b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd761094c0 0x55cd76165010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.446+0000 7fb7f702b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd76165550 con 0x55cd760f5e30 2024-03-11T19:19:12.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.446+0000 7fb7f702b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd761656c0 con 0x55cd761094c0 2024-03-11T19:19:12.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.446+0000 7fb7f702b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd76165880 con 0x55cd760f02d0 2024-03-11T19:19:12.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.446+0000 7fb7f702b740 10 client.0 _ll_register_callbacks cb 0x7fb7e800dd20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:12.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.446+0000 7fb7f21f1640 1 -- 172.21.15.29:0/3731954007 learned_addr learned my addr 172.21.15.29:0/3731954007 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.446+0000 7fb7f29f2640 1 --2- 172.21.15.29:0/3731954007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd761094c0 0x55cd76165010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.446+0000 7fb7f19f0640 1 --2- 172.21.15.29:0/3731954007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f5e30 0x55cd76108f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.446+0000 7fb7f29f2640 1 -- 172.21.15.29:0/3731954007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f02d0 msgr2=0x55cd76108a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.446+0000 7fb7f29f2640 1 --2- 172.21.15.29:0/3731954007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f02d0 0x55cd76108a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.446+0000 7fb7f29f2640 1 -- 172.21.15.29:0/3731954007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f5e30 msgr2=0x55cd76108f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.446+0000 7fb7f29f2640 1 --2- 172.21.15.29:0/3731954007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f5e30 0x55cd76108f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.446+0000 7fb7f29f2640 1 -- 172.21.15.29:0/3731954007 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e8020e70 con 0x55cd761094c0 2024-03-11T19:19:12.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.448+0000 7fb7f29f2640 1 --2- 172.21.15.29:0/3731954007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd761094c0 0x55cd76165010 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb7e40b8af0 tx=0x7fb7e400ac70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2083a562c3ea942 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:12.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.448+0000 7fb7d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:19:12.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.448+0000 7fb7d5ffb640 1 -- 172.21.15.29:0/3731954007 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e402f070 con 0x55cd761094c0 2024-03-11T19:19:12.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.448+0000 7fb7d5ffb640 1 -- 172.21.15.29:0/3731954007 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fb7e40d8020 con 0x55cd761094c0 2024-03-11T19:19:12.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.448+0000 7fb7f702b740 1 -- 172.21.15.29:0/3731954007 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7e8013070 con 0x55cd761094c0 2024-03-11T19:19:12.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.448+0000 7fb7d5ffb640 1 -- 172.21.15.29:0/3731954007 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7e40bbef0 con 0x55cd761094c0 2024-03-11T19:19:12.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.448+0000 7fb7f702b740 10 client.27182 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:12.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.449+0000 7fb7f702b740 1 -- 172.21.15.29:0/3731954007 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55cd760196a0 con 0x55cd761094c0 2024-03-11T19:19:12.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.449+0000 7fb7d5ffb640 1 -- 172.21.15.29:0/3731954007 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb7e40c0040 con 0x55cd761094c0 2024-03-11T19:19:12.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.449+0000 7fb7d5ffb640 1 -- 172.21.15.29:0/3731954007 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb7e403b030 con 0x55cd761094c0 2024-03-11T19:19:12.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.449+0000 7fb7f702b740 10 client.27182 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:12.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.449+0000 7fb7f702b740 10 client.27182 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:19:12.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.449+0000 7fb7f702b740 20 client.27182 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:12.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.449+0000 7fb7f702b740 1 --2- 172.21.15.29:0/3731954007 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cd76166e70 0x55cd76169260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:12.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.450+0000 7fb7f702b740 4 client.27182 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:19:12.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.450+0000 7fb7f702b740 1 -- 172.21.15.29:0/3731954007 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7fb7e8013070 con 0x55cd76166e70 2024-03-11T19:19:12.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.450+0000 7fb7f21f1640 1 --2- 172.21.15.29:0/3731954007 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cd76166e70 0x55cd76169260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:12.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.450+0000 7fb7d5ffb640 10 client.27182 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:12.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.451+0000 7fb7f21f1640 1 --2- 172.21.15.29:0/3731954007 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cd76166e70 0x55cd76169260 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=576f82d1562e6558 server_cookie=d899c5594c73470a in_seq=0 out_seq=0 2024-03-11T19:19:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.451+0000 7fb7d5ffb640 1 -- 172.21.15.29:0/3731954007 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7fb7e8013070 con 0x55cd76166e70 2024-03-11T19:19:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.451+0000 7fb7d5ffb640 10 client.27182 handle_command_reply: tid=0 2024-03-11T19:19:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.451+0000 7fb7f702b740 1 client.27182 shutdown 2024-03-11T19:19:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.452+0000 7fb7f702b740 1 -- 172.21.15.29:0/3731954007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd761094c0 msgr2=0x55cd76165010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.452+0000 7fb7f702b740 1 --2- 172.21.15.29:0/3731954007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd761094c0 0x55cd76165010 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb7e40b8af0 tx=0x7fb7e400ac70 comp rx=0 tx=0).stop 2024-03-11T19:19:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.452+0000 7fb7f702b740 1 -- 172.21.15.29:0/3731954007 shutdown_connections 2024-03-11T19:19:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.452+0000 7fb7f702b740 1 --2- 172.21.15.29:0/3731954007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd761094c0 0x55cd76165010 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.452+0000 7fb7f702b740 1 --2- 172.21.15.29:0/3731954007 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55cd76166e70 0x55cd76169260 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.452+0000 7fb7f702b740 1 --2- 172.21.15.29:0/3731954007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd760f5e30 0x55cd76108f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.452+0000 7fb7f702b740 1 --2- 172.21.15.29:0/3731954007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cd760f02d0 0x55cd76108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.452+0000 7fb7d5ffb640 0 client.27182 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:12.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.452+0000 7fb7f702b740 1 -- 172.21.15.29:0/3731954007 >> 172.21.15.29:0/3731954007 conn(0x55cd760eb7f0 msgr2=0x55cd760f2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:12.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.452+0000 7fb7f702b740 1 -- 172.21.15.29:0/3731954007 shutdown_connections 2024-03-11T19:19:12.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.452+0000 7fb7f702b740 1 -- 172.21.15.29:0/3731954007 wait complete. 2024-03-11T19:19:12.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.452+0000 7fb7f702b740 20 client.27182 trim_cache size 0 max 16384 2024-03-11T19:19:12.458 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:19:12.458 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:19:12.458 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:19:12.458 INFO:teuthology.orchestra.run.smithi029.stdout: "f86ca28a-38a6-413c-bfa7-26cb453f57c2": { 2024-03-11T19:19:12.458 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:19:12.459 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "f86ca28a-38a6-413c-bfa7-26cb453f57c2", 2024-03-11T19:19:12.459 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:19:12.459 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:19:12.459 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:19:12.459 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:19:12.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.454+0000 7fb7f4455640 1 -- 172.21.15.29:0/2789958109 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7d80a1a30 msgr2=0x7fb7d80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:12.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.454+0000 7fb7f4455640 1 --2- 172.21.15.29:0/2789958109 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7d80a1a30 0x7fb7d80a3ef0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb7e801dc40 tx=0x7fb7e8013000 comp rx=0 tx=0).stop 2024-03-11T19:19:12.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.454+0000 7fb7f4455640 1 -- 172.21.15.29:0/2789958109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7ec0788a0 msgr2=0x7fb7ec1b59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:12.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.454+0000 7fb7f4455640 1 --2- 172.21.15.29:0/2789958109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7ec0788a0 0x7fb7ec1b59b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4005f30 tx=0x7fb7e4004960 comp rx=0 tx=0).stop 2024-03-11T19:19:12.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.454+0000 7fb7f4455640 1 -- 172.21.15.29:0/2789958109 shutdown_connections 2024-03-11T19:19:12.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.454+0000 7fb7f4455640 1 --2- 172.21.15.29:0/2789958109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7ec0788a0 0x7fb7ec1b59b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.454+0000 7fb7f4455640 1 --2- 172.21.15.29:0/2789958109 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fb7d80a1a30 0x7fb7d80a3ef0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.454+0000 7fb7f4455640 1 --2- 172.21.15.29:0/2789958109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7ec075ef0 0x7fb7ec1b3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.454+0000 7fb7f4455640 1 --2- 172.21.15.29:0/2789958109 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7ec073010 0x7fb7ec1b0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:12.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.454+0000 7fb7f4455640 1 -- 172.21.15.29:0/2789958109 >> 172.21.15.29:0/2789958109 conn(0x7fb7ec06e200 msgr2=0x7fb7ec0780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:12.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.455+0000 7fb7f4455640 1 -- 172.21.15.29:0/2789958109 shutdown_connections 2024-03-11T19:19:12.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:12.455+0000 7fb7f4455640 1 -- 172.21.15.29:0/2789958109 wait complete. 2024-03-11T19:19:12.476 INFO:tasks.cephfs.filesystem:scrub status for tag:f86ca28a-38a6-413c-bfa7-26cb453f57c2 - {'path': '/', 'tag': 'f86ca28a-38a6-413c-bfa7-26cb453f57c2', 'options': 'recursive,force'} 2024-03-11T19:19:13.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:13 smithi029 ceph-mon[32439]: pgmap v633: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 61 MiB/s wr, 1.20k op/s 2024-03-11T19:19:13.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:13 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:19:13.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:13 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:19:13.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:13 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:19:13.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:13 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:19:14.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:13 smithi190 ceph-mon[36292]: pgmap v633: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 61 MiB/s wr, 1.20k op/s 2024-03-11T19:19:14.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:13 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:19:14.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:13 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:19:14.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:13 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:19:14.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:13 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:19:14.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:13 smithi170 ceph-mon[35690]: pgmap v633: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 61 MiB/s wr, 1.20k op/s 2024-03-11T19:19:14.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:13 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:19:14.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:13 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:19:14.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:13 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:19:14.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:13 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:19:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:15 smithi029 ceph-mon[32439]: pgmap v634: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 56 MiB/s wr, 1.11k op/s 2024-03-11T19:19:16.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:15 smithi190 ceph-mon[36292]: pgmap v634: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 56 MiB/s wr, 1.11k op/s 2024-03-11T19:19:16.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:15 smithi170 ceph-mon[35690]: pgmap v634: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 56 MiB/s wr, 1.11k op/s 2024-03-11T19:19:16.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:16 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:19:17.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:16 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:19:17.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:16 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:19:17.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:17 smithi029 ceph-mon[32439]: pgmap v635: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 60 MiB/s wr, 1.19k op/s 2024-03-11T19:19:18.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:17 smithi190 ceph-mon[36292]: pgmap v635: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 60 MiB/s wr, 1.19k op/s 2024-03-11T19:19:18.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:17 smithi170 ceph-mon[35690]: pgmap v635: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 60 MiB/s wr, 1.19k op/s 2024-03-11T19:19:19.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:19 smithi029 ceph-mon[32439]: pgmap v636: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 52 MiB/s wr, 1.05k op/s 2024-03-11T19:19:20.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:19 smithi190 ceph-mon[36292]: pgmap v636: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 52 MiB/s wr, 1.05k op/s 2024-03-11T19:19:20.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:19 smithi170 ceph-mon[35690]: pgmap v636: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 52 MiB/s wr, 1.05k op/s 2024-03-11T19:19:21.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:21 smithi029 ceph-mon[32439]: pgmap v637: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 57 MiB/s wr, 1.15k op/s 2024-03-11T19:19:22.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:21 smithi190 ceph-mon[36292]: pgmap v637: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 57 MiB/s wr, 1.15k op/s 2024-03-11T19:19:22.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:21 smithi170 ceph-mon[35690]: pgmap v637: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 57 MiB/s wr, 1.15k op/s 2024-03-11T19:19:23.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:23 smithi029 ceph-mon[32439]: pgmap v638: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 57 MiB/s wr, 1.15k op/s 2024-03-11T19:19:24.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:23 smithi190 ceph-mon[36292]: pgmap v638: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 57 MiB/s wr, 1.15k op/s 2024-03-11T19:19:24.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:23 smithi170 ceph-mon[35690]: pgmap v638: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 57 MiB/s wr, 1.15k op/s 2024-03-11T19:19:25.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:25 smithi029 ceph-mon[32439]: pgmap v639: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 54 MiB/s wr, 1.10k op/s 2024-03-11T19:19:26.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:25 smithi190 ceph-mon[36292]: pgmap v639: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 54 MiB/s wr, 1.10k op/s 2024-03-11T19:19:26.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:25 smithi170 ceph-mon[35690]: pgmap v639: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 54 MiB/s wr, 1.10k op/s 2024-03-11T19:19:27.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:27 smithi029 ceph-mon[32439]: pgmap v640: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 60 MiB/s wr, 1.22k op/s 2024-03-11T19:19:28.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:27 smithi190 ceph-mon[36292]: pgmap v640: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 60 MiB/s wr, 1.22k op/s 2024-03-11T19:19:28.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:27 smithi170 ceph-mon[35690]: pgmap v640: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 60 MiB/s wr, 1.22k op/s 2024-03-11T19:19:29.899 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:29 smithi170 ceph-mon[35690]: pgmap v641: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 54 MiB/s wr, 1.11k op/s 2024-03-11T19:19:29.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:29 smithi029 ceph-mon[32439]: pgmap v641: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 54 MiB/s wr, 1.11k op/s 2024-03-11T19:19:30.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:29 smithi190 ceph-mon[36292]: pgmap v641: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 54 MiB/s wr, 1.11k op/s 2024-03-11T19:19:32.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:31 smithi190 ceph-mon[36292]: pgmap v642: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 61 MiB/s wr, 1.24k op/s 2024-03-11T19:19:32.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:31 smithi170 ceph-mon[35690]: pgmap v642: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 61 MiB/s wr, 1.24k op/s 2024-03-11T19:19:32.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:31 smithi029 ceph-mon[32439]: pgmap v642: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 61 MiB/s wr, 1.24k op/s 2024-03-11T19:19:34.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:33 smithi190 ceph-mon[36292]: pgmap v643: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 61 MiB/s wr, 1.24k op/s 2024-03-11T19:19:34.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:33 smithi170 ceph-mon[35690]: pgmap v643: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 61 MiB/s wr, 1.24k op/s 2024-03-11T19:19:34.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:33 smithi029 ceph-mon[32439]: pgmap v643: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 61 MiB/s wr, 1.24k op/s 2024-03-11T19:19:36.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:35 smithi190 ceph-mon[36292]: pgmap v644: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 57 MiB/s wr, 1.17k op/s 2024-03-11T19:19:36.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:35 smithi170 ceph-mon[35690]: pgmap v644: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 57 MiB/s wr, 1.17k op/s 2024-03-11T19:19:36.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:35 smithi029 ceph-mon[32439]: pgmap v644: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 57 MiB/s wr, 1.17k op/s 2024-03-11T19:19:38.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:37 smithi190 ceph-mon[36292]: pgmap v645: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 61 MiB/s wr, 1.26k op/s 2024-03-11T19:19:38.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:37 smithi170 ceph-mon[35690]: pgmap v645: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 61 MiB/s wr, 1.26k op/s 2024-03-11T19:19:38.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:37 smithi029 ceph-mon[32439]: pgmap v645: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 61 MiB/s wr, 1.26k op/s 2024-03-11T19:19:40.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:39 smithi190 ceph-mon[36292]: pgmap v646: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 54 MiB/s wr, 1.11k op/s 2024-03-11T19:19:40.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:39 smithi170 ceph-mon[35690]: pgmap v646: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 54 MiB/s wr, 1.11k op/s 2024-03-11T19:19:40.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:39 smithi029 ceph-mon[32439]: pgmap v646: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 54 MiB/s wr, 1.11k op/s 2024-03-11T19:19:42.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:41 smithi190 ceph-mon[36292]: pgmap v647: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 58 MiB/s wr, 1.20k op/s 2024-03-11T19:19:42.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:41 smithi170 ceph-mon[35690]: pgmap v647: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 58 MiB/s wr, 1.20k op/s 2024-03-11T19:19:42.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:41 smithi029 ceph-mon[32439]: pgmap v647: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 58 MiB/s wr, 1.20k op/s 2024-03-11T19:19:42.477 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:19:42.477 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:19:42.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.662+0000 7fe0a10c3640 1 -- 172.21.15.29:0/3021911048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0940a4c40 msgr2=0x7fe0940a5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:42.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.662+0000 7fe0a10c3640 1 --2- 172.21.15.29:0/3021911048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0940a4c40 0x7fe0940a5060 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe090009900 tx=0x7fe09002bc30 comp rx=0 tx=0).stop 2024-03-11T19:19:42.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.664+0000 7fe0a10c3640 1 -- 172.21.15.29:0/3021911048 shutdown_connections 2024-03-11T19:19:42.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.664+0000 7fe0a10c3640 1 --2- 172.21.15.29:0/3021911048 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0940a6710 0x7fe0940aafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.664+0000 7fe0a10c3640 1 --2- 172.21.15.29:0/3021911048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0940a5d70 0x7fe0940a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.664+0000 7fe0a10c3640 1 --2- 172.21.15.29:0/3021911048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0940a4c40 0x7fe0940a5060 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.664+0000 7fe0a10c3640 1 -- 172.21.15.29:0/3021911048 >> 172.21.15.29:0/3021911048 conn(0x7fe0940a00d0 msgr2=0x7fe0940a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:42.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.664+0000 7fe0a10c3640 1 -- 172.21.15.29:0/3021911048 shutdown_connections 2024-03-11T19:19:42.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.664+0000 7fe0a10c3640 1 -- 172.21.15.29:0/3021911048 wait complete. 2024-03-11T19:19:42.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.665+0000 7fe0a10c3640 1 Processor -- start 2024-03-11T19:19:42.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.665+0000 7fe0a10c3640 1 -- start start 2024-03-11T19:19:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.666+0000 7fe0a10c3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0940a4c40 0x7fe094143310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.666+0000 7fe0a10c3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0940a5d70 0x7fe094143850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.666+0000 7fe0a10c3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0940a6710 0x7fe09414a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.666+0000 7fe0a10c3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09414ae00 con 0x7fe0940a5d70 2024-03-11T19:19:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.666+0000 7fe0a10c3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09414af70 con 0x7fe0940a6710 2024-03-11T19:19:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.666+0000 7fe0a10c3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09414b130 con 0x7fe0940a4c40 2024-03-11T19:19:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.666+0000 7fe09bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0940a4c40 0x7fe094143310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.666+0000 7fe09bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0940a4c40 0x7fe094143310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:60912/0 (socket says 172.21.15.29:60912) 2024-03-11T19:19:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.666+0000 7fe09bfff640 1 -- 172.21.15.29:0/4054091554 learned_addr learned my addr 172.21.15.29:0/4054091554 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.666+0000 7fe0a08c2640 1 --2- 172.21.15.29:0/4054091554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0940a6710 0x7fe09414a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:42.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.667+0000 7fe09bfff640 1 -- 172.21.15.29:0/4054091554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0940a6710 msgr2=0x7fe09414a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:42.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.667+0000 7fe09bfff640 1 --2- 172.21.15.29:0/4054091554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0940a6710 0x7fe09414a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.667+0000 7fe09bfff640 1 -- 172.21.15.29:0/4054091554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0940a5d70 msgr2=0x7fe094143850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:42.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.667+0000 7fe09bfff640 1 --2- 172.21.15.29:0/4054091554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0940a5d70 0x7fe094143850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.667+0000 7fe09bfff640 1 -- 172.21.15.29:0/4054091554 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe090009560 con 0x7fe0940a4c40 2024-03-11T19:19:42.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.667+0000 7fe09bfff640 1 --2- 172.21.15.29:0/4054091554 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0940a4c40 0x7fe094143310 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe090005ab0 tx=0x7fe0900040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:42.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.667+0000 7fe0997fa640 1 -- 172.21.15.29:0/4054091554 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0900044c0 con 0x7fe0940a4c40 2024-03-11T19:19:42.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.667+0000 7fe0997fa640 1 -- 172.21.15.29:0/4054091554 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe090003aa0 con 0x7fe0940a4c40 2024-03-11T19:19:42.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.667+0000 7fe0997fa640 1 -- 172.21.15.29:0/4054091554 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe090005510 con 0x7fe0940a4c40 2024-03-11T19:19:42.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.668+0000 7fe0a10c3640 1 -- 172.21.15.29:0/4054091554 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe09414b3b0 con 0x7fe0940a4c40 2024-03-11T19:19:42.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.668+0000 7fe0a10c3640 1 -- 172.21.15.29:0/4054091554 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe09414b870 con 0x7fe0940a4c40 2024-03-11T19:19:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.672+0000 7fe0a4fd2740 1 Processor -- start 2024-03-11T19:19:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.672+0000 7fe0a4fd2740 1 -- start start 2024-03-11T19:19:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.673+0000 7fe0a4fd2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d374b0 0x55deb8d378d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.673+0000 7fe0a4fd2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d37ea0 0x55deb8d3c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.673+0000 7fe0a4fd2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 0x55deb8d3f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.673+0000 7fe0a4fd2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb8ca7280 con 0x55deb8d37ea0 2024-03-11T19:19:42.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.673+0000 7fe0a4fd2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb8ca47b0 con 0x55deb8d3d010 2024-03-11T19:19:42.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.673+0000 7fe0a4fd2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb8c9ff90 con 0x55deb8d374b0 2024-03-11T19:19:42.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.674+0000 7fe09b7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 0x55deb8d3f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:42.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.674+0000 7fe09b7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 0x55deb8d3f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59348/0 (socket says 172.21.15.29:59348) 2024-03-11T19:19:42.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.674+0000 7fe09b7fe640 1 -- 172.21.15.29:0/3515830184 learned_addr learned my addr 172.21.15.29:0/3515830184 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:42.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.674+0000 7fe09b7fe640 1 -- 172.21.15.29:0/3515830184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d374b0 msgr2=0x55deb8d378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:42.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.674+0000 7fe09b7fe640 1 --2- 172.21.15.29:0/3515830184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d374b0 0x55deb8d378d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.674+0000 7fe09b7fe640 1 -- 172.21.15.29:0/3515830184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d37ea0 msgr2=0x55deb8d3c9a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:19:42.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.674+0000 7fe09b7fe640 1 --2- 172.21.15.29:0/3515830184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d37ea0 0x55deb8d3c9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.674+0000 7fe09b7fe640 1 -- 172.21.15.29:0/3515830184 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55deb8c5e570 con 0x55deb8d3d010 2024-03-11T19:19:42.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.675+0000 7fe09b7fe640 1 --2- 172.21.15.29:0/3515830184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 0x55deb8d3f4a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe088009be0 tx=0x7fe08800e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b651cb3d0ea7bed6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:42.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.676+0000 7fe0867fc640 1 -- 172.21.15.29:0/3515830184 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe088009730 con 0x55deb8d3d010 2024-03-11T19:19:42.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.676+0000 7fe0867fc640 1 -- 172.21.15.29:0/3515830184 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe0880054d0 con 0x55deb8d3d010 2024-03-11T19:19:42.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.676+0000 7fe0867fc640 1 -- 172.21.15.29:0/3515830184 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe088010690 con 0x55deb8d3d010 2024-03-11T19:19:42.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.677+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/3515830184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 msgr2=0x55deb8d3f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:42.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.677+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/3515830184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 0x55deb8d3f4a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe088009be0 tx=0x7fe08800e720 comp rx=0 tx=0).stop 2024-03-11T19:19:42.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.677+0000 7fe0997fa640 1 -- 172.21.15.29:0/4054091554 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe090003c10 con 0x7fe0940a4c40 2024-03-11T19:19:42.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.677+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/3515830184 shutdown_connections 2024-03-11T19:19:42.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.677+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/3515830184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 0x55deb8d3f4a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.677+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/3515830184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d37ea0 0x55deb8d3c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.677+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/3515830184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d374b0 0x55deb8d378d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.677+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/3515830184 >> 172.21.15.29:0/3515830184 conn(0x55deb8d32a20 msgr2=0x55deb8d32e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:42.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.678+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/3515830184 shutdown_connections 2024-03-11T19:19:42.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.678+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/3515830184 wait complete. 2024-03-11T19:19:42.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.678+0000 7fe0997fa640 1 --2- 172.21.15.29:0/4054091554 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe08c0a1a40 0x7fe08c0a3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.678+0000 7fe0997fa640 1 -- 172.21.15.29:0/4054091554 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe09002de40 con 0x7fe0940a4c40 2024-03-11T19:19:42.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.678+0000 7fe0a4fd2740 1 -- start start 2024-03-11T19:19:42.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.678+0000 7fe09b7fe640 1 --2- 172.21.15.29:0/4054091554 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe08c0a1a40 0x7fe08c0a3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:42.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.679+0000 7fe09b7fe640 1 --2- 172.21.15.29:0/4054091554 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe08c0a1a40 0x7fe08c0a3f00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe0880095b0 tx=0x7fe0880046c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:42.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.679+0000 7fe0a4fd2740 1 Processor -- start 2024-03-11T19:19:42.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.680+0000 7fe0a4fd2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d374b0 0x55deb8d412d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.680+0000 7fe0a4fd2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d37ea0 0x55deb8d41810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.680+0000 7fe0a4fd2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 0x55deb8d48850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.680+0000 7fe0a4fd2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb8973430 con 0x55deb8d37ea0 2024-03-11T19:19:42.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.680+0000 7fe0a4fd2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb892fe00 con 0x55deb8d3d010 2024-03-11T19:19:42.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.680+0000 7fe0a4fd2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb8c24fd0 con 0x55deb8d374b0 2024-03-11T19:19:42.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.680+0000 7fe0a4fd2740 10 client.0 _ll_register_callbacks cb 0x55deb8c7afd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:42.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.680+0000 7fe09bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d374b0 0x55deb8d412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:42.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.680+0000 7fe09bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d374b0 0x55deb8d412d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:60928/0 (socket says 172.21.15.29:60928) 2024-03-11T19:19:42.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.680+0000 7fe09bfff640 1 -- 172.21.15.29:0/2022761378 learned_addr learned my addr 172.21.15.29:0/2022761378 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:42.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.680+0000 7fe0a08c2640 1 --2- 172.21.15.29:0/2022761378 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 0x55deb8d48850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:42.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.680+0000 7fe09b7fe640 1 --2- 172.21.15.29:0/2022761378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d37ea0 0x55deb8d41810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:42.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.681+0000 7fe09bfff640 1 -- 172.21.15.29:0/2022761378 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 msgr2=0x55deb8d48850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:42.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.681+0000 7fe09bfff640 1 --2- 172.21.15.29:0/2022761378 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 0x55deb8d48850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.681+0000 7fe09bfff640 1 -- 172.21.15.29:0/2022761378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d37ea0 msgr2=0x55deb8d41810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:42.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.681+0000 7fe09bfff640 1 --2- 172.21.15.29:0/2022761378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d37ea0 0x55deb8d41810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.681+0000 7fe09bfff640 1 -- 172.21.15.29:0/2022761378 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe088009890 con 0x55deb8d374b0 2024-03-11T19:19:42.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.681+0000 7fe0a08c2640 1 --2- 172.21.15.29:0/2022761378 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 0x55deb8d48850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:42.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.681+0000 7fe09b7fe640 1 --2- 172.21.15.29:0/2022761378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d37ea0 0x55deb8d41810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:42.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.681+0000 7fe09bfff640 1 --2- 172.21.15.29:0/2022761378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d374b0 0x55deb8d412d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe0900b9080 tx=0x7fe09003cef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d33bb1e81b4e8e39 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:42.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.681+0000 7fe07b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:19:42.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.681+0000 7fe07b7fe640 1 -- 172.21.15.29:0/2022761378 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0900bcc20 con 0x55deb8d374b0 2024-03-11T19:19:42.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.681+0000 7fe07b7fe640 1 -- 172.21.15.29:0/2022761378 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe0900c0d70 con 0x55deb8d374b0 2024-03-11T19:19:42.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.681+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/2022761378 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55deb899f7e0 con 0x55deb8d374b0 2024-03-11T19:19:42.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.682+0000 7fe07b7fe640 1 -- 172.21.15.29:0/2022761378 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe0900c53c0 con 0x55deb8d374b0 2024-03-11T19:19:42.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.682+0000 7fe0a4fd2740 10 client.27229 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:42.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.682+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/2022761378 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55deb8c9e610 con 0x55deb8d374b0 2024-03-11T19:19:42.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.682+0000 7fe07b7fe640 1 -- 172.21.15.29:0/2022761378 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55deb899f7e0 con 0x55deb8d374b0 2024-03-11T19:19:42.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.683+0000 7fe07b7fe640 1 -- 172.21.15.29:0/2022761378 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe09003c070 con 0x55deb8d374b0 2024-03-11T19:19:42.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.683+0000 7fe0a4fd2740 10 client.27229 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:42.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.683+0000 7fe0a4fd2740 10 client.27229 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:19:42.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.683+0000 7fe0a4fd2740 20 client.27229 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:42.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.683+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/2022761378 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55deb8d4d1b0 0x55deb8d4f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.683+0000 7fe0a4fd2740 4 client.27229 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:19:42.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.683+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/2022761378 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55deb89dc720 con 0x55deb8d4d1b0 2024-03-11T19:19:42.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.683+0000 7fe09b7fe640 1 --2- 172.21.15.29:0/2022761378 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55deb8d4d1b0 0x55deb8d4f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:42.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.683+0000 7fe09b7fe640 1 --2- 172.21.15.29:0/2022761378 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55deb8d4d1b0 0x55deb8d4f5a0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76cfa6ee1a73cce9 server_cookie=152b6f77d898f80e in_seq=0 out_seq=0 2024-03-11T19:19:42.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.683+0000 7fe07b7fe640 10 client.27229 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:42.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.684+0000 7fe07b7fe640 1 -- 172.21.15.29:0/2022761378 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55deb89dc720 con 0x55deb8d4d1b0 2024-03-11T19:19:42.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.684+0000 7fe07b7fe640 10 client.27229 handle_command_reply: tid=0 2024-03-11T19:19:42.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.684+0000 7fe0a4fd2740 1 client.27229 shutdown 2024-03-11T19:19:42.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.687+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/2022761378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d374b0 msgr2=0x55deb8d412d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:42.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.687+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/2022761378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d374b0 0x55deb8d412d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe0900b9080 tx=0x7fe09003cef0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.688+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/2022761378 shutdown_connections 2024-03-11T19:19:42.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.688+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/2022761378 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 0x55deb8d48850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.688+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/2022761378 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55deb8d4d1b0 0x55deb8d4f5a0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.688+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/2022761378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d37ea0 0x55deb8d41810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.688+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/2022761378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d374b0 0x55deb8d412d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.688+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/2022761378 >> 172.21.15.29:0/2022761378 conn(0x55deb8d32a20 msgr2=0x55deb8d31c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:42.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.688+0000 7fe07b7fe640 0 client.27229 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:42.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.688+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/2022761378 shutdown_connections 2024-03-11T19:19:42.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.688+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/2022761378 wait complete. 2024-03-11T19:19:42.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.689+0000 7fe0a4fd2740 20 client.27229 trim_cache size 0 max 16384 2024-03-11T19:19:42.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.729+0000 7fe0a4fd2740 1 Processor -- start 2024-03-11T19:19:42.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.730+0000 7fe0a4fd2740 1 -- start start 2024-03-11T19:19:42.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.730+0000 7fe0a4fd2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d374b0 0x55deb8d3f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.730+0000 7fe0a4fd2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d37ea0 0x55deb8d487e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.730+0000 7fe0a4fd2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 0x55deb8d53f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.730+0000 7fe0a4fd2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb8d494d0 con 0x55deb8d374b0 2024-03-11T19:19:42.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.730+0000 7fe0a4fd2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb8d40b20 con 0x55deb8d3d010 2024-03-11T19:19:42.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.730+0000 7fe0a4fd2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb8d33510 con 0x55deb8d37ea0 2024-03-11T19:19:42.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.733+0000 7fe09bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d374b0 0x55deb8d3f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.733+0000 7fe09bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d374b0 0x55deb8d3f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43930/0 (socket says 172.21.15.29:43930) 2024-03-11T19:19:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.733+0000 7fe09bfff640 1 -- 172.21.15.29:0/1246316297 learned_addr learned my addr 172.21.15.29:0/1246316297 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.734+0000 7fe09bfff640 1 -- 172.21.15.29:0/1246316297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d37ea0 msgr2=0x55deb8d487e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.734+0000 7fe09bfff640 1 --2- 172.21.15.29:0/1246316297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d37ea0 0x55deb8d487e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.734+0000 7fe09bfff640 1 -- 172.21.15.29:0/1246316297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 msgr2=0x55deb8d53f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.734+0000 7fe09bfff640 1 --2- 172.21.15.29:0/1246316297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 0x55deb8d53f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.734+0000 7fe09bfff640 1 -- 172.21.15.29:0/1246316297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55deb8c3aac0 con 0x55deb8d374b0 2024-03-11T19:19:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.736+0000 7fe09bfff640 1 --2- 172.21.15.29:0/1246316297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d374b0 0x55deb8d3f400 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fe09002c210 tx=0x7fe0900bb130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1f531e7496c5517 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.737+0000 7fe085ffb640 1 -- 172.21.15.29:0/1246316297 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0900d9070 con 0x55deb8d374b0 2024-03-11T19:19:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.737+0000 7fe085ffb640 1 -- 172.21.15.29:0/1246316297 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe0900b8ec0 con 0x55deb8d374b0 2024-03-11T19:19:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.737+0000 7fe085ffb640 1 -- 172.21.15.29:0/1246316297 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0900edbd0 con 0x55deb8d374b0 2024-03-11T19:19:42.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.739+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/1246316297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d374b0 msgr2=0x55deb8d3f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:42.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.739+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/1246316297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d374b0 0x55deb8d3f400 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fe09002c210 tx=0x7fe0900bb130 comp rx=0 tx=0).stop 2024-03-11T19:19:42.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.739+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/1246316297 shutdown_connections 2024-03-11T19:19:42.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.739+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/1246316297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 0x55deb8d53f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.739+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/1246316297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d37ea0 0x55deb8d487e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.739+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/1246316297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d374b0 0x55deb8d3f400 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.739+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/1246316297 >> 172.21.15.29:0/1246316297 conn(0x55deb8d32a20 msgr2=0x55deb8d32e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:42.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.739+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/1246316297 shutdown_connections 2024-03-11T19:19:42.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.739+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/1246316297 wait complete. 2024-03-11T19:19:42.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.740+0000 7fe0a4fd2740 1 -- start start 2024-03-11T19:19:42.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.740+0000 7fe0a4fd2740 1 Processor -- start 2024-03-11T19:19:42.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.741+0000 7fe0a4fd2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d374b0 0x55deb8d464f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.741+0000 7fe0a4fd2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d37ea0 0x55deb8d46a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.741+0000 7fe0a4fd2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 0x55deb8d47a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.741+0000 7fe0a4fd2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb8d47f80 con 0x55deb8d374b0 2024-03-11T19:19:42.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.741+0000 7fe0a4fd2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb8d480f0 con 0x55deb8d3d010 2024-03-11T19:19:42.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.741+0000 7fe0a4fd2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55deb8d482b0 con 0x55deb8d37ea0 2024-03-11T19:19:42.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.741+0000 7fe0a4fd2740 10 client.0 _ll_register_callbacks cb 0x7fe088009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:42.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.741+0000 7fe09bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d374b0 0x55deb8d464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:42.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.741+0000 7fe09bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d374b0 0x55deb8d464f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43942/0 (socket says 172.21.15.29:43942) 2024-03-11T19:19:42.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.741+0000 7fe09bfff640 1 -- 172.21.15.29:0/2997888037 learned_addr learned my addr 172.21.15.29:0/2997888037 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:42.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.741+0000 7fe0a08c2640 1 --2- 172.21.15.29:0/2997888037 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 0x55deb8d47a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:42.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.741+0000 7fe09b7fe640 1 --2- 172.21.15.29:0/2997888037 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d37ea0 0x55deb8d46a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:42.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.741+0000 7fe0a08c2640 1 -- 172.21.15.29:0/2997888037 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d37ea0 msgr2=0x55deb8d46a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:42.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.741+0000 7fe0a08c2640 1 --2- 172.21.15.29:0/2997888037 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d37ea0 0x55deb8d46a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.741+0000 7fe0a08c2640 1 -- 172.21.15.29:0/2997888037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d374b0 msgr2=0x55deb8d464f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:42.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.741+0000 7fe0a08c2640 1 --2- 172.21.15.29:0/2997888037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d374b0 0x55deb8d464f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.741+0000 7fe0a08c2640 1 -- 172.21.15.29:0/2997888037 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55deb8c9e610 con 0x55deb8d3d010 2024-03-11T19:19:42.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.742+0000 7fe0a08c2640 1 --2- 172.21.15.29:0/2997888037 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 0x55deb8d47a40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe09c066e50 tx=0x7fe09c074c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6232c3585ca42c68 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:42.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.742+0000 7fe07b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:19:42.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.742+0000 7fe07b7fe640 1 -- 172.21.15.29:0/2997888037 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c079070 con 0x55deb8d3d010 2024-03-11T19:19:42.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.742+0000 7fe07b7fe640 1 -- 172.21.15.29:0/2997888037 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe09c073a00 con 0x55deb8d3d010 2024-03-11T19:19:42.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.742+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/2997888037 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe0880054d0 con 0x55deb8d3d010 2024-03-11T19:19:42.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.742+0000 7fe07b7fe640 1 -- 172.21.15.29:0/2997888037 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe09c073bc0 con 0x55deb8d3d010 2024-03-11T19:19:42.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.743+0000 7fe0a4fd2740 10 client.27197 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:42.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.743+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/2997888037 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55deb8c606a0 con 0x55deb8d3d010 2024-03-11T19:19:42.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.743+0000 7fe07b7fe640 1 -- 172.21.15.29:0/2997888037 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe0880054d0 con 0x55deb8d3d010 2024-03-11T19:19:42.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.743+0000 7fe07b7fe640 1 -- 172.21.15.29:0/2997888037 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe09c088400 con 0x55deb8d3d010 2024-03-11T19:19:42.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.743+0000 7fe0a4fd2740 10 client.27197 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:42.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.743+0000 7fe0a4fd2740 10 client.27197 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:19:42.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.743+0000 7fe0a4fd2740 20 client.27197 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:42.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.743+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/2997888037 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55deb8da46b0 0x55deb8da6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.743+0000 7fe0a4fd2740 4 client.27197 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:19:42.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.743+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/2997888037 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55deb8a2b890 con 0x55deb8da46b0 2024-03-11T19:19:42.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.743+0000 7fe09bfff640 1 --2- 172.21.15.29:0/2997888037 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55deb8da46b0 0x55deb8da6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:42.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.744+0000 7fe07b7fe640 10 client.27197 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:42.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.745+0000 7fe09bfff640 1 --2- 172.21.15.29:0/2997888037 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55deb8da46b0 0x55deb8da6ad0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57096cd1fb6a4a94 server_cookie=f0158adbec4c7fff in_seq=0 out_seq=0 2024-03-11T19:19:42.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.745+0000 7fe07b7fe640 1 -- 172.21.15.29:0/2997888037 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55deb8a2b890 con 0x55deb8da46b0 2024-03-11T19:19:42.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.745+0000 7fe07b7fe640 10 client.27197 handle_command_reply: tid=0 2024-03-11T19:19:42.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.745+0000 7fe0a4fd2740 1 client.27197 shutdown 2024-03-11T19:19:42.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.746+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/2997888037 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 msgr2=0x55deb8d47a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:42.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.746+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/2997888037 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 0x55deb8d47a40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe09c066e50 tx=0x7fe09c074c20 comp rx=0 tx=0).stop 2024-03-11T19:19:42.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.747+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/2997888037 shutdown_connections 2024-03-11T19:19:42.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.747+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/2997888037 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55deb8d3d010 0x55deb8d47a40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.747+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/2997888037 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55deb8d37ea0 0x55deb8d46a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.747+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/2997888037 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55deb8da46b0 0x55deb8da6ad0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.747+0000 7fe07b7fe640 0 client.27197 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:42.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.747+0000 7fe0a4fd2740 1 --2- 172.21.15.29:0/2997888037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55deb8d374b0 0x55deb8d464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.747+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/2997888037 >> 172.21.15.29:0/2997888037 conn(0x55deb8d32a20 msgr2=0x55deb8d35c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:42.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.748+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/2997888037 shutdown_connections 2024-03-11T19:19:42.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.748+0000 7fe0a4fd2740 1 -- 172.21.15.29:0/2997888037 wait complete. 2024-03-11T19:19:42.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.748+0000 7fe0a4fd2740 20 client.27197 trim_cache size 0 max 16384 2024-03-11T19:19:42.752 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:19:42.753 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:19:42.753 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:19:42.753 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:19:42.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.749+0000 7fe0a10c3640 1 -- 172.21.15.29:0/4054091554 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe08c0a1a40 msgr2=0x7fe08c0a3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:42.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.749+0000 7fe0a10c3640 1 --2- 172.21.15.29:0/4054091554 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe08c0a1a40 0x7fe08c0a3f00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe0880095b0 tx=0x7fe0880046c0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.750+0000 7fe0a10c3640 1 -- 172.21.15.29:0/4054091554 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0940a4c40 msgr2=0x7fe094143310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:42.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.750+0000 7fe0a10c3640 1 --2- 172.21.15.29:0/4054091554 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0940a4c40 0x7fe094143310 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe090005ab0 tx=0x7fe0900040c0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.750+0000 7fe0a10c3640 1 -- 172.21.15.29:0/4054091554 shutdown_connections 2024-03-11T19:19:42.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.750+0000 7fe0a10c3640 1 --2- 172.21.15.29:0/4054091554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0940a6710 0x7fe09414a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.750+0000 7fe0a10c3640 1 --2- 172.21.15.29:0/4054091554 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe08c0a1a40 0x7fe08c0a3f00 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.750+0000 7fe0a10c3640 1 --2- 172.21.15.29:0/4054091554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0940a5d70 0x7fe094143850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.750+0000 7fe0a10c3640 1 --2- 172.21.15.29:0/4054091554 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0940a4c40 0x7fe094143310 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.750+0000 7fe0a10c3640 1 -- 172.21.15.29:0/4054091554 >> 172.21.15.29:0/4054091554 conn(0x7fe0940a00d0 msgr2=0x7fe0940a0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:42.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.750+0000 7fe0a10c3640 1 -- 172.21.15.29:0/4054091554 shutdown_connections 2024-03-11T19:19:42.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.751+0000 7fe0a10c3640 1 -- 172.21.15.29:0/4054091554 wait complete. 2024-03-11T19:19:42.774 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:19:42.774 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:19:42.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.977+0000 7f10cf59e640 1 -- 172.21.15.29:0/275108503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10c8009510 msgr2=0x7f10c8007960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:42.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.977+0000 7f10cf59e640 1 --2- 172.21.15.29:0/275108503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10c8009510 0x7f10c8007960 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f10c0009900 tx=0x7f10c002bc30 comp rx=0 tx=0).stop 2024-03-11T19:19:42.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.978+0000 7f10cf59e640 1 -- 172.21.15.29:0/275108503 shutdown_connections 2024-03-11T19:19:42.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.978+0000 7f10cf59e640 1 --2- 172.21.15.29:0/275108503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10c809dcf0 0x7f10c80a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.978+0000 7f10cf59e640 1 --2- 172.21.15.29:0/275108503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10c8007f30 0x7f10c80083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.978+0000 7f10cf59e640 1 --2- 172.21.15.29:0/275108503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10c8009510 0x7f10c8007960 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.978+0000 7f10cf59e640 1 -- 172.21.15.29:0/275108503 >> 172.21.15.29:0/275108503 conn(0x7f10c80913a0 msgr2=0x7f10c80937e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:42.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.978+0000 7f10cf59e640 1 -- 172.21.15.29:0/275108503 shutdown_connections 2024-03-11T19:19:42.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.978+0000 7f10cf59e640 1 -- 172.21.15.29:0/275108503 wait complete. 2024-03-11T19:19:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.979+0000 7f10cf59e640 1 Processor -- start 2024-03-11T19:19:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.979+0000 7f10cf59e640 1 -- start start 2024-03-11T19:19:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.980+0000 7f10cf59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10c8007f30 0x7f10c81324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.980+0000 7f10cf59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10c8009510 0x7f10c8132a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.980+0000 7f10cf59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10c809dcf0 0x7f10c8139a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.980+0000 7f10cf59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c8139fb0 con 0x7f10c8009510 2024-03-11T19:19:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.980+0000 7f10cf59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c813a120 con 0x7f10c8007f30 2024-03-11T19:19:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.980+0000 7f10cf59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c813a2e0 con 0x7f10c809dcf0 2024-03-11T19:19:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.980+0000 7f10cdd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10c8009510 0x7f10c8132a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.980+0000 7f10ce59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10c8007f30 0x7f10c81324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.980+0000 7f10cdd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10c8009510 0x7f10c8132a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43956/0 (socket says 172.21.15.29:43956) 2024-03-11T19:19:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.980+0000 7f10cdd9b640 1 -- 172.21.15.29:0/3152659083 learned_addr learned my addr 172.21.15.29:0/3152659083 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.980+0000 7f10ce59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10c8007f30 0x7f10c81324c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59388/0 (socket says 172.21.15.29:59388) 2024-03-11T19:19:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.980+0000 7f10ced9d640 1 --2- 172.21.15.29:0/3152659083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10c809dcf0 0x7f10c8139a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.980+0000 7f10ce59c640 1 -- 172.21.15.29:0/3152659083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10c809dcf0 msgr2=0x7f10c8139a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.980+0000 7f10ce59c640 1 --2- 172.21.15.29:0/3152659083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10c809dcf0 0x7f10c8139a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.980+0000 7f10ce59c640 1 -- 172.21.15.29:0/3152659083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10c8009510 msgr2=0x7f10c8132a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.980+0000 7f10ce59c640 1 --2- 172.21.15.29:0/3152659083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10c8009510 0x7f10c8132a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.980+0000 7f10ce59c640 1 -- 172.21.15.29:0/3152659083 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d004f750 con 0x7f10c8007f30 2024-03-11T19:19:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.980+0000 7f10ced9d640 1 --2- 172.21.15.29:0/3152659083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10c809dcf0 0x7f10c8139a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.980+0000 7f10cdd9b640 1 --2- 172.21.15.29:0/3152659083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10c8009510 0x7f10c8132a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.981+0000 7f10ce59c640 1 --2- 172.21.15.29:0/3152659083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10c8007f30 0x7f10c81324c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f10c002c170 tx=0x7f10c0004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.981+0000 7f10bf7fe640 1 -- 172.21.15.29:0/3152659083 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c0004440 con 0x7f10c8007f30 2024-03-11T19:19:42.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.981+0000 7f10bf7fe640 1 -- 172.21.15.29:0/3152659083 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f10c0003aa0 con 0x7f10c8007f30 2024-03-11T19:19:42.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.981+0000 7f10cf59e640 1 -- 172.21.15.29:0/3152659083 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c0009560 con 0x7f10c8007f30 2024-03-11T19:19:42.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.981+0000 7f10bf7fe640 1 -- 172.21.15.29:0/3152659083 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c0005390 con 0x7f10c8007f30 2024-03-11T19:19:42.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.981+0000 7f10cf59e640 1 -- 172.21.15.29:0/3152659083 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c813a900 con 0x7f10c8007f30 2024-03-11T19:19:42.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.982+0000 7f10cf59e640 1 -- 172.21.15.29:0/3152659083 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c80a38f0 con 0x7f10c8007f30 2024-03-11T19:19:42.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.984+0000 7f10bf7fe640 1 -- 172.21.15.29:0/3152659083 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f10c00054f0 con 0x7f10c8007f30 2024-03-11T19:19:42.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.985+0000 7f10bf7fe640 1 --2- 172.21.15.29:0/3152659083 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f10b80a1ab0 0x7f10b80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:42.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.985+0000 7f10cdd9b640 1 --2- 172.21.15.29:0/3152659083 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f10b80a1ab0 0x7f10b80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:42.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.985+0000 7f10bf7fe640 1 -- 172.21.15.29:0/3152659083 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f10c002de40 con 0x7f10c8007f30 2024-03-11T19:19:42.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.986+0000 7f10cdd9b640 1 --2- 172.21.15.29:0/3152659083 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f10b80a1ab0 0x7f10b80a3f70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f10c81339d0 tx=0x7f10c4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:42.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:42.988+0000 7f10bf7fe640 1 -- 172.21.15.29:0/3152659083 <== mon.2 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f10c00b9370 con 0x7f10c8007f30 2024-03-11T19:19:43.463 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:19:43.463 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:19:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:19:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.457+0000 7f10cf59e640 1 -- 172.21.15.29:0/3152659083 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10c80a3ab0 con 0x7f10c8007f30 2024-03-11T19:19:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.458+0000 7f10bf7fe640 1 -- 172.21.15.29:0/3152659083 <== mon.2 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f10c0043020 con 0x7f10c8007f30 2024-03-11T19:19:43.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.462+0000 7f10cf59e640 1 -- 172.21.15.29:0/3152659083 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f10b80a1ab0 msgr2=0x7f10b80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:43.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.462+0000 7f10cf59e640 1 --2- 172.21.15.29:0/3152659083 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f10b80a1ab0 0x7f10b80a3f70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f10c81339d0 tx=0x7f10c4005f90 comp rx=0 tx=0).stop 2024-03-11T19:19:43.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.462+0000 7f10cf59e640 1 -- 172.21.15.29:0/3152659083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10c8007f30 msgr2=0x7f10c81324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:43.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.462+0000 7f10cf59e640 1 --2- 172.21.15.29:0/3152659083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10c8007f30 0x7f10c81324c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f10c002c170 tx=0x7f10c0004040 comp rx=0 tx=0).stop 2024-03-11T19:19:43.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.465+0000 7f10cf59e640 1 -- 172.21.15.29:0/3152659083 shutdown_connections 2024-03-11T19:19:43.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.465+0000 7f10cf59e640 1 --2- 172.21.15.29:0/3152659083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10c809dcf0 0x7f10c8139a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.466+0000 7f10cf59e640 1 --2- 172.21.15.29:0/3152659083 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f10b80a1ab0 0x7f10b80a3f70 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.466+0000 7f10cf59e640 1 --2- 172.21.15.29:0/3152659083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10c8009510 0x7f10c8132a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.466+0000 7f10cf59e640 1 --2- 172.21.15.29:0/3152659083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10c8007f30 0x7f10c81324c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.466+0000 7f10cf59e640 1 -- 172.21.15.29:0/3152659083 >> 172.21.15.29:0/3152659083 conn(0x7f10c80913a0 msgr2=0x7f10c80957e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:43.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.467+0000 7f10cf59e640 1 -- 172.21.15.29:0/3152659083 shutdown_connections 2024-03-11T19:19:43.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.467+0000 7f10cf59e640 1 -- 172.21.15.29:0/3152659083 wait complete. 2024-03-11T19:19:43.499 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:19:43.499 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:19:43.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.719+0000 7f97a759e640 1 -- 172.21.15.29:0/3461918900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a81094e0 msgr2=0x7f97a8109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:43.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.719+0000 7f97a759e640 1 --2- 172.21.15.29:0/3461918900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a81094e0 0x7f97a8109900 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9790009860 tx=0x7f979002bb70 comp rx=0 tx=0).stop 2024-03-11T19:19:43.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.719+0000 7f97a759e640 1 -- 172.21.15.29:0/3461918900 shutdown_connections 2024-03-11T19:19:43.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.719+0000 7f97a759e640 1 --2- 172.21.15.29:0/3461918900 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97a8108b40 0x7f97a81103e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.719+0000 7f97a759e640 1 --2- 172.21.15.29:0/3461918900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a8108180 0x7f97a8108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.719+0000 7f97a759e640 1 --2- 172.21.15.29:0/3461918900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a81094e0 0x7f97a8109900 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.719+0000 7f97a759e640 1 -- 172.21.15.29:0/3461918900 >> 172.21.15.29:0/3461918900 conn(0x7f97a806e060 msgr2=0x7f97a80704a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:43.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.719+0000 7f97a759e640 1 -- 172.21.15.29:0/3461918900 shutdown_connections 2024-03-11T19:19:43.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.720+0000 7f97a759e640 1 -- 172.21.15.29:0/3461918900 wait complete. 2024-03-11T19:19:43.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.720+0000 7f97a759e640 1 Processor -- start 2024-03-11T19:19:43.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.721+0000 7f97a759e640 1 -- start start 2024-03-11T19:19:43.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.721+0000 7f97a759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a8108180 0x7f97a819fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:43.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.721+0000 7f97a759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97a8108b40 0x7f97a81a2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:43.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.721+0000 7f97a759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a81094e0 0x7f97a81a49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:43.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.721+0000 7f97a759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a81a5150 con 0x7f97a81094e0 2024-03-11T19:19:43.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.721+0000 7f97a759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a81a52c0 con 0x7f97a8108b40 2024-03-11T19:19:43.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.721+0000 7f97a759e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a81a5480 con 0x7f97a8108180 2024-03-11T19:19:43.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.721+0000 7f97a659c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a8108180 0x7f97a819fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:43.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.721+0000 7f97a659c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a8108180 0x7f97a819fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:60986/0 (socket says 172.21.15.29:60986) 2024-03-11T19:19:43.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.721+0000 7f97a659c640 1 -- 172.21.15.29:0/1102722520 learned_addr learned my addr 172.21.15.29:0/1102722520 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:43.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.721+0000 7f97a6d9d640 1 --2- 172.21.15.29:0/1102722520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a81094e0 0x7f97a81a49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:43.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.721+0000 7f97a5d9b640 1 --2- 172.21.15.29:0/1102722520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97a8108b40 0x7f97a81a2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:43.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.721+0000 7f97a659c640 1 -- 172.21.15.29:0/1102722520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97a8108b40 msgr2=0x7f97a81a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:43.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.721+0000 7f97a659c640 1 --2- 172.21.15.29:0/1102722520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97a8108b40 0x7f97a81a2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.721+0000 7f97a659c640 1 -- 172.21.15.29:0/1102722520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a81094e0 msgr2=0x7f97a81a49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:43.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.721+0000 7f97a659c640 1 --2- 172.21.15.29:0/1102722520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a81094e0 0x7f97a81a49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.721+0000 7f97a659c640 1 -- 172.21.15.29:0/1102722520 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9790009490 con 0x7f97a8108180 2024-03-11T19:19:43.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.721+0000 7f97a5d9b640 1 --2- 172.21.15.29:0/1102722520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97a8108b40 0x7f97a81a2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:19:43.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.722+0000 7f97a659c640 1 --2- 172.21.15.29:0/1102722520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a8108180 0x7f97a819fe00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f979002c0b0 tx=0x7f9790032430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:43.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.722+0000 7f979f7fe640 1 -- 172.21.15.29:0/1102722520 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979002c510 con 0x7f97a8108180 2024-03-11T19:19:43.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.722+0000 7f979f7fe640 1 -- 172.21.15.29:0/1102722520 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f979002ce90 con 0x7f97a8108180 2024-03-11T19:19:43.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.722+0000 7f97a759e640 1 -- 172.21.15.29:0/1102722520 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a81a5700 con 0x7f97a8108180 2024-03-11T19:19:43.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.722+0000 7f97a759e640 1 -- 172.21.15.29:0/1102722520 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a81a5bb0 con 0x7f97a8108180 2024-03-11T19:19:43.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.722+0000 7f979f7fe640 1 -- 172.21.15.29:0/1102722520 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9790003e50 con 0x7f97a8108180 2024-03-11T19:19:43.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.725+0000 7f979f7fe640 1 -- 172.21.15.29:0/1102722520 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f979003b040 con 0x7f97a8108180 2024-03-11T19:19:43.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.726+0000 7f979f7fe640 1 --2- 172.21.15.29:0/1102722520 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97940a1a30 0x7f97940a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:43.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.726+0000 7f97a5d9b640 1 --2- 172.21.15.29:0/1102722520 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97940a1a30 0x7f97940a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:43.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.726+0000 7f979f7fe640 1 -- 172.21.15.29:0/1102722520 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f97900bd4f0 con 0x7f97a8108180 2024-03-11T19:19:43.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.727+0000 7f97a5d9b640 1 --2- 172.21.15.29:0/1102722520 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97940a1a30 0x7f97940a3ef0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f97a8110680 tx=0x7f9798009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:43.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.727+0000 7f97afa06740 1 Processor -- start 2024-03-11T19:19:43.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.727+0000 7f97afa06740 1 -- start start 2024-03-11T19:19:43.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.728+0000 7f97afa06740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612aa480 0x5555612aa8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:43.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.728+0000 7f97afa06740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aae70 0x5555612af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:43.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.728+0000 7f97afa06740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612affe0 0x5555612b2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:43.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.728+0000 7f97afa06740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555561266960 con 0x5555612aa480 2024-03-11T19:19:43.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.728+0000 7f97afa06740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55556121a310 con 0x5555612affe0 2024-03-11T19:19:43.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.728+0000 7f97afa06740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555561217840 con 0x5555612aae70 2024-03-11T19:19:43.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.728+0000 7f97a6d9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612affe0 0x5555612b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:43.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.728+0000 7f97a6d9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612affe0 0x5555612b2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59412/0 (socket says 172.21.15.29:59412) 2024-03-11T19:19:43.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.728+0000 7f97a6d9d640 1 -- 172.21.15.29:0/1308071648 learned_addr learned my addr 172.21.15.29:0/1308071648 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:43.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.728+0000 7f97a5d9b640 1 --2- 172.21.15.29:0/1308071648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aae70 0x5555612af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:43.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.728+0000 7f97a6d9d640 1 -- 172.21.15.29:0/1308071648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aae70 msgr2=0x5555612af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:43.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.728+0000 7f97a6d9d640 1 --2- 172.21.15.29:0/1308071648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aae70 0x5555612af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.728+0000 7f97a6d9d640 1 -- 172.21.15.29:0/1308071648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612aa480 msgr2=0x5555612aa8a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:19:43.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.729+0000 7f97a659c640 1 --2- 172.21.15.29:0/1308071648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612aa480 0x5555612aa8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:43.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.729+0000 7f97a6d9d640 1 --2- 172.21.15.29:0/1308071648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612aa480 0x5555612aa8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.729+0000 7f97a6d9d640 1 -- 172.21.15.29:0/1308071648 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5555611d5060 con 0x5555612affe0 2024-03-11T19:19:43.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.730+0000 7f97a6d9d640 1 --2- 172.21.15.29:0/1308071648 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612affe0 0x5555612b2470 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009fd0 tx=0x7f97a00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83611e90acb91993 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:43.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.730+0000 7f979cff9640 1 -- 172.21.15.29:0/1308071648 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a0004d20 con 0x5555612affe0 2024-03-11T19:19:43.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.730+0000 7f979cff9640 1 -- 172.21.15.29:0/1308071648 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f97a0004e80 con 0x5555612affe0 2024-03-11T19:19:43.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.731+0000 7f97afa06740 1 -- 172.21.15.29:0/1308071648 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612affe0 msgr2=0x5555612b2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:43.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.731+0000 7f97afa06740 1 --2- 172.21.15.29:0/1308071648 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612affe0 0x5555612b2470 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009fd0 tx=0x7f97a00118d0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.733+0000 7f97afa06740 1 -- 172.21.15.29:0/1308071648 shutdown_connections 2024-03-11T19:19:43.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.733+0000 7f97afa06740 1 --2- 172.21.15.29:0/1308071648 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612affe0 0x5555612b2470 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.733+0000 7f97afa06740 1 --2- 172.21.15.29:0/1308071648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aae70 0x5555612af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.733+0000 7f97afa06740 1 --2- 172.21.15.29:0/1308071648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612aa480 0x5555612aa8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.734+0000 7f97afa06740 1 -- 172.21.15.29:0/1308071648 >> 172.21.15.29:0/1308071648 conn(0x5555612a59f0 msgr2=0x5555612a5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:43.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.738+0000 7f97afa06740 1 -- 172.21.15.29:0/1308071648 shutdown_connections 2024-03-11T19:19:43.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.738+0000 7f97afa06740 1 -- 172.21.15.29:0/1308071648 wait complete. 2024-03-11T19:19:43.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.738+0000 7f97afa06740 1 -- start start 2024-03-11T19:19:43.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.739+0000 7f97afa06740 1 Processor -- start 2024-03-11T19:19:43.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.739+0000 7f97afa06740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aa480 0x5555612b4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:43.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.739+0000 7f97afa06740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612aae70 0x5555612b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:43.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.739+0000 7f97afa06740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612affe0 0x5555612bba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:43.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.740+0000 7f97a5d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612aae70 0x5555612b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:43.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.740+0000 7f97a5d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612aae70 0x5555612b49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59420/0 (socket says 172.21.15.29:59420) 2024-03-11T19:19:43.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.740+0000 7f97a5d9b640 1 -- 172.21.15.29:0/3666072978 learned_addr learned my addr 172.21.15.29:0/3666072978 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:43.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.740+0000 7f97afa06740 1 -- 172.21.15.29:0/3666072978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555561213020 con 0x5555612affe0 2024-03-11T19:19:43.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.740+0000 7f97a659c640 1 --2- 172.21.15.29:0/3666072978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aa480 0x5555612b4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:43.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.740+0000 7f97a6d9d640 1 --2- 172.21.15.29:0/3666072978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612affe0 0x5555612bba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:43.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.740+0000 7f97afa06740 1 -- 172.21.15.29:0/3666072978 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555560ee6430 con 0x5555612aae70 2024-03-11T19:19:43.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.740+0000 7f97afa06740 1 -- 172.21.15.29:0/3666072978 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555560ea2e00 con 0x5555612aa480 2024-03-11T19:19:43.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.740+0000 7f97afa06740 10 client.0 _ll_register_callbacks cb 0x5555611bacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:43.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.741+0000 7f97a6d9d640 1 -- 172.21.15.29:0/3666072978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aa480 msgr2=0x5555612b4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:43.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.741+0000 7f97a6d9d640 1 --2- 172.21.15.29:0/3666072978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aa480 0x5555612b4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.741+0000 7f97a6d9d640 1 -- 172.21.15.29:0/3666072978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612aae70 msgr2=0x5555612b49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:43.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.741+0000 7f97a6d9d640 1 --2- 172.21.15.29:0/3666072978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612aae70 0x5555612b49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.741+0000 7f97a6d9d640 1 -- 172.21.15.29:0/3666072978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a0009c80 con 0x5555612affe0 2024-03-11T19:19:43.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.741+0000 7f97a6d9d640 1 --2- 172.21.15.29:0/3666072978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612affe0 0x5555612bba10 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x5555612a6960 tx=0x7f97a0004550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c263369af5b7fc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:43.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.741+0000 7f978dffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:19:43.813 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:43 smithi029 ceph-mon[32439]: pgmap v648: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 57 MiB/s wr, 1.17k op/s 2024-03-11T19:19:43.813 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:43 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:19:43.813 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:43 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3152659083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:19:43.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.814+0000 7f978dffb640 1 -- 172.21.15.29:0/3666072978 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a0004c80 con 0x5555612affe0 2024-03-11T19:19:43.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.814+0000 7f97afa06740 1 -- 172.21.15.29:0/3666072978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555560f4f720 con 0x5555612affe0 2024-03-11T19:19:43.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.820+0000 7f978dffb640 1 -- 172.21.15.29:0/3666072978 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f97a00054a0 con 0x5555612affe0 2024-03-11T19:19:43.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.820+0000 7f978dffb640 1 -- 172.21.15.29:0/3666072978 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97a00056a0 con 0x5555612affe0 2024-03-11T19:19:43.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.820+0000 7f97afa06740 10 client.17718 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:43.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.820+0000 7f97afa06740 1 -- 172.21.15.29:0/3666072978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5555612116a0 con 0x5555612affe0 2024-03-11T19:19:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.824+0000 7f978dffb640 1 -- 172.21.15.29:0/3666072978 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x555560f4f720 con 0x5555612affe0 2024-03-11T19:19:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.824+0000 7f978dffb640 1 -- 172.21.15.29:0/3666072978 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f97a0021040 con 0x5555612affe0 2024-03-11T19:19:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.824+0000 7f97afa06740 10 client.17718 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.824+0000 7f97afa06740 10 client.17718 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:19:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.824+0000 7f97afa06740 20 client.17718 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.824+0000 7f97afa06740 1 --2- 172.21.15.29:0/3666072978 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5555612c0570 0x5555612c2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.824+0000 7f97afa06740 4 client.17718 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:19:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.824+0000 7f97afa06740 1 -- 172.21.15.29:0/3666072978 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555560f9e890 con 0x5555612c0570 2024-03-11T19:19:43.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.826+0000 7f97a659c640 1 --2- 172.21.15.29:0/3666072978 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5555612c0570 0x5555612c2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:43.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.831+0000 7f97a659c640 1 --2- 172.21.15.29:0/3666072978 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5555612c0570 0x5555612c2990 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c2c1a23561fd5a94 server_cookie=afca4b38d4faaafc in_seq=0 out_seq=0 2024-03-11T19:19:43.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.831+0000 7f978dffb640 10 client.17718 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:19:43.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.832+0000 7f978dffb640 1 -- 172.21.15.29:0/3666072978 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555560f9e890 con 0x5555612c0570 2024-03-11T19:19:43.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.832+0000 7f978dffb640 10 client.17718 handle_command_reply: tid=0 2024-03-11T19:19:43.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.832+0000 7f97afa06740 1 client.17718 shutdown 2024-03-11T19:19:43.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.832+0000 7f97afa06740 1 -- 172.21.15.29:0/3666072978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612affe0 msgr2=0x5555612bba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:43.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.832+0000 7f97afa06740 1 --2- 172.21.15.29:0/3666072978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612affe0 0x5555612bba10 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x5555612a6960 tx=0x7f97a0004550 comp rx=0 tx=0).stop 2024-03-11T19:19:43.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.832+0000 7f97afa06740 1 -- 172.21.15.29:0/3666072978 shutdown_connections 2024-03-11T19:19:43.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.832+0000 7f97afa06740 1 --2- 172.21.15.29:0/3666072978 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5555612c0570 0x5555612c2990 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.832+0000 7f97afa06740 1 --2- 172.21.15.29:0/3666072978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612affe0 0x5555612bba10 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.832+0000 7f97afa06740 1 --2- 172.21.15.29:0/3666072978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612aae70 0x5555612b49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.832+0000 7f97afa06740 1 --2- 172.21.15.29:0/3666072978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aa480 0x5555612b4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.832+0000 7f97afa06740 1 -- 172.21.15.29:0/3666072978 >> 172.21.15.29:0/3666072978 conn(0x5555612a59f0 msgr2=0x5555612a4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:43.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.832+0000 7f978dffb640 0 client.17718 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:19:43.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.833+0000 7f97afa06740 1 -- 172.21.15.29:0/3666072978 shutdown_connections 2024-03-11T19:19:43.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.833+0000 7f97afa06740 1 -- 172.21.15.29:0/3666072978 wait complete. 2024-03-11T19:19:43.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.833+0000 7f97afa06740 20 client.17718 trim_cache size 0 max 16384 2024-03-11T19:19:43.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.861+0000 7f97afa06740 1 Processor -- start 2024-03-11T19:19:43.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.862+0000 7f97afa06740 1 -- start start 2024-03-11T19:19:43.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.862+0000 7f97afa06740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aa480 0x5555612a5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:43.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.862+0000 7f97afa06740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612aae70 0x5555612bb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:43.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.862+0000 7f97afa06740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612affe0 0x5555612a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:43.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.862+0000 7f97afa06740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555561198130 con 0x5555612affe0 2024-03-11T19:19:43.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.862+0000 7f97afa06740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555560f9e890 con 0x5555612aae70 2024-03-11T19:19:43.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.862+0000 7f97afa06740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555560f4f720 con 0x5555612aa480 2024-03-11T19:19:43.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.862+0000 7f97a5d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612aae70 0x5555612bb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:43.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.862+0000 7f97a5d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612aae70 0x5555612bb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59428/0 (socket says 172.21.15.29:59428) 2024-03-11T19:19:43.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.862+0000 7f97a5d9b640 1 -- 172.21.15.29:0/903095085 learned_addr learned my addr 172.21.15.29:0/903095085 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:43.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.862+0000 7f97a659c640 1 --2- 172.21.15.29:0/903095085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aa480 0x5555612a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:43.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.862+0000 7f97a6d9d640 1 --2- 172.21.15.29:0/903095085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612affe0 0x5555612a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:43.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.862+0000 7f97a5d9b640 1 -- 172.21.15.29:0/903095085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aa480 msgr2=0x5555612a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:43.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.862+0000 7f97a5d9b640 1 --2- 172.21.15.29:0/903095085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aa480 0x5555612a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.862+0000 7f97a5d9b640 1 -- 172.21.15.29:0/903095085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612affe0 msgr2=0x5555612a8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:43.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.862+0000 7f97a5d9b640 1 --2- 172.21.15.29:0/903095085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612affe0 0x5555612a8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.862+0000 7f97a5d9b640 1 -- 172.21.15.29:0/903095085 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5555611adc20 con 0x5555612aae70 2024-03-11T19:19:43.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.863+0000 7f97a5d9b640 1 --2- 172.21.15.29:0/903095085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612aae70 0x5555612bb9d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f97980127a0 tx=0x7f9798014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af93b2a0551c852f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:43.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.863+0000 7f978ffff640 1 -- 172.21.15.29:0/903095085 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9798025070 con 0x5555612aae70 2024-03-11T19:19:43.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.863+0000 7f978ffff640 1 -- 172.21.15.29:0/903095085 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9798016070 con 0x5555612aae70 2024-03-11T19:19:43.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.863+0000 7f978ffff640 1 -- 172.21.15.29:0/903095085 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9798029960 con 0x5555612aae70 2024-03-11T19:19:43.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.864+0000 7f97afa06740 1 -- 172.21.15.29:0/903095085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612aae70 msgr2=0x5555612bb9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:43.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.864+0000 7f97afa06740 1 --2- 172.21.15.29:0/903095085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612aae70 0x5555612bb9d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f97980127a0 tx=0x7f9798014000 comp rx=0 tx=0).stop 2024-03-11T19:19:43.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.864+0000 7f97afa06740 1 -- 172.21.15.29:0/903095085 shutdown_connections 2024-03-11T19:19:43.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.864+0000 7f97afa06740 1 --2- 172.21.15.29:0/903095085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612affe0 0x5555612a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.864+0000 7f97afa06740 1 --2- 172.21.15.29:0/903095085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612aae70 0x5555612bb9d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.864+0000 7f97afa06740 1 --2- 172.21.15.29:0/903095085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aa480 0x5555612a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.864+0000 7f97afa06740 1 -- 172.21.15.29:0/903095085 >> 172.21.15.29:0/903095085 conn(0x5555612a59f0 msgr2=0x5555612c7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:43.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.864+0000 7f97afa06740 1 -- 172.21.15.29:0/903095085 shutdown_connections 2024-03-11T19:19:43.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.865+0000 7f97afa06740 1 -- 172.21.15.29:0/903095085 wait complete. 2024-03-11T19:19:43.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.866+0000 7f97afa06740 1 -- start start 2024-03-11T19:19:43.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.866+0000 7f97afa06740 1 Processor -- start 2024-03-11T19:19:43.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.866+0000 7f97afa06740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aa480 0x5555612bb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:43.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.866+0000 7f97afa06740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612aae70 0x5555612b9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:43.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.866+0000 7f97afa06740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612affe0 0x5555612ba600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:43.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.866+0000 7f97afa06740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555612bad00 con 0x5555612aae70 2024-03-11T19:19:43.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.866+0000 7f97afa06740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555612bae70 con 0x5555612affe0 2024-03-11T19:19:43.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.866+0000 7f97afa06740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555561316ad0 con 0x5555612aa480 2024-03-11T19:19:43.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.866+0000 7f97afa06740 10 client.0 _ll_register_callbacks cb 0x7f97a0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:43.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.866+0000 7f97a659c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aa480 0x5555612bb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:43.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.866+0000 7f97a659c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aa480 0x5555612bb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:32798/0 (socket says 172.21.15.29:32798) 2024-03-11T19:19:43.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.866+0000 7f97a659c640 1 -- 172.21.15.29:0/4023800287 learned_addr learned my addr 172.21.15.29:0/4023800287 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:43.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.866+0000 7f97a6d9d640 1 --2- 172.21.15.29:0/4023800287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612affe0 0x5555612ba600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:43.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.866+0000 7f97a5d9b640 1 --2- 172.21.15.29:0/4023800287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612aae70 0x5555612b9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:43.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.867+0000 7f97a6d9d640 1 -- 172.21.15.29:0/4023800287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aa480 msgr2=0x5555612bb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:43.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.867+0000 7f97a6d9d640 1 --2- 172.21.15.29:0/4023800287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aa480 0x5555612bb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.867+0000 7f97a6d9d640 1 -- 172.21.15.29:0/4023800287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612aae70 msgr2=0x5555612b9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:43.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.867+0000 7f97a6d9d640 1 --2- 172.21.15.29:0/4023800287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612aae70 0x5555612b9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.867+0000 7f97a6d9d640 1 -- 172.21.15.29:0/4023800287 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9798012450 con 0x5555612affe0 2024-03-11T19:19:43.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.867+0000 7f97a6d9d640 1 --2- 172.21.15.29:0/4023800287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612affe0 0x5555612ba600 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f97a0030040 tx=0x7f97a0005f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=378d065b3905653b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:43.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.867+0000 7f978dffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:19:43.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.867+0000 7f978dffb640 1 -- 172.21.15.29:0/4023800287 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a0025a80 con 0x5555612affe0 2024-03-11T19:19:43.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.868+0000 7f978dffb640 1 -- 172.21.15.29:0/4023800287 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f97a0025be0 con 0x5555612affe0 2024-03-11T19:19:43.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.868+0000 7f97afa06740 1 -- 172.21.15.29:0/4023800287 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555560f127e0 con 0x5555612affe0 2024-03-11T19:19:43.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.868+0000 7f978dffb640 1 -- 172.21.15.29:0/4023800287 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97a0009ea0 con 0x5555612affe0 2024-03-11T19:19:43.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.868+0000 7f97afa06740 10 client.27221 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:43.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.868+0000 7f97afa06740 1 -- 172.21.15.29:0/4023800287 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5555611d38c0 con 0x5555612affe0 2024-03-11T19:19:43.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.868+0000 7f978dffb640 1 -- 172.21.15.29:0/4023800287 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x555560f127e0 con 0x5555612affe0 2024-03-11T19:19:43.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.868+0000 7f978dffb640 1 -- 172.21.15.29:0/4023800287 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f97a0021040 con 0x5555612affe0 2024-03-11T19:19:43.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.869+0000 7f97afa06740 10 client.27221 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:43.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.869+0000 7f97afa06740 10 client.27221 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:19:43.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.869+0000 7f97afa06740 20 client.27221 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:43.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.869+0000 7f97afa06740 1 --2- 172.21.15.29:0/4023800287 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x555561318150 0x55556131a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:43.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.869+0000 7f97afa06740 4 client.27221 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:19:43.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.869+0000 7f97afa06740 1 -- 172.21.15.29:0/4023800287 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555560ebe700 con 0x555561318150 2024-03-11T19:19:43.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.869+0000 7f97a659c640 1 --2- 172.21.15.29:0/4023800287 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x555561318150 0x55556131a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:43.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.870+0000 7f978dffb640 10 client.27221 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:19:43.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.870+0000 7f97a659c640 1 --2- 172.21.15.29:0/4023800287 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x555561318150 0x55556131a570 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c2d037aadcafaf1a server_cookie=9a75bbb8651a16d0 in_seq=0 out_seq=0 2024-03-11T19:19:43.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.871+0000 7f978dffb640 1 -- 172.21.15.29:0/4023800287 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555560ebe700 con 0x555561318150 2024-03-11T19:19:43.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.871+0000 7f978dffb640 10 client.27221 handle_command_reply: tid=0 2024-03-11T19:19:43.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.871+0000 7f97afa06740 1 client.27221 shutdown 2024-03-11T19:19:43.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.871+0000 7f97afa06740 1 -- 172.21.15.29:0/4023800287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612affe0 msgr2=0x5555612ba600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:43.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.871+0000 7f97afa06740 1 --2- 172.21.15.29:0/4023800287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612affe0 0x5555612ba600 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f97a0030040 tx=0x7f97a0005f80 comp rx=0 tx=0).stop 2024-03-11T19:19:43.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.871+0000 7f97afa06740 1 -- 172.21.15.29:0/4023800287 shutdown_connections 2024-03-11T19:19:43.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.871+0000 7f97afa06740 1 --2- 172.21.15.29:0/4023800287 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x555561318150 0x55556131a570 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.871+0000 7f97afa06740 1 --2- 172.21.15.29:0/4023800287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5555612affe0 0x5555612ba600 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.871+0000 7f97afa06740 1 --2- 172.21.15.29:0/4023800287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555612aae70 0x5555612b9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.871+0000 7f97afa06740 1 --2- 172.21.15.29:0/4023800287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5555612aa480 0x5555612bb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.871+0000 7f97afa06740 1 -- 172.21.15.29:0/4023800287 >> 172.21.15.29:0/4023800287 conn(0x5555612a59f0 msgr2=0x5555612ac010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:43.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.871+0000 7f978dffb640 0 client.27221 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:19:43.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.871+0000 7f97afa06740 1 -- 172.21.15.29:0/4023800287 shutdown_connections 2024-03-11T19:19:43.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.871+0000 7f97afa06740 1 -- 172.21.15.29:0/4023800287 wait complete. 2024-03-11T19:19:43.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.872+0000 7f97afa06740 20 client.27221 trim_cache size 0 max 16384 2024-03-11T19:19:43.877 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:19:43.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.873+0000 7f97a759e640 1 -- 172.21.15.29:0/1102722520 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97940a1a30 msgr2=0x7f97940a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:43.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.873+0000 7f97a759e640 1 --2- 172.21.15.29:0/1102722520 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97940a1a30 0x7f97940a3ef0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f97a8110680 tx=0x7f9798009260 comp rx=0 tx=0).stop 2024-03-11T19:19:43.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.873+0000 7f97a759e640 1 -- 172.21.15.29:0/1102722520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a8108180 msgr2=0x7f97a819fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:43.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.873+0000 7f97a759e640 1 --2- 172.21.15.29:0/1102722520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a8108180 0x7f97a819fe00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f979002c0b0 tx=0x7f9790032430 comp rx=0 tx=0).stop 2024-03-11T19:19:43.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.875+0000 7f97a759e640 1 -- 172.21.15.29:0/1102722520 shutdown_connections 2024-03-11T19:19:43.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.875+0000 7f97a759e640 1 --2- 172.21.15.29:0/1102722520 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97940a1a30 0x7f97940a3ef0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.875+0000 7f97a759e640 1 --2- 172.21.15.29:0/1102722520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a81094e0 0x7f97a81a49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.875+0000 7f97a759e640 1 --2- 172.21.15.29:0/1102722520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97a8108b40 0x7f97a81a2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.875+0000 7f97a759e640 1 --2- 172.21.15.29:0/1102722520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97a8108180 0x7f97a819fe00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:43.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.875+0000 7f97a759e640 1 -- 172.21.15.29:0/1102722520 >> 172.21.15.29:0/1102722520 conn(0x7f97a806e060 msgr2=0x7f97a8070ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:43.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.875+0000 7f97a759e640 1 -- 172.21.15.29:0/1102722520 shutdown_connections 2024-03-11T19:19:43.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:43.876+0000 7f97a759e640 1 -- 172.21.15.29:0/1102722520 wait complete. 2024-03-11T19:19:43.912 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:19:43.912 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:19:44.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:43 smithi190 ceph-mon[36292]: pgmap v648: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 57 MiB/s wr, 1.17k op/s 2024-03-11T19:19:44.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:43 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:19:44.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:43 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3152659083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:19:44.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:43 smithi170 ceph-mon[35690]: pgmap v648: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 57 MiB/s wr, 1.17k op/s 2024-03-11T19:19:44.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:43 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:19:44.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:43 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3152659083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:19:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.107+0000 7f00e4088640 1 -- 172.21.15.29:0/846067168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00dc0ff3d0 msgr2=0x7f00dc0ff7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.107+0000 7f00e4088640 1 --2- 172.21.15.29:0/846067168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00dc0ff3d0 0x7f00dc0ff7f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f00d8009900 tx=0x7f00d802bc30 comp rx=0 tx=0).stop 2024-03-11T19:19:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.107+0000 7f00e4088640 1 -- 172.21.15.29:0/846067168 shutdown_connections 2024-03-11T19:19:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.107+0000 7f00e4088640 1 --2- 172.21.15.29:0/846067168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00dc100f80 0x7f00dc107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.107+0000 7f00e4088640 1 --2- 172.21.15.29:0/846067168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00dc1005c0 0x7f00dc100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.107+0000 7f00e4088640 1 --2- 172.21.15.29:0/846067168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00dc0ff3d0 0x7f00dc0ff7f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.107+0000 7f00e4088640 1 -- 172.21.15.29:0/846067168 >> 172.21.15.29:0/846067168 conn(0x7f00dc0fa940 msgr2=0x7f00dc0fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.107+0000 7f00e4088640 1 -- 172.21.15.29:0/846067168 shutdown_connections 2024-03-11T19:19:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.107+0000 7f00e4088640 1 -- 172.21.15.29:0/846067168 wait complete. 2024-03-11T19:19:44.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.108+0000 7f00e4088640 1 Processor -- start 2024-03-11T19:19:44.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.108+0000 7f00e4088640 1 -- start start 2024-03-11T19:19:44.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.108+0000 7f00e4088640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00dc0ff3d0 0x7f00dc1993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.108+0000 7f00e4088640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00dc1005c0 0x7f00dc199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.108+0000 7f00e4088640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00dc100f80 0x7f00dc1a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.108+0000 7f00e4088640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc1a0ee0 con 0x7f00dc1005c0 2024-03-11T19:19:44.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.108+0000 7f00e1e24640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00dc0ff3d0 0x7f00dc1993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.108+0000 7f00e4088640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc1a1050 con 0x7f00dc100f80 2024-03-11T19:19:44.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.108+0000 7f00e4088640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc1a1210 con 0x7f00dc0ff3d0 2024-03-11T19:19:44.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.109+0000 7f00e1e24640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00dc0ff3d0 0x7f00dc1993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:32816/0 (socket says 172.21.15.29:32816) 2024-03-11T19:19:44.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.109+0000 7f00e1e24640 1 -- 172.21.15.29:0/638345311 learned_addr learned my addr 172.21.15.29:0/638345311 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:44.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.109+0000 7f00e1623640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00dc1005c0 0x7f00dc199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.109+0000 7f00e2625640 1 --2- 172.21.15.29:0/638345311 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00dc100f80 0x7f00dc1a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.109+0000 7f00e1e24640 1 -- 172.21.15.29:0/638345311 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00dc100f80 msgr2=0x7f00dc1a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.109+0000 7f00e1e24640 1 --2- 172.21.15.29:0/638345311 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00dc100f80 0x7f00dc1a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.109+0000 7f00e1e24640 1 -- 172.21.15.29:0/638345311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00dc1005c0 msgr2=0x7f00dc199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.109+0000 7f00e1e24640 1 --2- 172.21.15.29:0/638345311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00dc1005c0 0x7f00dc199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.109+0000 7f00e1e24640 1 -- 172.21.15.29:0/638345311 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00cc009490 con 0x7f00dc0ff3d0 2024-03-11T19:19:44.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.109+0000 7f00e1e24640 1 --2- 172.21.15.29:0/638345311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00dc0ff3d0 0x7f00dc1993f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f00d8000c00 tx=0x7f00d8004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:44.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.109+0000 7f00d2ffd640 1 -- 172.21.15.29:0/638345311 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d8004500 con 0x7f00dc0ff3d0 2024-03-11T19:19:44.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.110+0000 7f00d2ffd640 1 -- 172.21.15.29:0/638345311 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f00d8003aa0 con 0x7f00dc0ff3d0 2024-03-11T19:19:44.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.110+0000 7f00d2ffd640 1 -- 172.21.15.29:0/638345311 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d8005430 con 0x7f00dc0ff3d0 2024-03-11T19:19:44.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.110+0000 7f00e4088640 1 -- 172.21.15.29:0/638345311 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d8009560 con 0x7f00dc0ff3d0 2024-03-11T19:19:44.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.110+0000 7f00e4088640 1 -- 172.21.15.29:0/638345311 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00dc1a17c0 con 0x7f00dc0ff3d0 2024-03-11T19:19:44.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.113+0000 7f00d2ffd640 1 -- 172.21.15.29:0/638345311 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f00d8003c10 con 0x7f00dc0ff3d0 2024-03-11T19:19:44.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.113+0000 7f00e6c65740 1 Processor -- start 2024-03-11T19:19:44.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.113+0000 7f00e6c65740 1 -- start start 2024-03-11T19:19:44.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.113+0000 7f00e6c65740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1ca80 0x5573b2f1cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.113+0000 7f00e6c65740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f1d470 0x5573b2f21f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.113+0000 7f00e6c65740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f225e0 0x5573b2f24a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.114+0000 7f00e6c65740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573b2ed0960 con 0x5573b2f225e0 2024-03-11T19:19:44.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.114+0000 7f00e6c65740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573b2e84310 con 0x5573b2f1ca80 2024-03-11T19:19:44.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.114+0000 7f00e6c65740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573b2e81840 con 0x5573b2f1d470 2024-03-11T19:19:44.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.114+0000 7f00e2625640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1ca80 0x5573b2f1cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.114+0000 7f00e2625640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1ca80 0x5573b2f1cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59466/0 (socket says 172.21.15.29:59466) 2024-03-11T19:19:44.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.114+0000 7f00e2625640 1 -- 172.21.15.29:0/3213565935 learned_addr learned my addr 172.21.15.29:0/3213565935 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:44.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.114+0000 7f00e1e24640 1 --2- 172.21.15.29:0/3213565935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f1d470 0x5573b2f21f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.114+0000 7f00e1623640 1 --2- 172.21.15.29:0/3213565935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f225e0 0x5573b2f24a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.114+0000 7f00e2625640 1 -- 172.21.15.29:0/3213565935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f1d470 msgr2=0x5573b2f21f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.114+0000 7f00e2625640 1 --2- 172.21.15.29:0/3213565935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f1d470 0x5573b2f21f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.114+0000 7f00e2625640 1 -- 172.21.15.29:0/3213565935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f225e0 msgr2=0x5573b2f24a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.114+0000 7f00e2625640 1 --2- 172.21.15.29:0/3213565935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f225e0 0x5573b2f24a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.114+0000 7f00e2625640 1 -- 172.21.15.29:0/3213565935 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573b2e3f060 con 0x5573b2f1ca80 2024-03-11T19:19:44.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.114+0000 7f00d2ffd640 1 --2- 172.21.15.29:0/638345311 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f00c80a1ab0 0x7f00c80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.114+0000 7f00e2625640 1 --2- 172.21.15.29:0/3213565935 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1ca80 0x5573b2f1cea0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f00d4009fc0 tx=0x7f00d4010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1de969c7169a0a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:44.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.115+0000 7f00e1e24640 1 --2- 172.21.15.29:0/638345311 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f00c80a1ab0 0x7f00c80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.115+0000 7f00c7fff640 1 -- 172.21.15.29:0/3213565935 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d4003b70 con 0x5573b2f1ca80 2024-03-11T19:19:44.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.115+0000 7f00c7fff640 1 -- 172.21.15.29:0/3213565935 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f00d4015070 con 0x5573b2f1ca80 2024-03-11T19:19:44.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.115+0000 7f00c7fff640 1 -- 172.21.15.29:0/3213565935 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d4011780 con 0x5573b2f1ca80 2024-03-11T19:19:44.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.115+0000 7f00d2ffd640 1 -- 172.21.15.29:0/638345311 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f00d802de40 con 0x7f00dc0ff3d0 2024-03-11T19:19:44.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.116+0000 7f00e6c65740 1 -- 172.21.15.29:0/3213565935 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1ca80 msgr2=0x5573b2f1cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.116+0000 7f00e6c65740 1 --2- 172.21.15.29:0/3213565935 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1ca80 0x5573b2f1cea0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f00d4009fc0 tx=0x7f00d4010860 comp rx=0 tx=0).stop 2024-03-11T19:19:44.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.116+0000 7f00e6c65740 1 -- 172.21.15.29:0/3213565935 shutdown_connections 2024-03-11T19:19:44.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.116+0000 7f00e6c65740 1 --2- 172.21.15.29:0/3213565935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f225e0 0x5573b2f24a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.116+0000 7f00e6c65740 1 --2- 172.21.15.29:0/3213565935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f1d470 0x5573b2f21f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.116+0000 7f00e6c65740 1 --2- 172.21.15.29:0/3213565935 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1ca80 0x5573b2f1cea0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.116+0000 7f00e6c65740 1 -- 172.21.15.29:0/3213565935 >> 172.21.15.29:0/3213565935 conn(0x5573b2f0f7e0 msgr2=0x5573b2f0fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:44.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.116+0000 7f00e1e24640 1 --2- 172.21.15.29:0/638345311 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f00c80a1ab0 0x7f00c80a3f70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x5573b2f1dee0 tx=0x7f00d80f5260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:44.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.116+0000 7f00e6c65740 1 -- 172.21.15.29:0/3213565935 shutdown_connections 2024-03-11T19:19:44.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.117+0000 7f00e6c65740 1 -- 172.21.15.29:0/3213565935 wait complete. 2024-03-11T19:19:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.117+0000 7f00e6c65740 1 -- start start 2024-03-11T19:19:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.117+0000 7f00e6c65740 1 Processor -- start 2024-03-11T19:19:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.118+0000 7f00e6c65740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f1ca80 0x5573b2f26c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.118+0000 7f00e6c65740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1d470 0x5573b2f271a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.118+0000 7f00e6c65740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f225e0 0x5573b2f2e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.118+0000 7f00e2625640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f1ca80 0x5573b2f26c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.118+0000 7f00e2625640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f1ca80 0x5573b2f26c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44070/0 (socket says 172.21.15.29:44070) 2024-03-11T19:19:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.118+0000 7f00e2625640 1 -- 172.21.15.29:0/904994600 learned_addr learned my addr 172.21.15.29:0/904994600 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.118+0000 7f00e6c65740 1 -- 172.21.15.29:0/904994600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573b2e7d020 con 0x5573b2f1ca80 2024-03-11T19:19:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.118+0000 7f00e6c65740 1 -- 172.21.15.29:0/904994600 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573b2b50430 con 0x5573b2f1d470 2024-03-11T19:19:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.118+0000 7f00e6c65740 1 -- 172.21.15.29:0/904994600 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573b2b0ce00 con 0x5573b2f225e0 2024-03-11T19:19:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.118+0000 7f00e6c65740 10 client.0 _ll_register_callbacks cb 0x5573b2e24cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.118+0000 7f00e1623640 1 --2- 172.21.15.29:0/904994600 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1d470 0x5573b2f271a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.118+0000 7f00e2625640 1 --2- 172.21.15.29:0/904994600 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f225e0 0x5573b2f2e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.119+0000 7f00e1623640 1 -- 172.21.15.29:0/904994600 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f225e0 msgr2=0x5573b2f2e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.119+0000 7f00e1623640 1 --2- 172.21.15.29:0/904994600 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f225e0 0x5573b2f2e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.119+0000 7f00e1623640 1 -- 172.21.15.29:0/904994600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f1ca80 msgr2=0x5573b2f26c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.119+0000 7f00e1623640 1 --2- 172.21.15.29:0/904994600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f1ca80 0x5573b2f26c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.119+0000 7f00e1623640 1 -- 172.21.15.29:0/904994600 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d4009c70 con 0x5573b2f1d470 2024-03-11T19:19:44.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.119+0000 7f00e1623640 1 --2- 172.21.15.29:0/904994600 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1d470 0x5573b2f271a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f00dc19a900 tx=0x7f00cc012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9af4799b64e823f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:44.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.119+0000 7f00c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:19:44.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.119+0000 7f00c57fa640 1 -- 172.21.15.29:0/904994600 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00cc017070 con 0x5573b2f1d470 2024-03-11T19:19:44.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.119+0000 7f00c57fa640 1 -- 172.21.15.29:0/904994600 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f00cc004030 con 0x5573b2f1d470 2024-03-11T19:19:44.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.120+0000 7f00e6c65740 1 -- 172.21.15.29:0/904994600 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5573b2b7c7e0 con 0x5573b2f1d470 2024-03-11T19:19:44.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.120+0000 7f00c57fa640 1 -- 172.21.15.29:0/904994600 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00cc004230 con 0x5573b2f1d470 2024-03-11T19:19:44.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.123+0000 7f00e6c65740 10 client.27233 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:44.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.123+0000 7f00e6c65740 1 -- 172.21.15.29:0/904994600 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5573b2e7b6a0 con 0x5573b2f1d470 2024-03-11T19:19:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.126+0000 7f00c57fa640 1 -- 172.21.15.29:0/904994600 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f00cc00fdd0 con 0x5573b2f1d470 2024-03-11T19:19:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.126+0000 7f00c57fa640 1 -- 172.21.15.29:0/904994600 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f00cc003a80 con 0x5573b2f1d470 2024-03-11T19:19:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.126+0000 7f00e6c65740 10 client.27233 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.126+0000 7f00e6c65740 10 client.27233 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:19:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.126+0000 7f00e6c65740 20 client.27233 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.126+0000 7f00e6c65740 1 --2- 172.21.15.29:0/904994600 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5573b2f32eb0 0x5573b2f352f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.127+0000 7f00e6c65740 4 client.27233 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:19:44.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.127+0000 7f00e6c65740 1 -- 172.21.15.29:0/904994600 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5573b2b7c7e0 con 0x5573b2f32eb0 2024-03-11T19:19:44.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.127+0000 7f00e2625640 1 --2- 172.21.15.29:0/904994600 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5573b2f32eb0 0x5573b2f352f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.128+0000 7f00e2625640 1 --2- 172.21.15.29:0/904994600 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5573b2f32eb0 0x5573b2f352f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fc68554f59d42c86 server_cookie=a197be2c5c302fa9 in_seq=0 out_seq=0 2024-03-11T19:19:44.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.129+0000 7f00c57fa640 10 client.27233 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:19:44.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.129+0000 7f00c57fa640 1 -- 172.21.15.29:0/904994600 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5573b2b7c7e0 con 0x5573b2f32eb0 2024-03-11T19:19:44.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.129+0000 7f00c57fa640 10 client.27233 handle_command_reply: tid=0 2024-03-11T19:19:44.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.129+0000 7f00e6c65740 1 client.27233 shutdown 2024-03-11T19:19:44.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.131+0000 7f00e6c65740 1 -- 172.21.15.29:0/904994600 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1d470 msgr2=0x5573b2f271a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.131+0000 7f00e6c65740 1 --2- 172.21.15.29:0/904994600 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1d470 0x5573b2f271a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f00dc19a900 tx=0x7f00cc012b60 comp rx=0 tx=0).stop 2024-03-11T19:19:44.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.131+0000 7f00e6c65740 1 -- 172.21.15.29:0/904994600 shutdown_connections 2024-03-11T19:19:44.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.131+0000 7f00e6c65740 1 --2- 172.21.15.29:0/904994600 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f225e0 0x5573b2f2e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.131+0000 7f00e6c65740 1 --2- 172.21.15.29:0/904994600 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1d470 0x5573b2f271a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.131+0000 7f00e6c65740 1 --2- 172.21.15.29:0/904994600 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5573b2f32eb0 0x5573b2f352f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.131+0000 7f00e6c65740 1 --2- 172.21.15.29:0/904994600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f1ca80 0x5573b2f26c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.131+0000 7f00e6c65740 1 -- 172.21.15.29:0/904994600 >> 172.21.15.29:0/904994600 conn(0x5573b2f0f7e0 msgr2=0x5573b2f0eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:44.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.131+0000 7f00c57fa640 0 client.27233 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:19:44.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.132+0000 7f00e6c65740 1 -- 172.21.15.29:0/904994600 shutdown_connections 2024-03-11T19:19:44.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.132+0000 7f00e6c65740 1 -- 172.21.15.29:0/904994600 wait complete. 2024-03-11T19:19:44.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.133+0000 7f00e6c65740 20 client.27233 trim_cache size 0 max 16384 2024-03-11T19:19:44.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.158+0000 7f00e6c65740 1 Processor -- start 2024-03-11T19:19:44.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.163+0000 7f00e6c65740 1 -- start start 2024-03-11T19:19:44.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.164+0000 7f00e6c65740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f1ca80 0x5573b2f39c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.164+0000 7f00e6c65740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1d470 0x5573b2f2e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.164+0000 7f00e6c65740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f225e0 0x5573b2f1e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.165+0000 7f00e6c65740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573b2e02130 con 0x5573b2f225e0 2024-03-11T19:19:44.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.165+0000 7f00e6c65740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573b2f0b320 con 0x5573b2f1d470 2024-03-11T19:19:44.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.165+0000 7f00e6c65740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573b2b7c7e0 con 0x5573b2f1ca80 2024-03-11T19:19:44.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.165+0000 7f00e2625640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f1ca80 0x5573b2f39c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.165+0000 7f00e2625640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f1ca80 0x5573b2f39c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:32844/0 (socket says 172.21.15.29:32844) 2024-03-11T19:19:44.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.165+0000 7f00e2625640 1 -- 172.21.15.29:0/1732062759 learned_addr learned my addr 172.21.15.29:0/1732062759 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:44.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.165+0000 7f00e2625640 1 --2- 172.21.15.29:0/1732062759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f225e0 0x5573b2f1e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.166+0000 7f00e2625640 1 -- 172.21.15.29:0/1732062759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1d470 msgr2=0x5573b2f2e1a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:19:44.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.166+0000 7f00e2625640 1 --2- 172.21.15.29:0/1732062759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1d470 0x5573b2f2e1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.166+0000 7f00e2625640 1 -- 172.21.15.29:0/1732062759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f225e0 msgr2=0x5573b2f1e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.166+0000 7f00e2625640 1 --2- 172.21.15.29:0/1732062759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f225e0 0x5573b2f1e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.166+0000 7f00e2625640 1 -- 172.21.15.29:0/1732062759 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573b2e17c20 con 0x5573b2f1ca80 2024-03-11T19:19:44.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.170+0000 7f00e2625640 1 --2- 172.21.15.29:0/1732062759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f1ca80 0x5573b2f39c90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f00d4000f80 tx=0x7f00d40099c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f01d94d28bb884c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:44.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.170+0000 7f00c77fe640 1 -- 172.21.15.29:0/1732062759 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d4033070 con 0x5573b2f1ca80 2024-03-11T19:19:44.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.170+0000 7f00c77fe640 1 -- 172.21.15.29:0/1732062759 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f00d402c8a0 con 0x5573b2f1ca80 2024-03-11T19:19:44.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.171+0000 7f00c77fe640 1 -- 172.21.15.29:0/1732062759 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d401f040 con 0x5573b2f1ca80 2024-03-11T19:19:44.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.173+0000 7f00e6c65740 1 -- 172.21.15.29:0/1732062759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f1ca80 msgr2=0x5573b2f39c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.173+0000 7f00e6c65740 1 --2- 172.21.15.29:0/1732062759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f1ca80 0x5573b2f39c90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f00d4000f80 tx=0x7f00d40099c0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.173+0000 7f00e6c65740 1 -- 172.21.15.29:0/1732062759 shutdown_connections 2024-03-11T19:19:44.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.173+0000 7f00e6c65740 1 --2- 172.21.15.29:0/1732062759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f225e0 0x5573b2f1e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.173+0000 7f00e6c65740 1 --2- 172.21.15.29:0/1732062759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1d470 0x5573b2f2e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.173+0000 7f00e6c65740 1 --2- 172.21.15.29:0/1732062759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f1ca80 0x5573b2f39c90 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.173+0000 7f00e6c65740 1 -- 172.21.15.29:0/1732062759 >> 172.21.15.29:0/1732062759 conn(0x5573b2f0f7e0 msgr2=0x5573b2f0fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.173+0000 7f00e6c65740 1 -- 172.21.15.29:0/1732062759 shutdown_connections 2024-03-11T19:19:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.173+0000 7f00e6c65740 1 -- 172.21.15.29:0/1732062759 wait complete. 2024-03-11T19:19:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.174+0000 7f00e6c65740 1 -- start start 2024-03-11T19:19:44.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.174+0000 7f00e6c65740 1 Processor -- start 2024-03-11T19:19:44.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.174+0000 7f00e6c65740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f1ca80 0x5573b2f2dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.175+0000 7f00e6c65740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1d470 0x5573b2f2e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.175+0000 7f00e6c65740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f225e0 0x5573b2f2c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.175+0000 7f00e6c65740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573b2f2ca30 con 0x5573b2f225e0 2024-03-11T19:19:44.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.175+0000 7f00e6c65740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573b2f2cba0 con 0x5573b2f1d470 2024-03-11T19:19:44.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.175+0000 7f00e6c65740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573b2f2cd60 con 0x5573b2f1ca80 2024-03-11T19:19:44.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.175+0000 7f00e6c65740 10 client.0 _ll_register_callbacks cb 0x7f00d4009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:44.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.175+0000 7f00e1623640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1d470 0x5573b2f2e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.175+0000 7f00e1623640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1d470 0x5573b2f2e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59480/0 (socket says 172.21.15.29:59480) 2024-03-11T19:19:44.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.175+0000 7f00e2625640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f1ca80 0x5573b2f2dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.175+0000 7f00e1623640 1 -- 172.21.15.29:0/3885650955 learned_addr learned my addr 172.21.15.29:0/3885650955 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:44.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.175+0000 7f00e2625640 1 --2- 172.21.15.29:0/3885650955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f225e0 0x5573b2f2c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.175+0000 7f00e1623640 1 -- 172.21.15.29:0/3885650955 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f1ca80 msgr2=0x5573b2f2dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.175+0000 7f00e1623640 1 --2- 172.21.15.29:0/3885650955 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f1ca80 0x5573b2f2dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.175+0000 7f00e1623640 1 -- 172.21.15.29:0/3885650955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f225e0 msgr2=0x5573b2f2c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.175+0000 7f00e1623640 1 --2- 172.21.15.29:0/3885650955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f225e0 0x5573b2f2c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.175+0000 7f00e1623640 1 -- 172.21.15.29:0/3885650955 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d4022040 con 0x5573b2f1d470 2024-03-11T19:19:44.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.176+0000 7f00e1623640 1 --2- 172.21.15.29:0/3885650955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1d470 0x5573b2f2e200 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f00dc19a900 tx=0x7f00cc00fbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e3831227d633c1f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:44.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.176+0000 7f00c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:19:44.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.176+0000 7f00c57fa640 1 -- 172.21.15.29:0/3885650955 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00cc017040 con 0x5573b2f1d470 2024-03-11T19:19:44.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.176+0000 7f00e6c65740 1 -- 172.21.15.29:0/3885650955 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f00d4015070 con 0x5573b2f1d470 2024-03-11T19:19:44.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.176+0000 7f00c57fa640 1 -- 172.21.15.29:0/3885650955 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f00cc010b00 con 0x5573b2f1d470 2024-03-11T19:19:44.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.176+0000 7f00c57fa640 1 -- 172.21.15.29:0/3885650955 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00cc010d00 con 0x5573b2f1d470 2024-03-11T19:19:44.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.176+0000 7f00e6c65740 10 client.27236 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:44.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.176+0000 7f00e6c65740 1 -- 172.21.15.29:0/3885650955 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5573b2e3d8c0 con 0x5573b2f1d470 2024-03-11T19:19:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.176+0000 7f00c57fa640 1 -- 172.21.15.29:0/3885650955 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f00cc036080 con 0x5573b2f1d470 2024-03-11T19:19:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.177+0000 7f00c57fa640 1 -- 172.21.15.29:0/3885650955 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f00cc003960 con 0x5573b2f1d470 2024-03-11T19:19:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.177+0000 7f00e6c65740 10 client.27236 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.177+0000 7f00e6c65740 10 client.27236 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:19:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.177+0000 7f00e6c65740 20 client.27236 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.177+0000 7f00e6c65740 1 --2- 172.21.15.29:0/3885650955 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5573b2f8aa20 0x5573b2f8ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.177+0000 7f00e6c65740 4 client.27236 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:19:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.177+0000 7f00e6c65740 1 -- 172.21.15.29:0/3885650955 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5573b2bb9720 con 0x5573b2f8aa20 2024-03-11T19:19:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.177+0000 7f00e2625640 1 --2- 172.21.15.29:0/3885650955 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5573b2f8aa20 0x5573b2f8ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.178+0000 7f00e2625640 1 --2- 172.21.15.29:0/3885650955 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5573b2f8aa20 0x5573b2f8ce40 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d97f8d152414384e server_cookie=5bd226799470f20d in_seq=0 out_seq=0 2024-03-11T19:19:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.178+0000 7f00c57fa640 10 client.27236 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:19:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.178+0000 7f00c57fa640 1 -- 172.21.15.29:0/3885650955 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5573b2bb9720 con 0x5573b2f8aa20 2024-03-11T19:19:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.178+0000 7f00c57fa640 10 client.27236 handle_command_reply: tid=0 2024-03-11T19:19:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.178+0000 7f00e6c65740 1 client.27236 shutdown 2024-03-11T19:19:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.178+0000 7f00e6c65740 1 -- 172.21.15.29:0/3885650955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1d470 msgr2=0x5573b2f2e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.178+0000 7f00e6c65740 1 --2- 172.21.15.29:0/3885650955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1d470 0x5573b2f2e200 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f00dc19a900 tx=0x7f00cc00fbb0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.178+0000 7f00e6c65740 1 -- 172.21.15.29:0/3885650955 shutdown_connections 2024-03-11T19:19:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.178+0000 7f00e6c65740 1 --2- 172.21.15.29:0/3885650955 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5573b2f8aa20 0x5573b2f8ce40 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.178+0000 7f00e6c65740 1 --2- 172.21.15.29:0/3885650955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5573b2f225e0 0x5573b2f2c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.178+0000 7f00e6c65740 1 --2- 172.21.15.29:0/3885650955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573b2f1d470 0x5573b2f2e200 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.178+0000 7f00e6c65740 1 --2- 172.21.15.29:0/3885650955 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573b2f1ca80 0x5573b2f2dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.178+0000 7f00e6c65740 1 -- 172.21.15.29:0/3885650955 >> 172.21.15.29:0/3885650955 conn(0x5573b2f0f7e0 msgr2=0x5573b2f39c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.178+0000 7f00c57fa640 0 client.27236 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:19:44.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.178+0000 7f00e6c65740 1 -- 172.21.15.29:0/3885650955 shutdown_connections 2024-03-11T19:19:44.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.178+0000 7f00e6c65740 1 -- 172.21.15.29:0/3885650955 wait complete. 2024-03-11T19:19:44.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.178+0000 7f00e6c65740 20 client.27236 trim_cache size 0 max 16384 2024-03-11T19:19:44.185 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:19:44.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.179+0000 7f00e4088640 1 -- 172.21.15.29:0/638345311 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f00c80a1ab0 msgr2=0x7f00c80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.179+0000 7f00e4088640 1 --2- 172.21.15.29:0/638345311 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f00c80a1ab0 0x7f00c80a3f70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x5573b2f1dee0 tx=0x7f00d80f5260 comp rx=0 tx=0).stop 2024-03-11T19:19:44.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.179+0000 7f00e4088640 1 -- 172.21.15.29:0/638345311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00dc0ff3d0 msgr2=0x7f00dc1993f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.179+0000 7f00e4088640 1 --2- 172.21.15.29:0/638345311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00dc0ff3d0 0x7f00dc1993f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f00d8000c00 tx=0x7f00d8004040 comp rx=0 tx=0).stop 2024-03-11T19:19:44.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.180+0000 7f00e4088640 1 -- 172.21.15.29:0/638345311 shutdown_connections 2024-03-11T19:19:44.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.180+0000 7f00e4088640 1 --2- 172.21.15.29:0/638345311 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00dc100f80 0x7f00dc1a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.180+0000 7f00e4088640 1 --2- 172.21.15.29:0/638345311 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f00c80a1ab0 0x7f00c80a3f70 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.180+0000 7f00e4088640 1 --2- 172.21.15.29:0/638345311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00dc1005c0 0x7f00dc199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.180+0000 7f00e4088640 1 --2- 172.21.15.29:0/638345311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00dc0ff3d0 0x7f00dc1993f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.180+0000 7f00e4088640 1 -- 172.21.15.29:0/638345311 >> 172.21.15.29:0/638345311 conn(0x7f00dc0fa940 msgr2=0x7f00dc0fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:44.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.180+0000 7f00e4088640 1 -- 172.21.15.29:0/638345311 shutdown_connections 2024-03-11T19:19:44.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.180+0000 7f00e4088640 1 -- 172.21.15.29:0/638345311 wait complete. 2024-03-11T19:19:44.220 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:19:44.220 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:19:44.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.405+0000 7f1ce9193640 1 -- 172.21.15.29:0/3741484191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ce4104590 msgr2=0x7f1ce4104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.405+0000 7f1ce9193640 1 --2- 172.21.15.29:0/3741484191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ce4104590 0x7f1ce4104a10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8009860 tx=0x7f1cd802bb70 comp rx=0 tx=0).stop 2024-03-11T19:19:44.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.405+0000 7f1ce9193640 1 -- 172.21.15.29:0/3741484191 shutdown_connections 2024-03-11T19:19:44.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.405+0000 7f1ce9193640 1 --2- 172.21.15.29:0/3741484191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ce4104f50 0x7f1ce410b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.405+0000 7f1ce9193640 1 --2- 172.21.15.29:0/3741484191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ce4104590 0x7f1ce4104a10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.405+0000 7f1ce9193640 1 --2- 172.21.15.29:0/3741484191 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ce41033a0 0x7f1ce41037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.405+0000 7f1ce9193640 1 -- 172.21.15.29:0/3741484191 >> 172.21.15.29:0/3741484191 conn(0x7f1ce40fe930 msgr2=0x7f1ce4100d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:44.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.405+0000 7f1ce9193640 1 -- 172.21.15.29:0/3741484191 shutdown_connections 2024-03-11T19:19:44.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.405+0000 7f1ce9193640 1 -- 172.21.15.29:0/3741484191 wait complete. 2024-03-11T19:19:44.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.406+0000 7f1ce9193640 1 Processor -- start 2024-03-11T19:19:44.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.406+0000 7f1ce9193640 1 -- start start 2024-03-11T19:19:44.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.407+0000 7f1ce9193640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ce41033a0 0x7f1ce419d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.407+0000 7f1ce9193640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ce4104590 0x7f1ce419dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.407+0000 7f1ce9193640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ce4104f50 0x7f1ce41a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.407+0000 7f1ce9193640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce41a52b0 con 0x7f1ce4104590 2024-03-11T19:19:44.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.407+0000 7f1ce359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ce4104f50 0x7f1ce41a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.407+0000 7f1ce359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ce4104f50 0x7f1ce41a4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:32874/0 (socket says 172.21.15.29:32874) 2024-03-11T19:19:44.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.407+0000 7f1ce359e640 1 -- 172.21.15.29:0/776019571 learned_addr learned my addr 172.21.15.29:0/776019571 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:44.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.407+0000 7f1ce9193640 1 -- 172.21.15.29:0/776019571 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce41a5420 con 0x7f1ce41033a0 2024-03-11T19:19:44.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.407+0000 7f1ce9193640 1 -- 172.21.15.29:0/776019571 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce41a55e0 con 0x7f1ce4104f50 2024-03-11T19:19:44.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.408+0000 7f1ce359e640 1 -- 172.21.15.29:0/776019571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ce41033a0 msgr2=0x7f1ce419d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.408+0000 7f1ce359e640 1 --2- 172.21.15.29:0/776019571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ce41033a0 0x7f1ce419d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.408+0000 7f1ce359e640 1 -- 172.21.15.29:0/776019571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ce4104590 msgr2=0x7f1ce419dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.408+0000 7f1ce359e640 1 --2- 172.21.15.29:0/776019571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ce4104590 0x7f1ce419dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.408+0000 7f1ce359e640 1 -- 172.21.15.29:0/776019571 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd8009490 con 0x7f1ce4104f50 2024-03-11T19:19:44.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.409+0000 7f1ce359e640 1 --2- 172.21.15.29:0/776019571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ce4104f50 0x7f1ce41a4d70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc009fd0 tx=0x7f1cdc00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:44.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.409+0000 7f1ccbfff640 1 -- 172.21.15.29:0/776019571 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc004b20 con 0x7f1ce4104f50 2024-03-11T19:19:44.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.409+0000 7f1ccbfff640 1 -- 172.21.15.29:0/776019571 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1cdc003e20 con 0x7f1ce4104f50 2024-03-11T19:19:44.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.409+0000 7f1ccbfff640 1 -- 172.21.15.29:0/776019571 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc004780 con 0x7f1ce4104f50 2024-03-11T19:19:44.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.409+0000 7f1ce9193640 1 -- 172.21.15.29:0/776019571 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce41a58c0 con 0x7f1ce4104f50 2024-03-11T19:19:44.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.410+0000 7f1ce9193640 1 -- 172.21.15.29:0/776019571 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce41a5d80 con 0x7f1ce4104f50 2024-03-11T19:19:44.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.416+0000 7f1cebd69740 1 Processor -- start 2024-03-11T19:19:44.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.416+0000 7f1ccbfff640 1 -- 172.21.15.29:0/776019571 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1cdc00a040 con 0x7f1ce4104f50 2024-03-11T19:19:44.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.416+0000 7f1cebd69740 1 -- start start 2024-03-11T19:19:44.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.417+0000 7f1cebd69740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afbb270 0x55cc0afbb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.417+0000 7f1cebd69740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbbc60 0x55cc0afc0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.417+0000 7f1cebd69740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc0afc0dd0 0x55cc0afc3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.417+0000 7f1cebd69740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc0af77960 con 0x55cc0afc0dd0 2024-03-11T19:19:44.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.417+0000 7f1cebd69740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc0af2b310 con 0x55cc0afbbc60 2024-03-11T19:19:44.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.417+0000 7f1cebd69740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc0af28840 con 0x55cc0afbb270 2024-03-11T19:19:44.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.417+0000 7f1ce259c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afbb270 0x55cc0afbb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.417+0000 7f1ce259c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afbb270 0x55cc0afbb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:32882/0 (socket says 172.21.15.29:32882) 2024-03-11T19:19:44.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.417+0000 7f1ce259c640 1 -- 172.21.15.29:0/740470364 learned_addr learned my addr 172.21.15.29:0/740470364 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:44.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.417+0000 7f1ce259c640 1 --2- 172.21.15.29:0/740470364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc0afc0dd0 0x55cc0afc3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.418+0000 7f1ce259c640 1 -- 172.21.15.29:0/740470364 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbbc60 msgr2=0x55cc0afc0760 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:19:44.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.418+0000 7f1ce259c640 1 --2- 172.21.15.29:0/740470364 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbbc60 0x55cc0afc0760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.418+0000 7f1ce259c640 1 -- 172.21.15.29:0/740470364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc0afc0dd0 msgr2=0x55cc0afc3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.418+0000 7f1ce259c640 1 --2- 172.21.15.29:0/740470364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc0afc0dd0 0x55cc0afc3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.418+0000 7f1ce259c640 1 -- 172.21.15.29:0/740470364 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc0aee6060 con 0x55cc0afbb270 2024-03-11T19:19:44.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.418+0000 7f1ce259c640 1 --2- 172.21.15.29:0/740470364 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afbb270 0x55cc0afbb690 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8034b10 tx=0x7f1cd802c090 comp rx=0 tx=0).ready entity=mon.1 client_cookie=846ad7f07364d064 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:44.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.419+0000 7f1cc97fa640 1 -- 172.21.15.29:0/740470364 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8034640 con 0x55cc0afbb270 2024-03-11T19:19:44.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.419+0000 7f1cc97fa640 1 -- 172.21.15.29:0/740470364 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1cd802ce90 con 0x55cc0afbb270 2024-03-11T19:19:44.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.419+0000 7f1cc97fa640 1 -- 172.21.15.29:0/740470364 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd802c510 con 0x55cc0afbb270 2024-03-11T19:19:44.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.420+0000 7f1ccbfff640 1 --2- 172.21.15.29:0/776019571 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1ccc0a1b80 0x7f1ccc0a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.420+0000 7f1cebd69740 1 -- 172.21.15.29:0/740470364 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afbb270 msgr2=0x55cc0afbb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.420+0000 7f1cebd69740 1 --2- 172.21.15.29:0/740470364 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afbb270 0x55cc0afbb690 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8034b10 tx=0x7f1cd802c090 comp rx=0 tx=0).stop 2024-03-11T19:19:44.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.420+0000 7f1ccbfff640 1 -- 172.21.15.29:0/776019571 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f1cdc0cd480 con 0x7f1ce4104f50 2024-03-11T19:19:44.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.421+0000 7f1cebd69740 1 -- 172.21.15.29:0/740470364 shutdown_connections 2024-03-11T19:19:44.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.421+0000 7f1cebd69740 1 --2- 172.21.15.29:0/740470364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc0afc0dd0 0x55cc0afc3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.421+0000 7f1cebd69740 1 --2- 172.21.15.29:0/740470364 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbbc60 0x55cc0afc0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.421+0000 7f1cebd69740 1 --2- 172.21.15.29:0/740470364 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afbb270 0x55cc0afbb690 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.422+0000 7f1cebd69740 1 -- 172.21.15.29:0/740470364 >> 172.21.15.29:0/740470364 conn(0x55cc0afb67e0 msgr2=0x55cc0afb6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:44.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.426+0000 7f1ce2d9d640 1 --2- 172.21.15.29:0/776019571 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1ccc0a1b80 0x7f1ccc0a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.426+0000 7f1cebd69740 1 -- 172.21.15.29:0/740470364 shutdown_connections 2024-03-11T19:19:44.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.427+0000 7f1ce2d9d640 1 --2- 172.21.15.29:0/776019571 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1ccc0a1b80 0x7f1ccc0a4040 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x55cc0afb7750 tx=0x7f1cd40096d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:44.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.427+0000 7f1cebd69740 1 -- 172.21.15.29:0/740470364 wait complete. 2024-03-11T19:19:44.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.427+0000 7f1cebd69740 1 -- start start 2024-03-11T19:19:44.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.428+0000 7f1cebd69740 1 Processor -- start 2024-03-11T19:19:44.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.428+0000 7f1cebd69740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afbb270 0x55cc0afc5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.428+0000 7f1cebd69740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbbc60 0x55cc0afc5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.428+0000 7f1cebd69740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc0afc0dd0 0x55cc0afcc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.429+0000 7f1ce259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbbc60 0x55cc0afc5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.429+0000 7f1ce259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbbc60 0x55cc0afc5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59508/0 (socket says 172.21.15.29:59508) 2024-03-11T19:19:44.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.429+0000 7f1ce259c640 1 -- 172.21.15.29:0/2455055591 learned_addr learned my addr 172.21.15.29:0/2455055591 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:44.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.429+0000 7f1cebd69740 1 -- 172.21.15.29:0/2455055591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc0af24020 con 0x55cc0afc0dd0 2024-03-11T19:19:44.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.429+0000 7f1cebd69740 1 -- 172.21.15.29:0/2455055591 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc0abf7430 con 0x55cc0afbbc60 2024-03-11T19:19:44.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.429+0000 7f1cebd69740 1 -- 172.21.15.29:0/2455055591 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc0abb3e00 con 0x55cc0afbb270 2024-03-11T19:19:44.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.429+0000 7f1cebd69740 10 client.0 _ll_register_callbacks cb 0x55cc0aecbcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:44.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.429+0000 7f1ce359e640 1 --2- 172.21.15.29:0/2455055591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc0afc0dd0 0x55cc0afcc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.430+0000 7f1ce359e640 1 -- 172.21.15.29:0/2455055591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afbb270 msgr2=0x55cc0afc5420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:19:44.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.430+0000 7f1ce359e640 1 --2- 172.21.15.29:0/2455055591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afbb270 0x55cc0afc5420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.430+0000 7f1ce359e640 1 -- 172.21.15.29:0/2455055591 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbbc60 msgr2=0x55cc0afc5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.430+0000 7f1ce359e640 1 --2- 172.21.15.29:0/2455055591 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbbc60 0x55cc0afc5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.430+0000 7f1ce359e640 1 -- 172.21.15.29:0/2455055591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce410c0e0 con 0x55cc0afc0dd0 2024-03-11T19:19:44.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.430+0000 7f1ce359e640 1 --2- 172.21.15.29:0/2455055591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc0afc0dd0 0x55cc0afcc9a0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc09a0f0 tx=0x7f1cdc09a240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f817b94ddad673a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:44.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.431+0000 7f1cc6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:19:44.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.431+0000 7f1cc6ff5640 1 -- 172.21.15.29:0/2455055591 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc09e320 con 0x55cc0afc0dd0 2024-03-11T19:19:44.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.431+0000 7f1ce259c640 1 --2- 172.21.15.29:0/2455055591 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbbc60 0x55cc0afc5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:19:44.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.431+0000 7f1cebd69740 1 -- 172.21.15.29:0/2455055591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cc0ac237e0 con 0x55cc0afc0dd0 2024-03-11T19:19:44.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.432+0000 7f1cc6ff5640 1 -- 172.21.15.29:0/2455055591 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f1cdc004e10 con 0x55cc0afc0dd0 2024-03-11T19:19:44.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.432+0000 7f1cc6ff5640 1 -- 172.21.15.29:0/2455055591 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1cdc09aaf0 con 0x55cc0afc0dd0 2024-03-11T19:19:44.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.432+0000 7f1cebd69740 10 client.17751 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:44.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.432+0000 7f1cebd69740 1 -- 172.21.15.29:0/2455055591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55cc0af226a0 con 0x55cc0afc0dd0 2024-03-11T19:19:44.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.433+0000 7f1cc6ff5640 1 -- 172.21.15.29:0/2455055591 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1cdc09a3f0 con 0x55cc0afc0dd0 2024-03-11T19:19:44.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.433+0000 7f1cc6ff5640 1 -- 172.21.15.29:0/2455055591 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1cdc0a2700 con 0x55cc0afc0dd0 2024-03-11T19:19:44.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.433+0000 7f1cebd69740 10 client.17751 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:44.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.433+0000 7f1cebd69740 10 client.17751 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:19:44.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.433+0000 7f1cebd69740 20 client.17751 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:44.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.433+0000 7f1cebd69740 1 --2- 172.21.15.29:0/2455055591 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55cc0afd1760 0x55cc0afd3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.433+0000 7f1cebd69740 4 client.17751 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:19:44.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.433+0000 7f1cebd69740 1 -- 172.21.15.29:0/2455055591 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cc0ac237e0 con 0x55cc0afd1760 2024-03-11T19:19:44.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.436+0000 7f1ce2d9d640 1 --2- 172.21.15.29:0/2455055591 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55cc0afd1760 0x55cc0afd3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.437+0000 7f1ce2d9d640 1 --2- 172.21.15.29:0/2455055591 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55cc0afd1760 0x55cc0afd3bf0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=691b6c26ea46ec55 server_cookie=b4f73bc5d310600d in_seq=0 out_seq=0 2024-03-11T19:19:44.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.437+0000 7f1cc6ff5640 10 client.17751 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:19:44.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.437+0000 7f1cc6ff5640 1 -- 172.21.15.29:0/2455055591 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cc0ac237e0 con 0x55cc0afd1760 2024-03-11T19:19:44.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.437+0000 7f1cc6ff5640 10 client.17751 handle_command_reply: tid=0 2024-03-11T19:19:44.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.437+0000 7f1cebd69740 1 client.17751 shutdown 2024-03-11T19:19:44.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.438+0000 7f1cebd69740 1 -- 172.21.15.29:0/2455055591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc0afc0dd0 msgr2=0x55cc0afcc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.438+0000 7f1cebd69740 1 --2- 172.21.15.29:0/2455055591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc0afc0dd0 0x55cc0afcc9a0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc09a0f0 tx=0x7f1cdc09a240 comp rx=0 tx=0).stop 2024-03-11T19:19:44.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.439+0000 7f1cebd69740 1 -- 172.21.15.29:0/2455055591 shutdown_connections 2024-03-11T19:19:44.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.439+0000 7f1cebd69740 1 --2- 172.21.15.29:0/2455055591 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55cc0afd1760 0x55cc0afd3bf0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.440+0000 7f1cebd69740 1 --2- 172.21.15.29:0/2455055591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc0afc0dd0 0x55cc0afcc9a0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.440+0000 7f1cc6ff5640 0 client.17751 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:19:44.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.440+0000 7f1cebd69740 1 --2- 172.21.15.29:0/2455055591 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbbc60 0x55cc0afc5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.440+0000 7f1cebd69740 1 --2- 172.21.15.29:0/2455055591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afbb270 0x55cc0afc5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.440+0000 7f1cebd69740 1 -- 172.21.15.29:0/2455055591 >> 172.21.15.29:0/2455055591 conn(0x55cc0afb67e0 msgr2=0x55cc0afb5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:44.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.440+0000 7f1cebd69740 1 -- 172.21.15.29:0/2455055591 shutdown_connections 2024-03-11T19:19:44.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.440+0000 7f1cebd69740 1 -- 172.21.15.29:0/2455055591 wait complete. 2024-03-11T19:19:44.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.440+0000 7f1cebd69740 20 client.17751 trim_cache size 0 max 16384 2024-03-11T19:19:44.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.470+0000 7f1cebd69740 1 Processor -- start 2024-03-11T19:19:44.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.470+0000 7f1cebd69740 1 -- start start 2024-03-11T19:19:44.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.471+0000 7f1cebd69740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbb270 0x55cc0afc5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.471+0000 7f1cebd69740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afbbc60 0x55cc0afc5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.471+0000 7f1cebd69740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc0afc0dd0 0x55cc0afcc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.472+0000 7f1cebd69740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc0aea9130 con 0x55cc0afc0dd0 2024-03-11T19:19:44.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.472+0000 7f1cebd69740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc0afb72d0 con 0x55cc0afbb270 2024-03-11T19:19:44.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.472+0000 7f1cebd69740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc0afbaf20 con 0x55cc0afbbc60 2024-03-11T19:19:44.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.472+0000 7f1ce2d9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbb270 0x55cc0afc5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.472+0000 7f1ce259c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afbbc60 0x55cc0afc5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.473+0000 7f1ce2d9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbb270 0x55cc0afc5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59524/0 (socket says 172.21.15.29:59524) 2024-03-11T19:19:44.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.473+0000 7f1ce2d9d640 1 -- 172.21.15.29:0/1191929584 learned_addr learned my addr 172.21.15.29:0/1191929584 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:44.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.474+0000 7f1ce2d9d640 1 -- 172.21.15.29:0/1191929584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afbbc60 msgr2=0x55cc0afc5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.474+0000 7f1ce2d9d640 1 --2- 172.21.15.29:0/1191929584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afbbc60 0x55cc0afc5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.474+0000 7f1ce2d9d640 1 -- 172.21.15.29:0/1191929584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc0afc0dd0 msgr2=0x55cc0afcc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.474+0000 7f1ce2d9d640 1 --2- 172.21.15.29:0/1191929584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc0afc0dd0 0x55cc0afcc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.474+0000 7f1ce2d9d640 1 -- 172.21.15.29:0/1191929584 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc0aebec20 con 0x55cc0afbb270 2024-03-11T19:19:44.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.475+0000 7f1ce2d9d640 1 --2- 172.21.15.29:0/1191929584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbb270 0x55cc0afc5420 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1ce41043f0 tx=0x7f1cd4015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2445baf22b06558 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:44.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.476+0000 7f1cc8ff9640 1 -- 172.21.15.29:0/1191929584 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4010dc0 con 0x55cc0afbb270 2024-03-11T19:19:44.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.476+0000 7f1cc8ff9640 1 -- 172.21.15.29:0/1191929584 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1cd4022040 con 0x55cc0afbb270 2024-03-11T19:19:44.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.476+0000 7f1cc8ff9640 1 -- 172.21.15.29:0/1191929584 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd401c420 con 0x55cc0afbb270 2024-03-11T19:19:44.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.477+0000 7f1cebd69740 1 -- 172.21.15.29:0/1191929584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbb270 msgr2=0x55cc0afc5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.477+0000 7f1cebd69740 1 --2- 172.21.15.29:0/1191929584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbb270 0x55cc0afc5420 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1ce41043f0 tx=0x7f1cd4015000 comp rx=0 tx=0).stop 2024-03-11T19:19:44.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.480+0000 7f1cebd69740 1 -- 172.21.15.29:0/1191929584 shutdown_connections 2024-03-11T19:19:44.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.480+0000 7f1cebd69740 1 --2- 172.21.15.29:0/1191929584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc0afc0dd0 0x55cc0afcc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.480+0000 7f1cebd69740 1 --2- 172.21.15.29:0/1191929584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afbbc60 0x55cc0afc5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.480+0000 7f1cebd69740 1 --2- 172.21.15.29:0/1191929584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbb270 0x55cc0afc5420 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.480+0000 7f1cebd69740 1 -- 172.21.15.29:0/1191929584 >> 172.21.15.29:0/1191929584 conn(0x55cc0afb67e0 msgr2=0x55cc0afb6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:44.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.480+0000 7f1cebd69740 1 -- 172.21.15.29:0/1191929584 shutdown_connections 2024-03-11T19:19:44.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.480+0000 7f1cebd69740 1 -- 172.21.15.29:0/1191929584 wait complete. 2024-03-11T19:19:44.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.481+0000 7f1cebd69740 1 -- start start 2024-03-11T19:19:44.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.481+0000 7f1cebd69740 1 Processor -- start 2024-03-11T19:19:44.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.481+0000 7f1cebd69740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc0afbb270 0x55cc0afcc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.481+0000 7f1cebd69740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbbc60 0x55cc0afcad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.481+0000 7f1cebd69740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afc0dd0 0x55cc0afcb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.481+0000 7f1cebd69740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc0afcb7f0 con 0x55cc0afbb270 2024-03-11T19:19:44.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.481+0000 7f1cebd69740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc0afcb960 con 0x55cc0afbbc60 2024-03-11T19:19:44.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.481+0000 7f1cebd69740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc0afcbb20 con 0x55cc0afc0dd0 2024-03-11T19:19:44.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.481+0000 7f1cebd69740 10 client.0 _ll_register_callbacks cb 0x7f1cd8037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:44.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.482+0000 7f1ce359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afc0dd0 0x55cc0afcb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.482+0000 7f1ce359e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afc0dd0 0x55cc0afcb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:32902/0 (socket says 172.21.15.29:32902) 2024-03-11T19:19:44.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.482+0000 7f1ce359e640 1 -- 172.21.15.29:0/3231232307 learned_addr learned my addr 172.21.15.29:0/3231232307 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:44.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.482+0000 7f1ce259c640 1 --2- 172.21.15.29:0/3231232307 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbbc60 0x55cc0afcad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.482+0000 7f1ce359e640 1 -- 172.21.15.29:0/3231232307 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbbc60 msgr2=0x55cc0afcad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.482+0000 7f1ce359e640 1 --2- 172.21.15.29:0/3231232307 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbbc60 0x55cc0afcad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.482+0000 7f1ce359e640 1 -- 172.21.15.29:0/3231232307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc0afbb270 msgr2=0x55cc0afcc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.482+0000 7f1ce359e640 1 --2- 172.21.15.29:0/3231232307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc0afbb270 0x55cc0afcc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.482+0000 7f1ce359e640 1 -- 172.21.15.29:0/3231232307 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc0aee48c0 con 0x55cc0afc0dd0 2024-03-11T19:19:44.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.483+0000 7f1ce359e640 1 --2- 172.21.15.29:0/3231232307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afc0dd0 0x55cc0afcb2b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc09a330 tx=0x7f1cdc0a2930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56e849de7cf3958f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:44.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.483+0000 7f1cc6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:19:44.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.483+0000 7f1cc6ff5640 1 -- 172.21.15.29:0/3231232307 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc09a850 con 0x55cc0afc0dd0 2024-03-11T19:19:44.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.483+0000 7f1cc6ff5640 1 -- 172.21.15.29:0/3231232307 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f1cdc004e10 con 0x55cc0afc0dd0 2024-03-11T19:19:44.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.483+0000 7f1cebd69740 1 -- 172.21.15.29:0/3231232307 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1cd802ce90 con 0x55cc0afc0dd0 2024-03-11T19:19:44.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.484+0000 7f1cc6ff5640 1 -- 172.21.15.29:0/3231232307 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1cdc004c80 con 0x55cc0afc0dd0 2024-03-11T19:19:44.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.485+0000 7f1cebd69740 10 client.27289 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:44.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.485+0000 7f1cebd69740 1 -- 172.21.15.29:0/3231232307 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55cc0aecb050 con 0x55cc0afc0dd0 2024-03-11T19:19:44.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.485+0000 7f1cc6ff5640 1 -- 172.21.15.29:0/3231232307 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1cdc0a6030 con 0x55cc0afc0dd0 2024-03-11T19:19:44.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.486+0000 7f1cc6ff5640 1 -- 172.21.15.29:0/3231232307 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1cdc099840 con 0x55cc0afc0dd0 2024-03-11T19:19:44.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.486+0000 7f1cebd69740 10 client.27289 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:44.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.486+0000 7f1cebd69740 10 client.27289 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:19:44.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.486+0000 7f1cebd69740 20 client.27289 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:44.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.486+0000 7f1cebd69740 1 --2- 172.21.15.29:0/3231232307 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55cc0afd1760 0x55cc0b02b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.486+0000 7f1cebd69740 4 client.27289 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:19:44.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.486+0000 7f1cebd69740 1 -- 172.21.15.29:0/3231232307 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cc0ac60720 con 0x55cc0afd1760 2024-03-11T19:19:44.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.487+0000 7f1ce2d9d640 1 --2- 172.21.15.29:0/3231232307 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55cc0afd1760 0x55cc0b02b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.487+0000 7f1ce2d9d640 1 --2- 172.21.15.29:0/3231232307 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55cc0afd1760 0x55cc0b02b2d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ba04b466adafa6c6 server_cookie=47e080e6a8dc8674 in_seq=0 out_seq=0 2024-03-11T19:19:44.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.488+0000 7f1cc6ff5640 10 client.27289 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:19:44.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.488+0000 7f1cc6ff5640 1 -- 172.21.15.29:0/3231232307 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cc0ac60720 con 0x55cc0afd1760 2024-03-11T19:19:44.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.488+0000 7f1cc6ff5640 10 client.27289 handle_command_reply: tid=0 2024-03-11T19:19:44.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.488+0000 7f1cebd69740 1 client.27289 shutdown 2024-03-11T19:19:44.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.489+0000 7f1cebd69740 1 -- 172.21.15.29:0/3231232307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afc0dd0 msgr2=0x55cc0afcb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.489+0000 7f1cebd69740 1 --2- 172.21.15.29:0/3231232307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afc0dd0 0x55cc0afcb2b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc09a330 tx=0x7f1cdc0a2930 comp rx=0 tx=0).stop 2024-03-11T19:19:44.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.489+0000 7f1cebd69740 1 -- 172.21.15.29:0/3231232307 shutdown_connections 2024-03-11T19:19:44.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.489+0000 7f1cebd69740 1 --2- 172.21.15.29:0/3231232307 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55cc0afd1760 0x55cc0b02b2d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.489+0000 7f1cebd69740 1 --2- 172.21.15.29:0/3231232307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc0afc0dd0 0x55cc0afcb2b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.489+0000 7f1cebd69740 1 --2- 172.21.15.29:0/3231232307 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc0afbbc60 0x55cc0afcad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.489+0000 7f1cebd69740 1 --2- 172.21.15.29:0/3231232307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cc0afbb270 0x55cc0afcc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.489+0000 7f1cebd69740 1 -- 172.21.15.29:0/3231232307 >> 172.21.15.29:0/3231232307 conn(0x55cc0afb67e0 msgr2=0x55cc0afc53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:44.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.489+0000 7f1cc6ff5640 0 client.27289 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:19:44.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.489+0000 7f1cebd69740 1 -- 172.21.15.29:0/3231232307 shutdown_connections 2024-03-11T19:19:44.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.489+0000 7f1cebd69740 1 -- 172.21.15.29:0/3231232307 wait complete. 2024-03-11T19:19:44.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.491+0000 7f1cebd69740 20 client.27289 trim_cache size 0 max 16384 2024-03-11T19:19:44.497 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:19:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.497+0000 7f1ce9193640 1 -- 172.21.15.29:0/776019571 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1ccc0a1b80 msgr2=0x7f1ccc0a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.498+0000 7f1ce9193640 1 --2- 172.21.15.29:0/776019571 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1ccc0a1b80 0x7f1ccc0a4040 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x55cc0afb7750 tx=0x7f1cd40096d0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.498+0000 7f1ce9193640 1 -- 172.21.15.29:0/776019571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ce4104f50 msgr2=0x7f1ce41a4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.498+0000 7f1ce9193640 1 --2- 172.21.15.29:0/776019571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ce4104f50 0x7f1ce41a4d70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc009fd0 tx=0x7f1cdc00c800 comp rx=0 tx=0).stop 2024-03-11T19:19:44.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.498+0000 7f1ce9193640 1 -- 172.21.15.29:0/776019571 shutdown_connections 2024-03-11T19:19:44.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.498+0000 7f1ce9193640 1 --2- 172.21.15.29:0/776019571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ce4104f50 0x7f1ce41a4d70 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.499+0000 7f1ce9193640 1 --2- 172.21.15.29:0/776019571 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1ccc0a1b80 0x7f1ccc0a4040 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.499+0000 7f1ce9193640 1 --2- 172.21.15.29:0/776019571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ce4104590 0x7f1ce419dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.499+0000 7f1ce9193640 1 --2- 172.21.15.29:0/776019571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ce41033a0 0x7f1ce419d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.499+0000 7f1ce9193640 1 -- 172.21.15.29:0/776019571 >> 172.21.15.29:0/776019571 conn(0x7f1ce40fe930 msgr2=0x7f1ce40775d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.502+0000 7f1ce9193640 1 -- 172.21.15.29:0/776019571 shutdown_connections 2024-03-11T19:19:44.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.503+0000 7f1ce9193640 1 -- 172.21.15.29:0/776019571 wait complete. 2024-03-11T19:19:44.527 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:19:44.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:19:44.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.708+0000 7f704325e640 1 -- 172.21.15.29:0/3743468372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f703c0ffea0 msgr2=0x7f703c1002c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.708+0000 7f704325e640 1 --2- 172.21.15.29:0/3743468372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f703c0ffea0 0x7f703c1002c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7038009860 tx=0x7f703802bb70 comp rx=0 tx=0).stop 2024-03-11T19:19:44.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.708+0000 7f704325e640 1 -- 172.21.15.29:0/3743468372 shutdown_connections 2024-03-11T19:19:44.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.708+0000 7f704325e640 1 --2- 172.21.15.29:0/3743468372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f703c1019b0 0x7f703c1082d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.708+0000 7f704325e640 1 --2- 172.21.15.29:0/3743468372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f703c100ff0 0x7f703c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.708+0000 7f704325e640 1 --2- 172.21.15.29:0/3743468372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f703c0ffea0 0x7f703c1002c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.708+0000 7f704325e640 1 -- 172.21.15.29:0/3743468372 >> 172.21.15.29:0/3743468372 conn(0x7f703c0fb450 msgr2=0x7f703c0fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:44.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.708+0000 7f704325e640 1 -- 172.21.15.29:0/3743468372 shutdown_connections 2024-03-11T19:19:44.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.708+0000 7f704325e640 1 -- 172.21.15.29:0/3743468372 wait complete. 2024-03-11T19:19:44.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.709+0000 7f704325e640 1 Processor -- start 2024-03-11T19:19:44.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.709+0000 7f704325e640 1 -- start start 2024-03-11T19:19:44.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.709+0000 7f704325e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f703c0ffea0 0x7f703c199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.709+0000 7f704325e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f703c100ff0 0x7f703c199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.709+0000 7f704325e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f703c1019b0 0x7f703c1a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.709+0000 7f704325e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c1a0c50 con 0x7f703c0ffea0 2024-03-11T19:19:44.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.709+0000 7f704325e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c1a0dc0 con 0x7f703c100ff0 2024-03-11T19:19:44.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.709+0000 7f704325e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c1a0f80 con 0x7f703c1019b0 2024-03-11T19:19:44.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.709+0000 7f704225c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f703c0ffea0 0x7f703c199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.709+0000 7f704225c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f703c0ffea0 0x7f703c199200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44182/0 (socket says 172.21.15.29:44182) 2024-03-11T19:19:44.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.709+0000 7f704225c640 1 -- 172.21.15.29:0/3904865232 learned_addr learned my addr 172.21.15.29:0/3904865232 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:44.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.709+0000 7f7041a5b640 1 --2- 172.21.15.29:0/3904865232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f703c100ff0 0x7f703c199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.709+0000 7f7042a5d640 1 --2- 172.21.15.29:0/3904865232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f703c1019b0 0x7f703c1a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.710+0000 7f7041a5b640 1 -- 172.21.15.29:0/3904865232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f703c1019b0 msgr2=0x7f703c1a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.710+0000 7f7041a5b640 1 --2- 172.21.15.29:0/3904865232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f703c1019b0 0x7f703c1a0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.710+0000 7f7041a5b640 1 -- 172.21.15.29:0/3904865232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f703c0ffea0 msgr2=0x7f703c199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.710+0000 7f7041a5b640 1 --2- 172.21.15.29:0/3904865232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f703c0ffea0 0x7f703c199200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.710+0000 7f7041a5b640 1 -- 172.21.15.29:0/3904865232 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7038009490 con 0x7f703c100ff0 2024-03-11T19:19:44.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.710+0000 7f704225c640 1 --2- 172.21.15.29:0/3904865232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f703c0ffea0 0x7f703c199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:44.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.710+0000 7f7041a5b640 1 --2- 172.21.15.29:0/3904865232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f703c100ff0 0x7f703c199740 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f702c005fd0 tx=0x7f702c00ec20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:44.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.710+0000 7f70337fe640 1 -- 172.21.15.29:0/3904865232 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c00ccc0 con 0x7f703c100ff0 2024-03-11T19:19:44.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.710+0000 7f704325e640 1 -- 172.21.15.29:0/3904865232 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703c1a1260 con 0x7f703c100ff0 2024-03-11T19:19:44.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.710+0000 7f704325e640 1 -- 172.21.15.29:0/3904865232 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f703c1a17c0 con 0x7f703c100ff0 2024-03-11T19:19:44.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.711+0000 7f70337fe640 1 -- 172.21.15.29:0/3904865232 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f702c0044d0 con 0x7f703c100ff0 2024-03-11T19:19:44.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.711+0000 7f70337fe640 1 -- 172.21.15.29:0/3904865232 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c003d60 con 0x7f703c100ff0 2024-03-11T19:19:44.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.713+0000 7f70337fe640 1 -- 172.21.15.29:0/3904865232 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f702c018430 con 0x7f703c100ff0 2024-03-11T19:19:44.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.714+0000 7f704716d740 1 Processor -- start 2024-03-11T19:19:44.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.715+0000 7f704716d740 1 -- start start 2024-03-11T19:19:44.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.715+0000 7f70337fe640 1 --2- 172.21.15.29:0/3904865232 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f70280a1ab0 0x7f70280a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.715+0000 7f704716d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 0x55d0a18ab0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.715+0000 7f704716d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18ab680 0x55d0a18b0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.715+0000 7f704716d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18b07f0 0x55d0a18b2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.715+0000 7f704716d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0a185e960 con 0x55d0a18aac90 2024-03-11T19:19:44.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.715+0000 7f704716d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0a1812310 con 0x55d0a18ab680 2024-03-11T19:19:44.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.715+0000 7f70337fe640 1 -- 172.21.15.29:0/3904865232 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f702c014070 con 0x7f703c100ff0 2024-03-11T19:19:44.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.715+0000 7f7041a5b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18ab680 0x55d0a18b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.715+0000 7f7041a5b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18ab680 0x55d0a18b0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59572/0 (socket says 172.21.15.29:59572) 2024-03-11T19:19:44.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.715+0000 7f7041a5b640 1 -- 172.21.15.29:0/2892920846 learned_addr learned my addr 172.21.15.29:0/2892920846 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:44.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.715+0000 7f7042a5d640 1 --2- 172.21.15.29:0/2892920846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18b07f0 0x55d0a18b2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.715+0000 7f704225c640 1 --2- 172.21.15.29:0/2892920846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 0x55d0a18ab0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.715+0000 7f7042a5d640 1 --2- 172.21.15.29:0/3904865232 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f70280a1ab0 0x7f70280a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.715+0000 7f704716d740 1 -- 172.21.15.29:0/2892920846 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0a180f840 con 0x55d0a18b07f0 2024-03-11T19:19:44.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.716+0000 7f704225c640 1 -- 172.21.15.29:0/2892920846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18b07f0 msgr2=0x55d0a18b2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.716+0000 7f704225c640 1 --2- 172.21.15.29:0/2892920846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18b07f0 0x55d0a18b2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.716+0000 7f704225c640 1 -- 172.21.15.29:0/2892920846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18ab680 msgr2=0x55d0a18b0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.716+0000 7f704225c640 1 --2- 172.21.15.29:0/2892920846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18ab680 0x55d0a18b0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.716+0000 7f704225c640 1 -- 172.21.15.29:0/2892920846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0a17cd060 con 0x55d0a18aac90 2024-03-11T19:19:44.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.716+0000 7f7041a5b640 1 --2- 172.21.15.29:0/2892920846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18ab680 0x55d0a18b0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:44.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.716+0000 7f704225c640 1 --2- 172.21.15.29:0/2892920846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 0x55d0a18ab0b0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f703802c180 tx=0x7f7038034ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=569c43c97fdd142f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:44.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.717+0000 7f7042a5d640 1 --2- 172.21.15.29:0/3904865232 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f70280a1ab0 0x7f70280a3f70 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7034005fd0 tx=0x7f7034009e90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:44.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.717+0000 7f7030bfe640 1 -- 172.21.15.29:0/2892920846 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038034640 con 0x55d0a18aac90 2024-03-11T19:19:44.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.717+0000 7f7030bfe640 1 -- 172.21.15.29:0/2892920846 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f703802ce90 con 0x55d0a18aac90 2024-03-11T19:19:44.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.717+0000 7f7030bfe640 1 -- 172.21.15.29:0/2892920846 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703802cb50 con 0x55d0a18aac90 2024-03-11T19:19:44.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.718+0000 7f704716d740 1 -- 172.21.15.29:0/2892920846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 msgr2=0x55d0a18ab0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.718+0000 7f704716d740 1 --2- 172.21.15.29:0/2892920846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 0x55d0a18ab0b0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f703802c180 tx=0x7f7038034ab0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.718+0000 7f704716d740 1 -- 172.21.15.29:0/2892920846 shutdown_connections 2024-03-11T19:19:44.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.718+0000 7f704716d740 1 --2- 172.21.15.29:0/2892920846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18b07f0 0x55d0a18b2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.718+0000 7f704716d740 1 --2- 172.21.15.29:0/2892920846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18ab680 0x55d0a18b0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.718+0000 7f704716d740 1 --2- 172.21.15.29:0/2892920846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 0x55d0a18ab0b0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.718+0000 7f704716d740 1 -- 172.21.15.29:0/2892920846 >> 172.21.15.29:0/2892920846 conn(0x55d0a189d9f0 msgr2=0x55d0a189ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:44.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.718+0000 7f704716d740 1 -- 172.21.15.29:0/2892920846 shutdown_connections 2024-03-11T19:19:44.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.718+0000 7f704716d740 1 -- 172.21.15.29:0/2892920846 wait complete. 2024-03-11T19:19:44.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.718+0000 7f704716d740 1 -- start start 2024-03-11T19:19:44.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.718+0000 7f704716d740 1 Processor -- start 2024-03-11T19:19:44.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.719+0000 7f704716d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18aac90 0x55d0a18b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.719+0000 7f704716d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18ab680 0x55d0a18b5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.719+0000 7f704716d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18b07f0 0x55d0a18bc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.719+0000 7f704225c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18aac90 0x55d0a18b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.719+0000 7f704225c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18aac90 0x55d0a18b4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:32928/0 (socket says 172.21.15.29:32928) 2024-03-11T19:19:44.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.719+0000 7f704225c640 1 -- 172.21.15.29:0/2894987470 learned_addr learned my addr 172.21.15.29:0/2894987470 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:44.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.719+0000 7f704716d740 1 -- 172.21.15.29:0/2894987470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0a180b020 con 0x55d0a18b07f0 2024-03-11T19:19:44.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.719+0000 7f704716d740 1 -- 172.21.15.29:0/2894987470 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0a14de430 con 0x55d0a18ab680 2024-03-11T19:19:44.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.719+0000 7f704716d740 1 -- 172.21.15.29:0/2894987470 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0a149ae00 con 0x55d0a18aac90 2024-03-11T19:19:44.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.719+0000 7f7041a5b640 1 --2- 172.21.15.29:0/2894987470 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18ab680 0x55d0a18b5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.719+0000 7f7042a5d640 1 --2- 172.21.15.29:0/2894987470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18b07f0 0x55d0a18bc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.719+0000 7f704716d740 10 client.0 _ll_register_callbacks cb 0x55d0a17b2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:44.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.720+0000 7f704225c640 1 -- 172.21.15.29:0/2894987470 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18ab680 msgr2=0x55d0a18b5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.720+0000 7f704225c640 1 --2- 172.21.15.29:0/2894987470 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18ab680 0x55d0a18b5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.720+0000 7f704225c640 1 -- 172.21.15.29:0/2894987470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18b07f0 msgr2=0x55d0a18bc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.720+0000 7f704225c640 1 --2- 172.21.15.29:0/2894987470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18b07f0 0x55d0a18bc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.720+0000 7f704225c640 1 -- 172.21.15.29:0/2894987470 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f703c108c00 con 0x55d0a18aac90 2024-03-11T19:19:44.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.720+0000 7f7041a5b640 1 --2- 172.21.15.29:0/2894987470 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18ab680 0x55d0a18b5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:44.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.720+0000 7f704225c640 1 --2- 172.21.15.29:0/2894987470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18aac90 0x55d0a18b4df0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7038009830 tx=0x7f7038034670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a78c14281cd1049d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:44.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.720+0000 7f7025ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:19:44.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.720+0000 7f7025ffb640 1 -- 172.21.15.29:0/2894987470 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038004450 con 0x55d0a18aac90 2024-03-11T19:19:44.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.720+0000 7f7025ffb640 1 -- 172.21.15.29:0/2894987470 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f703802c510 con 0x55d0a18aac90 2024-03-11T19:19:44.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.721+0000 7f704716d740 1 -- 172.21.15.29:0/2894987470 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d0a150a7e0 con 0x55d0a18aac90 2024-03-11T19:19:44.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.721+0000 7f7025ffb640 1 -- 172.21.15.29:0/2894987470 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7038032600 con 0x55d0a18aac90 2024-03-11T19:19:44.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.721+0000 7f704716d740 10 client.27301 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:44.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.721+0000 7f704716d740 1 -- 172.21.15.29:0/2894987470 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d0a18096a0 con 0x55d0a18aac90 2024-03-11T19:19:44.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.722+0000 7f7025ffb640 1 -- 172.21.15.29:0/2894987470 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f703804d070 con 0x55d0a18aac90 2024-03-11T19:19:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.722+0000 7f7025ffb640 1 -- 172.21.15.29:0/2894987470 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f703805a050 con 0x55d0a18aac90 2024-03-11T19:19:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.722+0000 7f704716d740 10 client.27301 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.722+0000 7f704716d740 10 client.27301 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:19:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.722+0000 7f704716d740 20 client.27301 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.722+0000 7f704716d740 1 --2- 172.21.15.29:0/2894987470 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55d0a18c0b40 0x55d0a18c3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.723+0000 7f704716d740 4 client.27301 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:19:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.723+0000 7f704716d740 1 -- 172.21.15.29:0/2894987470 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d0a150a7e0 con 0x55d0a18c0b40 2024-03-11T19:19:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.723+0000 7f7041a5b640 1 --2- 172.21.15.29:0/2894987470 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55d0a18c0b40 0x55d0a18c3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.724+0000 7f7041a5b640 1 --2- 172.21.15.29:0/2894987470 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55d0a18c0b40 0x55d0a18c3000 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=422384b8cb301145 server_cookie=bfd00b1267d07cca in_seq=0 out_seq=0 2024-03-11T19:19:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.724+0000 7f7025ffb640 10 client.27301 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:19:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.725+0000 7f7025ffb640 1 -- 172.21.15.29:0/2894987470 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d0a150a7e0 con 0x55d0a18c0b40 2024-03-11T19:19:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.725+0000 7f7025ffb640 10 client.27301 handle_command_reply: tid=0 2024-03-11T19:19:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.725+0000 7f704716d740 1 client.27301 shutdown 2024-03-11T19:19:44.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.726+0000 7f704716d740 1 -- 172.21.15.29:0/2894987470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18aac90 msgr2=0x55d0a18b4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.726+0000 7f704716d740 1 --2- 172.21.15.29:0/2894987470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18aac90 0x55d0a18b4df0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7038009830 tx=0x7f7038034670 comp rx=0 tx=0).stop 2024-03-11T19:19:44.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.726+0000 7f704716d740 1 -- 172.21.15.29:0/2894987470 shutdown_connections 2024-03-11T19:19:44.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.726+0000 7f704716d740 1 --2- 172.21.15.29:0/2894987470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18b07f0 0x55d0a18bc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.726+0000 7f704716d740 1 --2- 172.21.15.29:0/2894987470 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55d0a18c0b40 0x55d0a18c3000 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.726+0000 7f704716d740 1 --2- 172.21.15.29:0/2894987470 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18ab680 0x55d0a18b5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.726+0000 7f704716d740 1 --2- 172.21.15.29:0/2894987470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18aac90 0x55d0a18b4df0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.726+0000 7f704716d740 1 -- 172.21.15.29:0/2894987470 >> 172.21.15.29:0/2894987470 conn(0x55d0a189d9f0 msgr2=0x55d0a189ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:44.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.726+0000 7f7025ffb640 0 client.27301 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:19:44.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.726+0000 7f704716d740 1 -- 172.21.15.29:0/2894987470 shutdown_connections 2024-03-11T19:19:44.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.727+0000 7f704716d740 1 -- 172.21.15.29:0/2894987470 wait complete. 2024-03-11T19:19:44.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.727+0000 7f704716d740 20 client.27301 trim_cache size 0 max 16384 2024-03-11T19:19:44.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.761+0000 7f704716d740 1 Processor -- start 2024-03-11T19:19:44.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.761+0000 7f704716d740 1 -- start start 2024-03-11T19:19:44.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.762+0000 7f704716d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 0x55d0a189c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.762+0000 7f704716d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18ab680 0x55d0a18bc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.762+0000 7f704716d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18b07f0 0x55d0a189c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.762+0000 7f704225c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 0x55d0a189c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.762+0000 7f704225c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 0x55d0a189c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44198/0 (socket says 172.21.15.29:44198) 2024-03-11T19:19:44.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.762+0000 7f704225c640 1 -- 172.21.15.29:0/3191809576 learned_addr learned my addr 172.21.15.29:0/3191809576 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:44.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.762+0000 7f7041a5b640 1 --2- 172.21.15.29:0/3191809576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18ab680 0x55d0a18bc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.763+0000 7f7042a5d640 1 --2- 172.21.15.29:0/3191809576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18b07f0 0x55d0a189c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.763+0000 7f704716d740 1 -- 172.21.15.29:0/3191809576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0a1790130 con 0x55d0a18aac90 2024-03-11T19:19:44.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.763+0000 7f704716d740 1 -- 172.21.15.29:0/3191809576 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0a18b3340 con 0x55d0a18b07f0 2024-03-11T19:19:44.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.763+0000 7f704716d740 1 -- 172.21.15.29:0/3191809576 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0a150a7e0 con 0x55d0a18ab680 2024-03-11T19:19:44.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.763+0000 7f7041a5b640 1 -- 172.21.15.29:0/3191809576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18b07f0 msgr2=0x55d0a189c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.763+0000 7f7041a5b640 1 --2- 172.21.15.29:0/3191809576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18b07f0 0x55d0a189c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.763+0000 7f7041a5b640 1 -- 172.21.15.29:0/3191809576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 msgr2=0x55d0a189c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.763+0000 7f7041a5b640 1 --2- 172.21.15.29:0/3191809576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 0x55d0a189c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.763+0000 7f7041a5b640 1 -- 172.21.15.29:0/3191809576 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0a17a5c20 con 0x55d0a18ab680 2024-03-11T19:19:44.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.763+0000 7f704225c640 1 --2- 172.21.15.29:0/3191809576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 0x55d0a189c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:44.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.764+0000 7f7041a5b640 1 --2- 172.21.15.29:0/3191809576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18ab680 0x55d0a18bc330 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f702c000c00 tx=0x7f702c0cb010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a9749ef7a9240d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:44.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.764+0000 7f7027fff640 1 -- 172.21.15.29:0/3191809576 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c09acb0 con 0x55d0a18ab680 2024-03-11T19:19:44.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.764+0000 7f7027fff640 1 -- 172.21.15.29:0/3191809576 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f702c00ce20 con 0x55d0a18ab680 2024-03-11T19:19:44.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.764+0000 7f7027fff640 1 -- 172.21.15.29:0/3191809576 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c008ea0 con 0x55d0a18ab680 2024-03-11T19:19:44.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.765+0000 7f704716d740 1 -- 172.21.15.29:0/3191809576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18ab680 msgr2=0x55d0a18bc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.765+0000 7f704716d740 1 --2- 172.21.15.29:0/3191809576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18ab680 0x55d0a18bc330 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f702c000c00 tx=0x7f702c0cb010 comp rx=0 tx=0).stop 2024-03-11T19:19:44.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.765+0000 7f704716d740 1 -- 172.21.15.29:0/3191809576 shutdown_connections 2024-03-11T19:19:44.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.765+0000 7f704716d740 1 --2- 172.21.15.29:0/3191809576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18b07f0 0x55d0a189c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.766+0000 7f704716d740 1 --2- 172.21.15.29:0/3191809576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18ab680 0x55d0a18bc330 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.766+0000 7f704716d740 1 --2- 172.21.15.29:0/3191809576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 0x55d0a189c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.766+0000 7f704716d740 1 -- 172.21.15.29:0/3191809576 >> 172.21.15.29:0/3191809576 conn(0x55d0a189d9f0 msgr2=0x55d0a18acb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:44.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.766+0000 7f704716d740 1 -- 172.21.15.29:0/3191809576 shutdown_connections 2024-03-11T19:19:44.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.766+0000 7f704716d740 1 -- 172.21.15.29:0/3191809576 wait complete. 2024-03-11T19:19:44.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.766+0000 7f704716d740 1 -- start start 2024-03-11T19:19:44.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.766+0000 7f704716d740 1 Processor -- start 2024-03-11T19:19:44.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.767+0000 7f704716d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 0x55d0a18bc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.767+0000 7f704716d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18ab680 0x55d0a18ba7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.767+0000 7f704716d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18b07f0 0x55d0a18bace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.767+0000 7f704716d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0a18bb220 con 0x55d0a18aac90 2024-03-11T19:19:44.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.767+0000 7f704716d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0a18bb390 con 0x55d0a18ab680 2024-03-11T19:19:44.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.767+0000 7f704716d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0a18bb550 con 0x55d0a18b07f0 2024-03-11T19:19:44.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.767+0000 7f704716d740 10 client.0 _ll_register_callbacks cb 0x7f702c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:44.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.767+0000 7f704225c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 0x55d0a18bc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.767+0000 7f704225c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 0x55d0a18bc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44204/0 (socket says 172.21.15.29:44204) 2024-03-11T19:19:44.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.767+0000 7f704225c640 1 -- 172.21.15.29:0/1175665165 learned_addr learned my addr 172.21.15.29:0/1175665165 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:44.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.767+0000 7f7041a5b640 1 --2- 172.21.15.29:0/1175665165 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18ab680 0x55d0a18ba7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.767+0000 7f7042a5d640 1 --2- 172.21.15.29:0/1175665165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18b07f0 0x55d0a18bace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.767+0000 7f704225c640 1 -- 172.21.15.29:0/1175665165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18b07f0 msgr2=0x55d0a18bace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.767+0000 7f704225c640 1 --2- 172.21.15.29:0/1175665165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18b07f0 0x55d0a18bace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.767+0000 7f704225c640 1 -- 172.21.15.29:0/1175665165 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18ab680 msgr2=0x55d0a18ba7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:44.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.767+0000 7f704225c640 1 --2- 172.21.15.29:0/1175665165 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18ab680 0x55d0a18ba7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.767+0000 7f704225c640 1 -- 172.21.15.29:0/1175665165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d0a18096a0 con 0x55d0a18aac90 2024-03-11T19:19:44.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.767+0000 7f7041a5b640 1 --2- 172.21.15.29:0/1175665165 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18ab680 0x55d0a18ba7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:44.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.768+0000 7f704225c640 1 --2- 172.21.15.29:0/1175665165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 0x55d0a18bc0d0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f7038006010 tx=0x7f7038004480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0f1cad660b42eca server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:44.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.768+0000 7f7025ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:19:44.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.768+0000 7f7025ffb640 1 -- 172.21.15.29:0/1175665165 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038003800 con 0x55d0a18aac90 2024-03-11T19:19:44.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.768+0000 7f7025ffb640 1 -- 172.21.15.29:0/1175665165 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7038032600 con 0x55d0a18aac90 2024-03-11T19:19:44.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.768+0000 7f704716d740 1 -- 172.21.15.29:0/1175665165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f703802ce90 con 0x55d0a18aac90 2024-03-11T19:19:44.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.768+0000 7f7025ffb640 1 -- 172.21.15.29:0/1175665165 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70380327c0 con 0x55d0a18aac90 2024-03-11T19:19:44.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.768+0000 7f704716d740 10 client.17769 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:44.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.768+0000 7f704716d740 1 -- 172.21.15.29:0/1175665165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7f703c1a1260 con 0x55d0a18aac90 2024-03-11T19:19:44.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.769+0000 7f7025ffb640 1 -- 172.21.15.29:0/1175665165 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f703806d020 con 0x55d0a18aac90 2024-03-11T19:19:44.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.769+0000 7f7025ffb640 1 -- 172.21.15.29:0/1175665165 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f703805a070 con 0x55d0a18aac90 2024-03-11T19:19:44.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.769+0000 7f704716d740 10 client.17769 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:44.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.769+0000 7f704716d740 10 client.17769 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:19:44.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.769+0000 7f704716d740 20 client.17769 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:44.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.769+0000 7f704716d740 1 --2- 172.21.15.29:0/1175665165 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55d0a19184d0 0x55d0a191a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:44.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.769+0000 7f704716d740 4 client.17769 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:19:44.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.769+0000 7f704716d740 1 -- 172.21.15.29:0/1175665165 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d0a1547720 con 0x55d0a19184d0 2024-03-11T19:19:44.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.769+0000 7f7041a5b640 1 --2- 172.21.15.29:0/1175665165 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55d0a19184d0 0x55d0a191a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.770+0000 7f7041a5b640 1 --2- 172.21.15.29:0/1175665165 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55d0a19184d0 0x55d0a191a8c0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=73973eb74ed1c65b server_cookie=1f123d7350b83f69 in_seq=0 out_seq=0 2024-03-11T19:19:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.770+0000 7f7025ffb640 10 client.17769 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:19:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.770+0000 7f7025ffb640 1 -- 172.21.15.29:0/1175665165 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d0a1547720 con 0x55d0a19184d0 2024-03-11T19:19:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.770+0000 7f7025ffb640 10 client.17769 handle_command_reply: tid=0 2024-03-11T19:19:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.770+0000 7f704716d740 1 client.17769 shutdown 2024-03-11T19:19:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.770+0000 7f704716d740 1 -- 172.21.15.29:0/1175665165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 msgr2=0x55d0a18bc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.770+0000 7f704716d740 1 --2- 172.21.15.29:0/1175665165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 0x55d0a18bc0d0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f7038006010 tx=0x7f7038004480 comp rx=0 tx=0).stop 2024-03-11T19:19:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.771+0000 7f704716d740 1 -- 172.21.15.29:0/1175665165 shutdown_connections 2024-03-11T19:19:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.771+0000 7f704716d740 1 --2- 172.21.15.29:0/1175665165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0a18b07f0 0x55d0a18bace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.771+0000 7f704716d740 1 --2- 172.21.15.29:0/1175665165 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x55d0a19184d0 0x55d0a191a8c0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.771+0000 7f704716d740 1 --2- 172.21.15.29:0/1175665165 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d0a18ab680 0x55d0a18ba7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.771+0000 7f7025ffb640 0 client.17769 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:19:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.771+0000 7f704716d740 1 --2- 172.21.15.29:0/1175665165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0a18aac90 0x55d0a18bc0d0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.771+0000 7f704716d740 1 -- 172.21.15.29:0/1175665165 >> 172.21.15.29:0/1175665165 conn(0x55d0a189d9f0 msgr2=0x55d0a18a0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.771+0000 7f704716d740 1 -- 172.21.15.29:0/1175665165 shutdown_connections 2024-03-11T19:19:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.771+0000 7f704716d740 1 -- 172.21.15.29:0/1175665165 wait complete. 2024-03-11T19:19:44.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.772+0000 7f704716d740 20 client.17769 trim_cache size 0 max 16384 2024-03-11T19:19:44.778 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:19:44.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.772+0000 7f704325e640 1 -- 172.21.15.29:0/3904865232 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f70280a1ab0 msgr2=0x7f70280a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.772+0000 7f704325e640 1 --2- 172.21.15.29:0/3904865232 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f70280a1ab0 0x7f70280a3f70 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7034005fd0 tx=0x7f7034009e90 comp rx=0 tx=0).stop 2024-03-11T19:19:44.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.772+0000 7f704325e640 1 -- 172.21.15.29:0/3904865232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f703c100ff0 msgr2=0x7f703c199740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.773+0000 7f704325e640 1 --2- 172.21.15.29:0/3904865232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f703c100ff0 0x7f703c199740 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f702c005fd0 tx=0x7f702c00ec20 comp rx=0 tx=0).stop 2024-03-11T19:19:44.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.773+0000 7f704325e640 1 -- 172.21.15.29:0/3904865232 shutdown_connections 2024-03-11T19:19:44.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.773+0000 7f704325e640 1 --2- 172.21.15.29:0/3904865232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f703c1019b0 0x7f703c1a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.773+0000 7f704325e640 1 --2- 172.21.15.29:0/3904865232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f703c100ff0 0x7f703c199740 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.773+0000 7f704325e640 1 --2- 172.21.15.29:0/3904865232 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f70280a1ab0 0x7f70280a3f70 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.773+0000 7f704325e640 1 --2- 172.21.15.29:0/3904865232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f703c0ffea0 0x7f703c199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.773+0000 7f704325e640 1 -- 172.21.15.29:0/3904865232 >> 172.21.15.29:0/3904865232 conn(0x7f703c0fb450 msgr2=0x7f703c0682e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:44.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.773+0000 7f704325e640 1 -- 172.21.15.29:0/3904865232 shutdown_connections 2024-03-11T19:19:44.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.773+0000 7f704325e640 1 -- 172.21.15.29:0/3904865232 wait complete. 2024-03-11T19:19:44.802 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:19:44.802 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:19:44.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:19:44.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:19:44.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:19:44.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.993+0000 7fd8aaffd640 1 -- 172.21.15.29:0/576180452 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a40329d0 con 0x7fd8ac10d760 2024-03-11T19:19:44.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.993+0000 7fd8b250d640 1 -- 172.21.15.29:0/576180452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8ac10d760 msgr2=0x7fd8ac10fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:44.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.993+0000 7fd8b250d640 1 --2- 172.21.15.29:0/576180452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8ac10d760 0x7fd8ac10fb50 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4009cd0 tx=0x7fd8a402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.994+0000 7fd8b250d640 1 -- 172.21.15.29:0/576180452 shutdown_connections 2024-03-11T19:19:44.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.994+0000 7fd8b250d640 1 --2- 172.21.15.29:0/576180452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8ac10d760 0x7fd8ac10fb50 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.994+0000 7fd8b250d640 1 --2- 172.21.15.29:0/576180452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8ac073be0 0x7fd8ac074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.994+0000 7fd8b250d640 1 --2- 172.21.15.29:0/576180452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8ac0751c0 0x7fd8ac073610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:44.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.994+0000 7fd8b250d640 1 -- 172.21.15.29:0/576180452 >> 172.21.15.29:0/576180452 conn(0x7fd8ac0fc6e0 msgr2=0x7fd8ac0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:44.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.995+0000 7fd8b250d640 1 -- 172.21.15.29:0/576180452 shutdown_connections 2024-03-11T19:19:44.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.995+0000 7fd8b250d640 1 -- 172.21.15.29:0/576180452 wait complete. 2024-03-11T19:19:45.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.996+0000 7fd8b250d640 1 Processor -- start 2024-03-11T19:19:45.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.996+0000 7fd8b250d640 1 -- start start 2024-03-11T19:19:45.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.996+0000 7fd8b250d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8ac073be0 0x7fd8ac06fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.996+0000 7fd8b250d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8ac0751c0 0x7fd8ac06ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.996+0000 7fd8b250d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8ac10d760 0x7fd8ac1ad8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.996+0000 7fd8b250d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac072ff0 con 0x7fd8ac073be0 2024-03-11T19:19:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.996+0000 7fd8b250d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac073160 con 0x7fd8ac10d760 2024-03-11T19:19:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.996+0000 7fd8b250d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac104f50 con 0x7fd8ac0751c0 2024-03-11T19:19:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.997+0000 7fd8b0aaa640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8ac10d760 0x7fd8ac1ad8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.997+0000 7fd8b0aaa640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8ac10d760 0x7fd8ac1ad8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59616/0 (socket says 172.21.15.29:59616) 2024-03-11T19:19:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.997+0000 7fd8b0aaa640 1 -- 172.21.15.29:0/1441877251 learned_addr learned my addr 172.21.15.29:0/1441877251 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.997+0000 7fd8ab7fe640 1 --2- 172.21.15.29:0/1441877251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8ac0751c0 0x7fd8ac06ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.997+0000 7fd8b0aaa640 1 -- 172.21.15.29:0/1441877251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8ac0751c0 msgr2=0x7fd8ac06ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.997+0000 7fd8b0aaa640 1 --2- 172.21.15.29:0/1441877251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8ac0751c0 0x7fd8ac06ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.997+0000 7fd8b0aaa640 1 -- 172.21.15.29:0/1441877251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8ac073be0 msgr2=0x7fd8ac06fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.997+0000 7fd8b0aaa640 1 --2- 172.21.15.29:0/1441877251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8ac073be0 0x7fd8ac06fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.997+0000 7fd8b0aaa640 1 -- 172.21.15.29:0/1441877251 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a4009900 con 0x7fd8ac10d760 2024-03-11T19:19:45.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.998+0000 7fd8b0aaa640 1 --2- 172.21.15.29:0/1441877251 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8ac10d760 0x7fd8ac1ad8f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4009bf0 tx=0x7fd8a4032900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:45.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.998+0000 7fd8a97fa640 1 -- 172.21.15.29:0/1441877251 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a4034c60 con 0x7fd8ac10d760 2024-03-11T19:19:45.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.998+0000 7fd8b250d640 1 -- 172.21.15.29:0/1441877251 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ac1051d0 con 0x7fd8ac10d760 2024-03-11T19:19:45.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.998+0000 7fd8b250d640 1 -- 172.21.15.29:0/1441877251 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8ac1056d0 con 0x7fd8ac10d760 2024-03-11T19:19:45.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.998+0000 7fd8a97fa640 1 -- 172.21.15.29:0/1441877251 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd8a4003e20 con 0x7fd8ac10d760 2024-03-11T19:19:45.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:44.998+0000 7fd8a97fa640 1 -- 172.21.15.29:0/1441877251 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a40048a0 con 0x7fd8ac10d760 2024-03-11T19:19:45.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.002+0000 7fd8b51bb740 1 Processor -- start 2024-03-11T19:19:45.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.002+0000 7fd8a97fa640 1 -- 172.21.15.29:0/1441877251 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd8a403b040 con 0x7fd8ac10d760 2024-03-11T19:19:45.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.002+0000 7fd8b51bb740 1 -- start start 2024-03-11T19:19:45.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.003+0000 7fd8b51bb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d47270 0x55d210d47690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:45.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.003+0000 7fd8b51bb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d47c60 0x55d210d4c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:45.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.003+0000 7fd8b51bb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d4cdd0 0x55d210d4f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:45.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.003+0000 7fd8b51bb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d210d03960 con 0x55d210d47270 2024-03-11T19:19:45.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.003+0000 7fd8b51bb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d210cb7310 con 0x55d210d4cdd0 2024-03-11T19:19:45.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.003+0000 7fd8b51bb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d210cb4840 con 0x55d210d47c60 2024-03-11T19:19:45.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.003+0000 7fd8ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d47270 0x55d210d47690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:45.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.003+0000 7fd8abfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d47c60 0x55d210d4c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:45.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.003+0000 7fd8abfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d47c60 0x55d210d4c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:32972/0 (socket says 172.21.15.29:32972) 2024-03-11T19:19:45.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.003+0000 7fd8abfff640 1 -- 172.21.15.29:0/2360642781 learned_addr learned my addr 172.21.15.29:0/2360642781 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:45.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.003+0000 7fd8a97fa640 1 --2- 172.21.15.29:0/1441877251 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8940a1b80 0x7fd8940a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:45.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.003+0000 7fd8abfff640 1 -- 172.21.15.29:0/2360642781 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d4cdd0 msgr2=0x55d210d4f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:45.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.003+0000 7fd8abfff640 1 --2- 172.21.15.29:0/2360642781 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d4cdd0 0x55d210d4f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.003+0000 7fd8abfff640 1 -- 172.21.15.29:0/2360642781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d47270 msgr2=0x55d210d47690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:45.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.003+0000 7fd8abfff640 1 --2- 172.21.15.29:0/2360642781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d47270 0x55d210d47690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.003+0000 7fd8abfff640 1 -- 172.21.15.29:0/2360642781 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d210c72060 con 0x55d210d47c60 2024-03-11T19:19:45.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.004+0000 7fd8a97fa640 1 -- 172.21.15.29:0/1441877251 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd8a40ef970 con 0x7fd8ac10d760 2024-03-11T19:19:45.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.004+0000 7fd8b0aaa640 1 --2- 172.21.15.29:0/1441877251 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8940a1b80 0x7fd8940a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:45.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.005+0000 7fd8b0aaa640 1 --2- 172.21.15.29:0/1441877251 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8940a1b80 0x7fd8940a4040 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4005e30 tx=0x7fd8a4032b80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:45.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.007+0000 7fd8abfff640 1 --2- 172.21.15.29:0/2360642781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d47c60 0x55d210d4c760 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd89c00bb10 tx=0x7fd89c00d2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1532004ab69ace13 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:45.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.007+0000 7fd8927fc640 1 -- 172.21.15.29:0/2360642781 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c00dce0 con 0x55d210d47c60 2024-03-11T19:19:45.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.008+0000 7fd8927fc640 1 -- 172.21.15.29:0/2360642781 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd89c00de40 con 0x55d210d47c60 2024-03-11T19:19:45.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.008+0000 7fd8927fc640 1 -- 172.21.15.29:0/2360642781 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c0106c0 con 0x55d210d47c60 2024-03-11T19:19:45.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.009+0000 7fd8b51bb740 1 -- 172.21.15.29:0/2360642781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d47c60 msgr2=0x55d210d4c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:45.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.009+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/2360642781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d47c60 0x55d210d4c760 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd89c00bb10 tx=0x7fd89c00d2e0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:19:45.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:19:45.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:19:45.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.010+0000 7fd8b51bb740 1 -- 172.21.15.29:0/2360642781 shutdown_connections 2024-03-11T19:19:45.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.010+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/2360642781 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d4cdd0 0x55d210d4f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.010+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/2360642781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d47c60 0x55d210d4c760 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.010+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/2360642781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d47270 0x55d210d47690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.010+0000 7fd8b51bb740 1 -- 172.21.15.29:0/2360642781 >> 172.21.15.29:0/2360642781 conn(0x55d210d427e0 msgr2=0x55d210d42be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:45.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.011+0000 7fd8b51bb740 1 -- 172.21.15.29:0/2360642781 shutdown_connections 2024-03-11T19:19:45.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.011+0000 7fd8b51bb740 1 -- 172.21.15.29:0/2360642781 wait complete. 2024-03-11T19:19:45.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.011+0000 7fd8b51bb740 1 -- start start 2024-03-11T19:19:45.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.012+0000 7fd8b51bb740 1 Processor -- start 2024-03-11T19:19:45.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.012+0000 7fd8b51bb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d47270 0x55d210d514e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:45.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.012+0000 7fd8b51bb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d47c60 0x55d210d51a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:45.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.012+0000 7fd8b51bb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d4cdd0 0x55d210d58a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:45.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.012+0000 7fd8b51bb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d210cb0020 con 0x55d210d47c60 2024-03-11T19:19:45.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.012+0000 7fd8b51bb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d210983430 con 0x55d210d47270 2024-03-11T19:19:45.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.012+0000 7fd8b51bb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d21093fe00 con 0x55d210d4cdd0 2024-03-11T19:19:45.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.012+0000 7fd8b51bb740 10 client.0 _ll_register_callbacks cb 0x55d210c57cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:45.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.012+0000 7fd8ab7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d47270 0x55d210d514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:45.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.012+0000 7fd8ab7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d47270 0x55d210d514e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59632/0 (socket says 172.21.15.29:59632) 2024-03-11T19:19:45.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.012+0000 7fd8ab7fe640 1 -- 172.21.15.29:0/3845723237 learned_addr learned my addr 172.21.15.29:0/3845723237 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:45.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.012+0000 7fd8abfff640 1 --2- 172.21.15.29:0/3845723237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d47c60 0x55d210d51a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:45.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.012+0000 7fd8ab7fe640 1 --2- 172.21.15.29:0/3845723237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d4cdd0 0x55d210d58a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:45.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.013+0000 7fd8ab7fe640 1 -- 172.21.15.29:0/3845723237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d4cdd0 msgr2=0x55d210d58a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:45.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.013+0000 7fd8ab7fe640 1 --2- 172.21.15.29:0/3845723237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d4cdd0 0x55d210d58a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.013+0000 7fd8ab7fe640 1 -- 172.21.15.29:0/3845723237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d47c60 msgr2=0x55d210d51a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:45.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.013+0000 7fd8ab7fe640 1 --2- 172.21.15.29:0/3845723237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d47c60 0x55d210d51a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.013+0000 7fd8ab7fe640 1 -- 172.21.15.29:0/3845723237 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd89c00b740 con 0x55d210d47270 2024-03-11T19:19:45.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.013+0000 7fd8ab7fe640 1 --2- 172.21.15.29:0/3845723237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d47270 0x55d210d514e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0013b30 tx=0x7fd8a00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23907186b1d94c28 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:45.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.013+0000 7fd88b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:19:45.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.013+0000 7fd88b7fe640 1 -- 172.21.15.29:0/3845723237 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0011ab0 con 0x55d210d47270 2024-03-11T19:19:45.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.013+0000 7fd88b7fe640 1 -- 172.21.15.29:0/3845723237 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd8a0011c10 con 0x55d210d47270 2024-03-11T19:19:45.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.014+0000 7fd8b51bb740 1 -- 172.21.15.29:0/3845723237 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d2109af7e0 con 0x55d210d47270 2024-03-11T19:19:45.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.014+0000 7fd88b7fe640 1 -- 172.21.15.29:0/3845723237 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8a0011dd0 con 0x55d210d47270 2024-03-11T19:19:45.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.014+0000 7fd8b51bb740 10 client.27272 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:45.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.014+0000 7fd8b51bb740 1 -- 172.21.15.29:0/3845723237 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d210cae6a0 con 0x55d210d47270 2024-03-11T19:19:45.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.014+0000 7fd88b7fe640 1 -- 172.21.15.29:0/3845723237 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55d2109af7e0 con 0x55d210d47270 2024-03-11T19:19:45.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.014+0000 7fd88b7fe640 1 -- 172.21.15.29:0/3845723237 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd8a002a400 con 0x55d210d47270 2024-03-11T19:19:45.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.015+0000 7fd8b51bb740 10 client.27272 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:45.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.015+0000 7fd8b51bb740 10 client.27272 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:19:45.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.015+0000 7fd8b51bb740 20 client.27272 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:45.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.015+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/3845723237 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d210d5d6a0 0x55d210d5fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:45.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.015+0000 7fd8b51bb740 4 client.27272 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:19:45.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.015+0000 7fd8b51bb740 1 -- 172.21.15.29:0/3845723237 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d2109ec720 con 0x55d210d5d6a0 2024-03-11T19:19:45.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.015+0000 7fd8abfff640 1 --2- 172.21.15.29:0/3845723237 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d210d5d6a0 0x55d210d5fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:45.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.016+0000 7fd8abfff640 1 --2- 172.21.15.29:0/3845723237 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d210d5d6a0 0x55d210d5fac0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebb998979cbf6639 server_cookie=8c49eb38e0be84ec in_seq=0 out_seq=0 2024-03-11T19:19:45.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.021+0000 7fd88b7fe640 10 client.27272 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:45.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.021+0000 7fd88b7fe640 1 -- 172.21.15.29:0/3845723237 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d2109ec720 con 0x55d210d5d6a0 2024-03-11T19:19:45.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.021+0000 7fd88b7fe640 10 client.27272 handle_command_reply: tid=0 2024-03-11T19:19:45.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.021+0000 7fd8b51bb740 1 client.27272 shutdown 2024-03-11T19:19:45.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.021+0000 7fd8b51bb740 1 -- 172.21.15.29:0/3845723237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d47270 msgr2=0x55d210d514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:45.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.021+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/3845723237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d47270 0x55d210d514e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0013b30 tx=0x7fd8a00046c0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.022+0000 7fd8b51bb740 1 -- 172.21.15.29:0/3845723237 shutdown_connections 2024-03-11T19:19:45.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.022+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/3845723237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d4cdd0 0x55d210d58a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.022+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/3845723237 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d210d5d6a0 0x55d210d5fac0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.022+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/3845723237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d47c60 0x55d210d51a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.022+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/3845723237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d47270 0x55d210d514e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.022+0000 7fd8b51bb740 1 -- 172.21.15.29:0/3845723237 >> 172.21.15.29:0/3845723237 conn(0x55d210d427e0 msgr2=0x55d210d41b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:45.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.022+0000 7fd88b7fe640 0 client.27272 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:45.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.022+0000 7fd8b51bb740 1 -- 172.21.15.29:0/3845723237 shutdown_connections 2024-03-11T19:19:45.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.022+0000 7fd8b51bb740 1 -- 172.21.15.29:0/3845723237 wait complete. 2024-03-11T19:19:45.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.022+0000 7fd8b51bb740 20 client.27272 trim_cache size 0 max 16384 2024-03-11T19:19:45.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.044+0000 7fd8b51bb740 1 Processor -- start 2024-03-11T19:19:45.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.044+0000 7fd8b51bb740 1 -- start start 2024-03-11T19:19:45.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.045+0000 7fd8b51bb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d47270 0x55d210d491b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:45.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.045+0000 7fd8b51bb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d47c60 0x55d210d496f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:45.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.045+0000 7fd8b51bb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d4cdd0 0x55d210d589f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:45.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.045+0000 7fd8b51bb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d210c35130 con 0x55d210d4cdd0 2024-03-11T19:19:45.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.045+0000 7fd8b51bb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2109ec720 con 0x55d210d47270 2024-03-11T19:19:45.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.045+0000 7fd8b51bb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2109af7e0 con 0x55d210d47c60 2024-03-11T19:19:45.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.045+0000 7fd8ab7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d47270 0x55d210d491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:45.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.045+0000 7fd8ab7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d47270 0x55d210d491b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59638/0 (socket says 172.21.15.29:59638) 2024-03-11T19:19:45.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.045+0000 7fd8ab7fe640 1 -- 172.21.15.29:0/2694345913 learned_addr learned my addr 172.21.15.29:0/2694345913 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:45.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.045+0000 7fd8ab7fe640 1 --2- 172.21.15.29:0/2694345913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d4cdd0 0x55d210d589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:45.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.045+0000 7fd8ab7fe640 1 -- 172.21.15.29:0/2694345913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d47c60 msgr2=0x55d210d496f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:45.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.045+0000 7fd8ab7fe640 1 --2- 172.21.15.29:0/2694345913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d47c60 0x55d210d496f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.045+0000 7fd8ab7fe640 1 -- 172.21.15.29:0/2694345913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d4cdd0 msgr2=0x55d210d589f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:45.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.045+0000 7fd8ab7fe640 1 --2- 172.21.15.29:0/2694345913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d4cdd0 0x55d210d589f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.045+0000 7fd8ab7fe640 1 -- 172.21.15.29:0/2694345913 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d210c4ac20 con 0x55d210d47270 2024-03-11T19:19:45.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.046+0000 7fd8ab7fe640 1 --2- 172.21.15.29:0/2694345913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d47270 0x55d210d491b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0013b00 tx=0x7fd8a0015040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c7c953bdc6332fe server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:45.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.046+0000 7fd891ffb640 1 -- 172.21.15.29:0/2694345913 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a000b390 con 0x55d210d47270 2024-03-11T19:19:45.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.046+0000 7fd891ffb640 1 -- 172.21.15.29:0/2694345913 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd8a000bec0 con 0x55d210d47270 2024-03-11T19:19:45.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.046+0000 7fd891ffb640 1 -- 172.21.15.29:0/2694345913 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a002abd0 con 0x55d210d47270 2024-03-11T19:19:45.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.047+0000 7fd8b51bb740 1 -- 172.21.15.29:0/2694345913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d47270 msgr2=0x55d210d491b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:45.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.047+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/2694345913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d47270 0x55d210d491b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0013b00 tx=0x7fd8a0015040 comp rx=0 tx=0).stop 2024-03-11T19:19:45.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.047+0000 7fd8b51bb740 1 -- 172.21.15.29:0/2694345913 shutdown_connections 2024-03-11T19:19:45.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.047+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/2694345913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d4cdd0 0x55d210d589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.047+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/2694345913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d47c60 0x55d210d496f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.047+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/2694345913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d47270 0x55d210d491b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.047+0000 7fd8b51bb740 1 -- 172.21.15.29:0/2694345913 >> 172.21.15.29:0/2694345913 conn(0x55d210d427e0 msgr2=0x55d210d64440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:45.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.048+0000 7fd8b51bb740 1 -- 172.21.15.29:0/2694345913 shutdown_connections 2024-03-11T19:19:45.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.048+0000 7fd8b51bb740 1 -- 172.21.15.29:0/2694345913 wait complete. 2024-03-11T19:19:45.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.048+0000 7fd8b51bb740 1 -- start start 2024-03-11T19:19:45.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.048+0000 7fd8b51bb740 1 Processor -- start 2024-03-11T19:19:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.049+0000 7fd8b51bb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d47270 0x55d210d58780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.049+0000 7fd8b51bb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d47c60 0x55d210d56d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.049+0000 7fd8b51bb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d4cdd0 0x55d210d572d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.049+0000 7fd8b51bb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d210d57810 con 0x55d210d47c60 2024-03-11T19:19:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.049+0000 7fd8b51bb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d210d57980 con 0x55d210d4cdd0 2024-03-11T19:19:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.049+0000 7fd8b51bb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d210d57b40 con 0x55d210d47270 2024-03-11T19:19:45.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.049+0000 7fd8ab7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d47270 0x55d210d58780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:45.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.049+0000 7fd8ab7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d47270 0x55d210d58780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:32994/0 (socket says 172.21.15.29:32994) 2024-03-11T19:19:45.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.049+0000 7fd8ab7fe640 1 -- 172.21.15.29:0/56673213 learned_addr learned my addr 172.21.15.29:0/56673213 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:45.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.049+0000 7fd8b51bb740 10 client.0 _ll_register_callbacks cb 0x7fd89c00b2a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:45.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.049+0000 7fd8ab7fe640 1 --2- 172.21.15.29:0/56673213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d4cdd0 0x55d210d572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:45.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.049+0000 7fd8abfff640 1 --2- 172.21.15.29:0/56673213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d47c60 0x55d210d56d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:45.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.049+0000 7fd8ab7fe640 1 -- 172.21.15.29:0/56673213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d4cdd0 msgr2=0x55d210d572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:45.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.049+0000 7fd8ab7fe640 1 --2- 172.21.15.29:0/56673213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d4cdd0 0x55d210d572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.049+0000 7fd8ab7fe640 1 -- 172.21.15.29:0/56673213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d47c60 msgr2=0x55d210d56d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:45.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.049+0000 7fd8ab7fe640 1 --2- 172.21.15.29:0/56673213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d47c60 0x55d210d56d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.049+0000 7fd8ab7fe640 1 -- 172.21.15.29:0/56673213 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d210cae6a0 con 0x55d210d47270 2024-03-11T19:19:45.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.049+0000 7fd8ab7fe640 1 --2- 172.21.15.29:0/56673213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d47270 0x55d210d58780 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0005940 tx=0x7fd8a002a9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=536bcfb84876d100 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:45.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.049+0000 7fd88b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:19:45.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.050+0000 7fd88b7fe640 1 -- 172.21.15.29:0/56673213 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0020630 con 0x55d210d47270 2024-03-11T19:19:45.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.050+0000 7fd88b7fe640 1 -- 172.21.15.29:0/56673213 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fd8a0000e40 con 0x55d210d47270 2024-03-11T19:19:45.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.050+0000 7fd8b51bb740 1 -- 172.21.15.29:0/56673213 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d210a3b890 con 0x55d210d47270 2024-03-11T19:19:45.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.050+0000 7fd88b7fe640 1 -- 172.21.15.29:0/56673213 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8a0020790 con 0x55d210d47270 2024-03-11T19:19:45.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.050+0000 7fd8b51bb740 10 client.27322 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:45.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.050+0000 7fd8b51bb740 1 -- 172.21.15.29:0/56673213 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d210c708c0 con 0x55d210d47270 2024-03-11T19:19:45.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.051+0000 7fd88b7fe640 1 -- 172.21.15.29:0/56673213 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55d210a3b890 con 0x55d210d47270 2024-03-11T19:19:45.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.051+0000 7fd88b7fe640 1 -- 172.21.15.29:0/56673213 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd8a00046c0 con 0x55d210d47270 2024-03-11T19:19:45.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.051+0000 7fd8b51bb740 10 client.27322 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:45.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.051+0000 7fd8b51bb740 10 client.27322 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:19:45.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.051+0000 7fd8b51bb740 20 client.27322 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:45.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.051+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/56673213 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d210db4e60 0x55d210db7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:45.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.051+0000 7fd8b51bb740 4 client.27322 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:19:45.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.051+0000 7fd8b51bb740 1 -- 172.21.15.29:0/56673213 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d21095b700 con 0x55d210db4e60 2024-03-11T19:19:45.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.051+0000 7fd8abfff640 1 --2- 172.21.15.29:0/56673213 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d210db4e60 0x55d210db7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:45.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.052+0000 7fd88b7fe640 10 client.27322 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:45.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.052+0000 7fd8abfff640 1 --2- 172.21.15.29:0/56673213 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d210db4e60 0x55d210db7250 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afa6fc25394e0fd server_cookie=56f54bd328b6856f in_seq=0 out_seq=0 2024-03-11T19:19:45.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.052+0000 7fd88b7fe640 1 -- 172.21.15.29:0/56673213 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fd89c014020 con 0x55d210db4e60 2024-03-11T19:19:45.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.052+0000 7fd88b7fe640 10 client.27322 handle_command_reply: tid=0 2024-03-11T19:19:45.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.053+0000 7fd8b51bb740 1 client.27322 shutdown 2024-03-11T19:19:45.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.053+0000 7fd8b51bb740 1 -- 172.21.15.29:0/56673213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d47270 msgr2=0x55d210d58780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:45.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.053+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/56673213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d47270 0x55d210d58780 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0005940 tx=0x7fd8a002a9d0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.053+0000 7fd8b51bb740 1 -- 172.21.15.29:0/56673213 shutdown_connections 2024-03-11T19:19:45.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.053+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/56673213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d210d4cdd0 0x55d210d572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.053+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/56673213 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55d210db4e60 0x55d210db7250 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.053+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/56673213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d210d47c60 0x55d210d56d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.053+0000 7fd8b51bb740 1 --2- 172.21.15.29:0/56673213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d210d47270 0x55d210d58780 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.053+0000 7fd8b51bb740 1 -- 172.21.15.29:0/56673213 >> 172.21.15.29:0/56673213 conn(0x55d210d427e0 msgr2=0x55d210d491b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:45.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.053+0000 7fd88b7fe640 0 client.27322 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:45.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.053+0000 7fd8b51bb740 1 -- 172.21.15.29:0/56673213 shutdown_connections 2024-03-11T19:19:45.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.053+0000 7fd8b51bb740 1 -- 172.21.15.29:0/56673213 wait complete. 2024-03-11T19:19:45.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.054+0000 7fd8b51bb740 20 client.27322 trim_cache size 0 max 16384 2024-03-11T19:19:45.061 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:19:45.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.055+0000 7fd8b250d640 1 -- 172.21.15.29:0/1441877251 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8940a1b80 msgr2=0x7fd8940a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:45.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.055+0000 7fd8b250d640 1 --2- 172.21.15.29:0/1441877251 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8940a1b80 0x7fd8940a4040 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4005e30 tx=0x7fd8a4032b80 comp rx=0 tx=0).stop 2024-03-11T19:19:45.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.055+0000 7fd8b250d640 1 -- 172.21.15.29:0/1441877251 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8ac10d760 msgr2=0x7fd8ac1ad8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:45.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.055+0000 7fd8b250d640 1 --2- 172.21.15.29:0/1441877251 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8ac10d760 0x7fd8ac1ad8f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4009bf0 tx=0x7fd8a4032900 comp rx=0 tx=0).stop 2024-03-11T19:19:45.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.055+0000 7fd8b250d640 1 -- 172.21.15.29:0/1441877251 shutdown_connections 2024-03-11T19:19:45.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.055+0000 7fd8b250d640 1 --2- 172.21.15.29:0/1441877251 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8ac10d760 0x7fd8ac1ad8f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.055+0000 7fd8b250d640 1 --2- 172.21.15.29:0/1441877251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8ac0751c0 0x7fd8ac06ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.055+0000 7fd8b250d640 1 --2- 172.21.15.29:0/1441877251 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fd8940a1b80 0x7fd8940a4040 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.055+0000 7fd8b250d640 1 --2- 172.21.15.29:0/1441877251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8ac073be0 0x7fd8ac06fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:45.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.055+0000 7fd8b250d640 1 -- 172.21.15.29:0/1441877251 >> 172.21.15.29:0/1441877251 conn(0x7fd8ac0fc6e0 msgr2=0x7fd8ac100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:45.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.056+0000 7fd8b250d640 1 -- 172.21.15.29:0/1441877251 shutdown_connections 2024-03-11T19:19:45.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:45.056+0000 7fd8b250d640 1 -- 172.21.15.29:0/1441877251 wait complete. 2024-03-11T19:19:45.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:44 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:19:45.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:44 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:19:45.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:44 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:19:46.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:45 smithi190 ceph-mon[36292]: pgmap v649: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 53 MiB/s wr, 1.10k op/s 2024-03-11T19:19:46.081 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:19:46.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:19:46.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:19:46.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:45 smithi170 ceph-mon[35690]: pgmap v649: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 53 MiB/s wr, 1.10k op/s 2024-03-11T19:19:46.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:45 smithi029 ceph-mon[32439]: pgmap v649: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 53 MiB/s wr, 1.10k op/s 2024-03-11T19:19:46.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.258+0000 7f25a56c9640 1 -- 172.21.15.29:0/3573869684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25a01077c0 msgr2=0x7f25a0107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:46.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.258+0000 7f25a56c9640 1 --2- 172.21.15.29:0/3573869684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25a01077c0 0x7f25a0107be0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2590009860 tx=0x7f259002bb70 comp rx=0 tx=0).stop 2024-03-11T19:19:46.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.258+0000 7f25a56c9640 1 -- 172.21.15.29:0/3573869684 shutdown_connections 2024-03-11T19:19:46.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.258+0000 7f25a56c9640 1 --2- 172.21.15.29:0/3573869684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25a0109370 0x7f25a010fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.258+0000 7f25a56c9640 1 --2- 172.21.15.29:0/3573869684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25a01089b0 0x7f25a0108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.258+0000 7f25a56c9640 1 --2- 172.21.15.29:0/3573869684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25a01077c0 0x7f25a0107be0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.258+0000 7f25a56c9640 1 -- 172.21.15.29:0/3573869684 >> 172.21.15.29:0/3573869684 conn(0x7f25a0076750 msgr2=0x7f25a0078b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:46.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.258+0000 7f25a56c9640 1 -- 172.21.15.29:0/3573869684 shutdown_connections 2024-03-11T19:19:46.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.258+0000 7f25a56c9640 1 -- 172.21.15.29:0/3573869684 wait complete. 2024-03-11T19:19:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.259+0000 7f25a56c9640 1 Processor -- start 2024-03-11T19:19:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.260+0000 7f25a56c9640 1 -- start start 2024-03-11T19:19:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.260+0000 7f25a56c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25a01077c0 0x7f25a019d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.260+0000 7f25a56c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25a01089b0 0x7f25a019db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.260+0000 7f25a56c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25a0109370 0x7f25a01a4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.260+0000 7f25a56c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a01a50f0 con 0x7f25a0109370 2024-03-11T19:19:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.260+0000 7f25a56c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a01a5260 con 0x7f25a01089b0 2024-03-11T19:19:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.260+0000 7f25a56c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a01a5420 con 0x7f25a01077c0 2024-03-11T19:19:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.260+0000 7f259effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25a01077c0 0x7f25a019d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.260+0000 7f259effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25a01077c0 0x7f25a019d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:56400/0 (socket says 172.21.15.29:56400) 2024-03-11T19:19:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.260+0000 7f259effd640 1 -- 172.21.15.29:0/2232224001 learned_addr learned my addr 172.21.15.29:0/2232224001 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:46.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.261+0000 7f259f7fe640 1 --2- 172.21.15.29:0/2232224001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25a0109370 0x7f25a01a4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.261+0000 7f259e7fc640 1 --2- 172.21.15.29:0/2232224001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25a01089b0 0x7f25a019db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.261+0000 7f259effd640 1 -- 172.21.15.29:0/2232224001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25a01089b0 msgr2=0x7f25a019db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:46.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.261+0000 7f259effd640 1 --2- 172.21.15.29:0/2232224001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25a01089b0 0x7f25a019db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.261+0000 7f259effd640 1 -- 172.21.15.29:0/2232224001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25a0109370 msgr2=0x7f25a01a4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:46.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.261+0000 7f259effd640 1 --2- 172.21.15.29:0/2232224001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25a0109370 0x7f25a01a4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.261+0000 7f259effd640 1 -- 172.21.15.29:0/2232224001 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2590009490 con 0x7f25a01077c0 2024-03-11T19:19:46.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.261+0000 7f259e7fc640 1 --2- 172.21.15.29:0/2232224001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25a01089b0 0x7f25a019db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:19:46.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.261+0000 7f259effd640 1 --2- 172.21.15.29:0/2232224001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25a01077c0 0x7f25a019d600 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f259002c180 tx=0x7f2590004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:46.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.262+0000 7f2587fff640 1 -- 172.21.15.29:0/2232224001 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259002c710 con 0x7f25a01077c0 2024-03-11T19:19:46.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.262+0000 7f2587fff640 1 -- 172.21.15.29:0/2232224001 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f259002ce90 con 0x7f25a01077c0 2024-03-11T19:19:46.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.262+0000 7f2587fff640 1 -- 172.21.15.29:0/2232224001 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25900035f0 con 0x7f25a01077c0 2024-03-11T19:19:46.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.262+0000 7f25a56c9640 1 -- 172.21.15.29:0/2232224001 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a01a56a0 con 0x7f25a01077c0 2024-03-11T19:19:46.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.262+0000 7f25a56c9640 1 -- 172.21.15.29:0/2232224001 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a01a5b00 con 0x7f25a01077c0 2024-03-11T19:19:46.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.265+0000 7f25a837b740 1 Processor -- start 2024-03-11T19:19:46.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.265+0000 7f25a837b740 1 -- start start 2024-03-11T19:19:46.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.265+0000 7f25a837b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b632194c0 0x557b632198e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.265+0000 7f25a837b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b6321a690 0x557b6321ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.265+0000 7f25a837b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321b050 0x557b632218e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.265+0000 7f25a837b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b63188f90 con 0x557b6321a690 2024-03-11T19:19:46.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.265+0000 7f25a837b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b631864c0 con 0x557b6321b050 2024-03-11T19:19:46.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.265+0000 7f25a837b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b63181ca0 con 0x557b632194c0 2024-03-11T19:19:46.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.266+0000 7f2587fff640 1 -- 172.21.15.29:0/2232224001 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f259003b040 con 0x7f25a01077c0 2024-03-11T19:19:46.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.266+0000 7f259f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b632194c0 0x557b632198e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.266+0000 7f259f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b632194c0 0x557b632198e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:56414/0 (socket says 172.21.15.29:56414) 2024-03-11T19:19:46.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.266+0000 7f259f7fe640 1 -- 172.21.15.29:0/2220249029 learned_addr learned my addr 172.21.15.29:0/2220249029 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:46.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.266+0000 7f259e7fc640 1 --2- 172.21.15.29:0/2220249029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321b050 0x557b632218e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.266+0000 7f259effd640 1 --2- 172.21.15.29:0/2220249029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b6321a690 0x557b6321ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.266+0000 7f259f7fe640 1 -- 172.21.15.29:0/2220249029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321b050 msgr2=0x557b632218e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:46.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.266+0000 7f259f7fe640 1 --2- 172.21.15.29:0/2220249029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321b050 0x557b632218e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.266+0000 7f259f7fe640 1 -- 172.21.15.29:0/2220249029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b6321a690 msgr2=0x557b6321ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:46.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.266+0000 7f259f7fe640 1 --2- 172.21.15.29:0/2220249029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b6321a690 0x557b6321ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.266+0000 7f259f7fe640 1 -- 172.21.15.29:0/2220249029 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b631440b0 con 0x557b632194c0 2024-03-11T19:19:46.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.266+0000 7f2587fff640 1 --2- 172.21.15.29:0/2232224001 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f25880a1b00 0x7f25880a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.267+0000 7f259f7fe640 1 --2- 172.21.15.29:0/2220249029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b632194c0 0x557b632198e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f259800e580 tx=0x7f25980095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f08fedd4b05245d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:46.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.267+0000 7f2587fff640 1 -- 172.21.15.29:0/2232224001 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f25900ef080 con 0x7f25a01077c0 2024-03-11T19:19:46.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.267+0000 7f259effd640 1 --2- 172.21.15.29:0/2232224001 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f25880a1b00 0x7f25880a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.267+0000 7f25857fa640 1 -- 172.21.15.29:0/2220249029 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259801b070 con 0x557b632194c0 2024-03-11T19:19:46.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.267+0000 7f25857fa640 1 -- 172.21.15.29:0/2220249029 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2598003e20 con 0x557b632194c0 2024-03-11T19:19:46.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.267+0000 7f25857fa640 1 -- 172.21.15.29:0/2220249029 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2598013040 con 0x557b632194c0 2024-03-11T19:19:46.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.268+0000 7f25a837b740 1 -- 172.21.15.29:0/2220249029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b632194c0 msgr2=0x557b632198e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:46.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.268+0000 7f259effd640 1 --2- 172.21.15.29:0/2232224001 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f25880a1b00 0x7f25880a3fc0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2590032530 tx=0x7f2590000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:46.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.268+0000 7f25a837b740 1 --2- 172.21.15.29:0/2220249029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b632194c0 0x557b632198e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f259800e580 tx=0x7f25980095c0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.268+0000 7f25a837b740 1 -- 172.21.15.29:0/2220249029 shutdown_connections 2024-03-11T19:19:46.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.268+0000 7f25a837b740 1 --2- 172.21.15.29:0/2220249029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321b050 0x557b632218e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.268+0000 7f25a837b740 1 --2- 172.21.15.29:0/2220249029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b6321a690 0x557b6321ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.268+0000 7f25a837b740 1 --2- 172.21.15.29:0/2220249029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b632194c0 0x557b632198e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.268+0000 7f25a837b740 1 -- 172.21.15.29:0/2220249029 >> 172.21.15.29:0/2220249029 conn(0x557b632146a0 msgr2=0x557b63214aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:46.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.268+0000 7f25a837b740 1 -- 172.21.15.29:0/2220249029 shutdown_connections 2024-03-11T19:19:46.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.268+0000 7f25a837b740 1 -- 172.21.15.29:0/2220249029 wait complete. 2024-03-11T19:19:46.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.268+0000 7f25a837b740 1 -- start start 2024-03-11T19:19:46.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.269+0000 7f25a837b740 1 Processor -- start 2024-03-11T19:19:46.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.269+0000 7f25a837b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b632194c0 0x557b63223710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.269+0000 7f25a837b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321a690 0x557b63223c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.269+0000 7f25a837b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b6321b050 0x557b6322ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.269+0000 7f25a837b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b63126f40 con 0x557b6321b050 2024-03-11T19:19:46.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.269+0000 7f25a837b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b62e554b0 con 0x557b6321a690 2024-03-11T19:19:46.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.269+0000 7f25a837b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b63107670 con 0x557b632194c0 2024-03-11T19:19:46.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.269+0000 7f25a837b740 10 client.0 _ll_register_callbacks cb 0x557b63174f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:46.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.269+0000 7f259e7fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321a690 0x557b63223c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.269+0000 7f259e7fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321a690 0x557b63223c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:58656/0 (socket says 172.21.15.29:58656) 2024-03-11T19:19:46.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.269+0000 7f259e7fc640 1 -- 172.21.15.29:0/1286549147 learned_addr learned my addr 172.21.15.29:0/1286549147 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:46.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.269+0000 7f259f7fe640 1 --2- 172.21.15.29:0/1286549147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b6321b050 0x557b6322ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.269+0000 7f259f7fe640 1 --2- 172.21.15.29:0/1286549147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b632194c0 0x557b63223710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.270+0000 7f259e7fc640 1 -- 172.21.15.29:0/1286549147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b632194c0 msgr2=0x557b63223710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:46.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.270+0000 7f259e7fc640 1 --2- 172.21.15.29:0/1286549147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b632194c0 0x557b63223710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.270+0000 7f259e7fc640 1 -- 172.21.15.29:0/1286549147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b6321b050 msgr2=0x557b6322ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:46.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.270+0000 7f259e7fc640 1 --2- 172.21.15.29:0/1286549147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b6321b050 0x557b6322ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.270+0000 7f259e7fc640 1 -- 172.21.15.29:0/1286549147 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2598009d60 con 0x557b6321a690 2024-03-11T19:19:46.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.270+0000 7f259e7fc640 1 --2- 172.21.15.29:0/1286549147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321a690 0x557b63223c50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2594009fa0 tx=0x7f2594011870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f3996f9dee8ea21 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:46.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.270+0000 7f2582ff5640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:19:46.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.270+0000 7f2582ff5640 1 -- 172.21.15.29:0/1286549147 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2594012920 con 0x557b6321a690 2024-03-11T19:19:46.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.270+0000 7f25a837b740 1 -- 172.21.15.29:0/1286549147 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557b6315bf40 con 0x557b6321a690 2024-03-11T19:19:46.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.270+0000 7f2582ff5640 1 -- 172.21.15.29:0/1286549147 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f2594012a80 con 0x557b6321a690 2024-03-11T19:19:46.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.270+0000 7f2582ff5640 1 -- 172.21.15.29:0/1286549147 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2594012c40 con 0x557b6321a690 2024-03-11T19:19:46.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.270+0000 7f25a837b740 10 client.27287 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:46.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.270+0000 7f25a837b740 1 -- 172.21.15.29:0/1286549147 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x557b63180320 con 0x557b6321a690 2024-03-11T19:19:46.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.271+0000 7f2582ff5640 1 -- 172.21.15.29:0/1286549147 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x557b6315bf40 con 0x557b6321a690 2024-03-11T19:19:46.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.271+0000 7f2582ff5640 1 -- 172.21.15.29:0/1286549147 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f259400c070 con 0x557b6321a690 2024-03-11T19:19:46.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.271+0000 7f25a837b740 10 client.27287 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:46.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.271+0000 7f25a837b740 10 client.27287 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:19:46.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.271+0000 7f25a837b740 20 client.27287 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:46.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.271+0000 7f25a837b740 1 --2- 172.21.15.29:0/1286549147 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x557b6322f990 0x557b63231db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.271+0000 7f25a837b740 4 client.27287 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:19:46.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.271+0000 7f25a837b740 1 -- 172.21.15.29:0/1286549147 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557b63178310 con 0x557b6322f990 2024-03-11T19:19:46.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.272+0000 7f259f7fe640 1 --2- 172.21.15.29:0/1286549147 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x557b6322f990 0x557b63231db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.272+0000 7f259f7fe640 1 --2- 172.21.15.29:0/1286549147 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x557b6322f990 0x557b63231db0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1ac39fcbd78a1b2 server_cookie=e9f7b7bf64a5ddfc in_seq=0 out_seq=0 2024-03-11T19:19:46.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.272+0000 7f2582ff5640 10 client.27287 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:46.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.273+0000 7f2582ff5640 1 -- 172.21.15.29:0/1286549147 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557b63178310 con 0x557b6322f990 2024-03-11T19:19:46.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.273+0000 7f2582ff5640 10 client.27287 handle_command_reply: tid=0 2024-03-11T19:19:46.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.273+0000 7f25a837b740 1 client.27287 shutdown 2024-03-11T19:19:46.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.273+0000 7f25a837b740 1 -- 172.21.15.29:0/1286549147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321a690 msgr2=0x557b63223c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:46.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.273+0000 7f25a837b740 1 --2- 172.21.15.29:0/1286549147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321a690 0x557b63223c50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2594009fa0 tx=0x7f2594011870 comp rx=0 tx=0).stop 2024-03-11T19:19:46.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.273+0000 7f25a837b740 1 -- 172.21.15.29:0/1286549147 shutdown_connections 2024-03-11T19:19:46.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.273+0000 7f25a837b740 1 --2- 172.21.15.29:0/1286549147 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x557b6322f990 0x557b63231db0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.273+0000 7f25a837b740 1 --2- 172.21.15.29:0/1286549147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b6321b050 0x557b6322ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.273+0000 7f25a837b740 1 --2- 172.21.15.29:0/1286549147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321a690 0x557b63223c50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.273+0000 7f25a837b740 1 --2- 172.21.15.29:0/1286549147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b632194c0 0x557b63223710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.273+0000 7f25a837b740 1 -- 172.21.15.29:0/1286549147 >> 172.21.15.29:0/1286549147 conn(0x557b632146a0 msgr2=0x557b632138e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:46.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.273+0000 7f2582ff5640 0 client.27287 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:46.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.273+0000 7f25a837b740 1 -- 172.21.15.29:0/1286549147 shutdown_connections 2024-03-11T19:19:46.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.274+0000 7f25a837b740 1 -- 172.21.15.29:0/1286549147 wait complete. 2024-03-11T19:19:46.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.274+0000 7f25a837b740 20 client.27287 trim_cache size 0 max 16384 2024-03-11T19:19:46.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.298+0000 7f25a837b740 1 Processor -- start 2024-03-11T19:19:46.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.300+0000 7f25a837b740 1 -- start start 2024-03-11T19:19:46.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.300+0000 7f25a837b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b632194c0 0x557b63223710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.300+0000 7f25a837b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321a690 0x557b63223c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.301+0000 7f25a837b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b6321b050 0x557b6322ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.301+0000 7f25a837b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b63214ee0 con 0x557b632194c0 2024-03-11T19:19:46.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.301+0000 7f25a837b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b63209350 con 0x557b6321a690 2024-03-11T19:19:46.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.301+0000 7f25a837b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b6315bf40 con 0x557b6321b050 2024-03-11T19:19:46.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.301+0000 7f259f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b6321b050 0x557b6322ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.301+0000 7f259f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b6321b050 0x557b6322ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:56424/0 (socket says 172.21.15.29:56424) 2024-03-11T19:19:46.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.301+0000 7f259f7fe640 1 -- 172.21.15.29:0/3064505599 learned_addr learned my addr 172.21.15.29:0/3064505599 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:46.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.301+0000 7f259f7fe640 1 --2- 172.21.15.29:0/3064505599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b632194c0 0x557b63223710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.301+0000 7f259f7fe640 1 -- 172.21.15.29:0/3064505599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321a690 msgr2=0x557b63223c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:19:46.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.301+0000 7f259f7fe640 1 --2- 172.21.15.29:0/3064505599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321a690 0x557b63223c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.301+0000 7f259f7fe640 1 -- 172.21.15.29:0/3064505599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b632194c0 msgr2=0x557b63223710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:46.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.301+0000 7f259f7fe640 1 --2- 172.21.15.29:0/3064505599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b632194c0 0x557b63223710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.301+0000 7f259f7fe640 1 -- 172.21.15.29:0/3064505599 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b6311cc00 con 0x557b6321b050 2024-03-11T19:19:46.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.302+0000 7f259f7fe640 1 --2- 172.21.15.29:0/3064505599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b6321b050 0x557b6322ac50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f259800eed0 tx=0x7f2598010040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9e7ecaceb2ac37c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:46.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.302+0000 7f2584ff9640 1 -- 172.21.15.29:0/3064505599 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259802a040 con 0x557b6321b050 2024-03-11T19:19:46.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.302+0000 7f2584ff9640 1 -- 172.21.15.29:0/3064505599 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f259802b040 con 0x557b6321b050 2024-03-11T19:19:46.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.302+0000 7f2584ff9640 1 -- 172.21.15.29:0/3064505599 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2598013040 con 0x557b6321b050 2024-03-11T19:19:46.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.303+0000 7f25a837b740 1 -- 172.21.15.29:0/3064505599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b6321b050 msgr2=0x557b6322ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:46.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.303+0000 7f25a837b740 1 --2- 172.21.15.29:0/3064505599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b6321b050 0x557b6322ac50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f259800eed0 tx=0x7f2598010040 comp rx=0 tx=0).stop 2024-03-11T19:19:46.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.303+0000 7f25a837b740 1 -- 172.21.15.29:0/3064505599 shutdown_connections 2024-03-11T19:19:46.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.303+0000 7f25a837b740 1 --2- 172.21.15.29:0/3064505599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b6321b050 0x557b6322ac50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.303+0000 7f25a837b740 1 --2- 172.21.15.29:0/3064505599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321a690 0x557b63223c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.304+0000 7f25a837b740 1 --2- 172.21.15.29:0/3064505599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b632194c0 0x557b63223710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.304+0000 7f25a837b740 1 -- 172.21.15.29:0/3064505599 >> 172.21.15.29:0/3064505599 conn(0x557b632146a0 msgr2=0x557b63236710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:46.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.305+0000 7f25a837b740 1 -- 172.21.15.29:0/3064505599 shutdown_connections 2024-03-11T19:19:46.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.305+0000 7f25a837b740 1 -- 172.21.15.29:0/3064505599 wait complete. 2024-03-11T19:19:46.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.305+0000 7f25a837b740 1 -- start start 2024-03-11T19:19:46.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.305+0000 7f25a837b740 1 Processor -- start 2024-03-11T19:19:46.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.306+0000 7f25a837b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b632194c0 0x557b632296a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.306+0000 7f25a837b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321a690 0x557b63229be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.306+0000 7f25a837b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b6321b050 0x557b6322a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.306+0000 7f259f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b632194c0 0x557b632296a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.306+0000 7f259f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b632194c0 0x557b632296a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:56440/0 (socket says 172.21.15.29:56440) 2024-03-11T19:19:46.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.306+0000 7f259e7fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321a690 0x557b63229be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.307+0000 7f259f7fe640 1 -- 172.21.15.29:0/4272778745 learned_addr learned my addr 172.21.15.29:0/4272778745 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:46.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.307+0000 7f25a837b740 1 -- 172.21.15.29:0/4272778745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b63285f50 con 0x557b6321b050 2024-03-11T19:19:46.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.307+0000 7f25a837b740 1 -- 172.21.15.29:0/4272778745 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b632860c0 con 0x557b6321a690 2024-03-11T19:19:46.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.307+0000 7f25a837b740 1 -- 172.21.15.29:0/4272778745 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b63286280 con 0x557b632194c0 2024-03-11T19:19:46.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.307+0000 7f259f7fe640 1 --2- 172.21.15.29:0/4272778745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b6321b050 0x557b6322a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.307+0000 7f25a837b740 10 client.0 _ll_register_callbacks cb 0x7f2598009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:46.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.307+0000 7f259f7fe640 1 -- 172.21.15.29:0/4272778745 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321a690 msgr2=0x557b63229be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:46.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.307+0000 7f259f7fe640 1 --2- 172.21.15.29:0/4272778745 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321a690 0x557b63229be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.307+0000 7f259f7fe640 1 -- 172.21.15.29:0/4272778745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b6321b050 msgr2=0x557b6322a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:46.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.307+0000 7f259f7fe640 1 --2- 172.21.15.29:0/4272778745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b6321b050 0x557b6322a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.307+0000 7f259f7fe640 1 -- 172.21.15.29:0/4272778745 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557b631440b0 con 0x557b632194c0 2024-03-11T19:19:46.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.308+0000 7f259f7fe640 1 --2- 172.21.15.29:0/4272778745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b632194c0 0x557b632296a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f259802ee10 tx=0x7f2598000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77ec6c5e25cd7213 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:46.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.308+0000 7f2582ff5640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:19:46.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.308+0000 7f2582ff5640 1 -- 172.21.15.29:0/4272778745 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259802a070 con 0x557b632194c0 2024-03-11T19:19:46.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.308+0000 7f25a837b740 1 -- 172.21.15.29:0/4272778745 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557b631a7900 con 0x557b632194c0 2024-03-11T19:19:46.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.308+0000 7f2582ff5640 1 -- 172.21.15.29:0/4272778745 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f2598003970 con 0x557b632194c0 2024-03-11T19:19:46.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.308+0000 7f2582ff5640 1 -- 172.21.15.29:0/4272778745 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2598013040 con 0x557b632194c0 2024-03-11T19:19:46.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.308+0000 7f25a837b740 10 client.27340 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:46.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.308+0000 7f25a837b740 1 -- 172.21.15.29:0/4272778745 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x557b63142910 con 0x557b632194c0 2024-03-11T19:19:46.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.309+0000 7f2582ff5640 1 -- 172.21.15.29:0/4272778745 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x557b631a7900 con 0x557b632194c0 2024-03-11T19:19:46.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.309+0000 7f2582ff5640 1 -- 172.21.15.29:0/4272778745 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f25980046e0 con 0x557b632194c0 2024-03-11T19:19:46.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.309+0000 7f25a837b740 10 client.27340 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:46.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.309+0000 7f25a837b740 10 client.27340 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:19:46.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.309+0000 7f25a837b740 20 client.27340 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:46.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.309+0000 7f25a837b740 1 --2- 172.21.15.29:0/4272778745 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x557b63287a80 0x557b63289ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.309+0000 7f25a837b740 4 client.27340 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:19:46.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.309+0000 7f25a837b740 1 -- 172.21.15.29:0/4272778745 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x557b62e81860 con 0x557b63287a80 2024-03-11T19:19:46.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.310+0000 7f259e7fc640 1 --2- 172.21.15.29:0/4272778745 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x557b63287a80 0x557b63289ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.310+0000 7f259e7fc640 1 --2- 172.21.15.29:0/4272778745 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x557b63287a80 0x557b63289ea0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dad163fd66b510b8 server_cookie=f10fa7e531748099 in_seq=0 out_seq=0 2024-03-11T19:19:46.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.310+0000 7f2582ff5640 10 client.27340 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:46.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.311+0000 7f2582ff5640 1 -- 172.21.15.29:0/4272778745 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x557b62e81860 con 0x557b63287a80 2024-03-11T19:19:46.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.311+0000 7f2582ff5640 10 client.27340 handle_command_reply: tid=0 2024-03-11T19:19:46.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.311+0000 7f25a837b740 1 client.27340 shutdown 2024-03-11T19:19:46.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.312+0000 7f25a837b740 1 -- 172.21.15.29:0/4272778745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b632194c0 msgr2=0x557b632296a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:46.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.312+0000 7f25a837b740 1 --2- 172.21.15.29:0/4272778745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b632194c0 0x557b632296a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f259802ee10 tx=0x7f2598000f40 comp rx=0 tx=0).stop 2024-03-11T19:19:46.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.312+0000 7f25a837b740 1 -- 172.21.15.29:0/4272778745 shutdown_connections 2024-03-11T19:19:46.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.312+0000 7f25a837b740 1 --2- 172.21.15.29:0/4272778745 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x557b63287a80 0x557b63289ea0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.312+0000 7f25a837b740 1 --2- 172.21.15.29:0/4272778745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557b6321b050 0x557b6322a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.312+0000 7f25a837b740 1 --2- 172.21.15.29:0/4272778745 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557b6321a690 0x557b63229be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.312+0000 7f25a837b740 1 --2- 172.21.15.29:0/4272778745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557b632194c0 0x557b632296a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.312+0000 7f25a837b740 1 -- 172.21.15.29:0/4272778745 >> 172.21.15.29:0/4272778745 conn(0x557b632146a0 msgr2=0x557b63217900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:46.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.312+0000 7f2582ff5640 0 client.27340 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:46.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.312+0000 7f25a837b740 1 -- 172.21.15.29:0/4272778745 shutdown_connections 2024-03-11T19:19:46.317 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:19:46.317 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:19:46.317 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "f1a6978e-efe3-4b63-9ff6-8c7319efd68e", 2024-03-11T19:19:46.317 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:19:46.317 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:19:46.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.312+0000 7f25a837b740 1 -- 172.21.15.29:0/4272778745 wait complete. 2024-03-11T19:19:46.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.312+0000 7f25a837b740 20 client.27340 trim_cache size 0 max 16384 2024-03-11T19:19:46.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.313+0000 7f25a56c9640 1 -- 172.21.15.29:0/2232224001 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f25880a1b00 msgr2=0x7f25880a3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:46.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.313+0000 7f25a56c9640 1 --2- 172.21.15.29:0/2232224001 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f25880a1b00 0x7f25880a3fc0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2590032530 tx=0x7f2590000bc0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.313+0000 7f25a56c9640 1 -- 172.21.15.29:0/2232224001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25a01077c0 msgr2=0x7f25a019d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:46.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.313+0000 7f25a56c9640 1 --2- 172.21.15.29:0/2232224001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25a01077c0 0x7f25a019d600 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f259002c180 tx=0x7f2590004270 comp rx=0 tx=0).stop 2024-03-11T19:19:46.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.313+0000 7f25a56c9640 1 -- 172.21.15.29:0/2232224001 shutdown_connections 2024-03-11T19:19:46.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.313+0000 7f25a56c9640 1 --2- 172.21.15.29:0/2232224001 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f25880a1b00 0x7f25880a3fc0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.313+0000 7f25a56c9640 1 --2- 172.21.15.29:0/2232224001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25a0109370 0x7f25a01a4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.313+0000 7f25a56c9640 1 --2- 172.21.15.29:0/2232224001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25a01089b0 0x7f25a019db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.313+0000 7f25a56c9640 1 --2- 172.21.15.29:0/2232224001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25a01077c0 0x7f25a019d600 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.313+0000 7f25a56c9640 1 -- 172.21.15.29:0/2232224001 >> 172.21.15.29:0/2232224001 conn(0x7f25a0076750 msgr2=0x7f25a010bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:46.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.313+0000 7f25a56c9640 1 -- 172.21.15.29:0/2232224001 shutdown_connections 2024-03-11T19:19:46.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.314+0000 7f25a56c9640 1 -- 172.21.15.29:0/2232224001 wait complete. 2024-03-11T19:19:46.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:19:46.337 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:19:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.573+0000 7f4c5a849640 1 -- 172.21.15.29:0/2158979799 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c54109180 msgr2=0x7f4c5410da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.573+0000 7f4c5a849640 1 --2- 172.21.15.29:0/2158979799 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c54109180 0x7f4c5410da20 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4c44009900 tx=0x7f4c4402bc60 comp rx=0 tx=0).stop 2024-03-11T19:19:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.573+0000 7f4c5a849640 1 -- 172.21.15.29:0/2158979799 shutdown_connections 2024-03-11T19:19:46.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.573+0000 7f4c5a849640 1 --2- 172.21.15.29:0/2158979799 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c54109180 0x7f4c5410da20 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.573+0000 7f4c5a849640 1 --2- 172.21.15.29:0/2158979799 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c541087c0 0x7f4c54108c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.573+0000 7f4c5a849640 1 --2- 172.21.15.29:0/2158979799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c540724f0 0x7f4c54072910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.573+0000 7f4c5a849640 1 -- 172.21.15.29:0/2158979799 >> 172.21.15.29:0/2158979799 conn(0x7f4c5406dfa0 msgr2=0x7f4c540703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:46.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.574+0000 7f4c5a849640 1 -- 172.21.15.29:0/2158979799 shutdown_connections 2024-03-11T19:19:46.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.574+0000 7f4c5a849640 1 -- 172.21.15.29:0/2158979799 wait complete. 2024-03-11T19:19:46.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.575+0000 7f4c5a849640 1 Processor -- start 2024-03-11T19:19:46.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.575+0000 7f4c5a849640 1 -- start start 2024-03-11T19:19:46.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.575+0000 7f4c5a849640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c540724f0 0x7f4c541a5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.576+0000 7f4c5a849640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c541087c0 0x7f4c541a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.576+0000 7f4c5a849640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c54109180 0x7f4c541ad450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.576+0000 7f4c5a849640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c541ad990 con 0x7f4c540724f0 2024-03-11T19:19:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.576+0000 7f4c5a849640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c541adb00 con 0x7f4c541087c0 2024-03-11T19:19:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.576+0000 7f4c5a849640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c541adcc0 con 0x7f4c54109180 2024-03-11T19:19:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.576+0000 7f4c537fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c541087c0 0x7f4c541a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.576+0000 7f4c537fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c541087c0 0x7f4c541a63e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:58686/0 (socket says 172.21.15.29:58686) 2024-03-11T19:19:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.576+0000 7f4c537fe640 1 -- 172.21.15.29:0/2367238105 learned_addr learned my addr 172.21.15.29:0/2367238105 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.576+0000 7f4c53fff640 1 --2- 172.21.15.29:0/2367238105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c540724f0 0x7f4c541a5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.576+0000 7f4c58de6640 1 --2- 172.21.15.29:0/2367238105 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c54109180 0x7f4c541ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.576+0000 7f4c537fe640 1 -- 172.21.15.29:0/2367238105 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c54109180 msgr2=0x7f4c541ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.576+0000 7f4c537fe640 1 --2- 172.21.15.29:0/2367238105 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c54109180 0x7f4c541ad450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.576+0000 7f4c537fe640 1 -- 172.21.15.29:0/2367238105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c540724f0 msgr2=0x7f4c541a5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.576+0000 7f4c537fe640 1 --2- 172.21.15.29:0/2367238105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c540724f0 0x7f4c541a5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.576+0000 7f4c537fe640 1 -- 172.21.15.29:0/2367238105 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c44009560 con 0x7f4c541087c0 2024-03-11T19:19:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.576+0000 7f4c537fe640 1 --2- 172.21.15.29:0/2367238105 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c541087c0 0x7f4c541a63e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4c4800b470 tx=0x7f4c4800b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.577+0000 7f4c517fa640 1 -- 172.21.15.29:0/2367238105 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c48011890 con 0x7f4c541087c0 2024-03-11T19:19:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.577+0000 7f4c5a849640 1 -- 172.21.15.29:0/2367238105 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c541adfa0 con 0x7f4c541087c0 2024-03-11T19:19:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.577+0000 7f4c5a849640 1 -- 172.21.15.29:0/2367238105 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c541ae440 con 0x7f4c541087c0 2024-03-11T19:19:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.577+0000 7f4c517fa640 1 -- 172.21.15.29:0/2367238105 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4c480103e0 con 0x7f4c541087c0 2024-03-11T19:19:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.577+0000 7f4c517fa640 1 -- 172.21.15.29:0/2367238105 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c48011ea0 con 0x7f4c541087c0 2024-03-11T19:19:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.579+0000 7f4c517fa640 1 -- 172.21.15.29:0/2367238105 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4c480119f0 con 0x7f4c541087c0 2024-03-11T19:19:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.580+0000 7f4c5d416740 1 Processor -- start 2024-03-11T19:19:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.580+0000 7f4c517fa640 1 --2- 172.21.15.29:0/2367238105 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4c3c0a1bb0 0x7f4c3c0a4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.580+0000 7f4c5d416740 1 -- start start 2024-03-11T19:19:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.580+0000 7f4c5d416740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 0x56460eb788d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.580+0000 7f4c5d416740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb78ea0 0x56460eb7d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.580+0000 7f4c5d416740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb7e010 0x56460eb804a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.580+0000 7f4c5d416740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56460eae8280 con 0x56460eb7e010 2024-03-11T19:19:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.580+0000 7f4c5d416740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56460eae57b0 con 0x56460eb784b0 2024-03-11T19:19:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.580+0000 7f4c5d416740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56460eae0f90 con 0x56460eb78ea0 2024-03-11T19:19:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.581+0000 7f4c517fa640 1 -- 172.21.15.29:0/2367238105 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4c480cceb0 con 0x7f4c541087c0 2024-03-11T19:19:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.581+0000 7f4c537fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb78ea0 0x56460eb7d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.581+0000 7f4c53fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 0x56460eb788d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.581+0000 7f4c53fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 0x56460eb788d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:58702/0 (socket says 172.21.15.29:58702) 2024-03-11T19:19:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.581+0000 7f4c537fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb78ea0 0x56460eb7d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:56450/0 (socket says 172.21.15.29:56450) 2024-03-11T19:19:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.581+0000 7f4c53fff640 1 -- 172.21.15.29:0/3177671489 learned_addr learned my addr 172.21.15.29:0/3177671489 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.581+0000 7f4c537fe640 1 -- 172.21.15.29:0/3177671489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 msgr2=0x56460eb788d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.581+0000 7f4c58de6640 1 --2- 172.21.15.29:0/2367238105 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4c3c0a1bb0 0x7f4c3c0a4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.581+0000 7f4c537fe640 1 --2- 172.21.15.29:0/3177671489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 0x56460eb788d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.581+0000 7f4c537fe640 1 -- 172.21.15.29:0/3177671489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb7e010 msgr2=0x56460eb804a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.581+0000 7f4c537fe640 1 --2- 172.21.15.29:0/3177671489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb7e010 0x56460eb804a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.581+0000 7f4c537fe640 1 -- 172.21.15.29:0/3177671489 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56460ea9f570 con 0x56460eb78ea0 2024-03-11T19:19:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.581+0000 7f4c53fff640 1 --2- 172.21.15.29:0/3177671489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 0x56460eb788d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.581+0000 7f4c537fe640 1 --2- 172.21.15.29:0/3177671489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb78ea0 0x56460eb7d9a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4c480caa80 tx=0x7f4c48096250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4399d5f6c2f648b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.582+0000 7f4c3a7fc640 1 -- 172.21.15.29:0/3177671489 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c48008be0 con 0x56460eb78ea0 2024-03-11T19:19:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.582+0000 7f4c3a7fc640 1 -- 172.21.15.29:0/3177671489 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4c48008a00 con 0x56460eb78ea0 2024-03-11T19:19:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.582+0000 7f4c3a7fc640 1 -- 172.21.15.29:0/3177671489 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4809dff0 con 0x56460eb78ea0 2024-03-11T19:19:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.582+0000 7f4c58de6640 1 --2- 172.21.15.29:0/2367238105 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4c3c0a1bb0 0x7f4c3c0a4070 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4c44006010 tx=0x7f4c4402c180 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.582+0000 7f4c5d416740 1 -- 172.21.15.29:0/3177671489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb78ea0 msgr2=0x56460eb7d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.583+0000 7f4c5d416740 1 --2- 172.21.15.29:0/3177671489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb78ea0 0x56460eb7d9a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4c480caa80 tx=0x7f4c48096250 comp rx=0 tx=0).stop 2024-03-11T19:19:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.583+0000 7f4c5d416740 1 -- 172.21.15.29:0/3177671489 shutdown_connections 2024-03-11T19:19:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.583+0000 7f4c5d416740 1 --2- 172.21.15.29:0/3177671489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb7e010 0x56460eb804a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.583+0000 7f4c5d416740 1 --2- 172.21.15.29:0/3177671489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb78ea0 0x56460eb7d9a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.583+0000 7f4c5d416740 1 --2- 172.21.15.29:0/3177671489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 0x56460eb788d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.583+0000 7f4c5d416740 1 -- 172.21.15.29:0/3177671489 >> 172.21.15.29:0/3177671489 conn(0x56460eb73a20 msgr2=0x56460eb73e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.583+0000 7f4c5d416740 1 -- 172.21.15.29:0/3177671489 shutdown_connections 2024-03-11T19:19:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.583+0000 7f4c5d416740 1 -- 172.21.15.29:0/3177671489 wait complete. 2024-03-11T19:19:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.583+0000 7f4c5d416740 1 -- start start 2024-03-11T19:19:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.584+0000 7f4c5d416740 1 Processor -- start 2024-03-11T19:19:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.584+0000 7f4c5d416740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb784b0 0x56460eb93440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.584+0000 7f4c5d416740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb78ea0 0x56460eb93980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.584+0000 7f4c5d416740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb7e010 0x56460eb9a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.584+0000 7f4c5d416740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56460e7b4430 con 0x56460eb784b0 2024-03-11T19:19:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.584+0000 7f4c5d416740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56460e770e00 con 0x56460eb78ea0 2024-03-11T19:19:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.584+0000 7f4c5d416740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56460ea65fd0 con 0x56460eb7e010 2024-03-11T19:19:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.584+0000 7f4c5d416740 10 client.0 _ll_register_callbacks cb 0x56460eabbfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.584+0000 7f4c53fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb784b0 0x56460eb93440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.584+0000 7f4c53fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb784b0 0x56460eb93440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45362/0 (socket says 172.21.15.29:45362) 2024-03-11T19:19:46.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.584+0000 7f4c53fff640 1 -- 172.21.15.29:0/4266145340 learned_addr learned my addr 172.21.15.29:0/4266145340 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:46.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.584+0000 7f4c537fe640 1 --2- 172.21.15.29:0/4266145340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb78ea0 0x56460eb93980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.584+0000 7f4c58de6640 1 --2- 172.21.15.29:0/4266145340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb7e010 0x56460eb9a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.585+0000 7f4c537fe640 1 -- 172.21.15.29:0/4266145340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb7e010 msgr2=0x56460eb9a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:46.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.585+0000 7f4c537fe640 1 --2- 172.21.15.29:0/4266145340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb7e010 0x56460eb9a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.585+0000 7f4c537fe640 1 -- 172.21.15.29:0/4266145340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb784b0 msgr2=0x56460eb93440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:46.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.585+0000 7f4c537fe640 1 --2- 172.21.15.29:0/4266145340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb784b0 0x56460eb93440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.585+0000 7f4c537fe640 1 -- 172.21.15.29:0/4266145340 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c541ae440 con 0x56460eb78ea0 2024-03-11T19:19:46.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.585+0000 7f4c58de6640 1 --2- 172.21.15.29:0/4266145340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb7e010 0x56460eb9a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:46.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.585+0000 7f4c53fff640 1 --2- 172.21.15.29:0/4266145340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb784b0 0x56460eb93440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:19:46.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.585+0000 7f4c537fe640 1 --2- 172.21.15.29:0/4266145340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb78ea0 0x56460eb93980 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4c48006fd0 tx=0x7f4c48012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1050cad80f5d98b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:46.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.585+0000 7f4c337fe640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:19:46.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.585+0000 7f4c337fe640 1 -- 172.21.15.29:0/4266145340 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c480a2640 con 0x56460eb78ea0 2024-03-11T19:19:46.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.585+0000 7f4c337fe640 1 -- 172.21.15.29:0/4266145340 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4c4809dff0 con 0x56460eb78ea0 2024-03-11T19:19:46.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.585+0000 7f4c5d416740 1 -- 172.21.15.29:0/4266145340 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56460e7e07e0 con 0x56460eb78ea0 2024-03-11T19:19:46.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.585+0000 7f4c337fe640 1 -- 172.21.15.29:0/4266145340 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c48011ca0 con 0x56460eb78ea0 2024-03-11T19:19:46.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.585+0000 7f4c5d416740 10 client.27302 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:46.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.585+0000 7f4c5d416740 1 -- 172.21.15.29:0/4266145340 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56460eadf610 con 0x56460eb78ea0 2024-03-11T19:19:46.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.586+0000 7f4c337fe640 1 -- 172.21.15.29:0/4266145340 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4c480a8080 con 0x56460eb78ea0 2024-03-11T19:19:46.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.586+0000 7f4c337fe640 1 -- 172.21.15.29:0/4266145340 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4c48008570 con 0x56460eb78ea0 2024-03-11T19:19:46.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.586+0000 7f4c5d416740 10 client.27302 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:46.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.586+0000 7f4c5d416740 10 client.27302 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:19:46.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.586+0000 7f4c5d416740 20 client.27302 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:46.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.586+0000 7f4c5d416740 1 --2- 172.21.15.29:0/4266145340 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56460eb9f200 0x56460eba1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.586+0000 7f4c5d416740 4 client.27302 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:19:46.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.586+0000 7f4c5d416740 1 -- 172.21.15.29:0/4266145340 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56460e81d720 con 0x56460eb9f200 2024-03-11T19:19:46.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.586+0000 7f4c53fff640 1 --2- 172.21.15.29:0/4266145340 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56460eb9f200 0x56460eba1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.587+0000 7f4c53fff640 1 --2- 172.21.15.29:0/4266145340 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56460eb9f200 0x56460eba1620 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56b5cecb996148dd server_cookie=89de0b9afa2204c4 in_seq=0 out_seq=0 2024-03-11T19:19:46.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.587+0000 7f4c337fe640 10 client.27302 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:46.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.587+0000 7f4c337fe640 1 -- 172.21.15.29:0/4266145340 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56460e81d720 con 0x56460eb9f200 2024-03-11T19:19:46.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.587+0000 7f4c337fe640 10 client.27302 handle_command_reply: tid=0 2024-03-11T19:19:46.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.588+0000 7f4c5d416740 1 client.27302 shutdown 2024-03-11T19:19:46.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.588+0000 7f4c5d416740 1 -- 172.21.15.29:0/4266145340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb78ea0 msgr2=0x56460eb93980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:46.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.588+0000 7f4c5d416740 1 --2- 172.21.15.29:0/4266145340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb78ea0 0x56460eb93980 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4c48006fd0 tx=0x7f4c48012000 comp rx=0 tx=0).stop 2024-03-11T19:19:46.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.588+0000 7f4c5d416740 1 -- 172.21.15.29:0/4266145340 shutdown_connections 2024-03-11T19:19:46.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.588+0000 7f4c5d416740 1 --2- 172.21.15.29:0/4266145340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb7e010 0x56460eb9a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.588+0000 7f4c5d416740 1 --2- 172.21.15.29:0/4266145340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb78ea0 0x56460eb93980 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.588+0000 7f4c5d416740 1 --2- 172.21.15.29:0/4266145340 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56460eb9f200 0x56460eba1620 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.588+0000 7f4c5d416740 1 --2- 172.21.15.29:0/4266145340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb784b0 0x56460eb93440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.588+0000 7f4c337fe640 0 client.27302 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.588+0000 7f4c5d416740 1 -- 172.21.15.29:0/4266145340 >> 172.21.15.29:0/4266145340 conn(0x56460eb73a20 msgr2=0x56460eb72c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.588+0000 7f4c5d416740 1 -- 172.21.15.29:0/4266145340 shutdown_connections 2024-03-11T19:19:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.588+0000 7f4c5d416740 1 -- 172.21.15.29:0/4266145340 wait complete. 2024-03-11T19:19:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.589+0000 7f4c5d416740 20 client.27302 trim_cache size 0 max 16384 2024-03-11T19:19:46.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.624+0000 7f4c5d416740 1 Processor -- start 2024-03-11T19:19:46.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.624+0000 7f4c5d416740 1 -- start start 2024-03-11T19:19:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.624+0000 7f4c5d416740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 0x56460eba5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.624+0000 7f4c5d416740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb78ea0 0x56460eb9a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.624+0000 7f4c5d416740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb7e010 0x56460eb7a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.625+0000 7f4c53fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 0x56460eba5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.625+0000 7f4c53fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 0x56460eba5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:58712/0 (socket says 172.21.15.29:58712) 2024-03-11T19:19:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.625+0000 7f4c5d416740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56460eb749c0 con 0x56460eb7e010 2024-03-11T19:19:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.625+0000 7f4c5d416740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56460eb79280 con 0x56460eb784b0 2024-03-11T19:19:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.625+0000 7f4c5d416740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56460eb7d3d0 con 0x56460eb78ea0 2024-03-11T19:19:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.625+0000 7f4c53fff640 1 -- 172.21.15.29:0/250604891 learned_addr learned my addr 172.21.15.29:0/250604891 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.625+0000 7f4c537fe640 1 --2- 172.21.15.29:0/250604891 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb78ea0 0x56460eb9a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.625+0000 7f4c58de6640 1 --2- 172.21.15.29:0/250604891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb7e010 0x56460eb7a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.625+0000 7f4c53fff640 1 -- 172.21.15.29:0/250604891 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb78ea0 msgr2=0x56460eb9a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.625+0000 7f4c53fff640 1 --2- 172.21.15.29:0/250604891 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb78ea0 0x56460eb9a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.625+0000 7f4c53fff640 1 -- 172.21.15.29:0/250604891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb7e010 msgr2=0x56460eb7a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.625+0000 7f4c53fff640 1 --2- 172.21.15.29:0/250604891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb7e010 0x56460eb7a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.625+0000 7f4c53fff640 1 -- 172.21.15.29:0/250604891 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56460ea7bac0 con 0x56460eb784b0 2024-03-11T19:19:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.626+0000 7f4c53fff640 1 --2- 172.21.15.29:0/250604891 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 0x56460eba5fa0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00a920 tx=0x7f4c4c0096d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75f1f9806aa0a1df server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.626+0000 7f4c39ffb640 1 -- 172.21.15.29:0/250604891 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c010040 con 0x56460eb784b0 2024-03-11T19:19:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.627+0000 7f4c39ffb640 1 -- 172.21.15.29:0/250604891 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4c4c019cb0 con 0x56460eb784b0 2024-03-11T19:19:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.628+0000 7f4c39ffb640 1 -- 172.21.15.29:0/250604891 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c0165e0 con 0x56460eb784b0 2024-03-11T19:19:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.628+0000 7f4c5d416740 1 -- 172.21.15.29:0/250604891 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 msgr2=0x56460eba5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.628+0000 7f4c5d416740 1 --2- 172.21.15.29:0/250604891 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 0x56460eba5fa0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00a920 tx=0x7f4c4c0096d0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.628+0000 7f4c5d416740 1 -- 172.21.15.29:0/250604891 shutdown_connections 2024-03-11T19:19:46.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.628+0000 7f4c5d416740 1 --2- 172.21.15.29:0/250604891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb7e010 0x56460eb7a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.628+0000 7f4c5d416740 1 --2- 172.21.15.29:0/250604891 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb78ea0 0x56460eb9a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.628+0000 7f4c5d416740 1 --2- 172.21.15.29:0/250604891 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 0x56460eba5fa0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.628+0000 7f4c5d416740 1 -- 172.21.15.29:0/250604891 >> 172.21.15.29:0/250604891 conn(0x56460eb73a20 msgr2=0x56460eb6e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.628+0000 7f4c5d416740 1 -- 172.21.15.29:0/250604891 shutdown_connections 2024-03-11T19:19:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.628+0000 7f4c5d416740 1 -- 172.21.15.29:0/250604891 wait complete. 2024-03-11T19:19:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.628+0000 7f4c5d416740 1 -- start start 2024-03-11T19:19:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.629+0000 7f4c5d416740 1 Processor -- start 2024-03-11T19:19:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.629+0000 7f4c5d416740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 0x56460eb9a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.629+0000 7f4c5d416740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb78ea0 0x56460eb98d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.629+0000 7f4c5d416740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb7e010 0x56460eb992a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.629+0000 7f4c5d416740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56460eb998e0 con 0x56460eb78ea0 2024-03-11T19:19:46.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.629+0000 7f4c5d416740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56460eb99a50 con 0x56460eb784b0 2024-03-11T19:19:46.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.629+0000 7f4c5d416740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56460eb99c10 con 0x56460eb7e010 2024-03-11T19:19:46.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.629+0000 7f4c5d416740 10 client.0 _ll_register_callbacks cb 0x7f4c4c009ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:19:46.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.629+0000 7f4c53fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 0x56460eb9a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.629+0000 7f4c53fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 0x56460eb9a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:58716/0 (socket says 172.21.15.29:58716) 2024-03-11T19:19:46.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.629+0000 7f4c53fff640 1 -- 172.21.15.29:0/1400375945 learned_addr learned my addr 172.21.15.29:0/1400375945 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:19:46.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.629+0000 7f4c537fe640 1 --2- 172.21.15.29:0/1400375945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb78ea0 0x56460eb98d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.630+0000 7f4c58de6640 1 --2- 172.21.15.29:0/1400375945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb7e010 0x56460eb992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.630+0000 7f4c53fff640 1 -- 172.21.15.29:0/1400375945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb7e010 msgr2=0x56460eb992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.630+0000 7f4c53fff640 1 --2- 172.21.15.29:0/1400375945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb7e010 0x56460eb992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.630+0000 7f4c53fff640 1 -- 172.21.15.29:0/1400375945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb78ea0 msgr2=0x56460eb98d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:19:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.630+0000 7f4c53fff640 1 --2- 172.21.15.29:0/1400375945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb78ea0 0x56460eb98d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.630+0000 7f4c53fff640 1 -- 172.21.15.29:0/1400375945 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4c009390 con 0x56460eb784b0 2024-03-11T19:19:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.630+0000 7f4c53fff640 1 --2- 172.21.15.29:0/1400375945 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 0x56460eb9a750 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009950 tx=0x7f4c4c017620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa69bbd57db25f36 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:19:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.630+0000 7f4c337fe640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:19:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.631+0000 7f4c337fe640 1 -- 172.21.15.29:0/1400375945 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c010070 con 0x56460eb784b0 2024-03-11T19:19:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.631+0000 7f4c337fe640 1 -- 172.21.15.29:0/1400375945 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4c4c017920 con 0x56460eb784b0 2024-03-11T19:19:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.631+0000 7f4c5d416740 1 -- 172.21.15.29:0/1400375945 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c48008a00 con 0x56460eb784b0 2024-03-11T19:19:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.632+0000 7f4c337fe640 1 -- 172.21.15.29:0/1400375945 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c4c017ae0 con 0x56460eb784b0 2024-03-11T19:19:46.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.633+0000 7f4c5d416740 10 client.27308 fetch_fsmap learned FSMap version 27 2024-03-11T19:19:46.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.633+0000 7f4c5d416740 1 -- 172.21.15.29:0/1400375945 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56460eaa16a0 con 0x56460eb784b0 2024-03-11T19:19:46.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.633+0000 7f4c337fe640 1 -- 172.21.15.29:0/1400375945 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4c48008a00 con 0x56460eb784b0 2024-03-11T19:19:46.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.633+0000 7f4c337fe640 1 -- 172.21.15.29:0/1400375945 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4c4c008040 con 0x56460eb784b0 2024-03-11T19:19:46.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.633+0000 7f4c5d416740 10 client.27308 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:19:46.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.633+0000 7f4c5d416740 10 client.27308 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:19:46.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.633+0000 7f4c5d416740 20 client.27308 populate_metadata read hostname 'smithi029' 2024-03-11T19:19:46.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.633+0000 7f4c5d416740 1 --2- 172.21.15.29:0/1400375945 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56460ebf7060 0x56460ebf9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:19:46.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.633+0000 7f4c5d416740 4 client.27308 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:19:46.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.633+0000 7f4c5d416740 1 -- 172.21.15.29:0/1400375945 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56460e86c890 con 0x56460ebf7060 2024-03-11T19:19:46.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.634+0000 7f4c537fe640 1 --2- 172.21.15.29:0/1400375945 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56460ebf7060 0x56460ebf9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:19:46.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.635+0000 7f4c537fe640 1 --2- 172.21.15.29:0/1400375945 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56460ebf7060 0x56460ebf9480 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3358ea4d579a1adf server_cookie=338035b17e8a11ef in_seq=0 out_seq=0 2024-03-11T19:19:46.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.635+0000 7f4c337fe640 10 client.27308 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:46.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.635+0000 7f4c337fe640 1 -- 172.21.15.29:0/1400375945 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56460e86c890 con 0x56460ebf7060 2024-03-11T19:19:46.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.635+0000 7f4c337fe640 10 client.27308 handle_command_reply: tid=0 2024-03-11T19:19:46.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.635+0000 7f4c5d416740 1 client.27308 shutdown 2024-03-11T19:19:46.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.636+0000 7f4c5d416740 1 -- 172.21.15.29:0/1400375945 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 msgr2=0x56460eb9a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:46.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.636+0000 7f4c5d416740 1 --2- 172.21.15.29:0/1400375945 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 0x56460eb9a750 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009950 tx=0x7f4c4c017620 comp rx=0 tx=0).stop 2024-03-11T19:19:46.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.636+0000 7f4c5d416740 1 -- 172.21.15.29:0/1400375945 shutdown_connections 2024-03-11T19:19:46.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.636+0000 7f4c5d416740 1 --2- 172.21.15.29:0/1400375945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56460eb7e010 0x56460eb992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.636+0000 7f4c5d416740 1 --2- 172.21.15.29:0/1400375945 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56460ebf7060 0x56460ebf9480 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.636+0000 7f4c5d416740 1 --2- 172.21.15.29:0/1400375945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56460eb78ea0 0x56460eb98d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.636+0000 7f4c5d416740 1 --2- 172.21.15.29:0/1400375945 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56460eb784b0 0x56460eb9a750 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.636+0000 7f4c5d416740 1 -- 172.21.15.29:0/1400375945 >> 172.21.15.29:0/1400375945 conn(0x56460eb73a20 msgr2=0x56460eba5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:46.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.636+0000 7f4c337fe640 0 client.27308 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:19:46.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.636+0000 7f4c5d416740 1 -- 172.21.15.29:0/1400375945 shutdown_connections 2024-03-11T19:19:46.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.636+0000 7f4c5d416740 1 -- 172.21.15.29:0/1400375945 wait complete. 2024-03-11T19:19:46.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.636+0000 7f4c5d416740 20 client.27308 trim_cache size 0 max 16384 2024-03-11T19:19:46.641 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:19:46.641 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:19:46.641 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:19:46.641 INFO:teuthology.orchestra.run.smithi029.stdout: "f1a6978e-efe3-4b63-9ff6-8c7319efd68e": { 2024-03-11T19:19:46.641 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:19:46.641 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "f1a6978e-efe3-4b63-9ff6-8c7319efd68e", 2024-03-11T19:19:46.641 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:19:46.641 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:19:46.642 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:19:46.642 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:19:46.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.637+0000 7f4c5a849640 1 -- 172.21.15.29:0/2367238105 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4c3c0a1bb0 msgr2=0x7f4c3c0a4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:46.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.637+0000 7f4c5a849640 1 --2- 172.21.15.29:0/2367238105 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4c3c0a1bb0 0x7f4c3c0a4070 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4c44006010 tx=0x7f4c4402c180 comp rx=0 tx=0).stop 2024-03-11T19:19:46.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.637+0000 7f4c5a849640 1 -- 172.21.15.29:0/2367238105 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c541087c0 msgr2=0x7f4c541a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:19:46.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.637+0000 7f4c5a849640 1 --2- 172.21.15.29:0/2367238105 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c541087c0 0x7f4c541a63e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4c4800b470 tx=0x7f4c4800b930 comp rx=0 tx=0).stop 2024-03-11T19:19:46.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.637+0000 7f4c5a849640 1 -- 172.21.15.29:0/2367238105 shutdown_connections 2024-03-11T19:19:46.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.637+0000 7f4c5a849640 1 --2- 172.21.15.29:0/2367238105 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c54109180 0x7f4c541ad450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.637+0000 7f4c5a849640 1 --2- 172.21.15.29:0/2367238105 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c541087c0 0x7f4c541a63e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.637+0000 7f4c5a849640 1 --2- 172.21.15.29:0/2367238105 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4c3c0a1bb0 0x7f4c3c0a4070 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.637+0000 7f4c5a849640 1 --2- 172.21.15.29:0/2367238105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c540724f0 0x7f4c541a5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:19:46.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.637+0000 7f4c5a849640 1 -- 172.21.15.29:0/2367238105 >> 172.21.15.29:0/2367238105 conn(0x7f4c5406dfa0 msgr2=0x7f4c5406fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:19:46.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.637+0000 7f4c5a849640 1 -- 172.21.15.29:0/2367238105 shutdown_connections 2024-03-11T19:19:46.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:19:46.638+0000 7f4c5a849640 1 -- 172.21.15.29:0/2367238105 wait complete. 2024-03-11T19:19:46.658 INFO:tasks.cephfs.filesystem:scrub status for tag:f1a6978e-efe3-4b63-9ff6-8c7319efd68e - {'path': '/', 'tag': 'f1a6978e-efe3-4b63-9ff6-8c7319efd68e', 'options': 'recursive,force'} 2024-03-11T19:19:47.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:46 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:19:47.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:46 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:19:47.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:46 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:19:47.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:46 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:19:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:46 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:19:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:46 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:19:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:46 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:19:47.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:46 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:19:47.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:46 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:19:47.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:46 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:19:47.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:46 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:19:47.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:46 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:19:48.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:47 smithi190 ceph-mon[36292]: pgmap v650: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 58 MiB/s wr, 1.19k op/s 2024-03-11T19:19:48.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:47 smithi170 ceph-mon[35690]: pgmap v650: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 58 MiB/s wr, 1.19k op/s 2024-03-11T19:19:48.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:47 smithi029 ceph-mon[32439]: pgmap v650: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 58 MiB/s wr, 1.19k op/s 2024-03-11T19:19:49.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:48 smithi190 ceph-mon[36292]: pgmap v651: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 51 MiB/s wr, 1.06k op/s 2024-03-11T19:19:49.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:48 smithi170 ceph-mon[35690]: pgmap v651: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 51 MiB/s wr, 1.06k op/s 2024-03-11T19:19:49.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:48 smithi029 ceph-mon[32439]: pgmap v651: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 51 MiB/s wr, 1.06k op/s 2024-03-11T19:19:51.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:50 smithi029 ceph-mon[32439]: pgmap v652: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 57 MiB/s wr, 1.18k op/s 2024-03-11T19:19:51.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:50 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:19:51.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:50 smithi190 ceph-mon[36292]: pgmap v652: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 57 MiB/s wr, 1.18k op/s 2024-03-11T19:19:51.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:50 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:19:51.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:50 smithi170 ceph-mon[35690]: pgmap v652: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 57 MiB/s wr, 1.18k op/s 2024-03-11T19:19:51.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:50 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:19:53.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:52 smithi029 ceph-mon[32439]: pgmap v653: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 58 MiB/s wr, 1.19k op/s 2024-03-11T19:19:53.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:52 smithi190 ceph-mon[36292]: pgmap v653: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 58 MiB/s wr, 1.19k op/s 2024-03-11T19:19:53.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:52 smithi170 ceph-mon[35690]: pgmap v653: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 58 MiB/s wr, 1.19k op/s 2024-03-11T19:19:55.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:54 smithi029 ceph-mon[32439]: pgmap v654: 97 pgs: 97 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 55 MiB/s wr, 1.12k op/s 2024-03-11T19:19:55.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:54 smithi190 ceph-mon[36292]: pgmap v654: 97 pgs: 97 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 55 MiB/s wr, 1.12k op/s 2024-03-11T19:19:55.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:54 smithi170 ceph-mon[35690]: pgmap v654: 97 pgs: 97 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 55 MiB/s wr, 1.12k op/s 2024-03-11T19:19:57.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:56 smithi029 ceph-mon[32439]: pgmap v655: 97 pgs: 97 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 61 MiB/s wr, 1.23k op/s 2024-03-11T19:19:57.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:56 smithi190 ceph-mon[36292]: pgmap v655: 97 pgs: 97 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 61 MiB/s wr, 1.23k op/s 2024-03-11T19:19:57.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:56 smithi170 ceph-mon[35690]: pgmap v655: 97 pgs: 97 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 61 MiB/s wr, 1.23k op/s 2024-03-11T19:19:59.148 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:19:58 smithi170 ceph-mon[35690]: pgmap v656: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 55 MiB/s wr, 1.10k op/s 2024-03-11T19:19:59.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:19:58 smithi029 ceph-mon[32439]: pgmap v656: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 55 MiB/s wr, 1.10k op/s 2024-03-11T19:19:59.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:19:58 smithi190 ceph-mon[36292]: pgmap v656: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 55 MiB/s wr, 1.10k op/s 2024-03-11T19:20:01.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:00 smithi029 ceph-mon[32439]: pgmap v657: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 61 MiB/s wr, 1.21k op/s 2024-03-11T19:20:01.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:00 smithi029 ceph-mon[32439]: overall HEALTH_OK 2024-03-11T19:20:01.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:00 smithi190 ceph-mon[36292]: pgmap v657: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 61 MiB/s wr, 1.21k op/s 2024-03-11T19:20:01.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:00 smithi190 ceph-mon[36292]: overall HEALTH_OK 2024-03-11T19:20:01.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:00 smithi170 ceph-mon[35690]: pgmap v657: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 61 MiB/s wr, 1.21k op/s 2024-03-11T19:20:01.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:00 smithi170 ceph-mon[35690]: overall HEALTH_OK 2024-03-11T19:20:03.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:02 smithi029 ceph-mon[32439]: pgmap v658: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 61 MiB/s wr, 1.20k op/s 2024-03-11T19:20:03.209 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:02 smithi170 ceph-mon[35690]: pgmap v658: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 61 MiB/s wr, 1.20k op/s 2024-03-11T19:20:03.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:02 smithi190 ceph-mon[36292]: pgmap v658: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 61 MiB/s wr, 1.20k op/s 2024-03-11T19:20:05.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:04 smithi029 ceph-mon[32439]: pgmap v659: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 58 MiB/s wr, 1.14k op/s 2024-03-11T19:20:05.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:04 smithi190 ceph-mon[36292]: pgmap v659: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 58 MiB/s wr, 1.14k op/s 2024-03-11T19:20:05.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:04 smithi170 ceph-mon[35690]: pgmap v659: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 58 MiB/s wr, 1.14k op/s 2024-03-11T19:20:07.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:06 smithi190 ceph-mon[36292]: pgmap v660: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 62 MiB/s wr, 1.22k op/s 2024-03-11T19:20:07.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:06 smithi170 ceph-mon[35690]: pgmap v660: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 62 MiB/s wr, 1.22k op/s 2024-03-11T19:20:07.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:06 smithi029 ceph-mon[32439]: pgmap v660: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 62 MiB/s wr, 1.22k op/s 2024-03-11T19:20:09.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:08 smithi190 ceph-mon[36292]: pgmap v661: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 55 MiB/s wr, 1.07k op/s 2024-03-11T19:20:09.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:08 smithi170 ceph-mon[35690]: pgmap v661: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 55 MiB/s wr, 1.07k op/s 2024-03-11T19:20:09.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:08 smithi029 ceph-mon[32439]: pgmap v661: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 55 MiB/s wr, 1.07k op/s 2024-03-11T19:20:11.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:10 smithi190 ceph-mon[36292]: pgmap v662: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 60 MiB/s wr, 1.17k op/s 2024-03-11T19:20:11.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:10 smithi170 ceph-mon[35690]: pgmap v662: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 60 MiB/s wr, 1.17k op/s 2024-03-11T19:20:11.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:10 smithi029 ceph-mon[32439]: pgmap v662: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 60 MiB/s wr, 1.17k op/s 2024-03-11T19:20:13.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:12 smithi190 ceph-mon[36292]: pgmap v663: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 59 MiB/s wr, 1.16k op/s 2024-03-11T19:20:13.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:12 smithi170 ceph-mon[35690]: pgmap v663: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 59 MiB/s wr, 1.16k op/s 2024-03-11T19:20:13.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:12 smithi029 ceph-mon[32439]: pgmap v663: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 59 MiB/s wr, 1.16k op/s 2024-03-11T19:20:15.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:14 smithi190 ceph-mon[36292]: pgmap v664: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 55 MiB/s wr, 1.08k op/s 2024-03-11T19:20:15.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:14 smithi170 ceph-mon[35690]: pgmap v664: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 55 MiB/s wr, 1.08k op/s 2024-03-11T19:20:15.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:14 smithi029 ceph-mon[32439]: pgmap v664: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 55 MiB/s wr, 1.08k op/s 2024-03-11T19:20:16.659 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:20:16.659 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:20:16.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.845+0000 7f7b14e57640 1 -- 172.21.15.29:0/2571611209 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b100feef0 msgr2=0x7f7b100ff310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:16.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.845+0000 7f7b14e57640 1 --2- 172.21.15.29:0/2571611209 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b100feef0 0x7f7b100ff310 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009be0 tx=0x7f7b0802bef0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.845+0000 7f7b14e57640 1 -- 172.21.15.29:0/2571611209 shutdown_connections 2024-03-11T19:20:16.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.845+0000 7f7b14e57640 1 --2- 172.21.15.29:0/2571611209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b10106de0 0x7f7b101092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.845+0000 7f7b14e57640 1 --2- 172.21.15.29:0/2571611209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b100ff850 0x7f7b10106850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.845+0000 7f7b14e57640 1 --2- 172.21.15.29:0/2571611209 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b100feef0 0x7f7b100ff310 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.845+0000 7f7b14e57640 1 -- 172.21.15.29:0/2571611209 >> 172.21.15.29:0/2571611209 conn(0x7f7b100fa960 msgr2=0x7f7b100fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:16.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.846+0000 7f7b14e57640 1 -- 172.21.15.29:0/2571611209 shutdown_connections 2024-03-11T19:20:16.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.846+0000 7f7b14e57640 1 -- 172.21.15.29:0/2571611209 wait complete. 2024-03-11T19:20:16.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.847+0000 7f7b14e57640 1 Processor -- start 2024-03-11T19:20:16.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.847+0000 7f7b14e57640 1 -- start start 2024-03-11T19:20:16.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.847+0000 7f7b14e57640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b100feef0 0x7f7b10194ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:16.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.847+0000 7f7b14e57640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b100ff850 0x7f7b10195430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:16.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.847+0000 7f7b14e57640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b10106de0 0x7f7b1019c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:16.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.847+0000 7f7b14e57640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1019c9e0 con 0x7f7b100ff850 2024-03-11T19:20:16.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.847+0000 7f7b14e57640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1019cb50 con 0x7f7b10106de0 2024-03-11T19:20:16.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.847+0000 7f7b14e57640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1006abd0 con 0x7f7b100feef0 2024-03-11T19:20:16.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.847+0000 7f7b0ed9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b10106de0 0x7f7b1019c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:16.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.847+0000 7f7b0ed9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b10106de0 0x7f7b1019c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:36896/0 (socket says 172.21.15.29:36896) 2024-03-11T19:20:16.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.847+0000 7f7b0ed9d640 1 -- 172.21.15.29:0/1868284942 learned_addr learned my addr 172.21.15.29:0/1868284942 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:16.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.848+0000 7f7b0e59c640 1 --2- 172.21.15.29:0/1868284942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b100feef0 0x7f7b10194ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:16.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.848+0000 7f7b0ed9d640 1 -- 172.21.15.29:0/1868284942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b100feef0 msgr2=0x7f7b10194ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:16.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.848+0000 7f7b0ed9d640 1 --2- 172.21.15.29:0/1868284942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b100feef0 0x7f7b10194ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.848+0000 7f7b0ed9d640 1 -- 172.21.15.29:0/1868284942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b100ff850 msgr2=0x7f7b10195430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:16.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.848+0000 7f7b0ed9d640 1 --2- 172.21.15.29:0/1868284942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b100ff850 0x7f7b10195430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.848+0000 7f7b0ed9d640 1 -- 172.21.15.29:0/1868284942 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b08009840 con 0x7f7b10106de0 2024-03-11T19:20:16.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.848+0000 7f7b0e59c640 1 --2- 172.21.15.29:0/1868284942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b100feef0 0x7f7b10194ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:20:16.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.848+0000 7f7b0ed9d640 1 --2- 172.21.15.29:0/1868284942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b10106de0 0x7f7b1019c4a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7b0000e970 tx=0x7f7b0000ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:16.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.850+0000 7f7b077fe640 1 -- 172.21.15.29:0/1868284942 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0000ccc0 con 0x7f7b10106de0 2024-03-11T19:20:16.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.850+0000 7f7b077fe640 1 -- 172.21.15.29:0/1868284942 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7b0000ce20 con 0x7f7b10106de0 2024-03-11T19:20:16.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.851+0000 7f7b077fe640 1 -- 172.21.15.29:0/1868284942 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b00010430 con 0x7f7b10106de0 2024-03-11T19:20:16.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.852+0000 7f7b14e57640 1 -- 172.21.15.29:0/1868284942 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b1006aeb0 con 0x7f7b10106de0 2024-03-11T19:20:16.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.852+0000 7f7b14e57640 1 -- 172.21.15.29:0/1868284942 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b1006b340 con 0x7f7b10106de0 2024-03-11T19:20:16.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.855+0000 7f7b17a20740 1 Processor -- start 2024-03-11T19:20:16.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.857+0000 7f7b17a20740 1 -- start start 2024-03-11T19:20:16.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.857+0000 7f7b17a20740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bc2d0 0x55ea849bc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:16.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.857+0000 7f7b17a20740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bccc0 0x55ea849c17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:16.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.857+0000 7f7b17a20740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 0x55ea849c42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:16.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.857+0000 7f7b17a20740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea8492c280 con 0x55ea849bccc0 2024-03-11T19:20:16.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.857+0000 7f7b17a20740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea849297b0 con 0x55ea849bc2d0 2024-03-11T19:20:16.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.857+0000 7f7b17a20740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea84924f90 con 0x55ea849c1e30 2024-03-11T19:20:16.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.857+0000 7f7b077fe640 1 -- 172.21.15.29:0/1868284942 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7b00004680 con 0x7f7b10106de0 2024-03-11T19:20:16.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.858+0000 7f7b077fe640 1 --2- 172.21.15.29:0/1868284942 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7afc0a1aa0 0x7f7afc0a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:16.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.859+0000 7f7b077fe640 1 -- 172.21.15.29:0/1868284942 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7b00014070 con 0x7f7b10106de0 2024-03-11T19:20:16.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.861+0000 7f7b0dd9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bc2d0 0x55ea849bc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:16.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.862+0000 7f7b0e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bccc0 0x55ea849c17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:16.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.862+0000 7f7b0dd9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bc2d0 0x55ea849bc6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:36902/0 (socket says 172.21.15.29:36902) 2024-03-11T19:20:16.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.862+0000 7f7b0e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bccc0 0x55ea849c17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60834/0 (socket says 172.21.15.29:60834) 2024-03-11T19:20:16.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.862+0000 7f7b0e59c640 1 -- 172.21.15.29:0/1073230159 learned_addr learned my addr 172.21.15.29:0/1073230159 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:16.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.862+0000 7f7b0ed9d640 1 --2- 172.21.15.29:0/1868284942 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7afc0a1aa0 0x7f7afc0a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:16.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.862+0000 7f7b0dd9b640 1 --2- 172.21.15.29:0/1073230159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 0x55ea849c42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:16.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.862+0000 7f7b0dd9b640 1 -- 172.21.15.29:0/1073230159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 msgr2=0x55ea849c42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:16.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.862+0000 7f7b0dd9b640 1 --2- 172.21.15.29:0/1073230159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 0x55ea849c42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.862+0000 7f7b0dd9b640 1 -- 172.21.15.29:0/1073230159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bccc0 msgr2=0x55ea849c17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:16.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.862+0000 7f7b0dd9b640 1 --2- 172.21.15.29:0/1073230159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bccc0 0x55ea849c17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.862+0000 7f7b0dd9b640 1 -- 172.21.15.29:0/1073230159 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea848e3570 con 0x55ea849bc2d0 2024-03-11T19:20:16.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.864+0000 7f7b0ed9d640 1 --2- 172.21.15.29:0/1868284942 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7afc0a1aa0 0x7f7afc0a3f60 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7b00096ba0 tx=0x7f7b0000cf90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:16.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.866+0000 7f7b0dd9b640 1 --2- 172.21.15.29:0/1073230159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bc2d0 0x55ea849bc6f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7b10196400 tx=0x7f7af8009640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=296e2817585c098 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:16.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.868+0000 7f7b04ff9640 1 -- 172.21.15.29:0/1073230159 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8013110 con 0x55ea849bc2d0 2024-03-11T19:20:16.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.868+0000 7f7b04ff9640 1 -- 172.21.15.29:0/1073230159 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7af8019030 con 0x55ea849bc2d0 2024-03-11T19:20:16.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.868+0000 7f7b04ff9640 1 -- 172.21.15.29:0/1073230159 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8004b90 con 0x55ea849bc2d0 2024-03-11T19:20:16.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.869+0000 7f7b17a20740 1 -- 172.21.15.29:0/1073230159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bc2d0 msgr2=0x55ea849bc6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:16.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.869+0000 7f7b17a20740 1 --2- 172.21.15.29:0/1073230159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bc2d0 0x55ea849bc6f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7b10196400 tx=0x7f7af8009640 comp rx=0 tx=0).stop 2024-03-11T19:20:16.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.870+0000 7f7b17a20740 1 -- 172.21.15.29:0/1073230159 shutdown_connections 2024-03-11T19:20:16.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.870+0000 7f7b17a20740 1 --2- 172.21.15.29:0/1073230159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 0x55ea849c42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.870+0000 7f7b17a20740 1 --2- 172.21.15.29:0/1073230159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bccc0 0x55ea849c17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.870+0000 7f7b17a20740 1 --2- 172.21.15.29:0/1073230159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bc2d0 0x55ea849bc6f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.870+0000 7f7b17a20740 1 -- 172.21.15.29:0/1073230159 >> 172.21.15.29:0/1073230159 conn(0x55ea849b77f0 msgr2=0x55ea849b7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:16.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.872+0000 7f7b17a20740 1 -- 172.21.15.29:0/1073230159 shutdown_connections 2024-03-11T19:20:16.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.872+0000 7f7b17a20740 1 -- 172.21.15.29:0/1073230159 wait complete. 2024-03-11T19:20:16.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.873+0000 7f7b17a20740 1 -- start start 2024-03-11T19:20:16.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.873+0000 7f7b17a20740 1 Processor -- start 2024-03-11T19:20:16.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.873+0000 7f7b17a20740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 0x55ea849c6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:16.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.873+0000 7f7b17a20740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bccc0 0x55ea849c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:16.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.873+0000 7f7b17a20740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 0x55ea849cd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:16.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.873+0000 7f7b17a20740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea845f8430 con 0x55ea849bc2d0 2024-03-11T19:20:16.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.873+0000 7f7b17a20740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea845b4e00 con 0x55ea849bccc0 2024-03-11T19:20:16.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.873+0000 7f7b17a20740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea848a9fd0 con 0x55ea849c1e30 2024-03-11T19:20:16.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.874+0000 7f7b17a20740 10 client.0 _ll_register_callbacks cb 0x55ea848fffd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.874+0000 7f7b0dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 0x55ea849c6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.874+0000 7f7b0e59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bccc0 0x55ea849c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.874+0000 7f7b0dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 0x55ea849c6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60838/0 (socket says 172.21.15.29:60838) 2024-03-11T19:20:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.874+0000 7f7b0dd9b640 1 -- 172.21.15.29:0/1174651674 learned_addr learned my addr 172.21.15.29:0/1174651674 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.875+0000 7f7b0dd9b640 1 --2- 172.21.15.29:0/1174651674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 0x55ea849cd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.875+0000 7f7b0e59c640 1 -- 172.21.15.29:0/1174651674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 msgr2=0x55ea849cd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.875+0000 7f7b0e59c640 1 --2- 172.21.15.29:0/1174651674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 0x55ea849cd800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.875+0000 7f7b0e59c640 1 -- 172.21.15.29:0/1174651674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 msgr2=0x55ea849c6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.875+0000 7f7b0e59c640 1 --2- 172.21.15.29:0/1174651674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 0x55ea849c6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.875+0000 7f7b0e59c640 1 -- 172.21.15.29:0/1174651674 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af80134a0 con 0x55ea849bccc0 2024-03-11T19:20:16.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.875+0000 7f7b0dd9b640 1 --2- 172.21.15.29:0/1174651674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 0x55ea849c6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:20:16.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.875+0000 7f7af5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:20:16.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.875+0000 7f7b0e59c640 1 --2- 172.21.15.29:0/1174651674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bccc0 0x55ea849c67c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55ea849bd730 tx=0x7f7b08033710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40d6ddf331b0ca59 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.876+0000 7f7af5ffb640 1 -- 172.21.15.29:0/1174651674 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08003970 con 0x55ea849bccc0 2024-03-11T19:20:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.876+0000 7f7af5ffb640 1 -- 172.21.15.29:0/1174651674 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7b08044070 con 0x55ea849bccc0 2024-03-11T19:20:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.876+0000 7f7b17a20740 1 -- 172.21.15.29:0/1174651674 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ea846247e0 con 0x55ea849bccc0 2024-03-11T19:20:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.877+0000 7f7af5ffb640 1 -- 172.21.15.29:0/1174651674 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b08003970 con 0x55ea849bccc0 2024-03-11T19:20:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.877+0000 7f7b17a20740 10 client.27320 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.877+0000 7f7b17a20740 1 -- 172.21.15.29:0/1174651674 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ea84923610 con 0x55ea849bccc0 2024-03-11T19:20:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.877+0000 7f7af5ffb640 1 -- 172.21.15.29:0/1174651674 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55ea846247e0 con 0x55ea849bccc0 2024-03-11T19:20:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.877+0000 7f7af5ffb640 1 -- 172.21.15.29:0/1174651674 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7b080045e0 con 0x55ea849bccc0 2024-03-11T19:20:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.877+0000 7f7b17a20740 10 client.27320 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.877+0000 7f7b17a20740 10 client.27320 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:20:16.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.877+0000 7f7b17a20740 20 client.27320 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:16.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.877+0000 7f7b17a20740 1 --2- 172.21.15.29:0/1174651674 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ea849d2370 0x55ea849d4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:16.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.877+0000 7f7b17a20740 4 client.27320 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:20:16.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.877+0000 7f7b17a20740 1 -- 172.21.15.29:0/1174651674 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ea84661720 con 0x55ea849d2370 2024-03-11T19:20:16.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.878+0000 7f7b0dd9b640 1 --2- 172.21.15.29:0/1174651674 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ea849d2370 0x55ea849d4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:16.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.879+0000 7f7b0dd9b640 1 --2- 172.21.15.29:0/1174651674 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ea849d2370 0x55ea849d4760 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e9c62db3fcbfb89 server_cookie=85450168d5ad35f3 in_seq=0 out_seq=0 2024-03-11T19:20:16.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.879+0000 7f7af5ffb640 10 client.27320 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.880+0000 7f7af5ffb640 1 -- 172.21.15.29:0/1174651674 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ea84661720 con 0x55ea849d2370 2024-03-11T19:20:16.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.880+0000 7f7af5ffb640 10 client.27320 handle_command_reply: tid=0 2024-03-11T19:20:16.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.880+0000 7f7b17a20740 1 client.27320 shutdown 2024-03-11T19:20:16.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.880+0000 7f7b17a20740 1 -- 172.21.15.29:0/1174651674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bccc0 msgr2=0x55ea849c67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:16.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.880+0000 7f7b17a20740 1 --2- 172.21.15.29:0/1174651674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bccc0 0x55ea849c67c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55ea849bd730 tx=0x7f7b08033710 comp rx=0 tx=0).stop 2024-03-11T19:20:16.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.881+0000 7f7b17a20740 1 -- 172.21.15.29:0/1174651674 shutdown_connections 2024-03-11T19:20:16.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.881+0000 7f7b17a20740 1 --2- 172.21.15.29:0/1174651674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 0x55ea849cd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.881+0000 7f7b17a20740 1 --2- 172.21.15.29:0/1174651674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bccc0 0x55ea849c67c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.881+0000 7f7b17a20740 1 --2- 172.21.15.29:0/1174651674 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ea849d2370 0x55ea849d4760 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.881+0000 7f7b17a20740 1 --2- 172.21.15.29:0/1174651674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 0x55ea849c6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.881+0000 7f7b17a20740 1 -- 172.21.15.29:0/1174651674 >> 172.21.15.29:0/1174651674 conn(0x55ea849b77f0 msgr2=0x55ea849b6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:16.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.881+0000 7f7af5ffb640 0 client.27320 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:16.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.881+0000 7f7b17a20740 1 -- 172.21.15.29:0/1174651674 shutdown_connections 2024-03-11T19:20:16.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.881+0000 7f7b17a20740 1 -- 172.21.15.29:0/1174651674 wait complete. 2024-03-11T19:20:16.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.881+0000 7f7b17a20740 20 client.27320 trim_cache size 0 max 16384 2024-03-11T19:20:16.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.916+0000 7f7b17a20740 1 Processor -- start 2024-03-11T19:20:16.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.917+0000 7f7b17a20740 1 -- start start 2024-03-11T19:20:16.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.917+0000 7f7b17a20740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 0x55ea849c16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:16.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.917+0000 7f7b17a20740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bccc0 0x55ea849cd720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:16.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.917+0000 7f7b17a20740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 0x55ea849c6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:16.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.917+0000 7f7b17a20740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea849c4cf0 con 0x55ea849bc2d0 2024-03-11T19:20:16.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.917+0000 7f7b17a20740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea849b3350 con 0x55ea849bccc0 2024-03-11T19:20:16.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.917+0000 7f7b17a20740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea849b73b0 con 0x55ea849c1e30 2024-03-11T19:20:16.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.917+0000 7f7b0dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 0x55ea849c16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:16.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.917+0000 7f7b0dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 0x55ea849c16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60844/0 (socket says 172.21.15.29:60844) 2024-03-11T19:20:16.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.917+0000 7f7b0dd9b640 1 -- 172.21.15.29:0/4189499287 learned_addr learned my addr 172.21.15.29:0/4189499287 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:16.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.917+0000 7f7b0e59c640 1 --2- 172.21.15.29:0/4189499287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bccc0 0x55ea849cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:16.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.918+0000 7f7b0dd9b640 1 --2- 172.21.15.29:0/4189499287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 0x55ea849c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:16.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.918+0000 7f7b0e59c640 1 -- 172.21.15.29:0/4189499287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 msgr2=0x55ea849c6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:16.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.918+0000 7f7b0e59c640 1 --2- 172.21.15.29:0/4189499287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 0x55ea849c6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.918+0000 7f7b0e59c640 1 -- 172.21.15.29:0/4189499287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 msgr2=0x55ea849c16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:16.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.918+0000 7f7b0e59c640 1 --2- 172.21.15.29:0/4189499287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 0x55ea849c16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.918+0000 7f7b0e59c640 1 -- 172.21.15.29:0/4189499287 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea848bfac0 con 0x55ea849bccc0 2024-03-11T19:20:16.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.918+0000 7f7b0dd9b640 1 --2- 172.21.15.29:0/4189499287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 0x55ea849c16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:20:16.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.919+0000 7f7b0e59c640 1 --2- 172.21.15.29:0/4189499287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bccc0 0x55ea849cd720 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7b08047720 tx=0x7f7b08005e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ca76e8215f42f46 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:16.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.919+0000 7f7af7fff640 1 -- 172.21.15.29:0/4189499287 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08079070 con 0x55ea849bccc0 2024-03-11T19:20:16.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.919+0000 7f7af7fff640 1 -- 172.21.15.29:0/4189499287 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7b08042070 con 0x55ea849bccc0 2024-03-11T19:20:16.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.919+0000 7f7af7fff640 1 -- 172.21.15.29:0/4189499287 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08003970 con 0x55ea849bccc0 2024-03-11T19:20:16.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.921+0000 7f7b17a20740 1 -- 172.21.15.29:0/4189499287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bccc0 msgr2=0x55ea849cd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:16.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.921+0000 7f7b17a20740 1 --2- 172.21.15.29:0/4189499287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bccc0 0x55ea849cd720 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7b08047720 tx=0x7f7b08005e30 comp rx=0 tx=0).stop 2024-03-11T19:20:16.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.921+0000 7f7b17a20740 1 -- 172.21.15.29:0/4189499287 shutdown_connections 2024-03-11T19:20:16.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.921+0000 7f7b17a20740 1 --2- 172.21.15.29:0/4189499287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 0x55ea849c6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.921+0000 7f7b17a20740 1 --2- 172.21.15.29:0/4189499287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bccc0 0x55ea849cd720 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.921+0000 7f7b17a20740 1 --2- 172.21.15.29:0/4189499287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 0x55ea849c16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.921+0000 7f7b17a20740 1 -- 172.21.15.29:0/4189499287 >> 172.21.15.29:0/4189499287 conn(0x55ea849b77f0 msgr2=0x55ea849d90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:16.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.921+0000 7f7b17a20740 1 -- 172.21.15.29:0/4189499287 shutdown_connections 2024-03-11T19:20:16.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.921+0000 7f7b17a20740 1 -- 172.21.15.29:0/4189499287 wait complete. 2024-03-11T19:20:16.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.921+0000 7f7b17a20740 1 -- start start 2024-03-11T19:20:16.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.921+0000 7f7b17a20740 1 Processor -- start 2024-03-11T19:20:16.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.922+0000 7f7b17a20740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 0x55ea849cd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:16.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.922+0000 7f7b17a20740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bccc0 0x55ea849cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:16.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.922+0000 7f7b17a20740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 0x55ea849cc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:16.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.922+0000 7f7b17a20740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea849cc930 con 0x55ea849bc2d0 2024-03-11T19:20:16.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.922+0000 7f7b17a20740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea849ccaa0 con 0x55ea849bccc0 2024-03-11T19:20:16.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.922+0000 7f7b17a20740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea84a28830 con 0x55ea849c1e30 2024-03-11T19:20:16.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.922+0000 7f7b17a20740 10 client.0 _ll_register_callbacks cb 0x7f7af8009970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:16.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.922+0000 7f7b0dd9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 0x55ea849cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:16.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.922+0000 7f7b0dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 0x55ea849cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:16.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.922+0000 7f7b0dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 0x55ea849cd670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60860/0 (socket says 172.21.15.29:60860) 2024-03-11T19:20:16.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.922+0000 7f7b0dd9b640 1 -- 172.21.15.29:0/1172251231 learned_addr learned my addr 172.21.15.29:0/1172251231 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:16.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.923+0000 7f7b0e59c640 1 --2- 172.21.15.29:0/1172251231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bccc0 0x55ea849cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:16.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.923+0000 7f7b0dd9b640 1 -- 172.21.15.29:0/1172251231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bccc0 msgr2=0x55ea849cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:16.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.923+0000 7f7b0dd9b640 1 --2- 172.21.15.29:0/1172251231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bccc0 0x55ea849cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.923+0000 7f7b0dd9b640 1 -- 172.21.15.29:0/1172251231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 msgr2=0x55ea849cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:16.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.923+0000 7f7b0dd9b640 1 --2- 172.21.15.29:0/1172251231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 0x55ea849cd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.923+0000 7f7b0dd9b640 1 -- 172.21.15.29:0/1172251231 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ea848e3570 con 0x55ea849c1e30 2024-03-11T19:20:16.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.923+0000 7f7b0dd9b640 1 --2- 172.21.15.29:0/1172251231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 0x55ea849cc230 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x55ea849d5040 tx=0x7f7af8009640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9441df29903f221d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:16.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.923+0000 7f7af5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:20:16.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.924+0000 7f7af5ffb640 1 -- 172.21.15.29:0/1172251231 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8030070 con 0x55ea849c1e30 2024-03-11T19:20:16.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.924+0000 7f7af5ffb640 1 -- 172.21.15.29:0/1172251231 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7af802b330 con 0x55ea849c1e30 2024-03-11T19:20:16.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.924+0000 7f7b17a20740 1 -- 172.21.15.29:0/1172251231 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7af8019030 con 0x55ea849c1e30 2024-03-11T19:20:16.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.925+0000 7f7af5ffb640 1 -- 172.21.15.29:0/1172251231 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7af8025040 con 0x55ea849c1e30 2024-03-11T19:20:16.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.925+0000 7f7b17a20740 10 client.27373 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:16.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.925+0000 7f7b17a20740 1 -- 172.21.15.29:0/1172251231 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ea84923610 con 0x55ea849c1e30 2024-03-11T19:20:16.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.925+0000 7f7af5ffb640 1 -- 172.21.15.29:0/1172251231 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7af8019030 con 0x55ea849c1e30 2024-03-11T19:20:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.925+0000 7f7af5ffb640 1 -- 172.21.15.29:0/1172251231 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7af802b050 con 0x55ea849c1e30 2024-03-11T19:20:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.925+0000 7f7b17a20740 10 client.27373 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.925+0000 7f7b17a20740 10 client.27373 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:20:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.925+0000 7f7b17a20740 20 client.27373 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.925+0000 7f7b17a20740 1 --2- 172.21.15.29:0/1172251231 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ea84a2a020 0x55ea84a2c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.925+0000 7f7b17a20740 4 client.27373 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:20:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.925+0000 7f7b17a20740 1 -- 172.21.15.29:0/1172251231 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ea846b0890 con 0x55ea84a2a020 2024-03-11T19:20:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.926+0000 7f7b0e59c640 1 --2- 172.21.15.29:0/1172251231 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ea84a2a020 0x55ea84a2c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:16.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.926+0000 7f7b0e59c640 1 --2- 172.21.15.29:0/1172251231 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ea84a2a020 0x55ea84a2c440 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dafeb146a25a04a6 server_cookie=bd59c236aa571851 in_seq=0 out_seq=0 2024-03-11T19:20:16.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.926+0000 7f7af5ffb640 10 client.27373 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:16.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.927+0000 7f7af5ffb640 1 -- 172.21.15.29:0/1172251231 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ea846b0890 con 0x55ea84a2a020 2024-03-11T19:20:16.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.927+0000 7f7af5ffb640 10 client.27373 handle_command_reply: tid=0 2024-03-11T19:20:16.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.927+0000 7f7b17a20740 1 client.27373 shutdown 2024-03-11T19:20:16.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.928+0000 7f7b17a20740 1 -- 172.21.15.29:0/1172251231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 msgr2=0x55ea849cc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:16.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.928+0000 7f7b17a20740 1 --2- 172.21.15.29:0/1172251231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 0x55ea849cc230 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x55ea849d5040 tx=0x7f7af8009640 comp rx=0 tx=0).stop 2024-03-11T19:20:16.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.928+0000 7f7b17a20740 1 -- 172.21.15.29:0/1172251231 shutdown_connections 2024-03-11T19:20:16.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.928+0000 7f7b17a20740 1 --2- 172.21.15.29:0/1172251231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea849c1e30 0x55ea849cc230 secure :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0x55ea849d5040 tx=0x7f7af8009640 comp rx=0 tx=0).stop 2024-03-11T19:20:16.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.928+0000 7f7b17a20740 1 --2- 172.21.15.29:0/1172251231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ea849bccc0 0x55ea849cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.928+0000 7f7b17a20740 1 --2- 172.21.15.29:0/1172251231 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ea84a2a020 0x55ea84a2c440 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.928+0000 7f7b17a20740 1 --2- 172.21.15.29:0/1172251231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ea849bc2d0 0x55ea849cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.928+0000 7f7af5ffb640 0 client.27373 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:16.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.928+0000 7f7b17a20740 1 -- 172.21.15.29:0/1172251231 >> 172.21.15.29:0/1172251231 conn(0x55ea849b77f0 msgr2=0x55ea849d90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:16.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.928+0000 7f7b17a20740 1 -- 172.21.15.29:0/1172251231 shutdown_connections 2024-03-11T19:20:16.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.928+0000 7f7b17a20740 1 -- 172.21.15.29:0/1172251231 wait complete. 2024-03-11T19:20:16.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.928+0000 7f7b17a20740 20 client.27373 trim_cache size 0 max 16384 2024-03-11T19:20:16.933 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:20:16.933 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:20:16.933 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:20:16.933 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:20:16.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.932+0000 7f7b14e57640 1 -- 172.21.15.29:0/1868284942 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7afc0a1aa0 msgr2=0x7f7afc0a3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:16.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.932+0000 7f7b14e57640 1 --2- 172.21.15.29:0/1868284942 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7afc0a1aa0 0x7f7afc0a3f60 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7b00096ba0 tx=0x7f7b0000cf90 comp rx=0 tx=0).stop 2024-03-11T19:20:16.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.932+0000 7f7b14e57640 1 -- 172.21.15.29:0/1868284942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b10106de0 msgr2=0x7f7b1019c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:16.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.932+0000 7f7b14e57640 1 --2- 172.21.15.29:0/1868284942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b10106de0 0x7f7b1019c4a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7b0000e970 tx=0x7f7b0000ed70 comp rx=0 tx=0).stop 2024-03-11T19:20:16.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.932+0000 7f7b14e57640 1 -- 172.21.15.29:0/1868284942 shutdown_connections 2024-03-11T19:20:16.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.932+0000 7f7b14e57640 1 --2- 172.21.15.29:0/1868284942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b10106de0 0x7f7b1019c4a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.932+0000 7f7b14e57640 1 --2- 172.21.15.29:0/1868284942 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7afc0a1aa0 0x7f7afc0a3f60 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.933+0000 7f7b14e57640 1 --2- 172.21.15.29:0/1868284942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b100ff850 0x7f7b10195430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.933+0000 7f7b14e57640 1 --2- 172.21.15.29:0/1868284942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b100feef0 0x7f7b10194ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:16.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.933+0000 7f7b14e57640 1 -- 172.21.15.29:0/1868284942 >> 172.21.15.29:0/1868284942 conn(0x7f7b100fa960 msgr2=0x7f7b10109b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:16.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.933+0000 7f7b14e57640 1 -- 172.21.15.29:0/1868284942 shutdown_connections 2024-03-11T19:20:16.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:16.933+0000 7f7b14e57640 1 -- 172.21.15.29:0/1868284942 wait complete. 2024-03-11T19:20:16.956 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:16 smithi029 ceph-mon[32439]: pgmap v665: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 59 MiB/s wr, 1.15k op/s 2024-03-11T19:20:16.957 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:20:16.957 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:20:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.143+0000 7f49aac88640 1 -- 172.21.15.29:0/1277883472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49a41077c0 msgr2=0x7f49a4107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.143+0000 7f49aac88640 1 --2- 172.21.15.29:0/1277883472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49a41077c0 0x7f49a4107be0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f4994009be0 tx=0x7f499402bef0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.144+0000 7f49aac88640 1 -- 172.21.15.29:0/1277883472 shutdown_connections 2024-03-11T19:20:17.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.144+0000 7f49aac88640 1 --2- 172.21.15.29:0/1277883472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49a4109370 0x7f49a410fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.144+0000 7f49aac88640 1 --2- 172.21.15.29:0/1277883472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49a41089b0 0x7f49a4108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.145+0000 7f49aac88640 1 --2- 172.21.15.29:0/1277883472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49a41077c0 0x7f49a4107be0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.145+0000 7f49aac88640 1 -- 172.21.15.29:0/1277883472 >> 172.21.15.29:0/1277883472 conn(0x7f49a4076790 msgr2=0x7f49a4078bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:17.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.146+0000 7f49aac88640 1 -- 172.21.15.29:0/1277883472 shutdown_connections 2024-03-11T19:20:17.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.146+0000 7f49aac88640 1 -- 172.21.15.29:0/1277883472 wait complete. 2024-03-11T19:20:17.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.147+0000 7f49aac88640 1 Processor -- start 2024-03-11T19:20:17.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.147+0000 7f49aac88640 1 -- start start 2024-03-11T19:20:17.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.148+0000 7f49aac88640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49a41077c0 0x7f49a419d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.148+0000 7f49aac88640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49a41089b0 0x7f49a419dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.148+0000 7f49aac88640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49a4109370 0x7f49a41a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.148+0000 7f49aac88640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a41a5280 con 0x7f49a4109370 2024-03-11T19:20:17.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.148+0000 7f49aac88640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a4079a00 con 0x7f49a41089b0 2024-03-11T19:20:17.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.148+0000 7f49aac88640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a4079bc0 con 0x7f49a41077c0 2024-03-11T19:20:17.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.148+0000 7f49a8a24640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49a41077c0 0x7f49a419d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.148+0000 7f49a3fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49a41089b0 0x7f49a419dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.148+0000 7f49a3fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49a41089b0 0x7f49a419dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:36936/0 (socket says 172.21.15.29:36936) 2024-03-11T19:20:17.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.148+0000 7f49a3fff640 1 -- 172.21.15.29:0/3550584565 learned_addr learned my addr 172.21.15.29:0/3550584565 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:17.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.149+0000 7f49a3fff640 1 -- 172.21.15.29:0/3550584565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49a41077c0 msgr2=0x7f49a419d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:17.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.149+0000 7f49a3fff640 1 --2- 172.21.15.29:0/3550584565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49a41077c0 0x7f49a419d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.149+0000 7f49a3fff640 1 -- 172.21.15.29:0/3550584565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49a4109370 msgr2=0x7f49a41a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:17.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.149+0000 7f49a3fff640 1 --2- 172.21.15.29:0/3550584565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49a4109370 0x7f49a41a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.149+0000 7f49a3fff640 1 -- 172.21.15.29:0/3550584565 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498c009560 con 0x7f49a41089b0 2024-03-11T19:20:17.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.149+0000 7f49a3fff640 1 --2- 172.21.15.29:0/3550584565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49a41089b0 0x7f49a419dcd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f498c00e9c0 tx=0x7f498c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:17.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.150+0000 7f49a1ffb640 1 -- 172.21.15.29:0/3550584565 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c00cde0 con 0x7f49a41089b0 2024-03-11T19:20:17.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.150+0000 7f49a1ffb640 1 -- 172.21.15.29:0/3550584565 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f498c0044d0 con 0x7f49a41089b0 2024-03-11T19:20:17.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.150+0000 7f49a1ffb640 1 -- 172.21.15.29:0/3550584565 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c010430 con 0x7f49a41089b0 2024-03-11T19:20:17.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.151+0000 7f49aac88640 1 -- 172.21.15.29:0/3550584565 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499c009490 con 0x7f49a41089b0 2024-03-11T19:20:17.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.151+0000 7f49aac88640 1 -- 172.21.15.29:0/3550584565 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4994009840 con 0x7f49a41089b0 2024-03-11T19:20:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.153+0000 7f49aac88640 1 -- 172.21.15.29:0/3550584565 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a40676e0 con 0x7f49a41089b0 2024-03-11T19:20:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.155+0000 7f49a1ffb640 1 -- 172.21.15.29:0/3550584565 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f498c004000 con 0x7f49a41089b0 2024-03-11T19:20:17.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.156+0000 7f49a1ffb640 1 --2- 172.21.15.29:0/3550584565 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f49900a1c00 0x7f49900a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.156+0000 7f49a8a24640 1 --2- 172.21.15.29:0/3550584565 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f49900a1c00 0x7f49900a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.156+0000 7f49a1ffb640 1 -- 172.21.15.29:0/3550584565 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f498c014070 con 0x7f49a41089b0 2024-03-11T19:20:17.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.157+0000 7f49a8a24640 1 --2- 172.21.15.29:0/3550584565 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f49900a1c00 0x7f49900a40c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4994006010 tx=0x7f4994000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:17.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.163+0000 7f49a1ffb640 1 -- 172.21.15.29:0/3550584565 <== mon.2 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f498c096ab0 con 0x7f49a41089b0 2024-03-11T19:20:17.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:16 smithi190 ceph-mon[36292]: pgmap v665: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 59 MiB/s wr, 1.15k op/s 2024-03-11T19:20:17.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:16 smithi170 ceph-mon[35690]: pgmap v665: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 59 MiB/s wr, 1.15k op/s 2024-03-11T19:20:17.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.498+0000 7f49aac88640 1 -- 172.21.15.29:0/3550584565 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49a402ce40 con 0x7f49a41089b0 2024-03-11T19:20:17.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.499+0000 7f49a1ffb640 1 -- 172.21.15.29:0/3550584565 <== mon.2 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f498c09a170 con 0x7f49a41089b0 2024-03-11T19:20:17.506 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:20:17.506 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:20:17.507 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:20:17.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.502+0000 7f49aac88640 1 -- 172.21.15.29:0/3550584565 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f49900a1c00 msgr2=0x7f49900a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:17.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.502+0000 7f49aac88640 1 --2- 172.21.15.29:0/3550584565 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f49900a1c00 0x7f49900a40c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4994006010 tx=0x7f4994000f40 comp rx=0 tx=0).stop 2024-03-11T19:20:17.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.502+0000 7f49aac88640 1 -- 172.21.15.29:0/3550584565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49a41089b0 msgr2=0x7f49a419dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:17.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.502+0000 7f49aac88640 1 --2- 172.21.15.29:0/3550584565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49a41089b0 0x7f49a419dcd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f498c00e9c0 tx=0x7f498c00ee80 comp rx=0 tx=0).stop 2024-03-11T19:20:17.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.503+0000 7f49aac88640 1 -- 172.21.15.29:0/3550584565 shutdown_connections 2024-03-11T19:20:17.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.503+0000 7f49aac88640 1 --2- 172.21.15.29:0/3550584565 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f49900a1c00 0x7f49900a40c0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.503+0000 7f49aac88640 1 --2- 172.21.15.29:0/3550584565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49a4109370 0x7f49a41a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.503+0000 7f49aac88640 1 --2- 172.21.15.29:0/3550584565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49a41089b0 0x7f49a419dcd0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.503+0000 7f49aac88640 1 --2- 172.21.15.29:0/3550584565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49a41077c0 0x7f49a419d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.503+0000 7f49aac88640 1 -- 172.21.15.29:0/3550584565 >> 172.21.15.29:0/3550584565 conn(0x7f49a4076790 msgr2=0x7f49a410bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:17.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.503+0000 7f49aac88640 1 -- 172.21.15.29:0/3550584565 shutdown_connections 2024-03-11T19:20:17.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.503+0000 7f49aac88640 1 -- 172.21.15.29:0/3550584565 wait complete. 2024-03-11T19:20:17.526 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:20:17.526 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:20:17.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.715+0000 7f3ee6715640 1 -- 172.21.15.29:0/455347589 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ee01011e0 msgr2=0x7f3ee0107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:17.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.715+0000 7f3ee6715640 1 --2- 172.21.15.29:0/455347589 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ee01011e0 0x7f3ee0107a70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8009ca0 tx=0x7f3ed802bdd0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.715+0000 7f3ee6715640 1 -- 172.21.15.29:0/455347589 shutdown_connections 2024-03-11T19:20:17.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.715+0000 7f3ee6715640 1 --2- 172.21.15.29:0/455347589 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ee01011e0 0x7f3ee0107a70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.715+0000 7f3ee6715640 1 --2- 172.21.15.29:0/455347589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ee0100820 0x7f3ee0100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.715+0000 7f3ee6715640 1 --2- 172.21.15.29:0/455347589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ee00ff630 0x7f3ee00ffa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.715+0000 7f3ee6715640 1 -- 172.21.15.29:0/455347589 >> 172.21.15.29:0/455347589 conn(0x7f3ee00faba0 msgr2=0x7f3ee00fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:17.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.716+0000 7f3ee6715640 1 -- 172.21.15.29:0/455347589 shutdown_connections 2024-03-11T19:20:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.716+0000 7f3ee6715640 1 -- 172.21.15.29:0/455347589 wait complete. 2024-03-11T19:20:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.717+0000 7f3ee6715640 1 Processor -- start 2024-03-11T19:20:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.717+0000 7f3ee6715640 1 -- start start 2024-03-11T19:20:17.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.718+0000 7f3ee6715640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ee00ff630 0x7f3ee0194da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.718+0000 7f3ee6715640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ee0100820 0x7f3ee01952e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.718+0000 7f3ee6715640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ee01011e0 0x7f3ee019c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.718+0000 7f3ee4f12640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ee0100820 0x7f3ee01952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.718+0000 7f3ee4f12640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ee0100820 0x7f3ee01952e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:52226/0 (socket says 172.21.15.29:52226) 2024-03-11T19:20:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.718+0000 7f3ee4f12640 1 -- 172.21.15.29:0/2402179631 learned_addr learned my addr 172.21.15.29:0/2402179631 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.718+0000 7f3ee6715640 1 -- 172.21.15.29:0/2402179631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee019c880 con 0x7f3ee01011e0 2024-03-11T19:20:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.718+0000 7f3ee5f14640 1 --2- 172.21.15.29:0/2402179631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ee01011e0 0x7f3ee019c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.719+0000 7f3ee5713640 1 --2- 172.21.15.29:0/2402179631 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ee00ff630 0x7f3ee0194da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.719+0000 7f3ee6715640 1 -- 172.21.15.29:0/2402179631 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee019c9f0 con 0x7f3ee00ff630 2024-03-11T19:20:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.719+0000 7f3ee6715640 1 -- 172.21.15.29:0/2402179631 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee019cbb0 con 0x7f3ee0100820 2024-03-11T19:20:17.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.719+0000 7f3ee5f14640 1 -- 172.21.15.29:0/2402179631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ee0100820 msgr2=0x7f3ee01952e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:17.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.719+0000 7f3ee5f14640 1 --2- 172.21.15.29:0/2402179631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ee0100820 0x7f3ee01952e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.719+0000 7f3ee5f14640 1 -- 172.21.15.29:0/2402179631 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ee00ff630 msgr2=0x7f3ee0194da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:17.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.719+0000 7f3ee5f14640 1 --2- 172.21.15.29:0/2402179631 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ee00ff630 0x7f3ee0194da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.719+0000 7f3ee5f14640 1 -- 172.21.15.29:0/2402179631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed8009900 con 0x7f3ee01011e0 2024-03-11T19:20:17.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.720+0000 7f3ee4f12640 1 --2- 172.21.15.29:0/2402179631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ee0100820 0x7f3ee01952e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:20:17.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.720+0000 7f3ee5713640 1 --2- 172.21.15.29:0/2402179631 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ee00ff630 0x7f3ee0194da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:20:17.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.720+0000 7f3ee5f14640 1 --2- 172.21.15.29:0/2402179631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ee01011e0 0x7f3ee019c310 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f3ed802c310 tx=0x7f3ed8032b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:17.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.720+0000 7f3ed67fc640 1 -- 172.21.15.29:0/2402179631 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8034640 con 0x7f3ee01011e0 2024-03-11T19:20:17.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.720+0000 7f3ed67fc640 1 -- 172.21.15.29:0/2402179631 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3ed8034c50 con 0x7f3ee01011e0 2024-03-11T19:20:17.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.720+0000 7f3ed67fc640 1 -- 172.21.15.29:0/2402179631 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed803fd30 con 0x7f3ee01011e0 2024-03-11T19:20:17.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.720+0000 7f3ee6715640 1 -- 172.21.15.29:0/2402179631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee019ce30 con 0x7f3ee01011e0 2024-03-11T19:20:17.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.721+0000 7f3ee6715640 1 -- 172.21.15.29:0/2402179631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee019d1d0 con 0x7f3ee01011e0 2024-03-11T19:20:17.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.724+0000 7f3eea621740 1 Processor -- start 2024-03-11T19:20:17.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.725+0000 7f3ed67fc640 1 -- 172.21.15.29:0/2402179631 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3ed803b040 con 0x7f3ee01011e0 2024-03-11T19:20:17.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.725+0000 7f3eea621740 1 -- start start 2024-03-11T19:20:17.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.725+0000 7f3eea621740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594649fd280 0x5594649fd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.725+0000 7f3eea621740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594649fdc70 0x559464a02200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.725+0000 7f3eea621740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559464a02870 0x559464a04ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.725+0000 7f3eea621740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594649b0960 con 0x5594649fd280 2024-03-11T19:20:17.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.725+0000 7f3eea621740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559464964310 con 0x559464a02870 2024-03-11T19:20:17.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.725+0000 7f3eea621740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559464961840 con 0x5594649fdc70 2024-03-11T19:20:17.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.725+0000 7f3ee4f12640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594649fd280 0x5594649fd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.725+0000 7f3ee4f12640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594649fd280 0x5594649fd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60914/0 (socket says 172.21.15.29:60914) 2024-03-11T19:20:17.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.725+0000 7f3ee4f12640 1 -- 172.21.15.29:0/2665525800 learned_addr learned my addr 172.21.15.29:0/2665525800 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:17.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.725+0000 7f3ee4f12640 1 --2- 172.21.15.29:0/2665525800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559464a02870 0x559464a04ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.725+0000 7f3ee5713640 1 --2- 172.21.15.29:0/2665525800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594649fdc70 0x559464a02200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.726+0000 7f3ed67fc640 1 --2- 172.21.15.29:0/2402179631 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3ecc0a1ab0 0x7f3ecc0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.726+0000 7f3ee5713640 1 -- 172.21.15.29:0/2665525800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559464a02870 msgr2=0x559464a04ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:17.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.726+0000 7f3ee5713640 1 --2- 172.21.15.29:0/2665525800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559464a02870 0x559464a04ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.726+0000 7f3ee5713640 1 -- 172.21.15.29:0/2665525800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594649fd280 msgr2=0x5594649fd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:17.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.726+0000 7f3ee5713640 1 --2- 172.21.15.29:0/2665525800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594649fd280 0x5594649fd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.726+0000 7f3ee5713640 1 -- 172.21.15.29:0/2665525800 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55946491f060 con 0x5594649fdc70 2024-03-11T19:20:17.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.726+0000 7f3ed67fc640 1 -- 172.21.15.29:0/2402179631 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f3ed80efa10 con 0x7f3ee01011e0 2024-03-11T19:20:17.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.726+0000 7f3ee5713640 1 --2- 172.21.15.29:0/2665525800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594649fdc70 0x559464a02200 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0100680 tx=0x7f3edc00e5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19fbff1d76643355 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:17.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.726+0000 7f3ee5f14640 1 --2- 172.21.15.29:0/2402179631 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3ecc0a1ab0 0x7f3ecc0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.726+0000 7f3ecb7fe640 1 -- 172.21.15.29:0/2665525800 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc00eec0 con 0x5594649fdc70 2024-03-11T19:20:17.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.727+0000 7f3ecb7fe640 1 -- 172.21.15.29:0/2665525800 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3edc013920 con 0x5594649fdc70 2024-03-11T19:20:17.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.727+0000 7f3ecb7fe640 1 -- 172.21.15.29:0/2665525800 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc0127c0 con 0x5594649fdc70 2024-03-11T19:20:17.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.727+0000 7f3ee5f14640 1 --2- 172.21.15.29:0/2402179631 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3ecc0a1ab0 0x7f3ecc0a3f70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8037040 tx=0x7f3ed802bdd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:17.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.727+0000 7f3eea621740 1 -- 172.21.15.29:0/2665525800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594649fdc70 msgr2=0x559464a02200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:17.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.728+0000 7f3eea621740 1 --2- 172.21.15.29:0/2665525800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594649fdc70 0x559464a02200 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0100680 tx=0x7f3edc00e5a0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.728+0000 7f3eea621740 1 -- 172.21.15.29:0/2665525800 shutdown_connections 2024-03-11T19:20:17.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.728+0000 7f3eea621740 1 --2- 172.21.15.29:0/2665525800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559464a02870 0x559464a04ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.728+0000 7f3eea621740 1 --2- 172.21.15.29:0/2665525800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594649fdc70 0x559464a02200 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.728+0000 7f3eea621740 1 --2- 172.21.15.29:0/2665525800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594649fd280 0x5594649fd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.728+0000 7f3eea621740 1 -- 172.21.15.29:0/2665525800 >> 172.21.15.29:0/2665525800 conn(0x5594649ef970 msgr2=0x5594649efd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:17.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.728+0000 7f3eea621740 1 -- 172.21.15.29:0/2665525800 shutdown_connections 2024-03-11T19:20:17.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.728+0000 7f3eea621740 1 -- 172.21.15.29:0/2665525800 wait complete. 2024-03-11T19:20:17.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.728+0000 7f3eea621740 1 -- start start 2024-03-11T19:20:17.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.729+0000 7f3eea621740 1 Processor -- start 2024-03-11T19:20:17.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.729+0000 7f3eea621740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594649fd280 0x559464a06e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.729+0000 7f3eea621740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5594649fdc70 0x559464a07390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.729+0000 7f3eea621740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559464a02870 0x559464a0e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.729+0000 7f3eea621740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55946495d020 con 0x559464a02870 2024-03-11T19:20:17.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.729+0000 7f3eea621740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559464630430 con 0x5594649fdc70 2024-03-11T19:20:17.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.729+0000 7f3eea621740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594645ece00 con 0x5594649fd280 2024-03-11T19:20:17.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.729+0000 7f3ee4f12640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594649fd280 0x559464a06e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.729+0000 7f3eea621740 10 client.0 _ll_register_callbacks cb 0x559464904cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:17.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.729+0000 7f3ee4f12640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594649fd280 0x559464a06e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:52246/0 (socket says 172.21.15.29:52246) 2024-03-11T19:20:17.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.729+0000 7f3ee4f12640 1 -- 172.21.15.29:0/2120622137 learned_addr learned my addr 172.21.15.29:0/2120622137 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:17.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.729+0000 7f3ee4f12640 1 --2- 172.21.15.29:0/2120622137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559464a02870 0x559464a0e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.729+0000 7f3ee5713640 1 --2- 172.21.15.29:0/2120622137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5594649fdc70 0x559464a07390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.730+0000 7f3ee4f12640 1 -- 172.21.15.29:0/2120622137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5594649fdc70 msgr2=0x559464a07390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:17.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.730+0000 7f3ee4f12640 1 --2- 172.21.15.29:0/2120622137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5594649fdc70 0x559464a07390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.730+0000 7f3ee4f12640 1 -- 172.21.15.29:0/2120622137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559464a02870 msgr2=0x559464a0e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:17.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.730+0000 7f3ee4f12640 1 --2- 172.21.15.29:0/2120622137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559464a02870 0x559464a0e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.730+0000 7f3ee4f12640 1 -- 172.21.15.29:0/2120622137 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3edc0099e0 con 0x5594649fd280 2024-03-11T19:20:17.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.730+0000 7f3ee4f12640 1 --2- 172.21.15.29:0/2120622137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594649fd280 0x559464a06e50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3ed001cbc0 tx=0x7f3ed0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27e9d51a31a493c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:17.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.730+0000 7f3ec8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:20:17.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.730+0000 7f3ec8ff9640 1 -- 172.21.15.29:0/2120622137 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed001ae50 con 0x5594649fd280 2024-03-11T19:20:17.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.730+0000 7f3ec8ff9640 1 -- 172.21.15.29:0/2120622137 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f3ed0004050 con 0x5594649fd280 2024-03-11T19:20:17.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.730+0000 7f3eea621740 1 -- 172.21.15.29:0/2120622137 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55946465c7e0 con 0x5594649fd280 2024-03-11T19:20:17.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.730+0000 7f3ec8ff9640 1 -- 172.21.15.29:0/2120622137 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ed0004210 con 0x5594649fd280 2024-03-11T19:20:17.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.730+0000 7f3eea621740 10 client.27391 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:17.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.730+0000 7f3eea621740 1 -- 172.21.15.29:0/2120622137 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55946495b6a0 con 0x5594649fd280 2024-03-11T19:20:17.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.731+0000 7f3ec8ff9640 1 -- 172.21.15.29:0/2120622137 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3ed002d020 con 0x5594649fd280 2024-03-11T19:20:17.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.731+0000 7f3ec8ff9640 1 -- 172.21.15.29:0/2120622137 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3ed0004880 con 0x5594649fd280 2024-03-11T19:20:17.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.731+0000 7f3eea621740 10 client.27391 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:17.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.731+0000 7f3eea621740 10 client.27391 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:20:17.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.731+0000 7f3eea621740 20 client.27391 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:17.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.731+0000 7f3eea621740 1 --2- 172.21.15.29:0/2120622137 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x559464a12fa0 0x559464a15390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.731+0000 7f3eea621740 4 client.27391 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:20:17.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.731+0000 7f3eea621740 1 -- 172.21.15.29:0/2120622137 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559464699720 con 0x559464a12fa0 2024-03-11T19:20:17.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.731+0000 7f3ee5713640 1 --2- 172.21.15.29:0/2120622137 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x559464a12fa0 0x559464a15390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.732+0000 7f3ec8ff9640 10 client.27391 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:20:17.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.732+0000 7f3ee5713640 1 --2- 172.21.15.29:0/2120622137 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x559464a12fa0 0x559464a15390 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bf78fc0441f1f865 server_cookie=7008640d2da14deb in_seq=0 out_seq=0 2024-03-11T19:20:17.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.733+0000 7f3ec8ff9640 1 -- 172.21.15.29:0/2120622137 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559464699720 con 0x559464a12fa0 2024-03-11T19:20:17.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.733+0000 7f3ec8ff9640 10 client.27391 handle_command_reply: tid=0 2024-03-11T19:20:17.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.733+0000 7f3eea621740 1 client.27391 shutdown 2024-03-11T19:20:17.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.733+0000 7f3eea621740 1 -- 172.21.15.29:0/2120622137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594649fd280 msgr2=0x559464a06e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:17.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.733+0000 7f3eea621740 1 --2- 172.21.15.29:0/2120622137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594649fd280 0x559464a06e50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3ed001cbc0 tx=0x7f3ed0004730 comp rx=0 tx=0).stop 2024-03-11T19:20:17.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.733+0000 7f3eea621740 1 -- 172.21.15.29:0/2120622137 shutdown_connections 2024-03-11T19:20:17.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.733+0000 7f3eea621740 1 --2- 172.21.15.29:0/2120622137 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x559464a12fa0 0x559464a15390 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.733+0000 7f3eea621740 1 --2- 172.21.15.29:0/2120622137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559464a02870 0x559464a0e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.733+0000 7f3eea621740 1 --2- 172.21.15.29:0/2120622137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5594649fdc70 0x559464a07390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.733+0000 7f3eea621740 1 --2- 172.21.15.29:0/2120622137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594649fd280 0x559464a06e50 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.733+0000 7f3eea621740 1 -- 172.21.15.29:0/2120622137 >> 172.21.15.29:0/2120622137 conn(0x5594649ef970 msgr2=0x5594649fc910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:17.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.733+0000 7f3ec8ff9640 0 client.27391 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:20:17.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.733+0000 7f3eea621740 1 -- 172.21.15.29:0/2120622137 shutdown_connections 2024-03-11T19:20:17.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.734+0000 7f3eea621740 1 -- 172.21.15.29:0/2120622137 wait complete. 2024-03-11T19:20:17.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.734+0000 7f3eea621740 20 client.27391 trim_cache size 0 max 16384 2024-03-11T19:20:17.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.767+0000 7f3eea621740 1 Processor -- start 2024-03-11T19:20:17.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.771+0000 7f3eea621740 1 -- start start 2024-03-11T19:20:17.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.771+0000 7f3eea621740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594649fd280 0x5594649feb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.771+0000 7f3eea621740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594649fdc70 0x5594649ff0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.771+0000 7f3eea621740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559464a02870 0x559464a0e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.771+0000 7f3eea621740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594648e2130 con 0x5594649fd280 2024-03-11T19:20:17.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.771+0000 7f3eea621740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559464a053a0 con 0x559464a02870 2024-03-11T19:20:17.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.771+0000 7f3eea621740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594649eb1d0 con 0x5594649fdc70 2024-03-11T19:20:17.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.771+0000 7f3ee4f12640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559464a02870 0x559464a0e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.771+0000 7f3ee4f12640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559464a02870 0x559464a0e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:36988/0 (socket says 172.21.15.29:36988) 2024-03-11T19:20:17.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.771+0000 7f3ee4f12640 1 -- 172.21.15.29:0/2854854716 learned_addr learned my addr 172.21.15.29:0/2854854716 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:17.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.772+0000 7f3ee5713640 1 --2- 172.21.15.29:0/2854854716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594649fdc70 0x5594649ff0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.772+0000 7f3ee4f12640 1 --2- 172.21.15.29:0/2854854716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594649fd280 0x5594649feb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.772+0000 7f3ee5713640 1 -- 172.21.15.29:0/2854854716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559464a02870 msgr2=0x559464a0e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:17.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.772+0000 7f3ee5713640 1 --2- 172.21.15.29:0/2854854716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559464a02870 0x559464a0e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.772+0000 7f3ee5713640 1 -- 172.21.15.29:0/2854854716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594649fd280 msgr2=0x5594649feb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:17.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.772+0000 7f3ee5713640 1 --2- 172.21.15.29:0/2854854716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594649fd280 0x5594649feb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.772+0000 7f3ee5713640 1 -- 172.21.15.29:0/2854854716 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594648f7c20 con 0x5594649fdc70 2024-03-11T19:20:17.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.773+0000 7f3ee5713640 1 --2- 172.21.15.29:0/2854854716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594649fdc70 0x5594649ff0a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x5594649f08e0 tx=0x7f3edc009110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b7cc3a8aa311c3c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:17.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.773+0000 7f3ecaffd640 1 -- 172.21.15.29:0/2854854716 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc009880 con 0x5594649fdc70 2024-03-11T19:20:17.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.774+0000 7f3ecaffd640 1 -- 172.21.15.29:0/2854854716 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3edc013650 con 0x5594649fdc70 2024-03-11T19:20:17.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.774+0000 7f3ecaffd640 1 -- 172.21.15.29:0/2854854716 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc025610 con 0x5594649fdc70 2024-03-11T19:20:17.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.775+0000 7f3eea621740 1 -- 172.21.15.29:0/2854854716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594649fdc70 msgr2=0x5594649ff0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:17.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.775+0000 7f3eea621740 1 --2- 172.21.15.29:0/2854854716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594649fdc70 0x5594649ff0a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x5594649f08e0 tx=0x7f3edc009110 comp rx=0 tx=0).stop 2024-03-11T19:20:17.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.776+0000 7f3eea621740 1 -- 172.21.15.29:0/2854854716 shutdown_connections 2024-03-11T19:20:17.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.776+0000 7f3eea621740 1 --2- 172.21.15.29:0/2854854716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559464a02870 0x559464a0e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.776+0000 7f3eea621740 1 --2- 172.21.15.29:0/2854854716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594649fdc70 0x5594649ff0a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.776+0000 7f3eea621740 1 --2- 172.21.15.29:0/2854854716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594649fd280 0x5594649feb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.776+0000 7f3eea621740 1 -- 172.21.15.29:0/2854854716 >> 172.21.15.29:0/2854854716 conn(0x5594649ef970 msgr2=0x559464a19d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:17.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.777+0000 7f3eea621740 1 -- 172.21.15.29:0/2854854716 shutdown_connections 2024-03-11T19:20:17.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.777+0000 7f3eea621740 1 -- 172.21.15.29:0/2854854716 wait complete. 2024-03-11T19:20:17.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.777+0000 7f3eea621740 1 -- start start 2024-03-11T19:20:17.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.777+0000 7f3eea621740 1 Processor -- start 2024-03-11T19:20:17.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.778+0000 7f3eea621740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5594649fd280 0x559464a0d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.778+0000 7f3eea621740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594649fdc70 0x559464a0dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.778+0000 7f3eea621740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559464a02870 0x5594649f9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.778+0000 7f3eea621740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594649fa2f0 con 0x5594649fdc70 2024-03-11T19:20:17.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.778+0000 7f3eea621740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594649fa460 con 0x5594649fd280 2024-03-11T19:20:17.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.778+0000 7f3eea621740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594649fa620 con 0x559464a02870 2024-03-11T19:20:17.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.778+0000 7f3eea621740 10 client.0 _ll_register_callbacks cb 0x7f3edc0097d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:17.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.778+0000 7f3ee5713640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594649fdc70 0x559464a0dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.778+0000 7f3ee5713640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594649fdc70 0x559464a0dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60936/0 (socket says 172.21.15.29:60936) 2024-03-11T19:20:17.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.778+0000 7f3ee5713640 1 -- 172.21.15.29:0/459460194 learned_addr learned my addr 172.21.15.29:0/459460194 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:17.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.778+0000 7f3ee4f12640 1 --2- 172.21.15.29:0/459460194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559464a02870 0x5594649f9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.778+0000 7f3ee4f12640 1 --2- 172.21.15.29:0/459460194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5594649fd280 0x559464a0d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.778+0000 7f3ee5713640 1 -- 172.21.15.29:0/459460194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559464a02870 msgr2=0x5594649f9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:17.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.778+0000 7f3ee5713640 1 --2- 172.21.15.29:0/459460194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559464a02870 0x5594649f9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.778+0000 7f3ee5713640 1 -- 172.21.15.29:0/459460194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5594649fd280 msgr2=0x559464a0d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:17.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.778+0000 7f3ee5713640 1 --2- 172.21.15.29:0/459460194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5594649fd280 0x559464a0d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.778+0000 7f3ee5713640 1 -- 172.21.15.29:0/459460194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55946491f060 con 0x5594649fdc70 2024-03-11T19:20:17.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.778+0000 7f3ee4f12640 1 --2- 172.21.15.29:0/459460194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559464a02870 0x5594649f9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:20:17.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.779+0000 7f3ee5713640 1 --2- 172.21.15.29:0/459460194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594649fdc70 0x559464a0dee0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f3edc009cd0 tx=0x7f3edc0047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49e63dae2e506d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:17.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.779+0000 7f3ec8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:20:17.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.779+0000 7f3ec8ff9640 1 -- 172.21.15.29:0/459460194 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc025de0 con 0x5594649fdc70 2024-03-11T19:20:17.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.779+0000 7f3ec8ff9640 1 -- 172.21.15.29:0/459460194 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f3edc022070 con 0x5594649fdc70 2024-03-11T19:20:17.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.779+0000 7f3eea621740 1 -- 172.21.15.29:0/459460194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3edc013920 con 0x5594649fdc70 2024-03-11T19:20:17.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.779+0000 7f3ec8ff9640 1 -- 172.21.15.29:0/459460194 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3edc005e20 con 0x5594649fdc70 2024-03-11T19:20:17.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.779+0000 7f3eea621740 10 client.17844 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:17.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.779+0000 7f3eea621740 1 -- 172.21.15.29:0/459460194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55946491d8c0 con 0x5594649fdc70 2024-03-11T19:20:17.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.780+0000 7f3ec8ff9640 1 -- 172.21.15.29:0/459460194 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3edc013920 con 0x5594649fdc70 2024-03-11T19:20:17.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.780+0000 7f3ec8ff9640 1 -- 172.21.15.29:0/459460194 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3edc024070 con 0x5594649fdc70 2024-03-11T19:20:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.780+0000 7f3eea621740 10 client.17844 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.780+0000 7f3eea621740 10 client.17844 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:20:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.780+0000 7f3eea621740 20 client.17844 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.780+0000 7f3eea621740 1 --2- 172.21.15.29:0/459460194 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5594649fbc10 0x5594649fc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.780+0000 7f3eea621740 4 client.17844 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:20:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.780+0000 7f3eea621740 1 -- 172.21.15.29:0/459460194 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5594646e8890 con 0x5594649fbc10 2024-03-11T19:20:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.780+0000 7f3ee4f12640 1 --2- 172.21.15.29:0/459460194 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5594649fbc10 0x5594649fc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.781+0000 7f3ee4f12640 1 --2- 172.21.15.29:0/459460194 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5594649fbc10 0x5594649fc020 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=db664abe43d4881b server_cookie=612c977de86aaec in_seq=0 out_seq=0 2024-03-11T19:20:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.781+0000 7f3ec8ff9640 10 client.17844 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:20:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.781+0000 7f3ec8ff9640 1 -- 172.21.15.29:0/459460194 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5594646e8890 con 0x5594649fbc10 2024-03-11T19:20:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.781+0000 7f3ec8ff9640 10 client.17844 handle_command_reply: tid=0 2024-03-11T19:20:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.781+0000 7f3eea621740 1 client.17844 shutdown 2024-03-11T19:20:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.781+0000 7f3eea621740 1 -- 172.21.15.29:0/459460194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594649fdc70 msgr2=0x559464a0dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:17.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.781+0000 7f3eea621740 1 --2- 172.21.15.29:0/459460194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594649fdc70 0x559464a0dee0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f3edc009cd0 tx=0x7f3edc0047d0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.781+0000 7f3eea621740 1 -- 172.21.15.29:0/459460194 shutdown_connections 2024-03-11T19:20:17.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.781+0000 7f3eea621740 1 --2- 172.21.15.29:0/459460194 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x5594649fbc10 0x5594649fc020 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.781+0000 7f3eea621740 1 --2- 172.21.15.29:0/459460194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559464a02870 0x5594649f9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.781+0000 7f3eea621740 1 --2- 172.21.15.29:0/459460194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594649fdc70 0x559464a0dee0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.781+0000 7f3eea621740 1 --2- 172.21.15.29:0/459460194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5594649fd280 0x559464a0d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.781+0000 7f3eea621740 1 -- 172.21.15.29:0/459460194 >> 172.21.15.29:0/459460194 conn(0x5594649ef970 msgr2=0x5594649feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:17.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.781+0000 7f3ec8ff9640 0 client.17844 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:20:17.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.782+0000 7f3eea621740 1 -- 172.21.15.29:0/459460194 shutdown_connections 2024-03-11T19:20:17.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.782+0000 7f3eea621740 1 -- 172.21.15.29:0/459460194 wait complete. 2024-03-11T19:20:17.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.782+0000 7f3eea621740 20 client.17844 trim_cache size 0 max 16384 2024-03-11T19:20:17.788 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:20:17.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.782+0000 7f3ee6715640 1 -- 172.21.15.29:0/2402179631 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3ecc0a1ab0 msgr2=0x7f3ecc0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:17.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.782+0000 7f3ee6715640 1 --2- 172.21.15.29:0/2402179631 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3ecc0a1ab0 0x7f3ecc0a3f70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8037040 tx=0x7f3ed802bdd0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.783+0000 7f3ee6715640 1 -- 172.21.15.29:0/2402179631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ee01011e0 msgr2=0x7f3ee019c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:17.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.783+0000 7f3ee6715640 1 --2- 172.21.15.29:0/2402179631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ee01011e0 0x7f3ee019c310 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f3ed802c310 tx=0x7f3ed8032b80 comp rx=0 tx=0).stop 2024-03-11T19:20:17.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.783+0000 7f3ee6715640 1 -- 172.21.15.29:0/2402179631 shutdown_connections 2024-03-11T19:20:17.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.783+0000 7f3ee6715640 1 --2- 172.21.15.29:0/2402179631 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f3ecc0a1ab0 0x7f3ecc0a3f70 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.783+0000 7f3ee6715640 1 --2- 172.21.15.29:0/2402179631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ee01011e0 0x7f3ee019c310 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.783+0000 7f3ee6715640 1 --2- 172.21.15.29:0/2402179631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ee0100820 0x7f3ee01952e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.783+0000 7f3ee6715640 1 --2- 172.21.15.29:0/2402179631 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ee00ff630 0x7f3ee0194da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:17.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.783+0000 7f3ee6715640 1 -- 172.21.15.29:0/2402179631 >> 172.21.15.29:0/2402179631 conn(0x7f3ee00faba0 msgr2=0x7f3ee0103b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:17.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.783+0000 7f3ee6715640 1 -- 172.21.15.29:0/2402179631 shutdown_connections 2024-03-11T19:20:17.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.783+0000 7f3ee6715640 1 -- 172.21.15.29:0/2402179631 wait complete. 2024-03-11T19:20:17.807 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:20:17.807 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:20:17.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.995+0000 7f0e7d27d640 1 -- 172.21.15.29:0/2310198873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e78109230 msgr2=0x7f0e7810fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:17.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.995+0000 7f0e7d27d640 1 --2- 172.21.15.29:0/2310198873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e78109230 0x7f0e7810fac0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0e70009be0 tx=0x7f0e7002bef0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.995+0000 7f0e7d27d640 1 -- 172.21.15.29:0/2310198873 shutdown_connections 2024-03-11T19:20:18.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.995+0000 7f0e7d27d640 1 --2- 172.21.15.29:0/2310198873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e78109230 0x7f0e7810fac0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.995+0000 7f0e7d27d640 1 --2- 172.21.15.29:0/2310198873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e78108870 0x7f0e78108cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.995+0000 7f0e7d27d640 1 --2- 172.21.15.29:0/2310198873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e78107680 0x7f0e78107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.995+0000 7f0e7d27d640 1 -- 172.21.15.29:0/2310198873 >> 172.21.15.29:0/2310198873 conn(0x7f0e78076620 msgr2=0x7f0e78078a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.996+0000 7f0e7d27d640 1 -- 172.21.15.29:0/2310198873 shutdown_connections 2024-03-11T19:20:18.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.996+0000 7f0e7d27d640 1 -- 172.21.15.29:0/2310198873 wait complete. 2024-03-11T19:20:18.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.996+0000 7f0e7d27d640 1 Processor -- start 2024-03-11T19:20:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.996+0000 7f0e7d27d640 1 -- start start 2024-03-11T19:20:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.997+0000 7f0e7d27d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e78107680 0x7f0e7819d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.997+0000 7f0e7d27d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e78108870 0x7f0e7819db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.997+0000 7f0e7d27d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e78109230 0x7f0e781a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.997+0000 7f0e7d27d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e781a5110 con 0x7f0e78107680 2024-03-11T19:20:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.997+0000 7f0e7d27d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e781a5280 con 0x7f0e78109230 2024-03-11T19:20:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.997+0000 7f0e7d27d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e781a5440 con 0x7f0e78108870 2024-03-11T19:20:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.997+0000 7f0e77fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e78107680 0x7f0e7819d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.997+0000 7f0e77fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e78107680 0x7f0e7819d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60950/0 (socket says 172.21.15.29:60950) 2024-03-11T19:20:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.997+0000 7f0e77fff640 1 -- 172.21.15.29:0/454653277 learned_addr learned my addr 172.21.15.29:0/454653277 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.997+0000 7f0e777fe640 1 --2- 172.21.15.29:0/454653277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e78108870 0x7f0e7819db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.997+0000 7f0e7ca7c640 1 --2- 172.21.15.29:0/454653277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e78109230 0x7f0e781a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.997+0000 7f0e77fff640 1 -- 172.21.15.29:0/454653277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e78108870 msgr2=0x7f0e7819db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.997+0000 7f0e77fff640 1 --2- 172.21.15.29:0/454653277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e78108870 0x7f0e7819db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.997+0000 7f0e77fff640 1 -- 172.21.15.29:0/454653277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e78109230 msgr2=0x7f0e781a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.997+0000 7f0e77fff640 1 --2- 172.21.15.29:0/454653277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e78109230 0x7f0e781a4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.997+0000 7f0e77fff640 1 -- 172.21.15.29:0/454653277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e70009840 con 0x7f0e78107680 2024-03-11T19:20:18.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.997+0000 7f0e777fe640 1 --2- 172.21.15.29:0/454653277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e78108870 0x7f0e7819db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:20:18.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.998+0000 7f0e77fff640 1 --2- 172.21.15.29:0/454653277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e78107680 0x7f0e7819d620 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800e9f0 tx=0x7f0e6800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.998+0000 7f0e757fa640 1 -- 172.21.15.29:0/454653277 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6800ccc0 con 0x7f0e78107680 2024-03-11T19:20:18.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.998+0000 7f0e757fa640 1 -- 172.21.15.29:0/454653277 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0e680044d0 con 0x7f0e78107680 2024-03-11T19:20:18.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.998+0000 7f0e7d27d640 1 -- 172.21.15.29:0/454653277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e781a5720 con 0x7f0e78107680 2024-03-11T19:20:18.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.998+0000 7f0e757fa640 1 -- 172.21.15.29:0/454653277 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e68003de0 con 0x7f0e78107680 2024-03-11T19:20:18.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:17.998+0000 7f0e7d27d640 1 -- 172.21.15.29:0/454653277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e781a5cb0 con 0x7f0e78107680 2024-03-11T19:20:18.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.001+0000 7f0e757fa640 1 -- 172.21.15.29:0/454653277 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0e68003f40 con 0x7f0e78107680 2024-03-11T19:20:18.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.002+0000 7f0e757fa640 1 --2- 172.21.15.29:0/454653277 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0e600a1bb0 0x7f0e600a4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.002+0000 7f0e81193740 1 Processor -- start 2024-03-11T19:20:18.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.002+0000 7f0e757fa640 1 -- 172.21.15.29:0/454653277 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0e68014070 con 0x7f0e78107680 2024-03-11T19:20:18.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.003+0000 7f0e81193740 1 -- start start 2024-03-11T19:20:18.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.003+0000 7f0e777fe640 1 --2- 172.21.15.29:0/454653277 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0e600a1bb0 0x7f0e600a4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.003+0000 7f0e81193740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6acce480 0x55cb6acce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.003+0000 7f0e81193740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6accee70 0x55cb6acd3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.003+0000 7f0e81193740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6acd3fe0 0x55cb6acd6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.003+0000 7f0e81193740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb6ac8a960 con 0x55cb6acd3fe0 2024-03-11T19:20:18.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.003+0000 7f0e81193740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb6ac3e310 con 0x55cb6acce480 2024-03-11T19:20:18.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.003+0000 7f0e81193740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb6ac3b840 con 0x55cb6accee70 2024-03-11T19:20:18.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.003+0000 7f0e77fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6acce480 0x55cb6acce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.003+0000 7f0e777fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6accee70 0x55cb6acd3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.003+0000 7f0e77fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6acce480 0x55cb6acce8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37022/0 (socket says 172.21.15.29:37022) 2024-03-11T19:20:18.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.003+0000 7f0e77fff640 1 -- 172.21.15.29:0/480197796 learned_addr learned my addr 172.21.15.29:0/480197796 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.003+0000 7f0e7ca7c640 1 --2- 172.21.15.29:0/480197796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6acd3fe0 0x55cb6acd6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.004+0000 7f0e77fff640 1 -- 172.21.15.29:0/480197796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6accee70 msgr2=0x55cb6acd3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.004+0000 7f0e77fff640 1 --2- 172.21.15.29:0/480197796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6accee70 0x55cb6acd3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.004+0000 7f0e77fff640 1 -- 172.21.15.29:0/480197796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6acd3fe0 msgr2=0x55cb6acd6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.004+0000 7f0e77fff640 1 --2- 172.21.15.29:0/480197796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6acd3fe0 0x55cb6acd6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.004+0000 7f0e77fff640 1 -- 172.21.15.29:0/480197796 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cb6abf9060 con 0x55cb6acce480 2024-03-11T19:20:18.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.005+0000 7f0e777fe640 1 --2- 172.21.15.29:0/454653277 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0e600a1bb0 0x7f0e600a4070 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0e7819eb30 tx=0x7f0e6c012640 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.005+0000 7f0e77fff640 1 --2- 172.21.15.29:0/480197796 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6acce480 0x55cb6acce8a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0e68096900 tx=0x7f0e68096140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2f7dd03f1c26875 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.005+0000 7f0e5e7fc640 1 -- 172.21.15.29:0/480197796 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6809aba0 con 0x55cb6acce480 2024-03-11T19:20:18.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.005+0000 7f0e5e7fc640 1 -- 172.21.15.29:0/480197796 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0e6800ce20 con 0x55cb6acce480 2024-03-11T19:20:18.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.005+0000 7f0e5e7fc640 1 -- 172.21.15.29:0/480197796 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e680041f0 con 0x55cb6acce480 2024-03-11T19:20:18.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.008+0000 7f0e81193740 1 -- 172.21.15.29:0/480197796 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6acce480 msgr2=0x55cb6acce8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.008+0000 7f0e81193740 1 --2- 172.21.15.29:0/480197796 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6acce480 0x55cb6acce8a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0e68096900 tx=0x7f0e68096140 comp rx=0 tx=0).stop 2024-03-11T19:20:18.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.010+0000 7f0e81193740 1 -- 172.21.15.29:0/480197796 shutdown_connections 2024-03-11T19:20:18.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.010+0000 7f0e81193740 1 --2- 172.21.15.29:0/480197796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6acd3fe0 0x55cb6acd6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.010+0000 7f0e81193740 1 --2- 172.21.15.29:0/480197796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6accee70 0x55cb6acd3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.010+0000 7f0e81193740 1 --2- 172.21.15.29:0/480197796 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6acce480 0x55cb6acce8a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.010+0000 7f0e81193740 1 -- 172.21.15.29:0/480197796 >> 172.21.15.29:0/480197796 conn(0x55cb6acc99f0 msgr2=0x55cb6acc9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.014+0000 7f0e81193740 1 -- 172.21.15.29:0/480197796 shutdown_connections 2024-03-11T19:20:18.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.020+0000 7f0e81193740 1 -- 172.21.15.29:0/480197796 wait complete. 2024-03-11T19:20:18.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.020+0000 7f0e81193740 1 -- start start 2024-03-11T19:20:18.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.020+0000 7f0e81193740 1 Processor -- start 2024-03-11T19:20:18.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.021+0000 7f0e81193740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6acce480 0x55cb6acd86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.021+0000 7f0e81193740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6accee70 0x55cb6acd8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.021+0000 7f0e81193740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6acd3fe0 0x55cb6acdfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.021+0000 7f0e81193740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb6ac37020 con 0x55cb6acd3fe0 2024-03-11T19:20:18.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.021+0000 7f0e81193740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb6a90a430 con 0x55cb6accee70 2024-03-11T19:20:18.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.021+0000 7f0e81193740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb6a8c6e00 con 0x55cb6acce480 2024-03-11T19:20:18.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.021+0000 7f0e81193740 10 client.0 _ll_register_callbacks cb 0x55cb6abdecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:18.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.021+0000 7f0e77fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6acce480 0x55cb6acd86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.021+0000 7f0e77fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6acce480 0x55cb6acd86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:52306/0 (socket says 172.21.15.29:52306) 2024-03-11T19:20:18.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.021+0000 7f0e77fff640 1 -- 172.21.15.29:0/160390372 learned_addr learned my addr 172.21.15.29:0/160390372 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.022+0000 7f0e7ca7c640 1 --2- 172.21.15.29:0/160390372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6acd3fe0 0x55cb6acdfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.025+0000 7f0e777fe640 1 --2- 172.21.15.29:0/160390372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6accee70 0x55cb6acd8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.026+0000 7f0e77fff640 1 -- 172.21.15.29:0/160390372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6accee70 msgr2=0x55cb6acd8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.026+0000 7f0e77fff640 1 --2- 172.21.15.29:0/160390372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6accee70 0x55cb6acd8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.026+0000 7f0e77fff640 1 -- 172.21.15.29:0/160390372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6acd3fe0 msgr2=0x55cb6acdfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.026+0000 7f0e77fff640 1 --2- 172.21.15.29:0/160390372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6acd3fe0 0x55cb6acdfbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.026+0000 7f0e77fff640 1 -- 172.21.15.29:0/160390372 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e781a5cb0 con 0x55cb6acce480 2024-03-11T19:20:18.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.026+0000 7f0e777fe640 1 --2- 172.21.15.29:0/160390372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6accee70 0x55cb6acd8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:20:18.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.026+0000 7f0e77fff640 1 --2- 172.21.15.29:0/160390372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6acce480 0x55cb6acd86c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0e68009e50 tx=0x7f0e680037a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfc6b8eba0150a20 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.028+0000 7f0e577fe640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:20:18.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.028+0000 7f0e577fe640 1 -- 172.21.15.29:0/160390372 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6809a4a0 con 0x55cb6acce480 2024-03-11T19:20:18.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.028+0000 7f0e577fe640 1 -- 172.21.15.29:0/160390372 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0e6809e5f0 con 0x55cb6acce480 2024-03-11T19:20:18.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.028+0000 7f0e81193740 1 -- 172.21.15.29:0/160390372 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cb6a9367e0 con 0x55cb6acce480 2024-03-11T19:20:18.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.029+0000 7f0e577fe640 1 -- 172.21.15.29:0/160390372 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e680041f0 con 0x55cb6acce480 2024-03-11T19:20:18.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.029+0000 7f0e81193740 10 client.27409 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:18.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.030+0000 7f0e81193740 1 -- 172.21.15.29:0/160390372 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55cb6ac356a0 con 0x55cb6acce480 2024-03-11T19:20:18.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.031+0000 7f0e577fe640 1 -- 172.21.15.29:0/160390372 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55cb6a9367e0 con 0x55cb6acce480 2024-03-11T19:20:18.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.031+0000 7f0e577fe640 1 -- 172.21.15.29:0/160390372 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0e6801f060 con 0x55cb6acce480 2024-03-11T19:20:18.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.031+0000 7f0e81193740 10 client.27409 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:18.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.031+0000 7f0e81193740 10 client.27409 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:20:18.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.031+0000 7f0e81193740 20 client.27409 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:18.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.031+0000 7f0e81193740 1 --2- 172.21.15.29:0/160390372 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55cb6acf5410 0x55cb6acf7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.031+0000 7f0e81193740 4 client.27409 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:20:18.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.031+0000 7f0e81193740 1 -- 172.21.15.29:0/160390372 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cb6a973720 con 0x55cb6acf5410 2024-03-11T19:20:18.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.032+0000 7f0e777fe640 1 --2- 172.21.15.29:0/160390372 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55cb6acf5410 0x55cb6acf7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.032+0000 7f0e577fe640 10 client.27409 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:20:18.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.033+0000 7f0e777fe640 1 --2- 172.21.15.29:0/160390372 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55cb6acf5410 0x55cb6acf7830 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8638c528594ecac8 server_cookie=f034e6a081539b1 in_seq=0 out_seq=0 2024-03-11T19:20:18.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.034+0000 7f0e577fe640 1 -- 172.21.15.29:0/160390372 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cb6a973720 con 0x55cb6acf5410 2024-03-11T19:20:18.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.034+0000 7f0e577fe640 10 client.27409 handle_command_reply: tid=0 2024-03-11T19:20:18.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.034+0000 7f0e81193740 1 client.27409 shutdown 2024-03-11T19:20:18.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.035+0000 7f0e81193740 1 -- 172.21.15.29:0/160390372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6acce480 msgr2=0x55cb6acd86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.035+0000 7f0e81193740 1 --2- 172.21.15.29:0/160390372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6acce480 0x55cb6acd86c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0e68009e50 tx=0x7f0e680037a0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.035+0000 7f0e81193740 1 -- 172.21.15.29:0/160390372 shutdown_connections 2024-03-11T19:20:18.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.035+0000 7f0e81193740 1 --2- 172.21.15.29:0/160390372 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55cb6acf5410 0x55cb6acf7830 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.035+0000 7f0e81193740 1 --2- 172.21.15.29:0/160390372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6acd3fe0 0x55cb6acdfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.035+0000 7f0e81193740 1 --2- 172.21.15.29:0/160390372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6accee70 0x55cb6acd8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.035+0000 7f0e81193740 1 --2- 172.21.15.29:0/160390372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6acce480 0x55cb6acd86c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.035+0000 7f0e81193740 1 -- 172.21.15.29:0/160390372 >> 172.21.15.29:0/160390372 conn(0x55cb6acc99f0 msgr2=0x55cb6acc8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.035+0000 7f0e577fe640 0 client.27409 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:20:18.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.035+0000 7f0e81193740 1 -- 172.21.15.29:0/160390372 shutdown_connections 2024-03-11T19:20:18.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.035+0000 7f0e81193740 1 -- 172.21.15.29:0/160390372 wait complete. 2024-03-11T19:20:18.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.036+0000 7f0e81193740 20 client.27409 trim_cache size 0 max 16384 2024-03-11T19:20:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.056+0000 7f0e81193740 1 Processor -- start 2024-03-11T19:20:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.056+0000 7f0e81193740 1 -- start start 2024-03-11T19:20:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.057+0000 7f0e81193740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6acce480 0x55cb6acfc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.057+0000 7f0e81193740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6accee70 0x55cb6acdfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.057+0000 7f0e81193740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6acd3fe0 0x55cb6acd0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.057+0000 7f0e77fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6acce480 0x55cb6acfc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.057+0000 7f0e777fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6accee70 0x55cb6acdfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.057+0000 7f0e77fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6acce480 0x55cb6acfc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:52312/0 (socket says 172.21.15.29:52312) 2024-03-11T19:20:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.057+0000 7f0e777fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6accee70 0x55cb6acdfb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60966/0 (socket says 172.21.15.29:60966) 2024-03-11T19:20:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.057+0000 7f0e77fff640 1 -- 172.21.15.29:0/334981722 learned_addr learned my addr 172.21.15.29:0/334981722 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.057+0000 7f0e81193740 1 -- 172.21.15.29:0/334981722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb6abbc130 con 0x55cb6accee70 2024-03-11T19:20:18.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.057+0000 7f0e7ca7c640 1 --2- 172.21.15.29:0/334981722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6acd3fe0 0x55cb6acd0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.058+0000 7f0e81193740 1 -- 172.21.15.29:0/334981722 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb6a973720 con 0x55cb6acd3fe0 2024-03-11T19:20:18.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.058+0000 7f0e81193740 1 -- 172.21.15.29:0/334981722 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb6a9367e0 con 0x55cb6acce480 2024-03-11T19:20:18.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.058+0000 7f0e77fff640 1 -- 172.21.15.29:0/334981722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6acd3fe0 msgr2=0x55cb6acd0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.058+0000 7f0e77fff640 1 --2- 172.21.15.29:0/334981722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6acd3fe0 0x55cb6acd0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.058+0000 7f0e77fff640 1 -- 172.21.15.29:0/334981722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6accee70 msgr2=0x55cb6acdfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.058+0000 7f0e77fff640 1 --2- 172.21.15.29:0/334981722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6accee70 0x55cb6acdfb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.058+0000 7f0e77fff640 1 -- 172.21.15.29:0/334981722 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cb6abd1c20 con 0x55cb6acce480 2024-03-11T19:20:18.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.058+0000 7f0e77fff640 1 --2- 172.21.15.29:0/334981722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6acce480 0x55cb6acfc190 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0e6809a430 tx=0x7f0e680989b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=548c9adee3e63ec9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.059+0000 7f0e5dffb640 1 -- 172.21.15.29:0/334981722 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e68099fe0 con 0x55cb6acce480 2024-03-11T19:20:18.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.059+0000 7f0e5dffb640 1 -- 172.21.15.29:0/334981722 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0e680a2e90 con 0x55cb6acce480 2024-03-11T19:20:18.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.059+0000 7f0e5dffb640 1 -- 172.21.15.29:0/334981722 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6809a4a0 con 0x55cb6acce480 2024-03-11T19:20:18.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.060+0000 7f0e81193740 1 -- 172.21.15.29:0/334981722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6acce480 msgr2=0x55cb6acfc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.060+0000 7f0e81193740 1 --2- 172.21.15.29:0/334981722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6acce480 0x55cb6acfc190 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0e6809a430 tx=0x7f0e680989b0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.064+0000 7f0e81193740 1 -- 172.21.15.29:0/334981722 shutdown_connections 2024-03-11T19:20:18.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.064+0000 7f0e81193740 1 --2- 172.21.15.29:0/334981722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6acd3fe0 0x55cb6acd0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.064+0000 7f0e81193740 1 --2- 172.21.15.29:0/334981722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6accee70 0x55cb6acdfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.064+0000 7f0e81193740 1 --2- 172.21.15.29:0/334981722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6acce480 0x55cb6acfc190 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.064+0000 7f0e81193740 1 -- 172.21.15.29:0/334981722 >> 172.21.15.29:0/334981722 conn(0x55cb6acc99f0 msgr2=0x55cb6acd63d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.068+0000 7f0e81193740 1 -- 172.21.15.29:0/334981722 shutdown_connections 2024-03-11T19:20:18.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.068+0000 7f0e81193740 1 -- 172.21.15.29:0/334981722 wait complete. 2024-03-11T19:20:18.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.068+0000 7f0e81193740 1 -- start start 2024-03-11T19:20:18.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.069+0000 7f0e81193740 1 Processor -- start 2024-03-11T19:20:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.069+0000 7f0e81193740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6acce480 0x55cb6acdd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.070+0000 7f0e81193740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6accee70 0x55cb6acdd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.070+0000 7f0e81193740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6acd3fe0 0x55cb6acde910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.070+0000 7f0e81193740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb6acdee50 con 0x55cb6acce480 2024-03-11T19:20:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.070+0000 7f0e81193740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb6acdefc0 con 0x55cb6accee70 2024-03-11T19:20:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.070+0000 7f0e81193740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb6acdf180 con 0x55cb6acd3fe0 2024-03-11T19:20:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.070+0000 7f0e81193740 10 client.0 _ll_register_callbacks cb 0x7f0e68003690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.070+0000 7f0e77fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6acce480 0x55cb6acdd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.070+0000 7f0e77fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6acce480 0x55cb6acdd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60968/0 (socket says 172.21.15.29:60968) 2024-03-11T19:20:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.070+0000 7f0e77fff640 1 -- 172.21.15.29:0/2487618992 learned_addr learned my addr 172.21.15.29:0/2487618992 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.070+0000 7f0e777fe640 1 --2- 172.21.15.29:0/2487618992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6accee70 0x55cb6acdd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.070+0000 7f0e7ca7c640 1 --2- 172.21.15.29:0/2487618992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6acd3fe0 0x55cb6acde910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.070+0000 7f0e77fff640 1 -- 172.21.15.29:0/2487618992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6acd3fe0 msgr2=0x55cb6acde910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.070+0000 7f0e77fff640 1 --2- 172.21.15.29:0/2487618992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6acd3fe0 0x55cb6acde910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.070+0000 7f0e77fff640 1 -- 172.21.15.29:0/2487618992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6accee70 msgr2=0x55cb6acdd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.070+0000 7f0e77fff640 1 --2- 172.21.15.29:0/2487618992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6accee70 0x55cb6acdd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.070+0000 7f0e77fff640 1 -- 172.21.15.29:0/2487618992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cb6ac356a0 con 0x55cb6acce480 2024-03-11T19:20:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.070+0000 7f0e777fe640 1 --2- 172.21.15.29:0/2487618992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6accee70 0x55cb6acdd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:20:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.071+0000 7f0e77fff640 1 --2- 172.21.15.29:0/2487618992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6acce480 0x55cb6acdd3c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f0e680cd190 tx=0x7f0e6800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1985e2c363bdf2d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.071+0000 7f0e577fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:20:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.071+0000 7f0e577fe640 1 -- 172.21.15.29:0/2487618992 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e68099fe0 con 0x55cb6acce480 2024-03-11T19:20:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.072+0000 7f0e577fe640 1 -- 172.21.15.29:0/2487618992 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0e680a2c40 con 0x55cb6acce480 2024-03-11T19:20:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.072+0000 7f0e81193740 1 -- 172.21.15.29:0/2487618992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0e6800ce20 con 0x55cb6acce480 2024-03-11T19:20:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.072+0000 7f0e577fe640 1 -- 172.21.15.29:0/2487618992 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e6809a4a0 con 0x55cb6acce480 2024-03-11T19:20:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.072+0000 7f0e81193740 10 client.17859 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.072+0000 7f0e81193740 1 -- 172.21.15.29:0/2487618992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55cb6abf78c0 con 0x55cb6acce480 2024-03-11T19:20:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.073+0000 7f0e577fe640 1 -- 172.21.15.29:0/2487618992 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0e6800ce20 con 0x55cb6acce480 2024-03-11T19:20:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.073+0000 7f0e577fe640 1 -- 172.21.15.29:0/2487618992 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0e68081070 con 0x55cb6acce480 2024-03-11T19:20:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.073+0000 7f0e81193740 10 client.17859 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.073+0000 7f0e81193740 10 client.17859 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:20:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.073+0000 7f0e81193740 20 client.17859 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.073+0000 7f0e81193740 1 --2- 172.21.15.29:0/2487618992 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55cb6ad4bdb0 0x55cb6ad4e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.073+0000 7f0e81193740 4 client.17859 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:20:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.073+0000 7f0e81193740 1 -- 172.21.15.29:0/2487618992 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cb6a9c2890 con 0x55cb6ad4bdb0 2024-03-11T19:20:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.073+0000 7f0e777fe640 1 --2- 172.21.15.29:0/2487618992 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55cb6ad4bdb0 0x55cb6ad4e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.074+0000 7f0e777fe640 1 --2- 172.21.15.29:0/2487618992 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55cb6ad4bdb0 0x55cb6ad4e1a0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2d863cb059dc8e84 server_cookie=7bf68c12fef3fde4 in_seq=0 out_seq=0 2024-03-11T19:20:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.074+0000 7f0e577fe640 10 client.17859 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:20:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.074+0000 7f0e577fe640 1 -- 172.21.15.29:0/2487618992 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cb6a9c2890 con 0x55cb6ad4bdb0 2024-03-11T19:20:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.074+0000 7f0e577fe640 10 client.17859 handle_command_reply: tid=0 2024-03-11T19:20:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.074+0000 7f0e81193740 1 client.17859 shutdown 2024-03-11T19:20:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.076+0000 7f0e81193740 1 -- 172.21.15.29:0/2487618992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6acce480 msgr2=0x55cb6acdd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.076+0000 7f0e81193740 1 --2- 172.21.15.29:0/2487618992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6acce480 0x55cb6acdd3c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f0e680cd190 tx=0x7f0e6800cf90 comp rx=0 tx=0).stop 2024-03-11T19:20:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.076+0000 7f0e81193740 1 -- 172.21.15.29:0/2487618992 shutdown_connections 2024-03-11T19:20:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.076+0000 7f0e81193740 1 --2- 172.21.15.29:0/2487618992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cb6acd3fe0 0x55cb6acde910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.076+0000 7f0e81193740 1 --2- 172.21.15.29:0/2487618992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cb6accee70 0x55cb6acdd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.076+0000 7f0e81193740 1 --2- 172.21.15.29:0/2487618992 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x55cb6ad4bdb0 0x55cb6ad4e1a0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.076+0000 7f0e81193740 1 --2- 172.21.15.29:0/2487618992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cb6acce480 0x55cb6acdd3c0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.076+0000 7f0e81193740 1 -- 172.21.15.29:0/2487618992 >> 172.21.15.29:0/2487618992 conn(0x55cb6acc99f0 msgr2=0x55cb6acd63d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.077+0000 7f0e577fe640 0 client.17859 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:20:18.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.078+0000 7f0e81193740 1 -- 172.21.15.29:0/2487618992 shutdown_connections 2024-03-11T19:20:18.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.078+0000 7f0e81193740 1 -- 172.21.15.29:0/2487618992 wait complete. 2024-03-11T19:20:18.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.078+0000 7f0e81193740 20 client.17859 trim_cache size 0 max 16384 2024-03-11T19:20:18.082 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:20:18.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.081+0000 7f0e7d27d640 1 -- 172.21.15.29:0/454653277 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0e600a1bb0 msgr2=0x7f0e600a4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.081+0000 7f0e7d27d640 1 --2- 172.21.15.29:0/454653277 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0e600a1bb0 0x7f0e600a4070 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0e7819eb30 tx=0x7f0e6c012640 comp rx=0 tx=0).stop 2024-03-11T19:20:18.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.081+0000 7f0e7d27d640 1 -- 172.21.15.29:0/454653277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e78107680 msgr2=0x7f0e7819d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.081+0000 7f0e7d27d640 1 --2- 172.21.15.29:0/454653277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e78107680 0x7f0e7819d620 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800e9f0 tx=0x7f0e6800edf0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.082+0000 7f0e7d27d640 1 -- 172.21.15.29:0/454653277 shutdown_connections 2024-03-11T19:20:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.082+0000 7f0e7d27d640 1 --2- 172.21.15.29:0/454653277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e78109230 0x7f0e781a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.082+0000 7f0e7d27d640 1 --2- 172.21.15.29:0/454653277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e78108870 0x7f0e7819db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.082+0000 7f0e7d27d640 1 --2- 172.21.15.29:0/454653277 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0e600a1bb0 0x7f0e600a4070 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.082+0000 7f0e7d27d640 1 --2- 172.21.15.29:0/454653277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e78107680 0x7f0e7819d620 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.082+0000 7f0e7d27d640 1 -- 172.21.15.29:0/454653277 >> 172.21.15.29:0/454653277 conn(0x7f0e78076620 msgr2=0x7f0e7810bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.082+0000 7f0e7d27d640 1 -- 172.21.15.29:0/454653277 shutdown_connections 2024-03-11T19:20:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.082+0000 7f0e7d27d640 1 -- 172.21.15.29:0/454653277 wait complete. 2024-03-11T19:20:18.109 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:17 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3550584565' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:20:18.110 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:20:18.110 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:20:18.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:17 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3550584565' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:20:18.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.302+0000 7f9a46ffd640 1 -- 172.21.15.29:0/1436799095 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c02cb70 con 0x7f9a48100370 2024-03-11T19:20:18.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.303+0000 7f9a4e993640 1 -- 172.21.15.29:0/1436799095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a48100370 msgr2=0x7f9a481007d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.303+0000 7f9a4e993640 1 --2- 172.21.15.29:0/1436799095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a48100370 0x7f9a481007d0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009900 tx=0x7f9a3c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:20:18.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.303+0000 7f9a4e993640 1 -- 172.21.15.29:0/1436799095 shutdown_connections 2024-03-11T19:20:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.303+0000 7f9a4e993640 1 --2- 172.21.15.29:0/1436799095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a48100d10 0x7f9a48101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.303+0000 7f9a4e993640 1 --2- 172.21.15.29:0/1436799095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a48100370 0x7f9a481007d0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.303+0000 7f9a4e993640 1 --2- 172.21.15.29:0/1436799095 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a480ffa10 0x7f9a480ffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.303+0000 7f9a4e993640 1 -- 172.21.15.29:0/1436799095 >> 172.21.15.29:0/1436799095 conn(0x7f9a480fb630 msgr2=0x7f9a480fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.303+0000 7f9a4e993640 1 -- 172.21.15.29:0/1436799095 shutdown_connections 2024-03-11T19:20:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.303+0000 7f9a4e993640 1 -- 172.21.15.29:0/1436799095 wait complete. 2024-03-11T19:20:18.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.304+0000 7f9a4e993640 1 Processor -- start 2024-03-11T19:20:18.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.304+0000 7f9a4e993640 1 -- start start 2024-03-11T19:20:18.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.304+0000 7f9a4e993640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a480ffa10 0x7f9a48199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.305+0000 7f9a4e993640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a48100370 0x7f9a481997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.305+0000 7f9a47fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a480ffa10 0x7f9a48199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.305+0000 7f9a47fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a480ffa10 0x7f9a48199290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:52328/0 (socket says 172.21.15.29:52328) 2024-03-11T19:20:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.305+0000 7f9a47fff640 1 -- 172.21.15.29:0/2853963992 learned_addr learned my addr 172.21.15.29:0/2853963992 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.305+0000 7f9a4e993640 1 --2- 172.21.15.29:0/2853963992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a48100d10 0x7f9a481a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.305+0000 7f9a4e993640 1 -- 172.21.15.29:0/2853963992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a481a0d80 con 0x7f9a48100370 2024-03-11T19:20:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.305+0000 7f9a4e993640 1 -- 172.21.15.29:0/2853963992 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a481a0ef0 con 0x7f9a48100d10 2024-03-11T19:20:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.305+0000 7f9a4e993640 1 -- 172.21.15.29:0/2853963992 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a481a10b0 con 0x7f9a480ffa10 2024-03-11T19:20:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.306+0000 7f9a477fe640 1 --2- 172.21.15.29:0/2853963992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a48100370 0x7f9a481997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.306+0000 7f9a47fff640 1 -- 172.21.15.29:0/2853963992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a48100d10 msgr2=0x7f9a481a0840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:20:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.306+0000 7f9a47fff640 1 --2- 172.21.15.29:0/2853963992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a48100d10 0x7f9a481a0840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.306+0000 7f9a47fff640 1 -- 172.21.15.29:0/2853963992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a48100370 msgr2=0x7f9a481997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.306+0000 7f9a47fff640 1 --2- 172.21.15.29:0/2853963992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a48100370 0x7f9a481997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.306+0000 7f9a47fff640 1 -- 172.21.15.29:0/2853963992 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c009560 con 0x7f9a480ffa10 2024-03-11T19:20:18.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.307+0000 7f9a47fff640 1 --2- 172.21.15.29:0/2853963992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a480ffa10 0x7f9a48199290 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9a3800c920 tx=0x7f9a3800cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.308+0000 7f9a457fa640 1 -- 172.21.15.29:0/2853963992 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a38007c70 con 0x7f9a480ffa10 2024-03-11T19:20:18.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.308+0000 7f9a457fa640 1 -- 172.21.15.29:0/2853963992 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9a38007dd0 con 0x7f9a480ffa10 2024-03-11T19:20:18.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.308+0000 7f9a4e993640 1 -- 172.21.15.29:0/2853963992 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a481a1390 con 0x7f9a480ffa10 2024-03-11T19:20:18.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.309+0000 7f9a457fa640 1 -- 172.21.15.29:0/2853963992 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a38018800 con 0x7f9a480ffa10 2024-03-11T19:20:18.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.309+0000 7f9a4e993640 1 -- 172.21.15.29:0/2853963992 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a481a1870 con 0x7f9a480ffa10 2024-03-11T19:20:18.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.313+0000 7f9a457fa640 1 -- 172.21.15.29:0/2853963992 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9a38018960 con 0x7f9a480ffa10 2024-03-11T19:20:18.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.315+0000 7f9a457fa640 1 --2- 172.21.15.29:0/2853963992 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a300a1b80 0x7f9a300a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.315+0000 7f9a457fa640 1 -- 172.21.15.29:0/2853963992 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9a380cd3e0 con 0x7f9a480ffa10 2024-03-11T19:20:18.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.315+0000 7f9a5156f740 1 Processor -- start 2024-03-11T19:20:18.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.315+0000 7f9a5156f740 1 -- start start 2024-03-11T19:20:18.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.315+0000 7f9a477fe640 1 --2- 172.21.15.29:0/2853963992 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a300a1b80 0x7f9a300a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.315+0000 7f9a5156f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563faefa2270 0x563faefa2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.316+0000 7f9a5156f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa2c60 0x563faefa7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.316+0000 7f9a477fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa2c60 0x563faefa7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.316+0000 7f9a477fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa2c60 0x563faefa7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37056/0 (socket says 172.21.15.29:37056) 2024-03-11T19:20:18.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.316+0000 7f9a477fe640 1 -- 172.21.15.29:0/79722806 learned_addr learned my addr 172.21.15.29:0/79722806 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.316+0000 7f9a47fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563faefa2270 0x563faefa2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.316+0000 7f9a5156f740 1 --2- 172.21.15.29:0/79722806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa7dd0 0x563faefaa260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.317+0000 7f9a5156f740 1 -- 172.21.15.29:0/79722806 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563faef5e960 con 0x563faefa7dd0 2024-03-11T19:20:18.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.317+0000 7f9a5156f740 1 -- 172.21.15.29:0/79722806 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563faef12310 con 0x563faefa2c60 2024-03-11T19:20:18.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.317+0000 7f9a5156f740 1 -- 172.21.15.29:0/79722806 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563faef0f840 con 0x563faefa2270 2024-03-11T19:20:18.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.317+0000 7f9a4cf30640 1 --2- 172.21.15.29:0/79722806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa7dd0 0x563faefaa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.317+0000 7f9a477fe640 1 --2- 172.21.15.29:0/2853963992 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a300a1b80 0x7f9a300a4040 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009230 tx=0x7f9a3c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.317+0000 7f9a477fe640 1 -- 172.21.15.29:0/79722806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563faefa2270 msgr2=0x563faefa2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.317+0000 7f9a477fe640 1 --2- 172.21.15.29:0/79722806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563faefa2270 0x563faefa2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.317+0000 7f9a477fe640 1 -- 172.21.15.29:0/79722806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa7dd0 msgr2=0x563faefaa260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.317+0000 7f9a477fe640 1 --2- 172.21.15.29:0/79722806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa7dd0 0x563faefaa260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.317+0000 7f9a477fe640 1 -- 172.21.15.29:0/79722806 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563faeecd060 con 0x563faefa2c60 2024-03-11T19:20:18.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.318+0000 7f9a477fe640 1 --2- 172.21.15.29:0/79722806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa2c60 0x563faefa7760 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c035000 tx=0x7f9a3c047000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff42ce40f8deb9c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.318+0000 7f9a2e7fc640 1 -- 172.21.15.29:0/79722806 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c004040 con 0x563faefa2c60 2024-03-11T19:20:18.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.319+0000 7f9a2e7fc640 1 -- 172.21.15.29:0/79722806 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9a3c02de90 con 0x563faefa2c60 2024-03-11T19:20:18.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.319+0000 7f9a2e7fc640 1 -- 172.21.15.29:0/79722806 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c003ea0 con 0x563faefa2c60 2024-03-11T19:20:18.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.320+0000 7f9a5156f740 1 -- 172.21.15.29:0/79722806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa2c60 msgr2=0x563faefa7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.320+0000 7f9a5156f740 1 --2- 172.21.15.29:0/79722806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa2c60 0x563faefa7760 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c035000 tx=0x7f9a3c047000 comp rx=0 tx=0).stop 2024-03-11T19:20:18.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.320+0000 7f9a5156f740 1 -- 172.21.15.29:0/79722806 shutdown_connections 2024-03-11T19:20:18.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.320+0000 7f9a5156f740 1 --2- 172.21.15.29:0/79722806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa7dd0 0x563faefaa260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.321+0000 7f9a5156f740 1 --2- 172.21.15.29:0/79722806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa2c60 0x563faefa7760 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.321+0000 7f9a5156f740 1 --2- 172.21.15.29:0/79722806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563faefa2270 0x563faefa2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.321+0000 7f9a5156f740 1 -- 172.21.15.29:0/79722806 >> 172.21.15.29:0/79722806 conn(0x563faef9d7e0 msgr2=0x563faef9dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.321+0000 7f9a5156f740 1 -- 172.21.15.29:0/79722806 shutdown_connections 2024-03-11T19:20:18.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.321+0000 7f9a5156f740 1 -- 172.21.15.29:0/79722806 wait complete. 2024-03-11T19:20:18.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.322+0000 7f9a5156f740 1 -- start start 2024-03-11T19:20:18.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.322+0000 7f9a5156f740 1 Processor -- start 2024-03-11T19:20:18.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.323+0000 7f9a5156f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa2270 0x563faefb4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.323+0000 7f9a5156f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa2c60 0x563faefb5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.323+0000 7f9a5156f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563faefa7dd0 0x563faefbc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.323+0000 7f9a5156f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563faef0b020 con 0x563faefa2c60 2024-03-11T19:20:18.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.323+0000 7f9a5156f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563faebde430 con 0x563faefa2270 2024-03-11T19:20:18.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.324+0000 7f9a5156f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563faeb9ae00 con 0x563faefa7dd0 2024-03-11T19:20:18.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.324+0000 7f9a5156f740 10 client.0 _ll_register_callbacks cb 0x563faeeb2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:18.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.325+0000 7f9a4cf30640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563faefa7dd0 0x563faefbc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.325+0000 7f9a47fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa2270 0x563faefb4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.325+0000 7f9a47fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa2270 0x563faefb4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37070/0 (socket says 172.21.15.29:37070) 2024-03-11T19:20:18.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.325+0000 7f9a47fff640 1 -- 172.21.15.29:0/2105157177 learned_addr learned my addr 172.21.15.29:0/2105157177 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.325+0000 7f9a47fff640 1 -- 172.21.15.29:0/2105157177 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563faefa7dd0 msgr2=0x563faefbc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.326+0000 7f9a47fff640 1 --2- 172.21.15.29:0/2105157177 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563faefa7dd0 0x563faefbc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.326+0000 7f9a47fff640 1 -- 172.21.15.29:0/2105157177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa2c60 msgr2=0x563faefb5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.326+0000 7f9a47fff640 1 --2- 172.21.15.29:0/2105157177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa2c60 0x563faefb5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.326+0000 7f9a47fff640 1 -- 172.21.15.29:0/2105157177 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4810c0e0 con 0x563faefa2270 2024-03-11T19:20:18.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.326+0000 7f9a277fe640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:20:18.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.326+0000 7f9a47fff640 1 --2- 172.21.15.29:0/2105157177 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa2270 0x563faefb4c10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9a380caf20 tx=0x7f9a380cb160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=643f8aa3fe51ae7b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.327+0000 7f9a277fe640 1 -- 172.21.15.29:0/2105157177 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a380b8e70 con 0x563faefa2270 2024-03-11T19:20:18.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.327+0000 7f9a5156f740 1 -- 172.21.15.29:0/2105157177 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563faec0a7e0 con 0x563faefa2270 2024-03-11T19:20:18.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.327+0000 7f9a277fe640 1 -- 172.21.15.29:0/2105157177 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9a380b4030 con 0x563faefa2270 2024-03-11T19:20:18.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.327+0000 7f9a277fe640 1 -- 172.21.15.29:0/2105157177 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a380b8e70 con 0x563faefa2270 2024-03-11T19:20:18.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.331+0000 7f9a5156f740 10 client.27374 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:18.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.331+0000 7f9a5156f740 1 -- 172.21.15.29:0/2105157177 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563faef096a0 con 0x563faefa2270 2024-03-11T19:20:18.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.331+0000 7f9a277fe640 1 -- 172.21.15.29:0/2105157177 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x563faec0a7e0 con 0x563faefa2270 2024-03-11T19:20:18.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.331+0000 7f9a277fe640 1 -- 172.21.15.29:0/2105157177 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9a38016020 con 0x563faefa2270 2024-03-11T19:20:18.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.331+0000 7f9a5156f740 10 client.27374 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:18.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.331+0000 7f9a5156f740 10 client.27374 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:20:18.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.331+0000 7f9a5156f740 20 client.27374 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:18.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.331+0000 7f9a5156f740 1 --2- 172.21.15.29:0/2105157177 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x563faefb08f0 0x563faefb2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.331+0000 7f9a5156f740 4 client.27374 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:20:18.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.331+0000 7f9a5156f740 1 -- 172.21.15.29:0/2105157177 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563faec47720 con 0x563faefb08f0 2024-03-11T19:20:18.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.332+0000 7f9a477fe640 1 --2- 172.21.15.29:0/2105157177 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x563faefb08f0 0x563faefb2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.333+0000 7f9a477fe640 1 --2- 172.21.15.29:0/2105157177 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x563faefb08f0 0x563faefb2d30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e77e1ca6f3217cdd server_cookie=906c7fc644c1e291 in_seq=0 out_seq=0 2024-03-11T19:20:18.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.333+0000 7f9a277fe640 10 client.27374 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:20:18.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.333+0000 7f9a277fe640 1 -- 172.21.15.29:0/2105157177 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563faec47720 con 0x563faefb08f0 2024-03-11T19:20:18.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.333+0000 7f9a277fe640 10 client.27374 handle_command_reply: tid=0 2024-03-11T19:20:18.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.333+0000 7f9a5156f740 1 client.27374 shutdown 2024-03-11T19:20:18.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.334+0000 7f9a5156f740 1 -- 172.21.15.29:0/2105157177 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa2270 msgr2=0x563faefb4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.334+0000 7f9a5156f740 1 --2- 172.21.15.29:0/2105157177 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa2270 0x563faefb4c10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9a380caf20 tx=0x7f9a380cb160 comp rx=0 tx=0).stop 2024-03-11T19:20:18.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.334+0000 7f9a5156f740 1 -- 172.21.15.29:0/2105157177 shutdown_connections 2024-03-11T19:20:18.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.334+0000 7f9a5156f740 1 --2- 172.21.15.29:0/2105157177 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x563faefb08f0 0x563faefb2d30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.334+0000 7f9a5156f740 1 --2- 172.21.15.29:0/2105157177 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563faefa7dd0 0x563faefbc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.334+0000 7f9a5156f740 1 --2- 172.21.15.29:0/2105157177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa2c60 0x563faefb5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.334+0000 7f9a5156f740 1 --2- 172.21.15.29:0/2105157177 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa2270 0x563faefb4c10 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.334+0000 7f9a5156f740 1 -- 172.21.15.29:0/2105157177 >> 172.21.15.29:0/2105157177 conn(0x563faef9d7e0 msgr2=0x563faef9cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.334+0000 7f9a277fe640 0 client.27374 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:20:18.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.335+0000 7f9a5156f740 1 -- 172.21.15.29:0/2105157177 shutdown_connections 2024-03-11T19:20:18.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.335+0000 7f9a5156f740 1 -- 172.21.15.29:0/2105157177 wait complete. 2024-03-11T19:20:18.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.335+0000 7f9a5156f740 20 client.27374 trim_cache size 0 max 16384 2024-03-11T19:20:18.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:17 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3550584565' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:20:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.353+0000 7f9a5156f740 1 Processor -- start 2024-03-11T19:20:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.354+0000 7f9a5156f740 1 -- start start 2024-03-11T19:20:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.354+0000 7f9a5156f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa2270 0x563faef98c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.354+0000 7f9a5156f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa2c60 0x563faefbc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.354+0000 7f9a5156f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563faefa7dd0 0x563faef9c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.354+0000 7f9a5156f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563faee90130 con 0x563faefa2270 2024-03-11T19:20:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.354+0000 7f9a47fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa2270 0x563faef98c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.354+0000 7f9a5156f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563faec47720 con 0x563faefa2c60 2024-03-11T19:20:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.354+0000 7f9a5156f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563faefb5e00 con 0x563faefa7dd0 2024-03-11T19:20:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.354+0000 7f9a47fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa2270 0x563faef98c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32788/0 (socket says 172.21.15.29:32788) 2024-03-11T19:20:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.354+0000 7f9a47fff640 1 -- 172.21.15.29:0/2254749646 learned_addr learned my addr 172.21.15.29:0/2254749646 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.354+0000 7f9a477fe640 1 --2- 172.21.15.29:0/2254749646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa2c60 0x563faefbc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.355+0000 7f9a47fff640 1 -- 172.21.15.29:0/2254749646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563faefa7dd0 msgr2=0x563faef9c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.355+0000 7f9a47fff640 1 --2- 172.21.15.29:0/2254749646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563faefa7dd0 0x563faef9c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.355+0000 7f9a47fff640 1 -- 172.21.15.29:0/2254749646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa2c60 msgr2=0x563faefbc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.355+0000 7f9a47fff640 1 --2- 172.21.15.29:0/2254749646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa2c60 0x563faefbc150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.355+0000 7f9a47fff640 1 -- 172.21.15.29:0/2254749646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563faeea5c20 con 0x563faefa2270 2024-03-11T19:20:18.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.355+0000 7f9a47fff640 1 --2- 172.21.15.29:0/2254749646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa2270 0x563faef98c40 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f9a3809a210 tx=0x7f9a3800f860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79905fb884b8d9c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.355+0000 7f9a2dffb640 1 -- 172.21.15.29:0/2254749646 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a380b8e70 con 0x563faefa2270 2024-03-11T19:20:18.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.355+0000 7f9a2dffb640 1 -- 172.21.15.29:0/2254749646 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9a380cafd0 con 0x563faefa2270 2024-03-11T19:20:18.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.355+0000 7f9a2dffb640 1 -- 172.21.15.29:0/2254749646 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3800d340 con 0x563faefa2270 2024-03-11T19:20:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.356+0000 7f9a5156f740 1 -- 172.21.15.29:0/2254749646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa2270 msgr2=0x563faef98c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.356+0000 7f9a5156f740 1 --2- 172.21.15.29:0/2254749646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa2270 0x563faef98c40 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f9a3809a210 tx=0x7f9a3800f860 comp rx=0 tx=0).stop 2024-03-11T19:20:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.356+0000 7f9a5156f740 1 -- 172.21.15.29:0/2254749646 shutdown_connections 2024-03-11T19:20:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.357+0000 7f9a5156f740 1 --2- 172.21.15.29:0/2254749646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563faefa7dd0 0x563faef9c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.357+0000 7f9a5156f740 1 --2- 172.21.15.29:0/2254749646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa2c60 0x563faefbc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.357+0000 7f9a5156f740 1 --2- 172.21.15.29:0/2254749646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa2270 0x563faef98c40 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.357+0000 7f9a5156f740 1 -- 172.21.15.29:0/2254749646 >> 172.21.15.29:0/2254749646 conn(0x563faef9d7e0 msgr2=0x563faefa3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.357+0000 7f9a5156f740 1 -- 172.21.15.29:0/2254749646 shutdown_connections 2024-03-11T19:20:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.357+0000 7f9a5156f740 1 -- 172.21.15.29:0/2254749646 wait complete. 2024-03-11T19:20:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.357+0000 7f9a5156f740 1 -- start start 2024-03-11T19:20:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.357+0000 7f9a5156f740 1 Processor -- start 2024-03-11T19:20:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.358+0000 7f9a5156f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa2270 0x563faefbb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.358+0000 7f9a5156f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563faefa2c60 0x563faefbbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.358+0000 7f9a5156f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa7dd0 0x563faf01f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.358+0000 7f9a5156f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x563faf01f830 con 0x563faefa2270 2024-03-11T19:20:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.358+0000 7f9a5156f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x563faf01f970 con 0x563faefa7dd0 2024-03-11T19:20:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.358+0000 7f9a5156f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563faf01fb00 con 0x563faefa2c60 2024-03-11T19:20:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.358+0000 7f9a5156f740 10 client.0 _ll_register_callbacks cb 0x7f9a380a2c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.358+0000 7f9a47fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa2270 0x563faefbb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.358+0000 7f9a47fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa2270 0x563faefbb6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32794/0 (socket says 172.21.15.29:32794) 2024-03-11T19:20:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.358+0000 7f9a47fff640 1 -- 172.21.15.29:0/204111662 learned_addr learned my addr 172.21.15.29:0/204111662 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.358+0000 7f9a477fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563faefa2c60 0x563faefbbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.358+0000 7f9a4cf30640 1 --2- 172.21.15.29:0/204111662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa7dd0 0x563faf01f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.358+0000 7f9a47fff640 1 -- 172.21.15.29:0/204111662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563faefa2c60 msgr2=0x563faefbbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.358+0000 7f9a47fff640 1 --2- 172.21.15.29:0/204111662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563faefa2c60 0x563faefbbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.358+0000 7f9a47fff640 1 -- 172.21.15.29:0/204111662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa7dd0 msgr2=0x563faf01f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.358+0000 7f9a47fff640 1 --2- 172.21.15.29:0/204111662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa7dd0 0x563faf01f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.358+0000 7f9a47fff640 1 -- 172.21.15.29:0/204111662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563faef096a0 con 0x563faefa2270 2024-03-11T19:20:18.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.358+0000 7f9a4cf30640 1 --2- 172.21.15.29:0/204111662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa7dd0 0x563faf01f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:20:18.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.359+0000 7f9a47fff640 1 --2- 172.21.15.29:0/204111662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa2270 0x563faefbb6e0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f9a3809a120 tx=0x7f9a3800eae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7501f8dfd5298dd server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.359+0000 7f9a277fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:20:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.359+0000 7f9a277fe640 1 -- 172.21.15.29:0/204111662 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a380cb690 con 0x563faefa2270 2024-03-11T19:20:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.359+0000 7f9a5156f740 1 -- 172.21.15.29:0/204111662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9a3c02de90 con 0x563faefa2270 2024-03-11T19:20:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.359+0000 7f9a277fe640 1 -- 172.21.15.29:0/204111662 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9a380b4030 con 0x563faefa2270 2024-03-11T19:20:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.360+0000 7f9a277fe640 1 -- 172.21.15.29:0/204111662 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a380b8bd0 con 0x563faefa2270 2024-03-11T19:20:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.360+0000 7f9a5156f740 10 client.17874 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.360+0000 7f9a5156f740 1 -- 172.21.15.29:0/204111662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563faeecb8c0 con 0x563faefa2270 2024-03-11T19:20:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.360+0000 7f9a277fe640 1 -- 172.21.15.29:0/204111662 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9a3c02de90 con 0x563faefa2270 2024-03-11T19:20:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.360+0000 7f9a277fe640 1 -- 172.21.15.29:0/204111662 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9a38082070 con 0x563faefa2270 2024-03-11T19:20:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.360+0000 7f9a5156f740 10 client.17874 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.360+0000 7f9a5156f740 10 client.17874 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:20:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.360+0000 7f9a5156f740 20 client.17874 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.360+0000 7f9a5156f740 1 --2- 172.21.15.29:0/204111662 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x563faf020f90 0x563faf023380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.360+0000 7f9a5156f740 4 client.17874 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:20:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.360+0000 7f9a5156f740 1 -- 172.21.15.29:0/204111662 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563faec96890 con 0x563faf020f90 2024-03-11T19:20:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.361+0000 7f9a477fe640 1 --2- 172.21.15.29:0/204111662 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x563faf020f90 0x563faf023380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.361+0000 7f9a277fe640 10 client.17874 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:20:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.361+0000 7f9a477fe640 1 --2- 172.21.15.29:0/204111662 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x563faf020f90 0x563faf023380 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f031241693b1c7c server_cookie=585aa834ee491f8c in_seq=0 out_seq=0 2024-03-11T19:20:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.361+0000 7f9a277fe640 1 -- 172.21.15.29:0/204111662 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563faec96890 con 0x563faf020f90 2024-03-11T19:20:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.361+0000 7f9a277fe640 10 client.17874 handle_command_reply: tid=0 2024-03-11T19:20:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.361+0000 7f9a5156f740 1 client.17874 shutdown 2024-03-11T19:20:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.362+0000 7f9a5156f740 1 -- 172.21.15.29:0/204111662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa2270 msgr2=0x563faefbb6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.362+0000 7f9a5156f740 1 --2- 172.21.15.29:0/204111662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa2270 0x563faefbb6e0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f9a3809a120 tx=0x7f9a3800eae0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.362+0000 7f9a5156f740 1 -- 172.21.15.29:0/204111662 shutdown_connections 2024-03-11T19:20:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.362+0000 7f9a5156f740 1 --2- 172.21.15.29:0/204111662 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x563faf020f90 0x563faf023380 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.362+0000 7f9a5156f740 1 --2- 172.21.15.29:0/204111662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563faefa7dd0 0x563faf01f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.362+0000 7f9a5156f740 1 --2- 172.21.15.29:0/204111662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563faefa2c60 0x563faefbbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.362+0000 7f9a5156f740 1 --2- 172.21.15.29:0/204111662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563faefa2270 0x563faefbb6e0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.362+0000 7f9a5156f740 1 -- 172.21.15.29:0/204111662 >> 172.21.15.29:0/204111662 conn(0x563faef9d7e0 msgr2=0x563faefa0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.362+0000 7f9a277fe640 0 client.17874 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:20:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.365+0000 7f9a5156f740 1 -- 172.21.15.29:0/204111662 shutdown_connections 2024-03-11T19:20:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.365+0000 7f9a5156f740 1 -- 172.21.15.29:0/204111662 wait complete. 2024-03-11T19:20:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.365+0000 7f9a5156f740 20 client.17874 trim_cache size 0 max 16384 2024-03-11T19:20:18.370 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:20:18.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.368+0000 7f9a4e993640 1 -- 172.21.15.29:0/2853963992 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a300a1b80 msgr2=0x7f9a300a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.368+0000 7f9a4e993640 1 --2- 172.21.15.29:0/2853963992 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a300a1b80 0x7f9a300a4040 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009230 tx=0x7f9a3c005fd0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.368+0000 7f9a4e993640 1 -- 172.21.15.29:0/2853963992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a480ffa10 msgr2=0x7f9a48199290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.368+0000 7f9a4e993640 1 --2- 172.21.15.29:0/2853963992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a480ffa10 0x7f9a48199290 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9a3800c920 tx=0x7f9a3800cde0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.372+0000 7f9a4e993640 1 -- 172.21.15.29:0/2853963992 shutdown_connections 2024-03-11T19:20:18.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.372+0000 7f9a4e993640 1 --2- 172.21.15.29:0/2853963992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a48100d10 0x7f9a481a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.372+0000 7f9a4e993640 1 --2- 172.21.15.29:0/2853963992 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f9a300a1b80 0x7f9a300a4040 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.372+0000 7f9a4e993640 1 --2- 172.21.15.29:0/2853963992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a48100370 0x7f9a481997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.372+0000 7f9a4e993640 1 --2- 172.21.15.29:0/2853963992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a480ffa10 0x7f9a48199290 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.372+0000 7f9a4e993640 1 -- 172.21.15.29:0/2853963992 >> 172.21.15.29:0/2853963992 conn(0x7f9a480fb630 msgr2=0x7f9a480fd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.374+0000 7f9a4e993640 1 -- 172.21.15.29:0/2853963992 shutdown_connections 2024-03-11T19:20:18.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.374+0000 7f9a4e993640 1 -- 172.21.15.29:0/2853963992 wait complete. 2024-03-11T19:20:18.400 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:20:18.400 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:20:18.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.577+0000 7fcf196e6640 1 -- 172.21.15.29:0/2455588760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf14101aa0 msgr2=0x7fcf141083c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.577+0000 7fcf196e6640 1 --2- 172.21.15.29:0/2455588760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf14101aa0 0x7fcf141083c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c009c10 tx=0x7fcf0c02bef0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.577+0000 7fcf196e6640 1 -- 172.21.15.29:0/2455588760 shutdown_connections 2024-03-11T19:20:18.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.577+0000 7fcf196e6640 1 --2- 172.21.15.29:0/2455588760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf14101aa0 0x7fcf141083c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.577+0000 7fcf196e6640 1 --2- 172.21.15.29:0/2455588760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf141010e0 0x7fcf14101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.577+0000 7fcf196e6640 1 --2- 172.21.15.29:0/2455588760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf140fff10 0x7fcf14100350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.577+0000 7fcf196e6640 1 -- 172.21.15.29:0/2455588760 >> 172.21.15.29:0/2455588760 conn(0x7fcf140fb4f0 msgr2=0x7fcf140fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.577+0000 7fcf196e6640 1 -- 172.21.15.29:0/2455588760 shutdown_connections 2024-03-11T19:20:18.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.577+0000 7fcf196e6640 1 -- 172.21.15.29:0/2455588760 wait complete. 2024-03-11T19:20:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.578+0000 7fcf196e6640 1 Processor -- start 2024-03-11T19:20:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.578+0000 7fcf196e6640 1 -- start start 2024-03-11T19:20:18.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.579+0000 7fcf196e6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf140fff10 0x7fcf14068e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.579+0000 7fcf196e6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf141010e0 0x7fcf14199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.579+0000 7fcf196e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf14101aa0 0x7fcf1419a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.579+0000 7fcf196e6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1419a900 con 0x7fcf14101aa0 2024-03-11T19:20:18.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.579+0000 7fcf196e6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1419aa70 con 0x7fcf141010e0 2024-03-11T19:20:18.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.579+0000 7fcf196e6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1419ac30 con 0x7fcf140fff10 2024-03-11T19:20:18.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.579+0000 7fcf18ee5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf14101aa0 0x7fcf1419a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.579+0000 7fcf18ee5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf14101aa0 0x7fcf1419a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32816/0 (socket says 172.21.15.29:32816) 2024-03-11T19:20:18.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.579+0000 7fcf18ee5640 1 -- 172.21.15.29:0/1484635274 learned_addr learned my addr 172.21.15.29:0/1484635274 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.579+0000 7fcf137fe640 1 --2- 172.21.15.29:0/1484635274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf141010e0 0x7fcf14199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.579+0000 7fcf13fff640 1 --2- 172.21.15.29:0/1484635274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf140fff10 0x7fcf14068e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.580+0000 7fcf137fe640 1 -- 172.21.15.29:0/1484635274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf140fff10 msgr2=0x7fcf14068e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.580+0000 7fcf137fe640 1 --2- 172.21.15.29:0/1484635274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf140fff10 0x7fcf14068e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.580+0000 7fcf137fe640 1 -- 172.21.15.29:0/1484635274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf14101aa0 msgr2=0x7fcf1419a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.580+0000 7fcf137fe640 1 --2- 172.21.15.29:0/1484635274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf14101aa0 0x7fcf1419a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.580+0000 7fcf137fe640 1 -- 172.21.15.29:0/1484635274 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf08009560 con 0x7fcf141010e0 2024-03-11T19:20:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.580+0000 7fcf137fe640 1 --2- 172.21.15.29:0/1484635274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf141010e0 0x7fcf14199b10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fcf0400e9f0 tx=0x7fcf0400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.580+0000 7fcf117fa640 1 -- 172.21.15.29:0/1484635274 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0400ccc0 con 0x7fcf141010e0 2024-03-11T19:20:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.580+0000 7fcf117fa640 1 -- 172.21.15.29:0/1484635274 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcf0400ce20 con 0x7fcf141010e0 2024-03-11T19:20:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.580+0000 7fcf117fa640 1 -- 172.21.15.29:0/1484635274 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf040187c0 con 0x7fcf141010e0 2024-03-11T19:20:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.580+0000 7fcf196e6640 1 -- 172.21.15.29:0/1484635274 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf0c009840 con 0x7fcf141010e0 2024-03-11T19:20:18.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.581+0000 7fcf196e6640 1 -- 172.21.15.29:0/1484635274 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1419b240 con 0x7fcf141010e0 2024-03-11T19:20:18.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.584+0000 7fcf117fa640 1 -- 172.21.15.29:0/1484635274 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fcf04010430 con 0x7fcf141010e0 2024-03-11T19:20:18.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.584+0000 7fcf1d5f9740 1 Processor -- start 2024-03-11T19:20:18.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.585+0000 7fcf1d5f9740 1 -- start start 2024-03-11T19:20:18.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.585+0000 7fcf117fa640 1 --2- 172.21.15.29:0/1484635274 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcefc0a1b30 0x7fcefc0a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.585+0000 7fcf1d5f9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56305f6b1480 0x56305f6b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.585+0000 7fcf1d5f9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56305f6b1e70 0x56305f6b6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.585+0000 7fcf1d5f9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b6fe0 0x56305f6b9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.585+0000 7fcf1d5f9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56305f66d960 con 0x56305f6b1480 2024-03-11T19:20:18.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.585+0000 7fcf1d5f9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56305f621310 con 0x56305f6b6fe0 2024-03-11T19:20:18.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.585+0000 7fcf1d5f9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56305f61e840 con 0x56305f6b1e70 2024-03-11T19:20:18.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.585+0000 7fcf117fa640 1 -- 172.21.15.29:0/1484635274 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fcf04014070 con 0x7fcf141010e0 2024-03-11T19:20:18.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.586+0000 7fcf18ee5640 1 --2- 172.21.15.29:0/1484635274 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcefc0a1b30 0x7fcefc0a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.586+0000 7fcf13fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56305f6b1480 0x56305f6b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.586+0000 7fcf18ee5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b6fe0 0x56305f6b9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.586+0000 7fcf18ee5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b6fe0 0x56305f6b9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37134/0 (socket says 172.21.15.29:37134) 2024-03-11T19:20:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.586+0000 7fcf18ee5640 1 -- 172.21.15.29:0/4221159565 learned_addr learned my addr 172.21.15.29:0/4221159565 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.586+0000 7fcf137fe640 1 --2- 172.21.15.29:0/4221159565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56305f6b1e70 0x56305f6b6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.587+0000 7fcf18ee5640 1 -- 172.21.15.29:0/4221159565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56305f6b1e70 msgr2=0x56305f6b6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.587+0000 7fcf18ee5640 1 --2- 172.21.15.29:0/4221159565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56305f6b1e70 0x56305f6b6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.587+0000 7fcf18ee5640 1 -- 172.21.15.29:0/4221159565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56305f6b1480 msgr2=0x56305f6b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.587+0000 7fcf18ee5640 1 --2- 172.21.15.29:0/4221159565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56305f6b1480 0x56305f6b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.587+0000 7fcf18ee5640 1 -- 172.21.15.29:0/4221159565 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56305f5dc060 con 0x56305f6b6fe0 2024-03-11T19:20:18.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.587+0000 7fcf18ee5640 1 --2- 172.21.15.29:0/4221159565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b6fe0 0x56305f6b9470 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c036040 tx=0x7fcf0c033720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f268faddcad4c8e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.588+0000 7fcefa7fc640 1 -- 172.21.15.29:0/4221159565 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c033b30 con 0x56305f6b6fe0 2024-03-11T19:20:18.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.588+0000 7fcefa7fc640 1 -- 172.21.15.29:0/4221159565 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcf0c003e20 con 0x56305f6b6fe0 2024-03-11T19:20:18.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.589+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/4221159565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b6fe0 msgr2=0x56305f6b9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.589+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/4221159565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b6fe0 0x56305f6b9470 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c036040 tx=0x7fcf0c033720 comp rx=0 tx=0).stop 2024-03-11T19:20:18.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.590+0000 7fcf18ee5640 1 --2- 172.21.15.29:0/1484635274 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcefc0a1b30 0x7fcefc0a3ff0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c04a000 tx=0x7fcf0c034900 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.590+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/4221159565 shutdown_connections 2024-03-11T19:20:18.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.590+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/4221159565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b6fe0 0x56305f6b9470 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.590+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/4221159565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56305f6b1e70 0x56305f6b6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.590+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/4221159565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56305f6b1480 0x56305f6b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.590+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/4221159565 >> 172.21.15.29:0/4221159565 conn(0x56305f6ac9f0 msgr2=0x56305f6acdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.594+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/4221159565 shutdown_connections 2024-03-11T19:20:18.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.594+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/4221159565 wait complete. 2024-03-11T19:20:18.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.595+0000 7fcf1d5f9740 1 -- start start 2024-03-11T19:20:18.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.595+0000 7fcf1d5f9740 1 Processor -- start 2024-03-11T19:20:18.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.595+0000 7fcf1d5f9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56305f6b1480 0x56305f6cc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.595+0000 7fcf1d5f9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56305f6b1e70 0x56305f6ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.595+0000 7fcf1d5f9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b6fe0 0x56305f6d3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.595+0000 7fcf1d5f9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56305f61a020 con 0x56305f6b1e70 2024-03-11T19:20:18.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.595+0000 7fcf1d5f9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56305f2ed430 con 0x56305f6b6fe0 2024-03-11T19:20:18.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.595+0000 7fcf1d5f9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56305f2a9e00 con 0x56305f6b1480 2024-03-11T19:20:18.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.595+0000 7fcf1d5f9740 10 client.0 _ll_register_callbacks cb 0x56305f5c1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:18.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.596+0000 7fcf18ee5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b6fe0 0x56305f6d3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.596+0000 7fcf18ee5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b6fe0 0x56305f6d3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37140/0 (socket says 172.21.15.29:37140) 2024-03-11T19:20:18.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.596+0000 7fcf18ee5640 1 -- 172.21.15.29:0/1845191071 learned_addr learned my addr 172.21.15.29:0/1845191071 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.596+0000 7fcf137fe640 1 --2- 172.21.15.29:0/1845191071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56305f6b1e70 0x56305f6ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.596+0000 7fcf18ee5640 1 -- 172.21.15.29:0/1845191071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56305f6b1480 msgr2=0x56305f6cc630 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:20:18.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.596+0000 7fcf18ee5640 1 --2- 172.21.15.29:0/1845191071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56305f6b1480 0x56305f6cc630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.596+0000 7fcf18ee5640 1 -- 172.21.15.29:0/1845191071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56305f6b1e70 msgr2=0x56305f6ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.596+0000 7fcf18ee5640 1 --2- 172.21.15.29:0/1845191071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56305f6b1e70 0x56305f6ccb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.596+0000 7fcf18ee5640 1 -- 172.21.15.29:0/1845191071 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf14108c80 con 0x56305f6b6fe0 2024-03-11T19:20:18.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.598+0000 7fcf18ee5640 1 --2- 172.21.15.29:0/1845191071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b6fe0 0x56305f6d3bb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c006010 tx=0x7fcf0c009280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0f5f21781fcc400 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.598+0000 7fcef37fe640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:20:18.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.598+0000 7fcef37fe640 1 -- 172.21.15.29:0/1845191071 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c003b80 con 0x56305f6b6fe0 2024-03-11T19:20:18.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.598+0000 7fcef37fe640 1 -- 172.21.15.29:0/1845191071 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fcf0c033ad0 con 0x56305f6b6fe0 2024-03-11T19:20:18.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.601+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/1845191071 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56305f3197e0 con 0x56305f6b6fe0 2024-03-11T19:20:18.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.603+0000 7fcef37fe640 1 -- 172.21.15.29:0/1845191071 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf0c033c90 con 0x56305f6b6fe0 2024-03-11T19:20:18.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.604+0000 7fcf1d5f9740 10 client.27392 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:18.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.604+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/1845191071 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56305f6186a0 con 0x56305f6b6fe0 2024-03-11T19:20:18.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.607+0000 7fcef37fe640 1 -- 172.21.15.29:0/1845191071 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x56305f3197e0 con 0x56305f6b6fe0 2024-03-11T19:20:18.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.608+0000 7fcef37fe640 1 -- 172.21.15.29:0/1845191071 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcf0c041450 con 0x56305f6b6fe0 2024-03-11T19:20:18.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.609+0000 7fcf1d5f9740 10 client.27392 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:18.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.609+0000 7fcf1d5f9740 10 client.27392 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:20:18.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.609+0000 7fcf1d5f9740 20 client.27392 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:18.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.609+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/1845191071 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x56305f6e9580 0x56305f6eb990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.609+0000 7fcf1d5f9740 4 client.27392 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:20:18.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.609+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/1845191071 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56305f356720 con 0x56305f6e9580 2024-03-11T19:20:18.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.613+0000 7fcf13fff640 1 --2- 172.21.15.29:0/1845191071 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x56305f6e9580 0x56305f6eb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.614+0000 7fcf13fff640 1 --2- 172.21.15.29:0/1845191071 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x56305f6e9580 0x56305f6eb990 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7d962e12a6fed38f server_cookie=1d264b7385ae4970 in_seq=0 out_seq=0 2024-03-11T19:20:18.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.614+0000 7fcef37fe640 10 client.27392 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:20:18.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.615+0000 7fcef37fe640 1 -- 172.21.15.29:0/1845191071 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56305f356720 con 0x56305f6e9580 2024-03-11T19:20:18.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.615+0000 7fcef37fe640 10 client.27392 handle_command_reply: tid=0 2024-03-11T19:20:18.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.615+0000 7fcf1d5f9740 1 client.27392 shutdown 2024-03-11T19:20:18.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.615+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/1845191071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b6fe0 msgr2=0x56305f6d3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.615+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/1845191071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b6fe0 0x56305f6d3bb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c006010 tx=0x7fcf0c009280 comp rx=0 tx=0).stop 2024-03-11T19:20:18.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.615+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/1845191071 shutdown_connections 2024-03-11T19:20:18.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.615+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/1845191071 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x56305f6e9580 0x56305f6eb990 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.615+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/1845191071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b6fe0 0x56305f6d3bb0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.615+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/1845191071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56305f6b1e70 0x56305f6ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.615+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/1845191071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56305f6b1480 0x56305f6cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.615+0000 7fcef37fe640 0 client.27392 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:20:18.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.615+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/1845191071 >> 172.21.15.29:0/1845191071 conn(0x56305f6ac9f0 msgr2=0x56305f6abcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.615+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/1845191071 shutdown_connections 2024-03-11T19:20:18.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.615+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/1845191071 wait complete. 2024-03-11T19:20:18.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.616+0000 7fcf1d5f9740 20 client.27392 trim_cache size 0 max 16384 2024-03-11T19:20:18.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.638+0000 7fcf1d5f9740 1 Processor -- start 2024-03-11T19:20:18.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.638+0000 7fcf1d5f9740 1 -- start start 2024-03-11T19:20:18.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.638+0000 7fcf1d5f9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b1480 0x56305f6b3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.638+0000 7fcf1d5f9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56305f6b1e70 0x56305f6d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.638+0000 7fcf1d5f9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56305f6b6fe0 0x56305f6b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.638+0000 7fcf1d5f9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56305f59f130 con 0x56305f6b6fe0 2024-03-11T19:20:18.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.638+0000 7fcf1d5f9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56305f6b2560 con 0x56305f6b1480 2024-03-11T19:20:18.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.638+0000 7fcf1d5f9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56305f6e60b0 con 0x56305f6b1e70 2024-03-11T19:20:18.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.638+0000 7fcf13fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b1480 0x56305f6b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.638+0000 7fcf13fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b1480 0x56305f6b3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37148/0 (socket says 172.21.15.29:37148) 2024-03-11T19:20:18.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.638+0000 7fcf13fff640 1 -- 172.21.15.29:0/388096471 learned_addr learned my addr 172.21.15.29:0/388096471 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.638+0000 7fcf137fe640 1 --2- 172.21.15.29:0/388096471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56305f6b1e70 0x56305f6d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.639+0000 7fcf13fff640 1 -- 172.21.15.29:0/388096471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56305f6b1e70 msgr2=0x56305f6d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.639+0000 7fcf13fff640 1 --2- 172.21.15.29:0/388096471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56305f6b1e70 0x56305f6d3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.639+0000 7fcf13fff640 1 -- 172.21.15.29:0/388096471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56305f6b6fe0 msgr2=0x56305f6b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.639+0000 7fcf13fff640 1 --2- 172.21.15.29:0/388096471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56305f6b6fe0 0x56305f6b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.639+0000 7fcf13fff640 1 -- 172.21.15.29:0/388096471 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56305f5b4c20 con 0x56305f6b1480 2024-03-11T19:20:18.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.639+0000 7fcf13fff640 1 --2- 172.21.15.29:0/388096471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b1480 0x56305f6b3290 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x56305f6b26d0 tx=0x7fcf08014840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca92e38cb8ea0c64 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.639+0000 7fcef9ffb640 1 -- 172.21.15.29:0/388096471 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf08017040 con 0x56305f6b1480 2024-03-11T19:20:18.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.639+0000 7fcef9ffb640 1 -- 172.21.15.29:0/388096471 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcf08021070 con 0x56305f6b1480 2024-03-11T19:20:18.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.639+0000 7fcef9ffb640 1 -- 172.21.15.29:0/388096471 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf08004730 con 0x56305f6b1480 2024-03-11T19:20:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.640+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/388096471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b1480 msgr2=0x56305f6b3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.640+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/388096471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b1480 0x56305f6b3290 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x56305f6b26d0 tx=0x7fcf08014840 comp rx=0 tx=0).stop 2024-03-11T19:20:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.641+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/388096471 shutdown_connections 2024-03-11T19:20:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.641+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/388096471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56305f6b6fe0 0x56305f6b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.641+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/388096471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56305f6b1e70 0x56305f6d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.641+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/388096471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b1480 0x56305f6b3290 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.641+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/388096471 >> 172.21.15.29:0/388096471 conn(0x56305f6ac9f0 msgr2=0x56305f6cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.641+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/388096471 shutdown_connections 2024-03-11T19:20:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.641+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/388096471 wait complete. 2024-03-11T19:20:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.642+0000 7fcf1d5f9740 1 -- start start 2024-03-11T19:20:18.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.642+0000 7fcf1d5f9740 1 Processor -- start 2024-03-11T19:20:18.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.643+0000 7fcf1d5f9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b1480 0x56305f6d2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.643+0000 7fcf1d5f9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56305f6b1e70 0x56305f6d3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.643+0000 7fcf13fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b1480 0x56305f6d2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.643+0000 7fcf13fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b1480 0x56305f6d2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37160/0 (socket says 172.21.15.29:37160) 2024-03-11T19:20:18.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.643+0000 7fcf13fff640 1 -- 172.21.15.29:0/2603313844 learned_addr learned my addr 172.21.15.29:0/2603313844 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.643+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/2603313844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56305f6b6fe0 0x56305f73fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.644+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/2603313844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56305f740250 con 0x56305f6b1e70 2024-03-11T19:20:18.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.644+0000 7fcf18ee5640 1 --2- 172.21.15.29:0/2603313844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56305f6b6fe0 0x56305f73fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.644+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/2603313844 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56305f740390 con 0x56305f6b1480 2024-03-11T19:20:18.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.644+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/2603313844 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56305f740520 con 0x56305f6b6fe0 2024-03-11T19:20:18.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.644+0000 7fcf1d5f9740 10 client.0 _ll_register_callbacks cb 0x7fcf04005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:18.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.644+0000 7fcf137fe640 1 --2- 172.21.15.29:0/2603313844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56305f6b1e70 0x56305f6d3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.644+0000 7fcf13fff640 1 -- 172.21.15.29:0/2603313844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56305f6b6fe0 msgr2=0x56305f73fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.644+0000 7fcf13fff640 1 --2- 172.21.15.29:0/2603313844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56305f6b6fe0 0x56305f73fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.644+0000 7fcf13fff640 1 -- 172.21.15.29:0/2603313844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56305f6b1e70 msgr2=0x56305f6d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.644+0000 7fcf13fff640 1 --2- 172.21.15.29:0/2603313844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56305f6b1e70 0x56305f6d3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.644+0000 7fcf13fff640 1 -- 172.21.15.29:0/2603313844 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf08009d80 con 0x56305f6b1480 2024-03-11T19:20:18.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.645+0000 7fcf13fff640 1 --2- 172.21.15.29:0/2603313844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b1480 0x56305f6d2e00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcf080056e0 tx=0x7fcf08005710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44fab9503d51e25 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.645+0000 7fcef37fe640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:20:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.646+0000 7fcef37fe640 1 -- 172.21.15.29:0/2603313844 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0802b070 con 0x56305f6b1480 2024-03-11T19:20:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.646+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/2603313844 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf0c003e20 con 0x56305f6b1480 2024-03-11T19:20:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.646+0000 7fcef37fe640 1 -- 172.21.15.29:0/2603313844 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fcf08004030 con 0x56305f6b1480 2024-03-11T19:20:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.646+0000 7fcef37fe640 1 -- 172.21.15.29:0/2603313844 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf08017040 con 0x56305f6b1480 2024-03-11T19:20:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.646+0000 7fcf1d5f9740 10 client.27398 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.646+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/2603313844 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56305f5da8c0 con 0x56305f6b1480 2024-03-11T19:20:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.647+0000 7fcef37fe640 1 -- 172.21.15.29:0/2603313844 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcf080142d0 con 0x56305f6b1480 2024-03-11T19:20:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.647+0000 7fcef37fe640 1 -- 172.21.15.29:0/2603313844 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcf080049f0 con 0x56305f6b1480 2024-03-11T19:20:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.647+0000 7fcf1d5f9740 10 client.27398 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.647+0000 7fcf1d5f9740 10 client.27398 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:20:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.647+0000 7fcf1d5f9740 20 client.27398 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.647+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/2603313844 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x56305f741f70 0x56305f744390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.647+0000 7fcf1d5f9740 4 client.27398 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:20:18.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.647+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/2603313844 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7fcf0c003e20 con 0x56305f741f70 2024-03-11T19:20:18.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.648+0000 7fcf137fe640 1 --2- 172.21.15.29:0/2603313844 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x56305f741f70 0x56305f744390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.649+0000 7fcf137fe640 1 --2- 172.21.15.29:0/2603313844 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x56305f741f70 0x56305f744390 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7e895056d7cf8830 server_cookie=3e1de76f4213b7bf in_seq=0 out_seq=0 2024-03-11T19:20:18.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.649+0000 7fcef37fe640 10 client.27398 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:20:18.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.650+0000 7fcef37fe640 1 -- 172.21.15.29:0/2603313844 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fcf0c003e20 con 0x56305f741f70 2024-03-11T19:20:18.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.650+0000 7fcef37fe640 10 client.27398 handle_command_reply: tid=0 2024-03-11T19:20:18.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.650+0000 7fcf1d5f9740 1 client.27398 shutdown 2024-03-11T19:20:18.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.652+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/2603313844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b1480 msgr2=0x56305f6d2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.652+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/2603313844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b1480 0x56305f6d2e00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcf080056e0 tx=0x7fcf08005710 comp rx=0 tx=0).stop 2024-03-11T19:20:18.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.656+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/2603313844 shutdown_connections 2024-03-11T19:20:18.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.656+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/2603313844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56305f6b6fe0 0x56305f73fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.656+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/2603313844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56305f6b1e70 0x56305f6d3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.656+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/2603313844 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x56305f741f70 0x56305f744390 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.657+0000 7fcf1d5f9740 1 --2- 172.21.15.29:0/2603313844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56305f6b1480 0x56305f6d2e00 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.657+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/2603313844 >> 172.21.15.29:0/2603313844 conn(0x56305f6ac9f0 msgr2=0x56305f6cc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.657+0000 7fcef37fe640 0 client.27398 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:20:18.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.657+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/2603313844 shutdown_connections 2024-03-11T19:20:18.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.657+0000 7fcf1d5f9740 1 -- 172.21.15.29:0/2603313844 wait complete. 2024-03-11T19:20:18.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.661+0000 7fcf1d5f9740 20 client.27398 trim_cache size 0 max 16384 2024-03-11T19:20:18.666 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:20:18.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.667+0000 7fcf196e6640 1 -- 172.21.15.29:0/1484635274 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcefc0a1b30 msgr2=0x7fcefc0a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.667+0000 7fcf196e6640 1 --2- 172.21.15.29:0/1484635274 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcefc0a1b30 0x7fcefc0a3ff0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c04a000 tx=0x7fcf0c034900 comp rx=0 tx=0).stop 2024-03-11T19:20:18.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.667+0000 7fcf196e6640 1 -- 172.21.15.29:0/1484635274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf141010e0 msgr2=0x7fcf14199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.667+0000 7fcf196e6640 1 --2- 172.21.15.29:0/1484635274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf141010e0 0x7fcf14199b10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fcf0400e9f0 tx=0x7fcf0400eeb0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.670+0000 7fcf196e6640 1 -- 172.21.15.29:0/1484635274 shutdown_connections 2024-03-11T19:20:18.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.670+0000 7fcf196e6640 1 --2- 172.21.15.29:0/1484635274 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcefc0a1b30 0x7fcefc0a3ff0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.670+0000 7fcf196e6640 1 --2- 172.21.15.29:0/1484635274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf14101aa0 0x7fcf1419a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.670+0000 7fcf196e6640 1 --2- 172.21.15.29:0/1484635274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf141010e0 0x7fcf14199b10 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.670+0000 7fcf196e6640 1 --2- 172.21.15.29:0/1484635274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf140fff10 0x7fcf14068e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.670+0000 7fcf196e6640 1 -- 172.21.15.29:0/1484635274 >> 172.21.15.29:0/1484635274 conn(0x7fcf140fb4f0 msgr2=0x7fcf14104540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.671+0000 7fcf196e6640 1 -- 172.21.15.29:0/1484635274 shutdown_connections 2024-03-11T19:20:18.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.671+0000 7fcf196e6640 1 -- 172.21.15.29:0/1484635274 wait complete. 2024-03-11T19:20:18.693 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:20:18.694 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:20:18.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.875+0000 7f4d71c2a640 1 -- 172.21.15.29:0/1044936147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d6c1077a0 msgr2=0x7f4d6c107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.875+0000 7f4d71c2a640 1 --2- 172.21.15.29:0/1044936147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d6c1077a0 0x7f4d6c107bc0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4d60009900 tx=0x7f4d6002bc60 comp rx=0 tx=0).stop 2024-03-11T19:20:18.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.875+0000 7f4d71c2a640 1 -- 172.21.15.29:0/1044936147 shutdown_connections 2024-03-11T19:20:18.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.875+0000 7f4d71c2a640 1 --2- 172.21.15.29:0/1044936147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d6c109350 0x7f4d6c10fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.875+0000 7f4d71c2a640 1 --2- 172.21.15.29:0/1044936147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d6c108990 0x7f4d6c108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.875+0000 7f4d71c2a640 1 --2- 172.21.15.29:0/1044936147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d6c1077a0 0x7f4d6c107bc0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.875+0000 7f4d71c2a640 1 -- 172.21.15.29:0/1044936147 >> 172.21.15.29:0/1044936147 conn(0x7f4d6c076730 msgr2=0x7f4d6c078b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.875+0000 7f4d71c2a640 1 -- 172.21.15.29:0/1044936147 shutdown_connections 2024-03-11T19:20:18.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.875+0000 7f4d71c2a640 1 -- 172.21.15.29:0/1044936147 wait complete. 2024-03-11T19:20:18.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.876+0000 7f4d71c2a640 1 Processor -- start 2024-03-11T19:20:18.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.876+0000 7f4d71c2a640 1 -- start start 2024-03-11T19:20:18.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.876+0000 7f4d71c2a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d6c1077a0 0x7f4d6c19d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.876+0000 7f4d71c2a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d6c108990 0x7f4d6c19dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.877+0000 7f4d71c2a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d6c109350 0x7f4d6c1a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.877+0000 7f4d71c2a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6c1a5310 con 0x7f4d6c108990 2024-03-11T19:20:18.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.877+0000 7f4d71c2a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6c1a5480 con 0x7f4d6c109350 2024-03-11T19:20:18.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.877+0000 7f4d71c2a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6c1a5640 con 0x7f4d6c1077a0 2024-03-11T19:20:18.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.877+0000 7f4d6bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d6c109350 0x7f4d6c1a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.877+0000 7f4d6bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d6c109350 0x7f4d6c1a4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37190/0 (socket says 172.21.15.29:37190) 2024-03-11T19:20:18.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.877+0000 7f4d6bfff640 1 -- 172.21.15.29:0/3255501383 learned_addr learned my addr 172.21.15.29:0/3255501383 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.877+0000 7f4d6affd640 1 --2- 172.21.15.29:0/3255501383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d6c108990 0x7f4d6c19dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.877+0000 7f4d6bfff640 1 -- 172.21.15.29:0/3255501383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d6c1077a0 msgr2=0x7f4d6c19d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.877+0000 7f4d6bfff640 1 --2- 172.21.15.29:0/3255501383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d6c1077a0 0x7f4d6c19d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.877+0000 7f4d6bfff640 1 -- 172.21.15.29:0/3255501383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d6c108990 msgr2=0x7f4d6c19dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.877+0000 7f4d6bfff640 1 --2- 172.21.15.29:0/3255501383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d6c108990 0x7f4d6c19dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.877+0000 7f4d6bfff640 1 -- 172.21.15.29:0/3255501383 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d60009560 con 0x7f4d6c109350 2024-03-11T19:20:18.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.878+0000 7f4d6bfff640 1 --2- 172.21.15.29:0/3255501383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d6c109350 0x7f4d6c1a4dd0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4d6400e970 tx=0x7f4d6400ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.878+0000 7f4d68ff9640 1 -- 172.21.15.29:0/3255501383 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d64009720 con 0x7f4d6c109350 2024-03-11T19:20:18.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.878+0000 7f4d71c2a640 1 -- 172.21.15.29:0/3255501383 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d6c1a5920 con 0x7f4d6c109350 2024-03-11T19:20:18.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.878+0000 7f4d68ff9640 1 -- 172.21.15.29:0/3255501383 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4d640054d0 con 0x7f4d6c109350 2024-03-11T19:20:18.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.878+0000 7f4d71c2a640 1 -- 172.21.15.29:0/3255501383 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d6c1a5eb0 con 0x7f4d6c109350 2024-03-11T19:20:18.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.879+0000 7f4d68ff9640 1 -- 172.21.15.29:0/3255501383 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d64010430 con 0x7f4d6c109350 2024-03-11T19:20:18.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.881+0000 7f4d747fe740 1 Processor -- start 2024-03-11T19:20:18.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.881+0000 7f4d68ff9640 1 -- 172.21.15.29:0/3255501383 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4d6400ccc0 con 0x7f4d6c109350 2024-03-11T19:20:18.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.882+0000 7f4d747fe740 1 -- start start 2024-03-11T19:20:18.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.882+0000 7f4d747fe740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed75690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.882+0000 7f4d747fe740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 0x56268ed7a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.882+0000 7f4d747fe740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 0x56268ed7d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.882+0000 7f4d747fe740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268ed31960 con 0x56268ed75c60 2024-03-11T19:20:18.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.882+0000 7f4d747fe740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268ece5310 con 0x56268ed7add0 2024-03-11T19:20:18.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.882+0000 7f4d747fe740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268ece2840 con 0x56268ed75270 2024-03-11T19:20:18.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.882+0000 7f4d6affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed75690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.882+0000 7f4d6affd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 0x56268ed7d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.882+0000 7f4d68ff9640 1 --2- 172.21.15.29:0/3255501383 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4d540a1950 0x7f4d540a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.882+0000 7f4d6affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed75690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:52426/0 (socket says 172.21.15.29:52426) 2024-03-11T19:20:18.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.882+0000 7f4d6affd640 1 -- 172.21.15.29:0/4083603295 learned_addr learned my addr 172.21.15.29:0/4083603295 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.883+0000 7f4d6b7fe640 1 --2- 172.21.15.29:0/4083603295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 0x56268ed7a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.883+0000 7f4d68ff9640 1 -- 172.21.15.29:0/3255501383 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4d64014070 con 0x7f4d6c109350 2024-03-11T19:20:18.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.883+0000 7f4d6bfff640 1 --2- 172.21.15.29:0/3255501383 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4d540a1950 0x7f4d540a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.883+0000 7f4d6b7fe640 1 -- 172.21.15.29:0/4083603295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 msgr2=0x56268ed75690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.883+0000 7f4d6b7fe640 1 --2- 172.21.15.29:0/4083603295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed75690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.883+0000 7f4d6b7fe640 1 -- 172.21.15.29:0/4083603295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 msgr2=0x56268ed7d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.883+0000 7f4d6b7fe640 1 --2- 172.21.15.29:0/4083603295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 0x56268ed7d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.883+0000 7f4d6b7fe640 1 -- 172.21.15.29:0/4083603295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56268eca0060 con 0x56268ed75c60 2024-03-11T19:20:18.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.884+0000 7f4d6bfff640 1 --2- 172.21.15.29:0/3255501383 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4d540a1950 0x7f4d540a3e10 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4d64009380 tx=0x7f4d640962b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.884+0000 7f4d6b7fe640 1 --2- 172.21.15.29:0/4083603295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 0x56268ed7a760 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f4d60009230 tx=0x7f4d60004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99b54cc4b4b36928 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.884+0000 7f4d51ffb640 1 -- 172.21.15.29:0/4083603295 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d60004740 con 0x56268ed75c60 2024-03-11T19:20:18.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.884+0000 7f4d51ffb640 1 -- 172.21.15.29:0/4083603295 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4d600048a0 con 0x56268ed75c60 2024-03-11T19:20:18.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.884+0000 7f4d51ffb640 1 -- 172.21.15.29:0/4083603295 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d60034920 con 0x56268ed75c60 2024-03-11T19:20:18.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.885+0000 7f4d747fe740 1 -- 172.21.15.29:0/4083603295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 msgr2=0x56268ed7a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.885+0000 7f4d747fe740 1 --2- 172.21.15.29:0/4083603295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 0x56268ed7a760 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f4d60009230 tx=0x7f4d60004590 comp rx=0 tx=0).stop 2024-03-11T19:20:18.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.885+0000 7f4d747fe740 1 -- 172.21.15.29:0/4083603295 shutdown_connections 2024-03-11T19:20:18.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.885+0000 7f4d747fe740 1 --2- 172.21.15.29:0/4083603295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 0x56268ed7d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.885+0000 7f4d747fe740 1 --2- 172.21.15.29:0/4083603295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 0x56268ed7a760 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.885+0000 7f4d747fe740 1 --2- 172.21.15.29:0/4083603295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed75690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.885+0000 7f4d747fe740 1 -- 172.21.15.29:0/4083603295 >> 172.21.15.29:0/4083603295 conn(0x56268ed707e0 msgr2=0x56268ed70be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.886+0000 7f4d747fe740 1 -- 172.21.15.29:0/4083603295 shutdown_connections 2024-03-11T19:20:18.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.886+0000 7f4d747fe740 1 -- 172.21.15.29:0/4083603295 wait complete. 2024-03-11T19:20:18.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.886+0000 7f4d747fe740 1 -- start start 2024-03-11T19:20:18.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.886+0000 7f4d747fe740 1 Processor -- start 2024-03-11T19:20:18.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.887+0000 7f4d747fe740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed7f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.887+0000 7f4d747fe740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 0x56268ed7f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.887+0000 7f4d747fe740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 0x56268ed869e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.887+0000 7f4d747fe740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268ecde020 con 0x56268ed75c60 2024-03-11T19:20:18.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.887+0000 7f4d747fe740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268e9b1430 con 0x56268ed7add0 2024-03-11T19:20:18.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.887+0000 7f4d6affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed7f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.887+0000 7f4d6affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed7f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:52436/0 (socket says 172.21.15.29:52436) 2024-03-11T19:20:18.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.887+0000 7f4d6affd640 1 -- 172.21.15.29:0/741932336 learned_addr learned my addr 172.21.15.29:0/741932336 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.887+0000 7f4d747fe740 1 -- 172.21.15.29:0/741932336 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268e96de00 con 0x56268ed75270 2024-03-11T19:20:18.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.887+0000 7f4d6b7fe640 1 --2- 172.21.15.29:0/741932336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 0x56268ed7f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.887+0000 7f4d747fe740 10 client.0 _ll_register_callbacks cb 0x56268ec85cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:18.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.887+0000 7f4d6affd640 1 --2- 172.21.15.29:0/741932336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 0x56268ed869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.888+0000 7f4d6affd640 1 -- 172.21.15.29:0/741932336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 msgr2=0x56268ed869e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.888+0000 7f4d6affd640 1 --2- 172.21.15.29:0/741932336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 0x56268ed869e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.888+0000 7f4d6affd640 1 -- 172.21.15.29:0/741932336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 msgr2=0x56268ed7f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.888+0000 7f4d6affd640 1 --2- 172.21.15.29:0/741932336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 0x56268ed7f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.888+0000 7f4d6affd640 1 -- 172.21.15.29:0/741932336 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d60038040 con 0x56268ed75270 2024-03-11T19:20:18.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.889+0000 7f4d6affd640 1 --2- 172.21.15.29:0/741932336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed7f460 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c01ca90 tx=0x7f4d5c009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=687f540fdcd3ee69 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.889+0000 7f4d4affd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:20:18.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.889+0000 7f4d4affd640 1 -- 172.21.15.29:0/741932336 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c008a40 con 0x56268ed75270 2024-03-11T19:20:18.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.889+0000 7f4d4affd640 1 -- 172.21.15.29:0/741932336 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4d5c008ba0 con 0x56268ed75270 2024-03-11T19:20:18.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.890+0000 7f4d747fe740 1 -- 172.21.15.29:0/741932336 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56268e9dd7e0 con 0x56268ed75270 2024-03-11T19:20:18.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.890+0000 7f4d4affd640 1 -- 172.21.15.29:0/741932336 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d5c008d60 con 0x56268ed75270 2024-03-11T19:20:18.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.890+0000 7f4d747fe740 10 client.27451 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:18.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.890+0000 7f4d747fe740 1 -- 172.21.15.29:0/741932336 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56268ecdc6a0 con 0x56268ed75270 2024-03-11T19:20:18.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.891+0000 7f4d4affd640 1 -- 172.21.15.29:0/741932336 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x56268e9dd7e0 con 0x56268ed75270 2024-03-11T19:20:18.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.891+0000 7f4d4affd640 1 -- 172.21.15.29:0/741932336 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4d5c01d3f0 con 0x56268ed75270 2024-03-11T19:20:18.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.891+0000 7f4d747fe740 10 client.27451 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:18.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.891+0000 7f4d747fe740 10 client.27451 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:20:18.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.891+0000 7f4d747fe740 20 client.27451 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:18.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.891+0000 7f4d747fe740 1 --2- 172.21.15.29:0/741932336 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56268ed8b690 0x56268ed8dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.891+0000 7f4d747fe740 4 client.27451 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:20:18.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.891+0000 7f4d747fe740 1 -- 172.21.15.29:0/741932336 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56268ea1a720 con 0x56268ed8b690 2024-03-11T19:20:18.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.892+0000 7f4d6b7fe640 1 --2- 172.21.15.29:0/741932336 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56268ed8b690 0x56268ed8dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.893+0000 7f4d6b7fe640 1 --2- 172.21.15.29:0/741932336 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56268ed8b690 0x56268ed8dad0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5619da221e4e231 server_cookie=4f9909f652575d00 in_seq=0 out_seq=0 2024-03-11T19:20:18.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.893+0000 7f4d4affd640 10 client.27451 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:18.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.893+0000 7f4d4affd640 1 -- 172.21.15.29:0/741932336 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56268ea1a720 con 0x56268ed8b690 2024-03-11T19:20:18.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.893+0000 7f4d4affd640 10 client.27451 handle_command_reply: tid=0 2024-03-11T19:20:18.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.893+0000 7f4d747fe740 1 client.27451 shutdown 2024-03-11T19:20:18.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.894+0000 7f4d747fe740 1 -- 172.21.15.29:0/741932336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 msgr2=0x56268ed7f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.894+0000 7f4d747fe740 1 --2- 172.21.15.29:0/741932336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed7f460 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c01ca90 tx=0x7f4d5c009f70 comp rx=0 tx=0).stop 2024-03-11T19:20:18.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.894+0000 7f4d747fe740 1 -- 172.21.15.29:0/741932336 shutdown_connections 2024-03-11T19:20:18.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.894+0000 7f4d747fe740 1 --2- 172.21.15.29:0/741932336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 0x56268ed869e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.894+0000 7f4d747fe740 1 --2- 172.21.15.29:0/741932336 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56268ed8b690 0x56268ed8dad0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.894+0000 7f4d4affd640 0 client.27451 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:18.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.894+0000 7f4d747fe740 1 --2- 172.21.15.29:0/741932336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 0x56268ed7f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.894+0000 7f4d747fe740 1 --2- 172.21.15.29:0/741932336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed7f460 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.894+0000 7f4d747fe740 1 -- 172.21.15.29:0/741932336 >> 172.21.15.29:0/741932336 conn(0x56268ed707e0 msgr2=0x56268ed6fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.894+0000 7f4d747fe740 1 -- 172.21.15.29:0/741932336 shutdown_connections 2024-03-11T19:20:18.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.894+0000 7f4d747fe740 1 -- 172.21.15.29:0/741932336 wait complete. 2024-03-11T19:20:18.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.895+0000 7f4d747fe740 20 client.27451 trim_cache size 0 max 16384 2024-03-11T19:20:18.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.918+0000 7f4d747fe740 1 Processor -- start 2024-03-11T19:20:18.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.919+0000 7f4d747fe740 1 -- start start 2024-03-11T19:20:18.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.919+0000 7f4d747fe740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed70bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.919+0000 7f4d747fe740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 0x56268ed869a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.919+0000 7f4d747fe740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 0x56268ed7f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.919+0000 7f4d747fe740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268ec63130 con 0x56268ed75c60 2024-03-11T19:20:18.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.919+0000 7f4d747fe740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268e9dd7e0 con 0x56268ed7add0 2024-03-11T19:20:18.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.919+0000 7f4d747fe740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268ed705b0 con 0x56268ed75270 2024-03-11T19:20:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.919+0000 7f4d6b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 0x56268ed869a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.919+0000 7f4d6b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 0x56268ed869a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32888/0 (socket says 172.21.15.29:32888) 2024-03-11T19:20:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.919+0000 7f4d6b7fe640 1 -- 172.21.15.29:0/2698445741 learned_addr learned my addr 172.21.15.29:0/2698445741 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.919+0000 7f4d6affd640 1 --2- 172.21.15.29:0/2698445741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed70bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.919+0000 7f4d6affd640 1 --2- 172.21.15.29:0/2698445741 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 0x56268ed7f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.920+0000 7f4d6affd640 1 -- 172.21.15.29:0/2698445741 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 msgr2=0x56268ed7f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.920+0000 7f4d6affd640 1 --2- 172.21.15.29:0/2698445741 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 0x56268ed7f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.920+0000 7f4d6affd640 1 -- 172.21.15.29:0/2698445741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 msgr2=0x56268ed869a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.920+0000 7f4d6affd640 1 --2- 172.21.15.29:0/2698445741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 0x56268ed869a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.920+0000 7f4d6affd640 1 -- 172.21.15.29:0/2698445741 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56268ec78c20 con 0x56268ed75270 2024-03-11T19:20:18.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.920+0000 7f4d6affd640 1 --2- 172.21.15.29:0/2698445741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed70bc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c009a50 tx=0x7f4d5c01cf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9d4637aa650227c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.920+0000 7f4d517fa640 1 -- 172.21.15.29:0/2698445741 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c00a510 con 0x56268ed75270 2024-03-11T19:20:18.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.920+0000 7f4d517fa640 1 -- 172.21.15.29:0/2698445741 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4d5c01d3f0 con 0x56268ed75270 2024-03-11T19:20:18.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.921+0000 7f4d517fa640 1 -- 172.21.15.29:0/2698445741 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c031400 con 0x56268ed75270 2024-03-11T19:20:18.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.921+0000 7f4d747fe740 1 -- 172.21.15.29:0/2698445741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 msgr2=0x56268ed70bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.921+0000 7f4d747fe740 1 --2- 172.21.15.29:0/2698445741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed70bc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c009a50 tx=0x7f4d5c01cf50 comp rx=0 tx=0).stop 2024-03-11T19:20:18.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.922+0000 7f4d747fe740 1 -- 172.21.15.29:0/2698445741 shutdown_connections 2024-03-11T19:20:18.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.922+0000 7f4d747fe740 1 --2- 172.21.15.29:0/2698445741 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 0x56268ed7f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.922+0000 7f4d747fe740 1 --2- 172.21.15.29:0/2698445741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 0x56268ed869a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.922+0000 7f4d747fe740 1 --2- 172.21.15.29:0/2698445741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed70bc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.922+0000 7f4d747fe740 1 -- 172.21.15.29:0/2698445741 >> 172.21.15.29:0/2698445741 conn(0x56268ed707e0 msgr2=0x56268ed7de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.922+0000 7f4d747fe740 1 -- 172.21.15.29:0/2698445741 shutdown_connections 2024-03-11T19:20:18.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.922+0000 7f4d747fe740 1 -- 172.21.15.29:0/2698445741 wait complete. 2024-03-11T19:20:18.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.923+0000 7f4d747fe740 1 -- start start 2024-03-11T19:20:18.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.923+0000 7f4d747fe740 1 Processor -- start 2024-03-11T19:20:18.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.924+0000 7f4d747fe740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed868d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.924+0000 7f4d747fe740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 0x56268ed84ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.924+0000 7f4d6affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed868d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.924+0000 7f4d6affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed868d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:52442/0 (socket says 172.21.15.29:52442) 2024-03-11T19:20:18.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.924+0000 7f4d6affd640 1 -- 172.21.15.29:0/2764464766 learned_addr learned my addr 172.21.15.29:0/2764464766 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:18.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.925+0000 7f4d747fe740 1 --2- 172.21.15.29:0/2764464766 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 0x56268ed85420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.925+0000 7f4d747fe740 1 -- 172.21.15.29:0/2764464766 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268ed85a60 con 0x56268ed75c60 2024-03-11T19:20:18.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.925+0000 7f4d747fe740 1 -- 172.21.15.29:0/2764464766 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268ed85bd0 con 0x56268ed7add0 2024-03-11T19:20:18.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.925+0000 7f4d747fe740 1 -- 172.21.15.29:0/2764464766 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268ed85d90 con 0x56268ed75270 2024-03-11T19:20:18.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.925+0000 7f4d6affd640 1 --2- 172.21.15.29:0/2764464766 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 0x56268ed85420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.925+0000 7f4d6b7fe640 1 --2- 172.21.15.29:0/2764464766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 0x56268ed84ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.925+0000 7f4d6affd640 1 -- 172.21.15.29:0/2764464766 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 msgr2=0x56268ed85420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.925+0000 7f4d6affd640 1 --2- 172.21.15.29:0/2764464766 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 0x56268ed85420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.925+0000 7f4d6affd640 1 -- 172.21.15.29:0/2764464766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 msgr2=0x56268ed84ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:18.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.925+0000 7f4d6affd640 1 --2- 172.21.15.29:0/2764464766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 0x56268ed84ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.925+0000 7f4d6affd640 1 -- 172.21.15.29:0/2764464766 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56268ecdc6a0 con 0x56268ed75270 2024-03-11T19:20:18.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.925+0000 7f4d6b7fe640 1 --2- 172.21.15.29:0/2764464766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 0x56268ed84ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:20:18.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.926+0000 7f4d747fe740 10 client.0 _ll_register_callbacks cb 0x7f4d64005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:18.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.926+0000 7f4d6affd640 1 --2- 172.21.15.29:0/2764464766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed868d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c01d7d0 tx=0x7f4d5c004b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d9ab2d2d83b62aa server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:18.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.926+0000 7f4d747fe740 1 -- 172.21.15.29:0/2764464766 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4d600048a0 con 0x56268ed75270 2024-03-11T19:20:18.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.926+0000 7f4d4affd640 10 client.27457 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:20:18.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.926+0000 7f4d4affd640 1 -- 172.21.15.29:0/2764464766 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c008c70 con 0x56268ed75270 2024-03-11T19:20:18.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.926+0000 7f4d4affd640 1 -- 172.21.15.29:0/2764464766 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4d5c007000 con 0x56268ed75270 2024-03-11T19:20:18.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.927+0000 7f4d4affd640 1 -- 172.21.15.29:0/2764464766 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d5c01d560 con 0x56268ed75270 2024-03-11T19:20:18.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.927+0000 7f4d747fe740 10 client.27457 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:18.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.927+0000 7f4d747fe740 1 -- 172.21.15.29:0/2764464766 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x56268ec9e8c0 con 0x56268ed75270 2024-03-11T19:20:18.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.927+0000 7f4d4affd640 1 -- 172.21.15.29:0/2764464766 <== mon.1 v2:172.21.15.190:3300/0 4 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4d5c01dd20 con 0x56268ed75270 2024-03-11T19:20:18.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.927+0000 7f4d747fe740 10 client.27457 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:18.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.928+0000 7f4d747fe740 10 client.27457 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:20:18.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.928+0000 7f4d747fe740 20 client.27457 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:18.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.928+0000 7f4d747fe740 1 --2- 172.21.15.29:0/2764464766 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56268ed8b690 0x56268ede4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:18.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.928+0000 7f4d747fe740 4 client.27457 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:20:18.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.928+0000 7f4d747fe740 1 -- 172.21.15.29:0/2764464766 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f4d600048a0 con 0x56268ed8b690 2024-03-11T19:20:18.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.928+0000 7f4d6b7fe640 1 --2- 172.21.15.29:0/2764464766 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56268ed8b690 0x56268ede4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:18.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.928+0000 7f4d6b7fe640 1 --2- 172.21.15.29:0/2764464766 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56268ed8b690 0x56268ede4fd0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b926283a8ba05b84 server_cookie=1834e96792bbc8e6 in_seq=0 out_seq=0 2024-03-11T19:20:18.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.928+0000 7f4d4affd640 10 client.27457 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:18.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.929+0000 7f4d4affd640 1 -- 172.21.15.29:0/2764464766 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f4d600048a0 con 0x56268ed8b690 2024-03-11T19:20:18.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.929+0000 7f4d4affd640 10 client.27457 handle_command_reply: tid=0 2024-03-11T19:20:18.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.929+0000 7f4d747fe740 1 client.27457 shutdown 2024-03-11T19:20:18.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.929+0000 7f4d747fe740 1 -- 172.21.15.29:0/2764464766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 msgr2=0x56268ed868d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.929+0000 7f4d747fe740 1 --2- 172.21.15.29:0/2764464766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed868d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c01d7d0 tx=0x7f4d5c004b50 comp rx=0 tx=0).stop 2024-03-11T19:20:18.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.930+0000 7f4d747fe740 1 -- 172.21.15.29:0/2764464766 shutdown_connections 2024-03-11T19:20:18.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.930+0000 7f4d747fe740 1 --2- 172.21.15.29:0/2764464766 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56268ed7add0 0x56268ed85420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.930+0000 7f4d747fe740 1 --2- 172.21.15.29:0/2764464766 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56268ed8b690 0x56268ede4fd0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.930+0000 7f4d747fe740 1 --2- 172.21.15.29:0/2764464766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56268ed75c60 0x56268ed84ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.930+0000 7f4d4affd640 0 client.27457 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:18.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.930+0000 7f4d747fe740 1 --2- 172.21.15.29:0/2764464766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56268ed75270 0x56268ed868d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.930+0000 7f4d747fe740 1 -- 172.21.15.29:0/2764464766 >> 172.21.15.29:0/2764464766 conn(0x56268ed707e0 msgr2=0x56268ed70bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.930+0000 7f4d747fe740 1 -- 172.21.15.29:0/2764464766 shutdown_connections 2024-03-11T19:20:18.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.930+0000 7f4d747fe740 1 -- 172.21.15.29:0/2764464766 wait complete. 2024-03-11T19:20:18.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.930+0000 7f4d747fe740 20 client.27457 trim_cache size 0 max 16384 2024-03-11T19:20:18.936 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:20:18.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.931+0000 7f4d71c2a640 1 -- 172.21.15.29:0/3255501383 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4d540a1950 msgr2=0x7f4d540a3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.931+0000 7f4d71c2a640 1 --2- 172.21.15.29:0/3255501383 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4d540a1950 0x7f4d540a3e10 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4d64009380 tx=0x7f4d640962b0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.931+0000 7f4d71c2a640 1 -- 172.21.15.29:0/3255501383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d6c109350 msgr2=0x7f4d6c1a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:18.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.931+0000 7f4d71c2a640 1 --2- 172.21.15.29:0/3255501383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d6c109350 0x7f4d6c1a4dd0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4d6400e970 tx=0x7f4d6400ed70 comp rx=0 tx=0).stop 2024-03-11T19:20:18.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.932+0000 7f4d71c2a640 1 -- 172.21.15.29:0/3255501383 shutdown_connections 2024-03-11T19:20:18.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.932+0000 7f4d71c2a640 1 --2- 172.21.15.29:0/3255501383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d6c109350 0x7f4d6c1a4dd0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.932+0000 7f4d71c2a640 1 --2- 172.21.15.29:0/3255501383 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f4d540a1950 0x7f4d540a3e10 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.932+0000 7f4d71c2a640 1 --2- 172.21.15.29:0/3255501383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d6c108990 0x7f4d6c19dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.932+0000 7f4d71c2a640 1 --2- 172.21.15.29:0/3255501383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d6c1077a0 0x7f4d6c19d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:18.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.932+0000 7f4d71c2a640 1 -- 172.21.15.29:0/3255501383 >> 172.21.15.29:0/3255501383 conn(0x7f4d6c076730 msgr2=0x7f4d6c10bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:18.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.932+0000 7f4d71c2a640 1 -- 172.21.15.29:0/3255501383 shutdown_connections 2024-03-11T19:20:18.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:18.932+0000 7f4d71c2a640 1 -- 172.21.15.29:0/3255501383 wait complete. 2024-03-11T19:20:18.958 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:18 smithi029 ceph-mon[32439]: pgmap v666: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 52 MiB/s wr, 1.01k op/s 2024-03-11T19:20:19.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:18 smithi190 ceph-mon[36292]: pgmap v666: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 52 MiB/s wr, 1.01k op/s 2024-03-11T19:20:19.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:18 smithi170 ceph-mon[35690]: pgmap v666: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 52 MiB/s wr, 1.01k op/s 2024-03-11T19:20:19.960 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:20:19.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:20:19.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:20:20.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.165+0000 7f0fb4899640 1 -- 172.21.15.29:0/537071336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fa80a5600 msgr2=0x7f0fa80a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:20.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.165+0000 7f0fb4899640 1 --2- 172.21.15.29:0/537071336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fa80a5600 0x7f0fa80a61d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c009860 tx=0x7f0f9c02bb90 comp rx=0 tx=0).stop 2024-03-11T19:20:20.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.166+0000 7f0fb4899640 1 -- 172.21.15.29:0/537071336 shutdown_connections 2024-03-11T19:20:20.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.166+0000 7f0fb4899640 1 --2- 172.21.15.29:0/537071336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fa80ab410 0x7f0fa80ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.166+0000 7f0fb4899640 1 --2- 172.21.15.29:0/537071336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fa80a6710 0x7f0fa80a6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.166+0000 7f0fb4899640 1 --2- 172.21.15.29:0/537071336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fa80a5600 0x7f0fa80a61d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.166+0000 7f0fb4899640 1 -- 172.21.15.29:0/537071336 >> 172.21.15.29:0/537071336 conn(0x7f0fa80a0530 msgr2=0x7f0fa80a2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:20.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.169+0000 7f0fb4899640 1 -- 172.21.15.29:0/537071336 shutdown_connections 2024-03-11T19:20:20.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.169+0000 7f0fb4899640 1 -- 172.21.15.29:0/537071336 wait complete. 2024-03-11T19:20:20.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.170+0000 7f0fb4899640 1 Processor -- start 2024-03-11T19:20:20.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.170+0000 7f0fb4899640 1 -- start start 2024-03-11T19:20:20.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.170+0000 7f0fb4899640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fa80a5600 0x7f0fa8143440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.170+0000 7f0fb4899640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fa80a6710 0x7f0fa8143980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.171+0000 7f0fb4899640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fa80ab410 0x7f0fa814a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.171+0000 7f0fb4899640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa814af30 con 0x7f0fa80a5600 2024-03-11T19:20:20.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.171+0000 7f0fb4899640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa814b0a0 con 0x7f0fa80ab410 2024-03-11T19:20:20.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.171+0000 7f0fb4899640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa814b260 con 0x7f0fa80a6710 2024-03-11T19:20:20.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.171+0000 7f0faeffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fa80a6710 0x7f0fa8143980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.171+0000 7f0faeffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fa80a6710 0x7f0fa8143980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:52460/0 (socket says 172.21.15.29:52460) 2024-03-11T19:20:20.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.171+0000 7f0faeffd640 1 -- 172.21.15.29:0/2843147512 learned_addr learned my addr 172.21.15.29:0/2843147512 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:20.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.171+0000 7f0faf7fe640 1 --2- 172.21.15.29:0/2843147512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fa80a5600 0x7f0fa8143440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.171+0000 7f0faffff640 1 --2- 172.21.15.29:0/2843147512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fa80ab410 0x7f0fa814a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.171+0000 7f0faeffd640 1 -- 172.21.15.29:0/2843147512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fa80ab410 msgr2=0x7f0fa814a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:20.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.171+0000 7f0faeffd640 1 --2- 172.21.15.29:0/2843147512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fa80ab410 0x7f0fa814a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.171+0000 7f0faeffd640 1 -- 172.21.15.29:0/2843147512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fa80a5600 msgr2=0x7f0fa8143440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:20.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.171+0000 7f0faeffd640 1 --2- 172.21.15.29:0/2843147512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fa80a5600 0x7f0fa8143440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.171+0000 7f0faeffd640 1 -- 172.21.15.29:0/2843147512 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9c009490 con 0x7f0fa80a6710 2024-03-11T19:20:20.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.173+0000 7f0faeffd640 1 --2- 172.21.15.29:0/2843147512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fa80a6710 0x7f0fa8143980 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400b450 tx=0x7f0fa400b850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:20.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.173+0000 7f0facff9640 1 -- 172.21.15.29:0/2843147512 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa4011830 con 0x7f0fa80a6710 2024-03-11T19:20:20.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.173+0000 7f0facff9640 1 -- 172.21.15.29:0/2843147512 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0fa4011e40 con 0x7f0fa80a6710 2024-03-11T19:20:20.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.174+0000 7f0facff9640 1 -- 172.21.15.29:0/2843147512 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa400f2e0 con 0x7f0fa80a6710 2024-03-11T19:20:20.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.175+0000 7f0fb4899640 1 -- 172.21.15.29:0/2843147512 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa814b540 con 0x7f0fa80a6710 2024-03-11T19:20:20.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.175+0000 7f0fb4899640 1 -- 172.21.15.29:0/2843147512 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa814ba50 con 0x7f0fa80a6710 2024-03-11T19:20:20.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.179+0000 7f0fb87ac740 1 Processor -- start 2024-03-11T19:20:20.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.180+0000 7f0fb87ac740 1 -- start start 2024-03-11T19:20:20.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.180+0000 7f0fb87ac740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e4813690 0x5627e4813ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.180+0000 7f0fb87ac740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4814860 0x5627e4814ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.180+0000 7f0fb87ac740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e4815220 0x5627e481bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.180+0000 7f0fb87ac740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e4782f90 con 0x5627e4814860 2024-03-11T19:20:20.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.180+0000 7f0fb87ac740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e47804c0 con 0x5627e4815220 2024-03-11T19:20:20.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.180+0000 7f0fb87ac740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e477bca0 con 0x5627e4813690 2024-03-11T19:20:20.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.180+0000 7f0facff9640 1 -- 172.21.15.29:0/2843147512 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0fa4011990 con 0x7f0fa80a6710 2024-03-11T19:20:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.181+0000 7f0faeffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e4815220 0x5627e481bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.181+0000 7f0faeffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e4815220 0x5627e481bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37264/0 (socket says 172.21.15.29:37264) 2024-03-11T19:20:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.181+0000 7f0faeffd640 1 -- 172.21.15.29:0/933220277 learned_addr learned my addr 172.21.15.29:0/933220277 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.181+0000 7f0faffff640 1 --2- 172.21.15.29:0/933220277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e4813690 0x5627e4813ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.181+0000 7f0faeffd640 1 -- 172.21.15.29:0/933220277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e4813690 msgr2=0x5627e4813ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.181+0000 7f0faeffd640 1 --2- 172.21.15.29:0/933220277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e4813690 0x5627e4813ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.181+0000 7f0faeffd640 1 -- 172.21.15.29:0/933220277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4814860 msgr2=0x5627e4814ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:20:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.181+0000 7f0faeffd640 1 --2- 172.21.15.29:0/933220277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4814860 0x5627e4814ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.181+0000 7f0faeffd640 1 -- 172.21.15.29:0/933220277 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627e473e0b0 con 0x5627e4815220 2024-03-11T19:20:20.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.181+0000 7f0facff9640 1 --2- 172.21.15.29:0/2843147512 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0fa00a19f0 0x7f0fa00a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.182+0000 7f0facff9640 1 -- 172.21.15.29:0/2843147512 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0fa40ccb90 con 0x7f0fa80a6710 2024-03-11T19:20:20.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.182+0000 7f0faffff640 1 --2- 172.21.15.29:0/2843147512 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0fa00a19f0 0x7f0fa00a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.182+0000 7f0faeffd640 1 --2- 172.21.15.29:0/933220277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e4815220 0x5627e481bab0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0fa40d0a00 tx=0x7f0fa40d0ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b16a6812bc22c70 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:20.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.182+0000 7f0f99ffb640 1 -- 172.21.15.29:0/933220277 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa40092a0 con 0x5627e4815220 2024-03-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.182+0000 7f0f99ffb640 1 -- 172.21.15.29:0/933220277 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0fa40d3070 con 0x5627e4815220 2024-03-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.183+0000 7f0f99ffb640 1 -- 172.21.15.29:0/933220277 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa4099640 con 0x5627e4815220 2024-03-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.183+0000 7f0fb87ac740 1 -- 172.21.15.29:0/933220277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e4815220 msgr2=0x5627e481bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.183+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/933220277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e4815220 0x5627e481bab0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0fa40d0a00 tx=0x7f0fa40d0ec0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.183+0000 7f0faffff640 1 --2- 172.21.15.29:0/2843147512 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0fa00a19f0 0x7f0fa00a3eb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0fb00507e0 tx=0x7f0fb0074040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.184+0000 7f0fb87ac740 1 -- 172.21.15.29:0/933220277 shutdown_connections 2024-03-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.184+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/933220277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e4815220 0x5627e481bab0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.184+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/933220277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4814860 0x5627e4814ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.184+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/933220277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e4813690 0x5627e4813ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.184+0000 7f0fb87ac740 1 -- 172.21.15.29:0/933220277 >> 172.21.15.29:0/933220277 conn(0x5627e480e870 msgr2=0x5627e480ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.184+0000 7f0fb87ac740 1 -- 172.21.15.29:0/933220277 shutdown_connections 2024-03-11T19:20:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.185+0000 7f0fb87ac740 1 -- 172.21.15.29:0/933220277 wait complete. 2024-03-11T19:20:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.185+0000 7f0fb87ac740 1 -- start start 2024-03-11T19:20:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.185+0000 7f0fb87ac740 1 Processor -- start 2024-03-11T19:20:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.185+0000 7f0fb87ac740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e4813690 0x5627e481d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.185+0000 7f0fb87ac740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e4815220 0x5627e481df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.185+0000 7f0fb87ac740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4822f30 0x5627e4825320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.185+0000 7f0fb87ac740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e4720f40 con 0x5627e4822f30 2024-03-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.185+0000 7f0fb87ac740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e444f4b0 con 0x5627e4813690 2024-03-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.185+0000 7f0fb87ac740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e4701670 con 0x5627e4815220 2024-03-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.186+0000 7f0fb87ac740 10 client.0 _ll_register_callbacks cb 0x5627e476ef70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.186+0000 7f0faeffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e4815220 0x5627e481df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.186+0000 7f0faeffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e4815220 0x5627e481df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:52480/0 (socket says 172.21.15.29:52480) 2024-03-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.186+0000 7f0faeffd640 1 -- 172.21.15.29:0/3227772402 learned_addr learned my addr 172.21.15.29:0/3227772402 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.186+0000 7f0faf7fe640 1 --2- 172.21.15.29:0/3227772402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e4813690 0x5627e481d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.186+0000 7f0faffff640 1 --2- 172.21.15.29:0/3227772402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4822f30 0x5627e4825320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.186+0000 7f0faeffd640 1 -- 172.21.15.29:0/3227772402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e4813690 msgr2=0x5627e481d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.186+0000 7f0faeffd640 1 --2- 172.21.15.29:0/3227772402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e4813690 0x5627e481d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.186+0000 7f0faeffd640 1 -- 172.21.15.29:0/3227772402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4822f30 msgr2=0x5627e4825320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.186+0000 7f0faeffd640 1 --2- 172.21.15.29:0/3227772402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4822f30 0x5627e4825320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.186+0000 7f0faeffd640 1 -- 172.21.15.29:0/3227772402 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa40d6040 con 0x5627e4815220 2024-03-11T19:20:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.186+0000 7f0faffff640 1 --2- 172.21.15.29:0/3227772402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4822f30 0x5627e4825320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:20:20.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.187+0000 7f0faeffd640 1 --2- 172.21.15.29:0/3227772402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e4815220 0x5627e481df00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4095f10 tx=0x7f0fa40d0710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a58a0afa7d8812e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.187+0000 7f0f8effd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.187+0000 7f0f8effd640 1 -- 172.21.15.29:0/3227772402 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa4099640 con 0x5627e4815220 2024-03-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.187+0000 7f0fb87ac740 1 -- 172.21.15.29:0/3227772402 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5627e4755f40 con 0x5627e4815220 2024-03-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.187+0000 7f0f8effd640 1 -- 172.21.15.29:0/3227772402 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0fa409a200 con 0x5627e4815220 2024-03-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.188+0000 7f0f8effd640 1 -- 172.21.15.29:0/3227772402 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fa4099460 con 0x5627e4815220 2024-03-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.188+0000 7f0fb87ac740 10 client.27469 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.188+0000 7f0fb87ac740 1 -- 172.21.15.29:0/3227772402 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5627e477a320 con 0x5627e4815220 2024-03-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.188+0000 7f0f8effd640 1 -- 172.21.15.29:0/3227772402 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5627e4755f40 con 0x5627e4815220 2024-03-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.188+0000 7f0f8effd640 1 -- 172.21.15.29:0/3227772402 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0fa4096170 con 0x5627e4815220 2024-03-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.188+0000 7f0fb87ac740 10 client.27469 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.188+0000 7f0fb87ac740 10 client.27469 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.188+0000 7f0fb87ac740 20 client.27469 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.188+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/3227772402 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5627e483acf0 0x5627e483d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.188+0000 7f0fb87ac740 4 client.27469 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.188+0000 7f0fb87ac740 1 -- 172.21.15.29:0/3227772402 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5627e4772310 con 0x5627e483acf0 2024-03-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.190+0000 7f0faf7fe640 1 --2- 172.21.15.29:0/3227772402 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5627e483acf0 0x5627e483d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.192+0000 7f0faf7fe640 1 --2- 172.21.15.29:0/3227772402 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5627e483acf0 0x5627e483d110 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b69cadc0f649bf5 server_cookie=fd4e2484354a31c4 in_seq=0 out_seq=0 2024-03-11T19:20:20.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.192+0000 7f0f8effd640 10 client.27469 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:20.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.192+0000 7f0f8effd640 1 -- 172.21.15.29:0/3227772402 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5627e4772310 con 0x5627e483acf0 2024-03-11T19:20:20.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.192+0000 7f0f8effd640 10 client.27469 handle_command_reply: tid=0 2024-03-11T19:20:20.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.193+0000 7f0fb87ac740 1 client.27469 shutdown 2024-03-11T19:20:20.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.193+0000 7f0fb87ac740 1 -- 172.21.15.29:0/3227772402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e4815220 msgr2=0x5627e481df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:20.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.193+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/3227772402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e4815220 0x5627e481df00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4095f10 tx=0x7f0fa40d0710 comp rx=0 tx=0).stop 2024-03-11T19:20:20.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.193+0000 7f0fb87ac740 1 -- 172.21.15.29:0/3227772402 shutdown_connections 2024-03-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.193+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/3227772402 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5627e483acf0 0x5627e483d110 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.193+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/3227772402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4822f30 0x5627e4825320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.193+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/3227772402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e4815220 0x5627e481df00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.193+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/3227772402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e4813690 0x5627e481d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.193+0000 7f0fb87ac740 1 -- 172.21.15.29:0/3227772402 >> 172.21.15.29:0/3227772402 conn(0x5627e480e870 msgr2=0x5627e480dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.194+0000 7f0f8effd640 0 client.27469 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.194+0000 7f0fb87ac740 1 -- 172.21.15.29:0/3227772402 shutdown_connections 2024-03-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.194+0000 7f0fb87ac740 1 -- 172.21.15.29:0/3227772402 wait complete. 2024-03-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.195+0000 7f0fb87ac740 20 client.27469 trim_cache size 0 max 16384 2024-03-11T19:20:20.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.220+0000 7f0fb87ac740 1 Processor -- start 2024-03-11T19:20:20.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.220+0000 7f0fb87ac740 1 -- start start 2024-03-11T19:20:20.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.220+0000 7f0fb87ac740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e480e870 0x5627e4841a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.220+0000 7f0fb87ac740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e4813690 0x5627e484afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.220+0000 7f0fb87ac740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4815220 0x5627e484b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.220+0000 7f0fb87ac740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e4817cc0 con 0x5627e4815220 2024-03-11T19:20:20.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.221+0000 7f0fb87ac740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e4809d70 con 0x5627e4813690 2024-03-11T19:20:20.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.221+0000 7f0fb87ac740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e4836fb0 con 0x5627e480e870 2024-03-11T19:20:20.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.221+0000 7f0faf7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e480e870 0x5627e4841a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.221+0000 7f0faf7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e480e870 0x5627e4841a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:52494/0 (socket says 172.21.15.29:52494) 2024-03-11T19:20:20.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.221+0000 7f0faf7fe640 1 -- 172.21.15.29:0/969302143 learned_addr learned my addr 172.21.15.29:0/969302143 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:20.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.221+0000 7f0faffff640 1 --2- 172.21.15.29:0/969302143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4815220 0x5627e484b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.221+0000 7f0faeffd640 1 --2- 172.21.15.29:0/969302143 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e4813690 0x5627e484afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.222+0000 7f0faffff640 1 -- 172.21.15.29:0/969302143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e480e870 msgr2=0x5627e4841a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:20.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.222+0000 7f0faffff640 1 --2- 172.21.15.29:0/969302143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e480e870 0x5627e4841a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.222+0000 7f0faffff640 1 -- 172.21.15.29:0/969302143 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e4813690 msgr2=0x5627e484afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:20.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.222+0000 7f0faffff640 1 --2- 172.21.15.29:0/969302143 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e4813690 0x5627e484afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.222+0000 7f0faffff640 1 -- 172.21.15.29:0/969302143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627e4716c00 con 0x5627e4815220 2024-03-11T19:20:20.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.222+0000 7f0faffff640 1 --2- 172.21.15.29:0/969302143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4815220 0x5627e484b4f0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0000b60 tx=0x7f0fb0079000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5239041af1965bbe server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:20.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.223+0000 7f0f997fa640 1 -- 172.21.15.29:0/969302143 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0068d60 con 0x5627e4815220 2024-03-11T19:20:20.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.223+0000 7f0f997fa640 1 -- 172.21.15.29:0/969302143 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0fb0078040 con 0x5627e4815220 2024-03-11T19:20:20.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.223+0000 7f0f997fa640 1 -- 172.21.15.29:0/969302143 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb007e650 con 0x5627e4815220 2024-03-11T19:20:20.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.224+0000 7f0fb87ac740 1 -- 172.21.15.29:0/969302143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4815220 msgr2=0x5627e484b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:20.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.224+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/969302143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4815220 0x5627e484b4f0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0000b60 tx=0x7f0fb0079000 comp rx=0 tx=0).stop 2024-03-11T19:20:20.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.224+0000 7f0fb87ac740 1 -- 172.21.15.29:0/969302143 shutdown_connections 2024-03-11T19:20:20.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.224+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/969302143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4815220 0x5627e484b4f0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.224+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/969302143 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e4813690 0x5627e484afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.224+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/969302143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e480e870 0x5627e4841a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.224+0000 7f0fb87ac740 1 -- 172.21.15.29:0/969302143 >> 172.21.15.29:0/969302143 conn(0x5627e4809eb0 msgr2=0x5627e480ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:20.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.229+0000 7f0fb87ac740 1 -- 172.21.15.29:0/969302143 shutdown_connections 2024-03-11T19:20:20.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.231+0000 7f0fb87ac740 1 -- 172.21.15.29:0/969302143 wait complete. 2024-03-11T19:20:20.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.232+0000 7f0fb87ac740 1 -- start start 2024-03-11T19:20:20.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.232+0000 7f0fb87ac740 1 Processor -- start 2024-03-11T19:20:20.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.233+0000 7f0fb87ac740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e480e870 0x5627e488bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.233+0000 7f0fb87ac740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4813690 0x5627e488e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.233+0000 7f0faf7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e480e870 0x5627e488bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.233+0000 7f0faf7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e480e870 0x5627e488bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37292/0 (socket says 172.21.15.29:37292) 2024-03-11T19:20:20.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.233+0000 7f0faf7fe640 1 -- 172.21.15.29:0/1094782765 learned_addr learned my addr 172.21.15.29:0/1094782765 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:20.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.233+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/1094782765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e4815220 0x5627e48907e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.233+0000 7f0fb87ac740 1 -- 172.21.15.29:0/1094782765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e4890ee0 con 0x5627e4813690 2024-03-11T19:20:20.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.233+0000 7f0faeffd640 1 --2- 172.21.15.29:0/1094782765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4813690 0x5627e488e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.233+0000 7f0fb87ac740 1 -- 172.21.15.29:0/1094782765 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e4891050 con 0x5627e480e870 2024-03-11T19:20:20.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.233+0000 7f0fb87ac740 1 -- 172.21.15.29:0/1094782765 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e4891210 con 0x5627e4815220 2024-03-11T19:20:20.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.234+0000 7f0faffff640 1 --2- 172.21.15.29:0/1094782765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e4815220 0x5627e48907e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.234+0000 7f0fb87ac740 10 client.0 _ll_register_callbacks cb 0x7f0fa40d0370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:20.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.235+0000 7f0faffff640 1 -- 172.21.15.29:0/1094782765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e480e870 msgr2=0x5627e488bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:20.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.235+0000 7f0faffff640 1 --2- 172.21.15.29:0/1094782765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e480e870 0x5627e488bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.235+0000 7f0faffff640 1 -- 172.21.15.29:0/1094782765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4813690 msgr2=0x5627e488e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:20.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.235+0000 7f0faffff640 1 --2- 172.21.15.29:0/1094782765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4813690 0x5627e488e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.235+0000 7f0faffff640 1 -- 172.21.15.29:0/1094782765 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb0069cd0 con 0x5627e4815220 2024-03-11T19:20:20.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.236+0000 7f0faffff640 1 --2- 172.21.15.29:0/1094782765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e4815220 0x5627e48907e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0000b60 tx=0x7f0fb0080a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0bac8bd0aba7649 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:20.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.236+0000 7f0f8effd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:20:20.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.236+0000 7f0f8effd640 1 -- 172.21.15.29:0/1094782765 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0089840 con 0x5627e4815220 2024-03-11T19:20:20.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.237+0000 7f0fb87ac740 1 -- 172.21.15.29:0/1094782765 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5627e47a1900 con 0x5627e4815220 2024-03-11T19:20:20.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.237+0000 7f0f8effd640 1 -- 172.21.15.29:0/1094782765 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0fb0080bf0 con 0x5627e4815220 2024-03-11T19:20:20.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.237+0000 7f0f8effd640 1 -- 172.21.15.29:0/1094782765 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fb0069650 con 0x5627e4815220 2024-03-11T19:20:20.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.237+0000 7f0fb87ac740 10 client.27475 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:20.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.237+0000 7f0fb87ac740 1 -- 172.21.15.29:0/1094782765 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5627e473c910 con 0x5627e4815220 2024-03-11T19:20:20.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.238+0000 7f0f8effd640 1 -- 172.21.15.29:0/1094782765 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0fb0099030 con 0x5627e4815220 2024-03-11T19:20:20.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.238+0000 7f0f8effd640 1 -- 172.21.15.29:0/1094782765 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0fb00939a0 con 0x5627e4815220 2024-03-11T19:20:20.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.238+0000 7f0fb87ac740 10 client.27475 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:20.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.238+0000 7f0fb87ac740 10 client.27475 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:20:20.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.238+0000 7f0fb87ac740 20 client.27475 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:20.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.238+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/1094782765 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5627e4822f30 0x5627e4894b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.238+0000 7f0fb87ac740 4 client.27475 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:20:20.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.238+0000 7f0fb87ac740 1 -- 172.21.15.29:0/1094782765 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5627e47a1900 con 0x5627e4822f30 2024-03-11T19:20:20.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.238+0000 7f0faf7fe640 1 --2- 172.21.15.29:0/1094782765 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5627e4822f30 0x5627e4894b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.239+0000 7f0f8effd640 10 client.27475 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:20.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.239+0000 7f0faf7fe640 1 --2- 172.21.15.29:0/1094782765 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5627e4822f30 0x5627e4894b20 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e9f9304825c2ddd server_cookie=2b2b1a8e090d06a0 in_seq=0 out_seq=0 2024-03-11T19:20:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.241+0000 7f0f8effd640 1 -- 172.21.15.29:0/1094782765 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5627e47a1900 con 0x5627e4822f30 2024-03-11T19:20:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.241+0000 7f0f8effd640 10 client.27475 handle_command_reply: tid=0 2024-03-11T19:20:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.241+0000 7f0fb87ac740 1 client.27475 shutdown 2024-03-11T19:20:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.242+0000 7f0fb87ac740 1 -- 172.21.15.29:0/1094782765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e4815220 msgr2=0x5627e48907e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.242+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/1094782765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e4815220 0x5627e48907e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0000b60 tx=0x7f0fb0080a00 comp rx=0 tx=0).stop 2024-03-11T19:20:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.242+0000 7f0fb87ac740 1 -- 172.21.15.29:0/1094782765 shutdown_connections 2024-03-11T19:20:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.242+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/1094782765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627e4815220 0x5627e48907e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.242+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/1094782765 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5627e4822f30 0x5627e4894b20 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.242+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/1094782765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5627e4813690 0x5627e488e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.242+0000 7f0f8effd640 0 client.27475 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.242+0000 7f0fb87ac740 1 --2- 172.21.15.29:0/1094782765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627e480e870 0x5627e488bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.242+0000 7f0fb87ac740 1 -- 172.21.15.29:0/1094782765 >> 172.21.15.29:0/1094782765 conn(0x5627e4809eb0 msgr2=0x5627e48183d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:20.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.243+0000 7f0fb87ac740 1 -- 172.21.15.29:0/1094782765 shutdown_connections 2024-03-11T19:20:20.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.243+0000 7f0fb87ac740 1 -- 172.21.15.29:0/1094782765 wait complete. 2024-03-11T19:20:20.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.243+0000 7f0fb87ac740 20 client.27475 trim_cache size 0 max 16384 2024-03-11T19:20:20.248 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:20:20.248 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:20:20.249 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "a4bf4efd-2380-4c24-a048-6aa71877e498", 2024-03-11T19:20:20.249 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:20:20.249 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:20:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.244+0000 7f0fb4899640 1 -- 172.21.15.29:0/2843147512 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0fa00a19f0 msgr2=0x7f0fa00a3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.244+0000 7f0fb4899640 1 --2- 172.21.15.29:0/2843147512 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0fa00a19f0 0x7f0fa00a3eb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0fb00507e0 tx=0x7f0fb0074040 comp rx=0 tx=0).stop 2024-03-11T19:20:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.244+0000 7f0fb4899640 1 -- 172.21.15.29:0/2843147512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fa80a6710 msgr2=0x7f0fa8143980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.244+0000 7f0fb4899640 1 --2- 172.21.15.29:0/2843147512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fa80a6710 0x7f0fa8143980 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400b450 tx=0x7f0fa400b850 comp rx=0 tx=0).stop 2024-03-11T19:20:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.245+0000 7f0fb4899640 1 -- 172.21.15.29:0/2843147512 shutdown_connections 2024-03-11T19:20:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.245+0000 7f0fb4899640 1 --2- 172.21.15.29:0/2843147512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fa80ab410 0x7f0fa814a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.245+0000 7f0fb4899640 1 --2- 172.21.15.29:0/2843147512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fa80a6710 0x7f0fa8143980 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.245+0000 7f0fb4899640 1 --2- 172.21.15.29:0/2843147512 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f0fa00a19f0 0x7f0fa00a3eb0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.245+0000 7f0fb4899640 1 --2- 172.21.15.29:0/2843147512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fa80a5600 0x7f0fa8143440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.245+0000 7f0fb4899640 1 -- 172.21.15.29:0/2843147512 >> 172.21.15.29:0/2843147512 conn(0x7f0fa80a0530 msgr2=0x7f0fa80b0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.245+0000 7f0fb4899640 1 -- 172.21.15.29:0/2843147512 shutdown_connections 2024-03-11T19:20:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.245+0000 7f0fb4899640 1 -- 172.21.15.29:0/2843147512 wait complete. 2024-03-11T19:20:20.273 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:20:20.273 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:20:20.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.469+0000 7f8b35281640 1 -- 172.21.15.29:0/1257060234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b30101270 msgr2=0x7f8b301016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:20.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.469+0000 7f8b35281640 1 --2- 172.21.15.29:0/1257060234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b30101270 0x7f8b301016f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8b24009900 tx=0x7f8b2402bc30 comp rx=0 tx=0).stop 2024-03-11T19:20:20.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.469+0000 7f8b35281640 1 -- 172.21.15.29:0/1257060234 shutdown_connections 2024-03-11T19:20:20.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.469+0000 7f8b35281640 1 --2- 172.21.15.29:0/1257060234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b30101c30 0x7f8b301084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.469+0000 7f8b35281640 1 --2- 172.21.15.29:0/1257060234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b30101270 0x7f8b301016f0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.469+0000 7f8b35281640 1 --2- 172.21.15.29:0/1257060234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b30100080 0x7f8b301004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.469+0000 7f8b35281640 1 -- 172.21.15.29:0/1257060234 >> 172.21.15.29:0/1257060234 conn(0x7f8b300fb5f0 msgr2=0x7f8b300fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:20.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.469+0000 7f8b35281640 1 -- 172.21.15.29:0/1257060234 shutdown_connections 2024-03-11T19:20:20.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.469+0000 7f8b35281640 1 -- 172.21.15.29:0/1257060234 wait complete. 2024-03-11T19:20:20.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.469+0000 7f8b35281640 1 Processor -- start 2024-03-11T19:20:20.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.470+0000 7f8b35281640 1 -- start start 2024-03-11T19:20:20.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.470+0000 7f8b35281640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b30100080 0x7f8b30196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.470+0000 7f8b35281640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b30101270 0x7f8b30197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.470+0000 7f8b35281640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b30101c30 0x7f8b3019e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.470+0000 7f8b35281640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3019ea30 con 0x7f8b30101270 2024-03-11T19:20:20.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.470+0000 7f8b35281640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3019eba0 con 0x7f8b30101c30 2024-03-11T19:20:20.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.470+0000 7f8b2e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b30101270 0x7f8b30197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.470+0000 7f8b35281640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3019ed60 con 0x7f8b30100080 2024-03-11T19:20:20.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.470+0000 7f8b2effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b30100080 0x7f8b30196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.470+0000 7f8b2e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b30101270 0x7f8b30197480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32964/0 (socket says 172.21.15.29:32964) 2024-03-11T19:20:20.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.470+0000 7f8b2effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b30100080 0x7f8b30196f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:52518/0 (socket says 172.21.15.29:52518) 2024-03-11T19:20:20.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.470+0000 7f8b2e7fc640 1 -- 172.21.15.29:0/1329127494 learned_addr learned my addr 172.21.15.29:0/1329127494 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:20.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.470+0000 7f8b2f7fe640 1 --2- 172.21.15.29:0/1329127494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b30101c30 0x7f8b3019e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.470+0000 7f8b2e7fc640 1 -- 172.21.15.29:0/1329127494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b30100080 msgr2=0x7f8b30196f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:20.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.470+0000 7f8b2e7fc640 1 --2- 172.21.15.29:0/1329127494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b30100080 0x7f8b30196f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.470+0000 7f8b2e7fc640 1 -- 172.21.15.29:0/1329127494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b30101c30 msgr2=0x7f8b3019e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:20.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.470+0000 7f8b2e7fc640 1 --2- 172.21.15.29:0/1329127494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b30101c30 0x7f8b3019e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.470+0000 7f8b2e7fc640 1 -- 172.21.15.29:0/1329127494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b24009560 con 0x7f8b30101270 2024-03-11T19:20:20.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.470+0000 7f8b2effd640 1 --2- 172.21.15.29:0/1329127494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b30100080 0x7f8b30196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:20:20.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.471+0000 7f8b2e7fc640 1 --2- 172.21.15.29:0/1329127494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b30101270 0x7f8b30197480 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8b24036040 tx=0x7f8b240042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:20.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.471+0000 7f8b17fff640 1 -- 172.21.15.29:0/1329127494 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b24005d50 con 0x7f8b30101270 2024-03-11T19:20:20.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.471+0000 7f8b35281640 1 -- 172.21.15.29:0/1329127494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b30109010 con 0x7f8b30101270 2024-03-11T19:20:20.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.471+0000 7f8b17fff640 1 -- 172.21.15.29:0/1329127494 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8b2402de90 con 0x7f8b30101270 2024-03-11T19:20:20.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.471+0000 7f8b17fff640 1 -- 172.21.15.29:0/1329127494 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b24005570 con 0x7f8b30101270 2024-03-11T19:20:20.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.471+0000 7f8b35281640 1 -- 172.21.15.29:0/1329127494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b30109520 con 0x7f8b30101270 2024-03-11T19:20:20.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.473+0000 7f8b17fff640 1 -- 172.21.15.29:0/1329127494 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8b24003800 con 0x7f8b30101270 2024-03-11T19:20:20.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.473+0000 7f8b37e51740 1 Processor -- start 2024-03-11T19:20:20.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.473+0000 7f8b37e51740 1 -- start start 2024-03-11T19:20:20.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.473+0000 7f8b17fff640 1 --2- 172.21.15.29:0/1329127494 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8b180a1a30 0x7f8b180a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.473+0000 7f8b37e51740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba4bf192d0 0x55ba4bf196f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.473+0000 7f8b37e51740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf1e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.473+0000 7f8b37e51740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf1ee30 0x55ba4bf212c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.473+0000 7f8b37e51740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba4be89280 con 0x55ba4bf1ee30 2024-03-11T19:20:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.473+0000 7f8b37e51740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba4be867b0 con 0x55ba4bf19cc0 2024-03-11T19:20:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.473+0000 7f8b37e51740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba4be81f90 con 0x55ba4bf192d0 2024-03-11T19:20:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.473+0000 7f8b2f7fe640 1 --2- 172.21.15.29:0/1329127494 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8b180a1a30 0x7f8b180a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.473+0000 7f8b17fff640 1 -- 172.21.15.29:0/1329127494 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f8b240efd70 con 0x7f8b30101270 2024-03-11T19:20:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.473+0000 7f8b2e7fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf1e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.474+0000 7f8b2f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf1ee30 0x55ba4bf212c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.474+0000 7f8b2e7fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf1e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:37318/0 (socket says 172.21.15.29:37318) 2024-03-11T19:20:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.474+0000 7f8b2e7fc640 1 -- 172.21.15.29:0/1636867756 learned_addr learned my addr 172.21.15.29:0/1636867756 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.474+0000 7f8b2f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf1ee30 0x55ba4bf212c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32966/0 (socket says 172.21.15.29:32966) 2024-03-11T19:20:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.474+0000 7f8b2effd640 1 --2- 172.21.15.29:0/1636867756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba4bf192d0 0x55ba4bf196f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.474+0000 7f8b2e7fc640 1 -- 172.21.15.29:0/1636867756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba4bf192d0 msgr2=0x55ba4bf196f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:20.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.474+0000 7f8b2e7fc640 1 --2- 172.21.15.29:0/1636867756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba4bf192d0 0x55ba4bf196f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.474+0000 7f8b2e7fc640 1 -- 172.21.15.29:0/1636867756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf1ee30 msgr2=0x55ba4bf212c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:20.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.474+0000 7f8b2e7fc640 1 --2- 172.21.15.29:0/1636867756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf1ee30 0x55ba4bf212c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.474+0000 7f8b2e7fc640 1 -- 172.21.15.29:0/1636867756 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba4be40570 con 0x55ba4bf19cc0 2024-03-11T19:20:20.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.474+0000 7f8b2f7fe640 1 --2- 172.21.15.29:0/1329127494 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8b180a1a30 0x7f8b180a3ef0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8b28005fd0 tx=0x7f8b28009f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:20.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.474+0000 7f8b2e7fc640 1 --2- 172.21.15.29:0/1636867756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf1e7c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8b240c5110 tx=0x7f8b240bc9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d5d485778fd78a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:20.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.474+0000 7f8b157fa640 1 -- 172.21.15.29:0/1636867756 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2409aa50 con 0x55ba4bf19cc0 2024-03-11T19:20:20.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.474+0000 7f8b157fa640 1 -- 172.21.15.29:0/1636867756 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8b2409a870 con 0x55ba4bf19cc0 2024-03-11T19:20:20.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.474+0000 7f8b157fa640 1 -- 172.21.15.29:0/1636867756 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b240bcea0 con 0x55ba4bf19cc0 2024-03-11T19:20:20.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.475+0000 7f8b37e51740 1 -- 172.21.15.29:0/1636867756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf19cc0 msgr2=0x55ba4bf1e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:20.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.475+0000 7f8b37e51740 1 --2- 172.21.15.29:0/1636867756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf1e7c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8b240c5110 tx=0x7f8b240bc9e0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.475+0000 7f8b37e51740 1 -- 172.21.15.29:0/1636867756 shutdown_connections 2024-03-11T19:20:20.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.475+0000 7f8b37e51740 1 --2- 172.21.15.29:0/1636867756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf1ee30 0x55ba4bf212c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.475+0000 7f8b37e51740 1 --2- 172.21.15.29:0/1636867756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf1e7c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.475+0000 7f8b37e51740 1 --2- 172.21.15.29:0/1636867756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba4bf192d0 0x55ba4bf196f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.475+0000 7f8b37e51740 1 -- 172.21.15.29:0/1636867756 >> 172.21.15.29:0/1636867756 conn(0x55ba4bf147f0 msgr2=0x55ba4bf14bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.475+0000 7f8b37e51740 1 -- 172.21.15.29:0/1636867756 shutdown_connections 2024-03-11T19:20:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.475+0000 7f8b37e51740 1 -- 172.21.15.29:0/1636867756 wait complete. 2024-03-11T19:20:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.475+0000 7f8b37e51740 1 -- start start 2024-03-11T19:20:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.475+0000 7f8b37e51740 1 Processor -- start 2024-03-11T19:20:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b37e51740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf192d0 0x55ba4bf23060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b37e51740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf235a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b37e51740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba4bf1ee30 0x55ba4bf2a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b37e51740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba4bb55430 con 0x55ba4bf192d0 2024-03-11T19:20:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b37e51740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba4bb11e00 con 0x55ba4bf19cc0 2024-03-11T19:20:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b37e51740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba4be06fd0 con 0x55ba4bf1ee30 2024-03-11T19:20:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b37e51740 10 client.0 _ll_register_callbacks cb 0x55ba4be5cfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b2effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf192d0 0x55ba4bf23060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b2effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf192d0 0x55ba4bf23060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32982/0 (socket says 172.21.15.29:32982) 2024-03-11T19:20:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b2effd640 1 -- 172.21.15.29:0/1303496409 learned_addr learned my addr 172.21.15.29:0/1303496409 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b2f7fe640 1 --2- 172.21.15.29:0/1303496409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba4bf1ee30 0x55ba4bf2a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b2e7fc640 1 --2- 172.21.15.29:0/1303496409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf235a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b2effd640 1 -- 172.21.15.29:0/1303496409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba4bf1ee30 msgr2=0x55ba4bf2a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b2effd640 1 --2- 172.21.15.29:0/1303496409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba4bf1ee30 0x55ba4bf2a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b2effd640 1 -- 172.21.15.29:0/1303496409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf19cc0 msgr2=0x55ba4bf235a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b2effd640 1 --2- 172.21.15.29:0/1303496409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf235a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b2effd640 1 -- 172.21.15.29:0/1303496409 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b30109520 con 0x55ba4bf192d0 2024-03-11T19:20:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b2e7fc640 1 --2- 172.21.15.29:0/1303496409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf235a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:20:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b2effd640 1 --2- 172.21.15.29:0/1303496409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf192d0 0x55ba4bf23060 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f8b20009f40 tx=0x7f8b2000b750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ec3ec208692627 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b12ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:20:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b12ff5640 1 -- 172.21.15.29:0/1303496409 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b20015050 con 0x55ba4bf192d0 2024-03-11T19:20:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b12ff5640 1 -- 172.21.15.29:0/1303496409 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f8b20004440 con 0x55ba4bf192d0 2024-03-11T19:20:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.476+0000 7f8b37e51740 1 -- 172.21.15.29:0/1303496409 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ba4bb817e0 con 0x55ba4bf192d0 2024-03-11T19:20:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.477+0000 7f8b12ff5640 1 -- 172.21.15.29:0/1303496409 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b200035f0 con 0x55ba4bf192d0 2024-03-11T19:20:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.477+0000 7f8b37e51740 10 client.17925 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.477+0000 7f8b37e51740 1 -- 172.21.15.29:0/1303496409 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ba4be80610 con 0x55ba4bf192d0 2024-03-11T19:20:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.477+0000 7f8b12ff5640 1 -- 172.21.15.29:0/1303496409 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55ba4bb817e0 con 0x55ba4bf192d0 2024-03-11T19:20:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.477+0000 7f8b12ff5640 1 -- 172.21.15.29:0/1303496409 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8b2001b7a0 con 0x55ba4bf192d0 2024-03-11T19:20:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.477+0000 7f8b37e51740 10 client.17925 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.477+0000 7f8b37e51740 10 client.17925 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:20:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.477+0000 7f8b37e51740 20 client.17925 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.477+0000 7f8b37e51740 1 --2- 172.21.15.29:0/1303496409 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ba4bf2f2b0 0x55ba4bf316c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.477+0000 7f8b37e51740 4 client.17925 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:20:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.477+0000 7f8b37e51740 1 -- 172.21.15.29:0/1303496409 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ba4bbbe720 con 0x55ba4bf2f2b0 2024-03-11T19:20:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.477+0000 7f8b2e7fc640 1 --2- 172.21.15.29:0/1303496409 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ba4bf2f2b0 0x55ba4bf316c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.478+0000 7f8b2e7fc640 1 --2- 172.21.15.29:0/1303496409 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ba4bf2f2b0 0x55ba4bf316c0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47bea7334cacbdee server_cookie=4a38dd0b0d8dd8e9 in_seq=0 out_seq=0 2024-03-11T19:20:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.478+0000 7f8b12ff5640 10 client.17925 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.478+0000 7f8b12ff5640 1 -- 172.21.15.29:0/1303496409 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ba4bbbe720 con 0x55ba4bf2f2b0 2024-03-11T19:20:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.478+0000 7f8b12ff5640 10 client.17925 handle_command_reply: tid=0 2024-03-11T19:20:20.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.478+0000 7f8b37e51740 1 client.17925 shutdown 2024-03-11T19:20:20.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.478+0000 7f8b37e51740 1 -- 172.21.15.29:0/1303496409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf192d0 msgr2=0x55ba4bf23060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:20.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.478+0000 7f8b37e51740 1 --2- 172.21.15.29:0/1303496409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf192d0 0x55ba4bf23060 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f8b20009f40 tx=0x7f8b2000b750 comp rx=0 tx=0).stop 2024-03-11T19:20:20.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.478+0000 7f8b37e51740 1 -- 172.21.15.29:0/1303496409 shutdown_connections 2024-03-11T19:20:20.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.478+0000 7f8b37e51740 1 --2- 172.21.15.29:0/1303496409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba4bf1ee30 0x55ba4bf2a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.478+0000 7f8b37e51740 1 --2- 172.21.15.29:0/1303496409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf235a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.478+0000 7f8b37e51740 1 --2- 172.21.15.29:0/1303496409 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ba4bf2f2b0 0x55ba4bf316c0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.478+0000 7f8b37e51740 1 --2- 172.21.15.29:0/1303496409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf192d0 0x55ba4bf23060 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.478+0000 7f8b37e51740 1 -- 172.21.15.29:0/1303496409 >> 172.21.15.29:0/1303496409 conn(0x55ba4bf147f0 msgr2=0x55ba4bf13a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:20.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.478+0000 7f8b12ff5640 0 client.17925 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:20.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.479+0000 7f8b37e51740 1 -- 172.21.15.29:0/1303496409 shutdown_connections 2024-03-11T19:20:20.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.479+0000 7f8b37e51740 1 -- 172.21.15.29:0/1303496409 wait complete. 2024-03-11T19:20:20.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.479+0000 7f8b37e51740 20 client.17925 trim_cache size 0 max 16384 2024-03-11T19:20:20.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b37e51740 1 Processor -- start 2024-03-11T19:20:20.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b37e51740 1 -- start start 2024-03-11T19:20:20.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b37e51740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf192d0 0x55ba4bf131c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b37e51740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf2a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b37e51740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf1ee30 0x55ba4bf13700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b37e51740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba4bf157b0 con 0x55ba4bf192d0 2024-03-11T19:20:20.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b37e51740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba4bf100f0 con 0x55ba4bf1ee30 2024-03-11T19:20:20.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b37e51740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba4bf1a0f0 con 0x55ba4bf19cc0 2024-03-11T19:20:20.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b2effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf192d0 0x55ba4bf131c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b2effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf192d0 0x55ba4bf131c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32990/0 (socket says 172.21.15.29:32990) 2024-03-11T19:20:20.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b2effd640 1 -- 172.21.15.29:0/1698348756 learned_addr learned my addr 172.21.15.29:0/1698348756 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:20.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b2f7fe640 1 --2- 172.21.15.29:0/1698348756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf1ee30 0x55ba4bf13700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b2effd640 1 -- 172.21.15.29:0/1698348756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba4bf19cc0 msgr2=0x55ba4bf2a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:20.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b2effd640 1 --2- 172.21.15.29:0/1698348756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf2a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b2effd640 1 -- 172.21.15.29:0/1698348756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf1ee30 msgr2=0x55ba4bf13700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:20.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b2effd640 1 --2- 172.21.15.29:0/1698348756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf1ee30 0x55ba4bf13700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b2effd640 1 -- 172.21.15.29:0/1698348756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba4be1cac0 con 0x55ba4bf192d0 2024-03-11T19:20:20.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b2f7fe640 1 --2- 172.21.15.29:0/1698348756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf1ee30 0x55ba4bf13700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:20:20.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b2effd640 1 --2- 172.21.15.29:0/1698348756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf192d0 0x55ba4bf131c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f8b20009f10 tx=0x7f8b2000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f559184d83347cb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:20.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.494+0000 7f8b14ff9640 1 -- 172.21.15.29:0/1698348756 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b20015070 con 0x55ba4bf192d0 2024-03-11T19:20:20.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.495+0000 7f8b14ff9640 1 -- 172.21.15.29:0/1698348756 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8b20016040 con 0x55ba4bf192d0 2024-03-11T19:20:20.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.495+0000 7f8b14ff9640 1 -- 172.21.15.29:0/1698348756 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b200035f0 con 0x55ba4bf192d0 2024-03-11T19:20:20.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.495+0000 7f8b37e51740 1 -- 172.21.15.29:0/1698348756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf192d0 msgr2=0x55ba4bf131c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:20.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.495+0000 7f8b37e51740 1 --2- 172.21.15.29:0/1698348756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf192d0 0x55ba4bf131c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f8b20009f10 tx=0x7f8b2000bfd0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.495+0000 7f8b37e51740 1 -- 172.21.15.29:0/1698348756 shutdown_connections 2024-03-11T19:20:20.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.495+0000 7f8b37e51740 1 --2- 172.21.15.29:0/1698348756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf1ee30 0x55ba4bf13700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.495+0000 7f8b37e51740 1 --2- 172.21.15.29:0/1698348756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf2a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.495+0000 7f8b37e51740 1 --2- 172.21.15.29:0/1698348756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf192d0 0x55ba4bf131c0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.495+0000 7f8b37e51740 1 -- 172.21.15.29:0/1698348756 >> 172.21.15.29:0/1698348756 conn(0x55ba4bf147f0 msgr2=0x55ba4bf1afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:20.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.495+0000 7f8b37e51740 1 -- 172.21.15.29:0/1698348756 shutdown_connections 2024-03-11T19:20:20.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.495+0000 7f8b37e51740 1 -- 172.21.15.29:0/1698348756 wait complete. 2024-03-11T19:20:20.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.495+0000 7f8b37e51740 1 -- start start 2024-03-11T19:20:20.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.495+0000 7f8b37e51740 1 Processor -- start 2024-03-11T19:20:20.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b37e51740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf192d0 0x55ba4bf2a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b37e51740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf28b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b37e51740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba4bf1ee30 0x55ba4bf29090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b37e51740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba4bf29790 con 0x55ba4bf19cc0 2024-03-11T19:20:20.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b37e51740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba4bf29900 con 0x55ba4bf192d0 2024-03-11T19:20:20.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b37e51740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba4bf857d0 con 0x55ba4bf1ee30 2024-03-11T19:20:20.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b37e51740 10 client.0 _ll_register_callbacks cb 0x7f8b24004fb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:20.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b2e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf28b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b2e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf28b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33002/0 (socket says 172.21.15.29:33002) 2024-03-11T19:20:20.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b2e7fc640 1 -- 172.21.15.29:0/2506421786 learned_addr learned my addr 172.21.15.29:0/2506421786 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:20.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b2effd640 1 --2- 172.21.15.29:0/2506421786 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf192d0 0x55ba4bf2a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b2f7fe640 1 --2- 172.21.15.29:0/2506421786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba4bf1ee30 0x55ba4bf29090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b2e7fc640 1 -- 172.21.15.29:0/2506421786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba4bf1ee30 msgr2=0x55ba4bf29090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:20.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b2e7fc640 1 --2- 172.21.15.29:0/2506421786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba4bf1ee30 0x55ba4bf29090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b2e7fc640 1 -- 172.21.15.29:0/2506421786 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf192d0 msgr2=0x55ba4bf2a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:20.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b2e7fc640 1 --2- 172.21.15.29:0/2506421786 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf192d0 0x55ba4bf2a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b2e7fc640 1 -- 172.21.15.29:0/2506421786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ba4be80610 con 0x55ba4bf19cc0 2024-03-11T19:20:20.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b2e7fc640 1 --2- 172.21.15.29:0/2506421786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf28b50 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f8b240b9300 tx=0x7f8b2402c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74283c9baaa6d6d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:20.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b12ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:20:20.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b12ff5640 1 -- 172.21.15.29:0/2506421786 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b240056d0 con 0x55ba4bf19cc0 2024-03-11T19:20:20.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b12ff5640 1 -- 172.21.15.29:0/2506421786 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f8b24098070 con 0x55ba4bf19cc0 2024-03-11T19:20:20.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b37e51740 1 -- 172.21.15.29:0/2506421786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8b2409a870 con 0x55ba4bf19cc0 2024-03-11T19:20:20.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b12ff5640 1 -- 172.21.15.29:0/2506421786 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b24004dc0 con 0x55ba4bf19cc0 2024-03-11T19:20:20.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b37e51740 10 client.17931 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:20.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.496+0000 7f8b37e51740 1 -- 172.21.15.29:0/2506421786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ba4be426a0 con 0x55ba4bf19cc0 2024-03-11T19:20:20.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.497+0000 7f8b12ff5640 1 -- 172.21.15.29:0/2506421786 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8b2409a870 con 0x55ba4bf19cc0 2024-03-11T19:20:20.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.497+0000 7f8b12ff5640 1 -- 172.21.15.29:0/2506421786 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8b2403c070 con 0x55ba4bf19cc0 2024-03-11T19:20:20.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.497+0000 7f8b37e51740 10 client.17931 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:20.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.497+0000 7f8b37e51740 10 client.17931 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:20:20.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.497+0000 7f8b37e51740 20 client.17931 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:20.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.497+0000 7f8b37e51740 1 --2- 172.21.15.29:0/2506421786 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ba4bf86d80 0x55ba4bf891a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:20.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.497+0000 7f8b37e51740 4 client.17931 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:20:20.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.497+0000 7f8b37e51740 1 -- 172.21.15.29:0/2506421786 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ba4bc0d890 con 0x55ba4bf86d80 2024-03-11T19:20:20.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.497+0000 7f8b2effd640 1 --2- 172.21.15.29:0/2506421786 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ba4bf86d80 0x55ba4bf891a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:20.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.497+0000 7f8b2effd640 1 --2- 172.21.15.29:0/2506421786 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ba4bf86d80 0x55ba4bf891a0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b0eee3af63b78a6 server_cookie=bb18434fff78323a in_seq=0 out_seq=0 2024-03-11T19:20:20.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.497+0000 7f8b12ff5640 10 client.17931 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:20.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b12ff5640 1 -- 172.21.15.29:0/2506421786 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ba4bc0d890 con 0x55ba4bf86d80 2024-03-11T19:20:20.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b12ff5640 10 client.17931 handle_command_reply: tid=0 2024-03-11T19:20:20.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b37e51740 1 client.17931 shutdown 2024-03-11T19:20:20.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b37e51740 1 -- 172.21.15.29:0/2506421786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf19cc0 msgr2=0x55ba4bf28b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:20.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b37e51740 1 --2- 172.21.15.29:0/2506421786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf28b50 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f8b240b9300 tx=0x7f8b2402c450 comp rx=0 tx=0).stop 2024-03-11T19:20:20.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b37e51740 1 -- 172.21.15.29:0/2506421786 shutdown_connections 2024-03-11T19:20:20.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b37e51740 1 --2- 172.21.15.29:0/2506421786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ba4bf1ee30 0x55ba4bf29090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b37e51740 1 --2- 172.21.15.29:0/2506421786 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55ba4bf86d80 0x55ba4bf891a0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b37e51740 1 --2- 172.21.15.29:0/2506421786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ba4bf19cc0 0x55ba4bf28b50 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b12ff5640 0 client.17931 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:20.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b37e51740 1 --2- 172.21.15.29:0/2506421786 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ba4bf192d0 0x55ba4bf2a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b37e51740 1 -- 172.21.15.29:0/2506421786 >> 172.21.15.29:0/2506421786 conn(0x55ba4bf147f0 msgr2=0x55ba4bf17aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:20.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b37e51740 1 -- 172.21.15.29:0/2506421786 shutdown_connections 2024-03-11T19:20:20.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b37e51740 1 -- 172.21.15.29:0/2506421786 wait complete. 2024-03-11T19:20:20.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b37e51740 20 client.17931 trim_cache size 0 max 16384 2024-03-11T19:20:20.508 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:20:20.508 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:20:20.508 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:20:20.508 INFO:teuthology.orchestra.run.smithi029.stdout: "a4bf4efd-2380-4c24-a048-6aa71877e498": { 2024-03-11T19:20:20.508 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:20:20.508 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "a4bf4efd-2380-4c24-a048-6aa71877e498", 2024-03-11T19:20:20.509 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:20:20.509 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:20:20.509 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:20:20.509 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:20:20.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b35281640 1 -- 172.21.15.29:0/1329127494 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8b180a1a30 msgr2=0x7f8b180a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:20.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b35281640 1 --2- 172.21.15.29:0/1329127494 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8b180a1a30 0x7f8b180a3ef0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8b28005fd0 tx=0x7f8b28009f10 comp rx=0 tx=0).stop 2024-03-11T19:20:20.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b35281640 1 -- 172.21.15.29:0/1329127494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b30101270 msgr2=0x7f8b30197480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:20.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b35281640 1 --2- 172.21.15.29:0/1329127494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b30101270 0x7f8b30197480 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8b24036040 tx=0x7f8b240042f0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b35281640 1 -- 172.21.15.29:0/1329127494 shutdown_connections 2024-03-11T19:20:20.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b35281640 1 --2- 172.21.15.29:0/1329127494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b30101c30 0x7f8b3019e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b35281640 1 --2- 172.21.15.29:0/1329127494 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8b180a1a30 0x7f8b180a3ef0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b35281640 1 --2- 172.21.15.29:0/1329127494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b30101270 0x7f8b30197480 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b35281640 1 --2- 172.21.15.29:0/1329127494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b30100080 0x7f8b30196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:20.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b35281640 1 -- 172.21.15.29:0/1329127494 >> 172.21.15.29:0/1329127494 conn(0x7f8b300fb5f0 msgr2=0x7f8b301044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:20.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b35281640 1 -- 172.21.15.29:0/1329127494 shutdown_connections 2024-03-11T19:20:20.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:20.498+0000 7f8b35281640 1 -- 172.21.15.29:0/1329127494 wait complete. 2024-03-11T19:20:20.513 INFO:tasks.cephfs.filesystem:scrub status for tag:a4bf4efd-2380-4c24-a048-6aa71877e498 - {'path': '/', 'tag': 'a4bf4efd-2380-4c24-a048-6aa71877e498', 'options': 'recursive,force'} 2024-03-11T19:20:21.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:20 smithi190 ceph-mon[36292]: pgmap v667: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 56 MiB/s wr, 1.08k op/s 2024-03-11T19:20:21.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:20 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:20:21.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:20 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:20:21.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:20 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:20:21.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:20 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:20:21.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:20 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:20:21.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:20 smithi170 ceph-mon[35690]: pgmap v667: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 56 MiB/s wr, 1.08k op/s 2024-03-11T19:20:21.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:20 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:20:21.347 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:20 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:20:21.347 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:20 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:20:21.347 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:20 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:20:21.347 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:20 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:20:21.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:20 smithi029 ceph-mon[32439]: pgmap v667: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 56 MiB/s wr, 1.08k op/s 2024-03-11T19:20:21.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:20 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:20:21.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:20 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:20:21.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:20 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:20:21.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:20 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:20:21.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:20 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:20:23.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:22 smithi190 ceph-mon[36292]: pgmap v668: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 52 MiB/s wr, 1.01k op/s 2024-03-11T19:20:23.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:22 smithi170 ceph-mon[35690]: pgmap v668: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 52 MiB/s wr, 1.01k op/s 2024-03-11T19:20:23.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:22 smithi029 ceph-mon[32439]: pgmap v668: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 52 MiB/s wr, 1.01k op/s 2024-03-11T19:20:25.266 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:24 smithi190 ceph-mon[36292]: pgmap v669: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 52 MiB/s wr, 992 op/s 2024-03-11T19:20:25.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:24 smithi170 ceph-mon[35690]: pgmap v669: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 52 MiB/s wr, 992 op/s 2024-03-11T19:20:25.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:24 smithi029 ceph-mon[32439]: pgmap v669: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 52 MiB/s wr, 992 op/s 2024-03-11T19:20:27.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:26 smithi190 ceph-mon[36292]: pgmap v670: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 57 MiB/s wr, 1.08k op/s 2024-03-11T19:20:27.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:26 smithi170 ceph-mon[35690]: pgmap v670: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 57 MiB/s wr, 1.08k op/s 2024-03-11T19:20:27.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:26 smithi029 ceph-mon[32439]: pgmap v670: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 57 MiB/s wr, 1.08k op/s 2024-03-11T19:20:29.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:28 smithi190 ceph-mon[36292]: pgmap v671: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 51 MiB/s wr, 961 op/s 2024-03-11T19:20:29.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:28 smithi170 ceph-mon[35690]: pgmap v671: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 51 MiB/s wr, 961 op/s 2024-03-11T19:20:29.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:28 smithi029 ceph-mon[32439]: pgmap v671: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 51 MiB/s wr, 961 op/s 2024-03-11T19:20:31.227 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:30 smithi029 ceph-mon[32439]: pgmap v672: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 57 MiB/s wr, 1.08k op/s 2024-03-11T19:20:31.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:30 smithi190 ceph-mon[36292]: pgmap v672: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 57 MiB/s wr, 1.08k op/s 2024-03-11T19:20:31.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:30 smithi170 ceph-mon[35690]: pgmap v672: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 57 MiB/s wr, 1.08k op/s 2024-03-11T19:20:33.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:32 smithi190 ceph-mon[36292]: pgmap v673: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 56 MiB/s wr, 1.05k op/s 2024-03-11T19:20:33.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:32 smithi170 ceph-mon[35690]: pgmap v673: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 56 MiB/s wr, 1.05k op/s 2024-03-11T19:20:33.408 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:32 smithi029 ceph-mon[32439]: pgmap v673: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 56 MiB/s wr, 1.05k op/s 2024-03-11T19:20:35.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:34 smithi190 ceph-mon[36292]: pgmap v674: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 59 MiB/s wr, 1.10k op/s 2024-03-11T19:20:35.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:34 smithi170 ceph-mon[35690]: pgmap v674: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 59 MiB/s wr, 1.10k op/s 2024-03-11T19:20:35.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:34 smithi029 ceph-mon[32439]: pgmap v674: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 59 MiB/s wr, 1.10k op/s 2024-03-11T19:20:37.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:36 smithi190 ceph-mon[36292]: pgmap v675: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 61 MiB/s wr, 1.15k op/s 2024-03-11T19:20:37.267 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:36 smithi029 ceph-mon[32439]: pgmap v675: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 61 MiB/s wr, 1.15k op/s 2024-03-11T19:20:37.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:36 smithi170 ceph-mon[35690]: pgmap v675: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 61 MiB/s wr, 1.15k op/s 2024-03-11T19:20:39.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:38 smithi190 ceph-mon[36292]: pgmap v676: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 55 MiB/s wr, 1.02k op/s 2024-03-11T19:20:39.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:38 smithi170 ceph-mon[35690]: pgmap v676: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 55 MiB/s wr, 1.02k op/s 2024-03-11T19:20:39.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:38 smithi029 ceph-mon[32439]: pgmap v676: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 55 MiB/s wr, 1.02k op/s 2024-03-11T19:20:41.262 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:40 smithi190 ceph-mon[36292]: pgmap v677: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 60 MiB/s wr, 1.12k op/s 2024-03-11T19:20:41.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:40 smithi170 ceph-mon[35690]: pgmap v677: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 60 MiB/s wr, 1.12k op/s 2024-03-11T19:20:41.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:40 smithi029 ceph-mon[32439]: pgmap v677: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 60 MiB/s wr, 1.12k op/s 2024-03-11T19:20:43.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:42 smithi190 ceph-mon[36292]: pgmap v678: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 57 MiB/s wr, 1.05k op/s 2024-03-11T19:20:43.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:42 smithi170 ceph-mon[35690]: pgmap v678: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 57 MiB/s wr, 1.05k op/s 2024-03-11T19:20:43.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:42 smithi029 ceph-mon[32439]: pgmap v678: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 57 MiB/s wr, 1.05k op/s 2024-03-11T19:20:45.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:44 smithi190 ceph-mon[36292]: pgmap v679: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 59 MiB/s wr, 1.09k op/s 2024-03-11T19:20:45.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:44 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:20:45.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:44 smithi170 ceph-mon[35690]: pgmap v679: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 59 MiB/s wr, 1.09k op/s 2024-03-11T19:20:45.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:44 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:20:45.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:44 smithi029 ceph-mon[32439]: pgmap v679: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 59 MiB/s wr, 1.09k op/s 2024-03-11T19:20:45.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:44 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:20:47.817 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:47 smithi029 ceph-mon[32439]: pgmap v680: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 61 MiB/s wr, 1.12k op/s 2024-03-11T19:20:47.818 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:20:47.818 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:20:47.818 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:20:47.818 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:47 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:20:47.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:47 smithi170 ceph-mon[35690]: pgmap v680: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 61 MiB/s wr, 1.12k op/s 2024-03-11T19:20:47.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:20:47.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:20:47.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:20:47.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:47 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:20:48.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:47 smithi190 ceph-mon[36292]: pgmap v680: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 61 MiB/s wr, 1.12k op/s 2024-03-11T19:20:48.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:20:48.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:20:48.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:20:48.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:47 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:20:49.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:49 smithi170 ceph-mon[35690]: pgmap v681: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 54 MiB/s wr, 990 op/s 2024-03-11T19:20:49.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:49 smithi029 ceph-mon[32439]: pgmap v681: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 54 MiB/s wr, 990 op/s 2024-03-11T19:20:50.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:49 smithi190 ceph-mon[36292]: pgmap v681: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 54 MiB/s wr, 990 op/s 2024-03-11T19:20:50.514 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:20:50.514 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:20:50.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.706+0000 7fe317fff640 1 -- 172.21.15.29:0/1199728203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe310095e60 msgr2=0x7fe3100962e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:50.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.706+0000 7fe317fff640 1 --2- 172.21.15.29:0/1199728203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe310095e60 0x7fe3100962e0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009900 tx=0x7fe30c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:20:50.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.706+0000 7fe317fff640 1 -- 172.21.15.29:0/1199728203 shutdown_connections 2024-03-11T19:20:50.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.706+0000 7fe317fff640 1 --2- 172.21.15.29:0/1199728203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe310096820 0x7fe31009d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.707+0000 7fe317fff640 1 --2- 172.21.15.29:0/1199728203 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe310095e60 0x7fe3100962e0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.707+0000 7fe317fff640 1 --2- 172.21.15.29:0/1199728203 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe310094c70 0x7fe310095090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.707+0000 7fe317fff640 1 -- 172.21.15.29:0/1199728203 >> 172.21.15.29:0/1199728203 conn(0x7fe3100901c0 msgr2=0x7fe310092620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:50.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.707+0000 7fe317fff640 1 -- 172.21.15.29:0/1199728203 shutdown_connections 2024-03-11T19:20:50.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.707+0000 7fe317fff640 1 -- 172.21.15.29:0/1199728203 wait complete. 2024-03-11T19:20:50.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.707+0000 7fe317fff640 1 Processor -- start 2024-03-11T19:20:50.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.707+0000 7fe317fff640 1 -- start start 2024-03-11T19:20:50.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.707+0000 7fe317fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe310094c70 0x7fe3100052c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.707+0000 7fe317fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe310095e60 0x7fe310130a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.707+0000 7fe317fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe310096820 0x7fe3101330a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.707+0000 7fe317fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe310133830 con 0x7fe310094c70 2024-03-11T19:20:50.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.707+0000 7fe317fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3101339a0 con 0x7fe310095e60 2024-03-11T19:20:50.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.707+0000 7fe317fff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31012a8b0 con 0x7fe310096820 2024-03-11T19:20:50.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.708+0000 7fe316ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe310094c70 0x7fe3100052c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.708+0000 7fe316ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe310094c70 0x7fe3100052c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48426/0 (socket says 172.21.15.29:48426) 2024-03-11T19:20:50.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.708+0000 7fe316ffd640 1 -- 172.21.15.29:0/1232301993 learned_addr learned my addr 172.21.15.29:0/1232301993 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:50.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.708+0000 7fe3177fe640 1 --2- 172.21.15.29:0/1232301993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe310096820 0x7fe3101330a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.708+0000 7fe3167fc640 1 --2- 172.21.15.29:0/1232301993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe310095e60 0x7fe310130a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.708+0000 7fe316ffd640 1 -- 172.21.15.29:0/1232301993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe310096820 msgr2=0x7fe3101330a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:50.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.708+0000 7fe316ffd640 1 --2- 172.21.15.29:0/1232301993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe310096820 0x7fe3101330a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.708+0000 7fe316ffd640 1 -- 172.21.15.29:0/1232301993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe310095e60 msgr2=0x7fe310130a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:50.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.708+0000 7fe316ffd640 1 --2- 172.21.15.29:0/1232301993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe310095e60 0x7fe310130a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.708+0000 7fe316ffd640 1 -- 172.21.15.29:0/1232301993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30c009560 con 0x7fe310094c70 2024-03-11T19:20:50.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.708+0000 7fe316ffd640 1 --2- 172.21.15.29:0/1232301993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe310094c70 0x7fe3100052c0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fe30800e9c0 tx=0x7fe30800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:50.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.708+0000 7fe2fffff640 1 -- 172.21.15.29:0/1232301993 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe308009700 con 0x7fe310094c70 2024-03-11T19:20:50.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.708+0000 7fe2fffff640 1 -- 172.21.15.29:0/1232301993 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe3080044d0 con 0x7fe310094c70 2024-03-11T19:20:50.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.708+0000 7fe317fff640 1 -- 172.21.15.29:0/1232301993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe31012ab90 con 0x7fe310094c70 2024-03-11T19:20:50.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.708+0000 7fe2fffff640 1 -- 172.21.15.29:0/1232301993 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe308003e80 con 0x7fe310094c70 2024-03-11T19:20:50.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.708+0000 7fe317fff640 1 -- 172.21.15.29:0/1232301993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31012b070 con 0x7fe310094c70 2024-03-11T19:20:50.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.710+0000 7fe2fffff640 1 -- 172.21.15.29:0/1232301993 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe308003fe0 con 0x7fe310094c70 2024-03-11T19:20:50.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe32003a740 1 Processor -- start 2024-03-11T19:20:50.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe32003a740 1 -- start start 2024-03-11T19:20:50.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe32003a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a36e4f0 0x56226a36e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe32003a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 0x56226a3739e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe32003a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a374050 0x56226a3764e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe32003a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56226a2de280 con 0x56226a36eee0 2024-03-11T19:20:50.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe32003a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56226a2db7b0 con 0x56226a374050 2024-03-11T19:20:50.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe32003a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56226a2d6f90 con 0x56226a36e4f0 2024-03-11T19:20:50.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe2fffff640 1 --2- 172.21.15.29:0/1232301993 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3000a1a30 0x7fe3000a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe316ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 0x56226a3739e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe3177fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a36e4f0 0x56226a36e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe316ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 0x56226a3739e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48428/0 (socket says 172.21.15.29:48428) 2024-03-11T19:20:50.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe3177fe640 1 --2- 172.21.15.29:0/1232301993 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3000a1a30 0x7fe3000a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe316ffd640 1 -- 172.21.15.29:0/2469702516 learned_addr learned my addr 172.21.15.29:0/2469702516 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:50.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe3167fc640 1 --2- 172.21.15.29:0/2469702516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a374050 0x56226a3764e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe2fffff640 1 -- 172.21.15.29:0/1232301993 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe308014070 con 0x7fe310094c70 2024-03-11T19:20:50.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe316ffd640 1 -- 172.21.15.29:0/2469702516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a36e4f0 msgr2=0x56226a36e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:50.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe316ffd640 1 --2- 172.21.15.29:0/2469702516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a36e4f0 0x56226a36e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe316ffd640 1 -- 172.21.15.29:0/2469702516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a374050 msgr2=0x56226a3764e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:50.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe316ffd640 1 --2- 172.21.15.29:0/2469702516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a374050 0x56226a3764e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe316ffd640 1 -- 172.21.15.29:0/2469702516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56226a295570 con 0x56226a36eee0 2024-03-11T19:20:50.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe3167fc640 1 --2- 172.21.15.29:0/2469702516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a374050 0x56226a3764e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:20:50.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.711+0000 7fe316ffd640 1 --2- 172.21.15.29:0/2469702516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 0x56226a3739e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fe30800c6a0 tx=0x7fe30806da10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c27d859673a7b758 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:50.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.712+0000 7fe3177fe640 1 --2- 172.21.15.29:0/1232301993 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3000a1a30 0x7fe3000a3ef0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe3180668b0 tx=0x7fe31804ff00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:50.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.712+0000 7fe2fd7fa640 1 -- 172.21.15.29:0/2469702516 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe308065f00 con 0x56226a36eee0 2024-03-11T19:20:50.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.712+0000 7fe2fd7fa640 1 -- 172.21.15.29:0/2469702516 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe30809a710 con 0x56226a36eee0 2024-03-11T19:20:50.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.712+0000 7fe2fd7fa640 1 -- 172.21.15.29:0/2469702516 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe308096d80 con 0x56226a36eee0 2024-03-11T19:20:50.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.712+0000 7fe32003a740 1 -- 172.21.15.29:0/2469702516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 msgr2=0x56226a3739e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:50.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.712+0000 7fe32003a740 1 --2- 172.21.15.29:0/2469702516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 0x56226a3739e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fe30800c6a0 tx=0x7fe30806da10 comp rx=0 tx=0).stop 2024-03-11T19:20:50.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.712+0000 7fe32003a740 1 -- 172.21.15.29:0/2469702516 shutdown_connections 2024-03-11T19:20:50.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.712+0000 7fe32003a740 1 --2- 172.21.15.29:0/2469702516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a374050 0x56226a3764e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.712+0000 7fe32003a740 1 --2- 172.21.15.29:0/2469702516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 0x56226a3739e0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.712+0000 7fe32003a740 1 --2- 172.21.15.29:0/2469702516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a36e4f0 0x56226a36e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.712+0000 7fe32003a740 1 -- 172.21.15.29:0/2469702516 >> 172.21.15.29:0/2469702516 conn(0x56226a369a60 msgr2=0x56226a369e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:50.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.712+0000 7fe32003a740 1 -- 172.21.15.29:0/2469702516 shutdown_connections 2024-03-11T19:20:50.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.712+0000 7fe32003a740 1 -- 172.21.15.29:0/2469702516 wait complete. 2024-03-11T19:20:50.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.713+0000 7fe32003a740 1 -- start start 2024-03-11T19:20:50.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.713+0000 7fe32003a740 1 Processor -- start 2024-03-11T19:20:50.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.713+0000 7fe32003a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a36e4f0 0x56226a378400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.713+0000 7fe32003a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 0x56226a378940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.713+0000 7fe32003a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a374050 0x56226a37f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.713+0000 7fe32003a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562269faa430 con 0x56226a36eee0 2024-03-11T19:20:50.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.713+0000 7fe32003a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562269f66e00 con 0x56226a36e4f0 2024-03-11T19:20:50.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.713+0000 7fe32003a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56226a25bfd0 con 0x56226a374050 2024-03-11T19:20:50.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.713+0000 7fe32003a740 10 client.0 _ll_register_callbacks cb 0x56226a2b1fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:50.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.713+0000 7fe3167fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 0x56226a378940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.713+0000 7fe3167fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 0x56226a378940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48440/0 (socket says 172.21.15.29:48440) 2024-03-11T19:20:50.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.713+0000 7fe3167fc640 1 -- 172.21.15.29:0/3901663323 learned_addr learned my addr 172.21.15.29:0/3901663323 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:50.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.713+0000 7fe316ffd640 1 --2- 172.21.15.29:0/3901663323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a36e4f0 0x56226a378400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.713+0000 7fe3177fe640 1 --2- 172.21.15.29:0/3901663323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a374050 0x56226a37f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.713+0000 7fe3167fc640 1 -- 172.21.15.29:0/3901663323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a374050 msgr2=0x56226a37f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:50.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.713+0000 7fe3167fc640 1 --2- 172.21.15.29:0/3901663323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a374050 0x56226a37f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.713+0000 7fe3167fc640 1 -- 172.21.15.29:0/3901663323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a36e4f0 msgr2=0x56226a378400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:50.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.713+0000 7fe3167fc640 1 --2- 172.21.15.29:0/3901663323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a36e4f0 0x56226a378400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.713+0000 7fe3167fc640 1 -- 172.21.15.29:0/3901663323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31012b070 con 0x56226a36eee0 2024-03-11T19:20:50.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.714+0000 7fe3167fc640 1 --2- 172.21.15.29:0/3901663323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 0x56226a378940 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009900 tx=0x7fe30c0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=311457cb98d1049 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:50.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.714+0000 7fe2faff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:20:50.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.714+0000 7fe2faff5640 1 -- 172.21.15.29:0/3901663323 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c004ae0 con 0x56226a36eee0 2024-03-11T19:20:50.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.714+0000 7fe2faff5640 1 -- 172.21.15.29:0/3901663323 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe30c004250 con 0x56226a36eee0 2024-03-11T19:20:50.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.714+0000 7fe32003a740 1 -- 172.21.15.29:0/3901663323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562269fd67e0 con 0x56226a36eee0 2024-03-11T19:20:50.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.714+0000 7fe2faff5640 1 -- 172.21.15.29:0/3901663323 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe30c004410 con 0x56226a36eee0 2024-03-11T19:20:50.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.714+0000 7fe32003a740 10 client.17943 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:50.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.714+0000 7fe32003a740 1 -- 172.21.15.29:0/3901663323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56226a2d5610 con 0x56226a36eee0 2024-03-11T19:20:50.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.714+0000 7fe2faff5640 1 -- 172.21.15.29:0/3901663323 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x562269fd67e0 con 0x56226a36eee0 2024-03-11T19:20:50.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.714+0000 7fe2faff5640 1 -- 172.21.15.29:0/3901663323 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe30c03c040 con 0x56226a36eee0 2024-03-11T19:20:50.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.714+0000 7fe32003a740 10 client.17943 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:50.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.714+0000 7fe32003a740 10 client.17943 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:20:50.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.714+0000 7fe32003a740 20 client.17943 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:50.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.714+0000 7fe32003a740 1 --2- 172.21.15.29:0/3901663323 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56226a3846d0 0x56226a386af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.714+0000 7fe32003a740 4 client.17943 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:20:50.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.714+0000 7fe32003a740 1 -- 172.21.15.29:0/3901663323 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56226a013720 con 0x56226a3846d0 2024-03-11T19:20:50.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.715+0000 7fe316ffd640 1 --2- 172.21.15.29:0/3901663323 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56226a3846d0 0x56226a386af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.715+0000 7fe316ffd640 1 --2- 172.21.15.29:0/3901663323 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56226a3846d0 0x56226a386af0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=960e3e9d2a2b177e server_cookie=13382e094dcc809d in_seq=0 out_seq=0 2024-03-11T19:20:50.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.715+0000 7fe2faff5640 10 client.17943 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:50.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.715+0000 7fe2faff5640 1 -- 172.21.15.29:0/3901663323 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56226a013720 con 0x56226a3846d0 2024-03-11T19:20:50.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.715+0000 7fe2faff5640 10 client.17943 handle_command_reply: tid=0 2024-03-11T19:20:50.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.715+0000 7fe32003a740 1 client.17943 shutdown 2024-03-11T19:20:50.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.716+0000 7fe32003a740 1 -- 172.21.15.29:0/3901663323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 msgr2=0x56226a378940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:50.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.716+0000 7fe32003a740 1 --2- 172.21.15.29:0/3901663323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 0x56226a378940 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009900 tx=0x7fe30c0048f0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.716+0000 7fe32003a740 1 -- 172.21.15.29:0/3901663323 shutdown_connections 2024-03-11T19:20:50.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.716+0000 7fe32003a740 1 --2- 172.21.15.29:0/3901663323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a374050 0x56226a37f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.716+0000 7fe32003a740 1 --2- 172.21.15.29:0/3901663323 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56226a3846d0 0x56226a386af0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.716+0000 7fe32003a740 1 --2- 172.21.15.29:0/3901663323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 0x56226a378940 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.716+0000 7fe32003a740 1 --2- 172.21.15.29:0/3901663323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a36e4f0 0x56226a378400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.716+0000 7fe2faff5640 0 client.17943 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:50.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.716+0000 7fe32003a740 1 -- 172.21.15.29:0/3901663323 >> 172.21.15.29:0/3901663323 conn(0x56226a369a60 msgr2=0x56226a368d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:50.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.716+0000 7fe32003a740 1 -- 172.21.15.29:0/3901663323 shutdown_connections 2024-03-11T19:20:50.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.716+0000 7fe32003a740 1 -- 172.21.15.29:0/3901663323 wait complete. 2024-03-11T19:20:50.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.716+0000 7fe32003a740 20 client.17943 trim_cache size 0 max 16384 2024-03-11T19:20:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.730+0000 7fe32003a740 1 Processor -- start 2024-03-11T19:20:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe32003a740 1 -- start start 2024-03-11T19:20:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe32003a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a36e4f0 0x56226a378400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe32003a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 0x56226a378940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe32003a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a374050 0x56226a37f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe32003a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562269fd67e0 con 0x56226a36eee0 2024-03-11T19:20:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe32003a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56226a364170 con 0x56226a36e4f0 2024-03-11T19:20:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe32003a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56226a380e70 con 0x56226a374050 2024-03-11T19:20:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe3167fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 0x56226a378940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe3167fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 0x56226a378940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48450/0 (socket says 172.21.15.29:48450) 2024-03-11T19:20:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe3167fc640 1 -- 172.21.15.29:0/4161563823 learned_addr learned my addr 172.21.15.29:0/4161563823 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe3177fe640 1 --2- 172.21.15.29:0/4161563823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a374050 0x56226a37f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe316ffd640 1 --2- 172.21.15.29:0/4161563823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a36e4f0 0x56226a378400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe3167fc640 1 -- 172.21.15.29:0/4161563823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a374050 msgr2=0x56226a37f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe3167fc640 1 --2- 172.21.15.29:0/4161563823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a374050 0x56226a37f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe3167fc640 1 -- 172.21.15.29:0/4161563823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a36e4f0 msgr2=0x56226a378400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe3167fc640 1 --2- 172.21.15.29:0/4161563823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a36e4f0 0x56226a378400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe3167fc640 1 -- 172.21.15.29:0/4161563823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56226a271ac0 con 0x56226a36eee0 2024-03-11T19:20:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe3177fe640 1 --2- 172.21.15.29:0/4161563823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a374050 0x56226a37f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:20:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe3167fc640 1 --2- 172.21.15.29:0/4161563823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 0x56226a378940 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0050d0 tx=0x7fe30c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d339d76827f5e537 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe2fcff9640 1 -- 172.21.15.29:0/4161563823 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c004dc0 con 0x56226a36eee0 2024-03-11T19:20:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe2fcff9640 1 -- 172.21.15.29:0/4161563823 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe30c050030 con 0x56226a36eee0 2024-03-11T19:20:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.731+0000 7fe2fcff9640 1 -- 172.21.15.29:0/4161563823 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c003aa0 con 0x56226a36eee0 2024-03-11T19:20:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.732+0000 7fe32003a740 1 -- 172.21.15.29:0/4161563823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 msgr2=0x56226a378940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.732+0000 7fe32003a740 1 --2- 172.21.15.29:0/4161563823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 0x56226a378940 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0050d0 tx=0x7fe30c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:20:50.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.732+0000 7fe32003a740 1 -- 172.21.15.29:0/4161563823 shutdown_connections 2024-03-11T19:20:50.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.732+0000 7fe32003a740 1 --2- 172.21.15.29:0/4161563823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a374050 0x56226a37f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.732+0000 7fe32003a740 1 --2- 172.21.15.29:0/4161563823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36eee0 0x56226a378940 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.732+0000 7fe32003a740 1 --2- 172.21.15.29:0/4161563823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a36e4f0 0x56226a378400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.732+0000 7fe32003a740 1 -- 172.21.15.29:0/4161563823 >> 172.21.15.29:0/4161563823 conn(0x56226a369a60 msgr2=0x56226a369e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:50.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.732+0000 7fe32003a740 1 -- 172.21.15.29:0/4161563823 shutdown_connections 2024-03-11T19:20:50.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.732+0000 7fe32003a740 1 -- 172.21.15.29:0/4161563823 wait complete. 2024-03-11T19:20:50.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.732+0000 7fe32003a740 1 -- start start 2024-03-11T19:20:50.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.732+0000 7fe32003a740 1 Processor -- start 2024-03-11T19:20:50.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.732+0000 7fe32003a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36e4f0 0x56226a37f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.732+0000 7fe32003a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a36eee0 0x56226a37dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.732+0000 7fe32003a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a374050 0x56226a37e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.732+0000 7fe32003a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56226a37e710 con 0x56226a36e4f0 2024-03-11T19:20:50.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.732+0000 7fe32003a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56226a37e880 con 0x56226a36eee0 2024-03-11T19:20:50.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe32003a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56226a37ea40 con 0x56226a374050 2024-03-11T19:20:50.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe32003a740 10 client.0 _ll_register_callbacks cb 0x7fe3080046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:50.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe316ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36e4f0 0x56226a37f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe316ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36e4f0 0x56226a37f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48454/0 (socket says 172.21.15.29:48454) 2024-03-11T19:20:50.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe316ffd640 1 -- 172.21.15.29:0/527948908 learned_addr learned my addr 172.21.15.29:0/527948908 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:50.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe3177fe640 1 --2- 172.21.15.29:0/527948908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a374050 0x56226a37e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe3167fc640 1 --2- 172.21.15.29:0/527948908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a36eee0 0x56226a37dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe316ffd640 1 -- 172.21.15.29:0/527948908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a374050 msgr2=0x56226a37e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:50.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe316ffd640 1 --2- 172.21.15.29:0/527948908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a374050 0x56226a37e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe316ffd640 1 -- 172.21.15.29:0/527948908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a36eee0 msgr2=0x56226a37dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:50.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe316ffd640 1 --2- 172.21.15.29:0/527948908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a36eee0 0x56226a37dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe316ffd640 1 -- 172.21.15.29:0/527948908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56226a2d5610 con 0x56226a36e4f0 2024-03-11T19:20:50.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe2faff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:20:50.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe316ffd640 1 --2- 172.21.15.29:0/527948908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36e4f0 0x56226a37f680 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fe3080058c0 tx=0x7fe30809a170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2438e0049313b741 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:50.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe2faff5640 1 -- 172.21.15.29:0/527948908 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30809e860 con 0x56226a36e4f0 2024-03-11T19:20:50.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe2faff5640 1 -- 172.21.15.29:0/527948908 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fe3080a2eb0 con 0x56226a36e4f0 2024-03-11T19:20:50.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe32003a740 1 -- 172.21.15.29:0/527948908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe30809a710 con 0x56226a36e4f0 2024-03-11T19:20:50.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe2faff5640 1 -- 172.21.15.29:0/527948908 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3080b31f0 con 0x56226a36e4f0 2024-03-11T19:20:50.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe32003a740 10 client.17949 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:50.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe32003a740 1 -- 172.21.15.29:0/527948908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56226a2976a0 con 0x56226a36e4f0 2024-03-11T19:20:50.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.733+0000 7fe2faff5640 1 -- 172.21.15.29:0/527948908 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe30809a710 con 0x56226a36e4f0 2024-03-11T19:20:50.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.734+0000 7fe2faff5640 1 -- 172.21.15.29:0/527948908 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe30801f060 con 0x56226a36e4f0 2024-03-11T19:20:50.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.734+0000 7fe32003a740 10 client.17949 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:50.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.734+0000 7fe32003a740 10 client.17949 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:20:50.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.734+0000 7fe32003a740 20 client.17949 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:50.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.734+0000 7fe32003a740 1 --2- 172.21.15.29:0/527948908 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56226a3dc180 0x56226a3de5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.734+0000 7fe32003a740 4 client.17949 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:20:50.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.734+0000 7fe32003a740 1 -- 172.21.15.29:0/527948908 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56226a062890 con 0x56226a3dc180 2024-03-11T19:20:50.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.734+0000 7fe3167fc640 1 --2- 172.21.15.29:0/527948908 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56226a3dc180 0x56226a3de5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.734+0000 7fe2faff5640 10 client.17949 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:50.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.734+0000 7fe3167fc640 1 --2- 172.21.15.29:0/527948908 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56226a3dc180 0x56226a3de5a0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bddc65d729d586bc server_cookie=c1397541419f7e7a in_seq=0 out_seq=0 2024-03-11T19:20:50.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe2faff5640 1 -- 172.21.15.29:0/527948908 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56226a062890 con 0x56226a3dc180 2024-03-11T19:20:50.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe2faff5640 10 client.17949 handle_command_reply: tid=0 2024-03-11T19:20:50.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe32003a740 1 client.17949 shutdown 2024-03-11T19:20:50.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe32003a740 1 -- 172.21.15.29:0/527948908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36e4f0 msgr2=0x56226a37f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:50.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe32003a740 1 --2- 172.21.15.29:0/527948908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36e4f0 0x56226a37f680 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fe3080058c0 tx=0x7fe30809a170 comp rx=0 tx=0).stop 2024-03-11T19:20:50.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe32003a740 1 -- 172.21.15.29:0/527948908 shutdown_connections 2024-03-11T19:20:50.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe32003a740 1 --2- 172.21.15.29:0/527948908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56226a374050 0x56226a37e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe32003a740 1 --2- 172.21.15.29:0/527948908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56226a36eee0 0x56226a37dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe32003a740 1 --2- 172.21.15.29:0/527948908 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x56226a3dc180 0x56226a3de5a0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe32003a740 1 --2- 172.21.15.29:0/527948908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56226a36e4f0 0x56226a37f680 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe32003a740 1 -- 172.21.15.29:0/527948908 >> 172.21.15.29:0/527948908 conn(0x56226a369a60 msgr2=0x56226a3700c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:50.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe2faff5640 0 client.17949 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:50.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe32003a740 1 -- 172.21.15.29:0/527948908 shutdown_connections 2024-03-11T19:20:50.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe32003a740 1 -- 172.21.15.29:0/527948908 wait complete. 2024-03-11T19:20:50.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe32003a740 20 client.17949 trim_cache size 0 max 16384 2024-03-11T19:20:50.746 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:20:50.746 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:20:50.746 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:20:50.746 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:20:50.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe317fff640 1 -- 172.21.15.29:0/1232301993 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3000a1a30 msgr2=0x7fe3000a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:50.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe317fff640 1 --2- 172.21.15.29:0/1232301993 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3000a1a30 0x7fe3000a3ef0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe3180668b0 tx=0x7fe31804ff00 comp rx=0 tx=0).stop 2024-03-11T19:20:50.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe317fff640 1 -- 172.21.15.29:0/1232301993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe310094c70 msgr2=0x7fe3100052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:50.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe317fff640 1 --2- 172.21.15.29:0/1232301993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe310094c70 0x7fe3100052c0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fe30800e9c0 tx=0x7fe30800edc0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe317fff640 1 -- 172.21.15.29:0/1232301993 shutdown_connections 2024-03-11T19:20:50.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe317fff640 1 --2- 172.21.15.29:0/1232301993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe310096820 0x7fe3101330a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe317fff640 1 --2- 172.21.15.29:0/1232301993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe310095e60 0x7fe310130a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe317fff640 1 --2- 172.21.15.29:0/1232301993 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fe3000a1a30 0x7fe3000a3ef0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe317fff640 1 --2- 172.21.15.29:0/1232301993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe310094c70 0x7fe3100052c0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe317fff640 1 -- 172.21.15.29:0/1232301993 >> 172.21.15.29:0/1232301993 conn(0x7fe3100901c0 msgr2=0x7fe310099180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:50.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe317fff640 1 -- 172.21.15.29:0/1232301993 shutdown_connections 2024-03-11T19:20:50.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.735+0000 7fe317fff640 1 -- 172.21.15.29:0/1232301993 wait complete. 2024-03-11T19:20:50.749 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:20:50.750 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:20:50.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.848+0000 7fddcb89d640 1 -- 172.21.15.29:0/2291056654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc41011b0 msgr2=0x7fddc41015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:50.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.848+0000 7fddcb89d640 1 --2- 172.21.15.29:0/2291056654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc41011b0 0x7fddc41015d0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fddbc009900 tx=0x7fddbc02bc30 comp rx=0 tx=0).stop 2024-03-11T19:20:50.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.848+0000 7fddcb89d640 1 -- 172.21.15.29:0/2291056654 shutdown_connections 2024-03-11T19:20:50.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.848+0000 7fddcb89d640 1 --2- 172.21.15.29:0/2291056654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fddc4102d60 0x7fddc41095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.848+0000 7fddcb89d640 1 --2- 172.21.15.29:0/2291056654 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddc41023a0 0x7fddc4102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.848+0000 7fddcb89d640 1 --2- 172.21.15.29:0/2291056654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc41011b0 0x7fddc41015d0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.848+0000 7fddcb89d640 1 -- 172.21.15.29:0/2291056654 >> 172.21.15.29:0/2291056654 conn(0x7fddc40fc760 msgr2=0x7fddc40feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:50.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.848+0000 7fddcb89d640 1 -- 172.21.15.29:0/2291056654 shutdown_connections 2024-03-11T19:20:50.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.848+0000 7fddcb89d640 1 -- 172.21.15.29:0/2291056654 wait complete. 2024-03-11T19:20:50.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.849+0000 7fddcb89d640 1 Processor -- start 2024-03-11T19:20:50.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.849+0000 7fddcb89d640 1 -- start start 2024-03-11T19:20:50.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.849+0000 7fddcb89d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fddc41011b0 0x7fddc4074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.849+0000 7fddcb89d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc41023a0 0x7fddc4073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.849+0000 7fddcb89d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddc4102d60 0x7fddc4073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.849+0000 7fddcb89d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc4074210 con 0x7fddc41023a0 2024-03-11T19:20:50.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.849+0000 7fddcb89d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc4074380 con 0x7fddc4102d60 2024-03-11T19:20:50.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.849+0000 7fddcb89d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc41a11f0 con 0x7fddc41011b0 2024-03-11T19:20:50.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.849+0000 7fddc8e38640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc41023a0 0x7fddc4073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.849+0000 7fddc8e38640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc41023a0 0x7fddc4073570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48482/0 (socket says 172.21.15.29:48482) 2024-03-11T19:20:50.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.849+0000 7fddc8e38640 1 -- 172.21.15.29:0/4034402226 learned_addr learned my addr 172.21.15.29:0/4034402226 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:50.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.849+0000 7fddc9639640 1 --2- 172.21.15.29:0/4034402226 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fddc41011b0 0x7fddc4074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.849+0000 7fddc9e3a640 1 --2- 172.21.15.29:0/4034402226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddc4102d60 0x7fddc4073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.849+0000 7fddc8e38640 1 -- 172.21.15.29:0/4034402226 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fddc41011b0 msgr2=0x7fddc4074f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:50.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.849+0000 7fddc8e38640 1 --2- 172.21.15.29:0/4034402226 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fddc41011b0 0x7fddc4074f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.849+0000 7fddc8e38640 1 -- 172.21.15.29:0/4034402226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddc4102d60 msgr2=0x7fddc4073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:50.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.849+0000 7fddc8e38640 1 --2- 172.21.15.29:0/4034402226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddc4102d60 0x7fddc4073ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:50.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.850+0000 7fddc8e38640 1 -- 172.21.15.29:0/4034402226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddbc009560 con 0x7fddc41023a0 2024-03-11T19:20:50.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.850+0000 7fddc8e38640 1 --2- 172.21.15.29:0/4034402226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc41023a0 0x7fddc4073570 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fddb800ea20 tx=0x7fddb800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:50.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.850+0000 7fddc27fc640 1 -- 172.21.15.29:0/4034402226 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb8009700 con 0x7fddc41023a0 2024-03-11T19:20:50.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.850+0000 7fddc27fc640 1 -- 172.21.15.29:0/4034402226 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fddb800cd80 con 0x7fddc41023a0 2024-03-11T19:20:50.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.850+0000 7fddcb89d640 1 -- 172.21.15.29:0/4034402226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc41a14a0 con 0x7fddc41023a0 2024-03-11T19:20:50.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.850+0000 7fddc27fc640 1 -- 172.21.15.29:0/4034402226 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb8018710 con 0x7fddc41023a0 2024-03-11T19:20:50.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.850+0000 7fddcb89d640 1 -- 172.21.15.29:0/4034402226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc41a1a00 con 0x7fddc41023a0 2024-03-11T19:20:50.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.851+0000 7fddcb89d640 1 -- 172.21.15.29:0/4034402226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc41085a0 con 0x7fddc41023a0 2024-03-11T19:20:50.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.852+0000 7fddc27fc640 1 -- 172.21.15.29:0/4034402226 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fddb8010430 con 0x7fddc41023a0 2024-03-11T19:20:50.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.853+0000 7fddc27fc640 1 --2- 172.21.15.29:0/4034402226 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fddac0a1ab0 0x7fddac0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:50.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.853+0000 7fddc9639640 1 --2- 172.21.15.29:0/4034402226 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fddac0a1ab0 0x7fddac0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:50.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.853+0000 7fddc27fc640 1 -- 172.21.15.29:0/4034402226 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fddb8014070 con 0x7fddc41023a0 2024-03-11T19:20:50.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.853+0000 7fddc9639640 1 --2- 172.21.15.29:0/4034402226 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fddac0a1ab0 0x7fddac0a3f70 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fddbc02c7d0 tx=0x7fddbc005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:50.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:50.857+0000 7fddc27fc640 1 -- 172.21.15.29:0/4034402226 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fddb8096e20 con 0x7fddc41023a0 2024-03-11T19:20:51.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.099+0000 7fddcb89d640 1 -- 172.21.15.29:0/4034402226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fddc410ccc0 con 0x7fddc41023a0 2024-03-11T19:20:51.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.101+0000 7fddc27fc640 1 -- 172.21.15.29:0/4034402226 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7fddb8026090 con 0x7fddc41023a0 2024-03-11T19:20:51.105 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:20:51.105 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:20:51.105 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:20:51.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.107+0000 7fddcb89d640 1 -- 172.21.15.29:0/4034402226 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fddac0a1ab0 msgr2=0x7fddac0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.107+0000 7fddcb89d640 1 --2- 172.21.15.29:0/4034402226 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fddac0a1ab0 0x7fddac0a3f70 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fddbc02c7d0 tx=0x7fddbc005fd0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.107+0000 7fddcb89d640 1 -- 172.21.15.29:0/4034402226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc41023a0 msgr2=0x7fddc4073570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.107+0000 7fddcb89d640 1 --2- 172.21.15.29:0/4034402226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc41023a0 0x7fddc4073570 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fddb800ea20 tx=0x7fddb800eee0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.110+0000 7fddcb89d640 1 -- 172.21.15.29:0/4034402226 shutdown_connections 2024-03-11T19:20:51.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.110+0000 7fddcb89d640 1 --2- 172.21.15.29:0/4034402226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddc4102d60 0x7fddc4073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.110+0000 7fddcb89d640 1 --2- 172.21.15.29:0/4034402226 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fddac0a1ab0 0x7fddac0a3f70 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.110+0000 7fddcb89d640 1 --2- 172.21.15.29:0/4034402226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddc41023a0 0x7fddc4073570 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.110+0000 7fddcb89d640 1 --2- 172.21.15.29:0/4034402226 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fddc41011b0 0x7fddc4074f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.110+0000 7fddcb89d640 1 -- 172.21.15.29:0/4034402226 >> 172.21.15.29:0/4034402226 conn(0x7fddc40fc760 msgr2=0x7fddc4105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:51.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.110+0000 7fddcb89d640 1 -- 172.21.15.29:0/4034402226 shutdown_connections 2024-03-11T19:20:51.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.110+0000 7fddcb89d640 1 -- 172.21.15.29:0/4034402226 wait complete. 2024-03-11T19:20:51.133 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:20:51.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:20:51.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.339+0000 7fcfce7fc640 1 -- 172.21.15.29:0/2987779853 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc802cb70 con 0x7fcfd01087d0 2024-03-11T19:20:51.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.340+0000 7fcfd618a640 1 -- 172.21.15.29:0/2987779853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfd01087d0 msgr2=0x7fcfd0108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.340+0000 7fcfd618a640 1 --2- 172.21.15.29:0/2987779853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfd01087d0 0x7fcfd0108c50 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009900 tx=0x7fcfc802bc30 comp rx=0 tx=0).stop 2024-03-11T19:20:51.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.340+0000 7fcfd618a640 1 -- 172.21.15.29:0/2987779853 shutdown_connections 2024-03-11T19:20:51.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.340+0000 7fcfd618a640 1 --2- 172.21.15.29:0/2987779853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfd0109190 0x7fcfd010fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.340+0000 7fcfd618a640 1 --2- 172.21.15.29:0/2987779853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfd01087d0 0x7fcfd0108c50 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.340+0000 7fcfd618a640 1 --2- 172.21.15.29:0/2987779853 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfd01075e0 0x7fcfd0107a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.340+0000 7fcfd618a640 1 -- 172.21.15.29:0/2987779853 >> 172.21.15.29:0/2987779853 conn(0x7fcfd0076580 msgr2=0x7fcfd00789c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:51.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.341+0000 7fcfd618a640 1 -- 172.21.15.29:0/2987779853 shutdown_connections 2024-03-11T19:20:51.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.341+0000 7fcfd618a640 1 -- 172.21.15.29:0/2987779853 wait complete. 2024-03-11T19:20:51.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.341+0000 7fcfd618a640 1 Processor -- start 2024-03-11T19:20:51.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.342+0000 7fcfd618a640 1 -- start start 2024-03-11T19:20:51.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.342+0000 7fcfd618a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfd01075e0 0x7fcfd019d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.342+0000 7fcfd618a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfd01087d0 0x7fcfd019daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.342+0000 7fcfd618a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfd0109190 0x7fcfd01a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.342+0000 7fcfd618a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd01a50a0 con 0x7fcfd0109190 2024-03-11T19:20:51.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.342+0000 7fcfd618a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd01a5210 con 0x7fcfd01087d0 2024-03-11T19:20:51.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.342+0000 7fcfd618a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd01a53d0 con 0x7fcfd01075e0 2024-03-11T19:20:51.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.342+0000 7fcfcf7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfd01075e0 0x7fcfd019d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.342+0000 7fcfcf7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfd01075e0 0x7fcfd019d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:50620/0 (socket says 172.21.15.29:50620) 2024-03-11T19:20:51.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.342+0000 7fcfcf7fe640 1 -- 172.21.15.29:0/4070382396 learned_addr learned my addr 172.21.15.29:0/4070382396 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:51.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.342+0000 7fcfceffd640 1 --2- 172.21.15.29:0/4070382396 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfd01087d0 0x7fcfd019daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.342+0000 7fcfcf7fe640 1 -- 172.21.15.29:0/4070382396 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfd01087d0 msgr2=0x7fcfd019daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.342+0000 7fcfcf7fe640 1 --2- 172.21.15.29:0/4070382396 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfd01087d0 0x7fcfd019daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.342+0000 7fcfcf7fe640 1 -- 172.21.15.29:0/4070382396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfd0109190 msgr2=0x7fcfd01a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.342+0000 7fcfcf7fe640 1 --2- 172.21.15.29:0/4070382396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfd0109190 0x7fcfd01a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.342+0000 7fcfcf7fe640 1 -- 172.21.15.29:0/4070382396 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc8009560 con 0x7fcfd01075e0 2024-03-11T19:20:51.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.343+0000 7fcfcf7fe640 1 --2- 172.21.15.29:0/4070382396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfd01075e0 0x7fcfd019d5b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800d770 tx=0x7fcfb800dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:51.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.343+0000 7fcfccff9640 1 -- 172.21.15.29:0/4070382396 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8009c30 con 0x7fcfd01075e0 2024-03-11T19:20:51.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.344+0000 7fcfd618a640 1 -- 172.21.15.29:0/4070382396 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd01a56b0 con 0x7fcfd01075e0 2024-03-11T19:20:51.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.344+0000 7fcfd618a640 1 -- 172.21.15.29:0/4070382396 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd01a5c10 con 0x7fcfd01075e0 2024-03-11T19:20:51.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.344+0000 7fcfccff9640 1 -- 172.21.15.29:0/4070382396 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcfb80044d0 con 0x7fcfd01075e0 2024-03-11T19:20:51.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.344+0000 7fcfccff9640 1 -- 172.21.15.29:0/4070382396 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8009c30 con 0x7fcfd01075e0 2024-03-11T19:20:51.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.347+0000 7fcfccff9640 1 -- 172.21.15.29:0/4070382396 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fcfb8003b20 con 0x7fcfd01075e0 2024-03-11T19:20:51.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.348+0000 7fcfd8e35740 1 Processor -- start 2024-03-11T19:20:51.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.348+0000 7fcfd8e35740 1 -- start start 2024-03-11T19:20:51.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.348+0000 7fcfd8e35740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82480 0x562752b828a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.348+0000 7fcfd8e35740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b82e70 0x562752b87970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.348+0000 7fcfd8e35740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b87fe0 0x562752b8a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.348+0000 7fcfd8e35740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562752b3e960 con 0x562752b87fe0 2024-03-11T19:20:51.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.348+0000 7fcfd8e35740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562752af2310 con 0x562752b82480 2024-03-11T19:20:51.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.348+0000 7fcfd8e35740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562752aef840 con 0x562752b82e70 2024-03-11T19:20:51.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.348+0000 7fcfccff9640 1 --2- 172.21.15.29:0/4070382396 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcfbc0a1ae0 0x7fcfbc0a3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.349+0000 7fcfceffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b87fe0 0x562752b8a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.349+0000 7fcfceffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b87fe0 0x562752b8a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48514/0 (socket says 172.21.15.29:48514) 2024-03-11T19:20:51.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.349+0000 7fcfceffd640 1 -- 172.21.15.29:0/2891851526 learned_addr learned my addr 172.21.15.29:0/2891851526 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:51.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.349+0000 7fcfceffd640 1 -- 172.21.15.29:0/2891851526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b82e70 msgr2=0x562752b87970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:20:51.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.349+0000 7fcfceffd640 1 --2- 172.21.15.29:0/2891851526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b82e70 0x562752b87970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.349+0000 7fcfceffd640 1 -- 172.21.15.29:0/2891851526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82480 msgr2=0x562752b828a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:20:51.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.349+0000 7fcfceffd640 1 --2- 172.21.15.29:0/2891851526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82480 0x562752b828a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.349+0000 7fcfceffd640 1 -- 172.21.15.29:0/2891851526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562752aad060 con 0x562752b87fe0 2024-03-11T19:20:51.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.349+0000 7fcfccff9640 1 -- 172.21.15.29:0/4070382396 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fcfb809a030 con 0x7fcfd01075e0 2024-03-11T19:20:51.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.349+0000 7fcfceffd640 1 --2- 172.21.15.29:0/2891851526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b87fe0 0x562752b8a470 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009900 tx=0x7fcfc8005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fba066fdd15a90 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:51.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.350+0000 7fcfc5ffb640 1 -- 172.21.15.29:0/2891851526 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc802cbf0 con 0x562752b87fe0 2024-03-11T19:20:51.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.350+0000 7fcfc5ffb640 1 -- 172.21.15.29:0/2891851526 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcfc802de90 con 0x562752b87fe0 2024-03-11T19:20:51.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.351+0000 7fcfd8e35740 1 -- 172.21.15.29:0/2891851526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b87fe0 msgr2=0x562752b8a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.351+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/2891851526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b87fe0 0x562752b8a470 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009900 tx=0x7fcfc8005e80 comp rx=0 tx=0).stop 2024-03-11T19:20:51.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.351+0000 7fcfd8e35740 1 -- 172.21.15.29:0/2891851526 shutdown_connections 2024-03-11T19:20:51.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.351+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/2891851526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b87fe0 0x562752b8a470 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.351+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/2891851526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b82e70 0x562752b87970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.351+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/2891851526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82480 0x562752b828a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.351+0000 7fcfd8e35740 1 -- 172.21.15.29:0/2891851526 >> 172.21.15.29:0/2891851526 conn(0x562752b7d9f0 msgr2=0x562752b7ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:51.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.352+0000 7fcfcffff640 1 --2- 172.21.15.29:0/4070382396 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcfbc0a1ae0 0x7fcfbc0a3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.353+0000 7fcfcffff640 1 --2- 172.21.15.29:0/4070382396 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcfbc0a1ae0 0x7fcfbc0a3fa0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x562752b836d0 tx=0x7fcfc00096a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:51.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.360+0000 7fcfd8e35740 1 -- 172.21.15.29:0/2891851526 shutdown_connections 2024-03-11T19:20:51.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.360+0000 7fcfd8e35740 1 -- 172.21.15.29:0/2891851526 wait complete. 2024-03-11T19:20:51.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.361+0000 7fcfd8e35740 1 -- start start 2024-03-11T19:20:51.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.361+0000 7fcfd8e35740 1 Processor -- start 2024-03-11T19:20:51.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.361+0000 7fcfd8e35740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82480 0x562752b8c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.361+0000 7fcfd8e35740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b82e70 0x562752b8cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.361+0000 7fcfd8e35740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b87fe0 0x562752b93b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.361+0000 7fcfd8e35740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562752aeb020 con 0x562752b87fe0 2024-03-11T19:20:51.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.361+0000 7fcfd8e35740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627527be430 con 0x562752b82480 2024-03-11T19:20:51.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.361+0000 7fcfd8e35740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56275277ae00 con 0x562752b82e70 2024-03-11T19:20:51.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.361+0000 7fcfd8e35740 10 client.0 _ll_register_callbacks cb 0x562752a92cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:51.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.362+0000 7fcfceffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b82e70 0x562752b8cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.362+0000 7fcfceffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b82e70 0x562752b8cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:50624/0 (socket says 172.21.15.29:50624) 2024-03-11T19:20:51.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.362+0000 7fcfceffd640 1 -- 172.21.15.29:0/1571464044 learned_addr learned my addr 172.21.15.29:0/1571464044 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:51.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.362+0000 7fcfcffff640 1 --2- 172.21.15.29:0/1571464044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b87fe0 0x562752b93b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.362+0000 7fcfceffd640 1 -- 172.21.15.29:0/1571464044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82480 msgr2=0x562752b8c5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:20:51.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.362+0000 7fcfceffd640 1 --2- 172.21.15.29:0/1571464044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82480 0x562752b8c5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.362+0000 7fcfceffd640 1 -- 172.21.15.29:0/1571464044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b87fe0 msgr2=0x562752b93b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.362+0000 7fcfceffd640 1 --2- 172.21.15.29:0/1571464044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b87fe0 0x562752b93b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.362+0000 7fcfceffd640 1 -- 172.21.15.29:0/1571464044 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd0110320 con 0x562752b82e70 2024-03-11T19:20:51.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.362+0000 7fcfceffd640 1 --2- 172.21.15.29:0/1571464044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b82e70 0x562752b8cb30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcfc802c170 tx=0x7fcfc8004b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdf20e946571982a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:51.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.363+0000 7fcfaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:20:51.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.363+0000 7fcfaeffd640 1 -- 172.21.15.29:0/1571464044 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc80041c0 con 0x562752b82e70 2024-03-11T19:20:51.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.363+0000 7fcfd8e35740 1 -- 172.21.15.29:0/1571464044 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5627527ea7e0 con 0x562752b82e70 2024-03-11T19:20:51.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.363+0000 7fcfaeffd640 1 -- 172.21.15.29:0/1571464044 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fcfc8004dc0 con 0x562752b82e70 2024-03-11T19:20:51.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.364+0000 7fcfaeffd640 1 -- 172.21.15.29:0/1571464044 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfc8004f80 con 0x562752b82e70 2024-03-11T19:20:51.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.364+0000 7fcfd8e35740 10 client.27523 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:51.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.364+0000 7fcfd8e35740 1 -- 172.21.15.29:0/1571464044 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x562752ae96a0 con 0x562752b82e70 2024-03-11T19:20:51.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.367+0000 7fcfaeffd640 1 -- 172.21.15.29:0/1571464044 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5627527ea7e0 con 0x562752b82e70 2024-03-11T19:20:51.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.367+0000 7fcfaeffd640 1 -- 172.21.15.29:0/1571464044 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcfc803c040 con 0x562752b82e70 2024-03-11T19:20:51.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.367+0000 7fcfd8e35740 10 client.27523 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:51.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.367+0000 7fcfd8e35740 10 client.27523 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:20:51.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.367+0000 7fcfd8e35740 20 client.27523 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:51.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.367+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/1571464044 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x562752ba9370 0x562752bab790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.367+0000 7fcfd8e35740 4 client.27523 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:20:51.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.367+0000 7fcfd8e35740 1 -- 172.21.15.29:0/1571464044 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562752827720 con 0x562752ba9370 2024-03-11T19:20:51.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.370+0000 7fcfcf7fe640 1 --2- 172.21.15.29:0/1571464044 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x562752ba9370 0x562752bab790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.371+0000 7fcfcf7fe640 1 --2- 172.21.15.29:0/1571464044 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x562752ba9370 0x562752bab790 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=80b2094cce3b252a server_cookie=8ea834dccfaebcc2 in_seq=0 out_seq=0 2024-03-11T19:20:51.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.371+0000 7fcfaeffd640 10 client.27523 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:20:51.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.372+0000 7fcfaeffd640 1 -- 172.21.15.29:0/1571464044 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562752827720 con 0x562752ba9370 2024-03-11T19:20:51.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.372+0000 7fcfaeffd640 10 client.27523 handle_command_reply: tid=0 2024-03-11T19:20:51.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.372+0000 7fcfd8e35740 1 client.27523 shutdown 2024-03-11T19:20:51.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.376+0000 7fcfd8e35740 1 -- 172.21.15.29:0/1571464044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b82e70 msgr2=0x562752b8cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.376+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/1571464044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b82e70 0x562752b8cb30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcfc802c170 tx=0x7fcfc8004b70 comp rx=0 tx=0).stop 2024-03-11T19:20:51.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.376+0000 7fcfd8e35740 1 -- 172.21.15.29:0/1571464044 shutdown_connections 2024-03-11T19:20:51.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.376+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/1571464044 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x562752ba9370 0x562752bab790 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.376+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/1571464044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b87fe0 0x562752b93b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.376+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/1571464044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b82e70 0x562752b8cb30 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.376+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/1571464044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82480 0x562752b8c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.376+0000 7fcfd8e35740 1 -- 172.21.15.29:0/1571464044 >> 172.21.15.29:0/1571464044 conn(0x562752b7d9f0 msgr2=0x562752b7ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:51.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.377+0000 7fcfaeffd640 0 client.27523 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:20:51.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.377+0000 7fcfd8e35740 1 -- 172.21.15.29:0/1571464044 shutdown_connections 2024-03-11T19:20:51.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.377+0000 7fcfd8e35740 1 -- 172.21.15.29:0/1571464044 wait complete. 2024-03-11T19:20:51.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.378+0000 7fcfd8e35740 20 client.27523 trim_cache size 0 max 16384 2024-03-11T19:20:51.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.392+0000 7fcfd8e35740 1 Processor -- start 2024-03-11T19:20:51.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.392+0000 7fcfd8e35740 1 -- start start 2024-03-11T19:20:51.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.392+0000 7fcfd8e35740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82480 0x562752b8c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.392+0000 7fcfd8e35740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b82e70 0x562752b93b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.392+0000 7fcfd8e35740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b87fe0 0x562752b8caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.393+0000 7fcfd8e35740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562752a70130 con 0x562752b87fe0 2024-03-11T19:20:51.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.393+0000 7fcfd8e35740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627527ea810 con 0x562752b82480 2024-03-11T19:20:51.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.393+0000 7fcfd8e35740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562752b8aaa0 con 0x562752b82e70 2024-03-11T19:20:51.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.393+0000 7fcfcf7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82480 0x562752b8c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.393+0000 7fcfcf7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82480 0x562752b8c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49566/0 (socket says 172.21.15.29:49566) 2024-03-11T19:20:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.393+0000 7fcfcf7fe640 1 -- 172.21.15.29:0/3663725996 learned_addr learned my addr 172.21.15.29:0/3663725996 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.393+0000 7fcfcffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b87fe0 0x562752b8caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.393+0000 7fcfceffd640 1 --2- 172.21.15.29:0/3663725996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b82e70 0x562752b93b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.393+0000 7fcfcf7fe640 1 -- 172.21.15.29:0/3663725996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b82e70 msgr2=0x562752b93b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.393+0000 7fcfcf7fe640 1 --2- 172.21.15.29:0/3663725996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b82e70 0x562752b93b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.393+0000 7fcfcf7fe640 1 -- 172.21.15.29:0/3663725996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b87fe0 msgr2=0x562752b8caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.393+0000 7fcfcf7fe640 1 --2- 172.21.15.29:0/3663725996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b87fe0 0x562752b8caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.393+0000 7fcfcf7fe640 1 -- 172.21.15.29:0/3663725996 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562752a85c20 con 0x562752b82480 2024-03-11T19:20:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.393+0000 7fcfcf7fe640 1 --2- 172.21.15.29:0/3663725996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82480 0x562752b8c5b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8096840 tx=0x7fcfb809e930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb133cd2ebb361ab server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.394+0000 7fcfc57fa640 1 -- 172.21.15.29:0/3663725996 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8009c30 con 0x562752b82480 2024-03-11T19:20:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.394+0000 7fcfc57fa640 1 -- 172.21.15.29:0/3663725996 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcfb809e070 con 0x562752b82480 2024-03-11T19:20:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.394+0000 7fcfc57fa640 1 -- 172.21.15.29:0/3663725996 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb80ca5d0 con 0x562752b82480 2024-03-11T19:20:51.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.394+0000 7fcfd8e35740 1 -- 172.21.15.29:0/3663725996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82480 msgr2=0x562752b8c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.394+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/3663725996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82480 0x562752b8c5b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8096840 tx=0x7fcfb809e930 comp rx=0 tx=0).stop 2024-03-11T19:20:51.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.394+0000 7fcfd8e35740 1 -- 172.21.15.29:0/3663725996 shutdown_connections 2024-03-11T19:20:51.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.394+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/3663725996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b87fe0 0x562752b8caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.394+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/3663725996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b82e70 0x562752b93b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.394+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/3663725996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82480 0x562752b8c5b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.394+0000 7fcfd8e35740 1 -- 172.21.15.29:0/3663725996 >> 172.21.15.29:0/3663725996 conn(0x562752b7d9f0 msgr2=0x562752b84290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.394+0000 7fcfd8e35740 1 -- 172.21.15.29:0/3663725996 shutdown_connections 2024-03-11T19:20:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.395+0000 7fcfd8e35740 1 -- 172.21.15.29:0/3663725996 wait complete. 2024-03-11T19:20:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.395+0000 7fcfd8e35740 1 -- start start 2024-03-11T19:20:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.395+0000 7fcfd8e35740 1 Processor -- start 2024-03-11T19:20:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.395+0000 7fcfd8e35740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b82480 0x562752b92220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.395+0000 7fcfd8e35740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82e70 0x562752b92760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.395+0000 7fcfd8e35740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b87fe0 0x562752b92ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.395+0000 7fcfd8e35740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562752b93e00 con 0x562752b82480 2024-03-11T19:20:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.395+0000 7fcfd8e35740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562752bff860 con 0x562752b82e70 2024-03-11T19:20:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.395+0000 7fcfd8e35740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562752bff9a0 con 0x562752b87fe0 2024-03-11T19:20:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.395+0000 7fcfd8e35740 10 client.0 _ll_register_callbacks cb 0x7fcfc8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.395+0000 7fcfcf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b82480 0x562752b92220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.395+0000 7fcfcf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b82480 0x562752b92220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48560/0 (socket says 172.21.15.29:48560) 2024-03-11T19:20:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.395+0000 7fcfcf7fe640 1 -- 172.21.15.29:0/757335375 learned_addr learned my addr 172.21.15.29:0/757335375 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.395+0000 7fcfcffff640 1 --2- 172.21.15.29:0/757335375 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b87fe0 0x562752b92ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.395+0000 7fcfceffd640 1 --2- 172.21.15.29:0/757335375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82e70 0x562752b92760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.396+0000 7fcfcf7fe640 1 -- 172.21.15.29:0/757335375 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b87fe0 msgr2=0x562752b92ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.396+0000 7fcfcf7fe640 1 --2- 172.21.15.29:0/757335375 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b87fe0 0x562752b92ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.396+0000 7fcfcf7fe640 1 -- 172.21.15.29:0/757335375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82e70 msgr2=0x562752b92760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.396+0000 7fcfcf7fe640 1 --2- 172.21.15.29:0/757335375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82e70 0x562752b92760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.396+0000 7fcfcf7fe640 1 -- 172.21.15.29:0/757335375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd01a5c10 con 0x562752b82480 2024-03-11T19:20:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.396+0000 7fcfcf7fe640 1 --2- 172.21.15.29:0/757335375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b82480 0x562752b92220 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8096840 tx=0x7fcfb80ca7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d56ac33552d87ba server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.396+0000 7fcfaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:20:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.396+0000 7fcfaeffd640 1 -- 172.21.15.29:0/757335375 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb80961b0 con 0x562752b82480 2024-03-11T19:20:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.396+0000 7fcfaeffd640 1 -- 172.21.15.29:0/757335375 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fcfb809a030 con 0x562752b82480 2024-03-11T19:20:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.396+0000 7fcfd8e35740 1 -- 172.21.15.29:0/757335375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcfc802de90 con 0x562752b82480 2024-03-11T19:20:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.396+0000 7fcfaeffd640 1 -- 172.21.15.29:0/757335375 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfb80a20d0 con 0x562752b82480 2024-03-11T19:20:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.396+0000 7fcfd8e35740 10 client.17970 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.396+0000 7fcfd8e35740 1 -- 172.21.15.29:0/757335375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x562752aab8c0 con 0x562752b82480 2024-03-11T19:20:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.397+0000 7fcfaeffd640 1 -- 172.21.15.29:0/757335375 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcfc802de90 con 0x562752b82480 2024-03-11T19:20:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.397+0000 7fcfaeffd640 1 -- 172.21.15.29:0/757335375 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcfb809e9a0 con 0x562752b82480 2024-03-11T19:20:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.397+0000 7fcfd8e35740 10 client.17970 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.397+0000 7fcfd8e35740 10 client.17970 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:20:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.397+0000 7fcfd8e35740 20 client.17970 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.397+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/757335375 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x562752c01000 0x562752c03420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.397+0000 7fcfd8e35740 4 client.17970 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:20:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.397+0000 7fcfd8e35740 1 -- 172.21.15.29:0/757335375 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562752876890 con 0x562752c01000 2024-03-11T19:20:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.397+0000 7fcfceffd640 1 --2- 172.21.15.29:0/757335375 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x562752c01000 0x562752c03420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.398+0000 7fcfaeffd640 10 client.17970 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:20:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.398+0000 7fcfceffd640 1 --2- 172.21.15.29:0/757335375 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x562752c01000 0x562752c03420 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fa2680ea031d161 server_cookie=3c309a80c51d6d54 in_seq=0 out_seq=0 2024-03-11T19:20:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.398+0000 7fcfaeffd640 1 -- 172.21.15.29:0/757335375 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562752876890 con 0x562752c01000 2024-03-11T19:20:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.398+0000 7fcfaeffd640 10 client.17970 handle_command_reply: tid=0 2024-03-11T19:20:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.398+0000 7fcfd8e35740 1 client.17970 shutdown 2024-03-11T19:20:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.398+0000 7fcfd8e35740 1 -- 172.21.15.29:0/757335375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b82480 msgr2=0x562752b92220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.398+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/757335375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b82480 0x562752b92220 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8096840 tx=0x7fcfb80ca7e0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.398+0000 7fcfd8e35740 1 -- 172.21.15.29:0/757335375 shutdown_connections 2024-03-11T19:20:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.398+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/757335375 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x562752c01000 0x562752c03420 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.398+0000 7fcfaeffd640 0 client.17970 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:20:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.398+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/757335375 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562752b87fe0 0x562752b92ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.398+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/757335375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562752b82e70 0x562752b92760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.398+0000 7fcfd8e35740 1 --2- 172.21.15.29:0/757335375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562752b82480 0x562752b92220 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.398+0000 7fcfd8e35740 1 -- 172.21.15.29:0/757335375 >> 172.21.15.29:0/757335375 conn(0x562752b7d9f0 msgr2=0x562752b80c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.398+0000 7fcfd8e35740 1 -- 172.21.15.29:0/757335375 shutdown_connections 2024-03-11T19:20:51.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.398+0000 7fcfd8e35740 1 -- 172.21.15.29:0/757335375 wait complete. 2024-03-11T19:20:51.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.398+0000 7fcfd8e35740 20 client.17970 trim_cache size 0 max 16384 2024-03-11T19:20:51.408 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:20:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.399+0000 7fcfd618a640 1 -- 172.21.15.29:0/4070382396 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcfbc0a1ae0 msgr2=0x7fcfbc0a3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.399+0000 7fcfd618a640 1 --2- 172.21.15.29:0/4070382396 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcfbc0a1ae0 0x7fcfbc0a3fa0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x562752b836d0 tx=0x7fcfc00096a0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.399+0000 7fcfd618a640 1 -- 172.21.15.29:0/4070382396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfd01075e0 msgr2=0x7fcfd019d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.399+0000 7fcfd618a640 1 --2- 172.21.15.29:0/4070382396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfd01075e0 0x7fcfd019d5b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800d770 tx=0x7fcfb800dc30 comp rx=0 tx=0).stop 2024-03-11T19:20:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.399+0000 7fcfd618a640 1 -- 172.21.15.29:0/4070382396 shutdown_connections 2024-03-11T19:20:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.399+0000 7fcfd618a640 1 --2- 172.21.15.29:0/4070382396 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fcfbc0a1ae0 0x7fcfbc0a3fa0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.399+0000 7fcfd618a640 1 --2- 172.21.15.29:0/4070382396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfd0109190 0x7fcfd01a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.399+0000 7fcfd618a640 1 --2- 172.21.15.29:0/4070382396 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfd01087d0 0x7fcfd019daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.399+0000 7fcfd618a640 1 --2- 172.21.15.29:0/4070382396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfd01075e0 0x7fcfd019d5b0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.399+0000 7fcfd618a640 1 -- 172.21.15.29:0/4070382396 >> 172.21.15.29:0/4070382396 conn(0x7fcfd0076580 msgr2=0x7fcfd010bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:51.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.399+0000 7fcfd618a640 1 -- 172.21.15.29:0/4070382396 shutdown_connections 2024-03-11T19:20:51.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.399+0000 7fcfd618a640 1 -- 172.21.15.29:0/4070382396 wait complete. 2024-03-11T19:20:51.413 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:20:51.413 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:20:51.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.514+0000 7f772119f640 1 -- 172.21.15.29:0/702707288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f771c0fff10 msgr2=0x7f771c100350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.514+0000 7f772119f640 1 --2- 172.21.15.29:0/702707288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f771c0fff10 0x7f771c100350 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f770c009900 tx=0x7f770c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:20:51.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.514+0000 7f772119f640 1 -- 172.21.15.29:0/702707288 shutdown_connections 2024-03-11T19:20:51.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.514+0000 7f772119f640 1 --2- 172.21.15.29:0/702707288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f771c101aa0 0x7f771c1083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.514+0000 7f772119f640 1 --2- 172.21.15.29:0/702707288 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f771c1010e0 0x7f771c101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.514+0000 7f772119f640 1 --2- 172.21.15.29:0/702707288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f771c0fff10 0x7f771c100350 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.514+0000 7f772119f640 1 -- 172.21.15.29:0/702707288 >> 172.21.15.29:0/702707288 conn(0x7f771c0fb4f0 msgr2=0x7f771c0fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:51.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.514+0000 7f772119f640 1 -- 172.21.15.29:0/702707288 shutdown_connections 2024-03-11T19:20:51.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.514+0000 7f772119f640 1 -- 172.21.15.29:0/702707288 wait complete. 2024-03-11T19:20:51.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.514+0000 7f772119f640 1 Processor -- start 2024-03-11T19:20:51.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.514+0000 7f772119f640 1 -- start start 2024-03-11T19:20:51.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f772119f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f771c0fff10 0x7f771c1991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f772119f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f771c1010e0 0x7f771c199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f772119f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f771c101aa0 0x7f771c1a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f772119f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c1a0cd0 con 0x7f771c101aa0 2024-03-11T19:20:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f772119f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c1a0e40 con 0x7f771c0fff10 2024-03-11T19:20:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f772119f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c1a1000 con 0x7f771c1010e0 2024-03-11T19:20:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f772099e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f771c101aa0 0x7f771c1a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f772099e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f771c101aa0 0x7f771c1a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48580/0 (socket says 172.21.15.29:48580) 2024-03-11T19:20:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f772099e640 1 -- 172.21.15.29:0/449866038 learned_addr learned my addr 172.21.15.29:0/449866038 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f771b7fe640 1 --2- 172.21.15.29:0/449866038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f771c1010e0 0x7f771c199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f771bfff640 1 --2- 172.21.15.29:0/449866038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f771c0fff10 0x7f771c1991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f772099e640 1 -- 172.21.15.29:0/449866038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f771c1010e0 msgr2=0x7f771c199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f772099e640 1 --2- 172.21.15.29:0/449866038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f771c1010e0 0x7f771c199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f772099e640 1 -- 172.21.15.29:0/449866038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f771c0fff10 msgr2=0x7f771c1991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f772099e640 1 --2- 172.21.15.29:0/449866038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f771c0fff10 0x7f771c1991e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f772099e640 1 -- 172.21.15.29:0/449866038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770c009560 con 0x7f771c101aa0 2024-03-11T19:20:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f771bfff640 1 --2- 172.21.15.29:0/449866038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f771c0fff10 0x7f771c1991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:20:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f772099e640 1 --2- 172.21.15.29:0/449866038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f771c101aa0 0x7f771c1a0790 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f771400cce0 tx=0x7f771400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f77197fa640 1 -- 172.21.15.29:0/449866038 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771400f450 con 0x7f771c101aa0 2024-03-11T19:20:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f77197fa640 1 -- 172.21.15.29:0/449866038 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f771400fa60 con 0x7f771c101aa0 2024-03-11T19:20:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f772119f640 1 -- 172.21.15.29:0/449866038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771c1a12e0 con 0x7f771c101aa0 2024-03-11T19:20:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.515+0000 7f77197fa640 1 -- 172.21.15.29:0/449866038 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7714018b10 con 0x7f771c101aa0 2024-03-11T19:20:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.516+0000 7f772119f640 1 -- 172.21.15.29:0/449866038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771c1a17a0 con 0x7f771c101aa0 2024-03-11T19:20:51.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.518+0000 7f77197fa640 1 -- 172.21.15.29:0/449866038 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f771400fbd0 con 0x7f771c101aa0 2024-03-11T19:20:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.518+0000 7f77197fa640 1 --2- 172.21.15.29:0/449866038 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f77080a1a30 0x7f77080a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.518+0000 7f7724fcb740 1 Processor -- start 2024-03-11T19:20:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.518+0000 7f7724fcb740 1 -- start start 2024-03-11T19:20:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.518+0000 7f77197fa640 1 -- 172.21.15.29:0/449866038 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f77140cd7a0 con 0x7f771c101aa0 2024-03-11T19:20:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.518+0000 7f771bfff640 1 --2- 172.21.15.29:0/449866038 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f77080a1a30 0x7f77080a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.518+0000 7f7724fcb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 0x5636ef1ef880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.518+0000 7f7724fcb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1efe50 0x5636ef1f4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.518+0000 7f7724fcb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1f4fc0 0x5636ef1f7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.518+0000 7f7724fcb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636ef1ab960 con 0x5636ef1ef460 2024-03-11T19:20:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.518+0000 7f7724fcb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636ef15f310 con 0x5636ef1f4fc0 2024-03-11T19:20:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.518+0000 7f7724fcb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636ef15c840 con 0x5636ef1efe50 2024-03-11T19:20:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.518+0000 7f771bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 0x5636ef1ef880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.518+0000 7f771bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 0x5636ef1ef880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48588/0 (socket says 172.21.15.29:48588) 2024-03-11T19:20:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.519+0000 7f771bfff640 1 -- 172.21.15.29:0/486768935 learned_addr learned my addr 172.21.15.29:0/486768935 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.519+0000 7f771b7fe640 1 --2- 172.21.15.29:0/486768935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1efe50 0x5636ef1f4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.519+0000 7f772099e640 1 --2- 172.21.15.29:0/486768935 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1f4fc0 0x5636ef1f7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.519+0000 7f771bfff640 1 --2- 172.21.15.29:0/449866038 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f77080a1a30 0x7f77080a3ef0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f770c02c800 tx=0x7f770c0048c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.519+0000 7f771bfff640 1 -- 172.21.15.29:0/486768935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1efe50 msgr2=0x5636ef1f4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.519+0000 7f771bfff640 1 --2- 172.21.15.29:0/486768935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1efe50 0x5636ef1f4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.519+0000 7f771bfff640 1 -- 172.21.15.29:0/486768935 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1f4fc0 msgr2=0x5636ef1f7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.519+0000 7f771bfff640 1 --2- 172.21.15.29:0/486768935 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1f4fc0 0x5636ef1f7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.519+0000 7f771bfff640 1 -- 172.21.15.29:0/486768935 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636ef11a060 con 0x5636ef1ef460 2024-03-11T19:20:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.519+0000 7f771bfff640 1 --2- 172.21.15.29:0/486768935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 0x5636ef1ef880 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f770c02c630 tx=0x7f770c04a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66973b210afaac04 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.519+0000 7f77127fc640 1 -- 172.21.15.29:0/486768935 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c0037f0 con 0x5636ef1ef460 2024-03-11T19:20:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.519+0000 7f77127fc640 1 -- 172.21.15.29:0/486768935 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f770c003e00 con 0x5636ef1ef460 2024-03-11T19:20:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.519+0000 7f77127fc640 1 -- 172.21.15.29:0/486768935 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c005680 con 0x5636ef1ef460 2024-03-11T19:20:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.520+0000 7f7724fcb740 1 -- 172.21.15.29:0/486768935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 msgr2=0x5636ef1ef880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.520+0000 7f7724fcb740 1 --2- 172.21.15.29:0/486768935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 0x5636ef1ef880 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f770c02c630 tx=0x7f770c04a000 comp rx=0 tx=0).stop 2024-03-11T19:20:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.520+0000 7f7724fcb740 1 -- 172.21.15.29:0/486768935 shutdown_connections 2024-03-11T19:20:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.520+0000 7f7724fcb740 1 --2- 172.21.15.29:0/486768935 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1f4fc0 0x5636ef1f7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.520+0000 7f7724fcb740 1 --2- 172.21.15.29:0/486768935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1efe50 0x5636ef1f4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.520+0000 7f7724fcb740 1 --2- 172.21.15.29:0/486768935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 0x5636ef1ef880 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.520+0000 7f7724fcb740 1 -- 172.21.15.29:0/486768935 >> 172.21.15.29:0/486768935 conn(0x5636ef1ea9d0 msgr2=0x5636ef1eadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.520+0000 7f7724fcb740 1 -- 172.21.15.29:0/486768935 shutdown_connections 2024-03-11T19:20:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.520+0000 7f7724fcb740 1 -- 172.21.15.29:0/486768935 wait complete. 2024-03-11T19:20:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.520+0000 7f7724fcb740 1 -- start start 2024-03-11T19:20:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.520+0000 7f7724fcb740 1 Processor -- start 2024-03-11T19:20:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f7724fcb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 0x5636ef1f9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f7724fcb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1efe50 0x5636ef1f9b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f7724fcb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1f4fc0 0x5636ef200b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f7724fcb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636ef158020 con 0x5636ef1ef460 2024-03-11T19:20:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f7724fcb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636eee2b430 con 0x5636ef1f4fc0 2024-03-11T19:20:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f7724fcb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636eede7e00 con 0x5636ef1efe50 2024-03-11T19:20:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f771bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 0x5636ef1f9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f7724fcb740 10 client.0 _ll_register_callbacks cb 0x5636ef0ffcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f771bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 0x5636ef1f9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48600/0 (socket says 172.21.15.29:48600) 2024-03-11T19:20:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f771bfff640 1 -- 172.21.15.29:0/663159274 learned_addr learned my addr 172.21.15.29:0/663159274 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f771b7fe640 1 --2- 172.21.15.29:0/663159274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1efe50 0x5636ef1f9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f772099e640 1 --2- 172.21.15.29:0/663159274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1f4fc0 0x5636ef200b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f771bfff640 1 -- 172.21.15.29:0/663159274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1efe50 msgr2=0x5636ef1f9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f771bfff640 1 --2- 172.21.15.29:0/663159274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1efe50 0x5636ef1f9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f771bfff640 1 -- 172.21.15.29:0/663159274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1f4fc0 msgr2=0x5636ef200b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f771bfff640 1 --2- 172.21.15.29:0/663159274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1f4fc0 0x5636ef200b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f771bfff640 1 -- 172.21.15.29:0/663159274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771c108c80 con 0x5636ef1ef460 2024-03-11T19:20:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f771bfff640 1 --2- 172.21.15.29:0/663159274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 0x5636ef1f9610 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f770c02c6d0 tx=0x7f770c005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9ce021640a13bd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:51.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f76fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:20:51.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f76fb7fe640 1 -- 172.21.15.29:0/663159274 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c003a00 con 0x5636ef1ef460 2024-03-11T19:20:51.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f76fb7fe640 1 -- 172.21.15.29:0/663159274 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f770c003b60 con 0x5636ef1ef460 2024-03-11T19:20:51.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.521+0000 7f7724fcb740 1 -- 172.21.15.29:0/663159274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5636eee577e0 con 0x5636ef1ef460 2024-03-11T19:20:51.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.522+0000 7f76fb7fe640 1 -- 172.21.15.29:0/663159274 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f770c04d500 con 0x5636ef1ef460 2024-03-11T19:20:51.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.522+0000 7f7724fcb740 10 client.17982 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:51.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.522+0000 7f7724fcb740 1 -- 172.21.15.29:0/663159274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5636ef1566a0 con 0x5636ef1ef460 2024-03-11T19:20:51.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.522+0000 7f76fb7fe640 1 -- 172.21.15.29:0/663159274 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5636eee577e0 con 0x5636ef1ef460 2024-03-11T19:20:51.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.522+0000 7f76fb7fe640 1 -- 172.21.15.29:0/663159274 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f770c04b070 con 0x5636ef1ef460 2024-03-11T19:20:51.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.522+0000 7f7724fcb740 10 client.17982 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:51.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.522+0000 7f7724fcb740 10 client.17982 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:20:51.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.522+0000 7f7724fcb740 20 client.17982 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:51.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.522+0000 7f7724fcb740 1 --2- 172.21.15.29:0/663159274 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5636ef2054b0 0x5636ef2078a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.522+0000 7f7724fcb740 4 client.17982 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:20:51.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.522+0000 7f7724fcb740 1 -- 172.21.15.29:0/663159274 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5636eee94720 con 0x5636ef2054b0 2024-03-11T19:20:51.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.522+0000 7f771b7fe640 1 --2- 172.21.15.29:0/663159274 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5636ef2054b0 0x5636ef2078a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.523+0000 7f771b7fe640 1 --2- 172.21.15.29:0/663159274 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5636ef2054b0 0x5636ef2078a0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1a40eb644f38d522 server_cookie=4b247b2963932a2c in_seq=0 out_seq=0 2024-03-11T19:20:51.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.523+0000 7f76fb7fe640 10 client.17982 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:20:51.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.523+0000 7f76fb7fe640 1 -- 172.21.15.29:0/663159274 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5636eee94720 con 0x5636ef2054b0 2024-03-11T19:20:51.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.523+0000 7f76fb7fe640 10 client.17982 handle_command_reply: tid=0 2024-03-11T19:20:51.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.523+0000 7f7724fcb740 1 client.17982 shutdown 2024-03-11T19:20:51.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.523+0000 7f7724fcb740 1 -- 172.21.15.29:0/663159274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 msgr2=0x5636ef1f9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.523+0000 7f7724fcb740 1 --2- 172.21.15.29:0/663159274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 0x5636ef1f9610 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f770c02c6d0 tx=0x7f770c005c00 comp rx=0 tx=0).stop 2024-03-11T19:20:51.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.524+0000 7f7724fcb740 1 -- 172.21.15.29:0/663159274 shutdown_connections 2024-03-11T19:20:51.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.524+0000 7f7724fcb740 1 --2- 172.21.15.29:0/663159274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1f4fc0 0x5636ef200b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.524+0000 7f7724fcb740 1 --2- 172.21.15.29:0/663159274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1efe50 0x5636ef1f9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.524+0000 7f7724fcb740 1 --2- 172.21.15.29:0/663159274 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5636ef2054b0 0x5636ef2078a0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.524+0000 7f7724fcb740 1 --2- 172.21.15.29:0/663159274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 0x5636ef1f9610 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.524+0000 7f7724fcb740 1 -- 172.21.15.29:0/663159274 >> 172.21.15.29:0/663159274 conn(0x5636ef1ea9d0 msgr2=0x5636ef1e9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:51.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.524+0000 7f76fb7fe640 0 client.17982 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:20:51.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.524+0000 7f7724fcb740 1 -- 172.21.15.29:0/663159274 shutdown_connections 2024-03-11T19:20:51.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.524+0000 7f7724fcb740 1 -- 172.21.15.29:0/663159274 wait complete. 2024-03-11T19:20:51.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.524+0000 7f7724fcb740 20 client.17982 trim_cache size 0 max 16384 2024-03-11T19:20:51.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.535+0000 7f7724fcb740 1 Processor -- start 2024-03-11T19:20:51.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.535+0000 7f7724fcb740 1 -- start start 2024-03-11T19:20:51.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.535+0000 7f7724fcb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1ef460 0x5636ef1f9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.535+0000 7f7724fcb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1efe50 0x5636ef200b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.535+0000 7f7724fcb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1f4fc0 0x5636ef1f9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.535+0000 7f7724fcb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636ef0dd130 con 0x5636ef1f4fc0 2024-03-11T19:20:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.535+0000 7f7724fcb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636ef1efcd0 con 0x5636ef1ef460 2024-03-11T19:20:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.535+0000 7f7724fcb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636ef1e6520 con 0x5636ef1efe50 2024-03-11T19:20:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.536+0000 7f772099e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1f4fc0 0x5636ef1f9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.536+0000 7f772099e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1f4fc0 0x5636ef1f9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48602/0 (socket says 172.21.15.29:48602) 2024-03-11T19:20:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.536+0000 7f772099e640 1 -- 172.21.15.29:0/208190861 learned_addr learned my addr 172.21.15.29:0/208190861 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.536+0000 7f771b7fe640 1 --2- 172.21.15.29:0/208190861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1efe50 0x5636ef200b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.536+0000 7f772099e640 1 -- 172.21.15.29:0/208190861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1efe50 msgr2=0x5636ef200b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.536+0000 7f772099e640 1 --2- 172.21.15.29:0/208190861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1efe50 0x5636ef200b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.536+0000 7f771bfff640 1 --2- 172.21.15.29:0/208190861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1ef460 0x5636ef1f9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.536+0000 7f772099e640 1 -- 172.21.15.29:0/208190861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1ef460 msgr2=0x5636ef1f9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.536+0000 7f772099e640 1 --2- 172.21.15.29:0/208190861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1ef460 0x5636ef1f9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.536+0000 7f772099e640 1 -- 172.21.15.29:0/208190861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636ef0f2c20 con 0x5636ef1f4fc0 2024-03-11T19:20:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.536+0000 7f771b7fe640 1 --2- 172.21.15.29:0/208190861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1efe50 0x5636ef200b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:20:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.536+0000 7f772099e640 1 --2- 172.21.15.29:0/208190861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1f4fc0 0x5636ef1f9760 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f771409a740 tx=0x7f7714085040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37aa6c084c8c505b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.536+0000 7f7711ffb640 1 -- 172.21.15.29:0/208190861 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771409a7b0 con 0x5636ef1f4fc0 2024-03-11T19:20:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.536+0000 7f7711ffb640 1 -- 172.21.15.29:0/208190861 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f771409e900 con 0x5636ef1f4fc0 2024-03-11T19:20:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.536+0000 7f7711ffb640 1 -- 172.21.15.29:0/208190861 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77140aa3b0 con 0x5636ef1f4fc0 2024-03-11T19:20:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.537+0000 7f7724fcb740 1 -- 172.21.15.29:0/208190861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1f4fc0 msgr2=0x5636ef1f9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.537+0000 7f7724fcb740 1 --2- 172.21.15.29:0/208190861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1f4fc0 0x5636ef1f9760 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f771409a740 tx=0x7f7714085040 comp rx=0 tx=0).stop 2024-03-11T19:20:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.537+0000 7f7724fcb740 1 -- 172.21.15.29:0/208190861 shutdown_connections 2024-03-11T19:20:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.537+0000 7f7724fcb740 1 --2- 172.21.15.29:0/208190861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1f4fc0 0x5636ef1f9760 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.537+0000 7f7724fcb740 1 --2- 172.21.15.29:0/208190861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1efe50 0x5636ef200b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.537+0000 7f7724fcb740 1 --2- 172.21.15.29:0/208190861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1ef460 0x5636ef1f9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.537+0000 7f7724fcb740 1 -- 172.21.15.29:0/208190861 >> 172.21.15.29:0/208190861 conn(0x5636ef1ea9d0 msgr2=0x5636ef1f13a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.537+0000 7f7724fcb740 1 -- 172.21.15.29:0/208190861 shutdown_connections 2024-03-11T19:20:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.537+0000 7f7724fcb740 1 -- 172.21.15.29:0/208190861 wait complete. 2024-03-11T19:20:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.537+0000 7f7724fcb740 1 -- start start 2024-03-11T19:20:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.537+0000 7f7724fcb740 1 Processor -- start 2024-03-11T19:20:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.537+0000 7f7724fcb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 0x5636ef1ff670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.537+0000 7f7724fcb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1efe50 0x5636ef1ffbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.537+0000 7f7724fcb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1f4fc0 0x5636ef25b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.537+0000 7f7724fcb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636ef2003a0 con 0x5636ef1ef460 2024-03-11T19:20:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.537+0000 7f7724fcb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636ef200510 con 0x5636ef1efe50 2024-03-11T19:20:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.537+0000 7f7724fcb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636ef25bf00 con 0x5636ef1f4fc0 2024-03-11T19:20:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f771bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 0x5636ef1ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f7724fcb740 10 client.0 _ll_register_callbacks cb 0x7f770c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f771bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 0x5636ef1ff670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48616/0 (socket says 172.21.15.29:48616) 2024-03-11T19:20:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f771bfff640 1 -- 172.21.15.29:0/151042082 learned_addr learned my addr 172.21.15.29:0/151042082 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f771bfff640 1 -- 172.21.15.29:0/151042082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1f4fc0 msgr2=0x5636ef25b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f771bfff640 1 --2- 172.21.15.29:0/151042082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1f4fc0 0x5636ef25b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f771bfff640 1 -- 172.21.15.29:0/151042082 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1efe50 msgr2=0x5636ef1ffbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f771b7fe640 1 --2- 172.21.15.29:0/151042082 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1efe50 0x5636ef1ffbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f771bfff640 1 --2- 172.21.15.29:0/151042082 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1efe50 0x5636ef1ffbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f771bfff640 1 -- 172.21.15.29:0/151042082 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771c1a17a0 con 0x5636ef1ef460 2024-03-11T19:20:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f771bfff640 1 --2- 172.21.15.29:0/151042082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 0x5636ef1ff670 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f770c0049a0 tx=0x7f770c005300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dcb1ebd2d724be7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f76fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:20:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f76fb7fe640 1 -- 172.21.15.29:0/151042082 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c003800 con 0x5636ef1ef460 2024-03-11T19:20:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f76fb7fe640 1 -- 172.21.15.29:0/151042082 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f770c003960 con 0x5636ef1ef460 2024-03-11T19:20:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f7724fcb740 1 -- 172.21.15.29:0/151042082 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f770c003e00 con 0x5636ef1ef460 2024-03-11T19:20:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f76fb7fe640 1 -- 172.21.15.29:0/151042082 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f770c003b40 con 0x5636ef1ef460 2024-03-11T19:20:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f7724fcb740 10 client.17988 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f7724fcb740 1 -- 172.21.15.29:0/151042082 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5636ef1188c0 con 0x5636ef1ef460 2024-03-11T19:20:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f76fb7fe640 1 -- 172.21.15.29:0/151042082 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f770c003e00 con 0x5636ef1ef460 2024-03-11T19:20:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.538+0000 7f76fb7fe640 1 -- 172.21.15.29:0/151042082 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f770c04b040 con 0x5636ef1ef460 2024-03-11T19:20:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f7724fcb740 10 client.17988 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f7724fcb740 10 client.17988 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:20:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f7724fcb740 20 client.17988 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f7724fcb740 1 --2- 172.21.15.29:0/151042082 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5636ef25d500 0x5636ef25f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f7724fcb740 4 client.17988 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:20:51.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f7724fcb740 1 -- 172.21.15.29:0/151042082 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5636eeee3890 con 0x5636ef25d500 2024-03-11T19:20:51.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f771b7fe640 1 --2- 172.21.15.29:0/151042082 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5636ef25d500 0x5636ef25f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f76fb7fe640 10 client.17988 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:20:51.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f771b7fe640 1 --2- 172.21.15.29:0/151042082 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5636ef25d500 0x5636ef25f920 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1838c16c8b62c95c server_cookie=85eeb5b3ddfdb54f in_seq=0 out_seq=0 2024-03-11T19:20:51.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f76fb7fe640 1 -- 172.21.15.29:0/151042082 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5636eeee3890 con 0x5636ef25d500 2024-03-11T19:20:51.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f76fb7fe640 10 client.17988 handle_command_reply: tid=0 2024-03-11T19:20:51.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f7724fcb740 1 client.17988 shutdown 2024-03-11T19:20:51.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f7724fcb740 1 -- 172.21.15.29:0/151042082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 msgr2=0x5636ef1ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f7724fcb740 1 --2- 172.21.15.29:0/151042082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 0x5636ef1ff670 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f770c0049a0 tx=0x7f770c005300 comp rx=0 tx=0).stop 2024-03-11T19:20:51.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f7724fcb740 1 -- 172.21.15.29:0/151042082 shutdown_connections 2024-03-11T19:20:51.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f7724fcb740 1 --2- 172.21.15.29:0/151042082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636ef1f4fc0 0x5636ef25b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f7724fcb740 1 --2- 172.21.15.29:0/151042082 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5636ef1efe50 0x5636ef1ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f7724fcb740 1 --2- 172.21.15.29:0/151042082 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x5636ef25d500 0x5636ef25f920 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f7724fcb740 1 --2- 172.21.15.29:0/151042082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5636ef1ef460 0x5636ef1ff670 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f76fb7fe640 0 client.17988 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:20:51.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.539+0000 7f7724fcb740 1 -- 172.21.15.29:0/151042082 >> 172.21.15.29:0/151042082 conn(0x5636ef1ea9d0 msgr2=0x5636ef1f9220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:51.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.540+0000 7f7724fcb740 1 -- 172.21.15.29:0/151042082 shutdown_connections 2024-03-11T19:20:51.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.540+0000 7f7724fcb740 1 -- 172.21.15.29:0/151042082 wait complete. 2024-03-11T19:20:51.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.540+0000 7f7724fcb740 20 client.17988 trim_cache size 0 max 16384 2024-03-11T19:20:51.550 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:20:51.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.540+0000 7f772119f640 1 -- 172.21.15.29:0/449866038 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f77080a1a30 msgr2=0x7f77080a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.540+0000 7f772119f640 1 --2- 172.21.15.29:0/449866038 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f77080a1a30 0x7f77080a3ef0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f770c02c800 tx=0x7f770c0048c0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.540+0000 7f772119f640 1 -- 172.21.15.29:0/449866038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f771c101aa0 msgr2=0x7f771c1a0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.540+0000 7f772119f640 1 --2- 172.21.15.29:0/449866038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f771c101aa0 0x7f771c1a0790 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f771400cce0 tx=0x7f771400a620 comp rx=0 tx=0).stop 2024-03-11T19:20:51.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.540+0000 7f772119f640 1 -- 172.21.15.29:0/449866038 shutdown_connections 2024-03-11T19:20:51.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.540+0000 7f772119f640 1 --2- 172.21.15.29:0/449866038 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f77080a1a30 0x7f77080a3ef0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.540+0000 7f772119f640 1 --2- 172.21.15.29:0/449866038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f771c101aa0 0x7f771c1a0790 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.540+0000 7f772119f640 1 --2- 172.21.15.29:0/449866038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f771c1010e0 0x7f771c199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.540+0000 7f772119f640 1 --2- 172.21.15.29:0/449866038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f771c0fff10 0x7f771c1991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.540+0000 7f772119f640 1 -- 172.21.15.29:0/449866038 >> 172.21.15.29:0/449866038 conn(0x7f771c0fb4f0 msgr2=0x7f771c0fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:51.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.540+0000 7f772119f640 1 -- 172.21.15.29:0/449866038 shutdown_connections 2024-03-11T19:20:51.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.540+0000 7f772119f640 1 -- 172.21.15.29:0/449866038 wait complete. 2024-03-11T19:20:51.554 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:20:51.555 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:20:51.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.727+0000 7ff74cdc7640 1 -- 172.21.15.29:0/1448940241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff74810d4a0 msgr2=0x7ff74810d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.727+0000 7ff74cdc7640 1 --2- 172.21.15.29:0/1448940241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff74810d4a0 0x7ff74810d920 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff740009ca0 tx=0x7ff74002c120 comp rx=0 tx=0).stop 2024-03-11T19:20:51.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.727+0000 7ff74cdc7640 1 -- 172.21.15.29:0/1448940241 shutdown_connections 2024-03-11T19:20:51.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.727+0000 7ff74cdc7640 1 --2- 172.21.15.29:0/1448940241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff74810d4a0 0x7ff74810d920 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.727+0000 7ff74cdc7640 1 --2- 172.21.15.29:0/1448940241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff74810aaf0 0x7ff74810cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.727+0000 7ff74cdc7640 1 --2- 172.21.15.29:0/1448940241 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff748108180 0x7ff74810a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.727+0000 7ff74cdc7640 1 -- 172.21.15.29:0/1448940241 >> 172.21.15.29:0/1448940241 conn(0x7ff74806dfe0 msgr2=0x7ff748070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:51.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.728+0000 7ff74cdc7640 1 -- 172.21.15.29:0/1448940241 shutdown_connections 2024-03-11T19:20:51.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.728+0000 7ff74cdc7640 1 -- 172.21.15.29:0/1448940241 wait complete. 2024-03-11T19:20:51.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.729+0000 7ff74cdc7640 1 Processor -- start 2024-03-11T19:20:51.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.729+0000 7ff74cdc7640 1 -- start start 2024-03-11T19:20:51.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.730+0000 7ff74cdc7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff748108180 0x7ff7481a6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.730+0000 7ff74cdc7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff74810aaf0 0x7ff7481a6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.730+0000 7ff74cdc7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff74810d4a0 0x7ff7481ad5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.730+0000 7ff74cdc7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7481adb20 con 0x7ff748108180 2024-03-11T19:20:51.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.730+0000 7ff74cdc7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74810f8e0 con 0x7ff74810d4a0 2024-03-11T19:20:51.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.730+0000 7ff74cdc7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74810faa0 con 0x7ff74810aaf0 2024-03-11T19:20:51.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.732+0000 7ff74659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff748108180 0x7ff7481a6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.732+0000 7ff74659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff748108180 0x7ff7481a6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48632/0 (socket says 172.21.15.29:48632) 2024-03-11T19:20:51.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.732+0000 7ff74659c640 1 -- 172.21.15.29:0/4051332646 learned_addr learned my addr 172.21.15.29:0/4051332646 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:51.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.733+0000 7ff73dd9b640 1 --2- 172.21.15.29:0/4051332646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff74810aaf0 0x7ff7481a6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.733+0000 7ff74659c640 1 -- 172.21.15.29:0/4051332646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff74810aaf0 msgr2=0x7ff7481a6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.733+0000 7ff74659c640 1 --2- 172.21.15.29:0/4051332646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff74810aaf0 0x7ff7481a6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.733+0000 7ff74659c640 1 -- 172.21.15.29:0/4051332646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff74810d4a0 msgr2=0x7ff7481ad5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.733+0000 7ff74659c640 1 --2- 172.21.15.29:0/4051332646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff74810d4a0 0x7ff7481ad5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.733+0000 7ff74659c640 1 -- 172.21.15.29:0/4051332646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff740009900 con 0x7ff748108180 2024-03-11T19:20:51.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.734+0000 7ff74659c640 1 --2- 172.21.15.29:0/4051332646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff748108180 0x7ff7481a6030 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7ff73000b450 tx=0x7ff73000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:51.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.735+0000 7ff73ffff640 1 -- 172.21.15.29:0/4051332646 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff730011890 con 0x7ff748108180 2024-03-11T19:20:51.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.735+0000 7ff74cdc7640 1 -- 172.21.15.29:0/4051332646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff74810fd80 con 0x7ff748108180 2024-03-11T19:20:51.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.735+0000 7ff73ffff640 1 -- 172.21.15.29:0/4051332646 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff7300103e0 con 0x7ff748108180 2024-03-11T19:20:51.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.735+0000 7ff73ffff640 1 -- 172.21.15.29:0/4051332646 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff730011ea0 con 0x7ff748108180 2024-03-11T19:20:51.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.735+0000 7ff74cdc7640 1 -- 172.21.15.29:0/4051332646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff748110310 con 0x7ff748108180 2024-03-11T19:20:51.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.739+0000 7ff73ffff640 1 -- 172.21.15.29:0/4051332646 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff730010550 con 0x7ff748108180 2024-03-11T19:20:51.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.740+0000 7ff73ffff640 1 --2- 172.21.15.29:0/4051332646 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff7340a1bb0 0x7ff7340a4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.740+0000 7ff73ffff640 1 -- 172.21.15.29:0/4051332646 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff7300108c0 con 0x7ff748108180 2024-03-11T19:20:51.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.741+0000 7ff73dd9b640 1 --2- 172.21.15.29:0/4051332646 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff7340a1bb0 0x7ff7340a4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.741+0000 7ff73dd9b640 1 --2- 172.21.15.29:0/4051332646 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff7340a1bb0 0x7ff7340a4070 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff738005fd0 tx=0x7ff738005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:51.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.744+0000 7ff74f998740 1 Processor -- start 2024-03-11T19:20:51.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.744+0000 7ff74f998740 1 -- start start 2024-03-11T19:20:51.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.744+0000 7ff74f998740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f65d270 0x56394f65d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.744+0000 7ff74f998740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f65dc60 0x56394f662760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.744+0000 7ff74f998740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f662dd0 0x56394f665260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.744+0000 7ff74f998740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56394f619960 con 0x56394f662dd0 2024-03-11T19:20:51.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.744+0000 7ff74f998740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56394f5cd310 con 0x56394f65dc60 2024-03-11T19:20:51.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.744+0000 7ff74f998740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56394f5ca840 con 0x56394f65d270 2024-03-11T19:20:51.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.747+0000 7ff746d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f662dd0 0x56394f665260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.747+0000 7ff746d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f662dd0 0x56394f665260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48648/0 (socket says 172.21.15.29:48648) 2024-03-11T19:20:51.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.747+0000 7ff746d9d640 1 -- 172.21.15.29:0/1160220031 learned_addr learned my addr 172.21.15.29:0/1160220031 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:51.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.749+0000 7ff73dd9b640 1 --2- 172.21.15.29:0/1160220031 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f65dc60 0x56394f662760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.749+0000 7ff74659c640 1 --2- 172.21.15.29:0/1160220031 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f65d270 0x56394f65d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.750+0000 7ff746d9d640 1 -- 172.21.15.29:0/1160220031 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f65d270 msgr2=0x56394f65d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.750+0000 7ff746d9d640 1 --2- 172.21.15.29:0/1160220031 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f65d270 0x56394f65d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.750+0000 7ff746d9d640 1 -- 172.21.15.29:0/1160220031 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f65dc60 msgr2=0x56394f662760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.750+0000 7ff746d9d640 1 --2- 172.21.15.29:0/1160220031 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f65dc60 0x56394f662760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.750+0000 7ff746d9d640 1 -- 172.21.15.29:0/1160220031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56394f588060 con 0x56394f662dd0 2024-03-11T19:20:51.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.750+0000 7ff746d9d640 1 --2- 172.21.15.29:0/1160220031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f662dd0 0x56394f665260 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7ff74002c6b0 tx=0x7ff740005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec616741ac022209 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:51.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.750+0000 7ff73cd99640 1 -- 172.21.15.29:0/1160220031 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff740034bc0 con 0x56394f662dd0 2024-03-11T19:20:51.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.751+0000 7ff73cd99640 1 -- 172.21.15.29:0/1160220031 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff740034d20 con 0x56394f662dd0 2024-03-11T19:20:51.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.751+0000 7ff73cd99640 1 -- 172.21.15.29:0/1160220031 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff740003bd0 con 0x56394f662dd0 2024-03-11T19:20:51.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.751+0000 7ff74f998740 1 -- 172.21.15.29:0/1160220031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f662dd0 msgr2=0x56394f665260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.751+0000 7ff74f998740 1 --2- 172.21.15.29:0/1160220031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f662dd0 0x56394f665260 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7ff74002c6b0 tx=0x7ff740005f80 comp rx=0 tx=0).stop 2024-03-11T19:20:51.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.752+0000 7ff74f998740 1 -- 172.21.15.29:0/1160220031 shutdown_connections 2024-03-11T19:20:51.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.752+0000 7ff74f998740 1 --2- 172.21.15.29:0/1160220031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f662dd0 0x56394f665260 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.752+0000 7ff74f998740 1 --2- 172.21.15.29:0/1160220031 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f65dc60 0x56394f662760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.752+0000 7ff74f998740 1 --2- 172.21.15.29:0/1160220031 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f65d270 0x56394f65d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.752+0000 7ff74f998740 1 -- 172.21.15.29:0/1160220031 >> 172.21.15.29:0/1160220031 conn(0x56394f6587e0 msgr2=0x56394f658be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:51.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.753+0000 7ff74f998740 1 -- 172.21.15.29:0/1160220031 shutdown_connections 2024-03-11T19:20:51.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.753+0000 7ff74f998740 1 -- 172.21.15.29:0/1160220031 wait complete. 2024-03-11T19:20:51.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.753+0000 7ff74f998740 1 -- start start 2024-03-11T19:20:51.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.754+0000 7ff74f998740 1 Processor -- start 2024-03-11T19:20:51.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.754+0000 7ff74f998740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f65d270 0x56394f667440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.754+0000 7ff74f998740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f65dc60 0x56394f667980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.754+0000 7ff74f998740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f662dd0 0x56394f66e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.754+0000 7ff74f998740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56394f5c6020 con 0x56394f65d270 2024-03-11T19:20:51.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.754+0000 7ff74f998740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56394f299430 con 0x56394f65dc60 2024-03-11T19:20:51.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.754+0000 7ff74f998740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56394f255e00 con 0x56394f662dd0 2024-03-11T19:20:51.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.754+0000 7ff74f998740 10 client.0 _ll_register_callbacks cb 0x56394f56dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:51.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.754+0000 7ff74659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f65d270 0x56394f667440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.754+0000 7ff74659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f65d270 0x56394f667440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48654/0 (socket says 172.21.15.29:48654) 2024-03-11T19:20:51.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.754+0000 7ff74659c640 1 -- 172.21.15.29:0/3881049504 learned_addr learned my addr 172.21.15.29:0/3881049504 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:51.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.754+0000 7ff73dd9b640 1 --2- 172.21.15.29:0/3881049504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f65dc60 0x56394f667980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.755+0000 7ff746d9d640 1 --2- 172.21.15.29:0/3881049504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f662dd0 0x56394f66e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.755+0000 7ff74659c640 1 -- 172.21.15.29:0/3881049504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f662dd0 msgr2=0x56394f66e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.755+0000 7ff74659c640 1 --2- 172.21.15.29:0/3881049504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f662dd0 0x56394f66e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.755+0000 7ff74659c640 1 -- 172.21.15.29:0/3881049504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f65dc60 msgr2=0x56394f667980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.755+0000 7ff74659c640 1 --2- 172.21.15.29:0/3881049504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f65dc60 0x56394f667980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.755+0000 7ff74659c640 1 -- 172.21.15.29:0/3881049504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff74810e4b0 con 0x56394f65d270 2024-03-11T19:20:51.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.755+0000 7ff74659c640 1 --2- 172.21.15.29:0/3881049504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f65d270 0x56394f667440 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7ff730096520 tx=0x7ff730094bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=105bb44627c240aa server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:51.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.755+0000 7ff72dffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:20:51.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.756+0000 7ff72dffb640 1 -- 172.21.15.29:0/3881049504 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73009a7c0 con 0x56394f65d270 2024-03-11T19:20:51.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.756+0000 7ff74f998740 1 -- 172.21.15.29:0/3881049504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56394f2c57e0 con 0x56394f65d270 2024-03-11T19:20:51.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.758+0000 7ff72dffb640 1 -- 172.21.15.29:0/3881049504 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff7300108c0 con 0x56394f65d270 2024-03-11T19:20:51.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.758+0000 7ff72dffb640 1 -- 172.21.15.29:0/3881049504 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7300a2860 con 0x56394f65d270 2024-03-11T19:20:51.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.758+0000 7ff74f998740 10 client.18000 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:51.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.758+0000 7ff74f998740 1 -- 172.21.15.29:0/3881049504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56394f5c46a0 con 0x56394f65d270 2024-03-11T19:20:51.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.760+0000 7ff72dffb640 1 -- 172.21.15.29:0/3881049504 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x56394f2c57e0 con 0x56394f65d270 2024-03-11T19:20:51.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.760+0000 7ff72dffb640 1 -- 172.21.15.29:0/3881049504 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff7300b2020 con 0x56394f65d270 2024-03-11T19:20:51.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.760+0000 7ff74f998740 10 client.18000 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:51.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.760+0000 7ff74f998740 10 client.18000 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:20:51.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.760+0000 7ff74f998740 20 client.18000 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:51.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.760+0000 7ff74f998740 1 --2- 172.21.15.29:0/3881049504 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56394f673510 0x56394f675930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.760+0000 7ff74f998740 4 client.18000 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:20:51.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.760+0000 7ff74f998740 1 -- 172.21.15.29:0/3881049504 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56394f302720 con 0x56394f673510 2024-03-11T19:20:51.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.763+0000 7ff73dd9b640 1 --2- 172.21.15.29:0/3881049504 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56394f673510 0x56394f675930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.764+0000 7ff73dd9b640 1 --2- 172.21.15.29:0/3881049504 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56394f673510 0x56394f675930 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e75fdbecef4398fa server_cookie=6ff151b158912ce1 in_seq=0 out_seq=0 2024-03-11T19:20:51.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.765+0000 7ff72dffb640 10 client.18000 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:20:51.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.765+0000 7ff72dffb640 1 -- 172.21.15.29:0/3881049504 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56394f302720 con 0x56394f673510 2024-03-11T19:20:51.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.765+0000 7ff72dffb640 10 client.18000 handle_command_reply: tid=0 2024-03-11T19:20:51.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.765+0000 7ff74f998740 1 client.18000 shutdown 2024-03-11T19:20:51.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.766+0000 7ff74f998740 1 -- 172.21.15.29:0/3881049504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f65d270 msgr2=0x56394f667440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.766+0000 7ff74f998740 1 --2- 172.21.15.29:0/3881049504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f65d270 0x56394f667440 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7ff730096520 tx=0x7ff730094bc0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.766+0000 7ff74f998740 1 -- 172.21.15.29:0/3881049504 shutdown_connections 2024-03-11T19:20:51.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.766+0000 7ff74f998740 1 --2- 172.21.15.29:0/3881049504 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56394f673510 0x56394f675930 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.766+0000 7ff74f998740 1 --2- 172.21.15.29:0/3881049504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f662dd0 0x56394f66e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.766+0000 7ff74f998740 1 --2- 172.21.15.29:0/3881049504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f65dc60 0x56394f667980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.766+0000 7ff74f998740 1 --2- 172.21.15.29:0/3881049504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f65d270 0x56394f667440 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.766+0000 7ff74f998740 1 -- 172.21.15.29:0/3881049504 >> 172.21.15.29:0/3881049504 conn(0x56394f6587e0 msgr2=0x56394f657b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:51.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.766+0000 7ff72dffb640 0 client.18000 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:20:51.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.767+0000 7ff74f998740 1 -- 172.21.15.29:0/3881049504 shutdown_connections 2024-03-11T19:20:51.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.767+0000 7ff74f998740 1 -- 172.21.15.29:0/3881049504 wait complete. 2024-03-11T19:20:51.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.767+0000 7ff74f998740 20 client.18000 trim_cache size 0 max 16384 2024-03-11T19:20:51.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.787+0000 7ff74f998740 1 Processor -- start 2024-03-11T19:20:51.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.788+0000 7ff74f998740 1 -- start start 2024-03-11T19:20:51.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.788+0000 7ff74f998740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f65d270 0x56394f65f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.788+0000 7ff74f998740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f65dc60 0x56394f65f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.788+0000 7ff74f998740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f662dd0 0x56394f66e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.788+0000 7ff74f998740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56394f54b130 con 0x56394f65d270 2024-03-11T19:20:51.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.788+0000 7ff74f998740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56394f302720 con 0x56394f662dd0 2024-03-11T19:20:51.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.789+0000 7ff73dd9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f65dc60 0x56394f65f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.789+0000 7ff73dd9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f65dc60 0x56394f65f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:50718/0 (socket says 172.21.15.29:50718) 2024-03-11T19:20:51.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.789+0000 7ff73dd9b640 1 -- 172.21.15.29:0/399628562 learned_addr learned my addr 172.21.15.29:0/399628562 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:51.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.789+0000 7ff74f998740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56394f2c57e0 con 0x56394f65dc60 2024-03-11T19:20:51.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.789+0000 7ff746d9d640 1 --2- 172.21.15.29:0/399628562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f662dd0 0x56394f66e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.789+0000 7ff74659c640 1 --2- 172.21.15.29:0/399628562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f65d270 0x56394f65f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.789+0000 7ff73dd9b640 1 -- 172.21.15.29:0/399628562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f662dd0 msgr2=0x56394f66e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.789+0000 7ff73dd9b640 1 --2- 172.21.15.29:0/399628562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f662dd0 0x56394f66e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.789+0000 7ff73dd9b640 1 -- 172.21.15.29:0/399628562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f65d270 msgr2=0x56394f65f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.789+0000 7ff73dd9b640 1 --2- 172.21.15.29:0/399628562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f65d270 0x56394f65f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.789+0000 7ff73dd9b640 1 -- 172.21.15.29:0/399628562 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56394f560c20 con 0x56394f65dc60 2024-03-11T19:20:51.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.790+0000 7ff73dd9b640 1 --2- 172.21.15.29:0/399628562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f65dc60 0x56394f65f660 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x56394f65e6d0 tx=0x7ff738015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=488b28842df799d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:51.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.792+0000 7ff72ffff640 1 -- 172.21.15.29:0/399628562 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff738025040 con 0x56394f65dc60 2024-03-11T19:20:51.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.792+0000 7ff72ffff640 1 -- 172.21.15.29:0/399628562 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff738020e80 con 0x56394f65dc60 2024-03-11T19:20:51.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.792+0000 7ff72ffff640 1 -- 172.21.15.29:0/399628562 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7380116a0 con 0x56394f65dc60 2024-03-11T19:20:51.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.793+0000 7ff74f998740 1 -- 172.21.15.29:0/399628562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f65dc60 msgr2=0x56394f65f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.793+0000 7ff74f998740 1 --2- 172.21.15.29:0/399628562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f65dc60 0x56394f65f660 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x56394f65e6d0 tx=0x7ff738015000 comp rx=0 tx=0).stop 2024-03-11T19:20:51.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.793+0000 7ff74f998740 1 -- 172.21.15.29:0/399628562 shutdown_connections 2024-03-11T19:20:51.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.793+0000 7ff74f998740 1 --2- 172.21.15.29:0/399628562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f662dd0 0x56394f66e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.793+0000 7ff74f998740 1 --2- 172.21.15.29:0/399628562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f65dc60 0x56394f65f660 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.793+0000 7ff74f998740 1 --2- 172.21.15.29:0/399628562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f65d270 0x56394f65f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.793+0000 7ff74f998740 1 -- 172.21.15.29:0/399628562 >> 172.21.15.29:0/399628562 conn(0x56394f6587e0 msgr2=0x56394f658bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:51.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.797+0000 7ff74f998740 1 -- 172.21.15.29:0/399628562 shutdown_connections 2024-03-11T19:20:51.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.797+0000 7ff74f998740 1 -- 172.21.15.29:0/399628562 wait complete. 2024-03-11T19:20:51.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.798+0000 7ff74f998740 1 -- start start 2024-03-11T19:20:51.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.799+0000 7ff74f998740 1 Processor -- start 2024-03-11T19:20:51.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.800+0000 7ff74f998740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f65d270 0x56394f66e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.800+0000 7ff74f998740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f65dc60 0x56394f66cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.800+0000 7ff74f998740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f662dd0 0x56394f66d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.800+0000 7ff73dd9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f65dc60 0x56394f66cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.800+0000 7ff73dd9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f65dc60 0x56394f66cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49672/0 (socket says 172.21.15.29:49672) 2024-03-11T19:20:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.801+0000 7ff73dd9b640 1 -- 172.21.15.29:0/3324089759 learned_addr learned my addr 172.21.15.29:0/3324089759 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.801+0000 7ff74f998740 1 -- 172.21.15.29:0/3324089759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56394f66db20 con 0x56394f662dd0 2024-03-11T19:20:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.801+0000 7ff74f998740 1 -- 172.21.15.29:0/3324089759 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56394f66dc90 con 0x56394f65dc60 2024-03-11T19:20:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.801+0000 7ff74f998740 1 -- 172.21.15.29:0/3324089759 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56394f6c9a20 con 0x56394f65d270 2024-03-11T19:20:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.801+0000 7ff746d9d640 1 --2- 172.21.15.29:0/3324089759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f662dd0 0x56394f66d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.801+0000 7ff74f998740 10 client.0 _ll_register_callbacks cb 0x7ff740037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:51.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.803+0000 7ff73dd9b640 1 -- 172.21.15.29:0/3324089759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f65d270 msgr2=0x56394f66e890 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:20:51.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.803+0000 7ff73dd9b640 1 --2- 172.21.15.29:0/3324089759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f65d270 0x56394f66e890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.803+0000 7ff73dd9b640 1 -- 172.21.15.29:0/3324089759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f662dd0 msgr2=0x56394f66d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:51.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.803+0000 7ff73dd9b640 1 --2- 172.21.15.29:0/3324089759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f662dd0 0x56394f66d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.803+0000 7ff73dd9b640 1 -- 172.21.15.29:0/3324089759 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff73800fc10 con 0x56394f65dc60 2024-03-11T19:20:51.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.804+0000 7ff73dd9b640 1 --2- 172.21.15.29:0/3324089759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f65dc60 0x56394f66cea0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff73800f9e0 tx=0x7ff7380274f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=601664c9bc0de98d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:51.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.804+0000 7ff72dffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:20:51.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.804+0000 7ff72dffb640 1 -- 172.21.15.29:0/3324089759 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff738025070 con 0x56394f65dc60 2024-03-11T19:20:51.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.804+0000 7ff74f998740 1 -- 172.21.15.29:0/3324089759 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56394f351890 con 0x56394f65dc60 2024-03-11T19:20:51.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.804+0000 7ff72dffb640 1 -- 172.21.15.29:0/3324089759 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff738020be0 con 0x56394f65dc60 2024-03-11T19:20:51.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.804+0000 7ff72dffb640 1 -- 172.21.15.29:0/3324089759 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff738020da0 con 0x56394f65dc60 2024-03-11T19:20:51.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.804+0000 7ff74f998740 10 client.27509 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:51.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.805+0000 7ff74f998740 1 -- 172.21.15.29:0/3324089759 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56394f5868c0 con 0x56394f65dc60 2024-03-11T19:20:51.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.805+0000 7ff72dffb640 1 -- 172.21.15.29:0/3324089759 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x56394f351890 con 0x56394f65dc60 2024-03-11T19:20:51.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.805+0000 7ff72dffb640 1 -- 172.21.15.29:0/3324089759 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff738004b10 con 0x56394f65dc60 2024-03-11T19:20:51.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.805+0000 7ff74f998740 10 client.27509 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:51.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.805+0000 7ff74f998740 10 client.27509 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:20:51.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.805+0000 7ff74f998740 20 client.27509 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:51.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.805+0000 7ff74f998740 1 --2- 172.21.15.29:0/3324089759 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56394f6cb410 0x56394f6cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:51.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.805+0000 7ff74f998740 4 client.27509 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:20:51.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.805+0000 7ff74f998740 1 -- 172.21.15.29:0/3324089759 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56394f271700 con 0x56394f6cb410 2024-03-11T19:20:51.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.806+0000 7ff74659c640 1 --2- 172.21.15.29:0/3324089759 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56394f6cb410 0x56394f6cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:51.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.806+0000 7ff72dffb640 10 client.27509 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:20:51.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.806+0000 7ff74659c640 1 --2- 172.21.15.29:0/3324089759 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56394f6cb410 0x56394f6cd830 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=159cee298d3c6c51 server_cookie=c2b4d1024f3bc8c9 in_seq=0 out_seq=0 2024-03-11T19:20:51.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.806+0000 7ff72dffb640 1 -- 172.21.15.29:0/3324089759 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56394f271700 con 0x56394f6cb410 2024-03-11T19:20:51.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.806+0000 7ff72dffb640 10 client.27509 handle_command_reply: tid=0 2024-03-11T19:20:51.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.806+0000 7ff74f998740 1 client.27509 shutdown 2024-03-11T19:20:51.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.807+0000 7ff74f998740 1 -- 172.21.15.29:0/3324089759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f65dc60 msgr2=0x56394f66cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.807+0000 7ff74f998740 1 --2- 172.21.15.29:0/3324089759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f65dc60 0x56394f66cea0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff73800f9e0 tx=0x7ff7380274f0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.807+0000 7ff74f998740 1 -- 172.21.15.29:0/3324089759 shutdown_connections 2024-03-11T19:20:51.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.807+0000 7ff74f998740 1 --2- 172.21.15.29:0/3324089759 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x56394f6cb410 0x56394f6cd830 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.807+0000 7ff74f998740 1 --2- 172.21.15.29:0/3324089759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56394f662dd0 0x56394f66d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.807+0000 7ff74f998740 1 --2- 172.21.15.29:0/3324089759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56394f65dc60 0x56394f66cea0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.807+0000 7ff74f998740 1 --2- 172.21.15.29:0/3324089759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56394f65d270 0x56394f66e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.807+0000 7ff72dffb640 0 client.27509 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:20:51.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.807+0000 7ff74f998740 1 -- 172.21.15.29:0/3324089759 >> 172.21.15.29:0/3324089759 conn(0x56394f6587e0 msgr2=0x56394f65f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:51.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.807+0000 7ff74f998740 1 -- 172.21.15.29:0/3324089759 shutdown_connections 2024-03-11T19:20:51.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.807+0000 7ff74f998740 1 -- 172.21.15.29:0/3324089759 wait complete. 2024-03-11T19:20:51.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.807+0000 7ff74f998740 20 client.27509 trim_cache size 0 max 16384 2024-03-11T19:20:51.813 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:20:51.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.808+0000 7ff74cdc7640 1 -- 172.21.15.29:0/4051332646 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff7340a1bb0 msgr2=0x7ff7340a4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.808+0000 7ff74cdc7640 1 --2- 172.21.15.29:0/4051332646 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff7340a1bb0 0x7ff7340a4070 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff738005fd0 tx=0x7ff738005850 comp rx=0 tx=0).stop 2024-03-11T19:20:51.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.808+0000 7ff74cdc7640 1 -- 172.21.15.29:0/4051332646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff748108180 msgr2=0x7ff7481a6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:51.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.808+0000 7ff74cdc7640 1 --2- 172.21.15.29:0/4051332646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff748108180 0x7ff7481a6030 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7ff73000b450 tx=0x7ff73000b910 comp rx=0 tx=0).stop 2024-03-11T19:20:51.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.808+0000 7ff74cdc7640 1 -- 172.21.15.29:0/4051332646 shutdown_connections 2024-03-11T19:20:51.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.808+0000 7ff74cdc7640 1 --2- 172.21.15.29:0/4051332646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff74810d4a0 0x7ff7481ad5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.808+0000 7ff74cdc7640 1 --2- 172.21.15.29:0/4051332646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff74810aaf0 0x7ff7481a6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.808+0000 7ff74cdc7640 1 --2- 172.21.15.29:0/4051332646 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff7340a1bb0 0x7ff7340a4070 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.808+0000 7ff74cdc7640 1 --2- 172.21.15.29:0/4051332646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff748108180 0x7ff7481a6030 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:51.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.808+0000 7ff74cdc7640 1 -- 172.21.15.29:0/4051332646 >> 172.21.15.29:0/4051332646 conn(0x7ff74806dfe0 msgr2=0x7ff748109810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:51.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.808+0000 7ff74cdc7640 1 -- 172.21.15.29:0/4051332646 shutdown_connections 2024-03-11T19:20:51.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:51.808+0000 7ff74cdc7640 1 -- 172.21.15.29:0/4051332646 wait complete. 2024-03-11T19:20:51.834 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:51 smithi029 ceph-mon[32439]: pgmap v682: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 58 MiB/s wr, 1.06k op/s 2024-03-11T19:20:51.834 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:51 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/4034402226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:20:51.835 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:20:51.835 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:20:51.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:51 smithi170 ceph-mon[35690]: pgmap v682: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 58 MiB/s wr, 1.06k op/s 2024-03-11T19:20:51.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:51 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/4034402226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:20:52.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:51 smithi190 ceph-mon[36292]: pgmap v682: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 58 MiB/s wr, 1.06k op/s 2024-03-11T19:20:52.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:51 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/4034402226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:20:52.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.030+0000 7f71a1c2b640 1 -- 172.21.15.29:0/911182549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f719c1087b0 msgr2=0x7f719c108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:52.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.030+0000 7f71a1c2b640 1 --2- 172.21.15.29:0/911182549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f719c1087b0 0x7f719c108c30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7188009900 tx=0x7f718802bc30 comp rx=0 tx=0).stop 2024-03-11T19:20:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.031+0000 7f71a1c2b640 1 -- 172.21.15.29:0/911182549 shutdown_connections 2024-03-11T19:20:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.031+0000 7f71a1c2b640 1 --2- 172.21.15.29:0/911182549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f719c109170 0x7f719c10fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.031+0000 7f71a1c2b640 1 --2- 172.21.15.29:0/911182549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f719c1087b0 0x7f719c108c30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.031+0000 7f71a1c2b640 1 --2- 172.21.15.29:0/911182549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f719c1075c0 0x7f719c1079e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.031+0000 7f71a1c2b640 1 -- 172.21.15.29:0/911182549 >> 172.21.15.29:0/911182549 conn(0x7f719c076580 msgr2=0x7f719c0789a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.031+0000 7f71a1c2b640 1 -- 172.21.15.29:0/911182549 shutdown_connections 2024-03-11T19:20:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.031+0000 7f71a1c2b640 1 -- 172.21.15.29:0/911182549 wait complete. 2024-03-11T19:20:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.033+0000 7f71a1c2b640 1 Processor -- start 2024-03-11T19:20:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.033+0000 7f71a1c2b640 1 -- start start 2024-03-11T19:20:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.033+0000 7f71a1c2b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f719c1075c0 0x7f719c19d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.033+0000 7f71a1c2b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f719c1087b0 0x7f719c19da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.034+0000 7f71a1c2b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f719c109170 0x7f719c1a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.034+0000 7f71a0c29640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f719c1075c0 0x7f719c19d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.034+0000 7f71a0c29640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f719c1075c0 0x7f719c19d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:50734/0 (socket says 172.21.15.29:50734) 2024-03-11T19:20:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.034+0000 7f71a0c29640 1 -- 172.21.15.29:0/1535856154 learned_addr learned my addr 172.21.15.29:0/1535856154 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:52.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.034+0000 7f719bfff640 1 --2- 172.21.15.29:0/1535856154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f719c1087b0 0x7f719c19da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.034+0000 7f71a1c2b640 1 -- 172.21.15.29:0/1535856154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c1a4fb0 con 0x7f719c1087b0 2024-03-11T19:20:52.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.034+0000 7f71a1c2b640 1 -- 172.21.15.29:0/1535856154 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c1a5120 con 0x7f719c109170 2024-03-11T19:20:52.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.034+0000 7f71a1c2b640 1 -- 172.21.15.29:0/1535856154 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c1a52e0 con 0x7f719c1075c0 2024-03-11T19:20:52.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.034+0000 7f71a142a640 1 --2- 172.21.15.29:0/1535856154 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f719c109170 0x7f719c1a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.034+0000 7f71a142a640 1 -- 172.21.15.29:0/1535856154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f719c1075c0 msgr2=0x7f719c19d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:52.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.034+0000 7f71a142a640 1 --2- 172.21.15.29:0/1535856154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f719c1075c0 0x7f719c19d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.034+0000 7f71a142a640 1 -- 172.21.15.29:0/1535856154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f719c1087b0 msgr2=0x7f719c19da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:52.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.034+0000 7f71a142a640 1 --2- 172.21.15.29:0/1535856154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f719c1087b0 0x7f719c19da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.034+0000 7f71a142a640 1 -- 172.21.15.29:0/1535856154 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7188009560 con 0x7f719c109170 2024-03-11T19:20:52.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.035+0000 7f71a142a640 1 --2- 172.21.15.29:0/1535856154 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f719c109170 0x7f719c1a4a70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7194009fd0 tx=0x7f719400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:52.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.035+0000 7f7199ffb640 1 -- 172.21.15.29:0/1535856154 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719400ce20 con 0x7f719c109170 2024-03-11T19:20:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.035+0000 7f71a1c2b640 1 -- 172.21.15.29:0/1535856154 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f719c1a55c0 con 0x7f719c109170 2024-03-11T19:20:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.035+0000 7f7199ffb640 1 -- 172.21.15.29:0/1535856154 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7194005020 con 0x7f719c109170 2024-03-11T19:20:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.035+0000 7f7199ffb640 1 -- 172.21.15.29:0/1535856154 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7194010690 con 0x7f719c109170 2024-03-11T19:20:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.035+0000 7f71a1c2b640 1 -- 172.21.15.29:0/1535856154 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719c1a5aa0 con 0x7f719c109170 2024-03-11T19:20:52.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.039+0000 7f71a5a5c740 1 Processor -- start 2024-03-11T19:20:52.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.040+0000 7f71a5a5c740 1 -- start start 2024-03-11T19:20:52.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.040+0000 7f71a5a5c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d3847480 0x5556d38478a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.040+0000 7f71a5a5c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847e70 0x5556d384c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.040+0000 7f71a5a5c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5556d384cfe0 0x5556d384f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.040+0000 7f71a5a5c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556d3803960 con 0x5556d384cfe0 2024-03-11T19:20:52.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.040+0000 7f71a5a5c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556d37b7310 con 0x5556d3847e70 2024-03-11T19:20:52.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.040+0000 7f71a5a5c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556d37b4840 con 0x5556d3847480 2024-03-11T19:20:52.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.041+0000 7f71a0c29640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847e70 0x5556d384c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.041+0000 7f71a0c29640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847e70 0x5556d384c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49688/0 (socket says 172.21.15.29:49688) 2024-03-11T19:20:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.041+0000 7f71a0c29640 1 -- 172.21.15.29:0/2268881830 learned_addr learned my addr 172.21.15.29:0/2268881830 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.041+0000 7f7199ffb640 1 -- 172.21.15.29:0/1535856154 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7194005190 con 0x7f719c109170 2024-03-11T19:20:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.041+0000 7f71a0c29640 1 -- 172.21.15.29:0/2268881830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d3847480 msgr2=0x5556d38478a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:20:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.041+0000 7f71a0c29640 1 --2- 172.21.15.29:0/2268881830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d3847480 0x5556d38478a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.041+0000 7f71a0c29640 1 -- 172.21.15.29:0/2268881830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5556d384cfe0 msgr2=0x5556d384f470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T19:20:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.041+0000 7f71a0c29640 1 --2- 172.21.15.29:0/2268881830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5556d384cfe0 0x5556d384f470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.041+0000 7f71a0c29640 1 -- 172.21.15.29:0/2268881830 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556d3772060 con 0x5556d3847e70 2024-03-11T19:20:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.042+0000 7f7199ffb640 1 --2- 172.21.15.29:0/1535856154 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f718c0a1b30 0x7f718c0a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.042+0000 7f71a0c29640 1 --2- 172.21.15.29:0/2268881830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847e70 0x5556d384c970 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f719000ad60 tx=0x7f719000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e7cf4b32159777 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.042+0000 7f7199ffb640 1 -- 172.21.15.29:0/1535856154 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7194014070 con 0x7f719c109170 2024-03-11T19:20:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.042+0000 7f7186ffd640 1 -- 172.21.15.29:0/2268881830 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7190005220 con 0x5556d3847e70 2024-03-11T19:20:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.042+0000 7f7186ffd640 1 -- 172.21.15.29:0/2268881830 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f719000ae50 con 0x5556d3847e70 2024-03-11T19:20:52.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.042+0000 7f719bfff640 1 --2- 172.21.15.29:0/1535856154 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f718c0a1b30 0x7f718c0a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.042+0000 7f7186ffd640 1 -- 172.21.15.29:0/2268881830 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7190010910 con 0x5556d3847e70 2024-03-11T19:20:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.043+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2268881830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847e70 msgr2=0x5556d384c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.043+0000 7f719bfff640 1 --2- 172.21.15.29:0/1535856154 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f718c0a1b30 0x7f718c0a3ff0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7188006010 tx=0x7f7188000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.043+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/2268881830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847e70 0x5556d384c970 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f719000ad60 tx=0x7f719000f790 comp rx=0 tx=0).stop 2024-03-11T19:20:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.043+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2268881830 shutdown_connections 2024-03-11T19:20:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.043+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/2268881830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5556d384cfe0 0x5556d384f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.043+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/2268881830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847e70 0x5556d384c970 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.043+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/2268881830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d3847480 0x5556d38478a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.043+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2268881830 >> 172.21.15.29:0/2268881830 conn(0x5556d38429f0 msgr2=0x5556d3842df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.043+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2268881830 shutdown_connections 2024-03-11T19:20:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.043+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2268881830 wait complete. 2024-03-11T19:20:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.044+0000 7f71a5a5c740 1 -- start start 2024-03-11T19:20:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.044+0000 7f71a5a5c740 1 Processor -- start 2024-03-11T19:20:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.044+0000 7f71a5a5c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d3847480 0x5556d38516c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.044+0000 7f71a5a5c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5556d3847e70 0x5556d3851c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.044+0000 7f71a5a5c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d384cfe0 0x5556d3858bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.044+0000 7f71a5a5c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556d37b0020 con 0x5556d3847e70 2024-03-11T19:20:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.044+0000 7f71a5a5c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556d3483430 con 0x5556d384cfe0 2024-03-11T19:20:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.044+0000 7f71a5a5c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556d343fe00 con 0x5556d3847480 2024-03-11T19:20:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.045+0000 7f71a5a5c740 10 client.0 _ll_register_callbacks cb 0x5556d3757cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.045+0000 7f71a0c29640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d3847480 0x5556d38516c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.045+0000 7f71a142a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d384cfe0 0x5556d3858bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.045+0000 7f71a0c29640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d3847480 0x5556d38516c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:50744/0 (socket says 172.21.15.29:50744) 2024-03-11T19:20:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.045+0000 7f719bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5556d3847e70 0x5556d3851c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.045+0000 7f71a142a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d384cfe0 0x5556d3858bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49696/0 (socket says 172.21.15.29:49696) 2024-03-11T19:20:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.045+0000 7f71a0c29640 1 -- 172.21.15.29:0/3438405002 learned_addr learned my addr 172.21.15.29:0/3438405002 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.045+0000 7f71a0c29640 1 -- 172.21.15.29:0/3438405002 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d384cfe0 msgr2=0x5556d3858bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.045+0000 7f71a0c29640 1 --2- 172.21.15.29:0/3438405002 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d384cfe0 0x5556d3858bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.045+0000 7f71a0c29640 1 -- 172.21.15.29:0/3438405002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5556d3847e70 msgr2=0x5556d3851c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.045+0000 7f71a0c29640 1 --2- 172.21.15.29:0/3438405002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5556d3847e70 0x5556d3851c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.045+0000 7f71a0c29640 1 -- 172.21.15.29:0/3438405002 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719000aa20 con 0x5556d3847480 2024-03-11T19:20:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.045+0000 7f71a142a640 1 --2- 172.21.15.29:0/3438405002 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d384cfe0 0x5556d3858bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:20:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.046+0000 7f71a0c29640 1 --2- 172.21.15.29:0/3438405002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d3847480 0x5556d38516c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f719001a040 tx=0x7f7190005260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdde8313df688371 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.046+0000 7f717bfff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:20:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.046+0000 7f717bfff640 1 -- 172.21.15.29:0/3438405002 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71900106b0 con 0x5556d3847480 2024-03-11T19:20:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.046+0000 7f717bfff640 1 -- 172.21.15.29:0/3438405002 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7190010810 con 0x5556d3847480 2024-03-11T19:20:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.046+0000 7f71a5a5c740 1 -- 172.21.15.29:0/3438405002 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5556d34af7e0 con 0x5556d3847480 2024-03-11T19:20:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.046+0000 7f717bfff640 1 -- 172.21.15.29:0/3438405002 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7190010a10 con 0x5556d3847480 2024-03-11T19:20:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.046+0000 7f71a5a5c740 10 client.27562 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.046+0000 7f71a5a5c740 1 -- 172.21.15.29:0/3438405002 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5556d37ae6a0 con 0x5556d3847480 2024-03-11T19:20:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.046+0000 7f717bfff640 1 -- 172.21.15.29:0/3438405002 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5556d34af7e0 con 0x5556d3847480 2024-03-11T19:20:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.047+0000 7f717bfff640 1 -- 172.21.15.29:0/3438405002 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7190020040 con 0x5556d3847480 2024-03-11T19:20:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.047+0000 7f71a5a5c740 10 client.27562 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.047+0000 7f71a5a5c740 10 client.27562 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:20:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.047+0000 7f71a5a5c740 20 client.27562 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.047+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/3438405002 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5556d385d720 0x5556d385fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.047+0000 7f71a5a5c740 4 client.27562 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:20:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.047+0000 7f71a5a5c740 1 -- 172.21.15.29:0/3438405002 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5556d34ec720 con 0x5556d385d720 2024-03-11T19:20:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.047+0000 7f719bfff640 1 --2- 172.21.15.29:0/3438405002 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5556d385d720 0x5556d385fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.047+0000 7f719bfff640 1 --2- 172.21.15.29:0/3438405002 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5556d385d720 0x5556d385fb10 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c9498915d06affac server_cookie=f33d492eac2fcbd in_seq=0 out_seq=0 2024-03-11T19:20:52.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.047+0000 7f717bfff640 10 client.27562 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:20:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.048+0000 7f717bfff640 1 -- 172.21.15.29:0/3438405002 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5556d34ec720 con 0x5556d385d720 2024-03-11T19:20:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.048+0000 7f717bfff640 10 client.27562 handle_command_reply: tid=0 2024-03-11T19:20:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.048+0000 7f71a5a5c740 1 client.27562 shutdown 2024-03-11T19:20:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.048+0000 7f71a5a5c740 1 -- 172.21.15.29:0/3438405002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d3847480 msgr2=0x5556d38516c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.048+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/3438405002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d3847480 0x5556d38516c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f719001a040 tx=0x7f7190005260 comp rx=0 tx=0).stop 2024-03-11T19:20:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.049+0000 7f71a5a5c740 1 -- 172.21.15.29:0/3438405002 shutdown_connections 2024-03-11T19:20:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.049+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/3438405002 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5556d385d720 0x5556d385fb10 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.049+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/3438405002 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d384cfe0 0x5556d3858bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.049+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/3438405002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5556d3847e70 0x5556d3851c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.049+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/3438405002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d3847480 0x5556d38516c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.049+0000 7f71a5a5c740 1 -- 172.21.15.29:0/3438405002 >> 172.21.15.29:0/3438405002 conn(0x5556d38429f0 msgr2=0x5556d3841cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.049+0000 7f717bfff640 0 client.27562 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:20:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.050+0000 7f71a5a5c740 1 -- 172.21.15.29:0/3438405002 shutdown_connections 2024-03-11T19:20:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.050+0000 7f71a5a5c740 1 -- 172.21.15.29:0/3438405002 wait complete. 2024-03-11T19:20:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.050+0000 7f71a5a5c740 20 client.27562 trim_cache size 0 max 16384 2024-03-11T19:20:52.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.073+0000 7f71a5a5c740 1 Processor -- start 2024-03-11T19:20:52.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.073+0000 7f71a5a5c740 1 -- start start 2024-03-11T19:20:52.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.074+0000 7f71a5a5c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847480 0x5556d3846360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.074+0000 7f71a5a5c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5556d3847e70 0x5556d3858b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.074+0000 7f71a0c29640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847480 0x5556d3846360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.074+0000 7f71a0c29640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847480 0x5556d3846360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49698/0 (socket says 172.21.15.29:49698) 2024-03-11T19:20:52.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.074+0000 7f71a0c29640 1 -- 172.21.15.29:0/2804527754 learned_addr learned my addr 172.21.15.29:0/2804527754 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:52.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.075+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/2804527754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d384cfe0 0x5556d3850ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.075+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2804527754 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556d3735130 con 0x5556d3847e70 2024-03-11T19:20:52.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.075+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2804527754 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556d3848250 con 0x5556d3847480 2024-03-11T19:20:52.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.075+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2804527754 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556d383e4d0 con 0x5556d384cfe0 2024-03-11T19:20:52.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.075+0000 7f71a142a640 1 --2- 172.21.15.29:0/2804527754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d384cfe0 0x5556d3850ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.075+0000 7f719bfff640 1 --2- 172.21.15.29:0/2804527754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5556d3847e70 0x5556d3858b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.076+0000 7f71a142a640 1 -- 172.21.15.29:0/2804527754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847480 msgr2=0x5556d3846360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:52.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.076+0000 7f71a142a640 1 --2- 172.21.15.29:0/2804527754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847480 0x5556d3846360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.076+0000 7f71a142a640 1 -- 172.21.15.29:0/2804527754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5556d3847e70 msgr2=0x5556d3858b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:52.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.076+0000 7f71a142a640 1 --2- 172.21.15.29:0/2804527754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5556d3847e70 0x5556d3858b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.076+0000 7f71a142a640 1 -- 172.21.15.29:0/2804527754 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556d374ac20 con 0x5556d384cfe0 2024-03-11T19:20:52.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.076+0000 7f71a142a640 1 --2- 172.21.15.29:0/2804527754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d384cfe0 0x5556d3850ce0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f719400c760 tx=0x7f7194096db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63e3fa5c890562f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:52.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.076+0000 7f71867fc640 1 -- 172.21.15.29:0/2804527754 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71940a3230 con 0x5556d384cfe0 2024-03-11T19:20:52.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.076+0000 7f71867fc640 1 -- 172.21.15.29:0/2804527754 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f71940cb730 con 0x5556d384cfe0 2024-03-11T19:20:52.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.076+0000 7f71867fc640 1 -- 172.21.15.29:0/2804527754 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71940b34a0 con 0x5556d384cfe0 2024-03-11T19:20:52.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.077+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2804527754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d384cfe0 msgr2=0x5556d3850ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:52.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.077+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/2804527754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d384cfe0 0x5556d3850ce0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f719400c760 tx=0x7f7194096db0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.078+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2804527754 shutdown_connections 2024-03-11T19:20:52.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.078+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/2804527754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d384cfe0 0x5556d3850ce0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.078+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/2804527754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5556d3847e70 0x5556d3858b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.078+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/2804527754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847480 0x5556d3846360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.078+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2804527754 >> 172.21.15.29:0/2804527754 conn(0x5556d38429f0 msgr2=0x5556d3845c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:52.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.078+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2804527754 shutdown_connections 2024-03-11T19:20:52.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.078+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2804527754 wait complete. 2024-03-11T19:20:52.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.078+0000 7f71a5a5c740 1 -- start start 2024-03-11T19:20:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.079+0000 7f71a5a5c740 1 Processor -- start 2024-03-11T19:20:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.079+0000 7f71a5a5c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847480 0x5556d3857fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.079+0000 7f71a5a5c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d3847e70 0x5556d3858510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.079+0000 7f71a5a5c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5556d384cfe0 0x5556d38b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.079+0000 7f71a0c29640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847480 0x5556d3857fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.079+0000 7f71a0c29640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847480 0x5556d3857fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49712/0 (socket says 172.21.15.29:49712) 2024-03-11T19:20:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.079+0000 7f719bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d3847e70 0x5556d3858510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.080+0000 7f719bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d3847e70 0x5556d3858510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:50756/0 (socket says 172.21.15.29:50756) 2024-03-11T19:20:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.080+0000 7f71a0c29640 1 -- 172.21.15.29:0/2179620864 learned_addr learned my addr 172.21.15.29:0/2179620864 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.080+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2179620864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556d38b4530 con 0x5556d384cfe0 2024-03-11T19:20:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.080+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2179620864 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556d38b4670 con 0x5556d3847480 2024-03-11T19:20:52.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.080+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2179620864 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556d38b4800 con 0x5556d3847e70 2024-03-11T19:20:52.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.080+0000 7f71a142a640 1 --2- 172.21.15.29:0/2179620864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5556d384cfe0 0x5556d38b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.080+0000 7f71a5a5c740 10 client.0 _ll_register_callbacks cb 0x7f719000a790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:52.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.080+0000 7f71a0c29640 1 -- 172.21.15.29:0/2179620864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d3847e70 msgr2=0x5556d3858510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:52.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.080+0000 7f71a0c29640 1 --2- 172.21.15.29:0/2179620864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d3847e70 0x5556d3858510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.080+0000 7f71a0c29640 1 -- 172.21.15.29:0/2179620864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5556d384cfe0 msgr2=0x5556d38b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:52.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.080+0000 7f71a0c29640 1 --2- 172.21.15.29:0/2179620864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5556d384cfe0 0x5556d38b3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.080+0000 7f71a0c29640 1 -- 172.21.15.29:0/2179620864 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719c1a5aa0 con 0x5556d3847480 2024-03-11T19:20:52.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.080+0000 7f71a0c29640 1 --2- 172.21.15.29:0/2179620864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847480 0x5556d3857fd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f719001a040 tx=0x7f719000ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=360e34b1e5130c33 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:52.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.080+0000 7f717bfff640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:20:52.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.081+0000 7f717bfff640 1 -- 172.21.15.29:0/2179620864 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7190004d60 con 0x5556d3847480 2024-03-11T19:20:52.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.081+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2179620864 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f719000ae50 con 0x5556d3847480 2024-03-11T19:20:52.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.081+0000 7f717bfff640 1 -- 172.21.15.29:0/2179620864 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7190000e40 con 0x5556d3847480 2024-03-11T19:20:52.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.081+0000 7f717bfff640 1 -- 172.21.15.29:0/2179620864 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7190004ec0 con 0x5556d3847480 2024-03-11T19:20:52.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.081+0000 7f71a5a5c740 10 client.27527 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:52.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.081+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2179620864 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5556d37708c0 con 0x5556d3847480 2024-03-11T19:20:52.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.081+0000 7f717bfff640 1 -- 172.21.15.29:0/2179620864 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f71900188f0 con 0x5556d3847480 2024-03-11T19:20:52.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.081+0000 7f717bfff640 1 -- 172.21.15.29:0/2179620864 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7190020030 con 0x5556d3847480 2024-03-11T19:20:52.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.081+0000 7f71a5a5c740 10 client.27527 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:52.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.081+0000 7f71a5a5c740 10 client.27527 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:20:52.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.081+0000 7f71a5a5c740 20 client.27527 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:52.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.081+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/2179620864 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5556d38b5e00 0x5556d38b8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.082+0000 7f71a5a5c740 4 client.27527 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:20:52.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.082+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2179620864 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f719000ae50 con 0x5556d38b5e00 2024-03-11T19:20:52.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.082+0000 7f719bfff640 1 --2- 172.21.15.29:0/2179620864 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5556d38b5e00 0x5556d38b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.082+0000 7f719bfff640 1 --2- 172.21.15.29:0/2179620864 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5556d38b5e00 0x5556d38b8220 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=13e0f16ada8fdff9 server_cookie=eb9c53f11a285a7a in_seq=0 out_seq=0 2024-03-11T19:20:52.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.082+0000 7f717bfff640 10 client.27527 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:20:52.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.082+0000 7f717bfff640 1 -- 172.21.15.29:0/2179620864 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f719000ae50 con 0x5556d38b5e00 2024-03-11T19:20:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.082+0000 7f717bfff640 10 client.27527 handle_command_reply: tid=0 2024-03-11T19:20:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.083+0000 7f71a5a5c740 1 client.27527 shutdown 2024-03-11T19:20:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.083+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2179620864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847480 msgr2=0x5556d3857fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.083+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/2179620864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847480 0x5556d3857fd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f719001a040 tx=0x7f719000ac20 comp rx=0 tx=0).stop 2024-03-11T19:20:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.083+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2179620864 shutdown_connections 2024-03-11T19:20:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.083+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/2179620864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5556d384cfe0 0x5556d38b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.083+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/2179620864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556d3847e70 0x5556d3858510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.083+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/2179620864 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x5556d38b5e00 0x5556d38b8220 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.083+0000 7f71a5a5c740 1 --2- 172.21.15.29:0/2179620864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5556d3847480 0x5556d3857fd0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.083+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2179620864 >> 172.21.15.29:0/2179620864 conn(0x5556d38429f0 msgr2=0x5556d38464e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.083+0000 7f717bfff640 0 client.27527 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:20:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.083+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2179620864 shutdown_connections 2024-03-11T19:20:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.083+0000 7f71a5a5c740 1 -- 172.21.15.29:0/2179620864 wait complete. 2024-03-11T19:20:52.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.083+0000 7f71a5a5c740 20 client.27527 trim_cache size 0 max 16384 2024-03-11T19:20:52.091 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:20:52.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.084+0000 7f71a1c2b640 1 -- 172.21.15.29:0/1535856154 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f718c0a1b30 msgr2=0x7f718c0a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:52.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.084+0000 7f71a1c2b640 1 --2- 172.21.15.29:0/1535856154 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f718c0a1b30 0x7f718c0a3ff0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7188006010 tx=0x7f7188000bc0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.084+0000 7f71a1c2b640 1 -- 172.21.15.29:0/1535856154 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f719c109170 msgr2=0x7f719c1a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:52.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.084+0000 7f71a1c2b640 1 --2- 172.21.15.29:0/1535856154 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f719c109170 0x7f719c1a4a70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7194009fd0 tx=0x7f719400ef10 comp rx=0 tx=0).stop 2024-03-11T19:20:52.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.084+0000 7f71a1c2b640 1 -- 172.21.15.29:0/1535856154 shutdown_connections 2024-03-11T19:20:52.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.084+0000 7f71a1c2b640 1 --2- 172.21.15.29:0/1535856154 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f719c109170 0x7f719c1a4a70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.084+0000 7f71a1c2b640 1 --2- 172.21.15.29:0/1535856154 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f718c0a1b30 0x7f718c0a3ff0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.084+0000 7f71a1c2b640 1 --2- 172.21.15.29:0/1535856154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f719c1087b0 0x7f719c19da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.084+0000 7f71a1c2b640 1 --2- 172.21.15.29:0/1535856154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f719c1075c0 0x7f719c19d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.084+0000 7f71a1c2b640 1 -- 172.21.15.29:0/1535856154 >> 172.21.15.29:0/1535856154 conn(0x7f719c076580 msgr2=0x7f719c10bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:52.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.085+0000 7f71a1c2b640 1 -- 172.21.15.29:0/1535856154 shutdown_connections 2024-03-11T19:20:52.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.085+0000 7f71a1c2b640 1 -- 172.21.15.29:0/1535856154 wait complete. 2024-03-11T19:20:52.107 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:20:52.108 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:20:52.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.275+0000 7fbceca3b640 1 -- 172.21.15.29:0/894035387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbce81077a0 msgr2=0x7fbce8107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:52.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.275+0000 7fbceca3b640 1 --2- 172.21.15.29:0/894035387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbce81077a0 0x7fbce8107bc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8009900 tx=0x7fbcd802bc30 comp rx=0 tx=0).stop 2024-03-11T19:20:52.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.275+0000 7fbceca3b640 1 -- 172.21.15.29:0/894035387 shutdown_connections 2024-03-11T19:20:52.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.275+0000 7fbceca3b640 1 --2- 172.21.15.29:0/894035387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce8109350 0x7fbce810fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.275+0000 7fbceca3b640 1 --2- 172.21.15.29:0/894035387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbce8108990 0x7fbce8108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.275+0000 7fbceca3b640 1 --2- 172.21.15.29:0/894035387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbce81077a0 0x7fbce8107bc0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.275+0000 7fbceca3b640 1 -- 172.21.15.29:0/894035387 >> 172.21.15.29:0/894035387 conn(0x7fbce8076740 msgr2=0x7fbce8078b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:52.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.275+0000 7fbceca3b640 1 -- 172.21.15.29:0/894035387 shutdown_connections 2024-03-11T19:20:52.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.275+0000 7fbceca3b640 1 -- 172.21.15.29:0/894035387 wait complete. 2024-03-11T19:20:52.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.276+0000 7fbceca3b640 1 Processor -- start 2024-03-11T19:20:52.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.276+0000 7fbceca3b640 1 -- start start 2024-03-11T19:20:52.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.276+0000 7fbceca3b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbce81077a0 0x7fbce819d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.277+0000 7fbceca3b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbce8108990 0x7fbce819dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.277+0000 7fbceca3b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce8109350 0x7fbce81a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.277+0000 7fbce659c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbce81077a0 0x7fbce819d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.277+0000 7fbce659c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbce81077a0 0x7fbce819d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49730/0 (socket says 172.21.15.29:49730) 2024-03-11T19:20:52.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.277+0000 7fbce659c640 1 -- 172.21.15.29:0/245380699 learned_addr learned my addr 172.21.15.29:0/245380699 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:52.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.277+0000 7fbceca3b640 1 -- 172.21.15.29:0/245380699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce81a5280 con 0x7fbce8109350 2024-03-11T19:20:52.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.277+0000 7fbceca3b640 1 -- 172.21.15.29:0/245380699 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce81a53f0 con 0x7fbce81077a0 2024-03-11T19:20:52.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.277+0000 7fbceca3b640 1 -- 172.21.15.29:0/245380699 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce81a55b0 con 0x7fbce8108990 2024-03-11T19:20:52.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.277+0000 7fbcdffff640 1 --2- 172.21.15.29:0/245380699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbce8108990 0x7fbce819dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.277+0000 7fbce6d9d640 1 --2- 172.21.15.29:0/245380699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce8109350 0x7fbce81a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.277+0000 7fbce659c640 1 -- 172.21.15.29:0/245380699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbce8108990 msgr2=0x7fbce819dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:52.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.277+0000 7fbce659c640 1 --2- 172.21.15.29:0/245380699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbce8108990 0x7fbce819dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.277+0000 7fbce659c640 1 -- 172.21.15.29:0/245380699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce8109350 msgr2=0x7fbce81a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:52.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.277+0000 7fbce659c640 1 --2- 172.21.15.29:0/245380699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce8109350 0x7fbce81a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.277+0000 7fbce659c640 1 -- 172.21.15.29:0/245380699 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd8009560 con 0x7fbce81077a0 2024-03-11T19:20:52.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.277+0000 7fbce6d9d640 1 --2- 172.21.15.29:0/245380699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce8109350 0x7fbce81a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:20:52.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.278+0000 7fbce659c640 1 --2- 172.21.15.29:0/245380699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbce81077a0 0x7fbce819d790 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8000c00 tx=0x7fbcd8004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:52.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.278+0000 7fbcdf7fe640 1 -- 172.21.15.29:0/245380699 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8004500 con 0x7fbce81077a0 2024-03-11T19:20:52.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.278+0000 7fbcdf7fe640 1 -- 172.21.15.29:0/245380699 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbcd8003aa0 con 0x7fbce81077a0 2024-03-11T19:20:52.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.278+0000 7fbcdf7fe640 1 -- 172.21.15.29:0/245380699 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8005430 con 0x7fbce81077a0 2024-03-11T19:20:52.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.278+0000 7fbceca3b640 1 -- 172.21.15.29:0/245380699 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbce81a5830 con 0x7fbce81077a0 2024-03-11T19:20:52.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.278+0000 7fbceca3b640 1 -- 172.21.15.29:0/245380699 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce81a5cf0 con 0x7fbce81077a0 2024-03-11T19:20:52.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.281+0000 7fbcdf7fe640 1 -- 172.21.15.29:0/245380699 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fbcd8003c10 con 0x7fbce81077a0 2024-03-11T19:20:52.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.282+0000 7fbcef605740 1 Processor -- start 2024-03-11T19:20:52.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.282+0000 7fbcef605740 1 -- start start 2024-03-11T19:20:52.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.282+0000 7fbcef605740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 0x5566b5e79690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.282+0000 7fbcef605740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e79c60 0x5566b5e7e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.282+0000 7fbcef605740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e7edd0 0x5566b5e81260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.282+0000 7fbcef605740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b5e35960 con 0x5566b5e7edd0 2024-03-11T19:20:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.282+0000 7fbcef605740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b5de9310 con 0x5566b5e79270 2024-03-11T19:20:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.282+0000 7fbcef605740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b5de6840 con 0x5566b5e79c60 2024-03-11T19:20:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.283+0000 7fbce659c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e79c60 0x5566b5e7e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.283+0000 7fbce6d9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 0x5566b5e79690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.283+0000 7fbce6d9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 0x5566b5e79690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49744/0 (socket says 172.21.15.29:49744) 2024-03-11T19:20:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.283+0000 7fbce6d9d640 1 -- 172.21.15.29:0/882802478 learned_addr learned my addr 172.21.15.29:0/882802478 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.283+0000 7fbce659c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e79c60 0x5566b5e7e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:50790/0 (socket says 172.21.15.29:50790) 2024-03-11T19:20:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.283+0000 7fbcdffff640 1 --2- 172.21.15.29:0/882802478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e7edd0 0x5566b5e81260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.283+0000 7fbce659c640 1 -- 172.21.15.29:0/882802478 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 msgr2=0x5566b5e79690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:52.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.283+0000 7fbce659c640 1 --2- 172.21.15.29:0/882802478 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 0x5566b5e79690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.283+0000 7fbce659c640 1 -- 172.21.15.29:0/882802478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e7edd0 msgr2=0x5566b5e81260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:52.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.283+0000 7fbce659c640 1 --2- 172.21.15.29:0/882802478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e7edd0 0x5566b5e81260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.283+0000 7fbce659c640 1 -- 172.21.15.29:0/882802478 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566b5da4060 con 0x5566b5e79c60 2024-03-11T19:20:52.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.283+0000 7fbcdf7fe640 1 --2- 172.21.15.29:0/245380699 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbcd40a1ab0 0x7fbcd40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.284+0000 7fbcdf7fe640 1 -- 172.21.15.29:0/245380699 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fbcd802de40 con 0x7fbce81077a0 2024-03-11T19:20:52.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.284+0000 7fbce659c640 1 --2- 172.21.15.29:0/882802478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e79c60 0x5566b5e7e760 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80ed770 tx=0x7fbcd80f5b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a78287edb74c8c2a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:52.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.284+0000 7fbcdcff9640 1 -- 172.21.15.29:0/882802478 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd80c0020 con 0x5566b5e79c60 2024-03-11T19:20:52.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.284+0000 7fbcdcff9640 1 -- 172.21.15.29:0/882802478 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbcd80f56d0 con 0x5566b5e79c60 2024-03-11T19:20:52.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.284+0000 7fbcdcff9640 1 -- 172.21.15.29:0/882802478 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd80bc6c0 con 0x5566b5e79c60 2024-03-11T19:20:52.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.284+0000 7fbce6d9d640 1 --2- 172.21.15.29:0/245380699 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbcd40a1ab0 0x7fbcd40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.285+0000 7fbcef605740 1 -- 172.21.15.29:0/882802478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e79c60 msgr2=0x5566b5e7e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:52.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.285+0000 7fbcef605740 1 --2- 172.21.15.29:0/882802478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e79c60 0x5566b5e7e760 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80ed770 tx=0x7fbcd80f5b20 comp rx=0 tx=0).stop 2024-03-11T19:20:52.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.285+0000 7fbce6d9d640 1 --2- 172.21.15.29:0/245380699 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbcd40a1ab0 0x7fbcd40a3f70 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x5566b5e7a4c0 tx=0x7fbce0009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:52.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.285+0000 7fbcef605740 1 -- 172.21.15.29:0/882802478 shutdown_connections 2024-03-11T19:20:52.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.285+0000 7fbcef605740 1 --2- 172.21.15.29:0/882802478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e7edd0 0x5566b5e81260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.285+0000 7fbcef605740 1 --2- 172.21.15.29:0/882802478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e79c60 0x5566b5e7e760 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.285+0000 7fbcef605740 1 --2- 172.21.15.29:0/882802478 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 0x5566b5e79690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.285+0000 7fbcef605740 1 -- 172.21.15.29:0/882802478 >> 172.21.15.29:0/882802478 conn(0x5566b5e747e0 msgr2=0x5566b5e74be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:52.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.286+0000 7fbcef605740 1 -- 172.21.15.29:0/882802478 shutdown_connections 2024-03-11T19:20:52.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.286+0000 7fbcef605740 1 -- 172.21.15.29:0/882802478 wait complete. 2024-03-11T19:20:52.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.286+0000 7fbcef605740 1 -- start start 2024-03-11T19:20:52.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.287+0000 7fbcef605740 1 Processor -- start 2024-03-11T19:20:52.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.287+0000 7fbcef605740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 0x5566b5e94250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.287+0000 7fbcef605740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e79c60 0x5566b5e94790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.287+0000 7fbcef605740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e7edd0 0x5566b5e9b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.287+0000 7fbce659c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 0x5566b5e94250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.287+0000 7fbce659c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 0x5566b5e94250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49752/0 (socket says 172.21.15.29:49752) 2024-03-11T19:20:52.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.287+0000 7fbce659c640 1 -- 172.21.15.29:0/1051588168 learned_addr learned my addr 172.21.15.29:0/1051588168 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:52.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.288+0000 7fbcdffff640 1 --2- 172.21.15.29:0/1051588168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e79c60 0x5566b5e94790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.288+0000 7fbce6d9d640 1 --2- 172.21.15.29:0/1051588168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e7edd0 0x5566b5e9b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.288+0000 7fbcef605740 1 -- 172.21.15.29:0/1051588168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b5de2020 con 0x5566b5e79c60 2024-03-11T19:20:52.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.288+0000 7fbcef605740 1 -- 172.21.15.29:0/1051588168 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b5ab5430 con 0x5566b5e79270 2024-03-11T19:20:52.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.288+0000 7fbcef605740 1 -- 172.21.15.29:0/1051588168 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b5a71e00 con 0x5566b5e7edd0 2024-03-11T19:20:52.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.288+0000 7fbcef605740 10 client.0 _ll_register_callbacks cb 0x5566b5d89cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:52.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.288+0000 7fbce659c640 1 -- 172.21.15.29:0/1051588168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e7edd0 msgr2=0x5566b5e9b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:52.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.288+0000 7fbce659c640 1 --2- 172.21.15.29:0/1051588168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e7edd0 0x5566b5e9b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.288+0000 7fbce659c640 1 -- 172.21.15.29:0/1051588168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e79c60 msgr2=0x5566b5e94790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:52.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.288+0000 7fbce659c640 1 --2- 172.21.15.29:0/1051588168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e79c60 0x5566b5e94790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.288+0000 7fbce659c640 1 -- 172.21.15.29:0/1051588168 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce81a5cf0 con 0x5566b5e79270 2024-03-11T19:20:52.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.289+0000 7fbce659c640 1 --2- 172.21.15.29:0/1051588168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 0x5566b5e94250 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80ed7a0 tx=0x7fbcd8005bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=841461d31bf59304 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:52.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.289+0000 7fbccdffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:20:52.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.289+0000 7fbccdffb640 1 -- 172.21.15.29:0/1051588168 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd80c0070 con 0x5566b5e79270 2024-03-11T19:20:52.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.289+0000 7fbcef605740 1 -- 172.21.15.29:0/1051588168 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5566b5b1e720 con 0x5566b5e79270 2024-03-11T19:20:52.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.289+0000 7fbccdffb640 1 -- 172.21.15.29:0/1051588168 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fbcd8004dc0 con 0x5566b5e79270 2024-03-11T19:20:52.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.289+0000 7fbccdffb640 1 -- 172.21.15.29:0/1051588168 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbcd80b9000 con 0x5566b5e79270 2024-03-11T19:20:52.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.289+0000 7fbcef605740 10 client.27539 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:52.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.289+0000 7fbcef605740 1 -- 172.21.15.29:0/1051588168 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5566b5de06a0 con 0x5566b5e79270 2024-03-11T19:20:52.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.289+0000 7fbccdffb640 1 -- 172.21.15.29:0/1051588168 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbcd8086080 con 0x5566b5e79270 2024-03-11T19:20:52.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.290+0000 7fbccdffb640 1 -- 172.21.15.29:0/1051588168 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbcd803c070 con 0x5566b5e79270 2024-03-11T19:20:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.290+0000 7fbcef605740 10 client.27539 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.290+0000 7fbcef605740 10 client.27539 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:20:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.290+0000 7fbcef605740 20 client.27539 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.290+0000 7fbcef605740 1 --2- 172.21.15.29:0/1051588168 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5566b5e9fe40 0x5566b5ea2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.290+0000 7fbcef605740 4 client.27539 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:20:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.290+0000 7fbcef605740 1 -- 172.21.15.29:0/1051588168 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5566b5b6d890 con 0x5566b5e9fe40 2024-03-11T19:20:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.290+0000 7fbcdffff640 1 --2- 172.21.15.29:0/1051588168 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5566b5e9fe40 0x5566b5ea2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.290+0000 7fbcdffff640 1 --2- 172.21.15.29:0/1051588168 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5566b5e9fe40 0x5566b5ea2230 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e3c66b8a608ca55 server_cookie=cb89cd5759ec8310 in_seq=0 out_seq=0 2024-03-11T19:20:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.291+0000 7fbccdffb640 10 client.27539 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.291+0000 7fbccdffb640 1 -- 172.21.15.29:0/1051588168 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5566b5b6d890 con 0x5566b5e9fe40 2024-03-11T19:20:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.291+0000 7fbccdffb640 10 client.27539 handle_command_reply: tid=0 2024-03-11T19:20:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.291+0000 7fbcef605740 1 client.27539 shutdown 2024-03-11T19:20:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.292+0000 7fbcef605740 1 -- 172.21.15.29:0/1051588168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 msgr2=0x5566b5e94250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.292+0000 7fbcef605740 1 --2- 172.21.15.29:0/1051588168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 0x5566b5e94250 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80ed7a0 tx=0x7fbcd8005bc0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.292+0000 7fbcef605740 1 -- 172.21.15.29:0/1051588168 shutdown_connections 2024-03-11T19:20:52.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.292+0000 7fbcef605740 1 --2- 172.21.15.29:0/1051588168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e7edd0 0x5566b5e9b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.292+0000 7fbcef605740 1 --2- 172.21.15.29:0/1051588168 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5566b5e9fe40 0x5566b5ea2230 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.292+0000 7fbccdffb640 0 client.27539 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:52.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.292+0000 7fbcef605740 1 --2- 172.21.15.29:0/1051588168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e79c60 0x5566b5e94790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.292+0000 7fbcef605740 1 --2- 172.21.15.29:0/1051588168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 0x5566b5e94250 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.292+0000 7fbcef605740 1 -- 172.21.15.29:0/1051588168 >> 172.21.15.29:0/1051588168 conn(0x5566b5e747e0 msgr2=0x5566b5e73b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:52.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.292+0000 7fbcef605740 1 -- 172.21.15.29:0/1051588168 shutdown_connections 2024-03-11T19:20:52.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.292+0000 7fbcef605740 1 -- 172.21.15.29:0/1051588168 wait complete. 2024-03-11T19:20:52.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.292+0000 7fbcef605740 20 client.27539 trim_cache size 0 max 16384 2024-03-11T19:20:52.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.313+0000 7fbcef605740 1 Processor -- start 2024-03-11T19:20:52.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.313+0000 7fbcef605740 1 -- start start 2024-03-11T19:20:52.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.313+0000 7fbcef605740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e79270 0x5566b5e94570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.313+0000 7fbcef605740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e79c60 0x5566b5e9b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.314+0000 7fbcef605740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e7edd0 0x5566b5e732f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.314+0000 7fbcef605740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b5d67130 con 0x5566b5e79270 2024-03-11T19:20:52.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.314+0000 7fbcef605740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b5e933e0 con 0x5566b5e7edd0 2024-03-11T19:20:52.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.314+0000 7fbcef605740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b5b6d890 con 0x5566b5e79c60 2024-03-11T19:20:52.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.314+0000 7fbce659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e79270 0x5566b5e94570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.314+0000 7fbcdffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e79c60 0x5566b5e9b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.314+0000 7fbce659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e79270 0x5566b5e94570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48742/0 (socket says 172.21.15.29:48742) 2024-03-11T19:20:52.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.314+0000 7fbce659c640 1 -- 172.21.15.29:0/2351815900 learned_addr learned my addr 172.21.15.29:0/2351815900 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:52.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.314+0000 7fbce6d9d640 1 --2- 172.21.15.29:0/2351815900 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e7edd0 0x5566b5e732f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.314+0000 7fbcdffff640 1 -- 172.21.15.29:0/2351815900 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e7edd0 msgr2=0x5566b5e732f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:52.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.314+0000 7fbcdffff640 1 --2- 172.21.15.29:0/2351815900 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e7edd0 0x5566b5e732f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.314+0000 7fbcdffff640 1 -- 172.21.15.29:0/2351815900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e79270 msgr2=0x5566b5e94570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:52.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.314+0000 7fbcdffff640 1 --2- 172.21.15.29:0/2351815900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e79270 0x5566b5e94570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.314+0000 7fbcdffff640 1 -- 172.21.15.29:0/2351815900 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566b5d7cc20 con 0x5566b5e79c60 2024-03-11T19:20:52.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.314+0000 7fbce659c640 1 --2- 172.21.15.29:0/2351815900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e79270 0x5566b5e94570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:20:52.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.314+0000 7fbcdffff640 1 --2- 172.21.15.29:0/2351815900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e79c60 0x5566b5e9b760 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x5566b5ea2b30 tx=0x7fbcd0005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77bd780511310383 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:52.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.315+0000 7fbccffff640 1 -- 172.21.15.29:0/2351815900 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0017030 con 0x5566b5e79c60 2024-03-11T19:20:52.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.315+0000 7fbccffff640 1 -- 172.21.15.29:0/2351815900 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbcd0018040 con 0x5566b5e79c60 2024-03-11T19:20:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.315+0000 7fbccffff640 1 -- 172.21.15.29:0/2351815900 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0010a70 con 0x5566b5e79c60 2024-03-11T19:20:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.316+0000 7fbcef605740 1 -- 172.21.15.29:0/2351815900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e79c60 msgr2=0x5566b5e9b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.316+0000 7fbcef605740 1 --2- 172.21.15.29:0/2351815900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e79c60 0x5566b5e9b760 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x5566b5ea2b30 tx=0x7fbcd0005f90 comp rx=0 tx=0).stop 2024-03-11T19:20:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.316+0000 7fbcef605740 1 -- 172.21.15.29:0/2351815900 shutdown_connections 2024-03-11T19:20:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.316+0000 7fbcef605740 1 --2- 172.21.15.29:0/2351815900 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e7edd0 0x5566b5e732f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.316+0000 7fbcef605740 1 --2- 172.21.15.29:0/2351815900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e79c60 0x5566b5e9b760 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.316+0000 7fbcef605740 1 --2- 172.21.15.29:0/2351815900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e79270 0x5566b5e94570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.316+0000 7fbcef605740 1 -- 172.21.15.29:0/2351815900 >> 172.21.15.29:0/2351815900 conn(0x5566b5e747e0 msgr2=0x5566b5e93e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.317+0000 7fbcef605740 1 -- 172.21.15.29:0/2351815900 shutdown_connections 2024-03-11T19:20:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.317+0000 7fbcef605740 1 -- 172.21.15.29:0/2351815900 wait complete. 2024-03-11T19:20:52.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.318+0000 7fbcef605740 1 -- start start 2024-03-11T19:20:52.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.318+0000 7fbcef605740 1 Processor -- start 2024-03-11T19:20:52.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.318+0000 7fbcef605740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 0x5566b5e9a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.318+0000 7fbcef605740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e79c60 0x5566b5e9a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.318+0000 7fbcef605740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e7edd0 0x5566b5e9ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.319+0000 7fbce659c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 0x5566b5e9a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.319+0000 7fbcef605740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b5ef6440 con 0x5566b5e79c60 2024-03-11T19:20:52.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.319+0000 7fbcef605740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b5ef65b0 con 0x5566b5e79270 2024-03-11T19:20:52.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.319+0000 7fbcef605740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b5ef6770 con 0x5566b5e7edd0 2024-03-11T19:20:52.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.319+0000 7fbcdffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e79c60 0x5566b5e9a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.319+0000 7fbce659c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 0x5566b5e9a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49760/0 (socket says 172.21.15.29:49760) 2024-03-11T19:20:52.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.319+0000 7fbce659c640 1 -- 172.21.15.29:0/3379419413 learned_addr learned my addr 172.21.15.29:0/3379419413 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:52.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.319+0000 7fbcef605740 10 client.0 _ll_register_callbacks cb 0x7fbcd80f5400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:52.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.319+0000 7fbce6d9d640 1 --2- 172.21.15.29:0/3379419413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e7edd0 0x5566b5e9ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.319+0000 7fbce659c640 1 -- 172.21.15.29:0/3379419413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e7edd0 msgr2=0x5566b5e9ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.319+0000 7fbce659c640 1 --2- 172.21.15.29:0/3379419413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e7edd0 0x5566b5e9ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.319+0000 7fbce659c640 1 -- 172.21.15.29:0/3379419413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e79c60 msgr2=0x5566b5e9a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.319+0000 7fbce659c640 1 --2- 172.21.15.29:0/3379419413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e79c60 0x5566b5e9a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.319+0000 7fbce659c640 1 -- 172.21.15.29:0/3379419413 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5566b5de06a0 con 0x5566b5e79270 2024-03-11T19:20:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.319+0000 7fbcdffff640 1 --2- 172.21.15.29:0/3379419413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e79c60 0x5566b5e9a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:20:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.319+0000 7fbce659c640 1 --2- 172.21.15.29:0/3379419413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 0x5566b5e9a290 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80bccf0 tx=0x7fbcd8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73b401438eafa6f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.319+0000 7fbccdffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:20:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.320+0000 7fbccdffb640 1 -- 172.21.15.29:0/3379419413 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd80c0070 con 0x5566b5e79270 2024-03-11T19:20:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.320+0000 7fbccdffb640 1 -- 172.21.15.29:0/3379419413 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fbcd8004dc0 con 0x5566b5e79270 2024-03-11T19:20:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.320+0000 7fbcef605740 1 -- 172.21.15.29:0/3379419413 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5566b5ae17e0 con 0x5566b5e79270 2024-03-11T19:20:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.320+0000 7fbccdffb640 1 -- 172.21.15.29:0/3379419413 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbcd80f58e0 con 0x5566b5e79270 2024-03-11T19:20:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.320+0000 7fbcef605740 10 client.27545 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.320+0000 7fbcef605740 1 -- 172.21.15.29:0/3379419413 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7fbcd0009a50 con 0x5566b5e79270 2024-03-11T19:20:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.321+0000 7fbccdffb640 1 -- 172.21.15.29:0/3379419413 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5566b5ae17e0 con 0x5566b5e79270 2024-03-11T19:20:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.321+0000 7fbccdffb640 1 -- 172.21.15.29:0/3379419413 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbcd803c070 con 0x5566b5e79270 2024-03-11T19:20:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.321+0000 7fbcef605740 10 client.27545 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.321+0000 7fbcef605740 10 client.27545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:20:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.321+0000 7fbcef605740 20 client.27545 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.321+0000 7fbcef605740 1 --2- 172.21.15.29:0/3379419413 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5566b5e9fe40 0x5566b5ef9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.321+0000 7fbcef605740 4 client.27545 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:20:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.321+0000 7fbcef605740 1 -- 172.21.15.29:0/3379419413 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5566b5a8d700 con 0x5566b5e9fe40 2024-03-11T19:20:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.321+0000 7fbcdffff640 1 --2- 172.21.15.29:0/3379419413 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5566b5e9fe40 0x5566b5ef9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.322+0000 7fbccdffb640 10 client.27545 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.322+0000 7fbcdffff640 1 --2- 172.21.15.29:0/3379419413 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5566b5e9fe40 0x5566b5ef9de0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec3469e119588276 server_cookie=b4b425476b2923b7 in_seq=0 out_seq=0 2024-03-11T19:20:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.322+0000 7fbccdffb640 1 -- 172.21.15.29:0/3379419413 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5566b5a8d700 con 0x5566b5e9fe40 2024-03-11T19:20:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.322+0000 7fbccdffb640 10 client.27545 handle_command_reply: tid=0 2024-03-11T19:20:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.322+0000 7fbcef605740 1 client.27545 shutdown 2024-03-11T19:20:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.322+0000 7fbcef605740 1 -- 172.21.15.29:0/3379419413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 msgr2=0x5566b5e9a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.322+0000 7fbcef605740 1 --2- 172.21.15.29:0/3379419413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 0x5566b5e9a290 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80bccf0 tx=0x7fbcd8009260 comp rx=0 tx=0).stop 2024-03-11T19:20:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.322+0000 7fbcef605740 1 -- 172.21.15.29:0/3379419413 shutdown_connections 2024-03-11T19:20:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.322+0000 7fbcef605740 1 --2- 172.21.15.29:0/3379419413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5566b5e7edd0 0x5566b5e9ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.322+0000 7fbcef605740 1 --2- 172.21.15.29:0/3379419413 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5566b5e9fe40 0x5566b5ef9de0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.322+0000 7fbcef605740 1 --2- 172.21.15.29:0/3379419413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5566b5e79c60 0x5566b5e9a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.322+0000 7fbcef605740 1 --2- 172.21.15.29:0/3379419413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5566b5e79270 0x5566b5e9a290 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.322+0000 7fbcef605740 1 -- 172.21.15.29:0/3379419413 >> 172.21.15.29:0/3379419413 conn(0x5566b5e747e0 msgr2=0x5566b5e77a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.322+0000 7fbccdffb640 0 client.27545 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.322+0000 7fbcef605740 1 -- 172.21.15.29:0/3379419413 shutdown_connections 2024-03-11T19:20:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.322+0000 7fbcef605740 1 -- 172.21.15.29:0/3379419413 wait complete. 2024-03-11T19:20:52.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.322+0000 7fbcef605740 20 client.27545 trim_cache size 0 max 16384 2024-03-11T19:20:52.329 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:20:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.323+0000 7fbceca3b640 1 -- 172.21.15.29:0/245380699 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbcd40a1ab0 msgr2=0x7fbcd40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.323+0000 7fbceca3b640 1 --2- 172.21.15.29:0/245380699 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbcd40a1ab0 0x7fbcd40a3f70 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x5566b5e7a4c0 tx=0x7fbce0009500 comp rx=0 tx=0).stop 2024-03-11T19:20:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.323+0000 7fbceca3b640 1 -- 172.21.15.29:0/245380699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbce81077a0 msgr2=0x7fbce819d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.323+0000 7fbceca3b640 1 --2- 172.21.15.29:0/245380699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbce81077a0 0x7fbce819d790 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8000c00 tx=0x7fbcd8004040 comp rx=0 tx=0).stop 2024-03-11T19:20:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.323+0000 7fbceca3b640 1 -- 172.21.15.29:0/245380699 shutdown_connections 2024-03-11T19:20:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.323+0000 7fbceca3b640 1 --2- 172.21.15.29:0/245380699 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbcd40a1ab0 0x7fbcd40a3f70 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.323+0000 7fbceca3b640 1 --2- 172.21.15.29:0/245380699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce8109350 0x7fbce81a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.323+0000 7fbceca3b640 1 --2- 172.21.15.29:0/245380699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbce8108990 0x7fbce819dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.323+0000 7fbceca3b640 1 --2- 172.21.15.29:0/245380699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbce81077a0 0x7fbce819d790 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.323+0000 7fbceca3b640 1 -- 172.21.15.29:0/245380699 >> 172.21.15.29:0/245380699 conn(0x7fbce8076740 msgr2=0x7fbce810bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.323+0000 7fbceca3b640 1 -- 172.21.15.29:0/245380699 shutdown_connections 2024-03-11T19:20:52.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:52.323+0000 7fbceca3b640 1 -- 172.21.15.29:0/245380699 wait complete. 2024-03-11T19:20:53.340 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-11T19:20:53.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:20:53.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-11T19:20:53.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.512+0000 7f1e7949d640 1 -- 172.21.15.29:0/3373677729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e7406d610 msgr2=0x7f1e7406f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:53.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.512+0000 7f1e7949d640 1 --2- 172.21.15.29:0/3373677729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e7406d610 0x7f1e7406f490 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c009ca0 tx=0x7f1e6c02c100 comp rx=0 tx=0).stop 2024-03-11T19:20:53.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.513+0000 7f1e7949d640 1 -- 172.21.15.29:0/3373677729 shutdown_connections 2024-03-11T19:20:53.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.513+0000 7f1e7949d640 1 --2- 172.21.15.29:0/3373677729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e7406d610 0x7f1e7406f490 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.513+0000 7f1e7949d640 1 --2- 172.21.15.29:0/3373677729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e7406cc70 0x7f1e7406d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.513+0000 7f1e7949d640 1 --2- 172.21.15.29:0/3373677729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e7406e6f0 0x7f1e7406eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.513+0000 7f1e7949d640 1 -- 172.21.15.29:0/3373677729 >> 172.21.15.29:0/3373677729 conn(0x7f1e7406bb80 msgr2=0x7f1e7406bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:53.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.513+0000 7f1e7949d640 1 -- 172.21.15.29:0/3373677729 shutdown_connections 2024-03-11T19:20:53.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.513+0000 7f1e7949d640 1 -- 172.21.15.29:0/3373677729 wait complete. 2024-03-11T19:20:53.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.514+0000 7f1e7949d640 1 Processor -- start 2024-03-11T19:20:53.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.514+0000 7f1e7949d640 1 -- start start 2024-03-11T19:20:53.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.514+0000 7f1e7949d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e7406cc70 0x7f1e741a6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:53.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.514+0000 7f1e7949d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e7406d610 0x7f1e741a65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:53.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.514+0000 7f1e7949d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e7406e6f0 0x7f1e741ad610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:53.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.514+0000 7f1e7949d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e741adb50 con 0x7f1e7406d610 2024-03-11T19:20:53.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.514+0000 7f1e7949d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e741adcc0 con 0x7f1e7406e6f0 2024-03-11T19:20:53.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.514+0000 7f1e7949d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e741ade80 con 0x7f1e7406cc70 2024-03-11T19:20:53.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.515+0000 7f1e727fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e7406d610 0x7f1e741a65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:53.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.515+0000 7f1e72ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e7406cc70 0x7f1e741a6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:53.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.515+0000 7f1e72ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e7406cc70 0x7f1e741a6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:50830/0 (socket says 172.21.15.29:50830) 2024-03-11T19:20:53.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.515+0000 7f1e72ffd640 1 -- 172.21.15.29:0/2344181235 learned_addr learned my addr 172.21.15.29:0/2344181235 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:53.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.515+0000 7f1e737fe640 1 --2- 172.21.15.29:0/2344181235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e7406e6f0 0x7f1e741ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:53.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.515+0000 7f1e72ffd640 1 -- 172.21.15.29:0/2344181235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e7406e6f0 msgr2=0x7f1e741ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:53.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.515+0000 7f1e72ffd640 1 --2- 172.21.15.29:0/2344181235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e7406e6f0 0x7f1e741ad610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.515+0000 7f1e72ffd640 1 -- 172.21.15.29:0/2344181235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e7406d610 msgr2=0x7f1e741a65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:53.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.515+0000 7f1e72ffd640 1 --2- 172.21.15.29:0/2344181235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e7406d610 0x7f1e741a65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.515+0000 7f1e72ffd640 1 -- 172.21.15.29:0/2344181235 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c009900 con 0x7f1e7406cc70 2024-03-11T19:20:53.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.516+0000 7f1e72ffd640 1 --2- 172.21.15.29:0/2344181235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e7406cc70 0x7f1e741a6060 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400d950 tx=0x7f1e6400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:53.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.517+0000 7f1e5bfff640 1 -- 172.21.15.29:0/2344181235 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e64009a30 con 0x7f1e7406cc70 2024-03-11T19:20:53.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.518+0000 7f1e5bfff640 1 -- 172.21.15.29:0/2344181235 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1e640044d0 con 0x7f1e7406cc70 2024-03-11T19:20:53.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.518+0000 7f1e7949d640 1 -- 172.21.15.29:0/2344181235 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e741ae160 con 0x7f1e7406cc70 2024-03-11T19:20:53.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.518+0000 7f1e7949d640 1 -- 172.21.15.29:0/2344181235 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e741ae670 con 0x7f1e7406cc70 2024-03-11T19:20:53.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.518+0000 7f1e5bfff640 1 -- 172.21.15.29:0/2344181235 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e640047a0 con 0x7f1e7406cc70 2024-03-11T19:20:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.522+0000 7f1e7c073740 1 Processor -- start 2024-03-11T19:20:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.522+0000 7f1e7c073740 1 -- start start 2024-03-11T19:20:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.523+0000 7f1e7c073740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fb4c0 0x5568ea8fb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.523+0000 7f1e7c073740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5568ea8fc690 0x5568ea8fcb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.523+0000 7f1e7c073740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fd050 0x5568ea9038e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.523+0000 7f1e7c073740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568ea86af90 con 0x5568ea8fd050 2024-03-11T19:20:53.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.523+0000 7f1e7c073740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568ea8684c0 con 0x5568ea8fc690 2024-03-11T19:20:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.523+0000 7f1e7c073740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568ea863ca0 con 0x5568ea8fb4c0 2024-03-11T19:20:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.523+0000 7f1e727fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fd050 0x5568ea9038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.523+0000 7f1e727fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fd050 0x5568ea9038e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48762/0 (socket says 172.21.15.29:48762) 2024-03-11T19:20:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.523+0000 7f1e727fc640 1 -- 172.21.15.29:0/2838592985 learned_addr learned my addr 172.21.15.29:0/2838592985 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:53.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.523+0000 7f1e737fe640 1 --2- 172.21.15.29:0/2838592985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fb4c0 0x5568ea8fb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:53.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.524+0000 7f1e737fe640 1 -- 172.21.15.29:0/2838592985 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5568ea8fc690 msgr2=0x5568ea8fcb10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T19:20:53.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.524+0000 7f1e737fe640 1 --2- 172.21.15.29:0/2838592985 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5568ea8fc690 0x5568ea8fcb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.524+0000 7f1e737fe640 1 -- 172.21.15.29:0/2838592985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fd050 msgr2=0x5568ea9038e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:53.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.524+0000 7f1e737fe640 1 --2- 172.21.15.29:0/2838592985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fd050 0x5568ea9038e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.524+0000 7f1e737fe640 1 -- 172.21.15.29:0/2838592985 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568ea8260b0 con 0x5568ea8fb4c0 2024-03-11T19:20:53.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.525+0000 7f1e5bfff640 1 -- 172.21.15.29:0/2344181235 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1e64009b90 con 0x7f1e7406cc70 2024-03-11T19:20:53.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.525+0000 7f1e5bfff640 1 --2- 172.21.15.29:0/2344181235 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1e5c0a1bb0 0x7f1e5c0a4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:53.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.526+0000 7f1e737fe640 1 --2- 172.21.15.29:0/2838592985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fb4c0 0x5568ea8fb8e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c037040 tx=0x7f1e6c009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4e6d31a68c36281 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:53.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.526+0000 7f1e5bfff640 1 -- 172.21.15.29:0/2344181235 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f1e640962d0 con 0x7f1e7406cc70 2024-03-11T19:20:53.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.526+0000 7f1e597fa640 1 -- 172.21.15.29:0/2838592985 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c032cb0 con 0x5568ea8fb4c0 2024-03-11T19:20:53.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.526+0000 7f1e597fa640 1 -- 172.21.15.29:0/2838592985 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1e6c032e10 con 0x5568ea8fb4c0 2024-03-11T19:20:53.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.526+0000 7f1e597fa640 1 -- 172.21.15.29:0/2838592985 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c0348a0 con 0x5568ea8fb4c0 2024-03-11T19:20:53.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.526+0000 7f1e72ffd640 1 --2- 172.21.15.29:0/2344181235 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1e5c0a1bb0 0x7f1e5c0a4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.527+0000 7f1e7c073740 1 -- 172.21.15.29:0/2838592985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fb4c0 msgr2=0x5568ea8fb8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.527+0000 7f1e7c073740 1 --2- 172.21.15.29:0/2838592985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fb4c0 0x5568ea8fb8e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c037040 tx=0x7f1e6c009800 comp rx=0 tx=0).stop 2024-03-11T19:20:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.527+0000 7f1e7c073740 1 -- 172.21.15.29:0/2838592985 shutdown_connections 2024-03-11T19:20:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.527+0000 7f1e7c073740 1 --2- 172.21.15.29:0/2838592985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fd050 0x5568ea9038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.527+0000 7f1e7c073740 1 --2- 172.21.15.29:0/2838592985 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5568ea8fc690 0x5568ea8fcb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.527+0000 7f1e7c073740 1 --2- 172.21.15.29:0/2838592985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fb4c0 0x5568ea8fb8e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.527+0000 7f1e7c073740 1 -- 172.21.15.29:0/2838592985 >> 172.21.15.29:0/2838592985 conn(0x5568ea8f66a0 msgr2=0x5568ea8f6aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:53.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.527+0000 7f1e72ffd640 1 --2- 172.21.15.29:0/2344181235 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1e5c0a1bb0 0x7f1e5c0a4070 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1e64099960 tx=0x7f1e64004710 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:53.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.528+0000 7f1e7c073740 1 -- 172.21.15.29:0/2838592985 shutdown_connections 2024-03-11T19:20:53.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.528+0000 7f1e7c073740 1 -- 172.21.15.29:0/2838592985 wait complete. 2024-03-11T19:20:53.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.529+0000 7f1e7c073740 1 -- start start 2024-03-11T19:20:53.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.529+0000 7f1e7c073740 1 Processor -- start 2024-03-11T19:20:53.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.530+0000 7f1e7c073740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fb4c0 0x5568ea905770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:53.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.530+0000 7f1e7c073740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fc690 0x5568ea905cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:53.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.530+0000 7f1e737fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fb4c0 0x5568ea905770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.530+0000 7f1e737fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fb4c0 0x5568ea905770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:50858/0 (socket says 172.21.15.29:50858) 2024-03-11T19:20:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.530+0000 7f1e737fe640 1 -- 172.21.15.29:0/3077369010 learned_addr learned my addr 172.21.15.29:0/3077369010 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.530+0000 7f1e7c073740 1 --2- 172.21.15.29:0/3077369010 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5568ea8fd050 0x5568ea90cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.530+0000 7f1e7c073740 1 -- 172.21.15.29:0/3077369010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568ea808f40 con 0x5568ea8fc690 2024-03-11T19:20:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.531+0000 7f1e7c073740 1 -- 172.21.15.29:0/3077369010 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568ea5374b0 con 0x5568ea8fd050 2024-03-11T19:20:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.531+0000 7f1e7c073740 1 -- 172.21.15.29:0/3077369010 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568ea7e9670 con 0x5568ea8fb4c0 2024-03-11T19:20:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.531+0000 7f1e727fc640 1 --2- 172.21.15.29:0/3077369010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fc690 0x5568ea905cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:53.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.531+0000 7f1e7c073740 10 client.0 _ll_register_callbacks cb 0x5568ea856f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.531+0000 7f1e737fe640 1 -- 172.21.15.29:0/3077369010 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5568ea8fd050 msgr2=0x5568ea90cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.531+0000 7f1e737fe640 1 --2- 172.21.15.29:0/3077369010 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5568ea8fd050 0x5568ea90cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.531+0000 7f1e737fe640 1 -- 172.21.15.29:0/3077369010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fc690 msgr2=0x5568ea905cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.531+0000 7f1e737fe640 1 --2- 172.21.15.29:0/3077369010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fc690 0x5568ea905cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.531+0000 7f1e737fe640 1 -- 172.21.15.29:0/3077369010 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e7406fec0 con 0x5568ea8fb4c0 2024-03-11T19:20:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.531+0000 7f1e727fc640 1 --2- 172.21.15.29:0/3077369010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fc690 0x5568ea905cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:20:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.532+0000 7f1e737fe640 1 --2- 172.21.15.29:0/3077369010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fb4c0 0x5568ea905770 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c02c140 tx=0x7f1e6c034f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2268e2099e4360 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.532+0000 7f1e56ff5640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-03-11T19:20:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.532+0000 7f1e56ff5640 1 -- 172.21.15.29:0/3077369010 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c0327b0 con 0x5568ea8fb4c0 2024-03-11T19:20:53.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.532+0000 7f1e56ff5640 1 -- 172.21.15.29:0/3077369010 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f1e6c032910 con 0x5568ea8fb4c0 2024-03-11T19:20:53.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.532+0000 7f1e7c073740 1 -- 172.21.15.29:0/3077369010 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5568ea83df40 con 0x5568ea8fb4c0 2024-03-11T19:20:53.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.533+0000 7f1e56ff5640 1 -- 172.21.15.29:0/3077369010 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e6c003970 con 0x5568ea8fb4c0 2024-03-11T19:20:53.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.533+0000 7f1e7c073740 10 client.27598 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:53.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.533+0000 7f1e7c073740 1 -- 172.21.15.29:0/3077369010 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5568ea862320 con 0x5568ea8fb4c0 2024-03-11T19:20:53.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.534+0000 7f1e56ff5640 1 -- 172.21.15.29:0/3077369010 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5568ea83df40 con 0x5568ea8fb4c0 2024-03-11T19:20:53.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.534+0000 7f1e56ff5640 1 -- 172.21.15.29:0/3077369010 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1e6c0045c0 con 0x5568ea8fb4c0 2024-03-11T19:20:53.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.534+0000 7f1e7c073740 10 client.27598 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:53.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.534+0000 7f1e7c073740 10 client.27598 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:20:53.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.534+0000 7f1e7c073740 20 client.27598 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:53.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.534+0000 7f1e7c073740 1 --2- 172.21.15.29:0/3077369010 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5568ea911590 0x5568ea9139b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:53.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.534+0000 7f1e7c073740 4 client.27598 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:20:53.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.534+0000 7f1e7c073740 1 -- 172.21.15.29:0/3077369010 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5568ea85a310 con 0x5568ea911590 2024-03-11T19:20:53.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.534+0000 7f1e727fc640 1 --2- 172.21.15.29:0/3077369010 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5568ea911590 0x5568ea9139b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.535+0000 7f1e727fc640 1 --2- 172.21.15.29:0/3077369010 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5568ea911590 0x5568ea9139b0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d956b9dab43039bb server_cookie=f74540836a0bc049 in_seq=0 out_seq=0 2024-03-11T19:20:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.535+0000 7f1e56ff5640 10 client.27598 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.535+0000 7f1e56ff5640 1 -- 172.21.15.29:0/3077369010 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5568ea85a310 con 0x5568ea911590 2024-03-11T19:20:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.535+0000 7f1e56ff5640 10 client.27598 handle_command_reply: tid=0 2024-03-11T19:20:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.536+0000 7f1e7c073740 1 client.27598 shutdown 2024-03-11T19:20:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.536+0000 7f1e7c073740 1 -- 172.21.15.29:0/3077369010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fb4c0 msgr2=0x5568ea905770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.536+0000 7f1e7c073740 1 --2- 172.21.15.29:0/3077369010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fb4c0 0x5568ea905770 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c02c140 tx=0x7f1e6c034f30 comp rx=0 tx=0).stop 2024-03-11T19:20:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.536+0000 7f1e7c073740 1 -- 172.21.15.29:0/3077369010 shutdown_connections 2024-03-11T19:20:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.536+0000 7f1e7c073740 1 --2- 172.21.15.29:0/3077369010 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5568ea8fd050 0x5568ea90cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.536+0000 7f1e7c073740 1 --2- 172.21.15.29:0/3077369010 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5568ea911590 0x5568ea9139b0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.536+0000 7f1e56ff5640 0 client.27598 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.536+0000 7f1e7c073740 1 --2- 172.21.15.29:0/3077369010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fc690 0x5568ea905cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.536+0000 7f1e7c073740 1 --2- 172.21.15.29:0/3077369010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fb4c0 0x5568ea905770 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.536+0000 7f1e7c073740 1 -- 172.21.15.29:0/3077369010 >> 172.21.15.29:0/3077369010 conn(0x5568ea8f66a0 msgr2=0x5568ea8f58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.536+0000 7f1e7c073740 1 -- 172.21.15.29:0/3077369010 shutdown_connections 2024-03-11T19:20:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.536+0000 7f1e7c073740 1 -- 172.21.15.29:0/3077369010 wait complete. 2024-03-11T19:20:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.536+0000 7f1e7c073740 20 client.27598 trim_cache size 0 max 16384 2024-03-11T19:20:53.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.559+0000 7f1e7c073740 1 Processor -- start 2024-03-11T19:20:53.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.559+0000 7f1e7c073740 1 -- start start 2024-03-11T19:20:53.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.559+0000 7f1e7c073740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5568ea8fb4c0 0x5568ea905380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:53.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.559+0000 7f1e7c073740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fc690 0x5568ea90cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:53.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.559+0000 7f1e7c073740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fd050 0x5568ea9058c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:53.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.559+0000 7f1e7c073740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568ea9067e0 con 0x5568ea8fd050 2024-03-11T19:20:53.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.559+0000 7f1e7c073740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568ea85a310 con 0x5568ea8fb4c0 2024-03-11T19:20:53.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.559+0000 7f1e7c073740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568ea8fc520 con 0x5568ea8fc690 2024-03-11T19:20:53.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.559+0000 7f1e737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fd050 0x5568ea9058c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:53.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.560+0000 7f1e727fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fc690 0x5568ea90cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:53.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.560+0000 7f1e727fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fc690 0x5568ea90cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:50860/0 (socket says 172.21.15.29:50860) 2024-03-11T19:20:53.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.560+0000 7f1e737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fd050 0x5568ea9058c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48776/0 (socket says 172.21.15.29:48776) 2024-03-11T19:20:53.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.560+0000 7f1e737fe640 1 -- 172.21.15.29:0/2677475237 learned_addr learned my addr 172.21.15.29:0/2677475237 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:53.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.560+0000 7f1e737fe640 1 --2- 172.21.15.29:0/2677475237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5568ea8fb4c0 0x5568ea905380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:53.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.560+0000 7f1e727fc640 1 -- 172.21.15.29:0/2677475237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5568ea8fb4c0 msgr2=0x5568ea905380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:53.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.560+0000 7f1e727fc640 1 --2- 172.21.15.29:0/2677475237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5568ea8fb4c0 0x5568ea905380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.560+0000 7f1e727fc640 1 -- 172.21.15.29:0/2677475237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fd050 msgr2=0x5568ea9058c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:53.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.560+0000 7f1e727fc640 1 --2- 172.21.15.29:0/2677475237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fd050 0x5568ea9058c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.560+0000 7f1e727fc640 1 -- 172.21.15.29:0/2677475237 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568ea7fec00 con 0x5568ea8fc690 2024-03-11T19:20:53.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.560+0000 7f1e727fc640 1 --2- 172.21.15.29:0/2677475237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fc690 0x5568ea90cc40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x5568ea914260 tx=0x7f1e6800b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c980773cfc2a0dd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:53.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.560+0000 7f1e58ff9640 1 -- 172.21.15.29:0/2677475237 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e68017040 con 0x5568ea8fc690 2024-03-11T19:20:53.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.560+0000 7f1e58ff9640 1 -- 172.21.15.29:0/2677475237 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1e6801c860 con 0x5568ea8fc690 2024-03-11T19:20:53.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.560+0000 7f1e58ff9640 1 -- 172.21.15.29:0/2677475237 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e68011af0 con 0x5568ea8fc690 2024-03-11T19:20:53.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.561+0000 7f1e7c073740 1 -- 172.21.15.29:0/2677475237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fc690 msgr2=0x5568ea90cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:53.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.561+0000 7f1e7c073740 1 --2- 172.21.15.29:0/2677475237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fc690 0x5568ea90cc40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x5568ea914260 tx=0x7f1e6800b8f0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.561+0000 7f1e7c073740 1 -- 172.21.15.29:0/2677475237 shutdown_connections 2024-03-11T19:20:53.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.561+0000 7f1e7c073740 1 --2- 172.21.15.29:0/2677475237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fd050 0x5568ea9058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.561+0000 7f1e7c073740 1 --2- 172.21.15.29:0/2677475237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fc690 0x5568ea90cc40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.561+0000 7f1e7c073740 1 --2- 172.21.15.29:0/2677475237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5568ea8fb4c0 0x5568ea905380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.561+0000 7f1e7c073740 1 -- 172.21.15.29:0/2677475237 >> 172.21.15.29:0/2677475237 conn(0x5568ea8f66a0 msgr2=0x5568ea9002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.561+0000 7f1e7c073740 1 -- 172.21.15.29:0/2677475237 shutdown_connections 2024-03-11T19:20:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.561+0000 7f1e7c073740 1 -- 172.21.15.29:0/2677475237 wait complete. 2024-03-11T19:20:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e7c073740 1 -- start start 2024-03-11T19:20:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e7c073740 1 Processor -- start 2024-03-11T19:20:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e7c073740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fb4c0 0x5568ea90ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e7c073740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fc690 0x5568ea9094a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e7c073740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5568ea8fd050 0x5568ea9099e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e7c073740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568ea909f50 con 0x5568ea8fc690 2024-03-11T19:20:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e7c073740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568ea90a0c0 con 0x5568ea8fd050 2024-03-11T19:20:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e7c073740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568ea967ab0 con 0x5568ea8fb4c0 2024-03-11T19:20:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e727fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fc690 0x5568ea9094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e727fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fc690 0x5568ea9094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48786/0 (socket says 172.21.15.29:48786) 2024-03-11T19:20:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e727fc640 1 -- 172.21.15.29:0/1789649945 learned_addr learned my addr 172.21.15.29:0/1789649945 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e737fe640 1 --2- 172.21.15.29:0/1789649945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fb4c0 0x5568ea90ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:53.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e7c073740 10 client.0 _ll_register_callbacks cb 0x7f1e6c037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:53.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e727fc640 1 -- 172.21.15.29:0/1789649945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fb4c0 msgr2=0x5568ea90ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:53.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e727fc640 1 --2- 172.21.15.29:0/1789649945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fb4c0 0x5568ea90ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e737fe640 1 --2- 172.21.15.29:0/1789649945 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5568ea8fd050 0x5568ea9099e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:53.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e727fc640 1 -- 172.21.15.29:0/1789649945 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5568ea8fd050 msgr2=0x5568ea9099e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:53.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e727fc640 1 --2- 172.21.15.29:0/1789649945 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5568ea8fd050 0x5568ea9099e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e727fc640 1 -- 172.21.15.29:0/1789649945 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e68004320 con 0x5568ea8fc690 2024-03-11T19:20:53.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e737fe640 1 --2- 172.21.15.29:0/1789649945 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5568ea8fd050 0x5568ea9099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:20:53.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e727fc640 1 --2- 172.21.15.29:0/1789649945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fc690 0x5568ea9094a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800bfd0 tx=0x7f1e680057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b43ce1d0c129ff7c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:53.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.562+0000 7f1e56ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:20:53.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:53 smithi029 ceph-mon[32439]: pgmap v683: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 53 MiB/s wr, 966 op/s 2024-03-11T19:20:53.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:53 smithi170 ceph-mon[35690]: pgmap v683: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 53 MiB/s wr, 966 op/s 2024-03-11T19:20:53.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.869+0000 7f1e56ff5640 1 -- 172.21.15.29:0/1789649945 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e68017040 con 0x5568ea8fc690 2024-03-11T19:20:53.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.870+0000 7f1e56ff5640 1 -- 172.21.15.29:0/1789649945 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f1e68003b60 con 0x5568ea8fc690 2024-03-11T19:20:53.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.870+0000 7f1e7c073740 1 -- 172.21.15.29:0/1789649945 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5568ea889900 con 0x5568ea8fc690 2024-03-11T19:20:53.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.870+0000 7f1e56ff5640 1 -- 172.21.15.29:0/1789649945 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e68003d20 con 0x5568ea8fc690 2024-03-11T19:20:53.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.870+0000 7f1e7c073740 10 client.18054 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:53.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.870+0000 7f1e7c073740 1 -- 172.21.15.29:0/1789649945 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5568ea824910 con 0x5568ea8fc690 2024-03-11T19:20:53.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.871+0000 7f1e56ff5640 1 -- 172.21.15.29:0/1789649945 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1e68003e80 con 0x5568ea8fc690 2024-03-11T19:20:53.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.871+0000 7f1e56ff5640 1 -- 172.21.15.29:0/1789649945 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1e68039900 con 0x5568ea8fc690 2024-03-11T19:20:53.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.871+0000 7f1e7c073740 10 client.18054 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:53.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.871+0000 7f1e7c073740 10 client.18054 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:20:53.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.871+0000 7f1e7c073740 20 client.18054 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:53.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.871+0000 7f1e7c073740 1 --2- 172.21.15.29:0/1789649945 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5568ea969500 0x5568ea96b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:53.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.871+0000 7f1e7c073740 4 client.18054 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-11T19:20:53.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.871+0000 7f1e7c073740 1 -- 172.21.15.29:0/1789649945 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5568ea889900 con 0x5568ea969500 2024-03-11T19:20:53.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.871+0000 7f1e737fe640 1 --2- 172.21.15.29:0/1789649945 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5568ea969500 0x5568ea96b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:53.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.872+0000 7f1e56ff5640 10 client.18054 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:53.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.872+0000 7f1e737fe640 1 --2- 172.21.15.29:0/1789649945 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5568ea969500 0x5568ea96b920 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=948eb89cf9bc184f server_cookie=9dd5836ee1fc3887 in_seq=0 out_seq=0 2024-03-11T19:20:53.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.873+0000 7f1e56ff5640 1 -- 172.21.15.29:0/1789649945 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5568ea889900 con 0x5568ea969500 2024-03-11T19:20:53.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.873+0000 7f1e56ff5640 10 client.18054 handle_command_reply: tid=0 2024-03-11T19:20:53.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.873+0000 7f1e7c073740 1 client.18054 shutdown 2024-03-11T19:20:53.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.873+0000 7f1e7c073740 1 -- 172.21.15.29:0/1789649945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fc690 msgr2=0x5568ea9094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:53.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.873+0000 7f1e7c073740 1 --2- 172.21.15.29:0/1789649945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fc690 0x5568ea9094a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800bfd0 tx=0x7f1e680057a0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.873+0000 7f1e7c073740 1 -- 172.21.15.29:0/1789649945 shutdown_connections 2024-03-11T19:20:53.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.873+0000 7f1e7c073740 1 --2- 172.21.15.29:0/1789649945 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5568ea8fd050 0x5568ea9099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.873+0000 7f1e7c073740 1 --2- 172.21.15.29:0/1789649945 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x5568ea969500 0x5568ea96b920 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.873+0000 7f1e56ff5640 0 client.18054 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:53.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.873+0000 7f1e7c073740 1 --2- 172.21.15.29:0/1789649945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5568ea8fc690 0x5568ea9094a0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.873+0000 7f1e7c073740 1 --2- 172.21.15.29:0/1789649945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5568ea8fb4c0 0x5568ea90ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.873+0000 7f1e7c073740 1 -- 172.21.15.29:0/1789649945 >> 172.21.15.29:0/1789649945 conn(0x5568ea8f66a0 msgr2=0x5568ea905310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:53.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.874+0000 7f1e7c073740 1 -- 172.21.15.29:0/1789649945 shutdown_connections 2024-03-11T19:20:53.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.874+0000 7f1e7c073740 1 -- 172.21.15.29:0/1789649945 wait complete. 2024-03-11T19:20:53.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.874+0000 7f1e7c073740 20 client.18054 trim_cache size 0 max 16384 2024-03-11T19:20:53.879 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:20:53.879 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-03-11T19:20:53.879 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "6b614643-d4be-4c6f-a653-71bd887b3091", 2024-03-11T19:20:53.879 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-03-11T19:20:53.879 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:20:53.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.874+0000 7f1e7949d640 1 -- 172.21.15.29:0/2344181235 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1e5c0a1bb0 msgr2=0x7f1e5c0a4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:53.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.874+0000 7f1e7949d640 1 --2- 172.21.15.29:0/2344181235 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1e5c0a1bb0 0x7f1e5c0a4070 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1e64099960 tx=0x7f1e64004710 comp rx=0 tx=0).stop 2024-03-11T19:20:53.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.874+0000 7f1e7949d640 1 -- 172.21.15.29:0/2344181235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e7406cc70 msgr2=0x7f1e741a6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:53.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.874+0000 7f1e7949d640 1 --2- 172.21.15.29:0/2344181235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e7406cc70 0x7f1e741a6060 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400d950 tx=0x7f1e6400de10 comp rx=0 tx=0).stop 2024-03-11T19:20:53.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.874+0000 7f1e7949d640 1 -- 172.21.15.29:0/2344181235 shutdown_connections 2024-03-11T19:20:53.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.874+0000 7f1e7949d640 1 --2- 172.21.15.29:0/2344181235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e7406e6f0 0x7f1e741ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.875+0000 7f1e7949d640 1 --2- 172.21.15.29:0/2344181235 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f1e5c0a1bb0 0x7f1e5c0a4070 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.875+0000 7f1e7949d640 1 --2- 172.21.15.29:0/2344181235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e7406d610 0x7f1e741a65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.875+0000 7f1e7949d640 1 --2- 172.21.15.29:0/2344181235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e7406cc70 0x7f1e741a6060 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:53.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.875+0000 7f1e7949d640 1 -- 172.21.15.29:0/2344181235 >> 172.21.15.29:0/2344181235 conn(0x7f1e7406bb80 msgr2=0x7f1e7410a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:53.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.875+0000 7f1e7949d640 1 -- 172.21.15.29:0/2344181235 shutdown_connections 2024-03-11T19:20:53.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:53.875+0000 7f1e7949d640 1 -- 172.21.15.29:0/2344181235 wait complete. 2024-03-11T19:20:53.891 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:20:53.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:20:54.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:53 smithi190 ceph-mon[36292]: pgmap v683: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 53 MiB/s wr, 966 op/s 2024-03-11T19:20:54.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.098+0000 7f8d85ffb640 1 -- 172.21.15.29:0/2456252481 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7802cbc0 con 0x7f8d88104d10 2024-03-11T19:20:54.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.099+0000 7f8d8d5d0640 1 -- 172.21.15.29:0/2456252481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d88104d10 msgr2=0x7f8d88105190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:54.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.099+0000 7f8d8d5d0640 1 --2- 172.21.15.29:0/2456252481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d88104d10 0x7f8d88105190 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f8d78009900 tx=0x7f8d7802bc80 comp rx=0 tx=0).stop 2024-03-11T19:20:54.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.099+0000 7f8d8d5d0640 1 -- 172.21.15.29:0/2456252481 shutdown_connections 2024-03-11T19:20:54.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.099+0000 7f8d8d5d0640 1 --2- 172.21.15.29:0/2456252481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d88104d10 0x7f8d88105190 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.099+0000 7f8d8d5d0640 1 --2- 172.21.15.29:0/2456252481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d88102360 0x7f8d881047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.099+0000 7f8d8d5d0640 1 --2- 172.21.15.29:0/2456252481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d880ff9f0 0x7f8d88101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.099+0000 7f8d8d5d0640 1 -- 172.21.15.29:0/2456252481 >> 172.21.15.29:0/2456252481 conn(0x7f8d880fb5f0 msgr2=0x7f8d880fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:54.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.103+0000 7f8d8d5d0640 1 -- 172.21.15.29:0/2456252481 shutdown_connections 2024-03-11T19:20:54.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.107+0000 7f8d8d5d0640 1 -- 172.21.15.29:0/2456252481 wait complete. 2024-03-11T19:20:54.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.107+0000 7f8d8d5d0640 1 Processor -- start 2024-03-11T19:20:54.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.108+0000 7f8d8d5d0640 1 -- start start 2024-03-11T19:20:54.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.108+0000 7f8d8d5d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d88104d10 0x7f8d8806fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:54.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.108+0000 7f8d8d5d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d88070230 0x7f8d881ab9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:54.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.108+0000 7f8d8d5d0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d880706b0 0x7f8d881adf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:54.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.108+0000 7f8d8d5d0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d881ae5a0 con 0x7f8d88070230 2024-03-11T19:20:54.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.108+0000 7f8d8d5d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d88106040 con 0x7f8d88104d10 2024-03-11T19:20:54.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.108+0000 7f8d8d5d0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d88106200 con 0x7f8d880706b0 2024-03-11T19:20:54.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.109+0000 7f8d877fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d880706b0 0x7f8d881adf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:54.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.109+0000 7f8d877fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d880706b0 0x7f8d881adf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.29:50882/0 (socket says 172.21.15.29:50882) 2024-03-11T19:20:54.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.109+0000 7f8d877fe640 1 -- 172.21.15.29:0/2167063285 learned_addr learned my addr 172.21.15.29:0/2167063285 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:54.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.110+0000 7f8d877fe640 1 -- 172.21.15.29:0/2167063285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d88104d10 msgr2=0x7f8d8806fcf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-11T19:20:54.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.110+0000 7f8d877fe640 1 --2- 172.21.15.29:0/2167063285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d88104d10 0x7f8d8806fcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.110+0000 7f8d877fe640 1 -- 172.21.15.29:0/2167063285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d88070230 msgr2=0x7f8d881ab9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:54.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.110+0000 7f8d877fe640 1 --2- 172.21.15.29:0/2167063285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d88070230 0x7f8d881ab9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.110+0000 7f8d877fe640 1 -- 172.21.15.29:0/2167063285 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d78009560 con 0x7f8d880706b0 2024-03-11T19:20:54.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.110+0000 7f8d877fe640 1 --2- 172.21.15.29:0/2167063285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d880706b0 0x7f8d881adf00 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8d7802c290 tx=0x7f8d780042e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:54.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.111+0000 7f8d6ffff640 1 -- 172.21.15.29:0/2167063285 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d78004510 con 0x7f8d880706b0 2024-03-11T19:20:54.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.111+0000 7f8d8d5d0640 1 -- 172.21.15.29:0/2167063285 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d88106480 con 0x7f8d880706b0 2024-03-11T19:20:54.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.111+0000 7f8d8d5d0640 1 -- 172.21.15.29:0/2167063285 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d88106a10 con 0x7f8d880706b0 2024-03-11T19:20:54.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.112+0000 7f8d6ffff640 1 -- 172.21.15.29:0/2167063285 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8d7802de90 con 0x7f8d880706b0 2024-03-11T19:20:54.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.112+0000 7f8d6ffff640 1 -- 172.21.15.29:0/2167063285 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d78005540 con 0x7f8d880706b0 2024-03-11T19:20:54.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.115+0000 7f8d6ffff640 1 -- 172.21.15.29:0/2167063285 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8d7803e450 con 0x7f8d880706b0 2024-03-11T19:20:54.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.115+0000 7f8d6ffff640 1 --2- 172.21.15.29:0/2167063285 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8d700a1bb0 0x7f8d700a4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:54.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.116+0000 7f8d6ffff640 1 -- 172.21.15.29:0/2167063285 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f8d780f0c30 con 0x7f8d880706b0 2024-03-11T19:20:54.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.116+0000 7f8d86ffd640 1 --2- 172.21.15.29:0/2167063285 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8d700a1bb0 0x7f8d700a4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:54.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.120+0000 7f8d86ffd640 1 --2- 172.21.15.29:0/2167063285 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8d700a1bb0 0x7f8d700a4070 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8d880684b0 tx=0x7f8d80009250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:54.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.120+0000 7f8d901a1740 1 Processor -- start 2024-03-11T19:20:54.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.120+0000 7f8d901a1740 1 -- start start 2024-03-11T19:20:54.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.121+0000 7f8d901a1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888e52d0 0x55a7888e56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:54.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.121+0000 7f8d901a1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888e5cc0 0x55a7888ea7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:54.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.121+0000 7f8d901a1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888eae30 0x55a7888ed2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:54.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.121+0000 7f8d901a1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a788855280 con 0x55a7888eae30 2024-03-11T19:20:54.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.121+0000 7f8d901a1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7888527b0 con 0x55a7888e52d0 2024-03-11T19:20:54.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.121+0000 7f8d901a1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78884df90 con 0x55a7888e5cc0 2024-03-11T19:20:54.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.121+0000 7f8d877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888eae30 0x55a7888ed2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:54.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.121+0000 7f8d877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888eae30 0x55a7888ed2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48820/0 (socket says 172.21.15.29:48820) 2024-03-11T19:20:54.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.121+0000 7f8d877fe640 1 -- 172.21.15.29:0/4272745657 learned_addr learned my addr 172.21.15.29:0/4272745657 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:54.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.121+0000 7f8d867fc640 1 --2- 172.21.15.29:0/4272745657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888e5cc0 0x55a7888ea7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:54.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.122+0000 7f8d86ffd640 1 --2- 172.21.15.29:0/4272745657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888e52d0 0x55a7888e56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:54.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.122+0000 7f8d877fe640 1 -- 172.21.15.29:0/4272745657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888e5cc0 msgr2=0x55a7888ea7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:54.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.122+0000 7f8d877fe640 1 --2- 172.21.15.29:0/4272745657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888e5cc0 0x55a7888ea7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.122+0000 7f8d877fe640 1 -- 172.21.15.29:0/4272745657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888e52d0 msgr2=0x55a7888e56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:54.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.122+0000 7f8d877fe640 1 --2- 172.21.15.29:0/4272745657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888e52d0 0x55a7888e56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.122+0000 7f8d877fe640 1 -- 172.21.15.29:0/4272745657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a78880c570 con 0x55a7888eae30 2024-03-11T19:20:54.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.122+0000 7f8d877fe640 1 --2- 172.21.15.29:0/4272745657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888eae30 0x55a7888ed2c0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f8d780050b0 tx=0x7f8d78009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c0dd0c7edd572b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:54.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.123+0000 7f8d6d7fa640 1 -- 172.21.15.29:0/4272745657 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d780c63e0 con 0x55a7888eae30 2024-03-11T19:20:54.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.123+0000 7f8d6d7fa640 1 -- 172.21.15.29:0/4272745657 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8d780bbf40 con 0x55a7888eae30 2024-03-11T19:20:54.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.123+0000 7f8d6d7fa640 1 -- 172.21.15.29:0/4272745657 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d780c61b0 con 0x55a7888eae30 2024-03-11T19:20:54.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.125+0000 7f8d901a1740 1 -- 172.21.15.29:0/4272745657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888eae30 msgr2=0x55a7888ed2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:54.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.125+0000 7f8d901a1740 1 --2- 172.21.15.29:0/4272745657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888eae30 0x55a7888ed2c0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f8d780050b0 tx=0x7f8d78009260 comp rx=0 tx=0).stop 2024-03-11T19:20:54.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.125+0000 7f8d901a1740 1 -- 172.21.15.29:0/4272745657 shutdown_connections 2024-03-11T19:20:54.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.125+0000 7f8d901a1740 1 --2- 172.21.15.29:0/4272745657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888eae30 0x55a7888ed2c0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.125+0000 7f8d901a1740 1 --2- 172.21.15.29:0/4272745657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888e5cc0 0x55a7888ea7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.125+0000 7f8d901a1740 1 --2- 172.21.15.29:0/4272745657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888e52d0 0x55a7888e56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.125+0000 7f8d901a1740 1 -- 172.21.15.29:0/4272745657 >> 172.21.15.29:0/4272745657 conn(0x55a7888e07f0 msgr2=0x55a7888e0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:54.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.126+0000 7f8d901a1740 1 -- 172.21.15.29:0/4272745657 shutdown_connections 2024-03-11T19:20:54.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.127+0000 7f8d901a1740 1 -- 172.21.15.29:0/4272745657 wait complete. 2024-03-11T19:20:54.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.128+0000 7f8d901a1740 1 -- start start 2024-03-11T19:20:54.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.128+0000 7f8d901a1740 1 Processor -- start 2024-03-11T19:20:54.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.128+0000 7f8d901a1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e52d0 0x55a7888eee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:54.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.128+0000 7f8d901a1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888e5cc0 0x55a7888ef360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:54.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.128+0000 7f8d901a1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888eae30 0x55a7888f63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:54.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.128+0000 7f8d901a1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a788521430 con 0x55a7888e52d0 2024-03-11T19:20:54.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.128+0000 7f8d901a1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7884dde00 con 0x55a7888e5cc0 2024-03-11T19:20:54.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.128+0000 7f8d901a1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7887d2fd0 con 0x55a7888eae30 2024-03-11T19:20:54.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.128+0000 7f8d901a1740 10 client.0 _ll_register_callbacks cb 0x55a788828fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:54.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.138+0000 7f8d86ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e52d0 0x55a7888eee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:54.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.138+0000 7f8d86ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e52d0 0x55a7888eee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48830/0 (socket says 172.21.15.29:48830) 2024-03-11T19:20:54.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.138+0000 7f8d86ffd640 1 -- 172.21.15.29:0/1605566297 learned_addr learned my addr 172.21.15.29:0/1605566297 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:54.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.138+0000 7f8d877fe640 1 --2- 172.21.15.29:0/1605566297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888eae30 0x55a7888f63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:54.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.139+0000 7f8d867fc640 1 --2- 172.21.15.29:0/1605566297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888e5cc0 0x55a7888ef360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:54.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.140+0000 7f8d867fc640 1 -- 172.21.15.29:0/1605566297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888eae30 msgr2=0x55a7888f63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:54.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.140+0000 7f8d867fc640 1 --2- 172.21.15.29:0/1605566297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888eae30 0x55a7888f63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.141+0000 7f8d867fc640 1 -- 172.21.15.29:0/1605566297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e52d0 msgr2=0x55a7888eee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:54.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.141+0000 7f8d867fc640 1 --2- 172.21.15.29:0/1605566297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e52d0 0x55a7888eee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.141+0000 7f8d867fc640 1 -- 172.21.15.29:0/1605566297 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d88106a10 con 0x55a7888e5cc0 2024-03-11T19:20:54.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.141+0000 7f8d86ffd640 1 --2- 172.21.15.29:0/1605566297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e52d0 0x55a7888eee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:20:54.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.144+0000 7f8d6aff5640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:20:54.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.144+0000 7f8d867fc640 1 --2- 172.21.15.29:0/1605566297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888e5cc0 0x55a7888ef360 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x55a7888e60c0 tx=0x7f8d7c00d530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a2da9238ec0bd53 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:54.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.145+0000 7f8d6aff5640 1 -- 172.21.15.29:0/1605566297 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c018070 con 0x55a7888e5cc0 2024-03-11T19:20:54.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.145+0000 7f8d901a1740 1 -- 172.21.15.29:0/1605566297 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a78854d7e0 con 0x55a7888e5cc0 2024-03-11T19:20:54.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.145+0000 7f8d6aff5640 1 -- 172.21.15.29:0/1605566297 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f8d7c014430 con 0x55a7888e5cc0 2024-03-11T19:20:54.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.145+0000 7f8d6aff5640 1 -- 172.21.15.29:0/1605566297 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d7c018070 con 0x55a7888e5cc0 2024-03-11T19:20:54.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.146+0000 7f8d901a1740 10 client.27560 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:54.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.146+0000 7f8d901a1740 1 -- 172.21.15.29:0/1605566297 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a78884c610 con 0x55a7888e5cc0 2024-03-11T19:20:54.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.147+0000 7f8d6aff5640 1 -- 172.21.15.29:0/1605566297 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55a78854d7e0 con 0x55a7888e5cc0 2024-03-11T19:20:54.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.147+0000 7f8d6aff5640 1 -- 172.21.15.29:0/1605566297 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8d7c01b020 con 0x55a7888e5cc0 2024-03-11T19:20:54.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.147+0000 7f8d901a1740 10 client.27560 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:54.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.147+0000 7f8d901a1740 10 client.27560 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:20:54.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.147+0000 7f8d901a1740 20 client.27560 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:54.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.147+0000 7f8d901a1740 1 --2- 172.21.15.29:0/1605566297 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55a7888fb010 0x55a7888fd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:54.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.148+0000 7f8d901a1740 4 client.27560 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:20:54.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.148+0000 7f8d901a1740 1 -- 172.21.15.29:0/1605566297 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a78858a720 con 0x55a7888fb010 2024-03-11T19:20:54.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.148+0000 7f8d86ffd640 1 --2- 172.21.15.29:0/1605566297 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55a7888fb010 0x55a7888fd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:54.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.153+0000 7f8d86ffd640 1 --2- 172.21.15.29:0/1605566297 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55a7888fb010 0x55a7888fd430 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73e4749b67b70692 server_cookie=5c697906af6cf66a in_seq=0 out_seq=0 2024-03-11T19:20:54.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.153+0000 7f8d6aff5640 10 client.27560 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:54.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.154+0000 7f8d6aff5640 1 -- 172.21.15.29:0/1605566297 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a78858a720 con 0x55a7888fb010 2024-03-11T19:20:54.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.154+0000 7f8d6aff5640 10 client.27560 handle_command_reply: tid=0 2024-03-11T19:20:54.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.154+0000 7f8d901a1740 1 client.27560 shutdown 2024-03-11T19:20:54.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.155+0000 7f8d901a1740 1 -- 172.21.15.29:0/1605566297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888e5cc0 msgr2=0x55a7888ef360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:54.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.155+0000 7f8d901a1740 1 --2- 172.21.15.29:0/1605566297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888e5cc0 0x55a7888ef360 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x55a7888e60c0 tx=0x7f8d7c00d530 comp rx=0 tx=0).stop 2024-03-11T19:20:54.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.155+0000 7f8d901a1740 1 -- 172.21.15.29:0/1605566297 shutdown_connections 2024-03-11T19:20:54.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.155+0000 7f8d901a1740 1 --2- 172.21.15.29:0/1605566297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888eae30 0x55a7888f63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.155+0000 7f8d901a1740 1 --2- 172.21.15.29:0/1605566297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888e5cc0 0x55a7888ef360 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.155+0000 7f8d901a1740 1 --2- 172.21.15.29:0/1605566297 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55a7888fb010 0x55a7888fd430 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.155+0000 7f8d6aff5640 0 client.27560 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:54.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.155+0000 7f8d901a1740 1 --2- 172.21.15.29:0/1605566297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e52d0 0x55a7888eee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.155+0000 7f8d901a1740 1 -- 172.21.15.29:0/1605566297 >> 172.21.15.29:0/1605566297 conn(0x55a7888e07f0 msgr2=0x55a7888dfa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:54.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.155+0000 7f8d901a1740 1 -- 172.21.15.29:0/1605566297 shutdown_connections 2024-03-11T19:20:54.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.155+0000 7f8d901a1740 1 -- 172.21.15.29:0/1605566297 wait complete. 2024-03-11T19:20:54.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.156+0000 7f8d901a1740 20 client.27560 trim_cache size 0 max 16384 2024-03-11T19:20:54.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.185+0000 7f8d901a1740 1 Processor -- start 2024-03-11T19:20:54.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.185+0000 7f8d901a1740 1 -- start start 2024-03-11T19:20:54.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.186+0000 7f8d901a1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e52d0 0x55a7888e6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:54.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.186+0000 7f8d901a1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888e5cc0 0x55a7888e7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:54.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.186+0000 7f8d901a1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888eae30 0x55a7888f62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:54.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.186+0000 7f8d901a1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78854d7e0 con 0x55a7888e52d0 2024-03-11T19:20:54.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.186+0000 7f8d901a1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7888e6500 con 0x55a7888e5cc0 2024-03-11T19:20:54.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.186+0000 7f8d901a1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7888e60a0 con 0x55a7888eae30 2024-03-11T19:20:54.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.186+0000 7f8d86ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e52d0 0x55a7888e6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:54.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.186+0000 7f8d877fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888eae30 0x55a7888f62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:54.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.186+0000 7f8d867fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888e5cc0 0x55a7888e7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:54.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.186+0000 7f8d86ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e52d0 0x55a7888e6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48836/0 (socket says 172.21.15.29:48836) 2024-03-11T19:20:54.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.186+0000 7f8d86ffd640 1 -- 172.21.15.29:0/3314142446 learned_addr learned my addr 172.21.15.29:0/3314142446 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:54.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.186+0000 7f8d86ffd640 1 -- 172.21.15.29:0/3314142446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888eae30 msgr2=0x55a7888f62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:54.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.186+0000 7f8d86ffd640 1 --2- 172.21.15.29:0/3314142446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888eae30 0x55a7888f62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.186+0000 7f8d86ffd640 1 -- 172.21.15.29:0/3314142446 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888e5cc0 msgr2=0x55a7888e7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:54.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.186+0000 7f8d86ffd640 1 --2- 172.21.15.29:0/3314142446 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888e5cc0 0x55a7888e7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.186+0000 7f8d86ffd640 1 -- 172.21.15.29:0/3314142446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a7887e8ac0 con 0x55a7888e52d0 2024-03-11T19:20:54.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.186+0000 7f8d867fc640 1 --2- 172.21.15.29:0/3314142446 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888e5cc0 0x55a7888e7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:20:54.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.186+0000 7f8d877fe640 1 --2- 172.21.15.29:0/3314142446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888eae30 0x55a7888f62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:20:54.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.187+0000 7f8d86ffd640 1 --2- 172.21.15.29:0/3314142446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e52d0 0x55a7888e6e00 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f8d8000cfc0 tx=0x7f8d8000f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ddd5b6b8803728 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:54.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.187+0000 7f8d6cff9640 1 -- 172.21.15.29:0/3314142446 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8001f040 con 0x55a7888e52d0 2024-03-11T19:20:54.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.187+0000 7f8d6cff9640 1 -- 172.21.15.29:0/3314142446 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8d80018430 con 0x55a7888e52d0 2024-03-11T19:20:54.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.187+0000 7f8d6cff9640 1 -- 172.21.15.29:0/3314142446 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8002e620 con 0x55a7888e52d0 2024-03-11T19:20:54.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.188+0000 7f8d901a1740 1 -- 172.21.15.29:0/3314142446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e52d0 msgr2=0x55a7888e6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:54.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.189+0000 7f8d901a1740 1 --2- 172.21.15.29:0/3314142446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e52d0 0x55a7888e6e00 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f8d8000cfc0 tx=0x7f8d8000f000 comp rx=0 tx=0).stop 2024-03-11T19:20:54.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.189+0000 7f8d901a1740 1 -- 172.21.15.29:0/3314142446 shutdown_connections 2024-03-11T19:20:54.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.189+0000 7f8d901a1740 1 --2- 172.21.15.29:0/3314142446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888eae30 0x55a7888f62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.189+0000 7f8d901a1740 1 --2- 172.21.15.29:0/3314142446 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888e5cc0 0x55a7888e7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.189+0000 7f8d901a1740 1 --2- 172.21.15.29:0/3314142446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e52d0 0x55a7888e6e00 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.189+0000 7f8d901a1740 1 -- 172.21.15.29:0/3314142446 >> 172.21.15.29:0/3314142446 conn(0x55a7888e07f0 msgr2=0x55a788901db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:54.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.190+0000 7f8d901a1740 1 -- 172.21.15.29:0/3314142446 shutdown_connections 2024-03-11T19:20:54.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.190+0000 7f8d901a1740 1 -- 172.21.15.29:0/3314142446 wait complete. 2024-03-11T19:20:54.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.190+0000 7f8d901a1740 1 -- start start 2024-03-11T19:20:54.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.190+0000 7f8d901a1740 1 Processor -- start 2024-03-11T19:20:54.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.191+0000 7f8d901a1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888e52d0 0x55a7888f6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:54.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.191+0000 7f8d901a1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e5cc0 0x55a7888f4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:54.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.191+0000 7f8d901a1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888eae30 0x55a7888f4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:54.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.191+0000 7f8d901a1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7888f5190 con 0x55a7888e5cc0 2024-03-11T19:20:54.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.191+0000 7f8d901a1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7888f5300 con 0x55a7888eae30 2024-03-11T19:20:54.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.191+0000 7f8d901a1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7888f54c0 con 0x55a7888e52d0 2024-03-11T19:20:54.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.191+0000 7f8d901a1740 10 client.0 _ll_register_callbacks cb 0x7f8d780bda10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:20:54.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.191+0000 7f8d867fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e5cc0 0x55a7888f4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:54.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.191+0000 7f8d877fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888eae30 0x55a7888f4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:54.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.191+0000 7f8d877fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888eae30 0x55a7888f4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:49868/0 (socket says 172.21.15.29:49868) 2024-03-11T19:20:54.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.191+0000 7f8d877fe640 1 -- 172.21.15.29:0/951871153 learned_addr learned my addr 172.21.15.29:0/951871153 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:20:54.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.192+0000 7f8d877fe640 1 -- 172.21.15.29:0/951871153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888e52d0 msgr2=0x55a7888f6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:54.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.192+0000 7f8d877fe640 1 --2- 172.21.15.29:0/951871153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888e52d0 0x55a7888f6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.192+0000 7f8d877fe640 1 -- 172.21.15.29:0/951871153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e5cc0 msgr2=0x55a7888f4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:20:54.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.192+0000 7f8d877fe640 1 --2- 172.21.15.29:0/951871153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e5cc0 0x55a7888f4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.192+0000 7f8d877fe640 1 -- 172.21.15.29:0/951871153 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a78884c610 con 0x55a7888eae30 2024-03-11T19:20:54.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.192+0000 7f8d867fc640 1 --2- 172.21.15.29:0/951871153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e5cc0 0x55a7888f4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:20:54.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.192+0000 7f8d877fe640 1 --2- 172.21.15.29:0/951871153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888eae30 0x55a7888f4bd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8d780bd940 tx=0x7f8d780bddc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8012d8112a0576d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:20:54.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.192+0000 7f8d6aff5640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-11T19:20:54.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.193+0000 7f8d6aff5640 1 -- 172.21.15.29:0/951871153 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d780bd290 con 0x55a7888eae30 2024-03-11T19:20:54.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.193+0000 7f8d6aff5640 1 -- 172.21.15.29:0/951871153 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f8d780c63e0 con 0x55a7888eae30 2024-03-11T19:20:54.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.193+0000 7f8d901a1740 1 -- 172.21.15.29:0/951871153 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8d780bbf40 con 0x55a7888eae30 2024-03-11T19:20:54.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.193+0000 7f8d6aff5640 1 -- 172.21.15.29:0/951871153 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d780bd3f0 con 0x55a7888eae30 2024-03-11T19:20:54.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.193+0000 7f8d901a1740 10 client.27566 fetch_fsmap learned FSMap version 27 2024-03-11T19:20:54.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.193+0000 7f8d901a1740 1 -- 172.21.15.29:0/951871153 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a78880c570 con 0x55a7888eae30 2024-03-11T19:20:54.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.194+0000 7f8d6aff5640 1 -- 172.21.15.29:0/951871153 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8d780bbf40 con 0x55a7888eae30 2024-03-11T19:20:54.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.194+0000 7f8d6aff5640 1 -- 172.21.15.29:0/951871153 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8d780cf050 con 0x55a7888eae30 2024-03-11T19:20:54.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.194+0000 7f8d901a1740 10 client.27566 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:20:54.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.194+0000 7f8d901a1740 10 client.27566 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:20:54.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.194+0000 7f8d901a1740 20 client.27566 populate_metadata read hostname 'smithi029' 2024-03-11T19:20:54.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.194+0000 7f8d901a1740 1 --2- 172.21.15.29:0/951871153 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55a7888fb010 0x55a7889549a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:20:54.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.194+0000 7f8d901a1740 4 client.27566 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:20:54.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.194+0000 7f8d901a1740 1 -- 172.21.15.29:0/951871153 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a7885d9890 con 0x55a7888fb010 2024-03-11T19:20:54.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.194+0000 7f8d86ffd640 1 --2- 172.21.15.29:0/951871153 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55a7888fb010 0x55a7889549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:20:54.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.195+0000 7f8d86ffd640 1 --2- 172.21.15.29:0/951871153 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55a7888fb010 0x55a7889549a0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24e8f7e0e230318d server_cookie=ddfc5af6cd822497 in_seq=0 out_seq=0 2024-03-11T19:20:54.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.197+0000 7f8d6aff5640 10 client.27566 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:54.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.197+0000 7f8d6aff5640 1 -- 172.21.15.29:0/951871153 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55a7885d9890 con 0x55a7888fb010 2024-03-11T19:20:54.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.197+0000 7f8d6aff5640 10 client.27566 handle_command_reply: tid=0 2024-03-11T19:20:54.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.197+0000 7f8d901a1740 1 client.27566 shutdown 2024-03-11T19:20:54.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.199+0000 7f8d901a1740 1 -- 172.21.15.29:0/951871153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888eae30 msgr2=0x55a7888f4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:54.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.199+0000 7f8d901a1740 1 --2- 172.21.15.29:0/951871153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888eae30 0x55a7888f4bd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8d780bd940 tx=0x7f8d780bddc0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.200+0000 7f8d901a1740 1 -- 172.21.15.29:0/951871153 shutdown_connections 2024-03-11T19:20:54.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.200+0000 7f8d901a1740 1 --2- 172.21.15.29:0/951871153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a7888eae30 0x55a7888f4bd0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.200+0000 7f8d901a1740 1 --2- 172.21.15.29:0/951871153 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55a7888fb010 0x55a7889549a0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.200+0000 7f8d901a1740 1 --2- 172.21.15.29:0/951871153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a7888e5cc0 0x55a7888f4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.200+0000 7f8d901a1740 1 --2- 172.21.15.29:0/951871153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7888e52d0 0x55a7888f6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.200+0000 7f8d901a1740 1 -- 172.21.15.29:0/951871153 >> 172.21.15.29:0/951871153 conn(0x55a7888e07f0 msgr2=0x55a7888e6e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:54.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.200+0000 7f8d6aff5640 0 client.27566 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:20:54.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.200+0000 7f8d901a1740 1 -- 172.21.15.29:0/951871153 shutdown_connections 2024-03-11T19:20:54.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.200+0000 7f8d901a1740 1 -- 172.21.15.29:0/951871153 wait complete. 2024-03-11T19:20:54.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.200+0000 7f8d901a1740 20 client.27566 trim_cache size 0 max 16384 2024-03-11T19:20:54.205 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:20:54.206 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-11T19:20:54.206 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-03-11T19:20:54.206 INFO:teuthology.orchestra.run.smithi029.stdout: "6b614643-d4be-4c6f-a653-71bd887b3091": { 2024-03-11T19:20:54.206 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-03-11T19:20:54.206 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "6b614643-d4be-4c6f-a653-71bd887b3091", 2024-03-11T19:20:54.206 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-03-11T19:20:54.206 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:20:54.206 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:20:54.206 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:20:54.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.202+0000 7f8d8d5d0640 1 -- 172.21.15.29:0/2167063285 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8d700a1bb0 msgr2=0x7f8d700a4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:54.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.203+0000 7f8d8d5d0640 1 --2- 172.21.15.29:0/2167063285 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8d700a1bb0 0x7f8d700a4070 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8d880684b0 tx=0x7f8d80009250 comp rx=0 tx=0).stop 2024-03-11T19:20:54.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.203+0000 7f8d8d5d0640 1 -- 172.21.15.29:0/2167063285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d880706b0 msgr2=0x7f8d881adf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:20:54.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.203+0000 7f8d8d5d0640 1 --2- 172.21.15.29:0/2167063285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d880706b0 0x7f8d881adf00 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8d7802c290 tx=0x7f8d780042e0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.203+0000 7f8d8d5d0640 1 -- 172.21.15.29:0/2167063285 shutdown_connections 2024-03-11T19:20:54.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.203+0000 7f8d8d5d0640 1 --2- 172.21.15.29:0/2167063285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d880706b0 0x7f8d881adf00 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.203+0000 7f8d8d5d0640 1 --2- 172.21.15.29:0/2167063285 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f8d700a1bb0 0x7f8d700a4070 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.203+0000 7f8d8d5d0640 1 --2- 172.21.15.29:0/2167063285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d88070230 0x7f8d881ab9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.203+0000 7f8d8d5d0640 1 --2- 172.21.15.29:0/2167063285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d88104d10 0x7f8d8806fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:20:54.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.203+0000 7f8d8d5d0640 1 -- 172.21.15.29:0/2167063285 >> 172.21.15.29:0/2167063285 conn(0x7f8d880fb5f0 msgr2=0x7f8d88104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:20:54.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.204+0000 7f8d8d5d0640 1 -- 172.21.15.29:0/2167063285 shutdown_connections 2024-03-11T19:20:54.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:20:54.204+0000 7f8d8d5d0640 1 -- 172.21.15.29:0/2167063285 wait complete. 2024-03-11T19:20:54.223 INFO:tasks.cephfs.filesystem:scrub status for tag:6b614643-d4be-4c6f-a653-71bd887b3091 - {'path': '/', 'tag': '6b614643-d4be-4c6f-a653-71bd887b3091', 'options': 'recursive,force'} 2024-03-11T19:20:55.819 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:55 smithi029 ceph-mon[32439]: pgmap v684: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 52 MiB/s wr, 934 op/s 2024-03-11T19:20:55.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:55 smithi170 ceph-mon[35690]: pgmap v684: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 52 MiB/s wr, 934 op/s 2024-03-11T19:20:55.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:55 smithi170 ceph-mon[35690]: scrub queued for path: / 2024-03-11T19:20:55.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:55 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:20:55.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:55 smithi170 ceph-mon[35690]: scrub summary: active paths [/] 2024-03-11T19:20:55.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:55 smithi170 ceph-mon[35690]: scrub summary: idle+waiting paths [/] 2024-03-11T19:20:56.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:55 smithi190 ceph-mon[36292]: pgmap v684: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 52 MiB/s wr, 934 op/s 2024-03-11T19:20:56.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:55 smithi190 ceph-mon[36292]: scrub queued for path: / 2024-03-11T19:20:56.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:55 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:20:56.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:55 smithi190 ceph-mon[36292]: scrub summary: active paths [/] 2024-03-11T19:20:56.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:55 smithi190 ceph-mon[36292]: scrub summary: idle+waiting paths [/] 2024-03-11T19:20:56.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:55 smithi029 ceph-mon[32439]: scrub queued for path: / 2024-03-11T19:20:56.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:55 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:20:56.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:55 smithi029 ceph-mon[32439]: scrub summary: active paths [/] 2024-03-11T19:20:56.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:55 smithi029 ceph-mon[32439]: scrub summary: idle+waiting paths [/] 2024-03-11T19:20:56.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:56 smithi170 ceph-mon[35690]: scrub summary: idle 2024-03-11T19:20:56.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:56 smithi029 ceph-mon[32439]: scrub summary: idle 2024-03-11T19:20:57.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:56 smithi190 ceph-mon[36292]: scrub summary: idle 2024-03-11T19:20:57.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:57 smithi170 ceph-mon[35690]: pgmap v685: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 49 MiB/s wr, 870 op/s 2024-03-11T19:20:57.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:57 smithi029 ceph-mon[32439]: pgmap v685: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 49 MiB/s wr, 870 op/s 2024-03-11T19:20:58.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:57 smithi190 ceph-mon[36292]: pgmap v685: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 49 MiB/s wr, 870 op/s 2024-03-11T19:20:59.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:20:59 smithi170 ceph-mon[35690]: pgmap v686: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 40 MiB/s wr, 714 op/s 2024-03-11T19:20:59.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:20:59 smithi029 ceph-mon[32439]: pgmap v686: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 40 MiB/s wr, 714 op/s 2024-03-11T19:21:00.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:20:59 smithi190 ceph-mon[36292]: pgmap v686: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 40 MiB/s wr, 714 op/s 2024-03-11T19:21:01.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:01 smithi170 ceph-mon[35690]: pgmap v687: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 43 MiB/s wr, 768 op/s 2024-03-11T19:21:01.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:01 smithi029 ceph-mon[32439]: pgmap v687: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 43 MiB/s wr, 768 op/s 2024-03-11T19:21:02.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:01 smithi190 ceph-mon[36292]: pgmap v687: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 43 MiB/s wr, 768 op/s 2024-03-11T19:21:03.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:03 smithi170 ceph-mon[35690]: pgmap v688: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 39 MiB/s wr, 698 op/s 2024-03-11T19:21:03.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:03 smithi029 ceph-mon[32439]: pgmap v688: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 39 MiB/s wr, 698 op/s 2024-03-11T19:21:04.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:03 smithi190 ceph-mon[36292]: pgmap v688: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 39 MiB/s wr, 698 op/s 2024-03-11T19:21:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:05 smithi170 ceph-mon[35690]: pgmap v689: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 39 MiB/s wr, 711 op/s 2024-03-11T19:21:05.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:05 smithi029 ceph-mon[32439]: pgmap v689: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 39 MiB/s wr, 711 op/s 2024-03-11T19:21:06.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:05 smithi190 ceph-mon[36292]: pgmap v689: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 39 MiB/s wr, 711 op/s 2024-03-11T19:21:07.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:07 smithi029 ceph-mon[32439]: pgmap v690: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 41 MiB/s wr, 744 op/s 2024-03-11T19:21:07.408 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:21:07.408 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:07 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:21:07.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:07 smithi190 ceph-mon[36292]: pgmap v690: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 41 MiB/s wr, 744 op/s 2024-03-11T19:21:07.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:21:07.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:07 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:21:07.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:07 smithi170 ceph-mon[35690]: pgmap v690: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 41 MiB/s wr, 744 op/s 2024-03-11T19:21:07.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:21:07.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:07 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:21:08.408 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:21:08.408 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:21:08.409 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:21:08.512 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:21:08.513 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:21:08.513 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:21:08.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:21:08.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:21:08.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:21:09.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:09 smithi170 ceph-mon[35690]: pgmap v691: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 38 MiB/s wr, 695 op/s 2024-03-11T19:21:09.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:09 smithi029 ceph-mon[32439]: pgmap v691: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 38 MiB/s wr, 695 op/s 2024-03-11T19:21:09.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:09 smithi190 ceph-mon[36292]: pgmap v691: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 38 MiB/s wr, 695 op/s 2024-03-11T19:21:11.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:11 smithi170 ceph-mon[35690]: pgmap v692: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 45 MiB/s wr, 815 op/s 2024-03-11T19:21:11.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:11 smithi029 ceph-mon[32439]: pgmap v692: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 45 MiB/s wr, 815 op/s 2024-03-11T19:21:11.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:11 smithi190 ceph-mon[36292]: pgmap v692: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 45 MiB/s wr, 815 op/s 2024-03-11T19:21:13.096 INFO:tasks.workunit.client.0.smithi029.stdout:stopping iogen 2024-03-11T19:21:13.096 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 'stopping iogen' 2024-03-11T19:21:13.096 INFO:tasks.workunit.client.0.smithi029.stderr:+ ./iogen -k 2024-03-11T19:21:13.100 INFO:tasks.workunit.client.0.smithi029.stdout:OK 2024-03-11T19:21:13.100 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo OK 2024-03-11T19:21:13.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:13 smithi029 ceph-mon[32439]: pgmap v693: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 44 MiB/s wr, 805 op/s 2024-03-11T19:21:13.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:13 smithi190 ceph-mon[36292]: pgmap v693: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 44 MiB/s wr, 805 op/s 2024-03-11T19:21:13.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:13 smithi170 ceph-mon[35690]: pgmap v693: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 44 MiB/s wr, 805 op/s 2024-03-11T19:21:15.201 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-11T19:21:15.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-11T19:21:15.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:15 smithi029 ceph-mon[32439]: pgmap v694: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 49 MiB/s wr, 885 op/s 2024-03-11T19:21:15.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:15 smithi190 ceph-mon[36292]: pgmap v694: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 49 MiB/s wr, 885 op/s 2024-03-11T19:21:15.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:15 smithi170 ceph-mon[35690]: pgmap v694: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 49 MiB/s wr, 885 op/s 2024-03-11T19:21:16.024 INFO:tasks.workunit:Stopping ['suites/iogen.sh'] on client.0... 2024-03-11T19:21:16.024 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-11T19:21:16.347 DEBUG:teuthology.parallel:result is None 2024-03-11T19:21:16.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-11T19:21:16.388 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-11T19:21:16.388 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-11T19:21:16.398 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-11T19:21:17.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:17 smithi029 ceph-mon[32439]: pgmap v695: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 55 MiB/s wr, 981 op/s 2024-03-11T19:21:17.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:17 smithi190 ceph-mon[36292]: pgmap v695: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 55 MiB/s wr, 981 op/s 2024-03-11T19:21:17.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:17 smithi170 ceph-mon[35690]: pgmap v695: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 55 MiB/s wr, 981 op/s 2024-03-11T19:21:19.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:19 smithi029 ceph-mon[32439]: pgmap v696: 97 pgs: 97 active+clean; 9.9 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 50 MiB/s wr, 899 op/s 2024-03-11T19:21:19.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:19 smithi190 ceph-mon[36292]: pgmap v696: 97 pgs: 97 active+clean; 9.9 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 50 MiB/s wr, 899 op/s 2024-03-11T19:21:19.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:19 smithi170 ceph-mon[35690]: pgmap v696: 97 pgs: 97 active+clean; 9.9 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 50 MiB/s wr, 899 op/s 2024-03-11T19:21:21.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:21 smithi029 ceph-mon[32439]: pgmap v697: 97 pgs: 97 active+clean; 6.8 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 49 MiB/s wr, 939 op/s 2024-03-11T19:21:21.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:21 smithi190 ceph-mon[36292]: pgmap v697: 97 pgs: 97 active+clean; 6.8 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 49 MiB/s wr, 939 op/s 2024-03-11T19:21:21.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:21 smithi170 ceph-mon[35690]: pgmap v697: 97 pgs: 97 active+clean; 6.8 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 49 MiB/s wr, 939 op/s 2024-03-11T19:21:23.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:23 smithi029 ceph-mon[32439]: pgmap v698: 97 pgs: 97 active+clean; 3.4 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 39 MiB/s wr, 829 op/s 2024-03-11T19:21:23.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:23 smithi190 ceph-mon[36292]: pgmap v698: 97 pgs: 97 active+clean; 3.4 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 39 MiB/s wr, 829 op/s 2024-03-11T19:21:23.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:23 smithi170 ceph-mon[35690]: pgmap v698: 97 pgs: 97 active+clean; 3.4 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 39 MiB/s wr, 829 op/s 2024-03-11T19:21:24.224 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-11T19:21:24.224 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-11T19:21:24.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.533+0000 7efdcc7d7640 1 -- 172.21.15.29:0/1448055275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdc4101c30 msgr2=0x7efdc41084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:24.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.533+0000 7efdcc7d7640 1 --2- 172.21.15.29:0/1448055275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdc4101c30 0x7efdc41084c0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7efdbc009ca0 tx=0x7efdbc02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.533+0000 7efdcc7d7640 1 -- 172.21.15.29:0/1448055275 shutdown_connections 2024-03-11T19:21:24.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.533+0000 7efdcc7d7640 1 --2- 172.21.15.29:0/1448055275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdc4101c30 0x7efdc41084c0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.533+0000 7efdcc7d7640 1 --2- 172.21.15.29:0/1448055275 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdc4101270 0x7efdc41016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.533+0000 7efdcc7d7640 1 --2- 172.21.15.29:0/1448055275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdc4100080 0x7efdc41004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.533+0000 7efdcc7d7640 1 -- 172.21.15.29:0/1448055275 >> 172.21.15.29:0/1448055275 conn(0x7efdc40fb5f0 msgr2=0x7efdc40fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:24.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.533+0000 7efdcc7d7640 1 -- 172.21.15.29:0/1448055275 shutdown_connections 2024-03-11T19:21:24.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.533+0000 7efdcc7d7640 1 -- 172.21.15.29:0/1448055275 wait complete. 2024-03-11T19:21:24.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.534+0000 7efdcc7d7640 1 Processor -- start 2024-03-11T19:21:24.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.534+0000 7efdcc7d7640 1 -- start start 2024-03-11T19:21:24.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.534+0000 7efdcc7d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdc4100080 0x7efdc41970c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.534+0000 7efdcc7d7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdc4101270 0x7efdc4197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.534+0000 7efdcc7d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdc4101c30 0x7efdc419e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.534+0000 7efdcc7d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc419ebb0 con 0x7efdc4100080 2024-03-11T19:21:24.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.534+0000 7efdcc7d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc419ed20 con 0x7efdc4101c30 2024-03-11T19:21:24.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.534+0000 7efdcc7d7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc419eee0 con 0x7efdc4101270 2024-03-11T19:21:24.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.534+0000 7efdca573640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdc4100080 0x7efdc41970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.534+0000 7efdca573640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdc4100080 0x7efdc41970c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44684/0 (socket says 172.21.15.29:44684) 2024-03-11T19:21:24.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.534+0000 7efdca573640 1 -- 172.21.15.29:0/2386674243 learned_addr learned my addr 172.21.15.29:0/2386674243 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:24.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.535+0000 7efdc9d72640 1 --2- 172.21.15.29:0/2386674243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdc4101270 0x7efdc4197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.535+0000 7efdcad74640 1 --2- 172.21.15.29:0/2386674243 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdc4101c30 0x7efdc419e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.535+0000 7efdca573640 1 -- 172.21.15.29:0/2386674243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdc4101270 msgr2=0x7efdc4197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:24.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.535+0000 7efdca573640 1 --2- 172.21.15.29:0/2386674243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdc4101270 0x7efdc4197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.535+0000 7efdca573640 1 -- 172.21.15.29:0/2386674243 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdc4101c30 msgr2=0x7efdc419e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:24.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.535+0000 7efdca573640 1 --2- 172.21.15.29:0/2386674243 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdc4101c30 0x7efdc419e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.535+0000 7efdca573640 1 -- 172.21.15.29:0/2386674243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdbc009900 con 0x7efdc4100080 2024-03-11T19:21:24.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.535+0000 7efdc9d72640 1 --2- 172.21.15.29:0/2386674243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdc4101270 0x7efdc4197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:21:24.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.535+0000 7efdca573640 1 --2- 172.21.15.29:0/2386674243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdc4100080 0x7efdc41970c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7efdc000c970 tx=0x7efdc000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:24.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.535+0000 7efdbb7fe640 1 -- 172.21.15.29:0/2386674243 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0004220 con 0x7efdc4100080 2024-03-11T19:21:24.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.535+0000 7efdcc7d7640 1 -- 172.21.15.29:0/2386674243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc419f1c0 con 0x7efdc4100080 2024-03-11T19:21:24.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.535+0000 7efdbb7fe640 1 -- 172.21.15.29:0/2386674243 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efdc0007e60 con 0x7efdc4100080 2024-03-11T19:21:24.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.535+0000 7efdbb7fe640 1 -- 172.21.15.29:0/2386674243 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0003e40 con 0x7efdc4100080 2024-03-11T19:21:24.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.535+0000 7efdcc7d7640 1 -- 172.21.15.29:0/2386674243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc419f6d0 con 0x7efdc4100080 2024-03-11T19:21:24.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.538+0000 7efdbb7fe640 1 -- 172.21.15.29:0/2386674243 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7efdc0003610 con 0x7efdc4100080 2024-03-11T19:21:24.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.538+0000 7efdcf3ab740 1 Processor -- start 2024-03-11T19:21:24.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.538+0000 7efdcf3ab740 1 -- start start 2024-03-11T19:21:24.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.538+0000 7efdcf3ab740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edaf2d0 0x55948edaf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.538+0000 7efdcf3ab740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edafcc0 0x55948edb47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.538+0000 7efdcf3ab740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edb4e30 0x55948edb72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.538+0000 7efdcf3ab740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55948ed1f280 con 0x55948edb4e30 2024-03-11T19:21:24.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.538+0000 7efdcf3ab740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55948ed1c7b0 con 0x55948edaf2d0 2024-03-11T19:21:24.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.538+0000 7efdcf3ab740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55948ed17f90 con 0x55948edafcc0 2024-03-11T19:21:24.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.538+0000 7efdbb7fe640 1 --2- 172.21.15.29:0/2386674243 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7efdb00a1a30 0x7efdb00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.539+0000 7efdc9d72640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edb4e30 0x55948edb72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.539+0000 7efdc9d72640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edb4e30 0x55948edb72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44698/0 (socket says 172.21.15.29:44698) 2024-03-11T19:21:24.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.539+0000 7efdc9d72640 1 -- 172.21.15.29:0/1231651599 learned_addr learned my addr 172.21.15.29:0/1231651599 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:24.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.539+0000 7efdcad74640 1 --2- 172.21.15.29:0/1231651599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edaf2d0 0x55948edaf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.539+0000 7efdcad74640 1 --2- 172.21.15.29:0/2386674243 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7efdb00a1a30 0x7efdb00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.539+0000 7efdbb7fe640 1 -- 172.21.15.29:0/2386674243 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7efdc00cd1b0 con 0x7efdc4100080 2024-03-11T19:21:24.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.539+0000 7efdca573640 1 --2- 172.21.15.29:0/1231651599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edafcc0 0x55948edb47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.539+0000 7efdc9d72640 1 -- 172.21.15.29:0/1231651599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edafcc0 msgr2=0x55948edb47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:24.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.539+0000 7efdc9d72640 1 --2- 172.21.15.29:0/1231651599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edafcc0 0x55948edb47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.539+0000 7efdc9d72640 1 -- 172.21.15.29:0/1231651599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edaf2d0 msgr2=0x55948edaf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:24.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.539+0000 7efdc9d72640 1 --2- 172.21.15.29:0/1231651599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edaf2d0 0x55948edaf6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.539+0000 7efdc9d72640 1 -- 172.21.15.29:0/1231651599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55948ecd6570 con 0x55948edb4e30 2024-03-11T19:21:24.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.539+0000 7efdc9d72640 1 --2- 172.21.15.29:0/1231651599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edb4e30 0x55948edb72c0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7efdb4009cf0 tx=0x7efdb40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad38460bde1f2257 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:24.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.539+0000 7efdcad74640 1 --2- 172.21.15.29:0/2386674243 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7efdb00a1a30 0x7efdb00a3ef0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7efdbc006010 tx=0x7efdbc0095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:24.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.539+0000 7efdb8ff9640 1 -- 172.21.15.29:0/1231651599 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb4012450 con 0x55948edb4e30 2024-03-11T19:21:24.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.539+0000 7efdb8ff9640 1 -- 172.21.15.29:0/1231651599 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efdb4012a60 con 0x55948edb4e30 2024-03-11T19:21:24.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.539+0000 7efdb8ff9640 1 -- 172.21.15.29:0/1231651599 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb400fb50 con 0x55948edb4e30 2024-03-11T19:21:24.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.540+0000 7efdcf3ab740 1 -- 172.21.15.29:0/1231651599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edb4e30 msgr2=0x55948edb72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:24.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.540+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/1231651599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edb4e30 0x55948edb72c0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7efdb4009cf0 tx=0x7efdb40117b0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.540+0000 7efdcf3ab740 1 -- 172.21.15.29:0/1231651599 shutdown_connections 2024-03-11T19:21:24.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.540+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/1231651599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edb4e30 0x55948edb72c0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.540+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/1231651599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edafcc0 0x55948edb47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.540+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/1231651599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edaf2d0 0x55948edaf6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.540+0000 7efdcf3ab740 1 -- 172.21.15.29:0/1231651599 >> 172.21.15.29:0/1231651599 conn(0x55948edaa7f0 msgr2=0x55948edaabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:24.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.540+0000 7efdcf3ab740 1 -- 172.21.15.29:0/1231651599 shutdown_connections 2024-03-11T19:21:24.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.540+0000 7efdcf3ab740 1 -- 172.21.15.29:0/1231651599 wait complete. 2024-03-11T19:21:24.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdcf3ab740 1 -- start start 2024-03-11T19:21:24.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdcf3ab740 1 Processor -- start 2024-03-11T19:21:24.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdcf3ab740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 0x55948edb9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdcf3ab740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edafcc0 0x55948edb9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdcf3ab740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edb4e30 0x55948edc07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdcf3ab740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55948e9eb430 con 0x55948edaf2d0 2024-03-11T19:21:24.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdcf3ab740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55948e9a7e00 con 0x55948edafcc0 2024-03-11T19:21:24.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdcf3ab740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55948ec9cfd0 con 0x55948edb4e30 2024-03-11T19:21:24.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdcf3ab740 10 client.0 _ll_register_callbacks cb 0x55948ecf2fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:21:24.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdca573640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 0x55948edb9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdca573640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 0x55948edb9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44700/0 (socket says 172.21.15.29:44700) 2024-03-11T19:21:24.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdca573640 1 -- 172.21.15.29:0/3208286820 learned_addr learned my addr 172.21.15.29:0/3208286820 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:24.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdc9d72640 1 --2- 172.21.15.29:0/3208286820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edafcc0 0x55948edb9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdcad74640 1 --2- 172.21.15.29:0/3208286820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edb4e30 0x55948edc07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdca573640 1 -- 172.21.15.29:0/3208286820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edb4e30 msgr2=0x55948edc07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:24.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdca573640 1 --2- 172.21.15.29:0/3208286820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edb4e30 0x55948edc07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdca573640 1 -- 172.21.15.29:0/3208286820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edafcc0 msgr2=0x55948edb9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:24.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdca573640 1 --2- 172.21.15.29:0/3208286820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edafcc0 0x55948edb9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdca573640 1 -- 172.21.15.29:0/3208286820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb4009920 con 0x55948edaf2d0 2024-03-11T19:21:24.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.541+0000 7efdc9d72640 1 --2- 172.21.15.29:0/3208286820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edafcc0 0x55948edb9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:21:24.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.542+0000 7efdca573640 1 --2- 172.21.15.29:0/3208286820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 0x55948edb9240 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7efdc0096580 tx=0x7efdc009a960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d52507df8fbf103a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:24.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.542+0000 7efdadffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:21:24.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.542+0000 7efdadffb640 1 -- 172.21.15.29:0/3208286820 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc00a28c0 con 0x55948edaf2d0 2024-03-11T19:21:24.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.542+0000 7efdadffb640 1 -- 172.21.15.29:0/3208286820 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7efdc00cadc0 con 0x55948edaf2d0 2024-03-11T19:21:24.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.542+0000 7efdcf3ab740 1 -- 172.21.15.29:0/3208286820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55948ea177e0 con 0x55948edaf2d0 2024-03-11T19:21:24.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.542+0000 7efdadffb640 1 -- 172.21.15.29:0/3208286820 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdc00079b0 con 0x55948edaf2d0 2024-03-11T19:21:24.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.542+0000 7efdcf3ab740 10 client.18081 fetch_fsmap learned FSMap version 27 2024-03-11T19:21:24.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.542+0000 7efdcf3ab740 1 -- 172.21.15.29:0/3208286820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55948ed16610 con 0x55948edaf2d0 2024-03-11T19:21:24.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.542+0000 7efdadffb640 1 -- 172.21.15.29:0/3208286820 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55948ea177e0 con 0x55948edaf2d0 2024-03-11T19:21:24.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.542+0000 7efdadffb640 1 -- 172.21.15.29:0/3208286820 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efdc0016030 con 0x55948edaf2d0 2024-03-11T19:21:24.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.542+0000 7efdcf3ab740 10 client.18081 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:21:24.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.542+0000 7efdcf3ab740 10 client.18081 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:21:24.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.542+0000 7efdcf3ab740 20 client.18081 populate_metadata read hostname 'smithi029' 2024-03-11T19:21:24.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.542+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/3208286820 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55948edc5360 0x55948edc7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.542+0000 7efdcf3ab740 4 client.18081 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:21:24.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.542+0000 7efdcf3ab740 1 -- 172.21.15.29:0/3208286820 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55948ea54720 con 0x55948edc5360 2024-03-11T19:21:24.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.543+0000 7efdc9d72640 1 --2- 172.21.15.29:0/3208286820 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55948edc5360 0x55948edc7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.543+0000 7efdadffb640 10 client.18081 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:21:24.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.543+0000 7efdc9d72640 1 --2- 172.21.15.29:0/3208286820 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55948edc5360 0x55948edc7770 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36023138bbd3202d server_cookie=69ea3e647ba52b70 in_seq=0 out_seq=0 2024-03-11T19:21:24.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.544+0000 7efdadffb640 1 -- 172.21.15.29:0/3208286820 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55948ea54720 con 0x55948edc5360 2024-03-11T19:21:24.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.544+0000 7efdadffb640 10 client.18081 handle_command_reply: tid=0 2024-03-11T19:21:24.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.544+0000 7efdcf3ab740 1 client.18081 shutdown 2024-03-11T19:21:24.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.544+0000 7efdcf3ab740 1 -- 172.21.15.29:0/3208286820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 msgr2=0x55948edb9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:24.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.544+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/3208286820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 0x55948edb9240 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7efdc0096580 tx=0x7efdc009a960 comp rx=0 tx=0).stop 2024-03-11T19:21:24.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.544+0000 7efdcf3ab740 1 -- 172.21.15.29:0/3208286820 shutdown_connections 2024-03-11T19:21:24.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.544+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/3208286820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edb4e30 0x55948edc07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.544+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/3208286820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edafcc0 0x55948edb9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.544+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/3208286820 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55948edc5360 0x55948edc7770 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.544+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/3208286820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 0x55948edb9240 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.544+0000 7efdadffb640 0 client.18081 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:21:24.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.544+0000 7efdcf3ab740 1 -- 172.21.15.29:0/3208286820 >> 172.21.15.29:0/3208286820 conn(0x55948edaa7f0 msgr2=0x55948eda9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:24.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.544+0000 7efdcf3ab740 1 -- 172.21.15.29:0/3208286820 shutdown_connections 2024-03-11T19:21:24.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.544+0000 7efdcf3ab740 1 -- 172.21.15.29:0/3208286820 wait complete. 2024-03-11T19:21:24.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.544+0000 7efdcf3ab740 20 client.18081 trim_cache size 0 max 16384 2024-03-11T19:21:24.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.560+0000 7efdcf3ab740 1 Processor -- start 2024-03-11T19:21:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.560+0000 7efdcf3ab740 1 -- start start 2024-03-11T19:21:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.560+0000 7efdcf3ab740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 0x55948edb0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.560+0000 7efdcf3ab740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edafcc0 0x55948edb1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.560+0000 7efdcf3ab740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edb4e30 0x55948edc0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.560+0000 7efdcf3ab740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55948ea54720 con 0x55948edaf2d0 2024-03-11T19:21:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.560+0000 7efdcf3ab740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55948ea177e0 con 0x55948edafcc0 2024-03-11T19:21:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.560+0000 7efdcf3ab740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55948edb7780 con 0x55948edb4e30 2024-03-11T19:21:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.560+0000 7efdca573640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 0x55948edb0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.560+0000 7efdca573640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 0x55948edb0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44716/0 (socket says 172.21.15.29:44716) 2024-03-11T19:21:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.560+0000 7efdca573640 1 -- 172.21.15.29:0/4265114567 learned_addr learned my addr 172.21.15.29:0/4265114567 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.561+0000 7efdc9d72640 1 --2- 172.21.15.29:0/4265114567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edafcc0 0x55948edb1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.561+0000 7efdcad74640 1 --2- 172.21.15.29:0/4265114567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edb4e30 0x55948edc0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.561+0000 7efdca573640 1 -- 172.21.15.29:0/4265114567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edb4e30 msgr2=0x55948edc0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:24.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.561+0000 7efdca573640 1 --2- 172.21.15.29:0/4265114567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edb4e30 0x55948edc0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.561+0000 7efdca573640 1 -- 172.21.15.29:0/4265114567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edafcc0 msgr2=0x55948edb1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:24.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.561+0000 7efdca573640 1 --2- 172.21.15.29:0/4265114567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edafcc0 0x55948edb1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.561+0000 7efdca573640 1 -- 172.21.15.29:0/4265114567 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55948ecb2ac0 con 0x55948edaf2d0 2024-03-11T19:21:24.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.561+0000 7efdc9d72640 1 --2- 172.21.15.29:0/4265114567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edafcc0 0x55948edb1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:21:24.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.561+0000 7efdca573640 1 --2- 172.21.15.29:0/4265114567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 0x55948edb0f40 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7efdc009a0b0 tx=0x7efdc00038e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9ad4345b584bdd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:24.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.561+0000 7efdaffff640 1 -- 172.21.15.29:0/4265114567 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0095e70 con 0x55948edaf2d0 2024-03-11T19:21:24.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.561+0000 7efdaffff640 1 -- 172.21.15.29:0/4265114567 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efdc00a28c0 con 0x55948edaf2d0 2024-03-11T19:21:24.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.561+0000 7efdaffff640 1 -- 172.21.15.29:0/4265114567 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc00992e0 con 0x55948edaf2d0 2024-03-11T19:21:24.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.562+0000 7efdcf3ab740 1 -- 172.21.15.29:0/4265114567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 msgr2=0x55948edb0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:24.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.562+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/4265114567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 0x55948edb0f40 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7efdc009a0b0 tx=0x7efdc00038e0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.562+0000 7efdcf3ab740 1 -- 172.21.15.29:0/4265114567 shutdown_connections 2024-03-11T19:21:24.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.562+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/4265114567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edb4e30 0x55948edc0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.562+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/4265114567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edafcc0 0x55948edb1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.562+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/4265114567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 0x55948edb0f40 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.562+0000 7efdcf3ab740 1 -- 172.21.15.29:0/4265114567 >> 172.21.15.29:0/4265114567 conn(0x55948edaa7f0 msgr2=0x55948edcc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:24.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.562+0000 7efdcf3ab740 1 -- 172.21.15.29:0/4265114567 shutdown_connections 2024-03-11T19:21:24.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.562+0000 7efdcf3ab740 1 -- 172.21.15.29:0/4265114567 wait complete. 2024-03-11T19:21:24.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.562+0000 7efdcf3ab740 1 -- start start 2024-03-11T19:21:24.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.562+0000 7efdcf3ab740 1 Processor -- start 2024-03-11T19:21:24.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.562+0000 7efdcf3ab740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 0x55948edbe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.562+0000 7efdcf3ab740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edafcc0 0x55948edbee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.562+0000 7efdcf3ab740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edb4e30 0x55948edbfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.562+0000 7efdcf3ab740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55948edc02c0 con 0x55948edaf2d0 2024-03-11T19:21:24.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.562+0000 7efdcf3ab740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55948edc0430 con 0x55948edafcc0 2024-03-11T19:21:24.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.562+0000 7efdcf3ab740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55948edc05f0 con 0x55948edb4e30 2024-03-11T19:21:24.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.563+0000 7efdcf3ab740 10 client.0 _ll_register_callbacks cb 0x7efdb4009690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:21:24.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.563+0000 7efdca573640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 0x55948edbe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.563+0000 7efdca573640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 0x55948edbe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44722/0 (socket says 172.21.15.29:44722) 2024-03-11T19:21:24.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.563+0000 7efdca573640 1 -- 172.21.15.29:0/2856043013 learned_addr learned my addr 172.21.15.29:0/2856043013 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:24.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.563+0000 7efdc9d72640 1 --2- 172.21.15.29:0/2856043013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edafcc0 0x55948edbee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.563+0000 7efdca573640 1 -- 172.21.15.29:0/2856043013 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edb4e30 msgr2=0x55948edbfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:24.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.563+0000 7efdcad74640 1 --2- 172.21.15.29:0/2856043013 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edb4e30 0x55948edbfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.563+0000 7efdca573640 1 --2- 172.21.15.29:0/2856043013 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edb4e30 0x55948edbfd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.563+0000 7efdca573640 1 -- 172.21.15.29:0/2856043013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edafcc0 msgr2=0x55948edbee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:24.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.563+0000 7efdca573640 1 --2- 172.21.15.29:0/2856043013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edafcc0 0x55948edbee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.563+0000 7efdca573640 1 -- 172.21.15.29:0/2856043013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55948ed16610 con 0x55948edaf2d0 2024-03-11T19:21:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.563+0000 7efdc9d72640 1 --2- 172.21.15.29:0/2856043013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edafcc0 0x55948edbee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:21:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.563+0000 7efdca573640 1 --2- 172.21.15.29:0/2856043013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 0x55948edbe940 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7efdc0099b20 tx=0x7efdc0007be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d892546b7b87b79 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.563+0000 7efdadffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:21:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.563+0000 7efdadffb640 1 -- 172.21.15.29:0/2856043013 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0095e70 con 0x55948edaf2d0 2024-03-11T19:21:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.563+0000 7efdadffb640 1 -- 172.21.15.29:0/2856043013 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7efdc00b8020 con 0x55948edaf2d0 2024-03-11T19:21:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.563+0000 7efdcf3ab740 1 -- 172.21.15.29:0/2856043013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efdb4012a60 con 0x55948edaf2d0 2024-03-11T19:21:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.563+0000 7efdadffb640 1 -- 172.21.15.29:0/2856043013 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdc00963c0 con 0x55948edaf2d0 2024-03-11T19:21:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.564+0000 7efdcf3ab740 10 client.18087 fetch_fsmap learned FSMap version 27 2024-03-11T19:21:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.564+0000 7efdcf3ab740 1 -- 172.21.15.29:0/2856043013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55948ecd86a0 con 0x55948edaf2d0 2024-03-11T19:21:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.564+0000 7efdadffb640 1 -- 172.21.15.29:0/2856043013 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efdb4012a60 con 0x55948edaf2d0 2024-03-11T19:21:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.564+0000 7efdadffb640 1 -- 172.21.15.29:0/2856043013 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efdc0016040 con 0x55948edaf2d0 2024-03-11T19:21:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.564+0000 7efdcf3ab740 10 client.18087 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:21:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.564+0000 7efdcf3ab740 10 client.18087 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:21:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.564+0000 7efdcf3ab740 20 client.18087 populate_metadata read hostname 'smithi029' 2024-03-11T19:21:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.564+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/2856043013 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55948ee1c760 0x55948ee1eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.564+0000 7efdcf3ab740 4 client.18087 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-11T19:21:24.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.564+0000 7efdcf3ab740 1 -- 172.21.15.29:0/2856043013 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55948eaa3890 con 0x55948ee1c760 2024-03-11T19:21:24.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.564+0000 7efdc9d72640 1 --2- 172.21.15.29:0/2856043013 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55948ee1c760 0x55948ee1eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.565+0000 7efdadffb640 10 client.18087 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:21:24.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.565+0000 7efdc9d72640 1 --2- 172.21.15.29:0/2856043013 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55948ee1c760 0x55948ee1eb80 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a33d7219a14ff8c server_cookie=f524c400acf0f8e2 in_seq=0 out_seq=0 2024-03-11T19:21:24.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.565+0000 7efdadffb640 1 -- 172.21.15.29:0/2856043013 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55948eaa3890 con 0x55948ee1c760 2024-03-11T19:21:24.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.565+0000 7efdadffb640 10 client.18087 handle_command_reply: tid=0 2024-03-11T19:21:24.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.565+0000 7efdcf3ab740 1 client.18087 shutdown 2024-03-11T19:21:24.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.565+0000 7efdcf3ab740 1 -- 172.21.15.29:0/2856043013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 msgr2=0x55948edbe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:24.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.565+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/2856043013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 0x55948edbe940 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7efdc0099b20 tx=0x7efdc0007be0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.565+0000 7efdcf3ab740 1 -- 172.21.15.29:0/2856043013 shutdown_connections 2024-03-11T19:21:24.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.565+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/2856043013 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55948edb4e30 0x55948edbfd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.565+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/2856043013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55948edafcc0 0x55948edbee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.565+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/2856043013 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x55948ee1c760 0x55948ee1eb80 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.565+0000 7efdcf3ab740 1 --2- 172.21.15.29:0/2856043013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55948edaf2d0 0x55948edbe940 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.565+0000 7efdadffb640 0 client.18087 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:21:24.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.565+0000 7efdcf3ab740 1 -- 172.21.15.29:0/2856043013 >> 172.21.15.29:0/2856043013 conn(0x55948edaa7f0 msgr2=0x55948edb0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:24.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.565+0000 7efdcf3ab740 1 -- 172.21.15.29:0/2856043013 shutdown_connections 2024-03-11T19:21:24.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.565+0000 7efdcf3ab740 1 -- 172.21.15.29:0/2856043013 wait complete. 2024-03-11T19:21:24.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.565+0000 7efdcf3ab740 20 client.18087 trim_cache size 0 max 16384 2024-03-11T19:21:24.575 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:21:24.575 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-03-11T19:21:24.575 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-03-11T19:21:24.575 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:21:24.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.566+0000 7efdcc7d7640 1 -- 172.21.15.29:0/2386674243 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7efdb00a1a30 msgr2=0x7efdb00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:24.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.566+0000 7efdcc7d7640 1 --2- 172.21.15.29:0/2386674243 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7efdb00a1a30 0x7efdb00a3ef0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7efdbc006010 tx=0x7efdbc0095c0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.566+0000 7efdcc7d7640 1 -- 172.21.15.29:0/2386674243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdc4100080 msgr2=0x7efdc41970c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:24.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.566+0000 7efdcc7d7640 1 --2- 172.21.15.29:0/2386674243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdc4100080 0x7efdc41970c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7efdc000c970 tx=0x7efdc000ce30 comp rx=0 tx=0).stop 2024-03-11T19:21:24.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.566+0000 7efdcc7d7640 1 -- 172.21.15.29:0/2386674243 shutdown_connections 2024-03-11T19:21:24.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.566+0000 7efdcc7d7640 1 --2- 172.21.15.29:0/2386674243 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdc4101c30 0x7efdc419e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.566+0000 7efdcc7d7640 1 --2- 172.21.15.29:0/2386674243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdc4101270 0x7efdc4197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.566+0000 7efdcc7d7640 1 --2- 172.21.15.29:0/2386674243 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7efdb00a1a30 0x7efdb00a3ef0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.566+0000 7efdcc7d7640 1 --2- 172.21.15.29:0/2386674243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdc4100080 0x7efdc41970c0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.566+0000 7efdcc7d7640 1 -- 172.21.15.29:0/2386674243 >> 172.21.15.29:0/2386674243 conn(0x7efdc40fb5f0 msgr2=0x7efdc40fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:24.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.566+0000 7efdcc7d7640 1 -- 172.21.15.29:0/2386674243 shutdown_connections 2024-03-11T19:21:24.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.566+0000 7efdcc7d7640 1 -- 172.21.15.29:0/2386674243 wait complete. 2024-03-11T19:21:24.582 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-11T19:21:24.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:21:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.688+0000 7f808f2b5640 1 -- 172.21.15.29:0/1321128738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8088101cc0 msgr2=0x7f80881040b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.688+0000 7f808f2b5640 1 --2- 172.21.15.29:0/1321128738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8088101cc0 0x7f80881040b0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f807c009900 tx=0x7f807c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:21:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.688+0000 7f808f2b5640 1 -- 172.21.15.29:0/1321128738 shutdown_connections 2024-03-11T19:21:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.688+0000 7f808f2b5640 1 --2- 172.21.15.29:0/1321128738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8088106f20 0x7f8088109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.688+0000 7f808f2b5640 1 --2- 172.21.15.29:0/1321128738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80881045f0 0x7f80881069e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.688+0000 7f808f2b5640 1 --2- 172.21.15.29:0/1321128738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8088101cc0 0x7f80881040b0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.688+0000 7f808f2b5640 1 -- 172.21.15.29:0/1321128738 >> 172.21.15.29:0/1321128738 conn(0x7f80880fb630 msgr2=0x7f80880fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.688+0000 7f808f2b5640 1 -- 172.21.15.29:0/1321128738 shutdown_connections 2024-03-11T19:21:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.688+0000 7f808f2b5640 1 -- 172.21.15.29:0/1321128738 wait complete. 2024-03-11T19:21:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.688+0000 7f808f2b5640 1 Processor -- start 2024-03-11T19:21:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808f2b5640 1 -- start start 2024-03-11T19:21:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808f2b5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8088101cc0 0x7f8088197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808f2b5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80881045f0 0x7f8088197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808f2b5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8088106f20 0x7f808819e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808f2b5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808819ec20 con 0x7f8088101cc0 2024-03-11T19:21:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808f2b5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808819ed90 con 0x7f80881045f0 2024-03-11T19:21:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808f2b5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808819ef50 con 0x7f8088106f20 2024-03-11T19:21:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808d051640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8088101cc0 0x7f8088197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808d051640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8088101cc0 0x7f8088197130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44734/0 (socket says 172.21.15.29:44734) 2024-03-11T19:21:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808d051640 1 -- 172.21.15.29:0/3110351722 learned_addr learned my addr 172.21.15.29:0/3110351722 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808d852640 1 --2- 172.21.15.29:0/3110351722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8088106f20 0x7f808819e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808c850640 1 --2- 172.21.15.29:0/3110351722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80881045f0 0x7f8088197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808d051640 1 -- 172.21.15.29:0/3110351722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8088106f20 msgr2=0x7f808819e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808d051640 1 --2- 172.21.15.29:0/3110351722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8088106f20 0x7f808819e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808d051640 1 -- 172.21.15.29:0/3110351722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80881045f0 msgr2=0x7f8088197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808d051640 1 --2- 172.21.15.29:0/3110351722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80881045f0 0x7f8088197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808d051640 1 -- 172.21.15.29:0/3110351722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c009560 con 0x7f8088101cc0 2024-03-11T19:21:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808c850640 1 --2- 172.21.15.29:0/3110351722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80881045f0 0x7f8088197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:21:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808d852640 1 --2- 172.21.15.29:0/3110351722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8088106f20 0x7f808819e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:21:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808d051640 1 --2- 172.21.15.29:0/3110351722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8088101cc0 0x7f8088197130 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f807c005ab0 tx=0x7f807c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f807a7fc640 1 -- 172.21.15.29:0/3110351722 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c0044e0 con 0x7f8088101cc0 2024-03-11T19:21:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f808f2b5640 1 -- 172.21.15.29:0/3110351722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f808819f1d0 con 0x7f8088101cc0 2024-03-11T19:21:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.689+0000 7f807a7fc640 1 -- 172.21.15.29:0/3110351722 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f807c003cb0 con 0x7f8088101cc0 2024-03-11T19:21:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.690+0000 7f807a7fc640 1 -- 172.21.15.29:0/3110351722 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c005680 con 0x7f8088101cc0 2024-03-11T19:21:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.690+0000 7f808f2b5640 1 -- 172.21.15.29:0/3110351722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f808819f6e0 con 0x7f8088101cc0 2024-03-11T19:21:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.690+0000 7f808f2b5640 1 -- 172.21.15.29:0/3110351722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80880fd120 con 0x7f8088101cc0 2024-03-11T19:21:24.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.691+0000 7f807a7fc640 1 -- 172.21.15.29:0/3110351722 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f807c004e50 con 0x7f8088101cc0 2024-03-11T19:21:24.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.692+0000 7f807a7fc640 1 --2- 172.21.15.29:0/3110351722 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f80740a1a30 0x7f80740a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:24.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.692+0000 7f808c850640 1 --2- 172.21.15.29:0/3110351722 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f80740a1a30 0x7f80740a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:24.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.692+0000 7f807a7fc640 1 -- 172.21.15.29:0/3110351722 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f807c02de40 con 0x7f8088101cc0 2024-03-11T19:21:24.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.693+0000 7f808c850640 1 --2- 172.21.15.29:0/3110351722 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f80740a1a30 0x7f80740a3ef0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f8088198640 tx=0x7f8084005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:24.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.697+0000 7f807a7fc640 1 -- 172.21.15.29:0/3110351722 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f807c0b90c0 con 0x7f8088101cc0 2024-03-11T19:21:24.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.919+0000 7f808f2b5640 1 -- 172.21.15.29:0/3110351722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f808810cad0 con 0x7f8088101cc0 2024-03-11T19:21:24.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.920+0000 7f807a7fc640 1 -- 172.21.15.29:0/3110351722 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f808810cad0 con 0x7f8088101cc0 2024-03-11T19:21:24.924 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:21:24.924 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:21:24.924 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:21:24.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.922+0000 7f808f2b5640 1 -- 172.21.15.29:0/3110351722 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f80740a1a30 msgr2=0x7f80740a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:24.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.922+0000 7f808f2b5640 1 --2- 172.21.15.29:0/3110351722 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f80740a1a30 0x7f80740a3ef0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f8088198640 tx=0x7f8084005e50 comp rx=0 tx=0).stop 2024-03-11T19:21:24.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.922+0000 7f808f2b5640 1 -- 172.21.15.29:0/3110351722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8088101cc0 msgr2=0x7f8088197130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:24.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.922+0000 7f808f2b5640 1 --2- 172.21.15.29:0/3110351722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8088101cc0 0x7f8088197130 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f807c005ab0 tx=0x7f807c0042f0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.922+0000 7f808f2b5640 1 -- 172.21.15.29:0/3110351722 shutdown_connections 2024-03-11T19:21:24.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.922+0000 7f808f2b5640 1 --2- 172.21.15.29:0/3110351722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8088106f20 0x7f808819e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.922+0000 7f808f2b5640 1 --2- 172.21.15.29:0/3110351722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80881045f0 0x7f8088197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.922+0000 7f808f2b5640 1 --2- 172.21.15.29:0/3110351722 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f80740a1a30 0x7f80740a3ef0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.922+0000 7f808f2b5640 1 --2- 172.21.15.29:0/3110351722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8088101cc0 0x7f8088197130 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:24.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.922+0000 7f808f2b5640 1 -- 172.21.15.29:0/3110351722 >> 172.21.15.29:0/3110351722 conn(0x7f80880fb630 msgr2=0x7f8088102580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:24.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.922+0000 7f808f2b5640 1 -- 172.21.15.29:0/3110351722 shutdown_connections 2024-03-11T19:21:24.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:24.922+0000 7f808f2b5640 1 -- 172.21.15.29:0/3110351722 wait complete. 2024-03-11T19:21:24.937 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:21:24.937 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-11T19:21:25.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.039+0000 7f23a4481640 1 -- 172.21.15.29:0/2557092021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239c100080 msgr2=0x7f239c1004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.039+0000 7f23a4481640 1 --2- 172.21.15.29:0/2557092021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239c100080 0x7f239c1004a0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f2390009900 tx=0x7f239002bc30 comp rx=0 tx=0).stop 2024-03-11T19:21:25.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.039+0000 7f23a4481640 1 -- 172.21.15.29:0/2557092021 shutdown_connections 2024-03-11T19:21:25.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.039+0000 7f23a4481640 1 --2- 172.21.15.29:0/2557092021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f239c101c30 0x7f239c1084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.039+0000 7f23a4481640 1 --2- 172.21.15.29:0/2557092021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f239c101270 0x7f239c1016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.039+0000 7f23a4481640 1 --2- 172.21.15.29:0/2557092021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239c100080 0x7f239c1004a0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.039+0000 7f23a4481640 1 -- 172.21.15.29:0/2557092021 >> 172.21.15.29:0/2557092021 conn(0x7f239c0fb610 msgr2=0x7f239c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.039+0000 7f23a4481640 1 -- 172.21.15.29:0/2557092021 shutdown_connections 2024-03-11T19:21:25.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.039+0000 7f23a4481640 1 -- 172.21.15.29:0/2557092021 wait complete. 2024-03-11T19:21:25.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.039+0000 7f23a4481640 1 Processor -- start 2024-03-11T19:21:25.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.039+0000 7f23a4481640 1 -- start start 2024-03-11T19:21:25.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f23a4481640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239c100080 0x7f239c197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f23a4481640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f239c101270 0x7f239c197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f23a4481640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f239c101c30 0x7f239c19e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f23a4481640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c19ec40 con 0x7f239c100080 2024-03-11T19:21:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f23a4481640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c19edb0 con 0x7f239c101c30 2024-03-11T19:21:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f23a4481640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c19ef70 con 0x7f239c101270 2024-03-11T19:21:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f23a221d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239c100080 0x7f239c197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f23a221d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239c100080 0x7f239c197150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44750/0 (socket says 172.21.15.29:44750) 2024-03-11T19:21:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f23a221d640 1 -- 172.21.15.29:0/2406934927 learned_addr learned my addr 172.21.15.29:0/2406934927 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f23a2a1e640 1 --2- 172.21.15.29:0/2406934927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f239c101c30 0x7f239c19e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f23a1a1c640 1 --2- 172.21.15.29:0/2406934927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f239c101270 0x7f239c197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f23a221d640 1 -- 172.21.15.29:0/2406934927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f239c101270 msgr2=0x7f239c197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f23a221d640 1 --2- 172.21.15.29:0/2406934927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f239c101270 0x7f239c197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f23a221d640 1 -- 172.21.15.29:0/2406934927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f239c101c30 msgr2=0x7f239c19e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f23a221d640 1 --2- 172.21.15.29:0/2406934927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f239c101c30 0x7f239c19e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f23a221d640 1 -- 172.21.15.29:0/2406934927 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2390009560 con 0x7f239c100080 2024-03-11T19:21:25.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f23a1a1c640 1 --2- 172.21.15.29:0/2406934927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f239c101270 0x7f239c197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:21:25.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f23a221d640 1 --2- 172.21.15.29:0/2406934927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239c100080 0x7f239c197150 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f2390000c00 tx=0x7f23900042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f238f7fe640 1 -- 172.21.15.29:0/2406934927 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23900044e0 con 0x7f239c100080 2024-03-11T19:21:25.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f23a4481640 1 -- 172.21.15.29:0/2406934927 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239c19f1f0 con 0x7f239c100080 2024-03-11T19:21:25.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f238f7fe640 1 -- 172.21.15.29:0/2406934927 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f239002de90 con 0x7f239c100080 2024-03-11T19:21:25.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.040+0000 7f238f7fe640 1 -- 172.21.15.29:0/2406934927 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2390005410 con 0x7f239c100080 2024-03-11T19:21:25.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.041+0000 7f23a4481640 1 -- 172.21.15.29:0/2406934927 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f239c19f700 con 0x7f239c100080 2024-03-11T19:21:25.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.043+0000 7f238f7fe640 1 -- 172.21.15.29:0/2406934927 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2390005570 con 0x7f239c100080 2024-03-11T19:21:25.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.043+0000 7f23a704d740 1 Processor -- start 2024-03-11T19:21:25.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.043+0000 7f23a704d740 1 -- start start 2024-03-11T19:21:25.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f23a704d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830c6270 0x55b1830c6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f23a704d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830c6c60 0x55b1830cb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f23a704d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830cbdd0 0x55b1830ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f23a704d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b183082960 con 0x55b1830cbdd0 2024-03-11T19:21:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f23a704d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b183036310 con 0x55b1830c6270 2024-03-11T19:21:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f23a704d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b183033840 con 0x55b1830c6c60 2024-03-11T19:21:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f23a1a1c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830cbdd0 0x55b1830ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f23a1a1c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830cbdd0 0x55b1830ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44762/0 (socket says 172.21.15.29:44762) 2024-03-11T19:21:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f23a1a1c640 1 -- 172.21.15.29:0/1596440551 learned_addr learned my addr 172.21.15.29:0/1596440551 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f238f7fe640 1 --2- 172.21.15.29:0/2406934927 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23880a1a30 0x7f23880a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f23a2a1e640 1 --2- 172.21.15.29:0/1596440551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830c6270 0x55b1830c6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f23a221d640 1 --2- 172.21.15.29:0/1596440551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830c6c60 0x55b1830cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f23a2a1e640 1 --2- 172.21.15.29:0/2406934927 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23880a1a30 0x7f23880a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f23a1a1c640 1 -- 172.21.15.29:0/1596440551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830c6c60 msgr2=0x55b1830cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f23a1a1c640 1 --2- 172.21.15.29:0/1596440551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830c6c60 0x55b1830cb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f238f7fe640 1 -- 172.21.15.29:0/2406934927 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f23900ef700 con 0x7f239c100080 2024-03-11T19:21:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f23a1a1c640 1 -- 172.21.15.29:0/1596440551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830c6270 msgr2=0x55b1830c6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f23a1a1c640 1 --2- 172.21.15.29:0/1596440551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830c6270 0x55b1830c6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f23a1a1c640 1 -- 172.21.15.29:0/1596440551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b182ff1060 con 0x55b1830cbdd0 2024-03-11T19:21:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f23a1a1c640 1 --2- 172.21.15.29:0/1596440551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830cbdd0 0x55b1830ce260 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f2398009c20 tx=0x7f239800e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e478cfd3df6e56 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.044+0000 7f23a2a1e640 1 --2- 172.21.15.29:0/2406934927 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23880a1a30 0x7f23880a3ef0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x55b1830c74c0 tx=0x7f2394004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.045+0000 7f238cbfe640 1 -- 172.21.15.29:0/1596440551 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398011470 con 0x55b1830cbdd0 2024-03-11T19:21:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.045+0000 7f238cbfe640 1 -- 172.21.15.29:0/1596440551 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2398011a80 con 0x55b1830cbdd0 2024-03-11T19:21:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.045+0000 7f238cbfe640 1 -- 172.21.15.29:0/1596440551 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398010920 con 0x55b1830cbdd0 2024-03-11T19:21:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.045+0000 7f23a704d740 1 -- 172.21.15.29:0/1596440551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830cbdd0 msgr2=0x55b1830ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.045+0000 7f23a704d740 1 --2- 172.21.15.29:0/1596440551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830cbdd0 0x55b1830ce260 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f2398009c20 tx=0x7f239800e6c0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.045+0000 7f23a704d740 1 -- 172.21.15.29:0/1596440551 shutdown_connections 2024-03-11T19:21:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.045+0000 7f23a704d740 1 --2- 172.21.15.29:0/1596440551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830cbdd0 0x55b1830ce260 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.045+0000 7f23a704d740 1 --2- 172.21.15.29:0/1596440551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830c6c60 0x55b1830cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.045+0000 7f23a704d740 1 --2- 172.21.15.29:0/1596440551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830c6270 0x55b1830c6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.045+0000 7f23a704d740 1 -- 172.21.15.29:0/1596440551 >> 172.21.15.29:0/1596440551 conn(0x55b1830c17e0 msgr2=0x55b1830c1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.046+0000 7f23a704d740 1 -- 172.21.15.29:0/1596440551 shutdown_connections 2024-03-11T19:21:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.046+0000 7f23a704d740 1 -- 172.21.15.29:0/1596440551 wait complete. 2024-03-11T19:21:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.046+0000 7f23a704d740 1 -- start start 2024-03-11T19:21:25.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.046+0000 7f23a704d740 1 Processor -- start 2024-03-11T19:21:25.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.046+0000 7f23a704d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6270 0x55b1830d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.046+0000 7f23a704d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830c6c60 0x55b1830d0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.046+0000 7f23a704d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830cbdd0 0x55b1830d79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.046+0000 7f23a704d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b18302f020 con 0x55b1830c6270 2024-03-11T19:21:25.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.046+0000 7f23a704d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b182d02430 con 0x55b1830cbdd0 2024-03-11T19:21:25.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.046+0000 7f23a704d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b182cbee00 con 0x55b1830c6c60 2024-03-11T19:21:25.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.046+0000 7f23a704d740 10 client.0 _ll_register_callbacks cb 0x55b182fd6cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:21:25.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.046+0000 7f23a221d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6270 0x55b1830d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.046+0000 7f23a221d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6270 0x55b1830d0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44776/0 (socket says 172.21.15.29:44776) 2024-03-11T19:21:25.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.046+0000 7f23a221d640 1 -- 172.21.15.29:0/3514905484 learned_addr learned my addr 172.21.15.29:0/3514905484 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.046+0000 7f23a1a1c640 1 --2- 172.21.15.29:0/3514905484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830c6c60 0x55b1830d0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f23a2a1e640 1 --2- 172.21.15.29:0/3514905484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830cbdd0 0x55b1830d79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f23a221d640 1 -- 172.21.15.29:0/3514905484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830c6c60 msgr2=0x55b1830d0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f23a221d640 1 --2- 172.21.15.29:0/3514905484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830c6c60 0x55b1830d0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f23a221d640 1 -- 172.21.15.29:0/3514905484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830cbdd0 msgr2=0x55b1830d79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f23a221d640 1 --2- 172.21.15.29:0/3514905484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830cbdd0 0x55b1830d79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f23a221d640 1 -- 172.21.15.29:0/3514905484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2398009880 con 0x55b1830c6270 2024-03-11T19:21:25.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f23a2a1e640 1 --2- 172.21.15.29:0/3514905484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830cbdd0 0x55b1830d79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:21:25.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f23a221d640 1 --2- 172.21.15.29:0/3514905484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6270 0x55b1830d0430 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f2390005820 tx=0x7f239008c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=836046863ded4709 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f2385ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:21:25.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f2385ffb640 1 -- 172.21.15.29:0/3514905484 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2390004dc0 con 0x55b1830c6270 2024-03-11T19:21:25.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f2385ffb640 1 -- 172.21.15.29:0/3514905484 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f23900a5e40 con 0x55b1830c6270 2024-03-11T19:21:25.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f23a704d740 1 -- 172.21.15.29:0/3514905484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b182d2e7e0 con 0x55b1830c6270 2024-03-11T19:21:25.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f2385ffb640 1 -- 172.21.15.29:0/3514905484 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f239008b020 con 0x55b1830c6270 2024-03-11T19:21:25.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f23a704d740 10 client.18105 fetch_fsmap learned FSMap version 27 2024-03-11T19:21:25.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f23a704d740 1 -- 172.21.15.29:0/3514905484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55b18302d6a0 con 0x55b1830c6270 2024-03-11T19:21:25.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f2385ffb640 1 -- 172.21.15.29:0/3514905484 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55b182d2e7e0 con 0x55b1830c6270 2024-03-11T19:21:25.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f2385ffb640 1 -- 172.21.15.29:0/3514905484 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f239003c070 con 0x55b1830c6270 2024-03-11T19:21:25.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f23a704d740 10 client.18105 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:21:25.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f23a704d740 10 client.18105 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:21:25.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f23a704d740 20 client.18105 populate_metadata read hostname 'smithi029' 2024-03-11T19:21:25.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.047+0000 7f23a704d740 1 --2- 172.21.15.29:0/3514905484 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55b1830dc310 0x55b1830de700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.048+0000 7f23a704d740 4 client.18105 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:21:25.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.048+0000 7f23a704d740 1 -- 172.21.15.29:0/3514905484 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b182d6b720 con 0x55b1830dc310 2024-03-11T19:21:25.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.048+0000 7f23a1a1c640 1 --2- 172.21.15.29:0/3514905484 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55b1830dc310 0x55b1830de700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.048+0000 7f2385ffb640 10 client.18105 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:21:25.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.048+0000 7f23a1a1c640 1 --2- 172.21.15.29:0/3514905484 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55b1830dc310 0x55b1830de700 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=276d2b1fc2a04158 server_cookie=52155dfd2946fced in_seq=0 out_seq=0 2024-03-11T19:21:25.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.049+0000 7f2385ffb640 1 -- 172.21.15.29:0/3514905484 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b182d6b720 con 0x55b1830dc310 2024-03-11T19:21:25.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.049+0000 7f2385ffb640 10 client.18105 handle_command_reply: tid=0 2024-03-11T19:21:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.049+0000 7f23a704d740 1 client.18105 shutdown 2024-03-11T19:21:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.049+0000 7f23a704d740 1 -- 172.21.15.29:0/3514905484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6270 msgr2=0x55b1830d0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.049+0000 7f23a704d740 1 --2- 172.21.15.29:0/3514905484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6270 0x55b1830d0430 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f2390005820 tx=0x7f239008c760 comp rx=0 tx=0).stop 2024-03-11T19:21:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.049+0000 7f23a704d740 1 -- 172.21.15.29:0/3514905484 shutdown_connections 2024-03-11T19:21:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.049+0000 7f23a704d740 1 --2- 172.21.15.29:0/3514905484 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55b1830dc310 0x55b1830de700 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.049+0000 7f23a704d740 1 --2- 172.21.15.29:0/3514905484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830cbdd0 0x55b1830d79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.049+0000 7f23a704d740 1 --2- 172.21.15.29:0/3514905484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830c6c60 0x55b1830d0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.049+0000 7f2385ffb640 0 client.18105 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:21:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.049+0000 7f23a704d740 1 --2- 172.21.15.29:0/3514905484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6270 0x55b1830d0430 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.049+0000 7f23a704d740 1 -- 172.21.15.29:0/3514905484 >> 172.21.15.29:0/3514905484 conn(0x55b1830c17e0 msgr2=0x55b1830c0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.049+0000 7f23a704d740 1 -- 172.21.15.29:0/3514905484 shutdown_connections 2024-03-11T19:21:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.049+0000 7f23a704d740 1 -- 172.21.15.29:0/3514905484 wait complete. 2024-03-11T19:21:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.049+0000 7f23a704d740 20 client.18105 trim_cache size 0 max 16384 2024-03-11T19:21:25.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.061+0000 7f23a704d740 1 Processor -- start 2024-03-11T19:21:25.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.061+0000 7f23a704d740 1 -- start start 2024-03-11T19:21:25.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.061+0000 7f23a704d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6270 0x55b1830d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.061+0000 7f23a704d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830c6c60 0x55b1830d7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.061+0000 7f23a704d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830cbdd0 0x55b1830d0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.061+0000 7f23a704d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b182fb4130 con 0x55b1830c6270 2024-03-11T19:21:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.061+0000 7f23a704d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1830d1620 con 0x55b1830c6c60 2024-03-11T19:21:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.061+0000 7f23a704d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1830d1460 con 0x55b1830cbdd0 2024-03-11T19:21:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.061+0000 7f23a221d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6270 0x55b1830d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.061+0000 7f23a221d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6270 0x55b1830d0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44784/0 (socket says 172.21.15.29:44784) 2024-03-11T19:21:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.061+0000 7f23a221d640 1 -- 172.21.15.29:0/3030637733 learned_addr learned my addr 172.21.15.29:0/3030637733 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.062+0000 7f23a1a1c640 1 --2- 172.21.15.29:0/3030637733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830c6c60 0x55b1830d7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.062+0000 7f23a2a1e640 1 --2- 172.21.15.29:0/3030637733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830cbdd0 0x55b1830d0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.062+0000 7f23a221d640 1 -- 172.21.15.29:0/3030637733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830cbdd0 msgr2=0x55b1830d0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.062+0000 7f23a221d640 1 --2- 172.21.15.29:0/3030637733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830cbdd0 0x55b1830d0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.062+0000 7f23a221d640 1 -- 172.21.15.29:0/3030637733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830c6c60 msgr2=0x55b1830d7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.062+0000 7f23a221d640 1 --2- 172.21.15.29:0/3030637733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830c6c60 0x55b1830d7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.062+0000 7f23a221d640 1 -- 172.21.15.29:0/3030637733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b182fc9c20 con 0x55b1830c6270 2024-03-11T19:21:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.062+0000 7f23a221d640 1 --2- 172.21.15.29:0/3030637733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6270 0x55b1830d0040 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f239002c7d0 tx=0x7f23900c07c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d454ae78b5dd5bf server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.062+0000 7f2387fff640 1 -- 172.21.15.29:0/3030637733 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2390003e20 con 0x55b1830c6270 2024-03-11T19:21:25.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.062+0000 7f2387fff640 1 -- 172.21.15.29:0/3030637733 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f23900bc440 con 0x55b1830c6270 2024-03-11T19:21:25.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.062+0000 7f2387fff640 1 -- 172.21.15.29:0/3030637733 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23900a72b0 con 0x55b1830c6270 2024-03-11T19:21:25.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a704d740 1 -- 172.21.15.29:0/3030637733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6270 msgr2=0x55b1830d0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a704d740 1 --2- 172.21.15.29:0/3030637733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6270 0x55b1830d0040 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f239002c7d0 tx=0x7f23900c07c0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a704d740 1 -- 172.21.15.29:0/3030637733 shutdown_connections 2024-03-11T19:21:25.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a704d740 1 --2- 172.21.15.29:0/3030637733 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830cbdd0 0x55b1830d0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a704d740 1 --2- 172.21.15.29:0/3030637733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830c6c60 0x55b1830d7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a704d740 1 --2- 172.21.15.29:0/3030637733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6270 0x55b1830d0040 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a704d740 1 -- 172.21.15.29:0/3030637733 >> 172.21.15.29:0/3030637733 conn(0x55b1830c17e0 msgr2=0x55b1830c81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a704d740 1 -- 172.21.15.29:0/3030637733 shutdown_connections 2024-03-11T19:21:25.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a704d740 1 -- 172.21.15.29:0/3030637733 wait complete. 2024-03-11T19:21:25.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a704d740 1 -- start start 2024-03-11T19:21:25.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a704d740 1 Processor -- start 2024-03-11T19:21:25.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a704d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830c6270 0x55b1830d6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a704d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6c60 0x55b1830d66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a704d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830cbdd0 0x55b1830d6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a704d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1830d7d60 con 0x55b1830c6c60 2024-03-11T19:21:25.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a704d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1831327e0 con 0x55b1830c6270 2024-03-11T19:21:25.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a704d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b183132920 con 0x55b1830cbdd0 2024-03-11T19:21:25.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a704d740 10 client.0 _ll_register_callbacks cb 0x7f23980093e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:21:25.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a1a1c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6c60 0x55b1830d66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a1a1c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6c60 0x55b1830d66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44800/0 (socket says 172.21.15.29:44800) 2024-03-11T19:21:25.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a1a1c640 1 -- 172.21.15.29:0/833720897 learned_addr learned my addr 172.21.15.29:0/833720897 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a221d640 1 --2- 172.21.15.29:0/833720897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830c6270 0x55b1830d6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.063+0000 7f23a2a1e640 1 --2- 172.21.15.29:0/833720897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830cbdd0 0x55b1830d6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f23a1a1c640 1 -- 172.21.15.29:0/833720897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830cbdd0 msgr2=0x55b1830d6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f23a1a1c640 1 --2- 172.21.15.29:0/833720897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830cbdd0 0x55b1830d6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f23a1a1c640 1 -- 172.21.15.29:0/833720897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830c6270 msgr2=0x55b1830d6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f23a1a1c640 1 --2- 172.21.15.29:0/833720897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830c6270 0x55b1830d6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f23a1a1c640 1 -- 172.21.15.29:0/833720897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b18302d6a0 con 0x55b1830c6c60 2024-03-11T19:21:25.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f23a221d640 1 --2- 172.21.15.29:0/833720897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830c6270 0x55b1830d6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:21:25.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f23a1a1c640 1 --2- 172.21.15.29:0/833720897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6c60 0x55b1830d66a0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x55b1830c2750 tx=0x7f2398009370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d6a78ab1cc5e83 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f2385ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:21:25.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f2385ffb640 1 -- 172.21.15.29:0/833720897 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398020070 con 0x55b1830c6c60 2024-03-11T19:21:25.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f2385ffb640 1 -- 172.21.15.29:0/833720897 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f2398011880 con 0x55b1830c6c60 2024-03-11T19:21:25.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f23a704d740 1 -- 172.21.15.29:0/833720897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2398011a80 con 0x55b1830c6c60 2024-03-11T19:21:25.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f2385ffb640 1 -- 172.21.15.29:0/833720897 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2398005040 con 0x55b1830c6c60 2024-03-11T19:21:25.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f23a704d740 10 client.18111 fetch_fsmap learned FSMap version 27 2024-03-11T19:21:25.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f23a704d740 1 -- 172.21.15.29:0/833720897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55b182fef8c0 con 0x55b1830c6c60 2024-03-11T19:21:25.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f2385ffb640 1 -- 172.21.15.29:0/833720897 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2398011a80 con 0x55b1830c6c60 2024-03-11T19:21:25.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f2385ffb640 1 -- 172.21.15.29:0/833720897 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f239800c070 con 0x55b1830c6c60 2024-03-11T19:21:25.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f23a704d740 10 client.18111 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:21:25.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f23a704d740 10 client.18111 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-11T19:21:25.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f23a704d740 20 client.18111 populate_metadata read hostname 'smithi029' 2024-03-11T19:21:25.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f23a704d740 1 --2- 172.21.15.29:0/833720897 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55b1831341f0 0x55b1831365e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f23a704d740 4 client.18111 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:21:25.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.064+0000 7f23a704d740 1 -- 172.21.15.29:0/833720897 --> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b182dba890 con 0x55b1831341f0 2024-03-11T19:21:25.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.065+0000 7f23a221d640 1 --2- 172.21.15.29:0/833720897 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55b1831341f0 0x55b1831365e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.065+0000 7f23a221d640 1 --2- 172.21.15.29:0/833720897 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55b1831341f0 0x55b1831365e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8de87d24fd4078cd server_cookie=fe92d10337690cc7 in_seq=0 out_seq=0 2024-03-11T19:21:25.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.065+0000 7f2385ffb640 10 client.18111 ms_handle_connect on v2:172.21.15.190:6834/1048234253 2024-03-11T19:21:25.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.065+0000 7f2385ffb640 1 -- 172.21.15.29:0/833720897 <== mds.4 v2:172.21.15.190:6834/1048234253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b182dba890 con 0x55b1831341f0 2024-03-11T19:21:25.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.065+0000 7f2385ffb640 10 client.18111 handle_command_reply: tid=0 2024-03-11T19:21:25.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.065+0000 7f23a704d740 1 client.18111 shutdown 2024-03-11T19:21:25.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.065+0000 7f23a704d740 1 -- 172.21.15.29:0/833720897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6c60 msgr2=0x55b1830d66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.065+0000 7f23a704d740 1 --2- 172.21.15.29:0/833720897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6c60 0x55b1830d66a0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x55b1830c2750 tx=0x7f2398009370 comp rx=0 tx=0).stop 2024-03-11T19:21:25.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.065+0000 7f23a704d740 1 -- 172.21.15.29:0/833720897 shutdown_connections 2024-03-11T19:21:25.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.065+0000 7f23a704d740 1 --2- 172.21.15.29:0/833720897 >> [v2:172.21.15.190:6834/1048234253,v1:172.21.15.190:6835/1048234253] conn(0x55b1831341f0 0x55b1831365e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.065+0000 7f23a704d740 1 --2- 172.21.15.29:0/833720897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1830cbdd0 0x55b1830d6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.065+0000 7f23a704d740 1 --2- 172.21.15.29:0/833720897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b1830c6c60 0x55b1830d66a0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.065+0000 7f2385ffb640 0 client.18111 ms_handle_reset on v2:172.21.15.190:6834/1048234253 2024-03-11T19:21:25.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.065+0000 7f23a704d740 1 --2- 172.21.15.29:0/833720897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b1830c6270 0x55b1830d6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.065+0000 7f23a704d740 1 -- 172.21.15.29:0/833720897 >> 172.21.15.29:0/833720897 conn(0x55b1830c17e0 msgr2=0x55b1830cffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.066+0000 7f23a704d740 1 -- 172.21.15.29:0/833720897 shutdown_connections 2024-03-11T19:21:25.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.066+0000 7f23a704d740 1 -- 172.21.15.29:0/833720897 wait complete. 2024-03-11T19:21:25.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.066+0000 7f23a704d740 20 client.18111 trim_cache size 0 max 16384 2024-03-11T19:21:25.081 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:21:25.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.066+0000 7f23a4481640 1 -- 172.21.15.29:0/2406934927 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23880a1a30 msgr2=0x7f23880a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.066+0000 7f23a4481640 1 --2- 172.21.15.29:0/2406934927 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23880a1a30 0x7f23880a3ef0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x55b1830c74c0 tx=0x7f2394004a40 comp rx=0 tx=0).stop 2024-03-11T19:21:25.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.066+0000 7f23a4481640 1 -- 172.21.15.29:0/2406934927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239c100080 msgr2=0x7f239c197150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.066+0000 7f23a4481640 1 --2- 172.21.15.29:0/2406934927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239c100080 0x7f239c197150 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f2390000c00 tx=0x7f23900042f0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.066+0000 7f23a4481640 1 -- 172.21.15.29:0/2406934927 shutdown_connections 2024-03-11T19:21:25.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.066+0000 7f23a4481640 1 --2- 172.21.15.29:0/2406934927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f239c101c30 0x7f239c19e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.066+0000 7f23a4481640 1 --2- 172.21.15.29:0/2406934927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f239c101270 0x7f239c197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.066+0000 7f23a4481640 1 --2- 172.21.15.29:0/2406934927 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f23880a1a30 0x7f23880a3ef0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.066+0000 7f23a4481640 1 --2- 172.21.15.29:0/2406934927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239c100080 0x7f239c197150 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.066+0000 7f23a4481640 1 -- 172.21.15.29:0/2406934927 >> 172.21.15.29:0/2406934927 conn(0x7f239c0fb610 msgr2=0x7f239c104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.066+0000 7f23a4481640 1 -- 172.21.15.29:0/2406934927 shutdown_connections 2024-03-11T19:21:25.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.066+0000 7f23a4481640 1 -- 172.21.15.29:0/2406934927 wait complete. 2024-03-11T19:21:25.083 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:21:25.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-11T19:21:25.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.184+0000 7f925fac0640 1 -- 172.21.15.29:0/2379769390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9258101c30 msgr2=0x7f92581084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.184+0000 7f925fac0640 1 --2- 172.21.15.29:0/2379769390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9258101c30 0x7f92581084c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f9248009ca0 tx=0x7f924802c0f0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.184+0000 7f925fac0640 1 -- 172.21.15.29:0/2379769390 shutdown_connections 2024-03-11T19:21:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.184+0000 7f925fac0640 1 --2- 172.21.15.29:0/2379769390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9258101c30 0x7f92581084c0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.184+0000 7f925fac0640 1 --2- 172.21.15.29:0/2379769390 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9258101270 0x7f92581016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.184+0000 7f925fac0640 1 --2- 172.21.15.29:0/2379769390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9258100080 0x7f92581004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.184+0000 7f925fac0640 1 -- 172.21.15.29:0/2379769390 >> 172.21.15.29:0/2379769390 conn(0x7f92580fb610 msgr2=0x7f92580fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.184+0000 7f925fac0640 1 -- 172.21.15.29:0/2379769390 shutdown_connections 2024-03-11T19:21:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.184+0000 7f925fac0640 1 -- 172.21.15.29:0/2379769390 wait complete. 2024-03-11T19:21:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.185+0000 7f925fac0640 1 Processor -- start 2024-03-11T19:21:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.185+0000 7f925fac0640 1 -- start start 2024-03-11T19:21:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.185+0000 7f925fac0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9258100080 0x7f9258196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.185+0000 7f925fac0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9258101270 0x7f9258196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.185+0000 7f925fac0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9258101c30 0x7f925819d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.185+0000 7f925fac0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925819db10 con 0x7f9258101c30 2024-03-11T19:21:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.185+0000 7f925fac0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925819dc80 con 0x7f9258100080 2024-03-11T19:21:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.185+0000 7f925fac0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925819de40 con 0x7f9258101270 2024-03-11T19:21:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.185+0000 7f925e05d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9258101c30 0x7f925819d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.185+0000 7f925e05d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9258101c30 0x7f925819d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44806/0 (socket says 172.21.15.29:44806) 2024-03-11T19:21:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.185+0000 7f925e05d640 1 -- 172.21.15.29:0/3036567051 learned_addr learned my addr 172.21.15.29:0/3036567051 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.185+0000 7f925d85c640 1 --2- 172.21.15.29:0/3036567051 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9258100080 0x7f9258196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.186+0000 7f925e05d640 1 -- 172.21.15.29:0/3036567051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9258101270 msgr2=0x7f9258196560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.186+0000 7f925d05b640 1 --2- 172.21.15.29:0/3036567051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9258101270 0x7f9258196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.186+0000 7f925e05d640 1 --2- 172.21.15.29:0/3036567051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9258101270 0x7f9258196560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.186+0000 7f925e05d640 1 -- 172.21.15.29:0/3036567051 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9258100080 msgr2=0x7f9258196020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.186+0000 7f925e05d640 1 --2- 172.21.15.29:0/3036567051 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9258100080 0x7f9258196020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.186+0000 7f925e05d640 1 -- 172.21.15.29:0/3036567051 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9248009900 con 0x7f9258101c30 2024-03-11T19:21:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.186+0000 7f925d85c640 1 --2- 172.21.15.29:0/3036567051 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9258100080 0x7f9258196020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:21:25.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.186+0000 7f925d05b640 1 --2- 172.21.15.29:0/3036567051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9258101270 0x7f9258196560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:21:25.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.186+0000 7f925e05d640 1 --2- 172.21.15.29:0/3036567051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9258101c30 0x7f925819d5d0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f9248037040 tx=0x7f9248032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.186+0000 7f9256ffd640 1 -- 172.21.15.29:0/3036567051 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9248003970 con 0x7f9258101c30 2024-03-11T19:21:25.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.186+0000 7f925fac0640 1 -- 172.21.15.29:0/3036567051 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f925819e0c0 con 0x7f9258101c30 2024-03-11T19:21:25.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.186+0000 7f9256ffd640 1 -- 172.21.15.29:0/3036567051 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9248004040 con 0x7f9258101c30 2024-03-11T19:21:25.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.186+0000 7f9256ffd640 1 -- 172.21.15.29:0/3036567051 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924803d630 con 0x7f9258101c30 2024-03-11T19:21:25.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.186+0000 7f925fac0640 1 -- 172.21.15.29:0/3036567051 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f925819e570 con 0x7f9258101c30 2024-03-11T19:21:25.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.188+0000 7f9256ffd640 1 -- 172.21.15.29:0/3036567051 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f924803b040 con 0x7f9258101c30 2024-03-11T19:21:25.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f926276f740 1 Processor -- start 2024-03-11T19:21:25.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f926276f740 1 -- start start 2024-03-11T19:21:25.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f926276f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a1f480 0x557874a1f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f926276f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a1fe70 0x557874a24970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f926276f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a24fe0 0x557874a27470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f926276f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578749d9960 con 0x557874a24fe0 2024-03-11T19:21:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f926276f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55787498d310 con 0x557874a1fe70 2024-03-11T19:21:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f926276f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55787498a840 con 0x557874a1f480 2024-03-11T19:21:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f9256ffd640 1 --2- 172.21.15.29:0/3036567051 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f92400a1a30 0x7f92400a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f925d05b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a24fe0 0x557874a27470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f925d05b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a24fe0 0x557874a27470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44822/0 (socket says 172.21.15.29:44822) 2024-03-11T19:21:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f925d05b640 1 -- 172.21.15.29:0/969309127 learned_addr learned my addr 172.21.15.29:0/969309127 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f925d85c640 1 --2- 172.21.15.29:0/969309127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a1fe70 0x557874a24970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f925e05d640 1 --2- 172.21.15.29:0/3036567051 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f92400a1a30 0x7f92400a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f925d05b640 1 --2- 172.21.15.29:0/969309127 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a1f480 0x557874a1f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f9256ffd640 1 -- 172.21.15.29:0/3036567051 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f92480efdd0 con 0x7f9258101c30 2024-03-11T19:21:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f925d05b640 1 -- 172.21.15.29:0/969309127 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a1f480 msgr2=0x557874a1f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f925d05b640 1 --2- 172.21.15.29:0/969309127 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a1f480 0x557874a1f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f925d05b640 1 -- 172.21.15.29:0/969309127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a1fe70 msgr2=0x557874a24970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f925d05b640 1 --2- 172.21.15.29:0/969309127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a1fe70 0x557874a24970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.189+0000 7f925d05b640 1 -- 172.21.15.29:0/969309127 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557874948060 con 0x557874a24fe0 2024-03-11T19:21:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.190+0000 7f925d05b640 1 --2- 172.21.15.29:0/969309127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a24fe0 0x557874a27470 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f924c0144e0 tx=0x7f924c00d5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83f9e11304a2447 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.190+0000 7f925e05d640 1 --2- 172.21.15.29:0/3036567051 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f92400a1a30 0x7f92400a3ef0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9248048000 tx=0x7f9248090040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.190+0000 7f923ffff640 1 -- 172.21.15.29:0/969309127 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924c012650 con 0x557874a24fe0 2024-03-11T19:21:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.190+0000 7f923ffff640 1 -- 172.21.15.29:0/969309127 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f924c014300 con 0x557874a24fe0 2024-03-11T19:21:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.190+0000 7f923ffff640 1 -- 172.21.15.29:0/969309127 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924c011b10 con 0x557874a24fe0 2024-03-11T19:21:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.191+0000 7f926276f740 1 -- 172.21.15.29:0/969309127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a24fe0 msgr2=0x557874a27470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.191+0000 7f926276f740 1 --2- 172.21.15.29:0/969309127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a24fe0 0x557874a27470 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f924c0144e0 tx=0x7f924c00d5a0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.191+0000 7f926276f740 1 -- 172.21.15.29:0/969309127 shutdown_connections 2024-03-11T19:21:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.191+0000 7f926276f740 1 --2- 172.21.15.29:0/969309127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a24fe0 0x557874a27470 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.191+0000 7f926276f740 1 --2- 172.21.15.29:0/969309127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a1fe70 0x557874a24970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.191+0000 7f926276f740 1 --2- 172.21.15.29:0/969309127 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a1f480 0x557874a1f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.191+0000 7f926276f740 1 -- 172.21.15.29:0/969309127 >> 172.21.15.29:0/969309127 conn(0x557874a187e0 msgr2=0x557874a18be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.191+0000 7f926276f740 1 -- 172.21.15.29:0/969309127 shutdown_connections 2024-03-11T19:21:25.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.191+0000 7f926276f740 1 -- 172.21.15.29:0/969309127 wait complete. 2024-03-11T19:21:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.191+0000 7f926276f740 1 -- start start 2024-03-11T19:21:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.191+0000 7f926276f740 1 Processor -- start 2024-03-11T19:21:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.191+0000 7f926276f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a1f480 0x557874a29670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.191+0000 7f926276f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 0x557874a29bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.191+0000 7f926276f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a24fe0 0x557874a30bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f926276f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557874986020 con 0x557874a1fe70 2024-03-11T19:21:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f926276f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557874659430 con 0x557874a1f480 2024-03-11T19:21:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f926276f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557874615e00 con 0x557874a24fe0 2024-03-11T19:21:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f926276f740 10 client.0 _ll_register_callbacks cb 0x55787492dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:21:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f925d85c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 0x557874a29bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f925d85c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 0x557874a29bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44838/0 (socket says 172.21.15.29:44838) 2024-03-11T19:21:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f925d85c640 1 -- 172.21.15.29:0/1936520953 learned_addr learned my addr 172.21.15.29:0/1936520953 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f925d05b640 1 --2- 172.21.15.29:0/1936520953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a1f480 0x557874a29670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f925d05b640 1 --2- 172.21.15.29:0/1936520953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a24fe0 0x557874a30bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f925d85c640 1 -- 172.21.15.29:0/1936520953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a24fe0 msgr2=0x557874a30bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f925d85c640 1 --2- 172.21.15.29:0/1936520953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a24fe0 0x557874a30bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f925d85c640 1 -- 172.21.15.29:0/1936520953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a1f480 msgr2=0x557874a29670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f925d85c640 1 --2- 172.21.15.29:0/1936520953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a1f480 0x557874a29670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f925d85c640 1 -- 172.21.15.29:0/1936520953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f924c00dc90 con 0x557874a1fe70 2024-03-11T19:21:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f925d05b640 1 --2- 172.21.15.29:0/1936520953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a1f480 0x557874a29670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:21:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f925d85c640 1 --2- 172.21.15.29:0/1936520953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 0x557874a29bb0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f92500107a0 tx=0x7f9250010c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94fbe19ff548cb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f923d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:21:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f923d7fa640 1 -- 172.21.15.29:0/1936520953 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925000ed40 con 0x557874a1fe70 2024-03-11T19:21:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f923d7fa640 1 -- 172.21.15.29:0/1936520953 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9250018430 con 0x557874a1fe70 2024-03-11T19:21:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f926276f740 1 -- 172.21.15.29:0/1936520953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5578746857e0 con 0x557874a1fe70 2024-03-11T19:21:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f923d7fa640 1 -- 172.21.15.29:0/1936520953 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f925000eea0 con 0x557874a1fe70 2024-03-11T19:21:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f926276f740 10 client.18123 fetch_fsmap learned FSMap version 27 2024-03-11T19:21:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.192+0000 7f926276f740 1 -- 172.21.15.29:0/1936520953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5578749846a0 con 0x557874a1fe70 2024-03-11T19:21:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.193+0000 7f923d7fa640 1 -- 172.21.15.29:0/1936520953 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5578746857e0 con 0x557874a1fe70 2024-03-11T19:21:25.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.193+0000 7f923d7fa640 1 -- 172.21.15.29:0/1936520953 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9250022550 con 0x557874a1fe70 2024-03-11T19:21:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.193+0000 7f926276f740 10 client.18123 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:21:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.193+0000 7f926276f740 10 client.18123 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:21:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.193+0000 7f926276f740 20 client.18123 populate_metadata read hostname 'smithi029' 2024-03-11T19:21:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.193+0000 7f926276f740 1 --2- 172.21.15.29:0/1936520953 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x557874a35920 0x557874a37d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.193+0000 7f926276f740 4 client.18123 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:21:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.193+0000 7f926276f740 1 -- 172.21.15.29:0/1936520953 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5578746c2720 con 0x557874a35920 2024-03-11T19:21:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.193+0000 7f925d05b640 1 --2- 172.21.15.29:0/1936520953 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x557874a35920 0x557874a37d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.194+0000 7f925d05b640 1 --2- 172.21.15.29:0/1936520953 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x557874a35920 0x557874a37d40 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a25ade366dfc9155 server_cookie=54733489dd312b1d in_seq=0 out_seq=0 2024-03-11T19:21:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.194+0000 7f923d7fa640 10 client.18123 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:21:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.194+0000 7f923d7fa640 1 -- 172.21.15.29:0/1936520953 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5578746c2720 con 0x557874a35920 2024-03-11T19:21:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.194+0000 7f923d7fa640 10 client.18123 handle_command_reply: tid=0 2024-03-11T19:21:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.194+0000 7f926276f740 1 client.18123 shutdown 2024-03-11T19:21:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.194+0000 7f926276f740 1 -- 172.21.15.29:0/1936520953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 msgr2=0x557874a29bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.194+0000 7f926276f740 1 --2- 172.21.15.29:0/1936520953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 0x557874a29bb0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f92500107a0 tx=0x7f9250010c60 comp rx=0 tx=0).stop 2024-03-11T19:21:25.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.194+0000 7f926276f740 1 -- 172.21.15.29:0/1936520953 shutdown_connections 2024-03-11T19:21:25.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.194+0000 7f926276f740 1 --2- 172.21.15.29:0/1936520953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a24fe0 0x557874a30bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.194+0000 7f926276f740 1 --2- 172.21.15.29:0/1936520953 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x557874a35920 0x557874a37d40 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.194+0000 7f926276f740 1 --2- 172.21.15.29:0/1936520953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 0x557874a29bb0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.194+0000 7f926276f740 1 --2- 172.21.15.29:0/1936520953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a1f480 0x557874a29670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.194+0000 7f923d7fa640 0 client.18123 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:21:25.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.194+0000 7f926276f740 1 -- 172.21.15.29:0/1936520953 >> 172.21.15.29:0/1936520953 conn(0x557874a187e0 msgr2=0x557874a17b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.194+0000 7f926276f740 1 -- 172.21.15.29:0/1936520953 shutdown_connections 2024-03-11T19:21:25.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.194+0000 7f926276f740 1 -- 172.21.15.29:0/1936520953 wait complete. 2024-03-11T19:21:25.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.195+0000 7f926276f740 20 client.18123 trim_cache size 0 max 16384 2024-03-11T19:21:25.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.206+0000 7f926276f740 1 Processor -- start 2024-03-11T19:21:25.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.206+0000 7f926276f740 1 -- start start 2024-03-11T19:21:25.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.206+0000 7f926276f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a1f480 0x557874a18bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.206+0000 7f926276f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 0x557874a30bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.206+0000 7f926276f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a24fe0 0x557874a21330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.206+0000 7f926276f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55787490b130 con 0x557874a1fe70 2024-03-11T19:21:25.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.206+0000 7f926276f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557874a27b30 con 0x557874a24fe0 2024-03-11T19:21:25.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.206+0000 7f926276f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557874a20560 con 0x557874a1f480 2024-03-11T19:21:25.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.207+0000 7f925d85c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 0x557874a30bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.207+0000 7f925d85c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 0x557874a30bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44844/0 (socket says 172.21.15.29:44844) 2024-03-11T19:21:25.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.207+0000 7f925d85c640 1 -- 172.21.15.29:0/729630213 learned_addr learned my addr 172.21.15.29:0/729630213 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.207+0000 7f925d05b640 1 --2- 172.21.15.29:0/729630213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a1f480 0x557874a18bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.207+0000 7f925d05b640 1 --2- 172.21.15.29:0/729630213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a24fe0 0x557874a21330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.207+0000 7f925d85c640 1 -- 172.21.15.29:0/729630213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a1f480 msgr2=0x557874a18bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.207+0000 7f925d85c640 1 --2- 172.21.15.29:0/729630213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a1f480 0x557874a18bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.207+0000 7f925d85c640 1 -- 172.21.15.29:0/729630213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a24fe0 msgr2=0x557874a21330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.207+0000 7f925d85c640 1 --2- 172.21.15.29:0/729630213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a24fe0 0x557874a21330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.207+0000 7f925d85c640 1 -- 172.21.15.29:0/729630213 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557874920c20 con 0x557874a1fe70 2024-03-11T19:21:25.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.207+0000 7f925d85c640 1 --2- 172.21.15.29:0/729630213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 0x557874a30bb0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f9250005840 tx=0x7f9250004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad806eaadff5fd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.207+0000 7f923f7fe640 1 -- 172.21.15.29:0/729630213 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925000bed0 con 0x557874a1fe70 2024-03-11T19:21:25.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.207+0000 7f923f7fe640 1 -- 172.21.15.29:0/729630213 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f925000e780 con 0x557874a1fe70 2024-03-11T19:21:25.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.207+0000 7f923f7fe640 1 -- 172.21.15.29:0/729630213 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92500044b0 con 0x557874a1fe70 2024-03-11T19:21:25.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.208+0000 7f926276f740 1 -- 172.21.15.29:0/729630213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 msgr2=0x557874a30bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.208+0000 7f926276f740 1 --2- 172.21.15.29:0/729630213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 0x557874a30bb0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f9250005840 tx=0x7f9250004680 comp rx=0 tx=0).stop 2024-03-11T19:21:25.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.208+0000 7f926276f740 1 -- 172.21.15.29:0/729630213 shutdown_connections 2024-03-11T19:21:25.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.208+0000 7f926276f740 1 --2- 172.21.15.29:0/729630213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a24fe0 0x557874a21330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.208+0000 7f926276f740 1 --2- 172.21.15.29:0/729630213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 0x557874a30bb0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.208+0000 7f926276f740 1 --2- 172.21.15.29:0/729630213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a1f480 0x557874a18bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.208+0000 7f926276f740 1 -- 172.21.15.29:0/729630213 >> 172.21.15.29:0/729630213 conn(0x557874a187e0 msgr2=0x557874a13a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.208+0000 7f926276f740 1 -- 172.21.15.29:0/729630213 shutdown_connections 2024-03-11T19:21:25.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.208+0000 7f926276f740 1 -- 172.21.15.29:0/729630213 wait complete. 2024-03-11T19:21:25.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.208+0000 7f926276f740 1 -- start start 2024-03-11T19:21:25.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.208+0000 7f926276f740 1 Processor -- start 2024-03-11T19:21:25.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.208+0000 7f926276f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a1f480 0x557874a2f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.208+0000 7f926276f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 0x557874a2fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.208+0000 7f926276f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a24fe0 0x557874a2fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.208+0000 7f926276f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x557874a8be60 con 0x557874a1fe70 2024-03-11T19:21:25.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.208+0000 7f926276f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x557874a8bfd0 con 0x557874a24fe0 2024-03-11T19:21:25.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.208+0000 7f926276f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557874a8c190 con 0x557874a1f480 2024-03-11T19:21:25.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.208+0000 7f926276f740 10 client.0 _ll_register_callbacks cb 0x7f924c00d440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:21:25.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.209+0000 7f925d85c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 0x557874a2fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.209+0000 7f925d85c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 0x557874a2fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44860/0 (socket says 172.21.15.29:44860) 2024-03-11T19:21:25.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.209+0000 7f925d85c640 1 -- 172.21.15.29:0/3529311616 learned_addr learned my addr 172.21.15.29:0/3529311616 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.209+0000 7f925d05b640 1 --2- 172.21.15.29:0/3529311616 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a24fe0 0x557874a2fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.209+0000 7f925d05b640 1 --2- 172.21.15.29:0/3529311616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a1f480 0x557874a2f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.209+0000 7f925d85c640 1 -- 172.21.15.29:0/3529311616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a1f480 msgr2=0x557874a2f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.209+0000 7f925d85c640 1 --2- 172.21.15.29:0/3529311616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a1f480 0x557874a2f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.209+0000 7f925d85c640 1 -- 172.21.15.29:0/3529311616 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a24fe0 msgr2=0x557874a2fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.209+0000 7f925d85c640 1 --2- 172.21.15.29:0/3529311616 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a24fe0 0x557874a2fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.209+0000 7f925d85c640 1 -- 172.21.15.29:0/3529311616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5578749846a0 con 0x557874a1fe70 2024-03-11T19:21:25.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.209+0000 7f925d05b640 1 --2- 172.21.15.29:0/3529311616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a1f480 0x557874a2f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:21:25.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.209+0000 7f925d85c640 1 --2- 172.21.15.29:0/3529311616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 0x557874a2fab0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f925000bc20 tx=0x7f9250004440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36b47c742bdc5297 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.209+0000 7f923d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:21:25.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.209+0000 7f923d7fa640 1 -- 172.21.15.29:0/3529311616 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9250022660 con 0x557874a1fe70 2024-03-11T19:21:25.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.209+0000 7f923d7fa640 1 -- 172.21.15.29:0/3529311616 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f9250000e40 con 0x557874a1fe70 2024-03-11T19:21:25.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.209+0000 7f926276f740 1 -- 172.21.15.29:0/3529311616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f924c014300 con 0x557874a1fe70 2024-03-11T19:21:25.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.209+0000 7f923d7fa640 1 -- 172.21.15.29:0/3529311616 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f925000eb40 con 0x557874a1fe70 2024-03-11T19:21:25.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.209+0000 7f926276f740 10 client.18129 fetch_fsmap learned FSMap version 27 2024-03-11T19:21:25.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.209+0000 7f926276f740 1 -- 172.21.15.29:0/3529311616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5578749468c0 con 0x557874a1fe70 2024-03-11T19:21:25.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.210+0000 7f923d7fa640 1 -- 172.21.15.29:0/3529311616 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f924c014300 con 0x557874a1fe70 2024-03-11T19:21:25.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.210+0000 7f923d7fa640 1 -- 172.21.15.29:0/3529311616 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f925002a030 con 0x557874a1fe70 2024-03-11T19:21:25.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.210+0000 7f926276f740 10 client.18129 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:21:25.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.210+0000 7f926276f740 10 client.18129 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-11T19:21:25.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.210+0000 7f926276f740 20 client.18129 populate_metadata read hostname 'smithi029' 2024-03-11T19:21:25.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.210+0000 7f926276f740 1 --2- 172.21.15.29:0/3529311616 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x557874a8d960 0x557874a8fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.210+0000 7f926276f740 4 client.18129 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:21:25.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.210+0000 7f926276f740 1 -- 172.21.15.29:0/3529311616 --> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557874711890 con 0x557874a8d960 2024-03-11T19:21:25.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.210+0000 7f925d05b640 1 --2- 172.21.15.29:0/3529311616 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x557874a8d960 0x557874a8fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.211+0000 7f925d05b640 1 --2- 172.21.15.29:0/3529311616 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x557874a8d960 0x557874a8fd80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e92393a898dba97c server_cookie=4cd930d1dac4e821 in_seq=0 out_seq=0 2024-03-11T19:21:25.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.211+0000 7f923d7fa640 10 client.18129 ms_handle_connect on v2:172.21.15.170:6836/585739236 2024-03-11T19:21:25.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.211+0000 7f923d7fa640 1 -- 172.21.15.29:0/3529311616 <== mds.3 v2:172.21.15.170:6836/585739236 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557874711890 con 0x557874a8d960 2024-03-11T19:21:25.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.211+0000 7f923d7fa640 10 client.18129 handle_command_reply: tid=0 2024-03-11T19:21:25.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.211+0000 7f926276f740 1 client.18129 shutdown 2024-03-11T19:21:25.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.211+0000 7f926276f740 1 -- 172.21.15.29:0/3529311616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 msgr2=0x557874a2fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.211+0000 7f926276f740 1 --2- 172.21.15.29:0/3529311616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 0x557874a2fab0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f925000bc20 tx=0x7f9250004440 comp rx=0 tx=0).stop 2024-03-11T19:21:25.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.211+0000 7f926276f740 1 -- 172.21.15.29:0/3529311616 shutdown_connections 2024-03-11T19:21:25.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.211+0000 7f926276f740 1 --2- 172.21.15.29:0/3529311616 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557874a24fe0 0x557874a2fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.211+0000 7f926276f740 1 --2- 172.21.15.29:0/3529311616 >> [v2:172.21.15.170:6836/585739236,v1:172.21.15.170:6837/585739236] conn(0x557874a8d960 0x557874a8fd80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.211+0000 7f926276f740 1 --2- 172.21.15.29:0/3529311616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557874a1fe70 0x557874a2fab0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.211+0000 7f926276f740 1 --2- 172.21.15.29:0/3529311616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557874a1f480 0x557874a2f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.211+0000 7f923d7fa640 0 client.18129 ms_handle_reset on v2:172.21.15.170:6836/585739236 2024-03-11T19:21:25.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.211+0000 7f926276f740 1 -- 172.21.15.29:0/3529311616 >> 172.21.15.29:0/3529311616 conn(0x557874a187e0 msgr2=0x557874a18bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.211+0000 7f926276f740 1 -- 172.21.15.29:0/3529311616 shutdown_connections 2024-03-11T19:21:25.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.211+0000 7f926276f740 1 -- 172.21.15.29:0/3529311616 wait complete. 2024-03-11T19:21:25.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.211+0000 7f926276f740 20 client.18129 trim_cache size 0 max 16384 2024-03-11T19:21:25.221 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:21:25.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.212+0000 7f925fac0640 1 -- 172.21.15.29:0/3036567051 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f92400a1a30 msgr2=0x7f92400a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.212+0000 7f925fac0640 1 --2- 172.21.15.29:0/3036567051 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f92400a1a30 0x7f92400a3ef0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9248048000 tx=0x7f9248090040 comp rx=0 tx=0).stop 2024-03-11T19:21:25.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.212+0000 7f925fac0640 1 -- 172.21.15.29:0/3036567051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9258101c30 msgr2=0x7f925819d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.212+0000 7f925fac0640 1 --2- 172.21.15.29:0/3036567051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9258101c30 0x7f925819d5d0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f9248037040 tx=0x7f9248032980 comp rx=0 tx=0).stop 2024-03-11T19:21:25.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.212+0000 7f925fac0640 1 -- 172.21.15.29:0/3036567051 shutdown_connections 2024-03-11T19:21:25.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.212+0000 7f925fac0640 1 --2- 172.21.15.29:0/3036567051 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f92400a1a30 0x7f92400a3ef0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.212+0000 7f925fac0640 1 --2- 172.21.15.29:0/3036567051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9258101c30 0x7f925819d5d0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.212+0000 7f925fac0640 1 --2- 172.21.15.29:0/3036567051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9258101270 0x7f9258196560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.212+0000 7f925fac0640 1 --2- 172.21.15.29:0/3036567051 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9258100080 0x7f9258196020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.212+0000 7f925fac0640 1 -- 172.21.15.29:0/3036567051 >> 172.21.15.29:0/3036567051 conn(0x7f92580fb610 msgr2=0x7f92581045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.212+0000 7f925fac0640 1 -- 172.21.15.29:0/3036567051 shutdown_connections 2024-03-11T19:21:25.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.212+0000 7f925fac0640 1 -- 172.21.15.29:0/3036567051 wait complete. 2024-03-11T19:21:25.226 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:21:25.226 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-11T19:21:25.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.328+0000 7fbabfe6d640 1 -- 172.21.15.29:0/3924095946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab8102380 msgr2=0x7fbab8102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.328+0000 7fbabfe6d640 1 --2- 172.21.15.29:0/3924095946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab8102380 0x7fbab8102800 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8009900 tx=0x7fbaa802bc30 comp rx=0 tx=0).stop 2024-03-11T19:21:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.328+0000 7fbabfe6d640 1 -- 172.21.15.29:0/3924095946 shutdown_connections 2024-03-11T19:21:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.328+0000 7fbabfe6d640 1 --2- 172.21.15.29:0/3924095946 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8102d40 0x7fbab81095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.328+0000 7fbabfe6d640 1 --2- 172.21.15.29:0/3924095946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab8102380 0x7fbab8102800 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.328+0000 7fbabfe6d640 1 --2- 172.21.15.29:0/3924095946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbab8101190 0x7fbab81015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.328+0000 7fbabfe6d640 1 -- 172.21.15.29:0/3924095946 >> 172.21.15.29:0/3924095946 conn(0x7fbab80fc720 msgr2=0x7fbab80feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.328+0000 7fbabfe6d640 1 -- 172.21.15.29:0/3924095946 shutdown_connections 2024-03-11T19:21:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.328+0000 7fbabfe6d640 1 -- 172.21.15.29:0/3924095946 wait complete. 2024-03-11T19:21:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.328+0000 7fbabfe6d640 1 Processor -- start 2024-03-11T19:21:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.328+0000 7fbabfe6d640 1 -- start start 2024-03-11T19:21:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.328+0000 7fbabfe6d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbab8101190 0x7fbab8197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.328+0000 7fbabfe6d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab8102380 0x7fbab8197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.328+0000 7fbabfe6d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8102d40 0x7fbab819e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.328+0000 7fbabfe6d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab819ec40 con 0x7fbab8102380 2024-03-11T19:21:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.328+0000 7fbabfe6d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab819edb0 con 0x7fbab8102d40 2024-03-11T19:21:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.328+0000 7fbabfe6d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab819ef70 con 0x7fbab8101190 2024-03-11T19:21:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.329+0000 7fbabd408640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab8102380 0x7fbab8197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.329+0000 7fbabd408640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab8102380 0x7fbab8197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44878/0 (socket says 172.21.15.29:44878) 2024-03-11T19:21:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.329+0000 7fbabd408640 1 -- 172.21.15.29:0/2721879435 learned_addr learned my addr 172.21.15.29:0/2721879435 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.329+0000 7fbabe40a640 1 --2- 172.21.15.29:0/2721879435 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8102d40 0x7fbab819e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.329+0000 7fbabd408640 1 -- 172.21.15.29:0/2721879435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbab8101190 msgr2=0x7fbab8197150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.329+0000 7fbabd408640 1 --2- 172.21.15.29:0/2721879435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbab8101190 0x7fbab8197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.329+0000 7fbabd408640 1 -- 172.21.15.29:0/2721879435 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8102d40 msgr2=0x7fbab819e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.329+0000 7fbabd408640 1 --2- 172.21.15.29:0/2721879435 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8102d40 0x7fbab819e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.329+0000 7fbabd408640 1 -- 172.21.15.29:0/2721879435 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa8009560 con 0x7fbab8102380 2024-03-11T19:21:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.329+0000 7fbabe40a640 1 --2- 172.21.15.29:0/2721879435 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8102d40 0x7fbab819e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:21:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.329+0000 7fbabd408640 1 --2- 172.21.15.29:0/2721879435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab8102380 0x7fbab8197690 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8036040 tx=0x7fbaa8004140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.329+0000 7fbaaeffd640 1 -- 172.21.15.29:0/2721879435 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa80035f0 con 0x7fbab8102380 2024-03-11T19:21:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.329+0000 7fbaaeffd640 1 -- 172.21.15.29:0/2721879435 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbaa802de90 con 0x7fbab8102380 2024-03-11T19:21:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.329+0000 7fbabfe6d640 1 -- 172.21.15.29:0/2721879435 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab819f1f0 con 0x7fbab8102380 2024-03-11T19:21:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.329+0000 7fbaaeffd640 1 -- 172.21.15.29:0/2721879435 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8005490 con 0x7fbab8102380 2024-03-11T19:21:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.329+0000 7fbabfe6d640 1 -- 172.21.15.29:0/2721879435 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab819f750 con 0x7fbab8102380 2024-03-11T19:21:25.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.331+0000 7fbaaeffd640 1 -- 172.21.15.29:0/2721879435 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fbaa8003750 con 0x7fbab8102380 2024-03-11T19:21:25.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.332+0000 7fbac2a3d740 1 Processor -- start 2024-03-11T19:21:25.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.332+0000 7fbac2a3d740 1 -- start start 2024-03-11T19:21:25.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.332+0000 7fbac2a3d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69270 0x55e39fb69690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.332+0000 7fbac2a3d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb69c60 0x55e39fb6e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.332+0000 7fbac2a3d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb6edd0 0x55e39fb71260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.332+0000 7fbac2a3d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39fb25960 con 0x55e39fb6edd0 2024-03-11T19:21:25.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.332+0000 7fbac2a3d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39fad9310 con 0x55e39fb69270 2024-03-11T19:21:25.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.332+0000 7fbac2a3d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39fad6840 con 0x55e39fb69c60 2024-03-11T19:21:25.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.332+0000 7fbabd408640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb6edd0 0x55e39fb71260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.332+0000 7fbaaeffd640 1 --2- 172.21.15.29:0/2721879435 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbaa40a1ab0 0x7fbaa40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.332+0000 7fbabd408640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb6edd0 0x55e39fb71260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44884/0 (socket says 172.21.15.29:44884) 2024-03-11T19:21:25.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.332+0000 7fbabd408640 1 -- 172.21.15.29:0/2971124644 learned_addr learned my addr 172.21.15.29:0/2971124644 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.332+0000 7fbabe40a640 1 --2- 172.21.15.29:0/2971124644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69270 0x55e39fb69690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.332+0000 7fbabe40a640 1 --2- 172.21.15.29:0/2721879435 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbaa40a1ab0 0x7fbaa40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.332+0000 7fbabdc09640 1 --2- 172.21.15.29:0/2971124644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb69c60 0x55e39fb6e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.333+0000 7fbaaeffd640 1 -- 172.21.15.29:0/2721879435 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fbaa80ef650 con 0x7fbab8102380 2024-03-11T19:21:25.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.333+0000 7fbabd408640 1 -- 172.21.15.29:0/2971124644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb69c60 msgr2=0x55e39fb6e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.333+0000 7fbabd408640 1 --2- 172.21.15.29:0/2971124644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb69c60 0x55e39fb6e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.333+0000 7fbabd408640 1 -- 172.21.15.29:0/2971124644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69270 msgr2=0x55e39fb69690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.333+0000 7fbabd408640 1 --2- 172.21.15.29:0/2971124644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69270 0x55e39fb69690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.333+0000 7fbabd408640 1 -- 172.21.15.29:0/2971124644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e39fa94060 con 0x55e39fb6edd0 2024-03-11T19:21:25.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.333+0000 7fbabdc09640 1 --2- 172.21.15.29:0/2971124644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb69c60 0x55e39fb6e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:21:25.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.333+0000 7fbabe40a640 1 --2- 172.21.15.29:0/2971124644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69270 0x55e39fb69690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:21:25.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.333+0000 7fbabd408640 1 --2- 172.21.15.29:0/2971124644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb6edd0 0x55e39fb71260 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8006010 tx=0x7fbaa8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55d0744f79c7ac1a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.333+0000 7fbabe40a640 1 --2- 172.21.15.29:0/2721879435 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbaa40a1ab0 0x7fbaa40a3f70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x55e39fb6a4c0 tx=0x7fbab0009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.333+0000 7fbaa3fff640 1 -- 172.21.15.29:0/2971124644 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8003e20 con 0x55e39fb6edd0 2024-03-11T19:21:25.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.333+0000 7fbaa3fff640 1 -- 172.21.15.29:0/2971124644 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbaa8004dc0 con 0x55e39fb6edd0 2024-03-11T19:21:25.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.333+0000 7fbaa3fff640 1 -- 172.21.15.29:0/2971124644 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa80ba960 con 0x55e39fb6edd0 2024-03-11T19:21:25.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.334+0000 7fbac2a3d740 1 -- 172.21.15.29:0/2971124644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb6edd0 msgr2=0x55e39fb71260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.334+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/2971124644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb6edd0 0x55e39fb71260 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8006010 tx=0x7fbaa8009260 comp rx=0 tx=0).stop 2024-03-11T19:21:25.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.334+0000 7fbac2a3d740 1 -- 172.21.15.29:0/2971124644 shutdown_connections 2024-03-11T19:21:25.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.334+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/2971124644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb6edd0 0x55e39fb71260 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.334+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/2971124644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb69c60 0x55e39fb6e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.334+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/2971124644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69270 0x55e39fb69690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.334+0000 7fbac2a3d740 1 -- 172.21.15.29:0/2971124644 >> 172.21.15.29:0/2971124644 conn(0x55e39fb647e0 msgr2=0x55e39fb64be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.334+0000 7fbac2a3d740 1 -- 172.21.15.29:0/2971124644 shutdown_connections 2024-03-11T19:21:25.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.334+0000 7fbac2a3d740 1 -- 172.21.15.29:0/2971124644 wait complete. 2024-03-11T19:21:25.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.334+0000 7fbac2a3d740 1 -- start start 2024-03-11T19:21:25.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.334+0000 7fbac2a3d740 1 Processor -- start 2024-03-11T19:21:25.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbac2a3d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69270 0x55e39fb732d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbac2a3d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb69c60 0x55e39fb73810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbac2a3d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb6edd0 0x55e39fb7a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbac2a3d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39fad2020 con 0x55e39fb69c60 2024-03-11T19:21:25.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbac2a3d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39f7a5430 con 0x55e39fb69270 2024-03-11T19:21:25.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbac2a3d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39f761e00 con 0x55e39fb6edd0 2024-03-11T19:21:25.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbac2a3d740 10 client.0 _ll_register_callbacks cb 0x55e39fa79cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:21:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbabd408640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb69c60 0x55e39fb73810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbabd408640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb69c60 0x55e39fb73810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44888/0 (socket says 172.21.15.29:44888) 2024-03-11T19:21:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbabd408640 1 -- 172.21.15.29:0/859020852 learned_addr learned my addr 172.21.15.29:0/859020852 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbabe40a640 1 --2- 172.21.15.29:0/859020852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb6edd0 0x55e39fb7a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbabdc09640 1 --2- 172.21.15.29:0/859020852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69270 0x55e39fb732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbabd408640 1 -- 172.21.15.29:0/859020852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb6edd0 msgr2=0x55e39fb7a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbabd408640 1 --2- 172.21.15.29:0/859020852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb6edd0 0x55e39fb7a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbabd408640 1 -- 172.21.15.29:0/859020852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69270 msgr2=0x55e39fb732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbabd408640 1 --2- 172.21.15.29:0/859020852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69270 0x55e39fb732d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbabd408640 1 -- 172.21.15.29:0/859020852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab819f750 con 0x55e39fb69c60 2024-03-11T19:21:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbabe40a640 1 --2- 172.21.15.29:0/859020852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb6edd0 0x55e39fb7a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:21:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbabd408640 1 --2- 172.21.15.29:0/859020852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb69c60 0x55e39fb73810 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fbaa80ed240 tx=0x7fbaa80b89e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49b788a63a2d00c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbaa17fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:21:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbaa17fa640 1 -- 172.21.15.29:0/859020852 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa80c07b0 con 0x55e39fb69c60 2024-03-11T19:21:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbaa17fa640 1 -- 172.21.15.29:0/859020852 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fbaa80bcd60 con 0x55e39fb69c60 2024-03-11T19:21:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.335+0000 7fbac2a3d740 1 -- 172.21.15.29:0/859020852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e39f7d17e0 con 0x55e39fb69c60 2024-03-11T19:21:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.336+0000 7fbaa17fa640 1 -- 172.21.15.29:0/859020852 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbaa80dc0d0 con 0x55e39fb69c60 2024-03-11T19:21:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.336+0000 7fbac2a3d740 10 client.18141 fetch_fsmap learned FSMap version 27 2024-03-11T19:21:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.336+0000 7fbac2a3d740 1 -- 172.21.15.29:0/859020852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e39fad06a0 con 0x55e39fb69c60 2024-03-11T19:21:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.336+0000 7fbaa17fa640 1 -- 172.21.15.29:0/859020852 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55e39f7d17e0 con 0x55e39fb69c60 2024-03-11T19:21:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.336+0000 7fbaa17fa640 1 -- 172.21.15.29:0/859020852 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbaa803c070 con 0x55e39fb69c60 2024-03-11T19:21:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.336+0000 7fbac2a3d740 10 client.18141 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:21:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.336+0000 7fbac2a3d740 10 client.18141 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:21:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.336+0000 7fbac2a3d740 20 client.18141 populate_metadata read hostname 'smithi029' 2024-03-11T19:21:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.336+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/859020852 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55e39fb7efd0 0x55e39fb813c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.336+0000 7fbac2a3d740 4 client.18141 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:21:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.336+0000 7fbac2a3d740 1 -- 172.21.15.29:0/859020852 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e39f80e720 con 0x55e39fb7efd0 2024-03-11T19:21:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.336+0000 7fbabdc09640 1 --2- 172.21.15.29:0/859020852 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55e39fb7efd0 0x55e39fb813c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.337+0000 7fbaa17fa640 10 client.18141 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:21:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.337+0000 7fbabdc09640 1 --2- 172.21.15.29:0/859020852 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55e39fb7efd0 0x55e39fb813c0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b6bafd5f01b7a316 server_cookie=26d5e5242e16538a in_seq=0 out_seq=0 2024-03-11T19:21:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.337+0000 7fbaa17fa640 1 -- 172.21.15.29:0/859020852 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e39f80e720 con 0x55e39fb7efd0 2024-03-11T19:21:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.337+0000 7fbaa17fa640 10 client.18141 handle_command_reply: tid=0 2024-03-11T19:21:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.337+0000 7fbac2a3d740 1 client.18141 shutdown 2024-03-11T19:21:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.337+0000 7fbac2a3d740 1 -- 172.21.15.29:0/859020852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb69c60 msgr2=0x55e39fb73810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.337+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/859020852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb69c60 0x55e39fb73810 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fbaa80ed240 tx=0x7fbaa80b89e0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.338+0000 7fbac2a3d740 1 -- 172.21.15.29:0/859020852 shutdown_connections 2024-03-11T19:21:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.338+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/859020852 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55e39fb7efd0 0x55e39fb813c0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.338+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/859020852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb6edd0 0x55e39fb7a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.338+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/859020852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb69c60 0x55e39fb73810 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.338+0000 7fbaa17fa640 0 client.18141 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:21:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.338+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/859020852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69270 0x55e39fb732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.338+0000 7fbac2a3d740 1 -- 172.21.15.29:0/859020852 >> 172.21.15.29:0/859020852 conn(0x55e39fb647e0 msgr2=0x55e39fb63b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.338+0000 7fbac2a3d740 1 -- 172.21.15.29:0/859020852 shutdown_connections 2024-03-11T19:21:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.338+0000 7fbac2a3d740 1 -- 172.21.15.29:0/859020852 wait complete. 2024-03-11T19:21:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.338+0000 7fbac2a3d740 20 client.18141 trim_cache size 0 max 16384 2024-03-11T19:21:25.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.349+0000 7fbac2a3d740 1 Processor -- start 2024-03-11T19:21:25.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.349+0000 7fbac2a3d740 1 -- start start 2024-03-11T19:21:25.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.349+0000 7fbac2a3d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb69270 0x55e39fb71d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.349+0000 7fbac2a3d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69c60 0x55e39fb7a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.349+0000 7fbac2a3d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb6edd0 0x55e39fb722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.349+0000 7fbac2a3d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39fb6ebb0 con 0x55e39fb6edd0 2024-03-11T19:21:25.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.349+0000 7fbac2a3d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39fa57130 con 0x55e39fb69c60 2024-03-11T19:21:25.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.349+0000 7fbac2a3d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39fb86340 con 0x55e39fb69270 2024-03-11T19:21:25.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.350+0000 7fbabe40a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb6edd0 0x55e39fb722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.350+0000 7fbabe40a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb6edd0 0x55e39fb722a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44894/0 (socket says 172.21.15.29:44894) 2024-03-11T19:21:25.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.350+0000 7fbabe40a640 1 -- 172.21.15.29:0/2344319276 learned_addr learned my addr 172.21.15.29:0/2344319276 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.350+0000 7fbabdc09640 1 --2- 172.21.15.29:0/2344319276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb69270 0x55e39fb71d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.350+0000 7fbabd408640 1 --2- 172.21.15.29:0/2344319276 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69c60 0x55e39fb7a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.350+0000 7fbabe40a640 1 -- 172.21.15.29:0/2344319276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb69270 msgr2=0x55e39fb71d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.350+0000 7fbabe40a640 1 --2- 172.21.15.29:0/2344319276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb69270 0x55e39fb71d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.350+0000 7fbabe40a640 1 -- 172.21.15.29:0/2344319276 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69c60 msgr2=0x55e39fb7a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.350+0000 7fbabe40a640 1 --2- 172.21.15.29:0/2344319276 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69c60 0x55e39fb7a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.350+0000 7fbabe40a640 1 -- 172.21.15.29:0/2344319276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e39fa6cc20 con 0x55e39fb6edd0 2024-03-11T19:21:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.350+0000 7fbabd408640 1 --2- 172.21.15.29:0/2344319276 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69c60 0x55e39fb7a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:21:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.350+0000 7fbabdc09640 1 --2- 172.21.15.29:0/2344319276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb69270 0x55e39fb71d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:21:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.350+0000 7fbabe40a640 1 --2- 172.21.15.29:0/2344319276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb6edd0 0x55e39fb722a0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fbab0014a90 tx=0x7fbab0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0f67adee822b63c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.350+0000 7fbaa37fe640 1 -- 172.21.15.29:0/2344319276 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab0011ce0 con 0x55e39fb6edd0 2024-03-11T19:21:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.350+0000 7fbaa37fe640 1 -- 172.21.15.29:0/2344319276 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbab001a070 con 0x55e39fb6edd0 2024-03-11T19:21:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.350+0000 7fbaa37fe640 1 -- 172.21.15.29:0/2344319276 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab0022e10 con 0x55e39fb6edd0 2024-03-11T19:21:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.351+0000 7fbac2a3d740 1 -- 172.21.15.29:0/2344319276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb6edd0 msgr2=0x55e39fb722a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.351+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/2344319276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb6edd0 0x55e39fb722a0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fbab0014a90 tx=0x7fbab0009920 comp rx=0 tx=0).stop 2024-03-11T19:21:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.351+0000 7fbac2a3d740 1 -- 172.21.15.29:0/2344319276 shutdown_connections 2024-03-11T19:21:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.351+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/2344319276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb6edd0 0x55e39fb722a0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.351+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/2344319276 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69c60 0x55e39fb7a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.351+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/2344319276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb69270 0x55e39fb71d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.351+0000 7fbac2a3d740 1 -- 172.21.15.29:0/2344319276 >> 172.21.15.29:0/2344319276 conn(0x55e39fb647e0 msgr2=0x55e39fb6b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.351+0000 7fbac2a3d740 1 -- 172.21.15.29:0/2344319276 shutdown_connections 2024-03-11T19:21:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.351+0000 7fbac2a3d740 1 -- 172.21.15.29:0/2344319276 wait complete. 2024-03-11T19:21:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.351+0000 7fbac2a3d740 1 -- start start 2024-03-11T19:21:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.351+0000 7fbac2a3d740 1 Processor -- start 2024-03-11T19:21:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.351+0000 7fbac2a3d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb69270 0x55e39fb7a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.351+0000 7fbac2a3d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69c60 0x55e39fb78c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.351+0000 7fbac2a3d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb6edd0 0x55e39fb79170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.351+0000 7fbac2a3d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39fb796e0 con 0x55e39fb69270 2024-03-11T19:21:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.351+0000 7fbac2a3d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39fb79850 con 0x55e39fb69c60 2024-03-11T19:21:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.351+0000 7fbac2a3d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39fb79a10 con 0x55e39fb6edd0 2024-03-11T19:21:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbac2a3d740 10 client.0 _ll_register_callbacks cb 0x7fbaa80bc430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:21:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbabdc09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb69270 0x55e39fb7a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbabdc09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb69270 0x55e39fb7a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44904/0 (socket says 172.21.15.29:44904) 2024-03-11T19:21:25.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbabdc09640 1 -- 172.21.15.29:0/3442826586 learned_addr learned my addr 172.21.15.29:0/3442826586 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbabe40a640 1 --2- 172.21.15.29:0/3442826586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb6edd0 0x55e39fb79170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbabd408640 1 --2- 172.21.15.29:0/3442826586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69c60 0x55e39fb78c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbabdc09640 1 -- 172.21.15.29:0/3442826586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb6edd0 msgr2=0x55e39fb79170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbabdc09640 1 --2- 172.21.15.29:0/3442826586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb6edd0 0x55e39fb79170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbabdc09640 1 -- 172.21.15.29:0/3442826586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69c60 msgr2=0x55e39fb78c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbabdc09640 1 --2- 172.21.15.29:0/3442826586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69c60 0x55e39fb78c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbabdc09640 1 -- 172.21.15.29:0/3442826586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab0009d10 con 0x55e39fb69270 2024-03-11T19:21:25.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbabd408640 1 --2- 172.21.15.29:0/3442826586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69c60 0x55e39fb78c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:21:25.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbabdc09640 1 --2- 172.21.15.29:0/3442826586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb69270 0x55e39fb7a560 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fbab4000c00 tx=0x7fbab4018ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56bc00b746f234a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbaa17fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:21:25.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbaa17fa640 1 -- 172.21.15.29:0/3442826586 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4004250 con 0x55e39fb69270 2024-03-11T19:21:25.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbaa17fa640 1 -- 172.21.15.29:0/3442826586 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fbab40043b0 con 0x55e39fb69270 2024-03-11T19:21:25.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbac2a3d740 1 -- 172.21.15.29:0/3442826586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbaa8004dc0 con 0x55e39fb69270 2024-03-11T19:21:25.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbaa17fa640 1 -- 172.21.15.29:0/3442826586 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbab40035f0 con 0x55e39fb69270 2024-03-11T19:21:25.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbac2a3d740 10 client.18147 fetch_fsmap learned FSMap version 27 2024-03-11T19:21:25.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.352+0000 7fbac2a3d740 1 -- 172.21.15.29:0/3442826586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e39fa928c0 con 0x55e39fb69270 2024-03-11T19:21:25.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.353+0000 7fbaa17fa640 1 -- 172.21.15.29:0/3442826586 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbaa8004dc0 con 0x55e39fb69270 2024-03-11T19:21:25.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.353+0000 7fbaa17fa640 1 -- 172.21.15.29:0/3442826586 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbab40126b0 con 0x55e39fb69270 2024-03-11T19:21:25.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.353+0000 7fbac2a3d740 10 client.18147 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:21:25.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.353+0000 7fbac2a3d740 10 client.18147 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-11T19:21:25.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.353+0000 7fbac2a3d740 20 client.18147 populate_metadata read hostname 'smithi029' 2024-03-11T19:21:25.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.353+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/3442826586 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55e39fbd6d60 0x55e39fbd9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.353+0000 7fbac2a3d740 4 client.18147 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:21:25.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.353+0000 7fbac2a3d740 1 -- 172.21.15.29:0/3442826586 --> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e39f85d890 con 0x55e39fbd6d60 2024-03-11T19:21:25.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.353+0000 7fbabd408640 1 --2- 172.21.15.29:0/3442826586 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55e39fbd6d60 0x55e39fbd9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.353+0000 7fbabd408640 1 --2- 172.21.15.29:0/3442826586 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55e39fbd6d60 0x55e39fbd9180 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bf90ff62129c0baa server_cookie=ec0f3e70895f624b in_seq=0 out_seq=0 2024-03-11T19:21:25.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.353+0000 7fbaa17fa640 10 client.18147 ms_handle_connect on v2:172.21.15.190:6836/3595456607 2024-03-11T19:21:25.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.353+0000 7fbaa17fa640 1 -- 172.21.15.29:0/3442826586 <== mds.2 v2:172.21.15.190:6836/3595456607 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e39f85d890 con 0x55e39fbd6d60 2024-03-11T19:21:25.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.353+0000 7fbaa17fa640 10 client.18147 handle_command_reply: tid=0 2024-03-11T19:21:25.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.353+0000 7fbac2a3d740 1 client.18147 shutdown 2024-03-11T19:21:25.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbac2a3d740 1 -- 172.21.15.29:0/3442826586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb69270 msgr2=0x55e39fb7a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/3442826586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb69270 0x55e39fb7a560 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fbab4000c00 tx=0x7fbab4018ed0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbac2a3d740 1 -- 172.21.15.29:0/3442826586 shutdown_connections 2024-03-11T19:21:25.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/3442826586 >> [v2:172.21.15.190:6836/3595456607,v1:172.21.15.190:6837/3595456607] conn(0x55e39fbd6d60 0x55e39fbd9180 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/3442826586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e39fb6edd0 0x55e39fb79170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/3442826586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e39fb69c60 0x55e39fb78c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbac2a3d740 1 --2- 172.21.15.29:0/3442826586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e39fb69270 0x55e39fb7a560 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbac2a3d740 1 -- 172.21.15.29:0/3442826586 >> 172.21.15.29:0/3442826586 conn(0x55e39fb647e0 msgr2=0x55e39fb71d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbaa17fa640 0 client.18147 ms_handle_reset on v2:172.21.15.190:6836/3595456607 2024-03-11T19:21:25.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbac2a3d740 1 -- 172.21.15.29:0/3442826586 shutdown_connections 2024-03-11T19:21:25.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbac2a3d740 1 -- 172.21.15.29:0/3442826586 wait complete. 2024-03-11T19:21:25.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbac2a3d740 20 client.18147 trim_cache size 0 max 16384 2024-03-11T19:21:25.364 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:21:25.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbabfe6d640 1 -- 172.21.15.29:0/2721879435 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbaa40a1ab0 msgr2=0x7fbaa40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbabfe6d640 1 --2- 172.21.15.29:0/2721879435 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbaa40a1ab0 0x7fbaa40a3f70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x55e39fb6a4c0 tx=0x7fbab0009500 comp rx=0 tx=0).stop 2024-03-11T19:21:25.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbabfe6d640 1 -- 172.21.15.29:0/2721879435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab8102380 msgr2=0x7fbab8197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbabfe6d640 1 --2- 172.21.15.29:0/2721879435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab8102380 0x7fbab8197690 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8036040 tx=0x7fbaa8004140 comp rx=0 tx=0).stop 2024-03-11T19:21:25.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbabfe6d640 1 -- 172.21.15.29:0/2721879435 shutdown_connections 2024-03-11T19:21:25.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbabfe6d640 1 --2- 172.21.15.29:0/2721879435 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8102d40 0x7fbab819e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbabfe6d640 1 --2- 172.21.15.29:0/2721879435 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbaa40a1ab0 0x7fbaa40a3f70 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbabfe6d640 1 --2- 172.21.15.29:0/2721879435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab8102380 0x7fbab8197690 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbabfe6d640 1 --2- 172.21.15.29:0/2721879435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbab8101190 0x7fbab8197150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbabfe6d640 1 -- 172.21.15.29:0/2721879435 >> 172.21.15.29:0/2721879435 conn(0x7fbab80fc720 msgr2=0x7fbab81057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbabfe6d640 1 -- 172.21.15.29:0/2721879435 shutdown_connections 2024-03-11T19:21:25.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.354+0000 7fbabfe6d640 1 -- 172.21.15.29:0/2721879435 wait complete. 2024-03-11T19:21:25.369 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:21:25.369 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-11T19:21:25.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.470+0000 7f4914d56640 1 -- 172.21.15.29:0/1598866083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4910101c30 msgr2=0x7f49101084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.470+0000 7f4914d56640 1 --2- 172.21.15.29:0/1598866083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4910101c30 0x7f49101084c0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f4900009ca0 tx=0x7f490002c0f0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.470+0000 7f4914d56640 1 -- 172.21.15.29:0/1598866083 shutdown_connections 2024-03-11T19:21:25.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.470+0000 7f4914d56640 1 --2- 172.21.15.29:0/1598866083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4910101c30 0x7f49101084c0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.470+0000 7f4914d56640 1 --2- 172.21.15.29:0/1598866083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4910101270 0x7f49101016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.470+0000 7f4914d56640 1 --2- 172.21.15.29:0/1598866083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4910100080 0x7f49101004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.470+0000 7f4914d56640 1 -- 172.21.15.29:0/1598866083 >> 172.21.15.29:0/1598866083 conn(0x7f49100fb610 msgr2=0x7f49100fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.470+0000 7f4914d56640 1 -- 172.21.15.29:0/1598866083 shutdown_connections 2024-03-11T19:21:25.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.470+0000 7f4914d56640 1 -- 172.21.15.29:0/1598866083 wait complete. 2024-03-11T19:21:25.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.471+0000 7f4914d56640 1 Processor -- start 2024-03-11T19:21:25.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.471+0000 7f4914d56640 1 -- start start 2024-03-11T19:21:25.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.471+0000 7f4914d56640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4910100080 0x7f4910068e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.471+0000 7f4914d56640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4910101270 0x7f491019be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.471+0000 7f4914d56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4910101c30 0x7f491019e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.471+0000 7f4914d56640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491019ec80 con 0x7f4910101c30 2024-03-11T19:21:25.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.471+0000 7f4914d56640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491019edf0 con 0x7f4910100080 2024-03-11T19:21:25.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.471+0000 7f4914d56640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491019efb0 con 0x7f4910101270 2024-03-11T19:21:25.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.471+0000 7f490ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4910101c30 0x7f491019e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.471+0000 7f490ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4910101c30 0x7f491019e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44518/0 (socket says 172.21.15.29:44518) 2024-03-11T19:21:25.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.471+0000 7f490ed9d640 1 -- 172.21.15.29:0/602831078 learned_addr learned my addr 172.21.15.29:0/602831078 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.472+0000 7f490e59c640 1 --2- 172.21.15.29:0/602831078 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4910100080 0x7f4910068e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.472+0000 7f490dd9b640 1 --2- 172.21.15.29:0/602831078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4910101270 0x7f491019be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.472+0000 7f490ed9d640 1 -- 172.21.15.29:0/602831078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4910101270 msgr2=0x7f491019be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.472+0000 7f490ed9d640 1 --2- 172.21.15.29:0/602831078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4910101270 0x7f491019be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.472+0000 7f490ed9d640 1 -- 172.21.15.29:0/602831078 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4910100080 msgr2=0x7f4910068e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.472+0000 7f490ed9d640 1 --2- 172.21.15.29:0/602831078 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4910100080 0x7f4910068e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.472+0000 7f490ed9d640 1 -- 172.21.15.29:0/602831078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4900009900 con 0x7f4910101c30 2024-03-11T19:21:25.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.472+0000 7f490ed9d640 1 --2- 172.21.15.29:0/602831078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4910101c30 0x7f491019e4a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f4900037040 tx=0x7f4900032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.472+0000 7f48ff7fe640 1 -- 172.21.15.29:0/602831078 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4900034e70 con 0x7f4910101c30 2024-03-11T19:21:25.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.472+0000 7f4914d56640 1 -- 172.21.15.29:0/602831078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f491019f1d0 con 0x7f4910101c30 2024-03-11T19:21:25.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.472+0000 7f48ff7fe640 1 -- 172.21.15.29:0/602831078 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4900003e20 con 0x7f4910101c30 2024-03-11T19:21:25.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.472+0000 7f48ff7fe640 1 -- 172.21.15.29:0/602831078 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490003d3f0 con 0x7f4910101c30 2024-03-11T19:21:25.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.472+0000 7f4914d56640 1 -- 172.21.15.29:0/602831078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f491019f6d0 con 0x7f4910101c30 2024-03-11T19:21:25.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.475+0000 7f48ff7fe640 1 -- 172.21.15.29:0/602831078 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f490003b040 con 0x7f4910101c30 2024-03-11T19:21:25.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.475+0000 7f4917a07740 1 Processor -- start 2024-03-11T19:21:25.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.475+0000 7f4917a07740 1 -- start start 2024-03-11T19:21:25.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.475+0000 7f4917a07740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56002af56270 0x56002af56690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.475+0000 7f4917a07740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56c60 0x56002af5b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.475+0000 7f4917a07740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af5bdd0 0x56002af5e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.475+0000 7f4917a07740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56002af12960 con 0x56002af56c60 2024-03-11T19:21:25.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.475+0000 7f4917a07740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56002aec6310 con 0x56002af5bdd0 2024-03-11T19:21:25.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.475+0000 7f4917a07740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56002aec3840 con 0x56002af56270 2024-03-11T19:21:25.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.475+0000 7f490e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56c60 0x56002af5b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.476+0000 7f490e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56c60 0x56002af5b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44524/0 (socket says 172.21.15.29:44524) 2024-03-11T19:21:25.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.476+0000 7f490e59c640 1 -- 172.21.15.29:0/2325841584 learned_addr learned my addr 172.21.15.29:0/2325841584 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.476+0000 7f48ff7fe640 1 --2- 172.21.15.29:0/602831078 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f48f80a1a30 0x7f48f80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.476+0000 7f490dd9b640 1 --2- 172.21.15.29:0/2325841584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56002af56270 0x56002af56690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.476+0000 7f490e59c640 1 -- 172.21.15.29:0/2325841584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56002af56270 msgr2=0x56002af56690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.476+0000 7f490dd9b640 1 --2- 172.21.15.29:0/2325841584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af5bdd0 0x56002af5e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.476+0000 7f490e59c640 1 --2- 172.21.15.29:0/2325841584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56002af56270 0x56002af56690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.476+0000 7f490e59c640 1 -- 172.21.15.29:0/2325841584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af5bdd0 msgr2=0x56002af5e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.476+0000 7f490e59c640 1 --2- 172.21.15.29:0/2325841584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af5bdd0 0x56002af5e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.476+0000 7f490e59c640 1 -- 172.21.15.29:0/2325841584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56002ae81060 con 0x56002af56c60 2024-03-11T19:21:25.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.476+0000 7f490ed9d640 1 --2- 172.21.15.29:0/602831078 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f48f80a1a30 0x7f48f80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.476+0000 7f490dd9b640 1 --2- 172.21.15.29:0/2325841584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af5bdd0 0x56002af5e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:21:25.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.476+0000 7f48ff7fe640 1 -- 172.21.15.29:0/602831078 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f49000efa60 con 0x7f4910101c30 2024-03-11T19:21:25.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.476+0000 7f490e59c640 1 --2- 172.21.15.29:0/2325841584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56c60 0x56002af5b760 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f490800bcb0 tx=0x7f49080107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=219c77dac8901a6f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.476+0000 7f48fcff9640 1 -- 172.21.15.29:0/2325841584 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4908004020 con 0x56002af56c60 2024-03-11T19:21:25.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.476+0000 7f48fcff9640 1 -- 172.21.15.29:0/2325841584 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4908015070 con 0x56002af56c60 2024-03-11T19:21:25.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.476+0000 7f48fcff9640 1 -- 172.21.15.29:0/2325841584 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49080049c0 con 0x56002af56c60 2024-03-11T19:21:25.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.476+0000 7f490ed9d640 1 --2- 172.21.15.29:0/602831078 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f48f80a1a30 0x7f48f80a3ef0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f49000098d0 tx=0x7f490004bbd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.477+0000 7f4917a07740 1 -- 172.21.15.29:0/2325841584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56c60 msgr2=0x56002af5b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.477+0000 7f4917a07740 1 --2- 172.21.15.29:0/2325841584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56c60 0x56002af5b760 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f490800bcb0 tx=0x7f49080107a0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.477+0000 7f4917a07740 1 -- 172.21.15.29:0/2325841584 shutdown_connections 2024-03-11T19:21:25.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.477+0000 7f4917a07740 1 --2- 172.21.15.29:0/2325841584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af5bdd0 0x56002af5e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.477+0000 7f4917a07740 1 --2- 172.21.15.29:0/2325841584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56c60 0x56002af5b760 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.477+0000 7f4917a07740 1 --2- 172.21.15.29:0/2325841584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56002af56270 0x56002af56690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.477+0000 7f4917a07740 1 -- 172.21.15.29:0/2325841584 >> 172.21.15.29:0/2325841584 conn(0x56002af517e0 msgr2=0x56002af51be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.477+0000 7f4917a07740 1 -- 172.21.15.29:0/2325841584 shutdown_connections 2024-03-11T19:21:25.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.477+0000 7f4917a07740 1 -- 172.21.15.29:0/2325841584 wait complete. 2024-03-11T19:21:25.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.477+0000 7f4917a07740 1 -- start start 2024-03-11T19:21:25.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.477+0000 7f4917a07740 1 Processor -- start 2024-03-11T19:21:25.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f4917a07740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56002af56270 0x56002af60450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f4917a07740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af56c60 0x56002af60990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f4917a07740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af5bdd0 0x56002af679d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f4917a07740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56002aebf020 con 0x56002af5bdd0 2024-03-11T19:21:25.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f4917a07740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56002ab92430 con 0x56002af56c60 2024-03-11T19:21:25.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f4917a07740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56002ab4ee00 con 0x56002af56270 2024-03-11T19:21:25.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f4917a07740 10 client.0 _ll_register_callbacks cb 0x56002ae66cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:21:25.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f490dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af5bdd0 0x56002af679d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f490dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af5bdd0 0x56002af679d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44534/0 (socket says 172.21.15.29:44534) 2024-03-11T19:21:25.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f490dd9b640 1 -- 172.21.15.29:0/2792779404 learned_addr learned my addr 172.21.15.29:0/2792779404 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f490dd9b640 1 --2- 172.21.15.29:0/2792779404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56002af56270 0x56002af60450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f490e59c640 1 --2- 172.21.15.29:0/2792779404 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af56c60 0x56002af60990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f490dd9b640 1 -- 172.21.15.29:0/2792779404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56002af56270 msgr2=0x56002af60450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f490dd9b640 1 --2- 172.21.15.29:0/2792779404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56002af56270 0x56002af60450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f490dd9b640 1 -- 172.21.15.29:0/2792779404 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af56c60 msgr2=0x56002af60990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f490dd9b640 1 --2- 172.21.15.29:0/2792779404 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af56c60 0x56002af60990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f490dd9b640 1 -- 172.21.15.29:0/2792779404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490800b8e0 con 0x56002af5bdd0 2024-03-11T19:21:25.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f490dd9b640 1 --2- 172.21.15.29:0/2792779404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af5bdd0 0x56002af679d0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f4904012aa0 tx=0x7f49040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efdf7406aefaa590 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f48f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:21:25.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f48f5ffb640 1 -- 172.21.15.29:0/2792779404 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4904004310 con 0x56002af5bdd0 2024-03-11T19:21:25.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f48f5ffb640 1 -- 172.21.15.29:0/2792779404 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4904004470 con 0x56002af5bdd0 2024-03-11T19:21:25.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.478+0000 7f4917a07740 1 -- 172.21.15.29:0/2792779404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56002abbe7e0 con 0x56002af5bdd0 2024-03-11T19:21:25.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.479+0000 7f48f5ffb640 1 -- 172.21.15.29:0/2792779404 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4904010890 con 0x56002af5bdd0 2024-03-11T19:21:25.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.479+0000 7f4917a07740 10 client.18159 fetch_fsmap learned FSMap version 27 2024-03-11T19:21:25.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.479+0000 7f4917a07740 1 -- 172.21.15.29:0/2792779404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56002aebd6a0 con 0x56002af5bdd0 2024-03-11T19:21:25.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.479+0000 7f48f5ffb640 1 -- 172.21.15.29:0/2792779404 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x56002abbe7e0 con 0x56002af5bdd0 2024-03-11T19:21:25.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.479+0000 7f48f5ffb640 1 -- 172.21.15.29:0/2792779404 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4904005020 con 0x56002af5bdd0 2024-03-11T19:21:25.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.479+0000 7f4917a07740 10 client.18159 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:21:25.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.479+0000 7f4917a07740 10 client.18159 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:21:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.479+0000 7f4917a07740 20 client.18159 populate_metadata read hostname 'smithi029' 2024-03-11T19:21:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.479+0000 7f4917a07740 1 --2- 172.21.15.29:0/2792779404 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x56002af6c5c0 0x56002af6e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.479+0000 7f4917a07740 4 client.18159 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:21:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.479+0000 7f4917a07740 1 -- 172.21.15.29:0/2792779404 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56002abfb720 con 0x56002af6c5c0 2024-03-11T19:21:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.479+0000 7f490e59c640 1 --2- 172.21.15.29:0/2792779404 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x56002af6c5c0 0x56002af6e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.480+0000 7f490e59c640 1 --2- 172.21.15.29:0/2792779404 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x56002af6c5c0 0x56002af6e9b0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a4a06859abc6ac6d server_cookie=e729a08f1f355596 in_seq=0 out_seq=0 2024-03-11T19:21:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.480+0000 7f48f5ffb640 10 client.18159 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:21:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.480+0000 7f48f5ffb640 1 -- 172.21.15.29:0/2792779404 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56002abfb720 con 0x56002af6c5c0 2024-03-11T19:21:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.480+0000 7f48f5ffb640 10 client.18159 handle_command_reply: tid=0 2024-03-11T19:21:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.480+0000 7f4917a07740 1 client.18159 shutdown 2024-03-11T19:21:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.480+0000 7f4917a07740 1 -- 172.21.15.29:0/2792779404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af5bdd0 msgr2=0x56002af679d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.480+0000 7f4917a07740 1 --2- 172.21.15.29:0/2792779404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af5bdd0 0x56002af679d0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f4904012aa0 tx=0x7f49040046c0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.480+0000 7f4917a07740 1 -- 172.21.15.29:0/2792779404 shutdown_connections 2024-03-11T19:21:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.480+0000 7f4917a07740 1 --2- 172.21.15.29:0/2792779404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af5bdd0 0x56002af679d0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.480+0000 7f4917a07740 1 --2- 172.21.15.29:0/2792779404 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x56002af6c5c0 0x56002af6e9b0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.480+0000 7f4917a07740 1 --2- 172.21.15.29:0/2792779404 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af56c60 0x56002af60990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.480+0000 7f4917a07740 1 --2- 172.21.15.29:0/2792779404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56002af56270 0x56002af60450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.480+0000 7f48f5ffb640 0 client.18159 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:21:25.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.480+0000 7f4917a07740 1 -- 172.21.15.29:0/2792779404 >> 172.21.15.29:0/2792779404 conn(0x56002af517e0 msgr2=0x56002af50b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.481+0000 7f4917a07740 1 -- 172.21.15.29:0/2792779404 shutdown_connections 2024-03-11T19:21:25.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.481+0000 7f4917a07740 1 -- 172.21.15.29:0/2792779404 wait complete. 2024-03-11T19:21:25.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.481+0000 7f4917a07740 20 client.18159 trim_cache size 0 max 16384 2024-03-11T19:21:25.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.492+0000 7f4917a07740 1 Processor -- start 2024-03-11T19:21:25.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.492+0000 7f4917a07740 1 -- start start 2024-03-11T19:21:25.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.492+0000 7f4917a07740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56002af56270 0x56002af581b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.492+0000 7f4917a07740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56c60 0x56002af586f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.492+0000 7f4917a07740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af5bdd0 0x56002af67990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.492+0000 7f4917a07740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56002ae44130 con 0x56002af56c60 2024-03-11T19:21:25.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.492+0000 7f4917a07740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56002af61640 con 0x56002af5bdd0 2024-03-11T19:21:25.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.492+0000 7f4917a07740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56002af57090 con 0x56002af56270 2024-03-11T19:21:25.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.493+0000 7f490e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56c60 0x56002af586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.493+0000 7f490e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56c60 0x56002af586f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44542/0 (socket says 172.21.15.29:44542) 2024-03-11T19:21:25.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.493+0000 7f490e59c640 1 -- 172.21.15.29:0/96621812 learned_addr learned my addr 172.21.15.29:0/96621812 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.493+0000 7f490dd9b640 1 --2- 172.21.15.29:0/96621812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56002af56270 0x56002af581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.493+0000 7f490e59c640 1 -- 172.21.15.29:0/96621812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56002af56270 msgr2=0x56002af581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.493+0000 7f490e59c640 1 --2- 172.21.15.29:0/96621812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56002af56270 0x56002af581b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.493+0000 7f490dd9b640 1 --2- 172.21.15.29:0/96621812 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af5bdd0 0x56002af67990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.493+0000 7f490e59c640 1 -- 172.21.15.29:0/96621812 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af5bdd0 msgr2=0x56002af67990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.493+0000 7f490e59c640 1 --2- 172.21.15.29:0/96621812 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af5bdd0 0x56002af67990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.493+0000 7f490e59c640 1 -- 172.21.15.29:0/96621812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56002ae59c20 con 0x56002af56c60 2024-03-11T19:21:25.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.493+0000 7f490e59c640 1 --2- 172.21.15.29:0/96621812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56c60 0x56002af586f0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x56002af576d0 tx=0x7f4908000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af38b03428904bf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.493+0000 7f48f7fff640 1 -- 172.21.15.29:0/96621812 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4908021050 con 0x56002af56c60 2024-03-11T19:21:25.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.493+0000 7f48f7fff640 1 -- 172.21.15.29:0/96621812 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4908022040 con 0x56002af56c60 2024-03-11T19:21:25.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.493+0000 7f48f7fff640 1 -- 172.21.15.29:0/96621812 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49080035f0 con 0x56002af56c60 2024-03-11T19:21:25.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.494+0000 7f4917a07740 1 -- 172.21.15.29:0/96621812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56c60 msgr2=0x56002af586f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.494+0000 7f4917a07740 1 --2- 172.21.15.29:0/96621812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56c60 0x56002af586f0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x56002af576d0 tx=0x7f4908000bc0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.494+0000 7f4917a07740 1 -- 172.21.15.29:0/96621812 shutdown_connections 2024-03-11T19:21:25.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.494+0000 7f4917a07740 1 --2- 172.21.15.29:0/96621812 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af5bdd0 0x56002af67990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.494+0000 7f4917a07740 1 --2- 172.21.15.29:0/96621812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56c60 0x56002af586f0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.494+0000 7f4917a07740 1 --2- 172.21.15.29:0/96621812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56002af56270 0x56002af581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.494+0000 7f4917a07740 1 -- 172.21.15.29:0/96621812 >> 172.21.15.29:0/96621812 conn(0x56002af517e0 msgr2=0x56002af73350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.494+0000 7f4917a07740 1 -- 172.21.15.29:0/96621812 shutdown_connections 2024-03-11T19:21:25.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.494+0000 7f4917a07740 1 -- 172.21.15.29:0/96621812 wait complete. 2024-03-11T19:21:25.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.494+0000 7f4917a07740 1 -- start start 2024-03-11T19:21:25.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.494+0000 7f4917a07740 1 Processor -- start 2024-03-11T19:21:25.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f4917a07740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56270 0x56002af67420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f4917a07740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af56c60 0x56002af67960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f4917a07740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56002af5bdd0 0x56002af65a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f4917a07740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56002af66100 con 0x56002af56270 2024-03-11T19:21:25.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f4917a07740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56002af66270 con 0x56002af56c60 2024-03-11T19:21:25.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f4917a07740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56002af66430 con 0x56002af5bdd0 2024-03-11T19:21:25.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f4917a07740 10 client.0 _ll_register_callbacks cb 0x7f490800b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:21:25.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f490dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56270 0x56002af67420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f490dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56270 0x56002af67420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44552/0 (socket says 172.21.15.29:44552) 2024-03-11T19:21:25.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f490dd9b640 1 -- 172.21.15.29:0/3455948263 learned_addr learned my addr 172.21.15.29:0/3455948263 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f490dd9b640 1 -- 172.21.15.29:0/3455948263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56002af5bdd0 msgr2=0x56002af65a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f490dd9b640 1 --2- 172.21.15.29:0/3455948263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56002af5bdd0 0x56002af65a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f490dd9b640 1 -- 172.21.15.29:0/3455948263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af56c60 msgr2=0x56002af67960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f490dd9b640 1 --2- 172.21.15.29:0/3455948263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af56c60 0x56002af67960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f490dd9b640 1 -- 172.21.15.29:0/3455948263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56002ae81060 con 0x56002af56270 2024-03-11T19:21:25.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f490dd9b640 1 --2- 172.21.15.29:0/3455948263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56270 0x56002af67420 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f4904012aa0 tx=0x7f49040129b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec329b34ac107f17 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f48f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:21:25.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f48f5ffb640 1 -- 172.21.15.29:0/3455948263 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4904010860 con 0x56002af56270 2024-03-11T19:21:25.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f48f5ffb640 1 -- 172.21.15.29:0/3455948263 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f4904000e40 con 0x56002af56270 2024-03-11T19:21:25.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f4917a07740 1 -- 172.21.15.29:0/3455948263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4908015070 con 0x56002af56270 2024-03-11T19:21:25.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f48f5ffb640 1 -- 172.21.15.29:0/3455948263 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49040109c0 con 0x56002af56270 2024-03-11T19:21:25.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f4917a07740 10 client.18165 fetch_fsmap learned FSMap version 27 2024-03-11T19:21:25.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.495+0000 7f4917a07740 1 -- 172.21.15.29:0/3455948263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56002ae7f8c0 con 0x56002af56270 2024-03-11T19:21:25.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.496+0000 7f48f5ffb640 1 -- 172.21.15.29:0/3455948263 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4908015070 con 0x56002af56270 2024-03-11T19:21:25.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.496+0000 7f48f5ffb640 1 -- 172.21.15.29:0/3455948263 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4904003c30 con 0x56002af56270 2024-03-11T19:21:25.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.496+0000 7f4917a07740 10 client.18165 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:21:25.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.496+0000 7f4917a07740 10 client.18165 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-11T19:21:25.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.496+0000 7f4917a07740 20 client.18165 populate_metadata read hostname 'smithi029' 2024-03-11T19:21:25.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.496+0000 7f4917a07740 1 --2- 172.21.15.29:0/3455948263 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x56002afc3f90 0x56002afc63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.496+0000 7f4917a07740 4 client.18165 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:21:25.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.496+0000 7f4917a07740 1 -- 172.21.15.29:0/3455948263 --> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56002ac4a890 con 0x56002afc3f90 2024-03-11T19:21:25.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.496+0000 7f490e59c640 1 --2- 172.21.15.29:0/3455948263 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x56002afc3f90 0x56002afc63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.496+0000 7f48f5ffb640 10 client.18165 ms_handle_connect on v2:172.21.15.190:6838/2031437522 2024-03-11T19:21:25.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.496+0000 7f490e59c640 1 --2- 172.21.15.29:0/3455948263 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x56002afc3f90 0x56002afc63b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d9e0f34597865291 server_cookie=d2ff51f624c1fc07 in_seq=0 out_seq=0 2024-03-11T19:21:25.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f48f5ffb640 1 -- 172.21.15.29:0/3455948263 <== mds.1 v2:172.21.15.190:6838/2031437522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56002ac4a890 con 0x56002afc3f90 2024-03-11T19:21:25.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f48f5ffb640 10 client.18165 handle_command_reply: tid=0 2024-03-11T19:21:25.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4917a07740 1 client.18165 shutdown 2024-03-11T19:21:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4917a07740 1 -- 172.21.15.29:0/3455948263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56270 msgr2=0x56002af67420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4917a07740 1 --2- 172.21.15.29:0/3455948263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56270 0x56002af67420 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f4904012aa0 tx=0x7f49040129b0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4917a07740 1 -- 172.21.15.29:0/3455948263 shutdown_connections 2024-03-11T19:21:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4917a07740 1 --2- 172.21.15.29:0/3455948263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56002af5bdd0 0x56002af65a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4917a07740 1 --2- 172.21.15.29:0/3455948263 >> [v2:172.21.15.190:6838/2031437522,v1:172.21.15.190:6839/2031437522] conn(0x56002afc3f90 0x56002afc63b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4917a07740 1 --2- 172.21.15.29:0/3455948263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002af56c60 0x56002af67960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f48f5ffb640 0 client.18165 ms_handle_reset on v2:172.21.15.190:6838/2031437522 2024-03-11T19:21:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4917a07740 1 --2- 172.21.15.29:0/3455948263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002af56270 0x56002af67420 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4917a07740 1 -- 172.21.15.29:0/3455948263 >> 172.21.15.29:0/3455948263 conn(0x56002af517e0 msgr2=0x56002af581b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4917a07740 1 -- 172.21.15.29:0/3455948263 shutdown_connections 2024-03-11T19:21:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4917a07740 1 -- 172.21.15.29:0/3455948263 wait complete. 2024-03-11T19:21:25.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4917a07740 20 client.18165 trim_cache size 0 max 16384 2024-03-11T19:21:25.507 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:21:25.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4914d56640 1 -- 172.21.15.29:0/602831078 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f48f80a1a30 msgr2=0x7f48f80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4914d56640 1 --2- 172.21.15.29:0/602831078 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f48f80a1a30 0x7f48f80a3ef0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f49000098d0 tx=0x7f490004bbd0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4914d56640 1 -- 172.21.15.29:0/602831078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4910101c30 msgr2=0x7f491019e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4914d56640 1 --2- 172.21.15.29:0/602831078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4910101c30 0x7f491019e4a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f4900037040 tx=0x7f4900032980 comp rx=0 tx=0).stop 2024-03-11T19:21:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4914d56640 1 -- 172.21.15.29:0/602831078 shutdown_connections 2024-03-11T19:21:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4914d56640 1 --2- 172.21.15.29:0/602831078 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f48f80a1a30 0x7f48f80a3ef0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4914d56640 1 --2- 172.21.15.29:0/602831078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4910101c30 0x7f491019e4a0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4914d56640 1 --2- 172.21.15.29:0/602831078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4910101270 0x7f491019be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4914d56640 1 --2- 172.21.15.29:0/602831078 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4910100080 0x7f4910068e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.497+0000 7f4914d56640 1 -- 172.21.15.29:0/602831078 >> 172.21.15.29:0/602831078 conn(0x7f49100fb610 msgr2=0x7f49101045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.498+0000 7f4914d56640 1 -- 172.21.15.29:0/602831078 shutdown_connections 2024-03-11T19:21:25.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.498+0000 7f4914d56640 1 -- 172.21.15.29:0/602831078 wait complete. 2024-03-11T19:21:25.511 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:25 smithi029 ceph-mon[32439]: pgmap v699: 97 pgs: 97 active+clean; 421 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 30 MiB/s wr, 752 op/s 2024-03-11T19:21:25.511 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:25 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3110351722' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:21:25.512 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-11T19:21:25.512 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-11T19:21:25.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.614+0000 7f7194aeb640 1 -- 172.21.15.29:0/3424280156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7190102d50 msgr2=0x7f71901095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.614+0000 7f7194aeb640 1 --2- 172.21.15.29:0/3424280156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7190102d50 0x7f71901095e0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f7180009ca0 tx=0x7f718002c0f0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.614+0000 7f7194aeb640 1 -- 172.21.15.29:0/3424280156 shutdown_connections 2024-03-11T19:21:25.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.614+0000 7f7194aeb640 1 --2- 172.21.15.29:0/3424280156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7190102d50 0x7f71901095e0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.614+0000 7f7194aeb640 1 --2- 172.21.15.29:0/3424280156 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7190102390 0x7f7190102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.614+0000 7f7194aeb640 1 --2- 172.21.15.29:0/3424280156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71901011a0 0x7f71901015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.614+0000 7f7194aeb640 1 -- 172.21.15.29:0/3424280156 >> 172.21.15.29:0/3424280156 conn(0x7f71900fc6f0 msgr2=0x7f71900feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.614+0000 7f7194aeb640 1 -- 172.21.15.29:0/3424280156 shutdown_connections 2024-03-11T19:21:25.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.614+0000 7f7194aeb640 1 -- 172.21.15.29:0/3424280156 wait complete. 2024-03-11T19:21:25.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.615+0000 7f7194aeb640 1 Processor -- start 2024-03-11T19:21:25.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.615+0000 7f7194aeb640 1 -- start start 2024-03-11T19:21:25.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.615+0000 7f7194aeb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71901011a0 0x7f7190197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.615+0000 7f7194aeb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7190102390 0x7f7190197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.615+0000 7f7194aeb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7190102d50 0x7f719019e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.615+0000 7f7194aeb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719019ebf0 con 0x7f71901011a0 2024-03-11T19:21:25.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.615+0000 7f7194aeb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719019ed60 con 0x7f7190102390 2024-03-11T19:21:25.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.615+0000 7f7194aeb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719019ef20 con 0x7f7190102d50 2024-03-11T19:21:25.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.615+0000 7f718e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71901011a0 0x7f7190197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.615+0000 7f718e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71901011a0 0x7f7190197100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44562/0 (socket says 172.21.15.29:44562) 2024-03-11T19:21:25.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.615+0000 7f718e59c640 1 -- 172.21.15.29:0/2729884201 learned_addr learned my addr 172.21.15.29:0/2729884201 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.615+0000 7f718dd9b640 1 --2- 172.21.15.29:0/2729884201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7190102390 0x7f7190197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.615+0000 7f718ed9d640 1 --2- 172.21.15.29:0/2729884201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7190102d50 0x7f719019e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.615+0000 7f718e59c640 1 -- 172.21.15.29:0/2729884201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7190102d50 msgr2=0x7f719019e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.615+0000 7f718e59c640 1 --2- 172.21.15.29:0/2729884201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7190102d50 0x7f719019e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.615+0000 7f718e59c640 1 -- 172.21.15.29:0/2729884201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7190102390 msgr2=0x7f7190197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.615+0000 7f718e59c640 1 --2- 172.21.15.29:0/2729884201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7190102390 0x7f7190197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.615+0000 7f718e59c640 1 -- 172.21.15.29:0/2729884201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7180009900 con 0x7f71901011a0 2024-03-11T19:21:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.615+0000 7f718dd9b640 1 --2- 172.21.15.29:0/2729884201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7190102390 0x7f7190197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:21:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.616+0000 7f718e59c640 1 --2- 172.21.15.29:0/2729884201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71901011a0 0x7f7190197100 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f718800d770 tx=0x7f718800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.616+0000 7f71877fe640 1 -- 172.21.15.29:0/2729884201 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7188009c30 con 0x7f71901011a0 2024-03-11T19:21:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.616+0000 7f71877fe640 1 -- 172.21.15.29:0/2729884201 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f71880044d0 con 0x7f71901011a0 2024-03-11T19:21:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.616+0000 7f7194aeb640 1 -- 172.21.15.29:0/2729884201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f719019f200 con 0x7f71901011a0 2024-03-11T19:21:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.616+0000 7f71877fe640 1 -- 172.21.15.29:0/2729884201 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7188003de0 con 0x7f71901011a0 2024-03-11T19:21:25.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.616+0000 7f7194aeb640 1 -- 172.21.15.29:0/2729884201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719019f710 con 0x7f71901011a0 2024-03-11T19:21:25.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.618+0000 7f71877fe640 1 -- 172.21.15.29:0/2729884201 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7188003f40 con 0x7f71901011a0 2024-03-11T19:21:25.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.618+0000 7f7197799740 1 Processor -- start 2024-03-11T19:21:25.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.618+0000 7f7197799740 1 -- start start 2024-03-11T19:21:25.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.618+0000 7f71877fe640 1 --2- 172.21.15.29:0/2729884201 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f717c0a1a30 0x7f717c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f7197799740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b62680 0x558353b62aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f7197799740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b63070 0x558353b67b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f7197799740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b681e0 0x558353b6a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f7197799740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558353b1a960 con 0x558353b63070 2024-03-11T19:21:25.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f7197799740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558353ace310 con 0x558353b62680 2024-03-11T19:21:25.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f7197799740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558353acb840 con 0x558353b681e0 2024-03-11T19:21:25.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f718ed9d640 1 --2- 172.21.15.29:0/2729884201 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f717c0a1a30 0x7f717c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f71877fe640 1 -- 172.21.15.29:0/2729884201 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f71880cce20 con 0x7f71901011a0 2024-03-11T19:21:25.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f718dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b63070 0x558353b67b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f718dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b63070 0x558353b67b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44566/0 (socket says 172.21.15.29:44566) 2024-03-11T19:21:25.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f718dd9b640 1 -- 172.21.15.29:0/977533911 learned_addr learned my addr 172.21.15.29:0/977533911 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f718ed9d640 1 --2- 172.21.15.29:0/977533911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b681e0 0x558353b6a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f718e59c640 1 --2- 172.21.15.29:0/977533911 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b62680 0x558353b62aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f718dd9b640 1 -- 172.21.15.29:0/977533911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b681e0 msgr2=0x558353b6a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f718dd9b640 1 --2- 172.21.15.29:0/977533911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b681e0 0x558353b6a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f718dd9b640 1 -- 172.21.15.29:0/977533911 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b62680 msgr2=0x558353b62aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f718dd9b640 1 --2- 172.21.15.29:0/977533911 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b62680 0x558353b62aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f718dd9b640 1 -- 172.21.15.29:0/977533911 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558353a89060 con 0x558353b63070 2024-03-11T19:21:25.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f718ed9d640 1 --2- 172.21.15.29:0/2729884201 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f717c0a1a30 0x7f717c0a3ef0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7180006010 tx=0x7f7180032fc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f718dd9b640 1 --2- 172.21.15.29:0/977533911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b63070 0x558353b67b70 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f717800bd30 tx=0x7f717800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac9f5d8172ed0a31 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f7184ff9640 1 -- 172.21.15.29:0/977533911 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717801a660 con 0x558353b63070 2024-03-11T19:21:25.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f7184ff9640 1 -- 172.21.15.29:0/977533911 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f717800be20 con 0x558353b63070 2024-03-11T19:21:25.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.619+0000 7f7184ff9640 1 -- 172.21.15.29:0/977533911 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7178011b10 con 0x558353b63070 2024-03-11T19:21:25.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.620+0000 7f7197799740 1 -- 172.21.15.29:0/977533911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b63070 msgr2=0x558353b67b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.620+0000 7f7197799740 1 --2- 172.21.15.29:0/977533911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b63070 0x558353b67b70 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f717800bd30 tx=0x7f717800f730 comp rx=0 tx=0).stop 2024-03-11T19:21:25.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.620+0000 7f7197799740 1 -- 172.21.15.29:0/977533911 shutdown_connections 2024-03-11T19:21:25.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.620+0000 7f7197799740 1 --2- 172.21.15.29:0/977533911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b681e0 0x558353b6a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.620+0000 7f7197799740 1 --2- 172.21.15.29:0/977533911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b63070 0x558353b67b70 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.620+0000 7f7197799740 1 --2- 172.21.15.29:0/977533911 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b62680 0x558353b62aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.620+0000 7f7197799740 1 -- 172.21.15.29:0/977533911 >> 172.21.15.29:0/977533911 conn(0x558353b597e0 msgr2=0x558353b59be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.620+0000 7f7197799740 1 -- 172.21.15.29:0/977533911 shutdown_connections 2024-03-11T19:21:25.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.620+0000 7f7197799740 1 -- 172.21.15.29:0/977533911 wait complete. 2024-03-11T19:21:25.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f7197799740 1 -- start start 2024-03-11T19:21:25.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f7197799740 1 Processor -- start 2024-03-11T19:21:25.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f7197799740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b62680 0x558353b6c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f7197799740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b63070 0x558353b6cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f7197799740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b681e0 0x558353b73d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f7197799740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558353ac7020 con 0x558353b62680 2024-03-11T19:21:25.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f7197799740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55835379a430 con 0x558353b681e0 2024-03-11T19:21:25.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f7197799740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558353756e00 con 0x558353b63070 2024-03-11T19:21:25.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f7197799740 10 client.0 _ll_register_callbacks cb 0x558353a6ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:21:25.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f718e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b62680 0x558353b6c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f718e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b62680 0x558353b6c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44574/0 (socket says 172.21.15.29:44574) 2024-03-11T19:21:25.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f718e59c640 1 -- 172.21.15.29:0/572337243 learned_addr learned my addr 172.21.15.29:0/572337243 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f718ed9d640 1 --2- 172.21.15.29:0/572337243 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b681e0 0x558353b73d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f718dd9b640 1 --2- 172.21.15.29:0/572337243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b63070 0x558353b6cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f718e59c640 1 -- 172.21.15.29:0/572337243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b63070 msgr2=0x558353b6cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f718e59c640 1 --2- 172.21.15.29:0/572337243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b63070 0x558353b6cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f718e59c640 1 -- 172.21.15.29:0/572337243 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b681e0 msgr2=0x558353b73d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f718e59c640 1 --2- 172.21.15.29:0/572337243 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b681e0 0x558353b73d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f718e59c640 1 -- 172.21.15.29:0/572337243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f717800b960 con 0x558353b62680 2024-03-11T19:21:25.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f718ed9d640 1 --2- 172.21.15.29:0/572337243 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b681e0 0x558353b73d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:21:25.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f718dd9b640 1 --2- 172.21.15.29:0/572337243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b63070 0x558353b6cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:21:25.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f718e59c640 1 --2- 172.21.15.29:0/572337243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b62680 0x558353b6c7e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f7188099d20 tx=0x7f71880999c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a593da88873b1e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.621+0000 7f7175ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:21:25.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.622+0000 7f7175ffb640 1 -- 172.21.15.29:0/572337243 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7188099d90 con 0x558353b62680 2024-03-11T19:21:25.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.622+0000 7f7175ffb640 1 -- 172.21.15.29:0/572337243 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f718809dee0 con 0x558353b62680 2024-03-11T19:21:25.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.622+0000 7f7197799740 1 -- 172.21.15.29:0/572337243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5583537c67e0 con 0x558353b62680 2024-03-11T19:21:25.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.622+0000 7f7175ffb640 1 -- 172.21.15.29:0/572337243 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71880a2530 con 0x558353b62680 2024-03-11T19:21:25.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.622+0000 7f7197799740 10 client.18177 fetch_fsmap learned FSMap version 27 2024-03-11T19:21:25.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.622+0000 7f7197799740 1 -- 172.21.15.29:0/572337243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x558353ac56a0 con 0x558353b62680 2024-03-11T19:21:25.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.622+0000 7f7175ffb640 1 -- 172.21.15.29:0/572337243 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f71880b1020 con 0x558353b62680 2024-03-11T19:21:25.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.622+0000 7f7175ffb640 1 -- 172.21.15.29:0/572337243 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f71880a20a0 con 0x558353b62680 2024-03-11T19:21:25.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.622+0000 7f7197799740 10 client.18177 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:21:25.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.622+0000 7f7197799740 10 client.18177 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:21:25.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.622+0000 7f7197799740 20 client.18177 populate_metadata read hostname 'smithi029' 2024-03-11T19:21:25.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.622+0000 7f7197799740 1 --2- 172.21.15.29:0/572337243 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558353b78a80 0x558353b7ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.622+0000 7f7197799740 4 client.18177 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:21:25.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.622+0000 7f7197799740 1 -- 172.21.15.29:0/572337243 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558353803720 con 0x558353b78a80 2024-03-11T19:21:25.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.622+0000 7f718dd9b640 1 --2- 172.21.15.29:0/572337243 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558353b78a80 0x558353b7ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.623+0000 7f718dd9b640 1 --2- 172.21.15.29:0/572337243 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558353b78a80 0x558353b7ae90 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9116ce389c22f37 server_cookie=9bf6021bf8339d5 in_seq=0 out_seq=0 2024-03-11T19:21:25.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.623+0000 7f7175ffb640 10 client.18177 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:21:25.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.623+0000 7f7175ffb640 1 -- 172.21.15.29:0/572337243 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558353803720 con 0x558353b78a80 2024-03-11T19:21:25.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.623+0000 7f7175ffb640 10 client.18177 handle_command_reply: tid=0 2024-03-11T19:21:25.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.623+0000 7f7197799740 1 client.18177 shutdown 2024-03-11T19:21:25.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.623+0000 7f7197799740 1 -- 172.21.15.29:0/572337243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b62680 msgr2=0x558353b6c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.623+0000 7f7197799740 1 --2- 172.21.15.29:0/572337243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b62680 0x558353b6c7e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f7188099d20 tx=0x7f71880999c0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.624+0000 7f7197799740 1 -- 172.21.15.29:0/572337243 shutdown_connections 2024-03-11T19:21:25.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.624+0000 7f7197799740 1 --2- 172.21.15.29:0/572337243 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b681e0 0x558353b73d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.624+0000 7f7197799740 1 --2- 172.21.15.29:0/572337243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b63070 0x558353b6cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.624+0000 7f7197799740 1 --2- 172.21.15.29:0/572337243 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558353b78a80 0x558353b7ae90 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.624+0000 7f7197799740 1 --2- 172.21.15.29:0/572337243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b62680 0x558353b6c7e0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.624+0000 7f7175ffb640 0 client.18177 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:21:25.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.624+0000 7f7197799740 1 -- 172.21.15.29:0/572337243 >> 172.21.15.29:0/572337243 conn(0x558353b597e0 msgr2=0x558353b58b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.624+0000 7f7197799740 1 -- 172.21.15.29:0/572337243 shutdown_connections 2024-03-11T19:21:25.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.624+0000 7f7197799740 1 -- 172.21.15.29:0/572337243 wait complete. 2024-03-11T19:21:25.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.624+0000 7f7197799740 20 client.18177 trim_cache size 0 max 16384 2024-03-11T19:21:25.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.635+0000 7f7197799740 1 Processor -- start 2024-03-11T19:21:25.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.635+0000 7f7197799740 1 -- start start 2024-03-11T19:21:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.635+0000 7f7197799740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b62680 0x558353b6cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f7197799740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b63070 0x558353b73d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f7197799740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b681e0 0x558353b6b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f7197799740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558353a4c130 con 0x558353b62680 2024-03-11T19:21:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f7197799740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558353b4ba40 con 0x558353b681e0 2024-03-11T19:21:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f7197799740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558353b62320 con 0x558353b63070 2024-03-11T19:21:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f718e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b62680 0x558353b6cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f718e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b62680 0x558353b6cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44576/0 (socket says 172.21.15.29:44576) 2024-03-11T19:21:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f718e59c640 1 -- 172.21.15.29:0/693116494 learned_addr learned my addr 172.21.15.29:0/693116494 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f718ed9d640 1 --2- 172.21.15.29:0/693116494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b681e0 0x558353b6b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f718dd9b640 1 --2- 172.21.15.29:0/693116494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b63070 0x558353b73d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f718e59c640 1 -- 172.21.15.29:0/693116494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b63070 msgr2=0x558353b73d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f718e59c640 1 --2- 172.21.15.29:0/693116494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b63070 0x558353b73d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f718e59c640 1 -- 172.21.15.29:0/693116494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b681e0 msgr2=0x558353b6b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f718e59c640 1 --2- 172.21.15.29:0/693116494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b681e0 0x558353b6b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f718e59c640 1 -- 172.21.15.29:0/693116494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558353a61c20 con 0x558353b62680 2024-03-11T19:21:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f718dd9b640 1 --2- 172.21.15.29:0/693116494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b63070 0x558353b73d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:21:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f718ed9d640 1 --2- 172.21.15.29:0/693116494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b681e0 0x558353b6b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:21:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f718e59c640 1 --2- 172.21.15.29:0/693116494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b62680 0x558353b6cb00 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f7188099a90 tx=0x7f71880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d245450e3a5fc9de server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f7177fff640 1 -- 172.21.15.29:0/693116494 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718809dee0 con 0x558353b62680 2024-03-11T19:21:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f7177fff640 1 -- 172.21.15.29:0/693116494 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f71880b1020 con 0x558353b62680 2024-03-11T19:21:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.636+0000 7f7177fff640 1 -- 172.21.15.29:0/693116494 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71880ca9a0 con 0x558353b62680 2024-03-11T19:21:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f7197799740 1 -- 172.21.15.29:0/693116494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b62680 msgr2=0x558353b6cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f7197799740 1 --2- 172.21.15.29:0/693116494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b62680 0x558353b6cb00 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f7188099a90 tx=0x7f71880046c0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f7197799740 1 -- 172.21.15.29:0/693116494 shutdown_connections 2024-03-11T19:21:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f7197799740 1 --2- 172.21.15.29:0/693116494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b681e0 0x558353b6b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f7197799740 1 --2- 172.21.15.29:0/693116494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b63070 0x558353b73d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f7197799740 1 --2- 172.21.15.29:0/693116494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b62680 0x558353b6cb00 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f7197799740 1 -- 172.21.15.29:0/693116494 >> 172.21.15.29:0/693116494 conn(0x558353b597e0 msgr2=0x558353b6c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f7197799740 1 -- 172.21.15.29:0/693116494 shutdown_connections 2024-03-11T19:21:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f7197799740 1 -- 172.21.15.29:0/693116494 wait complete. 2024-03-11T19:21:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f7197799740 1 -- start start 2024-03-11T19:21:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f7197799740 1 Processor -- start 2024-03-11T19:21:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f7197799740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b62680 0x558353b73710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f7197799740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b63070 0x558353b73c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f7197799740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b681e0 0x558353b71d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f7197799740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558353b72440 con 0x558353b63070 2024-03-11T19:21:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f7197799740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558353b725b0 con 0x558353b62680 2024-03-11T19:21:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f7197799740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558353b72770 con 0x558353b681e0 2024-03-11T19:21:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f7197799740 10 client.0 _ll_register_callbacks cb 0x7f717800b4c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:21:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f718dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b63070 0x558353b73c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f718dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b63070 0x558353b73c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44586/0 (socket says 172.21.15.29:44586) 2024-03-11T19:21:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.637+0000 7f718dd9b640 1 -- 172.21.15.29:0/4224344975 learned_addr learned my addr 172.21.15.29:0/4224344975 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.638+0000 7f718ed9d640 1 --2- 172.21.15.29:0/4224344975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b681e0 0x558353b71d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.638+0000 7f718e59c640 1 --2- 172.21.15.29:0/4224344975 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b62680 0x558353b73710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.638+0000 7f718dd9b640 1 -- 172.21.15.29:0/4224344975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b681e0 msgr2=0x558353b71d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.638+0000 7f718dd9b640 1 --2- 172.21.15.29:0/4224344975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b681e0 0x558353b71d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.638+0000 7f718dd9b640 1 -- 172.21.15.29:0/4224344975 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b62680 msgr2=0x558353b73710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:21:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.638+0000 7f718dd9b640 1 --2- 172.21.15.29:0/4224344975 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b62680 0x558353b73710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.638+0000 7f718dd9b640 1 -- 172.21.15.29:0/4224344975 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558353ac56a0 con 0x558353b63070 2024-03-11T19:21:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.638+0000 7f718e59c640 1 --2- 172.21.15.29:0/4224344975 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b62680 0x558353b73710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:21:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.638+0000 7f718dd9b640 1 --2- 172.21.15.29:0/4224344975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b63070 0x558353b73c50 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f717800c010 tx=0x7f7178005a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2239fd42115ca17c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.638+0000 7f7175ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-11T19:21:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.638+0000 7f7175ffb640 1 -- 172.21.15.29:0/4224344975 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717801e040 con 0x558353b63070 2024-03-11T19:21:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.638+0000 7f7175ffb640 1 -- 172.21.15.29:0/4224344975 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7178029e40 con 0x558353b63070 2024-03-11T19:21:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.638+0000 7f7197799740 1 -- 172.21.15.29:0/4224344975 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f717800be20 con 0x558353b63070 2024-03-11T19:21:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.638+0000 7f7175ffb640 1 -- 172.21.15.29:0/4224344975 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7178011a70 con 0x558353b63070 2024-03-11T19:21:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.638+0000 7f7197799740 10 client.18183 fetch_fsmap learned FSMap version 27 2024-03-11T19:21:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.638+0000 7f7197799740 1 -- 172.21.15.29:0/4224344975 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x558353a878c0 con 0x558353b63070 2024-03-11T19:21:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.638+0000 7f7175ffb640 1 -- 172.21.15.29:0/4224344975 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f717800be20 con 0x558353b63070 2024-03-11T19:21:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.639+0000 7f7175ffb640 1 -- 172.21.15.29:0/4224344975 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 27) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7178029310 con 0x558353b63070 2024-03-11T19:21:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.639+0000 7f7197799740 10 client.18183 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T19:21:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.639+0000 7f7197799740 10 client.18183 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-11T19:21:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.639+0000 7f7197799740 20 client.18183 populate_metadata read hostname 'smithi029' 2024-03-11T19:21:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.639+0000 7f7197799740 1 --2- 172.21.15.29:0/4224344975 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558353bd03b0 0x558353bd27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.639+0000 7f7197799740 4 client.18183 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-11T19:21:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.639+0000 7f7197799740 1 -- 172.21.15.29:0/4224344975 --> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558353852890 con 0x558353bd03b0 2024-03-11T19:21:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.639+0000 7f718e59c640 1 --2- 172.21.15.29:0/4224344975 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558353bd03b0 0x558353bd27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.639+0000 7f718e59c640 1 --2- 172.21.15.29:0/4224344975 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558353bd03b0 0x558353bd27d0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a24ffd0e59a003ba server_cookie=177ad9afa6e8569d in_seq=0 out_seq=0 2024-03-11T19:21:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.639+0000 7f7175ffb640 10 client.18183 ms_handle_connect on v2:172.21.15.170:6838/4145997740 2024-03-11T19:21:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.639+0000 7f7175ffb640 1 -- 172.21.15.29:0/4224344975 <== mds.0 v2:172.21.15.170:6838/4145997740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558353852890 con 0x558353bd03b0 2024-03-11T19:21:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.639+0000 7f7175ffb640 10 client.18183 handle_command_reply: tid=0 2024-03-11T19:21:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.639+0000 7f7197799740 1 client.18183 shutdown 2024-03-11T19:21:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7197799740 1 -- 172.21.15.29:0/4224344975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b63070 msgr2=0x558353b73c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7197799740 1 --2- 172.21.15.29:0/4224344975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b63070 0x558353b73c50 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f717800c010 tx=0x7f7178005a70 comp rx=0 tx=0).stop 2024-03-11T19:21:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7197799740 1 -- 172.21.15.29:0/4224344975 shutdown_connections 2024-03-11T19:21:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7197799740 1 --2- 172.21.15.29:0/4224344975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558353b681e0 0x558353b71d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7197799740 1 --2- 172.21.15.29:0/4224344975 >> [v2:172.21.15.170:6838/4145997740,v1:172.21.15.170:6839/4145997740] conn(0x558353bd03b0 0x558353bd27d0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7197799740 1 --2- 172.21.15.29:0/4224344975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558353b63070 0x558353b73c50 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7175ffb640 0 client.18183 ms_handle_reset on v2:172.21.15.170:6838/4145997740 2024-03-11T19:21:25.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7197799740 1 --2- 172.21.15.29:0/4224344975 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558353b62680 0x558353b73710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7197799740 1 -- 172.21.15.29:0/4224344975 >> 172.21.15.29:0/4224344975 conn(0x558353b597e0 msgr2=0x558353b5ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7197799740 1 -- 172.21.15.29:0/4224344975 shutdown_connections 2024-03-11T19:21:25.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7197799740 1 -- 172.21.15.29:0/4224344975 wait complete. 2024-03-11T19:21:25.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7197799740 20 client.18183 trim_cache size 0 max 16384 2024-03-11T19:21:25.650 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-03-11T19:21:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7194aeb640 1 -- 172.21.15.29:0/2729884201 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f717c0a1a30 msgr2=0x7f717c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7194aeb640 1 --2- 172.21.15.29:0/2729884201 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f717c0a1a30 0x7f717c0a3ef0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7180006010 tx=0x7f7180032fc0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7194aeb640 1 -- 172.21.15.29:0/2729884201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71901011a0 msgr2=0x7f7190197100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7194aeb640 1 --2- 172.21.15.29:0/2729884201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71901011a0 0x7f7190197100 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f718800d770 tx=0x7f718800dc30 comp rx=0 tx=0).stop 2024-03-11T19:21:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7194aeb640 1 -- 172.21.15.29:0/2729884201 shutdown_connections 2024-03-11T19:21:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7194aeb640 1 --2- 172.21.15.29:0/2729884201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7190102d50 0x7f719019e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7194aeb640 1 --2- 172.21.15.29:0/2729884201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7190102390 0x7f7190197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7194aeb640 1 --2- 172.21.15.29:0/2729884201 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f717c0a1a30 0x7f717c0a3ef0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7194aeb640 1 --2- 172.21.15.29:0/2729884201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71901011a0 0x7f7190197100 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7194aeb640 1 -- 172.21.15.29:0/2729884201 >> 172.21.15.29:0/2729884201 conn(0x7f71900fc6f0 msgr2=0x7f7190105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7194aeb640 1 -- 172.21.15.29:0/2729884201 shutdown_connections 2024-03-11T19:21:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:25.640+0000 7f7194aeb640 1 -- 172.21.15.29:0/2729884201 wait complete. 2024-03-11T19:21:25.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:25 smithi190 ceph-mon[36292]: pgmap v699: 97 pgs: 97 active+clean; 421 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 30 MiB/s wr, 752 op/s 2024-03-11T19:21:25.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:25 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3110351722' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:21:25.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:25 smithi170 ceph-mon[35690]: pgmap v699: 97 pgs: 97 active+clean; 421 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 30 MiB/s wr, 752 op/s 2024-03-11T19:21:25.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:25 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3110351722' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:21:26.655 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-11T19:21:26.655 INFO:tasks.fwd_scrub:done joining 2024-03-11T19:21:26.656 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-11T19:21:26.666 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:21:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.778+0000 7f534a415640 1 -- 172.21.15.29:0/1828895324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53441011b0 msgr2=0x7f53441015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.778+0000 7f534a415640 1 --2- 172.21.15.29:0/1828895324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53441011b0 0x7f53441015d0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f5334009900 tx=0x7f533402bc30 comp rx=0 tx=0).stop 2024-03-11T19:21:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.778+0000 7f534a415640 1 -- 172.21.15.29:0/1828895324 shutdown_connections 2024-03-11T19:21:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.778+0000 7f534a415640 1 --2- 172.21.15.29:0/1828895324 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5344102d60 0x7f53441095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.778+0000 7f534a415640 1 --2- 172.21.15.29:0/1828895324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53441023a0 0x7f5344102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.778+0000 7f534a415640 1 --2- 172.21.15.29:0/1828895324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53441011b0 0x7f53441015d0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.778+0000 7f534a415640 1 -- 172.21.15.29:0/1828895324 >> 172.21.15.29:0/1828895324 conn(0x7f53440fc760 msgr2=0x7f53440feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.778+0000 7f534a415640 1 -- 172.21.15.29:0/1828895324 shutdown_connections 2024-03-11T19:21:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.778+0000 7f534a415640 1 -- 172.21.15.29:0/1828895324 wait complete. 2024-03-11T19:21:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.778+0000 7f534a415640 1 Processor -- start 2024-03-11T19:21:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.779+0000 7f534a415640 1 -- start start 2024-03-11T19:21:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.779+0000 7f534a415640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53441011b0 0x7f5344197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.779+0000 7f534a415640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53441023a0 0x7f53441976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.779+0000 7f534a415640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5344102d60 0x7f534419e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.779+0000 7f534a415640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534419ec50 con 0x7f53441011b0 2024-03-11T19:21:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.779+0000 7f534a415640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534419edc0 con 0x7f53441023a0 2024-03-11T19:21:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.779+0000 7f534a415640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534419ef80 con 0x7f5344102d60 2024-03-11T19:21:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.779+0000 7f5343fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53441011b0 0x7f5344197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.779+0000 7f5343fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53441011b0 0x7f5344197160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44604/0 (socket says 172.21.15.29:44604) 2024-03-11T19:21:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.779+0000 7f5343fff640 1 -- 172.21.15.29:0/3713300950 learned_addr learned my addr 172.21.15.29:0/3713300950 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.779+0000 7f53437fe640 1 --2- 172.21.15.29:0/3713300950 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53441023a0 0x7f53441976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.779+0000 7f53489b2640 1 --2- 172.21.15.29:0/3713300950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5344102d60 0x7f534419e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.779+0000 7f5343fff640 1 -- 172.21.15.29:0/3713300950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5344102d60 msgr2=0x7f534419e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.779+0000 7f5343fff640 1 --2- 172.21.15.29:0/3713300950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5344102d60 0x7f534419e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:26.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.779+0000 7f5343fff640 1 -- 172.21.15.29:0/3713300950 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53441023a0 msgr2=0x7f53441976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:26.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.779+0000 7f5343fff640 1 --2- 172.21.15.29:0/3713300950 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53441023a0 0x7f53441976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:26.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.779+0000 7f5343fff640 1 -- 172.21.15.29:0/3713300950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5334009560 con 0x7f53441011b0 2024-03-11T19:21:26.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.780+0000 7f5343fff640 1 --2- 172.21.15.29:0/3713300950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53441011b0 0x7f5344197160 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f533402c7d0 tx=0x7f53340042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:26.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.780+0000 7f53417fa640 1 -- 172.21.15.29:0/3713300950 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5334004aa0 con 0x7f53441011b0 2024-03-11T19:21:26.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.780+0000 7f53417fa640 1 -- 172.21.15.29:0/3713300950 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f533402de90 con 0x7f53441011b0 2024-03-11T19:21:26.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.780+0000 7f534a415640 1 -- 172.21.15.29:0/3713300950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f534419f200 con 0x7f53441011b0 2024-03-11T19:21:26.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.780+0000 7f53417fa640 1 -- 172.21.15.29:0/3713300950 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5334005440 con 0x7f53441011b0 2024-03-11T19:21:26.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.780+0000 7f534a415640 1 -- 172.21.15.29:0/3713300950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f534419f710 con 0x7f53441011b0 2024-03-11T19:21:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.781+0000 7f534a415640 1 -- 172.21.15.29:0/3713300950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53441085a0 con 0x7f53441011b0 2024-03-11T19:21:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.782+0000 7f53417fa640 1 -- 172.21.15.29:0/3713300950 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f53340055a0 con 0x7f53441011b0 2024-03-11T19:21:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.782+0000 7f53417fa640 1 --2- 172.21.15.29:0/3713300950 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f532c0a1a30 0x7f532c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.783+0000 7f53437fe640 1 --2- 172.21.15.29:0/3713300950 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f532c0a1a30 0x7f532c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.783+0000 7f53417fa640 1 -- 172.21.15.29:0/3713300950 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f53340ef460 con 0x7f53441011b0 2024-03-11T19:21:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.783+0000 7f53437fe640 1 --2- 172.21.15.29:0/3713300950 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f532c0a1a30 0x7f532c0a3ef0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5344198670 tx=0x7f533800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:26.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.787+0000 7f53417fa640 1 -- 172.21.15.29:0/3713300950 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f53340b8710 con 0x7f53441011b0 2024-03-11T19:21:26.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.970+0000 7f534a415640 1 -- 172.21.15.29:0/3713300950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f534402cdd0 con 0x7f53441011b0 2024-03-11T19:21:26.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.972+0000 7f53417fa640 1 -- 172.21.15.29:0/3713300950 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22559 (secure 0 0 0) 0x7f53340bbdd0 con 0x7f53441011b0 2024-03-11T19:21:26.976 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:21:26.976 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":89,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:08:44.981806+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:07:47.563184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:07:48.282461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:21:26.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.974+0000 7f534a415640 1 -- 172.21.15.29:0/3713300950 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f532c0a1a30 msgr2=0x7f532c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:26.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.974+0000 7f534a415640 1 --2- 172.21.15.29:0/3713300950 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f532c0a1a30 0x7f532c0a3ef0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5344198670 tx=0x7f533800a2d0 comp rx=0 tx=0).stop 2024-03-11T19:21:26.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.974+0000 7f534a415640 1 -- 172.21.15.29:0/3713300950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53441011b0 msgr2=0x7f5344197160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:26.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.974+0000 7f534a415640 1 --2- 172.21.15.29:0/3713300950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53441011b0 0x7f5344197160 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f533402c7d0 tx=0x7f53340042f0 comp rx=0 tx=0).stop 2024-03-11T19:21:26.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.974+0000 7f534a415640 1 -- 172.21.15.29:0/3713300950 shutdown_connections 2024-03-11T19:21:26.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.974+0000 7f534a415640 1 --2- 172.21.15.29:0/3713300950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5344102d60 0x7f534419e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:26.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.974+0000 7f534a415640 1 --2- 172.21.15.29:0/3713300950 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53441023a0 0x7f53441976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:26.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.974+0000 7f534a415640 1 --2- 172.21.15.29:0/3713300950 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f532c0a1a30 0x7f532c0a3ef0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:26.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.974+0000 7f534a415640 1 --2- 172.21.15.29:0/3713300950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53441011b0 0x7f5344197160 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:26.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.974+0000 7f534a415640 1 -- 172.21.15.29:0/3713300950 >> 172.21.15.29:0/3713300950 conn(0x7f53440fc760 msgr2=0x7f5344105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:26.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.974+0000 7f534a415640 1 -- 172.21.15.29:0/3713300950 shutdown_connections 2024-03-11T19:21:26.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:26.974+0000 7f534a415640 1 -- 172.21.15.29:0/3713300950 wait complete. 2024-03-11T19:21:26.989 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:03:09.860019+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T19:07:47.563184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T19:07:48.282461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T19:21:26.990 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:21:27.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.092+0000 7fbce622a640 1 -- 172.21.15.29:0/1201407723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce00ff470 msgr2=0x7fbce00ff890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:27.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.092+0000 7fbce622a640 1 --2- 172.21.15.29:0/1201407723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce00ff470 0x7fbce00ff890 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0009900 tx=0x7fbcd002bc30 comp rx=0 tx=0).stop 2024-03-11T19:21:27.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.092+0000 7fbce622a640 1 -- 172.21.15.29:0/1201407723 shutdown_connections 2024-03-11T19:21:27.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.092+0000 7fbce622a640 1 --2- 172.21.15.29:0/1201407723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbce0102e00 0x7fbce0109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.092+0000 7fbce622a640 1 --2- 172.21.15.29:0/1201407723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbce0102440 0x7fbce01028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.092+0000 7fbce622a640 1 --2- 172.21.15.29:0/1201407723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce00ff470 0x7fbce00ff890 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.092+0000 7fbce622a640 1 -- 172.21.15.29:0/1201407723 >> 172.21.15.29:0/1201407723 conn(0x7fbce00fad80 msgr2=0x7fbce00fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:27.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.092+0000 7fbce622a640 1 -- 172.21.15.29:0/1201407723 shutdown_connections 2024-03-11T19:21:27.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.092+0000 7fbce622a640 1 -- 172.21.15.29:0/1201407723 wait complete. 2024-03-11T19:21:27.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.093+0000 7fbce622a640 1 Processor -- start 2024-03-11T19:21:27.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.093+0000 7fbce622a640 1 -- start start 2024-03-11T19:21:27.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.093+0000 7fbce622a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce00ff470 0x7fbce0197190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:27.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.093+0000 7fbce622a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbce0102440 0x7fbce01976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:27.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.093+0000 7fbce622a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbce0102e00 0x7fbce019e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:27.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.093+0000 7fbce622a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce019ec80 con 0x7fbce00ff470 2024-03-11T19:21:27.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.093+0000 7fbce622a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce019edf0 con 0x7fbce0102440 2024-03-11T19:21:27.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.093+0000 7fbce622a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce019efb0 con 0x7fbce0102e00 2024-03-11T19:21:27.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.093+0000 7fbcdf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce00ff470 0x7fbce0197190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:27.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.093+0000 7fbcdf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce00ff470 0x7fbce0197190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44622/0 (socket says 172.21.15.29:44622) 2024-03-11T19:21:27.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.093+0000 7fbcdf7fe640 1 -- 172.21.15.29:0/2247431237 learned_addr learned my addr 172.21.15.29:0/2247431237 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:27.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.093+0000 7fbcdeffd640 1 --2- 172.21.15.29:0/2247431237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbce0102440 0x7fbce01976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:27.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.093+0000 7fbcdffff640 1 --2- 172.21.15.29:0/2247431237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbce0102e00 0x7fbce019e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:27.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.093+0000 7fbcdf7fe640 1 -- 172.21.15.29:0/2247431237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbce0102e00 msgr2=0x7fbce019e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:27.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.093+0000 7fbcdf7fe640 1 --2- 172.21.15.29:0/2247431237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbce0102e00 0x7fbce019e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.093+0000 7fbcdf7fe640 1 -- 172.21.15.29:0/2247431237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbce0102440 msgr2=0x7fbce01976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:27.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.093+0000 7fbcdf7fe640 1 --2- 172.21.15.29:0/2247431237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbce0102440 0x7fbce01976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.093+0000 7fbcdf7fe640 1 -- 172.21.15.29:0/2247431237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd0009560 con 0x7fbce00ff470 2024-03-11T19:21:27.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.094+0000 7fbcdeffd640 1 --2- 172.21.15.29:0/2247431237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbce0102440 0x7fbce01976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:21:27.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.094+0000 7fbcdf7fe640 1 --2- 172.21.15.29:0/2247431237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce00ff470 0x7fbce0197190 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0000c00 tx=0x7fbcd00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:27.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.094+0000 7fbcdcff9640 1 -- 172.21.15.29:0/2247431237 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd00044e0 con 0x7fbce00ff470 2024-03-11T19:21:27.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.094+0000 7fbce622a640 1 -- 172.21.15.29:0/2247431237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbce019f230 con 0x7fbce00ff470 2024-03-11T19:21:27.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.094+0000 7fbcdcff9640 1 -- 172.21.15.29:0/2247431237 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbcd0004aa0 con 0x7fbce00ff470 2024-03-11T19:21:27.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.094+0000 7fbcdcff9640 1 -- 172.21.15.29:0/2247431237 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd00055b0 con 0x7fbce00ff470 2024-03-11T19:21:27.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.094+0000 7fbce622a640 1 -- 172.21.15.29:0/2247431237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce019f790 con 0x7fbce00ff470 2024-03-11T19:21:27.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.095+0000 7fbce622a640 1 -- 172.21.15.29:0/2247431237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce0108640 con 0x7fbce00ff470 2024-03-11T19:21:27.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.096+0000 7fbcdcff9640 1 -- 172.21.15.29:0/2247431237 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fbcd0004dc0 con 0x7fbce00ff470 2024-03-11T19:21:27.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.096+0000 7fbcdcff9640 1 --2- 172.21.15.29:0/2247431237 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbcc80a1a30 0x7fbcc80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:27.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.097+0000 7fbcdeffd640 1 --2- 172.21.15.29:0/2247431237 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbcc80a1a30 0x7fbcc80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:27.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.097+0000 7fbcdcff9640 1 -- 172.21.15.29:0/2247431237 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fbcd00ef710 con 0x7fbce00ff470 2024-03-11T19:21:27.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.097+0000 7fbcdeffd640 1 --2- 172.21.15.29:0/2247431237 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbcc80a1a30 0x7fbcc80a3ef0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbce01986a0 tx=0x7fbcd4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:27.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.101+0000 7fbcdcff9640 1 -- 172.21.15.29:0/2247431237 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbcd00b8920 con 0x7fbce00ff470 2024-03-11T19:21:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.283+0000 7fbce622a640 1 -- 172.21.15.29:0/2247431237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbce01080b0 con 0x7fbce00ff470 2024-03-11T19:21:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.283+0000 7fbcdcff9640 1 -- 172.21.15.29:0/2247431237 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbcd00bbfe0 con 0x7fbce00ff470 2024-03-11T19:21:27.287 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-11T19:21:27.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.286+0000 7fbce622a640 1 -- 172.21.15.29:0/2247431237 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbcc80a1a30 msgr2=0x7fbcc80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.286+0000 7fbce622a640 1 --2- 172.21.15.29:0/2247431237 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbcc80a1a30 0x7fbcc80a3ef0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbce01986a0 tx=0x7fbcd4005f90 comp rx=0 tx=0).stop 2024-03-11T19:21:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.286+0000 7fbce622a640 1 -- 172.21.15.29:0/2247431237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce00ff470 msgr2=0x7fbce0197190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.286+0000 7fbce622a640 1 --2- 172.21.15.29:0/2247431237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce00ff470 0x7fbce0197190 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0000c00 tx=0x7fbcd00042f0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.286+0000 7fbce622a640 1 -- 172.21.15.29:0/2247431237 shutdown_connections 2024-03-11T19:21:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.286+0000 7fbce622a640 1 --2- 172.21.15.29:0/2247431237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbce0102e00 0x7fbce019e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.286+0000 7fbce622a640 1 --2- 172.21.15.29:0/2247431237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbce0102440 0x7fbce01976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.286+0000 7fbce622a640 1 --2- 172.21.15.29:0/2247431237 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fbcc80a1a30 0x7fbcc80a3ef0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.286+0000 7fbce622a640 1 --2- 172.21.15.29:0/2247431237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbce00ff470 0x7fbce0197190 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.286+0000 7fbce622a640 1 -- 172.21.15.29:0/2247431237 >> 172.21.15.29:0/2247431237 conn(0x7fbce00fad80 msgr2=0x7fbce0105810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.286+0000 7fbce622a640 1 -- 172.21.15.29:0/2247431237 shutdown_connections 2024-03-11T19:21:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.286+0000 7fbce622a640 1 -- 172.21.15.29:0/2247431237 wait complete. 2024-03-11T19:21:27.301 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T19:21:27.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.404+0000 7fef151e2640 1 -- 172.21.15.29:0/3311597361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef10101c30 msgr2=0x7fef101084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:27.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.404+0000 7fef151e2640 1 --2- 172.21.15.29:0/3311597361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef10101c30 0x7fef101084c0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fef00009ca0 tx=0x7fef0002c0f0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.404+0000 7fef151e2640 1 -- 172.21.15.29:0/3311597361 shutdown_connections 2024-03-11T19:21:27.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.404+0000 7fef151e2640 1 --2- 172.21.15.29:0/3311597361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef10101c30 0x7fef101084c0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.404+0000 7fef151e2640 1 --2- 172.21.15.29:0/3311597361 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef10101270 0x7fef101016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.404+0000 7fef151e2640 1 --2- 172.21.15.29:0/3311597361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fef10100080 0x7fef101004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.404+0000 7fef151e2640 1 -- 172.21.15.29:0/3311597361 >> 172.21.15.29:0/3311597361 conn(0x7fef100fb610 msgr2=0x7fef100fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:27.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.404+0000 7fef151e2640 1 -- 172.21.15.29:0/3311597361 shutdown_connections 2024-03-11T19:21:27.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.404+0000 7fef151e2640 1 -- 172.21.15.29:0/3311597361 wait complete. 2024-03-11T19:21:27.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.405+0000 7fef151e2640 1 Processor -- start 2024-03-11T19:21:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.405+0000 7fef151e2640 1 -- start start 2024-03-11T19:21:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.405+0000 7fef151e2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef10100080 0x7fef10195f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.405+0000 7fef151e2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef10101270 0x7fef101964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.405+0000 7fef151e2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fef10101c30 0x7fef1019d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.405+0000 7fef151e2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1019da60 con 0x7fef10100080 2024-03-11T19:21:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.405+0000 7fef151e2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1019dbd0 con 0x7fef10101c30 2024-03-11T19:21:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.405+0000 7fef151e2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1019dd90 con 0x7fef10101270 2024-03-11T19:21:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.405+0000 7fef0ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef10100080 0x7fef10195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.405+0000 7fef0ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef10100080 0x7fef10195f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44646/0 (socket says 172.21.15.29:44646) 2024-03-11T19:21:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.405+0000 7fef0ed9d640 1 -- 172.21.15.29:0/1298254654 learned_addr learned my addr 172.21.15.29:0/1298254654 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.405+0000 7fef0f59e640 1 --2- 172.21.15.29:0/1298254654 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fef10101c30 0x7fef1019d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.406+0000 7fef0e59c640 1 --2- 172.21.15.29:0/1298254654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef10101270 0x7fef101964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.406+0000 7fef0ed9d640 1 -- 172.21.15.29:0/1298254654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef10101270 msgr2=0x7fef101964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.406+0000 7fef0ed9d640 1 --2- 172.21.15.29:0/1298254654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef10101270 0x7fef101964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.406+0000 7fef0ed9d640 1 -- 172.21.15.29:0/1298254654 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fef10101c30 msgr2=0x7fef1019d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.406+0000 7fef0ed9d640 1 --2- 172.21.15.29:0/1298254654 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fef10101c30 0x7fef1019d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.406+0000 7fef0ed9d640 1 -- 172.21.15.29:0/1298254654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef00009900 con 0x7fef10100080 2024-03-11T19:21:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.406+0000 7fef0e59c640 1 --2- 172.21.15.29:0/1298254654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef10101270 0x7fef101964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:21:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.406+0000 7fef0ed9d640 1 --2- 172.21.15.29:0/1298254654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef10100080 0x7fef10195f70 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fef0800e9c0 tx=0x7fef0800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.406+0000 7feef7fff640 1 -- 172.21.15.29:0/1298254654 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef08009700 con 0x7fef10100080 2024-03-11T19:21:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.406+0000 7feef7fff640 1 -- 172.21.15.29:0/1298254654 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fef080044d0 con 0x7fef10100080 2024-03-11T19:21:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.406+0000 7feef7fff640 1 -- 172.21.15.29:0/1298254654 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef08010430 con 0x7fef10100080 2024-03-11T19:21:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.406+0000 7fef151e2640 1 -- 172.21.15.29:0/1298254654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef1019e070 con 0x7fef10100080 2024-03-11T19:21:27.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.406+0000 7fef151e2640 1 -- 172.21.15.29:0/1298254654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef1019e530 con 0x7fef10100080 2024-03-11T19:21:27.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.407+0000 7fef151e2640 1 -- 172.21.15.29:0/1298254654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef101073a0 con 0x7fef10100080 2024-03-11T19:21:27.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.408+0000 7feef7fff640 1 -- 172.21.15.29:0/1298254654 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fef08003f30 con 0x7fef10100080 2024-03-11T19:21:27.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.409+0000 7feef7fff640 1 --2- 172.21.15.29:0/1298254654 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7feef80a1a30 0x7feef80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:27.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.409+0000 7fef0e59c640 1 --2- 172.21.15.29:0/1298254654 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7feef80a1a30 0x7feef80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:27.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.409+0000 7feef7fff640 1 -- 172.21.15.29:0/1298254654 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fef08014070 con 0x7fef10100080 2024-03-11T19:21:27.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.409+0000 7fef0e59c640 1 --2- 172.21.15.29:0/1298254654 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7feef80a1a30 0x7feef80a3ef0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fef10197480 tx=0x7fef04005d90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:27.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.413+0000 7feef7fff640 1 -- 172.21.15.29:0/1298254654 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fef08096ae0 con 0x7fef10100080 2024-03-11T19:21:27.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.594+0000 7fef151e2640 1 -- 172.21.15.29:0/1298254654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fef1010bbb0 con 0x7fef10100080 2024-03-11T19:21:27.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.595+0000 7feef7fff640 1 -- 172.21.15.29:0/1298254654 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7fef0809a1a0 con 0x7fef10100080 2024-03-11T19:21:27.599 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-03-11T19:21:27.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.597+0000 7fef151e2640 1 -- 172.21.15.29:0/1298254654 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7feef80a1a30 msgr2=0x7feef80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:27.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.597+0000 7fef151e2640 1 --2- 172.21.15.29:0/1298254654 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7feef80a1a30 0x7feef80a3ef0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fef10197480 tx=0x7fef04005d90 comp rx=0 tx=0).stop 2024-03-11T19:21:27.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.597+0000 7fef151e2640 1 -- 172.21.15.29:0/1298254654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef10100080 msgr2=0x7fef10195f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:27.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.597+0000 7fef151e2640 1 --2- 172.21.15.29:0/1298254654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef10100080 0x7fef10195f70 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fef0800e9c0 tx=0x7fef0800ee80 comp rx=0 tx=0).stop 2024-03-11T19:21:27.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.597+0000 7fef151e2640 1 -- 172.21.15.29:0/1298254654 shutdown_connections 2024-03-11T19:21:27.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.597+0000 7fef151e2640 1 --2- 172.21.15.29:0/1298254654 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fef10101c30 0x7fef1019d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.597+0000 7fef151e2640 1 --2- 172.21.15.29:0/1298254654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef10101270 0x7fef101964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.597+0000 7fef151e2640 1 --2- 172.21.15.29:0/1298254654 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7feef80a1a30 0x7feef80a3ef0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.597+0000 7fef151e2640 1 --2- 172.21.15.29:0/1298254654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef10100080 0x7fef10195f70 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.597+0000 7fef151e2640 1 -- 172.21.15.29:0/1298254654 >> 172.21.15.29:0/1298254654 conn(0x7fef100fb610 msgr2=0x7fef10104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:27.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.597+0000 7fef151e2640 1 -- 172.21.15.29:0/1298254654 shutdown_connections 2024-03-11T19:21:27.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.597+0000 7fef151e2640 1 -- 172.21.15.29:0/1298254654 wait complete. 2024-03-11T19:21:27.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:27 smithi029 ceph-mon[32439]: pgmap v700: 97 pgs: 97 active+clean; 1.2 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 20 MiB/s wr, 571 op/s 2024-03-11T19:21:27.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:27 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3713300950' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:21:27.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:27 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2247431237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:21:27.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T19:21:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.714+0000 7f603b49a640 1 -- 172.21.15.29:0/131559021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60341011a0 msgr2=0x7f60341015c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.714+0000 7f603b49a640 1 --2- 172.21.15.29:0/131559021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60341011a0 0x7f60341015c0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f6030009900 tx=0x7f603002bc30 comp rx=0 tx=0).stop 2024-03-11T19:21:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.714+0000 7f603b49a640 1 -- 172.21.15.29:0/131559021 shutdown_connections 2024-03-11T19:21:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.714+0000 7f603b49a640 1 --2- 172.21.15.29:0/131559021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6034102d50 0x7f60341095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.714+0000 7f603b49a640 1 --2- 172.21.15.29:0/131559021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6034102390 0x7f6034102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.714+0000 7f603b49a640 1 --2- 172.21.15.29:0/131559021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60341011a0 0x7f60341015c0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.714+0000 7f603b49a640 1 -- 172.21.15.29:0/131559021 >> 172.21.15.29:0/131559021 conn(0x7f60340fc6f0 msgr2=0x7f60340feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.714+0000 7f603b49a640 1 -- 172.21.15.29:0/131559021 shutdown_connections 2024-03-11T19:21:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.714+0000 7f603b49a640 1 -- 172.21.15.29:0/131559021 wait complete. 2024-03-11T19:21:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.715+0000 7f603b49a640 1 Processor -- start 2024-03-11T19:21:27.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.715+0000 7f603b49a640 1 -- start start 2024-03-11T19:21:27.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.715+0000 7f603b49a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60341011a0 0x7f6034197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:27.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.715+0000 7f603b49a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6034102390 0x7f60341976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:27.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.715+0000 7f603b49a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6034102d50 0x7f603419e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:27.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.715+0000 7f603b49a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603419ec50 con 0x7f60341011a0 2024-03-11T19:21:27.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.715+0000 7f603b49a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603419edc0 con 0x7f6034102390 2024-03-11T19:21:27.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.715+0000 7f603b49a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603419ef80 con 0x7f6034102d50 2024-03-11T19:21:27.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.715+0000 7f6039236640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60341011a0 0x7f6034197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:27.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.715+0000 7f6039236640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60341011a0 0x7f6034197160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44660/0 (socket says 172.21.15.29:44660) 2024-03-11T19:21:27.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.715+0000 7f6039236640 1 -- 172.21.15.29:0/3495027399 learned_addr learned my addr 172.21.15.29:0/3495027399 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:27.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.715+0000 7f6039a37640 1 --2- 172.21.15.29:0/3495027399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6034102d50 0x7f603419e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:27.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.715+0000 7f6038a35640 1 --2- 172.21.15.29:0/3495027399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6034102390 0x7f60341976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:27.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.715+0000 7f6039236640 1 -- 172.21.15.29:0/3495027399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6034102d50 msgr2=0x7f603419e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:27.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.715+0000 7f6039236640 1 --2- 172.21.15.29:0/3495027399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6034102d50 0x7f603419e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.715+0000 7f6039236640 1 -- 172.21.15.29:0/3495027399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6034102390 msgr2=0x7f60341976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:27.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.715+0000 7f6039236640 1 --2- 172.21.15.29:0/3495027399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6034102390 0x7f60341976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.715+0000 7f6039236640 1 -- 172.21.15.29:0/3495027399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6030009560 con 0x7f60341011a0 2024-03-11T19:21:27.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.715+0000 7f6039236640 1 --2- 172.21.15.29:0/3495027399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60341011a0 0x7f6034197160 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f603002c7d0 tx=0x7f60300042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:27.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.716+0000 7f602a7fc640 1 -- 172.21.15.29:0/3495027399 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6030004aa0 con 0x7f60341011a0 2024-03-11T19:21:27.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.716+0000 7f602a7fc640 1 -- 172.21.15.29:0/3495027399 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f603002de90 con 0x7f60341011a0 2024-03-11T19:21:27.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.716+0000 7f603b49a640 1 -- 172.21.15.29:0/3495027399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f603419f200 con 0x7f60341011a0 2024-03-11T19:21:27.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.716+0000 7f602a7fc640 1 -- 172.21.15.29:0/3495027399 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6030005440 con 0x7f60341011a0 2024-03-11T19:21:27.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.716+0000 7f603b49a640 1 -- 172.21.15.29:0/3495027399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f603419f760 con 0x7f60341011a0 2024-03-11T19:21:27.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.717+0000 7f603b49a640 1 -- 172.21.15.29:0/3495027399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60341085e0 con 0x7f60341011a0 2024-03-11T19:21:27.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.718+0000 7f602a7fc640 1 -- 172.21.15.29:0/3495027399 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f60300055a0 con 0x7f60341011a0 2024-03-11T19:21:27.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.718+0000 7f602a7fc640 1 --2- 172.21.15.29:0/3495027399 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f60200a1ab0 0x7f60200a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:27.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.718+0000 7f602a7fc640 1 -- 172.21.15.29:0/3495027399 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f60300efa90 con 0x7f60341011a0 2024-03-11T19:21:27.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.718+0000 7f6038a35640 1 --2- 172.21.15.29:0/3495027399 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f60200a1ab0 0x7f60200a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:27.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.719+0000 7f6038a35640 1 --2- 172.21.15.29:0/3495027399 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f60200a1ab0 0x7f60200a3f70 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f6024005fd0 tx=0x7f6024005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:27.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.722+0000 7f602a7fc640 1 -- 172.21.15.29:0/3495027399 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f60300b8d40 con 0x7f60341011a0 2024-03-11T19:21:27.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:27 smithi190 ceph-mon[36292]: pgmap v700: 97 pgs: 97 active+clean; 1.2 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 20 MiB/s wr, 571 op/s 2024-03-11T19:21:27.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:27 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3713300950' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:21:27.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:27 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2247431237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:21:27.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:27 smithi170 ceph-mon[35690]: pgmap v700: 97 pgs: 97 active+clean; 1.2 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 20 MiB/s wr, 571 op/s 2024-03-11T19:21:27.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:27 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3713300950' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:21:27.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:27 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2247431237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:21:27.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.904+0000 7f603b49a640 1 -- 172.21.15.29:0/3495027399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f603402cda0 con 0x7f60341011a0 2024-03-11T19:21:27.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.905+0000 7f602a7fc640 1 -- 172.21.15.29:0/3495027399 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f60300430a0 con 0x7f60341011a0 2024-03-11T19:21:27.909 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-03-11T19:21:27.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.907+0000 7f603b49a640 1 -- 172.21.15.29:0/3495027399 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f60200a1ab0 msgr2=0x7f60200a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:27.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.907+0000 7f603b49a640 1 --2- 172.21.15.29:0/3495027399 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f60200a1ab0 0x7f60200a3f70 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f6024005fd0 tx=0x7f6024005850 comp rx=0 tx=0).stop 2024-03-11T19:21:27.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.907+0000 7f603b49a640 1 -- 172.21.15.29:0/3495027399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60341011a0 msgr2=0x7f6034197160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:27.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.907+0000 7f603b49a640 1 --2- 172.21.15.29:0/3495027399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60341011a0 0x7f6034197160 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f603002c7d0 tx=0x7f60300042f0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.907+0000 7f603b49a640 1 -- 172.21.15.29:0/3495027399 shutdown_connections 2024-03-11T19:21:27.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.907+0000 7f603b49a640 1 --2- 172.21.15.29:0/3495027399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6034102d50 0x7f603419e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.907+0000 7f603b49a640 1 --2- 172.21.15.29:0/3495027399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6034102390 0x7f60341976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.907+0000 7f603b49a640 1 --2- 172.21.15.29:0/3495027399 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f60200a1ab0 0x7f60200a3f70 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.907+0000 7f603b49a640 1 --2- 172.21.15.29:0/3495027399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60341011a0 0x7f6034197160 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:27.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.907+0000 7f603b49a640 1 -- 172.21.15.29:0/3495027399 >> 172.21.15.29:0/3495027399 conn(0x7f60340fc6f0 msgr2=0x7f60341057b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:27.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.907+0000 7f603b49a640 1 -- 172.21.15.29:0/3495027399 shutdown_connections 2024-03-11T19:21:27.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:27.907+0000 7f603b49a640 1 -- 172.21.15.29:0/3495027399 wait complete. 2024-03-11T19:21:27.922 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-11T19:21:28.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.022+0000 7f43d08c9640 1 -- 172.21.15.29:0/1225166730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43cc101cd0 msgr2=0x7f43cc108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:28.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.022+0000 7f43d08c9640 1 --2- 172.21.15.29:0/1225166730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43cc101cd0 0x7f43cc108560 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f43c4009ca0 tx=0x7f43c402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:21:28.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.022+0000 7f43d08c9640 1 -- 172.21.15.29:0/1225166730 shutdown_connections 2024-03-11T19:21:28.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.022+0000 7f43d08c9640 1 --2- 172.21.15.29:0/1225166730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43cc101cd0 0x7f43cc108560 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:28.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.022+0000 7f43d08c9640 1 --2- 172.21.15.29:0/1225166730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43cc101310 0x7f43cc101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:28.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.022+0000 7f43d08c9640 1 --2- 172.21.15.29:0/1225166730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43cc100120 0x7f43cc100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:28.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.022+0000 7f43d08c9640 1 -- 172.21.15.29:0/1225166730 >> 172.21.15.29:0/1225166730 conn(0x7f43cc0fb6b0 msgr2=0x7f43cc0fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:28.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.022+0000 7f43d08c9640 1 -- 172.21.15.29:0/1225166730 shutdown_connections 2024-03-11T19:21:28.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.022+0000 7f43d08c9640 1 -- 172.21.15.29:0/1225166730 wait complete. 2024-03-11T19:21:28.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43d08c9640 1 Processor -- start 2024-03-11T19:21:28.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43d08c9640 1 -- start start 2024-03-11T19:21:28.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43d08c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43cc100120 0x7f43cc1970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:28.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43d08c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43cc101310 0x7f43cc197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:28.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43d08c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43cc101cd0 0x7f43cc19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:28.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43d08c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc19ebe0 con 0x7f43cc101310 2024-03-11T19:21:28.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43d08c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc19ed50 con 0x7f43cc100120 2024-03-11T19:21:28.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43d08c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc19ef10 con 0x7f43cc101cd0 2024-03-11T19:21:28.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43c9d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43cc101310 0x7f43cc197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:28.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43c9d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43cc101310 0x7f43cc197630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44678/0 (socket says 172.21.15.29:44678) 2024-03-11T19:21:28.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43c9d9b640 1 -- 172.21.15.29:0/399025604 learned_addr learned my addr 172.21.15.29:0/399025604 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:21:28.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43ca59c640 1 --2- 172.21.15.29:0/399025604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43cc100120 0x7f43cc1970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:28.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43cad9d640 1 --2- 172.21.15.29:0/399025604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43cc101cd0 0x7f43cc19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:28.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43c9d9b640 1 -- 172.21.15.29:0/399025604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43cc101cd0 msgr2=0x7f43cc19e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:28.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43c9d9b640 1 --2- 172.21.15.29:0/399025604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43cc101cd0 0x7f43cc19e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:28.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43c9d9b640 1 -- 172.21.15.29:0/399025604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43cc100120 msgr2=0x7f43cc1970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:28.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43c9d9b640 1 --2- 172.21.15.29:0/399025604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43cc100120 0x7f43cc1970f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:28.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43c9d9b640 1 -- 172.21.15.29:0/399025604 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c4009900 con 0x7f43cc101310 2024-03-11T19:21:28.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43cad9d640 1 --2- 172.21.15.29:0/399025604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43cc101cd0 0x7f43cc19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:21:28.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.023+0000 7f43ca59c640 1 --2- 172.21.15.29:0/399025604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43cc100120 0x7f43cc1970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:21:28.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.024+0000 7f43c9d9b640 1 --2- 172.21.15.29:0/399025604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43cc101310 0x7f43cc197630 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f43c000d770 tx=0x7f43c000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:28.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.024+0000 7f43bb7fe640 1 -- 172.21.15.29:0/399025604 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c0009c30 con 0x7f43cc101310 2024-03-11T19:21:28.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.024+0000 7f43bb7fe640 1 -- 172.21.15.29:0/399025604 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f43c000de80 con 0x7f43cc101310 2024-03-11T19:21:28.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.024+0000 7f43d08c9640 1 -- 172.21.15.29:0/399025604 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43cc19f1f0 con 0x7f43cc101310 2024-03-11T19:21:28.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.024+0000 7f43bb7fe640 1 -- 172.21.15.29:0/399025604 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c0004e00 con 0x7f43cc101310 2024-03-11T19:21:28.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.024+0000 7f43d08c9640 1 -- 172.21.15.29:0/399025604 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43cc19f6b0 con 0x7f43cc101310 2024-03-11T19:21:28.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.025+0000 7f43d08c9640 1 -- 172.21.15.29:0/399025604 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43cc1074c0 con 0x7f43cc101310 2024-03-11T19:21:28.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.026+0000 7f43bb7fe640 1 -- 172.21.15.29:0/399025604 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f43c0004f60 con 0x7f43cc101310 2024-03-11T19:21:28.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.026+0000 7f43bb7fe640 1 --2- 172.21.15.29:0/399025604 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f43b40a1a30 0x7f43b40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:21:28.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.026+0000 7f43ca59c640 1 --2- 172.21.15.29:0/399025604 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f43b40a1a30 0x7f43b40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:21:28.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.026+0000 7f43bb7fe640 1 -- 172.21.15.29:0/399025604 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f43c00ccea0 con 0x7f43cc101310 2024-03-11T19:21:28.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.027+0000 7f43ca59c640 1 --2- 172.21.15.29:0/399025604 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f43b40a1a30 0x7f43b40a3ef0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f43cc101170 tx=0x7f43bc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:21:28.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.031+0000 7f43bb7fe640 1 -- 172.21.15.29:0/399025604 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f43c0096150 con 0x7f43cc101310 2024-03-11T19:21:28.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.262+0000 7f43d08c9640 1 -- 172.21.15.29:0/399025604 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f43cc10cc40 con 0x7f43cc101310 2024-03-11T19:21:28.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.438+0000 7f43bb7fe640 1 -- 172.21.15.29:0/399025604 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v20) v1 ==== 81+0+734469 (secure 0 0 0) 0x7f43c0020090 con 0x7f43cc101310 2024-03-11T19:21:28.452 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:21:28.452 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:21:28.452 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 20, 2024-03-11T19:21:28.452 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.452 INFO:teuthology.orchestra.run.smithi029.stdout: "active_gid": 14150, 2024-03-11T19:21:28.453 INFO:teuthology.orchestra.run.smithi029.stdout: "active_name": "x", 2024-03-11T19:21:28.453 INFO:teuthology.orchestra.run.smithi029.stdout: "active_addrs": { 2024-03-11T19:21:28.453 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-11T19:21:28.453 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.453 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-11T19:21:28.453 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6800", 2024-03-11T19:21:28.453 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1587871898 2024-03-11T19:21:28.453 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.453 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.453 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v1", 2024-03-11T19:21:28.453 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6801", 2024-03-11T19:21:28.453 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1587871898 2024-03-11T19:21:28.453 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.453 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-11T19:21:28.454 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.454 INFO:teuthology.orchestra.run.smithi029.stdout: "active_addr": "172.21.15.29:6801/1587871898", 2024-03-11T19:21:28.454 INFO:teuthology.orchestra.run.smithi029.stdout: "active_change": "2024-03-11T19:00:43.524986+0000", 2024-03-11T19:21:28.454 INFO:teuthology.orchestra.run.smithi029.stdout: "active_mgr_features": 4540701547738038271, 2024-03-11T19:21:28.454 INFO:teuthology.orchestra.run.smithi029.stdout: "available": true, 2024-03-11T19:21:28.454 INFO:teuthology.orchestra.run.smithi029.stdout: "standbys": [ 2024-03-11T19:21:28.454 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.454 INFO:teuthology.orchestra.run.smithi029.stdout: "gid": 24109, 2024-03-11T19:21:28.454 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "z", 2024-03-11T19:21:28.454 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_features": 4540701547738038271, 2024-03-11T19:21:28.454 INFO:teuthology.orchestra.run.smithi029.stdout: "available_modules": [ 2024-03-11T19:21:28.454 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.454 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "alerts", 2024-03-11T19:21:28.454 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.454 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.454 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.455 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-11T19:21:28.455 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-11T19:21:28.455 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.455 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.455 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.455 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-11T19:21:28.455 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.455 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.455 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.455 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How frequently to reexamine health status", 2024-03-11T19:21:28.455 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.455 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.455 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.455 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.456 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.456 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.456 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.456 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.456 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.456 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.456 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.456 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.456 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.456 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.456 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.456 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.456 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.456 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.456 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.456 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.456 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.457 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.457 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.457 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.457 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.457 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.457 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.457 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.457 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.457 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.457 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.457 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.457 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.457 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.457 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.457 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.458 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.458 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.458 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.458 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.458 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.458 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.458 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.458 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.458 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.458 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.458 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.458 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.458 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.458 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.458 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.458 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.459 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.459 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.459 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.459 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.459 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.459 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.459 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.459 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.459 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.459 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.459 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.459 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.459 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.459 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.459 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.459 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.460 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.460 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.460 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.460 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.460 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.460 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_destination": { 2024-03-11T19:21:28.460 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_destination", 2024-03-11T19:21:28.460 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.460 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.460 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.460 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.460 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.460 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.460 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.460 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Email address to send alerts to", 2024-03-11T19:21:28.460 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.461 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.461 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.461 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.461 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_from_name": { 2024-03-11T19:21:28.461 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_from_name", 2024-03-11T19:21:28.461 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.461 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.461 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.461 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "Ceph", 2024-03-11T19:21:28.461 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.461 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.461 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.461 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Email From: name", 2024-03-11T19:21:28.461 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.461 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.462 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.462 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.462 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_host": { 2024-03-11T19:21:28.462 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_host", 2024-03-11T19:21:28.462 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.462 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.462 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.462 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.462 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.462 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.462 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.462 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP server", 2024-03-11T19:21:28.462 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.462 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.462 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.462 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.462 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_password": { 2024-03-11T19:21:28.463 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_password", 2024-03-11T19:21:28.463 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.463 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.463 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.463 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.463 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.463 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.463 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.463 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Password to authenticate with", 2024-03-11T19:21:28.463 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.463 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.463 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.463 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.463 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_port": { 2024-03-11T19:21:28.463 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_port", 2024-03-11T19:21:28.463 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.464 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.464 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.464 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "465", 2024-03-11T19:21:28.464 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.464 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.464 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.464 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP port", 2024-03-11T19:21:28.464 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.464 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.464 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.464 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.464 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_sender": { 2024-03-11T19:21:28.464 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_sender", 2024-03-11T19:21:28.465 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.465 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.465 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.465 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.465 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.465 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.465 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.465 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP envelope sender", 2024-03-11T19:21:28.465 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.465 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.465 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.465 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.465 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_ssl": { 2024-03-11T19:21:28.465 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_ssl", 2024-03-11T19:21:28.465 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.466 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.466 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.466 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.466 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.466 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.466 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.466 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-11T19:21:28.466 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.466 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.466 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.466 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.466 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_user": { 2024-03-11T19:21:28.466 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_user", 2024-03-11T19:21:28.466 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.466 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.466 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.467 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.467 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.467 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.467 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.467 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "User to authenticate as", 2024-03-11T19:21:28.467 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.467 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.467 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.467 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.467 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.467 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.467 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.467 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "balancer", 2024-03-11T19:21:28.467 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.467 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.467 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: "active": { 2024-03-11T19:21:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "active", 2024-03-11T19:21:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically balance PGs across cluster", 2024-03-11T19:21:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.468 INFO:teuthology.orchestra.run.smithi029.stdout: "begin_time": { 2024-03-11T19:21:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "begin_time", 2024-03-11T19:21:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0000", 2024-03-11T19:21:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "beginning time of day to automatically balance", 2024-03-11T19:21:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-11T19:21:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: "begin_weekday": { 2024-03-11T19:21:28.469 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "begin_weekday", 2024-03-11T19:21:28.470 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-11T19:21:28.470 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.470 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.470 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-11T19:21:28.470 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0", 2024-03-11T19:21:28.470 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "6", 2024-03-11T19:21:28.470 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.470 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-11T19:21:28.470 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-11T19:21:28.470 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.470 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.470 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.470 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_max_iterations": { 2024-03-11T19:21:28.470 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_max_iterations", 2024-03-11T19:21:28.470 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-11T19:21:28.470 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.471 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.471 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "25", 2024-03-11T19:21:28.471 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-11T19:21:28.471 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "250", 2024-03-11T19:21:28.471 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.471 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-11T19:21:28.471 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.471 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.471 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.471 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.471 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_metrics": { 2024-03-11T19:21:28.471 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_metrics", 2024-03-11T19:21:28.471 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.471 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.471 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.471 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "pgs,objects,bytes", 2024-03-11T19:21:28.472 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.472 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.472 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.472 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-11T19:21:28.472 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-11T19:21:28.472 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.472 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.472 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.472 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_step": { 2024-03-11T19:21:28.472 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_step", 2024-03-11T19:21:28.472 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:28.472 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.472 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.472 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0.5", 2024-03-11T19:21:28.473 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0.001", 2024-03-11T19:21:28.473 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "0.999", 2024-03-11T19:21:28.473 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.473 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "aggressiveness of optimization", 2024-03-11T19:21:28.473 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-11T19:21:28.473 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.473 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.473 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.473 INFO:teuthology.orchestra.run.smithi029.stdout: "end_time": { 2024-03-11T19:21:28.473 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "end_time", 2024-03-11T19:21:28.473 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.473 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.473 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.473 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "2359", 2024-03-11T19:21:28.473 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.473 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.473 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.474 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "ending time of day to automatically balance", 2024-03-11T19:21:28.474 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-11T19:21:28.474 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.474 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.474 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.474 INFO:teuthology.orchestra.run.smithi029.stdout: "end_weekday": { 2024-03-11T19:21:28.474 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "end_weekday", 2024-03-11T19:21:28.474 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-11T19:21:28.474 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.474 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.474 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-11T19:21:28.474 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0", 2024-03-11T19:21:28.474 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "6", 2024-03-11T19:21:28.474 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.474 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-11T19:21:28.474 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-11T19:21:28.475 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.475 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.475 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.475 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.475 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.475 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.475 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.475 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.475 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.475 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.475 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.475 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.475 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.475 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.475 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.475 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.475 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.476 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.476 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.476 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.476 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.476 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.476 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.476 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.476 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.476 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.476 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.476 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.476 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.476 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.476 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.476 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.476 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.477 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.477 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.477 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.477 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.477 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.477 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.477 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.477 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.477 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.477 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.477 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.477 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.477 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.477 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.477 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.477 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.478 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.478 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.478 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.478 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.478 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.478 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.478 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.478 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.478 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.478 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.478 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.478 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.478 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.478 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.478 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.478 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.479 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.479 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.479 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.479 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.479 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.479 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.479 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.479 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.479 INFO:teuthology.orchestra.run.smithi029.stdout: "min_score": { 2024-03-11T19:21:28.479 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "min_score", 2024-03-11T19:21:28.479 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:28.479 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.479 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.479 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-11T19:21:28.479 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.479 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.479 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.480 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-11T19:21:28.480 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.480 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.480 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.480 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.480 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": { 2024-03-11T19:21:28.480 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mode", 2024-03-11T19:21:28.480 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.480 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.480 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.480 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "upmap", 2024-03-11T19:21:28.480 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.480 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.480 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.480 INFO:teuthology.orchestra.run.smithi029.stdout: "crush-compat", 2024-03-11T19:21:28.480 INFO:teuthology.orchestra.run.smithi029.stdout: "none", 2024-03-11T19:21:28.480 INFO:teuthology.orchestra.run.smithi029.stdout: "read", 2024-03-11T19:21:28.480 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap", 2024-03-11T19:21:28.481 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap-read" 2024-03-11T19:21:28.481 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.481 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Balancer mode", 2024-03-11T19:21:28.481 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.481 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.481 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.481 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.481 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_ids": { 2024-03-11T19:21:28.481 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pool_ids", 2024-03-11T19:21:28.481 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.481 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.481 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.481 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.481 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.481 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.481 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.481 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-11T19:21:28.482 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.482 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.482 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.482 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.482 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-11T19:21:28.482 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-11T19:21:28.482 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.482 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.482 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.482 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-11T19:21:28.482 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.482 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.483 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.483 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-11T19:21:28.483 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.483 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.483 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.483 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.483 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap_max_deviation": { 2024-03-11T19:21:28.483 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "upmap_max_deviation", 2024-03-11T19:21:28.483 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.483 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.483 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.483 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-11T19:21:28.483 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-11T19:21:28.483 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.484 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.484 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-11T19:21:28.484 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-11T19:21:28.484 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.484 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.484 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.484 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap_max_optimizations": { 2024-03-11T19:21:28.484 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "upmap_max_optimizations", 2024-03-11T19:21:28.484 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-11T19:21:28.484 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.484 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.484 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:28.484 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.484 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.485 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.485 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-11T19:21:28.485 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.485 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.485 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.485 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.485 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.485 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.485 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.485 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephadm", 2024-03-11T19:21:28.485 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.485 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.485 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.485 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_down_multiplier": { 2024-03-11T19:21:28.485 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_down_multiplier", 2024-03-11T19:21:28.486 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:28.486 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.486 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.486 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3.0", 2024-03-11T19:21:28.486 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.486 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.486 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.486 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-11T19:21:28.486 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.486 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.486 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.486 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.486 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_refresh_rate": { 2024-03-11T19:21:28.486 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_refresh_rate", 2024-03-11T19:21:28.486 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.486 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.487 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.487 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "20", 2024-03-11T19:21:28.487 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.487 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.487 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.487 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-11T19:21:28.487 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.487 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.487 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.487 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.487 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_starting_port": { 2024-03-11T19:21:28.487 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_starting_port", 2024-03-11T19:21:28.487 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.487 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.487 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.487 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "4721", 2024-03-11T19:21:28.488 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.488 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.488 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.488 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-11T19:21:28.488 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.488 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.488 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.488 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.488 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_ptrace": { 2024-03-11T19:21:28.488 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_ptrace", 2024-03-11T19:21:28.488 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.488 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.488 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.488 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.488 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.489 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.489 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.489 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-11T19:21:28.489 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-11T19:21:28.489 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.489 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.489 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.489 INFO:teuthology.orchestra.run.smithi029.stdout: "autotune_interval": { 2024-03-11T19:21:28.489 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "autotune_interval", 2024-03-11T19:21:28.489 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.489 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.489 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.489 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-11T19:21:28.489 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.489 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.489 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.489 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-11T19:21:28.490 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.490 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.490 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.490 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.490 INFO:teuthology.orchestra.run.smithi029.stdout: "autotune_memory_target_ratio": { 2024-03-11T19:21:28.490 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "autotune_memory_target_ratio", 2024-03-11T19:21:28.490 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:28.490 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.490 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.490 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0.7", 2024-03-11T19:21:28.490 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.490 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.490 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.490 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-11T19:21:28.490 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.490 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.490 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.490 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.491 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm_log_destination": { 2024-03-11T19:21:28.491 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephadm_log_destination", 2024-03-11T19:21:28.491 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.491 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.491 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.491 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.491 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.491 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.491 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.491 INFO:teuthology.orchestra.run.smithi029.stdout: "file", 2024-03-11T19:21:28.491 INFO:teuthology.orchestra.run.smithi029.stdout: "file,syslog", 2024-03-11T19:21:28.491 INFO:teuthology.orchestra.run.smithi029.stdout: "syslog" 2024-03-11T19:21:28.491 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.491 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-11T19:21:28.491 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.491 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.492 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.492 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.492 INFO:teuthology.orchestra.run.smithi029.stdout: "cgroups_split": { 2024-03-11T19:21:28.492 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cgroups_split", 2024-03-11T19:21:28.492 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.492 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.492 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.492 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.492 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.492 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.492 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.492 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-11T19:21:28.492 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.492 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.492 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.492 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.492 INFO:teuthology.orchestra.run.smithi029.stdout: "config_checks_enabled": { 2024-03-11T19:21:28.493 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_checks_enabled", 2024-03-11T19:21:28.493 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.493 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.493 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.493 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.493 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.493 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.493 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.493 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-11T19:21:28.493 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.493 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.493 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.493 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.493 INFO:teuthology.orchestra.run.smithi029.stdout: "config_dashboard": { 2024-03-11T19:21:28.493 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_dashboard", 2024-03-11T19:21:28.493 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.493 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.494 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.494 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.494 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.494 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.494 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.494 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-11T19:21:28.494 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.494 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.494 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.494 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.494 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_alertmanager": { 2024-03-11T19:21:28.494 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_alertmanager", 2024-03-11T19:21:28.494 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.494 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.494 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.494 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-11T19:21:28.494 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.495 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.495 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.495 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-11T19:21:28.495 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.495 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.495 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.495 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.495 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_base": { 2024-03-11T19:21:28.495 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_base", 2024-03-11T19:21:28.495 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.495 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.495 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.495 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-11T19:21:28.495 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.495 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.495 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.495 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Container image name, without the tag", 2024-03-11T19:21:28.495 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.496 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.496 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.496 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.496 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_elasticsearch": { 2024-03-11T19:21:28.496 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_elasticsearch", 2024-03-11T19:21:28.496 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.496 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.496 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.496 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-11T19:21:28.496 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.496 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.496 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.496 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "elasticsearch container image", 2024-03-11T19:21:28.496 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.496 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.496 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.496 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.497 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_grafana": { 2024-03-11T19:21:28.497 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_grafana", 2024-03-11T19:21:28.497 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.497 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.497 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.497 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-11T19:21:28.497 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.497 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.497 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.497 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-11T19:21:28.497 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.497 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.497 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.497 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.497 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_haproxy": { 2024-03-11T19:21:28.497 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_haproxy", 2024-03-11T19:21:28.497 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.497 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.498 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.498 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-11T19:21:28.498 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.498 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.498 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.498 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "HAproxy container image", 2024-03-11T19:21:28.498 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.498 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.498 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.498 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.498 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_agent": { 2024-03-11T19:21:28.498 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_agent", 2024-03-11T19:21:28.498 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.498 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.498 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.499 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-11T19:21:28.499 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.499 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.499 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.499 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger agent container image", 2024-03-11T19:21:28.499 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.499 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.499 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.499 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.499 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_collector": { 2024-03-11T19:21:28.499 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_collector", 2024-03-11T19:21:28.499 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.499 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.499 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.499 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-11T19:21:28.499 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.500 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.500 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.500 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger collector container image", 2024-03-11T19:21:28.500 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.500 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.500 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.500 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.500 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_query": { 2024-03-11T19:21:28.500 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_query", 2024-03-11T19:21:28.500 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.500 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.500 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.500 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-11T19:21:28.500 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.500 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.501 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.501 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger query container image", 2024-03-11T19:21:28.501 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.501 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.501 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.501 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.501 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_keepalived": { 2024-03-11T19:21:28.501 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_keepalived", 2024-03-11T19:21:28.501 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.501 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.501 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.501 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-11T19:21:28.501 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.501 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.501 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.501 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Keepalived container image", 2024-03-11T19:21:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_loki": { 2024-03-11T19:21:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_loki", 2024-03-11T19:21:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-11T19:21:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Loki container image", 2024-03-11T19:21:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.502 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.503 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.503 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.503 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_node_exporter": { 2024-03-11T19:21:28.503 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_node_exporter", 2024-03-11T19:21:28.503 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.503 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.503 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.503 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-11T19:21:28.503 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.503 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.503 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.503 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-11T19:21:28.503 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.503 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.503 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.504 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.504 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_nvmeof": { 2024-03-11T19:21:28.504 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_nvmeof", 2024-03-11T19:21:28.504 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.504 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.504 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.504 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-11T19:21:28.504 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.504 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.504 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.504 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Nvme-of container image", 2024-03-11T19:21:28.504 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.504 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.504 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.504 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.504 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_prometheus": { 2024-03-11T19:21:28.504 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_prometheus", 2024-03-11T19:21:28.505 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.505 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.505 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.505 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-11T19:21:28.505 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.505 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.505 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.505 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-11T19:21:28.505 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.505 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.505 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.505 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.505 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_promtail": { 2024-03-11T19:21:28.505 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_promtail", 2024-03-11T19:21:28.505 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.505 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.505 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.506 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-11T19:21:28.506 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.506 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.506 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.506 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Promtail container image", 2024-03-11T19:21:28.506 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.506 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.506 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.506 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.506 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_snmp_gateway": { 2024-03-11T19:21:28.506 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_snmp_gateway", 2024-03-11T19:21:28.506 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.506 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.506 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.506 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-11T19:21:28.506 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.506 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.507 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.507 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SNMP Gateway container image", 2024-03-11T19:21:28.507 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.507 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.507 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.507 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.507 INFO:teuthology.orchestra.run.smithi029.stdout: "container_init": { 2024-03-11T19:21:28.507 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_init", 2024-03-11T19:21:28.507 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.507 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.507 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.507 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.507 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.507 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.507 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.507 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Run podman/docker with `--init`", 2024-03-11T19:21:28.507 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.508 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.508 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.508 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.508 INFO:teuthology.orchestra.run.smithi029.stdout: "daemon_cache_timeout": { 2024-03-11T19:21:28.508 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "daemon_cache_timeout", 2024-03-11T19:21:28.508 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.508 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.508 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.508 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-11T19:21:28.508 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.508 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.508 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.508 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-11T19:21:28.508 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.508 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.508 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.509 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.509 INFO:teuthology.orchestra.run.smithi029.stdout: "default_cephadm_command_timeout": { 2024-03-11T19:21:28.509 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default_cephadm_command_timeout", 2024-03-11T19:21:28.509 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.509 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.509 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.509 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "900", 2024-03-11T19:21:28.509 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.509 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.509 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.509 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-11T19:21:28.509 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.509 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.509 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.509 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.509 INFO:teuthology.orchestra.run.smithi029.stdout: "default_registry": { 2024-03-11T19:21:28.509 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default_registry", 2024-03-11T19:21:28.509 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.510 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.510 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.510 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io", 2024-03-11T19:21:28.510 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.510 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.510 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.510 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-11T19:21:28.510 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.510 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.510 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.510 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.510 INFO:teuthology.orchestra.run.smithi029.stdout: "device_cache_timeout": { 2024-03-11T19:21:28.510 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_cache_timeout", 2024-03-11T19:21:28.510 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.510 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.511 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.511 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1800", 2024-03-11T19:21:28.511 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.511 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.511 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.511 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache device inventory", 2024-03-11T19:21:28.511 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.511 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.511 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.511 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.511 INFO:teuthology.orchestra.run.smithi029.stdout: "device_enhanced_scan": { 2024-03-11T19:21:28.511 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_enhanced_scan", 2024-03-11T19:21:28.511 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.511 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.511 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.512 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.512 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.512 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.512 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.512 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-11T19:21:28.512 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.512 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.512 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.512 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.512 INFO:teuthology.orchestra.run.smithi029.stdout: "facts_cache_timeout": { 2024-03-11T19:21:28.512 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "facts_cache_timeout", 2024-03-11T19:21:28.512 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.512 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.512 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.512 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-11T19:21:28.512 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.513 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.513 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.513 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache host facts data", 2024-03-11T19:21:28.513 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.513 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.513 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.513 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.513 INFO:teuthology.orchestra.run.smithi029.stdout: "host_check_interval": { 2024-03-11T19:21:28.513 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "host_check_interval", 2024-03-11T19:21:28.513 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.513 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.513 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.513 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-11T19:21:28.513 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.513 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.513 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.513 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to perform a host check", 2024-03-11T19:21:28.513 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.514 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.514 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.514 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.514 INFO:teuthology.orchestra.run.smithi029.stdout: "hw_monitoring": { 2024-03-11T19:21:28.514 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "hw_monitoring", 2024-03-11T19:21:28.514 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.514 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.514 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.514 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.514 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.514 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.514 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.514 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-11T19:21:28.514 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.514 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.514 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.515 INFO:teuthology.orchestra.run.smithi029.stdout: "inventory_list_all": { 2024-03-11T19:21:28.515 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "inventory_list_all", 2024-03-11T19:21:28.515 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.515 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.515 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.515 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.515 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.515 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.515 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.515 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-11T19:21:28.515 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.515 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.515 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.515 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.515 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.516 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.516 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.516 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.516 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.516 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.516 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.516 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.516 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.516 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.516 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.516 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.516 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.516 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.516 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.516 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.516 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.516 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.517 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.517 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.517 INFO:teuthology.orchestra.run.smithi029.stdout: "log_refresh_metadata": { 2024-03-11T19:21:28.517 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_refresh_metadata", 2024-03-11T19:21:28.517 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.517 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.517 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.517 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.517 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.517 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.517 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.517 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-11T19:21:28.517 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.517 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.517 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.517 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.517 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.517 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.518 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.518 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.518 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.518 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.518 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.518 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.518 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.518 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-11T19:21:28.518 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.518 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.518 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.518 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.518 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.518 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.518 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.518 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.519 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.519 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.519 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.519 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.519 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.519 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.519 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.519 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.519 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.519 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.519 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.519 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.519 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.519 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.519 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.519 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.520 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.520 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.520 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.520 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.520 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.520 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.520 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.520 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.520 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.520 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.520 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.520 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.520 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.520 INFO:teuthology.orchestra.run.smithi029.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-11T19:21:28.520 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-11T19:21:28.521 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.521 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.521 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.521 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.521 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.521 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.521 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.521 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-11T19:21:28.521 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.521 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.521 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.521 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.521 INFO:teuthology.orchestra.run.smithi029.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-11T19:21:28.521 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-11T19:21:28.521 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.521 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.522 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.522 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "*", 2024-03-11T19:21:28.522 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.522 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.522 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.522 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-11T19:21:28.522 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.522 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.522 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.522 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.522 INFO:teuthology.orchestra.run.smithi029.stdout: "max_count_per_host": { 2024-03-11T19:21:28.522 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_count_per_host", 2024-03-11T19:21:28.522 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.522 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.522 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.523 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:28.523 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.523 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.523 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.523 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "max number of daemons per service per host", 2024-03-11T19:21:28.523 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.523 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.523 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.523 INFO:teuthology.orchestra.run.smithi029.stdout: "max_osd_draining_count": { 2024-03-11T19:21:28.523 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_osd_draining_count", 2024-03-11T19:21:28.523 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.523 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.523 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.523 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:28.524 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.524 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.524 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.524 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-11T19:21:28.524 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.524 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.524 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.524 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.524 INFO:teuthology.orchestra.run.smithi029.stdout: "migration_current": { 2024-03-11T19:21:28.524 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "migration_current", 2024-03-11T19:21:28.524 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.524 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.524 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.524 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.524 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.524 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.525 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.525 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "internal - do not modify", 2024-03-11T19:21:28.525 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.525 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.525 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.525 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.525 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": { 2024-03-11T19:21:28.525 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mode", 2024-03-11T19:21:28.525 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.525 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.525 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.525 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "root", 2024-03-11T19:21:28.525 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.525 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.525 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.526 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm-package", 2024-03-11T19:21:28.526 INFO:teuthology.orchestra.run.smithi029.stdout: "root" 2024-03-11T19:21:28.526 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.526 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "mode for remote execution of cephadm", 2024-03-11T19:21:28.526 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.526 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.526 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.526 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.526 INFO:teuthology.orchestra.run.smithi029.stdout: "oob_default_addr": { 2024-03-11T19:21:28.526 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "oob_default_addr", 2024-03-11T19:21:28.526 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.526 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.526 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.526 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "169.254.1.1", 2024-03-11T19:21:28.526 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.526 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.527 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.527 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-11T19:21:28.527 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.527 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.527 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.527 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.527 INFO:teuthology.orchestra.run.smithi029.stdout: "prometheus_alerts_path": { 2024-03-11T19:21:28.527 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prometheus_alerts_path", 2024-03-11T19:21:28.527 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.527 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.527 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.527 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-11T19:21:28.527 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.527 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.527 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.527 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-11T19:21:28.528 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.528 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.528 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.528 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.528 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_insecure": { 2024-03-11T19:21:28.528 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_insecure", 2024-03-11T19:21:28.528 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.528 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.528 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.528 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.528 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.528 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.528 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.528 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-11T19:21:28.528 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.528 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.528 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.528 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.529 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_password": { 2024-03-11T19:21:28.529 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_password", 2024-03-11T19:21:28.529 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.529 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.529 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.529 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.529 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.529 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.529 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.529 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-11T19:21:28.529 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.529 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.529 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.529 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.529 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_url": { 2024-03-11T19:21:28.529 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_url", 2024-03-11T19:21:28.530 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.530 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.530 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.530 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.530 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.530 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.530 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.530 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-11T19:21:28.530 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.530 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.530 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.530 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.530 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_username": { 2024-03-11T19:21:28.530 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_username", 2024-03-11T19:21:28.531 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.531 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.531 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.531 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.531 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.531 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.531 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.531 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-11T19:21:28.531 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.531 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.531 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.531 INFO:teuthology.orchestra.run.smithi029.stdout: "secure_monitoring_stack": { 2024-03-11T19:21:28.531 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "secure_monitoring_stack", 2024-03-11T19:21:28.531 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.531 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.532 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.532 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.532 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.532 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.532 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.532 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-11T19:21:28.532 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.532 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.532 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.532 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.532 INFO:teuthology.orchestra.run.smithi029.stdout: "service_discovery_port": { 2024-03-11T19:21:28.532 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "service_discovery_port", 2024-03-11T19:21:28.532 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.532 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.532 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.532 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8765", 2024-03-11T19:21:28.533 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.533 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.533 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.533 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "cephadm service discovery port", 2024-03-11T19:21:28.533 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.533 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.533 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.533 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.533 INFO:teuthology.orchestra.run.smithi029.stdout: "ssh_config_file": { 2024-03-11T19:21:28.533 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssh_config_file", 2024-03-11T19:21:28.533 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.533 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.533 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.533 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.533 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.533 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.533 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.533 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-11T19:21:28.534 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.534 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.534 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.534 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.534 INFO:teuthology.orchestra.run.smithi029.stdout: "use_agent": { 2024-03-11T19:21:28.534 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "use_agent", 2024-03-11T19:21:28.534 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.534 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.534 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.534 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.534 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.534 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.534 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.535 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-11T19:21:28.535 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.535 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.535 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.535 INFO:teuthology.orchestra.run.smithi029.stdout: "use_repo_digest": { 2024-03-11T19:21:28.535 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "use_repo_digest", 2024-03-11T19:21:28.535 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.535 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.535 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.535 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.535 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.535 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.535 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.535 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-11T19:21:28.536 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.536 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.536 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.536 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.536 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_failed_host_check": { 2024-03-11T19:21:28.536 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_failed_host_check", 2024-03-11T19:21:28.536 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.536 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.536 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.536 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.536 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.536 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.536 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.536 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if the host check fails", 2024-03-11T19:21:28.536 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.536 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.536 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.537 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_stray_daemons": { 2024-03-11T19:21:28.537 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_stray_daemons", 2024-03-11T19:21:28.537 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.537 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.537 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.537 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.537 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.537 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.537 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.537 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-11T19:21:28.537 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.538 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.538 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.538 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_stray_hosts": { 2024-03-11T19:21:28.538 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_stray_hosts", 2024-03-11T19:21:28.538 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.538 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.538 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.538 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.538 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.538 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.538 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.538 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-11T19:21:28.538 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.538 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.538 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.539 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.539 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.539 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crash", 2024-03-11T19:21:28.539 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.539 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.539 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.539 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.539 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.539 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.539 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.539 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.539 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.539 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.539 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.540 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.540 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.540 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.540 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.540 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.540 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.540 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.540 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.540 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.540 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.540 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.540 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.540 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.540 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.540 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.540 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.541 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.541 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.541 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.541 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.541 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.541 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.541 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.541 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.541 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.541 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.541 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.541 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.541 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.541 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.541 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.542 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.542 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.542 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.542 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.542 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.542 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.542 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.542 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.542 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.542 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.542 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.542 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.542 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.542 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.543 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.543 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.543 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.543 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.543 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.543 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.543 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.543 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.543 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.543 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.543 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.543 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.543 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.543 INFO:teuthology.orchestra.run.smithi029.stdout: "retain_interval": { 2024-03-11T19:21:28.544 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "retain_interval", 2024-03-11T19:21:28.544 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.544 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.544 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.544 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "31536000", 2024-03-11T19:21:28.544 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.544 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.544 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.544 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-11T19:21:28.544 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.544 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.544 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.544 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_recent_interval": { 2024-03-11T19:21:28.544 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_recent_interval", 2024-03-11T19:21:28.544 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.545 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.545 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.545 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1209600", 2024-03-11T19:21:28.545 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.545 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.545 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.545 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-11T19:21:28.545 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.545 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.545 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.545 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.545 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.545 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.545 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.545 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "dashboard", 2024-03-11T19:21:28.545 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.545 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.546 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.546 INFO:teuthology.orchestra.run.smithi029.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-11T19:21:28.546 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-11T19:21:28.546 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.546 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.546 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.546 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:28.546 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.546 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.546 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.546 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.546 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.546 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.546 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.546 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.547 INFO:teuthology.orchestra.run.smithi029.stdout: "ALERTMANAGER_API_HOST": { 2024-03-11T19:21:28.547 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-11T19:21:28.547 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.547 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.547 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.547 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.547 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.547 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.547 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.547 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.547 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.547 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.547 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.547 INFO:teuthology.orchestra.run.smithi029.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-11T19:21:28.547 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-11T19:21:28.547 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.548 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.548 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.548 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.548 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.548 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.548 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.548 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.548 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.548 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.548 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.548 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.548 INFO:teuthology.orchestra.run.smithi029.stdout: "AUDIT_API_ENABLED": { 2024-03-11T19:21:28.548 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "AUDIT_API_ENABLED", 2024-03-11T19:21:28.548 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.548 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.548 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.548 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.548 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.549 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.549 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.549 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.549 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.549 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.549 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.549 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.549 INFO:teuthology.orchestra.run.smithi029.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-11T19:21:28.549 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-11T19:21:28.549 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.549 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.549 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.549 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.549 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.549 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.550 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.550 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.550 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.550 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.550 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.550 INFO:teuthology.orchestra.run.smithi029.stdout: "ENABLE_BROWSABLE_API": { 2024-03-11T19:21:28.550 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-11T19:21:28.550 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.550 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.550 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.550 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.550 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.550 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.551 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.551 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.551 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.551 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.551 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.551 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.551 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-11T19:21:28.551 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-11T19:21:28.551 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.551 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.551 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.551 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.551 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.551 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.551 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.551 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.552 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.552 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.552 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.552 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-11T19:21:28.552 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-11T19:21:28.552 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.552 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.552 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.552 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.552 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.552 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.552 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.552 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.552 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.552 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.552 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.553 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.553 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-11T19:21:28.553 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-11T19:21:28.553 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.553 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.553 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.553 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.553 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.553 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.553 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.553 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.553 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.553 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.553 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.554 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.554 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-11T19:21:28.554 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-11T19:21:28.554 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.554 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.554 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.554 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.554 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.554 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.554 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.554 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.554 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.554 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.554 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.554 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.554 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-11T19:21:28.555 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-11T19:21:28.555 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.555 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.555 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.555 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.555 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.555 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.555 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.555 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.555 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.555 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.555 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.555 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.556 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-11T19:21:28.556 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-11T19:21:28.556 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.556 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.556 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.556 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.556 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.556 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.556 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.556 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.556 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.556 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.556 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.556 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-11T19:21:28.556 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-11T19:21:28.557 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.557 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.557 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.557 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.557 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.557 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.557 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.557 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.557 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.557 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.557 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.557 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.557 INFO:teuthology.orchestra.run.smithi029.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-11T19:21:28.557 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-11T19:21:28.557 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.558 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.558 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.558 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.558 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.558 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.558 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.558 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.558 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.558 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.558 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.558 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_PASSWORD": { 2024-03-11T19:21:28.558 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-11T19:21:28.559 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.559 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.559 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.559 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-03-11T19:21:28.559 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.559 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.559 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.559 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.559 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.559 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.559 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.559 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.559 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-11T19:21:28.559 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-11T19:21:28.559 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.559 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.559 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.560 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.560 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.560 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.560 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.560 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.560 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.560 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.560 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.560 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.560 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_URL": { 2024-03-11T19:21:28.560 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_URL", 2024-03-11T19:21:28.560 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.560 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.560 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.560 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.560 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.560 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.560 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.561 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.561 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.561 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.561 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.561 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_USERNAME": { 2024-03-11T19:21:28.561 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-11T19:21:28.561 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.561 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.561 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.561 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-03-11T19:21:28.561 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.561 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.561 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.561 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.561 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.562 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.562 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.562 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.562 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-11T19:21:28.562 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-11T19:21:28.562 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.562 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.562 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.562 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.562 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.562 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.562 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.562 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.562 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.563 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.563 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.563 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.563 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-11T19:21:28.563 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-11T19:21:28.563 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.563 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.563 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.563 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.563 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.563 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.563 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.563 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.563 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.563 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.563 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.564 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.564 INFO:teuthology.orchestra.run.smithi029.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-11T19:21:28.564 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-11T19:21:28.564 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.564 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.564 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.564 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.564 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.564 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.564 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.564 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.564 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.564 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.564 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.564 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.564 INFO:teuthology.orchestra.run.smithi029.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-11T19:21:28.565 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-11T19:21:28.565 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.565 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.565 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.565 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.565 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.565 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.565 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.565 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.565 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.565 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.565 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.565 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.565 INFO:teuthology.orchestra.run.smithi029.stdout: "PROMETHEUS_API_HOST": { 2024-03-11T19:21:28.565 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-11T19:21:28.566 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.566 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.566 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.566 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.566 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.566 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.566 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.566 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.566 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.566 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.566 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.566 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.566 INFO:teuthology.orchestra.run.smithi029.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-11T19:21:28.566 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-11T19:21:28.566 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.566 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.567 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.567 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.567 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.567 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.567 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.567 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.567 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.567 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.567 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.567 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.567 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-11T19:21:28.567 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-11T19:21:28.567 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.567 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.567 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.568 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.568 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.568 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.568 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.568 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.568 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.568 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.568 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.568 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.568 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-11T19:21:28.568 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-11T19:21:28.568 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.568 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.568 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.568 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.568 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.568 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.569 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.569 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.569 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.569 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.569 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.569 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.569 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-11T19:21:28.569 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-11T19:21:28.569 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.569 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.569 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.569 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.569 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.569 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.569 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.569 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.569 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.569 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.570 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.570 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.570 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-11T19:21:28.570 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-11T19:21:28.570 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.570 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.570 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.570 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.570 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.570 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.570 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.570 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.570 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.570 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.571 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.571 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.571 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-11T19:21:28.571 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-11T19:21:28.571 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.571 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.571 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.571 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.571 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.571 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.571 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.571 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.571 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.571 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.571 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.571 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.572 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-11T19:21:28.572 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-11T19:21:28.572 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.572 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.572 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.572 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.572 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.572 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.572 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.572 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.572 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.572 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.572 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.572 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.572 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-11T19:21:28.572 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-11T19:21:28.572 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.572 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.573 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.573 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.573 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.573 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.573 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.573 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.573 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.573 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.573 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.573 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.573 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_ENABLED": { 2024-03-11T19:21:28.573 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-11T19:21:28.573 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.573 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.573 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.573 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.573 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.573 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.574 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.574 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.574 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.574 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.574 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.574 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.574 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-11T19:21:28.574 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-11T19:21:28.574 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.574 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.574 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.574 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-11T19:21:28.574 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.574 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.574 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.574 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.575 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.575 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.575 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.575 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.575 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-11T19:21:28.575 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-11T19:21:28.575 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.575 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.575 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.575 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:28.575 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.575 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.575 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.575 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.575 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.575 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.576 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.576 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-11T19:21:28.576 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-11T19:21:28.576 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.576 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.576 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.576 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8", 2024-03-11T19:21:28.576 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.576 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.576 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.576 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.576 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.576 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.576 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.577 INFO:teuthology.orchestra.run.smithi029.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-11T19:21:28.577 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-11T19:21:28.577 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.577 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.577 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.577 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "45", 2024-03-11T19:21:28.577 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.577 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.577 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.577 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.577 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.577 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.577 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.577 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_ACCESS_KEY": { 2024-03-11T19:21:28.578 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-11T19:21:28.578 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.578 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.578 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.578 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.578 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.578 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.578 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.578 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.578 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.578 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.578 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.578 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-11T19:21:28.578 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-11T19:21:28.579 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.579 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.579 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.579 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-03-11T19:21:28.579 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.579 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.579 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.579 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.579 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.579 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.579 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.579 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_SECRET_KEY": { 2024-03-11T19:21:28.579 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-11T19:21:28.579 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.579 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.580 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.580 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.580 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.580 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.580 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.580 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.580 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.580 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.580 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.580 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_SSL_VERIFY": { 2024-03-11T19:21:28.580 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-11T19:21:28.580 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.580 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.580 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.580 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.580 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.581 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.581 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.581 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.581 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.581 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.581 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.581 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-11T19:21:28.581 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-11T19:21:28.581 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.581 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.581 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.581 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-11T19:21:28.581 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.581 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.582 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.582 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.582 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.582 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.582 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.582 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-11T19:21:28.582 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-11T19:21:28.582 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.582 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.582 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.582 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:28.582 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.582 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.582 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.583 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.583 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.583 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.583 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.583 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.583 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-11T19:21:28.583 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-11T19:21:28.583 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.583 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.583 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.583 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-11T19:21:28.583 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.583 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.583 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.584 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.584 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.584 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.584 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.584 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.584 INFO:teuthology.orchestra.run.smithi029.stdout: "cross_origin_url": { 2024-03-11T19:21:28.584 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cross_origin_url", 2024-03-11T19:21:28.584 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.584 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.584 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.584 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.584 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.584 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.584 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.584 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.585 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.585 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.585 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.585 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.585 INFO:teuthology.orchestra.run.smithi029.stdout: "crt_file": { 2024-03-11T19:21:28.585 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crt_file", 2024-03-11T19:21:28.585 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.585 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.585 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.585 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.585 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.585 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.585 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.585 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.585 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.585 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.586 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.586 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.586 INFO:teuthology.orchestra.run.smithi029.stdout: "debug": { 2024-03-11T19:21:28.586 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "debug", 2024-03-11T19:21:28.586 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.586 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.586 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.586 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.586 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.586 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.586 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.586 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable/disable debug options", 2024-03-11T19:21:28.586 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.586 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.586 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.587 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.587 INFO:teuthology.orchestra.run.smithi029.stdout: "jwt_token_ttl": { 2024-03-11T19:21:28.587 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "jwt_token_ttl", 2024-03-11T19:21:28.587 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.587 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.587 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.587 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "28800", 2024-03-11T19:21:28.587 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.587 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.587 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.587 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.587 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.588 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.588 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.588 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.588 INFO:teuthology.orchestra.run.smithi029.stdout: "key_file": { 2024-03-11T19:21:28.588 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "key_file", 2024-03-11T19:21:28.588 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.588 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.588 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.588 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.588 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.588 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.589 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.589 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.589 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.589 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.589 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.589 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.589 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.589 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.589 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.589 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.589 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.589 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.589 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.589 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.590 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.590 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.590 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.590 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.590 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.590 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.590 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.590 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.590 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.590 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.590 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.590 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.590 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.591 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.591 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.591 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.591 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.591 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.591 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.591 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.591 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.591 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.591 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.591 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.591 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.592 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.592 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.592 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.592 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.592 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.592 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.592 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.592 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.592 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.593 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.593 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.593 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.593 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.593 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.593 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.593 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.593 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.593 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.593 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.593 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.593 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.594 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.594 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.594 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.594 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.594 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.594 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.594 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.594 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.594 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.594 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.594 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.594 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.594 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.595 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.595 INFO:teuthology.orchestra.run.smithi029.stdout: "motd": { 2024-03-11T19:21:28.595 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "motd", 2024-03-11T19:21:28.595 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.595 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.595 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.595 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.595 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.595 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.595 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.595 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "The message of the day", 2024-03-11T19:21:28.595 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.595 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.596 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.596 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.596 INFO:teuthology.orchestra.run.smithi029.stdout: "redirect_resolve_ip_addr": { 2024-03-11T19:21:28.596 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "redirect_resolve_ip_addr", 2024-03-11T19:21:28.596 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.596 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.596 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.596 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.596 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.596 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.596 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.596 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.596 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.596 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.597 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.597 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.597 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-03-11T19:21:28.597 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-03-11T19:21:28.597 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.597 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.597 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.597 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "::", 2024-03-11T19:21:28.597 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.597 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.597 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.597 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.597 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.597 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.598 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.598 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.598 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-03-11T19:21:28.598 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-03-11T19:21:28.598 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.598 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.598 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.598 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8080", 2024-03-11T19:21:28.598 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.598 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.598 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.598 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.598 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.598 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.599 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.599 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.599 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl": { 2024-03-11T19:21:28.599 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl", 2024-03-11T19:21:28.599 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.599 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.599 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.599 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.599 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.599 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.599 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.599 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.599 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.599 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.600 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.600 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.600 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl_server_port": { 2024-03-11T19:21:28.600 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl_server_port", 2024-03-11T19:21:28.600 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.600 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.600 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.600 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8443", 2024-03-11T19:21:28.600 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.600 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.600 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.600 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.600 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.600 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.601 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.601 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.601 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_behaviour": { 2024-03-11T19:21:28.601 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_behaviour", 2024-03-11T19:21:28.601 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.601 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.601 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.601 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "redirect", 2024-03-11T19:21:28.601 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.601 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.601 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.601 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.601 INFO:teuthology.orchestra.run.smithi029.stdout: "redirect" 2024-03-11T19:21:28.601 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.601 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.602 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.602 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.602 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.602 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.602 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_error_status_code": { 2024-03-11T19:21:28.602 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_error_status_code", 2024-03-11T19:21:28.602 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.602 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.602 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.602 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-03-11T19:21:28.602 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "400", 2024-03-11T19:21:28.602 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "599", 2024-03-11T19:21:28.602 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.602 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.602 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.602 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.603 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.603 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.603 INFO:teuthology.orchestra.run.smithi029.stdout: "url_prefix": { 2024-03-11T19:21:28.603 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "url_prefix", 2024-03-11T19:21:28.603 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.603 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.603 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.603 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.603 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.603 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.603 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.603 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.603 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.603 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.603 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.603 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.604 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.604 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.604 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.604 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "devicehealth", 2024-03-11T19:21:28.604 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.604 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.604 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.604 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_monitoring": { 2024-03-11T19:21:28.604 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enable_monitoring", 2024-03-11T19:21:28.604 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.604 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.604 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.604 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.604 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.604 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.604 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.604 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "monitor device health metrics", 2024-03-11T19:21:28.605 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.605 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.605 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.605 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.605 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.605 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.605 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.605 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.605 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.605 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.605 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.605 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.605 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.605 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.605 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.605 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.606 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.606 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.606 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.606 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.606 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.606 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.606 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.606 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.606 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.606 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.606 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.606 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.606 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.606 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.606 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.607 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.607 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.607 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.607 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.607 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.607 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.607 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.607 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.607 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.607 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.607 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.607 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.607 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.607 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.607 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.607 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.608 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.608 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.608 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.608 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.608 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.608 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.608 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.608 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.608 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.608 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.608 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.608 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.608 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.608 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.608 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.608 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.609 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.609 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.609 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.609 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.609 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.609 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.609 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.609 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.609 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.609 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.609 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.609 INFO:teuthology.orchestra.run.smithi029.stdout: "mark_out_threshold": { 2024-03-11T19:21:28.609 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mark_out_threshold", 2024-03-11T19:21:28.609 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.609 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.610 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.610 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "2419200", 2024-03-11T19:21:28.610 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.610 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.610 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.610 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-11T19:21:28.610 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.610 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.610 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.610 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.610 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": { 2024-03-11T19:21:28.610 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pool_name", 2024-03-11T19:21:28.610 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.611 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.611 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.611 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "device_health_metrics", 2024-03-11T19:21:28.611 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.611 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.611 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.611 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-11T19:21:28.611 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.611 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.611 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.611 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.611 INFO:teuthology.orchestra.run.smithi029.stdout: "retention_period": { 2024-03-11T19:21:28.611 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "retention_period", 2024-03-11T19:21:28.611 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.612 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.612 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.612 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15552000", 2024-03-11T19:21:28.612 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.612 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.612 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.612 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long to retain device health metrics", 2024-03-11T19:21:28.612 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.612 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.612 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.612 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.612 INFO:teuthology.orchestra.run.smithi029.stdout: "scrape_frequency": { 2024-03-11T19:21:28.612 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "scrape_frequency", 2024-03-11T19:21:28.612 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.613 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.613 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.613 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "86400", 2024-03-11T19:21:28.613 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.613 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.613 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.613 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-11T19:21:28.613 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.613 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.613 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.613 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.613 INFO:teuthology.orchestra.run.smithi029.stdout: "self_heal": { 2024-03-11T19:21:28.613 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "self_heal", 2024-03-11T19:21:28.613 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.613 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.614 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.614 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.614 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.614 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.614 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.614 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-11T19:21:28.614 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.614 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.614 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.614 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.614 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-11T19:21:28.614 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-11T19:21:28.614 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.614 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.614 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.615 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-11T19:21:28.615 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.615 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.615 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.615 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to wake up and check device health", 2024-03-11T19:21:28.615 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.615 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.615 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.615 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.615 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_threshold": { 2024-03-11T19:21:28.615 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_threshold", 2024-03-11T19:21:28.615 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.615 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.615 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.616 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7257600", 2024-03-11T19:21:28.616 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.616 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.616 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.616 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-11T19:21:28.616 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.616 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.616 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.616 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.616 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.616 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.616 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.616 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "diskprediction_local", 2024-03-11T19:21:28.616 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.616 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.617 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.617 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.617 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.617 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.617 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.617 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.617 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.617 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.617 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.617 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.617 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.617 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.617 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.617 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.617 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.618 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.618 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.618 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.618 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.618 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.618 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.618 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.618 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.618 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.618 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.618 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.618 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.618 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.619 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.619 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.619 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.619 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.619 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.619 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.619 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.619 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.619 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.619 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.619 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.619 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.619 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.619 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.619 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.620 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.620 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.620 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.620 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.620 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.620 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.620 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.620 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.620 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.620 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.620 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.620 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.620 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.620 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.620 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.621 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.621 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.621 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.621 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.621 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.621 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.621 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.621 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.621 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.621 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.621 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.621 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.622 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.622 INFO:teuthology.orchestra.run.smithi029.stdout: "predict_interval": { 2024-03-11T19:21:28.622 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "predict_interval", 2024-03-11T19:21:28.622 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.622 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.622 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.622 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "86400", 2024-03-11T19:21:28.622 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.622 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.622 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.622 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.622 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.622 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.622 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.622 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.623 INFO:teuthology.orchestra.run.smithi029.stdout: "predictor_model": { 2024-03-11T19:21:28.623 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "predictor_model", 2024-03-11T19:21:28.623 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.623 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.623 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.623 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "prophetstor", 2024-03-11T19:21:28.623 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.623 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.623 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.623 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.623 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.623 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.623 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.623 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.624 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-11T19:21:28.624 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-11T19:21:28.624 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.624 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.624 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.624 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-11T19:21:28.624 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.624 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.624 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.624 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.624 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.624 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.624 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.624 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.624 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.624 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.625 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.625 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "influx", 2024-03-11T19:21:28.625 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": false, 2024-03-11T19:21:28.625 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "influxdb python module not found", 2024-03-11T19:21:28.625 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.625 INFO:teuthology.orchestra.run.smithi029.stdout: "batch_size": { 2024-03-11T19:21:28.625 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "batch_size", 2024-03-11T19:21:28.625 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.625 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.625 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.625 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5000", 2024-03-11T19:21:28.625 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.625 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.625 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.626 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-11T19:21:28.626 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.626 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.626 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.626 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.626 INFO:teuthology.orchestra.run.smithi029.stdout: "database": { 2024-03-11T19:21:28.626 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "database", 2024-03-11T19:21:28.626 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.626 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.626 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.626 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "ceph", 2024-03-11T19:21:28.626 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.626 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.626 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.627 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-11T19:21:28.627 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.627 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.627 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.627 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.627 INFO:teuthology.orchestra.run.smithi029.stdout: "hostname": { 2024-03-11T19:21:28.627 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "hostname", 2024-03-11T19:21:28.627 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.627 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.627 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.627 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.627 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.627 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.627 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.627 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB server hostname", 2024-03-11T19:21:28.628 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.628 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.628 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.628 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.628 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-11T19:21:28.628 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-11T19:21:28.628 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.628 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.628 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.628 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "30", 2024-03-11T19:21:28.628 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "5", 2024-03-11T19:21:28.628 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.628 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.628 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-11T19:21:28.629 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.629 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.629 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.629 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.629 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.629 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.629 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.629 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.629 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.629 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.629 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.629 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.629 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.629 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.630 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.630 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.630 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.630 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.630 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.630 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.630 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.630 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.630 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.630 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.630 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.630 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.630 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.631 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.631 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.631 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.631 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.631 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.631 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.631 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.631 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.631 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.631 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.631 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.631 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.631 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.631 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.631 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.631 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.632 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.632 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.632 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.632 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.632 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.632 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.632 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.632 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.632 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.632 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.632 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.632 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.632 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.632 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.632 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.632 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.633 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.633 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.633 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.633 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.633 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.633 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.633 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.633 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.633 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.633 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.633 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.633 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.633 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.633 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.633 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.633 INFO:teuthology.orchestra.run.smithi029.stdout: "password": { 2024-03-11T19:21:28.634 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "password", 2024-03-11T19:21:28.634 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.634 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.634 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.634 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.634 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.634 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.634 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.634 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "password of InfluxDB server user", 2024-03-11T19:21:28.634 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.634 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.634 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.634 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.634 INFO:teuthology.orchestra.run.smithi029.stdout: "port": { 2024-03-11T19:21:28.634 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "port", 2024-03-11T19:21:28.634 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.635 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.635 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.635 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8086", 2024-03-11T19:21:28.635 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.635 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.635 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.635 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB server port", 2024-03-11T19:21:28.635 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.635 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.635 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.635 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.635 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl": { 2024-03-11T19:21:28.635 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl", 2024-03-11T19:21:28.636 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.636 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.636 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.636 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "false", 2024-03-11T19:21:28.636 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.636 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.636 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.636 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-11T19:21:28.636 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.636 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.636 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.636 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.636 INFO:teuthology.orchestra.run.smithi029.stdout: "threads": { 2024-03-11T19:21:28.636 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "threads", 2024-03-11T19:21:28.636 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.636 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.636 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.637 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-11T19:21:28.637 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-11T19:21:28.637 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "32", 2024-03-11T19:21:28.637 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.637 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-11T19:21:28.637 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.637 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.637 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.637 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.637 INFO:teuthology.orchestra.run.smithi029.stdout: "username": { 2024-03-11T19:21:28.637 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "username", 2024-03-11T19:21:28.637 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.637 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.637 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.637 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.637 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.638 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.638 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.638 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "username of InfluxDB server user", 2024-03-11T19:21:28.638 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.638 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.638 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.638 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.638 INFO:teuthology.orchestra.run.smithi029.stdout: "verify_ssl": { 2024-03-11T19:21:28.638 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "verify_ssl", 2024-03-11T19:21:28.638 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.638 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.638 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.638 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "true", 2024-03-11T19:21:28.638 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.638 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.638 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.639 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-11T19:21:28.639 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.639 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.639 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.639 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.639 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.639 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.639 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.639 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "insights", 2024-03-11T19:21:28.639 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.639 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.639 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.639 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.639 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.639 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.639 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.639 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.639 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.640 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.640 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.640 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.640 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.640 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.640 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.640 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.640 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.640 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.640 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.640 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.640 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.640 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.640 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.640 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.640 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.641 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.641 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.641 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.641 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.641 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.641 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.641 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.641 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.641 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.641 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.641 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.641 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.641 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.641 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.641 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.642 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.642 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.642 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.642 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.642 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.642 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.642 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.642 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.642 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.642 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.642 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.642 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.642 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.642 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.642 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.643 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.643 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.643 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.643 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.643 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.643 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.643 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.643 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.643 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.643 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.643 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.643 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.643 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.643 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.643 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.643 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.644 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.644 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.644 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.644 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.644 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.644 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "iostat", 2024-03-11T19:21:28.644 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.644 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.644 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.644 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.644 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.644 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.644 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.644 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.644 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.644 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.645 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.645 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.645 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.645 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.645 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.645 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.645 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.645 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.645 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.645 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.645 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.645 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.645 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.645 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.645 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.645 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.645 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.646 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.646 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.646 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.646 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.646 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.646 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.646 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.646 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.646 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.646 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.646 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.646 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.646 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.647 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.647 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.647 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.647 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.647 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.647 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.647 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.647 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.647 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.647 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.647 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.647 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.647 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.647 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.647 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.648 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.648 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.648 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.648 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.648 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.648 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.648 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.648 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.648 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.648 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.648 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.648 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.648 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.648 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.648 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.649 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.649 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.649 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.649 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.649 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.649 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.649 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "k8sevents", 2024-03-11T19:21:28.649 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.649 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.649 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.649 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_event_retention_days": { 2024-03-11T19:21:28.649 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ceph_event_retention_days", 2024-03-11T19:21:28.649 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.650 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.650 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.650 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7", 2024-03-11T19:21:28.650 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.650 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.650 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.650 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-11T19:21:28.650 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.650 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.650 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.650 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.650 INFO:teuthology.orchestra.run.smithi029.stdout: "config_check_secs": { 2024-03-11T19:21:28.650 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_check_secs", 2024-03-11T19:21:28.651 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.651 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.651 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.651 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:28.651 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "10", 2024-03-11T19:21:28.651 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.651 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.651 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-11T19:21:28.651 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.651 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.651 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.651 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.651 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.651 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.651 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.652 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.652 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.652 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.652 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.652 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.652 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.652 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.652 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.652 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.652 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.652 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.652 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.652 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.652 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.653 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.653 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.653 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.653 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.653 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.653 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.653 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.653 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.653 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.653 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.653 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.653 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.653 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.653 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.654 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.654 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.654 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.654 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.654 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.654 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.654 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.654 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.654 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.654 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.654 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.654 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.654 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.654 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.655 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.655 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.655 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.655 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.655 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.655 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.655 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.655 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.655 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.655 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.655 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.655 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.655 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.655 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.656 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.656 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.656 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.656 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.656 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.656 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.656 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.656 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.656 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.656 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.656 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.656 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.656 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.656 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.657 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "localpool", 2024-03-11T19:21:28.657 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.657 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.657 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.657 INFO:teuthology.orchestra.run.smithi029.stdout: "failure_domain": { 2024-03-11T19:21:28.657 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "failure_domain", 2024-03-11T19:21:28.657 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.657 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.657 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.657 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "host", 2024-03-11T19:21:28.657 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.657 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.657 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.657 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "failure domain for any created local pool", 2024-03-11T19:21:28.657 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-11T19:21:28.657 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.658 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.658 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.658 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.658 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.658 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.658 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.658 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.658 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.658 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.658 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.658 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.658 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.658 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.658 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.658 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.658 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.658 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.659 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.659 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.659 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.659 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.659 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.659 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.659 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.659 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.659 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.659 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.659 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.659 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.659 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.659 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.659 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.659 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.659 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.660 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.660 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.660 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.660 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.660 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.660 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.660 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.660 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.660 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.660 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.660 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.660 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.660 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.660 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.660 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.661 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.661 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.661 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.661 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.661 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.661 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.661 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.661 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.661 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.661 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.661 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.661 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.661 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.661 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.661 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.662 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.662 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.662 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.662 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.662 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.662 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.662 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.662 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.662 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": { 2024-03-11T19:21:28.662 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "min_size", 2024-03-11T19:21:28.662 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.662 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.662 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.662 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.662 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.662 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.663 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.663 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default min_size for any created local pool", 2024-03-11T19:21:28.663 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-11T19:21:28.663 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.663 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.663 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.663 INFO:teuthology.orchestra.run.smithi029.stdout: "num_rep": { 2024-03-11T19:21:28.663 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "num_rep", 2024-03-11T19:21:28.663 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.663 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.663 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.663 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3", 2024-03-11T19:21:28.663 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.663 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.663 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.664 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default replica count for any created local pool", 2024-03-11T19:21:28.664 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.664 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.664 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.664 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.664 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": { 2024-03-11T19:21:28.664 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pg_num", 2024-03-11T19:21:28.664 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.664 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.664 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.664 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "128", 2024-03-11T19:21:28.664 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.664 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.664 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.664 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default pg_num for any created local pool", 2024-03-11T19:21:28.665 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.665 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.665 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.665 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.665 INFO:teuthology.orchestra.run.smithi029.stdout: "prefix": { 2024-03-11T19:21:28.665 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prefix", 2024-03-11T19:21:28.665 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.665 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.665 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.665 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.665 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.665 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.665 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.665 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "name prefix for any created local pool", 2024-03-11T19:21:28.666 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.666 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.666 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.666 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.666 INFO:teuthology.orchestra.run.smithi029.stdout: "subtree": { 2024-03-11T19:21:28.666 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "subtree", 2024-03-11T19:21:28.666 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.666 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.666 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.666 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "rack", 2024-03-11T19:21:28.666 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.666 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.666 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.666 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-11T19:21:28.666 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-11T19:21:28.667 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.667 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.667 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.667 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.667 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.667 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.667 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mds_autoscaler", 2024-03-11T19:21:28.667 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.667 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.667 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.667 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.667 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.667 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.667 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.667 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.667 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.667 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.668 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.668 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.668 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.668 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.668 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.668 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.668 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.668 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.668 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.668 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.668 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.668 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.668 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.668 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.669 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.669 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.669 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.669 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.669 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.669 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.669 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.669 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.669 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.669 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.669 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.669 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.669 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.669 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.669 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.669 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.669 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.670 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.670 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.670 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.670 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.670 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.670 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.670 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.670 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.670 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.670 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.670 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.670 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.670 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.670 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.670 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.670 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.671 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.671 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.671 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.671 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.671 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.671 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.671 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.671 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.671 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.671 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.671 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.671 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.671 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.671 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.671 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.671 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.672 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.672 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.672 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.672 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mirroring", 2024-03-11T19:21:28.672 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.672 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.672 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.672 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.672 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.672 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.672 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.672 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.672 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.672 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.672 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.672 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.672 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.673 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.673 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.673 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.673 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.673 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.673 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.673 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.673 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.673 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.673 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.673 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.673 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.673 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.673 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.673 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.673 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.673 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.674 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.674 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.674 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.674 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.674 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.674 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.674 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.674 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.674 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.674 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.674 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.674 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.674 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.674 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.674 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.675 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.675 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.675 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.675 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.675 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.675 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.675 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.675 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.675 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.675 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.675 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.675 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.675 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.675 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.675 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.675 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.675 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.676 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.676 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.676 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.676 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.676 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.676 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.676 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.676 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.676 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.676 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.676 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.676 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.676 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.676 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.676 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "nfs", 2024-03-11T19:21:28.676 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.676 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.676 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.677 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.677 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.677 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.677 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.677 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.677 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.677 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.677 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.677 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.677 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.677 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.677 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.677 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.677 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.677 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.677 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.677 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.678 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.678 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.678 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.678 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.678 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.678 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.678 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.678 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.678 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.678 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.678 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.678 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.678 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.678 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.678 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.678 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.679 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.679 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.679 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.679 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.679 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.679 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.679 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.679 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.679 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.679 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.679 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.679 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.679 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.679 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.679 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.679 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.679 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.680 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.680 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.680 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.680 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.680 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.680 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.680 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.680 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.680 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.680 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.680 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.680 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.680 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.680 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.680 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.680 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.680 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.680 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.681 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.681 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.681 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.681 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.681 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.681 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "orchestrator", 2024-03-11T19:21:28.681 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.681 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.681 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.681 INFO:teuthology.orchestra.run.smithi029.stdout: "fail_fs": { 2024-03-11T19:21:28.681 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "fail_fs", 2024-03-11T19:21:28.681 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.681 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.681 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.681 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.681 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.682 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.682 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.682 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-11T19:21:28.682 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.682 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.682 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.682 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.682 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.682 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.682 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.682 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.682 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.683 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.683 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.683 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.683 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.683 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.683 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.683 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.683 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.683 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.683 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.683 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.683 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.683 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.683 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.683 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.683 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.683 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.684 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.684 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.684 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.684 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.684 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.684 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.684 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.684 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.684 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.684 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.684 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.684 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.684 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.684 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.684 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.684 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.685 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.685 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.685 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.685 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.685 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.685 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.685 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.685 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.685 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.685 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.685 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.685 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.685 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.685 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.685 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.685 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.685 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.685 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.686 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.686 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.686 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.686 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.686 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.686 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.686 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.686 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.686 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.686 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.686 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.686 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.686 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.686 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.686 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator": { 2024-03-11T19:21:28.686 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "orchestrator", 2024-03-11T19:21:28.687 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.687 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.687 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.687 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.687 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.687 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.687 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.687 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm", 2024-03-11T19:21:28.687 INFO:teuthology.orchestra.run.smithi029.stdout: "rook", 2024-03-11T19:21:28.687 INFO:teuthology.orchestra.run.smithi029.stdout: "test_orchestrator" 2024-03-11T19:21:28.687 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.687 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Orchestrator backend", 2024-03-11T19:21:28.687 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.687 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.687 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.688 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.688 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.688 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.688 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.688 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "osd_perf_query", 2024-03-11T19:21:28.688 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.688 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.688 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.688 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.688 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.688 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.688 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.688 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.688 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.688 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.688 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.688 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.689 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.689 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.689 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.689 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.689 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.689 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.689 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.689 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.689 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.689 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.689 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.689 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.689 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.689 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.689 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.689 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.690 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.690 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.690 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.690 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.690 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.690 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.690 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.690 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.690 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.690 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.690 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.690 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.690 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.690 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.691 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.691 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.691 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.691 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.691 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.691 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.691 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.691 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.691 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.691 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.691 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.691 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.691 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.691 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.691 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.692 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.692 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.692 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.692 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.692 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.692 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.692 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.692 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.692 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.692 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.692 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.692 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.692 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.692 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.692 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.692 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.693 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.693 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.693 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.693 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "osd_support", 2024-03-11T19:21:28.693 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.693 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.693 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.693 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.693 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.693 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.693 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.693 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.693 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.693 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.693 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.694 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.694 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.694 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.694 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.694 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.694 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.694 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.694 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.694 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.694 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.694 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.694 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.694 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.694 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.694 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.695 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.695 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.695 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.695 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.695 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.695 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.695 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.695 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.695 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.695 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.695 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.695 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.695 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.695 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.695 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.695 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.695 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.696 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.696 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.696 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.696 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.696 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.696 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.696 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.696 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.696 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.696 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.696 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.696 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.696 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.696 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.696 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.697 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.697 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.697 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.697 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.697 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.697 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.697 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.697 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.697 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.697 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.697 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.697 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.697 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.697 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.697 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.697 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.698 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.698 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.698 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pg_autoscaler", 2024-03-11T19:21:28.698 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.698 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.698 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.698 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.698 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.698 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.698 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.698 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.698 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.698 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.698 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.698 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.698 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.699 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.699 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.699 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.699 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.699 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.699 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.699 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.699 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.699 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.699 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.699 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.699 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.699 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.699 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.699 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.699 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.700 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.700 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.700 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.700 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.700 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.700 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.700 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.700 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.700 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.700 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.700 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.700 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.700 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.700 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.700 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.700 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.701 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.701 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.701 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.701 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.701 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.701 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.701 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.701 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.701 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.701 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.701 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.701 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.701 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.701 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.701 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.701 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.702 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.702 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.702 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.702 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.702 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.702 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.702 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.702 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.702 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.702 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.702 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.702 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.702 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-11T19:21:28.702 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-11T19:21:28.703 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.703 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.703 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.703 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-11T19:21:28.703 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.703 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.703 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.703 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.703 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.703 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.703 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.703 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.703 INFO:teuthology.orchestra.run.smithi029.stdout: "threshold": { 2024-03-11T19:21:28.703 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "threshold", 2024-03-11T19:21:28.703 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:28.703 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.704 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.704 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3.0", 2024-03-11T19:21:28.704 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1.0", 2024-03-11T19:21:28.704 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.704 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.704 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "scaling threshold", 2024-03-11T19:21:28.704 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-11T19:21:28.704 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.704 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.704 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.704 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.704 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.704 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.704 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "progress", 2024-03-11T19:21:28.704 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.705 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.705 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.705 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_pg_recovery_event": { 2024-03-11T19:21:28.705 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_pg_recovery_event", 2024-03-11T19:21:28.705 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.705 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.705 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.705 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.705 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.705 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.705 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.705 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-11T19:21:28.705 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.705 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.705 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.705 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.705 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": { 2024-03-11T19:21:28.705 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enabled", 2024-03-11T19:21:28.706 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.706 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.706 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.706 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.706 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.706 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.706 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.706 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.706 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.706 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.706 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.706 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.706 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.706 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.706 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.707 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.707 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.707 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.707 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.707 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.707 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.707 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.707 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.707 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.707 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.707 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.707 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.707 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.707 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.707 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.707 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.707 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.708 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.708 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.708 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.708 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.708 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.708 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.708 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.708 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.708 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.708 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.708 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.708 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.708 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.708 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.708 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.708 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.708 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.709 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.709 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.709 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.709 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.709 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.709 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.709 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.709 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.709 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.709 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.709 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.709 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.709 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.709 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.709 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.709 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.709 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.709 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.710 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.710 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.710 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.710 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.710 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.710 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.710 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.710 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.710 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.710 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.710 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.710 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.710 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.710 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.710 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.710 INFO:teuthology.orchestra.run.smithi029.stdout: "max_completed_events": { 2024-03-11T19:21:28.711 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_completed_events", 2024-03-11T19:21:28.711 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.711 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.711 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.711 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "50", 2024-03-11T19:21:28.711 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.711 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.711 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.711 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "number of past completed events to remember", 2024-03-11T19:21:28.711 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.711 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.711 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.711 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.711 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-11T19:21:28.711 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-11T19:21:28.711 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.711 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.712 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.712 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-11T19:21:28.712 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.712 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.712 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.712 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long the module is going to sleep", 2024-03-11T19:21:28.712 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.712 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.712 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.712 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.712 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.712 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.712 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.712 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prometheus", 2024-03-11T19:21:28.713 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.713 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.713 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.713 INFO:teuthology.orchestra.run.smithi029.stdout: "cache": { 2024-03-11T19:21:28.713 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cache", 2024-03-11T19:21:28.713 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.713 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.713 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.713 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.713 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.713 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.713 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.713 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.713 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.713 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.714 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.714 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.714 INFO:teuthology.orchestra.run.smithi029.stdout: "exclude_perf_counters": { 2024-03-11T19:21:28.714 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "exclude_perf_counters", 2024-03-11T19:21:28.714 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.714 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.714 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.714 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.714 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.714 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.714 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.714 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-11T19:21:28.714 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-11T19:21:28.714 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.714 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.715 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.715 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.715 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.715 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.715 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.715 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.715 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.715 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.715 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.715 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.715 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.715 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.715 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.715 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.715 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.715 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.716 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.716 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.716 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.716 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.716 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.716 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.716 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.716 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.716 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.716 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.716 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.716 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.716 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.716 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.716 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.717 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.717 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.717 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.717 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.717 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.717 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.717 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.717 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.717 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.717 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.717 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.717 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.717 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.717 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.717 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.717 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.718 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.718 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.718 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.718 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.718 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.718 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.718 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.718 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.718 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.718 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.718 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.718 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.718 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.718 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.718 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.718 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.719 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.719 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.719 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.719 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.719 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.719 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.719 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.719 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.719 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_stats_pools": { 2024-03-11T19:21:28.719 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_stats_pools", 2024-03-11T19:21:28.719 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.719 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.719 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.719 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.719 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.719 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.720 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.720 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.720 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.720 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.720 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.720 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.720 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-11T19:21:28.720 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-11T19:21:28.720 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.720 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.720 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.720 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "300", 2024-03-11T19:21:28.720 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.720 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.720 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.720 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.720 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.720 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.721 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.721 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.721 INFO:teuthology.orchestra.run.smithi029.stdout: "scrape_interval": { 2024-03-11T19:21:28.721 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "scrape_interval", 2024-03-11T19:21:28.721 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:28.721 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.721 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.721 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15.0", 2024-03-11T19:21:28.721 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.721 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.721 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.721 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.721 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.721 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.721 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.722 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.722 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-03-11T19:21:28.722 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-03-11T19:21:28.722 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.722 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.722 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.722 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "::", 2024-03-11T19:21:28.722 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.722 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.722 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.722 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-11T19:21:28.722 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.722 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.722 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.722 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.723 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-03-11T19:21:28.723 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-03-11T19:21:28.723 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.723 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.723 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.723 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "9283", 2024-03-11T19:21:28.723 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.723 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.723 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.723 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-11T19:21:28.723 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.723 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.723 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.723 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.723 INFO:teuthology.orchestra.run.smithi029.stdout: "stale_cache_strategy": { 2024-03-11T19:21:28.723 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "stale_cache_strategy", 2024-03-11T19:21:28.723 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.724 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.724 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.724 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "log", 2024-03-11T19:21:28.724 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.724 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.724 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.724 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.724 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.724 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.724 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.724 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.724 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_behaviour": { 2024-03-11T19:21:28.724 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_behaviour", 2024-03-11T19:21:28.724 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.724 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.724 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.725 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "default", 2024-03-11T19:21:28.725 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.725 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.725 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.725 INFO:teuthology.orchestra.run.smithi029.stdout: "default", 2024-03-11T19:21:28.725 INFO:teuthology.orchestra.run.smithi029.stdout: "error" 2024-03-11T19:21:28.725 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.725 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.725 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.725 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.725 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.725 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.725 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_error_status_code": { 2024-03-11T19:21:28.725 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_error_status_code", 2024-03-11T19:21:28.725 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.725 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.725 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-03-11T19:21:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "400", 2024-03-11T19:21:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "599", 2024-03-11T19:21:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_support", 2024-03-11T19:21:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.726 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.727 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.727 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.727 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.727 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.727 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.727 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.727 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.727 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.727 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.727 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.727 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.727 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.727 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.727 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.727 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.727 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.727 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.728 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.728 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.728 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.728 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.728 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.728 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.728 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.728 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.728 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.728 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.728 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.728 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.728 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.728 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.728 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.728 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.728 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.729 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.729 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.729 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.729 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.729 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.729 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.729 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.729 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.729 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.729 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.729 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.729 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.729 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.729 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.729 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.729 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.729 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.729 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.730 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.730 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.730 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.730 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.730 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.730 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.730 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.730 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.730 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.730 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.730 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.730 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.730 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.730 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.730 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.730 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.731 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.731 INFO:teuthology.orchestra.run.smithi029.stdout: "max_concurrent_snap_create": { 2024-03-11T19:21:28.731 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_concurrent_snap_create", 2024-03-11T19:21:28.731 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.731 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.731 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.731 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:28.731 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.731 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.731 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.731 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.731 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.731 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.731 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.731 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.732 INFO:teuthology.orchestra.run.smithi029.stdout: "mirror_snapshot_schedule": { 2024-03-11T19:21:28.732 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mirror_snapshot_schedule", 2024-03-11T19:21:28.732 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.732 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.732 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.732 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.732 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.732 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.732 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.732 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.732 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.732 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.732 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.732 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.732 INFO:teuthology.orchestra.run.smithi029.stdout: "trash_purge_schedule": { 2024-03-11T19:21:28.732 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "trash_purge_schedule", 2024-03-11T19:21:28.732 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.732 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.733 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.733 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.733 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.733 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.733 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.733 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.733 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.733 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.733 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.733 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.733 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.733 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.733 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.733 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "restful", 2024-03-11T19:21:28.734 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.734 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.734 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.734 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_auth": { 2024-03-11T19:21:28.734 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enable_auth", 2024-03-11T19:21:28.734 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.734 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.734 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.734 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.734 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.734 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.734 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.734 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.734 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.734 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.735 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.735 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.735 INFO:teuthology.orchestra.run.smithi029.stdout: "key_file": { 2024-03-11T19:21:28.735 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "key_file", 2024-03-11T19:21:28.735 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.735 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.735 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.735 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.735 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.735 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.735 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.735 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.735 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.735 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.736 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.736 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.736 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.736 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.736 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.736 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.736 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.736 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.736 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.736 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.736 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.736 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.736 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.737 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.737 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.737 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.737 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.737 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.737 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.737 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.737 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.737 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.737 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.737 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.737 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.737 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.737 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.737 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.737 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.737 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.738 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.738 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.738 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.738 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.738 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.738 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.738 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.738 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.738 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.738 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.738 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.738 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.739 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.739 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.739 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.739 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.739 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.739 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.739 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.739 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.739 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.739 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.739 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.739 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.739 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.739 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.739 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.739 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.739 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.740 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.740 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.740 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.740 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.740 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.740 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.740 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.740 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.740 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.740 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.740 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.740 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.740 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.740 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-03-11T19:21:28.740 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-03-11T19:21:28.740 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.741 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.741 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.741 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.741 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.741 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.741 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.741 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.741 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.741 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.741 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.741 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.741 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-03-11T19:21:28.741 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-03-11T19:21:28.741 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.741 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.741 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.742 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.742 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.742 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.742 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.742 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.742 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.742 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.742 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.742 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.742 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.742 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.742 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.742 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rgw", 2024-03-11T19:21:28.742 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.742 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.743 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.743 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.743 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.743 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.743 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.743 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.743 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.743 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.743 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.743 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.743 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.743 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.743 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.743 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.743 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.743 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.743 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.744 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.744 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.744 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.744 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.744 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.744 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.744 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.744 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.744 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.744 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.744 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.744 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.744 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.744 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.744 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.744 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.744 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.745 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.745 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.745 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.745 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.745 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.745 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.745 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.745 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.745 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.745 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.745 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.745 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.745 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.745 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.745 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.745 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.745 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.745 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.746 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.746 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.746 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.746 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.746 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.746 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.746 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.746 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.746 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.746 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.746 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.746 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.746 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.746 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.746 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.747 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.747 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.747 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.747 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.747 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.747 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.747 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.747 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rook", 2024-03-11T19:21:28.747 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.747 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.747 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.747 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.747 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.747 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.747 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.747 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.747 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.748 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.748 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.748 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.748 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.748 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.748 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.748 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.748 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.748 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.748 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.748 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.748 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.748 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.748 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.748 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.748 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.748 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.749 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.749 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.749 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.749 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.749 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.749 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.749 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.749 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.749 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.749 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.749 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.749 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.749 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.749 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.749 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.749 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.749 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.750 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.750 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.750 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.750 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.750 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.750 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.750 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.750 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.750 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.750 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.750 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.750 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.750 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.751 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.751 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.751 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.751 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.751 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.751 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.751 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.751 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.751 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.751 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.751 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.751 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.751 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.751 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.752 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.752 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.752 INFO:teuthology.orchestra.run.smithi029.stdout: "storage_class": { 2024-03-11T19:21:28.752 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "storage_class", 2024-03-11T19:21:28.752 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.752 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.752 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.752 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "local", 2024-03-11T19:21:28.752 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.752 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.752 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.752 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-11T19:21:28.752 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.752 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.752 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.752 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.753 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.753 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.753 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.753 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "selftest", 2024-03-11T19:21:28.753 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.753 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.753 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.753 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.753 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.753 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.753 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.753 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.753 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.753 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.753 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.753 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.753 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.754 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.754 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.754 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.754 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.754 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.754 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.754 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.754 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.754 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.754 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.754 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.754 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.754 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.754 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.755 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.755 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.755 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.755 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.755 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.755 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.755 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.755 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.755 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.755 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.755 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.755 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.755 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.755 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.755 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.756 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.756 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.756 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.756 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.756 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.756 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.756 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.756 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.756 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.756 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.756 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.756 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.756 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.756 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.756 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.757 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.757 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.757 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.757 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.757 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.757 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.757 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.757 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.757 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.757 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.757 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.757 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.757 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.757 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.757 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.758 INFO:teuthology.orchestra.run.smithi029.stdout: "roption1": { 2024-03-11T19:21:28.758 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "roption1", 2024-03-11T19:21:28.758 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.758 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.758 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.758 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.758 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.758 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.758 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.758 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.758 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.758 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.759 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.759 INFO:teuthology.orchestra.run.smithi029.stdout: "roption2": { 2024-03-11T19:21:28.759 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "roption2", 2024-03-11T19:21:28.759 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.759 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.759 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.759 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "xyz", 2024-03-11T19:21:28.759 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.759 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.759 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.759 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.759 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.759 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.760 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.760 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption1": { 2024-03-11T19:21:28.760 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption1", 2024-03-11T19:21:28.760 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.760 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.760 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.760 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.760 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.760 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.760 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.760 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.760 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.760 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.760 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.761 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.761 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption2": { 2024-03-11T19:21:28.761 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption2", 2024-03-11T19:21:28.761 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.761 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.761 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.761 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.761 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.761 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.761 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.761 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.761 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.761 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.761 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.761 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.762 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption3": { 2024-03-11T19:21:28.762 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption3", 2024-03-11T19:21:28.762 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:28.762 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.762 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.762 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.762 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.762 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.762 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.762 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.762 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.762 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.762 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.763 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.763 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption4": { 2024-03-11T19:21:28.763 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption4", 2024-03-11T19:21:28.763 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.763 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.763 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.763 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.763 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.763 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.763 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.763 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.763 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.764 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.764 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.764 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.764 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption5": { 2024-03-11T19:21:28.764 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption5", 2024-03-11T19:21:28.764 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.764 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.764 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.764 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.764 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.764 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.764 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.764 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.764 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.764 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.764 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.765 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.765 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption6": { 2024-03-11T19:21:28.765 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption6", 2024-03-11T19:21:28.765 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.765 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.765 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.765 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.765 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.765 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.765 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.765 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.765 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.765 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.765 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.765 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.765 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption7": { 2024-03-11T19:21:28.766 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption7", 2024-03-11T19:21:28.766 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.766 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.766 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.766 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.766 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-11T19:21:28.766 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "42", 2024-03-11T19:21:28.766 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.766 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.766 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.766 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.766 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.766 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.766 INFO:teuthology.orchestra.run.smithi029.stdout: "testkey": { 2024-03-11T19:21:28.767 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testkey", 2024-03-11T19:21:28.767 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.767 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.767 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.767 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.767 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.767 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.767 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.767 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.767 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.767 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.767 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.767 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.767 INFO:teuthology.orchestra.run.smithi029.stdout: "testlkey": { 2024-03-11T19:21:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testlkey", 2024-03-11T19:21:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "testnewline": { 2024-03-11T19:21:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testnewline", 2024-03-11T19:21:28.768 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.769 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snap_schedule", 2024-03-11T19:21:28.770 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.770 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.770 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.770 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_m_granularity": { 2024-03-11T19:21:28.770 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_m_granularity", 2024-03-11T19:21:28.770 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.770 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.770 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.770 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.770 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.770 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.770 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.770 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow minute scheduled snapshots", 2024-03-11T19:21:28.770 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.770 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.771 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.771 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.771 INFO:teuthology.orchestra.run.smithi029.stdout: "dump_on_update": { 2024-03-11T19:21:28.771 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "dump_on_update", 2024-03-11T19:21:28.771 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.771 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.771 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.771 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.771 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.771 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.771 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.771 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "dump database to debug log on update", 2024-03-11T19:21:28.771 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.771 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.771 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.771 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.772 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.772 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.772 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.772 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.772 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.772 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.772 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.772 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.772 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.772 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.772 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.772 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.772 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.772 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.772 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.773 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.773 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.773 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.773 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.773 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.773 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.773 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.773 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.773 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.773 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.773 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.773 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.773 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.773 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.773 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.774 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.774 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.774 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.774 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.774 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.774 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.774 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.774 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.774 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.774 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.774 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.774 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.774 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.775 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.775 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.775 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.775 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.775 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.775 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.775 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.775 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.775 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.775 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.775 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.775 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.775 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.775 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.775 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.775 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.776 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.776 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.776 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.776 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.776 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.776 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.776 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.776 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.776 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.776 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.776 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.776 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.776 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.776 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "stats", 2024-03-11T19:21:28.776 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.776 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.777 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.777 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.777 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.777 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.777 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.777 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.777 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.777 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.777 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.777 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.777 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.777 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.777 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.777 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.777 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.777 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.777 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.777 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.778 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.778 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.778 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.778 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.778 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.778 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.778 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.778 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.778 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.778 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.778 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.778 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.778 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.778 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.778 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.779 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.779 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.779 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.779 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.779 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.779 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.779 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.779 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.779 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.779 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.779 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.779 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.779 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.779 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.779 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.779 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.779 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.780 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.780 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.780 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.780 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.780 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.780 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.780 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.780 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.780 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.780 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.780 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.780 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.780 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.780 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.780 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.780 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.781 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.781 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.781 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.781 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.781 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.781 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.781 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.781 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.781 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "status", 2024-03-11T19:21:28.781 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.781 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.781 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.782 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.782 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.782 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.782 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.782 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.782 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.782 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.782 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.782 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.782 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.782 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.782 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.782 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.782 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.782 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.783 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.783 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.783 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.783 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.783 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.783 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.783 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.783 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.783 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.783 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.783 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.783 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.783 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.783 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.784 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.784 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.784 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.784 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.784 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.784 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.784 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.784 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.784 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.784 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.784 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.784 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.784 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.784 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.784 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.784 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.785 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.785 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.785 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.785 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.785 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.785 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.785 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.785 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.785 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.785 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.785 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.785 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.785 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.785 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.785 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.785 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.786 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.786 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.786 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.786 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.786 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.786 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.786 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.786 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.786 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.786 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.786 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.786 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.786 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "telegraf", 2024-03-11T19:21:28.786 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.786 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.787 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.787 INFO:teuthology.orchestra.run.smithi029.stdout: "address": { 2024-03-11T19:21:28.787 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "address", 2024-03-11T19:21:28.787 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.787 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.787 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.787 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-11T19:21:28.787 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.787 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.787 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.787 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.787 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.787 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.787 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.788 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.788 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-11T19:21:28.788 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-11T19:21:28.788 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.788 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.788 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.788 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15", 2024-03-11T19:21:28.788 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.788 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.788 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.788 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.788 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.788 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.788 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.788 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.789 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.789 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.789 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.789 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.789 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.789 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.789 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.789 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.789 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.789 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.789 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.789 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.789 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.790 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.790 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.790 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.790 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.790 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.790 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.790 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.790 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.790 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.790 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.790 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.790 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.790 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.790 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.791 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.791 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.791 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.791 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.791 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.791 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.791 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.791 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.791 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.791 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.791 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.791 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.791 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.791 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.792 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.792 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.792 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.792 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.792 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.792 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.792 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.792 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.792 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.792 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.792 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.792 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.792 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.792 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.792 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.793 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.793 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.793 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.793 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.793 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.793 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.793 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.793 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.793 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.793 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.793 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.793 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.793 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.793 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.794 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.794 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.794 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.794 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "telemetry", 2024-03-11T19:21:28.794 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.794 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.794 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.794 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_basic": { 2024-03-11T19:21:28.794 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_basic", 2024-03-11T19:21:28.794 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.794 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.794 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.794 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.795 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.795 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.795 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.795 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-11T19:21:28.795 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.795 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.795 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.795 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.795 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_crash": { 2024-03-11T19:21:28.795 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_crash", 2024-03-11T19:21:28.795 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.795 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.795 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.795 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.795 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.796 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.796 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.796 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-11T19:21:28.796 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.796 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.796 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.796 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.796 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_device": { 2024-03-11T19:21:28.796 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_device", 2024-03-11T19:21:28.796 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.796 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.796 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.796 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.796 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.796 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.796 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.796 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-11T19:21:28.797 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.797 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.797 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.797 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.797 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_ident": { 2024-03-11T19:21:28.797 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_ident", 2024-03-11T19:21:28.797 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.797 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.797 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.797 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.797 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.797 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.797 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.797 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-11T19:21:28.797 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.798 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.798 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.798 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.798 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_perf": { 2024-03-11T19:21:28.798 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_perf", 2024-03-11T19:21:28.798 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.798 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.798 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.798 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.798 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.798 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.798 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.798 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-11T19:21:28.798 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.798 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.798 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.799 INFO:teuthology.orchestra.run.smithi029.stdout: "contact": { 2024-03-11T19:21:28.799 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "contact", 2024-03-11T19:21:28.799 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.799 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.799 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.799 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.799 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.799 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.799 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.799 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.799 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.799 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.799 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.800 INFO:teuthology.orchestra.run.smithi029.stdout: "description": { 2024-03-11T19:21:28.800 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "description", 2024-03-11T19:21:28.800 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.800 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.800 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.800 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.800 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.800 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.800 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.800 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.800 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.800 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.800 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.800 INFO:teuthology.orchestra.run.smithi029.stdout: "device_url": { 2024-03-11T19:21:28.800 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_url", 2024-03-11T19:21:28.800 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.801 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.801 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.801 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-11T19:21:28.801 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.801 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.801 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.801 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.801 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.801 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.801 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.801 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": { 2024-03-11T19:21:28.801 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enabled", 2024-03-11T19:21:28.801 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.801 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.801 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.802 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.802 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.802 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.802 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.802 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.802 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.802 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.802 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.802 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-11T19:21:28.802 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-11T19:21:28.802 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.802 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.802 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.803 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "24", 2024-03-11T19:21:28.803 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "8", 2024-03-11T19:21:28.803 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.803 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.803 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.803 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.803 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.803 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.803 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.803 INFO:teuthology.orchestra.run.smithi029.stdout: "last_opt_revision": { 2024-03-11T19:21:28.803 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "last_opt_revision", 2024-03-11T19:21:28.803 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.803 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.803 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.803 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1", 2024-03-11T19:21:28.803 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.804 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.804 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.804 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.804 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.804 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.804 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.804 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.804 INFO:teuthology.orchestra.run.smithi029.stdout: "leaderboard": { 2024-03-11T19:21:28.804 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "leaderboard", 2024-03-11T19:21:28.804 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.804 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.804 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.804 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.804 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.804 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.805 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.805 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.805 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.805 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.805 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.805 INFO:teuthology.orchestra.run.smithi029.stdout: "leaderboard_description": { 2024-03-11T19:21:28.805 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "leaderboard_description", 2024-03-11T19:21:28.805 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.805 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.805 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.805 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.805 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.805 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.805 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.805 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.805 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.806 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.806 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.806 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.806 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.806 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.806 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.806 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.806 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.806 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.806 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.806 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.806 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.806 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.806 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.807 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.807 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.807 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.807 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.807 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.807 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.807 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.807 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.807 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.807 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.807 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.807 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.807 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.807 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.807 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.807 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.808 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.808 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.808 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.808 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.808 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.808 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.808 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.808 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.808 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.808 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.808 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.808 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.808 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.808 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.808 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.808 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.809 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.809 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.809 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.809 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.809 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.809 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.809 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.809 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.809 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.809 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.809 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.809 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.809 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.809 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.809 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.809 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.810 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.810 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.810 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.810 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.810 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.810 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.810 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.810 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.810 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.810 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.810 INFO:teuthology.orchestra.run.smithi029.stdout: "organization": { 2024-03-11T19:21:28.810 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "organization", 2024-03-11T19:21:28.810 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.810 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.810 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.811 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.811 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.811 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.811 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.811 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.811 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.811 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.811 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.811 INFO:teuthology.orchestra.run.smithi029.stdout: "proxy": { 2024-03-11T19:21:28.811 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "proxy", 2024-03-11T19:21:28.811 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.811 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.811 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.811 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.812 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.812 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.812 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.812 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.812 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.812 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.812 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.812 INFO:teuthology.orchestra.run.smithi029.stdout: "url": { 2024-03-11T19:21:28.812 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "url", 2024-03-11T19:21:28.812 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.812 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.812 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.812 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-11T19:21:28.813 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.813 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.813 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.813 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.813 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.813 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.813 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.813 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.813 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.813 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.813 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.813 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "test_orchestrator", 2024-03-11T19:21:28.813 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.813 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.813 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.814 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.814 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.814 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.814 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.814 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.814 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.814 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.814 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.814 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.814 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.814 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.814 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.814 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.815 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.815 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.815 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.815 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.815 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.815 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.815 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.815 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.815 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.815 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.815 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.815 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.815 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.815 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.815 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.816 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.816 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.816 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.816 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.816 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.816 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.816 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.816 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.816 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.816 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.816 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.816 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.816 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.817 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.817 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.817 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.817 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.817 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.817 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.817 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.817 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.817 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.817 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.817 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.817 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.817 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.817 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.818 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.818 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.818 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.818 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.818 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.818 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.818 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.818 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.818 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.818 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.818 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.818 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.818 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.819 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "volumes", 2024-03-11T19:21:28.819 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.819 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.819 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.819 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.819 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.819 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.819 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.819 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.819 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.819 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.819 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.819 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.819 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.819 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.819 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.820 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.820 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.820 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.820 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.820 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.820 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.820 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.820 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.820 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.820 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.820 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.820 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.820 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.820 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.820 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.821 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.821 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.821 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.821 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.821 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.821 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.821 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.821 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.821 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.821 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.821 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.821 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.821 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.821 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.821 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.822 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.822 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.822 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.822 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.822 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.822 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.822 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.822 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.822 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.822 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.822 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.822 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.822 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.822 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.822 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.823 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.823 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.823 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.823 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.823 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.823 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.823 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.823 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.823 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.823 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.823 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.823 INFO:teuthology.orchestra.run.smithi029.stdout: "max_concurrent_clones": { 2024-03-11T19:21:28.823 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_concurrent_clones", 2024-03-11T19:21:28.823 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.823 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.824 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.824 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "4", 2024-03-11T19:21:28.824 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.824 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.824 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.824 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-11T19:21:28.824 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.824 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.824 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.824 INFO:teuthology.orchestra.run.smithi029.stdout: "periodic_async_work": { 2024-03-11T19:21:28.824 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "periodic_async_work", 2024-03-11T19:21:28.824 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.824 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.824 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.824 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.824 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.825 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.825 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.825 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Periodically check for async work", 2024-03-11T19:21:28.825 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.825 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.825 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.825 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.825 INFO:teuthology.orchestra.run.smithi029.stdout: "snapshot_clone_delay": { 2024-03-11T19:21:28.825 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snapshot_clone_delay", 2024-03-11T19:21:28.825 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.825 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.825 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.825 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-11T19:21:28.825 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.825 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.825 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.826 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-11T19:21:28.826 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.826 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.826 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.826 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.826 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.826 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.826 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.826 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix", 2024-03-11T19:21:28.826 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.826 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.826 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.826 INFO:teuthology.orchestra.run.smithi029.stdout: "discovery_interval": { 2024-03-11T19:21:28.826 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "discovery_interval", 2024-03-11T19:21:28.826 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-11T19:21:28.826 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.827 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.827 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "100", 2024-03-11T19:21:28.827 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.827 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.827 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.827 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.827 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.827 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.827 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.827 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.827 INFO:teuthology.orchestra.run.smithi029.stdout: "identifier": { 2024-03-11T19:21:28.827 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "identifier", 2024-03-11T19:21:28.827 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.827 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.827 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.827 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.828 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.828 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.828 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.828 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.828 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.828 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.828 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.828 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-11T19:21:28.828 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-11T19:21:28.828 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.828 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.828 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.828 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-11T19:21:28.828 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.828 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.828 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.829 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.829 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.829 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.829 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.829 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.829 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.829 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.829 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.829 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.829 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.829 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.829 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.829 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.830 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.830 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.830 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.830 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.830 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.830 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.830 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.830 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.830 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.830 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.830 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.830 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.830 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.830 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.831 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.831 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.831 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.831 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.831 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.831 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.831 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.831 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.831 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.831 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.831 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.831 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.831 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.831 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.832 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.832 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.832 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.832 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.832 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.832 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.832 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.832 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.832 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.832 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.832 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.832 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.832 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.832 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.832 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.832 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.833 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.833 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.833 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.833 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.833 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.833 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.833 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.833 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.833 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.833 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.833 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.833 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.833 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.833 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.833 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.833 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_host": { 2024-03-11T19:21:28.833 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_host", 2024-03-11T19:21:28.834 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.834 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.834 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.834 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.834 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.834 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.834 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.834 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.834 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.834 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.834 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.834 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.834 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_port": { 2024-03-11T19:21:28.834 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_port", 2024-03-11T19:21:28.834 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.835 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.835 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.835 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10051", 2024-03-11T19:21:28.835 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.835 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.835 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.835 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.835 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.835 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.835 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.835 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.835 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_sender": { 2024-03-11T19:21:28.835 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_sender", 2024-03-11T19:21:28.836 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.836 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.836 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.836 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-11T19:21:28.836 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.836 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.836 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.836 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.836 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.836 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.836 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.836 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.836 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.836 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.836 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-11T19:21:28.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.837 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.837 INFO:teuthology.orchestra.run.smithi029.stdout: "gid": 24121, 2024-03-11T19:21:28.837 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "y", 2024-03-11T19:21:28.837 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_features": 4540701547738038271, 2024-03-11T19:21:28.837 INFO:teuthology.orchestra.run.smithi029.stdout: "available_modules": [ 2024-03-11T19:21:28.837 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.837 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "alerts", 2024-03-11T19:21:28.837 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.837 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.837 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.837 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-11T19:21:28.837 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-11T19:21:28.837 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.837 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.837 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.837 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-11T19:21:28.837 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.838 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.838 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.838 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How frequently to reexamine health status", 2024-03-11T19:21:28.838 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.838 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.838 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.838 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.838 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.838 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.838 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.838 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.838 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.838 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.839 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.839 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.839 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.839 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.839 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.839 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.839 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.839 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.839 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.839 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.839 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.839 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.839 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.839 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.839 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.839 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.840 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.840 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.840 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.840 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.840 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.840 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.840 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.840 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.840 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.840 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.840 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.840 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.840 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.841 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.841 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.841 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.841 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.841 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.841 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.841 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.841 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.841 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.841 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.841 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.841 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.842 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.842 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.842 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.842 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.842 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.842 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.842 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.842 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.842 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.842 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.842 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.842 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.842 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.842 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.843 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.843 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.843 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.843 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.843 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.843 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.843 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.843 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.843 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_destination": { 2024-03-11T19:21:28.843 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_destination", 2024-03-11T19:21:28.843 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.843 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.843 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Email address to send alerts to", 2024-03-11T19:21:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_from_name": { 2024-03-11T19:21:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_from_name", 2024-03-11T19:21:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.844 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.845 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.845 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "Ceph", 2024-03-11T19:21:28.845 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.845 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.845 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.845 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Email From: name", 2024-03-11T19:21:28.845 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.845 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.845 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.845 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.845 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_host": { 2024-03-11T19:21:28.845 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_host", 2024-03-11T19:21:28.845 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.845 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.846 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.846 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.846 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.846 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.846 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.846 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP server", 2024-03-11T19:21:28.846 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.846 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.846 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.846 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_password": { 2024-03-11T19:21:28.846 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_password", 2024-03-11T19:21:28.847 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.847 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.847 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.847 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.847 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.847 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.847 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.847 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Password to authenticate with", 2024-03-11T19:21:28.847 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.847 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.847 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.847 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.847 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_port": { 2024-03-11T19:21:28.847 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_port", 2024-03-11T19:21:28.847 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.848 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.848 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.848 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "465", 2024-03-11T19:21:28.848 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.848 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.848 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.848 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP port", 2024-03-11T19:21:28.848 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.848 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.848 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.848 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.848 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_sender": { 2024-03-11T19:21:28.848 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_sender", 2024-03-11T19:21:28.848 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.849 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.849 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.849 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.849 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.849 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.849 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.849 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP envelope sender", 2024-03-11T19:21:28.849 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.849 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.849 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.849 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.849 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_ssl": { 2024-03-11T19:21:28.849 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_ssl", 2024-03-11T19:21:28.849 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.849 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.850 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.850 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.850 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.850 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.850 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.850 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-11T19:21:28.850 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.850 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.850 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.850 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.850 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_user": { 2024-03-11T19:21:28.850 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_user", 2024-03-11T19:21:28.850 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.850 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.851 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.851 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.851 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.851 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.851 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.851 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "User to authenticate as", 2024-03-11T19:21:28.851 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.851 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.851 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.851 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.851 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.851 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.851 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.851 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "balancer", 2024-03-11T19:21:28.852 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.852 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.852 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.852 INFO:teuthology.orchestra.run.smithi029.stdout: "active": { 2024-03-11T19:21:28.852 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "active", 2024-03-11T19:21:28.852 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.852 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.852 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.852 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.852 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.852 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.852 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.852 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically balance PGs across cluster", 2024-03-11T19:21:28.852 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.852 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.852 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.853 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.853 INFO:teuthology.orchestra.run.smithi029.stdout: "begin_time": { 2024-03-11T19:21:28.853 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "begin_time", 2024-03-11T19:21:28.853 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.853 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.853 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.853 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0000", 2024-03-11T19:21:28.853 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.853 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.853 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.853 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "beginning time of day to automatically balance", 2024-03-11T19:21:28.853 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-11T19:21:28.853 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.853 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.854 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.854 INFO:teuthology.orchestra.run.smithi029.stdout: "begin_weekday": { 2024-03-11T19:21:28.854 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "begin_weekday", 2024-03-11T19:21:28.854 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-11T19:21:28.854 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.854 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.854 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-11T19:21:28.854 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0", 2024-03-11T19:21:28.854 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "6", 2024-03-11T19:21:28.854 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.854 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-11T19:21:28.854 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-11T19:21:28.854 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.854 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.854 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.855 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_max_iterations": { 2024-03-11T19:21:28.855 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_max_iterations", 2024-03-11T19:21:28.855 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-11T19:21:28.855 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.855 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.855 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "25", 2024-03-11T19:21:28.855 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-11T19:21:28.855 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "250", 2024-03-11T19:21:28.855 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.855 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-11T19:21:28.855 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.855 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.855 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.855 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.855 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_metrics": { 2024-03-11T19:21:28.855 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_metrics", 2024-03-11T19:21:28.856 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.856 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.856 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.856 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "pgs,objects,bytes", 2024-03-11T19:21:28.856 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.856 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.856 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.856 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-11T19:21:28.856 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-11T19:21:28.856 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.856 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.856 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.856 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_step": { 2024-03-11T19:21:28.856 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_step", 2024-03-11T19:21:28.856 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:28.857 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.857 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.857 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0.5", 2024-03-11T19:21:28.857 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0.001", 2024-03-11T19:21:28.857 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "0.999", 2024-03-11T19:21:28.857 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.857 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "aggressiveness of optimization", 2024-03-11T19:21:28.857 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-11T19:21:28.857 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.857 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.857 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.857 INFO:teuthology.orchestra.run.smithi029.stdout: "end_time": { 2024-03-11T19:21:28.857 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "end_time", 2024-03-11T19:21:28.857 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.857 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.858 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.858 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "2359", 2024-03-11T19:21:28.858 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.858 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.858 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.858 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "ending time of day to automatically balance", 2024-03-11T19:21:28.858 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-11T19:21:28.858 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.858 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.858 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.858 INFO:teuthology.orchestra.run.smithi029.stdout: "end_weekday": { 2024-03-11T19:21:28.858 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "end_weekday", 2024-03-11T19:21:28.858 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-11T19:21:28.858 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.858 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.859 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-11T19:21:28.859 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0", 2024-03-11T19:21:28.859 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "6", 2024-03-11T19:21:28.859 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.859 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-11T19:21:28.859 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-11T19:21:28.859 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.859 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.859 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.859 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.859 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.859 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.859 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.859 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.859 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.859 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.859 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.860 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.860 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.860 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.860 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.860 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.860 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.860 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.860 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.860 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.860 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.860 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.860 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.860 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.860 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.860 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.861 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.861 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.861 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.861 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.861 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.861 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.861 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.861 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.861 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.861 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.861 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.861 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.861 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.861 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.861 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.862 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.862 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.862 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.862 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.862 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.862 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.862 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.862 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.862 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.862 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.862 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.862 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.862 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.862 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.863 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.863 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.863 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.863 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.863 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.863 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.863 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.863 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.863 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.863 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.863 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.863 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.863 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.863 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.863 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.864 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.864 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.864 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.864 INFO:teuthology.orchestra.run.smithi029.stdout: "min_score": { 2024-03-11T19:21:28.864 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "min_score", 2024-03-11T19:21:28.864 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:28.864 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.864 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.864 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-11T19:21:28.864 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.864 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.864 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.864 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-11T19:21:28.864 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.864 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.865 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.865 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.865 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": { 2024-03-11T19:21:28.865 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mode", 2024-03-11T19:21:28.865 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.865 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.865 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.865 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "upmap", 2024-03-11T19:21:28.865 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.865 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.865 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.865 INFO:teuthology.orchestra.run.smithi029.stdout: "crush-compat", 2024-03-11T19:21:28.865 INFO:teuthology.orchestra.run.smithi029.stdout: "none", 2024-03-11T19:21:28.865 INFO:teuthology.orchestra.run.smithi029.stdout: "read", 2024-03-11T19:21:28.866 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap", 2024-03-11T19:21:28.866 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap-read" 2024-03-11T19:21:28.866 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.866 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Balancer mode", 2024-03-11T19:21:28.866 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.866 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.866 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.866 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.866 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_ids": { 2024-03-11T19:21:28.866 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pool_ids", 2024-03-11T19:21:28.866 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.866 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.866 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.866 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.867 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.867 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.867 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.867 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-11T19:21:28.867 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.867 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.867 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.867 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.867 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-11T19:21:28.867 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-11T19:21:28.867 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.867 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.867 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.867 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-11T19:21:28.867 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.868 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.868 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.868 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-11T19:21:28.868 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.868 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.868 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.868 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.868 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap_max_deviation": { 2024-03-11T19:21:28.868 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "upmap_max_deviation", 2024-03-11T19:21:28.868 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.868 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.868 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.868 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-11T19:21:28.868 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-11T19:21:28.869 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.869 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.869 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-11T19:21:28.869 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-11T19:21:28.869 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.869 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.869 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.869 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap_max_optimizations": { 2024-03-11T19:21:28.869 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "upmap_max_optimizations", 2024-03-11T19:21:28.869 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-11T19:21:28.869 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.869 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.869 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:28.869 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.869 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.870 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.870 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-11T19:21:28.870 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.870 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.870 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.870 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.870 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.870 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.870 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.870 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephadm", 2024-03-11T19:21:28.870 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.870 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.870 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.870 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_down_multiplier": { 2024-03-11T19:21:28.871 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_down_multiplier", 2024-03-11T19:21:28.871 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:28.871 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.871 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.871 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3.0", 2024-03-11T19:21:28.871 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.871 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.871 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.871 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-11T19:21:28.871 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.871 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.871 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.871 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.871 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_refresh_rate": { 2024-03-11T19:21:28.871 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_refresh_rate", 2024-03-11T19:21:28.872 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.872 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.872 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.872 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "20", 2024-03-11T19:21:28.874 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.874 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.874 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.874 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-11T19:21:28.874 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.874 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.874 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.874 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_starting_port": { 2024-03-11T19:21:28.875 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_starting_port", 2024-03-11T19:21:28.875 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.875 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.875 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.875 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "4721", 2024-03-11T19:21:28.875 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.875 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.875 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.875 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-11T19:21:28.875 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.875 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.875 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.875 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_ptrace": { 2024-03-11T19:21:28.875 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_ptrace", 2024-03-11T19:21:28.875 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.875 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.876 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.876 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.876 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.876 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.876 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.876 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-11T19:21:28.876 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-11T19:21:28.876 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.876 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.876 INFO:teuthology.orchestra.run.smithi029.stdout: "autotune_interval": { 2024-03-11T19:21:28.876 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "autotune_interval", 2024-03-11T19:21:28.876 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.876 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.876 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.877 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-11T19:21:28.877 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.877 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.877 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.877 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-11T19:21:28.877 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.877 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.877 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.877 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.877 INFO:teuthology.orchestra.run.smithi029.stdout: "autotune_memory_target_ratio": { 2024-03-11T19:21:28.877 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "autotune_memory_target_ratio", 2024-03-11T19:21:28.877 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:28.877 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.877 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.877 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0.7", 2024-03-11T19:21:28.877 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.877 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.878 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.878 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-11T19:21:28.878 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.878 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.878 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.878 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.878 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm_log_destination": { 2024-03-11T19:21:28.878 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephadm_log_destination", 2024-03-11T19:21:28.878 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.878 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.878 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: "file", 2024-03-11T19:21:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: "file,syslog", 2024-03-11T19:21:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: "syslog" 2024-03-11T19:21:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-11T19:21:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: "cgroups_split": { 2024-03-11T19:21:28.880 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cgroups_split", 2024-03-11T19:21:28.880 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.880 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.880 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.880 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.880 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.880 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.880 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.880 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-11T19:21:28.880 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.880 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.880 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.880 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.880 INFO:teuthology.orchestra.run.smithi029.stdout: "config_checks_enabled": { 2024-03-11T19:21:28.880 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_checks_enabled", 2024-03-11T19:21:28.880 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.881 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.881 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.881 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.881 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.881 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.881 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.881 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-11T19:21:28.881 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.881 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.881 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.881 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.881 INFO:teuthology.orchestra.run.smithi029.stdout: "config_dashboard": { 2024-03-11T19:21:28.881 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_dashboard", 2024-03-11T19:21:28.881 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.881 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.881 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.882 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.882 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.882 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.882 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.882 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-11T19:21:28.882 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.882 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.882 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.882 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.882 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_alertmanager": { 2024-03-11T19:21:28.882 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_alertmanager", 2024-03-11T19:21:28.882 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.882 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.882 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.883 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-11T19:21:28.883 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.883 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.883 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.883 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-11T19:21:28.883 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.883 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.883 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.883 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.883 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_base": { 2024-03-11T19:21:28.883 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_base", 2024-03-11T19:21:28.883 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.883 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.883 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.883 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-11T19:21:28.884 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.884 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.884 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.884 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Container image name, without the tag", 2024-03-11T19:21:28.884 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.884 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.884 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.884 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.884 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_elasticsearch": { 2024-03-11T19:21:28.884 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_elasticsearch", 2024-03-11T19:21:28.884 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.884 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.884 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.884 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-11T19:21:28.884 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.884 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.885 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.885 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "elasticsearch container image", 2024-03-11T19:21:28.885 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.885 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.885 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.885 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.885 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_grafana": { 2024-03-11T19:21:28.885 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_grafana", 2024-03-11T19:21:28.885 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.885 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.885 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.885 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-11T19:21:28.885 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.885 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.885 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.886 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-11T19:21:28.886 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.886 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.886 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.886 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.886 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_haproxy": { 2024-03-11T19:21:28.886 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_haproxy", 2024-03-11T19:21:28.886 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.886 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.886 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.886 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-11T19:21:28.886 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.886 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.886 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.886 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "HAproxy container image", 2024-03-11T19:21:28.886 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.887 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.887 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.887 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.887 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_agent": { 2024-03-11T19:21:28.887 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_agent", 2024-03-11T19:21:28.887 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.887 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.887 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.887 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-11T19:21:28.887 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.887 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.887 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.887 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger agent container image", 2024-03-11T19:21:28.887 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.887 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.887 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.887 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.888 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_collector": { 2024-03-11T19:21:28.888 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_collector", 2024-03-11T19:21:28.888 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.888 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.888 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.888 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-11T19:21:28.888 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.888 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.888 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.888 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger collector container image", 2024-03-11T19:21:28.888 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.888 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.888 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.888 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.888 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_query": { 2024-03-11T19:21:28.889 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_query", 2024-03-11T19:21:28.889 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.889 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.889 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.889 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-11T19:21:28.889 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.889 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.889 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.889 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger query container image", 2024-03-11T19:21:28.889 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.889 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.889 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.889 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.889 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_keepalived": { 2024-03-11T19:21:28.889 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_keepalived", 2024-03-11T19:21:28.889 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.890 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.890 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.890 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-11T19:21:28.890 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.890 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.890 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.890 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Keepalived container image", 2024-03-11T19:21:28.890 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.890 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.890 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.890 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.890 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_loki": { 2024-03-11T19:21:28.890 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_loki", 2024-03-11T19:21:28.890 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.891 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.891 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.891 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-11T19:21:28.891 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.891 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.891 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.891 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Loki container image", 2024-03-11T19:21:28.891 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.891 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.891 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.891 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.891 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_node_exporter": { 2024-03-11T19:21:28.891 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_node_exporter", 2024-03-11T19:21:28.891 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.891 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.892 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.892 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-11T19:21:28.892 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.892 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.892 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.892 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-11T19:21:28.892 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.892 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.892 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.892 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.892 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_nvmeof": { 2024-03-11T19:21:28.892 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_nvmeof", 2024-03-11T19:21:28.892 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.892 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.892 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.893 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-11T19:21:28.893 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.893 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.893 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.893 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Nvme-of container image", 2024-03-11T19:21:28.893 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.893 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.893 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.893 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.893 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_prometheus": { 2024-03-11T19:21:28.893 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_prometheus", 2024-03-11T19:21:28.893 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.893 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.893 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.893 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-11T19:21:28.893 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.893 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.893 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.894 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-11T19:21:28.894 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.894 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.894 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.894 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.894 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_promtail": { 2024-03-11T19:21:28.894 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_promtail", 2024-03-11T19:21:28.894 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.894 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.894 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.894 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-11T19:21:28.894 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.894 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.894 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.894 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Promtail container image", 2024-03-11T19:21:28.895 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.895 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.895 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.895 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.895 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_snmp_gateway": { 2024-03-11T19:21:28.895 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_snmp_gateway", 2024-03-11T19:21:28.895 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.895 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.895 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.895 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-11T19:21:28.895 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.895 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.895 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.895 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SNMP Gateway container image", 2024-03-11T19:21:28.895 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.895 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.896 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.896 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.896 INFO:teuthology.orchestra.run.smithi029.stdout: "container_init": { 2024-03-11T19:21:28.896 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_init", 2024-03-11T19:21:28.896 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.896 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.896 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.896 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.896 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.896 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.896 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.896 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Run podman/docker with `--init`", 2024-03-11T19:21:28.896 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.896 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.896 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.896 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.897 INFO:teuthology.orchestra.run.smithi029.stdout: "daemon_cache_timeout": { 2024-03-11T19:21:28.897 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "daemon_cache_timeout", 2024-03-11T19:21:28.897 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.897 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.897 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.897 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-11T19:21:28.897 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.897 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.897 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.897 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-11T19:21:28.897 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.897 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.897 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.897 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.897 INFO:teuthology.orchestra.run.smithi029.stdout: "default_cephadm_command_timeout": { 2024-03-11T19:21:28.897 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default_cephadm_command_timeout", 2024-03-11T19:21:28.898 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.898 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.898 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.898 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "900", 2024-03-11T19:21:28.898 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.898 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.898 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.898 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-11T19:21:28.898 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.898 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.898 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.898 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.898 INFO:teuthology.orchestra.run.smithi029.stdout: "default_registry": { 2024-03-11T19:21:28.898 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default_registry", 2024-03-11T19:21:28.898 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.898 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.899 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.899 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io", 2024-03-11T19:21:28.899 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.899 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.899 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.899 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-11T19:21:28.899 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.899 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.899 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.899 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.899 INFO:teuthology.orchestra.run.smithi029.stdout: "device_cache_timeout": { 2024-03-11T19:21:28.899 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_cache_timeout", 2024-03-11T19:21:28.899 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.899 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.899 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.899 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1800", 2024-03-11T19:21:28.900 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.900 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.900 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.900 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache device inventory", 2024-03-11T19:21:28.900 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.900 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.900 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.900 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.900 INFO:teuthology.orchestra.run.smithi029.stdout: "device_enhanced_scan": { 2024-03-11T19:21:28.900 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_enhanced_scan", 2024-03-11T19:21:28.900 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.900 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.900 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.900 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.900 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.900 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.901 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.901 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-11T19:21:28.901 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.901 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.901 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.901 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.901 INFO:teuthology.orchestra.run.smithi029.stdout: "facts_cache_timeout": { 2024-03-11T19:21:28.901 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "facts_cache_timeout", 2024-03-11T19:21:28.901 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.901 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.901 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.901 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-11T19:21:28.901 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.901 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.901 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.902 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache host facts data", 2024-03-11T19:21:28.902 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.902 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.902 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.902 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.902 INFO:teuthology.orchestra.run.smithi029.stdout: "host_check_interval": { 2024-03-11T19:21:28.902 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "host_check_interval", 2024-03-11T19:21:28.902 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.902 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.902 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.902 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-11T19:21:28.902 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.902 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.902 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.902 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to perform a host check", 2024-03-11T19:21:28.902 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.903 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.903 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.903 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.903 INFO:teuthology.orchestra.run.smithi029.stdout: "hw_monitoring": { 2024-03-11T19:21:28.903 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "hw_monitoring", 2024-03-11T19:21:28.903 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.903 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.903 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.903 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.903 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.903 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.903 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.903 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-11T19:21:28.903 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.903 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.904 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.904 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.904 INFO:teuthology.orchestra.run.smithi029.stdout: "inventory_list_all": { 2024-03-11T19:21:28.904 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "inventory_list_all", 2024-03-11T19:21:28.904 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.904 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.904 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.904 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.904 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.904 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.904 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.904 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-11T19:21:28.904 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.904 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.904 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.904 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.904 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.905 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.905 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.905 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.905 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.905 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.905 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.905 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.905 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.905 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.905 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.905 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.905 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.905 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.905 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.905 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.905 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.905 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.906 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.906 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.906 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.906 INFO:teuthology.orchestra.run.smithi029.stdout: "log_refresh_metadata": { 2024-03-11T19:21:28.906 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_refresh_metadata", 2024-03-11T19:21:28.906 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.906 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.906 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.906 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.906 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.906 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.906 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.906 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-11T19:21:28.906 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.906 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.906 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.907 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.907 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.907 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.907 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.907 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.907 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.907 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.907 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.907 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.907 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.907 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-11T19:21:28.907 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.907 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.907 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.907 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.907 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.907 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.908 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.908 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.908 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.908 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.908 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.908 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.908 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.908 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.908 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.908 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.908 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.908 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.908 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.908 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.908 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.908 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.908 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.909 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.909 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.909 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.909 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.909 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.909 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.909 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.909 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.909 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.909 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.909 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.909 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.909 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.909 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.909 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.909 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.910 INFO:teuthology.orchestra.run.smithi029.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-11T19:21:28.910 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-11T19:21:28.910 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.910 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.910 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.910 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.910 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.910 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.910 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.910 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-11T19:21:28.910 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.910 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.910 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.910 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.910 INFO:teuthology.orchestra.run.smithi029.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-11T19:21:28.911 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-11T19:21:28.911 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.911 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.911 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.911 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "*", 2024-03-11T19:21:28.911 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.911 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.911 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.911 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-11T19:21:28.911 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.911 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.911 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.911 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.911 INFO:teuthology.orchestra.run.smithi029.stdout: "max_count_per_host": { 2024-03-11T19:21:28.911 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_count_per_host", 2024-03-11T19:21:28.912 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.912 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.912 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.912 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:28.912 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.912 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.912 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.912 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "max number of daemons per service per host", 2024-03-11T19:21:28.912 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.912 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.912 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.912 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.912 INFO:teuthology.orchestra.run.smithi029.stdout: "max_osd_draining_count": { 2024-03-11T19:21:28.912 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_osd_draining_count", 2024-03-11T19:21:28.912 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.913 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.913 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.913 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:28.913 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.913 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.913 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.913 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-11T19:21:28.913 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.913 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.913 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.913 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.913 INFO:teuthology.orchestra.run.smithi029.stdout: "migration_current": { 2024-03-11T19:21:28.913 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "migration_current", 2024-03-11T19:21:28.913 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.913 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.913 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.914 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.914 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.914 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.914 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.914 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "internal - do not modify", 2024-03-11T19:21:28.914 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.914 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.914 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.914 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.914 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": { 2024-03-11T19:21:28.914 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mode", 2024-03-11T19:21:28.914 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.914 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.915 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.915 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "root", 2024-03-11T19:21:28.915 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.915 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.915 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.915 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm-package", 2024-03-11T19:21:28.915 INFO:teuthology.orchestra.run.smithi029.stdout: "root" 2024-03-11T19:21:28.915 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.915 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "mode for remote execution of cephadm", 2024-03-11T19:21:28.915 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.915 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.915 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.915 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.915 INFO:teuthology.orchestra.run.smithi029.stdout: "oob_default_addr": { 2024-03-11T19:21:28.915 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "oob_default_addr", 2024-03-11T19:21:28.916 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.916 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.916 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.916 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "169.254.1.1", 2024-03-11T19:21:28.916 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.916 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.916 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.916 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-11T19:21:28.916 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.916 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.916 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.916 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.916 INFO:teuthology.orchestra.run.smithi029.stdout: "prometheus_alerts_path": { 2024-03-11T19:21:28.916 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prometheus_alerts_path", 2024-03-11T19:21:28.917 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.917 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.917 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.917 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-11T19:21:28.917 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.917 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.917 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.917 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-11T19:21:28.917 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.917 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.917 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.917 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.917 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_insecure": { 2024-03-11T19:21:28.917 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_insecure", 2024-03-11T19:21:28.917 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.918 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.918 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.918 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.918 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.918 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.918 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.918 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-11T19:21:28.918 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.918 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.918 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.918 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.918 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_password": { 2024-03-11T19:21:28.918 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_password", 2024-03-11T19:21:28.918 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.918 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.919 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.919 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.919 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.919 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.919 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.919 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-11T19:21:28.919 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.919 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.919 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.919 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.919 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_url": { 2024-03-11T19:21:28.919 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_url", 2024-03-11T19:21:28.919 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.919 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.920 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.920 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.920 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.920 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.920 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.920 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-11T19:21:28.920 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.920 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.920 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.920 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.920 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_username": { 2024-03-11T19:21:28.920 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_username", 2024-03-11T19:21:28.920 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.920 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.920 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.921 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.921 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.921 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.921 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.921 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-11T19:21:28.921 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.921 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.921 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.921 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.921 INFO:teuthology.orchestra.run.smithi029.stdout: "secure_monitoring_stack": { 2024-03-11T19:21:28.921 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "secure_monitoring_stack", 2024-03-11T19:21:28.921 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.921 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.921 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.922 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.922 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.922 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.922 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.922 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-11T19:21:28.922 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.922 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.922 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.922 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.922 INFO:teuthology.orchestra.run.smithi029.stdout: "service_discovery_port": { 2024-03-11T19:21:28.922 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "service_discovery_port", 2024-03-11T19:21:28.922 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.922 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.922 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.922 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8765", 2024-03-11T19:21:28.922 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.923 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.923 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.923 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "cephadm service discovery port", 2024-03-11T19:21:28.923 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.923 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.923 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.923 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.923 INFO:teuthology.orchestra.run.smithi029.stdout: "ssh_config_file": { 2024-03-11T19:21:28.923 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssh_config_file", 2024-03-11T19:21:28.923 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.923 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.923 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.923 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.923 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.924 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.924 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.924 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-11T19:21:28.924 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.924 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.924 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.924 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.924 INFO:teuthology.orchestra.run.smithi029.stdout: "use_agent": { 2024-03-11T19:21:28.924 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "use_agent", 2024-03-11T19:21:28.924 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.924 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.924 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.924 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.924 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.924 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.924 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.924 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-11T19:21:28.925 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.925 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.925 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.925 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.925 INFO:teuthology.orchestra.run.smithi029.stdout: "use_repo_digest": { 2024-03-11T19:21:28.925 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "use_repo_digest", 2024-03-11T19:21:28.925 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.925 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.925 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.925 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.925 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.925 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.925 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.925 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-11T19:21:28.925 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.925 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.926 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.926 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.926 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_failed_host_check": { 2024-03-11T19:21:28.926 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_failed_host_check", 2024-03-11T19:21:28.926 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.926 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.926 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.926 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.926 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.926 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.926 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.926 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if the host check fails", 2024-03-11T19:21:28.926 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.926 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.926 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.927 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.927 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_stray_daemons": { 2024-03-11T19:21:28.927 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_stray_daemons", 2024-03-11T19:21:28.927 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.927 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.927 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.927 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.927 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.927 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.927 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.927 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-11T19:21:28.927 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.927 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.927 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.927 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.927 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_stray_hosts": { 2024-03-11T19:21:28.927 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_stray_hosts", 2024-03-11T19:21:28.928 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.928 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.928 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.928 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.928 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.928 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.928 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.928 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-11T19:21:28.928 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.928 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.928 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.928 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.928 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.928 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.928 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.928 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crash", 2024-03-11T19:21:28.928 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.929 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.929 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.929 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.929 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.929 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.929 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.929 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.929 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.929 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.929 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.929 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.929 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.929 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.929 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.929 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.929 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.929 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.929 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.930 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.930 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.930 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.930 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.930 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.930 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.930 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.930 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.930 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.930 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.930 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.930 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.930 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.930 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.930 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.930 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.931 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.931 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.931 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.931 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.931 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.931 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.931 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.931 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.931 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.931 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.931 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.931 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.931 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.931 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.931 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.931 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.931 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.932 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.932 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.932 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.932 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.932 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.932 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.932 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.932 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.932 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.932 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.932 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.932 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.932 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.932 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.932 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.932 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.933 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.933 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.933 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.933 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.933 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.933 INFO:teuthology.orchestra.run.smithi029.stdout: "retain_interval": { 2024-03-11T19:21:28.933 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "retain_interval", 2024-03-11T19:21:28.933 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.933 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.933 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.933 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "31536000", 2024-03-11T19:21:28.933 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.933 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.933 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.933 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-11T19:21:28.933 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.933 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.934 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.934 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.934 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_recent_interval": { 2024-03-11T19:21:28.934 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_recent_interval", 2024-03-11T19:21:28.934 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.934 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.934 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.934 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1209600", 2024-03-11T19:21:28.934 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.934 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.934 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.934 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-11T19:21:28.934 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.934 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.934 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.935 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.935 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.935 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.935 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.935 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "dashboard", 2024-03-11T19:21:28.935 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.935 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.935 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.935 INFO:teuthology.orchestra.run.smithi029.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-11T19:21:28.935 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-11T19:21:28.935 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.935 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.935 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.935 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:28.935 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.936 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.936 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.936 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.936 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.936 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.936 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.936 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.936 INFO:teuthology.orchestra.run.smithi029.stdout: "ALERTMANAGER_API_HOST": { 2024-03-11T19:21:28.936 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-11T19:21:28.936 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.936 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.936 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.936 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.936 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.936 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.936 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.936 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.937 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.937 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.937 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.937 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.937 INFO:teuthology.orchestra.run.smithi029.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-11T19:21:28.937 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-11T19:21:28.937 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.937 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.937 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.937 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.937 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.937 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.937 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.937 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.937 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.937 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.938 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.938 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.938 INFO:teuthology.orchestra.run.smithi029.stdout: "AUDIT_API_ENABLED": { 2024-03-11T19:21:28.938 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "AUDIT_API_ENABLED", 2024-03-11T19:21:28.938 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.938 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.938 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.938 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.938 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.938 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.938 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.939 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.939 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.939 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.939 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.939 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.939 INFO:teuthology.orchestra.run.smithi029.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-11T19:21:28.939 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-11T19:21:28.939 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.939 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.939 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.939 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.939 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.939 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.939 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.939 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.940 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.940 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.940 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.940 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.940 INFO:teuthology.orchestra.run.smithi029.stdout: "ENABLE_BROWSABLE_API": { 2024-03-11T19:21:28.940 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-11T19:21:28.940 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.940 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.940 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.940 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.940 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.940 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.940 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.940 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.940 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.940 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.940 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.941 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.941 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-11T19:21:28.941 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-11T19:21:28.941 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.941 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.941 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.941 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.941 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.941 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.941 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.941 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.941 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.941 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.941 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.941 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.942 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-11T19:21:28.942 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-11T19:21:28.942 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.942 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.942 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.942 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.942 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.942 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.942 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.942 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.942 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.942 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.942 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.942 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.943 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-11T19:21:28.943 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-11T19:21:28.943 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.943 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.943 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.943 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.943 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.943 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.943 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.943 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.943 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.943 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.943 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.943 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.943 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-11T19:21:28.943 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-11T19:21:28.944 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.944 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.944 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.944 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.944 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.944 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.944 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.944 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.944 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.944 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.944 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.944 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.944 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-11T19:21:28.944 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-11T19:21:28.944 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.944 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.944 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.945 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.945 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.945 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.945 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.945 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.945 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.945 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.945 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.945 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.945 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-11T19:21:28.945 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-11T19:21:28.945 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.945 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.945 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.945 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.945 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.945 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.946 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.946 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.946 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.946 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.946 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.946 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.946 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-11T19:21:28.946 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-11T19:21:28.946 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.946 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.946 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.946 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.946 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.946 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.946 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.947 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.947 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.947 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.947 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.947 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.947 INFO:teuthology.orchestra.run.smithi029.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-11T19:21:28.947 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-11T19:21:28.947 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.947 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.947 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.947 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.947 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.947 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.947 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.947 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.947 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.947 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.948 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.948 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.948 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_PASSWORD": { 2024-03-11T19:21:28.948 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-11T19:21:28.948 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.948 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.948 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.948 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-03-11T19:21:28.948 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.948 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.948 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.948 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.948 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.948 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.948 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.948 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.949 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-11T19:21:28.949 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-11T19:21:28.949 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.949 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.949 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.949 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.949 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.949 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.949 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.949 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.949 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.949 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.949 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.949 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.950 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_URL": { 2024-03-11T19:21:28.950 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_URL", 2024-03-11T19:21:28.950 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.950 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.950 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.950 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.950 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.950 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.950 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.950 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.950 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.950 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.950 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.950 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.951 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_USERNAME": { 2024-03-11T19:21:28.951 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-11T19:21:28.951 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.951 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.951 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.951 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-03-11T19:21:28.951 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.951 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.951 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.951 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.951 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.951 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.951 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.951 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.951 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-11T19:21:28.951 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-11T19:21:28.952 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.952 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.952 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.952 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.952 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.952 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.952 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.952 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.952 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.952 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.952 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.952 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.952 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-11T19:21:28.952 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-11T19:21:28.952 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.952 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.953 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.953 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.953 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.953 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.953 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.953 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.953 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.953 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.953 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.953 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.953 INFO:teuthology.orchestra.run.smithi029.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-11T19:21:28.953 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-11T19:21:28.953 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.953 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.953 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.953 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.953 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.954 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.954 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.954 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.954 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.954 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.954 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.954 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.954 INFO:teuthology.orchestra.run.smithi029.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-11T19:21:28.954 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-11T19:21:28.954 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.954 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.954 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.954 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.954 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.954 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.955 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.955 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.955 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.955 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.955 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.955 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.955 INFO:teuthology.orchestra.run.smithi029.stdout: "PROMETHEUS_API_HOST": { 2024-03-11T19:21:28.955 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-11T19:21:28.955 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.955 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.955 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.955 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.955 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.955 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.955 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.955 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.955 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.956 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.956 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.956 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.956 INFO:teuthology.orchestra.run.smithi029.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-11T19:21:28.956 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-11T19:21:28.956 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.956 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.956 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.956 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.956 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.956 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.956 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.956 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.956 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.956 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.956 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.957 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-11T19:21:28.957 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-11T19:21:28.957 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.957 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.957 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.957 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.957 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.957 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.957 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.957 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.957 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.957 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.957 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.958 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-11T19:21:28.958 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-11T19:21:28.958 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.958 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.958 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.958 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.958 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.958 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.958 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.958 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.958 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.958 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.958 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.958 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.958 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-11T19:21:28.959 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-11T19:21:28.959 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.959 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.959 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.959 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.959 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.959 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.959 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.959 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.959 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.959 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.959 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.959 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.959 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-11T19:21:28.959 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-11T19:21:28.959 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.960 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.960 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.960 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.960 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.960 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.960 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.960 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.960 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.960 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.960 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.960 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.960 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-11T19:21:28.960 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-11T19:21:28.960 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.960 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.960 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.961 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.961 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.961 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.961 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.961 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.961 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.961 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.961 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.961 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.961 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-11T19:21:28.961 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-11T19:21:28.961 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.961 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.961 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.962 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.962 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.962 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.962 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.962 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.962 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.962 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.962 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.962 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.962 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-11T19:21:28.962 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-11T19:21:28.962 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.963 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.963 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.963 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.963 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.963 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.963 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.963 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.963 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.963 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.963 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.963 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.963 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_ENABLED": { 2024-03-11T19:21:28.963 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-11T19:21:28.963 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.963 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.963 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.964 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.964 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.964 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.964 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.964 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.964 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.964 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.964 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.964 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.964 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-11T19:21:28.964 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-11T19:21:28.964 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.964 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.964 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.965 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-11T19:21:28.965 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.965 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.965 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.965 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.965 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.965 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.965 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.965 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.965 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-11T19:21:28.965 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-11T19:21:28.965 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.965 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.965 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.965 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:28.965 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.966 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.966 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.966 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.966 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.966 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.966 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.966 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.966 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-11T19:21:28.966 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-11T19:21:28.966 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.966 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.966 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.966 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8", 2024-03-11T19:21:28.966 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.967 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.967 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.967 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.967 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.967 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.967 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.967 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.967 INFO:teuthology.orchestra.run.smithi029.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-11T19:21:28.967 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-11T19:21:28.967 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.967 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.967 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.967 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "45", 2024-03-11T19:21:28.967 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.967 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.967 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.968 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.968 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.968 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.968 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.968 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.968 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_ACCESS_KEY": { 2024-03-11T19:21:28.968 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-11T19:21:28.968 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.968 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.968 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.968 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.968 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.968 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.968 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.968 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.969 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.969 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.969 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.969 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.969 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-11T19:21:28.969 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-11T19:21:28.969 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.969 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.969 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.969 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-03-11T19:21:28.969 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.969 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.969 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.969 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.969 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.970 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.970 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.970 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.970 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_SECRET_KEY": { 2024-03-11T19:21:28.970 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-11T19:21:28.970 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.970 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.970 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.970 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.970 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.970 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.970 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.970 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.971 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.971 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.971 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.971 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.971 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_SSL_VERIFY": { 2024-03-11T19:21:28.971 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-11T19:21:28.971 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.971 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.971 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.971 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.971 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.971 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.971 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.971 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.971 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.972 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.972 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.972 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.972 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-11T19:21:28.972 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-11T19:21:28.972 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.972 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.972 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.972 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-11T19:21:28.972 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.972 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.972 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.972 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.972 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.973 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.973 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.973 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.973 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-11T19:21:28.973 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-11T19:21:28.973 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.973 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.973 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.973 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:28.973 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.973 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.973 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.973 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.973 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.973 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.973 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.974 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.974 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-11T19:21:28.974 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-11T19:21:28.974 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.974 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.974 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.974 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-11T19:21:28.974 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.974 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.974 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.974 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.974 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.974 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.974 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.974 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.975 INFO:teuthology.orchestra.run.smithi029.stdout: "cross_origin_url": { 2024-03-11T19:21:28.975 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cross_origin_url", 2024-03-11T19:21:28.975 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.975 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.975 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.975 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.975 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.975 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.975 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.975 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.975 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.975 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.975 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.975 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.975 INFO:teuthology.orchestra.run.smithi029.stdout: "crt_file": { 2024-03-11T19:21:28.975 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crt_file", 2024-03-11T19:21:28.975 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.975 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.976 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.976 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.976 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.976 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.976 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.976 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.976 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.976 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.976 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.976 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.976 INFO:teuthology.orchestra.run.smithi029.stdout: "debug": { 2024-03-11T19:21:28.976 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "debug", 2024-03-11T19:21:28.976 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.976 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.977 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.977 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.977 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.977 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.977 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.977 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable/disable debug options", 2024-03-11T19:21:28.977 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.977 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.977 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.977 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.977 INFO:teuthology.orchestra.run.smithi029.stdout: "jwt_token_ttl": { 2024-03-11T19:21:28.977 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "jwt_token_ttl", 2024-03-11T19:21:28.977 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.977 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.977 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.977 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "28800", 2024-03-11T19:21:28.977 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.978 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.978 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.978 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.978 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.978 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.978 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.978 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.978 INFO:teuthology.orchestra.run.smithi029.stdout: "key_file": { 2024-03-11T19:21:28.978 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "key_file", 2024-03-11T19:21:28.978 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.978 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.978 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.978 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.978 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.978 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.979 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.979 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.979 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.979 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.979 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.979 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.979 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.979 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.979 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.979 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.979 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.979 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.979 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.979 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.979 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.980 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.980 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.980 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.980 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.980 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.980 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.980 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.980 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.980 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.980 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.980 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.980 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.980 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.980 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.980 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.980 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.981 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.981 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.981 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.981 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.981 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.981 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.981 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.981 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.981 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.981 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.981 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.981 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.981 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.981 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.981 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.982 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.982 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.982 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.982 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.982 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.982 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.982 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.982 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.982 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.982 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.982 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.982 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.982 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.983 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.983 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.983 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.983 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.983 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.983 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.983 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.983 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.983 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.983 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.983 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.983 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.983 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.983 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.983 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.984 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.984 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.984 INFO:teuthology.orchestra.run.smithi029.stdout: "motd": { 2024-03-11T19:21:28.984 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "motd", 2024-03-11T19:21:28.984 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.984 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.984 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.984 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.984 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.984 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.984 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.984 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "The message of the day", 2024-03-11T19:21:28.984 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.984 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.984 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.984 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.984 INFO:teuthology.orchestra.run.smithi029.stdout: "redirect_resolve_ip_addr": { 2024-03-11T19:21:28.985 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "redirect_resolve_ip_addr", 2024-03-11T19:21:28.985 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.985 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.985 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.985 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.985 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.985 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.985 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.985 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.985 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.985 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.985 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.985 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.985 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-03-11T19:21:28.985 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-03-11T19:21:28.985 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.986 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.986 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.986 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "::", 2024-03-11T19:21:28.986 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.986 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.986 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.986 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.986 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.986 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.986 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.986 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.986 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-03-11T19:21:28.986 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-03-11T19:21:28.986 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.986 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.987 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.987 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8080", 2024-03-11T19:21:28.987 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.987 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.987 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.987 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.987 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.987 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.987 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.987 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.987 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl": { 2024-03-11T19:21:28.987 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl", 2024-03-11T19:21:28.987 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.987 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.987 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.987 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.987 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.988 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.988 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.988 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.988 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.988 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.988 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.988 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.988 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl_server_port": { 2024-03-11T19:21:28.988 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl_server_port", 2024-03-11T19:21:28.988 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.988 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.988 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.988 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8443", 2024-03-11T19:21:28.988 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.989 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.989 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.989 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.989 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.989 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.989 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.989 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.989 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_behaviour": { 2024-03-11T19:21:28.989 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_behaviour", 2024-03-11T19:21:28.989 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.989 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.989 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.989 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "redirect", 2024-03-11T19:21:28.989 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.990 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.990 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.990 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.990 INFO:teuthology.orchestra.run.smithi029.stdout: "redirect" 2024-03-11T19:21:28.990 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.990 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.990 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.990 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.991 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.991 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.991 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_error_status_code": { 2024-03-11T19:21:28.991 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_error_status_code", 2024-03-11T19:21:28.991 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:28.991 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.991 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.991 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-03-11T19:21:28.991 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "400", 2024-03-11T19:21:28.991 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "599", 2024-03-11T19:21:28.991 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.992 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.992 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.992 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.992 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.992 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.992 INFO:teuthology.orchestra.run.smithi029.stdout: "url_prefix": { 2024-03-11T19:21:28.992 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "url_prefix", 2024-03-11T19:21:28.992 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.992 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.992 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:28.992 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.992 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.992 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.992 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.992 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.993 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.993 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.993 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.993 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.993 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:28.993 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.993 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:28.993 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "devicehealth", 2024-03-11T19:21:28.993 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:28.993 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:28.993 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:28.993 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_monitoring": { 2024-03-11T19:21:28.993 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enable_monitoring", 2024-03-11T19:21:28.993 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.993 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.994 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.994 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:28.994 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.994 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.994 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.994 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "monitor device health metrics", 2024-03-11T19:21:28.994 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.994 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.994 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.994 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.994 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:28.994 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:28.994 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.994 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.995 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.995 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:28.995 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.995 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.995 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.995 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.995 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.995 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.995 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.995 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.995 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.995 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.995 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.995 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.995 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.995 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.996 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.996 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:28.996 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:28.996 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.996 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.996 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.996 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.996 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.996 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.996 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.996 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.996 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.996 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.996 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.997 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.997 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:28.997 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:28.997 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:28.997 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.997 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.997 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:28.997 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.997 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.997 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:28.997 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:28.997 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:28.997 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:28.997 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:28.997 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:28.997 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:28.998 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:28.998 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.998 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.998 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.998 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.998 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.998 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:28.998 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:28.998 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:28.998 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.998 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.998 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:28.998 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.998 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.998 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.999 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:28.999 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.999 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.999 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:28.999 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:28.999 INFO:teuthology.orchestra.run.smithi029.stdout: "mark_out_threshold": { 2024-03-11T19:21:28.999 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mark_out_threshold", 2024-03-11T19:21:28.999 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:28.999 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:28.999 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:28.999 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "2419200", 2024-03-11T19:21:28.999 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:28.999 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:28.999 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:28.999 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-11T19:21:28.999 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:28.999 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:28.999 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.000 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.000 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": { 2024-03-11T19:21:29.000 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pool_name", 2024-03-11T19:21:29.000 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.000 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.000 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.000 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "device_health_metrics", 2024-03-11T19:21:29.000 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.000 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.000 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.000 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-11T19:21:29.000 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.000 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.000 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.000 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.000 INFO:teuthology.orchestra.run.smithi029.stdout: "retention_period": { 2024-03-11T19:21:29.000 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "retention_period", 2024-03-11T19:21:29.001 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.001 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.001 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.001 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15552000", 2024-03-11T19:21:29.001 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.001 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.001 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.001 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long to retain device health metrics", 2024-03-11T19:21:29.001 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.001 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.001 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.001 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.001 INFO:teuthology.orchestra.run.smithi029.stdout: "scrape_frequency": { 2024-03-11T19:21:29.001 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "scrape_frequency", 2024-03-11T19:21:29.001 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.002 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.002 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.002 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "86400", 2024-03-11T19:21:29.002 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.002 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.002 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.002 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-11T19:21:29.002 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.002 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.002 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.002 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.002 INFO:teuthology.orchestra.run.smithi029.stdout: "self_heal": { 2024-03-11T19:21:29.002 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "self_heal", 2024-03-11T19:21:29.002 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.003 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.003 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.003 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.003 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.003 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.003 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.003 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-11T19:21:29.003 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.003 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.003 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.003 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.003 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-11T19:21:29.003 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-11T19:21:29.003 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.003 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.003 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.003 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-11T19:21:29.004 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.004 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.004 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.004 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to wake up and check device health", 2024-03-11T19:21:29.004 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.004 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.004 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.004 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.004 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_threshold": { 2024-03-11T19:21:29.004 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_threshold", 2024-03-11T19:21:29.004 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.004 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.004 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.004 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7257600", 2024-03-11T19:21:29.004 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.005 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.005 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.005 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-11T19:21:29.005 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.005 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.005 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.005 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.005 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.005 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.005 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.005 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "diskprediction_local", 2024-03-11T19:21:29.005 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.005 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.005 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.005 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.005 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.006 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.006 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.006 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.006 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.006 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.006 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.006 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.006 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.006 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.006 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.006 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.006 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.006 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.006 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.006 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.007 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.007 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.007 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.007 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.007 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.007 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.007 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.007 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.007 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.007 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.007 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.007 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.007 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.007 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.007 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.007 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.007 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.008 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.008 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.008 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.008 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.008 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.008 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.008 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.008 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.008 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.008 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.008 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.008 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.008 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.008 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.008 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.008 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.009 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.009 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.009 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.009 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.009 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.009 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.009 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.009 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.009 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.009 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.009 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.009 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.009 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.009 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.009 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.010 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.010 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.010 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.010 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.010 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.010 INFO:teuthology.orchestra.run.smithi029.stdout: "predict_interval": { 2024-03-11T19:21:29.010 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "predict_interval", 2024-03-11T19:21:29.010 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.010 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.010 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.010 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "86400", 2024-03-11T19:21:29.010 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.010 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.010 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.010 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.011 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.011 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.011 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.011 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.011 INFO:teuthology.orchestra.run.smithi029.stdout: "predictor_model": { 2024-03-11T19:21:29.011 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "predictor_model", 2024-03-11T19:21:29.011 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.011 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.011 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.011 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "prophetstor", 2024-03-11T19:21:29.011 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.011 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.011 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.011 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.012 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.012 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.012 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.012 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.012 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-11T19:21:29.012 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-11T19:21:29.012 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.012 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.012 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.012 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-11T19:21:29.012 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.012 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.012 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.012 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.013 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.013 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.013 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.013 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.013 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.013 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.013 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.013 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "influx", 2024-03-11T19:21:29.013 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": false, 2024-03-11T19:21:29.013 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "influxdb python module not found", 2024-03-11T19:21:29.013 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.013 INFO:teuthology.orchestra.run.smithi029.stdout: "batch_size": { 2024-03-11T19:21:29.013 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "batch_size", 2024-03-11T19:21:29.013 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.013 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.013 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.014 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5000", 2024-03-11T19:21:29.014 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.014 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.014 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.014 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-11T19:21:29.014 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.014 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.014 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.014 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.014 INFO:teuthology.orchestra.run.smithi029.stdout: "database": { 2024-03-11T19:21:29.014 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "database", 2024-03-11T19:21:29.014 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.014 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.015 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.015 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "ceph", 2024-03-11T19:21:29.015 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.015 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.015 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.015 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-11T19:21:29.015 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.015 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.015 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.015 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.015 INFO:teuthology.orchestra.run.smithi029.stdout: "hostname": { 2024-03-11T19:21:29.015 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "hostname", 2024-03-11T19:21:29.015 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.015 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.016 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.016 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.016 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.016 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.016 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.016 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB server hostname", 2024-03-11T19:21:29.016 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.016 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.016 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.016 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.016 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-11T19:21:29.016 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-11T19:21:29.016 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.016 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.016 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.017 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "30", 2024-03-11T19:21:29.017 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "5", 2024-03-11T19:21:29.017 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.017 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.017 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-11T19:21:29.017 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.017 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.017 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.017 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.017 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.017 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.017 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.018 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.018 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.018 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.018 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.018 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.018 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.018 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.018 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.018 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.018 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.018 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.018 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.018 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.018 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.018 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.019 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.019 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.019 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.019 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.019 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.019 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.019 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.019 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.019 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.019 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.019 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.019 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.019 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.019 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.019 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.019 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.019 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.020 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.020 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.020 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.020 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.020 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.020 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.020 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.020 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.020 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.020 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.020 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.020 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.020 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.020 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.020 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.020 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.021 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.021 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.021 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.021 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.021 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.021 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.021 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.021 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.021 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.021 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.021 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.021 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.021 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.021 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.022 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.022 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.022 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.022 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.022 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.022 INFO:teuthology.orchestra.run.smithi029.stdout: "password": { 2024-03-11T19:21:29.022 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "password", 2024-03-11T19:21:29.022 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.022 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.022 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.022 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.022 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.022 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.022 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.022 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "password of InfluxDB server user", 2024-03-11T19:21:29.023 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.023 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.023 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.023 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.023 INFO:teuthology.orchestra.run.smithi029.stdout: "port": { 2024-03-11T19:21:29.023 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "port", 2024-03-11T19:21:29.023 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.023 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.023 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.023 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8086", 2024-03-11T19:21:29.023 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.023 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.023 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.023 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB server port", 2024-03-11T19:21:29.023 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.023 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.024 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.024 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.024 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl": { 2024-03-11T19:21:29.024 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl", 2024-03-11T19:21:29.024 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.024 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.024 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.024 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "false", 2024-03-11T19:21:29.024 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.024 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.024 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.024 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-11T19:21:29.024 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.024 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.024 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.024 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.025 INFO:teuthology.orchestra.run.smithi029.stdout: "threads": { 2024-03-11T19:21:29.025 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "threads", 2024-03-11T19:21:29.025 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.025 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.025 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.025 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-11T19:21:29.025 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-11T19:21:29.025 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "32", 2024-03-11T19:21:29.025 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.025 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-11T19:21:29.025 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.025 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.025 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.025 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.025 INFO:teuthology.orchestra.run.smithi029.stdout: "username": { 2024-03-11T19:21:29.025 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "username", 2024-03-11T19:21:29.026 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.026 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.026 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.026 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.026 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.026 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.026 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.026 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "username of InfluxDB server user", 2024-03-11T19:21:29.026 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.026 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.026 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.026 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.026 INFO:teuthology.orchestra.run.smithi029.stdout: "verify_ssl": { 2024-03-11T19:21:29.026 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "verify_ssl", 2024-03-11T19:21:29.026 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.027 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.027 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.027 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "true", 2024-03-11T19:21:29.027 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.027 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.027 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.027 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-11T19:21:29.027 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.027 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.027 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.027 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.027 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.027 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.027 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.027 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "insights", 2024-03-11T19:21:29.027 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.027 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.028 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.028 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.028 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.028 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.028 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.028 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.028 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.028 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.028 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.028 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.028 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.028 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.028 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.028 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.028 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.028 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.029 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.029 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.029 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.029 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.029 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.029 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.029 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.029 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.029 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.029 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.029 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.029 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.029 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.029 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.029 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.030 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.030 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.030 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.030 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.030 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.030 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.030 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.030 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.030 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.030 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.030 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.030 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.030 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.030 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.031 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.031 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.031 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.031 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.031 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.031 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.031 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.031 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.031 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.031 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.031 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.031 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.031 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.031 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.032 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.032 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.032 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.032 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.032 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.032 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.032 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.032 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.032 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.032 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.032 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.032 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.032 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.032 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.032 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.033 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "iostat", 2024-03-11T19:21:29.033 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.033 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.033 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.033 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.033 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.033 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.033 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.033 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.033 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.033 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.033 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.033 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.033 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.033 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.033 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.034 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.034 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.034 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.034 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.034 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.034 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.034 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.034 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.034 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.034 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.034 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.034 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.034 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.034 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.034 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.034 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.035 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.035 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.035 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.035 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.035 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.035 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.035 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.035 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.035 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.035 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.035 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.035 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.035 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.035 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.035 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.035 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.035 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.036 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.037 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.037 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.037 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.037 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.037 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.037 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.037 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.037 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.037 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.037 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.037 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "k8sevents", 2024-03-11T19:21:29.037 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.037 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.037 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.037 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_event_retention_days": { 2024-03-11T19:21:29.037 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ceph_event_retention_days", 2024-03-11T19:21:29.038 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.038 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.038 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.038 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7", 2024-03-11T19:21:29.038 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.038 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.038 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.038 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-11T19:21:29.038 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.038 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.038 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.038 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.038 INFO:teuthology.orchestra.run.smithi029.stdout: "config_check_secs": { 2024-03-11T19:21:29.038 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_check_secs", 2024-03-11T19:21:29.038 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.039 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.039 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.039 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:29.039 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "10", 2024-03-11T19:21:29.039 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.039 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.039 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-11T19:21:29.039 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.039 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.039 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.039 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.039 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.039 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.039 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.039 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.039 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.039 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.039 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.040 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.040 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.040 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.040 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.040 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.040 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.040 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.040 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.040 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.040 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.040 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.040 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.040 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.040 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.040 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.040 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.040 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.040 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.041 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.041 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.041 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.041 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.041 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.041 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.041 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.041 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.041 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.041 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.041 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.041 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.041 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.041 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.041 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.041 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.042 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.042 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.042 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.042 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.042 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.042 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.042 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.042 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.042 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.042 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.042 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.042 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.042 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.042 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.043 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.043 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.043 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.043 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.043 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.043 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.043 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.043 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.043 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.043 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.043 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.043 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.043 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.043 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.043 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.043 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.043 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.044 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.044 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "localpool", 2024-03-11T19:21:29.044 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.044 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.044 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.044 INFO:teuthology.orchestra.run.smithi029.stdout: "failure_domain": { 2024-03-11T19:21:29.044 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "failure_domain", 2024-03-11T19:21:29.044 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.044 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.044 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.044 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "host", 2024-03-11T19:21:29.044 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.044 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.044 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.044 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "failure domain for any created local pool", 2024-03-11T19:21:29.044 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-11T19:21:29.045 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.045 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.045 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.045 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.045 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.045 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.045 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.045 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.045 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.045 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.045 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.045 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.045 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.045 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.045 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.045 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.046 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.046 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.046 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.046 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.046 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.046 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.046 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.046 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.046 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.046 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.046 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.046 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.046 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.046 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.046 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.047 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.047 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.047 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.047 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.047 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.047 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.047 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.047 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.047 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.047 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.047 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.047 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.047 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.048 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.048 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.048 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.048 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.048 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.048 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.048 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.048 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.048 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.048 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.048 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.048 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.048 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.048 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.048 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.049 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.049 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.049 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.049 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.049 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.049 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.049 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.049 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.049 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.049 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.049 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.049 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.049 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.049 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.049 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": { 2024-03-11T19:21:29.049 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "min_size", 2024-03-11T19:21:29.049 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.050 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.050 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.050 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.050 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.050 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.050 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.050 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default min_size for any created local pool", 2024-03-11T19:21:29.050 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-11T19:21:29.050 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.050 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.050 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.050 INFO:teuthology.orchestra.run.smithi029.stdout: "num_rep": { 2024-03-11T19:21:29.050 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "num_rep", 2024-03-11T19:21:29.050 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.051 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.051 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.051 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3", 2024-03-11T19:21:29.051 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.051 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.051 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.051 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default replica count for any created local pool", 2024-03-11T19:21:29.051 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.051 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.051 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.051 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.051 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": { 2024-03-11T19:21:29.051 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pg_num", 2024-03-11T19:21:29.051 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.051 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.051 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.052 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "128", 2024-03-11T19:21:29.052 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.052 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.052 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.052 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default pg_num for any created local pool", 2024-03-11T19:21:29.052 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.052 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.052 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.052 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.052 INFO:teuthology.orchestra.run.smithi029.stdout: "prefix": { 2024-03-11T19:21:29.052 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prefix", 2024-03-11T19:21:29.052 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.052 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.052 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.052 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.052 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.052 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.053 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.053 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "name prefix for any created local pool", 2024-03-11T19:21:29.053 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.053 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.053 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.053 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.053 INFO:teuthology.orchestra.run.smithi029.stdout: "subtree": { 2024-03-11T19:21:29.053 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "subtree", 2024-03-11T19:21:29.053 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.053 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.053 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.053 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "rack", 2024-03-11T19:21:29.053 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.053 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.053 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.053 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-11T19:21:29.053 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-11T19:21:29.053 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.054 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.054 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.054 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.054 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.054 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.054 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mds_autoscaler", 2024-03-11T19:21:29.054 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.054 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.054 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.054 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.054 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.054 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.054 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.054 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.054 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.055 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.055 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.055 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.055 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.055 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.055 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.055 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.055 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.055 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.055 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.055 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.055 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.055 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.055 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.055 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.055 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.055 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.055 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.056 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.056 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.056 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.056 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.056 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.056 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.056 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.056 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.056 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.056 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.056 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.056 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.057 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.057 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.057 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.057 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.057 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.057 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.057 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.057 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.057 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.057 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.057 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.057 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.057 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.057 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.058 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.058 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.058 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.058 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.058 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.058 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.058 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.058 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.058 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.058 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.058 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.058 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.058 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.058 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.058 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.059 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.059 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.059 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.059 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.059 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.059 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.059 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.059 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.059 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mirroring", 2024-03-11T19:21:29.059 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.059 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.059 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.059 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.059 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.059 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.060 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.060 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.060 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.060 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.060 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.060 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.060 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.060 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.060 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.060 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.060 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.060 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.060 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.060 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.060 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.061 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.061 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.061 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.061 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.061 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.061 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.061 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.061 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.061 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.061 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.061 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.061 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.061 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.061 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.062 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.062 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.062 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.062 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.062 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.062 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.062 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.062 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.062 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.062 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.062 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.062 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.062 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.062 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.063 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.063 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.063 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.063 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.063 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.063 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.063 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.063 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.063 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.063 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.063 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.063 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.063 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.063 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.063 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.064 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.064 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.064 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.064 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.064 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.064 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.064 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.064 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.064 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.064 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.064 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.064 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "nfs", 2024-03-11T19:21:29.064 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.064 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.064 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.065 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.065 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.065 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.065 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.065 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.065 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.065 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.065 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.065 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.065 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.065 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.065 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.065 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.065 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.065 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.066 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.066 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.066 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.066 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.066 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.066 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.066 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.066 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.066 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.066 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.066 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.066 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.067 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.067 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.067 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.067 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.067 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.067 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.067 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.067 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.067 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.067 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.067 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.067 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.067 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.067 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.067 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.068 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.068 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.068 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.068 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.068 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.068 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.068 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.068 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.068 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.068 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.068 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.068 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.068 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.069 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.069 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.069 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.069 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.069 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.069 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.069 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.069 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.069 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.069 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.069 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.069 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.069 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.069 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.069 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.070 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.070 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.070 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.070 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "orchestrator", 2024-03-11T19:21:29.070 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.070 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.070 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.070 INFO:teuthology.orchestra.run.smithi029.stdout: "fail_fs": { 2024-03-11T19:21:29.070 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "fail_fs", 2024-03-11T19:21:29.070 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.070 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.070 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.070 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.071 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.071 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.071 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.071 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-11T19:21:29.071 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.071 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.071 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.071 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.071 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.071 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.071 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.071 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.071 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.071 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.071 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.072 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.072 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.072 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.072 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.072 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.072 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.072 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.072 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.072 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.072 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.072 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.072 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.072 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.072 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.072 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.072 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.072 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.073 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.073 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.073 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.073 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.073 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.073 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.073 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.073 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.073 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.073 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.073 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.073 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.073 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.073 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.073 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.073 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.073 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.074 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.074 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.074 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.074 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.074 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.074 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.074 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.074 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.074 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.074 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.074 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.074 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.074 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.075 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.075 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.075 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.075 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.075 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.075 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.075 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.075 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.077 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.077 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.077 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.077 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.078 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.078 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.078 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.078 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.078 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator": { 2024-03-11T19:21:29.078 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "orchestrator", 2024-03-11T19:21:29.078 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.078 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.078 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.078 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.078 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.078 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.078 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.078 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm", 2024-03-11T19:21:29.079 INFO:teuthology.orchestra.run.smithi029.stdout: "rook", 2024-03-11T19:21:29.079 INFO:teuthology.orchestra.run.smithi029.stdout: "test_orchestrator" 2024-03-11T19:21:29.079 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.079 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Orchestrator backend", 2024-03-11T19:21:29.079 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.079 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.079 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.079 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.079 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.079 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.079 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.079 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "osd_perf_query", 2024-03-11T19:21:29.079 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.079 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.079 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.079 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.080 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.080 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.080 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.080 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.080 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.080 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.080 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.080 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.080 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.080 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.080 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.080 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.080 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.080 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.081 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.081 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.081 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.081 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.081 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.081 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.081 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.081 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.081 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.081 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.081 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.081 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.081 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.081 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.081 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.082 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.082 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.082 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.082 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.082 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.082 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.082 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.082 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.082 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.082 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.082 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.082 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.082 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.083 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.083 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.083 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.083 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.083 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.083 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.083 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.083 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.083 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.083 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.083 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.083 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.083 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.083 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.083 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.084 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.084 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.084 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.084 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.084 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.084 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.084 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.084 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.084 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.084 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.084 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.084 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.084 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.084 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.084 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.085 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "osd_support", 2024-03-11T19:21:29.085 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.085 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.085 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.085 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.085 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.085 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.085 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.085 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.085 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.085 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.085 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.085 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.085 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.086 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.086 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.086 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.086 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.086 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.086 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.086 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.086 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.086 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.086 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.086 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.086 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.086 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.087 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.087 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.087 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.087 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.087 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.087 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.087 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.087 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.087 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.087 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.087 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.087 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.087 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.087 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.088 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.088 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.088 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.088 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.088 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.088 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.088 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.088 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.088 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.088 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.088 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.088 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.088 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.088 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.089 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.089 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.089 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.089 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.089 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.089 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.089 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.089 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.089 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.089 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.089 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.089 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.089 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.089 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.089 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.089 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.089 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.090 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.090 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.090 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.090 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.090 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.090 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pg_autoscaler", 2024-03-11T19:21:29.090 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.090 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.090 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.090 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.090 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.090 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.090 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.090 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.091 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.091 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.091 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.091 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.091 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.091 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.091 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.091 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.091 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.091 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.091 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.091 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.091 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.091 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.091 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.091 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.092 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.092 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.092 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.092 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.092 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.092 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.092 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.092 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.092 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.092 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.092 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.092 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.092 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.092 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.093 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.093 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.093 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.093 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.093 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.093 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.093 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.093 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.093 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.093 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.093 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.093 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.093 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.093 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.093 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.093 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.094 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.094 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.094 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.094 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.094 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.094 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.094 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.094 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.094 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.094 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.094 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.094 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.094 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.094 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.094 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.095 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.095 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.095 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.095 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.095 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-11T19:21:29.095 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-11T19:21:29.095 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.095 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.095 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.095 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-11T19:21:29.095 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.095 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.095 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.095 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.095 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.096 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.096 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.096 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.096 INFO:teuthology.orchestra.run.smithi029.stdout: "threshold": { 2024-03-11T19:21:29.096 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "threshold", 2024-03-11T19:21:29.096 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:29.096 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.096 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.096 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3.0", 2024-03-11T19:21:29.096 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1.0", 2024-03-11T19:21:29.096 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.096 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.096 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "scaling threshold", 2024-03-11T19:21:29.096 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-11T19:21:29.097 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.097 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.097 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.097 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.097 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.097 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.097 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "progress", 2024-03-11T19:21:29.097 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.097 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.097 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.097 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_pg_recovery_event": { 2024-03-11T19:21:29.097 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_pg_recovery_event", 2024-03-11T19:21:29.097 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.097 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.098 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.098 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.098 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.098 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.098 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.098 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-11T19:21:29.098 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.098 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.098 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.098 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.098 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": { 2024-03-11T19:21:29.098 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enabled", 2024-03-11T19:21:29.098 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.098 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.098 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.099 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.099 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.099 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.099 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.099 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.099 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.099 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.099 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.099 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.099 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.099 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.099 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.099 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.099 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.100 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.100 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.100 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.100 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.100 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.100 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.100 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.100 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.100 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.100 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.100 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.100 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.100 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.100 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.100 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.101 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.101 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.101 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.101 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.101 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.101 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.101 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.101 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.101 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.101 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.101 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.101 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.101 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.102 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.102 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.102 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.102 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.102 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.102 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.102 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.102 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.102 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.102 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.102 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.102 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.102 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.102 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.103 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.103 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.103 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.103 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.103 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.103 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.103 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.103 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.103 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.103 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.103 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.103 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.103 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.103 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.103 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.104 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.104 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.104 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.104 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.104 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.104 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.104 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.104 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.104 INFO:teuthology.orchestra.run.smithi029.stdout: "max_completed_events": { 2024-03-11T19:21:29.104 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_completed_events", 2024-03-11T19:21:29.104 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.104 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.104 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.104 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "50", 2024-03-11T19:21:29.104 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.105 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.105 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.105 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "number of past completed events to remember", 2024-03-11T19:21:29.105 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.105 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.105 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.105 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.105 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-11T19:21:29.105 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-11T19:21:29.105 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.105 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.105 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.105 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-11T19:21:29.105 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.105 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.106 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.106 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long the module is going to sleep", 2024-03-11T19:21:29.106 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.106 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.106 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.106 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.106 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.106 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.106 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.106 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prometheus", 2024-03-11T19:21:29.106 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.106 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.106 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.107 INFO:teuthology.orchestra.run.smithi029.stdout: "cache": { 2024-03-11T19:21:29.107 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cache", 2024-03-11T19:21:29.107 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.107 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.107 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.107 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.107 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.107 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.107 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.107 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.107 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.107 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.107 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.107 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.108 INFO:teuthology.orchestra.run.smithi029.stdout: "exclude_perf_counters": { 2024-03-11T19:21:29.108 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "exclude_perf_counters", 2024-03-11T19:21:29.108 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.108 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.108 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.108 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.108 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.108 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.108 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.108 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-11T19:21:29.108 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-11T19:21:29.108 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.108 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.108 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.109 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.109 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.109 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.109 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.109 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.109 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.109 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.109 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.109 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.109 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.109 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.109 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.109 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.110 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.110 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.110 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.110 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.110 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.110 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.110 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.110 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.110 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.110 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.110 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.110 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.111 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.111 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.111 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.111 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.111 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.111 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.111 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.111 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.111 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.111 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.111 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.111 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.111 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.111 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.112 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.112 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.112 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.112 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.112 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.112 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.112 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.112 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.112 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.112 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.112 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.112 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.113 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.113 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.113 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.113 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.113 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.113 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.113 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.113 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.113 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.113 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.113 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.113 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.113 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.113 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.114 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.114 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.114 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.114 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.114 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_stats_pools": { 2024-03-11T19:21:29.114 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_stats_pools", 2024-03-11T19:21:29.114 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.114 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.114 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.114 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.114 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.114 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.114 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.114 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.115 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.115 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.115 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.115 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.115 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-11T19:21:29.115 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-11T19:21:29.115 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.115 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.115 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.115 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "300", 2024-03-11T19:21:29.115 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.115 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.115 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.116 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.116 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.116 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.116 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.116 INFO:teuthology.orchestra.run.smithi029.stdout: "scrape_interval": { 2024-03-11T19:21:29.116 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "scrape_interval", 2024-03-11T19:21:29.116 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:29.116 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.116 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.116 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15.0", 2024-03-11T19:21:29.116 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.116 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.116 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.117 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.117 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.117 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.117 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.117 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.117 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-03-11T19:21:29.117 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-03-11T19:21:29.117 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.117 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.117 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.117 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "::", 2024-03-11T19:21:29.117 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.117 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.117 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.117 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-11T19:21:29.117 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.118 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.118 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.118 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.118 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-03-11T19:21:29.118 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-03-11T19:21:29.118 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.118 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.118 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.118 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "9283", 2024-03-11T19:21:29.118 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.118 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.118 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.118 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-11T19:21:29.119 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.119 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.119 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.119 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.119 INFO:teuthology.orchestra.run.smithi029.stdout: "stale_cache_strategy": { 2024-03-11T19:21:29.119 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "stale_cache_strategy", 2024-03-11T19:21:29.119 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.119 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.119 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.119 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "log", 2024-03-11T19:21:29.119 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.119 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.119 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.119 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.119 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.120 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.120 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.120 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.120 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_behaviour": { 2024-03-11T19:21:29.120 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_behaviour", 2024-03-11T19:21:29.120 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.120 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.120 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.120 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "default", 2024-03-11T19:21:29.120 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.120 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.120 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.120 INFO:teuthology.orchestra.run.smithi029.stdout: "default", 2024-03-11T19:21:29.120 INFO:teuthology.orchestra.run.smithi029.stdout: "error" 2024-03-11T19:21:29.120 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.120 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.121 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.121 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.121 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.121 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.121 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_error_status_code": { 2024-03-11T19:21:29.121 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_error_status_code", 2024-03-11T19:21:29.121 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.121 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.121 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.121 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-03-11T19:21:29.121 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "400", 2024-03-11T19:21:29.121 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "599", 2024-03-11T19:21:29.121 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.121 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.121 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.122 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.122 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.122 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.122 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.122 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.122 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.122 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_support", 2024-03-11T19:21:29.122 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.122 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.122 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.122 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.122 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.122 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.122 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.122 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.123 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.123 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.123 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.123 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.123 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.123 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.123 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.123 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.123 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.123 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.123 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.123 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.123 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.123 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.123 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.123 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.124 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.124 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.124 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.124 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.124 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.124 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.124 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.124 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.124 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.124 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.124 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.124 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.124 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.124 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.124 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.125 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.125 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.125 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.125 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.125 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.125 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.125 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.125 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.125 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.125 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.125 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.125 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.125 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.125 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.125 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.125 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.126 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.126 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.126 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.126 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.126 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.126 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.126 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.126 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.126 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.126 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.126 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.126 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.126 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.126 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.126 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.127 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.127 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.127 INFO:teuthology.orchestra.run.smithi029.stdout: "max_concurrent_snap_create": { 2024-03-11T19:21:29.127 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_concurrent_snap_create", 2024-03-11T19:21:29.127 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.127 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.127 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.127 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:29.127 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.127 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.127 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.127 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.127 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.127 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.128 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.128 INFO:teuthology.orchestra.run.smithi029.stdout: "mirror_snapshot_schedule": { 2024-03-11T19:21:29.128 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mirror_snapshot_schedule", 2024-03-11T19:21:29.128 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.128 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.128 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.128 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.128 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.128 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.128 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.128 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.128 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.128 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.128 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.129 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.129 INFO:teuthology.orchestra.run.smithi029.stdout: "trash_purge_schedule": { 2024-03-11T19:21:29.129 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "trash_purge_schedule", 2024-03-11T19:21:29.129 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.129 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.129 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.129 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.129 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.129 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.129 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.129 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.129 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.129 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.129 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.129 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.130 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.130 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.130 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.130 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "restful", 2024-03-11T19:21:29.130 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.130 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.130 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.130 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_auth": { 2024-03-11T19:21:29.130 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enable_auth", 2024-03-11T19:21:29.130 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.130 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.130 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.130 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.131 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.131 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.131 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.131 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.131 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.131 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.131 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.131 INFO:teuthology.orchestra.run.smithi029.stdout: "key_file": { 2024-03-11T19:21:29.131 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "key_file", 2024-03-11T19:21:29.131 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.131 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.131 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.131 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.131 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.132 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.132 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.132 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.132 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.132 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.132 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.132 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.132 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.132 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.132 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.132 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.132 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.132 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.132 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.133 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.133 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.133 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.133 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.133 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.133 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.133 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.133 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.133 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.133 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.133 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.133 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.133 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.133 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.133 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.134 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.134 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.134 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.134 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.134 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.134 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.134 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.134 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.134 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.134 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.134 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.134 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.134 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.134 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.134 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.135 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.135 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.135 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.135 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.135 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.135 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.135 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.135 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.135 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.135 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.135 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.135 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.135 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.136 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.136 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.136 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.136 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.136 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.136 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.136 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.136 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.136 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.136 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.136 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.136 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.136 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.136 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.137 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.137 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.137 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.137 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.137 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.137 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-03-11T19:21:29.137 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-03-11T19:21:29.137 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.137 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.137 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.137 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.137 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.137 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.137 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.138 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.138 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.138 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.138 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.138 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.138 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-03-11T19:21:29.138 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-03-11T19:21:29.138 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.138 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.138 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.138 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.138 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.138 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.138 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.139 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.139 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.139 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.139 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.139 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.139 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.139 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.139 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.139 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rgw", 2024-03-11T19:21:29.139 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.139 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.139 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.139 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.139 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.139 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.140 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.140 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.140 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.140 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.140 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.140 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.140 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.140 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.140 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.140 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.140 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.140 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.140 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.141 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.141 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.141 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.141 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.141 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.141 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.141 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.141 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.141 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.141 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.141 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.141 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.141 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.141 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.141 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.141 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.142 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.142 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.142 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.142 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.142 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.142 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.142 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.142 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.142 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.142 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.142 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.142 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.142 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.142 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.143 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.143 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.143 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.143 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.143 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.143 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.143 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.143 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.143 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.143 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.143 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.143 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.143 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.143 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.143 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.143 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.144 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.144 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.144 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.144 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.144 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.144 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.144 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.144 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.144 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.144 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.144 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.144 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rook", 2024-03-11T19:21:29.144 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.144 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.144 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.144 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.144 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.145 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.145 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.145 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.145 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.145 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.145 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.145 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.145 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.145 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.145 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.145 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.145 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.145 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.145 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.145 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.145 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.146 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.146 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.146 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.146 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.146 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.146 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.146 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.146 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.146 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.146 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.146 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.146 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.146 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.146 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.146 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.147 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.147 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.147 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.147 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.147 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.147 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.147 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.147 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.147 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.147 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.147 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.147 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.147 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.147 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.147 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.147 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.148 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.148 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.148 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.148 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.148 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.148 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.148 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.148 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.148 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.148 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.148 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.148 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.148 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.148 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.148 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.148 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.148 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.149 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.149 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.149 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.149 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.149 INFO:teuthology.orchestra.run.smithi029.stdout: "storage_class": { 2024-03-11T19:21:29.149 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "storage_class", 2024-03-11T19:21:29.149 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.149 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.149 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.149 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "local", 2024-03-11T19:21:29.149 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.149 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.149 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.150 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-11T19:21:29.150 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.150 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.150 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.150 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.150 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.150 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.150 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.150 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "selftest", 2024-03-11T19:21:29.150 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.150 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.150 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.150 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.150 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.151 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.151 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.151 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.151 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.151 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.151 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.151 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.151 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.151 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.151 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.151 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.151 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.151 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.151 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.151 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.152 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.152 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.152 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.152 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.152 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.152 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.152 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.152 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.152 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.152 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.152 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.152 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.152 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.152 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.152 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.152 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.153 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.153 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.153 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.153 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.153 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.153 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.153 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.153 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.153 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.153 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.153 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.153 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.153 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.153 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.153 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.153 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.153 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.154 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.154 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.154 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.154 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.154 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.154 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.154 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.154 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.154 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.154 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.154 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.154 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.154 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.154 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.154 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.155 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.155 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.155 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.155 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.155 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.155 INFO:teuthology.orchestra.run.smithi029.stdout: "roption1": { 2024-03-11T19:21:29.155 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "roption1", 2024-03-11T19:21:29.155 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.155 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.155 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.155 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.155 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.155 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.155 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.155 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.155 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.156 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.156 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.156 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.156 INFO:teuthology.orchestra.run.smithi029.stdout: "roption2": { 2024-03-11T19:21:29.156 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "roption2", 2024-03-11T19:21:29.156 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.156 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.156 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.156 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "xyz", 2024-03-11T19:21:29.156 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.156 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.156 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.156 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.156 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.156 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.156 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.156 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.157 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption1": { 2024-03-11T19:21:29.157 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption1", 2024-03-11T19:21:29.157 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.157 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.157 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.157 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.157 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.157 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.157 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.157 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.157 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.157 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.157 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.157 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.157 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption2": { 2024-03-11T19:21:29.157 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption2", 2024-03-11T19:21:29.157 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.157 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.158 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.158 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.158 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.158 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.158 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.158 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.158 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.158 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.158 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.158 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.158 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption3": { 2024-03-11T19:21:29.158 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption3", 2024-03-11T19:21:29.158 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:29.158 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.158 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.159 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.159 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.159 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.159 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.159 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.159 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.159 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.159 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.159 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.159 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption4": { 2024-03-11T19:21:29.159 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption4", 2024-03-11T19:21:29.159 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.159 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.159 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.159 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.159 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.159 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.159 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.160 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.160 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.160 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.160 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.160 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.160 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption5": { 2024-03-11T19:21:29.160 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption5", 2024-03-11T19:21:29.160 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.160 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.160 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.160 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.160 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.160 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.160 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.160 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.160 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.160 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.160 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.161 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.161 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption6": { 2024-03-11T19:21:29.161 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption6", 2024-03-11T19:21:29.161 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.161 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.161 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.161 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.161 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.161 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.161 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.161 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.161 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.161 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.161 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.162 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.162 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption7": { 2024-03-11T19:21:29.162 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption7", 2024-03-11T19:21:29.162 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.162 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.162 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.162 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.162 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-11T19:21:29.162 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "42", 2024-03-11T19:21:29.162 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.162 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.162 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.162 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.163 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.163 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.163 INFO:teuthology.orchestra.run.smithi029.stdout: "testkey": { 2024-03-11T19:21:29.163 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testkey", 2024-03-11T19:21:29.163 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.163 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.163 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.163 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.163 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.163 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.163 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.163 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.163 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.163 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.164 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.164 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.164 INFO:teuthology.orchestra.run.smithi029.stdout: "testlkey": { 2024-03-11T19:21:29.164 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testlkey", 2024-03-11T19:21:29.164 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.164 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.164 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.164 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.164 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.164 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.164 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.164 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.164 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.164 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.165 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.165 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.165 INFO:teuthology.orchestra.run.smithi029.stdout: "testnewline": { 2024-03-11T19:21:29.165 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testnewline", 2024-03-11T19:21:29.165 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.165 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.165 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.165 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.165 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.165 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.165 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.165 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.165 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.165 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.165 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.166 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.166 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.166 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.166 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.166 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snap_schedule", 2024-03-11T19:21:29.166 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.166 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.166 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.166 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_m_granularity": { 2024-03-11T19:21:29.166 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_m_granularity", 2024-03-11T19:21:29.166 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.166 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.166 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.166 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.166 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.167 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.167 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.167 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow minute scheduled snapshots", 2024-03-11T19:21:29.167 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.167 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.167 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.167 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.167 INFO:teuthology.orchestra.run.smithi029.stdout: "dump_on_update": { 2024-03-11T19:21:29.167 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "dump_on_update", 2024-03-11T19:21:29.167 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.167 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.167 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.167 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.167 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.168 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.168 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.168 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "dump database to debug log on update", 2024-03-11T19:21:29.168 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.168 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.168 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.168 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.168 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.168 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.168 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.168 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.169 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.169 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.169 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.169 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.169 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.169 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.169 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.169 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.169 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.169 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.169 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.169 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.169 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.169 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.170 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.170 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.170 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.170 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.170 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.170 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.170 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.170 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.170 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.170 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.170 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.170 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.170 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.170 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.170 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.171 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.171 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.171 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.171 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.171 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.171 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.171 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.171 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.171 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.171 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.171 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.171 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.171 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.171 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.172 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.172 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.172 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.172 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.172 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.172 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.172 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.172 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.172 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.172 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.172 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.172 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.172 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.172 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.172 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.173 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.173 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.173 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.173 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.173 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.173 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.173 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.173 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.173 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.173 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.173 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.173 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "stats", 2024-03-11T19:21:29.173 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.173 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.173 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.174 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.174 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.174 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.174 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.174 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.174 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.174 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.174 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.174 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.174 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.174 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.174 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.174 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.175 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.175 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.175 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.175 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.175 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.175 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.175 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.175 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.175 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.175 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.175 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.175 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.175 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.175 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.175 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.175 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.175 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.176 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.176 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.176 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.176 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.176 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.176 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.176 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.176 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.176 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.176 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.176 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.176 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.176 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.176 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.176 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.176 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.176 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.177 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.177 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.177 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.177 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.177 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.177 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.177 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.177 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.177 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.177 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.177 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.177 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.177 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.177 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.177 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.177 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.178 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.178 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.178 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.178 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.178 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.178 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.178 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.178 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.178 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.178 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.178 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "status", 2024-03-11T19:21:29.178 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.178 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.178 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.178 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.179 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.179 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.179 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.179 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.179 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.179 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.179 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.179 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.179 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.179 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.179 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.179 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.179 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.179 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.179 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.179 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.180 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.180 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.180 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.180 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.180 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.180 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.180 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.180 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.180 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.180 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.180 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.180 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.180 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.180 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.180 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.180 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.180 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.181 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.181 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.181 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.181 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.181 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.181 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.181 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.181 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.181 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.181 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.181 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.181 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.181 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.181 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.181 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.182 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.182 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.182 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.182 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.182 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.182 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.182 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.182 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.182 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.182 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.182 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.182 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.182 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.182 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.183 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.183 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.183 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.183 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.183 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.183 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.183 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.183 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.183 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.183 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.183 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "telegraf", 2024-03-11T19:21:29.183 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.183 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.183 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.183 INFO:teuthology.orchestra.run.smithi029.stdout: "address": { 2024-03-11T19:21:29.183 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "address", 2024-03-11T19:21:29.183 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.184 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.184 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.184 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-11T19:21:29.184 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.184 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.184 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.184 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.184 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.184 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.184 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.184 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.184 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-11T19:21:29.184 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-11T19:21:29.184 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.184 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.185 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.185 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15", 2024-03-11T19:21:29.185 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.185 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.185 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.185 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.185 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.185 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.185 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.185 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.185 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.185 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.185 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.185 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.186 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.186 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.186 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.186 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.186 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.186 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.186 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.186 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.186 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.186 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.186 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.186 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.186 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.186 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.186 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.187 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.187 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.187 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.187 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.187 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.187 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.187 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.187 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.187 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.187 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.187 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.187 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.187 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.187 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.187 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.187 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.187 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.188 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.188 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.188 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.188 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.188 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.188 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.188 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.188 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.188 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.188 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.188 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.188 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.188 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.188 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.188 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.188 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.189 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.189 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.189 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.189 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.189 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.189 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.189 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.189 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.189 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.189 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.189 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.189 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.189 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.189 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.189 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.189 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.190 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.190 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.190 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.190 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.190 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.190 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "telemetry", 2024-03-11T19:21:29.190 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.190 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.190 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.190 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_basic": { 2024-03-11T19:21:29.190 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_basic", 2024-03-11T19:21:29.190 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.190 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.190 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.190 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.190 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.191 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.191 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.191 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-11T19:21:29.191 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.191 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.191 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.191 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.191 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_crash": { 2024-03-11T19:21:29.191 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_crash", 2024-03-11T19:21:29.191 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.191 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.191 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.191 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.191 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.192 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.192 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.192 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-11T19:21:29.192 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.192 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.192 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.192 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.192 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_device": { 2024-03-11T19:21:29.192 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_device", 2024-03-11T19:21:29.192 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.192 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.192 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.192 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.192 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.192 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.193 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.193 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-11T19:21:29.193 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.193 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.193 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.193 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.193 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_ident": { 2024-03-11T19:21:29.193 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_ident", 2024-03-11T19:21:29.193 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.193 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.193 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.193 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.193 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.193 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.193 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.194 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-11T19:21:29.194 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.194 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.194 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.194 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.194 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_perf": { 2024-03-11T19:21:29.194 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_perf", 2024-03-11T19:21:29.194 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.194 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.194 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.194 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.194 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.194 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.195 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.195 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-11T19:21:29.195 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.195 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.195 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.195 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.195 INFO:teuthology.orchestra.run.smithi029.stdout: "contact": { 2024-03-11T19:21:29.195 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "contact", 2024-03-11T19:21:29.195 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.195 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.195 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.195 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.195 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.196 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.196 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.196 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.196 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.196 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.196 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.196 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.196 INFO:teuthology.orchestra.run.smithi029.stdout: "description": { 2024-03-11T19:21:29.196 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "description", 2024-03-11T19:21:29.196 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.196 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.196 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.196 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.196 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.196 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.196 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.197 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.197 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.197 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.197 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.197 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.197 INFO:teuthology.orchestra.run.smithi029.stdout: "device_url": { 2024-03-11T19:21:29.197 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_url", 2024-03-11T19:21:29.197 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.197 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.197 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.197 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-11T19:21:29.197 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.197 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.198 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.198 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.198 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.198 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.198 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.198 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.198 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": { 2024-03-11T19:21:29.198 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enabled", 2024-03-11T19:21:29.198 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.198 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.198 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.198 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.198 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.198 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.199 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.199 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.199 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.199 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.199 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.199 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.199 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-11T19:21:29.199 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-11T19:21:29.199 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.199 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.199 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.199 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "24", 2024-03-11T19:21:29.199 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "8", 2024-03-11T19:21:29.199 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.199 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.200 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.200 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.200 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.200 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.200 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.200 INFO:teuthology.orchestra.run.smithi029.stdout: "last_opt_revision": { 2024-03-11T19:21:29.200 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "last_opt_revision", 2024-03-11T19:21:29.200 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.200 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.200 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.200 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1", 2024-03-11T19:21:29.200 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.200 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.200 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.200 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.200 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "leaderboard": { 2024-03-11T19:21:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "leaderboard", 2024-03-11T19:21:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.201 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "leaderboard_description": { 2024-03-11T19:21:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "leaderboard_description", 2024-03-11T19:21:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.202 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.203 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.203 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.203 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.203 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.203 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.203 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.203 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.203 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.203 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.203 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.203 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.203 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.203 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.203 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.203 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.203 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.204 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.204 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.204 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.204 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.204 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.204 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.204 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.204 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.204 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.204 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.204 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.204 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.204 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.204 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.204 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.204 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.205 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.205 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.205 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.205 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.205 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.205 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.205 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.205 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.205 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.205 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.205 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.205 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.205 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.205 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.205 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.206 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.206 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.206 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.206 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.206 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.206 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.206 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.206 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.206 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.206 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.206 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.206 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.206 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.206 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.207 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.207 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.207 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.207 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.207 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.207 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.207 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.207 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.207 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.207 INFO:teuthology.orchestra.run.smithi029.stdout: "organization": { 2024-03-11T19:21:29.207 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "organization", 2024-03-11T19:21:29.207 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.207 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.207 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.207 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.207 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.207 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.208 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.208 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.208 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.208 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.208 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.208 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.208 INFO:teuthology.orchestra.run.smithi029.stdout: "proxy": { 2024-03-11T19:21:29.208 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "proxy", 2024-03-11T19:21:29.208 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.208 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.208 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.208 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.208 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.208 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.208 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.208 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.208 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.209 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.209 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.209 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.209 INFO:teuthology.orchestra.run.smithi029.stdout: "url": { 2024-03-11T19:21:29.209 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "url", 2024-03-11T19:21:29.209 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.209 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.209 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.209 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-11T19:21:29.209 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.209 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.209 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.209 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.209 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.209 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.210 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.210 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.210 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.210 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.210 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.210 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "test_orchestrator", 2024-03-11T19:21:29.210 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.210 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.210 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.210 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.210 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.210 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.210 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.210 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.210 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.211 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.211 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.211 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.211 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.211 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.211 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.211 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.211 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.211 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.211 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.211 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.211 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.211 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.211 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.211 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.212 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.212 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.212 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.212 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.212 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.212 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.212 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.212 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.212 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.212 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.212 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.212 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.212 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.212 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.212 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.212 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.213 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.213 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.213 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.213 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.213 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.213 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.213 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.213 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.213 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.213 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.213 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.213 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.213 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.213 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.213 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.214 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.214 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.214 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.214 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.214 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.214 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.214 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.214 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.214 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.214 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.214 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.214 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.214 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.214 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.215 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.215 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.215 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.215 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.215 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.215 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.215 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.215 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "volumes", 2024-03-11T19:21:29.215 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.215 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.215 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.215 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.215 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.215 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.216 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.216 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.216 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.216 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.216 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.216 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.216 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.216 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.216 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.216 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.216 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.216 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.216 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.216 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.216 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.217 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.217 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.217 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.217 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.217 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.217 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.217 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.217 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.217 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.217 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.217 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.217 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.217 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.217 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.218 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.218 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.218 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.218 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.218 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.218 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.218 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.218 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.218 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.218 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.218 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.218 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.218 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.218 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.219 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.219 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.219 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.219 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.219 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.219 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.219 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.219 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.219 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.219 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.219 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.219 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.220 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.220 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.220 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.220 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.220 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.220 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.220 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.220 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.220 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.220 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.220 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.220 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.220 INFO:teuthology.orchestra.run.smithi029.stdout: "max_concurrent_clones": { 2024-03-11T19:21:29.220 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_concurrent_clones", 2024-03-11T19:21:29.221 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.221 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.221 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.221 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "4", 2024-03-11T19:21:29.221 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.221 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.221 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.221 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-11T19:21:29.221 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.221 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.221 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.221 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.221 INFO:teuthology.orchestra.run.smithi029.stdout: "periodic_async_work": { 2024-03-11T19:21:29.221 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "periodic_async_work", 2024-03-11T19:21:29.221 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.221 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.222 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.222 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.222 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.222 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.222 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.222 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Periodically check for async work", 2024-03-11T19:21:29.222 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.222 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.222 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.222 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.222 INFO:teuthology.orchestra.run.smithi029.stdout: "snapshot_clone_delay": { 2024-03-11T19:21:29.222 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snapshot_clone_delay", 2024-03-11T19:21:29.222 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.222 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.223 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.223 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-11T19:21:29.223 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.223 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.223 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.223 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-11T19:21:29.223 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.223 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.223 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.223 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.223 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.223 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.223 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.224 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix", 2024-03-11T19:21:29.224 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.224 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.224 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.224 INFO:teuthology.orchestra.run.smithi029.stdout: "discovery_interval": { 2024-03-11T19:21:29.224 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "discovery_interval", 2024-03-11T19:21:29.224 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-11T19:21:29.224 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.224 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.224 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "100", 2024-03-11T19:21:29.224 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.224 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.224 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.224 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.224 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.224 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.224 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.225 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.225 INFO:teuthology.orchestra.run.smithi029.stdout: "identifier": { 2024-03-11T19:21:29.225 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "identifier", 2024-03-11T19:21:29.225 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.225 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.225 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.225 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.225 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.225 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.225 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.225 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.225 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.225 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.225 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.225 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.226 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-11T19:21:29.226 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-11T19:21:29.226 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.226 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.226 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.226 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-11T19:21:29.226 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.226 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.226 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.226 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.226 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.226 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.226 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.226 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.226 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.226 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.227 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.227 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.227 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.227 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.227 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.227 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.227 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.227 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.227 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.227 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.227 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.227 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.227 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.227 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.228 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.228 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.228 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.228 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.228 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.228 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.228 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.228 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.228 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.228 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.228 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.228 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.228 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.228 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.228 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.229 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.229 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.229 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.229 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.229 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.229 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.229 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.229 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.229 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.229 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.229 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.229 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.229 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.229 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.229 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.229 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.229 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.230 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.230 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.230 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.230 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.230 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.230 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.230 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.230 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.230 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.230 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.230 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.230 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.230 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.230 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.230 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.231 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.231 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.231 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.231 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.231 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.231 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.231 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.231 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_host": { 2024-03-11T19:21:29.231 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_host", 2024-03-11T19:21:29.231 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.231 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.231 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.231 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.231 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.231 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.232 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.232 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.232 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.232 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.232 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.232 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.232 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_port": { 2024-03-11T19:21:29.232 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_port", 2024-03-11T19:21:29.232 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.232 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.232 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.232 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10051", 2024-03-11T19:21:29.232 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.232 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.232 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.233 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.233 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.233 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.233 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.233 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.233 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_sender": { 2024-03-11T19:21:29.233 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_sender", 2024-03-11T19:21:29.233 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.233 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.233 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.233 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-11T19:21:29.233 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.233 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.233 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.233 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.234 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.234 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.234 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.234 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.234 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.234 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.234 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-11T19:21:29.234 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.234 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.234 INFO:teuthology.orchestra.run.smithi029.stdout: "modules": [ 2024-03-11T19:21:29.234 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm", 2024-03-11T19:21:29.234 INFO:teuthology.orchestra.run.smithi029.stdout: "dashboard", 2024-03-11T19:21:29.234 INFO:teuthology.orchestra.run.smithi029.stdout: "iostat", 2024-03-11T19:21:29.234 INFO:teuthology.orchestra.run.smithi029.stdout: "nfs", 2024-03-11T19:21:29.234 INFO:teuthology.orchestra.run.smithi029.stdout: "restful" 2024-03-11T19:21:29.235 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.235 INFO:teuthology.orchestra.run.smithi029.stdout: "available_modules": [ 2024-03-11T19:21:29.235 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.235 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "alerts", 2024-03-11T19:21:29.235 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.235 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.235 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.235 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-11T19:21:29.235 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-11T19:21:29.235 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.235 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.235 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.235 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-11T19:21:29.235 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.235 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.235 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.235 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How frequently to reexamine health status", 2024-03-11T19:21:29.236 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.236 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.236 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.236 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.236 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.236 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.236 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.236 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.236 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.236 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.236 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.236 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.236 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.236 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.236 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.237 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.237 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.237 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.237 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.237 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.237 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.237 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.237 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.237 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.237 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.237 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.237 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.237 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.237 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.237 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.238 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.238 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.238 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.238 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.238 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.238 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.238 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.238 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.238 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.238 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.238 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.238 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.238 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.238 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.238 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.239 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_destination": { 2024-03-11T19:21:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_destination", 2024-03-11T19:21:29.240 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.241 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.241 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.241 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.241 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.241 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.241 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.241 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Email address to send alerts to", 2024-03-11T19:21:29.241 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.241 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.241 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.241 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.241 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_from_name": { 2024-03-11T19:21:29.241 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_from_name", 2024-03-11T19:21:29.241 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.241 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.242 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.242 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "Ceph", 2024-03-11T19:21:29.242 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.242 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.242 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.242 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Email From: name", 2024-03-11T19:21:29.242 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.242 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.242 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.242 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.242 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_host": { 2024-03-11T19:21:29.242 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_host", 2024-03-11T19:21:29.242 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.242 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.242 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP server", 2024-03-11T19:21:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_password": { 2024-03-11T19:21:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_password", 2024-03-11T19:21:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Password to authenticate with", 2024-03-11T19:21:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_port": { 2024-03-11T19:21:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_port", 2024-03-11T19:21:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "465", 2024-03-11T19:21:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.245 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.245 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP port", 2024-03-11T19:21:29.245 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.245 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.245 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.245 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.245 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_sender": { 2024-03-11T19:21:29.245 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_sender", 2024-03-11T19:21:29.245 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.245 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.245 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.245 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.245 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.245 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.246 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.246 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP envelope sender", 2024-03-11T19:21:29.246 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.246 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.246 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.246 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.246 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_ssl": { 2024-03-11T19:21:29.246 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_ssl", 2024-03-11T19:21:29.246 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.246 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.246 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.246 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.246 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.247 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.247 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.247 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-11T19:21:29.247 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.247 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.247 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.247 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_user": { 2024-03-11T19:21:29.247 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_user", 2024-03-11T19:21:29.247 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.247 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.247 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.247 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.247 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.247 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.247 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.248 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "User to authenticate as", 2024-03-11T19:21:29.248 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.248 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.248 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.248 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.248 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.248 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.248 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.248 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "balancer", 2024-03-11T19:21:29.248 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.248 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.248 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.248 INFO:teuthology.orchestra.run.smithi029.stdout: "active": { 2024-03-11T19:21:29.248 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "active", 2024-03-11T19:21:29.248 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.248 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.249 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.249 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.249 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.249 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.249 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.249 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically balance PGs across cluster", 2024-03-11T19:21:29.249 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.249 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.249 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.249 INFO:teuthology.orchestra.run.smithi029.stdout: "begin_time": { 2024-03-11T19:21:29.249 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "begin_time", 2024-03-11T19:21:29.249 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.249 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.249 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.249 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0000", 2024-03-11T19:21:29.249 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.250 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.250 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.250 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "beginning time of day to automatically balance", 2024-03-11T19:21:29.250 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-11T19:21:29.250 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.250 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.250 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.250 INFO:teuthology.orchestra.run.smithi029.stdout: "begin_weekday": { 2024-03-11T19:21:29.250 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "begin_weekday", 2024-03-11T19:21:29.250 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-11T19:21:29.250 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.250 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.250 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-11T19:21:29.250 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0", 2024-03-11T19:21:29.250 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "6", 2024-03-11T19:21:29.251 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.251 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-11T19:21:29.251 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-11T19:21:29.251 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.251 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.251 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.251 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_max_iterations": { 2024-03-11T19:21:29.251 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_max_iterations", 2024-03-11T19:21:29.251 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-11T19:21:29.251 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.251 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.251 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "25", 2024-03-11T19:21:29.251 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-11T19:21:29.251 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "250", 2024-03-11T19:21:29.251 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.251 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-11T19:21:29.252 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.252 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.252 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.252 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.252 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_metrics": { 2024-03-11T19:21:29.252 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_metrics", 2024-03-11T19:21:29.252 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.252 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.252 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.252 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "pgs,objects,bytes", 2024-03-11T19:21:29.252 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.252 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.252 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.252 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-11T19:21:29.252 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-11T19:21:29.252 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.253 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.253 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.253 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_step": { 2024-03-11T19:21:29.253 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_step", 2024-03-11T19:21:29.253 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:29.253 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.253 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.253 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0.5", 2024-03-11T19:21:29.253 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0.001", 2024-03-11T19:21:29.253 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "0.999", 2024-03-11T19:21:29.253 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.253 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "aggressiveness of optimization", 2024-03-11T19:21:29.253 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-11T19:21:29.253 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.253 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.254 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.254 INFO:teuthology.orchestra.run.smithi029.stdout: "end_time": { 2024-03-11T19:21:29.254 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "end_time", 2024-03-11T19:21:29.254 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.254 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.254 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.254 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "2359", 2024-03-11T19:21:29.254 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.254 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.254 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.254 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "ending time of day to automatically balance", 2024-03-11T19:21:29.254 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-11T19:21:29.254 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.254 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.255 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.255 INFO:teuthology.orchestra.run.smithi029.stdout: "end_weekday": { 2024-03-11T19:21:29.255 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "end_weekday", 2024-03-11T19:21:29.255 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-11T19:21:29.255 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.255 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.255 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-11T19:21:29.255 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0", 2024-03-11T19:21:29.255 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "6", 2024-03-11T19:21:29.255 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.255 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-11T19:21:29.255 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-11T19:21:29.255 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.255 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.255 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.255 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.256 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.256 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.256 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.256 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.256 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.256 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.256 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.256 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.256 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.256 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.256 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.256 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.256 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.256 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.256 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.257 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.257 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.257 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.257 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.257 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.257 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.257 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.257 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.257 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.257 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.257 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.257 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.257 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.257 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.257 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.258 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.258 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.258 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.258 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.258 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.258 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.258 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.258 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.258 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.258 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.258 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.258 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.258 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.258 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.259 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.259 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.259 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.259 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.259 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.259 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.259 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.259 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.259 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.259 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.259 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.259 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.259 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.259 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.259 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.259 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.259 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.259 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.260 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.260 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.260 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.260 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.260 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.260 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.260 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.260 INFO:teuthology.orchestra.run.smithi029.stdout: "min_score": { 2024-03-11T19:21:29.260 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "min_score", 2024-03-11T19:21:29.260 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:29.260 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.261 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.261 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-11T19:21:29.261 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.261 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.261 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.261 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-11T19:21:29.261 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.261 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.261 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.261 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.261 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": { 2024-03-11T19:21:29.261 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mode", 2024-03-11T19:21:29.261 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.261 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.262 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.262 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "upmap", 2024-03-11T19:21:29.262 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.262 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.262 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.262 INFO:teuthology.orchestra.run.smithi029.stdout: "crush-compat", 2024-03-11T19:21:29.262 INFO:teuthology.orchestra.run.smithi029.stdout: "none", 2024-03-11T19:21:29.262 INFO:teuthology.orchestra.run.smithi029.stdout: "read", 2024-03-11T19:21:29.262 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap", 2024-03-11T19:21:29.262 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap-read" 2024-03-11T19:21:29.262 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.262 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Balancer mode", 2024-03-11T19:21:29.262 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.262 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.262 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.263 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.263 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_ids": { 2024-03-11T19:21:29.263 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pool_ids", 2024-03-11T19:21:29.263 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.263 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.263 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.263 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.263 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.263 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.263 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.263 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-11T19:21:29.263 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.263 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.263 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.263 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.263 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-11T19:21:29.263 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-11T19:21:29.263 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.264 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.264 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.264 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-11T19:21:29.264 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.264 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.264 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.264 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-11T19:21:29.264 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.264 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.264 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.264 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.264 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap_max_deviation": { 2024-03-11T19:21:29.264 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "upmap_max_deviation", 2024-03-11T19:21:29.264 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.264 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.264 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.264 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-11T19:21:29.265 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-11T19:21:29.265 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.265 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.265 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-11T19:21:29.265 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-11T19:21:29.265 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.265 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.265 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.265 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap_max_optimizations": { 2024-03-11T19:21:29.265 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "upmap_max_optimizations", 2024-03-11T19:21:29.265 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-11T19:21:29.265 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.265 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.265 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:29.265 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.265 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.265 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.266 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-11T19:21:29.266 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.266 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.266 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.266 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.266 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.266 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.266 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.266 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephadm", 2024-03-11T19:21:29.266 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.266 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.266 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.266 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_down_multiplier": { 2024-03-11T19:21:29.266 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_down_multiplier", 2024-03-11T19:21:29.266 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:29.267 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.267 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.267 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3.0", 2024-03-11T19:21:29.267 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.267 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.267 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.267 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-11T19:21:29.267 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.267 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.267 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.267 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.267 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_refresh_rate": { 2024-03-11T19:21:29.267 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_refresh_rate", 2024-03-11T19:21:29.267 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.268 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.268 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.268 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "20", 2024-03-11T19:21:29.268 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.268 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.268 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.268 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-11T19:21:29.268 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.268 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.268 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.268 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.268 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_starting_port": { 2024-03-11T19:21:29.268 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_starting_port", 2024-03-11T19:21:29.268 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.268 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.268 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.268 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "4721", 2024-03-11T19:21:29.269 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.269 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.269 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.269 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-11T19:21:29.269 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.269 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.269 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.269 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.269 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_ptrace": { 2024-03-11T19:21:29.269 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_ptrace", 2024-03-11T19:21:29.269 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.269 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.269 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.269 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.269 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.269 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.269 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.270 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-11T19:21:29.270 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-11T19:21:29.270 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.270 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.270 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.270 INFO:teuthology.orchestra.run.smithi029.stdout: "autotune_interval": { 2024-03-11T19:21:29.270 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "autotune_interval", 2024-03-11T19:21:29.270 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.270 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.270 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.270 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-11T19:21:29.270 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.270 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.270 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.271 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-11T19:21:29.271 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.271 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.271 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.271 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.271 INFO:teuthology.orchestra.run.smithi029.stdout: "autotune_memory_target_ratio": { 2024-03-11T19:21:29.271 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "autotune_memory_target_ratio", 2024-03-11T19:21:29.271 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:29.271 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.271 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.271 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0.7", 2024-03-11T19:21:29.271 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.271 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.271 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.272 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-11T19:21:29.272 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.272 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.272 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.272 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.272 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm_log_destination": { 2024-03-11T19:21:29.272 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephadm_log_destination", 2024-03-11T19:21:29.272 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.272 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.272 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.272 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.272 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.272 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.273 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.273 INFO:teuthology.orchestra.run.smithi029.stdout: "file", 2024-03-11T19:21:29.273 INFO:teuthology.orchestra.run.smithi029.stdout: "file,syslog", 2024-03-11T19:21:29.273 INFO:teuthology.orchestra.run.smithi029.stdout: "syslog" 2024-03-11T19:21:29.273 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.273 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-11T19:21:29.273 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.273 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.273 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.273 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.273 INFO:teuthology.orchestra.run.smithi029.stdout: "cgroups_split": { 2024-03-11T19:21:29.273 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cgroups_split", 2024-03-11T19:21:29.273 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.274 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.274 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.274 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.274 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.274 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.274 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.274 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-11T19:21:29.274 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.274 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.274 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.274 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.274 INFO:teuthology.orchestra.run.smithi029.stdout: "config_checks_enabled": { 2024-03-11T19:21:29.274 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_checks_enabled", 2024-03-11T19:21:29.274 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.275 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.275 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.275 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.275 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.275 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.275 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.275 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-11T19:21:29.275 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.275 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.275 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.275 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.275 INFO:teuthology.orchestra.run.smithi029.stdout: "config_dashboard": { 2024-03-11T19:21:29.275 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_dashboard", 2024-03-11T19:21:29.275 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.276 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.276 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.276 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.276 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.276 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.276 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.276 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-11T19:21:29.276 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.276 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.276 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.276 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.276 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_alertmanager": { 2024-03-11T19:21:29.276 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_alertmanager", 2024-03-11T19:21:29.276 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.276 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.277 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.277 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-11T19:21:29.277 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.277 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.277 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.277 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-11T19:21:29.277 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.277 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.277 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.277 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.277 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_base": { 2024-03-11T19:21:29.277 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_base", 2024-03-11T19:21:29.277 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.277 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.278 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.278 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-11T19:21:29.278 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.278 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.278 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.278 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Container image name, without the tag", 2024-03-11T19:21:29.278 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.278 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.278 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.278 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.278 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_elasticsearch": { 2024-03-11T19:21:29.278 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_elasticsearch", 2024-03-11T19:21:29.278 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.279 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.279 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.279 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-11T19:21:29.279 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.279 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.279 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.279 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "elasticsearch container image", 2024-03-11T19:21:29.279 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.279 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.279 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.280 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_grafana": { 2024-03-11T19:21:29.280 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_grafana", 2024-03-11T19:21:29.280 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.280 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.280 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.280 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-11T19:21:29.280 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.280 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.280 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.280 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-11T19:21:29.280 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.280 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.280 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.280 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.281 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_haproxy": { 2024-03-11T19:21:29.281 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_haproxy", 2024-03-11T19:21:29.281 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.281 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.281 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.281 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-11T19:21:29.281 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.281 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.281 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.281 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "HAproxy container image", 2024-03-11T19:21:29.281 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.281 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.281 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.282 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.282 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_agent": { 2024-03-11T19:21:29.282 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_agent", 2024-03-11T19:21:29.282 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.282 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.282 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.282 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-11T19:21:29.282 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.282 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.282 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.282 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger agent container image", 2024-03-11T19:21:29.282 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.282 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.283 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.283 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.283 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_collector": { 2024-03-11T19:21:29.283 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_collector", 2024-03-11T19:21:29.283 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.283 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.283 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.283 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-11T19:21:29.283 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.283 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.283 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.283 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger collector container image", 2024-03-11T19:21:29.283 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.283 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.284 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.284 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.284 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_query": { 2024-03-11T19:21:29.284 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_query", 2024-03-11T19:21:29.284 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.284 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.284 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.284 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-11T19:21:29.284 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.284 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.284 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.284 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger query container image", 2024-03-11T19:21:29.284 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.284 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.285 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.285 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.285 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_keepalived": { 2024-03-11T19:21:29.285 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_keepalived", 2024-03-11T19:21:29.285 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.285 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.285 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.285 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-11T19:21:29.285 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.285 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.285 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.285 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Keepalived container image", 2024-03-11T19:21:29.285 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.285 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.285 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.286 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.286 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_loki": { 2024-03-11T19:21:29.286 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_loki", 2024-03-11T19:21:29.286 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.286 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.286 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.286 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-11T19:21:29.286 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.286 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.286 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.286 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Loki container image", 2024-03-11T19:21:29.286 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.286 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.286 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.287 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.287 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_node_exporter": { 2024-03-11T19:21:29.287 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_node_exporter", 2024-03-11T19:21:29.287 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.287 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.287 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.287 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-11T19:21:29.287 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.287 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.287 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.287 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-11T19:21:29.287 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.287 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.287 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.287 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.287 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_nvmeof": { 2024-03-11T19:21:29.288 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_nvmeof", 2024-03-11T19:21:29.288 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.288 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.288 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.288 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-11T19:21:29.288 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.288 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.288 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.288 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Nvme-of container image", 2024-03-11T19:21:29.288 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.288 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.288 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.288 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.288 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_prometheus": { 2024-03-11T19:21:29.288 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_prometheus", 2024-03-11T19:21:29.289 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.289 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.289 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.289 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-11T19:21:29.289 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.289 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.289 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.289 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-11T19:21:29.289 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.289 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.289 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.289 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.289 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_promtail": { 2024-03-11T19:21:29.289 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_promtail", 2024-03-11T19:21:29.290 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.290 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.290 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.290 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-11T19:21:29.290 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.290 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.290 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.290 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Promtail container image", 2024-03-11T19:21:29.290 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.290 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.290 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.290 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.290 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_snmp_gateway": { 2024-03-11T19:21:29.290 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_snmp_gateway", 2024-03-11T19:21:29.291 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.291 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.291 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.291 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-11T19:21:29.291 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.291 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.291 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.291 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SNMP Gateway container image", 2024-03-11T19:21:29.291 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.291 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.291 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.291 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.291 INFO:teuthology.orchestra.run.smithi029.stdout: "container_init": { 2024-03-11T19:21:29.291 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_init", 2024-03-11T19:21:29.291 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.292 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.292 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.292 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.292 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.292 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.292 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.292 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Run podman/docker with `--init`", 2024-03-11T19:21:29.292 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.292 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.292 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.292 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.292 INFO:teuthology.orchestra.run.smithi029.stdout: "daemon_cache_timeout": { 2024-03-11T19:21:29.292 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "daemon_cache_timeout", 2024-03-11T19:21:29.292 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.292 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.292 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-11T19:21:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-11T19:21:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: "default_cephadm_command_timeout": { 2024-03-11T19:21:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default_cephadm_command_timeout", 2024-03-11T19:21:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "900", 2024-03-11T19:21:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.294 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.294 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-11T19:21:29.294 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.294 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.294 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.294 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.294 INFO:teuthology.orchestra.run.smithi029.stdout: "default_registry": { 2024-03-11T19:21:29.294 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default_registry", 2024-03-11T19:21:29.294 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.294 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.294 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.294 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io", 2024-03-11T19:21:29.294 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.294 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.294 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.294 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-11T19:21:29.295 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.295 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.295 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.295 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.295 INFO:teuthology.orchestra.run.smithi029.stdout: "device_cache_timeout": { 2024-03-11T19:21:29.295 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_cache_timeout", 2024-03-11T19:21:29.295 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.295 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.295 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.295 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1800", 2024-03-11T19:21:29.295 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.295 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.295 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.295 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache device inventory", 2024-03-11T19:21:29.295 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.296 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.296 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.296 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.296 INFO:teuthology.orchestra.run.smithi029.stdout: "device_enhanced_scan": { 2024-03-11T19:21:29.296 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_enhanced_scan", 2024-03-11T19:21:29.296 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.296 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.296 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.296 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.296 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.296 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.296 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.296 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-11T19:21:29.296 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.296 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.296 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.296 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.297 INFO:teuthology.orchestra.run.smithi029.stdout: "facts_cache_timeout": { 2024-03-11T19:21:29.297 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "facts_cache_timeout", 2024-03-11T19:21:29.297 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.297 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.297 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.297 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-11T19:21:29.297 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.297 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.297 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.297 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache host facts data", 2024-03-11T19:21:29.297 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.297 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.297 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.297 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.297 INFO:teuthology.orchestra.run.smithi029.stdout: "host_check_interval": { 2024-03-11T19:21:29.297 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "host_check_interval", 2024-03-11T19:21:29.297 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.298 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.298 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.298 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-11T19:21:29.298 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.298 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.298 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.298 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to perform a host check", 2024-03-11T19:21:29.298 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.298 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.298 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.298 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.298 INFO:teuthology.orchestra.run.smithi029.stdout: "hw_monitoring": { 2024-03-11T19:21:29.298 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "hw_monitoring", 2024-03-11T19:21:29.298 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.298 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.299 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.299 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.299 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.299 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.299 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.299 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-11T19:21:29.299 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.299 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.299 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.299 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.299 INFO:teuthology.orchestra.run.smithi029.stdout: "inventory_list_all": { 2024-03-11T19:21:29.299 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "inventory_list_all", 2024-03-11T19:21:29.299 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.299 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.299 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.300 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.300 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.300 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.300 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.300 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-11T19:21:29.300 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.300 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.300 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.300 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.300 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.300 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.300 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.300 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.300 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.300 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.300 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.301 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.301 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.301 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.301 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.301 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.301 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.301 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.301 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.301 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.301 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.301 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.301 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.301 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.301 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.301 INFO:teuthology.orchestra.run.smithi029.stdout: "log_refresh_metadata": { 2024-03-11T19:21:29.301 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_refresh_metadata", 2024-03-11T19:21:29.302 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.302 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.302 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.302 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.302 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.302 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.302 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.302 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-11T19:21:29.302 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.302 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.302 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.302 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.302 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.302 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.302 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.303 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.303 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.303 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.303 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.303 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.303 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.303 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-11T19:21:29.303 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.303 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.303 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.303 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.303 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.303 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.303 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.303 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.303 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.303 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.304 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.304 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.304 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.304 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.304 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.304 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.304 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.304 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.304 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.304 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.304 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.304 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.304 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.304 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.304 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.304 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.304 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.305 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.305 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.305 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.305 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.305 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.305 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.305 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.305 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.305 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.305 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.305 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.305 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.305 INFO:teuthology.orchestra.run.smithi029.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-11T19:21:29.305 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-11T19:21:29.305 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.305 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.305 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.306 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.306 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.306 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.306 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.306 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-11T19:21:29.306 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.306 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.306 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.306 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.306 INFO:teuthology.orchestra.run.smithi029.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-11T19:21:29.306 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-11T19:21:29.306 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.306 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.306 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.307 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "*", 2024-03-11T19:21:29.307 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.307 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.307 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.307 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-11T19:21:29.307 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.307 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.307 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.307 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.307 INFO:teuthology.orchestra.run.smithi029.stdout: "max_count_per_host": { 2024-03-11T19:21:29.307 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_count_per_host", 2024-03-11T19:21:29.307 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.307 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.307 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.308 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:29.308 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.308 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.308 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.308 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "max number of daemons per service per host", 2024-03-11T19:21:29.308 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.308 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.308 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.308 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.308 INFO:teuthology.orchestra.run.smithi029.stdout: "max_osd_draining_count": { 2024-03-11T19:21:29.308 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_osd_draining_count", 2024-03-11T19:21:29.308 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.308 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.308 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.308 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:29.308 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.309 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.309 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.309 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-11T19:21:29.309 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.309 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.309 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.309 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.309 INFO:teuthology.orchestra.run.smithi029.stdout: "migration_current": { 2024-03-11T19:21:29.309 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "migration_current", 2024-03-11T19:21:29.309 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.309 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.309 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.309 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.309 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.309 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.310 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.310 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "internal - do not modify", 2024-03-11T19:21:29.310 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.310 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.310 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.310 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.310 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": { 2024-03-11T19:21:29.310 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mode", 2024-03-11T19:21:29.310 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.310 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.310 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.310 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "root", 2024-03-11T19:21:29.310 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.311 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.311 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.311 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm-package", 2024-03-11T19:21:29.311 INFO:teuthology.orchestra.run.smithi029.stdout: "root" 2024-03-11T19:21:29.311 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.311 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "mode for remote execution of cephadm", 2024-03-11T19:21:29.311 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.311 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.311 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.311 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.311 INFO:teuthology.orchestra.run.smithi029.stdout: "oob_default_addr": { 2024-03-11T19:21:29.311 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "oob_default_addr", 2024-03-11T19:21:29.311 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.311 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.311 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.312 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "169.254.1.1", 2024-03-11T19:21:29.312 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.312 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.312 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.312 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-11T19:21:29.312 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.312 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.312 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.312 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.312 INFO:teuthology.orchestra.run.smithi029.stdout: "prometheus_alerts_path": { 2024-03-11T19:21:29.312 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prometheus_alerts_path", 2024-03-11T19:21:29.312 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.312 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.312 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.312 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-11T19:21:29.312 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.313 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.313 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.313 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-11T19:21:29.313 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.313 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.313 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.313 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.313 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_insecure": { 2024-03-11T19:21:29.313 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_insecure", 2024-03-11T19:21:29.313 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.313 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.313 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.313 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.313 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.313 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.313 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.314 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-11T19:21:29.314 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.314 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.314 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.314 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.314 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_password": { 2024-03-11T19:21:29.314 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_password", 2024-03-11T19:21:29.314 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.314 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.314 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.314 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.314 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.314 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.314 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.314 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-11T19:21:29.315 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.315 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.315 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.315 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.315 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_url": { 2024-03-11T19:21:29.315 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_url", 2024-03-11T19:21:29.315 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.315 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.315 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.315 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.315 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.315 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.315 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.315 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-11T19:21:29.315 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.315 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.316 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.316 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.316 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_username": { 2024-03-11T19:21:29.316 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_username", 2024-03-11T19:21:29.316 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.316 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.316 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.316 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.316 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.316 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.316 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.316 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-11T19:21:29.316 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.316 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.316 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.316 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.317 INFO:teuthology.orchestra.run.smithi029.stdout: "secure_monitoring_stack": { 2024-03-11T19:21:29.317 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "secure_monitoring_stack", 2024-03-11T19:21:29.317 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.317 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.317 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.317 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.317 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.317 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.317 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.317 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-11T19:21:29.317 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.317 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.317 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.317 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.317 INFO:teuthology.orchestra.run.smithi029.stdout: "service_discovery_port": { 2024-03-11T19:21:29.317 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "service_discovery_port", 2024-03-11T19:21:29.318 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.318 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.318 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.318 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8765", 2024-03-11T19:21:29.318 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.318 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.318 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.318 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "cephadm service discovery port", 2024-03-11T19:21:29.318 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.318 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.318 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.318 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.318 INFO:teuthology.orchestra.run.smithi029.stdout: "ssh_config_file": { 2024-03-11T19:21:29.318 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssh_config_file", 2024-03-11T19:21:29.318 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.318 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.319 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.319 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.319 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.319 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.319 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.319 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-11T19:21:29.319 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.319 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.319 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.319 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.319 INFO:teuthology.orchestra.run.smithi029.stdout: "use_agent": { 2024-03-11T19:21:29.319 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "use_agent", 2024-03-11T19:21:29.319 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.319 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.320 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.320 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.320 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.320 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.320 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.320 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-11T19:21:29.320 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.320 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.320 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.320 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.320 INFO:teuthology.orchestra.run.smithi029.stdout: "use_repo_digest": { 2024-03-11T19:21:29.320 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "use_repo_digest", 2024-03-11T19:21:29.320 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.320 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.320 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.321 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.321 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.321 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.321 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.321 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-11T19:21:29.321 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.321 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.321 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.321 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.321 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_failed_host_check": { 2024-03-11T19:21:29.321 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_failed_host_check", 2024-03-11T19:21:29.321 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.321 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.321 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.322 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.322 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.322 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.322 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.322 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if the host check fails", 2024-03-11T19:21:29.322 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.322 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.322 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.322 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.322 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_stray_daemons": { 2024-03-11T19:21:29.322 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_stray_daemons", 2024-03-11T19:21:29.322 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.322 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.323 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.323 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.323 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.323 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.323 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.323 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-11T19:21:29.323 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.323 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.323 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.323 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.323 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_stray_hosts": { 2024-03-11T19:21:29.323 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_stray_hosts", 2024-03-11T19:21:29.323 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.323 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.323 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.324 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.324 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.324 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.324 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.324 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-11T19:21:29.324 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.324 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.324 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.324 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.324 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.324 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.324 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.324 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crash", 2024-03-11T19:21:29.324 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.324 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.325 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.325 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.325 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.325 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.325 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.325 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.325 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.325 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.325 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.325 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.325 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.325 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.325 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.325 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.325 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.325 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.326 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.326 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.326 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.326 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.326 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.326 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.326 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.326 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.326 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.326 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.326 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.326 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.326 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.326 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.327 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.327 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.327 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.327 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.327 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.327 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.327 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.327 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.327 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.327 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.327 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.327 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.327 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.327 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.328 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.328 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.328 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.328 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.328 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.328 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.328 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.328 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.328 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.328 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.328 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.328 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.328 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.328 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.328 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.329 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.329 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.329 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.329 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.329 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.329 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.329 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.329 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.329 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.329 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.329 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.329 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.329 INFO:teuthology.orchestra.run.smithi029.stdout: "retain_interval": { 2024-03-11T19:21:29.329 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "retain_interval", 2024-03-11T19:21:29.329 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.329 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.329 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.330 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "31536000", 2024-03-11T19:21:29.330 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.330 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.330 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.330 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-11T19:21:29.330 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.330 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.330 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.330 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.330 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_recent_interval": { 2024-03-11T19:21:29.330 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_recent_interval", 2024-03-11T19:21:29.330 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.330 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.330 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.331 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1209600", 2024-03-11T19:21:29.331 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.331 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.331 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.331 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-11T19:21:29.331 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.331 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.331 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.331 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.331 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.331 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.331 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.331 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "dashboard", 2024-03-11T19:21:29.331 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.332 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.332 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.332 INFO:teuthology.orchestra.run.smithi029.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-11T19:21:29.332 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-11T19:21:29.332 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.332 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.332 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.332 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:29.332 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.332 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.332 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.332 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.332 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.332 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.332 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.333 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.333 INFO:teuthology.orchestra.run.smithi029.stdout: "ALERTMANAGER_API_HOST": { 2024-03-11T19:21:29.333 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-11T19:21:29.333 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.333 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.333 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.333 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.333 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.333 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.333 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.333 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.333 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.333 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.333 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.333 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.333 INFO:teuthology.orchestra.run.smithi029.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-11T19:21:29.334 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-11T19:21:29.334 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.334 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.334 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.334 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.334 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.334 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.334 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.334 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.334 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.334 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.334 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.334 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.334 INFO:teuthology.orchestra.run.smithi029.stdout: "AUDIT_API_ENABLED": { 2024-03-11T19:21:29.334 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "AUDIT_API_ENABLED", 2024-03-11T19:21:29.335 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.335 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.335 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.335 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.335 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.335 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.335 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.335 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.335 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.335 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.335 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.335 INFO:teuthology.orchestra.run.smithi029.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-11T19:21:29.335 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-11T19:21:29.335 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.335 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.335 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.336 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.336 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.336 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.336 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.336 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.336 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.336 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.336 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.336 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.336 INFO:teuthology.orchestra.run.smithi029.stdout: "ENABLE_BROWSABLE_API": { 2024-03-11T19:21:29.336 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-11T19:21:29.336 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.336 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.336 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.336 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.336 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.337 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.337 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.337 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.337 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.337 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.337 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.337 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-11T19:21:29.337 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-11T19:21:29.337 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.337 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.337 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.337 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.337 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.337 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.337 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.338 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.338 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.338 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.338 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.338 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.338 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-11T19:21:29.338 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-11T19:21:29.338 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.338 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.338 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.338 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.338 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.338 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.338 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.338 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.338 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.339 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.339 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.339 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.339 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-11T19:21:29.339 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-11T19:21:29.339 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.339 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.339 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.339 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.339 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.339 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.339 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.339 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.339 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.339 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.339 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.340 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.340 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-11T19:21:29.340 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-11T19:21:29.340 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.340 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.340 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.340 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.340 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.340 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.340 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.340 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.340 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.340 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.340 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.340 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.341 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-11T19:21:29.341 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-11T19:21:29.341 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.341 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.341 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.341 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.341 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.341 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.341 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.341 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.341 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.341 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.341 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.341 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.341 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-11T19:21:29.342 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-11T19:21:29.342 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.342 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.342 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.342 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.342 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.342 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.342 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.342 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.342 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.342 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.342 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.342 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.342 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-11T19:21:29.342 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-11T19:21:29.343 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.343 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.343 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.343 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.343 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.343 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.343 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.343 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.343 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.343 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.343 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.343 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.343 INFO:teuthology.orchestra.run.smithi029.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-11T19:21:29.343 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-11T19:21:29.344 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.344 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.344 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.344 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.344 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.344 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.344 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.344 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.344 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.344 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.344 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.344 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.344 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_PASSWORD": { 2024-03-11T19:21:29.344 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-11T19:21:29.344 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.345 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.345 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.345 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-03-11T19:21:29.345 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.345 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.345 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.345 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.345 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.345 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.345 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.345 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.345 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-11T19:21:29.345 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-11T19:21:29.346 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.346 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.346 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.346 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.346 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.346 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.346 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.346 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.346 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.346 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.346 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.346 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.347 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_URL": { 2024-03-11T19:21:29.347 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_URL", 2024-03-11T19:21:29.347 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.347 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.347 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.347 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.347 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.347 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.347 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.348 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.348 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.348 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.348 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.348 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.348 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_USERNAME": { 2024-03-11T19:21:29.348 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-11T19:21:29.348 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.348 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.348 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.349 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-03-11T19:21:29.349 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.349 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.349 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.349 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.349 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.349 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.349 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.349 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.349 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-11T19:21:29.349 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-11T19:21:29.349 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.349 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.349 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.349 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.349 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.350 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.350 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.350 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.350 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.350 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.350 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.350 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.350 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-11T19:21:29.350 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-11T19:21:29.350 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.350 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.350 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.350 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.350 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.351 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.351 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.351 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.351 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.351 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.351 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.351 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.351 INFO:teuthology.orchestra.run.smithi029.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-11T19:21:29.351 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-11T19:21:29.351 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.351 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.351 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.351 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.351 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.351 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.352 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.352 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.352 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.352 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.352 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.352 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.352 INFO:teuthology.orchestra.run.smithi029.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-11T19:21:29.352 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-11T19:21:29.352 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.352 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.352 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.352 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.352 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.353 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.353 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.353 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.353 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.353 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.353 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.353 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.353 INFO:teuthology.orchestra.run.smithi029.stdout: "PROMETHEUS_API_HOST": { 2024-03-11T19:21:29.353 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-11T19:21:29.353 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.353 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.353 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.353 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.353 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.353 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.353 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.354 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.354 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.354 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.354 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.354 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.354 INFO:teuthology.orchestra.run.smithi029.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-11T19:21:29.354 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-11T19:21:29.354 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.354 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.354 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.354 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.354 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.354 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.354 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.354 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.354 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.355 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.355 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.355 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.355 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-11T19:21:29.355 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-11T19:21:29.355 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.355 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.355 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.355 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.355 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.355 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.355 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.355 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.355 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.355 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.355 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.355 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.356 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-11T19:21:29.356 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-11T19:21:29.356 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.356 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.356 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.356 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.356 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.356 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.356 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.356 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.356 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.356 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.356 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.356 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.356 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-11T19:21:29.356 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-11T19:21:29.356 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.357 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.357 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.357 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.357 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.357 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.357 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.357 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.357 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.357 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.357 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.357 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.357 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-11T19:21:29.357 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-11T19:21:29.357 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.357 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.357 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.357 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.357 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.358 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.358 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.358 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.358 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.358 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.358 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.358 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.358 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-11T19:21:29.358 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-11T19:21:29.358 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.358 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.358 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.358 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.358 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.358 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.359 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.359 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.359 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.359 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.359 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.359 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.359 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-11T19:21:29.359 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-11T19:21:29.359 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.359 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.359 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.359 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.359 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.359 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.359 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.359 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.359 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.360 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.360 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.360 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.360 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-11T19:21:29.360 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-11T19:21:29.360 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.360 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.360 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.360 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.360 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.360 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.360 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.360 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.360 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.360 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.360 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.361 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.361 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_ENABLED": { 2024-03-11T19:21:29.361 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-11T19:21:29.361 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.361 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.361 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.361 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.361 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.361 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.361 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.361 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.361 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.361 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.361 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.361 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.362 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-11T19:21:29.362 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-11T19:21:29.362 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.362 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.362 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.362 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-11T19:21:29.362 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.362 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.362 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.362 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.362 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.362 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.363 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.363 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.363 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-11T19:21:29.363 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-11T19:21:29.363 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.363 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.363 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.363 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:29.363 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.363 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.363 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.363 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.363 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.363 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.364 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.364 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.364 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-11T19:21:29.364 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-11T19:21:29.364 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.364 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.364 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.364 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8", 2024-03-11T19:21:29.364 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.364 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.364 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.364 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.364 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.365 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.365 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.365 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.365 INFO:teuthology.orchestra.run.smithi029.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-11T19:21:29.365 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-11T19:21:29.365 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.365 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.365 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.365 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "45", 2024-03-11T19:21:29.365 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.365 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.365 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.365 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.365 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.365 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.365 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.366 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.366 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_ACCESS_KEY": { 2024-03-11T19:21:29.366 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-11T19:21:29.366 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.366 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.366 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.366 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.366 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.366 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.366 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.366 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.366 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.366 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.366 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.367 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.367 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-11T19:21:29.367 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-11T19:21:29.367 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.367 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.367 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.367 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-03-11T19:21:29.367 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.367 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.367 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.367 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.367 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.367 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.367 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.367 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.368 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_SECRET_KEY": { 2024-03-11T19:21:29.368 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-11T19:21:29.368 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.368 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.368 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.368 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.368 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.368 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.368 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.368 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.368 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.368 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.368 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.368 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.368 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_SSL_VERIFY": { 2024-03-11T19:21:29.368 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-11T19:21:29.369 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.369 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.369 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.369 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.369 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.369 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.369 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.369 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.369 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.369 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.369 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.369 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.369 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-11T19:21:29.369 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-11T19:21:29.369 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.369 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.369 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.370 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-11T19:21:29.370 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.370 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.370 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.370 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.370 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.370 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.370 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.370 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.370 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-11T19:21:29.370 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-11T19:21:29.370 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.370 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.370 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.371 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:29.371 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.371 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.371 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.371 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.371 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.371 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.371 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.371 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.371 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-11T19:21:29.371 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-11T19:21:29.371 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.371 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.371 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.371 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-11T19:21:29.371 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.372 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.372 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.372 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.372 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.372 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.372 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.372 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.372 INFO:teuthology.orchestra.run.smithi029.stdout: "cross_origin_url": { 2024-03-11T19:21:29.372 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cross_origin_url", 2024-03-11T19:21:29.372 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.372 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.372 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.372 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.372 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.373 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.373 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.373 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.373 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.373 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.373 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.373 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.373 INFO:teuthology.orchestra.run.smithi029.stdout: "crt_file": { 2024-03-11T19:21:29.373 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crt_file", 2024-03-11T19:21:29.373 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.373 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.373 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.373 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.373 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.374 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.374 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.374 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.374 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.374 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.374 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.374 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.374 INFO:teuthology.orchestra.run.smithi029.stdout: "debug": { 2024-03-11T19:21:29.374 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "debug", 2024-03-11T19:21:29.374 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.374 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.374 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.374 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.374 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.375 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.375 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.375 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable/disable debug options", 2024-03-11T19:21:29.375 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.375 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.375 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.375 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.375 INFO:teuthology.orchestra.run.smithi029.stdout: "jwt_token_ttl": { 2024-03-11T19:21:29.375 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "jwt_token_ttl", 2024-03-11T19:21:29.375 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.375 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.375 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.375 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "28800", 2024-03-11T19:21:29.376 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.376 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.376 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.376 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.376 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.376 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.376 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.376 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.376 INFO:teuthology.orchestra.run.smithi029.stdout: "key_file": { 2024-03-11T19:21:29.376 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "key_file", 2024-03-11T19:21:29.376 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.376 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.376 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.376 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.377 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.377 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.377 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.377 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.377 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.377 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.377 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.377 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.377 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.377 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.377 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.377 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.377 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.377 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.378 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.378 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.378 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.378 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.378 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.378 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.378 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.378 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.378 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.378 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.378 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.378 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.378 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.378 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.379 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.379 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.379 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.379 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.379 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.379 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.379 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.379 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.379 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.379 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.379 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.379 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.380 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.380 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.380 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.380 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.380 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.380 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.380 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.380 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.380 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.380 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.381 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.381 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.381 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.381 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.381 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.381 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.381 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.381 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.381 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.381 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.381 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.381 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.381 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.381 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.381 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.382 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.382 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.382 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.382 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.382 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.382 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.382 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.382 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.382 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.382 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.382 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.382 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.383 INFO:teuthology.orchestra.run.smithi029.stdout: "motd": { 2024-03-11T19:21:29.383 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "motd", 2024-03-11T19:21:29.383 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.383 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.383 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.383 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.383 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.383 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.383 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.383 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "The message of the day", 2024-03-11T19:21:29.383 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.383 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.383 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.383 INFO:teuthology.orchestra.run.smithi029.stdout: "redirect_resolve_ip_addr": { 2024-03-11T19:21:29.383 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "redirect_resolve_ip_addr", 2024-03-11T19:21:29.384 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.384 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.384 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.384 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.384 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.384 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.384 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.384 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.384 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.384 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.384 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.384 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.384 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-03-11T19:21:29.384 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-03-11T19:21:29.384 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.384 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.385 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.385 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "::", 2024-03-11T19:21:29.385 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.385 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.385 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.385 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.385 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.385 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.385 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.385 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-03-11T19:21:29.385 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-03-11T19:21:29.385 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.385 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.385 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.385 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8080", 2024-03-11T19:21:29.385 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.386 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.386 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.386 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.386 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.386 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.386 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.386 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.386 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl": { 2024-03-11T19:21:29.386 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl", 2024-03-11T19:21:29.386 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.386 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.386 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.386 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.386 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.386 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.386 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.387 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.387 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.387 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.387 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.387 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.387 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl_server_port": { 2024-03-11T19:21:29.387 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl_server_port", 2024-03-11T19:21:29.387 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.387 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.387 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.387 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8443", 2024-03-11T19:21:29.387 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.387 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.387 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.387 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.388 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.388 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.388 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.388 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.388 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_behaviour": { 2024-03-11T19:21:29.388 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_behaviour", 2024-03-11T19:21:29.388 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.388 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.388 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.388 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "redirect", 2024-03-11T19:21:29.388 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.388 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.388 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.388 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.389 INFO:teuthology.orchestra.run.smithi029.stdout: "redirect" 2024-03-11T19:21:29.389 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.389 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.389 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.389 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.389 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.389 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.389 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_error_status_code": { 2024-03-11T19:21:29.389 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_error_status_code", 2024-03-11T19:21:29.389 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.389 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.389 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.389 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-03-11T19:21:29.389 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "400", 2024-03-11T19:21:29.389 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "599", 2024-03-11T19:21:29.390 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.390 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.390 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.390 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.390 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.390 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.390 INFO:teuthology.orchestra.run.smithi029.stdout: "url_prefix": { 2024-03-11T19:21:29.390 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "url_prefix", 2024-03-11T19:21:29.390 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.390 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.390 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.390 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.390 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.390 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.390 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.390 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.391 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.391 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.391 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.391 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.391 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.391 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.391 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.391 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "devicehealth", 2024-03-11T19:21:29.391 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.391 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.391 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.391 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_monitoring": { 2024-03-11T19:21:29.391 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enable_monitoring", 2024-03-11T19:21:29.391 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.391 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.392 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.392 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.392 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.392 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.392 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.392 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "monitor device health metrics", 2024-03-11T19:21:29.392 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.392 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.392 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.392 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.392 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.392 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.392 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.392 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.393 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.393 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.393 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.393 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.393 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.393 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.393 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.393 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.393 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.393 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.393 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.393 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.393 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.393 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.393 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.393 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.394 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.394 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.394 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.394 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.394 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.394 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.394 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.394 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.394 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.394 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.394 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.394 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.394 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.395 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.395 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.395 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.395 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.395 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.395 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.395 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.395 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.395 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.395 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.395 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.395 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.395 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.395 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.395 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.395 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.396 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.396 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.396 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.396 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.396 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.396 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.396 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.396 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.396 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.396 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.396 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.396 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.396 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.396 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.396 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.396 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.397 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.397 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.397 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.397 INFO:teuthology.orchestra.run.smithi029.stdout: "mark_out_threshold": { 2024-03-11T19:21:29.397 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mark_out_threshold", 2024-03-11T19:21:29.397 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.397 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.397 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.397 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "2419200", 2024-03-11T19:21:29.397 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.397 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.397 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.397 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-11T19:21:29.397 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.397 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.398 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.398 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": { 2024-03-11T19:21:29.398 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pool_name", 2024-03-11T19:21:29.398 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.398 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.398 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.398 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "device_health_metrics", 2024-03-11T19:21:29.398 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.398 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.398 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.398 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-11T19:21:29.398 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.398 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.398 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.399 INFO:teuthology.orchestra.run.smithi029.stdout: "retention_period": { 2024-03-11T19:21:29.399 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "retention_period", 2024-03-11T19:21:29.399 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.399 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.399 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.399 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15552000", 2024-03-11T19:21:29.399 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.399 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.399 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.399 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long to retain device health metrics", 2024-03-11T19:21:29.399 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.399 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.399 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.399 INFO:teuthology.orchestra.run.smithi029.stdout: "scrape_frequency": { 2024-03-11T19:21:29.400 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "scrape_frequency", 2024-03-11T19:21:29.400 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.400 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.400 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.400 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "86400", 2024-03-11T19:21:29.400 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.400 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.400 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.400 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-11T19:21:29.400 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.400 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.400 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.400 INFO:teuthology.orchestra.run.smithi029.stdout: "self_heal": { 2024-03-11T19:21:29.400 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "self_heal", 2024-03-11T19:21:29.400 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.400 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.401 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.401 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.401 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.401 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.401 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.401 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-11T19:21:29.401 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.401 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.401 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.401 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.401 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-11T19:21:29.401 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-11T19:21:29.401 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.401 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.401 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.401 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-11T19:21:29.402 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.402 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.402 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.402 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to wake up and check device health", 2024-03-11T19:21:29.402 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.402 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.402 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.402 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_threshold": { 2024-03-11T19:21:29.402 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_threshold", 2024-03-11T19:21:29.402 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.402 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.402 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.402 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7257600", 2024-03-11T19:21:29.402 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.403 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.403 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.403 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-11T19:21:29.403 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.403 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.403 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.403 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.403 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.403 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.403 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.403 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "diskprediction_local", 2024-03-11T19:21:29.403 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.403 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.403 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.403 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.403 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.403 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.404 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.404 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.404 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.404 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.404 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.404 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.404 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.404 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.404 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.404 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.404 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.404 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.404 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.404 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.404 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.405 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.405 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.405 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.405 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.405 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.405 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.405 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.405 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.405 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.405 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.405 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.405 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.405 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.405 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.405 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.405 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.405 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.406 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.406 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.406 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.406 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.406 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.406 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.406 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.406 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.406 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.406 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.406 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.406 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.406 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.406 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.407 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.407 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.407 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.407 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.407 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.407 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.407 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.407 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.407 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.407 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.407 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.407 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.407 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.407 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.407 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.407 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.407 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.408 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.408 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.408 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.408 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.408 INFO:teuthology.orchestra.run.smithi029.stdout: "predict_interval": { 2024-03-11T19:21:29.408 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "predict_interval", 2024-03-11T19:21:29.408 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.408 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.408 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.408 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "86400", 2024-03-11T19:21:29.408 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.408 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.408 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.408 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.408 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.409 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.409 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.409 INFO:teuthology.orchestra.run.smithi029.stdout: "predictor_model": { 2024-03-11T19:21:29.409 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "predictor_model", 2024-03-11T19:21:29.409 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.409 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.409 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.409 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "prophetstor", 2024-03-11T19:21:29.409 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.409 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.409 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.409 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.409 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.409 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.409 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.410 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.410 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-11T19:21:29.410 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-11T19:21:29.410 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.410 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.410 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.410 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-11T19:21:29.410 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.410 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.410 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.410 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.410 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.410 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.410 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.410 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.411 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.411 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "influx", 2024-03-11T19:21:29.411 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": false, 2024-03-11T19:21:29.411 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "influxdb python module not found", 2024-03-11T19:21:29.411 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.411 INFO:teuthology.orchestra.run.smithi029.stdout: "batch_size": { 2024-03-11T19:21:29.411 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "batch_size", 2024-03-11T19:21:29.411 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.411 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.411 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.411 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5000", 2024-03-11T19:21:29.411 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.411 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.411 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.412 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-11T19:21:29.412 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.412 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.412 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.412 INFO:teuthology.orchestra.run.smithi029.stdout: "database": { 2024-03-11T19:21:29.412 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "database", 2024-03-11T19:21:29.412 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.412 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.412 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.412 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "ceph", 2024-03-11T19:21:29.412 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.412 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.412 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.412 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-11T19:21:29.412 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.413 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.413 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.413 INFO:teuthology.orchestra.run.smithi029.stdout: "hostname": { 2024-03-11T19:21:29.413 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "hostname", 2024-03-11T19:21:29.413 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.413 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.413 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.413 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.413 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.413 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.413 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.413 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB server hostname", 2024-03-11T19:21:29.413 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.413 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.413 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.414 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-11T19:21:29.414 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-11T19:21:29.414 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.414 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.414 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.414 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "30", 2024-03-11T19:21:29.414 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "5", 2024-03-11T19:21:29.414 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.414 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.414 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-11T19:21:29.414 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.414 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.414 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.415 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.415 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.415 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.415 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.415 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.415 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.415 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.415 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.415 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.415 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.415 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.415 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.415 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.415 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.415 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.416 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.416 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.416 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.416 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.416 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.416 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.416 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.416 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.416 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.416 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.416 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.416 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.416 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.416 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.416 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.417 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.417 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.417 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.417 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.417 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.417 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.417 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.417 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.417 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.417 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.417 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.417 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.417 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.418 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.418 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.418 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.418 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.418 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.418 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.418 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.418 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.418 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.418 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.419 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.419 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.419 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.419 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.419 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.419 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.419 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.419 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.419 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.419 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.419 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.419 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.420 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.420 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.420 INFO:teuthology.orchestra.run.smithi029.stdout: "password": { 2024-03-11T19:21:29.420 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "password", 2024-03-11T19:21:29.420 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.420 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.420 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.420 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.420 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.420 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.420 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.421 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "password of InfluxDB server user", 2024-03-11T19:21:29.421 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.421 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.421 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.421 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.421 INFO:teuthology.orchestra.run.smithi029.stdout: "port": { 2024-03-11T19:21:29.421 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "port", 2024-03-11T19:21:29.421 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.421 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.421 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.421 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8086", 2024-03-11T19:21:29.421 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.422 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.422 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.422 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB server port", 2024-03-11T19:21:29.422 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.422 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.422 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.422 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl": { 2024-03-11T19:21:29.422 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl", 2024-03-11T19:21:29.422 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.422 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.422 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.422 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "false", 2024-03-11T19:21:29.423 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.423 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.423 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.423 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-11T19:21:29.423 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.423 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.423 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.423 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.423 INFO:teuthology.orchestra.run.smithi029.stdout: "threads": { 2024-03-11T19:21:29.423 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "threads", 2024-03-11T19:21:29.423 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.423 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.423 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.423 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-11T19:21:29.423 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-11T19:21:29.423 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "32", 2024-03-11T19:21:29.424 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.424 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-11T19:21:29.424 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.424 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.424 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.424 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.424 INFO:teuthology.orchestra.run.smithi029.stdout: "username": { 2024-03-11T19:21:29.424 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "username", 2024-03-11T19:21:29.424 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.424 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.424 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.424 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.424 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.424 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.424 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.424 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "username of InfluxDB server user", 2024-03-11T19:21:29.425 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.425 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.425 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.425 INFO:teuthology.orchestra.run.smithi029.stdout: "verify_ssl": { 2024-03-11T19:21:29.425 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "verify_ssl", 2024-03-11T19:21:29.425 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.425 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.425 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.425 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "true", 2024-03-11T19:21:29.425 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.425 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.425 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.425 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-11T19:21:29.425 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.426 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.426 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.426 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.426 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.426 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.426 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.426 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "insights", 2024-03-11T19:21:29.426 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.426 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.426 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.426 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.426 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.426 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.426 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.426 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.427 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.427 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.427 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.427 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.427 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.427 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.427 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.427 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.427 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.427 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.427 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.427 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.427 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.427 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.427 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.427 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.428 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.428 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.428 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.428 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.428 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.428 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.428 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.428 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.428 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.428 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.428 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.428 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.428 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.428 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.428 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.428 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.429 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.429 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.429 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.429 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.429 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.429 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.429 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.429 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.429 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.429 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.429 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.429 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.429 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.429 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.429 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.429 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.429 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.430 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.430 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.430 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.430 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.430 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.430 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.430 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.430 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.430 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.430 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.430 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.430 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.430 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.430 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.430 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.431 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.431 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.431 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.431 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "iostat", 2024-03-11T19:21:29.431 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.431 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.431 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.431 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.431 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.431 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.431 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.431 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.431 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.431 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.431 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.431 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.432 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.432 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.432 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.432 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.432 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.432 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.432 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.432 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.432 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.432 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.432 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.432 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.432 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.432 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.432 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.432 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.432 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.433 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.433 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.433 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.433 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.433 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.433 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.433 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.433 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.433 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.433 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.433 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.433 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.433 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.433 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.433 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.433 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.433 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.434 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.434 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.434 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.434 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.434 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.434 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.434 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.434 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.434 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.434 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.434 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.434 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.434 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.434 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.434 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.434 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.434 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.435 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.435 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.435 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.435 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.435 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.435 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.435 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.435 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.435 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.435 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.435 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.435 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.435 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.435 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "k8sevents", 2024-03-11T19:21:29.435 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.435 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.436 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.436 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_event_retention_days": { 2024-03-11T19:21:29.436 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ceph_event_retention_days", 2024-03-11T19:21:29.436 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.436 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.436 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.436 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7", 2024-03-11T19:21:29.436 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.436 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.436 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.436 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-11T19:21:29.436 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.436 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.436 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.436 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.436 INFO:teuthology.orchestra.run.smithi029.stdout: "config_check_secs": { 2024-03-11T19:21:29.436 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_check_secs", 2024-03-11T19:21:29.437 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.437 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.437 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.437 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:29.437 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "10", 2024-03-11T19:21:29.437 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.437 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.437 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-11T19:21:29.437 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.437 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.437 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.437 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.437 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.437 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.437 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.437 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.437 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.438 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.438 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.438 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.438 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.438 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.438 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.438 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.438 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.438 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.438 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.438 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.438 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.438 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.438 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.438 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.438 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.439 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.439 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.439 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.439 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.439 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.439 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.439 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.439 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.439 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.439 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.439 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.439 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.439 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.439 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.439 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.439 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.439 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.439 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.440 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.440 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.440 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.440 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.440 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.440 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.440 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.440 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.440 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.440 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.440 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.440 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.440 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.440 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.440 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.441 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.441 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.441 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.441 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.441 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.441 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.441 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.441 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.441 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.441 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.441 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.441 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.441 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.441 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.441 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.441 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.441 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.442 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.442 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.442 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "localpool", 2024-03-11T19:21:29.442 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.442 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.442 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.442 INFO:teuthology.orchestra.run.smithi029.stdout: "failure_domain": { 2024-03-11T19:21:29.442 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "failure_domain", 2024-03-11T19:21:29.442 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.442 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.442 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.442 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "host", 2024-03-11T19:21:29.442 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.442 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.443 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.443 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "failure domain for any created local pool", 2024-03-11T19:21:29.443 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-11T19:21:29.443 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.443 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.443 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.443 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.443 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.443 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.443 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.443 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.443 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.443 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.443 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.443 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.443 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.444 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.444 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.444 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.444 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.444 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.444 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.444 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.444 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.444 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.444 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.444 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.444 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.444 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.444 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.444 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.444 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.445 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.445 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.445 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.445 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.445 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.445 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.445 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.445 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.445 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.445 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.445 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.445 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.445 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.445 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.445 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.445 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.445 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.446 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.446 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.446 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.446 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.446 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.446 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.446 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.446 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.446 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.446 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.446 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.446 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.446 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.446 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.447 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.447 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.447 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.447 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.447 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.447 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.447 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.447 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.447 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.447 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.447 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.447 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.447 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.447 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": { 2024-03-11T19:21:29.447 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "min_size", 2024-03-11T19:21:29.448 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.448 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.448 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.448 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.448 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.448 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.448 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.448 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default min_size for any created local pool", 2024-03-11T19:21:29.448 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-11T19:21:29.448 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.448 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.448 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.448 INFO:teuthology.orchestra.run.smithi029.stdout: "num_rep": { 2024-03-11T19:21:29.449 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "num_rep", 2024-03-11T19:21:29.449 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.449 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.449 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.449 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3", 2024-03-11T19:21:29.449 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.449 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.449 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.449 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default replica count for any created local pool", 2024-03-11T19:21:29.449 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.449 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.449 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.449 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.449 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": { 2024-03-11T19:21:29.450 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pg_num", 2024-03-11T19:21:29.450 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.450 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.450 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.450 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "128", 2024-03-11T19:21:29.450 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.450 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.450 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.450 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default pg_num for any created local pool", 2024-03-11T19:21:29.450 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.450 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.450 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.451 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.451 INFO:teuthology.orchestra.run.smithi029.stdout: "prefix": { 2024-03-11T19:21:29.451 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prefix", 2024-03-11T19:21:29.451 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.451 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.451 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.451 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.451 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.451 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.451 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.451 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "name prefix for any created local pool", 2024-03-11T19:21:29.451 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.451 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.451 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.451 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.452 INFO:teuthology.orchestra.run.smithi029.stdout: "subtree": { 2024-03-11T19:21:29.452 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "subtree", 2024-03-11T19:21:29.452 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.452 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.452 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.452 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "rack", 2024-03-11T19:21:29.452 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.452 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.452 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.452 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-11T19:21:29.452 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-11T19:21:29.452 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.452 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.453 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.453 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.453 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.453 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.453 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mds_autoscaler", 2024-03-11T19:21:29.453 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.453 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.453 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.453 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.453 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.453 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.453 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.453 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.453 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.454 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.454 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.454 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.454 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.454 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.454 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.454 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.454 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.454 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.454 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.454 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.454 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.454 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.455 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.455 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.455 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.455 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.455 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.455 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.455 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.455 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.455 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.455 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.455 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.455 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.455 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.455 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.455 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.456 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.456 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.456 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.456 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.456 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.456 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.456 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.456 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.456 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.456 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.456 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.456 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.456 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.456 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.456 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.456 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.456 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.457 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.457 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.457 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.457 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.457 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.457 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.457 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.457 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.457 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.457 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.457 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.457 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.457 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.457 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.457 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.457 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.457 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.457 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.458 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.458 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.458 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mirroring", 2024-03-11T19:21:29.458 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.458 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.458 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.458 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.458 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.458 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.458 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.458 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.458 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.458 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.458 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.458 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.458 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.459 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.459 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.459 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.459 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.459 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.459 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.459 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.459 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.459 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.459 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.459 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.459 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.459 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.459 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.459 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.459 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.460 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.460 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.460 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.460 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.460 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.460 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.460 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.460 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.460 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.460 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.460 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.460 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.460 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.460 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.460 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.460 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.461 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.461 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.461 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.461 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.461 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.461 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.461 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.461 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.461 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.461 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.461 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.461 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.461 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.461 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.461 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.462 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.462 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.462 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.462 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.462 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.462 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.462 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.462 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.462 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.462 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.462 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.462 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.462 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.462 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.462 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.463 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.463 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "nfs", 2024-03-11T19:21:29.463 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.463 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.463 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.463 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.463 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.463 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.463 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.463 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.463 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.463 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.463 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.463 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.463 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.463 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.464 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.464 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.464 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.464 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.464 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.464 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.464 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.464 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.464 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.464 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.464 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.464 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.464 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.464 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.464 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.464 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.465 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.465 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.465 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.465 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.465 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.465 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.465 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.465 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.465 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.465 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.465 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.465 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.465 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.465 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.465 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.465 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.465 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.466 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.466 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.466 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.466 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.466 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.466 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.466 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.466 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.466 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.466 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.466 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.466 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.466 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.466 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.466 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.466 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.467 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.467 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.467 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.467 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.467 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.467 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.467 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.467 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.467 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.467 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.467 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.467 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.467 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.467 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "orchestrator", 2024-03-11T19:21:29.467 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.467 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.467 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.468 INFO:teuthology.orchestra.run.smithi029.stdout: "fail_fs": { 2024-03-11T19:21:29.468 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "fail_fs", 2024-03-11T19:21:29.468 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.468 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.468 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.468 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.468 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.468 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.468 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.468 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-11T19:21:29.468 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.468 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.468 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.468 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.468 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.468 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.468 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.468 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.469 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.469 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.469 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.469 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.469 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.469 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.469 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.469 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.469 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.469 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.469 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.469 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.469 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.469 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.469 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.469 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.470 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.470 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.470 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.470 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.470 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.470 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.470 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.470 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.470 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.470 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.470 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.470 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.470 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.470 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.471 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.471 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.471 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.471 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.471 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.471 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.471 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.471 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.471 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.471 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.471 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.471 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.471 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.471 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.471 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.472 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.472 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.472 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.472 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.472 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.472 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.472 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.472 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.472 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.472 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.472 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.472 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.472 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.472 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.472 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.472 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.472 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.473 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.473 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.473 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.473 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.473 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator": { 2024-03-11T19:21:29.473 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "orchestrator", 2024-03-11T19:21:29.473 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.473 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.473 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.473 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.473 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.473 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.473 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.473 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm", 2024-03-11T19:21:29.473 INFO:teuthology.orchestra.run.smithi029.stdout: "rook", 2024-03-11T19:21:29.473 INFO:teuthology.orchestra.run.smithi029.stdout: "test_orchestrator" 2024-03-11T19:21:29.473 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.473 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Orchestrator backend", 2024-03-11T19:21:29.474 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.474 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.474 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.474 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.474 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.474 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.474 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.474 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "osd_perf_query", 2024-03-11T19:21:29.474 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.474 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.474 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.474 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.474 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.474 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.474 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.475 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.475 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.475 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.475 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.475 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.475 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.475 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.475 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.475 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.475 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.475 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.475 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.475 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.475 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.475 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.475 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.475 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.475 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.476 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.476 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.476 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.476 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.476 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.476 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.476 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.476 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.476 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.476 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.476 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.476 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.476 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.476 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.476 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.476 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.476 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.477 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.477 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.477 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.477 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.477 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.477 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.477 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.477 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.477 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.477 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.477 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.477 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.477 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.477 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.477 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.477 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.477 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.477 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.478 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.478 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.478 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.478 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.478 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.478 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.478 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.478 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.478 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.478 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.478 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.478 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.478 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.478 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.478 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.478 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.479 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "osd_support", 2024-03-11T19:21:29.479 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.479 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.479 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.479 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.479 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.479 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.479 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.479 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.479 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.479 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.479 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.479 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.479 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.479 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.479 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.479 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.479 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.480 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.480 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.480 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.480 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.480 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.480 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.480 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.480 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.480 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.480 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.480 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.480 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.480 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.480 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.480 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.480 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.481 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.481 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.481 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.481 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.481 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.481 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.481 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.481 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.481 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.481 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.481 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.481 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.481 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.481 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.481 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.482 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.482 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.482 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.482 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.482 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.482 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.482 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.482 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.482 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.482 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.482 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.482 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.482 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.482 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.483 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.483 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.483 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.485 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.485 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.485 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.486 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.486 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.486 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.486 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.486 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.486 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.486 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.486 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.486 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pg_autoscaler", 2024-03-11T19:21:29.486 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.486 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.486 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.486 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.486 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.486 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.487 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.487 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.487 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.487 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.487 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.487 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.487 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.487 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.487 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.487 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.487 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.487 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.487 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.487 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.487 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.487 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.487 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.488 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.488 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.488 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.488 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.488 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.488 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.488 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.488 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.488 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.488 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.488 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.488 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.488 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.488 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.488 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.488 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.488 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.489 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.489 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.489 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.489 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.489 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.489 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.489 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.489 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.489 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.489 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.489 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.489 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.489 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.489 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.489 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.489 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.490 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.490 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.490 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.490 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.490 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.490 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.490 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.490 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.490 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.490 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.490 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.490 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.490 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.490 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.490 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.491 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.491 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.491 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-11T19:21:29.491 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-11T19:21:29.491 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.491 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.491 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.491 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-11T19:21:29.491 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.491 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.491 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.491 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.491 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.491 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.491 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.492 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.492 INFO:teuthology.orchestra.run.smithi029.stdout: "threshold": { 2024-03-11T19:21:29.492 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "threshold", 2024-03-11T19:21:29.492 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:29.492 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.492 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.492 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3.0", 2024-03-11T19:21:29.492 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1.0", 2024-03-11T19:21:29.492 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.492 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.492 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "scaling threshold", 2024-03-11T19:21:29.492 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-11T19:21:29.492 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.492 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.493 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.493 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.493 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.493 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.493 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "progress", 2024-03-11T19:21:29.493 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.493 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.493 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.493 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_pg_recovery_event": { 2024-03-11T19:21:29.493 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_pg_recovery_event", 2024-03-11T19:21:29.493 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.493 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.493 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.493 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.493 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.493 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.493 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.494 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-11T19:21:29.494 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.494 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.494 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.494 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.494 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": { 2024-03-11T19:21:29.494 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enabled", 2024-03-11T19:21:29.494 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.494 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.494 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.494 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.494 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.494 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.494 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.494 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.495 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.495 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.495 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.495 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.495 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.495 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.495 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.495 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.495 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.495 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.495 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.495 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.495 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.495 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.495 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.495 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.496 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.496 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.496 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.496 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.496 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.496 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.496 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.496 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.496 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.496 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.496 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.496 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.496 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.496 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.496 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.497 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.497 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.497 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.497 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.497 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.497 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.497 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.497 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.497 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.497 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.497 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.497 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.497 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.497 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.498 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.498 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.498 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.498 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.498 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.498 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.498 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.498 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.498 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.498 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.498 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.498 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.498 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.499 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.499 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.499 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.499 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.499 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.499 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.499 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.499 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.499 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.499 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.499 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.499 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.499 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.499 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.499 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.499 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.500 INFO:teuthology.orchestra.run.smithi029.stdout: "max_completed_events": { 2024-03-11T19:21:29.500 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_completed_events", 2024-03-11T19:21:29.500 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.500 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.500 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.500 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "50", 2024-03-11T19:21:29.500 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.500 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.500 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.500 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "number of past completed events to remember", 2024-03-11T19:21:29.500 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.500 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.500 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.500 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.500 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-11T19:21:29.501 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-11T19:21:29.501 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.501 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.501 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.501 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-11T19:21:29.501 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.501 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.501 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.501 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long the module is going to sleep", 2024-03-11T19:21:29.501 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.501 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.501 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.501 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.501 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.502 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.502 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.502 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prometheus", 2024-03-11T19:21:29.502 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.502 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.502 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.502 INFO:teuthology.orchestra.run.smithi029.stdout: "cache": { 2024-03-11T19:21:29.502 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cache", 2024-03-11T19:21:29.502 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.502 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.502 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.502 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.502 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.502 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.502 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.503 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.503 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.503 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.503 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.503 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.503 INFO:teuthology.orchestra.run.smithi029.stdout: "exclude_perf_counters": { 2024-03-11T19:21:29.503 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "exclude_perf_counters", 2024-03-11T19:21:29.503 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.503 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.503 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.503 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.503 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.503 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.503 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.504 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-11T19:21:29.504 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-11T19:21:29.504 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.504 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.504 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.504 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.504 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.504 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.504 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.504 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.504 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.504 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.504 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.504 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.504 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.505 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.505 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.505 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.505 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.505 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.505 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.505 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.505 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.505 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.505 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.505 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.505 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.505 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.505 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.505 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.506 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.506 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.506 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.506 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.506 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.506 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.506 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.506 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.506 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.506 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.506 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.506 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.506 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.506 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.506 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.507 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.507 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.507 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.507 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.507 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.507 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.507 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.507 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.507 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.507 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.507 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.507 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.507 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.507 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.507 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.507 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.507 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.508 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.508 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.508 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.508 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.508 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.508 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.508 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.508 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.508 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.508 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.508 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.508 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.508 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.508 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_stats_pools": { 2024-03-11T19:21:29.508 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_stats_pools", 2024-03-11T19:21:29.508 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.509 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.509 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.509 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.509 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.509 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.509 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.509 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.509 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.509 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.509 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.509 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.509 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-11T19:21:29.509 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-11T19:21:29.509 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.509 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.509 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.509 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "300", 2024-03-11T19:21:29.510 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.510 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.510 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.510 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.510 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.510 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.510 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.510 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.510 INFO:teuthology.orchestra.run.smithi029.stdout: "scrape_interval": { 2024-03-11T19:21:29.510 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "scrape_interval", 2024-03-11T19:21:29.510 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:29.510 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.510 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.510 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15.0", 2024-03-11T19:21:29.510 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.511 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.511 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.511 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.511 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.511 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.511 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.511 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.511 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-03-11T19:21:29.511 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-03-11T19:21:29.511 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.511 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.511 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.511 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "::", 2024-03-11T19:21:29.511 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.511 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.511 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.512 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-11T19:21:29.512 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.512 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.512 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.512 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.512 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-03-11T19:21:29.512 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-03-11T19:21:29.512 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.512 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.512 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.512 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "9283", 2024-03-11T19:21:29.512 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.512 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.512 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.512 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-11T19:21:29.512 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.513 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.513 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.513 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.513 INFO:teuthology.orchestra.run.smithi029.stdout: "stale_cache_strategy": { 2024-03-11T19:21:29.513 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "stale_cache_strategy", 2024-03-11T19:21:29.513 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.513 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.513 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.513 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "log", 2024-03-11T19:21:29.513 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.513 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.513 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.513 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.513 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.513 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.513 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.513 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.514 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_behaviour": { 2024-03-11T19:21:29.514 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_behaviour", 2024-03-11T19:21:29.514 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.514 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.514 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.514 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "default", 2024-03-11T19:21:29.514 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.514 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.514 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.514 INFO:teuthology.orchestra.run.smithi029.stdout: "default", 2024-03-11T19:21:29.514 INFO:teuthology.orchestra.run.smithi029.stdout: "error" 2024-03-11T19:21:29.514 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.514 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.514 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.515 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.515 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.515 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_error_status_code": { 2024-03-11T19:21:29.515 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_error_status_code", 2024-03-11T19:21:29.515 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.515 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.515 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.515 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-03-11T19:21:29.515 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "400", 2024-03-11T19:21:29.515 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "599", 2024-03-11T19:21:29.515 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.515 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.515 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.515 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.516 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.516 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.516 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.516 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.516 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_support", 2024-03-11T19:21:29.516 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.516 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.516 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.516 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.516 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.516 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.516 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.516 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.516 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.516 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.517 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.517 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.517 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.517 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.517 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.517 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.517 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.517 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.517 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.517 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.517 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.517 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.517 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.517 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.517 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.518 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.518 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.518 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.518 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.518 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.518 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.518 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.518 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.518 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.518 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.518 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.518 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.518 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.518 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.519 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.519 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.519 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.519 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.519 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.519 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.519 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.519 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.519 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.519 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.519 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.519 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.519 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.519 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.519 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.519 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.520 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.520 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.520 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.520 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.520 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.520 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.520 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.520 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.520 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.520 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.520 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.520 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.520 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.520 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.520 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.520 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.521 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.521 INFO:teuthology.orchestra.run.smithi029.stdout: "max_concurrent_snap_create": { 2024-03-11T19:21:29.521 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_concurrent_snap_create", 2024-03-11T19:21:29.521 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.521 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.521 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.521 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-11T19:21:29.521 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.521 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.521 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.521 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.521 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.521 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.521 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.521 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.522 INFO:teuthology.orchestra.run.smithi029.stdout: "mirror_snapshot_schedule": { 2024-03-11T19:21:29.522 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mirror_snapshot_schedule", 2024-03-11T19:21:29.522 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.522 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.522 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.522 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.522 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.522 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.522 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.522 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.522 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.522 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.522 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.522 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.522 INFO:teuthology.orchestra.run.smithi029.stdout: "trash_purge_schedule": { 2024-03-11T19:21:29.523 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "trash_purge_schedule", 2024-03-11T19:21:29.523 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.523 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.523 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.523 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.523 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.523 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.523 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.523 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.523 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.523 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.523 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.523 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.523 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "restful", 2024-03-11T19:21:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_auth": { 2024-03-11T19:21:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enable_auth", 2024-03-11T19:21:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.524 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.525 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.525 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.525 INFO:teuthology.orchestra.run.smithi029.stdout: "key_file": { 2024-03-11T19:21:29.525 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "key_file", 2024-03-11T19:21:29.525 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.525 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.525 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.525 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.525 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.525 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.525 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.525 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.525 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.525 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.525 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.526 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.526 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.526 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.526 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.526 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.526 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.526 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.526 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.526 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.526 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.526 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.526 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.527 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.527 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.527 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.527 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.527 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.527 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.527 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.527 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.527 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.527 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.527 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.527 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.527 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.527 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.527 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.527 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.528 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.528 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.528 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.528 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.528 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.528 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.528 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.528 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.528 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.528 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.528 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.528 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.528 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.528 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.528 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.528 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.529 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.529 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.529 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.529 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.529 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.529 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.529 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.529 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.529 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.529 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.529 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.529 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.529 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.529 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.529 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.529 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.529 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.530 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.530 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.530 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.530 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.530 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.530 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.530 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.530 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.530 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.530 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.530 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-03-11T19:21:29.530 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-03-11T19:21:29.530 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.530 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.531 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.531 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.531 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.531 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.531 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.531 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.531 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.531 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.531 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.531 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-03-11T19:21:29.531 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-03-11T19:21:29.531 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.531 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.532 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.532 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.532 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.532 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.532 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.532 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.532 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.532 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.532 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.532 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.532 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.532 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.532 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.532 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rgw", 2024-03-11T19:21:29.533 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.533 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.533 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.533 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.533 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.533 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.533 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.533 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.533 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.533 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.533 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.533 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.533 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.533 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.533 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.534 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.534 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.534 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.534 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.534 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.534 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.534 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.534 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.534 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.534 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.534 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.534 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.534 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.534 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.534 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.535 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.535 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.535 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.535 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.535 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.535 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.535 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.535 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.535 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.535 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.535 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.535 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.535 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.535 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.535 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.536 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.536 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.536 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.536 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.536 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.536 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.536 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.536 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.536 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.536 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.536 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.536 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.536 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.536 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.536 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.536 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.537 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.537 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.537 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.537 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.537 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.537 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.537 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.537 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.537 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.537 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.537 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.537 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.538 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rook", 2024-03-11T19:21:29.538 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.538 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.538 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.538 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.538 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.538 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.538 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.538 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.538 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.538 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.538 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.538 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.538 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.539 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.539 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.539 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.539 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.539 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.539 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.539 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.539 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.539 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.539 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.539 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.539 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.539 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.539 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.540 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.540 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.540 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.540 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.540 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.540 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.540 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.540 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.540 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.540 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.540 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.540 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.540 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.540 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.541 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.541 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.541 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.541 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.541 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.541 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.541 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.541 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.541 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.541 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.541 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.541 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.541 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.541 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.542 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.542 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.542 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.542 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.542 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.542 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.542 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.542 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.542 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.542 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.542 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.543 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.543 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.543 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.543 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.543 INFO:teuthology.orchestra.run.smithi029.stdout: "storage_class": { 2024-03-11T19:21:29.543 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "storage_class", 2024-03-11T19:21:29.543 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.543 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.543 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.543 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "local", 2024-03-11T19:21:29.543 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.543 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.543 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.543 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-11T19:21:29.543 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.544 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.544 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.544 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.544 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.544 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.544 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "selftest", 2024-03-11T19:21:29.544 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.544 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.544 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.544 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.544 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.544 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.544 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.544 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.544 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.545 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.545 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.545 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.545 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.545 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.545 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.545 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.545 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.545 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.545 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.545 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.545 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.545 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.545 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.545 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.546 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.546 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.546 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.546 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.546 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.546 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.546 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.546 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.546 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.546 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.546 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.546 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.546 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.546 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.546 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.547 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.547 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.547 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.547 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.547 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.547 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.547 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.547 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.547 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.547 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.547 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.547 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.547 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.547 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.547 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.547 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.547 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.548 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.548 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.548 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.548 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.548 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.548 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.548 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.548 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.548 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.548 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.548 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.548 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.548 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.548 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.548 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.548 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.549 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.549 INFO:teuthology.orchestra.run.smithi029.stdout: "roption1": { 2024-03-11T19:21:29.549 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "roption1", 2024-03-11T19:21:29.549 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.549 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.549 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.549 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.549 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.549 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.549 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.549 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.549 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.549 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.549 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.549 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.549 INFO:teuthology.orchestra.run.smithi029.stdout: "roption2": { 2024-03-11T19:21:29.549 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "roption2", 2024-03-11T19:21:29.550 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.550 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.550 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.550 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "xyz", 2024-03-11T19:21:29.550 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.550 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.550 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.550 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.550 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.550 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.550 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.551 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.551 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption1": { 2024-03-11T19:21:29.551 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption1", 2024-03-11T19:21:29.551 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.551 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.551 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.551 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.551 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.551 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.551 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.551 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.551 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.551 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.551 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.552 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption2": { 2024-03-11T19:21:29.552 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption2", 2024-03-11T19:21:29.552 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.552 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.552 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.552 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.552 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.552 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.552 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.552 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.552 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.552 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.552 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.552 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption3": { 2024-03-11T19:21:29.552 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption3", 2024-03-11T19:21:29.553 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-11T19:21:29.553 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.553 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.553 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.553 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.553 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.553 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.553 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.553 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.553 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.553 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.553 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.553 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption4": { 2024-03-11T19:21:29.553 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption4", 2024-03-11T19:21:29.553 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.553 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.553 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.554 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.554 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.554 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.554 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.554 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.554 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.554 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.554 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.554 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.554 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption5": { 2024-03-11T19:21:29.554 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption5", 2024-03-11T19:21:29.554 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.554 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.554 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.554 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.554 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.555 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.555 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.555 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.555 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.555 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.555 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.555 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.555 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption6": { 2024-03-11T19:21:29.555 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption6", 2024-03-11T19:21:29.555 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.555 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.555 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.555 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.555 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.555 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.555 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.555 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.556 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.556 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.556 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.556 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption7": { 2024-03-11T19:21:29.556 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption7", 2024-03-11T19:21:29.556 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.556 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.556 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.556 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.556 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-11T19:21:29.556 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "42", 2024-03-11T19:21:29.556 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.556 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.556 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.556 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.556 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.557 INFO:teuthology.orchestra.run.smithi029.stdout: "testkey": { 2024-03-11T19:21:29.557 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testkey", 2024-03-11T19:21:29.557 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.557 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.557 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.557 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.557 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.557 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.557 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.557 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.557 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.557 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.557 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.557 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.557 INFO:teuthology.orchestra.run.smithi029.stdout: "testlkey": { 2024-03-11T19:21:29.557 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testlkey", 2024-03-11T19:21:29.557 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.557 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.558 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.558 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.558 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.558 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.558 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.558 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.558 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.558 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.558 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.558 INFO:teuthology.orchestra.run.smithi029.stdout: "testnewline": { 2024-03-11T19:21:29.558 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testnewline", 2024-03-11T19:21:29.558 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.558 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.558 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.559 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.559 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.559 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.559 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.559 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.559 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.559 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.559 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.559 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.559 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.559 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.559 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.559 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snap_schedule", 2024-03-11T19:21:29.559 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.559 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.559 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.559 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_m_granularity": { 2024-03-11T19:21:29.560 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_m_granularity", 2024-03-11T19:21:29.560 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.560 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.560 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.560 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.560 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.560 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.560 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.560 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow minute scheduled snapshots", 2024-03-11T19:21:29.560 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.560 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.560 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.560 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.560 INFO:teuthology.orchestra.run.smithi029.stdout: "dump_on_update": { 2024-03-11T19:21:29.560 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "dump_on_update", 2024-03-11T19:21:29.560 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.560 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.561 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.561 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.561 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.561 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.561 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.561 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "dump database to debug log on update", 2024-03-11T19:21:29.561 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.561 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.561 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.561 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.561 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.561 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.561 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.561 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.561 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.562 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.562 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.562 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.562 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.562 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.562 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.562 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.562 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.562 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.562 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.562 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.562 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.562 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.562 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.562 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.562 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.563 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.563 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.563 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.563 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.563 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.563 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.563 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.563 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.563 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.563 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.563 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.563 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.563 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.563 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.563 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.563 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.563 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.564 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.564 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.564 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.564 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.564 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.564 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.564 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.564 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.564 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.564 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.564 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.564 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.564 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.564 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.564 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.564 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.564 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.564 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.565 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.565 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.565 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.565 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.565 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.565 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.565 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.565 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.565 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.565 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.565 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.565 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.565 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.565 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.565 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.566 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.566 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "stats", 2024-03-11T19:21:29.566 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.566 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.566 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.566 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.566 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.566 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.566 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.566 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.566 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.566 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.566 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.567 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.567 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.567 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.567 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.567 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.567 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.567 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.567 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.567 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.567 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.567 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.567 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.567 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.567 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.567 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.567 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.567 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.567 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.568 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.568 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.568 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.568 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.568 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.568 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.568 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.568 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.568 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.568 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.568 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.568 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.568 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.568 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.568 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.569 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.569 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.569 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.569 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.569 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.569 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.569 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.569 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.569 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.569 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.569 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.569 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.569 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.569 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.569 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.569 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.569 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.570 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.570 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.570 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.570 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.570 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.570 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.570 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.570 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.570 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.570 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.570 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.570 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.570 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.570 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.570 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.570 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "status", 2024-03-11T19:21:29.571 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.571 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.571 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.571 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.571 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.571 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.571 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.571 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.571 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.571 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.571 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.571 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.571 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.571 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.571 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.571 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.572 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.572 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.572 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.572 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.572 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.572 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.572 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.572 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.572 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.572 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.572 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.572 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.572 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.572 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.572 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.572 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.572 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.573 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.573 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.573 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.573 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.573 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.573 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.573 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.573 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.573 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.573 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.573 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.573 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.573 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.573 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.573 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.573 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.573 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.574 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.574 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.574 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.574 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.574 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.574 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.574 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.574 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.574 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.574 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.574 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.574 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.574 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.574 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.574 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.574 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.574 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.575 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.575 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.575 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.575 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.575 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.575 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.575 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.575 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.575 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.575 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "telegraf", 2024-03-11T19:21:29.575 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.575 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.575 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.575 INFO:teuthology.orchestra.run.smithi029.stdout: "address": { 2024-03-11T19:21:29.575 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "address", 2024-03-11T19:21:29.575 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.575 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.576 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.576 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-11T19:21:29.576 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.576 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.576 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.576 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.576 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.576 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.576 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.576 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-11T19:21:29.576 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-11T19:21:29.576 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.576 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.576 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.576 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15", 2024-03-11T19:21:29.576 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.576 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.577 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.577 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.577 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.577 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.577 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.577 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.577 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.577 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.577 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.577 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.577 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.577 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.577 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.577 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.578 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.578 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.578 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.578 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.578 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.578 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.578 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.578 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.578 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.578 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.578 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.578 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.578 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.579 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.579 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.579 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.579 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.579 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.579 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.579 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.579 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.579 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.579 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.579 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.579 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.579 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.580 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.580 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.580 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.580 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.580 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.580 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.580 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.580 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.580 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.580 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.580 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.580 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.580 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.580 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.581 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.581 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.581 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.581 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.581 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.581 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.581 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.581 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.581 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.581 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.581 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.582 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.582 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.582 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.582 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.582 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.582 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.582 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.582 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.582 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "telemetry", 2024-03-11T19:21:29.582 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.582 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.583 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.583 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_basic": { 2024-03-11T19:21:29.583 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_basic", 2024-03-11T19:21:29.583 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.583 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.583 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.583 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.583 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.583 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.583 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.583 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-11T19:21:29.583 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.583 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.583 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.583 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.584 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_crash": { 2024-03-11T19:21:29.584 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_crash", 2024-03-11T19:21:29.584 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.584 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.584 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.584 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.584 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.584 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.584 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.584 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-11T19:21:29.584 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.584 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.584 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.584 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.584 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_device": { 2024-03-11T19:21:29.585 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_device", 2024-03-11T19:21:29.585 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.585 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.585 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.585 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-11T19:21:29.585 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.585 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.585 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.585 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-11T19:21:29.585 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.585 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.585 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.585 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.585 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_ident": { 2024-03-11T19:21:29.586 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_ident", 2024-03-11T19:21:29.586 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.586 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.586 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.586 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.586 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.586 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.586 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.586 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-11T19:21:29.586 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.586 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.586 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.586 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.586 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_perf": { 2024-03-11T19:21:29.586 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_perf", 2024-03-11T19:21:29.586 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.587 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.587 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.587 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.587 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.587 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.587 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.587 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-11T19:21:29.587 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.587 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.587 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.587 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.587 INFO:teuthology.orchestra.run.smithi029.stdout: "contact": { 2024-03-11T19:21:29.587 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "contact", 2024-03-11T19:21:29.587 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.587 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.587 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.587 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.588 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.588 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.588 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.588 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.588 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.588 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.588 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.588 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.588 INFO:teuthology.orchestra.run.smithi029.stdout: "description": { 2024-03-11T19:21:29.588 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "description", 2024-03-11T19:21:29.588 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.588 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.588 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.588 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.588 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.588 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.588 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.589 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.589 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.589 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.589 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.589 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.589 INFO:teuthology.orchestra.run.smithi029.stdout: "device_url": { 2024-03-11T19:21:29.589 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_url", 2024-03-11T19:21:29.589 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.589 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.589 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.589 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-11T19:21:29.589 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.589 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.589 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.590 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.590 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.590 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.590 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.590 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.590 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": { 2024-03-11T19:21:29.590 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enabled", 2024-03-11T19:21:29.590 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.590 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.590 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.590 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.590 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.590 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.590 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.590 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.590 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.591 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.591 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.591 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.591 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-11T19:21:29.591 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-11T19:21:29.591 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.591 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.591 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.591 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "24", 2024-03-11T19:21:29.591 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "8", 2024-03-11T19:21:29.591 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.591 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.591 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.591 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.591 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.591 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.592 INFO:teuthology.orchestra.run.smithi029.stdout: "last_opt_revision": { 2024-03-11T19:21:29.592 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "last_opt_revision", 2024-03-11T19:21:29.592 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.592 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.592 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.592 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1", 2024-03-11T19:21:29.592 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.592 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.592 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.592 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.592 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.592 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.592 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.592 INFO:teuthology.orchestra.run.smithi029.stdout: "leaderboard": { 2024-03-11T19:21:29.593 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "leaderboard", 2024-03-11T19:21:29.593 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.593 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.593 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.593 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.593 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.593 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.593 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.593 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.593 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.593 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.593 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.593 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.593 INFO:teuthology.orchestra.run.smithi029.stdout: "leaderboard_description": { 2024-03-11T19:21:29.593 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "leaderboard_description", 2024-03-11T19:21:29.593 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.593 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.593 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.594 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.594 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.594 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.594 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.594 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.594 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.594 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.594 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.594 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.594 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.594 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.594 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.594 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.595 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.595 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.595 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.595 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.595 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.595 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.595 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.595 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.595 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.595 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.595 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.595 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.595 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.595 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.595 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.596 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.596 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.596 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.596 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.596 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.596 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.596 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.596 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.596 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.596 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.596 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.596 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.596 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.596 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.596 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.597 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.597 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.597 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.597 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.597 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.597 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.597 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.597 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.597 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.597 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.597 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.597 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.597 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.597 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.597 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.597 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.598 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.598 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.598 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.598 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.598 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.598 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.598 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.598 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.598 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.598 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.598 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.598 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.598 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.598 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.598 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.599 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.599 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.599 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.599 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.599 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.599 INFO:teuthology.orchestra.run.smithi029.stdout: "organization": { 2024-03-11T19:21:29.599 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "organization", 2024-03-11T19:21:29.599 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.599 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.599 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.599 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.599 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.599 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.599 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.599 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.599 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.600 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.600 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.600 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.600 INFO:teuthology.orchestra.run.smithi029.stdout: "proxy": { 2024-03-11T19:21:29.600 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "proxy", 2024-03-11T19:21:29.600 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.600 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.600 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.600 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.600 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.600 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.600 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.600 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.600 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.600 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.601 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.601 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.601 INFO:teuthology.orchestra.run.smithi029.stdout: "url": { 2024-03-11T19:21:29.601 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "url", 2024-03-11T19:21:29.601 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.601 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.601 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.601 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-11T19:21:29.601 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.601 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.601 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.601 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.601 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.601 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.601 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.602 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.602 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.602 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.602 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.602 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "test_orchestrator", 2024-03-11T19:21:29.602 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.602 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.602 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.602 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.602 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.602 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.602 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.602 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.602 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.602 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.602 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.603 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.603 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.603 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.603 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.603 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.603 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.603 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.603 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.603 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.603 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.603 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.603 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.603 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.603 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.603 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.603 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.604 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.604 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.604 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.604 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.604 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.604 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.604 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.604 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.604 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.604 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.604 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.604 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.604 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.604 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.604 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.604 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.605 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.605 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.605 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.605 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.605 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.605 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.605 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.605 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.605 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.605 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.605 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.605 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.605 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.605 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.605 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.605 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.606 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.606 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.606 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.606 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.606 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.606 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.606 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.606 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.606 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.606 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.606 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.606 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.606 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.606 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.606 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.607 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.607 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.607 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "volumes", 2024-03-11T19:21:29.607 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.607 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.607 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.607 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.607 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.607 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.607 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.607 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.607 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.607 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.607 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.607 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.607 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.608 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.608 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.608 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.608 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.608 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.608 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.608 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.608 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.608 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.608 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.608 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.608 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.608 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.608 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.608 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.609 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.609 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.609 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.609 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.609 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.609 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.609 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.609 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.609 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.609 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.609 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.609 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.609 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.609 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.609 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.609 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.610 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.610 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.610 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.610 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.610 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.610 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.610 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.610 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.610 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.610 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.610 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.610 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.610 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.610 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.610 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.611 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.611 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.611 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.611 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.611 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.611 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.611 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.611 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.611 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.611 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.611 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.611 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.611 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.611 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.611 INFO:teuthology.orchestra.run.smithi029.stdout: "max_concurrent_clones": { 2024-03-11T19:21:29.612 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_concurrent_clones", 2024-03-11T19:21:29.612 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.612 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.612 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.612 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "4", 2024-03-11T19:21:29.612 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.612 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.612 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.612 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-11T19:21:29.612 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.612 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.612 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.612 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.612 INFO:teuthology.orchestra.run.smithi029.stdout: "periodic_async_work": { 2024-03-11T19:21:29.612 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "periodic_async_work", 2024-03-11T19:21:29.612 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.613 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.613 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.613 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.613 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.613 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.613 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.613 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Periodically check for async work", 2024-03-11T19:21:29.613 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.613 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.613 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.613 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.613 INFO:teuthology.orchestra.run.smithi029.stdout: "snapshot_clone_delay": { 2024-03-11T19:21:29.613 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snapshot_clone_delay", 2024-03-11T19:21:29.613 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.613 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.613 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.614 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-11T19:21:29.614 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.614 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.614 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.614 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-11T19:21:29.614 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.614 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.614 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.614 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.614 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.614 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.614 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.614 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix", 2024-03-11T19:21:29.614 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-11T19:21:29.614 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-11T19:21:29.615 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-11T19:21:29.615 INFO:teuthology.orchestra.run.smithi029.stdout: "discovery_interval": { 2024-03-11T19:21:29.615 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "discovery_interval", 2024-03-11T19:21:29.615 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-11T19:21:29.615 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.615 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.615 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "100", 2024-03-11T19:21:29.615 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.615 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.615 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.615 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.615 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.615 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.615 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.615 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.615 INFO:teuthology.orchestra.run.smithi029.stdout: "identifier": { 2024-03-11T19:21:29.616 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "identifier", 2024-03-11T19:21:29.616 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.616 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.616 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.616 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.616 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.616 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.616 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.616 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.616 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.616 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.616 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.616 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.616 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-11T19:21:29.617 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-11T19:21:29.617 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-11T19:21:29.617 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.617 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.617 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-11T19:21:29.617 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.617 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.617 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.617 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.617 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.617 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.617 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.617 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.617 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-11T19:21:29.617 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-11T19:21:29.618 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.618 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.618 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.618 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.618 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.618 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.618 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.618 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.618 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.618 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.618 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.618 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.618 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.619 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.619 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.619 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.619 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.619 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.619 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.619 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-11T19:21:29.619 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-11T19:21:29.619 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.619 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.619 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.619 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.619 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.619 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.620 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.620 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.620 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.620 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.620 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.620 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.620 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-11T19:21:29.620 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-11T19:21:29.620 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.620 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.620 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.620 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-11T19:21:29.620 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.621 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.621 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-11T19:21:29.621 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-11T19:21:29.621 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-11T19:21:29.621 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-11T19:21:29.621 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-11T19:21:29.621 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-11T19:21:29.621 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-11T19:21:29.621 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.621 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.621 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.621 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.621 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.621 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.622 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-11T19:21:29.622 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-11T19:21:29.622 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-11T19:21:29.622 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.622 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-11T19:21:29.622 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-11T19:21:29.622 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.622 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.622 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.622 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.622 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.622 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.622 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.622 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.623 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_host": { 2024-03-11T19:21:29.623 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_host", 2024-03-11T19:21:29.623 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.623 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.623 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.623 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-11T19:21:29.623 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.623 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.623 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.623 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.623 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.623 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.623 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.623 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.624 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_port": { 2024-03-11T19:21:29.624 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_port", 2024-03-11T19:21:29.624 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-11T19:21:29.624 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.624 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.624 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10051", 2024-03-11T19:21:29.624 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.624 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.624 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.624 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.624 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.624 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.624 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.625 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.625 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_sender": { 2024-03-11T19:21:29.625 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_sender", 2024-03-11T19:21:29.625 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-11T19:21:29.625 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-11T19:21:29.625 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-11T19:21:29.625 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-11T19:21:29.625 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-11T19:21:29.625 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-11T19:21:29.625 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-11T19:21:29.625 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-11T19:21:29.625 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-11T19:21:29.625 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-11T19:21:29.625 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-11T19:21:29.625 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.625 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.626 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.626 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.626 INFO:teuthology.orchestra.run.smithi029.stdout: "services": { 2024-03-11T19:21:29.626 INFO:teuthology.orchestra.run.smithi029.stdout: "dashboard": "https://172.21.15.29:8443/" 2024-03-11T19:21:29.626 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.626 INFO:teuthology.orchestra.run.smithi029.stdout: "always_on_modules": { 2024-03-11T19:21:29.626 INFO:teuthology.orchestra.run.smithi029.stdout: "octopus": [ 2024-03-11T19:21:29.626 INFO:teuthology.orchestra.run.smithi029.stdout: "balancer", 2024-03-11T19:21:29.626 INFO:teuthology.orchestra.run.smithi029.stdout: "crash", 2024-03-11T19:21:29.626 INFO:teuthology.orchestra.run.smithi029.stdout: "devicehealth", 2024-03-11T19:21:29.626 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator", 2024-03-11T19:21:29.626 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscaler", 2024-03-11T19:21:29.626 INFO:teuthology.orchestra.run.smithi029.stdout: "progress", 2024-03-11T19:21:29.626 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_support", 2024-03-11T19:21:29.626 INFO:teuthology.orchestra.run.smithi029.stdout: "status", 2024-03-11T19:21:29.626 INFO:teuthology.orchestra.run.smithi029.stdout: "telemetry", 2024-03-11T19:21:29.627 INFO:teuthology.orchestra.run.smithi029.stdout: "volumes" 2024-03-11T19:21:29.627 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.627 INFO:teuthology.orchestra.run.smithi029.stdout: "pacific": [ 2024-03-11T19:21:29.627 INFO:teuthology.orchestra.run.smithi029.stdout: "balancer", 2024-03-11T19:21:29.627 INFO:teuthology.orchestra.run.smithi029.stdout: "crash", 2024-03-11T19:21:29.627 INFO:teuthology.orchestra.run.smithi029.stdout: "devicehealth", 2024-03-11T19:21:29.627 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator", 2024-03-11T19:21:29.627 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscaler", 2024-03-11T19:21:29.627 INFO:teuthology.orchestra.run.smithi029.stdout: "progress", 2024-03-11T19:21:29.627 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_support", 2024-03-11T19:21:29.627 INFO:teuthology.orchestra.run.smithi029.stdout: "status", 2024-03-11T19:21:29.627 INFO:teuthology.orchestra.run.smithi029.stdout: "telemetry", 2024-03-11T19:21:29.627 INFO:teuthology.orchestra.run.smithi029.stdout: "volumes" 2024-03-11T19:21:29.627 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.628 INFO:teuthology.orchestra.run.smithi029.stdout: "quincy": [ 2024-03-11T19:21:29.628 INFO:teuthology.orchestra.run.smithi029.stdout: "balancer", 2024-03-11T19:21:29.628 INFO:teuthology.orchestra.run.smithi029.stdout: "crash", 2024-03-11T19:21:29.628 INFO:teuthology.orchestra.run.smithi029.stdout: "devicehealth", 2024-03-11T19:21:29.628 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator", 2024-03-11T19:21:29.628 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscaler", 2024-03-11T19:21:29.628 INFO:teuthology.orchestra.run.smithi029.stdout: "progress", 2024-03-11T19:21:29.628 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_support", 2024-03-11T19:21:29.628 INFO:teuthology.orchestra.run.smithi029.stdout: "status", 2024-03-11T19:21:29.628 INFO:teuthology.orchestra.run.smithi029.stdout: "telemetry", 2024-03-11T19:21:29.628 INFO:teuthology.orchestra.run.smithi029.stdout: "volumes" 2024-03-11T19:21:29.628 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.628 INFO:teuthology.orchestra.run.smithi029.stdout: "reef": [ 2024-03-11T19:21:29.628 INFO:teuthology.orchestra.run.smithi029.stdout: "balancer", 2024-03-11T19:21:29.629 INFO:teuthology.orchestra.run.smithi029.stdout: "crash", 2024-03-11T19:21:29.629 INFO:teuthology.orchestra.run.smithi029.stdout: "devicehealth", 2024-03-11T19:21:29.629 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator", 2024-03-11T19:21:29.629 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscaler", 2024-03-11T19:21:29.629 INFO:teuthology.orchestra.run.smithi029.stdout: "progress", 2024-03-11T19:21:29.629 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_support", 2024-03-11T19:21:29.629 INFO:teuthology.orchestra.run.smithi029.stdout: "status", 2024-03-11T19:21:29.629 INFO:teuthology.orchestra.run.smithi029.stdout: "telemetry", 2024-03-11T19:21:29.629 INFO:teuthology.orchestra.run.smithi029.stdout: "volumes" 2024-03-11T19:21:29.629 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-11T19:21:29.629 INFO:teuthology.orchestra.run.smithi029.stdout: "squid": [ 2024-03-11T19:21:29.629 INFO:teuthology.orchestra.run.smithi029.stdout: "balancer", 2024-03-11T19:21:29.629 INFO:teuthology.orchestra.run.smithi029.stdout: "crash", 2024-03-11T19:21:29.629 INFO:teuthology.orchestra.run.smithi029.stdout: "devicehealth", 2024-03-11T19:21:29.629 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator", 2024-03-11T19:21:29.629 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscaler", 2024-03-11T19:21:29.630 INFO:teuthology.orchestra.run.smithi029.stdout: "progress", 2024-03-11T19:21:29.630 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_support", 2024-03-11T19:21:29.630 INFO:teuthology.orchestra.run.smithi029.stdout: "status", 2024-03-11T19:21:29.630 INFO:teuthology.orchestra.run.smithi029.stdout: "telemetry", 2024-03-11T19:21:29.630 INFO:teuthology.orchestra.run.smithi029.stdout: "volumes" 2024-03-11T19:21:29.630 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-11T19:21:29.630 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.630 INFO:teuthology.orchestra.run.smithi029.stdout: "last_failure_osd_epoch": 3, 2024-03-11T19:21:29.630 INFO:teuthology.orchestra.run.smithi029.stdout: "active_clients": [ 2024-03-11T19:21:29.630 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.630 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "devicehealth", 2024-03-11T19:21:29.630 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-11T19:21:29.631 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.631 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-11T19:21:29.631 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:0", 2024-03-11T19:21:29.631 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 4241238840 2024-03-11T19:21:29.631 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.631 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-11T19:21:29.631 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.631 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.631 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "libcephsqlite", 2024-03-11T19:21:29.631 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-11T19:21:29.631 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.631 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-11T19:21:29.631 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:0", 2024-03-11T19:21:29.631 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 999793992 2024-03-11T19:21:29.631 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.632 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-11T19:21:29.632 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.632 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.632 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_support", 2024-03-11T19:21:29.632 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-11T19:21:29.632 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.632 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-11T19:21:29.632 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:0", 2024-03-11T19:21:29.632 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1189004412 2024-03-11T19:21:29.632 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.633 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-11T19:21:29.633 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:29.633 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.633 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "volumes", 2024-03-11T19:21:29.633 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-11T19:21:29.633 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-11T19:21:29.633 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-11T19:21:29.633 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:0", 2024-03-11T19:21:29.633 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2478434788 2024-03-11T19:21:29.633 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.633 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-11T19:21:29.633 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:29.633 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-11T19:21:29.633 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:21:29.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:28 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1298254654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:21:29.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:28 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3495027399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:21:29.635 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:28 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1298254654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:21:29.635 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:28 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3495027399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:21:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.444+0000 7f43d08c9640 1 -- 172.21.15.29:0/399025604 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f43b40a1a30 msgr2=0x7f43b40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.444+0000 7f43d08c9640 1 --2- 172.21.15.29:0/399025604 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f43b40a1a30 0x7f43b40a3ef0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f43cc101170 tx=0x7f43bc005f90 comp rx=0 tx=0).stop 2024-03-11T19:21:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.444+0000 7f43d08c9640 1 -- 172.21.15.29:0/399025604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43cc101310 msgr2=0x7f43cc197630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:21:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.444+0000 7f43d08c9640 1 --2- 172.21.15.29:0/399025604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43cc101310 0x7f43cc197630 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f43c000d770 tx=0x7f43c000dc30 comp rx=0 tx=0).stop 2024-03-11T19:21:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.444+0000 7f43d08c9640 1 -- 172.21.15.29:0/399025604 shutdown_connections 2024-03-11T19:21:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.444+0000 7f43d08c9640 1 --2- 172.21.15.29:0/399025604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43cc101cd0 0x7f43cc19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.444+0000 7f43d08c9640 1 --2- 172.21.15.29:0/399025604 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f43b40a1a30 0x7f43b40a3ef0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.444+0000 7f43d08c9640 1 --2- 172.21.15.29:0/399025604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43cc101310 0x7f43cc197630 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.444+0000 7f43d08c9640 1 --2- 172.21.15.29:0/399025604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43cc100120 0x7f43cc1970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:21:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.444+0000 7f43d08c9640 1 -- 172.21.15.29:0/399025604 >> 172.21.15.29:0/399025604 conn(0x7f43cc0fb6b0 msgr2=0x7f43cc104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:21:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.444+0000 7f43d08c9640 1 -- 172.21.15.29:0/399025604 shutdown_connections 2024-03-11T19:21:29.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:21:28.444+0000 7f43d08c9640 1 -- 172.21.15.29:0/399025604 wait complete. 2024-03-11T19:21:29.642 DEBUG:tasks.check_counter:Getting stats from a 2024-03-11T19:21:29.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-11T19:21:29.643 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:28 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1298254654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:21:29.643 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:28 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3495027399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:21:29.748 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:29 smithi170 ceph-mon[35690]: pgmap v701: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 7.3 MiB/s wr, 353 op/s 2024-03-11T19:21:29.748 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:29 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/399025604' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-11T19:21:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:29 smithi190 ceph-mon[36292]: pgmap v701: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 7.3 MiB/s wr, 353 op/s 2024-03-11T19:21:29.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:29 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/399025604' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-11T19:21:29.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:29 smithi029 ceph-mon[32439]: pgmap v701: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 7.3 MiB/s wr, 353 op/s 2024-03-11T19:21:29.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:29 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/399025604' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-11T19:21:30.069 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:21:31.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:31 smithi190 ceph-mon[36292]: pgmap v702: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 3.4 MiB/s wr, 282 op/s 2024-03-11T19:21:31.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:31 smithi170 ceph-mon[35690]: pgmap v702: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 3.4 MiB/s wr, 282 op/s 2024-03-11T19:21:31.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:31 smithi029 ceph-mon[32439]: pgmap v702: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 3.4 MiB/s wr, 282 op/s 2024-03-11T19:21:33.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:33 smithi029 ceph-mon[32439]: pgmap v703: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 227 KiB/s wr, 161 op/s 2024-03-11T19:21:33.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:33 smithi190 ceph-mon[36292]: pgmap v703: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 227 KiB/s wr, 161 op/s 2024-03-11T19:21:33.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:33 smithi170 ceph-mon[35690]: pgmap v703: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 227 KiB/s wr, 161 op/s 2024-03-11T19:21:34.919 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:21:34.920 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T19:21:34.920 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 1637, 2024-03-11T19:21:34.920 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 1639, 2024-03-11T19:21:34.920 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 490420, 2024-03-11T19:21:34.920 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 509108, 2024-03-11T19:21:34.920 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 6, 2024-03-11T19:21:34.921 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 2, 2024-03-11T19:21:34.921 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.132842727, 2024-03-11T19:21:34.921 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.041935322, 2024-03-11T19:21:34.921 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.034501522, 2024-03-11T19:21:34.921 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.046018434, 2024-03-11T19:21:34.921 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:34.921 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 1637, 2024-03-11T19:21:34.922 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.079384108, 2024-03-11T19:21:34.922 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000048493 2024-03-11T19:21:34.922 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.922 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:34.922 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.923 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.923 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.923 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.923 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T19:21:34.923 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T19:21:34.923 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.923 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T19:21:34.924 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 708, 2024-03-11T19:21:34.924 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 875, 2024-03-11T19:21:34.924 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 176742, 2024-03-11T19:21:34.924 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 597837, 2024-03-11T19:21:34.924 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 5, 2024-03-11T19:21:34.924 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 4, 2024-03-11T19:21:34.924 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.078586696, 2024-03-11T19:21:34.925 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.024242938, 2024-03-11T19:21:34.925 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.014378766, 2024-03-11T19:21:34.925 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.029220681, 2024-03-11T19:21:34.925 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:34.925 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 874, 2024-03-11T19:21:34.925 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.043128119, 2024-03-11T19:21:34.926 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000049345 2024-03-11T19:21:34.926 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.926 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:34.926 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.926 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.926 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.927 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.927 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-11T19:21:34.927 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 379136 2024-03-11T19:21:34.927 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.927 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T19:21:34.927 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 237, 2024-03-11T19:21:34.928 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 231, 2024-03-11T19:21:34.928 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 770860, 2024-03-11T19:21:34.928 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 135639, 2024-03-11T19:21:34.928 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 3, 2024-03-11T19:21:34.928 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 2, 2024-03-11T19:21:34.928 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.036938202, 2024-03-11T19:21:34.928 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.013587367, 2024-03-11T19:21:34.928 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.022741856, 2024-03-11T19:21:34.928 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.005190947, 2024-03-11T19:21:34.929 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:34.929 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 230, 2024-03-11T19:21:34.929 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.013593251, 2024-03-11T19:21:34.929 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000059101 2024-03-11T19:21:34.929 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.929 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:34.929 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.930 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.930 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.930 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.930 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 770592, 2024-03-11T19:21:34.930 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 135328 2024-03-11T19:21:34.930 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.931 INFO:teuthology.orchestra.run.smithi029.stdout: "cct": { 2024-03-11T19:21:34.931 INFO:teuthology.orchestra.run.smithi029.stdout: "total_workers": 1, 2024-03-11T19:21:34.931 INFO:teuthology.orchestra.run.smithi029.stdout: "unhealthy_workers": 0 2024-03-11T19:21:34.931 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.931 INFO:teuthology.orchestra.run.smithi029.stdout: "finisher-MDSRank": { 2024-03-11T19:21:34.931 INFO:teuthology.orchestra.run.smithi029.stdout: "queue_len": 0, 2024-03-11T19:21:34.932 INFO:teuthology.orchestra.run.smithi029.stdout: "complete_latency": { 2024-03-11T19:21:34.932 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 3834, 2024-03-11T19:21:34.932 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.114545670, 2024-03-11T19:21:34.932 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000029876 2024-03-11T19:21:34.932 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:34.932 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.933 INFO:teuthology.orchestra.run.smithi029.stdout: "finisher-PurgeQueue": { 2024-03-11T19:21:34.933 INFO:teuthology.orchestra.run.smithi029.stdout: "queue_len": 0, 2024-03-11T19:21:34.933 INFO:teuthology.orchestra.run.smithi029.stdout: "complete_latency": { 2024-03-11T19:21:34.933 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.933 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.933 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.934 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:34.934 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.934 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": { 2024-03-11T19:21:34.934 INFO:teuthology.orchestra.run.smithi029.stdout: "request": 0, 2024-03-11T19:21:34.934 INFO:teuthology.orchestra.run.smithi029.stdout: "reply": 0, 2024-03-11T19:21:34.934 INFO:teuthology.orchestra.run.smithi029.stdout: "reply_latency": { 2024-03-11T19:21:34.935 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.935 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.935 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.935 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.935 INFO:teuthology.orchestra.run.smithi029.stdout: "slow_reply": 0, 2024-03-11T19:21:34.935 INFO:teuthology.orchestra.run.smithi029.stdout: "forward": 0, 2024-03-11T19:21:34.936 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_fetch_complete": 0, 2024-03-11T19:21:34.936 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_fetch_keys": 0, 2024-03-11T19:21:34.936 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_commit": 0, 2024-03-11T19:21:34.936 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_split": 0, 2024-03-11T19:21:34.936 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_merge": 0, 2024-03-11T19:21:34.936 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes": 0, 2024-03-11T19:21:34.937 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_top": 0, 2024-03-11T19:21:34.937 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_bottom": 0, 2024-03-11T19:21:34.937 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_pin_tail": 0, 2024-03-11T19:21:34.937 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_pinned": 0, 2024-03-11T19:21:34.937 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_expired": 0, 2024-03-11T19:21:34.937 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_with_caps": 0, 2024-03-11T19:21:34.938 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": 0, 2024-03-11T19:21:34.938 INFO:teuthology.orchestra.run.smithi029.stdout: "subtrees": 1, 2024-03-11T19:21:34.938 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse": 0, 2024-03-11T19:21:34.939 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_hit": 0, 2024-03-11T19:21:34.939 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_forward": 0, 2024-03-11T19:21:34.939 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_discover": 0, 2024-03-11T19:21:34.939 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_dir_fetch": 0, 2024-03-11T19:21:34.939 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_remote_ino": 0, 2024-03-11T19:21:34.939 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_lock": 0, 2024-03-11T19:21:34.940 INFO:teuthology.orchestra.run.smithi029.stdout: "load_cent": 0, 2024-03-11T19:21:34.940 INFO:teuthology.orchestra.run.smithi029.stdout: "q": 0, 2024-03-11T19:21:34.940 INFO:teuthology.orchestra.run.smithi029.stdout: "exported": 0, 2024-03-11T19:21:34.940 INFO:teuthology.orchestra.run.smithi029.stdout: "exported_inodes": 0, 2024-03-11T19:21:34.940 INFO:teuthology.orchestra.run.smithi029.stdout: "imported": 0, 2024-03-11T19:21:34.940 INFO:teuthology.orchestra.run.smithi029.stdout: "imported_inodes": 0, 2024-03-11T19:21:34.941 INFO:teuthology.orchestra.run.smithi029.stdout: "openino_dir_fetch": 0, 2024-03-11T19:21:34.941 INFO:teuthology.orchestra.run.smithi029.stdout: "openino_backtrace_fetch": 0, 2024-03-11T19:21:34.941 INFO:teuthology.orchestra.run.smithi029.stdout: "openino_peer_discover": 0, 2024-03-11T19:21:34.941 INFO:teuthology.orchestra.run.smithi029.stdout: "root_rfiles": 0, 2024-03-11T19:21:34.941 INFO:teuthology.orchestra.run.smithi029.stdout: "root_rbytes": 0, 2024-03-11T19:21:34.941 INFO:teuthology.orchestra.run.smithi029.stdout: "root_rsnaps": 0, 2024-03-11T19:21:34.942 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T19:21:34.942 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_set_tag": 0, 2024-03-11T19:21:34.942 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T19:21:34.942 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_inotable_repaired": 0, 2024-03-11T19:21:34.943 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_dir_inodes": 0, 2024-03-11T19:21:34.943 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T19:21:34.943 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T19:21:34.943 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_file_inodes": 0, 2024-03-11T19:21:34.943 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_inode_file_caps": 0, 2024-03-11T19:21:34.943 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T19:21:34.944 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_grant": 0, 2024-03-11T19:21:34.944 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T19:21:34.944 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T19:21:34.944 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T19:21:34.944 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_caps": 0, 2024-03-11T19:21:34.944 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_caps_dirty": 0, 2024-03-11T19:21:34.944 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_cap_release": 0, 2024-03-11T19:21:34.945 INFO:teuthology.orchestra.run.smithi029.stdout: "process_request_cap_release": 0 2024-03-11T19:21:34.945 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.945 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache": { 2024-03-11T19:21:34.945 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_update": 0, 2024-03-11T19:21:34.945 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_update_receipt": 0, 2024-03-11T19:21:34.945 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_try_discover": 0, 2024-03-11T19:21:34.946 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_send_discover": 0, 2024-03-11T19:21:34.946 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_handle_discover": 0, 2024-03-11T19:21:34.946 INFO:teuthology.orchestra.run.smithi029.stdout: "num_strays": 0, 2024-03-11T19:21:34.946 INFO:teuthology.orchestra.run.smithi029.stdout: "num_strays_delayed": 0, 2024-03-11T19:21:34.946 INFO:teuthology.orchestra.run.smithi029.stdout: "num_strays_enqueuing": 0, 2024-03-11T19:21:34.946 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_created": 0, 2024-03-11T19:21:34.947 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_enqueued": 0, 2024-03-11T19:21:34.947 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_reintegrated": 0, 2024-03-11T19:21:34.947 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_migrated": 0, 2024-03-11T19:21:34.947 INFO:teuthology.orchestra.run.smithi029.stdout: "num_recovering_processing": 0, 2024-03-11T19:21:34.947 INFO:teuthology.orchestra.run.smithi029.stdout: "num_recovering_enqueued": 0, 2024-03-11T19:21:34.947 INFO:teuthology.orchestra.run.smithi029.stdout: "num_recovering_prioritized": 0, 2024-03-11T19:21:34.948 INFO:teuthology.orchestra.run.smithi029.stdout: "recovery_started": 0, 2024-03-11T19:21:34.948 INFO:teuthology.orchestra.run.smithi029.stdout: "recovery_completed": 0, 2024-03-11T19:21:34.948 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T19:21:34.948 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_exportdir": 0, 2024-03-11T19:21:34.948 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_flush": 0, 2024-03-11T19:21:34.948 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_fragmentdir": 0, 2024-03-11T19:21:34.949 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_fragstats": 0, 2024-03-11T19:21:34.949 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_inodestats": 0 2024-03-11T19:21:34.949 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.949 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log": { 2024-03-11T19:21:34.949 INFO:teuthology.orchestra.run.smithi029.stdout: "evlrg": 0, 2024-03-11T19:21:34.949 INFO:teuthology.orchestra.run.smithi029.stdout: "evadd": 0, 2024-03-11T19:21:34.949 INFO:teuthology.orchestra.run.smithi029.stdout: "evex": 0, 2024-03-11T19:21:34.949 INFO:teuthology.orchestra.run.smithi029.stdout: "evtrm": 0, 2024-03-11T19:21:34.950 INFO:teuthology.orchestra.run.smithi029.stdout: "ev": 3, 2024-03-11T19:21:34.950 INFO:teuthology.orchestra.run.smithi029.stdout: "evexg": 0, 2024-03-11T19:21:34.950 INFO:teuthology.orchestra.run.smithi029.stdout: "evexd": 0, 2024-03-11T19:21:34.950 INFO:teuthology.orchestra.run.smithi029.stdout: "segadd": 0, 2024-03-11T19:21:34.950 INFO:teuthology.orchestra.run.smithi029.stdout: "segex": 0, 2024-03-11T19:21:34.950 INFO:teuthology.orchestra.run.smithi029.stdout: "segtrm": 0, 2024-03-11T19:21:34.950 INFO:teuthology.orchestra.run.smithi029.stdout: "seg": 2, 2024-03-11T19:21:34.950 INFO:teuthology.orchestra.run.smithi029.stdout: "segmjr": 2, 2024-03-11T19:21:34.950 INFO:teuthology.orchestra.run.smithi029.stdout: "segexg": 0, 2024-03-11T19:21:34.951 INFO:teuthology.orchestra.run.smithi029.stdout: "segexd": 0, 2024-03-11T19:21:34.951 INFO:teuthology.orchestra.run.smithi029.stdout: "expos": 4194304, 2024-03-11T19:21:34.951 INFO:teuthology.orchestra.run.smithi029.stdout: "wrpos": 4195385, 2024-03-11T19:21:34.951 INFO:teuthology.orchestra.run.smithi029.stdout: "rdpos": 4194892, 2024-03-11T19:21:34.951 INFO:teuthology.orchestra.run.smithi029.stdout: "jlat": { 2024-03-11T19:21:34.951 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.951 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.951 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.951 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.952 INFO:teuthology.orchestra.run.smithi029.stdout: "replayed": 3 2024-03-11T19:21:34.952 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_mem": { 2024-03-11T19:21:34.952 INFO:teuthology.orchestra.run.smithi029.stdout: "ino": 3, 2024-03-11T19:21:34.952 INFO:teuthology.orchestra.run.smithi029.stdout: "ino+": 3, 2024-03-11T19:21:34.952 INFO:teuthology.orchestra.run.smithi029.stdout: "ino-": 0, 2024-03-11T19:21:34.952 INFO:teuthology.orchestra.run.smithi029.stdout: "dir": 1, 2024-03-11T19:21:34.952 INFO:teuthology.orchestra.run.smithi029.stdout: "dir+": 1, 2024-03-11T19:21:34.953 INFO:teuthology.orchestra.run.smithi029.stdout: "dir-": 0, 2024-03-11T19:21:34.953 INFO:teuthology.orchestra.run.smithi029.stdout: "dn": 0, 2024-03-11T19:21:34.953 INFO:teuthology.orchestra.run.smithi029.stdout: "dn+": 0, 2024-03-11T19:21:34.953 INFO:teuthology.orchestra.run.smithi029.stdout: "dn-": 0, 2024-03-11T19:21:34.953 INFO:teuthology.orchestra.run.smithi029.stdout: "cap": 0, 2024-03-11T19:21:34.953 INFO:teuthology.orchestra.run.smithi029.stdout: "cap+": 0, 2024-03-11T19:21:34.953 INFO:teuthology.orchestra.run.smithi029.stdout: "cap-": 0, 2024-03-11T19:21:34.954 INFO:teuthology.orchestra.run.smithi029.stdout: "rss": 48324, 2024-03-11T19:21:34.954 INFO:teuthology.orchestra.run.smithi029.stdout: "heap": 215420 2024-03-11T19:21:34.954 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_server": { 2024-03-11T19:21:34.954 INFO:teuthology.orchestra.run.smithi029.stdout: "dispatch_client_request": 0, 2024-03-11T19:21:34.954 INFO:teuthology.orchestra.run.smithi029.stdout: "dispatch_server_request": 0, 2024-03-11T19:21:34.954 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_request": 0, 2024-03-11T19:21:34.954 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_session": 0, 2024-03-11T19:21:34.954 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_peer_request": 0, 2024-03-11T19:21:34.954 INFO:teuthology.orchestra.run.smithi029.stdout: "req_create_latency": { 2024-03-11T19:21:34.954 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.954 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.955 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.955 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.955 INFO:teuthology.orchestra.run.smithi029.stdout: "req_getattr_latency": { 2024-03-11T19:21:34.955 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.955 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.955 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.955 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.955 INFO:teuthology.orchestra.run.smithi029.stdout: "req_getfilelock_latency": { 2024-03-11T19:21:34.955 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.955 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.955 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.955 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.955 INFO:teuthology.orchestra.run.smithi029.stdout: "req_link_latency": { 2024-03-11T19:21:34.955 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.956 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.956 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.956 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.956 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookup_latency": { 2024-03-11T19:21:34.956 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.956 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.956 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.956 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.956 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookuphash_latency": { 2024-03-11T19:21:34.956 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.956 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.956 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.956 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.956 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupino_latency": { 2024-03-11T19:21:34.956 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.957 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.957 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.957 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupname_latency": { 2024-03-11T19:21:34.957 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.957 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.957 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.957 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupparent_latency": { 2024-03-11T19:21:34.957 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.957 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.957 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.957 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupsnap_latency": { 2024-03-11T19:21:34.957 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.958 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.958 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.958 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.958 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lssnap_latency": { 2024-03-11T19:21:34.958 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.958 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.958 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.958 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.958 INFO:teuthology.orchestra.run.smithi029.stdout: "req_mkdir_latency": { 2024-03-11T19:21:34.958 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.958 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.958 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.958 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.959 INFO:teuthology.orchestra.run.smithi029.stdout: "req_mknod_latency": { 2024-03-11T19:21:34.959 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.959 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.959 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.959 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.960 INFO:teuthology.orchestra.run.smithi029.stdout: "req_mksnap_latency": { 2024-03-11T19:21:34.960 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.960 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.960 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.960 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.960 INFO:teuthology.orchestra.run.smithi029.stdout: "req_open_latency": { 2024-03-11T19:21:34.960 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.960 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.960 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.960 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.960 INFO:teuthology.orchestra.run.smithi029.stdout: "req_readdir_latency": { 2024-03-11T19:21:34.960 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.960 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.960 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.960 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.960 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rename_latency": { 2024-03-11T19:21:34.961 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.961 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.961 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.961 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.961 INFO:teuthology.orchestra.run.smithi029.stdout: "req_renamesnap_latency": { 2024-03-11T19:21:34.961 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.961 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.961 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.961 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.961 INFO:teuthology.orchestra.run.smithi029.stdout: "req_snapdiff_latency": { 2024-03-11T19:21:34.961 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.961 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.961 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.961 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.961 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rmdir_latency": { 2024-03-11T19:21:34.961 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.962 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.962 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.962 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.962 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rmsnap_latency": { 2024-03-11T19:21:34.962 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.962 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.962 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.962 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.962 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rmxattr_latency": { 2024-03-11T19:21:34.962 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.962 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.962 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.962 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.962 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setattr_latency": { 2024-03-11T19:21:34.962 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.963 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.963 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.963 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.963 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setdirlayout_latency": { 2024-03-11T19:21:34.963 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.963 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.963 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.963 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.963 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setfilelock_latency": { 2024-03-11T19:21:34.963 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.963 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.963 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.963 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.964 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setlayout_latency": { 2024-03-11T19:21:34.964 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.964 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.964 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.964 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.964 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setxattr_latency": { 2024-03-11T19:21:34.964 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.964 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.964 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.964 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.964 INFO:teuthology.orchestra.run.smithi029.stdout: "req_symlink_latency": { 2024-03-11T19:21:34.964 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.964 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.964 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.964 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.965 INFO:teuthology.orchestra.run.smithi029.stdout: "req_unlink_latency": { 2024-03-11T19:21:34.965 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.965 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.965 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.965 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.965 INFO:teuthology.orchestra.run.smithi029.stdout: "cap_revoke_eviction": 0, 2024-03-11T19:21:34.965 INFO:teuthology.orchestra.run.smithi029.stdout: "cap_acquisition_throttle": 0, 2024-03-11T19:21:34.965 INFO:teuthology.orchestra.run.smithi029.stdout: "req_getvxattr_latency": { 2024-03-11T19:21:34.965 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.965 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.965 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.965 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:34.965 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.965 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_sessions": { 2024-03-11T19:21:34.965 INFO:teuthology.orchestra.run.smithi029.stdout: "session_count": 1, 2024-03-11T19:21:34.965 INFO:teuthology.orchestra.run.smithi029.stdout: "session_add": 1, 2024-03-11T19:21:34.966 INFO:teuthology.orchestra.run.smithi029.stdout: "session_remove": 0, 2024-03-11T19:21:34.966 INFO:teuthology.orchestra.run.smithi029.stdout: "sessions_open": 1, 2024-03-11T19:21:34.966 INFO:teuthology.orchestra.run.smithi029.stdout: "sessions_stale": 0, 2024-03-11T19:21:34.966 INFO:teuthology.orchestra.run.smithi029.stdout: "total_load": 0, 2024-03-11T19:21:34.966 INFO:teuthology.orchestra.run.smithi029.stdout: "average_load": 0, 2024-03-11T19:21:34.966 INFO:teuthology.orchestra.run.smithi029.stdout: "avg_session_uptime": 1839, 2024-03-11T19:21:34.966 INFO:teuthology.orchestra.run.smithi029.stdout: "mdthresh_evicted": 0 2024-03-11T19:21:34.966 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.966 INFO:teuthology.orchestra.run.smithi029.stdout: "mempool": { 2024-03-11T19:21:34.966 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_bytes": 0, 2024-03-11T19:21:34.966 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_items": 0, 2024-03-11T19:21:34.966 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T19:21:34.966 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_items": 0, 2024-03-11T19:21:34.966 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T19:21:34.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_items": 0, 2024-03-11T19:21:34.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T19:21:34.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T19:21:34.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T19:21:34.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T19:21:34.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T19:21:34.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_items": 0, 2024-03-11T19:21:34.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T19:21:34.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T19:21:34.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_bytes": 0, 2024-03-11T19:21:34.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_items": 0, 2024-03-11T19:21:34.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_bytes": 0, 2024-03-11T19:21:34.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_items": 0, 2024-03-11T19:21:34.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T19:21:34.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T19:21:34.967 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T19:21:34.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T19:21:34.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T19:21:34.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_items": 0, 2024-03-11T19:21:34.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_bytes": 0, 2024-03-11T19:21:34.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_items": 0, 2024-03-11T19:21:34.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T19:21:34.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T19:21:34.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_bytes": 0, 2024-03-11T19:21:34.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_items": 0, 2024-03-11T19:21:34.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_bytes": 0, 2024-03-11T19:21:34.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_items": 0, 2024-03-11T19:21:34.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T19:21:34.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_items": 0, 2024-03-11T19:21:34.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T19:21:34.968 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_items": 0, 2024-03-11T19:21:34.968 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_bytes": 189383, 2024-03-11T19:21:34.969 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_items": 50, 2024-03-11T19:21:34.969 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_bytes": 0, 2024-03-11T19:21:34.969 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_items": 0, 2024-03-11T19:21:34.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bytes": 0, 2024-03-11T19:21:34.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_items": 0, 2024-03-11T19:21:34.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_bytes": 0, 2024-03-11T19:21:34.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_items": 0, 2024-03-11T19:21:34.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_bytes": 0, 2024-03-11T19:21:34.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_items": 0, 2024-03-11T19:21:34.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_bytes": 7336, 2024-03-11T19:21:34.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_items": 154, 2024-03-11T19:21:34.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T19:21:34.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_items": 0, 2024-03-11T19:21:34.969 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_bytes": 0, 2024-03-11T19:21:34.969 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_items": 0, 2024-03-11T19:21:34.969 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_bytes": 8320, 2024-03-11T19:21:34.970 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_items": 13, 2024-03-11T19:21:34.970 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_bytes": 0, 2024-03-11T19:21:34.970 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_items": 0, 2024-03-11T19:21:34.970 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_bytes": 0, 2024-03-11T19:21:34.970 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_items": 0 2024-03-11T19:21:34.970 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.970 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter": { 2024-03-11T19:21:34.970 INFO:teuthology.orchestra.run.smithi029.stdout: "op_active": 0, 2024-03-11T19:21:34.970 INFO:teuthology.orchestra.run.smithi029.stdout: "op_laggy": 0, 2024-03-11T19:21:34.970 INFO:teuthology.orchestra.run.smithi029.stdout: "op_send": 2341, 2024-03-11T19:21:34.970 INFO:teuthology.orchestra.run.smithi029.stdout: "op_send_bytes": 21, 2024-03-11T19:21:34.970 INFO:teuthology.orchestra.run.smithi029.stdout: "op_resend": 3, 2024-03-11T19:21:34.970 INFO:teuthology.orchestra.run.smithi029.stdout: "op_reply": 2341, 2024-03-11T19:21:34.970 INFO:teuthology.orchestra.run.smithi029.stdout: "op_latency": { 2024-03-11T19:21:34.970 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 2341, 2024-03-11T19:21:34.971 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 1.350978399, 2024-03-11T19:21:34.971 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000577094 2024-03-11T19:21:34.971 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.971 INFO:teuthology.orchestra.run.smithi029.stdout: "op_inflight": 0, 2024-03-11T19:21:34.971 INFO:teuthology.orchestra.run.smithi029.stdout: "oplen_avg": { 2024-03-11T19:21:34.971 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 2341, 2024-03-11T19:21:34.971 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 2342 2024-03-11T19:21:34.971 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.971 INFO:teuthology.orchestra.run.smithi029.stdout: "op": 2341, 2024-03-11T19:21:34.971 INFO:teuthology.orchestra.run.smithi029.stdout: "op_r": 2341, 2024-03-11T19:21:34.971 INFO:teuthology.orchestra.run.smithi029.stdout: "op_w": 0, 2024-03-11T19:21:34.971 INFO:teuthology.orchestra.run.smithi029.stdout: "op_rmw": 0, 2024-03-11T19:21:34.971 INFO:teuthology.orchestra.run.smithi029.stdout: "op_pg": 0, 2024-03-11T19:21:34.971 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_stat": 1517, 2024-03-11T19:21:34.971 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_create": 0, 2024-03-11T19:21:34.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_read": 822, 2024-03-11T19:21:34.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_write": 0, 2024-03-11T19:21:34.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_writefull": 0, 2024-03-11T19:21:34.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_writesame": 0, 2024-03-11T19:21:34.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_append": 0, 2024-03-11T19:21:34.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_zero": 0, 2024-03-11T19:21:34.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_truncate": 0, 2024-03-11T19:21:34.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_delete": 0, 2024-03-11T19:21:34.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_mapext": 0, 2024-03-11T19:21:34.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_sparse_read": 0, 2024-03-11T19:21:34.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_clonerange": 0, 2024-03-11T19:21:34.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_getxattr": 1, 2024-03-11T19:21:34.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_setxattr": 0, 2024-03-11T19:21:34.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_cmpxattr": 0, 2024-03-11T19:21:34.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_rmxattr": 0, 2024-03-11T19:21:34.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_resetxattrs": 0, 2024-03-11T19:21:34.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_call": 0, 2024-03-11T19:21:34.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_watch": 0, 2024-03-11T19:21:34.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_notify": 0, 2024-03-11T19:21:34.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T19:21:34.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_pgls": 0, 2024-03-11T19:21:34.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_pgls_filter": 0, 2024-03-11T19:21:34.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_other": 0, 2024-03-11T19:21:34.973 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_active": 0, 2024-03-11T19:21:34.973 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_send": 0, 2024-03-11T19:21:34.973 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_resend": 0, 2024-03-11T19:21:34.973 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_ping": 0, 2024-03-11T19:21:34.973 INFO:teuthology.orchestra.run.smithi029.stdout: "poolop_active": 0, 2024-03-11T19:21:34.973 INFO:teuthology.orchestra.run.smithi029.stdout: "poolop_send": 0, 2024-03-11T19:21:34.973 INFO:teuthology.orchestra.run.smithi029.stdout: "poolop_resend": 0, 2024-03-11T19:21:34.974 INFO:teuthology.orchestra.run.smithi029.stdout: "poolstat_active": 0, 2024-03-11T19:21:34.974 INFO:teuthology.orchestra.run.smithi029.stdout: "poolstat_send": 0, 2024-03-11T19:21:34.974 INFO:teuthology.orchestra.run.smithi029.stdout: "poolstat_resend": 0, 2024-03-11T19:21:34.974 INFO:teuthology.orchestra.run.smithi029.stdout: "statfs_active": 0, 2024-03-11T19:21:34.974 INFO:teuthology.orchestra.run.smithi029.stdout: "statfs_send": 0, 2024-03-11T19:21:34.974 INFO:teuthology.orchestra.run.smithi029.stdout: "statfs_resend": 0, 2024-03-11T19:21:34.974 INFO:teuthology.orchestra.run.smithi029.stdout: "command_active": 0, 2024-03-11T19:21:34.974 INFO:teuthology.orchestra.run.smithi029.stdout: "command_send": 0, 2024-03-11T19:21:34.974 INFO:teuthology.orchestra.run.smithi029.stdout: "command_resend": 0, 2024-03-11T19:21:34.974 INFO:teuthology.orchestra.run.smithi029.stdout: "map_epoch": 89, 2024-03-11T19:21:34.974 INFO:teuthology.orchestra.run.smithi029.stdout: "map_full": 0, 2024-03-11T19:21:34.974 INFO:teuthology.orchestra.run.smithi029.stdout: "map_inc": 2, 2024-03-11T19:21:34.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_sessions": 5, 2024-03-11T19:21:34.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_session_open": 5, 2024-03-11T19:21:34.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_session_close": 0, 2024-03-11T19:21:34.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_laggy": 0, 2024-03-11T19:21:34.975 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_wr": 0, 2024-03-11T19:21:34.975 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_rd": 2, 2024-03-11T19:21:34.975 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_del": 0 2024-03-11T19:21:34.975 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.975 INFO:teuthology.orchestra.run.smithi029.stdout: "oft": { 2024-03-11T19:21:34.975 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_objs": 0, 2024-03-11T19:21:34.975 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_kv_pairs": 0, 2024-03-11T19:21:34.975 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_updates": 0, 2024-03-11T19:21:34.975 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_removes": 0 2024-03-11T19:21:34.975 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd-slow-ops": { 2024-03-11T19:21:34.975 INFO:teuthology.orchestra.run.smithi029.stdout: "slow_ops_count": 0 2024-03-11T19:21:34.976 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.976 INFO:teuthology.orchestra.run.smithi029.stdout: "purge_queue": { 2024-03-11T19:21:34.976 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing_ops": 0, 2024-03-11T19:21:34.976 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T19:21:34.976 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing": 0, 2024-03-11T19:21:34.976 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing_high_water": 0, 2024-03-11T19:21:34.976 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executed_ops": 0, 2024-03-11T19:21:34.976 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executed": 0, 2024-03-11T19:21:34.976 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_item_in_journal": 0 2024-03-11T19:21:34.976 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.976 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T19:21:34.976 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-03-11T19:21:34.976 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-03-11T19:21:34.976 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-03-11T19:21:34.976 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 2578, 2024-03-11T19:21:34.976 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 1190887, 2024-03-11T19:21:34.976 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:34.977 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 2578, 2024-03-11T19:21:34.977 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-03-11T19:21:34.977 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-03-11T19:21:34.977 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 2578, 2024-03-11T19:21:34.977 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 1190887, 2024-03-11T19:21:34.977 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-03-11T19:21:34.977 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.977 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.977 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.977 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:34.977 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.977 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-objecter_bytes": { 2024-03-11T19:21:34.977 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-03-11T19:21:34.977 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-03-11T19:21:34.977 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-03-11T19:21:34.977 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-03-11T19:21:34.978 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-03-11T19:21:34.978 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:34.978 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:34.978 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 2341, 2024-03-11T19:21:34.978 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 1086, 2024-03-11T19:21:34.978 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 2341, 2024-03-11T19:21:34.978 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 1086, 2024-03-11T19:21:34.978 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-03-11T19:21:34.978 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.978 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.978 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.978 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:34.978 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.979 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-objecter_ops": { 2024-03-11T19:21:34.979 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-03-11T19:21:34.979 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 1024, 2024-03-11T19:21:34.979 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-03-11T19:21:34.979 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-03-11T19:21:34.979 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-03-11T19:21:34.979 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:34.979 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:34.979 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 2341, 2024-03-11T19:21:34.979 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 2341, 2024-03-11T19:21:34.979 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 2341, 2024-03-11T19:21:34.979 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 2341, 2024-03-11T19:21:34.979 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-03-11T19:21:34.979 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.979 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.980 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.980 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:34.980 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.980 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-write_buf_throttle": { 2024-03-11T19:21:34.980 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-03-11T19:21:34.980 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 3758096384, 2024-03-11T19:21:34.980 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-03-11T19:21:34.980 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-03-11T19:21:34.980 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-03-11T19:21:34.980 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:34.980 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:34.980 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-03-11T19:21:34.980 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-03-11T19:21:34.980 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 0, 2024-03-11T19:21:34.980 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 0, 2024-03-11T19:21:34.981 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-03-11T19:21:34.981 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.981 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.981 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.981 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:34.981 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:34.981 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-write_buf_throttle-0x55a6a0414960": { 2024-03-11T19:21:34.981 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-03-11T19:21:34.981 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 3758096384, 2024-03-11T19:21:34.981 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-03-11T19:21:34.981 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-03-11T19:21:34.981 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-03-11T19:21:34.981 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:34.981 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:34.981 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-03-11T19:21:34.982 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-03-11T19:21:34.982 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 0, 2024-03-11T19:21:34.982 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 0, 2024-03-11T19:21:34.982 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-03-11T19:21:34.982 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:34.982 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:34.982 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:34.982 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:34.982 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:34.982 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:21:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:35 smithi029 ceph-mon[32439]: pgmap v704: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.5 KiB/s wr, 84 op/s 2024-03-11T19:21:35.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:35 smithi190 ceph-mon[36292]: pgmap v704: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.5 KiB/s wr, 84 op/s 2024-03-11T19:21:35.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:35 smithi170 ceph-mon[35690]: pgmap v704: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.5 KiB/s wr, 84 op/s 2024-03-11T19:21:35.883 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-03-11T19:21:35.883 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-03-11T19:21:35.883 DEBUG:tasks.check_counter:Getting stats from d 2024-03-11T19:21:35.883 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-11T19:21:36.137 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:21:37.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:37 smithi029 ceph-mon[32439]: pgmap v705: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 18 op/s 2024-03-11T19:21:37.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:37 smithi190 ceph-mon[36292]: pgmap v705: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 18 op/s 2024-03-11T19:21:37.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:37 smithi170 ceph-mon[35690]: pgmap v705: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 18 op/s 2024-03-11T19:21:38.280 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:21:38.280 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T19:21:38.280 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 0, 2024-03-11T19:21:38.281 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 2, 2024-03-11T19:21:38.281 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 0, 2024-03-11T19:21:38.281 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 0, 2024-03-11T19:21:38.281 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 4, 2024-03-11T19:21:38.281 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 0, 2024-03-11T19:21:38.281 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.001205675, 2024-03-11T19:21:38.281 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.000000000, 2024-03-11T19:21:38.281 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.000295699, 2024-03-11T19:21:38.281 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T19:21:38.281 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:38.281 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:38.281 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:38.281 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:38.281 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:38.282 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:38.282 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:38.282 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:38.282 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:38.282 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:38.282 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T19:21:38.282 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T19:21:38.282 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:38.282 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T19:21:38.282 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 1, 2024-03-11T19:21:38.282 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 175, 2024-03-11T19:21:38.282 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 160, 2024-03-11T19:21:38.282 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 108064, 2024-03-11T19:21:38.282 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 3, 2024-03-11T19:21:38.283 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 2, 2024-03-11T19:21:38.283 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.012394241, 2024-03-11T19:21:38.283 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.009187890, 2024-03-11T19:21:38.283 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.001606613, 2024-03-11T19:21:38.283 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T19:21:38.283 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:38.283 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 173, 2024-03-11T19:21:38.283 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.006581478, 2024-03-11T19:21:38.283 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000038043 2024-03-11T19:21:38.283 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:38.283 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:38.283 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:38.283 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:38.283 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:38.283 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:38.284 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-11T19:21:38.284 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 108064 2024-03-11T19:21:38.284 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:38.284 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T19:21:38.284 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 238, 2024-03-11T19:21:38.284 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 231, 2024-03-11T19:21:38.284 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 752944, 2024-03-11T19:21:38.284 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 137872, 2024-03-11T19:21:38.284 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 2, 2024-03-11T19:21:38.284 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 1, 2024-03-11T19:21:38.284 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.034503789, 2024-03-11T19:21:38.284 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.013125443, 2024-03-11T19:21:38.284 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.018393814, 2024-03-11T19:21:38.284 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.005204469, 2024-03-11T19:21:38.284 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:38.285 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 231, 2024-03-11T19:21:38.285 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.013883057, 2024-03-11T19:21:38.285 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000060099 2024-03-11T19:21:38.285 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:38.285 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:38.285 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:38.285 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:38.285 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:38.285 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:38.285 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 752944, 2024-03-11T19:21:38.285 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 137872 2024-03-11T19:21:38.285 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:38.285 INFO:teuthology.orchestra.run.smithi029.stdout: "cct": { 2024-03-11T19:21:38.285 INFO:teuthology.orchestra.run.smithi029.stdout: "total_workers": 0, 2024-03-11T19:21:38.286 INFO:teuthology.orchestra.run.smithi029.stdout: "unhealthy_workers": 0 2024-03-11T19:21:38.286 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:38.286 INFO:teuthology.orchestra.run.smithi029.stdout: "mempool": { 2024-03-11T19:21:38.286 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_bytes": 0, 2024-03-11T19:21:38.286 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_items": 0, 2024-03-11T19:21:38.286 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T19:21:38.286 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_items": 0, 2024-03-11T19:21:38.286 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T19:21:38.286 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_items": 0, 2024-03-11T19:21:38.286 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T19:21:38.286 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T19:21:38.286 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T19:21:38.286 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T19:21:38.286 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T19:21:38.287 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_items": 0, 2024-03-11T19:21:38.287 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T19:21:38.287 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T19:21:38.287 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_bytes": 0, 2024-03-11T19:21:38.287 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_items": 0, 2024-03-11T19:21:38.287 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_bytes": 0, 2024-03-11T19:21:38.287 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_items": 0, 2024-03-11T19:21:38.287 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T19:21:38.287 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T19:21:38.287 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T19:21:38.287 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T19:21:38.287 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T19:21:38.287 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_items": 0, 2024-03-11T19:21:38.287 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_bytes": 0, 2024-03-11T19:21:38.287 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_items": 0, 2024-03-11T19:21:38.288 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T19:21:38.288 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T19:21:38.288 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_bytes": 0, 2024-03-11T19:21:38.288 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_items": 0, 2024-03-11T19:21:38.288 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_bytes": 0, 2024-03-11T19:21:38.288 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_items": 0, 2024-03-11T19:21:38.288 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T19:21:38.288 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_items": 0, 2024-03-11T19:21:38.288 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T19:21:38.288 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_items": 0, 2024-03-11T19:21:38.288 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_bytes": 148967, 2024-03-11T19:21:38.288 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_items": 29, 2024-03-11T19:21:38.288 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_bytes": 0, 2024-03-11T19:21:38.288 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_items": 0, 2024-03-11T19:21:38.289 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bytes": 0, 2024-03-11T19:21:38.289 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_items": 0, 2024-03-11T19:21:38.289 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_bytes": 0, 2024-03-11T19:21:38.289 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_items": 0, 2024-03-11T19:21:38.289 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_bytes": 0, 2024-03-11T19:21:38.289 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_items": 0, 2024-03-11T19:21:38.289 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_bytes": 0, 2024-03-11T19:21:38.289 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_items": 0, 2024-03-11T19:21:38.289 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T19:21:38.289 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_items": 0, 2024-03-11T19:21:38.289 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_bytes": 0, 2024-03-11T19:21:38.289 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_items": 0, 2024-03-11T19:21:38.289 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_bytes": 928, 2024-03-11T19:21:38.289 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_items": 2, 2024-03-11T19:21:38.289 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_bytes": 0, 2024-03-11T19:21:38.290 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_items": 0, 2024-03-11T19:21:38.290 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_bytes": 0, 2024-03-11T19:21:38.290 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_items": 0 2024-03-11T19:21:38.290 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:38.290 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T19:21:38.290 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-03-11T19:21:38.290 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-03-11T19:21:38.290 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-03-11T19:21:38.290 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 235, 2024-03-11T19:21:38.290 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 716880, 2024-03-11T19:21:38.290 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:38.290 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 235, 2024-03-11T19:21:38.290 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-03-11T19:21:38.290 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-03-11T19:21:38.291 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 235, 2024-03-11T19:21:38.291 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 716880, 2024-03-11T19:21:38.291 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-03-11T19:21:38.291 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:38.291 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:38.291 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:38.291 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:38.291 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:38.291 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:21:39.045 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-11T19:21:39.046 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-11T19:21:39.046 DEBUG:tasks.check_counter:Getting stats from g 2024-03-11T19:21:39.046 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-11T19:21:39.294 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:21:39.748 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:39 smithi029 ceph-mon[32439]: pgmap v706: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-11T19:21:39.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:39 smithi190 ceph-mon[36292]: pgmap v706: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-11T19:21:39.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:39 smithi170 ceph-mon[35690]: pgmap v706: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-11T19:21:41.109 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:21:41.109 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T19:21:41.110 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 824, 2024-03-11T19:21:41.110 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 826, 2024-03-11T19:21:41.110 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 276809, 2024-03-11T19:21:41.110 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 256265, 2024-03-11T19:21:41.110 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 6, 2024-03-11T19:21:41.110 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 2, 2024-03-11T19:21:41.110 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.090380902, 2024-03-11T19:21:41.110 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.033501823, 2024-03-11T19:21:41.110 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.020839467, 2024-03-11T19:21:41.110 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.028865031, 2024-03-11T19:21:41.110 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:41.110 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 824, 2024-03-11T19:21:41.110 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.051353609, 2024-03-11T19:21:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000062322 2024-03-11T19:21:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T19:21:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T19:21:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T19:21:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 1539, 2024-03-11T19:21:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 1708, 2024-03-11T19:21:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 395361, 2024-03-11T19:21:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 858219, 2024-03-11T19:21:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 5, 2024-03-11T19:21:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 4, 2024-03-11T19:21:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.114111603, 2024-03-11T19:21:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.037101637, 2024-03-11T19:21:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.025553280, 2024-03-11T19:21:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.044512817, 2024-03-11T19:21:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 1707, 2024-03-11T19:21:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.076062417, 2024-03-11T19:21:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000044559 2024-03-11T19:21:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-11T19:21:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 381088 2024-03-11T19:21:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T19:21:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 241, 2024-03-11T19:21:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 235, 2024-03-11T19:21:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 772484, 2024-03-11T19:21:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 137520, 2024-03-11T19:21:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 4, 2024-03-11T19:21:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 3, 2024-03-11T19:21:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.035724223, 2024-03-11T19:21:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.013498269, 2024-03-11T19:21:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.019416229, 2024-03-11T19:21:41.114 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.005356507, 2024-03-11T19:21:41.114 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:41.114 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 234, 2024-03-11T19:21:41.114 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.011042355, 2024-03-11T19:21:41.114 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000047189 2024-03-11T19:21:41.114 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.114 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:41.114 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.114 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.114 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.114 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.114 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 771136, 2024-03-11T19:21:41.114 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 136576 2024-03-11T19:21:41.114 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: "cct": { 2024-03-11T19:21:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: "total_workers": 1, 2024-03-11T19:21:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: "unhealthy_workers": 0 2024-03-11T19:21:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: "finisher-MDSRank": { 2024-03-11T19:21:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: "queue_len": 0, 2024-03-11T19:21:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: "complete_latency": { 2024-03-11T19:21:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 3836, 2024-03-11T19:21:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.100949689, 2024-03-11T19:21:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000026316 2024-03-11T19:21:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: "finisher-PurgeQueue": { 2024-03-11T19:21:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: "queue_len": 0, 2024-03-11T19:21:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: "complete_latency": { 2024-03-11T19:21:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": { 2024-03-11T19:21:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: "request": 0, 2024-03-11T19:21:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: "reply": 0, 2024-03-11T19:21:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: "reply_latency": { 2024-03-11T19:21:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: "slow_reply": 0, 2024-03-11T19:21:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: "forward": 0, 2024-03-11T19:21:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_fetch_complete": 0, 2024-03-11T19:21:41.117 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_fetch_keys": 0, 2024-03-11T19:21:41.117 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_commit": 0, 2024-03-11T19:21:41.117 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_split": 0, 2024-03-11T19:21:41.117 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_merge": 0, 2024-03-11T19:21:41.117 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes": 0, 2024-03-11T19:21:41.117 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_top": 0, 2024-03-11T19:21:41.117 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_bottom": 0, 2024-03-11T19:21:41.117 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_pin_tail": 0, 2024-03-11T19:21:41.117 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_pinned": 0, 2024-03-11T19:21:41.117 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_expired": 0, 2024-03-11T19:21:41.117 INFO:teuthology.orchestra.run.smithi029.stdout: "inodes_with_caps": 0, 2024-03-11T19:21:41.117 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": 0, 2024-03-11T19:21:41.117 INFO:teuthology.orchestra.run.smithi029.stdout: "subtrees": 1, 2024-03-11T19:21:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse": 0, 2024-03-11T19:21:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_hit": 0, 2024-03-11T19:21:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_forward": 0, 2024-03-11T19:21:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_discover": 0, 2024-03-11T19:21:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_dir_fetch": 0, 2024-03-11T19:21:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_remote_ino": 0, 2024-03-11T19:21:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: "traverse_lock": 0, 2024-03-11T19:21:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: "load_cent": 0, 2024-03-11T19:21:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: "q": 0, 2024-03-11T19:21:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: "exported": 0, 2024-03-11T19:21:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: "exported_inodes": 0, 2024-03-11T19:21:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: "imported": 0, 2024-03-11T19:21:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: "imported_inodes": 0, 2024-03-11T19:21:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: "openino_dir_fetch": 0, 2024-03-11T19:21:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: "openino_backtrace_fetch": 0, 2024-03-11T19:21:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "openino_peer_discover": 0, 2024-03-11T19:21:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "root_rfiles": 0, 2024-03-11T19:21:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "root_rbytes": 0, 2024-03-11T19:21:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "root_rsnaps": 0, 2024-03-11T19:21:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T19:21:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_set_tag": 0, 2024-03-11T19:21:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T19:21:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_inotable_repaired": 0, 2024-03-11T19:21:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_dir_inodes": 0, 2024-03-11T19:21:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T19:21:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T19:21:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_file_inodes": 0, 2024-03-11T19:21:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_inode_file_caps": 0, 2024-03-11T19:21:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T19:21:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_grant": 0, 2024-03-11T19:21:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T19:21:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T19:21:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T19:21:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_caps": 0, 2024-03-11T19:21:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_caps_dirty": 0, 2024-03-11T19:21:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_cap_release": 0, 2024-03-11T19:21:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: "process_request_cap_release": 0 2024-03-11T19:21:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache": { 2024-03-11T19:21:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_update": 0, 2024-03-11T19:21:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_update_receipt": 0, 2024-03-11T19:21:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_try_discover": 0, 2024-03-11T19:21:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_send_discover": 0, 2024-03-11T19:21:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: "dir_handle_discover": 0, 2024-03-11T19:21:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: "num_strays": 0, 2024-03-11T19:21:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: "num_strays_delayed": 0, 2024-03-11T19:21:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: "num_strays_enqueuing": 0, 2024-03-11T19:21:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_created": 0, 2024-03-11T19:21:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_enqueued": 0, 2024-03-11T19:21:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_reintegrated": 0, 2024-03-11T19:21:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: "strays_migrated": 0, 2024-03-11T19:21:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: "num_recovering_processing": 0, 2024-03-11T19:21:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: "num_recovering_enqueued": 0, 2024-03-11T19:21:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: "num_recovering_prioritized": 0, 2024-03-11T19:21:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: "recovery_started": 0, 2024-03-11T19:21:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: "recovery_completed": 0, 2024-03-11T19:21:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T19:21:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_exportdir": 0, 2024-03-11T19:21:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_flush": 0, 2024-03-11T19:21:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_fragmentdir": 0, 2024-03-11T19:21:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_fragstats": 0, 2024-03-11T19:21:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: "ireq_inodestats": 0 2024-03-11T19:21:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log": { 2024-03-11T19:21:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: "evlrg": 0, 2024-03-11T19:21:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: "evadd": 0, 2024-03-11T19:21:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: "evex": 0, 2024-03-11T19:21:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: "evtrm": 0, 2024-03-11T19:21:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: "ev": 5, 2024-03-11T19:21:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: "evexg": 0, 2024-03-11T19:21:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: "evexd": 0, 2024-03-11T19:21:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: "segadd": 0, 2024-03-11T19:21:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: "segex": 0, 2024-03-11T19:21:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: "segtrm": 0, 2024-03-11T19:21:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: "seg": 2, 2024-03-11T19:21:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: "segmjr": 2, 2024-03-11T19:21:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: "segexg": 0, 2024-03-11T19:21:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: "segexd": 0, 2024-03-11T19:21:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: "expos": 4194304, 2024-03-11T19:21:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: "wrpos": 4195917, 2024-03-11T19:21:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: "rdpos": 4195776, 2024-03-11T19:21:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: "jlat": { 2024-03-11T19:21:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: "replayed": 5 2024-03-11T19:21:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_mem": { 2024-03-11T19:21:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: "ino": 3, 2024-03-11T19:21:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: "ino+": 3, 2024-03-11T19:21:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: "ino-": 0, 2024-03-11T19:21:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: "dir": 1, 2024-03-11T19:21:41.124 INFO:teuthology.orchestra.run.smithi029.stdout: "dir+": 1, 2024-03-11T19:21:41.124 INFO:teuthology.orchestra.run.smithi029.stdout: "dir-": 0, 2024-03-11T19:21:41.124 INFO:teuthology.orchestra.run.smithi029.stdout: "dn": 0, 2024-03-11T19:21:41.124 INFO:teuthology.orchestra.run.smithi029.stdout: "dn+": 0, 2024-03-11T19:21:41.124 INFO:teuthology.orchestra.run.smithi029.stdout: "dn-": 0, 2024-03-11T19:21:41.124 INFO:teuthology.orchestra.run.smithi029.stdout: "cap": 0, 2024-03-11T19:21:41.124 INFO:teuthology.orchestra.run.smithi029.stdout: "cap+": 0, 2024-03-11T19:21:41.124 INFO:teuthology.orchestra.run.smithi029.stdout: "cap-": 0, 2024-03-11T19:21:41.124 INFO:teuthology.orchestra.run.smithi029.stdout: "rss": 49704, 2024-03-11T19:21:41.124 INFO:teuthology.orchestra.run.smithi029.stdout: "heap": 215420 2024-03-11T19:21:41.124 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.124 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_server": { 2024-03-11T19:21:41.124 INFO:teuthology.orchestra.run.smithi029.stdout: "dispatch_client_request": 0, 2024-03-11T19:21:41.124 INFO:teuthology.orchestra.run.smithi029.stdout: "dispatch_server_request": 0, 2024-03-11T19:21:41.124 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_request": 0, 2024-03-11T19:21:41.124 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_client_session": 0, 2024-03-11T19:21:41.124 INFO:teuthology.orchestra.run.smithi029.stdout: "handle_peer_request": 0, 2024-03-11T19:21:41.125 INFO:teuthology.orchestra.run.smithi029.stdout: "req_create_latency": { 2024-03-11T19:21:41.125 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.125 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.125 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.125 INFO:teuthology.orchestra.run.smithi029.stdout: "req_getattr_latency": { 2024-03-11T19:21:41.125 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.125 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.125 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.125 INFO:teuthology.orchestra.run.smithi029.stdout: "req_getfilelock_latency": { 2024-03-11T19:21:41.125 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.125 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.125 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.125 INFO:teuthology.orchestra.run.smithi029.stdout: "req_link_latency": { 2024-03-11T19:21:41.125 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.126 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.126 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.126 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.126 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookup_latency": { 2024-03-11T19:21:41.126 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.126 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.126 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.126 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.126 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookuphash_latency": { 2024-03-11T19:21:41.126 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.126 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.126 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.126 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.126 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupino_latency": { 2024-03-11T19:21:41.127 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.127 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.127 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.127 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.127 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupname_latency": { 2024-03-11T19:21:41.127 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.127 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.127 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.127 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.127 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupparent_latency": { 2024-03-11T19:21:41.127 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.127 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.127 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.127 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.127 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lookupsnap_latency": { 2024-03-11T19:21:41.127 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.127 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.128 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.128 INFO:teuthology.orchestra.run.smithi029.stdout: "req_lssnap_latency": { 2024-03-11T19:21:41.128 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.128 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.128 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.128 INFO:teuthology.orchestra.run.smithi029.stdout: "req_mkdir_latency": { 2024-03-11T19:21:41.128 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.128 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.128 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.129 INFO:teuthology.orchestra.run.smithi029.stdout: "req_mknod_latency": { 2024-03-11T19:21:41.129 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.129 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.129 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.129 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.129 INFO:teuthology.orchestra.run.smithi029.stdout: "req_mksnap_latency": { 2024-03-11T19:21:41.129 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.129 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.129 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.129 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.129 INFO:teuthology.orchestra.run.smithi029.stdout: "req_open_latency": { 2024-03-11T19:21:41.129 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.129 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.129 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.130 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.130 INFO:teuthology.orchestra.run.smithi029.stdout: "req_readdir_latency": { 2024-03-11T19:21:41.130 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.130 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.130 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.130 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.130 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rename_latency": { 2024-03-11T19:21:41.130 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.130 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.130 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.130 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.130 INFO:teuthology.orchestra.run.smithi029.stdout: "req_renamesnap_latency": { 2024-03-11T19:21:41.130 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.130 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.131 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.131 INFO:teuthology.orchestra.run.smithi029.stdout: "req_snapdiff_latency": { 2024-03-11T19:21:41.131 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.131 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.131 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.131 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rmdir_latency": { 2024-03-11T19:21:41.131 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.131 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.131 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.131 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rmsnap_latency": { 2024-03-11T19:21:41.131 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.131 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.131 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.132 INFO:teuthology.orchestra.run.smithi029.stdout: "req_rmxattr_latency": { 2024-03-11T19:21:41.132 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.132 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.132 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.132 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setattr_latency": { 2024-03-11T19:21:41.132 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.132 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.132 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.132 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setdirlayout_latency": { 2024-03-11T19:21:41.132 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.132 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.132 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.132 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setfilelock_latency": { 2024-03-11T19:21:41.133 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.133 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.133 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.133 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.133 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setlayout_latency": { 2024-03-11T19:21:41.133 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.133 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.133 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.133 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.133 INFO:teuthology.orchestra.run.smithi029.stdout: "req_setxattr_latency": { 2024-03-11T19:21:41.133 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.133 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.133 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.133 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.133 INFO:teuthology.orchestra.run.smithi029.stdout: "req_symlink_latency": { 2024-03-11T19:21:41.133 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.133 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.134 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.134 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.134 INFO:teuthology.orchestra.run.smithi029.stdout: "req_unlink_latency": { 2024-03-11T19:21:41.134 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.134 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.134 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.134 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.134 INFO:teuthology.orchestra.run.smithi029.stdout: "cap_revoke_eviction": 0, 2024-03-11T19:21:41.134 INFO:teuthology.orchestra.run.smithi029.stdout: "cap_acquisition_throttle": 0, 2024-03-11T19:21:41.134 INFO:teuthology.orchestra.run.smithi029.stdout: "req_getvxattr_latency": { 2024-03-11T19:21:41.134 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.134 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.134 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.134 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:41.135 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.135 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_sessions": { 2024-03-11T19:21:41.135 INFO:teuthology.orchestra.run.smithi029.stdout: "session_count": 1, 2024-03-11T19:21:41.135 INFO:teuthology.orchestra.run.smithi029.stdout: "session_add": 2, 2024-03-11T19:21:41.135 INFO:teuthology.orchestra.run.smithi029.stdout: "session_remove": 1, 2024-03-11T19:21:41.135 INFO:teuthology.orchestra.run.smithi029.stdout: "sessions_open": 2, 2024-03-11T19:21:41.135 INFO:teuthology.orchestra.run.smithi029.stdout: "sessions_stale": 0, 2024-03-11T19:21:41.135 INFO:teuthology.orchestra.run.smithi029.stdout: "total_load": 0, 2024-03-11T19:21:41.135 INFO:teuthology.orchestra.run.smithi029.stdout: "average_load": 0, 2024-03-11T19:21:41.135 INFO:teuthology.orchestra.run.smithi029.stdout: "avg_session_uptime": 2991, 2024-03-11T19:21:41.135 INFO:teuthology.orchestra.run.smithi029.stdout: "mdthresh_evicted": 0 2024-03-11T19:21:41.135 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.135 INFO:teuthology.orchestra.run.smithi029.stdout: "mempool": { 2024-03-11T19:21:41.135 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_bytes": 0, 2024-03-11T19:21:41.135 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_items": 0, 2024-03-11T19:21:41.136 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T19:21:41.136 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_items": 0, 2024-03-11T19:21:41.136 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T19:21:41.136 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_items": 0, 2024-03-11T19:21:41.136 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T19:21:41.136 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T19:21:41.136 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T19:21:41.136 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T19:21:41.136 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T19:21:41.136 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_items": 0, 2024-03-11T19:21:41.136 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T19:21:41.136 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T19:21:41.136 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_bytes": 0, 2024-03-11T19:21:41.136 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_items": 0, 2024-03-11T19:21:41.136 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_bytes": 0, 2024-03-11T19:21:41.137 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_items": 0, 2024-03-11T19:21:41.137 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T19:21:41.137 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T19:21:41.137 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T19:21:41.137 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T19:21:41.137 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T19:21:41.137 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_items": 0, 2024-03-11T19:21:41.137 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_bytes": 0, 2024-03-11T19:21:41.137 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_items": 0, 2024-03-11T19:21:41.137 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T19:21:41.137 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T19:21:41.137 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_bytes": 0, 2024-03-11T19:21:41.137 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_items": 0, 2024-03-11T19:21:41.137 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_bytes": 0, 2024-03-11T19:21:41.137 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_items": 0, 2024-03-11T19:21:41.137 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T19:21:41.138 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_items": 0, 2024-03-11T19:21:41.138 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T19:21:41.138 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_items": 0, 2024-03-11T19:21:41.138 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_bytes": 197463, 2024-03-11T19:21:41.138 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_items": 54, 2024-03-11T19:21:41.138 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_bytes": 0, 2024-03-11T19:21:41.138 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_items": 0, 2024-03-11T19:21:41.138 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bytes": 0, 2024-03-11T19:21:41.138 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_items": 0, 2024-03-11T19:21:41.138 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_bytes": 0, 2024-03-11T19:21:41.138 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_items": 0, 2024-03-11T19:21:41.138 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_bytes": 0, 2024-03-11T19:21:41.138 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_items": 0, 2024-03-11T19:21:41.138 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_bytes": 7336, 2024-03-11T19:21:41.138 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_items": 154, 2024-03-11T19:21:41.139 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T19:21:41.139 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_items": 0, 2024-03-11T19:21:41.139 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_bytes": 0, 2024-03-11T19:21:41.139 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_items": 0, 2024-03-11T19:21:41.139 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_bytes": 8320, 2024-03-11T19:21:41.139 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_items": 13, 2024-03-11T19:21:41.139 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_bytes": 0, 2024-03-11T19:21:41.139 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_items": 0, 2024-03-11T19:21:41.139 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_bytes": 0, 2024-03-11T19:21:41.139 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_items": 0 2024-03-11T19:21:41.139 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.139 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter": { 2024-03-11T19:21:41.139 INFO:teuthology.orchestra.run.smithi029.stdout: "op_active": 0, 2024-03-11T19:21:41.139 INFO:teuthology.orchestra.run.smithi029.stdout: "op_laggy": 0, 2024-03-11T19:21:41.139 INFO:teuthology.orchestra.run.smithi029.stdout: "op_send": 2361, 2024-03-11T19:21:41.140 INFO:teuthology.orchestra.run.smithi029.stdout: "op_send_bytes": 21, 2024-03-11T19:21:41.140 INFO:teuthology.orchestra.run.smithi029.stdout: "op_resend": 3, 2024-03-11T19:21:41.140 INFO:teuthology.orchestra.run.smithi029.stdout: "op_reply": 2361, 2024-03-11T19:21:41.140 INFO:teuthology.orchestra.run.smithi029.stdout: "op_latency": { 2024-03-11T19:21:41.140 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 2361, 2024-03-11T19:21:41.140 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 2.224964632, 2024-03-11T19:21:41.140 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000942382 2024-03-11T19:21:41.140 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.140 INFO:teuthology.orchestra.run.smithi029.stdout: "op_inflight": 0, 2024-03-11T19:21:41.140 INFO:teuthology.orchestra.run.smithi029.stdout: "oplen_avg": { 2024-03-11T19:21:41.140 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 2361, 2024-03-11T19:21:41.140 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 2362 2024-03-11T19:21:41.140 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.140 INFO:teuthology.orchestra.run.smithi029.stdout: "op": 2361, 2024-03-11T19:21:41.140 INFO:teuthology.orchestra.run.smithi029.stdout: "op_r": 2361, 2024-03-11T19:21:41.140 INFO:teuthology.orchestra.run.smithi029.stdout: "op_w": 0, 2024-03-11T19:21:41.141 INFO:teuthology.orchestra.run.smithi029.stdout: "op_rmw": 0, 2024-03-11T19:21:41.141 INFO:teuthology.orchestra.run.smithi029.stdout: "op_pg": 0, 2024-03-11T19:21:41.141 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_stat": 1529, 2024-03-11T19:21:41.141 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_create": 0, 2024-03-11T19:21:41.141 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_read": 830, 2024-03-11T19:21:41.141 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_write": 0, 2024-03-11T19:21:41.141 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_writefull": 0, 2024-03-11T19:21:41.141 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_writesame": 0, 2024-03-11T19:21:41.141 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_append": 0, 2024-03-11T19:21:41.141 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_zero": 0, 2024-03-11T19:21:41.141 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_truncate": 0, 2024-03-11T19:21:41.141 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_delete": 0, 2024-03-11T19:21:41.141 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_mapext": 0, 2024-03-11T19:21:41.141 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_sparse_read": 0, 2024-03-11T19:21:41.141 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_clonerange": 0, 2024-03-11T19:21:41.142 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_getxattr": 1, 2024-03-11T19:21:41.142 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_setxattr": 0, 2024-03-11T19:21:41.142 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_cmpxattr": 0, 2024-03-11T19:21:41.142 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_rmxattr": 0, 2024-03-11T19:21:41.142 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_resetxattrs": 0, 2024-03-11T19:21:41.142 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_call": 0, 2024-03-11T19:21:41.142 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_watch": 0, 2024-03-11T19:21:41.142 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_notify": 0, 2024-03-11T19:21:41.142 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T19:21:41.142 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_pgls": 0, 2024-03-11T19:21:41.142 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_pgls_filter": 0, 2024-03-11T19:21:41.142 INFO:teuthology.orchestra.run.smithi029.stdout: "osdop_other": 0, 2024-03-11T19:21:41.142 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_active": 0, 2024-03-11T19:21:41.142 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_send": 0, 2024-03-11T19:21:41.143 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_resend": 0, 2024-03-11T19:21:41.143 INFO:teuthology.orchestra.run.smithi029.stdout: "linger_ping": 0, 2024-03-11T19:21:41.143 INFO:teuthology.orchestra.run.smithi029.stdout: "poolop_active": 0, 2024-03-11T19:21:41.143 INFO:teuthology.orchestra.run.smithi029.stdout: "poolop_send": 0, 2024-03-11T19:21:41.143 INFO:teuthology.orchestra.run.smithi029.stdout: "poolop_resend": 0, 2024-03-11T19:21:41.143 INFO:teuthology.orchestra.run.smithi029.stdout: "poolstat_active": 0, 2024-03-11T19:21:41.143 INFO:teuthology.orchestra.run.smithi029.stdout: "poolstat_send": 0, 2024-03-11T19:21:41.143 INFO:teuthology.orchestra.run.smithi029.stdout: "poolstat_resend": 0, 2024-03-11T19:21:41.143 INFO:teuthology.orchestra.run.smithi029.stdout: "statfs_active": 0, 2024-03-11T19:21:41.143 INFO:teuthology.orchestra.run.smithi029.stdout: "statfs_send": 0, 2024-03-11T19:21:41.143 INFO:teuthology.orchestra.run.smithi029.stdout: "statfs_resend": 0, 2024-03-11T19:21:41.143 INFO:teuthology.orchestra.run.smithi029.stdout: "command_active": 0, 2024-03-11T19:21:41.143 INFO:teuthology.orchestra.run.smithi029.stdout: "command_send": 0, 2024-03-11T19:21:41.143 INFO:teuthology.orchestra.run.smithi029.stdout: "command_resend": 0, 2024-03-11T19:21:41.143 INFO:teuthology.orchestra.run.smithi029.stdout: "map_epoch": 89, 2024-03-11T19:21:41.144 INFO:teuthology.orchestra.run.smithi029.stdout: "map_full": 0, 2024-03-11T19:21:41.144 INFO:teuthology.orchestra.run.smithi029.stdout: "map_inc": 2, 2024-03-11T19:21:41.144 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_sessions": 6, 2024-03-11T19:21:41.144 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_session_open": 6, 2024-03-11T19:21:41.144 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_session_close": 0, 2024-03-11T19:21:41.144 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_laggy": 0, 2024-03-11T19:21:41.144 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_wr": 0, 2024-03-11T19:21:41.144 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_rd": 2, 2024-03-11T19:21:41.144 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_del": 0 2024-03-11T19:21:41.144 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.144 INFO:teuthology.orchestra.run.smithi029.stdout: "oft": { 2024-03-11T19:21:41.144 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_objs": 0, 2024-03-11T19:21:41.144 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_kv_pairs": 0, 2024-03-11T19:21:41.144 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_updates": 0, 2024-03-11T19:21:41.144 INFO:teuthology.orchestra.run.smithi029.stdout: "omap_total_removes": 0 2024-03-11T19:21:41.145 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.145 INFO:teuthology.orchestra.run.smithi029.stdout: "osd-slow-ops": { 2024-03-11T19:21:41.145 INFO:teuthology.orchestra.run.smithi029.stdout: "slow_ops_count": 0 2024-03-11T19:21:41.145 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.145 INFO:teuthology.orchestra.run.smithi029.stdout: "purge_queue": { 2024-03-11T19:21:41.145 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing_ops": 0, 2024-03-11T19:21:41.145 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T19:21:41.145 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing": 0, 2024-03-11T19:21:41.145 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executing_high_water": 0, 2024-03-11T19:21:41.145 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executed_ops": 0, 2024-03-11T19:21:41.145 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_executed": 0, 2024-03-11T19:21:41.145 INFO:teuthology.orchestra.run.smithi029.stdout: "pq_item_in_journal": 0 2024-03-11T19:21:41.145 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.145 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T19:21:41.145 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-03-11T19:21:41.146 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-03-11T19:21:41.146 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-03-11T19:21:41.146 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 2600, 2024-03-11T19:21:41.146 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 1195435, 2024-03-11T19:21:41.146 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:41.146 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 2600, 2024-03-11T19:21:41.146 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-03-11T19:21:41.146 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-03-11T19:21:41.146 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 2600, 2024-03-11T19:21:41.146 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 1195435, 2024-03-11T19:21:41.146 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-03-11T19:21:41.146 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.146 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.146 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.147 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:41.147 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.147 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-objecter_bytes": { 2024-03-11T19:21:41.147 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-03-11T19:21:41.147 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-03-11T19:21:41.147 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-03-11T19:21:41.147 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-03-11T19:21:41.147 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-03-11T19:21:41.147 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:41.147 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:41.147 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 2361, 2024-03-11T19:21:41.147 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 1618, 2024-03-11T19:21:41.147 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 2361, 2024-03-11T19:21:41.147 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 1618, 2024-03-11T19:21:41.147 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-03-11T19:21:41.147 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.148 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.148 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.148 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:41.148 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.148 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-objecter_ops": { 2024-03-11T19:21:41.148 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-03-11T19:21:41.148 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 1024, 2024-03-11T19:21:41.148 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-03-11T19:21:41.148 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-03-11T19:21:41.148 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-03-11T19:21:41.148 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:41.148 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:41.148 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 2361, 2024-03-11T19:21:41.148 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 2361, 2024-03-11T19:21:41.148 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 2361, 2024-03-11T19:21:41.149 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 2361, 2024-03-11T19:21:41.149 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-03-11T19:21:41.149 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.149 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.149 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.149 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:41.149 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.149 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-write_buf_throttle": { 2024-03-11T19:21:41.149 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-03-11T19:21:41.149 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 3758096384, 2024-03-11T19:21:41.149 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-03-11T19:21:41.149 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-03-11T19:21:41.149 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-03-11T19:21:41.149 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:41.149 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:41.150 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-03-11T19:21:41.150 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-03-11T19:21:41.150 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 0, 2024-03-11T19:21:41.150 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 0, 2024-03-11T19:21:41.150 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-03-11T19:21:41.150 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.150 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.150 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.150 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:41.150 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:41.150 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-write_buf_throttle-0x55f7792a2960": { 2024-03-11T19:21:41.150 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-03-11T19:21:41.150 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 3758096384, 2024-03-11T19:21:41.150 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-03-11T19:21:41.151 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 0, 2024-03-11T19:21:41.151 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 0, 2024-03-11T19:21:41.151 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:41.151 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:41.151 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-03-11T19:21:41.151 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-03-11T19:21:41.151 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 0, 2024-03-11T19:21:41.151 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 0, 2024-03-11T19:21:41.151 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-03-11T19:21:41.151 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:41.151 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:41.151 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:41.151 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:41.151 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:41.151 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:21:41.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:41 smithi190 ceph-mon[36292]: pgmap v707: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:21:41.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:41 smithi170 ceph-mon[35690]: pgmap v707: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:21:41.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:41 smithi029 ceph-mon[32439]: pgmap v707: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:21:41.913 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-03-11T19:21:41.914 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-03-11T19:21:41.914 DEBUG:tasks.check_counter:Getting stats from j 2024-03-11T19:21:41.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-11T19:21:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:21:43.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:43 smithi029 ceph-mon[32439]: pgmap v708: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:21:43.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:43 smithi190 ceph-mon[36292]: pgmap v708: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:21:43.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:43 smithi170 ceph-mon[35690]: pgmap v708: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:21:43.996 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-11T19:21:43.997 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T19:21:43.997 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 0, 2024-03-11T19:21:43.997 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 2, 2024-03-11T19:21:43.997 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 0, 2024-03-11T19:21:43.997 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 0, 2024-03-11T19:21:43.997 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 4, 2024-03-11T19:21:43.997 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 0, 2024-03-11T19:21:43.997 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.001409312, 2024-03-11T19:21:43.997 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.000000000, 2024-03-11T19:21:43.997 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.000533004, 2024-03-11T19:21:43.997 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T19:21:43.997 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:43.997 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:43.998 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:43.998 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:43.998 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:43.998 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:43.998 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:43.998 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:43.998 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:43.998 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:43.998 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T19:21:43.998 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T19:21:43.998 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:43.998 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T19:21:43.998 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 5, 2024-03-11T19:21:43.998 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 178, 2024-03-11T19:21:43.999 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 3152, 2024-03-11T19:21:43.999 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 108688, 2024-03-11T19:21:43.999 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 3, 2024-03-11T19:21:43.999 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 2, 2024-03-11T19:21:43.999 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.012914710, 2024-03-11T19:21:43.999 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.009596239, 2024-03-11T19:21:43.999 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.001695609, 2024-03-11T19:21:43.999 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T19:21:43.999 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:43.999 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 177, 2024-03-11T19:21:43.999 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.008456926, 2024-03-11T19:21:43.999 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000047779 2024-03-11T19:21:43.999 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:43.999 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:43.999 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:44.000 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:44.000 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:44.000 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:44.000 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-11T19:21:44.000 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 108688 2024-03-11T19:21:44.000 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:44.000 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T19:21:44.000 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 237, 2024-03-11T19:21:44.000 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 232, 2024-03-11T19:21:44.000 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 750768, 2024-03-11T19:21:44.000 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 139296, 2024-03-11T19:21:44.000 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 2, 2024-03-11T19:21:44.000 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 1, 2024-03-11T19:21:44.000 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.037644178, 2024-03-11T19:21:44.001 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.013397060, 2024-03-11T19:21:44.001 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.019223886, 2024-03-11T19:21:44.001 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.007486031, 2024-03-11T19:21:44.001 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:44.001 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 231, 2024-03-11T19:21:44.001 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.014580279, 2024-03-11T19:21:44.001 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000063118 2024-03-11T19:21:44.001 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:44.001 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:44.001 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:44.001 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:44.001 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:44.001 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:44.001 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 750768, 2024-03-11T19:21:44.001 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 139296 2024-03-11T19:21:44.002 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:44.002 INFO:teuthology.orchestra.run.smithi029.stdout: "cct": { 2024-03-11T19:21:44.002 INFO:teuthology.orchestra.run.smithi029.stdout: "total_workers": 0, 2024-03-11T19:21:44.002 INFO:teuthology.orchestra.run.smithi029.stdout: "unhealthy_workers": 0 2024-03-11T19:21:44.002 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:44.002 INFO:teuthology.orchestra.run.smithi029.stdout: "mempool": { 2024-03-11T19:21:44.002 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_bytes": 0, 2024-03-11T19:21:44.002 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_items": 0, 2024-03-11T19:21:44.002 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T19:21:44.002 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_items": 0, 2024-03-11T19:21:44.002 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T19:21:44.002 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_items": 0, 2024-03-11T19:21:44.002 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T19:21:44.002 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T19:21:44.003 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T19:21:44.003 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T19:21:44.003 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T19:21:44.003 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_items": 0, 2024-03-11T19:21:44.003 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T19:21:44.003 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T19:21:44.003 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_bytes": 0, 2024-03-11T19:21:44.003 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_items": 0, 2024-03-11T19:21:44.003 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_bytes": 0, 2024-03-11T19:21:44.003 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_items": 0, 2024-03-11T19:21:44.003 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T19:21:44.003 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T19:21:44.003 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T19:21:44.003 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T19:21:44.003 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T19:21:44.004 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_items": 0, 2024-03-11T19:21:44.004 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_bytes": 0, 2024-03-11T19:21:44.004 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_items": 0, 2024-03-11T19:21:44.004 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T19:21:44.004 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T19:21:44.004 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_bytes": 0, 2024-03-11T19:21:44.004 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_items": 0, 2024-03-11T19:21:44.004 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_bytes": 0, 2024-03-11T19:21:44.004 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_items": 0, 2024-03-11T19:21:44.004 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T19:21:44.004 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_items": 0, 2024-03-11T19:21:44.004 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T19:21:44.004 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_items": 0, 2024-03-11T19:21:44.004 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_bytes": 149998, 2024-03-11T19:21:44.005 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_items": 30, 2024-03-11T19:21:44.005 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_bytes": 0, 2024-03-11T19:21:44.005 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_items": 0, 2024-03-11T19:21:44.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bytes": 0, 2024-03-11T19:21:44.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_items": 0, 2024-03-11T19:21:44.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_bytes": 0, 2024-03-11T19:21:44.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_items": 0, 2024-03-11T19:21:44.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_bytes": 0, 2024-03-11T19:21:44.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_items": 0, 2024-03-11T19:21:44.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_bytes": 0, 2024-03-11T19:21:44.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_items": 0, 2024-03-11T19:21:44.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T19:21:44.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_items": 0, 2024-03-11T19:21:44.005 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_bytes": 0, 2024-03-11T19:21:44.005 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_items": 0, 2024-03-11T19:21:44.006 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_bytes": 928, 2024-03-11T19:21:44.006 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_items": 2, 2024-03-11T19:21:44.006 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_bytes": 0, 2024-03-11T19:21:44.006 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_items": 0, 2024-03-11T19:21:44.006 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_bytes": 0, 2024-03-11T19:21:44.006 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_items": 0 2024-03-11T19:21:44.006 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-11T19:21:44.006 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T19:21:44.006 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-03-11T19:21:44.006 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-03-11T19:21:44.006 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-03-11T19:21:44.006 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 238, 2024-03-11T19:21:44.006 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 717270, 2024-03-11T19:21:44.006 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:44.007 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 238, 2024-03-11T19:21:44.007 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-03-11T19:21:44.007 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-03-11T19:21:44.007 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 238, 2024-03-11T19:21:44.007 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 717270, 2024-03-11T19:21:44.007 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-03-11T19:21:44.007 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-11T19:21:44.007 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-11T19:21:44.007 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-11T19:21:44.007 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:44.007 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-11T19:21:44.007 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-11T19:21:44.711 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-11T19:21:44.711 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-11T19:21:44.711 DEBUG:tasks.check_counter:Getting stats from b 2024-03-11T19:21:44.711 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-11T19:21:44.964 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:21:45.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:45 smithi190 ceph-mon[36292]: pgmap v709: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:21:45.838 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:45 smithi170 ceph-mon[35690]: pgmap v709: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:21:45.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:45 smithi029 ceph-mon[32439]: pgmap v709: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:21:46.819 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-03-11T19:21:46.819 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T19:21:46.819 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 2254, 2024-03-11T19:21:46.819 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 2060, 2024-03-11T19:21:46.819 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 653429, 2024-03-11T19:21:46.819 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 1537741, 2024-03-11T19:21:46.819 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 32, 2024-03-11T19:21:46.820 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 5, 2024-03-11T19:21:46.820 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.175533480, 2024-03-11T19:21:46.820 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.046624771, 2024-03-11T19:21:46.820 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.053327788, 2024-03-11T19:21:46.820 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.053822654, 2024-03-11T19:21:46.820 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:46.820 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2057, 2024-03-11T19:21:46.820 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.307021474, 2024-03-11T19:21:46.820 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000149256 2024-03-11T19:21:46.820 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.820 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:46.820 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 800, 2024-03-11T19:21:46.820 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000203257, 2024-03-11T19:21:46.820 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000254 2024-03-11T19:21:46.820 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.820 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-11T19:21:46.821 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 448 2024-03-11T19:21:46.821 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.821 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T19:21:46.821 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 2469, 2024-03-11T19:21:46.821 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 2260, 2024-03-11T19:21:46.821 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 778564, 2024-03-11T19:21:46.821 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 1002906, 2024-03-11T19:21:46.821 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 9, 2024-03-11T19:21:46.821 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 6, 2024-03-11T19:21:46.821 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.197755825, 2024-03-11T19:21:46.821 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.058118705, 2024-03-11T19:21:46.821 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.061186816, 2024-03-11T19:21:46.821 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.068548221, 2024-03-11T19:21:46.821 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:46.822 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2260, 2024-03-11T19:21:46.822 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.471097369, 2024-03-11T19:21:46.822 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000208450 2024-03-11T19:21:46.822 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.822 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:46.822 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 1589, 2024-03-11T19:21:46.822 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000292034, 2024-03-11T19:21:46.822 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000183 2024-03-11T19:21:46.822 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.822 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-11T19:21:46.822 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 388464 2024-03-11T19:21:46.822 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.822 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T19:21:46.822 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 2863, 2024-03-11T19:21:46.822 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 1993, 2024-03-11T19:21:46.823 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 1700536, 2024-03-11T19:21:46.823 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 2084176, 2024-03-11T19:21:46.823 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 154, 2024-03-11T19:21:46.823 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 6, 2024-03-11T19:21:46.823 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.306179759, 2024-03-11T19:21:46.823 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.069365109, 2024-03-11T19:21:46.823 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.143859251, 2024-03-11T19:21:46.823 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.088456514, 2024-03-11T19:21:46.823 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:46.823 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 1992, 2024-03-11T19:21:46.823 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.244260846, 2024-03-11T19:21:46.823 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000122620 2024-03-11T19:21:46.823 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.823 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:46.823 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2558, 2024-03-11T19:21:46.823 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000503514, 2024-03-11T19:21:46.824 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000196 2024-03-11T19:21:46.824 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.824 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 893280, 2024-03-11T19:21:46.824 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 193120 2024-03-11T19:21:46.824 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.824 INFO:teuthology.orchestra.run.smithi170.stdout: "cct": { 2024-03-11T19:21:46.824 INFO:teuthology.orchestra.run.smithi170.stdout: "total_workers": 1, 2024-03-11T19:21:46.824 INFO:teuthology.orchestra.run.smithi170.stdout: "unhealthy_workers": 0 2024-03-11T19:21:46.824 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.824 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-MDSRank": { 2024-03-11T19:21:46.824 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-03-11T19:21:46.824 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-03-11T19:21:46.824 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 738, 2024-03-11T19:21:46.824 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.250789697, 2024-03-11T19:21:46.824 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000339823 2024-03-11T19:21:46.824 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:46.824 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.825 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-PurgeQueue": { 2024-03-11T19:21:46.825 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-03-11T19:21:46.825 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-03-11T19:21:46.825 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2477, 2024-03-11T19:21:46.825 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.076496941, 2024-03-11T19:21:46.825 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000030882 2024-03-11T19:21:46.825 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:46.825 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.825 INFO:teuthology.orchestra.run.smithi170.stdout: "mds": { 2024-03-11T19:21:46.825 INFO:teuthology.orchestra.run.smithi170.stdout: "request": 99, 2024-03-11T19:21:46.825 INFO:teuthology.orchestra.run.smithi170.stdout: "reply": 99, 2024-03-11T19:21:46.825 INFO:teuthology.orchestra.run.smithi170.stdout: "reply_latency": { 2024-03-11T19:21:46.825 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 99, 2024-03-11T19:21:46.825 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.135947040, 2024-03-11T19:21:46.825 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.001373202 2024-03-11T19:21:46.826 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.826 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_reply": 0, 2024-03-11T19:21:46.826 INFO:teuthology.orchestra.run.smithi170.stdout: "forward": 0, 2024-03-11T19:21:46.826 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_complete": 0, 2024-03-11T19:21:46.826 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_keys": 0, 2024-03-11T19:21:46.826 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_commit": 12, 2024-03-11T19:21:46.826 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_split": 0, 2024-03-11T19:21:46.826 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_merge": 0, 2024-03-11T19:21:46.826 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes": 23, 2024-03-11T19:21:46.826 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_top": 0, 2024-03-11T19:21:46.826 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_bottom": 23, 2024-03-11T19:21:46.826 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pin_tail": 0, 2024-03-11T19:21:46.826 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pinned": 23, 2024-03-11T19:21:46.827 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_expired": 0, 2024-03-11T19:21:46.827 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_with_caps": 2, 2024-03-11T19:21:46.827 INFO:teuthology.orchestra.run.smithi170.stdout: "caps": 2, 2024-03-11T19:21:46.827 INFO:teuthology.orchestra.run.smithi170.stdout: "subtrees": 2, 2024-03-11T19:21:46.827 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse": 151, 2024-03-11T19:21:46.827 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_hit": 140, 2024-03-11T19:21:46.827 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_forward": 0, 2024-03-11T19:21:46.827 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_discover": 0, 2024-03-11T19:21:46.827 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_dir_fetch": 0, 2024-03-11T19:21:46.827 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_remote_ino": 0, 2024-03-11T19:21:46.827 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_lock": 0, 2024-03-11T19:21:46.827 INFO:teuthology.orchestra.run.smithi170.stdout: "load_cent": 0, 2024-03-11T19:21:46.827 INFO:teuthology.orchestra.run.smithi170.stdout: "q": 0, 2024-03-11T19:21:46.827 INFO:teuthology.orchestra.run.smithi170.stdout: "exported": 0, 2024-03-11T19:21:46.828 INFO:teuthology.orchestra.run.smithi170.stdout: "exported_inodes": 0, 2024-03-11T19:21:46.828 INFO:teuthology.orchestra.run.smithi170.stdout: "imported": 0, 2024-03-11T19:21:46.828 INFO:teuthology.orchestra.run.smithi170.stdout: "imported_inodes": 0, 2024-03-11T19:21:46.828 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_dir_fetch": 0, 2024-03-11T19:21:46.828 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_backtrace_fetch": 597, 2024-03-11T19:21:46.828 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_peer_discover": 0, 2024-03-11T19:21:46.828 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rfiles": 4, 2024-03-11T19:21:46.828 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rbytes": 234, 2024-03-11T19:21:46.828 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rsnaps": 0, 2024-03-11T19:21:46.828 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_fetch": 597, 2024-03-11T19:21:46.828 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_set_tag": 0, 2024-03-11T19:21:46.828 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T19:21:46.828 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_inotable_repaired": 0, 2024-03-11T19:21:46.828 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_inodes": 211, 2024-03-11T19:21:46.828 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_base_inodes": 20, 2024-03-11T19:21:46.829 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dirfrag_rstats": 191, 2024-03-11T19:21:46.829 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_file_inodes": 386, 2024-03-11T19:21:46.829 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_inode_file_caps": 0, 2024-03-11T19:21:46.829 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_revoke": 42, 2024-03-11T19:21:46.829 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_grant": 139, 2024-03-11T19:21:46.829 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T19:21:46.829 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T19:21:46.829 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T19:21:46.829 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps": 179, 2024-03-11T19:21:46.829 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps_dirty": 143, 2024-03-11T19:21:46.829 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_cap_release": 3, 2024-03-11T19:21:46.829 INFO:teuthology.orchestra.run.smithi170.stdout: "process_request_cap_release": 81 2024-03-11T19:21:46.829 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.829 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cache": { 2024-03-11T19:21:46.829 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update": 0, 2024-03-11T19:21:46.829 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update_receipt": 0, 2024-03-11T19:21:46.830 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_try_discover": 0, 2024-03-11T19:21:46.830 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_send_discover": 0, 2024-03-11T19:21:46.830 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_handle_discover": 10, 2024-03-11T19:21:46.830 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays": 0, 2024-03-11T19:21:46.830 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_delayed": 0, 2024-03-11T19:21:46.830 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_enqueuing": 0, 2024-03-11T19:21:46.830 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_created": 22, 2024-03-11T19:21:46.830 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_enqueued": 22, 2024-03-11T19:21:46.830 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_reintegrated": 0, 2024-03-11T19:21:46.830 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_migrated": 0, 2024-03-11T19:21:46.830 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_processing": 0, 2024-03-11T19:21:46.830 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_enqueued": 0, 2024-03-11T19:21:46.830 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_prioritized": 0, 2024-03-11T19:21:46.830 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_started": 0, 2024-03-11T19:21:46.830 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_completed": 0, 2024-03-11T19:21:46.831 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_enqueue_scrub": 20, 2024-03-11T19:21:46.831 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_exportdir": 0, 2024-03-11T19:21:46.831 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_flush": 0, 2024-03-11T19:21:46.831 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragmentdir": 0, 2024-03-11T19:21:46.831 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragstats": 0, 2024-03-11T19:21:46.831 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_inodestats": 0 2024-03-11T19:21:46.831 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.831 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log": { 2024-03-11T19:21:46.831 INFO:teuthology.orchestra.run.smithi170.stdout: "evlrg": 0, 2024-03-11T19:21:46.831 INFO:teuthology.orchestra.run.smithi170.stdout: "evadd": 275, 2024-03-11T19:21:46.831 INFO:teuthology.orchestra.run.smithi170.stdout: "evex": 0, 2024-03-11T19:21:46.831 INFO:teuthology.orchestra.run.smithi170.stdout: "evtrm": 0, 2024-03-11T19:21:46.831 INFO:teuthology.orchestra.run.smithi170.stdout: "ev": 275, 2024-03-11T19:21:46.831 INFO:teuthology.orchestra.run.smithi170.stdout: "evexg": 0, 2024-03-11T19:21:46.831 INFO:teuthology.orchestra.run.smithi170.stdout: "evexd": 0, 2024-03-11T19:21:46.831 INFO:teuthology.orchestra.run.smithi170.stdout: "segadd": 2, 2024-03-11T19:21:46.832 INFO:teuthology.orchestra.run.smithi170.stdout: "segex": 0, 2024-03-11T19:21:46.832 INFO:teuthology.orchestra.run.smithi170.stdout: "segtrm": 0, 2024-03-11T19:21:46.832 INFO:teuthology.orchestra.run.smithi170.stdout: "seg": 2, 2024-03-11T19:21:46.832 INFO:teuthology.orchestra.run.smithi170.stdout: "segmjr": 2, 2024-03-11T19:21:46.832 INFO:teuthology.orchestra.run.smithi170.stdout: "segexg": 0, 2024-03-11T19:21:46.832 INFO:teuthology.orchestra.run.smithi170.stdout: "segexd": 0, 2024-03-11T19:21:46.832 INFO:teuthology.orchestra.run.smithi170.stdout: "expos": 4194304, 2024-03-11T19:21:46.832 INFO:teuthology.orchestra.run.smithi170.stdout: "wrpos": 4905530, 2024-03-11T19:21:46.832 INFO:teuthology.orchestra.run.smithi170.stdout: "rdpos": 0, 2024-03-11T19:21:46.832 INFO:teuthology.orchestra.run.smithi170.stdout: "jlat": { 2024-03-11T19:21:46.832 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 151, 2024-03-11T19:21:46.832 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.275618506, 2024-03-11T19:21:46.832 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.001825288 2024-03-11T19:21:46.832 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.832 INFO:teuthology.orchestra.run.smithi170.stdout: "replayed": 0 2024-03-11T19:21:46.833 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.833 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_mem": { 2024-03-11T19:21:46.833 INFO:teuthology.orchestra.run.smithi170.stdout: "ino": 23, 2024-03-11T19:21:46.833 INFO:teuthology.orchestra.run.smithi170.stdout: "ino+": 65, 2024-03-11T19:21:46.833 INFO:teuthology.orchestra.run.smithi170.stdout: "ino-": 42, 2024-03-11T19:21:46.833 INFO:teuthology.orchestra.run.smithi170.stdout: "dir": 18, 2024-03-11T19:21:46.833 INFO:teuthology.orchestra.run.smithi170.stdout: "dir+": 22, 2024-03-11T19:21:46.833 INFO:teuthology.orchestra.run.smithi170.stdout: "dir-": 4, 2024-03-11T19:21:46.833 INFO:teuthology.orchestra.run.smithi170.stdout: "dn": 23, 2024-03-11T19:21:46.833 INFO:teuthology.orchestra.run.smithi170.stdout: "dn+": 66, 2024-03-11T19:21:46.833 INFO:teuthology.orchestra.run.smithi170.stdout: "dn-": 43, 2024-03-11T19:21:46.833 INFO:teuthology.orchestra.run.smithi170.stdout: "cap": 2, 2024-03-11T19:21:46.833 INFO:teuthology.orchestra.run.smithi170.stdout: "cap+": 36, 2024-03-11T19:21:46.833 INFO:teuthology.orchestra.run.smithi170.stdout: "cap-": 34, 2024-03-11T19:21:46.833 INFO:teuthology.orchestra.run.smithi170.stdout: "rss": 58560, 2024-03-11T19:21:46.833 INFO:teuthology.orchestra.run.smithi170.stdout: "heap": 215420 2024-03-11T19:21:46.833 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.834 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_server": { 2024-03-11T19:21:46.834 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_client_request": 133, 2024-03-11T19:21:46.834 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_server_request": 1, 2024-03-11T19:21:46.834 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_request": 99, 2024-03-11T19:21:46.834 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_session": 20, 2024-03-11T19:21:46.834 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_peer_request": 2, 2024-03-11T19:21:46.834 INFO:teuthology.orchestra.run.smithi170.stdout: "req_create_latency": { 2024-03-11T19:21:46.834 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 22, 2024-03-11T19:21:46.834 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.017647207, 2024-03-11T19:21:46.834 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000802145 2024-03-11T19:21:46.834 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.834 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getattr_latency": { 2024-03-11T19:21:46.834 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 28, 2024-03-11T19:21:46.834 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.007490027, 2024-03-11T19:21:46.834 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000267500 2024-03-11T19:21:46.835 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.835 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getfilelock_latency": { 2024-03-11T19:21:46.835 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.835 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.835 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.835 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.835 INFO:teuthology.orchestra.run.smithi170.stdout: "req_link_latency": { 2024-03-11T19:21:46.835 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.835 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.835 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.835 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.835 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookup_latency": { 2024-03-11T19:21:46.835 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.835 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.835 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.836 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.836 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookuphash_latency": { 2024-03-11T19:21:46.836 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.836 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.836 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.836 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.836 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupino_latency": { 2024-03-11T19:21:46.836 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.836 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.836 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.836 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.836 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupname_latency": { 2024-03-11T19:21:46.836 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.836 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.836 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.837 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.837 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupparent_latency": { 2024-03-11T19:21:46.837 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.837 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.837 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.837 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.837 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupsnap_latency": { 2024-03-11T19:21:46.837 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.837 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.837 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.837 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.837 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lssnap_latency": { 2024-03-11T19:21:46.837 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.837 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.837 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.837 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.838 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mkdir_latency": { 2024-03-11T19:21:46.838 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 10, 2024-03-11T19:21:46.838 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.019977516, 2024-03-11T19:21:46.838 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.001997751 2024-03-11T19:21:46.838 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.838 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mknod_latency": { 2024-03-11T19:21:46.838 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.839 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.839 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.839 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.839 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mksnap_latency": { 2024-03-11T19:21:46.839 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.839 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.839 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.839 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.839 INFO:teuthology.orchestra.run.smithi170.stdout: "req_open_latency": { 2024-03-11T19:21:46.839 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 1, 2024-03-11T19:21:46.839 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000568451, 2024-03-11T19:21:46.839 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000568451 2024-03-11T19:21:46.839 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.839 INFO:teuthology.orchestra.run.smithi170.stdout: "req_readdir_latency": { 2024-03-11T19:21:46.839 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 5, 2024-03-11T19:21:46.840 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.012971959, 2024-03-11T19:21:46.840 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.002594391 2024-03-11T19:21:46.840 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.840 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rename_latency": { 2024-03-11T19:21:46.840 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2, 2024-03-11T19:21:46.840 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.001885580, 2024-03-11T19:21:46.840 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000942790 2024-03-11T19:21:46.840 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.840 INFO:teuthology.orchestra.run.smithi170.stdout: "req_renamesnap_latency": { 2024-03-11T19:21:46.840 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.840 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.840 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.840 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.840 INFO:teuthology.orchestra.run.smithi170.stdout: "req_snapdiff_latency": { 2024-03-11T19:21:46.840 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.840 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.841 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.841 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.841 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmdir_latency": { 2024-03-11T19:21:46.841 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 4, 2024-03-11T19:21:46.841 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.025504892, 2024-03-11T19:21:46.841 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.006376223 2024-03-11T19:21:46.841 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.841 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmsnap_latency": { 2024-03-11T19:21:46.841 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.841 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.841 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.841 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.841 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmxattr_latency": { 2024-03-11T19:21:46.841 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.841 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.841 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.842 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.842 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setattr_latency": { 2024-03-11T19:21:46.842 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 1, 2024-03-11T19:21:46.842 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.001848525, 2024-03-11T19:21:46.842 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.001848525 2024-03-11T19:21:46.842 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.842 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setdirlayout_latency": { 2024-03-11T19:21:46.842 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.842 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.842 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.842 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.842 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setfilelock_latency": { 2024-03-11T19:21:46.842 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.842 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.842 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.843 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.843 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setlayout_latency": { 2024-03-11T19:21:46.843 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.843 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.843 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.843 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.843 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setxattr_latency": { 2024-03-11T19:21:46.843 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 8, 2024-03-11T19:21:46.843 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.019688970, 2024-03-11T19:21:46.843 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.002461121 2024-03-11T19:21:46.843 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.843 INFO:teuthology.orchestra.run.smithi170.stdout: "req_symlink_latency": { 2024-03-11T19:21:46.843 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.843 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: "req_unlink_latency": { 2024-03-11T19:21:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 18, 2024-03-11T19:21:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.028363913, 2024-03-11T19:21:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.001575772 2024-03-11T19:21:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_revoke_eviction": 0, 2024-03-11T19:21:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_acquisition_throttle": 0, 2024-03-11T19:21:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getvxattr_latency": { 2024-03-11T19:21:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:46.844 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_sessions": { 2024-03-11T19:21:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: "session_count": 1, 2024-03-11T19:21:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: "session_add": 3, 2024-03-11T19:21:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: "session_remove": 2, 2024-03-11T19:21:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_open": 1, 2024-03-11T19:21:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_stale": 0, 2024-03-11T19:21:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: "total_load": 41, 2024-03-11T19:21:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: "average_load": 41, 2024-03-11T19:21:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: "avg_session_uptime": 712, 2024-03-11T19:21:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: "mdthresh_evicted": 0 2024-03-11T19:21:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: "mempool": { 2024-03-11T19:21:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_bytes": 0, 2024-03-11T19:21:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_items": 0, 2024-03-11T19:21:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T19:21:46.845 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_items": 0, 2024-03-11T19:21:46.846 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T19:21:46.846 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_items": 0, 2024-03-11T19:21:46.846 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T19:21:46.846 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T19:21:46.846 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T19:21:46.846 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T19:21:46.846 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T19:21:46.846 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_items": 0, 2024-03-11T19:21:46.846 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T19:21:46.846 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T19:21:46.846 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_bytes": 0, 2024-03-11T19:21:46.846 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_items": 0, 2024-03-11T19:21:46.846 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_bytes": 0, 2024-03-11T19:21:46.846 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_items": 0, 2024-03-11T19:21:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T19:21:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T19:21:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T19:21:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T19:21:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T19:21:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_items": 0, 2024-03-11T19:21:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_bytes": 0, 2024-03-11T19:21:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_items": 0, 2024-03-11T19:21:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T19:21:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T19:21:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_bytes": 0, 2024-03-11T19:21:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_items": 0, 2024-03-11T19:21:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_bytes": 0, 2024-03-11T19:21:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_items": 0, 2024-03-11T19:21:46.847 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T19:21:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_items": 0, 2024-03-11T19:21:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T19:21:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_items": 0, 2024-03-11T19:21:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_bytes": 316057, 2024-03-11T19:21:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_items": 125, 2024-03-11T19:21:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_bytes": 0, 2024-03-11T19:21:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_items": 0, 2024-03-11T19:21:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_bytes": 0, 2024-03-11T19:21:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_items": 0, 2024-03-11T19:21:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_bytes": 0, 2024-03-11T19:21:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_items": 0, 2024-03-11T19:21:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_bytes": 0, 2024-03-11T19:21:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_items": 0, 2024-03-11T19:21:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_bytes": 7336, 2024-03-11T19:21:46.848 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_items": 154, 2024-03-11T19:21:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T19:21:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_items": 0, 2024-03-11T19:21:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_bytes": 0, 2024-03-11T19:21:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_items": 0, 2024-03-11T19:21:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_bytes": 92778, 2024-03-11T19:21:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_items": 615, 2024-03-11T19:21:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_bytes": 0, 2024-03-11T19:21:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_items": 0, 2024-03-11T19:21:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_bytes": 0, 2024-03-11T19:21:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_items": 0 2024-03-11T19:21:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "objecter": { 2024-03-11T19:21:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "op_active": 0, 2024-03-11T19:21:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "op_laggy": 0, 2024-03-11T19:21:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send": 3962, 2024-03-11T19:21:46.849 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send_bytes": 734970, 2024-03-11T19:21:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: "op_resend": 19, 2024-03-11T19:21:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: "op_reply": 3962, 2024-03-11T19:21:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: "op_latency": { 2024-03-11T19:21:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 3962, 2024-03-11T19:21:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 48.655803729, 2024-03-11T19:21:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.012280616 2024-03-11T19:21:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: "op_inflight": 0, 2024-03-11T19:21:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: "oplen_avg": { 2024-03-11T19:21:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 3962, 2024-03-11T19:21:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 4012 2024-03-11T19:21:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: "op": 3962, 2024-03-11T19:21:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: "op_r": 640, 2024-03-11T19:21:46.850 INFO:teuthology.orchestra.run.smithi170.stdout: "op_w": 3322, 2024-03-11T19:21:46.851 INFO:teuthology.orchestra.run.smithi170.stdout: "op_rmw": 0, 2024-03-11T19:21:46.851 INFO:teuthology.orchestra.run.smithi170.stdout: "op_pg": 0, 2024-03-11T19:21:46.851 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_stat": 0, 2024-03-11T19:21:46.851 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_create": 11, 2024-03-11T19:21:46.851 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_read": 23, 2024-03-11T19:21:46.851 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_write": 154, 2024-03-11T19:21:46.851 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writefull": 28, 2024-03-11T19:21:46.851 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writesame": 0, 2024-03-11T19:21:46.851 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_append": 0, 2024-03-11T19:21:46.851 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_zero": 0, 2024-03-11T19:21:46.851 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_truncate": 0, 2024-03-11T19:21:46.851 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_delete": 3094, 2024-03-11T19:21:46.851 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_mapext": 0, 2024-03-11T19:21:46.851 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_sparse_read": 0, 2024-03-11T19:21:46.852 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_clonerange": 0, 2024-03-11T19:21:46.852 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_getxattr": 617, 2024-03-11T19:21:46.852 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_setxattr": 22, 2024-03-11T19:21:46.852 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_cmpxattr": 0, 2024-03-11T19:21:46.852 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_rmxattr": 0, 2024-03-11T19:21:46.852 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_resetxattrs": 0, 2024-03-11T19:21:46.852 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_call": 0, 2024-03-11T19:21:46.852 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_watch": 0, 2024-03-11T19:21:46.852 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_notify": 0, 2024-03-11T19:21:46.852 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T19:21:46.852 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls": 0, 2024-03-11T19:21:46.852 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls_filter": 0, 2024-03-11T19:21:46.852 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_other": 0, 2024-03-11T19:21:46.852 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_active": 0, 2024-03-11T19:21:46.852 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_send": 0, 2024-03-11T19:21:46.853 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_resend": 0, 2024-03-11T19:21:46.853 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_ping": 0, 2024-03-11T19:21:46.853 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_active": 0, 2024-03-11T19:21:46.853 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_send": 0, 2024-03-11T19:21:46.853 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_resend": 0, 2024-03-11T19:21:46.853 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_active": 0, 2024-03-11T19:21:46.853 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_send": 0, 2024-03-11T19:21:46.853 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_resend": 0, 2024-03-11T19:21:46.853 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_active": 0, 2024-03-11T19:21:46.853 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_send": 0, 2024-03-11T19:21:46.853 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_resend": 0, 2024-03-11T19:21:46.853 INFO:teuthology.orchestra.run.smithi170.stdout: "command_active": 0, 2024-03-11T19:21:46.853 INFO:teuthology.orchestra.run.smithi170.stdout: "command_send": 0, 2024-03-11T19:21:46.853 INFO:teuthology.orchestra.run.smithi170.stdout: "command_resend": 0, 2024-03-11T19:21:46.853 INFO:teuthology.orchestra.run.smithi170.stdout: "map_epoch": 89, 2024-03-11T19:21:46.853 INFO:teuthology.orchestra.run.smithi170.stdout: "map_full": 0, 2024-03-11T19:21:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: "map_inc": 3, 2024-03-11T19:21:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_sessions": 12, 2024-03-11T19:21:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_open": 12, 2024-03-11T19:21:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_close": 0, 2024-03-11T19:21:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_laggy": 0, 2024-03-11T19:21:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_wr": 50, 2024-03-11T19:21:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_rd": 0, 2024-03-11T19:21:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_del": 13 2024-03-11T19:21:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: "oft": { 2024-03-11T19:21:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_objs": 1, 2024-03-11T19:21:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_kv_pairs": 0, 2024-03-11T19:21:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_updates": 0, 2024-03-11T19:21:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_removes": 0 2024-03-11T19:21:46.854 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: "osd-slow-ops": { 2024-03-11T19:21:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_ops_count": 0 2024-03-11T19:21:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: "purge_queue": { 2024-03-11T19:21:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops": 0, 2024-03-11T19:21:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops_high_water": 518, 2024-03-11T19:21:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing": 0, 2024-03-11T19:21:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_high_water": 6, 2024-03-11T19:21:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed_ops": 2582, 2024-03-11T19:21:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed": 22, 2024-03-11T19:21:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_item_in_journal": 0 2024-03-11T19:21:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T19:21:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:46.855 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-03-11T19:21:46.856 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:46.856 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 7582, 2024-03-11T19:21:46.856 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 2422493, 2024-03-11T19:21:46.856 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:46.856 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 7582, 2024-03-11T19:21:46.856 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-11T19:21:46.856 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-11T19:21:46.856 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 7582, 2024-03-11T19:21:46.856 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 2422493, 2024-03-11T19:21:46.856 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:46.856 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.856 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.856 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.857 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:46.857 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.857 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_bytes": { 2024-03-11T19:21:46.857 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:46.857 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-03-11T19:21:46.857 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:46.857 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-11T19:21:46.857 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-11T19:21:46.857 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:46.857 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:46.857 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 3962, 2024-03-11T19:21:46.857 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 737192, 2024-03-11T19:21:46.858 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 3962, 2024-03-11T19:21:46.858 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 737192, 2024-03-11T19:21:46.858 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:46.858 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.858 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.858 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.858 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:46.858 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.858 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_ops": { 2024-03-11T19:21:46.858 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:46.858 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 1024, 2024-03-11T19:21:46.858 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:46.859 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-11T19:21:46.859 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-11T19:21:46.859 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:46.859 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:46.859 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 3962, 2024-03-11T19:21:46.859 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 3962, 2024-03-11T19:21:46.859 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 3962, 2024-03-11T19:21:46.859 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 3962, 2024-03-11T19:21:46.859 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:46.859 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.859 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.859 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.859 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:46.859 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.860 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle": { 2024-03-11T19:21:46.860 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:46.860 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-03-11T19:21:46.860 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:46.860 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 22, 2024-03-11T19:21:46.860 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 2222, 2024-03-11T19:21:46.860 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:46.860 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 22, 2024-03-11T19:21:46.860 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-11T19:21:46.860 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-11T19:21:46.860 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 3, 2024-03-11T19:21:46.860 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 2222, 2024-03-11T19:21:46.860 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:46.860 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.860 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.861 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.861 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:46.861 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:46.861 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle-0x564737d466e0": { 2024-03-11T19:21:46.861 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:46.861 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-03-11T19:21:46.861 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:46.861 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 275, 2024-03-11T19:21:46.861 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 711226, 2024-03-11T19:21:46.861 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:46.861 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 275, 2024-03-11T19:21:46.861 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-11T19:21:46.861 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-11T19:21:46.861 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 151, 2024-03-11T19:21:46.861 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 711226, 2024-03-11T19:21:46.862 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:46.862 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:46.862 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:46.862 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:46.862 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:46.862 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:46.862 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-03-11T19:21:47.671 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:47 smithi170 ceph-mon[35690]: pgmap v710: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:21:47.697 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-03-11T19:21:47.698 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-11T19:21:47.698 DEBUG:tasks.check_counter:Getting stats from e 2024-03-11T19:21:47.698 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-11T19:21:47.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:47 smithi190 ceph-mon[36292]: pgmap v710: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:21:47.809 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:47 smithi029 ceph-mon[32439]: pgmap v710: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:21:47.950 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:21:49.734 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:49 smithi170 ceph-mon[35690]: pgmap v711: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:21:49.759 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-03-11T19:21:49.759 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T19:21:49.759 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 1964, 2024-03-11T19:21:49.759 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 1956, 2024-03-11T19:21:49.759 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 1997904, 2024-03-11T19:21:49.759 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 674146, 2024-03-11T19:21:49.759 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 5, 2024-03-11T19:21:49.759 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 1, 2024-03-11T19:21:49.760 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.174430541, 2024-03-11T19:21:49.760 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.061514665, 2024-03-11T19:21:49.760 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.052678523, 2024-03-11T19:21:49.760 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.051950835, 2024-03-11T19:21:49.760 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:49.760 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 1956, 2024-03-11T19:21:49.760 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.063730438, 2024-03-11T19:21:49.760 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000032582 2024-03-11T19:21:49.760 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.760 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:49.760 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.760 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.760 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.760 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.761 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 776256, 2024-03-11T19:21:49.761 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 141088 2024-03-11T19:21:49.761 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.761 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T19:21:49.761 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 5, 2024-03-11T19:21:49.761 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 180, 2024-03-11T19:21:49.761 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 1816, 2024-03-11T19:21:49.761 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 384254, 2024-03-11T19:21:49.761 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 5, 2024-03-11T19:21:49.761 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 4, 2024-03-11T19:21:49.761 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.009135353, 2024-03-11T19:21:49.761 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.006352210, 2024-03-11T19:21:49.761 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.001353643, 2024-03-11T19:21:49.761 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.000077367, 2024-03-11T19:21:49.762 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:49.762 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 178, 2024-03-11T19:21:49.762 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.010735516, 2024-03-11T19:21:49.762 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000060311 2024-03-11T19:21:49.762 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.762 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:49.762 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.762 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.762 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.762 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.762 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-11T19:21:49.762 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 382944 2024-03-11T19:21:49.762 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.763 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T19:21:49.763 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 826, 2024-03-11T19:21:49.763 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 828, 2024-03-11T19:21:49.763 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 203302, 2024-03-11T19:21:49.763 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 256893, 2024-03-11T19:21:49.763 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 4, 2024-03-11T19:21:49.763 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 3, 2024-03-11T19:21:49.763 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.061756281, 2024-03-11T19:21:49.763 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.016715705, 2024-03-11T19:21:49.763 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.013858241, 2024-03-11T19:21:49.763 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.025537338, 2024-03-11T19:21:49.763 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:49.763 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 826, 2024-03-11T19:21:49.763 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.024701437, 2024-03-11T19:21:49.763 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000029904 2024-03-11T19:21:49.764 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.764 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:49.764 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.764 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.764 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.764 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.764 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T19:21:49.764 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T19:21:49.764 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.764 INFO:teuthology.orchestra.run.smithi170.stdout: "cct": { 2024-03-11T19:21:49.764 INFO:teuthology.orchestra.run.smithi170.stdout: "total_workers": 1, 2024-03-11T19:21:49.764 INFO:teuthology.orchestra.run.smithi170.stdout: "unhealthy_workers": 0 2024-03-11T19:21:49.764 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.764 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-MDSRank": { 2024-03-11T19:21:49.764 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-03-11T19:21:49.765 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-03-11T19:21:49.765 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 4202, 2024-03-11T19:21:49.765 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.103314143, 2024-03-11T19:21:49.765 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000024586 2024-03-11T19:21:49.765 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:49.765 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.765 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-PurgeQueue": { 2024-03-11T19:21:49.765 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-03-11T19:21:49.765 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-03-11T19:21:49.765 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.765 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.765 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.765 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:49.765 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.766 INFO:teuthology.orchestra.run.smithi170.stdout: "mds": { 2024-03-11T19:21:49.766 INFO:teuthology.orchestra.run.smithi170.stdout: "request": 0, 2024-03-11T19:21:49.766 INFO:teuthology.orchestra.run.smithi170.stdout: "reply": 0, 2024-03-11T19:21:49.766 INFO:teuthology.orchestra.run.smithi170.stdout: "reply_latency": { 2024-03-11T19:21:49.766 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.766 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.766 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.766 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.766 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_reply": 0, 2024-03-11T19:21:49.766 INFO:teuthology.orchestra.run.smithi170.stdout: "forward": 0, 2024-03-11T19:21:49.766 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_complete": 0, 2024-03-11T19:21:49.766 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_keys": 0, 2024-03-11T19:21:49.766 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_commit": 0, 2024-03-11T19:21:49.767 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_split": 0, 2024-03-11T19:21:49.767 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_merge": 0, 2024-03-11T19:21:49.767 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes": 36, 2024-03-11T19:21:49.767 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_top": 0, 2024-03-11T19:21:49.767 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_bottom": 36, 2024-03-11T19:21:49.767 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pin_tail": 0, 2024-03-11T19:21:49.767 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pinned": 36, 2024-03-11T19:21:49.767 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_expired": 0, 2024-03-11T19:21:49.767 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_with_caps": 0, 2024-03-11T19:21:49.767 INFO:teuthology.orchestra.run.smithi170.stdout: "caps": 0, 2024-03-11T19:21:49.767 INFO:teuthology.orchestra.run.smithi170.stdout: "subtrees": 2, 2024-03-11T19:21:49.767 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse": 0, 2024-03-11T19:21:49.767 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_hit": 0, 2024-03-11T19:21:49.767 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_forward": 0, 2024-03-11T19:21:49.767 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_discover": 0, 2024-03-11T19:21:49.767 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_dir_fetch": 0, 2024-03-11T19:21:49.768 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_remote_ino": 0, 2024-03-11T19:21:49.768 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_lock": 0, 2024-03-11T19:21:49.768 INFO:teuthology.orchestra.run.smithi170.stdout: "load_cent": 0, 2024-03-11T19:21:49.768 INFO:teuthology.orchestra.run.smithi170.stdout: "q": 0, 2024-03-11T19:21:49.768 INFO:teuthology.orchestra.run.smithi170.stdout: "exported": 0, 2024-03-11T19:21:49.768 INFO:teuthology.orchestra.run.smithi170.stdout: "exported_inodes": 0, 2024-03-11T19:21:49.768 INFO:teuthology.orchestra.run.smithi170.stdout: "imported": 0, 2024-03-11T19:21:49.768 INFO:teuthology.orchestra.run.smithi170.stdout: "imported_inodes": 0, 2024-03-11T19:21:49.768 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_dir_fetch": 0, 2024-03-11T19:21:49.768 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_backtrace_fetch": 0, 2024-03-11T19:21:49.768 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_peer_discover": 0, 2024-03-11T19:21:49.768 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rfiles": 4, 2024-03-11T19:21:49.768 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rbytes": 234, 2024-03-11T19:21:49.768 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rsnaps": 0, 2024-03-11T19:21:49.768 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T19:21:49.769 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_set_tag": 0, 2024-03-11T19:21:49.769 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T19:21:49.769 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_inotable_repaired": 0, 2024-03-11T19:21:49.769 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_inodes": 0, 2024-03-11T19:21:49.769 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T19:21:49.769 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T19:21:49.769 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_file_inodes": 0, 2024-03-11T19:21:49.769 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_inode_file_caps": 0, 2024-03-11T19:21:49.769 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T19:21:49.769 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_grant": 0, 2024-03-11T19:21:49.769 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T19:21:49.769 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T19:21:49.769 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T19:21:49.769 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps": 0, 2024-03-11T19:21:49.769 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps_dirty": 0, 2024-03-11T19:21:49.770 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_cap_release": 0, 2024-03-11T19:21:49.770 INFO:teuthology.orchestra.run.smithi170.stdout: "process_request_cap_release": 0 2024-03-11T19:21:49.770 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.770 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cache": { 2024-03-11T19:21:49.770 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update": 0, 2024-03-11T19:21:49.770 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update_receipt": 0, 2024-03-11T19:21:49.770 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_try_discover": 0, 2024-03-11T19:21:49.770 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_send_discover": 0, 2024-03-11T19:21:49.770 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_handle_discover": 0, 2024-03-11T19:21:49.770 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays": 0, 2024-03-11T19:21:49.770 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_delayed": 0, 2024-03-11T19:21:49.770 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_enqueuing": 0, 2024-03-11T19:21:49.770 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_created": 0, 2024-03-11T19:21:49.770 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_enqueued": 0, 2024-03-11T19:21:49.771 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_reintegrated": 0, 2024-03-11T19:21:49.771 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_migrated": 0, 2024-03-11T19:21:49.771 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_processing": 0, 2024-03-11T19:21:49.771 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_enqueued": 0, 2024-03-11T19:21:49.771 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_prioritized": 0, 2024-03-11T19:21:49.771 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_started": 0, 2024-03-11T19:21:49.771 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_completed": 0, 2024-03-11T19:21:49.771 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T19:21:49.771 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_exportdir": 0, 2024-03-11T19:21:49.771 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_flush": 0, 2024-03-11T19:21:49.771 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragmentdir": 0, 2024-03-11T19:21:49.771 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragstats": 0, 2024-03-11T19:21:49.771 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_inodestats": 0 2024-03-11T19:21:49.771 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.772 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log": { 2024-03-11T19:21:49.772 INFO:teuthology.orchestra.run.smithi170.stdout: "evlrg": 0, 2024-03-11T19:21:49.772 INFO:teuthology.orchestra.run.smithi170.stdout: "evadd": 0, 2024-03-11T19:21:49.772 INFO:teuthology.orchestra.run.smithi170.stdout: "evex": 0, 2024-03-11T19:21:49.772 INFO:teuthology.orchestra.run.smithi170.stdout: "evtrm": 0, 2024-03-11T19:21:49.772 INFO:teuthology.orchestra.run.smithi170.stdout: "ev": 275, 2024-03-11T19:21:49.772 INFO:teuthology.orchestra.run.smithi170.stdout: "evexg": 0, 2024-03-11T19:21:49.772 INFO:teuthology.orchestra.run.smithi170.stdout: "evexd": 0, 2024-03-11T19:21:49.772 INFO:teuthology.orchestra.run.smithi170.stdout: "segadd": 0, 2024-03-11T19:21:49.772 INFO:teuthology.orchestra.run.smithi170.stdout: "segex": 0, 2024-03-11T19:21:49.772 INFO:teuthology.orchestra.run.smithi170.stdout: "segtrm": 0, 2024-03-11T19:21:49.772 INFO:teuthology.orchestra.run.smithi170.stdout: "seg": 2, 2024-03-11T19:21:49.772 INFO:teuthology.orchestra.run.smithi170.stdout: "segmjr": 2, 2024-03-11T19:21:49.772 INFO:teuthology.orchestra.run.smithi170.stdout: "segexg": 0, 2024-03-11T19:21:49.772 INFO:teuthology.orchestra.run.smithi170.stdout: "segexd": 0, 2024-03-11T19:21:49.773 INFO:teuthology.orchestra.run.smithi170.stdout: "expos": 4194304, 2024-03-11T19:21:49.773 INFO:teuthology.orchestra.run.smithi170.stdout: "wrpos": 4905530, 2024-03-11T19:21:49.773 INFO:teuthology.orchestra.run.smithi170.stdout: "rdpos": 4900893, 2024-03-11T19:21:49.773 INFO:teuthology.orchestra.run.smithi170.stdout: "jlat": { 2024-03-11T19:21:49.773 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.773 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.773 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.773 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.773 INFO:teuthology.orchestra.run.smithi170.stdout: "replayed": 275 2024-03-11T19:21:49.773 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.773 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_mem": { 2024-03-11T19:21:49.773 INFO:teuthology.orchestra.run.smithi170.stdout: "ino": 14, 2024-03-11T19:21:49.773 INFO:teuthology.orchestra.run.smithi170.stdout: "ino+": 36, 2024-03-11T19:21:49.773 INFO:teuthology.orchestra.run.smithi170.stdout: "ino-": 22, 2024-03-11T19:21:49.774 INFO:teuthology.orchestra.run.smithi170.stdout: "dir": 9, 2024-03-11T19:21:49.774 INFO:teuthology.orchestra.run.smithi170.stdout: "dir+": 13, 2024-03-11T19:21:49.774 INFO:teuthology.orchestra.run.smithi170.stdout: "dir-": 4, 2024-03-11T19:21:49.774 INFO:teuthology.orchestra.run.smithi170.stdout: "dn": 36, 2024-03-11T19:21:49.774 INFO:teuthology.orchestra.run.smithi170.stdout: "dn+": 57, 2024-03-11T19:21:49.774 INFO:teuthology.orchestra.run.smithi170.stdout: "dn-": 21, 2024-03-11T19:21:49.774 INFO:teuthology.orchestra.run.smithi170.stdout: "cap": 0, 2024-03-11T19:21:49.774 INFO:teuthology.orchestra.run.smithi170.stdout: "cap+": 0, 2024-03-11T19:21:49.774 INFO:teuthology.orchestra.run.smithi170.stdout: "cap-": 0, 2024-03-11T19:21:49.774 INFO:teuthology.orchestra.run.smithi170.stdout: "rss": 50672, 2024-03-11T19:21:49.774 INFO:teuthology.orchestra.run.smithi170.stdout: "heap": 215420 2024-03-11T19:21:49.774 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.774 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_server": { 2024-03-11T19:21:49.774 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_client_request": 0, 2024-03-11T19:21:49.775 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_server_request": 0, 2024-03-11T19:21:49.775 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_request": 0, 2024-03-11T19:21:49.775 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_session": 0, 2024-03-11T19:21:49.775 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_peer_request": 0, 2024-03-11T19:21:49.775 INFO:teuthology.orchestra.run.smithi170.stdout: "req_create_latency": { 2024-03-11T19:21:49.775 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.775 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.775 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.775 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.775 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getattr_latency": { 2024-03-11T19:21:49.775 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.775 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.775 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.775 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.775 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getfilelock_latency": { 2024-03-11T19:21:49.776 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.776 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.776 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.776 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.776 INFO:teuthology.orchestra.run.smithi170.stdout: "req_link_latency": { 2024-03-11T19:21:49.776 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.776 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.776 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.776 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.776 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookup_latency": { 2024-03-11T19:21:49.776 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.776 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.776 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.776 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.777 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookuphash_latency": { 2024-03-11T19:21:49.777 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.777 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.777 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.777 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.777 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupino_latency": { 2024-03-11T19:21:49.777 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.777 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.777 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.777 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.777 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupname_latency": { 2024-03-11T19:21:49.777 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.777 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.777 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.777 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.778 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupparent_latency": { 2024-03-11T19:21:49.778 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.778 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.778 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.778 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.778 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupsnap_latency": { 2024-03-11T19:21:49.778 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.778 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.778 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.778 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.778 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lssnap_latency": { 2024-03-11T19:21:49.778 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.778 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.778 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.779 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.779 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mkdir_latency": { 2024-03-11T19:21:49.779 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.779 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.779 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.779 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:49 smithi190 ceph-mon[36292]: pgmap v711: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:21:49.780 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.780 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mknod_latency": { 2024-03-11T19:21:49.780 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.780 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.780 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.780 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.780 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mksnap_latency": { 2024-03-11T19:21:49.780 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.780 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.780 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.780 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.780 INFO:teuthology.orchestra.run.smithi170.stdout: "req_open_latency": { 2024-03-11T19:21:49.780 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.781 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.781 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.781 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.781 INFO:teuthology.orchestra.run.smithi170.stdout: "req_readdir_latency": { 2024-03-11T19:21:49.781 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.781 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.781 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.781 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.781 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rename_latency": { 2024-03-11T19:21:49.781 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.781 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.781 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.781 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.782 INFO:teuthology.orchestra.run.smithi170.stdout: "req_renamesnap_latency": { 2024-03-11T19:21:49.782 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.782 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.782 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.782 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.782 INFO:teuthology.orchestra.run.smithi170.stdout: "req_snapdiff_latency": { 2024-03-11T19:21:49.782 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.782 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.782 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.782 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.782 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmdir_latency": { 2024-03-11T19:21:49.782 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.782 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.783 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.783 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.783 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmsnap_latency": { 2024-03-11T19:21:49.783 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.783 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.783 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.783 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.783 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmxattr_latency": { 2024-03-11T19:21:49.783 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.783 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.783 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.783 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.783 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setattr_latency": { 2024-03-11T19:21:49.783 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.784 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.784 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.784 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.784 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setdirlayout_latency": { 2024-03-11T19:21:49.784 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.784 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.784 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.784 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.784 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setfilelock_latency": { 2024-03-11T19:21:49.784 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.784 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.784 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.784 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.784 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setlayout_latency": { 2024-03-11T19:21:49.784 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.785 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.785 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.785 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.785 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setxattr_latency": { 2024-03-11T19:21:49.785 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.785 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.785 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.785 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.785 INFO:teuthology.orchestra.run.smithi170.stdout: "req_symlink_latency": { 2024-03-11T19:21:49.785 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.785 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.785 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.785 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.785 INFO:teuthology.orchestra.run.smithi170.stdout: "req_unlink_latency": { 2024-03-11T19:21:49.786 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.786 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.786 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.786 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.786 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_revoke_eviction": 0, 2024-03-11T19:21:49.786 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_acquisition_throttle": 0, 2024-03-11T19:21:49.786 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getvxattr_latency": { 2024-03-11T19:21:49.786 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.786 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.786 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.786 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:49.786 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.786 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_sessions": { 2024-03-11T19:21:49.786 INFO:teuthology.orchestra.run.smithi170.stdout: "session_count": 1, 2024-03-11T19:21:49.787 INFO:teuthology.orchestra.run.smithi170.stdout: "session_add": 3, 2024-03-11T19:21:49.787 INFO:teuthology.orchestra.run.smithi170.stdout: "session_remove": 2, 2024-03-11T19:21:49.787 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_open": 2, 2024-03-11T19:21:49.787 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_stale": 0, 2024-03-11T19:21:49.787 INFO:teuthology.orchestra.run.smithi170.stdout: "total_load": 0, 2024-03-11T19:21:49.787 INFO:teuthology.orchestra.run.smithi170.stdout: "average_load": 0, 2024-03-11T19:21:49.787 INFO:teuthology.orchestra.run.smithi170.stdout: "avg_session_uptime": 3005, 2024-03-11T19:21:49.787 INFO:teuthology.orchestra.run.smithi170.stdout: "mdthresh_evicted": 0 2024-03-11T19:21:49.787 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.787 INFO:teuthology.orchestra.run.smithi170.stdout: "mempool": { 2024-03-11T19:21:49.787 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_bytes": 0, 2024-03-11T19:21:49.787 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_items": 0, 2024-03-11T19:21:49.787 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T19:21:49.788 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_items": 0, 2024-03-11T19:21:49.788 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T19:21:49.788 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_items": 0, 2024-03-11T19:21:49.788 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T19:21:49.788 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T19:21:49.788 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T19:21:49.788 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T19:21:49.788 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T19:21:49.788 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_items": 0, 2024-03-11T19:21:49.788 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T19:21:49.788 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T19:21:49.788 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_bytes": 0, 2024-03-11T19:21:49.788 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_items": 0, 2024-03-11T19:21:49.788 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_bytes": 0, 2024-03-11T19:21:49.788 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_items": 0, 2024-03-11T19:21:49.789 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T19:21:49.789 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T19:21:49.789 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T19:21:49.789 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T19:21:49.789 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T19:21:49.789 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_items": 0, 2024-03-11T19:21:49.789 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_bytes": 0, 2024-03-11T19:21:49.789 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_items": 0, 2024-03-11T19:21:49.789 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T19:21:49.789 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T19:21:49.789 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_bytes": 0, 2024-03-11T19:21:49.789 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_items": 0, 2024-03-11T19:21:49.789 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_bytes": 0, 2024-03-11T19:21:49.789 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_items": 0, 2024-03-11T19:21:49.790 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T19:21:49.790 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_items": 0, 2024-03-11T19:21:49.790 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T19:21:49.790 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_items": 0, 2024-03-11T19:21:49.790 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_bytes": 189423, 2024-03-11T19:21:49.790 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_items": 51, 2024-03-11T19:21:49.790 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_bytes": 0, 2024-03-11T19:21:49.790 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_items": 0, 2024-03-11T19:21:49.790 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_bytes": 0, 2024-03-11T19:21:49.790 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_items": 0, 2024-03-11T19:21:49.790 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_bytes": 0, 2024-03-11T19:21:49.790 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_items": 0, 2024-03-11T19:21:49.790 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_bytes": 0, 2024-03-11T19:21:49.790 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_items": 0, 2024-03-11T19:21:49.791 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_bytes": 7336, 2024-03-11T19:21:49.791 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_items": 154, 2024-03-11T19:21:49.791 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T19:21:49.791 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_items": 0, 2024-03-11T19:21:49.791 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_bytes": 0, 2024-03-11T19:21:49.791 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_items": 0, 2024-03-11T19:21:49.791 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_bytes": 67298, 2024-03-11T19:21:49.791 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_items": 333, 2024-03-11T19:21:49.791 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_bytes": 0, 2024-03-11T19:21:49.791 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_items": 0, 2024-03-11T19:21:49.791 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_bytes": 0, 2024-03-11T19:21:49.791 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_items": 0 2024-03-11T19:21:49.791 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.791 INFO:teuthology.orchestra.run.smithi170.stdout: "objecter": { 2024-03-11T19:21:49.791 INFO:teuthology.orchestra.run.smithi170.stdout: "op_active": 0, 2024-03-11T19:21:49.792 INFO:teuthology.orchestra.run.smithi170.stdout: "op_laggy": 0, 2024-03-11T19:21:49.792 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send": 2544, 2024-03-11T19:21:49.792 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send_bytes": 26, 2024-03-11T19:21:49.792 INFO:teuthology.orchestra.run.smithi170.stdout: "op_resend": 4, 2024-03-11T19:21:49.792 INFO:teuthology.orchestra.run.smithi170.stdout: "op_reply": 2544, 2024-03-11T19:21:49.792 INFO:teuthology.orchestra.run.smithi170.stdout: "op_latency": { 2024-03-11T19:21:49.792 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2544, 2024-03-11T19:21:49.792 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 1.200969866, 2024-03-11T19:21:49.792 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000472079 2024-03-11T19:21:49.792 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.792 INFO:teuthology.orchestra.run.smithi170.stdout: "op_inflight": 0, 2024-03-11T19:21:49.792 INFO:teuthology.orchestra.run.smithi170.stdout: "oplen_avg": { 2024-03-11T19:21:49.792 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2544, 2024-03-11T19:21:49.792 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 2545 2024-03-11T19:21:49.793 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.793 INFO:teuthology.orchestra.run.smithi170.stdout: "op": 2544, 2024-03-11T19:21:49.793 INFO:teuthology.orchestra.run.smithi170.stdout: "op_r": 2544, 2024-03-11T19:21:49.793 INFO:teuthology.orchestra.run.smithi170.stdout: "op_w": 0, 2024-03-11T19:21:49.793 INFO:teuthology.orchestra.run.smithi170.stdout: "op_rmw": 0, 2024-03-11T19:21:49.793 INFO:teuthology.orchestra.run.smithi170.stdout: "op_pg": 0, 2024-03-11T19:21:49.793 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_stat": 1655, 2024-03-11T19:21:49.793 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_create": 0, 2024-03-11T19:21:49.793 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_read": 886, 2024-03-11T19:21:49.793 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_write": 0, 2024-03-11T19:21:49.793 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writefull": 0, 2024-03-11T19:21:49.793 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writesame": 0, 2024-03-11T19:21:49.793 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_append": 0, 2024-03-11T19:21:49.793 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_zero": 0, 2024-03-11T19:21:49.793 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_truncate": 0, 2024-03-11T19:21:49.794 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_delete": 0, 2024-03-11T19:21:49.794 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_mapext": 0, 2024-03-11T19:21:49.794 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_sparse_read": 0, 2024-03-11T19:21:49.794 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_clonerange": 0, 2024-03-11T19:21:49.794 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_getxattr": 2, 2024-03-11T19:21:49.794 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_setxattr": 0, 2024-03-11T19:21:49.794 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_cmpxattr": 0, 2024-03-11T19:21:49.794 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_rmxattr": 0, 2024-03-11T19:21:49.794 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_resetxattrs": 0, 2024-03-11T19:21:49.794 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_call": 0, 2024-03-11T19:21:49.794 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_watch": 0, 2024-03-11T19:21:49.794 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_notify": 0, 2024-03-11T19:21:49.794 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T19:21:49.794 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls": 0, 2024-03-11T19:21:49.795 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls_filter": 0, 2024-03-11T19:21:49.795 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_other": 0, 2024-03-11T19:21:49.795 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_active": 0, 2024-03-11T19:21:49.795 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_send": 0, 2024-03-11T19:21:49.795 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_resend": 0, 2024-03-11T19:21:49.795 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_ping": 0, 2024-03-11T19:21:49.795 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_active": 0, 2024-03-11T19:21:49.795 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_send": 0, 2024-03-11T19:21:49.795 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_resend": 0, 2024-03-11T19:21:49.795 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_active": 0, 2024-03-11T19:21:49.795 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_send": 0, 2024-03-11T19:21:49.795 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_resend": 0, 2024-03-11T19:21:49.795 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_active": 0, 2024-03-11T19:21:49.795 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_send": 0, 2024-03-11T19:21:49.795 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_resend": 0, 2024-03-11T19:21:49.796 INFO:teuthology.orchestra.run.smithi170.stdout: "command_active": 0, 2024-03-11T19:21:49.796 INFO:teuthology.orchestra.run.smithi170.stdout: "command_send": 0, 2024-03-11T19:21:49.796 INFO:teuthology.orchestra.run.smithi170.stdout: "command_resend": 0, 2024-03-11T19:21:49.796 INFO:teuthology.orchestra.run.smithi170.stdout: "map_epoch": 89, 2024-03-11T19:21:49.796 INFO:teuthology.orchestra.run.smithi170.stdout: "map_full": 0, 2024-03-11T19:21:49.796 INFO:teuthology.orchestra.run.smithi170.stdout: "map_inc": 2, 2024-03-11T19:21:49.796 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_sessions": 5, 2024-03-11T19:21:49.796 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_open": 5, 2024-03-11T19:21:49.796 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_close": 0, 2024-03-11T19:21:49.796 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_laggy": 0, 2024-03-11T19:21:49.796 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_wr": 0, 2024-03-11T19:21:49.796 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_rd": 2, 2024-03-11T19:21:49.796 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_del": 0 2024-03-11T19:21:49.796 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.796 INFO:teuthology.orchestra.run.smithi170.stdout: "oft": { 2024-03-11T19:21:49.796 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_objs": 0, 2024-03-11T19:21:49.797 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_kv_pairs": 0, 2024-03-11T19:21:49.797 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_updates": 0, 2024-03-11T19:21:49.797 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_removes": 0 2024-03-11T19:21:49.797 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.797 INFO:teuthology.orchestra.run.smithi170.stdout: "osd-slow-ops": { 2024-03-11T19:21:49.797 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_ops_count": 0 2024-03-11T19:21:49.797 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.797 INFO:teuthology.orchestra.run.smithi170.stdout: "purge_queue": { 2024-03-11T19:21:49.797 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops": 0, 2024-03-11T19:21:49.797 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T19:21:49.797 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing": 0, 2024-03-11T19:21:49.797 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_high_water": 0, 2024-03-11T19:21:49.797 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed_ops": 0, 2024-03-11T19:21:49.797 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed": 0, 2024-03-11T19:21:49.797 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_item_in_journal": 0 2024-03-11T19:21:49.798 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.798 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T19:21:49.798 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:49.798 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-03-11T19:21:49.798 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:49.798 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 2791, 2024-03-11T19:21:49.798 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 1936203, 2024-03-11T19:21:49.798 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:49.798 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 2791, 2024-03-11T19:21:49.798 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-11T19:21:49.798 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-11T19:21:49.798 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 2791, 2024-03-11T19:21:49.798 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 1936203, 2024-03-11T19:21:49.798 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:49.799 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.799 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.799 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.799 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:49.799 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.799 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_bytes": { 2024-03-11T19:21:49.799 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:49.799 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-03-11T19:21:49.799 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:49.799 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-11T19:21:49.799 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-11T19:21:49.799 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:49.799 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:49.799 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 2544, 2024-03-11T19:21:49.799 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 711236, 2024-03-11T19:21:49.800 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 2544, 2024-03-11T19:21:49.800 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 711236, 2024-03-11T19:21:49.800 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:49.800 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.800 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.800 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.800 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:49.800 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.800 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_ops": { 2024-03-11T19:21:49.800 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:49.800 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 1024, 2024-03-11T19:21:49.800 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:49.800 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-11T19:21:49.800 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-11T19:21:49.800 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:49.801 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:49.801 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 2544, 2024-03-11T19:21:49.801 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 2544, 2024-03-11T19:21:49.801 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 2544, 2024-03-11T19:21:49.801 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 2544, 2024-03-11T19:21:49.801 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:49.801 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.801 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.801 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.801 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:49.801 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.801 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle": { 2024-03-11T19:21:49.801 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:49.801 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-03-11T19:21:49.802 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:49.802 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-11T19:21:49.802 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-11T19:21:49.802 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:49.802 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:49.802 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-11T19:21:49.802 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-11T19:21:49.802 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 0, 2024-03-11T19:21:49.802 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 0, 2024-03-11T19:21:49.802 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:49.802 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.802 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.802 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.802 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:49.802 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:49.803 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle-0x55afb416a960": { 2024-03-11T19:21:49.803 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:49.803 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-03-11T19:21:49.803 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:49.803 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-11T19:21:49.803 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-11T19:21:49.803 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:49.803 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:49.803 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-11T19:21:49.803 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-11T19:21:49.803 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 0, 2024-03-11T19:21:49.803 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 0, 2024-03-11T19:21:49.803 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:49.803 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:49.803 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:49.804 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:49.804 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:49.804 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:49.804 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-03-11T19:21:49.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:49 smithi029 ceph-mon[32439]: pgmap v711: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:21:50.618 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-11T19:21:50.619 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-11T19:21:50.619 DEBUG:tasks.check_counter:Getting stats from h 2024-03-11T19:21:50.619 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-11T19:21:50.862 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:21:51.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:51 smithi170 ceph-mon[35690]: pgmap v712: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-11T19:21:51.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:51 smithi190 ceph-mon[36292]: pgmap v712: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-11T19:21:51.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:51 smithi029 ceph-mon[32439]: pgmap v712: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-11T19:21:52.644 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-03-11T19:21:52.644 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T19:21:52.644 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 349, 2024-03-11T19:21:52.644 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 528, 2024-03-11T19:21:52.644 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 85289, 2024-03-11T19:21:52.644 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 209932, 2024-03-11T19:21:52.644 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 8, 2024-03-11T19:21:52.644 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 2, 2024-03-11T19:21:52.644 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.052120314, 2024-03-11T19:21:52.644 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.022082305, 2024-03-11T19:21:52.645 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.020910765, 2024-03-11T19:21:52.645 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.004170635, 2024-03-11T19:21:52.645 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:52.645 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 529, 2024-03-11T19:21:52.645 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.036894856, 2024-03-11T19:21:52.645 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000069744 2024-03-11T19:21:52.645 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.645 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:52.645 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 828, 2024-03-11T19:21:52.645 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000190222, 2024-03-11T19:21:52.645 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000229 2024-03-11T19:21:52.645 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.645 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-11T19:21:52.645 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 448 2024-03-11T19:21:52.645 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.646 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T19:21:52.646 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 284, 2024-03-11T19:21:52.646 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 275, 2024-03-11T19:21:52.646 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 789903, 2024-03-11T19:21:52.646 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 149905, 2024-03-11T19:21:52.646 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 6, 2024-03-11T19:21:52.646 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 5, 2024-03-11T19:21:52.646 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.038290896, 2024-03-11T19:21:52.646 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.012292345, 2024-03-11T19:21:52.646 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.023725660, 2024-03-11T19:21:52.646 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.005370400, 2024-03-11T19:21:52.646 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:52.646 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 274, 2024-03-11T19:21:52.646 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.023005353, 2024-03-11T19:21:52.647 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000083961 2024-03-11T19:21:52.647 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.647 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:52.647 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.647 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.647 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.647 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.647 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 786320, 2024-03-11T19:21:52.647 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 143696 2024-03-11T19:21:52.647 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.647 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T19:21:52.648 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 64, 2024-03-11T19:21:52.648 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 240, 2024-03-11T19:21:52.648 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 10656, 2024-03-11T19:21:52.648 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 819809, 2024-03-11T19:21:52.648 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 51, 2024-03-11T19:21:52.648 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 7, 2024-03-11T19:21:52.648 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.028190064, 2024-03-11T19:21:52.648 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.008169874, 2024-03-11T19:21:52.648 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.017563464, 2024-03-11T19:21:52.648 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.000102906, 2024-03-11T19:21:52.648 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:52.648 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 236, 2024-03-11T19:21:52.648 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.032688159, 2024-03-11T19:21:52.648 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000138509 2024-03-11T19:21:52.648 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.648 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:52.649 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 70, 2024-03-11T19:21:52.649 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000010757, 2024-03-11T19:21:52.649 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000153 2024-03-11T19:21:52.649 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.649 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-11T19:21:52.649 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 384368 2024-03-11T19:21:52.649 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.649 INFO:teuthology.orchestra.run.smithi170.stdout: "cct": { 2024-03-11T19:21:52.649 INFO:teuthology.orchestra.run.smithi170.stdout: "total_workers": 1, 2024-03-11T19:21:52.649 INFO:teuthology.orchestra.run.smithi170.stdout: "unhealthy_workers": 0 2024-03-11T19:21:52.649 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.649 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-MDSRank": { 2024-03-11T19:21:52.649 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-03-11T19:21:52.649 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-03-11T19:21:52.649 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 43, 2024-03-11T19:21:52.650 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.001443036, 2024-03-11T19:21:52.650 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000033558 2024-03-11T19:21:52.650 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:52.650 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.650 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-PurgeQueue": { 2024-03-11T19:21:52.650 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-03-11T19:21:52.650 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-03-11T19:21:52.650 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2, 2024-03-11T19:21:52.650 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000006520, 2024-03-11T19:21:52.650 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000003260 2024-03-11T19:21:52.650 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:52.650 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.650 INFO:teuthology.orchestra.run.smithi170.stdout: "mds": { 2024-03-11T19:21:52.650 INFO:teuthology.orchestra.run.smithi170.stdout: "request": 2, 2024-03-11T19:21:52.651 INFO:teuthology.orchestra.run.smithi170.stdout: "reply": 1, 2024-03-11T19:21:52.651 INFO:teuthology.orchestra.run.smithi170.stdout: "reply_latency": { 2024-03-11T19:21:52.651 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 1, 2024-03-11T19:21:52.651 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000882304, 2024-03-11T19:21:52.651 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000882304 2024-03-11T19:21:52.651 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.651 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_reply": 0, 2024-03-11T19:21:52.651 INFO:teuthology.orchestra.run.smithi170.stdout: "forward": 1, 2024-03-11T19:21:52.651 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_complete": 0, 2024-03-11T19:21:52.651 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_keys": 0, 2024-03-11T19:21:52.651 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_commit": 11, 2024-03-11T19:21:52.651 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_split": 0, 2024-03-11T19:21:52.651 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_merge": 0, 2024-03-11T19:21:52.651 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes": 14, 2024-03-11T19:21:52.651 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_top": 0, 2024-03-11T19:21:52.652 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_bottom": 14, 2024-03-11T19:21:52.652 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pin_tail": 0, 2024-03-11T19:21:52.652 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pinned": 14, 2024-03-11T19:21:52.652 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_expired": 0, 2024-03-11T19:21:52.652 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_with_caps": 1, 2024-03-11T19:21:52.652 INFO:teuthology.orchestra.run.smithi170.stdout: "caps": 1, 2024-03-11T19:21:52.652 INFO:teuthology.orchestra.run.smithi170.stdout: "subtrees": 2, 2024-03-11T19:21:52.652 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse": 3, 2024-03-11T19:21:52.652 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_hit": 1, 2024-03-11T19:21:52.652 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_forward": 0, 2024-03-11T19:21:52.652 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_discover": 1, 2024-03-11T19:21:52.652 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_dir_fetch": 0, 2024-03-11T19:21:52.652 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_remote_ino": 0, 2024-03-11T19:21:52.652 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_lock": 0, 2024-03-11T19:21:52.652 INFO:teuthology.orchestra.run.smithi170.stdout: "load_cent": 0, 2024-03-11T19:21:52.652 INFO:teuthology.orchestra.run.smithi170.stdout: "q": 0, 2024-03-11T19:21:52.653 INFO:teuthology.orchestra.run.smithi170.stdout: "exported": 0, 2024-03-11T19:21:52.653 INFO:teuthology.orchestra.run.smithi170.stdout: "exported_inodes": 0, 2024-03-11T19:21:52.653 INFO:teuthology.orchestra.run.smithi170.stdout: "imported": 0, 2024-03-11T19:21:52.653 INFO:teuthology.orchestra.run.smithi170.stdout: "imported_inodes": 0, 2024-03-11T19:21:52.653 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_dir_fetch": 0, 2024-03-11T19:21:52.653 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_backtrace_fetch": 0, 2024-03-11T19:21:52.653 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_peer_discover": 0, 2024-03-11T19:21:52.653 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rfiles": 22, 2024-03-11T19:21:52.653 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rbytes": 7436774654, 2024-03-11T19:21:52.653 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rsnaps": 0, 2024-03-11T19:21:52.653 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T19:21:52.653 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_set_tag": 0, 2024-03-11T19:21:52.653 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T19:21:52.653 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_inotable_repaired": 0, 2024-03-11T19:21:52.653 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_inodes": 0, 2024-03-11T19:21:52.653 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T19:21:52.654 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T19:21:52.654 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_file_inodes": 0, 2024-03-11T19:21:52.654 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_inode_file_caps": 0, 2024-03-11T19:21:52.654 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_revoke": 1, 2024-03-11T19:21:52.654 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_grant": 0, 2024-03-11T19:21:52.654 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T19:21:52.654 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T19:21:52.654 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T19:21:52.654 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps": 1, 2024-03-11T19:21:52.654 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps_dirty": 0, 2024-03-11T19:21:52.654 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_cap_release": 0, 2024-03-11T19:21:52.654 INFO:teuthology.orchestra.run.smithi170.stdout: "process_request_cap_release": 1 2024-03-11T19:21:52.654 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.654 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cache": { 2024-03-11T19:21:52.655 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update": 0, 2024-03-11T19:21:52.655 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update_receipt": 0, 2024-03-11T19:21:52.655 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_try_discover": 0, 2024-03-11T19:21:52.655 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_send_discover": 2, 2024-03-11T19:21:52.655 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_handle_discover": 0, 2024-03-11T19:21:52.655 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays": 0, 2024-03-11T19:21:52.655 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_delayed": 0, 2024-03-11T19:21:52.655 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_enqueuing": 0, 2024-03-11T19:21:52.655 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_created": 0, 2024-03-11T19:21:52.655 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_enqueued": 0, 2024-03-11T19:21:52.655 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_reintegrated": 0, 2024-03-11T19:21:52.655 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_migrated": 0, 2024-03-11T19:21:52.655 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_processing": 0, 2024-03-11T19:21:52.655 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_enqueued": 0, 2024-03-11T19:21:52.655 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_prioritized": 0, 2024-03-11T19:21:52.655 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_started": 0, 2024-03-11T19:21:52.655 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_completed": 0, 2024-03-11T19:21:52.656 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T19:21:52.656 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_exportdir": 0, 2024-03-11T19:21:52.656 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_flush": 0, 2024-03-11T19:21:52.656 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragmentdir": 0, 2024-03-11T19:21:52.656 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragstats": 0, 2024-03-11T19:21:52.656 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_inodestats": 0 2024-03-11T19:21:52.656 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.656 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log": { 2024-03-11T19:21:52.656 INFO:teuthology.orchestra.run.smithi170.stdout: "evlrg": 0, 2024-03-11T19:21:52.656 INFO:teuthology.orchestra.run.smithi170.stdout: "evadd": 3, 2024-03-11T19:21:52.656 INFO:teuthology.orchestra.run.smithi170.stdout: "evex": 0, 2024-03-11T19:21:52.656 INFO:teuthology.orchestra.run.smithi170.stdout: "evtrm": 0, 2024-03-11T19:21:52.656 INFO:teuthology.orchestra.run.smithi170.stdout: "ev": 3, 2024-03-11T19:21:52.656 INFO:teuthology.orchestra.run.smithi170.stdout: "evexg": 0, 2024-03-11T19:21:52.656 INFO:teuthology.orchestra.run.smithi170.stdout: "evexd": 0, 2024-03-11T19:21:52.656 INFO:teuthology.orchestra.run.smithi170.stdout: "segadd": 2, 2024-03-11T19:21:52.656 INFO:teuthology.orchestra.run.smithi170.stdout: "segex": 0, 2024-03-11T19:21:52.657 INFO:teuthology.orchestra.run.smithi170.stdout: "segtrm": 0, 2024-03-11T19:21:52.657 INFO:teuthology.orchestra.run.smithi170.stdout: "seg": 2, 2024-03-11T19:21:52.657 INFO:teuthology.orchestra.run.smithi170.stdout: "segmjr": 2, 2024-03-11T19:21:52.657 INFO:teuthology.orchestra.run.smithi170.stdout: "segexg": 0, 2024-03-11T19:21:52.657 INFO:teuthology.orchestra.run.smithi170.stdout: "segexd": 0, 2024-03-11T19:21:52.657 INFO:teuthology.orchestra.run.smithi170.stdout: "expos": 4194304, 2024-03-11T19:21:52.657 INFO:teuthology.orchestra.run.smithi170.stdout: "wrpos": 4195385, 2024-03-11T19:21:52.657 INFO:teuthology.orchestra.run.smithi170.stdout: "rdpos": 0, 2024-03-11T19:21:52.657 INFO:teuthology.orchestra.run.smithi170.stdout: "jlat": { 2024-03-11T19:21:52.657 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2, 2024-03-11T19:21:52.657 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.003413727, 2024-03-11T19:21:52.657 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.001706863 2024-03-11T19:21:52.657 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.657 INFO:teuthology.orchestra.run.smithi170.stdout: "replayed": 0 2024-03-11T19:21:52.657 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.657 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_mem": { 2024-03-11T19:21:52.658 INFO:teuthology.orchestra.run.smithi170.stdout: "ino": 17, 2024-03-11T19:21:52.658 INFO:teuthology.orchestra.run.smithi170.stdout: "ino+": 17, 2024-03-11T19:21:52.658 INFO:teuthology.orchestra.run.smithi170.stdout: "ino-": 0, 2024-03-11T19:21:52.658 INFO:teuthology.orchestra.run.smithi170.stdout: "dir": 15, 2024-03-11T19:21:52.658 INFO:teuthology.orchestra.run.smithi170.stdout: "dir+": 16, 2024-03-11T19:21:52.658 INFO:teuthology.orchestra.run.smithi170.stdout: "dir-": 1, 2024-03-11T19:21:52.658 INFO:teuthology.orchestra.run.smithi170.stdout: "dn": 14, 2024-03-11T19:21:52.658 INFO:teuthology.orchestra.run.smithi170.stdout: "dn+": 14, 2024-03-11T19:21:52.658 INFO:teuthology.orchestra.run.smithi170.stdout: "dn-": 0, 2024-03-11T19:21:52.658 INFO:teuthology.orchestra.run.smithi170.stdout: "cap": 1, 2024-03-11T19:21:52.658 INFO:teuthology.orchestra.run.smithi170.stdout: "cap+": 1, 2024-03-11T19:21:52.658 INFO:teuthology.orchestra.run.smithi170.stdout: "cap-": 0, 2024-03-11T19:21:52.658 INFO:teuthology.orchestra.run.smithi170.stdout: "rss": 51572, 2024-03-11T19:21:52.658 INFO:teuthology.orchestra.run.smithi170.stdout: "heap": 207228 2024-03-11T19:21:52.658 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.659 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_server": { 2024-03-11T19:21:52.659 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_client_request": 3, 2024-03-11T19:21:52.659 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_server_request": 0, 2024-03-11T19:21:52.659 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_request": 2, 2024-03-11T19:21:52.659 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_session": 10, 2024-03-11T19:21:52.659 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_peer_request": 0, 2024-03-11T19:21:52.659 INFO:teuthology.orchestra.run.smithi170.stdout: "req_create_latency": { 2024-03-11T19:21:52.659 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.659 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.659 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.659 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.659 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getattr_latency": { 2024-03-11T19:21:52.659 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 1, 2024-03-11T19:21:52.659 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000882304, 2024-03-11T19:21:52.659 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000882304 2024-03-11T19:21:52.659 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.660 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getfilelock_latency": { 2024-03-11T19:21:52.660 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.660 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.660 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.660 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.660 INFO:teuthology.orchestra.run.smithi170.stdout: "req_link_latency": { 2024-03-11T19:21:52.660 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.660 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.660 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.660 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.660 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookup_latency": { 2024-03-11T19:21:52.660 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.660 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.660 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.660 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.660 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookuphash_latency": { 2024-03-11T19:21:52.661 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.661 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.661 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.661 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.661 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupino_latency": { 2024-03-11T19:21:52.661 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.661 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.661 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.661 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.661 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupname_latency": { 2024-03-11T19:21:52.661 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.661 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.661 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.661 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.661 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupparent_latency": { 2024-03-11T19:21:52.662 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.662 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.662 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.662 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.662 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupsnap_latency": { 2024-03-11T19:21:52.662 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.662 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.662 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.662 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.662 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lssnap_latency": { 2024-03-11T19:21:52.662 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.662 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.662 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.663 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.663 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mkdir_latency": { 2024-03-11T19:21:52.663 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.663 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.663 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.663 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.663 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mknod_latency": { 2024-03-11T19:21:52.663 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.664 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.664 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.664 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.664 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mksnap_latency": { 2024-03-11T19:21:52.664 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.664 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.664 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.664 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.664 INFO:teuthology.orchestra.run.smithi170.stdout: "req_open_latency": { 2024-03-11T19:21:52.664 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.664 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.664 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.664 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.664 INFO:teuthology.orchestra.run.smithi170.stdout: "req_readdir_latency": { 2024-03-11T19:21:52.664 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.665 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.665 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.665 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.665 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rename_latency": { 2024-03-11T19:21:52.665 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.665 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.665 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.665 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.665 INFO:teuthology.orchestra.run.smithi170.stdout: "req_renamesnap_latency": { 2024-03-11T19:21:52.665 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.665 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.665 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.665 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.665 INFO:teuthology.orchestra.run.smithi170.stdout: "req_snapdiff_latency": { 2024-03-11T19:21:52.666 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.666 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.666 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.666 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.666 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmdir_latency": { 2024-03-11T19:21:52.666 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.666 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.666 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.666 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.666 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmsnap_latency": { 2024-03-11T19:21:52.666 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.666 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.666 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.666 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.667 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmxattr_latency": { 2024-03-11T19:21:52.667 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.667 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.667 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.667 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.667 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setattr_latency": { 2024-03-11T19:21:52.667 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.667 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.667 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.667 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.667 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setdirlayout_latency": { 2024-03-11T19:21:52.667 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.667 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.667 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.667 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.667 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setfilelock_latency": { 2024-03-11T19:21:52.668 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.668 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.668 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.668 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.668 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setlayout_latency": { 2024-03-11T19:21:52.668 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.668 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.668 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.668 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.668 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setxattr_latency": { 2024-03-11T19:21:52.668 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.668 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.668 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.668 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.668 INFO:teuthology.orchestra.run.smithi170.stdout: "req_symlink_latency": { 2024-03-11T19:21:52.668 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.668 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.669 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.669 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.669 INFO:teuthology.orchestra.run.smithi170.stdout: "req_unlink_latency": { 2024-03-11T19:21:52.669 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.669 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.669 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.669 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.669 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_revoke_eviction": 0, 2024-03-11T19:21:52.669 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_acquisition_throttle": 0, 2024-03-11T19:21:52.669 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getvxattr_latency": { 2024-03-11T19:21:52.669 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.669 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.669 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.669 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:52.669 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.669 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_sessions": { 2024-03-11T19:21:52.670 INFO:teuthology.orchestra.run.smithi170.stdout: "session_count": 1, 2024-03-11T19:21:52.670 INFO:teuthology.orchestra.run.smithi170.stdout: "session_add": 1, 2024-03-11T19:21:52.670 INFO:teuthology.orchestra.run.smithi170.stdout: "session_remove": 0, 2024-03-11T19:21:52.670 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_open": 1, 2024-03-11T19:21:52.670 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_stale": 0, 2024-03-11T19:21:52.670 INFO:teuthology.orchestra.run.smithi170.stdout: "total_load": 1, 2024-03-11T19:21:52.670 INFO:teuthology.orchestra.run.smithi170.stdout: "average_load": 1, 2024-03-11T19:21:52.670 INFO:teuthology.orchestra.run.smithi170.stdout: "avg_session_uptime": 715, 2024-03-11T19:21:52.670 INFO:teuthology.orchestra.run.smithi170.stdout: "mdthresh_evicted": 0 2024-03-11T19:21:52.670 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.670 INFO:teuthology.orchestra.run.smithi170.stdout: "mempool": { 2024-03-11T19:21:52.670 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_bytes": 0, 2024-03-11T19:21:52.670 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_items": 0, 2024-03-11T19:21:52.670 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T19:21:52.671 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_items": 0, 2024-03-11T19:21:52.671 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T19:21:52.671 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_items": 0, 2024-03-11T19:21:52.671 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T19:21:52.671 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T19:21:52.671 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T19:21:52.671 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T19:21:52.671 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T19:21:52.671 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_items": 0, 2024-03-11T19:21:52.671 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T19:21:52.671 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T19:21:52.671 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_bytes": 0, 2024-03-11T19:21:52.671 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_items": 0, 2024-03-11T19:21:52.671 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_bytes": 0, 2024-03-11T19:21:52.672 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_items": 0, 2024-03-11T19:21:52.672 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T19:21:52.672 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T19:21:52.672 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T19:21:52.672 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T19:21:52.672 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T19:21:52.672 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_items": 0, 2024-03-11T19:21:52.672 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_bytes": 0, 2024-03-11T19:21:52.672 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_items": 0, 2024-03-11T19:21:52.672 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T19:21:52.672 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T19:21:52.672 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_bytes": 0, 2024-03-11T19:21:52.672 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_items": 0, 2024-03-11T19:21:52.672 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_bytes": 0, 2024-03-11T19:21:52.672 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_items": 0, 2024-03-11T19:21:52.672 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T19:21:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_items": 0, 2024-03-11T19:21:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T19:21:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_items": 0, 2024-03-11T19:21:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_bytes": 246607, 2024-03-11T19:21:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_items": 80, 2024-03-11T19:21:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_bytes": 0, 2024-03-11T19:21:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_items": 0, 2024-03-11T19:21:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_bytes": 0, 2024-03-11T19:21:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_items": 0, 2024-03-11T19:21:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_bytes": 0, 2024-03-11T19:21:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_items": 0, 2024-03-11T19:21:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_bytes": 0, 2024-03-11T19:21:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_items": 0, 2024-03-11T19:21:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_bytes": 7336, 2024-03-11T19:21:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_items": 154, 2024-03-11T19:21:52.673 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T19:21:52.674 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_items": 0, 2024-03-11T19:21:52.674 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_bytes": 0, 2024-03-11T19:21:52.674 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_items": 0, 2024-03-11T19:21:52.674 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_bytes": 66949, 2024-03-11T19:21:52.674 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_items": 288, 2024-03-11T19:21:52.674 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_bytes": 0, 2024-03-11T19:21:52.674 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_items": 0, 2024-03-11T19:21:52.674 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_bytes": 0, 2024-03-11T19:21:52.674 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_items": 0 2024-03-11T19:21:52.674 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.674 INFO:teuthology.orchestra.run.smithi170.stdout: "objecter": { 2024-03-11T19:21:52.674 INFO:teuthology.orchestra.run.smithi170.stdout: "op_active": 0, 2024-03-11T19:21:52.674 INFO:teuthology.orchestra.run.smithi170.stdout: "op_laggy": 0, 2024-03-11T19:21:52.674 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send": 38, 2024-03-11T19:21:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send_bytes": 11562, 2024-03-11T19:21:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: "op_resend": 27, 2024-03-11T19:21:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: "op_reply": 38, 2024-03-11T19:21:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: "op_latency": { 2024-03-11T19:21:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 38, 2024-03-11T19:21:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.206994822, 2024-03-11T19:21:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.005447232 2024-03-11T19:21:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: "op_inflight": 0, 2024-03-11T19:21:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: "oplen_avg": { 2024-03-11T19:21:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 38, 2024-03-11T19:21:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 61 2024-03-11T19:21:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: "op": 38, 2024-03-11T19:21:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: "op_r": 0, 2024-03-11T19:21:52.675 INFO:teuthology.orchestra.run.smithi170.stdout: "op_w": 38, 2024-03-11T19:21:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: "op_rmw": 0, 2024-03-11T19:21:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: "op_pg": 0, 2024-03-11T19:21:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_stat": 0, 2024-03-11T19:21:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_create": 10, 2024-03-11T19:21:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_read": 0, 2024-03-11T19:21:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_write": 2, 2024-03-11T19:21:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writefull": 6, 2024-03-11T19:21:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writesame": 0, 2024-03-11T19:21:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_append": 0, 2024-03-11T19:21:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_zero": 0, 2024-03-11T19:21:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_truncate": 0, 2024-03-11T19:21:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_delete": 6, 2024-03-11T19:21:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_mapext": 0, 2024-03-11T19:21:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_sparse_read": 0, 2024-03-11T19:21:52.676 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_clonerange": 0, 2024-03-11T19:21:52.677 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_getxattr": 0, 2024-03-11T19:21:52.677 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_setxattr": 20, 2024-03-11T19:21:52.677 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_cmpxattr": 0, 2024-03-11T19:21:52.677 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_rmxattr": 0, 2024-03-11T19:21:52.677 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_resetxattrs": 0, 2024-03-11T19:21:52.677 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_call": 0, 2024-03-11T19:21:52.677 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_watch": 0, 2024-03-11T19:21:52.677 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_notify": 0, 2024-03-11T19:21:52.677 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T19:21:52.677 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls": 0, 2024-03-11T19:21:52.677 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls_filter": 0, 2024-03-11T19:21:52.677 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_other": 0, 2024-03-11T19:21:52.677 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_active": 0, 2024-03-11T19:21:52.677 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_send": 0, 2024-03-11T19:21:52.678 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_resend": 0, 2024-03-11T19:21:52.678 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_ping": 0, 2024-03-11T19:21:52.678 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_active": 0, 2024-03-11T19:21:52.678 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_send": 0, 2024-03-11T19:21:52.678 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_resend": 0, 2024-03-11T19:21:52.678 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_active": 0, 2024-03-11T19:21:52.678 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_send": 0, 2024-03-11T19:21:52.678 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_resend": 0, 2024-03-11T19:21:52.678 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_active": 0, 2024-03-11T19:21:52.678 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_send": 0, 2024-03-11T19:21:52.678 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_resend": 0, 2024-03-11T19:21:52.678 INFO:teuthology.orchestra.run.smithi170.stdout: "command_active": 0, 2024-03-11T19:21:52.678 INFO:teuthology.orchestra.run.smithi170.stdout: "command_send": 0, 2024-03-11T19:21:52.678 INFO:teuthology.orchestra.run.smithi170.stdout: "command_resend": 0, 2024-03-11T19:21:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: "map_epoch": 89, 2024-03-11T19:21:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: "map_full": 0, 2024-03-11T19:21:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: "map_inc": 2, 2024-03-11T19:21:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_sessions": 9, 2024-03-11T19:21:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_open": 9, 2024-03-11T19:21:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_close": 0, 2024-03-11T19:21:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_laggy": 0, 2024-03-11T19:21:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_wr": 16, 2024-03-11T19:21:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_rd": 0, 2024-03-11T19:21:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_del": 1 2024-03-11T19:21:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: "oft": { 2024-03-11T19:21:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_objs": 1, 2024-03-11T19:21:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_kv_pairs": 0, 2024-03-11T19:21:52.679 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_updates": 0, 2024-03-11T19:21:52.680 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_removes": 0 2024-03-11T19:21:52.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.680 INFO:teuthology.orchestra.run.smithi170.stdout: "osd-slow-ops": { 2024-03-11T19:21:52.680 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_ops_count": 0 2024-03-11T19:21:52.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.680 INFO:teuthology.orchestra.run.smithi170.stdout: "purge_queue": { 2024-03-11T19:21:52.680 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops": 0, 2024-03-11T19:21:52.680 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T19:21:52.680 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing": 0, 2024-03-11T19:21:52.680 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_high_water": 0, 2024-03-11T19:21:52.680 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed_ops": 0, 2024-03-11T19:21:52.680 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed": 0, 2024-03-11T19:21:52.680 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_item_in_journal": 0 2024-03-11T19:21:52.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T19:21:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-03-11T19:21:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 693, 2024-03-11T19:21:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 806695, 2024-03-11T19:21:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 693, 2024-03-11T19:21:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-11T19:21:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-11T19:21:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 693, 2024-03-11T19:21:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 806695, 2024-03-11T19:21:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.681 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.682 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.682 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:52.682 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.682 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_bytes": { 2024-03-11T19:21:52.682 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:52.682 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-03-11T19:21:52.682 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:52.682 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-11T19:21:52.682 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-11T19:21:52.682 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:52.682 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:52.682 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 38, 2024-03-11T19:21:52.682 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 11562, 2024-03-11T19:21:52.682 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 38, 2024-03-11T19:21:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 11562, 2024-03-11T19:21:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_ops": { 2024-03-11T19:21:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 1024, 2024-03-11T19:21:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-11T19:21:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-11T19:21:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:52.683 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:52.684 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 38, 2024-03-11T19:21:52.684 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 38, 2024-03-11T19:21:52.684 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 38, 2024-03-11T19:21:52.684 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 38, 2024-03-11T19:21:52.684 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:52.684 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.684 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.684 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.684 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:52.684 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.684 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle": { 2024-03-11T19:21:52.684 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:52.684 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-03-11T19:21:52.684 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:52.684 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-11T19:21:52.685 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-11T19:21:52.685 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:52.685 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:52.685 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-11T19:21:52.685 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-11T19:21:52.685 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 0, 2024-03-11T19:21:52.685 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 0, 2024-03-11T19:21:52.685 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:52.685 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.685 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.685 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.685 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:52.685 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:52.685 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle-0x560c49d806e0": { 2024-03-11T19:21:52.685 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:52.686 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-03-11T19:21:52.686 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:52.686 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 3, 2024-03-11T19:21:52.686 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 1081, 2024-03-11T19:21:52.686 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:52.686 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 3, 2024-03-11T19:21:52.686 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-11T19:21:52.686 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-11T19:21:52.686 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 2, 2024-03-11T19:21:52.686 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 1081, 2024-03-11T19:21:52.686 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:52.686 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:52.686 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:52.686 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:52.687 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:52.687 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:52.687 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-03-11T19:21:53.491 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-11T19:21:53.492 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-11T19:21:53.492 DEBUG:tasks.check_counter:Getting stats from k 2024-03-11T19:21:53.492 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-11T19:21:53.734 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:21:53.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:53 smithi190 ceph-mon[36292]: pgmap v713: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-11T19:21:53.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:53 smithi170 ceph-mon[35690]: pgmap v713: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-11T19:21:53.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:53 smithi029 ceph-mon[32439]: pgmap v713: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-11T19:21:55.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:55 smithi170 ceph-mon[35690]: pgmap v714: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-11T19:21:55.697 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-03-11T19:21:55.697 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T19:21:55.697 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 841, 2024-03-11T19:21:55.697 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 843, 2024-03-11T19:21:55.697 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 222030, 2024-03-11T19:21:55.697 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 261615, 2024-03-11T19:21:55.697 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 6, 2024-03-11T19:21:55.697 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 2, 2024-03-11T19:21:55.697 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.063948535, 2024-03-11T19:21:55.697 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.023553498, 2024-03-11T19:21:55.698 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.015354163, 2024-03-11T19:21:55.698 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.018084637, 2024-03-11T19:21:55.698 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:55.698 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 841, 2024-03-11T19:21:55.698 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.024180537, 2024-03-11T19:21:55.698 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000028752 2024-03-11T19:21:55.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.698 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:55.698 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.698 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.698 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.698 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T19:21:55.698 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T19:21:55.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.699 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T19:21:55.699 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 726, 2024-03-11T19:21:55.699 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 902, 2024-03-11T19:21:55.699 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 180540, 2024-03-11T19:21:55.699 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 611356, 2024-03-11T19:21:55.699 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 5, 2024-03-11T19:21:55.699 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 4, 2024-03-11T19:21:55.699 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.061147092, 2024-03-11T19:21:55.699 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.020705926, 2024-03-11T19:21:55.699 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.012732582, 2024-03-11T19:21:55.699 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.021790888, 2024-03-11T19:21:55.699 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:55.699 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 901, 2024-03-11T19:21:55.699 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.028877324, 2024-03-11T19:21:55.699 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000032050 2024-03-11T19:21:55.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.700 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:55.700 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.700 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.700 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.700 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.700 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-11T19:21:55.700 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 387120 2024-03-11T19:21:55.700 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.700 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T19:21:55.700 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 1084, 2024-03-11T19:21:55.700 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 1078, 2024-03-11T19:21:55.700 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 1054816, 2024-03-11T19:21:55.700 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 402348, 2024-03-11T19:21:55.700 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 3, 2024-03-11T19:21:55.700 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 2, 2024-03-11T19:21:55.701 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.115438885, 2024-03-11T19:21:55.701 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.037423036, 2024-03-11T19:21:55.701 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.040436324, 2024-03-11T19:21:55.701 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.032631071, 2024-03-11T19:21:55.701 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:55.701 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 1077, 2024-03-11T19:21:55.701 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.044591480, 2024-03-11T19:21:55.701 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000041403 2024-03-11T19:21:55.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.701 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:55.701 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.701 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.701 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.702 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 773920, 2024-03-11T19:21:55.702 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 142352 2024-03-11T19:21:55.702 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.702 INFO:teuthology.orchestra.run.smithi170.stdout: "cct": { 2024-03-11T19:21:55.702 INFO:teuthology.orchestra.run.smithi170.stdout: "total_workers": 1, 2024-03-11T19:21:55.702 INFO:teuthology.orchestra.run.smithi170.stdout: "unhealthy_workers": 0 2024-03-11T19:21:55.702 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.702 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-MDSRank": { 2024-03-11T19:21:55.702 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-03-11T19:21:55.702 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-03-11T19:21:55.702 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 4018, 2024-03-11T19:21:55.702 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.108543385, 2024-03-11T19:21:55.702 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000027014 2024-03-11T19:21:55.702 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:55.703 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.703 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-PurgeQueue": { 2024-03-11T19:21:55.703 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-03-11T19:21:55.703 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-03-11T19:21:55.703 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.703 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.703 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.703 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:55.703 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.703 INFO:teuthology.orchestra.run.smithi170.stdout: "mds": { 2024-03-11T19:21:55.703 INFO:teuthology.orchestra.run.smithi170.stdout: "request": 0, 2024-03-11T19:21:55.703 INFO:teuthology.orchestra.run.smithi170.stdout: "reply": 0, 2024-03-11T19:21:55.703 INFO:teuthology.orchestra.run.smithi170.stdout: "reply_latency": { 2024-03-11T19:21:55.703 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.703 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.704 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.704 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.704 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_reply": 0, 2024-03-11T19:21:55.704 INFO:teuthology.orchestra.run.smithi170.stdout: "forward": 0, 2024-03-11T19:21:55.704 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_complete": 0, 2024-03-11T19:21:55.704 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_keys": 0, 2024-03-11T19:21:55.704 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_commit": 0, 2024-03-11T19:21:55.704 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_split": 0, 2024-03-11T19:21:55.704 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_merge": 0, 2024-03-11T19:21:55.704 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes": 0, 2024-03-11T19:21:55.704 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_top": 0, 2024-03-11T19:21:55.705 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_bottom": 0, 2024-03-11T19:21:55.705 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pin_tail": 0, 2024-03-11T19:21:55.705 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pinned": 0, 2024-03-11T19:21:55.705 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_expired": 0, 2024-03-11T19:21:55.705 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_with_caps": 0, 2024-03-11T19:21:55.705 INFO:teuthology.orchestra.run.smithi170.stdout: "caps": 0, 2024-03-11T19:21:55.705 INFO:teuthology.orchestra.run.smithi170.stdout: "subtrees": 1, 2024-03-11T19:21:55.705 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse": 0, 2024-03-11T19:21:55.705 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_hit": 0, 2024-03-11T19:21:55.705 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_forward": 0, 2024-03-11T19:21:55.705 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_discover": 0, 2024-03-11T19:21:55.705 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_dir_fetch": 0, 2024-03-11T19:21:55.705 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_remote_ino": 0, 2024-03-11T19:21:55.705 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_lock": 0, 2024-03-11T19:21:55.705 INFO:teuthology.orchestra.run.smithi170.stdout: "load_cent": 0, 2024-03-11T19:21:55.706 INFO:teuthology.orchestra.run.smithi170.stdout: "q": 0, 2024-03-11T19:21:55.706 INFO:teuthology.orchestra.run.smithi170.stdout: "exported": 0, 2024-03-11T19:21:55.706 INFO:teuthology.orchestra.run.smithi170.stdout: "exported_inodes": 0, 2024-03-11T19:21:55.706 INFO:teuthology.orchestra.run.smithi170.stdout: "imported": 0, 2024-03-11T19:21:55.706 INFO:teuthology.orchestra.run.smithi170.stdout: "imported_inodes": 0, 2024-03-11T19:21:55.706 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_dir_fetch": 0, 2024-03-11T19:21:55.706 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_backtrace_fetch": 0, 2024-03-11T19:21:55.706 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_peer_discover": 0, 2024-03-11T19:21:55.706 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rfiles": 0, 2024-03-11T19:21:55.706 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rbytes": 0, 2024-03-11T19:21:55.706 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rsnaps": 0, 2024-03-11T19:21:55.706 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T19:21:55.706 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_set_tag": 0, 2024-03-11T19:21:55.706 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T19:21:55.707 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_inotable_repaired": 0, 2024-03-11T19:21:55.707 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_inodes": 0, 2024-03-11T19:21:55.707 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T19:21:55.707 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T19:21:55.707 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_file_inodes": 0, 2024-03-11T19:21:55.707 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_inode_file_caps": 0, 2024-03-11T19:21:55.707 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T19:21:55.707 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_grant": 0, 2024-03-11T19:21:55.707 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T19:21:55.707 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T19:21:55.707 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T19:21:55.707 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps": 0, 2024-03-11T19:21:55.707 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps_dirty": 0, 2024-03-11T19:21:55.707 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_cap_release": 0, 2024-03-11T19:21:55.708 INFO:teuthology.orchestra.run.smithi170.stdout: "process_request_cap_release": 0 2024-03-11T19:21:55.708 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.708 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cache": { 2024-03-11T19:21:55.708 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update": 0, 2024-03-11T19:21:55.708 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update_receipt": 0, 2024-03-11T19:21:55.708 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_try_discover": 0, 2024-03-11T19:21:55.708 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_send_discover": 0, 2024-03-11T19:21:55.708 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_handle_discover": 0, 2024-03-11T19:21:55.708 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays": 0, 2024-03-11T19:21:55.708 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_delayed": 0, 2024-03-11T19:21:55.708 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_enqueuing": 0, 2024-03-11T19:21:55.708 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_created": 0, 2024-03-11T19:21:55.708 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_enqueued": 0, 2024-03-11T19:21:55.708 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_reintegrated": 0, 2024-03-11T19:21:55.708 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_migrated": 0, 2024-03-11T19:21:55.709 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_processing": 0, 2024-03-11T19:21:55.709 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_enqueued": 0, 2024-03-11T19:21:55.709 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_prioritized": 0, 2024-03-11T19:21:55.709 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_started": 0, 2024-03-11T19:21:55.709 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_completed": 0, 2024-03-11T19:21:55.709 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T19:21:55.709 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_exportdir": 0, 2024-03-11T19:21:55.709 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_flush": 0, 2024-03-11T19:21:55.709 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragmentdir": 0, 2024-03-11T19:21:55.709 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragstats": 0, 2024-03-11T19:21:55.709 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_inodestats": 0 2024-03-11T19:21:55.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.709 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log": { 2024-03-11T19:21:55.709 INFO:teuthology.orchestra.run.smithi170.stdout: "evlrg": 0, 2024-03-11T19:21:55.709 INFO:teuthology.orchestra.run.smithi170.stdout: "evadd": 0, 2024-03-11T19:21:55.710 INFO:teuthology.orchestra.run.smithi170.stdout: "evex": 0, 2024-03-11T19:21:55.710 INFO:teuthology.orchestra.run.smithi170.stdout: "evtrm": 0, 2024-03-11T19:21:55.710 INFO:teuthology.orchestra.run.smithi170.stdout: "ev": 3, 2024-03-11T19:21:55.710 INFO:teuthology.orchestra.run.smithi170.stdout: "evexg": 0, 2024-03-11T19:21:55.710 INFO:teuthology.orchestra.run.smithi170.stdout: "evexd": 0, 2024-03-11T19:21:55.710 INFO:teuthology.orchestra.run.smithi170.stdout: "segadd": 0, 2024-03-11T19:21:55.710 INFO:teuthology.orchestra.run.smithi170.stdout: "segex": 0, 2024-03-11T19:21:55.710 INFO:teuthology.orchestra.run.smithi170.stdout: "segtrm": 0, 2024-03-11T19:21:55.710 INFO:teuthology.orchestra.run.smithi170.stdout: "seg": 2, 2024-03-11T19:21:55.710 INFO:teuthology.orchestra.run.smithi170.stdout: "segmjr": 2, 2024-03-11T19:21:55.710 INFO:teuthology.orchestra.run.smithi170.stdout: "segexg": 0, 2024-03-11T19:21:55.710 INFO:teuthology.orchestra.run.smithi170.stdout: "segexd": 0, 2024-03-11T19:21:55.710 INFO:teuthology.orchestra.run.smithi170.stdout: "expos": 4194304, 2024-03-11T19:21:55.710 INFO:teuthology.orchestra.run.smithi170.stdout: "wrpos": 4195385, 2024-03-11T19:21:55.710 INFO:teuthology.orchestra.run.smithi170.stdout: "rdpos": 4194892, 2024-03-11T19:21:55.711 INFO:teuthology.orchestra.run.smithi170.stdout: "jlat": { 2024-03-11T19:21:55.711 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.711 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.711 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.711 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.711 INFO:teuthology.orchestra.run.smithi170.stdout: "replayed": 3 2024-03-11T19:21:55.711 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.711 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_mem": { 2024-03-11T19:21:55.711 INFO:teuthology.orchestra.run.smithi170.stdout: "ino": 3, 2024-03-11T19:21:55.711 INFO:teuthology.orchestra.run.smithi170.stdout: "ino+": 3, 2024-03-11T19:21:55.711 INFO:teuthology.orchestra.run.smithi170.stdout: "ino-": 0, 2024-03-11T19:21:55.711 INFO:teuthology.orchestra.run.smithi170.stdout: "dir": 1, 2024-03-11T19:21:55.711 INFO:teuthology.orchestra.run.smithi170.stdout: "dir+": 1, 2024-03-11T19:21:55.711 INFO:teuthology.orchestra.run.smithi170.stdout: "dir-": 0, 2024-03-11T19:21:55.712 INFO:teuthology.orchestra.run.smithi170.stdout: "dn": 0, 2024-03-11T19:21:55.712 INFO:teuthology.orchestra.run.smithi170.stdout: "dn+": 0, 2024-03-11T19:21:55.712 INFO:teuthology.orchestra.run.smithi170.stdout: "dn-": 0, 2024-03-11T19:21:55.712 INFO:teuthology.orchestra.run.smithi170.stdout: "cap": 0, 2024-03-11T19:21:55.712 INFO:teuthology.orchestra.run.smithi170.stdout: "cap+": 0, 2024-03-11T19:21:55.712 INFO:teuthology.orchestra.run.smithi170.stdout: "cap-": 0, 2024-03-11T19:21:55.712 INFO:teuthology.orchestra.run.smithi170.stdout: "rss": 48936, 2024-03-11T19:21:55.712 INFO:teuthology.orchestra.run.smithi170.stdout: "heap": 215420 2024-03-11T19:21:55.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.712 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_server": { 2024-03-11T19:21:55.712 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_client_request": 0, 2024-03-11T19:21:55.712 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_server_request": 0, 2024-03-11T19:21:55.712 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_request": 0, 2024-03-11T19:21:55.712 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_session": 0, 2024-03-11T19:21:55.712 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_peer_request": 0, 2024-03-11T19:21:55.712 INFO:teuthology.orchestra.run.smithi170.stdout: "req_create_latency": { 2024-03-11T19:21:55.713 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.713 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.713 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.713 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.713 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getattr_latency": { 2024-03-11T19:21:55.713 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.713 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.713 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.713 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.713 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getfilelock_latency": { 2024-03-11T19:21:55.713 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.713 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.713 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.713 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.713 INFO:teuthology.orchestra.run.smithi170.stdout: "req_link_latency": { 2024-03-11T19:21:55.713 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.714 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.714 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.714 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookup_latency": { 2024-03-11T19:21:55.714 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.714 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.714 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.714 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookuphash_latency": { 2024-03-11T19:21:55.714 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.714 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.714 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.714 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupino_latency": { 2024-03-11T19:21:55.714 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.715 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.715 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.715 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.715 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupname_latency": { 2024-03-11T19:21:55.715 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.715 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.715 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.715 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.715 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupparent_latency": { 2024-03-11T19:21:55.715 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.715 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.715 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.715 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.715 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupsnap_latency": { 2024-03-11T19:21:55.715 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.716 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.716 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.716 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.716 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lssnap_latency": { 2024-03-11T19:21:55.716 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.716 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.716 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.716 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.716 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mkdir_latency": { 2024-03-11T19:21:55.716 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.716 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.716 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.716 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.716 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mknod_latency": { 2024-03-11T19:21:55.716 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.716 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.717 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.717 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mksnap_latency": { 2024-03-11T19:21:55.717 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.717 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.717 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.717 INFO:teuthology.orchestra.run.smithi170.stdout: "req_open_latency": { 2024-03-11T19:21:55.717 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.717 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.717 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.717 INFO:teuthology.orchestra.run.smithi170.stdout: "req_readdir_latency": { 2024-03-11T19:21:55.717 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.717 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.718 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.718 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.718 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rename_latency": { 2024-03-11T19:21:55.718 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.718 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.718 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.718 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.718 INFO:teuthology.orchestra.run.smithi170.stdout: "req_renamesnap_latency": { 2024-03-11T19:21:55.718 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.718 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.718 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.718 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.718 INFO:teuthology.orchestra.run.smithi170.stdout: "req_snapdiff_latency": { 2024-03-11T19:21:55.718 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.718 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.719 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.719 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.719 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmdir_latency": { 2024-03-11T19:21:55.719 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.719 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.719 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.719 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.719 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmsnap_latency": { 2024-03-11T19:21:55.719 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.719 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.719 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.719 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.719 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmxattr_latency": { 2024-03-11T19:21:55.719 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.720 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.720 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.720 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.720 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setattr_latency": { 2024-03-11T19:21:55.720 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.720 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.720 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.720 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.720 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setdirlayout_latency": { 2024-03-11T19:21:55.720 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.720 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.720 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.720 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.720 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setfilelock_latency": { 2024-03-11T19:21:55.720 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.721 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.721 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.721 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.721 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setlayout_latency": { 2024-03-11T19:21:55.721 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.721 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.721 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.721 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.721 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setxattr_latency": { 2024-03-11T19:21:55.721 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.721 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.721 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.721 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.721 INFO:teuthology.orchestra.run.smithi170.stdout: "req_symlink_latency": { 2024-03-11T19:21:55.721 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.722 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.722 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.722 INFO:teuthology.orchestra.run.smithi170.stdout: "req_unlink_latency": { 2024-03-11T19:21:55.722 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.722 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.722 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.722 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_revoke_eviction": 0, 2024-03-11T19:21:55.722 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_acquisition_throttle": 0, 2024-03-11T19:21:55.722 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getvxattr_latency": { 2024-03-11T19:21:55.722 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.722 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.722 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.722 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:55.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.723 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_sessions": { 2024-03-11T19:21:55.723 INFO:teuthology.orchestra.run.smithi170.stdout: "session_count": 1, 2024-03-11T19:21:55.723 INFO:teuthology.orchestra.run.smithi170.stdout: "session_add": 1, 2024-03-11T19:21:55.723 INFO:teuthology.orchestra.run.smithi170.stdout: "session_remove": 0, 2024-03-11T19:21:55.723 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_open": 1, 2024-03-11T19:21:55.723 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_stale": 0, 2024-03-11T19:21:55.723 INFO:teuthology.orchestra.run.smithi170.stdout: "total_load": 0, 2024-03-11T19:21:55.723 INFO:teuthology.orchestra.run.smithi170.stdout: "average_load": 0, 2024-03-11T19:21:55.723 INFO:teuthology.orchestra.run.smithi170.stdout: "avg_session_uptime": 1862, 2024-03-11T19:21:55.723 INFO:teuthology.orchestra.run.smithi170.stdout: "mdthresh_evicted": 0 2024-03-11T19:21:55.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.723 INFO:teuthology.orchestra.run.smithi170.stdout: "mempool": { 2024-03-11T19:21:55.723 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_bytes": 0, 2024-03-11T19:21:55.723 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_items": 0, 2024-03-11T19:21:55.724 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T19:21:55.724 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_items": 0, 2024-03-11T19:21:55.724 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T19:21:55.724 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_items": 0, 2024-03-11T19:21:55.724 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T19:21:55.724 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T19:21:55.724 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T19:21:55.724 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T19:21:55.724 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T19:21:55.724 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_items": 0, 2024-03-11T19:21:55.724 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T19:21:55.724 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T19:21:55.724 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_bytes": 0, 2024-03-11T19:21:55.724 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_items": 0, 2024-03-11T19:21:55.724 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_bytes": 0, 2024-03-11T19:21:55.725 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_items": 0, 2024-03-11T19:21:55.725 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T19:21:55.725 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T19:21:55.725 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T19:21:55.725 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T19:21:55.725 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T19:21:55.725 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_items": 0, 2024-03-11T19:21:55.725 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_bytes": 0, 2024-03-11T19:21:55.725 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_items": 0, 2024-03-11T19:21:55.725 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T19:21:55.725 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T19:21:55.725 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_bytes": 0, 2024-03-11T19:21:55.725 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_items": 0, 2024-03-11T19:21:55.725 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_bytes": 0, 2024-03-11T19:21:55.726 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_items": 0, 2024-03-11T19:21:55.726 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T19:21:55.726 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_items": 0, 2024-03-11T19:21:55.726 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T19:21:55.726 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_items": 0, 2024-03-11T19:21:55.726 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_bytes": 189400, 2024-03-11T19:21:55.726 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_items": 51, 2024-03-11T19:21:55.726 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_bytes": 0, 2024-03-11T19:21:55.726 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_items": 0, 2024-03-11T19:21:55.726 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_bytes": 0, 2024-03-11T19:21:55.726 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_items": 0, 2024-03-11T19:21:55.726 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_bytes": 0, 2024-03-11T19:21:55.726 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_items": 0, 2024-03-11T19:21:55.726 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_bytes": 0, 2024-03-11T19:21:55.727 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_items": 0, 2024-03-11T19:21:55.727 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_bytes": 7336, 2024-03-11T19:21:55.727 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_items": 154, 2024-03-11T19:21:55.727 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T19:21:55.727 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_items": 0, 2024-03-11T19:21:55.727 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_bytes": 0, 2024-03-11T19:21:55.727 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_items": 0, 2024-03-11T19:21:55.727 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_bytes": 8320, 2024-03-11T19:21:55.727 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_items": 13, 2024-03-11T19:21:55.727 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_bytes": 0, 2024-03-11T19:21:55.727 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_items": 0, 2024-03-11T19:21:55.727 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_bytes": 0, 2024-03-11T19:21:55.727 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_items": 0 2024-03-11T19:21:55.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.727 INFO:teuthology.orchestra.run.smithi170.stdout: "objecter": { 2024-03-11T19:21:55.728 INFO:teuthology.orchestra.run.smithi170.stdout: "op_active": 0, 2024-03-11T19:21:55.728 INFO:teuthology.orchestra.run.smithi170.stdout: "op_laggy": 0, 2024-03-11T19:21:55.728 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send": 2398, 2024-03-11T19:21:55.728 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send_bytes": 21, 2024-03-11T19:21:55.728 INFO:teuthology.orchestra.run.smithi170.stdout: "op_resend": 3, 2024-03-11T19:21:55.728 INFO:teuthology.orchestra.run.smithi170.stdout: "op_reply": 2398, 2024-03-11T19:21:55.728 INFO:teuthology.orchestra.run.smithi170.stdout: "op_latency": { 2024-03-11T19:21:55.728 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2398, 2024-03-11T19:21:55.728 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 3.924900785, 2024-03-11T19:21:55.728 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.001636739 2024-03-11T19:21:55.728 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.728 INFO:teuthology.orchestra.run.smithi170.stdout: "op_inflight": 0, 2024-03-11T19:21:55.728 INFO:teuthology.orchestra.run.smithi170.stdout: "oplen_avg": { 2024-03-11T19:21:55.728 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2398, 2024-03-11T19:21:55.728 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 2399 2024-03-11T19:21:55.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.729 INFO:teuthology.orchestra.run.smithi170.stdout: "op": 2398, 2024-03-11T19:21:55.729 INFO:teuthology.orchestra.run.smithi170.stdout: "op_r": 2398, 2024-03-11T19:21:55.729 INFO:teuthology.orchestra.run.smithi170.stdout: "op_w": 0, 2024-03-11T19:21:55.729 INFO:teuthology.orchestra.run.smithi170.stdout: "op_rmw": 0, 2024-03-11T19:21:55.729 INFO:teuthology.orchestra.run.smithi170.stdout: "op_pg": 0, 2024-03-11T19:21:55.729 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_stat": 1555, 2024-03-11T19:21:55.729 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_create": 0, 2024-03-11T19:21:55.729 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_read": 841, 2024-03-11T19:21:55.729 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_write": 0, 2024-03-11T19:21:55.729 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writefull": 0, 2024-03-11T19:21:55.729 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writesame": 0, 2024-03-11T19:21:55.729 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_append": 0, 2024-03-11T19:21:55.729 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_zero": 0, 2024-03-11T19:21:55.729 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_truncate": 0, 2024-03-11T19:21:55.729 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_delete": 0, 2024-03-11T19:21:55.730 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_mapext": 0, 2024-03-11T19:21:55.730 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_sparse_read": 0, 2024-03-11T19:21:55.730 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_clonerange": 0, 2024-03-11T19:21:55.730 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_getxattr": 1, 2024-03-11T19:21:55.730 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_setxattr": 0, 2024-03-11T19:21:55.730 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_cmpxattr": 0, 2024-03-11T19:21:55.730 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_rmxattr": 0, 2024-03-11T19:21:55.730 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_resetxattrs": 0, 2024-03-11T19:21:55.730 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_call": 0, 2024-03-11T19:21:55.730 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_watch": 0, 2024-03-11T19:21:55.730 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_notify": 0, 2024-03-11T19:21:55.730 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T19:21:55.730 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls": 0, 2024-03-11T19:21:55.730 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls_filter": 0, 2024-03-11T19:21:55.730 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_other": 0, 2024-03-11T19:21:55.731 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_active": 0, 2024-03-11T19:21:55.731 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_send": 0, 2024-03-11T19:21:55.731 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_resend": 0, 2024-03-11T19:21:55.731 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_ping": 0, 2024-03-11T19:21:55.731 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_active": 0, 2024-03-11T19:21:55.731 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_send": 0, 2024-03-11T19:21:55.731 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_resend": 0, 2024-03-11T19:21:55.731 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_active": 0, 2024-03-11T19:21:55.731 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_send": 0, 2024-03-11T19:21:55.731 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_resend": 0, 2024-03-11T19:21:55.731 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_active": 0, 2024-03-11T19:21:55.731 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_send": 0, 2024-03-11T19:21:55.731 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_resend": 0, 2024-03-11T19:21:55.731 INFO:teuthology.orchestra.run.smithi170.stdout: "command_active": 0, 2024-03-11T19:21:55.731 INFO:teuthology.orchestra.run.smithi170.stdout: "command_send": 0, 2024-03-11T19:21:55.732 INFO:teuthology.orchestra.run.smithi170.stdout: "command_resend": 0, 2024-03-11T19:21:55.732 INFO:teuthology.orchestra.run.smithi170.stdout: "map_epoch": 89, 2024-03-11T19:21:55.732 INFO:teuthology.orchestra.run.smithi170.stdout: "map_full": 0, 2024-03-11T19:21:55.732 INFO:teuthology.orchestra.run.smithi170.stdout: "map_inc": 2, 2024-03-11T19:21:55.732 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_sessions": 5, 2024-03-11T19:21:55.732 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_open": 5, 2024-03-11T19:21:55.732 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_close": 0, 2024-03-11T19:21:55.732 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_laggy": 0, 2024-03-11T19:21:55.732 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_wr": 0, 2024-03-11T19:21:55.732 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_rd": 2, 2024-03-11T19:21:55.732 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_del": 0 2024-03-11T19:21:55.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.732 INFO:teuthology.orchestra.run.smithi170.stdout: "oft": { 2024-03-11T19:21:55.732 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_objs": 0, 2024-03-11T19:21:55.733 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_kv_pairs": 0, 2024-03-11T19:21:55.733 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_updates": 0, 2024-03-11T19:21:55.733 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_removes": 0 2024-03-11T19:21:55.733 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.733 INFO:teuthology.orchestra.run.smithi170.stdout: "osd-slow-ops": { 2024-03-11T19:21:55.733 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_ops_count": 0 2024-03-11T19:21:55.733 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.733 INFO:teuthology.orchestra.run.smithi170.stdout: "purge_queue": { 2024-03-11T19:21:55.733 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops": 0, 2024-03-11T19:21:55.733 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T19:21:55.733 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing": 0, 2024-03-11T19:21:55.733 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_high_water": 0, 2024-03-11T19:21:55.733 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed_ops": 0, 2024-03-11T19:21:55.733 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed": 0, 2024-03-11T19:21:55.733 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_item_in_journal": 0 2024-03-11T19:21:55.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.734 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T19:21:55.734 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:55.734 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-03-11T19:21:55.734 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:55.734 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 2647, 2024-03-11T19:21:55.734 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 1203417, 2024-03-11T19:21:55.734 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:55.734 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 2647, 2024-03-11T19:21:55.734 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-11T19:21:55.734 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-11T19:21:55.734 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 2647, 2024-03-11T19:21:55.734 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 1203417, 2024-03-11T19:21:55.734 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:55.734 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.735 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.735 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.735 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:55.735 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.735 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_bytes": { 2024-03-11T19:21:55.735 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:55.735 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-03-11T19:21:55.735 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:55.735 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-11T19:21:55.735 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-11T19:21:55.735 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:55.735 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:55.735 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 2398, 2024-03-11T19:21:55.735 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 1086, 2024-03-11T19:21:55.735 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 2398, 2024-03-11T19:21:55.736 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 1086, 2024-03-11T19:21:55.736 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:55.736 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.736 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.736 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.736 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:55.736 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.736 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_ops": { 2024-03-11T19:21:55.736 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:55.736 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 1024, 2024-03-11T19:21:55.736 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:55.736 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-11T19:21:55.736 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-11T19:21:55.736 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:55.736 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:55.737 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 2398, 2024-03-11T19:21:55.737 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 2398, 2024-03-11T19:21:55.737 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 2398, 2024-03-11T19:21:55.737 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 2398, 2024-03-11T19:21:55.737 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:55.737 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.737 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.737 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.737 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:55.737 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.737 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle": { 2024-03-11T19:21:55.737 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:55.737 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-03-11T19:21:55.737 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:55.738 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-11T19:21:55.738 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-11T19:21:55.738 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:55.738 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:55.738 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-11T19:21:55.738 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-11T19:21:55.738 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 0, 2024-03-11T19:21:55.738 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 0, 2024-03-11T19:21:55.738 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:55.738 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.738 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.738 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.738 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:55.738 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-11T19:21:55.739 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle-0x563832faa960": { 2024-03-11T19:21:55.739 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-11T19:21:55.739 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-03-11T19:21:55.739 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-11T19:21:55.739 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-11T19:21:55.739 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-11T19:21:55.739 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:55.739 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:55.739 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-11T19:21:55.739 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-11T19:21:55.739 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 0, 2024-03-11T19:21:55.739 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 0, 2024-03-11T19:21:55.739 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-11T19:21:55.739 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-11T19:21:55.739 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-11T19:21:55.739 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-11T19:21:55.740 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:55.740 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-11T19:21:55.740 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-03-11T19:21:55.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:55 smithi190 ceph-mon[36292]: pgmap v714: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-11T19:21:55.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:55 smithi029 ceph-mon[32439]: pgmap v714: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-11T19:21:56.467 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-11T19:21:56.468 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-03-11T19:21:56.468 DEBUG:tasks.check_counter:Getting stats from c 2024-03-11T19:21:56.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-11T19:21:56.721 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:21:57.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:57 smithi190 ceph-mon[36292]: pgmap v715: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-11T19:21:57.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:57 smithi170 ceph-mon[35690]: pgmap v715: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-11T19:21:57.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:57 smithi029 ceph-mon[32439]: pgmap v715: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-11T19:21:58.639 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-11T19:21:58.639 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T19:21:58.639 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 33, 2024-03-11T19:21:58.639 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 35, 2024-03-11T19:21:58.639 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 7255, 2024-03-11T19:21:58.639 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 50154, 2024-03-11T19:21:58.639 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 12, 2024-03-11T19:21:58.639 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 3, 2024-03-11T19:21:58.639 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.009368488, 2024-03-11T19:21:58.639 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.000648009, 2024-03-11T19:21:58.640 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.004512479, 2024-03-11T19:21:58.640 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.000428959, 2024-03-11T19:21:58.640 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:58.640 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 33, 2024-03-11T19:21:58.640 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.022523419, 2024-03-11T19:21:58.640 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000682527 2024-03-11T19:21:58.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.640 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:58.640 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 26, 2024-03-11T19:21:58.640 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000005440, 2024-03-11T19:21:58.640 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000209 2024-03-11T19:21:58.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.640 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T19:21:58.640 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T19:21:58.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.641 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T19:21:58.641 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 60, 2024-03-11T19:21:58.641 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 234, 2024-03-11T19:21:58.641 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 13014, 2024-03-11T19:21:58.641 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 786261, 2024-03-11T19:21:58.641 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 46, 2024-03-11T19:21:58.641 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 5, 2024-03-11T19:21:58.641 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.023761282, 2024-03-11T19:21:58.641 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.008022449, 2024-03-11T19:21:58.641 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.013001858, 2024-03-11T19:21:58.641 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.000446265, 2024-03-11T19:21:58.641 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:58.641 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 233, 2024-03-11T19:21:58.641 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.016893995, 2024-03-11T19:21:58.642 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000072506 2024-03-11T19:21:58.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.642 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:58.642 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 40, 2024-03-11T19:21:58.642 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000004835, 2024-03-11T19:21:58.642 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000120 2024-03-11T19:21:58.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.642 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-11T19:21:58.642 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 388368 2024-03-11T19:21:58.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.642 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T19:21:58.642 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 585, 2024-03-11T19:21:58.642 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 774, 2024-03-11T19:21:58.642 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 863435, 2024-03-11T19:21:58.643 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 346926, 2024-03-11T19:21:58.643 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 5, 2024-03-11T19:21:58.643 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 4, 2024-03-11T19:21:58.643 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.077096316, 2024-03-11T19:21:58.643 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.028940719, 2024-03-11T19:21:58.643 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.038487232, 2024-03-11T19:21:58.643 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.010174482, 2024-03-11T19:21:58.643 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:21:58.643 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 773, 2024-03-11T19:21:58.643 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.032682216, 2024-03-11T19:21:58.643 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000042279 2024-03-11T19:21:58.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.643 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:21:58.643 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 793, 2024-03-11T19:21:58.644 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000205644, 2024-03-11T19:21:58.644 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000259 2024-03-11T19:21:58.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.644 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 792720, 2024-03-11T19:21:58.644 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 145904 2024-03-11T19:21:58.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.644 INFO:teuthology.orchestra.run.smithi190.stdout: "cct": { 2024-03-11T19:21:58.644 INFO:teuthology.orchestra.run.smithi190.stdout: "total_workers": 1, 2024-03-11T19:21:58.644 INFO:teuthology.orchestra.run.smithi190.stdout: "unhealthy_workers": 0 2024-03-11T19:21:58.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.644 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-MDSRank": { 2024-03-11T19:21:58.644 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-03-11T19:21:58.644 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-03-11T19:21:58.644 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 45, 2024-03-11T19:21:58.644 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.001768100, 2024-03-11T19:21:58.644 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000039291 2024-03-11T19:21:58.645 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:21:58.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.645 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-PurgeQueue": { 2024-03-11T19:21:58.645 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-03-11T19:21:58.645 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-03-11T19:21:58.645 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 2, 2024-03-11T19:21:58.645 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000009945, 2024-03-11T19:21:58.645 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000004972 2024-03-11T19:21:58.645 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:21:58.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.645 INFO:teuthology.orchestra.run.smithi190.stdout: "mds": { 2024-03-11T19:21:58.645 INFO:teuthology.orchestra.run.smithi190.stdout: "request": 1, 2024-03-11T19:21:58.645 INFO:teuthology.orchestra.run.smithi190.stdout: "reply": 1, 2024-03-11T19:21:58.645 INFO:teuthology.orchestra.run.smithi190.stdout: "reply_latency": { 2024-03-11T19:21:58.645 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1, 2024-03-11T19:21:58.646 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000609425, 2024-03-11T19:21:58.646 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000609425 2024-03-11T19:21:58.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.646 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_reply": 0, 2024-03-11T19:21:58.646 INFO:teuthology.orchestra.run.smithi190.stdout: "forward": 0, 2024-03-11T19:21:58.646 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_complete": 0, 2024-03-11T19:21:58.646 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_keys": 0, 2024-03-11T19:21:58.646 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_commit": 11, 2024-03-11T19:21:58.646 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_split": 0, 2024-03-11T19:21:58.646 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_merge": 0, 2024-03-11T19:21:58.646 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes": 11, 2024-03-11T19:21:58.646 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_top": 0, 2024-03-11T19:21:58.646 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_bottom": 11, 2024-03-11T19:21:58.646 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pin_tail": 0, 2024-03-11T19:21:58.646 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pinned": 11, 2024-03-11T19:21:58.647 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_expired": 0, 2024-03-11T19:21:58.647 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_with_caps": 1, 2024-03-11T19:21:58.647 INFO:teuthology.orchestra.run.smithi190.stdout: "caps": 1, 2024-03-11T19:21:58.647 INFO:teuthology.orchestra.run.smithi190.stdout: "subtrees": 2, 2024-03-11T19:21:58.647 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse": 2, 2024-03-11T19:21:58.647 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_hit": 1, 2024-03-11T19:21:58.647 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_forward": 0, 2024-03-11T19:21:58.647 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_discover": 1, 2024-03-11T19:21:58.647 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_dir_fetch": 0, 2024-03-11T19:21:58.647 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_remote_ino": 0, 2024-03-11T19:21:58.647 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_lock": 0, 2024-03-11T19:21:58.647 INFO:teuthology.orchestra.run.smithi190.stdout: "load_cent": 0, 2024-03-11T19:21:58.647 INFO:teuthology.orchestra.run.smithi190.stdout: "q": 0, 2024-03-11T19:21:58.647 INFO:teuthology.orchestra.run.smithi190.stdout: "exported": 0, 2024-03-11T19:21:58.647 INFO:teuthology.orchestra.run.smithi190.stdout: "exported_inodes": 0, 2024-03-11T19:21:58.647 INFO:teuthology.orchestra.run.smithi190.stdout: "imported": 0, 2024-03-11T19:21:58.648 INFO:teuthology.orchestra.run.smithi190.stdout: "imported_inodes": 0, 2024-03-11T19:21:58.648 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_dir_fetch": 0, 2024-03-11T19:21:58.648 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_backtrace_fetch": 0, 2024-03-11T19:21:58.648 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_peer_discover": 0, 2024-03-11T19:21:58.648 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rfiles": 22, 2024-03-11T19:21:58.648 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rbytes": 7436774654, 2024-03-11T19:21:58.648 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rsnaps": 0, 2024-03-11T19:21:58.648 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T19:21:58.648 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_set_tag": 0, 2024-03-11T19:21:58.648 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T19:21:58.648 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_inotable_repaired": 0, 2024-03-11T19:21:58.648 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_inodes": 0, 2024-03-11T19:21:58.648 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T19:21:58.648 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T19:21:58.648 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_file_inodes": 0, 2024-03-11T19:21:58.649 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_inode_file_caps": 0, 2024-03-11T19:21:58.649 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T19:21:58.649 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_grant": 0, 2024-03-11T19:21:58.649 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T19:21:58.649 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T19:21:58.649 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T19:21:58.649 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps": 0, 2024-03-11T19:21:58.649 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps_dirty": 0, 2024-03-11T19:21:58.649 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_cap_release": 0, 2024-03-11T19:21:58.649 INFO:teuthology.orchestra.run.smithi190.stdout: "process_request_cap_release": 0 2024-03-11T19:21:58.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.649 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_cache": { 2024-03-11T19:21:58.649 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update": 0, 2024-03-11T19:21:58.649 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update_receipt": 0, 2024-03-11T19:21:58.649 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_try_discover": 0, 2024-03-11T19:21:58.649 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_send_discover": 2, 2024-03-11T19:21:58.649 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_handle_discover": 0, 2024-03-11T19:21:58.650 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays": 0, 2024-03-11T19:21:58.650 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_delayed": 0, 2024-03-11T19:21:58.650 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_enqueuing": 0, 2024-03-11T19:21:58.650 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_created": 0, 2024-03-11T19:21:58.650 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_enqueued": 0, 2024-03-11T19:21:58.650 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_reintegrated": 0, 2024-03-11T19:21:58.650 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_migrated": 0, 2024-03-11T19:21:58.650 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_processing": 0, 2024-03-11T19:21:58.650 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_enqueued": 0, 2024-03-11T19:21:58.650 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_prioritized": 0, 2024-03-11T19:21:58.650 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_started": 0, 2024-03-11T19:21:58.650 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_completed": 0, 2024-03-11T19:21:58.650 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T19:21:58.650 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_exportdir": 0, 2024-03-11T19:21:58.651 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_flush": 0, 2024-03-11T19:21:58.651 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragmentdir": 0, 2024-03-11T19:21:58.651 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragstats": 0, 2024-03-11T19:21:58.651 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_inodestats": 0 2024-03-11T19:21:58.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.651 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_log": { 2024-03-11T19:21:58.651 INFO:teuthology.orchestra.run.smithi190.stdout: "evlrg": 0, 2024-03-11T19:21:58.651 INFO:teuthology.orchestra.run.smithi190.stdout: "evadd": 3, 2024-03-11T19:21:58.651 INFO:teuthology.orchestra.run.smithi190.stdout: "evex": 0, 2024-03-11T19:21:58.651 INFO:teuthology.orchestra.run.smithi190.stdout: "evtrm": 0, 2024-03-11T19:21:58.651 INFO:teuthology.orchestra.run.smithi190.stdout: "ev": 3, 2024-03-11T19:21:58.651 INFO:teuthology.orchestra.run.smithi190.stdout: "evexg": 0, 2024-03-11T19:21:58.651 INFO:teuthology.orchestra.run.smithi190.stdout: "evexd": 0, 2024-03-11T19:21:58.651 INFO:teuthology.orchestra.run.smithi190.stdout: "segadd": 2, 2024-03-11T19:21:58.652 INFO:teuthology.orchestra.run.smithi190.stdout: "segex": 0, 2024-03-11T19:21:58.652 INFO:teuthology.orchestra.run.smithi190.stdout: "segtrm": 0, 2024-03-11T19:21:58.652 INFO:teuthology.orchestra.run.smithi190.stdout: "seg": 2, 2024-03-11T19:21:58.652 INFO:teuthology.orchestra.run.smithi190.stdout: "segmjr": 2, 2024-03-11T19:21:58.652 INFO:teuthology.orchestra.run.smithi190.stdout: "segexg": 0, 2024-03-11T19:21:58.652 INFO:teuthology.orchestra.run.smithi190.stdout: "segexd": 0, 2024-03-11T19:21:58.652 INFO:teuthology.orchestra.run.smithi190.stdout: "expos": 4194304, 2024-03-11T19:21:58.652 INFO:teuthology.orchestra.run.smithi190.stdout: "wrpos": 4195385, 2024-03-11T19:21:58.652 INFO:teuthology.orchestra.run.smithi190.stdout: "rdpos": 0, 2024-03-11T19:21:58.652 INFO:teuthology.orchestra.run.smithi190.stdout: "jlat": { 2024-03-11T19:21:58.652 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 2, 2024-03-11T19:21:58.652 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.002990739, 2024-03-11T19:21:58.652 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.001495369 2024-03-11T19:21:58.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.652 INFO:teuthology.orchestra.run.smithi190.stdout: "replayed": 0 2024-03-11T19:21:58.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.653 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_mem": { 2024-03-11T19:21:58.653 INFO:teuthology.orchestra.run.smithi190.stdout: "ino": 14, 2024-03-11T19:21:58.653 INFO:teuthology.orchestra.run.smithi190.stdout: "ino+": 14, 2024-03-11T19:21:58.653 INFO:teuthology.orchestra.run.smithi190.stdout: "ino-": 0, 2024-03-11T19:21:58.653 INFO:teuthology.orchestra.run.smithi190.stdout: "dir": 12, 2024-03-11T19:21:58.653 INFO:teuthology.orchestra.run.smithi190.stdout: "dir+": 13, 2024-03-11T19:21:58.653 INFO:teuthology.orchestra.run.smithi190.stdout: "dir-": 1, 2024-03-11T19:21:58.653 INFO:teuthology.orchestra.run.smithi190.stdout: "dn": 11, 2024-03-11T19:21:58.653 INFO:teuthology.orchestra.run.smithi190.stdout: "dn+": 11, 2024-03-11T19:21:58.653 INFO:teuthology.orchestra.run.smithi190.stdout: "dn-": 0, 2024-03-11T19:21:58.653 INFO:teuthology.orchestra.run.smithi190.stdout: "cap": 1, 2024-03-11T19:21:58.653 INFO:teuthology.orchestra.run.smithi190.stdout: "cap+": 1, 2024-03-11T19:21:58.653 INFO:teuthology.orchestra.run.smithi190.stdout: "cap-": 0, 2024-03-11T19:21:58.653 INFO:teuthology.orchestra.run.smithi190.stdout: "rss": 50668, 2024-03-11T19:21:58.654 INFO:teuthology.orchestra.run.smithi190.stdout: "heap": 207228 2024-03-11T19:21:58.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.654 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_server": { 2024-03-11T19:21:58.654 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_client_request": 2, 2024-03-11T19:21:58.654 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_server_request": 0, 2024-03-11T19:21:58.654 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_request": 1, 2024-03-11T19:21:58.654 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_session": 10, 2024-03-11T19:21:58.654 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_peer_request": 0, 2024-03-11T19:21:58.654 INFO:teuthology.orchestra.run.smithi190.stdout: "req_create_latency": { 2024-03-11T19:21:58.654 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.654 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.654 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.654 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getattr_latency": { 2024-03-11T19:21:58.655 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1, 2024-03-11T19:21:58.655 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000609425, 2024-03-11T19:21:58.655 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000609425 2024-03-11T19:21:58.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.655 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getfilelock_latency": { 2024-03-11T19:21:58.655 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.655 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.655 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.655 INFO:teuthology.orchestra.run.smithi190.stdout: "req_link_latency": { 2024-03-11T19:21:58.655 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.655 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.655 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.655 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookup_latency": { 2024-03-11T19:21:58.655 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.656 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.656 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.656 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookuphash_latency": { 2024-03-11T19:21:58.656 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.656 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.656 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.656 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupino_latency": { 2024-03-11T19:21:58.656 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.656 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.656 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.656 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupname_latency": { 2024-03-11T19:21:58.657 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.657 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.657 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.657 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupparent_latency": { 2024-03-11T19:21:58.657 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.657 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.657 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.657 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupsnap_latency": { 2024-03-11T19:21:58.657 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.657 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.657 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.657 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lssnap_latency": { 2024-03-11T19:21:58.658 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.658 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.658 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.658 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mkdir_latency": { 2024-03-11T19:21:58.658 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.658 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.658 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.659 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mknod_latency": { 2024-03-11T19:21:58.659 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.659 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.659 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.659 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mksnap_latency": { 2024-03-11T19:21:58.659 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.659 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.659 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.659 INFO:teuthology.orchestra.run.smithi190.stdout: "req_open_latency": { 2024-03-11T19:21:58.659 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.659 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.660 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.660 INFO:teuthology.orchestra.run.smithi190.stdout: "req_readdir_latency": { 2024-03-11T19:21:58.660 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.660 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.660 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.660 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rename_latency": { 2024-03-11T19:21:58.660 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.660 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.660 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.660 INFO:teuthology.orchestra.run.smithi190.stdout: "req_renamesnap_latency": { 2024-03-11T19:21:58.660 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.660 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.661 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.661 INFO:teuthology.orchestra.run.smithi190.stdout: "req_snapdiff_latency": { 2024-03-11T19:21:58.661 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.661 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.661 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.661 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmdir_latency": { 2024-03-11T19:21:58.661 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.661 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.661 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.661 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmsnap_latency": { 2024-03-11T19:21:58.661 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.661 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.662 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.662 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmxattr_latency": { 2024-03-11T19:21:58.662 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.662 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.662 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.662 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setattr_latency": { 2024-03-11T19:21:58.662 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.662 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.662 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.662 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setdirlayout_latency": { 2024-03-11T19:21:58.662 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.662 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.663 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.663 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setfilelock_latency": { 2024-03-11T19:21:58.663 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.663 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.663 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.663 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setlayout_latency": { 2024-03-11T19:21:58.663 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.663 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.663 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.663 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setxattr_latency": { 2024-03-11T19:21:58.663 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.663 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.663 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.664 INFO:teuthology.orchestra.run.smithi190.stdout: "req_symlink_latency": { 2024-03-11T19:21:58.664 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.664 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.664 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.664 INFO:teuthology.orchestra.run.smithi190.stdout: "req_unlink_latency": { 2024-03-11T19:21:58.664 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.664 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.664 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.664 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_revoke_eviction": 0, 2024-03-11T19:21:58.664 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_acquisition_throttle": 0, 2024-03-11T19:21:58.664 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getvxattr_latency": { 2024-03-11T19:21:58.664 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.665 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.665 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.665 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:21:58.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.665 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_sessions": { 2024-03-11T19:21:58.665 INFO:teuthology.orchestra.run.smithi190.stdout: "session_count": 1, 2024-03-11T19:21:58.665 INFO:teuthology.orchestra.run.smithi190.stdout: "session_add": 1, 2024-03-11T19:21:58.665 INFO:teuthology.orchestra.run.smithi190.stdout: "session_remove": 0, 2024-03-11T19:21:58.665 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_open": 1, 2024-03-11T19:21:58.665 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_stale": 0, 2024-03-11T19:21:58.665 INFO:teuthology.orchestra.run.smithi190.stdout: "total_load": 1, 2024-03-11T19:21:58.665 INFO:teuthology.orchestra.run.smithi190.stdout: "average_load": 1, 2024-03-11T19:21:58.665 INFO:teuthology.orchestra.run.smithi190.stdout: "avg_session_uptime": 722, 2024-03-11T19:21:58.665 INFO:teuthology.orchestra.run.smithi190.stdout: "mdthresh_evicted": 0 2024-03-11T19:21:58.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.666 INFO:teuthology.orchestra.run.smithi190.stdout: "mempool": { 2024-03-11T19:21:58.666 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_bytes": 0, 2024-03-11T19:21:58.666 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_items": 0, 2024-03-11T19:21:58.666 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T19:21:58.666 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_items": 0, 2024-03-11T19:21:58.666 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T19:21:58.666 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_items": 0, 2024-03-11T19:21:58.666 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T19:21:58.666 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T19:21:58.666 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T19:21:58.666 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T19:21:58.666 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T19:21:58.666 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_items": 0, 2024-03-11T19:21:58.666 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T19:21:58.667 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T19:21:58.667 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_bytes": 0, 2024-03-11T19:21:58.667 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_items": 0, 2024-03-11T19:21:58.667 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_bytes": 0, 2024-03-11T19:21:58.667 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_items": 0, 2024-03-11T19:21:58.667 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T19:21:58.667 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T19:21:58.667 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T19:21:58.667 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T19:21:58.667 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T19:21:58.667 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_items": 0, 2024-03-11T19:21:58.667 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_bytes": 0, 2024-03-11T19:21:58.667 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_items": 0, 2024-03-11T19:21:58.667 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T19:21:58.667 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T19:21:58.668 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_bytes": 0, 2024-03-11T19:21:58.668 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_items": 0, 2024-03-11T19:21:58.668 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_bytes": 0, 2024-03-11T19:21:58.668 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_items": 0, 2024-03-11T19:21:58.668 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T19:21:58.668 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_items": 0, 2024-03-11T19:21:58.668 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T19:21:58.668 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_items": 0, 2024-03-11T19:21:58.668 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_bytes": 230230, 2024-03-11T19:21:58.668 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_items": 73, 2024-03-11T19:21:58.668 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_bytes": 0, 2024-03-11T19:21:58.668 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_items": 0, 2024-03-11T19:21:58.668 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_bytes": 0, 2024-03-11T19:21:58.668 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_items": 0, 2024-03-11T19:21:58.668 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_bytes": 0, 2024-03-11T19:21:58.668 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_items": 0, 2024-03-11T19:21:58.669 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_bytes": 0, 2024-03-11T19:21:58.669 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_items": 0, 2024-03-11T19:21:58.669 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_bytes": 7336, 2024-03-11T19:21:58.669 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_items": 154, 2024-03-11T19:21:58.669 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T19:21:58.669 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_items": 0, 2024-03-11T19:21:58.669 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_bytes": 0, 2024-03-11T19:21:58.669 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_items": 0, 2024-03-11T19:21:58.669 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_bytes": 54552, 2024-03-11T19:21:58.669 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_items": 215, 2024-03-11T19:21:58.669 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_bytes": 0, 2024-03-11T19:21:58.669 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_items": 0, 2024-03-11T19:21:58.669 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_bytes": 0, 2024-03-11T19:21:58.669 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_items": 0 2024-03-11T19:21:58.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.670 INFO:teuthology.orchestra.run.smithi190.stdout: "objecter": { 2024-03-11T19:21:58.670 INFO:teuthology.orchestra.run.smithi190.stdout: "op_active": 0, 2024-03-11T19:21:58.670 INFO:teuthology.orchestra.run.smithi190.stdout: "op_laggy": 0, 2024-03-11T19:21:58.670 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send": 38, 2024-03-11T19:21:58.670 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send_bytes": 11387, 2024-03-11T19:21:58.670 INFO:teuthology.orchestra.run.smithi190.stdout: "op_resend": 33, 2024-03-11T19:21:58.670 INFO:teuthology.orchestra.run.smithi190.stdout: "op_reply": 38, 2024-03-11T19:21:58.670 INFO:teuthology.orchestra.run.smithi190.stdout: "op_latency": { 2024-03-11T19:21:58.670 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 38, 2024-03-11T19:21:58.670 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.222993436, 2024-03-11T19:21:58.670 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.005868248 2024-03-11T19:21:58.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.670 INFO:teuthology.orchestra.run.smithi190.stdout: "op_inflight": 0, 2024-03-11T19:21:58.670 INFO:teuthology.orchestra.run.smithi190.stdout: "oplen_avg": { 2024-03-11T19:21:58.670 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 38, 2024-03-11T19:21:58.671 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 61 2024-03-11T19:21:58.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.671 INFO:teuthology.orchestra.run.smithi190.stdout: "op": 38, 2024-03-11T19:21:58.671 INFO:teuthology.orchestra.run.smithi190.stdout: "op_r": 0, 2024-03-11T19:21:58.671 INFO:teuthology.orchestra.run.smithi190.stdout: "op_w": 38, 2024-03-11T19:21:58.671 INFO:teuthology.orchestra.run.smithi190.stdout: "op_rmw": 0, 2024-03-11T19:21:58.671 INFO:teuthology.orchestra.run.smithi190.stdout: "op_pg": 0, 2024-03-11T19:21:58.671 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_stat": 0, 2024-03-11T19:21:58.671 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_create": 10, 2024-03-11T19:21:58.671 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_read": 0, 2024-03-11T19:21:58.671 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_write": 2, 2024-03-11T19:21:58.671 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writefull": 6, 2024-03-11T19:21:58.671 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writesame": 0, 2024-03-11T19:21:58.671 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_append": 0, 2024-03-11T19:21:58.671 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_zero": 0, 2024-03-11T19:21:58.672 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_truncate": 0, 2024-03-11T19:21:58.672 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_delete": 6, 2024-03-11T19:21:58.672 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_mapext": 0, 2024-03-11T19:21:58.672 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_sparse_read": 0, 2024-03-11T19:21:58.672 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_clonerange": 0, 2024-03-11T19:21:58.672 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_getxattr": 0, 2024-03-11T19:21:58.672 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_setxattr": 20, 2024-03-11T19:21:58.672 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_cmpxattr": 0, 2024-03-11T19:21:58.672 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_rmxattr": 0, 2024-03-11T19:21:58.672 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_resetxattrs": 0, 2024-03-11T19:21:58.672 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_call": 0, 2024-03-11T19:21:58.672 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_watch": 0, 2024-03-11T19:21:58.672 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_notify": 0, 2024-03-11T19:21:58.672 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T19:21:58.672 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls": 0, 2024-03-11T19:21:58.672 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls_filter": 0, 2024-03-11T19:21:58.673 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_other": 0, 2024-03-11T19:21:58.673 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_active": 0, 2024-03-11T19:21:58.673 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_send": 0, 2024-03-11T19:21:58.673 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_resend": 0, 2024-03-11T19:21:58.673 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_ping": 0, 2024-03-11T19:21:58.673 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_active": 0, 2024-03-11T19:21:58.673 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_send": 0, 2024-03-11T19:21:58.673 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_resend": 0, 2024-03-11T19:21:58.673 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_active": 0, 2024-03-11T19:21:58.673 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_send": 0, 2024-03-11T19:21:58.673 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_resend": 0, 2024-03-11T19:21:58.673 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_active": 0, 2024-03-11T19:21:58.673 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_send": 0, 2024-03-11T19:21:58.673 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_resend": 0, 2024-03-11T19:21:58.673 INFO:teuthology.orchestra.run.smithi190.stdout: "command_active": 0, 2024-03-11T19:21:58.674 INFO:teuthology.orchestra.run.smithi190.stdout: "command_send": 0, 2024-03-11T19:21:58.674 INFO:teuthology.orchestra.run.smithi190.stdout: "command_resend": 0, 2024-03-11T19:21:58.674 INFO:teuthology.orchestra.run.smithi190.stdout: "map_epoch": 89, 2024-03-11T19:21:58.674 INFO:teuthology.orchestra.run.smithi190.stdout: "map_full": 0, 2024-03-11T19:21:58.674 INFO:teuthology.orchestra.run.smithi190.stdout: "map_inc": 2, 2024-03-11T19:21:58.674 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_sessions": 8, 2024-03-11T19:21:58.674 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_open": 8, 2024-03-11T19:21:58.674 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_close": 0, 2024-03-11T19:21:58.674 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_laggy": 0, 2024-03-11T19:21:58.674 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_wr": 16, 2024-03-11T19:21:58.674 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_rd": 0, 2024-03-11T19:21:58.674 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_del": 1 2024-03-11T19:21:58.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.674 INFO:teuthology.orchestra.run.smithi190.stdout: "oft": { 2024-03-11T19:21:58.674 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_objs": 1, 2024-03-11T19:21:58.675 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_kv_pairs": 0, 2024-03-11T19:21:58.675 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_updates": 0, 2024-03-11T19:21:58.675 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_removes": 0 2024-03-11T19:21:58.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.675 INFO:teuthology.orchestra.run.smithi190.stdout: "osd-slow-ops": { 2024-03-11T19:21:58.675 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_ops_count": 0 2024-03-11T19:21:58.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.675 INFO:teuthology.orchestra.run.smithi190.stdout: "purge_queue": { 2024-03-11T19:21:58.675 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops": 0, 2024-03-11T19:21:58.675 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T19:21:58.675 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing": 0, 2024-03-11T19:21:58.675 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_high_water": 0, 2024-03-11T19:21:58.675 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed_ops": 0, 2024-03-11T19:21:58.675 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed": 0, 2024-03-11T19:21:58.675 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_item_in_journal": 0 2024-03-11T19:21:58.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.676 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T19:21:58.676 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:21:58.676 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-03-11T19:21:58.676 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:21:58.676 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 674, 2024-03-11T19:21:58.676 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 805785, 2024-03-11T19:21:58.676 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:58.676 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 674, 2024-03-11T19:21:58.676 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-03-11T19:21:58.676 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-03-11T19:21:58.676 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 674, 2024-03-11T19:21:58.676 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 805785, 2024-03-11T19:21:58.676 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:21:58.676 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.676 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.676 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.677 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:21:58.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.677 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_bytes": { 2024-03-11T19:21:58.677 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:21:58.677 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-03-11T19:21:58.677 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:21:58.677 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-03-11T19:21:58.677 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-03-11T19:21:58.677 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:58.677 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:58.677 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 38, 2024-03-11T19:21:58.677 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 11387, 2024-03-11T19:21:58.677 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 38, 2024-03-11T19:21:58.677 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 11387, 2024-03-11T19:21:58.677 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:21:58.678 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.678 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.678 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.678 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:21:58.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.678 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_ops": { 2024-03-11T19:21:58.678 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:21:58.678 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 1024, 2024-03-11T19:21:58.678 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:21:58.678 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-03-11T19:21:58.678 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-03-11T19:21:58.678 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:58.678 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:58.678 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 38, 2024-03-11T19:21:58.678 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 38, 2024-03-11T19:21:58.679 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 38, 2024-03-11T19:21:58.679 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 38, 2024-03-11T19:21:58.679 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:21:58.679 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.679 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.679 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.679 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:21:58.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.679 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle": { 2024-03-11T19:21:58.679 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:21:58.679 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-03-11T19:21:58.679 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:21:58.679 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-03-11T19:21:58.679 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-03-11T19:21:58.679 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:58.679 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-03-11T19:21:58.680 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-03-11T19:21:58.680 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-03-11T19:21:58.680 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 0, 2024-03-11T19:21:58.680 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 0, 2024-03-11T19:21:58.680 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:21:58.680 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.680 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.680 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.680 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:21:58.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:21:58.680 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle-0x55fad17ca6e0": { 2024-03-11T19:21:58.680 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:21:58.680 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-03-11T19:21:58.680 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:21:58.680 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 3, 2024-03-11T19:21:58.681 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 1081, 2024-03-11T19:21:58.681 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:21:58.681 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 3, 2024-03-11T19:21:58.681 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-03-11T19:21:58.681 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-03-11T19:21:58.681 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 2, 2024-03-11T19:21:58.681 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 1081, 2024-03-11T19:21:58.681 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:21:58.681 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:21:58.681 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:21:58.681 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:21:58.681 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:21:58.681 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:21:58.681 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-11T19:21:59.465 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-11T19:21:59.465 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-11T19:21:59.465 DEBUG:tasks.check_counter:Getting stats from f 2024-03-11T19:21:59.465 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-11T19:21:59.617 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:21:59 smithi170 ceph-mon[35690]: pgmap v716: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-11T19:21:59.713 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:21:59.749 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:21:59 smithi190 ceph-mon[36292]: pgmap v716: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-11T19:21:59.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:21:59 smithi029 ceph-mon[32439]: pgmap v716: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-11T19:22:01.474 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-11T19:22:01.474 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T19:22:01.474 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 308, 2024-03-11T19:22:01.474 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 301, 2024-03-11T19:22:01.474 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 787968, 2024-03-11T19:22:01.474 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 232241, 2024-03-11T19:22:01.474 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 16, 2024-03-11T19:22:01.474 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 2, 2024-03-11T19:22:01.475 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.045210033, 2024-03-11T19:22:01.475 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.013799359, 2024-03-11T19:22:01.475 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.025280032, 2024-03-11T19:22:01.475 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.005649015, 2024-03-11T19:22:01.475 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:22:01.475 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 300, 2024-03-11T19:22:01.475 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.021679059, 2024-03-11T19:22:01.475 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000072263 2024-03-11T19:22:01.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.475 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:22:01.475 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 28, 2024-03-11T19:22:01.475 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000005296, 2024-03-11T19:22:01.475 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000189 2024-03-11T19:22:01.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.475 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 782048, 2024-03-11T19:22:01.475 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 146528 2024-03-11T19:22:01.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.476 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T19:22:01.476 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 308, 2024-03-11T19:22:01.476 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 686, 2024-03-11T19:22:01.476 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 66344, 2024-03-11T19:22:01.476 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 604184, 2024-03-11T19:22:01.476 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 7, 2024-03-11T19:22:01.476 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 5, 2024-03-11T19:22:01.476 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.051168138, 2024-03-11T19:22:01.476 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.023711756, 2024-03-11T19:22:01.476 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.019430344, 2024-03-11T19:22:01.476 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.004164087, 2024-03-11T19:22:01.476 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:22:01.476 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 685, 2024-03-11T19:22:01.476 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.048139071, 2024-03-11T19:22:01.476 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000070276 2024-03-11T19:22:01.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.476 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:22:01.477 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 788, 2024-03-11T19:22:01.477 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000184266, 2024-03-11T19:22:01.477 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000233 2024-03-11T19:22:01.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.477 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-11T19:22:01.477 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 388544 2024-03-11T19:22:01.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.477 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T19:22:01.477 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 65, 2024-03-11T19:22:01.477 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 67, 2024-03-11T19:22:01.477 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 13027, 2024-03-11T19:22:01.477 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 359929, 2024-03-11T19:22:01.477 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 43, 2024-03-11T19:22:01.477 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 7, 2024-03-11T19:22:01.477 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.015615210, 2024-03-11T19:22:01.477 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.001424226, 2024-03-11T19:22:01.477 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.012796680, 2024-03-11T19:22:01.478 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.000621652, 2024-03-11T19:22:01.478 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:22:01.478 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 65, 2024-03-11T19:22:01.478 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.017915748, 2024-03-11T19:22:01.478 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000275626 2024-03-11T19:22:01.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.478 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:22:01.478 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 51, 2024-03-11T19:22:01.478 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000006551, 2024-03-11T19:22:01.478 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000128 2024-03-11T19:22:01.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.478 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 2176, 2024-03-11T19:22:01.478 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 3328 2024-03-11T19:22:01.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.479 INFO:teuthology.orchestra.run.smithi190.stdout: "cct": { 2024-03-11T19:22:01.479 INFO:teuthology.orchestra.run.smithi190.stdout: "total_workers": 1, 2024-03-11T19:22:01.479 INFO:teuthology.orchestra.run.smithi190.stdout: "unhealthy_workers": 0 2024-03-11T19:22:01.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.479 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-MDSRank": { 2024-03-11T19:22:01.479 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-03-11T19:22:01.479 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-03-11T19:22:01.479 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 49, 2024-03-11T19:22:01.479 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.002231735, 2024-03-11T19:22:01.479 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000045545 2024-03-11T19:22:01.479 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:01.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.479 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-PurgeQueue": { 2024-03-11T19:22:01.479 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-03-11T19:22:01.479 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-03-11T19:22:01.479 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 2, 2024-03-11T19:22:01.480 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000008476, 2024-03-11T19:22:01.480 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000004238 2024-03-11T19:22:01.480 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:01.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.480 INFO:teuthology.orchestra.run.smithi190.stdout: "mds": { 2024-03-11T19:22:01.480 INFO:teuthology.orchestra.run.smithi190.stdout: "request": 4, 2024-03-11T19:22:01.480 INFO:teuthology.orchestra.run.smithi190.stdout: "reply": 3, 2024-03-11T19:22:01.480 INFO:teuthology.orchestra.run.smithi190.stdout: "reply_latency": { 2024-03-11T19:22:01.480 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 3, 2024-03-11T19:22:01.480 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.001830856, 2024-03-11T19:22:01.480 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000610285 2024-03-11T19:22:01.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.480 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_reply": 0, 2024-03-11T19:22:01.480 INFO:teuthology.orchestra.run.smithi190.stdout: "forward": 1, 2024-03-11T19:22:01.480 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_complete": 0, 2024-03-11T19:22:01.481 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_keys": 0, 2024-03-11T19:22:01.481 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_commit": 11, 2024-03-11T19:22:01.481 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_split": 0, 2024-03-11T19:22:01.481 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_merge": 0, 2024-03-11T19:22:01.481 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes": 12, 2024-03-11T19:22:01.481 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_top": 0, 2024-03-11T19:22:01.481 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_bottom": 12, 2024-03-11T19:22:01.481 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pin_tail": 0, 2024-03-11T19:22:01.481 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pinned": 11, 2024-03-11T19:22:01.481 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_expired": 0, 2024-03-11T19:22:01.481 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_with_caps": 0, 2024-03-11T19:22:01.481 INFO:teuthology.orchestra.run.smithi190.stdout: "caps": 0, 2024-03-11T19:22:01.481 INFO:teuthology.orchestra.run.smithi190.stdout: "subtrees": 2, 2024-03-11T19:22:01.481 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse": 8, 2024-03-11T19:22:01.481 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_hit": 6, 2024-03-11T19:22:01.481 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_forward": 0, 2024-03-11T19:22:01.482 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_discover": 2, 2024-03-11T19:22:01.482 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_dir_fetch": 0, 2024-03-11T19:22:01.482 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_remote_ino": 0, 2024-03-11T19:22:01.482 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_lock": 0, 2024-03-11T19:22:01.482 INFO:teuthology.orchestra.run.smithi190.stdout: "load_cent": 0, 2024-03-11T19:22:01.482 INFO:teuthology.orchestra.run.smithi190.stdout: "q": 0, 2024-03-11T19:22:01.482 INFO:teuthology.orchestra.run.smithi190.stdout: "exported": 0, 2024-03-11T19:22:01.482 INFO:teuthology.orchestra.run.smithi190.stdout: "exported_inodes": 0, 2024-03-11T19:22:01.482 INFO:teuthology.orchestra.run.smithi190.stdout: "imported": 0, 2024-03-11T19:22:01.482 INFO:teuthology.orchestra.run.smithi190.stdout: "imported_inodes": 0, 2024-03-11T19:22:01.482 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_dir_fetch": 0, 2024-03-11T19:22:01.482 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_backtrace_fetch": 0, 2024-03-11T19:22:01.482 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_peer_discover": 0, 2024-03-11T19:22:01.482 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rfiles": 22, 2024-03-11T19:22:01.482 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rbytes": 7436774654, 2024-03-11T19:22:01.483 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rsnaps": 0, 2024-03-11T19:22:01.483 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T19:22:01.483 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_set_tag": 0, 2024-03-11T19:22:01.483 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T19:22:01.483 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_inotable_repaired": 0, 2024-03-11T19:22:01.483 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_inodes": 0, 2024-03-11T19:22:01.483 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T19:22:01.483 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T19:22:01.483 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_file_inodes": 0, 2024-03-11T19:22:01.483 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_inode_file_caps": 0, 2024-03-11T19:22:01.483 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T19:22:01.483 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_grant": 0, 2024-03-11T19:22:01.483 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T19:22:01.483 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T19:22:01.483 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T19:22:01.483 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps": 0, 2024-03-11T19:22:01.483 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps_dirty": 0, 2024-03-11T19:22:01.484 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_cap_release": 0, 2024-03-11T19:22:01.484 INFO:teuthology.orchestra.run.smithi190.stdout: "process_request_cap_release": 0 2024-03-11T19:22:01.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.484 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_cache": { 2024-03-11T19:22:01.484 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update": 0, 2024-03-11T19:22:01.484 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update_receipt": 0, 2024-03-11T19:22:01.484 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_try_discover": 0, 2024-03-11T19:22:01.484 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_send_discover": 3, 2024-03-11T19:22:01.484 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_handle_discover": 0, 2024-03-11T19:22:01.484 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays": 0, 2024-03-11T19:22:01.484 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_delayed": 0, 2024-03-11T19:22:01.484 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_enqueuing": 0, 2024-03-11T19:22:01.484 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_created": 0, 2024-03-11T19:22:01.484 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_enqueued": 0, 2024-03-11T19:22:01.484 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_reintegrated": 0, 2024-03-11T19:22:01.484 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_migrated": 0, 2024-03-11T19:22:01.484 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_processing": 0, 2024-03-11T19:22:01.485 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_enqueued": 0, 2024-03-11T19:22:01.485 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_prioritized": 0, 2024-03-11T19:22:01.485 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_started": 0, 2024-03-11T19:22:01.485 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_completed": 0, 2024-03-11T19:22:01.485 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T19:22:01.485 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_exportdir": 0, 2024-03-11T19:22:01.485 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_flush": 0, 2024-03-11T19:22:01.485 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragmentdir": 0, 2024-03-11T19:22:01.485 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragstats": 0, 2024-03-11T19:22:01.485 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_inodestats": 0 2024-03-11T19:22:01.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.485 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_log": { 2024-03-11T19:22:01.485 INFO:teuthology.orchestra.run.smithi190.stdout: "evlrg": 0, 2024-03-11T19:22:01.485 INFO:teuthology.orchestra.run.smithi190.stdout: "evadd": 5, 2024-03-11T19:22:01.485 INFO:teuthology.orchestra.run.smithi190.stdout: "evex": 0, 2024-03-11T19:22:01.485 INFO:teuthology.orchestra.run.smithi190.stdout: "evtrm": 0, 2024-03-11T19:22:01.485 INFO:teuthology.orchestra.run.smithi190.stdout: "ev": 5, 2024-03-11T19:22:01.486 INFO:teuthology.orchestra.run.smithi190.stdout: "evexg": 0, 2024-03-11T19:22:01.486 INFO:teuthology.orchestra.run.smithi190.stdout: "evexd": 0, 2024-03-11T19:22:01.486 INFO:teuthology.orchestra.run.smithi190.stdout: "segadd": 2, 2024-03-11T19:22:01.486 INFO:teuthology.orchestra.run.smithi190.stdout: "segex": 0, 2024-03-11T19:22:01.486 INFO:teuthology.orchestra.run.smithi190.stdout: "segtrm": 0, 2024-03-11T19:22:01.486 INFO:teuthology.orchestra.run.smithi190.stdout: "seg": 2, 2024-03-11T19:22:01.486 INFO:teuthology.orchestra.run.smithi190.stdout: "segmjr": 2, 2024-03-11T19:22:01.486 INFO:teuthology.orchestra.run.smithi190.stdout: "segexg": 0, 2024-03-11T19:22:01.486 INFO:teuthology.orchestra.run.smithi190.stdout: "segexd": 0, 2024-03-11T19:22:01.486 INFO:teuthology.orchestra.run.smithi190.stdout: "expos": 4194304, 2024-03-11T19:22:01.486 INFO:teuthology.orchestra.run.smithi190.stdout: "wrpos": 4195917, 2024-03-11T19:22:01.486 INFO:teuthology.orchestra.run.smithi190.stdout: "rdpos": 0, 2024-03-11T19:22:01.486 INFO:teuthology.orchestra.run.smithi190.stdout: "jlat": { 2024-03-11T19:22:01.486 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 4, 2024-03-11T19:22:01.486 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.007851246, 2024-03-11T19:22:01.487 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.001962811 2024-03-11T19:22:01.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.487 INFO:teuthology.orchestra.run.smithi190.stdout: "replayed": 0 2024-03-11T19:22:01.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.487 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_mem": { 2024-03-11T19:22:01.487 INFO:teuthology.orchestra.run.smithi190.stdout: "ino": 15, 2024-03-11T19:22:01.487 INFO:teuthology.orchestra.run.smithi190.stdout: "ino+": 15, 2024-03-11T19:22:01.487 INFO:teuthology.orchestra.run.smithi190.stdout: "ino-": 0, 2024-03-11T19:22:01.487 INFO:teuthology.orchestra.run.smithi190.stdout: "dir": 13, 2024-03-11T19:22:01.487 INFO:teuthology.orchestra.run.smithi190.stdout: "dir+": 14, 2024-03-11T19:22:01.487 INFO:teuthology.orchestra.run.smithi190.stdout: "dir-": 1, 2024-03-11T19:22:01.487 INFO:teuthology.orchestra.run.smithi190.stdout: "dn": 12, 2024-03-11T19:22:01.487 INFO:teuthology.orchestra.run.smithi190.stdout: "dn+": 12, 2024-03-11T19:22:01.487 INFO:teuthology.orchestra.run.smithi190.stdout: "dn-": 0, 2024-03-11T19:22:01.487 INFO:teuthology.orchestra.run.smithi190.stdout: "cap": 0, 2024-03-11T19:22:01.487 INFO:teuthology.orchestra.run.smithi190.stdout: "cap+": 3, 2024-03-11T19:22:01.488 INFO:teuthology.orchestra.run.smithi190.stdout: "cap-": 3, 2024-03-11T19:22:01.488 INFO:teuthology.orchestra.run.smithi190.stdout: "rss": 50956, 2024-03-11T19:22:01.488 INFO:teuthology.orchestra.run.smithi190.stdout: "heap": 207228 2024-03-11T19:22:01.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.488 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_server": { 2024-03-11T19:22:01.488 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_client_request": 6, 2024-03-11T19:22:01.488 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_server_request": 0, 2024-03-11T19:22:01.488 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_request": 4, 2024-03-11T19:22:01.488 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_session": 14, 2024-03-11T19:22:01.488 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_peer_request": 0, 2024-03-11T19:22:01.488 INFO:teuthology.orchestra.run.smithi190.stdout: "req_create_latency": { 2024-03-11T19:22:01.488 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.488 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.488 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.488 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getattr_latency": { 2024-03-11T19:22:01.489 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1, 2024-03-11T19:22:01.489 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000274313, 2024-03-11T19:22:01.489 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000274313 2024-03-11T19:22:01.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.489 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getfilelock_latency": { 2024-03-11T19:22:01.489 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.489 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.489 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.489 INFO:teuthology.orchestra.run.smithi190.stdout: "req_link_latency": { 2024-03-11T19:22:01.489 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.489 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.489 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.489 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookup_latency": { 2024-03-11T19:22:01.489 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 2, 2024-03-11T19:22:01.490 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.001556543, 2024-03-11T19:22:01.490 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000778271 2024-03-11T19:22:01.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.490 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookuphash_latency": { 2024-03-11T19:22:01.490 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.490 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.490 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.490 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupino_latency": { 2024-03-11T19:22:01.490 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.490 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.490 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.490 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupname_latency": { 2024-03-11T19:22:01.490 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.491 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.491 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.491 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupparent_latency": { 2024-03-11T19:22:01.491 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.491 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.491 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.491 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupsnap_latency": { 2024-03-11T19:22:01.491 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.491 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.491 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.491 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lssnap_latency": { 2024-03-11T19:22:01.491 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.492 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.492 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.492 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mkdir_latency": { 2024-03-11T19:22:01.492 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.492 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.492 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.492 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mknod_latency": { 2024-03-11T19:22:01.493 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.493 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.493 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.493 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mksnap_latency": { 2024-03-11T19:22:01.493 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.493 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.493 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.493 INFO:teuthology.orchestra.run.smithi190.stdout: "req_open_latency": { 2024-03-11T19:22:01.493 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.493 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.493 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.493 INFO:teuthology.orchestra.run.smithi190.stdout: "req_readdir_latency": { 2024-03-11T19:22:01.493 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.494 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.494 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.494 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rename_latency": { 2024-03-11T19:22:01.494 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.494 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.494 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.494 INFO:teuthology.orchestra.run.smithi190.stdout: "req_renamesnap_latency": { 2024-03-11T19:22:01.494 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.494 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.494 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.494 INFO:teuthology.orchestra.run.smithi190.stdout: "req_snapdiff_latency": { 2024-03-11T19:22:01.495 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.495 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.495 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.495 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmdir_latency": { 2024-03-11T19:22:01.495 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.495 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.495 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.495 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmsnap_latency": { 2024-03-11T19:22:01.495 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.495 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.495 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.495 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmxattr_latency": { 2024-03-11T19:22:01.496 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.496 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.496 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.496 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setattr_latency": { 2024-03-11T19:22:01.496 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.496 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.496 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.496 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setdirlayout_latency": { 2024-03-11T19:22:01.496 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.496 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.496 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.497 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setfilelock_latency": { 2024-03-11T19:22:01.497 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.497 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.497 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.497 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setlayout_latency": { 2024-03-11T19:22:01.497 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.497 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.497 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.497 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setxattr_latency": { 2024-03-11T19:22:01.497 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.497 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.497 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.498 INFO:teuthology.orchestra.run.smithi190.stdout: "req_symlink_latency": { 2024-03-11T19:22:01.498 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.498 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.498 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.498 INFO:teuthology.orchestra.run.smithi190.stdout: "req_unlink_latency": { 2024-03-11T19:22:01.498 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.498 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.498 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.498 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_revoke_eviction": 0, 2024-03-11T19:22:01.498 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_acquisition_throttle": 0, 2024-03-11T19:22:01.498 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getvxattr_latency": { 2024-03-11T19:22:01.498 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.498 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.499 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.499 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:01.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.499 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_sessions": { 2024-03-11T19:22:01.499 INFO:teuthology.orchestra.run.smithi190.stdout: "session_count": 1, 2024-03-11T19:22:01.499 INFO:teuthology.orchestra.run.smithi190.stdout: "session_add": 2, 2024-03-11T19:22:01.499 INFO:teuthology.orchestra.run.smithi190.stdout: "session_remove": 1, 2024-03-11T19:22:01.499 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_open": 1, 2024-03-11T19:22:01.499 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_stale": 0, 2024-03-11T19:22:01.499 INFO:teuthology.orchestra.run.smithi190.stdout: "total_load": 2, 2024-03-11T19:22:01.499 INFO:teuthology.orchestra.run.smithi190.stdout: "average_load": 1, 2024-03-11T19:22:01.499 INFO:teuthology.orchestra.run.smithi190.stdout: "avg_session_uptime": 724, 2024-03-11T19:22:01.499 INFO:teuthology.orchestra.run.smithi190.stdout: "mdthresh_evicted": 0 2024-03-11T19:22:01.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.499 INFO:teuthology.orchestra.run.smithi190.stdout: "mempool": { 2024-03-11T19:22:01.500 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_bytes": 0, 2024-03-11T19:22:01.500 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_items": 0, 2024-03-11T19:22:01.500 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T19:22:01.500 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_items": 0, 2024-03-11T19:22:01.500 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T19:22:01.500 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_items": 0, 2024-03-11T19:22:01.500 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T19:22:01.500 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T19:22:01.500 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T19:22:01.500 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T19:22:01.500 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T19:22:01.500 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_items": 0, 2024-03-11T19:22:01.500 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T19:22:01.500 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T19:22:01.500 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_bytes": 0, 2024-03-11T19:22:01.501 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_items": 0, 2024-03-11T19:22:01.501 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_bytes": 0, 2024-03-11T19:22:01.501 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_items": 0, 2024-03-11T19:22:01.501 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T19:22:01.501 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T19:22:01.501 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T19:22:01.501 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T19:22:01.501 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T19:22:01.501 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_items": 0, 2024-03-11T19:22:01.501 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_bytes": 0, 2024-03-11T19:22:01.501 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_items": 0, 2024-03-11T19:22:01.501 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T19:22:01.501 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T19:22:01.501 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_bytes": 0, 2024-03-11T19:22:01.501 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_items": 0, 2024-03-11T19:22:01.502 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_bytes": 0, 2024-03-11T19:22:01.502 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_items": 0, 2024-03-11T19:22:01.502 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T19:22:01.502 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_items": 0, 2024-03-11T19:22:01.502 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T19:22:01.502 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_items": 0, 2024-03-11T19:22:01.502 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_bytes": 247303, 2024-03-11T19:22:01.502 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_items": 89, 2024-03-11T19:22:01.502 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_bytes": 88, 2024-03-11T19:22:01.502 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_items": 1, 2024-03-11T19:22:01.502 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_bytes": 0, 2024-03-11T19:22:01.502 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_items": 0, 2024-03-11T19:22:01.502 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_bytes": 0, 2024-03-11T19:22:01.502 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_items": 0, 2024-03-11T19:22:01.503 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_bytes": 0, 2024-03-11T19:22:01.503 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_items": 0, 2024-03-11T19:22:01.503 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_bytes": 7336, 2024-03-11T19:22:01.503 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_items": 154, 2024-03-11T19:22:01.503 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T19:22:01.503 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_items": 0, 2024-03-11T19:22:01.503 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_bytes": 0, 2024-03-11T19:22:01.503 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_items": 0, 2024-03-11T19:22:01.503 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_bytes": 58376, 2024-03-11T19:22:01.503 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_items": 234, 2024-03-11T19:22:01.503 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_bytes": 0, 2024-03-11T19:22:01.503 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_items": 0, 2024-03-11T19:22:01.503 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_bytes": 0, 2024-03-11T19:22:01.503 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_items": 0 2024-03-11T19:22:01.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.504 INFO:teuthology.orchestra.run.smithi190.stdout: "objecter": { 2024-03-11T19:22:01.504 INFO:teuthology.orchestra.run.smithi190.stdout: "op_active": 0, 2024-03-11T19:22:01.504 INFO:teuthology.orchestra.run.smithi190.stdout: "op_laggy": 0, 2024-03-11T19:22:01.504 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send": 42, 2024-03-11T19:22:01.504 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send_bytes": 12140, 2024-03-11T19:22:01.504 INFO:teuthology.orchestra.run.smithi190.stdout: "op_resend": 33, 2024-03-11T19:22:01.504 INFO:teuthology.orchestra.run.smithi190.stdout: "op_reply": 42, 2024-03-11T19:22:01.504 INFO:teuthology.orchestra.run.smithi190.stdout: "op_latency": { 2024-03-11T19:22:01.504 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 42, 2024-03-11T19:22:01.504 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.228993239, 2024-03-11T19:22:01.504 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.005452219 2024-03-11T19:22:01.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.504 INFO:teuthology.orchestra.run.smithi190.stdout: "op_inflight": 0, 2024-03-11T19:22:01.504 INFO:teuthology.orchestra.run.smithi190.stdout: "oplen_avg": { 2024-03-11T19:22:01.504 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 42, 2024-03-11T19:22:01.505 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 66 2024-03-11T19:22:01.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.505 INFO:teuthology.orchestra.run.smithi190.stdout: "op": 42, 2024-03-11T19:22:01.505 INFO:teuthology.orchestra.run.smithi190.stdout: "op_r": 0, 2024-03-11T19:22:01.505 INFO:teuthology.orchestra.run.smithi190.stdout: "op_w": 42, 2024-03-11T19:22:01.505 INFO:teuthology.orchestra.run.smithi190.stdout: "op_rmw": 0, 2024-03-11T19:22:01.505 INFO:teuthology.orchestra.run.smithi190.stdout: "op_pg": 0, 2024-03-11T19:22:01.505 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_stat": 0, 2024-03-11T19:22:01.505 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_create": 10, 2024-03-11T19:22:01.505 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_read": 0, 2024-03-11T19:22:01.505 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_write": 4, 2024-03-11T19:22:01.505 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writefull": 7, 2024-03-11T19:22:01.505 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writesame": 0, 2024-03-11T19:22:01.505 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_append": 0, 2024-03-11T19:22:01.505 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_zero": 0, 2024-03-11T19:22:01.505 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_truncate": 0, 2024-03-11T19:22:01.506 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_delete": 6, 2024-03-11T19:22:01.506 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_mapext": 0, 2024-03-11T19:22:01.506 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_sparse_read": 0, 2024-03-11T19:22:01.506 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_clonerange": 0, 2024-03-11T19:22:01.506 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_getxattr": 0, 2024-03-11T19:22:01.506 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_setxattr": 20, 2024-03-11T19:22:01.506 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_cmpxattr": 0, 2024-03-11T19:22:01.506 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_rmxattr": 0, 2024-03-11T19:22:01.506 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_resetxattrs": 0, 2024-03-11T19:22:01.506 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_call": 0, 2024-03-11T19:22:01.506 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_watch": 0, 2024-03-11T19:22:01.506 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_notify": 0, 2024-03-11T19:22:01.506 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T19:22:01.506 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls": 0, 2024-03-11T19:22:01.506 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls_filter": 0, 2024-03-11T19:22:01.507 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_other": 0, 2024-03-11T19:22:01.507 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_active": 0, 2024-03-11T19:22:01.507 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_send": 0, 2024-03-11T19:22:01.507 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_resend": 0, 2024-03-11T19:22:01.507 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_ping": 0, 2024-03-11T19:22:01.507 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_active": 0, 2024-03-11T19:22:01.507 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_send": 0, 2024-03-11T19:22:01.507 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_resend": 0, 2024-03-11T19:22:01.507 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_active": 0, 2024-03-11T19:22:01.507 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_send": 0, 2024-03-11T19:22:01.507 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_resend": 0, 2024-03-11T19:22:01.507 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_active": 0, 2024-03-11T19:22:01.507 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_send": 0, 2024-03-11T19:22:01.507 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_resend": 0, 2024-03-11T19:22:01.507 INFO:teuthology.orchestra.run.smithi190.stdout: "command_active": 0, 2024-03-11T19:22:01.507 INFO:teuthology.orchestra.run.smithi190.stdout: "command_send": 0, 2024-03-11T19:22:01.508 INFO:teuthology.orchestra.run.smithi190.stdout: "command_resend": 0, 2024-03-11T19:22:01.508 INFO:teuthology.orchestra.run.smithi190.stdout: "map_epoch": 89, 2024-03-11T19:22:01.508 INFO:teuthology.orchestra.run.smithi190.stdout: "map_full": 0, 2024-03-11T19:22:01.508 INFO:teuthology.orchestra.run.smithi190.stdout: "map_inc": 2, 2024-03-11T19:22:01.508 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_sessions": 9, 2024-03-11T19:22:01.508 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_open": 9, 2024-03-11T19:22:01.508 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_close": 0, 2024-03-11T19:22:01.508 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_laggy": 0, 2024-03-11T19:22:01.508 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_wr": 17, 2024-03-11T19:22:01.508 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_rd": 0, 2024-03-11T19:22:01.508 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_del": 2 2024-03-11T19:22:01.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.508 INFO:teuthology.orchestra.run.smithi190.stdout: "oft": { 2024-03-11T19:22:01.508 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_objs": 1, 2024-03-11T19:22:01.508 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_kv_pairs": 0, 2024-03-11T19:22:01.508 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_updates": 0, 2024-03-11T19:22:01.508 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_removes": 0 2024-03-11T19:22:01.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.509 INFO:teuthology.orchestra.run.smithi190.stdout: "osd-slow-ops": { 2024-03-11T19:22:01.509 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_ops_count": 0 2024-03-11T19:22:01.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.509 INFO:teuthology.orchestra.run.smithi190.stdout: "purge_queue": { 2024-03-11T19:22:01.509 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops": 0, 2024-03-11T19:22:01.509 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T19:22:01.509 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing": 0, 2024-03-11T19:22:01.509 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_high_water": 0, 2024-03-11T19:22:01.509 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed_ops": 0, 2024-03-11T19:22:01.509 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed": 0, 2024-03-11T19:22:01.509 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_item_in_journal": 0 2024-03-11T19:22:01.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.509 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T19:22:01.509 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:22:01.509 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-03-11T19:22:01.509 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:22:01.510 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 677, 2024-03-11T19:22:01.510 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 788935, 2024-03-11T19:22:01.510 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:22:01.510 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 677, 2024-03-11T19:22:01.510 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-03-11T19:22:01.510 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-03-11T19:22:01.510 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 677, 2024-03-11T19:22:01.510 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 788935, 2024-03-11T19:22:01.510 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:22:01.510 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.510 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.510 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.510 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:01.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.510 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_bytes": { 2024-03-11T19:22:01.511 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:22:01.511 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-03-11T19:22:01.511 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:22:01.511 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-03-11T19:22:01.511 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-03-11T19:22:01.511 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:22:01.511 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-03-11T19:22:01.511 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 42, 2024-03-11T19:22:01.511 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 12140, 2024-03-11T19:22:01.511 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 42, 2024-03-11T19:22:01.511 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 12140, 2024-03-11T19:22:01.511 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:22:01.511 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.511 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.511 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.511 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:01.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.512 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_ops": { 2024-03-11T19:22:01.512 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:22:01.512 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 1024, 2024-03-11T19:22:01.512 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:22:01.512 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-03-11T19:22:01.512 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-03-11T19:22:01.512 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:22:01.512 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-03-11T19:22:01.512 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 42, 2024-03-11T19:22:01.512 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 42, 2024-03-11T19:22:01.512 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 42, 2024-03-11T19:22:01.512 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 42, 2024-03-11T19:22:01.512 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:22:01.512 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.512 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.513 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.513 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:01.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.513 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle": { 2024-03-11T19:22:01.513 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:22:01.513 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-03-11T19:22:01.513 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:22:01.513 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-03-11T19:22:01.513 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-03-11T19:22:01.513 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:22:01.513 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-03-11T19:22:01.513 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-03-11T19:22:01.513 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-03-11T19:22:01.513 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 0, 2024-03-11T19:22:01.513 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 0, 2024-03-11T19:22:01.513 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:22:01.514 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.514 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.514 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.514 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:01.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:01.514 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle-0x55d13f3ba6e0": { 2024-03-11T19:22:01.514 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:22:01.514 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-03-11T19:22:01.514 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:22:01.514 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 5, 2024-03-11T19:22:01.514 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 1613, 2024-03-11T19:22:01.514 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:22:01.514 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 5, 2024-03-11T19:22:01.514 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-03-11T19:22:01.514 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-03-11T19:22:01.514 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 4, 2024-03-11T19:22:01.515 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 1613, 2024-03-11T19:22:01.515 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:22:01.515 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:01.515 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:01.515 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:01.515 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:01.515 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:01.515 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-11T19:22:01.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:01 smithi190 ceph-mon[36292]: pgmap v717: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-11T19:22:01.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:01 smithi170 ceph-mon[35690]: pgmap v717: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-11T19:22:01.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:01 smithi029 ceph-mon[32439]: pgmap v717: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-11T19:22:02.252 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-11T19:22:02.252 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-11T19:22:02.252 DEBUG:tasks.check_counter:Getting stats from i 2024-03-11T19:22:02.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-11T19:22:02.494 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:22:03.641 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:03 smithi190 ceph-mon[36292]: pgmap v718: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:22:03.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:03 smithi170 ceph-mon[35690]: pgmap v718: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:22:03.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:03 smithi029 ceph-mon[32439]: pgmap v718: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:22:04.261 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-11T19:22:04.261 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T19:22:04.262 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 89, 2024-03-11T19:22:04.262 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 91, 2024-03-11T19:22:04.262 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 19127, 2024-03-11T19:22:04.262 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 440230, 2024-03-11T19:22:04.262 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 54, 2024-03-11T19:22:04.262 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 7, 2024-03-11T19:22:04.262 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.024062291, 2024-03-11T19:22:04.262 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.002157943, 2024-03-11T19:22:04.262 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.017321896, 2024-03-11T19:22:04.262 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.000536088, 2024-03-11T19:22:04.262 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:22:04.262 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 88, 2024-03-11T19:22:04.262 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.029247234, 2024-03-11T19:22:04.262 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000332354 2024-03-11T19:22:04.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.263 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:22:04.263 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 81, 2024-03-11T19:22:04.263 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000013245, 2024-03-11T19:22:04.263 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000163 2024-03-11T19:22:04.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.263 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 4928, 2024-03-11T19:22:04.263 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 2032 2024-03-11T19:22:04.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.263 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T19:22:04.263 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 329, 2024-03-11T19:22:04.263 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 698, 2024-03-11T19:22:04.263 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 77072, 2024-03-11T19:22:04.263 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 613548, 2024-03-11T19:22:04.263 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 7, 2024-03-11T19:22:04.263 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 4, 2024-03-11T19:22:04.264 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.054256485, 2024-03-11T19:22:04.264 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.023910477, 2024-03-11T19:22:04.264 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.022077079, 2024-03-11T19:22:04.264 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.005040889, 2024-03-11T19:22:04.264 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:22:04.264 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 698, 2024-03-11T19:22:04.264 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.036179898, 2024-03-11T19:22:04.264 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000051833 2024-03-11T19:22:04.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.264 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:22:04.264 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 824, 2024-03-11T19:22:04.264 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000218159, 2024-03-11T19:22:04.264 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000264 2024-03-11T19:22:04.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.265 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-11T19:22:04.265 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 389520 2024-03-11T19:22:04.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.265 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T19:22:04.265 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 286, 2024-03-11T19:22:04.265 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 273, 2024-03-11T19:22:04.265 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 797911, 2024-03-11T19:22:04.265 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 153991, 2024-03-11T19:22:04.265 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 6, 2024-03-11T19:22:04.265 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 5, 2024-03-11T19:22:04.265 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.036875248, 2024-03-11T19:22:04.265 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.012673428, 2024-03-11T19:22:04.265 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.021463227, 2024-03-11T19:22:04.265 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.005425027, 2024-03-11T19:22:04.265 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:22:04.266 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 272, 2024-03-11T19:22:04.266 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.021840681, 2024-03-11T19:22:04.266 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000080296 2024-03-11T19:22:04.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.266 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:22:04.266 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.266 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.266 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.266 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 795392, 2024-03-11T19:22:04.266 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 144480 2024-03-11T19:22:04.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.266 INFO:teuthology.orchestra.run.smithi190.stdout: "cct": { 2024-03-11T19:22:04.266 INFO:teuthology.orchestra.run.smithi190.stdout: "total_workers": 1, 2024-03-11T19:22:04.267 INFO:teuthology.orchestra.run.smithi190.stdout: "unhealthy_workers": 0 2024-03-11T19:22:04.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.267 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-MDSRank": { 2024-03-11T19:22:04.267 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-03-11T19:22:04.267 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-03-11T19:22:04.267 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 52, 2024-03-11T19:22:04.267 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.001742543, 2024-03-11T19:22:04.267 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000033510 2024-03-11T19:22:04.267 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:04.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.267 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-PurgeQueue": { 2024-03-11T19:22:04.267 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-03-11T19:22:04.267 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-03-11T19:22:04.267 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 2, 2024-03-11T19:22:04.267 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000009040, 2024-03-11T19:22:04.268 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000004520 2024-03-11T19:22:04.268 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:04.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.268 INFO:teuthology.orchestra.run.smithi190.stdout: "mds": { 2024-03-11T19:22:04.268 INFO:teuthology.orchestra.run.smithi190.stdout: "request": 5, 2024-03-11T19:22:04.268 INFO:teuthology.orchestra.run.smithi190.stdout: "reply": 3, 2024-03-11T19:22:04.268 INFO:teuthology.orchestra.run.smithi190.stdout: "reply_latency": { 2024-03-11T19:22:04.268 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 3, 2024-03-11T19:22:04.268 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.002316908, 2024-03-11T19:22:04.268 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000772302 2024-03-11T19:22:04.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.268 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_reply": 0, 2024-03-11T19:22:04.268 INFO:teuthology.orchestra.run.smithi190.stdout: "forward": 2, 2024-03-11T19:22:04.268 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_complete": 0, 2024-03-11T19:22:04.268 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_keys": 0, 2024-03-11T19:22:04.269 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_commit": 11, 2024-03-11T19:22:04.269 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_split": 0, 2024-03-11T19:22:04.269 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_merge": 0, 2024-03-11T19:22:04.269 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes": 12, 2024-03-11T19:22:04.269 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_top": 0, 2024-03-11T19:22:04.269 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_bottom": 12, 2024-03-11T19:22:04.269 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pin_tail": 0, 2024-03-11T19:22:04.269 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pinned": 12, 2024-03-11T19:22:04.269 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_expired": 0, 2024-03-11T19:22:04.269 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_with_caps": 2, 2024-03-11T19:22:04.269 INFO:teuthology.orchestra.run.smithi190.stdout: "caps": 2, 2024-03-11T19:22:04.269 INFO:teuthology.orchestra.run.smithi190.stdout: "subtrees": 2, 2024-03-11T19:22:04.269 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse": 9, 2024-03-11T19:22:04.269 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_hit": 5, 2024-03-11T19:22:04.270 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_forward": 0, 2024-03-11T19:22:04.270 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_discover": 2, 2024-03-11T19:22:04.270 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_dir_fetch": 0, 2024-03-11T19:22:04.270 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_remote_ino": 0, 2024-03-11T19:22:04.270 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_lock": 0, 2024-03-11T19:22:04.270 INFO:teuthology.orchestra.run.smithi190.stdout: "load_cent": 0, 2024-03-11T19:22:04.270 INFO:teuthology.orchestra.run.smithi190.stdout: "q": 0, 2024-03-11T19:22:04.270 INFO:teuthology.orchestra.run.smithi190.stdout: "exported": 0, 2024-03-11T19:22:04.270 INFO:teuthology.orchestra.run.smithi190.stdout: "exported_inodes": 0, 2024-03-11T19:22:04.270 INFO:teuthology.orchestra.run.smithi190.stdout: "imported": 0, 2024-03-11T19:22:04.270 INFO:teuthology.orchestra.run.smithi190.stdout: "imported_inodes": 0, 2024-03-11T19:22:04.270 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_dir_fetch": 0, 2024-03-11T19:22:04.270 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_backtrace_fetch": 0, 2024-03-11T19:22:04.270 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_peer_discover": 0, 2024-03-11T19:22:04.270 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rfiles": 22, 2024-03-11T19:22:04.271 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rbytes": 7436774654, 2024-03-11T19:22:04.271 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rsnaps": 0, 2024-03-11T19:22:04.271 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T19:22:04.271 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_set_tag": 0, 2024-03-11T19:22:04.271 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T19:22:04.271 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_inotable_repaired": 0, 2024-03-11T19:22:04.271 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_inodes": 0, 2024-03-11T19:22:04.271 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T19:22:04.271 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T19:22:04.271 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_file_inodes": 0, 2024-03-11T19:22:04.271 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_inode_file_caps": 0, 2024-03-11T19:22:04.271 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T19:22:04.271 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_grant": 0, 2024-03-11T19:22:04.271 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T19:22:04.271 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T19:22:04.272 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T19:22:04.272 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps": 0, 2024-03-11T19:22:04.272 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps_dirty": 0, 2024-03-11T19:22:04.272 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_cap_release": 0, 2024-03-11T19:22:04.272 INFO:teuthology.orchestra.run.smithi190.stdout: "process_request_cap_release": 2 2024-03-11T19:22:04.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.272 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_cache": { 2024-03-11T19:22:04.272 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update": 0, 2024-03-11T19:22:04.272 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update_receipt": 0, 2024-03-11T19:22:04.272 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_try_discover": 0, 2024-03-11T19:22:04.272 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_send_discover": 3, 2024-03-11T19:22:04.272 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_handle_discover": 0, 2024-03-11T19:22:04.272 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays": 0, 2024-03-11T19:22:04.272 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_delayed": 0, 2024-03-11T19:22:04.272 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_enqueuing": 0, 2024-03-11T19:22:04.273 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_created": 0, 2024-03-11T19:22:04.273 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_enqueued": 0, 2024-03-11T19:22:04.273 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_reintegrated": 0, 2024-03-11T19:22:04.273 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_migrated": 0, 2024-03-11T19:22:04.273 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_processing": 0, 2024-03-11T19:22:04.273 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_enqueued": 0, 2024-03-11T19:22:04.273 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_prioritized": 0, 2024-03-11T19:22:04.273 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_started": 0, 2024-03-11T19:22:04.273 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_completed": 0, 2024-03-11T19:22:04.273 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T19:22:04.273 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_exportdir": 0, 2024-03-11T19:22:04.273 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_flush": 0, 2024-03-11T19:22:04.273 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragmentdir": 0, 2024-03-11T19:22:04.273 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragstats": 0, 2024-03-11T19:22:04.273 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_inodestats": 0 2024-03-11T19:22:04.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.274 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_log": { 2024-03-11T19:22:04.274 INFO:teuthology.orchestra.run.smithi190.stdout: "evlrg": 0, 2024-03-11T19:22:04.274 INFO:teuthology.orchestra.run.smithi190.stdout: "evadd": 5, 2024-03-11T19:22:04.274 INFO:teuthology.orchestra.run.smithi190.stdout: "evex": 0, 2024-03-11T19:22:04.274 INFO:teuthology.orchestra.run.smithi190.stdout: "evtrm": 0, 2024-03-11T19:22:04.274 INFO:teuthology.orchestra.run.smithi190.stdout: "ev": 5, 2024-03-11T19:22:04.274 INFO:teuthology.orchestra.run.smithi190.stdout: "evexg": 0, 2024-03-11T19:22:04.274 INFO:teuthology.orchestra.run.smithi190.stdout: "evexd": 0, 2024-03-11T19:22:04.274 INFO:teuthology.orchestra.run.smithi190.stdout: "segadd": 2, 2024-03-11T19:22:04.274 INFO:teuthology.orchestra.run.smithi190.stdout: "segex": 0, 2024-03-11T19:22:04.274 INFO:teuthology.orchestra.run.smithi190.stdout: "segtrm": 0, 2024-03-11T19:22:04.274 INFO:teuthology.orchestra.run.smithi190.stdout: "seg": 2, 2024-03-11T19:22:04.274 INFO:teuthology.orchestra.run.smithi190.stdout: "segmjr": 2, 2024-03-11T19:22:04.274 INFO:teuthology.orchestra.run.smithi190.stdout: "segexg": 0, 2024-03-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi190.stdout: "segexd": 0, 2024-03-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi190.stdout: "expos": 4194304, 2024-03-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi190.stdout: "wrpos": 4195917, 2024-03-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi190.stdout: "rdpos": 0, 2024-03-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi190.stdout: "jlat": { 2024-03-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 4, 2024-03-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.006758820, 2024-03-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.001689705 2024-03-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi190.stdout: "replayed": 0 2024-03-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_mem": { 2024-03-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi190.stdout: "ino": 15, 2024-03-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi190.stdout: "ino+": 15, 2024-03-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi190.stdout: "ino-": 0, 2024-03-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi190.stdout: "dir": 13, 2024-03-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi190.stdout: "dir+": 14, 2024-03-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi190.stdout: "dir-": 1, 2024-03-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi190.stdout: "dn": 12, 2024-03-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi190.stdout: "dn+": 12, 2024-03-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi190.stdout: "dn-": 0, 2024-03-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi190.stdout: "cap": 2, 2024-03-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi190.stdout: "cap+": 3, 2024-03-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi190.stdout: "cap-": 1, 2024-03-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi190.stdout: "rss": 51044, 2024-03-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi190.stdout: "heap": 207228 2024-03-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_server": { 2024-03-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_client_request": 10, 2024-03-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_server_request": 0, 2024-03-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_request": 5, 2024-03-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_session": 14, 2024-03-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_peer_request": 1, 2024-03-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi190.stdout: "req_create_latency": { 2024-03-11T19:22:04.277 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.277 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.277 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.277 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getattr_latency": { 2024-03-11T19:22:04.277 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 3, 2024-03-11T19:22:04.277 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.002316908, 2024-03-11T19:22:04.277 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000772302 2024-03-11T19:22:04.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.277 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getfilelock_latency": { 2024-03-11T19:22:04.277 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.277 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.277 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.277 INFO:teuthology.orchestra.run.smithi190.stdout: "req_link_latency": { 2024-03-11T19:22:04.277 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.277 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.278 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.278 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookup_latency": { 2024-03-11T19:22:04.278 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.278 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.278 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.278 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookuphash_latency": { 2024-03-11T19:22:04.278 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.278 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.278 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.278 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupino_latency": { 2024-03-11T19:22:04.278 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.278 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.278 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.279 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupname_latency": { 2024-03-11T19:22:04.279 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.279 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.279 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.279 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupparent_latency": { 2024-03-11T19:22:04.279 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.279 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.279 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.279 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupsnap_latency": { 2024-03-11T19:22:04.279 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.279 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.279 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.280 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lssnap_latency": { 2024-03-11T19:22:04.280 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.280 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.280 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.280 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mkdir_latency": { 2024-03-11T19:22:04.280 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.280 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.280 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.281 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mknod_latency": { 2024-03-11T19:22:04.281 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.281 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.281 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.281 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mksnap_latency": { 2024-03-11T19:22:04.281 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.281 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.281 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.281 INFO:teuthology.orchestra.run.smithi190.stdout: "req_open_latency": { 2024-03-11T19:22:04.281 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.282 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.282 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.282 INFO:teuthology.orchestra.run.smithi190.stdout: "req_readdir_latency": { 2024-03-11T19:22:04.282 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.282 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.282 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.282 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rename_latency": { 2024-03-11T19:22:04.282 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.282 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.282 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.282 INFO:teuthology.orchestra.run.smithi190.stdout: "req_renamesnap_latency": { 2024-03-11T19:22:04.283 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.283 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.283 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.283 INFO:teuthology.orchestra.run.smithi190.stdout: "req_snapdiff_latency": { 2024-03-11T19:22:04.283 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.283 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.283 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.283 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmdir_latency": { 2024-03-11T19:22:04.283 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.283 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.283 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.284 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmsnap_latency": { 2024-03-11T19:22:04.284 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.284 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.284 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.284 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmxattr_latency": { 2024-03-11T19:22:04.284 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.284 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.284 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.284 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setattr_latency": { 2024-03-11T19:22:04.284 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.284 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.284 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.285 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setdirlayout_latency": { 2024-03-11T19:22:04.285 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.285 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.285 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.285 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setfilelock_latency": { 2024-03-11T19:22:04.285 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.285 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.285 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.285 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setlayout_latency": { 2024-03-11T19:22:04.285 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.285 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.285 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.286 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setxattr_latency": { 2024-03-11T19:22:04.286 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.286 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.286 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.286 INFO:teuthology.orchestra.run.smithi190.stdout: "req_symlink_latency": { 2024-03-11T19:22:04.286 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.286 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.286 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.286 INFO:teuthology.orchestra.run.smithi190.stdout: "req_unlink_latency": { 2024-03-11T19:22:04.286 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.286 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.287 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.287 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_revoke_eviction": 0, 2024-03-11T19:22:04.287 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_acquisition_throttle": 0, 2024-03-11T19:22:04.287 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getvxattr_latency": { 2024-03-11T19:22:04.287 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.287 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.287 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.287 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:04.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.287 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_sessions": { 2024-03-11T19:22:04.287 INFO:teuthology.orchestra.run.smithi190.stdout: "session_count": 1, 2024-03-11T19:22:04.287 INFO:teuthology.orchestra.run.smithi190.stdout: "session_add": 2, 2024-03-11T19:22:04.287 INFO:teuthology.orchestra.run.smithi190.stdout: "session_remove": 1, 2024-03-11T19:22:04.288 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_open": 1, 2024-03-11T19:22:04.288 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_stale": 0, 2024-03-11T19:22:04.288 INFO:teuthology.orchestra.run.smithi190.stdout: "total_load": 2, 2024-03-11T19:22:04.288 INFO:teuthology.orchestra.run.smithi190.stdout: "average_load": 2, 2024-03-11T19:22:04.288 INFO:teuthology.orchestra.run.smithi190.stdout: "avg_session_uptime": 728, 2024-03-11T19:22:04.288 INFO:teuthology.orchestra.run.smithi190.stdout: "mdthresh_evicted": 0 2024-03-11T19:22:04.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.288 INFO:teuthology.orchestra.run.smithi190.stdout: "mempool": { 2024-03-11T19:22:04.288 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_bytes": 0, 2024-03-11T19:22:04.288 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_items": 0, 2024-03-11T19:22:04.288 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T19:22:04.288 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_items": 0, 2024-03-11T19:22:04.288 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T19:22:04.288 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_items": 0, 2024-03-11T19:22:04.288 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T19:22:04.288 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T19:22:04.289 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T19:22:04.289 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T19:22:04.289 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T19:22:04.289 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_items": 0, 2024-03-11T19:22:04.289 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T19:22:04.289 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T19:22:04.289 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_bytes": 0, 2024-03-11T19:22:04.289 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_items": 0, 2024-03-11T19:22:04.289 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_bytes": 0, 2024-03-11T19:22:04.289 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_items": 0, 2024-03-11T19:22:04.289 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T19:22:04.289 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T19:22:04.289 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T19:22:04.289 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T19:22:04.290 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T19:22:04.290 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_items": 0, 2024-03-11T19:22:04.290 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_bytes": 0, 2024-03-11T19:22:04.290 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_items": 0, 2024-03-11T19:22:04.290 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T19:22:04.290 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T19:22:04.290 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_bytes": 0, 2024-03-11T19:22:04.290 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_items": 0, 2024-03-11T19:22:04.290 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_bytes": 0, 2024-03-11T19:22:04.290 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_items": 0, 2024-03-11T19:22:04.290 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T19:22:04.290 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_items": 0, 2024-03-11T19:22:04.290 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T19:22:04.290 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_items": 0, 2024-03-11T19:22:04.290 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_bytes": 247553, 2024-03-11T19:22:04.291 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_items": 89, 2024-03-11T19:22:04.291 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_bytes": 0, 2024-03-11T19:22:04.291 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_items": 0, 2024-03-11T19:22:04.291 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_bytes": 0, 2024-03-11T19:22:04.291 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_items": 0, 2024-03-11T19:22:04.291 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_bytes": 0, 2024-03-11T19:22:04.291 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_items": 0, 2024-03-11T19:22:04.291 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_bytes": 0, 2024-03-11T19:22:04.291 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_items": 0, 2024-03-11T19:22:04.291 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_bytes": 7336, 2024-03-11T19:22:04.291 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_items": 154, 2024-03-11T19:22:04.291 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T19:22:04.291 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_items": 0, 2024-03-11T19:22:04.291 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_bytes": 0, 2024-03-11T19:22:04.291 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_items": 0, 2024-03-11T19:22:04.292 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_bytes": 59088, 2024-03-11T19:22:04.292 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_items": 235, 2024-03-11T19:22:04.292 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_bytes": 0, 2024-03-11T19:22:04.292 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_items": 0, 2024-03-11T19:22:04.292 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_bytes": 0, 2024-03-11T19:22:04.292 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_items": 0 2024-03-11T19:22:04.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.292 INFO:teuthology.orchestra.run.smithi190.stdout: "objecter": { 2024-03-11T19:22:04.292 INFO:teuthology.orchestra.run.smithi190.stdout: "op_active": 0, 2024-03-11T19:22:04.292 INFO:teuthology.orchestra.run.smithi190.stdout: "op_laggy": 0, 2024-03-11T19:22:04.292 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send": 41, 2024-03-11T19:22:04.292 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send_bytes": 12070, 2024-03-11T19:22:04.292 INFO:teuthology.orchestra.run.smithi190.stdout: "op_resend": 28, 2024-03-11T19:22:04.292 INFO:teuthology.orchestra.run.smithi190.stdout: "op_reply": 41, 2024-03-11T19:22:04.292 INFO:teuthology.orchestra.run.smithi190.stdout: "op_latency": { 2024-03-11T19:22:04.293 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 41, 2024-03-11T19:22:04.293 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.231993147, 2024-03-11T19:22:04.293 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.005658369 2024-03-11T19:22:04.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.293 INFO:teuthology.orchestra.run.smithi190.stdout: "op_inflight": 0, 2024-03-11T19:22:04.293 INFO:teuthology.orchestra.run.smithi190.stdout: "oplen_avg": { 2024-03-11T19:22:04.293 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 41, 2024-03-11T19:22:04.293 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 64 2024-03-11T19:22:04.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.293 INFO:teuthology.orchestra.run.smithi190.stdout: "op": 41, 2024-03-11T19:22:04.293 INFO:teuthology.orchestra.run.smithi190.stdout: "op_r": 0, 2024-03-11T19:22:04.293 INFO:teuthology.orchestra.run.smithi190.stdout: "op_w": 41, 2024-03-11T19:22:04.293 INFO:teuthology.orchestra.run.smithi190.stdout: "op_rmw": 0, 2024-03-11T19:22:04.293 INFO:teuthology.orchestra.run.smithi190.stdout: "op_pg": 0, 2024-03-11T19:22:04.294 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_stat": 0, 2024-03-11T19:22:04.294 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_create": 10, 2024-03-11T19:22:04.294 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_read": 0, 2024-03-11T19:22:04.294 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_write": 4, 2024-03-11T19:22:04.294 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writefull": 7, 2024-03-11T19:22:04.294 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writesame": 0, 2024-03-11T19:22:04.294 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_append": 0, 2024-03-11T19:22:04.294 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_zero": 0, 2024-03-11T19:22:04.294 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_truncate": 0, 2024-03-11T19:22:04.294 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_delete": 6, 2024-03-11T19:22:04.294 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_mapext": 0, 2024-03-11T19:22:04.294 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_sparse_read": 0, 2024-03-11T19:22:04.294 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_clonerange": 0, 2024-03-11T19:22:04.294 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_getxattr": 0, 2024-03-11T19:22:04.295 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_setxattr": 20, 2024-03-11T19:22:04.295 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_cmpxattr": 0, 2024-03-11T19:22:04.295 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_rmxattr": 0, 2024-03-11T19:22:04.295 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_resetxattrs": 0, 2024-03-11T19:22:04.295 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_call": 0, 2024-03-11T19:22:04.295 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_watch": 0, 2024-03-11T19:22:04.295 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_notify": 0, 2024-03-11T19:22:04.295 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T19:22:04.295 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls": 0, 2024-03-11T19:22:04.295 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls_filter": 0, 2024-03-11T19:22:04.295 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_other": 0, 2024-03-11T19:22:04.295 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_active": 0, 2024-03-11T19:22:04.295 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_send": 0, 2024-03-11T19:22:04.295 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_resend": 0, 2024-03-11T19:22:04.295 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_ping": 0, 2024-03-11T19:22:04.296 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_active": 0, 2024-03-11T19:22:04.296 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_send": 0, 2024-03-11T19:22:04.296 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_resend": 0, 2024-03-11T19:22:04.296 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_active": 0, 2024-03-11T19:22:04.296 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_send": 0, 2024-03-11T19:22:04.296 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_resend": 0, 2024-03-11T19:22:04.296 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_active": 0, 2024-03-11T19:22:04.296 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_send": 0, 2024-03-11T19:22:04.296 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_resend": 0, 2024-03-11T19:22:04.296 INFO:teuthology.orchestra.run.smithi190.stdout: "command_active": 0, 2024-03-11T19:22:04.296 INFO:teuthology.orchestra.run.smithi190.stdout: "command_send": 0, 2024-03-11T19:22:04.296 INFO:teuthology.orchestra.run.smithi190.stdout: "command_resend": 0, 2024-03-11T19:22:04.296 INFO:teuthology.orchestra.run.smithi190.stdout: "map_epoch": 89, 2024-03-11T19:22:04.296 INFO:teuthology.orchestra.run.smithi190.stdout: "map_full": 0, 2024-03-11T19:22:04.296 INFO:teuthology.orchestra.run.smithi190.stdout: "map_inc": 2, 2024-03-11T19:22:04.297 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_sessions": 10, 2024-03-11T19:22:04.297 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_open": 10, 2024-03-11T19:22:04.297 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_close": 0, 2024-03-11T19:22:04.297 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_laggy": 0, 2024-03-11T19:22:04.297 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_wr": 16, 2024-03-11T19:22:04.297 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_rd": 0, 2024-03-11T19:22:04.297 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_del": 1 2024-03-11T19:22:04.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.297 INFO:teuthology.orchestra.run.smithi190.stdout: "oft": { 2024-03-11T19:22:04.297 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_objs": 1, 2024-03-11T19:22:04.297 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_kv_pairs": 0, 2024-03-11T19:22:04.297 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_updates": 0, 2024-03-11T19:22:04.297 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_removes": 0 2024-03-11T19:22:04.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.297 INFO:teuthology.orchestra.run.smithi190.stdout: "osd-slow-ops": { 2024-03-11T19:22:04.298 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_ops_count": 0 2024-03-11T19:22:04.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.298 INFO:teuthology.orchestra.run.smithi190.stdout: "purge_queue": { 2024-03-11T19:22:04.298 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops": 0, 2024-03-11T19:22:04.298 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T19:22:04.298 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing": 0, 2024-03-11T19:22:04.298 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_high_water": 0, 2024-03-11T19:22:04.298 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed_ops": 0, 2024-03-11T19:22:04.298 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed": 0, 2024-03-11T19:22:04.298 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_item_in_journal": 0 2024-03-11T19:22:04.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.298 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T19:22:04.298 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:22:04.298 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-03-11T19:22:04.298 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:22:04.299 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 700, 2024-03-11T19:22:04.299 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 813735, 2024-03-11T19:22:04.299 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:22:04.299 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 700, 2024-03-11T19:22:04.299 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-03-11T19:22:04.299 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-03-11T19:22:04.299 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 700, 2024-03-11T19:22:04.299 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 813735, 2024-03-11T19:22:04.299 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:22:04.299 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.299 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.299 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.299 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:04.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.299 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_bytes": { 2024-03-11T19:22:04.299 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:22:04.300 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-03-11T19:22:04.300 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:22:04.300 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-03-11T19:22:04.300 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-03-11T19:22:04.300 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:22:04.300 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-03-11T19:22:04.300 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 41, 2024-03-11T19:22:04.300 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 12070, 2024-03-11T19:22:04.300 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 41, 2024-03-11T19:22:04.300 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 12070, 2024-03-11T19:22:04.300 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:22:04.300 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.300 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.300 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.300 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:04.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.301 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_ops": { 2024-03-11T19:22:04.301 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:22:04.301 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 1024, 2024-03-11T19:22:04.301 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:22:04.301 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-03-11T19:22:04.301 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-03-11T19:22:04.301 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:22:04.301 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-03-11T19:22:04.301 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 41, 2024-03-11T19:22:04.301 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 41, 2024-03-11T19:22:04.301 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 41, 2024-03-11T19:22:04.301 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 41, 2024-03-11T19:22:04.301 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:22:04.301 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.301 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.301 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.302 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:04.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.302 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle": { 2024-03-11T19:22:04.302 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:22:04.302 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-03-11T19:22:04.302 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:22:04.302 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-03-11T19:22:04.302 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-03-11T19:22:04.302 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:22:04.302 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-03-11T19:22:04.302 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-03-11T19:22:04.302 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-03-11T19:22:04.302 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 0, 2024-03-11T19:22:04.302 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 0, 2024-03-11T19:22:04.302 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:22:04.303 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.303 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.303 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.303 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:04.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:04.303 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle-0x5644100086e0": { 2024-03-11T19:22:04.303 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:22:04.303 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-03-11T19:22:04.303 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:22:04.303 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 5, 2024-03-11T19:22:04.303 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 1613, 2024-03-11T19:22:04.303 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:22:04.303 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 5, 2024-03-11T19:22:04.303 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-03-11T19:22:04.303 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-03-11T19:22:04.304 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 4, 2024-03-11T19:22:04.304 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 1613, 2024-03-11T19:22:04.304 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:22:04.304 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:04.304 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:04.304 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:04.304 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:04.304 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:04.304 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-11T19:22:05.113 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-11T19:22:05.113 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-11T19:22:05.114 DEBUG:tasks.check_counter:Getting stats from l 2024-03-11T19:22:05.114 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-11T19:22:05.355 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:22:05.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:05 smithi190 ceph-mon[36292]: pgmap v719: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:22:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:05 smithi170 ceph-mon[35690]: pgmap v719: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:22:05.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:05 smithi029 ceph-mon[32439]: pgmap v719: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-11T19:22:07.136 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-11T19:22:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T19:22:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 2482, 2024-03-11T19:22:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 2482, 2024-03-11T19:22:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 704916, 2024-03-11T19:22:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 771113, 2024-03-11T19:22:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 6, 2024-03-11T19:22:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 2, 2024-03-11T19:22:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.175425355, 2024-03-11T19:22:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.053277907, 2024-03-11T19:22:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.044754261, 2024-03-11T19:22:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.065165598, 2024-03-11T19:22:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:22:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 2481, 2024-03-11T19:22:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.079549421, 2024-03-11T19:22:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000032063 2024-03-11T19:22:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:22:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-11T19:22:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 448 2024-03-11T19:22:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T19:22:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 254, 2024-03-11T19:22:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 248, 2024-03-11T19:22:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 775564, 2024-03-11T19:22:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 145532, 2024-03-11T19:22:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 3, 2024-03-11T19:22:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 2, 2024-03-11T19:22:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.036803865, 2024-03-11T19:22:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.012416707, 2024-03-11T19:22:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.022366254, 2024-03-11T19:22:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.005497074, 2024-03-11T19:22:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:22:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 247, 2024-03-11T19:22:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.009522030, 2024-03-11T19:22:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000038550 2024-03-11T19:22:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:22:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 775008, 2024-03-11T19:22:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 144848 2024-03-11T19:22:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T19:22:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 2, 2024-03-11T19:22:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 183, 2024-03-11T19:22:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 428, 2024-03-11T19:22:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 390711, 2024-03-11T19:22:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 4, 2024-03-11T19:22:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 3, 2024-03-11T19:22:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.008952984, 2024-03-11T19:22:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.006624652, 2024-03-11T19:22:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.000939929, 2024-03-11T19:22:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.000020442, 2024-03-11T19:22:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T19:22:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 181, 2024-03-11T19:22:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.008720869, 2024-03-11T19:22:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000048181 2024-03-11T19:22:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-03-11T19:22:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-11T19:22:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 390400 2024-03-11T19:22:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: "cct": { 2024-03-11T19:22:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: "total_workers": 1, 2024-03-11T19:22:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: "unhealthy_workers": 0 2024-03-11T19:22:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-MDSRank": { 2024-03-11T19:22:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-03-11T19:22:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-03-11T19:22:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 4156, 2024-03-11T19:22:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.100252329, 2024-03-11T19:22:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000024122 2024-03-11T19:22:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-PurgeQueue": { 2024-03-11T19:22:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-03-11T19:22:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-03-11T19:22:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "mds": { 2024-03-11T19:22:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "request": 0, 2024-03-11T19:22:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "reply": 0, 2024-03-11T19:22:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "reply_latency": { 2024-03-11T19:22:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_reply": 0, 2024-03-11T19:22:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "forward": 0, 2024-03-11T19:22:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_complete": 0, 2024-03-11T19:22:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_keys": 0, 2024-03-11T19:22:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_commit": 0, 2024-03-11T19:22:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_split": 0, 2024-03-11T19:22:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_merge": 0, 2024-03-11T19:22:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes": 0, 2024-03-11T19:22:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_top": 0, 2024-03-11T19:22:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_bottom": 0, 2024-03-11T19:22:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pin_tail": 0, 2024-03-11T19:22:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pinned": 0, 2024-03-11T19:22:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_expired": 0, 2024-03-11T19:22:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_with_caps": 0, 2024-03-11T19:22:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "caps": 0, 2024-03-11T19:22:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "subtrees": 1, 2024-03-11T19:22:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse": 0, 2024-03-11T19:22:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_hit": 0, 2024-03-11T19:22:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_forward": 0, 2024-03-11T19:22:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_discover": 0, 2024-03-11T19:22:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_dir_fetch": 0, 2024-03-11T19:22:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_remote_ino": 0, 2024-03-11T19:22:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_lock": 0, 2024-03-11T19:22:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "load_cent": 0, 2024-03-11T19:22:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "q": 0, 2024-03-11T19:22:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "exported": 0, 2024-03-11T19:22:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "exported_inodes": 0, 2024-03-11T19:22:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "imported": 0, 2024-03-11T19:22:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "imported_inodes": 0, 2024-03-11T19:22:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_dir_fetch": 0, 2024-03-11T19:22:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_backtrace_fetch": 0, 2024-03-11T19:22:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_peer_discover": 0, 2024-03-11T19:22:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rfiles": 0, 2024-03-11T19:22:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rbytes": 0, 2024-03-11T19:22:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rsnaps": 0, 2024-03-11T19:22:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T19:22:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_set_tag": 0, 2024-03-11T19:22:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T19:22:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_inotable_repaired": 0, 2024-03-11T19:22:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_inodes": 0, 2024-03-11T19:22:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T19:22:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T19:22:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_file_inodes": 0, 2024-03-11T19:22:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_inode_file_caps": 0, 2024-03-11T19:22:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T19:22:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_grant": 0, 2024-03-11T19:22:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T19:22:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T19:22:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T19:22:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps": 0, 2024-03-11T19:22:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps_dirty": 0, 2024-03-11T19:22:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_cap_release": 0, 2024-03-11T19:22:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "process_request_cap_release": 0 2024-03-11T19:22:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_cache": { 2024-03-11T19:22:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update": 0, 2024-03-11T19:22:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update_receipt": 0, 2024-03-11T19:22:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_try_discover": 0, 2024-03-11T19:22:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_send_discover": 0, 2024-03-11T19:22:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_handle_discover": 0, 2024-03-11T19:22:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays": 0, 2024-03-11T19:22:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_delayed": 0, 2024-03-11T19:22:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_enqueuing": 0, 2024-03-11T19:22:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_created": 0, 2024-03-11T19:22:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_enqueued": 0, 2024-03-11T19:22:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_reintegrated": 0, 2024-03-11T19:22:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_migrated": 0, 2024-03-11T19:22:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_processing": 0, 2024-03-11T19:22:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_enqueued": 0, 2024-03-11T19:22:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_prioritized": 0, 2024-03-11T19:22:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_started": 0, 2024-03-11T19:22:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_completed": 0, 2024-03-11T19:22:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T19:22:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_exportdir": 0, 2024-03-11T19:22:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_flush": 0, 2024-03-11T19:22:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragmentdir": 0, 2024-03-11T19:22:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragstats": 0, 2024-03-11T19:22:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_inodestats": 0 2024-03-11T19:22:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_log": { 2024-03-11T19:22:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "evlrg": 0, 2024-03-11T19:22:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "evadd": 0, 2024-03-11T19:22:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "evex": 0, 2024-03-11T19:22:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "evtrm": 0, 2024-03-11T19:22:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "ev": 5, 2024-03-11T19:22:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "evexg": 0, 2024-03-11T19:22:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "evexd": 0, 2024-03-11T19:22:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "segadd": 0, 2024-03-11T19:22:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "segex": 0, 2024-03-11T19:22:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "segtrm": 0, 2024-03-11T19:22:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "seg": 2, 2024-03-11T19:22:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "segmjr": 2, 2024-03-11T19:22:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "segexg": 0, 2024-03-11T19:22:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "segexd": 0, 2024-03-11T19:22:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "expos": 4194304, 2024-03-11T19:22:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "wrpos": 4195917, 2024-03-11T19:22:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "rdpos": 4195424, 2024-03-11T19:22:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "jlat": { 2024-03-11T19:22:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "replayed": 5 2024-03-11T19:22:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_mem": { 2024-03-11T19:22:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "ino": 3, 2024-03-11T19:22:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "ino+": 3, 2024-03-11T19:22:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "ino-": 0, 2024-03-11T19:22:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "dir": 1, 2024-03-11T19:22:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "dir+": 1, 2024-03-11T19:22:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "dir-": 0, 2024-03-11T19:22:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "dn": 0, 2024-03-11T19:22:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "dn+": 0, 2024-03-11T19:22:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "dn-": 0, 2024-03-11T19:22:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "cap": 0, 2024-03-11T19:22:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "cap+": 0, 2024-03-11T19:22:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "cap-": 0, 2024-03-11T19:22:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "rss": 51880, 2024-03-11T19:22:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "heap": 215420 2024-03-11T19:22:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_server": { 2024-03-11T19:22:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_client_request": 0, 2024-03-11T19:22:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_server_request": 0, 2024-03-11T19:22:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_request": 0, 2024-03-11T19:22:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_session": 0, 2024-03-11T19:22:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_peer_request": 0, 2024-03-11T19:22:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "req_create_latency": { 2024-03-11T19:22:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getattr_latency": { 2024-03-11T19:22:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getfilelock_latency": { 2024-03-11T19:22:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: "req_link_latency": { 2024-03-11T19:22:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookup_latency": { 2024-03-11T19:22:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookuphash_latency": { 2024-03-11T19:22:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupino_latency": { 2024-03-11T19:22:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupname_latency": { 2024-03-11T19:22:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupparent_latency": { 2024-03-11T19:22:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupsnap_latency": { 2024-03-11T19:22:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lssnap_latency": { 2024-03-11T19:22:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mkdir_latency": { 2024-03-11T19:22:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mknod_latency": { 2024-03-11T19:22:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mksnap_latency": { 2024-03-11T19:22:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "req_open_latency": { 2024-03-11T19:22:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: "req_readdir_latency": { 2024-03-11T19:22:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rename_latency": { 2024-03-11T19:22:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: "req_renamesnap_latency": { 2024-03-11T19:22:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "req_snapdiff_latency": { 2024-03-11T19:22:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmdir_latency": { 2024-03-11T19:22:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmsnap_latency": { 2024-03-11T19:22:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmxattr_latency": { 2024-03-11T19:22:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setattr_latency": { 2024-03-11T19:22:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setdirlayout_latency": { 2024-03-11T19:22:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setfilelock_latency": { 2024-03-11T19:22:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setlayout_latency": { 2024-03-11T19:22:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setxattr_latency": { 2024-03-11T19:22:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "req_symlink_latency": { 2024-03-11T19:22:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "req_unlink_latency": { 2024-03-11T19:22:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_revoke_eviction": 0, 2024-03-11T19:22:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_acquisition_throttle": 0, 2024-03-11T19:22:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getvxattr_latency": { 2024-03-11T19:22:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_sessions": { 2024-03-11T19:22:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "session_count": 1, 2024-03-11T19:22:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "session_add": 2, 2024-03-11T19:22:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "session_remove": 1, 2024-03-11T19:22:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_open": 1, 2024-03-11T19:22:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_stale": 0, 2024-03-11T19:22:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "total_load": 0, 2024-03-11T19:22:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "average_load": 0, 2024-03-11T19:22:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "avg_session_uptime": 1821, 2024-03-11T19:22:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "mdthresh_evicted": 0 2024-03-11T19:22:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "mempool": { 2024-03-11T19:22:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_bytes": 0, 2024-03-11T19:22:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_items": 0, 2024-03-11T19:22:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T19:22:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_items": 0, 2024-03-11T19:22:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T19:22:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_items": 0, 2024-03-11T19:22:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T19:22:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T19:22:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T19:22:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T19:22:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T19:22:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_items": 0, 2024-03-11T19:22:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T19:22:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T19:22:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_bytes": 0, 2024-03-11T19:22:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_items": 0, 2024-03-11T19:22:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_bytes": 0, 2024-03-11T19:22:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_items": 0, 2024-03-11T19:22:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T19:22:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T19:22:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T19:22:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T19:22:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T19:22:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_items": 0, 2024-03-11T19:22:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_bytes": 0, 2024-03-11T19:22:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_items": 0, 2024-03-11T19:22:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T19:22:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T19:22:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_bytes": 0, 2024-03-11T19:22:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_items": 0, 2024-03-11T19:22:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_bytes": 0, 2024-03-11T19:22:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_items": 0, 2024-03-11T19:22:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T19:22:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_items": 0, 2024-03-11T19:22:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T19:22:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_items": 0, 2024-03-11T19:22:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_bytes": 181303, 2024-03-11T19:22:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_items": 46, 2024-03-11T19:22:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_bytes": 0, 2024-03-11T19:22:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_items": 0, 2024-03-11T19:22:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_bytes": 0, 2024-03-11T19:22:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_items": 0, 2024-03-11T19:22:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_bytes": 0, 2024-03-11T19:22:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_items": 0, 2024-03-11T19:22:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_bytes": 0, 2024-03-11T19:22:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_items": 0, 2024-03-11T19:22:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_bytes": 7336, 2024-03-11T19:22:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_items": 154, 2024-03-11T19:22:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T19:22:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_items": 0, 2024-03-11T19:22:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_bytes": 0, 2024-03-11T19:22:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_items": 0, 2024-03-11T19:22:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_bytes": 8320, 2024-03-11T19:22:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_items": 13, 2024-03-11T19:22:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_bytes": 0, 2024-03-11T19:22:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_items": 0, 2024-03-11T19:22:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_bytes": 0, 2024-03-11T19:22:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_items": 0 2024-03-11T19:22:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "objecter": { 2024-03-11T19:22:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "op_active": 0, 2024-03-11T19:22:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "op_laggy": 0, 2024-03-11T19:22:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send": 2481, 2024-03-11T19:22:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send_bytes": 21, 2024-03-11T19:22:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "op_resend": 3, 2024-03-11T19:22:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "op_reply": 2481, 2024-03-11T19:22:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "op_latency": { 2024-03-11T19:22:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 2481, 2024-03-11T19:22:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 1.398959119, 2024-03-11T19:22:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000563869 2024-03-11T19:22:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "op_inflight": 0, 2024-03-11T19:22:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "oplen_avg": { 2024-03-11T19:22:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 2481, 2024-03-11T19:22:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 2482 2024-03-11T19:22:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "op": 2481, 2024-03-11T19:22:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "op_r": 2481, 2024-03-11T19:22:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "op_w": 0, 2024-03-11T19:22:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "op_rmw": 0, 2024-03-11T19:22:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "op_pg": 0, 2024-03-11T19:22:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_stat": 1623, 2024-03-11T19:22:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_create": 0, 2024-03-11T19:22:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_read": 856, 2024-03-11T19:22:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_write": 0, 2024-03-11T19:22:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writefull": 0, 2024-03-11T19:22:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writesame": 0, 2024-03-11T19:22:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_append": 0, 2024-03-11T19:22:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_zero": 0, 2024-03-11T19:22:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_truncate": 0, 2024-03-11T19:22:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_delete": 0, 2024-03-11T19:22:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_mapext": 0, 2024-03-11T19:22:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_sparse_read": 0, 2024-03-11T19:22:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_clonerange": 0, 2024-03-11T19:22:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_getxattr": 1, 2024-03-11T19:22:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_setxattr": 0, 2024-03-11T19:22:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_cmpxattr": 0, 2024-03-11T19:22:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_rmxattr": 0, 2024-03-11T19:22:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_resetxattrs": 0, 2024-03-11T19:22:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_call": 0, 2024-03-11T19:22:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_watch": 0, 2024-03-11T19:22:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_notify": 0, 2024-03-11T19:22:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T19:22:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls": 0, 2024-03-11T19:22:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls_filter": 0, 2024-03-11T19:22:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_other": 0, 2024-03-11T19:22:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_active": 0, 2024-03-11T19:22:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_send": 0, 2024-03-11T19:22:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_resend": 0, 2024-03-11T19:22:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_ping": 0, 2024-03-11T19:22:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_active": 0, 2024-03-11T19:22:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_send": 0, 2024-03-11T19:22:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_resend": 0, 2024-03-11T19:22:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_active": 0, 2024-03-11T19:22:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_send": 0, 2024-03-11T19:22:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_resend": 0, 2024-03-11T19:22:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_active": 0, 2024-03-11T19:22:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_send": 0, 2024-03-11T19:22:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_resend": 0, 2024-03-11T19:22:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "command_active": 0, 2024-03-11T19:22:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "command_send": 0, 2024-03-11T19:22:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "command_resend": 0, 2024-03-11T19:22:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "map_epoch": 89, 2024-03-11T19:22:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "map_full": 0, 2024-03-11T19:22:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "map_inc": 2, 2024-03-11T19:22:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_sessions": 4, 2024-03-11T19:22:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_open": 4, 2024-03-11T19:22:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_close": 0, 2024-03-11T19:22:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_laggy": 0, 2024-03-11T19:22:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_wr": 0, 2024-03-11T19:22:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_rd": 2, 2024-03-11T19:22:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_del": 0 2024-03-11T19:22:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "oft": { 2024-03-11T19:22:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_objs": 0, 2024-03-11T19:22:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_kv_pairs": 0, 2024-03-11T19:22:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_updates": 0, 2024-03-11T19:22:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_removes": 0 2024-03-11T19:22:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "osd-slow-ops": { 2024-03-11T19:22:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_ops_count": 0 2024-03-11T19:22:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "purge_queue": { 2024-03-11T19:22:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops": 0, 2024-03-11T19:22:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T19:22:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing": 0, 2024-03-11T19:22:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_high_water": 0, 2024-03-11T19:22:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed_ops": 0, 2024-03-11T19:22:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed": 0, 2024-03-11T19:22:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_item_in_journal": 0 2024-03-11T19:22:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T19:22:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:22:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-03-11T19:22:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:22:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 2734, 2024-03-11T19:22:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 1218901, 2024-03-11T19:22:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:22:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 2734, 2024-03-11T19:22:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-03-11T19:22:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-03-11T19:22:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 2734, 2024-03-11T19:22:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 1218901, 2024-03-11T19:22:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:22:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_bytes": { 2024-03-11T19:22:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:22:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-03-11T19:22:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:22:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-03-11T19:22:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-03-11T19:22:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:22:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-03-11T19:22:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 2481, 2024-03-11T19:22:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 1618, 2024-03-11T19:22:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 2481, 2024-03-11T19:22:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 1618, 2024-03-11T19:22:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:22:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_ops": { 2024-03-11T19:22:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:22:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 1024, 2024-03-11T19:22:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:22:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-03-11T19:22:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-03-11T19:22:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:22:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-03-11T19:22:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 2481, 2024-03-11T19:22:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 2481, 2024-03-11T19:22:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 2481, 2024-03-11T19:22:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 2481, 2024-03-11T19:22:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:22:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle": { 2024-03-11T19:22:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:22:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-03-11T19:22:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:22:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-03-11T19:22:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-03-11T19:22:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:22:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-03-11T19:22:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-03-11T19:22:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-03-11T19:22:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 0, 2024-03-11T19:22:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 0, 2024-03-11T19:22:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:22:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-11T19:22:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle-0x562c25f48960": { 2024-03-11T19:22:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-03-11T19:22:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-03-11T19:22:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-03-11T19:22:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-03-11T19:22:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-03-11T19:22:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-03-11T19:22:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-03-11T19:22:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-03-11T19:22:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-03-11T19:22:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 0, 2024-03-11T19:22:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 0, 2024-03-11T19:22:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-03-11T19:22:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-03-11T19:22:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-03-11T19:22:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-03-11T19:22:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:07.179 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-11T19:22:07.179 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-11T19:22:07.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:07 smithi190 ceph-mon[36292]: pgmap v720: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:22:07.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:07 smithi170 ceph-mon[35690]: pgmap v720: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:22:07.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:07 smithi029 ceph-mon[32439]: pgmap v720: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:22:08.197 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-11T19:22:08.197 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-11T19:22:08.197 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-11T19:22:08.209 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-11T19:22:08.209 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-11T19:22:08.209 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T19:22:08.238 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-11T19:22:08.238 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T19:22:08.295 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:22:08.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.438+0000 7f095b1a0640 1 -- 172.21.15.29:0/3311022560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0954073b60 msgr2=0x7f0954073fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:08.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.438+0000 7f095b1a0640 1 --2- 172.21.15.29:0/3311022560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0954073b60 0x7f0954073fc0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0950009900 tx=0x7f095002bcd0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.438+0000 7f095b1a0640 1 -- 172.21.15.29:0/3311022560 shutdown_connections 2024-03-11T19:22:08.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.438+0000 7f095b1a0640 1 --2- 172.21.15.29:0/3311022560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0954108e90 0x7f095410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.438+0000 7f095b1a0640 1 --2- 172.21.15.29:0/3311022560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0954073b60 0x7f0954073fc0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.438+0000 7f095b1a0640 1 --2- 172.21.15.29:0/3311022560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0954075140 0x7f0954073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.438+0000 7f095b1a0640 1 -- 172.21.15.29:0/3311022560 >> 172.21.15.29:0/3311022560 conn(0x7f09540fc550 msgr2=0x7f09540fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.438+0000 7f095b1a0640 1 -- 172.21.15.29:0/3311022560 shutdown_connections 2024-03-11T19:22:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.438+0000 7f095b1a0640 1 -- 172.21.15.29:0/3311022560 wait complete. 2024-03-11T19:22:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.439+0000 7f095b1a0640 1 Processor -- start 2024-03-11T19:22:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.439+0000 7f095b1a0640 1 -- start start 2024-03-11T19:22:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.439+0000 7f095b1a0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0954073b60 0x7f0954199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.439+0000 7f095b1a0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0954075140 0x7f09541996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.439+0000 7f095b1a0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0954108e90 0x7f09541a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.439+0000 7f095b1a0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09541a0c50 con 0x7f0954073b60 2024-03-11T19:22:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.439+0000 7f095b1a0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09541a0dc0 con 0x7f0954075140 2024-03-11T19:22:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.439+0000 7f095b1a0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09541a0f80 con 0x7f0954108e90 2024-03-11T19:22:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.439+0000 7f095a19e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0954073b60 0x7f0954199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.439+0000 7f095a19e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0954073b60 0x7f0954199160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54276/0 (socket says 172.21.15.29:54276) 2024-03-11T19:22:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.439+0000 7f095a19e640 1 -- 172.21.15.29:0/4245591860 learned_addr learned my addr 172.21.15.29:0/4245591860 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:22:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.439+0000 7f095999d640 1 --2- 172.21.15.29:0/4245591860 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0954075140 0x7f09541996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.439+0000 7f095a99f640 1 --2- 172.21.15.29:0/4245591860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0954108e90 0x7f09541a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.440+0000 7f095a19e640 1 -- 172.21.15.29:0/4245591860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0954108e90 msgr2=0x7f09541a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.440+0000 7f095a19e640 1 --2- 172.21.15.29:0/4245591860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0954108e90 0x7f09541a0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.440+0000 7f095a19e640 1 -- 172.21.15.29:0/4245591860 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0954075140 msgr2=0x7f09541996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.440+0000 7f095a19e640 1 --2- 172.21.15.29:0/4245591860 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0954075140 0x7f09541996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.440+0000 7f095a19e640 1 -- 172.21.15.29:0/4245591860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0950009560 con 0x7f0954073b60 2024-03-11T19:22:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.440+0000 7f095a19e640 1 --2- 172.21.15.29:0/4245591860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0954073b60 0x7f0954199160 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f094400e9f0 tx=0x7f094400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.440+0000 7f094b7fe640 1 -- 172.21.15.29:0/4245591860 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0944009700 con 0x7f0954073b60 2024-03-11T19:22:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.440+0000 7f095b1a0640 1 -- 172.21.15.29:0/4245591860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09541a1260 con 0x7f0954073b60 2024-03-11T19:22:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.440+0000 7f094b7fe640 1 -- 172.21.15.29:0/4245591860 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f094400cd80 con 0x7f0954073b60 2024-03-11T19:22:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.440+0000 7f094b7fe640 1 -- 172.21.15.29:0/4245591860 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0944018840 con 0x7f0954073b60 2024-03-11T19:22:08.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.440+0000 7f095b1a0640 1 -- 172.21.15.29:0/4245591860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09541a1740 con 0x7f0954073b60 2024-03-11T19:22:08.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.441+0000 7f095b1a0640 1 -- 172.21.15.29:0/4245591860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0954107c50 con 0x7f0954073b60 2024-03-11T19:22:08.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.442+0000 7f094b7fe640 1 -- 172.21.15.29:0/4245591860 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f09440189a0 con 0x7f0954073b60 2024-03-11T19:22:08.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.442+0000 7f094b7fe640 1 --2- 172.21.15.29:0/4245591860 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f09400a1a30 0x7f09400a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:08.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.443+0000 7f095999d640 1 --2- 172.21.15.29:0/4245591860 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f09400a1a30 0x7f09400a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:08.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.443+0000 7f094b7fe640 1 -- 172.21.15.29:0/4245591860 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f0944014070 con 0x7f0954073b60 2024-03-11T19:22:08.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.443+0000 7f095999d640 1 --2- 172.21.15.29:0/4245591860 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f09400a1a30 0x7f09400a3ef0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0950009230 tx=0x7f0950005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:08.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.447+0000 7f094b7fe640 1 -- 172.21.15.29:0/4245591860 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0944052d20 con 0x7f0954073b60 2024-03-11T19:22:08.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.633+0000 7f095b1a0640 1 -- 172.21.15.29:0/4245591860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f095402ce70 con 0x7f0954073b60 2024-03-11T19:22:08.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.635+0000 7f094b7fe640 1 -- 172.21.15.29:0/4245591860 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22559 (secure 0 0 0) 0x7f09440969f0 con 0x7f0954073b60 2024-03-11T19:22:08.639 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:22:08.639 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":89,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:08:44.981806+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:07:47.563184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:07:48.282461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:22:08.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.637+0000 7f095b1a0640 1 -- 172.21.15.29:0/4245591860 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f09400a1a30 msgr2=0x7f09400a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:08.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.637+0000 7f095b1a0640 1 --2- 172.21.15.29:0/4245591860 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f09400a1a30 0x7f09400a3ef0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0950009230 tx=0x7f0950005fd0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.637+0000 7f095b1a0640 1 -- 172.21.15.29:0/4245591860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0954073b60 msgr2=0x7f0954199160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:08.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.637+0000 7f095b1a0640 1 --2- 172.21.15.29:0/4245591860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0954073b60 0x7f0954199160 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f094400e9f0 tx=0x7f094400eeb0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.637+0000 7f095b1a0640 1 -- 172.21.15.29:0/4245591860 shutdown_connections 2024-03-11T19:22:08.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.637+0000 7f095b1a0640 1 --2- 172.21.15.29:0/4245591860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0954108e90 0x7f09541a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.637+0000 7f095b1a0640 1 --2- 172.21.15.29:0/4245591860 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0954075140 0x7f09541996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.637+0000 7f095b1a0640 1 --2- 172.21.15.29:0/4245591860 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f09400a1a30 0x7f09400a3ef0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.637+0000 7f095b1a0640 1 --2- 172.21.15.29:0/4245591860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0954073b60 0x7f0954199160 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.637+0000 7f095b1a0640 1 -- 172.21.15.29:0/4245591860 >> 172.21.15.29:0/4245591860 conn(0x7f09540fc550 msgr2=0x7f0954104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:08.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.637+0000 7f095b1a0640 1 -- 172.21.15.29:0/4245591860 shutdown_connections 2024-03-11T19:22:08.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.637+0000 7f095b1a0640 1 -- 172.21.15.29:0/4245591860 wait complete. 2024-03-11T19:22:08.651 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:22:08.651 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:22:08.651 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:22:08.651 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:22:08.651 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:08 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:22:08.653 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:03:09.860019+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T19:07:47.563184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T19:07:48.282461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T19:22:08.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:22:08.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:22:08.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:22:08.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:22:08.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:22:08.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:08 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:22:08.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.759+0000 7ff60ee3b640 1 -- 172.21.15.29:0/3197940280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff608104530 msgr2=0x7ff608106920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:08.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.759+0000 7ff60ee3b640 1 --2- 172.21.15.29:0/3197940280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff608104530 0x7ff608106920 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4009900 tx=0x7ff5f402bc30 comp rx=0 tx=0).stop 2024-03-11T19:22:08.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.759+0000 7ff60ee3b640 1 -- 172.21.15.29:0/3197940280 shutdown_connections 2024-03-11T19:22:08.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.759+0000 7ff60ee3b640 1 --2- 172.21.15.29:0/3197940280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff608106e60 0x7ff6081092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.759+0000 7ff60ee3b640 1 --2- 172.21.15.29:0/3197940280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff608104530 0x7ff608106920 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.759+0000 7ff60ee3b640 1 --2- 172.21.15.29:0/3197940280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff608101c00 0x7ff608103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.759+0000 7ff60ee3b640 1 -- 172.21.15.29:0/3197940280 >> 172.21.15.29:0/3197940280 conn(0x7ff6080fb610 msgr2=0x7ff6080fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:08.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.759+0000 7ff60ee3b640 1 -- 172.21.15.29:0/3197940280 shutdown_connections 2024-03-11T19:22:08.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.759+0000 7ff60ee3b640 1 -- 172.21.15.29:0/3197940280 wait complete. 2024-03-11T19:22:08.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.760+0000 7ff60ee3b640 1 Processor -- start 2024-03-11T19:22:08.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.760+0000 7ff60ee3b640 1 -- start start 2024-03-11T19:22:08.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.760+0000 7ff60ee3b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff608101c00 0x7ff608101570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:08.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.760+0000 7ff60ee3b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff608104530 0x7ff6080ffbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:08.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.760+0000 7ff60ee3b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff608106e60 0x7ff6081000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:08.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.760+0000 7ff60ee3b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff608100850 con 0x7ff608101c00 2024-03-11T19:22:08.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.760+0000 7ff60ee3b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6081009c0 con 0x7ff608104530 2024-03-11T19:22:08.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.760+0000 7ff60ee3b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6081a1240 con 0x7ff608106e60 2024-03-11T19:22:08.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.760+0000 7ff60cbd7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff608101c00 0x7ff608101570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:08.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.760+0000 7ff60cbd7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff608101c00 0x7ff608101570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54282/0 (socket says 172.21.15.29:54282) 2024-03-11T19:22:08.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.760+0000 7ff60cbd7640 1 -- 172.21.15.29:0/263197782 learned_addr learned my addr 172.21.15.29:0/263197782 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:22:08.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.760+0000 7ff60d3d8640 1 --2- 172.21.15.29:0/263197782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff608106e60 0x7ff6081000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:08.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.760+0000 7ff607fff640 1 --2- 172.21.15.29:0/263197782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff608104530 0x7ff6080ffbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:08.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.760+0000 7ff60cbd7640 1 -- 172.21.15.29:0/263197782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff608106e60 msgr2=0x7ff6081000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:08.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.760+0000 7ff60cbd7640 1 --2- 172.21.15.29:0/263197782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff608106e60 0x7ff6081000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.760+0000 7ff60cbd7640 1 -- 172.21.15.29:0/263197782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff608104530 msgr2=0x7ff6080ffbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:08.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.760+0000 7ff60cbd7640 1 --2- 172.21.15.29:0/263197782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff608104530 0x7ff6080ffbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.760+0000 7ff60cbd7640 1 -- 172.21.15.29:0/263197782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f4009560 con 0x7ff608101c00 2024-03-11T19:22:08.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.761+0000 7ff60cbd7640 1 --2- 172.21.15.29:0/263197782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff608101c00 0x7ff608101570 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc00ea20 tx=0x7ff5fc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:08.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.761+0000 7ff605ffb640 1 -- 172.21.15.29:0/263197782 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc009720 con 0x7ff608101c00 2024-03-11T19:22:08.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.761+0000 7ff605ffb640 1 -- 172.21.15.29:0/263197782 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff5fc00cd80 con 0x7ff608101c00 2024-03-11T19:22:08.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.761+0000 7ff605ffb640 1 -- 172.21.15.29:0/263197782 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc010430 con 0x7ff608101c00 2024-03-11T19:22:08.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.761+0000 7ff60ee3b640 1 -- 172.21.15.29:0/263197782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6081a14f0 con 0x7ff608101c00 2024-03-11T19:22:08.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.761+0000 7ff60ee3b640 1 -- 172.21.15.29:0/263197782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6081a1980 con 0x7ff608101c00 2024-03-11T19:22:08.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.762+0000 7ff60ee3b640 1 -- 172.21.15.29:0/263197782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6080fd100 con 0x7ff608101c00 2024-03-11T19:22:08.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.763+0000 7ff605ffb640 1 -- 172.21.15.29:0/263197782 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff5fc004fb0 con 0x7ff608101c00 2024-03-11T19:22:08.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.764+0000 7ff605ffb640 1 --2- 172.21.15.29:0/263197782 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff5f80a1a30 0x7ff5f80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:08.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.764+0000 7ff607fff640 1 --2- 172.21.15.29:0/263197782 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff5f80a1a30 0x7ff5f80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:08.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.764+0000 7ff605ffb640 1 -- 172.21.15.29:0/263197782 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff5fc014070 con 0x7ff608101c00 2024-03-11T19:22:08.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.765+0000 7ff607fff640 1 --2- 172.21.15.29:0/263197782 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff5f80a1a30 0x7ff5f80a3ef0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff5f402c7d0 tx=0x7ff5f4005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:08.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.768+0000 7ff605ffb640 1 -- 172.21.15.29:0/263197782 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff5fc096ba0 con 0x7ff608101c00 2024-03-11T19:22:08.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:22:08.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:22:08.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:22:08.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:22:08.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:08 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:22:08.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.952+0000 7ff60ee3b640 1 -- 172.21.15.29:0/263197782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff60802ce40 con 0x7ff608101c00 2024-03-11T19:22:08.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.953+0000 7ff605ffb640 1 -- 172.21.15.29:0/263197782 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff5fc09a260 con 0x7ff608101c00 2024-03-11T19:22:08.957 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-11T19:22:08.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.955+0000 7ff60ee3b640 1 -- 172.21.15.29:0/263197782 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff5f80a1a30 msgr2=0x7ff5f80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:08.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.955+0000 7ff60ee3b640 1 --2- 172.21.15.29:0/263197782 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff5f80a1a30 0x7ff5f80a3ef0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff5f402c7d0 tx=0x7ff5f4005fd0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.955+0000 7ff60ee3b640 1 -- 172.21.15.29:0/263197782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff608101c00 msgr2=0x7ff608101570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:08.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.955+0000 7ff60ee3b640 1 --2- 172.21.15.29:0/263197782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff608101c00 0x7ff608101570 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc00ea20 tx=0x7ff5fc00ee20 comp rx=0 tx=0).stop 2024-03-11T19:22:08.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.955+0000 7ff60ee3b640 1 -- 172.21.15.29:0/263197782 shutdown_connections 2024-03-11T19:22:08.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.955+0000 7ff60ee3b640 1 --2- 172.21.15.29:0/263197782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff608106e60 0x7ff6081000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.955+0000 7ff60ee3b640 1 --2- 172.21.15.29:0/263197782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff608104530 0x7ff6080ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.955+0000 7ff60ee3b640 1 --2- 172.21.15.29:0/263197782 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff5f80a1a30 0x7ff5f80a3ef0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.955+0000 7ff60ee3b640 1 --2- 172.21.15.29:0/263197782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff608101c00 0x7ff608101570 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:08.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.955+0000 7ff60ee3b640 1 -- 172.21.15.29:0/263197782 >> 172.21.15.29:0/263197782 conn(0x7ff6080fb610 msgr2=0x7ff608103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:08.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.955+0000 7ff60ee3b640 1 -- 172.21.15.29:0/263197782 shutdown_connections 2024-03-11T19:22:08.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:08.955+0000 7ff60ee3b640 1 -- 172.21.15.29:0/263197782 wait complete. 2024-03-11T19:22:08.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T19:22:09.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.072+0000 7fa9050c3640 1 -- 172.21.15.29:0/4025452029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9001045d0 msgr2=0x7fa9001069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:09.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.072+0000 7fa9050c3640 1 --2- 172.21.15.29:0/4025452029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9001045d0 0x7fa9001069c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8009900 tx=0x7fa8e802bc30 comp rx=0 tx=0).stop 2024-03-11T19:22:09.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.072+0000 7fa9050c3640 1 -- 172.21.15.29:0/4025452029 shutdown_connections 2024-03-11T19:22:09.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.072+0000 7fa9050c3640 1 --2- 172.21.15.29:0/4025452029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa900106f00 0x7fa9001092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.072+0000 7fa9050c3640 1 --2- 172.21.15.29:0/4025452029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9001045d0 0x7fa9001069c0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.072+0000 7fa9050c3640 1 --2- 172.21.15.29:0/4025452029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa900101ca0 0x7fa900104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.072+0000 7fa9050c3640 1 -- 172.21.15.29:0/4025452029 >> 172.21.15.29:0/4025452029 conn(0x7fa9000fb610 msgr2=0x7fa9000fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:09.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.072+0000 7fa9050c3640 1 -- 172.21.15.29:0/4025452029 shutdown_connections 2024-03-11T19:22:09.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.072+0000 7fa9050c3640 1 -- 172.21.15.29:0/4025452029 wait complete. 2024-03-11T19:22:09.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.073+0000 7fa9050c3640 1 Processor -- start 2024-03-11T19:22:09.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.073+0000 7fa9050c3640 1 -- start start 2024-03-11T19:22:09.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.073+0000 7fa9050c3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa900101ca0 0x7fa9001971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:09.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.073+0000 7fa9050c3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9001045d0 0x7fa900197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:09.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.073+0000 7fa9050c3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa900106f00 0x7fa90019e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:09.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.073+0000 7fa9050c3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90019ecb0 con 0x7fa900101ca0 2024-03-11T19:22:09.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.073+0000 7fa9050c3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90019ee20 con 0x7fa900106f00 2024-03-11T19:22:09.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.073+0000 7fa9050c3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90019efe0 con 0x7fa9001045d0 2024-03-11T19:22:09.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.073+0000 7fa8fed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa900101ca0 0x7fa9001971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:09.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.073+0000 7fa8fed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa900101ca0 0x7fa9001971c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54298/0 (socket says 172.21.15.29:54298) 2024-03-11T19:22:09.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.073+0000 7fa8fed9d640 1 -- 172.21.15.29:0/2284832379 learned_addr learned my addr 172.21.15.29:0/2284832379 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:22:09.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.073+0000 7fa8fe59c640 1 --2- 172.21.15.29:0/2284832379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9001045d0 0x7fa900197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:09.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.074+0000 7fa8ff59e640 1 --2- 172.21.15.29:0/2284832379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa900106f00 0x7fa90019e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:09.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.074+0000 7fa8fed9d640 1 -- 172.21.15.29:0/2284832379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9001045d0 msgr2=0x7fa900197700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:09.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.074+0000 7fa8fed9d640 1 --2- 172.21.15.29:0/2284832379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9001045d0 0x7fa900197700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.074+0000 7fa8fed9d640 1 -- 172.21.15.29:0/2284832379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa900106f00 msgr2=0x7fa90019e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:09.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.074+0000 7fa8fed9d640 1 --2- 172.21.15.29:0/2284832379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa900106f00 0x7fa90019e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.074+0000 7fa8fed9d640 1 -- 172.21.15.29:0/2284832379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e8009560 con 0x7fa900101ca0 2024-03-11T19:22:09.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.074+0000 7fa8ff59e640 1 --2- 172.21.15.29:0/2284832379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa900106f00 0x7fa90019e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:22:09.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.074+0000 7fa8fed9d640 1 --2- 172.21.15.29:0/2284832379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa900101ca0 0x7fa9001971c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800d950 tx=0x7fa8f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:09.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.074+0000 7fa8f7fff640 1 -- 172.21.15.29:0/2284832379 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8009c30 con 0x7fa900101ca0 2024-03-11T19:22:09.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.074+0000 7fa9050c3640 1 -- 172.21.15.29:0/2284832379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa90019f2c0 con 0x7fa900101ca0 2024-03-11T19:22:09.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.074+0000 7fa8f7fff640 1 -- 172.21.15.29:0/2284832379 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa8f8009d90 con 0x7fa900101ca0 2024-03-11T19:22:09.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.074+0000 7fa8f7fff640 1 -- 172.21.15.29:0/2284832379 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f800f6b0 con 0x7fa900101ca0 2024-03-11T19:22:09.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.074+0000 7fa9050c3640 1 -- 172.21.15.29:0/2284832379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa90019f7a0 con 0x7fa900101ca0 2024-03-11T19:22:09.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.075+0000 7fa9050c3640 1 -- 172.21.15.29:0/2284832379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9000fd100 con 0x7fa900101ca0 2024-03-11T19:22:09.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.076+0000 7fa8f7fff640 1 -- 172.21.15.29:0/2284832379 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fa8f8010aa0 con 0x7fa900101ca0 2024-03-11T19:22:09.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.077+0000 7fa8f7fff640 1 --2- 172.21.15.29:0/2284832379 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa8ec0a1a30 0x7fa8ec0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:09.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.077+0000 7fa8fe59c640 1 --2- 172.21.15.29:0/2284832379 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa8ec0a1a30 0x7fa8ec0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:09.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.077+0000 7fa8f7fff640 1 -- 172.21.15.29:0/2284832379 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fa8f80cd120 con 0x7fa900101ca0 2024-03-11T19:22:09.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.077+0000 7fa8fe59c640 1 --2- 172.21.15.29:0/2284832379 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa8ec0a1a30 0x7fa8ec0a3ef0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa8e802c7d0 tx=0x7fa8e8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:09.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.081+0000 7fa8f7fff640 1 -- 172.21.15.29:0/2284832379 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa8f8096330 con 0x7fa900101ca0 2024-03-11T19:22:09.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.263+0000 7fa9050c3640 1 -- 172.21.15.29:0/2284832379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa90002cdd0 con 0x7fa900101ca0 2024-03-11T19:22:09.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.263+0000 7fa8f7fff640 1 -- 172.21.15.29:0/2284832379 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa8f80999f0 con 0x7fa900101ca0 2024-03-11T19:22:09.267 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-03-11T19:22:09.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.266+0000 7fa9050c3640 1 -- 172.21.15.29:0/2284832379 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa8ec0a1a30 msgr2=0x7fa8ec0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:09.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.266+0000 7fa9050c3640 1 --2- 172.21.15.29:0/2284832379 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa8ec0a1a30 0x7fa8ec0a3ef0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa8e802c7d0 tx=0x7fa8e8005fd0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.266+0000 7fa9050c3640 1 -- 172.21.15.29:0/2284832379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa900101ca0 msgr2=0x7fa9001971c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:09.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.266+0000 7fa9050c3640 1 --2- 172.21.15.29:0/2284832379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa900101ca0 0x7fa9001971c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800d950 tx=0x7fa8f800de10 comp rx=0 tx=0).stop 2024-03-11T19:22:09.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.266+0000 7fa9050c3640 1 -- 172.21.15.29:0/2284832379 shutdown_connections 2024-03-11T19:22:09.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.266+0000 7fa9050c3640 1 --2- 172.21.15.29:0/2284832379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa900106f00 0x7fa90019e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.266+0000 7fa9050c3640 1 --2- 172.21.15.29:0/2284832379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9001045d0 0x7fa900197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.266+0000 7fa9050c3640 1 --2- 172.21.15.29:0/2284832379 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fa8ec0a1a30 0x7fa8ec0a3ef0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.266+0000 7fa9050c3640 1 --2- 172.21.15.29:0/2284832379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa900101ca0 0x7fa9001971c0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.266+0000 7fa9050c3640 1 -- 172.21.15.29:0/2284832379 >> 172.21.15.29:0/2284832379 conn(0x7fa9000fb610 msgr2=0x7fa900103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:09.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.266+0000 7fa9050c3640 1 -- 172.21.15.29:0/2284832379 shutdown_connections 2024-03-11T19:22:09.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.266+0000 7fa9050c3640 1 -- 172.21.15.29:0/2284832379 wait complete. 2024-03-11T19:22:09.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T19:22:09.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.383+0000 7f99bb908640 1 -- 172.21.15.29:0/2348495541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b4101c30 msgr2=0x7f99b41084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:09.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.383+0000 7f99bb908640 1 --2- 172.21.15.29:0/2348495541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b4101c30 0x7f99b41084c0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f99a4009ca0 tx=0x7f99a402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.383+0000 7f99bb908640 1 -- 172.21.15.29:0/2348495541 shutdown_connections 2024-03-11T19:22:09.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.383+0000 7f99bb908640 1 --2- 172.21.15.29:0/2348495541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b4101c30 0x7f99b41084c0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.383+0000 7f99bb908640 1 --2- 172.21.15.29:0/2348495541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99b4101270 0x7f99b41016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.383+0000 7f99bb908640 1 --2- 172.21.15.29:0/2348495541 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99b4100080 0x7f99b41004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.383+0000 7f99bb908640 1 -- 172.21.15.29:0/2348495541 >> 172.21.15.29:0/2348495541 conn(0x7f99b40fb610 msgr2=0x7f99b40fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:09.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.384+0000 7f99bb908640 1 -- 172.21.15.29:0/2348495541 shutdown_connections 2024-03-11T19:22:09.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.384+0000 7f99bb908640 1 -- 172.21.15.29:0/2348495541 wait complete. 2024-03-11T19:22:09.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.384+0000 7f99bb908640 1 Processor -- start 2024-03-11T19:22:09.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.384+0000 7f99bb908640 1 -- start start 2024-03-11T19:22:09.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.384+0000 7f99bb908640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99b4100080 0x7f99b4068e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:09.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.384+0000 7f99bb908640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99b4101270 0x7f99b419be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:09.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.384+0000 7f99bb908640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b4101c30 0x7f99b419e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:09.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.384+0000 7f99bb908640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b419ec80 con 0x7f99b4101c30 2024-03-11T19:22:09.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.384+0000 7f99bb908640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b419edf0 con 0x7f99b4101270 2024-03-11T19:22:09.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.384+0000 7f99bb908640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b419efb0 con 0x7f99b4100080 2024-03-11T19:22:09.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.384+0000 7f99b9ea5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b4101c30 0x7f99b419e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:09.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.384+0000 7f99b9ea5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b4101c30 0x7f99b419e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54324/0 (socket says 172.21.15.29:54324) 2024-03-11T19:22:09.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.384+0000 7f99b9ea5640 1 -- 172.21.15.29:0/1377968556 learned_addr learned my addr 172.21.15.29:0/1377968556 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:22:09.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.385+0000 7f99b96a4640 1 --2- 172.21.15.29:0/1377968556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99b4100080 0x7f99b4068e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:09.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.385+0000 7f99b8ea3640 1 --2- 172.21.15.29:0/1377968556 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99b4101270 0x7f99b419be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:09.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.385+0000 7f99b9ea5640 1 -- 172.21.15.29:0/1377968556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99b4100080 msgr2=0x7f99b4068e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:09.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.385+0000 7f99b9ea5640 1 --2- 172.21.15.29:0/1377968556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99b4100080 0x7f99b4068e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.385+0000 7f99b9ea5640 1 -- 172.21.15.29:0/1377968556 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99b4101270 msgr2=0x7f99b419be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:09.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.385+0000 7f99b9ea5640 1 --2- 172.21.15.29:0/1377968556 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99b4101270 0x7f99b419be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.385+0000 7f99b9ea5640 1 -- 172.21.15.29:0/1377968556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a4009900 con 0x7f99b4101c30 2024-03-11T19:22:09.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.385+0000 7f99b8ea3640 1 --2- 172.21.15.29:0/1377968556 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99b4101270 0x7f99b419be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:22:09.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.385+0000 7f99b9ea5640 1 --2- 172.21.15.29:0/1377968556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b4101c30 0x7f99b419e4a0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f99a4037040 tx=0x7f99a4032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:09.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.385+0000 7f99b27fc640 1 -- 172.21.15.29:0/1377968556 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a4034e70 con 0x7f99b4101c30 2024-03-11T19:22:09.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.385+0000 7f99bb908640 1 -- 172.21.15.29:0/1377968556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b419f1d0 con 0x7f99b4101c30 2024-03-11T19:22:09.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.385+0000 7f99b27fc640 1 -- 172.21.15.29:0/1377968556 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f99a4003e20 con 0x7f99b4101c30 2024-03-11T19:22:09.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.385+0000 7f99b27fc640 1 -- 172.21.15.29:0/1377968556 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a4004840 con 0x7f99b4101c30 2024-03-11T19:22:09.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.385+0000 7f99bb908640 1 -- 172.21.15.29:0/1377968556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b419f680 con 0x7f99b4101c30 2024-03-11T19:22:09.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.386+0000 7f99bb908640 1 -- 172.21.15.29:0/1377968556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b41073f0 con 0x7f99b4101c30 2024-03-11T19:22:09.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.387+0000 7f99b27fc640 1 -- 172.21.15.29:0/1377968556 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f99a403b040 con 0x7f99b4101c30 2024-03-11T19:22:09.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.388+0000 7f99b27fc640 1 --2- 172.21.15.29:0/1377968556 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f999c0a1a30 0x7f999c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:09.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.388+0000 7f99b96a4640 1 --2- 172.21.15.29:0/1377968556 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f999c0a1a30 0x7f999c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:09.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.388+0000 7f99b27fc640 1 -- 172.21.15.29:0/1377968556 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f99a40effe0 con 0x7f99b4101c30 2024-03-11T19:22:09.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.388+0000 7f99b96a4640 1 --2- 172.21.15.29:0/1377968556 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f999c0a1a30 0x7f999c0a3ef0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f99ac005fd0 tx=0x7f99ac005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:09.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.391+0000 7f99b27fc640 1 -- 172.21.15.29:0/1377968556 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f99a40b91f0 con 0x7f99b4101c30 2024-03-11T19:22:09.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.575+0000 7f99bb908640 1 -- 172.21.15.29:0/1377968556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f99b410bb40 con 0x7f99b4101c30 2024-03-11T19:22:09.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.575+0000 7f99b27fc640 1 -- 172.21.15.29:0/1377968556 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f99a4039020 con 0x7f99b4101c30 2024-03-11T19:22:09.579 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-03-11T19:22:09.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.577+0000 7f99bb908640 1 -- 172.21.15.29:0/1377968556 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f999c0a1a30 msgr2=0x7f999c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:09.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.577+0000 7f99bb908640 1 --2- 172.21.15.29:0/1377968556 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f999c0a1a30 0x7f999c0a3ef0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f99ac005fd0 tx=0x7f99ac005850 comp rx=0 tx=0).stop 2024-03-11T19:22:09.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.577+0000 7f99bb908640 1 -- 172.21.15.29:0/1377968556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b4101c30 msgr2=0x7f99b419e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:09.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.577+0000 7f99bb908640 1 --2- 172.21.15.29:0/1377968556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b4101c30 0x7f99b419e4a0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f99a4037040 tx=0x7f99a4032980 comp rx=0 tx=0).stop 2024-03-11T19:22:09.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.577+0000 7f99bb908640 1 -- 172.21.15.29:0/1377968556 shutdown_connections 2024-03-11T19:22:09.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.577+0000 7f99bb908640 1 --2- 172.21.15.29:0/1377968556 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f999c0a1a30 0x7f999c0a3ef0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.577+0000 7f99bb908640 1 --2- 172.21.15.29:0/1377968556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b4101c30 0x7f99b419e4a0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.577+0000 7f99bb908640 1 --2- 172.21.15.29:0/1377968556 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99b4101270 0x7f99b419be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.577+0000 7f99bb908640 1 --2- 172.21.15.29:0/1377968556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99b4100080 0x7f99b4068e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.577+0000 7f99bb908640 1 -- 172.21.15.29:0/1377968556 >> 172.21.15.29:0/1377968556 conn(0x7f99b40fb610 msgr2=0x7f99b41045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:09.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.577+0000 7f99bb908640 1 -- 172.21.15.29:0/1377968556 shutdown_connections 2024-03-11T19:22:09.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.577+0000 7f99bb908640 1 -- 172.21.15.29:0/1377968556 wait complete. 2024-03-11T19:22:09.591 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:09 smithi029 ceph-mon[32439]: pgmap v721: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:22:09.591 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:09 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/4245591860' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:22:09.591 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:09 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:22:09.591 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:09 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:22:09.591 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:09 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:22:09.592 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:09 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/263197782' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:22:09.592 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:09 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2284832379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:22:09.593 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-11T19:22:09.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.693+0000 7f45ddc1c640 1 -- 172.21.15.29:0/378362363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45d8102390 msgr2=0x7f45d8102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:09.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.693+0000 7f45ddc1c640 1 --2- 172.21.15.29:0/378362363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45d8102390 0x7f45d8102810 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009900 tx=0x7f45cc02bc30 comp rx=0 tx=0).stop 2024-03-11T19:22:09.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.693+0000 7f45ddc1c640 1 -- 172.21.15.29:0/378362363 shutdown_connections 2024-03-11T19:22:09.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.693+0000 7f45ddc1c640 1 --2- 172.21.15.29:0/378362363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45d8102d50 0x7f45d81095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.693+0000 7f45ddc1c640 1 --2- 172.21.15.29:0/378362363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45d8102390 0x7f45d8102810 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.693+0000 7f45ddc1c640 1 --2- 172.21.15.29:0/378362363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45d81011a0 0x7f45d81015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.693+0000 7f45ddc1c640 1 -- 172.21.15.29:0/378362363 >> 172.21.15.29:0/378362363 conn(0x7f45d80fc710 msgr2=0x7f45d80feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:09.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.693+0000 7f45ddc1c640 1 -- 172.21.15.29:0/378362363 shutdown_connections 2024-03-11T19:22:09.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.693+0000 7f45ddc1c640 1 -- 172.21.15.29:0/378362363 wait complete. 2024-03-11T19:22:09.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45ddc1c640 1 Processor -- start 2024-03-11T19:22:09.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45ddc1c640 1 -- start start 2024-03-11T19:22:09.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45ddc1c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45d81011a0 0x7f45d8074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:09.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45ddc1c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45d8102390 0x7f45d8073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:09.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45ddc1c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45d8102d50 0x7f45d8073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:09.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45ddc1c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d8074210 con 0x7f45d8102390 2024-03-11T19:22:09.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45ddc1c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d8074380 con 0x7f45d81011a0 2024-03-11T19:22:09.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45ddc1c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d81a11a0 con 0x7f45d8102d50 2024-03-11T19:22:09.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45d6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45d8102390 0x7f45d8073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:09.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45d6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45d8102390 0x7f45d8073570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54346/0 (socket says 172.21.15.29:54346) 2024-03-11T19:22:09.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45d6ffd640 1 -- 172.21.15.29:0/456155954 learned_addr learned my addr 172.21.15.29:0/456155954 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:22:09.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45d77fe640 1 --2- 172.21.15.29:0/456155954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45d81011a0 0x7f45d8074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:09.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45d7fff640 1 --2- 172.21.15.29:0/456155954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45d8102d50 0x7f45d8073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:09.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45d6ffd640 1 -- 172.21.15.29:0/456155954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45d8102d50 msgr2=0x7f45d8073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:09.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45d6ffd640 1 --2- 172.21.15.29:0/456155954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45d8102d50 0x7f45d8073ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45d6ffd640 1 -- 172.21.15.29:0/456155954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45d81011a0 msgr2=0x7f45d8074f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:09.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45d6ffd640 1 --2- 172.21.15.29:0/456155954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45d81011a0 0x7f45d8074f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45d6ffd640 1 -- 172.21.15.29:0/456155954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45cc009560 con 0x7f45d8102390 2024-03-11T19:22:09.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45d77fe640 1 --2- 172.21.15.29:0/456155954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45d81011a0 0x7f45d8074f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:22:09.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.694+0000 7f45d7fff640 1 --2- 172.21.15.29:0/456155954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45d8102d50 0x7f45d8073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:22:09.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.695+0000 7f45d6ffd640 1 --2- 172.21.15.29:0/456155954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45d8102390 0x7f45d8073570 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009430 tx=0x7f45cc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:09.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.695+0000 7f45d4ff9640 1 -- 172.21.15.29:0/456155954 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc004510 con 0x7f45d8102390 2024-03-11T19:22:09.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.695+0000 7f45d4ff9640 1 -- 172.21.15.29:0/456155954 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f45cc02de90 con 0x7f45d8102390 2024-03-11T19:22:09.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.695+0000 7f45ddc1c640 1 -- 172.21.15.29:0/456155954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d81a13f0 con 0x7f45d8102390 2024-03-11T19:22:09.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.695+0000 7f45d4ff9640 1 -- 172.21.15.29:0/456155954 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc003cc0 con 0x7f45d8102390 2024-03-11T19:22:09.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.695+0000 7f45ddc1c640 1 -- 172.21.15.29:0/456155954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d81a1900 con 0x7f45d8102390 2024-03-11T19:22:09.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.696+0000 7f45ddc1c640 1 -- 172.21.15.29:0/456155954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d81085e0 con 0x7f45d8102390 2024-03-11T19:22:09.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.697+0000 7f45d4ff9640 1 -- 172.21.15.29:0/456155954 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f45cc004fd0 con 0x7f45d8102390 2024-03-11T19:22:09.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.697+0000 7f45d4ff9640 1 --2- 172.21.15.29:0/456155954 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f45c00a1a30 0x7f45c00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:09.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.698+0000 7f45d77fe640 1 --2- 172.21.15.29:0/456155954 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f45c00a1a30 0x7f45c00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:09.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.698+0000 7f45d4ff9640 1 -- 172.21.15.29:0/456155954 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f45cc0ef650 con 0x7f45d8102390 2024-03-11T19:22:09.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.698+0000 7f45d77fe640 1 --2- 172.21.15.29:0/456155954 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f45c00a1a30 0x7f45c00a3ef0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f45c8005d00 tx=0x7f45c8005ef0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:09.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.702+0000 7f45d4ff9640 1 -- 172.21.15.29:0/456155954 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f45cc0b8900 con 0x7f45d8102390 2024-03-11T19:22:09.762 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:09 smithi190 ceph-mon[36292]: pgmap v721: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:22:09.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:09 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/4245591860' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:22:09.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:09 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:22:09.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:09 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:22:09.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:09 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:22:09.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:09 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/263197782' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:22:09.763 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:09 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2284832379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:22:09.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:09 smithi170 ceph-mon[35690]: pgmap v721: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-11T19:22:09.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:09 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/4245591860' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:22:09.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:09 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:22:09.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:09 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:22:09.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:09 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' 2024-03-11T19:22:09.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:09 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/263197782' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:22:09.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:09 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2284832379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:22:09.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.897+0000 7f45ddc1c640 1 -- 172.21.15.29:0/456155954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f45d8108050 con 0x7f45d8102390 2024-03-11T19:22:09.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.898+0000 7f45d4ff9640 1 -- 172.21.15.29:0/456155954 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+590 (secure 0 0 0) 0x7f45cc0bbfc0 con 0x7f45d8102390 2024-03-11T19:22:09.901 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2913819020 2024-03-12T19:00:43.524861+0000 2024-03-11T19:22:09.902 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/961617511 2024-03-12T19:00:43.524861+0000 2024-03-11T19:22:09.902 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6800/3585665950 2024-03-12T19:00:43.524861+0000 2024-03-11T19:22:09.902 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6801/3550575954 2024-03-12T19:00:15.449726+0000 2024-03-11T19:22:09.902 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6800/3550575954 2024-03-12T19:00:15.449726+0000 2024-03-11T19:22:09.902 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/385573026 2024-03-12T19:00:15.449726+0000 2024-03-11T19:22:09.902 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6801/3585665950 2024-03-12T19:00:43.524861+0000 2024-03-11T19:22:09.902 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2164849075 2024-03-12T19:00:15.449726+0000 2024-03-11T19:22:09.902 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1037268269 2024-03-12T19:00:43.524861+0000 2024-03-11T19:22:09.902 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2099360722 2024-03-12T19:00:15.449726+0000 2024-03-11T19:22:09.902 INFO:teuthology.orchestra.run.smithi029.stderr:listed 10 entries 2024-03-11T19:22:09.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.899+0000 7f45ddc1c640 1 -- 172.21.15.29:0/456155954 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f45c00a1a30 msgr2=0x7f45c00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:09.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.899+0000 7f45ddc1c640 1 --2- 172.21.15.29:0/456155954 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f45c00a1a30 0x7f45c00a3ef0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f45c8005d00 tx=0x7f45c8005ef0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.899+0000 7f45ddc1c640 1 -- 172.21.15.29:0/456155954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45d8102390 msgr2=0x7f45d8073570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:09.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.899+0000 7f45ddc1c640 1 --2- 172.21.15.29:0/456155954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45d8102390 0x7f45d8073570 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009430 tx=0x7f45cc0042f0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.900+0000 7f45ddc1c640 1 -- 172.21.15.29:0/456155954 shutdown_connections 2024-03-11T19:22:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.900+0000 7f45ddc1c640 1 --2- 172.21.15.29:0/456155954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45d8102d50 0x7f45d8073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.900+0000 7f45ddc1c640 1 --2- 172.21.15.29:0/456155954 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f45c00a1a30 0x7f45c00a3ef0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.900+0000 7f45ddc1c640 1 --2- 172.21.15.29:0/456155954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45d8102390 0x7f45d8073570 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.900+0000 7f45ddc1c640 1 --2- 172.21.15.29:0/456155954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45d81011a0 0x7f45d8074f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.900+0000 7f45ddc1c640 1 -- 172.21.15.29:0/456155954 >> 172.21.15.29:0/456155954 conn(0x7f45d80fc710 msgr2=0x7f45d81057b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.900+0000 7f45ddc1c640 1 -- 172.21.15.29:0/456155954 shutdown_connections 2024-03-11T19:22:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:09.900+0000 7f45ddc1c640 1 -- 172.21.15.29:0/456155954 wait complete. 2024-03-11T19:22:09.915 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-11T19:22:09.915 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T19:22:09.931 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:22:10.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.069+0000 7eff4a387640 1 -- 172.21.15.29:0/991597787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff441000a0 msgr2=0x7eff441004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:10.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.069+0000 7eff4a387640 1 --2- 172.21.15.29:0/991597787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff441000a0 0x7eff441004c0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7eff34009900 tx=0x7eff3402bc30 comp rx=0 tx=0).stop 2024-03-11T19:22:10.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.069+0000 7eff4a387640 1 -- 172.21.15.29:0/991597787 shutdown_connections 2024-03-11T19:22:10.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.069+0000 7eff4a387640 1 --2- 172.21.15.29:0/991597787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff44101c50 0x7eff441084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.069+0000 7eff4a387640 1 --2- 172.21.15.29:0/991597787 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff44101290 0x7eff44101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.069+0000 7eff4a387640 1 --2- 172.21.15.29:0/991597787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff441000a0 0x7eff441004c0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.069+0000 7eff4a387640 1 -- 172.21.15.29:0/991597787 >> 172.21.15.29:0/991597787 conn(0x7eff440fb610 msgr2=0x7eff440fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:10.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.069+0000 7eff4a387640 1 -- 172.21.15.29:0/991597787 shutdown_connections 2024-03-11T19:22:10.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.070+0000 7eff4a387640 1 -- 172.21.15.29:0/991597787 wait complete. 2024-03-11T19:22:10.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.070+0000 7eff4a387640 1 Processor -- start 2024-03-11T19:22:10.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.070+0000 7eff4a387640 1 -- start start 2024-03-11T19:22:10.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.070+0000 7eff4a387640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff441000a0 0x7eff44068e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:10.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.070+0000 7eff4a387640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff44101290 0x7eff4419bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:10.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.070+0000 7eff4a387640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff44101c50 0x7eff4419e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:10.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.070+0000 7eff4a387640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4419ed00 con 0x7eff44101290 2024-03-11T19:22:10.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.070+0000 7eff4a387640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4419ee70 con 0x7eff441000a0 2024-03-11T19:22:10.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.070+0000 7eff4a387640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4419f030 con 0x7eff44101c50 2024-03-11T19:22:10.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.070+0000 7eff437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff44101290 0x7eff4419bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.070+0000 7eff437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff44101290 0x7eff4419bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54366/0 (socket says 172.21.15.29:54366) 2024-03-11T19:22:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.070+0000 7eff437fe640 1 -- 172.21.15.29:0/3914446699 learned_addr learned my addr 172.21.15.29:0/3914446699 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:22:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.071+0000 7eff43fff640 1 --2- 172.21.15.29:0/3914446699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff441000a0 0x7eff44068e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.071+0000 7eff48924640 1 --2- 172.21.15.29:0/3914446699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff44101c50 0x7eff4419e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.071+0000 7eff437fe640 1 -- 172.21.15.29:0/3914446699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff44101c50 msgr2=0x7eff4419e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.071+0000 7eff437fe640 1 --2- 172.21.15.29:0/3914446699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff44101c50 0x7eff4419e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.071+0000 7eff437fe640 1 -- 172.21.15.29:0/3914446699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff441000a0 msgr2=0x7eff44068e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.071+0000 7eff437fe640 1 --2- 172.21.15.29:0/3914446699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff441000a0 0x7eff44068e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.071+0000 7eff437fe640 1 -- 172.21.15.29:0/3914446699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff34009560 con 0x7eff44101290 2024-03-11T19:22:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.071+0000 7eff437fe640 1 --2- 172.21.15.29:0/3914446699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff44101290 0x7eff4419bf00 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7eff3800d950 tx=0x7eff3800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.071+0000 7eff417fa640 1 -- 172.21.15.29:0/3914446699 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff38009a30 con 0x7eff44101290 2024-03-11T19:22:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.071+0000 7eff4a387640 1 -- 172.21.15.29:0/3914446699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff4419f2b0 con 0x7eff44101290 2024-03-11T19:22:10.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.071+0000 7eff417fa640 1 -- 172.21.15.29:0/3914446699 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7eff380044d0 con 0x7eff44101290 2024-03-11T19:22:10.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.071+0000 7eff417fa640 1 -- 172.21.15.29:0/3914446699 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff380047a0 con 0x7eff44101290 2024-03-11T19:22:10.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.071+0000 7eff4a387640 1 -- 172.21.15.29:0/3914446699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff4419f7c0 con 0x7eff44101290 2024-03-11T19:22:10.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.072+0000 7eff4a387640 1 -- 172.21.15.29:0/3914446699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff44107530 con 0x7eff44101290 2024-03-11T19:22:10.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.073+0000 7eff417fa640 1 -- 172.21.15.29:0/3914446699 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7eff38022020 con 0x7eff44101290 2024-03-11T19:22:10.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.074+0000 7eff417fa640 1 --2- 172.21.15.29:0/3914446699 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7eff2c0a1a30 0x7eff2c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:10.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.074+0000 7eff43fff640 1 --2- 172.21.15.29:0/3914446699 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7eff2c0a1a30 0x7eff2c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:10.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.074+0000 7eff417fa640 1 -- 172.21.15.29:0/3914446699 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7eff380ccd60 con 0x7eff44101290 2024-03-11T19:22:10.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.074+0000 7eff43fff640 1 --2- 172.21.15.29:0/3914446699 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7eff2c0a1a30 0x7eff2c0a3ef0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7eff3402c7d0 tx=0x7eff34005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:10.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.077+0000 7eff417fa640 1 -- 172.21.15.29:0/3914446699 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7eff38095f70 con 0x7eff44101290 2024-03-11T19:22:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.260+0000 7eff4a387640 1 -- 172.21.15.29:0/3914446699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff4410b960 con 0x7eff44101290 2024-03-11T19:22:10.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.261+0000 7eff417fa640 1 -- 172.21.15.29:0/3914446699 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22559 (secure 0 0 0) 0x7eff38017020 con 0x7eff44101290 2024-03-11T19:22:10.265 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:22:10.265 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":89,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:08:44.981806+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:07:47.563184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:07:48.282461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:22:10.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.264+0000 7eff4a387640 1 -- 172.21.15.29:0/3914446699 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7eff2c0a1a30 msgr2=0x7eff2c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:10.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.264+0000 7eff4a387640 1 --2- 172.21.15.29:0/3914446699 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7eff2c0a1a30 0x7eff2c0a3ef0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7eff3402c7d0 tx=0x7eff34005fd0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.264+0000 7eff4a387640 1 -- 172.21.15.29:0/3914446699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff44101290 msgr2=0x7eff4419bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:10.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.264+0000 7eff4a387640 1 --2- 172.21.15.29:0/3914446699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff44101290 0x7eff4419bf00 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7eff3800d950 tx=0x7eff3800de10 comp rx=0 tx=0).stop 2024-03-11T19:22:10.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.264+0000 7eff4a387640 1 -- 172.21.15.29:0/3914446699 shutdown_connections 2024-03-11T19:22:10.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.264+0000 7eff4a387640 1 --2- 172.21.15.29:0/3914446699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff44101c50 0x7eff4419e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.264+0000 7eff4a387640 1 --2- 172.21.15.29:0/3914446699 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7eff2c0a1a30 0x7eff2c0a3ef0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.264+0000 7eff4a387640 1 --2- 172.21.15.29:0/3914446699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff44101290 0x7eff4419bf00 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.264+0000 7eff4a387640 1 --2- 172.21.15.29:0/3914446699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff441000a0 0x7eff44068e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.264+0000 7eff4a387640 1 -- 172.21.15.29:0/3914446699 >> 172.21.15.29:0/3914446699 conn(0x7eff440fb610 msgr2=0x7eff44104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:10.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.264+0000 7eff4a387640 1 -- 172.21.15.29:0/3914446699 shutdown_connections 2024-03-11T19:22:10.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.264+0000 7eff4a387640 1 -- 172.21.15.29:0/3914446699 wait complete. 2024-03-11T19:22:10.279 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:03:09.860019+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T19:07:47.563184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T19:07:48.282461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T19:22:10.280 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:22:10.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.381+0000 7f95c6c9d640 1 -- 172.21.15.29:0/3325373855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95c0101280 msgr2=0x7f95c0101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:10.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.381+0000 7f95c6c9d640 1 --2- 172.21.15.29:0/3325373855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95c0101280 0x7f95c0101700 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009900 tx=0x7f95ac02bc30 comp rx=0 tx=0).stop 2024-03-11T19:22:10.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.381+0000 7f95c6c9d640 1 -- 172.21.15.29:0/3325373855 shutdown_connections 2024-03-11T19:22:10.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.381+0000 7f95c6c9d640 1 --2- 172.21.15.29:0/3325373855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95c0101c40 0x7f95c01084d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.381+0000 7f95c6c9d640 1 --2- 172.21.15.29:0/3325373855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95c0101280 0x7f95c0101700 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.381+0000 7f95c6c9d640 1 --2- 172.21.15.29:0/3325373855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95c0100090 0x7f95c01004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.381+0000 7f95c6c9d640 1 -- 172.21.15.29:0/3325373855 >> 172.21.15.29:0/3325373855 conn(0x7f95c00fb620 msgr2=0x7f95c00fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:10.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.381+0000 7f95c6c9d640 1 -- 172.21.15.29:0/3325373855 shutdown_connections 2024-03-11T19:22:10.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.381+0000 7f95c6c9d640 1 -- 172.21.15.29:0/3325373855 wait complete. 2024-03-11T19:22:10.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95c6c9d640 1 Processor -- start 2024-03-11T19:22:10.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95c6c9d640 1 -- start start 2024-03-11T19:22:10.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95c6c9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95c0100090 0x7f95c0197190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:10.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95c6c9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95c0101280 0x7f95c01976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:10.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95c6c9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95c0101c40 0x7f95c019e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:10.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95c6c9d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c019ec80 con 0x7f95c0100090 2024-03-11T19:22:10.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95c6c9d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c019edf0 con 0x7f95c0101c40 2024-03-11T19:22:10.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95c6c9d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c019efb0 con 0x7f95c0101280 2024-03-11T19:22:10.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95c4a39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95c0100090 0x7f95c0197190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:10.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95c4a39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95c0100090 0x7f95c0197190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54382/0 (socket says 172.21.15.29:54382) 2024-03-11T19:22:10.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95c4a39640 1 -- 172.21.15.29:0/2924088303 learned_addr learned my addr 172.21.15.29:0/2924088303 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:22:10.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95bffff640 1 --2- 172.21.15.29:0/2924088303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95c0101280 0x7f95c01976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:10.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95c523a640 1 --2- 172.21.15.29:0/2924088303 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95c0101c40 0x7f95c019e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:10.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95c4a39640 1 -- 172.21.15.29:0/2924088303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95c0101280 msgr2=0x7f95c01976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:10.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95c4a39640 1 --2- 172.21.15.29:0/2924088303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95c0101280 0x7f95c01976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95c4a39640 1 -- 172.21.15.29:0/2924088303 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95c0101c40 msgr2=0x7f95c019e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:10.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95c4a39640 1 --2- 172.21.15.29:0/2924088303 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95c0101c40 0x7f95c019e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95c4a39640 1 -- 172.21.15.29:0/2924088303 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ac009560 con 0x7f95c0100090 2024-03-11T19:22:10.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95c523a640 1 --2- 172.21.15.29:0/2924088303 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95c0101c40 0x7f95c019e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:22:10.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.382+0000 7f95c4a39640 1 --2- 172.21.15.29:0/2924088303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95c0100090 0x7f95c0197190 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f95b400e9c0 tx=0x7f95b400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:10.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.383+0000 7f95bdffb640 1 -- 172.21.15.29:0/2924088303 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b40096e0 con 0x7f95c0100090 2024-03-11T19:22:10.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.383+0000 7f95bdffb640 1 -- 172.21.15.29:0/2924088303 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f95b40044e0 con 0x7f95c0100090 2024-03-11T19:22:10.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.383+0000 7f95c6c9d640 1 -- 172.21.15.29:0/2924088303 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c019f290 con 0x7f95c0100090 2024-03-11T19:22:10.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.383+0000 7f95bdffb640 1 -- 172.21.15.29:0/2924088303 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b4010430 con 0x7f95c0100090 2024-03-11T19:22:10.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.383+0000 7f95c6c9d640 1 -- 172.21.15.29:0/2924088303 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c019f770 con 0x7f95c0100090 2024-03-11T19:22:10.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.384+0000 7f95c6c9d640 1 -- 172.21.15.29:0/2924088303 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c010b950 con 0x7f95c0100090 2024-03-11T19:22:10.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.385+0000 7f95bdffb640 1 -- 172.21.15.29:0/2924088303 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f95b4003f80 con 0x7f95c0100090 2024-03-11T19:22:10.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.385+0000 7f95bdffb640 1 --2- 172.21.15.29:0/2924088303 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f95b00a1ab0 0x7f95b00a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:10.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.385+0000 7f95bffff640 1 --2- 172.21.15.29:0/2924088303 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f95b00a1ab0 0x7f95b00a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:10.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.386+0000 7f95bdffb640 1 -- 172.21.15.29:0/2924088303 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f95b4014070 con 0x7f95c0100090 2024-03-11T19:22:10.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.386+0000 7f95bffff640 1 --2- 172.21.15.29:0/2924088303 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f95b00a1ab0 0x7f95b00a3f70 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009230 tx=0x7f95ac005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:10.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.390+0000 7f95bdffb640 1 -- 172.21.15.29:0/2924088303 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f95b40969a0 con 0x7f95c0100090 2024-03-11T19:22:10.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.572+0000 7f95c6c9d640 1 -- 172.21.15.29:0/2924088303 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f95c019fc60 con 0x7f95c0100090 2024-03-11T19:22:10.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.572+0000 7f95bdffb640 1 -- 172.21.15.29:0/2924088303 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f95b409a060 con 0x7f95c0100090 2024-03-11T19:22:10.576 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-11T19:22:10.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.574+0000 7f95c6c9d640 1 -- 172.21.15.29:0/2924088303 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f95b00a1ab0 msgr2=0x7f95b00a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:10.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.574+0000 7f95c6c9d640 1 --2- 172.21.15.29:0/2924088303 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f95b00a1ab0 0x7f95b00a3f70 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009230 tx=0x7f95ac005fd0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.574+0000 7f95c6c9d640 1 -- 172.21.15.29:0/2924088303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95c0100090 msgr2=0x7f95c0197190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:10.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.574+0000 7f95c6c9d640 1 --2- 172.21.15.29:0/2924088303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95c0100090 0x7f95c0197190 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f95b400e9c0 tx=0x7f95b400ee80 comp rx=0 tx=0).stop 2024-03-11T19:22:10.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.574+0000 7f95c6c9d640 1 -- 172.21.15.29:0/2924088303 shutdown_connections 2024-03-11T19:22:10.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.574+0000 7f95c6c9d640 1 --2- 172.21.15.29:0/2924088303 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95c0101c40 0x7f95c019e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.574+0000 7f95c6c9d640 1 --2- 172.21.15.29:0/2924088303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95c0101280 0x7f95c01976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.574+0000 7f95c6c9d640 1 --2- 172.21.15.29:0/2924088303 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f95b00a1ab0 0x7f95b00a3f70 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.574+0000 7f95c6c9d640 1 --2- 172.21.15.29:0/2924088303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95c0100090 0x7f95c0197190 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.574+0000 7f95c6c9d640 1 -- 172.21.15.29:0/2924088303 >> 172.21.15.29:0/2924088303 conn(0x7f95c00fb620 msgr2=0x7f95c0069770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:10.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.575+0000 7f95c6c9d640 1 -- 172.21.15.29:0/2924088303 shutdown_connections 2024-03-11T19:22:10.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.575+0000 7f95c6c9d640 1 -- 172.21.15.29:0/2924088303 wait complete. 2024-03-11T19:22:10.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:10 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/1377968556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:22:10.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:10 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/456155954' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T19:22:10.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:10 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3914446699' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:22:10.589 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T19:22:10.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.692+0000 7f37ea756640 1 -- 172.21.15.29:0/3675738244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37e4102d40 msgr2=0x7f37e41095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:10.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.692+0000 7f37ea756640 1 --2- 172.21.15.29:0/3675738244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37e4102d40 0x7f37e41095d0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f37dc009ca0 tx=0x7f37dc02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.692+0000 7f37ea756640 1 -- 172.21.15.29:0/3675738244 shutdown_connections 2024-03-11T19:22:10.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.692+0000 7f37ea756640 1 --2- 172.21.15.29:0/3675738244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37e4102d40 0x7f37e41095d0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.692+0000 7f37ea756640 1 --2- 172.21.15.29:0/3675738244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37e4102380 0x7f37e4102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.692+0000 7f37ea756640 1 --2- 172.21.15.29:0/3675738244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37e4101190 0x7f37e41015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.692+0000 7f37ea756640 1 -- 172.21.15.29:0/3675738244 >> 172.21.15.29:0/3675738244 conn(0x7f37e40fc720 msgr2=0x7f37e40feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:10.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.692+0000 7f37ea756640 1 -- 172.21.15.29:0/3675738244 shutdown_connections 2024-03-11T19:22:10.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.692+0000 7f37ea756640 1 -- 172.21.15.29:0/3675738244 wait complete. 2024-03-11T19:22:10.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.693+0000 7f37ea756640 1 Processor -- start 2024-03-11T19:22:10.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.693+0000 7f37ea756640 1 -- start start 2024-03-11T19:22:10.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.693+0000 7f37ea756640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37e4101190 0x7f37e41970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:10.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.693+0000 7f37ea756640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37e4102380 0x7f37e41975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:10.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.693+0000 7f37ea756640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37e4102d40 0x7f37e419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:10.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.693+0000 7f37ea756640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e419eba0 con 0x7f37e4102380 2024-03-11T19:22:10.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.693+0000 7f37ea756640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e419ed10 con 0x7f37e4102d40 2024-03-11T19:22:10.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.693+0000 7f37ea756640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e419eed0 con 0x7f37e4101190 2024-03-11T19:22:10.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.693+0000 7f37e37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37e4102380 0x7f37e41975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:10.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.693+0000 7f37e37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37e4102380 0x7f37e41975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54396/0 (socket says 172.21.15.29:54396) 2024-03-11T19:22:10.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.693+0000 7f37e37fe640 1 -- 172.21.15.29:0/2538029750 learned_addr learned my addr 172.21.15.29:0/2538029750 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:22:10.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.693+0000 7f37e3fff640 1 --2- 172.21.15.29:0/2538029750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37e4101190 0x7f37e41970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:10.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.693+0000 7f37e8cf3640 1 --2- 172.21.15.29:0/2538029750 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37e4102d40 0x7f37e419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:10.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.693+0000 7f37e37fe640 1 -- 172.21.15.29:0/2538029750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37e4101190 msgr2=0x7f37e41970b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:10.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.693+0000 7f37e37fe640 1 --2- 172.21.15.29:0/2538029750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37e4101190 0x7f37e41970b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.693+0000 7f37e37fe640 1 -- 172.21.15.29:0/2538029750 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37e4102d40 msgr2=0x7f37e419e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:10.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.693+0000 7f37e37fe640 1 --2- 172.21.15.29:0/2538029750 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37e4102d40 0x7f37e419e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.693+0000 7f37e37fe640 1 -- 172.21.15.29:0/2538029750 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37dc009900 con 0x7f37e4102380 2024-03-11T19:22:10.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.694+0000 7f37e37fe640 1 --2- 172.21.15.29:0/2538029750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37e4102380 0x7f37e41975f0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f37d400d770 tx=0x7f37d400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:10.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.694+0000 7f37e17fa640 1 -- 172.21.15.29:0/2538029750 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d4009c30 con 0x7f37e4102380 2024-03-11T19:22:10.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.694+0000 7f37e17fa640 1 -- 172.21.15.29:0/2538029750 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f37d400de80 con 0x7f37e4102380 2024-03-11T19:22:10.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.694+0000 7f37ea756640 1 -- 172.21.15.29:0/2538029750 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e419f1b0 con 0x7f37e4102380 2024-03-11T19:22:10.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.694+0000 7f37e17fa640 1 -- 172.21.15.29:0/2538029750 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d4004e00 con 0x7f37e4102380 2024-03-11T19:22:10.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.694+0000 7f37ea756640 1 -- 172.21.15.29:0/2538029750 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e419f6c0 con 0x7f37e4102380 2024-03-11T19:22:10.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.695+0000 7f37ea756640 1 -- 172.21.15.29:0/2538029750 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37e41084b0 con 0x7f37e4102380 2024-03-11T19:22:10.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.696+0000 7f37e17fa640 1 -- 172.21.15.29:0/2538029750 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f37d4004f60 con 0x7f37e4102380 2024-03-11T19:22:10.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.697+0000 7f37e17fa640 1 --2- 172.21.15.29:0/2538029750 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f37cc0a1ab0 0x7f37cc0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:10.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.697+0000 7f37e3fff640 1 --2- 172.21.15.29:0/2538029750 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f37cc0a1ab0 0x7f37cc0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:10.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.697+0000 7f37e17fa640 1 -- 172.21.15.29:0/2538029750 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f37d40ccde0 con 0x7f37e4102380 2024-03-11T19:22:10.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.697+0000 7f37e3fff640 1 --2- 172.21.15.29:0/2538029750 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f37cc0a1ab0 0x7f37cc0a3f70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f37e41021e0 tx=0x7f37d8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:10.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.701+0000 7f37e17fa640 1 -- 172.21.15.29:0/2538029750 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f37d4095ff0 con 0x7f37e4102380 2024-03-11T19:22:10.768 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:10 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/1377968556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:22:10.768 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:10 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/456155954' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T19:22:10.768 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:10 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3914446699' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:22:10.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:10 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/1377968556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:22:10.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:10 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/456155954' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T19:22:10.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:10 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3914446699' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:22:10.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.880+0000 7f37ea756640 1 -- 172.21.15.29:0/2538029750 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f37e402ce40 con 0x7f37e4102380 2024-03-11T19:22:10.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.880+0000 7f37e17fa640 1 -- 172.21.15.29:0/2538029750 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f37d4017020 con 0x7f37e4102380 2024-03-11T19:22:10.884 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-03-11T19:22:10.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.883+0000 7f37ea756640 1 -- 172.21.15.29:0/2538029750 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f37cc0a1ab0 msgr2=0x7f37cc0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:10.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.883+0000 7f37ea756640 1 --2- 172.21.15.29:0/2538029750 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f37cc0a1ab0 0x7f37cc0a3f70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f37e41021e0 tx=0x7f37d8005f90 comp rx=0 tx=0).stop 2024-03-11T19:22:10.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.883+0000 7f37ea756640 1 -- 172.21.15.29:0/2538029750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37e4102380 msgr2=0x7f37e41975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:10.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.883+0000 7f37ea756640 1 --2- 172.21.15.29:0/2538029750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37e4102380 0x7f37e41975f0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f37d400d770 tx=0x7f37d400dc30 comp rx=0 tx=0).stop 2024-03-11T19:22:10.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.883+0000 7f37ea756640 1 -- 172.21.15.29:0/2538029750 shutdown_connections 2024-03-11T19:22:10.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.883+0000 7f37ea756640 1 --2- 172.21.15.29:0/2538029750 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37e4102d40 0x7f37e419e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.883+0000 7f37ea756640 1 --2- 172.21.15.29:0/2538029750 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f37cc0a1ab0 0x7f37cc0a3f70 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.883+0000 7f37ea756640 1 --2- 172.21.15.29:0/2538029750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37e4102380 0x7f37e41975f0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.883+0000 7f37ea756640 1 --2- 172.21.15.29:0/2538029750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37e4101190 0x7f37e41970b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:10.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.883+0000 7f37ea756640 1 -- 172.21.15.29:0/2538029750 >> 172.21.15.29:0/2538029750 conn(0x7f37e40fc720 msgr2=0x7f37e4105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:10.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.883+0000 7f37ea756640 1 -- 172.21.15.29:0/2538029750 shutdown_connections 2024-03-11T19:22:10.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:10.883+0000 7f37ea756640 1 -- 172.21.15.29:0/2538029750 wait complete. 2024-03-11T19:22:10.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T19:22:11.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.000+0000 7f7b0e127640 1 -- 172.21.15.29:0/3413455131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b08100080 msgr2=0x7f7b081004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:11.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.000+0000 7f7b0e127640 1 --2- 172.21.15.29:0/3413455131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b08100080 0x7f7b081004a0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f7af8009900 tx=0x7f7af802bc30 comp rx=0 tx=0).stop 2024-03-11T19:22:11.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.000+0000 7f7b0e127640 1 -- 172.21.15.29:0/3413455131 shutdown_connections 2024-03-11T19:22:11.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.000+0000 7f7b0e127640 1 --2- 172.21.15.29:0/3413455131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b08101c30 0x7f7b081084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.000+0000 7f7b0e127640 1 --2- 172.21.15.29:0/3413455131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b08101270 0x7f7b081016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.000+0000 7f7b0e127640 1 --2- 172.21.15.29:0/3413455131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b08100080 0x7f7b081004a0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.000+0000 7f7b0e127640 1 -- 172.21.15.29:0/3413455131 >> 172.21.15.29:0/3413455131 conn(0x7f7b080fb610 msgr2=0x7f7b080fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.000+0000 7f7b0e127640 1 -- 172.21.15.29:0/3413455131 shutdown_connections 2024-03-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.000+0000 7f7b0e127640 1 -- 172.21.15.29:0/3413455131 wait complete. 2024-03-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.001+0000 7f7b0e127640 1 Processor -- start 2024-03-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.001+0000 7f7b0e127640 1 -- start start 2024-03-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.001+0000 7f7b0e127640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b08100080 0x7f7b081971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.001+0000 7f7b0e127640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b08101270 0x7f7b08197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.001+0000 7f7b0e127640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b08101c30 0x7f7b0819e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.001+0000 7f7b0e127640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0819ecb0 con 0x7f7b08100080 2024-03-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.001+0000 7f7b0e127640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0819ee20 con 0x7f7b08101270 2024-03-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.001+0000 7f7b0e127640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0819efe0 con 0x7f7b08101c30 2024-03-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.001+0000 7f7b077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b08100080 0x7f7b081971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.001+0000 7f7b077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b08100080 0x7f7b081971c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54410/0 (socket says 172.21.15.29:54410) 2024-03-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.001+0000 7f7b077fe640 1 -- 172.21.15.29:0/3701111886 learned_addr learned my addr 172.21.15.29:0/3701111886 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.001+0000 7f7b06ffd640 1 --2- 172.21.15.29:0/3701111886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b08101270 0x7f7b08197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.001+0000 7f7b07fff640 1 --2- 172.21.15.29:0/3701111886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b08101c30 0x7f7b0819e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.001+0000 7f7b077fe640 1 -- 172.21.15.29:0/3701111886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b08101c30 msgr2=0x7f7b0819e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.001+0000 7f7b077fe640 1 --2- 172.21.15.29:0/3701111886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b08101c30 0x7f7b0819e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.001+0000 7f7b077fe640 1 -- 172.21.15.29:0/3701111886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b08101270 msgr2=0x7f7b08197700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.001+0000 7f7b077fe640 1 --2- 172.21.15.29:0/3701111886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b08101270 0x7f7b08197700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.001+0000 7f7b077fe640 1 -- 172.21.15.29:0/3701111886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af8009560 con 0x7f7b08100080 2024-03-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.002+0000 7f7b077fe640 1 --2- 172.21.15.29:0/3701111886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b08100080 0x7f7b081971c0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f7af802c7d0 tx=0x7f7af8004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.002+0000 7f7b04ff9640 1 -- 172.21.15.29:0/3701111886 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8003800 con 0x7f7b08100080 2024-03-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.002+0000 7f7b04ff9640 1 -- 172.21.15.29:0/3701111886 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7af802de90 con 0x7f7b08100080 2024-03-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.002+0000 7f7b04ff9640 1 -- 172.21.15.29:0/3701111886 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af80055f0 con 0x7f7b08100080 2024-03-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.002+0000 7f7b0e127640 1 -- 172.21.15.29:0/3701111886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b0819f260 con 0x7f7b08100080 2024-03-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.002+0000 7f7b0e127640 1 -- 172.21.15.29:0/3701111886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b0819f720 con 0x7f7b08100080 2024-03-11T19:22:11.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.003+0000 7f7b0e127640 1 -- 172.21.15.29:0/3701111886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b08107510 con 0x7f7b08100080 2024-03-11T19:22:11.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.004+0000 7f7b04ff9640 1 -- 172.21.15.29:0/3701111886 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7af8004dc0 con 0x7f7b08100080 2024-03-11T19:22:11.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.004+0000 7f7b04ff9640 1 --2- 172.21.15.29:0/3701111886 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7af00a1a30 0x7f7af00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:11.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.004+0000 7f7b06ffd640 1 --2- 172.21.15.29:0/3701111886 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7af00a1a30 0x7f7af00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:11.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.004+0000 7f7b04ff9640 1 -- 172.21.15.29:0/3701111886 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f7af80ef750 con 0x7f7b08100080 2024-03-11T19:22:11.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.005+0000 7f7b06ffd640 1 --2- 172.21.15.29:0/3701111886 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7af00a1a30 0x7f7af00a3ef0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7b081986d0 tx=0x7f7afc00a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:11.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.009+0000 7f7b04ff9640 1 -- 172.21.15.29:0/3701111886 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7af80b8a00 con 0x7f7b08100080 2024-03-11T19:22:11.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.190+0000 7f7b0e127640 1 -- 172.21.15.29:0/3701111886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7b0802cda0 con 0x7f7b08100080 2024-03-11T19:22:11.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.191+0000 7f7b04ff9640 1 -- 172.21.15.29:0/3701111886 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7af80bc0c0 con 0x7f7b08100080 2024-03-11T19:22:11.195 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-03-11T19:22:11.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.193+0000 7f7b0e127640 1 -- 172.21.15.29:0/3701111886 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7af00a1a30 msgr2=0x7f7af00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:11.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.193+0000 7f7b0e127640 1 --2- 172.21.15.29:0/3701111886 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7af00a1a30 0x7f7af00a3ef0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7b081986d0 tx=0x7f7afc00a4c0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.193+0000 7f7b0e127640 1 -- 172.21.15.29:0/3701111886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b08100080 msgr2=0x7f7b081971c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:11.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.193+0000 7f7b0e127640 1 --2- 172.21.15.29:0/3701111886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b08100080 0x7f7b081971c0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f7af802c7d0 tx=0x7f7af8004360 comp rx=0 tx=0).stop 2024-03-11T19:22:11.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.193+0000 7f7b0e127640 1 -- 172.21.15.29:0/3701111886 shutdown_connections 2024-03-11T19:22:11.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.193+0000 7f7b0e127640 1 --2- 172.21.15.29:0/3701111886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b08101c30 0x7f7b0819e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.193+0000 7f7b0e127640 1 --2- 172.21.15.29:0/3701111886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b08101270 0x7f7b08197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.193+0000 7f7b0e127640 1 --2- 172.21.15.29:0/3701111886 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7af00a1a30 0x7f7af00a3ef0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.193+0000 7f7b0e127640 1 --2- 172.21.15.29:0/3701111886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b08100080 0x7f7b081971c0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.193+0000 7f7b0e127640 1 -- 172.21.15.29:0/3701111886 >> 172.21.15.29:0/3701111886 conn(0x7f7b080fb610 msgr2=0x7f7b081046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:11.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.193+0000 7f7b0e127640 1 -- 172.21.15.29:0/3701111886 shutdown_connections 2024-03-11T19:22:11.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.193+0000 7f7b0e127640 1 -- 172.21.15.29:0/3701111886 wait complete. 2024-03-11T19:22:11.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-11T19:22:11.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.308+0000 7ff883cf4640 1 -- 172.21.15.29:0/1637437946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c106ee0 msgr2=0x7ff87c1092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:11.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.308+0000 7ff883cf4640 1 --2- 172.21.15.29:0/1637437946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c106ee0 0x7ff87c1092d0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7ff86c009ca0 tx=0x7ff86c02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.308+0000 7ff883cf4640 1 -- 172.21.15.29:0/1637437946 shutdown_connections 2024-03-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.308+0000 7ff883cf4640 1 --2- 172.21.15.29:0/1637437946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c106ee0 0x7ff87c1092d0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.308+0000 7ff883cf4640 1 --2- 172.21.15.29:0/1637437946 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff87c1045b0 0x7ff87c1069a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.308+0000 7ff883cf4640 1 --2- 172.21.15.29:0/1637437946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff87c101c80 0x7ff87c104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.308+0000 7ff883cf4640 1 -- 172.21.15.29:0/1637437946 >> 172.21.15.29:0/1637437946 conn(0x7ff87c0fb610 msgr2=0x7ff87c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.308+0000 7ff883cf4640 1 -- 172.21.15.29:0/1637437946 shutdown_connections 2024-03-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.308+0000 7ff883cf4640 1 -- 172.21.15.29:0/1637437946 wait complete. 2024-03-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.309+0000 7ff883cf4640 1 Processor -- start 2024-03-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.309+0000 7ff883cf4640 1 -- start start 2024-03-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.309+0000 7ff883cf4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c101c80 0x7ff87c1015f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.309+0000 7ff883cf4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff87c1045b0 0x7ff87c0ffc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.309+0000 7ff883cf4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff87c106ee0 0x7ff87c100170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.309+0000 7ff883cf4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c19af80 con 0x7ff87c101c80 2024-03-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.309+0000 7ff883cf4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c1006b0 con 0x7ff87c106ee0 2024-03-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.309+0000 7ff883cf4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c100870 con 0x7ff87c1045b0 2024-03-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.309+0000 7ff881a90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c101c80 0x7ff87c1015f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.309+0000 7ff881a90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c101c80 0x7ff87c1015f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54426/0 (socket says 172.21.15.29:54426) 2024-03-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.309+0000 7ff881a90640 1 -- 172.21.15.29:0/2540964737 learned_addr learned my addr 172.21.15.29:0/2540964737 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.309+0000 7ff88128f640 1 --2- 172.21.15.29:0/2540964737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff87c1045b0 0x7ff87c0ffc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.310+0000 7ff882291640 1 --2- 172.21.15.29:0/2540964737 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff87c106ee0 0x7ff87c100170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.310+0000 7ff881a90640 1 -- 172.21.15.29:0/2540964737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff87c1045b0 msgr2=0x7ff87c0ffc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.310+0000 7ff881a90640 1 --2- 172.21.15.29:0/2540964737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff87c1045b0 0x7ff87c0ffc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.310+0000 7ff881a90640 1 -- 172.21.15.29:0/2540964737 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff87c106ee0 msgr2=0x7ff87c100170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.310+0000 7ff881a90640 1 --2- 172.21.15.29:0/2540964737 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff87c106ee0 0x7ff87c100170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.310+0000 7ff881a90640 1 -- 172.21.15.29:0/2540964737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86c009900 con 0x7ff87c101c80 2024-03-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.310+0000 7ff882291640 1 --2- 172.21.15.29:0/2540964737 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff87c106ee0 0x7ff87c100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.310+0000 7ff881a90640 1 --2- 172.21.15.29:0/2540964737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c101c80 0x7ff87c1015f0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7ff87400c890 tx=0x7ff87400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.310+0000 7ff87affd640 1 -- 172.21.15.29:0/2540964737 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff874004220 con 0x7ff87c101c80 2024-03-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.310+0000 7ff87affd640 1 -- 172.21.15.29:0/2540964737 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff874007e60 con 0x7ff87c101c80 2024-03-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.310+0000 7ff883cf4640 1 -- 172.21.15.29:0/2540964737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff87c1a5970 con 0x7ff87c101c80 2024-03-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.310+0000 7ff87affd640 1 -- 172.21.15.29:0/2540964737 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff874004a00 con 0x7ff87c101c80 2024-03-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.310+0000 7ff883cf4640 1 -- 172.21.15.29:0/2540964737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff87c1a5d30 con 0x7ff87c101c80 2024-03-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.311+0000 7ff883cf4640 1 -- 172.21.15.29:0/2540964737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff87c067f70 con 0x7ff87c101c80 2024-03-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.312+0000 7ff87affd640 1 -- 172.21.15.29:0/2540964737 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff8740079b0 con 0x7ff87c101c80 2024-03-11T19:22:11.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.312+0000 7ff87affd640 1 --2- 172.21.15.29:0/2540964737 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff8640a1a30 0x7ff8640a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:11.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.313+0000 7ff88128f640 1 --2- 172.21.15.29:0/2540964737 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff8640a1a30 0x7ff8640a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:11.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.313+0000 7ff87affd640 1 -- 172.21.15.29:0/2540964737 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7ff8740cd060 con 0x7ff87c101c80 2024-03-11T19:22:11.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.313+0000 7ff88128f640 1 --2- 172.21.15.29:0/2540964737 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff8640a1a30 0x7ff8640a3ef0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff87c101360 tx=0x7ff87000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:11.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.317+0000 7ff87affd640 1 -- 172.21.15.29:0/2540964737 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff874096310 con 0x7ff87c101c80 2024-03-11T19:22:11.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.512+0000 7ff883cf4640 1 -- 172.21.15.29:0/2540964737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff87c0fd100 con 0x7ff87c101c80 2024-03-11T19:22:11.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.515+0000 7ff87affd640 1 -- 172.21.15.29:0/2540964737 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+590 (secure 0 0 0) 0x7ff87401d260 con 0x7ff87c101c80 2024-03-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2913819020 2024-03-12T19:00:43.524861+0000 2024-03-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/961617511 2024-03-12T19:00:43.524861+0000 2024-03-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6800/3585665950 2024-03-12T19:00:43.524861+0000 2024-03-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6801/3550575954 2024-03-12T19:00:15.449726+0000 2024-03-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6800/3550575954 2024-03-12T19:00:15.449726+0000 2024-03-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/385573026 2024-03-12T19:00:15.449726+0000 2024-03-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6801/3585665950 2024-03-12T19:00:43.524861+0000 2024-03-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2164849075 2024-03-12T19:00:15.449726+0000 2024-03-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1037268269 2024-03-12T19:00:43.524861+0000 2024-03-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2099360722 2024-03-12T19:00:15.449726+0000 2024-03-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi029.stderr:listed 10 entries 2024-03-11T19:22:11.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.517+0000 7ff883cf4640 1 -- 172.21.15.29:0/2540964737 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff8640a1a30 msgr2=0x7ff8640a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:11.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.517+0000 7ff883cf4640 1 --2- 172.21.15.29:0/2540964737 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff8640a1a30 0x7ff8640a3ef0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff87c101360 tx=0x7ff87000b390 comp rx=0 tx=0).stop 2024-03-11T19:22:11.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.517+0000 7ff883cf4640 1 -- 172.21.15.29:0/2540964737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c101c80 msgr2=0x7ff87c1015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:11.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.517+0000 7ff883cf4640 1 --2- 172.21.15.29:0/2540964737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c101c80 0x7ff87c1015f0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7ff87400c890 tx=0x7ff87400cd50 comp rx=0 tx=0).stop 2024-03-11T19:22:11.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.517+0000 7ff883cf4640 1 -- 172.21.15.29:0/2540964737 shutdown_connections 2024-03-11T19:22:11.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.517+0000 7ff883cf4640 1 --2- 172.21.15.29:0/2540964737 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff87c106ee0 0x7ff87c100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.517+0000 7ff883cf4640 1 --2- 172.21.15.29:0/2540964737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff87c1045b0 0x7ff87c0ffc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.517+0000 7ff883cf4640 1 --2- 172.21.15.29:0/2540964737 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7ff8640a1a30 0x7ff8640a3ef0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.517+0000 7ff883cf4640 1 --2- 172.21.15.29:0/2540964737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c101c80 0x7ff87c1015f0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:11.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.517+0000 7ff883cf4640 1 -- 172.21.15.29:0/2540964737 >> 172.21.15.29:0/2540964737 conn(0x7ff87c0fb610 msgr2=0x7ff87c102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:11.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.517+0000 7ff883cf4640 1 -- 172.21.15.29:0/2540964737 shutdown_connections 2024-03-11T19:22:11.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:11.517+0000 7ff883cf4640 1 -- 172.21.15.29:0/2540964737 wait complete. 2024-03-11T19:22:11.532 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi029.front.sepia.ceph.com... 2024-03-11T19:22:11.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:22:11.532 DEBUG:teuthology.orchestra.run.smithi029:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-11T19:22:11.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi029.stderr:ceph-fuse[84458]: fuse finished with error 0 and tester_r 0 2024-03-11T19:22:11.573 INFO:teuthology.orchestra.run:waiting for 300 2024-03-11T19:22:11.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:11 smithi170 ceph-mon[35690]: pgmap v722: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:22:11.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:11 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2924088303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:22:11.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:11 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2538029750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:22:11.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:11 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3701111886' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:22:11.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:11 smithi029 ceph-mon[32439]: pgmap v722: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:22:11.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:11 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2924088303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:22:11.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:11 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2538029750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:22:11.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:11 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3701111886' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:22:12.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:11 smithi190 ceph-mon[36292]: pgmap v722: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:22:12.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:11 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2924088303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:22:12.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:11 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2538029750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:22:12.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:11 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3701111886' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:22:12.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:12 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2540964737' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T19:22:12.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:12 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2540964737' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T19:22:13.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:12 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2540964737' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T19:22:13.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:13 smithi170 ceph-mon[35690]: pgmap v723: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:22:13.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:13 smithi029 ceph-mon[32439]: pgmap v723: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:22:14.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:13 smithi190 ceph-mon[36292]: pgmap v723: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:22:15.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:15 smithi170 ceph-mon[35690]: pgmap v724: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-11T19:22:15.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:15 smithi029 ceph-mon[32439]: pgmap v724: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-11T19:22:16.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:15 smithi190 ceph-mon[36292]: pgmap v724: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-11T19:22:17.675 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2024-03-11T19:22:17.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:22:17.675 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-11T19:22:17.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:22:17.715 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-03-11T19:22:17.821 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-11T19:22:17.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:22:17.822 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-11T19:22:17.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:17 smithi170 ceph-mon[35690]: pgmap v725: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-11T19:22:17.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:17 smithi029 ceph-mon[32439]: pgmap v725: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-11T19:22:17.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:22:17.853 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2024-03-11T19:22:17.958 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-11T19:22:17.969 INFO:tasks.cephadm:Teardown begin 2024-03-11T19:22:17.969 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-11T19:22:17.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:22:18.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:17 smithi190 ceph-mon[36292]: pgmap v725: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-11T19:22:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.072+0000 7f367330f640 1 -- 172.21.15.29:0/2337619085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f366c1045d0 msgr2=0x7f366c1069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.072+0000 7f367330f640 1 --2- 172.21.15.29:0/2337619085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f366c1045d0 0x7f366c1069c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f3660009900 tx=0x7f366002bc30 comp rx=0 tx=0).stop 2024-03-11T19:22:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.072+0000 7f367330f640 1 -- 172.21.15.29:0/2337619085 shutdown_connections 2024-03-11T19:22:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.072+0000 7f367330f640 1 --2- 172.21.15.29:0/2337619085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f366c106f00 0x7f366c1092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.072+0000 7f367330f640 1 --2- 172.21.15.29:0/2337619085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f366c1045d0 0x7f366c1069c0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.072+0000 7f367330f640 1 --2- 172.21.15.29:0/2337619085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f366c101ca0 0x7f366c104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.072+0000 7f367330f640 1 -- 172.21.15.29:0/2337619085 >> 172.21.15.29:0/2337619085 conn(0x7f366c0fb610 msgr2=0x7f366c0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.072+0000 7f367330f640 1 -- 172.21.15.29:0/2337619085 shutdown_connections 2024-03-11T19:22:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.072+0000 7f367330f640 1 -- 172.21.15.29:0/2337619085 wait complete. 2024-03-11T19:22:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.072+0000 7f367330f640 1 Processor -- start 2024-03-11T19:22:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.072+0000 7f367330f640 1 -- start start 2024-03-11T19:22:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.072+0000 7f367330f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f366c101ca0 0x7f366c1971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.072+0000 7f367330f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f366c1045d0 0x7f366c1976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f367330f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f366c106f00 0x7f366c19e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f367330f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c19ec50 con 0x7f366c106f00 2024-03-11T19:22:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f367330f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c19edc0 con 0x7f366c101ca0 2024-03-11T19:22:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f367330f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c19ef80 con 0x7f366c1045d0 2024-03-11T19:22:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f36718ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f366c106f00 0x7f366c19e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f36718ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f366c106f00 0x7f366c19e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57598/0 (socket says 172.21.15.29:57598) 2024-03-11T19:22:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f36718ac640 1 -- 172.21.15.29:0/3090467502 learned_addr learned my addr 172.21.15.29:0/3090467502 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:22:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f36710ab640 1 --2- 172.21.15.29:0/3090467502 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f366c101ca0 0x7f366c1971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f36708aa640 1 --2- 172.21.15.29:0/3090467502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f366c1045d0 0x7f366c1976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f36718ac640 1 -- 172.21.15.29:0/3090467502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f366c1045d0 msgr2=0x7f366c1976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f36718ac640 1 --2- 172.21.15.29:0/3090467502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f366c1045d0 0x7f366c1976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f36718ac640 1 -- 172.21.15.29:0/3090467502 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f366c101ca0 msgr2=0x7f366c1971b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f36718ac640 1 --2- 172.21.15.29:0/3090467502 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f366c101ca0 0x7f366c1971b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f36718ac640 1 -- 172.21.15.29:0/3090467502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3660009560 con 0x7f366c106f00 2024-03-11T19:22:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f36708aa640 1 --2- 172.21.15.29:0/3090467502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f366c1045d0 0x7f366c1976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:22:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f36710ab640 1 --2- 172.21.15.29:0/3090467502 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f366c101ca0 0x7f366c1971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:22:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f36718ac640 1 --2- 172.21.15.29:0/3090467502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f366c106f00 0x7f366c19e710 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f366400c970 tx=0x7f366400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f366a7fc640 1 -- 172.21.15.29:0/3090467502 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3664007bb0 con 0x7f366c106f00 2024-03-11T19:22:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f367330f640 1 -- 172.21.15.29:0/3090467502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f366c19f260 con 0x7f366c106f00 2024-03-11T19:22:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f366a7fc640 1 -- 172.21.15.29:0/3090467502 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3664007d10 con 0x7f366c106f00 2024-03-11T19:22:18.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f366a7fc640 1 -- 172.21.15.29:0/3090467502 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36640186a0 con 0x7f366c106f00 2024-03-11T19:22:18.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.073+0000 7f367330f640 1 -- 172.21.15.29:0/3090467502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366c19f720 con 0x7f366c106f00 2024-03-11T19:22:18.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.074+0000 7f367330f640 1 -- 172.21.15.29:0/3090467502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f366c0fd100 con 0x7f366c106f00 2024-03-11T19:22:18.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.075+0000 7f366a7fc640 1 -- 172.21.15.29:0/3090467502 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f366400f450 con 0x7f366c106f00 2024-03-11T19:22:18.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.076+0000 7f366a7fc640 1 --2- 172.21.15.29:0/3090467502 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f36540a1a30 0x7f36540a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:18.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.076+0000 7f36710ab640 1 --2- 172.21.15.29:0/3090467502 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f36540a1a30 0x7f36540a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:18.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.076+0000 7f366a7fc640 1 -- 172.21.15.29:0/3090467502 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f36640cd580 con 0x7f366c106f00 2024-03-11T19:22:18.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.076+0000 7f36710ab640 1 --2- 172.21.15.29:0/3090467502 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f36540a1a30 0x7f36540a3ef0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f365c005c40 tx=0x7f365c00a430 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:18.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.080+0000 7f366a7fc640 1 -- 172.21.15.29:0/3090467502 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f36640968c0 con 0x7f366c106f00 2024-03-11T19:22:18.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.303+0000 7f367330f640 1 -- 172.21.15.29:0/3090467502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f366c10ca60 con 0x7f366c106f00 2024-03-11T19:22:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.304+0000 7f366a7fc640 1 -- 172.21.15.29:0/3090467502 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f3664099f80 con 0x7f366c106f00 2024-03-11T19:22:18.308 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:22:18.308 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:22:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:22:18.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.306+0000 7f367330f640 1 -- 172.21.15.29:0/3090467502 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f36540a1a30 msgr2=0x7f36540a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:18.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.306+0000 7f367330f640 1 --2- 172.21.15.29:0/3090467502 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f36540a1a30 0x7f36540a3ef0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f365c005c40 tx=0x7f365c00a430 comp rx=0 tx=0).stop 2024-03-11T19:22:18.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.306+0000 7f367330f640 1 -- 172.21.15.29:0/3090467502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f366c106f00 msgr2=0x7f366c19e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:18.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.306+0000 7f367330f640 1 --2- 172.21.15.29:0/3090467502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f366c106f00 0x7f366c19e710 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f366400c970 tx=0x7f366400ce30 comp rx=0 tx=0).stop 2024-03-11T19:22:18.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.306+0000 7f367330f640 1 -- 172.21.15.29:0/3090467502 shutdown_connections 2024-03-11T19:22:18.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.306+0000 7f367330f640 1 --2- 172.21.15.29:0/3090467502 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f36540a1a30 0x7f36540a3ef0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.306+0000 7f367330f640 1 --2- 172.21.15.29:0/3090467502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f366c106f00 0x7f366c19e710 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.306+0000 7f367330f640 1 --2- 172.21.15.29:0/3090467502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f366c1045d0 0x7f366c1976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.306+0000 7f367330f640 1 --2- 172.21.15.29:0/3090467502 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f366c101ca0 0x7f366c1971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.306+0000 7f367330f640 1 -- 172.21.15.29:0/3090467502 >> 172.21.15.29:0/3090467502 conn(0x7f366c0fb610 msgr2=0x7f366c102760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.306+0000 7f367330f640 1 -- 172.21.15.29:0/3090467502 shutdown_connections 2024-03-11T19:22:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.306+0000 7f367330f640 1 -- 172.21.15.29:0/3090467502 wait complete. 2024-03-11T19:22:18.322 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:22:18.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.423+0000 7f6d690a3640 1 -- 172.21.15.29:0/2348533408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d64101cd0 msgr2=0x7f6d64108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:18.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.423+0000 7f6d690a3640 1 --2- 172.21.15.29:0/2348533408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d64101cd0 0x7f6d64108560 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f6d54009ca0 tx=0x7f6d5402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.423+0000 7f6d690a3640 1 -- 172.21.15.29:0/2348533408 shutdown_connections 2024-03-11T19:22:18.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.423+0000 7f6d690a3640 1 --2- 172.21.15.29:0/2348533408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d64101cd0 0x7f6d64108560 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.423+0000 7f6d690a3640 1 --2- 172.21.15.29:0/2348533408 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d64101310 0x7f6d64101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.423+0000 7f6d690a3640 1 --2- 172.21.15.29:0/2348533408 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d64100120 0x7f6d64100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.423+0000 7f6d690a3640 1 -- 172.21.15.29:0/2348533408 >> 172.21.15.29:0/2348533408 conn(0x7f6d640fb6b0 msgr2=0x7f6d640fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:18.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.423+0000 7f6d690a3640 1 -- 172.21.15.29:0/2348533408 shutdown_connections 2024-03-11T19:22:18.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.423+0000 7f6d690a3640 1 -- 172.21.15.29:0/2348533408 wait complete. 2024-03-11T19:22:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.424+0000 7f6d690a3640 1 Processor -- start 2024-03-11T19:22:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.424+0000 7f6d690a3640 1 -- start start 2024-03-11T19:22:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.424+0000 7f6d690a3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d64100120 0x7f6d64199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.424+0000 7f6d690a3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d64101310 0x7f6d641998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.424+0000 7f6d690a3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d64101cd0 0x7f6d641a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.424+0000 7f6d690a3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d641a0de0 con 0x7f6d64101cd0 2024-03-11T19:22:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.424+0000 7f6d690a3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d641a0f50 con 0x7f6d64101310 2024-03-11T19:22:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.424+0000 7f6d690a3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d641a1110 con 0x7f6d64100120 2024-03-11T19:22:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.424+0000 7f6d6359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d64101cd0 0x7f6d641a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.424+0000 7f6d6359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d64101cd0 0x7f6d641a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57616/0 (socket says 172.21.15.29:57616) 2024-03-11T19:22:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.424+0000 7f6d6359e640 1 -- 172.21.15.29:0/2773701818 learned_addr learned my addr 172.21.15.29:0/2773701818 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:22:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.425+0000 7f6d62d9d640 1 --2- 172.21.15.29:0/2773701818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d64100120 0x7f6d64199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.425+0000 7f6d6259c640 1 --2- 172.21.15.29:0/2773701818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d64101310 0x7f6d641998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.425+0000 7f6d6359e640 1 -- 172.21.15.29:0/2773701818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d64100120 msgr2=0x7f6d64199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:18.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.425+0000 7f6d6359e640 1 --2- 172.21.15.29:0/2773701818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d64100120 0x7f6d64199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.425+0000 7f6d6359e640 1 -- 172.21.15.29:0/2773701818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d64101310 msgr2=0x7f6d641998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:18.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.425+0000 7f6d6359e640 1 --2- 172.21.15.29:0/2773701818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d64101310 0x7f6d641998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.425+0000 7f6d6359e640 1 -- 172.21.15.29:0/2773701818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d54009900 con 0x7f6d64101cd0 2024-03-11T19:22:18.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.425+0000 7f6d6359e640 1 --2- 172.21.15.29:0/2773701818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d64101cd0 0x7f6d641a08a0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f6d54037040 tx=0x7f6d54032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:18.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.425+0000 7f6d4bfff640 1 -- 172.21.15.29:0/2773701818 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d54034e70 con 0x7f6d64101cd0 2024-03-11T19:22:18.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.425+0000 7f6d4bfff640 1 -- 172.21.15.29:0/2773701818 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6d54003e20 con 0x7f6d64101cd0 2024-03-11T19:22:18.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.425+0000 7f6d690a3640 1 -- 172.21.15.29:0/2773701818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d641a1390 con 0x7f6d64101cd0 2024-03-11T19:22:18.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.425+0000 7f6d4bfff640 1 -- 172.21.15.29:0/2773701818 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5403d3f0 con 0x7f6d64101cd0 2024-03-11T19:22:18.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.425+0000 7f6d690a3640 1 -- 172.21.15.29:0/2773701818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d641a1840 con 0x7f6d64101cd0 2024-03-11T19:22:18.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.426+0000 7f6d690a3640 1 -- 172.21.15.29:0/2773701818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d64192ed0 con 0x7f6d64101cd0 2024-03-11T19:22:18.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.427+0000 7f6d4bfff640 1 -- 172.21.15.29:0/2773701818 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6d5403b040 con 0x7f6d64101cd0 2024-03-11T19:22:18.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.428+0000 7f6d4bfff640 1 --2- 172.21.15.29:0/2773701818 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6d4c0a1a10 0x7f6d4c0a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:18.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.428+0000 7f6d62d9d640 1 --2- 172.21.15.29:0/2773701818 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6d4c0a1a10 0x7f6d4c0a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:18.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.428+0000 7f6d4bfff640 1 -- 172.21.15.29:0/2773701818 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f6d540efc60 con 0x7f6d64101cd0 2024-03-11T19:22:18.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.428+0000 7f6d62d9d640 1 --2- 172.21.15.29:0/2773701818 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6d4c0a1a10 0x7f6d4c0a3ed0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c005fd0 tx=0x7f6d5c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:18.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.432+0000 7f6d4bfff640 1 -- 172.21.15.29:0/2773701818 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6d540b8e70 con 0x7f6d64101cd0 2024-03-11T19:22:18.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.655+0000 7f6d690a3640 1 -- 172.21.15.29:0/2773701818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d64192940 con 0x7f6d64101cd0 2024-03-11T19:22:18.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.656+0000 7f6d4bfff640 1 -- 172.21.15.29:0/2773701818 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f6d540bc530 con 0x7f6d64101cd0 2024-03-11T19:22:18.662 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:22:18.662 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/630953303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":630953303},{"type":"v1","addr":"172.21.15.29:6835","nonce":630953303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/368479661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":368479661},{"type":"v1","addr":"172.21.15.29:6837","nonce":368479661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:07:49.123624+0000","modified":"2024-03-11T19:09:55.569644+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24496,"mds_2":24475,"mds_3":24473,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/72765211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":72765211},{"type":"v1","addr":"172.21.15.29:6839","nonce":72765211}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/3154444865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":3154444865},{"type":"v1","addr":"172.21.15.29:6841","nonce":3154444865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/1035391240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1035391240},{"type":"v1","addr":"172.21.15.170:6833","nonce":1035391240}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6833/493725011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":493725011},{"type":"v1","addr":"172.21.15.190:6833","nonce":493725011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2555518515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2555518515},{"type":"v1","addr":"172.21.15.170:6835","nonce":2555518515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.190:6835/1048234253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1048234253},{"type":"v1","addr":"172.21.15.190:6835","nonce":1048234253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24473":{"gid":24473,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/585739236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":585739236},{"type":"v1","addr":"172.21.15.170:6837","nonce":585739236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.190:6837/3595456607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":3595456607},{"type":"v1","addr":"172.21.15.190:6837","nonce":3595456607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.190:6839/2031437522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":2031437522},{"type":"v1","addr":"172.21.15.190:6839","nonce":2031437522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/4145997740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":4145997740},{"type":"v1","addr":"172.21.15.170:6839","nonce":4145997740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:22:18.663 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-03-11T19:22:18.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.658+0000 7f6d690a3640 1 -- 172.21.15.29:0/2773701818 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6d4c0a1a10 msgr2=0x7f6d4c0a3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:18.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.658+0000 7f6d690a3640 1 --2- 172.21.15.29:0/2773701818 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6d4c0a1a10 0x7f6d4c0a3ed0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c005fd0 tx=0x7f6d5c005850 comp rx=0 tx=0).stop 2024-03-11T19:22:18.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.658+0000 7f6d690a3640 1 -- 172.21.15.29:0/2773701818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d64101cd0 msgr2=0x7f6d641a08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:18.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.658+0000 7f6d690a3640 1 --2- 172.21.15.29:0/2773701818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d64101cd0 0x7f6d641a08a0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f6d54037040 tx=0x7f6d54032980 comp rx=0 tx=0).stop 2024-03-11T19:22:18.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.658+0000 7f6d690a3640 1 -- 172.21.15.29:0/2773701818 shutdown_connections 2024-03-11T19:22:18.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.658+0000 7f6d690a3640 1 --2- 172.21.15.29:0/2773701818 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f6d4c0a1a10 0x7f6d4c0a3ed0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.658+0000 7f6d690a3640 1 --2- 172.21.15.29:0/2773701818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d64101cd0 0x7f6d641a08a0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.658+0000 7f6d690a3640 1 --2- 172.21.15.29:0/2773701818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d64101310 0x7f6d641998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.658+0000 7f6d690a3640 1 --2- 172.21.15.29:0/2773701818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d64100120 0x7f6d64199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.658+0000 7f6d690a3640 1 -- 172.21.15.29:0/2773701818 >> 172.21.15.29:0/2773701818 conn(0x7f6d640fb6b0 msgr2=0x7f6d641900a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:18.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.658+0000 7f6d690a3640 1 -- 172.21.15.29:0/2773701818 shutdown_connections 2024-03-11T19:22:18.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.658+0000 7f6d690a3640 1 -- 172.21.15.29:0/2773701818 wait complete. 2024-03-11T19:22:18.673 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:22:18.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.773+0000 7f5fcf1e9640 1 -- 172.21.15.29:0/1588215254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc8106f40 msgr2=0x7f5fc8109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:18.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.773+0000 7f5fcf1e9640 1 --2- 172.21.15.29:0/1588215254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc8106f40 0x7f5fc8109330 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0009ca0 tx=0x7f5fc002c0f0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.774+0000 7f5fcf1e9640 1 -- 172.21.15.29:0/1588215254 shutdown_connections 2024-03-11T19:22:18.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.774+0000 7f5fcf1e9640 1 --2- 172.21.15.29:0/1588215254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc8106f40 0x7f5fc8109330 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.774+0000 7f5fcf1e9640 1 --2- 172.21.15.29:0/1588215254 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fc8104610 0x7f5fc8106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.774+0000 7f5fcf1e9640 1 --2- 172.21.15.29:0/1588215254 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5fc8101ce0 0x7f5fc81040d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.774+0000 7f5fcf1e9640 1 -- 172.21.15.29:0/1588215254 >> 172.21.15.29:0/1588215254 conn(0x7f5fc80fb610 msgr2=0x7f5fc80fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:18.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.774+0000 7f5fcf1e9640 1 -- 172.21.15.29:0/1588215254 shutdown_connections 2024-03-11T19:22:18.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.774+0000 7f5fcf1e9640 1 -- 172.21.15.29:0/1588215254 wait complete. 2024-03-11T19:22:18.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.774+0000 7f5fcf1e9640 1 Processor -- start 2024-03-11T19:22:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.774+0000 7f5fcf1e9640 1 -- start start 2024-03-11T19:22:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.774+0000 7f5fcf1e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc8101ce0 0x7f5fc8101650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.774+0000 7f5fcf1e9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5fc8104610 0x7f5fc80ffc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.774+0000 7f5fcf1e9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fc8106f40 0x7f5fc81001d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.774+0000 7f5fcf1e9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc8100930 con 0x7f5fc8101ce0 2024-03-11T19:22:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.774+0000 7f5fcf1e9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc8100aa0 con 0x7f5fc8106f40 2024-03-11T19:22:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.774+0000 7f5fcf1e9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc81a11c0 con 0x7f5fc8104610 2024-03-11T19:22:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.774+0000 7f5fccf85640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc8101ce0 0x7f5fc8101650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.774+0000 7f5fccf85640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc8101ce0 0x7f5fc8101650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57622/0 (socket says 172.21.15.29:57622) 2024-03-11T19:22:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.774+0000 7f5fccf85640 1 -- 172.21.15.29:0/3010939559 learned_addr learned my addr 172.21.15.29:0/3010939559 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:22:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.775+0000 7f5fc7fff640 1 --2- 172.21.15.29:0/3010939559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5fc8104610 0x7f5fc80ffc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.775+0000 7f5fcd786640 1 --2- 172.21.15.29:0/3010939559 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fc8106f40 0x7f5fc81001d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.775+0000 7f5fccf85640 1 -- 172.21.15.29:0/3010939559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5fc8104610 msgr2=0x7f5fc80ffc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.775+0000 7f5fccf85640 1 --2- 172.21.15.29:0/3010939559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5fc8104610 0x7f5fc80ffc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.775+0000 7f5fccf85640 1 -- 172.21.15.29:0/3010939559 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fc8106f40 msgr2=0x7f5fc81001d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.775+0000 7f5fccf85640 1 --2- 172.21.15.29:0/3010939559 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fc8106f40 0x7f5fc81001d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.775+0000 7f5fccf85640 1 -- 172.21.15.29:0/3010939559 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc0009900 con 0x7f5fc8101ce0 2024-03-11T19:22:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.775+0000 7f5fcd786640 1 --2- 172.21.15.29:0/3010939559 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fc8106f40 0x7f5fc81001d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:22:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.775+0000 7f5fccf85640 1 --2- 172.21.15.29:0/3010939559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc8101ce0 0x7f5fc8101650 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc00d980 tx=0x7f5fbc00de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.775+0000 7f5fc5ffb640 1 -- 172.21.15.29:0/3010939559 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc009a30 con 0x7f5fc8101ce0 2024-03-11T19:22:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.775+0000 7f5fcf1e9640 1 -- 172.21.15.29:0/3010939559 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc81a1470 con 0x7f5fc8101ce0 2024-03-11T19:22:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.775+0000 7f5fc5ffb640 1 -- 172.21.15.29:0/3010939559 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5fbc0044d0 con 0x7f5fc8101ce0 2024-03-11T19:22:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.775+0000 7f5fc5ffb640 1 -- 172.21.15.29:0/3010939559 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc0047a0 con 0x7f5fc8101ce0 2024-03-11T19:22:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.775+0000 7f5fcf1e9640 1 -- 172.21.15.29:0/3010939559 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc81a1980 con 0x7f5fc8101ce0 2024-03-11T19:22:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.776+0000 7f5fcf1e9640 1 -- 172.21.15.29:0/3010939559 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc80fd100 con 0x7f5fc8101ce0 2024-03-11T19:22:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.777+0000 7f5fc5ffb640 1 -- 172.21.15.29:0/3010939559 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5fbc004020 con 0x7f5fc8101ce0 2024-03-11T19:22:18.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.778+0000 7f5fc5ffb640 1 --2- 172.21.15.29:0/3010939559 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5fb80a1a30 0x7f5fb80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:18.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.778+0000 7f5fc7fff640 1 --2- 172.21.15.29:0/3010939559 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5fb80a1a30 0x7f5fb80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:18.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.778+0000 7f5fc5ffb640 1 -- 172.21.15.29:0/3010939559 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7f5fbc0ccfb0 con 0x7f5fc8101ce0 2024-03-11T19:22:18.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.778+0000 7f5fc7fff640 1 --2- 172.21.15.29:0/3010939559 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5fb80a1a30 0x7f5fb80a3ef0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4005fd0 tx=0x7f5fb4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:18.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.782+0000 7f5fc5ffb640 1 -- 172.21.15.29:0/3010939559 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5fbc0961c0 con 0x7f5fc8101ce0 2024-03-11T19:22:18.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:18 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3090467502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:22:18.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:18 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3090467502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:22:18.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.966+0000 7f5fcf1e9640 1 -- 172.21.15.29:0/3010939559 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fc802ce40 con 0x7f5fc8101ce0 2024-03-11T19:22:18.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.967+0000 7f5fc5ffb640 1 -- 172.21.15.29:0/3010939559 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22559 (secure 0 0 0) 0x7f5fbc099880 con 0x7f5fc8101ce0 2024-03-11T19:22:18.972 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-11T19:22:18.972 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":89,"fsid":"638915c2-dfd9-11ee-95c8-87774f69a715","created":"2024-03-11T18:59:36.894847+0000","modified":"2024-03-11T19:08:44.981806+0000","last_up_change":"2024-03-11T19:06:58.567369+0000","last_in_change":"2024-03-11T19:06:38.007384+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:03:09.860019+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:07:47.563184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:07:48.282461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"62b6ad23-8635-47d9-9485-c91c1d1f942d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6803","nonce":827892152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6805","nonce":827892152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6809","nonce":827892152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":827892152},{"type":"v1","addr":"172.21.15.29:6807","nonce":827892152}]},"public_addr":"172.21.15.29:6803/827892152","cluster_addr":"172.21.15.29:6805/827892152","heartbeat_back_addr":"172.21.15.29:6809/827892152","heartbeat_front_addr":"172.21.15.29:6807/827892152","state":["exists","up"]},{"osd":1,"uuid":"4ae8a614-7d37-4e39-b8cc-542895cc5a30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6801","nonce":531888562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6803","nonce":531888562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6807","nonce":531888562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":531888562},{"type":"v1","addr":"172.21.15.170:6805","nonce":531888562}]},"public_addr":"172.21.15.170:6801/531888562","cluster_addr":"172.21.15.170:6803/531888562","heartbeat_back_addr":"172.21.15.170:6807/531888562","heartbeat_front_addr":"172.21.15.170:6805/531888562","state":["exists","up"]},{"osd":2,"uuid":"799a6a3b-63d6-47b8-a88f-c3a08ee03a53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6801","nonce":1010896180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6803","nonce":1010896180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6807","nonce":1010896180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1010896180},{"type":"v1","addr":"172.21.15.190:6805","nonce":1010896180}]},"public_addr":"172.21.15.190:6801/1010896180","cluster_addr":"172.21.15.190:6803/1010896180","heartbeat_back_addr":"172.21.15.190:6807/1010896180","heartbeat_front_addr":"172.21.15.190:6805/1010896180","state":["exists","up"]},{"osd":3,"uuid":"2efb0fb3-e388-49c3-aa14-9ceeb5d4b49e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6811","nonce":2103856055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6813","nonce":2103856055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6817","nonce":2103856055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":2103856055},{"type":"v1","addr":"172.21.15.29:6815","nonce":2103856055}]},"public_addr":"172.21.15.29:6811/2103856055","cluster_addr":"172.21.15.29:6813/2103856055","heartbeat_back_addr":"172.21.15.29:6817/2103856055","heartbeat_front_addr":"172.21.15.29:6815/2103856055","state":["exists","up"]},{"osd":4,"uuid":"8719e9e6-6389-4554-931e-722f213781b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6809","nonce":1979690226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6811","nonce":1979690226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6815","nonce":1979690226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1979690226},{"type":"v1","addr":"172.21.15.170:6813","nonce":1979690226}]},"public_addr":"172.21.15.170:6809/1979690226","cluster_addr":"172.21.15.170:6811/1979690226","heartbeat_back_addr":"172.21.15.170:6815/1979690226","heartbeat_front_addr":"172.21.15.170:6813/1979690226","state":["exists","up"]},{"osd":5,"uuid":"e8072585-c243-4a27-8a7a-b1af399aa467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6809","nonce":318560672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6811","nonce":318560672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6815","nonce":318560672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":318560672},{"type":"v1","addr":"172.21.15.190:6813","nonce":318560672}]},"public_addr":"172.21.15.190:6809/318560672","cluster_addr":"172.21.15.190:6811/318560672","heartbeat_back_addr":"172.21.15.190:6815/318560672","heartbeat_front_addr":"172.21.15.190:6813/318560672","state":["exists","up"]},{"osd":6,"uuid":"6ab9b5a8-7c19-4e5f-b211-6201fee37f11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6819","nonce":1792625214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6821","nonce":1792625214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6825","nonce":1792625214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":1792625214},{"type":"v1","addr":"172.21.15.29:6823","nonce":1792625214}]},"public_addr":"172.21.15.29:6819/1792625214","cluster_addr":"172.21.15.29:6821/1792625214","heartbeat_back_addr":"172.21.15.29:6825/1792625214","heartbeat_front_addr":"172.21.15.29:6823/1792625214","state":["exists","up"]},{"osd":7,"uuid":"f7d84364-5bf1-473c-9676-253fe01bff2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6817","nonce":3259207349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6819","nonce":3259207349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6823","nonce":3259207349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3259207349},{"type":"v1","addr":"172.21.15.170:6821","nonce":3259207349}]},"public_addr":"172.21.15.170:6817/3259207349","cluster_addr":"172.21.15.170:6819/3259207349","heartbeat_back_addr":"172.21.15.170:6823/3259207349","heartbeat_front_addr":"172.21.15.170:6821/3259207349","state":["exists","up"]},{"osd":8,"uuid":"7d27edd6-a75d-4b20-ab1b-dfa99909419d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6817","nonce":1474709011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6819","nonce":1474709011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6823","nonce":1474709011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1474709011},{"type":"v1","addr":"172.21.15.190:6821","nonce":1474709011}]},"public_addr":"172.21.15.190:6817/1474709011","cluster_addr":"172.21.15.190:6819/1474709011","heartbeat_back_addr":"172.21.15.190:6823/1474709011","heartbeat_front_addr":"172.21.15.190:6821/1474709011","state":["exists","up"]},{"osd":9,"uuid":"e593e338-5c8b-4d2e-a990-39fd8c3eff43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6827","nonce":3553345080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6829","nonce":3553345080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6833","nonce":3553345080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":3553345080},{"type":"v1","addr":"172.21.15.29:6831","nonce":3553345080}]},"public_addr":"172.21.15.29:6827/3553345080","cluster_addr":"172.21.15.29:6829/3553345080","heartbeat_back_addr":"172.21.15.29:6833/3553345080","heartbeat_front_addr":"172.21.15.29:6831/3553345080","state":["exists","up"]},{"osd":10,"uuid":"06dde09c-a014-4cac-aa88-352b63ff644b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6825","nonce":3766541425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6827","nonce":3766541425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6831","nonce":3766541425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3766541425},{"type":"v1","addr":"172.21.15.170:6829","nonce":3766541425}]},"public_addr":"172.21.15.170:6825/3766541425","cluster_addr":"172.21.15.170:6827/3766541425","heartbeat_back_addr":"172.21.15.170:6831/3766541425","heartbeat_front_addr":"172.21.15.170:6829/3766541425","state":["exists","up"]},{"osd":11,"uuid":"0cf7ece9-e288-49e6-b833-e9935c3abfc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6825","nonce":3926470159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6827","nonce":3926470159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6831","nonce":3926470159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":3926470159},{"type":"v1","addr":"172.21.15.190:6829","nonce":3926470159}]},"public_addr":"172.21.15.190:6825/3926470159","cluster_addr":"172.21.15.190:6827/3926470159","heartbeat_back_addr":"172.21.15.190:6831/3926470159","heartbeat_front_addr":"172.21.15.190:6829/3926470159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:18.982014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:02:42.104373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:05.578449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:31.119247+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:03:55.966288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:20.879680+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:04:47.101184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:12.801093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:05:37.581818+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:03.914859+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:30.174635+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:06:56.280673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2913819020":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/961617511":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6800/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:6801/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6800/3550575954":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/385573026":"2024-03-12T19:00:15.449726+0000","172.21.15.29:6801/3585665950":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2164849075":"2024-03-12T19:00:15.449726+0000","172.21.15.29:0/1037268269":"2024-03-12T19:00:43.524861+0000","172.21.15.29:0/2099360722":"2024-03-12T19:00:15.449726+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:22:18.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.970+0000 7f5fcf1e9640 1 -- 172.21.15.29:0/3010939559 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5fb80a1a30 msgr2=0x7f5fb80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:18.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.970+0000 7f5fcf1e9640 1 --2- 172.21.15.29:0/3010939559 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5fb80a1a30 0x7f5fb80a3ef0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4005fd0 tx=0x7f5fb4005850 comp rx=0 tx=0).stop 2024-03-11T19:22:18.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.970+0000 7f5fcf1e9640 1 -- 172.21.15.29:0/3010939559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc8101ce0 msgr2=0x7f5fc8101650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:18.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.970+0000 7f5fcf1e9640 1 --2- 172.21.15.29:0/3010939559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc8101ce0 0x7f5fc8101650 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc00d980 tx=0x7f5fbc00de40 comp rx=0 tx=0).stop 2024-03-11T19:22:18.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.970+0000 7f5fcf1e9640 1 -- 172.21.15.29:0/3010939559 shutdown_connections 2024-03-11T19:22:18.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.970+0000 7f5fcf1e9640 1 --2- 172.21.15.29:0/3010939559 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fc8106f40 0x7f5fc81001d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.970+0000 7f5fcf1e9640 1 --2- 172.21.15.29:0/3010939559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5fc8104610 0x7f5fc80ffc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.970+0000 7f5fcf1e9640 1 --2- 172.21.15.29:0/3010939559 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f5fb80a1a30 0x7f5fb80a3ef0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.970+0000 7f5fcf1e9640 1 --2- 172.21.15.29:0/3010939559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc8101ce0 0x7f5fc8101650 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:18.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.970+0000 7f5fcf1e9640 1 -- 172.21.15.29:0/3010939559 >> 172.21.15.29:0/3010939559 conn(0x7f5fc80fb610 msgr2=0x7f5fc8102760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:18.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.970+0000 7f5fcf1e9640 1 -- 172.21.15.29:0/3010939559 shutdown_connections 2024-03-11T19:22:18.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:18.970+0000 7f5fcf1e9640 1 -- 172.21.15.29:0/3010939559 wait complete. 2024-03-11T19:22:18.985 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-11T19:22:19.012 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:18 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3090467502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:22:19.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.088+0000 7fc8797b6640 1 -- 172.21.15.29:0/2138366405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874106ec0 msgr2=0x7fc8741092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:19.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.088+0000 7fc8797b6640 1 --2- 172.21.15.29:0/2138366405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874106ec0 0x7fc8741092b0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fc86c009ca0 tx=0x7fc86c02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:22:19.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.088+0000 7fc8797b6640 1 -- 172.21.15.29:0/2138366405 shutdown_connections 2024-03-11T19:22:19.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.088+0000 7fc8797b6640 1 --2- 172.21.15.29:0/2138366405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874106ec0 0x7fc8741092b0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:19.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.088+0000 7fc8797b6640 1 --2- 172.21.15.29:0/2138366405 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc874104590 0x7fc874106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:19.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.088+0000 7fc8797b6640 1 --2- 172.21.15.29:0/2138366405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc874101c60 0x7fc874104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:19.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.088+0000 7fc8797b6640 1 -- 172.21.15.29:0/2138366405 >> 172.21.15.29:0/2138366405 conn(0x7fc8740fb610 msgr2=0x7fc8740fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:19.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.088+0000 7fc8797b6640 1 -- 172.21.15.29:0/2138366405 shutdown_connections 2024-03-11T19:22:19.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.088+0000 7fc8797b6640 1 -- 172.21.15.29:0/2138366405 wait complete. 2024-03-11T19:22:19.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc8797b6640 1 Processor -- start 2024-03-11T19:22:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc8797b6640 1 -- start start 2024-03-11T19:22:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc8797b6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874101c60 0x7fc874197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc8797b6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc874104590 0x7fc874197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc8797b6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc874106ec0 0x7fc87419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc8797b6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87419ec10 con 0x7fc874101c60 2024-03-11T19:22:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc8797b6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87419ed80 con 0x7fc874104590 2024-03-11T19:22:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc8797b6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87419ef40 con 0x7fc874106ec0 2024-03-11T19:22:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc872ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874101c60 0x7fc874197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc872ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874101c60 0x7fc874197120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57636/0 (socket says 172.21.15.29:57636) 2024-03-11T19:22:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc872ffd640 1 -- 172.21.15.29:0/4053220720 learned_addr learned my addr 172.21.15.29:0/4053220720 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:22:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc8727fc640 1 --2- 172.21.15.29:0/4053220720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc874104590 0x7fc874197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc8737fe640 1 --2- 172.21.15.29:0/4053220720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc874106ec0 0x7fc87419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc872ffd640 1 -- 172.21.15.29:0/4053220720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc874106ec0 msgr2=0x7fc87419e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc872ffd640 1 --2- 172.21.15.29:0/4053220720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc874106ec0 0x7fc87419e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc872ffd640 1 -- 172.21.15.29:0/4053220720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc874104590 msgr2=0x7fc874197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:19.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc872ffd640 1 --2- 172.21.15.29:0/4053220720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc874104590 0x7fc874197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:19.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc872ffd640 1 -- 172.21.15.29:0/4053220720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc864009490 con 0x7fc874101c60 2024-03-11T19:22:19.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc8737fe640 1 --2- 172.21.15.29:0/4053220720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc874106ec0 0x7fc87419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:22:19.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.089+0000 7fc8727fc640 1 --2- 172.21.15.29:0/4053220720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc874104590 0x7fc874197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:22:19.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.090+0000 7fc872ffd640 1 --2- 172.21.15.29:0/4053220720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874101c60 0x7fc874197120 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fc86400ea60 tx=0x7fc86400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:19.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.090+0000 7fc85bfff640 1 -- 172.21.15.29:0/4053220720 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86400cde0 con 0x7fc874101c60 2024-03-11T19:22:19.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.090+0000 7fc8797b6640 1 -- 172.21.15.29:0/4053220720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc86c009900 con 0x7fc874101c60 2024-03-11T19:22:19.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.090+0000 7fc85bfff640 1 -- 172.21.15.29:0/4053220720 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc8640044d0 con 0x7fc874101c60 2024-03-11T19:22:19.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.090+0000 7fc8797b6640 1 -- 172.21.15.29:0/4053220720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc87419f5f0 con 0x7fc874101c60 2024-03-11T19:22:19.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.090+0000 7fc85bfff640 1 -- 172.21.15.29:0/4053220720 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc864010690 con 0x7fc874101c60 2024-03-11T19:22:19.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.091+0000 7fc8797b6640 1 -- 172.21.15.29:0/4053220720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8740fd100 con 0x7fc874101c60 2024-03-11T19:22:19.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.092+0000 7fc85bfff640 1 -- 172.21.15.29:0/4053220720 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc864003e60 con 0x7fc874101c60 2024-03-11T19:22:19.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.092+0000 7fc85bfff640 1 --2- 172.21.15.29:0/4053220720 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc85c0a1a30 0x7fc85c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:19.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.093+0000 7fc85bfff640 1 -- 172.21.15.29:0/4053220720 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8153+0+0 (secure 0 0 0) 0x7fc864014070 con 0x7fc874101c60 2024-03-11T19:22:19.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.093+0000 7fc8727fc640 1 --2- 172.21.15.29:0/4053220720 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc85c0a1a30 0x7fc85c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:19.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.093+0000 7fc8727fc640 1 --2- 172.21.15.29:0/4053220720 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc85c0a1a30 0x7fc85c0a3ef0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc874198630 tx=0x7fc868005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:19.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.097+0000 7fc85bfff640 1 -- 172.21.15.29:0/4053220720 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc864096b90 con 0x7fc874101c60 2024-03-11T19:22:19.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.318+0000 7fc8797b6640 1 -- 172.21.15.29:0/4053220720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc87402cda0 con 0x7fc874101c60 2024-03-11T19:22:19.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.321+0000 7fc85bfff640 1 -- 172.21.15.29:0/4053220720 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v28) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc86409a250 con 0x7fc874101c60 2024-03-11T19:22:19.325 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-11T19:22:19.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.324+0000 7fc8797b6640 1 -- 172.21.15.29:0/4053220720 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc85c0a1a30 msgr2=0x7fc85c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:19.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.324+0000 7fc8797b6640 1 --2- 172.21.15.29:0/4053220720 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc85c0a1a30 0x7fc85c0a3ef0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc874198630 tx=0x7fc868005f90 comp rx=0 tx=0).stop 2024-03-11T19:22:19.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.324+0000 7fc8797b6640 1 -- 172.21.15.29:0/4053220720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874101c60 msgr2=0x7fc874197120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:19.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.324+0000 7fc8797b6640 1 --2- 172.21.15.29:0/4053220720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874101c60 0x7fc874197120 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fc86400ea60 tx=0x7fc86400ef20 comp rx=0 tx=0).stop 2024-03-11T19:22:19.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.325+0000 7fc8797b6640 1 -- 172.21.15.29:0/4053220720 shutdown_connections 2024-03-11T19:22:19.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.325+0000 7fc8797b6640 1 --2- 172.21.15.29:0/4053220720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc874106ec0 0x7fc87419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:19.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.325+0000 7fc8797b6640 1 --2- 172.21.15.29:0/4053220720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc874104590 0x7fc874197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:19.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.325+0000 7fc8797b6640 1 --2- 172.21.15.29:0/4053220720 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7fc85c0a1a30 0x7fc85c0a3ef0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:19.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.325+0000 7fc8797b6640 1 --2- 172.21.15.29:0/4053220720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc874101c60 0x7fc874197120 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:19.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.325+0000 7fc8797b6640 1 -- 172.21.15.29:0/4053220720 >> 172.21.15.29:0/4053220720 conn(0x7fc8740fb610 msgr2=0x7fc8741025a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:19.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.325+0000 7fc8797b6640 1 -- 172.21.15.29:0/4053220720 shutdown_connections 2024-03-11T19:22:19.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.325+0000 7fc8797b6640 1 -- 172.21.15.29:0/4053220720 wait complete. 2024-03-11T19:22:19.346 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-11T19:22:19.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.469+0000 7f36b7977640 1 -- 172.21.15.29:0/4164899673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b0101270 msgr2=0x7f36b01016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:19.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.469+0000 7f36b7977640 1 --2- 172.21.15.29:0/4164899673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b0101270 0x7f36b01016f0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009900 tx=0x7f36a002bc30 comp rx=0 tx=0).stop 2024-03-11T19:22:19.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.469+0000 7f36b7977640 1 -- 172.21.15.29:0/4164899673 shutdown_connections 2024-03-11T19:22:19.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.469+0000 7f36b7977640 1 --2- 172.21.15.29:0/4164899673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36b0101c30 0x7f36b01084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:19.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.469+0000 7f36b7977640 1 --2- 172.21.15.29:0/4164899673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b0101270 0x7f36b01016f0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:19.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.469+0000 7f36b7977640 1 --2- 172.21.15.29:0/4164899673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36b0100080 0x7f36b01004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:19.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.469+0000 7f36b7977640 1 -- 172.21.15.29:0/4164899673 >> 172.21.15.29:0/4164899673 conn(0x7f36b00fb610 msgr2=0x7f36b00fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:19.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.469+0000 7f36b7977640 1 -- 172.21.15.29:0/4164899673 shutdown_connections 2024-03-11T19:22:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.469+0000 7f36b7977640 1 -- 172.21.15.29:0/4164899673 wait complete. 2024-03-11T19:22:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.470+0000 7f36b7977640 1 Processor -- start 2024-03-11T19:22:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.470+0000 7f36b7977640 1 -- start start 2024-03-11T19:22:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.470+0000 7f36b7977640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36b0100080 0x7f36b010a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.470+0000 7f36b7977640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36b0101270 0x7f36b0108d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.470+0000 7f36b7977640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b0101c30 0x7f36b0109240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.470+0000 7f36b7977640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b01099a0 con 0x7f36b0101c30 2024-03-11T19:22:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.470+0000 7f36b7977640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b0109b10 con 0x7f36b0100080 2024-03-11T19:22:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.470+0000 7f36b7977640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b01a1200 con 0x7f36b0101270 2024-03-11T19:22:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.470+0000 7f36b5f14640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b0101c30 0x7f36b0109240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.470+0000 7f36b4f12640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36b0101270 0x7f36b0108d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.470+0000 7f36b5f14640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b0101c30 0x7f36b0109240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57688/0 (socket says 172.21.15.29:57688) 2024-03-11T19:22:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.470+0000 7f36b5f14640 1 -- 172.21.15.29:0/3480897165 learned_addr learned my addr 172.21.15.29:0/3480897165 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:22:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.470+0000 7f36b5713640 1 --2- 172.21.15.29:0/3480897165 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36b0100080 0x7f36b010a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.470+0000 7f36b5f14640 1 -- 172.21.15.29:0/3480897165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36b0101270 msgr2=0x7f36b0108d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.470+0000 7f36b5f14640 1 --2- 172.21.15.29:0/3480897165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36b0101270 0x7f36b0108d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.470+0000 7f36b5f14640 1 -- 172.21.15.29:0/3480897165 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36b0100080 msgr2=0x7f36b010a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.470+0000 7f36b5f14640 1 --2- 172.21.15.29:0/3480897165 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36b0100080 0x7f36b010a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.470+0000 7f36b5f14640 1 -- 172.21.15.29:0/3480897165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a0009560 con 0x7f36b0101c30 2024-03-11T19:22:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.470+0000 7f36b5f14640 1 --2- 172.21.15.29:0/3480897165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b0101c30 0x7f36b0109240 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f36a800c9a0 tx=0x7f36a800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.471+0000 7f36a67fc640 1 -- 172.21.15.29:0/3480897165 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a8007bb0 con 0x7f36b0101c30 2024-03-11T19:22:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.471+0000 7f36a67fc640 1 -- 172.21.15.29:0/3480897165 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f36a8007d10 con 0x7f36b0101c30 2024-03-11T19:22:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.471+0000 7f36b7977640 1 -- 172.21.15.29:0/3480897165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b01a14b0 con 0x7f36b0101c30 2024-03-11T19:22:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.471+0000 7f36a67fc640 1 -- 172.21.15.29:0/3480897165 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a8004e10 con 0x7f36b0101c30 2024-03-11T19:22:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.471+0000 7f36b7977640 1 -- 172.21.15.29:0/3480897165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b01a1940 con 0x7f36b0101c30 2024-03-11T19:22:19.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.472+0000 7f36b7977640 1 -- 172.21.15.29:0/3480897165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b01074c0 con 0x7f36b0101c30 2024-03-11T19:22:19.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.473+0000 7f36a67fc640 1 -- 172.21.15.29:0/3480897165 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f36a8004f70 con 0x7f36b0101c30 2024-03-11T19:22:19.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.474+0000 7f36a67fc640 1 --2- 172.21.15.29:0/3480897165 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f369c0a1a30 0x7f369c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:19.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.474+0000 7f36a67fc640 1 -- 172.21.15.29:0/3480897165 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8583+0+0 (secure 0 0 0) 0x7f36a80cd5f0 con 0x7f36b0101c30 2024-03-11T19:22:19.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.474+0000 7f36b5713640 1 --2- 172.21.15.29:0/3480897165 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f369c0a1a30 0x7f369c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:19.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.475+0000 7f36b5713640 1 --2- 172.21.15.29:0/3480897165 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f369c0a1a30 0x7f369c0a3ef0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f36ac005c40 tx=0x7f36ac00a590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:19.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.478+0000 7f36a67fc640 1 -- 172.21.15.29:0/3480897165 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f36a8096750 con 0x7f36b0101c30 2024-03-11T19:22:19.656 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 11 19:22:19 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-a[74043]: ignoring --setuser ceph since I am not root 2024-03-11T19:22:19.657 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 11 19:22:19 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-a[74043]: ignoring --setgroup ceph since I am not root 2024-03-11T19:22:19.657 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 11 19:22:19 smithi029 ceph-mds[74047]: -- 172.21.15.29:0/3284360568 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e8898221a0 con 0x55e8897f8c00 2024-03-11T19:22:19.657 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 11 19:22:19 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-a[74043]: starting mds.a at 2024-03-11T19:22:19.657 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 11 19:22:19 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-g[73682]: ignoring --setuser ceph since I am not root 2024-03-11T19:22:19.657 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 11 19:22:19 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-g[73682]: ignoring --setgroup ceph since I am not root 2024-03-11T19:22:19.657 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 11 19:22:19 smithi029 ceph-mds[73686]: -- 172.21.15.29:0/3649684760 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557a7abe21a0 con 0x557a7abb9000 2024-03-11T19:22:19.657 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 11 19:22:19 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-g[73682]: starting mds.g at 2024-03-11T19:22:19.658 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:19 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-a[32435]: 2024-03-11T19:22:19.319+0000 7f700a141700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T19:22:19.658 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:19 smithi029 ceph-mon[32439]: pgmap v726: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 938 B/s wr, 12 op/s 2024-03-11T19:22:19.658 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:19 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2773701818' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:22:19.658 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:19 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3010939559' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:22:19.658 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:19 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/4053220720' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-11T19:22:19.658 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:19 smithi029 ceph-mon[32439]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-11T19:22:19.658 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:19 smithi029 ceph-mon[32439]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T19:22:19.658 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:19 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/4053220720' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-11T19:22:19.658 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:19 smithi029 ceph-mon[32439]: osdmap e90: 12 total, 12 up, 12 in 2024-03-11T19:22:19.658 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:19 smithi029 ceph-mon[32439]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-11T19:22:19.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:19.700+0000 7f36b7977640 1 -- 172.21.15.29:0/3480897165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f36b002ce40 con 0x7f36b0101c30 2024-03-11T19:22:19.763 INFO:journalctl@ceph.mds.c.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-c[66697]: ignoring --setuser ceph since I am not root 2024-03-11T19:22:19.763 INFO:journalctl@ceph.mds.c.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-c[66697]: ignoring --setgroup ceph since I am not root 2024-03-11T19:22:19.764 INFO:journalctl@ceph.mds.c.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-mds[66701]: -- 172.21.15.190:0/2327727285 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5559828d21a0 con 0x5559828a8800 2024-03-11T19:22:19.764 INFO:journalctl@ceph.mds.c.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-c[66697]: starting mds.c at 2024-03-11T19:22:19.764 INFO:journalctl@ceph.mds.i.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-i[67005]: ignoring --setuser ceph since I am not root 2024-03-11T19:22:19.764 INFO:journalctl@ceph.mds.i.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-i[67005]: ignoring --setgroup ceph since I am not root 2024-03-11T19:22:19.765 INFO:journalctl@ceph.mds.i.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-mds[67009]: -- 172.21.15.190:0/3345514563 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5635630601a0 con 0x563563036800 2024-03-11T19:22:19.765 INFO:journalctl@ceph.mds.i.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-i[67005]: starting mds.i at 2024-03-11T19:22:19.765 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-l[66073]: ignoring --setuser ceph since I am not root 2024-03-11T19:22:19.766 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-l[66073]: ignoring --setgroup ceph since I am not root 2024-03-11T19:22:19.766 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-mds[66077]: -- 172.21.15.190:0/3106048183 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c15c8621a0 con 0x55c15c838800 2024-03-11T19:22:19.766 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-l[66073]: starting mds.l at 2024-03-11T19:22:19.766 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-mon[36292]: pgmap v726: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 938 B/s wr, 12 op/s 2024-03-11T19:22:19.767 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2773701818' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:22:19.767 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3010939559' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:22:19.767 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/4053220720' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-11T19:22:19.767 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-mon[36292]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-11T19:22:19.767 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-mon[36292]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T19:22:19.767 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/4053220720' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-11T19:22:19.768 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-mon[36292]: osdmap e90: 12 total, 12 up, 12 in 2024-03-11T19:22:19.768 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-mon[36292]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-11T19:22:19.768 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-f[66392]: ignoring --setuser ceph since I am not root 2024-03-11T19:22:19.768 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-f[66392]: ignoring --setgroup ceph since I am not root 2024-03-11T19:22:19.768 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-mds[66396]: -- 172.21.15.190:0/646276244 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563d133781a0 con 0x563d1334e800 2024-03-11T19:22:19.769 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-f[66392]: starting mds.f at 2024-03-11T19:22:19.769 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:22:19 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-f[66392]: 2024-03-11T19:22:19.369+0000 7f0f6e8e7700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-03-11T19:22:19.847 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-b[66380]: ignoring --setuser ceph since I am not root 2024-03-11T19:22:19.847 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-b[66380]: ignoring --setgroup ceph since I am not root 2024-03-11T19:22:19.847 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-mds[66384]: -- 172.21.15.170:0/888898266 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5608fbb1e1a0 con 0x5608fbaf4c00 2024-03-11T19:22:19.847 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-b[66380]: starting mds.b at 2024-03-11T19:22:19.848 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-mon[35690]: pgmap v726: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 938 B/s wr, 12 op/s 2024-03-11T19:22:19.848 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2773701818' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:22:19.848 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3010939559' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:22:19.848 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/4053220720' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-11T19:22:19.849 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-mon[35690]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-11T19:22:19.849 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-mon[35690]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T19:22:19.849 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/4053220720' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-11T19:22:19.849 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-mon[35690]: osdmap e90: 12 total, 12 up, 12 in 2024-03-11T19:22:19.849 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-mon[35690]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-11T19:22:19.850 INFO:journalctl@ceph.mds.h.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-h[66069]: ignoring --setuser ceph since I am not root 2024-03-11T19:22:19.850 INFO:journalctl@ceph.mds.h.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-h[66069]: ignoring --setgroup ceph since I am not root 2024-03-11T19:22:19.850 INFO:journalctl@ceph.mds.h.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-mds[66073]: -- 172.21.15.170:0/3585296928 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d8711de1a0 con 0x55d8711b4c00 2024-03-11T19:22:19.850 INFO:journalctl@ceph.mds.h.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-h[66069]: starting mds.h at 2024-03-11T19:22:19.851 INFO:journalctl@ceph.mds.e.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-e[65760]: ignoring --setuser ceph since I am not root 2024-03-11T19:22:19.851 INFO:journalctl@ceph.mds.e.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-e[65760]: ignoring --setgroup ceph since I am not root 2024-03-11T19:22:19.851 INFO:journalctl@ceph.mds.e.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-mds[65764]: -- 172.21.15.170:0/1187341457 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e0753061a0 con 0x55e0752dcc00 2024-03-11T19:22:19.851 INFO:journalctl@ceph.mds.e.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-e[65760]: starting mds.e at 2024-03-11T19:22:19.852 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-k[65444]: ignoring --setuser ceph since I am not root 2024-03-11T19:22:19.852 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-k[65444]: ignoring --setgroup ceph since I am not root 2024-03-11T19:22:19.852 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-mds[65448]: -- 172.21.15.170:0/1133994530 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560f801661a0 con 0x560f8013cc00 2024-03-11T19:22:19.852 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:22:19 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-k[65444]: starting mds.k at 2024-03-11T19:22:20.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.322+0000 7f36a67fc640 1 -- 172.21.15.29:0/3480897165 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f36a8099e10 con 0x7f36b0101c30 2024-03-11T19:22:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.325+0000 7f36b7977640 1 -- 172.21.15.29:0/3480897165 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f369c0a1a30 msgr2=0x7f369c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.325+0000 7f36b7977640 1 --2- 172.21.15.29:0/3480897165 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f369c0a1a30 0x7f369c0a3ef0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f36ac005c40 tx=0x7f36ac00a590 comp rx=0 tx=0).stop 2024-03-11T19:22:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.325+0000 7f36b7977640 1 -- 172.21.15.29:0/3480897165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b0101c30 msgr2=0x7f36b0109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.325+0000 7f36b7977640 1 --2- 172.21.15.29:0/3480897165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b0101c30 0x7f36b0109240 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f36a800c9a0 tx=0x7f36a800ce60 comp rx=0 tx=0).stop 2024-03-11T19:22:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.325+0000 7f36b7977640 1 -- 172.21.15.29:0/3480897165 shutdown_connections 2024-03-11T19:22:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.325+0000 7f36b7977640 1 --2- 172.21.15.29:0/3480897165 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f369c0a1a30 0x7f369c0a3ef0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.325+0000 7f36b7977640 1 --2- 172.21.15.29:0/3480897165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b0101c30 0x7f36b0109240 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.325+0000 7f36b7977640 1 --2- 172.21.15.29:0/3480897165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36b0101270 0x7f36b0108d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.325+0000 7f36b7977640 1 --2- 172.21.15.29:0/3480897165 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36b0100080 0x7f36b010a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.325+0000 7f36b7977640 1 -- 172.21.15.29:0/3480897165 >> 172.21.15.29:0/3480897165 conn(0x7f36b00fb610 msgr2=0x7f36b0104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.325+0000 7f36b7977640 1 -- 172.21.15.29:0/3480897165 shutdown_connections 2024-03-11T19:22:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.325+0000 7f36b7977640 1 -- 172.21.15.29:0/3480897165 wait complete. 2024-03-11T19:22:20.344 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-11T19:22:20.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.448+0000 7f7e8fb9a640 1 -- 172.21.15.29:0/4256884090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e88106e80 msgr2=0x7f7e881092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:20.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.448+0000 7f7e8fb9a640 1 --2- 172.21.15.29:0/4256884090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e88106e80 0x7f7e881092d0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f7e80009ca0 tx=0x7f7e8002c0f0 comp rx=0 tx=0).stop 2024-03-11T19:22:20.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.448+0000 7f7e8fb9a640 1 -- 172.21.15.29:0/4256884090 shutdown_connections 2024-03-11T19:22:20.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.448+0000 7f7e8fb9a640 1 --2- 172.21.15.29:0/4256884090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e88106e80 0x7f7e881092d0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:20.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.448+0000 7f7e8fb9a640 1 --2- 172.21.15.29:0/4256884090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e88104550 0x7f7e88106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:20.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.448+0000 7f7e8fb9a640 1 --2- 172.21.15.29:0/4256884090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e88101c20 0x7f7e88104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:20.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.448+0000 7f7e8fb9a640 1 -- 172.21.15.29:0/4256884090 >> 172.21.15.29:0/4256884090 conn(0x7f7e880fb630 msgr2=0x7f7e880fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:20.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.448+0000 7f7e8fb9a640 1 -- 172.21.15.29:0/4256884090 shutdown_connections 2024-03-11T19:22:20.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.448+0000 7f7e8fb9a640 1 -- 172.21.15.29:0/4256884090 wait complete. 2024-03-11T19:22:20.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.449+0000 7f7e8fb9a640 1 Processor -- start 2024-03-11T19:22:20.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.449+0000 7f7e8fb9a640 1 -- start start 2024-03-11T19:22:20.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.449+0000 7f7e8fb9a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e88101c20 0x7f7e88101590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:20.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.449+0000 7f7e8fb9a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e88104550 0x7f7e880ffbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:20.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.449+0000 7f7e8fb9a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e88106e80 0x7f7e88100110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:20.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.449+0000 7f7e8fb9a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e88100870 con 0x7f7e88101c20 2024-03-11T19:22:20.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.449+0000 7f7e8fb9a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e881009e0 con 0x7f7e88104550 2024-03-11T19:22:20.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.449+0000 7f7e8fb9a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e881a11a0 con 0x7f7e88106e80 2024-03-11T19:22:20.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.449+0000 7f7e8d936640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e88101c20 0x7f7e88101590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:20.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.449+0000 7f7e8d936640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e88101c20 0x7f7e88101590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57712/0 (socket says 172.21.15.29:57712) 2024-03-11T19:22:20.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.449+0000 7f7e8d936640 1 -- 172.21.15.29:0/2227543976 learned_addr learned my addr 172.21.15.29:0/2227543976 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:22:20.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.449+0000 7f7e8d135640 1 --2- 172.21.15.29:0/2227543976 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e88104550 0x7f7e880ffbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:20.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.449+0000 7f7e8d936640 1 -- 172.21.15.29:0/2227543976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e88106e80 msgr2=0x7f7e88100110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:20.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.449+0000 7f7e8e137640 1 --2- 172.21.15.29:0/2227543976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e88106e80 0x7f7e88100110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:20.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.449+0000 7f7e8d936640 1 --2- 172.21.15.29:0/2227543976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e88106e80 0x7f7e88100110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:20.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.449+0000 7f7e8d936640 1 -- 172.21.15.29:0/2227543976 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e88104550 msgr2=0x7f7e880ffbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:20.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.449+0000 7f7e8d936640 1 --2- 172.21.15.29:0/2227543976 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e88104550 0x7f7e880ffbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:20.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.449+0000 7f7e8d936640 1 -- 172.21.15.29:0/2227543976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e80009900 con 0x7f7e88101c20 2024-03-11T19:22:20.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.449+0000 7f7e8d135640 1 --2- 172.21.15.29:0/2227543976 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e88104550 0x7f7e880ffbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:22:20.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.450+0000 7f7e8d936640 1 --2- 172.21.15.29:0/2227543976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e88101c20 0x7f7e88101590 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f7e8400c9a0 tx=0x7f7e8400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:20.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.450+0000 7f7e7effd640 1 -- 172.21.15.29:0/2227543976 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e84007bb0 con 0x7f7e88101c20 2024-03-11T19:22:20.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.450+0000 7f7e8fb9a640 1 -- 172.21.15.29:0/2227543976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e881a1450 con 0x7f7e88101c20 2024-03-11T19:22:20.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.450+0000 7f7e7effd640 1 -- 172.21.15.29:0/2227543976 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7e84007d10 con 0x7f7e88101c20 2024-03-11T19:22:20.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.450+0000 7f7e7effd640 1 -- 172.21.15.29:0/2227543976 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e84004e80 con 0x7f7e88101c20 2024-03-11T19:22:20.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.450+0000 7f7e8fb9a640 1 -- 172.21.15.29:0/2227543976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e881a1910 con 0x7f7e88101c20 2024-03-11T19:22:20.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.451+0000 7f7e8fb9a640 1 -- 172.21.15.29:0/2227543976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e880fd120 con 0x7f7e88101c20 2024-03-11T19:22:20.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.452+0000 7f7e7effd640 1 -- 172.21.15.29:0/2227543976 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7e840046c0 con 0x7f7e88101c20 2024-03-11T19:22:20.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.452+0000 7f7e7effd640 1 --2- 172.21.15.29:0/2227543976 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7e740a1a30 0x7f7e740a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:20.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.453+0000 7f7e8d135640 1 --2- 172.21.15.29:0/2227543976 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7e740a1a30 0x7f7e740a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:20.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.453+0000 7f7e7effd640 1 -- 172.21.15.29:0/2227543976 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8583+0+0 (secure 0 0 0) 0x7f7e840cd380 con 0x7f7e88101c20 2024-03-11T19:22:20.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.453+0000 7f7e8d135640 1 --2- 172.21.15.29:0/2227543976 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7e740a1a30 0x7f7e740a3ef0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7e88101300 tx=0x7f7e78005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:20.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.457+0000 7f7e7effd640 1 -- 172.21.15.29:0/2227543976 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7e84096430 con 0x7f7e88101c20 2024-03-11T19:22:20.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:20.645+0000 7f7e8fb9a640 1 -- 172.21.15.29:0/2227543976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7e8810ca60 con 0x7f7e88101c20 2024-03-11T19:22:20.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3480897165' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T19:22:20.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-11T19:22:20.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-11T19:22:20.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: Cluster is now healthy 2024-03-11T19:22:20.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3480897165' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-11T19:22:20.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.190:6833/564696652,v1:172.21.15.190:6835/564696652] up:boot 2024-03-11T19:22:20.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.170:6834/1876847968,v1:172.21.15.170:6835/1876847968] up:boot 2024-03-11T19:22:20.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.190:6832/3822038294,v1:172.21.15.190:6834/3822038294] up:boot 2024-03-11T19:22:20.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.170:6832/4059280164,v1:172.21.15.170:6833/4059280164] up:boot 2024-03-11T19:22:20.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.170:6836/1935977436,v1:172.21.15.170:6837/1935977436] up:boot 2024-03-11T19:22:20.848 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.170:6838/1815258142,v1:172.21.15.170:6839/1815258142] up:boot 2024-03-11T19:22:20.848 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.190:6837/3806586879,v1:172.21.15.190:6839/3806586879] up:boot 2024-03-11T19:22:20.848 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.190:6836/2334036684,v1:172.21.15.190:6838/2334036684] up:boot 2024-03-11T19:22:20.848 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.29:6838/2751195379,v1:172.21.15.29:6839/2751195379] up:boot 2024-03-11T19:22:20.848 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: mds.? [v2:172.21.15.29:6840/1834428861,v1:172.21.15.29:6841/1834428861] up:boot 2024-03-11T19:22:20.848 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: fsmap 12 up:standby 2024-03-11T19:22:20.848 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T19:22:20.848 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T19:22:20.848 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T19:22:20.848 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T19:22:20.848 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T19:22:20.848 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T19:22:20.848 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-11T19:22:20.848 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T19:22:20.849 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T19:22:20.849 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:20 smithi170 ceph-mon[35690]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T19:22:20.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3480897165' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T19:22:20.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-11T19:22:20.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-11T19:22:20.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: Cluster is now healthy 2024-03-11T19:22:20.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3480897165' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-11T19:22:20.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.190:6833/564696652,v1:172.21.15.190:6835/564696652] up:boot 2024-03-11T19:22:20.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.170:6834/1876847968,v1:172.21.15.170:6835/1876847968] up:boot 2024-03-11T19:22:20.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.190:6832/3822038294,v1:172.21.15.190:6834/3822038294] up:boot 2024-03-11T19:22:20.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.170:6832/4059280164,v1:172.21.15.170:6833/4059280164] up:boot 2024-03-11T19:22:20.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.170:6836/1935977436,v1:172.21.15.170:6837/1935977436] up:boot 2024-03-11T19:22:20.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.170:6838/1815258142,v1:172.21.15.170:6839/1815258142] up:boot 2024-03-11T19:22:20.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.190:6837/3806586879,v1:172.21.15.190:6839/3806586879] up:boot 2024-03-11T19:22:20.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.190:6836/2334036684,v1:172.21.15.190:6838/2334036684] up:boot 2024-03-11T19:22:20.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.29:6838/2751195379,v1:172.21.15.29:6839/2751195379] up:boot 2024-03-11T19:22:20.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: mds.? [v2:172.21.15.29:6840/1834428861,v1:172.21.15.29:6841/1834428861] up:boot 2024-03-11T19:22:20.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: fsmap 12 up:standby 2024-03-11T19:22:20.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T19:22:20.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T19:22:20.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T19:22:20.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T19:22:20.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T19:22:20.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T19:22:20.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-11T19:22:20.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T19:22:20.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T19:22:20.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:20 smithi029 ceph-mon[32439]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T19:22:21.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3480897165' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T19:22:21.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-11T19:22:21.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-11T19:22:21.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: Cluster is now healthy 2024-03-11T19:22:21.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3480897165' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-11T19:22:21.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.190:6833/564696652,v1:172.21.15.190:6835/564696652] up:boot 2024-03-11T19:22:21.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.170:6834/1876847968,v1:172.21.15.170:6835/1876847968] up:boot 2024-03-11T19:22:21.014 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.190:6832/3822038294,v1:172.21.15.190:6834/3822038294] up:boot 2024-03-11T19:22:21.014 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.170:6832/4059280164,v1:172.21.15.170:6833/4059280164] up:boot 2024-03-11T19:22:21.014 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.170:6836/1935977436,v1:172.21.15.170:6837/1935977436] up:boot 2024-03-11T19:22:21.014 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.170:6838/1815258142,v1:172.21.15.170:6839/1815258142] up:boot 2024-03-11T19:22:21.014 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.190:6837/3806586879,v1:172.21.15.190:6839/3806586879] up:boot 2024-03-11T19:22:21.014 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.190:6836/2334036684,v1:172.21.15.190:6838/2334036684] up:boot 2024-03-11T19:22:21.014 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.29:6838/2751195379,v1:172.21.15.29:6839/2751195379] up:boot 2024-03-11T19:22:21.014 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: mds.? [v2:172.21.15.29:6840/1834428861,v1:172.21.15.29:6841/1834428861] up:boot 2024-03-11T19:22:21.014 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: fsmap 12 up:standby 2024-03-11T19:22:21.014 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T19:22:21.014 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T19:22:21.015 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T19:22:21.015 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T19:22:21.015 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T19:22:21.015 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T19:22:21.015 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-11T19:22:21.015 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T19:22:21.015 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T19:22:21.015 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:20 smithi190 ceph-mon[36292]: from='mgr.14150 172.21.15.29:0/1748031752' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T19:22:21.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.542+0000 7f7e7effd640 1 -- 172.21.15.29:0/2227543976 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v91) v1 ==== 181+0+0 (secure 0 0 0) 0x7f7e84099af0 con 0x7f7e88101c20 2024-03-11T19:22:21.550 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2024-03-11T19:22:21.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.546+0000 7f7e8fb9a640 1 -- 172.21.15.29:0/2227543976 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7e740a1a30 msgr2=0x7f7e740a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:21.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.546+0000 7f7e8fb9a640 1 --2- 172.21.15.29:0/2227543976 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7e740a1a30 0x7f7e740a3ef0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7e88101300 tx=0x7f7e78005e50 comp rx=0 tx=0).stop 2024-03-11T19:22:21.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.546+0000 7f7e8fb9a640 1 -- 172.21.15.29:0/2227543976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e88101c20 msgr2=0x7f7e88101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:21.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.546+0000 7f7e8fb9a640 1 --2- 172.21.15.29:0/2227543976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e88101c20 0x7f7e88101590 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f7e8400c9a0 tx=0x7f7e8400ce60 comp rx=0 tx=0).stop 2024-03-11T19:22:21.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.546+0000 7f7e8fb9a640 1 -- 172.21.15.29:0/2227543976 shutdown_connections 2024-03-11T19:22:21.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.546+0000 7f7e8fb9a640 1 --2- 172.21.15.29:0/2227543976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e88106e80 0x7f7e88100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:21.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.546+0000 7f7e8fb9a640 1 --2- 172.21.15.29:0/2227543976 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e88104550 0x7f7e880ffbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:21.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.546+0000 7f7e8fb9a640 1 --2- 172.21.15.29:0/2227543976 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f7e740a1a30 0x7f7e740a3ef0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:21.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.546+0000 7f7e8fb9a640 1 --2- 172.21.15.29:0/2227543976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e88101c20 0x7f7e88101590 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:21.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.546+0000 7f7e8fb9a640 1 -- 172.21.15.29:0/2227543976 >> 172.21.15.29:0/2227543976 conn(0x7f7e880fb630 msgr2=0x7f7e881024c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:21.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.547+0000 7f7e8fb9a640 1 -- 172.21.15.29:0/2227543976 shutdown_connections 2024-03-11T19:22:21.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.547+0000 7f7e8fb9a640 1 -- 172.21.15.29:0/2227543976 wait complete. 2024-03-11T19:22:21.567 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-11T19:22:21.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.678+0000 7f972058c640 1 -- 172.21.15.29:0/2802325309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9718104550 msgr2=0x7f9718106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:21.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.678+0000 7f972058c640 1 --2- 172.21.15.29:0/2802325309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9718104550 0x7f9718106940 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f9708009900 tx=0x7f970802bc30 comp rx=0 tx=0).stop 2024-03-11T19:22:21.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.678+0000 7f972058c640 1 -- 172.21.15.29:0/2802325309 shutdown_connections 2024-03-11T19:22:21.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.678+0000 7f972058c640 1 --2- 172.21.15.29:0/2802325309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9718106e80 0x7f97181092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.678+0000 7f972058c640 1 --2- 172.21.15.29:0/2802325309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9718104550 0x7f9718106940 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.678+0000 7f972058c640 1 --2- 172.21.15.29:0/2802325309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9718101c20 0x7f9718104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.678+0000 7f972058c640 1 -- 172.21.15.29:0/2802325309 >> 172.21.15.29:0/2802325309 conn(0x7f97180fb630 msgr2=0x7f97180fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.678+0000 7f972058c640 1 -- 172.21.15.29:0/2802325309 shutdown_connections 2024-03-11T19:22:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.678+0000 7f972058c640 1 -- 172.21.15.29:0/2802325309 wait complete. 2024-03-11T19:22:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.679+0000 7f972058c640 1 Processor -- start 2024-03-11T19:22:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.679+0000 7f972058c640 1 -- start start 2024-03-11T19:22:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.679+0000 7f972058c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9718101c20 0x7f97181971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.679+0000 7f972058c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9718104550 0x7f97181976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.679+0000 7f972058c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9718106e80 0x7f971819e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.679+0000 7f972058c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971819ec90 con 0x7f9718101c20 2024-03-11T19:22:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.679+0000 7f972058c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971819ee00 con 0x7f9718104550 2024-03-11T19:22:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.679+0000 7f972058c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971819efc0 con 0x7f9718106e80 2024-03-11T19:22:21.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.679+0000 7f971e328640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9718101c20 0x7f97181971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:21.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.679+0000 7f971e328640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9718101c20 0x7f97181971a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57730/0 (socket says 172.21.15.29:57730) 2024-03-11T19:22:21.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.679+0000 7f971e328640 1 -- 172.21.15.29:0/3726796984 learned_addr learned my addr 172.21.15.29:0/3726796984 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-11T19:22:21.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.679+0000 7f971db27640 1 --2- 172.21.15.29:0/3726796984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9718104550 0x7f97181976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:21.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.680+0000 7f971e328640 1 -- 172.21.15.29:0/3726796984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9718106e80 msgr2=0x7f971819e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:21.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.680+0000 7f971e328640 1 --2- 172.21.15.29:0/3726796984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9718106e80 0x7f971819e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:21.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.680+0000 7f971e328640 1 -- 172.21.15.29:0/3726796984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9718104550 msgr2=0x7f97181976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:21.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.680+0000 7f971e328640 1 --2- 172.21.15.29:0/3726796984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9718104550 0x7f97181976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:21.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.680+0000 7f971e328640 1 -- 172.21.15.29:0/3726796984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9708009560 con 0x7f9718101c20 2024-03-11T19:22:21.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.680+0000 7f971e328640 1 --2- 172.21.15.29:0/3726796984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9718101c20 0x7f97181971a0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f971400c890 tx=0x7f971400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:21.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.680+0000 7f970f7fe640 1 -- 172.21.15.29:0/3726796984 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9714004220 con 0x7f9718101c20 2024-03-11T19:22:21.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.680+0000 7f972058c640 1 -- 172.21.15.29:0/3726796984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971819f2a0 con 0x7f9718101c20 2024-03-11T19:22:21.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.680+0000 7f970f7fe640 1 -- 172.21.15.29:0/3726796984 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9714007e60 con 0x7f9718101c20 2024-03-11T19:22:21.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.680+0000 7f970f7fe640 1 -- 172.21.15.29:0/3726796984 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9714004a00 con 0x7f9718101c20 2024-03-11T19:22:21.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.680+0000 7f972058c640 1 -- 172.21.15.29:0/3726796984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f971819f780 con 0x7f9718101c20 2024-03-11T19:22:21.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.681+0000 7f972058c640 1 -- 172.21.15.29:0/3726796984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97180fd120 con 0x7f9718101c20 2024-03-11T19:22:21.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.682+0000 7f970f7fe640 1 -- 172.21.15.29:0/3726796984 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f97140079b0 con 0x7f9718101c20 2024-03-11T19:22:21.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.683+0000 7f970f7fe640 1 --2- 172.21.15.29:0/3726796984 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97040a1a30 0x7f97040a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:22:21.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.683+0000 7f971db27640 1 --2- 172.21.15.29:0/3726796984 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97040a1a30 0x7f97040a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:22:21.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.683+0000 7f970f7fe640 1 -- 172.21.15.29:0/3726796984 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8087+0+0 (secure 0 0 0) 0x7f97140cd0d0 con 0x7f9718101c20 2024-03-11T19:22:21.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.683+0000 7f971db27640 1 --2- 172.21.15.29:0/3726796984 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97040a1a30 0x7f97040a3ef0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9708006010 tx=0x7f9708000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:22:21.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.688+0000 7f970f7fe640 1 -- 172.21.15.29:0/3726796984 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f97140963c0 con 0x7f9718101c20 2024-03-11T19:22:21.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:21 smithi170 ceph-mon[35690]: pgmap v728: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-03-11T19:22:21.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:21 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2227543976' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T19:22:21.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:21.872+0000 7f972058c640 1 -- 172.21.15.29:0/3726796984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f971810ca20 con 0x7f9718101c20 2024-03-11T19:22:21.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:21 smithi029 ceph-mon[32439]: pgmap v728: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-03-11T19:22:21.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:21 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2227543976' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T19:22:22.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:21 smithi190 ceph-mon[36292]: pgmap v728: 97 pgs: 97 active+clean; 1.3 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-03-11T19:22:22.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:21 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2227543976' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T19:22:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:22.544+0000 7f970f7fe640 1 -- 172.21.15.29:0/3726796984 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v92) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9714099a80 con 0x7f9718101c20 2024-03-11T19:22:22.548 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2024-03-11T19:22:22.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:22.548+0000 7f972058c640 1 -- 172.21.15.29:0/3726796984 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97040a1a30 msgr2=0x7f97040a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:22.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:22.548+0000 7f972058c640 1 --2- 172.21.15.29:0/3726796984 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97040a1a30 0x7f97040a3ef0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9708006010 tx=0x7f9708000bc0 comp rx=0 tx=0).stop 2024-03-11T19:22:22.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:22.548+0000 7f972058c640 1 -- 172.21.15.29:0/3726796984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9718101c20 msgr2=0x7f97181971a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:22:22.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:22.548+0000 7f972058c640 1 --2- 172.21.15.29:0/3726796984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9718101c20 0x7f97181971a0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f971400c890 tx=0x7f971400cd50 comp rx=0 tx=0).stop 2024-03-11T19:22:22.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:22.548+0000 7f972058c640 1 -- 172.21.15.29:0/3726796984 shutdown_connections 2024-03-11T19:22:22.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:22.548+0000 7f972058c640 1 --2- 172.21.15.29:0/3726796984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9718106e80 0x7f971819e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:22.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:22.548+0000 7f972058c640 1 --2- 172.21.15.29:0/3726796984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9718104550 0x7f97181976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:22.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:22.548+0000 7f972058c640 1 --2- 172.21.15.29:0/3726796984 >> [v2:172.21.15.29:6800/1587871898,v1:172.21.15.29:6801/1587871898] conn(0x7f97040a1a30 0x7f97040a3ef0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:22.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:22.548+0000 7f972058c640 1 --2- 172.21.15.29:0/3726796984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9718101c20 0x7f97181971a0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:22:22.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:22.548+0000 7f972058c640 1 -- 172.21.15.29:0/3726796984 >> 172.21.15.29:0/3726796984 conn(0x7f97180fb630 msgr2=0x7f9718103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:22:22.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:22.548+0000 7f972058c640 1 -- 172.21.15.29:0/3726796984 shutdown_connections 2024-03-11T19:22:22.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-11T19:22:22.548+0000 7f972058c640 1 -- 172.21.15.29:0/3726796984 wait complete. 2024-03-11T19:22:22.573 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T19:22:22.614 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T19:22:22.657 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T19:22:22.687 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-11T19:22:22.688 DEBUG:teuthology.orchestra.run.smithi029:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T19:22:22.710 DEBUG:teuthology.orchestra.run.smithi170:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T19:22:22.725 DEBUG:teuthology.orchestra.run.smithi190:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T19:22:22.747 INFO:tasks.cephadm:Stopping all daemons... 2024-03-11T19:22:22.747 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-11T19:22:22.747 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mon.a 2024-03-11T19:22:22.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:22 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/2227543976' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T19:22:22.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:22 smithi170 ceph-mon[35690]: osdmap e91: 12 total, 12 up, 12 in 2024-03-11T19:22:22.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:22 smithi170 ceph-mon[35690]: from='client.? 172.21.15.29:0/3726796984' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T19:22:22.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:22 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/2227543976' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T19:22:22.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:22 smithi029 ceph-mon[32439]: osdmap e91: 12 total, 12 up, 12 in 2024-03-11T19:22:22.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:22 smithi029 ceph-mon[32439]: from='client.? 172.21.15.29:0/3726796984' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T19:22:22.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:22 smithi029 systemd[1]: Stopping Ceph mon.a for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:22:23.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:22 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/2227543976' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T19:22:23.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:22 smithi190 ceph-mon[36292]: osdmap e91: 12 total, 12 up, 12 in 2024-03-11T19:22:23.013 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:22 smithi190 ceph-mon[36292]: from='client.? 172.21.15.29:0/3726796984' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T19:22:23.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:23 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-a[32435]: 2024-03-11T19:22:23.115+0000 7f700f94c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:22:23.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:23 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-a[32435]: 2024-03-11T19:22:23.115+0000 7f700f94c700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-11T19:22:23.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:23 smithi029 podman[107512]: 2024-03-11 19:22:23.279757746 +0000 UTC m=+0.283836618 container died 3d682d103af0cf3c34c3451d8fcc2bc22be7d675cede943758165c2d82c4d417 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:22:23.906 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:23 smithi029 podman[107512]: 2024-03-11 19:22:23.695513246 +0000 UTC m=+0.699592120 container cleanup 3d682d103af0cf3c34c3451d8fcc2bc22be7d675cede943758165c2d82c4d417 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:22:23.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 11 19:22:23 smithi029 bash[107512]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-a 2024-03-11T19:22:24.188 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mon.a.service' 2024-03-11T19:22:24.229 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:22:24.229 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-11T19:22:24.229 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-11T19:22:24.229 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mon.b 2024-03-11T19:22:24.568 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:24 smithi170 systemd[1]: Stopping Ceph mon.b for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:22:24.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:24 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-b[35684]: 2024-03-11T19:22:24.563+0000 7fa38297d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:22:24.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:24 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-b[35684]: 2024-03-11T19:22:24.563+0000 7fa38297d700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-11T19:22:24.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:24 smithi170 podman[70892]: 2024-03-11 19:22:24.758742131 +0000 UTC m=+0.322215593 container died c756922f87410a18357597d589284ac5e53f38733572816d3609a8ab0b58277e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:22:25.320 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:25 smithi170 podman[70892]: 2024-03-11 19:22:25.090159467 +0000 UTC m=+0.653632930 container cleanup c756922f87410a18357597d589284ac5e53f38733572816d3609a8ab0b58277e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397) 2024-03-11T19:22:25.320 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 11 19:22:25 smithi170 bash[70892]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-b 2024-03-11T19:22:25.542 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mon.b.service' 2024-03-11T19:22:25.588 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:22:25.588 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-11T19:22:25.588 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-11T19:22:25.589 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mon.c 2024-03-11T19:22:25.891 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:25 smithi190 systemd[1]: Stopping Ceph mon.c for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:22:26.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:25 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-c[36287]: 2024-03-11T19:22:25.887+0000 7fe06f24e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:22:26.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:25 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-c[36287]: 2024-03-11T19:22:25.887+0000 7fe06f24e700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-11T19:22:26.263 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:26 smithi190 podman[71563]: 2024-03-11 19:22:26.091925526 +0000 UTC m=+0.318244551 container died 632c5107f599ea62e70859ceee691ad4d31e932bedbf38e9450a70d0a8274b39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-11T19:22:26.738 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:26 smithi190 podman[71563]: 2024-03-11 19:22:26.499022557 +0000 UTC m=+0.725341580 container cleanup 632c5107f599ea62e70859ceee691ad4d31e932bedbf38e9450a70d0a8274b39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-11T19:22:26.738 INFO:journalctl@ceph.mon.c.smithi190.stdout:Mar 11 19:22:26 smithi190 bash[71563]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mon-c 2024-03-11T19:22:26.960 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mon.c.service' 2024-03-11T19:22:27.011 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:22:27.011 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-11T19:22:27.011 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-11T19:22:27.011 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mgr.x 2024-03-11T19:22:27.323 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:22:27 smithi029 systemd[1]: Stopping Ceph mgr.x for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:22:27.656 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:22:27 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:22:27.319+0000 7f36b5ec5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:22:27.657 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:22:27 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x[32667]: 2024-03-11T19:22:27.319+0000 7f36b5ec5700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-11T19:22:27.657 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:22:27 smithi029 podman[107632]: 2024-03-11 19:22:27.351826056 +0000 UTC m=+0.153281896 container died f43e9b0478f2758e59775146c378ee26e6d47d7fddc4fa997d9dad6119e10082 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-11T19:22:28.031 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:22:27 smithi029 podman[107632]: 2024-03-11 19:22:27.782069515 +0000 UTC m=+0.583525351 container cleanup f43e9b0478f2758e59775146c378ee26e6d47d7fddc4fa997d9dad6119e10082 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x, RELEASE=squid-94a5397, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-03-11T19:22:28.031 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 11 19:22:27 smithi029 bash[107632]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-x 2024-03-11T19:22:28.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-11T19:22:28.276 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:22:28.276 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-11T19:22:28.276 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-11T19:22:28.276 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mgr.y 2024-03-11T19:22:28.596 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:22:28 smithi170 systemd[1]: Stopping Ceph mgr.y for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:22:28.982 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:22:28 smithi170 podman[71014]: 2024-03-11 19:22:28.615961245 +0000 UTC m=+0.145938717 container died 1172ef466c1c3d0fc59e2df6a62e9d8b0fd1aa388ca242a7f50c2fd6be9a2c1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-11T19:22:29.346 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:22:29 smithi170 podman[71014]: 2024-03-11 19:22:29.087662288 +0000 UTC m=+0.617639766 container cleanup 1172ef466c1c3d0fc59e2df6a62e9d8b0fd1aa388ca242a7f50c2fd6be9a2c1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397) 2024-03-11T19:22:29.346 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Mar 11 19:22:29 smithi170 bash[71014]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-y 2024-03-11T19:22:29.479 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-11T19:22:29.517 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:22:29.517 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-11T19:22:29.517 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-11T19:22:29.517 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mgr.z 2024-03-11T19:22:29.846 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:22:29 smithi190 systemd[1]: Stopping Ceph mgr.z for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:22:30.151 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:22:29 smithi190 podman[71686]: 2024-03-11 19:22:29.841856794 +0000 UTC m=+0.137360784 container died 8b60ab5ea9125925525c2ed4248e8051f5014deef150dea51ad2f5f496b7ac82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=) 2024-03-11T19:22:30.505 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:22:30 smithi190 podman[71686]: 2024-03-11 19:22:30.255224912 +0000 UTC m=+0.550728916 container cleanup 8b60ab5ea9125925525c2ed4248e8051f5014deef150dea51ad2f5f496b7ac82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:22:30.505 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:22:30 smithi190 bash[71686]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z 2024-03-11T19:22:30.763 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:22:30 smithi190 podman[71700]: 2024-03-11 19:22:30.526763761 +0000 UTC m=+0.680942978 container remove 8b60ab5ea9125925525c2ed4248e8051f5014deef150dea51ad2f5f496b7ac82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mgr-z, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:22:30.763 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Mar 11 19:22:30 smithi190 systemd[1]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-03-11T19:22:30.848 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mgr.z.service' 2024-03-11T19:22:30.888 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:22:30.888 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-11T19:22:30.889 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-11T19:22:30.889 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.0 2024-03-11T19:22:31.210 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:22:30 smithi029 systemd[1]: Stopping Ceph osd.0 for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:22:31.482 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:22:31 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-0[46761]: 2024-03-11T19:22:31.205+0000 7fe8f5b48700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:22:31.482 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:22:31 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-0[46761]: 2024-03-11T19:22:31.205+0000 7fe8f5b48700 -1 osd.0 92 *** Got signal Terminated *** 2024-03-11T19:22:31.482 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:22:31 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-0[46761]: 2024-03-11T19:22:31.205+0000 7fe8f5b48700 -1 osd.0 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T19:22:37.005 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:22:36 smithi029 podman[107755]: 2024-03-11 19:22:36.685010056 +0000 UTC m=+5.592104210 container died 123983e8ccce257397d1b293b7a97bf6e6caf299cf15b9da6902dc5b9213d87f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-94a5397) 2024-03-11T19:22:37.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:22:37 smithi029 podman[107755]: 2024-03-11 19:22:37.100931123 +0000 UTC m=+6.008025280 container cleanup 123983e8ccce257397d1b293b7a97bf6e6caf299cf15b9da6902dc5b9213d87f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-11T19:22:37.333 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:22:37 smithi029 bash[107755]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-0 2024-03-11T19:22:37.656 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:22:37 smithi029 podman[107784]: 2024-03-11 19:22:37.330059607 +0000 UTC m=+0.641804634 container remove 123983e8ccce257397d1b293b7a97bf6e6caf299cf15b9da6902dc5b9213d87f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-0, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-11T19:22:38.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:22:37 smithi029 podman[107872]: 2024-03-11 19:22:37.688591026 +0000 UTC m=+0.018156072 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:22:38.050 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:22:37 smithi029 podman[107872]: 2024-03-11 19:22:37.807357812 +0000 UTC m=+0.136922896 container create 34e35b5c01b0427d472bc223179a609b4d631a6b8e85b216faff4abf923eeb47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True) 2024-03-11T19:22:38.050 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:22:38 smithi029 podman[107872]: 2024-03-11 19:22:38.046693729 +0000 UTC m=+0.376258774 container init 34e35b5c01b0427d472bc223179a609b4d631a6b8e85b216faff4abf923eeb47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, RELEASE=squid-94a5397) 2024-03-11T19:22:38.406 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:22:38 smithi029 podman[107872]: 2024-03-11 19:22:38.050437407 +0000 UTC m=+0.380002455 container start 34e35b5c01b0427d472bc223179a609b4d631a6b8e85b216faff4abf923eeb47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-0-deactivate, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-11T19:22:38.407 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:22:38 smithi029 podman[107872]: 2024-03-11 19:22:38.197363109 +0000 UTC m=+0.526928148 container attach 34e35b5c01b0427d472bc223179a609b4d631a6b8e85b216faff4abf923eeb47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-11T19:22:38.906 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 11 19:22:38 smithi029 podman[107872]: 2024-03-11 19:22:38.63675188 +0000 UTC m=+0.966316918 container died 34e35b5c01b0427d472bc223179a609b4d631a6b8e85b216faff4abf923eeb47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-11T19:22:39.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.0.service' 2024-03-11T19:22:39.338 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:22:39.338 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-11T19:22:39.338 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-11T19:22:39.338 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.1 2024-03-11T19:22:39.738 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:22:39 smithi170 systemd[1]: Stopping Ceph osd.1 for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:22:40.096 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:22:39 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-1[40296]: 2024-03-11T19:22:39.733+0000 7f2b33b00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:22:40.096 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:22:39 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-1[40296]: 2024-03-11T19:22:39.733+0000 7f2b33b00700 -1 osd.1 92 *** Got signal Terminated *** 2024-03-11T19:22:40.097 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:22:39 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-1[40296]: 2024-03-11T19:22:39.733+0000 7f2b33b00700 -1 osd.1 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T19:22:45.596 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:22:45 smithi170 podman[71152]: 2024-03-11 19:22:45.328457698 +0000 UTC m=+5.804051288 container died 1154df2ddc2974b43ee823946a597e66cbe2cd0423dfa5f909691a9914d655c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-1, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-11T19:22:45.897 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:22:45 smithi170 podman[71152]: 2024-03-11 19:22:45.675575348 +0000 UTC m=+6.151168935 container cleanup 1154df2ddc2974b43ee823946a597e66cbe2cd0423dfa5f909691a9914d655c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-1, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397) 2024-03-11T19:22:45.897 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:22:45 smithi170 bash[71152]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-1 2024-03-11T19:22:46.336 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:22:45 smithi170 podman[71171]: 2024-03-11 19:22:45.8941034 +0000 UTC m=+0.560050288 container remove 1154df2ddc2974b43ee823946a597e66cbe2cd0423dfa5f909691a9914d655c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:22:46.596 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:22:46 smithi170 podman[71267]: 2024-03-11 19:22:46.235751341 +0000 UTC m=+0.018406504 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:22:46.596 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:22:46 smithi170 podman[71267]: 2024-03-11 19:22:46.345254619 +0000 UTC m=+0.127909811 container create cc8934e2acfb262573ca7a434bbb4fe59121389a40c9cbe440642623f65ccb39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-11T19:22:46.906 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:22:46 smithi170 podman[71267]: 2024-03-11 19:22:46.609458878 +0000 UTC m=+0.392114028 container init cc8934e2acfb262573ca7a434bbb4fe59121389a40c9cbe440642623f65ccb39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-1-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-11T19:22:46.906 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:22:46 smithi170 podman[71267]: 2024-03-11 19:22:46.612595608 +0000 UTC m=+0.395250760 container start cc8934e2acfb262573ca7a434bbb4fe59121389a40c9cbe440642623f65ccb39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-11T19:22:46.906 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:22:46 smithi170 podman[71267]: 2024-03-11 19:22:46.714799081 +0000 UTC m=+0.497454261 container attach cc8934e2acfb262573ca7a434bbb4fe59121389a40c9cbe440642623f65ccb39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-1-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-11T19:22:47.262 INFO:journalctl@ceph.osd.1.smithi170.stdout:Mar 11 19:22:46 smithi170 podman[71267]: 2024-03-11 19:22:46.903269537 +0000 UTC m=+0.685924685 container died cc8934e2acfb262573ca7a434bbb4fe59121389a40c9cbe440642623f65ccb39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-1-deactivate, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-11T19:22:47.591 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.1.service' 2024-03-11T19:22:47.637 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:22:47.637 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-11T19:22:47.638 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-11T19:22:47.638 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.2 2024-03-11T19:22:47.938 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:22:47 smithi190 systemd[1]: Stopping Ceph osd.2 for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:22:48.263 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:22:47 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-2[41403]: 2024-03-11T19:22:47.934+0000 7f7fa2a15700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:22:48.263 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:22:47 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-2[41403]: 2024-03-11T19:22:47.934+0000 7f7fa2a15700 -1 osd.2 92 *** Got signal Terminated *** 2024-03-11T19:22:48.263 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:22:47 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-2[41403]: 2024-03-11T19:22:47.934+0000 7f7fa2a15700 -1 osd.2 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T19:22:53.855 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:22:53 smithi190 podman[71820]: 2024-03-11 19:22:53.547728155 +0000 UTC m=+5.717927818 container died 88797bf6b4f55d9290539e6d00d46049fd04d6ff9ee1bc9dcffcae201fef7718 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-11T19:22:54.207 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:22:53 smithi190 podman[71820]: 2024-03-11 19:22:53.952271159 +0000 UTC m=+6.122470830 container cleanup 88797bf6b4f55d9290539e6d00d46049fd04d6ff9ee1bc9dcffcae201fef7718 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-11T19:22:54.207 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:22:53 smithi190 bash[71820]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-2 2024-03-11T19:22:54.512 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:22:54 smithi190 podman[71837]: 2024-03-11 19:22:54.204181175 +0000 UTC m=+0.655007133 container remove 88797bf6b4f55d9290539e6d00d46049fd04d6ff9ee1bc9dcffcae201fef7718 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T19:22:54.836 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:22:54 smithi190 podman[71930]: 2024-03-11 19:22:54.421111005 +0000 UTC m=+0.018464902 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:22:54.836 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:22:54 smithi190 podman[71930]: 2024-03-11 19:22:54.547017959 +0000 UTC m=+0.144371899 container create af6994bbc359fa542b6ca7396c1a909b4094ec04b824b959b50d58390745d68d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-11T19:22:54.836 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:22:54 smithi190 podman[71930]: 2024-03-11 19:22:54.833364335 +0000 UTC m=+0.430718219 container init af6994bbc359fa542b6ca7396c1a909b4094ec04b824b959b50d58390745d68d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-11T19:22:55.131 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:22:54 smithi190 podman[71930]: 2024-03-11 19:22:54.836708158 +0000 UTC m=+0.434062043 container start af6994bbc359fa542b6ca7396c1a909b4094ec04b824b959b50d58390745d68d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-94a5397) 2024-03-11T19:22:55.131 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:22:54 smithi190 podman[71930]: 2024-03-11 19:22:54.92495564 +0000 UTC m=+0.522309555 container attach af6994bbc359fa542b6ca7396c1a909b4094ec04b824b959b50d58390745d68d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-11T19:22:55.406 INFO:journalctl@ceph.osd.2.smithi190.stdout:Mar 11 19:22:55 smithi190 podman[71930]: 2024-03-11 19:22:55.128503003 +0000 UTC m=+0.725856886 container died af6994bbc359fa542b6ca7396c1a909b4094ec04b824b959b50d58390745d68d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:22:55.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.2.service' 2024-03-11T19:22:55.831 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:22:55.831 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-11T19:22:55.831 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-11T19:22:55.831 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.3 2024-03-11T19:22:56.147 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:22:55 smithi029 systemd[1]: Stopping Ceph osd.3 for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:22:56.406 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:22:56 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-3[53739]: 2024-03-11T19:22:56.143+0000 7f94cc6a4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:22:56.407 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:22:56 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-3[53739]: 2024-03-11T19:22:56.143+0000 7f94cc6a4700 -1 osd.3 92 *** Got signal Terminated *** 2024-03-11T19:22:56.407 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:22:56 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-3[53739]: 2024-03-11T19:22:56.143+0000 7f94cc6a4700 -1 osd.3 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T19:22:57.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:22:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:22:57.385+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:22:57.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:22:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:22:57.367+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:22:58.156 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:22:57 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:22:57.859+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:22:58.656 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:22:58 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-3[53739]: 2024-03-11T19:22:58.221+0000 7f94c4c90700 -1 osd.3 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.744754+0000 front 2024-03-11T19:22:33.744783+0000 (oldest deadline 2024-03-11T19:22:57.845074+0000) 2024-03-11T19:22:58.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:22:58 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:22:58.377+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:22:58.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:22:58 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:22:58.410+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:22:59.156 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:22:58 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:22:58.848+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:22:59.657 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:22:59 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-3[53739]: 2024-03-11T19:22:59.173+0000 7f94c4c90700 -1 osd.3 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.744754+0000 front 2024-03-11T19:22:33.744783+0000 (oldest deadline 2024-03-11T19:22:57.845074+0000) 2024-03-11T19:22:59.664 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:22:59 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:22:59.393+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:22:59.664 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:22:59 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:22:59.416+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:00.156 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:22:59 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:22:59.868+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:00.656 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:23:00 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-3[53739]: 2024-03-11T19:23:00.164+0000 7f94c4c90700 -1 osd.3 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.744754+0000 front 2024-03-11T19:22:33.744783+0000 (oldest deadline 2024-03-11T19:22:57.845074+0000) 2024-03-11T19:23:00.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:00 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:00.374+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:23:00.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:00 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:00.429+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:01.131 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:00 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:00.872+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:01.131 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:00 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:00.761+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:01.406 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:23:01 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-3[53739]: 2024-03-11T19:23:01.127+0000 7f94c4c90700 -1 osd.3 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.744754+0000 front 2024-03-11T19:22:33.744783+0000 (oldest deadline 2024-03-11T19:22:57.845074+0000) 2024-03-11T19:23:01.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:01 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:01.422+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:01.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:01 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:01.351+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:23:01.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:01 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:01.454+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:01.984 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:01 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:01.859+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:01.985 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:01 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:01.770+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:01.985 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:23:01 smithi029 podman[107976]: 2024-03-11 19:23:01.677248409 +0000 UTC m=+5.650933403 container died f658c458ef8670e4b85d9c74f041a88c76b8703a513d169cee918a703d64b892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-11T19:23:02.379 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:23:02 smithi029 podman[107976]: 2024-03-11 19:23:02.0900119 +0000 UTC m=+6.063696894 container cleanup f658c458ef8670e4b85d9c74f041a88c76b8703a513d169cee918a703d64b892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-3, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-11T19:23:02.379 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:23:02 smithi029 bash[107976]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-3 2024-03-11T19:23:02.656 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:23:02 smithi029 podman[108002]: 2024-03-11 19:23:02.376120477 +0000 UTC m=+0.684201669 container remove f658c458ef8670e4b85d9c74f041a88c76b8703a513d169cee918a703d64b892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-3, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-11T19:23:02.762 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:02 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:02.451+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:02.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:02 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:02.378+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:23:02.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:02 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:02.461+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:02.946 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:02 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:02.899+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:02.946 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:23:02 smithi029 podman[108093]: 2024-03-11 19:23:02.580167666 +0000 UTC m=+0.017642127 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:23:02.946 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:23:02 smithi029 podman[108093]: 2024-03-11 19:23:02.702977923 +0000 UTC m=+0.140452365 container create b7f89cee9fd29362a6035badf30b8f412b876dd01b640ac4ab1601f8cdf03ee6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0) 2024-03-11T19:23:02.947 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:02 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:02.760+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:03.252 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:23:02 smithi029 podman[108093]: 2024-03-11 19:23:02.94298038 +0000 UTC m=+0.380454836 container init b7f89cee9fd29362a6035badf30b8f412b876dd01b640ac4ab1601f8cdf03ee6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-11T19:23:03.252 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:23:02 smithi029 podman[108093]: 2024-03-11 19:23:02.946437461 +0000 UTC m=+0.383911905 container start b7f89cee9fd29362a6035badf30b8f412b876dd01b640ac4ab1601f8cdf03ee6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:23:03.253 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:23:03 smithi029 podman[108093]: 2024-03-11 19:23:03.051466145 +0000 UTC m=+0.488940603 container attach b7f89cee9fd29362a6035badf30b8f412b876dd01b640ac4ab1601f8cdf03ee6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-94a5397) 2024-03-11T19:23:03.610 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 11 19:23:03 smithi029 podman[108093]: 2024-03-11 19:23:03.249308512 +0000 UTC m=+0.686782967 container died b7f89cee9fd29362a6035badf30b8f412b876dd01b640ac4ab1601f8cdf03ee6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-11T19:23:03.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:03 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:03.438+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:03.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:03 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:03.365+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:23:03.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:03 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:03.454+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:03.905 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:03 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:03.716+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:03.925 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.3.service' 2024-03-11T19:23:03.972 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:23:03.972 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-11T19:23:03.972 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-11T19:23:03.972 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.4 2024-03-11T19:23:04.156 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:03 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:03.924+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:04.275 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 11 19:23:04 smithi170 systemd[1]: Stopping Ceph osd.4 for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:23:04.596 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:04 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:04.450+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:04.597 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 11 19:23:04 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-4[47265]: 2024-03-11T19:23:04.270+0000 7f6596aaa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:23:04.597 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 11 19:23:04 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-4[47265]: 2024-03-11T19:23:04.270+0000 7f6596aaa700 -1 osd.4 92 *** Got signal Terminated *** 2024-03-11T19:23:04.597 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 11 19:23:04 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-4[47265]: 2024-03-11T19:23:04.270+0000 7f6596aaa700 -1 osd.4 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T19:23:04.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:04 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:04.407+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:23:04.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:04 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:04.391+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:04.966 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:04 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:04.685+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:05.406 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:04 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:04.962+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:05.766 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:05 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:05.372+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:23:05.766 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:05 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:05.389+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:05.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:05 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:05.461+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:05.958 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:05 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:05.677+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:05.958 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:05 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:05.677+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:06.407 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:05 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:05.954+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:06.407 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:05 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:05.954+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:06.753 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:06 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:06.476+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:06.762 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:06 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:06.428+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:06.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:06 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:06.366+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:23:07.096 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 11 19:23:06 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-4[47265]: 2024-03-11T19:23:06.749+0000 7f658f096700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.586234+0000 front 2024-03-11T19:22:41.586266+0000 (oldest deadline 2024-03-11T19:23:06.286370+0000) 2024-03-11T19:23:07.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:06 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:06.683+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:07.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:06 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:06.684+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:07.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:06 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:06.932+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:07.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:06 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:06.932+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:07.683 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:07 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:07.389+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:23:07.683 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:07 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:07.390+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:07.721 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:07 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:07.462+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:07.721 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:07 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:07.462+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:08.014 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:07 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:07.679+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:08.096 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 11 19:23:07 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-4[47265]: 2024-03-11T19:23:07.716+0000 7f658f096700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.586234+0000 front 2024-03-11T19:22:41.586266+0000 (oldest deadline 2024-03-11T19:23:06.286370+0000) 2024-03-11T19:23:08.096 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:07 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:07.899+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:08.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:07 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:07.729+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:08.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:07 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:07.729+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:08.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:07 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:07.888+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:08.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:07 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:07.888+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:08.700 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:08 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:08.437+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:23:08.701 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:08 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:08.416+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:08.846 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 11 19:23:08 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-4[47265]: 2024-03-11T19:23:08.681+0000 7f658f096700 -1 osd.4 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.586234+0000 front 2024-03-11T19:22:41.586266+0000 (oldest deadline 2024-03-11T19:23:06.286370+0000) 2024-03-11T19:23:08.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:08 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:08.439+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:08.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:08 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:08.439+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:09.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:08 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:08.697+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:09.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:08 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:08.729+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:09.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:08 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:08.729+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:09.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:08 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:08.875+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:09.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:08 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:08.875+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:09.346 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:08 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:08.918+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:09.763 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:09 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:09.681+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:09.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:09 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:09.467+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:23:09.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:09 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:09.467+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:42.138570+0000 front 2024-03-11T19:22:42.138587+0000 (oldest deadline 2024-03-11T19:23:09.139394+0000) 2024-03-11T19:23:09.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:09 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:09.440+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:09.764 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:09 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:09.440+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:09.790 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:09 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:09.434+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:09.790 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:09 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:09.434+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:10.096 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 11 19:23:09 smithi170 podman[71397]: 2024-03-11 19:23:09.906227377 +0000 UTC m=+5.746157346 container died 408bb11dcf7a7067f1904b495628943246da79b6874421fea4a6d9e90c0a287d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-4, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-11T19:23:10.097 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:09 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:09.885+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:10.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:09 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:09.771+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:10.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:09 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:09.771+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:10.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:09 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:09.855+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:10.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:09 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:09.855+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:10.568 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 11 19:23:10 smithi170 podman[71397]: 2024-03-11 19:23:10.296700591 +0000 UTC m=+6.136630568 container cleanup 408bb11dcf7a7067f1904b495628943246da79b6874421fea4a6d9e90c0a287d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-4, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-11T19:23:10.568 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 11 19:23:10 smithi170 bash[71397]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-4 2024-03-11T19:23:10.568 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:10 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:10.444+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:10.568 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:10 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:10.444+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:10.763 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:10 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:10.642+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:10.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:10 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:10.475+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:10.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:10 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:10.475+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:10.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:10 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:10.449+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:23:10.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:10 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:10.449+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:42.138570+0000 front 2024-03-11T19:22:42.138587+0000 (oldest deadline 2024-03-11T19:23:09.139394+0000) 2024-03-11T19:23:10.846 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 11 19:23:10 smithi170 podman[71416]: 2024-03-11 19:23:10.565068652 +0000 UTC m=+0.656000360 container remove 408bb11dcf7a7067f1904b495628943246da79b6874421fea4a6d9e90c0a287d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-4, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-11T19:23:11.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:10 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:10.901+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:11.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:10 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:10.901+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:11.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:10 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:10.781+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:11.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:10 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:10.781+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:11.188 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:10 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:10.928+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:11.188 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 11 19:23:10 smithi170 podman[71507]: 2024-03-11 19:23:10.812296202 +0000 UTC m=+0.018094369 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:23:11.188 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 11 19:23:10 smithi170 podman[71507]: 2024-03-11 19:23:10.915977097 +0000 UTC m=+0.121775256 container create ccf91af207834a030072a58974be8558ecab842fa35a8b7916c70e4ec8ea22b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-11T19:23:11.486 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 11 19:23:11 smithi170 podman[71507]: 2024-03-11 19:23:11.185220069 +0000 UTC m=+0.391018243 container init ccf91af207834a030072a58974be8558ecab842fa35a8b7916c70e4ec8ea22b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-11T19:23:11.486 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 11 19:23:11 smithi170 podman[71507]: 2024-03-11 19:23:11.188594028 +0000 UTC m=+0.394392192 container start ccf91af207834a030072a58974be8558ecab842fa35a8b7916c70e4ec8ea22b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-11T19:23:11.486 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 11 19:23:11 smithi170 podman[71507]: 2024-03-11 19:23:11.268696017 +0000 UTC m=+0.474494193 container attach ccf91af207834a030072a58974be8558ecab842fa35a8b7916c70e4ec8ea22b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-4-deactivate, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-11T19:23:11.486 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:11 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:11.426+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:11.486 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:11 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:11.426+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:11.679 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:11 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:11.405+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:23:11.680 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:11 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:11.405+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:42.138570+0000 front 2024-03-11T19:22:42.138587+0000 (oldest deadline 2024-03-11T19:23:09.139394+0000) 2024-03-11T19:23:11.680 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:11 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:11.501+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:11.680 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:11 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:11.501+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:11.825 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 11 19:23:11 smithi170 podman[71507]: 2024-03-11 19:23:11.481212638 +0000 UTC m=+0.687010803 container died ccf91af207834a030072a58974be8558ecab842fa35a8b7916c70e4ec8ea22b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-03-11T19:23:12.012 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:11 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:11.676+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:12.096 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:11 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:11.949+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:12.156 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:11 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:11.933+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:12.156 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:11 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:11.933+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:12.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:11 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:11.806+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:12.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:11 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:11.806+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:12.158 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.4.service' 2024-03-11T19:23:12.198 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:23:12.198 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-11T19:23:12.198 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-11T19:23:12.198 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.5 2024-03-11T19:23:12.509 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:12 smithi190 systemd[1]: Stopping Ceph osd.5 for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:23:12.510 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:12 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:12.372+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:23:12.510 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:12 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:12.372+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:42.138570+0000 front 2024-03-11T19:22:42.138587+0000 (oldest deadline 2024-03-11T19:23:09.139394+0000) 2024-03-11T19:23:12.763 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:12 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:12.642+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:12.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:12 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:12.503+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:12.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:12 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:12.503+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:12.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:12 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:12.507+0000 7fd2e4b41700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:23:12.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:12 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:12.507+0000 7fd2e4b41700 -1 osd.5 92 *** Got signal Terminated *** 2024-03-11T19:23:12.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:12 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:12.507+0000 7fd2e4b41700 -1 osd.5 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T19:23:12.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:12 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:12.459+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:12.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:12 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:12.459+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:13.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:12 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:12.805+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:13.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:12 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:12.805+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:13.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:12 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:12.917+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:13.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:12 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:12.917+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:13.346 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:12 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:12.902+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:13.680 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:13 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:13.397+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:23:13.681 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:13 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:13.397+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:42.138570+0000 front 2024-03-11T19:22:42.138587+0000 (oldest deadline 2024-03-11T19:23:09.139394+0000) 2024-03-11T19:23:13.681 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:13 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:13.482+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:13.681 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:13 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:13.482+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:13.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:13 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:13.490+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:13.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:13 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:13.490+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:14.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:13 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:13.677+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:14.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:13 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:13.893+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:14.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:13 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:13.893+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:14.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:13 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:13.804+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:14.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:13 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:13.804+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:14.349 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:13 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:13.931+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:14.695 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:14 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:14.358+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:23:14.695 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:14 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:14.358+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:42.138570+0000 front 2024-03-11T19:22:42.138587+0000 (oldest deadline 2024-03-11T19:23:09.139394+0000) 2024-03-11T19:23:14.695 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:14 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:14.521+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:14.695 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:14 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:14.521+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:14.696 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:14 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:14.521+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:14.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:14 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:14.503+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:14.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:14 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:14.503+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:14.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:14 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:14.503+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:15.012 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:14 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:14.692+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:15.156 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:14 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:14.872+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:15.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:14 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:14.872+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:15.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:14 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:14.824+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:15.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:14 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:14.824+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:15.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:14 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:14.824+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:15.346 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:14 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:14.938+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:15.659 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:15 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:15.321+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:23:15.660 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:15 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:15.321+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:42.138570+0000 front 2024-03-11T19:22:42.138587+0000 (oldest deadline 2024-03-11T19:23:09.139394+0000) 2024-03-11T19:23:15.660 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:15 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:15.321+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.139902+0000 front 2024-03-11T19:22:49.140002+0000 (oldest deadline 2024-03-11T19:23:15.039922+0000) 2024-03-11T19:23:15.660 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:15 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:15.506+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:15.660 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:15 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:15.506+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:15.660 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:15 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:15.506+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:15.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:15 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:15.531+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:15.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:15 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:15.531+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:15.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:15 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:15.531+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:16.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:15 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:15.655+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:16.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:15 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:15.816+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:16.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:15 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:15.816+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:16.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:15 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:15.816+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:16.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:15 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:15.907+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:16.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:15 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:15.907+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:16.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:15 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:15.907+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:51.611651+0000 front 2024-03-11T19:22:51.611739+0000 (oldest deadline 2024-03-11T19:23:15.111655+0000) 2024-03-11T19:23:16.346 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:15 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:15.908+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:16.346 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:15 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:15.908+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.759566+0000 front 2024-03-11T19:22:52.759620+0000 (oldest deadline 2024-03-11T19:23:15.560205+0000) 2024-03-11T19:23:16.710 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:16 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:16.338+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:23:16.710 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:16 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:16.338+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:42.138570+0000 front 2024-03-11T19:22:42.138587+0000 (oldest deadline 2024-03-11T19:23:09.139394+0000) 2024-03-11T19:23:16.710 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:16 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:16.338+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.139902+0000 front 2024-03-11T19:22:49.140002+0000 (oldest deadline 2024-03-11T19:23:15.039922+0000) 2024-03-11T19:23:16.710 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:16 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:16.464+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:16.710 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:16 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:16.464+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:16.710 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:16 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:16.464+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:16.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:16 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:16.526+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:16.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:16 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:16.526+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:16.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:16 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:16.526+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:17.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:16 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:16.705+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:17.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:16 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:16.705+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:17.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:16 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:16.770+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:17.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:16 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:16.770+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:17.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:16 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:16.770+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:17.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:16 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:16.876+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:17.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:16 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:16.876+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:17.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:16 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:16.876+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:51.611651+0000 front 2024-03-11T19:22:51.611739+0000 (oldest deadline 2024-03-11T19:23:15.111655+0000) 2024-03-11T19:23:17.346 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:16 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:16.871+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:17.347 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:16 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:16.871+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.759566+0000 front 2024-03-11T19:22:52.759620+0000 (oldest deadline 2024-03-11T19:23:15.560205+0000) 2024-03-11T19:23:17.735 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:17 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:17.348+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:31.736437+0000 front 2024-03-11T19:22:31.736423+0000 (oldest deadline 2024-03-11T19:22:56.936922+0000) 2024-03-11T19:23:17.735 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:17 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:17.348+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:42.138570+0000 front 2024-03-11T19:22:42.138587+0000 (oldest deadline 2024-03-11T19:23:09.139394+0000) 2024-03-11T19:23:17.735 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:17 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5[47923]: 2024-03-11T19:23:17.348+0000 7fd2dd12d700 -1 osd.5 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.139902+0000 front 2024-03-11T19:22:49.140002+0000 (oldest deadline 2024-03-11T19:23:15.039922+0000) 2024-03-11T19:23:17.735 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:17 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:17.419+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:17.735 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:17 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:17.419+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:17.735 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:17 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:17.419+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:17.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:17 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:17.514+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:17.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:17 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:17.514+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:17.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:17 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:17.514+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:17.985 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:17 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:17.730+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:17.985 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:17 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:17.730+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:18.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:17 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:17.827+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:18.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:17 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:17.827+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:18.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:17 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:17.827+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:51.611651+0000 front 2024-03-11T19:22:51.611739+0000 (oldest deadline 2024-03-11T19:23:15.111655+0000) 2024-03-11T19:23:18.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:17 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:17.727+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:18.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:17 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:17.727+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:18.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:17 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:17.727+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:18.263 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:17 smithi190 podman[72051]: 2024-03-11 19:23:17.982094268 +0000 UTC m=+5.593673772 container died 95cb041a90b7171adf83c0508821f7a80701b8a0d4b5def6c1ddd47bd6a87301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-11T19:23:18.346 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:17 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:17.857+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:18.347 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:17 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:17.857+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.759566+0000 front 2024-03-11T19:22:52.759620+0000 (oldest deadline 2024-03-11T19:23:15.560205+0000) 2024-03-11T19:23:18.663 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:18 smithi190 podman[72051]: 2024-03-11 19:23:18.366554764 +0000 UTC m=+5.978134259 container cleanup 95cb041a90b7171adf83c0508821f7a80701b8a0d4b5def6c1ddd47bd6a87301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5) 2024-03-11T19:23:18.663 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:18 smithi190 bash[72051]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5 2024-03-11T19:23:18.663 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:18 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:18.441+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:18.663 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:18 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:18.441+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:18.663 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:18 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:18.441+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:18.841 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:18 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:18.526+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:18.841 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:18 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:18.526+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:18.841 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:18 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:18.526+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:18.988 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:18 smithi190 podman[72067]: 2024-03-11 19:23:18.65959867 +0000 UTC m=+0.672511773 container remove 95cb041a90b7171adf83c0508821f7a80701b8a0d4b5def6c1ddd47bd6a87301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-11T19:23:18.989 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:18 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:18.777+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:18.989 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:18 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:18.777+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:19.096 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:18 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:18.836+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:19.097 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:18 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:18.836+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.759566+0000 front 2024-03-11T19:22:52.759620+0000 (oldest deadline 2024-03-11T19:23:15.560205+0000) 2024-03-11T19:23:19.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:18 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:18.697+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:19.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:18 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:18.697+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:19.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:18 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:18.697+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:19.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:18 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:18.787+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:19.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:18 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:18.787+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:19.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:18 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:18.787+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:51.611651+0000 front 2024-03-11T19:22:51.611739+0000 (oldest deadline 2024-03-11T19:23:15.111655+0000) 2024-03-11T19:23:19.241 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:18 smithi190 podman[72158]: 2024-03-11 19:23:18.985383512 +0000 UTC m=+0.113924092 container create e133a8aa3a697cb7c44408c6762bb25dc065d8342e6cc9c2600c68575041b7e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:23:19.242 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:18 smithi190 podman[72158]: 2024-03-11 19:23:18.889127483 +0000 UTC m=+0.017668001 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:23:19.513 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:19 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:19.416+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:19.513 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:19 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:19.416+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:19.513 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:19 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:19.416+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:19.513 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:19 smithi190 podman[72158]: 2024-03-11 19:23:19.238400457 +0000 UTC m=+0.366940994 container init e133a8aa3a697cb7c44408c6762bb25dc065d8342e6cc9c2600c68575041b7e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:23:19.513 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:19 smithi190 podman[72158]: 2024-03-11 19:23:19.242343074 +0000 UTC m=+0.370883597 container start e133a8aa3a697cb7c44408c6762bb25dc065d8342e6cc9c2600c68575041b7e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, ceph=True) 2024-03-11T19:23:19.513 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:19 smithi190 podman[72158]: 2024-03-11 19:23:19.313060335 +0000 UTC m=+0.441600870 container attach e133a8aa3a697cb7c44408c6762bb25dc065d8342e6cc9c2600c68575041b7e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T19:23:19.835 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:19 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:19.508+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:19.835 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:19 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:19.508+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:19.835 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:19 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:19.508+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:19.854 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 11 19:23:19 smithi190 podman[72158]: 2024-03-11 19:23:19.536508385 +0000 UTC m=+0.665048899 container died e133a8aa3a697cb7c44408c6762bb25dc065d8342e6cc9c2600c68575041b7e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-11T19:23:19.854 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:19 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:19.759+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:19.854 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:19 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:19.759+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:20.096 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:19 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:19.832+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:20.096 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:19 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:19.832+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.759566+0000 front 2024-03-11T19:22:52.759620+0000 (oldest deadline 2024-03-11T19:23:15.560205+0000) 2024-03-11T19:23:20.151 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.5.service' 2024-03-11T19:23:20.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:19 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:19.688+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:20.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:19 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:19.688+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:20.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:19 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:19.688+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:20.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:19 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:19.798+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:20.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:19 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:19.798+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:20.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:19 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:19.798+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:51.611651+0000 front 2024-03-11T19:22:51.611739+0000 (oldest deadline 2024-03-11T19:23:15.111655+0000) 2024-03-11T19:23:20.192 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:23:20.193 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-11T19:23:20.193 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-11T19:23:20.193 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.6 2024-03-11T19:23:20.502 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:20 smithi029 systemd[1]: Stopping Ceph osd.6 for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:23:20.502 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:20.498+0000 7f30a559e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:23:20.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:20 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:20.467+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:20.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:20 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:20.467+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:20.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:20 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:20.467+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:20.763 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:20 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:20.710+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:20.763 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:20 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:20.710+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:20.774 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:20.709+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:20.774 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:20.709+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:20.774 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:20.709+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:20.774 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:20.498+0000 7f30a559e700 -1 osd.6 92 *** Got signal Terminated *** 2024-03-11T19:23:20.774 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:20.498+0000 7f30a559e700 -1 osd.6 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T19:23:20.774 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:20.770+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:20.774 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:20.770+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:20.774 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:20.770+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:51.611651+0000 front 2024-03-11T19:22:51.611739+0000 (oldest deadline 2024-03-11T19:23:15.111655+0000) 2024-03-11T19:23:20.786 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:20 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:20.477+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:20.786 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:20 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:20.477+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:20.786 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:20 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:20.477+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:21.099 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:20 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:20.782+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:21.099 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:20 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:20.782+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.759566+0000 front 2024-03-11T19:22:52.759620+0000 (oldest deadline 2024-03-11T19:23:15.560205+0000) 2024-03-11T19:23:21.763 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:21 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:21.685+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:21.763 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:21 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:21.685+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:21.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:21 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:21.476+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:21.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:21 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:21.476+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:21.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:21 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:21.476+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:21.784 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:21 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:21.490+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:21.784 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:21 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:21.490+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:21.784 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:21 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:21.490+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:22.096 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:21 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:21.780+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:22.097 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:21 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:21.780+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.759566+0000 front 2024-03-11T19:22:52.759620+0000 (oldest deadline 2024-03-11T19:23:15.560205+0000) 2024-03-11T19:23:22.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:21 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:21.798+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:22.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:21 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:21.798+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:22.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:21 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:21.798+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:51.611651+0000 front 2024-03-11T19:22:51.611739+0000 (oldest deadline 2024-03-11T19:23:15.111655+0000) 2024-03-11T19:23:22.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:21 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:21.703+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:22.158 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:21 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:21.703+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:22.158 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:21 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:21.703+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:22.763 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:22 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:22.682+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:22.763 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:22 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:22.682+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:22.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:22 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:22.499+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:22.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:22 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:22.499+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:22.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:22 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:22.499+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:22.785 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:22 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:22.486+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:22.785 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:22 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:22.486+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:22.785 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:22 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:22.486+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:23.096 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:22 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:22.781+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:23.097 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:22 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:22.781+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.759566+0000 front 2024-03-11T19:22:52.759620+0000 (oldest deadline 2024-03-11T19:23:15.560205+0000) 2024-03-11T19:23:23.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:22 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:22.826+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:23.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:22 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:22.826+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:23.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:22 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:22.826+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:51.611651+0000 front 2024-03-11T19:22:51.611739+0000 (oldest deadline 2024-03-11T19:23:15.111655+0000) 2024-03-11T19:23:23.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:22 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:22.750+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:23.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:22 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:22.750+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:23.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:22 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:22.750+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:23.741 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:23 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:23.478+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:23.741 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:23 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:23.478+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:23.741 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:23 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:23.478+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:24.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:23 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:23.686+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:24.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:23 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:23.686+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:24.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:23 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:23.511+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:24.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:23 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:23.511+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:24.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:23 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:23.511+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:24.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:23 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:23.511+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:24.096 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:23 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:23.738+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:24.097 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:23 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:23.738+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.759566+0000 front 2024-03-11T19:22:52.759620+0000 (oldest deadline 2024-03-11T19:23:15.560205+0000) 2024-03-11T19:23:24.097 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:23 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:23.738+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.561575+0000 front 2024-03-11T19:22:59.561564+0000 (oldest deadline 2024-03-11T19:23:23.061627+0000) 2024-03-11T19:23:24.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:23 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:23.741+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:24.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:23 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:23.741+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:24.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:23 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:23.741+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:24.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:23 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:23.741+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:24.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:23 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:23.780+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:24.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:23 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:23.780+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:24.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:23 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:23.780+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:51.611651+0000 front 2024-03-11T19:22:51.611739+0000 (oldest deadline 2024-03-11T19:23:15.111655+0000) 2024-03-11T19:23:24.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:23 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:23.780+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.713085+0000 front 2024-03-11T19:22:59.713106+0000 (oldest deadline 2024-03-11T19:23:23.213254+0000) 2024-03-11T19:23:24.846 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:24 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:24.712+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:24.847 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:24 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:24.712+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.759566+0000 front 2024-03-11T19:22:52.759620+0000 (oldest deadline 2024-03-11T19:23:15.560205+0000) 2024-03-11T19:23:24.847 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:24 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:24.712+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.561575+0000 front 2024-03-11T19:22:59.561564+0000 (oldest deadline 2024-03-11T19:23:23.061627+0000) 2024-03-11T19:23:24.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:24 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:24.465+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:24.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:24 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:24.465+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:24.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:24 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:24.465+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:25.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:24 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:24.715+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:25.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:24 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:24.715+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:25.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:24 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:24.715+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.879349+0000 front 2024-03-11T19:23:00.879230+0000 (oldest deadline 2024-03-11T19:23:23.779266+0000) 2024-03-11T19:23:25.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:24 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:24.508+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:25.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:24 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:24.508+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:25.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:24 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:24.508+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:25.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:24 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:24.508+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:25.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:24 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:24.740+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:25.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:24 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:24.740+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:25.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:24 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:24.740+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:25.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:24 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:24.740+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:25.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:24 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:24.769+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:35.308689+0000 front 2024-03-11T19:22:35.308753+0000 (oldest deadline 2024-03-11T19:22:57.008854+0000) 2024-03-11T19:23:25.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:24 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:24.769+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:39.810572+0000 front 2024-03-11T19:22:39.810802+0000 (oldest deadline 2024-03-11T19:23:05.710556+0000) 2024-03-11T19:23:25.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:24 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:24.769+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:51.611651+0000 front 2024-03-11T19:22:51.611739+0000 (oldest deadline 2024-03-11T19:23:15.111655+0000) 2024-03-11T19:23:25.157 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:24 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6[60290]: 2024-03-11T19:23:24.769+0000 7f309db8a700 -1 osd.6 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.713085+0000 front 2024-03-11T19:22:59.713106+0000 (oldest deadline 2024-03-11T19:23:23.213254+0000) 2024-03-11T19:23:25.742 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:25 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:25.418+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:25.743 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:25 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:25.418+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:25.743 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:25 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:25.418+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:25.743 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:25 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:25.418+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:26.003 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:25 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:25.719+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:26.003 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:25 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:25.719+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:26.003 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:25 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:25.719+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:26.004 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:25 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:25.719+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:26.012 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:25 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:25.683+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:26.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:25 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:25.683+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:26.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:25 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:25.683+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.879349+0000 front 2024-03-11T19:23:00.879230+0000 (oldest deadline 2024-03-11T19:23:23.779266+0000) 2024-03-11T19:23:26.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:25 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:25.526+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:26.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:25 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:25.526+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:26.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:25 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:25.526+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:26.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:25 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:25.526+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:26.096 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:25 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:25.739+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:26.097 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:25 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:25.739+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.759566+0000 front 2024-03-11T19:22:52.759620+0000 (oldest deadline 2024-03-11T19:23:15.560205+0000) 2024-03-11T19:23:26.097 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:25 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:25.739+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.561575+0000 front 2024-03-11T19:22:59.561564+0000 (oldest deadline 2024-03-11T19:23:23.061627+0000) 2024-03-11T19:23:26.281 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:26 smithi029 podman[108197]: 2024-03-11 19:23:25.999866325 +0000 UTC m=+5.614879928 container died 4752209d7335ba51552e0b69f740c6b2f8ffd1b62d149660722a29c2bcda0ce7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.build-date=20240307) 2024-03-11T19:23:26.657 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:26 smithi029 podman[108197]: 2024-03-11 19:23:26.37761862 +0000 UTC m=+5.992632218 container cleanup 4752209d7335ba51552e0b69f740c6b2f8ffd1b62d149660722a29c2bcda0ce7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-11T19:23:26.657 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:26 smithi029 bash[108197]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6 2024-03-11T19:23:26.776 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:26 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:26.451+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:26.777 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:26 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:26.451+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:26.777 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:26 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:26.451+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:26.777 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:26 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:26.451+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:27.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:26 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:26.724+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:27.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:26 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:26.724+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:27.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:26 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:26.724+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.879349+0000 front 2024-03-11T19:23:00.879230+0000 (oldest deadline 2024-03-11T19:23:23.779266+0000) 2024-03-11T19:23:27.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:26 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:26.528+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:27.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:26 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:26.528+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:27.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:26 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:26.528+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:27.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:26 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:26.528+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:27.047 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:26 smithi029 podman[108213]: 2024-03-11 19:23:26.705458674 +0000 UTC m=+0.700712562 container remove 4752209d7335ba51552e0b69f740c6b2f8ffd1b62d149660722a29c2bcda0ce7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:23:27.047 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:26 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:26.701+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:27.047 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:26 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:26.701+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:27.047 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:26 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:26.701+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:27.047 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:26 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:26.701+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:27.096 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:26 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:26.773+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:27.097 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:26 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:26.773+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.759566+0000 front 2024-03-11T19:22:52.759620+0000 (oldest deadline 2024-03-11T19:23:15.560205+0000) 2024-03-11T19:23:27.097 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:26 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:26.773+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.561575+0000 front 2024-03-11T19:22:59.561564+0000 (oldest deadline 2024-03-11T19:23:23.061627+0000) 2024-03-11T19:23:27.316 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:27 smithi029 podman[108303]: 2024-03-11 19:23:26.947077735 +0000 UTC m=+0.018066103 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:23:27.316 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:27 smithi029 podman[108303]: 2024-03-11 19:23:27.050233151 +0000 UTC m=+0.121221509 container create 4fa39aeff0ab45ec7a5d0a924b1eee70a4f7455c54d14a4c9c8b491d2402f63c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-11T19:23:27.316 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:27 smithi029 podman[108303]: 2024-03-11 19:23:27.313114099 +0000 UTC m=+0.384102473 container init 4fa39aeff0ab45ec7a5d0a924b1eee70a4f7455c54d14a4c9c8b491d2402f63c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-11T19:23:27.618 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:27 smithi029 podman[108303]: 2024-03-11 19:23:27.316380666 +0000 UTC m=+0.387369030 container start 4fa39aeff0ab45ec7a5d0a924b1eee70a4f7455c54d14a4c9c8b491d2402f63c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6-deactivate, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T19:23:27.618 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:27 smithi029 podman[108303]: 2024-03-11 19:23:27.397097586 +0000 UTC m=+0.468085946 container attach 4fa39aeff0ab45ec7a5d0a924b1eee70a4f7455c54d14a4c9c8b491d2402f63c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-94a5397) 2024-03-11T19:23:27.778 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:27 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:27.491+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:27.778 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:27 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:27.491+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:27.778 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:27 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:27.491+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:27.778 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:27 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:27.491+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:27.906 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 11 19:23:27 smithi029 podman[108303]: 2024-03-11 19:23:27.614917671 +0000 UTC m=+0.685906048 container died 4fa39aeff0ab45ec7a5d0a924b1eee70a4f7455c54d14a4c9c8b491d2402f63c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-11T19:23:27.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:27 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:27.696+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:27.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:27 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:27.696+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:27.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:27 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:27.696+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:27.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:27 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:27.696+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:28.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:27 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:27.678+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:28.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:27 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:27.678+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:28.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:27 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:27.678+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.879349+0000 front 2024-03-11T19:23:00.879230+0000 (oldest deadline 2024-03-11T19:23:23.779266+0000) 2024-03-11T19:23:28.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:27 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:27.524+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:28.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:27 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:27.524+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:28.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:27 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:27.524+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:28.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:27 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:27.524+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:28.096 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:27 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:27.774+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:28.097 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:27 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:27.774+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.759566+0000 front 2024-03-11T19:22:52.759620+0000 (oldest deadline 2024-03-11T19:23:15.560205+0000) 2024-03-11T19:23:28.097 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:27 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:27.774+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.561575+0000 front 2024-03-11T19:22:59.561564+0000 (oldest deadline 2024-03-11T19:23:23.061627+0000) 2024-03-11T19:23:28.236 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.6.service' 2024-03-11T19:23:28.281 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:23:28.281 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-11T19:23:28.281 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-11T19:23:28.281 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.7 2024-03-11T19:23:28.587 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:28 smithi170 systemd[1]: Stopping Ceph osd.7 for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:23:28.587 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:28 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:28.512+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:28.587 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:28 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:28.512+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:28.587 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:28 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:28.512+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:28.587 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:28 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:28.512+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:28.763 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:28 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:28.687+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:28.763 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:28 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:28.687+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:28.763 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:28 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:28.687+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.879349+0000 front 2024-03-11T19:23:00.879230+0000 (oldest deadline 2024-03-11T19:23:23.779266+0000) 2024-03-11T19:23:28.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:28 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:28.485+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:28.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:28 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:28.485+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:28.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:28 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:28.485+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:28.764 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:28 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:28.485+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:28.846 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:28 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:28.584+0000 7f4a78ed5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:23:28.847 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:28 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:28.584+0000 7f4a78ed5700 -1 osd.7 92 *** Got signal Terminated *** 2024-03-11T19:23:28.847 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:28 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:28.584+0000 7f4a78ed5700 -1 osd.7 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T19:23:28.847 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:28 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:28.725+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:28.847 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:28 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:28.725+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.759566+0000 front 2024-03-11T19:22:52.759620+0000 (oldest deadline 2024-03-11T19:23:15.560205+0000) 2024-03-11T19:23:28.847 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:28 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:28.725+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.561575+0000 front 2024-03-11T19:22:59.561564+0000 (oldest deadline 2024-03-11T19:23:23.061627+0000) 2024-03-11T19:23:29.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:28 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:28.723+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:29.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:28 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:28.723+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:29.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:28 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:28.723+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:29.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:28 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:28.723+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:29.731 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:29 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:29.470+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:29.731 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:29 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:29.470+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:29.732 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:29 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:29.470+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:29.732 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:29 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:29.470+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:29.846 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:29 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:29.761+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:29.847 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:29 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:29.761+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.759566+0000 front 2024-03-11T19:22:52.759620+0000 (oldest deadline 2024-03-11T19:23:15.560205+0000) 2024-03-11T19:23:29.847 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:29 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:29.761+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.561575+0000 front 2024-03-11T19:22:59.561564+0000 (oldest deadline 2024-03-11T19:23:23.061627+0000) 2024-03-11T19:23:29.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:29 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:29.553+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:29.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:29 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:29.553+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:29.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:29 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:29.553+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:29.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:29 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:29.553+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:30.012 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:29 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:29.727+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:30.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:29 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:29.727+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:30.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:29 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:29.727+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.879349+0000 front 2024-03-11T19:23:00.879230+0000 (oldest deadline 2024-03-11T19:23:23.779266+0000) 2024-03-11T19:23:30.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:29 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:29.746+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:30.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:29 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:29.746+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:30.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:29 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:29.746+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:30.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:29 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:29.746+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:30.762 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:30 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:30.506+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:30.762 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:30 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:30.506+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:30.762 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:30 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:30.506+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:30.762 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:30 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:30.506+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:30.847 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:30 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:30.766+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:30.847 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:30 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:30.766+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.759566+0000 front 2024-03-11T19:22:52.759620+0000 (oldest deadline 2024-03-11T19:23:15.560205+0000) 2024-03-11T19:23:30.847 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:30 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:30.766+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.561575+0000 front 2024-03-11T19:22:59.561564+0000 (oldest deadline 2024-03-11T19:23:23.061627+0000) 2024-03-11T19:23:30.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:30 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:30.549+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:30.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:30 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:30.549+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:30.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:30 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:30.549+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:30.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:30 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:30.549+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:31.012 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:30 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:30.758+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:31.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:30 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:30.758+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:31.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:30 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:30.758+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.879349+0000 front 2024-03-11T19:23:00.879230+0000 (oldest deadline 2024-03-11T19:23:23.779266+0000) 2024-03-11T19:23:31.058 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:30 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:30.755+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:31.058 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:30 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:30.755+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:31.058 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:30 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:30.755+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:31.058 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:30 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:30.755+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:31.058 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:30 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:30.755+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.679831+0000 front 2024-03-11T19:23:04.679922+0000 (oldest deadline 2024-03-11T19:23:29.979907+0000) 2024-03-11T19:23:31.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:31 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:31.591+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:31.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:31 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:31.591+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:31.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:31 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:31.591+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:31.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:31 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:31.591+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:31.847 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:31 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:31.751+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:31.847 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:31 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:31.751+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.759566+0000 front 2024-03-11T19:22:52.759620+0000 (oldest deadline 2024-03-11T19:23:15.560205+0000) 2024-03-11T19:23:31.847 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:31 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:31.751+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.561575+0000 front 2024-03-11T19:22:59.561564+0000 (oldest deadline 2024-03-11T19:23:23.061627+0000) 2024-03-11T19:23:31.847 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:31 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:31.751+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:08.664443+0000 front 2024-03-11T19:23:08.664397+0000 (oldest deadline 2024-03-11T19:23:30.964579+0000) 2024-03-11T19:23:32.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:31 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:31.510+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:32.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:31 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:31.510+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:32.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:31 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:31.510+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:32.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:31 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:31.510+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:32.014 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:31 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:31.712+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:32.014 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:31 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:31.712+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:32.014 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:31 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:31.712+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.879349+0000 front 2024-03-11T19:23:00.879230+0000 (oldest deadline 2024-03-11T19:23:23.779266+0000) 2024-03-11T19:23:32.014 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:31 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:31.712+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:09.080612+0000 front 2024-03-11T19:23:09.080491+0000 (oldest deadline 2024-03-11T19:23:31.380313+0000) 2024-03-11T19:23:32.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:31 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:31.739+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:32.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:31 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:31.739+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:32.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:31 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:31.739+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:32.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:31 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:31.739+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:32.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:31 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:31.739+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.679831+0000 front 2024-03-11T19:23:04.679922+0000 (oldest deadline 2024-03-11T19:23:29.979907+0000) 2024-03-11T19:23:33.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:32 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:32.705+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:33.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:32 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:32.705+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:33.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:32 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:32.705+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.879349+0000 front 2024-03-11T19:23:00.879230+0000 (oldest deadline 2024-03-11T19:23:23.779266+0000) 2024-03-11T19:23:33.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:32 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:32.705+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:09.080612+0000 front 2024-03-11T19:23:09.080491+0000 (oldest deadline 2024-03-11T19:23:31.380313+0000) 2024-03-11T19:23:33.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:32 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:32.553+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:33.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:32 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:32.553+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:33.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:32 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:32.553+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:33.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:32 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:32.553+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:33.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:32 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:32.553+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:33.097 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:32 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:32.621+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:33.097 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:32 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:32.621+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:33.097 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:32 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:32.621+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:33.097 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:32 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:32.621+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:33.097 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:32 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:32.751+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.157438+0000 front 2024-03-11T19:22:44.157465+0000 (oldest deadline 2024-03-11T19:23:07.057618+0000) 2024-03-11T19:23:33.098 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:32 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:32.751+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.759566+0000 front 2024-03-11T19:22:52.759620+0000 (oldest deadline 2024-03-11T19:23:15.560205+0000) 2024-03-11T19:23:33.098 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:32 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:32.751+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.561575+0000 front 2024-03-11T19:22:59.561564+0000 (oldest deadline 2024-03-11T19:23:23.061627+0000) 2024-03-11T19:23:33.098 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:32 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7[53823]: 2024-03-11T19:23:32.751+0000 7f4a714c1700 -1 osd.7 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:08.664443+0000 front 2024-03-11T19:23:08.664397+0000 (oldest deadline 2024-03-11T19:23:30.964579+0000) 2024-03-11T19:23:33.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:32 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:32.768+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:33.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:32 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:32.768+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:33.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:32 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:32.768+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:33.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:32 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:32.768+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:33.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:32 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:32.768+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.679831+0000 front 2024-03-11T19:23:04.679922+0000 (oldest deadline 2024-03-11T19:23:29.979907+0000) 2024-03-11T19:23:34.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:33 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:33.671+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:34.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:33 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:33.671+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:34.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:33 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:33.671+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.879349+0000 front 2024-03-11T19:23:00.879230+0000 (oldest deadline 2024-03-11T19:23:23.779266+0000) 2024-03-11T19:23:34.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:33 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:33.671+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:09.080612+0000 front 2024-03-11T19:23:09.080491+0000 (oldest deadline 2024-03-11T19:23:31.380313+0000) 2024-03-11T19:23:34.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:33 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:33.579+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:34.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:33 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:33.579+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:34.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:33 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:33.579+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:34.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:33 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:33.579+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:34.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:33 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:33.579+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:34.014 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:33 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:33.650+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:34.014 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:33 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:33.650+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:34.014 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:33 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:33.650+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:34.014 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:33 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:33.650+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:34.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:33 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:33.783+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:34.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:33 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:33.783+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:34.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:33 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:33.783+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:34.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:33 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:33.783+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:34.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:33 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:33.783+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.679831+0000 front 2024-03-11T19:23:04.679922+0000 (oldest deadline 2024-03-11T19:23:29.979907+0000) 2024-03-11T19:23:34.279 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:34 smithi170 podman[71614]: 2024-03-11 19:23:34.011619008 +0000 UTC m=+5.543879133 container died 328c46bb7033d8ebd4266588d07d736452583d0b88f4bdab0fa20de6b72587b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:23:34.596 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:34 smithi170 podman[71614]: 2024-03-11 19:23:34.374654875 +0000 UTC m=+5.906914997 container cleanup 328c46bb7033d8ebd4266588d07d736452583d0b88f4bdab0fa20de6b72587b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-03-11T19:23:34.596 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:34 smithi170 bash[71614]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7 2024-03-11T19:23:35.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:34 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:34.644+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:35.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:34 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:34.644+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:35.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:34 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:34.644+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.879349+0000 front 2024-03-11T19:23:00.879230+0000 (oldest deadline 2024-03-11T19:23:23.779266+0000) 2024-03-11T19:23:35.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:34 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:34.644+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:09.080612+0000 front 2024-03-11T19:23:09.080491+0000 (oldest deadline 2024-03-11T19:23:31.380313+0000) 2024-03-11T19:23:35.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:34 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:34.548+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:35.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:34 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:34.548+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:35.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:34 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:34.548+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:35.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:34 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:34.548+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:35.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:34 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:34.548+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:35.030 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:34 smithi170 podman[71642]: 2024-03-11 19:23:34.684510577 +0000 UTC m=+0.675465267 container remove 328c46bb7033d8ebd4266588d07d736452583d0b88f4bdab0fa20de6b72587b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-11T19:23:35.030 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:34 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:34.626+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:35.031 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:34 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:34.626+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:35.031 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:34 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:34.626+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:35.031 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:34 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:34.626+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:35.031 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:34 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:34.626+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:35.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:34 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:34.737+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:35.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:34 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:34.737+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:35.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:34 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:34.737+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:35.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:34 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:34.737+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:35.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:34 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:34.737+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.679831+0000 front 2024-03-11T19:23:04.679922+0000 (oldest deadline 2024-03-11T19:23:29.979907+0000) 2024-03-11T19:23:35.299 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:35 smithi170 podman[71732]: 2024-03-11 19:23:34.929917348 +0000 UTC m=+0.017400877 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:23:35.300 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:35 smithi170 podman[71732]: 2024-03-11 19:23:35.043796977 +0000 UTC m=+0.131280499 container create af66ddd0c507777517cf75451909778522414748fb7c4a6a4b5509c0070c86e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-11T19:23:35.585 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:35 smithi170 podman[71732]: 2024-03-11 19:23:35.296679171 +0000 UTC m=+0.384162697 container init af66ddd0c507777517cf75451909778522414748fb7c4a6a4b5509c0070c86e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-11T19:23:35.585 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:35 smithi170 podman[71732]: 2024-03-11 19:23:35.3000123 +0000 UTC m=+0.387495825 container start af66ddd0c507777517cf75451909778522414748fb7c4a6a4b5509c0070c86e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-03-11T19:23:35.586 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:35 smithi170 podman[71732]: 2024-03-11 19:23:35.430284191 +0000 UTC m=+0.517767742 container attach af66ddd0c507777517cf75451909778522414748fb7c4a6a4b5509c0070c86e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.license=GPLv2) 2024-03-11T19:23:35.847 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 11 19:23:35 smithi170 podman[71732]: 2024-03-11 19:23:35.582652529 +0000 UTC m=+0.670136051 container died af66ddd0c507777517cf75451909778522414748fb7c4a6a4b5509c0070c86e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397) 2024-03-11T19:23:35.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:35 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:35.606+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:35.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:35 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:35.606+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:35.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:35 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:35.606+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:35.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:35 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:35.606+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:35.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:35 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:35.606+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:36.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:35 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:35.638+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:36.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:35 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:35.638+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:36.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:35 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:35.638+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.879349+0000 front 2024-03-11T19:23:00.879230+0000 (oldest deadline 2024-03-11T19:23:23.779266+0000) 2024-03-11T19:23:36.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:35 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:35.638+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:09.080612+0000 front 2024-03-11T19:23:09.080491+0000 (oldest deadline 2024-03-11T19:23:31.380313+0000) 2024-03-11T19:23:36.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:35 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:35.572+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:36.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:35 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:35.572+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:36.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:35 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:35.572+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:36.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:35 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:35.572+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:36.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:35 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:35.572+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:36.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:35 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:35.743+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:36.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:35 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:35.743+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:36.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:35 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:35.743+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:36.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:35 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:35.743+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:36.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:35 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:35.743+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.679831+0000 front 2024-03-11T19:23:04.679922+0000 (oldest deadline 2024-03-11T19:23:29.979907+0000) 2024-03-11T19:23:36.270 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.7.service' 2024-03-11T19:23:36.313 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:23:36.314 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-11T19:23:36.314 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-11T19:23:36.314 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.8 2024-03-11T19:23:36.614 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:36 smithi190 systemd[1]: Stopping Ceph osd.8 for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:23:36.615 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:36 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:36.526+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:36.615 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:36 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:36.526+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:36.615 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:36 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:36.526+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:36.615 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:36 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:36.526+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:36.615 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:36 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:36.526+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:36.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:36 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:36.572+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:36.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:36 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:36.572+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:36.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:36 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:36.572+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:36.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:36 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:36.572+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:36.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:36 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:36.572+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:37.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:36 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:36.610+0000 7f4c354a2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:23:37.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:36 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:36.610+0000 7f4c354a2700 -1 osd.8 92 *** Got signal Terminated *** 2024-03-11T19:23:37.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:36 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:36.610+0000 7f4c354a2700 -1 osd.8 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T19:23:37.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:36 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:36.686+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:37.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:36 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:36.686+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:37.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:36 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:36.686+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.879349+0000 front 2024-03-11T19:23:00.879230+0000 (oldest deadline 2024-03-11T19:23:23.779266+0000) 2024-03-11T19:23:37.014 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:36 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:36.686+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:09.080612+0000 front 2024-03-11T19:23:09.080491+0000 (oldest deadline 2024-03-11T19:23:31.380313+0000) 2024-03-11T19:23:37.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:36 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:36.696+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:37.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:36 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:36.696+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:37.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:36 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:36.696+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:37.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:36 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:36.696+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:37.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:36 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:36.696+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.679831+0000 front 2024-03-11T19:23:04.679922+0000 (oldest deadline 2024-03-11T19:23:29.979907+0000) 2024-03-11T19:23:37.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:37 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:37.540+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:37.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:37 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:37.540+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:37.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:37 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:37.540+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:37.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:37 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:37.540+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:37.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:37 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:37.540+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:38.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:37 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:37.661+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:38.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:37 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:37.661+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:38.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:37 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:37.661+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.879349+0000 front 2024-03-11T19:23:00.879230+0000 (oldest deadline 2024-03-11T19:23:23.779266+0000) 2024-03-11T19:23:38.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:37 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:37.661+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:09.080612+0000 front 2024-03-11T19:23:09.080491+0000 (oldest deadline 2024-03-11T19:23:31.380313+0000) 2024-03-11T19:23:38.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:37 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:37.539+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:38.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:37 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:37.539+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:38.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:37 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:37.539+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:38.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:37 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:37.539+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:38.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:37 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:37.539+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:38.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:37 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:37.663+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:38.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:37 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:37.663+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:38.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:37 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:37.663+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:38.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:37 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:37.663+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:38.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:37 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:37.663+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.679831+0000 front 2024-03-11T19:23:04.679922+0000 (oldest deadline 2024-03-11T19:23:29.979907+0000) 2024-03-11T19:23:38.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:38 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:38.563+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:38.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:38 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:38.563+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:38.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:38 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:38.563+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:38.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:38 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:38.563+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:38.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:38 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:38.563+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:38.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:38 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:38.645+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:38.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:38 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:38.645+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:38.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:38 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:38.645+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:38.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:38 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:38.645+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:38.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:38 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:38.645+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.679831+0000 front 2024-03-11T19:23:04.679922+0000 (oldest deadline 2024-03-11T19:23:29.979907+0000) 2024-03-11T19:23:39.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:38 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:38.697+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:39.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:38 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:38.697+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:39.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:38 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:38.697+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.879349+0000 front 2024-03-11T19:23:00.879230+0000 (oldest deadline 2024-03-11T19:23:23.779266+0000) 2024-03-11T19:23:39.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:38 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:38.697+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:09.080612+0000 front 2024-03-11T19:23:09.080491+0000 (oldest deadline 2024-03-11T19:23:31.380313+0000) 2024-03-11T19:23:39.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:38 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:38.532+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:39.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:38 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:38.532+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:39.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:38 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:38.532+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:39.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:38 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:38.532+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:39.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:38 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:38.532+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:39.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:39 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:39.562+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:39.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:39 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:39.562+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:39.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:39 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:39.562+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:39.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:39 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:39.562+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:39.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:39 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:39.562+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:39.906 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:39 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:39.610+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:39.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:39 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:39.610+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:39.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:39 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:39.610+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:39.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:39 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:39.610+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:39.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:39 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:39.610+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.679831+0000 front 2024-03-11T19:23:04.679922+0000 (oldest deadline 2024-03-11T19:23:29.979907+0000) 2024-03-11T19:23:39.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:39 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:39.610+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:15.781599+0000 front 2024-03-11T19:23:15.781497+0000 (oldest deadline 2024-03-11T19:23:39.281472+0000) 2024-03-11T19:23:40.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:39 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:39.690+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:40.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:39 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:39.690+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:40.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:39 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:39.690+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.879349+0000 front 2024-03-11T19:23:00.879230+0000 (oldest deadline 2024-03-11T19:23:23.779266+0000) 2024-03-11T19:23:40.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:39 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:39.690+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:09.080612+0000 front 2024-03-11T19:23:09.080491+0000 (oldest deadline 2024-03-11T19:23:31.380313+0000) 2024-03-11T19:23:40.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:39 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:39.578+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:40.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:39 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:39.578+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:40.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:39 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:39.578+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:40.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:39 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:39.578+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:40.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:39 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:39.578+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:40.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:40 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:40.572+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:40.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:40 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:40.572+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:40.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:40 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:40.572+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:40.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:40 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:40.572+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:40.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:40 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:40.572+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:40.906 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:40 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:40.584+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:40.906 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:40 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:40.584+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:40.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:40 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:40.584+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:40.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:40 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:40.584+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:40.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:40 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:40.584+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.679831+0000 front 2024-03-11T19:23:04.679922+0000 (oldest deadline 2024-03-11T19:23:29.979907+0000) 2024-03-11T19:23:40.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:40 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:40.584+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:15.781599+0000 front 2024-03-11T19:23:15.781497+0000 (oldest deadline 2024-03-11T19:23:39.281472+0000) 2024-03-11T19:23:41.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:40 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:40.661+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:43.377019+0000 front 2024-03-11T19:22:43.377002+0000 (oldest deadline 2024-03-11T19:23:06.876988+0000) 2024-03-11T19:23:41.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:40 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:40.661+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.377840+0000 front 2024-03-11T19:22:50.377866+0000 (oldest deadline 2024-03-11T19:23:16.277971+0000) 2024-03-11T19:23:41.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:40 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:40.661+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.879349+0000 front 2024-03-11T19:23:00.879230+0000 (oldest deadline 2024-03-11T19:23:23.779266+0000) 2024-03-11T19:23:41.013 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:40 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8[54760]: 2024-03-11T19:23:40.661+0000 7f4c31a96700 -1 osd.8 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:09.080612+0000 front 2024-03-11T19:23:09.080491+0000 (oldest deadline 2024-03-11T19:23:31.380313+0000) 2024-03-11T19:23:41.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:40 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:40.590+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:41.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:40 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:40.590+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:41.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:40 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:40.590+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:41.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:40 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:40.590+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:41.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:40 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:40.590+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:41.798 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:41 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:41.539+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:41.799 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:41 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:41.539+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:41.799 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:41 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:41.539+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:41.799 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:41 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:41.539+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:41.799 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:41 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:41.539+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.679831+0000 front 2024-03-11T19:23:04.679922+0000 (oldest deadline 2024-03-11T19:23:29.979907+0000) 2024-03-11T19:23:41.799 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:41 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:41.539+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:15.781599+0000 front 2024-03-11T19:23:15.781497+0000 (oldest deadline 2024-03-11T19:23:39.281472+0000) 2024-03-11T19:23:41.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:41 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:41.547+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:41.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:41 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:41.547+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:41.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:41 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:41.547+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:41.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:41 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:41.547+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:41.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:41 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:41.547+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:42.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:41 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:41.557+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:42.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:41 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:41.557+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:42.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:41 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:41.557+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:42.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:41 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:41.557+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:42.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:41 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:41.557+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:42.348 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:42 smithi190 podman[72262]: 2024-03-11 19:23:42.046790858 +0000 UTC m=+5.543041655 container died caeb4ff240470aedb4f186b75a1c472674b11751b341449367a195ad7adb02c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS) 2024-03-11T19:23:42.656 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:42 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:42.584+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:42.657 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:42 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:42.584+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:42.657 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:42 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:42.584+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:42.657 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:42 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:42.584+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:42.657 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:42 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:42.584+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.679831+0000 front 2024-03-11T19:23:04.679922+0000 (oldest deadline 2024-03-11T19:23:29.979907+0000) 2024-03-11T19:23:42.657 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:42 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:42.584+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:15.781599+0000 front 2024-03-11T19:23:15.781497+0000 (oldest deadline 2024-03-11T19:23:39.281472+0000) 2024-03-11T19:23:42.698 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:42 smithi190 podman[72262]: 2024-03-11 19:23:42.443793235 +0000 UTC m=+5.940044031 container cleanup caeb4ff240470aedb4f186b75a1c472674b11751b341449367a195ad7adb02c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, io.buildah.version=1.33.5) 2024-03-11T19:23:42.698 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:42 smithi190 bash[72262]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8 2024-03-11T19:23:42.698 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:42 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:42.567+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:42.699 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:42 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:42.567+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:42.699 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:42 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:42.567+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:42.699 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:42 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:42.567+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:42.699 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:42 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:42.567+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:42.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:42 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:42.540+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:42.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:42 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:42.540+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:42.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:42 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:42.540+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:42.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:42 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:42.540+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:42.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:42 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:42.540+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:42.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:42 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:42.540+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:14.934775+0000 front 2024-03-11T19:23:14.934654+0000 (oldest deadline 2024-03-11T19:23:41.934810+0000) 2024-03-11T19:23:43.011 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:42 smithi190 podman[72279]: 2024-03-11 19:23:42.69520607 +0000 UTC m=+0.643433453 container remove caeb4ff240470aedb4f186b75a1c472674b11751b341449367a195ad7adb02c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-11T19:23:43.263 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:43 smithi190 podman[72369]: 2024-03-11 19:23:42.910261538 +0000 UTC m=+0.017703795 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:23:43.263 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:43 smithi190 podman[72369]: 2024-03-11 19:23:43.012759578 +0000 UTC m=+0.120201865 container create ea091a3ce4d2782b6ea1c13ad9837cb02c9e25cf721d858fad7d7ccc74437cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-94a5397, ceph=True, org.label-schema.license=GPLv2) 2024-03-11T19:23:43.263 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:43 smithi190 podman[72369]: 2024-03-11 19:23:43.256929399 +0000 UTC m=+0.364371661 container init ea091a3ce4d2782b6ea1c13ad9837cb02c9e25cf721d858fad7d7ccc74437cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-11T19:23:43.550 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:43 smithi190 podman[72369]: 2024-03-11 19:23:43.260179193 +0000 UTC m=+0.367621440 container start ea091a3ce4d2782b6ea1c13ad9837cb02c9e25cf721d858fad7d7ccc74437cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-11T19:23:43.550 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:43 smithi190 podman[72369]: 2024-03-11 19:23:43.348322794 +0000 UTC m=+0.455765056 container attach ea091a3ce4d2782b6ea1c13ad9837cb02c9e25cf721d858fad7d7ccc74437cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T19:23:43.846 INFO:journalctl@ceph.osd.8.smithi190.stdout:Mar 11 19:23:43 smithi190 podman[72369]: 2024-03-11 19:23:43.547634126 +0000 UTC m=+0.655076371 container died ea091a3ce4d2782b6ea1c13ad9837cb02c9e25cf721d858fad7d7ccc74437cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-8-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-11T19:23:43.846 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:43 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:43.561+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:43.846 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:43 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:43.561+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:43.846 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:43 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:43.561+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:43.846 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:43 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:43.561+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:43.846 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:43 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:43.561+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:43.846 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:43 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:43.561+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:23:43.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:43 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:43.565+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:43.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:43 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:43.565+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:43.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:43 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:43.565+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:43.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:43 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:43.565+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:43.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:43 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:43.565+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:43.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:43 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:43.565+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:14.934775+0000 front 2024-03-11T19:23:14.934654+0000 (oldest deadline 2024-03-11T19:23:41.934810+0000) 2024-03-11T19:23:43.906 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:43 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:43.604+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:43.906 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:43 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:43.604+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:43.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:43 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:43.604+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:43.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:43 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:43.604+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:43.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:43 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:43.604+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.679831+0000 front 2024-03-11T19:23:04.679922+0000 (oldest deadline 2024-03-11T19:23:29.979907+0000) 2024-03-11T19:23:43.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:43 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:43.604+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:15.781599+0000 front 2024-03-11T19:23:15.781497+0000 (oldest deadline 2024-03-11T19:23:39.281472+0000) 2024-03-11T19:23:44.123 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.8.service' 2024-03-11T19:23:44.165 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:23:44.165 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-11T19:23:44.165 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-11T19:23:44.165 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.9 2024-03-11T19:23:44.496 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:44 smithi029 systemd[1]: Stopping Ceph osd.9 for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:23:44.497 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:44 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:44.492+0000 7f1504f53700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:23:44.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:44 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:44.575+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:44.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:44 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:44.575+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:44.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:44 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:44.575+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:44.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:44 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:44.575+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:44.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:44 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:44.575+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:44.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:44 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:44.575+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:23:44.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:44 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:44.530+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:44.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:44 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:44.530+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:44.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:44 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:44.530+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:44.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:44 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:44.530+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:44.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:44 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:44.530+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:44.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:44 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:44.530+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:14.934775+0000 front 2024-03-11T19:23:14.934654+0000 (oldest deadline 2024-03-11T19:23:41.934810+0000) 2024-03-11T19:23:44.906 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:44 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:44.492+0000 7f1504f53700 -1 osd.9 92 *** Got signal Terminated *** 2024-03-11T19:23:44.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:44 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:44.492+0000 7f1504f53700 -1 osd.9 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T19:23:44.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:44 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:44.644+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:44.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:44 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:44.644+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:44.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:44 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:44.644+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:44.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:44 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:44.644+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:44.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:44 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:44.644+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.679831+0000 front 2024-03-11T19:23:04.679922+0000 (oldest deadline 2024-03-11T19:23:29.979907+0000) 2024-03-11T19:23:44.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:44 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:44.644+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:15.781599+0000 front 2024-03-11T19:23:15.781497+0000 (oldest deadline 2024-03-11T19:23:39.281472+0000) 2024-03-11T19:23:45.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:45 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:45.497+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:45.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:45 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:45.497+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:45.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:45 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:45.497+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:45.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:45 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:45.497+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:45.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:45 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:45.497+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:45.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:45 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:45.497+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:14.934775+0000 front 2024-03-11T19:23:14.934654+0000 (oldest deadline 2024-03-11T19:23:41.934810+0000) 2024-03-11T19:23:45.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:45 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:45.649+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:45.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:45 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:45.649+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:45.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:45 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:45.649+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:45.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:45 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:45.649+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:45.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:45 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:45.649+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.679831+0000 front 2024-03-11T19:23:04.679922+0000 (oldest deadline 2024-03-11T19:23:29.979907+0000) 2024-03-11T19:23:45.907 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:45 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:45.649+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:15.781599+0000 front 2024-03-11T19:23:15.781497+0000 (oldest deadline 2024-03-11T19:23:39.281472+0000) 2024-03-11T19:23:46.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:45 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:45.582+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:46.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:45 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:45.582+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:46.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:45 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:45.582+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:46.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:45 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:45.582+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:46.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:45 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:45.582+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:46.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:45 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:45.582+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:23:46.596 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:46 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:46.496+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:46.597 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:46 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:46.496+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:46.597 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:46 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:46.496+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:46.597 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:46 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:46.496+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:46.597 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:46 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:46.496+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:46.597 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:46 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:46.496+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:14.934775+0000 front 2024-03-11T19:23:14.934654+0000 (oldest deadline 2024-03-11T19:23:41.934810+0000) 2024-03-11T19:23:47.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:46 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:46.572+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:47.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:46 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:46.572+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:47.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:46 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:46.572+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:47.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:46 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:46.572+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:47.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:46 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:46.572+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:47.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:46 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:46.572+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:23:47.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:46 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:46.682+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:47.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:46 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:46.682+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:47.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:46 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:46.682+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:47.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:46 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:46.682+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:47.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:46 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:46.682+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.679831+0000 front 2024-03-11T19:23:04.679922+0000 (oldest deadline 2024-03-11T19:23:29.979907+0000) 2024-03-11T19:23:47.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:46 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:46.682+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:15.781599+0000 front 2024-03-11T19:23:15.781497+0000 (oldest deadline 2024-03-11T19:23:39.281472+0000) 2024-03-11T19:23:47.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:47 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:47.475+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:47.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:47 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:47.475+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:47.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:47 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:47.475+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:47.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:47 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:47.475+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:47.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:47 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:47.475+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:47.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:47 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:47.475+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:14.934775+0000 front 2024-03-11T19:23:14.934654+0000 (oldest deadline 2024-03-11T19:23:41.934810+0000) 2024-03-11T19:23:47.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:47 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:47.475+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-11T19:23:21.935388+0000 front 2024-03-11T19:23:21.935406+0000 (oldest deadline 2024-03-11T19:23:47.235235+0000) 2024-03-11T19:23:48.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:47 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:47.608+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:48.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:47 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:47.608+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:48.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:47 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:47.608+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:48.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:47 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:47.608+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:48.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:47 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:47.608+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:48.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:47 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:47.608+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:23:48.156 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:47 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:47.723+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:48.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:47 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:47.723+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:48.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:47 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:47.723+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:48.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:47 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:47.723+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:48.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:47 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:47.723+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.679831+0000 front 2024-03-11T19:23:04.679922+0000 (oldest deadline 2024-03-11T19:23:29.979907+0000) 2024-03-11T19:23:48.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:47 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:47.723+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:15.781599+0000 front 2024-03-11T19:23:15.781497+0000 (oldest deadline 2024-03-11T19:23:39.281472+0000) 2024-03-11T19:23:48.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:48 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:48.488+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:48.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:48 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:48.488+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:48.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:48 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:48.488+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:48.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:48 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:48.488+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:48.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:48 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:48.488+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:48.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:48 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:48.488+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:14.934775+0000 front 2024-03-11T19:23:14.934654+0000 (oldest deadline 2024-03-11T19:23:41.934810+0000) 2024-03-11T19:23:48.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:48 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:48.488+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-11T19:23:21.935388+0000 front 2024-03-11T19:23:21.935406+0000 (oldest deadline 2024-03-11T19:23:47.235235+0000) 2024-03-11T19:23:49.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:48 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:48.634+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:49.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:48 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:48.634+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:49.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:48 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:48.634+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:49.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:48 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:48.634+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:49.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:48 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:48.634+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:49.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:48 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:48.634+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:23:49.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:48 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:48.760+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:34.772971+0000 front 2024-03-11T19:22:34.773073+0000 (oldest deadline 2024-03-11T19:23:00.073000+0000) 2024-03-11T19:23:49.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:48 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:48.760+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:40.073646+0000 front 2024-03-11T19:22:40.073690+0000 (oldest deadline 2024-03-11T19:23:05.373461+0000) 2024-03-11T19:23:49.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:48 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:48.760+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:52.176667+0000 front 2024-03-11T19:22:52.176535+0000 (oldest deadline 2024-03-11T19:23:13.876565+0000) 2024-03-11T19:23:49.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:48 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:48.760+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:00.678843+0000 front 2024-03-11T19:23:00.678870+0000 (oldest deadline 2024-03-11T19:23:22.978848+0000) 2024-03-11T19:23:49.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:48 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:48.760+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.679831+0000 front 2024-03-11T19:23:04.679922+0000 (oldest deadline 2024-03-11T19:23:29.979907+0000) 2024-03-11T19:23:49.157 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:48 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9[67329]: 2024-03-11T19:23:48.760+0000 7f1501547700 -1 osd.9 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:15.781599+0000 front 2024-03-11T19:23:15.781497+0000 (oldest deadline 2024-03-11T19:23:39.281472+0000) 2024-03-11T19:23:49.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:49 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:49.465+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:49.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:49 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:49.465+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:49.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:49 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:49.465+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:49.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:49 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:49.465+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:49.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:49 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:49.465+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:49.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:49 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:49.465+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:14.934775+0000 front 2024-03-11T19:23:14.934654+0000 (oldest deadline 2024-03-11T19:23:41.934810+0000) 2024-03-11T19:23:49.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:49 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:49.465+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-11T19:23:21.935388+0000 front 2024-03-11T19:23:21.935406+0000 (oldest deadline 2024-03-11T19:23:47.235235+0000) 2024-03-11T19:23:50.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:49 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:49.667+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:50.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:49 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:49.667+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:50.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:49 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:49.667+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:50.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:49 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:49.667+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:50.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:49 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:49.667+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:50.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:49 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:49.667+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:23:50.326 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:50 smithi029 podman[108421]: 2024-03-11 19:23:50.03476701 +0000 UTC m=+5.668935237 container died b37105f74307948d7386442a30bdb1679515940e79b3dadf9f5c5c62935e193b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T19:23:50.657 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:50 smithi029 podman[108421]: 2024-03-11 19:23:50.404646266 +0000 UTC m=+6.038814491 container cleanup b37105f74307948d7386442a30bdb1679515940e79b3dadf9f5c5c62935e193b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-11T19:23:50.657 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:50 smithi029 bash[108421]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9 2024-03-11T19:23:50.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:50 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:50.452+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:50.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:50 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:50.452+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:50.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:50 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:50.452+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:50.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:50 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:50.452+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:50.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:50 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:50.452+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:50.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:50 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:50.452+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:14.934775+0000 front 2024-03-11T19:23:14.934654+0000 (oldest deadline 2024-03-11T19:23:41.934810+0000) 2024-03-11T19:23:50.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:50 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:50.452+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-11T19:23:21.935388+0000 front 2024-03-11T19:23:21.935406+0000 (oldest deadline 2024-03-11T19:23:47.235235+0000) 2024-03-11T19:23:50.996 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:50 smithi029 podman[108439]: 2024-03-11 19:23:50.698625335 +0000 UTC m=+0.653969657 container remove b37105f74307948d7386442a30bdb1679515940e79b3dadf9f5c5c62935e193b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T19:23:51.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:50 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:50.623+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:51.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:50 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:50.623+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:51.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:50 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:50.623+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:51.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:50 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:50.623+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:51.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:50 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:50.623+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:51.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:50 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:50.623+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:23:51.301 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:50 smithi029 podman[108533]: 2024-03-11 19:23:50.993503032 +0000 UTC m=+0.114708413 container create 3896ca9b1ddbb49f1a8b6ee51ef9df2249e330c8a5e772d669c4f4e1b673d988 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9-deactivate, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-11T19:23:51.301 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:50 smithi029 podman[108533]: 2024-03-11 19:23:50.897015663 +0000 UTC m=+0.018221002 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:23:51.301 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:51 smithi029 podman[108533]: 2024-03-11 19:23:51.214513262 +0000 UTC m=+0.335718600 container init 3896ca9b1ddbb49f1a8b6ee51ef9df2249e330c8a5e772d669c4f4e1b673d988 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-11T19:23:51.301 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:51 smithi029 podman[108533]: 2024-03-11 19:23:51.217903774 +0000 UTC m=+0.339109119 container start 3896ca9b1ddbb49f1a8b6ee51ef9df2249e330c8a5e772d669c4f4e1b673d988 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-11T19:23:51.301 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:51 smithi029 podman[108533]: 2024-03-11 19:23:51.298008419 +0000 UTC m=+0.419213774 container attach 3896ca9b1ddbb49f1a8b6ee51ef9df2249e330c8a5e772d669c4f4e1b673d988 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-03-11T19:23:51.656 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 11 19:23:51 smithi029 podman[108533]: 2024-03-11 19:23:51.51172426 +0000 UTC m=+0.632929599 container died 3896ca9b1ddbb49f1a8b6ee51ef9df2249e330c8a5e772d669c4f4e1b673d988 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-9-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:23:51.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:51 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:51.501+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:51.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:51 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:51.501+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:51.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:51 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:51.501+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:51.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:51 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:51.501+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:51.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:51 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:51.501+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:51.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:51 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:51.501+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:14.934775+0000 front 2024-03-11T19:23:14.934654+0000 (oldest deadline 2024-03-11T19:23:41.934810+0000) 2024-03-11T19:23:51.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:51 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:51.501+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-11T19:23:21.935388+0000 front 2024-03-11T19:23:21.935406+0000 (oldest deadline 2024-03-11T19:23:47.235235+0000) 2024-03-11T19:23:52.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:51 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:51.595+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:52.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:51 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:51.595+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:52.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:51 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:51.595+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:52.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:51 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:51.595+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:52.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:51 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:51.595+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:52.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:51 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:51.595+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:23:52.103 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.9.service' 2024-03-11T19:23:52.145 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:23:52.145 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-11T19:23:52.145 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-11T19:23:52.145 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.10 2024-03-11T19:23:52.465 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:52 smithi170 systemd[1]: Stopping Ceph osd.10 for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:23:52.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:52 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:52.460+0000 7f6586a65700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:23:52.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:52 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:52.460+0000 7f6586a65700 -1 osd.10 92 *** Got signal Terminated *** 2024-03-11T19:23:52.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:52 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:52.460+0000 7f6586a65700 -1 osd.10 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T19:23:52.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:52 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:52.505+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:52.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:52 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:52.505+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:52.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:52 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:52.505+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:52.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:52 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:52.505+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:52.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:52 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:52.505+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:52.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:52 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:52.505+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:14.934775+0000 front 2024-03-11T19:23:14.934654+0000 (oldest deadline 2024-03-11T19:23:41.934810+0000) 2024-03-11T19:23:52.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:52 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:52.505+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-11T19:23:21.935388+0000 front 2024-03-11T19:23:21.935406+0000 (oldest deadline 2024-03-11T19:23:47.235235+0000) 2024-03-11T19:23:53.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:52 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:52.624+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:53.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:52 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:52.624+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:53.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:52 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:52.624+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:53.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:52 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:52.625+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:53.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:52 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:52.625+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:53.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:52 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:52.625+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:23:53.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:53 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:53.486+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:53.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:53 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:53.486+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:53.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:53 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:53.486+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:53.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:53 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:53.486+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:53.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:53 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:53.486+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:53.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:53 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:53.486+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:14.934775+0000 front 2024-03-11T19:23:14.934654+0000 (oldest deadline 2024-03-11T19:23:41.934810+0000) 2024-03-11T19:23:53.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:53 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:53.486+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-11T19:23:21.935388+0000 front 2024-03-11T19:23:21.935406+0000 (oldest deadline 2024-03-11T19:23:47.235235+0000) 2024-03-11T19:23:54.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:53 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:53.604+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:54.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:53 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:53.604+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:54.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:53 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:53.604+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:54.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:53 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:53.604+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:54.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:53 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:53.604+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:54.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:53 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:53.604+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:23:54.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:54 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:54.492+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:54.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:54 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:54.492+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:54.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:54 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:54.492+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:54.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:54 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:54.492+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:54.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:54 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:54.492+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:54.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:54 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:54.492+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:14.934775+0000 front 2024-03-11T19:23:14.934654+0000 (oldest deadline 2024-03-11T19:23:41.934810+0000) 2024-03-11T19:23:54.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:54 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:54.492+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-11T19:23:21.935388+0000 front 2024-03-11T19:23:21.935406+0000 (oldest deadline 2024-03-11T19:23:47.235235+0000) 2024-03-11T19:23:55.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:54 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:54.580+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:55.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:54 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:54.580+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:55.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:54 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:54.580+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:55.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:54 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:54.580+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:55.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:54 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:54.580+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:55.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:54 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:54.580+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:23:55.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:55 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:55.542+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:55.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:55 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:55.542+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:55.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:55 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:55.542+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:55.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:55 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:55.542+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:55.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:55 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:55.542+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:55.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:55 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:55.542+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:23:55.764 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:55 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:55.542+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-03-11T19:23:31.497538+0000 front 2024-03-11T19:23:31.497574+0000 (oldest deadline 2024-03-11T19:23:54.997421+0000) 2024-03-11T19:23:55.846 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:55 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:55.533+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:55.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:55 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:55.533+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:55.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:55 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:55.533+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:55.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:55 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:55.533+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:55.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:55 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:55.533+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:55.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:55 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:55.533+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:14.934775+0000 front 2024-03-11T19:23:14.934654+0000 (oldest deadline 2024-03-11T19:23:41.934810+0000) 2024-03-11T19:23:55.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:55 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:55.533+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-11T19:23:21.935388+0000 front 2024-03-11T19:23:21.935406+0000 (oldest deadline 2024-03-11T19:23:47.235235+0000) 2024-03-11T19:23:56.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:56 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:56.582+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:36.127881+0000 front 2024-03-11T19:22:36.127796+0000 (oldest deadline 2024-03-11T19:23:00.827932+0000) 2024-03-11T19:23:56.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:56 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:56.582+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:41.829367+0000 front 2024-03-11T19:22:41.829448+0000 (oldest deadline 2024-03-11T19:23:07.129490+0000) 2024-03-11T19:23:56.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:56 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:56.582+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:49.931236+0000 front 2024-03-11T19:22:49.931176+0000 (oldest deadline 2024-03-11T19:23:14.031072+0000) 2024-03-11T19:23:56.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:56 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:56.582+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:23:01.032710+0000 front 2024-03-11T19:23:01.032673+0000 (oldest deadline 2024-03-11T19:23:24.532524+0000) 2024-03-11T19:23:56.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:56 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:56.582+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:04.532781+0000 front 2024-03-11T19:23:04.532811+0000 (oldest deadline 2024-03-11T19:23:33.933250+0000) 2024-03-11T19:23:56.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:56 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:56.582+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:14.934775+0000 front 2024-03-11T19:23:14.934654+0000 (oldest deadline 2024-03-11T19:23:41.934810+0000) 2024-03-11T19:23:56.847 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:56 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10[60714]: 2024-03-11T19:23:56.582+0000 7f6583059700 -1 osd.10 92 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-11T19:23:21.935388+0000 front 2024-03-11T19:23:21.935406+0000 (oldest deadline 2024-03-11T19:23:47.235235+0000) 2024-03-11T19:23:57.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:56 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:56.590+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:57.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:56 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:56.590+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:57.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:56 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:56.590+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:57.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:56 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:56.590+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:57.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:56 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:56.590+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:57.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:56 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:56.590+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:23:57.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:56 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:56.590+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-03-11T19:23:31.497538+0000 front 2024-03-11T19:23:31.497574+0000 (oldest deadline 2024-03-11T19:23:54.997421+0000) 2024-03-11T19:23:58.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:57.598+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:58.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:57.598+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:58.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:57.598+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:58.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:57.598+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:58.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:57.598+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:58.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:57.598+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:23:58.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:57 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:57.598+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-03-11T19:23:31.497538+0000 front 2024-03-11T19:23:31.497574+0000 (oldest deadline 2024-03-11T19:23:54.997421+0000) 2024-03-11T19:23:58.315 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:58 smithi170 podman[71839]: 2024-03-11 19:23:58.042576433 +0000 UTC m=+5.706162436 container died 7f69282935bdcff3046c28b8060cafd8a694ac9b8596dd2108b2251cf27cc517 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-11T19:23:58.596 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:58 smithi170 podman[71839]: 2024-03-11 19:23:58.403159673 +0000 UTC m=+6.066745676 container cleanup 7f69282935bdcff3046c28b8060cafd8a694ac9b8596dd2108b2251cf27cc517 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10, RELEASE=squid-94a5397, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307) 2024-03-11T19:23:58.596 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:58 smithi170 bash[71839]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10 2024-03-11T19:23:59.000 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:58 smithi170 podman[71856]: 2024-03-11 19:23:58.687537274 +0000 UTC m=+0.637581106 container remove 7f69282935bdcff3046c28b8060cafd8a694ac9b8596dd2108b2251cf27cc517 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:23:59.012 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:58 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:58.587+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:59.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:58 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:58.587+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:59.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:58 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:58.587+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:59.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:58 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:58.587+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:59.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:58 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:58.587+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:59.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:58 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:58.587+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:23:59.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:58 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:58.587+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-03-11T19:23:31.497538+0000 front 2024-03-11T19:23:31.497574+0000 (oldest deadline 2024-03-11T19:23:54.997421+0000) 2024-03-11T19:23:59.296 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:58 smithi170 podman[71950]: 2024-03-11 19:23:58.99693554 +0000 UTC m=+0.103062968 container create eb8769576c27fbe44c5c7acd9653c5ab40b3188c11e066c8f4525a15247cfc96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-11T19:23:59.296 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:59 smithi170 podman[71950]: 2024-03-11 19:23:58.911511354 +0000 UTC m=+0.017638752 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:23:59.296 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:59 smithi170 podman[71950]: 2024-03-11 19:23:59.209006026 +0000 UTC m=+0.315133411 container init eb8769576c27fbe44c5c7acd9653c5ab40b3188c11e066c8f4525a15247cfc96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:23:59.296 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:59 smithi170 podman[71950]: 2024-03-11 19:23:59.211957376 +0000 UTC m=+0.318084763 container start eb8769576c27fbe44c5c7acd9653c5ab40b3188c11e066c8f4525a15247cfc96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:23:59.296 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:59 smithi170 podman[71950]: 2024-03-11 19:23:59.292920504 +0000 UTC m=+0.399047893 container attach eb8769576c27fbe44c5c7acd9653c5ab40b3188c11e066c8f4525a15247cfc96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:23:59.596 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:23:59 smithi170 podman[71950]: 2024-03-11 19:23:59.511540589 +0000 UTC m=+0.617667975 container died eb8769576c27fbe44c5c7acd9653c5ab40b3188c11e066c8f4525a15247cfc96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-94a5397, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-11T19:23:59.914 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:59 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:59.593+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:23:59.915 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:59 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:59.593+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:23:59.915 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:59 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:59.593+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:23:59.915 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:59 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:59.593+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:23:59.915 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:59 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:59.593+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:23:59.915 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:59 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:59.593+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:23:59.915 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:23:59 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:23:59.593+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-03-11T19:23:31.497538+0000 front 2024-03-11T19:23:31.497574+0000 (oldest deadline 2024-03-11T19:23:54.997421+0000) 2024-03-11T19:24:00.069 INFO:journalctl@ceph.osd.10.smithi170.stdout:Mar 11 19:24:00 smithi170 podman[71976]: 2024-03-11 19:24:00.046887381 +0000 UTC m=+0.530796783 container remove eb8769576c27fbe44c5c7acd9653c5ab40b3188c11e066c8f4525a15247cfc96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-10-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-11T19:24:00.074 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.10.service' 2024-03-11T19:24:00.117 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:24:00.117 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-11T19:24:00.117 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-11T19:24:00.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.11 2024-03-11T19:24:00.466 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:00 smithi190 systemd[1]: Stopping Ceph osd.11 for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:24:00.466 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:00 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:00.462+0000 7f05a5411700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:24:00.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:00 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:00.462+0000 7f05a5411700 -1 osd.11 92 *** Got signal Terminated *** 2024-03-11T19:24:00.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:00 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:00.462+0000 7f05a5411700 -1 osd.11 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T19:24:00.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:00 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:00.597+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:24:00.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:00 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:00.597+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:24:00.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:00 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:00.597+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:24:00.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:00 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:00.597+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:24:00.763 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:00 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:00.597+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:24:00.764 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:00 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:00.597+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:24:00.764 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:00 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:00.597+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-03-11T19:23:31.497538+0000 front 2024-03-11T19:23:31.497574+0000 (oldest deadline 2024-03-11T19:23:54.997421+0000) 2024-03-11T19:24:02.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:01 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:01.613+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:24:02.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:01 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:01.613+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:24:02.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:01 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:01.613+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:24:02.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:01 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:01.613+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:24:02.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:01 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:01.613+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:24:02.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:01 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:01.613+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:24:02.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:01 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:01.613+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-03-11T19:23:31.497538+0000 front 2024-03-11T19:23:31.497574+0000 (oldest deadline 2024-03-11T19:23:54.997421+0000) 2024-03-11T19:24:03.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:02 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:02.617+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:24:03.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:02 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:02.617+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:24:03.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:02 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:02.617+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:24:03.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:02 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:02.617+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:24:03.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:02 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:02.617+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:24:03.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:02 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:02.617+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:24:03.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:02 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:02.617+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-03-11T19:23:31.497538+0000 front 2024-03-11T19:23:31.497574+0000 (oldest deadline 2024-03-11T19:23:54.997421+0000) 2024-03-11T19:24:04.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:03 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:03.597+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:24:04.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:03 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:03.597+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:24:04.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:03 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:03.597+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:24:04.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:03 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:03.597+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:24:04.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:03 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:03.597+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:24:04.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:03 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:03.597+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:24:04.014 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:03 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:03.597+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-03-11T19:23:31.497538+0000 front 2024-03-11T19:23:31.497574+0000 (oldest deadline 2024-03-11T19:23:54.997421+0000) 2024-03-11T19:24:05.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:04 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:04.594+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-11T19:22:33.886138+0000 front 2024-03-11T19:22:33.886147+0000 (oldest deadline 2024-03-11T19:22:56.786105+0000) 2024-03-11T19:24:05.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:04 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:04.594+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-03-11T19:22:44.689551+0000 front 2024-03-11T19:22:44.689567+0000 (oldest deadline 2024-03-11T19:23:09.389477+0000) 2024-03-11T19:24:05.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:04 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:04.594+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2024-03-11T19:22:50.490539+0000 front 2024-03-11T19:22:50.490315+0000 (oldest deadline 2024-03-11T19:23:13.990391+0000) 2024-03-11T19:24:05.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:04 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:04.594+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-11T19:22:59.692494+0000 front 2024-03-11T19:22:59.692285+0000 (oldest deadline 2024-03-11T19:23:22.592360+0000) 2024-03-11T19:24:05.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:04 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:04.594+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-03-11T19:23:07.794077+0000 front 2024-03-11T19:23:07.794171+0000 (oldest deadline 2024-03-11T19:23:31.893961+0000) 2024-03-11T19:24:05.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:04 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:04.594+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-03-11T19:23:11.894549+0000 front 2024-03-11T19:23:11.894424+0000 (oldest deadline 2024-03-11T19:23:43.094951+0000) 2024-03-11T19:24:05.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:04 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:04.594+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-03-11T19:23:31.497538+0000 front 2024-03-11T19:23:31.497574+0000 (oldest deadline 2024-03-11T19:23:54.997421+0000) 2024-03-11T19:24:05.013 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:04 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11[61347]: 2024-03-11T19:24:04.594+0000 7f059d9fd700 -1 osd.11 92 heartbeat_check: no reply from 172.21.15.190:6820 osd.8 since back 2024-03-11T19:23:39.098279+0000 front 2024-03-11T19:23:39.098315+0000 (oldest deadline 2024-03-11T19:24:04.398306+0000) 2024-03-11T19:24:06.215 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:05 smithi190 podman[72486]: 2024-03-11 19:24:05.928308891 +0000 UTC m=+5.583828815 container died 609c688b2b95a51c2f599dbe8997d9dab0bd744d8c7ec8ba3ab28c70b43a0ba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-11T19:24:06.215 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:06 smithi190 podman[72486]: 2024-03-11 19:24:06.212083837 +0000 UTC m=+5.867603820 container cleanup 609c688b2b95a51c2f599dbe8997d9dab0bd744d8c7ec8ba3ab28c70b43a0ba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0) 2024-03-11T19:24:06.466 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:06 smithi190 bash[72486]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11 2024-03-11T19:24:06.762 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:06 smithi190 podman[72514]: 2024-03-11 19:24:06.463341056 +0000 UTC m=+0.536182651 container remove 609c688b2b95a51c2f599dbe8997d9dab0bd744d8c7ec8ba3ab28c70b43a0ba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-11T19:24:07.119 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:06 smithi190 podman[72606]: 2024-03-11 19:24:06.797336043 +0000 UTC m=+0.094677028 container create a72786f0395930e020bd8f2f70e395581ca3ace0e77018d507ae0b7aad2b11d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-11T19:24:07.119 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:06 smithi190 podman[72606]: 2024-03-11 19:24:06.720509806 +0000 UTC m=+0.017850795 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:24:07.120 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:07 smithi190 podman[72606]: 2024-03-11 19:24:07.033297231 +0000 UTC m=+0.330638226 container init a72786f0395930e020bd8f2f70e395581ca3ace0e77018d507ae0b7aad2b11d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T19:24:07.120 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:07 smithi190 podman[72606]: 2024-03-11 19:24:07.036566999 +0000 UTC m=+0.333907994 container start a72786f0395930e020bd8f2f70e395581ca3ace0e77018d507ae0b7aad2b11d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:24:07.120 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:07 smithi190 podman[72606]: 2024-03-11 19:24:07.116066341 +0000 UTC m=+0.413407341 container attach a72786f0395930e020bd8f2f70e395581ca3ace0e77018d507ae0b7aad2b11d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True) 2024-03-11T19:24:07.513 INFO:journalctl@ceph.osd.11.smithi190.stdout:Mar 11 19:24:07 smithi190 podman[72606]: 2024-03-11 19:24:07.31702962 +0000 UTC m=+0.614370613 container died a72786f0395930e020bd8f2f70e395581ca3ace0e77018d507ae0b7aad2b11d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-11T19:24:07.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@osd.11.service' 2024-03-11T19:24:07.929 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:24:07.929 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-11T19:24:07.929 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-11T19:24:07.929 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.a 2024-03-11T19:24:08.231 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 11 19:24:07 smithi029 systemd[1]: Stopping Ceph mds.a for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:24:08.656 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 11 19:24:08 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-a[74043]: 2024-03-11T19:24:08.226+0000 7fe3263a9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:24:08.657 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 11 19:24:08 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-a[74043]: 2024-03-11T19:24:08.226+0000 7fe3263a9700 -1 mds.a *** got signal Terminated *** 2024-03-11T19:24:11.734 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 11 19:24:11 smithi029 podman[108653]: 2024-03-11 19:24:11.428560359 +0000 UTC m=+3.300276873 container died bc90ce3586d6099f1a60853a1c77d8b636da4539ed540fdff18d31e435c94e03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-11T19:24:12.086 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 11 19:24:11 smithi029 podman[108653]: 2024-03-11 19:24:11.821802903 +0000 UTC m=+3.693519478 container cleanup bc90ce3586d6099f1a60853a1c77d8b636da4539ed540fdff18d31e435c94e03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-a, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-11T19:24:12.086 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 11 19:24:11 smithi029 bash[108653]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-a 2024-03-11T19:24:12.267 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.a.service' 2024-03-11T19:24:12.304 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:24:12.304 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-11T19:24:12.305 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-11T19:24:12.305 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.d 2024-03-11T19:24:12.631 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 11 19:24:12 smithi029 systemd[1]: Stopping Ceph mds.d for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:24:12.906 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 11 19:24:12 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-d[73073]: 2024-03-11T19:24:12.627+0000 7f56de142700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:24:12.907 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 11 19:24:12 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-d[73073]: 2024-03-11T19:24:12.627+0000 7f56de142700 -1 mds.d *** got signal Terminated *** 2024-03-11T19:24:14.956 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 11 19:24:14 smithi029 podman[108775]: 2024-03-11 19:24:14.686477935 +0000 UTC m=+2.164886159 container died 3e10f89d8bfd0151fc03c71070937771920a284616fc6032bc09f14c421608d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T19:24:15.333 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 11 19:24:15 smithi029 podman[108775]: 2024-03-11 19:24:15.047565381 +0000 UTC m=+2.525973598 container cleanup 3e10f89d8bfd0151fc03c71070937771920a284616fc6032bc09f14c421608d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:24:15.333 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 11 19:24:15 smithi029 bash[108775]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-d 2024-03-11T19:24:15.495 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.d.service' 2024-03-11T19:24:15.531 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:24:15.531 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-11T19:24:15.531 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-11T19:24:15.531 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.g 2024-03-11T19:24:15.861 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 11 19:24:15 smithi029 systemd[1]: Stopping Ceph mds.g for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:24:16.156 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 11 19:24:15 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-g[73682]: 2024-03-11T19:24:15.857+0000 7f5d18d46700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:24:16.157 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 11 19:24:15 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-g[73682]: 2024-03-11T19:24:15.857+0000 7f5d18d46700 -1 mds.g *** got signal Terminated *** 2024-03-11T19:24:19.680 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 11 19:24:19 smithi029 podman[108897]: 2024-03-11 19:24:19.416686877 +0000 UTC m=+3.669017598 container died f31bf8334228613f858d1d2aa4bf491798ee0e5c8faacbd823aa1250b0851e75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-g, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T19:24:19.975 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 11 19:24:19 smithi029 podman[108897]: 2024-03-11 19:24:19.768257165 +0000 UTC m=+4.020587898 container cleanup f31bf8334228613f858d1d2aa4bf491798ee0e5c8faacbd823aa1250b0851e75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-g, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-11T19:24:19.975 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 11 19:24:19 smithi029 bash[108897]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-g 2024-03-11T19:24:20.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.g.service' 2024-03-11T19:24:20.225 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 11 19:24:19 smithi029 podman[108911]: 2024-03-11 19:24:19.972187227 +0000 UTC m=+0.549246359 container remove f31bf8334228613f858d1d2aa4bf491798ee0e5c8faacbd823aa1250b0851e75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-g, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-11T19:24:20.225 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 11 19:24:20 smithi029 systemd[1]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.g.service: Deactivated successfully. 2024-03-11T19:24:20.225 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 11 19:24:20 smithi029 systemd[1]: Stopped Ceph mds.g for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:24:20.225 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 11 19:24:20 smithi029 systemd[1]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.g.service: Consumed 2.086s CPU time. 2024-03-11T19:24:20.234 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:24:20.234 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-11T19:24:20.235 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-11T19:24:20.235 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.j 2024-03-11T19:24:20.553 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 11 19:24:20 smithi029 systemd[1]: Stopping Ceph mds.j for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:24:20.906 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 11 19:24:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-j[72682]: 2024-03-11T19:24:20.549+0000 7fa9fe0e3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:24:20.907 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 11 19:24:20 smithi029 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-j[72682]: 2024-03-11T19:24:20.549+0000 7fa9fe0e3700 -1 mds.j *** got signal Terminated *** 2024-03-11T19:24:23.996 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 11 19:24:23 smithi029 podman[109017]: 2024-03-11 19:24:23.727938368 +0000 UTC m=+3.273093621 container died bbbf59d44f5797c2c253c2a3e709e3df3e722e40ca41fb437042c09ffb3c4147 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-j, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:24:24.306 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 11 19:24:24 smithi029 podman[109017]: 2024-03-11 19:24:24.101705273 +0000 UTC m=+3.646860512 container cleanup bbbf59d44f5797c2c253c2a3e709e3df3e722e40ca41fb437042c09ffb3c4147 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-j, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-94a5397) 2024-03-11T19:24:24.306 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 11 19:24:24 smithi029 bash[109017]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-j 2024-03-11T19:24:24.532 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.j.service' 2024-03-11T19:24:24.557 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 11 19:24:24 smithi029 podman[109031]: 2024-03-11 19:24:24.303542845 +0000 UTC m=+0.563632114 container remove bbbf59d44f5797c2c253c2a3e709e3df3e722e40ca41fb437042c09ffb3c4147 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-j, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-11T19:24:24.557 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 11 19:24:24 smithi029 systemd[1]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.j.service: Deactivated successfully. 2024-03-11T19:24:24.557 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 11 19:24:24 smithi029 systemd[1]: Stopped Ceph mds.j for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:24:24.557 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 11 19:24:24 smithi029 systemd[1]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.j.service: Consumed 1.052s CPU time. 2024-03-11T19:24:24.566 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:24:24.566 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-11T19:24:24.566 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-11T19:24:24.566 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.b 2024-03-11T19:24:24.846 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:24:24 smithi170 systemd[1]: Stopping Ceph mds.b for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:24:25.346 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:24:24 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-b[66380]: 2024-03-11T19:24:24.881+0000 7f73b29df700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:24:25.347 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:24:24 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-b[66380]: 2024-03-11T19:24:24.881+0000 7f73b29df700 -1 mds.b *** got signal Terminated *** 2024-03-11T19:24:27.802 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:24:27 smithi170 podman[72069]: 2024-03-11 19:24:27.40173702 +0000 UTC m=+2.645910782 container died 32ee158abb0682e2a718c2a286f999af1e4cd98bb3b7e02dc349428721a37fdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-b, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-11T19:24:28.096 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:24:27 smithi170 podman[72069]: 2024-03-11 19:24:27.889287055 +0000 UTC m=+3.133460803 container cleanup 32ee158abb0682e2a718c2a286f999af1e4cd98bb3b7e02dc349428721a37fdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-b, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-11T19:24:28.097 INFO:journalctl@ceph.mds.b.smithi170.stdout:Mar 11 19:24:27 smithi170 bash[72069]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-b 2024-03-11T19:24:28.268 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.b.service' 2024-03-11T19:24:28.304 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:24:28.304 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-11T19:24:28.304 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-11T19:24:28.304 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.e 2024-03-11T19:24:28.596 INFO:journalctl@ceph.mds.e.smithi170.stdout:Mar 11 19:24:28 smithi170 systemd[1]: Stopping Ceph mds.e for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:24:29.096 INFO:journalctl@ceph.mds.e.smithi170.stdout:Mar 11 19:24:28 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-e[65760]: 2024-03-11T19:24:28.600+0000 7f3c55d06700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:24:29.096 INFO:journalctl@ceph.mds.e.smithi170.stdout:Mar 11 19:24:28 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-e[65760]: 2024-03-11T19:24:28.600+0000 7f3c55d06700 -1 mds.e *** got signal Terminated *** 2024-03-11T19:24:31.596 INFO:journalctl@ceph.mds.e.smithi170.stdout:Mar 11 19:24:31 smithi170 podman[72191]: 2024-03-11 19:24:31.398538353 +0000 UTC m=+2.875649703 container died ce0f0e897c09006ae16ec94a50e179ee076aaa86c799d91a54ba86a9b5ff94fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-e, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-11T19:24:32.023 INFO:journalctl@ceph.mds.e.smithi170.stdout:Mar 11 19:24:31 smithi170 podman[72191]: 2024-03-11 19:24:31.809176773 +0000 UTC m=+3.286288134 container cleanup ce0f0e897c09006ae16ec94a50e179ee076aaa86c799d91a54ba86a9b5ff94fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-e, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-11T19:24:32.023 INFO:journalctl@ceph.mds.e.smithi170.stdout:Mar 11 19:24:31 smithi170 bash[72191]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-e 2024-03-11T19:24:32.234 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.e.service' 2024-03-11T19:24:32.272 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:24:32.272 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-11T19:24:32.272 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-11T19:24:32.272 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.h 2024-03-11T19:24:32.596 INFO:journalctl@ceph.mds.h.smithi170.stdout:Mar 11 19:24:32 smithi170 systemd[1]: Stopping Ceph mds.h for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:24:32.596 INFO:journalctl@ceph.mds.h.smithi170.stdout:Mar 11 19:24:32 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-h[66069]: 2024-03-11T19:24:32.580+0000 7f2664dd3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:24:32.597 INFO:journalctl@ceph.mds.h.smithi170.stdout:Mar 11 19:24:32 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-h[66069]: 2024-03-11T19:24:32.580+0000 7f2664dd3700 -1 mds.h *** got signal Terminated *** 2024-03-11T19:24:35.728 INFO:journalctl@ceph.mds.h.smithi170.stdout:Mar 11 19:24:35 smithi170 podman[72324]: 2024-03-11 19:24:35.38361922 +0000 UTC m=+2.896130514 container died 9a47d71e1dc3db0db8cdbd2d4dd82c350188c83d4ed3a170d45b1a06f24e43d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-h, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-11T19:24:36.010 INFO:journalctl@ceph.mds.h.smithi170.stdout:Mar 11 19:24:35 smithi170 podman[72324]: 2024-03-11 19:24:35.806296683 +0000 UTC m=+3.318807985 container cleanup 9a47d71e1dc3db0db8cdbd2d4dd82c350188c83d4ed3a170d45b1a06f24e43d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-h, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:24:36.011 INFO:journalctl@ceph.mds.h.smithi170.stdout:Mar 11 19:24:35 smithi170 bash[72324]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-h 2024-03-11T19:24:36.234 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.h.service' 2024-03-11T19:24:36.270 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:24:36.270 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-11T19:24:36.270 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-11T19:24:36.271 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.k 2024-03-11T19:24:36.580 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:24:36 smithi170 systemd[1]: Stopping Ceph mds.k for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:24:36.846 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:24:36 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-k[65444]: 2024-03-11T19:24:36.575+0000 7fb632266700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:24:36.846 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:24:36 smithi170 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-k[65444]: 2024-03-11T19:24:36.575+0000 7fb632266700 -1 mds.k *** got signal Terminated *** 2024-03-11T19:24:39.647 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:24:39 smithi170 podman[72444]: 2024-03-11 19:24:39.39153137 +0000 UTC m=+2.899630289 container died f35ef38595e55d38d2f499a2468eac9713d8d4aa79842c18933cce171b57add0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-k, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-11T19:24:39.902 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:24:39 smithi170 podman[72444]: 2024-03-11 19:24:39.718218491 +0000 UTC m=+3.226317466 container cleanup f35ef38595e55d38d2f499a2468eac9713d8d4aa79842c18933cce171b57add0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-k, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-11T19:24:39.902 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:24:39 smithi170 bash[72444]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-k 2024-03-11T19:24:40.133 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.k.service' 2024-03-11T19:24:40.156 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:24:39 smithi170 podman[72458]: 2024-03-11 19:24:39.899434257 +0000 UTC m=+0.500995503 container remove f35ef38595e55d38d2f499a2468eac9713d8d4aa79842c18933cce171b57add0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-k, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-11T19:24:40.156 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:24:40 smithi170 systemd[1]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.k.service: Deactivated successfully. 2024-03-11T19:24:40.156 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:24:40 smithi170 systemd[1]: Stopped Ceph mds.k for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:24:40.156 INFO:journalctl@ceph.mds.k.smithi170.stdout:Mar 11 19:24:40 smithi170 systemd[1]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.k.service: Consumed 2.102s CPU time. 2024-03-11T19:24:40.167 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:24:40.167 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-11T19:24:40.167 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-11T19:24:40.167 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.c 2024-03-11T19:24:40.457 INFO:journalctl@ceph.mds.c.smithi190.stdout:Mar 11 19:24:40 smithi190 systemd[1]: Stopping Ceph mds.c for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:24:40.763 INFO:journalctl@ceph.mds.c.smithi190.stdout:Mar 11 19:24:40 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-c[66697]: 2024-03-11T19:24:40.452+0000 7fa140c1e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:24:40.763 INFO:journalctl@ceph.mds.c.smithi190.stdout:Mar 11 19:24:40 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-c[66697]: 2024-03-11T19:24:40.452+0000 7fa140c1e700 -1 mds.c *** got signal Terminated *** 2024-03-11T19:24:43.762 INFO:journalctl@ceph.mds.c.smithi190.stdout:Mar 11 19:24:43 smithi190 podman[72713]: 2024-03-11 19:24:43.395197042 +0000 UTC m=+3.042103960 container died 656094af5d2608fc1c96ddd144fe5a6dd814113f1cdc80d15e8656c900f486c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-c, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-11T19:24:44.248 INFO:journalctl@ceph.mds.c.smithi190.stdout:Mar 11 19:24:43 smithi190 podman[72713]: 2024-03-11 19:24:43.9859026 +0000 UTC m=+3.632809518 container cleanup 656094af5d2608fc1c96ddd144fe5a6dd814113f1cdc80d15e8656c900f486c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-94a5397) 2024-03-11T19:24:44.249 INFO:journalctl@ceph.mds.c.smithi190.stdout:Mar 11 19:24:43 smithi190 bash[72713]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-c 2024-03-11T19:24:44.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.c.service' 2024-03-11T19:24:44.444 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:24:44.445 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-11T19:24:44.445 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-11T19:24:44.445 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.f 2024-03-11T19:24:44.763 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:24:44 smithi190 systemd[1]: Stopping Ceph mds.f for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:24:45.262 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:24:44 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-f[66392]: 2024-03-11T19:24:44.815+0000 7f0f6c8e3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:24:45.263 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:24:44 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-f[66392]: 2024-03-11T19:24:44.815+0000 7f0f6c8e3700 -1 mds.f *** got signal Terminated *** 2024-03-11T19:24:47.660 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:24:47 smithi190 podman[72845]: 2024-03-11 19:24:47.385979159 +0000 UTC m=+2.730946603 container died 5cb8dff6283739da1dfc1043fe1fce1d0ec129a63ef9ba3f6c81e589e2933e57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-f, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-11T19:24:48.002 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:24:47 smithi190 podman[72845]: 2024-03-11 19:24:47.7564086 +0000 UTC m=+3.101376059 container cleanup 5cb8dff6283739da1dfc1043fe1fce1d0ec129a63ef9ba3f6c81e589e2933e57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-f, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.license=GPLv2) 2024-03-11T19:24:48.002 INFO:journalctl@ceph.mds.f.smithi190.stdout:Mar 11 19:24:47 smithi190 bash[72845]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-f 2024-03-11T19:24:48.123 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.f.service' 2024-03-11T19:24:48.157 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:24:48.158 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-11T19:24:48.158 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-11T19:24:48.158 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.i 2024-03-11T19:24:48.263 INFO:journalctl@ceph.mds.i.smithi190.stdout:Mar 11 19:24:48 smithi190 systemd[1]: Stopping Ceph mds.i for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:24:48.762 INFO:journalctl@ceph.mds.i.smithi190.stdout:Mar 11 19:24:48 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-i[67005]: 2024-03-11T19:24:48.467+0000 7fc04571f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:24:48.763 INFO:journalctl@ceph.mds.i.smithi190.stdout:Mar 11 19:24:48 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-i[67005]: 2024-03-11T19:24:48.467+0000 7fc04571f700 -1 mds.i *** got signal Terminated *** 2024-03-11T19:24:51.763 INFO:journalctl@ceph.mds.i.smithi190.stdout:Mar 11 19:24:51 smithi190 podman[72969]: 2024-03-11 19:24:51.398519601 +0000 UTC m=+3.029988234 container died b94348ec9f1e6eb0654d18f3864c99b7842b250c5af33978f3fb99f18ef4ba87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True) 2024-03-11T19:24:52.231 INFO:journalctl@ceph.mds.i.smithi190.stdout:Mar 11 19:24:52 smithi190 podman[72969]: 2024-03-11 19:24:52.001504472 +0000 UTC m=+3.632973149 container cleanup b94348ec9f1e6eb0654d18f3864c99b7842b250c5af33978f3fb99f18ef4ba87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-i, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-11T19:24:52.231 INFO:journalctl@ceph.mds.i.smithi190.stdout:Mar 11 19:24:52 smithi190 bash[72969]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-i 2024-03-11T19:24:52.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.i.service' 2024-03-11T19:24:52.404 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:24:52.404 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-11T19:24:52.404 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-11T19:24:52.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.l 2024-03-11T19:24:52.763 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:24:52 smithi190 systemd[1]: Stopping Ceph mds.l for 638915c2-dfd9-11ee-95c8-87774f69a715... 2024-03-11T19:24:52.763 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:24:52 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-l[66073]: 2024-03-11T19:24:52.730+0000 7faf424aa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:24:52.763 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:24:52 smithi190 ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-l[66073]: 2024-03-11T19:24:52.730+0000 7faf424aa700 -1 mds.l *** got signal Terminated *** 2024-03-11T19:24:55.750 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:24:55 smithi190 podman[73090]: 2024-03-11 19:24:55.384932797 +0000 UTC m=+2.757449705 container died abe8b2b6897f4542c598c1fa04b6a91671964b15b1fd1446e292bdbb060f23a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-l, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-11T19:24:56.013 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:24:55 smithi190 podman[73090]: 2024-03-11 19:24:55.813314137 +0000 UTC m=+3.185831046 container cleanup abe8b2b6897f4542c598c1fa04b6a91671964b15b1fd1446e292bdbb060f23a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-l, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-11T19:24:56.013 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:24:55 smithi190 bash[73090]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-l 2024-03-11T19:24:56.292 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:24:56 smithi190 podman[73104]: 2024-03-11 19:24:56.039294196 +0000 UTC m=+0.650106649 container remove abe8b2b6897f4542c598c1fa04b6a91671964b15b1fd1446e292bdbb060f23a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-638915c2-dfd9-11ee-95c8-87774f69a715-mds-l, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-03-11T19:24:56.293 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:24:56 smithi190 systemd[1]: ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.l.service: Deactivated successfully. 2024-03-11T19:24:56.293 INFO:journalctl@ceph.mds.l.smithi190.stdout:Mar 11 19:24:56 smithi190 systemd[1]: Stopped Ceph mds.l for 638915c2-dfd9-11ee-95c8-87774f69a715. 2024-03-11T19:24:56.294 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-638915c2-dfd9-11ee-95c8-87774f69a715@mds.l.service' 2024-03-11T19:24:56.328 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T19:24:56.329 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-11T19:24:56.329 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T19:24:56.527 INFO:teuthology.orchestra.run.smithi029.stdout:Deleting cluster with fsid: 638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T19:24:59.940 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T19:25:00.130 INFO:teuthology.orchestra.run.smithi170.stdout:Deleting cluster with fsid: 638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T19:25:02.776 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T19:25:02.965 INFO:teuthology.orchestra.run.smithi190.stdout:Deleting cluster with fsid: 638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T19:25:05.263 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T19:25:05.298 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T19:25:05.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T19:25:05.362 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-11T19:25:05.365 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591250/remote/smithi029/crash 2024-03-11T19:25:05.366 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/crash -- . 2024-03-11T19:25:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:tar: /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T19:25:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:tar: Error is not recoverable: exiting now 2024-03-11T19:25:05.412 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591250/remote/smithi170/crash 2024-03-11T19:25:05.413 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/crash -- . 2024-03-11T19:25:05.491 INFO:teuthology.orchestra.run.smithi170.stderr:tar: /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T19:25:05.491 INFO:teuthology.orchestra.run.smithi170.stderr:tar: Error is not recoverable: exiting now 2024-03-11T19:25:05.493 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591250/remote/smithi190/crash 2024-03-11T19:25:05.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/crash -- . 2024-03-11T19:25:05.540 INFO:teuthology.orchestra.run.smithi190.stderr:tar: /var/lib/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T19:25:05.540 INFO:teuthology.orchestra.run.smithi190.stderr:tar: Error is not recoverable: exiting now 2024-03-11T19:25:05.541 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-11T19:25:05.541 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-11T19:25:05.653 INFO:teuthology.orchestra.run.smithi029.stdout:2024-03-11T19:01:44.792176+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T19:25:05.653 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-11T19:25:05.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[SEC\]' /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-11T19:25:05.687 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]' /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-11T19:25:05.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[WRN\]' /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-11T19:25:05.834 INFO:teuthology.orchestra.run.smithi029.stdout:2024-03-11T19:01:44.792176+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T19:25:05.834 INFO:tasks.cephadm:Compressing logs... 2024-03-11T19:25:05.835 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T19:25:05.877 DEBUG:teuthology.orchestra.run.smithi170:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T19:25:05.879 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T19:25:05.924 INFO:teuthology.orchestra.run.smithi029.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T19:25:05.924 INFO:teuthology.orchestra.run.smithi029.stderr:: No such file or directory 2024-03-11T19:25:05.927 INFO:teuthology.orchestra.run.smithi170.stderr:find: ‘/var/log/rbd-target-api’gzip -5: No such file or directory --verbose 2024-03-11T19:25:05.928 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/cephadm.log 2024-03-11T19:25:05.928 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.10.log 2024-03-11T19:25:05.928 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T19:25:05.928 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.b.log 2024-03-11T19:25:05.928 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.4.log 2024-03-11T19:25:05.928 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-11T19:25:05.928 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.b.log: /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.4.log: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T19:25:05.928 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.k.log 2024-03-11T19:25:05.928 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.h.log 2024-03-11T19:25:05.928 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.7.log 2024-03-11T19:25:05.928 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.e.log 2024-03-11T19:25:05.928 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.7.log: 90.2% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T19:25:05.929 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-11T19:25:05.931 INFO:teuthology.orchestra.run.smithi190.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T19:25:05.931 INFO:teuthology.orchestra.run.smithi190.stderr:: No such file or directory 2024-03-11T19:25:05.931 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.l.log 2024-03-11T19:25:05.931 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T19:25:05.931 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.i.log 2024-03-11T19:25:05.931 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.11.log 2024-03-11T19:25:05.931 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mgr.z.log 2024-03-11T19:25:05.931 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.i.log: /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.11.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T19:25:05.931 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.8.log 2024-03-11T19:25:05.931 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.5.log 2024-03-11T19:25:05.931 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.f.log 2024-03-11T19:25:05.932 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T19:25:05.932 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.f.log: 90.2% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T19:25:05.932 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.log 2024-03-11T19:25:05.932 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.cephadm.log: 82.7% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T19:25:05.932 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-11T19:25:05.932 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.log: 87.0% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T19:25:05.932 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.c.log 2024-03-11T19:25:05.932 INFO:teuthology.orchestra.run.smithi190.stderr: 92.9%/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.2.log: -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mgr.z.log.gz 2024-03-11T19:25:05.932 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T19:25:05.936 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-11T19:25:05.936 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T19:25:05.936 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-03-11T19:25:05.936 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.6.log 2024-03-11T19:25:05.936 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-11T19:25:05.936 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T19:25:05.936 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.g.log 2024-03-11T19:25:05.937 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.j.log 2024-03-11T19:25:05.937 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.3.log 2024-03-11T19:25:05.937 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-11T19:25:05.937 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-11T19:25:05.937 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T19:25:05.937 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.log 2024-03-11T19:25:05.937 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T19:25:05.937 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.9.log 2024-03-11T19:25:05.937 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.d.log 2024-03-11T19:25:05.937 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.a.log 2024-03-11T19:25:05.937 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.d.log: /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.a.log: 87.0% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T19:25:05.940 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mgr.y.log: 92.9% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-11T19:25:05.946 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.log 2024-03-11T19:25:05.947 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.cephadm.log: 82.7% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T19:25:05.955 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T19:25:05.961 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T19:25:05.969 INFO:teuthology.orchestra.run.smithi190.stderr: 93.4% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.i.log.gz 2024-03-11T19:25:05.970 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-11T19:25:05.971 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.f.log.gz 2024-03-11T19:25:05.971 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-volume.log: 87.0% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T19:25:05.974 INFO:teuthology.orchestra.run.smithi029.stderr: 94.6% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.g.log.gz 2024-03-11T19:25:05.975 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.c.log.gz 2024-03-11T19:25:05.976 INFO:teuthology.orchestra.run.smithi190.stderr: 94.6% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.l.log.gz 2024-03-11T19:25:05.977 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.h.log.gz 2024-03-11T19:25:05.978 INFO:teuthology.orchestra.run.smithi029.stderr: 94.6% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.a.log.gz 2024-03-11T19:25:05.982 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mon.b.log: 94.6% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.k.log.gz 2024-03-11T19:25:05.984 INFO:teuthology.orchestra.run.smithi170.stderr: 91.0% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T19:25:06.017 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.e.log.gz 2024-03-11T19:25:06.023 INFO:teuthology.orchestra.run.smithi029.stderr: 82.3% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T19:25:06.202 INFO:teuthology.orchestra.run.smithi029.stderr: 94.7% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.d.log.gz 2024-03-11T19:25:06.284 INFO:teuthology.orchestra.run.smithi170.stderr: 93.0% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.b.log.gz 2024-03-11T19:25:06.286 INFO:teuthology.orchestra.run.smithi029.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T19:25:06.323 INFO:teuthology.orchestra.run.smithi029.stderr: 94.7% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mds.j.log.gz 2024-03-11T19:25:06.382 INFO:teuthology.orchestra.run.smithi029.stderr: 90.1% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T19:25:06.409 INFO:teuthology.orchestra.run.smithi170.stderr: 92.2% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-11T19:25:06.475 INFO:teuthology.orchestra.run.smithi190.stderr: 92.0% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-11T19:25:06.538 INFO:teuthology.orchestra.run.smithi029.stderr: 91.1% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T19:25:09.423 INFO:teuthology.orchestra.run.smithi029.stderr: 90.2% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-11T19:25:25.499 INFO:teuthology.orchestra.run.smithi029.stderr: 90.4% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-11T19:25:37.734 INFO:teuthology.orchestra.run.smithi029.stderr: 90.9% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.9.log.gz 2024-03-11T19:25:40.253 INFO:teuthology.orchestra.run.smithi170.stderr: 91.9% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.4.log.gz 2024-03-11T19:25:41.174 INFO:teuthology.orchestra.run.smithi190.stderr: 90.8% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.8.log.gz 2024-03-11T19:25:45.282 INFO:teuthology.orchestra.run.smithi190.stderr: 91.1% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-11T19:25:48.412 INFO:teuthology.orchestra.run.smithi170.stderr: 90.9% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.10.log.gz 2024-03-11T19:25:48.770 INFO:teuthology.orchestra.run.smithi190.stderr: 91.1% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.5.log.gz 2024-03-11T19:25:50.183 INFO:teuthology.orchestra.run.smithi170.stderr: 90.6% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.7.log.gz 2024-03-11T19:25:53.853 INFO:teuthology.orchestra.run.smithi170.stderr: 90.7% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-11T19:25:53.855 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-11T19:25:53.855 INFO:teuthology.orchestra.run.smithi170.stderr:real 0m47.960s 2024-03-11T19:25:53.855 INFO:teuthology.orchestra.run.smithi170.stderr:user 2m44.404s 2024-03-11T19:25:53.855 INFO:teuthology.orchestra.run.smithi170.stderr:sys 0m5.380s 2024-03-11T19:25:54.015 INFO:teuthology.orchestra.run.smithi190.stderr: 90.7% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.11.log.gz 2024-03-11T19:25:54.017 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-11T19:25:54.017 INFO:teuthology.orchestra.run.smithi190.stderr:real 0m48.123s 2024-03-11T19:25:54.017 INFO:teuthology.orchestra.run.smithi190.stderr:user 2m40.891s 2024-03-11T19:25:54.017 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m5.164s 2024-03-11T19:29:53.119 INFO:teuthology.orchestra.run.smithi029.stderr: 90.7% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-11T19:32:01.312 INFO:teuthology.orchestra.run.smithi029.stderr: 90.7% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.6.log.gz 2024-03-11T19:32:59.751 INFO:teuthology.orchestra.run.smithi029.stderr: 90.7% -- replaced with /var/log/ceph/638915c2-dfd9-11ee-95c8-87774f69a715/ceph-osd.3.log.gz 2024-03-11T19:35:19.870 INFO:teuthology.orchestra.run.smithi029.stderr: 94.3% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-03-11T19:35:19.872 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-03-11T19:35:19.872 INFO:teuthology.orchestra.run.smithi029.stderr:real 10m13.983s 2024-03-11T19:35:19.872 INFO:teuthology.orchestra.run.smithi029.stderr:user 5m29.584s 2024-03-11T19:35:19.872 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m14.307s 2024-03-11T19:35:19.873 INFO:tasks.cephadm:Archiving logs... 2024-03-11T19:35:19.873 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591250/remote/smithi029/log 2024-03-11T19:35:19.874 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T19:36:10.620 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591250/remote/smithi170/log 2024-03-11T19:36:10.621 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T19:36:40.725 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591250/remote/smithi190/log 2024-03-11T19:36:40.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T19:37:10.599 INFO:tasks.cephadm:Removing cluster... 2024-03-11T19:37:10.600 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 --force 2024-03-11T19:37:10.797 INFO:teuthology.orchestra.run.smithi029.stdout:Deleting cluster with fsid: 638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T19:37:11.279 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 --force 2024-03-11T19:37:11.471 INFO:teuthology.orchestra.run.smithi170.stdout:Deleting cluster with fsid: 638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T19:37:12.036 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 638915c2-dfd9-11ee-95c8-87774f69a715 --force 2024-03-11T19:37:12.227 INFO:teuthology.orchestra.run.smithi190.stdout:Deleting cluster with fsid: 638915c2-dfd9-11ee-95c8-87774f69a715 2024-03-11T19:37:12.767 INFO:tasks.cephadm:Removing cephadm ... 2024-03-11T19:37:12.767 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T19:37:12.784 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T19:37:12.804 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T19:37:12.821 INFO:tasks.cephadm:Teardown complete 2024-03-11T19:37:12.822 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-11T19:37:12.845 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-11T19:37:12.846 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T19:37:12.850 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T19:37:12.857 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T19:37:12.975 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-11T19:37:12.987 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-11T19:37:12.988 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T19:37:12.991 DEBUG:teuthology.orchestra.run.smithi170:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T19:37:12.994 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T19:37:13.002 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-03-11T19:37:13.005 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-03-11T19:37:13.006 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T19:37:13.006 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-03-11T19:37:13.006 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 +1102us[+1102us] +/- 131ms 2024-03-11T19:37:13.006 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -2172us[-2203us] +/- 63ms 2024-03-11T19:37:13.006 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +1895us[+1864us] +/- 67ms 2024-03-11T19:37:13.006 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T19:37:13.008 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-03-11T19:37:13.009 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T19:37:13.009 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-03-11T19:37:13.009 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 +1057us[+1057us] +/- 131ms 2024-03-11T19:37:13.009 INFO:teuthology.orchestra.run.smithi170.stdout:^* hv02.front.sepia.ceph.com 3 6 377 35 -2229us[-2212us] +/- 63ms 2024-03-11T19:37:13.009 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +1823us[+1840us] +/- 67ms 2024-03-11T19:37:13.009 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T19:37:13.011 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T19:37:13.011 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-03-11T19:37:13.011 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 +1027us[+1027us] +/- 131ms 2024-03-11T19:37:13.011 INFO:teuthology.orchestra.run.smithi190.stdout:^* hv02.front.sepia.ceph.com 3 6 377 32 -2235us[-2226us] +/- 63ms 2024-03-11T19:37:13.012 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 +1811us[+1821us] +/- 67ms 2024-03-11T19:37:13.012 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T19:37:13.012 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-11T19:37:13.023 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-11T19:37:13.023 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-11T19:37:13.034 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T19:37:13.248 DEBUG:teuthology.orchestra.run.smithi170:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T19:37:13.318 DEBUG:teuthology.orchestra.run.smithi190:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T19:37:13.390 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T19:37:13.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T19:37:13.421 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T19:37:13.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T19:37:13.450 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T19:37:13.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T19:37:13.480 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-11T19:37:13.493 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-11T19:37:13.505 INFO:teuthology.task.internal:Duration was 2702.610191 seconds 2024-03-11T19:37:13.506 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-11T19:37:13.516 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-11T19:37:13.517 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T19:37:13.519 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T19:37:13.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T19:37:13.563 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T19:37:13.565 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T19:37:13.576 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T19:37:13.975 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-11T19:37:13.975 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2024-03-11T19:37:13.976 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T19:37:13.995 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi170.front.sepia.ceph.com 2024-03-11T19:37:13.996 DEBUG:teuthology.orchestra.run.smithi170:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T19:37:14.019 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2024-03-11T19:37:14.020 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T19:37:14.040 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-11T19:37:14.041 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T19:37:14.043 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T19:37:14.063 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T19:37:14.109 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-11T19:37:14.109 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T19:37:14.112 DEBUG:teuthology.orchestra.run.smithi170:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T19:37:14.132 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T19:37:14.515 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-11T19:37:14.526 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-11T19:37:14.526 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T19:37:14.560 DEBUG:teuthology.orchestra.run.smithi170:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T19:37:14.588 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T19:37:14.617 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-11T19:37:14.629 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T19:37:14.632 DEBUG:teuthology.orchestra.run.smithi170:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T19:37:14.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T19:37:14.657 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = core 2024-03-11T19:37:14.662 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2024-03-11T19:37:14.683 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2024-03-11T19:37:14.708 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T19:37:14.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T19:37:14.741 DEBUG:teuthology.orchestra.run.smithi170:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T19:37:14.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T19:37:14.755 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T19:37:14.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T19:37:14.769 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-11T19:37:14.780 INFO:teuthology.task.internal:Transferring archived files... 2024-03-11T19:37:14.781 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591250/remote/smithi029 2024-03-11T19:37:14.781 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T19:37:14.826 DEBUG:teuthology.misc:Transferring archived files from smithi170:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591250/remote/smithi170 2024-03-11T19:37:14.826 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T19:37:14.865 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591250/remote/smithi190 2024-03-11T19:37:14.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T19:37:14.904 INFO:teuthology.task.internal:Removing archive directory... 2024-03-11T19:37:14.904 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T19:37:14.906 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T19:37:14.908 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T19:37:14.954 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-11T19:37:14.967 INFO:teuthology.task.internal:Not uploading archives. 2024-03-11T19:37:14.967 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-11T19:37:14.980 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-11T19:37:14.980 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T19:37:14.983 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T19:37:14.986 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T19:37:14.994 INFO:teuthology.orchestra.run.smithi029.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 19:37 /home/ubuntu/cephtest 2024-03-11T19:37:14.998 INFO:teuthology.orchestra.run.smithi170.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 19:37 /home/ubuntu/cephtest 2024-03-11T19:37:15.009 INFO:teuthology.orchestra.run.smithi190.stdout: 265716 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 19:37 /home/ubuntu/cephtest 2024-03-11T19:37:15.010 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-11T19:37:15.024 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-11T19:37:15.048 INFO:teuthology.nuke:Checking targets against current locks 2024-03-11T19:37:15.074 DEBUG:teuthology.nuke:shortname: smithi029 2024-03-11T19:37:15.074 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T19:37:15.099 DEBUG:teuthology.nuke:shortname: smithi170 2024-03-11T19:37:15.099 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T19:37:15.104 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591250', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 18:41:46.997298', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGYZilMnOnggr1Y/gS2ZXlbZcuw//vXJ/iU1NMGtqA6Wd8bQYlifRUH//C0wNj5O5nuN56O4JOfysQGUkthuPV8='} 2024-03-11T19:37:15.125 DEBUG:teuthology.nuke:shortname: smithi190 2024-03-11T19:37:15.126 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T19:37:15.130 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591250', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 18:41:46.995827', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBETp9xZZU8hpJWpm3QSjIWV/gEDDHQ3IfwEvgofjAyO0IozeUhdwLNWbcmTLw4cRsz0hmpFImxcBetisVFs1014='} 2024-03-11T19:37:15.159 INFO:teuthology.orchestra.console.smithi029:Power off 2024-03-11T19:37:15.159 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T19:37:15.173 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591250', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 18:41:46.998544', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMS0O7LVYEEcLwSKh+rOHokIgBjlbuuLyRFdpoBkfP/KLb72O3NsTW8FVvwqQN0Yky+HYVXQZ/8iaJEUUrEva28='} 2024-03-11T19:37:15.186 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off 2024-03-11T19:37:15.186 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T19:37:15.203 INFO:teuthology.orchestra.console.smithi170:Power off 2024-03-11T19:37:15.203 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T19:37:15.216 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-03-11T19:37:15.226 DEBUG:teuthology.orchestra.console.smithi170:power off output: Chassis Power Control: Down/Off 2024-03-11T19:37:15.226 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T19:37:15.247 INFO:teuthology.orchestra.console.smithi190:Power off 2024-03-11T19:37:15.248 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T19:37:15.259 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2024-03-11T19:37:15.270 DEBUG:teuthology.orchestra.console.smithi190:power off output: Chassis Power Control: Down/Off 2024-03-11T19:37:15.271 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T19:37:15.294 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2024-03-11T19:37:19.217 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T19:37:19.260 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T19:37:19.296 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T19:37:19.328 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-03-11T19:37:19.370 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2024-03-11T19:37:19.406 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2024-03-11T19:37:23.330 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T19:37:23.371 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T19:37:23.407 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T19:37:23.440 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2024-03-11T19:37:23.482 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off 2024-03-11T19:37:23.517 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is off 2024-03-11T19:37:23.542 INFO:teuthology.orchestra.console.smithi029:Power off completed 2024-03-11T19:37:23.583 INFO:teuthology.orchestra.console.smithi170:Power off completed 2024-03-11T19:37:23.618 INFO:teuthology.orchestra.console.smithi190:Power off completed 2024-03-11T19:37:23.723 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} duration: 2702.610190629959 failure_reason: '"2024-03-11T19:01:44.792176+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-11T19:37:23.723 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T19:37:23.802 INFO:teuthology.run:FAIL